experiencing visual analytics from trenches & labs fileevaluation of a guided exploratory...
TRANSCRIPT
Evaluation of a Guided
Exploratory Visualization System:
a Mixed-Approach
INRIA, Université Paris-Sud, INRA
N. Boukhelifa, A. Bezerianos, E. Lutton
Examples of Guided EVS
2
BROWN E. T., LIU J., BRODLEY C. E., CHANG R.: Dis-function: learning distance functions interactively. In IEEE VAST (2012), IEEE Computer Society, pp. 83–92. 1
Examples of Guided EVS
3
SHAO L., BEHRISCH M., SCHRECK T., VON LAN- DESBERGER T., SCHERER M., BREMM S., KEIM D. A.: Guided sketching for visual search and exploration in large scat- ter plot spaces. In Proc. EuroVA International Workshop on Vi- sual Analytics (2014).
Examples of Guided EVS
4 N. Boukhelifa, W. Cancino, A. Bezerianos and E. Lutton. Evolutionary Visual Exploration: Evaluation With Expert Users. Computer Graphics Forum (EuroVis 2013, June 17--21, 2013, Leipzig, Germany), Eurographics Association, 2013, 32 (3).
5
6
7
8
9
10
How to explore combined dimensions?
11
12
How to explore a huge
search space?
Evolutionary Visual Exploration
13
Primordial Soup Evolved Species
NA
TU
RA
L E
VO
LU
TIO
N
Evolutionary Visual Exploration
14
Primordial Soup Evolved Species
NA
TU
RA
L E
VO
LU
TIO
N
n-D data set Interesting 2D projections EV
E
IEA
Evolutionary Visual Exploration
15
Primordial Soup Evolved Species
NA
TU
RA
L E
VO
LU
TIO
N
n-D data set Interesting 2D projections EV
E
IEA
Interactive Evolutionary
Algorithm
Evaluation of Projections
21
User assessment Complexity Surrogate function
Scagnostics, Wilkinson and Wills, 2008
Evaluation of Projections
22
User assessment Complexity Surrogate function
Scagnostics, Wilkinson and Wills, 2008
learned
Evaluation of EVE: Challenges
• exploratory nature of tasks: the user
does not know what they are looking for
• learning component: how accurate is
the user model and how informative is the
system feedback
23
A Mixed Approach
24
User Centered Evaluation
Algorithm Centered Evaluation
A Mixed Approach
25
User Centered Evaluation
Algorithm Centered Evaluation
VALIDATION
A Mixed Approach
26
User Centered Evaluation
Algorithm Centered Evaluation
VALIDATION
VERIFICATION
User-Centered Evaluation
Aim
• are experts able to confirm old knowledge?
• are experts able to gain new insight?
Qualitative study methodology
• think aloud, observe, interview & questionnaire
• videotaped and log data capture
27
• 5 domain experts
• mean 34.2 years
• own datasets
• pre-questionnaire
• 2.5 hours
Participants
28
Training
T1: show in the tool what you already know about the data
T2: explore the data in light of a research question
Tasks
29
Insight-based Evaluation
Results
30
Insight-based Evaluation
Results
31
EVE Results
Hypothesis Generation
32
“this combination
may be an
important finding
because it involves
parameters that
affect only one part
of the simulation
model ...”
City emergence model
EVE Results
Hypothesis Quantification
‘‘we always talk
about this
qualitatively. This is
the first time I see
concrete weights ...’’
33
Electricity consumption profiles
EVE Results
Experts are able to:
learn how to use our tool
reproduce known findings
generate new findings
34
Evaluation with Experts
High ecological validity, but:
– time consuming
– difficult to recruit experts
– we cannot share data
– results may not be replicable or
generalisable
36
Reproducibility of Results
Due to the observational study methodology / case studies with experts:
• Not possible to reproduce findings across subjects
• Can reproduce testing methodologies and coding of the analysis
• Reproducible specific tasks, e.g. for training.
37
A Mixed Approach
38
User Centered Evaluation
Algorithm Centered Evaluation
VERIFICATION
Algorithm-Centered Evaluation
Aim
• is the IEA able to steer the exploration toward
an interesting area of the search space?
• are the proposed solutions varied?
Quantitative study methodology
• synthetic dataset
• pre-specified task
39
• 12 participants
• mean 28.5 years
• no experience required
• Synthetic dataset 5D
• 20 minutes
Participants
40
Game
separate two point clusters
Task
41
IEA is able to follow the order of user ranking
Convergence
Analysis
42
Diverse solutions observed at the start of search and less towards the end
Diversity
Analysis
43
RVV Issues and EVE
Validation: are you building the right thing
(user-centered)
• studying how users learn and explore using
EVE
• usability issues: efficient interactions and
richer user feedback
44
RVV Issues and EVE
Verification: are you building it right?
(algorithm-centered)
• comparison of user ranking and predicted
system evaluation
• algorithmic issues: e.g. diversity and
premature convergence
45
RVV Issues and EVE
Reproducibility: are the evaluations
reproducible ?
• task description and test dataset (for training
task) are provided online.
46
RVV Issues and EVE
Reproducibility: are insights reproducible ?
• through a synthetic task, simple dataset +
example solutions (combined dimensions)
• other toy tasks, data sets ?
47
RVV Issues and EVE
Reproducibility: are insights reproducible ?
• more challenging, esp. for real data and tasks:
– stochastic behaviour of the underlying algorithm
– uncertainty in user goals
– reproducing the exploration path: provenance,
collaboration
– co-adaptive behaviour of the system
48
In Summary …
Reproducibility in guided / co-adaptive systems
What is the baseline for reproducibility ?
- indication of convergence?
Reproducibility use cases:
- success cases: insight vs. process
- failure cases: limitation of process
49
