Visualizing and Interacting with Diamond
Jason I. Hong
March 14 2006
Overview
• Interaction techniques for Diamond• Integrated Diamond Environment• Metrics for evaluating Diamond UIs
• None of what I will talk about has been built• Lots of feedback will be useful
Information Visualization
• Problem: How to view lots of data simultaneously?• Traditionally done through information visualization
0
200
400
600
800
1000
1200
A M J J A S O N D J F M
Prior 12 months
Latest 12 months
Information Visualization
• Problem: How to view lots of data simultaneously?• Traditionally done through information visualization
Idea #1: Summarize as Metadata
• Problem: How to view lots of data simultaneously?• Traditionally done through information visualization
– These kinds of views good for seeing trends and relations
– Views are also interactive
• Observation: Showing pictures (ie search results) makes it hard to see any trends and relations
• Idea: Diamond should focus first on showing metadata rather than pictures
Example: Stanford Photo Browser
Example: Stanford Photo Browser
Example: Stanford Photo Browser
Idea #1: Summarize as MetadataDiscussion
• Faceted metadata– Can explore categories in any order
– Ex. Location then time, or time then weather then location
• Provides feedforward (like feedback)– United States (1822)
– Gives an idea if drilling down into a category is a good idea
• So how might this work for Diamond?
Idea #1: Summarize as MetadataRough Interface Mockup #1
People0 people (2135…)1 person (1029…)2 people (613…)3 people (323…)4+ people (705…)
Cars0 cars (1036)1 car (1664)2 cars (1324)3 cars (1156)4+ cars (1422)
GrassNo grass (1603)Some grass (1400)Lots of grass (1311)
Search Filters
Search Results
Photo Metadata
Idea #1: Summarize as MetadataRough Interface Mockup #1
People0 people (1635)1 person (829)2 people (213)3 people (123)4+ people (305)
Cars (all > 1 car)
GrassNo grass (703)Some grass (505)Lots of grass (611)
Photo Metadata Search Filters
Search Results (1664 results)
1 Car (1664 results)
Idea #1: Summarize as MetadataRough Interface Mockup #1
People (all > 1 person)
Cars (all > 1 car)
GrassNo grass (703)Some grass (505)Lots of grass (611)
Photo Metadata Search Filters
Search Results (829 results)
1 Car (1664 results)
1 Person (829 results)
Idea #1: Summarize as MetadataRough Interface Mockup #2
People
Cars
Grass
Search Results Search Filters
28,438 photos
Server 1
53,129 photos
Server 2
10,298 photos
Server 3
Add Filter
This side gives an overview of how many results there are, where pictures are coming from
Two-headed scrollbars let you easily select a range of values rather than just a single one (as in previous UI)
Idea #1: Summarize as MetadataRough Interface Mockup #2
People
Cars
Grass
Search Results Search Filters
Server 1
Server 2
Server 3
Add Filter
28,438 photos
53,129 photos
10,298 photos
1,352 photos
Idea #1: Summarize as MetadataDiscussion
• Metadata approach good for exploratory tasks– When you want to get a feel for the data set
– Probably suboptimal for confirmatory tasks
• Digression: Exploratory vs Confirmatory tasks– Exploratory more about understanding the data
• Let the data suggest interesting questions• Interactive overviews useful here
– Confirmatory more about validating a hypothesis• Use data to prove something true or false
– Diamond needs to support both kinds of tasks
Idea #1: Summarize as MetadataDiscussion
• Metadata approach good for stovepiping– Some organizations don’t want to or can’t share all info
– Legal or market reasons
– Metadata approach a way of sharing enough useful info without giving everything away
– Ex. FBI, CIA, NSA, let them explore each other’s metadata, and if probable cause, then examine relevant pictures (and then log it for accountability)
Idea #1: Summarize as MetadataDiscussion
• Issue: how to get good labels for metadata?– Semantic gap between people and vision algorithms
– Each filter can be a label, but not often useful• RGB values not useful as metadata here
• Issue: confidence intervals for metadata?– Ex. 90% sure there are 3 people in this image
Idea #2: Scatter / Gather for Clustering
• Problem: Lots of ways of ways of organizing pictures, how to do so in a flexible and meaningful way?
• Example: 2000 books on European History– Could organize by author, by nation, by era, political, etc
– But hard to do this in advance if you aren’t sure what categories are going to be useful
• Scatter / Gather developed at PARC for text retrieval– System scatters documents into a number of clusters
– User can gather clusters together
– System re-scatters
Idea #2: Scatter / Gather for Clustering
• Retrieve top 250 documents containing “star”– From an encyclopedia
Idea #2: Scatter / Gather for Clustering
Idea #2: Scatter / Gather for ClusteringNo semantics a priori, just some clustering algorithms to
group things by similarity
Idea #2: Scatter / Gather for Clustering
Idea #2: Scatter / Gather for Clustering
• Can we apply Scatter / Gather to Diamond?– Easier to cluster documents (vector of words) than images
• Text is simple, images have “hidden” content
– Scale and performance also issues
• Currently unclear how effective it would be– Open research question
– Need lots of help here from vision and learning people
Idea #2: Scatter / Gather for Clustering
• Digression: Different kinds of tools seem more useful for different stages of use in Diamond
• Opening– Coarse-grained, cull out obvious misses
– Exploration and metadata overview
– Query specification
• Midgame– Refine and drill-down
– Scatter/Gather
• Endgame– Fine-grained
Idea #3: Why and Why Not Questions
• Problem: It can be confusing as to why a certain picture is shown (or not shown)
• Example: search for “faces”
Idea #3: Why and Why Not Questions
• Whyline– reduced novice programmers' debugging time ~7.8 and
helping them get 40% further through their tasks
Idea #3: Why and Why Not Questions
Idea #3: Why and Why Not Questions
• Possible alternative interface for relevance feedback– Relevance feedback often a black box
– ‘Why’ and ‘why not’ questions can expose more info for developing and tuning filters
• Why questions– Why do my filters say there is a face here?
– Why is this image in this category?
– Also, how to fix?
• Why not questions– Harder to ask, may not know image is missing
– More useful with known data set
Overview
• Interaction techniques for Diamond• Integrated Diamond Environment• Metrics for evaluating Diamond UIs
Idea #4: Integrated Diamond Environment
• Problem: How to develop new user interfaces?– Discussed lots of UI ideas
– Don’t want to custom build a new UI for each domain
– UI toolkits, but not quite right, lots of duplicated effort
• Alternative idea: An Integrated Diamond Environment– Like Integrated Developer Environment for programming
Idea #4: Integrated Diamond EnvironmentEclipse IDE
• A single UI for managing all tasks related to development
• Comes with large set of tools• Supports plug-ins for new tools• End-users can easily re-arrange tools (windows) to fit their needs
Idea #4: Integrated Diamond Environment
• Provides standard tools for manipulating data sets• Advantages
– Better re-use of code– Fosters a developer community (network effects)– Foster more deployment of Diamond?
• Setting up a UI for new domain would consist of:– setting up the image filters– setting up the windows– creating useful macros
Idea #4: Integrated Diamond Environment
• Issue: Need a common way of integrating Diamond data flow and new tools
• Idea: Provide abstract notion of data set, plus belt of tools for viewing and manipulating
Current Tool
Tool
Tool
ToolData Sets
History
Idea #4: Integrated Diamond Environment
People0 people (2135…)1 person (1029…)2 people (613…)3 people (323…)4+ people (705…)
Cars0 cars (1036)1 car (1664)2 cars (1324)3 cars (1156)4+ cars (1422)
Scatter / GatherTool
Set ManipulationTool
Tool
Working Data Set (5326)
Metadata Overview
Start (5326)
Idea #4: Integrated Diamond Environment
People (all > 1 person)Cars
0 cars (1036)1 car (1664)2 cars (1324)3 cars (1156)4+ cars (1422)
Scatter / GatherTool
Set ManipulationTool
Tool
Working Data Set (1029)
Metadata Overview
Start (5326) 1 person (1029)
Idea #4: Integrated Diamond Environment
Yellow, Door, Grass
Face
MetadataOverview Tool
Set ManipulationTool
Tool
Working Data Set (1029)
Scatter / Gather Tool
Start (5326) 1 person (1029)
Idea #4: Integrated Diamond Environment
Yellow, Door, Grass
Face
MetadataOverview Tool
Set ManipulationTool
Tool
Pics with 1 Person (1029)
Scatter / Gather Tool
Start (5326)
Working Data Set (5326)
Idea #4: Integrated Diamond Environment
Yellow, Door, Grass
Face
MetadataOverview Tool
Set ManipulationTool
Tool
Pics with 1 Person (1029)
Scatter / Gather Tool
Start (5326)
Working Data Set (200)
Tool A (629) Tool B (200)
Idea #4: Integrated Diamond Environment
• Data Set– Abstract set of images + History of how it got there
• Tool– Manipulates current data set– Provides view of that data set– Developers create these
Idea #4: Integrated Diamond Environment
• Macros– Would hide some of the tools– Automate repetitive actions– Describe things in a domain-specific manner– UI designers and advanced end-users create these
• Collection of tools, loosely joined– Easy to add new tools w/o a lot of overhead– Easy to assemble tools into an ensemble– Like Unix philosophy
Idea #4: Integrated Diamond Environment
• Idea: Model-based user interfaces– Given an abstract task model, generate a high quality
user interface with domain-specific terminology
– Alternative to macros described earlier
– Task model might include things like:• Kinds of images interested in• Higher-level description of objects in images
– Lots of green near bottom of image is “grass”
• Workflow (first do A, then do B)
– Model-based UIs researched for 20+ years• Diamond might be constrained enough though
Idea #5: Metrics
• How to measure the effectiveness of Diamond UIs?– Developing a new Diamond UI
– Using a Diamond UI
Idea #5: MetricsMethod #1 – Standard HCI task-based approach
• Define standard tasks– Ex. 1000 pictures
– Warmup task, find any picture that has “1 car + 1 person”
– Medium task (1-2 pictures with answer), see if a tractor trailer ever passed through this gate
– Harder tasks
• Measures– Time on task
– #errors
– #tasks successfully completed
– Outside judges measure quality of work
– Subjective satisfaction
• This would be a standard way of evaluating a UI
Idea #5: MetricsMethod #1 – Standard HCI task-based approach
• Problem: Diamond intended for use by experts whose time is very valuable, making it hard to go this route– Hard to get lots of experts
– Hard to get statistically significant results
– Unclear what UI to compare against (only useful if there is another UI that does something like Diamond)
• Not clear to me how to address this problem– Usually we would try to find good approximate users,
ex. medical students instead of doctors
– Focus here likely to be qualitative rather than quantitative
Idea #5: MetricsMethod #2 – TREC-like Bakeoff
• TREC stands for Text REtrieval Conference
“Existing research groups (such as the Cornell SMART system) report a doubling in performance over the six years of TREC, whereas systems new to TREC typically double their performance in the first year as they move their techniques into current state-of-the-art.”
American Society for Information Science Bulletin, Apr/May 98
Idea #5: MetricsMethod #2 – TREC-like Bakeoff
• TREC worked because it targeted the generic problem of text retrieval first rather than specific problems– Initially focused on searching generic text corpora– Only recently has there been a genomics track, legal track,
blogs, video, terabyte storage, etc
• With Diamond, we are focusing on specific apps first– Pharmaceutical, Medical, Homeland security– Right from an interaction design standpoint, might not be
from an optimization standpoint
Idea #5: MetricsMethod #2 – TREC-like Bakeoff
• Idea: Push for a new track in TREC for large-scale image retrieval
• Idea: Focus on a domain that will have LOTS of users– Improvements in the general case will also help specific cases
• Idea: Provide a standard large corpora of images– Bettman archive, AP photo archive, web image search– Provide an open Diamond server for others– Foster competitions on top of Diamond– Open source code + “open source” data– Student could just download code and run it
Summary
• Interaction techniques for Diamond– Faceted metadata for overviews
– Scatter / Gather for refining
– Why and Why Not questions for debugging
• Integrated Diamond Environment– A framework for developing tools and UIs
• Metrics for evaluating Diamond UIs– Standard human-computer interaction evaluation
– TREC-like bakeoffs
Backup slides
Idea: Summarize as MetadataRough Interface Mockup
People0 people (2135)1 person (1029)2 people (613)3 people (323)4+ people (705)
Cars0 cars (1036)1 car (1664)2 cars (1324)3 cars (1156)4+ cars (1422)
GrassNo grass (1603)Some grass (1400)Lots of grass (1311)
Other Diamond Issues
• Sketch Retrieval– Draw what you want and retrieve
– Spatial as well
• Spatial retrieval– Car in corner
• Handling false negatives• Multiple people evaluating search results• Model-based user interfaces
– Given an abstract task model, generate a high quality user interface with domain-specific terminology
Idea: Other Domains for Diamond
• Digital Libraries• Web search