1 shengdong zhao department of computer science university of toronto july 9, 2008 earpod:...
TRANSCRIPT
![Page 1: 1 Shengdong Zhao Department of Computer Science University of Toronto July 9, 2008 earPod: Efficient, Hierarchical, Eyes-free Menu Selection](https://reader035.vdocument.in/reader035/viewer/2022062717/56649e365503460f94b26051/html5/thumbnails/1.jpg)
1
Shengdong ZhaoDepartment of Computer Science
University of TorontoJuly 9, 2008
earPod: Efficient, Hierarchical, Eyes-free Menu Selection
![Page 2: 1 Shengdong Zhao Department of Computer Science University of Toronto July 9, 2008 earPod: Efficient, Hierarchical, Eyes-free Menu Selection](https://reader035.vdocument.in/reader035/viewer/2022062717/56649e365503460f94b26051/html5/thumbnails/2.jpg)
Outline• Research problem • Previous research• Research questions• Research strategy and focus• Prototype• Empirical studies• Contributions• Future work
2
![Page 3: 1 Shengdong Zhao Department of Computer Science University of Toronto July 9, 2008 earPod: Efficient, Hierarchical, Eyes-free Menu Selection](https://reader035.vdocument.in/reader035/viewer/2022062717/56649e365503460f94b26051/html5/thumbnails/3.jpg)
Research problems and objectives
• Motivating problem– How can users interact with information devices in mobile
environments when their visual attention is taken up by other tasks (e.g navigation through the environment)
• Objective– To develop interactive techniques that support user
control and feedback without relying on the visual modality
• Scope – This research focuses on menu selection because this is an
elemental task (Foley et al. ‘84)
3
![Page 4: 1 Shengdong Zhao Department of Computer Science University of Toronto July 9, 2008 earPod: Efficient, Hierarchical, Eyes-free Menu Selection](https://reader035.vdocument.in/reader035/viewer/2022062717/56649e365503460f94b26051/html5/thumbnails/4.jpg)
Previous research• Visual menus
– Linear menus (e.g., Sear and Schneiderman ’94; Cockburn and Gin ‘06, etc.)
– Radial menus (e.g., Callahan et al. ‘88; Kurtenbach ’93, etc.)
• Auditory applications– Accessibility for the blind (e.g., Edwards ‘89; Mynatt and Weber ‘94, etc.)
– Mobile auditory interfaces (e.g., Roy and Schmandt ‘96; Pirhonen et al. ’02, etc.)
• Auditory menus– Interactive Voice Response (IVR) systems (e.g., Marics and Engelbeck
1997; Resnick and Virzi ’92, etc.)
– Head gesture auditory pie menu (Brewster et al. ‘03)
4
![Page 5: 1 Shengdong Zhao Department of Computer Science University of Toronto July 9, 2008 earPod: Efficient, Hierarchical, Eyes-free Menu Selection](https://reader035.vdocument.in/reader035/viewer/2022062717/56649e365503460f94b26051/html5/thumbnails/5.jpg)
Research questions
• How to design usable eyes-free menus?– Designing interfaces with reasonable levels of
efficiency and ease of use – Facilitating a smooth novice to expert transition
• How to work around the problems with serial and temporal audio feedback?– Facilitating scan and compare of menu items– Providing users with a sense of control and
responsiveness5
![Page 6: 1 Shengdong Zhao Department of Computer Science University of Toronto July 9, 2008 earPod: Efficient, Hierarchical, Eyes-free Menu Selection](https://reader035.vdocument.in/reader035/viewer/2022062717/56649e365503460f94b26051/html5/thumbnails/6.jpg)
Research strategy
• Define research problem• Review literature• Define general design concept• Perform iteratively design and pilot studies for
prototype• Conduct empirical studies to explore
prototype properties • Derive design recommendations based on
empirical results6
![Page 7: 1 Shengdong Zhao Department of Computer Science University of Toronto July 9, 2008 earPod: Efficient, Hierarchical, Eyes-free Menu Selection](https://reader035.vdocument.in/reader035/viewer/2022062717/56649e365503460f94b26051/html5/thumbnails/7.jpg)
Selected input and output modalities
• Touch input– Combines relative and absolute pointing (Hinckley ‘01)
– Can be robustly operated in motion (Buxton et al. ‘85)
– Can be combined with other input devices (Hinckley and Sinclair ‘99; Rekimoto et al. ‘03)
• Auditory feedback– The other primary remote sense (Gaver ‘97)
– The only other sense with a natural language*
7
![Page 8: 1 Shengdong Zhao Department of Computer Science University of Toronto July 9, 2008 earPod: Efficient, Hierarchical, Eyes-free Menu Selection](https://reader035.vdocument.in/reader035/viewer/2022062717/56649e365503460f94b26051/html5/thumbnails/8.jpg)
earPod prototype
• A touch-sensitive device with software to perform eyes-free, hierarchical menu selection
• earPod prototype used commercial touchpad augmented with custom-built plastic cover, with cable to computer for voice output
• Built in University of Toronto in 2006 with Pierre Dragicevic
8
![Page 9: 1 Shengdong Zhao Department of Computer Science University of Toronto July 9, 2008 earPod: Efficient, Hierarchical, Eyes-free Menu Selection](https://reader035.vdocument.in/reader035/viewer/2022062717/56649e365503460f94b26051/html5/thumbnails/9.jpg)
earPod Prototype
9
![Page 10: 1 Shengdong Zhao Department of Computer Science University of Toronto July 9, 2008 earPod: Efficient, Hierarchical, Eyes-free Menu Selection](https://reader035.vdocument.in/reader035/viewer/2022062717/56649e365503460f94b26051/html5/thumbnails/10.jpg)
earPod interaction
10
Novice Intermediate
Intermediate Expert
![Page 11: 1 Shengdong Zhao Department of Computer Science University of Toronto July 9, 2008 earPod: Efficient, Hierarchical, Eyes-free Menu Selection](https://reader035.vdocument.in/reader035/viewer/2022062717/56649e365503460f94b26051/html5/thumbnails/11.jpg)
Video - browsing
11
![Page 12: 1 Shengdong Zhao Department of Computer Science University of Toronto July 9, 2008 earPod: Efficient, Hierarchical, Eyes-free Menu Selection](https://reader035.vdocument.in/reader035/viewer/2022062717/56649e365503460f94b26051/html5/thumbnails/12.jpg)
Video - fast browsing
12
![Page 13: 1 Shengdong Zhao Department of Computer Science University of Toronto July 9, 2008 earPod: Efficient, Hierarchical, Eyes-free Menu Selection](https://reader035.vdocument.in/reader035/viewer/2022062717/56649e365503460f94b26051/html5/thumbnails/13.jpg)
Video - very fast browsing
13
![Page 14: 1 Shengdong Zhao Department of Computer Science University of Toronto July 9, 2008 earPod: Efficient, Hierarchical, Eyes-free Menu Selection](https://reader035.vdocument.in/reader035/viewer/2022062717/56649e365503460f94b26051/html5/thumbnails/14.jpg)
Video - expert use
14
![Page 15: 1 Shengdong Zhao Department of Computer Science University of Toronto July 9, 2008 earPod: Efficient, Hierarchical, Eyes-free Menu Selection](https://reader035.vdocument.in/reader035/viewer/2022062717/56649e365503460f94b26051/html5/thumbnails/15.jpg)
Prototype design• Efficiency
– By-pass audio – Use interruptible audio feedback– Use non-speech audio
• Sense of control – Synchronous communication (reactive audio feedback)
• Simplicity and power – Gliding and tapping
• Easy to learn– Use spatial audio to reinforce mental model – Self-discoverable transition
15
![Page 16: 1 Shengdong Zhao Department of Computer Science University of Toronto July 9, 2008 earPod: Efficient, Hierarchical, Eyes-free Menu Selection](https://reader035.vdocument.in/reader035/viewer/2022062717/56649e365503460f94b26051/html5/thumbnails/16.jpg)
Empirical studies
Motivating questions: 1. How does earPod compare with a popular visual menu
selection technique used by iPod?
2. How does earPod compare with a number of competitor techniques with different modalities and menu styles?
3. What are learning behaviors for earPod and the related techniques?
4. How does earPod and the related techniques perform with a visually demanding primary task?
16
![Page 17: 1 Shengdong Zhao Department of Computer Science University of Toronto July 9, 2008 earPod: Efficient, Hierarchical, Eyes-free Menu Selection](https://reader035.vdocument.in/reader035/viewer/2022062717/56649e365503460f94b26051/html5/thumbnails/17.jpg)
Evaluation issues • Modality
– Visual
– Audio
– Dual
• Menu style– Linear
– Radial
17
Menu Style
Radial (absolute) Linear (relative)
Modality
Audio Audio radial Audio linear
Visual Visual radial Visual linear
Dual Audio visual radial Audio visual linear
![Page 18: 1 Shengdong Zhao Department of Computer Science University of Toronto July 9, 2008 earPod: Efficient, Hierarchical, Eyes-free Menu Selection](https://reader035.vdocument.in/reader035/viewer/2022062717/56649e365503460f94b26051/html5/thumbnails/18.jpg)
4 empirical studies
• 4 empirical studies were conducted at UofT in 2006-08
• Experiment 1: earPod vs. iPod (audio radial vs. visual linear)
• Experiment 2: 3 x 2 study (3 modalities x 2 menu styles)
• Experiment 3: 2 x 2 longitudinal study (audio linear, visual
linear, audio radial, visual radial) • Experiment 4: 3 x 2 dual-task study (3 modalities x 2 menu
styles)
18
![Page 19: 1 Shengdong Zhao Department of Computer Science University of Toronto July 9, 2008 earPod: Efficient, Hierarchical, Eyes-free Menu Selection](https://reader035.vdocument.in/reader035/viewer/2022062717/56649e365503460f94b26051/html5/thumbnails/19.jpg)
Experimental 1 design
• Goal: compare earPod with iPod-like linear menu
• 60-90 minute session, within-subject, counter-balanced
• Setup:12 participants x 2 techniques (audio and visual) x (40+80) items for the 2 menu configurations (8 and
8x8) x 4 blocks= 11,520 menu selections in total.
19
![Page 20: 1 Shengdong Zhao Department of Computer Science University of Toronto July 9, 2008 earPod: Efficient, Hierarchical, Eyes-free Menu Selection](https://reader035.vdocument.in/reader035/viewer/2022062717/56649e365503460f94b26051/html5/thumbnails/20.jpg)
Results: response time
Block number Block number
20
![Page 21: 1 Shengdong Zhao Department of Computer Science University of Toronto July 9, 2008 earPod: Efficient, Hierarchical, Eyes-free Menu Selection](https://reader035.vdocument.in/reader035/viewer/2022062717/56649e365503460f94b26051/html5/thumbnails/21.jpg)
Results: response time
Block number Block number
21
![Page 22: 1 Shengdong Zhao Department of Computer Science University of Toronto July 9, 2008 earPod: Efficient, Hierarchical, Eyes-free Menu Selection](https://reader035.vdocument.in/reader035/viewer/2022062717/56649e365503460f94b26051/html5/thumbnails/22.jpg)
Results: response time
Block number Block number
22
![Page 23: 1 Shengdong Zhao Department of Computer Science University of Toronto July 9, 2008 earPod: Efficient, Hierarchical, Eyes-free Menu Selection](https://reader035.vdocument.in/reader035/viewer/2022062717/56649e365503460f94b26051/html5/thumbnails/23.jpg)
Experiment 1 - results and discussion
• earPod has comparable performance with iPod-like linear menu
• earPod user performance improves more rapidly than that of iPod-like linear menu
• With practice, earPod can outperform iPod-like linear menu
23
![Page 24: 1 Shengdong Zhao Department of Computer Science University of Toronto July 9, 2008 earPod: Efficient, Hierarchical, Eyes-free Menu Selection](https://reader035.vdocument.in/reader035/viewer/2022062717/56649e365503460f94b26051/html5/thumbnails/24.jpg)
Experiment 2 design
• Goal: systematic investigation of modality of feedback and menu style
• 60-minute, within subject, counter-balanced • Setup
12 participants x6 techniques x8 menu items x13 blocks (12 blocks + 1 practice block)= 7488 menu selections in total
24
![Page 25: 1 Shengdong Zhao Department of Computer Science University of Toronto July 9, 2008 earPod: Efficient, Hierarchical, Eyes-free Menu Selection](https://reader035.vdocument.in/reader035/viewer/2022062717/56649e365503460f94b26051/html5/thumbnails/25.jpg)
25
Response time
Audio radial
Audio linear
Visual radial
Visual linear
Audio visual radial
Audio visual linear
![Page 26: 1 Shengdong Zhao Department of Computer Science University of Toronto July 9, 2008 earPod: Efficient, Hierarchical, Eyes-free Menu Selection](https://reader035.vdocument.in/reader035/viewer/2022062717/56649e365503460f94b26051/html5/thumbnails/26.jpg)
Experiment 2 - results and discussion
• Accuracy– Comparable performance among all techniques
• Speed – visual radial ~ dual radial
< audio radial ~ visual linear ~ dual linear < audio linear
• Modality– Visual ~ dual < audio
• Menu style– Radial < linear
26
![Page 27: 1 Shengdong Zhao Department of Computer Science University of Toronto July 9, 2008 earPod: Efficient, Hierarchical, Eyes-free Menu Selection](https://reader035.vdocument.in/reader035/viewer/2022062717/56649e365503460f94b26051/html5/thumbnails/27.jpg)
Experiment 3: longitudinal study
Goal: learning of earPod and related techniques5 one-hour sessions in a week, between-subjectSetup:
2 participants (per technique) x 4 techniques x64 items for 1 menu configuration (8 x 8) x7 blocks per day x5 days= 17920 menu selections in total
27
![Page 28: 1 Shengdong Zhao Department of Computer Science University of Toronto July 9, 2008 earPod: Efficient, Hierarchical, Eyes-free Menu Selection](https://reader035.vdocument.in/reader035/viewer/2022062717/56649e365503460f94b26051/html5/thumbnails/28.jpg)
Accuracy
28
1 2 3 4 50.88
0.9
0.92
0.94
0.96
0.98
1
Audio linear Audio radial Visual linear Visual radial
Day
![Page 29: 1 Shengdong Zhao Department of Computer Science University of Toronto July 9, 2008 earPod: Efficient, Hierarchical, Eyes-free Menu Selection](https://reader035.vdocument.in/reader035/viewer/2022062717/56649e365503460f94b26051/html5/thumbnails/29.jpg)
Response time
29
1 2 3 4 50
1000
2000
3000
4000
5000
6000
Audio linear Audio radial Visual linear Visual radial
Day
![Page 30: 1 Shengdong Zhao Department of Computer Science University of Toronto July 9, 2008 earPod: Efficient, Hierarchical, Eyes-free Menu Selection](https://reader035.vdocument.in/reader035/viewer/2022062717/56649e365503460f94b26051/html5/thumbnails/30.jpg)
Gliding to tapping transition
30
Items explored Tapping percentage
1 2 3 4 50
1
2
3
4
5
6
7
Audio linear Audio radial Visual linear Visual radial
1 2 3 4 50
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Audio radial Visual radial
Day Day
![Page 31: 1 Shengdong Zhao Department of Computer Science University of Toronto July 9, 2008 earPod: Efficient, Hierarchical, Eyes-free Menu Selection](https://reader035.vdocument.in/reader035/viewer/2022062717/56649e365503460f94b26051/html5/thumbnails/31.jpg)
Experiment 3 - results and discussion
• Novice performance is determined by modality, expert performance is determined by menu style
• Rapid learning is observed for earPod• Users can perform tapping for 70-80% of the
64 menu items from day 3 onwards• Audio linear has comparable performance
with visual linear from day 3 onwards
31
![Page 32: 1 Shengdong Zhao Department of Computer Science University of Toronto July 9, 2008 earPod: Efficient, Hierarchical, Eyes-free Menu Selection](https://reader035.vdocument.in/reader035/viewer/2022062717/56649e365503460f94b26051/html5/thumbnails/32.jpg)
Experiment 4: dual-task study
Goal: Investigate the properties of using earPod and the related techniques as a secondary task in a dual-task setting and their impact on the visually demanding primary task
32
![Page 33: 1 Shengdong Zhao Department of Computer Science University of Toronto July 9, 2008 earPod: Efficient, Hierarchical, Eyes-free Menu Selection](https://reader035.vdocument.in/reader035/viewer/2022062717/56649e365503460f94b26051/html5/thumbnails/33.jpg)
Experiment 4: dual-task study
• Dual-task– Primary task: simulated driving– Secondary task: menu selection
• External validity: similar settings have been
successfully used by Salvucci et al. ‘01, ‘02, ‘05
33
![Page 34: 1 Shengdong Zhao Department of Computer Science University of Toronto July 9, 2008 earPod: Efficient, Hierarchical, Eyes-free Menu Selection](https://reader035.vdocument.in/reader035/viewer/2022062717/56649e365503460f94b26051/html5/thumbnails/34.jpg)
Experiment 4 design
• Conditions– Desktop
• 12 participants X 6 techniques X 8 items of 1 menu configurations X (3 + 1) blocks = 2304 menu selections
– Driving• 12 participants X 6 techniques X 8 items of 1 menu configurations X (1 + 1)
blocks = 1152 menu selections
• Measures– Desktop
• Speed & accuracy of menu selection
– Driving• Speed & accuracy of menu selection (secondary task)• Driving performance (primary task)
34
![Page 35: 1 Shengdong Zhao Department of Computer Science University of Toronto July 9, 2008 earPod: Efficient, Hierarchical, Eyes-free Menu Selection](https://reader035.vdocument.in/reader035/viewer/2022062717/56649e365503460f94b26051/html5/thumbnails/35.jpg)
Experiment 4 setup
35
![Page 36: 1 Shengdong Zhao Department of Computer Science University of Toronto July 9, 2008 earPod: Efficient, Hierarchical, Eyes-free Menu Selection](https://reader035.vdocument.in/reader035/viewer/2022062717/56649e365503460f94b26051/html5/thumbnails/36.jpg)
Response time: modality
36
1 2 30
500
1000
1500
2000
2500
3000
3500
4000
4500
DesktopDriving
Audio Visual Dual
![Page 37: 1 Shengdong Zhao Department of Computer Science University of Toronto July 9, 2008 earPod: Efficient, Hierarchical, Eyes-free Menu Selection](https://reader035.vdocument.in/reader035/viewer/2022062717/56649e365503460f94b26051/html5/thumbnails/37.jpg)
1 20
500
1000
1500
2000
2500
3000
3500
4000
4500
DesktopDriving
Response time: menu style
37
PRadial Linear
![Page 38: 1 Shengdong Zhao Department of Computer Science University of Toronto July 9, 2008 earPod: Efficient, Hierarchical, Eyes-free Menu Selection](https://reader035.vdocument.in/reader035/viewer/2022062717/56649e365503460f94b26051/html5/thumbnails/38.jpg)
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Audio Visual Dual
Late
ral V
eloc
ity
(m/s
)Lateral velocity
38
![Page 39: 1 Shengdong Zhao Department of Computer Science University of Toronto July 9, 2008 earPod: Efficient, Hierarchical, Eyes-free Menu Selection](https://reader035.vdocument.in/reader035/viewer/2022062717/56649e365503460f94b26051/html5/thumbnails/39.jpg)
0
10
20
30
40
50
60
70
80
Audio Visual Dual
Follo
win
g D
ista
nce
(m
)Following distance
39
![Page 40: 1 Shengdong Zhao Department of Computer Science University of Toronto July 9, 2008 earPod: Efficient, Hierarchical, Eyes-free Menu Selection](https://reader035.vdocument.in/reader035/viewer/2022062717/56649e365503460f94b26051/html5/thumbnails/40.jpg)
Experiment 4 - results and discussion
• Audio techniques work better with a visually demanding primary task– Comparable performance with visual techniques,
less impact on primary tasks • Although dual-channel techniques have
similar performance with audio techniques, users prefer single channel audio feedback
40
![Page 41: 1 Shengdong Zhao Department of Computer Science University of Toronto July 9, 2008 earPod: Efficient, Hierarchical, Eyes-free Menu Selection](https://reader035.vdocument.in/reader035/viewer/2022062717/56649e365503460f94b26051/html5/thumbnails/41.jpg)
Summary of empirical studies
• earPod has comparable performance with visual linear menus
• Transition from novice to expert for earPod is fast
• Audio techniques work better with a visually demanding primary task
• Users only pay attention to the appropriate channel when are presented with both channels of feedback
41
![Page 42: 1 Shengdong Zhao Department of Computer Science University of Toronto July 9, 2008 earPod: Efficient, Hierarchical, Eyes-free Menu Selection](https://reader035.vdocument.in/reader035/viewer/2022062717/56649e365503460f94b26051/html5/thumbnails/42.jpg)
Contributions
• Developed a novel interaction technique called earPod
• Conducted empirical evaluations on earPod and related techniques
• Derived design recommendation for incorporating earPod and related techniques to mobile devices
42
![Page 43: 1 Shengdong Zhao Department of Computer Science University of Toronto July 9, 2008 earPod: Efficient, Hierarchical, Eyes-free Menu Selection](https://reader035.vdocument.in/reader035/viewer/2022062717/56649e365503460f94b26051/html5/thumbnails/43.jpg)
Future work
• Explore eyes-free menu techniques for long and dynamic lists
• Investigate the role of compression and dichotic listening
• Extend earPod to other mobile scenarios (e.g., walking, running, etc.)
• Develop eyes-free interaction techniques for other tasks and applications (e.g., drag & drop, text entry, etc.)
43
![Page 44: 1 Shengdong Zhao Department of Computer Science University of Toronto July 9, 2008 earPod: Efficient, Hierarchical, Eyes-free Menu Selection](https://reader035.vdocument.in/reader035/viewer/2022062717/56649e365503460f94b26051/html5/thumbnails/44.jpg)
Questions
44
![Page 45: 1 Shengdong Zhao Department of Computer Science University of Toronto July 9, 2008 earPod: Efficient, Hierarchical, Eyes-free Menu Selection](https://reader035.vdocument.in/reader035/viewer/2022062717/56649e365503460f94b26051/html5/thumbnails/45.jpg)
45
The End