![Page 1: CS-525H: Immersive HCIweb.cs.wpi.edu/~gogo/courses/cs525H/slides/cs525H... · R.W. Lindeman -WPI Dept. of Computer Science Interactive Media & Game Development 20 Design of the User](https://reader034.vdocument.in/reader034/viewer/2022043006/5f8f502367403766ca2667dc/html5/thumbnails/1.jpg)
CS-525H:Immersive HCI
Augmented Reality
Robert W. Lindeman
Worcester Polytechnic InstituteDepartment of Computer Science
![Page 2: CS-525H: Immersive HCIweb.cs.wpi.edu/~gogo/courses/cs525H/slides/cs525H... · R.W. Lindeman -WPI Dept. of Computer Science Interactive Media & Game Development 20 Design of the User](https://reader034.vdocument.in/reader034/viewer/2022043006/5f8f502367403766ca2667dc/html5/thumbnails/2.jpg)
R.W. Lindeman - WPI Dept. of Computer ScienceInteractive Media & Game Development
2
Motivation Augmented Reality
Mixing of real-world (RW) and computer-generated(CG) stimuli
Graphical overlays on the real world Adding information to real experiences
Much work on visual sense Can be extended to auditory sense
Other senses?
For the user to merge RW and CG, attributesmust be matched Visual: Lighting & shadows, level of fidelity Audio: CG and RW sound occlusion and reflection
![Page 3: CS-525H: Immersive HCIweb.cs.wpi.edu/~gogo/courses/cs525H/slides/cs525H... · R.W. Lindeman -WPI Dept. of Computer Science Interactive Media & Game Development 20 Design of the User](https://reader034.vdocument.in/reader034/viewer/2022043006/5f8f502367403766ca2667dc/html5/thumbnails/3.jpg)
R.W. Lindeman - WPI Dept. of Computer ScienceInteractive Media & Game Development
3
Direct
Real-World Stimulus Paths
1 2 3 4
Real-WorldSignal Environment Sensory
Subsystem Nerves Brain
Real-WorldSignal Environment Capture
DevicePost-
ProcessingCaptured
Signal
Captured/Mediated
![Page 4: CS-525H: Immersive HCIweb.cs.wpi.edu/~gogo/courses/cs525H/slides/cs525H... · R.W. Lindeman -WPI Dept. of Computer Science Interactive Media & Game Development 20 Design of the User](https://reader034.vdocument.in/reader034/viewer/2022043006/5f8f502367403766ca2667dc/html5/thumbnails/4.jpg)
R.W. Lindeman - WPI Dept. of Computer ScienceInteractive Media & Game Development
4
Visual Sense
PC
Projection Mixing in the environment (far)
![Page 5: CS-525H: Immersive HCIweb.cs.wpi.edu/~gogo/courses/cs525H/slides/cs525H... · R.W. Lindeman -WPI Dept. of Computer Science Interactive Media & Game Development 20 Design of the User](https://reader034.vdocument.in/reader034/viewer/2022043006/5f8f502367403766ca2667dc/html5/thumbnails/5.jpg)
R.W. Lindeman - WPI Dept. of Computer ScienceInteractive Media & Game Development
5
Visual Sense (cont.)
PC
Optical-see-through AR Mixing in the environment (near)
![Page 6: CS-525H: Immersive HCIweb.cs.wpi.edu/~gogo/courses/cs525H/slides/cs525H... · R.W. Lindeman -WPI Dept. of Computer Science Interactive Media & Game Development 20 Design of the User](https://reader034.vdocument.in/reader034/viewer/2022043006/5f8f502367403766ca2667dc/html5/thumbnails/6.jpg)
R.W. Lindeman - WPI Dept. of Computer ScienceInteractive Media & Game Development
6
Visual Sense (cont.)
PC
Video-see-through AR Mixing in the Computer
![Page 7: CS-525H: Immersive HCIweb.cs.wpi.edu/~gogo/courses/cs525H/slides/cs525H... · R.W. Lindeman -WPI Dept. of Computer Science Interactive Media & Game Development 20 Design of the User](https://reader034.vdocument.in/reader034/viewer/2022043006/5f8f502367403766ca2667dc/html5/thumbnails/7.jpg)
R.W. Lindeman - WPI Dept. of Computer ScienceInteractive Media & Game Development
7
Video-See-Through HMD
(Image: Fuchs, et al., Medical Image Computing and Computer-Assisted Intervention (MICCAI) ’98, LNCS, 1998, Vol.1496/1998, 934)
![Page 8: CS-525H: Immersive HCIweb.cs.wpi.edu/~gogo/courses/cs525H/slides/cs525H... · R.W. Lindeman -WPI Dept. of Computer Science Interactive Media & Game Development 20 Design of the User](https://reader034.vdocument.in/reader034/viewer/2022043006/5f8f502367403766ca2667dc/html5/thumbnails/8.jpg)
R.W. Lindeman - WPI Dept. of Computer ScienceInteractive Media & Game Development
8
Video-See-Through HMD (cont.)
(Image: Fuchs, et al., Medical Image Computing and Computer-Assisted Intervention (MICCAI) ’98, LNCS, 1998, Vol.1496/1998, 934)
![Page 9: CS-525H: Immersive HCIweb.cs.wpi.edu/~gogo/courses/cs525H/slides/cs525H... · R.W. Lindeman -WPI Dept. of Computer Science Interactive Media & Game Development 20 Design of the User](https://reader034.vdocument.in/reader034/viewer/2022043006/5f8f502367403766ca2667dc/html5/thumbnails/9.jpg)
R.W. Lindeman - WPI Dept. of Computer ScienceInteractive Media & Game Development
9
Video-See-Through HMD (cont.)NVIS: nVisor MH60-V (2010)
http://www.nvisinc.com/product2009.php?id=57
![Page 10: CS-525H: Immersive HCIweb.cs.wpi.edu/~gogo/courses/cs525H/slides/cs525H... · R.W. Lindeman -WPI Dept. of Computer Science Interactive Media & Game Development 20 Design of the User](https://reader034.vdocument.in/reader034/viewer/2022043006/5f8f502367403766ca2667dc/html5/thumbnails/10.jpg)
R.W. Lindeman - WPI Dept. of Computer ScienceInteractive Media & Game Development
10
Using Visual AR: SDKsARToolKit
http://www.hitl.washington.edu/artoolkit/ Earliest usable kit Now Open Source (free) Commercial versions for iPhone & Android
http://www.artoolworks.com/
Studierstube ES & Tracker http://studierstube.icg.tu-graz.ac.at/handheld_ar/ ES sits on top of Tracker Not free
![Page 11: CS-525H: Immersive HCIweb.cs.wpi.edu/~gogo/courses/cs525H/slides/cs525H... · R.W. Lindeman -WPI Dept. of Computer Science Interactive Media & Game Development 20 Design of the User](https://reader034.vdocument.in/reader034/viewer/2022043006/5f8f502367403766ca2667dc/html5/thumbnails/11.jpg)
R.W. Lindeman - WPI Dept. of Computer ScienceInteractive Media & Game Development
11
Using Visual AR: SDKs ExamplesARToolKit
http://www.youtube.com/watch?v=5M-oAmBDcZk (local clip)
Studierstube ES http://www.youtube.com/watch?v=JwluCuVKO9c (local clips)
![Page 12: CS-525H: Immersive HCIweb.cs.wpi.edu/~gogo/courses/cs525H/slides/cs525H... · R.W. Lindeman -WPI Dept. of Computer Science Interactive Media & Game Development 20 Design of the User](https://reader034.vdocument.in/reader034/viewer/2022043006/5f8f502367403766ca2667dc/html5/thumbnails/12.jpg)
R.W. Lindeman - WPI Dept. of Computer ScienceInteractive Media & Game Development
12
Using Visual AR: ToolsGoogle SketchUp + ARMedia Plugin
http://www.youtube.com/watch?v=wsQ-YGgVUT0 (local clip) (live demo) http://sketchup.google.com/
Layar for mobile devices http://www.layar.com/ Layering tool for layar browser
"Like HTML for AR" (local clip)
![Page 13: CS-525H: Immersive HCIweb.cs.wpi.edu/~gogo/courses/cs525H/slides/cs525H... · R.W. Lindeman -WPI Dept. of Computer Science Interactive Media & Game Development 20 Design of the User](https://reader034.vdocument.in/reader034/viewer/2022043006/5f8f502367403766ca2667dc/html5/thumbnails/13.jpg)
R.W. Lindeman - WPI Dept. of Computer ScienceInteractive Media & Game Development
13
Using Visual AR: Tools (cont.) Cereal?
http://www.youtube.com/watch?v=jGdSslAJRwM (local clip)
Slot Cars? http://www.youtube.com/watch?v=WMWEYqYPDfc (local clip)
Magic Tricks? http://www.youtube.com/watch?v=Mk1xjbA-ISE (local clip)
Heads-up Display in Cars (play GE clip) Mobile AR (play Nokia clip) Mobile 3rd Party
http://news.bbc.co.uk/2/hi/technology/8193951.stm
![Page 14: CS-525H: Immersive HCIweb.cs.wpi.edu/~gogo/courses/cs525H/slides/cs525H... · R.W. Lindeman -WPI Dept. of Computer Science Interactive Media & Game Development 20 Design of the User](https://reader034.vdocument.in/reader034/viewer/2022043006/5f8f502367403766ca2667dc/html5/thumbnails/14.jpg)
R.W. Lindeman - WPI Dept. of Computer ScienceInteractive Media & Game Development
14
Sound Paths & Mixing Points
Real-WorldSound Environment Outer
EarNerves Brain
Typical VR/AR systems use speakers (1)or headphones (2a)
Our approach performs the mixing at thecochlea (2b)
InnerEar
MiddleEar
Ear System
1 2 3 4a b
![Page 15: CS-525H: Immersive HCIweb.cs.wpi.edu/~gogo/courses/cs525H/slides/cs525H... · R.W. Lindeman -WPI Dept. of Computer Science Interactive Media & Game Development 20 Design of the User](https://reader034.vdocument.in/reader034/viewer/2022043006/5f8f502367403766ca2667dc/html5/thumbnails/15.jpg)
R.W. Lindeman - WPI Dept. of Computer ScienceInteractive Media & Game Development
15
Auditory SenseAcoustic-Hear-Through AR (Speakers)
Mixing in the environment (far)
PC
![Page 16: CS-525H: Immersive HCIweb.cs.wpi.edu/~gogo/courses/cs525H/slides/cs525H... · R.W. Lindeman -WPI Dept. of Computer Science Interactive Media & Game Development 20 Design of the User](https://reader034.vdocument.in/reader034/viewer/2022043006/5f8f502367403766ca2667dc/html5/thumbnails/16.jpg)
R.W. Lindeman - WPI Dept. of Computer ScienceInteractive Media & Game Development
16
Auditory Sense (cont.)Mic-Through AR
Mixing in the computer
PC
![Page 17: CS-525H: Immersive HCIweb.cs.wpi.edu/~gogo/courses/cs525H/slides/cs525H... · R.W. Lindeman -WPI Dept. of Computer Science Interactive Media & Game Development 20 Design of the User](https://reader034.vdocument.in/reader034/viewer/2022043006/5f8f502367403766ca2667dc/html5/thumbnails/17.jpg)
R.W. Lindeman - WPI Dept. of Computer ScienceInteractive Media & Game Development
17
Auditory Sense (cont.)Hear-Through AR
Bone conduction Mixing at the sensory subsystem
PC
![Page 18: CS-525H: Immersive HCIweb.cs.wpi.edu/~gogo/courses/cs525H/slides/cs525H... · R.W. Lindeman -WPI Dept. of Computer Science Interactive Media & Game Development 20 Design of the User](https://reader034.vdocument.in/reader034/viewer/2022043006/5f8f502367403766ca2667dc/html5/thumbnails/18.jpg)
R.W. Lindeman - WPI Dept. of Computer ScienceInteractive Media & Game Development
18
Bone-Conduction ExampleThe sound of your own voice is a
combination of: Sound reaching your ears through the air Vibrations reaching your cochlea though your
head
ExampleSound heard through the airSound heard through the headCombined sound
Mauldin & Scordilis, 2004
![Page 19: CS-525H: Immersive HCIweb.cs.wpi.edu/~gogo/courses/cs525H/slides/cs525H... · R.W. Lindeman -WPI Dept. of Computer Science Interactive Media & Game Development 20 Design of the User](https://reader034.vdocument.in/reader034/viewer/2022043006/5f8f502367403766ca2667dc/html5/thumbnails/19.jpg)
R.W. Lindeman - WPI Dept. of Computer ScienceInteractive Media & Game Development
19
Research QuestionsHow well can people localize sound using
bone conduction?What types of sound works best?
Ambient sound Spoken voice Sound FX Music
We looked at basic sounds (sine waves)of various frequencies Stationary and moving sounds
![Page 20: CS-525H: Immersive HCIweb.cs.wpi.edu/~gogo/courses/cs525H/slides/cs525H... · R.W. Lindeman -WPI Dept. of Computer Science Interactive Media & Game Development 20 Design of the User](https://reader034.vdocument.in/reader034/viewer/2022043006/5f8f502367403766ca2667dc/html5/thumbnails/20.jpg)
R.W. Lindeman - WPI Dept. of Computer ScienceInteractive Media & Game Development
20
Design of the User Study 24 Computer science students (22 male)
3 Main treatments (Audio Devices) Speakers, Headphones, Bone-Conduction Device
Each subject performed 63 trials with eachdevice 3 Frequencies
Low (200Hz), Medium (500Hz), High (1kHz) 7 sound samples (5 sound locations + 2 directions)
Left, Center-Left, Center, Center-Right, Right Moving, right-to-left moving
3 repetitions of each combination 3 * 7 * 3 = 63
![Page 21: CS-525H: Immersive HCIweb.cs.wpi.edu/~gogo/courses/cs525H/slides/cs525H... · R.W. Lindeman -WPI Dept. of Computer Science Interactive Media & Game Development 20 Design of the User](https://reader034.vdocument.in/reader034/viewer/2022043006/5f8f502367403766ca2667dc/html5/thumbnails/21.jpg)
R.W. Lindeman - WPI Dept. of Computer ScienceInteractive Media & Game Development
21
User StudyPhysical/Virtual sound locations
Ө = 45o
ӨӨ
ӨLEFT RIGHT
CENTER
CENTER-LEFT
CENTER-RIGHT
r = 1m
![Page 22: CS-525H: Immersive HCIweb.cs.wpi.edu/~gogo/courses/cs525H/slides/cs525H... · R.W. Lindeman -WPI Dept. of Computer Science Interactive Media & Game Development 20 Design of the User](https://reader034.vdocument.in/reader034/viewer/2022043006/5f8f502367403766ca2667dc/html5/thumbnails/22.jpg)
R.W. Lindeman - WPI Dept. of Computer ScienceInteractive Media & Game Development
22
User Study (cont.)Each sample was played for 1 secondSubjects wore a blindfoldNo HRTFs usedSubjects had to identify location/direction
![Page 23: CS-525H: Immersive HCIweb.cs.wpi.edu/~gogo/courses/cs525H/slides/cs525H... · R.W. Lindeman -WPI Dept. of Computer Science Interactive Media & Game Development 20 Design of the User](https://reader034.vdocument.in/reader034/viewer/2022043006/5f8f502367403766ca2667dc/html5/thumbnails/23.jpg)
R.W. Lindeman - WPI Dept. of Computer ScienceInteractive Media & Game Development
23
Results Accuracy for Stationary Sounds
Speakers > headphones > bone conduction High-Freq. == Low Freq., both > Medium Freq.
Accuracy for Moving Sounds Speakers == Bone conduction Bone Conduction == Headphones Speakers > headphones (α = .05)
nsnsInteractionnsHIGH LOW MEDFrequency
S B HS H BAudio DeviceMovingStationary
![Page 24: CS-525H: Immersive HCIweb.cs.wpi.edu/~gogo/courses/cs525H/slides/cs525H... · R.W. Lindeman -WPI Dept. of Computer Science Interactive Media & Game Development 20 Design of the User](https://reader034.vdocument.in/reader034/viewer/2022043006/5f8f502367403766ca2667dc/html5/thumbnails/24.jpg)
R.W. Lindeman - WPI Dept. of Computer ScienceInteractive Media & Game Development
24
Results (cont.) Problems with the "in-between" locations
Center-Left/Center-Right
![Page 25: CS-525H: Immersive HCIweb.cs.wpi.edu/~gogo/courses/cs525H/slides/cs525H... · R.W. Lindeman -WPI Dept. of Computer Science Interactive Media & Game Development 20 Design of the User](https://reader034.vdocument.in/reader034/viewer/2022043006/5f8f502367403766ca2667dc/html5/thumbnails/25.jpg)
R.W. Lindeman - WPI Dept. of Computer ScienceInteractive Media & Game Development
25
AnalysisReal-world sound
High fidelity Low control
Computer-generated sound Low(er) fidelity Complete control
Later mixing point = Closer to the brain More personalized, but More processing for transforming and mixing
![Page 26: CS-525H: Immersive HCIweb.cs.wpi.edu/~gogo/courses/cs525H/slides/cs525H... · R.W. Lindeman -WPI Dept. of Computer Science Interactive Media & Game Development 20 Design of the User](https://reader034.vdocument.in/reader034/viewer/2022043006/5f8f502367403766ca2667dc/html5/thumbnails/26.jpg)
R.W. Lindeman - WPI Dept. of Computer ScienceInteractive Media & Game Development
26
Analysis (cont.)Bone-conduction/headphone approaches
Require head tracking for CG sound Require processing for spatialization (e.g.,
HRTF or BRTF)
Speaker-based Allows for shared experience (like projection
systems in visual field)
![Page 27: CS-525H: Immersive HCIweb.cs.wpi.edu/~gogo/courses/cs525H/slides/cs525H... · R.W. Lindeman -WPI Dept. of Computer Science Interactive Media & Game Development 20 Design of the User](https://reader034.vdocument.in/reader034/viewer/2022043006/5f8f502367403766ca2667dc/html5/thumbnails/27.jpg)
R.W. Lindeman - WPI Dept. of Computer ScienceInteractive Media & Game Development
27
Haptic Sense
Mixing at Sensory Subsystem(Novint Falcon)
Mixing in Computer (teleoperation) orin Environment(Immersion CyberGrasp)
![Page 28: CS-525H: Immersive HCIweb.cs.wpi.edu/~gogo/courses/cs525H/slides/cs525H... · R.W. Lindeman -WPI Dept. of Computer Science Interactive Media & Game Development 20 Design of the User](https://reader034.vdocument.in/reader034/viewer/2022043006/5f8f502367403766ca2667dc/html5/thumbnails/28.jpg)
R.W. Lindeman - WPI Dept. of Computer ScienceInteractive Media & Game Development
28
Haptic Sense (cont.)Mixing in the Environment
(Lindeman, VRST 2004)
![Page 29: CS-525H: Immersive HCIweb.cs.wpi.edu/~gogo/courses/cs525H/slides/cs525H... · R.W. Lindeman -WPI Dept. of Computer Science Interactive Media & Game Development 20 Design of the User](https://reader034.vdocument.in/reader034/viewer/2022043006/5f8f502367403766ca2667dc/html5/thumbnails/29.jpg)
R.W. Lindeman - WPI Dept. of Computer ScienceInteractive Media & Game Development
29
Olfactory SenseMixing in the Environment (far)[AirCanon (Yanagida et al., 2004)]
Mixing in the Computer(Hirose et al. 1997)
![Page 30: CS-525H: Immersive HCIweb.cs.wpi.edu/~gogo/courses/cs525H/slides/cs525H... · R.W. Lindeman -WPI Dept. of Computer Science Interactive Media & Game Development 20 Design of the User](https://reader034.vdocument.in/reader034/viewer/2022043006/5f8f502367403766ca2667dc/html5/thumbnails/30.jpg)
R.W. Lindeman - WPI Dept. of Computer ScienceInteractive Media & Game Development
30
Olfactory Sense (cont.)
Mixing in the Environment (mid)[AirCanon (Yanagida et al., 2004)]
Mixing in the Environment (near)(Nakamoto & Min, 2007)
![Page 31: CS-525H: Immersive HCIweb.cs.wpi.edu/~gogo/courses/cs525H/slides/cs525H... · R.W. Lindeman -WPI Dept. of Computer Science Interactive Media & Game Development 20 Design of the User](https://reader034.vdocument.in/reader034/viewer/2022043006/5f8f502367403766ca2667dc/html5/thumbnails/31.jpg)
R.W. Lindeman - WPI Dept. of Computer ScienceInteractive Media & Game Development
31
Gustatory SenseBite interface
Really haptics (near)
Iwata, 2004(photos: Sid Fels)
![Page 32: CS-525H: Immersive HCIweb.cs.wpi.edu/~gogo/courses/cs525H/slides/cs525H... · R.W. Lindeman -WPI Dept. of Computer Science Interactive Media & Game Development 20 Design of the User](https://reader034.vdocument.in/reader034/viewer/2022043006/5f8f502367403766ca2667dc/html5/thumbnails/32.jpg)
R.W. Lindeman - WPI Dept. of Computer ScienceInteractive Media & Game Development
32
Gustatory Sense (cont.)Edible bitsStraw-like interface
Mixing in the env.
(Maynes-Aminzade, 2005)(Nakamoto, 2007)
![Page 33: CS-525H: Immersive HCIweb.cs.wpi.edu/~gogo/courses/cs525H/slides/cs525H... · R.W. Lindeman -WPI Dept. of Computer Science Interactive Media & Game Development 20 Design of the User](https://reader034.vdocument.in/reader034/viewer/2022043006/5f8f502367403766ca2667dc/html5/thumbnails/33.jpg)
R.W. Lindeman - WPI Dept. of Computer ScienceInteractive Media & Game Development
33
Gustatory Sense (cont.)
![Page 34: CS-525H: Immersive HCIweb.cs.wpi.edu/~gogo/courses/cs525H/slides/cs525H... · R.W. Lindeman -WPI Dept. of Computer Science Interactive Media & Game Development 20 Design of the User](https://reader034.vdocument.in/reader034/viewer/2022043006/5f8f502367403766ca2667dc/html5/thumbnails/34.jpg)
R.W. Lindeman - WPI Dept. of Computer ScienceInteractive Media & Game Development
34
Final Thoughts What about a 3D printer+robot arm? RW stimuli
High fidelity / low control
CG stimuli Low(er) fidelity / complete control
Later mixing point = more "personal" stimuli Closer to the brain
Multi-sensory approaches are interesting Compensate for weaknesses in one sense with
another sense Use speakers for environmental, bone-conduction for
virtual characters