the reality of augmented reality: are we there yet?
TRANSCRIPT
The Reality of Augmented Reality: Are we there yet?
Mark Billinghurst
The HIT Lab NZ, University of Canterbury
March 24th 2015
My Childhood Home
The Vision - Auckland
The Road Trip
Happy Trails
The Reality
Mount Messenger
Are We There Yet?
Vision vs. Reality
Lesson’s Learned ! Have a clear driving vision ! Enjoy the journey ! Travel with others ! Have a well equipped vehicle ! The end always seems nearer than it really is
THE AR VISION
The Ultimate Display The ultimate display would .. be a room within which the computer can control the existence of matter. A chair displayed in such a room would be good enough to sit in. Handcuffs .. would be confining, and a bullet .. would be fatal. With appropriate programming such a display could literally be the Wonderland into which Alice walked.
Ivan E. Sutherland
Sutherland, I.E. The ultimate display. Information Processing 1965, Proc. IFIP Congress 65, 506-508.
Realizing the Vision ! 3D computer graphics ! Physical input devices ! Speech interaction ! Eye-gaze input ! Kinesthetic force feedback
Sutherland, I. E. (1968, December). A head-mounted three dimensional display. In Proceedings of the December 9-11, 1968, fall joint computer conference, part I (pp. 757-764). ACM.
Star Wars (1970’s-80’s)
! 3D projection ! Collaborative systems
The Terminator (1984 - )
! Scene recognition ! Wearable AR HUD
The Super Cockpit (1980’s)
! Furness - USAF
Furness, T. A. (1986, September). The super cockpit and its human factors challenges. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (Vol. 30, No. 1, pp. 48-52). SAGE Publications.
Star Trek (1987 – 94)
! HoloDeck, Sensory glasses ! Ubiquitous Computing
Minority Report (2002)
! Natural gesture interaction
Ironman (2008-)
! Natural gesture input ! Wearable AR system
Trends ! 1960’s – 70’s: Display of Information
! Ultimate Displays, Star Wars
! 1980’s Information Overlay ! Terminator, Super cockpit
! 1990’s Environmental Enhancement ! Star Trek HoloDeck
! 2000’s Natural Interaction, Wearable Systems ! Minority Report, Iron-man
ARE WE THERE YET?
2012 – Iron Man 2
To Make the Vision Real.. ! Automatically detecting real environment
! Environmental awareness, Physically based interaction
! Gesture interaction ! Free-hand interaction
! Multimodal input ! Speech and gesture interaction
! Intelligent interfaces ! Implicit rather than Explicit interaction
Environmental Awareness ! AR MicroMachines
! AR experience with environment awareness and physically-based interaction
! Based on MS Kinect RGB-D sensor
! Augmented environment supports ! occlusion, shadows ! physically-based interaction between real and
virtual objects
Clark, A., & Piumsomboon, T. (2011). A realistic augmented reality racing game using a depth-sensing camera. In Proceedings of the 10th International Conference on Virtual Reality Continuum and Its Applications in Industry (pp. 499-502). ACM.
Physics Simulation
! Create virtual mesh over real world
! Update at 10 fps – can move real objects
! Use by physics engine for collision detection (virtual/real)
! Use by OpenScenegraph for occlusion and shadows
Rendering
Occlusion Shadows
Natural Hand Interaction
! Using bare hands to interact with AR content ! MS Kinect depth sensing ! Real time hand tracking ! Physics based simulation model
Skeleton Based Interaction
! 3 Gear Systems ! Kinect/Primesense Sensor ! Two hand tracking ! http://www.threegear.com
AR Rift Display
Skeleton Interaction + AR
! HMD AR View ! Viewpoint tracking
! Two hand input ! Skeleton interaction, occlusion
Piumsomboon, T., Altimira, D., Kim, H., Clark, A., Lee, G., & Billinghurst, M. (2014, September). Grasp-Shell vs gesture-speech: A comparison of direct and indirect natural interaction techniques in augmented reality. In Mixed and Augmented Reality (ISMAR), 2014 IEEE International Symposium on (pp. 73-82). IEEE.
Multimodal Interaction ! Combined speech and gesture input ! Gesture and Speech complimentary
! Speech: modal commands, quantities ! Gesture: selection, motion, qualities
! Previous work found multimodal interfaces intuitive for 2D/3D graphics interaction ! However, few multimodal AR interfaces
Free Hand Multimodal Input
! Use free hand to interact with AR content ! Recognize simple gestures
! Open hand, closed hand, pointing
Point Move Pick/Drop
Lee, M., Billinghurst, M., Baek, W., Green, R., & Woo, W. (2013). A usability study of multimodal input in an augmented reality environment. Virtual Reality, 17(4), 293-305.
Multimodal Architecture
Experimental Setup
Change object shape and colour
Results - Performance ! Average performance time
! Gesture: 15.44s ! Speech: 12.38s ! Multimodal: 11.78s
! Significant difference across conditions (p < 0.01) ! Difference between gesture and speech/MMI
Subjective Results (Likert 1-7)
! User subjective survey ! Gesture significantly worse, MMI and Speech same ! MMI perceived as most efficient
! Preference ! 70% MMI, 25% speech only, 5% gesture only
Gesture Speech MMI
Naturalness 4.60 5.60 5.80
Ease of Use 4.00 5.90 6.00
Efficiency 4.45 5.15 6.05
Physical Effort 4.75 3.15 3.85
Lessons Learned ! Multimodal interaction significantly better than
gesture alone in AR interfaces for 3D tasks ! Shorter task time, more efficient
! Multimodal input was more natural, easier, and more effective that gesture/speech only ! Simultaneous input rarely used
! More studies need to be conducted ! What gesture/speech patterns? Richer input
Intelligent Interfaces
! AR interface + intelligent tutoring system ! ASPIRE constraint based system (from UC) ! Constraints
- relevance cond., satisfaction cond., feedback
Westerfield, G., Mitrovic, A., & Billinghurst, M. (2013). Intelligent Augmented Reality Training for Assembly Tasks. In Artificial Intelligence in Education (pp. 542-551). Springer Berlin Heidelberg.
Intelligent Feedback
! Actively monitors user behaviour ! Implicit vs. explicit interaction
! Provides corrective feedback
Evaluation Results ! 16 subjects, with and without ITS ! Improved task completion
! Improved learning
MagicLeap (2014)
! How much is real?
! Digital Imperceptible from the Real World ! Multi-sensory Display ! Natural unencumbered input ! Seamless blend between real and virtual
! Huge technical and non-technical challenges
The Ultimate AR Display
AR Research Trends ! Zhou 10 year ISMAR survey (1998-2008)
! 276 papers reviewed
! Most researched topics ! (1) Tracking techniques (20%) ! (2) Interaction techniques (15%) ! (3) Calibration and registration (14%) ! (4) AR applications (14%) ! (5) Display techniques (12%)
Feng Zhou, Henry B.L. Duh, Mark Billinghurst. Trends in Augmented Reality Tracking, Interaction and Display: A Review of Ten Years of ISMAR. In proceedings of ISMAR 2008, Cambridge, UK, 15-18th of Sept. 2008.
Tracking
2005 2015
Marker Based ARToolKit Single Sensor Local
Object Based Vuforia Smart Terrain Hybrid Systems Wide area
Input
2005 2015
Tangible AR Custom controller Sensor based
Natural gesture Mobile touch Vision based
2005 2015
Video See-through Desktop Projected Custom
Optical See-through Wearable Handheld Consumer
Display
Rendering
2005 2015
Simple graphics No environmental effects No lighting capture
Photorealistic Scene capture/shading Real time lighting
Better Tools Than Every Before ! Cheap/Free Software
! Vuforia, Junaio, etc
! Wide Range of Displays ! AR, VR, Projected, etc
! Input Devices ! Eye-tracking, depth sensing, body measuring
! New Funding Models ! Kickstarter, Indiegogo, Investor
Research Gaps ! Least Researched Topics
! (6) Evaluation/testing (6%); ! (7) Mobile/wearable AR (6%); ! (8) Visualization (5%); ! (9) AR authoring (4%); ! (10) Multimodal AR (3%); ! (11) Rendering (2%).
Research Opportunities ! Tracking
! 3D scene capture, ubiquitous tracking, middleware
! Display ! Projection displays, retinal displays, contact lens
! Interaction ! Collaborative, Ubiquitous computing, Tangible Input
Is AR the Right Tool ?
! Which tasks are suitable for AR? ! Eg AR vs Map for outdoor navigation
No One Wants to be a Glasshole
! Social impact of AR technology ! Privacy, Social Interactions, Cultural Impact
MY VISION
! sdfs
! sdfgs
! axcvxca
Vision
“Using technology to create shared emotional experiences between users
and so create a deeper sense of empathy and understanding”
Empathic Computing 1. Computing systems that can understand your feelings and emotions 2. Computing systems that help you better understand the feelings of others
Affective Computing
! Ros Picard – MIT Media Lab ! http://affect.media.mit.edu
Appliances That Make You Happy
! Jun Rekimoto – Univ. Tokyo ! Smile detection + smart devices
Can we develop interfaces that allow us to be more empathetic to others?
Movies are like a machine that generates Empathy
Roger Ebert
Empathy Computing Requirements
! Basic Requirements ! Making the technology transparent
! Empathy Definition ! Seeing with the eyes of another ! Hearing with the ears of another ! Feeling with the heart of another
Using AR for Empathy
! Augmented Reality can: ! Remove technology barriers ! Enhance communication ! Change perspective ! Share experiences ! Enhance interaction in real world
Wearable Computer
▪ Wearable computer – Ego-Vision view ▪ camera + processing + display + connectivity
Current Collaboration on Wearables
! First person remote conferencing/hangouts ! Limitations
! Single POV, no spatial cues, no annotations, etc
Sharing Space: Social Panoramas
! Capture and share social spaces in real time ! Enable remote people to feel like they’re with you
Technology
! Google Glass ! Image capture, viewpoint sharing
! Remote device (desktop, tablet) ! Immersive viewing, live annotation
Key Research Questions
! Where is my partner looking? ! Enhanced radar display, Context compass
! How can we interact together? ! Shared pointers, Shared drawing
Sensor Systems/Software
! Different systems developed/evaluated ! SSI framework (emoVoice, etc) ! e-Health, bitalino, arduino bio-sensors ! Wearscript wearable eye-tracker
!
!
Approach Data Capture
Feature Detection
Emotion Recognition
Emotion Representation
Empathic User Interface
Hardware
User Interface
Wearable Interface
! Google Glass + e-Health + Spydroid + SSI ! Measure GSR, pulse oxygen, ECG, pitch ! Share video and audio remotely ! Represent emotions back to Glass user (4 states)
! !
Desktop Interface
! Live video, real time emotion data ! See what sender sees, emotion representation
!
Emotion Representation
! How can we show what you’re feeling? ! Tested - Raw data, Visual tinting, Emotion labeling
!
! !Excited Happy
Early Results ! Colour overlay + video stream preferred
! Easier to understand
! Disagreement over best colour coding ! Different emotional response with diff. stimuli
! Scary movie best
! Remote users felt connected to local user ! Understanding of emotions
Table 1. Correlation Analysis between the local users and the remote users
Analysis
Table 2. Correlation between different streaming scenarios obtained from the feedback of the local and remote users
Experiment 3 (Creating a shared experience while watching an audio Visual Watching an audio visual)
Happy Sad Neutral Excited Case1
(Only video stream) 0.77 0.84 0.85 0.85 Case 2 (Case 1 + Color Overlay) 0.92 0.91 0.85 0.98
Case 3 (Case 2 + Graphical Analysis) 0.86 0.75 0.85 0.94
Capturing Space: Real World Capture
! Hands free AR ! Portable scene capture (color + depth)
! Projector/Kinect combo, Remote controlled pan/tilt
! Remote expert annotation interface
Remote Expert View
Ghostman
! Use AR to capture and overlay your actions into remote persons space ! Eg remote therapy
Looking to the Future
What’s Next?
Brain to Brain Control
! Rajesh Rao, University of Washington ! First Brain to Brain control
System Architecture
Scaling Up
! Seeing actions of millions of users in the world ! Augmentation on city/country level
AR + Smart Sensors + Social Networks
! Track population at city scale (mobile networks) ! Match population data to external sensor data
! medical, environmental, etc
! Mine data to improve social services
Research Challenges ! How to convey emotion? ! How to measure empathy? ! Interface/interaction models? ! How to communicate emotion? ! Scaling up to city/country scale?
CONCLUSIONS
Take Home Messages ! Have a Vision ! Find people to travel with ! Equip the vehicle ! Enjoy the journey
Most Important: We’re nowhere near the end of interesting research in Augmented Reality
