Introduction to Optical See-Through Head-Mounted Displays in Augmented Reality

Yuta Itoh23th May, 2016

~2016.03: Ph.D. at TU Munich,@Germany

2016.04~: Project Assistant Prof.@Sugimoto lab

Yuta Itoh

Research topics:OST-HMDs, Eye tracking, AH

Research: http://campar.in.tum.de/Main/YutaItoh

Augmented Reality (AR)

©IKEA, 2013

Augmented Reality (AR)

Feiner, Steven, Blair Macintyre, and DoréeSeligmann. "Knowledge-based augmented reality." Communications of the ACM 36.7 (1993): 53-62.

Hirota, Gentaro, et al. "Superior augmented reality registration by integrating landmark tracking and magnetic tracking." ACM SIGGRAPH 1996.

Fuse virtual contents into the reality

Augmented Reality (AR)

[1] Klinker et al. ISMR’99, [2] Kato and Bilinghurst, [3] Kaufmann and Schmalstieg 2003 [3] IWAR’99, Sony Computer Entertainment, [4] Schwerdtfeger et al. VRST 2008, [5] Bichlmeier et al. ISMAR 20017

Augmented Reality (AR)

Research collections from FAR, TU Munich (http://campar.in.tum.de/Chair/ResearchAr)

Fuchs, Henry, et al. "Optimizing a head-tracked stereo display system to guide hepatic tumor ablation." Studies in health

technology and informatics 132 (2007): 126-131.

Augmented Reality (AR)

Fuchs, Henry, et al. Augmented reality visualization for laparoscopic surgery. Springer Berlin Heidelberg, 1998.

Non-Optical (Video) See-Through

Head-Mounted Displays (HMD)

Optical See-Through (OST-HMD)

VR HMDs

VideoSee-Thro.

OpticalSee-Thro.

Opaque display

Scene camera

Sight ©Robot Genius Film, 2012https://vimeo.com/46304267

Hugo Gernsback, TV Glasses

1963

Ivan Sutherland, The Sword of Damocles

1968

Ivan Sutherland, The Sword of Damocles

1968

2000- (VR HMD)

Toshiba, 2006 Oculus Rift DK2, 2014Photo from IFIXIT

2013-

2013-

Google Glass, 2013Photo by Mikepanhu

Don't Be A Glasshole | Mashable

2016 (VR HMDs)

Oculus Rift, 2016HTC Vive, 2016Photo from Tom's Hardware

2016

Meta 2, 2016 ODG R-7, 2016

EPSON BT300, 2016

Kotaku, Minecraft Hololens demo at E3 2015https://www.youtube.com/watch?v=xgakdcEzVwg

2016-

https://twitter.com/ksasao/status/715569511684911105

Microsoft, Hololens

2016-

Magic Leap, 2016

©Microsoft, 2015, 2016

How to make AR real?

Issues on OST-HMDsDesign Factors

Photo by Mikepanhu

Field of ViewIssues on OST-HMD

https://twitter.com/ksasao/status/715569511684911105

Field of ViewIssues on OST-HMD

“Pinlight Displays: Wide Field of View Augmented Reality Eyeglassesusing Defocused Point Light Sources” Maimone et al., TOG’14

Issues on OST-HMD

“Pinlight Displays: Wide Field of View Augmented Reality Eyeglassesusing Defocused Point Light Sources” Maimone et al., TOG’14

Field of View

Realism in AR (with OST HMDs)

Visual

[Kiyokawa ’01] [Liu ’08] [Lee ’09]

Spatial

[Azuma ’95] [Tuceryan ’00] [Genc ’02] [Grubert ‘08]

Temporal

[Zheng ’14] [Lincoln ’16]

Realism in AR (with OST HMDs)

Visual

[Kiyokawa ’01] [Liu ’08] [Lee ’09]

Spatial

[Azuma ’95] [Tuceryan ’00] [Genc ’02] [Grubert ‘08]

Temporal

[Zheng ’14] [Lincoln ’16]

Temporal Realism in OST-HMDs

“How fast is fast enough? : a study of the effects of latency in direct-touch pointing tasks” Jota et al. CH’13

https://www.youtube.com/watch?v=PCbSTj7LjJg

“latencies down to 2.38 ms are required to alleviate user perception when dragging”

Temporal Realism in OST-HMDsDigital Light Processing Projector

“Minimizing Latency for Augmented Reality Displays: Frames Considered Harmful” Zheng et al. ISMAR’14

Temporal Realism in OST-HMDs

Lincoln, Peter, et al. "From Motion to Photons in 80 Microseconds: Towards Minimal Latency for Virtual and Augmented Reality." (2016), IEEE VR 2016 (Best paper award)

Realism in AR (with OST HMDs)

Visual

[Kiyokawa ’01] [Liu ’08] [Lee ’09]

Spatial

[Azuma ’95] [Tuceryan ’00] [Genc ’02] [Grubert ‘08]

Temporal

[Zheng ’14] [Lincoln ’16]

Visual Realism (including photorealism)

Occlusion, Depth, Color, etc..

Lee and woo, 2009

Kiyokawa et al. 2003Liu et al. 2008

Occlusion (physical “Z-buffer”)

Kiyokawa et al. 2003

Depth (eye accommodation)

Liu et al. 2008

Konrad et al. 2016

Depth (eye accommodation)

The Light Field Stereoscope,Huan et al., SIGGRPAH 2015

Realism in AR (with OST HMDs)

Visual

[Kiyokawa ’01] [Liu ’08] [Lee ’09]

Spatial

[Azuma ’95] [Tuceryan ’00] [Genc ’02] [Grubert ‘08]

Temporal

[Zheng ’14] [Lincoln ’16]

Grubert, J., Tümler, J., & Mecke, R. (2008). Untersuchungen zur optimierung der see-through-kalibrierung fuer mobile augmented reality assistenzsysteme. Michael Schenk, Hrsg, 6(7).

Spatial Realism in OST-HMDs

Spatial Realism in OST-HMDs

Optics Designs of OST-HMDs

(Free-form)prism

Point source

Waveguide

Laser source

Half-mirror

Eye

Prism-based

Waveguide-based

Half-mirror

Hainich and Bimber, Displays Fundamentals and Applications, 2011

Retinal Scanning

Eye-HMD Calibration

3D2D

Eye-HMD Calibration

3D

Off-axis Pinhole camera

3D2D

Manual method (De facto standard)

[Tuceryan ’00] [Genc ’02]

SPAAMSingle Point Active Alignment Method

SPAAM (Manual Calibration) video

Track the position!

Itoh and Klinker, IEEE 3DUI 2014, IEEE ISMAR 2014

Automated method

OST-HMD with eye-tracking

Hua and Gao, SPIE, 2012

VR-HMD with eye-tracking

Stengel et al., ACM Multimedia, 2015

Better understandings ofHMD and Eye-ball optics

Locational Realism revisited

EyeEnvironment

Medium

View-Dependent Lens Distortion

Distorted ray

HMDoptics

Image source

Itoh and Klinker IEEE TVCG ‘15 (VR ’15) & IEEE ISMAR ‘15

Light Field Calibration

World camera

User-view camera

Itoh and Klinker IEEE TVCG ‘15 (VR ’15) & IEEE ISMAR ‘15

Learned distortion maps

IEEE TVCG ‘15 (VR ’15) & IEEE ISMAR ‘15

View-Dependent Point-Spread Func.

Corneal-Imaging Calibration

Plopski et al., IEEE TVCG ‘15 (VR ’15)

Schematic Eye for Virtual Environments

Jones et al., IEEE 3DUI 2016

What would happen in 20XX???

Sight ©Robot Genius Film, 2012

Vision Augmentation

Analysis of individual eyes

Computational photography

+

Low-vision devices

Peli, Eli. "Vision multiplexing: an engineering approach to vision rehabilitation device development." Optometry & Vision Science 78.5 (2001): 304-315.

Peli, Eli. "Treating with spectacle lenses: a novel idea!?." Optometry & Vision Science 79.9 (2002): 569-580.

Enhancing Color

Toshiyuki Amano, Oliver Bimber, and Anselm Grundhöfer. "Appearance Enhancement for Visually Impaired with Projector Camera Feedback.“, TECHNICAL REPORT, BAUHAUS-UNIVERSITY WEIMAR, JANUARY 2009

Broecker, Markus, Ross T. Smith, and Bruce H. Thomas. "Adaptive substrate for enhanced spatial augmented reality contrast and resolution." transport 3.6 (2011): 22.

Enhancing Contrast

Huang, Fu-Chung, et al. "Computational light field display for correcting visual aberrations." ACM SIGGRAPH 2013 Posters. ACM, 2013.

De-blurring displays

De-blurring OST-HMD screen

Oshima et al. ISMAR’15

De-blurring eye via OST-HMD

Itoh and Klinker, Augmented Human ‘15

Compensation layer

Blurred

Enhanced

Itoh and Klinker, Augmented Human ‘15

Sharp

In Science Fiction, a precog sees his short-term future…

Next, Nicholas Cage, TM & © Paramount (2007)

Vision Predication

Itoh et al., Augmented Human ‘16

A proof-of-concept system

Tracking system

User-view Camera

OST-HMD

Virtual Image plane

Tracking system

User-view Camera

OST-HMD

Virtual Image plane

A proof-of-concept system

Realism in OST-HMDs Better understandings of

Tracking, Graphics, HMD optics, Eye balls, etc.

Summary

Vision Augmentation Extend human capability

Questions & comments: yuta.itoh@keio.jpResearch: http://campar.in.tum.de/Main/YutaItoh

Resources related to OST-HMD&AR

“Optical See-Through AR Calibration: Methods for Current & Next Generation Head-Mounted Displays”, Kenneth Moserhttps://sites.google.com/site/krm104/project-updates/ieeevr2016tutorialslidesandmaterials

“How to calibrate your Optical See-Through Head Mounted Displays”, Jens Grubert and Yuta Itohenneth Moserhttp://www.slideshare.net/JensGrubert/google-glass-the-meta-and-co-how-to-calibrate-your-optical-seethrough-head-mounted-displays

“2016 AR Summer School ”, Mark Billinghurst,http://www.slideshare.net/marknb00/2016-ar-summer-school-lecture1