Download - COMP 4010: Lecture8 - AR Technology
![Page 1: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/1.jpg)
LECTURE 8: AR TECHNOLOGY
COMP 4010 – Virtual RealitySemester 5 – 2017
Bruce Thomas, Mark BillinghurstUniversity of South Australia
October 5th 2017
![Page 2: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/2.jpg)
Augmented Reality
1977 – Star Wars
![Page 3: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/3.jpg)
Augmented Reality Definition
• Defining Characteristics [Azuma 97]:• Combines Real and Virtual Images
• Both can be seen at the same time• Interactive in real-time
• The virtual content can be interacted with• Registered in 3D
• Virtual objects appear fixed in space
Azuma, R. T. (1997). A survey of augmented reality. Presence, 6(4), 355-385.
![Page 4: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/4.jpg)
Technology Requirements
• Combining Real and Virtual Images• Needs - Display technologies
• Interactive in Real-Time• Needs - Input and interactive technologies
• Registered in 3D• Needs - Viewpoint tracking technologies Display
Processing
Input Tracking
![Page 5: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/5.jpg)
AR DISPLAYS
![Page 6: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/6.jpg)
Display Technologies• Types (Bimber/Raskar 2005)• Head attached
• Head mounted display/projector• Body attached
• Handheld display/projector• Spatial
• Spatially aligned projector/monitor
![Page 7: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/7.jpg)
Bimber, O., & Raskar, R. (2005). Spatial augmented reality: merging real and virtual worlds. CRC press.
Display Taxonomy
![Page 8: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/8.jpg)
HEAD MOUNTED DISPLAYS
![Page 9: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/9.jpg)
Head Mounted Displays (HMD)• Display and Optics mounted on Head• May or may not fully occlude real world• Provide full-color images• Considerations
• Cumbersome to wear• Brightness• Low power consumption• Resolution limited• Cost is high?
![Page 10: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/10.jpg)
Types of Head Mounted Displays
OccludedSee-thru
Multiplexed
![Page 11: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/11.jpg)
Immersive VR Architecture
HeadTracker
Host Processor
Data BaseModel
RenderingEngine Frame
Buffer
head position/orientation
to networkDisplayDriver
Non see-thru
Image source &
optics
virtualobject
VirtualWorld
![Page 12: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/12.jpg)
See-thru AR Architecture
HeadTracker
Host Processor
Data BaseModel
RenderingEngine Frame
Buffer
head position/orientation
to networkDisplayDriver
see-thrucombiner
Virtual Image superimposedover real world object
real world
Image source
![Page 13: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/13.jpg)
Optical see-through Head-Mounted Display
Virtual imagesfrom monitors
RealWorld
OpticalCombiners
![Page 14: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/14.jpg)
Optical See-Through HMD
![Page 15: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/15.jpg)
Example: Epson Moverio BT-200
▪ Stereo see-through display ($500)▪ 960 x 540 pixels, 23 degree FOV, 60Hz, 88g▪ Android Powered, separate controller▪ VGA camera, GPS, gyro, accelerometer
![Page 16: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/16.jpg)
View Through Optical See-Through HMD
![Page 17: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/17.jpg)
Other Optical See-Through Displays• Microsoft HoloLens - $3,000 USD
• Wearable computer• Waveguide displays• https://www.microsoft.com/hololens
• Meta Meta2 - $1,500 USD• Wide field of view (90 degrees)• Tethered display• https://www.metavision.com/
• Mira Prism - $100 USD• Smart phone based• Wide field of view• https://www.mirareality.com/
![Page 18: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/18.jpg)
Example: HoloLens
• https://www.youtube.com/watch?v=AaTyeDtht-8
![Page 19: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/19.jpg)
Strengths of Optical see-through AR
• Simpler (cheaper)• Direct view of real world
• Full resolution, no time delay (for real world)• Safety• Lower distortion
• No eye displacement• but some video see-through displays avoid this
![Page 20: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/20.jpg)
Video AR Architecture
HeadTracker
Host Processor
Graphicsrenderer
DigitalMixer Frame
Buffer
head position/orientation
to networkDisplayDriver
Non see-thruImage source &
optics
Head-mounted camera aligned to
display optics
VideoProcessor
Video image of real world
Virtual image inset into video of real world
![Page 21: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/21.jpg)
Video see-through HMD
Videocameras
Monitors
Graphics
Combiner
Video
![Page 22: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/22.jpg)
Video See-Through HMD
![Page 23: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/23.jpg)
Vuzix Wrap 1200DXAR
▪ Stereo video see-through display ($1500)■Twin 852 x 480 LCD displays, 35 deg. FOV ■Stereo VGA cameras■3 DOF head tracking■https://www.vuzix.com/Products/LegacyProduct/4
![Page 24: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/24.jpg)
View Through a Video See-Through HMD
![Page 25: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/25.jpg)
Strengths of Video See-Through AR• True occlusion
• AR pixels block video pixels
• Digitized image of real world• Flexibility in composition• Matchable time delays• More registration, calibration strategies
• Wide FOV is easier to support
![Page 26: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/26.jpg)
Eye Multiplexed AR Architecture
HeadTracker
Host Processor
Data BaseModel
RenderingEngine Frame
Buffer
head position/orientation
to networkDisplayDriver
Virtual Image inset intoreal world scene
real world
OpaqueImage source
![Page 27: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/27.jpg)
Virtual Image ‘inset’ into Real World
![Page 28: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/28.jpg)
Example: Google Glass
![Page 29: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/29.jpg)
View Through Google Glass
https://www.youtube.com/watch?v=zKNv505sYAM
![Page 30: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/30.jpg)
![Page 31: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/31.jpg)
Vuzix M-100
▪ Monocular multiplexed display ($1000)■852 x 480 LCD display, 15 deg. FOV ■5 MP camera, HD video■GPS, gyro, accelerometer
![Page 32: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/32.jpg)
Display Technology
• Curved Mirror• off-axis projection• curved mirrors in front of eye• high distortion, small eye-box
• Waveguide• use internal reflection• unobstructed view of world• large eye-box
![Page 33: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/33.jpg)
Example: Sony Waveguide Display
• https://www.youtube.com/watch?v=G2MtI7asLcA
![Page 34: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/34.jpg)
See-through Thin Displays (Waveguide)
• Waveguide techniques for thin see-through displays• Wider FOV, enable AR applications• Social acceptability
Opinvent Ora
Lumus DK40
![Page 35: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/35.jpg)
Example: Sony Smart EyeGlasses
• https://www.youtube.com/watch?v=kYPWaMsarss
![Page 36: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/36.jpg)
SPATIAL AUGMENTED REALITY
![Page 37: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/37.jpg)
Spatial Augmented Reality
• Project onto irregular surfaces• Geometric Registration• Projector blending, High dynamic range
• Book: Bimber, Rasker “Spatial Augmented Reality”
![Page 38: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/38.jpg)
Projector-based AR
Examples:Raskar, MIT Media LabInami, Tachi Lab, U. Tokyo
Projector
Real objectswith retroreflectivecovering
User (possiblyhead-tracked)
![Page 39: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/39.jpg)
Example of Projector-Based AR
Ramesh Raskar, MIT
![Page 40: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/40.jpg)
Example of Projector-Based AR
Ramesh Raskar, MIT
![Page 41: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/41.jpg)
Head Mounted Projector
• NVIS P-50 HMPD• 1280x1024/eye• Stereoscopic• 50 degree FOV• www.nvis.com
![Page 42: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/42.jpg)
HMD vs. HMPD
Head Mounted Display Head Mounted Projected Display
![Page 43: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/43.jpg)
CastAR - http://technicalillusions.com/
• Stereo head worn projectors• Interactive wand• Rollable retro-reflective sheet
![Page 44: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/44.jpg)
•Designed for shared interaction
![Page 45: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/45.jpg)
Demo: CastAR
https://www.youtube.com/watch?v=AOI5UW9khoQ#t=47
![Page 46: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/46.jpg)
Pico Projectors
• Microvision - www.mvis.com• 3M, Samsung, Philips, etc
![Page 47: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/47.jpg)
Demo: Pico Projector
https://www.youtube.com/watch?v=vtdH3CLKEuY
![Page 48: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/48.jpg)
MIT Sixth Sense
• Body worn camera and projector• http://www.pranavmistry.com/projects/sixthsense/
![Page 49: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/49.jpg)
Demo: Sixth Sense
https://www.youtube.com/watch?v=Q4Z9sOtiWUY
![Page 50: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/50.jpg)
OTHER AR DISPLAYS
![Page 51: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/51.jpg)
Video Monitor AR
Videocameras Monitor
Graphics Combiner
Video
Stereoglasses
![Page 52: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/52.jpg)
Examples
![Page 53: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/53.jpg)
Virtual Showcase
• Mirrors on a projection table• Head tracked stereo• Up to 4 users• Merges graphic and real objects• Exhibit/museum applications
• Fraunhofer Institute (2001)• Bimber, Frohlich
Bimber, O., Fröhlich, B., Schmalstieg, D., & Encarnação, L. M. (2006, July). The virtual showcase. In ACM SIGGRAPH 2006 Courses (p. 9). ACM.
![Page 54: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/54.jpg)
Demo: Virtual Showcase
https://www.youtube.com/watch?v=iXl4FolFUzc
![Page 55: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/55.jpg)
Augmented Paleontology
Bimber et. al. IEEE Computer Sept. 2002
![Page 56: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/56.jpg)
Alternate Displays
LCD Panel Laptop PDA
![Page 57: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/57.jpg)
Handheld Displays
•Mobile Phones• Camera• Display • Input
![Page 58: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/58.jpg)
Other Types of AR Display
• Audio• spatial sound• ambient audio
• Tactile • physical sensation
• Haptic• virtual touch
![Page 59: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/59.jpg)
Haptic Input
• AR Haptic Workbench• CSIRO 2003 – Adcock et. al.
![Page 60: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/60.jpg)
Phantom
• Sensable Technologies (www.sensable.com)• 6 DOF Force Feedback Device
![Page 61: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/61.jpg)
Example: Haptic Augmented Reality
• https://www.youtube.com/watch?v=3meAlle8kZs
![Page 62: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/62.jpg)
AR TRACKING AND REGISTRATION
![Page 63: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/63.jpg)
AR Requires Tracking and Registration
• Registration• Positioning virtual object wrt real world
• Fixing virtual object on real object when view is fixed • Tracking
• Continually locating the users viewpoint when view moving • Position (x,y,z), Orientation (r,p,y)
![Page 64: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/64.jpg)
AR TRACKING
![Page 65: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/65.jpg)
Tracking Requirements
• Augmented Reality Information Display • World Stabilized• Body Stabilized• Head Stabilized
Increasing Tracking Requirements
Head Stabilized Body Stabilized World Stabilized
![Page 66: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/66.jpg)
Tracking Technologies
§ Active• Mechanical, Magnetic, Ultrasonic• GPS, Wifi, cell location
§ Passive• Inertial sensors (compass, accelerometer, gyro)• Computer Vision
• Marker based, Natural feature tracking
§ Hybrid Tracking• Combined sensors (eg Vision + Inertial)
![Page 67: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/67.jpg)
Tracking Types
Magnetic Tracker
Inertial Tracker
Ultrasonic Tracker
Optical Tracker
Marker-Based Tracking
Markerless Tracking
Specialized Tracking
Edge-Based Tracking
Template-Based Tracking
Interest Point Tracking
Mechanical Tracker
![Page 68: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/68.jpg)
Mechanical Tracker
• Idea: mechanical arms with joint sensors
•++: high accuracy, haptic feedback • -- : cumbersome, expensive
Microscribe
![Page 69: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/69.jpg)
Magnetic Tracker• Idea: coil generates current when moved in magnetic field. Measuring current gives position and orientation relative to magnetic source.
• ++: 6DOF, robust • -- : wired, sensible to metal, noisy, expensive
Flock of Birds (Ascension)
![Page 70: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/70.jpg)
Inertial Tracker• Idea: measuring linear and angular orientation rates (accelerometer/gyroscope)
• ++: no transmitter, cheap, small, high frequency, wireless • -- : drifts over time, hysteresis effect, only 3DOF
IS300 (Intersense) Wii Remote
![Page 71: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/71.jpg)
Ultrasonic Tracker
• Idea: time of Flight or phase-Coherence Sound Waves
• ++: Small, Cheap • -- : 3DOF, Line of Sight, Low resolution, Affected by environmental conditons (pressure, temperature)
Ultrasonic Logitech IS600
![Page 72: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/72.jpg)
Global Positioning System (GPS)
• Created by US in 1978• Currently 29 satellites
• Satellites send position + time • GPS Receiver positioning
• 4 satellites need to be visible• Differential time of arrival• Triangulation
• Accuracy• 5-30m+, blocked by weather, buildings etc.
![Page 73: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/73.jpg)
Mobile Sensors
• Inertial compass• Earth’s magnetic field• Measures absolute orientation
• Accelerometers• Measures acceleration about axis• Used for tilt, relative rotation• Can drift over time
![Page 74: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/74.jpg)
OPTICAL TRACKING
![Page 75: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/75.jpg)
Why Optical Tracking for AR?
• Many AR devices have cameras• Mobile phone/tablet, Video see-through display
• Provides precise alignment between video and AR overlay• Using features in video to generate pixel perfect alignment• Real world has many visual features that can be tracked from
• Computer Vision well established discipline• Over 40 years of research to draw on• Old non real time algorithms can be run in real time on todays devices
![Page 76: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/76.jpg)
Common AR Optical Tracking Types
• Marker Tracking• Tracking known artificial markers/images
• e.g. ARToolKit square markers
• Markerless Tracking• Tracking from known features in real world
• e.g. Vuforia image tracking
• Unprepared Tracking• Tracking in unknown environment
• e.g. SLAM tracking
![Page 77: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/77.jpg)
Marker tracking
• Available for more than 10 years• Several open source solutions exist
• ARToolKit, ARTag, ATK+, etc
• Fairly simple to implement• Standard computer vision methods
• A rectangle provides 4 corner points• Enough for pose estimation!
![Page 78: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/78.jpg)
Demo: ARToolKit
• https://www.youtube.com/watch?v=TqGAqAFlGg0
![Page 79: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/79.jpg)
Marker Based Tracking: ARToolKit
http://www.artoolkit.org
![Page 80: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/80.jpg)
Goal: Find Camera Pose
• Goal is to find the camera pose in maker coordinate frame• Knowing:
• Position of key points in on-screen video image• Camera properties (focal length, image distortion)
![Page 81: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/81.jpg)
Coordinates for Marker Tracking
![Page 82: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/82.jpg)
Coordinates for Marker Tracking
Marker Camera•Final Goal•Rotation & Translation
1: Camera Ideal Screen•Perspective model•Obtained from Camera Calibration
2: Ideal Screen Observed Screen•Nonlinear function (barrel shape)•Obtained from Camera Calibration
3: Marker Observed Screen•Correspondence of 4 vertices•Real time image processing
![Page 83: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/83.jpg)
Marker Tracking – OverviewFiducial
DetectionRectangle
Fitting
Pattern Checking
Lens Undistortion
Pose Estimation
Identified Markers
Camera Image Contours
Rectangles
Estimated Poses
Undistorted Corners
![Page 84: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/84.jpg)
Marker Tracking – Fiducial Detection
• Threshold the whole image to black and white• Search scanline by scanline for edges (white to black)• Follow edge until either
• Back to starting pixel• Image border
• Check for size• Reject fiducials early that are too small (or too large)
![Page 85: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/85.jpg)
Marker Tracking – Rectangle Fitting• Start with an arbitrary point “x” on the contour• The point with maximum distance must be a corner c0• Create a diagonal through the center• Find points c1 & c2 with maximum distance left and right of diag.• New diagonal from c1 to c2• Find point c3 right of diagonal with maximum distance
![Page 86: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/86.jpg)
Marker Tracking – Pattern checking• Calculate homography using the 4 corner points
• “Direct Linear Transform” algorithm• Maps normalized coordinates to marker coordinates
(simple perspective projection, no camera model)
• Extract pattern by sampling and check• Id (implicit encoding)• Template (normalized cross correlation)
![Page 87: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/87.jpg)
Marker Tracking – Corner refinement• Refine corner coordinates
• Critical for high quality tracking• Remember: 4 points is the bare minimum!• So these 4 points should better be accurate…
• Detect sub-pixel coordinates• E.g. Harris corner detector
• Specialized methods can be faster and more accurate
• Strongly reduces jitter!• Undistort corner coordinates
• Remove radial distortion from lens
![Page 88: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/88.jpg)
Marker tracking – Pose estimation
• Calculates marker pose relative to the camera• Initial estimation directly from homography
• Very fast, but coarse with error• Jitters a lot…
• Iterative Refinement using Gauss-Newton method• 6 parameters (3 for position, 3 for rotation) to refine• At each iteration we optimize on the error• Iterat
![Page 89: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/89.jpg)
From Marker To Camera• Rotation & Translation
TCM : 4x4 transformation matrixfrom marker coord. to camera coord.
![Page 90: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/90.jpg)
Tracking challenges in ARToolKit
False positives and inter-marker confusion(image by M. Fiala)
Image noise(e.g. poor lens, block
coding /compression, neon tube)
Unfocused camera, motion blur
Dark/unevenly lit scene, vignetting
Jittering (Photoshop illustration)
Occlusion(image by M. Fiala)
![Page 91: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/91.jpg)
Other Marker Tracking Libraries
![Page 92: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/92.jpg)
But - You can’t cover world with ARToolKit Markers!
![Page 93: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/93.jpg)
Markerless Tracking
Magnetic Tracker Inertial Tracker
Ultrasonic Tracker
Optical Tracker
Marker-Based Tracking
Markerless Tracking
Specialized Tracking
Edge-Based Tracking
Template-Based Tracking
Interest Point Tracking
• No more Markers! èMarkerless Tracking
Mechanical Tracker
![Page 94: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/94.jpg)
Natural Feature Tracking• Use Natural Cues of Real Elements
• Edges• Surface Texture • Interest Points
• Model or Model-Free• No visual pollution
Contours
Features Points
Surfaces
![Page 95: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/95.jpg)
Texture Tracking
![Page 96: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/96.jpg)
Tracking by Keypoint Detection
• This is what most trackers do…• Targets are detected every frame• Popular because
tracking and detectionare solved simultaneously
Keypoint detection
Descriptor creationand matching
Outlier Removal
Pose estimationand refinement
Camera Image
Pose
Rec
ogni
tion
![Page 97: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/97.jpg)
What is a Keypoint?
• It depends on the detector you use!• For high performance use the FAST corner detector
• Apply FAST to all pixels of your image• Obtain a set of keypoints for your image• Describe the keypoints
Rosten, E., & Drummond, T. (2006, May). Machine learning for high-speed corner detection. In European conference on computer vision (pp. 430-443). Springer Berlin Heidelberg.
![Page 98: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/98.jpg)
FAST Corner Keypoint Detection
![Page 99: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/99.jpg)
Example: FAST Corner Detection
https://www.youtube.com/watch?v=pJ2xSrIXy_s
![Page 100: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/100.jpg)
Descriptors• Describe the Keypoint features• Can use SIFT
• Estimate the dominant keypointorientation using gradients
• Compensate for detectedorientation
• Describe the keypoints in termsof the gradients surrounding it
WagnerD.,Reitmayr G.,Mulloni A.,DrummondT.,Schmalstieg D.,Real-TimeDetectionandTrackingforAugmentedRealityonMobilePhones.IEEETransactionsonVisualizationandComputerGraphics,May/June,2010
![Page 101: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/101.jpg)
Database Creation• Offline step – create database of known features• Searching for corners in a static image• For robustness look at corners on multiple scales
• Some corners are more descriptive at larger or smaller scales• We don’t know how far users will be from our image
• Build a database file with all descriptors and their position on the original image
![Page 102: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/102.jpg)
Real-time tracking• Search for known keypoints
in the video image• Create the descriptors• Match the descriptors from the
live video against thosein the database• Brute force is not an option• Need the speed-up of special
data structures• E.g., we use multiple spill trees
Keypoint detection
Descriptor creationand matching
Outlier Removal
Pose estimationand refinement
Camera Image
Pose
Rec
ogni
tion
![Page 103: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/103.jpg)
NFT – Outlier removal• Removing outlining features• Cascade of removal techniques• Start with cheapest, finish with most expensive…
• First simple geometric tests• E.g., line tests
• Select 2 points to form a line• Check all other points being on correct side of line
• Then, homography-based tests
![Page 104: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/104.jpg)
NFT – Pose refinement• Pose from homography makes good starting point• Based on Gauss-Newton iteration
• Try to minimize the re-projection error of the keypoints
• Part of tracking pipeline that mostly benefits from floating point usage
• Can still be implemented effectively in fixed point• Typically 2-4 iterations are enough…
![Page 105: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/105.jpg)
NFT – Real-time tracking• Search for keypoints
in the video image• Create the descriptors• Match the descriptors from the
live video against thosein the database
• Remove the keypoints thatare outliers
• Use the remaining keypointsto calculate the pose of the camera
Keypoint detection
Descriptor creationand matching
Outlier Removal
Pose estimationand refinement
Camera Image
Pose
Rec
ogni
tion
![Page 106: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/106.jpg)
Demo: Vuforia Texture Tracking
https://www.youtube.com/watch?v=1Qf5Qew5zSU
![Page 107: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/107.jpg)
Edge Based Tracking• Example: RAPiD [Drummond et al. 02]
• Initialization, Control Points, Pose Prediction (Global Method)
![Page 108: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/108.jpg)
Demo: Edge Based Tracking
https://www.youtube.com/watch?v=Z1J0VLizVDs
![Page 109: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/109.jpg)
Line Based Tracking• Visual Servoing [Comport et al. 2004]
![Page 110: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/110.jpg)
Model Based Tracking• Tracking from 3D object shape• Example: OpenTL - www.opentl.org
• General purpose library for model based visual tracking
![Page 111: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/111.jpg)
Demo: OpenTL Model Tracking
https://www.youtube.com/watch?v=laiykNbPkgg
![Page 112: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/112.jpg)
Demo: OpenTL Face Tracking
https://www.youtube.com/watch?v=WIoGdhkfNVE
![Page 113: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/113.jpg)
Marker vs. Natural Feature Tracking• Marker tracking
• Usually requires no database to be stored• Markers can be an eye-catcher• Tracking is less demanding• The environment must be instrumented • Markers usually work only when fully in view
• Natural feature tracking• A database of keypoints must be stored/downloaded• Natural feature targets might catch the attention less• Natural feature targets are potentially everywhere• Natural feature targets work also if partially in view
![Page 114: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/114.jpg)
Tracking from an Unknown Environment• What to do when you don’t know any features?
• Very important problem in mobile robotics - Where am I?
• SLAM• Simultaneously Localize And Map the environment• Goal: to recover both camera pose and map structure
while initially knowing neither. �• Mapping:
• Building a map of the environment which the robot is in• Localisation:
• Navigating this environment using the map while keeping track of the robot’s relative position and orientation
![Page 115: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/115.jpg)
Visual SLAM
• Early SLAM systems (1986 - )• Computer visions and sensors (e.g. IMU, laser, etc.)• One of the most important algorithms in Robotics
• Visual SLAM• Using cameras only, such as stereo view • MonoSLAM (single camera) developed in 2007 (Davidson)
![Page 116: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/116.jpg)
Example: Kudan MonoSLAM
https://www.youtube.com/watch?v=g2SFJGDz9cQ
![Page 117: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/117.jpg)
How SLAM Works
• Three main steps1. Tracking a set of points through successive camera frames2. Using these tracks to triangulate their 3D position3. Simultaneously use the estimated point locations to calculate
the camera pose which could have observed them
• By observing a sufficient number of points can solve for for both structure and motion (camera path and scene structure).
![Page 118: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/118.jpg)
SLAM Optimization• SLAM is an optimisation problem
• compute the best configuration of camera poses and point positions in order to minimise reprojection error • difference between a point's tracked location and where it is expected to be
• Can be solved using bundle adjustment• a nonlinear least squares algorithm that finds minimum error• But – time taken grows as size of map increases
• Multi-core machines can do localization and mapping on different threads
• Relocalisation• Allows tracking to be restarted when it fails
![Page 119: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/119.jpg)
Evolution of SLAM Systems
• MonoSLAM (Davidson, 2007)• Real time SLAM from single camera
• PTAM (Klein, 2009)• First SLAM implementation on mobile phone
• FAB-MAP (Cummins, 2008)• Probabilistic Localization and Mapping
• DTAM (Newcombe, 2011)• 3D surface reconstruction from every pixel in image
• KinectFusion (Izadi, 2011)• Realtime dense surface mapping and tracking using RGB-D
![Page 120: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/120.jpg)
Demo: MonoSLAM
https://www.youtube.com/watch?v=saUE7JHU3P0
![Page 121: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/121.jpg)
LSD-SLAM (Engel 2014)
• A novel, direct monocular SLAM technique• Uses image intensities both for tracking and mapping.
• The camera is tracked using direct image alignment, while • Geometry is estimated as semi-dense depth maps
• Supports very large scale tracking• Runs in real time on CPU and smartphone
![Page 122: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/122.jpg)
Demo: LSD-SLAM
https://www.youtube.com/watch?v=GnuQzP3gty4
![Page 123: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/123.jpg)
Direct Method vs. Feature Based
• Direct uses all information in image, cf feature based approach that only use small patches around corners and edges
![Page 124: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/124.jpg)
Applications of SLAM Systems• Many possible applications
• Augmented Reality camera tracking• Mobile robot localisation• Real world navigation aid• 3D scene reconstruction• 3D Object reconstruction• Etc..
• Assumptions• Camera moves through an unchanging scene• So not suitable for person tracking, gesture recognition
• Both involve non-rigidly deforming objects and a non-static map
![Page 125: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/125.jpg)
Hybrid TrackingCombining several tracking modalities together
![Page 126: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/126.jpg)
Sensor Tracking
• Used by many “AR browsers”• GPS, compass, accelerometer, gyroscope• Not sufficient alone (drift, interference)
Inertial Compass Drifting Over Time
![Page 127: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/127.jpg)
Combining Sensors and Vision
• Sensors• Produces noisy output (= jittering augmentations)• Are not sufficiently accurate (= wrongly placed augmentations)• Gives us first information on where we are in the world,
and what we are looking at
• Vision• Is more accurate (= stable and correct augmentations)• Requires choosing the correct keypoint database to track from• Requires registering our local coordinate frame (online-
generated model) to the global one (world)
![Page 128: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/128.jpg)
Example: Outdoor Hybrid Tracking
• Combines• computer vision • inertial gyroscope sensors
• Both correct for each other• Inertial gyro
• provides frame to frame prediction of camera orientation, fast sensing
• drifts over time• Computer vision
• Natural feature tracking, corrects for gyro drift• Slower, less accurate
![Page 129: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/129.jpg)
Outdoor AR Tracking System
You, Neumann, Azuma outdoor AR system (1999)
![Page 130: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/130.jpg)
Robust Outdoor Tracking
• Hybrid Tracking• Computer Vision, GPS, inertial
• Going Out• Reitmayr & Drummond (Univ. Cambridge)
Reitmayr, G., & Drummond, T. W. (2006). Going out: robust model-based tracking for outdoor augmented reaity. In Mixed and Augmented Reality, 2006. ISMAR 2006. IEEE/ACM International Symposium on (pp. 109-118). IEEE.
![Page 131: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/131.jpg)
Handheld Display
![Page 132: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/132.jpg)
Example: Apple’s ARKit
https://www.youtube.com/watch?v=6xDyVBsBtX8
![Page 133: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/133.jpg)
REGISTRATION
![Page 134: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/134.jpg)
Spatial Registration
![Page 135: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/135.jpg)
The Registration Problem
• Virtual and Real content must stay properly aligned• If not:
• Breaks the illusion that the two coexist• Prevents acceptance of many serious applications
t = 0 seconds t = 1 second
![Page 136: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/136.jpg)
Sources of Registration Errors
•Static errors• Optical distortions (in HMD)• Mechanical misalignments• Tracker errors• Incorrect viewing parameters
•Dynamic errors• System delays (largest source of error)
• 1 ms delay = 1/3 mm registration error
![Page 137: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/137.jpg)
Reducing Static Errors
•Distortion compensation• For lens or display distortions
•Manual adjustments• Have user manually alighn AR and VR content
•View-based or direct measurements• Have user measure eye position
•Camera calibration (video AR)• Measuring camera properties
![Page 138: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/138.jpg)
View Based Calibration (Azuma 94)
![Page 139: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/139.jpg)
Dynamic errors
• Total Delay = 50 + 2 + 33 + 17 = 102 ms• 1 ms delay = 1/3 mm = 33mm error
Tracking Calculate ViewpointSimulation
Render Scene
Draw to Display
x,y,zr,p,y
Application Loop
20 Hz = 50ms 500 Hz = 2ms 30 Hz = 33ms 60 Hz = 17ms
![Page 140: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/140.jpg)
Reducing dynamic errors (1)
•Reduce system lag•Faster components/system modules
•Reduce apparent lag• Image deflection• Image warping
![Page 141: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/141.jpg)
Reducing System Lag
Tracking Calculate ViewpointSimulation
Render Scene
Draw to Display
x,y,zr,p,y
Application Loop
Faster Tracker Faster CPU Faster GPU Faster Display
![Page 142: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/142.jpg)
Reducing Apparent Lag
TrackingUpdate
x,y,zr,p,y
Virtual Display
Physical Display
(640x480)
1280 x 960
Last known position
Virtual Display
Physical Display
(640x480)
1280 x 960
Latest position
Tracking Calculate ViewpointSimulation
Render Scene
Draw to Display
x,y,zr,p,y
Application Loop
![Page 143: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/143.jpg)
Reducing dynamic errors (2)• Match video + graphics input streams (video AR)
• Delay video of real world to match system lag• User doesn’t notice
• Predictive Tracking• Inertial sensors helpful
Azuma / Bishop 1994
![Page 144: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/144.jpg)
Predictive Tracking
Time
Position
Past Future
Can predict up to 80 ms in future (Holloway)
Now
![Page 145: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/145.jpg)
Predictive Tracking (Azuma 94)
![Page 146: COMP 4010: Lecture8 - AR Technology](https://reader034.vdocument.in/reader034/viewer/2022050614/5a6479797f8b9a5d568b46df/html5/thumbnails/146.jpg)
Wrap-up
• Tracking and Registration are key problems• Registration error
• Measures against static error• Measures against dynamic error
• AR typically requires multiple tracking technologies• Computer vision most popular
• Research Areas: • SLAM systems, Deformable models, Mobile outdoor tracking