A Survey of Augmented A Survey of Augmented RealityReality

Ronald T. Azuma, 1997Ronald T. Azuma, 1997

A Survey of Augmented A Survey of Augmented RealityReality

2

SummarySummary

1) What is Augmented Reality (AR)?

2) Applications

3) Characteristics

4) Problems

5) Solving problems using sensing

6) Future directions

7) Conclusions

3

1.1 What is Augmented Reality?1.1 What is Augmented Reality?

Augmented Reality (AR) is a variation of Virtual Environments (VE), or Virtual Reality. These technologies completely immerse a user inside a synthetic environment. AR allows the user to see the real world (directly or indirectly), with virtual objects superimposed upon or composited with the real world.

Augmented Reality enhances a user's perception of and interaction with the real world.

Fred Brooks calls AR Intelligence Amplification (IA): using the computer as a tool to make a task easier for a human to perform.

AR supplements reality, rather than completely replacing it.

4

1.2 What is Augmented Reality?1.2 What is Augmented Reality?

Some researchers define AR in a way that requires the use of Head-Mounted Displays (HMDs). To avoid limiting AR to specific technologies, Ronald Azuma defines AR as systems that have the following three characteristics:

1) Combines real and virtual 2) Interactive in real time 3) Registered in 3D

This definition allows other technologies besides HMDs while retaining the essential components of AR.

There are three major display techniques for Augmented Reality:

Head Mounted Displays Handheld Displays Spatial Displays

AR does not include film,2D overlays, or any non-interactive technologies.

5

2.1 Applications - Summary2.1 Applications - Summary

At least six classes of potential AR applications have been explored: medical visualization, assembly, maintenance and repair, annotation and visualization, robot path planning, entertainment, military aircraft navigation and targeting.

6

2.2 Medical2.2 Medical

(MRI+CT+US) + AR = “X-ray vision”

AR surgery simulator

7

2.3 Manufacturing & Repair2.3 Manufacturing & Repair

For complex machinery: assembly, maintenance, repair.

Superimposed 3D drawings can be animated, making directions even more explicit for a manual.

Technology implies smaller costs and storage space.

BMW Augmented Reality

8

2.4 Annotation and visualization2.4 Annotation and visualization

Annotations for parts of an engine

a library with handheld device,

mapping (Bing maps 3D),

real-life labels, architecture, os interface (p-ism).

AR Tourist Guide

9

2.5 Robot path planning2.5 Robot path planning

teleoperation for time-delay and human-induced oscillations – planning and previewing tool.

the ARGOS system has demonstrated that stereoscopic AR is an easier and more accurate way of doing robot path planning than traditional monoscopic interfaces

Augmented Reality - HelicopteriPhone Control

10

2.6 Entertainment2.6 Entertainment used in 3D television sets. could be used in theater. computer games

(cargocollective.com/moreyellow). tabletop games (SCOPE) arts clothes shopping

AR Pool

11

2.7 Military Aircraft2.7 Military Aircraft

Head-up displays Helmet-mounted sights Navigation and flight

information Aiming / Targeting

Apache Targeting

12

3.1 Characteristics - 3.1 Characteristics - AugmentationAugmentation

AR also has the ability to remove objects.Could also be applied to sound.Upgraded with Haptics.

13

3.2 3.2 OpticalOptical vs. Video vs. Video

Advantages: Simple to design; Nanoseconds delay; No resolution reduction; Safe in case of no power; No eye offset;

Disadvantages: Reduces amount of light; Can’t display solid virtual

objects because of brightness differences.

Can’t perform “full” AR immersion.

14

3.2 Optical vs. 3.2 Optical vs. VideoVideo Advantages:

Flexibility in blending; Additional registration

strategies; Wide field of view; Can solve distortions; Easy to match brightness.

Disadvantages: Delay; Camera position might

confuse the user; Expensive; Dangerous in case of power

failure; More difficult to design.

15

3.3 Focus and Contrast3.3 Focus and Contrast

Focus in optical systems is difficult to achieve because the virtual image is projected at a distance away from the user. This distance should be adjusted in order for the projected objects to be able to view both elements clearly.

Contrast with optical systems is difficult because if the brightness of the virtual elements is not matched to the one of the original environment, then it could either be invisible or remove the environment.

Focus for video systems can be tricky because the graphics should be rendered to simulate a limited depth-of-field, and the video camera should have an autofocus lens.

Contrast is not such a big problem for video systems because the cameras themselves have limited dynamic response and everything has to be adjusted for the monitor.

16

3.4 Requirements3.4 Requirements

Portability is not such a big problem anymore because devices have been invented that are light enough for people to wear. Also AR has been implemented into hand-held devices.

Scene generators are not a big problem because usually the elements to not need to have such a great quality (for example annotations).

Display devices are deal with the rendering of the images, for optical it is not a problem, but for video it might reduce the resolution.

Tracking and sensing is a big requirement for AR.

17

4.1 Registration (tracking 4.1 Registration (tracking system)system)

The objects in the real and virtual worlds must be properly aligned with respect to each other. Is a big problem for video systems because visual information tends to override all other senses

(visual capture).

Static errors Dynamic errors are the ones that have no effect until either

the viewpoint or the objects begin moving. End to end system delay occurs especially in

the video systems.Solutions: Reducing system lag by sacrificing throughput

for delay. Reduce apparent lag using image deflection Match temporal streams Prediction is used to compute the movement,

better with inertial sensors.

cause registration errors even when the user's viewpoint and the objects in the environment remain completely still.

Optical distortion Additional optics Digital compensation Predistortion functions

Errors in the tracking system Discussed later

Mechanical misalignments Incorrect viewing parameters

For optical based systems can be achieved through various calibration techniques.

For video systems it is solved by taking different angle pictures and computing parameters.

18

4.2 Vision-based techniques4.2 Vision-based techniques

Using image processing or computer vision techniques to aid registration. Closed-loop feedback: feature detection in the video stream.

On some applications fiducials (markers) are used in the environment – only relative position.

Template matching searches the images for stored images.

Using laser-rangefinders to acquire an initial depth-map.

Asking the user for corrections of the differences between fusion of real and virtual.

19

5 Solving problems using 5 Solving problems using sensingsensing

Accurate tracking of the user's head and sensing the locations of other objects in the environment.

Input variety and bandwidth: CT, MRI, US, IR and UV, remote sensors would let users view hidden objects. Anything not detectable by human senses but detectable by machines is good for

AR. Range data is vital for many AR applications (laser, sonar, stereo vision) –

difficult in real time. Annotation systems require database as input.

Higher accuracy: Sensors that are accurate to around a millimeter and a tiny fraction of a degree,

across the entire working range. Every sensor type has disadvantages so combining them can compensate from

each other. Longer range:

A scalable system is one that can be expanded to cover any desired range, simply by adding more modular components to the system.

GPS might be useful as a long-range tracker for AR systems, though accuracy is low.

20

5 Future directions5 Future directions

Hybrid approaches: Using multiple techniques to cover more weaknesses; Combining for example vision-based techniques with prediction.

Real-time systems and time-critical computing: AR systems need to be build to synchronize the real world which runs in real time

with the virtual part; Problem occurs mostly for video-based systems.

Perceptual and psychophysical studies: How much delay, registration error, etc can a user detect? How do we deal with possible optical illusions?

Portability: How to make the systems portable, even for outdoors. There are video eyewear for mobile phones and pda’s, gaming consoles, etc. (

Vuzix iWear). Multimodal displays:

Adding information for more of senses like sound and touch. Social and political issues:

Implications relating to the use of certain technologies in AR like lasers; The way people perceive AR, as a threat or as a tool to Augment Intelligence.

21

6 Conclusions6 Conclusions

We need both optical and video based systems because they both have advantages and disadvantages.

Research should go on to better optical technologies to allow more blending capabilities. These are more useful for civilian, business and military applications.

Video based systems are also useful because they can be the only solution for some difficult medical tasks with teleoperated robots.

The ultimate goal would be to generate virtual objects that are indistinguishable from real objects. (Photorealism)

Head-up-displays (HUD) and Helmet-mounted-displays (HMD) are currently in use in military applications and decrease significantly the amount of collateral damage.

There are applications for the iPhone for example that use annotations when the user uses the camera. (tourism for example)

AR has also been used in cars to enhance visibility at night with IR. Several setups have been used in museums to enhance the expositions.