7/31/2019 T-111 5700 Lecture Slides - Part 2

1/44

Mobile AR History

7/31/2019 T-111 5700 Lecture Slides - Part 2

2/44

A Brief History of AR (1) 1960s: Sutherland / Sproulls

first HMD system was see-through

7/31/2019 T-111 5700 Lecture Slides - Part 2

3/44

A Brief History of AR (2)

1960 - 70s: US Air Force helmet mounted displays(T. Furness)

7/31/2019 T-111 5700 Lecture Slides - Part 2

4/44

A Brief History of AR (3)

1970 - 80s: US Air Force Super Cockpit (T. Furness)

7/31/2019 T-111 5700 Lecture Slides - Part 2

5/44

A Brief History of AR (4)

Early 1990s: Boeing coined the term AR. Wire harness assemblyapplication begun (T. Caudell, D. Mizell).

Early to mid 1990s: UNC ultrasound visualization project

7/31/2019 T-111 5700 Lecture Slides - Part 2

6/44

A Brief History of AR (5)

1994: Motion stabilized display [Azuma] 1995: Fiducial tracking in video see-through [Bajura / Neumann]

1996: UNC hybrid magnetic-vision tracker

7/31/2019 T-111 5700 Lecture Slides - Part 2

7/44

A Brief History of AR (6)

1996: MIT Wearable Computing efforts 1998: Dedicated conferences begin Late 90s: Collaboration, outdoor, interaction Late 90s: Augmented sports broadcasts 1998 - 2001: Mixed Reality Systems Lab

7/31/2019 T-111 5700 Lecture Slides - Part 2

8/44

Evolution of Mobile AR

Wearable AR

HandheldAR Displays

Camera phone

1995 1997 2001 2003 2004

Camera phone- Self contained AR

WearableComputers

PDAs-Thin client AR

PDAs-Self contained AR

Camera phone- Thin client AR

7/31/2019 T-111 5700 Lecture Slides - Part 2

9/44

Handheld Displays

Tethered Applications Fitzmaurice Chameleon (1994)

Rekimotos Transvision (1995)

Tethered LCD

PC Processing and Tracking

7/31/2019 T-111 5700 Lecture Slides - Part 2

10/44

Handheld AR Display - Tethered

1995, 1996 Handheld AR ARPad, Cameleon Rekimotos NaviCam, Transvision

Tethered LCD

PC Processing and Tracking

7/31/2019 T-111 5700 Lecture Slides - Part 2

11/44

NaviCam (Rekimoto, 1995)

Information is registered to

real-world context Hand held AR displays

Interaction

Manipulation of a windowinto information space

Applications

Context-aware information displays

7/31/2019 T-111 5700 Lecture Slides - Part 2

12/44

NaviCam Architecture

Jun Rekimoto and Katashi Nagao, "The World through the Computer:Computer Augmented Interaction with Real World Environments", UserInterface Software and Technology (UIST '95)

7/31/2019 T-111 5700 Lecture Slides - Part 2

13/44

AR Pad (Mogilev 2002)

Handheld AR Display

LCD screen

Camera

SpaceOrb 3 DOF controller

Peripheral awareness

Viewpoint awareness

7/31/2019 T-111 5700 Lecture Slides - Part 2

14/44

Mobile AR: Touring Machine (1997)

University of Columbia

Feiner, MacIntyre, Hllerer, Webster Combines

See through head mounted display

GPS tracking

Orientation sensor

Backpack PC (custom)

Tablet input

7/31/2019 T-111 5700 Lecture Slides - Part 2

15/44

MARS View

Virtual tags overlaid on the real world

Information in place

7/31/2019 T-111 5700 Lecture Slides - Part 2

16/44

Backpack/Wearable AR

1997 Backpack AR Feiners Touring Machine AR Quake (Thomas)

Tinmith (Piekarski)

MCAR (Reitmayr) Bulky, HMD based

7/31/2019 T-111 5700 Lecture Slides - Part 2

17/44

PCI 3D Graphics Board

Hard Drive

Serial

Ports

CPU

PC104 Sound Card

PC104 PCMCIA

GPSAntenna

RTK correction AntennaRTK correction AntennaRTK correction Antenna

HMDController

HMDHMDControllerController

TrackerController

DC to DCConverter

Battery

WearableComputer

GPS RTKcorrection

Radio

Example self-built workingsolution with PCI-based 3D graphics

Columbia Touring Machine

Mobile AR - Hardware

7/31/2019 T-111 5700 Lecture Slides - Part 2

18/44

1997 Philip Kahn invents camera phone

1999 First commercial camera phone

Sharp J-SH04

7/31/2019 T-111 5700 Lecture Slides - Part 2

19/44

Millions of Camera Phones

0

200

400

600

800

1000

1200

2002 2003 2004 2005 2006 2007 2008 2009 2010

DSC

Phone

7/31/2019 T-111 5700 Lecture Slides - Part 2

20/44

Handheld AR Thin Client

2001 BatPortal (AT&T Cambridge)

PDA used as I/O device Wireless connection to workstation

Room-scale ultrasonic tracking (Bat)

2001 AR-PDA (C Lab) PDA thin graphics client

Remote image processing

www.ar-pda.com

7/31/2019 T-111 5700 Lecture Slides - Part 2

21/44

2003 ARphone (Univ. of Sydney) Transfer images via Bluetooth (slow 30 sec/image)

Remote processing AR Server

Mobile Phone AR Thin Client

7/31/2019 T-111 5700 Lecture Slides - Part 2

22/44

Early Phone Computer Vision Apps2003 Mozzies Game - Best mobile game

Optical motion flow detecting phone orientationSiemens SX1 Symbian, 120Mhz, VGA Camera

2005 Marble Revolution (Bit-Side GmbH)

Winner of Nokia's Series 60 Challenge 2005

2005 SymBall (VTT)

7/31/2019 T-111 5700 Lecture Slides - Part 2

23/44

Computer Vision on Mobile Phone Cameras and Phone CPU sufficient for computer

vision applications

Pattern Recognition (Static Processing)

QR Code

Shotcode (http://www.shotcode.com/)

Motion Flow (2D Image Processing)

GestureTek- http://www.gesturetekmobile.com/

TinyMotion

3D Pose Calculation

Augmented Reality

7/31/2019 T-111 5700 Lecture Slides - Part 2

24/44

Handheld AR Self Contained2003 PDA-based AR

ARToolKit port to PDA Studierstube ported to PDA

AR Kanji Educational App.

Mr Virtuoso AR character

Wagners Invisible Train- Collaborative AR

7/31/2019 T-111 5700 Lecture Slides - Part 2

25/44

Mobile Phone AR Self Contained2004 Mobile Phone AR

Moehring, Bimber

Henrysson (ARToolKit)

Camera, processor, display together

7/31/2019 T-111 5700 Lecture Slides - Part 2

26/44

AR Advertising

Txt message to download AR application (200K)

See virtual content popping out of real paper advert

Tested May 2007 by Saatchi and Saatchi

7/31/2019 T-111 5700 Lecture Slides - Part 2

27/44

Location Aware Phones

Nokia NavigatorMotorola Droid

7/31/2019 T-111 5700 Lecture Slides - Part 2

28/44

Real World Information Overlay Tag real world locations

GPS + Compass input

Overlay graphics data on live video

Applications Travel guide, Advertising, etc

Eg: Mobilizy Wikitude (www.mobilizy.com) Android based, Public API released

Other companies Layar, AcrossAir, Tochnidot, RobotVision, etc

7/31/2019 T-111 5700 Lecture Slides - Part 2

29/44

Layar www.layar.com

7/31/2019 T-111 5700 Lecture Slides - Part 2

30/44

Wikitude www.mobilizy.com Overlays Points of Interest on real world

GPS, compass data

Uses data feeds Flickr

Wikipedia Google

Web authoring

7/31/2019 T-111 5700 Lecture Slides - Part 2

31/44

1995 Handheld Display: NaviCam, AR-PAD, Transvision

1997 Wearable AR: Touring Machine, AR Quake

2001 Handheld AR Thin Client: AR-PDA, Bat Portal

2003 Handheld AR Self contained: Invisible Train

2003 Mobile Phone 2D Vision: Mozzies, Symball

2003 Mobile Phone Thin Client: ARphone

2004 Mobile Phone Self contained: Moehring, Symbian

History of Handheld and MobileAR

7/31/2019 T-111 5700 Lecture Slides - Part 2

32/44

Mobile AR by Weight

Backpack+HMD:5-8kg

Scale it down:

VespR[Kruijff ISMAR07]:Sony UMPC 1.1GHz1.5kgstill >$5K

Scale it down more:

Smartphone$500All-in-one0.1kgbillions of units

1996

20032007

7/31/2019 T-111 5700 Lecture Slides - Part 2

33/44

2010 State of the ArtHandheld Hardware available

PDA, mobile phones, external cameras

Sensors: GPS, accelerometer, compassSoftware Tools are Available

Tracking: ARToolKitPlus, stbTracker

Graphics: OpenGL ES

Authoring: Studierstube, Layar, Wikitude, Unifye

What is needed:

High level authoring tools

Content development tools

Novel interaction techniques

User evaluation and usability

7/31/2019 T-111 5700 Lecture Slides - Part 2

34/44

2007 - AR Reaches Mainstream

MIT Technology Review

March 2007 list of the 10 most exciting

technologies

Economist

Dec 6th 2007

Reality, only better

7/31/2019 T-111 5700 Lecture Slides - Part 2

35/44

Google Searches for AR

7/31/2019 T-111 5700 Lecture Slides - Part 2

36/44

Gartners top 10 disruptive technologies 2008-2012: Multicore and hybrid processors

Virtualisation and fabric computing

Social networks and social software Cloud computing and cloud/Web platforms

Web mashups

User Interface

Ubiquitous computing Contextual computing

Augmented reality

Semantics

7/31/2019 T-111 5700 Lecture Slides - Part 2

37/44

Gartner Hype Cycle

7/31/2019 T-111 5700 Lecture Slides - Part 2

38/44

Mobile AR Companies Mobile AR

GPS + compass

Many Companies

Layar

Wikitude

Acrossair PressLite

Yelp

Robot vision

Etc..

7/31/2019 T-111 5700 Lecture Slides - Part 2

39/44

$2 million USD in 2010$732 million USD in 2014

7/31/2019 T-111 5700 Lecture Slides - Part 2

40/44

Handset Manufacturers Qualcomm

$100 million USD investment

Nokia 25+ people in NRC

Samsung Exploring the space

Apple 586 AR Applications on App Store

Google Google goggles/Android AR Applications

7/31/2019 T-111 5700 Lecture Slides - Part 2

41/44

Qualcomm

Acquired Imagination

October 2010 - Releases free Android AR SDK

Computer vision tracking - marker, markerless Integrated with Unity 3D renderer

http://developer.qualcomm.com/ar

7/31/2019 T-111 5700 Lecture Slides - Part 2

42/44

Rock-em Sock-em

Shared AR Demo

Markerless tracking

7/31/2019 T-111 5700 Lecture Slides - Part 2

43/44

Apple

iPhone 4 SDK supports direct camera access Launches AR theme on App Store

7/31/2019 T-111 5700 Lecture Slides - Part 2

44/44

43 AR apps on App Store