augmented reality: connecting physical and digital worlds
DESCRIPTION
I made this presentation at the MPEG Multimedia Ecosystem 2013 in Incheon. It includes a summary of MPEG technologies related to Augmented Reality and is focuses on the separation between the AR creation and AR consuming. A system architecture for AR is also presented. Photo credits: Lisa Blum, Richard Wetzel, Veronica Scurtu Note: many pictures used in this presentation are downloaded from the Internet; I'll be happy to add credits to the original authors if they let me knowTRANSCRIPT
![Page 1: Augmented Reality: Connecting physical and digital worlds](https://reader030.vdocument.in/reader030/viewer/2022012913/5557a5b8d8b42a696c8b4753/html5/thumbnails/1.jpg)
CONNECTING PHYSICAL AND DIGITAL WORLDSMARIUS PREDA, MPEG 3DG CHAIR, INSTITUT MINES TELECOM
www.slideshare.net/MariusPreda
![Page 2: Augmented Reality: Connecting physical and digital worlds](https://reader030.vdocument.in/reader030/viewer/2022012913/5557a5b8d8b42a696c8b4753/html5/thumbnails/2.jpg)
From centuries, we are building two different types of worlds
PhysicalInformational
Immersion, from a human user point of view, is making abstraction of the Physical world (remove all possible connection to it) and have stimuli only from the Informational one
Immersion, presence, stimuli, perception, …
stories
music
films
knowledge
![Page 3: Augmented Reality: Connecting physical and digital worlds](https://reader030.vdocument.in/reader030/viewer/2022012913/5557a5b8d8b42a696c8b4753/html5/thumbnails/3.jpg)
In which world our users are?
The stimuli from physical worlds are familiar or at very low intensities
![Page 4: Augmented Reality: Connecting physical and digital worlds](https://reader030.vdocument.in/reader030/viewer/2022012913/5557a5b8d8b42a696c8b4753/html5/thumbnails/4.jpg)
We intentionally block our interfaces with the physical world
Improving immersion feeling
![Page 5: Augmented Reality: Connecting physical and digital worlds](https://reader030.vdocument.in/reader030/viewer/2022012913/5557a5b8d8b42a696c8b4753/html5/thumbnails/5.jpg)
Strategies for blocking the interfaces with the Physical world
Improving immersion feeling
![Page 6: Augmented Reality: Connecting physical and digital worlds](https://reader030.vdocument.in/reader030/viewer/2022012913/5557a5b8d8b42a696c8b4753/html5/thumbnails/6.jpg)
Strategies for extending the space of the Informational world stimuli
Improving immersion feeling
![Page 7: Augmented Reality: Connecting physical and digital worlds](https://reader030.vdocument.in/reader030/viewer/2022012913/5557a5b8d8b42a696c8b4753/html5/thumbnails/7.jpg)
How far we’ve gone with this strategy?
Relatively far …
Improving immersion feeling
![Page 8: Augmented Reality: Connecting physical and digital worlds](https://reader030.vdocument.in/reader030/viewer/2022012913/5557a5b8d8b42a696c8b4753/html5/thumbnails/8.jpg)
Probably too far …
Improving immersion feelingHow far?
![Page 9: Augmented Reality: Connecting physical and digital worlds](https://reader030.vdocument.in/reader030/viewer/2022012913/5557a5b8d8b42a696c8b4753/html5/thumbnails/9.jpg)
It’s the time to connect our worlds
![Page 10: Augmented Reality: Connecting physical and digital worlds](https://reader030.vdocument.in/reader030/viewer/2022012913/5557a5b8d8b42a696c8b4753/html5/thumbnails/10.jpg)
Why now?
• For the first time, smartphones enable “everyone” to access Informational world while sensing also the Physical world
• They change our view of the environment• Enhance the rich history
and meaningfulness ofthe real world withpowerful digitalelements
![Page 11: Augmented Reality: Connecting physical and digital worlds](https://reader030.vdocument.in/reader030/viewer/2022012913/5557a5b8d8b42a696c8b4753/html5/thumbnails/11.jpg)
Physical & Informational Augment The Reality
An invitation: become architects of a new kind of bridges
![Page 12: Augmented Reality: Connecting physical and digital worlds](https://reader030.vdocument.in/reader030/viewer/2022012913/5557a5b8d8b42a696c8b4753/html5/thumbnails/12.jpg)
In AR, almost everything remains to be invented
Las Vegas
This place some years ago
![Page 13: Augmented Reality: Connecting physical and digital worlds](https://reader030.vdocument.in/reader030/viewer/2022012913/5557a5b8d8b42a696c8b4753/html5/thumbnails/13.jpg)
In AR, almost everything remains to be invented
![Page 14: Augmented Reality: Connecting physical and digital worlds](https://reader030.vdocument.in/reader030/viewer/2022012913/5557a5b8d8b42a696c8b4753/html5/thumbnails/14.jpg)
• Marketing• Tourism• Culture• Education• Serious and less serious games• …
• Hardware manufacturers• Software vendors
In AR, almost everything remains to be invented
![Page 15: Augmented Reality: Connecting physical and digital worlds](https://reader030.vdocument.in/reader030/viewer/2022012913/5557a5b8d8b42a696c8b4753/html5/thumbnails/15.jpg)
We (and few others) already started and invite you to be part of
The AR Revolution
In AR, almost everything remains to be invented
![Page 16: Augmented Reality: Connecting physical and digital worlds](https://reader030.vdocument.in/reader030/viewer/2022012913/5557a5b8d8b42a696c8b4753/html5/thumbnails/16.jpg)
Tidy City
![Page 17: Augmented Reality: Connecting physical and digital worlds](https://reader030.vdocument.in/reader030/viewer/2022012913/5557a5b8d8b42a696c8b4753/html5/thumbnails/17.jpg)
Portal Hunt
![Page 18: Augmented Reality: Connecting physical and digital worlds](https://reader030.vdocument.in/reader030/viewer/2022012913/5557a5b8d8b42a696c8b4753/html5/thumbnails/18.jpg)
Elements
![Page 19: Augmented Reality: Connecting physical and digital worlds](https://reader030.vdocument.in/reader030/viewer/2022012913/5557a5b8d8b42a696c8b4753/html5/thumbnails/19.jpg)
ARQuiz
![Page 20: Augmented Reality: Connecting physical and digital worlds](https://reader030.vdocument.in/reader030/viewer/2022012913/5557a5b8d8b42a696c8b4753/html5/thumbnails/20.jpg)
Augmented Books
![Page 21: Augmented Reality: Connecting physical and digital worlds](https://reader030.vdocument.in/reader030/viewer/2022012913/5557a5b8d8b42a696c8b4753/html5/thumbnails/21.jpg)
Summer School games
![Page 22: Augmented Reality: Connecting physical and digital worlds](https://reader030.vdocument.in/reader030/viewer/2022012913/5557a5b8d8b42a696c8b4753/html5/thumbnails/22.jpg)
TECHNOLOGY INSIGHTSMPEG-V AND MPEG ARAF
![Page 23: Augmented Reality: Connecting physical and digital worlds](https://reader030.vdocument.in/reader030/viewer/2022012913/5557a5b8d8b42a696c8b4753/html5/thumbnails/23.jpg)
MPEG technologies related to AR
MPEG-1/2(AV content)
1992/4
VRML
1997
• Part 11 - BIFS: -Binarisation of VRML -Extensions for streaming -Extensions for server command -Extensions for 2D graphics - Real time augmentation with audio & video• Part 2 - Visual: - 3D Mesh compression - Face animation
1998
• Part 2 – Visual - Body animation
1999
MPEG-4 v.1
MPEG-4 v.2
First form of broadcast signal augmentation
![Page 24: Augmented Reality: Connecting physical and digital worlds](https://reader030.vdocument.in/reader030/viewer/2022012913/5557a5b8d8b42a696c8b4753/html5/thumbnails/24.jpg)
MPEG technologies related to AR
MPEG-4
2003
•AFX 2nd Edition: - Animation by morphing - Multi-texturing
2005
• AFX 3rd Edition - WSS for terrain and cities - Frame based animation
2007
MPEG-4
MPEG-4
• Part 16 - AFX: - A rich set of 3D graphics tools - Compression of geometry, appearance, animation
• AFX 4th Edition - Scalable complexity mesh coding
2011
MPEG-4A rich set of 3D Graphics representation and compression tools
![Page 25: Augmented Reality: Connecting physical and digital worlds](https://reader030.vdocument.in/reader030/viewer/2022012913/5557a5b8d8b42a696c8b4753/html5/thumbnails/25.jpg)
MPEG technologies related to AR
MPEG-4
2003
•AFX 2nd Edition: - Animation by morphing - Multi-texturing
2005
• AFX 3rd Edition - WSS for terrain and cities - Frame based animation
2007
MPEG-4
MPEG-4
• Part 16 - AFX: - A rich set of 3D graphics tools - Compression of geometry, appearance, animation
• AFX 4th Edition - Scalable complexity mesh coding
2011
MPEG-4
2009
• Part 25 - Compression of third-party XML (X3D, COLLADA)
MPEG-4
2004
• Part 16 - X3D Interactive Profile
MPEG-4
![Page 26: Augmented Reality: Connecting physical and digital worlds](https://reader030.vdocument.in/reader030/viewer/2022012913/5557a5b8d8b42a696c8b4753/html5/thumbnails/26.jpg)
MPEG technologies related to AR
MPEG-V - Media Context and Control
2011
• 2nd Edition: - GPS - Biosensors - 3D Camera
201x
• Compression of video + depth
201x
MPEG-V
- 3D Video
• 1st Edition - Sensors and actuators - Interoperability between Virtual Worlds
• Feature-point based descriptors for image recognition
201x
CDVS
MPEG-U – Advanced User Interface
2012
A rich set of sensors and actuators
- 3D Audio
MPEG-H
![Page 27: Augmented Reality: Connecting physical and digital worlds](https://reader030.vdocument.in/reader030/viewer/2022012913/5557a5b8d8b42a696c8b4753/html5/thumbnails/27.jpg)
MPEG technologies related to ARMPEG-V – Media Context and Control
![Page 28: Augmented Reality: Connecting physical and digital worlds](https://reader030.vdocument.in/reader030/viewer/2022012913/5557a5b8d8b42a696c8b4753/html5/thumbnails/28.jpg)
ActuatorsLight Flash Heating Cooling Wind Vibration Sprayer Scent Fog Color correction Initialize color correction parameter Rigid body motion Tactile Kinesthetic
Global position command
SensorsLight Ambient noise Temperature Humidity Distance Atmospheric pressure Position Velocity Acceleration Orientation Angular velocity Angular acceleration Force Torque Pressure Motion Intelligent camera typeMulti Interaction point Gaze tracking Wind
Dust Body height Body weight Body temperature Body fat Blood type Blood pressure Blood sugar Blood oxygen Heart rate
Electrograph EEG , ECG, EMG, EOG , GSR Weather Facial expression Facial morphology Facial expression characteristics Geomagnetic
Global position Altitude Bend Gas
MPEG technologies related to ARMPEG-V – Media Context and Control
![Page 29: Augmented Reality: Connecting physical and digital worlds](https://reader030.vdocument.in/reader030/viewer/2022012913/5557a5b8d8b42a696c8b4753/html5/thumbnails/29.jpg)
•All AR-related data is available from MPEG standards•Real time composition of synthetic and natural objects•Access to
• Remotely/locally stored BIFS/compressed 2D/3D mesh objects • Streamed real-time BIFS/compressed 2D/3D mesh objects
•Inherent object scalability (e.g. for streaming)•User interaction & server generated scene changes•Physical context
• Captured by a broad range of standard sensors• Affected by a broad range of standard actuators
Main features of MPEG AR technologies
![Page 30: Augmented Reality: Connecting physical and digital worlds](https://reader030.vdocument.in/reader030/viewer/2022012913/5557a5b8d8b42a696c8b4753/html5/thumbnails/30.jpg)
MPEG vision on AR: the MPEG AR Browser
Point to a URL – no need to download new applications for each contextThe browser• Retrieves scenario from the internet• Starts video acquisition• Tracks objects• Recognizes objects from visual signatures• Recovers camera pose• Gets streamed 3D graphics• Composes new scenes• Gets inputs from various sensors• Offers optimal AR experience by constantly adapting interaction possibilities and
objects from a remote server.
Industry • Maximize number of customers through MPEG-compliant authoring tools and
browsers• No need to develop a new application for each use case and device platform
![Page 31: Augmented Reality: Connecting physical and digital worlds](https://reader030.vdocument.in/reader030/viewer/2022012913/5557a5b8d8b42a696c8b4753/html5/thumbnails/31.jpg)
MPEG vision on AR
MPEG-4 + MPEG-7 + MPEG-21 ++ MPEG-U+ MPEG-V
= ARAFMPEG Player
CompressionAuthoring Tool
Produce
Download
![Page 32: Augmented Reality: Connecting physical and digital worlds](https://reader030.vdocument.in/reader030/viewer/2022012913/5557a5b8d8b42a696c8b4753/html5/thumbnails/32.jpg)
Technology insights
Modeling the content chain– Creation– Consumption– The pivot : a representation formalism
![Page 33: Augmented Reality: Connecting physical and digital worlds](https://reader030.vdocument.in/reader030/viewer/2022012913/5557a5b8d8b42a696c8b4753/html5/thumbnails/33.jpg)
Technology insights: the chain
ARAF Browser
MediaServers
ServiceServers
User
LocalSensors & Actuators
RemoteSensors & Actuators
MPEG ARAF
Local Real World
Environment
Local Real World
EnvironmentRemote
Real World Environment
Remote Real World
Environment
Auth. Tools
![Page 34: Augmented Reality: Connecting physical and digital worlds](https://reader030.vdocument.in/reader030/viewer/2022012913/5557a5b8d8b42a696c8b4753/html5/thumbnails/34.jpg)
Technology insights: Infrastructure
ARAF Browser
MediaServers
ServiceServers
User
LocalSensors & Actuators
RemoteSensors & Actuators
MPEG ARAF
Local Real World
Environment
Local Real World
EnvironmentRemote
Real World Environment
Remote Real World
Environment
Auth. Tools
![Page 35: Augmented Reality: Connecting physical and digital worlds](https://reader030.vdocument.in/reader030/viewer/2022012913/5557a5b8d8b42a696c8b4753/html5/thumbnails/35.jpg)
The ARAF browserARAF Browser
• Full scene graph representation (2D and 3D)• Static and dynamic objects and scenes• Programmable behavior (JS)• Access to sensors and cameras• Access to remote resources and services (http
request and web sockets)• Support for user interaction• Support for compressed resources
![Page 36: Augmented Reality: Connecting physical and digital worlds](https://reader030.vdocument.in/reader030/viewer/2022012913/5557a5b8d8b42a696c8b4753/html5/thumbnails/36.jpg)
The ARAF browserARAF Browser
ARAF Browser
JavaScript Engine
MPEG Totem Services
WebSockets Server
Communication service
WebSockets Client
ARAF Browser
SC3DMC Decoder
BBA Decoder
ARAF File
BIFS Stream
SC3DMC Stream
BBA Stream
BIFS DecoderMPEG-4 Scene
IndexedFaceSet
SBVCAnimation
Camera
Image data
Reference Images
Image data
Image data
Image data
AFAR Browser
Reference Signal
Detection
Detected Images
![Page 37: Augmented Reality: Connecting physical and digital worlds](https://reader030.vdocument.in/reader030/viewer/2022012913/5557a5b8d8b42a696c8b4753/html5/thumbnails/37.jpg)
MyMultimediaWorld.com
• Convert media assets in a common format• Describe, index, organize, search digital assets• Compressed assets, ready for transmission
A multimedia repository and processing platform supporting both creation and consumption Media
& ServiceServers
![Page 38: Augmented Reality: Connecting physical and digital worlds](https://reader030.vdocument.in/reader030/viewer/2022012913/5557a5b8d8b42a696c8b4753/html5/thumbnails/38.jpg)
Technology insights: the formalism
ARAF Player
MediaServers
ServiceServers
User
LocalSensors & Actuators
RemoteSensors & Actuators
MPEG ARAF
Local Real World
Environment
Local Real World
EnvironmentRemote
Real World Environment
Remote Real World
Environment
Auth. Tools
![Page 39: Augmented Reality: Connecting physical and digital worlds](https://reader030.vdocument.in/reader030/viewer/2022012913/5557a5b8d8b42a696c8b4753/html5/thumbnails/39.jpg)
ARAF Description Language
ARAF : an XML representation
Player
CompressionAuthoring Tool
Produce
Download
ARAF
![Page 40: Augmented Reality: Connecting physical and digital worlds](https://reader030.vdocument.in/reader030/viewer/2022012913/5557a5b8d8b42a696c8b4753/html5/thumbnails/40.jpg)
A set of scene graph nodes defined in MPEG-4• Audio, image, video, graphics, programming, communication, user
interactivity, animation
• Map, MapMarker, Overlay, ReferenceSignal, ReferenceSignalLocation, CameraCalibration, AugmentedRegion
Connection to sensors defined in MPEG-V• Orientation, Position, Angular Velocity, Acceleration, GPS, Geomagnetic,
Altitude, Local camera(s)
Compressed media• Image, sound, video, 2D/3D graphics
Augmented Reality Application Format
ARAF
![Page 41: Augmented Reality: Connecting physical and digital worlds](https://reader030.vdocument.in/reader030/viewer/2022012913/5557a5b8d8b42a696c8b4753/html5/thumbnails/41.jpg)
Technology insights: creation
ARAF Browser
MediaServers
ServiceServers
User
LocalSensors & Actuators
RemoteSensors & Actuators
MPEG ARAF
Local Real World
Environment
Local Real World
EnvironmentRemote
Real World Environment
Remote Real World
Environment
Auth. Tools
![Page 42: Augmented Reality: Connecting physical and digital worlds](https://reader030.vdocument.in/reader030/viewer/2022012913/5557a5b8d8b42a696c8b4753/html5/thumbnails/42.jpg)
Authoring ARAF
• Specific tools• Generic tools
– TOTEM.Designer– TOTEM.Scout
Auth. Tools
![Page 43: Augmented Reality: Connecting physical and digital worlds](https://reader030.vdocument.in/reader030/viewer/2022012913/5557a5b8d8b42a696c8b4753/html5/thumbnails/43.jpg)
Specific Authoring Tools
• Scenario driven approach– Tidy City, Portal Hunt, ARQuiz, AR Book, Treasure
Hunt
Auth. Tools
![Page 44: Augmented Reality: Connecting physical and digital worlds](https://reader030.vdocument.in/reader030/viewer/2022012913/5557a5b8d8b42a696c8b4753/html5/thumbnails/44.jpg)
Towards a generic Designer
• Create templates for app. objects• Fill structure with data• Create GUI• Create application logic • Export data
![Page 45: Augmented Reality: Connecting physical and digital worlds](https://reader030.vdocument.in/reader030/viewer/2022012913/5557a5b8d8b42a696c8b4753/html5/thumbnails/45.jpg)
On site designer
![Page 46: Augmented Reality: Connecting physical and digital worlds](https://reader030.vdocument.in/reader030/viewer/2022012913/5557a5b8d8b42a696c8b4753/html5/thumbnails/46.jpg)
Conclusions
• AR – the next digital revolution• For making revolutions, one needs weapons:
– MPEG technologies represent an efficient and powerful set of weapons
• Current MPEG activities– Improve and enrich the weapons– Package them in consistent and easy to use tools