fondamentals of virtual and augmented reality click to edit … · 2019-10-02 · 3 “virtual and...
TRANSCRIPT
Click to edit Master title style
“Virtual and Augmented Reality”
Interaction speciality – Computer Science Master - University Paris-Saclay
Fondamentals of Virtual and Augmented Reality
AR: an introduction
Jeanne Vezien
2
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
Plan of the lecture
1. Augmented reality:
what and why
2. User tracking
3. Real world reconstruction
4. 3D Graphics
5. Augmentation
3
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
Wikipedia says…
Augmented reality (AR) is a term for a live
direct or indirect view of a physical, real-world
environment whose elements are augmented by
computer-generated sound, video, graphics,
haptic or GPS data.
Augmentation is conventionally in real-time and
in semantic context with environmental
elements, such as sports scores on TV during a
match.
With the help of advanced AR technology
(computer vision and object recognition) the
information about the surrounding world
becomes interactive, e.g. artificial information
about the environment can be overlaid.
Augmented reality was coined by Thomas
Caudell, working at Boeing, in 1990. ARToolkit (Kato & Billinghurst, 2001)
4
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
AR
InteractiveReal-time
Contextualized
5
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
The dinosaur
is rendered:
- Interactively
- In real-time
- In context
AR example
6
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
2018: the competition began
• Hololens (Microsoft, 2016)
• Commercial in 2017
• 50 000 units sold.
• V2 in 2019
• Magic Leap (Magic Leap Inc, 2018)
• Released august 2018
• Maximum Hype
• No business (yet)
7
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
2019: the competition is on
Hololens 2
3500$
Real time hand tracking
(grab, pinch, push,
slide…)
Gaze tracking
Magic Leap 1
2.295 $
8
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
Real/virtual Continuum (Milgram 1994)
9
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
Real/virtual Continuum (Milgram 1994)
10
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
Real/virtual Continuum (Milgram 1994)
11
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
Real/virtual Continuum (Milgram 1994)
12
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
Real/virtual Continuum (Milgram 1994)
13
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
Real/virtual Continuum (Milgram 1994)
14
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
Real/virtual Continuum (Milgram 1994)
15
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
AR: examples
Not a new idea….
• User position and gaze provides context ,
sometimes with help (markers)
• Audio is nice (non-obstrusive), can be
stopped anytime.
• No computer involved
• No sensing involved : limited interaction
Now associated with head-tracking,
localization…
Started in 1957 (Roosevelt home)
Principio system
(2007)
16
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
AR: examples
See-through augmented vision: « classic AR »
Tourism: Archeoguide (2002)
17
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
AR: examples
Assemply / Maintenance / Repair
Maintenance: help technician with
contextualized content
BMW, 2010
Matris project, 2007
Fiducial Text
ActionGraphics
Icons
18
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
AR: examples
Way-findingIphone app: NY nearest subway (and many cities, airports have their own app)
https://www.youtube.com/watch?v=ps49T0iJwVg
Augmented Car Finder
Note:
• Data is collected off-line
• Access to database is native on mobile phones
(http protocol)
19
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
AR: examples
Safe landing
Hunter system: Landing guiding system by AR
tracking techniques (computer vision based)
www.novemberkiloecho.com
20
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
War "games" (1/2)
Track target w.r.t. weapon
Coupled with (off-line) Geographical Information
system
… and strategic realtime info (GPS, detectors)
gun-mounted display
21
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
War "games" (2/2)
Tanagram.com concept (closed)
22
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
Heads Up Displays
TopOwl helmet (helicopter)
In operation since 2008
Thalesgroup.com
Eye tracking
Symbology
Gun pointing
Night VisionHeavy and $$$ !
23
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
Heads Up Displays (2)
Skully web helmet (2015)
(+ rear cameras)
Intuitive Aerial (Sweden, 2013)
24
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
MR: examples
Games and promotional contents
EyePet for PSP (2009)
Topps 3D baseball cards (2009)
PSVita (2012)
Father.io (2016): Massive Multiplayer Laser Tag
25
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
Pokémon GO !
The Pokemon Company (Nintendo) + Niantic (Ingress)
Since July 2016
Game brought in more than $1.2 Billion !
2018: 104M$ / month !
Success = the right mix of game + social
Possible only with AR (or connected VR ?)
26
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
Other examples
Medical: Surgery planning, nurse and student training…
University of North Carolina
Augmented virtuality (presented on screen)
Track hand-held device or body
Coupled with (off-line) anatomic data
27
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
DaVinci Robot
Intuitive Surgical
3 000 ex.
1 M.$ per unit30 problems over 17000
interventions, 1 death (bad
manip.)
Less stress
More precision
Comfortable
Complete asepsis
No tremor
abdomen
surgery
28
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
• Militaires: « future soldier », BARS
• Médical : assistance chirurgicale
• Tourisme
• Customization (mode)
Virtual try-ons and Customize
Augmented virtuality or
AR
Track users anatomy
and motion (better still)
Coupled with external
data on-line
Augment.com (2017)Ikea (2014)
ForeverMark (2011)
29
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
MR: the SACARI example
Mixed virtuality: user not moving with camera,
indirect view of real world
Tele-immersion : provide sense of presence of remote environment
Internship possibilities in VENISE team !
N. Khenak et al. 2019
30
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
Blurred boundaries: augmented movies
Is this real ? Augmented ? Virtual ?
On-line ? Off-line ?
Avatar (2009)Zoe Zaldana
31
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
Blurred boundaries
Is this still AR ?
AR will be everywhere
when efficient, cheap see-
through displays will
become available
Google moto: The World's
Information in Context
Street View is close to
Augmented Virtuality
Google glass 2 (2019)
32
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
The Ingredients for AR
Capture real world
33
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
The Ingredients for AR
Capture real world
+
Capture virtual world
(Computer Graphics)
34
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
The Ingredients for AR
Capture real world
+
Capture virtual world
(Computer Graphics)
Present to user
(Augmentation)
35
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
Real 3D world capture
Two ingredients necessary for
successful AR:
Real-time 3D tracking of user
viewpoint w.r.t. world
3D scene analysis realistic
augmentation WorldViz tracking
Structure sensor
(Occipital, 2019) https://structure.io/
36
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
Real 3D world
Two ingredients necessary for
successful AR:
Real-time 3D tracking of
user viewpoint w.r.t. world
3D scene analysis
realistic augmentation
37
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
Real-Time tracking
Many means to compute positioning information:
Geolocalization
Electro-magnetic
Acoustic
Inertial
Vision-based (active or passive)
Covered in « stereo and tracking » course
38
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
Image-based 3D tracking
It is a special case of adaptive
rendering ( for HMDs)
Environment control ++
Markers are still needed
(often)
Special case of
structure/motion estimation
AR meets movie industry:
the user is the director !
markersKnown as motion estimation / match moving
39
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
From [3]
= user
From [3]
Inside-out vs. Outside-in
= user
Inside-out: user wears camera Outside-in: camera observes user
• Inside-out better at estimating relative rotation
• Outside-in much more convenient : don’t have to wear a camera
• Special case: console gaming
• Inside-out easier for wearable AR: mobility, ego-centered
reference frame
• Needed for nomadic apps
• Cameras are getting small and commodity items
40
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
Projection is necessary to map 3D points on 2D pixels:
P(X,Y,Z)
o
p(x,y)
f
z
x
Focal point
Focal plane
11
Z
Y
X
Z
Y
X
y
x
s
Z
Yy
Z
Xx
0100
0010
0001
3D tracking: equations
41
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
Objective: find camera parameters and position
110001
333231
232221
131211
Z
Y
X
TzRRR
TyRRR
TxRRR
y
x
s
0100
0010
0001
1100
0
0
1
y
x
v
u
v
u
cv
cu
Tracker
Camera
(R,T)
Perspective equation (extrinsic parameters):
Cam. Position,
(R,T)
Intrinsic parameters (scale, offset):
Pixels
M(X,Y,Z)
3D tracking: equations
42
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
M
Z
Y
X
P
TtTtTt
v
u
v
u
sz
y
x
cv
cu
110
k
j
i
0100
0010
0001
100
0
0
1
Projection equation:
image 3D worldm = P . M
motion structure
The Maths
43
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
M
Z
Y
X
P
TtTtTt
v
u
v
u
sz
y
x
cv
cu
110
k
j
i
0100
0010
0001
100
0
0
1
Projection equation:
image 3D worldm = P . M
motion structure
The Maths
World camera =
Camera position
44
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
M
Z
Y
X
P
TtTtTt
v
u
v
u
sz
y
x
cv
cu
110
k
j
i
0100
0010
0001
100
0
0
1
Projection equation:
image 3D worldm = P . M
motion structure
The Maths
3D 2D projection
45
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
M
Z
Y
X
P
TtTtTt
v
u
v
u
sz
y
x
cv
cu
110
k
j
i
0100
0010
0001
100
0
0
1
image 3D worldm = P . M
motion structure
The Maths
Sensor image (pixels)
Projection equation:
46
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
M
Z
Y
X
P
TtTtTt
v
u
v
u
sz
y
x
cv
cu
110
k
j
i
0100
0010
0001
100
0
0
1
Projection equation:
image 3D worldm = P . M
motion structure
The Maths
(i,j,k,Tx,Ty,Tz) can be computed if m and M are known.
Not linear in the motion parameters !!
47
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
Dementhon-Davis (1992)
Several methods to recover calibration exist (Tsai, Lowe…)
" Model-Based object pose in 25 lines of code" is a simple
yet elegant method to obtain pose rapidly if a rigid 3D model
is provided.
i
i
i
i
i
i
i
ii
xPI
PTz
kTz
TxP
Tz
i
TzPk
TxPi
Z
Xx
1
.
1.
.
.
. 0
i
ii
yPJy
1
. 0
),( 00 yx Tz
Pk ii
.
Projection of T(Tx,Ty,Tz) Relative depth around Z0=Tz
Idea consists in linearizing xi and yi to compute iJI ,,
ziz TTZ
48
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
i
o
i
i
o
i
i PJyy
PIxx
.
.
0
011
1
Geometrical interpretation:
P
O
Image plane
P0
Reference plane
Z0=Tz
Orthographic
projection
Z
Y
i Z0
Perspective
projection
Linear approximation of perspective
iio
y
iii
o
i
o
x
y
xxx
49
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
POSIT algorithm
Summarize:
Tz
jPJyyyy
Tz
iPIxxxx
i
o
iii
i
o
iii
=Jwith.)1(
=Iwith.)1(
00
00
If εi is known then I,J,Tz,X0=x0Tz,Y0=y0Tz can be computed
If I,J,Tz k is known
Iterate starting with
Converges rapidly:
Tz
Pk ii
.
0i
50
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
3D tracking: off-the-shelf solution
• Kato & Billinghurst (HIT lab, univ. of Washington) introduce
in 1999 a tool for teleconferencing:
"Marker Tracking and HMD Calibration for a video-
based Augmented Reality Conferencing System.“
Soon to become ARToolkit
51
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
ARToolkit: main characteristics
• Fast and cheap 6D marker tracking.
• Distributed with complete source code.
OpenSource with GPL license for
noncommercial usage.
• Multiplatform (Linux, MacOS and Windows) .
• Multiple input sources (USB, Firewire) ,
multiple format (RGB, YUV) supported.
• Multiple camera tracking supported.
• GUI initializing interface.
• Easy calibration routine.
• Fast rendering based on OpenGL.
• 3D VRML support.
• Simple and modular API (in C and C++).
• Complete set of samples and utilities.
• Supports both video and optical see-through
AR.
52
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
ARToolkit: main processing loop
Since v4, Pose estimation is performed using the
Iterative Closest Point (ICP) algorithm (Besl, 1992)
53
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
Important details
Virtual objects appear only when complete markers are visible:
Size
Movement
Orientation
Lighting conditions
Pattern Size (cm.) Usable Range (cm.)
6.98 40.64
8.89 63.5
10.79 86.36
In practice: close range only !
54
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
3D tracking: off-the-shelf solution
www.layar.com
web-based, per page service
www.wikitude.com: AR SDK
Vuforia (qualcomm) multiplatform SDK
Including Unity
Inglobe AR: AR solutions, ARPlayer+plugins (40.000 registered users)
2490€
$499
focus on printed text
55
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
Markerless 3D tracking and AR dev: battle of
the SDKs
Spatial anchors
Cloud anchors
1.2ARKit 2
ARWorldMap
56
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
Real 3D world
Two ingredients necessary for
successful AR:
Real-time 3d tracking of user
viewpoint w.r.t. world
3D scene analysis
realistic augmentation
57
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
2 components: Light and Geometry
Lighting coherency Motion coherency
Geometric coherency
Light probe
3D real world analysis
58
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
Aim: Capture light coming from light sources in real world
10,000:1 is the (static) human eye ratio between brightest and
darkest shade: cannot be represented on 8 bits
Stored in High Dynamic Range images (.hdr image format)
From all directions at a given point
Light Probe
u
v
59
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
Half-life 2: normal and HDR
Note: advanced graphics techniques can
use hdr images (Unreal engine)
u
v
1,1),( vu
u
v1tan 22 vu
A direction vector in the world (Dx, Dy, Dz)
u = r. Dx and v = r.Dy with
unit vector pointing in the direction (u,v) is obtained by rotating
(0,0,-1) by:
1) degrees around the y (up) axis
2) degrees around the -z (forward) axis.
22
1 )(cos1
DyDx
Dzr
Light Probe (2)
60
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
World content
Geometry: 3D reconstruction : what is
where ?
Semantic : augmentation of context
object recognition
interpretation of content
Comprendre l’environnement 3D en termes : http://www.truevisionsys.com
61
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
Markers cannot always be present !
If structure is known: a priori model tracking (see before):
an object is a (collection of) marker
Remember : projection combines structure and motion (m
= P.M)
Idea: recover both simultaneouly !
+ gives tracking and reconstruction at the same time
- highly non-linear : requires iterations + data filtering (remove
outliers )
Geometry: 3D reconstruction
Factorization method introduced by Tomasi and Kanade (1992)
62
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
mij = Pj. Mi
Factorization method
2x1 = 2x3 3x1
FPF
FPF
P
P
yy
xx
yy
xx
m
1
1
111
111
jth frame
ith point
P
P
P
ZZ
YY
XX
M
1
1
1
FP
P
P 1
• Compact representation:
• Consider P points projecting on F frames:
• There is an of possible (P,M) pairs!
• Solved via SVD decomposition (but not uniquely)
Carlo Tomasi and Takeo Kanade. (November 1992)
"Shape and motion from image streams under orthography: a factorization method."
International Journal of Computer Vision, 9 (2): 137–154.
Images !
m is big, but rank 3 ! m= P.M
63
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
Modern Approach: SfM
• Structure from Motion (or SLAM)
Inertial-Aided Sequential 3D Metric Surface
Reconstruction from Monocular Image Streams
Aufderheide et al. (2013).
• Initial pose estimation with factorization with a few keyframes
• Non-linear (costly) optimization (bundle adjustment)
• Real-time solutions are starting to emerge
64
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
S. Christy et R. Horaud : "Euclidean Shape and Motion from Multiple Perspective Views by Affine Iterations"
IEEE Transactions on Pattern Analysis and Machine Intelligence, Volume 18, Number 11, Pages 1098--1104 - November 1996
Factorization: results
Hanno Ackermann, University of Hanovre (2008)
65
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
Computer Vision is good at locating features:
points
regions
textures
contours
… at different scales !
Image-based 3D reconstruction
Harris (1988)
SURF detector (OpenCV), Bay (2006)
2D region detection (region-growing)
66
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
Problems :
Robustness of detection
Matching (space/time)
Is becoming robust enough
For now: limited use , constraints
(model-based)
Moving fast with new AR devices
Markerless tracking
Wang & Popovic, MIT, 2010
67
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
Boujou: http://vicon.com/boujou/
Commercial solutions
Pricey (3500€)
68
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
New real-time sensors: depth cameras (Zcam)
Infer depth for ALL pixels
Early : laser, stereoPerceptron
LIDAR , 1995
Devernay, 1994
Image-based 3D tracking
69
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
Now: structured lighting : Zcam for 100 € !
Kinect, 2010
Augmented Reality Magic
Mirror using the Kinect
Tobias Blum
Image-based 3D tracking
Solves the "where" but not the "what":
objects must still be identified (segmented)
70
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
Image-based 3D tracking
Hololens (2016, 3000$)
Augmented reality « see-through »
glasses
Markerless 3D scanner with :
depth camera (Kinect-like) Spatial
Mapping
4 environment cameras
Inertial Measurement UnitHead tracking
SLAM = Simultaneous Localization and Mapping
= markerless tracking !
}
71
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
Image-based 3D tracking
Magic Leap (2018, 2295$)
Augmented reality « see-through »
glasses
2 focus: near/far !
Markerless 3D scanner with :
depth camera (Kinect-like) Spatial
Mapping
4 environment cameras
Inertial Measurement Unit
Magnetic tracker
Head tracking
SLAM based on Deep Neural Network
}DeTone & Malisiewicz
Towards Geometriv Deep SLAM
https://arxiv.org/pdf/1707.07410.pdf
72
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
Stereo for AR: augmentation
Object location (table…)
Object occlusion
Shadows and light
interactions
In real-time !!
Will soon happen in
movie industry ($$$)
3D reconstruction is necessary for realistic CG blending:
Source: X3d consortium
73
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
3D analysis of
real images
(camera +
reconstruction)
Augmentation: CG blending
74
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
CG graphics+3D analysis of
real images
(camera +
reconstruction)
Augmentation: CG blending
75
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
CG graphics
Compositing masks
(shadows, occlusions…)
+3D analysis of
real images
(camera +
reconstruction)
Augmentation: CG blending
match move
76
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
CG graphics
Compositing masks
(shadows, occlusions…)
Augmented image
+3D analysis of
real images
(camera +
reconstruction)
Augmentation: CG blending
match move
77
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
Matching constraints:
• geometry (epipolar)
• photometry
H. Jin, P. Favaro, and S. Soatto.
A Semi-direct Approach to Structure From Motion. The Visual Computer, 19(6): 377-394, October 2003.
Example of augmentation pipeline (1)
image analysis
Region matching (1992)
78
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
Example of augmentation pipeline (2)
3D reconstruction
3D reconstruction of regions based on planar equations:
Hypothesis: world is piece-wise planar
79
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
Prior information:
explicit 3D model
motion constraints
3D registration by ICP = Iterative
Closest Point (Besl 92):
always converges
model points do not coincide with
reconstruction points
Example of augmentation pipeline (3)
3D reconstruction
80
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
Example of augmentation pipeline (4)
3D virtual content
Two steps:
Geometric modeling
(sometimes based on 3D
scans): Maya, Unity,
Blender, Sketchup, etc.
Photo-realistic rendering:
textures, lights, reflectance,
shadows, etc.
Note: Light probes
environment mapping
81
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
Light interaction: a must
Geometric coherency is good
but… light is essential !
Shadows in real-time =
geometry + light
Standard technique for CG renderers:
Ground (real)
(virtual)PSP
.'
dnEc
.nLd
.
E
S
82
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
Shadow projection techniques
Many different techniques !
Plane Projected
Shadows
Projected
Shadows
Depth Shadow
MappingVertex Projection
Shadow
Volumes
Quick, not much
calculations
Quick, almost no
calculations
Very quick, not
much calculations
Slow with high-res
meshes
Slow, lots of
calculations
High detail Detail depends
on texture
Detail depends
on textureHigh Detail High detail
No self-
shadowingNo self-
shadowingSelf-shadowing
No self-shadowing Self-shadowing
No shadow
receivers
Shadow
receivers
Shadow
receivers
No shadow
receivers
Shadow
receivers
83
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
Shadow mapping
M. Haller, S. Drab, and W. Hartmann, 2003.
"A real-time shadow approach for an augmented reality application using shadow
volumes," in VRST 03: Proceedings of the ACM symposium on Virtual reality software and
technology, New York, NY, USA, 2003, pp. 56-65
Shadow volume with stencil drawing
84
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
Example of augmentation pipeline (5)
animations
Two steps:
Virtual motions must be
coherent with real ones :
match-moving
Motion is camera-centric
Virtual camera must be
identical to real one (off-line
calibration is necessary)
Accommodate zoom !
85
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
Occlusion mask
Virtual object rendering
(alone)
Example of augmentation pipeline (6)
Step by step rendering
86
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
Real objects: reference white
(albedo)
"black" virtual objects:
Shadow computation
Example of augmentation pipeline (7)
Step by step rendering
87
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
Final composition =
real image *(1-mask) * attenuation
+
Virtual objects * mask
+ reflexions * (1-mask)
Example of augmentation pipeline (8)
Step by step rendering
88
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
Final compositing
89
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
Other example (1)
Objets déformables + occultations réel/virtuel
90
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
Other example (2)
Reference white
Shadows of
virtual objects on
the real scene
91
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
Other example (3)
Final result
92
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
Conclusion
• Augmented reality is a reality
• Is a complicated process, requires
a lot of expertise and hard/software
• In-depth coherency of geometry
and photometry
• Real-time challenges
Elaborate rendering (GPU)
Match moving (markerless)
3D reconstruction (occlusions)
User interaction: hand and
body tracking
93
“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay
THE END !
Denno Coil, 2007