3d imaging motion. extract visual information from image sequences assume illumination conditions...
TRANSCRIPT
![Page 1: 3D IMAGING Motion. Extract visual information from image sequences Assume illumination conditions do not change Differences in image Camera moves](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f395503460f94c55e51/html5/thumbnails/1.jpg)
3D IMAGING
Motion
![Page 2: 3D IMAGING Motion. Extract visual information from image sequences Assume illumination conditions do not change Differences in image Camera moves](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f395503460f94c55e51/html5/thumbnails/2.jpg)
Motion
Extract visual information from image sequences
Assume illumination conditions do not change
Differences in image Camera moves (Some) Objects in a scene move Both camera and objects move
![Page 3: 3D IMAGING Motion. Extract visual information from image sequences Assume illumination conditions do not change Differences in image Camera moves](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f395503460f94c55e51/html5/thumbnails/3.jpg)
Difference from stereo
Stereo from multiple views Images taken at the same time moment
Motion Images taken in small time intervals
Differences Camera position might not be known Changes are smaller Not necessary caused by a single 3D rigid transform
Rotation and Translation Multiple objects
![Page 4: 3D IMAGING Motion. Extract visual information from image sequences Assume illumination conditions do not change Differences in image Camera moves](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f395503460f94c55e51/html5/thumbnails/4.jpg)
Why analyze motion
Understand scene structure Objects, Distance, Velocities
Understand scene properties without a priori knowledge on the world
![Page 5: 3D IMAGING Motion. Extract visual information from image sequences Assume illumination conditions do not change Differences in image Camera moves](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f395503460f94c55e51/html5/thumbnails/5.jpg)
Random Dot
![Page 6: 3D IMAGING Motion. Extract visual information from image sequences Assume illumination conditions do not change Differences in image Camera moves](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f395503460f94c55e51/html5/thumbnails/6.jpg)
Time to Collision
f
L
v
L
Do
l(t)
An object of height L moves with constant velocity v:•At time t=0 the object is at:• D(0) = Do
•At time t it is at • D(t) = Do – vt
•It will crash with the camera at time:• D(t) = Do – vt = 0
• t = Do/v
t=0t
D(t)
![Page 7: 3D IMAGING Motion. Extract visual information from image sequences Assume illumination conditions do not change Differences in image Camera moves](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f395503460f94c55e51/html5/thumbnails/7.jpg)
Time to Collision
f
L
v
L
Do
l(t)
t=0t
D(t)
The image of the object has size l(t):
Taking derivative wrt time:
![Page 8: 3D IMAGING Motion. Extract visual information from image sequences Assume illumination conditions do not change Differences in image Camera moves](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f395503460f94c55e51/html5/thumbnails/8.jpg)
Time to Collision
f
L
v
L
Do
l(t)
t=0t
D(t)
And their ratio is:
![Page 9: 3D IMAGING Motion. Extract visual information from image sequences Assume illumination conditions do not change Differences in image Camera moves](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f395503460f94c55e51/html5/thumbnails/9.jpg)
Time to Collision
f
L
v
L
Do
l(t)
t=0t
D(t)
And time to collision:
Can be directly measured from image
Can be found, without knowing L or Do or v !!
![Page 10: 3D IMAGING Motion. Extract visual information from image sequences Assume illumination conditions do not change Differences in image Camera moves](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f395503460f94c55e51/html5/thumbnails/10.jpg)
Parallax
Parallax=apparent change in position when viewed from different lines of sights
Closer objects have larger parallax
![Page 11: 3D IMAGING Motion. Extract visual information from image sequences Assume illumination conditions do not change Differences in image Camera moves](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f395503460f94c55e51/html5/thumbnails/11.jpg)
Depth cues
![Page 12: 3D IMAGING Motion. Extract visual information from image sequences Assume illumination conditions do not change Differences in image Camera moves](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f395503460f94c55e51/html5/thumbnails/12.jpg)
Kinetic Depth Effect
![Page 13: 3D IMAGING Motion. Extract visual information from image sequences Assume illumination conditions do not change Differences in image Camera moves](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f395503460f94c55e51/html5/thumbnails/13.jpg)
Motion Field
3D velocity field Each point has a velocity vector
2D Motion field Projection of these 3D velocity vector onto
image plane
![Page 14: 3D IMAGING Motion. Extract visual information from image sequences Assume illumination conditions do not change Differences in image Camera moves](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f395503460f94c55e51/html5/thumbnails/14.jpg)
2D Motion Field
![Page 15: 3D IMAGING Motion. Extract visual information from image sequences Assume illumination conditions do not change Differences in image Camera moves](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f395503460f94c55e51/html5/thumbnails/15.jpg)
2D Optical Flow
Apparent motion of image brightness pattern
![Page 16: 3D IMAGING Motion. Extract visual information from image sequences Assume illumination conditions do not change Differences in image Camera moves](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f395503460f94c55e51/html5/thumbnails/16.jpg)
Assuming that illumination does not change:
Image changes are due to the RELATIVE MOTION between the scene and the camera.
There are 3 possibilities: Camera still, moving scene Moving camera, still scene Moving camera, moving scene
![Page 17: 3D IMAGING Motion. Extract visual information from image sequences Assume illumination conditions do not change Differences in image Camera moves](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f395503460f94c55e51/html5/thumbnails/17.jpg)
Motion Analysis Problems
Correspondence Problem Track corresponding elements across frames
Reconstruction Problem Given a number of corresponding elements, and
camera parameters, what can we say about the 3D motion and structure of the observed scene?
Segmentation Problem What are the regions of the image plane which
correspond to different moving objects?
![Page 18: 3D IMAGING Motion. Extract visual information from image sequences Assume illumination conditions do not change Differences in image Camera moves](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f395503460f94c55e51/html5/thumbnails/18.jpg)
Motion Field (MF)
The MF assigns a velocity vector to each pixel in the image.
These velocities are INDUCED by the RELATIVE MOTION btw the camera and the 3D scene
The MF can be thought as the projection of the 3D velocities on the image plane.
![Page 19: 3D IMAGING Motion. Extract visual information from image sequences Assume illumination conditions do not change Differences in image Camera moves](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f395503460f94c55e51/html5/thumbnails/19.jpg)
2D Motion Field and 2D Optical Flow Motion field: projection of 3D motion vectors on image plane
• Optical flow field: apparent motion of brightness patterns
• We equate motion field with optical flow field
0 0
00
Object point has velocity , induces in imagei
ii
P
d d
dt dt
v v
r rv v
![Page 20: 3D IMAGING Motion. Extract visual information from image sequences Assume illumination conditions do not change Differences in image Camera moves](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f395503460f94c55e51/html5/thumbnails/20.jpg)
2 Cases Where this Assumption Clearly is not Valid
(a) (b)
(a) A smooth sphere is rotating under constant illumination. Thus the optical flow field is zero, but the motion field is not.
(b) A fixed sphere is illuminated by a moving source—the shading of the image changes. Thus the motion field is zero, but the optical flow field is not.
![Page 21: 3D IMAGING Motion. Extract visual information from image sequences Assume illumination conditions do not change Differences in image Camera moves](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f395503460f94c55e51/html5/thumbnails/21.jpg)
What is Meant by Apparent Motion of Brightness Pattern?
The apparent motion of brightness patterns is an awkward concept. It is not easy to decide which point P' on a contour C' of constant brightness in the second image corresponds to a particular point P on the corresponding contour C in the first image.
![Page 22: 3D IMAGING Motion. Extract visual information from image sequences Assume illumination conditions do not change Differences in image Camera moves](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f395503460f94c55e51/html5/thumbnails/22.jpg)
Aperture Problem
(a) Line feature observed through a small aperture at time t.
(b) At time t+t the feature has moved to a new position. It is not possible to determine exactly where each point has moved. From local image measurements only the flow component perpendicular to the line feature can be computed.
Normal flow: Component of flow perpendicular to line feature.
(a) (b)
![Page 23: 3D IMAGING Motion. Extract visual information from image sequences Assume illumination conditions do not change Differences in image Camera moves](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f395503460f94c55e51/html5/thumbnails/23.jpg)
Brightness Constancy Equation Let P be a moving point in 3D:
At time t, P has coords (X(t),Y(t),Z(t)) Let p=(x(t),y(t)) be the coords. of its image
at time t. Let E(x(t),y(t),t) be the brightness at p at
time t. Brightness Constancy Assumption:
As P moves over time, E(x(t),y(t),t) remains constant.
![Page 24: 3D IMAGING Motion. Extract visual information from image sequences Assume illumination conditions do not change Differences in image Camera moves](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f395503460f94c55e51/html5/thumbnails/24.jpg)
Brightness Constraint Equation
flow. optical of components the, ,, and irradiance thebe ,,Let yxvyxutyxE
expansionTaylor
,,,, tyxEtttvytuxE
0limit takingand by dividing
,,,,
tt
tyxEet
Et
y
Ey
x
ExtyxE
0
derivative total theofexpansion theiswhich
0
dt
dE
t
E
dt
dy
y
E
dt
dx
x
E
short: 0 tyx EvEuE
![Page 25: 3D IMAGING Motion. Extract visual information from image sequences Assume illumination conditions do not change Differences in image Camera moves](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f395503460f94c55e51/html5/thumbnails/25.jpg)
Brightness Constancy Equation
Taking derivative wrt time:
![Page 26: 3D IMAGING Motion. Extract visual information from image sequences Assume illumination conditions do not change Differences in image Camera moves](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f395503460f94c55e51/html5/thumbnails/26.jpg)
Brightness Constancy Equation
Let(Frame spatial gradient)
(optical flow)
and(derivative across frames)
![Page 27: 3D IMAGING Motion. Extract visual information from image sequences Assume illumination conditions do not change Differences in image Camera moves](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f395503460f94c55e51/html5/thumbnails/27.jpg)
Brightness Constancy Equation
Becomes:
The OF is CONSTRAINED to be on a line !
![Page 28: 3D IMAGING Motion. Extract visual information from image sequences Assume illumination conditions do not change Differences in image Camera moves](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f395503460f94c55e51/html5/thumbnails/28.jpg)
Normal Motion/Aperture Problem
![Page 29: 3D IMAGING Motion. Extract visual information from image sequences Assume illumination conditions do not change Differences in image Camera moves](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f395503460f94c55e51/html5/thumbnails/29.jpg)
Barber Pole Illusion
![Page 30: 3D IMAGING Motion. Extract visual information from image sequences Assume illumination conditions do not change Differences in image Camera moves](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f395503460f94c55e51/html5/thumbnails/30.jpg)
Solving the aperture problem How to get more equations for a pixel?
Basic idea: impose additional constraints most common is to assume that the flow field is
smooth locally one method: pretend the pixel’s neighbors have the
same (u,v) If we use a 5x5 window, that gives us 25 equations per
pixel!
![Page 31: 3D IMAGING Motion. Extract visual information from image sequences Assume illumination conditions do not change Differences in image Camera moves](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f395503460f94c55e51/html5/thumbnails/31.jpg)
Constant flow Prob: we have more equations than unknowns
– The summations are over all pixels in the K x K window
• Solution: solve least squares problem
– minimum least squares solution given by solution (in d) of:
![Page 32: 3D IMAGING Motion. Extract visual information from image sequences Assume illumination conditions do not change Differences in image Camera moves](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f395503460f94c55e51/html5/thumbnails/32.jpg)
Taking a closer look at (ATA)
This is the same matrix we used for corner detection!
![Page 33: 3D IMAGING Motion. Extract visual information from image sequences Assume illumination conditions do not change Differences in image Camera moves](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f395503460f94c55e51/html5/thumbnails/33.jpg)
Taking a closer look at (ATA)
The matrix for corner detection:
is singular (not invertible) when det(ATA) = 0
One e.v. = 0 -> no corner, just an edgeTwo e.v. = 0 -> no corner, homogeneous region
Aperture Problem !
But det(ATA) = Õ li = 0 -> one or both e.v. are 0
![Page 34: 3D IMAGING Motion. Extract visual information from image sequences Assume illumination conditions do not change Differences in image Camera moves](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f395503460f94c55e51/html5/thumbnails/34.jpg)
A Pattern of Hajime Ouchi
![Page 36: 3D IMAGING Motion. Extract visual information from image sequences Assume illumination conditions do not change Differences in image Camera moves](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f395503460f94c55e51/html5/thumbnails/36.jpg)
Reconstruction
![Page 37: 3D IMAGING Motion. Extract visual information from image sequences Assume illumination conditions do not change Differences in image Camera moves](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f395503460f94c55e51/html5/thumbnails/37.jpg)
Rotation
P
rrr
rrr
rrr
3,32,31,3
3,22,21,2
3,12,11,1 Represents a 3D rotation of the points in P.
![Page 38: 3D IMAGING Motion. Extract visual information from image sequences Assume illumination conditions do not change Differences in image Camera moves](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f395503460f94c55e51/html5/thumbnails/38.jpg)
First, look at 2D rotation (easier)
n
n
yyy
xxx
21
21 ...
cossin
sincos
cossin
sincosR
Matrix R acts on points by rotating them.
• Also, RRT = Identity. RT is also a rotation matrix, in the opposite direction to R.
![Page 39: 3D IMAGING Motion. Extract visual information from image sequences Assume illumination conditions do not change Differences in image Camera moves](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f395503460f94c55e51/html5/thumbnails/39.jpg)
Simple 3D Rotation
n
n
n
zzz
yyy
xxx
21
21
21...
100
0cossin
0sincos
Rotation about z axis.
Rotates x,y coordinates. Leaves z coordinates fixed.
![Page 40: 3D IMAGING Motion. Extract visual information from image sequences Assume illumination conditions do not change Differences in image Camera moves](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f395503460f94c55e51/html5/thumbnails/40.jpg)
Full 3D Rotation
cossin0
sincos0
001
cos0sin
010
sin0cos
100
0cossin
0sincos
R
• Any rotation can be expressed as combination of three rotations about three axes.
100
010
001TRR
• Rows (and columns) of R are orthonormal vectors.
• R has determinant 1 (not -1).
![Page 41: 3D IMAGING Motion. Extract visual information from image sequences Assume illumination conditions do not change Differences in image Camera moves](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f395503460f94c55e51/html5/thumbnails/41.jpg)
Alper Yilmaz, Fall 2004 UCF
3D Motion
Displacement model Velocity model
Z
Y
X
TZYXZ
TZYXY
TZYXX
Z
Y
X
T
T
T
Z
Y
X
Z
Y
X
Z
Y
X
1
1
1
TR
![Page 42: 3D IMAGING Motion. Extract visual information from image sequences Assume illumination conditions do not change Differences in image Camera moves](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f395503460f94c55e51/html5/thumbnails/42.jpg)
Displacement Model and Its Projection onto Image Space
Z
Y
X
TZYXZ
TZYXY
TZYXX
Perspective projection
ZT
yx
ZT
yxy
ZT
yx
ZT
yxx
Z
Y
Z
X
1
1
x X Y Z TX
X Y Z TZ
y X Y Z TY
X Y Z TZ
![Page 43: 3D IMAGING Motion. Extract visual information from image sequences Assume illumination conditions do not change Differences in image Camera moves](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f395503460f94c55e51/html5/thumbnails/43.jpg)
Alper Yilmaz, Fall 2004 UCF
Velocity Model in 3DOptical Flow
Z
Y
X
T
T
T
Z
Y
X
Z
Y
X
1
1
1
Z
Y
X
T
T
T
Z
Y
X
ZZ
YY
XX
0
0
0
Z
Y
X
T
T
T
Z
Y
X
Z
Y
X
100
010
001
0
0
0
![Page 44: 3D IMAGING Motion. Extract visual information from image sequences Assume illumination conditions do not change Differences in image Camera moves](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f395503460f94c55e51/html5/thumbnails/44.jpg)
Alper Yilmaz, Fall 2004 UCF
Velocity Model in 3DOptical Flow
3
2
1
12
13
23
0
0
0
V
V
V
Z
Y
X
Z
Y
X
321
213
132
VXYZ
VZXY
VYZX
Z
Y
X
T
T
T
Z
Y
X
ZZ
YY
XX
0
0
0
rotational velocities
translational velocities
3D optical flow
VXX
3
2
1
Z
Y
X
X
cross product
![Page 45: 3D IMAGING Motion. Extract visual information from image sequences Assume illumination conditions do not change Differences in image Camera moves](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f395503460f94c55e51/html5/thumbnails/45.jpg)
Velocity Model in 2D
Perspective projection
321
213
132
VXYZ
VZXY
VYZX
Z
Yfy
Z
Xfx
yv
xu
212331
2
2213
32
1
)(
)(
yf
xyf
yZ
Vx
Z
Vfv
xf
xyf
yxZ
V
Z
Vfu