digital visual effects, spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4....
TRANSCRIPT
![Page 1: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/1.jpg)
Motion estimation
Digital Visual Effects, Spring 2008Yung-Yu Chuang2008/4/8
with slides by Michael Black and P. Anandan
![Page 2: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/2.jpg)
Announcements
• Artifacts #1 votinghttp://140.112.29.103/~hsiao/cgi-bin/votepage.cgi
• Some scribes are online• Project #2 checkpoint due on this Friday. Send
an image to TA.
![Page 3: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/3.jpg)
Motion estimation
• Parametric motion (image alignment)• Tracking• Optical flow
![Page 4: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/4.jpg)
Parametric motion
direct method for image stitching
![Page 5: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/5.jpg)
Tracking
![Page 6: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/6.jpg)
Optical flow
![Page 7: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/7.jpg)
Three assumptions
• Brightness consistency• Spatial coherence• Temporal persistence
![Page 8: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/8.jpg)
Brightness consistency
Image measurement (e.g. brightness) in a small region remain the same although their location may change.
![Page 9: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/9.jpg)
Spatial coherence
• Neighboring points in the scene typically belong to the same surface and hence typically have similar motions.
• Since they also project to nearby pixels in the image, we expect spatial coherence in image flow.
![Page 10: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/10.jpg)
Temporal persistence
The image motion of a surface patch changes gradually over time.
![Page 11: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/11.jpg)
Image registration
Goal: register a template image T(x) and an input image I(x), where x=(x,y)T. (warp I so that it matches T)
Image alignment: I(x) and T(x) are two imagesTracking: T(x) is a small patch around a point p in
the image at t. I(x) is the image at time t+1. Optical flow: T(x) and I(x) are patches of images
at t and t+1.
T
fixed
Iwarp
![Page 12: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/12.jpg)
Simple approach (for translation)
• Minimize brightness difference
( )∑ −++=yx
yxTvyuxIvuE,
2),(),(),(
![Page 13: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/13.jpg)
Simple SSD algorithm
For each offset (u, v)compute E(u,v);
Choose (u, v) which minimizes E(u,v);
Problems:• Not efficient• No sub-pixel accuracy
![Page 14: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/14.jpg)
Lucas-Kanade algorithm
![Page 15: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/15.jpg)
Newton’s method
• Root finding for f(x)=0• March x and test signs• Determine Δx (small→slow; large→ miss)
![Page 16: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/16.jpg)
Newton’s method
• Root finding for f(x)=0Taylor’s expansion:
K+++=+ 20000 )(''
21)(')()( εεε xfxfxfxf
εε )(')()( 000 xfxfxf +≈+
)(')(
n
nn xf
xf−=ε
)(')(
1n
nnn xf
xfxx −=+
![Page 17: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/17.jpg)
Newton’s method
• Root finding for f(x)=0
x0x1x2
![Page 18: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/18.jpg)
Newton’s method
pick up x=x0iterate
compute
update x by x+Δxuntil converge
Finding root is useful for optimization because Minimize g(x) → find root for f(x)=g’(x)=0
)(')(
xfxfx −=Δ
![Page 19: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/19.jpg)
Lucas-Kanade algorithm
( )∑ −++=yx
yxTvyuxIvuE,
2),(),(),(
yx vIuIyxIvyuxI ++≈++ ),(),(
( )∑ ++−=yx
yx vIuIyxTyxI,
2),(),(
( )∑ ++−=∂∂
=yx
yxx vIuIyxTyxIIuE
,),(),(20
( )∑ ++−=∂∂
=yx
yxy vIuIyxTyxIIvE
,),(),(20
![Page 20: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/20.jpg)
Lucas-Kanade algorithm
( )∑ ++−=∂∂
=yx
yxx vIuIyxTyxIIuE
,),(),(20
( )∑ ++−=∂∂
=yx
yxy vIuIyxTyxIIvE
,),(),(20
( )
( )⎪⎩
⎪⎨
⎧
−=+
−=+
∑∑∑∑ ∑∑
yxy
yxy
yxyx
yx yxxyx
yxx
yxIyxTIvIuII
yxIyxTIvIIuI
,,
2
,
, ,,
2
),(),(
),(),(
( )
( )⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
−
−=⎥
⎦
⎤⎢⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
∑∑
∑∑∑∑
yxy
yxx
yxy
yxyx
yxyx
yxx
yxIyxTI
yxIyxTI
vu
III
III
,
,
,
2
,
,,
2
),(),(
),(),(
![Page 21: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/21.jpg)
Lucas-Kanade algorithm
iterateshift I(x,y) with (u,v)compute gradient image Ix, Iycompute error image T(x,y)-I(x,y)compute Hessian matrixsolve the linear system(u,v)=(u,v)+(∆u,∆v)
until converge
( )
( )⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
−
−=⎥
⎦
⎤⎢⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
∑∑
∑∑∑∑
yxy
yxx
yxy
yxyx
yxyx
yxx
yxIyxTI
yxIyxTI
vu
III
III
,
,
,
2
,
,,
2
),(),(
),(),(
![Page 22: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/22.jpg)
Parametric model
( )∑ −++=yx
yxTvyuxIvuE,
2),(),(),(
( )∑ −=x
xp)W(x;p 2)()()( TIE
Tyx
y
x ddpdydx
),(, =⎟⎟⎠
⎞⎜⎜⎝
⎛++
=p)W(x;translation
Tyxyyyxxyxx
yyyyx
xxyxx
ddddddp
yx
dddddd
),,,,,(
,1
11
=
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
⎟⎟⎠
⎞⎜⎜⎝
⎛+
+=+= dAxp)W(x;affine
Our goal is to find p to minimize E(p)
for all x in T’s domain
![Page 23: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/23.jpg)
Parametric model
( )∑ −+x
xΔp)pW(x; 2)()( TIminimize
with respect to Δp
ΔppWp)W(x;Δp)pW(x;∂∂
+≈+
)()( ΔppWp)W(x;Δp)pW(x;∂∂
+≈+ II
ΔppW
xp)W(x;
∂∂
∂∂
+≈II )(
∑ ⎟⎟⎠
⎞⎜⎜⎝
⎛−
∂∂
∇+x
xΔppWp)W(x;
2
)()( TIIminimize
![Page 24: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/24.jpg)
Parametric model
∑ ⎟⎟⎠
⎞⎜⎜⎝
⎛−
∂∂
∇+x
xΔppWp)W(x;
2
)()( TII
image gradient
Jacobian of the warp
warped image
⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜
⎝
⎛
∂
∂
∂
∂
∂
∂∂∂
∂∂
∂∂
=
⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜
⎝
⎛
∂
∂∂∂
=∂∂
n
yyy
n
xxx
y
x
pW
pW
pW
pW
pW
pW
pWpW
pW
L
L
21
21
target image
![Page 25: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/25.jpg)
Jacobian matrix
• The Jacobian matrix is the matrix of all first-order partial derivatives of a vector-valued function.
mnF RR →:)),,(),,,(),,,((
),,(
21212211
21
nmnn
n
xxxfxxxfxxxfxxxF
KKKK
K
=
),,(),,( or ),,(
21
21
21
n
m
nF
xxxfff
xxxJ
K
K
K
∂∂
=
⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢
⎣
⎡
∂∂
∂∂
∂∂
∂∂
n
mm
n
xf
xf
xf
xf
L
MOM
L
1
1
1
1
ΔxxxΔxx )()()( FJFF +≈+
![Page 26: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/26.jpg)
Jacobian matrix
[ ] [ ] 32,0,0: RR →×× ππF
φθφθφ
cossinsincossin
rvrurt
===
),,(),,( vutrF =θφ
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
−
−=
⎥⎥⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢⎢⎢
⎣
⎡
∂∂
∂∂
∂∂
∂∂
∂∂
∂∂
∂∂
∂∂
∂∂
=
0sincoscossinsincossinsinsinsincoscoscossin
),,(
φφθφθφθφθφθφθφ
θφ
θφ
θφ
θφ
rrrrr
vvrv
uuru
ttrt
rJ F
![Page 27: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/27.jpg)
Parametric model
∑ ⎟⎟⎠
⎞⎜⎜⎝
⎛−
∂∂
∇+x
xΔppWp)W(x;
2
)()( TII
image gradient
Jacobian of the warp
warped image
⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜
⎝
⎛
∂
∂
∂
∂
∂
∂∂∂
∂∂
∂∂
=
⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜
⎝
⎛
∂
∂∂∂
=∂∂
n
yyy
n
xxx
y
x
pW
pW
pW
pW
pW
pW
pWpW
pW
L
L
21
21
target image
![Page 28: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/28.jpg)
Jacobian of the warp
For example, for affine
⎟⎟⎠
⎞⎜⎜⎝
⎛++++++
=⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
⎟⎟⎠
⎞⎜⎜⎝
⎛+
+=
yyyyx
xxyxx
yyyyx
xxyxx
dydxddydxd
yx
dddddd
)1()1(
11
1p)W(x;
⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜
⎝
⎛
∂
∂
∂
∂
∂
∂∂∂
∂∂
∂∂
=
⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜
⎝
⎛
∂
∂∂∂
=∂∂
n
yyy
n
xxx
y
x
pW
pW
pW
pW
pW
pW
pWpW
pW
L
L
21
21
⎟⎟⎠
⎞⎜⎜⎝
⎛=
∂∂
10000100
yxyx
pW
dxx dyx dxy dyy dx dy
![Page 29: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/29.jpg)
Parametric model
∑ ⎟⎟⎠
⎞⎜⎜⎝
⎛−
∂∂
∇+x
xΔppWp)W(x;
2
)()( TII
∑ ⎥⎦
⎤⎢⎣
⎡−
∂∂
∇+⎥⎦
⎤⎢⎣
⎡∂∂
∇=x
xΔppWp)W(x;
pW )()(0 TIII
T
[ ]∑ −⎥⎦
⎤⎢⎣
⎡∂∂
∇= −
xp)W(x;x
pWHΔp )()(1 ITI
T
∑ ⎥⎦
⎤⎢⎣
⎡∂∂
∇⎥⎦
⎤⎢⎣
⎡∂∂
∇=x p
WpWH II
T
(Approximated) Hessian
Δpminarg
![Page 30: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/30.jpg)
Lucas-Kanade algorithm
iterate1) warp I with W(x;p)2) compute error image T(x,y)-I(W(x,p))3) compute gradient image with W(x,p)4) evaluate Jacobian at (x;p)5) compute 6) compute Hessian7) compute 8) solve 9) update p by p+until converge
pW∂∂
pW∂∂
∇I
[ ]∑ −⎥⎦
⎤⎢⎣
⎡∂∂
∇x
p)W(x;xpW )()( ITI
T
ΔpΔp
[ ]∑ −⎥⎦
⎤⎢⎣
⎡∂∂
∇= −
xp)W(x;x
pWHΔp )()(1 ITI
T
I∇
![Page 31: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/31.jpg)
[ ]∑ −⎥⎦
⎤⎢⎣
⎡∂∂
∇= −
xp)W(x;x
pWHΔp )()(1 ITI
T
![Page 32: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/32.jpg)
[ ]∑ −⎥⎦
⎤⎢⎣
⎡∂∂
∇= −
xp)W(x;x
pWHΔp )()(1 ITI
T
![Page 33: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/33.jpg)
[ ]∑ −⎥⎦
⎤⎢⎣
⎡∂∂
∇= −
xp)W(x;x
pWHΔp )()(1 ITI
T
![Page 34: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/34.jpg)
[ ]∑ −⎥⎦
⎤⎢⎣
⎡∂∂
∇= −
xp)W(x;x
pWHΔp )()(1 ITI
T
![Page 35: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/35.jpg)
[ ]∑ −⎥⎦
⎤⎢⎣
⎡∂∂
∇= −
xp)W(x;x
pWHΔp )()(1 ITI
T
![Page 36: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/36.jpg)
[ ]∑ −⎥⎦
⎤⎢⎣
⎡∂∂
∇= −
xp)W(x;x
pWHΔp )()(1 ITI
T
![Page 37: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/37.jpg)
[ ]∑ −⎥⎦
⎤⎢⎣
⎡∂∂
∇= −
xp)W(x;x
pWHΔp )()(1 ITI
T
![Page 38: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/38.jpg)
[ ]∑ −⎥⎦
⎤⎢⎣
⎡∂∂
∇= −
xp)W(x;x
pWHΔp )()(1 ITI
T
![Page 39: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/39.jpg)
[ ]∑ −⎥⎦
⎤⎢⎣
⎡∂∂
∇= −
xp)W(x;x
pWHΔp )()(1 ITI
T
![Page 40: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/40.jpg)
[ ]∑ −⎥⎦
⎤⎢⎣
⎡∂∂
∇= −
xp)W(x;x
pWHΔp )()(1 ITI
T
![Page 41: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/41.jpg)
[ ]∑ −⎥⎦
⎤⎢⎣
⎡∂∂
∇= −
xp)W(x;x
pWHΔp )()(1 ITI
T
![Page 42: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/42.jpg)
Coarse-to-fine strategy
J Jw Iwarp refine
inav
avΔ+
J Jw Iwarp refine
av
avΔ+
J
pyramid construction
J Jw Iwarp refine
avΔ+
I
pyramid construction
outav
![Page 43: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/43.jpg)
Application of image alignment
![Page 44: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/44.jpg)
Direct vs feature-based
• Direct methods use all information and can be very accurate, but they depend on the fragile “brightness constancy” assumption.
• Iterative approaches require initialization.• Not robust to illumination change and noise
images.• In early days, direct method is better.
• Feature based methods are now more robust and potentially faster.
• Even better, it can recognize panorama without initialization.
![Page 45: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/45.jpg)
Tracking
![Page 46: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/46.jpg)
Tracking
I(x,y,t) I(x,y,t+1)
![Page 47: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/47.jpg)
Tracking
0),,()1,,( =−+++ tyxItvyuxI
0),,(),,(),,(),,(),,( ≈−+++ tyxItyxItyxvItyxuItyxI tyx
0),,(),,(),,( =++ tyxItyxvItyxuI tyx
0=++ tyx IvIuI optical flow constraint equation
brightness constancy
![Page 48: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/48.jpg)
Optical flow constraint equation
![Page 49: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/49.jpg)
Multiple constraints
![Page 50: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/50.jpg)
Area-based method• Assume spatial smoothness
![Page 51: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/51.jpg)
Area-based method
• Assume spatial smoothness
( )∑ ++=yx
tyx IvIuIvuE,
2),(
![Page 52: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/52.jpg)
Area-based method
must be invertible
![Page 53: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/53.jpg)
Area-based method• The eigenvalues tell us about the local image
structure.• They also tell us how well we can estimate the
flow in both directions.• Link to Harris corner detector.
![Page 54: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/54.jpg)
Textured area
![Page 55: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/55.jpg)
Edge
![Page 56: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/56.jpg)
Homogenous area
![Page 57: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/57.jpg)
KLT tracking
• Select feature by • Monitor features by measuring dissimilarity
λλλ >),( 21min
![Page 58: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/58.jpg)
Aperture problem
![Page 59: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/59.jpg)
Aperture problem
![Page 60: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/60.jpg)
Aperture problem
![Page 61: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/61.jpg)
Demo for aperture problem
• http://www.sandlotscience.com/Distortions/Breathing_Square.htm
• http://www.sandlotscience.com/Ambiguous/Barberpole_Illusion.htm
![Page 62: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/62.jpg)
Aperture problem
• Larger window reduces ambiguity, but easily violates spatial smoothness assumption
![Page 63: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/63.jpg)
![Page 64: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/64.jpg)
![Page 65: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/65.jpg)
![Page 66: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/66.jpg)
KLT tracking
http://www.ces.clemson.edu/~stb/klt/
![Page 67: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/67.jpg)
KLT tracking
http://www.ces.clemson.edu/~stb/klt/
![Page 68: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/68.jpg)
SIFT tracking (matching actually)
Frame 0 Frame 10
![Page 69: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/69.jpg)
SIFT tracking
Frame 0 Frame 100
![Page 70: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/70.jpg)
SIFT tracking
Frame 0 Frame 200
![Page 71: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/71.jpg)
KLT vs SIFT tracking
• KLT has larger accumulating error; partly because our KLT implementation doesn’t have affine transformation?
• SIFT is surprisingly robust• Combination of SIFT and KLT (example)
http://www.frc.ri.cmu.edu/projects/buzzard/smalls/
![Page 72: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/72.jpg)
Rotoscoping (Max Fleischer 1914)
1937
![Page 73: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/73.jpg)
Tracking for rotoscoping
![Page 74: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/74.jpg)
Tracking for rotoscoping
![Page 75: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/75.jpg)
Waking life (2001)
![Page 76: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/76.jpg)
A Scanner Darkly (2006)
• Rotoshop, a proprietary software. Each minute of animation required 500 hours of work.
![Page 77: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/77.jpg)
Optical flow
![Page 78: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/78.jpg)
Single-motion assumption
Violated by• Motion discontinuity• Shadows• Transparency• Specular reflection• …
![Page 79: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/79.jpg)
Multiple motion
![Page 80: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/80.jpg)
Multiple motion
![Page 81: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/81.jpg)
Simple problem: fit a line
![Page 82: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/82.jpg)
Least-square fit
![Page 83: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/83.jpg)
Least-square fit
![Page 84: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/84.jpg)
Robust statistics
• Recover the best fit for the majority of the data
• Detect and reject outliers
![Page 85: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/85.jpg)
Approach
![Page 86: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/86.jpg)
Robust weighting
Truncated quadratic
![Page 87: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/87.jpg)
Robust weighting
Geman & McClure
![Page 88: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/88.jpg)
Robust estimation
![Page 89: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/89.jpg)
Fragmented occlusion
![Page 90: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/90.jpg)
![Page 91: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/91.jpg)
![Page 92: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/92.jpg)
Regularization and dense optical flow
• Neighboring points in the scene typically belong to the same surface and hence typically have similar motions.
• Since they also project to nearby pixels in the image, we expect spatial coherence in image flow.
![Page 93: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/93.jpg)
![Page 94: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/94.jpg)
![Page 95: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/95.jpg)
![Page 96: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/96.jpg)
![Page 97: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/97.jpg)
![Page 98: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/98.jpg)
![Page 99: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/99.jpg)
Input image Horizontal motion
Vertical motion
![Page 100: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/100.jpg)
![Page 101: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/101.jpg)
![Page 102: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/102.jpg)
Application of optical flow
videomatching
![Page 103: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/103.jpg)
![Page 104: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/104.jpg)
Input for the NPR algorithm
![Page 105: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/105.jpg)
Brushes
![Page 106: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/106.jpg)
Edge clipping
![Page 107: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/107.jpg)
Gradient
![Page 108: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/108.jpg)
Smooth gradient
![Page 109: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/109.jpg)
Textured brush
![Page 110: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/110.jpg)
Edge clipping
![Page 111: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/111.jpg)
Temporal artifacts
Frame-by-frame application of the NPR algorithm
![Page 112: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/112.jpg)
Temporal coherence
![Page 113: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/113.jpg)
What dreams may come
![Page 114: Digital Visual Effects, Spring 2008 - 國立臺灣大學cyy/courses/vfx/08spring/... · 2008. 4. 8. · Motion estimation Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/8](https://reader034.vdocument.in/reader034/viewer/2022051607/60384bb667d4707e690f3998/html5/thumbnails/114.jpg)
References• B.D. Lucas and T. Kanade, An Iterative Image Registration Technique with
an Application to Stereo Vision, Proceedings of the 1981 DARPA Image Understanding Workshop, 1981, pp121-130.
• Bergen, J. R. and Anandan, P. and Hanna, K. J. and Hingorani, R., Hierarchical Model-Based Motion Estimation, ECCV 1992, pp237-252.
• J. Shi and C. Tomasi, Good Features to Track, CVPR 1994, pp593-600. • Michael Black and P. Anandan, The Robust Estimation of Multiple Motions:
Parametric and Piecewise-Smooth Flow Fields, Computer Vision and Image Understanding 1996, pp75-104.
• S. Baker and I. Matthews, Lucas-Kanade 20 Years On: A Unifying Framework, International Journal of Computer Vision, 56(3), 2004, pp221 - 255.
• Peter Litwinowicz, Processing Images and Video for An Impressionist
Effects, SIGGRAPH 1997.
• Aseem Agarwala, Aaron Hertzman, David Salesin and Steven Seitz, Keyframe-Based Tracking for Rotoscoping and Animation, SIGGRAPH 2004, pp584-591.