lecture 20: motion estimationahowens/eecs504/w20/slides/...most slides from s. lazebnik, which are...
Post on 30-Jan-2021
2 Views
Preview:
TRANSCRIPT
-
1
Lecture 20: Motion estimation
Most slides from S. Lazebnik, which are based on other slides from S. Seitz, R. Szeliski, M. Pollefeys
-
2
Announcements• PS9 and PS10 deadlines extended to April 21
-
Optical flow
3 Source: S. Lazebnik
What moved where?
-
Motion is a powerful perceptual cue
4 Source: S. Lazebnik
-
Motion is a powerful perceptual cue
5
G. Johansson, “Visual Perception of Biological Motion and a Model For Its Analysis", Perception and Psychophysics 14, 201-211, 1973. Source: S. Lazebnik
-
Motion is a powerful perceptual cue
6
G. Johansson, “Visual Perception of Biological Motion and a Model For Its Analysis", Perception and Psychophysics 14, 201-211, 1973. Source: S. Lazebnik
-
Motion field
7 Source: S. Lazebnik
-
Estimating optical flow
8
• Common assumptions
• Brightness constancy: projection of the same point looks the
same in every frame
• Small motion: points do not move very far
• Spatial coherence: points move like their neighbors
I(x,y,t–1) I(x,y,t)
• Given two subsequent frames, estimate the motion field u(x,y) and v(x,y) between them
Source: S. Lazebnik
-
The brightness constancy constraint
I(x,y,t–1) I(x,y,t)
Simple loss function [Lucas & Kanade 1981]. Find flow that minimizes:
L(u, v) =X
x,y
[I(x, y, t� 1)� I(x+ u(x, y), y + v(x, y), t)]2AAACOXicbVBLSwMxEM76tr6qHr0MFqHFWnZV0IsgelHwUMGq0K4lm6YazD5IJsWy9G958V94E7x4UMSrf8BsreBrYMj3fTPDZL4gkUKj6z44Q8Mjo2PjE5O5qemZ2bn8/MKpjo1ivMZiGavzgGouRcRrKFDy80RxGgaSnwXX+1n9rMOVFnF0gt2E+yG9jERbMIpWauarjZDiFaMyPeoVTRk6JdiBhjZhM70pQ7dXPyxmbxlwzSvBGlgKq2D6YsmmJZ0vgiX/Yr2ZL7gVtx/wF3gDUCCDqDbz941WzEzII2SSal333AT9lCoUTPJermE0Tyi7ppe8bmFEQ679tH95D1as0oJ2rGxGCH31+0RKQ627YWA7szv171om/lerG2xv+6mIEoM8Yp+L2kYCxpDZCC2hOEPZtYAyJexfgV1RRRlas3PWBO/3yX/B6XrF26i4x5uF3b2BHRNkiSyTIvHIFtklB6RKaoSRW/JInsmLc+c8Oa/O22frkDOYWSQ/wnn/ALA2pq8=
Source: S. Lazebnik
-
),()1,,( ),,(),( tyxyx vyuxItyxI ++=−
),(),(),,()1,,( yxvIyxuItyxItyxI yx ++≈−
Linearizing the right side using Taylor expansion:
The brightness constancy constraint
10
I(x,y,t–1) I(x,y,t)
0≈++ tyx IvIuIHence,
Brightness Constancy Equation:
Derivative in y direction
Source: S. Lazebnik
-
The brightness constancy constraint
11
• How many equations and unknowns per pixel?
• One equation, two unknowns
• What does this constraint mean?
• The component of the flow perpendicular to the gradient (i.e., parallel to the edge) is unknown!
0=++ tyx IvIuI
0),( =+⋅∇ tIvuI
0)','( =⋅∇ vuI
edge
(u,v)
(u’,v’)
gradient
(u+u’,v+v’)
If (u, v) satisfies the equation, so does (u+u’, v+v’) if
Source: S. Lazebnik
-
The aperture problem
12
Perceived motion
Source: S. Lazebnik
-
The aperture problem
13
Actual motion
Source: S. Lazebnik
-
The barber pole illusion
14http://en.wikipedia.org/wiki/Barberpole_illusion
Source: S. Lazebnik
http://en.wikipedia.org/wiki/Barberpole_illusion
-
The barber pole illusion
15http://en.wikipedia.org/wiki/Barberpole_illusion
Source: S. Lazebnik
http://en.wikipedia.org/wiki/Barberpole_illusion
-
• How to get more equations for a pixel?
• Spatial coherence constraint: assume the pixel’s
neighbors have the same (u,v)
• E.g., if we use a 5x5 window, that gives us 25 equations per pixel
Solving the aperture problem
16
0)(],[)( =+⋅∇ iti IvuI xx
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
−=⎥⎦
⎤⎢⎣
⎡
⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢
⎣
⎡
)(
)()(
)()(
)()()()(
2
1
22
11
nt
t
t
nynx
yx
yx
I
II
vu
II
IIII
x
xx
xx
xxxx
!!!… … …
[Lucas & Kanade 1981] Source: S. Lazebnik
-
Lucas-Kanade flow
17
When is this system solvable?
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
−=⎥⎦
⎤⎢⎣
⎡
⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢
⎣
⎡
)(
)()(
)()(
)()()()(
2
1
22
11
nt
t
t
nynx
yx
yx
I
II
vu
II
IIII
x
xx
xx
xxxx
!!!
[Lucas & Kanade 1981]
Least squares problem:
… … …Source: S. Lazebnik
-
Lucas-Kanade optical flow
18
(summations are over all pixels in the window)
• Solution given by
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
−=⎥⎦
⎤⎢⎣
⎡
⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢
⎣
⎡
)(
)()(
)()(
)()()()(
2
1
22
11
nt
t
t
nynx
yx
yx
I
II
vu
II
IIII
x
xx
xx
xxxx
!!! 1122 ×××=nnbdA
bAA)dA TT =(
⎥⎥⎦
⎤
⎢⎢⎣
⎡−=⎥
⎦
⎤⎢⎣
⎡
⎥⎥⎦
⎤
⎢⎢⎣
⎡
∑∑
∑∑∑∑
ty
tx
yyyx
yxxx
IIII
vu
IIIIIIII
M = ATA is the “second moment” matrix
[Lucas & Kanade 1981]
… … …Source: S. Lazebnik
-
Analyzing the second moment matrix
19 λ1
λ2
“Corner” λ1 and λ2 are large, λ1 ~ λ2
λ1 and λ2 are small “Edge” λ1 >> λ2
“Edge” λ2 >> λ1
“Flat” region
• Estimation of optical flow is well-conditioned precisely for regions with high “cornerness”:
Eigenvalues
Source: S. Lazebnik
-
Conditions for solvability
20 Source: S. Lazebnik
-
Conditions for solvability
21 Source: S. Lazebnik
-
Lucas-Kanade flow example
22
Input frames Output
Source: MATLAB Central File Exchange Source: S. Lazebnik
https://www.mathworks.com/matlabcentral/fileexchange/48744-lucas-kanade-tutorial-example-1
-
Fixing the errors in Lucas-Kanade
23
• The motion is large (larger than a pixel)
• Iterative refinement
• Multi-resolution (coarse-to-fine) estimation
• Local ambiguity
• Smooth using graphical model refinement
Source: S. Lazebnik
-
Large motions
24 Source: Khurram Hassan-Shafique CAP5415 Computer Vision 2003
-
Initialize flow: u0(x, y) = v0(x, y) = 0 For each iteration i:
1. Linearize around current solution. Find an update for problem: by solving linear least squares problem for each pixel:
2. Apply updates: and
Idea #1: iterative estimationGoal: minimize matching error
Iterative algorithm:
L(u, v) =X
x,y
x+NX
x0=x�N
y+NX
y0=y�N[I(x0, y0, t� 1)� I(x0 + u(x, y), y0 + v(x, y), t)]2
AAACcHicbZFda9swFIZlt1vb7CttbgrbmLYwmqxtsNtCd1MI3c0GY2SwJIXEDbKitKLyB9JRsDG+7v/rXX9Eb/oLJjsebMkOCL96n3Ns6bUfC67Ace4te239ydONza3as+cvXr6qb+8MVKQlZX0aiUhe+EQxwUPWBw6CXcSSkcAXbOjffCn4cM6k4lH4C9KYeQG5CvmMUwLGmtRvxwGBa0pE9j1v6QM8b+MzPFY6mGTJAU7zSu4ZN8GH+Ed+mSV43zyrprQgaUXSkoy+tZI9M2sWHLptwwrDIN0qXtkukNnN/+yg7V0eTepNp+OUhVeFW4kmqqo3qd+NpxHVAQuBCqLUyHVi8DIigVPB8tpYKxYTekOu2MjIkARMeVkZWI4/GmeKZ5E0KwRcun9PZCRQKg1801nEo5ZZYf6PjTTMPnsZD2MNLKSLD820wBDhIn085ZJREKkRhEpuzorpNZGEgvlHNROCu3zlVTE46rjHHefnSbN7XsWxiV6jD6iFXHSKuugr6qE+oujBalhvrLfWo71rv7PfL1ptq5ppoH/K/vQbOZqyVw==
where the window width/height is 2N+1
�u,�v = min�u,�v
L(ui +�u, vi +�v)AAACQ3icbVDNS8MwHE39nPOr6tFLcAgTZbQq6EUY6sGDhwnuA7dR0izbwtK0JGlhlP5vXvwHvPkPePGgiFfBdCs4Nx8EXt57P5LfcwNGpbKsF2NufmFxaTm3kl9dW9/YNLe2a9IPBSZV7DNfNFwkCaOcVBVVjDQCQZDnMlJ3B1epX4+IkNTn92oYkLaHepx2KUZKS4750LomTCEYHsGMRfACtjzKnXjWSrSDVB8jFt8mxdCh8BD+pqLJe3TgmAWrZI0AZ4mdkQLIUHHM51bHx6FHuMIMSdm0rUC1YyQUxYwk+VYoSYDwAPVIU1OOPCLb8aiDBO5rpQO7vtCHKzhSJydi5Ek59FydTFeQ014q/uc1Q9U9b8eUB6EiHI8f6oYMKh+mhcIOFQQrNtQEYUH1XyHuI4Gw0rXndQn29MqzpHZcsk9K1t1poXyZ1ZEDu2APFIENzkAZ3IAKqAIMHsEreAcfxpPxZnwaX+PonJHN7IA/ML5/APHGreo=
argminAAAB9HicbVDLSgMxFL3js9ZX1aWbYBFclRkVdFl047KCfUA7lEyaaUPzGJNMsQz9DjcuFHHrx7jzb0zbWWjrgQuHc+5N7j1Rwpmxvv/trayurW9sFraK2zu7e/ulg8OGUakmtE4UV7oVYUM5k7RumeW0lWiKRcRpMxreTv3miGrDlHyw44SGAvclixnB1klhR0TqKcO6L5icdEtlv+LPgJZJkJMy5Kh1S1+dniKpoNISjo1pB35iQ/ecZYTTSbGTGppgMsR92nZUYkFNmM2WnqBTp/RQrLQradFM/T2RYWHMWESuU2A7MIveVPzPa6c2vg4zJpPUUknmH8UpR1ahaQKoxzQllo8dwUQztysiA6wxsS6nogshWDx5mTTOK8FFxb+/LFdv8jgKcAwncAYBXEEV7qAGdSDwCM/wCm/eyHvx3r2PeeuKl88cwR94nz9vZ5KG�u,�v = min
�u,�vL(ui +�u, vi +�v)
AAACQ3icbVDNS8MwHE39nPOr6tFLcAgTZbQq6EUY6sGDhwnuA7dR0izbwtK0JGlhlP5vXvwHvPkPePGgiFfBdCs4Nx8EXt57P5LfcwNGpbKsF2NufmFxaTm3kl9dW9/YNLe2a9IPBSZV7DNfNFwkCaOcVBVVjDQCQZDnMlJ3B1epX4+IkNTn92oYkLaHepx2KUZKS4750LomTCEYHsGMRfACtjzKnXjWSrSDVB8jFt8mxdCh8BD+pqLJe3TgmAWrZI0AZ4mdkQLIUHHM51bHx6FHuMIMSdm0rUC1YyQUxYwk+VYoSYDwAPVIU1OOPCLb8aiDBO5rpQO7vtCHKzhSJydi5Ek59FydTFeQ014q/uc1Q9U9b8eUB6EiHI8f6oYMKh+mhcIOFQQrNtQEYUH1XyHuI4Gw0rXndQn29MqzpHZcsk9K1t1poXyZ1ZEDu2APFIENzkAZ3IAKqAIMHsEreAcfxpPxZnwaX+PonJHN7IA/ML5/APHGreo=
Ad = bAAAB7XicbVBNSwMxEJ2tX7V+VT16CRbBU9nVgl6EqhePFewHtEvJZrNtbDZZkqxQlv4HLx4U8er/8ea/MW33oK0PBh7vzTAzL0g408Z1v53Cyura+kZxs7S1vbO7V94/aGmZKkKbRHKpOgHWlDNBm4YZTjuJojgOOG0Ho9up336iSjMpHsw4oX6MB4JFjGBjpdZ1iK5Q0C9X3Ko7A1omXk4qkKPRL3/1QknSmApDONa667mJ8TOsDCOcTkq9VNMEkxEe0K6lAsdU+9ns2gk6sUqIIqlsCYNm6u+JDMdaj+PAdsbYDPWiNxX/87qpiS79jIkkNVSQ+aIo5chINH0dhUxRYvjYEkwUs7ciMsQKE2MDKtkQvMWXl0nrrOqdV937WqV+k8dRhCM4hlPw4ALqcAcNaAKBR3iGV3hzpPPivDsf89aCk88cwh84nz85l448
ui+1 = ui +�u, vi+1 = vi +�vAAACHHicbVDLSsNAFJ34rPUVdelmsAhCpSRW0I1Q1IXLCvYBbQiT6bQdOpmEeQRK6Ie48VfcuFDEjQvBv3HaBqmtBy4czrmXe+8JYkalcpxva2l5ZXVtPbeR39za3tm19/brMtICkxqOWCSaAZKEUU5qiipGmrEgKAwYaQSDm7HfSIiQNOIPahgTL0Q9TrsUI2Uk3y5rP6VFdwSvoPYpLML2LWEKQX0Kk18nmXUS6NsFp+RMABeJm5ECyFD17c92J8I6JFxhhqRsuU6svBQJRTEjo3xbSxIjPEA90jKUo5BIL508N4LHRunAbiRMcQUn6uxEikIph2FgOkOk+nLeG4v/eS2tupdeSnmsFeF4uqirGVQRHCcFO1QQrNjQEIQFNbdC3EcCYWXyzJsQ3PmXF0n9rOSWS879eaFyncWRA4fgCJwAF1yACrgDVVADGDyCZ/AK3qwn68V6tz6mrUtWNnMA/sD6+gEQTp48
ui+1 = ui +�u, vi+1 = vi +�vAAACHHicbVDLSsNAFJ34rPUVdelmsAhCpSRW0I1Q1IXLCvYBbQiT6bQdOpmEeQRK6Ie48VfcuFDEjQvBv3HaBqmtBy4czrmXe+8JYkalcpxva2l5ZXVtPbeR39za3tm19/brMtICkxqOWCSaAZKEUU5qiipGmrEgKAwYaQSDm7HfSIiQNOIPahgTL0Q9TrsUI2Uk3y5rP6VFdwSvoPYpLML2LWEKQX0Kk18nmXUS6NsFp+RMABeJm5ECyFD17c92J8I6JFxhhqRsuU6svBQJRTEjo3xbSxIjPEA90jKUo5BIL508N4LHRunAbiRMcQUn6uxEikIph2FgOkOk+nLeG4v/eS2tupdeSnmsFeF4uqirGVQRHCcFO1QQrNjQEIQFNbdC3EcCYWXyzJsQ3PmXF0n9rOSWS879eaFyncWRA4fgCJwAF1yACrgDVVADGDyCZ/AK3qwn68V6tz6mrUtWNnMA/sD6+gEQTp48
-
Idea #2: Multi-scale estimation
26 Source: Khurram Hassan-Shafique CAP5415 Computer Vision 2003
-
For each iteration i: 1. Linearize around current solution. Find an update for problem: by solving linear least squares problem for each pixel:
2. Apply updates: and
Idea #2: Multi-scale estimation
�u,�v = min�u,�v
L(ui +�u, vi +�v)AAACQ3icbVDNS8MwHE39nPOr6tFLcAgTZbQq6EUY6sGDhwnuA7dR0izbwtK0JGlhlP5vXvwHvPkPePGgiFfBdCs4Nx8EXt57P5LfcwNGpbKsF2NufmFxaTm3kl9dW9/YNLe2a9IPBSZV7DNfNFwkCaOcVBVVjDQCQZDnMlJ3B1epX4+IkNTn92oYkLaHepx2KUZKS4750LomTCEYHsGMRfACtjzKnXjWSrSDVB8jFt8mxdCh8BD+pqLJe3TgmAWrZI0AZ4mdkQLIUHHM51bHx6FHuMIMSdm0rUC1YyQUxYwk+VYoSYDwAPVIU1OOPCLb8aiDBO5rpQO7vtCHKzhSJydi5Ek59FydTFeQ014q/uc1Q9U9b8eUB6EiHI8f6oYMKh+mhcIOFQQrNtQEYUH1XyHuI4Gw0rXndQn29MqzpHZcsk9K1t1poXyZ1ZEDu2APFIENzkAZ3IAKqAIMHsEreAcfxpPxZnwaX+PonJHN7IA/ML5/APHGreo=
argminAAAB9HicbVDLSgMxFL3js9ZX1aWbYBFclRkVdFl047KCfUA7lEyaaUPzGJNMsQz9DjcuFHHrx7jzb0zbWWjrgQuHc+5N7j1Rwpmxvv/trayurW9sFraK2zu7e/ulg8OGUakmtE4UV7oVYUM5k7RumeW0lWiKRcRpMxreTv3miGrDlHyw44SGAvclixnB1klhR0TqKcO6L5icdEtlv+LPgJZJkJMy5Kh1S1+dniKpoNISjo1pB35iQ/ecZYTTSbGTGppgMsR92nZUYkFNmM2WnqBTp/RQrLQradFM/T2RYWHMWESuU2A7MIveVPzPa6c2vg4zJpPUUknmH8UpR1ahaQKoxzQllo8dwUQztysiA6wxsS6nogshWDx5mTTOK8FFxb+/LFdv8jgKcAwncAYBXEEV7qAGdSDwCM/wCm/eyHvx3r2PeeuKl88cwR94nz9vZ5KG�u,�v = min
�u,�vL(ui +�u, vi +�v)
AAACQ3icbVDNS8MwHE39nPOr6tFLcAgTZbQq6EUY6sGDhwnuA7dR0izbwtK0JGlhlP5vXvwHvPkPePGgiFfBdCs4Nx8EXt57P5LfcwNGpbKsF2NufmFxaTm3kl9dW9/YNLe2a9IPBSZV7DNfNFwkCaOcVBVVjDQCQZDnMlJ3B1epX4+IkNTn92oYkLaHepx2KUZKS4750LomTCEYHsGMRfACtjzKnXjWSrSDVB8jFt8mxdCh8BD+pqLJe3TgmAWrZI0AZ4mdkQLIUHHM51bHx6FHuMIMSdm0rUC1YyQUxYwk+VYoSYDwAPVIU1OOPCLb8aiDBO5rpQO7vtCHKzhSJydi5Ek59FydTFeQ014q/uc1Q9U9b8eUB6EiHI8f6oYMKh+mhcIOFQQrNtQEYUH1XyHuI4Gw0rXndQn29MqzpHZcsk9K1t1poXyZ1ZEDu2APFIENzkAZ3IAKqAIMHsEreAcfxpPxZnwaX+PonJHN7IA/ML5/APHGreo=
Ad = bAAAB7XicbVBNSwMxEJ2tX7V+VT16CRbBU9nVgl6EqhePFewHtEvJZrNtbDZZkqxQlv4HLx4U8er/8ea/MW33oK0PBh7vzTAzL0g408Z1v53Cyura+kZxs7S1vbO7V94/aGmZKkKbRHKpOgHWlDNBm4YZTjuJojgOOG0Ho9up336iSjMpHsw4oX6MB4JFjGBjpdZ1iK5Q0C9X3Ko7A1omXk4qkKPRL3/1QknSmApDONa667mJ8TOsDCOcTkq9VNMEkxEe0K6lAsdU+9ns2gk6sUqIIqlsCYNm6u+JDMdaj+PAdsbYDPWiNxX/87qpiS79jIkkNVSQ+aIo5chINH0dhUxRYvjYEkwUs7ciMsQKE2MDKtkQvMWXl0nrrOqdV937WqV+k8dRhCM4hlPw4ALqcAcNaAKBR3iGV3hzpPPivDsf89aCk88cwh84nz85l448
ui+1 = ui +�u, vi+1 = vi +�vAAACHHicbVDLSsNAFJ34rPUVdelmsAhCpSRW0I1Q1IXLCvYBbQiT6bQdOpmEeQRK6Ie48VfcuFDEjQvBv3HaBqmtBy4czrmXe+8JYkalcpxva2l5ZXVtPbeR39za3tm19/brMtICkxqOWCSaAZKEUU5qiipGmrEgKAwYaQSDm7HfSIiQNOIPahgTL0Q9TrsUI2Uk3y5rP6VFdwSvoPYpLML2LWEKQX0Kk18nmXUS6NsFp+RMABeJm5ECyFD17c92J8I6JFxhhqRsuU6svBQJRTEjo3xbSxIjPEA90jKUo5BIL508N4LHRunAbiRMcQUn6uxEikIph2FgOkOk+nLeG4v/eS2tupdeSnmsFeF4uqirGVQRHCcFO1QQrNjQEIQFNbdC3EcCYWXyzJsQ3PmXF0n9rOSWS879eaFyncWRA4fgCJwAF1yACrgDVVADGDyCZ/AK3qwn68V6tz6mrUtWNnMA/sD6+gEQTp48
ui+1 = ui +�u, vi+1 = vi +�vAAACHHicbVDLSsNAFJ34rPUVdelmsAhCpSRW0I1Q1IXLCvYBbQiT6bQdOpmEeQRK6Ie48VfcuFDEjQvBv3HaBqmtBy4czrmXe+8JYkalcpxva2l5ZXVtPbeR39za3tm19/brMtICkxqOWCSaAZKEUU5qiipGmrEgKAwYaQSDm7HfSIiQNOIPahgTL0Q9TrsUI2Uk3y5rP6VFdwSvoPYpLML2LWEKQX0Kk18nmXUS6NsFp+RMABeJm5ECyFD17c92J8I6JFxhhqRsuU6svBQJRTEjo3xbSxIjPEA90jKUo5BIL508N4LHRunAbiRMcQUn6uxEikIph2FgOkOk+nLeG4v/eS2tupdeSnmsFeF4uqirGVQRHCcFO1QQrNjQEIQFNbdC3EcCYWXyzJsQ3PmXF0n9rOSWS879eaFyncWRA4fgCJwAF1yACrgDVVADGDyCZ/AK3qwn68V6tz6mrUtWNnMA/sD6+gEQTp48
Initialize flow: u0(x, y) = v0(x, y) = 0 For each scale s: Initialize flow from previous (coarser) scale
3. Extra trick for smoother flow: apply median filter to ui+1 and vi+1
2x scale
2x flow magnitude
-
Example
28
Flow visualization
Coarse-to-fine LK with median filtering
Coarse-to-fine LK Input two frames
Source: Ce Liu
-
Smoothness assumption
X
x,y
[I(x, y, t� 1)� I(u(x), v(y), t)]2 +X
p
X
p02N(u(p)� u(p0))2 + (v(p)� v(p0))2
AAACYHicbVFBT9swFHYCjNLBKOM2Lk+rJhKtVAkgbUe0XeAyMWkFpCZUjuuCheNE9ktHFOVPctthl/0SnDaTGOxJ9vv0vffZfp+TXAqDQfDLcVdW116tdza6rze33mz3dt5emKzQjI9YJjN9lVDDpVB8hAIlv8o1p2ki+WVy97WpX865NiJTP7DMeZzSGyVmglG01KT3MzJFOqnuB1DW4zOvyQPAg9CHAzjzCu/eH8DcK+2Ofnx9CB9hKcjrv2AfIqEgSineMiqrb3UNVpc3B9i07/sLlTdfUvOWmvT6wTBYBLwEYQv6pI3zSe8hmmasSLlCJqkx4zDIMa6oRsEkr7tRYXhO2R294WMLFU25iauFQTV8sMwUZpm2SyEs2KeKiqbGlGliO5s5zPNaQ/6vNi5w9jmuhMoL5IotL5oVEjCDxm2YCs0ZytICyrSwbwV2SzVlaP+ka00In4/8ElwcDsOjYfD9uH/ypbWjQ/bIe+KRkHwiJ+SUnJMRYeS3s+JsOlvOH7fjbrs7y1bXaTW75J9w3z0CWeOtTQ==
Goal: minimize matching error + smoothness [Horn and Schunck 1981]X
x,y
[I(x, y, t� 1)� I(u(x), v(y), t)]2 +X
p
X
p02N(u(p)� u(p0))2 + (v(p)� v(p0))2
AAACYHicbVFBT9swFHYCjNLBKOM2Lk+rJhKtVAkgbUe0XeAyMWkFpCZUjuuCheNE9ktHFOVPctthl/0SnDaTGOxJ9vv0vffZfp+TXAqDQfDLcVdW116tdza6rze33mz3dt5emKzQjI9YJjN9lVDDpVB8hAIlv8o1p2ki+WVy97WpX865NiJTP7DMeZzSGyVmglG01KT3MzJFOqnuB1DW4zOvyQPAg9CHAzjzCu/eH8DcK+2Ofnx9CB9hKcjrv2AfIqEgSineMiqrb3UNVpc3B9i07/sLlTdfUvOWmvT6wTBYBLwEYQv6pI3zSe8hmmasSLlCJqkx4zDIMa6oRsEkr7tRYXhO2R294WMLFU25iauFQTV8sMwUZpm2SyEs2KeKiqbGlGliO5s5zPNaQ/6vNi5w9jmuhMoL5IotL5oVEjCDxm2YCs0ZytICyrSwbwV2SzVlaP+ka00In4/8ElwcDsOjYfD9uH/ypbWjQ/bIe+KRkHwiJ+SUnJMRYeS3s+JsOlvOH7fjbrs7y1bXaTW75J9w3z0CWeOtTQ==
where p and p’ are neighboring pixels
Ed(u, v)AAAB8HicbVBNSwMxEJ2tX7V+VT16CRahgpRdFfRYFMFjBfsh7VKy2bQNTbJLki2Upb/CiwdFvPpzvPlvTNs9aOuDgcd7M8zMC2LOtHHdbye3srq2vpHfLGxt7+zuFfcPGjpKFKF1EvFItQKsKWeS1g0znLZiRbEIOG0Gw9up3xxRpVkkH804pr7Afcl6jGBjpae7blhOztDotFssuRV3BrRMvIyUIEOtW/zqhBFJBJWGcKx123Nj46dYGUY4nRQ6iaYxJkPcp21LJRZU++ns4Ak6sUqIepGyJQ2aqb8nUiy0HovAdgpsBnrRm4r/ee3E9K79lMk4MVSS+aJewpGJ0PR7FDJFieFjSzBRzN6KyAArTIzNqGBD8BZfXiaN84p3UXEfLkvVmyyOPBzBMZTBgyuowj3UoA4EBDzDK7w5ynlx3p2PeWvOyWYO4Q+czx9F1Y9m
match cost Es(u, v)AAAB8HicbVBNSwMxEJ2tX7V+VT16CRahgpRdFfRYFMFjBfsh7VKyabYNTbJLki2Upb/CiwdFvPpzvPlvTNs9aOuDgcd7M8zMC2LOtHHdbye3srq2vpHfLGxt7+zuFfcPGjpKFKF1EvFItQKsKWeS1g0znLZiRbEIOG0Gw9up3xxRpVkkH804pr7AfclCRrCx0tNdV5eTMzQ67RZLbsWdAS0TLyMlyFDrFr86vYgkgkpDONa67bmx8VOsDCOcTgqdRNMYkyHu07alEguq/XR28ASdWKWHwkjZkgbN1N8TKRZaj0VgOwU2A73oTcX/vHZiwms/ZTJODJVkvihMODIRmn6PekxRYvjYEkwUs7ciMsAKE2MzKtgQvMWXl0njvOJdVNyHy1L1JosjD0dwDGXw4AqqcA81qAMBAc/wCm+Ocl6cd+dj3ppzsplD+APn8wdc6491
smoothness
• This is a probabilistic graphical model! • Can solve using least squares (as in Lucas-Kanade) • Or using discrete optimization (e.g. belief propagation)
-
30
Flow visualization Coarse-to-fine LK
Input two frames Horn-Schunck
Smoothness assumption
Source: Ce Liu
-
Flow networks
[Sun et al., “PWC-Net”, 2018]
Traditional coarse-to-fine flow PWC-net
-
Simple application: slow motion• Flow used in lots of familiar places!
• E.g., video compression, denoising, action recognition, …
• One application: use flow to estimate where pixel will be between frames
• Synthesize intermediate frames
(x, y) at time t-1
(x + u, y + v) at time t
-
34
Next lecture: light and color
top related