cpsc 425: computer vision
TRANSCRIPT
![Page 1: CPSC 425: Computer Vision](https://reader030.vdocument.in/reader030/viewer/2022012519/6193d6a735536a4eeb4f2bd9/html5/thumbnails/1.jpg)
Lecture 17: Optical Flow (cont.)
CPSC 425: Computer Vision
!1
![Page 2: CPSC 425: Computer Vision](https://reader030.vdocument.in/reader030/viewer/2022012519/6193d6a735536a4eeb4f2bd9/html5/thumbnails/2.jpg)
Menu for Today (March 11, 2020)Topics:
— Optical Flow (cont) — Classification
Redings: — Today’s Lecture: Forsyth & Ponce (2nd ed.) 15.1, 15.2
— Next Lecture: Forsyth & Ponce (2nd ed.) 16.1.3, 16.1.4, 16.1.9
Reminders: — Assignment 4: Local Invariant Features and RANSAC due Tuesday — Midterm graded. Grades will be released soon.
— Naive Bayes Classifier — Bayes' Risk
![Page 3: CPSC 425: Computer Vision](https://reader030.vdocument.in/reader030/viewer/2022012519/6193d6a735536a4eeb4f2bd9/html5/thumbnails/3.jpg)
!3
Today’s “fun” Example: Visual Microphone
Follow-up work to previous lecture’s example of Eulerian video magnification
![Page 4: CPSC 425: Computer Vision](https://reader030.vdocument.in/reader030/viewer/2022012519/6193d6a735536a4eeb4f2bd9/html5/thumbnails/4.jpg)
!3
Today’s “fun” Example: Visual Microphone
Follow-up work to previous lecture’s example of Eulerian video magnification
![Page 5: CPSC 425: Computer Vision](https://reader030.vdocument.in/reader030/viewer/2022012519/6193d6a735536a4eeb4f2bd9/html5/thumbnails/5.jpg)
!4
Lecture 16: Re-cap
Optical flow is the apparent motion of brightness patterns in the image
Applications — image and video stabilization in digital cameras, camcorders — motion-compensated video compression schemes such as MPEG — image registration for medical imaging, remote sensing — action recognition — motion segmentation
![Page 6: CPSC 425: Computer Vision](https://reader030.vdocument.in/reader030/viewer/2022012519/6193d6a735536a4eeb4f2bd9/html5/thumbnails/6.jpg)
!5Figure credit: M. Srinivasan
Lecture 16: Re-cap
![Page 7: CPSC 425: Computer Vision](https://reader030.vdocument.in/reader030/viewer/2022012519/6193d6a735536a4eeb4f2bd9/html5/thumbnails/7.jpg)
Aperture Problem
!6
In which direction is the line moving?
Image Credit: Ioannis (Yannis) Gkioulekas (CMU)
![Page 8: CPSC 425: Computer Vision](https://reader030.vdocument.in/reader030/viewer/2022012519/6193d6a735536a4eeb4f2bd9/html5/thumbnails/8.jpg)
Aperture Problem
!7
In which direction is the line moving?
Image Credit: Ioannis (Yannis) Gkioulekas (CMU)
![Page 9: CPSC 425: Computer Vision](https://reader030.vdocument.in/reader030/viewer/2022012519/6193d6a735536a4eeb4f2bd9/html5/thumbnails/9.jpg)
Aperture Problem
!8 Image Credit: Ioannis (Yannis) Gkioulekas (CMU)
![Page 10: CPSC 425: Computer Vision](https://reader030.vdocument.in/reader030/viewer/2022012519/6193d6a735536a4eeb4f2bd9/html5/thumbnails/10.jpg)
Aperture Problem
!9 Image Credit: Ioannis (Yannis) Gkioulekas (CMU)
![Page 11: CPSC 425: Computer Vision](https://reader030.vdocument.in/reader030/viewer/2022012519/6193d6a735536a4eeb4f2bd9/html5/thumbnails/11.jpg)
Aperture Problem
!10 Image Credit: Ioannis (Yannis) Gkioulekas (CMU)
![Page 12: CPSC 425: Computer Vision](https://reader030.vdocument.in/reader030/viewer/2022012519/6193d6a735536a4eeb4f2bd9/html5/thumbnails/12.jpg)
Aperture Problem
!11 Image Credit: Ioannis (Yannis) Gkioulekas (CMU)
![Page 13: CPSC 425: Computer Vision](https://reader030.vdocument.in/reader030/viewer/2022012519/6193d6a735536a4eeb4f2bd9/html5/thumbnails/13.jpg)
Aperture Problem
— Without distinct features to track, the true visual motion is ambiguous
— Locally, one can compute only the component of the visual motion in the direction perpendicular to the contour
!12
![Page 14: CPSC 425: Computer Vision](https://reader030.vdocument.in/reader030/viewer/2022012519/6193d6a735536a4eeb4f2bd9/html5/thumbnails/14.jpg)
Aperture Problem
— Without distinct features to track, the true visual motion is ambiguous
— Locally, one can compute only the component of the visual motion in the direction perpendicular to the contour
!13
![Page 15: CPSC 425: Computer Vision](https://reader030.vdocument.in/reader030/viewer/2022012519/6193d6a735536a4eeb4f2bd9/html5/thumbnails/15.jpg)
Visual Motion
Visual motion is determined when there are distinct features to track, provided: — the features can be detected and localized accurately; and — the features can be correctly matched over time
!14
![Page 16: CPSC 425: Computer Vision](https://reader030.vdocument.in/reader030/viewer/2022012519/6193d6a735536a4eeb4f2bd9/html5/thumbnails/16.jpg)
Motion as Matching
!15
![Page 17: CPSC 425: Computer Vision](https://reader030.vdocument.in/reader030/viewer/2022012519/6193d6a735536a4eeb4f2bd9/html5/thumbnails/17.jpg)
Consider image intensity also to be a function of time, . We write
Optical Flow Constraint Equation
!16
I(x, y, t)
t
![Page 18: CPSC 425: Computer Vision](https://reader030.vdocument.in/reader030/viewer/2022012519/6193d6a735536a4eeb4f2bd9/html5/thumbnails/18.jpg)
Consider image intensity also to be a function of time, . We write
Applying the chain rule for differentiation, we obtain
where subscripts denote partial differentiation
Optical Flow Constraint Equation
!17
I(x, y, t)
t
dI(x, y, t)
dt
= I
x
dx
dt
+ I
y
dy
dt
+ I
t
![Page 19: CPSC 425: Computer Vision](https://reader030.vdocument.in/reader030/viewer/2022012519/6193d6a735536a4eeb4f2bd9/html5/thumbnails/19.jpg)
Consider image intensity also to be a function of time, . We write
Applying the chain rule for differentiation, we obtain
where subscripts denote partial differentiation
Define . and . Then is the 2-D motion and the space of all
such and is the 2-D velocity space
Optical Flow Constraint Equation
!18
I(x, y, t)
t
dI(x, y, t)
dt
= I
x
dx
dt
+ I
y
dy
dt
+ I
t
v =dy
dtu =
dx
dt
[u, v]
u v
![Page 20: CPSC 425: Computer Vision](https://reader030.vdocument.in/reader030/viewer/2022012519/6193d6a735536a4eeb4f2bd9/html5/thumbnails/20.jpg)
Consider image intensity also to be a function of time, . We write
Applying the chain rule for differentiation, we obtain
where subscripts denote partial differentiation
Define . and . Then is the 2-D motion and the space of all
such and is the 2-D velocity space
Suppose . Then we obtain the (classic) optical flow constraint equation
Optical Flow Constraint Equation
!19
I(x, y, t)
t
dI(x, y, t)
dt
= I
x
dx
dt
+ I
y
dy
dt
+ I
t
v =dy
dtu =
dx
dt
dI(x, y, t)
dt
= 0
Ix
u+ Iy
v + It
= 0
[u, v]
u v
![Page 21: CPSC 425: Computer Vision](https://reader030.vdocument.in/reader030/viewer/2022012519/6193d6a735536a4eeb4f2bd9/html5/thumbnails/21.jpg)
Consider image intensity also to be a function of time, . We write
Applying the chain rule for differentiation, we obtain
where subscripts denote partial differentiation
Define . and . Then is the 2-D motion and the space of all
such and is the 2-D velocity space
Suppose . Then we obtain the (classic) optical flow constraint equation
Optical Flow Constraint Equation
!20
I(x, y, t)
t
dI(x, y, t)
dt
= I
x
dx
dt
+ I
y
dy
dt
+ I
t
v =dy
dtu =
dx
dt
dI(x, y, t)
dt
= 0
Ix
u+ Iy
v + It
= 0
[u, v]
u v
![Page 22: CPSC 425: Computer Vision](https://reader030.vdocument.in/reader030/viewer/2022012519/6193d6a735536a4eeb4f2bd9/html5/thumbnails/22.jpg)
Consider image intensity also to be a function of time, . We write
Applying the chain rule for differentiation, we obtain
where subscripts denote partial differentiation
Define . and . Then is the 2-D motion and the space of all
such and is the 2-D velocity space
Suppose . Then we obtain the (classic) optical flow constraint equation
Optical Flow Constraint Equation
!21
dI(x, y, t)
dt
= 0
Ix
u+ Iy
v + It
= 0
What does this mean, and why is it reasonable?
![Page 23: CPSC 425: Computer Vision](https://reader030.vdocument.in/reader030/viewer/2022012519/6193d6a735536a4eeb4f2bd9/html5/thumbnails/23.jpg)
Consider image intensity also to be a function of time, . We write
Applying the chain rule for differentiation, we obtain
where subscripts denote partial differentiation
Define . and . Then is the 2-D motion and the space of all
such and is the 2-D velocity space
Suppose . Then we obtain the (classic) optical flow constraint equation
Optical Flow Constraint Equation
!22
dI(x, y, t)
dt
= 0
Ix
u+ Iy
v + It
= 0
What does this mean, and why is it reasonable?
Scene point moving through image sequence
Image Credit: Ioannis (Yannis) Gkioulekas (CMU)
![Page 24: CPSC 425: Computer Vision](https://reader030.vdocument.in/reader030/viewer/2022012519/6193d6a735536a4eeb4f2bd9/html5/thumbnails/24.jpg)
Consider image intensity also to be a function of time, . We write
Applying the chain rule for differentiation, we obtain
where subscripts denote partial differentiation
Define . and . Then is the 2-D motion and the space of all
such and is the 2-D velocity space
Suppose . Then we obtain the (classic) optical flow constraint equation
Optical Flow Constraint Equation
!23
dI(x, y, t)
dt
= 0
Ix
u+ Iy
v + It
= 0
What does this mean, and why is it reasonable?
Scene point moving through image sequence
Image Credit: Ioannis (Yannis) Gkioulekas (CMU)
![Page 25: CPSC 425: Computer Vision](https://reader030.vdocument.in/reader030/viewer/2022012519/6193d6a735536a4eeb4f2bd9/html5/thumbnails/25.jpg)
Consider image intensity also to be a function of time, . We write
Applying the chain rule for differentiation, we obtain
where subscripts denote partial differentiation
Define . and . Then is the 2-D motion and the space of all
such and is the 2-D velocity space
Suppose . Then we obtain the (classic) optical flow constraint equation
Optical Flow Constraint Equation
!24
dI(x, y, t)
dt
= 0
Ix
u+ Iy
v + It
= 0
What does this mean, and why is it reasonable?
constant
Brightness Constancy Assumption: Brightness of the point remains the same
Image Credit: Ioannis (Yannis) Gkioulekas (CMU)
![Page 26: CPSC 425: Computer Vision](https://reader030.vdocument.in/reader030/viewer/2022012519/6193d6a735536a4eeb4f2bd9/html5/thumbnails/26.jpg)
Aside: Derivation of Optical Flow Constraint
!25
For small space-time step, brightness of a point is the same
Slide Credit: Ioannis (Yannis) Gkioulekas (CMU)
),( yx
),( tvytux δδ ++
ttime tttime δ+
),( yx
![Page 27: CPSC 425: Computer Vision](https://reader030.vdocument.in/reader030/viewer/2022012519/6193d6a735536a4eeb4f2bd9/html5/thumbnails/27.jpg)
Aside: Derivation of Optical Flow Constraint
!26
For small space-time step, brightness of a point is the same
Insight: If the time step is really small,
we can linearize the intensity function
Slide Credit: Ioannis (Yannis) Gkioulekas (CMU)
![Page 28: CPSC 425: Computer Vision](https://reader030.vdocument.in/reader030/viewer/2022012519/6193d6a735536a4eeb4f2bd9/html5/thumbnails/28.jpg)
Aside: Derivation of Optical Flow Constraint
!27
Multivariable Taylor Series Expansion (First order approximation, two variables)
Slide Credit: Ioannis (Yannis) Gkioulekas (CMU)
![Page 29: CPSC 425: Computer Vision](https://reader030.vdocument.in/reader030/viewer/2022012519/6193d6a735536a4eeb4f2bd9/html5/thumbnails/29.jpg)
Aside: Derivation of Optical Flow Constraint
!28
Multivariable Taylor Series Expansion (First order approximation, two variables)
assuming small motion
Slide Credit: Ioannis (Yannis) Gkioulekas (CMU)
![Page 30: CPSC 425: Computer Vision](https://reader030.vdocument.in/reader030/viewer/2022012519/6193d6a735536a4eeb4f2bd9/html5/thumbnails/30.jpg)
Aside: Derivation of Optical Flow Constraint
!29
Multivariable Taylor Series Expansion (First order approximation, two variables)
assuming small motionfixed point
partial derivative
cancel terms
Slide Credit: Ioannis (Yannis) Gkioulekas (CMU)
![Page 31: CPSC 425: Computer Vision](https://reader030.vdocument.in/reader030/viewer/2022012519/6193d6a735536a4eeb4f2bd9/html5/thumbnails/31.jpg)
Aside: Derivation of Optical Flow Constraint
!30
Multivariable Taylor Series Expansion (First order approximation, two variables)
assuming small motion
cancel terms
Slide Credit: Ioannis (Yannis) Gkioulekas (CMU)
![Page 32: CPSC 425: Computer Vision](https://reader030.vdocument.in/reader030/viewer/2022012519/6193d6a735536a4eeb4f2bd9/html5/thumbnails/32.jpg)
Aside: Derivation of Optical Flow Constraint
!31
Multivariable Taylor Series Expansion (First order approximation, two variables)
assuming small motion
divide by take limit
Slide Credit: Ioannis (Yannis) Gkioulekas (CMU)
![Page 33: CPSC 425: Computer Vision](https://reader030.vdocument.in/reader030/viewer/2022012519/6193d6a735536a4eeb4f2bd9/html5/thumbnails/33.jpg)
Aside: Derivation of Optical Flow Constraint
!32
Multivariable Taylor Series Expansion (First order approximation, two variables)
assuming small motion
divide by take limit
Brightness Constancy Equation
Slide Credit: Ioannis (Yannis) Gkioulekas (CMU)
![Page 34: CPSC 425: Computer Vision](https://reader030.vdocument.in/reader030/viewer/2022012519/6193d6a735536a4eeb4f2bd9/html5/thumbnails/34.jpg)
How do we compute …
!33 Slide Credit: Ioannis (Yannis) Gkioulekas (CMU)
![Page 35: CPSC 425: Computer Vision](https://reader030.vdocument.in/reader030/viewer/2022012519/6193d6a735536a4eeb4f2bd9/html5/thumbnails/35.jpg)
How do we compute …
!34
spatial derivative
Slide Credit: Ioannis (Yannis) Gkioulekas (CMU)
![Page 36: CPSC 425: Computer Vision](https://reader030.vdocument.in/reader030/viewer/2022012519/6193d6a735536a4eeb4f2bd9/html5/thumbnails/36.jpg)
How do we compute …
!35
spatial derivative
Forward difference Sobel filter Scharr filter
…
Slide Credit: Ioannis (Yannis) Gkioulekas (CMU)
![Page 37: CPSC 425: Computer Vision](https://reader030.vdocument.in/reader030/viewer/2022012519/6193d6a735536a4eeb4f2bd9/html5/thumbnails/37.jpg)
How do we compute …
!36
spatial derivative
Forward difference Sobel filter Scharr filter
…
temporal derivative
Slide Credit: Ioannis (Yannis) Gkioulekas (CMU)
![Page 38: CPSC 425: Computer Vision](https://reader030.vdocument.in/reader030/viewer/2022012519/6193d6a735536a4eeb4f2bd9/html5/thumbnails/38.jpg)
How do we compute …
!37
spatial derivative
Forward difference Sobel filter Scharr filter
…
temporal derivative
Frame differencing
Slide Credit: Ioannis (Yannis) Gkioulekas (CMU)
![Page 39: CPSC 425: Computer Vision](https://reader030.vdocument.in/reader030/viewer/2022012519/6193d6a735536a4eeb4f2bd9/html5/thumbnails/39.jpg)
Frame Differencing: Example
!38
1 1 1 1 11 1 1 1 11 10 10 10 101 10 10 10 101 10 10 10 101 10 10 10 10
1 1 1 1 11 1 1 1 11 1 1 1 11 1 10 10 101 1 10 10 101 1 10 10 10
0 0 0 0 00 0 0 0 00 -9 -9 -9 -90 -9 0 0 00 -9 0 0 00 -9 0 0 0
- =
(example of a forward temporal difference)
Slide Credit: Ioannis (Yannis) Gkioulekas (CMU)
![Page 40: CPSC 425: Computer Vision](https://reader030.vdocument.in/reader030/viewer/2022012519/6193d6a735536a4eeb4f2bd9/html5/thumbnails/40.jpg)
!39
1 1 1 1 11 1 1 1 11 10 10 10 101 10 10 10 101 10 10 10 101 10 10 10 10
1 1 1 1 11 1 1 1 11 1 1 1 11 1 10 10 101 1 10 10 101 1 10 10 10
- 0 0 0 -- 0 0 0 -- 9 0 0 -- 9 0 0 -- 9 0 0 -- 9 0 0 -
- - - - -0 0 0 0 00 9 9 9 90 0 0 0 00 0 0 0 0- - - - -
0 0 0 0 00 0 0 0 00 -9 -9 -9 -90 -9 0 0 00 -9 0 0 00 -9 0 0 0
y
x
-1 0 1
-1 0 1
y
x
y
x
y
x
y
x
Slide Credit: Ioannis (Yannis) Gkioulekas (CMU)
![Page 41: CPSC 425: Computer Vision](https://reader030.vdocument.in/reader030/viewer/2022012519/6193d6a735536a4eeb4f2bd9/html5/thumbnails/41.jpg)
How do we compute …
!40
spatial derivative optical flow
Forward difference Sobel filter Scharr filter
…
temporal derivative
Frame differencingHow do you compute this?
Slide Credit: Ioannis (Yannis) Gkioulekas (CMU)
![Page 42: CPSC 425: Computer Vision](https://reader030.vdocument.in/reader030/viewer/2022012519/6193d6a735536a4eeb4f2bd9/html5/thumbnails/42.jpg)
How do we compute …
!41
spatial derivative optical flow
Forward difference Sobel filter Scharr filter
…
temporal derivative
Frame differencingWe need to solve for this! (this is the unknown in the
optical flow problem)
Slide Credit: Ioannis (Yannis) Gkioulekas (CMU)
![Page 43: CPSC 425: Computer Vision](https://reader030.vdocument.in/reader030/viewer/2022012519/6193d6a735536a4eeb4f2bd9/html5/thumbnails/43.jpg)
How do we compute …
!42
spatial derivative optical flow
Forward difference Sobel filter Scharr filter
…
temporal derivative
Frame differencingSolution lies on a line
Cannot be found uniquely with a single constraint
Slide Credit: Ioannis (Yannis) Gkioulekas (CMU)
![Page 44: CPSC 425: Computer Vision](https://reader030.vdocument.in/reader030/viewer/2022012519/6193d6a735536a4eeb4f2bd9/html5/thumbnails/44.jpg)
!43
Equation determines a straight line in velocity space
many combinations of u and v will satisfy the equality
Optical Flow Constraint Equation
Slide Credit: Ioannis (Yannis) Gkioulekas (CMU)
![Page 45: CPSC 425: Computer Vision](https://reader030.vdocument.in/reader030/viewer/2022012519/6193d6a735536a4eeb4f2bd9/html5/thumbnails/45.jpg)
Lucas-Kanade
Observations: 1. The 2-D motion, , at a given point, , has two degrees-of-freedom 2. The partial derivatives, , provide one constraint 3. The 2-D motion, , cannot be determined locally from alone
!44
[u, v] [x, y]
Ix
, Iy
, It
[u, v] Ix
, Iy
, It
![Page 46: CPSC 425: Computer Vision](https://reader030.vdocument.in/reader030/viewer/2022012519/6193d6a735536a4eeb4f2bd9/html5/thumbnails/46.jpg)
Lucas-Kanade
Observations: 1. The 2-D motion, , at a given point, , has two degrees-of-freedom 2. The partial derivatives, , provide one constraint 3. The 2-D motion, , cannot be determined locally from alone
Lucas–Kanade Idea: Obtain additional local constraint by computing the partial derivatives, , in a window centered at the given
!45
[u, v] [x, y]
Ix
, Iy
, It
[u, v] Ix
, Iy
, It
Ix
, Iy
, It
[x, y]
![Page 47: CPSC 425: Computer Vision](https://reader030.vdocument.in/reader030/viewer/2022012519/6193d6a735536a4eeb4f2bd9/html5/thumbnails/47.jpg)
Lucas-Kanade
Observations: 1. The 2-D motion, , at a given point, , has two degrees-of-freedom 2. The partial derivatives, , provide one constraint 3. The 2-D motion, , cannot be determined locally from alone
Lucas–Kanade Idea: Obtain additional local constraint by computing the partial derivatives, , in a window centered at the given
Constant Flow Assumption: nearby pixels will likely have same optical flow
!46
[u, v] [x, y]
Ix
, Iy
, It
[u, v] Ix
, Iy
, It
Ix
, Iy
, It
[x, y]
![Page 48: CPSC 425: Computer Vision](https://reader030.vdocument.in/reader030/viewer/2022012519/6193d6a735536a4eeb4f2bd9/html5/thumbnails/48.jpg)
Suppose is the (original) center point in the window. Let be any other point in the window. This gives us two equations that we can write
and that can be solved locally for and as
provided that and are the same in both equations and provided that the required matrix inverse exists.
!47
[x1, y1] = [x, y] [x2, y2]
x
I
x1u+ I
y1v = �I
t1
I
x2u+ I
y2v = �I
t2
u v
uv
�= �
Ix1 I
y1
Ix2 I
y2
��1 It1
It2
�
u v
Lucas-Kanade Ix
u+ Iy
v + It
= 0Optical Flow Constraint Equation:
![Page 49: CPSC 425: Computer Vision](https://reader030.vdocument.in/reader030/viewer/2022012519/6193d6a735536a4eeb4f2bd9/html5/thumbnails/49.jpg)
Considering all n points in the window, one obtains
which can be written as the matrix equation
where , and
!48
Ix1u+ I
y1v = �It1
Ix2u+ I
y2v = �It2
...Ixnu+ I
ynv = �Itn
A =
2
6664
Ix1 I
y1
Ix2 I
y2
......
Ixn I
yn
3
7775
b = �
2
6664
It1
It2
...Itn
3
7775
Av = b
v = [u, v]T
Lucas-Kanade
A =
2
6664
Ix1 I
y1
Ix2 I
y2
......
Ixn I
yn
3
7775
b = �
2
6664
It1
It2
...Itn
3
7775
Ix
u+ Iy
v + It
= 0Optical Flow Constraint Equation:
![Page 50: CPSC 425: Computer Vision](https://reader030.vdocument.in/reader030/viewer/2022012519/6193d6a735536a4eeb4f2bd9/html5/thumbnails/50.jpg)
The standard least squares solution, , to is
again provided that and are the same in all equations and provided that the rank of is 2 (so that the required inverse exists)
!49
v̄ = (ATA)�1ATb
ATA
v̄ = (ATA)�1ATb
u v
Lucas-Kanade
![Page 51: CPSC 425: Computer Vision](https://reader030.vdocument.in/reader030/viewer/2022012519/6193d6a735536a4eeb4f2bd9/html5/thumbnails/51.jpg)
The standard least squares solution, , to is
again provided that and are the same in all equations and provided that the rank of is 2 (so that the required inverse exists)
!50
v̄ = (ATA)�1ATb
ATA
v̄ = (ATA)�1ATb
u v
Lucas-Kanade
A =
2
6664
Ix1 I
y1
Ix2 I
y2
......
Ixn I
yn
3
7775
b = �
2
6664
It1
It2
...Itn
3
7775
A=
2 6 6 6 4
I x1
I y1
I x2
I y2
. . .. . .
I xn
I yn
3 7 7 7 5
b=
�
2 6 6 6 4
I t1
I t2 . . . I tn
3 7 7 7 5
![Page 52: CPSC 425: Computer Vision](https://reader030.vdocument.in/reader030/viewer/2022012519/6193d6a735536a4eeb4f2bd9/html5/thumbnails/52.jpg)
Note that we can explicitly write down an expression for as
which is identical to the matrix that we saw in the context of Harris corner detection
!51
ATA =
PI2x
PIx
IyP
Ix
Iy
I2y
�
ATA
C
Lucas-Kanade
![Page 53: CPSC 425: Computer Vision](https://reader030.vdocument.in/reader030/viewer/2022012519/6193d6a735536a4eeb4f2bd9/html5/thumbnails/53.jpg)
Note that we can explicitly write down an expression for as
which is identical to the matrix that we saw in the context of Harris corner detection
!52
ATA =
PI2x
PIx
IyP
Ix
Iy
I2y
�
ATA
C
Lucas-Kanade
What does that mean?
![Page 54: CPSC 425: Computer Vision](https://reader030.vdocument.in/reader030/viewer/2022012519/6193d6a735536a4eeb4f2bd9/html5/thumbnails/54.jpg)
A dense method to compute motion, , at every location in an image
Key Assumptions:
1. Motion is slow enough and smooth enough that differential methods apply (i.e., that the partial derivatives, , are well-defined)
2. The optical flow constraint equation holds (i.e., )
3. A window size is chosen so that motion, , is constant in the window
4. A window size is chosen so that the rank of is 2 for the window
!53
Lucas-Kanade Summary
[u, v]
Ix
, Iy
, It
[u, v]
ATA
dI(x, y, t)
dt
= 0
![Page 55: CPSC 425: Computer Vision](https://reader030.vdocument.in/reader030/viewer/2022012519/6193d6a735536a4eeb4f2bd9/html5/thumbnails/55.jpg)
Aside: Optical Flow Smoothness Constraint
Many methods trade off a ‘departure from the optical flow constraint’ cost with a ‘departure from smoothness’ cost.
The optimization objective to minimize becomes
where is a weighing parameter.
!54
E =
Z Z(I
x
u+ Iy
v + It
)2 + �(||5 u||2 + ||5 v||2)
�
![Page 56: CPSC 425: Computer Vision](https://reader030.vdocument.in/reader030/viewer/2022012519/6193d6a735536a4eeb4f2bd9/html5/thumbnails/56.jpg)
!55
smoothness brightness constancy
weight
Horn-Schunck Optical Flow
Slide Credit: Ioannis (Yannis) Gkioulekas (CMU)
![Page 57: CPSC 425: Computer Vision](https://reader030.vdocument.in/reader030/viewer/2022012519/6193d6a735536a4eeb4f2bd9/html5/thumbnails/57.jpg)
Horn-Schunck Optical Flow
!56
Brightness constancy
Smoothness
Slide Credit: Ioannis (Yannis) Gkioulekas (CMU)
![Page 58: CPSC 425: Computer Vision](https://reader030.vdocument.in/reader030/viewer/2022012519/6193d6a735536a4eeb4f2bd9/html5/thumbnails/58.jpg)
SummaryMotion, like binocular stereo, can be formulated as a matching problem. That is, given a scene point located at in an image acquired at time , what is its position, , in an image acquired at time ?
Assuming image intensity does not change as a consequence of motion, we obtain the (classic) optical flow constraint equation
where , is the 2-D motion at a given point, , and are the partial derivatives of intensity with respect to , , and
Lucas–Kanade is a dense method to compute the motion, , at every location in an image
!57
(x0, y0) t0(x1, y1) t1
Ix
u+ Iy
v + It
= 0
[u, v] [x, y] Ix
, Iy
, It
x y t
[u, v]