lecture: motion - stanford university
TRANSCRIPT
![Page 1: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/1.jpg)
Lecture 17 -Stanford University
Lecture:Motion
JuanCarlosNiebles andRanjayKrishnaStanfordVisionandLearningLab
28-Nov-171
![Page 2: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/2.jpg)
Lecture 17 -Stanford University
Whatwewilllearntoday?
• Opticalflow• Lucas-Kanade method• Horn-Schunk method• Pyramidsforlargemotion• Commonfate• Applications
28-Nov-172
Reading:[Szeliski]Chapters:8.4,8.5[Fleet&Weiss,2005]http://www.cs.toronto.edu/pub/jepson/teaching/vision/2503/opticalFlow.pdf
![Page 3: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/3.jpg)
Lecture 17 -Stanford University
Whatwewilllearntoday?
• Opticalflow• Horn-Schunk method• Lucas-Kanade method• Pyramidsforlargemotion• Commonfate• Applications
28-Nov-173
Reading:[Szeliski]Chapters:8.4,8.5[Fleet&Weiss,2005]http://www.cs.toronto.edu/pub/jepson/teaching/vision/2503/opticalFlow.pdf
![Page 4: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/4.jpg)
Lecture 17 -Stanford University
Fromimagestovideos• Avideoisasequenceofframescapturedovertime• Nowourimagedataisafunctionofspace(x,y)andtime(t)
28-Nov-174
![Page 5: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/5.jpg)
Lecture 17 -Stanford University
Whyismotionuseful?
28-Nov-175
![Page 6: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/6.jpg)
Lecture 17 -Stanford University
Whyismotionuseful?
28-Nov-176
![Page 7: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/7.jpg)
Lecture 17 -Stanford University
Opticalflow• Definition:opticalflowistheapparentmotionofbrightnesspatternsintheimage
• Note:apparentmotioncanbecausedbylightingchangeswithoutanyactualmotion– Thinkofauniformrotatingsphereunderfixedlightingvs.astationarysphereundermovingillumination
GOAL: Recoverimagemotionateachpixelfromopticalflow
Source:SilvioSavarese
28-Nov-177
![Page 8: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/8.jpg)
Lecture 17 -Stanford University
PicturecourtesyofSelim Temizer - LearningandIntelligentSystems(LIS)Group,MIT
Opticalflow
Vectorfieldfunctionofthespatio-temporalimagebrightnessvariations
28-Nov-178
![Page 9: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/9.jpg)
Lecture 17 -Stanford University
Estimatingopticalflow
• Giventwosubsequentframes,estimatetheapparentmotionfieldu(x,y),v(x,y)betweenthem
• Keyassumptions• Brightnessconstancy:projectionofthesamepointlooksthesamein
everyframe• Smallmotion: pointsdonotmoveveryfar• Spatialcoherence: pointsmoveliketheirneighbors
I(x,y,t–1) I(x,y,t)
Source:SilvioSavarese
28-Nov-179
![Page 10: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/10.jpg)
Lecture 17 -Stanford University 10
KeyAssumptions:smallmotions
*SlidefromMichaelBlack,CS143 2003
![Page 11: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/11.jpg)
Lecture 17 -Stanford University 11
KeyAssumptions:spatialcoherence
*SlidefromMichaelBlack,CS143 2003
![Page 12: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/12.jpg)
Lecture 17 -Stanford University 12
KeyAssumptions:brightnessConstancy
*SlidefromMichaelBlack,CS143 2003
![Page 13: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/13.jpg)
Lecture 17 -Stanford University
I(x +u, y+ v, t) ≈ I(x, y, t −1)+ Ix ⋅u(x, y)+ Iy ⋅ v(x, y)+ It
• BrightnessConstancyEquation:I(x, y, t −1) = I(x +u(x, y), y+ v(x, y), t)
LinearizingtherightsideusingTaylorexpansion:
I(x,y,t–1) I(x,y,t)
0»+×+× tyx IvIuIHence,
Imagederivativealongx
→∇I ⋅ u v[ ]T + It = 0I(x +u, y+ v, t)− I(x, y, t −1) = Ix ⋅u(x, y)+ Iy ⋅ v(x, y)+ It
Source:SilvioSavarese
28-Nov-1713
Thebrightnessconstancyconstraint
![Page 14: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/14.jpg)
Lecture 17 -Stanford University
Filtersusedtofindthederivatives
28-Nov-1714
𝐼" 𝐼# 𝐼$
![Page 15: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/15.jpg)
Lecture 17 -Stanford University
• Howmanyequationsandunknownsperpixel?
Thecomponentoftheflowperpendiculartothegradient(i.e.,paralleltotheedge)cannotbemeasured
edge
(u,v)
(u’,v’)
gradient
(u+u’,v+v’)
If(u,v)satisfiestheequation,sodoes(u+u’,v+v’)if
•Oneequation(thisisascalarequation!),twounknowns(u,v)
∇I ⋅ u ' v '[ ]T = 0
Canweusethisequationtorecoverimagemotion(u,v)ateachpixel?
Source:SilvioSavarese
28-Nov-1715
Thebrightnessconstancyconstraint
∇I ⋅ u v[ ]T + It = 0
![Page 16: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/16.jpg)
Lecture 17 -Stanford University
Theapertureproblem
Actualmotion
Source:SilvioSavarese
28-Nov-1716
![Page 17: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/17.jpg)
Lecture 17 -Stanford University
Theapertureproblem
Perceivedmotion Source:SilvioSavarese
28-Nov-1717
![Page 18: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/18.jpg)
Lecture 17 -Stanford University
Thebarberpoleillusion
http://en.wikipedia.org/wiki/Barberpole_illusion Source:SilvioSavarese
28-Nov-1718
![Page 19: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/19.jpg)
Lecture 17 -Stanford University
Thebarberpoleillusion
http://en.wikipedia.org/wiki/Barberpole_illusion Source:SilvioSavarese
28-Nov-1719
![Page 20: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/20.jpg)
Lecture 17 -Stanford University
Whatwewilllearntoday?
• Opticalflow• Lucas-Kanade method• Horn-Schunk method• Pyramidsforlargemotion• Commonfate• Applications
28-Nov-1720
Reading:[Szeliski]Chapters:8.4,8.5[Fleet&Weiss,2005]http://www.cs.toronto.edu/pub/jepson/teaching/vision/2503/opticalFlow.pdf
![Page 21: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/21.jpg)
Lecture 17 -Stanford University
Solvingtheambiguity…
• Howtogetmoreequationsforapixel?• Spatialcoherenceconstraint:• Assumethepixel’sneighborshavethesame(u,v)
– Ifweusea5x5window,thatgivesus25equationsperpixel
B.LucasandT.Kanade.Aniterativeimageregistrationtechniquewithanapplicationtostereovision.InProceedingsoftheInternationalJointConferenceonArtificialIntelligence,pp.674–679,1981.
Source:SilvioSavarese
28-Nov-1721
![Page 22: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/22.jpg)
Lecture 17 -Stanford University
• Overconstrained linearsystem:
Lucas-Kanade flow
Source:SilvioSavarese
28-Nov-1722
![Page 23: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/23.jpg)
Lecture 17 -Stanford University
• Overconstrained linearsystem
ThesummationsareoverallpixelsintheKxKwindow
Leastsquaressolutionford givenby
Source:SilvioSavarese
28-Nov-1723
Lucas-Kanade flow
![Page 24: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/24.jpg)
Lecture 17 -Stanford University
Conditionsforsolvability– Optimal(u,v)satisfiesLucas-Kanadeequation
Doesthisremindanythingtoyou?
WhenisThisSolvable?• ATA shouldbeinvertible• ATA shouldnotbetoosmallduetonoise
– eigenvaluesl1 andl 2 ofATA shouldnotbetoosmall• ATA shouldbewell-conditioned
– l 1/l 2 shouldnotbetoolarge(l 1 =largereigenvalue)
Source:SilvioSavarese
28-Nov-1724
![Page 25: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/25.jpg)
Lecture 17 -Stanford University
• EigenvectorsandeigenvaluesofATArelatetoedgedirectionandmagnitude• Theeigenvectorassociatedwiththelargereigenvaluepointsin
thedirectionoffastestintensitychange• Theothereigenvectorisorthogonaltoit
M=ATAisthesecondmomentmatrix!(Harriscornerdetector…)
Source:SilvioSavarese
28-Nov-1725
![Page 26: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/26.jpg)
Lecture 17 -Stanford University
Interpretingtheeigenvalues
l1
l2
“Corner”l1 and l2 are large,l1 ~ l2
l1 and l2 are small “Edge” l1 >> l2
“Edge” l2 >> l1
“Flat” region
Classificationofimagepointsusingeigenvaluesofthesecondmomentmatrix:
Source:SilvioSavarese
28-Nov-1726
![Page 27: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/27.jpg)
Lecture 17 -Stanford University
Edge
– gradientsverylargeorverysmall– large l1,smalll2
Source:SilvioSavarese
28-Nov-1727
![Page 28: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/28.jpg)
Lecture 17 -Stanford University
Low-textureregion
– gradientshavesmallmagnitude– small l1,smalll2
Source:SilvioSavarese
28-Nov-1728
![Page 29: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/29.jpg)
Lecture 17 -Stanford University
High-textureregion
– gradientsaredifferent,largemagnitudes– large l1,largel2
Source:SilvioSavarese
28-Nov-1729
![Page 30: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/30.jpg)
Lecture 17 -Stanford University 30
ErrorsinLukas-Kanade
Whatarethepotentialcausesoferrorsinthisprocedure?– SupposeATAiseasilyinvertible– Supposethereisnotmuchnoiseintheimage
• When our assumptions are violated– Brightness constancy is not satisfied– The motion is not small– A point does not move like its neighbors
• window size is too large• what is the ideal window size?
*FromKhurram Hassan-Shafique CAP5415ComputerVision2003
![Page 31: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/31.jpg)
Lecture 17 -Stanford University 31
– Can solve using Newton’s method (out of scope for this class)
– Lukas-Kanade method does one iteration of Newton’s method• Better results are obtained via more iterations
Improvingaccuracy• Recalloursmallmotionassumption
• This is not exact– To do better, we need to add higher order terms back in:
• This is a polynomial root finding problem
It-1(x,y)
It-1(x,y)
It-1(x,y)
*FromKhurramHassan-ShafiqueCAP5415ComputerVision2003
![Page 32: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/32.jpg)
Lecture 17 -Stanford University 32
IterativeRefinement• Iterative Lukas-Kanade Algorithm
1. Estimate velocity at each pixel by solving Lucas-Kanade equations
2. Warp I(t-1) towards I(t) using the estimated flow field- use image warping techniques
3. Repeat until convergence
*FromKhurramHassan-ShafiqueCAP5415ComputerVision2003
![Page 33: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/33.jpg)
Lecture 17 -Stanford University
Whendotheopticalflowassumptionsfail?
28-Nov-1733
Inotherwords,inwhatsituationsdoesthedisplacementofpixelpatchesnotrepresentphysicalmovementofpointsinspace?
1. Well, TV is based on illusory motion – the set is stationary yet things seem to move
2. A uniform rotating sphere – nothing seems to move, yet it is rotating
3. Changing directions or intensities of lighting can make things seem to move – for example, if the specular highlight on a rotating sphere moves.
4. Muscle movement can make some spots on a cheetah move opposite direction of motion. – And infinitely more break downs of optical flow.
![Page 34: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/34.jpg)
Lecture 17 -Stanford University
Whatwewilllearntoday?
• Opticalflow• Lucas-Kanade method• Horn-Schunk method• Pyramidsforlargemotion• Commonfate• Applications
28-Nov-1734
Reading:[Szeliski]Chapters:8.4,8.5[Fleet&Weiss,2005]http://www.cs.toronto.edu/pub/jepson/teaching/vision/2503/opticalFlow.pdf
![Page 35: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/35.jpg)
Lecture 17 -Stanford University
Horn-Schunk methodforopticalflow
• Theflowisformulatedasaglobalenergy function whichisshouldbeminimized:
28-Nov-1735
![Page 36: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/36.jpg)
Lecture 17 -Stanford University
Horn-Schunk methodforopticalflow
• Theflowisformulatedasaglobalenergy function whichisshouldbeminimized:
• Thefirstpartofthefunctionisthebrightnessconsistency.
28-Nov-1736
![Page 37: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/37.jpg)
Lecture 17 -Stanford University
Horn-Schunk methodforopticalflow
• Theflowisformulatedasaglobalenergy function whichisshouldbeminimized:
• Thesecondpartisthesmoothnessconstraint.It’stryingtomakesurethatthechangesbetweenframesaresmall.
28-Nov-1737
![Page 38: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/38.jpg)
Lecture 17 -Stanford University
Horn-Schunk methodforopticalflow
• Theflowisformulatedasaglobalenergy function whichisshouldbeminimized:
• 𝛼 isaregularizationconstant.Largervaluesof𝛼 leadtosmootherflow.
28-Nov-1738
![Page 39: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/39.jpg)
Lecture 17 -Stanford University
Horn-Schunk methodforopticalflow
• Theflowisformulatedasaglobalenergy function whichisshouldbeminimized:
• Bytakingthederivativewithrespecttouandv,wegetthefollowing2equations:
28-Nov-1739
![Page 40: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/40.jpg)
Lecture 17 -Stanford University
Horn-Schunk methodforopticalflow
• Bytakingthederivativewithrespecttouandv,wegetthefollowing2equations:
• WhereiscalledtheLagrangeoperator.Inpractice,itismeasuredusing:
• whereistheweightedaverageofumeasuredat(x,y).
28-Nov-1740
![Page 41: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/41.jpg)
Lecture 17 -Stanford University
Horn-Schunk methodforopticalflow
• Nowwesubstitutein:
• Toget:
• Whichislinearinuandvandcanbesolvedforeachpixelindividually.
28-Nov-1741
![Page 42: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/42.jpg)
Lecture 17 -Stanford University
IterativeHorn-Schunk
• Butsincethesolutiondependsontheneighboringvaluesoftheflowfield,itmustberepeatedoncetheneighborshavebeenupdated.
• Soinstead,wecaniterativelysolveforuandvusing:
28-Nov-1742
![Page 43: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/43.jpg)
Lecture 17 -Stanford University 43
Whatdoesthesmoothnessregularizationdoanyway?
• It’sasumofsquaredterms(aEuclidiandistancemeasure).• We’reputtingitintheexpressiontobeminimized.• =>Intexturefreeregions,thereisnoopticalflow
Regularizedflow
Opticalflow
• =>Onedges,pointswillflowtonearestpoints,solvingtheapertureproblem.
Slidecredit:SebastianThurn
![Page 44: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/44.jpg)
Lecture 17 -Stanford University
DenseOpticalFlowwithMichaelBlack’smethod
• MichaelBlacktookHorn-Schunk’s methodonestepfurther,startingfromtheregularizationconstant:
• Whichlookslikeaquadratic:
• Andreplaceditwiththis:
• Whydoesthisregularizationworkbetter?
28-Nov-1744
![Page 45: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/45.jpg)
Lecture 17 -Stanford University
Whatwewilllearntoday?
• Opticalflow• Lucas-Kanade method• Horn-Schunk method• Pyramidsforlargemotion• Commonfate• Applications
28-Nov-1745
![Page 46: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/46.jpg)
Lecture 17 -Stanford University
• Keyassumptions(ErrorsinLucas-Kanade)
• Smallmotion: pointsdonotmoveveryfar
• Brightnessconstancy:projectionofthesamepointlooksthesameineveryframe
• Spatialcoherence: pointsmoveliketheirneighbors
Recap
Source:SilvioSavarese
28-Nov-1746
![Page 47: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/47.jpg)
Lecture 17 -Stanford University
Revisitingthesmallmotionassumption
• Isthismotionsmallenough?– Probablynot—it’smuchlargerthanonepixel(2nd ordertermsdominate)– Howmightwesolvethisproblem?
* Fro
m K
hurra
mH
assa
n-Sh
afiq
ueC
AP54
15 C
ompu
ter V
isio
n 20
03
28-Nov-1747
![Page 48: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/48.jpg)
Lecture 17 -Stanford University
Reducetheresolution!
* Fro
m K
hurra
mH
assa
n-Sh
afiq
ueC
AP54
15 C
ompu
ter V
isio
n 20
03
28-Nov-1748
![Page 49: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/49.jpg)
Lecture 17 -Stanford University
Source:SilvioSavarese
image Iimage H
Gaussianpyramidofimage1 Gaussianpyramidofimage2
image2image1 u=10 pixels
u=5 pixels
u=2.5 pixels
u=1.25 pixels
Coarse-to-fineopticalflowestimation
28-Nov-1749
![Page 50: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/50.jpg)
Lecture 17 -Stanford University
image Iimage J
Gaussianpyramidofimage1(t) Gaussianpyramidofimage2(t+1)
image2image1
Coarse-to-fineopticalflowestimation
runiterativeL-K
runiterativeL-K
warp&upsample
.
.
.
Source:SilvioSavarese
28-Nov-1750
![Page 51: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/51.jpg)
Lecture 17 -Stanford University
OpticalFlowResults
* Fro
m K
hurra
mH
assa
n-Sh
afiq
ueC
AP54
15 C
ompu
ter V
isio
n 20
03
28-Nov-1751
![Page 52: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/52.jpg)
Lecture 17 -Stanford University
OpticalFlowResults
* Fro
m K
hurra
mH
assa
n-Sh
afiq
ueC
AP54
15 C
ompu
ter V
isio
n 20
03
• http://www.ces.clemson.edu/~stb/klt/• OpenCV
28-Nov-1752
![Page 53: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/53.jpg)
Lecture 17 -Stanford University
Whatwewilllearntoday?
• Opticalflow• Lucas-Kanade method• Horn-Schunk method• Pyramidsforlargemotion• Commonfate• Applications
28-Nov-1753
![Page 54: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/54.jpg)
Lecture 17 -Stanford University
• Keyassumptions(ErrorsinLucas-Kanade)
• Smallmotion: pointsdonotmoveveryfar
• Brightnessconstancy:projectionofthesamepointlooksthesameineveryframe
• Spatialcoherence: pointsmoveliketheirneighbors
Recap
Source:SilvioSavarese
28-Nov-1754
![Page 55: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/55.jpg)
Lecture 17 -Stanford University
Reminder:Gestalt– commonfate
28-Nov-1755
![Page 56: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/56.jpg)
Lecture 17 -Stanford University
Motionsegmentation• Howdowerepresentthemotioninthisscene?
Source:SilvioSavarese
28-Nov-1756
![Page 57: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/57.jpg)
Lecture 17 -Stanford University
• Breakimagesequenceinto“layers”eachofwhichhasacoherent(affine)motion
MotionsegmentationJ.WangandE.Adelson.LayeredRepresentationforMotionAnalysis.CVPR1993.
Source:SilvioSavarese
28-Nov-1757
![Page 58: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/58.jpg)
Lecture 17 -Stanford University
• Substitutingintothebrightnessconstancyequation:
yaxaayxvyaxaayxu
654
321
),(),(
++=++=
0»+×+× tyx IvIuI
Affinemotion
Source:SilvioSavarese
28-Nov-1758
![Page 59: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/59.jpg)
Lecture 17 -Stanford University
0)()( 654321 »++++++ tyx IyaxaaIyaxaaI
• Substitutingintothebrightnessconstancyequation:
yaxaayxvyaxaayxu
654
321
),(),(
++=++=
• Eachpixelprovides1linearconstraintin6unknowns
[ ] 2å ++++++= tyx IyaxaaIyaxaaIaErr )()()( 654321!
• Leastsquaresminimization:
Affinemotion
Source:SilvioSavarese
28-Nov-1759
![Page 60: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/60.jpg)
Lecture 17 -Stanford University
Howdoweestimatethelayers?• 1.Obtainasetofinitialaffinemotionhypotheses
– Dividetheimageintoblocksandestimateaffinemotionparametersineachblockbyleastsquares• Eliminatehypotheseswithhighresidualerror
• Mapintomotionparameterspace• Performk-meansclusteringonaffinemotionparameters
–Mergeclustersthatarecloseandretainthelargestclusterstoobtainasmallersetofhypothesestodescribeallthemotionsinthescene
Source:SilvioSavarese
28-Nov-1760
![Page 61: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/61.jpg)
Lecture 17 -Stanford University
Howdoweestimatethelayers?• 1.Obtainasetofinitialaffinemotionhypotheses
– Dividetheimageintoblocksandestimateaffinemotionparametersineachblockbyleastsquares• Eliminatehypotheseswithhighresidualerror
• Mapintomotionparameterspace• Performk-meansclusteringonaffinemotionparameters
–Mergeclustersthatarecloseandretainthelargestclusterstoobtainasmallersetofhypothesestodescribeallthemotionsinthescene
Source:SilvioSavarese
28-Nov-1761
![Page 62: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/62.jpg)
Lecture 17 -Stanford University
Howdoweestimatethelayers?• 1.Obtainasetofinitialaffinemotionhypotheses
– Dividetheimageintoblocksandestimateaffinemotionparametersineachblockbyleastsquares• Eliminatehypotheseswithhighresidualerror
• Mapintomotionparameterspace• Performk-meansclusteringonaffinemotionparameters
–Mergeclustersthatarecloseandretainthelargestclusterstoobtainasmallersetofhypothesestodescribeallthemotionsinthescene
2.Iterateuntilconvergence:•Assigneachpixeltobesthypothesis
–Pixelswithhighresidualerrorremainunassigned•Performregionfilteringtoenforcespatialconstraints•Re-estimateaffinemotionsineachregion
Source:SilvioSavarese
28-Nov-1762
![Page 63: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/63.jpg)
Lecture 17 -Stanford University
Exampleresult
J. Wang and E. Adelson. Layered Representation for Motion Analysis. CVPR 1993. Source:SilvioSavarese
28-Nov-1763
![Page 64: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/64.jpg)
Lecture 17 -Stanford University
Whatwewilllearntoday?
• Opticalflow• Lucas-Kanade method• Horn-Schunk method• Pyramidsforlargemotion• Commonfate• Applications
28-Nov-1764
![Page 65: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/65.jpg)
Lecture 17 -Stanford University
Usesofmotion
• Trackingfeatures• Segmentingobjectsbasedonmotioncues• Learningdynamicalmodels• Improvingvideoquality
– Motionstabilization– Superresolution
• Trackingobjects• Recognizingeventsandactivities
28-Nov-1765
![Page 66: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/66.jpg)
Lecture 17 -Stanford University
Estimating3Dstructure
Source:SilvioSavarese
28-Nov-1766
![Page 67: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/67.jpg)
Lecture 17 -Stanford University
Segmentingobjectsbasedonmotioncues
• Backgroundsubtraction– Astaticcameraisobservingascene– Goal:separatethestaticbackground fromthemovingforeground
Source:SilvioSavarese
28-Nov-1767
![Page 68: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/68.jpg)
Lecture 17 -Stanford University
• Motionsegmentation– Segmentthevideointomultiplecoherentlymovingobjects
Segmentingobjectsbasedonmotioncues
S.J.Pundlik andS.T.Birchfield,MotionSegmentationatAnySpeed,ProceedingsoftheBritishMachineVisionConference(BMVC)2006
Source:SilvioSavarese
28-Nov-1768
![Page 69: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/69.jpg)
Lecture 17 -Stanford University
Z.Yin andR.Collins,"On-the-flyObjectModelingwhileTracking,"IEEEComputerVisionandPatternRecognition(CVPR'07),Minneapolis,MN,June2007.
Trackingobjects
Source:SilvioSavarese
28-Nov-1769
![Page 70: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/70.jpg)
Lecture 17 -Stanford University
Synthesizingdynamictextures
28-Nov-1770
![Page 71: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/71.jpg)
Lecture 17 -Stanford University 71
Super-resolution
Example:Asetoflowqualityimages
Source:SilvioSavarese
28-Nov-17
![Page 72: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/72.jpg)
Lecture 17 -Stanford University 72
Super-resolution
Eachoftheseimageslookslikethis:
Source:SilvioSavarese
28-Nov-17
![Page 73: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/73.jpg)
Lecture 17 -Stanford University 73
Super-resolution
Therecoveryresult:
Source:SilvioSavarese
28-Nov-17
![Page 74: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/74.jpg)
Lecture 17 -Stanford University
D. Ramanan, D. Forsyth, and A. Zisserman. Tracking People by Learning their Appearance. PAMI 2007.
Tracker
Recognizingeventsandactivities
Source:SilvioSavarese
28-Nov-1774
![Page 75: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/75.jpg)
Lecture 17 -Stanford University
JuanCarlosNiebles,HongchengWangandLiFei-Fei,UnsupervisedLearningofHumanActionCategoriesUsingSpatial-TemporalWords,(BMVC),Edinburgh,2006.
Recognizingeventsandactivities
28-Nov-1775
![Page 76: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/76.jpg)
Lecture 17 -Stanford University
Crossing– Talking– Queuing– Dancing– jogging
W.Choi &K.Shahid&S.SavareseWMC2010
Recognizingeventsandactivities
Source:SilvioSavarese
28-Nov-1776
![Page 77: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/77.jpg)
Lecture 17 -Stanford University
W.Choi,K.Shahid,S.Savarese,"Whataretheydoing?:CollectiveActivityClassificationUsingSpatio-TemporalRelationshipAmongPeople",9thInternationalWorkshoponVisualSurveillance(VSWS09)inconjuction withICCV09
28-Nov-1777
![Page 78: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/78.jpg)
Lecture 17 -Stanford University
Opticalflowwithoutmotion!
28-Nov-1778
![Page 79: Lecture: Motion - Stanford University](https://reader036.vdocument.in/reader036/viewer/2022062222/62a432bef3e39b36505fda63/html5/thumbnails/79.jpg)
Lecture 17 -Stanford University
Whatwehavelearnedtoday?
• Opticalflow• Lucas-Kanade method• Horn-Schunk method• Pyramidsforlargemotion• Commonfate• Applications
28-Nov-1779
[Fleet&Weiss,2005]http://www.cs.toronto.edu/pub/jepson/teaching/vision/2503/opticalFlow.pdf
Reading:[Szeliski]Chapters:8.4,8.5