motion from normal flow. optical flow difficulties the aperture problemdepth discontinuities
Post on 22-Dec-2015
223 views
TRANSCRIPT
![Page 1: Motion from normal flow. Optical flow difficulties The aperture problemDepth discontinuities](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7e5503460f94a61c4f/html5/thumbnails/1.jpg)
Motion from normal flow
![Page 2: Motion from normal flow. Optical flow difficulties The aperture problemDepth discontinuities](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7e5503460f94a61c4f/html5/thumbnails/2.jpg)
Optical flow difficulties
• The aperture problem • Depth discontinuities
![Page 3: Motion from normal flow. Optical flow difficulties The aperture problemDepth discontinuities](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7e5503460f94a61c4f/html5/thumbnails/3.jpg)
Translational Normal Flow
• In the case of translation each normal flow vector constrains the location of the FOE to ahalf-plane.
• Intersection of half-planes provides FOE.
nu
u Zn
tr
![Page 4: Motion from normal flow. Optical flow difficulties The aperture problemDepth discontinuities](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7e5503460f94a61c4f/html5/thumbnails/4.jpg)
Egoestimation from normal flow
• Idea: choose particular directions: patterns defined on the sign of normal flow along particular orientation fields
• positive depth constraint
• 2 classes of orientation fields: copoint vectors and coaxis vectors
![Page 5: Motion from normal flow. Optical flow difficulties The aperture problemDepth discontinuities](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7e5503460f94a61c4f/html5/thumbnails/5.jpg)
Optical flow and normal flow
![Page 6: Motion from normal flow. Optical flow difficulties The aperture problemDepth discontinuities](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7e5503460f94a61c4f/html5/thumbnails/6.jpg)
Optical flow and normal flow
![Page 7: Motion from normal flow. Optical flow difficulties The aperture problemDepth discontinuities](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7e5503460f94a61c4f/html5/thumbnails/7.jpg)
Coaxis vectorswith respect to axis (A,B,C)
![Page 8: Motion from normal flow. Optical flow difficulties The aperture problemDepth discontinuities](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7e5503460f94a61c4f/html5/thumbnails/8.jpg)
Coaxis vectors
![Page 9: Motion from normal flow. Optical flow difficulties The aperture problemDepth discontinuities](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7e5503460f94a61c4f/html5/thumbnails/9.jpg)
Translational coaxis vectors
![Page 10: Motion from normal flow. Optical flow difficulties The aperture problemDepth discontinuities](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7e5503460f94a61c4f/html5/thumbnails/10.jpg)
Translational coaxis vectors
h passes through FOE and (Af/C, Bf/C), defined by 2 parameters
![Page 11: Motion from normal flow. Optical flow difficulties The aperture problemDepth discontinuities](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7e5503460f94a61c4f/html5/thumbnails/11.jpg)
Rotational coaxis vectors
![Page 12: Motion from normal flow. Optical flow difficulties The aperture problemDepth discontinuities](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7e5503460f94a61c4f/html5/thumbnails/12.jpg)
Rotational coaxis vectors
![Page 13: Motion from normal flow. Optical flow difficulties The aperture problemDepth discontinuities](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7e5503460f94a61c4f/html5/thumbnails/13.jpg)
Rotational coaxis vectors
g passes through AOR and (Af/C, Bf/C), defined by 1 parameter
![Page 14: Motion from normal flow. Optical flow difficulties The aperture problemDepth discontinuities](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7e5503460f94a61c4f/html5/thumbnails/14.jpg)
Combine translation and rotation
Positive + positive positive
Negative + negative negative
Positive + negative don’t know (depends on structure)
![Page 15: Motion from normal flow. Optical flow difficulties The aperture problemDepth discontinuities](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7e5503460f94a61c4f/html5/thumbnails/15.jpg)
Coaxis patterntranslational
rotational
combined
![Page 16: Motion from normal flow. Optical flow difficulties The aperture problemDepth discontinuities](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7e5503460f94a61c4f/html5/thumbnails/16.jpg)
-vectors: Translation
![Page 17: Motion from normal flow. Optical flow difficulties The aperture problemDepth discontinuities](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7e5503460f94a61c4f/html5/thumbnails/17.jpg)
-vectors: Rotation
cossin22
rot
f
r
f
ru
![Page 18: Motion from normal flow. Optical flow difficulties The aperture problemDepth discontinuities](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7e5503460f94a61c4f/html5/thumbnails/18.jpg)
-vectors: Translation and Rotation
UW
VW
![Page 19: Motion from normal flow. Optical flow difficulties The aperture problemDepth discontinuities](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7e5503460f94a61c4f/html5/thumbnails/19.jpg)
alpha beta gamma
Three coaxis vector fields
![Page 20: Motion from normal flow. Optical flow difficulties The aperture problemDepth discontinuities](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7e5503460f94a61c4f/html5/thumbnails/20.jpg)
Copoint vectors
![Page 21: Motion from normal flow. Optical flow difficulties The aperture problemDepth discontinuities](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7e5503460f94a61c4f/html5/thumbnails/21.jpg)
O
copoint vectors
![Page 22: Motion from normal flow. Optical flow difficulties The aperture problemDepth discontinuities](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7e5503460f94a61c4f/html5/thumbnails/22.jpg)
Copoint vectors
defined by point (r,s)
![Page 23: Motion from normal flow. Optical flow difficulties The aperture problemDepth discontinuities](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7e5503460f94a61c4f/html5/thumbnails/23.jpg)
Translational copoint vectors
![Page 24: Motion from normal flow. Optical flow difficulties The aperture problemDepth discontinuities](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7e5503460f94a61c4f/html5/thumbnails/24.jpg)
FOE
AOR
FOE
AOR
FOE
AOR
: Negative
: Positive
: Don't know
FOE
AOR
![Page 25: Motion from normal flow. Optical flow difficulties The aperture problemDepth discontinuities](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7e5503460f94a61c4f/html5/thumbnails/25.jpg)
Translational copoint vectors
k passes through FOE and (r,s) defined by 1 parameter
(r, s)
FOE
![Page 26: Motion from normal flow. Optical flow difficulties The aperture problemDepth discontinuities](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7e5503460f94a61c4f/html5/thumbnails/26.jpg)
Rotational copoint vectors
![Page 27: Motion from normal flow. Optical flow difficulties The aperture problemDepth discontinuities](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7e5503460f94a61c4f/html5/thumbnails/27.jpg)
Rotational copoint vectors
l passes through AOR and (r,s), is defined by 2 parameters
(r, s)
AOR
![Page 28: Motion from normal flow. Optical flow difficulties The aperture problemDepth discontinuities](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7e5503460f94a61c4f/html5/thumbnails/28.jpg)
(r, s)
FOE
(r, s)
AOR
FOE
(r, s)
AOR
translational component rotational component
![Page 29: Motion from normal flow. Optical flow difficulties The aperture problemDepth discontinuities](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7e5503460f94a61c4f/html5/thumbnails/29.jpg)
(a) (b) (c)
Three coaxis vector fields
![Page 30: Motion from normal flow. Optical flow difficulties The aperture problemDepth discontinuities](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7e5503460f94a61c4f/html5/thumbnails/30.jpg)
a,b,c : positive and negative vectors
c,d,e: Fitting of patterns
g: Separation of (coaxis patternh: Separation of (x0, y0) copoint pattern
![Page 31: Motion from normal flow. Optical flow difficulties The aperture problemDepth discontinuities](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7e5503460f94a61c4f/html5/thumbnails/31.jpg)
FOE
AOR
FOE
AOR
FOE
AOR
: Negative
: Positive
: Don't know
FOE
AOR
![Page 32: Motion from normal flow. Optical flow difficulties The aperture problemDepth discontinuities](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7e5503460f94a61c4f/html5/thumbnails/32.jpg)
![Page 33: Motion from normal flow. Optical flow difficulties The aperture problemDepth discontinuities](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7e5503460f94a61c4f/html5/thumbnails/33.jpg)
Opticalillusion
![Page 34: Motion from normal flow. Optical flow difficulties The aperture problemDepth discontinuities](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7e5503460f94a61c4f/html5/thumbnails/34.jpg)
![Page 35: Motion from normal flow. Optical flow difficulties The aperture problemDepth discontinuities](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7e5503460f94a61c4f/html5/thumbnails/35.jpg)
What is the Problem?
• Flow can be accurately estimated in an image patch corresponding to a smooth scene patch,
• But erroneous flow estimates are obtained for image patches corresponding to scene patches containing discontinuities
Image Flow
3D MotionScene structureDiscontinuities
![Page 36: Motion from normal flow. Optical flow difficulties The aperture problemDepth discontinuities](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7e5503460f94a61c4f/html5/thumbnails/36.jpg)
Depth variability constraint
• Errors in motion estimates lead to distortion of the scene estimates.
• The distortion is such that the correct motion gives the “smoothest” (least varying) scene structure.
![Page 37: Motion from normal flow. Optical flow difficulties The aperture problemDepth discontinuities](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7e5503460f94a61c4f/html5/thumbnails/37.jpg)
• Scene depth can be estimated from normal flow measurements:
ntu
nu
)(ˆ
ωˆ1
tr
rotnu
Z
Depth estimation
nununu rottrn
1
Zu
![Page 38: Motion from normal flow. Optical flow difficulties The aperture problemDepth discontinuities](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7e5503460f94a61c4f/html5/thumbnails/38.jpg)
Visual Space Distortion
• Wrong 3D motion gives rise to a rugged (unsmooth) depth function (surface).
• The correct 3D motion leads to the “smoothest” estimated depth.
nutu
ntu
ωδ
ˆ ,ˆ
rottr
trDDZZ
![Page 39: Motion from normal flow. Optical flow difficulties The aperture problemDepth discontinuities](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7e5503460f94a61c4f/html5/thumbnails/39.jpg)
Inverse depth estimates
correct motion
incorrect motion
![Page 40: Motion from normal flow. Optical flow difficulties The aperture problemDepth discontinuities](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7e5503460f94a61c4f/html5/thumbnails/40.jpg)
The error function
• A normal flow measurement:
For an estimate
• The error function to be minimized:
nunu rottrn
1
Zu
2
))ˆ()(ˆ1
()ˆ(
ii
ii n
ZnuW tuωu trrotin
nωuntu )ˆ()ˆ(1
rottrn Zu
ωt ˆ,ˆ
![Page 41: Motion from normal flow. Optical flow difficulties The aperture problemDepth discontinuities](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7e5503460f94a61c4f/html5/thumbnails/41.jpg)
The error function• Estimated normal flow
• The error function to be minimized:
• Global parameters:• Local parameter: locally planar patches:
nωuntu )ˆ()ˆ(ˆ1
ˆ rottrnZ
u
2
nnˆ R i
i uuW
ωt ˆ,ˆZ
cbyaxZ
ˆ1
![Page 42: Motion from normal flow. Optical flow difficulties The aperture problemDepth discontinuities](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7e5503460f94a61c4f/html5/thumbnails/42.jpg)
Error function evaluation
• Given a translation candidate , each local depth can be computed as a linear function of the rotation .
• We obtain a second order function of the rotation; its minimization provides both the rotation and the value of the error function.
t
ω
![Page 43: Motion from normal flow. Optical flow difficulties The aperture problemDepth discontinuities](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7e5503460f94a61c4f/html5/thumbnails/43.jpg)
Is derived from image gradients only
Brightness consistency:
Flow:
Planar patch:
![Page 44: Motion from normal flow. Optical flow difficulties The aperture problemDepth discontinuities](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7e5503460f94a61c4f/html5/thumbnails/44.jpg)
Handling depth discontinuities
• Given a candidate motion, the scene depth can be estimated and further processed to find depth discontinuities.
• Split a region if it corresponds to two depth values separated in space.
![Page 45: Motion from normal flow. Optical flow difficulties The aperture problemDepth discontinuities](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7e5503460f94a61c4f/html5/thumbnails/45.jpg)
The algorithm
• Compute spatio-temporal image derivatives and normal flow.
• Find the direction of translation that minimizes the depth-variability criterion.– Hierarchical search of the 2D space.– Iterative minimization.– Utilize continuity of the solution in time;
usually the motion changes slowly over time.
![Page 46: Motion from normal flow. Optical flow difficulties The aperture problemDepth discontinuities](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7e5503460f94a61c4f/html5/thumbnails/46.jpg)
Divide image into small patches
Search in the 2D space of translations
For each candidate 3D motion, using normal flow
measurements in each patch, compute depth of
the scene.
For each image patch
NO
Depth variation small?
Distinguish between two cases
wrong 3D motionexistence of a
discontinuity at the patch
Use the error Split the patch and repeat the
process
Use the error
YES
The Algorithm
![Page 47: Motion from normal flow. Optical flow difficulties The aperture problemDepth discontinuities](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7e5503460f94a61c4f/html5/thumbnails/47.jpg)
![Page 48: Motion from normal flow. Optical flow difficulties The aperture problemDepth discontinuities](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7e5503460f94a61c4f/html5/thumbnails/48.jpg)
3D reconstruction
• Comparison of the original sequence and the re-projection of the 3D reconstruction.
![Page 49: Motion from normal flow. Optical flow difficulties The aperture problemDepth discontinuities](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7e5503460f94a61c4f/html5/thumbnails/49.jpg)
3D model construction