steve seitz carnegie mellon university university of washington seitz passive 3d photography...

31
Steve Seitz Steve Seitz Carnegie Mellon University Carnegie Mellon University University of Washington University of Washington http://www.cs.cmu.edu/~seitz http://www.cs.cmu.edu/~seitz Passive 3D Photography Passive 3D Photography SIGGRAPH 2000 Course on SIGGRAPH 2000 Course on 3D Photography 3D Photography

Upload: augusta-ellis

Post on 22-Dec-2015

223 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Steve Seitz Carnegie Mellon University University of Washington seitz Passive 3D Photography SIGGRAPH 2000 Course on 3D Photography

Steve SeitzSteve Seitz

Carnegie Mellon UniversityCarnegie Mellon University

University of WashingtonUniversity of Washingtonhttp://www.cs.cmu.edu/~seitzhttp://www.cs.cmu.edu/~seitz

Passive 3D PhotographyPassive 3D Photography

SIGGRAPH 2000 Course onSIGGRAPH 2000 Course on3D Photography3D Photography

Page 2: Steve Seitz Carnegie Mellon University University of Washington seitz Passive 3D Photography SIGGRAPH 2000 Course on 3D Photography

ShadingShading

Visual CuesVisual Cues

Merle Norman Cosmetics, Los Angeles

Page 3: Steve Seitz Carnegie Mellon University University of Washington seitz Passive 3D Photography SIGGRAPH 2000 Course on 3D Photography

Visual CuesVisual Cues

ShadingShading

TextureTexture

The Visual Cliff, by William Vandivert, 1960

Page 4: Steve Seitz Carnegie Mellon University University of Washington seitz Passive 3D Photography SIGGRAPH 2000 Course on 3D Photography

Visual CuesVisual Cues

ShadingShading

TextureTexture

FocusFocus

From The Art of Photography, Canon

Page 5: Steve Seitz Carnegie Mellon University University of Washington seitz Passive 3D Photography SIGGRAPH 2000 Course on 3D Photography

ShadingShading

TextureTexture

FocusFocus

MotionMotion

Visual CuesVisual Cues

Page 6: Steve Seitz Carnegie Mellon University University of Washington seitz Passive 3D Photography SIGGRAPH 2000 Course on 3D Photography

Visual CuesVisual Cues

MotionMotion

ShadingShading

TextureTexture

FocusFocus

Others:Others:• HighlightsHighlights

• ShadowsShadows

• SilhouettesSilhouettes

• Inter-reflectionsInter-reflections

• SymmetrySymmetry

• Light PolarizationLight Polarization

• ......

Page 7: Steve Seitz Carnegie Mellon University University of Washington seitz Passive 3D Photography SIGGRAPH 2000 Course on 3D Photography

Reconstruction AlgorithmsReconstruction Algorithms

Shape From XShape From X• Stereo (shape from parallax)Stereo (shape from parallax)• Structure from motionStructure from motion• Shape from shadingShape from shading• Photometric stereoPhotometric stereo• Shape from textureShape from texture• Shape from focus/defocusShape from focus/defocus• Shape from silhouettes, ...Shape from silhouettes, ...

Talk OutlineTalk Outline1.1. Single View ModelingSingle View Modeling

2.2. Stereo ReconstructionStereo Reconstruction

3.3. Structure from MotionStructure from Motion

Page 8: Steve Seitz Carnegie Mellon University University of Washington seitz Passive 3D Photography SIGGRAPH 2000 Course on 3D Photography

Single View ModelingSingle View Modeling

What is This Scene?What is This Scene?

Page 9: Steve Seitz Carnegie Mellon University University of Washington seitz Passive 3D Photography SIGGRAPH 2000 Course on 3D Photography

How do Humans do This?How do Humans do This?

Good Guesswork based on PriorsGood Guesswork based on Priors• ““these lines these lines looklook parallel” parallel”

• ““this this lookslooks like a cube” like a cube”

• ““this this lookslooks like a shadow” like a shadow”

Computers can do this tooComputers can do this too• Shape from shading Shape from shading [Horn 89][Horn 89]

• User-aided modelingUser-aided modeling

> ““Tour into the Picture” [Horry 97]Tour into the Picture” [Horry 97]

> ““Facade” [Debevec 96] Facade” [Debevec 96]

> ““Single View Metrology” [Criminisi 99]Single View Metrology” [Criminisi 99]

• Learning approachesLearning approaches

> ““Morphable Models” [Blanz 99]Morphable Models” [Blanz 99]

Page 10: Steve Seitz Carnegie Mellon University University of Washington seitz Passive 3D Photography SIGGRAPH 2000 Course on 3D Photography

Perspective CuesPerspective Cues

Page 11: Steve Seitz Carnegie Mellon University University of Washington seitz Passive 3D Photography SIGGRAPH 2000 Course on 3D Photography

Perspective CuesPerspective Cues

Page 12: Steve Seitz Carnegie Mellon University University of Washington seitz Passive 3D Photography SIGGRAPH 2000 Course on 3D Photography

Perspective CuesPerspective Cues

Page 13: Steve Seitz Carnegie Mellon University University of Washington seitz Passive 3D Photography SIGGRAPH 2000 Course on 3D Photography

Vanishing PointsVanishing Points

Page 14: Steve Seitz Carnegie Mellon University University of Washington seitz Passive 3D Photography SIGGRAPH 2000 Course on 3D Photography

Measuring HeightMeasuring Height

1

2

3

4

55.4

2.8

3.3

Same Concepts EnableSame Concepts Enable• Reconstructing X, Y, and ZReconstructing X, Y, and Z

• Computing camera projection matrixComputing camera projection matrix

• Eliminating the rulerEliminating the ruler

Page 15: Steve Seitz Carnegie Mellon University University of Washington seitz Passive 3D Photography SIGGRAPH 2000 Course on 3D Photography

““Single View Metrology” [Criminisi 99]Single View Metrology” [Criminisi 99]

Page 16: Steve Seitz Carnegie Mellon University University of Washington seitz Passive 3D Photography SIGGRAPH 2000 Course on 3D Photography

““Single View Metrology” [Criminisi 99]Single View Metrology” [Criminisi 99]

The Music Lesson, Jan Vermeer, 1662-65 Royal Collection of Her Majesty Queen Elizabeth II

Page 17: Steve Seitz Carnegie Mellon University University of Washington seitz Passive 3D Photography SIGGRAPH 2000 Course on 3D Photography

““Morphable Models” [Blanz 99]Morphable Models” [Blanz 99]

VideoVideo

Page 18: Steve Seitz Carnegie Mellon University University of Washington seitz Passive 3D Photography SIGGRAPH 2000 Course on 3D Photography

Stereo ReconstructionStereo Reconstruction

The Stereo ProblemThe Stereo Problem• Shape from two (or more) imagesShape from two (or more) images

• Biological motivationBiological motivation

knownknowncameracamera

viewpointsviewpoints

Page 19: Steve Seitz Carnegie Mellon University University of Washington seitz Passive 3D Photography SIGGRAPH 2000 Course on 3D Photography

StereoStereo

scene pointscene point

focal pointfocal point

image planeimage plane

Page 20: Steve Seitz Carnegie Mellon University University of Washington seitz Passive 3D Photography SIGGRAPH 2000 Course on 3D Photography

StereoStereo

Basic Principle: TriangulationBasic Principle: Triangulation• Gives reconstruction as intersection of two raysGives reconstruction as intersection of two rays• Requires Requires point correspondencepoint correspondence

Page 21: Steve Seitz Carnegie Mellon University University of Washington seitz Passive 3D Photography SIGGRAPH 2000 Course on 3D Photography

Stereo CorrespondenceStereo Correspondence

Determine Pixel CorrespondenceDetermine Pixel Correspondence• Pairs of points that correspond to same scene pointPairs of points that correspond to same scene point

Epipolar ConstraintEpipolar Constraint• Reduces correspondence problem to 1D search along Reduces correspondence problem to 1D search along

conjugateconjugate epipolar linesepipolar lines

epipolar planeepipolar planeepipolar lineepipolar lineepipolar lineepipolar line

Page 22: Steve Seitz Carnegie Mellon University University of Washington seitz Passive 3D Photography SIGGRAPH 2000 Course on 3D Photography

Stereo Matching AlgorithmsStereo Matching Algorithms

Match Pixels in Conjugate Epipolar LinesMatch Pixels in Conjugate Epipolar Lines• Assume color of point does not changeAssume color of point does not change

• PitfallsPitfalls

> specularities specularities

> low-contrast regionslow-contrast regions

> occlusionsocclusions

> image errorimage error

> camera calibration errorcamera calibration error

• Numerous approachesNumerous approaches

> dynamic programming [Baker 81,Ohta 85]dynamic programming [Baker 81,Ohta 85]

> smoothness functionalssmoothness functionals

> more images (trinocular, N-ocular) [Okutomi 93]more images (trinocular, N-ocular) [Okutomi 93]

> graph cuts [Boykov 00]graph cuts [Boykov 00]

Page 23: Steve Seitz Carnegie Mellon University University of Washington seitz Passive 3D Photography SIGGRAPH 2000 Course on 3D Photography

Structure from MotionStructure from Motion

Reconstruct Reconstruct • Scene Scene geometrygeometry

• Camera Camera motionmotion

UnknownUnknowncameracamera

viewpointsviewpoints

Page 24: Steve Seitz Carnegie Mellon University University of Washington seitz Passive 3D Photography SIGGRAPH 2000 Course on 3D Photography

Structure from MotionStructure from Motion

The SFM ProblemThe SFM Problem• Reconstruct scene Reconstruct scene geometrygeometry and camera and camera motionmotion from from

two or more imagestwo or more images

Track2D Points Estimate

3D OptimizeFit Surfaces

Page 25: Steve Seitz Carnegie Mellon University University of Washington seitz Passive 3D Photography SIGGRAPH 2000 Course on 3D Photography

Structure From MotionStructure From Motion

Step 1: Track FeaturesStep 1: Track Features• Detect good featuresDetect good features

> corners, line segmentscorners, line segments

• Find correspondences between framesFind correspondences between frames

> window-based correlationwindow-based correlation

Page 26: Steve Seitz Carnegie Mellon University University of Washington seitz Passive 3D Photography SIGGRAPH 2000 Course on 3D Photography

Structure From MotionStructure From Motion

Step 2: Estimate Motion and StructureStep 2: Estimate Motion and Structure• Orthographic projection, e.g., Orthographic projection, e.g., [Tomasi 92][Tomasi 92]

• 2 or 3 views at a time 2 or 3 views at a time [Hartley 00][Hartley 00]

n21

f

2

1

f

2

1

XXX

Π

Π

Π

I

I

I

Images Motion

Structure

Page 27: Steve Seitz Carnegie Mellon University University of Washington seitz Passive 3D Photography SIGGRAPH 2000 Course on 3D Photography

Structure From MotionStructure From Motion

Step 3: Refine EstimatesStep 3: Refine Estimates• Nonlinear optimization over cameras and pointsNonlinear optimization over cameras and points

> [Hartley 94][Hartley 94]

• ““Bundle adjustment” in photogrammetryBundle adjustment” in photogrammetry

Page 28: Steve Seitz Carnegie Mellon University University of Washington seitz Passive 3D Photography SIGGRAPH 2000 Course on 3D Photography

Structure From MotionStructure From Motion

Step 4: Recover SurfacesStep 4: Recover Surfaces• Image-based triangulation Image-based triangulation [Morris 00, Baillard 99][Morris 00, Baillard 99]

• Silhouettes Silhouettes [Fitzgibbon 98][Fitzgibbon 98]

• Stereo Stereo [Pollefeys 99][Pollefeys 99]

Poor meshPoor mesh Good meshGood meshMorris and Kanade, 2000Morris and Kanade, 2000

Page 29: Steve Seitz Carnegie Mellon University University of Washington seitz Passive 3D Photography SIGGRAPH 2000 Course on 3D Photography

ResourcesResources

Computer Vision Home PageComputer Vision Home Page• http://www.cs.cmu.edu/afs/cs/project/cil/ftp/html/vision.htmlhttp://www.cs.cmu.edu/afs/cs/project/cil/ftp/html/vision.html

Computer Vision TextbooksComputer Vision Textbooks• O. Faugeras, O. Faugeras, Three-Dimensional Computer VisionThree-Dimensional Computer Vision, MIT Press, 1993. , MIT Press, 1993.

• E. Trucco and A. Verri, E. Trucco and A. Verri, Introductory Techniques for 3-D Computer VisionIntroductory Techniques for 3-D Computer Vision, Prentice-Hall, , Prentice-Hall, 1998. 1998.

• V. S. Nalwa, V. S. Nalwa, A Guided Tour of Computer VisionA Guided Tour of Computer Vision, Addison-Wesley, 1993. , Addison-Wesley, 1993.

• R. Jain, R. Kasturi and B. G. Schunck, R. Jain, R. Kasturi and B. G. Schunck, Machine VisionMachine Vision, McGraw-Hill, 1995. , McGraw-Hill, 1995.

• R. Klette, K. Schluns and A. Koschan, R. Klette, K. Schluns and A. Koschan, Computer Vision: Three-Dimensional Data from Computer Vision: Three-Dimensional Data from ImagesImages, Springer-Verlag, 1998. , Springer-Verlag, 1998.

• M. Sonka, V. Hlavac and R. Boyle, M. Sonka, V. Hlavac and R. Boyle, Image Processing, Analysis, and Machine VisionImage Processing, Analysis, and Machine Vision, , Brooks/Cole Publishing, 1999. Brooks/Cole Publishing, 1999.

• D. H. Ballard and C. M. Brown, D. H. Ballard and C. M. Brown, Computer VisionComputer Vision, Prentice-Hall, 1982., Prentice-Hall, 1982.

• B. K. P. Horn, B. K. P. Horn, Robot VisionRobot Vision, McGraw-Hill, 1986. , McGraw-Hill, 1986.

• J. Koenderink, J. Koenderink, Solid ShapeSolid Shape, MIT Press, 1990. , MIT Press, 1990.

• D. Marr, D. Marr, VisionVision, Freeman, 1982. , Freeman, 1982.

Page 30: Steve Seitz Carnegie Mellon University University of Washington seitz Passive 3D Photography SIGGRAPH 2000 Course on 3D Photography

Single View ModelingSingle View Modeling• V. Blanz & T. Vetter, “A Morphable Model for the Synthesis of 3D Faces”, SIGGRAPH 99, V. Blanz & T. Vetter, “A Morphable Model for the Synthesis of 3D Faces”, SIGGRAPH 99,

pp. 187-194.pp. 187-194.

• A. Criminisi, I. Reid, & A. Zisserman, “Single View Metrology”, ICCV 2000, pp. 434-441.A. Criminisi, I. Reid, & A. Zisserman, “Single View Metrology”, ICCV 2000, pp. 434-441.

• B. K. P. Horn & M. Brooks, “Shape from Shading”, 1989, MIT Press, Cambridge, M.A.B. K. P. Horn & M. Brooks, “Shape from Shading”, 1989, MIT Press, Cambridge, M.A.

• Y. Horry, K. Anjyo, & K. Arai, “Tour into the Picture”, SIGGRAPH 97, pp. 225-232.Y. Horry, K. Anjyo, & K. Arai, “Tour into the Picture”, SIGGRAPH 97, pp. 225-232.

• R. Zhang, P-S. Tsai, J. Cryer, & M. Shah, “Shape from Shading: A Survey”, IEEE Trans. R. Zhang, P-S. Tsai, J. Cryer, & M. Shah, “Shape from Shading: A Survey”, IEEE Trans. on PAMI, 21(8), 1999.on PAMI, 21(8), 1999.

StereoStereo• Y. Boykov, O. Veksler, & R. Zabih, “Fast Approximate Energy Minimization via Graph Y. Boykov, O. Veksler, & R. Zabih, “Fast Approximate Energy Minimization via Graph

Cuts”, ICCV, 1999.Cuts”, ICCV, 1999.

• Y. Ohta & T. Kanade, "Stereo by Intra- and Inter-Scanline Search Using Dynamic Y. Ohta & T. Kanade, "Stereo by Intra- and Inter-Scanline Search Using Dynamic Programming", IEEE Trans. on PAMI, 7(2), 1985, pp. 129-154.Programming", IEEE Trans. on PAMI, 7(2), 1985, pp. 129-154.

• M. Okutomi & T. Kanade, ”A Multiple-Baseline Stereo", IEEE Trans. on Pattern Analysis M. Okutomi & T. Kanade, ”A Multiple-Baseline Stereo", IEEE Trans. on Pattern Analysis and Machine Intelligence", 15(4), 1993, 353-363.and Machine Intelligence", 15(4), 1993, 353-363.

BibliographyBibliography

Page 31: Steve Seitz Carnegie Mellon University University of Washington seitz Passive 3D Photography SIGGRAPH 2000 Course on 3D Photography

Structure from MotionStructure from Motion• C. Baillard & A. Zisserman, “Automatic Reconstruction of Planar Models from Multiple C. Baillard & A. Zisserman, “Automatic Reconstruction of Planar Models from Multiple

Views”, CVPR 99, pp. 559-565.Views”, CVPR 99, pp. 559-565.

• A.W. Fitzgibbon, G. Cross, & A. Zisserman, “Automatic 3D Model Construction for Turn-A.W. Fitzgibbon, G. Cross, & A. Zisserman, “Automatic 3D Model Construction for Turn-Table Sequences”, SMILE Workshop, 1998. Table Sequences”, SMILE Workshop, 1998.

• R. Hartley & A. Zisserman, “Multiple View Geometry”, Cambridge Univ. Press, 2000.R. Hartley & A. Zisserman, “Multiple View Geometry”, Cambridge Univ. Press, 2000.

• R. Hartley, “Euclidean Reconstruction from Uncalibrated Views”, In Applications of R. Hartley, “Euclidean Reconstruction from Uncalibrated Views”, In Applications of Invariance in Computer Vision, Springer-Verlag, 1994, pp. 237-256.Invariance in Computer Vision, Springer-Verlag, 1994, pp. 237-256.

• D. Morris & T. Kanade, “Image-Consistent Surface Triangulation”, CVPR 00, pp. 332-338.D. Morris & T. Kanade, “Image-Consistent Surface Triangulation”, CVPR 00, pp. 332-338.

• M. Pollefeys, R. Koch & L. Van Gool, “Self-Calibration and Metric Reconstruction in spite M. Pollefeys, R. Koch & L. Van Gool, “Self-Calibration and Metric Reconstruction in spite of Varying and Unknown Internal Camera Parameters”, Int. J. of Computer Vision, 32(1), of Varying and Unknown Internal Camera Parameters”, Int. J. of Computer Vision, 32(1), 1999, pp. 7-25.1999, pp. 7-25.

• C. Tomasi & T. Kanade, ”Shape and Motion from Image Streams Under Orthography: A C. Tomasi & T. Kanade, ”Shape and Motion from Image Streams Under Orthography: A Factorization Method", Int. Journal of Computer Vision, 9(2), 1992, pp. 137-154. Factorization Method", Int. Journal of Computer Vision, 9(2), 1992, pp. 137-154.

BibliographyBibliography