3d photography - cvg · • cover basic 3d photography concepts and approaches. • further...

59
3D Photography Kevin Köser and Marc Pollefeys Spring 2011

Upload: others

Post on 21-Sep-2020

3 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: 3D Photography - CVG · • Cover basic 3D photography concepts and approaches. • Further lectures: • Short introduction to topic • Paper presentations (seminal papers and state-of-the-art,

3D Photography

Kevin Köser and Marc Pollefeys

Spring 2011

Page 2: 3D Photography - CVG · • Cover basic 3D photography concepts and approaches. • Further lectures: • Short introduction to topic • Paper presentations (seminal papers and state-of-the-art,

3D Photography

•  Understanding geometric relations between images and the 3D world, as well as between images

•  Obtaining 3D information from our 3D world

Page 3: 3D Photography - CVG · • Cover basic 3D photography concepts and approaches. • Further lectures: • Short introduction to topic • Paper presentations (seminal papers and state-of-the-art,

Motivation

•  Applications in many different areas

•  A few examples …

Page 4: 3D Photography - CVG · • Cover basic 3D photography concepts and approaches. • Further lectures: • Short introduction to topic • Paper presentations (seminal papers and state-of-the-art,

Architecture

Survey Stability analysis Plan renovations

Page 5: 3D Photography - CVG · • Cover basic 3D photography concepts and approaches. • Further lectures: • Short introduction to topic • Paper presentations (seminal papers and state-of-the-art,

Architecture

Survey Stability analysis Plan renovations

Page 6: 3D Photography - CVG · • Cover basic 3D photography concepts and approaches. • Further lectures: • Short introduction to topic • Paper presentations (seminal papers and state-of-the-art,

Interactive 3D Modeling

(Sinha et al. Siggraph Asia 08) collaboration with Microsoft Research (and licensed to MS)

Page 7: 3D Photography - CVG · • Cover basic 3D photography concepts and approaches. • Further lectures: • Short introduction to topic • Paper presentations (seminal papers and state-of-the-art,
Page 8: 3D Photography - CVG · • Cover basic 3D photography concepts and approaches. • Further lectures: • Short introduction to topic • Paper presentations (seminal papers and state-of-the-art,

3D urban modeling

UNC/UKY UrbanScape project

Page 9: 3D Photography - CVG · • Cover basic 3D photography concepts and approaches. • Further lectures: • Short introduction to topic • Paper presentations (seminal papers and state-of-the-art,

Virtual Tourism

Page 10: 3D Photography - CVG · • Cover basic 3D photography concepts and approaches. • Further lectures: • Short introduction to topic • Paper presentations (seminal papers and state-of-the-art,

Clothing

•  Scan a person, custom-fit clothing

Page 11: 3D Photography - CVG · • Cover basic 3D photography concepts and approaches. • Further lectures: • Short introduction to topic • Paper presentations (seminal papers and state-of-the-art,

Industrial inspection

•  Verify specifications •  Compare measured model with CAD

Page 12: 3D Photography - CVG · • Cover basic 3D photography concepts and approaches. • Further lectures: • Short introduction to topic • Paper presentations (seminal papers and state-of-the-art,

Scanning industrial sites

as-build 3D model of off-shore oil platform

Page 13: 3D Photography - CVG · • Cover basic 3D photography concepts and approaches. • Further lectures: • Short introduction to topic • Paper presentations (seminal papers and state-of-the-art,

Robot navigation

small tethered rover

pan/tilt stereo head

ESA project our task: Calibration + Terrain modelling + Visualization

Page 14: 3D Photography - CVG · • Cover basic 3D photography concepts and approaches. • Further lectures: • Short introduction to topic • Paper presentations (seminal papers and state-of-the-art,

Cultural heritage Virtual Monticello

Allow virtual visits

Page 15: 3D Photography - CVG · • Cover basic 3D photography concepts and approaches. • Further lectures: • Short introduction to topic • Paper presentations (seminal papers and state-of-the-art,

Cultural heritage

Stanford s Digital Michelangelo

Digital archive Art historic studies

Page 16: 3D Photography - CVG · • Cover basic 3D photography concepts and approaches. • Further lectures: • Short introduction to topic • Paper presentations (seminal papers and state-of-the-art,

Archaeology

accuracy ~1/500 from DV video (i.e. 140kb jpegs 576x720)

Page 17: 3D Photography - CVG · • Cover basic 3D photography concepts and approaches. • Further lectures: • Short introduction to topic • Paper presentations (seminal papers and state-of-the-art,

Record different excavation layers

Archaeology

Generate & verify construction hypothesis

Layer 1 Layer 2

Generate 4D excavation record

Page 18: 3D Photography - CVG · • Cover basic 3D photography concepts and approaches. • Further lectures: • Short introduction to topic • Paper presentations (seminal papers and state-of-the-art,

Forensics

•  Crime scene recording and analysis

Page 19: 3D Photography - CVG · • Cover basic 3D photography concepts and approaches. • Further lectures: • Short introduction to topic • Paper presentations (seminal papers and state-of-the-art,
Page 20: 3D Photography - CVG · • Cover basic 3D photography concepts and approaches. • Further lectures: • Short introduction to topic • Paper presentations (seminal papers and state-of-the-art,

Forensics

Page 21: 3D Photography - CVG · • Cover basic 3D photography concepts and approaches. • Further lectures: • Short introduction to topic • Paper presentations (seminal papers and state-of-the-art,

Surgery - simulation

•  simulate results of surgery •  allows preoperative planning

Page 22: 3D Photography - CVG · • Cover basic 3D photography concepts and approaches. • Further lectures: • Short introduction to topic • Paper presentations (seminal papers and state-of-the-art,

Surgery - teaching

Capture models of surgery for interactive learning

Page 23: 3D Photography - CVG · • Cover basic 3D photography concepts and approaches. • Further lectures: • Short introduction to topic • Paper presentations (seminal papers and state-of-the-art,

Surveillance

Page 24: 3D Photography - CVG · • Cover basic 3D photography concepts and approaches. • Further lectures: • Short introduction to topic • Paper presentations (seminal papers and state-of-the-art,

Surveillance

Page 25: 3D Photography - CVG · • Cover basic 3D photography concepts and approaches. • Further lectures: • Short introduction to topic • Paper presentations (seminal papers and state-of-the-art,

Computer games Content capture

HCI

Page 26: 3D Photography - CVG · • Cover basic 3D photography concepts and approaches. • Further lectures: • Short introduction to topic • Paper presentations (seminal papers and state-of-the-art,

Raw Kinect output: Color + Depth

Computational 3D Photography 26

http://grouplab.cpsc.ucalgary.ca/cookbook/index.php/Technologies/Kinect

Page 27: 3D Photography - CVG · • Cover basic 3D photography concepts and approaches. • Further lectures: • Short introduction to topic • Paper presentations (seminal papers and state-of-the-art,

Bodytracking

Computer Vision, Kinect and beyond 27

� � � �

� � � � � � � � � � � � � �

Randomized forest (Shotton et al.)

based on simple depth difference tests at each node

Page 28: 3D Photography - CVG · • Cover basic 3D photography concepts and approaches. • Further lectures: • Short introduction to topic • Paper presentations (seminal papers and state-of-the-art,

Human-machine interface

28

Page 29: 3D Photography - CVG · • Cover basic 3D photography concepts and approaches. • Further lectures: • Short introduction to topic • Paper presentations (seminal papers and state-of-the-art,

3D Video with Kinect

29

Page 30: 3D Photography - CVG · • Cover basic 3D photography concepts and approaches. • Further lectures: • Short introduction to topic • Paper presentations (seminal papers and state-of-the-art,

3D free-viewpoint video with Kinect

30

(Kuster 2011, under review)

Intended application: 3D tele-presence (BeingThere UNC-ETH-NTU center)

Page 31: 3D Photography - CVG · • Cover basic 3D photography concepts and approaches. • Further lectures: • Short introduction to topic • Paper presentations (seminal papers and state-of-the-art,

3D free-viewpoint video with Kinect

31

(Kuster 2011, under review)

Page 32: 3D Photography - CVG · • Cover basic 3D photography concepts and approaches. • Further lectures: • Short introduction to topic • Paper presentations (seminal papers and state-of-the-art,

Autonomous micro-helicopter navigation

32 Use Kinect to map out obstacles and avoid collisions

Page 33: 3D Photography - CVG · • Cover basic 3D photography concepts and approaches. • Further lectures: • Short introduction to topic • Paper presentations (seminal papers and state-of-the-art,

Motion capture

•  Facial capture

•  Full body motion

Page 34: 3D Photography - CVG · • Cover basic 3D photography concepts and approaches. • Further lectures: • Short introduction to topic • Paper presentations (seminal papers and state-of-the-art,

Sport analysis

Page 35: 3D Photography - CVG · • Cover basic 3D photography concepts and approaches. • Further lectures: • Short introduction to topic • Paper presentations (seminal papers and state-of-the-art,

Course objectives

•  To understand the concepts that relate images to the 3D world and images to other images

•  Explore the state of the art in 3D photography

•  Implement a 3D photography system/algorithm

Page 36: 3D Photography - CVG · • Cover basic 3D photography concepts and approaches. • Further lectures: • Short introduction to topic • Paper presentations (seminal papers and state-of-the-art,

Material

Slides and more

http://www.cvg.ethz.ch/teaching/2011spring/3dphoto/ Also check out on-line “shape-from-video” tutorial:

http://www.cs.unc.edu/~marc/tutorial.pdf http://www.cs.unc.edu/~marc/tutorial/

Other interesting stuff: •  Book by Hartley & Zisserman, Multiple View Geometry

•  Peter Allen’s 3D photography class webpage http://www1.cs.columbia.edu/~allen/S10/

Page 37: 3D Photography - CVG · • Cover basic 3D photography concepts and approaches. • Further lectures: • Short introduction to topic • Paper presentations (seminal papers and state-of-the-art,

Learning approach

•  Introductory lectures: •  Cover basic 3D photography concepts and approaches.

•  Further lectures: •  Short introduction to topic •  Paper presentations

(seminal papers and state-of-the-art, related to projects)

•  3D photography project: •  Choose topic, define scope (by week 4) •  Implement algorithm/system •  Presentation/demo and paper report

Grade distribution •  Paper presentation & discussions: 20% •  3D photography project: 80%

Page 38: 3D Photography - CVG · • Cover basic 3D photography concepts and approaches. • Further lectures: • Short introduction to topic • Paper presentations (seminal papers and state-of-the-art,

Content

•  camera model and calibration •  single-view metrology •  triangulation •  epipolar geometry, stereo and rectification •  multi-view stereo •  structured-light, active techniques •  feature tracking and matching •  structure-from-motion and V-SLAM •  shape-from-silhouettes •  …

Page 39: 3D Photography - CVG · • Cover basic 3D photography concepts and approaches. • Further lectures: • Short introduction to topic • Paper presentations (seminal papers and state-of-the-art,

3D photography course schedule (tentative)

Topic

Feb 21 Introduction + Geometry and Camera Model

Feb 28 Lecture: Camera calibration + Epipolar Geometry

Mar 7 Lecture: Active Ranging, Structured Light, Kinect

Mar 14 Project Proposals

Mar 21 Lecture: Features, Tracking/Matching

Mar 28 Topic-intro + 2 papers (2 teams of 2 students)

Apr 4 Topic-intro + 2 papers

Apr 11 Topic-intro +2 papers

Apr 18 Project Updates

Apr. 25 --- Easter ---

May 2 Topic-intro + 2 papers

May 9 Topic-intro + 2 papers

May 16 Topic-intro + 2 papers

May 23 Topic-intro + 2 papers

May 30 Final Project presentation/demos

Page 40: 3D Photography - CVG · • Cover basic 3D photography concepts and approaches. • Further lectures: • Short introduction to topic • Paper presentations (seminal papers and state-of-the-art,

Fast Forward!

•  Quick overview of what is coming…

Page 41: 3D Photography - CVG · • Cover basic 3D photography concepts and approaches. • Further lectures: • Short introduction to topic • Paper presentations (seminal papers and state-of-the-art,

Camera models and geometry Pinhole camera

Geometric transformations in 2D and 3D

or

Page 42: 3D Photography - CVG · • Cover basic 3D photography concepts and approaches. • Further lectures: • Short introduction to topic • Paper presentations (seminal papers and state-of-the-art,

Camera Calibration

•  Know 2D/3D correspondences, compute projection matrix

also radial distortion (non-linear)

Page 43: 3D Photography - CVG · • Cover basic 3D photography concepts and approaches. • Further lectures: • Short introduction to topic • Paper presentations (seminal papers and state-of-the-art,

Single View Metrology

Page 44: 3D Photography - CVG · • Cover basic 3D photography concepts and approaches. • Further lectures: • Short introduction to topic • Paper presentations (seminal papers and state-of-the-art,

Single View Metrology

Antonio Criminisi

Page 45: 3D Photography - CVG · • Cover basic 3D photography concepts and approaches. • Further lectures: • Short introduction to topic • Paper presentations (seminal papers and state-of-the-art,

Feature tracking and matching

Harris corners, KLT features, SIFT features key concepts: invariance of extraction, descriptors to viewpoint, exposure and illumination changes

Page 46: 3D Photography - CVG · • Cover basic 3D photography concepts and approaches. • Further lectures: • Short introduction to topic • Paper presentations (seminal papers and state-of-the-art,

l2

3D from images

C1 m1 M? L1

m2

L2 M

C2

Triangulation -  calibration -  correspondences

Page 47: 3D Photography - CVG · • Cover basic 3D photography concepts and approaches. • Further lectures: • Short introduction to topic • Paper presentations (seminal papers and state-of-the-art,

Epipolar Geometry

Fundamental matrix Essential matrix

Also how to robustly compute from images

Page 48: 3D Photography - CVG · • Cover basic 3D photography concepts and approaches. • Further lectures: • Short introduction to topic • Paper presentations (seminal papers and state-of-the-art,

Stereo and rectification Warp images to simplify epipolar geometry Warp images to simplify epipolar geometry Warp images to simplify epipolar geometry

Compute correspondences for all pixels

Page 49: 3D Photography - CVG · • Cover basic 3D photography concepts and approaches. • Further lectures: • Short introduction to topic • Paper presentations (seminal papers and state-of-the-art,

Multi-View Stereo

Page 50: 3D Photography - CVG · • Cover basic 3D photography concepts and approaches. • Further lectures: • Short introduction to topic • Paper presentations (seminal papers and state-of-the-art,

Structured-light

•  Projector = camera •  Use specific patterns to obtain

correspondences

Page 51: 3D Photography - CVG · • Cover basic 3D photography concepts and approaches. • Further lectures: • Short introduction to topic • Paper presentations (seminal papers and state-of-the-art,

Initialize Motion (P1,P2 compatibel with F)

Initialize Structure (minimize reprojection error)

Extend motion (compute pose through matches seen in 2 or more previous views)

Extend structure (Initialize new structure, refine existing structure)

Structure from motion

Page 52: 3D Photography - CVG · • Cover basic 3D photography concepts and approaches. • Further lectures: • Short introduction to topic • Paper presentations (seminal papers and state-of-the-art,

Visual SLAM

•  Visual Simultaneous Navigation and Mapping

(Clipp et al. ICCV 09)

Page 53: 3D Photography - CVG · • Cover basic 3D photography concepts and approaches. • Further lectures: • Short introduction to topic • Paper presentations (seminal papers and state-of-the-art,

ω* Ω*

projection

constraints

Auto-calibration

Page 54: 3D Photography - CVG · • Cover basic 3D photography concepts and approaches. • Further lectures: • Short introduction to topic • Paper presentations (seminal papers and state-of-the-art,

Shape-from-Silhouette

Page 55: 3D Photography - CVG · • Cover basic 3D photography concepts and approaches. • Further lectures: • Short introduction to topic • Paper presentations (seminal papers and state-of-the-art,

Shape-from-X

Shape-from-focus

Shape-from-texture

time-of-flight

Page 56: 3D Photography - CVG · • Cover basic 3D photography concepts and approaches. • Further lectures: • Short introduction to topic • Paper presentations (seminal papers and state-of-the-art,

3D modeling and texturing

Multiple depth images Surface model

Texture integration

patchwork texture map

Page 57: 3D Photography - CVG · • Cover basic 3D photography concepts and approaches. • Further lectures: • Short introduction to topic • Paper presentations (seminal papers and state-of-the-art,

Papers and discussion

•  Will cover recent state of the art Each student will present a paper, discussion Select paper related to project

Page 58: 3D Photography - CVG · • Cover basic 3D photography concepts and approaches. • Further lectures: • Short introduction to topic • Paper presentations (seminal papers and state-of-the-art,

Course project: Build your own 3D scanner!

Example: Bouguet ICCV’98

Students can work in alone or in pairs

Page 59: 3D Photography - CVG · • Cover basic 3D photography concepts and approaches. • Further lectures: • Short introduction to topic • Paper presentations (seminal papers and state-of-the-art,

3D photography course team

Marc Pollefeys CNB G105 [email protected]

Tel. 044 63 23105

Gim Hee Lee CAB G 84.2 [email protected]

044 63 24 854

Kevin Köser CNB G104 [email protected]

Tel: 044 63 29 438