3d photography - cvg · • cover basic 3d photography concepts and approaches. • further...
TRANSCRIPT
3D Photography
Kevin Köser and Marc Pollefeys
Spring 2011
3D Photography
• Understanding geometric relations between images and the 3D world, as well as between images
• Obtaining 3D information from our 3D world
Motivation
• Applications in many different areas
• A few examples …
Architecture
Survey Stability analysis Plan renovations
Architecture
Survey Stability analysis Plan renovations
Interactive 3D Modeling
(Sinha et al. Siggraph Asia 08) collaboration with Microsoft Research (and licensed to MS)
3D urban modeling
UNC/UKY UrbanScape project
Virtual Tourism
Clothing
• Scan a person, custom-fit clothing
Industrial inspection
• Verify specifications • Compare measured model with CAD
Scanning industrial sites
as-build 3D model of off-shore oil platform
Robot navigation
small tethered rover
pan/tilt stereo head
ESA project our task: Calibration + Terrain modelling + Visualization
Cultural heritage Virtual Monticello
Allow virtual visits
Cultural heritage
Stanford s Digital Michelangelo
Digital archive Art historic studies
Archaeology
accuracy ~1/500 from DV video (i.e. 140kb jpegs 576x720)
Record different excavation layers
Archaeology
Generate & verify construction hypothesis
Layer 1 Layer 2
Generate 4D excavation record
Forensics
• Crime scene recording and analysis
Forensics
Surgery - simulation
• simulate results of surgery • allows preoperative planning
Surgery - teaching
Capture models of surgery for interactive learning
Surveillance
Surveillance
Computer games Content capture
HCI
Raw Kinect output: Color + Depth
Computational 3D Photography 26
http://grouplab.cpsc.ucalgary.ca/cookbook/index.php/Technologies/Kinect
Bodytracking
Computer Vision, Kinect and beyond 27
�
� � � �
� � � � � � � � � � � � � �
Randomized forest (Shotton et al.)
based on simple depth difference tests at each node
Human-machine interface
28
3D Video with Kinect
29
3D free-viewpoint video with Kinect
30
(Kuster 2011, under review)
Intended application: 3D tele-presence (BeingThere UNC-ETH-NTU center)
3D free-viewpoint video with Kinect
31
(Kuster 2011, under review)
Autonomous micro-helicopter navigation
32 Use Kinect to map out obstacles and avoid collisions
Motion capture
• Facial capture
• Full body motion
Sport analysis
Course objectives
• To understand the concepts that relate images to the 3D world and images to other images
• Explore the state of the art in 3D photography
• Implement a 3D photography system/algorithm
Material
Slides and more
http://www.cvg.ethz.ch/teaching/2011spring/3dphoto/ Also check out on-line “shape-from-video” tutorial:
http://www.cs.unc.edu/~marc/tutorial.pdf http://www.cs.unc.edu/~marc/tutorial/
Other interesting stuff: • Book by Hartley & Zisserman, Multiple View Geometry
• Peter Allen’s 3D photography class webpage http://www1.cs.columbia.edu/~allen/S10/
Learning approach
• Introductory lectures: • Cover basic 3D photography concepts and approaches.
• Further lectures: • Short introduction to topic • Paper presentations
(seminal papers and state-of-the-art, related to projects)
• 3D photography project: • Choose topic, define scope (by week 4) • Implement algorithm/system • Presentation/demo and paper report
Grade distribution • Paper presentation & discussions: 20% • 3D photography project: 80%
Content
• camera model and calibration • single-view metrology • triangulation • epipolar geometry, stereo and rectification • multi-view stereo • structured-light, active techniques • feature tracking and matching • structure-from-motion and V-SLAM • shape-from-silhouettes • …
3D photography course schedule (tentative)
Topic
Feb 21 Introduction + Geometry and Camera Model
Feb 28 Lecture: Camera calibration + Epipolar Geometry
Mar 7 Lecture: Active Ranging, Structured Light, Kinect
Mar 14 Project Proposals
Mar 21 Lecture: Features, Tracking/Matching
Mar 28 Topic-intro + 2 papers (2 teams of 2 students)
Apr 4 Topic-intro + 2 papers
Apr 11 Topic-intro +2 papers
Apr 18 Project Updates
Apr. 25 --- Easter ---
May 2 Topic-intro + 2 papers
May 9 Topic-intro + 2 papers
May 16 Topic-intro + 2 papers
May 23 Topic-intro + 2 papers
May 30 Final Project presentation/demos
Fast Forward!
• Quick overview of what is coming…
Camera models and geometry Pinhole camera
Geometric transformations in 2D and 3D
or
Camera Calibration
• Know 2D/3D correspondences, compute projection matrix
also radial distortion (non-linear)
Single View Metrology
Single View Metrology
Antonio Criminisi
Feature tracking and matching
Harris corners, KLT features, SIFT features key concepts: invariance of extraction, descriptors to viewpoint, exposure and illumination changes
l2
3D from images
C1 m1 M? L1
m2
L2 M
C2
Triangulation - calibration - correspondences
Epipolar Geometry
Fundamental matrix Essential matrix
Also how to robustly compute from images
Stereo and rectification Warp images to simplify epipolar geometry Warp images to simplify epipolar geometry Warp images to simplify epipolar geometry
Compute correspondences for all pixels
Multi-View Stereo
Structured-light
• Projector = camera • Use specific patterns to obtain
correspondences
Initialize Motion (P1,P2 compatibel with F)
Initialize Structure (minimize reprojection error)
Extend motion (compute pose through matches seen in 2 or more previous views)
Extend structure (Initialize new structure, refine existing structure)
Structure from motion
Visual SLAM
• Visual Simultaneous Navigation and Mapping
(Clipp et al. ICCV 09)
ω* Ω*
projection
constraints
Auto-calibration
Shape-from-Silhouette
Shape-from-X
Shape-from-focus
Shape-from-texture
time-of-flight
3D modeling and texturing
Multiple depth images Surface model
Texture integration
patchwork texture map
Papers and discussion
• Will cover recent state of the art Each student will present a paper, discussion Select paper related to project
Course project: Build your own 3D scanner!
Example: Bouguet ICCV’98
Students can work in alone or in pairs
3D photography course team
Marc Pollefeys CNB G105 [email protected]
Tel. 044 63 23105
Gim Hee Lee CAB G 84.2 [email protected]
044 63 24 854
Kevin Köser CNB G104 [email protected]
Tel: 044 63 29 438