ENGN2502 3D Photography Spring 2012

Gabriel Taubin Brown University

What do we mean by 3D Photography ?

• Techniques and systems using cameras and lights to capture the shape and appearance of 3D objects

• The geometry of triangulation • Surface representations and data structures • Methods to smooth, denoise, edit, compress, transmit,

simplify, and optimize very large polygonal models • Applications to computer animation, game development,

electronic commerce, heritage preservation, reverse engineering, medicine, virtual reality, etc.

• Project oriented • Goal: publication quality final projects • Instructor permission required

3D Shape Capture

Representation / Data Structures

Simplification / Compression

Smoothing / Parameterization

Remeshing / Segmentation

Interactive Modeling

Optimization / Resampling

Surface Reconstruction

Out-of-Core Algorithms

ENGN2502 : 3D Photography *Spring’2012+

• Medical Imaging – Visualization – Segmentation

Motivation

• Industry – Reverse engineering – Fast metrology – Physical simulations

• Entertainment

– Animating digital clays for movies or games

• Archeology and Art – Digitization of cultural

heritage and artistic works

Laser range scanning devices Multi-camera systems

3D Shape and Appearance Capture

Structured lighting systems

http://mesh.brown.edu/byo3d/

http://mesh.brown.edu/byo3d

Stereoscopic Photography

Screen shots of our real-time stereo system working on the field

Real-Time High-Definition Stereo on GPGPU using Progressive Multi-Resolution Adaptive Windows

Y. Zhao, and G. Taubin, Image and Vision Computing 2011.

¼ Resolution ½ Resolution Full Resolution¼ Resolution ½ Resolution Full Resolution

Coarse-to-fine matching on multiple resolutions

Stereo Frame Grabbing

Stereo Rectification

Multi-Resolution Pyramid Generating

Background Modeling With Shadow Removal Foreground Detection with Dilation & Erosion

Stereo Matching Using Adaptive Window with Cross-Checking

Disparity Refinement R

eso

luti

on

Sca

n f

rom

Lo

w t

o

Hig

h

Processing Pipeline

Real-Time High-Definition Stereo on GPGPU using Progressive Multi-Resolution Adaptive Windows

Y. Zhao, and G. Taubin, Image and Vision Computing 2011.

Time of Flight 3D Scanning

nssm

m

c

dt 17

103

0.528

Single Shot Structured Lighting: MS Kinect

3D Triangulation: Ray-Plane Intersection

ray

plane

intersection point

projector /

coordinate systems

Triangulation by Line-Plane Intersection

object being scanned

pqn

}0)(:{ pt qpnpP

projected light plane

p

illuminated point on object

}{ vqpL L

camera ray

Lq

v

intersection of light plane

with object

same

coordinate

system

Triangulation by Line-Line Intersection

object being scanned

}{ 2222 vqpL

camera ray

2q

2v

projected light ray

1q1v

}{ 1111 vqpL

p

lines may not intersect !

Triangulation and Scanning with Swept-Planes

Structured Lighting using Projector-Camera Systems

Gray Code Structured Lighting

3D Reconstruction using Structured Light [Inokuchi 1984] • Recover 3D depth for each pixel using ray-plane intersection • Determine correspondence between camera pixels and projector planes by

projecting a temporally-multiplexed binary image sequence • Each image is a bit-plane of the Gray code for each projector row/column

Point Grey Flea2 (15 Hz @ 1024 x 768)

Mitsubishi XD300U (50-85 Hz @ 1024 x 768)

Gray Code Structured Lighting

3D Reconstruction using Structured Light [Inokuchi 1984] • Recover 3D depth for each pixel using ray-plane intersection • Determine correspondence between camera pixels and projector planes by

projecting a temporally-multiplexed binary image sequence • Each image is a bit-plane of the Gray code for each projector row/column • Encoding algorithm: integer row/column index binary code Gray code

Point Grey Flea2 (15 Hz @ 1024 x 768)

Mitsubishi XD300U (50-85 Hz @ 1024 x 768)

Gray Code Structured Lighting Results

3D Photography in Android

Reconstruction Method

Surface Representation

Oriented Points

Positions & Normals Implicit Surface Polygon Mesh

Typical Surface Reconstruction Pipeline

SSD: Smooth Signed Distance Surface Reconstruction F. Calakli, G. Taubin, Computer Graphics Forum, 2011.

• A new mathematical formulation

• And a particular algorithm

• To reconstruct a watertight surface

From a static oriented point cloud

Particularly Good at Extrapolating Missing Data

http://mesh.brown.edu/ssd

3D Reconstruction & Analysis of Bat Flight Maneuvers

• 3D Reconstruction of Bat Flight Kinematics from Sparse Multiple Views, by A. Bergou, S. Swartz, and G. Taubin, and K. Breuer, 4DMOD, 2011.

• 3D Reconstruction and Analysis of Bat Flight Maneuvers from Sparse Multiple View Video, by A. Bergou, S. Swartz, S. K. Breuer, G. Taubin, BioVis, 2011.

• Falling with Style - The Role of Wing Inertia in Bat Flight Maneuvers, by A. Bergou, D. Riskin, G. Taubin, S. Swartz, and K. Breuer, Annual Meeting, Society for Integrative and Comparative Biology, 2011.

• Falling with Style-Bat Flight Maneuvers, by A. Bergou, D. Riskin, G. Taubin, S. Swartz, and K. Breuer, Bulletin of the American Physical Society, Vol. 55, 2010.

• Multiple synchronized 1000fps+ cameras

• Controlled environment (backdrop & illumination)

• Bats trained to land on landing pad

• Experiments with several species

How do we measure bats ?

• Bats have highly articulated wings • Very complex wing motion • Current goal: Detailed reconstruction of wing and body

kinematics and derivatives from visual data • Skeleton model with 52 degrees of freedom • Geometry parameterized by 37 constants • Future Goal: Model-less Dynamic Shape Reconstruction

Some Methods to Capture 3D Point Clouds

Shadow

Turntable

Backdrop

8 Megapixel Camera

Multi-Flash Attachment

Multi-Flash Camera

Beyond Silhouettes: Surface Reconstruction using Multi-Flash Photography

D. Crispell, D. Lanman, P. Sibley, Y. Zhao and G. Taubin [3DPVT 2006]

30

Shadow

Turntable

Backdrop

8 Megapixel Camera

Multi-Flash Attachment

Multi-Flash Camera

Multi-Flash 3D Photography:

Capturing the Shape and Appearance of 3D Objects

Turntable Rotation

A new approach for reconstructing 3D objects using shadows cast by depth discontinuities, as detected by a multi-flash camera. Unlike existing stereo vision algorithms, this method works even with plain surfaces, including unpainted ceramics and architecture.

Estimated Shape: 3D Point Cloud

Recovered Appearance: Phong BRDF Model

Multi-Flash Turntable Sequence: Input Image

Data Capture: A turntable and a digital camera are used to acquire data from 670 viewpoints. For each viewpoint, we capture a set of images using illumination from four different flashes. Future embodiments will include a small, inexpensive handheld multi-flash camera.

Recovering a Smooth Surface

The reconstructed point cloud can possess errors, including gaps and noise. To minimize these effects, we find an implicit surface which interpolates the 3D points. This method can be applied to any 3D point cloud, including those generated by laser scanners.

Raskar et al. [Siggraph 2004]

• Depth discontinuity estimation for Non-Photorealistic Rendering

– Camera static with respect to object

– 4 Images are captured with object

– Each illuminated by a different flash

– Flashes located close to camera lens

– Image processing extracts and combines shadows

What do we gain?

Using only silhouettes Using all depth discontinuities

Silhouettes vs Depth Discontinuities [2D]

Silhouette

Depth Discontinuity

Differential Shape From Silhouette [Cipolla & Blake 92]

Known from camera motion

Measured in epipolar slice

Measured in depth discontinuity image

Computing r’(t) : Depth Discontinuity Edge Tracking

1

2 3

4

1 2

3 4

VFIso Results [2006 110x110x110 grid]

40

Using the implicit surface, we can determine which points are visible

from each viewpoint. To model the material properties of the surface,

we fit a per-point Phong BRDF model to the set of visible reflectance

observations (using a total of 67 viewpoints).

Multi-Flash 3D Photography: Photometric Reconstruction

Multi-Flash Turntable Sequence

Images

Phong (Specular)

Phong (Diffuse) Estimated Phong Appearance Model

Diffuse Specular Ambient

…

3D Point

Cloud

Implicit Surface

Surround Structured Lighting: 3-D Scanning with Orthographic Illumination D. Lanman, D. Crispell, G. Taubin [CVIU 2009]

3D Slit Scanning with Planar Constraints M. Leotta, A. Vandergon, and G. Taubin [CGF 2008]

Camera 1

Camera 2 Laser pointer +

cylindrical lens

Catadioptric Stereo Implementation

Can Estimate Points Visible From One Camera

Surface Reconstruction from Multi-View Data

NSF Digital Archaeology Project

• The main goal is to automate the tedious processes of data collection and documentation at the excavation site, as well as to provide visualization tools to explore the data collected in the database

• Also to solve specific problems in Archaeology using computer vision techniques.

• We first used a network of cameras to capture the activity at the excavation site, to reconstruct the shape of the environment as it is being excavated, to reconstruct layers, and to locate finds in 3D

• Now we use multi-view stereo from photographs captured by a handheld digital camera

Semi-automatically Assemble Fragments Into Artifacts

Assisted Data Acquisition, Algorithmic Reconstruction, Integrated multi-format analysis

REVEAL Archaeological Data Acquisition

Objects: Artifacts

Excavations Areas Sites

Data Acquisition

Data: Text Photos Video 3D Models

Automatically Convert Photos to 3D Models

Site Plans

Import photos, videos, and laser scans and connect them to database objects

Improve speed and accuracy with computer assisted data entry

Advanced Algorithms

Import External Data

Laser Scanned Models

Rule-based Reconstructions

REVEAL Database

Typical Activity Sequence

Data integrated and synchronized in tabular, plan drawing, 3D spatial, image, and video formats

REVEAL Archaeological Analysis

Select Artifacts on Site Plan

Artifacts Excavations

Areas Sites

Display Photos of Selected Artifacts

Examine Relationship of Artifacts in-situ in auto-generated 3D Excavation Model

Export Formatted Artifact Data for inclusion in Site Publication

[Furukawa and Ponce 2008]

[Snavely et. al. 2006]

MVS software

Patch-based Multi-View Stereo (PMVS) http://grail.cs.washington.edu/software/pmvs/

http://phototour.cs.washington.edu/bundler/

Accurate 3D Footwear Impression Recovery From Photographs, F. A. Andalo, F. Calakli, G. Taubin, and S. Goldenstein,

International Conference on Imaging for Crime Detection and Prevention (ICDP-2011).

Comparable to 3D Laser Scanner

Challenges

Uniform sampling

Non-uniform sampling

Noisy data

Misaligned scans

From Multi-View Video Cameras

View Interpolation From Multi-View Video Cameras

View Interpolation From Multi-View Video Cameras

View Interpolation From Multi-View Video Cameras

View Interpolation From Multi-View Video Cameras

With Background Segmentation

Ongoing work

ENGN2502 3D Photography Spring 2012

I hope to see you in class !