real-time acquisition and rendering of large 3d models szymon rusinkiewicz

Real-time Acquisition and Real-time Acquisition and Rendering of Large 3D Rendering of Large 3D

ModelsModels

Szymon RusinkiewiczSzymon Rusinkiewicz

Computer Graphics PipelineComputer Graphics Pipeline

• Human time = expensiveHuman time = expensive• Sensors = cheapSensors = cheap

– Computer graphics increasingly relies onComputer graphics increasingly relies onmeasurements of the real worldmeasurements of the real world

RenderingRenderingRenderingRendering

ShapeShapeShapeShape

LightingLightingandand

ReflectanceReflectance

LightingLightingandand

ReflectanceReflectance

MotionMotionMotionMotion

ShapeShape 3D Scanning3D Scanning

3D Scanning Applications3D Scanning Applications

• Computer Computer graphicsgraphics

• Product Product inspectioninspection

• Robot Robot navigationnavigation

• As-built As-built floorplansfloorplans

• Product designProduct design

• ArchaeologyArchaeology

• Clothes fittingClothes fitting

• Art historyArt history

The Digital Michelangelo The Digital Michelangelo ProjectProject

• Push state of the art in range Push state of the art in range scanning and demonstrate scanning and demonstrate applications in art and art historyapplications in art and art history

Working inWorking inthe museumthe museum

ScanningScanninggeometrygeometry

ScanningScanningcolorcolor

Traditional Range Scanning Traditional Range Scanning PipelinePipeline

• High-quality, robust pipeline for High-quality, robust pipeline for producing 3D models:producing 3D models:– ScanScan object with laser triangulation object with laser triangulation

scanner: many views from different anglesscanner: many views from different angles

– AlignAlign pieces into single coordinate frame: pieces into single coordinate frame:initial manual alignment, refined with ICPinitial manual alignment, refined with ICP

– MergeMerge overlapping regions: compute overlapping regions: compute “average” surface using VRIP [Curless & “average” surface using VRIP [Curless & Levoy 96]Levoy 96]

– DisplayDisplay resulting model resulting model

3D Scan of David: 3D Scan of David: StatisticsStatistics

• Over 5 meters tallOver 5 meters tall• 1/4 mm resolution1/4 mm resolution• 22 people22 people• 30 nights of scanning30 nights of scanning• Efficiency max : min = 8 : 1Efficiency max : min = 8 : 1

– Needed view planningNeeded view planning

• Weight of gantry: 800 kgWeight of gantry: 800 kg• Putting model together:Putting model together:

1000+ man-hours and 1000+ man-hours and countingcounting

New 3D Scanning PipelineNew 3D Scanning Pipeline

• Need for a fast, inexpensive,Need for a fast, inexpensive,easy-to-use 3D scanning systemeasy-to-use 3D scanning system

• Wave a (small, rigid) Wave a (small, rigid) object by hand in front of object by hand in front of the scannerthe scanner

• Automatically align data Automatically align data asasit is acquiredit is acquired

• Let user see partial Let user see partial model as itmodel as itis being built – fill holesis being built – fill holes

Real-Time Model AcquisitionReal-Time Model AcquisitionReal-Time Model AcquisitionReal-Time Model Acquisition

Real-Time 3D Model Real-Time 3D Model AcquisitionAcquisition

• Prototype real-time model Prototype real-time model acquisition systemacquisition system– 3D scanning of moving objects3D scanning of moving objects

– Fast alignmentFast alignment

– Real-time merging and displayReal-time merging and display

Applications of Easy-to-UseApplications of Easy-to-Use3D Model Acquisition3D Model Acquisition

• AdvertisingAdvertising

• More capabilities in PhotoshopMore capabilities in Photoshop

• Movie setsMovie sets

• Augmented realityAugmented reality

• User interfacesUser interfaces

3D Scanning Technologies3D Scanning Technologies

• Contact-based:Contact-based: touch probes touch probes• Passive:Passive: shape from stereo, motion, shape from stereo, motion,

shadingshading• Active:Active: time-of-flight, defocus, time-of-flight, defocus,

photometric stereo, triangulationphotometric stereo, triangulation• Triangulation systems are Triangulation systems are

inexpensive, robust, and flexibleinexpensive, robust, and flexible– Take advantage of trends in DLP Take advantage of trends in DLP

projectorsprojectors

Laser TriangulationLaser Triangulation

• Project laser stripe onto objectProject laser stripe onto object

ObjectObject

LaserLaser

CameraCameraCameraCamera

CameraCameraCameraCamera

Laser TriangulationLaser Triangulation

• Depth from ray-plane triangulationDepth from ray-plane triangulation

ObjectObject

LaserLaser

(x,y)(x,y)

TriangulationTriangulation

• Faster acquisition: project Faster acquisition: project multiple stripesmultiple stripes

• Correspondence problem: which Correspondence problem: which stripestripeis which?is which?

TriangulationTriangulation

Slow, robust Fast, fragile

Multi-stripeMulti-frame

Single-frameSingle-stripe

Time-Coded Light PatternsTime-Coded Light Patterns

• Assign each stripe a unique Assign each stripe a unique illumination codeillumination codeover time [Posdamer 82]over time [Posdamer 82]

SpaceSpace

TimeTime

Gray-Code PatternsGray-Code Patterns

• To minimize effects of quantization To minimize effects of quantization error:error:each point may be a boundary only onceeach point may be a boundary only once

SpaceSpace

TimeTime

Structured-Light Structured-Light AssumptionsAssumptions

• Structured-light systems make certain Structured-light systems make certain assumptions about the scene:assumptions about the scene:

• SpatialSpatial continuity assumption: continuity assumption:– Assume scene is one objectAssume scene is one object

– Project a grid, pattern of dots, etc.Project a grid, pattern of dots, etc.

• TemporalTemporal continuity assumption: continuity assumption:– Assume scene is staticAssume scene is static

– Assign stripes a code over timeAssign stripes a code over time

Codes for Moving ScenesCodes for Moving Scenes

• We make a different assumption:We make a different assumption:– Object may moveObject may move

– Velocity low enough to permit Velocity low enough to permit trackingtracking

– ““Spatio-temporal” continuitySpatio-temporal” continuity

Illumination history = (WB),(BW),(WB)Illumination history = (WB),(BW),(WB)Illumination history = (WB),(BW),(WB)Illumination history = (WB),(BW),(WB)

CodeCodeCodeCode

Codes for Moving ScenesCodes for Moving Scenes

• Code stripe Code stripe boundariesboundariesinstead of stripesinstead of stripes

• Perform frame-to-framePerform frame-to-frametracking of correspondingtracking of correspondingboundariesboundaries– Propagate illumination Propagate illumination

historyhistory

[Hall-Holt & Rusinkiewicz, [Hall-Holt & Rusinkiewicz, ICCV 2001]ICCV 2001]

New Scanning PipelineNew Scanning Pipeline

ProjectProjectCodeCode


CaptureCaptureImagesImagesCaptureCaptureImagesImages

FindFindBoundariesBoundaries


MatchMatchBoundariesBoundaries

MatchMatchBoundariesBoundaries DecodeDecodeDecodeDecode ComputeCompute

RangeRangeComputeCompute

RangeRange

Designing a CodeDesigning a Code

• Biggest problem is ghosts – WW or Biggest problem is ghosts – WW or BB “boundaries” that can’t be BB “boundaries” that can’t be seen directlyseen directly









RangeRange


• Design a code to make tracking Design a code to make tracking possible:possible:– Do not allow two Do not allow two spatiallyspatially adjacent adjacent

ghostsghosts

– Do not allow two Do not allow two temporallytemporally adjacent adjacent ghostsghosts tt


• Graph (for 4 frames):Graph (for 4 frames):

– Edges: boundaries (over time)Edges: boundaries (over time)

– Nodes: stripes (over time)Nodes: stripes (over time)

0011001111101110

10111011 01100110

01000100 10011001

00010001 11001100

00000000 11011101

10101010 01110111

100010000101010111111111 00100010

SpaceSpace

TimeTime


• Graph (for 4 frames):Graph (for 4 frames):

• Path with alternating colors:Path with alternating colors:55 edges in graph 55 edges in graph maximal-length traversal has 110 boundaries (111 maximal-length traversal has 110 boundaries (111 stripes)stripes)

– Edges: boundaries (over time)Edges: boundaries (over time)

Boundary visible at even timesBoundary visible at even times

Boundary visible at odd timesBoundary visible at odd times

– Nodes: stripes (over time)Nodes: stripes (over time)

0011001111101110

10111011 01100110

01000100 10011001

00010001 11001100

00000000 11011101

10101010 01110111

100010000101010111111111 00100010

Image CaptureImage Capture

• Standard video camera: fields at Standard video camera: fields at 60 Hz60 Hz

• Genlock camera to projectorGenlock camera to projector









RangeRange

Finding BoundariesFinding Boundaries









RangeRange

• Standard edge detection problemStandard edge detection problem

• Current solution: find minima and Current solution: find minima and maxima of intensity, boundary is maxima of intensity, boundary is between thembetween them

Matching Stripe BoundariesMatching Stripe Boundaries









RangeRange

• Even if number of ghosts isEven if number of ghosts isminimized, matching is not easyminimized, matching is not easy

??

Matching Stripe BoundariesMatching Stripe Boundaries

• Resolve ambiguity by constraining Resolve ambiguity by constraining maximum stripe velocitymaximum stripe velocity

• Could accommodate higher speeds Could accommodate higher speeds by estimating velocitiesby estimating velocities

• Could take advantage of methods Could take advantage of methods inintracking literature (e.g., Kalman tracking literature (e.g., Kalman filters)filters)

Decoding BoundariesDecoding Boundaries









RangeRange

• Propagate illumination historyPropagate illumination history

• Table lookup based on illumination Table lookup based on illumination history and position in four-frame history and position in four-frame sequencesequence– Once a stripe has been tracked for at least Once a stripe has been tracked for at least

four frames,four frames,it contributes useful data on it contributes useful data on everyevery subsequent framesubsequent frame

Computing 3D PositionComputing 3D Position

• Ray-plane intersectionRay-plane intersection

• Requires calibration of:Requires calibration of:– Camera, projector intrinsicsCamera, projector intrinsics

– Relative position and orientationRelative position and orientation









RangeRange

ResultsResults

VideoVideoframesframes

StripeStripeboundariesboundaries

unknowunknownnknownknownghostsghosts

ResultsResults

• Single range image of Single range image of movingmoving objectobject

Gray codes, no trackingGray codes, no tracking Boundary codes and trackingBoundary codes and tracking

Top ViewTop View Top ViewTop View Front ViewFront ViewFront ViewFront View

Aligning 3D DataAligning 3D Data

• This range scanner can be used This range scanner can be used for any moving objectsfor any moving objects

• For For rigid objectsrigid objects, range images , range images can be aligned to each other as can be aligned to each other as object movesobject moves


• If correct correspondences are If correct correspondences are known,known,it is possible to find correct it is possible to find correct relative rotation/translationrelative rotation/translation


• How to find corresponding points?How to find corresponding points?

• Previous systems based on user Previous systems based on user input,input,feature matching, surface feature matching, surface signatures, etc.signatures, etc.


• Alternative: assume Alternative: assume closestclosest points points correspond to each other, correspond to each other, compute the best transform…compute the best transform…


• … … and iterate to find alignmentand iterate to find alignment– Iterated Closest Points (ICP) [Besl & Iterated Closest Points (ICP) [Besl &

McKay 92]McKay 92]

• Converges if starting position Converges if starting position “close enough““close enough“

ICP VariantsICP Variants

• Classic ICP algorithm not real-timeClassic ICP algorithm not real-time• To improve speed: examine stages of ICP and evaluate proposed variantsTo improve speed: examine stages of ICP and evaluate proposed variants

[Rusinkiewicz & Levoy, 3DIM 2001][Rusinkiewicz & Levoy, 3DIM 2001]1.1. SelectingSelecting source points (from one or both meshes) source points (from one or both meshes)2.2. MatchingMatching to points in the other mesh to points in the other mesh3.3. WeightingWeighting the correspondences the correspondences4.4. RejectingRejecting certain (outlier) point pairs certain (outlier) point pairs5.5. Assigning an Assigning an error metricerror metric to the current transform to the current transform6.6. MinimizingMinimizing the error metric the error metric

ICP Variant – ICP Variant – Point-to-Plane Error MetricPoint-to-Plane Error Metric

• Using point-to-plane distance Using point-to-plane distance instead of point-to-point lets flat instead of point-to-point lets flat regions slide along each other regions slide along each other more easily [Chen & Medioni 91]more easily [Chen & Medioni 91]

Finding Corresponding Finding Corresponding PointsPoints

• Finding closest point is most expensive Finding closest point is most expensive stage of ICPstage of ICP– Brute force search – O(n)Brute force search – O(n)

– Spatial data structure (e.g., k-d tree) – Spatial data structure (e.g., k-d tree) – O(log n)O(log n)

– Voxel grid – O(1), but large constant, slow Voxel grid – O(1), but large constant, slow preprocessingpreprocessing

Finding Corresponding Finding Corresponding PointsPoints

• For range images, simply project point For range images, simply project point [Blais 95][Blais 95]– Constant-time, fastConstant-time, fast

– Does not require precomputing a spatial data Does not require precomputing a spatial data structurestructure

High-Speed ICP AlgorithmHigh-Speed ICP Algorithm

• ICP algorithm with projection-ICP algorithm with projection-based correspondences, point-to-based correspondences, point-to-plane matchingplane matchingcan align meshes in a few tens of can align meshes in a few tens of ms.ms.(cf. over 1 sec. with closest-point)(cf. over 1 sec. with closest-point)

Anchor ScansAnchor Scans

• Alignment of consecutive scans Alignment of consecutive scans leads to accumulation of ICP leads to accumulation of ICP errorserrors

• Alternative: align all scans to an Alternative: align all scans to an “anchor” scan, only switch anchor “anchor” scan, only switch anchor when overlap lowwhen overlap low

• Given anchor scans, restart after Given anchor scans, restart after failed ICP becomes easierfailed ICP becomes easier

Merging and RenderingMerging and Rendering

• Goal: visualize the model well enoughGoal: visualize the model well enoughto be able to see holesto be able to see holes

• Cannot display all the scanned data – Cannot display all the scanned data – accumulates linearly with timeaccumulates linearly with time

• Standard high-quality merging Standard high-quality merging methods:methods:processing time ~ 1 minute per scanprocessing time ~ 1 minute per scan

Merging and RenderingMerging and Rendering

• Real-time incremental merging Real-time incremental merging and rendering:and rendering:– Quantize samples to a 3D gridQuantize samples to a 3D grid

– Maintain average normal of all pointsMaintain average normal of all pointsat a grid cellat a grid cell

– Point (splat) renderingPoint (splat) rendering

– Can be made hierarchical to conserve Can be made hierarchical to conserve memorymemory

PhotographPhotograph

Real-time Scanning DemoReal-time Scanning Demo

PostprocessingPostprocessing

• Goal of real-time display is to let Goal of real-time display is to let user evaluate coverage, fill holesuser evaluate coverage, fill holes– Quality/speed tradeoff Quality/speed tradeoff

• Offline postprocessing for high-Offline postprocessing for high-quality modelsquality models

Merged ResultMerged Result

PhotographPhotograph Aligned scansAligned scans

MergedMerged

Future WorkFuture Work

• Technological improvements:Technological improvements:– Use full resolution of projectorUse full resolution of projector– Higher-resolution camerasHigher-resolution cameras– Ideas from design of single-stripe 3D Ideas from design of single-stripe 3D

scannersscanners

• Pipeline improvements:Pipeline improvements:– Better detection of failed alignmentBetter detection of failed alignment– Better handling of object texture – combine Better handling of object texture – combine

with stereo?with stereo?– Global registration to eliminate driftGlobal registration to eliminate drift– More sophisticated mergingMore sophisticated merging– Improve user interaction during scanningImprove user interaction during scanning


• Faster scanningFaster scanning– Better stripe boundary matchingBetter stripe boundary matching

– Multiple cameras, projectorsMultiple cameras, projectors

– High-speed camerasHigh-speed cameras

• Application in different contextsApplication in different contexts– Small, hand-heldSmall, hand-held

– Cart- or shoulder-mounted for digitizing Cart- or shoulder-mounted for digitizing roomsrooms

– Infrared for imperceptibilityInfrared for imperceptibility

Rendering of Large ModelsRendering of Large Models

• Range scanners increasingly capable of Range scanners increasingly capable of producing very large modelsproducing very large models– DMich models are 100 million to 1 billion samplesDMich models are 100 million to 1 billion samples

• Challenge:Challenge: how to allow viewing in real time how to allow viewing in real time– Fast startup, progressive loadingFast startup, progressive loading

• Traditional answer:Traditional answer: triangle meshes, triangle meshes, simplification, hardware-accelerated simplification, hardware-accelerated renderingrendering– Impractical for such large modelsImpractical for such large models

• Alternative:Alternative: revisit basic data structure revisit basic data structure

QSplat [Rusinkiewicz & Levoy, SIGGRAPH 00]QSplat [Rusinkiewicz & Levoy, SIGGRAPH 00]

QSplatQSplat

• Key observation: a single Key observation: a single bounding sphere hierarchy can be bounding sphere hierarchy can be used forused for– Hierarchical frustum and backface Hierarchical frustum and backface

cullingculling

– Level of detail controlLevel of detail control

– Splat rendering [Westover 89]Splat rendering [Westover 89]

QSplat Node StructureQSplat Node Structure

PositionPositionandand

RadiusRadius

TreeTreeStructureStructure NormalNormal

Width ofWidth ofCone ofCone ofNormalsNormals

ColorColor(Optional)(Optional)

13 bits13 bits 3 bits3 bits 14 bits14 bits 2 bits2 bits 16 bits16 bits

6 bytes6 bytes


• Position and radius encoded Position and radius encoded relative to parent noderelative to parent node– Hierarchical coding vs. delta Hierarchical coding vs. delta

coding along a path for vertex coding along a path for vertex positionspositions


RadiusRadius





Center OffsetCenter Offset

Radius RatioRadius Ratio



RadiusRadius





UncompressedUncompressed



RadiusRadius





Delta CodingDelta Coding[Deering 96][Deering 96]



RadiusRadius





HierarchicalHierarchicalCodingCoding

QSplat Rendering AlgorithmQSplat Rendering Algorithm

• Traverse hierarchy recursivelyTraverse hierarchy recursively

if if (node not visible)(node not visible)Skip this branchSkip this branch

else ifelse if (leaf node) (leaf node)Draw a splatDraw a splat

else ifelse if (size on screen < (size on screen < threshold)threshold)

Draw a splatDraw a splat

elseelseTraverse childrenTraverse children

if if (node not visible)(node not visible)Skip this branchSkip this branch

else ifelse if (leaf node) (leaf node)Draw a splatDraw a splat

else ifelse if (size on screen < (size on screen < threshold)threshold)

Draw a splatDraw a splat

elseelseTraverse childrenTraverse children

Hierarchical frustum /Hierarchical frustum /backface cullingbackface culling

Point renderingPoint rendering

Adjusted to maintainAdjusted to maintaindesired frame ratedesired frame rate

Level of detailLevel of detailcontrolcontrol

Demo – St. MatthewDemo – St. Matthew

• 3D scan of 2.7 meter 3D scan of 2.7 meter statue at 0.25 mmstatue at 0.25 mm

• 102,868,637 points102,868,637 points

• File size: 644 MBFile size: 644 MB

• Preprocessing time: Preprocessing time: 1 hour1 hour


• Splats as primitiveSplats as primitive– Unify rendering of meshes, volumes, point Unify rendering of meshes, volumes, point

cloudsclouds

– Compatible with shading after rasterizationCompatible with shading after rasterization

– Hybrid point/polygon systemsHybrid point/polygon systems

• High-level visibility / LOD frameworksHigh-level visibility / LOD frameworks– Store different kinds of data at each node: Store different kinds of data at each node:

alpha, BRDF, scattering function, etc. alpha, BRDF, scattering function, etc.

– Potentially could be used to unify image-Potentially could be used to unify image-based-rendering (IBR) techniquesbased-rendering (IBR) techniques

ContributionsContributions

• Real-time 3D model acquisition systemReal-time 3D model acquisition system– Video-rate 3D scanner for moving objectsVideo-rate 3D scanner for moving objects

– Analysis of ICP variants; real-time algorithmAnalysis of ICP variants; real-time algorithm

– Real-time merging and renderingReal-time merging and rendering

– Allows user to see model and fill holesAllows user to see model and fill holes

• QSplat: interactive rendering of large 3D QSplat: interactive rendering of large 3D meshesmeshes– Single data structure used for visibility culling,Single data structure used for visibility culling,

level-of-detail control, point rendering, level-of-detail control, point rendering, compressioncompression

– Extension to network streaming [I3D 2001]Extension to network streaming [I3D 2001]

AcknowledgmentsAcknowledgments

• Olaf Hall-HoltOlaf Hall-Holt• Lucas PereiraLucas Pereira• The Original DMich Gang: Dave Koller, Sean The Original DMich Gang: Dave Koller, Sean

Anderson, James Davis, Kari Pulli, Matt Ginzton, Jon Anderson, James Davis, Kari Pulli, Matt Ginzton, Jon ShadeShade

• DMich, the next generation: Gary King, Steve DMich, the next generation: Gary King, Steve MarschnerMarschner

• Graphics labGraphics lab• Advisor: Marc LevoyAdvisor: Marc Levoy• Committee: Pat Hanrahan, Leo Guibas, Mark Committee: Pat Hanrahan, Leo Guibas, Mark

Horowitz, Bernd GirodHorowitz, Bernd Girod• Family, friendsFamily, friends• Sponsors: NSF, Interval, Honda, Sony, IntelSponsors: NSF, Interval, Honda, Sony, Intel

real-time acquisition and rendering of large 3d models szymon rusinkiewicz

Documents