real-time 3d model acquisition
DESCRIPTION
Real-Time 3D Model Acquisition. Princeton University Stanford University. Szymon Rusinkiewicz Olaf Hall-Holt Marc Levoy. 3D Scanning. Possible Research Goals. Low noise Guaranteed high accuracy High speed Low cost Automatic operation No holes. 3D Model Acquisition Pipeline. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Real-Time 3D Model Acquisition](https://reader035.vdocument.in/reader035/viewer/2022062422/56813075550346895d965415/html5/thumbnails/1.jpg)
Real-Time 3D Model Real-Time 3D Model AcquisitionAcquisition
Szymon RusinkiewiczSzymon Rusinkiewicz
Olaf Hall-HoltOlaf Hall-Holt
Marc LevoyMarc Levoy
Princeton UniversityPrinceton University
Stanford UniversityStanford University
![Page 2: Real-Time 3D Model Acquisition](https://reader035.vdocument.in/reader035/viewer/2022062422/56813075550346895d965415/html5/thumbnails/2.jpg)
3D Scanning3D Scanning
![Page 3: Real-Time 3D Model Acquisition](https://reader035.vdocument.in/reader035/viewer/2022062422/56813075550346895d965415/html5/thumbnails/3.jpg)
Possible Research GoalsPossible Research Goals
• Low noiseLow noise
• Guaranteed high accuracyGuaranteed high accuracy
• High speedHigh speed
• Low costLow cost
• Automatic operationAutomatic operation
• No holesNo holes
![Page 4: Real-Time 3D Model Acquisition](https://reader035.vdocument.in/reader035/viewer/2022062422/56813075550346895d965415/html5/thumbnails/4.jpg)
3D Model Acquisition Pipeline3D Model Acquisition Pipeline
3D Scanner3D Scanner3D Scanner3D Scanner
![Page 5: Real-Time 3D Model Acquisition](https://reader035.vdocument.in/reader035/viewer/2022062422/56813075550346895d965415/html5/thumbnails/5.jpg)
3D Model Acquisition Pipeline3D Model Acquisition Pipeline
3D Scanner3D Scanner3D Scanner3D Scanner
View PlanningView PlanningView PlanningView Planning
![Page 6: Real-Time 3D Model Acquisition](https://reader035.vdocument.in/reader035/viewer/2022062422/56813075550346895d965415/html5/thumbnails/6.jpg)
3D Model Acquisition Pipeline3D Model Acquisition Pipeline
3D Scanner3D Scanner3D Scanner3D Scanner
AlignmentAlignmentAlignmentAlignmentView PlanningView PlanningView PlanningView Planning
![Page 7: Real-Time 3D Model Acquisition](https://reader035.vdocument.in/reader035/viewer/2022062422/56813075550346895d965415/html5/thumbnails/7.jpg)
3D Model Acquisition Pipeline3D Model Acquisition Pipeline
3D Scanner3D Scanner3D Scanner3D Scanner
AlignmentAlignmentAlignmentAlignment
MergingMergingMergingMerging
View PlanningView PlanningView PlanningView Planning
![Page 8: Real-Time 3D Model Acquisition](https://reader035.vdocument.in/reader035/viewer/2022062422/56813075550346895d965415/html5/thumbnails/8.jpg)
3D Model Acquisition Pipeline3D Model Acquisition Pipeline
3D Scanner3D Scanner3D Scanner3D Scanner
AlignmentAlignmentAlignmentAlignment
MergingMergingMergingMergingDone?Done?Done?Done?
View PlanningView PlanningView PlanningView Planning
![Page 9: Real-Time 3D Model Acquisition](https://reader035.vdocument.in/reader035/viewer/2022062422/56813075550346895d965415/html5/thumbnails/9.jpg)
3D Model Acquisition Pipeline3D Model Acquisition Pipeline
3D Scanner3D Scanner3D Scanner3D Scanner
AlignmentAlignmentAlignmentAlignment
MergingMergingMergingMergingDone?Done?Done?Done?
View PlanningView PlanningView PlanningView Planning
DisplayDisplayDisplayDisplay
![Page 10: Real-Time 3D Model Acquisition](https://reader035.vdocument.in/reader035/viewer/2022062422/56813075550346895d965415/html5/thumbnails/10.jpg)
3D Model Acquisition Difficulties3D Model Acquisition Difficulties
• Much (often most) time spent on “last Much (often most) time spent on “last 20%”20%”
• Pipeline not optimized for hole-fillingPipeline not optimized for hole-filling
• Not sufficient just to speed up scanner –Not sufficient just to speed up scanner –must design pipeline for fast feedbackmust design pipeline for fast feedback
![Page 11: Real-Time 3D Model Acquisition](https://reader035.vdocument.in/reader035/viewer/2022062422/56813075550346895d965415/html5/thumbnails/11.jpg)
Real-Time 3D Model AcquisitionReal-Time 3D Model Acquisition
![Page 12: Real-Time 3D Model Acquisition](https://reader035.vdocument.in/reader035/viewer/2022062422/56813075550346895d965415/html5/thumbnails/12.jpg)
Real-Time 3D Model Acquisition Real-Time 3D Model Acquisition PipelinePipeline
3D Scanner3D Scanner3D Scanner3D Scanner
AlignmentAlignmentAlignmentAlignment
MergingMergingMergingMergingDone?Done?Done?Done?
View PlanningView PlanningView PlanningView Planning
DisplayDisplayDisplayDisplay
HumanHuman
![Page 13: Real-Time 3D Model Acquisition](https://reader035.vdocument.in/reader035/viewer/2022062422/56813075550346895d965415/html5/thumbnails/13.jpg)
Real-Time 3D Model Acquisition Real-Time 3D Model Acquisition PipelinePipeline
3D Scanner3D Scanner3D Scanner3D Scanner
AlignmentAlignmentAlignmentAlignment
MergingMergingMergingMergingDone?Done?Done?Done?
View PlanningView PlanningView PlanningView Planning
DisplayDisplayDisplayDisplay
Challenge:Challenge:Real TimeReal Time
![Page 14: Real-Time 3D Model Acquisition](https://reader035.vdocument.in/reader035/viewer/2022062422/56813075550346895d965415/html5/thumbnails/14.jpg)
Real-Time 3D Model Acquisition Real-Time 3D Model Acquisition PipelinePipeline
3D Scanner3D Scanner3D Scanner3D Scanner
AlignmentAlignmentAlignmentAlignment
MergingMergingMergingMergingDone?Done?Done?Done?
View PlanningView PlanningView PlanningView Planning
DisplayDisplayDisplayDisplay
Part I:Part I:Structured-LightStructured-Light
TriangulationTriangulation
![Page 15: Real-Time 3D Model Acquisition](https://reader035.vdocument.in/reader035/viewer/2022062422/56813075550346895d965415/html5/thumbnails/15.jpg)
Real-Time 3D Model Acquisition Real-Time 3D Model Acquisition PipelinePipeline
3D Scanner3D Scanner3D Scanner3D Scanner
AlignmentAlignmentAlignmentAlignment
MergingMergingMergingMergingDone?Done?Done?Done?
View PlanningView PlanningView PlanningView Planning
DisplayDisplayDisplayDisplay
Part II:Part II:Fast ICPFast ICP
![Page 16: Real-Time 3D Model Acquisition](https://reader035.vdocument.in/reader035/viewer/2022062422/56813075550346895d965415/html5/thumbnails/16.jpg)
Real-Time 3D Model Acquisition Real-Time 3D Model Acquisition PipelinePipeline
3D Scanner3D Scanner3D Scanner3D Scanner
AlignmentAlignmentAlignmentAlignment
MergingMergingMergingMergingDone?Done?Done?Done?
View PlanningView PlanningView PlanningView Planning
DisplayDisplayDisplayDisplay
Part III:Part III:Voxel GridVoxel Grid
![Page 17: Real-Time 3D Model Acquisition](https://reader035.vdocument.in/reader035/viewer/2022062422/56813075550346895d965415/html5/thumbnails/17.jpg)
TriangulationTriangulation
• Project laser stripe onto objectProject laser stripe onto object
ObjectObject
LaserLaser
CameraCameraCameraCamera
![Page 18: Real-Time 3D Model Acquisition](https://reader035.vdocument.in/reader035/viewer/2022062422/56813075550346895d965415/html5/thumbnails/18.jpg)
CameraCameraCameraCamera
TriangulationTriangulation
• Depth from ray-plane triangulationDepth from ray-plane triangulation
LaserLaser
(x,y(x,y))
ObjectObject
![Page 19: Real-Time 3D Model Acquisition](https://reader035.vdocument.in/reader035/viewer/2022062422/56813075550346895d965415/html5/thumbnails/19.jpg)
TriangulationTriangulation
• Faster acquisition: project multiple Faster acquisition: project multiple stripesstripes
• Correspondence problem: which stripeCorrespondence problem: which stripeis which?is which?
![Page 20: Real-Time 3D Model Acquisition](https://reader035.vdocument.in/reader035/viewer/2022062422/56813075550346895d965415/html5/thumbnails/20.jpg)
Continuum of Triangulation Continuum of Triangulation MethodsMethods
Slow, robustSlow, robust Fast, fragileFast, fragile
Multi-stripeMulti-stripeMulti-frameMulti-frame
Single-frameSingle-frameSingle-stripeSingle-stripe
![Page 21: Real-Time 3D Model Acquisition](https://reader035.vdocument.in/reader035/viewer/2022062422/56813075550346895d965415/html5/thumbnails/21.jpg)
Time-Coded Light PatternsTime-Coded Light Patterns
• Assign each stripe a unique illumination Assign each stripe a unique illumination codecodeover time [Posdamer 82]over time [Posdamer 82]
SpaceSpace
TimeTime
![Page 22: Real-Time 3D Model Acquisition](https://reader035.vdocument.in/reader035/viewer/2022062422/56813075550346895d965415/html5/thumbnails/22.jpg)
Illumination history = (WB),(BW),(WB)Illumination history = (WB),(BW),(WB)Illumination history = (WB),(BW),(WB)Illumination history = (WB),(BW),(WB)
CodeCodeCodeCode
Codes for Moving ScenesCodes for Moving Scenes
• Assign time codesAssign time codesto stripe to stripe boundariesboundaries
• Perform frame-to-framePerform frame-to-frametracking of correspondingtracking of correspondingboundariesboundaries– Propagate illumination historyPropagate illumination history
[Hall-Holt & Rusinkiewicz, ICCV 2001][Hall-Holt & Rusinkiewicz, ICCV 2001]
![Page 23: Real-Time 3D Model Acquisition](https://reader035.vdocument.in/reader035/viewer/2022062422/56813075550346895d965415/html5/thumbnails/23.jpg)
Designing a CodeDesigning a Code
• Want many “features” to track:Want many “features” to track:lots of black/white edges at each framelots of black/white edges at each frame
• Try to minimize ghosts – WW or BB Try to minimize ghosts – WW or BB “boundaries” that can’t be seen directly“boundaries” that can’t be seen directly
![Page 24: Real-Time 3D Model Acquisition](https://reader035.vdocument.in/reader035/viewer/2022062422/56813075550346895d965415/html5/thumbnails/24.jpg)
Designing a CodeDesigning a Code
0011001111101110
10111011 01100110
01000100 10011001
0001000100010001 1100110011001100
00000000 11011101
10101010 01110111
10001000010101011111111111111111 0010001000100010
[Hall-Holt & Rusinkiewicz, ICCV 2001][Hall-Holt & Rusinkiewicz, ICCV 2001]
![Page 25: Real-Time 3D Model Acquisition](https://reader035.vdocument.in/reader035/viewer/2022062422/56813075550346895d965415/html5/thumbnails/25.jpg)
ImplementationImplementation
• Pipeline:Pipeline:
• DLP projector illuminates scene @ 60 DLP projector illuminates scene @ 60 Hz.Hz.
• Synchronized NTSC camera captures Synchronized NTSC camera captures videovideo
• Pipeline returns range images @ 60 Hz.Pipeline returns range images @ 60 Hz.
ProjectProjectCodeCode
ProjectProjectCodeCode
CaptureCaptureImagesImagesCaptureCaptureImagesImages
FindFindBoundariesBoundaries
FindFindBoundariesBoundaries
MatchMatchBoundariesBoundaries
MatchMatchBoundariesBoundaries DecodeDecodeDecodeDecode ComputeCompute
RangeRangeComputeCompute
RangeRange
![Page 26: Real-Time 3D Model Acquisition](https://reader035.vdocument.in/reader035/viewer/2022062422/56813075550346895d965415/html5/thumbnails/26.jpg)
Real-Time 3D Model Acquisition Real-Time 3D Model Acquisition PipelinePipeline
3D Scanner3D Scanner3D Scanner3D Scanner
AlignmentAlignmentAlignmentAlignment
MergingMergingMergingMergingDone?Done?Done?Done?
View PlanningView PlanningView PlanningView Planning
DisplayDisplayDisplayDisplay
Part II:Part II:Fast ICPFast ICP
![Page 27: Real-Time 3D Model Acquisition](https://reader035.vdocument.in/reader035/viewer/2022062422/56813075550346895d965415/html5/thumbnails/27.jpg)
Aligning 3D DataAligning 3D Data
• ICP (Iterative Closest Points): for each ICP (Iterative Closest Points): for each point on one scan, minimize distance to point on one scan, minimize distance to closest point on other scan…closest point on other scan…
![Page 28: Real-Time 3D Model Acquisition](https://reader035.vdocument.in/reader035/viewer/2022062422/56813075550346895d965415/html5/thumbnails/28.jpg)
Aligning 3D DataAligning 3D Data
• … … and iterate to find alignmentand iterate to find alignment– Iterated Closest Points (ICP) [Besl & McKay Iterated Closest Points (ICP) [Besl & McKay
92]92]
![Page 29: Real-Time 3D Model Acquisition](https://reader035.vdocument.in/reader035/viewer/2022062422/56813075550346895d965415/html5/thumbnails/29.jpg)
ICP in the Real-Time PipelineICP in the Real-Time Pipeline
• Potential problem with ICP: local minimaPotential problem with ICP: local minima– In this pipeline, scans close togetherIn this pipeline, scans close together
– Very likely to converge to correct (global) Very likely to converge to correct (global) minimumminimum
• Basic ICP algorithm too slow (~ seconds)Basic ICP algorithm too slow (~ seconds)– Point-to-plane minimizationPoint-to-plane minimization
– Projection-based matchingProjection-based matching
– With these tweaks, running time ~ millisecondsWith these tweaks, running time ~ milliseconds
[Rusinkiewicz & Levoy, 3DIM 2001][Rusinkiewicz & Levoy, 3DIM 2001]
![Page 30: Real-Time 3D Model Acquisition](https://reader035.vdocument.in/reader035/viewer/2022062422/56813075550346895d965415/html5/thumbnails/30.jpg)
Real-Time 3D Model Acquisition Real-Time 3D Model Acquisition PipelinePipeline
3D Scanner3D Scanner3D Scanner3D Scanner
AlignmentAlignmentAlignmentAlignment
MergingMergingMergingMergingDone?Done?Done?Done?
View PlanningView PlanningView PlanningView Planning
DisplayDisplayDisplayDisplay
Part III:Part III:Voxel GridVoxel Grid
![Page 31: Real-Time 3D Model Acquisition](https://reader035.vdocument.in/reader035/viewer/2022062422/56813075550346895d965415/html5/thumbnails/31.jpg)
Merging and RenderingMerging and Rendering
• Goal: visualize the model well enoughGoal: visualize the model well enoughto be able to see holesto be able to see holes
• Cannot display all the scanned data – Cannot display all the scanned data – accumulates linearly with timeaccumulates linearly with time
• Standard high-quality merging methods:Standard high-quality merging methods:processing time ~ 1 minute per scanprocessing time ~ 1 minute per scan
![Page 32: Real-Time 3D Model Acquisition](https://reader035.vdocument.in/reader035/viewer/2022062422/56813075550346895d965415/html5/thumbnails/32.jpg)
Merging and RenderingMerging and Rendering
![Page 33: Real-Time 3D Model Acquisition](https://reader035.vdocument.in/reader035/viewer/2022062422/56813075550346895d965415/html5/thumbnails/33.jpg)
Merging and RenderingMerging and Rendering
![Page 34: Real-Time 3D Model Acquisition](https://reader035.vdocument.in/reader035/viewer/2022062422/56813075550346895d965415/html5/thumbnails/34.jpg)
Merging and RenderingMerging and Rendering
![Page 35: Real-Time 3D Model Acquisition](https://reader035.vdocument.in/reader035/viewer/2022062422/56813075550346895d965415/html5/thumbnails/35.jpg)
Merging and RenderingMerging and Rendering
++
![Page 36: Real-Time 3D Model Acquisition](https://reader035.vdocument.in/reader035/viewer/2022062422/56813075550346895d965415/html5/thumbnails/36.jpg)
Merging and RenderingMerging and Rendering
• Point rendering, using accumulated normals for lightingPoint rendering, using accumulated normals for lighting
![Page 37: Real-Time 3D Model Acquisition](https://reader035.vdocument.in/reader035/viewer/2022062422/56813075550346895d965415/html5/thumbnails/37.jpg)
Example: PhotographExample: Photograph
18 cm.18 cm.
![Page 38: Real-Time 3D Model Acquisition](https://reader035.vdocument.in/reader035/viewer/2022062422/56813075550346895d965415/html5/thumbnails/38.jpg)
ResultResult
![Page 39: Real-Time 3D Model Acquisition](https://reader035.vdocument.in/reader035/viewer/2022062422/56813075550346895d965415/html5/thumbnails/39.jpg)
PostprocessingPostprocessing
• Real-time displayReal-time display– Quality/speed tradeoffQuality/speed tradeoff
– Goal: let user evaluate coverage, fill holesGoal: let user evaluate coverage, fill holes
• Offline postprocessing for high-quality Offline postprocessing for high-quality modelsmodels– Global registrationGlobal registration
– High-quality merging (e.g., using VRIP High-quality merging (e.g., using VRIP [Curless 96])[Curless 96])
![Page 40: Real-Time 3D Model Acquisition](https://reader035.vdocument.in/reader035/viewer/2022062422/56813075550346895d965415/html5/thumbnails/40.jpg)
Postprocessed ModelPostprocessed Model
![Page 41: Real-Time 3D Model Acquisition](https://reader035.vdocument.in/reader035/viewer/2022062422/56813075550346895d965415/html5/thumbnails/41.jpg)
Recapturing AlignmentRecapturing Alignment
![Page 42: Real-Time 3D Model Acquisition](https://reader035.vdocument.in/reader035/viewer/2022062422/56813075550346895d965415/html5/thumbnails/42.jpg)
SummarySummary
• 3D model acquisition pipeline optimized 3D model acquisition pipeline optimized for obtaining complete, hole-free modelsfor obtaining complete, hole-free models
• Use human’s time most efficientlyUse human’s time most efficiently
• Pieces of pipeline selected for real-time Pieces of pipeline selected for real-time use:use:– Structured-light scanner for moving objectsStructured-light scanner for moving objects
– Fast ICP variantFast ICP variant
– Simple grid-based merging, point renderingSimple grid-based merging, point rendering
![Page 43: Real-Time 3D Model Acquisition](https://reader035.vdocument.in/reader035/viewer/2022062422/56813075550346895d965415/html5/thumbnails/43.jpg)
LimitationsLimitations
• Prototype noisier than commercial systemsPrototype noisier than commercial systems– Could be made equivalent with careful engineeringCould be made equivalent with careful engineering
– Ultimate limitations on quality: focus, textureUltimate limitations on quality: focus, texture
• Scan-to-scan ICP not perfect Scan-to-scan ICP not perfect alignment drift alignment drift– Due to noise, miscalibration, degenerate geometryDue to noise, miscalibration, degenerate geometry
– Reduced, but not eliminated, by “anchor scans”Reduced, but not eliminated, by “anchor scans”
– Possibly combine ICP with separate trackersPossibly combine ICP with separate trackers
![Page 44: Real-Time 3D Model Acquisition](https://reader035.vdocument.in/reader035/viewer/2022062422/56813075550346895d965415/html5/thumbnails/44.jpg)
Future WorkFuture Work
• Faster scanningFaster scanning– Better stripe boundary trackingBetter stripe boundary tracking
– Multiple cameras, projectorsMultiple cameras, projectors
– High-speed cameras, projectorsHigh-speed cameras, projectors
• Application in different contextsApplication in different contexts– Cart- or shoulder-mounted for digitizing Cart- or shoulder-mounted for digitizing
roomsrooms
– Infrared for imperceptibilityInfrared for imperceptibility
![Page 45: Real-Time 3D Model Acquisition](https://reader035.vdocument.in/reader035/viewer/2022062422/56813075550346895d965415/html5/thumbnails/45.jpg)
AcknowledgmentsAcknowledgments
• Collaborators:Collaborators:– Li-Wei HeLi-Wei He
– James DavisJames Davis
– Lucas PereiraLucas Pereira
– Sean AndersonSean Anderson
• Sponsors:Sponsors:– SonySony
– IntelIntel
– IntervalInterval