david luebke11-17-98 modeling and rendering architecture from photographs a hybrid geometry- and...

David Luebke 11-17-98

Modeling and Rendering Modeling and Rendering Architecture from PhotographsArchitecture from Photographs

A hybrid geometry- and image-based approachA hybrid geometry- and image-based approach

Debevec, Taylor, and MalikDebevec, Taylor, and Malik

SIGGRAPH 96SIGGRAPH 96Presented by David LuebkePresented by David Luebke


OverviewOverview

The Problem and the IdeaThe Problem and the Idea BackgroundBackground Model Representation and Model Representation and

ReconstructionReconstruction View-dependent Texture MappingView-dependent Texture Mapping Model-based StereoModel-based Stereo Conclusion and DiscussionConclusion and Discussion


The ProblemThe Problem

Architectural walkthroughs and flybys Architectural walkthroughs and flybys are an important applicationare an important application

Creating detailed models is Creating detailed models is hardhard– Start with blueprints (if they exist…)Start with blueprints (if they exist…)– Survey an existing buildingSurvey an existing building

Resulting systems don’t look greatResulting systems don’t look great– Hard to get all the detailsHard to get all the details– Hard to get realistic Hard to get realistic exteriorsexteriors


The IdeaThe Idea

Wanted: a system to generate realistic Wanted: a system to generate realistic architectural scenesarchitectural scenes

Idea: Model and render from Idea: Model and render from photos!photos!– Take a few widely spaced photographsTake a few widely spaced photographs– Build simple underlying model of sceneBuild simple underlying model of scene– Use correspondences between photos to Use correspondences between photos to

adjust scene parametersadjust scene parameters– Paste photos back onto simple geometry Paste photos back onto simple geometry

of scene for realistic façadeof scene for realistic façade


BackgroundBackground

Computer vision: Computer vision: recover 3D geometry recover 3D geometry from 2D imagesfrom 2D images

Debevec uses some CV concepts:Debevec uses some CV concepts:– Camera calibrationCamera calibration: simplify problem by : simplify problem by

finding exact pixel finding exact pixel ray mappings ray mappings– Structure from motionStructure from motion and and stereo stereo

correspondencecorrespondence: triangulating for depth: triangulating for depth– Image-based renderingImage-based rendering: given image & : given image &

depth map, re-render from other viewsdepth map, re-render from other views


Photogrammetric ModelingPhotogrammetric Modeling

Extracting 3D surfaces from multiple Extracting 3D surfaces from multiple images is images is hardhard

Constrain the problem:Constrain the problem:– User builds a simple notional model using User builds a simple notional model using

blocksblocks: primitive solid shapes : primitive solid shapes Example: boxes, wedges, prisms, frustaExample: boxes, wedges, prisms, frusta

– User marks correspondences between User marks correspondences between images and modelimages and model

– System fits model to imagesSystem fits model to images



Now system need only solve Now system need only solve parameters of blocks!parameters of blocks!– Height, width, translation, rotation, etc.Height, width, translation, rotation, etc.



Even better: build in architectural Even better: build in architectural constraints!constraints!– Roof prism lies flush on building blockRoof prism lies flush on building block– Stacked tower blocks share center axisStacked tower blocks share center axis



Knowns: image Knowns: image block edge block edge correspondences correspondences

Unknowns: block parameters, Unknowns: block parameters, camera position/orientationcamera position/orientation

Constraints reduce # unknownsConstraints reduce # unknowns Generally, # correspondences must Generally, # correspondences must

equal # unknowns for reconstructionequal # unknowns for reconstruction



Represent block parameters as Represent block parameters as instances of shared variablesinstances of shared variables

Lots of math…Lots of math…– Tweaking model edges to correspond Tweaking model edges to correspond

to recovered edgesto recovered edges– Computing an initial estimateComputing an initial estimate



Results:Results:


View-Dependent View-Dependent Texture MappingTexture Mapping

Given the model, treat each camera Given the model, treat each camera position as a “slide projector”position as a “slide projector”

Some images overlap!Some images overlap!– Idea: pick image taken from viewpoint Idea: pick image taken from viewpoint

closest to desired rendering viewpointclosest to desired rendering viewpoint– Better: use weighted average (Fig 12)Better: use weighted average (Fig 12)


View-Dependent View-Dependent Texture MappingTexture Mapping

Best: Do view-dependent texture Best: Do view-dependent texture mapping on per-pixel basismapping on per-pixel basis

Okay: Do it on a per-face basisOkay: Do it on a per-face basis– Subdivide large facesSubdivide large faces– Use texture hardware!Use texture hardware!


View-DependentView-DependentTexture MappingTexture Mapping


Model-Based StereoModel-Based Stereo

Problem: fine architectural details Problem: fine architectural details still not capturedstill not captured– recessed windows, friezes, cornicesrecessed windows, friezes, cornices

Stereo depth extraction can help!Stereo depth extraction can help!– Problem: when images are taken from Problem: when images are taken from

distant viewpoints, corresponding pixel distant viewpoints, corresponding pixel neighborhoods can look very differentneighborhoods can look very different


Model-Based StereoModel-Based Stereo

Key observation: Key observation: – Even though two images of the same Even though two images of the same

scene may look very different, they look scene may look very different, they look similar after being projected onto the similar after being projected onto the approximate modelapproximate model..

– Idea: Warp offset image by projecting onto Idea: Warp offset image by projecting onto the approximate model and re-renderingthe approximate model and re-rendering

– Use McMillian warp to render image-with-Use McMillian warp to render image-with-depth from novel viewpointsdepth from novel viewpoints


Conclusion and DiscussionConclusion and Discussion

Results speak for themselvesResults speak for themselves What problems do you see? What problems do you see?

david luebke11-17-98 modeling and rendering architecture from photographs a hybrid geometry- and...

Documents

david luebke11

images slide

model system

david luebke slide

reconstruction slide

stereo model

discussion slide

views slide