image-based rendering to accelerate interactive walkthroughs

62
Image-Based Rendering Image-Based Rendering to Accelerate to Accelerate Interactive Interactive Walkthroughs Walkthroughs Lucent Technologies Bell Labs Innovations Daniel G. Aliaga Daniel G. Aliaga Visual Communications Visual Communications Research Research Lucent Technologies Bell Lucent Technologies Bell Labs Labs

Upload: kadeem-chase

Post on 04-Jan-2016

38 views

Category:

Documents


0 download

DESCRIPTION

Lucent Technologies. Bell Labs Innovations. Image-Based Rendering to Accelerate Interactive Walkthroughs. Daniel G. Aliaga Visual Communications Research Lucent Technologies Bell Labs . 3D Models. 2.0M tris. 1.7M tris. 1.0M tris. 0.9M tris. Why Use Images?. - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: Image-Based Rendering to Accelerate Interactive Walkthroughs

Image-Based Rendering Image-Based Rendering to Accelerate to Accelerate Interactive Interactive

WalkthroughsWalkthroughs

Lucent TechnologiesBell Labs Innovations

Daniel G. AliagaDaniel G. Aliaga

Visual Communications Visual Communications ResearchResearch

Lucent Technologies Bell LabsLucent Technologies Bell Labs

<[email protected]><[email protected]>

Page 2: Image-Based Rendering to Accelerate Interactive Walkthroughs

E2

3D Models3D Models

1.7Mtris

0.9Mtris

2.0Mtris

1.0Mtris

Page 3: Image-Based Rendering to Accelerate Interactive Walkthroughs

E3

Why Use Images?Why Use Images?

• Independent of scene complexityIndependent of scene complexity

640x480 pixels640x480 pixels 640x480 pixels640x480 pixels

Page 4: Image-Based Rendering to Accelerate Interactive Walkthroughs

E4

Flight SimulatorsFlight Simulators

• Mid-1980’sMid-1980’s– E&S CT-6 one of first to use real-time E&S CT-6 one of first to use real-time

photo texturesphoto textures

• Hand-selected objects:Hand-selected objects:– Terrain, trees, airplanes, buildings, Terrain, trees, airplanes, buildings,

etc.etc.

• 30-60 Hz30-60 Hz– High visual fidelityHigh visual fidelity

Page 5: Image-Based Rendering to Accelerate Interactive Walkthroughs

E5

OverviewOverview

• Replacing Geometry with ImagesReplacing Geometry with Images• Displaying ImagesDisplaying Images

– Texture-mapping and error metricsTexture-mapping and error metrics– Geometry and image warping Geometry and image warping – Meshes, Lightfield/LumigraphMeshes, Lightfield/Lumigraph

• Image PlacementImage Placement– Automatically Bounding Model ComplexityAutomatically Bounding Model Complexity– Cells and PortalsCells and Portals

• ConclusionsConclusions

Page 6: Image-Based Rendering to Accelerate Interactive Walkthroughs

E6

Replacing Geometry with Replacing Geometry with ImagesImages

• AlgorithmAlgorithm– Select subset of modelSelect subset of model– Create image of the subsetCreate image of the subset– Cull subset and replace with imageCull subset and replace with image

• Why?Why?– Image displayed in (approx.) constant Image displayed in (approx.) constant

timetime– Image reused for several framesImage reused for several frames

Page 7: Image-Based Rendering to Accelerate Interactive Walkthroughs

E7

Simple ExampleSimple Example

Page 8: Image-Based Rendering to Accelerate Interactive Walkthroughs

E8

Simple ExampleSimple Example

Page 9: Image-Based Rendering to Accelerate Interactive Walkthroughs

E9

Simple ExampleSimple Example

Page 10: Image-Based Rendering to Accelerate Interactive Walkthroughs

E10

OverviewOverview

• Replacing Geometry with ImagesReplacing Geometry with Images• Displaying ImagesDisplaying Images

– Texture-mapping and error metricsTexture-mapping and error metrics– Geometry and image warping Geometry and image warping – Meshes, Lightfield/LumigraphMeshes, Lightfield/Lumigraph

• Image PlacementImage Placement– Automatically Bounding Model ComplexityAutomatically Bounding Model Complexity– Cells and PortalsCells and Portals

• ConclusionsConclusions

Page 11: Image-Based Rendering to Accelerate Interactive Walkthroughs

E11

Geometric Geometric DiscontinuityDiscontinuity

• If we move from the center-of-projection, If we move from the center-of-projection, discontinuities appear at the borderdiscontinuities appear at the border

Page 12: Image-Based Rendering to Accelerate Interactive Walkthroughs

E12

Temporal DiscontinuityTemporal Discontinuity

• While moving, if we switch between While moving, if we switch between geometry and image, a sudden geometry and image, a sudden poppop occurs occurs

Page 13: Image-Based Rendering to Accelerate Interactive Walkthroughs

E13

ApproachesApproaches

• Geometric and Temporal Geometric and Temporal ContinuityContinuity– Error metricsError metrics– Geometry warpingGeometry warping– Image warpingImage warping– Lightfield/LumigraphLightfield/Lumigraph

Page 14: Image-Based Rendering to Accelerate Interactive Walkthroughs

E14

Error MetricsError Metrics

• Use an Use an error metricerror metric to control to control amount of discontinuityamount of discontinuity

c1

c2

[Maciel95][Shade96][Schaufler96]

Page 15: Image-Based Rendering to Accelerate Interactive Walkthroughs

E15

Error MetricError Metric

• Relies on “angular-deviation” Relies on “angular-deviation” measuring the visual quality of measuring the visual quality of using the (same) imageusing the (same) image

Page 16: Image-Based Rendering to Accelerate Interactive Walkthroughs

E16

Video Segment IVideo Segment I

• Pre-rendered ImpostorsPre-rendered Impostors– Maciel95Maciel95

• Dynamic Image-CachingDynamic Image-Caching– Shade96, Schaufler96Shade96, Schaufler96

Page 17: Image-Based Rendering to Accelerate Interactive Walkthroughs

E17

Geometry WarpingGeometry Warping

[Aliaga96]

Page 18: Image-Based Rendering to Accelerate Interactive Walkthroughs

E18

Geometry WarpingGeometry Warping

Page 19: Image-Based Rendering to Accelerate Interactive Walkthroughs

E19

Geometry WarpingGeometry Warping

Page 20: Image-Based Rendering to Accelerate Interactive Walkthroughs

E20

Geometry WarpingGeometry Warping

Page 21: Image-Based Rendering to Accelerate Interactive Walkthroughs

E21

Geometry WarpingGeometry Warping

Page 22: Image-Based Rendering to Accelerate Interactive Walkthroughs

E22

Geometry WarpingGeometry Warping

Page 23: Image-Based Rendering to Accelerate Interactive Walkthroughs

E23

Geometry WarpingGeometry Warping

Page 24: Image-Based Rendering to Accelerate Interactive Walkthroughs

E24

Geometry WarpingGeometry Warping

Page 25: Image-Based Rendering to Accelerate Interactive Walkthroughs

E25

Geometry WarpingGeometry Warping

Surrounding geometry warpedSurrounding geometry warped(incorrect perspective)(incorrect perspective)

Correct perspectiveCorrect perspective

Page 26: Image-Based Rendering to Accelerate Interactive Walkthroughs

E26

Geometry WarpingGeometry Warping

• Surrounding geometry warped to Surrounding geometry warped to match imagematch image

Center-of-projection

Viewpoint

SurroundingGeometry

Page 27: Image-Based Rendering to Accelerate Interactive Walkthroughs

E27

Video Segment IIVideo Segment II

• Geometry WarpingGeometry Warping– Aliaga96Aliaga96

Page 28: Image-Based Rendering to Accelerate Interactive Walkthroughs

E28

Image WarpingImage Warping

• Change the image itselfChange the image itself– Re-project the image to the current Re-project the image to the current

viewpointviewpoint• [Chen93][McMillan95][Max95][Shade98][Chen93][McMillan95][Max95][Shade98]

– Display image as a (simplified, Display image as a (simplified, textured) meshtextured) mesh• [Darsa97][Sillion97][Darsa97][Sillion97]

Page 29: Image-Based Rendering to Accelerate Interactive Walkthroughs

E29

Image WarpingImage Warping

• A raster scan of each sheet produces a A raster scan of each sheet produces a back-to-front ordering of warped pixelsback-to-front ordering of warped pixels

eye

Reference COP

Page 30: Image-Based Rendering to Accelerate Interactive Walkthroughs

E30

Image WarpingImage Warping

• McMillan and Bishop’s Warping McMillan and Bishop’s Warping EquationEquation

~Texture mappingMove pixels based on distance to eye

xx22 = = (x(x11) P) P22-1-1

(c(c11 - c - c22) + P) + P22-1-1

PP1 1 xx11

x1c1

c2x2

d

Page 31: Image-Based Rendering to Accelerate Interactive Walkthroughs

E31

Example...Example...

• Image outlined in yellowImage outlined in yellow• Viewed from image’s center-of-projectionViewed from image’s center-of-projection

Page 32: Image-Based Rendering to Accelerate Interactive Walkthroughs

E32

3D Image Warp3D Image Warp

• Single sample per pixelSingle sample per pixel

Page 33: Image-Based Rendering to Accelerate Interactive Walkthroughs

E33

Layered Depth Image Layered Depth Image WarpWarp

• Multiple samples per pixel Multiple samples per pixel – Previous occlusions are filled-inPrevious occlusions are filled-in

[Popescu98]

Page 34: Image-Based Rendering to Accelerate Interactive Walkthroughs

E34

MeshesMeshes

• (Simplified) Textured Depth Mesh(Simplified) Textured Depth Mesh– Per-pixel depth creates mesh that Per-pixel depth creates mesh that

approximates 3D parallax effectsapproximates 3D parallax effects– Image is texture-mapped onto meshImage is texture-mapped onto mesh

Page 35: Image-Based Rendering to Accelerate Interactive Walkthroughs

E35

Video Segment IIIVideo Segment III

• 3D Image Warping3D Image Warping– McMillan95McMillan95

• Textured MeshesTextured Meshes– Darsa97, Sillion97Darsa97, Sillion97

Page 36: Image-Based Rendering to Accelerate Interactive Walkthroughs

E36

Lightfield/LumigraphLightfield/Lumigraph

• Flow of light at all positions and Flow of light at all positions and directionsdirections– [Levoy96][Gortler96][Levoy96][Gortler96]

• Large number of images are used as Large number of images are used as 2D slices of a 4D light function2D slices of a 4D light function

(u,v)

(s,t)

Page 37: Image-Based Rendering to Accelerate Interactive Walkthroughs

E37

Video Segment IVVideo Segment IV

• Light fieldLight field– Levoy96Levoy96

• LumigraphLumigraph– Gortler96Gortler96

Page 38: Image-Based Rendering to Accelerate Interactive Walkthroughs

E38

OverviewOverview

• Replacing Geometry with ImagesReplacing Geometry with Images• Displaying ImagesDisplaying Images

– Texture-mapping and error metricsTexture-mapping and error metrics– Geometry and image warping Geometry and image warping – Meshes, Lightfield/LumigraphMeshes, Lightfield/Lumigraph

• Image PlacementImage Placement– Automatically Bounding Model ComplexityAutomatically Bounding Model Complexity– Cells and PortalsCells and Portals

• ConclusionsConclusions

Page 39: Image-Based Rendering to Accelerate Interactive Walkthroughs

E39

Automatic Image-Automatic Image-PlacementPlacement

• As a preprocessAs a preprocess– Select geometry to replace with an Select geometry to replace with an

image in order to limit the number of image in order to limit the number of primitives to render for any frameprimitives to render for any frame

• At run timeAt run time– Display selected geometry as a Display selected geometry as a

(depth) image(depth) image– Render remaining geometry normallyRender remaining geometry normally

Page 40: Image-Based Rendering to Accelerate Interactive Walkthroughs

E40

Automatic Image-Automatic Image-PlacementPlacement

PreprocessPreprocess

Run timeRun time

ModelAutomatic

Image-Placement

Create Images

Cull GeometryReplaced with

Images

RenderGeometry and

ImagesDisplayModel

Page 41: Image-Based Rendering to Accelerate Interactive Walkthroughs

E41

Example RenderingExample Rendering

Final Scene

Geometry

Image

+ =

Page 42: Image-Based Rendering to Accelerate Interactive Walkthroughs

E42

Viewpoint

- Overview- Image Placement- Displaying Images- ConclusionsKey ObservationKey Observation

• Example modelExample model• Too much Too much

geometry in geometry in view frustumview frustum

Page 43: Image-Based Rendering to Accelerate Interactive Walkthroughs

E43

Viewpoint

Key ObservationKey Observation

• Geometry is Geometry is replaced by replaced by image to limit image to limit the number of the number of primitives to primitives to renderrender

Page 44: Image-Based Rendering to Accelerate Interactive Walkthroughs

E44

Eye

Grid viewpoint

Key ObservationKey Observation

• Less geometry Less geometry is in the view is in the view frustum from frustum from the eye than the eye than the one from the one from the grid the grid viewpoint viewpoint

Page 45: Image-Based Rendering to Accelerate Interactive Walkthroughs

E45

Recursive Subdivision Recursive Subdivision AlgorithmAlgorithm

2k 2k+1

2k+22k+1

Even to odd

Odd to even

Page 46: Image-Based Rendering to Accelerate Interactive Walkthroughs

E46

Example GridExample Grid

3D grid of 1557 viewpoints3D grid of 1557 viewpoints

-1

-0.50

0.51

-1

-0.5

0

0.5

1-1

-0.5

0

0.5

1

Wireframe renderingWireframe rendering

Page 47: Image-Based Rendering to Accelerate Interactive Walkthroughs

E47

Sample PathSample Path

0

100000

200000

300000

400000

500000

600000

700000

0 50 100 150 200 250 300Frame

View Frustum CullingImages Present

Pol

ygon

s

Power Plant, 2M tris, SGI Onyx2 w/ IR

Page 48: Image-Based Rendering to Accelerate Interactive Walkthroughs

E48

Images Per TriangleImages Per Triangle

0

0.001

0.002

0.003

0.004

0 10 20 30 40 50

Maximum Rendered Triangles (% of Model)

Imag

es P

er M

odel

Tri

angl

e

Power PlantTorpedo RoomBrooks HousePipesBest CaseWorst Case

Page 49: Image-Based Rendering to Accelerate Interactive Walkthroughs

E49

Preprocessing Preprocessing SummarySummary

Model No. of Images Prep. Time(hours)

EstimatedSpace (MB)

Power Plant(2M)

239-5815 1.2-21.7 156-3802

Torpedo Room(850k)

181-2333 1.1-11.8 72- 933

Brooks House(1.7M)

561-2492 11.4-28.4 388-1725

Pipes(1M)

282- 893 2.4- 4.6 175- 554

Page 50: Image-Based Rendering to Accelerate Interactive Walkthroughs

E50

Video Segment VVideo Segment V

• Automatically Bounding Geometric Automatically Bounding Geometric Complexity by Using ImagesComplexity by Using Images– Aliaga99Aliaga99

Page 51: Image-Based Rendering to Accelerate Interactive Walkthroughs

E51

Cells and PortalsCells and Portals

[Airey90, Teller91, Luebke95][Airey90, Teller91, Luebke95]

Page 52: Image-Based Rendering to Accelerate Interactive Walkthroughs

E52

Portal ImagesPortal Images

[Aliaga97][Aliaga97]

Page 53: Image-Based Rendering to Accelerate Interactive Walkthroughs

E53

Creating Portal ImagesCreating Portal Images

portaleye

Ideal portal Ideal portal image would be image would be one sampled one sampled from the current from the current eye positioneye position

Page 54: Image-Based Rendering to Accelerate Interactive Walkthroughs

E54

Creating Portal ImagesCreating Portal Images

Reference COPs

Display one of a large number of pre-computed images (~120) portal

Page 55: Image-Based Rendering to Accelerate Interactive Walkthroughs

E55

Creating Portal ImagesCreating Portal Images

portal

Reference COPs

or…

Warp one of a much smaller number of reference images

Page 56: Image-Based Rendering to Accelerate Interactive Walkthroughs

E56

Brooks HouseBrooks House

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5

0 50 100 150 200

Frame

Ren

der

ing

Tim

e (s

ec.)

View Frustum Culling

Portal Culling

Static Portal Images

Onyx IR

Page 57: Image-Based Rendering to Accelerate Interactive Walkthroughs

E57

Brooks HouseBrooks House

0

0.1

0.2

0.3

0.4

0.5

0 100 200 300 400 500

Frame

Re

nd

erin

g T

ime

(sec

.)

Portal Culling

Warped Portal Images

Onyx IR

Page 58: Image-Based Rendering to Accelerate Interactive Walkthroughs

E58

Video Segment VIVideo Segment VI

• Architectural Walkthroughs using Architectural Walkthroughs using Portal ImagesPortal Images– Aliaga97, Rafferty98Aliaga97, Rafferty98

Page 59: Image-Based Rendering to Accelerate Interactive Walkthroughs

E59

OverviewOverview

• Replacing Geometry with ImagesReplacing Geometry with Images• Displaying ImagesDisplaying Images

– Texture-mapping and error metricsTexture-mapping and error metrics– Geometry and image warping Geometry and image warping – Meshes, Lightfield/LumigraphMeshes, Lightfield/Lumigraph

• Image PlacementImage Placement– Automatically Bounding Model ComplexityAutomatically Bounding Model Complexity– Cells and PortalsCells and Portals

• ConclusionsConclusions

Page 60: Image-Based Rendering to Accelerate Interactive Walkthroughs

E60

Image QualityImage Quality

• What about measuring quality? What about measuring quality? – Need a perceptual quality metric!Need a perceptual quality metric!

• We knowWe know– Texture-mapping: bad perspective, Texture-mapping: bad perspective,

small distortions believable (geometry small distortions believable (geometry warping)warping)

– IBR: correct perspective, disocclusionsIBR: correct perspective, disocclusions– Meshes: stretching of skinsMeshes: stretching of skins

Page 61: Image-Based Rendering to Accelerate Interactive Walkthroughs

E61

LimitationsLimitations

• Diffuse illuminationDiffuse illumination– Deferred shading?Deferred shading?

• Static modelsStatic models– Incremental updating?Incremental updating?

• Cannot sample all visible surfacesCannot sample all visible surfaces– Smarter reconstruction/resampling?Smarter reconstruction/resampling?

• Can only sample surfaces at a fixed Can only sample surfaces at a fixed resolutionsresolutions– Multi-resolution reference images?Multi-resolution reference images?

Page 62: Image-Based Rendering to Accelerate Interactive Walkthroughs

E62

AcknowledgmentsAcknowledgments

• Authors of the Video SegmentsAuthors of the Video Segments• ModelsModels

– Discreet Logic, UNC Walkthrough Discreet Logic, UNC Walkthrough GroupGroup

• UNC-Chapel HillUNC-Chapel Hill– Walkthrough, PixelFlow, ImageFlowWalkthrough, PixelFlow, ImageFlow

• NSF, NIH, DARPANSF, NIH, DARPA• Lucent Technologies Bell LabsLucent Technologies Bell Labs