lightfields, lumigraphs, and image-based rendering
Post on 17-Dec-2015
226 Views
Preview:
TRANSCRIPT
Lightfields, Lumigraphs, Lightfields, Lumigraphs, and Image-based and Image-based
RenderingRendering
2D to 4D2D to 4D
• More than can be captured in one More than can be captured in one imageimage
• All light passing through a region All light passing through a region of spaceof space
• General (5D) vs. unoccluded space General (5D) vs. unoccluded space (4D)(4D)
“Rebinning” pixels
Image-Based Modeling and Image-Based Modeling and RenderingRendering
• Generate new views of a scene Generate new views of a scene directly from existing viewsdirectly from existing views
• ““Pure” IBR (such as lightfields): no Pure” IBR (such as lightfields): no model of scenemodel of scene
• Other IBR techniques try to obtain Other IBR techniques try to obtain higher quality with less storage by higher quality with less storage by building a modelbuilding a model
LightfieldsLightfields
• Advantages:Advantages:– Simpler computation vs. traditional CGSimpler computation vs. traditional CG
– Cost independent of scene complexityCost independent of scene complexity
– Cost independent of material Cost independent of material properties and other optical effectsproperties and other optical effects
• Disadvantages:Disadvantages:– Static geometryStatic geometry
– Fixed lightingFixed lighting
– High storage costHigh storage cost
Using LightfieldsUsing Lightfields
• Obtain 2D slices of 4D data setObtain 2D slices of 4D data set
• Arbitrary views: take other 2D Arbitrary views: take other 2D slicesslices
• Challenges:Challenges:– CaptureCapture
– ParameterizationParameterization
– CompressionCompression
– RenderingRendering
Capturing LightfieldsCapturing Lightfields
• Need a 2D set of (2D) imagesNeed a 2D set of (2D) images
• Choices:Choices:– Camera motion: human vs. computerCamera motion: human vs. computer
– Constraints on camera motionConstraints on camera motion
– Coverage and sampling uniformityCoverage and sampling uniformity
– AliasingAliasing
• Point / anglePoint / angle
• Two points on a sphereTwo points on a sphere
• Points on two planesPoints on two planes
• Original images and camera Original images and camera positionspositions
Lightfield ParameterizationLightfield Parameterization
CompressionCompression
• Compress individual images (JPEG, Compress individual images (JPEG, etc.)etc.)
• Adapt video compression to 2D Adapt video compression to 2D arraysarrays
• Decomposition into basis functionsDecomposition into basis functions
• Vector quantizationVector quantization
RenderingRendering
• How to select rays?How to select rays?
• InterpolationInterpolation
• Taking advantage of hardwareTaking advantage of hardware– Graphics hardwareGraphics hardware
– Compression hardwareCompression hardware
ImplementationsImplementations
• Lightfields, SIGGRAPH 96,Lightfields, SIGGRAPH 96,Levoy and HanrahanLevoy and Hanrahan
• Lumigraphs, SIGGRAPH 96,Lumigraphs, SIGGRAPH 96,Gortler et al.Gortler et al.
• Unstructured lumigraphs, Unstructured lumigraphs, SIGGRAPH 01, Buehler et al.SIGGRAPH 01, Buehler et al.
Light Field RenderingLight Field Rendering
• Capture:Capture:– Computer-controlled camera rigComputer-controlled camera rig
– Move camera to grid of locations on a Move camera to grid of locations on a planeplane
Light Field RenderingLight Field Rendering
• Parameterization:Parameterization:– Two planes, evenly sampled: “light Two planes, evenly sampled: “light
slab”slab”
– In general, planes in arbitrary In general, planes in arbitrary orientationsorientations
– In practice, one plane = camera In practice, one plane = camera locationslocations• Minimizes resamplingMinimizes resampling
RenderingRendering
• For each desired ray:For each desired ray:– Compute intersection with (u,v) andCompute intersection with (u,v) and
(s,t) planes(s,t) planes
– Take closest rayTake closest ray
• Variants: interpolationVariants: interpolation– Bilinear in (u,v) onlyBilinear in (u,v) only
– Bilinear in (s,t) onlyBilinear in (s,t) only
– Quadrilinear in (u,v,s,t)Quadrilinear in (u,v,s,t)
Light Field CompressionLight Field Compression
• Based on vector quantizationBased on vector quantization
• Preprocessing: build a Preprocessing: build a representative codebook of 4D representative codebook of 4D tilestiles
• Each tile in lightfield represented Each tile in lightfield represented by indexby index
• Example: 2x2x2x2 tiles, 16 bit Example: 2x2x2x2 tiles, 16 bit index = 24:1 compressionindex = 24:1 compression
Lightfield Compression byLightfield Compression byVector QuantizationVector Quantization
• Advantages:Advantages:– Fast at runtimeFast at runtime
– Reasonable compressionReasonable compression
• Disadvantages:Disadvantages:– Preprocessing slowPreprocessing slow
– Manually-selected codebook sizeManually-selected codebook size
– Does not take advantage of structureDoes not take advantage of structure
The LumigraphThe Lumigraph
• Capture: move camera by handCapture: move camera by hand
• Camera intrinsics assumed Camera intrinsics assumed calibratedcalibrated
• Camera pose recovered from Camera pose recovered from markersmarkers
Lumigraph PostprocessingLumigraph Postprocessing
• Obtain rough geometric modelObtain rough geometric model– Chroma keying (blue screen) to Chroma keying (blue screen) to
extract silhouettesextract silhouettes
– Octree-based space carvingOctree-based space carving
• Resample images to two-plane Resample images to two-plane parameterizationparameterization
Lumigraph RenderingLumigraph Rendering
• Use rough depth information to Use rough depth information to improve rendering qualityimprove rendering quality
Lumigraph RenderingLumigraph Rendering
• Use rough depth information to Use rough depth information to improve rendering qualityimprove rendering quality
Unstructured Lumigraph Unstructured Lumigraph RenderingRendering
• Further enhancement of Further enhancement of lumigraphs:lumigraphs:do not use two-plane do not use two-plane parameterizationparameterization
• Store original pictures: no Store original pictures: no resamplingresampling
• Hand-held camera, moved around Hand-held camera, moved around an environmentan environment
Unstructured Lumigraph Unstructured Lumigraph RenderingRendering
• To reconstruct views, assign To reconstruct views, assign penalty to each original raypenalty to each original ray– Distance to desired ray, usingDistance to desired ray, using
approximate geometryapproximate geometry
– ResolutionResolution
– Feather near edges of imageFeather near edges of image
• Construct “camera blending field”Construct “camera blending field”
• Render using texture mappingRender using texture mapping
Other Lightfield Acquisition Other Lightfield Acquisition DevicesDevices
• Spherical motionSpherical motionof camera aroundof camera aroundan objectan object
• Samples space ofSamples space ofdirections uniformlydirections uniformly
• Second arm toSecond arm tomove light source –move light source –measure BRDFmeasure BRDF
Other Lightfield Acquisition Other Lightfield Acquisition DevicesDevices
• Acquire an entireAcquire an entirelight field at oncelight field at once
• Video ratesVideo rates
• Integrated MPEG2Integrated MPEG2compression forcompression foreach cameraeach camera
(Bennett Wilburn, Michal Smulski, Mark Horowitz)
top related