a signal-processing framework for forward and inverse rendering ravi ramamoorthi...

A Signal-Processing Framework for A Signal-Processing Framework for Forward and Inverse RenderingForward and Inverse Rendering

A Signal-Processing Framework for A Signal-Processing Framework for Forward and Inverse RenderingForward and Inverse Rendering

Ravi Ramamoorthiravir@graphics.stanford.edu

Ravi Ramamoorthiravir@graphics.stanford.edu

Pat Hanrahanhanrahan@graphics.stanford.edu

Pat Hanrahanhanrahan@graphics.stanford.edu

OutlineOutlineOutlineOutline

• Motivation• Forward Rendering• Inverse Rendering• Object Recognition

• Reflection as Convolution

• Efficient Rendering: Environment Maps

• Lighting Variability in Object Recognition

• Deconvolution, Inverse Rendering

• Summary

• Motivation• Forward Rendering• Inverse Rendering• Object Recognition

• Reflection as Convolution

• Efficient Rendering: Environment Maps

• Lighting Variability in Object Recognition

• Deconvolution, Inverse Rendering

• Summary

Interactive RenderingInteractive RenderingInteractive RenderingInteractive Rendering

Directional Source Complex Illumination

Ramamoorthi and Hanrahan, SIGGRAPH 2001b

Reflection MapsReflection MapsReflection MapsReflection Maps

Blinn and Newell, 1976

Environment MapsEnvironment MapsEnvironment MapsEnvironment Maps

Miller and Hoffman, 1984Later, Greene 86, Cabral et al. 87

Reflectance Space ShadingReflectance Space ShadingReflectance Space ShadingReflectance Space Shading

Cabral, Olano, Nemec 1999

Reflectance MapsReflectance MapsReflectance MapsReflectance Maps

• Reflectance Maps (Index by N)• Horn, 1977

• Irradiance (N) and Phong (R) Reflection Maps• Hoffman and Miller, 1984

• Reflectance Maps (Index by N)• Horn, 1977

• Irradiance (N) and Phong (R) Reflection Maps• Hoffman and Miller, 1984

Mirror Sphere Chrome Sphere Matte (Lambertian) Sphere

Irradiance Environment Map

Complex IlluminationComplex Illumination Complex IlluminationComplex Illumination

• Must (pre)compute hemispherical integral of lighting• Efficient Prefiltering (> 1000x faster)

• Traditionally, requires irradiance map textures• Real-Time Procedural Rendering (no textures)

• New representation for lighting design, IBR

• Must (pre)compute hemispherical integral of lighting• Efficient Prefiltering (> 1000x faster)

• Traditionally, requires irradiance map textures• Real-Time Procedural Rendering (no textures)

• New representation for lighting design, IBR

Directional Source Complex LightingIllumination Irradiance

Environment Map

Photorealistic RenderingPhotorealistic RenderingPhotorealistic RenderingPhotorealistic Rendering

Geometry

70s, 80s: Splines 90s: Range Data

Materials/Lighting(Texture, reflectance [BRDF], Lighting)

Realistic Input Models Required

Arnold Renderer: Marcos Fajardo

Rendering Algorithm

80s,90s: Physically-based

Inverse RenderingInverse RenderingInverse RenderingInverse Rendering• How to measure realistic material models, lighting?

• From real photographs by inverse rendering

• Can then change viewpoint, lighting, reflectance• Rendered images very realistic: they use real data

• How to measure realistic material models, lighting?• From real photographs by inverse rendering

• Can then change viewpoint, lighting, reflectance• Rendered images very realistic: they use real data

Illumination:Mirror Sphere

Grace Cathedralcourtesy

Paul Debevec

BRDF (reflectance): Images using

point light source

FlowchartFlowchartFlowchartFlowchart

Lighting

BRDF NewView

NewView,Light

ResultsResultsResultsResults

Photograph Computer rendering

New view, new lighting

Ramamoorthi and Hanrahan, SIGGRAPH 2001a

Inverse Rendering: GoalsInverse Rendering: GoalsInverse Rendering: GoalsInverse Rendering: Goals

• Complex (possibly unknown) illumination

• Estimate both lighting and reflectance (factorization)

• Complex (possibly unknown) illumination

• Estimate both lighting and reflectance (factorization)

Photographs of 4 spheres in 3 different

lighting conditions

courtesy Dror and Adelson

Factorization AmbiguitiesFactorization AmbiguitiesFactorization AmbiguitiesFactorization Ambiguities

Width of Light Source

SurfaceRoughness

Inverse ProblemsInverse ProblemsInverse ProblemsInverse Problems

• Sometimes ill-posed• No solution or several solutions given data

• Often numerically ill-conditioned• Answer not robust, sensitive to noise

• Need general framework to address these issues• Mathematical theory for complex illumination

• Sometimes ill-posed• No solution or several solutions given data

• Often numerically ill-conditioned• Answer not robust, sensitive to noise

• Need general framework to address these issues• Mathematical theory for complex illumination

Directional

Source

Area source

Same reflectance

Lighting effects in recognitionLighting effects in recognitionLighting effects in recognitionLighting effects in recognition

• Space of Images (Lighting) is Infinite Dimensional• Prior empirical work: 5D subspace captures variability

• We explain empirical data, subspace methods

• Space of Images (Lighting) is Infinite Dimensional• Prior empirical work: 5D subspace captures variability

• We explain empirical data, subspace methods

Peter Belhumeur: Yale Face Database A

OutlineOutlineOutlineOutline

• Motivation

• Signal Processing Framework: Reflection as Convolution• Reflection Equation (2D)• Fourier Analysis (2D)• Spherical Harmonic Analysis (3D)• Examples

• Efficient Rendering: Environment Maps

• Lighting Variability in Object Recognition

• Deconvolution, Inverse Rendering

• Summary

• Motivation

• Signal Processing Framework: Reflection as Convolution• Reflection Equation (2D)• Fourier Analysis (2D)• Spherical Harmonic Analysis (3D)• Examples

• Efficient Rendering: Environment Maps

• Lighting Variability in Object Recognition

• Deconvolution, Inverse Rendering

• Summary

Reflection as Convolution (2D)Reflection as Convolution (2D)Reflection as Convolution (2D)Reflection as Convolution (2D)

2

2

ˆ( , ) ( , ) ( , )o i i o iB L d

( , ) ( ) ( )i i iL L L

o

L

i i

o B

BRDF

( , )i o

2

2

ˆ( , ) ( ) ( , )o i i o iB L d

B L

Fourier Analysis (2D)Fourier Analysis (2D)Fourier Analysis (2D)Fourier Analysis (2D)

2

2

ˆ( , ) ( ) ( , )o i i o iB L d

iIl

pleL ,ˆ i oIl I

lp

qp

p

e e ,oIp

pIl

ql

p

B e e

, ,ˆ2l p l l pB L

Spherical Harmonics (3D)Spherical Harmonics (3D)Spherical Harmonics (3D)Spherical Harmonics (3D)

-1-2 0 1 2

0

1

2

.

.

.

( , )lmY

xy z

xy yz 23 1z zx 2 2x y

Spherical Harmonic AnalysisSpherical Harmonic AnalysisSpherical Harmonic AnalysisSpherical Harmonic Analysis

, ,ˆ2l p l l pB L

lL ,ˆl p,l pB

2

2

ˆ( , ) ( ) ( , )o i i o iB L d

2D:

3D:,lm pqB

lmL ,ˆlq pqIsotropic

, ,ˆlm pq l lm lq pqB L

InsightsInsightsInsightsInsights

• Signal processing framework for reflection• Light is the signal• BRDF is the filter• Reflection on a curved surface is convolution

• Inverse rendering is deconvolution

• Our contribution: Formal Frequency-space analysis

• Signal processing framework for reflection• Light is the signal• BRDF is the filter• Reflection on a curved surface is convolution

• Inverse rendering is deconvolution

• Our contribution: Formal Frequency-space analysis

Example: Mirror BRDFExample: Mirror BRDFExample: Mirror BRDFExample: Mirror BRDF

• BRDF is delta function• Harmonic Transform is constant (infinite width)

• Reflected light field corresponds directly to lighting

• Mirror Sphere (Gazing Ball)

• BRDF is delta function• Harmonic Transform is constant (infinite width)

• Reflected light field corresponds directly to lighting

• Mirror Sphere (Gazing Ball)

Phong, Microfacet ModelsPhong, Microfacet ModelsPhong, Microfacet ModelsPhong, Microfacet Models

• Rough surfaces blur highlight

• Analytic Formula• Approximately Gaussian

• Rough surfaces blur highlight

• Analytic Formula• Approximately Gaussian

Mirror Matte

2

exp2l l

l

s

Roughness

Example: Lambertian BRDFExample: Lambertian BRDFExample: Lambertian BRDFExample: Lambertian BRDF

2 1

2

2

2 1 ( 1) !ˆ 2

4 ( 2)( 1) 2 !

l

l l l

l ll even

l l

Ramamoorthi and Hanrahan, JOSA 2001

Second-Order ApproximationSecond-Order ApproximationSecond-Order ApproximationSecond-Order Approximation

Lambertian: 9 parameters only• order 2 approx. suffices• Quadratic polynomial

Lambertian: 9 parameters only• order 2 approx. suffices• Quadratic polynomial

L=0 L=1 L=2 Exact

-1-2 0 1 2

0

1

2

( , )lmY

xy z

xy yz 23 1z zx 2 2x ySimilar to Basri & Jacobs 01

Dual RepresentationDual Representation Dual RepresentationDual Representation

• Practical Representation• Diffuse localized in frequency space• Specular localized in angular space• Dual Angular, Frequency-Space representation

• Practical Representation• Diffuse localized in frequency space• Specular localized in angular space• Dual Angular, Frequency-Space representation

Frequency9 param.

B Bd

diffuse

Bs,slow

slow specular(area sources)

Bs,fast

fast specular(directional)

= + +

Angular SpaceFrequency9 param.

OutlineOutlineOutlineOutline

• Motivation

• Reflection as Convolution

• Efficient Rendering: Environment Maps

• Lighting Variability in Object Recognition

• Deconvolution, Inverse Rendering

• Summary

• Motivation

• Reflection as Convolution

• Efficient Rendering: Environment Maps

• Lighting Variability in Object Recognition

• Deconvolution, Inverse Rendering

• Summary

VideoVideoVideoVideo

Ramamoorthi and Hanrahan, SIGGRAPH 2001b

OutlineOutlineOutlineOutline

• Motivation

• Reflection as Convolution

• Efficient Rendering: Environment Maps

• Lighting Variability in Object Recognition

• Deconvolution, Inverse Rendering

• Summary

• Motivation

• Reflection as Convolution

• Efficient Rendering: Environment Maps

• Lighting Variability in Object Recognition

• Deconvolution, Inverse Rendering

• Summary

Lighting effects in recognitionLighting effects in recognitionLighting effects in recognitionLighting effects in recognition

• Space of Images (Lighting) is Infinite Dimensional• Prior empirical work: 5D subspace captures variability

• We explain empirical data, subspace methods

• Space of Images (Lighting) is Infinite Dimensional• Prior empirical work: 5D subspace captures variability

• We explain empirical data, subspace methods

Peter Belhumeur: Yale Face Database A

Face Basis FunctionsFace Basis FunctionsFace Basis FunctionsFace Basis Functions

• 5 basis functions capture 95% of image variability

• Linear combinations of spherical harmonics

• Complex illumination not much harder than points

• 5 basis functions capture 95% of image variability

• Linear combinations of spherical harmonics

• Complex illumination not much harder than points

Frontal Lighting Side Above/Below Extreme Side Corner

Inverse LightingInverse LightingInverse LightingInverse Lighting

• Well-posed unless equals zero in denominator• Cannot recover radiance from irradiance:

contradicts theorem in Preisendorfer 76

• Well-conditioned unless small• BRDF should contain high frequencies : Sharp highlights• Diffuse reflectors are ill-conditioned : Low pass filters

• Well-posed unless equals zero in denominator• Cannot recover radiance from irradiance:

contradicts theorem in Preisendorfer 76

• Well-conditioned unless small• BRDF should contain high frequencies : Sharp highlights• Diffuse reflectors are ill-conditioned : Low pass filters

B L

BL

Inverse LambertianInverse LambertianInverse LambertianInverse LambertianSum l=2 Sum l=4True Lighting

Mirror

Teflon

OutlineOutlineOutlineOutline

• Motivation

• Reflection as Convolution

• Efficient Rendering: Environment Maps

• Lighting Variability in Object Recognition

• Deconvolution, Inverse Rendering

• Summary

• Motivation

• Reflection as Convolution

• Efficient Rendering: Environment Maps

• Lighting Variability in Object Recognition

• Deconvolution, Inverse Rendering

• Summary

Inverse Rendering: GoalsInverse Rendering: GoalsInverse Rendering: GoalsInverse Rendering: Goals• Formal study: Well-posedness, conditioning

• General Complex (Unknown) Illumination

• Formal study: Well-posedness, conditioning

• General Complex (Unknown) Illumination

Bronze Delrin Paint Rough Steel

Photographs

Renderings(Recovered

BRDF)

Quantitative Pixel error approximately 5%

Factoring the Light FieldFactoring the Light FieldFactoring the Light FieldFactoring the Light Field

• The light field may be factored to estimate both the BRDF and the lightingKnowns B (4D)Unknowns L (2D)

(½ 3D) -- Make use of reciprocity

• The light field may be factored to estimate both the BRDF and the lightingKnowns B (4D)Unknowns L (2D)

(½ 3D) -- Make use of reciprocity

,,

1 lm pqlq pq

l lm

B

L

,00

00, 0

1 lmlm

l l

BL

B

00

1

l

L

B L

Algorithms ValidationAlgorithms ValidationAlgorithms ValidationAlgorithms ValidationRendering

Unknown

Lighting

Photograph

Known

Lighting

Kd 0.91 0.89 0.87

Ks 0.09 0.11 0.13

1.85 1.78 1.48

0.13 0.12 .14

Recovered

Light

Estimate by ratio of

intensity andtotal energy

Marschner

Complex GeometryComplex GeometryComplex GeometryComplex Geometry

3 photographs of a sculptureComplex unknown illuminationGeometry KNOWNEstimate BRDF and Lighting

FlowchartFlowchartFlowchartFlowchart

Lighting

BRDF NewView

NewView,Light

ComparisonComparisonComparisonComparison

RenderedKnown Lighting

Photograph RenderedUnknown Lighting

New View, LightingNew View, LightingNew View, LightingNew View, Lighting

Photograph Computer rendering

Textured ObjectsTextured ObjectsTextured ObjectsTextured Objects

Real RenderingComplex, Known Lighting

OutlineOutlineOutlineOutline

• Motivation

• Reflection as Convolution

• Efficient Rendering: Environment Maps

• Lighting Variability in Object Recognition

• Deconvolution, Inverse Rendering

• Summary • Conclusions• Pointers

• Motivation

• Reflection as Convolution

• Efficient Rendering: Environment Maps

• Lighting Variability in Object Recognition

• Deconvolution, Inverse Rendering

• Summary • Conclusions• Pointers

SummarySummarySummarySummary

• Reflection as Convolution

• Signal-Processing Framework

• Frequency-space analysis yields insights• Lambertian: approximated with 9 parameters• Phong/Microfacet: acts like Gaussian filter

• Inverse Rendering• Formal Study: Well-posedness, conditioning• Dual Representations• Practical Algorithms: Complex Lighting, Factorization

• Efficient Forward Rendering (Environment Maps)

• Lighting Variability in Object Recognition

• Reflection as Convolution

• Signal-Processing Framework

• Frequency-space analysis yields insights• Lambertian: approximated with 9 parameters• Phong/Microfacet: acts like Gaussian filter

• Inverse Rendering• Formal Study: Well-posedness, conditioning• Dual Representations• Practical Algorithms: Complex Lighting, Factorization

• Efficient Forward Rendering (Environment Maps)

• Lighting Variability in Object Recognition

PapersPapersPapersPapers

• http://graphics.stanford.edu/~ravir/research.html

• Theory• Flatland or 2D using Fourier analysis [SPIE 01]• Lambertian: radiance from irradiance [JOSA 01]• General 3D, Isotropic BRDFs [SIGGRAPH 01a]

• Applications• Inverse Rendering [SIGGRAPH 01a]• Forward Rendering [SIGGRAPH 01b]• Lighting variability [In preparation]

• http://graphics.stanford.edu/~ravir/research.html

• Theory• Flatland or 2D using Fourier analysis [SPIE 01]• Lambertian: radiance from irradiance [JOSA 01]• General 3D, Isotropic BRDFs [SIGGRAPH 01a]

• Applications• Inverse Rendering [SIGGRAPH 01a]• Forward Rendering [SIGGRAPH 01b]• Lighting variability [In preparation]

AcknowledgementsAcknowledgementsAcknowledgementsAcknowledgements

• Marc Levoy

• Szymon Rusinkiewicz

• Steve Marschner

• John Parissenti

• Jean Gleason

• Scanned cat sculpture is “Serenity” by Sue Dawes

• Hodgson-Reed Stanford Graduate Fellowship

• NSF ITR grant #0085864: “Interacting with the Visual World”

• Marc Levoy

• Szymon Rusinkiewicz

• Steve Marschner

• John Parissenti

• Jean Gleason

• Scanned cat sculpture is “Serenity” by Sue Dawes

• Hodgson-Reed Stanford Graduate Fellowship

• NSF ITR grant #0085864: “Interacting with the Visual World”

The EndThe EndThe EndThe End

Related WorkRelated WorkRelated WorkRelated Work

Graphics: Prefiltering Environment Maps• Qualitative observation of reflection as convolution• Miller and Hoffman 84, Greene 86• Cabral, Max, Springmeyer 87 (use spherical harmonics)• Cabral et al. 99

Vision, Perception• D’Zmura 91: Reflection as frequency-space operator• Basri and Jacobs 01: Lambertian reflection as convolution• Recognition: Appearance models e.g. Belhumeur et al.

Graphics: Prefiltering Environment Maps• Qualitative observation of reflection as convolution• Miller and Hoffman 84, Greene 86• Cabral, Max, Springmeyer 87 (use spherical harmonics)• Cabral et al. 99

Vision, Perception• D’Zmura 91: Reflection as frequency-space operator• Basri and Jacobs 01: Lambertian reflection as convolution• Recognition: Appearance models e.g. Belhumeur et al.

Related WorkRelated WorkRelated WorkRelated Work

Graphics: Prefiltering Environment Maps• Qualitative observation of reflection as convolution• Miller and Hoffman 84, Greene 86• Cabral, Max, Springmeyer 87 (use spherical harmonics)• Cabral et al. 99

Vision, Perception• D’Zmura 91: Reflection as frequency-space operator• Basri and Jacobs 01: Lambertian reflection as convolution• Recognition: Appearance models e.g. Belhumeur et al.

Our Contributions• Explicitly derive frequency-space convolution formula• Formal Quantitative Analysis in General 3D Case

Graphics: Prefiltering Environment Maps• Qualitative observation of reflection as convolution• Miller and Hoffman 84, Greene 86• Cabral, Max, Springmeyer 87 (use spherical harmonics)• Cabral et al. 99

Vision, Perception• D’Zmura 91: Reflection as frequency-space operator• Basri and Jacobs 01: Lambertian reflection as convolution• Recognition: Appearance models e.g. Belhumeur et al.

Our Contributions• Explicitly derive frequency-space convolution formula• Formal Quantitative Analysis in General 3D Case

Example: Directional SourceExample: Directional SourceExample: Directional SourceExample: Directional Source

• Lighting is delta function• Harmonic Transform is constant (infinite width)

• Reflected light field corresponds directly to BRDF• Impulse response of BRDF filter

• Lighting is delta function• Harmonic Transform is constant (infinite width)

• Reflected light field corresponds directly to BRDF• Impulse response of BRDF filter

Practical IssuesPractical IssuesPractical IssuesPractical Issues• Incomplete sparse data: Few views

• Use practical Dual Representation

• Incomplete sparse data: Few views• Use practical Dual Representation

B Bd

diffuse

Bs,slow

slow specular(area sources)

Bs,fast

fast specular(directional)

= + +

Angular SpaceFrequency

Practical IssuesPractical IssuesPractical IssuesPractical Issues• Incomplete sparse data: Few views

• Use practical Dual Representation

• Concavities: Self Shadowing

• Incomplete sparse data: Few views• Use practical Dual Representation

• Concavities: Self Shadowing

B Bd

diffuse

Bs,slow

slow specular(area sources)

Bs,fast

fast specular(directional)

= + +

Reflected RayShadowed?

IntegrateLighting

SourceShadowed?

Practical IssuesPractical IssuesPractical IssuesPractical Issues• Incomplete sparse data: Few views

• Use practical Dual Representation

• Concavities: Self Shadowing

• Textures: Spatially Varying Reflectance

• Incomplete sparse data: Few views• Use practical Dual Representation

• Concavities: Self Shadowing

• Textures: Spatially Varying Reflectance

B Bd

diffuse

Bs,slow

slow specular(area sources)

Bs,fast

fast specular(directional)

= + +

Reflected RayShadowed?

IntegrateLighting

SourceShadowed?

Kd(x) Ks(x)

Inverse BRDFInverse BRDFInverse BRDFInverse BRDF

• Well-conditioned unless L small• Lighting should have sharp features (point sources, edges)• Ill-conditioned for soft lighting

• Well-conditioned unless L small• Lighting should have sharp features (point sources, edges)• Ill-conditioned for soft lighting

Directional

Source

Area source

Same BRDF

B L

B

L

ComparisonComparisonComparisonComparisonRenderingPhotograph

Kd 0.91 0.89

Ks 0.09 0.11

1.85 1.78

0.13 0.12

Marschner Our method

KnownLighting

Pixel ErrorApproximately 5%