Slide 1

Inverse Volume Rendering with Material Dictionaries Ioannis Gkioulekas 1 Shuang Zhao 2 Kavita Bala 2 Todd Zickler 1 Anat Levin 3 1 Harvard 3 Weizmann 2 Cornell 1

Slide 2

Most materials are translucent 2 jewelry skin architecture Photo credit: Bei Xiao, Ted Adelson food

Slide 3

We know how to render them 3 Monte-Carlo rendering material parameters Veach 1997, Dutr et al. 2006 ? rendered image

Slide 4

We show how to measure them 4 inverse rendering material parameters rendered image captured photograph

Slide 5

Our contributions 5 material 1. exact inverse volume rendering with arbitrary phase functions! 2. validation with calibration materials known parameters 3. database of broad range of materials thinthick non- dilutable solids

Slide 6

material sample Why is inverse rendering so hard? 6 radiative transfer random walk of photons inside volume volume light transport has very complex dependence material parameters thinthick non- dilutable solids

Slide 7

thinthick non- dilutable solids Light transport approximations 7 Previous approach: single-scattering random walk of photons inside volume single-bounce random walk Narasimhan et al. 2006

Slide 8

Light transport approximations 8 Previous approach: diffusion Jensen et al. 2001 Papas et al. 2013 isotropic distribution of photons parameter ambiguity material 1 material 2 random walk of photons inside volume thinthick non- dilutable solids

Slide 9

Inverse rendering without approximations 9 random walk of photons inside volume exact inversion of random walk thinthick non- dilutable solids

Slide 10

Our approach 10 appearance matching ii. operator-theoretic analysis i. material representation iii. stochastic optimization

Slide 11

Background 11 phase function p() scattering coefficient s extinction coefficient t m = ( t s p()) random walk of photons inside medium

Slide 12

Papas et al. 2013 Phase function parameterization 12 not general enough Henyey-Greenstein lobes Chen et al. 2006 Donner et al. 2008 Fuchs et al. 2007 Goesele et al. 2004 Gu et al. 2008 Hawkins et al. 2005 Holroyd et al. 2011 McCormick et al. 1981 Pine et al. 1990 Prahl et al. 1993 Wang et al. 2008 Gkioulekas et al. 2013 Narasimhan et al. 2006 Jensen et al. 2001 Previous approach: single-parameter families

Slide 13

m = q q m q p = q q p q D = {m 1, m 2, , m Q } Dictionary parameterization 13 tent phase functions D = {p 1, p 2, , p Q } p1p1 p2p2 p3p3 p4p4 p5p5 p6p6 p7p7 p8p8 p9p9 p 10 p 11 dictionary of arbitrary p similarly for t and s 11 22 33 44 55 66 77 88 99 10 11 D phase functions materials t = q q t,q s = q q s,q

Slide 14

Our approach 14 appearance matching ii. operator-theoretic analysis i. material representation iii. stochastic optimization m = q q m q

Slide 15

Operator-theoretic analysis 15 m = ( t s p()) random walk of photons inside medium discretized random walk paths propagation step

Slide 16

total radiance K() = q q K q Operator-theoretic analysis 16 m = ( t s p()) discretized random walk paths propagation step L(x, ) radiance at all medium points and directions L n+1 (x, ) = L n (x, )K rendering operator R = (I - K) -1 L input L = n L n L(x, ) = R L input (x, ) radiance after n steps radiance after n+1 steps R()= (I - q q K q ) -1 dictionary representation: m = q q m q

Slide 17

Our approach 17 appearance matching ii. operator-theoretic analysis i. material representation iii. stochastic optimization m = q q m q R()= (I - q q K q ) -1

Slide 18

Stochastic optimization 18 appearance matching analytic operator expression for gradient! R() render()single-step q render() R()KqKq gradient descent optimization for inverse rendering min photo - render() 2

Slide 19

Stochastic optimization 19 exact gradient descent for k = 1, , N, k = k - 1 - a k end N = a few hundreds several CPU hours * = intractable exact

Slide 20

Stochastic optimization 20 Monte-Carlo rendering to compute 10 2 samples noisy + fast 10 4 samples 10 6 samples accurate + slow

Slide 21

Stochastic optimization 21 exact gradient descent for k = 1, , N, k = k - 1 - a k end N = a few hundreds several CPU hours * = intractable stochastic gradient descent for k = 1, , N, k = k - 1 - a k end N = a few hundreds few CPU seconds * = solvable exactnoisy

Slide 22

Theory wrap-up 22 appearance matching ii. operator-theoretic analysis i. material representation iii. stochastic optimization m = q q m q R()= (I - q q K q ) -1 noisy min photo - render() 2

Slide 23

Our contributions 23 material 1. exact inverse volume rendering with arbitrary phase functions! 2. validation with calibration materials known parameters 3. database of broad range of materials thinthick non- dilutable solids

Slide 24

Measurements 24 multiple lighting multiple viewpoints appearance matching min photo - render() 2

Slide 25

Acquisition setup 25 material sample frontlighting backlighting camera

Slide 26

Acquisition setup 26 bottom rotation stage top rotation stage material sample frontlighting backlighting material sample frontlighting camera backlighting bottom rotation stage top rotation stage camera

Slide 27

Validation 27 Frisvad et al. 2007 polystyrene monodispersions aluminum oxide polydispersions very precise dispersions (NIST Traceable Standards) calibration materials known parameters Mie theory size % particle material medium material

Slide 28

Parameter accuracy 28 polystyrene 1polystyrene 2polystyrene 3aluminum oxide all parameters estimated within 4% error comparison of ground-truth and measured parameters ground-truth measured Henyey-Greenstein fit -0 p()

Slide 29

Matching novel measurements 29 captured rendered rendered with HGprofiles polystyrene 3 comparison of captured and rendered images images under unseen geometries predicted within 5% RMS error ground-truth measured Henyey-Greenstein fit

Slide 30

Our contributions 30 material 1. exact inverse volume rendering with arbitrary phase functions! 2. validation with calibration materials known parameters 3. database of broad range of materials thinthick non- dilutable solids

Slide 31

thinthick non- dilutable solids Measured materials 31 mustard whole milk shampoo hand cream coffee wine robitussin olive oil curacao mixed soap milk soap liquid clay reduced milk

Slide 32

Measured phase functions 32 whole milkreduced milk mustard shampoohand cream liquid claymilk soapmixed soapglycerine soap robitussin coffee olive oil curacao wine -0 p() measured Henyey-Greenstein fit

Slide 33

whole milkreduced milk mustard shampoohand cream liquid claymilk soapmixed soapglycerine soap robitussin coffee olive oil curacao wine Measured phase functions 33 -0 p() measured Henyey-Greenstein fit

Slide 34

Synthetic images 34 mixed soap glycerine soapolive oilcuracaowhole milk rendered image

Slide 35

Synthetic images 35 chromaticity

Slide 36

Synthetic images 36 mixed soap glycerine soapolive oilcuracaowhole milk rendered image

Slide 37

Effect of phase function 37 mixed soap measured phase function Henyey-Greenstein fit -0 p() rendered image chromaticity measured Henyey-Greenstein fit

Slide 38

Discussion 38 faster capture and convergence: trade-offs between accuracy, generality, mobility, and usability more interesting materials: more general solids, heterogeneous volumes, fluorescing materials other setups: alternative lighting (basis, adaptive, high- frequency), geometries, or imaging (transient imaging)

Slide 39

Take-home messages 39 material 1. exact inverse volume rendering with arbitrary phase functions! 2. validation with calibration materials known parameters 3. database of broad range of materials thinthick non- dilutable solids

Slide 40

Acknowledgements 40 Henry Sarkas (Nanophase) Wenzel Jakob (Mitsuba) Funding: National Science Foundation European Research Council Binational Science Foundation Feinberg Foundation Intel Amazon http://tinyurl.com/sa2013-inverse Database of measured materials:

Slide 41

Error surface 41 appearance matching min photo - render() 2

Slide 42

Light generation 42 MEMS light switch RGB combiner blue (480 nm) laser green (535 nm) laser red (635 nm) laser