A Rough Guide to Shape from Shading

Tom S. F. Hainesthaines@eecs.qmul.ac.uk

Queen Mary, University of London

14th June 2012

Roadmap

1. The Original Model

2. A Little Bit Of Theoretical Stuff

3. Alternate Assumptions

4. SfS with. . .

Shape from Shading (SfS)

• Image formulation rules tell you how to go from a 3D modeland its materials to a 2D image.

• Shape from shading is the inverse problem.

• It can be seen as a constraint on the set of possible realities.

• Justifying working with it can take several arguments - thesimplest is that multiple species of animals, ourselvesincluded, use it.

Roadmap

1. The Original Model

2. A Little Bit Of Theoretical Stuff

3. Alternate Assumptions

4. SfS with. . .

Pre-Horn

• In other fields the term photoclinometry is used instead ofShape from Shading.

• Earliest reference I know of: Diggelen, A photometricinvestigation of the slopes of the heights of the ranges of hillsin the maria of the moon, 1951. Observed that the moon wasa Lambertian surface and inferred the relative depths of 1Dslices. By hand.

• First use of a computer that I know of: Rindfleisch,Photometric method for lunar topography, 1966.

Horn

• Shape From Shading: A Method For Obtaining The Shape OfA Smooth Opaque Object From One View, 1970.

• First to work with entire surfaces, rather than 1D slices.

• Used the assumptions: Unknown object, Lambertianreflectance, orthographic projection, constant known albedo, asmooth surface, valid surface not required, no surfaceinter-reflectance and a single infinitely distant and known lightsource, with no falloff.

• Still ill defined - 2 degrees of freedom for each surface normalwhen only 1 degree is constrained.

• Will explain the model in geometric terms later.

Lee & Kuo

• Shape from shading with perspective projection, 1994.

• Best approach in 1999 according to a review paper: Zhang,Tsai, Cryer, and Shah, Shape from shading: A survey, 1999.

• This and all other approaches in the survey used Horn’sassumptions - this was a comparison of who had the bestoptimiser.

• Solved by repeatedly linearising the equations and solving theconstraints for each linearisation - closely related to Newtonsmethod.

• Also used something called a ’multigrid method’. This isbasically solving at various resolutions, to avoid the really badlocal minima.

• Actually contains an integration constraint, which we willcome to later. . .

Worthington & Hancock• New constraints on data-closeness and needle map

consistency for shape-from-shading, 1999.• Used a hard constraint - that re-rendering the model must

produce the original image. This leads to a simple geometricinterpretation of the model:

L̂

n̂cos−1(I/a)

• n̂ · L̂ = I/a• For each pixel - L̂ is the direction to the light source; I is the

image brightness; a is the albedo and n̂ is the (unknown)surface normal.

Worthington & Hancock, continued. . .

• Solved the model by iterating between smoothing the field ofsurface normals and then projecting the normals back ontothe closest point on the cone.

• Initialised the field using image gradients, pointing away fromthe brightest direction. This is an implicit assumption that thelight source is probably near the camera.

• Whole bunch of different smoothing approaches tried, but theinitialisation is has the greatest impact on the results.

Haines & Wilson

• Belief propagation with directional statistics for solving theshape-from-shading problem, 2008.

• A probabilistic interpretation of Worthington & Hancock -solved the same model but used belief propagation withdirectional statistics to solve a Markov random field.

• Directional statistics are probability distributions overdirections - to make it work an approximation of convolving aFisher-Bingham-8 distribution by a Fisher distribution has tobe developed.

Directional Statistics Visualisation

Bingham distribution, α = β =

5. Can represent gradient infor-

mation.

Bingham-Mardia distribution,

k = 8, angle = 45◦. Can repre-

sent the cone constraint.

Results: Mozart at 90◦

Mozart, lit head on. Ground truth

Worthington & Hancock Lee & Kuo Haines & Wilson

Results: Mozart at 45◦

Mozart, lit at 45◦

from head on.Ground truth

Worthington & Hancock Lee & Kuo Haines & Wilson

Results: Head

Head, lit head on. Ground truth

Worthington & Hancock Lee & Kuo Haines & Wilson

Results: Venus

Venus, lit head on. Ground truth

Worthington & Hancock Lee & Kuo Haines & Wilson

Results: Bard

Bard, lit head on. Ground truth

Worthington & Hancock Lee & Kuo Haines & Wilson

Results: Sunev

Sunev, lit head on. Ground truth

Worthington & Hancock Haines & Wilson

Roadmap

1. The Original Model

2. A Little Bit Of Theoretical Stuff

3. Alternate Assumptions

4. SfS with. . .

Error Measurement

• Shading provides directional information, not depth.

• If you integrate to get depth the errors in surface orientationwill add up - the further you go the larger they will be. This iscalled curl.

• Consequentially it makes more sense to measure SfSalgorithms in terms of their angular error.

• Most papers use some function of depth error however,including both of the SfS review papers (Though the 2008review is pointless anyway.).

• Funnily enough some of the ’best’ algorithms have strongfronto-planar assumptions, that push the answer towardsbeing a flat plane.

Calibration

• Cameras do not output a linear measure of irradiance - thefunction between pixel values and actual irradiance oftenforms an ’S’ shaped curve.

• SfS algorithms need to know it - if you are doing SfS youmust calibrate your camera, as otherwise it will destroyperformance.

• In the event that the camera does not provide it or there is noway to infer it from the available data offline calibration canbe done in the same way that a multiple exposure HDR photois taken.

Bas-relief Ambiguity

• If the light source position is unknown then there is a 1Dfamily of valid surfaces.

• If the light source position is known then two members of thatfamily are valid, as illustrated by the optical illusion below.This is the concave/convex ambiguity. (Does not exist if thelight source is at the camera.)

• Not a major problem once you move beyond just SfS, as othersources of information can resolve it.

• One trick often used is to observe that most objects areapproximately convex, and initialise as such, so you at leastget stuck in a convex local minima.

Roadmap

1. The Original Model

2. A Little Bit Of Theoretical Stuff

3. Alternate Assumptions

4. SfS with. . .

Integration

• So far needle maps (Surface orientation at every pixel.) havebeen generated.

• Depth is usually desired, and as orientation is depthsdifferential we have to integrate to get it.

• If you start travelling and end up at the same place as whereyou started the sum of how far you travelled in everydimension must be zero. An arbitrary needle map does notnecessarily enforce this condition, and most of the approachesso far do not enforce it.

• There are actually papers on obtaining surfaces from dodgyneedle maps (I have always used Gaussian belief propagation.).

• Note that the +c of integration exists - depth output can onlybe relative, unless you have further information.

Frankot & Chellappa

• A method for enforcing integrability in shape from shadingalgorithms, 1988.

• Added an integrability constraint.

• Represented by δ2zδxδy = δ2z

δyδx .

• They formulate a method of projecting a needle map onto thenearest needle map that satisfies this constraint.

• Can therefore work with any SfS algorithm.

Potetz

• Efficient belief propagation for vision using linear constraintnodes, 2007.

• Easiest way to add an integration constraint is to use asurface representation that enforces it, such as a depth map.

• Potetz did this (Not the first), and solved it using anextremely hardcore BP method, with large cliques andarbitrary continuous multidimensional probability distributions.

• Insanely demanding in terms of both memory andcomputation, but the results are probably the best possible forthis kind of model.

Potetz Results: Penny

Coin, lit at 45◦. Ground truth

Lee & Kuo Haines & Wilson Potetz

Smooth Surface

• Assuming a smooth surface causes issues, particularly atdiscontinuities, and explicitly handling them has a similareffect to the same in stereopsis.

• Zheng & Chellappa, Estimation of illuminant direction,albedo, and shape from shading, 1991: Used the basic ideathat large changes in irradiance probably indicate adiscontinuity in the surface.

Non-Lambertian Shading

• The Lambertian Bidirectional Reflectance DistributionFunction (BRDF) is obviously not exhibited by many objects.

• Some simpler BRDF models can be converted into Lambertianas a pre-processing step - as long as it forms a cone constraintthe irradiance can be converted so it makes the correct angle.

• The biggest issue is knowing the model - for Lambertianreflectance it is possible to estimate its albedo (Using pointsthat face the light source), but as the model gets harder sodoes estimating it.

Non-Lambertian Shading Examples

• Healey & Binford, Local shape from specularity, 1988: Usedthe Torrence-Sparrow model, which is a model ofspecularities. Assumes that there is no range issues and fitsthe parameters after segmenting small bright regions.

• Ahmed & Farag, A new formulation for shape from shadingfor non-lambertian surfaces, 2006: Oren-Nayar, learns theparameters by expressing the entire problem as a partialdifferential equation and using a sophisticated solver.

• Approaches such as Vega & Yang, Shading logic: A heuristicapproach to recover shape from shading, 1997; Lee & Kuo,Shape from shading with a generalized reflectance mapmodel, 1997; only ever need to evaluate the BRDF, andcalculate its differential, which can be done using finitedifferences. Any BRDF can be used with them.

Specularity Detection

• Specularities will trip up a SfS algorithm.

• Using a specularity detector as a pre-processing step isextremely helpful.

• A more integrated example is Ragheb & Hancock, Separatinglambertian and specular reflectance components using iteratedconditional modes, 2001; where belief propagation is used tolearn the ratio of Lambertian and specular contribution toeach pixel, in combination with a SfS algorithm.

Variable Surface Parameters

• SfS is usually posed for objects with a single material, e.g. aconstant albedo Lambertian model.

• Variable albedo, or other parameters is usually trivial if known- you just have different parameters for each pixel. Knowing itis the problem.

• The area of intrinsic image research is dedicated to inferringalbedo from an image, though Tappen, Freeman & Adelson,Recovering intrinsic images from a single image, 2005 is aboutas good as it gets.

• Integrated solving for albedo and SfS is demonstrated by Fua& Leclerc, Object-centered surface reconstruction: combiningmulti-image stereo and shading, 1995. It optimises thevertices of a 3D mesh, also making use of stereopsis. Has acost term to prefer piecewise constant albedo.

Perspective

• As cameras tend to use perspective projection assumingorthographic will distort objects badly, particularly when closeto the camera.

• Many of the more recent algorithms given over the followingslides include perspective.

• Algorithms that consider depth can often get perspective forfree.

Lighting falloff

• Prados & Faugeras. Shape from shading: a well-posedproblem, 2005: Demonstrate that if you include perspectiveand model the falloff from the light source the problem is nolonger ill-posed and there is only one answer.

• The only issue is it breaks if lighting falloff is not detectable -you will probably need to use 12bit per channel capture.

• There is an entire sequence of these papers, including ones bydifferent authors. The model was initially limited to the lightsource being at the camera, which has been relaxed. Supportfor alternate shading models has also been added.

• They all formulate the problem as a PDE, and find ’viscositysolutions’.

Multiple lights

•• Prados, Camilli & Faugeras, A unifying and rigorous shape

from shading method adapted to realistic data andapplications, 2006: Consider a light source at the camera,with perspective and lighting falloff, e.g. a flash on a camera.

• Tian, Tsui, Yeung & Ma, Shape from shading for multiplelight sources, 1999: Consider multiple area lights. Theapproach requires the depth of singular points (The pointsthat face a light source) however, which is hardly practical,and then propagate values from these points, which is errorprone.

SfS outside the dark room• Langer & Zucker, Shape from shading on a cloudy day, 1994:

Considers uniform light coming from a hemisphere, e.g. thesky, consequentially pixel brightness is a function of occlusion.Uses a voxel grid in a space carving style algorithm - iteffectively digs pits until they are deep enough to match thesolid angle of visible sky implied by the pixels irradiance.

• A more recent instance of the above is Prados, Jindal &Soatto, A Non-Local Approach to Shape From AmbientShading, 2009, which formulates the problem using (horrible)PDEs.

• Whilst a very cool idea these approaches just don’t work - allof these papers give abstract results, i.e. runs on syntheticimages that are not even recognisable as real objects.

• Note that Brooks & Horn, Shape and source from shading,1985, introduced the idea of a uniform hemisphere torepresent the sky. Whilst they gave an algorithm no resultswere presented, and I am doubtful it could have ever worked.

Inter-reflections

• Nayar, Ikeuchi & Kanade, Shape from interreflections, 1990:The only approach that considers interreflections.

• Reasons that interreflections lighten surfaces, and make themflatter/less concave.

• Therefore if you estimate the surface using any SfS algorithmand then use that to estimate the contribution frominterreflections it will always be an underestimate.

• Therefore iteratively run SfS and interreflection estimation,using the interreflection estimate each time to reduce theinterreflections in the image.

Shadows

• One approach is to use a shadow detection method andswitch off SfS in such areas.

• Deformable model based approaches can model shadowsexplicitly, e.g. Samaras & Metaxas, Incorporating illuminationconstraints in deformable models, 1998.

Inpainting

• As strange as it sounds detecting areas where SfS is going tofail (e.g. texture), deleting them and applying an inpaintingalgorithm to fill in the gaps, before running SfS on the entireimage, is a thing.

• It probably works because inpainting algorithms actually workquite well.

• A recent example is Zhang, Yip, Brown & Tan, A UnifiedFramework for Document Restoration using Inpainting andShape-from-Shading, 2009, which uses this approach to inferthe shape of pages to obtain a flat ’scan’ of the page withoutdamaging the book.

Roadmap

1. The Original Model

2. A Little Bit Of Theoretical Stuff

3. Alternate Assumptions

4. SfS with. . .

. . . Statistical models

• If the object in question belongs to a known class, for which astatistical model is avalible, then the model can be fitted tothe SfS data.

• Atick,Griffin & Redlich, Statistical approach to shape fromshading: Reconstruction of three-dimensional face surfacesfrom single two-dimensional images, 1996: First instance ofthis, using a model of human faces.

• Will Smith (Uni. of York) has done an awful lot of work onhuman faces with SfS, such as his thesis, Statistical MethodsFor Facial Shape-from-shading and Recognition, 2007.

• A further example is Dovgard & Basri, Statistical symmetricshape from shading for 3d structure recovery of faces, 2004,who make use of symmetry.

. . . Image Statistics

• Barron & Malik, Shape, Albedo, and Illumination from aSingle image of an Unknown Object, 2012: Learns everything,by using the SfS constraint alongside various costs thatprovide it with clues as to which SfS solution to choose.

• For albedo estimation they have a smoothness prior and aminimal entropy - the first to encourage regions of contiguousor smoothly varying albedo, the second to encourage albedoreuse around the image.

• For shape they have a prior towards flat objects, towardsoccluding contours having surface orientation in the imageplane, and a term to encourage constant changes in meancurvature (To encourage spherical objects.).

Figure 1 from the previous

. . . (Multi-view) Stereopsis

• Stereo is the obvious thing to combine SfS with:• SfS does well in smoothly shaded areas where stereo has no

information.• Stereo does well in areas where there is texture to match, areas

where SfS fails because we can’t infer the albedo.• SfS tends to provide fine detail, but gets the large scale details

wrong, whilst stereo gets the large details right but is not goodwith the fine details.

• Blake, Zisserman & Knowles, Surface descriptions from stereoand shading,1985 : Discussed the idea of using SfS tointerpolate between sparse points found by stereo. Noalgorithm was given however.

Modular approaches

• Where you run them separately, without integrating them.

• Leclerc & Bobick, The direct computation of height fromshading, 1991: An Sfs algorithm, but they initialise a depthmap using stereo to avoid getting stuck in a nasty localminima.

• Mostafa, Yamany & Farag, Integrating stereo and shape fromshading, 1999: Run both, using sparse Stereo, then subtractthe SfS from the stereo and fit a smooth surface, beforeadding back the SfS.

• Cryer, Tsai & Shah, Integration of shape from shading andstereo, 1995: Take the stereo is low frequency information andSfS is high literally - use the fast Fourier transform to applylow and high pass filters respectively.

Haines & Wilson

• Another modular approach, though it iterates between themso each estimate can inform the other.

• Uses Gaussian belief propagation on a Markov random field tocombine them - Stereo provides the data term, SfS providesthe smoothness term. Error estimates are used to set theStandard deviations of these sources correctly.

• Used Worthington & Hancock for SfS, and a BP basedalgorithm for stereo.

• Includes piece-wise constant albedo estimation, usingsegmentation to define the constant areas.

Results: Me, part 1

Left image. Estimated Albedo

Stereo needle map SfS needle map

Results: Me, part 2

Discrete stereo

result

Smoothed stereo

result

Ground truth Output Zombie

Object centred approaches• Many examples of these - they all involve making some initial

estimate of the model using stereo, then updating the modelto obey SfS derived orientation data. The previous approachis technically one of them, though most of these use multiplecameras rather than just a stereo pair.

• Fua & Leclerc, Object-centered surface reconstruction:combining multi-image stereo and shading, 1995: Probablythe first such algorithm - works exactly as described above.Includes albedo estimation. The model is a mesh, with perface albedo.

• There is a long chain of papers on the same theme - shadowhandling, alternate shading models and specularity handlinghave all been explored.

• The biggest issue with them is initialisation and optimisationmethod - the early approaches did poorly because of weakoptimisation. Initialisation remains an issue - a bad initialmodel will not get better due to SfS.

Wu, Wilburn, Matsushita & Theobalt

• High-quality shape from multi-view stereo and shading undergeneral illumination, 2011.

• The best results I’ve seen with multi-view stereo and SfS. Itsthe deformable model approach with up to date techniques,polished to give good results.

• MVS is doing most of the work however - it provides theinitial model, which has to be accurate.

• It estimates arbitrary lighting - they use spherical harmonicsto represent the lighting environment, in the same form as alight probe - MVS gives surface orientation and the irradianceis a linear combination of the light emitted by each harmonic -a linear equation. It factors in shadows.

• Surface refinement uses the light model and irradiance topush the surface to comply. Adaptively decides if it should usethe information or not, so stereo can remain king where thereis texture.

Figure 1 from the previous

Conclusions

• SfS on its own is (mostly) a silly idea.

• There are valid applications - e.g. book page shape recovery.

• Combined with other cues is can work very well.

• It is an incredibly varied area, and people have worked onsome really crazy formulations - this talk has only scratchedthe surface.

• Light source estimation, shadow and specularity detection andintrinsic images are all closely related areas that have onlybeen touched on.