Interactive Material Replacement in Photographs

Steve Zelinka Hui Fang Michael Garland John C. Hart

Department of Computer ScienceUniversity of Illinois at Urbana–Champaign

Figure 1: Replaced materials on Mount Rushmore. Totaltime, including user interaction, was under one minute.

AbstractMaterial replacement has wide application through-

out the entertainment industry, particularly for post-production make-up application or wardrobe adjustment.More generally, any low-cost mock-up object can be pro-cessed to have the appearance of expensive, high-qualitymaterials. We demonstrate a new system that allows fast,intuitive material replacement in photographs. We ex-tend recent work in object selection and fast texture syn-thesis, as well as develop a novel approach to shape-from-shading capable of handling objects with albedochanges. Each component of our system runs with inter-active speed, allowing for easy experimentation and re-finement of results.

Key words: material replacement, texture synthesis, im-age segmentation, jump maps, shape from shading

1 Introduction

In its simplest form, material replacement has beenwidely used for decades. Chroma-keying, or blue-screenmatting [19], replaces objects of a particular colour (typ-ically blue or green) with new objects or materials, andis a staple of movie and television production. Whilechroma-keying is most often used to replace the back-ground, placing actors in new virtual sets, it is increas-ingly used in special effects production to replace simple

mock-up objects with computer-generated imagery. Theadvantages to this approach over simply directly insertingthe computer-generated imagery is that actors may natu-rally interact with the mock-up object, and the mock-upalso produces lighting effects that are difficult to achievewith computer-generated imagery, such as casting shad-ows into the rest of the scene.

A key limitation of chroma-key systems is that onemust know before-hand which objects and materials onewishes to replace in the final output, since only objectsof a particular distinctive colour may be replaced. In thispaper, we address a more general material replacementproblem. Given any diffuse, possibly textured objectwithin a photograph, and an image swatch of a desiredmaterial, we replace the apparent material of the objectwith the desired material. Thus, the makeup of an actormay be easily adjusted after a photo shoot if errors arediscovered or a different look is desired. Logos or adver-tisements on clothing may be removed, or new coloursand patterns applied. Internet-based vendors can showpeople how user-supplied objects would look using theirfabrics, patterns, or upholstery. Since editing is fast andintuitive, there is no user penalty for experimenting withreplacing different objects or using different textures, sothe system is useable even by novices for personal use.

Textureshop [6] pioneered the idea of using a very sim-ple but fast approach to shape-from-shading to recover arough set of normals for an object in a photograph, andusing these normals to distort texture synthesis over theobject. They note that to some extent, texture variationsmask the inaccuracy of the recovered normals, and so de-spite the low quality of the normals, they are still able toobtain visually pleasing results. However, Textureshopcan be cumbersome to use, since object selection is basedon manual outlining, and Graphcut Textures [9] is usedas the texture synthesis engine. Manually outlining ob-jects from photographs can be tedious, even when aidedwith tools such as intelligent scissors [14]. As GraphcutTextures are relatively slow to synthesize, experimenta-tion with synthesis parameters is discouraged. Further-more, the normal recovery algorithm used in Textureshopis highly inaccurate for all but uniformly coloured, dif-

fuse objects, often requiring a non-trivial amount of userinteraction to fix normal recovery errors.

Our system makes three important, practical contribu-tions over the state of the art:

• Integration of advanced object selection [11] with aGaussian mixture model [18] to allow simple, fast,and intuitive object selection in images.

• A novel shape-from-shading algorithm making ex-plicit use of the Gaussian mixture model of the ob-ject selection phase. The most reliable colour clusterdrives the normal recovery, producing higher qualitynormals and allowing fortexturedinput surfaces.

• The use of jump map-based texture synthesis,which, while lower quality than other algorithms,runs in interactive time, allowing for easy experi-mentation with different textures and texturing pa-rameters, such as scale and orientation.

Each step of the resulting system – selection, normalrecovery, and texture synthesis – runs in interactive time,allowing for more intensive experimentation and refine-ment of parameters. Users can update the selection area,adjust recovered normals, and change textures or textur-ing parameters, and view the effects of their changes in-teractively. Previous methods for material replacementrequire a non-trivial amount of computation time. Thislag time, especially during texture synthesis, is a severedetriment to experimentation. It is also important to notethat once a user is happy with the parameters of the algo-rithm, better quality texturing and rendering may be usedto produce a final high-quality result.

2 Related Work

Recently, graph cut-based techniques [2, 11, 18] havebeen very successfully applied in selecting objects fromimages. Previous methods [14, 15, 7] require the userto trace the boundary of the object directly, which canoften be tedious and error-prone. In contrast, graph cut-based approaches use strokes within or outside the object,and compute the actual boundary by formulating an en-ergy minimization problem to be solved by computing aminimum cost graph cut. Grabcut [18] additionally usesGaussian mixtures to statistically model each region, andrequires only a box to be placed around the object. Lazysnapping [11] instead pre-segments the image to reducethe size of the graph cut problem and achieve real-timeupdates after each brush stroke. While the focus of thesemethods is on hard segmentation, high-quality mattes canbe extracted with both Grabcut or Poisson matting [21].

There are a variety of methods for computing theshape of a 3D object from its shading in a 2D im-

age [8]. Bichsel and Pentland [1] propagate depth fromthe brightest points to the rest of the surface using a min-imum downhill principle. Most similar to ours, Lee andRosenfeld [10] assume local surface regions are spheri-cal patches, yielding fast but poor results. More recently,Cho and Chow [3] iteratively combine surfaces indepen-dently recovered from each colour channel of an image.Oh et al. [16] extract lighting from texture via bilateralfiltering. Zhanget al. [27] survey and compare severalshape-from-shading methods, discovering generally poorquality and limited generality. However, our results rein-force the conclusion of Fang and Hart [6], that high qual-ity normals are not required to produce good synthesizedresults. For objects with regular or near-regular textures,Liu et al. [12] demonstrate excellent results by manuallyfitting a regular lattice to the texture.

Recent texture synthesis methods can be classified intotwo approaches. Pixel-based techniques [5] iterativelycopy pixels by searching for the best match for the cur-rent output neighbourhood within the input image. Patch-based methods [4, 9] instead place entire patches of tex-ture while attempting to find a good boundary betweenthe existing output and the new patch. Methods to tex-ture 3D surfaces have been demonstrated by generalizingboth pixel-based [23, 22, 26, 24] and patch-based tech-niques [17, 20, 13]. For material replacement, we do notnecessarily have a coherent 3D surface, so the more localpixel-based methods are a more natural solution. Tex-ton maps [26], utilize a coarse map of prominent texturefeatures to yield the highest quality results for most tex-tures. Jump map-based synthesis [24] generally producesthe lowest-quality but fastest results, and is the only tech-nique to produce results in interactive time.

3 System Overview

Our material replacement system is composed of threemain components. First, the userselectsan object fromthe input image. Once satisfied with the selection,nor-mal recoverywith our novel colour cluster-based shape-from-shading algorithm approximates the surface normalat each pixel of the object. Finally, these normals are usedto produce a set of distortions to be applied during jumpmap-basedtexture synthesis. In the following sections,we describe each of these steps in detail.

4 Object Selection

Given a photograph containing objects needing mate-rial replacement, the first task is to identify the objectsfrom the background. In our system, we use Lazy Snap-ping [11] as the basic interaction mechanism, as it of-fers the highest interactivity and natural refinement. LazySnapping is a graph cut-based approach, posing the im-

Figure 2: From left to right: original image; textured re-sult with our texture-sensitive procedure; result withouttexture-awareness.

age segmentation problem as energy minimization solvedvia minimum cost graph cut. The energy is a sum of twoterms: a smoothness term, which penalizes segmentationboundaries cutting smooth areas of the image, and an ap-proximation error term. The system statistically approx-imates both the foreground and background, and the ap-proximation error term ensures pixels more likely to be fitby one model over the other are penalized for assumingcontradicting labels. The key idea of Lazy Snapping overalternate approaches is to pre-segment areas of the imageunlikely to be cut in object selection. The graph problemuses the resulting segments, rather than individual pixels,as nodes, so its size (and solution time) is vastly reduced.

To augment Lazy Snapping for material replacement,we first note that as we are concerned with textured, dif-fuse objects, it is especially important that colour imagesare handled well. Lazy Snapping’s approximation modelis ak-means clustering of the pixels that are selected bythe user as definite-foreground or definite-background.We instead use a limited Gaussian mixture. Under a nor-mal Gaussian mixture, each pixel is a weighted combi-nation of the colours in the mixture, each of which ismodelled with a Gaussian distribution. Following Grab-Cut [18], we assume each pixel is drawn from only onecolour of the mixture. In comparison tok-means cluster-ing, the limited Gaussian mixtures include the covarianceof each colour cluster, allowing for a slightly more accu-rate approximation error term during segmentation. Forefficiency, we do not iterate our algorithm, but once seg-mentation is complete, we update the foreground mixtureaccording to the pixels actually selected. As discussedbelow, this mixture is used for better normal recovery.

5 Normal Recovery

Once an object is identified, the next phase of materialreplacement is to recover its normals. We generalizeTextureshop-style normal recovery to textured objects.For simplicity of description, we assume that the lightsource is at the viewer; the extension to other positionsis straightforward, though we require the user to specifythe light position if not at the viewer. The basic idea is

to assume a Lambertian reflection model, so that at thebrightest point in the image, the surface normal points tothe light source, while the darkest point of the image lieson a silhouette and its normal aligns with the reverse im-age gradient. For other pixels, thex andy components ofthe normal are given by the reverse image gradient, whilethe z component is linearly mapped from its luminance,with the mapping fixed by the brightest and darkest points(if lp is the luminance of pixelp, and lmin and lmax arethe minimum and maximum observed luminances, thenzp = (lp − lmin)/(lmax− lmin)). In general, the resultingnormals are very approximate, but after some smoothingare quite useable for our application.

We improve on this approach by making use of theGaussian mixture model for the selected object. Eachpixel has an approximating colour cluster from the Gaus-sian mixture. We first recover the normals over pixels ofthe mostreliable colour cluster in the mixture using theabove approach. Then, normals are propagated across theimage-space boundaries of that cluster to seed the nor-mal recovery of each subsequent colour cluster. After allclusters are processed, some smoothing of the recoverednormals is typically required.

The reliability rc of each colour clusterc is based onits areaac (number of pixels) and dynamic rangevc (lu-minance variation):rc = α ∗ ac +(1−α) ∗ vc, whereα

is a user parameter (for all images in this paper, we usedα = 0.75). Intuitively, we desire the largest cluster as itis most visually important, while we desire the cluster oflargest dynamic range as it likely contains a wide normalvariation in accordance with our assumptions above.

To propagate normals across cluster boundaries, weassume the change in colour is entirely due to albedochange, and the underlying surface remains smooth, al-lowing us to copy normals across boundaries. We theniteratively choose the cluster with the longest boundarywith already-processed clusters. Thex andy componentsof the normals in this new cluster remain the reverse im-age gradient, and thez components are still derived froma linear mapping of luminance. However, each pixel onthe boundary of the cluster is a sample of this mapping(its luminance and normal being copied from the neigh-bouring cluster). We thus use these samples to compute aleast-squares fit of the luminance-to-z mapping.

In some cases, pixels on the boundary between twocolour clusters may be blends of the cluster means. Here,boundary propagation will not work since the boundarypixels will differ significantly from the rest of the colourcluster, and the computed luminance-to-z mapping willbe highly inaccurate. We instead first erode the bound-aries and extrapolate from each cluster to a sharp bound-ary. Generally, we erode until the pixels are sufficiently

Image Plane

nnz A

B

ApproximatingSurface

di

ds

Figure 3: The image plane is viewed from above, and twoneighbouring pixels are in green. The surface betweenthe pixels is approximated by a plane whose normal is n,the average of the recovered normals at each pixel. Thetexture-space distance between the two pixels, ||ds||, isderived from the similar triangles A and B.

close to the mean of the cluster, or until a user-specifiedmaximum erosion is reached.

As can be seen in Figure 2, our results are much betterthan simply recovering the normals over the entire object.Note that we modulate the textured output according tothe luminance of the object in order to preserve its shad-ing. With albedo changes, the means of various colourclusters may have different luminances, and thus the orig-inal luminances are not adequate for this task (this is themain reason the “9” is very clearly visible in the right-most image of Figure 2). Since the luminance is directlyproportional to thez component of the surface normal,we have effectively recovered consistent luminance lev-els for the object as well, and we use these to modulatethe textured output.

6 Texture Synthesis

Once normals for the image have been recovered, the fi-nal step for material replacement is to synthesize new tex-ture. We use texture synthesis based onjump maps[25].While jump map-based texture synthesis produces lowerquality textures than alternate techniques, it is the fastestknown method for synthesizing textures on surfaces.Thus, users can interactively experiment with differenttextures, or texture orientations or scales.

A jump map is a parallel image to a texture whichrecords for each pixel a small set of similar pixels withinthe image. Texture is produced from a seed point by iter-atively copying pixels from the input to the output. Occa-sionally, or near input image boundaries, a jump is madeusing the jump map, and copying proceeds instead fromthe jump destination. On surfaces, the method is simi-lar, walking across vertices instead of pixels. To acco-modate the irregular topology and sampling of surfaces,each edge is individually mapped into a 2D texture spaceoffset according to a supplied orientation field and texturescale. Instead of moving one pixel in the input, movingacross vertices moves the corresponding offset.

We adapt jump map-based texture synthesis for ma-terial replacement in images by treating the image se-lection as a surface with varying edge lengths betweenpixels. Since the synthesis procedure considers only theimmediate neighbours when synthesizing a given pixel,we need not have a physically-realizable surface; this isa distinct advantage over Textureshop, whose Graphcut-based approach required the recovery of consistent sur-face patches. The remaining question is how to computetexture space offsets between neighbouring pixels.

We assume that at each pixel the user has specified ascale,sp, as the number of texture pixels per target imagepixel, and an angleθp, which specifies the orientation ofthe texture with respect to the target image. These aretypically specified globally, but may be gradually variedacross the image as well [26] if desired. From the pre-vious step, we have a recovered normalnp at each pixelas well. On each edge between neighbouring pixels, wefirst compute average normals, scales, and orientations,n, s, θ . We approximate the local surface by a planewhose normal isn. As shown in Figure 3, the lengthof ds, the distance on the surface (equivalently, in texturespace) between these two pixels, can be derived via sim-ilar triangles. The direction ofds is given by projectingthe image-space offset between the pixels (di) onto theapproximating plane. Thus, we have:

ds =s||di ||||nz||

di − (ndi)n||di − (ndi)n||

(1)

Finally, we construct a basis for texture space by rotat-ing the image axes byθ (the user-specified orientation),and projecting them onto the approximating plane. The3D vectords is finally projected into this basis to get thetexture space offset. The resulting graph of offsets is notnecessarily coherent, but as shown in the next section, itis sufficient to give good synthesis results.

Texture synthesis produces texture coordinates at eachpixel of the object. We render a final result by splatting asmall texture patch at each pixel, both filtering the textureand obscuring discontinuities. As in Textureshop, nor-mals may also be recovered for the source texture, andsplatted together with the recovered object normals forbump mapping. Finally, the result is modulated by therecovered luminance to preserve the object’s shading.

7 Results and Discussion

All of the results presented in this paper took about aminute or two to generate, mostly for user interaction.Selection occurs in real-time with each brush stroke, nor-mal recovery takes up to five seconds, and synthesis andrendering takes up to two seconds; both operations scalelinearly with the number of pixels, while normal recovery

Figure 4: Selecting and retexturing multiple objects isfast and efficient with our system.

Figure 5: Jump map-based texture synthesis allowsfor easy experimentation with different textures, texturescales, and texture orientations. As shown in the video,each image is produced with virtually no lag time.

also depends on the number of clusters in the Gaussianmixture. These timings, as well as the real-time capturesin the video, are on a 1.8 GHz G5 machine. As shownin Figures 1, 4 and 5, as well as the accompanying video,the speed of the system easily allows for selecting mul-tiple objects from a scene, using multiple textures, andvarying texturing parameters. Updates to the result occurinteractively, usually under a second.

We demonstrate results of material replacement on ob-jects with albedo changes in Figures 2 and 6. Note that inthe dancing image (Figure 6, bottom), the high-frequencycolour changes in the shirt of the left dancer are inaccu-rately treated as a single colour cluster by our system,producing some brightness artifacts, but the result is stillquite convincing. With Textureshop, the recovered lumi-nances are incorrect (Figure 7).

The main limitations of our system are shown in Fig-ure 8. Clustering can succeed even on high frequencyalbedo changes, as with the patterned shirt example, butthe resulting clusters are not coherent enough to providegood shape-from-shading results. Thus, the luminance

Figure 6: Material replacement with albedo changes.

Figure 7: Results using Textureshop-style recovery. Thetextures are severely distorted and the luminance incor-rect in areas with different albedo.

Figure 8: Material replacement failures. Left to right:original image, colour clusters, recovered brightnesses.

recovery is incorrect, revealing the original pattern in-stead of removing it.

A more subtle problem can cause the clustering itself tofail. In the dish cloth example, which uses 3 clusters, theshading variation outweighs the albedo changes, so pixelswith different albedo are incorrectly clustered together,causing later steps in the algorithm to fail. Clustering indifferent colour spaces can help, but no one colour spaceis good over a wide variety of images. We expect a morefruitful approach is for the user to provide some sort ofextra information to make the recovery more accurate.From our results, simply specifying the expected numberof colour clusters is not sufficient (as using too many clus-ters can lead to the same problem, we leave the numberof clusters as a user parameter).

Another limitation of our approach is that since nor-mals are examined locally, two parts of the surface atvarying depth but with similar normals will be given tex-ture at the same scale, and may appear to be at the samedepth after material replacement. This can be fixed bymanually altering the scale of the texture appropriately,but a more automatic solution would be preferable.

8 Conclusion

We have presented a new system for interactive materialreplacement. Our novel normal recovery procedure reli-ably estimates the normals of diffuse objects even withalbedo changes, broadly extending the utility of mate-rial replacement techniques. Integrating Lazy Snappinginto the system vastly reduces the tedium of object selec-tion. Jump map-based texture synthesis produces slightlylower-quality textured results, but produces results inter-actively, allowing for real-time texture brushing and veryquick overall texturing. By achieving interactive speedfor all elements of the system, we remove the perceivedpenalty for experimentation with different selections, tex-tures, and rendering options.

Acknowledgements

We would like to thank the anonymous reviewers for allof their helpful comments. This research was supportedin part under grants from the NSF (CCR-0086084 andCCF-0121288).

References[1] M. Bichsel and A. P. Pentland. A simple algorithm for shape from shading.

Computer Vision and Pattern Recognition, pages 459–463, 1992.

[2] Yuri Boykov and Marie-Pierre Jolly. Interactive graph cuts for optimalboundary and region segmentation of objects in n-d images. InProceedingsof the Eighth IEEE International Conference on Computer Vision, volume 1,pages 105–112, 2001.

[3] S. Y. Cho and T. W. S. Chow. A new colour 3d sfs methodology using neu-ral based colour reflectance models and iterative recursive method.NeuralComputation, 14(11):2751–2789, 2002.

[4] Alexei A. Efros and William T. Freeman. Image quilting for texture synthe-sis and transfer. InProceedings of SIGGRAPH 2001, pages 341–346, LosAngeles, CA, August 2001. ACM SIGGRAPH.

[5] Alexei A. Efros and Thomas K. Leung. Texture synthesis by non-parametricsampling. InIEEE International Conference on Computer Vision, pages1033–1038, Corfu, Greece, September 1999. IEEE Computer Society.

[6] Hui Fang and John C. Hart. Textureshop: texture synthesis as a photographediting tool.ACM Transactions on Graphics, 23(3):354–359, 2004.

[7] Michael Gleicher. Image snapping. InProceedings of SIGGRAPH 95, pages183–190, 1995.

[8] Berthold K.P. Horn. Height and gradient from shading.International Jour-nal of Computer Vision, 5(1):37–75, 1990.

[9] Vivek Kwatra, Arno Schodl, Irfan Essa, Grek Turk, and Aaron Bobick.Graphcut textures: Image and video synthesis using graph cuts.ACM Trans-actions on Graphics, SIGGRAPH 2003, 22(2):277–286, July 2003.

[10] C.H. Lee and A. Rosenfeld. Improved methods of estimating shape fromshading using the light source coordinate system.Artificial Intelligence,26(2):125–143, 1985.

[11] Yin Li, Jian Sun, and Chi-Keung Tang abd Heung-Yeung Shum. Lazy snap-ping. ACM Transactions on Graphics, 23(3):303–308, 2004.

[12] Yanxi Liu, Wen-Chieh Lin, and James Hays. Near regular texture analysisand manipulation.ACM Transactions on Graphics, 23(3):368–376, 2004.

[13] Sebastian Magda and David Kriegman. Fast texture synthesis on arbitrarymeshes. InProceedings of the Eurographics Symposium on Rendering 2003,pages 82–89, Leuven, Belgium, June 2003. Eurographics Association.

[14] E. Mortensen and W. Barrett. Intelligent scissors for image composition. InProceedings of SIGGRAPH 95, pages 191–198, 1995.

[15] E. Mortensen and W. Barrett. Tobogan-based intelligent scissors with a fourparameter edge model. InProceedings of the IEEE Conference on ComputerVision and Pattern Recognition, volume 2, pages 452–458, 1999.

[16] Byong Mok Oh, Max Chen, Julie Dorsey, and Fredo Durand. Image-basedmodeling and photo editing. InProceedings of SIGGRAPH 2001, pages433–442, Los Angeles, CA, August 2001. ACM SIGGRAPH.

[17] Emil Praun, Adam Finkelstein, and Hugues Hoppe. Lapped textures. InProceedings of SIGGRAPH 2000, pages 465–470, New Orleans, LA, July2000. ACM SIGGRAPH.

[18] Carsten Rother, Vladimir Kolmogorov, and Andrew Blake. Grabcut: inter-active foreground extraction using iterated graph cuts.ACM Transactionson Graphics, 23(3):309–314, 2004.

[19] A. R. Smith and J. F. Blinn. Blue screen matting. InProceedings of SIG-GRAPH 96, pages 259–268, 1996.

[20] Cyril Soler, Marie-Paule Cani, and Alexis Angelidis. Hierarchical patternmapping. InProceedings of SIGGRAPH 2002, pages 673–680, San Anto-nio, TX, July 2002. ACM SIGGRAPH.

[21] Jian Sun, Jiaya Jia, Chi-Keung Tang, and Heung-Yeung Shum. Poissonmatting.ACM Transactions on Graphics, 23(3):315–321, 2004.

[22] Greg Turk. Texture synthesis on surfaces. InProceedings of SIGGRAPH2001, pages 347–354, Los Angeles, CA, August 2001. ACM SIGGRAPH.

[23] Li-Yi Wei and Mark Levoy. Texture synthesis over arbitrary manifold sur-faces. InProceedings of SIGGRAPH 2001, pages 355–360, Los Angeles,CA, August 2001. ACM SIGGRAPH.

[24] Steve Zelinka and Michael Garland. Interactive texture synthesis on sur-faces using jump maps. InProceedings of the Eurographics Symposium onRendering 2003, pages 90–96, Leuven, Belgium, June 2003. EurographicsAssociation.

[25] Steve Zelinka and Michael Garland. Jump map-based interactive texturesynthesis.ACM Transactions on Graphics, 23(4):930–962, 2004.

[26] Jingdan Zhang, Kun Zhou, Luiz Velho, Baining Guo, and Heung-YeungShum. Synthesis of progressively variant texture on arbitrary surfaces.ACMTransactions on Graphics, SIGGRAPH 2003, 22(2):295–302, July 2003.

[27] Ruo Zhang, Ping-Sing Tsai, James Edwin Cryer, and Mubarak Shah. Shapefrom shading: A survey.IEEE Transactions on Pattern Analysis and Ma-chine Intelligence, 21(8):690–706, 1999.

[28] Qinfen Zheng and Rama Chellappa. Estimation of illuminant direction,albedo and shape from shading.IEEE Transactions on Pattern Analysis andMachine Intelligence, 13:680–702, 1991.