Towards High Fidelity Monocular Face Reconstruction with Rich Reflectanceusing Self-supervised Learning and Ray Tracing

Abdallah Dib1* Cedric Thebault1∗ Junghyun Ahn1∗ Philippe-Henri Gosselin1

Christian Theobalt2 Louis Chevallier1

1InterDigital R&I 2Max-Planck-Institute for Informatics

Figure 1. A deep neural network trained in a self-supervised manner with a differentiable ray tracer estimates a complete set of facialattributes – 3D head pose, geometry, personalized albedo (diffuse and specular) from unconstrained monocular image. Reconstructionbased on these attributes enables a variety of applications, such as relighting.

AbstractRobust face reconstruction from monocular image in

general lighting conditions is challenging. Methods com-bining deep neural network encoders with differentiablerendering have opened up the path for very fast monocularreconstruction of geometry, lighting and reflectance. Theycan also be trained in self-supervised manner for increasedrobustness and better generalization. However, their dif-ferentiable rasterization-based image formation models, aswell as underlying scene parameterization, limit them toLambertian face reflectance and to poor shape details.More recently, ray tracing was introduced for monocu-lar face reconstruction within a classic optimization-basedframework and enables state-of-the art results. However,optimization-based approaches are inherently slow and lackrobustness. In this paper, we build our work on the afore-mentioned approaches and propose a new method thatgreatly improves reconstruction quality and robustness ingeneral scenes. We achieve this by combining a CNN en-coder with a differentiable ray tracer, which enables usto base the reconstruction on much more advanced per-

*Equal contribution

sonalized diffuse and specular albedos, a more sophisti-cated illumination model and a plausible representation ofself-shadows. This enables to take a big leap forward inreconstruction quality of shape, appearance and lightingeven in scenes with difficult illumination. With consistentface attributes reconstruction, our method leads to prac-tical applications such as relighting and self-shadows re-moval. Compared to state-of-the-art methods, our resultsshow improved accuracy and validity of the approach.

1. Introduction

Fast and accurate image-based face reconstruction hasmany applications in several domains including rig basedrealistic videoconferencing, interactive AR/VR experiencesand special effects for professionals like facial attributemanipulation/transfer or relighting. Also, supporting un-constrained pose and in-the-wild capture conditions with-out specific hardware, such as multi-view setup, allows forenhanced flexibility and extended applicability. However,captured images reflect the complex interaction betweenlight and faces including shadows and specularities, whichposes a real challenge for face reconstruction. Speed is also

12819

a key factor for interactive scenario and other real-time usecases. Great multi-view approaches exist ([1, 2, 3, 4, 5]),but they may not be easily applied in many applicationssuch as VR or movies / special effects. Significant progresshas been made on monocular face reconstruction wheremost methods resort to some form of parametric prior;high-quality analysis-by-synthesis monocular optimizationmethods exist ([6, 7, 8]), but besides being rather slow, theywould fail for difficult head poses and lighting conditions.More recently, and to improve this robustness against light-ing conditions, [9] introduced ray tracing for face recon-struction within an optimization-based framework. But thequality of their reconstruction remains sensitive to the land-marks used for initialization.

Real-time analysis-by-synthesis approaches have alsobeen presented, however they often sacrifice reconstructiondetails. To increase reconstruction efficiency, CNN basedapproaches ([10, 11, 12, 13]) that directly regress 3D recon-struction parameters from images have been investigated.To overcome the challenge of creating large amounts of la-beled data, while enabling reconstruction on the basis ofmeaningful scene parameters, methods combining CNNswith differentiable image formation models trained in a self-supervised way have been presented ([10, 11, 14]). Theyenable reconstruction performance in the range of millisec-onds, and can be applied to more general scenes and sub-jects ([11, 14]). However, even the best of these highly ef-ficient monocular reconstruction methods fall short of thequality and robustness requirements expected in profes-sional visual effects (VFX) pipelines. They rely on sim-ple parametric diffuse reflectance models based on low-frequency spherical harmonics (SH) illumination model,whereas more detailed models would be needed to recon-struct at the level of quality that is typically required. Theinability to model self-shadows is also a prime reason fortheir instability under challenging scene conditions.

To overcome these limitations, we present a new ap-proach that is the first to jointly provide the following ca-pabilities: it enables monocular reconstruction of detailedface geometry, spatially varying face reflectance and com-plex scene illumination at very high speed on the basis ofsemantically meaningful scene parameters. To achieve this,our model resorts to a parametric face and scene model thatrepresents geometry using 3DMM statistical model, illumi-nation as high-order spherical harmonics and reflectancemodel with diffuse and specular components. Our newCNN-based approach can be trained in a self-supervisedway on unlabeled image data. It features a CNN encoderprojecting the input image into the parametric scene rep-resentation. We also use an end-to-end differentiable raytracing image formation model which, in contrast to ear-lier rasterization-based models, provides a more accuratelight-geometry interaction and is able to synthesize images

with complex illumination accounting for self-shadows. Tothe best of our knowledge, this is the first time a differen-tiable ray tracer is used for deep-based face reconstructionin an inverse-rendering setup. While ray tracing enables[9] to improve the state of the art, their method is basedon costly and slow iterative optimization and the final re-construction remains sensitive to the quality of the land-marks. Our method overcomes these limitations: it ab-sorbs the complexity of ray tracing at training time, achievesrobust and competitive results with near real-time perfor-mance, and, being completely independent of landmarksat test time, is more suitable for in-the-wild conditions.Finally, with an appropriate training strategy, our methodis the first self-supervised method to achieve robust facereconstruction in challenging lighting conditions and cap-tures person-specific shadow-free albedo details (such asfacial hair or makeup) not restricted by the 3DMM space.Our rich and consistent facial attributes reconstruction natu-rally allows various types of applications such as relighting,light/albedo edit and transfer. Our comparison with recentstate-of-the-art methods shows improved robustness, accu-racy and versatility of our method.

2. Related works

While methods such as [2, 1, 3, 4, 5] produce high-quality face reconstruction from multi-view camera and/orfrom multi-light illumination setup, they are not applicablefor in-the-wild images. We focus therefore on monocularimage-based face reconstruction approaches that do not re-quire any external hardware setup.

Statistical morphable model To make the highly ill-posedproblem of monocular face reconstruction tractable, statisti-cal priors have been introduced [15] such as 3D morphablemodels (3DMMs) [16, 17, 18]. 3DMM was the main build-ing block for most recent approaches because of its effi-ciency. However, 3DMM limits face reconstruction to alow-dimensional space and does not capture person spe-cific details (such as beards and makeup), and its statisti-cal albedo prior bakes some illumination in it. Recently, toovercome these limitations, [19] proposes a drop-in replace-ment to 3DMM statistical albedo model with more sophis-ticated diffuse and specular albedo priors. In this work, webase our reconstruction on the 3DMM geometry and albedoprior (diffuse and specular) of [19] and we learn an incre-ment, similarly to [13, 9], on top of the albedo prior to cap-ture more person-specific details outside of the statisticalprior space.

Optimization-based approaches [8, 6, 7, 20, 21, 9] useoptimization formulation for face (geometry and diffusealbedo) reconstruction. In [8, 6, 7, 9], a 3DMM based sta-tistical priors act as optimization regularizer while minimiz-ing a photo-consistency loss function. While such methods

12820

work well for controlled scene conditions, often they do notgeneralize well for in-the-wild images scenarios and can becomputationally expensive.

Illumination and reflectance models The aforementionedmethods generally use low-order spherical harmonics (SH)to model light and assume Lambertian surface, while ourmethod uses higher-order SH in order to better model lightinteraction with non-Lambertian surface (with diffuse andspecular). While [22] extracts diffuse and specular albe-dos from a single image using SH illumination, they do notexplicitly model self-shadows and show only results in con-trolled conditions. [9] introduced a novel virtual light stageto model area lights and used a Cook-Torrance BRDF (dif-fuse, specular and roughness) to model skin reflectance.

Differentiable rendering A simple and efficient vertex-wise differentiable rendering is proposed in [10]. Twoshortcomings of this approach are the simple LambertianBRDF and its inability to handle self-shadows. To addressthese limitations, [23, 9] introduce the use of differentiableray tracing for face reconstruction. It can handle more com-plex illumination and BRDF models, and naturally accountsfor self-shadows. However, in addition to being computa-tionally expensive, differentiable ray tracing exhibits noisygradients on the geometry edges as they are sampled byvery few points (solutions, such as [24, 25], exist but re-main computationally expensive).

Deep learning based approaches When trained on largecorpus, convolution neural networks (CNNs) are wellproven face deep representation decoders and encoders suchas [26, 27]. Several novel approaches have been pro-posed ([12, 13, 10, 11, 14, 28, 29]). [10] was among thefirst to show the use of self-supervised autoencoder-likeinverse rendering based architecture to infer semantic at-tributes. These unsupervised learning approaches have thepotential to leverage vast amount of face images. Theyare however limited by their differentiable rasterizer, re-lying on a low-order spherical harmonics parametrizationand a pure-Lambertian skin reflectance model, which lim-its their performance and reconstruction quality under chal-lenging lighting conditions. [10] relies on diffuse albedoobtained from 3DMM, which restricts generalization of thetrue diversity in face reflectance. [11] uses self-supervisionto learn a corrective space to capture more person-specificalbedo/geometry details outside of the 3DMM space. Morerecently, [30] learns user-specific expression blendshapesand dynamic albedo maps by predicting personalized cor-rections on top of a 3DMM prior. However, these meth-ods do not separate diffuse and specular albedos, whichget mixed in their final estimated reflectance. They alsodo not show reconstruction under challenging lighting con-ditions. Our method uses self-supervision to train a CNNand extracts personalized diffuse and specular albedos, out-

side of the 3DMM space. In contrast to the aforementionedmethods, our method uses high-order SH for better light ap-proximation and an image formation layer based on a dif-ferentiable ray tracing that handle advanced lighting andself-shadows. All this contributes to a robust face recon-struction even in scenes with challenging lighting condi-tions. Our loss functions and training strategy allow us toobtain personalized diffuse and specular albedos, outsidethe span of 3DMM, with faithful separation between themavoiding baking residual self-shadows in the albedo. Morerecently, [12, 13] use supervised training to learn a non-linear 3DMM model to produce more detailed albedo andgeometry. [31, 32, 33] show vast improvements in geome-try reconstruction. [34] improves on these approaches fur-ther by inferring mesoscopic facial attributes given monoc-ular facial images, an attribute we do not model in our finalreconstruction. Finally, [35] and [36] use image-to-imagetranslation networks to directly regress diffuse and specu-lar albedos but they do not model light. Self-shadows canbe observed in the albedo they obtain, whereas we modelself-shadows implicitly. Additionally, these methods re-quire ground-truth data to train the generative model, whichis not easy to acquire.

3. MethodOur method is inspired by the work of [9], which

achieves face reconstruction with personalized albedos out-side the statistical albedo prior space and in challenginglighting conditions using ray-tracing within an optimizationframework. While their method achieves state-of-the art re-sults, it lacks robustness against initial starting point and isinherently slow, which makes their method not suitable forin-the-wild conditions. Our method overcomes these lim-itations by leveraging the generalization capacity of self-supervised learning.

Our method is composed of two stages as shown in Fig-ure 2. In stage I, an input image IR is passed through a deepnetwork E1 (a pre-trained ResNet-152) followed by a fullyconnected layer to predict a semantic attribute vector χ for3DMM shape (α), expression blendshapes (δ), camera pose(ϕ = {R, T}, composed of rotation and translation), light(γ) modeled with 9 spherical harmonics bands, albedo prior(β), which encodes diffuse and specular priors from [19].Statistical diffuse D and specular S textures are obtainedfrom β. These parameters are fed to a differentiable raytracer to generate a ray traced image IS

1 . This encoder Eis trained end-to-end in a non-supervised manner to obtaina ‘base’ reconstruction. At this level, the estimated albe-dos only capture low-frequency skin attributes. In stage II,to enhance these albedo priors, we train two additional de-coders, D1 and D2, in a self-supervised way to estimate

1For the sake of conciseness, in the rest of the paper, we use E to denotethe combination of this encoder with the fully connected layer.

12821

Figure 2. Method overview: A network E regresses semantic face attributes (shape, expression, light, statistical diffuse and specularalbedos and camera). Two additional networks D1,D2 are used to estimate increments ∆d,∆s on top of the statistical albedos to capturepersonalized reflectance (diffuse and specular) outside of the statistical prior space. A differentiable ray tracer is used for image formation.

diffuse ∆d and specular ∆s increments to be added on topof the previously estimated textures, D and S, respectively.The resultant textures, D and S, are used to generate a newimage IS

2 . The challenges is to avoid mixing the diffuse andspecular parts and baking unexplained shadows (residual)in the personalized albedos. In the next section, we brieflydescribe the scene attributes used for image formation, andthen we discuss the training of the different networks.

3.1. Scene attributes

Geometry We use [16, 37]’s statistical face model,where identity is given by e = as + Σsα. e a vector offace geometry vertices with N vertices. The identity-shapespace is spanned by Σs ∈ R3N×Ks composed of Ks = 80principal components of this space. α ∈ RKs describesweights for each coefficient of the 3DMM and as ∈ R3N isthe average face mesh. We use linear blendshapes to modelface expressions over the neutral identity e. v = e + Σeδ,where v is the final vertex position displaced from e byblendshape weights vector δ ∈ RKe and Σe ∈ R3N×Ke

composed of Ke = 75 principal components of the expres-sion space.

Reflectance A simplified Cook-Torrance BRDF [38, 39],with a constant roughness term, is used to model skin re-flectance. In contrast to Lambertian BRDF, the Cook-Torrance BRDF can model specular reflections, and thusdefines for each vertex vi a diffuse ci ∈ R3 and a specularsi ∈ R3 albedos. The statistical diffuse albedo c ∈ R3N

is derived from 3DMM as c = ar + Σrβ, where Σr ∈R3N×Kr defines the PCA diffuse reflectance with Kr = 80and β ∈ RKr the coefficients. ar is the average skin dif-fuse reflectance. Similarly to [9], we employ the statisticalspecular prior introduced by [19] to model the specular re-flectance: s = ab + Σbβ where Σb ∈ R3N×Kr definesthe PCA specular reflectance. ab is the average specularreflectance. We use the same coefficients β to sample forthe diffuse and specular albedos as suggested by [19]. In

unwrapped (UV) image texture space, D ∈ RM×M×3 andS ∈ RM×M×3 are the statistical diffuse and specular albe-dos, with M ×M being the texture resolution.

Illumination Specular reflections are much more sensitiveto light direction than diffuse reflections. So while in the lit-erature 3-order spherical harmonics have been widely usedin conjunction with Lambertian BRDF, we resort to nine SHbands with the Cook-Torrance model. We actually foundSH to be better suited to our deep learning framework thanan explicit spatial representation as the one used by [9].We show in section 5 that high-order SH produces bettershadows estimation than low-order SH. To use with the raytracer, an environment map of 64 × 64 is derived from thislight representation. We define γ ∈ R9×9×3 as the light co-efficients to be predicted by the network E.

Camera We use the pinhole camera model with rotationR ∈ SO(3) and translation T ∈ R3. We define ϕ = {T,R}the parameters predicted by E.

3.2. Training

First stage (Base reconstruction) In this stage, we usestatistical geometry and albedo priors to obtain a first esti-mation of geometry, reflectance, light and camera attributes.We define χ = {α, δ, ϕ, γ, β} as the semantic attribute vec-tor used by the differentiable ray tracer to obtain a ray tracedimage IS

1 . We train a deep encoder E to directly regress χ.We use a pixel-wise photo-consistency loss between IS

1 andthe input image IR:

Eph(χ) =∑i∈I

|pSi (χ)− pRi | (1)

Here, pSi , pRi ∈ R3 are ray traced and real image pixel

colors, respectively. Rendered pixel colors are given bypSi = F(χ), where F is the Monte Carlo estimator of therendering equation [40]. We also define a sparse landmarkloss Eland, which measures the distance between the pro-jection of L = 68 facial landmarks and their corresponding

12822

Figure 3. Yellow box: Base reconstruction (Stage I section 3) with the estimated statistical albedo priors. Red box: Final reconstruction(Stage II section 3) with the final albedos and light.

pixel projections zl on the input image (more details in [9]).These landmarks are obtained using an off-the-shelf land-marks detector [41]. During training, we minimize thefollowing energy function:

argmin(α,δ,ϕ,γ,β)

Ed(χ) + Ep(α, β) + Eb(δ) (2)

Ed(χ) = Eph(χ) + α1 Eland(χ) and Ep is the statisticalface (shape and albedo) prior ([9]) that regularizes againstimplausible face geometry and reflectance deformations.Eb(δ) is a soft-box constraint that restricts δ to range [0, 1].Second stage (personalized albedos) In Figure 3 (yellowboxes) we show the final reconstruction together with theestimated statistical albedos (D and S) obtained by the basereconstruction of stage I. This result misses subject specificskin features, like beards and make-up, as 3DMM does notsupport them. We aim to personalize the albedo. However,the challenges are to avoid mixing the diffuse and specularparts and to avoid baking unexplained shadows (residual)in the personalized albedos. We train two additional net-works D1 and D2, which take as input the latent space of Eand estimate a diffuse and specular increments ∆d and ∆s

which are added to D and S, respectively. D = D+∆d andS = S+∆s are then used by the differentiable ray tracer togenerate a new synthetic image IS

2 . Predicting an incrementon statistical texture priors instead of directly estimating acomplete texture is important to force E to produce a goodalbedo prior while at the same time estimating a good in-crement over these priors. We define χ = {α, δ, ϕ, γ, D, S}which is used in the photo-consistency loss (eq 1). The fol-lowing energy function is minimized:

argmin(∆d,∆s)

Ed(χ) + w1(Es(D) + Es(S))+

w2DEc(D,D) + w2SEc(S,S)+w3(Em(D) + Em(S))+w4(Eb(D) + Eb(S)) (3)

where Eb is the soft box constraint that restricts the albedosto remain in an acceptable range [0, 1]. Em is a constraintterm which ensures local smoothness at each vertex, with

respect to its first ring neighbors in the UV space, and isgiven by Em(A) =

∑xj∈Nxi

||(A(xj) − A(xi)||22, whereNxi is 4-pixel neighborhood of pixel xi.

We use regularization similar to [13, 9] to preventresidual shadows to leak into the albedos. Es(A) =∑

i∈M |A(xi))−flip(A(xi)))|1 is a symmetry constraint,where flip() is the horizontal flip operator. Ec(A,A)is a consistency regularizer, which weakly regularizes Awith respect to the previously optimized albedo A basedon the chromaticity κ of each pixel in the texture given byEc(A,A) =

∑i∈M |κ(A(xi))− κ(A(xi))|1.

Figure 4. Row I: Comparison of the estimated light using 3 and9 SH bands. Row II: Comparison of a single network vs. ourdual-network approach to estimate albedo increments. Row III:Comparison of results obtained with and without the symmetryand consistency regularizers. Row IV: Comparison of base vs.final geometry reconstruction. Row V: comparison of vertex-basedrenderer and ray tracing.

Training strategy Obtaining faithful and personalized dif-fuse and specular albedos (outside of the statistical priorspace) in uncontrolled lighting conditions is an ill-posedproblem. For instance, one can easily overfit the diffusealbedo increment ∆d to explain the remaining informationin the input image. To avoid this case, we proceed with the

12823

following strategy to separate the diffuse and specular albe-dos: After training E for few epochs, we fix E and starttraining D1 and D2 with a high regularization weight w2D

for the diffuse consistency regularizer in order to keep Dcloser to D. Next, we progressively relax the w2D con-straint during the training to let the diffuse increment ∆s

capture more details. Finally, we train for all networksjointly (E, D1 and D2). So, for a given image, we generatetwo images IS

1 and IS2 and minimize the energy functions

in 2 and 3 respectively and back-propagate over the wholeattributes. This last step is important to stress E to producebetter priors, light and pose while at the same time pushingD1 and D2 to capture more details in the albedos.

Figure 5. Comparison against [11] (subjects from authors paper).

4. Results

For training, we used a total of 250K images, partiallyfrom CelebA dataset [42]. For E, we use a pre-trainedResNet-152, and both D1 and D2 networks use a cascadeof 7 convolution layers. Ray tracing is based on the methodof [25]. Because ray tracing is very memory consuming, weuse 256×256 as resolution for the texture and the input im-ages. The inference takes 54 ms (47 ms for E and 7 ms forD1,D2). During training, we use 8 samples per pixels forray tracing the images. Other implementation details canbe found in the supplementary material (section I). Pleasenote that images used in the figures were not used for train-ing. The attributes estimated by our method are compatiblewith most rendering engines, nevertheless all results shownin the paper were rendered using ray tracing.Figure 3 (red boxes) shows reconstruction results from in-the-wild images for different subjects with various face at-tributes, head pose and lighting conditions. Our methodsuccessfully captures the facial hair and the lipstick – out-side of statistical albedo prior space – for subjects on the leftand in the middle respectively. Subject on the right is underchallenging lighting conditions. Our method robustly esti-mates meaningful albedos and avoid baking residual shad-ows in the final albedos. We also show on Figure 3 theeffectiveness of albedo personalization provided by the net-works D1 and D2 (red box) to refine the estimated statis-tical priors obtained by E (yellow box) from outside of thestatistical albedo prior space. More results are shown onFigure 1 and in supplementary material (section 5).

Figure 6. Comparison against [9] (right subject: authors paper).

5. AblationLow-order SH parameterization In this experiment,

we compare 9-order with 3-order SH. Estimated illumi-nations are shown in Figure 4 (first row). This showsthat using higher-order spherical harmonics results in bet-ter shadow reconstruction, which prevents baking residualshadows in the albedo. In a similar experiment, 9-order SHhas proven itself to be slightly better than 7-order.

Dual networks In this experiment, we trained a single net-work to regress diffuse ∆d and specular ∆s increments. Asshown in Figure 4 (second row), a single network producespoor albedo separation compared to our approach that usestwo separate networks, possibly because the diffuse andspecular components capture different skin features that in-terfere when using a single network.

Symmetry and consistency regularizers Figure 4 (row III)shows the importance of the regularizers used in equation 3.In fact, because our light model may not always perfectlycapture the real shadows, these regularizers prevent frombaking residual shadows in the albedo.

Joint training Training E jointly with D1 and D2 (referto section 3) improves the final geometry as shown in Fig-ure 4 (row IV). The ‘final’ mesh better fits the input imagecompared to the ‘base’ mesh obtained from the base recon-struction (when training E only).

Vertex-based renderer In this experiment, we train thesame architecture, except that we used a ‘vertex-based’ ren-derer, with the same illumination model (9-order SH) [43]and the simplified Cook-Torrance BRDF (more details insupp. material, section 2). Figure 4 (last row) shows that raytracing produces smoother and natural projected-shadows(especially around the nose). This is because ray tracingcan naturally models self-shadows while vertex-based ren-derer cannot. For instance, spherical harmonics (SH) coef-ficients are converted to an environment map (EM) for theuse with ray tracing. Each pixel in EM acts as a light sourceat infinity. Shadows rays (rays shot from a surface point (P)towards a light source sampled from EM) are used to calcu-late a visibility mask for P. On the other hand, vertex-basedrenderers do not naturally model visibility of light sources.Work such as [44] tries to solve for this.

6. ComparisonIn the next, we compare, qualitatively and quantitatively,

our method against recent methods and to the ground-truthDigital Emily project and NoW benchmark [45].

12824

Figure 7. Comparison between methods - [35], [36], [13], [19], and ours. The red arrows indicate the shadow baked in diffuse albedo.

Visual comparison Figure 5 shows comparison results ofour method against [11] (results and images are taken fromauthors paper). Both methods achieve visually compara-ble results on the final reconstructed images. However, ourmethod separates diffuse and specular albedos while [11]only estimates a reflectance map, which mixes diffuse andspecular components. This makes our method more suitablefor applications such as relighting.

Figure 6 shows comparison against [9]. For the left sub-ject, their method infers incorrect shape (around the mouth),head pose and illumination estimation. For the right subject,their estimated shape is inaccurate (right part of the head).This illustrates the sensitivity of [9] to the landmarks quality(as reported by the authors). Our method, completely inde-pendent of landmarks, is more robust to the lighting condi-tions and generates more convincing results. In the supp.material (section 4), we show that our method achieves vi-sually comparable results ([9] being slightly better) whilebeing an order of magnitude faster (54 ms vs. 6.4 min). Wenote, that similarly to [46], combining both approaches byusing our predicted attributes as initialization for the opti-mization produces better reconstruction quality. Finally, wenote that [9] can capture shadows projected by point lights,while our method can only handle lights at infinite distance;nevertheless, our training strategy, together with carefullydesigned loss functions helps our method from baking un-explained shadows in the final personalized albedos at theexpense of some albedo details (please refer to supp. mate-rial for more results in challenging lighting conditions).

Figure 7 shows comparison against [35, 36, 13, 19]. Re-sults for [19] are obtained using their open-source imple-mentations. For [13, 35, 36], results are from original au-thors. [35] and [36] methods do not estimate light and thusfinal reconstruction is not available. Because [35] does notmodel light, they bake some shadows in the diffuse albedofor both subjects. [19] estimates light using 3-order SHbut their reconstruction is bounded by the statistical albedospace and cannot capture personalized albedos outside ofthis space. Compared to [13], our method has ‘visually’better light estimation and smoother shadows (for first sub-ject). Additionally, their diffuse and geometry have someartifacts visible around the nose. Also, their method does

Table 1. Vertex position and normal error µ and standard deviationσ for each method on all subjects.

[35] [36] [13] [9] OursPosition error µ (cm) 0.236 0.184 0.176 0.174 0.181Position error σ (cm) 0.114 0.072 0.065 0.064 0.069Normal error µ (rad) 0.152 0.156 0.159 0.139 0.148Normal error σ (rad) 0.051 0.043 0.046 0.046 0.048

not estimate the specular component. [36] produces con-vincing shadow-free diffuse albedo. However, they do notestimate light and head pose.

Geometric comparison We evaluate the quality of the re-constructed geometry on 23 images with 3D ground truth(GT) mesh from 3DFAW [47], AFLW2000 fitting package[48][49][50], and from the wikihuman project2 [51]. Wecompared our method to [35, 36, 13, 9] (Figure 8 and Table1), using vertex position and normal errors calculated withrespect to the GT meshes. Table 1 reports, for each methodon all subjects, the average error (µ) and standard deviation(σ) for vertex position and normal direction. For vertex-position error, [9] scores the best average error (0.174),while [13] reports the second lowest measure. Compar-ing to the above methods, our method shows similar per-formance (0.181). We note that [13] learns a non-linear3DMM model to improve the geometry while our method,which only uses 3DMM geometry, achieves comparable re-sults. Our score is also on par with optimization-basedmethod [9], while being order of magnitude faster. For nor-mal error, our method reports the second-best error (0.148),very close to [9]. Also, our score is better than [13] (0.159),since the mesh estimated by the latter has noise that ap-pears around the nose (visible in Figure 7 and Figure 8).We also evaluate our method on the NoW benchmark [45]that only evaluates neutral mesh (no expression, albedo andpose evaluation). Nevertheless, we obtain very competitiveresults: 1.26/1.57/1.31mm (median/mean/std).

Digital Emily As shown in Fig.9, we compare ourmethod with Digital Emily [51] ground truth (GT). We notethat our method bakes some albedo in the estimated light(as shown in the recovered environment map).

2More details in supp. material section 3.

12825

Figure 8. Vertex position and normal error for each method (from left [35], [36], [13], [9], and Ours) compared to GT mesh.

We also compare quantitatively our image reconstruc-tion quality against state-of-the-art (see Table 2). For eachmethod, we compute SSIM [52] and PSNR scores ver-sus GT, for final render, diffuse, and specular. Since eachmethod uses a different UV mapping, we compare the pro-jection of the albedo on the input image (using the GT cam-era pose) and not on the unwrapped texture. For the ‘Final’rendered image, our method is on par with the method of[9] and achieves better performance than [19]. Since [35]and [36] do not estimate scene light, they do not have a fi-nal render image, so no comparison is available for thesemethods. For diffuse and specular albedos, [35] is globallybetter than the other methods except for ‘Diffuse SSIM’ and‘Specular SSIM’, where [9] and our method measure highersimilarity, respectively.

Figure 9. Our final, diffuse, and specular render images (Ours box)compared to GT (GT box). All GTs are rendered by Maya.

Table 2. Final render, diffuse and specular albedos in comparisonwith GT Maya renders. SSIM and PSNR (dB): higher is better.

vs GT Final Final Diffuse Diffuse Spec. Spec.Render (SSIM) (PSNR) (SSIM) (PSNR) (SSIM) (PSNR)Ours 0.933 35.932 0.653 29.548 0.642 29.274[9] 0.965 36.390 0.722 29.812 0.547 29.670[19] 0.906 35.389 0.639 29.006 0.452 28.833[36] - - 0.540 28.633 0.516 28.926[35] - - 0.679 30.061 0.604 30.923

7. Limitations, Future works and ConclusionLimitations Our data-driven method inherits the bias of

the training data, which does not provide high diversity inlighting and facial expression (most subjects in the datasetare with smiling faces, eyes opened and well lit). Thisleads to sub-optimal reconstruction for less frequent expres-sion/lighting (incorrect light estimation for left subject inFigure 10). This limitation can be mitigated by using a morebalanced dataset while the method remains the same. Also,with images with extreme shadows some artifacts may ap-pear in the estimated albedos (right subject in Figure 10).

Disentangling light color from skin color from a singleimage is an ill-posed problem and is not solved in this work.

The limitations of 3DMM statistical albedo prior (unable tomodel all skin types e.g. non-Caucasian) make this sepa-ration even more difficult. Adding light color regulariza-tion, together with a more complete albedo prior can miti-gate this.

Future works Our method could be extended by using anon-linear morphable model such as [53] to improve the ge-ometry at finer level. Using a more complex skin reflectancemodel such as BSSRDF/dielectric materials [54] is also in-teresting. Our method could also benefit from a work suchas [55], which tackles the foreign (external) shadows chal-lenge. Finally, our method can naturally extend to video-based reconstruction [14], which should improve the accu-racy of the estimated facial attributes.

Figure 10. Limitations of our method.

Conclusion In this work, we address the problem of face re-construction under general illumination conditions, an im-portant challenge to tackle for in-the-wild face reconstruc-tion. For this, we introduced the first deep-based and self-supervised method that achieves state-of-the art monocularface reconstruction in challenging lighting conditions. Webuild our work on recent methods, namely [10] which com-bines deep neural networks with differentiable renderingand [9] which uses ray tracing for face reconstruction withinan optimization-based framework. Our method solves thelimitations of [10] by using a better light and BRDF mod-els and captures personalized diffuse and specular albedosoutside of 3DMM space, while being robust against harshshadows. Our approach also solves the limitations of [9]and achieves near-real time performance while at the sametime being completely independent of landmarks at testtime. Our method naturally benefits from large-scale un-labeled data-sets. By comparing to recent approaches, weachieve better results in terms of robustness in scenes withchallenging lighting conditions, while producing plausiblereconstruction of subject-specific albedos. Beyond its ro-bustness to lighting conditions, the rich reflectance decom-position produced by our method is compatible with exist-ing rendering engines and allows for several style – illumi-nation and albedo – transfer, edit applications, avatar cre-ation and relighting.

12826

References

[1] Thabo Beeler, Fabian Hahn, Derek Bradley, Bernd Bickel,Paul Beardsley, Craig Gotsman, Robert W Sumner, andMarkus Gross. High-quality passive facial performance cap-ture using anchor frames. In ACM Transactions on Graphics(TOG), volume 30, page 75. ACM, 2011. 2

[2] Chenglei Wu, Kiran Varanasi, Yebin Liu, Hans-Peter Seidel,and Christian Theobalt. Shading-based dynamic shape re-finement from multi-view video under general illumination.In 2011 International Conference on Computer Vision, pages1108–1115. IEEE, 2011. 2

[3] Levi Valgaerts, Chenglei Wu, Andres Bruhn, Hans-PeterSeidel, and Christian Theobalt. Lightweight binocular fa-cial performance capture under uncontrolled lighting. ACMTrans. Graph., 31(6):187–1, 2012. 2

[4] Abhijeet Ghosh, Graham Fyffe, Borom Tunwattanapong, JayBusch, Xueming Yu, and Paul Debevec. Multiview facecapture using polarized spherical gradient illumination. InACM Transactions on Graphics (TOG), volume 30, page129. ACM, 2011. 2

[5] P Gotardo, J Riviere, D Bradley, et al. Practical DynamicFacial Appearance Modeling and Acquisition. ACM SIG-GRAPH Asia, 2018. 2

[6] Pablo Garrido, Levi Valgaert, Chenglei Wu, and ChristianTheobalt. Reconstructing detailed dynamic face geometryfrom monocular video. ACM Transactions on Graphics,2013. 2

[7] Supasorn Suwajanakorn, Ira Kemelmacher-Shlizerman, andSteven M Seitz. Total moving face reconstruction. In Eu-ropean Conference on Computer Vision, pages 796–812.Springer, 2014. 2

[8] Pablo Garrido, Michael Zollhofer, Dan Casas, Levi Val-gaerts, Kiran Varanasi, Patrick Perez, and ChristianTheobalt. Reconstruction of Personalized 3D Face Rigs fromMonocular Video. {ACM} Trans. Graph. (Presented at SIG-GRAPH 2016), 35(3):28:1–28:15, 2016. 2

[9] Abdallah Dib, Gaurav Bharaj, Junghyun Ahn, CedricThebault, Philippe-Henri Gosselin, Marco Romeo, andLouis Chevallier. Practical face reconstruction via differen-tiable ray tracing. Computer Graphics Forum, 2021. 2, 3, 4,5, 6, 7, 8

[10] Ayush Tewari, Michael Zollofer, Hyeongwoo Kim, PabloGarrido, Florian Bernard, Patrick Perez, and Theobalt Chris-tian. MoFA: Model-based Deep Convolutional Face Autoen-coder for Unsupervised Monocular Reconstruction. In TheIEEE International Conference on Computer Vision (ICCV),2017. 2, 3, 8

[11] Ayush Tewari, Michael Zollhofer, Pablo Garrido, FlorianBernard, Hyeongwoo Kim, Patrick Perez, and ChristianTheobalt. Self-supervised multi-level face model learningfor monocular reconstruction at over 250 hz. In 2018 IEEEConference on Computer Vision and Pattern Recognition,CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018,pages 2549–2559. IEEE Computer Society, 2018. 2, 3, 6,7

[12] Luan Tran and Xiaoming Liu. Nonlinear 3d face morphablemodel. In Proceedings of the IEEE conference on computervision and pattern recognition, pages 7346–7355, 2018. 2, 3

[13] Luan Tran, Feng Liu, and Xiaoming Liu. Towards high-fidelity nonlinear 3d face morphable model. In In Proceed-ing of IEEE Computer Vision and Pattern Recognition, LongBeach, CA, June 2019. 2, 3, 5, 7, 8

[14] A Tewari, F Bernard, P Garrido, G Bharaj, et al. FML: FaceModel Learning from Videos. CVPR, 2019. 2, 3, 8

[15] Michael Zollhofer, Justus Thies, Darek Bradley, PabloGarrido, Thabo Beeler, Patrick Perez, Marc Stamminger,Matthias Nießner, and Christian Theobalt. State of the Arton Monocular 3D Face Reconstruction, Tracking, and Ap-plications. 2018. 2

[16] Volker Blanz and Thomas Vetter. A morphable model forthe synthesis of 3D faces. In Proceedings of the 26th an-nual conference on Computer graphics and interactive tech-niques, pages 187–194. ACM Press/Addison-Wesley Pub-lishing Co., 1999. 2, 4

[17] Tianye Li, Timo Bolkart, Michael J Black, Hao Li, andJavier Romero. Learning a model of facial shape and expres-sion from 4d scans. ACM Transactions on Graphics (ToG),36(6):194, 2017. 2

[18] Bernhard Egger, William AP Smith, Ayush Tewari, StefanieWuhrer, Michael Zollhoefer, Thabo Beeler, Florian Bernard,Timo Bolkart, Adam Kortylewski, Sami Romdhani, et al.3d morphable face models—past, present, and future. ACMTransactions on Graphics (TOG), 39(5):1–38, 2020. 2

[19] William AP Smith, Alassane Seck, Hannah Dee, BernardTiddeman, Joshua B Tenenbaum, and Bernhard Egger. Amorphable face albedo model. In Proceedings of theIEEE/CVF Conference on Computer Vision and PatternRecognition, pages 5011–5020, 2020. 2, 3, 4, 7, 8

[20] Chenglei Wu, Derek Bradley, Markus Gross, and ThaboBeeler. An anatomically-constrained local deformationmodel for monocular face capture. ACM transactions ongraphics (TOG), 35(4):1–12, 2016. 2

[21] Curtis Andrus, Junghyun Ahn, Michele Alessi, AbdallahDib, Philippe Gosselin, Cedric Thebault, Louis Chevallier,and Marco Romeo. Facelab: Scalable facial performancecapture for visual effects. In The Digital Production Sympo-sium, DigiPro ’20, New York, NY, USA, 2020. Associationfor Computing Machinery. 2

[22] Chen Li, Kun Zhou, and Stephen Lin. Intrinsic face im-age decomposition with human face priors. In David Fleet,Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars, edi-tors, Computer Vision – ECCV 2014, pages 218–233, Cham,2014. Springer International Publishing. 3

[23] Abdallah Dib, Gaurav Bharaj, Junghyun Ahn, Cedric The-bault, Philippe-Henri Gosselin, and Louis Chevallier. Facereflectance and geometry modeling via differentiable raytracing. ACM SIGGRAPH European Conference on VisualMedia Production (CVMP), 2019. 3

[24] Guillaume Loubet, Nicolas Holzschuch, and Wenzel Jakob.Reparameterizing discontinuous integrands for differentiablerendering. Transactions on Graphics (Proceedings of SIG-GRAPH Asia), 38(6), Dec. 2019. 3

12827

[25] Tzu-Mao Li, Miika Aittala, Fredo Durand, and Jaakko Lehti-nen. Differentiable monte carlo ray tracing through edgesampling. ACM Trans. Graph., 37(6):222:1–222:11, Dec.2018. 3, 6

[26] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang.Large-scale celebfaces attributes (celeba) dataset. RetrievedAugust, 15:2018, 2018. 3

[27] Tero Karras, Samuli Laine, and Timo Aila. A style-basedgenerator architecture for generative adversarial networks.In Proceedings of the IEEE Conference on Computer Visionand Pattern Recognition, pages 4401–4410, 2019. 3

[28] Tatsuro Koizumi and William A. P. Smith. Look ma, no land-marks! – unsupervised, model-based dense face alignment.In IEEE European Conference on Computer Vision (ECCV),2020. 3

[29] Soumyadip Sengupta, Angjoo Kanazawa, Carlos D Castillo,and David W Jacobs. Sfsnet: Learning shape, reflectanceand illuminance of facesin the wild’. In Proceedings of theIEEE Conference on Computer Vision and Pattern Recogni-tion, pages 6296–6305, 2018. 3

[30] Bindita Chaudhuri, Noranart Vesdapunt, Linda Shapiro, andBaoyuan Wang. Personalized face modeling for improvedface reconstruction and motion retargeting. In IEEE Euro-pean Conference on Computer Vision (ECCV), 2020. 3

[31] Shunsuke Saito, Lingyu Wei, Liwen Hu, Koki Nagano, andHao Li. Photorealistic facial texture inference using deepneural networks. In Proceedings of the IEEE Conferenceon Computer Vision and Pattern Recognition, pages 5144–5153, 2017. 3

[32] Samuli Laine, Tero Karras, Timo Aila, Antti Herva, Shun-suke Saito, Ronald Yu, Hao Li, and Jaakko Lehtinen.Production-level facial performance capture using deep con-volutional neural networks. In Proceedings of the ACM SIG-GRAPH/Eurographics Symposium on Computer Animation,pages 1–10, 2017. 3

[33] Timur Bagautdinov, Chenglei Wu, Jason Saragih, PascalFua, and Yaser Sheikh. Modeling facial geometry usingcompositional vaes. In Proceedings of the IEEE Conferenceon Computer Vision and Pattern Recognition, pages 3877–3886, 2018. 3

[34] Loc Huynh, Weikai Chen, Shunsuke Saito, Jun Xing, KokiNagano, Andrew Jones, Paul Debevec, and Hao Li. Meso-scopic Facial Geometry Inference Using Deep Neural Net-works. In Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition, pages 8407–8416, 2018. 3

[35] S Yamaguchi, S Saito, et al. High-fidelity Facial Re-flectance and Geometry Inference from an UnconstrainedImage. ACM TOG, 2018. 3, 7, 8

[36] Alexandros Lattas, Stylianos Moschoglou, Baris Gecer,Stylianos Ploumpis, Vasileios Triantafyllou, AbhijeetGhosh, and Stefanos Zafeiriou. Avatarme: Realistically ren-derable 3d facial reconstruction” in-the-wild”. In Proceed-ings of the IEEE/CVF Conference on Computer Vision andPattern Recognition, pages 760–769, 2020. 3, 7, 8

[37] Thomas Gerig, Andreas Morel-Forster, Clemens Blumer,Bernhard Egger, Marcel Luthi, Sandro Schonborn, andThomas Vetter. Morphable face models-an open framework.

In 2018 13th IEEE International Conference on AutomaticFace & Gesture Recognition (FG 2018), pages 75–82. IEEE,2018. 4

[38] Robert L Cook and Kenneth E. Torrance. A reflectancemodel for computer graphics. ACM Transactions on Graph-ics (TOG), 1(1):7–24, 1982. 4

[39] Bruce Walter, Stephen Marschner, Hongsong Li, and Ken-neth Torrance. Microfacet models for refraction throughrough surfaces. pages 195–206, 01 2007. 4

[40] James T. Kajiya. The rendering equation. In Proceedings ofthe 13th Annual Conference on Computer Graphics and In-teractive Techniques, SIGGRAPH ’86, pages 143–150, NewYork, NY, USA, 1986. ACM. 4

[41] Adrian Bulat and Georgios Tzimiropoulos. How far are wefrom solving the 2d & 3d face alignment problem? (and adataset of 230,000 3d facial landmarks). In InternationalConference on Computer Vision, 2017. 5

[42] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang.Deep learning face attributes in the wild. In Proceedings ofInternational Conference on Computer Vision (ICCV), De-cember 2015. 6

[43] Dhruv Mahajan, Ravi Ramamoorthi, and Brian Curless. Atheory of frequency domain invariants: Spherical harmonicidentities for brdf/lighting transfer and image consistency.IEEE transactions on pattern analysis and machine intelli-gence, 30(2):197–213, 2007. 6

[44] Linjie Lyu, Marc Habermann, Lingjie Liu, Ayush Tewari,Christian Theobalt, et al. Efficient and differentiableshadow computation for inverse problems. arXiv preprintarXiv:2104.00359, 2021. 6

[45] Soubhik Sanyal, Timo Bolkart, Haiwen Feng, and MichaelBlack. Learning to regress 3D face shape and expressionfrom an image without 3D supervision. In Proceedings IEEEConf. on Computer Vision and Pattern Recognition (CVPR),pages 7763–7772, June 2019. 6, 7

[46] Ayush Tewari, Michael Zollofer, Florian Bernard, PabloGarrido, Hyeongwoo Kim, Patrick Perez, and ChristianTheobalt. High-fidelity monocular face reconstruction basedon an unsupervised model-based face autoencoder. IEEETransactions on Pattern Analysis and Machine Intelligence,pages 1–1, 2018. 7

[47] Rohith Krishnan Pillai, Laszlo A. Jeni, Huiyuan Yang, ZhengZhang, Lijun Yin, and Jeffrey F. Cohn. The 2nd 3d facealignment in the wild challenge (3dfaw-video): Dense recon-struction from video. In In Proceedings of the 2019 IEEE In-ternational Conference on Computer Vision Workshops, Oc-tober 2019. 7

[48] Xiangyu Zhu, Xiaoming Liu, Zhen Lei, and Stan Z Li. Facealignment in full pose range: A 3d total solution. IEEE trans-actions on pattern analysis and machine intelligence, 2017.7

[49] Jianzhu Guo, Xiangyu Zhu, and Zhen Lei. 3ddfa. https://github.com/cleardusk/3DDFA, 2018. 7

[50] Jianzhu Guo, Xiangyu Zhu, Yang Yang, Fan Yang, Zhen Lei,and Stan Z Li. Towards fast, accurate and stable 3d denseface alignment. In Proceedings of the European Conferenceon Computer Vision (ECCV), 2020. 7

12828

[51] Emily. The wikihuman project. https://vgl.ict.usc.edu/Data/DigitalEmily2/, 2017. Accessed:2020-05-21. 7

[52] Zhou Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli.Image quality assessment: From error visibility to structuralsimilarity. Trans. Img. Proc., 13(4):600–612, Apr. 2004. 8

[53] Ruilong Li, Karl Bladin, Yajie Zhao, Chinmay Chinara,Owen Ingraham, Pengda Xiang, Xinglei Ren, PratushaPrasad, Bipin Kishore, Jun Xing, et al. Learning forma-tion of physically-based face attributes. In Proceedings ofthe IEEE/CVF Conference on Computer Vision and PatternRecognition, pages 3410–3419, 2020. 8

[54] Tim Weyrich, Wojciech Matusik, Hanspeter Pfister, BerndBickel, Craig Donner, Chien Tu, Janet McAndless, JinhoLee, Addy Ngan, Henrik Wann Jensen, and Others. Anal-ysis of human faces using a measurement-based skin re-flectance model. In ACM Transactions on Graphics (TOG),volume 25, pages 1013–1024. ACM, 2006. 8

[55] Xuaner Zhang, Jonathan T. Barron, Yun-Ta Tsai, XiumingZhang, Ren Ng, and David E. Jacobs. Portrait shadow ma-nipulation. volume 39, 2020. 8

12829