Reference-guided Face Component EditingQiyao Deng1,4 , Jie Cao1,4 , Yunfan Liu1,4 , Zhenhua Chai5 , Qi Li1,2,4∗ , Zhenan Sun1,3,4

1Center for Research on Intelligent Perception and Computing, NLPR, CASIA, Beijing, China2Artificial Intelligence Research, CAS, Qingdao, China

3Center for Excellence in Brain Science and Intelligence Technology, CAS, Beijing, China4School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China

5Vision Intelligence Center, AI Platform, Meituandianping Group{qiyao.deng, jie.cao, yunfan.liu}@cripac.ia.ac.cn, {qli, znsun}@nlpr.ia.ac.cn,

chaizhenhua@meituan.com

AbstractFace portrait editing has achieved great progressin recent years. However, previous methods ei-ther 1) operate on pre-defined face attributes, lack-ing the flexibility of controlling shapes of high-level semantic facial components (e.g., eyes, nose,mouth), or 2) take manually edited mask or sketchas an intermediate representation for observablechanges, but such additional input usually requiresextra efforts to obtain. To break the limitations(e.g. shape, mask or sketch) of the existing meth-ods, we propose a novel framework termed r-FACE(Reference-guided FAce Component Editing) fordiverse and controllable face component editingwith geometric changes. Specifically, r-FACE takesan image inpainting model as the backbone, utiliz-ing reference images as conditions for controllingthe shape of face components. In order to encour-age the framework to concentrate on the target facecomponents, an example-guided attention moduleis designed to fuse attention features and the targetface component features extracted from the refer-ence image. Through extensive experimental vali-dation and comparisons, we verify the effectivenessof the proposed framework.

1 IntroductionFace portrait editing is of great interest in the computer visioncommunity due to its potential applications in movie indus-try, photo manipulation, and interactive entertainment, etc.With advances in Generative Adversarial Networks [Good-fellow et al., 2014] in recent years, tremendous progresshas been made in face portrait editing [Yang et al., 2018;Choi et al., 2018; Liu et al., 2019]. These approaches gener-ally fall into three main categories: label-conditioned meth-ods, geometry-guided methods and reference-guided meth-ods. Specifically, label-conditioned methods [He et al., 2019;Choi et al., 2018] only focus on several pre-defined conspicu-ous attributes thus lacking the flexibility of controlling shapes

∗Contact Author

Figure 1: The illustration of reference guided face component edit-ing. The first row is the definition diagram of the task, and the sec-ond row is the synthesized result based on the given different refer-ence images.

of high-level semantic facial components (e.g., eyes, nose,mouth). This is because it is hard to produce results withobservable geometric changes merely based on attribute la-bels. In order to tackle this, geometry-guided methods [Jo andPark, 2019; Gu et al., 2019] propose to take manually editedmask or sketch as an intermediate representation for obvi-ous face component editing with large geometric changes.However, directly taking such precise representations as ashape guide is inconvenient for users, which is laborious andrequires painting skills. To solve this problem, reference-guided methods directly learns shape information from ref-erence images without requiring precise auxiliary representa-tion, relieving the dependence on face attribute annotation orprecise sketch/color/mask information. As far as we know,reference-guided methods are less studied than the first twomethods. ExGANs [Dolhansky and Canton Ferrer, 2018] uti-lizes exemplar information in the form of a reference imageof the region for eye editing (in-painting). However, Ex-GANs can only edit eyes and requires reference images withthe same identity, which is inconvenient to collect in practice.ELEGANT [Xiao et al., 2018] transfers exactly the same typeof attributes from a reference image to the source image byexchanging certain part of their encodings. However, ELE-GANT is only used for editing obvious semantic attributes,and could not change abstract shapes.

To overcome the aforementioned problems, we propose a

arX

iv:2

006.

0205

1v2

[cs

.CV

] 1

4 Ju

l 202

0

Figure 2: Different methods for face portrait editing. (a) AttGAN, (b) SC-FEGAN and (c) Our network.

new framework: Reference guided FAce Component Edit-ing (r-FACE for short), which can achieve diverse and con-trollable face semantic components editing (e.g., eyes, nose,mouth) without requiring paired images. The ideal editingis to transfer single or multiple face components from a ref-erence image to the source image, while still preserving theconsistency of pose, skin color and topological structure(seeFigure 1). Our framework breaks the limitations of existingmethods: 1) shape limitation. r-FACE can flexibly controldiverse shapes of high-level semantic facial components bydifferent reference images; 2) intermediate presentation lim-itation. There is no need to manually edit precise masks orsketches for observable geometric changes.

Our framework is based on an image inpainting model forediting face components by reference images even withoutpaired images. r-FACE has two main streams including 1)an inpainting network Gi and 2) an embedding network Er.As shown in Figure 3, Gi takes the source image with targetface components corrupted and the corresponding mask im-age as input, and outputs the generated image with semanticfeatures extracted by Er. To encourage the framework to con-centrate on the target face components, an example-guidedattention module is introduced to combine features extractedby Gi and Er. To supervise the proposed model, a contex-tual loss is adopted to constrain the similarity of shape be-tween generated images and reference images, while a styleloss and a perceptual loss are adopted to preserve the consis-tency of skin color and topological structure between gener-ated images and source images. Both qualitative and quantita-tive results demonstrate that our model is superior to existingliterature by generating high-quality and diverse faces withobservable changes for face component editing.

In summary, the contributions of this paper are as follows:

• We propose a novel framework named reference guidedface component editing for diverse and controllableface component editing with geometric changes, whichbreaks the shape and intermediate presentation (e.g.,precise masks or sketches) limitation of existing meth-ods.

• An example-guided attention module is designed to en-courage the framework to concentrate on the target facecomponents by combining attention features and the tar-get face component features of the reference image, fur-ther boosting the performance of face portrait editing.

• Both qualitative and quantitative results demonstrate thesuperiority of our method compared with other bench-mark methods.

2 Related WorkFace Portrait Editing. Face portrait editing aims at ma-nipulating single or multiple attributes or components of aface image towards given conditions. Depending on differ-ent conditions, face portrait editing methods can be classifiedinto three categories: label-conditioned methods, geometry-guided methods and reference-guided methods. Label-conditioned methods change predefined attributes, such ashair color [Choi et al., 2018], age [Liu et al., 2019] andpose [Cao et al., 2019]. However, these methods focus onseveral conspicuous attributes [Liu et al., 2015; Langner etal., 2010], lacking the flexibility of controlling the shapes ofdifferent semantic facial parts. As shown in Figure 2(a),AttGAN [He et al., 2019] attempts to edit attributes withshape changes, such as ′Narrow Eyes′, ′Pointy Nose′and ′Mouth Slightly Open′, but it can only achieve subtlechanges hard to be observed. Moreover, lacking of labeleddata will extremely limit the performance of these meth-ods. To tackle above problems, geometry-guided methodsuse an intermediate representation to guide observable shapechanges. [Gu et al., 2019] proposes a framework based onmask-guided conditional GANs which can change the shapeof face components by manual editing precise masks. Asshown in Figure 2(b), SC-FEGAN [Jo and Park, 2019] re-quires directly taking mask, precise sketch and color as inputfor editing the shape of face components. However, such pre-cise input is difficult and inconvenient to obtain. Reference-guided methods can directly learn shape information fromreference images without precise auxiliary representation, re-lieving the dependence on face attribute annotation or pre-cise sketch/color/mask information for face portrait editing.Inspired by this, we propose a new framework (see Figure2(c)), which can achieve diverse and controllable face se-mantic components editing (e.g., eyes, nose, mouth), whichis shape free and precise landmark or sketch free.

Face Completion/Inpainting. Face completion, alsoknown as face inpainting, aims to complete a face imagewith a masked region or missing content. Early face com-pletion works [Bertalmio et al., 2000; Criminisi et al., 2003;Bertalmio et al., 2003] fill semantic contents based on theoverall image and structural continuity between the maskedand unmasked regions, which aims to reconstruct missing re-gions according to the ground-truth image. Recently, somelearning-based method [Zheng et al., 2019; Song et al., 2019]are proposed for generating multiple and diverse plausible re-sults. [Zheng et al., 2019] proposes a probabilistically prin-cipled framework with two parallel paths, the VAE-based re-constructive path is used to impose smooth priors for the la-

Figure 3: The overall structure of proposed framework. On the top left corner is the generator. The top-right figure shows detailed attentionmodule. The constrains among the source image, the reference image and the generated image is shown on the bottom.

tent space of complement regions, the generative path is usedto predict the latent prior distribution for missing regions,which can be sampled to generate diverse results. However,this method lacks controllability for diverse results. In lightof this, [Song et al., 2019] generate controllable diverse re-sults from the same masked input by manual modifying faciallandmark heatmaps and parsing maps.

3 MethodIn this section, we first introduce the framework of reference-based face component editing. Then, the example-guided at-tention module are presented. Finally, objective functions ofthe proposed model are provided.

3.1 Reference Guided Face Component EditingWe propose a framework (See Figure 3), named referenceguided face component editing, that transfers one or multi-ple face components of a reference image to correspondingcomponents of the source image. The framework requiresthree inputs, a source image Is, a reference image Ir and thetarget face component mask of the reference image Ism. Thesource mask Ism merely needs to roughly represent the targetface components, which can be obtained by a face parsingor landmark detection network. The corrupted image can beobtained by Equation 1:

Ic = Is ∗ Ism, (1)

where ∗ is an element-wise multiplication operator. The goalof this framework is to generate a photo-realistic image Ig ,in which shape is semantically similar to corresponding facecomponents of the reference image while the face color andtopological structure are consistent with the source image.

In this work, we utilize an image inpainting generator Gi asbackbone that can generate the completed image without con-straints on shape, while a discriminator D is used for distin-guishing face images from the real and synthesized domains.Gi is consist of an encoder, seven dilated residual blocks, anattention module and a decoder. To fill missing parts with se-mantically meaningful face components of a reference image,

an reference-guided encoder Er is introduced to extract fea-tures of the reference image. The encoder of Gi and Er havesame structure but parameters are not shared. Attention mod-ule, to be described next, is effectively transferring semanticcomponents from high-level features of the reference imageto Gi, further improving the performance of our framework.The generated image Ig can be expressed as:

Ig = Gi(Ic, Ism, Er(Ir)), (2)

3.2 Example-guided Attention ModuleInspired by the short+long term attention of PICNet [Zhenget al., 2019], we propose an example-guided attention. Theshort+long term attention uses the self-attention map to har-ness distant spatial context and the contextual flow to cap-ture feature-feature context for finer-grained image inpaint-ing. The example-guided attention replaces the contextualflow by the example-guided flow which combines the atten-tion features and the reference features for clearly transform-ing the corresponding face component features of referenceimages to source images.

The proposed structure is shown in the upper right cornerof Figure 3. Following [Zheng et al., 2019], the self-attentionmap is calculated from the source feature, which can be ex-pressed as Fa = Fs⊗Fm. The attention map Fm is obtainedby Fm = Softmax(FT

q Fq), in which Fq = Conv(Fs) andConv is a 1× 1 convolution filter. To transfer the target facecomponent features of reference images, the reference featureis embedded by multiplying the attention map with the sourcemask Ism. The example-guided flow is expressed as follows:

Fe = Ism ∗ F′e + (1− Ism) ∗ Fr, (3)

where F′e = Fr ⊗ Fm. Finally, the fused feature Ff =Fa ⊕ Fe is sent to the decoder for generating results withthe target components of the reference image.

3.3 ObjectiveWe use the combination of a per-pixel loss, a style loss, aperceptual loss, a contextual loss, a total variation loss and anadversarial loss for training the framework.

To capture fine facial details we adopt the perceptualloss [Johnson et al., 2016] and the style loss [Johnson etal., 2016], which are widely adopted in style transfer, superresolution, and face synthesis. The perceptual loss aims tomeasure the similarity of the high dimensional features (e.g.,overall spatial structure) between two images, while the styleloss measure the similarity of styles (e.g., colors). The per-ceptual loss can be expressed as follows:

Lperc =∑l

1

ClHlWl‖φl(Ig)− φl(Is)‖1, (4)

The style loss compare the content of two images by usingGram matrix, which can be expressed as follows:

Lstyle =∑l

1

ClCl‖Gl(Ig ∗ Ism)−Gl(Ic)

ClHlWl‖1, (5)

where ‖ · ‖1 is the `1 norm. φl(·) ∈ RCl×Hl×Wl representsthe feature map of the l-th layer of the VGG-19 network [Si-monyan and Zisserman, 2014] pretrained on the ImageNet.Gl(·) = φl(·)Tφl(·) represents the Gram matrix correspond-ing to φl(·).

The contextual loss [Mechrez et al., 2018] measures thesimilarity between non-aligned images, which in our modeleffectively guarantees the consistent shape of the targetface components in generated images and reference images.Given an image x and its target image y, each of which isrepresented as a collection of points(e.g. VGG-19 [Simonyanand Zisserman, 2014] features): X = {xi} and Y = {yj},|X| = |Y | = N . The similarity between the images can becalculated that for each feature yj , finding the feature xi thatis most similar to it, and then sum the corresponding featuresimilarity values over all yj . Formally, it is defined as:

CX(x, y) = CX(X,Y ) =1

N

∑j

maxiCXij , (6)

where CXij is the similarity between features xi and yj . Attraining stage, we need the target components mask of ref-erence images Irm for calculating the contextual loss. Thecontextual loss can be expressed as follows:

Lcx = − log(CX(φl(Ig ∗ Ism), φl(Ir ∗ Irm))), (7)

The per-pixel loss can be expressed as follows:

Lpixel = ‖φl(Ig ∗ Ism)− φl(Is ∗ Ism)‖1, (8)

Lastly, we use an adversarial loss for minimizing the dis-tance between the distribution of the real image and the gen-erated image. Here, LSGAN [Mao et al., 2017] is adoptedfor stable training. The adversarial loss can be expressed asfollows:

LadG= E[(D(Ig)− 1)2], (9)

LadD= E[D(Ig)2] + E[(D(Is)− 1)2], (10)

With total variational regularization loss Ltv [Johnson etal., 2016] added to encourage the spatial smoothness in thegenerated images, we obtain our full objective as:

L = λ1Lperc + λ2Lstyle + λ3Lcx

+λ4Lpixel + λ5Ltv + λ6LadG

(11)

Figure 4: Comparisons among (c) copy-and-paste, (d) AttGAN, (e)ELEGANT, (f) the commercial state of the art algorithm in AdobePhotoshop and (g) the proposed r-FACE technique. The source andreference images are shown in (a) and (b), respectively. AttGANand ELEGANT edit attributes: ′Narrow Eyes′, ′Pointy Nose′

,′Mouth Slightly Open′ and all of them respectively.

4 Experiments4.1 Dataset and PreprocessingThe face attribute dataset CelebAMask-HQ [Lee et al., 2019]contains 30000 aligned facial images with the size of 1024×1024 and corresponding 30000 semantic segmentation labelswith the size of 512 × 512. Each label in the dataset has19 classes (e.g., ”left eye”, ”nose”, ”mouth”). In our experi-ments, three face components are considered, i.e., eyes (”lefteye & right eye”), nose (”nose”), and mouth (”mouth & u lip& l lip”). We obtain rough version of face components fromsemantic segmentation labels by an image dilation operation,which are defined as mask images. We take 2,000 images asthe test set for performance evaluation, using rest images totrain our model. All images are resized to 256× 256.

4.2 Implementation DetailsOur end-to-end network is trained on four GeForce GTX TI-TAN X GPUs of 12GB memory. Adam optimizer is usedin experiments with β1 = 0.0 and β2 = 0.9. The hyperpa-rameters from λ1 to λ6 are assigned as 0.1, 250, 1, 0.5, 0.1,0.01 respectively. For each source image, we remove two orthree target face components for training. During testing, ourmodel can change one or more face components.

4.3 Qualitative EvaluationComparison with Other Methods. In order to demon-strate the superiority of the proposed method, we comparethe quality of sample synthesis results to several benchmarkmethods. In addition to face editing model AttGAN [He etal., 2019] and ELEGANT [Xiao et al., 2018], we also con-sider copy-and-paste as a naive baseline and Adobe Photo-shop image editing as an interactive way to produce synthe-sized face images. According to the results presented in Fig-ure 4, although margins of edited facial components in Adobe

Figure 5: Illustrations of multi-modal face components editing, in-cluding ’eyes’ and ’mouth’. The first column represents source im-ages, the first and third rows are reference images, and the secondand fourth rows are synthesized images according to reference im-ages above.

Photoshop are much smoother than those in results of copy-and-paste, obvious ghosting artifacts and color distortionsstill exist and more fine-grained manual labour is required toimprove the quality. In contrast, AttGAN could generate real-istic synthetic images which are indistinguishable to genericones in an end-to-end manner. However, pre-defined facialattribute labels are used to guide AttGAN to transform facialcomponents thus the diversity of generated images is limited.Moreover, as can be seen in Figure 4, AttGAN only producessubtle changes that could hardly reflect the desired transla-tion. ELEGANT, as a reference-guided face attribute edit-ing method, can learn obvious semantic attributes (e.g., openeyes or close mouth), but could not learn abstract shape infor-mation (e.g., nose editing). Moreover, ELEGANT produceslarge deformation and unexpected artifacts at other attribute-unrelated areas, especially the editing of multiple compo-nents. On the contrary, our method takes arbitrary face im-ages as reference, which significantly increases the diversityof generated images.

Multi-Modal Face Components Editing. Reference-guided face components editing improves the diversity andcontrollability of generated faces, as the stylistic informationis designated by arbitrary reference images. As shown inFigure 5, target face components of interest (e.g. eyes andmouths) are transformed to be of the same style as the corre-sponding reference image. For example, synthesized mouthsin faces of the last row accurately simulate the counterpart inreference images, in terms of both overall shape (e.g. raisedcorners of pursed mouth) and local details (e.g. partiallycovered teeth). Meanwhile, they are naturally blended in

Method FID ↓ MS-SSIM ↑GLCIC[Iizuka et al., 2017] 8.09 0.95AttGAN[He et al., 2019] 6.28 0.96

ELEGANT [Xiao et al., 2018] 15.97 0.82

r-FACE (Ours) 5.81 0.92w / o attention 6.25 0.90

w / o contextual loss 5.27 0.90w / o style loss 8.28 0.90

w / o perceptual loss 8.49 0.89

Table 1: Comparisons of FID and MS-SSIM on the CelebA-HQdataset.

to the source face without observable color distortions andghosting artifacts, demonstrating the effectiveness of pro-posed method.

4.4 Quantitative EvaluationFollowing most of face portrait methods [He et al., 2019; Wuet al., 2019], we leverage Frechet Inception Distance (FID,lower value indicates better quality) [Heusel et al., 2017] andMulti-Scale Structural SIMilarity (MS-SSIM, higher valueindicates better quality) [Wang et al., 2003] to evaluate theperformance of our model. FID is used to measure the qual-ity and diversity of generated images. MS-SSIM is used toevaluate the similarity of two images from luminance, con-trast, and structure.

We compare our method with AttGAN[He et al., 2019],one of label-conditioned methods, and ELEGANT [Xiaoet al., 2018], one of reference-guided methods. ForAttGAN and ELEGANT, the value of FID or MS-SSIMare the result of averaging three pre-defined attributes forshape changing, including ’Narrow Eyes’, ’Pointy Nose’ and’Mouth Slightly Open’. The backbone of r-FACE is similarto image inpainting task, so we also compare our method withGLCIC[Iizuka et al., 2017], one of popular face inpaintingmethods. As is shown in TABLE 1, comparing with thesemethods, the FID of our method is much lower. With theobservation that the MS-SSIM of our method is lower thanAttGAN and GLCIC, we analyze the reasons: MS-SSIMis sensitive to luminance, contrast, and structure, however,1) GLCIC does not have any constraints on the structure orshape of components, just requiring the missing regions to becompleted; 2) AttGAN edits shape-changing attributes withsubtle changes that are hard to be observed as shown in Fig-ure 4, so the change of luminance, contrast, and structure islimited. In contrast, r-FACE imposes a geometric similarityconstraint on the components of source images and referenceimages, which changes the shape or structure drastically andeven affects the identity of the face.

4.5 Ablation StudyExample-guided Attention Module. To investigate the ef-fectiveness of the example-guided attention module, we con-duct a variant of our model, denoted as ”r-FACE w / o atten-tion”. In ”r-FACE w / o attention”, we train r-FACE with-out example-guided attention module. To learn face compo-nents from a reference image, we directly concatenate the fea-

tures of reference images with that of source images, whichintroduced the features of the whole reference image. TA-BLE 1 shows the comparison on FID and MS-SSIM between”r-FACE w / o attention” and ”r-FACE”. Our method out-performs ”r-FACE w / o attention” on both FID and MS-SSIM, which indicates that example-guided attention modulecan explicitly transfer the corresponding face components ofa reference image and reduce the impact of other informationof the reference image.

Loss Configurations. In this section, we conduct ablationstudies on different loss, such as the contextual loss, the styleloss and the perceptual loss, for evaluating their individualcontributions on our framework. As shown in TABLE 1,the quantitative comparison among ”r-FACE w / o contextualloss”, ”r-FACE w / o style loss”, ”r-FACE w / o perceptualloss” and ”r-FACE” are presented. ”r-FACE” outperformsother loss configurations on MS-SSIM, which indicates theeffectiveness of the three losses. It is observed that the FIDof ”r-FACE” is also better than other loss configurations ex-cept ”r-FACE w/o contextual loss”. We argue that contextualloss plays an important role in restricting the shape of facecomponents. The framework has no constraints on the shapeafter removing the contextual loss, so ”r-FACE w / o contex-tual loss” is easier to generate missing components, merelyrequiring generated images to look real. Figure 6 comparesthe visual effects of different loss configurations. We find ”r-FACE w / o contextual loss” could not synthesize componentswith the corresponding shape of reference images, which in-dicates the contextual loss is significantly important in shapeconstraints. In the results of ”r-FACE w / o style loss” and”r-FACE w / o perceptual loss”, color distortions and obviousghosting artifacts are observed, which show the style loss andthe perceptual loss are able to preserve skin color of sourceimage. Above all, we prove that three losses contribute tothe performance of our framework and the combination of alllosses achieves the best results.

4.6 Discussion and LimitationReference guided face component editing has wide real-lifeapplications in interactive entertainment, security and dataaugmentation. Given whole reference images of any iden-tity, r-FACE performs various face component editing, whichcan achieve the effect of ”plastic surgery”. Besides, whenall face components of a reference face are transformed tothe source face, r-FACE is further extended to face swappingtask. Moreover, As the face component is a part of face, edit-ing face components may change the identity of the person.Therefore, r-FACE can be used for data augmentation to gen-erate face images of different identities. Although r-FACEobtains diverse and controllable face component editing re-sults, reference images with significant differences in poseare still challenging for our model. We will continue to ex-plore solving extreme pose problems in further work.

5 ConclusionIn this work, we have proposed a novel framework, refer-ence guided face components editing (r-FACE), for high-level face components editing, such as eyes, nose and mouth.

Figure 6: Visual comparisons for different loss configurations, in-cluding ”r-FACE w / o contextual loss”, ”r-FACE w / o style loss”,”r-FACE w / o perceptual loss” and ”r-FACE”.

r-FACE can achieve diverse, high-quality, and controllablecomponent changes in shape from given references, whichbreaks the shape and precise intermediate presentation lim-itation of existing methods. For embedding the target facecomponents of reference images to source images specifi-cally, an example-guided attention module is designed to fusethe features of source images and reference images, furtherboosting the performance of face component editing. Theextensive experiments demonstrate that our framework canachieve state-of-art face editing results with observable geo-metric changes.

AcknowledgmentsThis work was partially supported by the Natural Sci-ence Foundation of China (Grant No. U1836217, GrantNo. 61702513, Grant No. 61721004, and Grant No.61427811). This research was also supported by Meituan-Dianping Group, CAS-AIR and Shandong Provincial KeyResearch and Development Program (Major Scientific andTechnological Innovation Project) (NO.2019JZZY010119).

References[Bertalmio et al., 2000] Marcelo Bertalmio, Guillermo

Sapiro, Vincent Caselles, and Coloma Ballester. Imageinpainting. In SIGGRAPH, pages 417–424, 2000.

[Bertalmio et al., 2003] Marcelo Bertalmio, Luminita Vese,Guillermo Sapiro, and Stanley Osher. Simultaneous struc-ture and texture image inpainting. IEEE TIP, 12(8):882–889, 2003.

[Cao et al., 2019] Jie Cao, Yibo Hu, Bing Yu, Ran He, andZhenan Sun. 3D aided duet GANs for multi-view faceimage synthesis. IEEE TIFS, 14(8):2028–2042, 2019.

[Choi et al., 2018] Yunjey Choi, Minje Choi, MunyoungKim, Jung-Woo Ha, Sunghun Kim, and Jaegul Choo. Star-GAN: Unified generative adversarial networks for multi-domain image-to-image translation. In CVPR, pages8789–8797, 2018.

[Criminisi et al., 2003] Antonio Criminisi, Patrick Perez,and Kentaro Toyama. Object removal by exemplar-basedinpainting. In CVPR, volume 2, pages II–II, 2003.

[Dolhansky and Canton Ferrer, 2018] Brian Dolhansky andCristian Canton Ferrer. Eye in-painting with exemplargenerative adversarial networks. In CVPR, pages 7902–7911, 2018.

[Goodfellow et al., 2014] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley,Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Gen-erative adversarial nets. In NeurIPS, pages 2672–2680,2014.

[Gu et al., 2019] Shuyang Gu, Jianmin Bao, Hao Yang,Dong Chen, Fang Wen, and Lu Yuan. Mask-guided por-trait editing with conditional gans. In CVPR, pages 3436–3445, 2019.

[He et al., 2019] Zhenliang He, Wangmeng Zuo, Meina Kan,Shiguang Shan, and Xilin Chen. AttGAN: Facial attributeediting by only changing what you want. IEEE TIP, 2019.

[Heusel et al., 2017] Martin Heusel, Hubert Ramsauer,Thomas Unterthiner, Bernhard Nessler, and Sepp Hochre-iter. GANs trained by a two time-scale update ruleconverge to a local Nash Equilibrium. In NeurIPS, pages6626–6637, 2017.

[Iizuka et al., 2017] Satoshi Iizuka, Edgar Simo-Serra, andHiroshi Ishikawa. Globally and locally consistent imagecompletion. ACM ToG, 36(4):107, 2017.

[Jo and Park, 2019] Youngjoo Jo and Jongyoul Park. SC-FEGAN: Face editing generative adversarial network withuser’s sketch and color. In ICCV, October 2019.

[Johnson et al., 2016] Justin Johnson, Alexandre Alahi, andLi Fei-Fei. Perceptual losses for real-time style transferand super-resolution. In ECCV, pages 694–711, 2016.

[Langner et al., 2010] Oliver Langner, Ron Dotsch, Gijs-bert Bijlstra, Daniel HJ Wigboldus, Skyler T Hawk, andAD Van Knippenberg. Presentation and validation ofthe Radboud Faces Database. Cognition and Emotion,24(8):1377–1388, 2010.

[Lee et al., 2019] Cheng-Han Lee, Ziwei Liu, Lingyun Wu,and Ping Luo. MaskGAN: Towards diverse and in-teractive facial image manipulation. arXiv preprintarXiv:1907.11922, 2019.

[Liu et al., 2015] Ziwei Liu, Ping Luo, Xiaogang Wang, andXiaoou Tang. Deep learning face attributes in the wild. InICCV, pages 3730–3738, 2015.

[Liu et al., 2019] Yunfan Liu, Qi Li, and Zhenan Sun.Attribute-aware face aging with wavelet-based genera-tive adversarial networks. In CVPR, pages 11877–11886,2019.

[Mao et al., 2017] Xudong Mao, Qing Li, Haoran Xie, Ray-mond YK Lau, Zhen Wang, and Stephen Paul Smolley.Least squares generative adversarial networks. In ICCV,pages 2794–2802, 2017.

[Mechrez et al., 2018] Roey Mechrez, Itamar Talmi, andLihi Zelnik-Manor. The contextual loss for image transfor-mation with non-aligned data. In ECCV, pages 768–783,2018.

[Simonyan and Zisserman, 2014] Karen Simonyan and An-drew Zisserman. Very deep convolutional networksfor large-scale image recognition. arXiv preprintarXiv:1409.1556, 2014.

[Song et al., 2019] Linsen Song, Jie Cao, Linxiao Song,Yibo Hu, and Ran He. Geometry-aware face completionand editing. In AAAI, pages 2506–2513, 2019.

[Wang et al., 2003] Zhou Wang, Eero P Simoncelli, andAlan C Bovik. Multiscale structural similarity for imagequality assessment. In ACSSC, volume 2, pages 1398–1402, 2003.

[Wu et al., 2019] Po-Wei Wu, Yu-Jing Lin, Che-Han Chang,Edward Y Chang, and Shih-Wei Liao. RelGAN: Multi-domain image-to-image translation via relative attributes.In ICCV, pages 5914–5922, 2019.

[Xiao et al., 2018] Taihong Xiao, Jiapeng Hong, and JinwenMa. Elegant: Exchanging latent encodings with gan fortransferring multiple face attributes. In ECCV, pages 168–184, 2018.

[Yang et al., 2018] Hongyu Yang, Di Huang, YunhongWang, and Anil K Jain. Learning face age progression:A pyramid architecture of gans. In CVPR, pages 31–39,2018.

[Zheng et al., 2019] Chuanxia Zheng, Tat-Jen Cham, andJianfei Cai. Pluralistic image completion. In CVPR, pages1438–1447, 2019.