photo enhancement with inpaintingphoto manipulations enhance quality remove scratches and spots...

Photo Enhancement with Inpainting

Gustaf Kylberg1, Paul Nemes2, Martin Maska3, andSandor Fazekas4

1Uppsala University, Sweden2Technical University of Cluj-Napoca, Romania

3Masaryk University, Czech Republic4Computer and Automation Institute (SZTAKI), Hungary

SSIP 2008

Project PresentationJuly 15, 2008

G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting

The group

Gustaf Kylberg, Paul Nemes, Martin Maska, and Sandor Fazekas

G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting

Photo manipulations

Enhance qualityRemove scratches and spots“Highly adaptive median filter”Photo restoration

Enhance contentRemove unwanted parts

Objects, peopleWires, graffitiTV logo, text

Change the appearance of peopleGlassesMustache, beardTattoo

Make people look nicerRemove wrinkles, freckles

. . .Nikolai Yezhov removed

G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting

Work flow of the proposed method

G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting

Defect detection

Color based segmentationHough transform

Detect linesLong scratchesWires (disturbing the view)

Detect circlesCircular blobsDirty spots

Parallel close edgesThreshold on large gradientsThin defected areas assumed

Semi-automatic defect detectionRefine a manually selected regionSuitable for thin scratches and small blobs

G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting

Implemented inpainting methods

Input and mask1

Output of inpainting

Isotropic diffusion Anisotropic diffusion Distance transform

1Intentionally so bad :)G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting

Inpainting with diffusion

Diffusion process

∂tρ =∑α,β

∂α

(Dαβ∂βρ

)(1)

Applied to inpaint a region Ω (defined by µ(Ω) = 0)ρ and the derivatives of ρ are known outside the maskAssume some initial value inside the maskSolve Eq. (1) on Ω with fixed boundary conditionsLet the system evolve until a steady state is reached∑

α,β

∂α

(Dαβ∂βρ

)= 0 (2)

Result = Smooth patchTexture synthesized in an additional step

G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting

Inpainting with diffusion

Isotropic diffusionDαβ = Dδαβ (3)

Tends to smooth away edgesAnisotropic diffusion

Eigenvectors parallel and perpendicular to image gradientsDiffusion only perpendicular to gradient⇒ Keeps edges

Can be applied to multichannel images tooThe diffusivity Dαβ should be the same for all channelsDefine a norm in the “color space”Diffusion along isophote lines

Convergence speeded up with a multiscale approachBuild a Gaussian pyramid and apply diffusion on all scalesPropagate information for lower to higher scalesStart diffusion form the result obtained on lower scale

G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting

Discretization

Isotropic diffusion

ρt+1 − ρt = D(ρt − 4ρt+1) (4)

where

ρ =

0 1 01 0 10 1 0

? ρ (5)

Information taken from neighboring pixelsIterative solution

ρt+1 =ρt + Dρt

1 + 4D(6)

Anisotropic diffusion discretized in a similar wayMultiscale approach can be applied to speed up convergence

G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting

Inpainting Based on Distance Transform

Distance to closest non-zero pixel

Input OutputApply distance transform on the mask µ→ d(µ)

Update image within the mask

ρ(r)← ρ

(r + d(µ)

∇d(µ)

‖∇d(µ)‖

)(7)

G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting

Applications

G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting

Image restoration

G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting

Image restoration

Original Enhanced

G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting

Image restoration

Original Enhanced

G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting

Image restoration

Original Enhanced

G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting

Image restoration

Original Enhanced

G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting

Removing unwantedimage regions

G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting

Removing objects

Original Enhanced

G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting

Removing text

Original Enhanced

Fully automatic!Using color based segmentation

G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting

Removing wires

Original Enhanced

Fully automatic!Using Hough transform

G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting

Removing wires

Original Enhanced

Fully automatic!Using Hough transform

G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting

Removing wires

Original Enhanced

G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting

Making people look niceron photographs

G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting

Removing freckles

Original Enhanced

G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting

Removing freckles

Original Enhanced

G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting

Removing wrinkles

Original Enhanced

High frequency components of the texture kept

G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting

Removing wrinkles

Original Enhanced

High frequency components of the texture kept

G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting

Removing wrinkles

Original Enhanced

High frequency components of the texture kept

G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting

Removing wrinkles

Original Enhanced

High frequency components of the texture kept

G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting

Changing the appearanceof people

G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting

Removing glasses and mustache

Original Enhanced

G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting

Removing glasses and mustache

Original Enhanced

G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting

Limitations!

G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting

Limitations

Original Enhanced

G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting

Thank you for yourattention!

G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting