1

Image Completion Image Completion using Global using Global OptimizationOptimizationPresented by Tingfan WuPresented by Tingfan Wu

2

The Image Inpainting ProblThe Image Inpainting Problemem

3

OutlineOutline

IntroductionIntroduction History of InpaintingHistory of Inpainting Camps – Greedy & Global Opt.Camps – Greedy & Global Opt.

Model and AlgorithmModel and Algorithm Markov Random Fields (MRF) & InpaintingMarkov Random Fields (MRF) & Inpainting Belief Propagation (BP)Belief Propagation (BP) Priority BPPriority BP

ResultsResults Structural PropagationStructural Propagation

4

Method TypeMethod TypeInpainting

Bertalmio et al. SIGGRAPH 2000

Statistical Example BasedTexture Synth./Patch

GreedyCriminisi et al. 2003

Global Optimization

Priority BP[This paper]

Structural Propagation[Sun et. al, SIGGRAPH 05]

PDEBertalmio et al. 2001

PriorityTexture Syn

th.

Need User Guidance

5

Exampled Based MethodExampled Based Method—Jigsaw Puzzle—Jigsaw Puzzle

Patch

es

Not A

vaila

ble

6

Method TypeMethod TypeInpainting

Bertalmio et al. SIGGRAPH 2000

Statistical Example BasedTexture Synth./Patch

GreedyCriminisi et al. 2003

Global Optimization

Priority BP[This paper]

Structural Propagation[Sun et. al, SIGGRAPH 05]

PDEBertalmio et al. 2001

PriorityTexture Syn

th.

Need User Guidance

7

Greedy v.s Global OptmizatiGreedy v.s Global Optmizationon

Greedy Method

Cannot go back

Global Optimization

Ooops

Refine Globally

8

OutlineOutline

IntroductionIntroduction History of InpaintingHistory of Inpainting Camps – Greedy & Global Opt.Camps – Greedy & Global Opt.

Model and AlgorithmModel and Algorithm Markov Random Fields (MRF) & InpaintingMarkov Random Fields (MRF) & Inpainting Belief Propagation (BP)Belief Propagation (BP) Priority BPPriority BP

ResultsResults Structural PropagationStructural Propagation

9

Random Fields / Belief Random Fields / Belief NetworkNetwork

Good Project Writer?(High Project grade)

Good Test Taker?(High test score)

Smart Student?(High GPA)

Good Employee(No Observation yet)Edge:

Dependency

RFRF：： Random Variables on GraphRandom Variables on Graph Node : Random Var. (Hidden State) Node : Random Var. (Hidden State) Belief : from Neighbors, and Observation Belief : from Neighbors, and Observation

Random Variable(Observation)

10

Story about MRFStory about MRF

(Bayesian) Belief Network (DAG)(Bayesian) Belief Network (DAG) Markov Random Fields (Undirected, Markov Random Fields (Undirected,

Loopy) Loopy) Special Case:Special Case:

1D - Hidden Markov Model (HMM)1D - Hidden Markov Model (HMM)

Office Helper Wizard Hidden Markov Model (HMM)Hidden Markov Model (HMM)

11

Inpainting as MRF optimizaInpainting as MRF optimizationtion

Node : Grid on target region, overlapped patchesNode : Grid on target region, overlapped patches Edge : A Edge : A nodenode depends only on its depends only on its neighborsneighbors Optimal labeling (hidden state) that minimizing Optimal labeling (hidden state) that minimizing

mismatch energymismatch energy

12

MRF Potential FunctionsMRF Potential Functions

Mismatch (Energy) between ..Mismatch (Energy) between .. VVp p (X(Xp p ) : Source Image vs. New Label) : Source Image vs. New Label VVpqpq(X(Xpp, X, Xqq) : Adjacent Labels) : Adjacent Labels SSum of um of SSquare quare DDistances (SSD) in Overlapping Regioistances (SSD) in Overlapping Regio

nn

13

Global OptimizatoinGlobal Optimizatoin

min

14

OutlineOutline

IntroductionIntroduction History of InpaintingHistory of Inpainting Camps – Greedy & Global Opt.Camps – Greedy & Global Opt.

Model and AlgorithmModel and Algorithm Markov Random Fields (MRF) & InpaintingMarkov Random Fields (MRF) & Inpainting Belief Propagation (BP)Belief Propagation (BP) Priority BPPriority BP

ResultsResults Structural PropagationStructural Propagation

15

Belief Propagation(1/3)Belief Propagation(1/3)

•Undirected and Loopy•Propagate forward and backward

Good Project Writer?(High Project grade)

Good Test Taker?(High test score)

Smart Student?(High GPA)

Good Employee(No Observation yet)

16

Belief Propagation(2/3)Belief Propagation(2/3) Message ForwardingMessage Forwarding Iterative algorithm until convergeIterative algorithm until converge

Xq

Xp

Candidates at Node Q

Candidates at Node P

Neighbors (P)

O(|Candidate|2)

17

Belief Propagation(3/3)Belief Propagation(3/3)

18

Priority BPPriority BP BP too slow: BP too slow:

Huge #candidates Huge #candidates Time Timemsgmsg = O(|Candidat = O(|Candidates|es|22) )

Huge #Pairs Huge #Pairs Cannot cache pairwise SSDs.Cannot cache pairwise SSDs. ObservationsObservations

Non-Informative messages in unfilled regiNon-Informative messages in unfilled regions ons

Solution to some nodes is obvious (fewer cSolution to some nodes is obvious (fewer candidates.)andidates.)

19

Human WisdomHuman Wisdom

Nobody start from here

Start from non-ambiguous part

AndSearch for

Brown feather+green grass

Candidates

20

Priority BPPriority BP ObservationsObservations

needless messages in unfilled regions needless messages in unfilled regions Solution to some nodes is obvious (fewer Solution to some nodes is obvious (fewer

candidates.)candidates.) Solution: Enhanced BP:Solution: Enhanced BP:

Easy nodes goes first (priority message Easy nodes goes first (priority message scheduling)scheduling)

Keep only highly possible candidates Keep only highly possible candidates (maintain a Active Set)(maintain a Active Set)

21

Priority & PruningPriority & Pruning

?? ?

? ? ?

?? High Priorityprune a lot

Pruning may miss correct label

Low Priority

Candidates sorted by relative belief

Discard Blue Points

22

#Candidates after #Candidates after PruningPruning

Active Set (Darker means smaller)

Histogram of #candidates Similar candidates

23

A closer look at Priority A closer look at Priority BPBP Priority CalculationPriority Calculation

Priority : 1/(#significant candidate)Priority : 1/(#significant candidate) Pruning (on the fly )Pruning (on the fly )

Discard Low Confidence CandidatesDiscard Low Confidence Candidates Similar patches Similar patches One representative (by One representative (by

clustering)clustering) ResultResult

More Confident More Confident More Pruning More Pruning Confident node helps increase neighbor’s Confident node helps increase neighbor’s

confidence.confidence. Warning:Warning:

PBP and Pruning must be used PBP and Pruning must be used togethertogether

25

OutlineOutline IntroductionIntroduction

History of InpaintingHistory of Inpainting Camps – Greedy & Global Opt.Camps – Greedy & Global Opt.

Model and AlgorithmModel and Algorithm Markov Random Fields (MRF) & InpaintingMarkov Random Fields (MRF) & Inpainting Belief Propagation (BP)Belief Propagation (BP) Priority BPPriority BP

ResultsResults ConclusionConclusion Structural PropagationStructural Propagation

26

Darker pixels higher priorityAutomatically start from salient parts.

Results-Inpainting(1/3)Results-Inpainting(1/3)

27

Results-Inpainting(2/3)Results-Inpainting(2/3)

28

Results-Inpainting(3/3)Results-Inpainting(3/3)Up to 2minutes / image (256x170) on P4-2.4GUp to 2minutes / image (256x170) on P4-2.4G

29

More : Texture SynthesisMore : Texture SynthesisInterpolation as well as extrapolationInterpolation as well as extrapolation

30

31

ConclusionConclusion

Priority BPPriority BP {Confident node first} + {candidate pruning}{Confident node first} + {candidate pruning} Generic – applicable to other MRF problems.Generic – applicable to other MRF problems. Speed upSpeed up

MRF for InpaintingMRF for Inpainting Global optimization Global optimization

avoid visually inconsistence by greedyavoid visually inconsistence by greedy Priority BP for InpaintingPriority BP for Inpainting

Automatically start from salient point. Automatically start from salient point.

32

Sometimes …Sometimes … Image contains hard high-level Image contains hard high-level

structurestructure Hard for computersHard for computers Interactive completion guided by Interactive completion guided by

human.human.

33

Potential Func. For Structural PropagaPotential Func. For Structural Propagationtion

User input a guideline by human User input a guideline by human region.region.

Potential Function respect distance Potential Function respect distance between linesbetween lines

Jian Sun et al, SIGGRAPH 2005Jian Sun et al, SIGGRAPH 2005

34

VideoVideo

Link:Link:MicrosoftMicrosoft Research Research