Image Decomposition, Inpainting, and Impulse Noise Removal by
Sparse & Redundant Representations
Michael EladThe Computer Science DepartmentThe Technion – Israel Institute of technologyHaifa 32000, Israel
Sparse representations for Image Decomposition
2
General
Sparsity and over-completeness have important roles in analyzing and representing signals.
We have already discussed the image separation task (image=cartoon + texture) and inpainting based on it, adopting a global treatment of these problems.
We have seen in the context of denoising that migration from global to local processing led to far better results.
We Now return to these tasks (and beyond), with a patch-based approach, with the hope to do better.
Before – Lets recall the global treatment
Sparse representations for Image Decomposition
3
Global Separation &
Inpainting
Sparse representations for Image Decomposition
4
Decomposition via Sparsity
yx
00,
s.t.sArgMinˆˆ
y
x
= s+
The dictionaries x y should be chosen such that they•sparsify the desired content, while •giving poor service to the opposite content
Sparse representations for Image Decomposition
5
Noise Considerations
2
2yx11,
sArgMinˆˆ
yx
11,
s.t.sArgMinˆˆ
Forcing exact representation is sensitive to additive noise
and model mismatch
Sparse representations for Image Decomposition
6
Original ‘Barbara’ image
Separated texture using local overlapped DCT (32×32 blocks)
Separated Cartoon using Curvelets (5 resolution layers)
Results – Good old ‘Barbara’
Sparse representations for Image Decomposition
7
Results – Zoom in
Zoom in on the result
shown in the previous slide
(the texture part)
Zoom in on the results
shown in the previous slide
(the cartoon part)
The same part taken from Vese’s et. al.
The same part taken from Vese’s et. al.
Sparse representations for Image Decomposition
8
Application - Inpainting
2
2yx11,
sArgMinˆˆ
For separation
What if some values in s are unknown (with known locations!!!)?
2x y1 1 2,
ˆArgMin s
ˆM
The image will be the inpainted outcome. Interesting comparison to [Bertalmio et.al.
’02]
βΦαΦ yx
Sparse representations for Image Decomposition
9
Results - Inpainting
Source
Cartoon Part
Texture Part
Outcome
Sparse representations for Image Decomposition
10
Results - Inpainting
Source
Cartoon Part
Texture Part
Outcome
Sparse representations for Image Decomposition
11
Results - Inpainting
Outcome
Source
Sparse representations for Image Decomposition
12
Patch-Based (Local)
Separation
Sparse representations for Image Decomposition
13
MCA for Patches
CXTX T CY X X V
Lets impose the MCA model on each and every patch-set in these images
kk T CT Ck T Ck k 2k 2
YX X V
R D DR D D
Sparse representations for Image Decomposition
14
Start by Denoising
kT C k
kand qD D D
k k
22kk k2 0 2X, , q k
min X Y q q XD
D R
We define ,
and formulate a patch-based MAP denoising problem:
We are familiar with this expression: it aims to remove the noise V from Y, bypassing altogether the separation task.
k kq Compute
Sparse Coding of patches
Compute D K-SVD Update
Compute X Patch-
averaging
kk T C
k 2
YR D D
Sparse representations for Image Decomposition
15
Decompose the Dictionary
The denoising process leads to a dictionary D, assumed to be composed of DC and DT.
We need to divide the content of D into these two “sub-dictionaries”, and we can do this by measuring the Total-Variation.
Any atom with “activity” below a threshold is assigned as a cartoon atom, and similarly, to the texture one if above the threshold.
DC & DT
Sparse representations for Image Decomposition
16
Final Image Separation
Now that DC and DT are defined, we can separate.
Starting with
the coefficients in qk are split between the two dictionaries.
We obtain the system:
22kk k2 0 2X k
min X Y q q XD R
22
T C T CT C k kk k2 2X kmin X X Y X XD D R R
22
T CT k C kk k2 2X XD R D R
T Tk k k T kT
k kT Tk k k C kC k k
YX
YX
I R R I R D
I I R R R D
Sparse representations for Image Decomposition
17
Results
The obtained results are of similar quality (and slightly better) than the
global approach
Sparse representations for Image Decomposition
18
Results
Zoom in on the texture
global result shown before
Zoom in on the cartoon
global result shown before
The same part taken from the local approach
The same part taken from the local approach
Sparse representations for Image Decomposition
19
Patch-Based (Local)
Inpainting
Sparse representations for Image Decomposition
20
Lets Start with a Simple Experiment
We assume that the image is corrupted by additive noise (=20), and z% missing pixels (the Mask is given)
Lets break the image into a set of non-overlapping patches, and inpaint each separately by (D: 2D-DCT)
Once done, the patches are put into position to create the inpainted image.
0
2 2k k k2
min
s.t. Y cnM D R
Sparse representations for Image Decomposition
21
Lets Start with a Simple Experiment
25% missing 50% missing 75% missing
RMSE for 75%missing
RMSE for 50% missing
RMSE for 25% missing
Algorithm
29.70 19.61 14.55 No-overlap
Sparse representations for Image Decomposition
22
How About Overlapped Patches?
RMSE for 75%missing
RMSE for 50% missing
RMSE for 25% missing
Algorithm
29.70 19.61 14.55 No-overlap
18.18 11.55 9.00 Overlap
25% missing 50% missing 75% missing
Sparse representations for Image Decomposition
23
Formulating This Problem: K-SVD
k k
2 2G kk k0 22X, , k
min X Y XD
M D R
k k Compute
Sparse Coding of patches
Compute D K-SVD Update
Compute X Patch-
averaging
2 2
k k k0 2min s.t. Y cnM D R
Initialize with X=MGY
Sparse coding:
Dictionary Update:
Image Update:
2k k 2
kmin YD
M D R
2 2G k 22X k
min X Y XM D R
Sparse representations for Image Decomposition
24
Results
RMSE for 75%missing
RMSE for 50% missing
RMSE for 25% missing
Alg.
29.70 19.61 14.55 No-overlap
18.18 11.55 9.00 Overlap
17.74 10.05 8.1 K-SVD
For the Peppers image
This is a more challenging case, where the DCT is not a suitable dictionary. • For Redundant DCT we get
RMSE=16.13, and • For K-SVD (15 iterations) we
get RMSE=12.74
Sparse representations for Image Decomposition
25
Inpainting Color Images
Sparse representations for Image Decomposition
26
Inpainting Video
Sparse representations for Image Decomposition
27
Patch-Based Impulse-Noise
Removal
Sparse representations for Image Decomposition
28
Introduction
In the inpainting problem discussed above, the mask is KNOWN. What do we do if the mask (missing pixels) is unknown – this is the impulse noise problem.
Definition: we obtain an image with p% of the pixels in random locations corrupted by adding ±50 to the gray-level. This is known as “salt-and-pepper” noise.
This is an example with 10% corrupted pixels:
Most common method for filtering this noise: Median filtering
Sparse representations for Image Decomposition
29
Sparse-Land Alternative
We will use the median filter as a simple (and primitive) detector of missing pixels (there are better ways!):
Once the mask is available, we shall proceed with regular inpainting
… BUT …
Applying the inpainting ONLY for the masked pixels
For a fair comparison, we will adopt the same mask for the median filter., i.e., modify the pixels having Mask=0 only.
k0 Y Median Y 27
Mask1 Otherwise
Sparse representations for Image Decomposition
30
Results
The image with the
detected mask
Masked median filter,
RMSE=6.52
Inpainting by DCT, RMSE=5.86
Regular Median filter, RMSE=7.42