image super-resolution using sparse representation by: michael elad single image super-resolution...

Image Super-Resolution Using Sparse RepresentationBy: Michael Elad

Single Image Super-Resolution Using Sparse Representation

Michael EladThe Computer Science DepartmentThe Technion – Israel Institute of technologyHaifa 32000, Israel

MS45: Recent Advances in Sparse and Non-local Image Regularization - Part III of

III Wednesday, April 14, 2010


*

* Joint work with Roman Zeyde and Matan Protter



The Super-Resolution Problem

2

hz y v SH

hyz

Blur (H) and

Decimation (S)

+WGN

v

Our Task: Reverse the process – recover the high-resolution image from the

low-resolution one



Single Image Super-Resolution

3

The reconstruction process should rely on:The given low-resolution imageThe knowledge of S, H, and statistical properties of v,

and Image behavior (prior).

Recovery Algorithm

hyz ?

In our work: We use patch-based sparse and redundant

representation based prior, and We follow the work by Yang, Wright, Huang, and Ma

[CVPR 2008, IEEE-TIP – to appear], proposing an improved algorithm.



Core Idea (1) - Work on Patches

4

z

hy

Interp.Q

y

Enhance Resolution

Every patch from should go through a process of resolution enhancement. The improved patches are then merged together into the final image (by averaging).

y

y zQ

We interpolate the low-res. Image in order to align the coordinate systems



Core Idea (2) – Learning [Yang et. al. ’08]

5

Enhance Resolution?

We shall perform a sparse decomposition of the low-res. patch, w.r.t. a learned dictionary A

We shall construct the high res. patch using the same sparse representation, imposed on a dictionary

hA

and now, lets go into the details …



The Sparse-Land Prior [Aharon & Elad, ’06]

6

hy Extraction of a patch from y

h in location k is

performed by hk k hp yRn

n

Model Assumption: Every such patch can be represented sparsely over the dictionary Ah:

Position k

hk k

hk h k k 0

p q such that

p q and q nA

m

n Akq

hkp

h



Low Versus High-Res. Patches

7

hy

n

n

Position k

z

InterpolationQ

y

hk k 2

p p L

L = blur + decimation + interpolation. This expression relates the low-res.

patch to the high-res. one, based on

hz y v SH

hkp

n

n

kp?



Low Versus High-Res. Patches

8

hk h kp qAh

k k 2p p L

We interpolate the low-res. Image in order to align the coordinate systems

Interpolation

z

hy

n

n

Position ky

n

n

h k k 2q p LA qk is ALSO the

sparse representation of the low-resolution patch,

with respect to the dictionary LAh.

A



Training the Dictionaries – General

9

Training pair(s)

Interpolation

hk

k k

p

p

We obtain a

set of matching

patch-pairs

Pre-process(low-res)

Pre-process(high-res)

k kp

hk k

p These will be

used for the

dictionary training



Alternative: Bootstrapping

10

z

The given image to be scaled-up

hx z vSH

xBlur (H)

and Decimation

(S)+WGN

v

Simulate the degradation process

Use this pair of images to generate the patches for the

training



Pre-Processing High-Res. Patches

11

z

x

Interpolation

-

High-Res

Low-Res

hk k

p

Patch size: 9×9



Pre-Processing Low-Res. Patches

12

x

Interpolation

Low-Res

* [0 1 -1]

* [0 1 -1]T

* [-1 2 -1]* [-1 2 -1]T

We extract patches of size 9×9 from each of these images, and concatenate them to form one vector of length 324

k kp

Dimensionality Reduction

By PCA

k kp

Patch size: ~30h h

k kp pB



Training the Dictionaries:

13

A

K-SVD Am=1000

N=30

40 iterations, L=3

k k

pFor an image of size 1000×1000 pixels, there are ~12,000 examples to train on

kk k k 02q

min p q s.t. q LA

Given a low-res. Patch to be scaled-up, we start the resolution enhancement by sparse coding it, to find qk:

Remember




14

hA

kk k k 02q

min p q s.t. q LA

Given a low-res. Patch to be scaled-up, we start the resolution enhancement by sparse coding it, to find qk:

Remember

And then, the high-res. Patch is obtained by h

k h kp qA

Thus, Ah should be designed such that

h

2hk h k 2

k

p q minAA




15

hA

h

2hk h k 2

k

p q minAA

However, this approach disregards the fact that the resulting high-res. patches are not used directly as the final result, but averaged due to overlaps between them.

A better method (leading to better final scaled-up image) would be – Find Ah such that the following error is minimized:

h h

Tk

22

h h h kh2k 2

ˆmin y y min y y qA A

R AW

The constructed image



Overall Block-Diagram

16

Training the low-res. dictionary

zA

Training the high-res. dictionary

hA

The recovery algorithm

(next slide)

On-line or off-line



The Super-Resolution Algorithm

17

z

Interpolation

y

multiply by B to reduce dimension

* [0 1 -1]

* [0 1 -1]T

* [-1 2 1]

* [-1 2 1]T

Extract patches

k kp

Sparse coding using Aℓ to compute

kq

k k

pMultiply by Ah and combine by averaging the overlaps

hy



Relation to the Work by Yang et. al.

18

Training the low-res. dictionary

zA

Training the high-res. dictionary

hA

The recovery algorithm

(previous slide)

We can train on the given image or a training set

We reduce dimension prior

to training

We train using the K-SVD algorithm

We use a direct approach to the

training of Ah

We train on a difference-

image Bottom line: The proposed algorithm is much simpler,

much faster, and, as we show next, it also leads to better

results

We use OMP for sparse-

coding

We avoid post-processing

We interpolate to avoid coordinate

ambiguity



Results (1) – Off-Line Training

19

The training image: 717×717 pixels, providing a set of 54,289 training patch-pairs.



20

Results (1) – Off-Line Training

Ideal Image

Given Image

SR ResultPSNR=16.95dB

Bicubic interpolation PSNR=14.68

dB



21

Results (2) – On-Line Training

Given image

Scaled-Up (factor 2:1) using the proposed algorithm, PSNR=29.32dB (3.32dB improvement over bicubic)



22


The Original Bicubic Interpolation SR result



23


The Original Bicubic Interpolation SR result



Comparative Results – Off-Line

24

Our alg. Yang et. al. Bicubic

SSIM PSNR SSIM PSNR SSIM PNSR

0.78 26.77 0.76 26.39 0.75 26.24 Barbara

0.66 27.12 0.64 27.02 0.61 26.55 Coastguard

0.82 33.52 0.80 33.11 0.80 32.82 Face

0.93 33.19 0.91 32.04 0.91 31.18 Foreman

0.88 33.00 0.86 32.64 0.86 31.68 Lenna

0.79 27.91 0.77 27.76 0.75 27.00 Man

0.94 31.12 0.93 30.71 0.92 29.43 Monarch

0.89 34.05 0.87 33.33 0.87 32.39 Pepper

0.91 25.22 0.89 24.98 0.87 23.71 PPT3

0.84 28.52 0.83 27.95 0.79 26.63 Zebra

0.85 30.04 0.83 29.59 0.81 28.76 Average



25

Comparative Results - Example

Yang et. al.Ours



Summary

26

Single-image scale-up –

an important

problem, with many

attempts to solve it in

the past decade.

The rules of the game:

use the given image,

the known degradation,

and a sophisticated

image prior.

Yang et. al. [2008] – a

very elegant way to

incorporate sparse &

redundant

representation prior

We introduced

modifications to Yang’s

work, leading to a simple,

more efficient alg. with

better results. More work is required to

improve further the results

obtained.


Michael Elad

A New Book


27

SPARSE AND REDUNDANT REPRESENTATIONS

From Theory to Applications in Signal

and Image Processing

Michael Elad

Thank You Very Much !!

image super-resolution using sparse representation by: michael elad single image super-resolution...

Documents

superresolution problem

michael elad low

highresolution image

final image

sparse decomposition

redundant representation

michael elad core idea

given lowresolution