ee565 advanced image processing copyright xin li 20081 motivating applications hdtv internet video...

25
EE565 Advanced Image Processi ng Copyright Xin Li 2008 1 Motivating Applications HDTV Internet video Artistic reproduction Widescreen movie

Upload: ashlynn-caldwell

Post on 04-Jan-2016

218 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: EE565 Advanced Image Processing Copyright Xin Li 20081 Motivating Applications HDTV Internet video Artistic reproduction Widescreen movie

EE565 Advanced Image Processing Copyright Xin Li 2008

1

Motivating Applications

HDTVInternet video

Artistic reproduction Widescreen movie

Page 2: EE565 Advanced Image Processing Copyright Xin Li 20081 Motivating Applications HDTV Internet video Artistic reproduction Widescreen movie

EE565 Advanced Image Processing Copyright Xin Li 2008

2

Image Interpolation

Image Interpolation Importance of geometry Edge directed interpolation PDE-based techniques

Applications Super-resolution Inpainting (error concealment) Compressive sensing

Page 3: EE565 Advanced Image Processing Copyright Xin Li 20081 Motivating Applications HDTV Internet video Artistic reproduction Widescreen movie

EE565 Advanced Image Processing Copyright Xin Li 2008

3

Importance of Geometry

It is geometry that distinguishes image signals (2D) from audio and speech (1D)

2D image is the projection of 3D geometry (a seemingly-trivial statement)

Geometric elements and properties: edge (location and orientation), object (convexity) … …

Page 4: EE565 Advanced Image Processing Copyright Xin Li 20081 Motivating Applications HDTV Internet video Artistic reproduction Widescreen movie

EE565 Advanced Image Processing Copyright Xin Li 2008

4

Geometric Constraint of Edges

3D visualizationsingle-edge image

• Along the edge orientation, intensity field is homogeneous

• Across the edge orientation, intensity field evolves fast

Observations:

Page 5: EE565 Advanced Image Processing Copyright Xin Li 20081 Motivating Applications HDTV Internet video Artistic reproduction Widescreen movie

EE565 Advanced Image Processing Copyright Xin Li 2008

5

Why geometry is difficult?

Geometry is embedded in the array of image pixels, which is not straightforward to be exploited by any linear operations (filtering or transform)

Image signals form a manifold in an extremely high-dimensional space (What is a manifold? Think of a string in the real world, it is a 3D object and a 1D manifold)

Page 6: EE565 Advanced Image Processing Copyright Xin Li 20081 Motivating Applications HDTV Internet video Artistic reproduction Widescreen movie

EE565 Advanced Image Processing Copyright Xin Li 2008

6

Interpolation Problem

downsampling

originalhigh-resolution

image X

low-resolution image Y

interpolatedhigh-resolution

image Z

How to make Z as close to X as possible?

Page 7: EE565 Advanced Image Processing Copyright Xin Li 20081 Motivating Applications HDTV Internet video Artistic reproduction Widescreen movie

EE565 Advanced Image Processing Copyright Xin Li 2008

7

bilinear Interpolation Review

(e+w)/2

(n+s)/2

(ne+nw+se+sw)/4

Page 8: EE565 Advanced Image Processing Copyright Xin Li 20081 Motivating Applications HDTV Internet video Artistic reproduction Widescreen movie

EE565 Advanced Image Processing Copyright Xin Li 2008

8

Why bilinear is bad?

Edge blurring Jagged artifacts

X Z

Jagged artifacts

X

Z

Edge blurring

Page 9: EE565 Advanced Image Processing Copyright Xin Li 20081 Motivating Applications HDTV Internet video Artistic reproduction Widescreen movie

EE565 Advanced Image Processing Copyright Xin Li 2008

9

Where does image quality degradation come from?

Violation of geometric constraint Keep in mind an edge could be

arbitrarily oriented

Edge blurring Jagged artifacts

Along the edge orientation,intensity field is inhomogeneous

Across the edge orientation, intensity field evolves slower

Page 10: EE565 Advanced Image Processing Copyright Xin Li 20081 Motivating Applications HDTV Internet video Artistic reproduction Widescreen movie

EE565 Advanced Image Processing Copyright Xin Li 2008

10

Ideas to Do Better

Recognize the importance of edges Location: where are they? Orientation: how are they oriented?

Identify appropriate mathematical tools for exploiting geometric constraints Heuristics Statistical vs. PDE

Page 11: EE565 Advanced Image Processing Copyright Xin Li 20081 Motivating Applications HDTV Internet video Artistic reproduction Widescreen movie

EE565 Advanced Image Processing Copyright Xin Li 2008

11

Heuristics: Edge Location

Intuitively, edges are characterized by fast intensity value variation (i.e., large gradient)

To avoid edge blurring, we need to spatially adaptively adjust interpolation coefficients

Question: how to incorporate gradient cue into a spatiallyadaptive interpolation scheme?

Page 12: EE565 Advanced Image Processing Copyright Xin Li 20081 Motivating Applications HDTV Internet video Artistic reproduction Widescreen movie

EE565 Advanced Image Processing Copyright Xin Li 2008

12

Desired Improvement

X Y Zbil Zesi

horizontaledge

verticaledge

Page 13: EE565 Advanced Image Processing Copyright Xin Li 20081 Motivating Applications HDTV Internet video Artistic reproduction Widescreen movie

EE565 Advanced Image Processing Copyright Xin Li 2008

13

Here Comes the Trick

Step 1: interpolate the missing pixels along the diagonal

black or white?

Step 2: interpolate the other half missing pixels

a b

c d

Since |a-c|=|b-d|x

x has equal probabilityof being black or white

a

b

c

dSince |a-c|>|b-d|

x=(b+d)/2=black

x

Page 14: EE565 Advanced Image Processing Copyright Xin Li 20081 Motivating Applications HDTV Internet video Artistic reproduction Widescreen movie

EE565 Advanced Image Processing Copyright Xin Li 2008

14

Edge Sensitive Interpolation (ESI)

b

c

d x

a

If |a-c|>|b-d|

x=(b+d)/2

If |a-c|<|b-d|

x=(a+c)/2

If |a-c|=|b-d|

x=min or max

a b

c d

x

In simple words, we want to avoid interpolate along thedirection which has a large gradient

Page 15: EE565 Advanced Image Processing Copyright Xin Li 20081 Motivating Applications HDTV Internet video Artistic reproduction Widescreen movie

EE565 Advanced Image Processing Copyright Xin Li 2008

15

Experiment Result

Bilinear interpolation Edge sensitive interpolation

Page 16: EE565 Advanced Image Processing Copyright Xin Li 20081 Motivating Applications HDTV Internet video Artistic reproduction Widescreen movie

EE565 Advanced Image Processing Copyright Xin Li 2008

16

Heuristics: Edge Orientation

Can we do better? Yes! Gradient is only a first-order

characteristics of edge location ESI makes binary decision with two

orthogonal directions How to do better?

We need some mathematical tool that can work with arbitrary edge orientation

Page 17: EE565 Advanced Image Processing Copyright Xin Li 20081 Motivating Applications HDTV Internet video Artistic reproduction Widescreen movie

EE565 Advanced Image Processing Copyright Xin Li 2008

17

Motivation

x

y

Along the edge orientation,We observe repeated pattern

(0,0)

(-1,2)

(-2,4)

(1,-2)

:

:.

.

pattern

Page 18: EE565 Advanced Image Processing Copyright Xin Li 20081 Motivating Applications HDTV Internet video Artistic reproduction Widescreen movie

EE565 Advanced Image Processing Copyright Xin Li 2008

18

Geometric Duality

same pattern

downsampling

Page 19: EE565 Advanced Image Processing Copyright Xin Li 20081 Motivating Applications HDTV Internet video Artistic reproduction Widescreen movie

EE565 Advanced Image Processing Copyright Xin Li 2008

19

Bridge across the resolution

High-resolution

Low-resolution

2i

2j

2i+2

2i-2

2j-2 2j+2

Cov(X2i,2j,X2i+k,2j+l)≈Cov(X2i,2j,X2i+2k,2j+2l)

(k,l)={(0,1),(1,1),(1,0),(1,-1),(0,-1),(-1,-1),(-1,0),(-1,1)}

Page 20: EE565 Advanced Image Processing Copyright Xin Li 20081 Motivating Applications HDTV Internet video Artistic reproduction Widescreen movie

EE565 Advanced Image Processing Copyright Xin Li 2008

20

Least-Square (LS) Method

nnnnnnnnnnnn YAXXAY 1

Solve over-determined system

Solve square linear system

)()( 111 YAAAXXAY TT

nnmm

Page 21: EE565 Advanced Image Processing Copyright Xin Li 20081 Motivating Applications HDTV Internet video Artistic reproduction Widescreen movie

EE565 Advanced Image Processing Copyright Xin Li 2008

21

LS-based estimation

X1X2X3

X4

X5 X6 X7

X8

X

8

1iii XaX

For all pixels in 7x7 window,we can write an equation likeabove, which renders anover-determined systemwith 49 equations and 8 unknown variables

Use LS method to solve

Page 22: EE565 Advanced Image Processing Copyright Xin Li 20081 Motivating Applications HDTV Internet video Artistic reproduction Widescreen movie

EE565 Advanced Image Processing Copyright Xin Li 2008

22

Step 1: Interpolate diagonal pixels

0

1

2 3

40

1 4

2 3

-Formulate LS estimationproblem with pixels atlow resolution and solve{a1,a2,a3,a4}

-Use {a1,a2,a3,a4} tointerpolate the pixel0 at the high resolution

Implementation:

Page 23: EE565 Advanced Image Processing Copyright Xin Li 20081 Motivating Applications HDTV Internet video Artistic reproduction Widescreen movie

EE565 Advanced Image Processing Copyright Xin Li 2008

23

Step 2: Interpolate the Other Half

0

1

2

3

4

0

1

2

3

4

-Formulate LS estimationproblem with pixels atlow resolution and solve{a1,a2,a3,a4}

-Use {a1,a2,a3,a4} tointerpolate the pixel0 at the high resolution

Implementation:

Page 24: EE565 Advanced Image Processing Copyright Xin Li 20081 Motivating Applications HDTV Internet video Artistic reproduction Widescreen movie

EE565 Advanced Image Processing Copyright Xin Li 2008

24

Experiment Result

bilinear Edge directed interpolation

Page 25: EE565 Advanced Image Processing Copyright Xin Li 20081 Motivating Applications HDTV Internet video Artistic reproduction Widescreen movie

EE565 Advanced Image Processing Copyright Xin Li 2008

25

After Thoughts

Pro Improve visual quality dramatically

Con Computationally expensive

Further optimization Translation invariant derivation of

interpolation coefficients a’s