ee565 advanced image processing copyright xin li 20081 motivating applications hdtv internet video...
TRANSCRIPT
EE565 Advanced Image Processing Copyright Xin Li 2008
1
Motivating Applications
HDTVInternet video
Artistic reproduction Widescreen movie
EE565 Advanced Image Processing Copyright Xin Li 2008
2
Image Interpolation
Image Interpolation Importance of geometry Edge directed interpolation PDE-based techniques
Applications Super-resolution Inpainting (error concealment) Compressive sensing
EE565 Advanced Image Processing Copyright Xin Li 2008
3
Importance of Geometry
It is geometry that distinguishes image signals (2D) from audio and speech (1D)
2D image is the projection of 3D geometry (a seemingly-trivial statement)
Geometric elements and properties: edge (location and orientation), object (convexity) … …
EE565 Advanced Image Processing Copyright Xin Li 2008
4
Geometric Constraint of Edges
3D visualizationsingle-edge image
• Along the edge orientation, intensity field is homogeneous
• Across the edge orientation, intensity field evolves fast
Observations:
EE565 Advanced Image Processing Copyright Xin Li 2008
5
Why geometry is difficult?
Geometry is embedded in the array of image pixels, which is not straightforward to be exploited by any linear operations (filtering or transform)
Image signals form a manifold in an extremely high-dimensional space (What is a manifold? Think of a string in the real world, it is a 3D object and a 1D manifold)
EE565 Advanced Image Processing Copyright Xin Li 2008
6
Interpolation Problem
downsampling
originalhigh-resolution
image X
low-resolution image Y
interpolatedhigh-resolution
image Z
How to make Z as close to X as possible?
EE565 Advanced Image Processing Copyright Xin Li 2008
7
bilinear Interpolation Review
(e+w)/2
(n+s)/2
(ne+nw+se+sw)/4
EE565 Advanced Image Processing Copyright Xin Li 2008
8
Why bilinear is bad?
Edge blurring Jagged artifacts
X Z
Jagged artifacts
X
Z
Edge blurring
EE565 Advanced Image Processing Copyright Xin Li 2008
9
Where does image quality degradation come from?
Violation of geometric constraint Keep in mind an edge could be
arbitrarily oriented
Edge blurring Jagged artifacts
Along the edge orientation,intensity field is inhomogeneous
Across the edge orientation, intensity field evolves slower
EE565 Advanced Image Processing Copyright Xin Li 2008
10
Ideas to Do Better
Recognize the importance of edges Location: where are they? Orientation: how are they oriented?
Identify appropriate mathematical tools for exploiting geometric constraints Heuristics Statistical vs. PDE
EE565 Advanced Image Processing Copyright Xin Li 2008
11
Heuristics: Edge Location
Intuitively, edges are characterized by fast intensity value variation (i.e., large gradient)
To avoid edge blurring, we need to spatially adaptively adjust interpolation coefficients
Question: how to incorporate gradient cue into a spatiallyadaptive interpolation scheme?
EE565 Advanced Image Processing Copyright Xin Li 2008
12
Desired Improvement
X Y Zbil Zesi
horizontaledge
verticaledge
EE565 Advanced Image Processing Copyright Xin Li 2008
13
Here Comes the Trick
Step 1: interpolate the missing pixels along the diagonal
black or white?
Step 2: interpolate the other half missing pixels
a b
c d
Since |a-c|=|b-d|x
x has equal probabilityof being black or white
a
b
c
dSince |a-c|>|b-d|
x=(b+d)/2=black
x
EE565 Advanced Image Processing Copyright Xin Li 2008
14
Edge Sensitive Interpolation (ESI)
b
c
d x
a
If |a-c|>|b-d|
x=(b+d)/2
If |a-c|<|b-d|
x=(a+c)/2
If |a-c|=|b-d|
x=min or max
a b
c d
x
In simple words, we want to avoid interpolate along thedirection which has a large gradient
EE565 Advanced Image Processing Copyright Xin Li 2008
15
Experiment Result
Bilinear interpolation Edge sensitive interpolation
EE565 Advanced Image Processing Copyright Xin Li 2008
16
Heuristics: Edge Orientation
Can we do better? Yes! Gradient is only a first-order
characteristics of edge location ESI makes binary decision with two
orthogonal directions How to do better?
We need some mathematical tool that can work with arbitrary edge orientation
EE565 Advanced Image Processing Copyright Xin Li 2008
17
Motivation
x
y
Along the edge orientation,We observe repeated pattern
(0,0)
(-1,2)
(-2,4)
(1,-2)
:
:.
.
pattern
EE565 Advanced Image Processing Copyright Xin Li 2008
18
Geometric Duality
same pattern
downsampling
EE565 Advanced Image Processing Copyright Xin Li 2008
19
Bridge across the resolution
High-resolution
Low-resolution
2i
2j
2i+2
2i-2
2j-2 2j+2
Cov(X2i,2j,X2i+k,2j+l)≈Cov(X2i,2j,X2i+2k,2j+2l)
(k,l)={(0,1),(1,1),(1,0),(1,-1),(0,-1),(-1,-1),(-1,0),(-1,1)}
EE565 Advanced Image Processing Copyright Xin Li 2008
20
Least-Square (LS) Method
nnnnnnnnnnnn YAXXAY 1
Solve over-determined system
Solve square linear system
)()( 111 YAAAXXAY TT
nnmm
EE565 Advanced Image Processing Copyright Xin Li 2008
21
LS-based estimation
X1X2X3
X4
X5 X6 X7
X8
X
8
1iii XaX
For all pixels in 7x7 window,we can write an equation likeabove, which renders anover-determined systemwith 49 equations and 8 unknown variables
Use LS method to solve
EE565 Advanced Image Processing Copyright Xin Li 2008
22
Step 1: Interpolate diagonal pixels
0
1
2 3
40
1 4
2 3
-Formulate LS estimationproblem with pixels atlow resolution and solve{a1,a2,a3,a4}
-Use {a1,a2,a3,a4} tointerpolate the pixel0 at the high resolution
Implementation:
EE565 Advanced Image Processing Copyright Xin Li 2008
23
Step 2: Interpolate the Other Half
0
1
2
3
4
0
1
2
3
4
-Formulate LS estimationproblem with pixels atlow resolution and solve{a1,a2,a3,a4}
-Use {a1,a2,a3,a4} tointerpolate the pixel0 at the high resolution
Implementation:
EE565 Advanced Image Processing Copyright Xin Li 2008
24
Experiment Result
bilinear Edge directed interpolation
EE565 Advanced Image Processing Copyright Xin Li 2008
25
After Thoughts
Pro Improve visual quality dramatically
Con Computationally expensive
Further optimization Translation invariant derivation of
interpolation coefficients a’s