4/99super-resolution2 super-resolution 4/99super-resolution3 out line
Post on 22-Dec-2015
224 views
TRANSCRIPT
![Page 1: 4/99Super-Resolution2 Super-resolution 4/99Super-Resolution3 Out Line](https://reader031.vdocument.in/reader031/viewer/2022031909/56649d795503460f94a5c55e/html5/thumbnails/1.jpg)
TECHNIONIsrael Institute of Technology
Department of Electrical EngineeringThe Vision Research and Image Science Laboratory
By
Elad Aharoni Guy Abramovich
Supervised by
Dr. Michael Elad
Summer 1998 - Winter 1999
Supported by the Ollendorff Research Center Fund
![Page 2: 4/99Super-Resolution2 Super-resolution 4/99Super-Resolution3 Out Line](https://reader031.vdocument.in/reader031/viewer/2022031909/56649d795503460f94a5c55e/html5/thumbnails/2.jpg)
4/99 Super-Resolution 2
Super-resolution
Given a collection of measured images of thesame scene, it is possible to fuse them to a onehigher resolution image, called the "SuperResolution Image"
Definition
Project Goal
1. To reconstruct a super-resolution image, whilepreserving its edges
2. Develop fast algorithm for the pure translationmotion case
![Page 3: 4/99Super-Resolution2 Super-resolution 4/99Super-Resolution3 Out Line](https://reader031.vdocument.in/reader031/viewer/2022031909/56649d795503460f94a5c55e/html5/thumbnails/3.jpg)
4/99 Super-Resolution 3
Out Line
Part One
1. Simulate the creation of measured images
2. The super-resolution as a quadratic problem
3. The basic Steepest-Descent algorithm
4. Improvements of the SD algorithm
5. The Normalized-Steepest-Descent algorithm
6. The Conjugate-Gradient algorithm
7. Regularization
8. Simulations and Results
Part Two
1. Motion estimation algorithm
2. Getting input images
3. Estimating the super-resolution image
![Page 4: 4/99Super-Resolution2 Super-resolution 4/99Super-Resolution3 Out Line](https://reader031.vdocument.in/reader031/viewer/2022031909/56649d795503460f94a5c55e/html5/thumbnails/4.jpg)
4/99 Super-Resolution 4
Simulate Creation Process of
Target Images
Warp1F
1
Blur1
1H
Decimate1D
1
noise 1
1 measuredlow resolutionimage
st
1Y
Warp NF
N
Blur N
NH
Decimate ND
N
noise N
N measuredlow resolutionimage
th
NY
sourcehigh - resolutionimage X
Creation of Yk images
Images Yk1k
Nmr supposed to be taken by CCD
camera
We assume Yk kNmr1are different representations of a
single high resolution image X of size LL
![Page 5: 4/99Super-Resolution2 Super-resolution 4/99Super-Resolution3 Out Line](https://reader031.vdocument.in/reader031/viewer/2022031909/56649d795503460f94a5c55e/html5/thumbnails/5.jpg)
4/99 Super-Resolution 5
Simulate Creation Process of Target Images (Cont.)
M o d e l i n g o f c r e a t i o n p r o c e s s : Yk
Dk
Hk
Fk
X Ek
Fk
L L 2 2 g e o m e t r i c w a r p i n g
Hk
L L 2 2 l i n e a r - s p a c e i n v a r i a n t b l u r
Dk
M L 2 2 d e c i m a t i o n o p e r a t o r
Ek
M 2 1 a d d i t i v e z e r o m e a n G a u s s i a n n o i s e
Yk
M 2 1 v e c t o r , C S r e p r e s e n t a t i o n o f t h e m e a s u r e d
i m a g e s
X L 2 1 v e c t o r , C S r e p r e s e n t a t i o n o f t h e
s u p e r - r e s o l u t i o n i m a g e
All matrices assumed to be known in advance
![Page 6: 4/99Super-Resolution2 Super-resolution 4/99Super-Resolution3 Out Line](https://reader031.vdocument.in/reader031/viewer/2022031909/56649d795503460f94a5c55e/html5/thumbnails/6.jpg)
4/99 Super-Resolution 6
Simulate Creation Process of Target Images (Cont.)
Assumptions: Hk = H
Dk = D
Fk – Pure displacement.
G r o u p i n g N e q u a t i o n s i n t o o n eY
YN
D H F
DN
HN
FN
X
E
E N
C
CN
X E
1 1 1 1 1 1
L
N
MMMMM
O
Q
PPPPP
L
N
MMMMM
O
Q
PPPPP
L
N
MMMMM
O
Q
PPPPP
L
N
MMMMM
O
Q
PPPPP
w h e r e C D H Fk k k k
Final result YCXE
![Page 7: 4/99Super-Resolution2 Super-resolution 4/99Super-Resolution3 Out Line](https://reader031.vdocument.in/reader031/viewer/2022031909/56649d795503460f94a5c55e/html5/thumbnails/7.jpg)
4/99 Super-Resolution 7
Creation of Yk Images from Source
Image
Source Image Yk Images
![Page 8: 4/99Super-Resolution2 Super-resolution 4/99Super-Resolution3 Out Line](https://reader031.vdocument.in/reader031/viewer/2022031909/56649d795503460f94a5c55e/html5/thumbnails/8.jpg)
4/99 Super-Resolution 8
The Super-resolution Problem as Quadratic Problem
YCXE Classic model of restoration problem
Given a solution X the Error function:
E Y C Xrr
2 2
D if f e r e n t i a t i n g w i th r e s p e c t t o X a n d e q u a t i n g t o z e r o :
2 0 C Y C XT a fC Y C C XT T c h
t h e n :
X C C C YT T c h1
P C YT - ו R C CT
R X P
Well known classic pseudoinverse result XRP1
Practically impossible to invert
Requires alternative solutions – iterative algorithms
R - Size of L L2 2 2 2400 400 c h
![Page 9: 4/99Super-Resolution2 Super-resolution 4/99Super-Resolution3 Out Line](https://reader031.vdocument.in/reader031/viewer/2022031909/56649d795503460f94a5c55e/html5/thumbnails/9.jpg)
4/99 Super-Resolution 9
Steepest-Descent Algorithm
Described by the following equations:
X X R X Pj j 1
R F H D D H F
P F H D Y
kT
k
NT T
k
kT T
k
NT
k
1
1
F1 H D DT HT FT1
XK
H D DT HTFN FNT
Y1
XK1
XK
YN
Xk
Xk
FjT
j
NHT DT D H F
jX
kY
j IKFH1
1
The basic Steepest-Descent algorithm
Error function (quality factor):
2 2
1
Y DHF Xk k
k
N
Relatively slow (H implemented as convolution)
constant chosen manually: =0.1 – slowconvergence
=0.5 – fast convergence
![Page 10: 4/99Super-Resolution2 Super-resolution 4/99Super-Resolution3 Out Line](https://reader031.vdocument.in/reader031/viewer/2022031909/56649d795503460f94a5c55e/html5/thumbnails/10.jpg)
4/99 Super-Resolution 10
Restoration Results ofSteepest-Descent Algorithm
![Page 11: 4/99Super-Resolution2 Super-resolution 4/99Super-Resolution3 Out Line](https://reader031.vdocument.in/reader031/viewer/2022031909/56649d795503460f94a5c55e/html5/thumbnails/11.jpg)
4/99 Super-Resolution 11
Enhanced Steepest-Descent Algorithm
Basic algorithm is very slow
Blurring operator is equal for all images
F i r s t e n h a n c e m e n t : F k a n d H a r e c o m m u t a t i v e
T h e n :
2 2
1 1
1
LNM OQP LNM OQP
Y D H F X R H F D D F H
P H F D Y
k kk
NT
kT T
kk
N
Tk
T
k
NT
k
A n d t h e n e w e q u a t i o n :
X X H F D D F H X Yk kT
kT T
j
N
j k j
11
a f
![Page 12: 4/99Super-Resolution2 Super-resolution 4/99Super-Resolution3 Out Line](https://reader031.vdocument.in/reader031/viewer/2022031909/56649d795503460f94a5c55e/html5/thumbnails/12.jpg)
4/99 Super-Resolution 12
Enhanced Steepest-Descent Algorithm (Cont.)
Blurring operator performed twice instead of 2N in
every iteration
Save of 30% in run-time while retaining result
F1 D DT
HT
FT1
XK
D DTFN FNT
Y1
XK1
XK
YN
H
Commutative warping and blurring operators
![Page 13: 4/99Super-Resolution2 Super-resolution 4/99Super-Resolution3 Out Line](https://reader031.vdocument.in/reader031/viewer/2022031909/56649d795503460f94a5c55e/html5/thumbnails/13.jpg)
4/99 Super-Resolution 13
Enhanced Steepest-Descent Algorithm (Cont.)
S e c o n d e n h a n c e m e n t :
R H F D D F H H R HTk
T Tk
k
NT LNM OQP
10
R F D D FkT T
kk
N
01
P H F D Y H PTk
T Tk
k
NT LNM OQP
10
P F D YkT T
kk
N
01
T h e n e w e q u a t i o n : X X H R H X Pj jT
j 1 0 0
Matrices P0 and R0 calculated once only for the
whole algorithm
If warping is based on Nearest-Neighbor then:
R0 is diagonal
R0 implemented as mask
![Page 14: 4/99Super-Resolution2 Super-resolution 4/99Super-Resolution3 Out Line](https://reader031.vdocument.in/reader031/viewer/2022031909/56649d795503460f94a5c55e/html5/thumbnails/14.jpg)
4/99 Super-Resolution 14
Enhanced Steepest-Descent Algorithm (Cont.)
XK H R0
P0
HT XK1
The enhanced Steepest-Descent algorithm
Save of 80% in run-time while retaining result,compared to the original SD algorithm
![Page 15: 4/99Super-Resolution2 Super-resolution 4/99Super-Resolution3 Out Line](https://reader031.vdocument.in/reader031/viewer/2022031909/56649d795503460f94a5c55e/html5/thumbnails/15.jpg)
4/99 Super-Resolution 15
The Normalized SD algorithm
SD algorithm - constant chosen by user
Normalized - SD: calculates optimal for every
iteration
X X Ej j j j 1
E R X Pj j
and jjT
j
jT
j
E E
E R E
1
Calculation of optimal adds 20% to run-time
![Page 16: 4/99Super-Resolution2 Super-resolution 4/99Super-Resolution3 Out Line](https://reader031.vdocument.in/reader031/viewer/2022031909/56649d795503460f94a5c55e/html5/thumbnails/16.jpg)
4/99 Super-Resolution 16
Restoration Results of NSD Algorithm
![Page 17: 4/99Super-Resolution2 Super-resolution 4/99Super-Resolution3 Out Line](https://reader031.vdocument.in/reader031/viewer/2022031909/56649d795503460f94a5c55e/html5/thumbnails/17.jpg)
4/99 Super-Resolution 17
Conjugate Gradient Algorithm
Complicated calculations
Comparison to NSD algorithm:
On synthetic square problem
On our current simulation
T h e C G a l g o r i t h m : G i v e n R , P , X k a n d V k - 1
1a f E R X Pk k
2 11
1 1
a f kkT
k
kT
k
E R VV R V
3 1 1a f V E Vk k k k
4a f kkT
k
kT
k
E VV R V
T h e f i n a l e q u a t i o n : X X Vk k k k 1
![Page 18: 4/99Super-Resolution2 Super-resolution 4/99Super-Resolution3 Out Line](https://reader031.vdocument.in/reader031/viewer/2022031909/56649d795503460f94a5c55e/html5/thumbnails/18.jpg)
4/99 Super-Resolution 18
Conjugate Gradient Algorithm (Cont.)
E KK
K 1 V K
1
MUL
2 3
4
K KV XK1
XK
P
R
-
+
VK 1
The Conjugate-Gradient algorithm
CG iteration run-time is 50% longer than in the NSD
![Page 19: 4/99Super-Resolution2 Super-resolution 4/99Super-Resolution3 Out Line](https://reader031.vdocument.in/reader031/viewer/2022031909/56649d795503460f94a5c55e/html5/thumbnails/19.jpg)
4/99 Super-Resolution 19
Comparison of CG vs. NSD
Comparison parameters: Error function 2 xafNorma X X true 2
Both algorithms converge to same result
The norma factor getting worse because of ringing effect
Xtrue calculation is explained later
![Page 20: 4/99Super-Resolution2 Super-resolution 4/99Super-Resolution3 Out Line](https://reader031.vdocument.in/reader031/viewer/2022031909/56649d795503460f94a5c55e/html5/thumbnails/20.jpg)
4/99 Super-Resolution 20
Comparison of CG vs. NSD (Cont.)
Concentrating on the first few iterations
CG vs NSD in the first few interations
![Page 21: 4/99Super-Resolution2 Super-resolution 4/99Super-Resolution3 Out Line](https://reader031.vdocument.in/reader031/viewer/2022031909/56649d795503460f94a5c55e/html5/thumbnails/21.jpg)
4/99 Super-Resolution 21
Comparison of CG vs. NSD (Cont.)
CG and NSD after 7 iterations
CG converges faster - damaged by stronger ringing effect
![Page 22: 4/99Super-Resolution2 Super-resolution 4/99Super-Resolution3 Out Line](https://reader031.vdocument.in/reader031/viewer/2022031909/56649d795503460f94a5c55e/html5/thumbnails/22.jpg)
4/99 Super-Resolution 22
Regularization
Ringing effect damages images as we converge
Possible reasons: True R matrix is not inversible
Error function doesn't reflect realminimization
We're interested in - min X Xrestore true 2
Regularization component in the error function
2 2
1
2
Y D H F X D Xk k
k
N ~
Differentiating with respect to X and equating to zeroyields:
R F H D D H F D D
P F H D Y
kT T T
kk
NT
kT T
kT
k
N
1
1
~ ~
~~DDT- Laplasian operator
- determines the regularization scale. Constant chosen
manually
![Page 23: 4/99Super-Resolution2 Super-resolution 4/99Super-Resolution3 Out Line](https://reader031.vdocument.in/reader031/viewer/2022031909/56649d795503460f94a5c55e/html5/thumbnails/23.jpg)
4/99 Super-Resolution 23
Regular Regularization
XK H R0
P0
HT XK1
~ ~D DXT
K
Regular regularization implementation in NSD
What is the optimal ?
Run for various values of :
Convergence factor for various values of
Optimal result for =0.6
![Page 24: 4/99Super-Resolution2 Super-resolution 4/99Super-Resolution3 Out Line](https://reader031.vdocument.in/reader031/viewer/2022031909/56649d795503460f94a5c55e/html5/thumbnails/24.jpg)
4/99 Super-Resolution 24
Regularization Results for Various Values of
![Page 25: 4/99Super-Resolution2 Super-resolution 4/99Super-Resolution3 Out Line](https://reader031.vdocument.in/reader031/viewer/2022031909/56649d795503460f94a5c55e/html5/thumbnails/25.jpg)
4/99 Super-Resolution 25
Comparison Between CG and NSD Including Regularization
Conclusion: CG and NSD converge to same resultCG converges faster in the first fewiterations
![Page 26: 4/99Super-Resolution2 Super-resolution 4/99Super-Resolution3 Out Line](https://reader031.vdocument.in/reader031/viewer/2022031909/56649d795503460f94a5c55e/html5/thumbnails/26.jpg)
4/99 Super-Resolution 26
Adaptive Regularization
Is it possible to implement regularization while
preserving edges?
W e ig h te d re g u la riza tio n c o m p o n e n ts in th e e rro rfu n c tio n :
2 2
1
2
Y C X D Xk k
k
N
W
~
D iffe re n tia tin g w ith re sp ec t to X an d e q u a tin g to z e roy ie ld s :
R F H D D H F D W D
P F H D Y
kT T T
kk
NT
kT T
kT
k
N
1
1
~ ~
Implementation of adaptive regularization in NSDalgorithm:
XK H R0
P0
HT XK1
~ ~D WDXT
K
![Page 27: 4/99Super-Resolution2 Super-resolution 4/99Super-Resolution3 Out Line](https://reader031.vdocument.in/reader031/viewer/2022031909/56649d795503460f94a5c55e/html5/thumbnails/27.jpg)
4/99 Super-Resolution 27
W Matrix
Prevents edges from being regularized
Detects edges by calculating derivatives
Built according to derivatives values
Implemented as mask
a
W
b ערךרת גז הנ
1
a
W
ערךרת גז הנ
1
a
W
b ערךרת גז הנ
1
Different threshold functions
Image of W mask
![Page 28: 4/99Super-Resolution2 Super-resolution 4/99Super-Resolution3 Out Line](https://reader031.vdocument.in/reader031/viewer/2022031909/56649d795503460f94a5c55e/html5/thumbnails/28.jpg)
4/99 Super-Resolution 28
W Matrix (Cont.)
Low threshold High threshold
Adaptive regularization Regular
![Page 29: 4/99Super-Resolution2 Super-resolution 4/99Super-Resolution3 Out Line](https://reader031.vdocument.in/reader031/viewer/2022031909/56649d795503460f94a5c55e/html5/thumbnails/29.jpg)
4/99 Super-Resolution 29
Motion Estimation
Relative translation is not known
M o tio nE stim atio n ן בי ת סי ח י זה ו תז
ת ונו מ ת שתי ה
I (x, y)1
I (x, y)2
(dx, dy)
Algorithm based on Taylor series
Accurate for small warping (0-2 pixels)
We deal with large warping (0-10 pixels)
Implement iterative algorithm
![Page 30: 4/99Super-Resolution2 Super-resolution 4/99Super-Resolution3 Out Line](https://reader031.vdocument.in/reader031/viewer/2022031909/56649d795503460f94a5c55e/html5/thumbnails/30.jpg)
4/99 Super-Resolution 30
Motion Estimation Implementation as Iterative Algorithm
M o ti o nE stim ati o n
I (x, y)1 I (x, y)2
I nv erseW arp ing
M o ti o nE stim ati o n
I nv erseW arp ing
M o ti o nE stim ati o n
Tota lW arp ing(dx , dy)
(dx1 , dy1)
(dx2 , dy2)
(dxn , dyn)
![Page 31: 4/99Super-Resolution2 Super-resolution 4/99Super-Resolution3 Out Line](https://reader031.vdocument.in/reader031/viewer/2022031909/56649d795503460f94a5c55e/html5/thumbnails/31.jpg)
4/99 Super-Resolution 31
Input Images
Take 60 images from video sequence
Create difference images
Choose best 15 with minimal rotation and scaling
Difference images including rotation and scaling
Difference images including minimal rotation and scaling
![Page 32: 4/99Super-Resolution2 Super-resolution 4/99Super-Resolution3 Out Line](https://reader031.vdocument.in/reader031/viewer/2022031909/56649d795503460f94a5c55e/html5/thumbnails/32.jpg)
4/99 Super-Resolution 32
Final Results
Implemented in NSD algorithm
Specific blurring kernel of CCD camera -
Regularization parameter -
Optimal result for =1.8
![Page 33: 4/99Super-Resolution2 Super-resolution 4/99Super-Resolution3 Out Line](https://reader031.vdocument.in/reader031/viewer/2022031909/56649d795503460f94a5c55e/html5/thumbnails/33.jpg)
4/99 Super-Resolution 33
Final Results (Cont.)
Optimal Result for =1.8 and =2
![Page 34: 4/99Super-Resolution2 Super-resolution 4/99Super-Resolution3 Out Line](https://reader031.vdocument.in/reader031/viewer/2022031909/56649d795503460f94a5c55e/html5/thumbnails/34.jpg)
4/99 Super-Resolution 34
Final Results (Cont.)
Zoom on the final results
Good obtained results
Insufficient due to technical problems:
Strong anti-aliasing effect
Rotation and scaling
![Page 35: 4/99Super-Resolution2 Super-resolution 4/99Super-Resolution3 Out Line](https://reader031.vdocument.in/reader031/viewer/2022031909/56649d795503460f94a5c55e/html5/thumbnails/35.jpg)
4/99 Super-Resolution 35
Aliasing Effect
The Original Image Sampled Image The Restored Image
The Original Image Sampled Image The Restored Image
The original images decimated in order to "add"aliasing effect
Very strong anti-aliasing effect caused by the camera
![Page 36: 4/99Super-Resolution2 Super-resolution 4/99Super-Resolution3 Out Line](https://reader031.vdocument.in/reader031/viewer/2022031909/56649d795503460f94a5c55e/html5/thumbnails/36.jpg)
4/99 Super-Resolution 36
Aliasing Effect (Cont.)
No resolution improvement in the result image
compared to the original
Quality improvement in the result image compared to
the original
The Original Image Sampled Image The Restored Image
![Page 37: 4/99Super-Resolution2 Super-resolution 4/99Super-Resolution3 Out Line](https://reader031.vdocument.in/reader031/viewer/2022031909/56649d795503460f94a5c55e/html5/thumbnails/37.jpg)
4/99 Super-Resolution 37
Summing
We developed a fast version algorithm forSuper-Resolution Reconstruction, assuming puredisplacement between the images
Simulations on synthetic data verified that the takenshortcuts are valid and produce 80% reduction nrun-time, while preserving output quality
Results on true video images show slightimprovement, due to too strong anti-aliasing effect