frequency domain methods for demosaicking of bayer sampled color images eric dubois
Post on 12-Jan-2016
219 Views
Preview:
TRANSCRIPT
Frequency domain methods for demosaicking of Bayer sampled color
images Eric Dubois
Frequency-domain Bayer demosaicking 2
Problem Statement
• Problem: Most digital color cameras capture only one color component at each spatial location. The remaining components must be reconstructed by interpolation from the captured samples. Cameras provide hardware or software to do this, but the quality may be inadequate.
• Objective: Develop new algorithms to interpolate each color plane (called demosaicking) with better quality reconstruction, and with minimal computational complexity.
Frequency-domain Bayer demosaicking 3
Retinal Cone Mosaic
The human visual system must solve a similar problem!
Frequency-domain Bayer demosaicking 4
Construction of color image from color planes
+
Lighthouseoriginal
Lighthousered original
Lighthousegreen original
Lighthouseblue original
Frequency-domain Bayer demosaicking 9
Formation of Color planes
Lighthousered subsampled
Lighthousegreen subsampled
Lighthouseblue subsampled
LighthouseBayer CFA image
Frequency-domain Bayer demosaicking 14
Color plane interpolation
GA
GB
GL GR
)(4
1ABRLI GGGGG
GI
Green channel: bilinear interpolation
Frequency-domain Bayer demosaicking 15
Color plane interpolation
)(4
1SESWNENWC RRRRR
RC
Red channel: bilinear interpolation
RNWRNE
RSWRSE RS
SESWS RRR 2
1
Lighthousered interpolated
Lighthousegreen interpolated
Lighthouseblue interpolated
LighthouseInterpolated color image
Lighthouseoriginal
Frequency-domain Bayer demosaicking 21
Can we do better?
• Color planes have severe aliasing. Better interpolation of the individual planes has little effect.
Lighthousered interpolatedwith bilinear interpolator
Lighthousered interpolatedwith bicubic interpolator
Frequency-domain Bayer demosaicking 24
Can we do better?
• Color planes have severe aliasing. Better interpolation of the individual planes has little effect.
• We could optically prefilter the image (blur it) so that aliasing is less severe.
Lighthousered interpolatedwith bilinear interpolator
Lighthouseprefiltered red interpolatedwith bilinear interpolator
LighthouseInterpolated color image
Lighthouse Prefiltered & Interpolated color image
Lighthouse original
Frequency-domain Bayer demosaicking 30
Can we do better?
• Color planes have severe aliasing. Better interpolation of the individual planes has little effect.
• We could optically prefilter the image (blur it) so that aliasing is less severe.
• We can process the three color planes together to gather details from all three components.
Frequency-domain Bayer demosaicking 31
Can we do better?• There have been numerous papers and patents
describing different algorithms to interpolate the color planes – they all work on the three planes together, exploiting the correlation between the three components.
• Gunturk et al. published an extensive survey in March 2005. The best methods were the projection on convex sets (POCS) algorithm (lowest MSE) and the adaptive homogeneity directed (AHD) algorithm (best subjective quality).
• We present here a novel frequency-domain algorithm.
Frequency-domain Bayer demosaicking 32
Spatial multiplexing model
subsampling multiplexing
Frequency-domain Bayer demosaicking 33
Spatial multiplexing model
))1(1)()1(1](,[4
1
))1(1)()1(1](,[4
1))1(1](,[
2
1],[
2121
2121
212121CFA
nn
B
nn
R
nn
G
nnf
nnfnnfnnf
Frequency-domain Bayer demosaicking 34
Frequency-domain multiplexing model
212121
21212121
212121
2121
2121
212121CFA
)1()1(],[4
1],[
4
1
)1(],[4
1],[
2
1],[
4
1
],[4
1],[
2
1],[
4
1
))1(1)()1(1](,[4
1
))1(1)()1(1](,[4
1))1(1](,[
2
1],[
nn
BR
nn
BGR
BGR
nn
B
nn
R
nn
G
nnfnnf
nnfnnfnnf
nnfnnfnnf
nnf
nnfnnfnnf
Re-arranging the spatial multiplexing expression
Frequency-domain Bayer demosaicking 35
Frequency-domain multiplexing model
)2/2exp()2/2exp(],[
)2/)(2exp(],[],[
)1()1(],[
)1](,[],[],[
21212
2121121
21212
212112121CFA
njnjnnf
nnjnnfnnf
nnf
nnfnnfnnf
C
CL
nn
C
nn
CL
)5.0,(),5.0()5.0,5.0(),(),( 221CFA vuFvuFvuFvuFvuF CCCL
David Alleysson, EPFL
Frequency-domain Bayer demosaicking 36
Luma and chrominance components
2
1
41
41
41
21
41
41
21
41
2
1
211
011
211
0
C
C
L
B
G
R
B
G
R
C
C
L
f
f
f
f
f
f
f
f
f
f
f
f
Frequency-domain Bayer demosaicking 37
Luma and chrominance components
Luma fL Chroma_1 fC1 Chroma_2 fC2
Lighthouse BilinearlyInterpolated color image
Frequency-domain Bayer demosaicking 39
Frequency-domain demosaicking algorithm
1. Extract modulated C1 using a band-pass filter at (0.5,0.5) and demodulate to baseband
2. Extract modulated C2 using band-pass filters at (0.5,0.0) and (0.0, 0.5), demodulate to baseband, and combine in some suitable fashion (the key)
3. Subtract modulated C1 and remodulated C2 from the CFA to get the estimated luma component L.
4. Matrix the L, C1 and C2 components to get the RGB representation.
Frequency-domain Bayer demosaicking 40
Spectrum of CFA signal
ab
Using C2a only Using C2b only
Frequency-domain Bayer demosaicking 42
Original From C2a only From C2b only
Demosaicking using C2a only or C2b only -- details
Frequency-domain Bayer demosaicking 43
Demosaicking Block Diagram
h2a
h2b
+ -
fCFA
(-1)n1+n2
-(-1)n2
(-1)n1
h1
combine
(-1)n1-(-1)n2
+
-
matrix
fR
fG
fB
fC2am
fC2bm
fC1m fC1
fC2a
fC2b
fC2
fL
fC1
fC2
21212
212112121CFA )1()1(],[)1](,[],[],[
nn
C
nn
CL nnfnnfnnfnnf
Frequency-domain Bayer demosaicking 44
Spectrum of CFA signal
ab
Frequency-domain Bayer demosaicking 45
Design Issues
• How to choose the filters h1, h2a and h2b
– Frequency domain design methods
– Least-squares design methods
– Size of the filters
• How to combine the two estimates and – Choice of features to guide weighting
• The two above issues may be inter-related.
aCf 2ˆ
bCf 2ˆ
Frequency-domain Bayer demosaicking 46
Filter design
• Gaussian filters (Alleysson)• Window design or minimax design
– Define ideal response, with pass, stop and transition bands
– Approximate using the window design method
– Refine using minimax or least pth optimization
– Can design low-pass filters and modulate to the center frequency
Frequency-domain Bayer demosaicking 47
Filter specification u v val0.000 0.00 1.00.110 0.00 1.00.110 0.02 1.00.000 0.10 1.00.030 0.10 1.00.070 0.06 1.00.338 0.00 0.00.338 0.05 0.00.050 0.36 0.00.000 0.36 0.00.184 .205 0.00.500 0.00 0.00.000 0.50 0.00.500 0.50 0.0
0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
u
v
filter specification
1
0
Frequency-domain Bayer demosaicking 48
-0.5-0.4
-0.3-0.2
-0.10
0.10.2
0.30.4
0.5
-0.5
0
0.50
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Horizontal frequency
Perspective plot of ideal filter response
Vertical frequency
Mag
nitu
de o
f fil
ter
resp
onse
Ideal response – perspective view
Frequency-domain Bayer demosaicking 49
Ideal response – contour plot
Contour plot of ideal filter response
Horizontal frequency
Vert
ical fr
equency
-0.5 -0.4 -0.3 -0.2 -0.1 0 0.1 0.2 0.3 0.4-0.5
-0.4
-0.3
-0.2
-0.1
0
0.1
0.2
0.3
0.4
Frequency-domain Bayer demosaicking 50
Window design – perspective view
-0.5-0.4
-0.3-0.2
-0.10
0.10.2
0.30.4
0.5
-0.5
0
0.5-0.2
0
0.2
0.4
0.6
0.8
1
1.2
Perspective plot of lowpass filter
Frequency-domain Bayer demosaicking 51
Window design – contour plot
Contour plot of lowpass filter
-0.5 -0.4 -0.3 -0.2 -0.1 0 0.1 0.2 0.3 0.4
-0.5
-0.4
-0.3
-0.2
-0.1
0
0.1
0.2
0.3
0.4
Frequency-domain Bayer demosaicking 52
Least pth filter – perspective view
-0.5-0.4
-0.3-0.2
-0.10
0.10.2
0.30.4
0.5
-0.5
0
0.5-0.2
0
0.2
0.4
0.6
0.8
1
1.2
Perspective plot of lowpass filter
Frequency-domain Bayer demosaicking 53
Least pth filter – contour plot
0
0
0
0.4
Contour plot of lowpass filter
-0.5 -0.4 -0.3 -0.2 -0.1 0 0.1 0.2 0.3 0.4-0.5
-0.4
-0.3
-0.2
-0.1
0
0.1
0.2
0.3
0.4
21 x 21 filters in SPL published algorithm
-1-0.5
00.5
1
-1
-0.5
0
0.5
1
0
0.2
0.4
0.6
0.8
1
Horizontal frequency (c/pixel)Vertical frequency (c/pixel)
Mag
nitu
de
-1-0.5
00.5
1
-1
-0.5
0
0.5
1
0
0.2
0.4
0.6
0.8
1
Horizontal frequency (c/pixel)Vertical frequency (c/pixel)
Mag
nitu
de
-1-0.5
00.5
1
-1
-0.5
0
0.5
1
0
0.2
0.4
0.6
0.8
1
Horizontal frequency (c/pixel)Vertical frequency (c/pixel)
Mag
nitu
de
u
v
h2a h2b
h1
Frequency-domain Bayer demosaicking 55
Adaptive weighting of C2a and C2b
• We want to form the estimate of C2 by choosing the best between C2a and C2b, or perhaps by a weighted average. We have used
• should be near 1 when C2a is the best choice, and near 0 when C2b is the best choice
],[ˆ]),[1(],[ˆ],[],[ˆ2122121221212 nnfnnwnnfnnwnnf bCaCC
],[ 21 nnw
Frequency-domain Bayer demosaicking 56
Typical scenarios for local spectrum
L C2aC2a
C2b
C2b
C1C1
C1 C1
B: C2b is better estimate
L C2aC2a
C2b
C2b
C1C1
C1 C1
A: C2a is better estimate
u
v v
u
Scenario A
Scenario B
Frequency-domain Bayer demosaicking 58
Typical scenarios for local spectrum
L C2aC2a
C2b
C2b
C1C1
C1 C1
B: C2b is better estimate
L C2aC2a
C2b
C2b
C1C1
C1 C1
A: C2a is better estimate
u
v v
u
Frequency-domain Bayer demosaicking 59
Weight selection strategy
• Scenario A: average local energy near (fm, 0) is smaller than near (0, fm ).
• Scenario B: average local energy near (0, fm ) is smaller than near (fm, 0).
• Let be a measure of the average local energy near (fm, 0), and be a measure of the average local energy near (0, fm ).
],[ 21 nneX
],[ 21 nneY
],[],[
],[],[
2121
2121 nnenne
nnennw
YX
Y
Frequency-domain Bayer demosaicking 60
Gaussian filters for local energy measurement
-1-0.5
00.5
1
-1
-0.5
0
0.5
1
0
0.2
0.4
0.6
0.8
1
Horizontal frequency (c/pixel)Vertical frequency (c/pixel)
Mag
nitu
de
-1-0.5
00.5
1
-1
-0.5
0
0.5
1
0
0.2
0.4
0.6
0.8
1
Horizontal frequency (c/pixel)Vertical frequency (c/pixel)M
agni
tude
u
vfm = 0.375
Frequency-domain Bayer demosaicking 61
Results
• Results with this adaptive frequency-domain demosaicking method were published in IEEE Signal Processing Letters in Dec. 2005. All filters were of size 21 x 21. Filters h1, h2a and h2b were designed with the window method, with band parameters determined by trial and error. The method gave the lowest mean-square reconstruction error on the standard set of Kodak test images compared to other published methods.
Frequency-domain Bayer demosaicking 62
Mean square error comparison
top related