SVMsSVMsSVMsSVMs IIIInterpolationnterpolationnterpolationnterpolation BBBBasedasedasedased EEEEdgedgedgedge CCCCorrectionorrectionorrectionorrection SSSSchemechemechemecheme forforforfor CCCColorolorolorolor FFFFilterilterilterilter AAAArrayrrayrrayrray

Baiting ZHAO , Xiaofen JIAzhaobaiting@gmail.com jxfzbt2008@163.com

School of Electrical and Information Engineering, Anhui University of Science & Technology, Huainan 232001,China

Abstract-To settle the problem of blurring andvisible artifacts around the edge regions of color filterarray (CFA) interpolation images, a Support VectorMachines (SVMs) interpolation based edge correctionscheme is proposed. In this scheme, a simple CFAinterpolation method is used, and the support vectorregression (SVR) is trained to rectify the color valuesat the edge of the result image. This scheme canproduce visually pleasing full-color images and obtainbetter PSNR results than other conventional CFAinterpolation algorithms. The correction isconcentrated on the edge region, because the humanperception is mainly focusing on edge region. Thescheme can reduce the training time effectively, andcan combine with other CFA interpolation algorithmsfreely. Simulation studies indicate that the proposedalgorithm is effective.

Keywords-edge correction, color filter array,support vector machines, support vector regression

1.1.1.1. IntroductionIntroductionIntroductionIntroduction

Image interpolation has been widely used inelectronics, remote sensing, medical imaging, militaryand so on. Fig.1 shows the basic structure for digitalstill cameras (DSCs). To produce a color image, threesensors (CCD or CMOS ) is needed to obtain red (R)、 green (G) and blue (B). This is a very expensiveapproach, and actually only professional DSCs use thisstructure. In order to minimize the size, cost andcomplexity of the DSCs, digital camera designers use asingle sensor overlaid with a color filter array (CFA) toacquire a color image, as shown in Fig.2. There is onlyone of the three primary color values available in eachsensor pixel. To restore a full-color image from CFAsamples, the two missing color values at each pixelneed to be estimated from the neighboring samples.This process is commonly known as the CFAInterpolation or Demosaicking.

OpticalOpticalOpticalOpticals y s t e ms y s t e ms y s t e ms y s t e m

R filterR filterR filterR filter++++sensorsensorsensorsensor

G filterG filterG filterG filter++++sensorsensorsensorsensor

B filterB filterB filterB filter++++sensorsensorsensorsensor

+

FigureFigureFigureFigure1111 Three-sensorThree-sensorThree-sensorThree-sensor digitaldigitaldigitaldigital cameracameracameracamera structurestructurestructurestructure

C F AC F AC F AC F AInterpolationInterpolationInterpolationInterpolation

OpticalOpticalOpticalOpticals y s t e ms y s t e ms y s t e ms y s t e m

Figure2Figure2Figure2Figure2 Single-sensorSingle-sensorSingle-sensorSingle-sensor digitaldigitaldigitaldigital cameracameracameracamera structurestructurestructurestructure

Bilinear interpolation is the most widely usedmethod due to its simplicity and speediness. It fills in a

missing color value with the average of its neighboringCFA samples of the same color --- four samples for

Image scene

Sensor data RGB color data

Camera output

Color filters+sensors

Image scene CFA+sensor Sensordata

RGB color dataCFA interpolation(demosaicking)

Camera output

2009 Second International Conference on Machine Vision

978-0-7695-3944-7/10 $26.00 © 2010 IEEE

DOI 10.1109/ICMV.2009.27

194

each missing G value and two for a missing R or Bvalue in the Bayer CFA pattern (see Fig.3). Goodvisual quality can be obtained in the smooth region.However, it introduces large errors in the edge regionthat blurs the result image and produces highly visibleartifacts around the image edge region.

To restore more accurate and visually pleasingresults, many sophisticated demosacking methods havebeen proposed exploiting with image spatial or spectralcorrelation, or both [1-2]. Soo-Chang Pei [1] provideseffective color interpolation in CCD color filter arraysusing signal correlation (ECI), which is a low-complexity interpolation method. For typical images, itproduces superior PSNR and better image qualityperformance than other algorithms. Effective use ofspatial and spectral correlations for color filter arraydemosaicking [2] (EECI) is proposed by Lanlan Chang,who stresses the importance of well exploiting bothimage spatial and spectral correlations, and proposingeffective schemes to enhance the two existing state-of-the-art interpolation methods.

Support Vector Machine (SVM) is built on the basisof statistical learning theory with optimal ways to solvethe problem of machine learning [3]. In recent years,SVMs have been used successfully in imageinterpolation [4-5] with the advantages of computationor image quality. Recently, a novel SVMs based imageinterpolation method employing information about thelocal spatial properties of the source image provideshigh quality result images [6].

In almost all developed SVMs based imageinterpolation algorithms, people take each entire imageas a complete set of data to estimate the result image.Nevertheless, it is easy to understand that asophisticated method and simple interpolation tend tobe very similar in smooth regions. Consequently, amore efficient approach is to separate image pixels intothe edge region and the smooth region. One of themajor contributions of this article is to reduce trainingtime, and more pleasant visual quality could beachieved as the training task is now concentrated onthe edge region. The other main advantage of thisscheme is that it can combine with most interpolationalgorithms.

2.2.2.2. SupportSupportSupportSupport vectorvectorvectorvector regressionregressionregressionregression

SVR is a convex optimization problem, which cangain the optimal solution. -SVR and -SVR are theε γtwo widely used styles.

In -SVR, the parameter of -insensitive lossε ε εfunction should be ascertained firstly, but, it is verydifficult. -SVR can calculate automatically, whichγ ε

is the transformation of -SVR, as expressed asεfollows [3]:

(1)2

1

1 1min [ ( * )]2

l

i ii

cl

γε ξ ξ=

+ + +∑ωωωω� �

s.t. ( ( ) )i i iy xφ ε ξ− ⋅ + ≤ +ωωωω bbbb( ( ) ) *i i ix yφ ε ξ⋅ + − ≤ +ωωωω bbbb, , * 0, 1i i i lε ξ ξ ≥ =⋯ ⋯

where is the punishment factor, which is used tocrealize the compromise between the experiential riskand the fiducial range. and are the slackiξ *iξvariables for handling the data that can not beestimated by function with accuracy . is a( )f x ε εvariable of optimization, given as a part of the solution.Finally, we can obtain the function:

(2)1

( ) ( * ) ( , )l

ii ii

f x Kα α=

= − +∑ x x bx x bx x bx x b

where and are Lagrange multipliers,iα *iαand is the kernel.( , ) ( ) ( )i iK = Φ ⋅Φx x x xx x x xx x x xx x x x

3.3.3.3. CorrectionCorrectionCorrectionCorrection schemeschemeschemescheme

Theoretically, SVR can gain the optimal solution,because it is a convex optimization problem. Thatmeans the missing color values which are estimated bySVR are the proximal to the actual values. So, high-quality full-color image can be gained by SVMs basedCFA interpolation. And simple CFA interpolationmethods can gain satisfied result with high speed insmooth regions. Therefore, we intend to apply the SVRalgorithm only on the edge pixels. The overview of thealgorithm is as follows: First, the entire image isinterpolated by a simple CFA interpolation method.Second, the edge detection is done on the result image,and then SVR is trained with the training samples.Finally, the color values at the edge of the result imagecan be rectified by the trained SVR.

3333.1.1.1.1.... EdgeEdgeEdgeEdge detectiondetectiondetectiondetection

The full-color image is firstly obtained using asimple interpolation method. Then, the Prewitt edgedetector is applied on the interpolated image toascertain all edge pixels that will be used for thefurther processing. Comparing with other edgedetection operator, the Prewitt edge detector has theoperation of difference filter operation, and the edgelocalization is more accurate and complete. Since edgedetection generates an edge map that often consists ofsome lines of one-pixel width, we need to extend theseone-pixel lines, that is, to “lengthen” or “broaden”

195

them, so as to generate the representation of wholeedge along two sides. Then, a dilation is performed onthe edge map image in order to locate a fewneighboring pixels which are related to the edge pixels.The structuring element for dilation is a 3 by 3diamond mask.

Prewitt edge detector is an operator of gray image,so, the R, G and B image planes should be detectedindependently for color images.

3333.2.2.2.2.... InterpolationInterpolationInterpolationInterpolation edgeedgeedgeedge usingusingusingusing SVMSVMSVMSVM

Generally, all images have strong spatial andspectral correlations. Image spatial correlation refers tothe fact that within a homogeneous image region,neighboring pixels share similar color values, whilespectral correlation indicates that there is a highcorrelation between the R, G and B image planes, sothat the difference (or radio) between two color planesis likely to be a constant within a local imageneighborhood. According to spectral correlation, wedefine the constant Kr as G minus R (Kr=G-R) and Kbas G minus B (Kb=G-B) [1]. To obtain a high-qualityinterpolation image, either correlation should beexploited adequately when using SVR.

Next, we will present the interpolation algorithmwith Bayer CFA.

The first step: interpolate G image plane. We useGR, GB to denote the inserted G components at R and Blocation.

(1) On the planes of Kr and Kb, we can select properneighbor pixels region according to spatial correlation,then the trained SVR model is obtained by trainingSVR, and then the trained SVR is employed tocalculate for every pixels values need to be estimated.The principle of selecting neighbor pixels region is thetrained mode similar with the forecast mode. Theforecast mode is determined by the position of thesame color pixels around the neighbor region.

Firstly, construct the plane of Kr. In Fig. 3, G hasdifferent positions, such as G3 which is in the place ofodd row and column, the corresponding Kr3=G3-(R2+R4)/2. And G8 is in even row and column, thecorresponding Kr8=G8-(R2+R14)/2. The G positionaround edge, for instance, G13 and G34,we can gain thecorresponding Kr by Kr13=G13-R14, Kr34=G34-R28. Krplane will be formed after all G spots were calculated,as shown in Fig. 4.

Secondly, interpolate all Kr pixels corresponding toR using SVR on the plane of Kr. To train SVR, aregion around the pixel to be interpolated must bedecided prior to the interpolation calculation. In Fig. 4,

the input patterns of SVR are 4 dimensions, the outputpattern is 1 dimension. Such as, select Kr3, Kr17, Kr27,Kr13 as input pattern, the corresponding output patternis Kr15. As to Kr15, the input pattern can also be selectedas Kr8, Kr10, Kr22, Kr20 or Kr1, Kr5, Kr29, Kr25. Then theSVR training is performed. Finally, the trained SVR isemployed to the interpolation calculation for Krcorresponding to R14 with the input pattern Kr8, Kr15,Kr20, Kr13. Similarly, all Kr pixels corresponding to Rcan be obtained.

In the same way, the plane of Kb can be constructed,and all Kb pixels corresponding to B can be estimatedwith SVR. Now, the interpolation of G image plane isfinished.

(2) The G components corresponding to R and Bcan be obtained with the formulae of GR=R +Kr, GB=B+Kb.

The second step: interpolate B image plane. Thediacritical signs of the inserted B componentscorresponding to R and G are BR and BG, and Kb pixelscorresponding to R and G are KbR and KbG.

(1) Estimate pixels Kb of the position of R and G.First of all, construct the plane of Kb. After the first

step, every pixels of G image plane have pixel values.As shown in Fig. 5, the Kb plane can be gained by theformula of Kb=GB-B in the position of B.

Then, on the plane of Kb interpolate all Kb pixelscorresponding to R. In Fig. 5, if the output pattern isKb21, the corresponding input pattern can be selected asKb7, Kb11, Kb35, Kb31 or Kb9, Kb23, Kb33, Kb19. Then theSVR training is performed with the selected samples.Next, with the input model of Kb7, Kb9, Kb21, Kb19, thetrained SVR can interpolate the KbR14 of the position ofR14. Similarly, all KbR corresponding to R can beobtained.

Finally, In Fig. 5, interpolate all Kb correspondingto G. According to the position of G, there are twosituations to be considered. One is in the place of oddrow and column, such as G15, the corresponding KbG15can be obtained with the input model of Kb9, KbR16,Kb21, KbR14 and the trained SVR that be trained wheninterpolating Kb. The other situation is in the place ofeven row and column, the KbG10 corresponding to G10can be gained by the trained SVR that be trained wheninterpolating Kb and the input model of KbR4, Kb11,KbR16, Kb9. In the same way, we can interpolate all Krcorresponding to G.

(2) The B components corresponding to R and Gcan be obtained with the formulae of BR=GR-KbR，BG=G-KbG.

The third step: interpolate R image plane. Themethod is similar with the second step.

196

G2 7R2 6

G2 0 B2 1 B2 3

G1 7G1 5R1 4

B1 1B9G8

G3R2

G2 9

G5

R2 8

G2 2

R1 6

G1 0

R4

B1 9

G1 3

B7

G2 5

G1

R3 0

G2 4

R1 8

G1 2

R6

G3 2 B3 3 B3 5G3 4B3 1 G3 6

K r2 7R2 6

K r2 0 B2 1 B2 3

K r1 7K r1 5R1 4

B1 1B9K r8

K r 3R2

K r2 9

K r5

R2 8

K r 2 2

R1 6

K r 1 0

R4

B1 9

K r1 3

B7

K r2 5

K r1

R3 0

K r2 4

R1 8

K r1 2

R6

K r3 2 B3 3 B3 5K r 3 4B3 1 K r3 6

G2 7G r2 6R2 6

G2 0 K b 2 1 K b 2 3

G1 7G1 5G r1 4R1 4

K b 1 1K b 9G8

G3G r2R2

G2 9

G5

G r 2 8R2 8

G2 2

G r 1 6R1 6

G1 0

G r4R4

K b 1 9

G1 3

K b 7

G2 5

G1

G r3 0R3 0

G2 4

G r1 8R1 8

G1 2

G r6R6

G3 2 K b 3 3 K b 3 5G3 4K b 3 1 G3 6

Fig.3Fig.3Fig.3Fig.3 BayerBayerBayerBayer CFACFACFACFA Fig.4Fig.4Fig.4Fig.4 KrKrKrKr planeplaneplaneplane ofofofof interpolateinterpolateinterpolateinterpolate GGGG Fig.5Fig.5Fig.5Fig.5 KKKKbbbb planeplaneplaneplane ofofofof interpolatinterpolatinterpolatinterpolateeee BBBB

4.4.4.4. Experiments

The experiments are performed in Matlab7.4 andthe SVMs tools for Matlab [7] are used. In order toverify the effect of the proposed algorithm, somestandard test images are used in our experimentsshown in Fig.6. The quality of result images can beevaluated by objective and subjective indicators. Theresult images are compared with the original images tocalculate the peak signal to noise radio (PSNR), andPSNR are calculated as follow.

(3)2

10 3 2

z 1 1 1

25510log( , , ) ( , , )

M N

m n

M NPSNRI m n z I m n z

= = =

⋅ ⋅=

−∑ ∑∑

where and are the original image and theI Iinterpolated result image, respectively; the size of

and is .The pixel in and the pixelI I M N× × 3 Iin are denoted as and ,I ( , , )I m n z ( , , )I m n zrespectively, where , and .1 m M≤ ≤ 1 n N≤ ≤ 1 3z≤ ≤

In experiments, the -SVR with RBF kernel isγemployed in the SVR interpolation experiments, allparameters in formula (1) and the threshold parameterof Prewitt operator are defaulted. The interpolationprocess of R plane of Bridge can be seen in Fig. 7,

which indicates that the quality of the approachproposed in this article is visually pleasing thanbilinear interpolation. The PSNR of differentinterpolation approaches can be seen in Tab.1.

In order to observe the details of image results moreclearly, some local amplificatory result images ofdifferent interpolation approaches for Bridge and Wallare illustrated in Fig. 8-9 respectively. It can beobserved that the bilinear interpolation, ECIinterpolation and EECI interpolation blur the imageedges. Especially the bilinear interpolation, it also hasvisible artifacts around image edge regions. Ourproposed SVMs interpolation based edge correctionscheme obtains the best result image which is verysimilar with the standard image. These observationresults are consistent with the calculation results inTab.1.

Fig.6Fig.6Fig.6Fig.6 standardstandardstandardstandard imageimageimageimageTab.Tab.Tab.Tab.1111 PSNRPSNRPSNRPSNR ofofofof differentdifferentdifferentdifferent interpolationinterpolationinterpolationinterpolation approachapproachapproachapproacheseseses

image Bilinear SVM +Bilinear ECI SVM+ECI EECI SVM+EECI

Nest 16.3467 20.5758 21.0814 22.6174 20.6900 21.1689

Bridge 20.6344 26.0190 20.9420 26.0160 23.3905 25.5842

Face 13.6626 19.4035 18.1249 21.1044 19.5262 21.3416

House1 23.1911 28.6964 25.6434 30.9172 27.2380 29.5592

House 23.3086 25.7191 26.3930 28.2074 27.8547 28.2803

Lighthouse 22.3971 26.0556 25.3098 27.8631 27.3210 28.1940

Book 15.7092 19.6705 20.2747 21.4948 21.7949 22.1265

Wall 20.2676 29.7617 26.1521 31.5860 25.3302 30.3924

197

Fig.Fig.Fig.Fig. 7777 interpolationinterpolationinterpolationinterpolation processprocessprocessprocess ofofofof RRRR planeplaneplaneplane ofofofof BridgeBridgeBridgeBridge

Fig.8Fig.8Fig.8Fig.8 interpolationinterpolationinterpolationinterpolation resultsresultsresultsresults ofofofof BridgeBridgeBridgeBridge

Fig.9Fig.9Fig.9Fig.9 interpolationinterpolationinterpolationinterpolation resultsresultsresultsresults ofofofof WallWallWallWall

5555. ConclusionsConclusionsConclusionsConclusions andandandand futurefuturefuturefuture prospectsprospectsprospectsprospects

This article presents a simple but effective methodto generate a CFA interpolation image with pleasantvisual quality. In the proposed algorithm, we separatethe edge region from the smooth region so as to applydifferent processing methods in these two regions. Thesmooth region is reconstructed by a simple CFAinterpolation scheme, whereas the edge region ishandled by the SVR. The SVR based interpolation isperformed on the plane of Kr and Kb, which wellexploits both the image spatial and the spectralcorrelations. The simulation results show that theproposed algorithm produces visually pleasing full-color images and obtains higher PSNR.

6666.... ReferencesReferencesReferencesReferences

[1] S. C. Pei, I. K. Tam, “Effective color interpolation inCCD color filter array using signal correlation”, IEEE

Trans.on Circuits and Systems for Video Technology,2003, 13(6): 503-513.

[2] L. L. Chang, Y. P. Tan, “Effective use of spatial andspectral correlations for color filter array demosaicking”,IEEE Trans. on Consumer Electronics, 2004, 50(1):355-365.

[3] N. Y. Deng, Y. J. Tian, A novel data mining method:SVM, Science Press, Beijing, 2004.

[4] S. Zheng, J. W. Tian, and J. Liu, “Research of SVM-based image interpolation algorithm optimization”,Journal of Image and Graphics, 2005, 10(3):338-343.

[5] J. Wang, L. Ji, “Image interpolation and errorconcealment scheme based on support vector machine”,Journal of Image and Graphics, 2002, 7(A) (6):558-564.

[6] L. Y. Ma, Y. Shen, and J. C, Ma. “Local spatialproperties based image interpolation scheme usingSVMs”, Journal of Systems Engineering and Electronics,2008, 19(3):618-623.

[7] C.-C. Chang and C.-J. Lin, LIBSVM: a library forsupport vector machines, 2001. Software available athttp://www.csie.ntu.edu.tw/~ cjlin /libsvm.

(a)Standard image (b)Bilinear (c)Result of Prewitt detect (d)Result of diamond dilate (e)SVM +Bilinear

(a) Standard image

(b) Bilinear (c) ECI (d)EECI

(e) SVM +Bilinear (g) SVM +EECI(f) SVM +ECI

(a) Standard image

(b) Bilinear (c) ECI (d)EECI

(e) SVM +Bilinear (f) SVM +ECI (g) SVM +EECI

198