Sensor interpixel correlation analysis and 

reduction for color filter array high dynamic 

range image reconstruction Mikael Lindstrand

The self-archived postprint version of this journal article is available at Linköping University Institutional Repository (DiVA): http://urn.kb.se/resolve?urn=urn:nbn:se:liu:diva-154156   

  

N.B.: When citing this work, cite the original publication. Lindstrand, M., (2019), Sensor interpixel correlation analysis and reduction for color filter array high dynamic range image reconstruction, Color Research and Application, , 1-13. https://doi.org/10.1002/col.22343

Original publication available at: https://doi.org/10.1002/col.22343

Copyright: This is an open access article under the terms of the Creative Commons Attribution-NonCommercial-NoDerivs License, which permits use and distribution in any medium, provided the original work is properly cited, the use is non-commercial and no modifications or adaptations are made. © 2019 The Authors. Color Research & Application published by Wiley Periodicals, Inc. http://eu.wiley.com/WileyCDA/

  

 

 

 

RE S EARCH ART I C L E

Sensor interpixel correlation analysis and reduction for colorfilter array high dynamic range image reconstruction

Mikael Lindstrand1,2

1gonioLabs AB, Stockholm, Sweden2Image Reproduction and Graphics Design,Campus Norrköping, ITN, Linköping University,Linköping, Sweden

CorrespondencegonioLabs AB, Stockholm, Sweden.Email: mikael.lindstrand@goniolabs.com

Funding informationgonioLabs AB, Stockholm, Sweden

AbstractHigh dynamic range imaging (HDRI) by bracketing of low dynamic range (LDR)images is demanding, as the sensor is deliberately operated at saturation. This exac-erbates any crosstalk, interpixel capacitance, blooming and smear, all causing inter-pixel correlations (IC) and a deteriorated modulation transfer function (MTF).Established HDRI algorithms exclude saturated pixels, but generally overlookIC. This work presents a calibration method to estimate the affected region fromsaturated pixels for a color filter array (CFA) sensor, using the native CFA as amatched filter. The method minimizes color crosstalk given a set of candidates forproximity regions, and requires no special setup. Results are shown for a 21-bitHDR output image with improved color fidelity and reduced noise. The calibrationreduces IC in the LDR images and is performed only once for a given sensor. Theimprovement is applicable to any HDRI algorithm based on CFA image bracket-ing, irrespective of sensor technology. Generalizations to subsaturated and super-saturated pixels are described, facilitating a suggested irradiance-exposuredependent point spread function charge repatriation strategy.

KEYWORDS

blooming, color filter array, crosstalk, high dynamic range, Interpixel correlation,saturation

1 | INTRODUCTION

Because interpixel correlation (IC) is sparsely treated in thebody of previous work, with individual papers generally cover-ing only selective relevant aspects, the introduction begins withan overview of the problem. The introduction continues withIC causes and remedies and finally ends pointing to the contri-bution of this work.

1.1 | Sensor interpixel correlation

IC in an image sensor is an undesired statistical dependencebetween sensor pixel outputs. In a sensor of nonzero IC, anincrease of the irradiance in exactly one source pixel results inan overestimation of irradiance in the output of one or (most

likely) several pixels, and an underestimation of irradiance inthe output of the source pixel. These overestimations andunderestimations lower the signal-to-noise ratio (SNR) of thedetected signal and worsen the modulation transfer function(MTF) of the sensor. Note, for imaging fidelity enhancementoperations in general SNR and MTF are often opposite aspects,one is improved on the expense of the other. However, for lownoise applications in general, minimizing IC results bothimproved SNR and MTF, which makes the minimization of ICparticularly relevant when aiming for high fidelity imaging.

The term IC is not yet an established concept within theimaging science community. It is used only in a more infor-mal manner, without a clear definition. Within this work, wedefine IC as an aggregate of six established distortion con-cepts from the image sensor literature: optical crosstalk,

Received: 10 April 2018 Revised: 3 December 2018 Accepted: 4 December 2018

DOI: 10.1002/col.22343

This is an open access article under the terms of the Creative Commons Attribution-NonCommercial-NoDerivs License, which permits use and distribution inany medium, provided the original work is properly cited, the use is non-commercial and no modifications or adaptations are made.© 2019 The Authors. Color Research & Application published by Wiley Periodicals, Inc.

Color Res Appl. 2019;1–13. wileyonlinelibrary.com/journal/col 1

charge diffusion crosstalk, the so-called “brighter-fattereffect”, blooming, interpixel capacitance, and smear. Theseconcepts represent different laws of physics; they mayinclude significant elements of nonlinearities and representstochastic as well as deterministic correlation processes, allof which present challenges for analysis and reduction (dec-orrelation) of IC. It is worth noting that this work aims atbeing sensor technology neutral, even though the describedconcrete implementation has some technology specificdetails. No single technology in current use suffers all sixdistortion concepts mentioned above, but every technologymay suffer from a majority of them. Therefore, IC has acomplex, multidimensional character.

In Figure 1, two neighboring pixels are shown, with thephoton entry at pixel p0. Photons and charges ending up inthe intermediate region, i0, only reduce the photon transfer(sensor efficiency), while photons and charges crossing overto the neighboring pixel, p1, also contribute to the IC.

Optical crosstalk may either be circuit-related scatteringin the air-sensor surface boundary causing a photon to departfrom the intended optical path and enter another pixel,1 oroptical diffusion or other lateral redistribution of photonswithin the sensor epitaxial layer or bulk, causing the photonto generate charge in the wrong pixel.2

Charge diffusion crosstalk has two causes: the linear dif-fusion of charge within the sensor from the pixel where thecharge is generated to another pixel where the charge is

collected,2 and the nonlinear brighter-fatter effect, which is aradiant energy dependent widening of the point spread func-tion (PSF).3

Blooming is caused by charge overflow from a full-wellpixel into neighborhood pixel(s).4 In Figure 1, we symbolizethis by an avalanche diode, DB, which is nonconducting inthe back direction (no blooming) up to the avalanche voltage(blooming radiant energy), above which the diode is con-ducting (blooming). Note that the diode model in this illus-tration is conceptual and only describes blooming from p0 top1, that is, cases where the photon is entering pixel p0.

Interpixel capacitance5 is a coupling in CMOS andhybrid CMOS pixel arrays employing source follower pixelamplifiers.6

Smear is due to both electrical and optical effects. Itsmain cause is a lateral diffusion of charges between thephotosensor and the CCD shift register, but it is also causedby light piping underneath or directly through the lightshields that may cover these registers.7–9 Although interpixelcapacitance and smear are caused by different physical pro-cesses related to two different technology platforms (hybridCMOS and CCD, respectively), they are jointly symbolizedhere by one coupling capacitance Cc. In literature, however,Cc is usually related to interpixel capacitance only.5

If the sensor is equipped with a color filter array (CFA)and/or a microlens array, those may be seen as an integralpart of the sensor, and their influence on IC are thus seen assensor-caused effects. This includes reflection, refractionand optical scattering as well as effects from possible voidspace between layers on top of the sensor.10

IC according to the definition in this work characterizesonly sensor-caused effects. Distortions before the sensor ascaused by for example, the lens system and sensor housingveiling glare11 are therefore not considered as IC. The IC gen-erally decays strongly with pixel distance: components of ICsuch as the brighter-fatter effect may drop at a r-2.5 power-lawrate,12 r being the distance between two neighboring pixels.The consequence is therefore small for conventional imagingwith moderate dynamic range (DR) and contrast. For calibratedand radiometrically accurate sensors, and for HDRI and poten-tially high contrast imaging, the lateral extent of the IC mayhowever influence the entire sensor,13 which is of significantimportance for the SNR and MTF.

1.2 | Implications of operating the sensor in saturation

The decorrelation of IC is a challenge already in conven-tional nonsaturated imaging, in part because several differentlaws of physics contribute to the IC as described above. TheIC generally increases with charge accumulation, with thepossible exception of interpixel capacitance where a non-linear reduction of the coupling may occur with increasedcharge accumulation.5,14,15 An increase has also beenreported,16 which complicates the issue further, but the inter-pixel capacitance definitely shows a nonlinear dependence

FIGURE 1 Illustration of intended and unintended collection of photonsand charges in a model sensor, not drawn to scale. The sensor pixel p0 (left)is the photon entry position and the intended charge collection target, whilep1 (right) is a nearby pixel. The small coupling capacitance Cc symbolizesinterpixel capacitance and smear. The avalanche diode, DB, symbolizespixel blooming. The intermediate area i0 potentially contains circuitry forexample, anti-blooming. The vertical position of the photoelectricconversion is a function of a probabilistic process with a spectrallydependent distribution

2 LINDSTRAND

on charge accumulation. Therefore, HDRI by bracketing,where the sensor is deliberately operated also in saturation,makes decorrelation of IC more challenging. Because ofnonlinear effects, conventional linear deconvolution will failto provide effective decorrelation of IC.

1.3 | Previous work

It is established knowledge that there is an interpixel spatialdependence, and the effort to reduce it using conventionallinear signal processing methods is far from new.17 The citedwork concerns astronomy imaging of poorly sampled pointsources using data matching by profile fitting of precharac-terized small variable size PSF pixel masks to the measuredresponse. This facilitates super-resolution in determining theposition of the centroid and better accuracy for the bright-ness. The idea of preanalyzed multiple PSF traces as a func-tion of sub-pixel position of a point source is similar to thepresent work, since the detailed PSF analysis provides infor-mation on how the neighborhood is affected by a local brightspot. It is also suggested that the mask may be applied forzeroing “the saturated pixels and their (likely) contaminatedneighbors”.17 It remains unclear how to estimate the radiantpower of saturated pixels without exposure bracketing,declipping18 or other techniques. This is important, becausethe extent of the contaminated neighborhood increases withthe radiant energy impinging on the saturated pixel. Poorerresults were reported for cases having saturated pixels.17

Another study19 takes a more explicit and straight-forward approach in specifically limiting the effect ofblooming. Around a saturated pixel, a circular neighborhoodis excluded, having a radius of three pixels motivated byempirics. The empirical knowledge of the three pixel dis-tance is of limited general applicability, for example, if thesensor equipment, the HDRI implementation or the dynamicranges do not closely resemble those in the study. A similarstudy20 excludes the region affected by a saturated pixel bya two-pixel erosion around any saturated pixel value andstudying cross-histograms identifying the outlier nonlinear-ities next to high-contrast regions. Although more generaland based on the cross-histogram findings, the rationale forthe exactly two pixel wide erosion remains undescribed.

An existing method for HDRI lens flare removal21 doesnot in a strict sense meet the definitions to be included in thisstudy of IC effects, as lens flare is a pre-sensor defect. How-ever, the mitigation method is relevant also for the decorrela-tion of IC and will be discussed further in section 5.3.

HDR reconstruction has been described22 based on a5 × 5 pixel neighborhood adaptive filtering of LDR sam-ples, taking into account if the centroid is saturated and let-ting the number of saturated pixels in the neighborhoodinfluence the blending in order to “limit the proximity effects(e.g., leakage or pixel cross-talk on the sensor)”. The algo-rithm is commendably ambitious in its analysis and decisionrules. However, where proximity effects from saturated

donor pixels to nearby recipient pixels are present, they willbe reduced but not eliminated. A systematic overestimationerror will remain in the output, because the proximity effectis always considered positive: only the recipient pixels areconsidered, while the saturated pixels are excluded. Theblending in the algorithm's case 2:3 and the estimation incase 3:3, hence also the blending in case 3:4 will all overesti-mate a true (unaffected) output value.

A generalized mosaicing method23 fusing data from mul-tiple views through an optical filter with spatially varyingdensity also takes an ambitious approach in avoiding satura-tion proximity effects. Such effects may result in abruptchanges in the certainty function controlling the constructionof a maximum likelihood (ML) pyramid, thereby causingartificial seams in the result. An introduced fuzzy classifica-tion rule for “saturation-associated” pixels mitigates theseseams. “Low” and “high” saturation control and separateisolated high radiant energy pixels having high certaintyfrom “saturation-associated” pixels with low certainty. Thiscertainty measure defines the pixel weight controlling afeathering effect that mitigates the seam.23 The ML pyramidassumes pixels to be independent (note 13 in the reference),but subsaturation crosstalk, which is present in all physicalsensors, makes that independence assumption invalid. Thismeans that the effect of “saturation-associated” pixels willnot be fully eliminated. In addition, due to the introducedcontrol parameter “low saturation”, high contrast regionsaffected by saturation proximity effects causes pixel valuesbelow “low saturation” to remain undetected, which willcause errors in the result.

A generic image fusion approach, as described already in199324, may have a potential also for the decorrelation ofIC. The algorithm uses a local contrast dependent choice ofeither pixel selection or averaging. Minor generalizations ofthe neighborhood analysis and of the process steps, mostimportantly a saturation rejection, would make it applicablefor IC and HDR reconstruction with improved fidelity.

A method to avoid vignetting caused by a change ofaperture25 may similarly be effective for the decorrelation ofIC, with minor modifications. The method compares pixelradiant energies for a set of LDR images with varying expo-sures irrespective of position, whereby outliers are identifiedand excluded. With only small changes in the algorithm, itmay instead detect IC. The challenge would be to defineeffective outlier criteria.

Early attempts to model and reduce color crosstalk sim-plified the problem by handling for example, only one colorchannel of the tri-chromatic CFA.26 However, the continu-ing trend of reduced pixel pitch requires more effective androbust methods for color crosstalk reduction. A generaliza-tion of a shading correction algorithm achieves a generalizedcalibration of not only color crosstalk, but also minor mis-matches due to micro-lens and imaging lens defects, reduc-ing color noise.27 It is possible to separate a quantum

LINDSTRAND 3

efficiency (QE) characterization of a CMOS sensor with aCFA into effects from the color filters and the bare sensorQE.28 This includes color crosstalk due to pixel neighbors ofa different color.

The characterization principle is developed further byconfiguring tailor-made CFA having spatially repeating5 × 5 pixel evaluation clusters, with each cluster center pixelin one color and the neighborhood pixels in a single differentcolor.29 By using different two-color configurations, themore varied color configurations of a traditional CFA werecharacterized. A monochromatic illuminator was used tofacilitate more detailed study of QE and the color crosstalkdependence on the depth position within the sensor.29

Color crosstalk models and simulations are generallyconsidered in the design, construction and verification ofnew sensors.10 Modified CFA configurations have beenpresented, which reduce the color error and SNR deteriora-tion due to color crosstalk relative to a conventional Bayerconfiguration.30 This in-depth analysis also includes sensi-tivity studies for sources of error, as well as characteriza-tions of metamerism.

Reducing the effects of interpixel capacitance by decon-volution is suggested in the literature,31 but even fairly lowlevels of noise in the input make the implementationchallenging,32 as the deconvolution is a de-facto high-passfiltering in the presence of noise.33

The radiant energy dependence of the PSF due to thebrighter-fatter effect, has been reduced, but not eliminated,by simulating reverse charge shifts at the pixel level.12

1.4 | Remaining challenges

The majority of published methods for decorrelation of ICare limited to non-saturation operation, while the remainingones ignore the non-saturation operation. An aggregation ofmethods to cover the full envelope of operation, saturationas well as non-saturation, appears nontrivial. With fewexceptions,12,22–24,32 the works described generally lackdetailed descriptions and motivations for important algo-rithm decisions. This is understandable, as IC has not beentheir focus but rather is included as a secondary problem.The need for further research on the analysis and decorrela-tion of IC is apparent.

It is desirable, but unquestionably challenging, to gener-alize the PSF to an irradiance-exposure dependent (IED)PSF able to characterize the described IC effects at an appro-priate high fidelity. The subject of IC analysis and reductionrelated to conventional as well as HDR imaging is describedmore thoroughly by the author.34

Because the described LDR problem of optimizing theSNR and MTF is inherently difficult, simplifications andapproximations, including nonconventional approaches, aremotivated to at least improve SNR and MTF performance andto gain knowledge. With the same motivation, in this explor-atory first step, no rigorous SNR improvement analysis is

made. Presented is an image comparison, before and after ICreduction, only. However, the visual comparison leaves nodoubt concerning the notable and substantial improvement.Subsequent studies may then build upon the results to furtherimprove IC decorrelation, advisably including also more rigor-ous SNR and MTF improvement analyses.

1.5 | Contribution

This work presents and implements a calibration method thatfacilitates the detection, analysis, and reduction of IC in theaffected neighborhood of saturated pixels in the LDR CFA sen-sor images, thus improving the LDR SNR and, consequently,also the HDR reconstruction. The calibration method is novelas no special setup is needed, facilitated utilizing the sensornative CFA as a matched filter for IC detection.

Reflecting on the results, concrete suggestions for gen-eralization are described to estimate and reduce both sub-saturation and supersaturation IC, along with a finalgeneralization to an irradiance-exposure dependent (IED)PSF, extending to supersaturation levels.

2 | PROBLEM FORMULATION

Given a CFA HDR capture device imaging a static scene withnegligible level of motion blur for the set of LDR exposuresneeded, we seek a method that specifically reduces the ICcaused by saturation in the LDR images. Based on a definedIC error metric, the method is able to estimate the error and theaffected environment using an appropriate test target.

The given HDR reconstruction is evaluated in terms ofthe visually apparent differences between using and notusing IC decorrelation. To show the influence of the LDRimprovements clearly, a deliberately simplistic HDR recon-struction algorithm will be used, inspired by an early HDRIalgorithm20 but without the ad hoc IC treatment.

The method has a potential for generalization to estimateand exclude subsaturated as well as supersaturated IC-affected environments.

3 | METHOD

A calibration method is introduced to estimate the affectedregion from saturated pixels for a color filter array (CFA)sensor, using the native CFA as a matched filter. Themethod minimizes color crosstalk given a set of candidatesfor proximity regions.

The method is general and requires no special setup. How-ever, to test and verify the method for a challenging input, anexisting goniophotometric measurement system was utilized,consisting of an illumination system, a cylinder-shaped sampleholder and a camera. The setup is configured such that part ofthe image area provides specular reflection, while other parts

4 LINDSTRAND

provide bulk reflections. These two reflection modes, includingan in-between transition region, represent a large difference inreflectance levels and require HDRI. The illumination systemis a halogen white light and a telecentric lens (Sill opticsTC55). The imaging system consists of a telecentric lens (Silloptics T30/0.375), a near-IR blocking filter and a 12-bit CCD,non-cooled, anti-blooming Bayer CFA camera (VosskühlerCCD1300-QCB) delivering raw, non-demosaiced pixelinformation.

3.1 | Sensor calibration

The sensor values were subtracted by the per-pixel biasframes for the exposure durations used, in this work 1, 2,4 … 1024 ms, 11 durations in all. Imaging a constant flatfield scene with varying exposure durations, the sensorresponse is plotted in Figure 2. The response is very close tolinear. This is the only calibration required to execute theproposed IC decorrelation algorithm.

In general, it is preferable to also perform absolute cali-bration of HDR irradiance35 information. Doing so would befully compatible with the work presented, but was not per-formed. Hence, the experimental measurements are referredto as “intensities”, where the general reasoning instead refersto the radiometric values.

3.2 | HDRI—IC contaminated

Using a deliberately simplistic but functional algorithm forHDRI, selecting the highest per-pixel nonsaturated valuefrom the set of exposures and a subsequent bilinear demosai-cing, two different test targets were imaged: a high-glossplastic label and a matt uncoated office paper. The imagescontain a wide dynamic range from high radiant power spec-ular reflections to low radiant power bulk reflections, see

Figure 3. Note, an achromatic original reproduced underdemanding conditions in terms of dynamic range or contrastis indeed a significant test of a color imaging system. Only ahigh performance imaging system (hardware and software)will reproduce the achromatic original without color tint.

Indeed, analyzing the two plain white areas on the leftand right hand side of the barcode in Figure 3, banding andcolor tints are apparent in what should ideally be smoothachromatic gradients across the curved samples. This islargely due to IC contamination, which will be further dis-cussed in section 3.3. Comment: note the inclined bandingin the upper left corner in Figure 3 which is a consequenceof an unintended and undesirable illumination inhomogene-ity. An ideal illumination without this inhomogeneity wouldhave caused a horizontal banding, similar to the other band-ings in the figure. This undesirable imperfection of the illu-mination do however not influence the statistical analysis ofthe color data or the implied other conclusions described inthe following, as the purpose of the motif is to yield a HDRinput of varying achromatic reflectance, which remains truedespite the imperfect illumination.

The RGB values for all the pixels within these two whiteareas are plotted in Figure 4. The image in Figure 3, waswhite balanced for presentation, but Figure 4 shows theunscaled sensor values. Given the homogeneous test targetsand the linear sensor, an image free from IC contaminationwould have shown a range of achromatic intensities plusnoise, that is, data points clustered around a straight line inthe 3D color space. This is not the case in Figure 4: streaks,gaps, and irregularities are clearly visible in the point cloud.

3.3 | Sensor native CFA—an IC matched filter

IC causes undesired modifications of sensor pixel values. Acentral finding in this work is that a CFA sensor may actually

100 101 102 10310−1

100

101

102

103

104

sens

or le

vel [

arb.

sca

le]

sensor integration time [ms]

FIGURE 2 The sensor output versus exposure (integration time). Solidred—red channel; dashed green—green channel; dotted blue—blue channel.Dashed-dotted black—an ideal linear response (with an arbitrary offset) forreference

FIGURE 3 Image of the white areas of the test targets, mounted on acylinder which results in a specular highlight requiring HDRI. Themeasurement is performed with no IC reduction. Left: Glossy plastic label.Right: Matte office paper. Image width 21.8 mm, pixel pitch 17 μm

LINDSTRAND 5

facilitate the detection and characterization of IC. Assuming apoint source input, the source (donor) pixel will decline in valuedue to IC, while the neighboring (receiver) pixels are increasingin value. The CFA causes the modification to potentially operateacross different color channels, depending on the CFA configu-ration. For a Bayer sensor, we note that the four-connectivityneighbor receiver pixels are all of different colors than the donorpixel. Thus, the Bayer CFA provides something akin to a tem-plate matched filter for detecting and characterization of theIC. The declining value and the set of increasing values all con-tribute to the deviation in pixel color vectors. IC can thus bedetected as the donor pixel decline and the receiver pixels' valueincrease, both separately and as a combined effect, which makesfor a sensitive filter.

Generally, the color deviation will be at its strongest for asub-pixel source (approximating a point source). For a brightarea of larger spatial extent against a darker background, thestrongest color deviation occurs at the bright-to-dark

transition, that is, the edge. However, the effect occurs for anynon-flat field, that is, any image with contrast. The experi-ments presented below use the gradients of the white regionsin Figure 3 as input, which give satisfactory results.

The IC may extend beyond the closest four-connectivityneighbors, somewhat reducing the selectivity of the matchedfilter, but assuming the IC is monotonically declining withdistance in any given direction, the described matched filtercharacteristic will essentially remain valid.

The combined CFA and spectral sensitivity differs betweencolor channels, causing the described matched filter characteris-tic to be further improved. For an achromatic input, the mostsensitive channel will donate more than the other channels. If adonor pixel is of the most sensitive channel, the 4-connectivityneighbors have lower sensitivities, and the receiving values areamplified in post-processing. This causes IC caused color devi-ations to be amplified, which further facilitates the IC detectionand characterization.

FIGURE 4 Plot of the reconstructed HDR RGB intensities of the white sample areas of Figure 3, without IC reduction. The dashed–dotted line is theprincipal component (highest variance) of a principal component analysis derived by singular value decomposition, (A) 3D plot of RGB intensities.(B) Projection to R,G only (omitting the blue channel) (C) Projection to R,B

6 LINDSTRAND

In Figure 4, the color data deviates substantially from theideal achromatic straight line. Distinct cluster branches alongthe principal coordinate directions occur at a spacing ofpowers of two, corresponding to saturation levels of theLDR images. Note, however, that all saturated LDR pixelsare already excluded, and the defects are due to secondaryIC influence from saturated donor pixels to neighbors.

3.4 | Defining and evaluating candidate regions for ICinfluence

To characterize the region of IC influence, we define candi-date regions of various shapes and extensions and evaluatethem in an optimization process. Choosing candidatesdeserves some consideration. The IC is a consequence of atleast six distortion components and different laws of physics,as described in section 1.1. Without specific prior knowl-edge it is difficult to foresee one particular shape and extentof the region being more likely than another. Instead, wechoose candidates differing significantly in shape and size,to make a broader search. The goal is to have enough candi-dates to cover a wide range of shapes with reasonable den-sity, such that the detected best candidate is close to theglobal optimum.

The optimization criterion is the minimization of themean square color distortion in RGB space after HDR recon-struction, as evaluated for each of the candidate regions ofFigure 5. The distortion is measured as the orthogonal dis-tance from the achromatic line, see Figure 4. The achromaticline, in turn, was characterized by linear regression analysis,using singular value decomposition (SVD) to determine theprincipal component (highest variance).

More formally: let A 2 Rm × n, m rows of test samples(corresponding the pixels defining the previously described twoplain white areas on the left and right hand side of the barcodein Figure 3) of a n dimensional color space (RGB yield n = 3).Let B = round(A + E), B 2 Zm × n, E 2 Rm × n, be the infor-mation detected by the sensor, where the additive noiseE includes interpixel correlation (IC) and other sensor noise.Using SVD to decompose A

A ¼ UΣVT ð1Þwhere U and V are orthogonal matrices of sizes m × m and n ×n respectively, and Σ = diag(α1, α2, …, αn) is a diagonalmatrix of descending order singular values of A such that α1 ≥α2 ≥ … ≥ αn. The number of nonzero singular values definesthe rank, r, of A, where r ≤ min (m, n). Ideally our monochro-matic test areas A have a first singular vector (1, 1, 1)×3-1/2

only, corresponding to achromatic and neutral gray (the factor3-1/2 for normalization). That is, α2 = α3 = 0, and rank(A) = 1.

However, we detect B including a nonzero E, causingrank(B) = 3 reflecting the detection is not monochromatic,but includes undesired color tints. By instead using SVD onB, B ¼ UBΣBVT

B, the singular value β1 and corresponding(ie, first) singular vector of VB defines the orientation ofmaximum variance of B. By assuming that the orientation ofthe maximum variance of B corresponds to the orientation ofmaximum variance of ideally our monochromatic test areasA, we define an estimate of E:

E ¼ UBΣBrVTB

�� ��F ¼

Xn

i¼2βi

h i1=2ð2Þ

where k•kF is the Frobenious norm, ΣBr is diag 0, β2,…, βnð Þ ,that is, singular value β1 set to zero and thereby keeping onlythe residue of B, after elimination of the first singular vector

FIGURE 5 The evaluated candidate regions. The integer values below and to the left of each sub-figure denote the vertical and horizontal extent of theregion. The value above each sub-figure is the estimated relative color distortion, compared to the null neighborhood at the top left, Ê/Ê0 where Ê0 is theestimate color distortion of the null neighborhood at the top left. The regions in gray rectangles are referred to in the text

LINDSTRAND 7

energy36 (c.f. a residue of a principal component analysis ofB). E and hence Ê are dependent on the candidate regions ofFigure 5. Aiming for a small Ê, an unfavorably large elimi-nation region will cause the HDRI reconstruction to be inef-fective, due to unnecessary pixel eliminations, henceunnecessary high quantization and other sensor noise. A toosmall elimination region will cause an unnecessary high IC,and the IC will dominate the total error Ê. This optimizationis further explained in the next section 3.5. By evaluating Êfor each candidate region of Figure 5 we identify the regioncorresponding to the minimal Ê.

The candidates regions which were evaluated are shownin Figure 5. The patterns of white pixels define the neighbor-hood to exclude around a saturated center pixel. The ordering,left-to-right, top-to-bottom, is in order from worst to best. Thenull 1 × 1 neighborhood, where only the saturated pixel itselfis excluded, is at the top left. The error measures in the figureare the mean square deviations from the achromatic line, inrelation to the same error metric for the null neighborhood.

Among the regions tested, the most effective to reducethe color distortion was the 5 × 5 square, in the lower rightcorner in Figure 5, circumscribed by a gray rectangle. Thisregion was used in the following calculations. This is notnecessarily the optimum. A more thorough search might finda better candidate. However, the error is roughly the samefor similar regions, and the 5 × 5 square is sufficient for thealgorithm to improve the HDR result significantly.

3.5 | IC reduction

The IC related errors are reduced by first excluding pixelsaround saturated pixels within the best candidate regionaccording to section 3.4, and then performing the simplisticHDR reconstruction as before. Where a pixel in one LDRimage is discarded, its irradiance will instead be estimatedfrom a shorter exposure, replacing a sample with IC contami-nation by a sample with higher quantization and read noise.The objective is to find the best middle ground: a region smallenough not to discard useful values, but large enough not tokeep values strongly affected by IC. Optimizing on globalSNR is a reasonable criterion to find the best middle ground.

4 | RESULTS

The results are twofold. First, we estimated a neighborhoodof IC influence, determining where a saturated pixel reducesthe SNR in the HDR reconstruction and should be excluded.Pixels outside that neighborhood increase the SNR andshould be included. The error measure was the color distor-tion in achromatic (white) image areas. Second, the esti-mated IC neighborhood and pixel exclusion were applied inan HDRI reconstruction to illustrate the visual impact of thepixel exclusion on the result HDR image.

The IC-reduced HDR reconstruction is plotted inFigure 6 (c.f. Figure 4) and imaged in Figure 7(c.f. Figure 3). The relative sensitivity of the color channelsare compensated for in Figure 7 (as was done for Figure 3).

Note that Figure 6 shows essentially a range of achromaticintensities plus noise, that is, data points clustered around theachromatic straight line in the 3D color space. This is in con-trast to Figure 4: where streaks, gaps and irregularities areclearly visible in the point cloud. The two plain white areas onthe left and right hand side of the barcode in Figure 7, showsmooth achromatic gradients across the curved samples. This isin contrast to the banding and color tints apparent in Figure 3.

In Figure 8 the exposure duration for each position isillustrated using the same motif as in Figures 3 and 7. Thehighest radiant power specular reflections are imaged withthe shortest exposures.

In Figure 9, the IC reduced HDR reconstruction (Figure 7)is shown again, using different intensity scale factors to illus-trate in print the range of intensities captured better.

Notice that, the bluish tint in the lower part of Figure 9E,and to some degree also in Figure 9F, corresponds to signifi-cant amplifications. This indication of further potentialimprovement is the topic of next section and do not detractfrom the verified main novel work of this study.

5 | GENERALIZATION OF IC REDUCTION

The estimated IC neighborhood is an average, based on allsaturated pixels under the illumination and exposure condi-tions represented in the set of LDR images. Hence the analy-sis did not address irradiance-exposure dependent (IED)IC. An IED IC neighborhood determined by a more rigorousanalysis and estimation would reasonably result in a smallerIC neighborhood for lower radiant energies and a largerneighborhood for higher radiant energies, because mostunderlying causes of IC are exacerbated with increasingradiant energy. With such refined characterization, it wouldbe possible to achieve better IC reduction while discardingfewer pixels. The generalizations are suggestions for futurework to perform such refined characterization, using experi-ence gained in the course of the current work.

First, we describe how to characterize IED IC neighbor-hoods, initially for subsaturation, then for supersaturation.Then, a more advanced generalization is suggested to replacethe crude, qualitative “keep or discard” strategy for LDR pixelswith an IED PSF, and perform a potentially more efficient dec-orrelation of the IC by a quantitative charge repatriationstrategy.

5.1 | Generalization to multilevel subsaturation ICreduction

In order to generalize the findings to multilevel subsaturationIC neighborhoods, the candidate regions would need to be

8 LINDSTRAND

evaluated for a set of subsaturated irradiant exposures orsensor output digital numbers (DN), coarsely or finelyspaced depending on ambition, using interpolation tocover intermediate levels. Properly characterizing the

nonlinear IC effects may require analysis at many differentDN levels, but the characterization may be fully auto-mated. Of course, such more detailed characterizationswould require more and better tailored calibration data, butstill no special setup.

FIGURE 7 HDR image reconstructed with IC reduction, otherwiseidentical to Figure 3

1

2

4

8

16

32

64

128

256

512

1024

FIGURE 8 Illustration of the exposure durations used: 1, 2, 4 … 1024 ms.(11 different exposure durations)

FIGURE 6 Plot of the RGB intensities for HDR reconstruction with IC reduction, otherwise identical to Figure 4

LINDSTRAND 9

A multilevel subsaturation IC characterization can bedescribed as a 3D aggregate of individual 2D subsaturationIC neighborhoods.

5.2 | Generalization to multilevel supersaturation ICreduction

To generalize the method to multilevel supersaturation ICneighborhoods, the candidate regions would need to be eval-uated for a set of supersaturated levels with the same clippedDN. The level of irradiance exposure of a saturated pixelmay be estimated from a shorter LDR exposure of the pixelwith nonsaturated values. If saturated values appear in theshortest exposure, declipping18 may be applied. The reportedrange extension potential for declipping is about 5σ for thenoise in the signal. Past this level, clipping is beyond salva-tion using the described approach.

The resulting multilevel supersaturation IC neighbor-hood may be represented as a 3D structure and aggregatedwith the 3D multilevel subsaturation IC neighborhood,described in section 5.1, to a general multilevel IC neighbor-hood. This multilevel IC neighborhood defines, for the char-acterized range of pixel irradiance exposure below andabove saturation, the spatial extent of pixels to exclude toreduce the joint negative effect of IC distortion, quantizationnoise and read noise.

5.3 | Generalization to an irradiance-exposuredependent PSF charge repatriation strategy

The binary decision to either keep a LDR pixel value for fur-ther processing or to discard it for being too contaminated bydistortion and noise, may be refined further by addressing tworesidual sources of error. First, signal will be lost in the elimina-tion of the IC affected neighborhood. This signal may be sal-vaged if the neighborhood is instead characterized andprocessed. Second, pixels outside the neighborhood are pro-cessed as if free from IC, which is not correct as IC also influ-ences more distant pixels. This influence may be reduced if theIC effects for more distant pixels are characterized and used toprocess data differently. Generalizing the multilevel IC neigh-borhood into an Irradiance-Exposure Dependent (IED) PSFhence has a potential of gaining SNR within the IC neighbor-hood as well as outside this neighborhood.

This quantitative characterization requires a special testmotif and a more controlled collection of larger amounts ofcalibration data. Measuring a radiant point source, it is possi-ble to characterize the detailed shape of the IC influence,similarly to a conventional PSF characterization. For a CFAsensor, the influence of an imperfect point source may bereduced by spectral matching of the source to the CFA chan-nels to evoke a response predominantly in a single colorchannel. This is another example of the CFA being used tofacilitate the IC analysis.

Pixel-individual variations may motivate pixel-individualcharacterization or, less ambitiously, at least characterizingdifferent sets of pixels. Multiple characterizations may beperformed in parallel, provided that the IC influences fromdifferent point sources do not interfere. One possible test tar-get would be a light blocking foil with an array of pinholes.

Using this set of point sources and making sure they hitpixels of all color channels with a dense enough distributionon the sensor, the multiple different subsaturation excitationsdescribed in section 5.1 and the multiple different supersatu-ration excitations described in section 5.2 together facilitatean IED PSF characterization.

The IED PSF opens up for what may be named a“charge repatriation strategy”, with the objective of decorre-lation of multilevel subsaturation and supersaturationIC. The purpose of the repatriation is similar to a linear PSFdeconvolution, that is, to undo the impact of the PSF, but arepatriation strategy for decorrelation of IC would also con-sider the nonlinearity and the supersaturation aspects of IC.

A published method for HDRI lens flare method21 mayinspire a design of a repatriation strategy. The method isrestricted to subsaturation correlations, performing a PSF analy-sis for one exposure and assuming the effect is linear, which isreasonable in the given context. In addition, the PSF is assumedradially symmetric,21 whereas the IED PSF in the present workcan attain arbitrary shape. Nevertheless, an arbitrary shapedIED PSF should not add further challenges compared to the

FIGURE 9 HDR image reconstructed with IC reduction, rendered with sixdifferent intensity scale factors: (A) 0.72, (B) 5; (C) 50, (D) 100; (E) 2000,(F) 20 000

10 LINDSTRAND

algorithm referred to,21 besides the main problem of incorporat-ing the irradiance exposure dependence.

The IED PSF characterization may be facilitated by HDRIexposure bracketing. As each position is sampled at differentexposures, this may be described as an overdetermined equa-tion system, albeit with noise and IC. The relative influenceof read noise, quantization noise and IC distortion will varywith irradiant exposure, such that the noise will dominate thelow irradiant exposures and the IC distortion will dominatethe high irradiant exposures. This will cause sensor IC toincrease with exposure, resulting in a characteristic and pre-dictable IC related color tint that facilitates the IED PSF char-acterization and optimizations of repatriation, and hence apotential SNR improvement in the HDR reconstruction.

6 | DISCUSSION

The most effective region to reduce the color distortion wasthe 5 × 5 square, among the candidate regions tested. Thismay indicate that the IC to a considerable degree stem fromcrosstalk (charge diffusion or light scattering), prone to bemore circular-symmetric in distribution, than from bloom-ing, prone to be more column oriented. If true, it would indi-cate the anti-blooming functionality of the CCD sensorbeing effective up to the radiant exposures evaluated, asblooming is often a challenge for this sensor technology ifnot equipped with anti-blooming functionality.

The approximately circular footprint is also in agreementwith previous results for one aspect of IC, the brighter-fattereffect,12 where the closest neighbors show an anisotropiccorrelation but the further reaches of the effect are isotropic.The minor deviation from an isotropic shape in the currentwork may be a consequence of the configuration of theBayer mosaic. Omitting the corners of the 5 × 5 pixelsquare, the lower right corner in Figure 5, circumscribed bya gray rectangle, the optimal IC neighborhood in this study,corresponds to a more isotropic shape, the lower left cornerin Figure 5, circumscribed by a gray rectangle. However,this would also keep pixels of the same color channel as thesensor position causing the saturation, which may be the rea-son for the inferior performance compared to the 5 × 5square. Also, the data set used in this study was fairly small,and the optimization was performed on all saturated pixelsirrespective of irradiant exposure. More experiments wouldbe needed to determine the definite optimum.

In addition to the three generalizations presented, section 5,a detailed characterization of IC may also be resolved in termsof position in the sensor focal plane, similarly to crosstalk37

and PSF17 characterizations. The optimization may further beperformed for each color channel individually, as it is reason-able to assume the IC being spectrally dependent due to contri-butions from spectrally dependent optical crosstalk.4

There are other findings worth mentioning which are rele-vant to the understanding of IC, although their applicability is

not immediately obvious. Janesick comments on page 144 inhis reference book38 that pixel crosstalk at saturation acts as anaveraging filter, causing an apparent reduction of noise. This isimportant not only when estimating the noise level, but alsowhen characterizing sensor gain by the commonly used photontransfer curve method, or performing any other noise-dependent characterizations. Although more pronounced at sat-uration, the effect is present also at subsaturation.

Cross-histogram (joint histogram, 2D-histogram) analy-sis may be useful to gain a deeper understanding of IC. Thediscrepancy between the sensor output relations for differentexposures,20,39–41 referred to as “widening ridge” in cross-histogram terms, illustrates one aspect of IC, albeit notexplicitly identified as a correlation effect in the references.

Interpixel charge transfer effects have been introduced toexplain image deterioration in x-ray astronomy.6 Althoughlacking references by others, the effect may be of relevanceto refine the characterization of the IC into its sensor-causedsubcomponents, such as for example, charge transfer.

Although the interpixel capacitance and the PSF widen-ing as described in section 1.3 may be mitigated by othermethods,12,31,32 our presented method adds the ability toreduce the IC effect also of saturated pixels in a controlledmanner, hence potentially widening the useful range of irra-diance exposures.

Established HDRI algorithms implicitly reduce aspects ofIC, but run the risk of losing or distorting information as a nega-tive side effect. The complexity of IC at subsaturation as well assupersaturation, having different physical causes and exhibitingnonlinearities, makes it particularly problematic to adapt theworkarounds of established HDRI work to mitigate an identifieddistortion at its source. A tailor-made IC reduction method thatfollows the best practice in signal processing and operatesdirectly on the systematic errors in the LDR data set will have ahigher potential of effectiveness in terms of reducingcorrelation-related errors. This in comparison with a less specificsignal processing methods of established HDR reconstructionalgorithms. This, irrespective of whether the less specific algo-rithms are based on design and optimization of weight func-tions, operating jointly with the HDR reconstruction, or appliedas post-processing. Consequently, the present work shows asubstantial decorrelation of the IC even with fairly simplemeans, by attacking the specific problem at the source in a man-ner clearly motivated by analysis.

Inspiration for methods to reduce IC may be found inother fields than HDR imaging, as related further to by theauthor.34

The presented work may be applicable to a wider field ofHDRI, including both exposure fusion and radiant powercalibrated applications, and also to non-HDRI applications,for example, Wide Dynamic Range sensors and conven-tional imaging. In sensor design, IC resilience is always anoptimization with competing characteristics.

LINDSTRAND 11

The algorithm introduced in the current work is a prepro-cessing strategy to mitigate IC in the LDR images. EstablishedCFA HDR reconstruction algorithms that either do not considerIC, or perform the IC reduction less effectively, may adopt theproposed method as a preprocessor without a need to otherwisemodify the HDR algorithm, yielding a higher SNR irrespectiveof the HDR reconstruction algorithm used.

The algorithm presented has been implemented in anindustrial security document imaging application42 withHDR requirements.

6.1 | Limitations of the presented work

The rescaling of the sensitivity of the color channels is per-formed for the illustration purpose of Figure 3 and Figure 7only, but not for the singular value decomposition withrespect to the achromatic line. Depending on the application,it may be desirable instead to perform a rescaling first, forexample, a white balancing based on visual importance ofthe color channels. It may have been more pedagogical toperform such white balancing prior to the optimization,because in this manuscript the application is for visualiza-tion, demonstrating the algorithm in images, which moti-vates a white balance calibrated optimization. However andmost importantly: whichever calibration target that best suitthe application at hand, the principal benefit and the algo-rithm presented remains otherwise unchained.

Spectral crosstalk,1 caused by chromatic imperfections inthe CFA resulting also non-intended colors to pass, is neitherdetected as IC nor reduced by the present algorithm. Like-wise, irradiance spectral variations within the spectral pass-bands of each of the color channel of the color filter array,will influence the diffusion crosstalk, hence IC, but will notbe detected using the suggested method, including the gener-alizations. However, this influence is (a) limited to one (opti-cal crosstalk) of the six described components of IC only,and (b) generally very limited on the overall optical crosstalkcomponent, given the context described. Therefore, this limi-tation is likely of practical importance to only a mostextreme and niche application.

ACKNOWLEDGMENT

The author would like to thank Stefan Gustavson, SasanGooran, Jonas Unger, and Reiner Lenz, all at ITN, Linköp-ing University, for insightful comments and valuable feed-back improving the earlier versions of this article, and StefanGustavson for assistance in preparing the final text.

CONFLICT OF INTEREST

Mikael Lindstrand is the sole owner of gonioLabs AB andgonioLabs sole proprietorship, and has related patents US6147750, and SE 532553 (family; also AU, CA, CN, HK,US, and IN) issued to gonioLabs sole proprietorship.

ORCID

Mikael Lindstrand https://orcid.org/0000-0002-2537-2703

REFERENCES

[1] Agranov G, Berezin V, Tsai RH. Crosstalk and microlens study in a colorCMOS image sensor. IEEE Trans Electron Devices. 2003;50(1):4-11.

[2] Kavaldjiev D, Ninkov Z. Subpixel sensitivity map for a charge-coupleddevice sensor. Opt Eng. 1998;37(3):948-954.

[3] Antilogus P, Astier P, Doherty P, Guyonnet A, Regnault N. The brighter-fatter effect and pixel correlations in CCD sensors. J Instrum. 2014;9:C03048. https://doi.org/10.1088/1748-0221/9/03/c03048.

[4] Janesick JR. Scientific Charge-Coupled Devices. Vol PM83. Bellingham,WA, USA: SPIE Press; 2001. DOI: https://doi.org/10.1117/3.374903.

[5] Moore AC, Ninkov Z, Forrest WJ. Quantum efficiency overestimation anddeterministic cross talk resulting from interpixel capacitance. Opt Eng.2006;45(7):076402. https://doi.org/10.1117/1.2219103.

[6] Simms LM. Hybrid CMOS SiPIN detectors as astronomical imagers [PhDDissertation]. Stanford, USA: Department of Applied Physics, StanfordUniversity; 2010. http://pqdtopen.proquest.com/pubnum/3395872.html?FMT=AI, Accessed May 7, 2018.

[7] Stevens EG, Lee YR, Burkey BC. The effects of smear on Antibloomingprotection and dynamic range of interline CCD image sensors. IEEE TransElectron Devices. 1992;39:2508-2514.

[8] Teranishi N, Ishihara Y. Smear reduction in the interline CCD image sen-sor. IEEE Trans Electron Devices. 1987;34:1052-1056.

[9] Tanabe A, Kudoh Y, Kawakami Y, et al. Dynamic range improvement byNarrow-Channel effect suppression and smear reduction Technologies inSmall Pixel IT-CCD image sensors. IEEE Trans Electron Devices. 2000;47:1700-1706.

[10] Agranov G, Mauritzson R, Barna S, et al. Super Small, Sub 2μm Pixels ForNovel CMOS Image Sensors. In: International Image Sensor Workshop;June 7–10, 2007; Ogunquit Maine, USA. http://www.imagesensors.org/Past%20Workshops/2007%20Workshop/2007%20Papers/079%20Agranov%20et%20al.pdf, Accessed May 7, 2018.

[11] ISO. Optics and Optical Instruments - Veiling Glare of Image Forming Sys-tems - Definitions and Methods of Measurement. Vol 9358. Geneva, Swit-zerland: ISO; 1994.

[12] Gruen D, Bernstein GM, Jarvis M, et al. Characterization and correction ofcharge-induced pixel shifts in DECam. J Instrum. 2015;10(5):C05032.https://doi.org/10.1088/1748-0221/10/05/C05032.

[13] Du H, Voss KJ. Effects of point-spread function on calibration and radio-metric accuracy of CCD camera. Appl Opt. January 20, 2004;43(3):665-670.

[14] Cheng L. Interpixel capacitive coupling [M. Sc. Thesis]. Rochester, UnitedStates, Rochester Institute of Technology; 2009. http://scholarworks.rit.edu/theses/2871, Accessed May 7, 2018.

[15] Donlon K, Ninkov Z, Baum S, Cheng L. Modeling of hybridized infraredarrays for characterization of interpixel capacitive coupling. Opt Eng. 2017;56(2):024103. https://doi.org/10.1117/1.oe.56.2.024103.

[16] Giardino G, Sirianni M, Birkmann SM, et al. Noise properties and signal-dependent interpixel crosstalk of the detectors of the near-infrared spectro-graph of the James Webb space telescope. Opt Eng. 2013;52(3):034001.https://doi.org/10.1117/1.oe.52.3.034001.

[17] Howell SB, Koehn B, Bowell E, Hoffman M. Detection and measurementof poorly sampled point sources imaged with 2-D arrays. Astron J (NY).1996;112:1302-1311.

[18] Foi A. Clipped noisy images: heteroskedastic modeling and practicaldenoising. Signal Process. 2009;89:2609-2629.

[19] Tsin Y, Ramesh V, Kanade T. Statistical Calibration of CCD Imaging Pro-cess. In: International Conference on Computer Vision; 7–14 July, 2001;Vancouver, BC, Canada. DOI: 10.1109/iccv.2001.937555.

[20] Madden BC. Extended Intensity Range Imaging: University of Pennsyl-vania, Department of Computer & Information Science;1993.Paper 248. http://repository.upenn.edu/cis_reports/248, Accessed May7, 2018.

[21] Reinhard E, Ward G, Pattanaik SN, Debevec P. High Dynamic RangeImaging; Acquisition, Display, and Image-Based Lighting. San Francisco,CA, USA: Morgan Kaufmann Publisher; 2006.

12 LINDSTRAND

[22] Tocci MD, Kiser C, Tocci N, Sen P. A Versatile HDR video productionsystem. In: ACM SIGGRAPH; 7–11 August, 2011; Vancouver, BritishColumbia, Canada. DOI: https://doi.org/10.1145/1964921.1964936.

[23] Schechner YY, Generalized Mosaicing: High dynamic range in a wide fieldof view. Int J Comput Vis. 2003;53:245-267.

[24] Burt PJ, Kolczynski RJ. Enhanced image capture through fusion. In: Inter-national Conference on Computer Vision; 11–14 May, 1993; Berlin, Ger-many DOI: https://doi.org/10.1109/ICCV.1993.378222.

[25] Mitsunaga T, Nayar SK. Radiometric self calibration. In: Computer SocietyConference on Computer Vision and Pattern Recognition; June 23–25,1999; Fort Collins, CO, USA. DOI: https://doi.org/10.1109/CVPR.1999.786966.

[26] Li W, Ogunbona P, Shi Y, Kharitonenko I. Modelling of color cross-talk inCMOS image sensors. In: IEEE International Conference on Acoustics,Speech, and Signal Processing (ICASSP); 13–17 May, 2002; Orlando, FL,USA. DOI: https://doi.org/10.1109/icassp.2002.5745428.

[27] Getman A, Uvarov T, Han YI, Kim B, Ahn JC, Lee YH. Crosstalk, colortint and shading correction for small pixel size image sensor. In: Interna-tional Image Sensor Workshop; 7–10 June, 2007; Ogunquit, ME, USA.http://www.imagesensors.org/Past%20Workshops/2007%20Workshop/2007%20Papers/042%20Getman%20et%20al.pdf, AccessedAugust 30, 2018.

[28] Wu Y, Cizdziel PJ, Rhodes HE. Decoupling photo collection efficiency andcolor crosstalk from the quantum efficiency spectrum for the CMOS imagesensor pixel development. In: Digital Photography V; January 18, 2009;San Jose, CA, USA. DOI: https://doi.org/10.1117/12.805743.

[29] Vaillant J, Mornet C, Decroux T, Hérault D, Schanen I. Characterization ofpixel crosstalk and impact of Bayer patterning by quantum efficiency mea-surement. In: Digital Photography VII; 23 January, 2011; San FranciscoAirport, California, USA. DOI: https://doi.org/10.1117/12.879603.

[30] Anzagira L, Fossum ER. Color filter array patterns for small-pixel imagesensors with substantial cross talk. J Opt Soc Am A 2015;32:28-34.

[31] Finger G, Dorn R, Meyer M, Mehrgan L, Moorwood AFM, Stegmeier J.Interpixel capacitance in large format CMOS hybrid arrays. In: HighEnergy, Optical, and Infrared Detectors for Astronomy II; 24–31 May,2006; Orlando, Florida , USA. DOI: https://doi.org/10.1117/12.671828.

[32] McCullough P. Inter-pixel capacitance: prospects for deconvolution: SpaceTelescope Science Institute;2008. Instrument Science Report WFC32008–26. http://www.stsci.edu/hst/wfc3/documents/ISRs/WFC3-2008-26.pdf, Accessed May 7, 2018.

[33] Starck JL, Pantin E, Murtagh F. Deconvolution in astronomy: a review.Publ Astron Soc Pac. 2002;114(800):1051-1069. https://doi.org/10.1086/342606.

[34] Lindstrand M. Sensor Interpixel correlation analysis and reduction: areview. J Imag Sci Technol. 2019;63(1). https://doi.org/10.2352/J.ImagingSci.Technol.2019.63.1.010502.

[35] Debevec PE, Malik J. Recovering high dynamic range radiance maps fromphotographs. In: SIGGRAPH; 3–8 August, 1997; Los Angeles, CA, USA.DOI: https://doi.org/10.1145/258734.258884.

[36] Konstantinides K, Yao K. Statistical analysis of effective singular values inmatrix rank determination. IEEE Trans Acoust Speech Signal Process.1988;36(5):757-763. https://doi.org/10.1109/29.1585.

[37] Hirakawa K. Cross-talk explained. In: IEEE International Conference onImage Processing; 12–15 October, 2008; San Diego, CA, USA. DOI:https://doi.org/10.1109/icip.2008.4711845.

[38] Janesick JR. Photon Transfer. Vol PM170. Bellingham, Washington, USA:SPIE Press; 2007. DOI: https://doi.org/10.1117/3.725073.

[39] Mann S, Picard RW. On being ‘undigital’ with digital cameras: extendingdynamic range by combining differently exposed pictures. In: Imaging Sci-ence & Technology Conference; 7–11 May, 1995; Cambridge, MA, USA.http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.21.3141,Accessed May 7, 2018.

[40] Robertson MA, Borman S, Stevenson RL. Estimation-theoretic approach todynamic range enhancement using multiple exposures. J Electron Imaging.2003;12:219-228.

[41] Battiato S, Castorina A, Mancuso M. High dynamic range imaging for digi-tal still camera: an overview. J Electron Imaging. 2003;12:459-469.

[42] Lindstrand M. Spatially and angularly resolved high dynamic range reflec-tance measurements for forensic document inspection. Optical DocumentSecurity - the Conference on Optical Security and Counterfeit Deterrence;January 23–25. USA: San Francisco, CA; 2008.

AUTHOR BIOGRAPHY

MIKAEL LINDSTRAND is the founder and sole owner ofgonioLabs AB. He received an M.S. degree in computer sci-ence and engineering, 1996, and a Ph.D. in computer sciencefrom Linköping University, Sweden, 2018. For more thantwo decades he has developed and implemented image anal-ysis measurement and characterization tools in the context ofhuman visual perception-based evaluation. Since 2007, afterfounding gonioLabs, the work is focused on measurementmethods and equipment for perceptually meaningful charac-terization of optically variable image devices as generallycritically features of security documents such as travel docu-ments (Passports) and bank notes. Such features, often basedon diffractive or interference optics, typically exhibit highchromatic contrast and high dynamic range of reflectance.Therefore these materials are especially demanding in termsof interpixel correlation, dynamic range capability and mod-ulation transfer function of the image sensor. Currently hedoubles as a consultant data scientist in assignments “Artifi-cial intelligence ability development” and “Analyst for pre-dictive analysis” at two Swedish Government agencies.

How to cite this article: Lindstrand M. Sensor inter-pixel correlation analysis and reduction for color filterarray high dynamic range image reconstruction. ColorRes Appl. 2019;1–13. https://doi.org/10.1002/col.22343

LINDSTRAND 13