The Linear Monge-Kantorovitch Linear Colour Mapping forExample-Based Colour Transfer.

F. Pitie, A. Kokaram

Sigmedia.tv,Trinity College Dublin, Ireland,

fpitie@mee.tcd.ie

Keywords: Colour Transfer, Colour Grading, Monge-Kantorovitch.

Abstract

A common task in image editing is to change the coloursof a picture to match the desired colour grade of anotherpicture. Finding the correct colour mapping is tricky becauseit involves numerous interrelated operations, like balancingthe colours, mixing the colour channels or adjusting thecontrast. Recently, a number of automated tools have beenproposed to find an adequate one-to-one colour mapping.The focus in this paper is on finding the best linear colourtransformation. Linear transformations have been proposed inthe literature but independently. The aim of this paper is thusto establish a common mathematical background to all thesemethods. Also, this paper proposes a novel transformation,which is derived from the Monge-Kantorovicth theory of masstransportation. The proposed solution is optimal in the sensethat it minimises the amount of changes in the picture colours.It favourably compares theoretically and experimentally withother techniques for various images and under various colourspaces.

1 Introduction

Adjusting the colour grade of pictures is an important step inprofessional photography and in the movie post-production.This process is part of the larger activity of grading in whichthe colour and grain aspects of the photographic materialare digitally manipulated. In particular, a main problem inthe movie industry is to adjust the colour consistently acrossall the shots, even though the movie has been edited withheterogeneous video material. Shots taken at different timesunder natural light can have a substantially different feel dueto even slight changes in lighting. Colour grading is a delicatetask since the slightest of colour variations can alter the moodof a picture.

Typical tuning operations comprise adjusting the exposure,brightness and contrast, calibrating the white point or findinga colour mapping curve for the luminance levels and the threecolour channels. For instance, in an effort to balance thecontrast of the red colour, the digital samples in the red channelin one frame may be multiplied by some gain factor and theoutput image viewed and compared to the colour of some

other (a target) frame. The gain is then adjusted if the matchin colour is not quite right. The amount of adjustment andwhether it is an increase or decrease depends crucially on theexperience of the artist. An another problem is that most ofthese operations are interdependent. For instance, the contraston the red channel can change after a mixing of the colourchannels. It would be therefore beneficial to automate this taskin some way.

This paper presents techniques which aim at facilitating thechoice of the colour mappings. These techniques belong tothe class of example-based colour transfer methods. Theidea, first formulated by Reinhard et al.[13], has raised a lotof interest recently [1, 8, 19, 11]. Figure 1 illustrates thiswith an example. The original picture (a) is transformed sothat its colours match the palette of the image (b), regardlessof the content of the pictures. Consider the two pictures astwo sets of three dimensional colour pixels. A way of treatingthe re-colouring problem would be to find a one-to-one colourmapping that is applied for every pixel in the original image.For example in Figure 1, every blue pixel is re-coloured ingreen. The new picture is identical in every aspect to theoriginal picture, except that the picture now exhibits the samecolour statistics, or palette, as the target picture.

Employing a one-to-one colour mapping is a standard methodin the industry, but note that more complicated gradingtechniques based on local manipulations have also beenproposed [16],[3],[20]. These techniques will not be discussedhere but a comprehensive review can be found in [12].

The notion of transfer of colour statistics encompasses anentire range of possibilities from the simple match of themean, variances [13], covariance [1, 6] to the exact transfer ofthe exact distribution of the colour samples [7, 8, 11]. Thus,depending on how close the graded picture should match thecolour distribution of the example image, multiple techniquescould be used. In this paper, the focus is on linear colourtransformations, which correspond to the transfer of the secondstatistical moments, i.e. the mean and the covariance. A lineartransformation is usually enough to fix most of the colourchanges. Non-linear refinements can been done afterwards ifnecessary using non-linear mappings as presented by Pitie etal. [12]. Solving the problem for linear colour transformationsis indeed of high interest. To obtain a correct linear colourtransformation under Adobe Photoshop R©, one must correctly

(a) Original (b) Target Colour Palette (c) Result

Figure 1: Colour transfer example. A colour mapping is applied on the original picture (a) to match the palette of an example(b) provided by the user.

combine the following 6 adjustment functions: “levels”,“brightness/contrast”, “color balance”, “Hue/Saturation”,“Channel Mixer” and “Photo Filter”. Finding automaticallythe linear transformation would save a lot of time.

Linear colour transfer techniques actually exist in the literature.In particular Abadpour and Kasaei [1] and Kotera [19] havepresented simple solutions to this problem. In fact it transpiresfrom the literature that numerous linear transformationscan achieve an exact transfer of the statistics. The firstcontribution of this paper is to mathematically characterisethe ensemble of these linear transformations. This gives acommon mathematical background to these methods. Thesecond contribution of this paper is to introduce a newtransformation which is based on the Monge-Kantorovitch(MK) of mass transportation. This solution is optimal in thesense that it minimises the amount of colour changes. This MKtransformation turns out to be very intuitive from an artist pointof view because it has monotonous properties. For instancethe brightest and the darkest points remains the brightest anddarkest points after transformation. It is also shown here thatthe MK transformation is independent to orthogonal colourspace conversions.

Organisation of the Paper. The problem is first formalisedin section 2. The ensemble of the all possible lineartransformations is then characterised in section 3. The reviewof existing techniques is then presented in section 4. Thereview is accompanied with a table of comparative results.The optimal Monge-Kantorovitch solution is then presentedin section 5. The last section 6 examines the influence onthe Monge-Kantorovitch transformation when working underdifferent colour-spaces.

2 Mathematical Background of Linear ColourDistribution Transfer

The notion of colour statistics can be understood if an imageis represented as a set of colour samples. When workingin RGB colour space, the image is then represented by theset of the RGB colour samples (R(i), G(i), B(i))1≤i≤M . In a

Mapping u → t(u)

?

pdf f of u pdf g of t(u)

Figure 2: Distribution Transfer Concept. How to find amapping that transforms the distribution on the left to thedistribution on the right?

probabilistic sense, these colour samples are realisations of a 3-dimensional colour random variable and which will be denotedas u for the original image and v for the target palette image.The colour palette of the original and target pictures correspondthen to the distributions of u and v.

To simplify the presentation of the problem, it is supposed herethat both distributions have absolutely continuous probabilitydensity function (pdf) f and g. The problem of colour transferis to find a C1 continuous mapping u → t(u), such that thenew colour distribution of t(u) matches the target distributiong. This latter problem, illustrated in Figure 2 is also knownas the mass preserving transport problem in the mathematicliterature[4, 5, 18].

The mapping is in essence a change of variables. Thus thetransfer equation can be written as:

f(u)du = g(v)dv ⇒ f(u) = g(t(u))|detJt(u)| (1)

where Jt(u) is the Jacobian of t taken at u. The constraintis very complicated in the general case, but in this paper, theproblem is restricted to linear mappings. Mappings are thenof the form t(u) = Tu + t0 where T is a N×N matrix (widthN = 3 for colour). In that form, the Jacobian is then Jt(u) = Tand the quantity |detJt(u)| = |detT | is also constant, thus

f(u) ∝ g(t(u)) (2)

It is not necessarily possible to find a linear mapping in the

general case, but, as it shown hereafter, this can always beachieved when both the original distributions f and the targetdistributions g are multivariate Gaussian distributions (MVG)denoted as N (µu,Σu) and N (µv,Σv):

f(u) ∝ exp(−1

2(u− µu)T Σ−1

u (u− µu))

g(v) ∝ exp(−1

2(v − µv)T Σ−1

v (v − µv)) (3)

with Σu and Σv the covariance matrices of u and v. Note thatwhen the distributions are not MVG, a MVG approximationcan always be obtained by estimating the mean and thecovariance matrices of the distributions. To have the pdftransfer condition of equation (2), i.e. g(t(u)) ∝ f(u), it musthold that:

(t(u)− µv)T Σ−1v (t(u)− µv) = (u− µu)T Σ−1

u (u− µu) (4)

Thus t must satisfy t(u) = T (u− µu) + µv with TT Σ−1v T =

Σ−1u , or equivalently TΣuTT = Σu:{

t(u) = T (u− µu) + µv

TΣuTT = Σv

(5)

It turns out that there are numerous solutions for the matrix Tand thus multiple ways of transferring the colour statistics.

3 Characterisation of the Transformations

The main reason why there is more than one solution is becausethe covariance matrices Σu and Σv admit more than one squareroot, i.e. that there is more than one square matrix A such thatΣu = AAT . To understand this, consider two particular squareroots A and B such that Σu = AAT and Σv = BBT . Forinstance the square roots can be found by taking the Choleskydecomposition: A = chol(Σu) and B = chol(Σv). In thiscase both A and B are lower triangle matrices. Consider thenthe development of equation (5):

TΣuTT = Σv (6)T

(AAT

)TT = BBT (7)

(TA) (TA)T = BBT (8)

One solution for T will be to set TA = B and take T = BA−1

as a solution. However this is not the only solution. Observethat both TA and B are square roots of Σv . An interestingresult about matrices square root, is that all square roots of apositive matrix are related by orthogonal transformations. Thismeans here that there exists an orthogonal matrix Q that linksTA to B by:

TA = BQ (9)

Rearranging this equation yields to the completecharacterisation of the solutions. Any valid transformation canthus be derived from two given square root matrices A and Bof Σu and Σv by:

T = BQA−1 with QT Q = Id (10)

Thus all the following solutions in this paper only differ for thechoice of this inner orthogonal matrix Q. This choice yieldshowever to very different properties for the transformations.

4 Review of Existing Linear Transfers

This section presents some particular transformations that havebeen explored in the colour grading literature. The differenttechniques are compared to each other on a practical examplein Figure 3. The original image is depicted in Figure 3-a and thetarget palette is displayed in Figure 3-b. All colour transfers aredone in the RGB space. Note that further comparison analysisfor different colour spaces is carried out in section 6.

In addition to the comparison on this colour transfer example,it would be interesting to visualise the geometrical differencesbetween the colour mappings. This is presented in Figure 4,which shows some the mappings for a 2D case. The originalMVG f is displayed Figure 4-a, and the target MVG g isdisplayed on Figure 4-b. Figure 4-c to Figure 4-g overlay theestimated 2D transformation on the original distribution.

4.1 Independent Transfer

The first linear method, used by Reinhard et al.[14] in theiroriginal paper on colour transfer, is to simply match the meansand the variances of each component independently. Thistransformation is not a limited solution as it assumes thatboth distributions are separable. The covariance matrices areassumed to be diagonal: Σu = diag(var(u1), . . . , var(uN ))and Σv = diag(var(v1), . . . , var(vN )). It yields for themapping that

T =

√

var(v1)var(u1)

0. . .

0√

var(vN )var(uN )

(11)

The independence assumption is simplistic and is rarely validfor real images. The poor quality of the transfer in the resultsin Figure 3-c shows that this is indeed not always the case. Thesolution proposed by Reinhard is to work in the decorrelatedcolour space lαβ of Ruderman [15]. This helps to some extentbut cannot guarantee a full decorrelation between components.

4.2 Cholesky Decomposition

Following the characterisation of section 3, an exactsolution can be obtain using the Cholesky decompositionof Σu = LuLT

u and Σv = LvLTv where Lu and Lv are lower

triangular matrices with strictly positive diagonal elements.This decomposition yields the following solution:

T = LvL−1u (12)

Figure 3-d shows that some improvements on the previousmethod have been achieved. However, by the iterative natureof the Cholesky decomposition, the way the colour channelsare ordered has actually an influence on the form of thetransformation. As shown on Figure 4-d and Figure 4-e,

the Cholesky decomposition can lead to radically differenttransformation if instead of u = (u2, u1) it is consideredu = (u1, u2). This mean for instance that the channel orderingRGB leads to different results that with the channel orderingBGR.

4.3 Square Root Decomposition

Another popular solution [10, 6, 1, 2, 17] is to find the mappingthat realigns the principal axes of Σv to that of Σu. Inthis solution the covariances matrices are decomposed intosymmetric definite positive matrices. This done by usingthe square root operator for symmetric positive semidefinitematrices. There are multiple square roots for matrices, butif the matrix is symmetric positive then there is only onesymmetric positive square root matrix. Denote as Σ1/2

u andΣ1/2

v these positive square roots. They can be obtained via thespectral decomposition of Σu and Σv:

Σu = PTu DuPu and Σ1/2

u = PTu D1/2

u Pu (13)Σv = PT

v DvPv and Σ1/2v = PT

v D1/2v Pv (14)

where Pu and Pv are orthogonal matrices and Du and Dv

the diagonal matrix containing the (positive) eigenvalues ofΣu and Σv . Since Σ1/2

u and Σ1/2v are uniquely defined, no

special arrangement of the eigenvectors is necessary. Thesedecompositions lead to the following mapping:

T = Σ1/2v Σ−1/2

u (15)

Results displayed in Figure 3-e show an improvement overthe mapping based on the Cholesky decomposition. Note inparticular that the violet colour of the grass in Figure 3-e is notpresent here. This is due to fact that the mapping here does notdepend on the channel ordering.

In fact, this paper shows hereafter that this transformationis independent of orthogonal transformations. Consider anorthogonal transformation defined as u → Hu. To beindependent under the transformation H , it must be shown thatthe transformation T is the same up to the transformation H ,i.e. that THu→Hv = HT Tu→vH . To show this, it is sufficientto realise that:

Σ1/2Hu =

(HT ΣuH

)1/2(16)

=(HT PT

u DuPuH)1/2

(17)

since PuH is also orthogonal, it yields that

Σ1/2Hu = HT PT

u (Du)1/2PuH (18)

= HT (Σu)1/2H (19)

Thus

THu→Hv = Σ1/2Hv Σ−1/2

Hu (20)

= HT Σ1/2v HHT Σ−1/2

u H (21)= HT Tu→vH (22)

This property of independence to orthogonal transformationsis a major advantage over the Choleski based method. Thismeans that the colour transfer has mainly the same effect underdifferent colour spaces, which is clearly not the case withCholeski.

5 The Linear Monge-Kantorovitch Solution

The problem encountered with most of the presentedtransformations is that the geometry of the resulting mappingmight not be as it was intended. For example, it is notguaranteed that a transfer will not map black pixels to whiteand conversely white pixels to black. The resulting picturecould have the colour proportions as expected, but locally thecolours would be swapped. To avoid this problem, a goodsolution is to further constrain the transfer problem and lookfor a mapping that also minimise its displacement cost:

I[t] =∫

u

‖t(u)− u‖2f(u)du (23)

Finding this minimal displacement mapping in the general caseis known as the Monge’s optimal transportation problem.Monge’s problem has raised a major interest in mathematics inrecent years [4, 5, 18] as it has been found to be relevant formany scientific fields like fluid mechanics.

To understand the interest of the MK solution in colour grading,three aspects of the MK solutions will be reported here. Thefirst result of importance is that the MK solution always existsfor continuous pdfs and is unique. This means that there isno room left for ambiguity. The second result, which is ofinterest here, is that the MK solution is the gradient of a convexfunction1:

t = ∇φ where φ : RN → R is convex (24)

This property might seem quite obscure at first sight, but thissimply is the equivalent of monotonicity for one dimensionalfunctions in R. It implies for instance that the brightest areas ofa picture still remain the brightest areas after mapping. The MKsolution is thus geometrically more intuitive than the Choleskyor the Principal Axes solution. One could be concerned thatthe MK solution might not be linear for MVG distributions.Fortunately the solution for MVGs is actually linear and admitsa simple closed form. The detailed proof of how to find the MKmapping can be found in [9]. The mainstay of the reasoning isthat since the MK solution is the gradient of a convex function,the matrix T has to be symmetric definite positive, which leadsto this unique solution for T :

T = Σ−1/2u

(Σ1/2

u ΣvΣ1/2u

)1/2

Σ−1/2u (25)

The corresponding results in Figure 3-f are convincing. Theresults are slightly better than the ones of the Principal Axesmethod. For instance a pink trace on the grass in front of the

1A convex function [5] φ : RN → R is such that ∀u1, u2 ∈ RN , α ∈[0; 1], φ(αu1 + (1− α)u2) ≤ αφ(u1) + (1− α)φ(u2)

house is visible in (e) but not in (f). The difference betweenthe PCA and the MK mappings can also be seen on Figure 4-fwhere both mappings have been overlaid.

Note that, like the PCA based transformation, the MKtransformation is independent of orthogonal transformations.The MK solution is thus interesting as it provides an intuitiveand probably better mapping for a similar computationalcomplexity to the popular methods.

6 Comparisons in different Colour Spaces

If a method can exactly transfer the complete colour statistics,then this method will work regardless of the chosen colourspace. Thus, in that respect, the colour space is not importantto transfer the colour ‘feel‘ of an image.

The colour space has however an influence on the geometricalform of the colour mapping. It is now known that thePCA and MK transformations are independent to orthogonaltransformations, so the influence of the colour space is notas dramatic as it can be for the Choleski method. Howeverthese methods are not independent to non-linear colour spacetransformations or even colour scaling.

In the MK formulation for instance, the cost of transportationis related to the colour difference, which depends on the chosencolour space. Thus uniform colour spaces like YUV, or betterCIELAB and CIELUV, are preferable to the basic RGB toobtain coherent colour mappings. Comparative results forRGB, YUV, CIE XYZ, CIELAB and CIELUV are displayedin Figure 5 for the linear MK solution. Differences might bedifficult to see on printed images, but these are very significantwhen displayed on a big screen, especially if they are imagesin a sequence which is played back. The CIELAB colour spaceoverall offers better renderings, for the linear and the non-linearcase. This is because it is designed to measure the differencebetween colours under different illuminants.

7 Conclusion

The study conducted in the paper has established a solidbackground to understand differences in linear colourtransformations for example-based colour transfer. The linearMonge-Kantorovicth solution, which is to our knowledgeintroduced here for the first time in the image processingcommunity, offers a number of advantages over previouslyproposed transformations. It gives to the user intuitiveguarantees on the colour transfer:

1. it minimises the amount of colour changes

2. it has some nice monotonicity property and thus preservesthe relative positions of colours

3. it produces relatively similar results under different colourspaces

The linear Monge-Kantorovitch seems therefore to be a goodchoice in an interactive post-production environment.

Acknowledgements

This work supported by Adobe Inc, SJ, USA. The authorswould also like to thanks the people at The Foundry for theirremarks and feedback.

References

[1] A. Abadpour and S. Kasaei. A fast and efficientfuzzy color transfer method. In Proceedings of theIEEE Symposium on Signal Processing and InformationTechnology, pages 491–494, 2004.

[2] A. Abadpour and S. Kasaei. An efficient pca-based colortransfer method. Journal of Visual Communication andImage Representation, 18(1):15–34, 2007.

[3] S. Bae, S. Paris, and F. Durand. Two-scale tonemanagement for photographic look. ACM Transactionson Graphics (Proceedings of ACM SIGGRAPH 2006),pages 637–645, 2006.

[4] L. C. Evans. Partial differential equations and Monge-Kantorovich mass transfer. Current Developments inMathematics, 1997, pages 65–126, 1998.

[5] W. Gangbo and R. McCann. The geometry of optimaltransport. Acta Mathematica, pages 113–161, 1996.

[6] H. Kotera. A scene-referred color transfer for pleasantimaging on display. In Proceedings of the IEEEInternational Conference on Image Processing, pages 5–8, 2005.

[7] J. Morovic and P-L. Sun. Accurate 3d image colourhistogram transformation. Pattern Recognition Letters,24(11):1725–1735, 2003.

[8] L. Neumann and A. Neumann. Color style transfertechniques using hue, lightness and saturation histogrammatching. In Proceedings of Computational Aesteticsin Graphics, Visualization and Imaging, pages 111–122,2005.

[9] I Olkin and F. Pukelsheim. The distance between tworandom vectors with given dispersion matrices. LinearAlgebra and its Applications, 48:257–263, 1982.

[10] F. Pitie. Statistical Signal Processing Techniques forVisual Post-Production. PhD thesis, University of Dublin,Trinity College, April 2006.

[11] F. Pitie, A. Kokaram, and R. Dahyot. N-DimensionalProbability Density Function Transfer and its Applicationto Colour Transfer. In International Conference onComputer Vision (ICCV’05), Beijing, October 2005.

[12] F. Pitie, A. Kokaram, and R. Dahyot. Automated colourgrading using colour distribution transfer. Journal ofComputer Vision and Image Understanding, February2007.

(a) Original (b) Target palette

(c) Separable linear transfer (d) Cholesky based transfer

(e) Principal Axes transfer (f) Linear Monge-Kantorovitch transfer

Figure 3: Results for linear techniques. All transfers are done in the RGB colour space.

(a) (b)

(c) (d) (e)

(f) (g) (h)

Figure 4: Maps for different linear transformations. On (a) the original MVG in dimension 2, on (b) the target MVGdistribution. The map for the separable assumption is displayed on (c). The choleski method on (d). Choleski decompositionfor a different axis sequence on (e). The PCA method on (f) and the Monge-Kantorovitch (MK) method on (g). On (h) thecomparison between the PCA (red) and the MK (black) methods.

(a) target colour palette (b) RGB

(c) YUV (d) XYZ

(e) CIELAB (f) CIELUV

Figure 5: Colour-Space. Results of the Linear Monge-Kantorovitch transfer for different colour spaces.

[13] E. Reinhard, M. Ashikhmin, B. Gooch, and P. Shirley.Color transfer between images. IEEE Computer GraphicsApplications, 21(5):34–41, 2001.

[14] E. Reinhard, M. Stark, P. Shirley, and J. Ferwerda.Photographic tone reproduction for digital images. ACMTransactions on Graphics, 21(3):267–276, 2002.

[15] D. L. Ruderman, T. W. Cronin, and C. C. Chiao. Statisticsof Cone Responses to Natural Images: Implications forVisual Coding. Journal of the Optical Society of America,(8):2036–2045, 1998.

[16] H. L. Shen and J. H. Xin. Transferring color betweenthree-dimensional objects. Applied Optics, 44(10):1969–1976, April 2005.

[17] H. J. Trussell and M. J. Vrhel. Color correction usingprinciple components. In M. R. Civanlar, S. K. Mitra,and R. J. Moorhead, II, editors, Proc. of Society of Photo-Optical Instrumentation Engineers (SPIE), volume 1452,pages 2–9, June 1991.

[18] C. Villani. Topics in Optimal Transportation, volume 58of Graduate Studies in Mathematics. AmericanMathematical Society, Providence, RI, 2003.

[19] C. M. Wang and Y. H. Huang. A novel color transferalgorithm for image sequences. Journal of InformationScience and Engineering, 20(6):1039–1056, November2004.

[20] T. Welsh, M. Ashikhmin, and K. Mueller. TransferringColor to Greyscale Images. In Proceedings of ACMSIGGRAPH, pages 227–280, San Antonio, 2002.