Exploiting Color Strength to Improve Color Correction

Lisa Brown, Ankur Datta, Sharathchandra Pankanti IBM T.J. Watson Research Center

Yorktown, NY 10598 USA (lisabr, ankurd, sharat)@us.ibm.com

Abstract—Color information is an important feature for many vision algorithms including color correction, image retrieval and tracking. In this paper, we study the limitations of color measurement accuracy and explore how this information can be used to improve the performance of color correction. In particular, we show that a strong correlation exists between the error in hue measurements on one hand and saturation and intensity on the other hand. We introduce the notion of color strength, which is a combination of saturation and intensity information to determine when hue information in a scene is reliable. We verify the predictive capability of this model on two different datasets with ground truth color information. Further, we show how color strength information can be used to significantly improve color correction accuracy for the 11K real-world SFU gray ball dataset.

Keywords-component; color, color constancy, color correction, chromatic adaptation

I. INTRODUCTION Color correction methods are used to compensate for

illumination conditions. In human perception such correction is called color constancy – the ability to perceive a relatively constant color for an object even under varying illumination. Most computer methods are pixel-based, correcting an image so that its statistics satisfy assumptions such as the average intensity of the scene under neutral light is achromatic, or that for a given illuminant, there is a limited number of expected colors in a real-world scene.

Various schemes have been proposed to use features instead of pixels including higher order derivatives or homogeneous color regions. These features are chosen based on their likelihood to best characterize the illuminant color and ignore the specific color of the objects in the scene. For example, higher order derivatives are used based on the assumption that the average of reflectance differences in a scene is achromatic. As another example, homogeneously colored segments can be used to catalog the colors that appear in a scene, thereby reducing the influence of a single large region.

However, to the best of our knowledge, none of the existing methods account for the fact that even at the level of the individual pixels, the reliability of the color information varies. In this paper, we introduce the

notion of color strength, a measure of color information accuracy. Color strength postulates that for color information to be reliable, it requires both saturation and intensity to have strong information content. As demonstrated by our experiments, neither saturation nor intensity alone, can predict the reliability of color information but a combination thereof can predict reliability of color information accurately.

In Fig. 1, we show three example images from the SFU dataset [1] and their corresponding color strength images. Notice the objects whose color is clearly perceived, such as the green trees and shrubs in the leftmost image or the person’s red shirt in the middle image appear white (highest color strength) in their corresponding color strength images. We will show that it is the color of the pixels with high color strength that can be robustly measured.

II. RELATED WORK Color constancy methods have been categorized into three major groups (static, gamut-mapping and learning-based) and extensively evaluated in the recent work by Gijsenij et al.[2] These methods are an important precursor to improve algorithms which rely on color such as image retrieval, matching color across cameras and long term background modeling for surveillance. The typical methodology, whether based on learning or not, and regardless of the underlying assumptions, still rely on global image statistics.

Figure 1. Top row: original images from SFU gray ball dataset [1]. Bottom row: corresponding image showing the color strength for

each pixel, where white is the highest color strength.

Preprocessing for color correction often involves local averaging or Gaussian smoothing. This serves to reduce noise and has been shown to be beneficial for color correction [3, 4]. However, this type of noise

2012 IEEE International Symposium on Multimedia

978-0-7695-4875-3/12 $26.00 © 2012 IEEE

DOI 10.1109/ISM.2012.43

188

2012 IEEE International Symposium on Multimedia

978-0-7695-4875-3/12 $26.00 © 2012 IEEE

DOI 10.1109/ISM.2012.43

179

2012 IEEE International Symposium on Multimedia

978-0-7695-4875-3/12 $26.00 © 2012 IEEE

DOI 10.1109/ISM.2012.43

179

reduction, while it may improve performance overall, introduces artifacts along high gradients and potentially ignores relevant information. The color strength model proposed here is related to earlier work in color histograms, in particular, methods which are based on one of the perceptually uniform color spaces. Several investigators have concluded that the features derived from perceptually uniform space are in many ways optimal for color image retrieval [5]. In order to avoid instability along the gray axis in hue-based spaces such HSV/HSB/HSI, a weighting system was developed by [6], a non-uniform binning system by [7] and chromatic/achromatic splitting method by [8]. In spite of such technology, it remains the case that hue information becomes unreliable near the gray axis: the transformation to hue-based spaces is ill-conditioned near the gray axes and the noise inherent in the raw RGB images is therefore amplified. In this paper, we explore how to identify and either remove or utilize this noise. We then show how this model of hue error can be used to significantly improve color correction accuracy.

We experimentally corroborate our model with the measurement of known color charts. We then show the validity of our postulate on two datasets with ground truth color information and demonstrate its practical value to improve a range of color correction approaches.

III. RELIABILITY OF COLOR MEASUREMENT In order to understand the relationship between

reliability of color measurement and ground-truth color, we undertook the following experiment. We took three color charts of basic colors (red, green, and blue – see Fig. 2) composed of multiple patches that ranged over a wide-range of intensity (from 0.2 to 0.6) and saturation (from 0.05 to 0.5), and obtained their corresponding image views across three cameras. Next, we measured the error between ground-truth hue values and their corresponding image measurements.

We noticed that hue error grew as both intensity and saturation became lower. Our experiment provided strong empirical evidence of a correlation between the error in hue and the combination of saturation and intensity. We empirically found that among the many potential combination possibilities for saturation and intensity, a product of them appeared to provide good approximation power for the correlation. In Fig. 3, the error in the hue measurement is shown with respect to color strength for one of the indoor cameras. Except for a few green outliers, all large hue errors occur at low color strength. In addition, we also note that the error bars are much higher at lower color strength as compared to when the color strength is higher suggesting reduction in hue error. For regions of

homogeneous color, local hue variation provides a cue to determine whether hue will accurately be measured at the corresponding local color strength.

Figure 2. Three color charts were tested in 3 camera views.

Figure 3. As the color strength weakens, both the hue error and hue deviation increases. Hue deviation is shown in the error bar for each point. Measurements from the red/green/blue patches are shown in

their respective colors.

Additionally, we have tested the predictive power of color strength for determination of hue error on two different datasets. See Fig. 4 and 6. These two datasets use very different approaches to perform color correction which shows that our result is not dependent on the type of ground truth methodology. The first dataset is a video sequence provided by Sunkavalli [9]. In their work, they model time-varying changes in both direct sunlight and ambient skylight using static background pixels and perform color correction from frame to frame. We use their color corrected sequence as ground truth in order to evaluate the hue error with respect to color strength.

Figure 4 shows the average hue error for all pixels in the image below the specified color strength for 16 different frames of varying lighting changes (approx. every 500th frame from the original sequence). An example of a one of these images and its color corrected counterpart is shown in Fig. 5.

In the second dataset, we use the very large (11K), real-world images taken by Ciurea [1]. Every image in this set contains a small gray sphere which was physically connected to the camera. Ground truth color correction is computed based on the primary color mode of the brightest region on the gray sphere.

189180180

Figure 4. Hue error is significantly larger for pixels at low color strength for 16 images from [14].

Figure 5. An example frame (left) from [14] and its corresponding

color corrected frame (right).

Fig. 6 shows a similar hue error plot for the 11K images of the SFU gray ball dataset [1]. In this case, we plotted the median of the average hue error over all images between the original image and its corresponding color corrected version. The hue error is computed using only the specified percent of pixels with the lowest color strength. In this way, we guarantee there are a known (and a reasonable) number of pixels for each measurement. In both Fig. 4 and 6, there is significant reduction in hue error as color strength is increased.

IV. RESULTS OF COLOR STRENGTH MODEL In this section, we show the results of using color strength to improve several color correction methods on the SFU dataset (see Table I and Fig. 7 for results). In this experiment, we used four color correction methods: gray world (GW), general gray world (GGW), gray edge (GE) and second order gray edge (GE2) provided by [2] to compare the performance of each method with and without color strength information. These methods can all be derived as instantiations of the following framework:

�∫ �����,ℴ()��� �� �� �

� ��= ���

�,�,ℴ (1)

where ��,ℴ() is the color of the pixel at location x, p is the Minkowski norm (p=1 is the average and p=∞ is the maximum.), σ is a Gaussian smoothing scale factor, n is the order of the derivative and A is the set of all pixels in the image. On the right hand side, �� represents the color vector of the light source normalized by a multiplicative constant k chosen so the vector has unit length.

Figure 6. Median hue error for 11K images of SFU gray ball

dataset [12] with .1 standard deviation error bar. Error decreases significantly (from an average of 27° to an average greater than 37°)

for smaller color strength.

For each method, we used the optimal parameterization of the derivative order (n), Minkowski norm (p), and the smoothing scale (σ) as computed in (1) for the SFU dataset and shown below the method name in Table 1. Based on our model, hue error is largest at low color strength. Therefore, the most valuable information for computing correction will be found in pixels with low color strength. We compute the result of color correction using all pixels (100%) (A = the entire image in equation [1]) and using only the 50% of the pixels with the lowest color strength. Specifically, let us define a function g(x,y) such that

�(�, �) = �1 �� �(�, �) ∗ �(�, �) < !0 "#ℎ�%&�'�

where S(x,y) and I(x,y) is the saturation and intensity at pixel location (x,y) . We then find K such that,

* * �(�, �) +

-.�

/

�.�= .5 ∗ 3 ∗ 4

Lastly, we change the set of pixels A in equation [1] to be,

6 = {�, � | �(�, �) ∗ �(�, �) < !}

190181181

The error is computed in two ways. RMS error is computed between the ground-truth corrected image and the corresponding color corrected image for each method. Angular error is computed between the ground truth illuminant and the estimated illuminant. For all methods and for both error types, we can observe that incorporation of color strength information leads to significant improvement in the results.

V. CONCLUSION In this paper, we studied the empirical relationship between hue error, and intensity and saturation in camera images. We found that there exists a strong correlation between hue error and color strength, which is a product of intensity and saturation. The principle benefit of the color strength model is that it can be used to estimate the reliability of the color information contained in a pixel. We empirically tested this model on two datasets with ground-truth color information, and in both datasets the color strength model had strong predictive power for hue error. Finally, we showed that the color strength model predicts that the pixels with the lowest color strength will provide the most information for color correction for four standard color constancy methods. This was substantiated using the largest available real-world dataset with illuminant color ground truth. We believe the color strength model can be used to improve the performance of many other algorithms which rely on hue information such as image retrieval by color, object tracking, and person re-identification.

REFERENCES [1] F. Ciurea, and B. Funt, "A Large Image Database for Color

Constancy Research," Proceedings of the Imaging Science and Technology Eleventh Color Imaging Conference, pp. 160-164, Scottsdale, Nov. 2003.

Table I. Color correction error for four methods using all pixels (100%) compared to the four methods using 50% of the pixels with

the lowest color strength based on the proposed color strength model. The proposed algorithm has significantly lower error for all the color

correction methods.

Error Type

%Pixels

Method Type (n,p,σ) GW GGW GE GE2 0,1,0 0,13,2 1,1,6 2,1,5

rms 50 6.81 4.32 4.53 4.60 100 7.70 4.85 4.72 4.91

angular 50 5.46 4.16 4.05 4.24 100 6.24 4.47 4.37 4.60

[2] E. A. Gijsenij, T. Gevers, and J. van de Wiejer, “Computational

Color Constancy: Survey and Experiments,” IEEE Trans. On Image Processing, Vol 20, No. 9, Sept. 2011.

[3] K. Barnard, L. Martin, A. Coath, and B. Funt, “A comparison of computational color constancy algorithms – part ii: experiments with image data,” IEEE Trans. on Image Processing, vol. 11, no. 9, pp 985-996, September 2002.

[4] J. van de Weijer, Th. Gebers, A. Gijsenij, “Edge-based color constancy,” IEEE Trans. on Image Processing, vol. 16, no. 9, September 2007.

[5] D. Borhesani et al., “Color Features Performance Comparison for Image Retrieval,” Image Analysis and Processing – ICIAP 2009, Lecture Notes in Computer Science, Springer Berlin/Heidelberg, p902-210, August 2009.

[6] J. van de Weijer, T. Gevers, & A. Bagdanov, “Boosting Color Saliency in Image Feature Detection,” IEEE Trans. Pattern Analysis and Machine Intelligence (PAMI), Vol 28, No. 1, p150-156, 2006.

[7] Z. Lei, et al., “A CBIR Method Based on Color-Spatial Feature,” Proc. IEEE Region 10 Annual International Conference 1999 (TENCON'99), Cheju, Korea. p166-169, 1999.

[8] L. Brown, “Example-based Color Vehicle Retrieval for Surveillance,” IEEE MMSS Workshop, Boston, MA Sept 2010.

[9] K. Sunkavalli, F. Romeiro, W. Matusik, T. Zickler and H. Pfister, “What do color changes reveal about an outdoor scene?” IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) 2008.

Figure 7.Top row from left to right: Ground truth image, First pair: GW (50% - Our result), GW (100%), Second Pair: GGW(50% - Our result),

GGW(100%). Bottom row from left to right: Ground truth image, First pair: GE (50% - Our result), GE (100%), Second Pair: GE2 (50% - Our result), GE2 (100%). Angular error for each color correction method is shown at the bottom right of each image. Figure best seen when zoomed in.

191182182