comparison between different implementation s of dark

CS534: Computational Photography – Final Project Report

Fig. 1. Example of Haze Images

Comparison between Different Implementations of Dark Channel Prior Dehazing Method

Shiwei Zhou, Sicong Jiang

Link to Project Website: http://go.wisc.edu/u19c0x Abstract - Photos taken outdoors often suffered from haze, be it smoke or particles in the air. In this project, we are going to implement an image dehazing algorithm based on the paper “Single Image Haze Removal Using Dark Channel Prior”. The dark channel is a statistics of haze-free outdoor images based on observations that most local patches in haze-free outdoor images contain some pixels with very low intensities in at least one color channel. With this prior, and if the atmospheric light is known, the transmission map can be estimated by taking a minimum operation in the local patch on the haze imaging equation. Soft matting is applied to refine the transmission map and the scene radiance can then be recovered using the haze imaging equation with restriction the transmission to some lower bound. The atmospheric light is estimated by picking the top brightest pixels in the dark channel, which is more robust than existing methods. With this prior on the haze imaging model, we can estimate the thickness of the haze and recover a high quality haze-free image. Moreover, the depth map can be estimated as a byproduct of the algorithm, which is useful in many other applications. We plan to divide the project into three major parts, that is, the transmission map estimation, recovering the scene radiance, and estimation of atmospheric light. Because of the speed limitation of soft matting method, we also implemented several alternatives to soft matting based on the Retinex theory. We also compare the performance of the above mentioned implementations.

1

http://go.wisc.edu/u19c0x


1. Motivation and Problem Statement

Haze is a usual phenomenon in our daily life, it is defined as dust and particles in the air that make the sky unclear. As shown in Fig. 1, images taken in hazy weather often suffers from low contrast and objects in the picture looks fuzzy. This will cause problem in many different areas. For travelers who went to other places to travel, the hazy image will make travel experience became a bad one; for video surveillance, where accuracy is important, hazy image reduce the accuracy. Therefore, haze-free images is highly desirable for both consumers and other industries. In our project, we recognize this problem and try to compare the efficiency and effectiveness of different implementation based on the dark channel prior method proposed by He et.al.

The remaining part of the paper will organize as following, in Section 2, we will explore previous methods of image dehazing. Follow by that will be He’s method [1] of image dehazing using dark channel prior. Xie’s method [2] of haze removal using Multi-scale Retinex theory will be presented in Section 4. We will do a qualitative comparison in Section 5 and finally conclude the paper in Section 6.

2. Related Works

Many methods have been proposed to create haze-free image. Some of the methods apply image enhancement algorithms to improve the over contrast of the image; some other methods would implement algorithms that based on physical model to restore the image. The classic method for image enhancement include histogram equalization, homomorphic filter, wavelet transform, Retinex algorithm, luminance and contrast transform and so on. For methods based on physical model, it usually involve the step of creating a degradation model and estimate the original image based on this degradation model. Methods based on physical model usually provide a more natural result and still maintaining the details in the image. The most recent research of the first category was conducted by Tarel et al [3], their proposed method was not able to remove haze from small artifacts.

McCartney [4] is the first person to propose the widely used degradation model used to describe hazy image. Many researchers have adapted this method, including the polarization based methods which propose a method to create haze-free image by using two or more images taken from different degrees of polarization. The challenge for all of the methods, however, lies on the speed of processing as well as the effectiveness. In this paper, we will compare the method of haze removal by using dark channel prior proposed by He et.al [1] with the method of proposed Xie et.al [2] as well as one variation on Xie’s method. We will provide several examples and compare the time consumed as well as the effectiveness of each of these methods.

3. Haze Removal using Dark Channel Prior

A. Dark Channel Prior The dark channel prior is based on the observation on haze-free outdoor images saying that in most

non-sky patches, at least one color channel has very low intensity at some pixels. For an image J, its dark channel Jdark is defined as

, , ( )( ) min ( min ( ( )))dark c

c r g b y xJ x J y

∈ ∈Ω= , (1)

2


Fig. 2. Dark channels of haze-free images (left) and haze image (right)

where Jc is a color channel of J and Ω(x) is a local patch centered at pixel x. Observation says that if J is a haze-free outdoor image, then except for the sky region, the minimum intensity in a local patch among three color channels should has very low value and tends to be zero. In other words, Jdark should be very small for most haze-free images.

Fig. 2 shows an example of dark channels of different outdoor images from the internet with sky region cropped out. As can be observed, the dark channels of haze-free images tends to have very small intensity values and appears very dark, while hazy image usually has bright values in its dark channel. The low intensities in dark channels of haze-free images are mainly caused by three factors: a) shadows, that is, dark regions that is blocked by objects so that light cannot reach; b) colorful objects or surfaces, in which at least one of the color channels has low values so that they appear as colorful (otherwise they would appear as monochrome); c) dark objects or surfaces, which has low intensity values in all three color channels. On the other hand, due to the additive air light, a hazy image tends to be brighter than its haze-free version, and therefore, its dark channel normally have higher intensity values in dense haze areas. Using this prior information, transmission and the atmospheric light can be estimated and haze in the image can be removed gracefully.

B. Estimating the Transmission Assume the atmospheric light A is given, and the transmission in a local patch Ω(x) is constant, then

the transmission can be estimated by taking the min operation in the local patch on the haze imaging Equation (1).

( ) ( )min ( ( )) ( ) min ( ( )) (1 ( ))c c c

y x y xI y t x J y t x A

∈Ω ∈Ω= + − (2)

→ ( ) ( )

( ) ( )min( min ( )) ( ) min( min ( )) (1 ( ))c c

c cc y x c y x

I y J yt x t xA A∈Ω ∈Ω

= + − (3)

According to dark channel prior, the dark channel Jdark of the haze-free image J tends to be zero:

( )( ) min( min ( ( ))) 0dark c

c y xJ x J y

∈Ω= = . (4)

Substituting the dark channel prior (4) into Equation (3), we get the estimate for transmission as

( )

( )( ) 1 min( min ( ))c

cc y x

I yt xA∈Ω

= − . (5)

Even though sky regions does not fully satisfies the dark channel prior, it is found that the color of sky is very similar to the atmospheric light A in haze images, and therefore we have

3


(a) (b) (c) (d)

Fig. 3. (a) Haze image, (b) estimated transmission, (c) refined transmission, and (d) recovered scene radiance

Fig. 4. Estimating the atmospheric light

( )

( )min( min ( )) 1c

cc y x

I yA∈Ω

→ , and ( ) 0t x →

in the sky regions. This observation gracefully handles the sky regions for the estimation of transmission in Equation (5) as sky is at infinite and tends to have zero transmission, which means that there is no need to separate the image into sky and non-sky regions.

In practice, a constant parameter ω (0<ω≤1) is added into Equation (5) as

( )

( )( ) 1 min( min ( ))c

cc y x

I yt xA

ω∈Ω

= − . (6)

This parameter stands for the aerial perspective which means that the presence of haze facilitates human to perceive depth, and if haze is completely removed, the image would seem unnatural and lost the feeling of depth.

Using Equation (6), we can estimated the transmission map from the haze image. Fig. 3 shows one of the haze images and its estimated transmission map. We can see that there are some block effects due to the use of patches and our assumption that transmission in a local patch is constant. This effect is solved with soft matting method proposed by Levin [5]. The refined transmission map after soft matting is also shown in Fig. 3. As can be seen, the refined transmission map is able to captures sharp edges and outline of object profiles.

C. Recovering the Scene Radiance With the transmission map, scene radiance can be easily recovered from Equation (1). The final scene

radiance J(x) is recovered using

0

( )( )max( ( ), )

xxt x t−

= +I AJ A , (7)

where t0 is a lower bound of t(x), which means that a small amount of haze are preserved in very dense

4


haze regions. An example of recovered haze-free scene radiance is shown in Fig. 3.

D. Estimating the Atmospheric Light Previous steps assumes that atmospheric light is known, which is not true in most cases. However,

with the haze image, the atmospheric light A can be easily estimated from the most haze-opaque pixel. Instead of directly taking the brightest pixel in the image as the atmospheric light, we pick the top 0.1% brightest pixels in the dark channel, and among them, select the pixel with highest intensity in the input image as the atmospheric light. Such pixel may not necessarily be the brightest pixel in the image because the brightest pixel could be a white object. For example, as shown in Fig. 4, the yellow line indicates the region of brightest pixels in the dark channel, from which the final atmospheric light is selected as indicated by the red square. The blue squares are two brighter pixels in the input image which turn out to be a white car and a white building.

E. An Alternative to Soft Matting: Guided Filter Guided filter [6] is an image filter that computes the filtered output by considering the content of a

guidance image, which can be the input image itself or another different image. Since it can transfer the structures of the guidance image to the filtering output, we can apply it to image dehazing to replace the computational expensive soft matting method. By setting the raw transmission map as image to be filtered, and the haze image as guidance image, a similar refinement to the transmission as sift matting can be achieved, and guided filter is much more computational efficient. Fig. 5 shows comparison of guided filter output and soft matting output. From the figure, it can be seen that both methods have similar results, with soft matting performs a little better for the sky region (as shown in “Tiananmen.jpg”), and guided filter has a little advantage in dense haze area (as shown in “Forest.jpg”). Table I shows a comparison of soft matting and guided filter in terms of running time in MATLAB for these three images (“Forest.jpg”, “Tiananmen.jpg”, “Sweden.jpg”). The machine used for tests has Intel® Core™ i7-4700MQ CPU (2.40GHz).

TABLE I. RUNNING TIME OF SOFT MATTING AND GUIDED FILTER

Image Soft matting Guided Filter

Forest.jpg 6.963 sec 1.743 sec

Tiananmen.jpg 7.037 sec 1.792 sec

Sweden.jpg 6.230 sec 1.640 sec

5


Forest.jpg Transmission (soft matting) Recovered radiance (soft matting)

Transmission (guided filter) Recovered radiance (guided filter)

Tiananmen.jpg Transmission (soft matting) Recovered radiance (soft matting)


Sweden.jpg Transmission (soft matting) Recovered radiance (soft matting)


Fig. 5. Comparison of transmission and recovered scene radiance between soft matting and guided filter

6


Fig. 6. Rentinex theory decompose given image S

Forest.jpg Untouched Transmission Adjusted Transmission Soft matting

Toys.jpg Untouched Transmission Adjusted Transmission Soft matting

Sweden.jpg Untouched Transmission Adjusted Transmission Soft matting

Fig. 7. Comparison of transmission before and after adjustment

4. Estimate Transmission Map Using Multi-Scale Retinex

A. Retinex Theory Retinex theory can be used on evaluating the illumination effects in the images. The main idea of this

theory lies on decomposing a given image into two different images which is motivated by the research on human visual system. In Fig. 6, S represents the image received by human eyes, the Retinex theory states that the image can be decomposed into an illumination image L, and a reflectance image R. Therefore, for each point in the image, the following function can be derived:

( , ) ( , ) ( , )S x y R x y L x y= . This can be further transformed into logarithm form as following:

log ( , ) log ( , ) log ( , )S x y R x y L x y= + . From this, the following equation can be derived:

log ( , ) log ( , ) log ( , )R x y S x y L x y= − . The goal of applying Retinex theory is to provide an alternative way to the transmission map in the

7


original method (see Section 3B). The essential part of this algorithm is to eliminate the effect of L which is defined as the luminance component of the image.

B. Transition Map Estimation Based on MSR The original method used soft matting to do estimate the transmission map and recover the scene

radiance which proved to be taken some time to get the result. In this alternative method, the YCbCr color space is used in combination with Retinex theory to estimate the transmission map. This color space is chosen specifically to increase the speed of processing, as other color space, such as RGB will cost more time. Note that only the Y component of the YCbCr space is used as it represents the luminance of the image and fit the algorithm well. The specific step of the algorithm is as follows.

First, we need transform a RGB image to YCbCr, this can be done through MATLAB easily, and we will be ready to separate the luminance component of this space which is Y. Then we can apply the MSR algorithm to the luminance component, this is done through the following equation:

1( , ) (log ( , ) log( ( , )* ( , )))

N

m n nn

R x y W Y x y F x y Y x y=

= −∑ .

In this equation, R means the transmission map that we expect to get, Wn means the weight of each scale where n is the number of scale. More specifically, we chose 3 scales and each of the have the same weight of 1/3. Y(x, y) is a point in the luminance component whose coordinate is x and y. This is what we separate from the YCbCr space. Finally, Fn(x, y) means the nth surrounding function which will be detailed later and “*” is the convolution operator.

The surrounding equation that was used here is the classic Gaussian function which can be expressed as following:

2 2

2( )

( , )x y

cF x y Ke− +

= . Here, c is used as the scale parameter for the Gaussian function. The value of c determines how the dynamic range is compressed as well as the detail in the image. The larger the value of c, the natural the color and the overall visual effect will be. For the purpose of this implementation, we chose 15, 80, and 250 as the value of c. K is a normalization factor that satisfies the following equation:

( , ) 1F x y dxdy =∫∫ .

C. Adjustment After the initial steps of applying the MSR to the luminance component, we found that the remission

map we got was not similar to the refined transmission map in He’s method. Therefore we applied the following steps after MSR is applied: 1) subtract each of the pixel in the remission map with an adjusting parameter which we set to 1.08; 2) Apply a median filter to the transmission map where the window size is selected to be 5 * 5; 3) Scale the pixel in the image to the range of 0-1 as He’s transmission map is actually a percentage of the luminance and that some of the pixels in our transmission map were not within this range. Fig. 7 shows a comparison of the transmission map we get before adjustment and after the adjustment.

It can be seen that the untouched map is quite different from the result of soft matting obtained by He’s method. After the adjustment, the result is similar. Some noticeable difference is that the result obtained by Retinex have higher brightness than the result of soft matting. Therefore, when the adjustment is done, the transmission map can used to directly to replace the refined transmission map in He’s algorithm to recover the scene radiance, a comparison of the result will be shown in Section 5.

8


Forest.jpg Adjusted Transmission Soft matting

Toys.jpg Adjusted Transmission Soft matting

Sweden.jpg Adjusted Transmission Soft matting

Fig. 8. Transmission of using HSV color space

D. Other Methods for Estimating Transmission Map

Another similar method to estimate the transmission map would be using the HSV color space. For this method, we just convert the RGB image to HSV color space and then apply the same MSR and adjustment mentioned above to the V component which represents brightness. The obtained results is shown in Fig. 8.

Similar to using YCrCb color space, the brightness of the transmission map obtained by using MSR on V component of HSV color space is higher than the soft matting result.

5. Results Comparison Having implemented method proposed by He and Xie using MATLAB, a quantitative comparison

was done on a PC with Intel® Core™ i5-4570 (3.2GHz) and 7.6GB RAM (different from Table I). Table II shows the speed of each of the methods, and Fig. 9 shows the comparison of visual results of different methods.

TABLE II. RUNNING TIME OF DIFFERENT METHODS

Image He’s Method (Soft matting)

He’s Method (Guided Filter)

Xie’s method (MSR on Luminance)

Variation on Xie’s Method (MSR on Value)

Toys.jpg 40.301 sec 4.967 sec 4.359 sec 2.069 sec

Sweden.jpg 42.254 sec 5.936 sec 3.671 sec 4.202 sec

Forest.jpg 44.068 sec 6.498 sec 4.168 sec 2.749 sec

9


Forest.jpg He’s Result MSR on Y of YCbCr MSR on V of HSV

Toys.jpg He’s Result MSR on Y of YCbCr MSR on V of HSV

Sweden.jpg He’s Result MSR on Y of YCbCr MSR on V of HSV

Fig. 9. Recovering results of different methods

From the above comparison, it can be seen that most of the MSR on V of HSV color space will have

the best performance. The result of applying MSR on luminance component of YCbCr color space proposed by Xie et al have better contrast and clarity than the result obtained by He’s method in Swenden.jpg and Forest.jpg. While in Toys.jpg, He’s method yield a more vivid color than the other two methods.

Using MATLAB’s built-in function, we were also able to compare the histogram of each of the image. It is shown as in Fig.10. It can be seen from the above comparison that He’s method yields histograms with higher dynamic range while results obtained from MSR algorithm have a compressed dynamic range.

6. Conclusion

In this paper, we implemented and compared serval methods of haze removal based on the dark channel prior. The original method proposed by He et al were slow in that it is computationally expensive to do the soft matting. Therefore, we proposed several ways to bypass this. The first method is to use guided filter instead of soft matting. This method yield similar result as He’s, however, soft matting method have better performance in the sky region than guided filter. The second method we implement was the Multi-scale Retinex algorithm proposed by Xie et al. Thanks to the use of luminance component and MSR algorithm, we were able to estimate the transmission map in He’s method without using soft matting or guided filter. This method costs much less time than the previous methods. However, the recovered images were not always better. Lastly, we also proposed a variation on Xie’s method, that is, use the V component of HSV color space to estimate the transmission map. Compared to Xie’s method, this method yields slightly performance improvement.

10


The limitation for dark channel prior and all the above mentioned methods is that the performance is still not ideal for real-time processing. When trying to use Xie’s method to do a real-time processing on a video of 3 seconds in length, the time taken was around 5 minutes. Therefore, future research on real-time processing method for video dehazing is still desirable.

11


Forest.jpg Original Histogram He’s Result

MSR on Y of YCbCr MSR on V of HSV

Toys.jpg Original Histogram He’s Result


Sweden.jpg Original Histogram He’s Result


Fig. 10. Histograms of different methods

12


References [1] K. He, J. Sun, and X. Tang, “Single Image Haze Removal using Dark Channel Prior,” CVPR, 2009, pp.1956-1963. [2] B. Xie, G. Fan, and Z. Cai, “Improved Single Image Dehazing Using Dark Channel Prior and Multi-scale Retinex,” in

Proceedings of International Conference on Intelligent System Design and Engineering Application, vol. 1, 2010, pp. 848–851

[3] J. Tarel, and N. Hautiere, “Fast Visibility Restoration from a Single Color or Gray Level Image, ” ICCV, 2009 [4] E. J. McCartney, Optics of Atmosphere: Scattering by Molecules and Particles[M]. New York: John Wiley and Sons, 1976,

pp.23-32. [5] A. Levin, D. Lischinski, and Y. Weiss. A closed form solution to natural image matting. CVPR, 1:61–68, 2006. [6] K. He, J. Sun, and X. Tang. Guided image filtering. Pattern Analysis and Machine Intelligence, IEEE Transactions on 35,

no. 6 (2013): 1397-1409. [7] W. Wang and L. Xu, “Retinex Algorithm on Changing Scales for Haze Removal with Depth Map,” International Journal of

Hybrid Information Technology, vol. 7, no. 4, pp. 353–364, Jul. 2014. Summary of Work Done:

Shiwei Zhou wrote the part that implements haze removal using dark channel prior and the guided filter alternative for refining transmission (about 120 lines of codes). The code for computing Matting Laplacian matrix L is taken from Levin’s webpage for [5]. The guided filter code is from He’s webpage for [6].

Sicong Jiang wrote the part that implements the multi-scale retinex algorithm. Approximately 150 lines of code were written. Part of the single scale retinex code was taken from “The INface toolbox v2.0 for illumination invariant face recognition” written by Vitomir Struc, it was vectorized and modified to fit the need of this project.

13

comparison between different implementation s of dark

Documents