ONLINE SUNFLICKER REMOVAL USING DYNAMIC TEXTUREPREDICTION

ASM Shihavuddin, Nuno Gracias and Rafael GarciaComputer Vision and Robotics Group,Universitat de Girona, Girona, Spain

{asm.shihavuddin, ngracias, rafael.garcia}@udg.edu

Keywords: SUNFLICKER; DYNAMIC TEXTURE; ILLUMINATION FIELD; SHALLOW UNDERWATER IMAGES

Abstract: An underwater vision system operating in shallow water faces unique challenges, which often degrade thequality of the acquired data. One of these challenges is the sunflicker effect, created from refracted sunlightcasting fast moving patterns on the seafloor. Surprisingly few previous works exist to address this topic. Thebest performing available method mitigates the sunflickering effect using offline motion compensated filtering.In the present work, we propose an online sunflicker removal method targeted at producing better registrationaccuracy. The illumination field of the sunflicker effect is considered as a dynamic texture, since it producesrepetitive dynamic patterns. With that assumption, the dynamic model of the sunflicker is learned from theregistered illumination fields of the previous frames and is used for predicting that of the next coming frame.Such prediction allows for removing the sunflicker patterns from the new frame and successfully register itagainst previous frames. Comparative results are presented using challenging test sequences which illustratethe better performance of the approach against the closest related method in the literature.

1 INTRODUCTION

In the field of underwater image sensing, new andcomplex challenges need to be addressed like lightscattering, sunflicker effects, color shifts, shape dis-tortion, visibility degradation, blurring effects andmany others. The research work presented in this pa-per deals with the presence of strong light fluctuationsdue to refraction, commonly found in shallow under-water imaging. Refracted sunlight generates dynamicpatterns, which degrade the image quality and the in-formation content of the acquired data. The sunflick-ering effect creates a difficult challenge to the scien-tists to understand and to interpret the benthos. Devel-opment of an online method to completely or partiallyeliminate this effect is a prerequisite to ensure optimalperformance of underwater imaging algorithms.

2 RELATED WORK

Motivated by the work by on effective shadow re-moval method (Weiss, 2001) in land scenes and thework on shadow elimination (Matsushita et al., 2002)on video surveillance, Schechner and Karpel devel-oped an approach (Schechner and Karpel, 2004a) tosolve the sunflicker removal problem based on the

observation that the spatial intensity gradients of thecaustics tend to be sparse. Under that assumption,doing the temporal median over the gradients of asmall number of images, a gradient field was ob-tained where the illumination effect was greatly re-duced. The flicker free image was obtained by inte-grating the median gradient field. This sunflicker re-moval method (Schechner and Karpel, 2004a) did nottake into consideration the camera motion for whichregistration inaccuracies are likely to appear. Us-ing the method by Sarel and Irani (Sarel and Irani,2004; Ukrainitz and Irani, 2006) two transparent over-lapped video can be separated using the informationexchange theory, if one of the video mentioned con-tains a repetitive dynamic sequence. This method(Sarel and Irani, 2004) claims to work for the repeti-tive dynamic sequence with variation in each repeatedcycle, though the variation amount in the repetitivedynamic sequence is not quantitatively defined. Onthe other hand, a large number of frames would berequired to grab a complete cycle of a repetitive dy-namic sequence, making it impractical for movingcameras. An interesting approach to this problemcan be to use polarization information (Schechner andKarpel, 2004b), to improve visibility underwater andin deep space. The refracted sunlight underwater hasunique polarization characteristics. The exploitation

of such characteristics is possible, but requires spe-cial cameras with polarized filters or imaging sensors,which are not commonly deployed. Another way oflooking in to the same problem can be to recover thelow rank matrix (Candes et al., 2009) which presentsthe illumination field. But this method only workswhen there is no camera motion which in our case isimpractical.

The work by Gracias et al. (Gracias et al., 2008)can be considered as the state of art among the meth-ods that have addressed the removal of sunflicker inshallow underwater images. This motion compen-sated filtering approach accounted for the basic re-quirement that the camera should be allowed to moveduring acquisition, thus being more adequate to realoptical survey work. The method is based on the as-sumption that video sequences allow several observa-tions of the same area of the seafloor, over time. If theassumption is valid, it is possible to compute the im-age difference between a given reference frame andthe temporal median of a registered set of neighbor-ing images. The most important observation is thatthis difference will have two components with sepa-rable spectral content. One is related to the illumi-nation field (which has lower spatial frequencies) andthe other to the registration inaccuracies (mainly hav-ing higher frequencies). The illumination field can beapproximately recovered by using low–pass filtering.The main limitation is that the median image for eachframe is obtained from both past and future frames,thus making it non causal.

3 APPROACH

The presented method uses dynamic texture modelingand synthesizing (Doretto et al., 2003), using princi-pal component analysis, to predict the sunflicker pat-tern of the next frame from the previous few frames.The prediction is then used to coarsely remove the il-lumination field from the current working frame. Thepresented approach attains higher registration per-formance even under heavy illumination fluctuation.Also this method is strictly causal (i.e. does not relyon future observations), fulfilling the important con-dition of online operation required for visual-basedrobot navigation.

3.1 Dynamic Texture Modeling andSynthesizing

The Open Loop Linear Dynamic system (OLLDS)model (Doretto et al., 2003) is used to learn the dy-namic model of sunflicker illumination pattern and to

synthesize it for unknown cases. Once modeling isdone, the OLLDS can potentially be used for extrap-olating synthetic sequences of any duration at negli-gible computational cost. The underlying assumptionin this approach is that, the individual images are real-izations of the output of a dynamical system driven byan independent and identically distributed (IID) pro-cess.

For learning the model parameters, the OLLDSmodel uses one of two criteria: total likelihood or pre-diction error. Under the hypothesis of second-orderstationarity, a closed-form sub-optimal solution of thelearning problem can be obtained as follows (Dorettoet al., 2003).

1. A linear dynamic texture is modeled as an auto-regressive moving average process (ARMA) with un-known input distribution, in the form,

x(t +1) = Ax(t)+Bv(t) v(t)≈ N(0,Q) x(0) = xoy(t) =Cx(t)+w(t) w(t)≈ N(0,R)

where y(t) belongs to Rn is the observation vector; x(t)belonging to Rr, is the hidden state vector (r is muchsmaller than n); A is the system matrix; C is the outputmatrix and v(t), w(t) are Gaussian white noises. Herey(t) presents the noisy output, in this case the imagesequences.

2. Taking the SVD of y(t), C can be found.

Y τ1 =UΣV T

C(τ) =U X(τ) = ΣV T

3. A can be determined uniquely by,

A(τ) = ΣV T D1V (V T D2V )−1Σ−1

Where

D1 =

[0 0

Iτ−1 0

]D2 =

[Iτ−1 0

0 0

]Also, Q and B can be found by

Q(τ) =1

1− τ

τ−1

∑i=1

v(i)vT (i)

BBT = Q

3.2 Motion Compensated Filtering

Let us consider a set of registered images. We referto a given image by the discrete parameter i whichindexes the images temporally. The radiance L of agiven pixel with coordinates (x, y) can be modelledas

Li(x,y) = Ei(x,y) ·Ri(x,y)

where Ei is the irradiance of the sunlight over the 3Dscene at the location defined by pixel (x, y) at time i,after absorption in the water. R(x, y) is the bidirec-tional reflectance distribution function. For underwa-ter natural scenes, where diffuse reflectance modelsare applicable, R is assumed independent of both lightand view directions.

Converting the expression for Li to a logarithmicscale allows the use of linear filtering over the illumi-nation and reflectance.

li(x,y) = ei(x,y)+ ri(x,y)

For approximately constant water depth, and forrealistic finite cases, the median converges signifi-cantly faster to average value of li than the samplemean.

Imed(x,y) = med[i0,i1]Ii(x,y)≈ e+ rmed(x,y)

Here rmed(x, y) stands for an approximation to themedian of reflectance. The difference di(x, y) of agiven image li(x, y) with the median radiance lmed(x,y) is used to recover the approximate background im-age.

dli(x,y) = li(x,y)− Imed(x,y)

≈ (ei(x,y)− e)+(ri(x,y)− rmed(x,y))

This difference dli (x,y) has two main components.The first component relates to the instant fluctuationof the illumination field and has lower spatial frequen-cies. This component will have positive values in theover-illuminated areas where there is convergence ofthe refracted sunlight and will have negative valuesin the areas where the sunlight is diverted away. Thesecond component relates to inaccuracies in the imageregistration and has higher spatial frequencies. Af-ter applying a low pass filter on the difference image,the low frequency components which resembles theillumination field is kept in the output. This approx-imated illumination field is later used to correct themain input image and recover a flicker–free image.

4 PROPOSED ALGORITHM

In the online sunflicker removal method proposed inthis work, the approximate illumination field of thecurrent frame is found from the dynamic texture pre-diction step. The homography is mainly calculatedbetween the temporary recovered current image (us-ing the approximate illumination of the current frame)

and the last recovered image. Finally, the new imagecan be recovered by using a median image calculatedfrom the current and previous few images warpedwith respect to the current image using the calculatedhomography. The median image and the current im-ages can be used to find the difference image for thecurrent frame. The difference images after being fil-tered through a low pass filter only holds the low fre-quency components which represents the correct illu-mination field. Using this correct illumination fieldthe working image is recovered from the sunflickereffect. The steps are demonstrated as in fig. 1

Figure 1: Step by step flow diagram of the proposed onlinemethod for sunflicker removal. From left to right, (1)warp-ing previous illumination field to the current frame, (2)pre-dicting the current illumination field, (3)coarsely recover-ing the current image, (4)finding homography between thecurrent and the previous frame and (5)removing sunflickerpattern from the image using the calculated homography

The proposed approach considers the followingmentioned assumption to be valid

• Illumination field is a dynamic texture

• Camera movement in the video sequence is smooth

• Bottom of the sea is approximately flat

The main algorithm contains several steps whichare presented next, using the following nomenclature:

• I0,k - Original input image obtained at time instant k

• IR,k - Recovered image obtained from I0,k after sun-flicker removal

• Hk - low pass filtered version of the difference imageat time k, which is used as estimate of the illuminationfield at time k

• IM,k - Median image obtained from N frames after be-ing warped into the frame of image I0,k

• N - the number of images present in the learning se-quence.

• Mk,k−1 - Homography relating the image frames at timek and k−1.

The algorithm comprises the following steps:

1. Apply the motion compensated filteringmethod (Gracias et al., 2008) to register and recoverfirst few images from the sun flickering effects. In ourimplemented system, we used first the 25 frames forthis step.

2. Get the new image in the sequence I0,k assum-ing that the previous images I0,k−1, . . . , I0,k−N havebeen recovered after sunflicker removal (IR,k−1, . . . ,IR,k−N). Advance time k (i.e. all previous data struc-tures that had index k now have index k−1).

3. Predict the flicker pattern by

• Warping all the filtered version of the difference im-ages, Hk−1, . . . , Hk−N with respect to the current frameI0,k to be recovered , assuming that Mk,k−1 ≈Mk−1,k−2.All other previous homographies were obtained fromactual image matches and thus known previously.

• Learning the sunflicker pattern from the registered fil-tered difference images. After registration, the motionof the camera is being compensated. In the learningphase, all the difference images of the previous framesH (Hk−1, . . . , Hk−N ) are converted into a column ma-trix. Using the array of all the registered H (Hk−1, . . . ,Hk−N ) a large matrix called Wt−1 is created having Prows and N columns. P is the number of pixels perframe and N is the total number of frames in the learn-ing sequence.

• Predicting the Hk using the learned model. For learn-ing, open loop linear dynamic model (Doretto et al.,2003) is being used. In this step the last frame Hk−1of the learned sequence is considered as the first framefor the synthesize part.

4. Create the correction factor Ck using the pre-dicted low pass filtered version of the difference im-age Hk and the approximate difference image Id,k. Tofind the approximate difference image, the previouslyrecovered image IR,K−1 is warped into the currentframe I0,k position using the last homography. Usingthe warped portion of the previous recovered imageand the rest from current original image, a approxi-mate median image is created, which is fused into thesystem to find the approximate difference image, Id,kof the current frame. Using this approximated differ-ence image Id,k and the predicted Hk, the correctionfactor Ck is found.

5. Apply predicted correction factor to I0,k. Us-ing the correction factor Ck found in the last step, thecurrent image is approximately recovered from sun-flicker effect. This recovered image is denoted by IR,k.

6. Perform image registration between IR,k andIR,k−1. From this obtain the real Mk,k−1.

7. Update IM,k. Using the motion compensatedfiltering method (Gracias et al., 2008) create a medianimage for the current frame using the last few originalframes. In this case, use the IR,k to do the registrationduring finding the current median image, IM,k.

8. Obtain final IR,k. Using IM,k find the real differ-ence image for the current frame, Id,k and the correctsunflicker pattern Hk and later the correct recover im-age IR,k removing the sunlight properly.

9. Go to step 2 and do the same for the next framesThe image registration is performed using the

classic approach of robust model based estimationusing Harris corner detection (Harris and Stephens,1988) and normalized cross–correlation (Zhao et al.,2006). This method proved more resilient to strongillumination differences when compared with the re-sult doing the same with SIFT (Lowe, 2004). Further-more, the operation is considerably faster since thesearch areas are small.

We assume the knowledge of the gamma valuesfor each color channel. For unknown gamma val-ues one can apply blind gamma estimation. An ef-ficient method is described in (Farid, 2001; Farid andPopescu, 2001), which exploits the fact that gammacorrection introduces specific higher-order correla-tions in the frequency domain. Having the gammavalues we transform the intensities to linear scale.After the deflickering the final output images aretransformed into the sRGB space with the prescribedgamma value.

The steps above are applied over each color chan-nel independently. Strong caustics lead to overex-posure and intensity clipping in one or more of thecolor channels, resulting in chromaticity changes inthe original images. These clippings typically affectdifferent regions of the images over time, given thenon-stationary nature of the caustics. The median isnot affected by such transient clippings, whereas theaverage is. The low pass filtering is performed usinga fourth order Butterworth filter (Kovesi, 2009), witha manually adjusted cutoff frequency.

Due to the camera motion, the stack of warped dif-ference images described in step 3, may not cover theentire area of the current frame. If one considers thewhole area of the current frame, this creates a condi-tion of missing data in the W matrix for PCA. To cir-cumvent this condition, only the maximum sized areaof the current frame which is present in each warpedframe at current frame location, is considered.

5 SELECTED RESULTS

The performance evaluation of the proposed systemwas done comparing with the previously available of-fline method (Gracias et al., 2008) by evaluating bothon several test datasets of shallow water video se-quences having distinct refracted sunlight conditions.The main evaluation criterion is the number of inliers

found per time–consecutive image pair in each reg-istration step. This criterion was found to be a goodindicator the image de–flickering performance. Thebetter the sunflicker removal is achieved, the largerthe number of found inliers per time–consecutive im-age pair, considering all the other influencing factorsconstant.

5.1 Grounding Sequence

The first test sequence was obtained from a shipgrounding survey and contains illumination patternsof relatively low spatial frequency. Fig. 2 shows anexample of how in each step the image is coarsely re-covered, registered and finally cleaned. The top leftand bottom left parts of the image 2 show the originalinput image and real illumination field accordingly.Using this initial estimate, the warped illuminationfield is learnt and predicted for the current frame; thenew image is temporarily cleaned with the predictionof illumination field and registered with respect to thelast recovered image. In the bottom middle of Fig. 2,the predicted illumination field is shown and the uppermiddle part of the Fig. 2 represents the intermediatecondition of the illumination field. The prediction il-lumination fields approximates the real illuminationfield with some spatial displacements and intensityvariation. On the top right of Fig 2 shows the finalrecovered image and the bottom right of Fig 2 showsthe final median image. Some examples of the finallyrecovered images for difference video sequences aregiven in Figs. 3(b), 3(d) and 3(f).

Figure 2: Step by step results for the grounding sequence.The top left is the original input image and bottom left isthe original illumination field in the input image. In thetop middle, the image is of the temporary recovered im-age, IR,k and the bottom middle is the predicted illumina-tion field. This was done using the predicted illuminationfield as shown in the bottom middle. On the top right is thefinally recovered image, IR,k and on the bottom right is thefinal median image

In Fig. 4 the proposed online method and the ex-isting offline method (Gracias et al., 2008) are beingcompared in terms of the number of inliers during the

(a) Original Image (b) Recovered Image

(c) Original Image (d) Recovered Image

(e) Original Image (f) Recovered ImageFigure 3: Illustration of the online sunflicker removal per-formance

registration step. From the graph, it can be seen thatthe new method is outperforming the old method inalmost all the frames. Quantitatively, matching per-formance is improved by 46% based on the inliers de-tected in every pair of frames. Without prediction, theregistration or the number of found inliers are vary-ing very rapidly because of complete dependency onthe camera motion. In the case of prediction, it in-cludes the displacement errors creating a generalizedsolution which provides approximately constant out-comes.

Figure 4: Comparison between proposed method and theoffline method for grounding sequence

5.2 Rock Sequence

The Rock sequence is more challenging than thegrounding sequence, captured in shallow waters hav-ing a very rocky bottom. Both the methods are be-ing tested for sunflicker removal keeping other pa-rameters unchanged. The proposed online methodis also performing significantly better than the offlinemethod in terms of found inliers per registration stepsas shown in Fig. 5. Overall gain in the matching per-formance in the proposed method is about 67% com-pared with the offline method.

Figure 5: Comparison between proposed method and theoffline method for Rock sequence

5.3 Andros Sequence

This is the most challenging sequence of the three,captured in very shallow waters of less than 2 metersdepth under intense sunlight. In this case, the illumi-nation patterns have simultaneously very high spatialand temporal frequencies which leads to the methodof (Gracias et al., 2008) to fail. In such situation it isvery hard to get a good registration performance. Inthe Fig. 6, the graph shows that the proposed methodperforms 15% better. This performance is achievedbecause of the steady motion of the camera in this par-ticular sequence. It proves the robustness of the onlinemethod achieving better results even in such a difficultvideo sequence. In case of large registration errors,the prediction model performance degrades rapidly.

6 CONCLUSION AND FUTUREDIRECTION

This work addresses the specific problem of sun-flicker in shallow underwater images, by presentinga method which is suited for de–flickering on the fly.In the current online method, only the previous fewframes with the current frame are used to create me-dian image. In this case, the homography is calculated

Figure 6: Comparison between proposed method and theoffline method for Andros sequence

by registering the sunflicker removed version (usingprediction from dynamic texture model learned fromthe last few frames) of current image with the lastflicker–free image. This results in higher image reg-istration accuracy than the offline method where theregistration is carried out over the original images af-fected by the illumination patterns. The better regis-tration then reflects in better median images estima-tion and ultimately in better sunflickering correction.

This research was motivated by the fact that theflicker–free images will help creating more accuratemosaics of the sea floor. Currently the method is im-plemented in Matlab, and the code has not been op-timized for speed. It takes 6.39 seconds per frame,on average, when executed on an Intel core 2 Duo 2GHz processor. However, being an on-line approach,the method has the potential to be implemented forreal–time operation.

An extension to the current work is to relate theillumination frequency with the number of frames re-quired to perform sunflicker removal optimally. It canbe a way to know beforehand the minimum frame rateof the camera required to remove the sunflicker effectproperly. Also, for instrumented imaging platforms,the camera motion can be estimated using a motionsensor, such as a rate gyro or an accelerometer. Thisestimate can be used during the initialization phase ofthe method, or whenever the image registration is notpossible.

Another extension addresses highly slanted cam-era configuration, where the illumination patternspresent distinct spatio-temporal frequencies for dif-ferent regions of the image. In the described method,the cutoff frequency of the the low-pass filter is con-stant. As future work, this cutoff frequency will bemade adaptive to the spatial location and estimatedfrom image data.

Acknowledgments

This work was partially funded by the SpanishMCINN under grant CTM2010-15216, and by theEU Project FP7-ICT-2009-248497. ASM Shihavud-din and Nuno Gracias were supported by the MCINNunder the FI and Ramon y Cajal programs.

REFERENCES

Candes, E. J., Li, X., Ma, Y., and Wright, J. (2009). Robustprincipal component analysis? CoRR, abs/0912.3599.

Doretto, G., Chiuso, A., Wu, Y. N., and Soatto, S. (2003).Dynamic textures. International Journal of ComputerVision, 51:91–109.

Farid, H. (2001). Blind inverse gamma correction. Im-age Processing, IEEE Transactions on, 10(10):1428–1433.

Farid, H. and Popescu, A. C. (2001). Blind removal of im-age non-linearities. Computer Vision, IEEE Interna-tional Conference on, 1:76.

Gracias, N., Negahdaripour, S., Neumann, L., Prados, R.,and Garcia, R. (2008). A motion compensated filter-ing approach to remove sunlight flicker in shallow wa-ter images. In OCEANS 2008.

Harris, C. and Stephens, M. (1988). A combined cornerand edge detector. In Fourth Alvey Vision Conference,Manchester, UK, pages 147 –151.

Kovesi, P. D. (2009). Matlab and octave functions for com-puter vision and image processing. School of Com-puter Science Software Engineering.

Lowe, D. G. (2004). Distinctive image features from scale-invariant keypoints. International Journal of Com-puter Vision, 60:91–110.

Matsushita, Y., Nishino, K., Ikeuchi, K., and Sakauchi, M.(2002). Shadow elimination for robust video surveil-lance. Motion and Video Computing, IEEE Workshopon, page 15.

Sarel, B. and Irani, M. (2004). Separating transparent lay-ers through layer information exchange. In Pajdla, T.and Matas, J., editors, Computer Vision - ECCV 2004,volume 3024 of Lecture Notes in Computer Science,pages 328–341. Springer Berlin / Heidelberg.

Schechner, Y. and Karpel, N. (2004a). Attenuating natu-ral flicker patterns. In IEEE TECHNO-OCEAN, 04,volume 3, pages 1262 –1268.

Schechner, Y. and Karpel, N. (2004b). Recoveringscenes by polarization analysis. In OCEANS ’04.MTTS/IEEE TECHNO-OCEAN ’04, volume 3, pages1255 –1261 Vol.3.

Ukrainitz, Y. and Irani, M. (2006). Aligning sequencesand actions by maximizing space-time correlations.In Leonardis, A., Bischof, H., and Pinz, A., edi-tors, Computer Vision ECCV 2006, volume 3953 ofLecture Notes in Computer Science, pages 538–550.Springer Berlin / Heidelberg.

Weiss, Y. (2001). Deriving intrinsic images from image se-quences. In Computer Vision, 2001. ICCV 2001. Pro-ceedings. Eighth IEEE International Conference on,volume 2, pages 68 –75.

Zhao, F., Huang, Q., and Gao, W. (2006). Image matchingby normalized cross-correlation. In Acoustics, Speechand Signal Processing, 2006. ICASSP 2006 Proceed-ings. 2006 IEEE International Conference on, vol-ume 2.