BIDIRECTIONAL BIAS CORRECTION FOR GRADIENT-BASED SHIFT ESTIMATION

Tuan Q. Pham Matthew Duggan

Canon Information Systems Research Australia (CiSRA)1 Thomas Holt drive, North Ryde, NSW 2113, Australia

http://www.cisra.com.au

ABSTRACT

This paper characterizes the bias of a Gradient-Based ShiftEstimator (GBSE) and proposes a novel scheme to correct forthe bias. The bias of GBSE comes from an inaccurate es-timate of the image gradient energy, which is due to noise,aliasing and built-in low-pass filtering. For subpixel shift,the bias is linearly proportional to the true shift. The linearbias factor can be blindly estimated using two or more sub-pixel shift estimates of the same image pair, each estimate isperformed on a slightly shifted version of the second image.Compared to the original GBSE, the new method reduces thebias by half at the same time improving the precision.

Index Terms— image registration, gradient-based shiftestimation, bidirectional bias correction, optical flow.

1. INTRODUCTION

Shift estimation is a basic building block in many com-puter vision systems. The task is to estimate an apparenttranslational motion between two images of a moving sceneor two images captured by a moving camera. There arevarious methods to handle this task, for example, cross-correlation, phase correlation, and Gradient-Based Shift Esti-mation (GBSE). GBSE [1], a.k.a. differential-based methodor optic flow, is an efficient spatial-domain method for sub-pixel shift estimation. However, GBSE has been shown tobe biased [2, 3, 4]. The bias is due to a truncation of theTaylor series approximation within the algorithm as well asnoise [5]. This paper estimates the bias of GBSE directlyfrom the input images and hence produces a bias-reducedshift estimation.

Previous attempts to reduce the bias of GBSE mainly fo-cus on the design of gradient filters. Because finite gradi-ents are more accurate at low frequencies, the image is oftenpre-filtered prior to differentiation. Simoncelli [6] designed ageneral-purpose gradient filter by matching it with the deriva-tive of the pre-filter. Elad et.al. [7] designed two pre-filtersand a gradient filter for GBSE by minimizing the error of Tay-lor series approximation. Robinson and Milanfar [8] designeda gradient filter to minimize the bias due to Taylor series ap-proximation. Both the latter methods use a prior knowledge

of the image content and the distribution of the motion vec-tors. However, none of the above methods considers bias dueto noise or aliasing. This explains why [8] fails to produce asatisfactory result for low Signal-to-Noise Ratio images (e.g.,SNR < 25dB).

To our knowledge, only iterative GBSE can reduce theestimation bias to almost zero [5]. The method successivelyshifts one image closer to the other using a previously esti-mated shift. Since the displacement between the two imagesgets smaller after each iteration, so is the bias. However, themethod requires multiple passes over the input images, eachwith an expensive motion compensation operation. It is there-fore not suitable for many real-time applications.

In the remaining part of this paper, we will show that thebias of GBSE mainly comes from a poor estimation of the im-age gradient. Whether it is due to image prefiltering or noiseor aliasing, this introduces an estimation bias linearly propor-tional to the true shift. Using at most four shift estimations onsingle-pixel shifted versions of the same image pair, the biascan be computed and compensated for.

2. GRADIENT-BASED SHIFT ESTIMATION

Gradient-based shift estimation, a.k.a. optic flow computa-tion [1], is a least-squares method to solve for a small shiftv between two images I1 and I2. The method makes use ofthe Taylor series expansion of a slightly shifted version of thereference image I1:

I2(x) = I1(x + v) = I1(x) + v∇I(x) + ε (1)

where x is the image coordinate, ∇I is the first-order deriva-tives of I1, and ε represents the sum of higher-order polyno-mial terms (ε is negligible for ||v|| < 1). The solution is ob-tained by minimizing an approximated Mean Squared Error(MSE) of the two images with respect to the unknown shiftv: ∑

ε2 =∑

(I2 − I1 − v∇I)2 (2)

where∑

denotes an average operator over the whole image.For 1-D signals, this yields a least-squares solution:

v =∑

ItI′

∑I ′2

(3)

where I ′ is the spatial gradient image and It = I2 − I1 is thetemporal gradient image.

For 2-D images, minimizing (2) yields a linear system:

Av = b (4)

where A =[ ∑

I2x

∑IxIy∑

IxIy

∑I2y

]is the second moment

matrix, b =[ ∑

ItIx∑ItIy

]is the spatio-temporal gradient cor-

relation vector, and v =[

vx

vy

]is the unknown shift. The

solution of this linear system is v = A−1b.

3. BIAS AND PRECISION OF GBSE

Due to Taylor series approximation and inaccurate gradientestimation, GBSE has a systematic error (i.e. bias) and astochastic variation about the biased value (i.e. nonzero preci-sion). Of all the mentioned bias sources, Taylor series approx-imation incurs the least bias, which is proportional to a cubicpower of the subpixel shift [5]. We therefore only model thebias due to inaccurate gradient estimation in this analysis.

3.1. 1-D bias factor model

The 1-D shift estimator in (3) is biased if the gradient I ′ iscomputed from a noisy image I1 = I + n1. This is becausethe denominator of the solution is overestimated:

E[∑

I ′2]

=∑

I ′2 + E[∑

n′2]

(5)

As a result, t is underestimated by a constant factor k = SS+N

where S = var(I ′) is the variance of the uncorrupted imagegradient and N = var(n′) is the variance of the noise gra-dient. Theoretically, S and N can be estimated from a noisyimage, hence the shift bias can be corrected [9]. In practice,however, this is a difficult task because noise is not alwaysGaussian nor is it uncorrelated with the signal. Furthermore,a noise-corrected value

∑I ′2 − var(nx) is not necessarily a

good approximation of the true gradient energy∑

I ′2 whena pre-filter is used or when the image is aliased or small.

Different from bias, which mainly comes from an incor-rect estimation of the solution’s denominator, precision is dueto stochastic variation in the nominator. Even when the twoimages are perfectly aligned, noise introduces a nonzero shiftestimation (3) because

∑ItI

′ =∑

(n2 − n1)(I ′ + n′) 6= 0.This gradient correlation term only reduces to zero for in-finitely large images.

3.2. 2-D bias matrix model

In 2-D, the diagonal entries of the second moment matrix Ais overestimated in the presence of noise:

E[A

]=

[ ∑I2x + var(nx)

∑IxIy + cov(nx, ny)∑

IxIy + cov(nx, ny)∑

I2y + var(ny)

]6=A,

but E[b] = b. Equating b and b, the estimated shift v islinearly proportional to the true shift v via a bias matrix:

Av = Av ⇒ v = A−1Av = Mv (6)

Similar to the bias factor in 1D, this bias matrix M is inde-pendent of the second image I2.

Denote A =[

a bb c

]and A =

[a + ∆a b + ∆b

b + ∆b c + ∆c

],

where ∆a, ∆b and ∆c are noise-induced offsets of the gra-dient auto-correlation and cross-correlation terms a, b and c,respectively, the bias matrix M can be rewritten as:

M=[

1 + (a∆c − b∆b)/∆ (b∆c − c∆b)/∆(−a∆b + b∆a)/∆ 1 + (c∆a − b∆b)/∆

](7)

where ∆ = (a+∆a)×(c+∆c)−(b+∆b)2 is the determinantof A. Note that the diagonal terms of M are close to oneand the off-diagonal terms are close to zero because a, c >>b and ∆a,∆c >> ∆b (b and ∆b are statistically zero forinfinitely large images). If the off-diagonal terms of M areapproximated by zeros, this 2-D bias matrix model reduces toa 1-D bias factor model for each shift component along thex− and y−directions.

4. BIDIRECTIONAL BIAS CORRECTION

Because both the bias factor in 1-D and the bias matrix in 2-D depend only on the reference image, all shift estimationsagainst a common reference image share the same bias fac-tor/matrix. We can therefore derive this bias factor/matrixdirectly from the input images using additional shift estima-tions. The additional shift is estimated between a shifted ver-sion of I2 by a known translation against a fixed referenceimage I1.

4.1. 1-D bias factor correction

GBSE assumes that the two input images I1 and I2 are alreadyaligned to one pixel accuracy. By shifting image I2 one pixeltowards image I1, the displacement from I1(x) to I2(x±1) isalso less than one (the sign of ±1 equals −sign(v0) where v0

is an initial shift estimate between I1 and I2). In the exampleillustrated by Figure 1, a plus sign is used because v0 ≤ 0.Because the same reference image I1(x) is used, the two es-timated shifts are biased by the same factor:

v0 = findshift(I1(x), I2(x)) = kv (8)v1 = findshift(I1(x), I2(x + 1)) = k(v + 1) (9)

Subtracting (8) from (9) resolves the bias factor:

k = v1 − v0 (10)

I1(x) = I2(x – v)

I2(x) Actual shifts I2(x + 1)

First shift estimation Second shift estimation

v v + 1

v0 = kv v1 = k(v + 1)

Fig. 1. Bidirectional 1-D shift bias factor estimation.

I1(x,y) = I2(x–vx, y–vy)

I2(x,y) I2(x+1,y+1)

v00 I2(x+1,y) I2(x,y+1)

v10 v11 v10

Fig. 2. Bidirectional 2-D shift bias matrix estimation.

4.2. 2-D bias matrix correction

Similar to the 1-D case, the 2-D bias matrix can be derivedfrom multiple subpixel shift estimates between I1 and shiftedversions of I2. As illustrated in Figure 2, four following sub-pixel shift estimates can be computed for the case of v00 < 0:

v00 = findshift(I1, I2(x, y)) = Mv (11)

v10 = findshift(I1, I2(x+1, y)) = M(v+[1 0]T) (12)

v01 = findshift(I1, I2(x, y+1)) = M(v+[0 1]T) (13)

v11 = findshift(I1, I2(x+1, y+1))=M(v+[1 1]T)(14)

Subtracting (11) from the above equations eliminates theunknown shift v:

v10 − v00 = M.[1 0]T (15)

v01 − v00 = M.[0 1]T (16)

v11 − v00 = M.[1 1]T (17)

It is clear that the bias matrix M can be resolved from anytwo of the equations above. As a result, a minimum of threesubpixel shift estimates are required to estimate M. However,if all four estimates (11-14) are available, M can be estimatedby least-squares from:

M.

[1 0 10 1 1

]=

[v10−v00 v01−v00 v11−v00

]

(18)

using Moore-Penrose pseudo-inverse. The bias-reduced shiftestimate is then computed as: v = M−1v00.

4.2.1. Weighted least-squares estimation

There are two problems with the estimation of the bias matrixin (18). First, the resulting second moment matrix A = A.Mis not symmetric. Second, all shift measurements vij aretreated equally despite their varying levels of accuracy. Be-cause GBSE estimates small shift more accurately, each shiftestimation should be weighted by its inverse magnitude.

Pre-multiplying A to (15-17) yields:

A.(v10 − v00) ≈ A.[1 0]T = [a b]T (19)

A.(v01 − v00) ≈ A.[0 1]T = [b c]T (20)

A.(v11 − v00) ≈ A.[1 1]T = [a + b b + c]T (21)

If the left-hand side of the above equations is assigned to[p1 q1]T, [p2 q2]T and [p3 q3]T, with associated weightsw1 = ‖|v10|+ |v00|‖−2, w2 = ‖|v01|+ |v00|‖−2 andw3 = ‖|v11|+ |v00|‖−2, respectively1, a weighted least-squares solution of [a b c]T aims to minimize:

ε =w1(a− p1)2 + w1(b− q1)2 + w2(b− p2)2 (22)

+w2(c− q2)2 + w3(a + b− p3)2 + w3(b + c− q3)2

Setting the derivatives of ε against a, b and c to zeros yields:

w1 + w3 w3 0w3 w1 + w2 + 2w3 w3

0 w3 w2 + w3

.

abc

=

w1p1 + w3p3

w1q1 + w2p2 + w3(p3 + q3)w2q2 + w3q3

, (23)

from which a solution [a b c]T can be derived.

5. EXPERIMENT

In this section, the performance of the bias-corrected GBSE insection 4.2.1 is compared against those of other GBSE’s. Fif-teen shift estimates are computed from a 90× 90× 16 noisyand highly aliased image sequence2 whose first frame is thereference. Since the ground-truth motion vectors are avail-able, the bias and precision of each estimator can be computedfrom the average and standard deviation of the estimation er-rors. Figure 3 plots these results in the form of a rectanglefor each shift estimator. Each rectangle is centered at the av-eraged error [∆vx, ∆vy] with its dimensions being twice thestandard deviation of the errors [std(∆vx), std(∆vy)]. Bydefinition, an estimator is less biased if its corresponding rect-angle is closer to the origin. The estimator is also precise ifthe area of the rectangle is small.

1|v| and ‖v‖ denote absolute value and norm of vector v, respectively2EIA sequence available at http://www.soe.ucsc.edu/∼milanfar/DataSets/

−0.2 −0.15 −0.1 −0.05 0 0.05−0.1

−0.05

0

0.05

0.1

0.15

∆tx ± std(∆t

x)

∆ty ±

std

(∆t y)

GBSE (1 iteration of [1])CLS (appendix E of [9])bidirectional (this paper)iterative GBSE [1]

Fig. 3. Performance of different shift estimators.

According to figure 3, the bidirectional bias-correctedGBSE is the least biased as well as most precise shift es-timator. The Corrected Least Squares (CLS) estimator [9]has a similar level of precision with the original GBSE inequation (4) because both are single-iteration GBSE. TheCLS method is slightly less biased than GBSE due to a noise-corrected second moment matrix as described in section 3.1.Surprisingly, the iterative GBSE [5] performs worst with thelargest bias and standard deviation. This is most likely due toa high level of aliasing in the test sequence, which prevents agood motion compensation during the iterative estimation.

6. CONCLUSION AND FUTURE RESEARCH

We have presented a general scheme to correct for the biasof an n−parameter gradient-based registration estimator us-ing at least n+1 estimations of the registration between trans-formed versions of the input images. Before each estimation,the second image is pre-transformed by a set of predefinedtransformation parameters. The predefined parameters arechosen such that for each registration parameter, there are atleast one estimate above and one estimate below the true value(hence bidirectional). For 1D shift estimation, a coordinateshift of the second image by one pixel is required. For 2Dshift estimation, two coordinate shifts of the second image,each by one pixel along a sampling direction, are sufficient.

For future work, the bidirectional bias correction schemecan be applied to other shift estimation methods, as long asthe bias model is known. Cross-correlation, for example, mayhave a linear bias model because its iterative version con-verges monotonically [10]. Phase correlation may also pro-duce a biased estimate because of mismatched border wrapand non-overlapping areas between the input images.

Furthermore, the estimation bias should not be restrictedto a linear model. Bias of a cross-correlation peak due toparabola fitting, for example, is nonlinear [11], so does thebias of gradient correlation [12]. Even the bias of the origi-

nal GBSE is nonlinear considering the truncation error of theTaylor series approximation [5]. In these cases, more estima-tion points are needed to fit a higher-order bias model.

Finally, the bidirectional bias correction scheme can beapplied to the registration of images under more complextransformations models such as rigid, affine, projective trans-formations and optical flow.

7. REFERENCES

[1] B.D. Lucas and T. Kanade, “An iterative image registra-tion technique with an application to stereo vision,” inProc. of DARPA’81, 1981, pp. 121–130.

[2] J. Kearney, W. Thompson, and D. Boley, “Optical flowestimation: an error analysis of gradient-based methodswith local optimization,” PAMI, vol. 9, no. 2, pp. 229–244, 1987.

[3] J.W. Brandt, “Analysis of bias in gradient-based opticalflow estimation,” in Proc. IEEE Asilomar Conf. SignalsSyst. Comput., 1995, pp. 721–725.

[4] M.D. Robinson and P. Milanfar, “Fundamental perfor-mance limits in image registration,” IEEE Trans. on IP,vol. 13, no. 9, pp. 1185–1199, 2004.

[5] T.Q. Pham, M. Bezuijen, L.J. van Vliet, K. Schutte, andC.L. Luengo Hendriks, “Performance of optimal reg-istration estimators,” in Visual Information ProcessingXIV, 2005, vol. 5817 of SPIE, pp. 133–144.

[6] E.P. Simoncelli, “Design of multi-dimensional deriva-tives filters,” in Proc. ICIP, 1994.

[7] M. Elad, P. Teo, and Y. Hel-Or, “On the design of opti-mal filters for gradient-based motion estimation,” JMIV,vol. 23, no. 3, pp. 345–265, 2005.

[8] M.D. Robinson and P. Milanfar, “Bias minimizing filterdesign for gradient-based image registration,” SP:IC,vol. 20, no. 6, pp. 554–568, 2005.

[9] H. Ji and C. Fermueller, “Noise causes slant underesti-mation in stereo and motion,” Vision Research, vol. 46,pp. 3105–3120, 2006.

[10] F. Scarano, “On the stability of iterative PIV im-age interrogation methods,” in Proc. of 12th Interna-tional Symposium on Applications of Laser Techniquesto Fluid Mechanics, 2004.

[11] M. Shimizu and M. Okutomi, “Precise subpixel estima-tion on area-based matching,” Systems and Computersin Japan, vol. 33, no. 7, pp. 1–10, 2002.

[12] V. Argyriou and T. Vlachos, “Estimation of sub-pixelmotion using gradient cross-correlation,” ElectronicsLetters, vol. 39, no. 13, pp. 980–982, 2003.