FLUid Image analysis and Description (FLUID)Project No. FP6-513663

FET - Open Domain, Priority IST

Deliverable 2.1

Report on Filter Bank Design forLocal Fluid Motion Estimation

Jing Yuan, Florian Becker, Christoph SchnorrCVGPR group, University of Mannheim

Due date of deliverable: 30 November 2005Actual submission date: 30 November 2005

Start date of project: 1 December 2004Duration: 3 years

Project cofunded by the European Comissionwithin the Sixth Framework Programme (2002-2006).

Dissemination level: PU (Public) Revision: 1

Contents

1 Introduction 11.1 Design Requirements . . . . . . . . . . . . . . . . . . . . . . . 11.2 Local Filters and Motion Estimation: A Brief Overview . . . . 21.3 Design Decisions . . . . . . . . . . . . . . . . . . . . . . . . . 4

2 Multiscale Local Motion Estimation 52.1 Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52.2 Filter Design . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2.2.1 Low-pass filters . . . . . . . . . . . . . . . . . . . . . . 72.2.2 Derivative Filters . . . . . . . . . . . . . . . . . . . . . 7

2.3 Image Pyramids . . . . . . . . . . . . . . . . . . . . . . . . . . 82.4 Interpolation with Splines . . . . . . . . . . . . . . . . . . . . 9

2.4.1 Uniform B-Splines . . . . . . . . . . . . . . . . . . . . 92.4.2 Computing the Coefficients . . . . . . . . . . . . . . . 102.4.3 Image Interpolation . . . . . . . . . . . . . . . . . . . . 12

2.5 Multiscale Motion Estimation . . . . . . . . . . . . . . . . . . 13

3 From Local to Global Motion Estimation 163.1 Ill-Posedness of Local Motion Estimation . . . . . . . . . . . . 163.2 Regularization . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

3.2.1 Numerical Rank and Truncation . . . . . . . . . . . . . 173.2.2 Filter Factors . . . . . . . . . . . . . . . . . . . . . . . 17

3.3 Global Variational Motion Estimation . . . . . . . . . . . . . . 183.3.1 Data Term . . . . . . . . . . . . . . . . . . . . . . . . . 183.3.2 Global Regularization . . . . . . . . . . . . . . . . . . 20

i

Chapter 1

Introduction

This chapter gives a brief overview over local filter design in connection withlocal image motion estimation (section 1.2). The overview is based on therequirements that we impose on the design of a multiscale stage for localmotion estimation. This requirements are specified and discussed in (section1.1).

The stage itself and the design of corresponding filters are described insection 2. Next, we specify how it can be used as part of any non-localvariational approach to motion estimation within the FLUID project (section3).

This report accompanies a second report [BYS05] which describes alltechnical details of a corresponding implementation (demonstrator). Theimplementation is equipped with an interface allowing for its use as partof non-local variational approaches to motion estimation, to be developedwithin the overall project.

1.1 Design Requirements

Local motion estimation is concerned with estimation the apparent imagevelocity u(x, t) at some image position x ∈ Ω and some point of time t ∈[0, T ]. Depending on the local image structure, it is possible to recover thefull flow vector, or only a component of it (e.g., the so-called normal flow).Moreover, in homogeneous image regions, no motion can be estimated at all.This problem is known as the aperture problem.

Target of the design of a computational stage for local motion estimationare non-local variational approaches of the form

u∗ = argminu∈U

J(u) , J(u) = D(u) +R(u) (1.1)

1

The stage is incorporated into the so-called data-term D(u) which is com-plemented by a regularizing functional R(u). We refer to [HS81, WS01] fora prototypical example and a survey of extensions.

In connection with (1.1), the following requirements arise for the designof a computational stage for local motion:

(i) Motion estimation should operate on multiple scales. ’Multiscale’ refersto the capability to deal with large motions leading to aliasing alongthe time axis at the original image resolution.

(ii) Motion estimation should be local. ’Local’ refers to small spatial sup-port: As part of a non-local variational approach, using a large spa-tial supports is not reasonable because the corresponding unspecificsmoothing effect interferes in a rather uncontrolled way with the ef-fect of variational regularization terms, which are a major topic of theinvestigations within this project.

(iii) Motion estimation should provide a confidence measure allowing forproper mathematical models of the interplay in (1.1) between localmotion estimation and non-local variational regularization.

The subsequent overview over local filters for motion estimation focusesmainly on the requirements (i) and (ii) above. Issue (iii) is dealt with insection 3.

1.2 Local Filters and Motion Estimation: A

Brief Overview

Filter design for local motion estimation has a long history in image pro-cessing and computer vision. Whereas the filters used in the seminal paperby Horn and Schunck [HS81] were rather crude from the viewpoint of signalprocessing, closer investigations of the subject were published rather early.

Hashimoto and Sklansky [HS87], for example, presented a thorough studyof Gaussian derivative filtering, taking into account aspects of discretizationand efficient implementation. Using Krawtchouk polynomials as discreteanalogs of Hermite polynomials, which are orthogonal with respect to theGaussian-weighted L2 inner product, they exhibited some optimal propertiesof the resulting filters, such as maximal energy concentration within theNyquist frequency band for filters with small spatial support. Moreover,they enjoy the following favourable properties:

2

• They are robust in terms of (i) approximating the ideal frequency re-sponse (iω)k of the k-th derivative for small frequencies, and (ii) havinglow-pass characteristics for noise suppression at higher frequencies. In-deed, the latter decay is exponential and hence very fast.While it is straightforward to develop impulse responses providing goodpointwise approximations within ω ∈ [−ωg, ωg] of the ideal impulse re-sponse for larger ωg, this is always at the cost of a considerably largersensitivity to noise.

• They are computationally efficient due to separability, and becauseall filter coefficients are simple rational numbers, allowing for specialimplementations with integer arithmetic, if required.

Major attempts to improve filters for local motion estimation include studiesof their invariance properties [VF92, DV94, FS97]. For example, computationof the gradient should not depend on the orientation of the local coordinatesystem. While marginal improvements are possible, the resulting coefficientsare on longer rational numbers. This seems to be a high price comparedto the good rotational invariance already achieved with the filter design in[HS87].

A second line of research concerns local motion computation using ana-lytical bandpass filters. Starting point was work by Fleet and Jepson [FJ90]who showed the increased robustness of using the phase of such filters in-stead of image intensity. On the other hand, locally constant motion overa 15 × 15 × 15 pixel spatio-temporal support was assumed, and more than20 filters were used for sampling the frequencey space, in order to determinethe orientation of the frequency plane which corresponds to the translationof local image patches.

To alleviate these computational costs and to reduce the filter support,wavelet-based versions were investigated [SFAH92, MK98, Kin01, Ber01].Using standard dyadic multiresolution schemes [Mal98] causes a severe lossof invariance, however. Therefore, by introducing redundant signal repre-sentations, attempts were made to optimize the compromise between ap-proximate invariance (translation, rotation) and computational efficiency[MK98, Kin01].

Our own investigation of these schemes [Neu04, chapter 3], in connectionwith invariant feature extraction for pattern recognition, showed however,that the approximation to rotational invariance is not satisfying. While thesedeficiencies can be improved [NS05], it leads again to elaborate filter banksrepresented in frequency space that do not allow for efficient implementationsin the spatial or spatio-temporal domain.

3

1.3 Design Decisions

All partners agreed on a first project meeting that it would be worthwhileto design and to implement a first version of a standard data term D(u), tobe used by any partner as part of variational approaches of the form (1.1).This will allow in subsequent project phases for the comparability of differentvariational approaches to motion estimation.

Based on

– the discussion in the previous section,

– our experience with optical flow computation in computer vision (e.g.,[BWS05] and references),

– and our recent experience with applying a suitable modification of theprototypical variational approach of Horn and Schunck [HS81] to Par-ticle Image Velocimetry [RKNS05],

we decided to design a data term D(u) as a first standard, by modifyingthe well-known Lukas-Kanade approach [LK81, BFB94, BM04] in view ofits incorporation into (1.1) and the corresponding requirements discussed insection 1.1.

4

Chapter 2

Multiscale Local MotionEstimation

In this section, we describe

– the local approach to motion estimation,

– the corresponding filter design,

– the scheme used for image interpolation and warping, and

– the multiscale version of the motion estimation approach.

The incorporation of the approach into a non-local variational approach willbe exemplified in section 3.

2.1 Approach

Let

Gσ(x− y) =1

(2πσ2)d/2exp

(

− 1

2σ2‖x− y‖2

)

, d = 2 or 3

denote a Gaussian located at x in the image domain. Assuming constantimage motion within a region size related to the parameter σ, the approachconsists in minimizing

E(u) =

∫

Gσ(x− y)‖∇g>u+ gt‖2 dy (2.1)

5

Here, g(x, t) denotes the image sequence function with gradient and partialderivatives

∇g =

(∂

∂x1

∂∂x2

)

g , gt =∂

∂tg

Furthermore, g(x, t) is assumed not to change during motion, leading to thecommon contraint equation [HS81]

∇g>u+ gt = 0 (2.2)

Expanding the integrand of (2.1) and computing the derivative with respectto u leads to the linear system

Au = b (2.3)

where

A = Gσ ∗[(∇g)(∇g)>

](2.4a)

b = −Gσ ∗ (gt∇g) (2.4b)

and convolution ’∗’ is applied element-wise. Matrix A is known as structuretensor in the literature.

Note: Both Gσ and the partial derivatives ∇g, gt in (2.3) and (2.4) areimplemented with the filters described in the next section.

2.2 Filter Design

We describe a class of filters with the following properties:

• The filters are separable, that is multi-dimensional filters are formallyconstructed by taking the outer product of one-dimensional filters. Forexample, if h0 denotes the impulse response of a low-pass filter (zero’th-derivative and noise suppression), and h1 the impulse response of afirst-derivative filter, then the partial derivative with respect to x2 ofan image sequence g(x, t) is estimated:

∂

∂x2g(x1, x2, t) ≈ (h0 ⊗ h1 ⊗ h0) ∗ g

Of course, the implementation of the right-hand side is done by subse-quent one-dimensional convolutions.

As a result, it suffices in the following to consider the design of one-dimensional filters.

6

• The filters are “multiscale” in the sense that a larger scale can be com-puted from a smaller scale by “adding” the scale difference, similarlyto the semigroup property in the linear continuous scale space.

• All filter coefficients are simple rational numbers that in principle canbe efficiently implemented on dedicated hardware.

• Maximal noise suppression through exponential decay as ω → π.

2.2.1 Low-pass filters

Let 3 ≤ n ∈ N, n odd, denote the size of the (discrete) impulse response.Then the definition is:

h0n(x) =

1

2n−1

(n− 1x

)

, x = 0, . . . , n− 1 (2.5)

Convolution approximates Gaussian smoothing with parameters µ = n−12

and σ2 = n−14

. Figure 2.1 below demonstrates why this filter is preferableinstead of sampling the Gaussian, besides its efficient implementation.

p2

p

0.2

0.4

0.6

0.8

1

p2

p

0.2

0.4

0.6

0.8

1

p2

p

0.2

0.4

0.6

0.8

1

Figure 2.1: Frequency responses h0n(ω) for n = 3, 5, 11 (blue), and for the

sampled Gaussian low-pass (black). For filters with small spatial support,the energy of the approximate filter (blue curves) is better contained withinthe Nyqist interval.

2.2.2 Derivative Filters

Computation of derivatives by convolution is based on the following repre-sentation through the Fourier transform:

d

dx

(h0 ∗ g(x)

)↔ (iω)

(h0(ω)g(ω)

)=((iω)h0(ω)

)g(ω)

anddk

dxk

(h0 ∗ g(x)

)=

d

dx

( dk−1

dxk−1

(h0 ∗ g(x)

))

7

These equations say that derivative estimation, in conjunction with noisesuppression, amounts to convolve the data with the derivative of a low-passfilter.

The following modification of the definition of low-pass filters (2.5) takesthis into account:

hkn(x) =

1

2n−k−1

k∑

i=0

(−1)i

(n− k − 1x− i

)(ki

)

, x = 0, . . . , n− 1 (2.6)

The implementation is very simple:

(i) Convolve n− k − 1 times with the impulse response (1, 1).

(ii) Convolve k times with the impulse response (1,−1).

(iii) Normalize.

While it is straightforward to design filters with excellent pointwise approxi-mation of the ideal derivative filter, this also results in poor noise suppression– see figure 2.2. Therefore, when working with real image data, (2.6) is ourchoice.

0.5 1 1.5 2 2.5 3

0.5

1

1.5

2

2.5

3

0.5 1 1.5 2 2.5 3

0.5

1

1.5

2

2.5

3

Figure 2.2: Frequency responses of two classes of derivative filters in com-parison with the ideal derivative filter (blue line). Left: Approximation ofGaussian derivative filters (2.6). The low-pass band shrinks for increasingsize n of the impulse response, due to larger noise suppression. Right: An-other class of derivative filters which much better approximates the idealfilter the larger is n. Noise suppression is poor, however. For real imagedata, therefore, the filter class shown on the left is preferable.

2.3 Image Pyramids

For each point of time t, a given image g(x) = g(x, t) is transformed intoa so-called image pyramid by low-pass filtering and subsampling. Proper

8

filtering depends on the sampling theorem and on the distribution of thespectral energy |g(ω)|.

Figure 2.3 illustrates this process. More details relevant for motion esti-mation are given in section 2.5.

Figure 2.3: From left to right: Image pyramid by low-pass filtering andsubsampling. Top: Improper filtering leads to aliasing. Bottom: Properfiltering does not create spurious structures.

2.4 Interpolation with Splines

Image motion leads to transformations of functions which do not conformto the image grid. Consequently, interpolation is needed for transferringarbitrary functions to and from image grids.

Since several decades [Pra78] cubic splines are known to accurately ap-proximate the ideal interpolation function sinc(x), while being compactlysupported. We describe next a corresponding efficient interpolation schemebased on the work of Unser [Uns99, UAE93a, UAE93b].

We consider again the one-dimensional case first, and then consider imageinterpolation.

2.4.1 Uniform B-Splines

Splines are smooth piecewise-polynomial functions. A spline of degree n isn− 1 times continuously differentiable. As a result, n degrees of freedom of

9

each polynomial piece are already fixed through these continuity constraints,leaving just a single degree of freedom.

In the 1-D case and for a uniform grid, the n-degree polynomial spline isdefined as

f(x) =∑

k∈Z

c(k)βn(x− k) (2.7)

where βn(x) denotes the uniform B-spline basis functions

βn = β0 ∗ . . . ∗ β0(x)︸ ︷︷ ︸

(n+1) times

, β0 =

1, −12< x < 1

212, |x| = 1

2

0, otherwise(2.8)

The commonly used cubic B-spline (n = 3)

β3(x) =

23− |x|2 + |x|3

2, 0 < |x| < 1

(2−|x|)3

6, 1 < |x| < 2

0, |x| ≥ 2

; (2.9)

performs high-equality interpolation at a low computational cost. Of course,it is possible to use a higher-order spline interpolation. The accuracy doesnot benefit much, however, in comparison to the increased computationalcosts.

Due to (2.7), each spline function is represented by the coefficients se-quence c(k). To perform interpolation, the main work is to compute thesecoefficients. This will be addressed in the next section.

Once the coefficients have been computed, it is easy to sample arbitraryfunction values f(x), or to find the derivative or integral of the function fbased on corresponding operations applied to the basis function βn(x):

dβn(x)

dx= βn−1(x + 1/2)− βn−1(x− 1/2); (2.10)

∫ x

−∞

βn(x) dx =

+∞∑

k=0

βn+1(x− 1/2− k). (2.11)

2.4.2 Computing the Coefficients

Given the samples f(k), the interpolation problem based on (2.7) amountsto find the spline coefficients from the conditions

∑

l∈Z

c(l)βn(x− l)|x=k = f(k). (2.12)

10

This can be rewritten as a convolution

f(k) = (bn ∗ c)(k) (2.13)

where bn = βn(k − l) are filter coefficients. Then the spline coefficients c(k)can be found by inverse filtering [UAE93b]

c(k) = (bn)−1 ∗ f(k) (2.14)

The convolution kernel (bn)−1 can be easily computed by the z-tranform ofbn, and be implemented efficiently by a cascade of first-order causal andanti-causal recursive filters.

For example, the z-tranform of the cubic B-spline b3 from (2.9) is

B3(z) =z + 4 + z−1

6

Thus, the corresponding z-transform of the filter (b3)−1 is

(b3)−1(k) ↔ 6

z + 4 + z−1

The right-hand transform leads to the following causal and anti-causal re-cursive algorithm [Uns99]:

c+(k) = f(k) + z1c+(k − 1) , k = 1, . . . , N − 1 (2.15)

c−(k) = z1

(c−(k + 1)− c+(k)

), k = N − 2, . . . , 0 (2.16)

where z1 = −2 +√

3 and c−(k) = c(k)/6. Assuming the mirror-symmetricboundary conditions, the two initializing value c+(0) and c−(N − 1) aredefined as

c+(0) =1

1− z2N−21

2N−3∑

k=0

f(k)zk1 (2.17)

c−(N − 1) =z1

1− z21

(c+(N − 1) + z1c

+(N − 2))

(2.18)

The algorithm is numerically stable and fast. Interpolation can then by doneby a simple convolution. Since each basis function is compactly supported,this scheme provides a very good compromise between accuracy and compu-tational complexity [Uns99].

The spline-based interpolation (2.7) generalizes often-used interpolationmethods of the form

f(x) =∑

k∈Z

fkψ(x− k)

11

by using an extra series of coefficients ck, providing additional flexibility soas to construct reliable and fast interpolation schemes with compact support[TBU00]. Conversely, as the ck coefficients are still related to the samplesfk, spline-based interpolation may still be regarded as a normal interpolationmethod where the coefficients are defined implicitly.

In the following section, we compare the spline-based interpolation withtwo other common interpolation methods: Bilinear interpolation and bicubicinterpolation.

2.4.3 Image Interpolation

For the 2-D case and a uniform grid, the spline function reads

f(x, y) =

(k1+K−l)∑

k=k1

(l1+K−l)∑

l=l1

c(k, l)βn(x− k)βn(y − l) (2.19)

withk1 = k1(x) = [x− (n+ 1)/2], l1 = l1(y) = [y − (n + 1)/2]

andK = supportβn = n+ 1

The parameters c(k, l) are constructed by applying 1-D filtering (2.14) suc-cessively along x and y axis, respectively. It is obvious that when the splinecoefficients c(k, l) are given, the interpolated value at position (x, y) can becomputed by the sum within a compactly supported area around (x, y). Inthis work, the spline-based interpolation is used to implement image transfor-mation (warping) and image resampling, i.e. the rescaling procedures betweenthe finer scale and the coarser scale.

In order to show the main advantages of spline-based interpolation forimage warping, we use a highly relevant geometric transformation, rotation,which is successively applied k times to a given image. Figures (2.4) show thedifference between spline-based image interpolation and bilinear interpolationand bicubic interpolation, respectively (both latter methods are implementedas function interp2 in Matlab). It can be clearly seen that spline-based inter-polation generates much smaller errors than other two interpolation methodswith much less running time. Although the bicubic interpolation is more ac-curate than the bilinear method, it needs more running time to reach thehigher accuracy.

In the algorithm for multiscale motion estimation, resampling proceduresare necessary to construct 2-D functions at some finer level from a coarserlevel, or at some coarser level from a finer level with some zooming factor k.

12

According to (2.19), the spline-based resampling is implemented by selectingthe coordinate displacements (i + 1/k, j + 1/k), i = 1, . . . , m, j = 1, . . . , n,where k is the rescaling factor.

The comparison bewteen spline-based resampling and bilinear resamplingis shown in figure (2.5). The close-up views clearly exhibit the very goodperformance of the spline-based method, whereas the bilinear-based methodgenerates some visible artifacts.

2.5 Multiscale Motion Estimation

We summarize the overall algorithm for multiscale motion estimation. Adetailed description of all functions is given in the second part of this report[BYS05].

Algorithm: Multiscale optical flow estimation

Input: image pair g1, g2

Output: estimated warp u0

g(0)1 , . . . , g

(dcoarsest)1 ← multiscaleFilter2(g1) ;

g(0)2 , . . . , g

(dcoarsest)2 ← multiscaleFilter2(g2) ;

u(dcoarsest) ← Identity ;for i from level dcoarsest down to 0 do

repeat

g1 ←warpImage(g(i)1 ,+1

2u(i)) ;

g2 ←warpImage(g(i)2 ,−1

2u(i)) ;

∆u← estimateWarp(g1,g2) ;if ∆u improves some error measurement then

u(i) ← u(i) + ∆u ;end

until some stopping criterion is met ;if i > 0 then

u(i−1) ← rescaleWarp(u(i),g(i−1)) ;end

endAlgorithm 1: Algorithmic description of a multiscale optical flow es-timator.

13

Figure 2.4: The upper left image is an original 2-D PIV imagedownloaded from the website of The Visualization Society of Japan:http://piv.vsj.or.jp/piv/image-e.html. This image is rotated by 360 degreein 23 steps with different interpolation methods. The upper right imageshows the result using spline-based image interpolation. The bottom leftimage results from bicubic interpolation, and the bottom right imageshows the result from bilinear interpolation. The detailed error measure-ment in the table below is based on the average and standard derivation of‖g(x)− grot(x)‖2.

err. spline bilinear bicubic

mean 3.62e-3 1.56e-2 7.25 e-3st. deviation 1.36e-2 4.84e-2 2.47e-2

max 0.36 0.70 0.49run. time (sec.) 3.4 10.6 26.4

14

Figure 2.5: Top: original image. This image is zoomed 10-times by spline-based resampling, and by bilinear resampling. The respective two imagesare shown in the second row. Left: bilinear. Right: spline-based method.Corresponding close-up views in the third row show artifacts generated bythe bilinear method.

15

Chapter 3

From Local to Global MotionEstimation

In this section, we first make the ill-posedness of the local estimation scheme(2.3) explicit. Based on this, we discuss regularizations as well as the em-bedding of the regularized local motion estimation into a global variationalapproach (1.1).

3.1 Ill-Posedness of Local Motion Estimation

Definition (2.4) shows that the structure tensor A is symmetric and positivedefinite. Its spectral decomposition into eigenvectors e1, e2 and eigenvaluesλ1, λ2 reads:

A =(e1 e2

)(λ1 00 λ2

)(e1 e2

)>, λ1 ≥ λ2 ≥ 0

Solvability of (2.3) depends on the rank of A, that is the number of non-vanishing eigenvalues λ1, λ2, which in turn depends on the local grayvaluestructure defining A in (2.4).

We distinguish the following cases:

Rank = 2: Both components of u can be locally computed by solving (2.3).

Rank = 1: Image structure varies locally in a single direction only. As aresult, λ2 = 0, and only a single component of u, the so-called normalflow, can be computed. This is called the aperture problem.

Rank = 0: Both eigenvalues vanish λ1,2 = 0 due to a homogeneous imageregion. No motion information be computed.

16

Given the expansion of u and b in (2.3) in the corresponding coordinatesystem1

u = u1e1 + u2e2 , b = b1e1 + b2e2

the solution to (2.3) can be expressed as:

uloc =

b1

λ1

e1 + b2

λ2

e2 , λ1 > 0 , λ2 > 0b1

λ1

e1 , λ1 > 0 , λ2 = 0

0 , λ1 = λ2 = 0

(3.1)

Unfortunatly, it may quite often happen in practice, that both or one eigen-value is very small, λ2 1, providing a typical rank-deficient problem[Han98]. In this case, even when the data b on the right-hand side in (2.3) arecontaminated with small errors δb only – which is unavoidable in practicalsituations – then the results can easily become corrupted and useless.

We describe next two regularization approaches to cope with such situa-tions.

3.2 Regularization

3.2.1 Numerical Rank and Truncation

The simplest way to deal with the ill-conditioned structure tensor A in (2.3)is to define a suitable bound to determine the numerical rank. The latteris defined by two tolerance parameters ε and η and leads to the followingmodification of (3.1):

uloc =

b1

λ1

e1 + b2

λ2

e2 , λ21 + λ2

2 > η2 , λ2

λ1

> εb1

λ1

e1 , λ21 + λ2

2 > η2 , λ2

λ1

≤ ε

0 , λ21 + λ2

2 ≤ η2

(3.2)

Parameter η is used to determine numerically the zero-rank case. Parameterε measures the condition number of the matrix A to distinguish between therank-1 and rank-2 cases.

3.2.2 Filter Factors

A disadvantage of the previous method is the extra computational cost ofthe spectral decomposition of A decomposition. An alternative is to solve

1We use superscripts to distinguish the original coefficients, written with subscripts,

related to the global image coordinate system.

17

the least square problem

minu‖Au− b‖2 + α ‖u‖2 (3.3)

It is easy to see that problem (3.3) leads to the system

(A>A + αI)ureg = A>b (3.4)

Because the matrix (A>A + αI) has the same eigenvectors as A and botheigenvalues are always larger than zero, we do no longer have to distinguishexplicitly the three cases related to the rank of A.

Nevertheless, it is instructive to compare the solution of (3.4) with thatof (3.2). The former reads

ureg = f1b1

λ1e1 + f2

b2

λ2e2 (3.5)

where we call fi, i = 1, 2, the filter factors:

fi =λ2

i

λ2i + α

, i = 1, 2 (3.6)

Clearly, when λi 0, we have fi → 1. On the other hand, bad directionsλi → 0+ are filtered out by having fi → 0. As the result, f1, f2 can be con-sidered as indicator functions which may be used for analyzing the structuretensor:

Rank = 2: f1 ' 1 and f2 ' 1

Rank = 1: f1 ' 1 and f2 ' 0

Rank = 0: f1 ' 0 and f2 ' 0

Figure (3.1) shows an example illustrating the information provided by thefilter factors f1,2.

3.3 Global Variational Motion Estimation

3.3.1 Data Term

We address next the qestion of how to design a data term D(u) in (1.1),based on the previous considerations.

18

It is clear that reasonable data terms D(u) have to account for the infor-mation actually provided by uloc. According to (3.1), we make the ansatz:

D(u) =

∫

Ω

‖f(u;A, uloc)‖2dx

with

f(u;A, uloc) =

u− uloc , λ1 > 0 , λ2 > 0(u>e1)e1 − uloc , λ1 > 0 , λ2 = 00 , λ1 = λ2 = 0

(3.7)

At each location x ∈ Ω in the image domain, function f(u;A, uloc) definedin (3.7) requires the global motion field to be compatible with the localmeasurements uloc only within the subspace N(A)⊥, i.e.

f(u;A, uloc) = PN(A)⊥u− uloc , (3.8)

where PN(A)⊥ is the orthogonal projection operator to the complement ontothe nullspace N(A) of A at location x:

PN(A)⊥ = A+A =

e1e>1 + e2e

>2 , λ1 > 0 , λ2 > 0

e1e>1 , λ1 > 0 , λ2 = 0

0 , λ1 = λ2 = 0(3.9)

Like in the previous section, we wish to avoid the explicit distinction of thesethree cases. In view of (3.4), this leads us to the regularized version of (3.8)

f(u;A, ureg) = A+αAu− ureg = A+

α (Au− b) (3.10)

and the corresponding data term [Sch93]

D(u) =

∫

Ω

‖A+α (Au− b)‖2dx (3.11)

whereA+

α = (A>A + αI)−1A>

Using the representation (3.5), we may rewrite integrand of the data term

‖f(u;A, ureg)‖2 = f 21 (u1 − u1

reg)2 + f 2

2 (u2 − u2reg)

2

which makes very explicit how confidence measures are incorporated throughthe filter factors defined in (3.6).

19

3.3.2 Global Regularization

Functional (1.1) with the data term as defined in (3.11) is well-posed forconvex regularizers R(u). This has been shown for quadratic regularizers in[Sch93], and for non-quadratic regularizers in [WS01].

Proposition 1 ([Sch93, WS01]) Let R(u) in (1.1) be convex with respect

to(H1(Ω)

)2, and the functions a11, a12, a22 defined by

(a11 a12

a12 a22

)

:= A+αA

be measurable and (almost everywhere) bounded. Suppose that a11, a12 anda12, a22, respectively, are linearly independent as elements of L2(Ω). Thenthe functional (1.1) is strictly convex and has a unique global minimum u.

20

Figure 3.1: The structure tensor A is computed for the image shown on thetop. The two columns with coloured images show the filter factors f1, f2,respectively, for different paramter values. Blue indicates fi = 0, and redfi = 1. 2nd row: α = 10−6, size of Gσ: n = 15. 3rd row: α = 10−8, size ofGσ: n = 15. 4th row: α = 10−6, size of Gσ: n = 5.

21

Bibliography

[Ber01] C.P. Bernard. Discrete wavelet analysis for fast optic flow com-putation. Appp. Comput. Harm. Anal., 11(1):32–63, 2001.

[BFB94] J.L. Barron, D.J. Fleet, and S.S. Beauchemin. Performance ofoptical flow techniques. Int. J. of Comp. Vision, 12(1):43–77,1994.

[BM04] S. Baker and I. Matthews. Lucas-kanade 20 years on: A unifyingframework. Int. J. of Comp. Vision, 56(3):221 – 255, 2004.

[BWS05] A. Bruhn, J. Weickert, and C. Schnorr. Lucas/kanade meetshorn/schunck: Combining local and global optic flow methods.Int. J. of Comp. Vision, 61(3):211–231, 2005.

[BYS05] F. Becker, J. Yuan, and C. Schnorr. Demonstrator on multiscalemotion estimation. Activity Report, EU-Project FLUID Deliv-erable 2.2, CVGPR Group, University of Mannheim, Germany,Nov. 30, 2005.

[DV94] J. De Vriendt. Fast computation of unbiased intensity deriva-tives in images using separable filters. Int. J. of Comp. Vision,13(3):259–269, 1994.

[FJ90] D.J. Fleet and A.D. Jepson. Computation of component imagevelocity from local phase information. Int. J. of Comp. Vision,5(1):77–104, 1990.

[FS97] H. Farid and E.P. Simoncelli. Optimally rotation-equivariantdirectional derivative kernels. In Computer Analysis of Imagesand Patterns (CAIP’97), volume 1296 of Lect. Not. Comp. Sci.Springer, 1997.

[Han98] P. C. Hansen. Rank-Deficient and Discrete Ill-Posed Problems.SIAM, Philadelphia, 1998.

22

[HS81] B. Horn and B. Schunck. Determining optical flow. ArtificialIntelligence, 17:185–203, 1981.

[HS87] M. Hashimoto and J. Sklansky. Multiple-order derivatives fordetecting local image characteristics. Comp. Vision, Graphics,and Image Proc., 39:28–55, 1987.

[Kin01] N.G. Kingsbury. Complex wavelets for shift invariant analysis andfiltering of signals. J. Appl. Comput. Harm. Anal., 10(3):234–253,2001.

[LK81] B.D. Lucas and T. Kanade. An iterative image registration tech-nique with an application to stereo vision. In Image Understand-ing Workshop, pages 121–130, 1981.

[Mal98] S. Mallat. A Wavelet Tour of Signal Processing. Academic Press,1998.

[MK98] J.F.A. Magaray and N.G. Kingsbury. Motion estimation usinga complex-valued wavelet transform. IEEE Trans. Signal Proc. ,46(4):1069–1084, 1998.

[Neu04] J. Neumann. Adaptive Conjoint Wavelet-Support Vector Clas-sifiers. PhD thesis, Dept. Math. Comp. Science, University ofMannheim, Germany, Dec. 2004.

[NS05] J. Neumann and G. Steidl. Dual–tree complex wavelet transformin the frequency domain and an application to signal classifica-tion. Int. J. Wavelets, Multires. Inform. Proc., 3(1):43–66, 2005.

[Pra78] W.R. Pratt. Digital Image Processing. Wiley & Sons, 1978.

[RKNS05] P. Ruhnau, T. Kohlberger, H. Nobach, and C. Schnorr. Varia-tional optical flow estimation for particle image velocimetry. Ex-periments in Fluids, 38:21–32, 2005.

[Sch93] C. Schnorr. On functionals with greyvalue-controlledsmoothness terms for determining optical flow. IEEETrans. Patt. Anal. Mach. Intell., 15(10):1074–1079, 1993.

[SFAH92] E.P. Simoncelli, W.T. Freeman, E.H. Adelson, and D.J. Heeger.Shiftable multiscale transforms. IEEE Trans. Inform. Theory,38(2):587–607, 1992.

23

[TBU00] P. Thevenaz, T. Blu, and M. Unser. Image interpolation and re-sampling. In I.N. Bankman, editor, Handbook of Medical Imaging,Processing and Analysis, chapter 25, pages 393–420. AcademicPress, San Diego CA, USA, 2000.

[UAE93a] M. Unser, A. Aldroubi, and M. Eden. B-Spline signal process-ing: Part I—Theory. IEEE Transactions on Signal Processing,41(2):821–833, February 1993.

[UAE93b] M. Unser, A. Aldroubi, and M. Eden. B-Spline signal processing:Part II—Efficient design and applications. IEEE Transactions onSignal Processing, 41(2):834–848, February 1993.

[Uns99] M. Unser. Splines: A perfect fit for signal and image processing.IEEE Signal Processing Magazine, 16(6):22–38, November 1999.IEEE Signal Processing Society’s 2000 magazine award.

[VF92] T. Vieville and O. Faugeras. Robust and fast estima-tion of unbiased intensity derivatives in images. In Sec-ond European Conf. Comp. Vision (ECCV’92), volume 588 ofLect. Not. Comp. Sci., pages 203–211. Springer, 1992.

[WS01] J. Weickert and C. Schnorr. A theoretical framework for con-vex regularizers in pde–base d computation of image motion.Int. J. Computer Vision, 45(3):245–264, 2001.

24