performance of optimal registration estimator

Performance of Optimal Registration Estimator

Tuan [email protected]

Quantitative Imaging GroupDelft University of Technology

The Netherlands

Conference 5817-15: VisualInformation Processing XIV

© 2004 Tuan Pham [email protected] 2

• Image registration– Bias of gradient-based shift estimation– Bias correction by iterative shift estimation

• Cramer-Rao bound of registration:– Optimal shift estimation– Optimal 2D projective registration

• Applications:– Panoramic image reconstruction– Super-resolution

Outline: accurate registration

© 2004 Tuan Pham [email protected] 3-0.26 -0.24 -0.22 -0.2 -0.18 -0.16

-0.3

-0.28

-0.26

-0.24

-0.22

-0.2

-0.18

-0.16

Δx = -0.2286

Δy =

-0.2

477

256x256 image at SNR=15dB

NCCMADTaylorphasetrue shift

Shift accuracy and precision

NCC:Normalized cross-correlation

MAD:Minimum absolute difference

Taylor:Local Taylor expansion

Phase:Plane fit of phase difference

100 noise realizations


1D Taylor shift estimator

• For a small shift vx: (vx < 1 pixel)

21

2 11 ( ) ( ) x

S

sMSE s x s x vN x

∂⎛ ⎞= − −⎜ ⎟∂⎝ ⎠∑

Minimize mean squared error ε over a neighborhood S:

ε

yields a least-squares solution:

( ) 12 1

21

( ) ( )LS Sx

S

ss x s xxv

sx

∂−

∂=∂⎛ ⎞

⎜ ⎟∂⎝ ⎠

∑

∑

221 1

2 1 1 21( ) ( ) ( ) ...2!x x x

s ss x s x v s x v vx x

∂ ∂= + = + + +

∂ ∂


• Bias due to truncation of Taylor expansion:– Least squares solves but ε ≠ 0

• Bias due to noise:– Noise modifies signals:

Biases of Taylor method

12 1( ) ( ) 0x

ss x s x vx

ε ∂= − − =

∂

*1 1 1*2 1 2

s ( ) ( ) ( )

s ( ) ( ) ( )

= +

= + Δ +

x s x n x

x s x x n x

2 3 51 1 1 1 1

3 5

2 2 2 21 1 1

5

1

31 1 ...3! 5!

Sx x x

S S

S S S S

n s s s sx x x x xbias

s n s sx x

v

x x

v v

∂ ∂ ∂ ∂ ∂⎛ ⎞⎜ ⎟∂ ∂ ∂ ∂ ∂⎝ ⎠= + + +

∂ ∂ ∂ ∂⎛ ⎞ ⎛ ⎞ ⎛ ⎞ ⎛ ⎞+⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟∂ ∂ ∂ ∂⎝ ⎠ ⎝ ⎠ ⎝ ⎠ ⎝ ⎠

∑ ∑ ∑

∑ ∑ ∑ ∑

• Total bias under brightness consistency:


Iterative bias correction

• Since bias→ 0 when vx → 0 3 5 ...x x xbias Av Bv Cv= + + +

ii = 0, Δx = 0, s2(0) = s2

Δx(ii) = findshift ( s1 , s2(ii) )

Δx += Δx(ii)

s2(ii+1) = warp ( s2 , Δx )

Δx(ii) < threshold No, loop again

Yes, finish

Initialization:

Shift estimation:

Update displacement:

Shift correction:

Break condition:

• Iterative shift refinement:


Performance of iterative shift estimation

0.44 0.46 0.48 0.5 0.52 0.54 0.56 0.580.42

0.44

0.46

0.48

0.5

0.52

0.54

0.56

0.58

Δ x

Δ y

precision of shift estimation

Tayloriterativetrue shift

100 noise realizations

Bias = 0 but the estimated shift is not precise:

stdev(Vx)> 0


Cramér-Rao bound on registration

• A lower bound on the variance of any unbiased estimator :2 1ˆ( ) ( )i i iiE m m −⎡ ⎤− ≥⎣ ⎦ F m

where m = [m1 m2 … mn]T & F is the Fisher Information Matrix (FIM).

• For 2D shift estimation: I2(x,y) = I1(x+vx,y+vy) the FIM is:2x x y

S S2 2

x y yS S

I I I1( )

I I Inσ

⎡ ⎤⎢ ⎥= ⎢ ⎥⎢ ⎥⎣ ⎦

∑ ∑∑ ∑

F v

where Ix Iy are image derivatives, is noise variance

• Registration variance:1 2 2 2 2

11

1 2 2 2 222

var( ) I I

var( ) I I

x n y n xS S

y n x n yS S

v Det

v Det

σ σ

σ σ

−

−

≥ = ≈

≥ = ≈

∑ ∑

∑ ∑

F

Fwhere Det = determinant ( F )

2nσ

m̂


0.45 0.5 0.550.45

0.5

0.55 σnoise=5

0.45 0.5 0.550.45

0.5

0.55 σnoise=10

0.45 0.5 0.550.45

0.5

0.55 σnoise=20

Iterative shift estimator is optimal

0 10 20 30 400

0.01

0.02

0.03

0.04

0.05

0.06

σnoise

x

Cramer-Rao Lower Boundwithout pre-smoothingadaptive pre-smoothing σs=σn/20


2D projective registration

' /' /==

x u wy v w

mosaic image

0 1 2

3 4 5

6 7 1 1

⎡ ⎤ ⎡ ⎤ ⎡ ⎤⎢ ⎥ ⎢ ⎥ ⎢ ⎥= ×⎢ ⎥ ⎢ ⎥ ⎢ ⎥⎢ ⎥ ⎢ ⎥ ⎢ ⎥⎣ ⎦ ⎣ ⎦ ⎣ ⎦

u m m m xv m m m yw m m

• Transformation:

• Levenberg-Marquardt iteratively minimizes:

2' = ×Dp M p

2[ '( ') ( )]= −∑ I Ii ii

E p p


2D projective registration byLevenberg-Marquardt optimization

11 iters: RMSE = 10.0 14 iters: RMSE = 1.5 19 iters: RMSE = 1.3

Original Distorted image Register after 19 iters

Warperror


Levenberg-Marquardt is locally optimal

0 10 200

0.5

1 x 10-3

std(

m1 )

0 10 200

0.5

1 x 10-3

std(

m2 )

0 10 200

0.05

0.1

std(

m3 )

0 10 200

5 x 10-4

std(

m4 )

0 10 200

0.5

1 x 10-3

std(

m5 )

0 10 200

0.05

0.1

std(

m6 )

0 10 200

2

4 x 10-6

std(

m7 )

0 10 200

2

4 x 10-6

σnoise

std(

m8 )

Cramer-Raoestimated


Application: Panoramic image reconstruction

128 x 128 x 100 long-IR sequence


Application: Super-resolution

Low-resolution input 2x High-Resolution output


Conclusion and Future Research

• Conclusion

– All conventional shift estimators are inaccurate due to bias

– Iterative Taylor shift estimator reaches Cramer-Rao bound

– Good super-resolution is achieved with the iterative registration

• Future research

– Precision of optic flow for certain neighborhood size

– Improving registration of JPEG compressed images


?


Image registration

Regis-tration

Irregular samples of LRimages on a HR grid

Set of LRimages

Fusion

FilteredHR image

DeblurredHR image

Deblur

• 3-step approach:

• Registration:– Shift estimation– 2D projective– Optical flow

• Inaccurate registration → poor super-resolution