The basics

Computing motion between images

Matching images We will deal with two similar tasks:

– A. Given a point in one image, find its matching point in the other image - feature matching/optical flow.

– B. Find the transformation mapping one image to the other - Image alignment/ Image Registration.

Feature matching/optical flow

Alignment: Scale by 3/4

Why is it useful

Camera stabilization Tracking moving objects Finding the camera motion and 3D

reconstruction. Image processing by combining several

images (mosaicing, super-resolution…) Compression. …..

Matching by sliding window A basic method to find matching points

between images. (correspondence) a neighborhood of the point (Window) in

image I1 is compared to several Windows in I2.

The window with the best score wins. Various scores can be used.

Matching by sliding window

I1I2

I2

I2

I2 I2

The winning windowThe window

in image 1 A sliding window in I2

Matching Criteria - Difference

Common matching criteria:– SSD - Sum of Squared Differences

k - the size of the window.

p1 = (x1, y1) is the center of the window in I1.

p2 = (x2, y2) is the center of the window in I2.

– SAD - Sum of Absolute Differences

k

kj

k

ki

jyixIjyixIppd 222211121 ),(),(),(

k

kj

k

ki

jyixIjyixIppd ),(),(),( 22211121

Matching Criteria - Correlation Cross Correlation is similar to SSD, but can

be implemented more efficiently:

k

kj

k

ki

jyixIjyixIppd ),(*),(),( 22211121

The window to search

The image to search (I2)

The cross correlation score

The maximum cross correlation score

Handling illumination changes

SSD, SAD and Cross Correlation assume constant brightness.

On order to handle illumination changes, Normalized cross-correlation can be used.

I1 I2

Normalized Cross Correlation When ordering the pixels in the windows in

vectors v1, v2:

The Normalized Cross Correlation is:

212121

2

212

2121

,)(*)(),(

))()((),(

vvivivvvcorr

vvivivvvssd

i

i

21

21

2211

21

21 *

,

)(*)()(*)(

)(*)(),(

vv

vv

iviviviv

ivivvvNorCorr

ii

i

Inner product

Squared Vector norm

The aperture problem

?

?

?

The aperture problem (cont’)

Hard to track vertically.

Easy to track in both directions.

The aperture problem (cont’)

normalflow

T junction

realmotion

And what about smooth areas ?

??

Use bigger windows:– Less noise– Smaller aperture problem– Might include different motions

Smooth the image !

Using a Pyramid for optical flow Smoothing the images: we get an

estimation of the motion in uniform regions.

Handling large motions: Even if the motion in the original image is big, the motion in the small level is small.

Once the solution was find in level k, there is only small motion to fix in the level k-1: The optical flow calculations become much faster !

Pyramid[0]

Pyramid[1]

Pyramid[2]

Auto-Correlation (and how does the SSD handle a combination of motions)

j i

jyixIjiIyxAC ),(*),(),(

The AC meaning: How similar does the image looks to it selves in different shifts.

It has a strong connection with the aperture problem.

The SSD should find the best motion – depending on the auto-correlation function.

Deformations

The window matching assumes a pure image translation in small regions.

Possible solutions for deformations:– Invariant features– Iterations of motion computations + warping

Limitations of Window Matching

Accuracy: – A pixel is always matched to integer location

on the grid. The image motion is usually not integer.

Neighborhood/Scene constraints:– High level knowledge about the

scene/camera may help in limiting the search, and reducing errors. (for example, the scene is planar)