Motion Analysis (contd.)Motion Analysis (contd.)

Slides are from RPI Slides are from RPI Registration Class.Registration Class.

22

TransformationsTransformations

• Geometric transformations Geometric transformations

• Intensity transformationsIntensity transformations

33

Forward Geometric Forward Geometric TransformationTransformation

“Moving” image, Im “Fixed” image, If

Transformed moving image, Im’. It is mapped into the coordinate system of If

44

Mapping Pixel Values - Going Mapping Pixel Values - Going BackwardsBackwards

“Moving” image, Im “Fixed” image, If

Need to go backward to get pixel value. This will generally not “land” on a single pixel location. Instead you need to do interpolation.

55

Backward TransformationBackward Transformation

• Because of the above, some techniques Because of the above, some techniques estimate the parameters of the estimate the parameters of the “backwards” transformation, “backwards” transformation, TT-1-1::– From the fixed image to the moving imageFrom the fixed image to the moving image

• This can sometimes become a problem This can sometimes become a problem if the mapping function is non-invertibleif the mapping function is non-invertible

• Intensity-based algorithms generally Intensity-based algorithms generally estimate the backwards transformationestimate the backwards transformation

66

Intensity TransformationsIntensity Transformations

• When the intensities must be When the intensities must be transformed as well, the equations transformed as well, the equations get more complicated:get more complicated:

Intensity mapping function Intensity mapping parameters

77

MRI image registration, similarity transformation (rotated by MRI image registration, similarity transformation (rotated by 10 degrees, with a translation of 17mm and 13mm)10 degrees, with a translation of 17mm and 13mm)

88

Intensity-Based RegistrationIntensity-Based Registration• Use the intensities more or Use the intensities more or

less directlyless directly• Compare intensities Compare intensities

between between – Mapped (transformed) version Mapped (transformed) version

of the moving image of the moving image IImm (based (based on an estimated on an estimated transformation) andtransformation) and

– Fixed image Fixed image IIff

• Need:Need:– Pixel-by-pixel error measurePixel-by-pixel error measure– Mapping techniqueMapping technique– Minimization techniqueMinimization technique

99

Example Error Measure: Example Error Measure: SSDSSD

Region of intersection between images

Pixel location within region

1010

SSD Example: Initial SSD Example: Initial AlignmentAlignment

1111

SSD: Sum of Squared Errors SSD: Sum of Squared Errors

• Advantages:Advantages:– Simple to computeSimple to compute– DifferentiableDifferentiable– Optimal for Gaussian error distributionsOptimal for Gaussian error distributions

• Disadvantages:Disadvantages:– Doesn’t allow varying “gain” between the Doesn’t allow varying “gain” between the

images, which may be caused by different images, which may be caused by different illuminations or different camera settingsilluminations or different camera settings

– Biased by large errors in intensity Biased by large errors in intensity •E.g. caused by contrast agent injectionE.g. caused by contrast agent injection

1212

Working in the ParametersWorking in the Parameters

• Remember:Remember:

• This means that to evaluate the This means that to evaluate the effect of a transformation estimate effect of a transformation estimate what we really want to evaluate iswhat we really want to evaluate is

1313

The Role of the RegionThe Role of the Region

• Observe: the region over which the Observe: the region over which the transformation is evaluated depends on transformation is evaluated depends on the parameters:the parameters:

• This can cause problems This can cause problems in practice:in practice:– A transformation resulting A transformation resulting

in no overlap leads to 0 in no overlap leads to 0 error!error!

1414

Evaluating the Objective Evaluating the Objective FunctionFunction

• Pixel-by-pixel evaluation within the Pixel-by-pixel evaluation within the regionregion

• Apply the inverse mapping at each Apply the inverse mapping at each pixelpixel

• Problem: inverse mapping of pixel Problem: inverse mapping of pixel does not “land” on a discrete pixel does not “land” on a discrete pixel location!location!

1515

Many Interpolation OptionsMany Interpolation Options

• Nearest neighborNearest neighbor

• Bilinear (or trilinear in 3d)Bilinear (or trilinear in 3d)

• SplineSpline

1616

Bilinear Interpolation in Moving Bilinear Interpolation in Moving ImageImage

• Weighted average of Weighted average of 4 surrounding pixels4 surrounding pixels– 8 surrounding pixels in 8 surrounding pixels in

3d3d

• Weight proportional Weight proportional to distance in x and to distance in x and in yin y

1717

Bilinear: Resulting IntensityBilinear: Resulting Intensity

1818

Two Options In PracticeTwo Options In Practice

• Create intensity, pixel-by-pixel, but Create intensity, pixel-by-pixel, but don’t create an explicit image don’t create an explicit image IImm’’

• Create actual image Create actual image IImm’’

1919

Resetting the StageResetting the Stage

• We have:We have:– Formulated the SSD objective functionFormulated the SSD objective function– Discussed how to evaluate itDiscussed how to evaluate it

• Next step is how to minimize it with Next step is how to minimize it with respect to the transformation respect to the transformation parametersparameters

2020

Before ProceedingBefore Proceeding

• We will estimate the parameters of the We will estimate the parameters of the backward transformationbackward transformation

• Abusing notation, we will minimize the Abusing notation, we will minimize the equationequation

• It should be understood (implicitly) that this It should be understood (implicitly) that this is the inverse transformation and the is the inverse transformation and the parameter values will be different parameter values will be different

2121

Function MinimizationFunction Minimization

• Function to minimize:Function to minimize:

• PossibilitiesPossibilities– Amoeba (simplex) methods - Amoeba (simplex) methods -

non-differentialnon-differential– Gradient / steepest descentGradient / steepest descent– Linearization (leading to least-Linearization (leading to least-

squares)squares)– Newton’s methodNewton’s method– Many more …Many more …

2222

Gradient / Steepest DescentGradient / Steepest Descent

• Compute gradient of objective function (with respect to transformation Compute gradient of objective function (with respect to transformation parameters), evaluated at current parameter estimateparameters), evaluated at current parameter estimate

• Make tentative small change in parameters in the negative gradient Make tentative small change in parameters in the negative gradient directiondirection

– is called the “learning rate”is called the “learning rate”• Re-evaluate objective function and accept change if it is reduced Re-evaluate objective function and accept change if it is reduced

(otherwise reduce the learning rate)(otherwise reduce the learning rate)• Continue until no further changes are possibleContinue until no further changes are possible

2323

Computing the DerivativeComputing the Derivative

• Issue:Issue:– Images are discreteImages are discrete– Parameters are continuousParameters are continuous

• Two methodsTwo methods– NumericalNumerical– Continuous (eventually numerical as well)Continuous (eventually numerical as well)

• Abstract definition of parameter vector:Abstract definition of parameter vector:

2424

Numerical DerivativesNumerical Derivatives

• Form each partial derivative by taking a Form each partial derivative by taking a small step in each parameter, small step in each parameter, i i = 1,..,= 1,..,kk::

• Choice of step size can be difficultChoice of step size can be difficult

• Requires Requires kk+1 function evaluations to +1 function evaluations to compute the derivativecompute the derivative

• Sometimes this is the only thing you can do!Sometimes this is the only thing you can do!

2525

Continuous Computation of Continuous Computation of DerivativeDerivative

Apply chain rule:Apply chain rule:

Intensity gradient in moving image

Change in transformation wrt change in parameters

Current error at pixel location

2626

Computing Image Computing Image DerivativesDerivatives• Many ways. Many ways.

– Simplest is pixel differences.Simplest is pixel differences.

– More sophisticated methods account for image More sophisticated methods account for image noisenoise

• Computed at each pixelComputed at each pixel

2727

Derivative In Moving ImageDerivative In Moving Image

• EquationEquation

• In detailIn detail– Pre-compute derivatives in moving image Pre-compute derivatives in moving image IImm

– During minimization, map pixels back into During minimization, map pixels back into moving image coordinate system and moving image coordinate system and interpolateinterpolate

2828

Image Derivative ExampleImage Derivative Example

2929

dT/ddT/d

• Similarity transform:Similarity transform:

– WhereWhere

• So derivative is 2x4 matrix (Jacobian):So derivative is 2x4 matrix (Jacobian):

3030

Putting It All TogetherPutting It All Together• At each pixel in overlap region:At each pixel in overlap region:

– Calculate intensity difference (scalar)Calculate intensity difference (scalar)– Multiply by 1x2 intensity gradient vector Multiply by 1x2 intensity gradient vector

computed by mapping pixel location back computed by mapping pixel location back to moving imageto moving image

– Multiply by 2x4 Jacobian of the Multiply by 2x4 Jacobian of the transformation, evaluated at pixel locationtransformation, evaluated at pixel location

– Result is 1x4 gradient vector at each pixelResult is 1x4 gradient vector at each pixel

• Sum each component of vector over Sum each component of vector over all pixelsall pixels

3131

Algorithm OutlineAlgorithm Outline

• Initialize transformationInitialize transformation

• RepeatRepeat– Compute gradientCompute gradient– Make step in gradient directionMake step in gradient direction– Update mapping equationUpdate mapping equation– Remap imageRemap image

• Until convergenceUntil convergence

3232

InitializationInitialization• Since this is a minimization technique, Since this is a minimization technique,

an initial estimate is required,an initial estimate is required,

• There are many ways to generate this There are many ways to generate this estimate:estimate:– Identity transformation, e.g. Identity transformation, e.g.

– Prior informationPrior information– Different techniqueDifferent technique

• Steepest descent only finds a local Steepest descent only finds a local minimum of the objective function minimum of the objective function

3333

ConvergenceConvergence

• Ideal is that gradient is 0.Ideal is that gradient is 0.

• In practice, algorithm is stopped In practice, algorithm is stopped when:when:– Step size becomes too smallStep size becomes too small– Objective function change is sufficiently Objective function change is sufficiently

smallsmall– Maximum number of iterations is Maximum number of iterations is

reachedreached

3434

ExampleExample

Initial errors Iteration 100 Iteration 200

Iteration 300 Final: 498 iterations

3535

DiscussionDiscussion

• Steepest descent is simple, but has Steepest descent is simple, but has limitations:limitations:– Local minimaLocal minima– Slow (linear) convergenceSlow (linear) convergence

3636

SummarySummary

• Intensity-based registration is driven by direct Intensity-based registration is driven by direct functions of image intensityfunctions of image intensity

• SSD is a common, though simple exampleSSD is a common, though simple example• Evaluating the SSD objective function (and most Evaluating the SSD objective function (and most

other intensity-based functions) requires image other intensity-based functions) requires image interpolationinterpolation

• Gradient descent is a simple, commonly-used Gradient descent is a simple, commonly-used minimization techniqueminimization technique

• Derivatives may be calculated using either Derivatives may be calculated using either numerical approximations or differentiation of the numerical approximations or differentiation of the objective function. objective function.