Motion in Video

Presented By:

Dr. S. K. SinghIndian Institute of Technology (B. H.U.)Varanasi-221005.

Road Map

• Motion in Video• Motion classification• Motion in Video: Displacement, Velocity.• Object Motion in Scene(3-D)• Motion Model • Image formation model • TEMPORAL MOTION MODEL• Calculation of displacement• REGION OF SUPPORT MODELS• OBSERVATIONAL MOTION MODEL

Road Map• Why Motion MODEL• Motion Operation• 2-D Motion and Apparent Motion• Solution Of Aperture Problem• 2-D MOTION ESTIMATION• PROBLEM WITH 2-D MOTION• 2-D MOTION ESTIMATION• Representation Of Forward And Backward Motion Estimation

Road Map

• Estimation of forward and backward Motion• NON PARAMETRIC MODEL• BLOCK MATCHING METHOD• Pixel Based Motion Estimation• Feature based methods (“Indirect Method")• Pixel Based Motion Estimation• PEL-RECURSIVE METHOD• Optical Flow Based Method For Motion Estimation• Multiple Choice Question

Motion in Video

Motion is a change in position of object with respect to time and its reference

point(like origin or other point ).

It is described in terms of displacement (d) velocity(V), acceleration(a), and time(t).

Motion is observed by attaching a frame of reference(F1) to a body and measuring

its change in position relative to another reference frame(F2).

http://en.wikipedia.org/wiki/Motion_%28physics%29

Motion classification

• In physics, Motion in the universe is described through two sets of apparently

(superficial) contradictory laws of mechanics.

• Motions of all large scale and familiar objects in the universe ( projectiles, planets,

cells, and humans) are described by classical mechanics.

• Whereas the motion of very small atomic and sub-atomic objects is described by

quantum mechanics.

Motion in Video: Displacement, Velocity

Object Motion in Scene

MOTION

Motion is a fundamental technique to understanding the scene(3-D Object Motion).

It also helps to provides a sparse representation of scene with respect to time (t).

A time varying image is such that the spatial intensity pattern changes with time.

Hence time varying image is spatial temporal intensity pattern.

It is denoted by Sc(X1 ,X2,T), where X1 and X2 is spatial coordinates and t is time

coordinate.

Cont…

Source: macroevolution.livejournal.com - 749 × 501

Motion Model

A motion model is used to describe object motion in an image sequence.

Fundamental Motion Model

1. SPATIAL MOTION MODEL

The main aim to estimate the motion of image points in the 2-D motion (apparent

trajectory motion).

Such motion is combination of projection s of the motion of objects in a 3-D scene

and of 3-D camera motion.

Cont…

Although camera motion affects the movement of all or almost all image points.

The motion of 3-D object only affects a subset of image points compensated for by

either estimating it or by physically measuring it at the camera.

Image formation model Image formation model works on the following projection. Perspective Projection. Orthogonal projection.

Motion model of 3-D object.Example: Rigid body with 3-d translation and rotation 3-D affine motion.

TEMPORAL MOTION MODEL

TEMPORAL MOTION MODEL

This model finds the trajectory(Path) of individual points drawn in the (x ,y ,t) space

of an image sequence.

It can be fairly arbitrary since they depends on object motion.

The trajectory is linear function of velocity or displacement of moving object at

each x.

TEMPORAL MOTION MODEL: Calculation of displacement

It can be represented by as follows:

X(T)=x(t)+vt(t)*(T-t)

=x(t)+dt ,T (x) where dt,T(x)= vt(x)*(T-t). It is known as

displacement vector(distance between moving pixel value).

X(T)=x(t)+vt(t)*(T-t)+1∕2 *at (x).(T-t)2 .

This model is based on two velocity (linear) variables and two acceleration

(quadratic) variable.

REGION OF SUPPORT MODELS

The set of points (x, y ,t)={(x1, x2 , x3, …..xn) ,(y1, y2, y3, y4……yn) and ( t1 , t2, t3 ,

t4 ….tn)} are the values of spatial and temporal motion models.

On the basis of these values ,we find the region in given image by apply the region

of support model of the moving object.

Cont…

There are following types cases in the region

1. 1-R=the whole image.

2. This model is suitable for estimation of camera induced motion in a simple

static scene(image).

3. 2-R= one pixel value.

4. 3-R=Rectangular block of pixel.

OBSERVATIONAL MOTION MODEL

OBSERVATIONAL MOTION MODEL

The observation model is based on the intensity variations of moving object with

respect to time(t).During motion ,object does not change their appearance .

The value of intensity remains constant .

We observe the continuous function of intensity along trajectory path.

We find partial derivatives of intensity function (I) with respect to x, y , t andput zero .Note: Intensity function (I) must be remain constant through out trajectorypath.It means ∂I/∂s=0

∂I∕∂x(v1) +∂I∕∂y( v2 )+∂I∕∂t=(∂∕∂x ,∂∕∂y)T v +∂I∕∂t =0.= (∂∕∂x ,∂∕∂y)T

= spatial gradientWhere v=(v1 ,v2)T

Note:It can apply at single position (x, y) under constrained (one equation and two

known).

Why Motion MODEL

Why Motion model is required for Digital Video Processing?It is mathematical model describes various event in video processing system inparametric form. This is used to estimate these parameter( Velocity vector,displacement vector etc.)

Motion OperationMotion model consist of following motion operation. Translation. Affine. Perspective. Rotation

2-D Motion and Apparent Motion2-D Motion(Actual Motion)

• 2-D motion gives emphasize on moving object in (2-D) image plane.

• It is known as projected motion or pattern motion.

• There is no relative motion .

Apparent Motion( Superficial Motion)

• Apparent motion emphasizes on motion of object in 2-D image plane projected from 3-D scene.

• It is not real motion of object what we observe is known as apparent(superficial motion ) into 2-D image plane.

• It deals with relative motion between moving object and observer (device like camera).

Solution Of Aperture Problem • The resolution of aperture problem given by a simple process known as

intersection of constraints (IOC) for solving the aperture problem.

• IOC method uses only the local motions of two edge pieces to compute the global motion by finding the intersection of all possible global motions.

• It consistent with the two local motions detected.

Cont…• The global motion velocity of the moving object can be simply calculated using

IOC from the two edges of the 2D object associated with the top vertex of theobject.

• As the outcome of aperture problem, the motion of a pictorial object composed ofmany components of different orientations.

• It known as pattern or global motion cannot be detected.• It is shown below

2-D MOTION ESTIMATION

PROBLEM WITH 2-D MOTION

2-D MOTION ESTIMATIONBACKWARD ESTIMATIONIf the motion vector V is defined from time t to t- l∆t . It is known as backward estimation.

Representation Of Forward And Backward Motion Estimation

In the below figure ,which explain the motion of trajectory of moving object in 2-Dplane.Here L is trajectory find the apparent motion trajectory with the help forward and

backward motion.

Estimation of forward and backward Motion

In below figure.1 the point A has same spatial coordinates all frames. Here L

denotes the path (trajectory) of moving object in 3-D scene. There are two velocity

vectors Vf and Vb known as forward and backward velocity vector.

The compact notation of spatial coordinate S=(x , y). We will show the arrow from

A which is same as S.

A

AA

Vb

Vf

LFrame 1 Frame 3Main frame=Frame 2

NON PARAMETRIC MODELNON PARAMETRIC MODELAccording to this model ,Whenever a moving object under 3-D projection. It is not consider under rigidness of object. It provides the concept of smoothness of object edge and surface.

NOTE:It can be classified under deterministic and stochastic model.1. Optical Flow Filed Estimation.2. Block Motion Estimation.3. Pel- Recursive Algorithm.

BLOCK MATCHING METHODThe block based motion estimation is preferred over pixel based method inparticular implementations.It searches best matching block location which is in fixed size block.It depends on the frame translation of block.

TYPE OF BLOCK SEARCH ALGORITHM

• Full search block motion estimation algorithm.• 2-D logarithmic search algorithm.• Three steps search algorithm.• New three steps search algorithm.

Pixel Based Motion Estimation

The methods for finding motion vectors can be categorized into two parts.• Pixel based methods (“Direct Method").• Feature based methods (“Indirect Method").

Pixel based methods (“Direct Method")• Block-matching algorithm.

Source :www.sciencedirect.com

Pixel Based Motion Estimation

• Phase correlation(frequency domain methods).

Pel -Recursive motion estimation

• Netravali and Robbins developed a pel recursive spatio-temporal steepest descent

gradient technique in which the displacement of a pel was predicted from

previously transmitted information.

• Since then various algorithms have been proposed to improve the performance of

pel recursive motion estimation (PRME) techniques.

• The most important contribution was the modification of the steepest-descent

algorithm which was introduced by Walker and Rao.

Pel -Recursive motion estimation

• where dx and dy correspond to the horizontal and vertical components of the motion displacement.

• Assuming that are known for each x, y, t and defini

Pixel Based Motion Estimation

Real-time Frame Rate Up-conversion for Video Games (SIGGRAPH 2010 Slides)and.intercon.ru

Feature based methods (“Indirect Method")Feature based methods (“Indirect Method")Indirect methods use features, like• Corner detection.• Match corresponding features between frames.

Methodology used :• The main objective of the statistical function is:• To Remove matches that do not correspond to the actual motion.• With a statistical function applied over a local or global area.• Statistical functions that have been successfully used include RANSAC.

Pixel Based Motion Estimation

Horn-schunck methodIt seeks a motion vector that satisfies the optical flow field estimation withminimum pixel to pixel variation among the flow vector.

Multipoint neighborhood method:Assuming that every pixel in small block surrounding a pixel has same motionvector(MV).

Pel recursive Method:In this method, motion vector(MV) is updated from those of its pixels.

PEL-RECURSIVE METHOD

It is predicator –corrector type displacement estimator.

The predicator can be taken as the value of the motion estimate the previous pixel

location. Linear combination of motion estimation in neighborhood of current pixel.

Optical Flow Based Method For Motion EstimationOPTICAL FLOWThe optical flow is pixel level representation model for motion pattern.In each point in the image is assigned a motion vector.

There are following method for estimation of optical flow.• Bayesian method.• Gibbs random field motion estimation.• Optical flow equation.• Second order derivatives of optical flow field.

Digital Video Processing Textbooks

Yao Wang, Jörn Ostermann,Ya-Qin Zhang PrenticeHall, 2002

A.M. Tekalp, Prentice-Hall, 1995

References

1. Tekalp,A.Murat “Digital Video Processing.” P. cm (Prentice Hall signal

processing series),ISBN 0-13-190075-7, 1995.

2. AL Bovik “Handbook of image and digital video processing", academic press, A

Harcourt Science and Technology Company,ISBN-0-12-119790-5,2000.

3. Yao Wang, Jörn Ostermann,and Ya-Qin Zhang Digital Video processing

published in Prentice Hall, 2002.

References

4. R. G. Gonzalez and R. E. Woods “Digital Image Processing”. Addison

Wesley,2nd edition, 1992.

5. http://en.wikipedia.org/wiki/Motion_%28physics%29.

6. Oge Marques “Practical Image and Video Processing Using MATLAB.” John

Wiley & Sons, Technology & Engineering, 2011.

7. H. Gharavi and H. Reza-Alikhani “Pel-Recursive Motion Estimation Algorithm

” National Institute of Standards & Technology (NIST)