CSSE463: Image Recognition CSSE463: Image Recognition Day 30Day 30This week This week

Today: motion vectors and trackingToday: motion vectors and trackingFriday: Project workday. Friday: Project workday.

First First status report status report due. Please bring short due. Please bring short printed copy to class: printed copy to class: what you didwhat you didwhat you didn’t and the technical challengeswhat you didn’t and the technical challengeswhat you will do next week. what you will do next week.

Questions?Questions?

What is image flow?What is image flow?

Notice that we can Notice that we can take partial take partial derivatives with derivatives with respect to x, y, and respect to x, y, and time.time.

Image flow equationsImage flow equations Goal: to find where each pixel in frame t moves in frame t+Goal: to find where each pixel in frame t moves in frame t+t t

E.g. for 2 adjacent frames, E.g. for 2 adjacent frames, t = 1t = 1 That is, That is, x, x, y are unknowny are unknown

Assume: Assume: Illumination of object doesn’t changeIllumination of object doesn’t change Distances of object from camera or lighting don’t changeDistances of object from camera or lighting don’t change Each small intensity neighborhood can be observed in consecutive Each small intensity neighborhood can be observed in consecutive

frames: f(x,y,t)frames: f(x,y,t)f(x+f(x+x, y+x, y+y, t+y, t+t) t) for some for some x, x, y (the correct motion y (the correct motion vector)vector)..

Compute a Taylor-series expansion around a point in (x,y,t) Compute a Taylor-series expansion around a point in (x,y,t) coordinates.coordinates.

Gives edge gradient and temporal gradientGives edge gradient and temporal gradient Solve for (Solve for (x, x, y)y) See answers to first quiz question nowSee answers to first quiz question now

LimitationsLimitations

Assumptions don’t always hold in real-world Assumptions don’t always hold in real-world images.images.

Doesn’t give a unique solution for flowDoesn’t give a unique solution for flow Sometimes motion is ambiguousSometimes motion is ambiguous

Look at last question on last quiz.Look at last question on last quiz.

““Live demo”Live demo”

http://www.cs.mcgill.ca/~chundt/coherence/

Aperture problemAperture problem When we only see a small part of the image, motion can When we only see a small part of the image, motion can

be ambiguousbe ambiguous

Lesson: By constraining ourselves to a small Lesson: By constraining ourselves to a small neighborhood, we lose the “big picture”neighborhood, we lose the “big picture”

Solutions:Solutions: Assume pixels belong to same object and have same motionAssume pixels belong to same object and have same motion

Propagate constraints between pixelsPropagate constraints between pixels Can still take many frames to convergeCan still take many frames to converge

Doesn‘t work at occlusions (hidden parts of object)Doesn‘t work at occlusions (hidden parts of object) Sensitive to outliersSensitive to outliers

Only track motion of “interesting points”Only track motion of “interesting points”

Q1

Interest point trackingInterest point tracking

Idea: corners and other easily-identifiable Idea: corners and other easily-identifiable points can be tracked, and their motion is points can be tracked, and their motion is unambiguous.unambiguous.

Technique for calculating a set of interest Technique for calculating a set of interest points: points: Loop over image:Loop over image:

Calculate an interest score for a small Calculate an interest score for a small neighborhood around each pixelneighborhood around each pixel

If the score is above threshold, then add it to a If the score is above threshold, then add it to a point set to be returned.point set to be returned.

Q2

Stretch breakStretch break

When is your group willing to present?When is your group willing to present?

Q3,4

Interest scoresInterest scores One solution: One solution:

Take minimum variance of those found in 4 directions:Take minimum variance of those found in 4 directions:

(3x3 shown, but would use bigger window)(3x3 shown, but would use bigger window) Another related texture-based operator is based on Another related texture-based operator is based on

directions with significant edge gradient.directions with significant edge gradient. Could also detect corners using Kirsch operators (similar Could also detect corners using Kirsch operators (similar

to other edge operators).to other edge operators).

Tracking interest pointsTracking interest points

Search for them directly using a template.Search for them directly using a template.Choose one with highest correlationChoose one with highest correlationJust need to search in small neighborhood if Just need to search in small neighborhood if

frame rate is high enoughframe rate is high enoughShapiro’s Alg. 9.3Shapiro’s Alg. 9.3

Review: CorrelationReview: Correlation Just the dot product between Just the dot product between

the template and a the template and a neighborhood in the image. neighborhood in the image.

Idea: high correlation when Idea: high correlation when the template matches. the template matches.

Problem: Problem: alwaysalways high high correlation when matching correlation when matching with bright regionwith bright region

Solution: Normalize each region Solution: Normalize each region by subtracting mean by subtracting mean before before taking dot producttaking dot product

Matlab libraries exist to extract frames (still Matlab libraries exist to extract frames (still images) from video for processing.images) from video for processing.

Computing trajectoriesComputing trajectories Problem: what if unlabeled moving points belong to Problem: what if unlabeled moving points belong to

multiple trajectories?multiple trajectories? Idea: want to maximize smoothness, defined as change Idea: want to maximize smoothness, defined as change

in direction and change of speed.in direction and change of speed. Given two points pGiven two points ptt, p, pt+1t+1, thought to be related, calculate v, thought to be related, calculate vtt

= p= pt+1t+1 – p – pt. t. Then:Then:

tt

tt

tt

ttt VV

VVw

VV

VVwS

1

1

1

12

)1(

direction Average speed compared to geom. mean

Computing trajectoriesComputing trajectories

tt

tt

tt

ttt VV

VVw

VV

VVwS

1

1

1

12

)1(

Then the “Greedy-Exchange” algorithm is used Then the “Greedy-Exchange” algorithm is used to choose which points belong to which to choose which points belong to which trajectory. trajectory. Operates by creating trajectories and exchanging Operates by creating trajectories and exchanging

points if it would increase total smoothness. points if it would increase total smoothness. Local optimum.Local optimum.

Example: Tracking Pedestrians Example: Tracking Pedestrians with Human Modelswith Human Models

http://www.umiacs.umd.edu/users/hismail/Ghost_outline.htm

Thanks to Thomas Root for linkThanks to Thomas Root for link

Another approachAnother approach

The Kalman filter is a probabilistic model The Kalman filter is a probabilistic model that combines noisy measurements with that combines noisy measurements with the expected trajectory of the object. It the expected trajectory of the object. It works even with occlusion.works even with occlusion.

See See http://www.cs.unc.edu/~welch/kalman/ for for a start.a start.