An Introduction to Feature Tracking

An Introduction to Feature TrackingN. D.OutlineIntroduction: What is feature tracking?FeaturesFeature Tracking AlgorithmAlternativesApplicationsQ & A

Introduction: MotivationHow does object tracking work?How do you determine the similarity between images?Introduction: PreviewYou have two images. How does the first map onto the second?

Introduction: Feature TrackingFrom Wikipedia: Motion estimation is...

Process of determining motion vectors that describe the transformation from one 2D image to another, usually from adjacent frames in a video sequence.

Features: DefinitionOfficial Definition:a n-tuple or vector whose components are functions of the initial measurement pattern variables or some subset of the initial measurement pattern variables.

Human-Speak Definition:A location on an image where you track change.Features(A location on an image where you track change==Any location) ?

Features(A location on an image where you track change==Any location) ?trueFeaturesWhat is a good feature?Features: An Example

(1)(3)(5)(2)(4)(6)Features: Computed Answer

Features: Computed Answer

Features: The Good, the Bad, the UglyA good feature should be locally unique and invariant to geometric transformations.

Good Bad Ugly Features: Local UniquenessTo determine local uniqueness, look at sub-parts of the feature.

Good Bad Ugly

. . .. . .. . .Tracking: Algorithm OverviewInputs: (a: Image, b: Image)

Step 1:vectora = Extract Features ( a )vectorb = Extract Features ( b )Step 2:matches = Compute Difference ( vectora , vectorb )Step 3:result = Eliminate False Positives ( matches )

Tracking: Shoes ExampleInputs:

Tracking: Shoes ExampleOutput:Inputs:

Tracking: Shoes Example --- Step 1Step 1: Feature Extraction using the Harris OperatorTracking: Shoes Example --- Corner Detection The Harris Operator is used in corner detection.

Tracking: Shoes Example --- Harris Points Harris Operator output: bright points = locally unique candidates

Tracking: Shoes Example --- Choosing Points Choose the brightest points that are at least delta distance apart.

Tracking: Shoes Example --- Choosing Points Same result, on the colored images:

Tracking: Shoes Example --- Harris Equationwhere

W is a weights kernel (Gaussian kernel; smoothing)Ix is the horizontal Sobel operator (horizontal edge detection)Iy is the vertical Sobel operator (vertical edge detection)

Tracking: Shoes Example --- Sobel OperatorSobelx and Sobely applied to Image a

[ Horizontal edge detection ][ Vertical edge detection ]Tracking: Shoes Example --- Step 2Step 2: Feature Matching using the Sum of Squared Differences(alternative: Normalized Cross Correlation) to measure error and find similarity between images.

Tracking: Shoes Example --- SSDLow error measurements between features means a potential match.

The SSD can give us false positives. Consider the case where one feature is detected on the original image, but not detected on the new image (maybe because it the new angle obscures it or it is out of scope). In that case, the SSD gives us its best guess, which will be the wrong answer.Tracking: Shoes Example --- SSD Equationwhere

E(u, v) is the error measurement at (u, v) compared to (x, y)W(x, y) is a weights kernel (Gaussian kernel; smoothing)I(x, y) is the Sobel operator (edge detection)

Tracking: Shoes Example --- Step 3Step 3: Eliminate False Positives using the Random Sample Consensus (RANSAC) methodTracking: Shoes Example --- RANSACAn iterative method to estimate parameters of a mathematical model from a set of observed data which contains outliers.

talk about the inlier

Statistical method: Given a difference threshold (how much variation you want to allow), randomly select a match vector and count how many inliers it has. After a certain number of iterations, you select the vector that has the most inliers and disregard its outliers. The magenta vector is the best vector in this case.Tracking: Flowers ExampleExample #2:

Tracking: Museum ExampleExample #3:

(A Few) AlternativesPixel-by-pixel computation (horribly inefficient)Relational matchingBag-of-WordsBoth generative or discriminative modelsParticle FilterMulti-Scale Oriented PatchesEigenfaces / Space of Faces

Applications: Feature Extraction & TrackingFilmVideo trackingReplacement sets or characters (ex: Gollum)Motion tracking markers

BiometricsFacial recognition (Eigenfaces)Xbox Kinect (?)

Questions ? Ask : Clap

(You may clap.)SourcesGollumMotion Estimation - WikiRansac - WikiCar with Tracking MarkersMotion Tracking MarkersJoffrey Clapping GIFGlossary of Computer Vision TermsSkeleton code for reading in/writing out images from ColorFrontFlowers Pictures from ColorFront Computer Vision: Algorithms and Applications by Richard Szeliski