cs231a section 6: problem set 3 - artificial...

Review 6 - !Kevin Wong!

CS231A Section 6: Problem Set 3

Kevin Wong

11/09/2012 1


Announcements

•  PS3 Due 2:15pm Tuesday, Nov 13

•  Extra Office Hours: – Friday 6 – 8pm Huang Common Area,

Basement Level.

11/11/2011 2


Topics

•  Review – Optical Flow

•  PS3 – Keypoints and features – SIFT Descriptors – SIFT Matching

•  Projects – OpenCV with Matlab

11/09/2012 3


Review: Optical Flow

•  Overview •  Review of Theory •  Applications

11/09/2012 4


Optical Flow

11/09/2012 5 Learning OpenCV, page 322


Optical Flow

•  A low level motion tracking algorithm that tracks apparent motion from frame to frame of a video.

•  Uses brightness patterns as features, not corners.

•  Key Assumptions – Brightness consistency:

11/09/2012 6

I(x, y, t� 1) = I (x+ u(x, y), y + v(x, y), t)


Optical Flow

•  Key Assumptions Cont. – Spatial Coherence

•  Neighboring points are from the same surface and will have similar motion.

–  Small Motion •  The points do not move

very far between frames

11/09/2012 7 * Image from Michael Black, CS143 2003


Optical Flow

•  Taylor Expansion -> Brightness consistency

•  We have one equation and two unknowns, motion perpendicular to the gradient, parallel to an edge is ambiguously, Aperture problem/Barber pole illusion.

11/09/2012 8

I (x+ u, y + v, t) ⇡ I(x, y, t� 1) + I

x

· u+ I

y

· v + I

t

I (x+ u, y + v, t)� I(x, y, t� 1) = I

x

·+I

y

· v + I

t

I

x

· u+ I

y

· v + I

t

⇡ 0

5I [u v]T + I

t

= 0

Review 6 - !Kevin Wong! 11/09/2012 9


Lucas-Kanade Flow

•  Solvable when ATA is invertible – Eigenvectors of ATA describe the patch

•  Good patches turn out to be Harris corners.

•  Improvements? – Spatial consistency doesn’t really hold across

the scene. •  Solution? Motion segmentation: Divide the region

into blocks, cluster pixel movement, reassign pixels to closets clusters and repeat.

11/09/2012 10


Optical flow applications

•  Object tracking via motion segmentation •  Traffic analysis •  Crowd flow/behavior analysis.

11/09/2012 11


SIFT

•  Motivations •  What makes a point “interesting” •  What to do with Keypoints •  SIFT detector •  SIFT descriptor •  Object Recognition with SIFT

11/11/2011 12


Where are we?

11/11/2011

Scene Object Local Patches Pixels Camera Models

13

CV

Tim

elin

e


What to identify? (Keypoints)

•  Salient points that would be present across a set of (reasonably) related images – Repeatable – Distinctive

11/11/2011 14


What information? (Descriptors)

11/11/2011 15


SIFT

•  Ubiquitous: 10,000+ Citations •  David Lowe, 2001 and 2004, from UBC •  Both a detector and a descriptor •  Invariant to:

–  Illumination – Scale – Rotation – Affine – Perspective

11/11/2011 16


SIFT: Keypoint Detector •  Difference of Gaussians using Scale Space

Pyramid •  Section 3 and 4 of Lowe, 2004

11/11/2011 17

−

=


Extract and Prune max(DoG)

•  A point is extreme if it larger/smaller than its 26 neighboring points

•  Prune for: – Low Contrast – Edge Points

11/11/2011 18


What can we do with keypoints?

11/11/2011 19


SIFT Descriptor •  Sections 5 and 6 in Lowe, 2004. •  Inspired by neurological research and models •  Keypoint is center of square patch of pixels, blurred at the scale of the

keypoint •  Construction of Orientation Histogram for each 4x4 set of pixels •  All pixels are rotated by the orientation of the keypoint

11/11/2011 20


Steps of finding descriptor (for PS3) Given a keypoint P 1.  Find gradient:

1.  Apply Gaussian filter to the image using the scale of P as sigma (standard deviation)

2.  Find gradient in x and y directions (down - up, right - left) 3.  Magnitude: sqrt(Ix2+Iy2); Theta: atan2(Iy, Ix) 4.  Account for keypoint orientation.

2.  Find the patch associated with P 3.  For each bin b of the 4 x 4 bins in the patch

1.  Initialize the histogram of the 8 angle bins 2.  For each pixel p in b

1.  Find the angle bin it falls into 2.  In the histogram, increase the value of this angle bin by the gradient

magnitude of p 3.  Append this histogram to the end of the descriptor

11/11/2011 21


Many little details •  Many, many experimentally determined

parameters for descriptor. •  The standard patch size is 16

–  Feature Vector = 16 patches * 8 bins = 128 elements •  Normalize each feature vector •  Avoid large (and potentially erroneous) gradients

by capping each element at 0.2 •  Add extra blur for camera sensor •  [More steps…] •  We try to make the process as linear as

possible…

11/11/2011 22


The Matching Problem

•  Locate arbitrary objects despite environmental difficulties

11/11/2011 23


Object Recognition with SIFT (Sec 7, Lowe 2004)

•  Step 1: Feature matching •  Step 2: Hough transform in pose space •  Step 3: Geometric verification via affine

transformation

11/11/2011 24


Step 1: Feature matching

•  A match is determined by distance between closest neighbor and second closest neighbor – Euclidean distance between descriptor

vectors •  Nearest Neighbor problem

– k-d tree is inefficient – Best-Bin-First (BBF) (Beis and Lowe, 1997),

modified from k-d tree, giving approximated result

11/11/2011 25


Step 2: Hough transform in pose space

•  Goal: given test image and training image, find object pose

•  Input: Descriptor matches (pi -> pi’) of an object, many are false matches

•  Output: estimated object pose

•  A match is specified by 4 parameters <x, y, scale, orientation>

11/11/2011 26



1.  Discrete bins in 4D space; 2.  Assign each match to the bin whose

object pose is consistent with it; 3.  Find bins with > 3 votes.

11/11/2011 27



•  Have to use broad bins since there are only 4 parameters but 6 dof – bin size of 30 degrees for orientation, a factor

of 2 for scale, etc. For the problem, you can use a uniform bins, which isn’t optimal, good enough for the problem.

11/11/2011 28


Step 3: Geometric verification via affine transformation

•  Verify each bin with at least 3 entries •  3 matches determine an affine transformation

(An approximation of finding the fundamental matrix which requires more matches)

11/11/2011 29


Parameters of Affine Transformation

•  Affine transformation:

•  Least-squares solution for the best affine projection parameters

11/11/2011 30


Results

11/11/2011 31


Object Matching in PS3

•  PS3 uses a simplified version of object matching – Match on keypoints

descriptors – Use Hough to determine

bounding box parameters.

11/11/2011 32

http

://w

ww

.cs.

was

hing

ton.

edu/

hom

es/a

nkit/

rese

arch

_pro

ject

s/su

ra/s

ura-

web

site

/sift

.htm


Projects •  OpenCV with Matlab •  http://xanthippi.ceid.upatras.gr/people/evangelidis/

matlab_opencv/ –  Guide to building OpenCV Mex files for windows that

can be called in Matlab. –  Note that the Guide is for OpenCV 2.1. –  Macs: Tested using OSX 10.8, Xcode 4.5, OpenCV

2.4.2. •  Other options:

–  Use files for I/O to standalone OpenCV applications –  Use OpenCV exclusively, with CGAL for potential

matrix operations.

11/09/2012 33

cs231a section 6: problem set 3 - artificial...

Documents