cs231a section 6: problem set 3 - artificial...
Post on 24-Jul-2020
0 Views
Preview:
TRANSCRIPT
Review 6 - !Kevin Wong!
CS231A Section 6: Problem Set 3
Kevin Wong
11/09/2012 1
Review 6 - !Kevin Wong!
Announcements
• PS3 Due 2:15pm Tuesday, Nov 13
• Extra Office Hours: – Friday 6 – 8pm Huang Common Area,
Basement Level.
11/11/2011 2
Review 6 - !Kevin Wong!
Topics
• Review – Optical Flow
• PS3 – Keypoints and features – SIFT Descriptors – SIFT Matching
• Projects – OpenCV with Matlab
11/09/2012 3
Review 6 - !Kevin Wong!
Review: Optical Flow
• Overview • Review of Theory • Applications
11/09/2012 4
Review 6 - !Kevin Wong!
Optical Flow
11/09/2012 5 Learning OpenCV, page 322
Review 6 - !Kevin Wong!
Optical Flow
• A low level motion tracking algorithm that tracks apparent motion from frame to frame of a video.
• Uses brightness patterns as features, not corners.
• Key Assumptions – Brightness consistency:
11/09/2012 6
I(x, y, t� 1) = I (x+ u(x, y), y + v(x, y), t)
Review 6 - !Kevin Wong!
Optical Flow
• Key Assumptions Cont. – Spatial Coherence
• Neighboring points are from the same surface and will have similar motion.
– Small Motion • The points do not move
very far between frames
11/09/2012 7 * Image from Michael Black, CS143 2003
Review 6 - !Kevin Wong!
Optical Flow
• Taylor Expansion -> Brightness consistency
• We have one equation and two unknowns, motion perpendicular to the gradient, parallel to an edge is ambiguously, Aperture problem/Barber pole illusion.
11/09/2012 8
I (x+ u, y + v, t) ⇡ I(x, y, t� 1) + I
x
· u+ I
y
· v + I
t
I (x+ u, y + v, t)� I(x, y, t� 1) = I
x
·+I
y
· v + I
t
I
x
· u+ I
y
· v + I
t
⇡ 0
5I [u v]T + I
t
= 0
Review 6 - !Kevin Wong! 11/09/2012 9
Review 6 - !Kevin Wong!
Lucas-Kanade Flow
• Solvable when ATA is invertible – Eigenvectors of ATA describe the patch
• Good patches turn out to be Harris corners.
• Improvements? – Spatial consistency doesn’t really hold across
the scene. • Solution? Motion segmentation: Divide the region
into blocks, cluster pixel movement, reassign pixels to closets clusters and repeat.
11/09/2012 10
Review 6 - !Kevin Wong!
Optical flow applications
• Object tracking via motion segmentation • Traffic analysis • Crowd flow/behavior analysis.
11/09/2012 11
Review 6 - !Kevin Wong!
SIFT
• Motivations • What makes a point “interesting” • What to do with Keypoints • SIFT detector • SIFT descriptor • Object Recognition with SIFT
11/11/2011 12
Review 6 - !Kevin Wong!
Where are we?
11/11/2011
Scene Object Local Patches Pixels Camera Models
13
CV
Tim
elin
e
Review 6 - !Kevin Wong!
What to identify? (Keypoints)
• Salient points that would be present across a set of (reasonably) related images – Repeatable – Distinctive
11/11/2011 14
Review 6 - !Kevin Wong!
What information? (Descriptors)
11/11/2011 15
Review 6 - !Kevin Wong!
SIFT
• Ubiquitous: 10,000+ Citations • David Lowe, 2001 and 2004, from UBC • Both a detector and a descriptor • Invariant to:
– Illumination – Scale – Rotation – Affine – Perspective
11/11/2011 16
Review 6 - !Kevin Wong!
SIFT: Keypoint Detector • Difference of Gaussians using Scale Space
Pyramid • Section 3 and 4 of Lowe, 2004
11/11/2011 17
−
=
Review 6 - !Kevin Wong!
Extract and Prune max(DoG)
• A point is extreme if it larger/smaller than its 26 neighboring points
• Prune for: – Low Contrast – Edge Points
11/11/2011 18
Review 6 - !Kevin Wong!
What can we do with keypoints?
11/11/2011 19
Review 6 - !Kevin Wong!
SIFT Descriptor • Sections 5 and 6 in Lowe, 2004. • Inspired by neurological research and models • Keypoint is center of square patch of pixels, blurred at the scale of the
keypoint • Construction of Orientation Histogram for each 4x4 set of pixels • All pixels are rotated by the orientation of the keypoint
11/11/2011 20
Review 6 - !Kevin Wong!
Steps of finding descriptor (for PS3) Given a keypoint P 1. Find gradient:
1. Apply Gaussian filter to the image using the scale of P as sigma (standard deviation)
2. Find gradient in x and y directions (down - up, right - left) 3. Magnitude: sqrt(Ix2+Iy2); Theta: atan2(Iy, Ix) 4. Account for keypoint orientation.
2. Find the patch associated with P 3. For each bin b of the 4 x 4 bins in the patch
1. Initialize the histogram of the 8 angle bins 2. For each pixel p in b
1. Find the angle bin it falls into 2. In the histogram, increase the value of this angle bin by the gradient
magnitude of p 3. Append this histogram to the end of the descriptor
11/11/2011 21
Review 6 - !Kevin Wong!
Many little details • Many, many experimentally determined
parameters for descriptor. • The standard patch size is 16
– Feature Vector = 16 patches * 8 bins = 128 elements • Normalize each feature vector • Avoid large (and potentially erroneous) gradients
by capping each element at 0.2 • Add extra blur for camera sensor • [More steps…] • We try to make the process as linear as
possible…
11/11/2011 22
Review 6 - !Kevin Wong!
The Matching Problem
• Locate arbitrary objects despite environmental difficulties
11/11/2011 23
Review 6 - !Kevin Wong!
Object Recognition with SIFT (Sec 7, Lowe 2004)
• Step 1: Feature matching • Step 2: Hough transform in pose space • Step 3: Geometric verification via affine
transformation
11/11/2011 24
Review 6 - !Kevin Wong!
Step 1: Feature matching
• A match is determined by distance between closest neighbor and second closest neighbor – Euclidean distance between descriptor
vectors • Nearest Neighbor problem
– k-d tree is inefficient – Best-Bin-First (BBF) (Beis and Lowe, 1997),
modified from k-d tree, giving approximated result
11/11/2011 25
Review 6 - !Kevin Wong!
Step 2: Hough transform in pose space
• Goal: given test image and training image, find object pose
• Input: Descriptor matches (pi -> pi’) of an object, many are false matches
• Output: estimated object pose
• A match is specified by 4 parameters <x, y, scale, orientation>
11/11/2011 26
Review 6 - !Kevin Wong!
Step 2: Hough transform in pose space
1. Discrete bins in 4D space; 2. Assign each match to the bin whose
object pose is consistent with it; 3. Find bins with > 3 votes.
11/11/2011 27
Review 6 - !Kevin Wong!
Step 2: Hough transform in pose space
• Have to use broad bins since there are only 4 parameters but 6 dof – bin size of 30 degrees for orientation, a factor
of 2 for scale, etc. For the problem, you can use a uniform bins, which isn’t optimal, good enough for the problem.
11/11/2011 28
Review 6 - !Kevin Wong!
Step 3: Geometric verification via affine transformation
• Verify each bin with at least 3 entries • 3 matches determine an affine transformation
(An approximation of finding the fundamental matrix which requires more matches)
11/11/2011 29
Review 6 - !Kevin Wong!
Parameters of Affine Transformation
• Affine transformation:
• Least-squares solution for the best affine projection parameters
11/11/2011 30
Review 6 - !Kevin Wong!
Results
11/11/2011 31
Review 6 - !Kevin Wong!
Object Matching in PS3
• PS3 uses a simplified version of object matching – Match on keypoints
descriptors – Use Hough to determine
bounding box parameters.
11/11/2011 32
http
://w
ww
.cs.
was
hing
ton.
edu/
hom
es/a
nkit/
rese
arch
_pro
ject
s/su
ra/s
ura-
web
site
/sift
.htm
Review 6 - !Kevin Wong!
Projects • OpenCV with Matlab • http://xanthippi.ceid.upatras.gr/people/evangelidis/
matlab_opencv/ – Guide to building OpenCV Mex files for windows that
can be called in Matlab. – Note that the Guide is for OpenCV 2.1. – Macs: Tested using OSX 10.8, Xcode 4.5, OpenCV
2.4.2. • Other options:
– Use files for I/O to standalone OpenCV applications – Use OpenCV exclusively, with CGAL for potential
matrix operations.
11/09/2012 33
top related