![Page 1: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/1.jpg)
Generic object recognitionMay 19th, 2015
Yong Jae LeeUC Davis
![Page 2: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/2.jpg)
Announcements
• PS3 out; due 6/3, 11:59 pm• Sign attendance sheet (3rd one)
2
![Page 3: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/3.jpg)
Indexing local features
…
Kristen Grauman3
![Page 4: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/4.jpg)
Visual words• Map high-dimensional descriptors to tokens/words by
quantizing the feature space
Descriptor’s feature space
• Quantize via clustering, let cluster centers be the prototype “words”
• Determine which word to assign to each new image region by finding the closest cluster center.
Word #2
Kristen Grauman4
![Page 5: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/5.jpg)
Visual words• Example: each
group of patches belongs to the same visual word
Figure from Sivic & Zisserman, ICCV 2003 Kristen Grauman5
![Page 6: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/6.jpg)
Inverted file index
• Database images are loaded into the index mapping words to image numbers
Kristen Grauman6
![Page 7: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/7.jpg)
• New query image is mapped to indices of database images that share a word.
Inverted file indexWhen will this give us a significant gain in efficiency?
Kristen Grauman7
![Page 8: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/8.jpg)
Bags of visual words
• Summarize entire image based on its distribution (histogram) of word occurrences.
• Analogous to bag of words representation commonly used for documents.
8
![Page 9: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/9.jpg)
Comparing bags of words• Rank frames by normalized scalar product between their
(possibly weighted) occurrence counts---nearest neighborsearch for similar images.
[5 1 1 0][1 8 1 4]
jd q
𝑠𝑠𝑠𝑠𝑠𝑠 𝑑𝑑𝑗𝑗 ,𝑞𝑞 =𝑑𝑑𝑗𝑗 , 𝑞𝑞𝑑𝑑𝑗𝑗 𝑞𝑞
=∑𝑖𝑖=1𝑉𝑉 𝑑𝑑𝑗𝑗 𝑠𝑠 ∗ 𝑞𝑞(𝑠𝑠)
∑𝑖𝑖=1𝑉𝑉 𝑑𝑑𝑗𝑗(𝑠𝑠)2 ∗ ∑𝑖𝑖=1𝑉𝑉 𝑞𝑞(𝑠𝑠)2
for vocabulary of V words
Kristen Grauman9
![Page 10: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/10.jpg)
Application: Large-Scale Retrieval
[Philbin CVPR’07]
Query Results from 5k Flickr images (demo available for 100k set)10
![Page 11: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/11.jpg)
Spatial Verification: two basic strategies
• RANSAC– Typically sort by BoW similarity as initial filter– Verify by checking support (inliers) for possible
transformations • e.g., “success” if find a transformation with > N inlier
correspondences
• Generalized Hough Transform– Let each matched feature cast a vote on location,
scale, orientation of the model object – Verify parameters with enough votes
Kristen Grauman11
![Page 12: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/12.jpg)
RANSAC verification
12
![Page 13: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/13.jpg)
Voting: Generalized Hough Transform• If we use scale, rotation, and translation invariant local
features, then each feature match gives an alignment hypothesis (for scale, translation, and orientation of model in image).
Model Novel image
Adapted from Lana Lazebnik13
![Page 14: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/14.jpg)
Voting: Generalized Hough Transform• A hypothesis generated by a single match may be unreliable,• So let each match vote for a hypothesis in Hough space
Model Novel image14
![Page 15: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/15.jpg)
China is forecasting a trade surplus of $90bn (£51bn) to $100bn this year, a threefold increase on 2004's $32bn. The Commerce Ministry said the surplus would be created by a predicted 30% jump in exports to $750bn, compared with a 18% rise in imports to $660bn. The figures are likely to further annoy the US, which has long argued that China's exports are unfairly helped by a deliberately undervalued yuan. Beijing agrees the surplus is too high, but says the yuan is only one factor. Bank of China governor Zhou Xiaochuan said the country also needed to do more to boost domestic demand so more goods stayed within the country. China increased the value of the yuan against the dollar by 2.1% in July and permitted it to trade within a narrow band, but the US wants the yuan to be allowed to trade freely. However, Beijing has made it clear that it will take its time and tread carefully before allowing the yuan to rise further in value.
China, trade, surplus, commerce,
exports, imports, US, yuan, bank, domestic,
foreign, increase, trade, value
What else can we borrow from text retrieval?
15
![Page 16: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/16.jpg)
tf-idf weighting• Term frequency – inverse document frequency• Describe frame by frequency of each word within it,
downweight words that appear often in the database• (Standard weighting for text retrieval)
Total number of documents in database
Number of documents word i occurs in, in whole database
Number of occurrences of word i in document d
Number of words in document d
Kristen Grauman16
![Page 17: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/17.jpg)
Query expansion
Query: golf green
Results:
- How can the grass on the greens at a golf course be so perfect?- For example, a skilled golfer expects to reach the green on a par-four hole in ...- Manufactures and sells synthetic golf putting greens and mats.
Irrelevant result can cause a `topic drift’:
- Volkswagen Golf, 1999, Green, 2000cc, petrol, manual, , hatchback, 94000miles, 2.0 GTi, 2 Registered Keepers, HPI Checked, Air-Conditioning, Front and Rear Parking Sensors, ABS, Alarm, Alloy
Slide credit: Ondrej Chum17
![Page 18: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/18.jpg)
Query expansion
…
Query image
Results
New query
Spatial verification
New results
Chum, Philbin, Sivic, Isard, Zisserman: Total Recall…, ICCV 2007Slide credit: Ondrej Chum
18
![Page 19: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/19.jpg)
Recognition via alignment
Pros: • Effective when we are able to find reliable features
within clutter• Great results for matching specific instances
Cons:• Scaling with number of models• Spatial verification as post-processing – not seamless,
expensive for large-scale problems• Not suited for generic category recognition
Kristen Grauman19
![Page 20: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/20.jpg)
Summary
• Matching local invariant features– Useful to find objects and scenes
• Bag of words representation: quantize feature space to make discrete set of visual words– Summarize image by distribution of words– Index individual words
• Inverted index: pre-compute index to enable faster search at query time
• Recognition of instances via alignment: matching local features followed by spatial verification– Robust fitting : RANSAC, GHT
Kristen Grauman20
![Page 21: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/21.jpg)
Making the Sky Searchable:Fast Geometric Hashing for
Automated Astrometry
Sam Roweis, Dustin Lang & Keir MierleUniversity of Toronto
David Hogg & Michael BlantonNew York University
2121
![Page 22: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/22.jpg)
Example
A shot of the Great Nebula, by Jerry Lodriguss (c.2006), from astropix.comhttp://astrometry.net/gallery.html 22
![Page 23: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/23.jpg)
Example
An amateur shot of M100, by Filippo Ciferri (c.2007) from flickr.comhttp://astrometry.net/gallery.html 23
![Page 24: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/24.jpg)
Example
A beautiful image of Bode's nebula (c.2007) by Peter Bresseler, from starlightfriend.de http://astrometry.net/gallery.html 24
![Page 25: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/25.jpg)
Today
• Generic object recognition
25
![Page 26: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/26.jpg)
What does recognition involve?
Source: Fei-Fei Li, Rob Fergus, Antonio Torralba.26
![Page 27: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/27.jpg)
Verification: is that a lamp?
Source: Fei-Fei Li, Rob Fergus, Antonio Torralba.27
![Page 28: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/28.jpg)
Detection: are there people?
Source: Fei-Fei Li, Rob Fergus, Antonio Torralba.28
![Page 29: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/29.jpg)
Identification: is that Potala Palace?
Source: Fei-Fei Li, Rob Fergus, Antonio Torralba.29
![Page 30: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/30.jpg)
Object categorization
mountain
buildingtree
banner
vendorpeople
street lamp
Source: Fei-Fei Li, Rob Fergus, Antonio Torralba.30
![Page 31: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/31.jpg)
Scene and context categorization
• outdoor• city• …
Source: Fei-Fei Li, Rob Fergus, Antonio Torralba.31
![Page 32: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/32.jpg)
Instance-level recognition problem
John’s car
32
![Page 33: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/33.jpg)
Generic categorization problem
33
![Page 34: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/34.jpg)
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Visu
al O
bjec
t Rec
ogni
tion
Tuto
rial
K. Grauman, B. Leibe
Object Categorization
• Task Description “Given a small number of training images of a category,
recognize a-priori unknown instances of that category and assign the correct category label.”
• Which categories are feasible visually?
Germanshepherd
animaldog livingbeing
“Fido”
34
![Page 35: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/35.jpg)
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Visu
al O
bjec
t Rec
ogni
tion
Tuto
rial
K. Grauman, B. Leibe
Visual Object Categories
• Basic Level Categories in human categorization [Rosch 76, Lakoff 87] The highest level at which category members have similar
perceived shape The highest level at which a single mental image reflects the
entire category The level at which human subjects are usually fastest at
identifying category members The first level named and understood by children The highest level at which a person uses similar motor actions
for interaction with category members
35
![Page 36: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/36.jpg)
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Visu
al O
bjec
t Rec
ogni
tion
Tuto
rial
K. Grauman, B. Leibe
Visual Object Categories
• Basic-level categories in humans seem to be defined predominantly visually.
• There is evidence that humans (usually)start with basic-level categorization before doing identification.
Basic level
Individual level
Abstract levels
“Fido”
dog
animal
quadruped
Germanshepherd
Doberman
cat cow
…
…
……
… …
36
![Page 37: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/37.jpg)
How many object categories are there?
Biederman 1987Source: Fei-Fei Li, Rob Fergus, Antonio Torralba.
37
![Page 38: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/38.jpg)
38
![Page 39: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/39.jpg)
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Visu
al O
bjec
t Rec
ogni
tion
Tuto
rial
K. Grauman, B. Leibe
Other Types of Categories
• Functional Categories e.g. chairs = “something you can sit on”
39
![Page 40: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/40.jpg)
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Visu
al O
bjec
t Rec
ogni
tion
Tuto
rial
K. Grauman, B. Leibe
Other Types of Categories
• Ad-hoc categories e.g. “something you can find in an office environment”
40
![Page 41: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/41.jpg)
Why recognition?– Recognition a fundamental part of perception
• e.g., robots, autonomous agents
– Organize and give access to visual content• Connect to information • Detect trends and themes
41
![Page 42: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/42.jpg)
Posing visual queries
Kooaba, Bay & Quack et al.
Yeh et al., MIT
Belhumeur et al.
42
![Page 43: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/43.jpg)
Autonomous agents able to detect objects
43
![Page 44: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/44.jpg)
Finding visually similar objects
44
![Page 45: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/45.jpg)
Discovering visual patterns
Sivic & Zisserman
Lee & Grauman
Wang et al.
Objects
Actions
Categories
Kristen Grauman
45
![Page 46: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/46.jpg)
Auto-annotation
Gammeter et al. T. Berg et al.
Kristen Grauman
46
![Page 47: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/47.jpg)
Challenges: robustness
Illumination Object pose Clutter
ViewpointIntra-class appearance
Occlusions
Kristen Grauman
47
![Page 48: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/48.jpg)
Challenges: robustness
Realistic scenes are crowded, cluttered, have overlapping objects.
48
![Page 49: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/49.jpg)
Challenges: importance of context
slide credit: Fei-Fei, Fergus & Torralba
49
![Page 50: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/50.jpg)
Challenges: importance of context
50
![Page 51: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/51.jpg)
51
6 billion images 70 billion images 1 billion images served daily
10 billion images
100 hours uploaded per minute
Almost 90% of web traffic is visual!
Challenges: complexity
![Page 52: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/52.jpg)
Challenges: complexity• Thousands to millions of pixels in an image• 30+ degrees of freedom in the pose of articulated
objects (humans)• About half of the cerebral cortex in primates is
devoted to processing visual information [Fellemanand van Essen 1991]
52
Kristen Grauman
![Page 53: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/53.jpg)
Challenges: learning with minimal supervision MoreLess
Kristen Grauman53
![Page 54: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/54.jpg)
What works most reliably today
• Reading license plates, zip codes, checks
Source: Lana Lazebnik
54
![Page 55: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/55.jpg)
What works most reliably today
• Reading license plates, zip codes, checks• Fingerprint recognition
Source: Lana Lazebnik
55
![Page 56: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/56.jpg)
What works most reliably today
• Reading license plates, zip codes, checks• Fingerprint recognition• Face detection
Source: Lana Lazebnik
56
![Page 57: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/57.jpg)
What works most reliably today• Reading license plates, zip codes, checks• Fingerprint recognition• Face detection• Recognition of flat textured objects (CD covers, book
covers, etc.)
Source: Lana Lazebnik
57
![Page 58: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/58.jpg)
What works most reliably today• Reading license plates, zip codes, checks• Fingerprint recognition• Face detection• Recognition of flat textured objects (CD covers, book
covers, etc.)• Recognition of generic categories beginning to work!
58
![Page 59: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/59.jpg)
Generic category recognition:basic framework
• Build/train object model
– Choose a representation
– Learn or fit parameters of model / classifier
• Generate candidates in new image
• Score the candidates
Kristen Grauman59
![Page 60: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/60.jpg)
Generic category recognition:representation choice
Window-based Part-based
Kristen Grauman60
![Page 61: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/61.jpg)
Supervised classification• Given a collection of labeled examples, come up with a
function that will predict the labels of new examples.
• How good is some function we come up with to do the classification?
• Depends on– Mistakes made– Cost associated with the mistakes
“four”
“nine”
?Training examples Novel input
Kristen Grauman
61
![Page 62: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/62.jpg)
Supervised classification• Given a collection of labeled examples, come up with a
function that will predict the labels of new examples.
• Consider the two-class (binary) decision problem– L(4→9): Loss of classifying a 4 as a 9– L(9→4): Loss of classifying a 9 as a 4
• Risk of a classifier s is expected loss:
• We want to choose a classifier so as to minimize this total risk
( ) ( ) ( ) ( )49 using|49Pr94 using|94Pr)( →→+→→= LsLssR
Kristen Grauman
62
![Page 63: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/63.jpg)
Supervised classification
Feature value x
If we choose class “four” at boundary, expected loss is:
If we choose class “nine” at boundary, expected loss is:
4)(9 )|9 is class(4)(4) | 4 is (class4)(9 )|9 is class(
→=→+→=
LPLPLP
xxx
9)(4 )|4 is class( →= LP xKristen Grauman
63
Optimal classifier will minimize total risk.
At decision boundary, either choice of label yields same expected loss.
![Page 64: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/64.jpg)
Supervised classification
Feature value x
So, best decision boundary is at point x where
To classify a new point, choose class with lowest expected loss; i.e., choose “four” if
9)(4) |4 is P(class4)(9 )|9 is class( →=→ LLP xx
)49()|9()94()|4( →>→ LPLP xxKristen Grauman
64
Optimal classifier will minimize total risk.
At decision boundary, either choice of label yields same expected loss.
![Page 65: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/65.jpg)
Supervised classification
Feature value x
So, best decision boundary is at point x where
To classify a new point, choose class with lowest expected loss; i.e., choose “four” if
9)(4) |4 is P(class4)(9 )|9 is class( →=→ LLP xx
)49()|9()94()|4( →>→ LPLP xxHow to evaluate these probabilities?
P(4 | x) P(9 | x)
Kristen Grauman
65
Optimal classifier will minimize total risk.
At decision boundary, either choice of label yields same expected loss.
![Page 66: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/66.jpg)
ProbabilityBasic probability
• X is a random variable• P(X) is the probability that X achieves a certain value
•
• or
• Conditional probability: P(X | Y)– probability of X given that we already know Y
continuous X discrete X
called a PDF-probability distribution/density function
Source: Steve Seitz
66
![Page 67: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/67.jpg)
Example: learning skin colors• We can represent a class-conditional density using a
histogram (a “non-parametric” distribution)
Feature x = Hue
P(x|skin)
Feature x = Hue
P(x|not skin)
Percentage of skin pixels in each bin
Kristen Grauman67
![Page 68: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/68.jpg)
Example: learning skin colors• We can represent a class-conditional density using a
histogram (a “non-parametric” distribution)
Feature x = Hue
P(x|skin)
Feature x = Hue
P(x|not skin)Now we get a new image, and want to label each pixel as skin or non-skin. What’s the probability we care about to do skin detection?
Kristen Grauman68
![Page 69: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/69.jpg)
Bayes rule
)()()|()|(
xPskinPskinxPxskinP =
posterior priorlikelihood
)()|( )|( skinPskinxPxskinP α
Where does the prior come from?
Why use a prior?69
![Page 70: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/70.jpg)
Example: classifying skin pixelsNow for every pixel in a new image, we can estimate probability that it is generated by skin.
Classify pixels based on these probabilities
Brighter pixels higher probability of being skin
70
![Page 71: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/71.jpg)
Gary Bradski, 1998
Example: classifying skin pixels
Using skin color-based face detection and pose estimation as a video-based interface
72
![Page 72: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/72.jpg)
Supervised classification
• Want to minimize the expected misclassification• Two general strategies
– Use the training data to build representative probability model; separately model class-conditional densities and priors (generative)
– Directly construct a good decision boundary, model the posterior (discriminative)
73
![Page 73: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/73.jpg)
Coming up
Face detectionCategorization with local features and part-based modelsDeep convolutional neural networks
74
![Page 74: Generic object recognition - University of California, …yjlee/teaching/ecs189g...Generic object recognition May 19th, 2015 Yong Jae Lee UC Davis Announcements • PS3 out; due 6/3,](https://reader036.vdocument.in/reader036/viewer/2022071014/5fccacf75a2b980d9b3a06d8/html5/thumbnails/74.jpg)
Questions?
See you Thursday!
75