evaluation of color descriptors for object and scene
TRANSCRIPT
Evaluation of Color Descriptors for Object and Scene Recognition
Authors: Koen E. A. van de Sande, Theo Gevers, and Cees G. M. Snoek @
University of AmsterdamPresenter: Shao-Chuan Wang
Evaluation of Color Descriptors for Object and Scene Recognition
• Focus: Color features/descriptors on obj. and scene recognition
• Summary: The invariance of photometric transform aces and its effect on discriminative power.
• Conclusion: The usefulness of invariance is category-specific!
Photometric transforms (1/2)
• Light intensity scale invariant
• Light intensity shift invariant
B
G
R
a
a
a
00
00
00
3
2
1
o
o
o
B
G
R
Photometric transforms (1/2)
• Light intensity scale and shift invariant
• Light color change
• Light color change and shift
3
2
1
00
00
00
o
o
o
B
G
R
a
a
a
B
G
R
c
b
a
00
00
00
3
2
1
00
00
00
o
o
o
B
G
R
c
b
a
Color Descriptors (1/1)
• Histograms – RGB, Hue, Saturation, rgHistogram, …
• Color Moments: contain spatial info.
• Color SIFT: combined color and SIFT– HSV-SIFT, Hue-SIFT, …
dxdyyxIyxIyxIyxM cB
bG
aR
qpabcpq )],([)],([)],([
Color Histograms (1/2)
• RGB-histogram• Hue-histogram– H and S are scale-invariant and
shift-invariant w.r.t light intensity
• rg-histogram– Scale-invariant– Not shift-invariant– Not that b is redudant
BGRB
BGRG
BGRR
b
g
r
Image from wikipedia
Color Histograms (2/2)
• Transformed color– Scale and shift-invariant w.r.t
light intensity.
• Opponent color histogram– O1,O2 shift invariant– O3: intensity, no invariant
B
B
G
G
R
R
B
G
R
B
G
R
3
622
3
2
1
BGR
BGR
GR
O
O
O
Color SIFT Descriptors (1/3)
• HSV-SIFT– SIFT over HSV channels– Hue is unstable in gray axis
• Hue-SIFT (Van de Weijer 2006)
– Used Hue histogram weighing by its saturation
– Concatenate Hue histogram with SIFT– Only SIFT is invariant; Hue histogram is not!
(partial invariance)
Hue Instability
Color SIFT Descriptors (2/3)
• OpponentSIFT– SIFT over all channels in the opponent color space.– Shift-invariant to light intensity.
• W-SIFT– Eliminate O1 and O2’s intensity information– Scale-invariant to light intensity
• rg-SIFT– SIFT over r,g spaces– Scale and shift invariant, but not invariant to light color
changes/shifts
32
31
2
1
OO
OO
O
O
Color SIFT Descriptors (3/3)
• Transformed color SIFT– SIFT over normalized transformed channels.– Scale- and shift-invariant to light color changes
and shift.
Experiments
• Implementation:– Scale-invariants points are detected by Harris-
Laplace point detectors– Color descriptors are computed over the area
around the points; all regions are proportionally re-sampled to a uniform 60x60 patch.
– Cluster descriptors with k = 40 (images) k = 4000 (video)
– SVM classifier with EMD/chi-square kernel
Benchmark (1/3)
• Image: PASCAL VOC 2007, over 20 object categories
Benchmark (2/3)
• Most objs were categorized better under scale- and shift- invariant to light intensity
• Some, such as car and dining table, do not benefit from such invariance.
Benchmark (3/3)
• Video: Mediamill Challenge, 39 object and scene categories
Evaluation of Color Descriptors for Object and Scene Recognition
• Conclusion:– W-SIFT and rgSIFT, in general, outperform other
color descriptors.– Light intensity info. Is important for some
categories– Usefulness of invariance is category-specific.