image representation and description - columbia...
TRANSCRIPT
4/15/2008 1
Lecture 10 (4.14.07)
Image Representation and Description
Shahram Ebadollahi
DIP ELEN E4830
4/15/2008 2
Image
Description
Recognition
High-level Image Representation
Image Understanding
4/15/2008 3
Lecture Outline
� Image Description� Shape Descriptors� Texture & Texture Descriptors� SIFT� Motion Descriptors� Color Descriptors
4/15/2008 4
Shape Description
� Shape Represented by its Boundary� Shape Numbers, � Fourier Descriptors, � Statistical Moments
� Shape Represented by its Interior� Topological Descriptors� Moment Invariants
4/15/2008 5
Boundary Representation: (Freeman) Chain Code
Sub-sampled boundary Chain code of boundaryOriginal boundary
Chain code for 4-neighborhood
Chain code for 8-neighborhood
Boundary representation = 0766666453321212
4/15/2008 6
8-directional chain code � 00006066666666444444242222202202
Rotation normalized chain code � 0006200000006000006260000620626
Chain Code: example
Starting point normalized chain code � 00006066666666444444242222202202
First difference of chain code
4/15/2008 7
Shape Number –A boundary descriptor
4/15/2008 8
1,,2,1,0)()()( −=+= Kkkjykxks �
1,,2,1,0)()(1
0
/2 −== ∑−
=
− KueksuaK
k
Kukj�
π
1,,2,1,0)(1
)(1
0
/2 −== ∑−
=
KkeuaK
ksK
u
Kukj�
π
Fourier Descriptor
Boundary descriptor – Fourier
4/15/2008 9
50% 10% 5%
2.5% 1.25%
100%
0.63% 0.28%
Only 8 descriptors
2868 descriptors
Boundary Reconstruction using Fourier Descriptors
4/15/2008 10
Boundary Representation: Signatures
• Represent 2-D boundary shape using 1-D signature signal
4/15/2008 11
Boundary Representation: Signatures
4/15/2008 12
Boundary Description using Statistical Moments
( )∑−
=
−=1
0
)()(A
ii
nin vpmvvµ n-th moment of v
∑−
=
=1
1
)(A
iii vpvm
4/15/2008 13
� Area� Perimeter� Compactness� Circularity Ratio� Mean/Median intensity� Max/Min intensity� Normalized area
Region Descriptors - Simple
(perimeter)2/Area
r rπ
2r
b
a=2b
π4 16π5
π4/2P
ARc =
Area of circle with same perimeter as the shape
:cR
:C
1 8.054 ≈ 78.04 ≈π
4/15/2008 14
4/15/2008 15
Topological Region Descriptors• Topological properties: Properties of image preserved under rubber-sheet distortions
H: # holes in the image
C: # connected components
E = C-H: Euler Number
H=2, C=1, E=-1
H=0, C=3, E=3
H=1, C=1, E=0 H=2, C=1, E=-1
EHCFQV =−=+−
4/15/2008 16
Geometric Moment Invariants
∫ ∫= dxdyyxfyxm qppq ),( (p+q)-th 2D geometric moment
Projection of f(x,y) onto monomial qp yx
• Why use moments?
• Geometric moments of different orders represent spatial characteristics of the image intensity distribution
00m
00010
00100
/
/
mmy
mmx
==
Total intensity of image. For binary image � area
Intensity centroid
binary image � geometrical center
∑∑−
=
−
=
=1
0
1
0
),(M
x
N
y
qppq yxfyxm
4/15/2008 17
Central Moments
∑∑−
=
−
=
−−=1
0
1
000 ),()()(
M
x
N
y
qppq yxfyyxxµ
[Translation invariance]
0000 m=µ00110 == µµ
2002,µµ
11µ
Variance about the centroid
covariance
2/)2(00 )'/(' ++= qp
pqpq µµλ Scale and translation invariant
Scaled Central Moment
2' ++=
qp
pqpq α
µµ
2/)2(00)/( ++= qp
pqpq µµη
Normalized Un-Scaled Central Moment
4/15/2008 18
Moment Invariants (translation, scale, mirroring, rotation)
ηηϕ += 201
02201 ηηφ +=211
202202 4)( ηηηφ +−=
20321
212303 )()( ηηηηφ −+−=
20321
212304 )()( ηηηηφ +++=
�=5φ
�=6φ�=7φ
4/15/2008 19
Affine Transform & Affine Moment Invariants
),(' yxTx x=
),(' yxTy y=
∑∑=
−
=
=m
r
rm
k
krrk yxax
0 0
'
∑∑=
−
=
=m
r
rm
k
krrk yxby
0 0
'
In practice: bilinear transformxyayaxaax 3210' +++=xybybxbby 3210' +++=4 pairs of corresponding points
needed to find coefficients
In practice: affine transform
3 pairs of corresponding points needed to find coefficients
yaxaax 210' ++=ybxbby 210' ++=
φφ sincos' yxx +=φφ cossin' yxy +−=
axx ='byy ='
φtan' yxx +=yy ='
Rotation: Scale: Skew:
4/15/2008 20
2
]4)[()(
2
]4)[()(
2/1211
202200220
2
2/1211
202200220
1
µµµµµ
µµµµµ
+−−+=
+−++=
I
I
Principal moment of inertia:
θ
)2
(tan5.00220
111
µµµθ−
= −
2/1002
2/1001 )/(2)/(2 µµ IbIa ==
Image ellipse characterizes fundamental shape features and also 2D position and orientation
20021 /)( mII +
)/()( 2112 IIII +−
spreadness
elongation
Elliptical Shape Descriptors
4/15/2008 21
Texture - Definition
4/15/2008 22
Texture – Quantification Methods
� Statistical: compute local features at each point in image and derive a set of statistics from the distribution of local features
� 1st, 2nd, and higher-order statistics based on how many points are used to define local features
� Structural: texture is considered to be composed of “texture elements”. Properties of the “texture elements” and their spatial placement rules characterizes the texture
� Original texture can be reconstructed from its structural description
4/15/2008 23
Statistical Texture Analysis1st order statistics
image histogramfhf →
• Obtain statistics of the histogram:
Mean:
Variance:
Skewness:
Entropy:
∑−
=
1
0
)(L
i
iih
∑−
=
−1
0
2 )()(L
i
ihi µ
∑−
=
−1
0
3 )()(L
i
ihi µ
∑−
=
−1
0
)(log)(L
i
ihih Measure of variability of intensity
: average intensity
: measure of intensity contrast
4/15/2008 24
4/15/2008 25
Statistical Texture Analysis1st order statistics
imag
ehi
stog
ram
Skewness = 2.08
Entropy = 0.88
Skewness = 2.44
Entropy = 0.77
Skewness = -0.092
Entropy = 0.97stat
istic
s
4/15/2008 26
Statistical Texture Analysis2nd order statistics: Co-occurrence
� Joint gray-level histogram of pairs of pixels� 2D histogram
inmf =),( 11
jnmf =),( 22
θd
1
2
L
1 2 L
i
j
]),(,),(Pr[),( 2211),( jnmfinmfjiPd ==≈θ
),(),( jiP d θ
4/15/2008 27
Statistical Texture Analysis2nd order statistics: Co-occurrence
j
k
°0
°45
°90°135
|}),(,),(,||,0
:)()()),(),,{((|),()0,(
jnmfilkfdnlmk
NMNMnmlkjiPd
===−=−
×××∈=°=θ
|}),(,),(),,(),(
:)()()),(),,{((|),()45,(
jnmfilkfdnldmkdnldmk
NMNMnmlkjiPd
===−−==∨−=−=−
×××∈=°=θ
),()90,(
jiPd °=θ
),()135,(
jiPd °=θ
{ }� is set cardinality
4/15/2008 28
Statistical Texture Analysis2nd order statistics: Co-occurrence (example)
image Co-occurrence matrix
4/15/2008 29
Statistical Texture Analysis2nd order statistics: Co-occurrence (statistics)
Angular 2nd moment (energy):
Maximum Probability:
Contrast:
Correlation:
),(1 1
2),( jiP
L
i
L
jd∑∑
= =θ
),(max ),(,
jiPdji
θ
),(log),( ),(1 1
),( jiPjiP d
L
i
L
jd θθ∑∑
= =
−Entropy:
∑∑= =
−L
i
L
jd jiPji
1 1),( ),(λ
θκ
yx
L
iyx
L
jd jiijP
σσ
µµθ∑∑= =
−1 1
),( )],([
∑ ∑= =
=L
i
L
jdx jiPi
1 1),( ),(θµ ∑ ∑
= =
−=L
i
L
jdxx jiPi
1 1),(
2 ),()( θµσ
(measure of image homogeneity)
(measure of local variations)
(measure of image linearity)
4/15/2008 30
4/15/2008 31
Statistical Texture Analysis2nd order statistics: Difference Statistics
Angular 2nd moment (energy):
Mean:
Contrast:
)(1
0
2),( kP
L
kd∑
−
=θ
∑−
=
1
0),( )(
L
kd kkP θ
∑−
=
−1
0),(),( )(log)(
L
kdd kPkP θθEntropy:
∑−
=
1
0),(
2 )(L
kd kPk θ
∑=−
∈=
kji
Ljidd jiPkP
||
},,1{,),(),( ),()(
�
θθ is a subset of co-occurrence matrix
4/15/2008 32
Statistical Texture Analysis2nd order statistics: Autocorrelation
∑∑
∑∑
= =
−
=
−
=
++
−−=
M
k
N
l
pM
k
qN
lff
lkf
qlpkflkf
qNpM
MNqpC
1 1
2
1 1
),(
),(),(
))((),(
Large texture elements � autoccorrelation decreases slowly with increasing distance
Small texture elements � autoccorrelation decreases rapidly with increasing distance
Periodic texture elements � periodic increase & decrease in autocorrelation value
4/15/2008 33
Statistical Texture Analysis2nd order statistics: Fourier Power Spectrum
),(),( vuFyxf ↔
2|),(|),( vuFvuP =
∑=
=π
θθ
0
),(2)( rPrP
∑=
=2/
0
),()(L
r
rPP θθ
Power Spectrum
u
v
u
v
Indicator for size of dominant texture element or texture coarseness
Indicator for the directionality of the texture
Note:
}|),({| 21 vuFFC ff−=
4/15/2008 34
Law’s Texture Energy Measures
]1,2,1[3 =L ]1,0,1[3 −=E ]1,2,1[3 −−=S
]1,2,0,2,1[
]1,4,6,4,1[
]1,0,2,0,1[
]1,2,0,2,1[
]1,4,6,4,1[
5
5
5
5
5
−−−=−−=
−−=−−=
=
W
R
S
E
L
−−−−−−−−−−
=×
10201
40804
601206
40804
10201
55 SLT•Convolute different Law’s masks with image
• Compute energy statistics
4/15/2008 35
4/15/2008 36
4/15/2008 37
4/15/2008 38
4/15/2008 39
4/15/2008 40
Motion – object
1f 2f
object motion
ε≤−= |),(),(|0),( 21 jifjififjid
otherwise1
Difference image:
Cumulative difference image
absolute positive negative
No motion direction information !
∑=
−=n
kkkcum jifjifajid
11 |),(),(|),(
Tells us how often the image gray level was different from gray-level of reference image
4/15/2008 41
Motion Field
• A velocity vector is assigned to each pixel in the image
• Velocities due to relative motion between camera and the 3D scene
• Image change due to motion during a time interval dt
• Velocity field that represents 3-dimensional motion of object points across 2-dimensional image
Motion field
4/15/2008 42
Optical flow
• Motion of brightness patterns in image sequence
• Assumptions for computing optical flow:
• Observed brightness of any object point is constant over time
• Nearby points in the image plane move in a similar manner
)(),,(),,( 2∂+∂∂+
∂∂+
∂∂+=+++ Odt
t
fdy
y
fdx
x
ftyxfdttdyydxxf
)(),,( 2∂++++= Odtfdyfdxftyxf tyx
dt
dyf
dt
dxfftyxfdttdyydxxf yxt +=−⇒=+++ ),,(),,(
),(),( vudt
dy
dt
dxc ==
cfvfuff yxt .∇=+=−
Gray-level difference at same location over time is
equivalent to product of spatial gray-level difference and
velocity
known unknown
4/15/2008 43
Optical Flow Constraints
• no spatial change in brightness, induce no temporal change in brightness � no discernible motion
• motion perpendicular to local gradient induce no temporal change in brightness � no discernible motion
• motion in direction of local gradient, induce temporal change in brightness � discernible motion
• only motion in direction of local gradient induces temporal change in brightness and discernible motion
vfuff yxt +=−
4/15/2008 44
Optical flow != Motion Field
0
0
=≠
OF
MF0
0
≠=
OF
MF
4/15/2008 45
Which descriptors?Image Feature Evaluation
1. Prototype Performance� Classify (Segment) image using
different features� Evaluate which feature is optimal
(minimum classification error)
2. Figure of Merit� Establish functional distance
measurements between set of image features (large distance �low classification error)
� Bhattacharyya distance