![Page 1: Robust texture image representation by scale selective](https://reader031.vdocument.in/reader031/viewer/2022012201/61de3643a3a2135e2f5623c9/html5/thumbnails/1.jpg)
郭振华清华大学深圳研究生院
Robust texture image
representation by scale
selective local binary patterns
(TIP2016)
![Page 2: Robust texture image representation by scale selective](https://reader031.vdocument.in/reader031/viewer/2022012201/61de3643a3a2135e2f5623c9/html5/thumbnails/2.jpg)
清华大学深圳研究生院
清华大学深圳研究生院成立于2001年。为清华大学唯一的异地办学机构。同一学校,同一品牌,秉承同一文化传统。
有在校全日制研究生3000余人,其中博士生380余人。
有专职教师150余人,博士后80余人,双基地教师280余人,兼职教师40余人。
2http://www.sz.tsinghua.edu.cn
![Page 3: Robust texture image representation by scale selective](https://reader031.vdocument.in/reader031/viewer/2022012201/61de3643a3a2135e2f5623c9/html5/thumbnails/3.jpg)
设生命与健康、能源与环境、信息科学与技术、物流与交通、先进制造、海洋科学与技术、社会科学与管理七个学部。
着力发展信息、先进制造、网络与媒体技术、环境、材料、新能源、物流、海洋等学科。
校园建筑面积10万平米,创新基地规划10万平米。
3
清华大学深圳研究生院
![Page 4: Robust texture image representation by scale selective](https://reader031.vdocument.in/reader031/viewer/2022012201/61de3643a3a2135e2f5623c9/html5/thumbnails/4.jpg)
Outline
Texture definition and challenge
Texton (Statistical vs. Binary)
Overview of LBP and CLBP
Proposed SSLBP
Experimental Results and Discussion
4
![Page 5: Robust texture image representation by scale selective](https://reader031.vdocument.in/reader031/viewer/2022012201/61de3643a3a2135e2f5623c9/html5/thumbnails/5.jpg)
Texture Definition
Definition
Wiki Dictionary:
The feel or shape of a surface or substance;
the smoothness, roughness, softness, etc. of
something.
In fact, the definition of texture is still an open
issue.
![Page 6: Robust texture image representation by scale selective](https://reader031.vdocument.in/reader031/viewer/2022012201/61de3643a3a2135e2f5623c9/html5/thumbnails/6.jpg)
Texture is everywhere
6
草地(纹理)
树林(纹理)
楼房(纹理)
天空(纹理)
![Page 7: Robust texture image representation by scale selective](https://reader031.vdocument.in/reader031/viewer/2022012201/61de3643a3a2135e2f5623c9/html5/thumbnails/7.jpg)
Wide Application
7
Desert vs Mountain
Normal vs Abnormal
Defect Detection
![Page 8: Robust texture image representation by scale selective](https://reader031.vdocument.in/reader031/viewer/2022012201/61de3643a3a2135e2f5623c9/html5/thumbnails/8.jpg)
Common issues
Lightness, rotation and scale
8
![Page 9: Robust texture image representation by scale selective](https://reader031.vdocument.in/reader031/viewer/2022012201/61de3643a3a2135e2f5623c9/html5/thumbnails/9.jpg)
9
Structural approach: a set of texels in some regular or repeated pattern
Structural Approach
![Page 10: Robust texture image representation by scale selective](https://reader031.vdocument.in/reader031/viewer/2022012201/61de3643a3a2135e2f5623c9/html5/thumbnails/10.jpg)
10
How do you decide what is a texel?
Limitation of Structure
Approach
grass leaves
What/Where are the texels?
![Page 11: Robust texture image representation by scale selective](https://reader031.vdocument.in/reader031/viewer/2022012201/61de3643a3a2135e2f5623c9/html5/thumbnails/11.jpg)
Statistical Texton
11
![Page 12: Robust texture image representation by scale selective](https://reader031.vdocument.in/reader031/viewer/2022012201/61de3643a3a2135e2f5623c9/html5/thumbnails/12.jpg)
Binary Texton
12
Two advantages:
Fast
Insensitive to training set
![Page 13: Robust texture image representation by scale selective](https://reader031.vdocument.in/reader031/viewer/2022012201/61de3643a3a2135e2f5623c9/html5/thumbnails/13.jpg)
LBP in spatial domain
13T. Ojala, M. Pietikäinen, and T. T. Mäenpää. Multiresolution gray-scale and rotation invariant texture classification
with Local Binary Pattern. IEEE Transactions on Pattern Analysis and Machine Intelligence 24(7), pp. 971-987, 2002.
![Page 14: Robust texture image representation by scale selective](https://reader031.vdocument.in/reader031/viewer/2022012201/61de3643a3a2135e2f5623c9/html5/thumbnails/14.jpg)
LBP and contrast operators
14
![Page 15: Robust texture image representation by scale selective](https://reader031.vdocument.in/reader031/viewer/2022012201/61de3643a3a2135e2f5623c9/html5/thumbnails/15.jpg)
Circle-LBP
15
![Page 16: Robust texture image representation by scale selective](https://reader031.vdocument.in/reader031/viewer/2022012201/61de3643a3a2135e2f5623c9/html5/thumbnails/16.jpg)
Multiscale LBP
16
![Page 17: Robust texture image representation by scale selective](https://reader031.vdocument.in/reader031/viewer/2022012201/61de3643a3a2135e2f5623c9/html5/thumbnails/17.jpg)
An example of LBP image
and histogram
17
![Page 18: Robust texture image representation by scale selective](https://reader031.vdocument.in/reader031/viewer/2022012201/61de3643a3a2135e2f5623c9/html5/thumbnails/18.jpg)
Rotation Invariance (ri)
18
![Page 19: Robust texture image representation by scale selective](https://reader031.vdocument.in/reader031/viewer/2022012201/61de3643a3a2135e2f5623c9/html5/thumbnails/19.jpg)
Rotation Invariance
19
![Page 20: Robust texture image representation by scale selective](https://reader031.vdocument.in/reader031/viewer/2022012201/61de3643a3a2135e2f5623c9/html5/thumbnails/20.jpg)
Completed LBP
Central pixel and its P circularly and evenly spaced neighbours with radius R.
(a) (b) (c) (d)
(a) A 33 sample block; (b) the local differences; (c) the sign and (d) magnitude components.
24
![Page 21: Robust texture image representation by scale selective](https://reader031.vdocument.in/reader031/viewer/2022012201/61de3643a3a2135e2f5623c9/html5/thumbnails/21.jpg)
Completed LBP
21
OriginalImage
LDSMT
CLBP Map
Center Gray Level CLBP_C
CLBP_SS
CLBP_MM
Local Difference
CLBPHistogram
Classifier
![Page 22: Robust texture image representation by scale selective](https://reader031.vdocument.in/reader031/viewer/2022012201/61de3643a3a2135e2f5623c9/html5/thumbnails/22.jpg)
Representation Example
Histogram of CLBP_S of a sample. Histogram of CLBP_M of the sample.
Histogram of CLBP_S_M. Histogram of CLBP_S/M.Zhenhua Guo, Lei Zhang, David Zhang, A Completed Modeling of Local Binary Pattern Operator for Texture
Classification, IEEE Transactions on Image Processing, vol. 19, no. 6, pp. 1657-1663, 2010.
![Page 23: Robust texture image representation by scale selective](https://reader031.vdocument.in/reader031/viewer/2022012201/61de3643a3a2135e2f5623c9/html5/thumbnails/23.jpg)
Scale Variation
Scale invariance is more difficult.
Two popular ways:
Local scale invariance
Global scale invariance
23
![Page 24: Robust texture image representation by scale selective](https://reader031.vdocument.in/reader031/viewer/2022012201/61de3643a3a2135e2f5623c9/html5/thumbnails/24.jpg)
Scale Invariance (I)
24
Local scale invariance
Detect a Harris or Laplacian Region->Normalize the
region->Feature Extraction
![Page 25: Robust texture image representation by scale selective](https://reader031.vdocument.in/reader031/viewer/2022012201/61de3643a3a2135e2f5623c9/html5/thumbnails/25.jpg)
Scale Invariance (I)
25
Local scale invariance
Estimating local scale or extracting local fractal
feature
![Page 26: Robust texture image representation by scale selective](https://reader031.vdocument.in/reader031/viewer/2022012201/61de3643a3a2135e2f5623c9/html5/thumbnails/26.jpg)
Scale Invariance (II)
26
Global scale invariance
Global fractal feature
![Page 27: Robust texture image representation by scale selective](https://reader031.vdocument.in/reader031/viewer/2022012201/61de3643a3a2135e2f5623c9/html5/thumbnails/27.jpg)
Scale Invariance (II)
27
Global scale invariance
Polar transform
![Page 28: Robust texture image representation by scale selective](https://reader031.vdocument.in/reader031/viewer/2022012201/61de3643a3a2135e2f5623c9/html5/thumbnails/28.jpg)
Scale Invariance (II)
28
Global scale invariance
Scale shift matching
![Page 29: Robust texture image representation by scale selective](https://reader031.vdocument.in/reader031/viewer/2022012201/61de3643a3a2135e2f5623c9/html5/thumbnails/29.jpg)
Assumption
29
Statistical dominant local patterns provide
discriminant information for texture classification.
When an image changes scale, percentage of
dominant patterns does not change.
S. Liao, M. W. K. Law, and A. C. S. Chung. Dominant local binary patterns for texture classification. IEEE
Transactions on Image Processing 18(5), pp. 1107-1118, 2009.
![Page 30: Robust texture image representation by scale selective](https://reader031.vdocument.in/reader031/viewer/2022012201/61de3643a3a2135e2f5623c9/html5/thumbnails/30.jpg)
Algorithm 1:
Feature Learning
Step 1: for one training sample, build a scale space by a 2D
Gaussian filter;
Step 2: compute local pattern histogram for each image;
Step 3: only maximal frequency among different scale is
kept;
Step 4: compute average frequency for the whole training
set;
Step 5: dominant patterns with high frequency are learnt.
30
1
, =1
, 1< L, "*" is the convolution operator
i
l
l
f ls
s g l
1 2( )=max( ( ), ( ), ..., ( ))i Lf s s s
CLBP _ S / C CLBP _ S / C CLBP _ S / C CLBP _ S / CH k H k H k H k
( )( )= ( )+
if
CLBP _ S / CT T
CLBP _ S / C CLBP _ S / C
H kH k H k
N
![Page 31: Robust texture image representation by scale selective](https://reader031.vdocument.in/reader031/viewer/2022012201/61de3643a3a2135e2f5623c9/html5/thumbnails/31.jpg)
Algorithm 2:
Feature Extraction
Step 1: for one test sample, build a scale space by a 2D
Gaussian filter;
Step 2: compute histogram for selected patterns by algorithm
1;
Step 3: only maximal frequency among different scale is kept.
31
1
, =1
, 1< L, "*" is the convolution operatorl
l
I ls
s g l
1 2( )=max( ( ), ( ), ..., ( ))Ls s sI
CLBP _ S / C CLBP _ S / C CLBP _ S / C CLBP _ S / CDPH k DPH k DPH k DPH k
![Page 32: Robust texture image representation by scale selective](https://reader031.vdocument.in/reader031/viewer/2022012201/61de3643a3a2135e2f5623c9/html5/thumbnails/32.jpg)
Scale selective LBP
(SSLBP)
32
.
.
.
1 : Original ImageIs
Convolved by 2D
Gaussian function g
Image
Scale
Space
2
Is
I
Ls
0 05. 0 1. 0 08. 0 03. 0 02.
Feature Size: K
Feature frequency of riP ,R
T
CLBP _ M / CDP
Feature frequency
of riP ,R
T
CLBP _ S / CDP
Feature Size: K
0 14. 0 01. 0 02. 0 05. 0 0.
0 06. 0 07. 0 06. 0 05. 0 0.
Feature frequency of riP ,R
T
CLBP _ M / CDP
Feature frequency
of riP ,R
T
CLBP _ S / CDP
0 15. 0 01. 0 02. 0 01. 0 04.
Feature frequency of riP ,R
T
CLBP _ M / CDP
Feature frequency
of riP ,R
T
CLBP _ S / CDP
Feature
Scale
Space
0 15. 0 1. 0 08. 0 05. 0 04.
operationMax
0 14. 0 02. 0 07. 0 05. 0 02.
0 09. 0 02. 0 04. 0 03. 0 02.
0 13. 0 01. 0 07. 0 01. 0 01.
Output feature
for image I
Feature Size: 2K
operationMax
![Page 33: Robust texture image representation by scale selective](https://reader031.vdocument.in/reader031/viewer/2022012201/61de3643a3a2135e2f5623c9/html5/thumbnails/33.jpg)
An example
33
5 10 15 20 25 30 35 40 45 500
0.01
0.02
0.03
0.04
0.05
0.06
Pattern Index
Fre
quency(%
)
5 10 15 20 25 30 35 40 45 500
0.01
0.02
0.03
0.04
0.05
0.06
Pattern Index
Fre
quency(%
)
5 10 15 20 25 30 35 40 45 500
0.01
0.02
0.03
0.04
0.05
0.06
0.07
0.08
Pattern Index
Fre
quency(%
)
5 10 15 20 25 30 35 40 45 500
0.01
0.02
0.03
0.04
0.05
0.06
0.07
0.08
Pattern Index
Fre
quency(%
)
0.35
0.17
![Page 34: Robust texture image representation by scale selective](https://reader031.vdocument.in/reader031/viewer/2022012201/61de3643a3a2135e2f5623c9/html5/thumbnails/34.jpg)
5 Dominant Patterns
34
![Page 35: Robust texture image representation by scale selective](https://reader031.vdocument.in/reader031/viewer/2022012201/61de3643a3a2135e2f5623c9/html5/thumbnails/35.jpg)
Scale Estimation
351 2 3 4 5 6 7 8 91.7
1.75
1.8
1.85
1.9
1.95
2
2.05
2.1
Image Scale
Avera
ge F
eatu
re S
cale
Scale parameter of KTH-TIPS.
, 1, 2,...,990
z
I
I IS
z
FS
AFS z
K
kScaleIndex
FS
K
k
I
I
1
)(
![Page 36: Robust texture image representation by scale selective](https://reader031.vdocument.in/reader031/viewer/2022012201/61de3643a3a2135e2f5623c9/html5/thumbnails/36.jpg)
Texture Databases
36
Texture
Database
Name
Imaging property Image Size
Number
of
classes
Number
of
samples
CUReTThe images are captured under different illumination and
viewing directions. Fixed, 200*200 61 92
KTH-
TIPS
It extends CUReT by imaging new samples of ten of the
CUReT textures at a subset of the viewing and lighting
angles used in CUReT but also over a range of scales.
Varied,
196*201 10 81
UIUC
Textures are acquired under significant scale and viewpoint
changes, arbitrary rotations, and uncontrolled illumination
conditions, even including textures with non-rigid
deformation.
Fixed, 640*480 25 40
UMDIt has been designed in a similar way as UIUC, while the
image resolution is 4 times of UIUC.
Fixed, 1280*
96025 40
A LOTIt is systematically collected with varied viewing angles,
illumination angles, and illumination colors for each
material.
Varied, 1536*
891250 100
![Page 37: Robust texture image representation by scale selective](https://reader031.vdocument.in/reader031/viewer/2022012201/61de3643a3a2135e2f5623c9/html5/thumbnails/37.jpg)
CUReT Database
37
![Page 38: Robust texture image representation by scale selective](https://reader031.vdocument.in/reader031/viewer/2022012201/61de3643a3a2135e2f5623c9/html5/thumbnails/38.jpg)
KTH-TIPS Database
38
![Page 39: Robust texture image representation by scale selective](https://reader031.vdocument.in/reader031/viewer/2022012201/61de3643a3a2135e2f5623c9/html5/thumbnails/39.jpg)
39
UIUC Database
![Page 40: Robust texture image representation by scale selective](https://reader031.vdocument.in/reader031/viewer/2022012201/61de3643a3a2135e2f5623c9/html5/thumbnails/40.jpg)
40
UMD Database
![Page 41: Robust texture image representation by scale selective](https://reader031.vdocument.in/reader031/viewer/2022012201/61de3643a3a2135e2f5623c9/html5/thumbnails/41.jpg)
ALOT Database
41
![Page 42: Robust texture image representation by scale selective](https://reader031.vdocument.in/reader031/viewer/2022012201/61de3643a3a2135e2f5623c9/html5/thumbnails/42.jpg)
Parameters
Scale Space: 4
2D Gaussian filter: 20.25
Radius: 3, 9
Neighbor: 24
Feature extractor: CLBP_S/C, CLBP_M/C
Feature Length: 2400
NNC: NNC+Chi-square distance
Feature preprocessing: 42
( ), 1,2,...,k kH sqrt H k K
![Page 43: Robust texture image representation by scale selective](https://reader031.vdocument.in/reader031/viewer/2022012201/61de3643a3a2135e2f5623c9/html5/thumbnails/43.jpg)
Nearest Subspace
Classifier (NSC)
There are C classes of textures.
n training samples in each class, a set of
histograms for one class:
project hy into the subspace spanned by Hc :
The projection residuals is computed as:
43
1( )T T yc c c cH H H h
2
yc c cerr h H
,1 ,2 ,, ,...,c c c c nH h h h
K. Lee, J. Ho, and D. Kriegman. Acquiring linear subspaces for face recognition under variable lighting. IEEE
Transactions on Pattern Analysis and Machine Intelligence 27(5), pp. 684–698, 2005.
![Page 44: Robust texture image representation by scale selective](https://reader031.vdocument.in/reader031/viewer/2022012201/61de3643a3a2135e2f5623c9/html5/thumbnails/44.jpg)
Experimental ResultsMethod CUReT
(46)
KTH-TIPS
(40)
UIUC
(20)
UMD
(20)
ALOT
(50)
SRP [ICCV2011] (SVM) 99.37 99.29 98.56 99.30 -
RP [TPAMI2012] (NNC) 98.52 97.71 96.27 99.13 -
Caputo et al. [IVC2010] (SVM) 98.46 94.8 92.0 - -
BIF [IJCV2010] (Shift Matching NNC) 98.6 98.5 98.8 - -
OTF [CVIU2010] (SVM) - - 97.44 98.42 95.6
WMFS [TIP2013] (SVM) - - 97.62 98.68 96.94
PLS [CVPR2014](SVM) - 98.4 96.57 98.99 93.35
PFS [IVC2014](SVM) - 97.35 97.92 99.38 97.5
LEP [TIP2013] (Shift Matching NNC) - 97.56 - - -
scLBP [TIP2015] (SVM) 99.29 - 98.45 99.25 -
COV-LBPD [TIP2014] (NNC) - 98.0 - - -
RPICoLBP [TPAMI2014] (SVM) 98.4 98.4 - - -
RLBP [BMVC2013] (NNC) - - 96.7 - -
DLBP [TIP2009] (NNC) 84.93 86.99 60.73 89.87 78.38
LBPSRI [TIP2012] (NNC) 85.00 89.73 70.05 91.71 71.29
LBP [TPAMI2002] (NNC) 80.63 82.67 55.26 88.23 63.33
CLBP [TIP2010] (NNC) 97.40 97.19 93.26 98.00 93.28
SSLBP (NNC) 98.55 97.80 97.02 98.84 96.69
SSLBP (NSC) 99.51 99.39 99.36 99.46 99.7144
![Page 45: Robust texture image representation by scale selective](https://reader031.vdocument.in/reader031/viewer/2022012201/61de3643a3a2135e2f5623c9/html5/thumbnails/45.jpg)
Time Cost (I)
Method
Scale Space
Building
Pattern/
Patch
Processing
HistogrammingClassification
(NNC)
RP [TPAMI2012] - SP·SR·Ip SR·C·ST·Ip C·ST·(C·Tn)
VZ_Patch
[TPAMI2009]- - SP·C·ST·Ip C·ST·(C·Tn)
SSLBP (L-1)·Sg· Ip 2·L·P·Ip 2·L·Ip 4·K·(C·Tn)
45
Here L=4 is the size of scale space, P=24 is the number of neighbours for LBP, Ip denotes the number of pixels
per image sample, Sg denotes the size of Gaussian smooth kernel, K=600 is the number of selected dominant
patterns. SP represents the size of a local patch, usually 7*7, SR is the dimension of random projection, usually
15, C denotes the number of classes, ST represents the number of clustered textons per class, here C·ST≈4·K. Tn
is the number of training samples per class.
![Page 46: Robust texture image representation by scale selective](https://reader031.vdocument.in/reader031/viewer/2022012201/61de3643a3a2135e2f5623c9/html5/thumbnails/46.jpg)
CUReT KTH-TIPS UIUC UMD ALOT
Feature extraction of MFS
(ICCV2009) (Unit: Second)0.09 0.08 0.62 2.60 2.67
Feature extraction of VZ_MR8
(ICCV2005) (Unit: Second)1.03 0.93 9.98 37.35 40.11
Feature extraction of VZ_Patch
(TPAMI2009) (Unit: Second)12.44 11.28 96.97 309.51 346.64
Feature extraction of the
proposed scheme
(Unit: Second)
0.24 0.23 1.80 7.63 8.46
Matching (NNC)
(Unit: Millisecond)177.47 25.09 30.78 31.1 297.42
Matching (NSC)
(Unit: Millisecond)2.25 0.64 0.90 0.92 5.05
46
77
Time Cost (II)
![Page 47: Robust texture image representation by scale selective](https://reader031.vdocument.in/reader031/viewer/2022012201/61de3643a3a2135e2f5623c9/html5/thumbnails/47.jpg)
Robust to image size
Image size
\#Training
Sample
20 15 10 5
1280*960 99.46+0.46 99.31+0.51 98.81+0.70 96.41+1.28
640*480 99.71+0.26 99.48+0.37 98.77+0.68 95.68+1.45
320*240 99.38+0.45 98.80+0.69 97.41+1.06 92.74+1.84
47
Test on UMD
![Page 48: Robust texture image representation by scale selective](https://reader031.vdocument.in/reader031/viewer/2022012201/61de3643a3a2135e2f5623c9/html5/thumbnails/48.jpg)
Dominant patterns
analysis
CUReT KTH-TIPS UMD UIUC
CUReT 100% 80.33% 73% 74.54%
KTH-TIPS - 100% 76.12% 80.16%
UMD - - 100% 87.87%
UIUC - - - 100%
48
Percentage of identical dominant patterns between different training sets.
![Page 49: Robust texture image representation by scale selective](https://reader031.vdocument.in/reader031/viewer/2022012201/61de3643a3a2135e2f5623c9/html5/thumbnails/49.jpg)
Robust to pattern selection
49
![Page 50: Robust texture image representation by scale selective](https://reader031.vdocument.in/reader031/viewer/2022012201/61de3643a3a2135e2f5623c9/html5/thumbnails/50.jpg)
Discussion
Traditional methods try to extract local or
global scale invariant features.
From implementation view, extract local
scale variant feature first, then apply a
global transformation to achieve invariance.
From scale space view, instead of analyzing
scale spaces locally, analyze scale space
globally.
50
![Page 51: Robust texture image representation by scale selective](https://reader031.vdocument.in/reader031/viewer/2022012201/61de3643a3a2135e2f5623c9/html5/thumbnails/51.jpg)
Conclusion
A simple and effective method to address
scale variation issue for texture image.
Fast enough for many applications, 0.24
second for a 200*200 image.
LBP with scale selection can get promising
result for challenge databases, such as UIUC
and ALOT.
51
![Page 52: Robust texture image representation by scale selective](https://reader031.vdocument.in/reader031/viewer/2022012201/61de3643a3a2135e2f5623c9/html5/thumbnails/52.jpg)
谢谢!For any inquiry, please contact with [email protected]