fetch.php

118 May 2008 1Introduction to Statistical Pattern Recognition

Introduction to Statistical Pattern Recognition

R.P.W. Duin

Pattern Recognition GroupDelft University of Technology

The Netherlands

[email protected]


Pattern Recognition Problem

What is this? What occasion? Where are the faces? Who is who?


Pattern Recognition Problems

Which group?

A

B

C


Pattern Recognition Problems

To which classbelongs an image

To which class (segment)belongs every pixel?

Where is an object ofinterest (detection);

What is it (classification)?


Pattern Recognition BooksFukunaga, K., Introduction to statistical pattern recognition, second edition, Academic Press, 1990.

Kohonen, T., Self-organizing maps, Springer Series in Information Sciences, Volume 30, Berlin, 1995.

Ripley, B.D., Pattern Recognition and Neural Networks, Cambridge University Press, 1996.

Devroye, L., Gyorfi, L., and Lugosi, G., A probabilistic theory of pattern recognition, Springer, 1996.

Schurmann, J. Pattern classification, a unified view of statistical and neural approaches, Wiley, 1996

Gose, E., Johnsonbaugh, R., and Jost, S., Pattern Recognition and Image Analysis, Prentice-Hall, 1996.

Vapnik, V.N., Statistical Learning Theory, Wiley, New York, 1998.

Duda, R.O., Hart, P.E., and Stork, D.G. Pattern Classification, 2d Edition Wiley, New York, 2001.

Hastie, T., Tibishirani, R., Friedman, J., The Elements of Statistical Learning, Springer, Berlin, 2001.

Webb, A. Statistical Pattern Recognition Wiley, 2002.

F. van der Heiden, R.P.W. Duin, D. de Ridder, and D.M.J. Tax, Classification, Parameter Estimation, State Estimation: An Engineering Approach Using MatLab, Wiley, New York, 2004, 1-440.

Bishop, C.M., Pattern Recognition and Machine Learning, Springer 2006

Theodoridis, S. and Koutroumbas, K. Pattern Recognition, 4th ed., Academic Press, New York, 2008.

Statsoft, Electronic Textbook, http://www.statsoft.com/textbook/stathome.html18 May 2008 6Introduction to Statistical Pattern Recognition

Pattern Recognition: OCR

Kurzweil Reading Edge

Automatic text reading machine with speech synthesizer


Pattern Recognition: Traffic

Road sign detectionRoad sign recognition


Pattern Recognition: Traffic

License Plates


Pattern Recognition: Images

100 Objects


Pattern Recognition: Faces

Is he in the database? Yes!


Pattern Recognition: Speech


Pattern Recognition: Pathology

MRI Brain Image


Pattern Recognition: Pathology

0 50 100 150 200 250 3000

500

1000

1500

2000

2500

3000

Flow cytometry


Pattern Recognition: Seismics

Earthquakes


Pattern Recognition: Shape Recognition

Pattern Recognition is very often Shape Recognition: Images: B/W, grey value, color, 2D, 3D, 4D Time Signals Spectra


Pattern Recognition: Shapes

Examples of objects for different classes

Object of unknown class to be classified

A B?


Pattern Recognition System

Representation GeneralizationSensor

B

A

B

A

perimeter

area

perimeter

area

Feature Representation




B

A

B

A

pixel_1

pixe

l_2

Pixel Representation




B

A

B

A

D(x,xA1)

D(x

,xB

1)

Dissimilarity Representation




B

A

B

A

Classifier_1

B

A

B

A

B

A

B

A

Combining Classifiers

Clas

sifie

r_2


Applications

Biomedical: EEG, ECG, Rntgen, Nuclear, Tomography, Tissues, Cells, Chromosomes, Bio-informatics

Speech Recognition, Speaker Identification. Character Recognition Signature Verification Remote Sensing, Meteorology Industrial Inspection Robot Vision Digital Microscopy

Notes: not always, but very often image basedno strong models available


Requirements and GoalsSet of Examples Physical Knowledge

Representation Representation (Model)

Classification

size

complexity

cost(speed)

error

Minimize desired set of examplesMinimize amount of explicit knowledgeMinimize complexity of representationMinimize cost of recognitionMinimize probability of classification error

Pattern RecognitionSystem


Possible Object RepresentationsMeasurement samples

Feature vector

Sets of segments or primitives

Outline samples (shape)

Symbolic structures

(Attributed) graphs

Dissimilarities

Atf(t)

A -> A||BB -> B1B2B3..BnA B

C DE

F

area

perimeter

d(y,x2)

d(y,x1)



Statistics needed to solve class overlap

Test object classified as A


Classification Feature Space

1x

2xBBA


Bayes decision rule, formal

p(A|x) > p(B|x) A else B

p(x|A) p(A) > p(x|B) p(B) A else B

Bayes: p(x|A) p(A) p(x|B) p(B) p(x) p(x)>

A else B

2-class problems: S(x) = p(x|A) p(A) - p(x|B) p(B) > 0 A else B

n-class problems: Class(x) = argmax(p(x|) p())


Decisions based on densities

length

p(length | male)p(length | female)

What is the gender of somebody with this length?

p(female | length) = p(length | female) p(female) / p(length)p(male | length) = p(length | male) p(male) / p(length)Bayes: {


Decisions based on densities

length

p(length | male) p(male)p(length | female) p(female)

What is the gender of somebody with this length?

p(female | length) = p(length | female) p(female) / p(length)p(male | length) = p(length | male) p(male) / p(length)Bayes: {

p(female) = 0.3p(male) = 0.7

1.0


Bayes decision rule

p(female | length) > p(male | length) female else maleBayes:

p(length | female) p(female) p(length | male) p(male) p(length) p(length)>

p(length | female) p(female) > p(length | male) p(male) female else male

pdf estimated from training set class prior probabilitiesknown, guessed or estimated


What are Good Features?

x

A B

Good features are discriminative.

Good features are have to be defined by experts

Sometimes to be found by statistics

e.g. Fisher Criterion:

JF A B,( )A B 2

A2 B2+--------------------------=A B


Compactness

The compactness hypothesis is notsufficient for perfect classificationas dissimilar objects may be close. class overlap probabilities

Representations of real world similar objects are close. There is no ground for any generalization (induction) on representationsthat do not obey this demand.

(A.G. Arkedev and E.M. Braverman, Computers and Pattern Recognition, 1966.)

1x

2x

(perimeter)

(area)


Distances and Densities

? to be classified as

B because it is mostclose to an object A

A because the localdensity of B is larger.

1x

2x

(perimeter)

(area)

A

B?


Distances: Scaling Problem

Before scaling: D(X,A) < D(X,B) After scaling: D(X,A) > D(X,B)

perimeter

area

X B

Aarea

perimeter

X B

A


How to Scale in Case of no Natural Features

Make variances equal:

color

perimeter

color'

)colorvar(colorcolor'=

)perimetervar(perimeterperimeter'=

perimeter


Density Estimation:

What is the probability of finding an objectof class A (B) on this place in the 2D space?

What is the probability of finding an objectof class A (B) on this place in the 1D space?

1x(perimeter)

(area)

A

B?

2x

(perimeter) 1x?


The Gaussian distribution (3)

1-dimensional density:

= 2

2

2

)(21exp

21)(

xxp

Normal distribution =Gaussian distribution

Standard normaldistribution: = 0, 2 = 1

95% of data between[ - 2, + 2 ] (in 1D!)

-5 0 50

0.1

0.2

0.3

0.4

0.5


Multivariate Gaussians

k - dimensional density:

= )()(

21exp

)det(21)( 1T xGx

Gxp

k

G G == 3 13 1

1 21 2


Multivariate Gaussians (2)

G == 1 01 00 10 1

G == 3 03 00 10 1

G == 3 3 --11--11 11

G == 1 01 00 30 3


Density estimation (1)

The density is defined on the whole feature space. Around object x, the density is defined as:

Given n measured objects, e.g. persons height (m) how can we estimate p(x)?

==volume

objectsoffraction)()(dxxdPxp


Density estimation (2)

Parametric estimation: Assume a parameterized model, e.g. Gaussian Estimate parameters from data Resulting density is of the assumed form

Non parametric estimation: Assume no formal structure/model, choose approach Estimate density with chosen approach Resulting density has no formal form


Parametric estimation

Assume Gaussian model Estimate mean and covariance from data

-1.5 -1 -0.5 0 0.5 1 1.5 2 2.5-1.5

-1

-0.5

0

0.5

1

1.5

2

=

=n

iin x

1

1 )()( T1

11 xxG i

n

iin =

=


Nonparametric estimation

Histogram method:1. Divide feature space in

N2 bins2. Count the number of

objects in each bin3. Normalize:

, 1

( ) ijNij

i j

np x

n dxdy=

=

x

y

dx

dy


Parzen density estimation (1)

Fix volume of bin, vary positions of bins, add contribution of each binDefine bin-shape (kernel):

For test object z sum all bins

-2 -1 0 1 2 3 4 5 6 7-3

-2

-1

0

1

2

3

40)( >rK

= ii

hK

hnp xzz 1)(

1)( = rr dK


Parzen density estimation (2)

)(xp

x

Parzen:

With Gaussian kernel: ( )22221 exp)( hxhxK =


Parametric vs. Nonparametric

Parametric estimation, based on some model: Model parameters to estimate More samples required than parameters Model assumption could be incorrect resulting in

erroneous conclusions Non parametric estimation, hangs on data directly:

Assume no formal structure/model Almost no parameters to estimate Erroneous estimates are less likely

fetch.php

Documents