url:.../publications/courses/ece_8443/lectures/current/exam/2004/ ece 8443 – pattern recognition...

12
• URL: .../publications/courses/ece_8443/lectures/current/exam/2 004/ ECE 8443 – Pattern Recognition LECTURE 15: EXAM NO. 1 (CHAP. 2) Spring 2004 Solutions: 1a 1b 1c 2a 2b 2c 2d 2e 2f 2g

Upload: ronald-thomas

Post on 03-Jan-2016

214 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: URL:.../publications/courses/ece_8443/lectures/current/exam/2004/ ECE 8443 – Pattern Recognition LECTURE 15: EXAM NO. 1 (CHAP. 2) Spring 2004 Solutions:

• URL: .../publications/courses/ece_8443/lectures/current/exam/2004/

ECE 8443 – Pattern Recognition

LECTURE 15: EXAM NO. 1 (CHAP. 2)

• Spring 2004

• Solutions:

1a 1b 1c 2a 2b 2c 2d 2e 2f 2g

Page 2: URL:.../publications/courses/ece_8443/lectures/current/exam/2004/ ECE 8443 – Pattern Recognition LECTURE 15: EXAM NO. 1 (CHAP. 2) Spring 2004 Solutions:

Problem No. 1: Let for a two-category one-dimensional problem with

(a) Show that the minimum probability of error is

given by: where

),i

(N~)i

|x(p2

./)(P)(P 2121

du

a

ue

eP

22

2

1 212/a

Solution:

1 1 2 2

2 1

( ) ( ) ( / ) ( ) ( / )R R

P error P p x dx P p x dx The probability of error is given by:

1 2

LECTURE 15: EXAM NO. 1 (CHAP. 2)PROBLEM 1

Where R1 denotes class region and R2, . To determine R1 and R2 , the decision region must

be determined.

Calculate the decision rule with the Likelihood Ratio : For a two–category problem place in if

the likelihood ratio exceeds a threshold value . Otherwise decide

)(

)(

)|(

)|(

1

2

2

1

P

P

xp

xp

12

x

--------------------- ( 2 )

--------------------- ( 1 )

Page 3: URL:.../publications/courses/ece_8443/lectures/current/exam/2004/ ECE 8443 – Pattern Recognition LECTURE 15: EXAM NO. 1 (CHAP. 2) Spring 2004 Solutions:

The class conditional densities are

222

212

)(2

1

2

)(2

1

12

1)/(,

2

1)/(

xx

expexp

and the Prior probabilities are ./)(P)(P 2121

For a given feature , the likelihood ratio (1) can be used to determine the decision rule:

2

)(

0))(2)((

0)()(2

22

1)(

)(

2/1

2/1

2

12

1

21

1212

21

2212

222

2211

2

22

21

)(2

1

)(2

1

222

212

x

x

x

xxxx

x

x

e

e

x

x

,changing signs

,taking log

, decide 1Else If

2

)( 21 x , decide 2

x

Page 4: URL:.../publications/courses/ece_8443/lectures/current/exam/2004/ ECE 8443 – Pattern Recognition LECTURE 15: EXAM NO. 1 (CHAP. 2) Spring 2004 Solutions:

Assume , then, Let

Where,

ax

1 dadx

1 1*

( ) 2* ( ) ( / )x

P error P p x dx

21

*

1

*

1

)/(

)/(2

1*2

x

x

xp

xp

PROBLEM 1LECTURE 15: EXAM NO. 1 (CHAP. 2)

The decision region is half way between the two means. This result makes sense from an intuitive point of

view since the likelihoods are identical and differ only in their mean value.

Probability of error ( 1 ) becomes:

*

11

*

11 )/()()/()()(xx

dxxpPdxxpPerrorP

2

)(* 21

xwhere ,

211

2

2*

1( )

2

x

x

P error e dx

*

2

1

*

2

1 22

2

1

2

1)(

a

a

a

adaedaeerrorP

22* 12

121

a

Page 5: URL:.../publications/courses/ece_8443/lectures/current/exam/2004/ ECE 8443 – Pattern Recognition LECTURE 15: EXAM NO. 1 (CHAP. 2) Spring 2004 Solutions:

Combining these two results we get:

a

adaeerrorP

2

2

1

2

1)(

1 2| |

2a

We get a similar result if 2<1.

*

2

1 2

2

1)(

a

adaeerrorP

22* 21

221

aWhere,

Where,

PROBLEM 1LECTURE 15: EXAM NO. 1 (CHAP. 2)

OK. But how to be sure if this is the minimum probability of error?

P[error] in terms of the posterior P[error|x]

The optimal decision rule will minimize P[error|x] for every value of x. At each feature value x,

P[error|x] = P[ω2|x] when ω1 was chosen.

Therefore, when we integrate over the limit, the decision rule yields a minimum P[error] . This probability of

error is the Bayes Error.

dxxpxerrorPerrorP )()|(][

Page 6: URL:.../publications/courses/ece_8443/lectures/current/exam/2004/ ECE 8443 – Pattern Recognition LECTURE 15: EXAM NO. 1 (CHAP. 2) Spring 2004 Solutions:

(b)    Use the inequality to show that P e goes to zero as

goes to infinity.

22

2

122

2

1 ae

a

dt

a

te

eP

/

12

2

2

2

2

1

2

1

2

1

2

1

2

1

02

1lim

a

a

a

e

a

a

ea

daeP

ea

and

1 2| |

2a

as the distance between the means of the two distribution

tend to infinity.

0eP 1 2| |

2a

PROBLEM 1LECTURE 15: EXAM NO. 1 (CHAP. 2)

Solution:

Page 7: URL:.../publications/courses/ece_8443/lectures/current/exam/2004/ ECE 8443 – Pattern Recognition LECTURE 15: EXAM NO. 1 (CHAP. 2) Spring 2004 Solutions:

(c) How does your result for (b) change if the prior probabilities are not equal?

Making , puts infinite distance between the classes . The probability of

error will still tend to zero if the Prior probabilities are not equal.

2

2121&

PROBLEM 1LECTURE 15: EXAM NO. 1 (CHAP. 2)

Page 8: URL:.../publications/courses/ece_8443/lectures/current/exam/2004/ ECE 8443 – Pattern Recognition LECTURE 15: EXAM NO. 1 (CHAP. 2) Spring 2004 Solutions:

PROBLEM 2Problem No. 2: Given a two-class two-dimensional classification problem (x = {x1,x2}) with the following parameters

(uniform distributions):

3 / 4 1/ 4 0 11 11 1

( | ) ( | )3 / 4 1/ 4 0 11 22 2

0 0

x x

p px x

elsewhere elsewhere

x x

( ) ( ) 1/ 2.1 2P P

LECTURE 15: EXAM NO. 1 (CHAP. 2)

a) Compute the mean and covariance (hint: plot the distributions).

I. Using Independent class approach, we get joint pdf as

1

2

1

2

1

2

1

2

1

2

( )

3/ 4 01/ 2

3/ 4 1/ 4

0 1/ 41/ 2

3/ 4 0

0 1/ 41

0 1/ 4

0 1/ 41/ 2

1/ 4 1

1/ 4 11/ 2

0 1

0

p

x

x

x

x

x

x

x

x

x

x

otherwise

x

1

11

1

( )

1/ 2 3/ 4 0

1 0 1/ 4

1/ 2 1/ 4 1

0

p x

x

x

x

otherwise

2

22

2

( )

1/ 2 3/ 4 0

1 0 1/ 4

1/ 2 1/ 4 1

0

p x

x

x

x

otherwise

We can obtain marginal probability distribution function as

0

2040

6080

0

2040

60800

0.2

0.4

0.6

0.8

1

Page 9: URL:.../publications/courses/ece_8443/lectures/current/exam/2004/ ECE 8443 – Pattern Recognition LECTURE 15: EXAM NO. 1 (CHAP. 2) Spring 2004 Solutions:

PROBLEM 2LECTURE 15: EXAM NO. 1 (CHAP. 2)

1 11

2 2 2

( ) 1/ 8

1/8( )E

x p xE x

E x x p x

x

1 1 11 1 1 1

1

1 1 1 1 1 1 1

2

1 1 1 2 1 21 1 1 1 1 1 2 211 12

221 22

2 2 1 1 2 2 2 2 2 1 2 1 2 2

E x E x x E EE x x E x x

E x x E x x E x x E E E x

1 01210 12

2

2

2

11 22 2

2 2 2

// 1/ 2

// 1/ 2

EE x

p dxE x

x x x

2 2 22 2 2 2

2

2 2 2 2 2 2 2

2

1 1 1 2 1 21 1 1 1 1 1 2 211 12

221 22

2 2 1 1 2 2 2 2 2 1 2 1 2 2

E x E x x E EE x x E x x

E x x E x x E x x E E E x

1 01210 12

II. Using class-dependent approach,

1

2

1 1 1 2 1 21 1 1 1 1 1 2 211 12

221 22

2 2 1 1 2 2 2 2 2 1 2 1 2 2

43 9192 649 4364 192

E x E x x E EE x x E x x

E x x E x x E x x E E E x

1

1

1

11 11 1

2 1 2

// 1/ 4

// 1/ 4

EE x

p dxE x

x x x

We can calculate mean and covariance as,

Page 10: URL:.../publications/courses/ece_8443/lectures/current/exam/2004/ ECE 8443 – Pattern Recognition LECTURE 15: EXAM NO. 1 (CHAP. 2) Spring 2004 Solutions:

PROBLEM 2LECTURE 15: EXAM NO. 1 (CHAP. 2)

Figure showing the decision surface and the distribution of two classes

Page 11: URL:.../publications/courses/ece_8443/lectures/current/exam/2004/ ECE 8443 – Pattern Recognition LECTURE 15: EXAM NO. 1 (CHAP. 2) Spring 2004 Solutions:

PROBLEM 2b) Find the discriminant functions(e.g., gi (x) ).

There are infinite numbers of solutions for the case of P(w1)=P(w2) and P(x/w1)=P(x/w2) in the overlap area.

The simplest one can be defined asg(x)= x1+ x2-1/4 such that

if g(x)>0 decide class w2

else class w1

C) Write the Bayes decision rule for this case (hint: draw the decision boundary). Is this solution unique? Explain.

Since P(w1)=P(w2) and P(x/w1)=P(x/w2) in the overlap area, the posterior probability will be same in the overlap area.

Hence, the solution won’t be unique.

d) Compute the probability of error.

LECTURE 15: EXAM NO. 1 (CHAP. 2)

1 1 2 2

2 1

1 1 1 1 1( ) ( ) ( / ) ( ) ( / )

2 32 2 32 32R R

P error P p x dx P p x dx

Page 12: URL:.../publications/courses/ece_8443/lectures/current/exam/2004/ ECE 8443 – Pattern Recognition LECTURE 15: EXAM NO. 1 (CHAP. 2) Spring 2004 Solutions:

PROBLEM 2e) How will the decision surface change if the priors are not equal? Explain.

When the priors are equal, the decision region is at the point where two distribution meet if the distributions are similar. If the priors are not equal the decision region moves away from the higher prior class.

f) How will the probability of error change if the priors are not equal?

As the prior probability changes, the decision surface changes. Hence the probability of error changes. For ex, if P(w 1)=1 and P(w2)=0. In this case, the decision line will move such that the overlapped rectangle region belongs to region 1. Hence probability of error is zero.

g) Draw the minimax decision surface. Compare and contrast this to your answer in part (c).

The requirement for the minimax decision surface is

Since P(x/w1)=P(x/w2), we need to obtain R1= R2 The minimax decision surface also will have infinite solution compared to Bayes’ decision surface.

The contrast is the overlap region needs to be divided into equal area to get R1= R2

LECTURE 15: EXAM NO. 1 (CHAP. 2)

1 2

2 1( | ) ( | )R R

p x dx p x dx