scientific methods 1

10 Dec 2012 COMP80131-SEEDSM8 1

Scientific Methods 1

Barry & Goran

‘Scientific evaluation, experimental design

& statistical methods’

COMP80131

Lecture 8: Statistical Methods-Significance tests & confidence limits

www.cs.man.ac.uk/~barry/mydocs/MyCOMP80131


Introduction

• Statistical significance testing has so far been applied on the assumption of a

1) discrete population with binomial distribution

2) continuous population with known normal pdf & stdev.• Before proceeding further, take a quick look at a few more

prob distributions & pdfs.• Significance testing can be adapted to any of these.


Exponential pdf• Lifetimes, e.g. of light bulbs, follow an exponential distribution:

0:)/1(

0: 0 )( / xe

xxpdf x

0 1 2 3 4 5 6 7 8 9 100

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5

pdf

x

mean = 2;x = 0:0.1:10;y = exppdf(x,mean);plot(x,y);

Mean =

Stdev = also


Poisson Distribution

• λ, is both mean & variance of the distribution. • Poisson & exponential distributions are related. • If number of counts follows a Poisson distribution, then interval

between individual counts follows exponential distribution.• As λ gets larger, Poisson pdf normal with µ = λ, σ2 = λ.

integeran is x where!

)(x

exprob

xx

• For applications that involve counting number of times a random event occurs in a given amount of time, e.g. number of people walking into a store in an hour.


Poisson distributions in MATLABx=0:16y = poisspdf(x,5);stem(x,y);

0 10 20 30 40 50 600

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

0.09

prob

(x)

x0 2 4 6 8 10 12 14 16

0

0.02

0.04

0.06

0.08

0.1

0.12

0.14

0.16

0.18

prob

(x)

x

x=0:60y = poisspdf(x,20);stem(x,y);


Chi-squared distribution

0:

)2/(2

10: 0

)( 2/12/2/

2

xexV

xx xV

VV

•Given a population of normally distrib random variables with mean = 0 & stdev =1. •Randomly choose a sample of V observations of them.•Let x be the sum of their squares. •Then pdf of x has the 2 distribution:

(‘Gamma function’ (x) is generalisation of x! to non-integers).

If s = stdev of the V observations, pdf(s2) (1/V)V2(s2)

If pop mean = & stdev = , pdf (s2 ) (1/V)V2(s2/2+ 2)


Plot chi2 pdf with V = 4

0 5 10 150

0.02

0.04

0.06

0.08

0.1

0.12

0.14

0.16

0.18

0.2

pdf

x

x = 0:0.2:15; y = chi2pdf(x,4); plot(x,y)


Student’s t-distribution pdf

tVtVV

Vtpdf

V: )/1

)2/(

) 2/)1( ()(

2/)1(2

Depends on a single parameter V (degrees of freedom).

As V, t-pdf approaches standard normal distribution

If x is random sample of size n from a normal distribution with mean μ, then the t-statistic

stdev) samples&mean samplex(with /

ns

x

has Student's t-pdf with V = n – 1 degrees of freedom.


Compare t-pdf(V=5) with normal

x = -5:0.1:5;y = tpdf(x,5);z = normpdf(x,0,1);plot(x,y,'b',x,z,'r');

-5 -4 -3 -2 -1 0 1 2 3 4 50

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

T-p

df(b

lue)

N

orm

-pdf

(red

)


MATLAB functions for t-dist

• pdf for t-distribution with V degrees of freedom: y = tpdf ( t,V);

(With samples with n values, V = n-1)

• Cumulative df with V degrees of freedom p = tcdf ( t , V) Prob of rand var being t

• Complementary df (area under ‘tail’ from t to ) p = 1 – tcdf ( t , V) Prob of rand var being > t


Inverse-cdf in MATLAB

• If p = tcdf(t,V) then t = tinv(p,V)

Value of t such that prob of rand var being t is p• If p = normcdf(z,m,) then z = norminv(p,m, )

Value of z such that prob of rand var being z is p

• Complementary version:

t = tinv(1-p,V)

Value of t such that prob of rand var being > t is p.• Similarly for complementary version of norminv

t-pdf

x

t

p• Inverse of cumulative distrib function

t-pdf

x

t

p


Significance testing: z-test• Assume Normal population with known stdev = .• Null-hypothesis: pop-mean = 0

• Alternative hyp: pop-mean < 0• Take one sample of n values & calculate the z-statistic:

stdev) pop&mean samplex(with /

0

n

xz

If pop-mean = 0, dist of z will be standard Normal (mean=0, std=1)

-2 -1 0 1 2 40

0.1

0.2

0.3

0.4

Std

Nor

mal

pdf

z

If mean of z is 0, how likely is a value z as just calculated?

p-value = prob (x z)

= 1-normcdf(z,0,1)

If p-value < significance level alpha () reject null-hyp.


Alternative formulation


0

n

xz

Assuming we need 95% confidence, = 0.05Let z() = norminv(1-, 0, 1) = 1.65Prob of getting rand var 1.65 is less than 0.05If z 1.65, it is outside our 95% ‘confidence limit’ that the null-hyp may be true.So reject null-hyp.Confidence limit is for z is - to 1.65Neglect possibility that z may be negative.(1-tailed test)Confidence limit for sample-mean is - to 1.65/n + 0


2-tailed teststdev) pop&mean samplex(with

/0

n

xz

Assuming we need 95% confidence, = 0.05.Allowing possibility that z < 0, extreme portions of tails are for z > z(/2)) and for z < -z(/2)). prob(z z(/2)) + prob(z -z((/2) ) = 2 prob(z z(/2))Now, z(/2) = norminv(1-/2,0,1) = 1.96Prob of getting rand var 1.96 or -1.96 is 0.05If z > 1.96 or z < - 1.96, it is outside our 95% ‘confidence limit’ that the null hyp may be true. So reject null-hyp.Confidence limits for z are -1.96 to 1.96Confidence limits for sample-mean are: 0 - 1.96/n to 0 + 1.96/n


Significance testing: t-test• Assume Normal population with unknown stdev.• Null-hypothesis: pop-mean =0

• Alternative hyp: pop-mean < 0• Take one sample of n values & calculate the t-statistic:

stdev) sample&mean samplex(with /

0

sns

xt

If pop-mean = 0, dist of t will be standard t-pdf (blue) with V=n-1.

How likely is calculated value of t?

‘1-tailed’ p-value = prob (x t)

= 1 - tcdf(t , n-1)

If p-value < significance level alpha () reject null hyp.

t

-5 -4 -3 -2 -1 0 1 2 3 4 50

0.1

0.2

0.3

0.4

T-p

df(b

lue)

Nor

m-p

df(r

ed)


Alternative formulation (2-tailed)


0

sns

xt

• Assuming we need 95% confidence, = 0.05• Confidence limits for 0 is:

• Null-Hyp is that pop-mean is 0

If value of 0 is outside these limits, reject the null-hyp that population mean is 0

If 0 is within these confidence limits, cannot reject null-hyp.

nsntinvxnsntinvx /)1,2/1( to/)1,2/1(


Difference betw z-test & t-test(2-tailed)• With z-test pop-std () is known; with t-test is unknown.


0

n

xz


0

sns

xt

For z-test, p-value = prob ( x z) = 1- normcdf(z,0,1)For t-test, p-value = prob( x t) = 1 – tcdf(t,n-1)Same Null-hyp: pop-mean = 0 : reject if 0 outside conf limits

Confidence limits for z-test:

Confidence limits for t-test:

nxnx /)1,0 ,2/1(norminv to/)1,0, 2/1(norminv

nsntinvxnsntinvx /)1,2/1( to/)1,2/1(


Non-Gaussian populations

• If samples of size n are ‘randomly’ chosen from a pop with mean & std , the pdf of their sample-means approaches a Normal (Gaussian) pdf with mean & stdev /n as n ∞.

• Regardless of whether the population is Gaussian or not!• This is Central Limit Theorem• Tests can be made to work for non-Gaussian populations

provided n is ‘large enough’.


Meaning of confidence limits

If =0.5, there is 95% probability that the confidence limits for a given sample will contain the true population statistic say.


A really subtle point

• Does this mean that there a 95% probability that lies within the 95% confidence limits for the given sample?


A really subtle point

• Does this mean that there a 95% probability that lies within the 95% confidence limits for the given sample?

• No! A common mistake!• We have just one sample – we have no idea whether it is

one whose confidence limits contain or not.• Only 95% of possible samples will have conf limits which

contain .


P-values & confidence limits in MATLAB

• Come for free with most measurements. For example: x= [1;2;3;4;5;6]; y =[1.1; 3;2;4;6;4]; [R, p_value, Rlo, Rup] = corrcoef(x,y)• Returns Pearson corr coeff R= 0.79, • p_value = 0.061, • Also 95% confidence limits: Rlo=-0.06, Rup = 0.98• 95% prob that the true corr lies between -0.06 & 0.98

• “ Returns p-values for testing the hypothesis of no correlation. Each p-value is probability of getting a correlation as large as the observed value by random chance, when the true correlation is zero. If p_value is small, say < 0.05, then the correlation is significant”.


Credibility limits

• Baysian equivalent of ‘confidence limits’• If limits are C1 to C2, & = 0.05• Now there is 95% probability of the statistic, say, lying

between C1 & C2.• ‘Confidence limits are ‘frequentist’• Jonas explained why many people distrust the

frequentist approach and consider the Bayesian approach to be much more reliable.


Reminder: Binomial distribution• If p=prob(Heads), prob of getting Heads exactly r times in n

independent coin-tosses is:

nCr pr (1-p)(n-r)

• For a fair coin. p=0.5, this becomes nCr /2n

0 2 4 6 8 10 12 14 16 18 2000.020.04

0.10.12

0.16

0.2

No of heads obtainable with n coin-tosses

Tru

e pr

obab

ility

of g

ettin

g th

at n

o of

hea

ds


Binomist dist with n=6

0 1 2 3 4 5 60

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

No of heads obtainable with n coin-tosses

Tru

e pr

obab

ility

of g

ettin

g th

at n

o of

hea

ds

0.0156


MATLAB Script

p = 0.5; % for coin tossing

n=6;

for r=0:n

nCr = prod(n:-1:(n-r+1))/prod(1:r);

Prob(1+r) = nCr * (p^r) * (1-p)^(n-r);

end;

Prob

figure(2); stem(0:n,Prob);


Geometric distribution• p(x) = (1-p)px-1 (p = prob of success).• Number of trials (coin tosses) up to & including that in

which first failure occurs

1 2 3 4 5 6 7 8 9 100

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5

x: number of trials

prob

of

first

fai

lure

at

x

p = 0.5x=1:10; prob = (1-p)*p.^(x-1);stem(x,prob);


Geometric distribution (again)

1 2 3 4 5 6 7 8 9 100

0.05

0.1

0.2

0.3

0.4

0.5

x: number of trials

prob

of

first

fai

lure

at

x

prob(6) = 0.0156

prob(5) = 0.0313


Barry’s Assignment

• Deadline 20 Dec 2012

• Email to [email protected] with ‘SEEDSM12’ in title

• or

• Hand in paper copy to SSO

• Exam statistics are in examdata.dat and examdata.xls in

• www.cs.man.ac.uk/~barry/mydocs/MyCOMP80131

• (or navigate from www.cs.man.ac.uk/~barry)

mailto:[email protected]

http://www.cs.man.ac.uk/~barry/mydocs/MyCOMP80131

scientific methods 1

Documents