P Values - part 3The P value as a ‘statistic’

Robin Beaumont1/03/2012

With much help from

Professor Geoff Cumming

P values - Putting it all together

P Valuesampling

probability

statisticRule Alternatives

Summary so far

• A P value is a conditional probability which considers a range of outcomes – shown as a ‘area’ in a graph.

• The SEM formula allows us to: predict the accuracy of your estimate ( i.e. the mean value of our sample) across a infinite number of samples!

Review

Summary so far• A statistic is just a summary measure, technically

we have reduced a set of data to one or two values:

• Range (smallest – largest)• Mean, median etc.• Inter-quartile range, SD Variance• Z score, T value, chi square value, F value etc• P value

What is a statistic?

T value• T statistic – different types, simplest 1 sample:

observed difference in estimated mean and population valuesampling variability in means

observed difference in estimated mean and population valueSEM

observed difference

statistic

statistic

T

T

in estimated mean and population valueexpected variability in means due to random samping

SignalNoise

So when t = 0 means 0/anything = estimated and hypothesised population mean are equal

So when t = 1 observed different same as SEMSo when t = 10 observed different much greater than

SEM

T statistic exampleSerum amylase values from a random sample of 15 apparently healthy

subjects. The mean = 96 SD= 35 units/100 ml. How likely would such a ‘unusual’ sample be obtained from a

population of serum amylase determinations with a mean of 120. (taken from Daniel 1991 p.202 adapted)

96 120 24

35 9.03715

2.656statisticT

This looks like a rare occurrence?

The population value = the null hypothesis

t density: sx = 9.037 n =15

0

12096

-2.656t 2.656

Shaded area=0.0188

Original units:

0

Serum amylase values from a random sample of 15 apparently healthy subjects. mean =96 SD= 35 units/100 ml. How likely would such a unusual sample be obtained from a population of serum amylase determinations with a mean of 120. (taken from Daniel 1991 p.202 adapted)

What does the shaded area mean!

Given that the sample was obtained from a population with a mean of 120 a sample with a T(n=15) statistic of -2.656 or 2.656 or one more extreme will occur 1.8% of the time = just under two samples per hundred on average. . . . .Given that the sample was obtained from a population with a mean of 120 a sample of 15 producing a mean of 96 (120-x where x=24) or 144 (120+x where x=24) or one more extreme will occur 1.8% of the time, that is just under two samples per hundred on average.

=P valueP value = 2 · P(t(n−1) < t| Ho is true) = 2 · [area to the left of t under a t distribution with df = n − 1]

P value and probability for the one sample t statistic

p value

= 2 x P(t(n-1) values more extreme than obtained t(n-1) | Ho is true)

= 2 X [area to the left of t under a t distribution with n − 1 shape]

Statistic -> sampling distribution -> PDF -> p value

No sampling distribution! Create a virtual one

P Value Variability

Taking another random sample the P value be different

How different? – Does not follow a normal distribution

Depends upon the probability of the null hypothesis being true! Remember we have assumed so far that the null hypothesis is true.

Dance of the p values – Geoff Cummings

Simplified dance of the p values when the null hypothesis is true

Example from Geoff Cummings dance of the p values

The take home message is that we can obtain very small p values even when the null hypothesis is true.

• P value -> statistic but• Not all statistics represent values that are reflected in a

population value• Other ways of getting an idea of variability across trials:• Reproducibility Probability Value (RP)

Why no CI for the P Value if it varies across trials

Goodman 1992 and also 2001 journal articlesHung, O’Neill, Bauer & Kohne 1997 Biometrics journalShao & Chow 2002 – Statistics in Medicine journalBoos & Stefanki 2011 – Journal of the American statistical associationCummings 2008 + and book

Cumming’s Reproducibility (replication) Probability ValueGiven Pobtained = 0.05What is the interval in which we are likely to see 80% of subsequent P values?

Answer:We have 80% of seeing subsequent p values fall within the zero to 0.22 boundary0, 0.22 [One sided]This means that we have a 20% of them being subsequently > 0.22

What about when the null hypothesis is not true?

statistic RuleAlternatives

P Value