STATISTICAL THEORY How do we know what is real?

HOW DO WE KNOW WHAT IS REAL?

CARGO CULT SCIENCE

Feynman spoke of what he called cargo cult science

Cargo cults were observed by anthropologists - formed by South Sea Islanders living in pre-scientific cultures that came into contact with manufactured goods through small aircraft pilots landing their planes

The Islanders wanted the planes (and their cargo) to return, so they built “airplanes” and “runways”

The planes didn’t return as the Islanders hoped

TOOTH FAIRY SCIENCE

Coined by Harriet Hall, MD

“You could measure how much money the Tooth Fairy leaves under the pillow, whether she leaves more cash for the first or last tooth, whether the payoff is greater if you leave the tooth in a plastic baggie versus wrapped in Kleenex. You can get all kinds of good data that is reproducible and statistically significant. Yes, you have learned something. But you haven’t learned what you think you’ve learned, because you haven’t bothered to establish whether the Tooth Fairy really exists.”

IS THIS A RANDOM SEQUENCE?

OXXXOXXXOXXOOOXOOXXOO

ANSWER

The pattern was generated randomly!

“What this ingenious experiment shows is just how bad we are at correctly identifying random sequences. We are wrong about what they should look like: we expect too much alternation, and to us, even truly random sequences seem somehow too lumpy and ordered.” – Ben Goldacre

HUMANS ARE EXPERT PATTERN-FINDERS

Did our propensity for Type 1 errors help us survive as a species?

BUT WE ALSO HAVE A LIMITED VIEWPOINT

“You cannot sense whether a pill improves intelligence, or cures the common cold, or whether MMR causes autism. Your tiny, beautiful ingot of human experience in the world does not present you with sufficient information to spot patterns on that scale: it’s like looking at the ceiling of the Sistine chapel with one eye through a very long cardboard tube.”

-Ben Goldacre

WHAT CAN STATISTICS DO FOR US? When used correctly…

Separate the signal from the noise

Reveal the pattern in the chaos

Find the enduring among the transient

THE UNIVERSE IS A BIG PLACE

One of the fundamental issues of scientific inquiry is we can’t just measure things directly

There are usually far too many instances of what we would like to observe to examine each one

We can’t gather all people with a particular disease at our hospital to undergo an intervention or answer our questions

So we have to resort to taking samples of the total population

THAT DRESSWhat if we want to know how many people in the Cincinnati metro area see that dress as blue and black and how many see it as white and gold?

We can’t show that dress to and poll 2.2 million people (although some people attempted this through social media)

If we take a sample, however, it might or might not be representative of the true proportions of people who see the 2 different color combos

If we take repeated samples and average, we will get closer, but taking repeated samples is not usually practical due to limited time and resources

We have to factor in ERROR

We have to rely on PROBABILITIES

EVERY STATISTICAL MODEL EVER

predicted outcome = model + error

IS OUR MODEL ANY GOOD?

Question: How well does it explain the outcome data based on our hypothesis?

The hypothesis must be decided in advance of data collection

The statistical model must reflect the alternate hypothesis

H0 = There is no relationship between age and seeing that dress as blue and black

HA = There is a relationship between age and seeing that dress as blue and black

ASSESSING GOODNESS What is the probability that the alternate hypothesis is WRONG?

What is the probability the data can be explained equally well by just CHANCE?

Recall:

predicted outcome = model + error

Need a test statistic

test statistic = =

TEST STATISTIC Recall that the whole point of this exercise is to determine what is (most likely) happening in the real world

The test statistic is based on known distributions

If the probability of obtaining the test statistic is low (conventionally < .05), then we say that we fail to reject the null hypothesis

We have some confidence that the alternate hypothesis reflects something happening in the real world and our model is a good one

FACING REALITY

Type 1 error:

We think there is an effect in the real world population based on our test statistic but there is none

alpha = .05

Acceptable level of Type 1 error: 5 out of 100 collections of sample data will result in a test statistic that leads us to conclude there is a real effect when there is none

Type 2 error:

We think there is no effect in the real world population based on our model but there is one

beta = .20

Acceptable level of Type 2 error: 20 out of 100 collection of sample data will lead us to conclude there is no real effect when there is one