Statistical Inference Using Scrambles and Bootstraps

Robin LockBurry Professor of Statistics

St. Lawrence University

MAA Allegheny Mountain 2014 Section Spring Meeting

Westminster College

The Lock5 Team

DennisIowa State

KariHarvard/Duke

EricUNC/Duke/UMinn

Robin & PattiSt. Lawrence

What is Statistical Inference?

Hypothesis Test Is an effect observed in a sample true for a population or just due to random chance?

Confidence Interval Based on the data in a sample, find a range of plausible values for a quantity in a population.

Example #1: Beer & Mosquitoes• Volunteers were randomly assigned to drink either a

liter of beer or a liter of water.• Mosquitoes were caught in nets as they approached

each volunteer and counted . n mean

Beer 25 23.60

Water

18 19.22

Does this provide convincing evidence that mosquitoes tend to be more attracted to beer drinkers or could this difference be just due to random chance? Hypothesis Test

Example #2: Mustang Prices• A student selected a random sample of n=25

Mustang (cars) from an internet site and recorded the prices in $1,000’s.

n mean std. dev.

Price 25 15.98 11.11

Find a range of plausible values where the mean price for all Mustangs at this website is likely to be. Confidence Interval

Price (in $1,000’s)

Two Approaches to InferenceTraditional: • Assume some distribution (e.g. normal or t) to

describe the behavior of sample statistics• Estimate parameters for that distribution from

sample statistics• Calculate the desired quantities from the

theoretical distribution

Simulation: • Generate many samples (by computer) to show

the behavior of sample statistics • Calculate the desired quantities from the

simulation distribution

“New” Simulation Methods?

"Actually, the statistician does not carry out this very simple and very tedious process, but his conclusions have no justification beyond the fact that they agree with those which could have been arrived at by thiselementary method."

-- Sir R. A. Fisher, 1936

Example #1: Beer & Mosquitoesµ = mean number of attracted mosquitoes

H0: μB = μW

Ha: μB > μW

Competing claims about the population means

Based on the sample data:

P-value: The proportion of samples, when H0 is true, that would give results as (or more) extreme as the original sample.

Is this a “significant” difference?

Traditional Inference2. Which formula?

3. Calculate numbers and plug into formula

4. Chug with calculator

5. Which theoretical distribution?

6. df?

7. Find p-value

0.0005 < p-value < 0.001

1. Check conditions

𝑡=𝑥𝐵−𝑥𝑊

√ 𝑠𝐵2𝑛𝐵

+𝑠𝑊

2

𝑛𝑊

𝑡=23.6−19.22

√ 4.12

25+ 3.7❑

2

18

𝑡=3.68

8. Interpret a decision

Simulation Approach

0

Water 21 22 15 12 21 16 19 15 24 19 23 13 22 20 24 18 20 22

Beer 27 20 21 26 27 31 24 19 23 24 28 19 24 29 20 17 31 20 25 28 21 27 21 18 20

Number of Mosquitoes

𝑥𝑊=19.22

𝑥𝐵−𝑥𝑊=4.38

To simulate samples under H0 (no difference):• Re-randomize the values into

Beer & Water groups • Compute

Original Sample

Simulation Approach

0

Water 21 22 15 12 21 16 19 15 24 19 23 13 22 20 24 18 20 22

Beer 27 20 21 26 27 31 24 19 23 24 28 19 24 29 20 17 31 20 25 28 21 27 21 18 20

Number of Mosquitoes

𝑥𝑊=19.22

𝑥𝐵−𝑥𝑊=4.38

To simulate samples under H0 (no difference):27 19 21 24

20 24 18 1921 29 20 2326 20 21 1327 27 22 2231 31 15 2024 20 12 2419 25 21 1823 28 16 2024 21 19 2228 27 15

Simulation Approach

𝑥𝐵=21.76

Number of Mosquitoes

𝑥𝑊=22.50

𝑥𝐵−𝑥𝑊=−0.84

To simulate samples under H0 (no difference):• Re-randomize the values into

Beer & Water groups • Compute

24 20 24 18 1921 29 20 2326 20 21 1327 27 22 2231 31 15 2024 20 12 2419 25 21 1823 28 16 2024 21 19 2228 27 15

Beer Water

2024192024311318242521181521162822192720232221

27 19 2120263119231522122429202721172428

Repeat this process 1000’s of times to see how “unusual” is the original difference of 4.38.

We need technology!

www.lock5stat.com/statkey

StatKey

Freely available web apps with no login requiredRuns in (almost) any browser (incl. smartphones/tablets) Google Chrome App available (no internet needed)Standalone or supplement to existing technology

p-value = proportion of samples, when H0 is true, that are as (or more) extreme as the original sample.

p-value

Price0 5 10 15 20 25 30 35 40 45

MustangPrice Dot Plot

𝑛=25 𝑥=15.98 𝑠=11.11

Key concept: How much can we expect the sample means to vary just by random chance?

Example #2: Mustang PricesStart with a random sample of 25 prices (in $1,000’s)

Goal: Find an interval that is likely to contain the mean price for all Mustangs

Traditional Inference2. Which formula?

3. Calculate summary stats

6. Plug and chug

𝑥± 𝑡∗ ∙𝑠

√𝑛𝑥± 𝑧∗ ∙𝜎√𝑛

,

4. Find t*

95% CI

5. df?

df=251=24

OR

t*=2.064

15.98±2 .064 ∙11.11

√25

15.98±4.59=(11.39 ,20.57)7. Interpret in context

CI for a mean1. Check conditions

Bootstrapping

To create a bootstrap distribution: • Assume the “population” is many, many copies

of the original sample. • Simulate many samples from the population by

sampling with replacement from the original sample

“Let your data be your guide.”

Brad Efron Stanford University

Original Sample (n=6)

Bootstrap Sample(sample with replacement from the original sample)

Finding a Bootstrap Sample

A simulated “population” to sample from

Original Sample Bootstrap Sample

𝑥=15.98 𝑥=17.51

Repeat 1,000’s of times!

Original Sample

BootstrapSample

BootstrapSample

BootstrapSample

●●●

Bootstrap Statistic

Sample Statistic

Bootstrap Statistic

Bootstrap Statistic

●●●

Bootstrap Distribution

StatKey

StatKey

Standard Error𝑠

√𝑛=

11.114

√25=2.2

)

A 95% Confidence Level

Keep 95% in middle

Chop 2.5% in each tail

Chop 2.5% in each tail

We are 95% sure that the mean price for Mustangs is between $11,800 and $20,190

The same method is used for any statistic, including new statistics that are being defined in areas like genetics.

This is very powerful for practioners!(and appreciated by students – especially visual learners)

Why does the bootstrap

work?

Sampling Distribution

Population

µ

BUT, in practice we don’t see the “tree” or all of the “seeds” – we only have ONE seed

Bootstrap Distribution

Bootstrap“Population”

What can we do with just one seed?

Grow a NEW tree!

𝑥

Estimate the distribution and variability (SE) of ’s from the bootstraps

µUse the bootstrap errors that we CAN see to estimate the sampling errors that we CAN’T see.

Golden Rule of Bootstraps

The bootstrap statistics are to the original statistic

as the original statistic is to the population parameter.

Example #3: Malevolent Uniforms

Sample Correlation r = 0.43

Do football teams with more malevolent uniforms tend to get more penalty yards?

H0: ρ = 0Ha: ρ > 0

Simulation Approach

Find out how extreme this correlation would be, if there is no relationship between uniform malevolence and penalties.

i.e., What kinds of results (correlations) would we see, just by random chance?

Sample Correlation = 0.43

Randomization by ScramblingOriginal sample

MalevolentUniformsNFL

NFLTeam NFL_Ma... ZPenYds <new>

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

LA Raiders 5.1 1.19

Pittsburgh 5 0.48

Cincinnati 4.97 0.27

New Orl... 4.83 0.1

Chicago 4.68 0.29

Kansas ... 4.58 -0.19

Washing... 4.4 -0.07

St. Louis 4.27 -0.01

NY Jets 4.12 0.01

LA Rams 4.1 -0.09

Cleveland 4.05 0.44

San Diego 4.05 0.27

Green Bay 4 -0.73

Philadel... 3.97 -0.49

Minnesota 3.9 -0.81

Atlanta 3.87 0.3

Indianap... 3.83 -0.19

San Fra... 3.83 0.09

Seattle 3.82 0.02

Denver 3.8 0.24

Tampa B... 3.77 -0.41

New Eng... 3.6 -0.18

Buffalo 3.53 0.63

Scrambled MalevolentUniformsNFL

NFLTeam NFL_Ma... ZPenYds <new>

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

LA Raiders 5.1 0.44

Pittsburgh 5 -0.81

Cincinnati 4.97 0.38

New Orl... 4.83 0.1

Chicago 4.68 0.63

Kansas ... 4.58 0.3

Washing... 4.4 -0.41

St. Louis 4.27 -1.6

NY Jets 4.12 -0.07

LA Rams 4.1 -0.18

Cleveland 4.05 0.01

San Diego 4.05 1.19

Green Bay 4 -0.19

Philadel... 3.97 0.27

Minnesota 3.9 -0.01

Atlanta 3.87 0.02

Indianap... 3.83 0.23

San Fra... 3.83 0.04

Seattle 3.82 -0.09

Denver 3.8 -0.49

Tampa B... 3.77 -0.19

New Eng... 3.6 -0.73

Buffalo 3.53 0.09

Scrambled sample

StatKey

Repeat 1000’s of times

P-value

Small p-value Strong evidence of a positive association between uniform malevolence and penalty yards.

How does everything fit together?• We use simulation methods to build understanding of the key statistical ideas.

• We then cover traditional normal and t-based procedures as “short-cut formulas”.

• Students continue to see all the standard methods but with a deeper understanding of the meaning.

Intro Stat – Revise the Topics • Descriptive Statistics – one and two samples• Normal distributions• Data production (samples/experiments)

• Sampling distributions (mean/proportion)

• Confidence intervals (means/proportions)

• Hypothesis tests (means/proportions)

• ANOVA for several means, Inference for regression, Chi-square tests

• Data production (samples/experiments)• Bootstrap confidence intervals• Randomization-based hypothesis tests• Normal distributions

• Bootstrap confidence intervals• Randomization-based hypothesis tests

• Descriptive Statistics – one and two samples

Transitioning to Traditional Inference

Confidence Interval:

Hypothesis Test:

The Next Big Thing...“... the consensus curriculum is still an unwitting prisoner of history. What we teach is largely the technical machinery of numerical approximations based on the normal distribution and its many subsidiary cogs. This machinery was once necessary, because the conceptually simpler alternative based on permutations was computationally beyond our reach. Before computers statisticians had no choice. These days we have no excuse. Randomization-based inference makes a direct connection between data production and the logic of inference that deserves to be at the core of every introductory course.”

-- Professor George Cobb, 2007

Thanks for listening!

rlock@stlawu.edu

www.lock5stat.com