technology for teaching bootstraps and randomizationskfl5/lock_uscots_2011.pdf · technology for...

Technology for Teaching Bootstraps and Randomizations

Data for these problems in various formats can be found at www.lock5stat.com under “Datasets”.

Problem #1: Mustang Prices. A student collected data on the selling prices for a sample of used Mustang cars being offered for sale at an internet website. The price (in $1,000’s), age (in years) and miles driven (in 1,000’s) for the 25 cars in the sample are given in the table below. Use these data to construct a 90% confidence interval for the mean price (in $1,000’s) for used Mustangs at this website.

Price 32 45 11.9 24.8 22 10 5 9 23 37.9 32.5 3 9 Age 6 7 9 2 3 15 10 9 1 1 4 14 8 Miles 8.5 33 82.8 7 23 111 136.2 78.2 26.1 1.1 18.2 144.9 100.8 Price 13 14.9 7 16 21 7 8.2 9.7 8 11.8 12.9 4.9 Age 10 5 9 6 1 12 14 10 13 13 12 14 Miles 51.4 38.5 61.9 71.2 26.4 117.4 102 86.4 72.7 71.8 72.9 115.1

Problem #2: NFL overtimes. The National Football League (NFL) uses an overtime period to determine a winner for games that are tied at the end of regulation time. The first team to score in the overtime wins the game and a coin flip is used to determine which team gets the ball first. Is there an advantage to winning the coin flip? Data from the 1974 through 2009 seasons show that the coin flip winner won 240 of the 428 games where a winner was determined in overtime. Treat these as a sample of NFL games to test whether there is sufficient evidence to show that the proportion of overtime games won by the coin flip winner is more than one half.

Problem #3: Sleep vs. Caffeine. In an experiment on memory1, students were given lists of 24 words to memorize. After hearing the words they were assigned at random to different groups. One group of 12 students took a nap for 1.5 hours while a second group of 12 students stayed awake and was given a caffeine pill. The table below shows the number of words each participant was able to recall after the break. Test whether the data indicate a difference in mean number of words recalled between the two treatments.

Sleep 14 18 11 13 18 17 21 9 16 17 14 15 mean=15.25 Caffeine 12 12 14 13 6 18 14 16 10 7 15 10 mean=12.25

(Optional) Bonus problem: The data on Mustang prices in Problem #1 also contains the number of miles each car had been driven (in thousands). Find a 95% confidence interval for the correlation between price and mileage.

1Mednicj, Cai, Kanady and Drumond, “Comparing the benefit of caffeine, naps and placebo on verbal, motor and perceptual memory”, Behavioural Brain Research, 193 (2008), 79-86.

Minitab Mintab Instructions

Bootstrap Confidence Interval for Mean Mustang Price Data: A sample of n=25 Mustang cars offered for sale on a website (thanks to Gabe McBride). We are interested in estimating the mean Price (in $1,000) and finding confidence interval for that mean.

1. Find the mean for the original sample.

• Open the MustangPrice.mtw worksheet. Use Stat > Basic Statistics to find statistics (including the mean) for the sample of prices. Mean price for original sample: _________________

2. Create a column of bootstrap means. • Run* the oneboot macro with syntax such as MTB > %oneboot c3 c4

where C3 = column with the original sample C4 = column to hold the bootstrap means

• Respond to the prompts to select the mean as a statistic to store and 1000 bootstrap samples.

3. Analyze the bootstrap distribution

• Create a graph (dotplot or histogram) of the column of bootstrap means. • Use Stat> Basic Statistics to find the mean and standard deviation of the bootstrap means. The

standard deviation is an estimate of the standard error (SE) for the sample mean. Standard deviation of bootstrap means = ___________________ (estimates SE of �̅)

• Compute a rough interval estimate for the mean Mustang price using the mean price for the original sample �̅ ± 2�� =

• To find percentiles of the bootstrap distribution use Calc>Calculator and formulas like • Percentile(C4,0.025)= Percentile(C4,0.975)=

* To run a macro in Minitab, from the session window

• Select Editor > Enable Commands to show the Minitab command prompt. • Use %macroname to run a macro stored as macroname.mac . • The macro file should be stored in Minitab’s default macro folder or the current data folder.

Minitab macro ################################################### #### # MACRO: OneBoot.MAC # Writen by: Robin Lock February 2011 # Purpose: Construct a distribution of bootstrap s tatistics for one sample # with a quantitative variable ################################################### ####### oneboot x bootstat mcolumn x tempx bootstat cint mconstant nboot stat n i mreset brief 1 noecho notitle let n=count(x) #sample size for original sample # Code below is to let user enter stat note note Which statistic? note 1=mean 2=median 3=std dev set cint; file "Terminal"; nobs 1. copy cint stat # Code below is to let user number of bootstraps note note How many bootstrap samples? set cint; file "Terminal"; nobs 1. copy cint nboot #clear out the column to contain bootstrap samples erase bootstat do i=1:nboot sample n x tempx; replace. if stat=1 let bootstat(i)=mean(tempx) elseif stat=2 let bootstat(i)=median(tempx) elseif stat=3 let bootstat(i)=stdev(tempx) else note note ***Oops*** You need to specify 1, 2 , or 3 for statistic exit endif enddo endmacro

Minitab Minitab Instructions

Advantage for Coin Flip Winner in NFL OT?

Parameter: p = proportion of NFL overtime games won by the winner of the coin flip.

Hypotheses: Ho: p=1/2 vs. Ha: p>1/2

Data: Coin flip winner won 240 of 428 past games.

Key question: If the null hypothesis is true (p=1/2), how likely is it to see 240 (or more) wins for the coin flip winner in a sample of 428 games.

Approach: Use Minitab to simulate lots of samples of size 428 (when p=1/2). Count the number of “wins” in each sample. See what proportion of the samples give 240 (or more) wins.

*************************************************** ************************** Run the onePTest.mac macro

MTB> %onePTest C7

where C7 is a column to hold the randomization counts

Follow the prompts to specify an original sample size (428), hypothesized proportion (0.5), number of randomization samples (1000) and to save the count for each sample (2).

Analyze the counts generated in the new columns to show the distribution of the randomization counts (use a dotplot or histogram) and see what proportion are greater than or equal to the count of 240 in the original sample. Hint: Use Data > Sort to put the randomization counts in order to make counting easier.

-----------------------------------------------------------------------------------------------------------------------------------*************************************************** ***********************************

Sleep vs. Caffeine for Memory

Response: Number of words recalled (out of 24)

Conditions: Caffeine pill vs. Sleep nap (12 assigned at random to each group)

Test: H0:µC=µS vs. Ha: µC≠µS

*************************************************** ************************** Run the TwoMeanRand.mac macro

MTB> %TwoMeanRand C2 C1 C5

where C2 is the column with a quantitative response variable

C1 is the column with a binary grouping variable

C5 is a column to contain the randomization differences

Follow the prompts to specify a number of randomization samples (1000).

Analyze the randomization differences in means generated in the new column to show the distribution of the randomization statistics (use a dotplot or histogram) and see what proportion are greater than or equal to the difference of 3.0 in the original sample. Hint: Use Data > Sort to put the randomization stats in order to make counting easier.

Minitab

Minitab Instructions

CI for Correlation of Price and Miles for Used Mustangs

Parameter: ρ = correlation between price and mileage for Mustangs offered at an internet site.

Data: Price and Miles (both in thousands) for a sample of 25 Mustangs.

Task: Find a 95% confidence interval for the correlation between price and miles for Mustangs.

1. Find the correlation for the original sample

• Open the worksheet MustangPrice.mtw.

• Choose Stat > Basic Statistics > Correlation to find the correlation between Price and Miles.

Original correlation: r = ____________________________

2. Create a column of bootstrap correlations

• Run the twoboot macro with syntax such as MTB > %twoboot c3 c2 C5

where C3 = column with a quantitative response variable C2 = column with a quantitative predictor variable C5 = column tp hole bootstrap correlatoins

• Respond to the prompts to select correlation as a statistic and store and 1000 bootstrap samples. See the earlier handout on CI for mean Mustang price for instructions on creating a confidence interval from the column of bootstrap statistics.

R

Bootstrapping Confidence Intervals and Randomization Tests in R

OPTION 1: Resample, Reallocate Functions

To make bootstrapping and randomization tests incredibly easy to do, we’ve written functions to

automate the process. These are available in LockJMM_functions.R, and are described below. See the

file for code.

resample = function(y1, y2 = NULL, num = 10000, lev el=95, statistic=NULL, distribution = FALSE) The resample function creates a bootstrap distribution by sampling with replacement, and produces

the corresponding confidence interval.

y1 : variable

y2 : second variable (if applicable)

num: number of bootstrap samples to take, 10,000 by default

level : confidence level, 95% by default

statistic : any statistic can be entered in the form of a function (e.g. median, sd,…) or any

predefined function on y, y2. If unspecified, the default statistics are

quantitative y: mean

binary y: proportion

quantitative y, binary y2: difference in means

binary y, binary y2: difference in proportions

quantitative y, quantitative y2: correlation

distribution : set to TRUE to return entire bootstrap distribution

reallocate = function(y1, y2, num=10000, statistic = NULL, distribution = FALSE) The reallocate function creates a randomization distribution by simulating new randomizations (each

time randomly permuting the group vector), and performs the corresponding randomization test.

y1 : response variable

y2 : explanatory variable

num: number of bootstrap samples to take, 10,000 by default

statistic : any statistic can be entered in the form of a function of the form function(y,

group). If unspecified, the default statistics are

quantitative y, binary group: difference in means

binary y, binary group: difference in proportions

quantitative y, quantitative group: correlation

distribution : set to TRUE to return entire randomization distribution coin.flips = function(n, p, num = 10000, phat = NUL L, distribution = FALSE) Simulates coin flips, recording the number of heads in num iterations of n coin flips, where each flip

has probability p of landing heads. If phat is given a p-value is returned, if distribution is true

the entire distribution is returned.

Before doing the problems below, source this file to make the functions available: > source(“LockJMM_functions.R”)

R

The code listed below is available in “LockUSCOTS.R”.

Problem 1: Create a 90% confidence interval for the mean Mustang price.

With the data loaded as “mustang”, use the resample function to create a bootstrap confidence

interval for the mean Mustang price, setting level=90 for a 90% confidence interval.

> resample(mustang$Price, level = 90)

This gives the following output:

One quantitative variable Single Mean Observed Sample Mean: 15.98 SE = 2.166 90% Confidence Interval: 5% 95% 12.548 19.664

You can also create an interval using the standard error, either by looking at the output above, or by

calculating the standard deviation of the bootstrap distribution directly by returning the entire

distribution:

> boot.dist = resample(mustang$Price, level = 90, d istribution = TRUE) > se = sd(boot.dist) > c(mean(mustang$Price) – 1.96*se, mean(mustang$Pri ce) + 1.96*se) [1] 11.72704 20.23296 This option (distribution = TRUE) can also be used with the resample function to perform tests based

on resampling.

Bootstrap Distribution

Sample Mean

Fre

quen

cy

10 15 20 25

050

010

0015

00

12.548 19.664

R Problem 2: Is the proportion of overtime games won by the coin flip winner more than ½? We use the coin.flips function to simulate coin flips. We set n=428 , the number of flips in our

sample, and p=.5 , since we want to simulate coin flips under the null hypothesis of p = ½. From our

sample data, p̂ = 240/428 = 0.561. We test the significance of this with:

> coin.flips(n = 428, p = .5, phat = 240/428) This yields the following output:

[1] "Observed Sample Proportion: 0.560747663551402" [1] "Smaller Tail Probability:" [1] 0.0066

This is a one-tailed test, so the p-value is the area in the upper tail, 0.0066. There is evidence that the

proportion of overtime games won by the coin flip winner is more than ½.

Problem 3: Is there a significant difference in the mean number of words recalled between the group

randomized to sleep and the group randomized to caffeine?

With the data loaded as “sleepcaffeine”, use the reallocate function to create a randomization

distribution for the difference in means, each time reallocating units to either sleep or caffeine. > reallocate(sleepcaffeine$Words, sleepcaffeine$Gro up)


Variable 1: quantitative , Variable 2: categorical Test for a Difference in Means Observed Sample Difference in Means: -3 SE = 1.517 Smaller Tail Probability: 0.027

R

Randomization Distribution

Sample Proportion

Fre

quen

cy

0.40 0.45 0.50 0.55

050

010

0015

00

0.007

This is a two-tailed test, so we double the probability in the lower tail for a p-value of 0.054.

Bonus Problem: Find a 95% confidence interval for the price and mileage of Mustangs.

With the data loaded as “mustang”, use the resample function to create a bootstrap confidence

interval for the correlation between Mustang price and mileage: > resample(mustang$Price, mustang$Miles)


Variable 1: quantitative , Variable 2: quantitative Correlation Observed Sample Correlation: -0.825 SE = 0.056 95% Confidence Interval: 2.5% 97.5% -0.929 -0.724

A 95% confidence interval for the correlation between price and mileage is (-0.929, -0.724).

Randomization Distribution

Sample Difference in Means

Fre

quen

cy

-4 -2 0 2 40

200

600

1000

0.027

Bootstrap Distribution

Sample Correlation

Fre

quen

cy

-0.9 -0.8 -0.7 -0.6

020

060

010

00

-0.929 -0.724

R

Bootstrapping Confidence Intervals and Randomization Tests in R

OPTION 2: Students Code Everything

You can also have the students code everything themselves, giving them guidance the first few times.

This is great pedagogically because it forces the students to understand each step of the process, but

requires the students to be sufficient in using R.

Code for each problem is given below, and is written to be most pedagogically understandable, not the

most computationally efficient.

#Problem 1 mustang = read.table("MustangPrice.csv", sep=",", h eader=TRUE) num = 10000 #number of simulations y = mustang$Price boot.y = matrix(NA, length(y), num) #each column will be a bootstrap sample

boot.dist = rep(NA, num) #bootstrap distribution for (i in 1:num) { boot.y[,i] = sample(y, length(y), replace=TRUE) boot.dist[i] = mean(boot.y[,i]) } hist(boot.dist, main="Bootstrap Distribution", xlab ="Sample Mean") quantile(boot.dist, c(.05, .95)) #90% confidence interval #Problem 2 num = 10000 n = 428 sim.dist = rbinom(num, size = n, p = .5)/n #simulated distribution under the null hist(sim.dist) abline(v = 240/428, col="red") mean(sim.dist >= 240/428) #p-value #Problem 3 sleepcaffeine = read.table("SleepCaffeine.csv", sep =",", header=TRUE) y = sleepcaffeine$Words #outcome g = sleepcaffeine$Group #group diff.means = function(y, g) mean(y[g==levels(as.fac tor(g))[1]]) - mean(y[g == levels(as.factor(g))[2]]) num = 10000 sim.g = matrix(NA, length(g), num) #each column will be a new random assignment

rand.dist = rep(NA, num) #randomization distribution for (i in 1:num) { sim.g[,i] = sample(g, length(g), replace=FALSE) rand.dist[i] = diff.means(y, sim.g[,i]) } hist(rand.dist, main="Randomization Distribution", xlab="Sample Difference in Means") observed = diff.means(y, g) abline(v = observed, col="red") 2*mean(rand.dist >= abs(observed)) #p-value

R

#Bonus Problem y = cbind(mustang$Price, mustang$Miles) n = dim(y)[1] num = 10000 boot.indices = matrix(NA, n, num) #each column gives indices of units sampled boot.dist = rep(NA, num) for (i in 1:num) { boot.indices[,i] = sample(1:n, n, replace=TRUE) boot.dist[i] = cor(y[boot.indices[,i],1], y[boot. indices[,i], 2]) } hist(boot.dist, main="Bootstrap Distribution", xlab ="Sample Correlation") quantile(boot.dist, c(.025, .975)) #95% confidence interval

Excel

Excel Instructions

Microsoft Excel is a standard spreadsheet software package that should be familiar to most students.

However, there are no built-in methods to do statistical inference through randomization in Excel. The examples

below use PopTools, a free and universally available add-in for Excel that incorporates bootstrapping and

randomization tests. PopTools can be downloaded and installed at www.poptools.org.

Bootstrap Confidence Interval for Mean Mustang Price

Data: A sample of n=25 Mustang cars offered for sale on a website (thanks to Gabe McBride). We are

interested in estimating the mean Price (in $1,000) and finding a confidence interval for that mean.

1.) Open ‘MustangPrice.xls’

2.) Enter the mean price in an unoccupied cell

=AVERAGE(C2:C26)

3.) Select two cells, and use the PopTools ‘BootMean’ function to create a 90% confidence interval for

the mean price based on 5000 bootstrap means.

=BOOTMEAN(C2:C26,5000,0.1)

Instead of Enter, press Ctrl+Shift+Enter. The lower confidence bound should be on the top cell, the

upper bound on the bottom cell.





1.) What is the test statistic, �̂?

=240/428

2.) Select an unoccupied column, and simulate 1000 samples of size 428 under Ho: p=1/2.

=BINOMIALDEVA(428,0.5,1000)/428

(Press Ctrl+Shift+Enter).

3.) Find the p-value (proportion of simulations with result > �̂).

=COUNTIF(C1:C1000,">0.5607")/1000

4.) What can you conclude?

Excel





1.) Open ‘SleepCaffeine.xls’

2.) Find the mean difference.

= AVERAGE(B2:B13)-AVERAGE(B14:B25) (in cell C1)

3.) Use PopTools -> Resample to generate a column of shuffled data

4.) Find the mean difference for the shuffled data

=AVERAGE(D2:D13)-AVERAGE(D14:D25) ( in cell E1)

5.) Use PopTools -> Monte Carlo analysis to repeat 1000 times

6.) Use the output from step (5) to find the p-value (the number of differences after reshuffling that are

greater than the true difference should be in cell J2).

=(J2/1000)*2

Fathom Fathom Instructions

Bootstrap Confidence Intervals for Mean Mustang Price Data: A sample of n=25 Mustang cars offered for sale on a website (thanks to Gabe McBride). We are interested in estimating the mean Price (in $1,000) and finding confidence interval for that mean.


• Open the MustangPrice.ftm file. Double click on the MustangPrice collection box (with gold balls) to bring up its inspector.

• Select the Measures tab and define a new measure and name it xbar. Double-click in the Formula area and enter the formula mean(Price). Record the mean price for the original sample below. Mean price for original sample: _________________

2. Create a bootstrap sample. • Click to select the MustangPrice collection box, choose Collection > Sample Cases. • Double-click on the new “Sample of MustangPrice” collection box and

choose the Sample tab in its inspector. • Adjust the sample size to match the original sample (25), turn off

animation and be sure samples are with replacement. Click on Sample More Cases to generate the new sample. Drag down a Table if you want to see the sample results.

• Choose the Measures tab in the inspector and record the mean for this sample below.

• Use “Sample More Cases” (or select the collection box and hit Ctrl-Y) to generate another sample. Repeat and record the sample means below.

Bootstrap means

3. Collect measures for lots of bootstrap samples

• Click to select the sample collection is and choose Collection > Collect Measures. This should create a new collection with (by default) the means for five samples. Drag down a Table to see the results (be sure the “Measures from …” collection box is selected).

• Double-click on the new collection to bring up its inspector. Turn animation OFF, replace existing cases and change the number of measures to 1000. Collect More Measures.

• Create a graph of the bootstrap distribution. (drag down an empty graph and drag the xbar variable from the data table to the horizontal axis).

4. Compute a bootstrap confidence interval for mean Mustang price

• Via SE: Drag down a Summary table, drag the xbar variable to the summary, use Summary > Add formula to add s( ) to find the standard deviation.

• Visa percentiles: On the graph (right-click > Plot value) or in a summary table, use a formula like percentile(2.5,xbar) to find a desired percentile in the bootstrap distribution.

CI for mean Mustang price?

Fathom Advantage for Coin Flip Winner in NFL OT?





Approach: Use Fathom to simulate lots of samples of size 428 (when p=1/2). Count the number of “wins” in each sample. See what proportion of the samples give 240 (or more) wins.

*************************************************** ************************** 1. Create a “population” with p=1/2.

� Drag down an empty data table. Label a new variable as Result to represent what happens to the coin flip winner.

� Fill in values for Result with an equal number of y and n values to represent win or loss. For example 5 y’s and 5 n’s. Note: The actual number is irrelevant as long as the proportion agrees with Ho:

2. Create a sample from the “population” and count the number of correct matches.

� Click to select the original collection and choose Collection> Sample Cases.

� Change sample size n=428 of the actual sample (with replacement, no animation).

� Define a measure for the sample collection (find the Measures tab in its inspector). Name the measure Wins and use the formula count(Result=”y”) to compute the simulated number of matches. Record the count value in the table below.

� With the sample collection selected, hit Ctrl-Y to generate new samples and record the number of matches for each sample in the table below.

Wins in 428 trials

3. Collect the measure (count of wins) for lots of samples.

� With the sample collection selected, choose Collection > Collect Measures.

� Adjust the new Measures collection to do collect results for 1000 simulations.

� Create a graph of the Wins values for all the simulations.

4. How many of the 1000 simulations have 240 (or more) matches? _________

Method #1: Right-click on the Wins variable in a table and choose Sort Descending.

Method #2: Use a summary table and add the formula count(Wins ≥≥≥≥ 240) . Hint: To get the ≥≥≥≥ hold down the ctrl key to see it in Fathom’s formula editor.

What proportion of simulated samples had 240 (or more) wins in 428 games? ________

Does this appear to be very unusual when p=1/2?

Fathom Sleep vs. Caffeine for Memory




1. Find the difference in means for the original sample

• Open the data in SleepCaffeine.ftm. Double-click on the collection box to open its inspector.

• Click the Measure tab and define a new measure called Diff and enter a formula to compute the difference in means between the two groups.

mean(Words,Group=”Sleep”)- mean(Words,Group=”Caffei ne”)

Original difference in means: ____________________________

2. Create a randomization sample

• Click to select the SleepCaffeine collection box, choose Collection > Scramble Attribute Values.

• Drag down a Table to look at the new collection. You should see the group assignments reallocated at random (to satisfy the null hypothesis).

• Click on the Measures tab in the inspector for the scrambled collection to see the difference in means for this randomization. Record the value below.

• Click to select the Scrambled collection and hit Ctrl-Y to generate a new randomization. Record the difference in means. Repeat a few more times. Mean difference

3. Collect mean differences for lots of randomization samples

• Click to select the scrambled collection and choose Collection > Collect Measures. Drag down a Table to see the mean differences for five randomizations.

• Double-click on the Measures collection to bring its inspector. Put animation off, replace existing cases and request 1000 measures. Collect More Measures. Drag down a Table (with Measures collection selected) to show the randomization differences.

• Drag down an empty graph and create a dotplot of the Diff variable from the Measures collection to show the randomization distribution of mean differences.

How unusual is the original difference in mean?

To count the number of randomization samples as extreme as the original sample:

• Right–click on the Diff variable and choose Sort Descending, then count how many match or exceed the original difference. OR

• Use a formula in a summary table for the measures collection such as

Count(Diff> 3.0)

• Double the count and divide by 1000 to get an approximate p-value. p-value≈ _______________

SleepCaffeine

Group Words

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

Sleep 14

Sleep 18

Sleep 11

Sleep 13

Sleep 18

Sleep 17

Sleep 21

Sleep 9

Sleep 16

Sleep 17

Sleep 14

Sleep 15

Caffeine 12

Caffeine 12

Caffeine 14

Caffeine 13

Caffeine 6

Caffeine 18

Caffeine 14

Caffeine 16

Caffeine 10

Caffeine 7

Caffeine 15

Caffeine 10

Fathom Correlation of Price and Miles for Used Mustangs

Parameter: ρ = correlation between price and mileage for Mustangs offered at an internet site.

Data: Price and Miles (both in thousands) for a sample of 25 Mustangs.

Task: Find a 95% confidence interval for the correlation between price and miles for Mustangs.

1. Find the correlation for the original sample

• Open the data in MustangPrice.ftm. Double-click on the collection box to open its inspector.

• Click the Measure tab and define a new measure called corr and enter a formula to compute the correlation.

correlation(Price,Miles)

Original correlation: r = ____________________________

2. Create a bootstrap sample

• Click to select the MustangPrice collection box, choose Collection > Sample Cases. Double-click on the new “Sample of MustangPrice” collection box and choose the Sample tab in its inspector.

• Adjust the sample size to match the original sample (25), turn off animation and be sure samples are with replacement. Click on Sample More Cases to generate the new sample. Drag down a Table if you want to see the sample results.

• Choose the Measures tab in the inspector and record the correlation for this sample below. • Select the sample collection and hit Ctrl-Y to generate a few more samples and see how the

correlation varies. 3. Collect measures for lots of bootstrap samples

• Click to select the sample collection is and choose Collection > Collect Measures. This should create a new collection with (by default) the means for five samples. Drag down a Table to see the results (be sure the “Measures from …” collection box is selected).

• Double-click on the new collection to bring up its inspector. Turn animation OFF, replace existing cases and change the number of measures to 1000. Collect More Measures.

• Create a graph of the bootstrap distribution. (drag down an empty graph and drag the corr variable from the data table to the horizontal axis).

4. Compute a bootstrap confidence interval for mean Mustang price

• Via SE: Drag down a Summary table, drag the corr variable to the summary, use Summary > Add formula to add s( ) to find the standard deviation.

• Visa percentiles: On the graph (right-click > Plot value) or in a summary table, use a formula like percentile(2.5,corr) to find a desired percentile in the bootstrap distribution.

CI for correlation between Mustang price and miles?

Matlab

Matlab Instructions

Matlab (http://www.mathworks.com/) is a computing environment widely used in statistics, mathematics,

and other scientific fields. The Matlab language is flexible, which can make bootstrapping, permutations tests and

simulations easy. The command line interface, however, may be intimidating at the introductory level.

Bootstrap Confidence Interval for Mean Mustang Price

Data: A sample of n=25 Mustang cars offered for sale on a website (thanks to Gabe McBride). We are

interested in estimating the mean Price (in $1,000) and finding a confidence interval for that mean.

4.) Load the data. >> load MustangPrice

5.) Find the mean price for the true data.

>> Mean = mean(Price)

Mean price (in $1000): ____________

6.) Generate 1000 bootstrap means.

>> BootStats = bootstrp(1000,@mean,Price)

7.) Plot a histogram of the bootstrap means.

>> hist(BootStats)

8.) Use percentiles of the bootstrap means to create a 90% confidence interval for the mean price.

>> LowerBnd = prctile(BootStats, 5) >> UpperBnd = prctile(BootStats, 95)

90% confidence interval (in $1000): _________________





5.) What is the test statistic, �̂?

>> P_hat = 240/428

6.) Simulate 1000 samples of size 428 under Ho: p=1/2.

>>Sims = binornd(428,0.5,1000,1); >>Sims_P_hat = Sims/428;

7.) Plot the simulated �̂‘s.

>>hist(Sims_P_hat)

8.) Find the p-value (proportion of simulations with result > �̂).

>>p = sum(Sims_P_hat > P_hat)/1000

9.) What can you conclude?

Matlab





7.) Load the data.

>> load(‘SleepCaffeine.mat’)

8.) Find the mean for both groups.

>>mean(Sleep) >>mean(Caffeine)

Mean words (Sleep): _____________ Mean words (Caffeine):_______________

9.) Find the mean difference.

>>Diff = mean(Sleep)-mean(Caffeine)

Difference:_________________

10.) Find the mean difference for 1000 sample permutations.

>>Diff = mean(Sleep)-mean(Caffeine) >>AllData = [Sleep Caffeine]; >>for (i=1:1000) >>Rand = randsample(AllData,24); >>Sleep_rand = Rand(1:12); >>Caffeine_rand = Rand(13:24); >>Diff_rand(i) = mean(Sleep_rand)-mean(Caffeine_ran d); >>end

11.) Find the p-value

>> p = (sum(Diff_rand>3)+sum(Diff_rand<-3))/1000

SAS SAS Instructions

Bootstrap Confidence Interval for Mean Mustang Price Data: A sample of n=25 Mustang cars offered for sale on a website (thanks to Gabe McBride) with three variables price, mileage, and age for each vehicle.

Problem 1: We are interested in estimating the mean Price (in $1,000) and finding a 90% confidence interval for that mean.


• Input the data into SAS. Use proc univariate to find statistics (including the mean) for the sample of prices. Mean price for original sample: _________________

proc univariate data = mustang; var price;

run ;

2. Create a set of bootstrap samples

• Use a new data statement to generate a set of 1,000 bootstrap samples. data bootsamp; do sampnum = 1 to 1000 ; /* 1,000 replicates */ do i = 1 to nobs; x = round(ranuni( 0) * nobs); set mustang nobs = nobs point = x; output; end; end; stop; run ;

3. Collect a mean from each individual bootstrap sample

• Use the output statement in proc univariate with by statement for sample number to collect a data set of bootstrap means.

proc univariate data = bootsamp; var price; by sampnum; output out = bootmeans mean = means; run ;

4. Analyze the distribution of bootstrap means

• Use the proc univariate to calculate basic summary statistics on the boostrap means. The inclusion of the histogram statement plots the distribution.

proc univariate data = bootmeans; var means; histogram means; run ;

Standard deviation of bootstrap means = ___________________ (estimates SE of �̅)

• Compute a rough interval estimate for the mean Mustang price using the mean price for the original sample. �̅ ± 1.5�� =

SAS • Use the quantiles section of the output to find percentiles for a 90% interval estimate Percentile(0.05)= Percentile(0.95)=

Bonus Problem: We are interested in estimating a 95% confidence interval for the correlation between price and mileage.

1. Find the correlation for the original sample.

• Use proc corr to find statistics (including the mean) for the sample of prices. Correlation for original sample: _________________

proc corr data = mustang; var price miles; run ;

2. Create a set of bootstrap samples

• The same set of bootstrap samples from Problem 1 part 2 can be used here. 3. Collect a correlation from each individual bootstrap sample

• Use the outp option in proc corr with a by statement for each sample number to collect a data set of bootstrap correlations.

proc corr data = bootsamp outp = output; var Price Miles; by sampnum; run ;

• The output gives you a correlation matrix as well as means of each variable, clean this up for just the

correlations with respect to price. data outcorr; set output; if _TYPE_ = "CORR"; if _NAME_ = "Price"; run ;

4. Analyze the distribution of bootstrap correlations

• Use the proc univariate to calculate basic summary statistics on the boostrap correlations. The inclusion of the histogram statement plots the distribution.

proc univariate data = outcorr; var miles; histogram miles; run ;

Standard deviation of bootstrap means = _____________ (estimates SE of correlation)

• The percentiles for a 95% confidence interval (2.5,97.5) are not given in the quantiles section of the output. Add these percentiles to caclulate a 95% CI.

proc univariate data = outcorr; var miles; histogram miles; output out = percentiles pctlpre=P_ pctlpts= 2.5 , 97.5 ; run ; proc print data = percentiles; run ;

SAS Advantage for Coin Flip Winner in NFL OT?

Data: A sample of n = 428 NFL overtime games, 240 of which the winner of overtime's opening coin flip won the game.




Approach: Use SAS to simulate many (1,000) samples of size 428 (with p=1/2), counting the number of “wins” in each sample. See what proportion of the samples give 240 (or more) wins.

Problem 2: Is the team winning the overtime coin flip is more likely to win the football game?

1. Generate 1,00 samples data NFL; do samp = 1 to 1000 ; Wins = rand('Binomial', 0.5 , 428 ); output; end; run ;

2. Observe the distribution of win counts proc univariate data = nfl; histogram Wins; run ;

3. Count the number of times win count is equal to or greater then 240 data more; set NFL; if Wins >= 240 ; run ;

• The count is the number of observations in the data set more seen in the log. Randomization test p-value = ___________

SAS





Problem 3: Comparing the number of words recalled in the caffeine and sleep groups.

1. Reallocate 1,000 times to test

• Randomly assign the Sleep and Caffeine treatments to the 24 data values. Count how often the difference is greater than the observed difference (3). This is all done automatically in proc npar1way. proc npar1way scores = data;

class caff; var words; exact scores = data /N = 1000; run ;

• Pay attention to one-tailed or two-tailed test!

technology for teaching bootstraps and randomizationskfl5/lock_uscots_2011.pdf · technology for...

Documents