Estimation: How Large is the Effect?Chapter 2

Chapter OverviewSo far, we can only say things like

◦“We have strong evidence that the long-run probability Buzz pushes the correct button is larger than 0.5.”

◦“We do not have strong evidence kids have a preference between candy and a toy when trick-or-treating.”

We want a method that says ◦“I believe 68 to 75% of all elections can

be correctly predicted by the competent face method.”

Chapter OverviewEstimation tells how large the effect is, through an interval of values. We can be 95% confident that the

“true” effect of taking bi-daily aspirin will reduce the rate of heart attacks somewhere between 30% and 50%.

“If the election were held today, would you vote for Barack Obama or Mitt Romney?” 51% responded Obama (margin of error

is ± 3 percentage points)

Confidence IntervalsThese interval estimates of a population

parameter are called confidence intervals.

We will find confidence intervals three ways.◦Through a series of tests of significance to see

which proportions are plausible values for the parameter.

◦Using the standard deviation of the simulated null distribution to help us determine the width of the interval.

◦Through traditional theory-based methods.

Statistical Inference – Confidence IntervalsSection 2.1

Can Dogs Sniff Out Cancer?Example 2.1

Marine sniffing samples

Can Dogs Sniff Out Cancer?Marine, a dog originally trained for water rescues, was tested to see if she could detect whether a patient had colorectal cancer by smelling a sample of their breath.She first smells a bag from a patient with

colorectal cancer.Then she smells 5 other samples; 4 from

normal patients and 1 from a person with colorectal cancer

She is trained to sit next to the bag that matches the scent of the initial bag (the “cancer scent”) by being rewarded with a tennis ball.

Can Dogs Sniff Out Cancer?Marine was tested with a set of 33 trials.

Null hypothesis: Marine is randomly guessing which bag is the cancer specimen ( = 0.20)

Alternative hypothesis: Marine can detect cancer better than guessing ( > 0.20)

represents her long-run probability of identifying the cancer specimen.

Can Dogs Sniff Out Cancer?30 out of 33 trials resulted in

Marine correctly identifying the bag from the cancer patient

So our sample proportion is

Do you think Marine can detect cancer?

What sort of p-value will we get?

Can Dogs Sniff Out Cancer?

Our sample proportion lies more than 10 standard deviation above the mean and hence our p-value is 0.

Can Dogs Sniff Out Cancer?While we found that Marine’s probability

of picking the correct specimen is greater than 0.2, but what is it?

Since our sample proportion is about 0.909, it is plausible that 0.909 is a value for the probability. What about other values?

Is it plausible that Marine’s probability is actually 0.70 and she had a lucky day?

Is a sample proportion of 0.909 unlikely if = 0.70?

Can Dogs Sniff Out Cancer?H0: = 0.70 Ha: ≠ 0.70We get a small p-value (0.011) so

we can rule out 0.70 as a probability

Can Dogs Sniff Out Cancer?What about 0.80? Is 0.909 unlikely if = 0.80?

Can Dogs Sniff Out Cancer?

H0: = 0.80 Ha: ≠ 0.80We get a large p-value (0.146) so 0.80 is

a plausible value for Marine’s long-run probability.

Developing a range of plausible values

If I get a small p-value (like I did with 0.70) I will conclude that the value under the null is not plausible. This is when we conclude the alternative hypothesis.

If I get a large p-value (like I did with 0.80) I will conclude the value under the null is plausible. This is when I can’t conclude the alternative.

Developing a range of plausible values

Let’s use the one-proportion inference applet to find a range of plausible values for Marine’s long term probability of choosing the correct specimen.

We will keep the sample proportion the same and change the possible values

We will first use 0.10 as our cutoff value for if a p-value is small or large. (This is called the significance level.)

Can Dogs Sniff Out Cancer?

We should have found that values between 0.79 and 0.96 are plausible values for Marine’s probability of picking the correct specimen.

We can do more tests and find a more precise interval to be 0.787 to 0.966.Probability

under null 0.785 0.786

0.787

0.788 …………

0.965

0.966 0.967 0.968

p-value 0.085 0.093 0.110

0.144

0.109

0.102 0.094 0.088

Plausible? No No Yes Yes … Yes … Yes Yes No No

Can Dogs Sniff Out Cancer?

(0.787, 0.966) is called a confidence interval.

Since we used 10% as our significance level, this is a 90% confidence interval. (100% − 10%)

90% is the confidence level of the interval of plausible values.

Can Dogs Sniff Out Cancer?We would say we are 90%

confident that Marine’s probability of correctly picking the bag with breath from the cancer patient from among 5 bags is between 0.797 and 0.966.

This is a more precise statement than our initial significance test which concluded Marine’s probability was more than 0.20.

Significance Level Confidence Level

Typically we use 0.05 for our significance level. There is nothing magical about 0.05. We could set up our test to make it harder to conclude the alternative (smaller significance level) or easier (larger significance level).

If we increase the confidence level from 90% to 95%, what will happen to the width of the confidence interval?

Can Dogs Sniff Out Cancer?

Since the confidence level gives an indication of how sure we are that we captured the actual value of the parameter in our interval, to be more sure our interval should be wider.

How would we obtain a wider interval of plausible values to represent a 95% confidence level?◦Use a 5% significance level in the tests.

Can Dogs Sniff Out Cancer?

Values that correspond to 2-sided p-values larger than 0.05 should now be in our interval.

Using the table we developed, what is the 95% confidence interval for Marine’s long-run probability?

Exploration 2.1: Kissing Right

As you work through this exploration …

Exploration 2.1Your first test will be one-sided, but after

that everything is a two-sided test.The sample proportion stays constant.The parameter under the null will change

since we are testing to see if different parameters are plausible or not.

For small p-values we can rule the parameter out.

For large p-values, the parameter is plausible.

2SD and Theory-Based Methods for Confidence IntervalsSection 2.2

IntroductionIn Section 2.1 we found confidence intervals

by doing repeated tests of significance (changing the value in the null hypothesis) to find a range of values that were plausible for the population parameter (long run relative frequency or probability).

This is a very tedious way to construct a confidence interval.

Today, we will look at two other ways to construct confidence intervals.◦Two Standard Deviations (2SD).◦Theory-based.

In example 1.5 we looked at a study where researchers investigated whether children show a preference to toys or candy

Test households in five Connecticut neighborhoods offered children two plates: ◦One with candy ◦One with small, inexpensive toys

Example 2.2: Halloween Treats (continued)

We tested the hypotheses:◦Null: The proportion of trick-or-

treaters who choose candy is 0.5. (π= 0.5)

◦Alternative: The proportion of trick-or-treaters who choose candy is not 0.5. (π ≠ 0.5)

Our sample proportion was:◦148 out of 283 (52.3%) chose candy

Halloween Treats

Halloween Treats

When we ran this test, we got a very large p-value so we could not conclude that π was not 0.5. This means 0.5 is a plausible value for π.

Halloween TreatsWhat are some other plausible

values for π?Through repeated tests we come

up with:Prob. under null

0.463

0.464

0.465

…………

0.581

0.582

0.583

2-sided p-value

0.048

0.049

0.057

…………

0.053

0.047

0.046

Plausible value (@0.05)?

No No Yes …………

Yes No No

Halloween TreatsThus we found a 95% confidence interval

of 0.465 to 0.581.This means, we are 95% confident that

the probability a child will choose candy while trick-or-treating is between 0.465 and 0.581

Does it make sense that 0.5 is in the interval? Why?

Halloween TreatsRemember that a p-value of less

than 0.05 corresponds with a standardized statistic of 2 or larger (or -2 or smaller)

Hence for most symmetric, bell-shaped distributions, about 95% of the values in the distribution fall within 2 SD of the mean.

Halloween TreatsSo we could say that a parameter

value is plausible if it is within 2 standard deviations (SD) from our best estimate of the parameter, our observed sample statistic.

This gives us the simple formula for a 95% confidence interval of

Halloween TreatsUsing the 2SD method on our Halloween

treat data we get a 95% confidence interval

The 0.06 in the above is called a margin of error.

The interval can also be written as we did before using just the endpoints; (0.463, 0.583)

This is approximately what we got with our range of plausible values method

Halloween TreatsThe 2SD method only gives us a 95%

confidence intervalIf we want a different level of confidence,

we can us the range of plausible values (hard) or theory-based methods (easy).

In the theory-based applet, you can input any level of confidence and the applet will calculate the confidence interval for you. This is valid provided there are at least 10 successes and 10 failures in your sample (validity conditions).

AppletsLet’s check out this example

using applets and doing the 2SD method and theory-based method.

Remember 52.3% of 283 trick-or-treaters chose candy.

Predicting Elections from Faces (continued)Exploration 2.2

Factors that affect the width of a confidence intervalSection 2.3

Factors AffectingConfidence Interval Widths

Level of confidence (e.g., 90% vs. 95%)◦As we increase the confidence level, we

increase the width of the interval.Sample size

◦As sample size increases, variability decreases and hence the standard deviation will be smaller. This will result in a narrower interval.

Sample proportion ◦Wider intervals when is closer to 0.5.

Level of ConfidenceLet’s take another look at the St.

George Hospital heart transplant data.

In the sample of 361 patients, 71of them died within 30 days of their heart transplant, and 290 survived.

Each of these counts of “successes” and “failures” is greater than 10, so we can use our theory-based applet to play around with confidence intervals.

Level of Confidence Since the standard deviation is predictable, we can use the Theory-Based Inference Applet to easily find a confidence interval for the 30 day mortality rate for heart transplant patients at St. George’s. (Notice that the confidence level can be adjusted.)

Level of ConfidenceWhat happens to the width of the

confidence interval as we change the level of confidence?

As the level of confidence increases, the width of our confidence interval increases (and hence the margin of error increases).

We are more confident of capturing our parameter in a wider range of values.

Level of Confidence 80% 90% 95% 99%Confidence

Interval(0.170, 0.224)

(0.163, 0.231)

(0.156, 0.238)

(0.143, 0.251)

0.197±0.027 0.197±0.034 0.197±0.041 0.197±0.054

Sample SizeWe know as sample size

increases, the variability (and thus standard deviation) in our null distribution decreases

n = 90 (SD = 0.054) n = 361 (SD = 0.026) n = 1444 (SD = 0.013)

Sample size 90 361 1444SD of null distr. 0.053 0.027 0.013Margin of error 2 x SD = 0.106 2 × SD = 0.054 2 × SD = 0.026Confidence interval

(0.091, 0.303) (0.143, 0.251) (0.171, 0.223)

Sample Size(With everything else staying the

same) increasing the sample size will make a confidence interval narrower.

Notice:The observed sample proportion is

the midpoint. (that won’t change)Margin of error is a multiple of the

standard deviation so as the standard deviation decreases, so will the margin of error.

Formula for Theory-Based Confidence IntervalThe Theory-Based Inference applet uses the

one proportion z-interval formula to calculate confidence intervals:

+ z* The width of the confidence interval

increases as level of confidence increases (see z*)

The width of the confidence interval decreases as the sample size increases

The value also has a more subtle effect. The farther it is from 0.5 the smaller the width.

SummaryIncreasing the level of confidence

will make a confidence interval wider.

Increasing the sample size will make a confidence interval narrower.

The farther is from 0.5 will make a confidence interval narrower.

What does 95% confidence mean?If we repeatedly sampled from a

population and constructed 95% confidence intervals, 95% of our intervals should contain the population parameter.

Notice the interval is the random event here.

The population parameter is a fixed number, we just don’t know what it is.

Simulating Confidence Intervals Applet.

Type I errorThink back to the St. George’s example that

looked at deaths of heart transplant patients.

We concluded that their death rate was higher than the national average. Suppose this resulted in ceasing operations at the hospital.

Also suppose that in reality their rate was really the same as the national average.

What we have done is to reject a true null hypothesis. This is called a type I error and is sometimes referred to a false alarm.

Type II errorNow suppose we obtained a large p-value so

we didn’t get significant results in the St. George’s example.

Hence, we could not conclude that their death rate was higher than the national average. And heart operations continued at the hospital.

Also suppose that in reality their rate was in fact higher than the national average.

What we have done is to not reject a false null hypothesis. This is called a type II error and is sometimes referred to a missed opportunity.

Type I and Type II Errors

    What is true (unbeknownst to us)

    Null hypothesis is true

Null hypothesis is

false

What we decide(based

on data)

Reject null hypothesis

Type I Error(false alarm)

Correct Decision

Do not reject null hypothesis

Correct DecisionType II Error

(missed opportunity)

Type I and Type II errorsNull is true Defendant is

innocent

Null is falseDefendant is

guilty

Reject nullJury finds

defendant guilty

Type I ErrorInnocent person

goes to prison

Correct DecisionGuilty person goes to prison

Not reject nullJury finds

defendant not guilty

Correct DecisionInnocent person

is set free

Type II ErrorGuilty person is

set free

The probabilities of Type I and Type II errors

The significance level is the criterion used to reject the null hypothesis. We have been using 0.05 as our significance level.

The probability of a type I error is the significance level. (Suppose the significance level is 0.05. If the null is true we would reject it 5% of the time and thus make a type I error 5% of the time.)

Type I and Type II ErrorsThink back to the dog sniffing

cancer study:◦Describe what a type I error would

be.◦Describe what a type II error would

be.

Competitive advantage to Uniform Color (continued)Exploration 2.3A