lecture 12: introduction to hypothesis testing api-201zhypothesis testing example: trial for malaria...

$Page 1: Lecture 12: Introduction to Hypothesis Testing API-201ZHypothesis Testing Example: Trial for Malaria Vaccine I From the Guardian (UK): I \The researchers found that vaccinated children$

Lecture 12:Introduction to Hypothesis Testing

API-201Z

Maya Sen

Harvard Kennedy Schoolhttp://scholar.harvard.edu/msen

http://scholar.harvard.edu/msen

Announcements

I Nice job on the midterms!

I Now available for pick up (Melissa Kappotis, Belfer 4th Floor)

Announcements

Roadmap

I Continuing building statistical inference

I Quick review of CLT

I Start hypothesis testing

I One and Two-tailed tests

I z-tests and t-tests

Roadmap

Central Limit Theorem

Central Limit Theorem (CLT) has two parts:

I (1) The sums and means of independent random variableshave an approximately normal distribution

I (2) This distribution becomes “more and more normal” themore observations are included in the sum or the mean

I Under CLT → As n goes up, distribution of X̄ approaches:

X̄ ∼ N(µ,σ2/n)

Central Limit Theorem Example

Take 500 samples of n = 2, average, and plot:

Distribution of the Sample Mean

Pounds

Den

sity

100 120 140 160 180

0.00

0.05

0.10

0.15

0.20

Pounds

Den

sity

100 120 140 160 180

0.00

0.05

0.10

0.15

0.20

Pounds

Den

sity

100 120 140 160 180

0.00

0.05

0.10

0.15

0.20

Pounds

Den

sity

100 120 140 160 180

0.00

0.05

0.10

0.15

0.20

Pounds

Den

sity

100 120 140 160 180

0.00

0.05

0.10

0.15

0.20

Pounds

Den

sity

100 120 140 160 180

0.00

0.05

0.10

0.15

0.20

Pounds

Den

sity

100 120 140 160 180

0.0

0.1

0.2

0.3

0.4

0.5

0.6

Pounds

Den

sity

100 120 140 160 180

0.0

0.1

0.2

0.3

0.4

0.5

0.6

Pounds

Den

sity

100 120 140 160 180

0.0

0.2

0.4

0.6

0.8

1.0

1.2

Hypothesis Testing

I How can we leverage one sample to tell us about thepopulation we might care about?

I Hypothesis tests: Use sample data to address a question ofinterest about a population (or populations)

I Commonly used tool in translating public policy questions intoa quantitative framework

I Follows rough steps every time

Hypothesis Testing

Hypothesis Testing Example: Lady Tasting Tea

I Lady claims she can tell whether milk or tea poured first

I She chooses 4 cups out of 8 correctly

I Question: Does she choose differently than what you wouldexpect if she just had guessed?

Hypothesis Testing Example: Trial for Malaria Vaccine

I From the Guardian (UK):

I “The researchers found that vaccinated children were 30%less likely to have suffered at least one episode of clinicalmalaria (that needed treatment) by the end of the 6-monthtrial, compared with unvaccinated children”

I Question: Does this suggest that vaccinated children in thepopulation suffer less from malaria than unvaccinatedchildren?

Hypothesis Testing Example: Household Size

I Census data from 2000 give an average of 2.5 people perhousehold

I In 2018, you collect random sample of 400 households,calculated mean household size of 2.36 people and SD of 2.0

I Question: Taking 2000 as a benchmark, does the 2014 datasupport belief household size has changed?

Hypothesis Testing: Key Idea

I Will a hypothesis test tell us for certain the answer to ourquestion of interest?

I No: Hypothesis testing does not provide w/ certain answers

I Instead provides “strength of evidence” based upon sampledata

I Not: “How likely is my hypothesis?”

I Instead: “How likely is it that we would have drawn a sample‘like’ the one we got, given/not given my hypothesis?”

Hypothesis Testing for a Single Mean

Conducted using several steps:

1. Generate your null and alternative hypotheses

2. Collect sample of data

3. Calculate appropriate test statistic

4. Use that to calculate a probability called a p-value (morelater)

5. Use the p-value to decide whether to reject null hypothesisand interpret results

Null and Alternative Hypotheses

I Null Hypothesis (H0): Some statement about the populationparameters

I The “Devil’s Advocate” hypothesis → Assumes whatever youseek to prove did not happen

I Ex) Lady Tasting Tea guessing randomlyI Ex) Household size in 2000 and 2018 are the sameI Ex) Medical treatment has no positive effect on preventing

diseasesI Usually “no effect” or “no difference” or “due to chance”

I Alternative Hypothesis (Ha or H1) The statement we hope orsuspect is true instead of H0

I Ex) Lady Tasting Tea guessing non-randomlyI Ex) Household size in 2018 has gone down from 2000I Ex) Medical treatment has positive effect on preventing

diseases

I → We move forward by trying to disprove the null (asopposed to proving the alternative)

diseases

I Ex) Lady Tasting Tea guessing randomly

I Ex) Household size in 2000 and 2018 are the sameI Ex) Medical treatment has no positive effect on preventing

diseases

I Ex) Lady Tasting Tea guessing randomlyI Ex) Household size in 2000 and 2018 are the same

I Ex) Medical treatment has no positive effect on preventingdiseases

I Usually “no effect” or “no difference” or “due to chance”

diseases

I Usually “no effect” or “no difference” or “due to chance”

diseases

I Ex) Lady Tasting Tea guessing non-randomly

I Ex) Household size in 2018 has gone down from 2000I Ex) Medical treatment has positive effect on preventing

diseases

I Ex) Lady Tasting Tea guessing non-randomlyI Ex) Household size in 2018 has gone down from 2000

I Ex) Medical treatment has positive effect on preventingdiseases

diseases

I Let’s walk through this using this example

I 2000 U.S. Census: Shows 2.5 people per household

I Separate study in 2018: Shows 2.36 people per household, w/sample size of n = 400 w/ SD of 2.0

I Question: Do 2018 data support belief household size haschanged since 2000?

I What is our population under study?I U.S. households

I What is the parameter we are interested in finding out about?I The average number of people per household, µ

I How would you state the null and alternative hypotheses?I H0 : µ2000 = µ2018I Ha : µ2000 6= µ2018

I Always stated in terms of population parameters (why?because the estimates from the sample are known, nouncertainty)

I What is our population under study?

I U.S. households

I What is the parameter we are interested in finding out about?

I The average number of people per household, µ

I How would you state the null and alternative hypotheses?

I H0 : µ2000 = µ2018I Ha : µ2000 6= µ2018

I How would you state the null and alternative hypotheses?I H0 : µ2000 = µ2018

I Ha : µ2000 6= µ2018I Always stated in terms of population parameters (why?

because the estimates from the sample are known, nouncertainty)

I Step 1: Null and Alternative HypothesesI H0 : µ2000 = µ2018I Ha : µ2000 6= µ2018

I Step 2: Collect sample dataI Done → (Ok, well, “data” collected for us)

I Step 3: Calculate appropriate test statisticI This test statistic will help us compare our data to the

distribution of the data under the null hypothesis (the nulldistribution)

I Allows us to assess evidence against null hypothesisI But how to we calculate it?

I Step 1: Null and Alternative Hypotheses

I H0 : µ2000 = µ2018I Ha : µ2000 6= µ2018

I Step 1: Null and Alternative HypothesesI H0 : µ2000 = µ2018

I Ha : µ2000 6= µ2018I Step 2: Collect sample data

I Done → (Ok, well, “data” collected for us)

I Step 2: Collect sample data

I Done → (Ok, well, “data” collected for us)

I Step 3: Calculate appropriate test statistic

I This test statistic will help us compare our data to thedistribution of the data under the null hypothesis (the nulldistribution)

I Allows us to assess evidence against null hypothesis

I But how to we calculate it?

Calculating a Test Statistic for Means

Remember:

1. CLT: Estimators such as sample mean Normally distributedwhen n goes up

X̄ ∼ N(µ,σ/√n)

2. These are normals that can be standardized to get a standardnormal

Z =X̄ − µ

σ/√n

Make sure to understand each step (if not, go back to previouslectures)

Remember:

X̄ ∼ N(µ,σ/√n)

Z =X̄ − µ

σ/√n

Remember:

X̄ ∼ N(µ,σ/√n)

Z =X̄ − µ

σ/√n

Remember:

X̄ ∼ N(µ,σ/√n)

Z =X̄ − µ

σ/√n

Remember:

X̄ ∼ N(µ,σ/√n)

Z =X̄ − µ

σ/√n

Remember:

X̄ ∼ N(µ,σ/√n)

Z =X̄ − µ

σ/√n

Remember:

X̄ ∼ N(µ,σ/√n)

Z =X̄ − µ

σ/√n

I Following from previous slide, 2018 sample mean can beapproximated using normal and standardized

I So:

Z =X̄2018 − µ2018σ2018/

√n2018

I Magic step: Assume null hypothesis is true: µ2000 = µ2018(No diff between 2000 and 2018)

I Estimate test statistic, z , using sample values:

z =X̄2018 − µ2010σ2018/

√n2018

≈ 2.36 − 2.50

2.0/√

400

= −1.4

I Note: Using sample statistics introduces somenoise/additional variance, which we’ll deal with later

I So:

Z =X̄2018 − µ2018σ2018/

√n2018

z =X̄2018 − µ2010σ2018/

√n2018

≈ 2.36 − 2.50

2.0/√

400

= −1.4

I So:

Z =X̄2018 − µ2018σ2018/

√n2018

z =X̄2018 − µ2010σ2018/

√n2018

≈ 2.36 − 2.50

2.0/√

400

= −1.4

I So:

Z =X̄2018 − µ2018σ2018/

√n2018

z =X̄2018 − µ2010σ2018/

√n2018

≈ 2.36 − 2.50

2.0/√

400

= −1.4

I So:

Z =X̄2018 − µ2018σ2018/

√n2018

z =X̄2018 − µ2010σ2018/

√n2018

≈ 2.36 − 2.50

2.0/√

400

= −1.4

I So:

Z =X̄2018 − µ2018σ2018/

√n2018

z =X̄2018 − µ2010σ2018/

√n2018

≈ 2.36 − 2.50

2.0/√

400

= −1.4

I So:

Z =X̄2018 − µ2018σ2018/

√n2018

z =X̄2018 − µ2010σ2018/

√n2018

≈ 2.36 − 2.50

2.0/√

400

= −1.4

I Step 4: Use test statistic to calculate p-value

I If null is true (which we assumed for purposes of hypothesistest), what distribution should z come from? (standardnormal)

I p-value: Given that null is true (steps on previous slide), whatis probability of seeing test statistic as extreme as the one wegot?

I Plain English: How likely would it be to see the test statisticthat extreme given a standard normal

I → Implies smaller the p-value the more compelling is evidencethat null hypothesis should be rejected

I How to calculate? Refer to the standard normal

I If null is true (which we assumed for purposes of hypothesistest), what distribution should z come from?

(standardnormal)

Can use z-score table to calculate these:

−4 −2 0 2 4

0.0

0.1

0.2

0.3

0.4

0.5

z

−4 −2 0 2 4

0.0

0.1

0.2

0.3

0.4

0.5

z

z = −1.4

One-sided versus Two-sided Tests

I Note: p-value depends on which alternate hypothesis isrelevant

I Distinguish between one-sided and two-sided tests:

I Ha : µ < 2.5 (1-sided)

I Ha : µ > 2.5 (1-sided)

I Ha : µ 6= 2.5 (2-sided)

I Ha : µ < 2.5 (1-sided)

I Ha : µ > 2.5 (1-sided)

I Ha : µ 6= 2.5 (2-sided)

I Ha : µ < 2.5 (1-sided)

I Ha : µ > 2.5 (1-sided)

I Ha : µ 6= 2.5 (2-sided)

I Ha : µ < 2.5 (1-sided)

I Ha : µ > 2.5 (1-sided)

I Ha : µ 6= 2.5 (2-sided)

I Ha : µ < 2.5 (1-sided)

I Ha : µ > 2.5 (1-sided)

I Ha : µ 6= 2.5 (2-sided)

I Ha : µ < 2.5 (1-sided)

I Ha : µ > 2.5 (1-sided)

I Ha : µ 6= 2.5 (2-sided)

I Should decide before carrying out study whether to use a one-or two-sided hypothesis test

I When might we choose to use a one-sided hypothesis testversus a two-sided hypothesis test?

I 2-sided test more conservative (why?), so best to use that ifany uncertainty exists about the relationship of µ and µ0

−4 −2 0 2 4

0.0

0.1

0.2

0.3

0.4

0.5

z

z = −1.4

0.08

−4 −2 0 2 4

0.0

0.1

0.2

0.3

0.4

0.5

z

0.08 0.08

z = −1.4 −z = 1.4

−4 −2 0 2 4

0.0

0.1

0.2

0.3

0.4

0.5

z

0.08 0.08

z = −1.4 −z = 1.4

p−value = 0.16

I Step 5: Decide whether to reject null and interpret results

I But how do you know a p-value is too extreme?

I Usually rely on critical values, denoted as α

I α: is the probability level we set to determine that p-value isso low as to be ”unlikely”

I Common to use α = 0.05

I Note: α represents your tolerance of rejecting the null, giventhat null is true

I This is a Type I error

I So compare p-value to critical value, α

I Consider rules of thumb in policy/social science

p-value Accepted Interpretation

p-value > 0.10 No evidence to reject H0

0.05 < p-value 6 0.10 Weak evidence to reject H0

0.01 < p-value 6 0.05 Some evidence to reject H0

0.001 < p-value 6 0.01 Strong evidence to reject H0

p-value 6 0.001 Very strong evidence to reject H0

I What is your conclusion in our example, where p-value is0.16?

I → No evidence to reject null hypothesis that household sincehas changed since 2000

Two Potential Problems

Used standard normal in example, but might not be wise – why?

1. Reasoning relies on CLTI CLT relies on sampling distributions approximating normal as n

goes upI What if you have small n (and non-normal distributions)?I Using CLT questionable

2. We used sample parameters to approximate the standardnormal (e.g., sample standard deviation)

I But standardizing assumes you know population parametersI Thus, we did our best, but introduced additional uncertainty

I To take this into account, nearly all hypothesis testing usesStudent’s t distribution instead of normal

1. Reasoning relies on CLT

I CLT relies on sampling distributions approximating normal as ngoes up

I What if you have small n (and non-normal distributions)?I Using CLT questionable

goes up

I What if you have small n (and non-normal distributions)?I Using CLT questionable

goes upI What if you have small n (and non-normal distributions)?

I Using CLT questionable

I But standardizing assumes you know population parameters

I Thus, we did our best, but introduced additional uncertainty

Who was “Student”?

Student’s t-distribution

I Similar in shape to Normal distribution, but with fatter tails

I For sample sizes > 100, t distribution and N(0, 1)distributions virtually identical

I Thus: Use t distribution to be conservative, but inferencesconverge as n goes up

I t distribution shape determined by size of sample

I Exact shape requires knowing the degrees of freedom, df or ν

I Degrees of freedom takes into account # of observations andfact that you need data to estimate parameters

I For the sample mean, ν = n − 1

I Thus, if three observations ν = 3 − 1 = 2

I The probability density function (pdf) for the t-distribution is:

f (x) =Γ(ν+1

2 )√νπ× Γ(ν2 )

(1 +x2

ν)−(ν+1)/2

I where ν is degrees of freedom

I and Γ(n) = (n − 1)!

I Test statistic calculated similarly to before:

tdf =X̄ − µ0s/√n

I and we compare this to the appropriate t distribution (withdf ) as opposed to standard normal

f (x) =Γ(ν+1

2 )√νπ× Γ(ν2 )

(1 +x2

ν)−(ν+1)/2

I and Γ(n) = (n − 1)!

f (x) =Γ(ν+1

2 )√νπ× Γ(ν2 )

(1 +x2

ν)−(ν+1)/2

I and Γ(n) = (n − 1)!

f (x) =Γ(ν+1

2 )√νπ× Γ(ν2 )

(1 +x2

ν)−(ν+1)/2

I and Γ(n) = (n − 1)!

f (x) =Γ(ν+1

2 )√νπ× Γ(ν2 )

(1 +x2

ν)−(ν+1)/2

I and Γ(n) = (n − 1)!

f (x) =Γ(ν+1

2 )√νπ× Γ(ν2 )

(1 +x2

ν)−(ν+1)/2

I and Γ(n) = (n − 1)!

f (x) =Γ(ν+1

2 )√νπ× Γ(ν2 )

(1 +x2

ν)−(ν+1)/2

I and Γ(n) = (n − 1)!

f (x) =Γ(ν+1

2 )√νπ× Γ(ν2 )

(1 +x2

ν)−(ν+1)/2

I and Γ(n) = (n − 1)!

When to use z-test versus t-tests

I z-tests follow normal distribution while t-tests follow studentst-distribution

I t-test appropriate w/ small samples (n 6 30) while z-testprobably ok w/ moderate to large samples (n > 30)

I However, t-tests more conservative than z-test, b/c z-testsassume you know true population standard deviation

I B/c you usually don’t: t-tests more common than z-tests(used in STATA, R)

I However: z-tests more accurate than t-tests when populationstandard deviations known

Work Week Example

I Suppose you study workers rights provisions

I Fact: 40 hr work week standard in U.S. → true elsewhere inworld?

I Your RA records average workweek for a sample of 36countries with mean of 40.31 and SD of 2.97

I Conduct a hypothesis test to examine whether the averageworld-wide work week length differs from 40 hours

Work Week Example

I Step 1: Null and Alternative HypothesesI H0 : µ = 40I Ha : µ 6= 40

I Step 2: Collect sample dataI n= 36 countries, X̄ = 40.3, s = 2.97

Work Week Example

I Step 1: Null and Alternative Hypotheses

I H0 : µ = 40I Ha : µ 6= 40

Work Week Example

I Step 1: Null and Alternative HypothesesI H0 : µ = 40

I Ha : µ 6= 40

Work Week Example

I Step 2: Collect sample data

I n= 36 countries, X̄ = 40.3, s = 2.97

Work Week Example

I Step 3: Calculate test statistic

t35 =40.31 − 40

2.97/√

36

≈ 0.626

I Step 4: Calculate p-valuedisp 2*ttail(35,.626)

.53537632

. ttesti 36 40.31 2.97 40

One-sample t test

------------------------------------------------------------------------------

| Obs Mean Std. Err. Std. Dev. [95% Conf. Interval]

---------+--------------------------------------------------------------------

x | 36 40.31 .495 2.97 39.3051 41.3149

------------------------------------------------------------------------------

mean = mean(x) t = 0.6263

Ho: mean = 40 degrees of freedom = 35

Ha: mean < 40 Ha: mean != 40 Ha: mean > 40

Pr(T < t) = 0.7324 Pr(|T| > |t|) = 0.5352 Pr(T > t) = 0.2676

Work Week ExampleI Step 3: Calculate test statistic

t35 =40.31 − 40

2.97/√

36

≈ 0.626

.53537632

. ttesti 36 40.31 2.97 40

One-sample t test

------------------------------------------------------------------------------

---------+--------------------------------------------------------------------

x | 36 40.31 .495 2.97 39.3051 41.3149

------------------------------------------------------------------------------

Pr(T < t) = 0.7324 Pr(|T| > |t|) = 0.5352 Pr(T > t) = 0.2676

t35 =40.31 − 40

2.97/√

36

≈ 0.626

.53537632

. ttesti 36 40.31 2.97 40

One-sample t test

------------------------------------------------------------------------------

---------+--------------------------------------------------------------------

x | 36 40.31 .495 2.97 39.3051 41.3149

------------------------------------------------------------------------------

Pr(T < t) = 0.7324 Pr(|T| > |t|) = 0.5352 Pr(T > t) = 0.2676

t35 =40.31 − 40

2.97/√

36

≈ 0.626

I Step 4: Calculate p-value

disp 2*ttail(35,.626)

.53537632

. ttesti 36 40.31 2.97 40

One-sample t test

------------------------------------------------------------------------------

---------+--------------------------------------------------------------------

x | 36 40.31 .495 2.97 39.3051 41.3149

------------------------------------------------------------------------------

Pr(T < t) = 0.7324 Pr(|T| > |t|) = 0.5352 Pr(T > t) = 0.2676

t35 =40.31 − 40

2.97/√

36

≈ 0.626

.53537632

. ttesti 36 40.31 2.97 40

One-sample t test

------------------------------------------------------------------------------

---------+--------------------------------------------------------------------

x | 36 40.31 .495 2.97 39.3051 41.3149

------------------------------------------------------------------------------

Pr(T < t) = 0.7324 Pr(|T| > |t|) = 0.5352 Pr(T > t) = 0.2676

Work Week Example

I Step 5: Based on p-value = 0.5352, decide whether to rejectnull

I What’s your conclusion?

Work Week Example

When to use t and when to use Std Normal?

I Use Standard Normal distribution when:I You know the population standard deviation (σ)I Unusual, but we will tell you if you can make that assumption

on problem sets/exams

I Use t distribution when:I You don’t know population standard deviationI You use the sample standard deviation to estimate it, and use

that to calculate test statisticI You wish to be more conservative (you have a very small

sample size)

I Use Standard Normal distribution when:

I You know the population standard deviation (σ)I Unusual, but we will tell you if you can make that assumption

sample size)

I Use Standard Normal distribution when:I You know the population standard deviation (σ)

I Unusual, but we will tell you if you can make that assumptionon problem sets/exams

sample size)

I Use t distribution when:

I You don’t know population standard deviationI You use the sample standard deviation to estimate it, and use

sample size)

I Use t distribution when:I You don’t know population standard deviation

I You use the sample standard deviation to estimate it, and usethat to calculate test statistic

I You wish to be more conservative (you have a very smallsample size)

that to calculate test statistic

I You wish to be more conservative (you have a very smallsample size)

sample size)

Next Time

I More on hypothesis tests

I Extending this framework to comparing two populationsmeans

I Type I and Type II errors

lecture 12: introduction to hypothesis testing api-201zhypothesis testing example: trial for malaria...

Documents