Stat BootcampSession 2: Confidence Intervals

and Hypothesis Testing

Professor PK Kannan1

Agenda

1. Overview + review

2. Confidence Intervals

3. Hypothesis Testing

2

Part I: Overview + review

3

Population versus sampleLast week, we learned the concepts of population and sample:

• Population: all members of a defined group that we are interested in studying

• Sample: a part of the population• Our interest is to know the population, but can only observe

samples• So, we use the sample statistics to make inference on the

population

4

Sampling Error or Sampling Bias• Definition: the difference between a value (statistic)

computed from a sample and the corresponding value (a parameter) computed from a population

In the example below: parameter of interest is the mean

Population Mean Sample Mean

5

Sampling Error - x µ=

µ x

we say is an estima fte or x µ

• Suppose we are interested in knowing the number of hours people spend on social media per week

• Suppose the true population mean is 18.5 hours per week

• We gather data from a sample of respondents and yield a sample mean of 20.4 hours per week

• The sampling error (or bias) is 1.9 hoursPopulation Mean Sample Mean

6

Example

18.5 hoursµ = 20.4 hoursx =

20.4 18.5 1.9 hoursx µ− = − =

• In practice, however, the population value is often unknown to us (otherwise, we don’t need to conduct statistical analysis), and we only observe the sample data

• To make inference on the population, we would use the sample data to (1) construct a confidence interval and/or (2) perform hypothesis testing.

• Examples• We use data from multiple stores to estimate the

average unit sales of Tide Pods• We want to know whether more than 40% of American

adults can recall at least one Super Bowl Commercial

7

Overview of inference

8

Continuousvariables

Discrete variables

Mean Proportion

Mean Proportion

Inference

EstimationTo construct a confidence interval for a population

parameter

Hypothesis TestingTo assess a statement (hypothesis) about a

population parameter

Part II: Confidence Intervals

9

Topics to cover for CI

• Conceptual understanding of CI• Computing a CI of the mean

• using the formula• using StatTools

• Computing a CI of the proportion • using the formula• using StatTools

• Relationship between margin of error and the sample size

10

Central Limit Theorem revisited

• We are interested in estimating the population mean for a random variable X (for example, the average nights per week that people eat out)

• To form an estimate, we can generate a sample with n people and measure X

• Each time we generate a new sample, we would likely get a different group of n people and thus get a different sample mean

• If we repeatedly draw samples of size n and plot the sample means, we get the sampling distribution of the mean

• Central limit Theorem: the sample means will be approximately normal as long as the sample size is large enough

• How large is large enough: n > 30

11

• Point estimate • is a point estimate for • But a single sample mean does not convey the uncertainty

associated with the estimation

• A confidence interval provides additional information about variability of the sampling distribution

• Gives a range of values based on observations from one sample

• Gives information about the closeness to the unknown population mean

• Stated in terms of level of confidence: never 100% sure

12

Estimating population mean

x µ

• Symmetric around the point estimate• Margin of error = (Multiple) X (standard error)

• Reflects the uncertainty or precision of the estimation• Multiplier: determined by the level of confidence • Standard error: to be computed using a formula or using StatTools

• Now, let’s do a poll…

13

Confidence interval calculation

CI = Point Estimate (Multiple) (Standard Error)± ×

Margin of Error

Point EstimateLower Confidence Limit

Upper Confidence Limit

Width of CI = 2 X (Multiple) X (Standard Error)

Poll questions

1. Given a confidence level, any value within the confidence interval of the mean could possibly be the population mean.Is this statement true or false?a) Trueb) False

2. To have more precise estimate, you would like the confidence interval to be (a) wider? or (b) narrower?a) Widerb) Narrower

14

Computing a confidence interval

15

CI = Point Estimate (Multiplier) (Standard Error)± ×

Discrete variables

sample meanContinuous variables

sample proportion

x

p

sn

(1 )p pn−

s=sample standard deviation

Common multiple:

• Suppose a random sample of 100 people are selected• Sample mean: • Sample standard deviation:

• 95% confidence interval calculation • α = 0.05

• standard error:

• Lower bound:

• Upper bound:

16

Mean example: hours spent on social media

1.960Multiple =

5.9 0.59 hours100

sn= =

17.4 hoursx =5.9 hourss =

17.4 1.96 0.59 16.2 hourssx Multiplen

− ⋅ = − ⋅ =

17.4 1.96 0.59 18.6 hourssx Multiplen

+ ⋅ = + ⋅ =

17

Mean example: confidence interval (cont.)

Point EstimateLower Confidence Limit

Upper Confidence Limit

17.4 1.96 0.5916.2 hours

sx Multiplen

− ⋅

= − ⋅=

17.4 1.96 0.5918.6 hours

sx Multiplen

+ ⋅

= + ⋅=

17.4 hoursx =

95% confidence interval = (16.2, 18.6) is interpreted as:we are 95% confident that the population mean is between 16.2 hours and 18.6 hours

Or, with 95% confident level, we conclude that the population mean is between 16.2 hours and 18.6 hours

Proportion example: estimate customer retention rate• A firm estimates its customer retention rate for the next year• A random sample of 400 customers is surveyed, out of whom 320

respondents claim that they would renew the service

• Calculation of the estimation• Sample proportion is

• The standard error is

• Multiplier for 95% confidence level is 1.96• The 95% confidence interval is

18

320 0.8400

p = =

(1 ) 0.8 0.2 0.02400

p pSEn− ×

= = =

1.96 (0.80 1.96 0.02) (0.76,0.84)p SE± × = ± × =

A Note on Std. Error of Sample Proportion

• Sample proportion is

• The standard error is

• What if the proportion is ?

• The standard error is

Multiplier for 95% confidence level is 1.96• The 95% confidence interval is

19

320 0.8400

p = =

(1 ) 0.8 0.2 0.02400

p pSEn− ×

= = =

𝑝𝑝 ± 1.96 × 𝑆𝑆𝑆𝑆 = (0.50 ± 1.96 × 0.025) = (0.451,0.549)

𝑝𝑝 =200400 = 0.5

𝑆𝑆𝑆𝑆 =𝑝𝑝(1 − 𝑝𝑝)

𝑛𝑛 =0.5 × 0.5

400 = 0.025

p = 0.5 maximizes the Standard Error!

CI of the mean in StatTools

• Dataset: FastFoodDataSample.xls (n=30)Step 1: select the column(s) of your data, click on “Data Set Manager”, click on “Yes” to confirm creating a data and then “OK” to create the data set

20

CI of the mean in StatTools (cont.)

Step 2: select the menu, Statistical Inference Confidence IntervalMean/Std. DeviationThen, select your variable

21

CI of the mean in StatTools (cont.)

Step 3: Interpret the results

22

Point estimate of the mean

Lower and upper bounds for your 95% confidence interval

23

Properties of confidence intervals

margin of error sMultiplen

= ×

Margin of Error

Point EstimateLower Confidence Limit

Upper Confidence Limit

Higher confidence level (for example, from 90% to 95%) higher multiple Larger margin of error wider confidence interval

Larger sample size smaller standard error Smaller margin of error narrower confidence interval

2

margin of errorsn Multiple

= ×

For a given margin of error, we can then compute the required sample size n:

CI of the proportion in StatTools

• Suppose we know that service time less than 60 is considered satisfactory

• Now, we have a new binary (discrete) column called “satisfactory”, which equals 1 if the value is less than 60

• From the data, we can then compute the proportion of data being satisfactory as well as its 95% confidence interval

24

CI of the proportion in StatTools (cont.)

Step 1: Data Set Manager create the dataStep 2: Statistical Inference Confidence Interval Proportion, select the variable name and the value to compute proportion on (in our example, the value of 1)

25

CI of the proportion in StatTools (cont.)

Step 3: interpreting the results

26

80% of the sample are satisfactory

With 95% confidence level, the proportion of satisfactory service cases is between 65.7% and 94.3%.

Part III: Hypothesis Testing

27

Topics to cover for hypothesis testing

• Steps of hypothesis testing • How to construct null and alternative hypotheses

• Two-sided test versus one-sided test

• P-value and results interpretation • Hypothesis testing of the mean

• Using StatTools

• Hypothesis testing of the proportion • Using StatTools

28

Steps of hypothesis testing

Form the null hypothesisH0: a claim about population parameterHA: the alternative hypothesis

Set a level of significance α, usually set at 0.05 Choose an appropriate test and perform the test using

sample data (this step takes time and experiences to master)

Use the evidence provided by the test to make a conclusion

• If p-value < α we reject H0 and conclude that we are in favor of the alternative hypothesis HA

• If p-value >= α we conclude that we do not have evidence to reject H0

29

• A hypothesis is a claim (assumption, initial statement) about a population parameter

• Example: on average people spend 18.5 hours on social media each week

• The Null Hypothesis, • States the assumption (numerical) to be tested,

for example:

• The Alternative Hypothesis,• Is the opposite of the null hypothesis, for example:

30

Step 1: construct hypothesis

0 0H : µ µ=

0H

AH

0H : A µ µ≠

Two-sided versus one-sided• Using the same example: time spent on social media per week• Suppose you are interested in testing whether an average

American adult spends roughly 20 hours per week on social media• The two-sided hypotheses look like this:

H0: μ=20HA: μ≠20

• Suppose you want to test if the average is less than 20 hours per week. Then this becomes a one-sided test and your hypotheses look like this:

H0: μ>=20HA: μ<20

• When is it appropriate to use a one-sided test?• Note that it is easier for you to claim significance when performing a one-

sided test than a two-sided test• Thus, only do one-sided test if you have a strong reason why you are

interested in the pattern of one direction (in our example, the mean is greater than or equal to 20)

• Otherwise, a two-sided test is more conservative and would be preferred

31

• The significance level α for a given hypothesis testing is a value for which p-values less than or equal to α is considered statistically significant

• Typical values are .01, .05, or .10

• Should be selected by you at the beginning of the test (typically set as 0.05)

• For very large sample sizes may consider α = 0.01 • For very small sample sizemay consider α = 0.10

32

Step 2: choose level of significance, α

α = 0.01 p<0.01 are considered statistically significant α = 0.05 p<0.05 are considered statistically significant α = 0.10 p<0.10 are considered statistically significant

Step 3: choose and perform a test

• Well, this is the step where most people find challenging.

• There are many different tests in statistics. Which one to use depends on the nature of your variable and the hypotheses.

• It does take lots of practice and experiences to be proficient with testing.

• For this class, we focus on two tests:• Testing the population mean against a null value• Testing the population proportion against a null value

• You will see other tests in future weeks…

33

Step 4: make a conclusion

• After performing the test, you just need to do three things:

1. Obtain the p-value from your test2. Recall the null and alternative hypotheses 3. Use the above two items and state your conclusion (better

to do so in plain English)• Again, the rule for p-value is:

• If p-value < α we reject H0 and conclude that we are in favor of the alternative hypothesis HA

• If p-value >= α we conclude that we do not have evidence to reject H0

• Let’s walk through the steps using one example for the mean and one for the proportion

34

Hypothesis testing example: the mean

Use the same fast food data set. Also, assume that the dataset has been set up in StatTools as we did in the confidence interval procedure.

H0: mean service time >= 60Ha: mean service time < 60 Significance level: 0.05

StatTools Statistical Inference Hypothesis testing Mean/Std. Deviationset H0 and Ha as shownun-select “Standard Deviation”

35

Hypothesis testing example: the mean

Results

36

p-value is less than 0.05

With 95% confidence level, we reject the nullhypothesis and conclude that the mean servicetime is statistically lower than 60.

H0: mean service time > = 60Ha: mean service time < 60 Significance level: 0.05

Hypothesis testing example: the proportion

• Let’s do testing on the proportion of satisfactory case (denoted as p).

• Suppose we have:H0: p = 0.5Ha: p ≠ 0.5 Significance level: 0.05

StatTools Statistical InferenceHypothesis testing Proportionset H0 and Ha as shown

37

Hypothesis testing example: the mean

Results

38

p-value is less than 0.05

With 95% confidence level, we reject the nullhypothesis and conclude that population proportion of satisfactory cases is significantly different from 50%.

H0: p = 0.5Ha: p ≠ 0.5 Significance level: 0.05

Confidence Interval (Slide 26) Hypothesis Testing (Slide 37)

39

Putting things together: the proportion example

65.7% 94.3%80%p =

Point

Estimate

Lower Confidence Limit

Upper Confidence Limit

With 95% confidence level, the proportion of satisfactory service cases is between 65.7% and 94.3%.

Note that this confidence interval does not contain 50%, the null value from the hypothesis testing approach on the right.

The p value is 0.001, less than 0.05. At significance level of 0.05, we reject the nullhypothesis and conclude that population proportion of satisfactory cases is significantly different from 50%.

Note that this is consistent from what we observe from the confidence interval, which does not include 50%.

H0: p = 0.5Ha: p ≠ 0.5 Significance level: 0.05