Download - Hypothesis Testing: Significance
![Page 1: Hypothesis Testing: Significance](https://reader036.vdocument.in/reader036/viewer/2022062302/568164a7550346895dd69e39/html5/thumbnails/1.jpg)
Statistics: Unlocking the Power of Data Lock5
Hypothesis Testing: Significance
STAT 101
Dr. Kari Lock Morgan
9/27/12
SECTION 4.3• Significance level• Statistical conclusions• Type I and II errors
![Page 2: Hypothesis Testing: Significance](https://reader036.vdocument.in/reader036/viewer/2022062302/568164a7550346895dd69e39/html5/thumbnails/2.jpg)
Statistics: Unlocking the Power of Data Lock5
Office HoursMy office hours next week will be Wednesday
1 – 3pm, NOT Monday
(and Thurs 1 – 2:30 as always)
![Page 3: Hypothesis Testing: Significance](https://reader036.vdocument.in/reader036/viewer/2022062302/568164a7550346895dd69e39/html5/thumbnails/3.jpg)
Statistics: Unlocking the Power of Data Lock5
Randomization Distributionsp-values can be calculated by randomization
distributions:
simulate samples, assuming H0 is true calculate the statistic of interest for each sample find the p-value as the proportion of simulated
statistics as extreme as the observed statistic
Let’s do a randomization distribution for a randomized experiment…
![Page 4: Hypothesis Testing: Significance](https://reader036.vdocument.in/reader036/viewer/2022062302/568164a7550346895dd69e39/html5/thumbnails/4.jpg)
Statistics: Unlocking the Power of Data Lock5
• In a randomized experiment on treating cocaine addiction, 48 people were randomly assigned to take either Desipramine (a new drug), or Lithium (an existing drug), and then followed to see who relapsed
• Question of interest: Is Desipramine better than Lithium at treating cocaine addiction?
Cocaine Addiction
![Page 5: Hypothesis Testing: Significance](https://reader036.vdocument.in/reader036/viewer/2022062302/568164a7550346895dd69e39/html5/thumbnails/5.jpg)
Statistics: Unlocking the Power of Data Lock5
•What are the null and alternative hypotheses?
•What are the possible conclusions?
Cocaine Addiction
pD, pL: proportion of cocaine addicts who relapse after taking Desipramine or Lithium, respectively
H0: pD = pLHa: pD < pL
Reject H0; Desipramine is better than LithiumDo not reject H0: We cannot determine from these data whether Desipramine is better than Lithium
![Page 6: Hypothesis Testing: Significance](https://reader036.vdocument.in/reader036/viewer/2022062302/568164a7550346895dd69e39/html5/thumbnails/6.jpg)
Statistics: Unlocking the Power of Data Lock5
R R R R R R
R R R R R R
R R R R R R
R R R R R R
R R R R R R
R R R R R R
R R R R R R
R R R R R R
R R R R
R R R R R R
R R R R R R
R R R R R R
R R R R
R R R R R R
R R R R R R
R R R R R R
Desipramine Lithium
1. Randomly assign units to treatment groups
![Page 7: Hypothesis Testing: Significance](https://reader036.vdocument.in/reader036/viewer/2022062302/568164a7550346895dd69e39/html5/thumbnails/7.jpg)
Statistics: Unlocking the Power of Data Lock5
R R R R
R R R R R R
R R R R R R
N N N N N N
RRR R R R
R R R R N N
N N N N N N
RR
N N N N N N
R = RelapseN = No Relapse
R R R R
R R R R R R
R R R R R R
N N N N N N
RRR R R R
R R R R RR
R R N N N N
RR
N N N N N N
2. Conduct experiment
3. Observe relapse counts in each group
LithiumDesipramine
10 relapse, 14 no relapse 18 relapse, 6 no relapse
1. Randomly assign units to treatment groups
10 1824
ˆ ˆ
24.333
D Lp p
![Page 8: Hypothesis Testing: Significance](https://reader036.vdocument.in/reader036/viewer/2022062302/568164a7550346895dd69e39/html5/thumbnails/8.jpg)
Statistics: Unlocking the Power of Data Lock5
To see if a statistic provides evidence against H0, we need to
see what kind of sample statistics we would observe,
just by random chance, if H0 were true
Measuring Evidence against H0
![Page 9: Hypothesis Testing: Significance](https://reader036.vdocument.in/reader036/viewer/2022062302/568164a7550346895dd69e39/html5/thumbnails/9.jpg)
Statistics: Unlocking the Power of Data Lock5
• “by random chance” means by the random assignment to the two treatment groups
• “if H0 were true” means if the two drugs were equally effective at preventing relapses (equivalently: whether a person relapses or not does not depend on which drug is taken)
• Simulate what would happen just by random chance, if H0 were true…
Cocaine Addiction
![Page 10: Hypothesis Testing: Significance](https://reader036.vdocument.in/reader036/viewer/2022062302/568164a7550346895dd69e39/html5/thumbnails/10.jpg)
Statistics: Unlocking the Power of Data Lock5
R R R R
R R R R R R
R R R R R R
N N N N N N
RRR R R R
R R R R N N
N N N N N N
RR
N N N N N N
10 relapse, 14 no relapse 18 relapse, 6 no relapse
![Page 11: Hypothesis Testing: Significance](https://reader036.vdocument.in/reader036/viewer/2022062302/568164a7550346895dd69e39/html5/thumbnails/11.jpg)
Statistics: Unlocking the Power of Data Lock5
R R R R R R
R R R R N N
N N N N N N
N N N N N N
R R R R R R
R R R R R R
R R R R R R
N N N N N N
R N R N
R R R R R R
R N R R R N
R N N N R R
N N N R
N R R N N N
N R N R R N
R N R R R R
Simulate another randomization
Desipramine Lithium
16 relapse, 8 no relapse 12 relapse, 12 no relapse
ˆ ˆ16 1224 240.167
LDp p
![Page 12: Hypothesis Testing: Significance](https://reader036.vdocument.in/reader036/viewer/2022062302/568164a7550346895dd69e39/html5/thumbnails/12.jpg)
Statistics: Unlocking the Power of Data Lock5
R R R R
R R R R R R
R R R R R R
N N N N N N
RRR R R R
R N R R N N
R R N R N R
RR
R N R N R R
Simulate another randomization
Desipramine Lithium
17 relapse, 7 no relapse 11 relapse, 13 no relapse
ˆ ˆ17 1124 240.250
D Lp p
![Page 13: Hypothesis Testing: Significance](https://reader036.vdocument.in/reader036/viewer/2022062302/568164a7550346895dd69e39/html5/thumbnails/13.jpg)
Statistics: Unlocking the Power of Data Lock5
Simulate Your Own SampleIn the experiment, 28 people relapsed and 20 people
did not relapse. Create cards or slips of paper with 28 “R” values and 20 “N” values.
Pool these response values together, and randomly divide them into two groups (representing Desipramine and Lithium)
Calculate your difference in proportions
Plot your statistic on the class dotplot
To create an entire randomization distribution, we simulate this process many more times with technology: StatKey
![Page 14: Hypothesis Testing: Significance](https://reader036.vdocument.in/reader036/viewer/2022062302/568164a7550346895dd69e39/html5/thumbnails/14.jpg)
Statistics: Unlocking the Power of Data Lock5
www.lock5stat.com/statkey
p-value
![Page 15: Hypothesis Testing: Significance](https://reader036.vdocument.in/reader036/viewer/2022062302/568164a7550346895dd69e39/html5/thumbnails/15.jpg)
Statistics: Unlocking the Power of Data Lock5
Formal DecisionsIf the p-value is small:
REJECT H0
the sample would be extreme if H0 were true the results are statistically significant we have evidence for Ha
If the p-value is not small: DO NOT REJECT H0
the sample would not be too extreme if H0 were true the results are not statistically significant the test is inconclusive; either H0 or Ha may be true
![Page 16: Hypothesis Testing: Significance](https://reader036.vdocument.in/reader036/viewer/2022062302/568164a7550346895dd69e39/html5/thumbnails/16.jpg)
Statistics: Unlocking the Power of Data Lock5
A formal hypothesis test has only two possible conclusions:
1. The p-value is small: reject the null hypothesis in favor of the alternative
2. The p-value is not small: do not reject the null hypothesis
Formal Decisions
How small?
![Page 17: Hypothesis Testing: Significance](https://reader036.vdocument.in/reader036/viewer/2022062302/568164a7550346895dd69e39/html5/thumbnails/17.jpg)
Statistics: Unlocking the Power of Data Lock5
Significance Level
The significance level, , is the threshold below which the p-value is deemed small enough to reject the null hypothesis
p-value < Reject H0
p-value > Do not Reject H0
![Page 18: Hypothesis Testing: Significance](https://reader036.vdocument.in/reader036/viewer/2022062302/568164a7550346895dd69e39/html5/thumbnails/18.jpg)
Statistics: Unlocking the Power of Data Lock5
Significance LevelIf the p-value is less than , the results are
statistically significant, and we reject the null hypothesis in favor of the alternative
If the p-value is not less than , the results are not statistically significant, and our test is inconclusive
Often = 0.05 by default, unless otherwise specified
![Page 19: Hypothesis Testing: Significance](https://reader036.vdocument.in/reader036/viewer/2022062302/568164a7550346895dd69e39/html5/thumbnails/19.jpg)
Statistics: Unlocking the Power of Data Lock5
• Resveratrol, an ingredient in red wine and grapes, has been shown to promote weight loss in rodents, and has recently been investigated in primates (specifically, the Grey Mouse Lemur).
• A sample of lemurs had various measurements taken before and after receiving resveratrol supplementation for 4 weeks
Red Wine and Weight Loss
BioMed Central (2010, June 22). “Lemurs lose weight with ‘life-extending’ supplement resveratrol. Science Daily.
![Page 20: Hypothesis Testing: Significance](https://reader036.vdocument.in/reader036/viewer/2022062302/568164a7550346895dd69e39/html5/thumbnails/20.jpg)
Statistics: Unlocking the Power of Data Lock5
Red Wine and Weight Loss
In the test to see if the mean resting metabolic rate is higher after treatment, the p-value is 0.013.
Using = 0.05, is this difference statistically significant? (should we reject H0: no difference?)
a) Yesb) No The p-value is lower than
= 0.05, so the results are statistically significant and we reject H0.
![Page 21: Hypothesis Testing: Significance](https://reader036.vdocument.in/reader036/viewer/2022062302/568164a7550346895dd69e39/html5/thumbnails/21.jpg)
Statistics: Unlocking the Power of Data Lock5
Red Wine and Weight Loss
In the test to see if the mean body mass is lower after treatment, the p-value is 0.007.
Using = 0.05, is this difference statistically significant? (should we reject H0: no difference?)
a) Yesb) No The p-value is lower than
= 0.05, so the results are statistically significant and we reject H0.
![Page 22: Hypothesis Testing: Significance](https://reader036.vdocument.in/reader036/viewer/2022062302/568164a7550346895dd69e39/html5/thumbnails/22.jpg)
Statistics: Unlocking the Power of Data Lock5
Red Wine and Weight Loss
In the test to see if locomotor activity changes after treatment, the p-value is 0.980.
Using = 0.05, is this difference statistically significant? (should we reject H0: no difference?)
a) Yesb) No
The p-value is not lower than = 0.05, so the results are not statistically significant and we do not reject H0.
![Page 23: Hypothesis Testing: Significance](https://reader036.vdocument.in/reader036/viewer/2022062302/568164a7550346895dd69e39/html5/thumbnails/23.jpg)
Statistics: Unlocking the Power of Data Lock5
Red Wine and Weight Loss
In the test to see if mean food intake changes after treatment, the p-value is 0.035.
Using = 0.05, is this difference statistically significant? (should we reject H0: no difference?)
a) Yesb) No
The p-value is lower than = 0.05, so the results are statistically significant and we reject H0.
![Page 24: Hypothesis Testing: Significance](https://reader036.vdocument.in/reader036/viewer/2022062302/568164a7550346895dd69e39/html5/thumbnails/24.jpg)
Statistics: Unlocking the Power of Data Lock5
H0 : X is an elephantHa : X is not an elephant
Would you conclude, if you get the following data?
• X walks on two legs
• X has four legs
Elephant Example
Reject H0; evidence that X is not an elephant
Although we can never be certain!
Do not reject H0; we do not have sufficient evidence to determine whether X is an elephant
![Page 25: Hypothesis Testing: Significance](https://reader036.vdocument.in/reader036/viewer/2022062302/568164a7550346895dd69e39/html5/thumbnails/25.jpg)
Statistics: Unlocking the Power of Data Lock5
“For the logical fallacy of believing that a hypothesis has been proved to be true, merely because it is not contradicted by the available facts, has no more right to insinuate itself in statistical than in other kinds of scientific reasoning…”
-Sir R. A. Fisher
Never Accept H0
•“Do not reject H0” is not the same as “accept H0”!
• Lack of evidence against H0 is NOT the same as evidence for H0!
![Page 26: Hypothesis Testing: Significance](https://reader036.vdocument.in/reader036/viewer/2022062302/568164a7550346895dd69e39/html5/thumbnails/26.jpg)
Statistics: Unlocking the Power of Data Lock5
Statistical Conclusions
In a hypothesis test of
H0: = 10 vs Ha: < 10
the p-value is 0.002. With α = 0.05, we conclude:
a) Reject H0
b) Do not reject H0
c) Reject Ha
d) Do not reject Ha
The p-value of 0.002 is less than α = 0.05, so we reject H0
![Page 27: Hypothesis Testing: Significance](https://reader036.vdocument.in/reader036/viewer/2022062302/568164a7550346895dd69e39/html5/thumbnails/27.jpg)
Statistics: Unlocking the Power of Data Lock5
Statistical Conclusions
In a hypothesis test of
H0: = 10 vs Ha: < 10
the p-value is 0.002. With α = 0.01, we conclude:
a) There is evidence that = 10 b) There is evidence that < 10
c) We have insufficient evidence to conclude anything
![Page 28: Hypothesis Testing: Significance](https://reader036.vdocument.in/reader036/viewer/2022062302/568164a7550346895dd69e39/html5/thumbnails/28.jpg)
Statistics: Unlocking the Power of Data Lock5
Statistical Conclusions
In a hypothesis test of
H0: = 10 vs Ha: < 10
the p-value is 0.21. With α = 0.01, we conclude:
a) Reject H0
b) Do not reject H0
c) Reject Ha
d) Do not reject Ha
The p-value of 0.21 is not less than α = 0.01, so we do not reject H0
![Page 29: Hypothesis Testing: Significance](https://reader036.vdocument.in/reader036/viewer/2022062302/568164a7550346895dd69e39/html5/thumbnails/29.jpg)
Statistics: Unlocking the Power of Data Lock5
Statistical Conclusions
In a hypothesis test of
H0: = 10 vs Ha: < 10
the p-value is 0.21. With α = 0.01, we conclude:
a) There is evidence that = 10 b) There is evidence that < 10
c) We have insufficient evidence to conclude anything
![Page 30: Hypothesis Testing: Significance](https://reader036.vdocument.in/reader036/viewer/2022062302/568164a7550346895dd69e39/html5/thumbnails/30.jpg)
Statistics: Unlocking the Power of Data Lock5
Informal strength of evidence against H0:
Formal decision of hypothesis test, based on = 0.05 :
statistically significant
not statistically significant
Statistical Conclusions
![Page 31: Hypothesis Testing: Significance](https://reader036.vdocument.in/reader036/viewer/2022062302/568164a7550346895dd69e39/html5/thumbnails/31.jpg)
Statistics: Unlocking the Power of Data Lock5
Multiple Sclerosis and Sunlight• It is believed that sunlight offers some protection
against multiple sclerosis, but the reason is unknown
• Researchers randomly assigned mice to one of:• Control (nothing)• Vitamin D Supplements• UV Light
• All mice were injected with proteins known to induce a mouse form of MS, and they observed which mice got MS
Seppa, Nathan. “Sunlight may cut MS risk by itself”, Science News, April 24, 2010 pg 9, reporting on a study appearing March 22, 2010 in the Proceedings of the National Academy of Science.
![Page 32: Hypothesis Testing: Significance](https://reader036.vdocument.in/reader036/viewer/2022062302/568164a7550346895dd69e39/html5/thumbnails/32.jpg)
Statistics: Unlocking the Power of Data Lock5
Multiple Sclerosis and SunlightFor each situation below, write down
Null and alternative hypotheses Informal description of the strength of evidence against H0
Formal decision about H0, using α = 0.05 Conclusion in the context of the question
In testing whether UV light provides protection against MS (UV light vs control group), the p-value is 0.002.
In testing whether Vitamin D provides protection against MS (Vitamin D vs control group), the p-value is 0.47.
![Page 33: Hypothesis Testing: Significance](https://reader036.vdocument.in/reader036/viewer/2022062302/568164a7550346895dd69e39/html5/thumbnails/33.jpg)
Statistics: Unlocking the Power of Data Lock5
Multiple Sclerosis and SunlightIn testing whether UV light provides
protection against MS (UV light vs control group), the p-value is 0.002.
• H0: pUV – pC = 0 Ha: pUV – pC < 0• We have strong evidence against H0
• Reject H0
• We have strong evidence that UV light provides protection against MS, at least in mice.
![Page 34: Hypothesis Testing: Significance](https://reader036.vdocument.in/reader036/viewer/2022062302/568164a7550346895dd69e39/html5/thumbnails/34.jpg)
Statistics: Unlocking the Power of Data Lock5
Multiple Sclerosis and SunlightIn testing whether Vitamin D provides
protection against MS (Vitamin D vs control group), the p-value is 0.47.
• H0: pD – pC = 0 Ha: pD – pC < 0• We have little evidence against H0
• Do not reject H0
• We cannot conclude anything about Vitamin D and MS.
![Page 35: Hypothesis Testing: Significance](https://reader036.vdocument.in/reader036/viewer/2022062302/568164a7550346895dd69e39/html5/thumbnails/35.jpg)
Statistics: Unlocking the Power of Data Lock5
There are four possibilities:Errors
Reject H0 Do not reject H0
H0 true
H0 false TYPE I ERROR
TYPE II ERRORTrut
h
Decision
• A Type I Error is rejecting a true null
• A Type II Error is not rejecting a false null
![Page 36: Hypothesis Testing: Significance](https://reader036.vdocument.in/reader036/viewer/2022062302/568164a7550346895dd69e39/html5/thumbnails/36.jpg)
Statistics: Unlocking the Power of Data Lock5
• In the test to see if resveratrol is associated with food intake, the p-value is 0.035.
o If resveratrol is not associated with food intake, a Type I Error would have been made
• In the test to see if resveratrol is associated with locomotor activity, the p-value is 0.980.
o If resveratrol is associated with locomotor activity, a Type II Error would have been made
Red Wine and Weight Loss
![Page 37: Hypothesis Testing: Significance](https://reader036.vdocument.in/reader036/viewer/2022062302/568164a7550346895dd69e39/html5/thumbnails/37.jpg)
Statistics: Unlocking the Power of Data Lock5
A person is innocent until proven guilty.
Evidence must be beyond the shadow of a doubt.
Types of mistakes in a verdict?
Convict an innocent
Release a guilty
Ho Ha
Type I error
Type II error
Analogy to Law
p-value from data
![Page 38: Hypothesis Testing: Significance](https://reader036.vdocument.in/reader036/viewer/2022062302/568164a7550346895dd69e39/html5/thumbnails/38.jpg)
Statistics: Unlocking the Power of Data Lock5
• The probability of making a Type I error (rejecting a true null) is the significance level, α
• α should be chosen depending how bad it is to make a Type I error
Probability of Type I Error
![Page 39: Hypothesis Testing: Significance](https://reader036.vdocument.in/reader036/viewer/2022062302/568164a7550346895dd69e39/html5/thumbnails/39.jpg)
Statistics: Unlocking the Power of Data Lock5
If the null hypothesis is true:• 5% of statistics will be in the most extreme 5% • 5% of statistics will give p-values less than 0.05• 5% of statistics will lead to rejecting H0 at α = 0.05• If α = 0.05, there is a 5% chance of a Type I error
Distribution of statistics, assuming H0 true:
Probability of Type I Error
![Page 40: Hypothesis Testing: Significance](https://reader036.vdocument.in/reader036/viewer/2022062302/568164a7550346895dd69e39/html5/thumbnails/40.jpg)
Statistics: Unlocking the Power of Data Lock5
If the null hypothesis is true:• 1% of statistics will be in the most extreme 1% • 1% of statistics will give p-values less than 0.01• 1% of statistics will lead to rejecting H0 at α = 0.01• If α = 0.01, there is a 1% chance of a Type I error
Distribution of statistics, assuming H0 true:
Probability of Type I Error
![Page 41: Hypothesis Testing: Significance](https://reader036.vdocument.in/reader036/viewer/2022062302/568164a7550346895dd69e39/html5/thumbnails/41.jpg)
Statistics: Unlocking the Power of Data Lock5
Probability of Type II ErrorThe probability of making a Type II Error (not
rejecting a false null) depends on
Effect size (how far the truth is from the null)
Sample size
Variability
Significance level
![Page 42: Hypothesis Testing: Significance](https://reader036.vdocument.in/reader036/viewer/2022062302/568164a7550346895dd69e39/html5/thumbnails/42.jpg)
Statistics: Unlocking the Power of Data Lock5
Choosing αBy default, usually α = 0.05
If a Type I error (rejecting a true null) is much worse than a Type II error, we may choose a smaller α, like α = 0.01
If a Type II error (not rejecting a false null) is much worse than a Type I error, we may choose a larger α, like α = 0.10
![Page 43: Hypothesis Testing: Significance](https://reader036.vdocument.in/reader036/viewer/2022062302/568164a7550346895dd69e39/html5/thumbnails/43.jpg)
Statistics: Unlocking the Power of Data Lock5
Come up with a hypothesis testing situation in which you may want to…
• Use a smaller significance level, like = 0.01
• Use a larger significance level, like = 0.10
Significance Level
![Page 44: Hypothesis Testing: Significance](https://reader036.vdocument.in/reader036/viewer/2022062302/568164a7550346895dd69e39/html5/thumbnails/44.jpg)
Statistics: Unlocking the Power of Data Lock5
• Results are statistically significant if the p-value is less than the significance level, α• In making formal decisions, reject H0 if the p-value is less than α, otherwise do not reject H0
• Not rejecting H0 is NOT the same as accepting H0
• There are two types of errors: rejecting a true null (Type I) and not rejecting a false null (Type II)
Summary
![Page 45: Hypothesis Testing: Significance](https://reader036.vdocument.in/reader036/viewer/2022062302/568164a7550346895dd69e39/html5/thumbnails/45.jpg)
Statistics: Unlocking the Power of Data Lock5
To DoProject 1 proposal due TODAY at 5pm
Read Section 4.3
Do Homework 4 (due Thursday, 10/4)