synthesis and review 3/26/12
DESCRIPTION
Synthesis and Review 3/26/12. Multiple Comparisons Review of Concepts Review of Methods - Prezi. Essential Synthesis 3. Professor Kari Lock Morgan Duke University. To Do. Study and prepare for Exam 2 (Wednesday and Thursday). Exam Policies. - PowerPoint PPT PresentationTRANSCRIPT
Synthesis and Review3/26/12
• Multiple Comparisons• Review of Concepts• Review of Methods - Prezi
Essential Synthesis 3 Professor Kari Lock MorganDuke University
• Study and prepare for Exam 2 (Wednesday and Thursday)
To Do
• An exam absence is only excused if a short term illness form is submitted before the exam
• In this case, your final exam grade will be substituted
• Keep in mind that you will be responsible for a LOT more material on the final exam, and it is already worth 25% of your grade
• You can ONLY take the lab exam during your designated section. Set two alarms if needed.
Exam Policies
• Any cheating (either on the in-class exam or the lab exam) will result in an automatic 0, and will be treated as a serious case of academic misconduct
• This includes, but is not limited to,• Looking at someone else’s exam or computer screen• For the in-class exam, using pages of notes prepared
by someone else• Communicating (in any form) with anyone besides
myself or your TAs during the exam• Communicating (in any way) with any classmates
about the lab exam, or sharing any code or materials related to the lab exam, before 4pm on Thursday, 3/29
Exam Policies
Analytic Approaches to BasketballMike Zarren (Boston Celtics)
Tuesday, 3/27, 5pm in 2231 French Family Science
Michael Zarren is the Boston Celtics’ Assistant General Manager and Associate Team Counsel. Mike is widely recognized as one of the leaders in the field of advanced statistical analysis of basketball players and teams, and is an important part of the team’s strategic planning and player personnel evaluation processes. Mike is also the team’s salary cap expert and lead in-house counsel, and is responsible for the development of new technologies for team use, including the team’s statistical database and video archive/delivery system. Read more here: http://goo.gl/l4P3I.
Talk
You have LOTS of opportunities for help!
• Monday, 3 – 4 pm (Prof Morgan)• Monday, 4 – 6 pm (Christine)• Tuesday, 3 – 6 pm (Prof Morgan)• Tuesday, 6 – 8 pm (Yue)
Office Hours before Exam
• RStudio no longer supports importing data from a google doc
Importing from a Google Doc
Extrasensory Perception•Is there such a thing as ESP?
• Let’s find out by conducting our own experiments!
Extrasensory Perception
•Get into pairs.•“Randomly” choose A, B, C, or D, and write it down•Try to transmit this information to your partner, without communicating the letter in any way that can be perceived by any of the five senses! •Partner: guess the letter. •Repeat this 10 times each, and keep track of the number of correct guesses.•Once you have n = 20, come to the board and plot your sample proportion•Test whether your experiment provides evidence of ESP
Extrasensory Perception
Did your experiment provide evidence of extrasensory perception, using = 0.05?
(a) Yes
(b) No
Test for a ProportionWhich of the following ways are appropriate to test whether your sample proportion is significantly different from p = ¼?a) Randomization Test (only)b) Normal distribution (only)c) t-distribution (only)d) Either (a) or (b)e) Either (a), (b), or (c)
Randomization DistributionIF there is no such thing as ESP, then you all just created a randomization distribution.
0.25(1 0.
20
25)0.097SE
Extrasensory Perception
If there is no such thing as ESP, what percentage of experiments on ESP will get results that are significant, using = 0.05?
(a) None(b) All of them
(c) 95%(d) 5%
Multiple Comparisons
• Consider a topic that is being investigated by research teams all over the world
5% of teams are going to find something significant, even if the null is true
Multiple Comparisons
•Consider a research team/company doing many hypothesis tests
Þ 5% of tests are going to be significant, even if the nulls are all true
Multiple Comparisons
• Consider an experiment that randomizes units to treatment groups, and then looks at many response variables
Þ 5% of variables are going to be significantly different between the groups, just by random chance
Pairwise Comparisons
• Consider a study with many different treatment groups, and so many possible pairwise comparisons
Þ 5% of comparisons are going to be significantly different, even if no differences actually exist
(This is the main reason for only testing pairwise comparisons if the overall ANOVA is found to be significant)
Publication Bias
• publication bias: usually, only the significant results get published
• The one study that turns out significant gets published, and no one knows about all the insignificant results
• This is a serious problem
• The most important thing is to simply be aware of this issue, and not to trust claims that are obviously one of many tests (unless they specifically mention an adjustment for multiple testing)
Multiple Comparisons
REVIEW
Was the sample randomly selected?
Possible to generalize to
the population
Yes
Should not generalize to
the population
No
Was the explanatory variable randomly
assigned?
Possible to make
conclusions about causality
Yes
Can not make conclusions
about causality
No
Data Collection
Confidence Interval
• A confidence interval for a parameter is an interval computed from sample data by a method that will capture the parameter for a specified proportion of all samples
• A 95% confidence interval will contain the true parameter for 95% of all samples
• How unusual would it be to get results as extreme (or more extreme) than those observed, if the null hypothesis is true?
• If it would be very unusual, then the null hypothesis is probably not true!
• If it would not be very unusual, then there is not evidence against the null hypothesis
Hypothesis Testing
• The p-value is the probability of getting a statistic as extreme (or more extreme) as that observed, just by random chance, if the null hypothesis is true
• The p-value measures evidence against the null hypothesis
p-value
Hypothesis Testing
1.State Hypotheses
2.Calculate a test statistic, based on your sample data
3.Create a distribution of this test statistic, as it would be observed if the null hypothesis were true
4.Use this distribution to measure how extreme your test statistic is
Distribution of the Sample Statistic
1.Sampling distribution: distribution of the statistic based on many samples from the population
2.Bootstrap Distribution: distribution of the statistic based on many samples with replacement from the original sample
3.Randomization Distribution: distribution of the statistic assuming the null hypothesis is true
4.Normal, t,2, F: Theoretical distributions used to approximate the distribution of the statistic
Sample Size Conditions
• For large sample sizes, either simulation methods or theoretical methods work
• If sample sizes are too small, only simulation methods can be used
• For confidence intervals, you find the desired percentage in the middle of the distribution, then find the corresponding value on the x-axis
• For p-values, you find the value of the observed statistic on the x-axis, then find the area in the tail(s) of the distribution
Using Distributions
Confidence IntervalsBest Guess at Sampling Distribution
Statistic
2 3 4 5 6 7 8
Best Guess at Sampling Distribution
Statistic
2 3 4 5 6 7 8
Observed Statistic
Best Guess at Sampling Distribution
Statistic
2 3 4 5 6 7 8
Observed Statistic
P%
Best Guess at Sampling Distribution
Statistic
2 3 4 5 6 7 8
Observed Statistic
P%P%P%
Upper BoundUpper Bound
Lower Bound
Confidence IntervalsN(0,1)
-3 -2 -1 0 1 2 3
N(0,1)
-3 -2 -1 0 1 2 3
P%
N(0,1)
-3 -2 -1 0 1 2 3
P% z*
*sample statistic z SE Return to original scale with
Hypothesis TestingDistribution of Statistic Assuming Null
Statistic
-3 -2 -1 0 1 2 3
Observed Statistic
Distribution of Statistic Assuming Null
Statistic
-3 -2 -1 0 1 2 3
Distribution of Statistic Assuming Null
Statistic
-3 -2 -1 0 1 2 3
Observed Statistic
p-value
General Formulas• When performing inference for a single
parameter (or difference in two parameters), the following formulas are used:
sample statistic null value
SEz
*sample statistic z SE
Standard Error
• The standard error is the standard deviation of the sample statistic
• The formula for the standard error depends on the type of statistic (which depends on the type of variable(s) being analyzed)
Multiple Categories• These formulas do not work for categorical
variables with more than two categories, because there are multiple parameters
• For one or two categorical variables with multiple categories, use 2 tests
• For testing for a difference in means across multiple groups, use ANOVA
