topics ahead - zhang.ist.psu.eduzhang.ist.psu.edu/teaching/505/pdf/week10.pdf · one slide on...
TRANSCRIPT
![Page 1: Topics Ahead - zhang.ist.psu.eduzhang.ist.psu.edu/teaching/505/pdf/Week10.pdf · One slide on research question Literature review: structured; holes you will fill in Your research](https://reader033.vdocument.in/reader033/viewer/2022052021/603523f33f264507ab091626/html5/thumbnails/1.jpg)
Topics Ahead Week 10-11: Experimental design; Running
experiment
Week 12: Survey Design; ANOVA
Week 13: Correlation and Regression; Non Parametric Statistics
Week 14: Computational Methods; Simulation; Design Research
Week 15: Research Report Writing
Week 16: Final Presentation
![Page 2: Topics Ahead - zhang.ist.psu.eduzhang.ist.psu.edu/teaching/505/pdf/Week10.pdf · One slide on research question Literature review: structured; holes you will fill in Your research](https://reader033.vdocument.in/reader033/viewer/2022052021/603523f33f264507ab091626/html5/thumbnails/2.jpg)
Presentations Week 13
One slide on research question
Literature review: structured; holes you will fill in
Your research design General design and key components
Data: type, sources, possible analysis methods
Week 16:
Everything
Detailed design Data collection method
Data analysis method
Instruments: survey, experimental protocols, etc.
![Page 3: Topics Ahead - zhang.ist.psu.eduzhang.ist.psu.edu/teaching/505/pdf/Week10.pdf · One slide on research question Literature review: structured; holes you will fill in Your research](https://reader033.vdocument.in/reader033/viewer/2022052021/603523f33f264507ab091626/html5/thumbnails/3.jpg)
Experimental Design
![Page 4: Topics Ahead - zhang.ist.psu.eduzhang.ist.psu.edu/teaching/505/pdf/Week10.pdf · One slide on research question Literature review: structured; holes you will fill in Your research](https://reader033.vdocument.in/reader033/viewer/2022052021/603523f33f264507ab091626/html5/thumbnails/4.jpg)
Previous Example
New Tradition
![Page 5: Topics Ahead - zhang.ist.psu.eduzhang.ist.psu.edu/teaching/505/pdf/Week10.pdf · One slide on research question Literature review: structured; holes you will fill in Your research](https://reader033.vdocument.in/reader033/viewer/2022052021/603523f33f264507ab091626/html5/thumbnails/5.jpg)
Goal?New Tradition
=?
![Page 6: Topics Ahead - zhang.ist.psu.eduzhang.ist.psu.edu/teaching/505/pdf/Week10.pdf · One slide on research question Literature review: structured; holes you will fill in Your research](https://reader033.vdocument.in/reader033/viewer/2022052021/603523f33f264507ab091626/html5/thumbnails/6.jpg)
Challenges
Internal validity
How to guarantee what you have observed is true?
External validity
How to guarantee what you have observed can be applied in other situations?
Construct validity
To what extend does your study measure the construct of interest?
![Page 7: Topics Ahead - zhang.ist.psu.eduzhang.ist.psu.edu/teaching/505/pdf/Week10.pdf · One slide on research question Literature review: structured; holes you will fill in Your research](https://reader033.vdocument.in/reader033/viewer/2022052021/603523f33f264507ab091626/html5/thumbnails/7.jpg)
Threats to Internal Validity: Biased Samples
Group selection Self-identified
Assigned by researchers
Assigned arbitrarily
Are the groups equal?
Choosing groups based on their differences results in having groups that are different Difference should be less than 5%.
![Page 8: Topics Ahead - zhang.ist.psu.eduzhang.ist.psu.edu/teaching/505/pdf/Week10.pdf · One slide on research question Literature review: structured; holes you will fill in Your research](https://reader033.vdocument.in/reader033/viewer/2022052021/603523f33f264507ab091626/html5/thumbnails/8.jpg)
How About Matching?
Almost impossible to perfectly match individual participants
No identical participants
Too many relevant factors/variables Some we know, but some we don’t.
Matching on pretest scores Selection by maturation interactions: participants
growing in different ways
How can you make sure equal scores mean subjects are equal?
![Page 9: Topics Ahead - zhang.ist.psu.eduzhang.ist.psu.edu/teaching/505/pdf/Week10.pdf · One slide on research question Literature review: structured; holes you will fill in Your research](https://reader033.vdocument.in/reader033/viewer/2022052021/603523f33f264507ab091626/html5/thumbnails/9.jpg)
So …
Trying to keep everything except the treatment constant is very difficult, if not impossible
Selection is a big problem
Internal validity is threatened.
Only option is to rule out extraneous variables.
Use random assignment
Identify extraneous variables and then try to rule
them out
![Page 10: Topics Ahead - zhang.ist.psu.eduzhang.ist.psu.edu/teaching/505/pdf/Week10.pdf · One slide on research question Literature review: structured; holes you will fill in Your research](https://reader033.vdocument.in/reader033/viewer/2022052021/603523f33f264507ab091626/html5/thumbnails/10.jpg)
Dilemma
Attempts to rule out threats to internal validity may hurt external validity
Not studying a heterogeneous group of participants
Not studying participants in a naturalistic setting
![Page 11: Topics Ahead - zhang.ist.psu.eduzhang.ist.psu.edu/teaching/505/pdf/Week10.pdf · One slide on research question Literature review: structured; holes you will fill in Your research](https://reader033.vdocument.in/reader033/viewer/2022052021/603523f33f264507ab091626/html5/thumbnails/11.jpg)
You Need To
Be cautious about accepting cause-effect statements
If the study is not an experiment, the study’s internal validity is probably threatened by at least one of 8 threats to validity argued by Campbell and Stanley (1963)
![Page 12: Topics Ahead - zhang.ist.psu.eduzhang.ist.psu.edu/teaching/505/pdf/Week10.pdf · One slide on research question Literature review: structured; holes you will fill in Your research](https://reader033.vdocument.in/reader033/viewer/2022052021/603523f33f264507ab091626/html5/thumbnails/12.jpg)
![Page 13: Topics Ahead - zhang.ist.psu.eduzhang.ist.psu.edu/teaching/505/pdf/Week10.pdf · One slide on research question Literature review: structured; holes you will fill in Your research](https://reader033.vdocument.in/reader033/viewer/2022052021/603523f33f264507ab091626/html5/thumbnails/13.jpg)
Some Threats To Internal Validity
History: events that occur between treatments
Testing: changes resulting from the practice and experience gained in the test
Instrumentation: the change of the way subjects are measured
Regression effects: selection of subjects based on extreme scores
Mortality: the loss of subjects
Selection: groups are selected differently
Maturation: treatment effects are really due to natural growth or development
Selection by maturation interactions: participants growing in different ways
![Page 14: Topics Ahead - zhang.ist.psu.eduzhang.ist.psu.edu/teaching/505/pdf/Week10.pdf · One slide on research question Literature review: structured; holes you will fill in Your research](https://reader033.vdocument.in/reader033/viewer/2022052021/603523f33f264507ab091626/html5/thumbnails/14.jpg)
Experimental Study
Experimental designs can rule out most threats to internal validity.
![Page 15: Topics Ahead - zhang.ist.psu.eduzhang.ist.psu.edu/teaching/505/pdf/Week10.pdf · One slide on research question Literature review: structured; holes you will fill in Your research](https://reader033.vdocument.in/reader033/viewer/2022052021/603523f33f264507ab091626/html5/thumbnails/15.jpg)
What Is An Experimental
Study?
![Page 16: Topics Ahead - zhang.ist.psu.eduzhang.ist.psu.edu/teaching/505/pdf/Week10.pdf · One slide on research question Literature review: structured; holes you will fill in Your research](https://reader033.vdocument.in/reader033/viewer/2022052021/603523f33f264507ab091626/html5/thumbnails/16.jpg)
Basic Logic
Get a hypotheses
Causal relationship
Independent Variable Dependent Variable
Manipulate the independent variable (IV) and measure the dependent variable (DV)
Have experimental and control groups
Similar, but treated differently
Random selected
Statistically analyze the difference
Significant result or not?
![Page 17: Topics Ahead - zhang.ist.psu.eduzhang.ist.psu.edu/teaching/505/pdf/Week10.pdf · One slide on research question Literature review: structured; holes you will fill in Your research](https://reader033.vdocument.in/reader033/viewer/2022052021/603523f33f264507ab091626/html5/thumbnails/17.jpg)
Key Issue
Apply treatment on one group but not the other.
The only difference between two groups is the treatment.
If DV shows any difference, it is due to the treatment!
All other factors should be the same, at least in theory.
Task, observation instrument, procedure.
Subjects
![Page 18: Topics Ahead - zhang.ist.psu.eduzhang.ist.psu.edu/teaching/505/pdf/Week10.pdf · One slide on research question Literature review: structured; holes you will fill in Your research](https://reader033.vdocument.in/reader033/viewer/2022052021/603523f33f264507ab091626/html5/thumbnails/18.jpg)
HOW CAN WE GUARANTEE SUBJECTS IN TWO GROUPS ARE THE SAME?
![Page 19: Topics Ahead - zhang.ist.psu.eduzhang.ist.psu.edu/teaching/505/pdf/Week10.pdf · One slide on research question Literature review: structured; holes you will fill in Your research](https://reader033.vdocument.in/reader033/viewer/2022052021/603523f33f264507ab091626/html5/thumbnails/19.jpg)
Group Assignment
Random assignment Select subjects for two groups from the same
subject pool.
Assign them to two groups randomly.
Without random assignment, you do not have an experiment
![Page 20: Topics Ahead - zhang.ist.psu.eduzhang.ist.psu.edu/teaching/505/pdf/Week10.pdf · One slide on research question Literature review: structured; holes you will fill in Your research](https://reader033.vdocument.in/reader033/viewer/2022052021/603523f33f264507ab091626/html5/thumbnails/20.jpg)
Methods for Random Selection
Random
Pick up a series of numbers to sort
Excel
Generate random numbers and then sort.
Toss a coin
Tail experimental group
Head control group
![Page 21: Topics Ahead - zhang.ist.psu.eduzhang.ist.psu.edu/teaching/505/pdf/Week10.pdf · One slide on research question Literature review: structured; holes you will fill in Your research](https://reader033.vdocument.in/reader033/viewer/2022052021/603523f33f264507ab091626/html5/thumbnails/21.jpg)
Using a Random Table
![Page 22: Topics Ahead - zhang.ist.psu.eduzhang.ist.psu.edu/teaching/505/pdf/Week10.pdf · One slide on research question Literature review: structured; holes you will fill in Your research](https://reader033.vdocument.in/reader033/viewer/2022052021/603523f33f264507ab091626/html5/thumbnails/22.jpg)
Using Random Table
![Page 23: Topics Ahead - zhang.ist.psu.eduzhang.ist.psu.edu/teaching/505/pdf/Week10.pdf · One slide on research question Literature review: structured; holes you will fill in Your research](https://reader033.vdocument.in/reader033/viewer/2022052021/603523f33f264507ab091626/html5/thumbnails/23.jpg)
Steps
![Page 24: Topics Ahead - zhang.ist.psu.eduzhang.ist.psu.edu/teaching/505/pdf/Week10.pdf · One slide on research question Literature review: structured; holes you will fill in Your research](https://reader033.vdocument.in/reader033/viewer/2022052021/603523f33f264507ab091626/html5/thumbnails/24.jpg)
Using Excel
Generate a set of random numbers
Using the rand() function
Copy the values!
![Page 25: Topics Ahead - zhang.ist.psu.eduzhang.ist.psu.edu/teaching/505/pdf/Week10.pdf · One slide on research question Literature review: structured; holes you will fill in Your research](https://reader033.vdocument.in/reader033/viewer/2022052021/603523f33f264507ab091626/html5/thumbnails/25.jpg)
IV and DV CONTROL
![Page 26: Topics Ahead - zhang.ist.psu.eduzhang.ist.psu.edu/teaching/505/pdf/Week10.pdf · One slide on research question Literature review: structured; holes you will fill in Your research](https://reader033.vdocument.in/reader033/viewer/2022052021/603523f33f264507ab091626/html5/thumbnails/26.jpg)
Data Collection and Analysis
Knowing what to measure Variables
Calculating means Sample means
Comparing sample means Statistic methods
![Page 27: Topics Ahead - zhang.ist.psu.eduzhang.ist.psu.edu/teaching/505/pdf/Week10.pdf · One slide on research question Literature review: structured; holes you will fill in Your research](https://reader033.vdocument.in/reader033/viewer/2022052021/603523f33f264507ab091626/html5/thumbnails/27.jpg)
Independent Variable
Treatment
Simple experiments
Treatment: yes and no
Experimental group: applied
Control group: not applied
Treatment: different levels and no
Multiple experimental groups and one control group
![Page 28: Topics Ahead - zhang.ist.psu.eduzhang.ist.psu.edu/teaching/505/pdf/Week10.pdf · One slide on research question Literature review: structured; holes you will fill in Your research](https://reader033.vdocument.in/reader033/viewer/2022052021/603523f33f264507ab091626/html5/thumbnails/28.jpg)
Manipulation IV
Making the treatment observable and significant
![Page 29: Topics Ahead - zhang.ist.psu.eduzhang.ist.psu.edu/teaching/505/pdf/Week10.pdf · One slide on research question Literature review: structured; holes you will fill in Your research](https://reader033.vdocument.in/reader033/viewer/2022052021/603523f33f264507ab091626/html5/thumbnails/29.jpg)
Statistic Analysis
Rule out random errors Errors = random errors + systematic errors by the
treatment
Statistically significant results
You can declare the causal effects.
May not be the same direction you expected
Non-significant results
You didn’t find it
But it doesn’t mean it doesn’t exist
![Page 30: Topics Ahead - zhang.ist.psu.eduzhang.ist.psu.edu/teaching/505/pdf/Week10.pdf · One slide on research question Literature review: structured; holes you will fill in Your research](https://reader033.vdocument.in/reader033/viewer/2022052021/603523f33f264507ab091626/html5/thumbnails/30.jpg)
Type of Experiments
![Page 31: Topics Ahead - zhang.ist.psu.eduzhang.ist.psu.edu/teaching/505/pdf/Week10.pdf · One slide on research question Literature review: structured; holes you will fill in Your research](https://reader033.vdocument.in/reader033/viewer/2022052021/603523f33f264507ab091626/html5/thumbnails/31.jpg)
Often Seen Types
Simple experiment
One independent variable
Two groups
Multi-group experiment
One independent variable
Multiple groups (>2)
Factorial design
Two or more independent variables
At least four groups
![Page 32: Topics Ahead - zhang.ist.psu.eduzhang.ist.psu.edu/teaching/505/pdf/Week10.pdf · One slide on research question Literature review: structured; holes you will fill in Your research](https://reader033.vdocument.in/reader033/viewer/2022052021/603523f33f264507ab091626/html5/thumbnails/32.jpg)
Simple Experiment
t test
Could be between- or within-subject design
![Page 33: Topics Ahead - zhang.ist.psu.eduzhang.ist.psu.edu/teaching/505/pdf/Week10.pdf · One slide on research question Literature review: structured; holes you will fill in Your research](https://reader033.vdocument.in/reader033/viewer/2022052021/603523f33f264507ab091626/html5/thumbnails/33.jpg)
Questions Raised by Results
Questions raised by non-significant results Enough participants?
Participants homogeneous enough?
Experiment sufficiently standardized?
Data coded carefully?
DV sensitive and reliable enough?
![Page 34: Topics Ahead - zhang.ist.psu.eduzhang.ist.psu.edu/teaching/505/pdf/Week10.pdf · One slide on research question Literature review: structured; holes you will fill in Your research](https://reader033.vdocument.in/reader033/viewer/2022052021/603523f33f264507ab091626/html5/thumbnails/34.jpg)
Pilot Study Testing various aspects of study
Subjects
IV
DV
Procedure and protocols
With a smaller group of subjects
With the SAME procedure and protocols Must have all experimental materials ready before the pilot study
Issues to look at Procedure and protocols (e.g., instruction, task, etc.)
Data collection method
The number of subjects required