validity and reliability in instrumentation 47.469: research i: basics dr. leonard february 24, 2010
Post on 21-Dec-2015
222 views
TRANSCRIPT
![Page 1: Validity and Reliability in Instrumentation 47.469: Research I: Basics Dr. Leonard February 24, 2010](https://reader030.vdocument.in/reader030/viewer/2022033103/56649d545503460f94a3029a/html5/thumbnails/1.jpg)
Validity and Reliability in Instrumentation
47.469: Research I: Basics
Dr. Leonard
February 24, 2010
![Page 2: Validity and Reliability in Instrumentation 47.469: Research I: Basics Dr. Leonard February 24, 2010](https://reader030.vdocument.in/reader030/viewer/2022033103/56649d545503460f94a3029a/html5/thumbnails/2.jpg)
Recap Research design can be…
experimental or non-experimental (maybe quasi-experimental) basic or applied research laboratory or field setting quantitative or qualitative data collection
Research must be based in solid theory and testable hypotheses
Research must include clear conceptual and operational definitions
![Page 3: Validity and Reliability in Instrumentation 47.469: Research I: Basics Dr. Leonard February 24, 2010](https://reader030.vdocument.in/reader030/viewer/2022033103/56649d545503460f94a3029a/html5/thumbnails/3.jpg)
Quasi-experimental Occurring more commonly in psychology
Apply experimental principles like cause and effect or group comparison to field, or less controlled settings More like correlational research
Less control over extraneous variables but can take place outside of lab, which may decrease the artificial feeling
Interpretation of results not as clean as in experimental research but closer to “real world” application
![Page 4: Validity and Reliability in Instrumentation 47.469: Research I: Basics Dr. Leonard February 24, 2010](https://reader030.vdocument.in/reader030/viewer/2022033103/56649d545503460f94a3029a/html5/thumbnails/4.jpg)
Scientific method
1. Formulate theories √√
2. Develop testable hypotheses (operational definitions) √√
3. Conduct research, gather data √√
4. Evaluate hypotheses based on data
5. Cautiously draw conclusions
![Page 5: Validity and Reliability in Instrumentation 47.469: Research I: Basics Dr. Leonard February 24, 2010](https://reader030.vdocument.in/reader030/viewer/2022033103/56649d545503460f94a3029a/html5/thumbnails/5.jpg)
Next steps…gather data Once you have explicitly clear conceptual and
operational definitions to guide the research, you must develop your measures for collecting data Operational definition proposes type of measures
Instrumentation is the process of selecting or creating measures for a study (the measure is your instrument)
Two overarching goals for instrumentation Validity: the extent to which a measure (operationally
defined) taps the concept it’s designed to measure and not some other concept
Reliability: the consistency or stability of a measure, i.e., same results obtained if measure used again
![Page 6: Validity and Reliability in Instrumentation 47.469: Research I: Basics Dr. Leonard February 24, 2010](https://reader030.vdocument.in/reader030/viewer/2022033103/56649d545503460f94a3029a/html5/thumbnails/6.jpg)
Caveats
Can never be certain of the validity (or reliability) of our instruments so we try to speculate the degree of validity We might claim “modest” or “partial” validity Hard to capture true essence of a concept/construct and
some concepts/constructs are more elusive than others! An estimate of the validity of our measures depends on the
purpose of the study Keep focused on the hypotheses and operational definitions!
Two types of validity we estimate Judgmental validity Empirical validity
![Page 7: Validity and Reliability in Instrumentation 47.469: Research I: Basics Dr. Leonard February 24, 2010](https://reader030.vdocument.in/reader030/viewer/2022033103/56649d545503460f94a3029a/html5/thumbnails/7.jpg)
Types of validity: Judgmental Content validity: whether the concept being
measured is a real concept AND whether the measurement being used is the most appropriate one to be using
Is our operationally defined variable (concrete) really capturing the hypothetical concept (abstract) we are interested in studying?
Are we capturing the central meaning?
Concept
Variable/Measure
![Page 8: Validity and Reliability in Instrumentation 47.469: Research I: Basics Dr. Leonard February 24, 2010](https://reader030.vdocument.in/reader030/viewer/2022033103/56649d545503460f94a3029a/html5/thumbnails/8.jpg)
Types of validity: Judgmental Content validity, or any other type of validity alone, is
never enough to determine if our measure is valid so we consider other types…
Face validity: measure is valid because it makes sense; on the surface, it seems to tap into construct of interest
Face Validity is neither sufficient nor absolutely necessary for overall validity, but is a helpful clue
Could have high face validity but low content validity!
![Page 9: Validity and Reliability in Instrumentation 47.469: Research I: Basics Dr. Leonard February 24, 2010](https://reader030.vdocument.in/reader030/viewer/2022033103/56649d545503460f94a3029a/html5/thumbnails/9.jpg)
Good face validity?Rosenberg Self-Esteem Scale
1= Strongly Disagree, 7 = Strongly Agree
_____1. I feel that I am a person of worth, at least on an equal basis with others.
_____2. I feel that I have a number of good qualities. _____3. All in all, I am inclined to think that I am a failure.*_____4. I am able to do things as well as most people. _____5. I feel that I do not have much to be proud of.* _____6. I take a positive attitude towards myself. _____7. On the whole, I am satisfied with myself. _____8. I wish I could have more respect for myself.* _____9. I certainly feel useless at times.*_____10. At times I think I am no good at all.*
*Reverse scored
![Page 10: Validity and Reliability in Instrumentation 47.469: Research I: Basics Dr. Leonard February 24, 2010](https://reader030.vdocument.in/reader030/viewer/2022033103/56649d545503460f94a3029a/html5/thumbnails/10.jpg)
Types of validity: Empirical• Criterion-related Validity: extent to which your
measure of a concept relates to a theoretically meaningful criterion for that concept, a “gold standard” for that concept
• Predictive validity: The measure should be able to predict future behavior that is related to the concept
• E.g., Job skills test and future ratings of performance• Concurrent (convergent) validity: The measure
should be meaningfully related or correlated to some other measure of the behavior
• E.g., Scores on two different job skills tests• Predicitve or concurent validity coefficient: a
number (0-1) based on correlation that quantifies whether the measure is in fact related to other measures it should be related to
![Page 11: Validity and Reliability in Instrumentation 47.469: Research I: Basics Dr. Leonard February 24, 2010](https://reader030.vdocument.in/reader030/viewer/2022033103/56649d545503460f94a3029a/html5/thumbnails/11.jpg)
Predictive Validity
Job skills test
QualificationFor job
Job skills test
Future performance
ratings
Correlation coefficient = .60
![Page 12: Validity and Reliability in Instrumentation 47.469: Research I: Basics Dr. Leonard February 24, 2010](https://reader030.vdocument.in/reader030/viewer/2022033103/56649d545503460f94a3029a/html5/thumbnails/12.jpg)
Concurrent (convergent) Validity
Job skills test A
Qualification for job
Job skillsTest A
Job skillsTest B
Job skills test B
![Page 13: Validity and Reliability in Instrumentation 47.469: Research I: Basics Dr. Leonard February 24, 2010](https://reader030.vdocument.in/reader030/viewer/2022033103/56649d545503460f94a3029a/html5/thumbnails/13.jpg)
Types of validity: Judgmental-Empirical Construct validity represents a combined
approach for estimating validity using 1) a subjective prediction about what other concepts
(indicators) the concept being measured should relate to and.. May relate positively OR negatively
2) an empirical test of whether the concept is in fact related to those other indicators E.g., Depression should be linked to disengagement
from schoolwork among college students so test relationship between depression scores and GPA among a sample of students
![Page 14: Validity and Reliability in Instrumentation 47.469: Research I: Basics Dr. Leonard February 24, 2010](https://reader030.vdocument.in/reader030/viewer/2022033103/56649d545503460f94a3029a/html5/thumbnails/14.jpg)
Construct Validity exercise Take heart rate for 30 seconds and multiply by 2,
record on separate paper Repeat Average two heart rate measurements Turn in paper Complete Manifest Anxiety Scale Score Turn in sheet Why is this as a test of construct validity?
![Page 15: Validity and Reliability in Instrumentation 47.469: Research I: Basics Dr. Leonard February 24, 2010](https://reader030.vdocument.in/reader030/viewer/2022033103/56649d545503460f94a3029a/html5/thumbnails/15.jpg)
Reliability
The consistency or stability of a measure; easier to establish when measure is unidimensional
Related to validity? Yes!
Generally, more valid measures tend to be more reliable BUT you could have a highly reliable measure that is low in validity Think of gun shooting a target example
Like validity, reliability can be estimated by a correlation coefficient (0-1) Generally, to be respectable in the scientific community, reliability
should be .80 (80%) or higher
![Page 16: Validity and Reliability in Instrumentation 47.469: Research I: Basics Dr. Leonard February 24, 2010](https://reader030.vdocument.in/reader030/viewer/2022033103/56649d545503460f94a3029a/html5/thumbnails/16.jpg)
Relationship between reliability and validity
Is our measure RELIABLE? Does it have consistency and stability in measurement?
Is our measure VALID? Does it measure what it’s supposed to measure?
Validity is more important to a research study; reliability can’t tell us if we are measuring the correct concept, only if we are measuring something consistently.
![Page 17: Validity and Reliability in Instrumentation 47.469: Research I: Basics Dr. Leonard February 24, 2010](https://reader030.vdocument.in/reader030/viewer/2022033103/56649d545503460f94a3029a/html5/thumbnails/17.jpg)
Classical Test Theory
An observed measurement (or score, X) is comprised of a true score (T, the score that would be obtained if there was no measurement error) and some random measurement error (E).
X = T + E
X is the observed scoreT is the true scoreE is the measurement error
![Page 18: Validity and Reliability in Instrumentation 47.469: Research I: Basics Dr. Leonard February 24, 2010](https://reader030.vdocument.in/reader030/viewer/2022033103/56649d545503460f94a3029a/html5/thumbnails/18.jpg)
Types of Reliability Test-retest reliability - consistent results from same
measure under same conditions two times Across time
Inter-rater reliability - consistent results when same measure is given twice, but with different test givers, or have two independent observers code some behavior Across raters or observers
Alpha reliability - individual items/questions from a scale measuring same concept are correlated Across items
Split-half reliability - items from one part of a scale are correlated and measure same concept as another part Across items
![Page 19: Validity and Reliability in Instrumentation 47.469: Research I: Basics Dr. Leonard February 24, 2010](https://reader030.vdocument.in/reader030/viewer/2022033103/56649d545503460f94a3029a/html5/thumbnails/19.jpg)
Test-retest reliability (across time)
ID
Time 1
X
Time 2
X
1
2
3
4
18
12
29
25
19
13
28
25
![Page 20: Validity and Reliability in Instrumentation 47.469: Research I: Basics Dr. Leonard February 24, 2010](https://reader030.vdocument.in/reader030/viewer/2022033103/56649d545503460f94a3029a/html5/thumbnails/20.jpg)
Inter-rater reliability (across raters)
ID
Rater 1
X
Rater 2
X
Rater 3
X
1
2
3
4
18
12
29
25
19
13
28
25
20
14
27
24
![Page 21: Validity and Reliability in Instrumentation 47.469: Research I: Basics Dr. Leonard February 24, 2010](https://reader030.vdocument.in/reader030/viewer/2022033103/56649d545503460f94a3029a/html5/thumbnails/21.jpg)
Alpha reliability (across items)
ID
Item 1
X1
Item 2
X2
(Reversed)
Item 3
X3
Item 4
X4
(Reversed)
Item 5
X5
(Reversed)
Item 6
X6
1
2
3
4
4
3
2
1
4
2
1
1
5
4
3
2
4
3
1
1
4
3
2
1
5
4
4
2
Sometimes called internal consistency
![Page 22: Validity and Reliability in Instrumentation 47.469: Research I: Basics Dr. Leonard February 24, 2010](https://reader030.vdocument.in/reader030/viewer/2022033103/56649d545503460f94a3029a/html5/thumbnails/22.jpg)
Split-half reliability (across items)
ID
First 1/2 items
Second 1/2 items
1
2
3
4
5
6
7
8
9
10
1
2
3
3
2
3
1
2
2
1
1
2
2
3
1
2
1
2
2
2
Sometimes called internal consistency
![Page 23: Validity and Reliability in Instrumentation 47.469: Research I: Basics Dr. Leonard February 24, 2010](https://reader030.vdocument.in/reader030/viewer/2022033103/56649d545503460f94a3029a/html5/thumbnails/23.jpg)
The more, the better
As with validity, it is always better if you can estimate or test for multiple forms of reliability! Sometimes called parallel-forms reliability if
measure is available in more than one version and can be given in both ways and then compared
![Page 24: Validity and Reliability in Instrumentation 47.469: Research I: Basics Dr. Leonard February 24, 2010](https://reader030.vdocument.in/reader030/viewer/2022033103/56649d545503460f94a3029a/html5/thumbnails/24.jpg)
Valid? Reliable? Concept: Parental engagement in child’s academic
development
How often do you help your child with his/her homework (please check one)?
_Never_Rarely_Sometimes_Often_Everyday
![Page 25: Validity and Reliability in Instrumentation 47.469: Research I: Basics Dr. Leonard February 24, 2010](https://reader030.vdocument.in/reader030/viewer/2022033103/56649d545503460f94a3029a/html5/thumbnails/25.jpg)
Concept? Valid? Reliable?
Is there a chance that you could get HIV/AIDS?
1--------------------2--------------------3 Not at Small chance Yes
All Definitely
Do you worry about getting HIV/AIDS? (circle one number)
1--------------------2--------------------3--------------------4--------------------5 Never Almost Sometimes Often
Very Never Often
![Page 26: Validity and Reliability in Instrumentation 47.469: Research I: Basics Dr. Leonard February 24, 2010](https://reader030.vdocument.in/reader030/viewer/2022033103/56649d545503460f94a3029a/html5/thumbnails/26.jpg)
Concept? Valid? Reliable?
How important is financial success to you? _Very important _Somewhat important _Not at all important
How important is it for you to have nice things?_Very important _Somewhat important _Not at all important
![Page 27: Validity and Reliability in Instrumentation 47.469: Research I: Basics Dr. Leonard February 24, 2010](https://reader030.vdocument.in/reader030/viewer/2022033103/56649d545503460f94a3029a/html5/thumbnails/27.jpg)
Our total MAS scores and average heart rate were only correlated at -.04; correlational relationships can have a magnitude from 0-1 but also a direction (+ or -)
Correlation between Heart Rate and MAS Score
Linear Regression
10 20 30
MAS_total
30.00
35.00
40.00
45.00
Heart_rate
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
Heart_rate = 40.04 + -0.02 * MAS_total
R-Square = 0.00
Good construct validity? Why?
Good predictive validity? Why?
![Page 28: Validity and Reliability in Instrumentation 47.469: Research I: Basics Dr. Leonard February 24, 2010](https://reader030.vdocument.in/reader030/viewer/2022033103/56649d545503460f94a3029a/html5/thumbnails/28.jpg)
Beginning APA style for your proposal
Author last name, author first and middle initials. (Year published). Title of article. Title of Journal, Volume number (Issue number), pg.-pg.
Morelli, G. A., Rogoff, B., Oppenheim, D., & Goldsmith, D. (1996). Cultural variations in infants’ sleeping arrangements: Questions of independence. Developmental Psychology, 28(4), 604-613.
Put the following three articles into an APA style reference
![Page 29: Validity and Reliability in Instrumentation 47.469: Research I: Basics Dr. Leonard February 24, 2010](https://reader030.vdocument.in/reader030/viewer/2022033103/56649d545503460f94a3029a/html5/thumbnails/29.jpg)
Three APA references
Samuolis, J., Layburn, K., & Schiaffino, K. M. (2001). Identity development and attachment to parents in college students. Journal of Youth and Adolescence, 30 (3), 373-383.
Tripodi, S. J., Bender, K., Litschge, C., & Vaughn, M. G. (2010). Interventions for reducing adolescent alcohol abuse: A meta-analytic review. Archives of Pediatric and Adolescent Medicine, 164 (1), 85-91.