© a. taylor do not duplicate without author’s permission 1 reliability and validity
TRANSCRIPT
© A. Taylor
Do not duplicate without author’s permission 2
Objectives
• There are five (5) main objectives for this module, to:
1. Be able to recognize various tools of measurement
2. Understand what is meant by reliability
3. Be able to evaluate the reliability of a measure
4. Understand what is meant by validity
5. Be able to evaluate the validity of a measure
© A. Taylor
Do not duplicate without author’s permission 3
We have already learned about internal and external validity
of a study
Now we are going to learn about reliability and validity
of a measure
© A. Taylor
Do not duplicate without author’s permission 4
We have already learned about internal and external validity
of a study
Now we are going to learn about reliability and validity
of a measure
Don’t get confused. Even though they both use the terms validity, they use them in a slightly different way.
© A. Taylor
Do not duplicate without author’s permission 5
In Other Words
• When evaluating a Study discuss the– Internal Validity– External Validity
• When evaluating a Measure discuss the– Reliability– Validity
You already learned about this in the Integrating Research Concepts tutorial
We are learning about this now
© A. Taylor
Do not duplicate without author’s permission 6
What is Reliability?
• Reliability is:– the consistency of your measurement
instrument
– the degree to which an instrument measures the same way each time it is used under the same condition with the same subjects
© A. Taylor
Do not duplicate without author’s permission 7
What is Reliability?
• Reliability is:– the consistency of your measurement
instrument
– the degree to which an instrument measures the same way each time it is used under the same condition with the same subjects
What do I mean by…?
© A. Taylor
Do not duplicate without author’s permission 8
What in the world is a measurement instrument?
• Any tool that you use to measure with…• What “instrument” might you use to measure the
following items?
1. How heavy the apples are
2. How hot the meat is
3. How much orange juice there is
4. How tall the wall is
a scale
a meat thermometer
a measuring cup
a yardstick
© A. Taylor
Do not duplicate without author’s permission 9
More on Instruments
What would the following instruments measure?
The size of someone’s foot
EyesightThe perfect amount of spaghetti!
© A. Taylor
Do not duplicate without author’s permission 10
More on InstrumentsOften there are many different instruments that could be
used to measure the same thing.
Take distance for example…
Or an odometer
You could use a measuring tapeMeasure it out on a map
Or walk it with a pedometer
You can count paces
Or get really creative…Use a distance wheel
© A. Taylor
Do not duplicate without author’s permission 11
What instrument you choose depends on several factors
• Ease of use – do you know how to use it?I still can’t figure out my pedometer.
• Access – do you have access to the instrument?Where do you get a distance wheel from? What if you don’t have a car – and therefore no odometer?
• Appropriateness – is it the best tool for what you are measuring? You wouldn’t use a ruler to measure the distance from here to Ohio; you wouldn’t use an odometer to measure the length of a room.
• Accuracy – how precise do your measurements have to be?Counting paces is easy and cheap but only an estimate.
• Cost – how expensive is it to use the instrument or interpret the results?You can also use a laser but that can be really expensive.
© A. Taylor
Do not duplicate without author’s permission 12
Back to Reliability…
• Reliability is:– the consistency of your measurement
instrument
– the degree to which an instrument measures the same way each time it is used under the same condition with the same subjects
© A. Taylor
Do not duplicate without author’s permission 13
Reliability
• Imagine that you are using a ruler
to measure a book
And it measures about 7 inches across
What do you think would happen if you waited 10 minutes and measured the book again, how long would it be then?
…Probably still 7 inches
What if you spun the ruler around!
And shook it up really good?!
Now what would it say?
…Probably still 7 inches
© A. Taylor
Do not duplicate without author’s permission 14
Reliability
• Your ruler… – was consistent– measured the same way each time it
was used under the same condition with the same object
The book did not change and therefore the ruler reported back the same measurement
Your ruler is RELIABLE
© A. Taylor
Do not duplicate without author’s permission 15
Reliability• Every time the water begins to
boil the kettle whistles
• Every time it is 6:30 my alarm clock goes off
• Every time I use the little plastic up I have 2 tablespoons of medicine
All of these tools of measurement are reliablereliable given the same temperature, time and volume, they measure the same consistentlyconsistently.
© A. Taylor
Do not duplicate without author’s permission 16
Reliability• Reliability alone does not mean that you
have a good instrument however.
• Imagine the following reliable instrument:
0165Every morning Margie gets on the scale and every morning it reads 165 lbs.
It seems pretty reliable since Margie hasn’t gained or lost any weight.
One day an elephant got on Margie’s scale and it still read 165 lbs.!
How is this scale reliable?
It does measure the same way under the same conditions (Margie’s) – but a lot of other conditions too (the elephant and who knows what else).
© A. Taylor
Do not duplicate without author’s permission 17
Reliability
• What about this reliable instrument…
This clock reads 6:15
If nothing changes – if time stands still, will the clock still say the same thing?
YES! It’s very reliable. You always know exactly what it is going to say.
The problem is, even if time doesn’t stand still, the clock will not move…but it IS still reliable.
© A. Taylor
Do not duplicate without author’s permission 19
What is Validity?
• Validity asks – if an instrument measures what it is
supposed to
– how “true” or accurate the measurement is
© A. Taylor
Do not duplicate without author’s permission 20
Reliable but not Valid165
These instruments are very RELIABLE
They both report consistently – too consistently
But, neither measures what it is supposed to:
• The scale is not really measuring weight
• The clock is not measuring time
They are NOT VALID
© A. Taylor
Do not duplicate without author’s permission 21
Reliable but not Valid
• Remember our reliable ruler?
Can it measure how loud the radio is? how full the
glass is?
how smart the girl is?
The ruler may be reliable (and perhaps even valid) but not in these situations!
It is only valid for measuring length.
© A. Taylor
Do not duplicate without author’s permission 22
Putting Reliability and Validity Together
• Every instrument can be evaluated on two dimensions:– Reliability
• How consistent it is given the same conditions
– Validity• If it measures what it is supposed to and how
accurate it is
© A. Taylor
Do not duplicate without author’s permission 23
Putting Reliability and Validity Together
• Imagine that I have 3 fish tank thermometers, a blue one, a red one, and a green one.
• The blue one always reads the same temperature no matter how hot or cold the water is.
• The red one shows a different temperature every time even if I just measured it 5 seconds earlier.
• The green one seems to read accurately, warm when the water is warm and cold when the water is cool.
© A. Taylor
Do not duplicate without author’s permission 24
Complete the chart below
Is it consistent?
Is it measuring what it is supposed
to?
Is it reliable?
Is it valid?
Blue always reads the same temperature no matter what
Red different temperature every time even if nothing has changed
Green warm when the water is warm and cold when the water is cool
Yes
No
Yes, the thermometer only changes if the temperature changes
No
No
Yes
Reliable but Not valid
Not reliable Not valid
Reliable and Valid
© A. Taylor
Do not duplicate without author’s permission 25
What can be said of the reliability and validity of the following?
• A spelling test with the following item: 2 + 5 = ____ – Probably reliable, if you get it wrong once you will probably get it wrong again
(assuming no new learning) – same with getting it right.
– Lacks validity, this is more appropriate for a math test, not a spelling test
• An elastic ruler (every time you use it is stretches to a different length)– Lack reliability
– You can’t have validity without reliability
• A thermometer used to measure volume – Probably reliable
– Lacks validity for this task
• A scale that reads 40 pounds at baseline – Reliable, will consistently be 40 lbs. off
– Not valid
© A. Taylor
Do not duplicate without author’s permission 26
Consider the Peabody Picture Inventory
Is it reliable? – if you don’t know the vocabulary today, are you going to suddenly know it tomorrow or next week? Probably not.
We’ll say it’s reliable
Imagine someone wanted to use it as a measure of cognitive development, is it valid?
Well, there are a lot of ways to operationalize cognitive development. Some might say that this is a better measure of English language competence rather than general cognition.
We’ll say that it lacks some validity for general cognition.
© A. Taylor
Do not duplicate without author’s permission 27
Consider a Maze Task to measure fine motor skills
Is it reliable? – if you do the maze once, do you think you need the same amount of time to finish it a second time?
You’ll probably be faster the second time just from practicing – even though your motor skills didn’t change any.
We’ll say it lacks reliability.Is it valid?
No, a measure must be reliable in order to be valid.
It is probably a more valid measure of memory.
© A. Taylor
Do not duplicate without author’s permission 28
Is it reliable? – if you don’t know the answers, are you going to suddenly know them tomorrow or next week? Probably not.
We’ll say it’s reliable
Is it valid?
This doesn’t look like a measure of World Geography to me, it focuses on America. It doesn’t even seem like American geography – maybe American Geographical Trivia.
It’s not a valid measure of World Geography.
1. The nickname for the state of New Hampshire is: a. Vacationland b. The Pine Tree State c. The Granite State
2. The most populated U.S. city is: a. New York b. Los Angeles c. Chicago
3. The "Coyote" State is: a. New Mexico b. Nebraska c. South Dakota
4. The largest state is: a. Texas b. Alaska c. Hawaii
5. The highest mountain is the U.S. is: a. Mt. McKinley b. Mt. Shasta c. Mt. Logan
Consider the World Geography quiz below:
© A. Taylor
Do not duplicate without author’s permission 29
Thinking More about Validity
Validity is whether or not the instrument measures what it is designed to measure.
Below are three constructs that you can use to evaluate the validity of a measure:
• Face Validity
• Predictive Validity
• Concurrent Validity
© A. Taylor
Do not duplicate without author’s permission 30
Three Validities
• Face Validity -- Do the questions look like measure what they are supposed to?
• Predictive Validity -- Do you imagine that this measure would predict something that it logically should?
• Concurrent Validity – Do you think this measure correlates strongly with something that it logically should?
© A. Taylor
Do not duplicate without author’s permission 31
Face Validity
Do the questions look like they measure what they are supposed to?
• What does the question below look like it would measure?
Choose the item that best describes you:(0) I do not feel sad. (1) I feel sad. (2) I am sad all the time and I can't snap out of it. (3) I am so sad or unhappy that I can't stand it.
If you said depression you would be correct. It looks like an item from a depression scale – and it is!
It has face validity.
If someone said this was from a parental attachment scale, then you could say that it lacks face validity.
© A. Taylor
Do not duplicate without author’s permission 32
Predictive Validity
Does the measure predict something that it logically should?
The SATAccording to their website, “the SAT is the one test that shows colleges what you know”. College admissions officers use this test to determine who will do well in college.
What future measures do you expect that the SAT would be correlated with?
If you said grades in college you would be right, and the SAT IS correlated with that! – somewhat…
It has predictive validity
© A. Taylor
Do not duplicate without author’s permission 33
Concurrent Validity
Does this measure correlate strongly with something that it logically should right now?Unlike predictive validity, concurrent validity compares measures taken at the same time.
What current measures do you imagine that GPA should correlate with?
If you said current test scores and teacher ratings you would be correct.
GPA has some concurrent validity
© A. Taylor
Do not duplicate without author’s permission 34
You should now be ready to begin the IQ Measures
assignment
Feel free to review
this module as needed
© A. Taylor
Do not duplicate without author’s permission 35
Resources• The College Board (2008). SAT: Know more. Retrieved September
8, 2008 from http://www.collegeboard.com/sat-achieve-more/?s_kwcid=sat%20test|2592657944
• Creasey, G. L. (2005). Research methods in lifespan development. Upper Saddle River, NJ: Pearson Education.
• Family Education Network (2008). American Geography. In Back to school with family education American geography. Retrieved September 8, 2008 from http://quizzes.familyeducation.com/maps/united-states/55376.html
• Korbin, J. L., Patterson, B. F., Shaw, E. J., Mattern, K. D., & Barbuti, S. M. (2008). Validity of the SAT for Predicting First Year College Grade Point Average. In College Board research report #2008-5. Retrieved September 8, 2008 from http://professionals.collegeboard.com/profdownload/Validity_of_the_SAT_for_Predicting_First_Year_College_Grade_Point_Average.pdf