thursday validity, reliability and generalizabilityupdated

8/14/2019 Thursday Validity, Reliability and Generalizabilityupdated

1/16

Validity, Reliability andValidity, Reliability and

GeneralizabilityGeneralizability


2/16

ValidityValidity

HOW MIGHT YOU BE WRONG?HOW MIGHT YOU BE WRONG?

Internal ValidityInternal Validity

External ValidityExternal ValidityConstruct ValidityConstruct Validity


3/16

ValidityValidity

Are your findings really about what theyAre your findings really about what they

appear to be about or did something elseappear to be about or did something else

cause them?cause them?


4/16

Internal ValidityInternal Validity

Did your treatment actually cause theDid your treatment actually cause the

outcomes?outcomes?

Did anything else potentially cause theDid anything else potentially cause the

outcome instead of your treatment?outcome instead of your treatment?


5/16

INTERNAL VALIDITYINTERNAL VALIDITY

Internal validity addresses the "true" causes of theInternal validity

addresses the "true" causes of theoutcomes that you observed in your study.outcomes that y

ou observed in your study. StrongStronginternal validity means that you not only haveinternal validity means that you not only havereliable measures of your independent andreliable measures of your independent and

dependent variables BUT a strong justification thatdependent variables BUT a strong justification thatyou are able to rule out extraneous variables, oryou are able to rule out extraneous variables, oralternative, often unanticipated, causes for youralternative, often unanticipated, causes for yourfindings.findings. Thus strong internal validity refers to theThus strong

internal validity refers to theunambiguous assignment of causes to effects.unambig

uous assignment of causes to effects.

Internal validity is aboutInternal validity is about causal controlcausal control..


6/16

EXTERNAL VALIDITYEXTERNAL VALIDITY

External validity addresses the abilityExternal validity

addresses the abilityto generalize your study to otherto g

eneralize your study to other

people and other situations.

people and other situations. To haveTo havestrong external validity (ideally), youstrong external validity (ideally), youneed a probability sample of subjectsneed a probability sample of subjects

or respondents drawn using "chanceor respondents drawn using "chancemethods" from a clearly definedmethods" from a clearly definedpopulationpopulation


7/16

External ValidityExternal Validity

When you have strong external validity,When you have strong external validity,

you can generalize to other people andyou can generalize to other people and

situations with confidence.situations with confidence.

External validity refers to the population weExternal validity refers to the population we

use and the setting. If I do a study with fifthuse and the setting. If I do a study with fifth

graders, then the results can be applied to agraders, then the results can be applied to a

similar set of fifth graders.similar set of fifth graders.


8/16

Construct ValidityConstruct Validity

Does your test measure what you think itDoes your test measure what you think it

really measures. What are you measuringreally measures. What are you measuring

or observing?or observing?

Keep it simple and account for it.Keep it simple and account for it.


9/16

CONSTRUCT VALIDITYCONSTRUCT VALIDITY

Construct validity is about the correspondenceConstruct validity is about the correspondencebetween your concepts (constructs) and the actualbetween your concepts (constructs) and the actualmeasurements that you use.measurements that you use.A measure with highA measure with high

concept validity accurately reflects the abstract conceptconcept validity accurately reflects the abstract conceptthat you are trying to study. Since we can only knowthat you are trying to study. Since we can only knowabout our concepts through the concrete measures thatabout our concepts through the concrete measures thatwe use, you can see that construct validity is extremelywe use, you can see that construct validity is extremelyimportant. It also becomes clear why it is so important toimportant. It also becomes clear why it is so important to

have very clear conceptual definitions of our variables.have very clear conceptual definitions of our variables.


10/16

Construct ValidityConstruct Validity

Construct validity is often establishedConstruct validity is often established

through the use ofthrough the use ofa multi-trait, multi-methoda multi-trait, multi-method

matrix.matrix.At least two constructs are measured.At least two constructs are measured.

Each construct is measured at least two differentEach construct is measured at least two differentways, and the type of measures is repeatedways, and the type of measures is repeated

across constructs. For example, each constructacross constructs. For example, each construct

first might be measured using a questionnaire,first might be measured using a questionnaire,

then each construct would be measured using athen each construct would be measured using asimilar set of behavioral observation categories.similar set of behavioral observation categories.


11/16

RELIABILITYRELIABILITY

In order to make any kind of causalIn order to make any kind of causal

assessments in your research situation,assessments in your research situation,

you must first haveyou must first have reliablereliable measures,measures,

i.e., stable and/or repeatable measures. Ifi.e., stable and/or repeatable measures. If

the random error variation in yourthe random error variation in your

measurements is so large that there ismeasurements is so large that there is

almost no stability in your measures, youalmost no stability in your measures, youcan't explain anything!can't explain anything!


12/16

ReliabilityReliability

Researcher Bias-How might you haveResearcher Bias-How might you have

effected the study? Account for it!effected the study? Account for it!


13/16


Reliability is required to make statements aboutReliability is required to make statements aboutvalidity. However, reliable measures could bevalidity. However, reliable measures could bebiasedbiased and hence "untrue" measures of aand hence "untrue" measures of aphenomenon)phenomenon) or confoundedor confounded with other factorswith other factorssuch as acquiescence response set. Picture asuch as acquiescence response set. Picture ascale that always weighs five pounds too light.scale that always weighs five pounds too light.The results are reliable, but inaccurate orThe results are reliable, but inaccurate orbiased. Or, picture an intelligence test on whichbiased. Or, picture an intelligence test on which

women or people of color always score lowerwomen or people of color always score lower((even if this doesn't occur on other testeven if this doesn't occur on other tests). Again,s). Again,the measure may be reliable but biased.the measure may be reliable but biased.


14/16


Identifying and controlling threats toIdentifying and controlling threats to

reliability and validity in qualitativereliability and validity in qualitative

research is critical. Asking the question ofresearch is critical. Asking the question of

whether another researcher, going into thewhether another researcher, going into the

field and asking the same interviewfield and asking the same interview

questions would get the same answers isquestions would get the same answers is

the overarching premise of reliability.the overarching premise of reliability.


15/16

GeneralizabilityGeneralizability

Can you apply your findings across similarCan you apply your findings across similar

populations? If so how and whatpopulations? If so how and what

boundaries exist?boundaries exist?


16/16

Generalizability (Be Honest)Generalizability (Be Honest)

The study is generalizable to other school headsThe study is generalizable to other school headsfrom international schools in Asia. However, onefrom international schools in Asia. However, onelimit to this study was that the study will belimit to this study was that the study will beconducted with schools only in Asia. Therefore,conducted with schools only in Asia. Therefore,

the findings will not apply universally. A secondthe findings will not apply universally. A secondlimit to the study is that the entire study waslimit to the study is that the entire study wasfunded by the researcher, with limited funds forfunded by the researcher, with limited funds forair travel and miscellaneous expenses.air travel and miscellaneous expenses.Therefore, schools were selected within a fourTherefore, schools were selected within a four

hour flight range of the researchers home town.hour flight range of the researchers home town.Telephone interviews were deemed less reliableTelephone interviews were deemed less reliableas they did allow for interpretation of bodyas they did allow for interpretation of bodylanguage and visual clues.language and visual clues.

thursday validity, reliability and generalizabilityupdated

Documents