Construct ValidityAnd its Threats

Jill HoxmeierH615: Advanced Research Design

October 10, 2013

“Boy, a few more like that and I’ll be ready for Gamblers Anonymous”

Twin Problems of Construct Validity

Construct Validity: making inferences from the sampling particulars of a study to the higher-order constructs they represent• Understanding them • Assessing them

“Thinking without the positing of categories and concepts in general would be as

impossible as breathing in a vacuum” (Einstein, 1949)

Meehl & Cronbach

Construct Validity in Psychological Tests• Three steps to evaluate construct validity:• Articulation of a set of theoretical constructs and their

interrelations• Developing ways to measure the hypothetical constructs

proposed by a theory• Empirically testing the hypothesized relations

Cronbach & Meehl, 1955

• Evolution of “Attitude”• From a loose variable comprised of beliefs, norms, intentions… • … to Fishbein-ian perspective that attitude = person’s evaluation

of behavioral consequence x strength of belief that behavior will lead to those consequences

Why do we need valid constructs?• Research cannot be done without constructs • Constructs are the central means we have for connecting the

operations used in an experiment to theory• Construct labels carry social, political, and economic

implications• The creation and defense of basic constructs is a fundamental

task of all science

Construct Validity• Fostered by:• Starting with a clear explication of the person, setting, treatment,

and outcome constructs of interest• Carefully selecting instances that match those constructs• Assessing the match between instances and constructs for

“slippage”• Revising construct descriptions accordingly

Understanding Psychological Domains

“…there are often no natural units of measurement…”

(Shonkoff & Philllips, 2000, p.83-83)• Pattern-matching logic: does the given instance sufficiently match the prototypical features to warrant using the category label?

How do you measure your domain?

• Think about your own research interest… • What are the domains or constructs that are measured?

• Have you discovered from the literature any challenges in measuring those domains?

• What are ways investigators in your field have revised constructs to overcome those challenges?

Program of Research• A theory of constructs must emphasize • 1) operationalizing each construct several ways within

and across studies• 2) probing the pattern match between the multivariate

characteristics of instances and characteristics of the target construct• 3) acknowledge legitimate debate about the quality of

the match given the socially constructed nature of both operations and constructs

Threats to Construct Validity• Inadequate explication of constructs• Construct confounding • Mono-operation bias• Mono-method bias• Confounding constructs with levels of constructs• Treatment sensitive factorial structure• Reactive self-report changes • Reactivity to the experimental situation• Experimenter expectancies • Novelty and disruptive effects • Compensatory rivalry • Resentful demoralization• Treatment diffusion

What (potential) threats are most relevant to your research?And how do you plan to guard against such threats?

Questions for You!• What role does qualitative research play in contributing to

construct validity?

• How do we balance the theory construct testing of quantitative research and the “naturalistic generalizations” that may come from readers of “thick descriptions” that qualitative researchers provide?