© 2012 by W. W. Norton & Company

CHAPTER 5

Identifying Good Measurement

Detailed Learning Objectives1. Recognize the difference between a conceptual variable and its operationalization.2. List three ways psychologists typically operationalize variables: self-report, observational, and physiological. 3. Classify variable scales as categorical or quantitative.4. Describe the difference between the validity and the reliability of a measure.5. Identify three types of reliability (test-retest, interrater, and internal), and know when each type is relevant.

Detailed Learning Objectives6. Review scatterplots, focusing on how scatterplots show the direction and strength of a relationship.7. Apply the correlation coefficient, r, as a way to describe the direction and strength of a relationship. (In this chapter, r is relevant as a common statistic to describe reliability and validity.) 8. Identify face and content validity.9. Identify predictive, concurrent, convergent, and discriminant validity.10. Describe how scatterplots, r, and known groups can be used to evaluate predictive, concurrent, convergent, and discriminant validity.

Ways to Measure Variables

Ways to Measure Variables

• Conceptual and operational variables• Three common types of measures• Scales of measurement

Constructs and Operationalizations

Three types of operationalization– Self-report– Observational– Physiological

5-item scale

Well-being

No. of smiles

Brain scan

Construct

Operationalization

Scales of Measurement

• Categorical • Quantitative– Ordinal (meaningful values but unequal intervals

between units)– Interval (equal intervals between units but no

meaningful zero)– Ratio (equal intervals and a meaningful zero)

The claim: “College students are getting more narcissistic.”

Discussion starter

NPI Example Items: Forced-Choice Format

(Ames, Rose, & Cameron, 2006)Narcissistic responseI know that I am good because everybody keeps telling me so.

I like to be the center of attention.

I think I am a special person.

I insist upon getting the respect that is due me.

Everybody likes to hear my stories.

I am going to be a great person.

Non-narcissistic responseWhen people compliment me I sometimes get embarrassed.

I prefer to blend in with the crowd.

I am no better nor no worse than most people.

I usually get the respect that I deserve.

Sometimes I tell good stories.

I hope I am going to be successful.

Reliability of Measurement

Reliability of Measurement

• Three types of reliability– Test-retest– Interrater– Internal

• Using a scatterplot to evaluate reliability• Using the correlation coefficient r to evaluate

reliability

Reliability of Measurement

Reliability of Measurement

When is each kind of reliability necessary?

Why is reliability an empirical question?

What does reliability tell us?

Test-retest reliabilityInterrater reliabilityInternal reliability

Test-Retest Reliability: Consistent scores every time we test

Interrater Reliability: Consistent scores no matter who is rating

Interrater Reliability ExampleDemo of interrater reliabilityPair up:Three kids 0:38 to 2:02

Girl in pinkGirl in yellowGirl in blue

Two people in each group count: How many times does she look away from the teacher? How many times does she clap?How many times does she put her hands in her lap?

Internal Reliability: Consistent scores no matter how you ask

Internal Reliability

Internal reliability (not to be confused with internal validity!)

The extent to which multiple measures, or items, are all answered the same by the same set of people.

Cronbach’s alpha: An average of all of the possible item-total correlations.

I know that I am good because everybody keeps telling me so.

I like to be the center of attention.

I think I am a special person.

I insist upon getting the respect that is due me.

Everybody likes to hear my stories.

I am going to be a great person.

When people compliment me I sometimes get embarrassed.

I prefer to blend in with the crowd.

I am no better nor no worse than most people.

I usually get the respect that I deserve.

Sometimes I tell good stories.

I hope I am going to be successful.

Narcissistic response Non-narcissistic response

“The NPI-16 had an alpha of .72, while the full 40-item measure revealed an alpha of .84” (Ames et al., 2006, p. 442).

Validity of Measurement

Validity of Measurement

• Measurement validity of abstract constructs• Face validity and content validity• Predictive validity and concurrent validity• Convergent validity and discriminant validity• Relationship between reliability and validity

Validity of Measurement

Validity of Measurement

Subjective forms

Empirically derived forms

Face and Content Validity

Face validity: Does it look like a good measure? (often assessed by asking experts)

Content validity: Does it include all the important components of the construct?

Correlation method

Predictive and Concurrent Validity

Known groups method

Predictive and Concurrent Validity

Known groups method

Predictive and Concurrent Validity

Convergent and Discriminant Validity

Convergent and Discriminant Validity

Homework: Reliability

What kind(s) of reliability would

need to be evaluated?

Draw a

scatterplot or describe a result

that would indicate that the

measure has good reliability and one that shows it has

poor reliability.

1. Researchers place unobtrusive video recording devices in the living rooms of 20 children. Later, coders view tapes of the living areas and code how many minutes each child spends playing video games.2. Clinical psychologists have developed a 7-item self-report measure to quickly identify people who are at risk for panic disorder. 3. A restaurant owner uses a response card with four items in order to evaluate how satisfied customers are with the food, service, ambience, and overall experience.

Homework: Validity

How might you show that this measure has predictive

validity?

How might you show that this measure

has convergent

and discriminant

validity?

4. Clinical psychologists have developed a 7-item self-report measure to quickly identify people who are at risk for panic disorder.

5. A restaurant owner uses a response card to evaluate how satisfied customers are with the food. It contains one item, “Please rate the quality of the food:” on a scale from 1 (very dissatisfied) to 4 (very satisfied).

Relationship Between Reliability and ValidityCan a measure be reliable but not valid?Examples:

• Shoe size as an intelligence test (reliable, not valid)• Number of children you have as a measure of interest in

children (reliably measured, but correlated with interest?)Can a measure be valid but not reliable? (No)

Reliable and valid Reliable but not validNot reliable and not valid

Reliability Is Necessary, But Not Sufficient for, Validity

Interrogating Construct Validity as a Consumer

Interrogating Construct Validity as a Consumer

• Diener’s measure of happiness

• Gallup poll’s measure of happiness

Ames et al. (2006, p. 444)

What kind of validity are these correlations supporting?

Correlations of the 16-item NPI with:

40-item NPI: .90**Extraversion: .36**Agreeableness: -.23**Self-esteem: .30**Belief in a just world: .04

Reliability in articles

“The NPI-16 had an alpha of .72, while the full 40-item measure revealed an alpha of .84” (Ames et al., 2006, p. 442).