chapter 2 personality assessment, measurement, and research design © 2015 m. guthrie yarwood

CHAPTER 2

Personality Assessment, Measurement, andResearch Design

Outline

I. 4 Sources of personality data

II. Reliability / Validity in Personality

III. A little more on observer reports

I. Sources of Personality Data

Self-Report Data (S-Data)Observer-Report Data (O-Data) Test-Data (T-Data) Life-Outcome Data (L-Data)

Self-Report Data (S-Data)

Information provided by a person, such as through a survey or interview

Limitations of S-data?

O-Data

Information provided by someone else about another person Professional personality assessors People who actually know the target person

Naturalistic vs. Artificial Observation

Limitations?

Test-Data (T-Data)

Information provided by standardized tests or testing situations

Situation designed to elicit behaviors that serve as indicators of personality

Test-Data Creativity Example

What are unusual uses for common objects – bricks, knives, newspapers?

Answers to hypothetical events What would happen if people went blind? What would happen is people shrank to 12 inches tall?

(Paul Silvia’s work)

Test-Data: Other Examples

Mechanical recording devices, e.g., “Actometer” used to assess children’s activity

Physiological data

Projective Tests

Ex: Fairy Tale Test

Limitations?

Life-Outcome Data (L-Data)

Information that can be gleaned from events, activities, and outcomes in a person’s life that is available for public scrutiny—e.g., marriage, speeding tickets

Can serve as important source of “real life” information about personality

Ex: implicit egotism: people gravitate toward people, places, things that resemble the self

(Pelham and colleagues’ work)

Issues in Personality Assessment

Links among different data sourcesFallibility of personality measurement

All sources of data have limitations Results that replicate through “triangulation” are

most powerful

You are a personality psychologist and would like to measure the personality trait risk-taking (i.e., sensation seeking).

How could you measure risk-taking using each of the four data sources?S-DataO-DataT-DataL-Data

Evaluation of Personality Measures How do we know whether our scale is a “good scale?”

Types of ErrorsReliability ValidityThreats to Reliability and Validity

Extraneous vs. Confounding Variables

Extraneous variables are variables that may compete with the independent variable in explaining the outcome of a study.

Confounding variable: an extraneous variable that does indeed influence the dependent variable.

A confounding variable systematically varies or influences the independent variable and also influences the dependent variable.

Random vs. Constant Errors

Random Errors (unsystematic): extraneous variables whose average influence on the outcome is the same in both (or all) conditions (Aronson et al., 1990) Affects reliability AND validity

Constant (systematic) Errors: influences all the scores in one condition in the same direction and has no effect or a different effect on the other condition. Affects ONLY validity Confounding variables

Clarification Check!

IV: Type of show Condition 1: Extraverts watch comedy

show Condition 2: Extraverts watch neutral

DV: Self-reported emotions Extraverts report more positive emotions

in Comedy Condition than in Neutral Condition

Clarification Check! – Random Errors

Extraverts in conditions 1 and 2 are placed in very hot rooms. The hot temperature lowers their self-reported levels of positive emotions.

But, we still find that extraverts report more positive emotions when watching the comedy show than the neutral show.

Accurate: We find differences in self-reported positive emotions (on 5-point scale): Comedy Cond. = 4.8, Neutral Cond. = 2.5

Random Errors: In presence of hot temperature, we find differences in self-reported positive emotions Comedy Cond. = 3.0, Neutral Cond. = 1.5

Clarification Check! – Random Errors

Extraverts in conditions 1 and 2 are placed in very hot rooms. The hot temperature lowers their self-reported levels of positive emotions.

But, we still find that extraverts report more positive emotions when watching the comedy show than the neutral show.

Accurate: We find differences in self-reported positive emotions (on 5-point scale): Comedy Cond. 1 = 4.8, Neutral Cond. 2 = 2.5

Random Errors: In presence of hot temperature, we find differences in self-reported positive emotions Comedy Cond = 3.0, Neutral Cond.2 = 1.5

Hot temperature lowers the scores in both

conditions!!

Clarification Check! – Constant Errors

Extraverts in Comedy Cond. are in a 70◦ room. Extraverts in Neutral Cond. are placed in a 78◦ room. In Neutral Cond. only, the hot temperature lowers self-reported levels of positive emotions.

We find that the comedy show results in more positive emotions than the neutral show.

Accurate: We do not find differences in self-reported positive emotions Comedy Cond. = 4.8, Neutral Cond. = 4.6

Constant Error: We find differences in positive emotions due to the hot temperature, not due to the manipulation! Comedy Cond. = 4.8, Neutral Cond. = 2.5

Clarification Check! – Constant Errors

Extraverts in Comedy Cond. are in a 70◦ room. Extraverts in Neutral Cond. are placed in a 78◦ room. In Neutral Cond. only, the hot temperature lowers self-reported levels of positive emotions.

We find that the comedy show results in more positive emotions than the neutral show.

Accurate: We do not find differences in self-reported positive emotions Comedy Cond. = 4.8, Neutral Cond. = 4.6

Constant Error: We find differences in positive emotions due to the hot temperature, not due to the manipulation! Comedy Cond. = 4.8, Neutral Cond. = 2.5

Hot temperature lowers the scores in Condition 2, but not Condition 1!!

Reliability

The ability of a test to measure an attribute consistently Does this extraversion scale measure the true

level of each participant’s extraversion (over time)?

Are coders following the same method?

To achieve reliability we want to reduce random error.

Threats to Reliability (random error)

Participant Characteristics: fatigue, motivation, boredom

Testing situation: time of day, room temperature

Testing Instrument: instructions, rating scale, items, reading level.

Experimenter Characteristics and Errors: interactions with participants; incorrect observations

of participants; incorrect coding of behavior

Estimating Reliability

Test-Retest Coefficient

Parallel-Forms Coefficient

Internal Consistency Coefficient

Interrater (interobserver) Reliability

Validity

Validity: The degree to which a test or measurement accurately measures or reflects what it claims to measure.

Internal Validity: Did the experimental treatments make a difference in this specific experimental instance?

External Validity: Generalizability; To what populations, settings, treatment variables, and measurement variables can the effect be generalized? **Never completely answerable

Internal Validity

Degree to which test measures what it claims to measure

5 types of internal validity Face validity Predictive or criterion validity Convergent validity Discriminant validity Construct validity

Threats to Internal Validity (errors)

Affected by random and constant errors

Random (unsystematic) Errors: same errors that affect reliability

Constant Errors (systematic): errors that affect measurement of variable; does not affect reliability

Knowledge Check!You are conducting a study on the personality traits associated with the frequency of exercising.

For your study, which of the following poses a threat to validity, but not reliability? A. All the participants are bored.B. The construction outside the laboratory

window is very loud.C. You recruited participants from the Rec Hall.D. The experimenter who greets all

participants is very rude.

Knowledge Check!You are conducting a study on the personality traits associated with the frequency of exercising.

For your study, which of the following poses a threat to validity, but not reliability? A. All the participants are bored.B. The construction outside the laboratory

window is very loud.C. You recruited participants from the Rec

Hall.D. The experimenter who greets all participants

is very rude.

Threats to Internal Validity – MRS SMITH

MaturationRegression to the MeanSelection of SubjectsSelection by Maturation InteractionMortalityInstrumentationTestingHistory

Regression to the Mean Example

In an experiment involving reading instruction, subjects grouped because of poor pre-test reading scores show considerably greater gain than do the groups who scored average and high on the pre-test.

Poor Average / Mean

HighPre-test

Poor Average / Mean

HighPre-test

After reading Instruction

Selection by Maturation Interaction

Group Pre-test Post-Test (after head start)

Head Start Intervention

Middle-class children 65/100

Disadvantaged children

65/100

Control Group – No Intervention

Middle-class children 65/100

Disadvantaged children

65/100

Group Pre-test

Post-Test: 6 months

Post-Test: 12 months

65/100

70/100 75/100 80/100

65/100

65/100 65/100 65/100

Group Pre-test

Post-Test: 6 months

65/100

70/100 75/100 80/100

65/100

65/100 65/100 65/100

Ss in Intervention are middle class, while Ss in control group are disadvantaged. Over time, Intervention Ss show

improvement of post-test due to better health care, greater parental support, greater access to resources, etc.

Identify the Threat to Validity!

In a short experiment designed to investigate the effect of computer-based instruction, Ss missed some instruction because of a power failure at school.

A. HistoryB. Mortality C. TestingD. Instrumentation

In a short experiment designed to investigate the effect of computer-based instruction, Ss missed some instruction because of a power failure at school.

A. HistoryB. Mortality C. TestingD. Instrumentation

In a health experiment designed to determine the effect of various exercises, those Ss who find the exercise most difficult stop participating.

A. Selection of SubjectsB. Mortality C. TestingD. Maturation

In a health experiment designed to determine the effect of various exercises, those Ss who find the exercise most difficult stop participating.

A. Selection of SubjectsB. Mortality C. TestingD. Maturation

Estimating Internal Validity

Content ValidityCriterion-Related Validity

Concurrent Validity Predictive Validity

Construct Validity Convergent Validity Discriminant Validity

Content Validity

Definition: whether the content of a test elicits a range of responses that are representative of the entire domain or universe of skills, understandings, and other behaviors a test is designed to measure.

To assess: compare tests’ content with an outline of specifications concerning subject matter to be covered in test.

Openness to Experience

Behaviors?

Perceptions?

Thoughts/Cognitions?

Feelings/Emotions?

Criterion-Related Validity

Definition: procedures in which the test scores of a group of people are compared with ratings, classifications, or other measures of performance.

Concurrent Validity – A type of criterion-related validity

Concurrent Validity: when a test is administered to people in various categories, to determine whether test scores of people in 1 category are significantly different from people in other categories. Clinical vs. non-clinical group Different socioeconomic levels

Group A shows more openness to experience

Group B shows less openness to experience

Concurrent Validity

High IQ group shows more openness

to experience

Low IQ group shows less openness to

experience

Concurrent Validity

Predictive Validity – A type of criterion-related validity

Predictive Validity: how accurately test scores predict criterion scores.

Indicated by correlation between test score (the predictor) and a criterion of future performance (what the test predicts)

Predictive Validity

(Predictor)

(Criterion)

Predictive Validity

College Major

Well-being, Psychological Adjustment

Construct-Related Validity

Definition: extent to which scale measures a particular construct or psychological concept

To assess: need to determine whether an assessment instrument that presumably measures a certain personality variable is actually doing so

Convergent Validity – A Type of Construct-Related Validity

Convergent Validity: the measure has high correlations with other measures or methods of measuring the same construct

Convergent Validity

Our Openness to Experience

Self-Report Scale

Same Construct, Same Measurement

Self-Report Scale

Same Construct,Different Measurement

Convergent Validity

Our Openness to Experience Self-

Report Scale

Costa & McCrae’s Self-Report OE

Dimension

Observer-Report OE Dimension

Our Openness to Experience Self-

Report Scale

Discriminant Validity – A Type of Construct-Related Validity

Discriminant Validity: the measure has low correlations with measures of different constructs

Discriminant Validity

Self-Report Scale

Different Construct, Same Measurement

Different Construct Different Measurement

Self-Report Scale

Self-Report Sensation Seeking

Clinical Diagnosis of Schizotypal

PersonalityDisorder

Self-Report Scale

Self-Report Sensation Seeking

Clinical Diagnosis of Schizotypal

PersonalityDisorder

Self-Report Scale

These correlations should be significant,

but lower than correlations for

convergent validity.

To assess convergent/discriminant validity

The same construct using the same method (c)The same construct using different methods

(c)Different constructs using the same method

(d)Different constructs using different methods

Note: c = convergent; d=discriminant

Reliability & Validity

A test can be reliable, but not valid

In other words, to be valid a test must first be reliable.

NO VALDITY WITHOUT RELIABILITY!

Reliability Internal Validity

Test-Retest Coefficient

Parallel-Forms Coefficient

Internal Consistency Coefficient

Interrater (interobserver) Reliability

Content ValidityCriterion-Related

Validity Concurrent Validity Predictive Validity

Construct Validity Convergent Validity Discriminant Validity

Reliability and Validity: Summary

A LITTLE MORE ON OBSERVER REPORTS (O-DATA)

•Self-report

•Observer report 1

Correlation 1

Correlation 2

•Self-report

•Several different observers

Correlation 3, 4, 5…

Aggregating Scores

Averaging the self-report and the observer report/s provides a clearer picture of personality than the self or observer report alone (Kolar et al., 1996)

Let’s discuss the purpose of observer reports

What is the purpose of an observer report?Compared to a self-report measure, what

results would you expect from an observer report?

What would you think if you did not obtain these results?

Can you think of any threats to validity that observer reports may pose?

Aggregating Scores

A positive correlation between an observer report and a self-report, would provide evidence for which type of validity?

A. ConvergentB. PredictiveC. ConcurrentD. Discriminant

Recall Reflection Post and TIPI!

Disagree

Strongly

Disagree

Moderately

Disagree a

little

Neither

Agree nor

Disagree

Agree a

little

Moderately

Strongly

1 Extraverted,

enthusiastic

2 Reserved, quiet r

3 Critical,

quarrelsomer

4 Sympathetic,

5 Dependable,

self-disciplined

6 Disorganized,

carelessr

7 Anxious, easily

upset r

8 Calm,

emotionally

stable

9 Open to new

experiences,

complex

10 Conventional,

uncreativer

Self Scores!

ablene

ess to

erienc

5.135.54

TIPI Dimensions

Observer Scores!

ablene

ess to

erienc

4.294.92 5.13

TIPI Scores

Based on the following graphs, which dimension do you think shows the weakest correlation between self and observer scores?

A. ExtraversionB. AgreeablenessC. ConscientiousnessD. Emotional StabilityE. Openness to Experience

1 2 3 4 5 6 71

SELF SCORE

1 2 3 4 5 6 71234567

Emotional StabilityLOW NEUROTICISM

SELF SCORE

1 2 3 4 5 6 71

Conscientiousness

SELF SCORE

1 2 3 4 5 6 71

AGREEABLENEESS

SELF SCOREOB

1 2 3 4 5 6 71234567

EXTRAVERSION

SELF SCORE

Based on the following graphs, which dimension do you think shows the weakest correlation between self and observer scores?

A. ExtraversionB. AgreeablenessC. ConscientiousnessD. Emotional StabilityE. Openness to Experience

LET’S CORRELATE THE SCORES!

TIPI DIMENSION TIPI TRAIT r

1EXTRAVERSION

Extraverted, enthusiastic

.742 Reserved, quiet r

3AGREEABLENESS

Critical, quarrelsomer

.484 Sympathetic, warm

CONSCIENTIOUSNESSDependable, self-disciplined

.526 Disorganized, carelessr

7EMOTIONAL STABILITY

(LOW NEUROTICISM)

Anxious, easily upset r

.078 Calm, emotionally stable

9 OPENNESS TO

EXPERIENCE

Open to new experiences,

complex .3910 Conventional, uncreativer

Note. Red = Self; Yellow = Observer

1 2 3 4 5 6 7 8 9 10 11 121

EXTRAVERSION

STUDENT

1 2 3 4 5 6 7 8 9 10 11 121

AGREEABLENESS

STUDENT

1 2 3 4 5 6 7 8 9 10 11 121

CONSCIENTIOUSNESS

STUDENT

1 2 3 4 5 6 7 8 9 10 11 121

EMOTIONAL STABILITY(LOW NEUROTICISM)

STUDENT

1 2 3 4 5 6 7 8 9 10 11 121

OPENNESS TO EXPERIENCE

STUDENT

Note. Red = Self; Yellow = Observer

1 2 3 4 5 6 7 8 9 10 11 121

EXTRAVERSION

STUDENT

1 2 3 4 5 6 7 8 9 10 11 121

AGREEABLENESS

STUDENT

1 2 3 4 5 6 7 8 9 10 11 121

CONSCIENTIOUSNESS

STUDENT

1 2 3 4 5 6 7 8 9 10 11 121

EMOTIONAL STABILITY(LOW NEUROTICISM)

STUDENT

1 2 3 4 5 6 7 8 9 10 11 121

OPENNESS TO EXPERIENCE

STUDENT

Summary and Evaluation

Decisions about data source and research design depend on (1) the purpose of study and (2) threats to validity/reliability

There is no perfect data sourceThere is no perfect research designAssessing threats to reliability and validity will

assist in selecting a data source and research design.

Observer reports improve validity and reduce social desirability concerns

Reminder – Paper Topic!!

Find a group member (can do on ANGEL!)Select a topic!Submit Paper topic to drop-box

Due: Friday, January 30th at 9 AM ETPaper Topic Outline available on ANGEL.Guidance: Meet with Michelle or Celina

Questions???

Chapter 3 (Dispositional Domain) Survey

HEXACO Follow link to access survey Score your survey according to the instructions on

Big Five Model Access through Chapter 3 Survey on course website

chapter 2 personality assessment, measurement, and research design © 2015 m. guthrie yarwood

Documents

chapter 7 physiological approaches to personality © m....

chapter 3 culture © 2015 m. guthrie yarwood. outline 3 ways...

review guthrie

disorders of personality chapter 19 1 © 2015 m. guthrie...

e motion, body, and brain (c) 2015 m. guthrie yarwood

informing the public about immunisation by jo yarwood -...

chapter 6 genetics and personality click to edit master...

guthrie - ecf 15 - 2013-04-02 - guthrie motion for...

george guthrie presentation

woody guthrie essay

ralph morris, bonyoung koo, jeremiah, johnson and greg...

guthrie general inc

greg yarwood, sue kemball-cook and jeremiah johnson ...

capri - yarwood leather

guthrie cemetary lots - guthrie...

guthrie hospital case study

guthrie vol 1 thesis

chapter 1 introduction to personality psychology © 2015 m....

guthrie county newsletter

technical information pack - yarwood leather