Representing Situations in Assessment –

extracting better value from our investment

Division of Occupational Psychology

Annual Conference 2011

Michael Burnett & Almuth McDowall

• The Problem with Situations

• Disentangling Situational Influences:

• Part 1: Predominance of situational effects in high stakes

Assessment Centres

• Part 2: Understanding trait v. situational effects in low stakes

Situational Judgement Tests

• Part 3: How cognitive Situational Models address sources of

situational effects

• Implications

Overview

•Techniques used today range from ‘context free’ cognitive ability

and personality measures to simulations (ACs and SJTs) of work

or organisational situations

•The first problem is that simulations give significant (ACs .281,

SJTs .18-.382) but in absolute terms low additional incremental

validity over ‘context free’ measures

• The second problem is explaining what simulations actually

measure – e.g. the methods effect in ACs3, construct validity of

SJTs4

• The implication is that applied psychology needs to provide a

better account of individual differences in dealing with situations *

1AC Criterion Validity –Harmelin et. al. (2007) 2SJT Criterion Validity - Christian et. al. (2010) 3Sackett & Dreher (1982) 4McDaniel & Whetzel (2005)

The Problem with Situations

Multimedia based simulations will happen

• Multimedia-based ‘simulations’ in assessment will grow quickly

• Video-based, Avatar-based simulations (Second LifeTM), 2-D

Animations, Apps (i-phone/ipad/mobile), Games

• Occ. Psych. Design Methodology needs to adapt

• Maintain control over what is being measured!

Model x2 df GFI AGFI RMSEA NNFI SRMR

Study 1a

Traits Only

(CT0E)

3034.6 67 0.69 0.51 0.22 0.53 0.17

Exercises Only

(0TCE)

96.6 57 0.99 0.97 0.027 0.99 0.014

Correlated Traits

& Exercises

(CTCE)

208.05 59 0.97 0.95 0.052 0.98 0.047

Correlated

Exercises &

Traits (CECT)*

62.45 44 0.99 0.98 0.021 1.0 0.013

Exercises Measures

T

r

a

i

t

s

Part 1: The predominance

of the situation

Structural Equation Models of assessment centre results based on

situations (exercises) provide a good fit to the data, trait-only

(competency dimensions) models do not fit

The best fit combines situations moderated by trait (competency

dimensions) factors

Adding in Cognitive ability and Big 5 Personality dimensions

showed best fit when added to the Exercise model, cognitive ability

predominated

Models that did not fit included Exercise Type model and Trait

Activation model (Lievens et. al., 2006)

Situational effects in high stakes simulations

• Use of low fidelity simulation based on SJT method to explore

comparative effect of situational and ‘trait’ effects

•Academic situations used with samples drawn from University

and job applicants seeking SJT practise (n. 339)

• Situational influence assessed by examining within-situation item

relationships (zero correlation expected if competency/traits

determine performance)

• Competency and personality trait influence assessed by

examining across-situation scale relationships between SJT items

and independent measures of competencies & traits

Part 2: Exploring situational effects

Item 1 Item 2 Item 3 Item 4

Plan &

Organise

Relate &

Network Analysis

Persuade &

Influence

Conscientiousness Openness Agreeableness Extraversion

Item 1 Item 2 Item 3 Item 4

Situation

1

Situation

2

Design of Situational Judgement Items

You have been selected to present a lecture to your course in

front of the head of Department and senior tutors

1. You set yourself objectives for what you are going to say to

include new and innovative ideas (Planning_Open)

2. You try and make sure that the content of your lecture will appeal

to both students and staff (Relating_Agreeable)

3. You aim to keep the audience captivated by questioning

established assumptions (Analysis_Extravert)

4. You check your material and rehearse the presentation to ensure

you appear credible (Persuading_Conscientious)

•

Situational Judgement Item

Plan &

Organise

Relate &

Network Analysis

Persuade &

Influence

Conscientiousness Openness Agreeableness Extraversion

Plan &

Organise

Relate &

Network Analysis

Persuade &

Influence

Conscientiousness Openness Agreeableness Extraversion

.36 .33 .24 .10

.22 .31 .05 .13

Likert what I ‘would do’ responses

Average

Situation .12

(.06 to.20)

• Situations are multidimensional - consider effects on encoding & retrieval:

(Zwaan & Radvansky, 1998; Magliano et.al. 2008):

- Space – context, layout and movement

- Time – chronology of events, temporal shifts

- Protagonists - Actors & Objects

- Intention – Goals & Motivations

- Causation – inferences of what has caused events or what could

happen next

•

Part 3: Understanding situational effects

Goals 1. Space

2. Time

Manager3

Participant

as

Protagonist3

Supplier3

Goals & motivations4

Goals & motivations4

Goals & motivations4

Assumed ‘agreement’ becomes disagreement

‘Non-performing’ supplier becomes threat to supply chain

Manager’s expectation becomes “use of initiative”

Causality5

Assumption

about power

relationship5

Assumption

about power

relationship5

The Situational Judgement determines these…………..

Cognitive Model of Situation

3. Actors 4. Motivations 5. Causes

Goals Space

Time

Tutor

Protagonist

Student

Goals & Motivations

Goals & Motivations

Goals & Motivations

Influence on future evaluation

Influence on future relationships

A

B

C

D

E

F

G

Objective

characteristics

of Situation

a

b

c

d

e

f

g

Situation

as construed

& encoded

I

II

III

IV

V

VI

VII

Potential

behaviours

afforded

i

ii

iii

iv

v

vi

vii

Actual

behaviour

enacted

Adapted from Reis, 2008

Person Variables e.g. goals, domain knowledge

competences, personality traits, state

Personality & Competency effects on

encoding & judgements

•Performance in Simulation-based assessments is determined by

both situational and dispositional (trait & competency) factors

•AC feedback should reflect both aspects

•SJTs can be structured to return required facets

•Situational Models hold promise in improving the yield from

simulations (if we change the paradigm)

•assessing encoding as well as judgement to widen the range

of information gathered from assessments

•providing a systematic way of describing situations to

understand how these affect behaviour (IRT correlates)

•enabling greater control over how situations are used in

simulations including use of multimedia & gaming techniques

(impact on Occ Psych methodology)

Implications