all about performance innovative methods of selecting new employees: simulations and in-baskets...

All About Performance www.allaboutperformance.biz

Innovative Methods of Selecting Innovative Methods of Selecting New Employees: Simulations and New Employees: Simulations and

In-Baskets In-Baskets Warren Bobrow, Ph.D.Warren Bobrow, Ph.D.

All About Performance, LLCAll About Performance, [email protected]@allaboutperformance.biz

(310) 670-4175(310) 670-4175


Some History In Personnel Some History In Personnel SelectionSelection

• Personality Now Of Great Personality Now Of Great Interest to I/O PsychologistsInterest to I/O Psychologists

• Performance Testing Has a Long Performance Testing Has a Long History in I/O PsychologyHistory in I/O Psychology

• Computer-Based Tests Are Computer-Based Tests Are Increasingly PopularIncreasingly Popular


Validity of Various Selection Validity of Various Selection ProceduresProcedures

(Schmidt & Hunter, 1998)(Schmidt & Hunter, 1998)

Personnel measure Validity (r) Personnel measure Validity (r)

Work Sample Tests .54 Biographical Data .35

Cognitive Ability (IQ) .51 Conscientiousness Tests .31

Interviews (structured) .51 Reference Checks .26

Peer Ratings .49 Job Experience (years) .18

Job Knowledge Tests .48 Education (years) .10

Integrity Tests .41 Interests .10

Job Tryout Procedures .44 Graphology (handwriting) .02

Interviews (unstructured) .38 Age -.01


What We’ll Be What We’ll Be AddressingAddressing

• The Criterion Validity of The Criterion Validity of Various Innovative Various Innovative Procedures.Procedures.

• How Streamlining How Streamlining Assessments Can Increase Assessments Can Increase ROI and Usability.ROI and Usability.


Comparison of Validity Comparison of Validity Between Live and Computer-Between Live and Computer-

Based SimulationsBased Simulations• Many organizations are looking Many organizations are looking

to leverage computers in to leverage computers in selectionselection– More efficiencyMore efficiency– More accurate data gatheringMore accurate data gathering– Projects a more positive image with Projects a more positive image with

applicantsapplicants

• Are they getting good value?Are they getting good value?


Why is This Important?Why is This Important?

• If Computer-Based Simulations If Computer-Based Simulations Are As Effective as Live Ones, Are As Effective as Live Ones, Then:Then:

– Objective Way To Measure Skill Objective Way To Measure Skill and Abilityand Ability

– Cost Savings Can Be RealizedCost Savings Can Be Realized


Live and Computer Based Live and Computer Based SimulationsSimulations

• Live Simulations Have Been Shown To Live Simulations Have Been Shown To Be Valid Predictors of Job PerformanceBe Valid Predictors of Job Performance

• Computers Are Limited in Their Ability Computers Are Limited in Their Ability to Analyze Open-Ended Responsesto Analyze Open-Ended Responses

• Is the Lower Cost of Computer Is the Lower Cost of Computer Simulations Worth the (Presumed) Simulations Worth the (Presumed) Lower ValidityLower Validity


Item Writing Possibilities: Item Writing Possibilities: All Applicants Get the Same TestAll Applicants Get the Same Test

Item1

Item2


Item Writing Possibilities: Item Writing Possibilities: All Applicants Potentially Get A Unique All Applicants Potentially Get A Unique

TestTest

Item1

Item2

Item4

Item3


Item Writing Possibilities: Item Writing Possibilities: All Applicants Get Similar But Different All Applicants Get Similar But Different

TestsTests

Item1

Item2

Item3

Item4


Simulation Simulation DemonstrationsDemonstrations

• Supervisor In-BasketSupervisor In-Basket

• Supervisor CoachingSupervisor Coaching

• The CSR Computer Based The CSR Computer Based Simulation TestSimulation Test

• The CSR Live Simulation TestThe CSR Live Simulation Test


ComparisonComparison

• But Do These ‘Innovative’ But Do These ‘Innovative’ Methods work as well (and Methods work as well (and have the same ROI) as have the same ROI) as Traditional Ones?Traditional Ones?

• Same Construct Validity?Same Construct Validity?

• Worse/Equal/Better Criterion-Worse/Equal/Better Criterion-Related Validity?Related Validity?


What the CBT MeasuresWhat the CBT MeasuresAbility Call A Call B Call C

Typing Call A XTyping Call B XTyping Call C XSpeed Call A XSpeed Call B XSpeed Call C XEmpathy Call A XEmpathy Call B XEmpathy Call C XListening Call A XListening Call B XListening Call C X


Live SimulationsLive Simulations

• Designed as a Traditional Designed as a Traditional Simulation ExerciseSimulation Exercise

• Trained Role-Players/RatersTrained Role-Players/Raters

• 1 Practice and 2 Real Calls1 Practice and 2 Real Calls


Live Simulation Factor Live Simulation Factor StructureStructure

Ability Call 1 Call 2Speed Call 1 XSpeed Call 2 XProblem Solving Call 1 XProblem Solving Call 2 XEfficiency Call 1 XEfficiency Call 2 X


Validities:Validities:Computer vs. LiveComputer vs. Live

Live ComputerCost Per Applicant $70 $35Median Correlation with Productivity 0.24 0.23Median Correlation with Customer Satisfaction 0.21 0.22

Live k=3, Computer k=21


Moral to the StoryMoral to the Story

• Computer Based Simulations Have Computer Based Simulations Have the Same Construct Validity as the Same Construct Validity as Their Traditional CounterpartsTheir Traditional Counterparts

• Multiple Choice Format Does Not Multiple Choice Format Does Not Lead to a Decrement in ValidityLead to a Decrement in Validity

• Innovative Methods Provided Innovative Methods Provided Higher ROIHigher ROI


Improvements to Improvements to Computer TestsComputer Tests

•Better, More Better, More Sophisticated Branching Sophisticated Branching TechniquesTechniques

•Voice Recognition of Tone, Voice Recognition of Tone, Not Just WordsNot Just Words

all about performance innovative methods of selecting new employees: simulations and in-baskets...

Documents