marvin d. dunnette distinguished professor...marvin d. dunnette distinguished professor my...
TRANSCRIPT
![Page 1: Marvin D. Dunnette Distinguished Professor...Marvin D. Dunnette Distinguished Professor My Toothbrushes “No significant differences in plaque reductions were found between manual](https://reader030.vdocument.in/reader030/viewer/2022040514/5e6be1349a967c36df4f5ed5/html5/thumbnails/1.jpg)
Validation:We Are Doing It Wrong
Nathan KuncelMarvin D. Dunnette Distinguished Professor
![Page 2: Marvin D. Dunnette Distinguished Professor...Marvin D. Dunnette Distinguished Professor My Toothbrushes “No significant differences in plaque reductions were found between manual](https://reader030.vdocument.in/reader030/viewer/2022040514/5e6be1349a967c36df4f5ed5/html5/thumbnails/2.jpg)
My Toothbrushes
![Page 3: Marvin D. Dunnette Distinguished Professor...Marvin D. Dunnette Distinguished Professor My Toothbrushes “No significant differences in plaque reductions were found between manual](https://reader030.vdocument.in/reader030/viewer/2022040514/5e6be1349a967c36df4f5ed5/html5/thumbnails/3.jpg)
“No significant differences in plaquereductions were found between
manual brushing and any of the 2electric brushes.”
Mantokoudis, et al., 2001, p. 65
![Page 4: Marvin D. Dunnette Distinguished Professor...Marvin D. Dunnette Distinguished Professor My Toothbrushes “No significant differences in plaque reductions were found between manual](https://reader030.vdocument.in/reader030/viewer/2022040514/5e6be1349a967c36df4f5ed5/html5/thumbnails/4.jpg)
Tooth Brushing Times (Sec) for Nathan KuncelElectric versus Manual
![Page 5: Marvin D. Dunnette Distinguished Professor...Marvin D. Dunnette Distinguished Professor My Toothbrushes “No significant differences in plaque reductions were found between manual](https://reader030.vdocument.in/reader030/viewer/2022040514/5e6be1349a967c36df4f5ed5/html5/thumbnails/5.jpg)
Today’s Talk• We currently study the correlations between
predictor and criterion measures.– Answers the question: “Is there a correlation
between predictors and criteria?”– The Real question of interest: “Do better people
get hired?”• What we need to study is how assessments
affect the decision making process of:– Hiring Managers– Applicants
![Page 6: Marvin D. Dunnette Distinguished Professor...Marvin D. Dunnette Distinguished Professor My Toothbrushes “No significant differences in plaque reductions were found between manual](https://reader030.vdocument.in/reader030/viewer/2022040514/5e6be1349a967c36df4f5ed5/html5/thumbnails/6.jpg)
The Textbook Model of Validationis a Nice Safe Starting Place…
but we’renot threeanymore.
![Page 7: Marvin D. Dunnette Distinguished Professor...Marvin D. Dunnette Distinguished Professor My Toothbrushes “No significant differences in plaque reductions were found between manual](https://reader030.vdocument.in/reader030/viewer/2022040514/5e6be1349a967c36df4f5ed5/html5/thumbnails/7.jpg)
Note: I’m Guilty Too
![Page 8: Marvin D. Dunnette Distinguished Professor...Marvin D. Dunnette Distinguished Professor My Toothbrushes “No significant differences in plaque reductions were found between manual](https://reader030.vdocument.in/reader030/viewer/2022040514/5e6be1349a967c36df4f5ed5/html5/thumbnails/8.jpg)
Note: Not a Strawman Argument
“The primary inference of concern in an employmentcontext is that test scores predict subsequent workbehavior.” – Van Iddekinge & Ployhart (2008)
SchmidtHunter1998
cited3,460x
![Page 9: Marvin D. Dunnette Distinguished Professor...Marvin D. Dunnette Distinguished Professor My Toothbrushes “No significant differences in plaque reductions were found between manual](https://reader030.vdocument.in/reader030/viewer/2022040514/5e6be1349a967c36df4f5ed5/html5/thumbnails/9.jpg)
“The primary inference of concern inan employment context is that hiringsystems cause better people to behired.” – Kuncel (right now)
![Page 10: Marvin D. Dunnette Distinguished Professor...Marvin D. Dunnette Distinguished Professor My Toothbrushes “No significant differences in plaque reductions were found between manual](https://reader030.vdocument.in/reader030/viewer/2022040514/5e6be1349a967c36df4f5ed5/html5/thumbnails/10.jpg)
Note: I’m Not the First Person toPoint Out Some of These Issues
Research by Dawes, Highhouse,Slaughter, Bazerman, Kahneman,Tversky, and others have pointed out anddone research on some of these issues.
However, right now traditional validationis the primary focus and JDM is asideshow. The emphasis needs to flip.
![Page 11: Marvin D. Dunnette Distinguished Professor...Marvin D. Dunnette Distinguished Professor My Toothbrushes “No significant differences in plaque reductions were found between manual](https://reader030.vdocument.in/reader030/viewer/2022040514/5e6be1349a967c36df4f5ed5/html5/thumbnails/11.jpg)
New Hiring System
Quality criterion measureCareful predictor build
r = .45
Good job analysis
All is right in the world, yes?
![Page 12: Marvin D. Dunnette Distinguished Professor...Marvin D. Dunnette Distinguished Professor My Toothbrushes “No significant differences in plaque reductions were found between manual](https://reader030.vdocument.in/reader030/viewer/2022040514/5e6be1349a967c36df4f5ed5/html5/thumbnails/12.jpg)
No• In this case, the correlation between hiring
decisions and your assessment scores areZERO.
• Judgments were unrelated to your input.• The organization has realized negative utility
from your work.– I hope you’re proud of yourself.
The Effect of Our Work on Decision Making is the Key
![Page 13: Marvin D. Dunnette Distinguished Professor...Marvin D. Dunnette Distinguished Professor My Toothbrushes “No significant differences in plaque reductions were found between manual](https://reader030.vdocument.in/reader030/viewer/2022040514/5e6be1349a967c36df4f5ed5/html5/thumbnails/13.jpg)
The Cassandra Problem
![Page 14: Marvin D. Dunnette Distinguished Professor...Marvin D. Dunnette Distinguished Professor My Toothbrushes “No significant differences in plaque reductions were found between manual](https://reader030.vdocument.in/reader030/viewer/2022040514/5e6be1349a967c36df4f5ed5/html5/thumbnails/14.jpg)
Incremental Validity Part 1:Adding a New Predictor to theHiring SystemPredictor A, r = .30, ΔR2 = .00
Predictor A = Failure, Drop
Right?
![Page 15: Marvin D. Dunnette Distinguished Professor...Marvin D. Dunnette Distinguished Professor My Toothbrushes “No significant differences in plaque reductions were found between manual](https://reader030.vdocument.in/reader030/viewer/2022040514/5e6be1349a967c36df4f5ed5/html5/thumbnails/15.jpg)
No• Predictor 1 is highly face valid, sexy (for an
assessment), and gets a heavy weight indecision making…forcing out irrelevantfactors that used to weaken the decision.
![Page 16: Marvin D. Dunnette Distinguished Professor...Marvin D. Dunnette Distinguished Professor My Toothbrushes “No significant differences in plaque reductions were found between manual](https://reader030.vdocument.in/reader030/viewer/2022040514/5e6be1349a967c36df4f5ed5/html5/thumbnails/16.jpg)
Incremental Validity Part 2:Adding a New Predictor to the HiringSystemPredictor B, r = .30, ΔR2 = .04
Predictor 2 = Success, Keep
Right?
![Page 17: Marvin D. Dunnette Distinguished Professor...Marvin D. Dunnette Distinguished Professor My Toothbrushes “No significant differences in plaque reductions were found between manual](https://reader030.vdocument.in/reader030/viewer/2022040514/5e6be1349a967c36df4f5ed5/html5/thumbnails/17.jpg)
No• No one likes Predictor B and it is largely
ignored costing time and money.• People ignore predictors because:
– Don’t like it.– Don’t get it.– Don’t trust it.
![Page 18: Marvin D. Dunnette Distinguished Professor...Marvin D. Dunnette Distinguished Professor My Toothbrushes “No significant differences in plaque reductions were found between manual](https://reader030.vdocument.in/reader030/viewer/2022040514/5e6be1349a967c36df4f5ed5/html5/thumbnails/18.jpg)
Incremental Validity Part 4:This Time It’s Personal
Predictor C, r = .20, ΔR2 = .01Fast, Low cost, People like it
Predictor C = OK, keep
Right?
![Page 19: Marvin D. Dunnette Distinguished Professor...Marvin D. Dunnette Distinguished Professor My Toothbrushes “No significant differences in plaque reductions were found between manual](https://reader030.vdocument.in/reader030/viewer/2022040514/5e6be1349a967c36df4f5ed5/html5/thumbnails/19.jpg)
No• Predictor C has strong narrative qualities
that cause it to get excessive judgmentweight dragging down overall decisionquality.
• In other words, people now ignore the otherhigher quality information in the system.
![Page 20: Marvin D. Dunnette Distinguished Professor...Marvin D. Dunnette Distinguished Professor My Toothbrushes “No significant differences in plaque reductions were found between manual](https://reader030.vdocument.in/reader030/viewer/2022040514/5e6be1349a967c36df4f5ed5/html5/thumbnails/20.jpg)
Imagine we have goodpredictors that are correctlyused and we make offers tothe top people.
Good?
![Page 21: Marvin D. Dunnette Distinguished Professor...Marvin D. Dunnette Distinguished Professor My Toothbrushes “No significant differences in plaque reductions were found between manual](https://reader030.vdocument.in/reader030/viewer/2022040514/5e6be1349a967c36df4f5ed5/html5/thumbnails/21.jpg)
No, The Best May Not Accept
Murphy, 1986
They don’tlove you
![Page 22: Marvin D. Dunnette Distinguished Professor...Marvin D. Dunnette Distinguished Professor My Toothbrushes “No significant differences in plaque reductions were found between manual](https://reader030.vdocument.in/reader030/viewer/2022040514/5e6be1349a967c36df4f5ed5/html5/thumbnails/22.jpg)
Fine, fine, fine, samesituation but the bestaccept, happy now?
![Page 23: Marvin D. Dunnette Distinguished Professor...Marvin D. Dunnette Distinguished Professor My Toothbrushes “No significant differences in plaque reductions were found between manual](https://reader030.vdocument.in/reader030/viewer/2022040514/5e6be1349a967c36df4f5ed5/html5/thumbnails/23.jpg)
Kuncel & Klieger, 2007
Nope, The Best May Not Even Apply
Weakest LSAT,Lowest Ranked
Schools
Strongest LSAT,Lowest Ranked
Schools
![Page 24: Marvin D. Dunnette Distinguished Professor...Marvin D. Dunnette Distinguished Professor My Toothbrushes “No significant differences in plaque reductions were found between manual](https://reader030.vdocument.in/reader030/viewer/2022040514/5e6be1349a967c36df4f5ed5/html5/thumbnails/24.jpg)
New Hiring System
Quality criterion measure
Careful predictor builds
Adjusted Multiple R = .45
Good job analysis
All is right in the world, yes?
Best applicants acceptBest applicants apply
Predictors are used
![Page 25: Marvin D. Dunnette Distinguished Professor...Marvin D. Dunnette Distinguished Professor My Toothbrushes “No significant differences in plaque reductions were found between manual](https://reader030.vdocument.in/reader030/viewer/2022040514/5e6be1349a967c36df4f5ed5/html5/thumbnails/25.jpg)
No• Virtually no one makes decisions using beta
weighted (or unit weighted) predictor scores.
Kuncel, Klieger, Connelly, & Ones, 2013
![Page 26: Marvin D. Dunnette Distinguished Professor...Marvin D. Dunnette Distinguished Professor My Toothbrushes “No significant differences in plaque reductions were found between manual](https://reader030.vdocument.in/reader030/viewer/2022040514/5e6be1349a967c36df4f5ed5/html5/thumbnails/26.jpg)
Please Note• In fact, it is entirely possible that the expert
judgments, despite lower predictive power,result in better people getting hired.
• Preference for expert judgement may deterpeople from using their own, off the cuff, “Ifyou were a tree, what kind of a tree wouldyou be”-untrained-gut reaction-nonsense.
• Algorithmic decision making may invite theopposite reaction. (“There is room for my gutreaction in the decision”).
![Page 27: Marvin D. Dunnette Distinguished Professor...Marvin D. Dunnette Distinguished Professor My Toothbrushes “No significant differences in plaque reductions were found between manual](https://reader030.vdocument.in/reader030/viewer/2022040514/5e6be1349a967c36df4f5ed5/html5/thumbnails/27.jpg)
A barely explored world awaits usin understanding how to integratealgorithmic decision making withexpert judgement to have apositive affect on the decisionmaking of hiring mangers.
![Page 28: Marvin D. Dunnette Distinguished Professor...Marvin D. Dunnette Distinguished Professor My Toothbrushes “No significant differences in plaque reductions were found between manual](https://reader030.vdocument.in/reader030/viewer/2022040514/5e6be1349a967c36df4f5ed5/html5/thumbnails/28.jpg)
Incremental Validity Part 3:Things Are Not As They Appear
• New predictor is added to system.• Both are believed to be solid predictors• Single predictor was good (r=.40)• Joint predictive power is large R=.60• Hurray!
Right?
![Page 29: Marvin D. Dunnette Distinguished Professor...Marvin D. Dunnette Distinguished Professor My Toothbrushes “No significant differences in plaque reductions were found between manual](https://reader030.vdocument.in/reader030/viewer/2022040514/5e6be1349a967c36df4f5ed5/html5/thumbnails/29.jpg)
Dawes (1975) Science
Compensatory Selection EffectsWe must know the decision policy
No
![Page 30: Marvin D. Dunnette Distinguished Professor...Marvin D. Dunnette Distinguished Professor My Toothbrushes “No significant differences in plaque reductions were found between manual](https://reader030.vdocument.in/reader030/viewer/2022040514/5e6be1349a967c36df4f5ed5/html5/thumbnails/30.jpg)
Positive Predictor IntercorrelationWhat’s Going On?
![Page 31: Marvin D. Dunnette Distinguished Professor...Marvin D. Dunnette Distinguished Professor My Toothbrushes “No significant differences in plaque reductions were found between manual](https://reader030.vdocument.in/reader030/viewer/2022040514/5e6be1349a967c36df4f5ed5/html5/thumbnails/31.jpg)
People who get hired
*
*
*
*
**
*
*
*
*
![Page 32: Marvin D. Dunnette Distinguished Professor...Marvin D. Dunnette Distinguished Professor My Toothbrushes “No significant differences in plaque reductions were found between manual](https://reader030.vdocument.in/reader030/viewer/2022040514/5e6be1349a967c36df4f5ed5/html5/thumbnails/32.jpg)
Resulting Intercorrelation
*
**
**
**
***
![Page 33: Marvin D. Dunnette Distinguished Professor...Marvin D. Dunnette Distinguished Professor My Toothbrushes “No significant differences in plaque reductions were found between manual](https://reader030.vdocument.in/reader030/viewer/2022040514/5e6be1349a967c36df4f5ed5/html5/thumbnails/33.jpg)
Compensatory Selection Effects• Suppressed or negative predictor
intercorrelations wreck havoc on ourestimates.
• This is all due to people making decisions– Hiring manager– Applicant
• A traditional criterion related validity studydoes not tell this story.
![Page 34: Marvin D. Dunnette Distinguished Professor...Marvin D. Dunnette Distinguished Professor My Toothbrushes “No significant differences in plaque reductions were found between manual](https://reader030.vdocument.in/reader030/viewer/2022040514/5e6be1349a967c36df4f5ed5/html5/thumbnails/34.jpg)
The Prom Dateor
Waiting for the One
![Page 35: Marvin D. Dunnette Distinguished Professor...Marvin D. Dunnette Distinguished Professor My Toothbrushes “No significant differences in plaque reductions were found between manual](https://reader030.vdocument.in/reader030/viewer/2022040514/5e6be1349a967c36df4f5ed5/html5/thumbnails/35.jpg)
Top Two ApplicantsWho Do They Select?
Applicant A– Structured Interview 4.5– Job Knowledge 4.0
Applicant B– Structured Interview 4.0– Job Knowledge 4.5
![Page 36: Marvin D. Dunnette Distinguished Professor...Marvin D. Dunnette Distinguished Professor My Toothbrushes “No significant differences in plaque reductions were found between manual](https://reader030.vdocument.in/reader030/viewer/2022040514/5e6be1349a967c36df4f5ed5/html5/thumbnails/36.jpg)
It Depends: Decoy Effect• Applicant A
– Structured Interview 4.50– Job Knowledge 4.00
• Applicant B– Structured Interview 4.00– Job Knowledge 4.50
• Applicant C1– Structured Interview 4.25– Job Knowledge 3.75
• Applicant C2– Structured Interview 3.75– Job Knowledge 4.25
![Page 37: Marvin D. Dunnette Distinguished Professor...Marvin D. Dunnette Distinguished Professor My Toothbrushes “No significant differences in plaque reductions were found between manual](https://reader030.vdocument.in/reader030/viewer/2022040514/5e6be1349a967c36df4f5ed5/html5/thumbnails/37.jpg)
Preference Flipping
Applicant J Written 30 Programs in last 2 years3.0 GPA in computer science (j)
Applicant S Written 10 Programs in last 2 years4.0 GPA in computer science (s)
MP3 Player 2 Gig capacity .003% THD (j)MP3 Player 5 Gig capacity .01% THD (s)
Is the meaning of our scores clear?Are users calibrated on them?
![Page 38: Marvin D. Dunnette Distinguished Professor...Marvin D. Dunnette Distinguished Professor My Toothbrushes “No significant differences in plaque reductions were found between manual](https://reader030.vdocument.in/reader030/viewer/2022040514/5e6be1349a967c36df4f5ed5/html5/thumbnails/38.jpg)
Phantoms
![Page 39: Marvin D. Dunnette Distinguished Professor...Marvin D. Dunnette Distinguished Professor My Toothbrushes “No significant differences in plaque reductions were found between manual](https://reader030.vdocument.in/reader030/viewer/2022040514/5e6be1349a967c36df4f5ed5/html5/thumbnails/39.jpg)
The Top Candidate Turns You DownWho Do We Take?
![Page 40: Marvin D. Dunnette Distinguished Professor...Marvin D. Dunnette Distinguished Professor My Toothbrushes “No significant differences in plaque reductions were found between manual](https://reader030.vdocument.in/reader030/viewer/2022040514/5e6be1349a967c36df4f5ed5/html5/thumbnails/40.jpg)
Procrastination
• Decision makers are prone to delay adecision in this situation.
• But NOT when the alternative is average onboth.
• Does your system reduce or enhance thiseffect? When should it?
![Page 41: Marvin D. Dunnette Distinguished Professor...Marvin D. Dunnette Distinguished Professor My Toothbrushes “No significant differences in plaque reductions were found between manual](https://reader030.vdocument.in/reader030/viewer/2022040514/5e6be1349a967c36df4f5ed5/html5/thumbnails/41.jpg)
Questions• What assessments are resistant to Decoys,
Phantoms, and Procrastination?• What assessment reports are resistant to
Decoys, Phantoms, and Procrastination?• Can we train people out of these effects?• Do the effects hold in more complex
assessments? Do they get worse?
![Page 42: Marvin D. Dunnette Distinguished Professor...Marvin D. Dunnette Distinguished Professor My Toothbrushes “No significant differences in plaque reductions were found between manual](https://reader030.vdocument.in/reader030/viewer/2022040514/5e6be1349a967c36df4f5ed5/html5/thumbnails/42.jpg)
Hiring Decision Making is NOT Top Down
![Page 43: Marvin D. Dunnette Distinguished Professor...Marvin D. Dunnette Distinguished Professor My Toothbrushes “No significant differences in plaque reductions were found between manual](https://reader030.vdocument.in/reader030/viewer/2022040514/5e6be1349a967c36df4f5ed5/html5/thumbnails/43.jpg)
The Lens Model Is Your Friend
Kuncel, Klieger, Connelly, & Ones, 2013
![Page 44: Marvin D. Dunnette Distinguished Professor...Marvin D. Dunnette Distinguished Professor My Toothbrushes “No significant differences in plaque reductions were found between manual](https://reader030.vdocument.in/reader030/viewer/2022040514/5e6be1349a967c36df4f5ed5/html5/thumbnails/44.jpg)
Yu, Kuncel, Shu, McNeal, & Goff (2016, April)
Modeling the Expert
![Page 45: Marvin D. Dunnette Distinguished Professor...Marvin D. Dunnette Distinguished Professor My Toothbrushes “No significant differences in plaque reductions were found between manual](https://reader030.vdocument.in/reader030/viewer/2022040514/5e6be1349a967c36df4f5ed5/html5/thumbnails/45.jpg)
Who shouldget the job
offer?
How should Icombine (use) this
information?
What predictorsshould I use?
What will I doif the offer is
turned down?
What will I do if noone exceptional
applies?
?
Hiring DecisionMaker
![Page 46: Marvin D. Dunnette Distinguished Professor...Marvin D. Dunnette Distinguished Professor My Toothbrushes “No significant differences in plaque reductions were found between manual](https://reader030.vdocument.in/reader030/viewer/2022040514/5e6be1349a967c36df4f5ed5/html5/thumbnails/46.jpg)
Should Ibother toapply?
How shouldI perform onthe tests?
Should Iaccept the
offer?
How should I workgiven my
experiences?
?
Applicant Decision Makers
Applicant Population
Applicant Group
Job Offers
![Page 47: Marvin D. Dunnette Distinguished Professor...Marvin D. Dunnette Distinguished Professor My Toothbrushes “No significant differences in plaque reductions were found between manual](https://reader030.vdocument.in/reader030/viewer/2022040514/5e6be1349a967c36df4f5ed5/html5/thumbnails/47.jpg)
What To Do: Validation Methods1. Focus on the average performance of
hired group versus expected averageperformance of applicant pool/population.
2. Capture hiring recommendation, hiringdecision, and who made the decision.
3. Know the Lens Model.
![Page 48: Marvin D. Dunnette Distinguished Professor...Marvin D. Dunnette Distinguished Professor My Toothbrushes “No significant differences in plaque reductions were found between manual](https://reader030.vdocument.in/reader030/viewer/2022040514/5e6be1349a967c36df4f5ed5/html5/thumbnails/48.jpg)
What To Do: ConsultingOur job is to get good correlationsbetween predictors and criteria.Our job is to positively affect thedecision making process of bothapplicants and hiring managers.