public safety hiring tutorial
TRANSCRIPT
Public Safety Hiring: A Tutorial on Improving Your Selection Process From the RFP to the First Day of the Academy
John M. Ford, Ph.D.
Chris W. Hornick, Ph.D.
Kathryn A. Fox, M.A.
IPAC 2014
Important Considerations Before You Start Developing (or Evaluating) Your Process
2
Factors That Should Drive Public Safety Hiring Decisions
Hiring Decisions
Professional Standards
Legal Guidelines
Scientific Literature
Practical Experience
Best Interests of
Organization
Best Interests of Community
3
Factors That Should Not (But Often Do) Drive Public Safety Hiring Decisions
4
Hiring Decisions
Political Pressure
Past Precedence
Archaic or Misguided
Rules
Self Interest
Reactive Emotional Responses
Rumors
Fear
Define Your Selection Process Goals
5
Hiring Process Goals
Quality Candidates
Legal Defensibility
Candidate / Public
Acceptance
Cost & Required Resources
Diversity
6
Evaluate the Scope of Your Job Analysis
• Evaluate the extent to which your job analysis encompasses the complete job domain.– Ensure that your job analysis captures the full range of
KSAOs relevant to the job.– Identify the broad range of non-task elements that
employees need to be successful.
• Ensure that technical and task-related KSAOs are grouped similarly to how non-technical and non-task KSAOs are grouped.– Over emphasis on “knowledge of” and task-related
abilities will drown out other critical elements of the job.
Example – Importance Ratings of a Firefighter Job Analysis with a
Consortium of Fire Departments
Very ImportantEssential for successful performance of the job
Critically ImportantFailure to perform results in extreme negative consequences
3.8
4.3
4.3
4.3
4.4
3.8
4.3
4.3
3 3.5 4 4.5
Basic Educational Skills
Emotional Outlook
Interpersonal Skills
Practical Skills
Physical Skills
Mechanical Aptitude
Firefighting Skills
EMS Skills
Example – Frequency Ratings of a Firefighter Job Analysis with a
Consortium of Fire Departments3.6
4.2
4.3
4.2
4
3.4
3.6
4.2
3 3.5 4 4.5
Basic Educational Skills
Emotional Outlook
Interpersonal Skills
Practical Skills
Physical Skills
Mechanical Aptitude
Firefighting Skills
EMS Skills
Fairly Often Approximately once per shift
Very OftenMore often than once per shift
Example – Necessity Ratings of a Firefighter Job Analysis with a
Consortium of Fire Departments3.8
4.2
4.1
4
4.3
3.1
2.7
3.3
2.5 3 3.5 4 4.5
Basic Educational Skills
Emotional Outlook
Interpersonal Skills
Practical Skills
Physical Skills
Mechanical Aptitude
Firefighting Skills
EMS Skills
Required at a minimum performance level at time of hire
Required at a fullperformance level at time of hire
10
Example – Police Officer Job Analysis Linkage Results
55%
85%
13%
70%
23%
0% 25% 50% 75% 100%
Basic EducationalSkills
Emotional Outlook
Interpersonal Skills
Practical Skills
Physical Skills
% of Relevant Job Duties Linked to KSAOs
RFPs / Evaluating Proposals
11
Take Control of Your Proposal Evaluation Process
• The way many proposal evaluation processes currently function is not conducive to making informed decisions.–Don’t let test providers decide what
information you should consider.
• Evaluating tests requires professional judgment.– You must ask the right questions and evaluate
the evidence.
13
Write Clear and Specific RFPs
• Define and prioritize your organization’s goals with regard to testing.
• Clearly define the project (or clearly specify when you want guidance).
• Allow consultants to demonstrate their strengths and the value they can provide.
• Define your evaluation criteria based on your values.• Determine appropriate weights for evaluating or scoring
tests.• Give yourself adequate time to review tests.• Don’t be afraid to ask follow-up questions.
14
Proposal Evaluation• Select proposals that utilize assessment tools that are
consistent with your job analysis results.• Evaluate the strength of the validation evidence.• Pay attention to samples.• Don’t compare corrected validity coefficients to
uncorrected validity coefficients.• Evaluate adverse impact in a manner that is consistent
with how you will use the test.• Recognize that adverse impact in applicant samples will be
larger than in incumbent samples.• Has the test been involved in litigation?• Don’t just evaluate the tests, evaluate the consulting
company / test provider.
Remember Hidden Costs
• Poor performance• Increased training/remedial
training/retraining• Lawsuits• Turnover• Grievances• Disciplinary problems• Accidents
Hiring Guidelines & Policies
16
17
Some Agencies Have Policies that Function as a Bureaucratic
Straightjacket• Many agencies have policies or agreements
that hinder their ability to develop an effective hiring process.– Mandatory Cut Scores– Mandated Selection Tools– Mandated Preference Points– Mandated Weighting of Components– Banding Requirements– Mandated Minimum Requirements
18
Strategies for Changing Policies & Agreements
• Build long-term relationships.• Establish your credibility.• Demonstrate fairness and lack of bias.• Show that you have their best interests at heart.• Learn the organization (organizational structure, key
decision makers and potential advocates, available resources, culture and values, lingo, etc.).
• Identify areas of agreement.• Frame communications around strategic goals, belief
systems, and areas of agreement.• Provide information that reduces fear.• Measure! Measure! Measure!• Trumpet successes.
Minimum Qualifications
19
20
Important Considerations Regarding Minimum Qualifications (MQs)
• Represent the minimum competence level needed to perform the job.
• MQs are often highly subjective.• MQs often result in adverse impact or less
diversity in your candidate pool.• MQs sometimes screen out outstanding
candidates.• Processing applications can be very time
consuming – evaluate when in your process you should review applications.
Hiring Announcements
21
22
Best Practices for Hiring Announcements
• Emphasize EEO and diversity values.• Describe the application and hiring process.• Be specific about the job description and required
qualifications.• Provide information on the benefits and available
career paths.• Advertise through want ads, minority
publications, television (including Cable/Local Access), and radio stations.
23
How Applicants Learned About Fire Fighter Job Openings
4240
36 3639
23
4
15
7
15
7
12
0
5
10
15
20
25
30
35
40
45
50
CurrentMember
Friend orFamily
Website Radio CalledDept.
Want Ads
White %
Black %
Recruitment
24
25
Recruitment Best Practices• Involve women and minorities in the recruitment process.• Show diversity in ads brochures, recruiters, media
coverage, etc.• Ensure that women and minorities in the department have
a visible presence.• Encourage “word of mouth” recruiting among women and
minorities.• Go where the women and minorities are!• Have a presence at cultural events and job fairs.• Build positive relationships with local minority community
leaders and minority groups.• Provide mentoring and training/tutoring for the tests.
26
Targeted Recruiting Do’s & Don’ts
• Do not exclude non-minorities from any of your recruitment initiatives or opportunities.
• Work with leaders in the minority community to encourage minorities to take advantage of recruitment initiatives or opportunities.
• Do not use quotas.• Be consistent and fair to every candidate in your
hiring process.• Hire highly qualified candidates.
Assessment Methodology
27
28
Best Practices for Making Decisions about Your Assessment Process
• Don’t make assumptions – Investigate and Measure.
• Recognize that there is no perfect process or perfect test.
• Conduct a validation study or transportability study.
• Be careful about making decisions based on single administrations or small samples.
• Understand sampling error.
29
Example: One Client’s AI Ratios Over Multiple Administrations
1 2 3 4 5 6 7 80
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.41
0.840.69 0.67
0.870.78 0.72
0.44
Administrations
Ad
vers
e Im
pac
t R
atio
Combined Sample = .75
30
Example – Simpson’s Paradox
Male Female
Applicants Hired%
Hired Applicants Hired % Hired
Year 1 150 30 20% 40 16 40%
Year 2 200 70 35% 50 35 70%
Year 3 100 15 15% 50 15 30%
Year 4 50 2 4% 600 48 8%
Total 500 117 23% 740 114 15%
Adapted from Bobko & Roth (2004)
31
Evaluate Testing Medium Just as You Would Any Test
• Research on the use of video or computer-based tests is inconsistent.
• Research suggests that reduced AI may sometimes be the result of something other than the change in medium:– These tests are assumed to reduce AI by reducing the reading
comprehension demands of a test. However, this can hurt validity if the acquisition of job knowledge or classroom training is required.
– These formats may expand (or reduce) the domain of constructs being measured
– Reduced AI may be result of reduced reliability.• Computer-based and video-based tests should be evaluated just
like any other test—in terms of validity, reliability, AI, cost, and practicality.
Written Tests
32
33
Ask the Right Questions When Evaluating the Validity of Potential Tests
• What does the test measure? Are those dimensions consistent with your job analysis results?
• Does the test have a adequate validation report?• Was the test based on a job analysis of the position you are
hiring for?• Is the recommended use for the test consistent with your
process?• Where was the test validated?• Is the validation sample representative of your position?• What was the criterion used for the validation?• Does that criterion represent critical performance
dimensions of your position?
34
Ask the Right Questions When Evaluating the Adverse Impact of Potential Tests
• What was the size of the sample(s) used to evaluate adverse impact?
• Was the sample similar to our applicant pool?• Is the adverse impact sample representative of our labor
market?• Was adverse impact evaluated in an incumbent or
applicant sample?• What are the adverse impact ratios for relevant protected
groups at pass points you are likely to use?• Is the pass point at which adverse impact is reported
similar to the pass point you will use?• What is the SD Difference between relevant protected
groups.
35
Measure a Broad Range of Skills Early in the Process
• Some agencies administer a cognitive test in the first stage and measure non-cognitive skills in a later stage (e.g., interview).
• This strategy will screen out many individuals who have strong non-cognitive skills and would make more complete candidates.
• Measuring a broad range of skills can increase the validity (i.e., the quality of the candidate pool) and minimize the AI of your first stage (as well as your total process).
36
Example—Which Candidate Would be the Best Hire?
Cognitive InterpersonalEmotional Outlook
Practical
Candidate A
87 60 60 60
Candidate B 85 70 70 70
Candidate C
83 90 90 90
Physical Ability Tests
37
38
Legal Background of PATs
• Many Title VII standards have never been interpreted by the Supreme Court, and lower court decisions have been inconsistent and contradictory (Brooks, 2001).– “It is not an overstatement to say that no other aspect of
the personnel selection process is fraught with more difficulty or clouded with more ambiguity than physical ability testing.” Larry Hoover, “Trends in Police Physical Ability Selection Testing”
– A 1997 survey of court reviewed police and fire PATs found that less than 10% were successfully defended.
– Most valid PATs will demonstrate at least some adverse impact with regard to Female candidates.
39
PAT Recommendations
• Use a PAT that utilizes job simulation exercises.– They tend to have lower adverse impact.– They tend to be highly correlated with physical job
performance.– They are the easiest to defend legally as being job-
related.• In addition to measuring strength, including events
emphasizing balance, flexibility, and endurance can increase validity while reducing adverse impact.
• Use a compensatory model.• Ensure that administrators and proctors understand that
making changes to the test can invalidate the test.
40
More PAT Recommendations• Make sure individual events are supported by the job
analysis.• Record data for individual events.• If women are underrepresented in the department, recruit
women from colleges and gyms to run through the PAT before setting a cut score.
• Communicate the importance of physical fitness to the candidates early in the process.
• Show a video of your PAT to the candidates and put the video on your website.
• Provide mentoring and practice to candidates.
Oral Interviews
41
42
Oral Interview Best Practices
• Use Structured Interview format.• Ask questions that assess situational
judgment and past behavior.• Create scoring benchmarks with Subject
Matter Experts.• Use mock candidates to allow assessor
panels to practice working as a team.• Use a multiple hurdle or compensatory
model with written test scores.
Accommodating Disabilities
43
44
Best Practices in Accommodating Disabilities
• Require a letter from a doctor with a diagnosis and a recommended accommodation.
• Establish a time frame for the letter from the Doctor.
• Remember that accommodations need to be reasonable – the candidate must be able to reasonably perform the job.
Preference Points
45
46
Preference Points Considerations
• Make sure your preference points don’t overwhelm the variability in your scores.
• Measure the impact of your preference points on the distribution of your scores and your hiring process.– Do they prevent the top candidates from being
hired?– Do they reduce the diversity at the top of your
rank-order list?
Background Checks
47
48
Evaluate Your Background Check Criteria
• There is very little research investigating the validity and adverse impact of many background check criteria.
• There is very little consistency in the background check criteria used throughout the country.
• Make sure you can defend the criteria you use.
49
Example: Investigating Credit History During Background Check
• EEOC has publicly urged organizations to avoid using credit history in making pre-employment screening decisions.
• Credit history has substantial AI– Freddie Mac study found that the race-credit
correlation is stronger than the income-credit correlation.
• Bad credit histories are often the result of factors unrelated to job performance (such as income loss, medical problems, family breakup).
• Women sometimes have poor credit histories due to ex-partners who have failed to live up to their legal obligations (child support, alimony).
50
Using Credit History as a Disqualifier in Background Investigations Is Difficult to
Defend in Most Cases
• There is no evidence that credit history is related to job performance.
• There is no evidence linking poor credit history to theft, fraud, or criminality.
• There is no evidence to guide creating criteria for rejecting someone due to poor credit history (how much bad credit history is too much?)
• Credit records are notoriously susceptible to errors.
Final Hiring Decision
51
52
Final Considerations
• Remember that Rome was not built in a day – making changes may take a substantial amount of time.
• Focus on the changes you can make.• Continue your professional development and stay
aware of current research, litigation, and best practices.
• Continue to measure!!!• Periodically re-evaluate your process.