Selection and Assessment in Safety-Sensitive Industries
Selection System Development and Validation Portfolio
Lynne R. Sherman, Ph.D. Industrial and Organizational Psychologist
Selection System Development and Validation Portfolio i Lynne R. Sherman, Ph.D.
Table of Contents Introduction ................................................................................................................................. 1 Selection and Assessment Testimonials ..................................................................................... 2 Selection System Development and Validation Projects ............................................................. 3 Selection System Validation Documentation ............................................................................... 4 Training Programs that Support the Selection System Development and
Administration Processes ................................................................................................ 7 Expert Witness Testimony and Litigation Support Activities ........................................................ 8 Sample Job Knowledge Test Items ............................................................................................. 9
Reading Comprehension Test ......................................................................................... 9 Mathematical Ability Test ............................................................................................... 10 Instrumentation Reading Test ........................................................................................ 11 Troubleshooting Test ..................................................................................................... 13
Selection System Development and Validation Portfolio 1 Lynne R. Sherman, Ph.D.
Introduction Selecting candidates in safety-sensitive industries – such as healthcare, aviation, and public safety – presents a number of unique challenges not found in other industries, the most notable of which is the potential catastrophe associated with making a bad hiring decision. Public health can be at risk, fines can be levied, negative press can result, large-scale property damage can occur, and most importantly, lives can be lost. Thus, it is critical that selection systems used for hiring employees in safety-sensitive industries follow all professional, legal, and regulatory requirements that apply to selection system development and validation. Included in this portfolio are several documents that highlight my expertise in selection and assessment in healthcare, aviation, and public safety organizations. These documents include:
Selection and Assessment Testimonials
Selection System Development and Validation Projects: Organization x Selection System Component Grid
Selection System Validation Documentation
Training Programs that Support the Selection System Development and Administration Processes
Expert Witness Testimony and Litigation Support
Sample Job Knowledge Test Items For additional information, feel free to contact me at 203-641-6212 or at [email protected].
Selection System Development and Validation Portfolio 2 Lynne R. Sherman, Ph.D.
Selection and Assessment Testimonials
“Lynne, you have earned the respect of all the people you work with, from front-line mechanics and pilots to your colleagues in the People division and senior leaders. In your role as an expert witness for United, you saved the organization hundreds of thousands of dollars in employment litigation costs and penalties. In short, you exemplify the best in us all: you can always be counted on to deliver a professionally-developed, high-quality product when we need it.” Dennis Armstrong, Ph.D. Director, Organizational Effectiveness United Airlines – World Headquarters
“Lynne is regarded as an asset by those who have worked with her, and her skills are clearly in line with our needs. She is an exceptionally competent practitioner in her field and has had a tremendous impact on helping us become a more highly functioning organization. Personally I really enjoy working with Lynne!” Elizabeth Conrad Vice President of Human Resources Yale-New Haven Hospital / Saint Raphael Campus
“On behalf of the Airframe Maintenance SFOOV Training Department, I want to express my sincere appreciation for all that you have done for us. As a direct result of your expertise and hard work, we have shown an immediate return on investment in the maintenance division as a result of the new skills assessments and structured interviews you developed and validated for use in hiring new airframe mechanics. We have seen a higher quality, better prepared workforce with an improved work ethic. These results are especially important in light of the recent heightened safety awareness among major air carriers. Without your tenacity and quest for accurate maintenance knowledge, we would not be able to hire and promote such high quality candidates. Lynne, you have proven your worth over and over again in our quest for quality. We’ve also learned a great deal from you!” Tony Barker Team Leader, SFOOV Training Department United Airlines – Maintenance Operations Center, San Francisco
Selection System Development and Validation Portfolio 3 Lynne R. Sherman, Ph.D.
Selection System Development and Validation Projects Organization x Selection System Component Grid
Organization
Beh
avio
rally-B
ased
S
tru
ctu
red
In
terv
iew
Jo
b K
no
wle
dg
e
Te
st(
s)*
Skills
Assessm
en
t
Wri
tin
g S
am
ple
Assessm
en
t
Wo
rk S
tyle
s /
Pers
on
ality
In
ven
tory
Assessm
en
t C
en
ter –
In-B
asket
Exerc
ise
Assessm
en
t C
en
ter –
Ora
l B
riefi
ng
Ph
ysic
al A
bil
itie
s
Te
st
Valid
ati
on
Do
cu
men
ts o
n F
ile
Su
ccessfu
lly
Wit
hs
too
d L
eg
al
Ch
all
en
ge
Waterbury Hospital
Registered Nurse n/a
Patient Care Associate (CNA) n/a
Unit Clerk n/a
OD Consultant n/a
Saint Raphael Healthcare System (now Yale-New Haven Healthcare)
Patient Care Manager n/a
Registered Nurse n/a
Graduate Nurse n/a
Support Associate n/a
United Airlines
Flight Officer (Pilot / Copilot / Flight Engineer)
Yes
Airframe and Powerplant Mechanic
Yes
Radio and Electric Base Mechanic
Yes
Radio and Electric Line Mechanic Yes
Radio and Electric Shop Mechanic
Yes
Powerplant Base Mechanic n/a
Powerplant Line Mechanic n/a
General Mechanic n/a
Mechanic Helper n/a
Storekeeper Yes
Painter n/a
Technical Recruiter – Flight Operations
Yes
Technical Recruiter – Fleet Operations
n/a
Flight Officer Training Database Manager – Fight Safety
n/a
Senior Staff Analyst – Flight Safety
n/a
Data Entry Associate – Flight Safety
n/a
Administrative Assistant – Flight Safety
n/a
Director of Maintenance Operations
n/a
* Job Knowledge Tests include:
Critical Thinking Test
Data Entry Test
Floor Plan Reading Test
Following Directions Test
Instrumentation Reading Test
Map Reading Test
Mathematical Ability Test
Mechanical Reasoning Test
Med Administration Test
Problem, Cause, Solution Test
Radio and Electric Mechanic Test
Reading Comprehension Test
Troubleshooting Test
Selection System Development and Validation Portfolio 4 Lynne R. Sherman, Ph.D.
Organization
Beh
avio
rally-B
ased
Str
uctu
red
In
terv
iew
Jo
b K
no
wle
dg
e
Te
st(
s)*
Skills
Assessm
en
t
Wri
tin
g S
am
ple
Assessm
en
t
Wo
rk S
tyle
s /
P
ers
on
ality
In
ven
tory
Assessm
en
t C
en
ter –
In-B
asket
Exerc
ise
Assessm
en
t C
en
ter –
Ora
l B
riefi
ng
Ph
ysic
al A
bil
itie
s
Te
st
Valid
ati
on
D
oc
um
en
ts o
n F
ile
Su
ccessfu
lly
Wit
hs
too
d L
eg
al
Ch
all
en
ge
General Electric
Collections Adjuster n/a
Greater Cleveland RTA
Upholsterer n/a
Merchant Bank
Account Processor n/a
Bank Teller n/a
Clerk Typist n/a
Collections Adjuster n/a
Encoder n/a
Exception Item Processor n/a
Sorter Operator n/a
City of Akron, OH
Fire Department – All ranks Yes
Police Department – All ranks Yes
City of Cleveland, OH
Fire Department – All ranks Yes
Police Department – All ranks Yes
City of Chicago, IL
Fire Department – All ranks Yes
Police Department – All ranks Yes
City of Jacksonville, FL
Fire Department – All ranks Yes
Police Department – All ranks Yes
City of Kent, OH
Fire Department – All ranks Yes
Police Department – All ranks Yes
City of Lakewood, OH
Fire Department – All ranks Yes
Building Inspector Yes
Clerk Typist Yes
Wastewater Plant Operator Yes
Water Treatment Plant Operator Yes
City of Mansfield, OH
Fire Department – All ranks Yes
Police Department – All ranks Yes
City of Willoughby, OH
Fire Department – All ranks Yes
Police Department – All ranks Yes
* Job Knowledge Tests include:
Critical Thinking Test
Data Entry Test
Floor Plan Reading Test
Following Directions Test
Instrumentation Reading Test
Map Reading Test
Mathematical Ability Test
Mechanical Reasoning Test
Med Administration Test
Problem, Cause, Solution Test
Radio and Electric Mechanic Test
Reading Comprehension Test
Troubleshooting Test
Selection System Development and Validation Portfolio 5 Lynne R. Sherman, Ph.D.
Selection System Validation Documentation The following documents are routinely developed as part of the selection system validation process and provided to clients / organizations as appropriate.
Technical Report A formal technical report is written for each selection system that is developed, and it contains detailed information about the steps that were taken to create and validate the selection system. In most cases, a multiple hurdle selection system is developed in which case, the development and validation of each component of the selection system is included in the report. This document serves as the foundation of legal defensibility. Several appendices are typically referenced in and attached to the report; these appendices may include the following information:
Job description with major work behaviors, task analysis, and cognitive decision points Note: In the case of positions for which there are variations or specializations (e.g., Emergency Department Registered Nurse vs. Behavioral Health Registered Nurse vs. Critical Care Registered Nurse, etc.), a core job description is developed which represents commonalities across all similar positions and targeted job descriptions representing different specializations supplement the core job description.
Job specifications including minimum qualifications
ADA documentation
Challenges and responses to job knowledge test items for Civil Service exams
Job Analysis Data
Demographic information for each person interviewed / observed during the job analysis process
Job analysis notes generated from supervisor and incumbent interviews
Job analysis notes generated from on-the-job observations
List of source material which represents knowledge, skills, and abilities required prior to job entry
List of resources available to incumbents
Draft job descriptions reviewed and approved by incumbents
Statistical Analyses
Item analyses
Analyses used to determine cut scores for job knowledge tests, assessment center exercises, skills assessments, and physical abilities tests
Adverse impact analyses (4/5ths Rule and chi-square)
Selection System Development and Validation Portfolio 6 Lynne R. Sherman, Ph.D.
Behaviorally-Based Structured Interviews
List of source materials used for content development
List of behavioral dimensions to be assessed in each selection system component with definitions / descriptions of each dimension and behavioral anchors associated with varying levels of performance (e.g., minimally acceptable performance, below minimally acceptable performance, and above minimally acceptable performance)
Structured, behaviorally-based interview questions for each behavioral dimension
Rating forms for each dimension assessed
Voluntary withdrawal form
Job Knowledge Tests
Test plan for job knowledge tests including domains assessed and number of questions for each domain
List of source material used to generate test items
Item documentation (i.e., keyed test items with documented rationales for both correct and incorrect alternatives, citations for each item and alternative)
Cut score development process
Statistical analyses to support use of individual items / tests
Voluntary withdrawal form
Skills Assessments
List of skills assessed linked back to KSAs required at time of job entry
Detailed steps outlining how to perform skill
Subject matter expert ratings of varying performance levels
Behavioral criteria to objectively evaluate performance
Cut score development process
Statistical analyses to support use of cut scores
Rating forms
Involuntary withdrawal / disqualification form due to safety concerns / violations
Voluntary withdrawal form
Assessment Centers – In-Basket Exercises
List of KSAs assessed linked back to KSAs required at time of job entry
Storyboards / scenarios for each exercise
Subject matter expert ratings for varying performance levels
Behavioral criteria to objectively evaluate performance
Cut score development process
In-basket exercise materials (e.g., memos, reports, phone messages, performance reports)
Voluntary withdrawal form
Selection System Development and Validation Portfolio 7 Lynne R. Sherman, Ph.D.
Assessment Centers – Oral Briefing Exercises
List of KSAs assessed linked back to KSAs required at time of job entry
Subject matter expert ratings for varying performance levels
Behavioral criteria to objectively evaluate performance
Cut score development process
Oral briefing materials (e.g., memos, reports, phone messages, performance reports)
Voluntary withdrawal form
Physical Abilities Test
List of activities assessed in test linked back to KSAs required at time of job entry
Course layout / map of course for participants
Equipment required provided for test (e.g., bunker coat, fire hose)
Subject matter expert ratings of varying performance levels
Behavioral criteria to objectively evaluate performance
Cut score development process
Statistical analyses to support use of cut scores
Rating forms
List of safety precautions taken before, during, and after administrations (e.g., medical personnel at site, change in start time due to dangerously hot weather, food and water stations throughout course)
Involuntary withdrawal / disqualification form due to safety concerns / violations
Voluntary withdrawal form
Selection System Administration Manual
Each component of the selection system has its own administration manual
Selection System Scoring Manual
Each component of the selection system has its own scoring manual
Selection System Development and Validation Portfolio 8 Lynne R. Sherman, Ph.D.
Training Programs that Support the Selection System Development and Administration Processes
The ability to clearly demonstrate that all relevant personnel have received formal training as needed to complete the selection system development, administration, and/or scoring processes is critical to the defensibility of both the selection system used as well as the administration process. Listed below are the most common training programs provided to organizations,
Cognitive Job Analysis Training Participants include:
Industrial and Organizational Psychologists
Human Resources professionals
Item Writing Training Participants include:
Industrial and Organizational Psychologists
Human Resources professionals
Technical item writers (e.g., education professionals, commercial airline pilots, airframe and powerplant mechanics, radio and electric mechanics)
Behaviorally-Based Structured Interview Training Participants include:
Human resources professionals
Management personnel
Technical interviewers (e.g., commercial airline pilots, airframe and powerplant mechanics, radio and electric mechanics)
Selection System Administration Training A separate manual is created for each component of the selection system. Participants include:
Industrial and Organizational Psychologists
Human Resources personnel
Management personnel
Assessment center administrators and raters
Selection System Development and Validation Portfolio 9 Lynne R. Sherman, Ph.D.
Expert Witness Testimony and Litigation Support Activities
Expert Witness Testimony Cases in which expert witness testimony, reports, and opinions were provided are listed below.
Marion S. Bullington v. United Air Lines, Inc., United States District Court District of Colorado, Case No. 97-WY-240-AJ
Alleged discrimination in the flight officer selection system / process based on age and gender; case included charges of adverse impact and disparate treatment.
David Ebeling v. United Airlines, Inc., United States District Court Western District of Washington, Case No. C97-347C
Alleged discrimination in the flight officer selection system / process based on age; case included charges of adverse impact and disparate treatment.
Litigation Support Litigation support includes reviewing case / court documents, developing case summaries, engaging clients’ in-house and/or trial attorneys in strategy development activities, critiquing opposing experts’ arguments and developing counterarguments, conducting literature searches and compiling relevant literature, compiling data for Discovery, writing direct and cross-examination questions for depositions and trial, conducting statistical analyses, and writing reports. Cases for which litigation support was provided are listed below.
Adams v. Burroughs-Wellcome Company
Adams v. City of Chicago
Adams v. SouthWest Publishing
Bettes v. Jenkens Gilchrist
Brown v. City of Chicago
Carmichael v. Martin Marietta
Gilner v. Saint-Gobain / Norton Industries
Porter v. McKinsey & Co.
Skalka v. Fermco
Slover v. Steris Corporation
Sperling v. Hoffman-LaRoche
United Black Firefighters Association v. City of Akron
Ware v. GCRTA
Woods v. Green Tree Financial Corporation
Wright v. South Central Bell
Selection System Development and Validation Portfolio 10 Lynne R. Sherman, Ph.D.
Sample Job Knowledge Test Items As with all components of a selection system, the constructs assessed (e.g., reading comprehension, mathematical ability) must be factors that are required at the time of job entry. Support for the inclusion of these constructs can be found in the job description for the target position. The following test items were included in a battery of tests for the position of Water Treatment Plant Operator for a city in Lakewood, Ohio. The test items presented below were used as SAMPLE questions for the candidates to review prior to starting the actual test and are no longer in use, thus their presentation below does not, in any way, compromise the integrity of the actual tests. Instructions on how to complete each test are not included here; however, detailed instructions were included in the actual test booklets.
Reading Comprehension
SAMPLE PASSAGE
Regulating every chemical that may appear in drinking water and that theoretically may adversely affect health in some remote circumstances is impractical and irrational. What is needed is some priority determination for regulation so that a reasonable number of contaminants of sufficient concern can be addressed in regulations that will advance the goals of the SDWA and provide definitive guidance to address potential human health effects of exposure to hazardous materials in drinking water. The most relevant considerations in selection of contaminants are (1) the occurrence or potential for occurrence in drinking water, and (2) the potential health risk.
SAMPLE 1: Plant Operator Smith is conducting a series of tests on a sample of water to determine
what, if any, chemicals are present in the water that may adversely affect people’s health. Operator Smith knows that trying to identify and regulate every chemical that may be found in drinking water is:
A. compatible with the goals of the SDWA. B. both irrational and impractical. C. the most relevant consideration in water treatment.
The best answer is alternative B, “both irrational and impractical.” The first sentence in the passage states that “Regulating every chemical that may appear in drinking water and that theoretically may adversely affect health in some remote circumstances is impractical and irrational.”
Selection System Development and Validation Portfolio 11 Lynne R. Sherman, Ph.D.
Mathematical Ability Test SAMPLE 1: A plant operator must determine the amount of a chemical that has been used during the
past two days. Plant records indicate that on Monday 14.60 lbs. of the chemical were used, and on Tuesday 15.70 lbs. of the chemical were used. Based on the information contained in plant records, the plant operator determined that the amount of the chemical used during the past two days is __________ lbs.
A. 29.30 B. 30.30 C. 31.30
The best answer is alternative B, “30.30” because 15.70 plus 14.60 equals 30.30.
USE THE FOLLOWING FORMULAS TO ANSWER SAMPLE 2
FORMULA A: Average Daily Flow = Total Flow Number of Days
FORMULA B: Average Flow, gpm = Total Flow Number of Minutes
SAMPLE 2: A plant operator must determine the average daily flow over a 5-day period. The total
flow, in terms of millions of gallons (MG) during that 5-day period was 21,000,000. Based on this information, the plant operator determined that the average daily flow during that 5-day period was __________ MG.
A. 2,050,000 B. 3,600,000 C. 4.200,000
The best answer is alternative C, “4,200,000.” Because the question specifically asks you to determine the average daily flow, the most appropriate formula to use is FORMULA A. Take the total flow of 21,000,000 and divide that number by the number of days over which the flow rate must be determined; in this case the target time period is 5 days. Thus, the required calculation is 21,000,000 divided by 5 which equals 4,200,000. Therefore, the best answer to SAMPLE 2 is “4,200,000.”
Selection System Development and Validation Portfolio 12 Lynne R. Sherman, Ph.D.
Instrumentation Reading Test
CARBON DIOXIDE (CO2)
FEED RATE
SAMPLE 1: Plant Operator Smith is taking a reading from a carbon dioxide (CO2) tank. The current
reading is __________ units.
A. 7.4 B. 7.6 C. 7.8
The best alternative is A, “7.4.” The shaded area in the instrument represents the amount of the substance (i.e., carbon dioxide) being measured, and each horizontal line or hash mark on the instrument corresponds 0.2 units. The uppermost level of the substance contained in the instrument corresponds with the number 7.4; therefore, the current reading for this instrument is 7.4 units.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
Selection System Development and Validation Portfolio 13 Lynne R. Sherman, Ph.D.
CARBON DIOXIDE (CO2)
FEED RATE
SAMPLE 2: Plant Operator Smith needs to determine whether the amount of carbon dioxide (CO2)
being fed into the water treatment system needs to be adjusted. Operator Smith knows that readings between 2 and 12 units of CO2 are normal, and NO adjustments are needed to the feed rate. If the feed rate is below 2 units of CO2, then the feed rate must be increased, whereas if the feed rate is above 12 units of CO2, then the feed rate must be decreased. Based on the current reading, Operator Smith must __________ the CO2 feed rate.
A. increase B. decrease C. make NO adjustments to
The best alternative is B, “decrease.” The shaded area in the instrument represents the amount of the substance (i.e., carbon dioxide) being measured, and each horizontal line or hash mark on the instrument corresponds 0.2 units. The uppermost level of the substance contained in the instrument corresponds with the number 12.2 which is outside the desired feed rate range of 2 to 12. More specifically, the feed rate is greater than 12. Therefore, the plant operator needs to decrease the feed rate.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
Selection System Development and Validation Portfolio 14 Lynne R. Sherman, Ph.D.
Troubleshooting Test Rules for Reading Diagrams 1. Water always flows from left to right unless indicated otherwise by an arrow. 2. For each question, always assume water is running through the water treatment process. 3. The cause of a problem in the normal water treatment operations will always occur prior to the
problem or earlier in the water treatment process. 4. In measuring rate of flow, you will always be measuring the flow from the component that is
located just prior to the point at which the rate of flow measurement was taken. 5. Each of the components of the water treatment system is labelled on the diagram, and the
numbers that correspond to each of the tasks performed at the water treatment plant are also provided. The diagram also shows where in the process certain chemicals and gases are added to the water. A Water Treatment Plant Operator can adjust the amount of chemicals and gases added to the water at those specified locations.
SAMPLE DIAGRAM
IRON AND MANGANESE TREATMENT
Contact Tank
Aeration Filtration Clearwell
Backwash
Distribution System Backwash Drain
to Waste
Waste Sludge
Groundwater Well
Chemical Oxidant Added
Disinfectant Added
1 1 2 2 2 2
Selection System Development and Validation Portfolio 15 Lynne R. Sherman, Ph.D.
SAMPLE TABLE
IRON AND MANGANESE TREATMENT
Component Number Measure /
Test Control / Adjust
Water Characteristic
Groundwater Well 1 X X Rate of flow
Aeration 2 X Water level
Contact Tank 2 X X Water level
Filtration 1 X X Rate of flow
2 X X Water level
Clearwell 2 X X Water level
SAMPLE 1: Water Treatment Plan Operator Smith is responsible for measuring the rate of flow in
different locations in the iron and manganese process. Based on the SAMPLE TABLE, Operator Smith will measure the rate of flow at __________ different locations.
A. 1 B. 2 C. 4
The best answer to SAMPLE 1 is alternative B, “2.” Based on the information presented in the SAMPLE TABLE, rate of flow is measure at the groundwater well and at filtration.
SAMPLE 2: Water Treatment Plant Operator Smith has measured the water level at aeration and
notices an unusually high water level. Operator Smith concludes that the high water level must be caused by an increase in the:
A. rate of flow between groundwater well and aeration. B. water level in clearwell. C. rate of flow between filtration and clearwell.
The best answer to SAMPLE 2 is alternative A, “rate of flow between groundwater well and aeration.” Based on the information presented in the rules of water flow, SAMPLE DIAGRAM, and SAMPLE TABLE, an increase in water level at aeration must be due to an earlier process or occurrence. Therefore, the only process before aeration that could increase the water level at aeration is an increase in rate of flow between groundwater well and aeration.