comparing models of calculating the criticality index in job analysis · louisiana tech university...

LOUISIANA TECH UNIVERSITYDepartment of Psychology & Behavioral Sciences

Department of Psychology & Behavioral Sciences ￭ 118 Woodard Hall ￭ (318) 257-4315

Comparing Models of

Calculating the Criticality Index

in Job Analysis

Bharati Belwalkar, M.S.

Lindsey Anderson, M.A.

Frank Igou, Ph.D.


Outline

• Background

• Research Question

• Methodology

• Results

• Implications of Findings

• Limitations & Future Directions


Background

Job analysis for a state-wide law enforcement

agency in southern United States

Captain

Lieutenant

Sergeant


What is job analysis?

• A systematic process of discovering the nature of job(Brannick et al., 2007)

• A systematic investigation of work role requirements and work context (Morgeson & Dierdoff, 2011)

– Tasks

– KSAOs

– Physical context


Why did we conduct job analysis?

• Hiring/ Promotion

• Performance Management

• Training

• Other HR activities– Job description

– Classification and evaluation

– Compensation Planning

Job

Description

Selection

Test Development


How did we collect data?

• Multiple data collection methods (Morgeson & Dierdorff,

2011)

• Direct Observation

• Focus Groups

• Interviews

• Questionnaires

• O*NET

Questionnaires

Focus Groups

O*NET


What did we come up with?

Work Behavior

Task Statement

Knowledge Skills

Abilities

• 8 broad WBs

• Task Statements:

– 95: Captain

– 183: Lieutenant

– 316: Sergeant

• 50 KSAs


Who was involved in the JA process?

• N = 56

• Captains, Lieutenants, and Sergeants

• Generated task statements as incumbents and supervisors

Group 1: Specific SMEs


Who was involved in the JA process?

• N = 56

• Captains, Lieutenants, and Sergeants

• Generated task statements as incumbents and supervisors

Group 1: Specific SMEs

• N = 12

• Majors, Captains, and Lieutenants

• Rated task statements, generated KSAs, and linked KSAs to WBs

Group 2:

Global SMEs

• N = 6

• Assessment Specialists and doctoral students

• Rewrote task statements, applied cut-offs, etc.

Group 3:

Analysts


Which rating scales did we use?

• Importance

• Frequency

• Difficulty

• Consequences of Error

• Necessity Upon Entry 96

6720


Why are rating scales

used?

• Job performance is

impossible without

‘critical’ tasks (Laabs &

Baker, 1989)

• Individuals perceive

and interpret their

jobs differently (Conte et

al., 2005)

How are rating scales

chosen?

• Historical precedents

and personal

preferences (Sanchez &

Fraser, 1992)

• Other factors (e.g.,

economic efficiency,

anticipation of Title VII

litigation)


Research Question

• Unique information (Sanchez & Fraser, 1992)

• Ambiguity in the past literature on how to

combine ratings to obtain criticality index (Curtin,

2003)

FrequencyImportance


Models of Criticality Index• Only importance and frequency as a tie-

breaker (Seberhagen, 2012)

Only Importance Model

• Importance + Frequency (Gatewood, Field, & Barrick, 2011)

Additive Model

• Importance x Frequency (Aguinis, 2013)Multiplicative Model

• (2 x Importance) + Frequency – 2 (Locklear et al., 1993)Arithmetic Model

• Zimportance + Zfrequency (Curtin, 2003)Standardized Model

• Logimportance + Logfrequency (Curtin, 2003)Logarithmic Model


Preliminary Analysis

• Missing Ratings • Not included in the analysis

• Veracity Checks• Duplicates

• Bogus items Bogus

Duplicate

Missing


CODE TASK STATEMENTS

CP001 Certifies timesheets/ payroll records of commissioned and non-commissioned personnel in the section, to ensure proper compensation and

documentation, using the knowledge of policy and procedure related to the accrual of time

CP002 Develops after-action reports for all departmental responses to critical incidents and for tactical operations, using critical thinking skills,

knowledge, and inputs from others, to improve future operations

CP003 Approves or deny leave requests from personnel, to ensure adequate staffing and proper documentation, using the knowledge of policy and

procedure, established section work schedules, and understanding of the section staffing needs

CP004 Approves training and travel requests from section personnel, by reviewing work schedules, policy and procedure, and the training budget, to

ensure personnel receive required training and are provided opportunities for advanced training and skill development

CP005 Assesses available resources in order to formulate a plan and conduct crime investigations by utilizing personnel, equipment/ materials, and

funding resources

CP006 Deploys Louisiana State Police resources, in compliance with the Department's strategic plan, using the knowledge of the Department's

performance indicators

CP007 Maintains the early identification system file, using knowledge of computers and Use of Force policies, in order to identify employees who may

require intervention for their repetitive/ excessive use of force

CP008 Forms alliances with the local law enforcement agencies, by networking and information sharing, to establish common ground in addressing

crime problems

CP009 Offers support to law enforcement agencies, to fulfill the low enforcement mission, using Louisiana State Police resources.

CP010 Distributes all department communications (procedural or executive orders, directives, policy statements, legislative statutes, etc.) to all the

personnel in the section using tools/ processes, such as, email, hand-delivery, staff meeting, etc.

CP011 Ensures command notification to the chain of command, to update command staff with the significant events, using the knowledge of

command notification procedure order

CP012 Ensures timely and accurate reporting of critical or unusual information, via tools such as Command Duty Journal, Lotus Notes,

telecommunications, etc., in order to assist the Command Staff in decision making

CP013 Reviews and approve subordinates' purchases of supplies and equipment, to ensure compliance, using the knowledge of Louisiana State Police

policy and procedure

CP014 Submits a request-to-fill of all vacancies for approval by the Superintendent, in accordance with Department policy, to ensure adequate

personnel level/ staffing

CP015 Keeps oneself updated with the latest trends and legal issues in order to maintain efficient and successful field offices

CP016 Directs/orders subordinate to participate in a random drug screening upon direction of the Command Staff, in order to adhere to Departmental

policy

CP017 Supervises the Evidence Custodian within own troop in effort to ensure compliance with the Departmental Policies specific to handling and

storage of evidence

CP018 Mentors subordinates, utilizing prior knowledge of the Department's mission, to enable the subordinate to become a productive employee

CP019 Reviews actions of subordinates, utilizing knowledge of the Louisiana ethics laws, to ensure compliance with current ethics laws

CP020 Ensures appropriate maintenance and security of personnel files, unique to each employee assigned to his command, as per Louisiana State

Police policy concerning personnel file, to maintain confidentiality.


Tasks Only IMP ADD MULTI ARITH STD LOG

CP001 2 2 4 3 15 2

CP002 7 14 18 17 9 13

CP003 10 5 6 8 13 5

CP004 13 9 11 15 4 7

CP005 8 7 7 9 12 7

CP006 1 1 1 1 1 1

CP007 5 9 13 10 14 9

CP008 4 4 5 6 8 4

CP009 8 8 9 11 10 10

CP010 2 3 4 4 6 3

CP011 13 11 10 17 10 12

CP012 3 3 3 5 18 2

CP013 10 9 12 13 7 8

CP014 9 12 16 16 16 11

CP015 12 9 10 14 3 8

CP016 6 10 15 12 19 9

CP017 11 13 14 18 11 15

CP018 3 1 2 2 17 1

CP019 12 15 17 19 2 14

CP020 8 6 8 7 5 6


Add Multi Arith Std LogOnlyImp

.712**

.606**

.825**

-.283 .673**

Add.964

**.974

**-.093 .963

**

Multi.901

**-.039 .911

**

Arith-.164 .941

**

Std-.062

Note: *p < .05, **p < .01


Add Multi Arith Std LogOnlyImp

.712**

.606**

.825**

-.283 .673**

Add.964

**.974

**-.093 .963

**

Multi.901

**-.039 .911

**

Arith-.164 .941

**

Std-.062

Note: *p < .05, **p < .01

Additional Analysis: Fisher’s r to z transformation


Results

• Additive, Multiplicative, Arithmetic, and

Logarithmic models were fairly consistent with

each other

• Standardized Model operates differently.


Implications of Findings

• Importance ratings are subjective (judgmental),

frequency ratings are objective (Morgeson & Campion, 1997)

• Only Importance Model: Frequency not crucial

• Training: ‘Necessity Upon Entry’ ratings

• Performance Appraisal: Task Difficulty

• Other HR processes: Limited knowledge and

resources

Purpose and nature of the job dictate the choice of a model


Limitations

• Generalizability– Small number of SMEs

– Limited sample of 20 Task

Statements

– Only one rank was

considered

Future Directions

• Comparing outcomes

from different models


Some Important References Aguinis, H. (2013). Performance Management (3rd Ed). Upper Saddle River, New Jersey: Pearson Education Inc.

Curtin, P. (2003). Task analysis: Examining four rating formulas. Paper presentation session at the 26th annual conference

for the International Public Management Association Assessment Council, Baltimore, MD.

Levine, E. L., & Sanchez, J. I. (2012). EVALUATING WORK ANALYSIS IN THE 21 ST CENTURY. The Handbook of Work

Analysis: Methods, Systems, Applications and Science of Work Measurement in Organizations, 127.

Gatewood, R. D., Field, H. S., & Barrick, M. (2011). Human Resource Section. Southern-Western Cengage Learning,

Mason, OH.

Kane, Kingsbury, Colton, & Estes (1989). Combining Data on Criticality and Frequency in Developing Test Plans for

Licensure and Certification Examinations. Journal of Educational Measurement, 26(1), 17-27.

Laabs, G. J., & Baker, H. G. (1989). Selection of critical tasks for Navy job performance measures. Military

Psychology, 1(1), 3-16.

Locklear, T. S., Prewett, A. J., Campion, C. H., Livingston, S., & Verses, J. G., III. (1993). Developing job-related selection

procedures: Alabama job analysis method training manual (Vol. 1). Montgomery, AL: Auburn University Montgomery.

Morgeson, F. P., & Dierdorff, E. C. (2011). Work analysis: From technique to theory. APA handbook of industrial and

organizational psychology, 2, 3-41.

Morgeson, F.P. & Campion, M.A. (1997). Social and cognitive sources of potential inaccuracy in job analysis. Journal of

Applied Psychology, 82(5), 627-655.

Sanchez, Juan I., and Scott L. Fraser. "On the choice of scales for task analysis." Journal of Applied Psychology 77, no. 4

(1992): 545.

Seberhagen, L. (2012, November 26). Re: Hello, my question goes out to those who have been working extensively in the

field of job analysis for test development purposes [Online discussion comment]. Retrieved from

http://www.linkedin.com/groupItem?view= &gid=1778581&type =member&item =190785402&qid=80f236d7-701e-43f8-

b0ef-38112506f850&trk=group _most_popular-mc-rr-ttl&goback=%2Egmp_1778581


Any Questions?

comparing models of calculating the criticality index in job analysis · louisiana tech university...

Documents