survey of rater training programs
TRANSCRIPT
Current Trends in Rater Training: A Survey of Rater Training Programs in
U. S. OrganizationsC. Allen Gorman
East Tennessee State University
Joshua L. RayTusculum College
John P. MeriacUniversity of Missouri-St. Louis
Thomas W. RoddyEast Tennessee State University
Introduction
• The accuracy of performance ratings is important to the success of a performance management system (Werner & Bolino, 1997)
• Two general strategies for improving rating accuracy (Woehr & Huffcutt, 1994)
– Rating scale development– Rater training
• Rater training has become the most widely accepted strategy (Roch, Woehr, Mishra, & Kieszczynska, 2011)
Purpose
• No published research on the prevalence of rater training programs in organizations
• Purpose is to fill the void by conducting a survey of U.S. organizations to determine– Do organizations utilize rater training
programs?– If so, what types of training programs?
Rater Training
• In general, rater training is effective for improving the quality of performance ratings (Smith, 1986; Spool, 1978)
• Two major benefits of rater training (McIntyre, Smith, & Hassett, 1984)
– Enhance raters’ knowledge and skills for carrying out evaluations– Motivate raters to use the knowledge and skills learned in the
training program
• Two meta-analyses have empirically demonstrated the overall effectiveness of rater training programs for improving rating accuracy (Roch, et al., 2011; Woehr & Huffcutt, 1994)
Approaches to Rater Training
• From Woehr & Huffcutt (1994)– Rater Error Training (RET)– Performance Dimension Training (PDT)– Frame-of-Reference Training (FORT)– Behavioral Observation Training (BOT)
Rater Error Training
• Developed as a way to combat the prevalence of psychometric errors in performance appraisal ratings (Borman, 2001)
• Generally focuses on recognizing and avoiding halo, leniency, and central tendency errors (Woehr & Huffcutt, 1994)
• RET reduces halo and leniency errors (Smith, 1986)
But….
• RET inadvertently lowers levels of rating accuracy (Bernardin & Pence, 1980; Borman, 1979; Landy & Farr, 1980)
• Smith (1986) argued that RET actually produces a meaningless redistribution of ratings
• Rater errors may not be errors, but could actually reflect true score variance (Arvey & Murphy, 1998; Hedge & Kavanagh, 1988)
Performance Dimension Training
• Criticisms of RET shifted focus of rater training literature toward rating accuracy (Athey & McIntyre, 1987)
• PDT emphasizes the cognitive processing of raters as the key to the success of rater training
• Typically involves having raters review the rating scale or participating in the development of the scale (Woehr & Huffcutt, 1994)
• Generally effective for improving rating accuracy (Woehr & Huffcutt, 1994)
Frame-of-Reference Training
• Proposed by Bernardin & Buckley (1981) in response to the disappointing results of RET
• Essentially an extension of PDT, but incorporates a practice and feedback session (Woehr & Huffcutt, 1994)
• Involves categorizing behaviors into appropriate dimensions and correctly judging the effectiveness of those behaviors (Sulsky & Day, 1992; 1994)
FORT
• Has emerged as the most popular approach for improving rating accuracy (Roch et al., 2011)
• Meta-analytic effect sizes– Cohen’s d = .83 (Woehr & Huffcutt, 1994)
– Cohen’s d = .50 (Roch et al., 2011)
Criticisms of FORT
• Does not instruct raters on how to process behavior information with goal of remembering the behavior at a later time (Noonan & Sulsky, 2001)
• May cause raters to see certain behaviors that were never exhibited (Noonan & Sulsky, 2001; Sulsky & Day, 1992)
• Little attempt to measure the information processing that supposedly occurs during training (Arvey & Murphy, 1998)
• Overreliance on standard videos of performance and student raters in contrived rating situations (Arvey & Murphy, 1998; Noonan & Sulsky, 2001)
Behavioral Observation Training
• Emphasizes the accuracy of behavioral observations
• Important when considering that raters often must observe performance in noisy environments where competing demands deplete cognitive resources (Noonan & Sulsky, 2001)
• Typically involves note taking or keeping a diary (Woehr & Huffcutt, 1994)
BOT
• Reduces rating errors (Bernardin & Walter, 1977; Latham, Wexley, & Pursell, 1975)
• Leads to increased observational accuracy (Thornton & Zorich, 1980)
• Significantly increases rating accuracy (Hedge & Kavanagh, 1988; Noonan & Sulsky, 2001; Pulakos, 1986)
• Criticisms– Lack of agreement on what constitutes an observational
training program (Noonan & Sulsky, 2001)
– Note taking and diary keeping is likely impractical
Combinations of Rater Training Approaches
• RET + FORT = no significant increase in rating accuracy (McIntyre et al., 1984; Pulakos, 1984)
• RET + other approaches = no increase in rating accuracy (Smith, 1986)
• FORT + BOT = no significant increase in rating accuracy beyond FORT alone (Noonan & Sulsky, 2001; Roch & O’Sullivan, 2003)
Summary of Rater Training Research
• FORT has become the “go to” training– Although may have limited generalizability
• RET is effective– At reducing rating accuracy
• Practice and feedback appear to be important components of any successful rater training program (Borman, 2001; Latham, 1986; Smith, 1986)
• Accumulation of empirical evidence suggests that rater training programs should be worthwhile interventions for improving ratings in organizations
However….
• Lack of widespread adoption of rater training programs in applied settings (Bernardin, Buckley, Tyler, & Wiese, 2001)
– Time consuming and expensive to implement (Stamoulis & Hauenstein, 1993)
– Developing target scores for computing rating accuracy indices is complex and time consuming (Bernardin et al., 2001; Ilgen & Favero, 1985)
– May be insufficient due to low levels of user acceptance and political influence (Carroll & Schneier, 1982; Longnecker, Gioia, & Sims, 1987)
– Has yet to be shown to generalize across jobs and members in organizations (Arvey & Murphy, 1998)
The Present Study
• No scholarly evidence of the prevalence and types of rater training programs in organizations today
• Some anecdotal evidence– TVA, JP Morgan Chase, Lucent Technologies, AT&T
have adopted rater training programs (Levy, 2010)
– Employers Resource Council (2008) – 46% of the 73 organizations surveyed provide rater training
• Exploratory research question: Is rater training related to organizational performance?
Method
• Procedure– Survey part of a larger data collection effort on current performance
management practices (Gorman, Ray, Nugent, et al., 2012)
– Recruited HR executives to complete survey• Directly e-mailing HR departments in Fortune 500 companies• Advertising on popular online business forums• Asking HR execs to forward survey to other HR execs
• Participants• HR executives from 101 U.S. organizations• 88% report revenues of 1 million + dollars• 88% employ at least 100 employees• Largest percentage of the organizations were headquartered in the
Southeastern U.S. (44%)
Measures• Rater Training
– 8 items (e.g., Does your company train managers how to conduct performance appraisals?)
• Performance appraisal system effectiveness– 1 item (Overall, how would you rate the effectiveness of your company’s performance
appraisal system?)– 1 (extremely ineffective) to 5 (extremely effective)
• Performance appraisal system fairness– 1 item (Overall, how would you rate the fairness of your company’s performance
appraisal system?)– 1 (extremely unfair) to 5 (extremely fair)
• Firm-level performance– 1 item (Approximately how much revenue does your company make annually?)– 1 (less than $1 million) to 4 (more than $100 million)
Results
Do Organizations Conduct Rater Training?Response Train Managers Train Non-Managers Refresher/Recalibration
Training
Yes 77 31 50
No 24 70 19
Results
Frequency of Rater Training ApproachesRater Training Approach Frequency Percent
No training 24 23.76%
Rater error training 13 12.87%
Performance dimension training 23 22.77%
Frame-of-reference training 31 30.69%
Behavioral observation training 8 7.92%
Other 2 1.98%
Results
Who Conducts Rater Training?
Training Conducted by Frequency Percent
External consultant 2 2.63%
Internal consultant 2 2.63%
Human resource personnel 61 80.26%
Department manager 6 7.89%
Other 5 6.58%
Results
Frequency of Rater Training
Frequency of Rater
Training
Frequency Percent
Less than one time a year 6 8.00%
One time per year 28 37.33%
Two times per year 13 17.33%
Three times per year 0 0.00%
Four times per year 3 4.00%
As needed 25 33.33%
Exploratory Analyses
• Control variable:– Company size
• Performance appraisal systems that utilize managerial rater training were judged to be more effective (M = 3.70, SD = 1.51) than those that do not (M = 3.37, SD = 1.61), t(98) = 1.77, p < .05.
Exploratory Results
• No significant difference in perceived fairness of performance appraisal system
• Organizations that utilized managerial rater training generated higher revenue (M = 3.09, SD = 1.03) than those that did not (M = 2.71, SD = 1.04), t(98) = 3.07, p < .01.
• Performance appraisal systems were perceived as significantly more legally defensible when the system included a rater training program, χ2(1, N = 101) = 4.13, p < .05.
Rater Training Focus and Perceived Performance Appraisal Effectiveness
Rater Training Focus and Company Revenue
Discussion
• Rater training is alive and well– 76% of organizations surveyed utilize
managerial rater training– 31% train non-managers– FORT (40%) and PDT (30%) most popular– Preliminary evidence that rater training is
linked to firm-level performance
Discussion
• Encouraging results– In contrast to the presumed scientist-
practitioner gap in performance appraisal (Banks & Murphy, 1985; Bretz, Milkovich, & Read, 1992)
– Evidence-based approaches are the predominant rater training methods in use today
– Widespread adoption across many organizations and industries
Areas for Improvement
• Only 22 of the 77 organizations that offer rater training have evaluated the training
• Majority of rater training sessions are only offered either once per year or as needed
Limitations
• Single source data• Links between rater training and firm
performance are not causal• Training programs in practice may not
contain all elements of what is described in the literature
• Small number of organizations; may not be generalizable
