-1 •

\

\

'Y\\W -' . .1 \ \ \ \ \ .

v ; \ • •

I \ \

RESEARCH OK THE DEVBLOK^kirT OF

SHIPBOARD PERFORMANCE METRES

*•»« - v. y

FINAL REPORT (IN FIVE rain)

PARTI*

••.-• •"*•.,

^SHEl*6*ÖEuÄ PERFORMANCE RATING SCALE "I*TTH£ MEASUREMENT OF SHIPBOARD;

FERFORÄ^NmOF ENLISTED NAVAL PERSONNEL

§3^ - i •

:^^ •

1 :'

5 " •• • > "• ' -

- •/<

S - • • ' :^>s

-'•

*'. ' . a •- . - -•

'-.. • »•

•S •' .» . { •

:..': vy. ••. «

1 * V ' P ,'

.'•:'•'

PREPARMN^ '•:;.

PERSONNEL AND TRAINING BRANCH OFFICE OF NAVAL RESEABCK DE?A|H*reNT 0? TH^L-NAVY

• s i

\ \

>*? \y V.*.

> r f v

•'/I A\

vr- _-:-\- r ;i:*^LANAOEMENT;ANO MARKETING RES?AICH JÖORPORatflÖN

LOS ANGELES, CALIFORNIA

V

-'•'•'A

(.

\ , '— <

THIS REPORT HAS BEEN DELIMITED

AND CLEARED FOR PUBLIC RELEASE

UNDER DOD DIRECTIVE 5200,20 AND

NO RESTRICTIONS ARE IMPOSED UPON

ITS USE AND DISCLOSURE,

DISTRIBUTION STATEMENT A

APPROVED FOR PUBLIC RELEASE;

DISTRIBUTION UNLIMITED.

RESEARCH ON THE DEVELGPi.iZNT CF

SHIPBOARD PERFORMANCE MEASURES

FINAL REPORT (In Five Parts)

PART II

TEE USE OF A PERFORMANCE RATING SCALE IN THE I.EASUREMENT OF SHIPBOARD

By

Clark L. Uilson, Ph.D.

Robert R, Mackie, Ph.D.

Donald N, Cuckner

February 1954

Prepared for Personnel and Training Branch Office of Naval Research Department of the Navy Washington 25, D, C.

Contractor Managsuent and marketing

Research Corporation 3004 South Crand Avenue Los Angeles 7. California

for Psychological Research Center

Los Angeles, California

Contract NO onr 70001

ACIINGiJLEDGEUENTS

lie wish to express oi-r sincere thanks for the cooperation of

many Naval personnel, afloat and ashore. Particularly, we wish to thank

Capt. G. R, Donaho, USN, for permission to her. in the work afloat, in Sub-

marine Fiotiiia One, ü. S, Pacific Fleet, Subsequent Commanders of Sub-

marine Flotilla One have been most helpful in the continuing research.

Force, Squadron, Division, and Tender staffs ir. both Pacific and

Atlantic Fleets have contributed greatly to the progress of the work,

and our especial appreciation goes to the Officers ü.-.d leading Petty

Gfficers aboard the more than thirty submarines where \.'\e work was done.

Capt. T. L. Uillmon, MC, USN, and Commander Gerald J. Duffner,

IX, USN, as Officers-in-Charre, Medical Research Laboratory, U. S, Naval

Submarine Case, Mew London, have botli done much to further the wo/'; of

this project.

><'e also wish to acknowledge the editorial assistance of Forrest IC,

Strayer and the contributions of William F. Dossett and Weiis D. De Loach

during the data analysis phase of the project.

TKDLE L'F CONTENTS r/.cc

ABSTRACT

CHATTER

I. SUMMARY AND OPERATIONAL IMPLICATIONS 1

II. BACKGROUND AND üEVELGBiEiYl' CF TIE RATING SCALE 4

A. The Problem 4 B. The Quarterly Marks System ........_ 5 C. r'"-,:id;raiiOiiS for the Design of a Rating Scale 6 D. Requirements of Rating Scale Construction and Design 6 E. Format of the Experimental Scale 8 F. The Selection of Performance Traits for the Rating Scale ... 11 G. Use of the Performance Rating Scale Aboard Ship 11 H, Instructions to the Raters , 12 I, Scoring the Ratings 14

III. RESULTS FROM SHirBQf.RD USE OF TIE RATING SCALE, MEASURING CHARACTERISTICS OF THE SCALE 15

A. Average Rating and Siread of Ratings over the Scale ..... 15 D. Reliability of the Ratings , , 18 C. Agreement among the Raters 19 D. Intercorrelations of Traits and Results of Factorial Analysis. 22 E. Intei-ccrrelations of Traits, Procedure I 23 F. Intercorrelations of Traits, Procedure II. , * 25 G. Intercorrelations of Traits, Procedure III ..«*.- -... . 26 H. Results of the Factor Analyses 20 I. Identification of Factors. .,..,, 30

IV. RESULTS CF THE SHIPBOARD USE OF THE RATING SCALE, FELATIONSHIPS WITH OTHER PERFORMANCE MEASURES 34

A. Correlation of Rating Scale ana Check List Evaluation .... 34 B. Correlations Detveen Rating Scale Scores and Performance Test.

Scores 3C C. Conslusion ....-=. 40

V. CONCLUSIONS , 42

r I 1

CliAPTER PACE

SELECTED BIBLIOGRAPHY „ e ,...,., 44

APPENDIX A, Iutercorrelation Matrices and Results of Factorial Analyses. 50

APPENDIX 3, Sample Performance Rating Scale 69

ABSTRACT

A performance rating scale that Included 10 traits reflecting various

technical and non-technical aspects of Navy shipboard performance was developed

and used to evaluate the performance of Electrician's iiiates (EM's) and Erig.inemen

(EN's) serving aboard submarines in the Atlantic and Pacific Fleets. Analysis of

results indicated that officers and petty officers using the scale tended:

(1) to agree with one another when they evaluated the same men

(2) to be consistent in their own evaluations from one time to the

next

(3) to discriminate reliably among men of the same pay grade

(4) to differentiate, to an appreciable degree, the technical from

the adjustive aspects of shipboard performance.

In addition, a factorial analysis indicated that:

(5) at least two broad "factors" of shipboard performance - one

representing technical skill, and the other, adjustment to Navy

life - accounted for most of the intercorrelations among traits

(o) the traits representing the technical side of perfcmT.ance cor-

related moderately high with independent measures of technical

skill, but the traits representing the adjustment side of per-

formance were not related appreciably to any other measures

obtained.

As a part of this overall research project practical performance tests and

performance check lists for Eu's and EN's were also developed (see Parts I and III

of this Final Report for details).

Relationships between these two measures and the rating scale arc reported

and discussed in this report.

Chapter

3U.iT.iAFY AND OPERATIONAL EDUCATIONS

At the time this research was initiated, shipboard performance in the Navy

was evaluated by means of Quarterly Marks assigned by division officers. It ivas

considered by many, in the Navy and out, that the Quarterly iiarks did not adequately

reflect the differences among enlisted men's abilities to perform their duties.

There was a marked tendency for the assigned scores to pile up around the highest

possible value (4.0). Unless it were assumed that most men's performance was of

equal excellence, these marks were failing tc give discriminating and realistic

indications of performance.

For this reason, research wss conducted to determine if more adequate means

of rating shipboard performance could be developed. This report, which is Part II

of a Final Report of a study of shipboard performance measures, describes a per-

formance rating scale that ivas developed for use by submarine officers and petty

officers in evaluating the performance of Electrician's Mates (Ell's) and Engineraen

(EN's) serving under them.

In its final form, the performance rating scale, following a format that

permitted man-to-man comparisons, included these ten traits:

1

Cooperation Knowledge of the Job Application and Initiative Judgment and Common Sense Leadership

Neatness of Work Care of Equipment Ability tc Troubleshoot Sincerity in Doing a Good Job Discipline

In addition, a Rank Oder Preference (relative overall value in the gang)

was included «s a summary evaluation.

A total of 320 Efi's and 487 EN's were rated on these traits. The majority

- 1 -

^B^ST-TJf^Sm

I

of the men were rated by at least three officers and/cr petty officers. The

reliability (consistency) of the ratings was calculated by having a representative

group of raters re-rate their men after a suitable lapse of time. It was found to

ha mi-itn hint; (f — CD* p.», -1-- 1 ...»«.i. 4.1 ...l..»..»;.il han -w v,-*.^, •.iyn vi«. — ,w/ , naicis axau atjiccj witu unc em/Luc* suuMdimaiij' WHen

they rated the same men, tiie average inter-rater agreement on total score being

rt = .70.

The ratings assigned by all raters on each of the traits were intercorrelated

and analyzed to find what general elements (factors) of performance could account

for the intercorrelations. Two factors were isolated in each of two independent

analyses:

1, Technical competence (ability to meet the practical or technical demands

of a job)

2. Personal adjustment (attitude towards both job and other elements of

Navy life).

The measures obtained with the performance rating scale were also correlated

with scores from two other measures of shipboard performance: performance check

lists and practical performance tests. The check lists were designed to provide

ratings and the performance tests objective scores of ability to perform specific

tasks from Ell's and EN's jobs.

It was found that the rating scale and check list scores were highly cor-

related (about .85) when men in all pay grades were rated together (the substantial

correlation between pay grade and performance ratings in part induced this r.or-

relation). but the correlations between rating scale trait scores and performance

test scores were low to moderate (.25 to .62). On a wlthin-pay-grade basis, the

two rating device*: correlated moderately (.23 to .61) while the rating scale scores

and performance test scores correlated low to moderately (.02 to .40), The

- 2 -

correlations were generally higher for the EM'S than for the EN's.

It was concluded that discriminating and reliable ratings of goneral traits

can be obtained which will reflect important aspects of shipboard performance.

Such ratings say not, however, indicate very much about ability to perfuri.i spöuifii

tasks from shipboard jobs.

The present and probable continued use of rating devices as measures of

shipboard performance gives operational significance to many of the findings of

this study.

Specifically:

1. Reliable and discriminating ratings can be made by Navy officer

and petty officer personnel.

2. A man-to-man rating format appears to promote discriminating

ratings and inter-rater agreement.

o. Ratings should be performed separately for different gangs and

pay grades.

4. Both technical and adjustment, aspects of shipboard performance

can be rated.

5- Inter-rater differences in leniency and spread of ratings must be

accounted for in order to achieve maximum usefulness from ratings.

6. Comparisons of the technical skill of men from different boats

demand the use of performance tests as well as ratines.

- 3

Chapter II

BACKGROUND AND DEVELOPMENT CF THE RATING SCALE

The problem

"In wartime ... elaborate routines of test administration aboard ship may

not be feasible, and a more convenient procedure for evaluating performance is

needed. The use of rating methods would provide the means for making such evalu-

ations. M1

The problem of evaluating the shipboard performance of Navy men is an

exceedingly important one. The success and constant improvement of selection,

training and placement programs depend in large part on the availability of suitable

performance measures which can be used as criteria against which the effectiveness

of these programs can be judged.

In the Navy, and particularly in the submarine Navy, evaluations of per-

formance require consideration of a complexity of factors. Technical know-how and

skill are essential; even more important, however, may be factors of an attitudinal

or adjustive nature toward Navy life and the men with whom one lives 24 hours a day.

This study is concerned with an effort to devise a rating scale winch could

be used to obtain measures of the many important aspects of shipboard performance.

It, together with other studies on the problem of shipboard performance measures,

provides partial answers to this important problem.«-

Human Factors in Military Efficiencyt Summary Technical Report of the Applied Psychology Panel, M3RC, Vol. I, Washington, D. C., iväö, p. 130.

^ See also Part I, "The use of practical performance tests in the measurement of shipboard performance of Naval enlisted personnel." and Part III. "Ili_e use, QC performance check lists in the measurement of shipboard performance of enlisted Naval personnel."of this final report.

- 4 -

r

Thg Quarterly Marks System

At the time this research was initiated, shipboard performance was regularly

evaluated in '«lie Navy through the assigning of Quarterly Marks. Enlisted men were

rated on their Proficiency in Rate and Conduct, and, depending en their job and pay

grade, on their Leadership, Seamanship, and Mechanical Ability.

The Quarterly Marks scale had a score range from 0.0 to 4.0. However, a

»ark below 2.5 indicated unsatisfactory performance, and there was a tendency, in

practice, for both Division Officers and the men being rated to consider an; n«3rk

below 3,5 unsatisfactory. As a result, the marks tended to pile up between 3.5

and 4.0, with the majority for the higher pay grades ax or near 4.O.. Little dis-

crimination resulted from such ratings and a man's scores seldom gave a clear

indication of whether the job he was doing was excellent, good, ordinary, or just

fair.

The problems faced by the Division Officer (or whoever does the ratine) in

assigning Quarterly Marks may be illustrated by a few extracts from the DuPors

Manual;

In determining marks, it shall be borne in mind that the mark

for proficiency in rate is intended to be sufficient in itself

to denote a person's ability, habits, and character; in short,

the individual's value to the service in the particular rate.

The following qualifications are then listed as the determining factors in assign-

ing marks of 4.0 and 3.5:

4.0 Competent and reliable in rate. Not less than 3.5 in conduct.

3.5 Competent and qualified in all duties of rate; has qualities suf- j i

ficient to justify advancement. Not less than 3.5 in conduct.^ J

3 Dureau of Naval Personnel Manual, pp. 181-1B2.

4 Ibid., H; 183; .. s -

i

1 This evident lack of distinction between the marks, and the emphasis put on

a single, broad, performance characteristic may account, in part, for the fact that

most of the Quarterly Marks were at, or near, 4.0.

With this brief critique of the characteristics of Quarterly Marks, let us

pass on to the considerations which governed the design of the rating scales

developed for this study.

Considerations for the design of a rsting scale

A rating scale is a device that should help an officer or petty officer

organize and express his evaluations of the men who work under him. In order that

they be useful, such evaluations depend to a great extent on three things:

1. The format of the scale. Does it provide reference points by which

the rater can reliably determine where each man should be placed on the

seal»?

2. The wav relevant aspects of o< . foi,.iance are described. Do they help

the rater "break down" the job? Do they bring out the different

features of performance? Do the trait descriptions help the rater

visualize the man or. the job?

3. The wav "levels of performance" are defined. Do they help the rater

separate the "good" from the "not so good", and the "superior" from the

"above average"?

Requirements of rating scale construction and design

Investigation of *he rating scale literature and consideration of the goals

of this research have led to the following general requirements which affect

rating scale construction and design:

1. A rating scale should be designed to reflect, insofar as possible, the

- 6 -

r differences among the Performance levels of the men bei«? rated. That

is, ratings given to a group of men should be spread over a relatively

large portion of the scale and few men, if any, should receive the same

score. The scale should yield scores that discriminate.5

2. A rating scale should be designed so that '.he "average rating" of a

typical group of men falls somewhere near the middle of the scale.

The bunching of scores at the upper end of the scale, so typical of

ratings, is unrealistic and reduces discrimination as well. (It will

be remembered that the Quarterly Marks had a tendency to be bunched at

the upper end of the scale. This was a fault of the scale design as

well as a result of overly lenient rating practices.)

3. A rating scale should be designed so that different raters will agree

on the position occupied on the scale by each roan rated. This is very

difficult to achieve when raters are forced to rely on descriptive

adjectives or numerical values in establishing key points on the scale.

More complete verbal descriptions of observable bits of behavior should

be of some help to raters in agreeing on the positions their men occupy.

However, the format of the rating device can also be such as to promote

agreement.

4. The design of a rating scale should be such that it will help the

raters using it to be consistent in their ratings of the same men from

one rating period to the next. This is reliability in the test-retest

sense. Assuming raters had no recollection of scores given at an

** This assumes, of course, that men are actually different fror" each other in the performance characteristics on which they are to be rated. From present day knowledge of individual differences this appears a reasonable assumption.

1

- 7 -

earlier time, there s';ii 1 should be a high relationship between scores

given to a group of men at interval? of, say, 90 to 180 days.

With these requirements in mind, the actual construction of a performance

rating scale was begun.

Format of the experimental scale

Rating scale« typically are single-page arrangements with the name of the

man to be rated at the top of the page and a series of traits on which he is to be

rated listed down the side. Performance levels on this type of rating form are

usually expressed in terms of some scale of numbers or adjectives, or both. For

example, a man may be rated as "excellent" or "4.0" on one characteristic, or

"good" or "3.5" on another, and so forth. This kind of rating form, though simple

in design and easy to >se; minimizes the likelihood of obtaining useful ratings.

It fails to provide the raters with a satisfactory framework for comparison. The

adjectives or numbers used usually have quite different meanings to different

raters at different times. Either, or both, of the*» conditions leads to unreliable

ratings. Words and numbers on a rating scale lack absolute meaning and are thus

subject to as many interpretations as there are raters.

It is exceedingly difficult to state how absolutely good or poor a man is

in his work. It is much easier to say he is good compared to this or poor compared

to that. The more concrete and meaningful are the reference points against which

these necessary comparisons must be made, the more objective and reliable will be

the resulting ratings. Something more than numbers or adjectives is needed although

Liicjf nay be of limited help. The history of ratings suggests that this something

mote may be achieved by evaluating a man's performance relative to that of his

peers.

In most job situations, men belong to an identifiable work group. Each mar.

- 8 -

within the group either does work similar to that of the others, or works under

approximately the same conditions, or bcth. The members of this group provide

meaningful, stable points of reference for ell raters who are sufficiently well

acquainted with the group to do the rating,

A man's assigned rank in a group on any one trait, or his average rank in a

group on all traits, is an indication of his level of performance based on meanina-

iül reference points — namely, the other members of the work group.

The performance rating scale wns designed with this man-to-man comparative

procedure in mind.° The following features were considered important: (A copy

of the final form of the rating scale may be seen in Appendix B.)

1. Space was provided on each page of the rating scale booklet for the

names of all men within a given work group. Each rater rated each man

aj. ä member of a. particular wprk group. In this study the group was

either the EM or EN gang aboard submarines.^

2. A single performance trait was listed on each page of the booklet. All

men were therefore rated as a group on ONE trait at a time.

This procedure of rating a group of men on each traic separately

is to be contrasted with the usual procedure of rating one man on all

traits without direct reference to the other persons to be rated.

3. The definition of each trait was followed by four series of statements

down the side of the page defining various degrees of possession of the

6 The design was not completely original. It was first tested by Stevens and Wonderlic, "An Effective Re-.'ision of the Rating Scale Technique," Personnel Journal, 13; 1934, 125-134. It was recommended by Guilford, Psychometric Methods McGraw-Hill Book Co., New York, iy3t>, Ü&3-264. It was further tested by Gilinski "The Influence of the Procedure of Judging on the Halo Effect," American Psy- chologist, 2:309-319, 1947.

' In subsequent work, groups have been homogeneous with respect to pay grade as well as rate.

- 9 -

trait in question. The top-most group of statements described

exceptionally outstanding behavior. The next i^ioup described

clearly above average performance; the third group described

average or satisfactory performance and the last group of

statements described below average or undesirable performance.

Thus, three of the four groups of statements described

acceptable performance or better. This was done in an effort to

keep the average rating down toward the physical center of the

scale, recognizing both the tendency of raters to rate theiT

men high and, at the same time, the natural selection and admit-

tedly superior performance of most higher rated petty officers.

4. In a column underneath each man's name was a continuum line.

This line was unbroken except for short horizontal marks which

located on the continuum the position of each group of descrip-

tive statements. The raters could thus locate where, on the

line, a give« description of behavior would fall* They could

then decide whether the performance of a man being rated was

equal to, better or worse than 3 particular description and rate

him at any appropriate point along the entire continuum.

On eacli page of the rating scale booklet, therefore, appeared

a definition of the trait on which the men were to be rated, four

statements defining the degree of possession of the trait, and

the names of the men being rated. Below each of the latter was

a column, representing a continued, in which the ratings were to

be made with a check mark.

Each time a man was rated on a niven trait, the ratinns

assioned other men had to be taken into consideration by ;the

- 10 -

1 rater. Tims, each man was directly compared with each other man

in the group on each of the traits.

The selection of performance traits for the ratine scale

To explore as many facets of shipboard performance as possible, a wide

selection of traits was made for the preliminary form of the rating scale.

These traits, which are listed below, were cleaned from a study of shipboard

jobs and from conversations with Naval personnel:

Social Adjustment Adaptability Quality of Work Leadership Neatnesb of Appearance Overall Efficiency

Cooperation Neatness of Work Watch Standing Ability to be Taught Knowledge of the Job Care of Equipment

Discipline Ability to Troubleshoot Application and Initiative Sincerity in the Job Dependability Manual Skill

Overall Efficiency in Hate (within pay grade)

In additiv to an overall rank order preference, the final form of the

rating scale consisted of ten traits. These were:

Cooperation Neatness of Work Knowledge of the Job Care of Equipment Application and Initiative Ability to Troubieshoot Judgment and Common Sense Sincerity in Doing a Good Job Leadership Discipline

Deiails of the analysis that led to this final selection of traits are

reported in Chapter III,

Use of the performance rating scale aboard ship

As an exploratory study, the preliminary rating scale was used to obtain

evaluations of 107 El.i's and EN's of Submarine Squadrons 3 and 7 at San Diego.

il -

1 AH pay grades fron strikers through chief petty officers (CTC's) were included.

The men rated were selected solely on the basis of availability.

'<• revised preliminary form was used later to obtain evaluations of

206 submarine Et.i's and EN's» Analysis of these two preliminary forms led to

the development of the final form which is included in Appendix B to tiiis

report.

The final form of the rating scale was used to evaluate the same groups

of EM's and EN's that were subsequently evaluated with the performance check

lists and tested with the practical performance tests described in Parts I and

III of this report. These groups included 320 EM's and 4C7 EN's, the greater

majority of the men being strikers and third class petty officers.

An effort was made to get as many raters as possible who i'elt they

knew the men to be rated well enough to rate them fairly. In most instances,

this meant the Engineering Officer, the two leading Chief Ell's, and the two

loading Chief En's. In some instances, ratings were made by the Executive

Cfficer or Assistant Engineering Cfficer.

The majority of the E'.i' s and EN's were rated by either one officer and

two petty officers or by two officers and ono petty officer; so, typically,

at least three ratings were obtained on each man.

Instrnctlons to the raters

Written instructions, together with sample ratings, were given to

all raters. (Examp'->s may be seen on pages 1 and 2 of the rating scale in

Mie Appendix.) In addition, in the preliminary study raters were instructed

personally by project personnel. Common pitfalls of rating, such as excessive

leniency, halo effect, and high interrelationships among traits due to logical

errors were explained to the raters in simple terms and they were asked to

m 19 -

guard against then-,. During later administrations, time considerations prevented

personal instruction and the written instructions hod to suffice.

The written instructions emphasized the following:

1. The differences, no matter how small, in the performance

levels of the men being rated should be brought out. Men

in each work group should be compared with one another on

each trait. It was Suggested that the poorest and best

man in each group ("best" or "poorest" on the trait in question)

should be rated first, and then the other men rated in between.

Haters were instructed to avoid giving tie ratings if at all

possible.

2. The men's pay grades should not be unduely influential in the

in the ratings assigned, iler. were to be rated on each char-

acteristic strictly on the basis of their demonstrated perform-

ance. Haters were cautioned against the general "halo-effect"

of pay grade.

The halo created by pay grade is a real problem in analyzing ratings

given to Naval personnel. It is natural for an officer or petty officer to be

affected by the man's pay grade when he evaluates a man's performance. Part

of this influence, no doubt, is the result of genuine differences in ability

reflected in the different pay grades, but part very likely is spurious.

The effect of pay grade on the usefulness and reliability of ratings

was examined in considerable detail during this study. Most statistical

analysis reported in later chapters '..'ere performed both on a within and across

pay grade basis in order to answer the many questions involved.

'3 -

f

Scoring the ratings

It will be remembered that each definition of a performance trait on

the pages of the rating scale booklet was accompanied by four groups of state-

ments down the side of the page defining the degree of possession of that

trait. Across the top of the page were listed the names of the men to be rated.

A check mark in a column opposite one of the series of qualifying statements

or between two such series, and under a particular nr.n's name, indicated that

man's rated performance level on a particular trait.

The distance of the check mark from the lower end of the column was

converted into a numerical score by measuring it with a centimeter rule, ail

columns were thirteen centimeters long, and so possible scores ranged from

0,0 to 13.0, Each check mark was read to the nearest half centimeter*

These raw scores varied a good deal from rater to rater in central

tendency and spread. To make ratings from different raters comparable, it

was necessary to convert them into standard scores. This had the effect of

producing a common average for all ratings and reduced the effect of differen-

tial spread.

'Standard scores were expressed in terms of the Sten-scale, Scores on this scale ranged from 0 to 9 inclusive, with the mean at 4.5. Each unit on the scale covered one-half standard deviation.

- 14 -

B ü

Chapter III

RESULTS FROM SHIPBCARD 'JSE OF THE RATING SCALC

A, Measuring Characteristics of the Scale

Average rating and spread of ratings over the scale

In Table Ic appearing on page 17, the mean and standard deviation of

ratings ore reported for each trait of the preliminary form of the rating

scale. This analysis was not repeated for the revised form, there being no

suggestion of any differences in results. If anything, the variability of

ratings on the revised form probably was slightly greater than that reported

here because of the specific instruction in later administrations to avoid

tie ratings. To indicate the variability of the means and standard deviations,

the middle C& rar.a's of values has bee» inr.ludpd in the Table.

In many rating scales there is a notable tendency for raters to reduce

the usefulness of their evaluations by;

(1) Assigning high ratings, on the average

(2) Indicating little or no differences between men (small

dispersions).

As indicated in the previous chapter, it was thought that the format of this

scale, together with the manner in which the trait descriptions were graduated,

would reduce both these tendencies appreciably,

In large part these expectations were realized. There was, of course,

variability in the leniency - stringency factor, mean ratings for individual

raters extending all the way from 5.9 to 10.C on the 13-point scale. However,

the mean of mean ratings for 43 raters in the preliminary study was 7.C — quite

acceptably near the center of the continuum, 6.5.

- 15 -

In general, the dispetsicr.c of ratines were sufficiently large to

provide meaningful and useful discriminations. There were a few raters who

assigned a large percentage of tie ratings, thus producing small standard

deviations and little discrimination. This led to the instruction, for later

administrations, to avoid tie ratings if at all possible.

Considerable variability can be seen in the means and spread of scores

from trait to trait. This could be due to one of several factors or in part

due to all of them. One possibility is that the variability resulted from

differences in the extremeness of phraseology of the descriptive phrases for

the several traits. A second possible explanation is that Täters may have

considered it more logical or essential to assign high mean ratings for some

traits than others. For example, a rater might reason that it was perfectly

satisfactory to assign moderate or even low ratings in a trait sach as

iianual 31:111. but night consider it a reflection en his own performance if

anything but high ratings were assigned in a trait such as Discipline.

Finally, a third possible explanation of differences in means and variabil-

ities of the several traits is that they reflect differences which, in fact,

äfi exist. For example, the mean score in Leadership may be low because a

substantial proportion of the men rated actually have had no opportunity to

display leadership. The mean rating in Discipline may be high because the

majority of men offer no disciplinary problems. The variability in Manual

Skill may be small because there is little opportunity to demonstrate dif-

ferences in motor skills in the particular rates studied.

1

- 16 -

r «r-~

p5

&-S-

fcj 0> C»

WJ w t- O * CO 2 3^

Be-

0) o C (/]

Vi • Q

••» •

B W U «M M O

1/1

O 0) t/5

CO* ra M o a> .

*•* --» if".

0>

Ö «0 W> li B

0) (0

?£ I

M H B <-4 « n v

o- (0

B H re v

_« ••> 5 «

re

cJcvievicoc^f^fococsicorsc^eocNicoconcOcJco l • ; • : : i i i l i 1 l I I i I I l l

OOWOi-iWOOOCOCNOCSOO'rJCOCO^H

000»-«^**-<©i-«O^Q'"*0000-«00©

c>' d o o' o e« H" es d o o r-' d c>* o o o> o :' o

I I I I I I I I I I I I I I I I I I t I

ft) o> •»* « a •J s ^-2 ^Mj

s *2 0> o •w tn o B

B •J «-> JS •PH

E •M E JS to c O >x en o> a-

•J ca >-! i Ü 3 *J r-H u B4) .O E js! « C J3 -O B V) v^ a o BMC D>»-S B j «J hH 0) BO

•HO g o••? 0) B

** o -: c C3 >» « — 0) & M •H J3 >H H S

u t) W 2 «P4 VH T3 »J >> i-< •H 3 <M C 73 O B c •-« «-> <M <H 3 —< f—i >^ •nvi o a a o> C lOH'H 6*1 o c c- o •»( <M ^^

"w O >H (8 A) c •»* ••H f—( trf U » U «•» >.* W ^ ^ <£ V) «-> <-> C> •^ <-> |J J'rt «Si </» «-> l/J re — *J o u £)

*—( f8 a ca JO </5 H CO >,t« >,.H —1 £ c U 4> -o a E E «->

M p-i 4) «-> O «-> H ^H l—t 0) •M r-l *i B, U 5

•i-i •rt 01 (3 E -H .^ 0) C5 co > o •—1 O 0> C\ T3 H fri (1H U 3 k o

o a c o » c </> CD C C3 re 0) «1 -rt U -M c E 0) •H e. 3 a> -a 4> > 0) .a « .a -H re ^

I/11HZU2^ Ui < TU^i —j w Z si U <4 vl M o

i- CJ (O T m <o r^ OPOH NnTin^soc-o

en

CO

>0 o

o I

CO

t in

o CO

p—i

i

1 i

Reliability of the ratings

"!i important feature of any evaluating instrument is the consistency

with v/iieh it measures. Few reports on rating scales of the traditional type

indicate satisfactory reliability and, in nany studies, reliability in the

ratc-rerat.e sense is not even reported.

Having established that raters could assign meaningful and discriminat-

ing ratings, it was next desired to ascertain how consistently (reliably)

they did this.

Cne hundred fifteen subjects were common to the two preliminary

administrations of the rating scale and also had the same raters each time.

These were non-selected cases from six submarines, being singled out solely

because of participation in both trial runs. The elapsed period of time

between the first and second ratings varied from five to as much as nine

months for the various raters.

Since fairly high reliability would be expected on the basis of knowl-

edge of a man's pay grade alone, it was decided that the reliability should

be estimated on a within pav grade basis. That is, for each pay grade, it

iras determined whether a given rater placed his men in essentially the seme order

(based on total score) during the second evaluation as he did on the first. The

self-agreements and disagreements then were plotted in a four-lold table (split at

the median) and the correlation (tetrachoric) was computed. Tor the 115 cases

(over 300)ratings) the correlation was rt = .80, This compares very favorably with

reliability figures usually obtained with more objective measuring devices,

„..„u ~ .-. .-..- ...-.•„-. *.-..-. I ..». 1 •«,,*,. 1 i~ „„„,.; .1~~-U 1 .. t«Jnl 4.L„_ »i. _ _. 1 j _i_ ; l _• » _• OUl'll Ui) MJIVIII/lUtUUUl bUALüi UIIU AO 1/UUOiUCl UU1] J1 X S | *' C L lllUtl LUC 1 fl 1 ÜU4 1 1 Llt'3

usually attributed to rating scales (.60 to .70).

- 16 -

(

acreement amonn th*> raters

In addition to reliability in the rate-rerate sense, a useful rating

device must promote substantial agreement among raters on the relative pro-

ficiency of the men they are rating. Indeed, ratings probably can be consid-

ered a satisfactory criterion of performance in direct proportion to the amount

of agreement among rrters, provided, of course, the characteristics of perform-

ance being rated are relevant to the ultimate criterion.

Inter-rater agreement was determined by correlating one ratev's judg-

ments on each trait with those of each other vater w!;o had rated the same

subjects. Thus, if a pair of raters both considered a man either above or

below the median on a certain trait, it constituted an agreement. If, on

the other hand, one rater considered the man above the median and another

considered him below it, a disagreement was recorded. Dy considering the

regression of each rater on each other rater, a four-fold table of frequencies

representing agreements and disagreements was established for each trait

in the scale and for average score. From these four-fold tables, tetrachoric

correlation coefficients were computed which served as indices of agreement.

The question again arose as to the role of pay grade in determining

Ihu Magnitude of inter-rater agreement. For example, it seemed a reasonable

assumption that most raters would generally agree that a first-class petty

officer was better than a third-class petty officer or a striker, Thus 'the

factor of pay grade a]one would create considerable agrecnent among raters.

It was considered necessary, therefore, to determine whether there was a:iy

basis for arreement amonc; raters independent of the factor of pay grade.

Each rater's subjects were grouped by pay grade. On each trait

these pay-grade croups were divided into upper and lower sub-groups with the

division at the median. (If there was only one subject in any given pay

- 10 -

i

grade, he was omitted from the calculation.) Those subjects in the uppiT

sub-group were assigned a passing score (-») v;!iilc these in the lower sub-

group received a failing score (0).

With these rcores assigned for each rater for each trait for each

pay grade, the intercoirelations of raters' judgments was again a matter of

plotting a four-fold table of frequencies. All pay yvades were plotted on the

same table and ouch subject appeared in the table as many times as there were

pairs of raters who judged him.

The coefficient computed from this table rosy be termed a within nay

grade coefficient. The results for each trait, under each of the two conditions,

and for both of the preliminary samples, are shown in Table II on the following

page. In addition to tho agreement on the single traits, the agreement on

average scores is indicated. The values for the traits of Neatness of Arnearance.

Dependability. Adaptability. Ability to Troubleshoot. and Manual Skill do not

aopear for the second sample since those traits were eliminated from the second

form of the scale.

Severa1 of the results reported in Table II require comment. There

is considerable variability in the size of the coefficients from trait tc

trait as might be expected. Considering both samples and both conditions

of correlation, the trait Social Adjustment seems to be about the least objec-

tive. This is reasonable in view of the obvious difficulties involved both

in defining Social Adiustment and in agreeing on objc cive indicators for such

a trait.

Among the traits leading to the highest degree of agreement under both

conditions of correlation were Knowledge of the Jojj, Discipline. Judgment and

Common Sense and Leadership. It is feit that the behavioral referents for

20 -

r 1

TABUS II

INTER-RATER AGREEMENT* ON TIE VARIOUS RATING SCALE TRAITS AND ON AVERAGE SCORE, FOR BOTH SAUPLUS, IJITii PAY GRADE

IN AS A VARIADLE AND ',/ITll PAY GRADE HELD CONSTANT

Trait First Sample Second Sample (N=1G7) CJ=2G6)

1. Social Adjustment 2. Quality cf >«urk 3. Neatness of Appearance 4. Cooperation 5. Watch Standing 6. Knowledge of Job 7. Discipline 8. Application and Initiative 9. Judgment and Common Sense

10. Dependability 11. Adaptability 12. Leadership 13. Overall Efficiency 14. Neatness cf Work 15. Ability to be Taught 16. Care of Equipment 17. Ability to Troubleshoot 18. Sincerity in Job 19. f.ianual Skill 20. Overall Proficiency in Rate

Pay Pay Pay Pay Grade Grade Grade Grade

In Out In Out .32 .26 .46 .40 .53 .56 .55 .39 .42 .42 - - .30 .58 .50 .57 .45 .44 .60 .59 .66 .61 .7U .50 .60 .07 .62 .58 .49 .59 .60 .54 .61 .67 .69 .46 .47 .43 mm mm

.S3 .63 M -

.60 .55 .72 .56

.54 .62 .65 .51

.51 .48 .64 AA

AH .53 ,55 -42 .42 .61 .56 .35 .60 ,54 «. mm

.44 .60 .60 .44

.58 .45 - mm

.57 .63 .66 .40

Average Rating Scale Score .67 .73 .7< .59

* Tpt.rachoric Correlation Coefficients

I !

1 these traits are much easier to define and their manifestations easier to

observe, which may explain the greater degree of agreement on them. Further,

they are all traits which reflect a basic ability to perform and, as such,

may tend to correlate with each other.

The factor of pay grade raised or lowered the inter-ruter agreement

depending upon circumstances. In the first sample, the removal of pay grade

variance caused the majority of the coefficients to increase, v.'hile a few

remained the same and a few decreased in size. In general, the increase

in agreement occurred with traits which are not so highly related to pay

grade. Thus it may have been that instructions designed to decrease inter-

trait correlations and reduce halo were somewhat effective with the prelim-

inary sample, and this fact showed up in the inter-rater agreement, It was

apparent, in seoriüy the ratings, that pay grade was affecting various raters

differently. Some were overcome by its influence and rated strictly in

accordance with pay grade level on every trait. Others remained more objective,

their ratings in many traits showing much smaller relationships to pay grade.

If both kinds of raters picked the same men within pay grade level as best

and poorest, then removal of the effect of pay grade should increase inter-

rater agreement. It is believed that this is what happened in the first

sample.

In the revised preliminary sample, licever. the general effect ot

removing pay grade variance was to reduce the inter-rater agreement. It

is c-onr.luueu Llitit tlic reasoii for this reversal of the cp.rlicr results was

due to the instructions (this time the lack of them) given to the raters,

Since there was practically no personal contact with the raters during the

second administration of the rating scale, and since many of the raters in

21

I

the second sample had not participated ir: the first ratings, it is reasonable

to hypothesize that the logical and halo effects created by the presence of

pay grade were greater in the second sample than in the first. This hypoth-

esis is substantiated by the fact that the correlations of pay grade with

other rating scale variables was higher in the second sample than in the first,

and also by tne fact that trait intercorrelations were higher in general for

the second sample. It is suggested, then, that pay grade variance contributed

relatively more to the magnitude of the inter-rater agreement in the second

sample, and its removal caused a reduction in agreement.

In concluding the discussion on ihe agreement ol raters, a word is

in order about the level of agreement obtained. In general it was more

substantial, both on individual traits and on average score, than is usually

reported with conventional scales. Undoubtedly exclusion of some of the less

objective traits would have increased the agreement on average score appreciably,

Intercorrelations of traits and results of factorial analysis

A survey of the literature on rating scales revealed few studies which

had been carried as far as factorial analysis.X,|W These studies, with the

exception of that by Bolanovich, plus well established indications of rating

fallacies such as halo effect and logical.errors leave one with the impression

that seldot.i are more than one or two factors required to account for the

high inter-trait correlations which typically are. found in rating scale studies.

Holanovich» D, J„ "Statistical analysis of an industrial relations chart," J. of ADDI. Psvcholocv. 30, 1946, 22-31.

-Chi, fan-Lin, "Statistical analysis of personality »ratings,"J, of Expprinr-n- tal Education. 5, 1937, 229-245.

3Cwart, E., Seashore, S, E, and Tiffin, J., "A factor analysis of an industrial merit rating scale," J. of Appl. Psychology. 25, 1941, 401-4C6.

- 22 -

r k part of the moiysis of the present rating scale wa.> to determine

the magnitude of the inter-trait correlations and to pertor:,! factorial analyses

of the resulting natrices.:;; This was done for each of two preliminary samples

of submarine personnel, under three conditions for each sample with a resultant

total of six analyses. The various conditions of these analyses are described

below.

Intercorrclation of traits. Procedure I. In the initial analysis,

traits were intercorrelated in a straightforward manner. Individuals

were assigned plus scores in all traits in which they received overage

STEN-scores of five or better, and minus scores in all traits in which

they received STEN-scores of four or less. Tetrachoric coefficients

were then computed.

In addition to the rating scale traits, each man's general Classifica-

tion Test Score (CCT), his age, length of time on board, education

and ;.ay grade level were included in the matrix for analysis.

In the f'avy, pay grade is quite naturally related to inost traits that

make for success. This fact tends to increase the correlation between

traits on a rating scale such as the one under discussion, »".s in any

organization, those indiviudals who are at the most advanced levels

are rated higher because of their positions and, in turn, they have

been advanced to those levels because th~y possess more desirable

amounts of the tra. that are necessary for advancement.

In view of this, and the relative naivete of the raters, one would

expect the intercorrelations among the traits to be quite high. This

•Tables III through VIII, the intercorrelation matrices, will be found in Appendix /.,

- 23 -

r

expectation ivas realized as tne matrices in TabL? Ill (first sample)

and Table IV (second sanple) indicate. (Because of their size, these

and t!ie other correlational matrices, Tables V, VI, VII and VIII

appear in Appendix «'•.) The inlercorrelations in the second sample

were somewhat higher than those obtained from the first sample.

This was due probably to less personal instruction of the raters in

the second sample, and to the greater dispersion of scores as a result

of the instruction that no tie scores be given.

[« word is in order in regard to the reliability of the coefficients

in these and subsequent calculations. The standard errors aie less

than the N's of 1G7 and 286 for the two samples would indicate. This

is true because of the fact that the cell frequencies in the four-

fold tables did not total N, but rather added up to the total number

of pairs of ratings. Since there were two to three raters for every

ratee, this total was considerably greater than N«]

Inspection of Tables III and IV indicates that pay grade is highly

associated with technical skill as reflected in the traits of Iinowledre

of the Job. Judrnent and Common Sense. Dependability. Leadership.

Ability lp Tioublcshooi. and /.re. Less highly related to pay grade

is a group of traits which is eorsidered to indicate the device of

personal adjustment to Navy life. In this group arc Social Adjustment.

Neatness of Appearance. Cooperation. Discipline. ;'.ppl ination and

I illative. Ability to; b_e Taucht, and Neatness of. Work. Length on Board

is seen to have a small relationship to most traits, and Education and

CCT have practically none, (Submarine personnel are pie-selected on

GCT.)

- 24 -

^R3^

interCyrre.Tation of traits. Procedure II (Variance within pj}_y. crade).

Because of the substantial relationship of pay grade to most of the

rating scale trait scores, and because of the great range of abilities

represented in groups containing everyone fron strikers to chief

petty officers, within pav grade intercorrelations were conputed

next, treating a man's scores as high or low relative only to his

own pay grade average. If some foot or (s) other than pay grade was

in part producing the intercorrelations, then the resulting matrices

should contain significant coefficients. If, however, pay grade were

the only variable operating to produce the intercorrelations. then

the matrices would contain only near-zevo coefficients.

In Tables V end VI it is possible to obtain a very general picture

of the effect of removing pay grade variance. It will be noted that

either method of removing pay grade leduced the intercoiTelations

of the traits appreciably, but did not by any means reduce them to

near-zero. The amount of reduction can be discerned from the sums

of the coefficients of the matrices. In the first sample, the sum of

the coeffecients, Table III, was 162.39 (25 variables -disregarding

the signs). In the reduced matrix, Table V (24 variables), it was

115.37. In the second sample, the sum of the coefficients, Table IV

(21 variables), was 125.C9. In the reduced matrix, Table VI (20

variables), the sum was 74.61 . This greater change in the second

sample supports tiie hypothesis that lack of instructions during the

second administration increased the halo due to pay grade, ?nd thus

its contribution to the total variance.

/»s testimony to the fact that the within pry grade variance technique

- 25 -

was effectively removing tha variance due to pay grade level alone,

the correlation of the rating scale variables with the trait of Age

may be examined. In both samples, age con elated very iiir;iiiy with

pay trade (,03 in the first sample, .94 in the second). P.e^ovvnr the

effects of pay grade, then, should be tantamount to removing the

effects of age, and the correlations of Ace with the rating scale

variables should distribute themselves rather closely about zero.

Examinations of the correlations with age in Tables V and VI reveals

that in both samples this did occur, over three-fourths of the coef-

ficients having values between plus and minus ,15, From this evidence

it appears that the reducing procedure accomplished its purpose.

Intercorreiation of traits. Procedure III (pay evade prit railed out).

As an addition?.! check on the technique of reducing within pc.y grade

variance, however,.it was decided to compare these results with those

obtained by partialling out statistically the effects of pay grade.

This was accomplished by starting with the original matrices and re-

moving the pay grade variance from each coefficient by conventional

partialling techniques,H The pcrtialled matrices resulting may be

seen in Tables VII and VIII for the first and second samples respect-

ively.

In comparing specific corresponding coefficients in the reduced and

partialled matrices, considerable variation can be seen. Apparently

the two procedures did no^ produce precise!" the same effects. However

a comparison of the sums of these matrices for both samples yields

striking similarities. In the first sample, the sum of the reduced

%J, F, Cuilford, Fundamental Statistics in Psychology and Education (New York: McGrctHüll Cool: Coupany, 1942), pp. 26C-271.

- IiÜ -

matrix, Table V, was llf>.07 and that of the lartialled matrix, Table VII,

was 117.67. In the second sample, the sum of the reduced matrix, Table

VI, was 74.61 while that of the partialled matrix, Table VIII, was

79.C6 . Thus the two methods removed approximately the same amount

of variance in each sample, but they treated individual coefficients

somewhat differently. The explanation for this must lie in the fact

that what was accomplished precisely by mathematics was dependent on

the correlation of pay grade with the other variables derived from

the entire sample, while in the reducing procedure each individual

case required a decision as to who was high and who was low within

each pay grade. In other words, if the same amount of variance were

due to pay grade in the case of each coefficient, then the pcvtialling

procedure should correspond very closely tö the reducing procedure.

It is believed that this was not the case in every instance and thus

discrepancies between the matrices resulting iron the two procedures

were to be expected.

r.s a final comment on the pertialling procedure, it should be pointed

out that the correlations of Age with the rating scale vaiiables again

distributed themselves about zero as they did in the reducing procedure.

Again, over 70 percent of the corielations with age were within the

range of plus or minus ,15. There were a few more lauje coefficients than

with the reducing procedure, however, and a laiger number of negative

coefficients..

The discussion on the reducing and partial liny procedures should not

be concluded without recognizing thrt these techniques probably removed

m i. I

loo rauch vüiü-.nce. That is to say, if many of Lliö qualities under

consideration are genuinely related to pay grade, then removal of tlie

effects of pay grade eliminated real, as well as spurious, variance,

Evidence that this was the case occurred particularly during the analysis

of the second sample of data. ".Jhen pay grade was partial led from this

motrix, a substantial number of negative correlations arose between

the rating scale traits and such variables as Aoe. Lencth in Service,

and scores in the General Clrssification Test. This, in turn, required

a reflection of those variables during *.!.* factoring procedure, which

led eventually to a few rather substantial negative factor loadings

for these variables. These loadings cannot be interpreted in the

usual sense, and are regrrde<l as artifacts of the reducing and partial-

iing procedure?.

Cesults of the factor analyses. frying computed the inter-cor;elation

of traits under three conditions for each of the two camples, six

factor analyses were next performed using the Thurstone method. The

purpose of these analyses was two-fold: (1) to determine the extent

and nature of the factor structure of the rating scale under each of

the three'conditions of corieiations, and (2) to determine how similar

the factor structure was from one sample to the next.

extraction of factors was continued as long cs the cross-product

öf any pair of factor loadings exc^nded <:he standard error of the

zero-order coefficient between the corresponding pair of traits in

the original matrix. For the sake of consistency and in order to intro-

duce a minimum of assumptions, all rotations were orthooona1 and the

- 20 -

ciiteric for otation were positive manifold end sinp'e structure.

Flotations were accomplished according to c. graphic netliod Cose:ibed

, ... 0 jy wir.ir.iei-nnn.

In assembling the traits which comprised the rating scale, it was

felt that there were temperamental components of performance aboard

submarines which were of as much importance zz technical knowledge

and skills. Theoretically, at least, the temperamental components

should be independent, of the technical components, although for any

given population they might be conelated. For this reason., orthogo-

nal rotations were utilized which, in fact, rave very satisfactory

solutions for the original data from both samples. For the sake of

determining whether or not the factor structure was altered by the

reducing and -»artialling procedures, orthogonal rotations we:e emp'oyed

in those analyses also. Here the results were not nearly as satisfactory

from the standnoint of identifying factors. It may be that an oljlirue

solution would hove resulted in more meaning for these sets of dztr..

However, such an investigation was not considered advisable because of

the labor :ecuired.

In Tables IX._ II and .XJ (h ipendix /-.), the results of the factoria1

analyses neriormed on the data from the first samp'e may be seen.

In Tables XII, XIII and XIV, are the corresponding results from the

second sample. The most remarkable feature of these results is the

fact thrt the factor structure increased in dimensiona'ity when variance

due to pay grade was removed. In the first samp'e, there were four

^Zimmerman, W, S,, "A simple graphical method for orthogonal rotation of axes", Psychometrika, 11:51-55, 1946.

- 29 -

r i factors before tue effects of ?oy gratis were removed, five factors

in the reduced data, o»id six in the data in which pay grade was held

constant by partial correlation. In the second samp1.« this increase

uns fron three to five and seven factors for the matrices f:om which

pay grade was removed. Ho obvious explanation for this is readily

available. Cne reasonable hypothesis might be, however, that in the

first analysis for each sample, pay grade acted as a general factor

and increased the correlation between factors to such an extent that

two or more factors emerged as one.

Identification of factors. The most readily identifiable factors were

extracted from the original matrices of both samp'es. In both these

analyses a factor called Technical Competence emerged which was

practically svnonomous with pr.y grade. The traits and other vailables

having the highest loadings in these factors are listed below:

First Sample

fay Grade Knowledge of Job

Leadership Judgment and Common Sense Ability to Trcubleshoct

Second Sample

Pay Grade Kge Length in Service Judgment and Common Sense Knowledge cf Job Leadership Quality of Work Overall Efficiency in Rate

Technical Competence

.97

.70

.75

.69

.64

.63

Technical Conpetence

no

.95

.92

.00

.60

.70

.74

.73

- 30 -

In both original analyses a factor also emerged which seemed best

identified as Personal Adjustment of some kind. Involved in it were

attitude toward the job and shipmates, effort, sincerity, dependability,

etc. The traits having tiie l'nhest loadings in this factor for each

sample are listed below:

First Sample

Cooperation Application rnd Initiative Ability to be Taucht Adaptability Sincerity in the Job Overall Efficiency Care of Equipment Dependability

Second Sample

Ability to be Taught Watch Standing Cooperation Application and Initiative Care of Equipment Sincerity in the Job Social Adjustment Quality of Work Overall Efficiency

Personal Adjustment

.79

.77

.75 ,73 .69 .69 .67 .64

Personal Adjustment

.75 -70 .6C .65 .64 .64 .62 .60 .60

Also identified from the original data of the first samp'e was a factor

which apoears best described as Carefulness or Neatness in Work and

I'erson. Traits having the highest loadings were:

First 5ano\e

CuaTity of Work üeotness oi Work Care of Equipment Overall Efficiency in Este Discipline Neatness of Appearance

Carefulness or Neatness

.56

.52

.50

.45

.40

31 -

This rather well defined factor was not. identified in t'19 analysis of

the data iro<r. the second sample.

The fourth and final factor emerging from the original data of the

first sample appeared to be Sö(?C sort of efficiency or Job Performance

factor. The traits with principal loadings were:

First Sample

Neatness of iJork Overall Efficiency in Rate Watch Standing Overall Efficiency Sincerity in the Job Manual Skill Ability to Troubleshoot Judgment and Common Sense Quality of Work

Efficiency or Job Performance

.59

.57

.55

.46

.45

.44

.37

.37

.34

The third and final factor emerging from the original data of the second

sample was found only in that snmp e. While all loadings were low, it

appeared to be best identified with Maturity or Experience;

Second Samo e

Length on Board Education Knowledge of Job Judgment Leadership

Maturity or Experience

.46 «41 .30 .35 .33

ihe iactors extracted irom me reaucea ana -rrtiHiien matrices were

very difficult to identify. Traces of the factors extracted from the

ui luiiia; uula. 11SILU oJUvCi v^uuiu D*3 occll LUX UUU:IWU L. A J5« IUUUJHI\J;J;

of course, were very different due to the removal of pay grade vaiiance.

An attempt was made to classify the factors into two brood groups —

those of a Technical nature and those of an Adjustment nature. Even

32 -

with such a bvoati classification, some factors seemed to be as much

a member of one class as of the other. Because of th" possibly

limited significance of these patterns of locdings, they will not

be reported here.

The difficulty in identifying these last groups of factors piobably

rests paitly with the fact that too few experimental variables were

included for identification of the large number of factors, which

theoretically might be rated. To this difficulty may be added the

fact that the rating scale traits were not sufficiently definitive

to permit a distinction between two factors which were comprised in

large ptut of the same traits. For example, in the field of aptitudes,

identification of a factor in which many tests have loadings can be

made v:ith considerable confidence if a relatively pure test (such r.s

number operations or vocabulary) is highly saturated with the factor.

In the present analysis, however, one would hesitate to identify a

factor as Cooperation, for example, simply because the trait called

Cooperation had the highest loading In that factor.

Identification of factors extracted from the reduced and partialled

matrices was incieased in difficulty also by the fact that these pro-

cedures removed a good proportion of genuine as well as spurious

variance. This left many of the traits with reduced loadings in certain

factors, and decreased differences in loadings between traits. This

resulted in less distinct patterns of loadings which were correspond-

ingly more difficult to identify.

- 33

Chapter IV

RESULTS OF THE SHXPDOMU) USE OF THE FATING SCALE

3, Relationships with other Peiformance Measures

The lerfornance rating scale was the first of three ciiterion measures

developed in the study of shipboard Performance of Nuvy eniisted personnel.

The results reported in previous chapters indicate that it provides a ueans

for obtaining reliable, disciii.iinat.inc measures of at least two ceneral aspects

of shipboard performance — one technical and one adjustive in nature.

Latcu in the study, performance check lists were developed to deterr.iine

how well officers and petty officeis could rate their men on ability to pei-

form specific tasks from their jobs. These were much more specific and

technical in nature tiian the general traits of the rating scale.

Finally, actual peiformance tests were constructed around tasks renie-

sentative of the Eti and EN jobs aboaid submarines. These provided an opior-

tynity to compare rated performance with actual tested performance.

Correlation of rating scale and check list evaluation?,

It would be expected that scores from the two rating measures would

correlate appreciably, simply because scores on both were substantially re-

lated to pay grade. Fart of the relationship of pay grrde to the ratings

reflected the genuinely higher average performance ievei uf fieri in the higher

pay grades, but pa;t, no doubt also reflected the tendency of some raters

to rate men high simply because they were of Mnlwsr pay grade and "should" be

able to perform better. It was decided, therefore, thrt the relationships

See Part III, this report.

^See Pait I, this report.

34 -

•. !

i

between the two measures should be investigated on a .vithin pay rrade basis.

Tn order to have an estimate of the highest likely relationship, however, it

was decided to correlate, across all pay grades, scores from the check lists

with perhaps the most technical rating scale trait, Ability to TJ oubleshoot.

The results, broken down by officer and enlisted rater, and by ob-

served and total check list scores, appear in Table XV, The very high

relationships obtained, particularly in the case of check list total scores,

suggest that the check list and technical traits of the rating SGale were

measuring essentially the same things if scorss were taken across pay grade,

T/.BLE XV

crccziÄTiCNS BZT.JZII; SCCT.ZS Cfl TIE n&TiriG GG.L: TTJ.IT 'V.3ILITY TC Tr.cuBLzsHcoT" um V/.MCüG eisen LIST scenzs

C.LL r.-Y GH/.tES INCLUDED)

scenz a: zr.

unlisted rater, observed scores ,78 (T! = 229) .71 (N = 246) Officer rater, observer scores .53 (N - .100) .52 (11 - .42)

Enlisted rater, total scores .89 (N = 234) .63 ill - 293) Officer rater, total scores .90 (N = '59) .79 (N = 215)

The within pay grade study «as made on striker and third-class petty

officers only. These were the groups singled out for special study as a

resvlt of the overall design of the pioject. O'-s a part of this study,

specially designed aptitude tests had been administered to men entering the

Submarine School, New London, It was intended thrt the relationships between

these tests and the shipboard criterion measures would i examined. There-

fore attention was focused on strikers and thirds since few higher rated

petty officers would have been recent Submarine School graduates.)

See fart III, this report for a discussion of "observed" check list scores.

The results appear in Table VI. C^neiaiions aie leportec! for strikers

and thirds separately and also across oay grade to study the influence of that

variable at the lower pay grade levels.

TABLE XVT

RELATIONSHIPS BETWEEN CiECl'. LIST AND RATING SCALE TOTAL SCORES

(Striker and Third-Class Petty officers, Siserial Correlations)

I 1

EM Efi

Strikers Strikers Strikers Thirds 6 Thirds Strikers Thirds C Thirds

Enlisted rater, observed scores .65 .53 Officer rater, observed scores .13 .40

Enlisted rater, total scores Officer rater, total scores

.45

.50

,5C .37

.61

.51

2C .42 25 .40

41 .61 On .23

.50

A'.

.56

.40

The last two lines in Table '.VI reveal the correlations between tote' scores

on the check lists and total ratine scale scores to be substantial even within

the lower ;>ay grades and in spite of the fact that some of the rating scale

variance was non-technical in nature. The inclusion of pay grade variance does

not appear to increase these relationships systematically at these job levels.

As a final check on the relationships between check list and rating

SC3le, Scores OH ??ch of th«? tw^ntv inriivirtiia1 traits on tlip nrol i m -i nnrv rntinn

scale -were correlated with total check list scores. These results appear in

Table XVII. It can be seen that, even on a within pay <;rade basis, the more

technically oriented rating scale ttaits tended to correlate higher with the

check list scores than those which were more adjustive or attitudinal in nature.

36 -

TABLE XVII

*I£ f tcr TfrpM crnorc RELATIONSHIPS BETWEEN PERFORMANCE CHECK LJ AND SCORES ON INDIVIDUAL TRAITS OF THE PRELIMINARY RATING SCALE

(Within Pay Grade N = 187)

Rating Scale Traits

1. Social Adjustment 2. Quality of Work 3. Neatness of Appearance 4. Cooperation 5. Watch Standing 6. Knowledge of Job 7« Discipline 8. Application and Initiative 9, Judgment and Common Sense

10. Dependability 11. Adaptability 12. Leadership 13. Overall Efficiency 14. Neatness of Work 15. Ability to be Taught 16. Care of Equipment 17. Ability to Troubleshoot 18. Sincerity in the Job 19. Manual Skill 20. Overall Efficiency in Rate

* Significant at the 5% level. «ft c*nni*-• JO/ 1~..-1

Correlation (Tetrachoric) wfth Check List Scores

-.10 .32** ,07 .14 ,32** .49** .29* .16 .40** .40** .37** .35** .46** .40**

.35**

.41**

.43**

.43**

.31**

For example, Knowledge of the Job. Judgment and Common Sense. Overall

Efficiency, Ability to Troubleshoot and Manual Skill all correlated very signi-

ficantly with total check list scores on a within pay grade basis. However, certain

traits associated with the adjustment factor in the rating scale, including Social

Adjustment. Neatness o£ Appearance, Cooperation. Application and Initiative, and

Ability to be Taught did not correlate significantly.

- 37 -

Cor r el at ions between rating scale score? and perform«?nee test_ scores

As a final study of the nature of rating scale scores as criteria of ship-

board performance, correlations were computed with scores on the job sample tests

and written job knowledge examinations. Again, these relationships were studied

first as they existed for all pay grades and later as they existed within the

striker and third class petty officer levels.

Table XVIJI reveals the across pay grade relationships between the ten

traits on the final version of the rating scale and average job sample test scores

and written job knowledge test scores. Generally, the traits Ability to Trouble-

shoot T Knowledge si the Job, Judgment and Common Sens«, Leader*l«ip and Care of

Equipment were more highly related to performance test scores than were other

traits. This tendency was more pronounced for Ef.l's than for EN's.

•

-

TADLE XVIII

CORRELATION BETWEEN ACROSS-PAY GRADE R'TING SCALE TRAIT SCORES, AVERAGE JOB SAFJPLE TEST SCORES , AND JOB KNOWLEDGE TEST SCORES

(Tetrachoric Coefficients)

Eh's» EN's*

dvpranp Job Averane Job Knnun PHHP Job Sample Knowledge

Score Score

.36

Score

.38

Score

.25 .40 ,50 .48 .34 .51 .30 .32 .36 .35 .41 .30 .;>£ .tu .53 .47 .38 .41 .57 .52 .45 .44 .43 .44 .46 .45 CO

. •-••-•

A-t •su.

,62 .53 .55 .56 .43 .36 .44 .47

Characteristic

Cooperation Knowledge of Job Discipline Application and Initiative Judgment and Common Sense Leadership Neatness of Work

Ability to Troubleshoot Sincerity in Job

* For EM's, N = 315 Strikers through CPO's; for EN's, N = 403 Strikers through CPO's.

- 38 -

t- similar analysis next was done on a within pay grade basis, including

strikers and third class petty officers only. These results appear in Table XIX.

It will be noticed that the relationships are smaller in every case. The cor-

relations still are appreciable for the Er.I's but in the case of EN's, the cor-

relations of job sample test scores with the rating scale traits reduced to nearly

iero. This was not true of the correlations between rating traits and the written

job knowledge test, however. For the written test the drop in correlations w3S

quite similar for bot« EM's and EN's.

•

TABLE XIX

CORRELATIONS BETWEEN WITHIN-PAY GRADE RATING SCALE TRAIT SCORES, AVERAGE JOB-SAMPLE TEST SCORES, AND WRITTEN JOB-KNOVLEDGE TEST SCORES

(Tetrachoric Coefficients)

EM's* EN's*

Average Job Average Job Job Sample Knowledge Job Sample Knowledge

Score Score Score Score

.20 .20 .02 .27

.48 .25 .16 .32 ,26 -.16 .04 !Io or» or» f\A 1A • uu ,S.\J ,\J-I m <-»-» **c» *>t- 1 1 •"t(~i

.*£• , ÖO • •"•«-> • v>v

.38 .27 .04 .25

.45 .27 .10 .28

.38 .23 .10 .35

.39 .32 .23 .36

.'iü .21 ,23 .39

Characteristic

Cooperation Knowledge oi' Job Discipline Appxiüütiüii and initiative Judgment and CuiTüiüii Sense Leadership Neatness of Work Care of Equipment Ability to Troubleshoot Sincerity in Job

* For EM's, N = 147 Strikers and 3rd Class PO's; for EN's, N = 190 Strikers and 3rd Class PO's.

In addition to the correlations for the individual traits of the rating

scale, total scores also were correlated with average performance test scores.

These values proved to be .14 for EN's and .38 for EM's. Thus, as might be

expected, certain of the more technically oriented rating scale traits correlated

higher with tested performance than did the average of scores from all traits.

- 39 -

-

Conclusion

Some of the results of the foregoing analyses may be regarded as encouraging

and some discouraging as far as the relationship between rated and tested per-

formance is concerned.

Relationships between the two kinds of measures tend to be substantial if

considered across pay grade. This would be true, however, because both are, them-

selves, related to pay grade.*

Within pay grade the correlation was appreciable for EK's but not so for

EN's. It would be highly desirable if rated performance would accurately reflect

tested performance within each pay grade level. However, this is demanding reliable

and valid observation of technical performance within very narrow ranges of skill

and job experience. Probably most Navy raters have not gathered sufficient spe-

cific evidence from their incidental observations of men on the job to know the

precise nature and rank order of skills among men in the lower pay grades.*

It is also quite possible that the performance tasks used in this study were

not sufficiently comprehensive to reflect the variety and depth of skills demanded

in the two rates studied. This probably was more true of the EN than of the £M

battery and the correlations throughout the study lend credence to this notion.

Finally, it must be remembered that the rating scale was designed to reflect

aspects of shipboard performance other than the technical ones. Job attitude or

adjustment to Navy life must be considered highly important, particularly in the

Submarine Navy, These aspects of performance are possibly greater determiners of

1 The correlation between pay grade and performance test scores was ,36, That be- tween pay grade and ratings was ,62 in the preliminary and ,69 in the revised preliminary scale.

2 This conclusion also is borne out by the results reported in Part III of this report,

- 40 -

the ratings a man receives from his superiors than are technical knowledge and

cH i \

It would seem, then, that reliable and discriminating ratings by superiors

would fOtsa a necessary part of the criterion of any man's shipboard performance.

However, the within-pay-grade correlations between ratings and tests of performance

in this study were so low that it must be considered desirable, in addition, to

actually test a man on his practical factors wherever possible.

.

?

1 Chapter V

CONCLUSIONS

Tue use by Navy personnel of a rating scale designed to measure the ship-

board performance of submarine personnel has led to the following conclusion:»:

1. Reliable and discriminating evaluations of performance aboard ship can

be obtained with a rating scale that is properly designed and used.

The "man-to-man" format adopted for this study appears to be practical

and tends to promote better ratings than conventional scales.

2. Reliable ratings can be obtained on single traits as well as on total

scores; reasonably reliable ratings can be obtained within-pay-grade,

even at the striker level.

3. Navy raters need specific instructions on the following if their

ratings are to be highly useful:

a) halo effect, particularly that due to pay grade;

b) the logical error leading to high intercorrflatior of

traits;

c) the need for maximum discrimination in ratings;

d) the need for a realistic average in the ratings assigned.

To a certain extent written instructions accomplish the needed edu-

cation of raters* Personal instruction is to be preferred, however.

4. At least two broad aspects of shipboard performance can be tapped by

means of ratings. In this study these appeared to be technical compe-

tence or iob skill on the one hand, and personal adjustment or atti-

tude toward the job on the other.

5. Appraisals of performance made with the rating scale correlate with

- 42 -

actual tests of performance to a very limited extent if the range of

experience is restricted to a single pay grade. This is apparently

mere true for some r?te? ?nH pay grades than others,

6, Inter-rater differences in leniency must be accounted for if high

agreement and reliability are to be realized. This is particularly

true if such agreement is sought on a within pay grade basis.

The usefulness of a rating device depends on how it is designed and how

well it is used. The device should include well defined traits and performance

levels. It should be designed so that men may be readily compared with one

another within the framework of a single performance characteristic, It should

promote discriminating ratings. Ideally, the raters should be aware of the

cemmon pitfalls of rating and tUe many uses for <jood ratings, under these

conditions, highly useful evaluations of performance can be made.

Because of the limited correlations between ratings and performance test

scores obtained in this study, it must be concluded further, that a satisfactory

criterion of shipboard performance demands the inclusion both of ratings by

superiors and actual tests of performance,

- 43 -

r*

SELECTED "BIBLIOGRAPHY

BIBLIOGRAPHY*

nnnr-c;

Guilford, J, P. Fundamental Statistics jj\ Psvcholonv and Education. New York: McGraw-Hill Cook Cor.ipany, 1942. 333 pp.

, Psychometric Methods. New York: McGraw-Hill Book Company, 1936, 566 pp =

Mahler, U. R., Twenty Years si Merit Rat inn (1926-1946). New York: The Psychological Corporation, 1947. 73pp.

B. PERIODICAL ARTICLES

Almy, H, C., and H, Sorenson, "A Teacher Rnting Scale of Determined Reliability and Validity," Educational Ad- ministration and Supervision. 16, 1930, 179-106.

Barteaj. Charles E., "A New Conception in Personnel Rat- ing," Personnel. 13, i936, 20-27.

Bittner, R. H.. "Developing an Industrial Merit Rating Pro- cedure," Personnel Psychology. 1, 1948, 403-432,

Bittner, R. H., and E. A. Rundquist, "The Rank-Comparison Hating Method," Journal £f Applied Psychology. 34, 1950, 171-177.

Blum, LI, L., "A Contribution to Manual Aptitude Measure- ment in Industry," Journal of Applied Psychology. 24, 1940, 301-416.

Bolanovich, D, J., "Statistical Analysis of an industrial Relations Chart," Journal of Applied Psychology, 30, 1946, 22-31,

Bradshaw, F, F., "The American Council on Education Rating Scale: Its Reliability, Validity and Use," Archives of Psychology. 119, 1930, 80 pp.

This bibliography does not include references after 1950 since construction of the rating scale was completed at that time.

Champney, H,, and II, Marshall, "Chtimal Refinement of the Rating Scale," Journal of Applied Psyc/ioiogy. 23, 1939, 323-331.

"•••I - •»•• ~ * •• f m* v •>• V — W h A VUA •*«# *•*!«**.*» W *. » V*XVUM*** I --•*»,

incis." Journal of Experimental Education. 5, 1937, 229- 245.

Davis, William S., "Factor Merit Rating System," Personnel. 22, 1946, 309-319,

Driver, R. S., "The Validity and Reliability of Ratings," Personnel. 17. 1941, 105-191.

Evans, J. W«, "Emotional Bias in Merit Rating," Personnel Journal. 28, 1950, 290-291.

., "The Rater's Task in Merit Rating,' Personnel Jour- nal, 28, 1950, 375.

Ewart, E,, S. E. Seashore, and J. Tiffin, "A Factor Analysis of an Industrial Merit Rating Scale," Journal of Ap- plied Psychology. 25, 1941, 4C1-4Ö6,,

Ferruson, L, '..'., "The Value of Acquaintance Ratings in Crite- rion Rpy.pnrr.il." Persnnr.fi1. Psynhnl ngy. 2, 1949^ 93-102.

., "A Brief Description of a Reliable Criterion of Job Performance, Journal o£ Psvchoionv. 25, 1940, 309- 399.

Fiannegan. J, C., "A New Approach to Evaluating Personnel," Personnel. 26, 35-42, 1949.

., "Critical Requirements: A New Abroach to Em- ployee Evaluation," Personnel Psychology. 2, 1949, 419- 425,

Furfey, P. H, "~n Improved Rating Scale Technique " Jour- nal Q£ Educational Psychology. 17, 1926, 45-48.

r-Uiup, G. H., "Tvaits of Successful Retail Salespeople." Journal of Personnel Research. 4, 1926, 474-482.

Garrison, K. C., and S. C. Howell, "The Relationship Between Character Trait Ratings and Certain Mental Abilities," Journal of Aooliea Psvchoionv. 15, 1931, 370-309,

•

George, Wally E,, MKov; Ue Rvte Apprentices," Factory Man- agement ajip; Maintenance. 95, 1937, 55pp.

Giiinski, A, S,, '"The Influence of the Procedure of Judg- ing on the Halo Effect," American Psychologist. 2, is<u, 309-310,

Jurgmsen, C. E,, "A Fallacy in the Use of Median Scale Values in Employee Check Lists," Journal Q£ Applied Psychology, 33, 1949, 56-59.

Kidder, J. H, T,, "Employee Rating Methods of Appraising Ability, Efficiency, and Potentialities," National In- dustrial Conference Board Management Record. 4. 1942; 33-40.

King, J. E. "Multiple-Item Anproach to Merit Rating," American Psychologist. 4, 1949, 278 pp. (Abstract),

Htusman, H, J., J. T, Begley« and H. L. Parris, "Selected Measures of Proficiency for B-29 Mechanics: Study No. 1," Human Resources Research Laboratories Report £7. July, 1949.

Knauft, Z. D,, "A Classification and Evaluation of Person- nel Rating Methods," JQurnr.l a£ A-iplied Psychology. 31, 1947, 617-625.

. "Construction and Use of Weighted Check List Rat- ing Scales for Two Industrial Situations," Journal of Applied Psychology. 32, 1940, 63-70.

Kornhauser, A, UM "Reliability of Average Ratings." Jour-

•Uii 2i Personnel Rr.search. 5, 1926, 309-317.

Lawshe, C. H., N. C, Keohart, and E, J. McCormick, "The Paired-Comparison Technique for Rating Performance of Industrial Employees," Journal of Applied Psychology. 33, 1949, 69-77

Mahler, W, ft«, "Some Common errors in Employee Rating Prac- tices," Personnel Journal. 26, 1947, 60-74.

,, AR Experimental Study of Two Methods nf uniing Employees," Personnel. 25, 1940, 211-220.

Marble, S. D,, "A Performance Oasis for Employee Evalua- tion," Personnel, 10, 1942, 217-226.

1

Markey, S, C,, "Consistency of Descriptive Personality Phrases in the Forced-Choice Technique," Americ»n Psy- chologist. 2, 1947, 310-311.

Marsh, S., and F. Perrin, "An Experimental Study of Rat- ina Scale Technique," Journal Q£ Abnormal and Social Psychology. 19, 1925, 303-399.

L'.orrow, L,t "/. Neu Approach tu Merit Rating,'' Modern ilan- arc-r^nt., ?, 1949, 19-22.

Patterson, C. II,, "On the Probler. of the Criterion in Prediction Studies," Journal pX Consulting Psychology. 10, 1946, 277-2C0.

Pockrass, J. II., "Rating Training and experience in Merit System Selection," Public Personnel Review. 2, 1941, 211-222.

Richardson, fi, V.'., "Forced-Choice Performance Reports," Personnel. 26, 1949. 205-212.

, "An Experimental Study of the Forced-Choice Performance Report. ' American Psychologist. 4, 1949, 270-279. (Abstract).

Robertson, A, E., and C, L, Stromberg, "The Agreement Be- tween Associates' Ratings and Self-Ratings of Person- ality," School and Society. 50, 1939, 126-127.

Rvan- T. A.. "Merit Ratinn Criticized." Personnel Journal. 24, 1945, 6-15.

Sisson, E, D,, "Forced-Choice—The New Avmy Rating," Per- sonnel Psychology. 1, 1940, 365-301.

Smith, I, S,, "Developi.'.c; a Service Rating System," Educa- te jjona), and Psychplgflica] Measurement. 4, 1944, 327-337,

Stevens, 5, N,, and E, F, Uonderi.'.c, "An effective Revi- sion of the Rating Technique," Personnel Jourr.al. 13, 1934, 125-134.

Stoclcford, L,, and II, \t, Oisseli, "Factors involved in estab- lishing a Merit Rating Scale," Personnel. 26, 1949, 94- 116.

• «

Taylor, E, K,, "lihot Raters Hate," American Psychologist. 3, 194Ü, 2bV-üyO, (Abstract),

Tiffin; J,, and U, Musser, "Weighting Merit Hating Items," Journal j>f Apsliei Psychology. 26, 1942, 575-503.

Turner, U, D,, "A Multiple Committee Method of Merit Rat- ing," Personnel. 25, 1948, 176-194.

Viteies, M, S,, "A Psychologist Looks at. Job evaluation," Personnel. 17, 1941, 165-176.

Uadsworth Jr., G. '.'., "The Field Review Method of Employee Valuation and Internal Placement," Personnel Journal. 27, 194C, 47-54.

VJeinstock, I., "Merit Rating—A Restatement of Principles," Personnel Journal. 27, 1940, 223-226.

Weiss, R0 A., "Rating Scales," Psychological Bulletin. 30, 1933, 185-208.

Wherry, R. H,, and D. H. Fryer, "Buddy Ratings: Popularity Contest or Leadership Criteria?" Personnel Psychology. 2, 1949, 147-159.

Wiebe, G. D,, "A Comparison of Various Rating Scales used in Judging the Merits of Popular Songs," Journal of Applied Psychology. 23, 1939, 18-22.

Wilke, W, H., "The Reliability of Summaries of Rating Scale Evaluations of Student Personality Traits," Journal of. Genetic Psvcholoov. 53, 1938, 313-320.

Zsrga, J. E,, "Developing an Industrial Merit Rating Scale," Journal o£ Aonlled Psychology. 27, 1943, 190-195.

Zimmerman, W, S., "A Simple Graphical Method for Orthogonal dotation of Axes," Psvchometrika. 11, 1946, 51-55.

n i

AFPCIIDIX A

CM CMI—I-OCM—ICO^-COI—lOinco—«

CO

in

CM

1 CM CO in in

CO CO in m

CM CO

in en in -o ~ m

in (->

, 8 O- CO in o

>o © o in © «-

r- i^ r~ f~ h- CM

1 r* in -»o w CM >~< © CO F-

CM CM "ST r-.

«O CM CM CM CM vO O r- O >£> **- <-*

•o ^ v o in -o o ;". ^- vC^Wh-nl'-C

cooinr~cMincoocoo,<5,rH-oooinos,r

1 CO*TCO^,CO*3,L0^3''5r^J,'*3,*3,lO,TCOC0^1,»-<

1 ^ONMfiNUO^t'OUht-OOI-NO

•>x m ** *-* in i "c i—i o in co

1

COmTt'-t-OCMOCMCMCMCOCOOCO cimT'3linvo^"!r,T'3,inv)'"H

co vO in eo •** —« 1 © C C -N -H C NOHOOOOOOO«OC — O

f

1 III

t< T vO CO T' O1 O CM o •—< >—• «-« CM *a*

1 1 1

ocMi-iio>oinomr-ocO'3<r^in>-< CMCMCOCMCOriCOr-l'q'CM'CrcOCMCMC

CO 1 CM

1

Q CD © © —t CO CO © © ^r co co T vO

1

TfC-OTONTOONNNHh-.C

1 CO '*l> *-* —* if) *f CO !*• CO COCOCMO^CMOCOr^OCMCM^HCO^r coeoooo©inL"3T«o©vor->oco

u H

"2 C3 O

CO

5j

c ta SH (0 Cv XI Gu o C D>»-» < B

«/) r» 'Min 3 <<-« E "D O

•<~>VH o o e

O

O O -

0) •** co o> E

©•H «0 «-> *J

V & O P .1 11 'H B 4) -

o o H H P a u 2 u >,«.'C30cacB©«->0l'> cDoa}*qosa)oraB->^

> V

*> C to 0>

•H E C O

-o o ß o ro

•o a c o (? *. *J re. e O (j .« E M c> fc"0

o c ,

<-> ü c

*-> A A V) © 0) 3 *J ,-j n B ^ A H d) 3 o

E O •-> <U n. H

u

10 0) H

B O

•<-< A •-* H E .—i •« HI

B «-> O C c r £

"t/1 1—<

a> « "3 I«

o o cr o >HM n

•" «•></) co W «— C W -~i • 1 i ' ^ B 'H -IH fli m n •-) p-l 4) i—I U 3 >H • to •!-• u •<-< e p ") B

-<MrtTtn>oNcoaOrtM(,5irinoNüc-OH(\in>irin HHHHP-irtrt-(HP,NN!MNNtN

*-• s w

o •FH

«H

a> o

E o

o

u

4) H

tt

CM

co CM

CM

O CM

co

-o 3 S

o o

to

m

CD • o

h- CM i r~ CM

! PSo

O O -O CO I C3 00 0

• » • •

•—I i—( lO ^ O I t- O «O h- CM • • • • •

n c ^ o n i i r- r- i> t- o o

• •••••

CM %' o —* co o —< I OM-OOCJO

• ••••• a I

H V N vM> LT O -i I OüOOOt-ON

h-OOIv-C0t~l--t--<—i

1 c-o-Ht-oocoocom

o h- m • »A ^""» **"• • «w^> w_/ y,/ w » -

co I—1 LO o T r» c2 co co äI

CO CO 1 CO CO

C? CM CO •*? cO co t- >-

m-4

to 51

co o m t- ONO-H

<D

(0

i/1 CO *-" r. «J CS 1« o> o CO

C/5 o e H 4-> .= •**

c -'3 W e g >, w' CJ >> o

O 3 +J —4 o •* g K ü o c .T ja C 4-> o $ ^H O 3 o CO CO o ••-1 o n o <-> •rt O

>. U2 « CM O -~s "O *- >> tf-4 _£k •«* C_* r! m-J • X <_J

C •* 4-> Vi <n 3 * ^ i-< m -H A3 ^H -iH c«n c o c ~ *t-l VI

•c« I—< •H fc-J •-> t-3 «•> >>iS S.J VJ *1^ "-*- ;-' v« V-

n « ja (/) i~< OT >,«H >>•** i-H i-H co "o m 1 (3 *»

U r-\ O •-» « ••> l-l r-* -H O «0 19 B'K —1 CO CO CO

DOI C-O u +J !-• a> ~! " s (H • T3 G, ro n 1) CO •>-: !-; -^ C C co a 3 0> T3 co > 0) .-f5 C3 XI -^ CO

o a < u <: w 2s > cu

'"» CO ^ «J u c«?

covinvOt^cooo-^cicoTrm CM CM

m

I

4 &

CO

NonmNHOONC^oo

icsoiscccocoQnoocoinN

t or-oor-or-oecoi'-Ou-'-i

I COOOO^O—iOO-HOOCOOCM

i* i* * i* i* * i* * r i*

t~cD^irjcoc'3'3,-*©-.ot-m—<~o—«r^r- I CJ (.: N H H N O M IH M in t? PJ -ifj N C Ü

I I*

HOiohOcr,3cr;c . --. a• t- c- —i o c-

vrvioOOirtO«') h- 5- v n "if M C' o O C' i* c i c>-«iaTNo»Ow'j'ifioOMnv?NCsCC'n

i

•rt V) C9 +J «-> c Ci y» eg 0) a>

«H CO c H •J *J .1-1 •<H C -S c CO c> >> >> o

«-> v-t S 3 +J O O 0> -M C -Q S J£ <3 c J3 r: c O <->

-a a a c -s c ^ H «s s s i« s * S :-i o»-> c u o E ;< ^ .H > o a «JO n ra r^ a> c. w u M •*< o «^ •<-•<«-« -3 x^ »H c 'IH •»-< a> <n

« c3 3 a -a o c c *H a •<•» «H «H co '« >-. —»«« OC 4> O CO Q. O O C ViV «5

f-( CD C C "O O »H CO 0> B '<•< «H *J tJ >>ÜJ t'j c w *o o o -j; *-> «-> OT-rt +> «•> ä w *> ..•* to

CO «H >i B W t) r-l * G "1 W >,'« .H H rH i-H

LT

nr

n

CM

—(C*gC3Tl.TOf~COOOfHCMCO*J'm>Ot^COC"C I-H __.-H.-l.-IHHrtHHCMN

f/1

0) o u e o

•<-i •J CO

o

CO (-< 4) H

I I

CM

O CM

o

c o

£

lO

CM

T CM CO 1 C- O CM • • •

1 CM in t- CM

• • • •

O ix >0 O >H 1 O C5 O O —i

CM O » CD t-

O © Q 1- C5 F T CM

M- <> C> H H C I t- O O .*- Oh-N • • •

c n « in a o -o ^ » Ot^OOOOt^CM

CM'S'O'q'COOt-OO i ocoooooor~«-i

t— lf5f-iO,3'O^''T,3'C0

is

0) > t> «-> c "H [.I <-> •H a M §

y •o o c o a

a c t/> O OJ M

—< B-o co «a r. 3 o>

• t—* *

a

H

3 +> •M r: c o 8> •—* V c

^3 • -1

V: 3 o o C

!-' 1J w ji >><*H to «-> o c • M «-> i—* 0> CO •>H >-l

4) »o o -~* I * r i

Ü .Q B O di ^ -rt

O C -r4

<-<

U H i> ca u u c o> .,* >

LI Q

u

0) -<i O ^J rt CO > O h «H

«M (/) .rt 1*4 </> li s •>

•*H CO •-H F-H « Ä U M *J IN O • 0/ C B '•> 0) 0) O J (j

~ CM (O Vf UT O N CO O* O -* i—lr-ir-IPHI—I H rl H rt N N

I I

9

CO

I I—(

°s «

OS» H M C M s co

HE'S

J O o v.- ,z u c 3 --^ -—< u H y a Sat

•a B CO

g5

* a SSW o o i-i M t-- H H CO

SB O M

S u E 4) c

c

II

I

i-t a a

CM

CO

in

io

CM I

tn »r f) c> o « i vo so o *o in vr

i so o in m -o m n

O CO C7> r~> C- CO

CM cj C' co *o *3* •O LO O "V O CM

O <i -H CO CM t co co o CM CM

co r- g ^* CO CO

•O liO C- CO f» CM LO —I T CO CM r—

r- in m o o o co ^r co O CO <l LO NO r-H

in CM CM co co t- r- i >o co -o >o o in vO

in o o PHCWOH LO o i- m «T CM

NOOOOOCMTCOr-C'O ! TT m CM R m o o •v o co LO

NHHOOO LO LO o m LO CM

COOCOh-h-Om^TCMCOCOO i co^rcomcoin^r^rinmcoco

?ONoc>rt •*? CO TT CM CO O

, ^mmvoinvoinr-inm r~ co co t- r- in

h- co co co co a r- o o o LO <-*

in p o Q n N c^ m c^ H n 1 in © '."5 CO '-f? —< ** CM W T CO

CO O CO m —< v

1- CO —I O C4 CO CO V TT LO V H

i c <• -HOOOI-HOP^OC-I-I C C —i r- v c-J c> co i- c cimCiCN c

FHfMM CM O C"

1

^r <* n « rt CM N n o w roO^Hi-HOO—HCO

1 !

CO r-1 O l-H -H Ö

i

CO O CO C- CO CO O ---: C. C t ; Ö

III I

- I^OQf~>OCMOinCOCM COOCMC'-<0>-II-IO'WCMOI-I CM CO ~H CO

~ O "

r OOOCMOV—ILO-H © o o i-t ~* o o o o

i

a c5 c o

c

in 3

U U a co »- co 4' a. r.

o

B

a> > <i

«J c co a>

•« E s o

TJ O C U _ CO >>

o c . 4)

O -4

b> ü •«* a> s 3 u

^ • j ui in

VH O o

t'i >> w

«-> 0)

C T3 O C O C 4> O •rl CO 4) C •« *J «J CT-i-t *J CO (/) "O <-H (0 )H 4> a. O

"5 *J >, .«

i-H -rt £S.'«-i O O

C O 2 o *-> o w

c 4)

b O.T5 P. 3

U l-H

J2 CO •a

S a, 4)

3 f3 H

<u .a

o o

4>

C XI O O 3 O S O >-> Ei »-

'^i o

> zr. O ^: fci Q .4: r» Q <

X) in CO >4 ^— w CO C6 iu

•O 4)

I« 4)

*< *J 1-1 •»H iM •-C c5^

4> c H

•i-t (3

>, O U -i-i C *-! c co

O —i •i-l [-1 3 I—( 'It '•-I 3 .M i—I <(-( . = i CO -H VH t/>

L.1 *» >>^U ifl *>(.1 CO

<*H >TiH 1—I .—I O *J h 1-1 — rj

>m (U CO (3 ~ 4) i-H O 3 U • M >M a e « e CO X5 —i CO > 4» U < CO ^ O (J

—c CM ro T m o r~ O O O *-* CM CO T in >Q Is- CO O O -H f\l o <s* ^l^<f_i^)1_ll_,rt,_)r_(^CMCMCMCMrvJ

c 4>

Cv <J

B o

4) s- M O o

o CO

<-> 41 H

CM

co CM

CM 1 CM

CM

O CM

a«

CO

o i <-H

*

i—i CO 1 LO CM • =

r^. <q' c 1 -O t~ CM • • •

CO O O in i 3 o ^. c • • • «

'JON 8 CM i oon r^

co r- i— o o vn I in -o in m LO c

C

c o

in

rg< _ CM t- CM C- 1 CNh-Oh-lO C3

1 CM O CO LO O —i O t~ r~ r- o o o o

s CM t^ O O O vC CO LO co in o >o ^r o

n

OOOCMCOLO—ir-coco i -oomoot^t^oocM

CM O ~H CM O —«CMCMTTO-O O NO CD o m r~ r- r- in LO *•*

CTJ U H

w «-> n ••'• «-> £3 </)

o <U o C H

*J ^s si. in S3 >> v> (1) >i Ö

U 3 ••> «-H Ü -rt c ^4 O s .a .a a «-> V »1 H <U 3 O 0) CS

o a o »-5 •« o >> o *'* 0) a. »- O -pi

<-> >-. •pH .Q •* H C •—t •«H VI *|H <-> 'TH t-4 3 •<•< (—4 «ft -H .-« '11 C«iH o o L.- O •1-4 IrH 1/1 •>•< r-H •H td »^ L: <-> >» £* Cd w „Q • f«l 0) •-> 1.0 r- ^ M£ •*l , , •"« •"""l ••-1 i-"> •*"! ^-* »i •o o M ~H 0) ••-> O •- >H i—1 r-l O c <-> ft) Rl c •-« •M 4) CO ra 0) c •a u <-» »-•« 0) —1 O 3 h • c C3 <1 0) o •F* >- -i"4 C e <U r.

a -J 3£ o J2

W C 1/5 rt

Ö o

CO T LO O CO O O I-H CM CO '«T CM CM CM

I I

c I—(

o o co to CM in —«t^cMin

i ^r <r v rr ^r m T w ^f H c

e> i co o r» Is- in to in -o o Is» •—• c

i NO « « m io n h' in >o o t- CM

i v U"J -sr r- o in in o o >o in o •-< <

m IT T M in i' c o o T T fi n o H 11

i oor~*rr--ht~»ot^t-t~ot—CM—<

i to t-- rr •^ CM to uo *r in to co to in o r CM

•-. 3 o -H

a "si C3 IJ3 < CO

H R

0)

CM C '-0 r- O CM LO O T CO CM CM O C TT T C3 CO I C CO —< C 1 ^ CM CM —< F4 CM CM {N —< CM —* —i CM

t^ o v N o H i- h o M m c. oi c- v o H ^j CM I —i CM —H C NWOHHrtf s< K H [M H O O

o v •* to o co ^r o co co o o o o CM ca -o m CM I O c .-i C. c ÜNOrtWOCCX —t O O TT O

</3 0> H

c a) u

•p* «H

"S 5 u

a S

"E O n

•J O

r> o -.: •a c

co ••-« •••< •—! o co O 3

J2 ^ .3 S2 -Q C S o "c h H ti o u «j

CT-J £ O C •"» •»* -rt c co ^: >u o. o o

•M VH ja f\ c ••* •« C"00 C *H 3-iH«M«»H

>p( « «J rs "* -«H «-> to >,r_:j [-j p «J U'H f> f> £ »I *> r< V5 "3 --; cs = t* us :~v— .«. — — M 0) Ci. ü 01- ^ r- w 5 y .—i o-o O *-» O V) CX O CO CO '»"I one'rirstiiif

o gj M a) «./ o u —i ^-< M* ß 4) S «H 0) CO CO

& o u u M B 41 tl CO >H > >

ti^-n^ojzüiui/joo

o o •-<

••H CO > o

vi >H

•i-l CJ

c c

IM

o o

u CO

i-twnvin^NOO-OHWcTirinoNüCM'

CM

I

CO • M3

, CM t - ITj I ot r • • •

rp CM < • o | r- t- r-l c

• • • • I

m o o UT rr sr I O t"~ O C •-* • « • • •

I

•a 8)

H o ^ n ?• o I o o t- t-- c —i

co c o

CO C^ V CM.T N Li

• % • • *> • % I

CM

4 ft

o «5* o v ^ r- LO -i m o in o i- t— r-< o

C3 C ; CJ o •*? *- o v c - O L-) O Wi O h- I"- O CM

H

ra •-> a <«

0) C H %j

• M ^ c (_i s-> >> o 3 *J O O O "H

l*ij R C J3 s c u *-> >- H CJ o <u co -H n o kal I-} .,-. •« > o

•-J o r. O O >-i -rt •a •£ e -H •« a> <M

<*H •~» •H >H <w n •-« r. o o c VI «M t/>

•-> CJ >>U CJ C </> +J — 10 »-> •>-< c *— 1*7 r/> Jan. <J_J •••* —* —'. "H 4> M n <u

0) »->' Ö h i-c i~i 5 U r? •r* aj « n *•> W3 *•> -* CJ u M h o i

T3 C3 M —t >-> U «J CJ c c 3 CO a -Q C3 •rH > > CO CO

'.o O o -i o •"> *J ~ • 4 U

.-( CM co v tr> ot-oc-o

I

•

t-H

H 3 • *—*

O O t n 10

W ^ w M *J

>M n > H 12 ä^ M

Cl • • c

££.2 M O O

fin ^

u 3 o »-( •-I

^rfu

f 1 «SB-a H vi B

•»»

I— M[J C HQ C 5 -4 E

o - B x '3 -::- i£ t-J LO

3so i-< O .-I H HI ^H II

C- S c- o U L=« C.J

e

8

C\.

I -

CO

r-

LO

co

CM

t— O CO CO •—I o I >o so -o r- Is- Is- •O in IS- T' IS- CM

o er- CM vO f- o in I sO Is- >0 -O >0 vO P-

i

*r co T co in o fiOO"TOO

i-ivJlONCTsr •OCMTCMrrC

I in co o ^r co in in in io ^r in o

I TNO-OLTOvO^i^mvO

i

CM TT 0s CO -^ CM Is- co o CM in —<

i >ocococom>o-oo>ciOsO

i

Q O CM O h- CO

CM i—i^-Jl^-MCMh-CCO^TOt- CM n o « PJ N n i j - rt N N

I

nocorto T CM H o n o

co N npjinNcircJO-^ l CM in m vo co >o VQ r- in LO Is-

O CM sO Is- CO •-C O O s0 u j

IM C ' O CM

i i—t '3'

CO o o m co

CO CO CM in

-^ CO <3

co 3 c- co

Is- T M< in

CO Is- ©

1 8 o £2 i

>0 —i c o

CM o co in CO ~H CM C s CO CM "3" ^r

-H CM CM -H 1- CO

c © 1—t

C^CI o © ~< CM

CM .—4

1 T in o CM O —I

CM c Is- ©

TOOON o © o -H f-«

o a cS O

CO I'lrtO O >-" —

vC CO o

o 1—1

i i 1 1 1 1 1 1 i 1 1 1

m ! CM

32 T c^ Sg O CM

CM —1 m co o Is- o © o o

O CM s c i—1 tfi m CM -^ O CO CM CM

r—< p—I

co CM

T3 U CO

O

s CS CS I»

*-> co e v cv i: «. E ^ a •» o < 3

H

c co

4-> 10

«S3

-C mi td

IM o o c O _ —1 «

>. w « en

—< ••> o. ^ CO CO o «-> 2 4> O in

.-<• 2i O 2

CO 4) •>i CO

w-l B

•C3 O CS CJ> CO

•o c a

fli rt m

•—« ra B a- o CD

>M .,-. g O •-« 3> l/l Q.T3

•-« r. ;-s Q < ^

>, Ü

IM

M

o o

ca

0)

c J J:

H H K P—I

—• —< cx^-i e o v c • •4 Q, »--J

'rj ,_< -1-1 Ci3

CO X) V) f-4 "O CO IM -H e «J u it a> co u c. a x * <U T3 4J >

Q < -J ©

«-" U •- >>>is

0) 4-> CO «J

9> a H

B >. O u •« = •••>

4) :3

•-M m Vi -rt

t.-J W

-4 01

MM W < u rtNC1, ^mONCBCfO-w.-n 55 O

•J M .—i •>^ 4> « .-( O 3 •« E C |M •»-? CO <c V) :2

• • • O —I CM CM CM CM

—< O CO »- • Cv C > 0> O CJ3

co rr CM CM

VI 4->

• >M

Ü

0) o u c o

u u c

c_>

IM O

M •-» V H

-

CM

CO

CM !

CM

© CM

"8 s

r:

a in

o i <—'

1

1 o o 1

M o o . i"S r-\

• • • 1

o TT r» f- 1 t- V.O t- CO • • • •

i

uoo s o <5J" h •>y o

vO o o o rt \n X o >o \D CO —i

O N N N -O T H I- vO O h- I- CO CM

n o f 2 n o N CM

h--HCO^J,COOOO » o in. in v IO o o m FH

CO

% s 1

O vT Crf vi- O O N N o c- 1 NO O h o m i- N m o o

^T ^1* •—« O O ^J* >o t- o CM r 1 OOCOI^-t-OTt^LO h- o

41 +•* es ••>

<-i CX V) £ 4> O CH

W « •« x: </i C >. e> a> >> o

o 3 «J H o •« c ^g CJ 'Z JZ X» c «-> 0) J-« F—« 4? S3 O

1 o »-s w vw

• F« o • -i U ~ , CJ ^7 4> C >•> ü »(H «-> :•» >«H Xi -•> H c «-« •»H <•-(

• »* 4-> «W *M Ü 'rt it <w •*•. r-« •-* CM o O CO •-< «M t/1 •F* rH •« W »i ui «-> >i« r-j <0 XI •H .S </> *-> to CO C3 XI W —1 V) >>«H >,..* f—1 |-H

f—4 \^/

C 4-> 4> CO a •-< Ml 4) CO a

S. c-a H «-> l-H 4) —1 Ü 3 u • C3 (Q 4) ra • rt M >H c c 4) B

4> •a i> > <C -J O

5J 43 co XI -^ c? $2 a :— -J: o < en s

CO rf to o 1- O O^ O <-H CM ra CM

I I

>

(3 Q

lg ,0

s3 >) >•* <-> > H « si V) 2

H v> . *•<• ..

'/> H E

tig S

d d ö <C HJ "-1

bau

gS § HO. C •-4 <*>

feä e

* c

?: •-: O CU sj

H »CM

a H «HZ

& <

C3

! s;

CO

CO

CM

COOO-OTf^COI—OO I so in h- vo r- sD in «ft c o

r r

l"

t^pt-TOOor t- o r~- co .......... ^.

l r- tft t- P- o h- r~ t— r- r~ Is- CM t I

<. ooor-ocomoccoTTco

r r

I S O N 11 O t" O t- O t- NN N M c,

OLor-wcAio>comcNJOOf-iLOirico i iftoiniftcoiniftotniftino^Tincoc «••*••««•••!•• = •

I I

(MO'JHOrtHrt^vCr T C- ö vr .—< t- |C CIH^MMH^C -H CM C CC < -H CM

l" * " * .... (.

I N(Hi-iHNOHMHHC —I C "H •—< •—' O I I I I* I'

i—ir^^oonoTOvOCMr- c- n m o vS n o

:* r i* i* * * i* " * i* r i* J* * r

> o «->

~ s &s « •<•< to c H •* S <B C CO CT >. >> O

.-> t-t g 3 *J O '_* ty .H C SI E .Ü ft H ,ß C C «j u

-3 - _: o "w c ;- H <u o OJ <U •'' « « C !-i U>1 C U O £ II .H .« » U CJ 4->0 C "3 T SJ i UUU-rH O W *"•• T1 V( T3 £ 'H C 'H 'H (J lu

C3 a =: -a o cc v* 3 -H «H m/i H •'->«•-( OkJ i>orsü.üocr <«<«H VS

C B T3 © "* C3 OJ e '** -H *J fcj >.fcj r_J c « o o ._; <-> <- c:— <-> ~» — w <-> -FH es

p£j *-**»**— #-s wi^wwr^ii^^w^.—• »—* .ü O «J C3 C3 •** 4) Ä t-< •'! "HEOJC'rt ]) (] r) fi

0>c3ooo«JO</>Q,,oc3rt-.-<>-<ci;a>cc

« N n ^r n o t-- w c^ o --• N n T -.1 vo h- o c> c" .-(—<•—(•—(>—l»-t|—I M ^(^.f\J

c J

o t«-l

o o u c o

M p u

o o a H o H

•

-

X) <i>

rz

c o

rs

CM

CO

in

CO

CM

l § • i

i CM r- • • I i

1 o w *r co o c • • •

1 o • • • •

r- in 1 r- o

co CM ^r • * • • •

in o n O CM T O O —1

o rr —i o i- r-. ocMfon o -o o o

COTTUOHON N vO O O ^ O H O

l" l"

^rooooiooflw t- o o o o o r- o r-i

I I

0) « •J </» <3 4-1 c d l/l <u CD

CO •u

C •ri

H c .£3 c o o >> >, o £ 7J *-> o o o •F*

2 *^ »3 C .3 C C o 4J o H '—4 (D O O 0) • rH ra

«.«> C. E ^ H "* > 6 "^ D CH U C H .f-^

•o JQ •>-l B >H -i-l O VH c >M 3 •« <<H <»-t O •FH

ra c o O C,* <4H <«-l to •i-t 4-> £5 SM 3 Ed s in

*J ,<~ V) <J •r* n aiha>4->o>-i--j—!^='_j E a» R '<H c. n n «-> 9TI 4<n S U h H O •

i-i N f •) <r n o t* o <> c HHrtr-l.-lHHHHN

FACTOR LOADINGS AFTER ROTATION .'.IJD CCÜ.IUNALITIE3 BEFORE AND AFTER RCTATIOT! (FIRST SAUPLE, N = 167,

ACrOSS PAY GRADE)

n

2** 2*i;:

Frctor I II III IV "u "r

Trait

1. Pay Grade 27* 97 -7 °6 1.04 1.03 2. Age 22 75 -10 !5 66 64 3. Length on Beard -06 30 30 11 31 31 4. Education -11 -12 39 02 17 10 5. Social adjustment 60 00 07 17 41 A 1

6, Quality of Work 56 31 "A 03 03 7. Neatness of Appearance 34 25 40 -02 35 34 B, Cooperation 79 10 16 20 74 74 9. Watch Standing 52 A4 24 55 83 62

10. Knowledge of Jot 40 70 26 23 ni / ^ 89 11. Discipline 51 14 45 02 49 49 12. Anplication and Initiative77 14 29 26 77 77 13. Judgment and Common Sens e 58 64 13 37 91 91 14. Dependability 64 59 27 26 91 90 15. Adaptability 73 45 27 22 06 05 16, Leadership 52 69 19 27 06 86 17. Overall Efficiency- 69 34 32 46 90 91 10. neatness of '.lor!: 42 27 52 59 no 07 19. Ability to be Taught 75 35 29 21 02 Cl 20. Care of Equipment 67 37 5C 10 05 05 21. Ability to Troubleshoot 48 63 37 C5 05 rxry

Sincerity in Job 69 30 29 45 91 91 23, Manual Skill 42 56 22 44 74 74 24. Overall Efficiency in Ra te4C 37 50 57 94 94 25. Gen, Classification Test -00 20 07 12 10 10

¥ Decimal points omitted

,2 ,,.,_ __ communalities before rotation

u

h'- communalities after rotation r

r

a i

Ö H

«a

OH

aa

k 3

o to a a

r- _ o o ~ M H li

sä W H tJJ 1/5

W I-«

b J

S

* h a

, i o o o OOOH

OC'NC'OHHHNNC'a'OnONHN^OOOOÜ

Q CO m CO vO » ITS O >0 CM LT5 —( © 50 1/5 CM t- C '.:0 P5 C3 M O <* n c H o ^ n H n n rt w i.o is »« n ci n v H N LT o c N

t» N o H c- H o CM iß R r- r, n a v rt o ^ n o n o N n o o -i 11» H o n if) M N « c « w n c rt o v o -• o ü

ir^QfJNOCC-HOO

~H .-< CM CM co o o {•I <•+ C'J O CM I I

o », v o —i o o i— co Is- co o o CM *r :-» «TCOTcoo'S'inTj'Tufi'g'c^cjTrcovcMmcM

o

«T? i- Ö CM ^ i) tN W LJ m co o to n - cpc^cp- ij"i C*i CM li"; N C Ü V O O O CM

o co c —> LI CM CM o <N CM CM c A

re H

8 c. re M

*J re B 0)

EMC <-> O < M "^ 3 «iH

"-><-( 3 S E *0 O

•i* "" >, </> Ä *> '•< f 0) *-» re re •<-> —. C7> o •<-< ••+ *->

0) C 3 O t,5 4> "O o < -o r-3 t/)

C3 O

CO

o B

O B H re a

v > 0) .« w> «-> B re a>

— B B C

•-) B •"

*c o E U re B C

<D O CO r-t "-I

o o

>» u B ..S

•r-S 0 O 3

O

>-l "*

c V-l O ,•3

S .O V 3 i o C M

c- o

m «-> re ••»

CJ </>

B H •H

B >> O *> —*

re 3 0)

a io -a •—i re s cs .a H a> c o co "a re ^ 'i> ä ^t —< -H E c *J ai CO 2 0«-< CiflJ C.T3 O «~ O irt CO C "9 re OreB'MC34>-Ot>

WH in

re s <>> O •-> •-1 •« FH CO t-i

Si re X3 ^ o <

w a> re -J -HO o •* S i-H •« <H H r—( <M •«

•IH «M «> >.ü tJ to

>•> >-i n O ai re re u 3 M . «SOB

•<* a j» tu nnou

in^f'.lrtOOOOOCMT- I CM ** I-H —i o in LO CM h- CM LO CM c

-HCJ^^O-OhOC- CHCMnTOONüC-Ci»HCMC1<J

3 o

re o

c o

re

'S

o

h U 0) CO

re re

O -<H

re B n

re o c o

o (OCMMCM;

Q s ä

r %

* CM u

noinin>0(MN(0>oifioa5CO'-£iooo( c>coon JS

B • • •-I H

g * CM 3 ^cMf5tMin-inc.c3f-*^1cOH<inrr^in,!rrtfJO{M

COrtlOLOOCOh-C^sOinOCOCDC-vOC-OOI-OC-OC^CO JE c= • « CJ »-H M

£ < ^* >-( OTTT,!rc,c>N,cno-inf>i(Minü,T(MTiomciHco

ci-i^OHOifi-iHOonoonrtNrJwnnorJ Sg > K c

es _l CvOfcnncONO'HrtvO^'HCfj'-< o n « C"T c n

> cHNonTH'n'g'inrtOüCNNHnvtMnrtMr o 3 ii ii L> .-3 C£- > or-cocoocMO^Hco^Tr-sOcO'-ioh-vn^cc^-ioc^o

c oriN'j'Hin'rTrco'tiH'^irto^'HCoMTrj T I I

1—1

H y M a t-i -J -~ M Q C « T "» N IQ CJ C> T M O M V N M <I C- CM O Li ^ If) t»

(yi^^rtiMLTCMrtoco^TLOinh-r^inmrtt^ot^^roi^f^ i l 1

l—1

h-H now^'io-mnHrtN^^HfiooOHnnoo ntMcoNHOOonoirtHnHNOMocHHH 1—I o •H

'" a t I \ ^ o n <; i—<

i-H C T O ^ « If! O CM O P W f Mfl C- O t'. V- v vj u vO l'i ^O-H^-HOlftMCMHinTCMnCOCMpO^CMHCMr! £ JZ II o

1-1 Ü

•-I

o •r-t

E O

M § CJ es < 4-> ra g-3 ai a>

O ^ O

£ H > 0) «-> u . ? t* -.-! » ro ,-1 o> " ci *-» «= 4-1 CJ W I-« i-t

«/) M a» .-a a> o a> o 0>

gö O »1 W O C H T3 V-l «-* B *•> ** A •«* 0> o> V-l s CO «H E A Ml E •^ X! re

C2 M i-> co >> c> a> *, o «J

•5 O «J <-> fl) i—I E O 3 «J «-« :} •.-. »»^ v. V3 4-> •« E 4> J3 E E ,.^ (9 E .Q Si HI *-> 5 <D 0)

,j o rS -X3 4) £Z a. O 13 O O l-i H O 3 O UW O •^^ »H n M >* E VH a, CT^ e u -*< o so >-? ••* u 4-> W e u. H o «-> o ..* E re >> o "i a) n. u o -H <s> •M •M

o O </> ti M-4 «M "O «J >J -- ' .3 —* H E —l —» «rt 4-1 ^H ^•H

B E5 => <*.' E "0 O S C ••* *•< <M<t-i 3 .H ,-H «<H •* E (0 m •-•><*-! O O B <L> O ffl FH 'f. CM O O C" O •« VI t/j •^ E E

< EETJO «H « 4) C '»1 •«•—I'^iLi -i Ü M >i^ 'J W O 3 u. O C ^S •• (J »J Ü* »J »J A .H i t/s *J 01 cj C

| —* >i » B «i T3 M « C M /i « H «1 'oVt >,.« >-< I-H £? J5 «J —* *r" 0} 1» ty CO a)T3rj>-i»-iC;*JC)^ --"•—'--. -^ /-^ *- «w iw *»-i •- w — ^^ -^ -*^ B B *-» V ra B »M *H O CO fl ro 6 ö C»0*<H'-«*'ff-OSOi-iO>4)C-T3h4J^Ha)I~t0 3»-i . E

gieäuicnat'PMa'saiiigOig-Hh'rtEavB • "t I 1 i cio-po3«uocor,^r34>"«34/>o-.oi5^a'^rt>(i)

i < j ü w ;>2 ü ?J ii Q •••{ ^ Q < J o L: ^ u v; a .•;! o u O ÜCM3 CNT-<

l Q •e — M CM n T o -o N o o o ^ « n T •-•; c i- ü o o « C>J co T > *

ir

TABU: XII

FACTOR LOADINGS AFTER ROTATION AND CCÜiiüNALITIEG BEFORE AND AFTiiR ROTATICN (SECOND Si.HPLE, N = 2G6,

ACROSS PAY GRADE)

Factor I II III n2** u hf*

Trait

1. Pay Grade 99* 04 14 1.00 1.00 2. Age 95 02 10 90 91 3. Length on Board 05 04 46 22 22 4. Education -22 19 41 25 25 5. Social Adjustment 44 62 01 56 50 6. Quality of Work 74 60 14 93 94 '. Cooperation 56 60 21 81 82 8. Watch Standing 58 70 22 67 88 9. Knowledge of Job 00 10

'It. 38 95 96 10. Discipline 32 59 27 53 53 ii. Application and initiative 39 » ' o 10 7n

1 w 78 12. Judgment and Common Sense 60 41 35 93 93 13. Leadership 78 46 33 92 92 14. Neatness of Work 73 55 15 35 C5 15. Ability to be Taught d? 75 30 84 05 16. Care of Equipment 60 64 11 80 80 17. Sincerity in Job 66 64 14 86 06 18. Overall Efficiency in Rate 73 54 05 03 83 19. Overall Efficiency 72 60 06 on 88 0'i Length in Service 92 -02 8C cc 21 Gen. Classification Test 32 C9 -26 10 10

* Decimal points omitted

** h^ - communal ities before rotation it

h^ - communal itics after rotation

• a

g

£S tj Q

o N a a-

HE

to E bj < M HH H fcJ M .." J > ^ .-, B o

8a O tO

H 2 o

es ••

p co n SP 5-

QH

o H

fc.

*

8

we inc,TCMrrcMvorjcococo-<c-mor',T n CJ n o " o m o H v ff '? f i f T w « in o o

onnoNOvOooNOHinoprvoooh t~cr^^H^'-^'-t^r>—i *-( ^ W •-< T o o »H CM CM r

rtCOONO^Ortr5ocJincNj>--<p-iF^CN4^<Trir5c->j

NQOOM»- •ocaccra'Tr^oin>ocoorjr~o T in ^r t- c-. in o * >o n i* ^ o H

o o re

ra

H

ID > 0)

•M V) •!-» K re 0) •n V)

c o «-> H £

•a cu .x o -a 6 « ** O C (0 s

e s» ET3C r. r vi •-)Vi o B a> o a o. e

c c -o o •»* re a> c « O Q <* t! 41 giwt w ^

•-H >> w to "V •-< re c .c «-> —i «-> u <L> a. o a> •J C3 CO ••-! 0J J2 i-l "H ..H p

a> c ~ w re o ••> o >/> eve ciciTJOaosc'Hts

CT n w es EJ

Ö £•"""

>»««•.•< <-> C i- •H V .— QJ t; 'H k C -a «j —<

HCjn^inoNCjff c *•* ;••: eo rr in o

09

B H •rH

c ;>> >> o 0 O 0J « B C ü ** eu o —< re

U O tl'H •it -rt Q; Vt <H (M V) "«•<

UU c i« •IH (0

«—I ^ »-t HHSU to « «-> M I-l O • 01 <JJ c c s» > as ai o o „.•} o

i- co o c H-iH(M

"8

E O

O c

o IH

o 1-4 o OJ

a

n o

n

o J-.

a)

w o

o

I ! c ü

E! I

0JCM SCJ

§

I * CM U-

* C\l 3

U o o

T c; o N g t-, M c -o ^ cc i: •£ n o> o T c-1^ in

lOsO T O H CO l> O 01

o M Is- T -H -< ^r r o o r» co CM in —i o co ^r co co n co ^ n -< ^r -•( o o rt n w H H n ü TJ -i PJ N

Wi-iOOrtCCOHMMrtC-HOOCOnCMON i l

^HOOOCMCMr-TOCvDOC-J cuncM'T'C^'cim^-NTro

O ^3* CO CJ ^3* ^H *J* ^ ._ _ i - CM^.-M.-H^'COt.OCOinCMT^TCOmT

! I

cJNCJvrJvcMcoort^'H'TOCMrtT'CfcMn I I I I I

ontOHNOH<vOon^HOocwino

1 <*; -•< J y ^O C « UJ in -H TT -< O m C O — "7 lOOHOHOOl/iCOPIOOOOONO

CO uo

K co \r> oc: o i

innnprjc3CMnmh-oNiocjint'C> nNoifiNHvoco^'uio<oifimmHÖ

0) > 0)

« S

B O

V

.0 o e

'D B co

o •a oi ü >- G U rs «-> o a 3 • c "O e Be

•—l Vl O B <U O CO E S *0 O •>* r3 01 E •»•* -^ OO'^ t< «I Q«< •» •» *

•<-t ^iSWO^ » E »1 .s «j —< »J v< u a ü oj k

O">O*aO3OC0E"-<E.34>

<<-4 ex o

3 E -G a> a

a.

c u E E tu 4J

•l-l *r+ o o

3 -H

o

u H E O

V •* o «->

•*H CO > o t-C ....

D V,

w E «

•H CO

a;«-; O I^HPH JU

•rt

0) o

« —< U C7)

>-> I*

£6 E E 0) V

E o E O

+•• CO

E o w <-> E

o

O U

i> M o

<4-(

cu

</> 4)

O

in

(0 E 3

o 0) CM 3 CM M

KPPCNDIX B

I

LU LU

d o CO K r-1

<

>

z Q

z 02

LU

c O

CO

2

o

u

LU a:

< > <

o LU

u

o

z Q

«/> > 3 Ö

2 O LU

< z

<

a: O Z 2

LU I— < Q

c

u -c .5 u c

1- 1_ <o o at M- V) IX <D «3

0O «* U

- "S « — o a)

o £ u_ a.

0 CP

<

1 KNOWLEDGE OF THE JOB

How well does each man know

o

s

o

to

S w 3

< as w

to

1 !

r 1 1

the technical duties of his rate? Does he know all the fine points or does he just know enough to get by? How often does someone have to help him get the job dore? Does he know more or less than most men of the same rate?

H CD

s O u PQ

•

a o to t, «J o ~t

1-3

a o (0 •H u u (0 •

E4

a> u o o

• •

u <D

a CO a,

•

a o W a a o

a •

BE

h a> a u

•

CO

•

i

i

i i

i

Has as much or more job knowl-

! ! 1 i

! V 1

! 1

edge as the very best of his rate. Knows all the fine points of his

•! : 1 _L 1 "i 1 T 1

job. Never needs help on the 'ob. ! 1 V |V;

I i

Practically never needs help on 1 i i

tne job. Has very little to learn ab:>ut his iob except for c few of

_ - i - - - - i

_L - - - - " - - _ .. - - i the mcst technical aspects. i

1 V V i

Has adequate knowledge ts turn 1 1

i 1 i

n a satisfactory job. Sometimes 1 1 .

needs help on the job. since fie - - - -- - --- - -+- - - - - - - - _ - - - - .. _ - — - still has a few things to earn. j

1 V 1 1 j

Has much to learn about his joo.

1 J

Needs supervision very freqjently V f

since he lacks much of tie -;unda- _ _ _ .. _ _:_ .. _. — - - - _ _. -V- _ _ - - _ .. — _ ... ._ _ — _ _ _. _ .. _ .. mental knowledge necessary to do . the job alone.

j 1

i 1 j 1

<

IS» Z O

U =>

* ©

O -t- -Q c

© * I Q. •- = © ID

"t" «« 1= IM

O CP

> w

c P

o CP

© »• CP-P

3 _P

2 i

© <0

©

<D > ©

C o E

ID

s > -D-° .E-c M- © _ •+- 7= ID >-.

_ © © g-QZ

I.E.

-P © ©

© D

<D —

a? © = © Q_ CT'-Q J2 -O 3 <o_M_ -c o Q-— O U bL > -»_ oo CP

c- 2

» p ©.>>£. w !_ »"t: *~

=; E s c 9 « Q- © >•£ c

"£ P -P as = o 2 u

© E © CPE-2 5 •

o © ^ jS o

© . © cn-o r S.-2.2

.c 8-S C .E O

3 Q. o ^ ©

O „ •*. © Q_-t- 10 o « 3 i- n ° 2 «•" o £ CP y

A M Ö -P . ,

+- 3 0 U

ID 5 c

"D *~ Q. c „ Ü.-fc ©J-g §

© _

o S © '

©•£~o

o-c O

•-So 3 CP t7 © -^

^ en o 2

© ©

0)

O © :>- CL v*

© 3 >» © _ .2£i E_c IS 2 © _g .t > CL-Ü+ i

T3 © © -p £ E

!§-|! o-o ° S u © ^_ Q.

ID 2 O <D

o vc £- -c iu C °

© 52 8- 2 "Üj © — '-^ O-^ © .i2

c _ <*- •- © c o

r- ©. <0

•*• ^3 S -= -s c .•*: © u

o * o o

O ^

©

«3 O ~c,

3 O

o E © o

_c u o © ft © _c

.E © . -t- J3 © rp 10 - r£ C i- oo -•- ._

IQ , -<-

-2-«= c 2 N- o !o F © E "> - '-.• .•:; it' -^ '+1 '•/>

<y © •*- "i: S.8 „-a r^ © D» 9

.E -1- j> © * .->.• u

_y o o •- ^-2>5_2 „ +- CL

•£ CT1 O © c'E^ ^ •

\J G1 ° > a1

ID <D

_© J3

ID 3

ID > to 0 E

"5

3 o

©

ID

E _Q_

©

i ©

©

±:-p <D O

CP

U O ID W> © +-

© o -4- P ID •- ©

u

|J ID *

3 ©

2^~ CT> ©

'•* V p

© S, _C CP O-P

u _ ID © - P

©

p o-° O fc P to .—

'S ? o tJ c: t_ S-^D O = c * O ID > "«' ul p

c' .E 3 o •— O w

J 113

3

© v o I- a

-P <Cr=

- * S J

C « © a.

S SjJ 7^ Of *

Q

©

©

«= E -»- ©

c •v o

"tj c ID O

- © p •- 2 p

t; © S E

© *-fc ^: O

"D © P "D

M — © t

_a ©

J 2

'0

C O u

3 cr

o u ' ' in

CP ^ LO

P c •*- ID © p © -C

© s ©

— —- o

O O P © CO i_ CL ©

ID CP--

©

t E

-2 © •- •£

©r^ a.- *" ^ E^>

ti P m

>- © P p

* ELL". —

.E S^ o •f. « ~ — © © 2 u

© © -Ü- ."> c w ^_

g o S J -•- "? ~D 'f-

2 o c o c "O §_Q

© ID ©

^E-t-Q _, p

L. © O ©

-S81E

!.£-« b t "äT* E © > © © £j> 4= E

© O c c c © ©

© >-^>^ •~"p p ID o 5

© S

3 © -^

3

-c 2 u

c © o *- "3> CP S ^

•_ <-> P — o- u. 2

•£ p o ©

o B^ g © >. © ^

j= © © > *• P -Q

c £J? IE' -c ©

CP >* © ^

-p f= © cii 1« r © © D'1 ^

©

U

5 P © 1-

Qj © o ' c ©

c ID

© w ©

© -r< Q.

Ü c O

c ©

« -P P 1 UJ o

* 5 C

« • o o . © <- -2

•fc -D >- u g •?= jo -ü<

C CP m v © O "-

>- L. ©

E a. CL-U

CL E

•B o 5L o -R

I— -P © S _P

in ©

<_ o Q. en 11 E ^0

r 0

E 1c

C

(u _C

H3 o O

1D

z g

IT!

E

I)

<0 en c a m

o o E

E

~<5 •— «0 U c c < SO a: txl » a. a O ~D

O U 5

5 Ü

I

CLM- 0 J?_>-° E ui en £ -t

C_ 0 <V t>. 1= _0 <D

»* £ J5 ^- * E _ o -^ . « <- S3 0 -'

c: 'S f £ ° 3 2 S

io "O 3f "D o E ^E

r n i u}

m T~ o F ^ ^ l/>

ir\ If

> * Q. l_

-+- TJ 0 «J r i_ >

o 0 0) n o V- z 0 n

-<— D_ >-

o o 3 -t- () M-

o E

0 o

CO

< 0 0

L-

0 -*—

-o_* _*— f) <D 3 3 O O-O o >.-a

_0 — Q-> VJ

< 5.

t_ 1

0 — 0 ~D > 0 .£:_0

T3 -+- ~ o 11

M 0 >- O

u u M _^ •c

O '•"=

c '0

O o

T3 >

3 ~- ^ ID U

* g»-t. en i= Ö

v_ I 0 > •*• E ;') ... 0 z 5 0 ^E

-+— J— •I ^- i/l

Ü n •-^ r U "" *: O x» r -t

' (/> 3J 0 o G) '-t— "O 0 . -•- m CT> </i ' — 0 -i- * , ,

O D. O ^p c o

D () 0) E

1c

O

U o

< «0 c •4-

i ) cr> ° > u o c +- m IX Q >*• • j

m tfl c <o L_

QC i_ CP 5 ts

>~ 0 =

o — -° «" o

TJ TJ

O i .-r>^-n

^£ O > >~ .

S o S.5 -o SjE-f 05^ -P

o c 3 _Q

1— T• 1— r " i i i i j

! I ' \

1 1 ! I 1 i > i : j i 1 i 1 i 1 1 1

1 ! 1 1 1 i 1 1

lit! 1 I 1 1

! 1 1 1 II ! '

1 1 1 1 II!!

i ! 1 1 lit!

1 1 1 1 ill!

lil

1 i 1 1 1 1 1 1

till 1 1 1 1

till III!

1 i i !

1 i J 1 1 I i I

iiii i i i i

IIII ttii

IIII i i t i

t t i

KN

OW

LE

DG

E O

F T

HE J

OB

How w

ell

does e

ach

mar

krow

th

e te

chni

cal

dutie

s o

f hi

s ra

te?

Doe

s he

kno

w a

ll th

e fin

e p

oint

s or

do

es

he j

ust

know

eno

ugh

to y

et

by?

H

ow

often d

oes

so

meo

ne

have

to h

elp

him

get

the

job d

one?

D

oes

he

know

mor

e o

r le

ss

than

m

ost

men

of

the

sam

e ra

te?

Has a

i m

uch

or

mor

e jo

b

know

l- ed

ge a

s th

e ve

ry b

est

of

his

rato

. K

now

s a

ll +

h*»

tinp

poin

ts

of

his

job.

Nev

er

need

s he

lp o

n th

e jo

b.

r Pra

ctic

ally

re

ver n

eeds

he

lp

on

the

job.

Ha

; ve

ry

little

to

lear

n about

his

job e

xce

pt

for

a fe

w o

f th

us m

ost

Tec

hnic

al a

spec

ts.

1 i

Has a

dequate

know

ledg

e to

turn

in

a sa

ris.

act

ory

job.

Som

etim

e;;

need

s he

lp

on

the

job,

sinc

e he

'

st

ill h

as e f

ew t

hing

s to l

earn

.

!

Has m

uch

to l

earn a

bo

ut

his

job.

Nee

d:;

supe

rvis

ion

very f

recu

en

tly

sinc

e hs (

acks m

uch

of

the f

unda

- m

enta

l kn

owle

dge

nece

ssar

y to

do

th

e jo

b «

ione

.

i i ~r- ' • i

1 1 . i

1 1 1 1

i i i i

I i i i

i i i i II!!

II!! 1 1 1 1 »

1 1 1 1

III! 1 1 1 1

i 1 I 1

1 1 1 1

1 1 1 1 1 1 I 1

1 1 1 1 1 i 1 1

i 1 1 i II i i 1 1

1 | I | I 1 1 1

1 1 ! 1

III! 1 1 1 1

1 1 1 i 1 1 1 1

III! 1 1 1 1

III!

DIS

CIP

LIN

E

Ho

w

often

d

oes

ea

ch m

an g

et

limse

lf in

to t

rou

b'e

? (

Co

r.si

de

r co

nd

uct

bo

th

ash

ore

and

aflo

at)

. H

as

he eve

r had fo

rma

l dis

ciplin

- a

ry a

ctio

n?

Has

he e

ver

been

re-

str

icte

d?

Do

ss

ie

ha

ve

to

ha

ve

verb

al

warn

ings

ob

ou

t hi

s co

nd

uct

or

the w

ay

in w

hic

h h

e re

sponds

to

ord

ers

?

Ne

ver

g

ets

in

to t

rou

ble

e

ithe

r ash

ore

or

aflo

at. O

beys

all

reg

ula

- tio

ns

and. r

esp

onds

to o

rde

rs w

ith-

ou

t q

ue

stio

n.

Neve

r n

eeds

to

be

sp

o<en

to

.

Ha

rdly

e

ver

g

ets

in

to t

rouble

, iv

lay

bre

ak

a ru

le once

in

a g

rea

t w

hie

, b

ur

is

qu

ickl

y sn

aig

lire

ne

d

ou

t if g

ive

n a

wa

rnin

g.

Coic

uct

is

usu

ally oka

y,

but

ge

ts

into t

rouble

once

in

a w

hile

. E

asily

str

aig

hte

ned o

ut

most o

f th

e t

ime

.

Quiio

o

fte

n g

eJs

in

to t

rouble

. O

fte

n

ign

ors w

arn

ing

s and

has

to

bavi

!

form

al

dis

cip

linary

a

ction.

Conduct

frequently i

s a

rea

l pro

b-

iem

.

1 •

LU .£•

O O :- "tT- -Tl * >

< 0

jsu

aliy k

eep

run

ou

t of

w

ob o

n hi

r. ow

av

e to

be

tol

on t

he j

ob 1

or

does

he

t. rt

t o

f e

ffo

rt ?

1- z

•4-

Q Z < -7 O

0

0)

• s Ü? « i -; 2 <" 0 ->* <0 < -t- <o $ ,J £ - - O © <o — ^. a. t-

tfJ . *» ÖJ © O -—• F c m D)XX rt .- _1 Q. IX <

0 U

man 1

or

do

D

oes

does

do

i.

he r

e«

a m

in

er- o ***• V .. -n

®-o£ E o- o —1 o^

• .E "O '(1)

m *- .JJ

>.§*! v> <D * —^

©os £ u * fc Q L 10 "^ J c - §•,

"d _>- E 0 -t:

-I cr> tr O c © T3 © *-*.. •;>

n a 3 -*- c rr 3 •0 0 t_

-C "D LO

« c *_c

•" *- « 3 © ®

-O > n

© C ffl

2kl >• 1 1

5 0 -Q"0

-K-8 o-n *"ö +- £ "> ©

© ^--JC O

5 C c ID 0)

V? i- -•-

-2 ? 0) J- O > .«2 E itJ

"O c 0 <t/l ^/ O ©TJ •Ti

C-s

O IU © CT> >

*• « 2 3 O C O *• ®

"si &-«=**- ^ l/> w-

0 a«, *• ©tt

© O LL t_ -+-

"*""»$ -fi c >» V ^ ~ © >~>

©«^_Q - - Z3 . C -w

o-tf *~ • ^ © Q © 0

ix ;* .-; a J

| r 1— ' • r "i

i i i i

JUD

GM

EN

T A

ND C

OM

MO

N

SE

NS

E

How

wel

l ca

n yo

u de

pend

on

each

man t

o m

ake

corr

ect d

ecis

- io

ns?

Wo

uld

you

f.ust

his

judg

- m

ent

in a

n e

me

rge

ncy

' W

he

n

plac

ed

on

his

own

can

he fig

ure

th

ings

out

for

him

se'f

and

mak

e pro

per

deci

sion

s,

or

do

you

fre

- quently

hav

e to

che

ck t

o s

ee t

hat

he h

as n

ot

foule

d t

hing

s up

?

1 C

an r

ely

com

ple

tely o

n hi

s a

bili

ty

to m

ake

pro

pe

r de

cisi

ons. C

an b

e de

pend

ed

upon

to

hand

le a

lmos

t an

y em

erge

ncy

Ne

/er

fails

to d

is-

play

exc

elle

nt j

udgm

ent.

Has

ve

ry g

oo

d j

ud

gm

en

t an

d se

l- do

m r

uns

into

a s

ituat

ion

whi

ch h

o ca

nnot h

andl

e. N

eeds a

dvic

e on

ly

occa

sion

ally o

n w

hat

is th

e p

rop

er

5

deci

sion

.

Usu

ally

uses

conm

on

sens

e,

but

does

not

have

o p

art

icu

larly

ma-

tu

re j

udgm

ent.

Nee

cs a

dvic

e fa

irly

fr

eq

ue

ntly

in

mak

ing

deci

sion

s.

Som

etim

es

mak

es

unso

und

jud

g-

men

ts

or

deci

sion

s.

Often

does

no

*-

use

com

mon

sens

e.

Ca

nn

ot

rely

very

high

ly

on

his

ab

ility to

d

eci

de c

orr

ect

ly.

«JD+; £ "O

<o <a a) o. o -»- er _C •»- Jo

^ n, l- <D

• - ^> u> 'T m *r- a o

Q

Oi

Q 'S

:- y' O C ^ •-

15 *• !•>. v. C O

„-=^ • « | - P -c -n c

„ •- <—K y

13s si §"£

"9 i a>~£ 10 £ °

•£< g:> O •*- .5 (0 a *-£ .

2 "q> "* o —

« §"!-£

§:i= §1 < 'O ' a O

T1 •- 0 £ C

~°^-^ c^^ (1> iO U, ®

0

gang

, d.

Is

adm

c E

•__ ip 111 .. a) TJ o .=:•+-£ -t- IQ -"- u. <o a> •*• s fc

o H"o 'o <o !-•-

a> <D _Q > <D Q.

k/1 c 0

-4- © * $ a

lyl •+- > ID c c

rect

ient

i -*- I— u^

>• o o

£ fl) O

til > -+- (O (DO 2 C. O

-o t £ £ ° == iS)

"** £ * 2 ID O U <U w '-ft ">r£

•a a> ID

< 3^ S

I ""I I I

_ 1_ ! ! _ i i i I 1

I i J_ I

H

c >. UO -4- O •"

o <J 3 «0 CD

3 -J b 0)

o X c •-

_o O c

0 2 Si 2

t/% <D~T> c -£i a 111 O „ — _J £C <D h f ,n CO i-r- *"~ f^ -t- rr: Z3 CD •*• >si b o b o i—

"2 45 <0 ,Q

-2* o

cr o

M- c

8*-

0 >—

_*: —- O CD

u 0 u. 3 C (!- 0

r~ rr <s • ~> - _-J ü .

v . a-. T — 9) ^ •s.. 10 ... u -C L.

CD * S -D

OS»« O ?^£ t/> b <D

5 o

>- hi' — <c O o

A o

,_ b

71

o >~ . rr> a. i" O „ <o

a)

0).O

3

ID

ID

o b

3 o

E<

,«? © T. 0) t 0) Q 0

o ~7 n S^= 2 O * - -i c

CD 0 ~5

C71 m M-

?J ?JQ C.

• C > < CL (T

o

o in O ° ^ ^ _ ö O

— id c .2- 8? <s

0) -c: iO •*- >.

-? -t-

."t: «o c .a -° o 2 «

<o a. b lc «* -«-

<D .i

3 _Q CL' cr 3 o cj> .„ ü u 5

ts4=-o-a a> oc ll) O' (j IB

ID <" CD CD c:

>- CD c o <D :•-

u U' c

CD

"I _u 0

<n n o

. 0 a c m i r O L. >

-^ 0 h- I.T

E V OJ n •u o ^ 0 (1) •*-

> <fl 0) (!) r M

3= O

a 70 o

! ; 1 . i i i i

i t i i iii'

1 1 ! 1 1 1 !

I 1 1 (

1 ! 1 1

j 1 ! 1

1 1 1 i 1 1 \ 1

! . . i

i i i i i i i i

i i i i i i i i

ill! I 1 1 1

1 i 1 1 ill!

1 1 > 1

ill i 1

III! I 1 1 1

1 j „ . , , 1 1 1 1 1 1

i 1 1

"~| 1 i 1

1 1 1 1

SIN

CtR

ITY I

N D

OIN

G A

G

OO

D J

OB

Ho

w s

ince

re i

s each

man

in

turn

- in

g o

ut

th i b

est j

ob o

f w

hic

h h

e is

ca

pable

?

Doe

s he

»tic

k to t

he j

ob

u

ntil i

t is d

on

e i

n th

e b

est p

oss

ible

m

an

rer, o

r is h

e conte

nt

to g

et

by

with

som

eth

ing l

ess

tha

n t

he b

est

?

Doe

s he ta

ke pride

in

his

wc

k o>

- is it j

ust s

om

eth

ing t

hat

has

to b

e

done?

Alw

ays

sta

ys w

ith t

he jo

b u

ntil

he

has

done th

e b

est p

oss

ible jo

b o

f w

hic

h

he

is

capable

. T

akes

gre

at

pride i

n se

ein

g a g

oo

d jo

b d

one.

Very

seld

om

sett

les

for

an

yth

ing

b

ut

the

be

st

po

ssib

le

job T

akes

co

nsi

de

rab

le p

ride in

h

is w

ork

. P

ract

ically

neve

r tu

rns

ouJ"

a se

c-

ond-r

ate

job.

Genera

lly

dees

a sa

tisfa

cto

ry j

ob,

bu

t

som

erirr

es

d

oe

s

no

t fo

llow

th

rough.

Occ

asi

on

ally

do

es

no

t d

o

the j

ob o

f w

hic

h h

e is c

ap

ab

le.

Oft

en t

urn

s oui

a sl

oppy

job.

Hu

r-

ries

thro

ugh

the

wo

rk

with

little

re

gard

for

the q

ua

lity o

f th

e

job.

Tcik

es lit

fie p

ride

in doin

g

a g

oo

d

job.

INSTRUCTIONS:

l)'i ycxi were transferr3d to another ship from the one you are now on and were to be the divisici 0"senior petty officsr, you would be interested in both the petty officer qualities (leadership, coof initiative, sincerity, etc.) of your men and in the r ability in rating [iechnical skill).

Considering both petty officer qualities and ability in rating, which one of these men would you in- to take v-Tth you if you were transferred to another ship? Which man would be of greatest vaiue

Indicate your first choice by placing number I opposite the man's name in the columr headed "< Choice," Place a 2 opposite your second choice, a 3 opposite your third cho:ce end so on until > asiigriöd a number to each man in the gang.

2) In the space headed "Remarks," make any comment that you feel should ba consid^ed in evaluating The ratings you have assigned to the men in your gang. For example,

"Only been aboard 60 days" "Stands only look-out watches" "Will never make a submarine sailor" "Finest man I have ever known

OI1I.S'

»ratio*»,

st wa, t •o you' rder c-!

-JU hav

REMARKS I*MC , on »ATP

I