esources position based r testingmission.sfgov.org/document_center_documents/dc3259.pdf · section...

Position

Based

Testing

Manual

January 1, 2012 Part 1

DE

PA

RT

ME

NT

OF

HU

MA

NR

ES

OU

RC

ES

POSITION BASED TESTING MANUAL

TABLE OF CONTENTS

Forms and Samples

Section I Examination Assignment and Research Forms:

Request to Hire Form………………………………………………………………4-5

Position Based Testing Examination Plan Checklist……………………………10-12

Sample:

Section II Job Analysis Forms:

Subject Matter Expert Background & Qualifications Form………………………...12

Test Security Agreement and Statement of Responsibility for

Test Development and Administration Staff…………………………………….13-15

Task Rating Form…………………………………………………………..........25-26

Knowledge, Skills and Abilities Form…………………………………………..27-28

Task and KSA Linkup Form………………………………………………………...29

Qualifications Form…………………………………………………………………36

Examination Outline………………………………………………………………...43

Section III Recruitment Sample:

Forms:

Request to Cancel an Examination/Recruitment/Eligible List…………………..17-18

Section IV Exam Development Samples:

Combined Point Method and Task (Skill) – Based Method T&E Supplemental...9-11

Point Method and Task Based Training and Experience Evaluation Sheet…...........12

Task Based Supplemental Questionnaire………………………………………..13-14

Behavioral Consistency Example Supplemental Questionnaires,

Instructions, Rating Guidelines and Scales……………………………………...17-23

Supplemental Questionnaire for Screening Committee, and Rating

Guidelines and Forms………………………………………………………..…..34-31

Oral Communication Scales……………………………………………………..51-53

Performance Exam Rating Guidelines and Instructions…………………………57-60

Written Communication Rating Sheet………………………………………………64

Request for Promotive-Only Exam………………………………………………..…8

Standard Text for All Job Announcements………………………………………...6-9

Section V Screening of Applications Forms:

Human Resources Director’s Application Approval Form…………………………..7

Veteran’s Preference Application…………………………………………………...19

Employment Verification Form………………………………………………....21-22

Samples: Letters to Applicants Who Do Not Qualify……………………………………...10-16

Section VI Examination Administration Forms:

Request for Reasonable Accommodation in Civil Service

Examination Form………………………………………………………………….4-5

Identification Verification Form……………………………………………………...8

Written Examination Checklist……………………………………………………...12

Review of Ratings by Candidates in the Written Examination Form………………14

Checklist for Oral Examinations……………………………………………………23

Information for Candidates Form………………………………………………...…25

Rater Background Information and Qualifications Form…………………………...29

Expense Reimbursement Receipt Form………………………………………….….33

Request for Cash Advance to defray Business-Related Expense Form…………….34

Field Expense Report………………………………………………………………..35

Samples: Proctor Instructions for Written-Multiple Choice Examination…………………10-11

Cover Letter to Oral/Performance Examination Raters……………………………..16

Proctor Instructions for Oral/Performance Examination……………………………21

Examination Instructions to Candidate for Oral/Performance Exam……………….22

Section VII Examination Scoring

Section VIII Eligible Lists Sample:

Letter to Eligible………………………………………………………………..…….4

Inspection Instructions…………………………………………………………..13-21

Forms: Authorization to Adopt a List of Eligibles………………………………………...…7

Restriction Removal Form……………………………………………………………9

Request for Extension of Eligible List Form……………………………………….23

Section IX Exam Documentation, Finalization and Storage Forms:

Examination Final Report Form……………………………………………………...3

Checklist for Active File Folder…………………………………………………...…6

Checklist for Examination Materials Storage………………………………………...7

Section X Appeals

Section I – Page 1

SECTION I

Examination Assignment and Research

Section I - Page 2

SECTION I – EXAM ASSIGNMENT AND RESEARCH

TABLE OF CONTENTS

Page

REQUEST FOR POSITION BASED TESTING 3

Considerations to Determine Best Approach 4

REQUEST TO HIRE FORM 5

PROMOTIVE-ONLY EXAMINATION 7

Sample Request 8

EXAMINATION PLAN CHECKLIST & EXAMINATION REPORTING FORM 9

POSITION BASED TESTING EXAMINATION PLAN CHECKLIST 10

Section I - Page 3

REQUEST FOR POSITION BASED TESTING

PURPOSE:

To request approval to conduct an exam under Rule 111A Position Based Testing and to document

the rationale for the request.

FORM:

Request to Hire Form

PROCEDURE:

The table that follows this section serves as a guide to help analysts determine whether a Class

Based or Position Based Testing approach should be used the best selection process to use to fill a

vacant position. If a personnel analyst believes that a Position Based Testing approach should be

used, s/he will need to submit a Request to Hire Form to DHR via his or her respective

supervisor/manager.

Before submitting a Request to Hire Form, the analyst should research personnel data so that the

hiring department and DHR will have the necessary information to determine whether Position

Based Testing is appropriate to use to fill a particular vacancy. Research in this regard may require

from automated systems PeopleSoft and/or JobAps (e.g., to obtain appointment and eligible list

information respectively).

Specifically, is the analyst’s responsibility to obtain and provide answers to the following questions

when preparing a Request to Hire Form. DHR will provide assistance, if necessary. 1. Does the target classification belong to a pre-approved PBT program designated by DHR (e.g.,

Personnel Analyst)? 2. How many positions are there in the class overall and where is the class used? 3. Are there permanent (PCS), provisional (PV) and/or exempt (E) incumbents serving in the class,

(if known)? 4. Do recruitments for this class tend to result in diverse candidate populations?

5. What was the applicant population size for the prior exam? How many satisfied the minimum

requirements and how many were tested?

6. Were there any departmental or union protests to any aspect of the previous selection process?

If yes, what did the protest(s) involve and how were they resolved?

7. Was the previous examination issued as an entrance, promotive, or combined promotive and

entrance examination?

8. If there is an existing PBT eligible list for this class? If so, is there is a unique special condition

that would make this list inappropriate?

When the Request to Hire Form is completed, the analyst should give the form to his or her

supervisor/manager or personnel officer. That individual should review and, upon approval, sign

the form. It should then be submitted electronically to the appropriate Client Services representative

at DHR.

Section I - Page 4

When evaluating the request, DHR will consider issues such as:

grievances associated with the class;

protests/appeals filed in past recruitments (e.g., announcement or MQ protests);

discrimination complaints involving the selection procedure(s);

pending investigations related to the class;

complaints from members in the class or class series;

professional considerations related to Subject Matter Experts, and;

other factors related to merit principles.

The existence of past complaints, grievances, etc. may not necessarily prevent use of Position Based

Testing, but it may require implementation of certain strategies to ensure that a valid and defensible

process is used going forward.

DHR may require a draft of the announcement be submitted for review prior to approving the

Request to Hire.

After the Recruitment and Assessment Services Unit (RAS) at DHR approves the request, copies of

the approved request will be provided to the hiring department and DHR’s Client Services.

If DHR denies a request for Position Based Testing, it will inform the department of the method the

department can use to fill the position. DHR will also notify other units as appropriate.

The department is to retain a copy of the approved Request to Hire Form in the Active File folder.

Section I – Page 4

Does the hiring department have an existing eligible list

or holdover roster appropriate for this position?

Position Based Test

City-Wide Test

Do many departments hire in this classification?

Is the hiring need extremely urgent? (e.g., 311 Call Center)

Is the classification highly competitive and are very large

applicant pools expected?

Are there complex issues associated with this classification

or does past exam history indicate high sensitivity when

testing this classification?

Does this classification belong a to pre-determined PBT

program designated by DHR?

If yes, PBT examination is not appropriate; use eligible list

If no, PBT may be an appropriate alternative to city-wide test

If yes, city-wide test is not appropriate; use existing eligible list

If yes, a city-wide test may be more appropriate

Considerations

to determine best approach

Does the work associated with this classification vary

considerably from position to position? Is the work

of the target position unique within this job classification?

If the work varies little from position to position, a city-

wide test is appropriate


If no, a city-wide test may be more appropriate

If no, a city-wide test may be appropriate


If no, PBT may be appropriate alternative to city-wide test

If yes, PBT may be appropriate alternative to city-wide test

If yes, a PBT shall be used.

If no, PBT may be appropriate alternative to city-wide test

If the work varies considerably or if the position is unique, a

PBT may be appropriate

Section I - Page 5

City and County of San Francisco REQUEST TO HIRE FORM DEPARTMENT OF HUMAN RESOURCES Position Based Testing Provisional Recruitment

Section I – Department Information Instructions: Department completes Sections I through III for PBT Requests, and Sections I, II and V (on second page) for

Provisional Recruitment Requests. Send form to your Client Service Representative.

Department: Division: Date:

Hiring Administrator: Phone No.:

Supervisor: Phone No.:

Section II – Position Information

Division: Section:

Class Code and Title:

Working Title (if applicable):

DHR Requisition Number(s): Requisition Approved? Yes No

Special Condition, if applicable:

Section III – Request for Position Based Testing

Justification for requesting PBT:

Section IV – Position Based Testing Review & Approval (DHR Use Only)

PBT Approved: Denied: List ID Number: List Type: E CPE P Scope:

Comments:

Signature of DHR Examination Team Leader: Date:

Section V – Request for Provisional Recruitment

Justification for requesting Provisional recruitment: Note: Oral Authorization will be required (Section VII)

Section VI – Provisional Recruitment Review & Approval (for DHR Use Only)

Provisional Recruitment Approved: Denied: List ID Number:

Comments:

Name of DHR, Examination Team Leader: Date:

Section I - Page 6

Section VII – Request for Oral Authorization (Use only for Provisional Hire request)

At the completion of a provisional selection process and prior to appointment, please complete this form and send as an attachment to an email to your DHR Department Representative (please, no hardcopies). Your Representative will obtain approval and return the form to you electronically or call if clarification is required. Your copy of the approved form should be maintained for five years. Use the tab key to move between fields.

Request date: Submitted by:

Appointment Information

No eligible list or test in progress

Exam in progress; selected from applicant pool

Selected from eligible list pending referral

Pending completion of medical or background

Continued PV in same class, but different requisition

Transfer, reinstatement, or reappointment pending

PCS

Check box to confirm that the MQs are identical to the last permanent job announcement. If not, explain any differences below.

Recruitment Process (required)

Opening Date: Final Closing Date:

Announcement posted within department

Announcement distributed city wide

Announcement posted at DHR

Announcement posted on City’s website (SFGOV)

Selection Hurdles (not all required) and Applicant Statistics

Screen for MQs– Analyst Name:

Training and Experience (T&E) ratings

Written examination

Performance test (e.g., writing sample)

Oral Interview Other

Number of applications received

Number meeting minimum qualifications

Number of candidates invited to participate in process

Number of vacancies (approved requisitions)

Candidates Selected (List only those to be hired in same order as requisitions listed above)

Rank

Name

SSN

Current or previous

employee?

State briefly why each individual was deemed more qualified than others in the pool of candidates for this position.

Department Certification DHR Concurrence

This provisional process was conducted in a fair and open manner and the selection was based on merit.

By: Date

Based on the information and discussions with the department (when necessary) DHR concurs with this selection decision.

By: Date

Section I - Page 7

PROMOTIVE-ONLY EXAMINATION

PURPOSE:

To ensure that promotive-only examinations are announced in compliance with Civil Service

Commission (CSC) policy.

AUTHORITY:

The Civil Service Commission re-issued its policy on “promotive only” examinations in a

memorandum dated 3/26/99 (see Appendix II). It delegated authority to the Human

Resources Director to issue “promotive only” examinations.

PROCEDURE:

Promotive-only examinations are prohibited except as mandated by law or when there is

agreement by concerned Department(s), Employee Organization(s) and subject to the review

of the DHR – Equal Employment Opportunity Unit and Human Resources Director.

Therefore, the personnel analyst shall review qualifications and appointment status of all

employees in promotive classes in the occupational series, including provisional and exempt

employees, to:

(1) determine if there would be any problems qualifying them under the terms of a promotive-only announcement, and

(2) evaluate the ethnic and gender composition of employees within the promotive class, and lower level classes to ensure the workforce is diverse.

The analyst shall also obtain written confirmation from the union stating that it will agree to

promotive-only examination.

Upon receipt, the analyst shall:

(1) prepare a memorandum that includes analysis of above factors, (2) point out relevant issues (e.g., if there are some provisional employees who may not

be eligible to compete for the promotional examination based on the MQ

requirements) and

(3) submit the draft memorandum to his/her supervisor/manager for review and approval.

The approved memorandum should be initialed and submitted to the Human Resources

Director for approval. (See next page for sample memo to the Human Resources Director)

Section I - Page 8

City and County of San Francisco Department of Human Resources

Mayor Human Resources Director

__________________________________________________________________

SAMPLE

DATE: Month x, 20xx

TO: ______________

Human Resources Director

THROUGH: _____________

Senior Personnel Analyst

DHR Recruitment & Assessment Services Unit

FROM: _________________

(Hiring Department’s Personnel Supervisor/Manager)

SUBJECT: Request for a Promotive-Only Examination for Class 7334 Stationary

Engineer

I request permission to conduct a promotive-only examination for class 7334 Stationary

Engineer. Eight employees in class 7333, Apprentice Stationary Engineer will soon complete

their four-year apprenticeship programs and become eligible to work at the journey level.

The Department of _______ requests that these apprentices have an opportunity to promote to

the journey level, since four years have been invested in their training. Some well qualified

people are graduating in this apprentice class.

The pool of potential promotional applicants will be more representative of gender and ethnic

diversity than the population of 7334 Stationary Engineers.

The union has agreed to a promotive-only examination for this classification. [See attached.]

Should this request be approved, we plan to post an eligible list by ___ (month/year).

Section I - Page 9

EXAMINATION PLAN CHECKLIST AND

EXAMINATION REPORTING FORM

PURPOSE:

To provide personnel analysts with a document to guide them through the Position Based

Testing process.

FORM:

Position Based Testing Examination Plan Checklist

PROCEDURE:

Position Based Testing Examination Plan Checklist

Use the Position Based Testing Examination Plan Checklist (see below) as a planning tool to help guide you throughout the examination process. Make note of the points

in the process where approval by the Department of Human Resources is required.

Complete the Examination Planning Form and submit it to the DHR RAS for consultation and approval.

o The shaded task statements are the responsibility of the Department of Human Resources or the authorized Decentralized Exam Unit supervisor.

o If the department has an authorized Decentralized Exam Unit supervisor, the supervisor may review and approve the Position Based Testing process

administered by the departmental analyst. The authorized Decentralized Exam

Unit supervisor must sign off on the form indicating that the supervisor reviewed

and approved the examination documents and process at the required points

indicated on the checklist.

o Additional review points may be required depending on the experience of the analyst and/or the issues or complexities associated with the exam.

The analyst may wish to indicate the date when each specific examination task is completed.

Upon approval of the eligible list, the checklist should be filed for exam record purposes.

Section I - Page 10

POSITION BASED TESTING EXAMINATION PLAN CHECKLIST

The outline below identifies the general steps and procedures to follow when conducting PBT examinations.

Important: Please note each step (indicated by a box with an X in it) where DHR review or attention is required.

DHR Review

Point

Notes:

(i.e. Target Dates, Completion Dates)

PLANNING

1. Identify funding & budgeted position.

2. Identify appropriate job class.

3. Obtain concurrence on classification, if needed.

4. Issue requisition.

5. Determine if there are provisional employees or anticipated hires, and contact respective

departments.

6. Obtain approval from DHR to conduct a PBT process for the position.

7. Obtain agreement from the union on the certification rule, if necessary.

8. Identify Subject Matter Experts & schedule position analysis and exam development

meetings.

JOB ANALYSIS AND DEVELOPMENT OF THE

RECRUITMENT PLAN

1. Review prior announcement, recruitment file and exam active file, if available.

2. Review job duty information, KSAs, and minimum qualifications based on existing

documentation. Determine if amendments to

the documentation are needed.

3. Complete position analysis documentation forms and attachments.

4. If position analysis differs significantly from the job analysis documentation, confer with

DHR before drafting announcement.

5. Assess recruitment needs (e.g., check previous recruitment and workforce statistics

and confer with Subject Matter Experts, if

needed).

EXAM DEVELOPMENT AND

ANNOUNCEMENT

1. Develop test (i.e. T&E, oral exam, etc.)

2. Develop the proposed PBT announcement.

3. Upon request, submit to DHR that portion of the Compliance Review Form covering

position analysis, recruitment plan, test

information (e.g., KSA/test linkage, test

questions, rating guidelines, etc.), rater

information, announcement, and exam plan.

DHR will review and respond.

4. DHR will check for holdovers prior to issuance of list ID, and will consult with

department if provisional employees exist

Section I - Page 11

5. DHR assesses the need for updates to the class specification and, if needed, posts the

amended class specification in advance of

the announcement posting.

6. The Department may wish to send a courtesy copy of the announcement to the union

approximately five days prior to the

scheduled posting of the announcement.

[However, this is not a requirement.]

7. Issue and post the announcement in JobAps.

RECRUITMENT

1. Distribute the announcement to outside parties (media, professional organizations,

professional magazines, etc.) if needed.

2. Respond to any announcement inquiries.

3. Notify and copy RAS with respect to any protests or appeals received.

4. Review protest and conduct research as necessary. Draft and forward reply to RAS

for review and approval. As appropriate,

complainants are notified of rules relating to

appeals.

5. DHR or CSC determines if exam process or list issuance will continue with a pending

appeal.

APPLICATION SCREENING

1. Collect applications and/or resumes from interested job seekers.

2. Screen applications for minimum qualifications.

3. Notify applicants via email (or letter) who do not meet the minimum qualifications and

provide five days for

clarification/verification.

4. Respond to questions and issues from applicants.

5. Enter applicant disposition information in JobAps.

6. Notify candidates of time/date/place for exam process.

7. EXAMINATION ADMINISTRATION

1. Implement logistics of exam such as securing exam site; recruiting proctors; contacting

and/or recruiting raters; and preparing exam

materials, etc.

2. Train raters and proctors.

3. Administer the exam process. Inform candidates in writing that protests involving

the manner in which the test is administered

must be filed prior to the candidate leaving

the test center. This policy is designed to

address issues on-the-spot rather than after-

the-fact when there may be no remedy.

Section I - Page 12

4. Bring all test administration protests to the attention of RAS.

5. DATA ENTRY AND SCORING

1. Score exam results and apply the appropriate pass point.

2. Upon request, submit to DHR a Compliance Review Form to document the exam process,

any problems encountered and the

justification for the pass point.

3. Add promotive and veteran points if needed. City employees with six consecutive months

(1040 hours) of verifiable experience in any

job classification in any appointment type

qualify as receiving promotive

4. Enter exam results into JobAps.

ELIGIBLE LIST AND RESPONDING TO

APPEALS

1. Send out exam results and inspection materials to candidates. Inform eligibles of

the eligible list’s duration and/or the duration

of their eligibility. Exam participants have

five working days to verify the accuracy of

their score calculations.

2. Appeals to the CSC at this time can only be about claims of inconsistency in exam

administration, bias of raters and/or failure of

raters to apply uniform standards. All other

protests regarding exam administration will

be submitted to the HR Director.

3. Notify DHR of any protest; confer with DHR on response and forward draft response to

DHR for review.

4. Direct appeals of inconsistency of the exam administration, bias of raters and failure to

apply uniform standards to the CSC. Notify

DHR of appeals directed to CSC, and

unresolved protests of any other matters.

5. Prepare materials for presentation to the Civil Service Commission in response to

appeal and forward to DHR for review.

6. Conduct Review of Rating.

7. Send Authorization to Adopt the Eligible List via email to DHR’s EIS and Referral

Units. DHR adopts and posts the eligible list.

REFERRAL AND FINAL SELECTION

1. DHR issues Referral to departments within 15 business days from date of adoption.

2. Conduct hiring interview, make selection decision and notify DHR.

DOCUMENT PROCESS

1. Document unusual circumstances, if

applicable.

2. Prepare and maintain active file.

3. Retain storage materials. Follow record

retention schedule.

Section I - Page 13

Section II – Page 1

SECTION II

Job Analysis


SECTION II – JOB ANALYSIS

TABLE OF CONTENTS

Page

JOB ANALYSIS 3

Frequently Used Terms 4

Updating Job Analyses 5

New Job Analyses 6

SUBJECT MATTER EXPERTS 9

Subject Matter Expert Forms 11

SME Background and Qualifications Form 12

Test Security Agreement and Statement of Responsibility Form 13

Role of the Job Analyst 16

DESCRIBING JOB BEHAVIORS 17

Writing Critical Incidents 17

Writing Task Statements 18

WORKER CHARACTERISTICS 19

Writing KSA Statements 19

Evaluating KSA Statements 20

On-The-Job Learning or Brought-To-Job? 20

GENERAL INTERVIEWING GUIDE FOR AN ANALYST 22

JOB ANALYSIS FORMS 24

MINIMUM AND DESIRABLE QUALIFICATIONS 30

Minimum Qualifications 30

Minimum Qualifications with Special Conditions 32

Common Pitfalls Associating with Minimum Requirements 34

Qualifications Forms 35

Desirable Qualifications 37

ESTABLISHING A CERTIFICATION RULE 38

VERIFICATION AS PART OF THE APPLICATION PROCESS 39

SELECTION PLAN EVALUATION 40

Selection Plan Worksheet 42


JOB ANALYSIS

Job analysis is a generic term for the process of critically examining job components to provide a

functional description of a job. The job analysis is a basic tool of personnel administration

which can serve a number of personnel needs including job classification and specification,

worker compensation, training, performance evaluation, and of course examination for

employment and promotion. Because of the diversity of jobs and job analysis applications, a

variety of job analysis strategies, methods and formats exist.

All job analysis techniques deal with behavior, either directly or indirectly. Behavior is dealt

with directly when the job analysis specifies as its elements those activities which are necessary

for achieving the objectives of the job. Behavior is dealt with indirectly when work products are

used as elements in place of behaviors which are not directly observable. The descriptions of the

job behaviors spelled out in the job analysis are the foundation blocks upon which the analyst

constructs the selection instrument. This is the basic linkage between the job and the test.

In the City and County of San Francisco, selection procedures are generally developed with a

content validity approach. This manual, therefore, shall focus on one type of job analysis which

is legally defensible and supports this type of validity. It supports validity because the job

analysis procedures capture the most important aspects of the position(s) which are then reflected

in what the selection instrument measures. In other words, the validity of the selection

instrument is inferred from the nature of the test content’s relationship to the job analysis

information. It should be noted that, although this manual focuses on job analysis for specific

positions under the Position Based Testing Program, concepts underlying the process are the

same as those for the analysis of all positions in a class.

Basic Principles

Job analysis is a process; not a product.

Job analysis is crucial to the establishment of content validity.

Content validity job analysis begins with observable job behavior.

Content validation requires that rational and logical linkage be made between observable

job behaviors and the selection instrument.

Elements embodied in a job analysis, such as tasks and worker characteristics (KSAs), must

never be so vague or abstract that the behaviors, and hence the job itself, cannot be perceived.


Frequently Used Terms

To avoid confusion, analysts should be clear on the descriptions of terms used to describe job

analysis and its components. Here are some definitions of common terms:

A position is a related set of purposeful activities for an individual. Only one individual occupies a position, although it is possible for a position to be vacant. Several positions

may entail performance of the same job.

A task constitutes a discrete unit of purposeful (for the organization) activity. It is behavior that produces a result. All the job tasks taken together constitute all the

activities of the job. “Task” is used synonymously with “behavior,” “function,” or

“activity”; to the extent that “task” differs from the other three terms, the difference is

that “task” connotes accomplishment of the organization’s purpose at a basic level.

A dimension is a defined category under which behavior can be reliably classified. The set of behaviors classified under the dimension operationally define the dimension.

A duty is a set of related tasks. Tasks are elementary components; duties on components taken at a higher level of abstraction. “Duty” and “responsibility” are taken as

synonymous.

The principal divisions of worker characteristics are knowledge, skills and abilities, frequently referred to as KSAs. In order to allow for the possibility of other worker

characteristics (e.g., personality variables), some agencies include “other”; in this case,

the generic designation for worker characteristics is KASO. Knowledge is a body of

information applied directly to the performance of a function. A skill is a present

observable competence to perform a learned psychomotor act. An ability is a present

competence to perform an observable behavior or a behavior which results in an

observable product.

Subject Matter Experts (SMEs) – are typically incumbents and supervisors who provide important job information in the job analysis process. Although incumbents and

supervisors are termed “Subject Matter Experts” with respect to the job they do or

supervise, this term is something of a misnomer because incumbents, even superior

performers when in the entry level of their title series, are not necessarily “experts.” The

term “SME” is also used within the test development field to designate consultants who

develop test material.


Updating Job Analyses

Jobs do change, at least in some of their aspects over time. For example, it sometimes happens

that a job will change radically in a short span of time, as when there is major organizational

restructuring or when there is implementation of new technology. In other cases, jobs may

change at a glacial rate, one minor aspect at a time. In these latter cases, a new job analysis need

not be completed each time a test is held for the title. In other words, a prior job analysis can be

verified to see if change has occurred since it was last done and, if necessary, it can be amended.

An expedited data gathering process or a previous job analysis may be employed to update a job

analysis conducted within the last five years if the basic duties and responsibilities and

requirements of the position as described by the last job analysis have not changed substantially

in that time. Analysts should consult their team leaders or DHR to determine if an expedited

process is appropriate.

For expedited data gathering, it is recommended that analysts, at a minimum, do the following:

Review previous examination files (i.e., Active Folder information) - to obtain prior job announcements and job analysis reports which should include tasks determined to be

important to the job as well as the knowledge, skills, and abilities needed to perform these

tasks. These can provide a tentative list of job tasks and KSAs, as well as the names of

SMEs used for job analysis and/or test development. The analyst should pay particular

attention to how the prior job analyses were conducted and whether the job analysis elements

were clearly described. [Note: Although a prior file can serve as a good starting point, it

should be remembered that it documents the job only as it existed at the time of the last

examination administration. If this occurred some time ago, then the information may be

outdated and have limited utility.]

Review the class specification for this position – to obtain major job activities and the KSAs to perform these activities. Also, the minimum qualifications (MQs) stated in the

specification should always be the same as that of the target announcement with certain

exceptions (e.g., manager titles, IT titles, special conditions). [It should be remembered that

job specifications should be considered as providing “bare bones” information for selection

purposes. Because job specifications are oriented toward classification and perhaps

compensation as well, descriptions of activities may be written in a manner so broadly as to

not identify work behaviors specific to a particular position within that class. It is also quite

possible that the job description is out-of-date. Indeed the current job specification may be

quite different from the one that existed at the time of the original job analysis.]

Review position classification information - such as Job Analysis Questionnaires, recent job descriptions or performance appraisals if available, and/or recent provisional announcements.

Review class specifications and announcements for other classes in the same series, both above and below the classification for the position. The duties of the class tested under

Position Based Testing should be related and consistent when compared with duties and

responsibilities of the class in general.


Conduct a Telephone or Personal Interview – Given the concerns noted with regard to the review of active examination files, existing job specifications, etc., the job analyst should

gather and verify job analysis data by conducting a telephone or personal interview with a

Subject Matter Expert. Providing the SME time to review in advance the job analysis

questionnaire/booklet with draft statements of job behaviors and worker characteristics will

speed the process considerably. If a personal interview, it is also recommended that the

analyst notify the SME in advance to bring, as appropriate, necessary job-related documents

(e.g., training materials, organizational charts, etc.) to the interview.

Ideally, the SME used for verification purposes will be the one who participated in the

original job analysis, although this is not a necessary condition. In any event, the SME is to

examine the prior job analysis for changes in job behaviors, worker characteristics, linkages,

and ratings. The SME should be alerted to specify new versions of manuals or other changes

that might not be reflected in the wording of the last job analysis. The SME might find minor

changes in the job since the initial job analysis but they may be of no consequence. What

constitutes “no consequence” is a matter of judgment; however, the following should be used

as a guide:

Changes involving on-the-job learning status or ranker/qualifier status are never trivial.

Where job behaviors or worker characteristics are rated on scales, a fluctuation of one scale point is trivial unless that change has impact on the test development plan.

Any change in a job behavior or worker characteristic must be noted. Additions or deletions must be noted. Any change in linkages must be noted.

New Job Analyses

For classes that have not been studied within the last five-to-seven years, a new job analysis

should normally be conducted. When a new job analysis is performed, the analyst will often need

to select the type of data collection method based on the amount of job-relevant information

already available from previous job analyses and the title’s sensitivity (e.g., number of appeals

associated with the title). The amount of time that an analyst has to spend on a job analysis,

however, is often the most significant factor that enters into the decision of the data collection

method to be used.

Typical methods used to collect job-relevant information in the job analysis process are

presented below. More than one method may be used in any given job analysis.

Questionnaire – Those knowledgeable about the job provide answers to questions presented on

a standardized form which is known as either a job analysis questionnaire or job analysis

booklet. Although the opportunity for personal interaction is lost, the time saved by avoiding

such interaction may be an important consideration. The questionnaire is particularly useful

when the duties of the position haven’t changed much since the previous job analysis. Although

highly unlikely in the case of PBTs, questionnaires are also very useful when there are numerous

SMEs to be contacted and many positions to be covered.


This procedure can be taken a step further, by mailing or electronically forwarding the job

behavior and worker characteristic statements to the SMEs in advance of a panel session or

interview (discussed later in Section II). Indeed, the drafting of a tentative set of job behaviors

and worker characteristic statements by the analyst into a questionnaire, gives SMEs an initial

frame of reference and the need to reconcile differences of language can be mostly eliminated

where no substantive issues are involved. In this way, the SMEs can do much of the job analysis

before the formal session. The session then has its primary purpose the integration, rather than

the generation, of information.

It is essential that the supervisors of the SMEs allow on-the-job time for completion of the

materials. It is always good practice by the analyst to notify the HR office in advance of any

plans to use SMEs from that department for job analysis purposes and to alert the HR office of

the analyst’s plans to forward the questionnaire to the SMEs. In many instances, the HR office

may be needed to recommend particular SMEs. Such notification, regardless of the job analysis

method, also allows the HR office to determine whether there are any concerns about using a

particular SME. It is also essential that SMEs who complete the questionnaire have the reading

and communication skills to work with the job analysis materials (if this is not the case, this

method should not be used). Detailed instructions must be prepared for the SMEs working on

the material. A “due date” for completion should also be provided, along with the job analyst’s

contact information.

Group discussion – The components of the job are brought out through a discussion or

brainstorming session with incumbents, their supervisors, or others knowledgeable about the job.

It is preferable to meet jointly with all SMEs when conducting job analyses because a group has

the advantage of providing information from different perspectives that can be immediately

discussed by all. That is, group discussion allows for immediate feedback on the information

generated. Consequently, it tends to produce quality information. However, meeting with a

group of SMEs can be quite time-consuming. Structuring the group or panel discussion (for

example, by starting the panel with a draft set of task statements and KSAs selected from job

specifications or other documentation) can often speed the process. No further discussion of

large SME groups or multiple SME panels will be discussed here given that this approach is

more likely to be used in association with Class-Based Testing.

Interview – Incumbents and supervisors are asked a series of interactive questions about the job.

This method allows the analyst to gather details about the job which may be especially important

if the job is highly technical. In fact, information can be obtained in more depth in an interview,

yielding information superior to that which can be obtained in a panel situation. The interview

may be done via a telephone call but a personal interview is preferable because it also allows the

analyst to examine job related material (procedure manuals, equipment, forms) while someone

familiar with that material explains its usage. When possible, the analyst should interview at

least two incumbents, and interview them together. This gives two perspectives on the job and

enables the incumbents to cross-check information with each other.

The job analyst, at a minimum, should have gathered information from the job specification or a

previous job analysis, prior to speaking with the SME. Ideally, the analyst should prepare a draft

questionnaire with tentative task and/or KSA statements in advance of the interview. This will


allow the SME to verify the accuracy of the conclusions drawn by the job analyst from his/her

review and synthesis of job-related data gathered.

The interview must have as a minimum goal a set of job behavior statements and a

differentiation of those statements by task frequency and importance. The generation of worker

characteristics should be done as well along with evaluations of those KSAs in terms of

importance at entry, usefulness in differentiating levels of work performance, and usefulness as a

qualifying or ranking element in selection. It should be noted that various other scales do exist

(e.g., Independence/Supervision of tasks performed), many of which are variations on a theme

(e.g., Learning Time/Difficulty). However, the task and KSA scales identified in the job analysis

forms presented later in this section are the ones recommended for the majority of titles that fall

within the PBT program.

It should be noted that there are several ways in which an analyst may finalize the task and KSA

statements.

As indicated previously, it is recommended that the analyst do homework beforehand and have a list of tentative tasks and/or KSAs available in advance of the interview and this

provides a structural guide which can be amended as the interview proceeds.

It may be feasible in some cases to send a questionnaire to the interviewee, have it returned and then conduct the interview to extend and to clarify the questionnaire information.

The analyst always has the option of gathering questionnaire information during the interview, editing the statements after the interview, sending the edited statements back to

the SME for review and/or evaluation. Upon completion, the SME would return the

questionnaire back to the analyst.

Observation – The job analyst observes the activities of one or more people and records the

observations. This is a wonderful method for analysts to understand what is done by someone in

the target position. It is especially helpful with jobs involving physical activity and the use of

tools and equipment. However, it is very time consuming. Indeed, when possible, the analyst

should observe a complete work cycle, noting tools and activities performed. Because of the

time commitment, direct observation, may not always be a viable option for PBT recruitments.

Literature Review - As previously noted under the discussion of expedited data gathering,

various job descriptive material may be used. This can include job specifications, position

questionnaires, prior job analyses, research reports, announcements and class specifications from

other agencies, job descriptions outlined in the Dictionary of Occupational Titles, training

manuals, etc. This method is particularly useful when working with a new class or a class that

has only a small number of available SMEs.


SUBJECT MATTER EXPERTS

It is impossible to do a good job analysis without considering what people are actually doing on

the job, and virtually impossible to do a good job analysis without some form of interaction

between the analyst and those with first-hand knowledge of the job. As previously indicated,

incumbents and supervisors typically serve the purpose of providing this important job

information. Incumbents and supervisors are termed “Subject Matter Experts” (SMEs) with

respect to the job they do or supervise. This term is something of a misnomer; incumbents, even

superior performers when in the entry level of their title series, are not necessarily “experts.” The

term is also used, within the test develop field, to designate consultants who develop test

material.

The analyst needs SMEs to generate, add to, delete from, modify or confirm materials

comprising the job analysis, which could consist of job behaviors (task statements), critical

incidents, job elements/sub-elements, activity elements, process statements, or worker

characteristics (KSAs). Consequently, a SME should have first-hand knowledge about the job;

if possible, each SME should have at least one year of experience performing or supervising the

job. Of course, when supervisors are used, the general rule is to try to use supervisors who

supervise the position. However, for some jobs, a representative of higher management may be

appropriate (e.g., where there are suspected differences in job activity by unit, shift, location,

work crew, etc. and the employee may be subject to reassignment). Also, in Position Based

Testing, oftentimes the hiring manager might be the only individual who can serve as a subject

matter expert when no permanent incumbents available. Additional considerations in the

selection of SMEs:

Permanent incumbents and supervisors should be used to: develop job tasks and the KSAs required to perform those tasks rate those tasks and KSAs in terms of importance, frequency, etc. establish linkages between the important tasks and KSAs, and the content used in the

selection process.

If possible, it is often advantageous to use supervisors who were promoted in the not-too-distant past from the job being analyzed.

Provisional employees, as a rule, should never be used. However, if no Permanent Civil Service or supervisory SMEs are available, a provisional employee may be used to only

identify job tasks. They should never be used to rate or weigh tasks or identify, rate or

weigh KSAs since they may be eligible to take the examination. DHR should be

consulted prior to any consideration concerning use of provisional incumbents in the job

analysis or test development process. Exceptions may be made if the provisional

employee signs a form or statement electing to not participate in the examination.

In unusual circumstances, SMEs from other departments or jurisdictions who are at or above the level being tested may be used.


Incumbents should have permanent status and preferably should be superior performers.

When panels of SMEs are used in the job analysis, they should be a representative sample of ethnic and gender workforce composition and range of experience, including different

position locations, shifts, etc., if and when appropriate.

While analysts may meet jointly with all SMEs when conducting job analyses, it is often a good practice to meet with the supervisor first, separately from the employees to avoid

any distraction or inhibition on the part of incumbents. The supervisor can initially

provide an overview of the job; give a perspective on how the job under consideration fits

in with other jobs within the organization; indicate what is expected by management from

incumbents in the job; and discuss special conditions, problems, procedures, or tools

associated with the job.

An important point to keep in mind is that the interviewer should elicit a description of the job behaviors from the supervisor, rather than have the supervisor merely confirm the

analyst’s summary of the duties and job behaviors or to confirm what was contained in a

previous job analysis or class specification.

Interviewing supervisors without interviewing employees should be done only when there are no permanent incumbents or the position is structured so explicitly that it is not

subject to modification by the employee’s style or approach.


SUBJECT MATTER EXPERT FORMS

PURPOSE: To ensure that participants in the job analysis and exam development

process are qualified and have no conflict of interest, and to document

participation and communication of confidentiality requirements.

FORM: Subject Matter Expert forms

PROCEDURE: All SMEs must complete the following two forms:

Subject Matter Expert Background & Qualifications

Test Security Agreement and Statement of Responsibility form.

The examination analyst provides the SME(s) with a copy of the

completed form and files the original with his/her examination records.


SUBJECT MATTER EXPERT

BACKGROUND & QUALIFICATIONS FORM

Classification Title _____________________________________________ Classification Number ___________________ Date: ________________________________ Name: _______________________________ Work Phone: __________________________ Current job class #: Title: Sex: Male Female Ethnic group: White

African-American Hispanic Asian/Pacific Islander

Filipino American Indian/Alaskan Native

Other _______________________ Department: ____________________________ Section/Division: ________________________________ Years of work experience in tested class: _________________ Years of experience supervising tested class: ________________ Briefly describe your experience, training and education in this area: _ ________________________________________________________________________________

Signature: Date signed: ______ -------------------------------------------------------------------------------------------------------------------------------- FOR DHR USE ONLY Participated in:

First Job Analysis Meeting Second Job Analysis Meeting

Test development: indicate type(s) of test Test review: indicate type(s) of test


City and County of San Francisco Department of Human Resources

Test Security Agreement and Statement of Responsibility Form

For

Test Development and Administration Staff

I understand and expressly acknowledge that:

The loss or disclosure of examination information or material, unintentional or otherwise, is a very serious matter as it can render a test invalid and useless. Since examinations represent a significant investment in time and money to develop and administer, any loss of test security can be very costly, disruptive and harmful to the operations of the Department of Human Resources (DHR) and/or other City and County of San Francisco (CCSF) departments. Further, the loss of examination and exam-related material undermines the public’s trust and confidence in this Department, the CCSF and the merit system.

Similarly, applicant or candidate information (including, but not limited to, test answers, test scores, and personal information such as addresses, social security numbers, disabilities, etc.) must be safeguarded and kept confidential. Everyone who is involved in test development and administration therefore has a special responsibility to uphold the public trust and merit system principles. Moreover, test development and administration staff is legally and ethically obligated to protect examination material and to maintain the confidentiality of applicant and test-related information. Everyone who participates in test development or administration must protect the value of secure examinations and observe security precautions when working with tests and test-related information. Indeed, there is no time during a test’s development or administration that the security of an examination or examination-related material is not the responsibility of those who are entrusted with these activities.

I am aware of the confidential nature of my work and therefore expressly acknowledge that:

1. I may be given access to confidential data, examination material or test-related information in association with work that I perform for the CCSF. Any such material or information for which I am given access is the property of the CCSF. Dissemination of this information or material to persons other than designated, authorized CCSF representatives is strictly prohibited.

2. I may be a party to conversations or discussion wherein confidential

examination-related information is discussed. Dissemination of the content of


these conversations to persons other than authorized CCSF employees is strictly prohibited. I am strictly prohibited from disseminating the contents of these conversations to third parties without the express written consent of authorized Human Resource representatives of the CCSF.

3. I must keep the content of all test questions and test-related material involving

examinations developed or administered by the DHR and/or other CCSF HR departments in the strictest confidence. I may not discuss or otherwise make available to anyone outside of my test development or administration duties any test questions, answers or stimulus material.

4. I am responsible for maintaining the confidentiality of all examinations and

examination-related material and must never leave such material or information unattended or unsecured. These materials at all times need to be properly safeguarded.

5. I may not participate in any examination for which I, a relative or a close

personal friend, have applied in any way. Should I discover that a relative or close personal friend of mine intends to apply or actually does apply, for a particular examination in which I am or may be a participant in either test development or administration activities, or otherwise have access to examination materials, I must immediately notify DHR’s Director of Recruitment and Assessment Services.

6. It is against the law to have in one’s possession at any time a copyrighted test

item or protected intellectual property for which one does not have permission. Test administration staff may only have such permission at the test center. Simply memorizing a test question and/or answer and writing it down later can be construed as a violation of copyright law. If test administration staff is asked to describe a test, test item(s) or answer(s) and the requested information is provided, this also may be construed as copyright infringement or a violation of intellectual property rights.

7. Study groups, test preparation businesses, etc., are known to try to acquire

test material. I understand that I must be especially careful in my interactions and conversations with others regarding my test-related work and report any unusual occurrences regarding inquiries, etc. from anyone who is not an authorized Human Resource representative. Neither I nor, to the best of my knowledge, any of my relatives or close friends have any personal or business affiliation or relations with any group or organization which may have an interest in examinations developed or administered by DHR and/or other CCSF departments that conflicts with the objectives of preserving test security and maintaining the confidentiality of test-related information.


In addition, I understand that I am subject to Department policies and State and local laws and rules governing the conduct of public officers and employees, including but not limited to:

Political Reform Act, California Government Code § 87100 et seq.;

California Government Code § 1090;

San Francisco Charter;

San Francisco Campaign and Governmental Conduct Code;

San Francisco Sunshine Ordinance;

Statement of Incompatible Activities;

Civil Service Rules; and

Any applicable departmental policies. I understand that I am required to report any violations, upon discovery, of the provisions stated herein committed by others, to the Director of Recruitment and Assessment, Department of Human Resources. I understand that engaging in the activities that are prohibited by this Agreement and Statement of Responsibility, or non-adherence to the terms of this Agreement and Statement of Responsibility, may subject me to discipline, up to and including possible termination of employment or removal from office, as well as prosecution, monetary fines and penalties. By signing this agreement and statement of responsibility, I acknowledge that it has been received, read and understood. PRINT NAME_______________________________________ SIGNATURE_____________________________ DATE______________________


ROLE OF THE JOB ANALYST

When possible, the analyst should prepare draft job analysis documents for the Subject Matter

Experts to review in advance of the meeting. Possible changes to this information can be

discussed during the job analysis meeting. Analysts may also want to gather and provide each

SME with a copy of the following documents:

previous job analysis report

previous examination announcement

classification documents submitted for the position Although an analyst may find an incumbent’s performance plan or appraisal useful to learn the

duties and responsibilities of a position, these documents should NOT be provided to the Subject

Matter Experts.

When first meeting with SMEs, generally, the analyst should explain:

what a job analysis is

what the components of a job analysis are

what the SME role in the process is

how the job analysis information will be used

the need for confidentiality and why SMEs must complete necessary forms (i.e., Background and Qualifications, and Test Security Agreement and Statement of

Responsibility forms)

In general, the analyst must carefully review and assess job analysis information provided by

SMEs and be in a position to confirm and/or contest the following:

General description of the position – Does it align with the class?

Essential duties and job behaviors – Are they appropriate to the class and position?

Knowledge, skills, and abilities – Are they linked to the duties and behaviors?

Minimum qualifications – Are they tied to the job duties?

Desirable qualifications (For possible use in the post-referral process) – Are they related to the job?

Legally mandated licenses, certificates or other requirements

For documentation purposes, the analyst must include the following information and file it for

record purposes:

Who conducted the job analysis and contact information for that individual

Date(s) and location(s) of the job analysis meetings

A description of how the job was analyzed

Steps taken to ensure the accuracy and completeness of the collection, analysis and report of the data and results (e.g., SME instructions)

Identification of all job analysis participants (incumbents? supervisors? race? gender? tenure?) and how they were selected for participation


For each SME, a completed Background and Qualifications form and a Test Security Agreement and Statement of Responsibility form.

Signed and dated job analysis rating forms completed by SMEs

If a questionnaire was used to collect job analysis data, a copy of that questionnaire and all of its rating scales should be documented, along with data collected by that

questionnaire

Descriptions of the work behaviors, tasks, and/or work products identified as important, along with measures of their criticality and/or relative importance and level of difficulty.

Where appropriate, description of the knowledge (in terms of a body of learned information), skill or ability (in terms of observable behaviors) that are used in the work

behavior(s) and their relative importance

Evidence of how the KSAs are related to the job behaviors

When a job analysis is updated, documentation regarding each SME’s revision of the tasks, KSAs, ratings, and linkups. [In some cases it may be important to note the reason

for the change(s)]

Significant changes to the job analysis often will require an official amendment to the job

specification. In particular, changes to the minimum requirements will require such an

amendment. In these cases, the analyst should consult with the Department of Human Resources.

Based on the results of the job analysis, some classes may be deemed suitable for flexible

staffing. Generally, this is used for trainee or entry-level classes to progress to the journey level

without an additional Civil Service examination process. Consult with the Department of

Human Resources, Recruitment and Assessment Services Unit if you believe that the recruitment

you are working on may be appropriate for flexible staffing. Flexible staffing will be a rare

occurrence under Position Based Testing, but it may be possible in certain situations involving

highly specialized positions within a class series.

DESCRIBING JOB BEHAVIOR

As noted above, one of the job analyst’s key objectives is to obtain information about the worker

activities or behavior performed on a given job. Another is to seek information as to the job

requirements, which are measured in terms of needed worker characteristics.

Writing Critical Incidents

When the analyst wishes to focus on a worker’s primary activities or behavior, but not

necessarily on worker characteristics, the Critical Incident method of information gathering is

often used. This method is particularly applicable to physical or operational activities which are

part of a job but are not specified by a task approach to describing a job.

Writing a behavior description using this technique, the following basic components should be

included:

The situation, circumstances or the setting of the “story” The behavior or activity performed The consequence or result of the behaviors


To elicit critical incidents, SMEs may be asked one of the following questions:

“Think of an incident indicating outstanding (extremely effective) performance in the ___________position. Describe the situation, the behavior and the consequences. It may

help to think of a person who is outstanding, and then think of incidents observed.”

“Think of an incident that indicated less than effective (poor) performance by a _______. Describe specifically the situation, the behavior, and the consequence.”

Here are two examples of critical incidents that correspond to the above questions respectively:

“A modification in the salary grades produced some discontent among employees. After

detecting this discontent, the manager brought his supervisors together for a meeting. In the

meeting he explained both the benefits and drawbacks of the new compensation

arrangements. He then asked his supervisors to return to their units and explain the new

compensation plan. As a result, morale was improved and the employees were more satisfied

with their treatment by management.”

“When changes in salary ranges were made, a supervisor explained to his subordinates that

the change was a matter of company policy and that no actions on their behalf could be

taken. As a result, morale deteriorated in this department.”

Writing Task Statements

A typical way of describing job behavior is by a task statement. A task statement should reflect

a single, discrete, purposeful activity. It is the smallest component of a purposeful job activity

but not so small that it no longer meets the definition of “purposeful work.” A common sense

way to decide on the scope of a given statement is to ask oneself, “Would one normally be

considered to be paid to do that activity?”

There are no firm rules on the number of task statements to be used to describe a job. However,

a practical way to decide how many is to divide the job into broad functional areas (duties) and

then determine the minimum number of task statements necessary to cover the purposeful

activities in each area. An analyst has a sufficient number of statements when they are

collectively exhaustive and mutually exclusive. That is, it is collectively exhaustive when it

includes all the significant purposeful activity that comprises the job. It is mutually exclusive

when none of the task statements overlap.

At a very minimum, a task should contain an action and an object of that action. However, a

widely used format involving task statements includes five components that are combined to

form a simple declarative sentence as seen below. Note that, the task statement generally starts

with a verb in the third person singular; the subject of the sentence is not written, but is always

implied to be, “the worker.” The following are the five, commonly-accepted, components of a

task statement.


Performs what action? – This is the verb that tells what action is being done. The verb should

have a single, unambiguous meaning. If the verb is indefinite, unclear or ambiguous,

misunderstandings may occur, or else the verb might not really say anything. Below are some

common verbs that are usually not definite enough for task statements. Their use should be

avoided or clarified.

administers, analyzes, arranges, assesses, assists, assumes, assures, audits, collaborates,

cooperates, coordinates, counsels, develops, discusses, examines, facilitates, follow-up,

handles, helps, interviews, investigates, keeps, maintains liaison, manages, participates,

prepares, processes, is responsible for, reviews studies, supports, teaches.

To whom or what? – Grammatically, this is the indirect object, or a phrase that serves the same

purpose. Not all task statements will allow for this component.

To produce what? /Why? - This component tells the purpose of the activity and, grammatically,

is usually the direct object of the verb.

Using what tools, equipment, work aids procedures? – Things mentioned should be specific and

tangible, if possible. If there is no tangible and identifiable tool or procedure used, then this

component may be omitted from the task statement. KSAs should not be substituted for tools or

procedures.

Upon what instructions? - Instructions may be directions from a supervisor or procedures

specified in a document.

Two examples of task statements using the component format described above:

“Evaluates proposed construction projects to determine environmental impact according to laws

and regulations.”

“Initiates contact with client to explain program services either in person or by telephone taking

the names from ‘Intake Roster’.”

WORKER CHARACTERISTICS

Writing KSA Statements

Task statements describe behavior; KSA statements describe the inferred qualities which

underlie the behaviors. KSA statements do not have a standardized format but there are a

number of considerations to keep in mind when writing KSA statements:

Scope and Number of KSA Statements – In any job, it is usually possible to infer a large number

of worker characteristics that have “something” to do with the job; how large a number depends

on the level of abstraction used. The key is to find those, for which the degree of possession,

will lead to discernible and significant differences in the quantity or quality of job performance

(ranking KSAs), as well as those that represent essential characteristics for meeting a minimum

level of job competency (qualifying KSAs).


Specificity – It is important that the KSA statement be specific, terse and direct, telling exactly

what characteristic is needed. This does not mean that a KSA needs to be specific to one task.

KSA statements should not be so nebulous that, although related to task performance, there is

confusion over exactly what is required. For example, writing “mathematical ability” may leave

us wondering what that means, especially when the true requirement is the “ability to add and

subtract whole numbers.”

Overlap – A KSA statement should never be overloaded by having unrelated characteristics

subsumed within it. As far as possible, the KSA statements, as a set, should be mutually

exclusive and collectively exhaustive, the same goal set for task statements.

Linkage – The KSA must be “linkable” to at least one job behavior; if it cannot be linked to any

behavior it definitely does not belong in the job analysis.

Intrinsic or Ancillary – Some KSAs may be linked to behaviors without having much bearing on

those behaviors. Such worker characteristics are not important to the performance of the

behavior (i.e., they are not intrinsic to the behavior). This is not to say that these worker

characteristics have no relations to the behavior; rather, they are “nice-to-know,” but are trivial,

ancillary details associated with the job. If a KSA has a direct bearing on performance of the

behavior, however, it should be included in the job analysis. [Another way of handling trivial or

ancillary KSAs is to include all KSAs in the job analysis, have them rated for importance, and

then drop out the trivial KSAs based on those ratings.]

Evaluating KSA Statements

When a worker characteristic is proposed during a job analysis it is necessary to evaluate that

worker characteristic in several ways:

1) To determine if the worker characteristic is really part of the job; 2) To edit the worker characteristic statement so that it constitutes a precise operational

definition in terms of job observables;

3) To determine if the worker characteristic can be used to rank candidates in order of merit for the job (ranker) or if it is a pass/fail qualification for minimal job competency

(qualifier); and

4) To determine if the worker characteristic is acquired through on-the-job learning (OJL) or whether it is something the worker must bring to the job (BTJ)

On-The-Job Learning or Brought-To-Job?

Worker characteristics which can be acquired after beginning the job (OJL) should not be

represented on a test. However, this doesn’t mean that OJL KSAs shouldn’t be stated in the job

analysis. To the contrary, there are at least three good reasons to include OJL KSAs in the job

analysis.

1. If nothing else, accounting for all KSAs provides protection against later appeals that the test did not cover all relevant areas. Indeed, since OJL’s are part of the job and workers


are expected to acquire them at some point, they are needed to ensure completeness of the

job analysis.

2. If the title is not a trainee title, and if a large proportion of the KSAs are learned on the job, perhaps the title is being used inappropriately. Conversely, if the KSAs are rated as

needed at job entry and the minimum qualifications suggest otherwise, there also may be

a classification issue.

3. Elimination of OJL KSAs can exaggerate the importance of the remaining KSAs if only the remaining KSAs were rated for importance relative to a given behavior.

The type of questions asked by the job analyst may have a bearing on whether the KSA will be

classified as OJL or BTJ. The analysts should avoid asking SMEs if the KSA can be learned

during the probationary period. In most cases, the honest answer is that any worker

characteristic, considered in isolation, could be object of intensive study or practice and possibly

learned by a new employee within three months. But the reality of the job is that one typically

does not receive three months’ salary to learn a single KSA. Hence, a better strategy would be

for the analyst to use the following pair of questions:

Is this worker characteristic going to be taught?

Is this worker characteristic something that most workers could learn on their own quickly?

In some situations the job analyst will have to use careful judgment to determine if a proposed

worker characteristic is actually OJL or BTJ, regardless of the initial claims of SMEs. Two such

situations are:

1. Although worker characteristics taught on the job are not proper material upon which to base a test, there are situations where training programs presuppose a certain level of

competence in worker characteristics. In such cases, the training program builds upon

what is brought to the job. The training, then, provides an OJL extension for the BTJ

worker characteristic. It is permissible to test for the worker characteristic at the level

prerequisite for training. The content of the test, however, must not cover that content

pertaining to the worker characteristic which will be presented during training.

2. The job analyst must be alert to situations where worker characteristics are being proposed as qualifying or important ranking KSAs when a more reasonable judgment

would be that these could be acquired on the job easily by anyone proficient in the other

aspects of the job. It is important, therefore, to consider how the present incumbents

acquired these worker characteristics, but it is also important that the job analyst make a

prudent judgment, regardless of the claims of the SME’s, of the reasonableness of

requiring these worker characteristics at entry.


GENERAL INTERVIEWING GUIDE FOR AN ANALYST

(Assumes a single employee interview performed without tasks and KSAs drafted in advance.)

Introduction

Introduce self, explain the purpose of the meeting and that you will be taking notes. [Make notes unobtrusively.] Consider using a laptop/project.

Reassure the employee that the interview does not concern performance evaluation or reclassification.

Job Behaviors and Job Tools

Have the interviewee give an overview of the job. Ask the interviewee to describe the major job duties in his own words; get purposes and broad duties. Do not go through the

job specification point by point.

Ask the employee to describe exactly what he does on the job during a typical day. This is to break down the duties just described into specific behaviors.

Ask the employee about how important they are and how frequently they are performed.

Ask the employee about the tools or manuals that s/he uses on the job. Ask if manuals, references and tool usage have to be known from memory, or if they can be looked up as

needed?

If the interview does not uncover important expected activities or tasks, ask about them, but do not “force” them upon the employee. Resolve every ambiguity immediately.

Place the job in the organization

Ask the employee to describe how the performance of this job relates to other jobs and people.

Determine to whom the employee reports on a regular basis and whether there are others to whom the employee may report on a sporadic basis. If a supervisory position,

determine who is supervised and how much of the work is done by directing others.

Determine how this job fits into the overall organization and whether interaction is needed with other employees or outsiders to accomplish the purpose of the job, or

whether the work is “self-contained.”

Determine whether all work is performed at a work station, if there is fieldwork, traveling, meetings, etc.

Determine if the job entails working under any “unusual” conditions. Worker Characteristics

Ask about the knowledge that the employee needs to do this job.

Ask about the kinds of abilities and skills that a person must be able to do on this job.

Ask the incumbent about on-the-job learning and whether a person must know or do this when first hired or promoted into this job. Ask whether orientation is normally given,

whether courses are taken, etc.

Ask if there are “qualifying” KSAs; things which are necessary to do on the job but which do not lead to better performance.

Differentiating the Elements

Ask the employee for the hardest or most demanding part of the job.

Ask about what gives newcomers the most trouble and in what areas are newcomers best/least prepared.


Ask the employee if there are things done on the job that an unprepared person could “let slide” for a while.

Ask the employee if there are things which, if not done properly, will get the person in trouble almost immediately.

Ask about the qualities or characteristics of the people who do well on this job. Explore things such as volume of work, meeting deadlines, staying within budget, high-volume

productivity, absence of errors, speed in performance, or quality-of-product indicators.

Ask about the deficiencies of people who cannot do the job. Determine the things that would lead to job failure.

Wrap-Up

Ask if there is additional information that hasn’t already been covered.

If the analyst’s notes are adequate for this purpose, allow the employee to rate job behaviors or worker characteristics statements taken from the notes. Otherwise, these

ratings may be postponed until the analyst has properly drafted these statements.

Secure, or make note of, forms or manuals which might be of value either to document the job analysis or to provide test material.

Thank the employee and end the interview.


JOB ANALYSIS FORMS

As indicated earlier, analysts will need to document the job analysis. To do this, it is

recommended that analysts use the job analysis forms found on the following pages. [You may

also use the job analysis forms and instructions found in WRIPAC’s Job Analysis Manual.]

Procedure:

Ask each SME to complete the Task Rating Form and the Knowledge, Skills, and Abilities Form.

For the Task Rating Form, the activities to be performed by the position should be listed.

In preparing the draft of the form for SMEs to review, the analyst may want to extract duties and responsibilities listed in the class specification.

The analyst may also extract the knowledge, skills, and abilities listed in the class specification to populate the Knowledge, Skills, and Abilities Form.

Number each of the tasks and KSAs on the form.

Compile the results to determine highest rated tasks and KSAs. Drop tasks/KSAs as necessary based on cut-off criteria.

Complete the Task and KSA Linkup Worksheet. Identify KSAs within each task and then sum these identifications across the tasks li

esources position based r testingmission.sfgov.org/document_center_documents/dc3259.pdf · section...

Documents