a needs assessment of the letc’s internal self-assessment · conducting a needs assessment”...

1

© OPWL 2013. Do not distribute without written permission from the OPWL department at Boise State University.

A Needs Assessment of the LETC’s Internal Self-Assessment

James Corbett, Dyan Ellinger, Savannah McEntarffer, and Alan Wheaton

OPWL 529 Needs Assessment

Boise State University

Fall 2013

2


Table of Contents

Executive Summary……………………………………………………………………..3

Introduction……………………………………………………………………………...5

Define Needs…………………………………………………………………………….6

Interventions…………………………………………………………………………….13

Recommendations………………………………………………………………………19

References………………………………………………………………………………20

Appendix A: Supporting Data…………………………………………………………..22

Appendix B: Performance Technology Frameworks…………………………………...23

Appendix C: Lessons Learned…………………………………………………………..30

Appendix D: Triangulation Data………………………………………………………..31

Appendix E: Interview Questions……………………………………………………….34

3


Executive Summary

Purpose

A needs assessment was conducted to evaluate the ABC Law Enforcement Training

Corporation’s (LETC) internal self-assessment process in preparation for federal accreditation and to

provide the CEO with recommendations to improve the quality of the internal assessments.

Current state of performance: 60% to 75% of the training accreditation standards are NOT met

during the internal self-assessment.

Desired state of performance: 100% of the training accreditation standards are met during the

internal self-assessment.

Objectives

Based on the purpose and goals of the needs assessment, four objectives were established for the

needs assessment.

1. Given the available data, define the performance gap related to the internal self-assessment results

in preparation for federal accreditation.

2. Using systematic tools, identify causes of the performance gap through analysis of evidence

based data.

3. Once the causes have been identified, develop interventions that address the identified causal

factors.

4. Using the determined interventions and performance tools, prioritize the interventions to ensure

effective and efficient recommendations are made.

Process and Findings

After identifying the actual and desired state of performance, the causes and potential

interventions were sought. The process utilized several HPT frameworks followed by rigorous data

collection which included several interviews with key stakeholders, numerous observations by the client

liaison, task analysis and extensive document review of emails, past assessments, and various doctrinal

instructions for the LETC organization.

Using the frameworks and data collected, the team identified the following primary deficiencies

affecting desired performance:

Lack of clear processes for internal assessments

Lack of constructive feedback to improve the overall process

Lack of a viable tool to manage the entire process

These are the general causes of poor performance; however, these causes of poor performance

and others were identified during the team’s assessment and are discussed in greater detail in the final

report. The HPT frameworks and the analysis combined with the data collected identified an operational

level performance issue with the potential for systemic effects at the tactical and strategic levels of the

organization. The fishbone diagram included below represents this examination, utilized with Thomas

4


Gilbert’s Behavior Engineering Model and its categories of data, instruments, incentives, knowledge,

capacity, and motives.

Recommendations

Following the identification of the causes of the performance deficiency, the team was able to

identify several performance interventions that would remove the barriers to desired performance. Using a

multicriteria analysis to compare these interventions based on expected results, feasibility, cost, ability to

accomplish the desired outcomes, and acceptability, the team recommends that ABC LETC consider the

following performance interventions.

1) Checklists for accreditation preparation processes

2) Tools for tracking accreditation processes

3) SharePoint Training

4) Process and Communication Flow Chart

5) Updates to LETC Organization and Regulations Manual

6) Dispute Resolution Process

While these six interventions were chosen based on their potential for a significant impact on

closing the performance gap, if resources are available, the LETC should also consider implementing

some of the remaining interventions that were identified. The analysis team recommends the organization

develop a comprehensive plan to implement the alternative sets of interventions and continuously monitor

and evaluate the performance of the internal self-assessment.

5


Introduction

Client Organization

The ABC Law Enforcement Training Corporation (LETC) is the client organization of this needs

assessment. The LETC was established in 2004 to prepare ABC personnel to perform as law enforcement

officers. The LETC enhances the law enforcement skills of students from other federal, state and local

agencies, as well as the international community. This is accomplished through the delivery of high

quality training that provides the knowledge and skills necessary for graduates to perform in a safe, legal

and professional manner. The LETC also develops, maintains and makes readily available up-to-date

training materials that support the standardization and professionalism of the LETC’s entire law

enforcement training system. The LETC and other law enforcement agencies also obtain accreditation for

some courses, through their accrediting body, BOSS.

Reporting

The needs assessment team reported to John Smith, Chief Executive Officer (CEO), ABC Law

Enforcement Training Corporation.

Team & Other Stakeholders

The LETC members that participated in the needs assessment are Ken Jones, Accreditation

Manager (Training Division), Bill Durango, Assistant Accreditation Manager (Training Division), Lee

Clayton, Instructional Designer (BOSS Assessor/Performance Systems Branch), Ken Crooker, Training

Specialist (BOSS Assessor/Performance Systems Branch), and Tom Brady, Training Officer.

The needs assessment team included graduate students from Boise State University: James

Corbett, Dyan Ellinger, Savannah McEntarffer, and Alan Wheaton. While Mr. Wheaton was the client

liaison and member of the LETC, all four team members conducted the research, contributed to project

deliverables, tracked the project schedule, collected and analyzed data, and developed potential

interventions related to the findings. The project was conducted remotely because all team members are

located in different states.

Purpose

A needs assessment drives thoughtful organizational decisions. Starting with identification of a

performance gap and the desired performance, a needs assessment can evaluate causes of a gap, identify

and analyze data related to the gap, and ideally populate effective ideas to address the gap. “By examining

what results you are achieving today and what results you want to accomplish tomorrow, you are

conducting a needs assessment” (Watkins et al., 2012, p. 19).

In the case of the LETC, the needs assessment’s purpose was to provide the CEO with

recommendations to improve the quality of their internal self-assessment used during the accreditation

process.

Goals

In order to achieve the needs assessment’s purpose, goals include the collection and analysis of

data to create a comprehensive set of performance solutions using human performance technology (HPT).

6


Directly tied to the needs assessment graduate students’ program, the HPT process was both

systematic and systemic. The needs assessment adopted a systematic methodical approach, as well as a

systemic consideration of how the focus impacted the whole organization. By driving a systematic

process, each step informed the questions and answers at the next level (Rossett, 2009, p. 42). Likewise,

although the focus of the needs assessment was at the operational level, inputs and outputs systemically

impact tactical and strategic levels, in addition to the interconnected systems at work in any organization,

including social systems.

Ethical considerations and professional standards were actively discussed and utilized throughout

the process. Particularly of concern was the political climate that may be roused with the needs

assessment and, the ethical factors of an internal team member with deeper knowledge and relationships

beyond this needs assessment.

Other goals included alignment of the recommended solutions at all levels of the organization,

even though the focus spotlighted the operational level. Most importantly, a goal was to provide solutions

that impacted results of the organization. The thought process of the team maintained focus on the

internal self-assessment with consideration to the ripple effect among the organization’s staff and

operations.

As a team, the goals included respectful practice and dialogue among internal team members and

with the client organization members. This was accomplished by discussing expectations at the beginning

of the project. Finally, the team vowed to learn about evidence-based needs assessment work, from the

collection and analysis of data to the solutions ideated by the team. Although professional “hunches” and

suspected issues were discussed, the team committed to only utilizing findings that were supported by

data-driven evidence.

Objectives

Based on the purpose and goals of the needs assessment, four objectives were established for the

needs assessment.

1. Given the available data, define the performance gap related to the internal self-assessment results

in preparation for federal accreditation.

2. Using systematic tools, identify causes of the performance gap through analysis of evidence

based data.

3. Once the causes have been identified, develop interventions that address the identified causal

factors.

4. Using the determined interventions and performance tools, prioritize the interventions to ensure

effective and efficient recommendations are made.

Define Needs

Methods Used to Identify Needs

Several HPT frameworks were used during the performance analysis including Joseph Harless’

front-end analysis model (1973), the Robert Mager and Peter Pipe performance analysis flow diagram

(1997, p. 5), Geary Rummler and Alan Brache’s three levels of performance (1995, p. 19), Watkins et

7


al.’s logic model (2012, p. 61), and Thomas Gilbert’s Behavior Engineering Model (1996, p. 83). Each

helped to evaluate the current and desired performance. All of these frameworks together provide a path

of analysis from the more granular details of the work, how those details connect to the operational goals

within the organization, and finally align to the more external impacts of the organization in its

community

Performance Analysis Flow Diagram

At the initial onset of the project, the performance analysis flow diagram (Appendix B, Figure 1)

was used as an efficient and effective way to examine what the performance gaps of the problem are and

whether it’s worth solving. In the case of this performance problem it appeared that there was a lack of

clear expectations, feedback was lacking, and consequences were misaligned. It also appeared that

resources were adequate and there wasn’t a genuine skill deficiency as evidenced in the success of

previous assessments. The data used to make these determinations came from observations, informal

interviews and a comparison of past and present assessment reports. This tool quickly revealed that the

performance problem was worthy to pursue. In a systematic way, the team could proceed through each

part of the performance analysis flow diagram and answer each question.

13 Smart Questions

The 13 Smart Questions (Appendix B, Figure 2) verified the information studied in the

performance analysis flow diagram. In addition, Harless’ questions helped to identify general types of

solutions, costs and development times, constraints, and overall goals. These questions encapsulate what a

manager, trainer, or performance technology consultant should ask before choosing solutions for a

performance problem (Harless, 1973, p. 231). The combination of the Performance Analysis Flow

Diagram and Harless’ 13 Smart Questions provided the team the evidence needed to continue pursuing

the performance problem.

Anatomy of Performance

The team used Rummler and Brache’s Anatomy of Performance (AoP) model (Appendix B,

Figure 3) to evaluate the LETC’s internal accreditation processes because this model can be used for

analyzing complicated issues that span multiple levels of an organization. The AoP illuminated that the

LETC is experiencing a process related performance deficiency, among other smaller issues, at the

operational level of the organization that has repercussions that span both the tactical and strategic levels

of the organization.

Logic Model

Logic models are common planning, monitoring, evaluating, and communicating tools used to

guide development projects (Watkins et al., 2012, p. 61). The logic model visually represents the

relationships among the resources operating in the LETC, the activities to plan, and the results the team

hopes to achieve. After completing the logic model, the team identified that the gap was located at the

operational level of the organization. Next, the team processed how the gap is connected to the tactical

and strategic levels of the organization. The logic model kept focus on the outputs and operational level,

while not straying out of scope when using data and interventions. By examining these relationships in

the project, the needs assessment team could determine the sequential connections that will deliver short

and long-term desired results.

8


Behavior Engineering Model (BEM)

The BEM was used to classify the root causes of the performance gap. Combined with other

models and frameworks, the BEM made sense to the project team, because the performance problem

primarily resides at the operational level of the organization. The BEM is a powerful tool to gather data at

the worker (operational) level and to distinguish whether performance problems are related to the person

or the environment. Additionally, the BEM dovetails easily into the nine performance variables of the

Anatomy of Performance (AoP) framework that allows for a clear connection to all three levels of the

organization, namely the worker, the process, and the organizational levels.

Data Identifying Needs

Using the methods to identify the needs the data was collected and scrutinized in a systematic

process to determine causes. The desired performance is linked to the operational, tactical, and strategic

goals of the LETC, which is also a goal of ABC Headquarters. The accreditation manager at the LETC

performs the work and is responsible to establish effective and efficient processes to conduct the work

(Accreditation Manager job description).

Current state of performance: 60% to 75% of the training accreditation standards are NOT met

during the internal self-assessment.

Desired state of performance: 100% of the training accreditation standards are met during the

internal self-assessment.

Evidence of the current performance is documented in two separate emails from Lee Clayton

(2013, September 25) and Ken Crooker (2013, September 24). Evidence was also collected in a face-to-

face interview with a third LETC assessor. Each assessor stated that only 25% to 40% of the standards

were met during the last internal self-assessment held in August 2013 (Clayton and Crooker, 2013

September). Personal observation of the accreditation files by Alan Wheaton substantiates that a low

percentage of files meet the standards. In addition, some assessors reported an overall decrease in the

quality of the BOSS files as compared to previous years. A comparison of past and present internal self-

assessment reports indicates a decrease of compliant files per the ABC internal self-assessments (2013)

and the LETC Facilitator Course self-assessment report (2010).

The LETC is required to achieve federal accreditation every year. They have successfully

achieved this goal each year; however, it appears that some of the internal processes leading to

achievement of this goal are beginning to break down. Most notably is what appears to be a breakdown in

the preparation and feedback processes. During the internal self-assessments prior to the actual

accreditation, the LETC has internal experts conduct self-assessments of each account that needs to be

accredited. Over time the scores achieved during this process have declined and the assessors have

noticed that the same mistakes are being repeated. As a result of the poor results during the internal

assessments, the LETC is forced to expend massive amounts of resources in the form of staffing, money,

and time to ensure that the accounts are ready prior to the actual accreditation. This reactionary posture

has placed a strain on the organization’s resources. Further complicating this situation is the fact that the

LETC, even with the failed internal assessments, has still managed to achieve accreditation each year,

which provides a false perception that the accreditation process is working properly.

The desired performance of 100% of the training accreditation standards met would ensure the

files were ready for the assessors review and provide a more efficient, consistent process, saving the

assessors’ time during the internal self-assessment, and the accreditation managers’ time during the

correction of mistakes prior the actual self-assessment. Once the preparatory processes are improved, the

9


LETC can begin using the internal assessment process as intended, which is as a measurement tool that

determines the readiness of the LETC for accreditation, and not as method for fixing inefficient processes.

Based on the results of the logic model the team determined that the performance gap should be

addressed due to the importance of its impacts at the tactical and strategic levels.

Input -time of the staff to conduct the internal self-assessment

-financial costs of staff and actual resources (technology, paper, facilities)

-procedures

-data collection tools

-databases

-external resources

Activities -observations

-interviews

-evaluations

-benchmarking

-change management

-corrective action program

-peer reviews

-building the files with evidence for review

-collecting evidence for the files

-reviewing of the files by assessors Outputs

(Operational)

-the LETC meets 100% of the training accreditation standards during the internal self-

assessment

-the LETC meets the BOSS accreditation standards Outcomes

(Tactical)

-supply the ABC with knowledgeable and skilled law enforcement professionals ready to

meet the demands of the nation at any time Impacts

(Strategic)

-provide the persistent presence of law enforcement officers

-provide safety, security and environmental stewardship

-provide the community with a sense of protection and security

The team utilized multiple data sources and tools in order to acquire various views on the

performance gap (Watkins et al, 2012, p. 48). This included extensive document review, task analysis and

several interviews with key stakeholders. Following the data collection, the data was coded and further

analyzed using a task analysis and completion of an Ishikawa Fishbone Diagram to determine the root

causes of the flawed internal accreditation process. Data were triangulated with more than one source

when possible (Appendix D, Triangulation Data). The steps below show steps we took to analyze the

data.

First, the team started with interviews because there is no current task analysis for the internal

self- assessment. Interview questions were developed to obtain data about current processes and

performance gaps. The questions used to obtain this data are listed in Appendix E.

The data obtained in the interviews can be found in Appendices B and D. A brief summary of the

major points found during the interviews are listed below:

There is no current consistent process for internal self assessments.

Feedback is not provided.

There is no incentive to change the current process.

Using the data from the interviews we completed a task analysis for the internal self-assessment process.

10


Using the data collected (interviews, observations, and document review) the team further

analyzed the problem by answering Rummler and Brache's questions as provided in the AoP model

(Appendix B, Figure 3). From that analysis, the team noted that several factors influencing the internal

accreditation process involved the Human Performance System.

To further investigate the causes of human performance deficiencies, an analysis was conducted

using the Factors Affecting Human Performance System (Wallace, 2013). This analysis was conducted

from the perspectives of the accreditation manager, the assistant accreditation manager, the internal

assessors, and the accreditation file managers(Appendix B, Figure 4).

The Ishikawa fishbone diagram then allowed the team to visually identify the root causes of the

problem. The problem- the flawed internal accreditation process- was placed to the right at the head of

11


the fish (per the model below). Just as the team used the BEM for data coding purposes, the team further

used the BEM’s six categories of Information, Instrumentation, Motivation, Knowledge, Capacity, and

Motives to organize the fishbone diagram. The secondary causal factors of poor performance were then

indicated on the smaller ribs parallel to the spine of the diagram. This simple depiction provides a clear

visual of all of the potential causes leading to the problem of the flawed internal accreditation process.

12


From the diagram the team could immediately see the causes, or what Rossett (2009) terms

“performance drivers...the factors that block or aid performance” (p. 49), such as available data, processes

and tools in the internal self-assessment, as well as motives and incentives. For example, no established

process is followed for the preparation and maintenance of the accreditation folders, nor is there a process

to communicate with all stakeholders about the status of each accreditation folder. The accreditation

manager, assistant accreditation manager, and assessors do not prescribe to any checklist that would allow

for consistent review of the folders. Regarding incentives, the assistant accreditation manager receives

unintentional positive incentives to perform poorly by others correcting the training standards for him

during the internal self-assessment. These causes are further described below in the identification of

interventions.

The relationship of the performance gap, the root causes, and the classification of the root causes

was also helpful to examine in the six cells of the BEM.

Performance Gap Root Causes Classification (BEM)

~60% of the training accreditation standards are NOT met during the internal self-assessment.

Data sources: (Performers, Supervisors, Policy Documents, Other Documents)

Data collection methods: (Interview, Observation, Document Review)

No process for folder management

Data Feedback

Environment

No documented internal self-assessment process

Key stakeholder job tasks are unclear

No ongoing feedback process

No tool used to track and communicate accreditation status

Support Tools Resources No specific checklist

used by assessors

Accreditation managers awarded for poor performance Incentives

No consequences for poor performance

Accreditation managers do not know how to develop tracking tools

Knowledge Skills

Person

Assistant Accreditation Manager is overwhelmed with the accreditation processes

Capacity Master Training Specialists are no longer available to provide additional support to the Assistant Accreditation Manager

Internal self- assessors are no longer volunteering

Motives

13


Interventions

Methods for Identifying Alternative Sets of Interventions

After identifying the needs, defining the performance gap, and evaluating the causes of the

performance gap, interventions were explored. To identify interventions, the team processed verbally at a

synchronous meeting rather than through electronic, asynchronous communication to allow brainstorming

and quick dialogue of possible solutions. This allowed accurate understanding of each idea and

perspective, as well as synergy through the dialogue as we reviewed the intersections and patterns of the

data.

First, the team reviewed methods to identify alternative sets of interventions. The BEM and the

Anatomy of Performance (AoP) models provided structure to do this. For example, many data points

appeared in the environmental supports section of data. Problems with lack of process and

communication jumped off the BEM as primary issues. Considering the related organizational elements

from the AoP, solutions for these problems would be supported by senior leadership and be low cost to

the LETC. At the organizational level, any solutions must efficiently utilize resources and make sense.

In addition to the BEM and AoP, Appendix A4 in Watkins, West Meiers and Visser (2012) was

utilized throughout the needs assessment. Particularly at this stage, the “select potential solutions” section

for organizational needs and team performance encouraged the use of the multicriteria analysis (p. 271-

272), which is noted below in the criteria used to compare interventions.

Data and Criteria Used to Compare Interventions

Given the fact that the team identified 12 separate interventions, the multicriteria analysis was

used to determine which ones should be used to close the performance gap. Watkins et al. (2012) note

that the multicriteria analysis provides a systematic method to quantitatively compare multiple

interventions (p.171).

As part of the multicriteria analysis the team had to identify the appropriate criteria to consider

when scoring each intervention. Watkins et al. recommend considering five to eight attributes for this

process (2012, p. 173); the team decided upon five. The attributes the team considered were: expected

results after six months, feasibility of implementation, cost of activity over first year, ability to

accomplish desired outcomes, and acceptability.

Deciding upon appropriate attributes to consider was critical to distinguish among the multitudes

of interventions. ‘Expected results after six months’ was chosen as an attribute because it would provide

an idea of which interventions would have immediate and substantial impacts. The ‘feasibility of

implementation’ provided a reality check as to whether a considered intervention was reasonable to

implement. The ‘cost of activity over the first year’, which is also linked to feasibility, provided an idea if

the intervention could be implemented at a low cost, or if it was cost prohibitive.

Perhaps the most important attribute considered, was the ‘ability to accomplish desired

outcomes,’ which took into account the ability of the proposed intervention to close the performance gap.

Lastly, ‘acceptability’ was considered from the perspective of the organization’s likelihood of embracing

the proposed intervention. Considering that the LETC is a vertically structured organization that closely

follows the chain of command, ‘acceptability’ was weighted based on the perceived likelihood of the

LETC’s leadership accepting the intervention.

14


After identifying the appropriate criteria to distinguish between the interventions, the team

considered each intervention individually and scored each attribute between one and ten (one being the

least effective and ten being the most effective). After scoring the five attributes for each individual

intervention, the scores were totaled and divided by five. This provided an average score for each

intervention and provided a simple method to compare the interventions.

Interventions and criteria scores (note a more thorough explanation of each intervention is

provided later in this report):

Interventions Criteria for Multicriteria Analysis Expected

results after six months

Feasibility of Implementa-tion

Cost of activity over first year

Ability to accomplish desired outcomes

Acceptability Average Score

Accreditation Job aid/

process flow

5 6 7 10 9 7.4

Updates to LETC Organization & Regulations Manual

2 7 6 10 1

0

7.0

Dashboard for tracking accreditation status

7 9 9 9 1

0

8.8

Checklists (for

accreditation

preparation

processes)

8 8 9 9 1

0

8.8

Performance Management (expectations/ accountability)

5 5 9 7 8 6.8

Schedule for

assessors

3 1

0

0

1

0

3 7 6.6

Incentives for assessors

3 5 8 4 5 5

Dispute resolution process

5 7 9 7 7 7

Documented role

clarity with task list 2 9 1

0

5 9 7

Onboarding

checklist for

stakeholders

3 5 6 7 8 5.8

SharePoint training (For AM/AAM/ Stakeholders)

5 9 8 8 9 7.8

Community of Practice

1 3 1

0

8 6 5.6

15


Prioritized List of Interventions

Based on the above criteria comparing possible solutions, the interventions were discussed and

prioritized. Each intervention possesses advantages and disadvantages.

1. Process and Communication Flowchart: Multicriteria Score 7.4 (Classification- Environment/

Data)

A process and communication flowchart, or an algorithmic visual display, of the internal self-

assessment will provide all stakeholders an important reference. The flowchart must include all steps as

well as the communication and feedback points from the point of preparing folders, to assessing the

folders, to what happens after the actual self-assessment. To assist accountability and true buy-in of the

process, the flowchart should be vetted with senior leadership at the LETC, and then be displayed for all

in an agreed upon shared site, either physically in the office or/and electronically via SharePoint, as well

as maintained in the organization’s manual. Visio could be a program to use that would allow the flow

chart to reflect the sequencing and if/then scenarios during the process.

Ideally, this intervention would envelope other interventions noted, such as the checklist created,

established schedule for assessors, documented role clarity, and a dispute resolution process.

Data from document review, multiple interviews and the task analysis concur that no consistent

process or exchange of communication exists (Appendix D, Triangulation Data). Currently, the AAM

collects the evidence when an assessment is nearing, and the assessors conduct the internal self-

assessment from their prior knowledge of the training standards. The assessors receive no feedback to

whether what they provided was needed (or appreciated) and are confused when the same mistakes occur

in folders multiple times. Others involved in the process share information on accreditation status verbally

in update meetings, however, most are not aware of the status or upcoming needs for assessments. The

flowchart provides the roadmap to solve this significant performance gap.

2. Updates to LETC Organization and Regulations Manual (OrgMan) and Documented Role

Clarity: Multicriteria Score 7 (Classification- Environment/ Data)

Recommend that the organization include BOSS standards numbers when there is an update to

the LETC OrgMan. By ensuring BOSS is included in the update means that all changes will comply with

both accreditation and the organization standards. This will prevent loss of accreditation standards while

maintaining organizational standards. If a committee is included in the process of updating the OrgMan,

the loss of accreditation standards will be prevented. The committee will include all contributing key

stakeholders to the accreditation process, and they will meet to approve any changes. Before approving

any change the key stakeholders will be able to review and edit proposed changes to the OrgMan.

Role clarity with a task list for all the key stakeholders is also recommended for this intervention.

By having a written task list for each stakeholder, the process will not become clouded or overwhelm any

one person. Each stakeholder will be aware of his/her specific contribution to the update process in the

OrgMan. This will prevent confusion about what needs to be done to update the OrgMan as well as who

participates in the process as the turnover rate is high at the LETC. There will be no confusion left about

what specific role they are expected to complete and what each role entails. This will benefit both the

proposed changes and stakeholders involved.

3. Dashboard for Tracking Accreditation Process: Multicriteria Score 8.8 (Classification -

Environment/Data)

Recommend that the organization use a dashboard or other dynamic display which will provide

all accreditation stakeholders with awareness of the current status of all critical accreditation milestones.

16


As mentioned by several stakeholders during the interview process, many of the personnel involved in the

accreditation process never know what the status of the greater process is. Instead, they are typically

notified at various times during the process and asked to provide assistance in a reactionary capacity.

By using a display in an area where all stakeholders, including leadership, can see a current

snapshot of the current accreditation status, all stakeholders will then be empowered to take action prior

to milestones becoming delinquent. The dashboard would be displayed on a large monitor or whiteboard

in the main briefing room for LETC personnel to see on a daily or weekly basis.

4. Checklists: Multicriteria Score 8.8 (Classification - Environment/Tools, Resources)

Recommend the organization coordinates with other accredited agencies to develop a standard

checklist to be used throughout the entire accreditation community. If this is not feasible, the organization

should create their own checklist to prepare for and conduct internal self-assessments.

From 2003-2013 standardized checklists were used by accreditation managers as a job aid to

populate folders and were also used by assessors to evaluate each folder/standard. These checklists were

provided by BOSS and were required to be submitted to BOSS with the final assessment report. In the

summer of 2013 BOSS removed the requirement to fill out and submit checklists for each folder. In fact,

BOSS removed the checklist from their website and the checklists are no longer available. BOSS

removed this requirement because they were unable to administratively handle the amount of checklists

received. As a result, some agencies have created their own checklists for internal use. These checklists

are not standard amongst each agency. Over the past 10 years the use of a standardized checklist has

proven to be an effective tool (job aid) for the accreditation manager to prepare folders and the

assessment teams to objectively evaluate each standard. The checklists were also used to by the assessors

to provide objective feedback to the accreditation manager based on the standardized criteria.

5. Performance Management: Multicriteria Score 6.8 (Classification – Environment/Incentives)

Recommend that the organization incorporate completion of accreditation tasks as part of each

stakeholder’s greater performance goals and evaluations. From interviews with several stakeholders, it

was determined that positive and negative feedback relating to individuals’ participation in the

accreditation process is not factored into performance goals and appraisals.

This process would involve all supervisors communicating their expectations for the accreditation

process to each subordinate involved with accreditation. This would further involve each subordinate

communicating to their supervisors what tasks they have accomplished in completing their part of the

accreditation process. By doing this, ownership of the entire process is expanded from a few individuals

to all stakeholders involved. This further ensures expanded accountability which will lead to better care

taken to ensure accreditation folders are maintained in accordance with BOSS standards.

6. Set Schedule for Assessors: Multicriteria Score 6.6 (Classification- Environment/Incentives)

Recommend that a set schedule be made for assessors to participate in the self-assessment.

Currently assessments are done by volunteers and the volunteers (low number) are repeatedly the same.

Having a set schedule will ensure everyone contributes equally to the self-assessment preparation as well

as creating diversity among assessors. By rotating assessors, the same person is not repeatedly sacrificing

their allotted work hours to review folders, and the same person is not repeatedly contributing their

knowledge. The schedule will include all personnel and rotate for each assessment preparation.

17


7. Incentives for Assessors: Multicriteria Score 5.0 (Classification – Environment/Incentives)

Recommend assessors that volunteer for internal self-assessments are incentivized. The

organization has mechanisms readily available such as; time off awards, letters of appreciation, public

recognition, cash awards, and performance appraisals, to name a few.

Recently, several internal assessors have quit volunteering their time to conduct internal self-

assessments. These assessors are the same ones that normally volunteer and have done so for several

years. These assessors not only conduct internal assessments but also conduct assessments for other

agencies; they are experts at this job. The internal assessors stopped volunteering because they don’t

receive any feedback and they feel their time and expertise is not valued. Since the assessors quit

volunteering, a recent internal self-assessment was cancelled. As a result of not conducting an internal

self-assessment, the external self-assessment team (the next step in the process) reported numerous

deficiencies with many of the standards.

8. Dispute Resolution Process: Multicriteria Score 7.0 (Classification – Environment/Incentives)

Recommend that an informal dispute resolution process be created to provide a method for

ensuring divergences of opinions between the assessors and the assistant accreditation manager (AAM)

be identified and resolved quickly during the internal accreditation process.

During the interview process it was discovered that the assessors had become frustrated during

the internal assessment process when they noticed that the same discrepancies that they had already

identified during earlier assessments reappeared during subsequent internal assessments. They also noted

that the assistant accreditation manager did not communicate to them which specific recommendations

that they had provided that he was going to incorporate to ensure the folders were ready for accreditation.

This was confirmed in an interview with the AAM when he verified that he does not provide the assessors

with feedback and at times he chooses not to implement their feedback.

This intervention can only be implemented after intervention 1 (above), process and

communication flowchart, is implemented. This means that the assessors must be informed following

each internal self-assessment regarding what feedback is being implemented or not implemented after

they conduct their internal assessments. They will then consult with the assistant accreditation manager

on items that they did not agree upon and then come to a final mutual agreement on how similar situations

will be handled in the future. In the rare event that the AAM and the assessors cannot come to an

agreement, the final resolution can be determined by the AM or the training officer. Upon resolution, the

results need to be communicated to the appropriate stakeholders to ensure that all LETC parties have a

shared understanding of the accreditation process moving forward.

9. Documented Role Clarity: Combined with #2

10. Onboarding: Multicriteria Score 5.8 (Classification- Person/ Knowledge)

Establishing knowledge of and expectations for the accreditation process right when staff begin

employment at the LETC is the goal with adding this item to any existing onboarding checklists utilized

at the organization. Just as the organization would cover topics related to job expectations, benefits, and

organizational standards, onboarding can include discussion of the accreditation process and reference to

the process and communication flowchart that includes expectations for all staff involved.

At this time, not all involved in training are aware of the training standards or accreditation

process at all. Even at the leadership level, there exists a frustration with how much time is spent on the

18


accreditation process, as verified in interviews, resulting in the work happening at times “under the

radar.” Without true leadership support and understanding of the process, those involved have little

understanding or misinformation. Onboarding would create a clear picture from the start of the

requirements.

11. SharePoint Training: Multicriteria Score 7.8 (Classification – Person/Knowledge)

Recommend providing SharePoint Training for the assistant accreditation manager, the

accreditation manager, and other accreditation stakeholders. During the data collection process it was

noted that there is not a standard project management program for managing the accreditation process.

SharePoint has been identified as a viable resource for overseeing the accreditation process and it is

currently in use by the BOSS organization which is the largest federal law enforcement training center in

the U.S. and is also being used by several of LETC’s partner organizations. Further, SharePoint is

currently being used by several of LETC’s stakeholders for other non-accreditation purposes. However,

the personnel identified as key managers for the accreditation process currently do not have the skill base

necessary to leverage SharePoint as a process management tool.

The team recommends that the LETC work with a professional SharePoint trainer to ensure that

the managers learn how to use SharePoint to manage the accreditation process. In order for this training to

be successful and also lead to continued use, the team recommends that the training for the accreditation

manager and assistant accreditation manager go beyond simple use of the program, and actually

incorporate how to create a functional accreditation management project interface with all stakeholders

involved; they can then allow the managers to use the tool as they will in the future. As Peggy Ertmer and

Timothy Newby state in their article Behaviorism, Cognitivism, Constructivism: Comparing Critical

Features from an Instructional Design Perspective, “the ultimate measure of learning is based on how

effective the learner's knowledge structure is in facilitating thinking and performing in the system in

which those tools are used” (1993, p. 64). This training will be most successful in achieving transfer in

the workspace if the managers learn to use the program as they will actually use it at the LETC. This

intervention will also require the support of LETC leadership in setting the appropriate expectations that

SharePoint be used as the primary accreditation management tool in the future.

12. Community of Practice: Multicriteria Score 5.6 (Classification- Person/ Motives)

The needs assessment team discussed this as the most controversial potential intervention, as the

culture of the LETC has not fully embraced accreditation so establishing a community of practice may

require a shift in the culture. A shared exchange of the challenges and opportunities of the internal self-

assessment and accreditation process overall could be a motive for involved staff by sharing the load,

increasing collegial interactions and ideating future improvements together.

At this time, the relationship between the accreditation managers and the assessors has become

somewhat tenuous. Misunderstandings exist on both sides. A community of practice would provide an

informal learning opportunity for all involved to access one another. Wenger (2009) describes that a

community of practice reveals “emergent structure, complex relationships, self-organisation, dynamic

boundaries, ongoing negotiation of identity and cultural meaning” (p. 179-180). The expectation would

not be that the community of practice will immediately tie the LETC staff together, but it would allow an

informal framework to allow the relationships to grow with common goals of improving the internal self-

assessment and the overall accreditation process.

19


Recommendations

The needs assessment produced several recommendations from the team, which address the

operational, tactical and strategic goals of the organization. Due to the causes of the performance

deficiency, the team identified and recommends several performance interventions that would remove the

barriers to desired performance at the LETC.

Checklists for accreditation preparation processes (Classification Environment/Tools, Resources)

Data for tracking accreditation processes (Classification – Environment/Data)

SharePoint Training (Classification – Person/Knowledge)

Process and Communication Flow Chart (Classification – Environment/ Data)

Updates to LETC Organization and Regulations Manual (Classification – Environment/ Data,

Instruments)

Dispute Resolution Process (Classification – Environment/ Data, Instruments)

At the operational level, these recommendations plug the holes of lack of process, communication,

and tracking. All staff directly involved in the internal self-assessment will benefit by following agreed

upon processes, communication loops, and maintaining updated status reports via a tracking tool. In

addition, a dispute resolution process, even an informal one, establishes an agreed-upon procedure when

there may be confusion or disagreement throughout the process.

Likewise at the tactical level, by ensuring these processes establish consistency and preparedness, the

internal self-assessment is more likely to successfully renew accreditation at the agency level. The LETC

strives to ensure their courses prepare staff to the highest performance. Setting up the accreditation

process for success accomplishes this tactical level goal.

The intervention recommendations address the strategic level of the organization, as well. The

operational and tactical goals align with the strategic goal of maintaining an agency that instills “a high

degree of public confidence in the competence and professionalism of federal agents and officers” (BOSS

procedures and standards manual, 2010, p. 3). A small example is the tracking tools recommended, which

would allow transparency of the organization’s status, in addition to keeping all levels of the organization

informed.

The recommended interventions directly impact all three levels of the organization, which the team

focused on from the beginning of the needs assessment. The recommendations would produce efficient,

consistent process and communication among staff about the internal self-assessment, while setting up the

accreditation reviews successfully, which in turn meets the organization’s commitment to the community.

Last, the needs assessment team recommends the organization develop a comprehensive plan to

implement the alternative sets of interventions and continuously monitor and evaluate the performance of

the internal self-assessment. While these six interventions were chosen based on their potential for a

significant impact on closing the performance gap, if resources are available, the LETC should also

consider implementing the remaining interventions from the multicriteria analysis.

20


References

Clayton, L. (2013, September 25). Re: Needs assessment question (BOSS self-

assessments) [Electronic mailing list message]. Retrieved from ABC.com email server.

Crooker, K. (2013, September 24). Re: Needs assessment question (BOSS self-

assessments) [Electronic mailing list message]. Retrieved from ABC.com email server.

Ertmer, P. A., & Newby, T. J. (1993). Behaviorism, cognitivism, constructivism: Comparing critical

features from a design perspective. Performance Improvement Quarterly, 6(4), 50-72. doi:

10.1111/j.1937-8327.1993.tb00605.x

Gilbert, T. F. (1996). Human competence: Engineering worthy performance (Tribute ed.). Silver Spring,

MD: International Society for Performance Improvement. (Original work published 1978).

Governing Body of the LETC Accreditation (2010). BOSS Standards and

Procedures Manual. Retrieved from http://www.ABC.com

Harless, J. (1973). An analysis of front-end analysis. Improving Human Performance:

A Research Quarterly, 4, 229-244.

LETC (2010, April 13-14). Self-assessment Report for the The ABC

Law Enforcement Training Corporation Facilitator

Course Accreditation. Retrieved from the LETC database.

Mager, R., & Pipe, P. (1997). Performance analysis flow diagram. In Analyzing

performance problems, or you really oughta wanna: How to figure out why

people aren’t doing what they should be, and what to do about it (third edition) (p. 5). Atlanta,

GA: Center for Effective Performance.

Position Description (2010, February 4). Organization title: assistant training officer

and accreditation manager. Retrieved from LETC database.

Rossett, A. (2009). First things fast: A handbook for performance analysis (second edition). San

Francisco: Jossey-Bass/Pfeiffer.

Rummler, G. A., & Brache, A. P. (2013). Improving performance: How to manage the

white space on the organization chart (3rd ed.). San Francisco, CA: Jossey-

Bass.

Schensul, J., & LeCompte, M. (2013). Essential ethnographic methods: A mixed

methods approach (2nd edition). Lanham, MD: AltaMira Press.

Wallace, G. (2013, February 3). Geary A. Rummler on performance engineering 1986.

Retrieved from http://www.youtube.com/watch?v=XrzmZLislDs

Watkins, R., West Meiers, M., & Visser, Y. (2012). A Guide to Assessing Needs:

Essential Tools for Collecting Information, Making Decisions and Achieving

Development Results. Washington, D.C.: The World Bank.

http://www.youtube.com/watch?v=XrzmZLislDs

21


Wenger, E. (2010). Communities of practice and social learning systems: The career of a concept. In C.

Blackmore (Ed.), Social learning systems and communities of practice (pp. 179-198). London:

Springer.

22


Appendix A: Supporting Data Inventory List

(Note: The supporting documents are located in a separate PDF file)

Item Description Note 1: Yellow highlighting was used to indicate relevant data. Specific page numbers are indicated by an asterisk - 40 pages contain highlighting. Note 2: Text boxes (also highlighted in yellow) were used to provide additional comments by the internal analyst.

PDF Page Numbers

1 Interview: TO *1

2 Interview: Assistant Accreditation Manager *2 - *3

3 Interview: Accreditation manager and an Internal Assessor *4, *5, *6, & *7

4 Interview: Internal Assessor *8 - *9

5 Intentionally left blank Intentionally left blank

6 Interview: External Team Leader Assessor *10-11

7 Email: Internal assessor *12

8 Email: Internal assessor *13

9 ORGMAN Chapter 1 (Unit Organization)

15-28 (*16, *21, *23, *24)

10 ORGMAN Chapter 4. (Training Division)

29-40 (*32, *33, *39, * 40)

11 ORGMAN Chapter 5. (Performance Systems Division)

*41

12 Standards Manual 42-47 (*45, *46, *47)

13 Training material for Accreditation Managers 48-52 (*49)

14 Accreditation Schedule *53

15 Accreditation Tracking Document 55-68 (*55)

16 Position Description (Accreditation Manager, GS-13) *69

17 Position Description (Assistant Accreditation Manager, Contractor) *70

18 Standards Responsibilities Chart *71

19 AM Accreditation Checklist 72-76 (*72, *74)

20 Accreditation Status Report 77-81 (*77)

21 Assignment to Duties Instruction 82-89 (*82)

22 Intentionally left blank Intentionally left blank

23 2006 Program A Self Assessment Report 90-100 (*90, *93)

24 2010 Program B Self Assessment Report 101-111 (*101, *103)

25 August 2013 Academy Self Assessment Report 112-138 (*113)

26 November 2013 Program C Self Assessment Report 139-141 (*139)

23


Appendix B: Performance Technology Frameworks

Figure 1: Mager and Pipe - Performance Analysis Flow Diagram

24


Figure 2: Harless - 13 Smart Questions

1. Do we have a problem? Yes.

2. Do we have a

performance problem?

Yes.

3. How will we know when

the problem is solved?

When accreditation is achieved using less staff hours and less interruptions

to workers regular work schedule. Other projects will be completed on time

and quality of other projects will improve.

4. What is the performance

problem?

Work is being conducted inefficiently causing other projects to missing

deadlines and/or with reduced quality.

5. Should we allocate

resources to solve it?

Yes.

6. What are the possible

causes of the problem?

Expectations, roles, feedback, are unclear. Processes and procedures are

ineffective.

7. What evidence bears on

each possibility?

Workers are complaining that their feedback is not used to mitigate

recurring problems. Other projects are being rushed or timelines are moved.

Quality of work is reduced.

8. What is the probable

cause?

Processes and procedures are unclear or nonexistent.

9. What general solution

type is indicated?

Create a process chart and assign various parts to the appropriate

stakeholders.

10. What are the alternate

subclasses of solution?

Use project management techniques to manage each accreditation project.

Regularly communicate with key stakeholders. Conduct debriefs to identify

lessons learned and best practices. Consider using SharePoint as the

document management system.

11. What are the costs,

effects, and development

times of each solution?

Costs include staff hours to create new processes, communicate processes,

and develop the skills to manage SharePoint.

Effects: The accreditation manager will be required to work with a

performance technologist to create process charts, etc. and learn how to use

SharePoint (if used).

Six months to implement new processes and learn how to use SharePoint.

12. What are the

constraints?

Willingness of the accreditation managers to accept and learn new processes

and procedures.

13. What are the overall

goals?

Develop alternative sets of interventions to maintain LETC accreditation.

25


Figure 3: Anatomy of Performance, Three Levels of Performance

26


27


Figure 4: Anatomy of Performance, Human Performance System

28


29


30


Appendix C: Lessons Learned

In hindsight, the needs assessment graduate school team members identified some things they

would have done differently during the project process. One of the lessons learned is to communicate and

develop partnerships with the CEO and all key stakeholders of the client organization throughout the

entire project. Even though this was a school project, the client and those closest to the work should feel

as if they own (or at least acknowledge) they have a performance problem. Otherwise the data collection

efforts may be seen as in invasion of their work place. It was also discovered that some of the people

closest to the work opine there is not a performance problem or an opportunity for improvement and that

they might be at risk as part of this assessment. Rossett (2009) warns of this and recommends to

“understand your sources” in order to prohibit misconceptions (p. 127). Developing a “cover story” at the

very beginning of the project would have been an effective tactic to ensuring the client was on board with

the goal of the needs assessment. The team established a cover story prior to data collection and

interviews, however, in hindsight one would have aided establishment of the relationship from the start.

Establishing a communication plan seems like an obvious thing to do, however, the project moved so

quickly that this was not a priority. Midway through the project and once the interviews were underway

with the managers, it appeared as if there was some friction between the partnership. Fostering a healthy

partnership with the client organization would increase the likelihood that the interventions would be

implemented with the ultimate goal of adding value to the organization.

31


Appendix D: Triangulation of Data

32


33


34


Appendix E: Interview Questions

1. Al has explained that BOSS may be onsite or offsite to do the actual self assessment, but as far as preparing for the internal self assessment, walk us through the process you and your team follow to prepare for it?

2. Have you always prepared the folders the way you do today?

3. What do you do with the feedback from the assessors of the internal self assessment?

4. How do you track the status of the standards and folders?

5. How do Jones and other senior leaders support your efforts?

6. What would you change about the process to make it more effective?

7. What training were you provided or did you do to prepare for the accreditation process?

8. As I said, the LETC is seen as a leader in accreditation among all agencies. What are your expectations for Durango and Jones in succeeding in this important process?

9. What is your opinion of how the process runs today?

10. How are you updated on the status of accreditation and reaccreditation of the courses?

11. What would be the consequence if for some reason the LETC did not pass accreditation or reaccreditation of a course?

12. Do Durango and Jones have the capacity to do what is needed for this process?

13. How do you support the accreditation process?