rdpc level 3 evaluation program study...rdpc level 3 evaluation program study: mgt 335, 2009 | 9 1.0...

RDPC Level 3Evaluation Program Study:MGT 335 Event Security Planningfor Public Safety Professionals, 2009

Rural Domestic Preparedness Consortium2009 RDPC Level 3 Evaluation Program StudyAward #2006-GD-T6-K001

RDPC Level 3 Evaluation Program Study: MGT 335, 2009 | 1

RDPC Level 3 Evaluation Program Study:

MGT 335 Event Security Planning for Public Safety Professionals, 2009

Submitted to U.S. Department of Homeland Security/Federal Emergency Management Agency Training & Exercise Integration Secretariat

By

The Justice and Safety Center at Eastern Kentucky University

| 2 This research was supported under award #2006‐GD‐T6‐K001 from the U.S. Department of Homeland Security/Federal Emergency Management Agency Training & Exercise Integration (DHS/FEMA TEI) Secretariat. The products, manufacturers, and organizations discussed in this publication are presented for informational purposes only and do not constitute product approval or endorsement by the U.S. Department of Homeland Security/Federal Emergency Management Agency Training & Exercise Integration Secretariat or Eastern Kentucky University.

Points of view are those of the authors and do not necessarily represent the official position of DHS/FEMA TEI or Eastern Kentucky University. This document is not intended to create, does not create, and may not be relied upon to create any rights, substantive or procedural, enforceable by any party in any matter civil or criminal.


Table of Contents List of Tables ................................................................................................................................... 4

List of Acronyms and Abbreviations ............................................................................................... 5

Executive Summary ......................................................................................................................... 6

1.0 Program Purpose ................................................................................................................ 9

1.1 Overview of MGT 335 ........................................................................................... 10

2.0 Methodology ..................................................................................................................... 13

3.0 Results ............................................................................................................................... 16

4.0 Discussion.......................................................................................................................... 20

4.1 Key Findings .......................................................................................................... 20

4.2 Limitations............................................................................................................. 21

4.3 Recommendations ................................................................................................ 22

5.0 Appendix A: Modified Version of First Invitation Letter ................................................... 26

6.0 Appendix B: Survey Instrument ........................................................................................ 28

7.0 Appendix C: Responses to Survey Questions ................................................................... 30

| 4

List of Tables Tables Table 3.1: Percentage of Respondents Who Applied the KSAs to Action Tasks ........................... 16


List of Acronyms and Abbreviations

C3 Command, Control, and Communications

DHS U.S. Department of Homeland Security

DHS/FEMA U.S. Department of Homeland Security/Federal Emergency Management Agency

DHS/FEMA TEI U.S. Department of Homeland Security/Federal Emergency Management Agency Training & Exercise Integration Secretariat

ICS Incident Command System

KSAs Knowledge, Skills, and Abilities

MGT 335 MGT 335 Event Security Planning for Public Safety Professionals

RDPC Rural Domestic Preparedness Consortium

SNS Strategic National Stockpile

SPSS® Statistical Package for the Social Sciences

TEI‐TO Training & Exercise Integration, Training Operations division

TLO Terminal Learning Objective

| 6

Executive Summary

The Rural Domestic Preparedness Consortium (RDPC) was established in 2005 by Congress to develop and deliver all‐hazards preparedness training to rural communities across America. The mission of RDPC is to coordinate the development and delivery of preparedness training in support of rural homeland security requirements and facilitate relevant information sharing. It is essential that emergency responders in small, rural, and remote communities are properly trained to deal with all‐hazards events. It is also important that the training delivered to rural emergency responders be effective in meeting its goals and objectives.

In 2009, RDPC implemented a post‐training evaluation program to evaluate the training effectiveness of its courses. The purpose of the program is to measure the transfer in behavior that has occurred in the participant due to his/her completion of the training course. In addition, the program will assess whether the knowledge, skills, and abilities (KSAs) that each participant acquires via the training course are being applied in the daily work setting of the participant.

The current study provides the results of the first evaluation conducted on a RDPC training course. The course, MGT 335 Event Security Planning for Public Safety Professionals (MGT 335), is one of several management‐level, performance‐based courses that RDPC currently offers to trainees in rural communities. MGT 335 is designed to educate small and rural community public safety personnel and local officials regarding security concerns and considerations involved with the planning of any type of event. The sample drawn for Phase 1 of the Level 3 evaluation of MGT 335 was comprised of participants who had completed the course from three separate training locations. Of the 130 participants who were invited to participate in the survey, a total of 39 surveys were completed surveys, resulting in an adjusted response rate of 32.8%. The adjusted sample size was 119 due to 11 (or 8.5%) subjects being no longer employed with their agency or having undeliverable mailing addresses.

Overall, the response from respondents was very positive. Respondents indicated that the course had helped them to better understand specific core topics of event security planning, such as the utilization of intelligence resources, the types of security information that are allowed to be disseminated to the public via the media, and the role of a security committee when planning for events. Furthermore, of those respondents who indicated that they have had an opportunity to utilize the KSAs acquired from the course, almost all of them indicated that they employed the KSAs taught in MGT 335 for various types of events, such as Strategic National Stockpile (SNS) dispensing locations, county fairs, multi‐cultural events held at local


fairgrounds, high profile court trials, execution of search warrants, local youth camps, community‐wide holiday related events, outdoor music concerts, local school related events, and a national political campaign rally.

In summary, the findings from this study suggest that the training needs assessment and curriculum development process adopted by the U.S. Department of Homeland Security/Federal Emergency Management Agency (DHS/FEMA), Training & Exercise Integration, Training Operations division (TEI‐TO) is effective at producing training that achieves the goal of increasing the knowledge, skills, and abilities of participants. Additionally, the findings suggest that training in the areas of information gathering and intelligence analysis should be held under strong consideration by RDPC for the purposes of training development. RDPC will use the information obtained from this study to refine the course curriculum and the evaluation program, as appropriate, in order to continue offering high‐quality training courses.

1.0 Program Purpose


1.0 Program Purpose

In 2009, RDPC implemented a post‐training (Level 3, from this point forward) evaluation program to evaluate the training effectiveness of its performance‐level courses. This program is based on Level 3 of Donald Kirkpatrick’s “Four Levels” of evaluating training programs — behavior.1 The purpose of the program is to measure the transfer in behavior that has occurred in the participant due to his/her completion of the training course. In addition, the program assesses whether the KSAs that each participant acquires via the training course are being applied in the daily work setting of the participant.

Kirkpatrick claims that four conditions are necessary for change to occur in a participant’s behavior once he/she has attended a training course. The first two conditions — the person must have a desire to change and the person must know what to do and how to do it — can be accomplished through a training course by “…creating a positive attitude toward the desired change and by teaching the necessary knowledge and skills.”2 Furthermore, these two conditions are contingent on the participants and their willingness to learn the training curriculum, as well as the training instructor and his/her ability to educate the participants to meet the learning objectives. The third condition — the person must work in the right climate — is outside of the training program’s (e.g., RDPC) control, as this condition pertains to the participant’s immediate supervisor. Kirkpatrick lists five different kinds of climate, which range from a supervisor intentionally preventing a participant from implementing the KSA that he/she acquired from the training course to a kind of climate in which a supervisor requires the participant’s learning transfer to the job. It is likely that participants in all RDPC training courses will work in climates more like the latter, since the participants of such courses are middle‐to‐senior management level, and the training courses are essential to participants’ job duties. The final condition — the person must be rewarded for changing — can be either intrinsic or extrinsic, according to Kirkpatrick. He explains that “intrinsic” rewards may include “the feelings of satisfaction, pride, and achievement that can occur when change in behavior has positive results,” while “extrinsic” rewards include “praise from the boss, recognition by others, and monetary rewards, such as merit pay increases and bonuses.”3 RDPC may contribute to either type of reward by simply encouraging participants throughout the training process and by providing an incentive to participants, such as continuing education units.

1 See Kirkpatrick, D. (1998). Evaluating training programs: The four levels (2nd ed.). San Francisco, CA: Berrett‐Koehler Publishers, Inc. 2 Ibid page 21. 3 Ibid page 22.

| 10

RDPC used these conditions as a framework in developing the criteria that courses must meet in the evaluation program. One, the Terminal Learning Objective (TLO) for each training module must be observable and measurable for research and training purposes. Two, a process and the needed tools are in place to be able to evaluate the transfer in behavior from the classroom to the workplace. An example of this criterion would be an evaluation survey, like the one used in the present study. Three, participants can and must use the tools that are in place to fulfill their responsibilities in the evaluation of the transfer in behavior. Four, participants will be provided with on‐the‐job opportunities to demonstrate the TLO for each module learned in the training course. Five, the benefits of the course must equate to or outweigh the costs of implementing Level 3 evaluation. Lastly, certain provisions must be in place to reinforce the performance outcomes of the course (e.g., rewards).

1.1 Overview of MGT 335

MGT 335 is a performance‐level course that is designed to educate small and rural community public safety personnel and local officials regarding security concerns and considerations involved with the planning of any type of event. Training participants are introduced to the basic principles and skills that are associated with planning security for events in small and rural communities. The course enables participants to recognize and plan adequate strategies and security measures to prevent or mitigate security incidents that are related to events. In addition, the course reinforces the importance and magnitude of security planning that is required to execute a safe and effective event, regardless of its size. There are a total of 10 modules, each having its own TLOs, in the training course. The tenth module, however, is a practical exercise for participants, so a measure of it was not included in the current study. Below is a list of the modules and their corresponding TLOs for the training course that were used to design the research study:

Module TLOs

1. Relationship Between Event Planning and Security Planning

1.1 Students will be able to outline the components of planned events and security planning, and the inter‐relationship of each.

1.2 Identify and explain the roles and responsibilities of the event security planner and the security committee’s role in the successful execution of a planned event.

2. Intelligence Planning 2.1 Students will be able to describe the basic concepts of intelligence, assess their organization’s intelligence capability, and identify resources available to prevent and monitor criminal activity.


3. Developing and Using Risk Assessments

3.1 Students will be able to identify realistic threats, vulnerabilities, and impacts facing special events and use risk assessment as a means to plan for the prevention/mitigation and management of risks.

4. Law Enforcement Activities Associated with Event Security Planning

4.1 Students will be able to differentiate between routine law enforcement activities and the special considerations required of those same activities during planning events.

4.2 Students will be able to give examples of how plans for routine law enforcement activities have to be modified to ensure safety and security during planned events.

5. Planning for Command, Control and Communications (C3)

5.1 Students will be able to explain the importance of C3 planning and give examples of the special considerations required when developing a command, control and communications plan for security during planned events.

6. Planning for Staffing Needs and Other Resources

6.1 Students will be able to assess the personnel, supplies, equipment and other resources needed to staff and enhance the security posture at planned events or planned events with associated risks.

7. Planning for Incident Management

7.1 Students will be able to illustrate the need and importance of planning incident management in support of planned events, and they will be able to identify the security considerations in the planning process for incident management.

8. Role of Public Information and Media Relations in Security Planning

8.1 Students will be able to state the role and functions of public information and media relations for planned events.

9. Developing the Security Plan and Briefing

9.1 Students will be able to write a security plan and prepare briefing materials to conduct a briefing internally to other law enforcement professionals, as well as to the overall event planner or the event committee.

2.0 Methodology


2.0 Methodology

The sample drawn for the Level 3 evaluation of MGT 335 was comprised of participants who had completed the course from three separate training locations: Raleigh, North Carolina – June 17‐18, 2008; Frederick, Maryland, October 30‐31, 2008; and Topeka, Kansas – February 2‐4, 2009. The training in Raleigh, North Carolina was held at the RBC Center, and was considered a pilot. This particular site was selected because it was in close proximity to North Carolina Central University, an academic partner of RDPC. The training in Frederick, Maryland was approved by DHS to be held at the Charles V. Main Training Center. Lastly, the training in Topeka, Kansas was held in conjunction with the Kansas Law Enforcement Training Center annual training conference. Participants from these locations were specifically selected to participate in this study because it had been at least four months since the training was completed. This time frame was determined to be an adequate amount of time for participants to have possibly utilized the KSAs acquired from the training. A total of 130 participants were invited to participate in the survey. The mailing address that participants provided at the time of enrollment in the training course was used to send invitation letters to the participants. The adjusted sample size was 119 due to 11 (or 8.5%) subjects being no longer employed with their agency or providing insufficient contact information. Overall, a total of 39 completed surveys were received, resulting in an adjusted response rate of 32.8%.

Multiple methods were implemented to collect data for the study. Respondents were initially contacted via postal mail with a letter inviting them to participate in the study (see Appendix A for a copy of the invitation letter). The mailing date for this first mailing was June 19, 2009. Approximately two weeks later on July 3, 2009, as a courtesy reminder of RDPC’s invitation to participate in the survey, postcards were mailed to all subjects who did not respond to the initial mailing. In a third attempt to solicit a response via postal mail, approximately two weeks following the postcard mailing, a second mailing of the invitation letters were made on July 16, 2009 to all subjects who had not responded to the two previous attempts. Then, for subjects who had not yet participated in the study by this point, research staff members contacted each subject individually via telephone on August 3‐5, 2009. Finally, on August 10, 2009, subjects who were not contacted directly via telephone were sent invitations to participate via the email address that they provided at the time of enrollment in the training course.

The survey instrument was an online self‐administered, self‐report survey in which research staff members were able to download the data from the software Web site. The software used to create the instrument and collect the data was the Statistical Package for the Social

Sciences (SPSS®) Dimensions’ mrInterview™ program. Respondents were provided the online

| 14

link to the survey in the invitation letters. Additionally, each subject was assigned a unique three‐digit survey code as an identifier to track his/her completion of the survey, which was also provided within the invitation letters. Subjects had to enter their survey codes in order to access the online survey.

The survey instrument included 14 single‐choice and three open‐ended questions. All of the questions directly aligned with the modules and TLOs that were listed in Section 1.1. Questions 1 and 4 aligned with Module 2, as they related to “intelligence resources.” Questions 2, 6, and 16 pertained to the development of a security plan and briefing; therefore, they aligned with Module 9. Questions 3 and 13 aligned with Module 3, which addresses the development and use of risk assessments. Question 5 related to the differentiation between standard law enforcement activities and security planning for events, so it aligned with Module 4. Questions 7 and 12 aligned with Module 5, as they pertained to the importance of command, control, and communications (C3) planning. Question 8 related to public information and the media, so it aligned with Module 8. Questions 9, 10, 11, and 14 aligned with Module 6, which addresses staffing needs and other resources. Finally, question 15 was very general in its scope, so it aligned with all nine modules. Appendix B provides the hard‐copy version of the survey instrument.

3.0 Results

| 16

3.0 Results

The data from each of the 39 surveys were analyzed using the SPSS® 16.0. Statistical analyses to include frequencies and percentages were conducted to analyze the data. It was determined that these methods of univariate analysis were the most appropriate given the research objective. Appendix C provides the responses to each question on the survey.

With regard to the single‐response questions, subjects were asked whether the training course helped them to better understand specific core topics, such as the utilization of intelligence resources as they relate to event security planning (Question 1), the types of security information that are allowed to be disseminated to the public via the media (Question 8), and the role of a security committee when planning for events (Question 13). For the first topic, the utilization of intelligence resources, all subjects (n=39) indicated that the course had helped in this area. A majority of the respondents (89.7%, n=35) indicated that the course had helped them to better understand the dissemination of public information via the media. Lastly, practically all of the respondents (97.4%, n=38) reported that the course had helped to better understand the role of a security committee. Since the subjects’ behavior and actions were being evaluated, several questions asked respondents whether they had had an opportunity to utilize the KSAs acquired from the training course. Of those respondents who indicated that they had had an opportunity, nearly all of them indicated that they had applied the KSAs acquired from MGT 335. As Table 3.1 displays, respondents applied their KSAs across many action tasks that are related to event security planning.

Table 3.1: Percentage of Respondents Who Applied the KSAs to Action Tasks

Action Task % (n)

Identifying Ways to Maximize Department Capabilities (Question 14) 100.0 (32)

Identifying External Resource Requirements (Question 10) 100.0 (29)

Employing the Four Types of Intelligence Resources (Question 4) 100.0 (25)

Developing/Implementing a Security Plan (Question 2) 100.0 (23)

Conducting Analysis of a Security Environment (Question 3) 100.0 (23)


Action Task % (n)

Presenting an Event Security Briefing (Question 6) 100.0 (19)

Identifying Internal Resource Requirements (Question 9) 96.7 (29)

Applying the ICS Structure and Concepts (Question 7) 96.2 (25)

Employing Risk Assessments (Question 5) 95.2 (20)

Developing Effective Commands and Control Plans (Question 12) 92.3 (24)

Preparing Formal Agreements among Organizations and/or Jurisdictions (Question 11) 90.0 (9)

As noted above, respondents were asked to answer three open‐ended questions on the survey. The first open‐ended question was an elaboration to the single‐response on the types of intelligence resources that subjects have employed since taking the course (Question 4). Specifically, subjects were asked the types of resources that they had employed and how those resources were employed. Approximately 92 percent (n=23) of the respondents who indicated that they had employed one or more of the types of intelligence resources provided responses about the types of resources used and how those resources were used. Both “Open Source” and “Internal” resources were employed on 15 separate occasions, which were the resources most often used among the sample. Respondents reported that they had employed “External” resources approximately 11 times, and they had used “Covert” resources nine times. The techniques that were employed by respondents in use of the intelligence resources are provided in Appendix C.

Respondents were also asked to explain how they and their departments had employed event security planning tactics as relayed in the training course (Question 15). Over two‐thirds (69.2%, n=27) of the sample provided a usable response to this question. Respondents identified a variety of events for which they had employed the KSAs learned in the course. As a final question, due to what they learned in the training course, respondents were asked to explain whether they had been responsible for outlining the key components of event security planning and/or developing security plans for events supported by their departments (Question 16). Of the 37 subjects who responded to this question, over half (51.4%) stated that they had assumed such a role with their departments, while just slightly under half (45.9%) stated that they had not. Finally, less than three percent (2.7%) indicated that such a role was not

| 18

applicable to their job duties. Appendix C provides the detailed responses of these final two questions.

4.0 Discussion

| 20

4.0 Discussion

The objective of this study was to determine how, if at all, trainees who successfully completed MGT 335 utilized the KSAs acquired from the course for the purposes of event security planning. This objective was adequately met by the findings of the study, which will be discussed in further detail in this section. The limitations of this study, particularly with regard to the methodology, are also outlined below. Lastly, the findings of this study allow RDPC to refine, where appropriate, MGT 335 course curriculum and future training evaluations and to develop training courses.

4.1 Key Findings

Overwhelmingly, respondents indicated the KSAs that they acquired from MGT 335 were beneficial to them in planning security for a variety of events. Of particular interest was the indication among respondents of the use of intelligence resources. Respondents reported that when faced with real‐world situations in their jurisdictions, they employed one or more of the four types of intelligence resources: Open Source, Internal, Covert, and/or External. In explaining how he employed the resources, one subject stated:

In developing the plan for securing the National Strategic Stockpile [SNS] and Dispensing locations, we worked extensively with the Health Department, Emergency Management, and KHP [Kansas Highway Patrol] in coordinating activities. Utilizing the four types of Intelligence Resources, we were able to develop a plan that has been tested in a scenario.

This finding is important, because it provided training curriculum staff with insight into how intelligence resources are utilized by local public safety personnel in rural communities during event security planning. This component is usually reserved for use by federal authorities at the regional fusion centers, who then disseminate the intelligence to the local public safety officials. The results of this study suggest, however, that if adequately trained and provided appropriate resources, local public safety personnel in rural communities may also utilize intelligence resources as well.

The relevance of this finding also cuts across unmet training needs of rural emergency responders. In the two National Training Needs Surveys of rural emergency responders that RDPC has conducted, responders, particularly law enforcement officers, have indicated a strong need for training in the areas of collecting information and intelligence analysis in order to


prevent the occurrence of emergencies and unforeseen hazards. As the results of the current study indicate, these areas may also be applicable to event security planning. Overall, it seems that rural emergency responders utilize intelligence resources on a continuous basis in order to be effectively prepared for any type of emergency and/or event.

Another interesting finding in the current study was the many ways and the variety of events in which respondents utilized the KSAs acquired from the training course. Not only did respondents seem to apply their acquired KSAs in the security preparation of events, respondents also reported using them in the daily operation of their departments’ activities. For instance, one subject reported that his department developed better coordination and use of resources:

Better coordination of Department resources and non‐Department resources and identification of sector responsibilities. Having officers use the highly reflective traffic vests at all times to give the illusion that more officers are present and make them more identifiable to each other. Using elevated platforms that officers can stand on giving patrons an unobstructed view of the officers and vice versa.

In addition, respondents did not seem to be limited in the types of events in which they could employ the tactics learned in the course. These events included SNS dispensing locations, county fairs, multi‐cultural events held at local fairgrounds, high profile court trials, execution of search warrants, local youth camps, community‐wide holiday related events, outdoor music concerts, local school related events, and a national political campaign rally.

4.2 Limitations

The limitations of this study are noteworthy, particularly in regard to the refinement of course curriculum and future evaluations. Being the initial study for RDPC’s Level 3 evaluation program, it is not surprising to find the limitations to be related to the methodology. Although the adjusted response rate (32.8%) was adequate, it was lower than what RDPC intends to reach in future evaluation studies. There are several factors, however, that contributed to this response rate. First, trainees were not informed of the evaluation study at the conclusion of the training course. This allowed for some ambiguity about the study on behalf of the trainees. For future evaluation studies, RDPC will ensure that trainees are informed at the conclusion of the training course that they can expect to receive an invitation to participate in the study via postal mail or email. Furthermore, trainees should be informed that they can expect to receive the invitations approximately 4‐6 months from when the training course was attended. Notifying trainees such as this should be an essential component of the evaluation process.

| 22

Second, the survey instrument was only made available in an online format, which may have limited the number of responses that were received from subjects. The justification for using an online format only was due to the belief that all respondents had Internet‐access, since all of them provided an email address at the time of registering for the training course. What could not be determined, however, was the ability level of respondents in using a computer to access the online survey. In fact, numerous respondents contacted the RDPC survey administrator during the data collection because they did not know how to access the web link to the survey. Furthermore, many potential respondents may have completely dismissed the survey once they realized they had to access it via online. Some respondents may have been more comfortable in completing a hard copy of the survey and returning it via postal mail. Any future evaluation study will use a mixed‐modal approach: survey available in both online and mail‐in formats.

Finally, the contact information that was used to communicate with respondents while administering the survey was incorrect at times. The telephone numbers and email addresses that some respondents provided at the time of registering for the training course were incorrect. RDPC research staff members were able to retrieve the correct contact information on some respondents via the Internet, since such information was publicly accessible. Additionally, some respondents were no longer employed with the agency with whom they were employed at the time of completing the training course. This was discovered when RDPC research staff members conducted follow‐up telephone calls to respondents. Overall, the total number of respondents who had incorrect contact information or were no longer employed with their agency was 11, or 8.5 percent of the total surveys mailed. This finding indicates a need for RDPC to develop an automated system that collects contact information regularly from respondents of training courses that may undergo an evaluation.

4.3 Recommendations

Overall, the findings of the Level 3 evaluation of MGT 335 suggest that the training needs assessment and curriculum development process adopted by the U.S. Department of Homeland Security/Federal Emergency Management Agency (DHS/FEMA), Training & Exercise Integration, Training Operations division (TEI‐TO) is effective at producing training that achieves the goal of increasing the KSAs of participants. This process is structured on the Planning, Analysis, Design, Development, Implementation, and Evaluation (PADDIE) model, which is set forth on the TEI‐TO Responder Training Development Center.1 The PADDIE model is instrumental in establishing common standards and processes for the development of training. Moreover, the model provides a step‐by‐step system for the analysis of students’ needs, the

1 For more information about the PADDIE Model, see http://portal.vertexsolutions.com/RTDC_1.8/rtdc_state/.

http://portal.vertexsolutions.com/RTDC_1.8/rtdc_state/


design and development of training materials, and the evaluation of the effectiveness of the training intervention. Adoption of this model has greatly assisted RDPC with promoting best practices and establishing a consistent course development and evaluation methodology.

With regard to the development of new training courses, the findings of the current study suggest that training in the areas of information gathering and intelligence analysis should be held under strong consideration by RDPC. As mentioned in Section 4.1, in both of RDPC’s National Training Needs Surveys, respondents in the law enforcement discipline have indicated unmet training needs in these areas. RDPC identifies training needs based on the Target Capabilities List (TCL), which is central to the mission and strategic goals of DHS. The TCL identifies and defines capabilities that communities may need to achieve and sustain, depending on relevant risks and threats, in order to prevent, protect against, respond to, and recover from major events. It gives communities the framework that allows them to define areas in which they may need training to ensure that they are prepared for all‐hazards events. In the 2006 survey, law enforcement officers identified “Information Gathering and Recognition of Indicators and Warnings” and “Intelligence/Information Sharing and Dissemination” as areas in which training was needed by agency personnel. The former entails the gathering, consolidation, and retention of raw data and information, and recognizing indicators and warnings in this gathered data of potential threats to homeland security; the latter capability directly concerns the collection of data in efforts to detect threats to homeland security. In the 2009 survey, law enforcement officers identified “Intelligence Analysis and Production” as a training need, which directly concerns the analysis of intelligence in efforts to detect threats to homeland security.

A component of the MGT 335 training is a specialized leave‐behind tool which assists law enforcement in preparing a basic event security plan that can be modified to fit their needs. This computer‐based tool was reviewed and approved within the TEI‐TO course development process. The intent has been to distribute the software to all students participating in the training, but due to required modifications to the content, the tool was not yet available to students participating in the classes selected for this survey. In future Level 3 evaluations, it is recommended that questions be added to the survey instrument to capture data on the relevance, effectiveness, and usefulness of the tool, in addition to the curriculum.

In summary, it seems that rural law enforcement officers have not only indicated a need for training in these areas, but when they have been provided with related training, as indicative of the current study, they have utilized the KSAs they acquired from the training in preparation for homeland security‐related events. RDPC will use the information obtained from this study to

| 24

refine the course curriculum and the evaluation program, as appropriate, in order to continue offering high‐quality training courses.

5.0 Appendix A: Modified Version of First Invitation Letter

| 26

5.0 Appendix A: Modified Version of First Invitation Letter DATE Dear SURVEY RESPONDENT: You have been selected to participate in an evaluation research study regarding your completion of the course MGT 335 Event Security Planning for Public Safety Professionals, a training sponsored by the Rural Domestic Preparedness Consortium (RDPC). This study is being conducted by the Justice and Safety Center at Eastern Kentucky University and is funded by the U.S. Department of Homeland Security/Federal Emergency Management Agency Training & Exercises Integration Secretariat. In this study, researchers are assessing the transfer of knowledge from the classroom to the job in order to determine the success of the training, as well as to help guide the development and delivery of future training. It is important to emphasize that this survey is NOT A TEST; therefore, there are no right or wrong answers. Please understand that your participation in this study is completely voluntary. There will not be any consequences for refusal to participate, nor will we identify those who refuse to participate. Your willingness to participate, however, will result in highly beneficial information for RDPC. It is important that your unique perspective is represented, so we ask that you not allow another individual to complete the survey in your place. This project was reviewed and approved by Eastern Kentucky University’s Institutional Review Board (IRB) for the Protection of Human Research Subjects. Questions regarding your rights as a participant in this research may be addressed to: IRB Chairperson, Sponsored Programs, Jones 414/Coates CPO 20, 521 Lancaster Avenue, Richmond, KY 40475‐3102. You may access the survey online at the following link: http://surveys.jsc.eku.edu/MGT335Evaluation.html. Once you access the survey, you will be required to enter a three‐digit survey code. Your survey code is XXX. The survey will take approximately 20 minutes to complete. Please complete your survey by July 17, 2009. If you have any questions, please contact Nathan Lowe at (859) 622‐6763 or [email protected]. Thank you for your willingness to share your experiences with us. We appreciate your participation. Sincerely,

Dr. Pam Collins Principal Investigator – RDPC / Executive Director – JSC‐EKU

http://surveys.jsc.eku.edu/MGT335Evaluation.html

mailto:[email protected]

6.0 Appendix B: Survey Instrument

| 28

6.0 Appendix B: Survey Instrument

Rural Domestic Preparedness Consortium MGT 335 Event Security Planning for Public Safety Professionals

2009 Evaluation Research Study Instructions: Please answer the questions based on your experiences since taking MGT 335 Event Security Planning for Public Safety Professionals. If you encounter any problems while taking the survey, or have any questions or comments in general, please contact Nathan Lowe at (859) 622‐6763 or [email protected]. Thank you in advance for your participation. 1. Did the course help you to better understand the utilization of intelligence resources as they

relate to event security planning? Yes No

2. Since taking the course, have you had an opportunity to develop or implement a security plan based on an analysis of an event’s security requirements?

Yes No (If “no,” please skip to Question #3)

a. If "yes," did you use what you learned in this course when you developed or implemented the security plan?

Yes No

3. Since taking the course, have you conducted an analysis of a security environment for a planned event?


a. If "yes," did you use what you learned in this course when you conducted the analysis?

Yes No

4. Since taking this course, have you employed any of the four types of intelligence resources — Open Source, Internal, Covert, and/or External?


a. If "yes," did you use what you learned in this course when you employed the four types of intelligence resources?

Yes No

mailto:[email protected]


b. What types did you employ and how were they employed? Please explain. __________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

5. Since taking this course, have you employed risk assessment that anticipates the potential danger associated with an event?


a. If "yes," did you use what you learned in this course when you employed risk assessment with an event?

Yes No

6. Since taking this course, have you presented an event security briefing? Yes No (If “no,” please skip to Question #7)

a. If "yes," did you use what you learned in this course when you presented the briefing?

Yes No

7. Since taking this course, have you had an opportunity to apply the ICS structure and concepts when planning security for events?


a. If "yes," did you use what you learned in this course when you applied the ICS structure and concepts to the planning of security for events?

Yes No

8. Did the course provide you with a better understanding of the types of security information that are allowed to be disseminated to the public via the media?

Yes No

| 30

9. Since taking this course, have you had an opportunity to identify INTERNAL resource requirements for planned events?


a. If "yes," did you use what you learned in this course when you identified INTERNAL resource requirements for the planned events?

Yes No

10. Since taking this course, have you had an opportunity to identify EXTERNAL resource requirements for planned events? Yes No (If “no,” please skip to Question #11)

a. If "yes," did you use what you learned in this course when you identified EXTERNAL resource requirements for the planned events?

Yes No

11. Since taking this course, have you prepared a formal agreement among organizations and/or jurisdictions that facilitates the sharing of resources necessary to conduct security during an event? Yes No (If “no,” please skip to Question #12)

a. If "yes," did you use what you learned in this course when you prepared the agreement?

Yes No

12. Since taking this course, have you developed effective command and control plans for planned events? Yes No (If “no,” please skip to Question #13)

a. If "yes," did you use what you learned in this course when you developed the plans?

Yes No

13. Since taking this course, do you now have a better understanding regarding the role of a security committee when planning for events? Yes No

14. Since taking this course, have you identified ways to maximize department capabilities when planning security for major events? Yes No (If “no,” please skip to Question #15)


a. If "yes," did you use what you learned in this course when you identified ways to maximize department capabilities to the planning of security for major events?

Yes No

15. Please explain how you and your department have employed Event Security Planning tactics as relayed in MGT 335 Event Security Planning for Public Safety Professionals. _________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

16. Because of what you learned in MGT 335 Event Security Planning for Public Safety Professionals, have you been responsible for outlining the key components of event security planning and/or developing security plans for events supported by your department? If so, please explain. _________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

7.0 Appendix C: Responses to Survey Questions

| 30

7.0 Appendix C: Responses to Survey Questions

1. Did the course help you to better understand the utilization of intelligence resources as they relate to event security planning? (n = 39)

Frequency Percent

Yes 39 100.0%

No 0 0.0% 2. Since taking the course, have you had an opportunity to develop or implement a security

plan based on an analysis of an event’s security requirements? (n = 39)

Frequency Percent

Yes 23 59.0%

No 16 41.0%

a. If “yes,” did you use what you learned in this course when you developed or implemented the security plan? (n = 23)

Frequency Percent

Yes 23 100.0%

No 0 0.0% 3. Since taking the course, have you conducted an analysis of a security environment for a

planned event? (n = 39)

Frequency Percent

Yes 23 59.0%

No 16 41.0%

a. If “yes,” did you use what you learned in this course when you conducted the analysis? (n = 23)

Frequency Percent

Yes 23 100.0%

No 0 0.0%


4. Since taking this course, have you employed any of the four types of intelligence resources – Open Source, Internal, Covert, and/or External? (n = 39)

Frequency Percent

Yes 25 64.1%

No 14 35.9%

a. If “yes,” did you use what you learned in this course when you employed the four types of intelligence resources? (n = 25)

Frequency Percent

Yes 25 100.0%

No 0 0.0%

b. What types did you employ and how were they employed? Please explain. • “All four sources were used. Triple homicide suspect warrant service.

Intelligence provided by Missouri detectives, local LEO, family members, and covert surveillance.”

• “All four. Please note, using these was not a result of the course, but rather SOP for our agency. Still in process of using them due to yet another high‐profile trial getting ready to start.”

• “Covert, plain clothes during rock concert.”

• “I used all four types for a large scale event I planned for Halloween. The information was very helpful.”

• “Intelligence from our Intelligence section – information provided to me from our internal sources mostly – these would be from a combination of the four sources.”

• “Internal.”

• “Internal – officers assigned to positions. External – security personnel from the event venue.”

• “Internal and external in preparation for the July 4th celebration in a local park.”

• “Internal External.”

• “Internal, External, & Covert mainly for a large 50‐75K 4th of July event. Mainly to practice what was learned but very beneficial as a first time use.”

• “Internet sources looking at chat rooms etc. to determine if certain groups were targeting the event and if so in what manner.”

• “Open Source and internal. Used in planning event where the threat of Animal Activist was a concern. Used their web sites as well as industry web sites to check for information.”

| 32

• “Open source was used to determine what types of people would be attending rock concert; internal sources were used to assist LEO in monitoring and detecting criminal behaviors.”

• “Open source, Internet information.”

• “Open Source: News media and posters to get times, locations, predicted weather conditions and dates of the events. Internal: to ascertain any local problems that were foreseen surrounding the events, such as possible disruptive influences.”

• “Planning for dignitary visit.”

• “Used Internet sites to review past events with similar performers or content. Fusion Center information concerning previous developed intelligence with respect to certain people or groups.”

• “We complete internal and covert surveillance for a dignitary visiting our campus.”

• “We conducted an open source and covert intelligence gathering program for a closed event that was supposed to take place in our jurisdiction.”

• “We recently set up procedures for assisting the Health Dept. with locating, assessing and establishing Warehouse and Dispensing Sites for National Strategic Stockpile. We did a scenario setting up and securing dispensing sites. In this process, Deputy [name omitted for purposes of anonymity] utilized all four of the intelligence gathering types in establishing the procedures for securing the facilities.”

• “We relied on open source, internal, and external intelligence. Our area and events are small (2000‐3000 people). Still we felt better about our events in regard to security risks.”

• “We used Open Source and External in reviewing the security requirements for a local event.”

• “Yes, I use open source/internet resources to conduct research into events that were held at other locations and view Best Practice information to include in my planning.”

5. Since taking this course, have you employed risk assessment that anticipates the potential danger associated with an event? (n = 39)

Frequency Percent

Yes 21 53.8%

No 18 46.2%


a. If “yes,” did you use what you learned in this course when you employed risk assessment with an event? (n = 21)

Frequency Percent

Yes 20 95.2%

No 1 4.8%

6. Since taking this course, have you presented an event security briefing? (n = 39)

Frequency Percent

Yes 19 48.7%

No 20 51.3%

a. If “yes,” did you use what you learned in this course when you presented the briefing? (n = 19)

Frequency Percent

Yes 19 100.0%

No 0 0.0% 7. Since taking this course, have you had an opportunity to apply the ICS structure and

concepts when planning security for events? (n = 39)

Frequency Percent

Yes 26 66.7%

No 13 33.3%

a. If “yes,” did you use what you learned in this course when you applied the ICS structure and concepts to the planning of security for events? (n = 26)

Frequency Percent

Yes 25 96.2%

No 1 3.8%

8. Did the course provide you with a better understanding of the types of security information that are allowed to be disseminated to the public via the media? (n = 39)

Frequency Percent

Yes 35 89.7%

No 4 10.3%

| 34

9. Since taking this course, have you had an opportunity to identify INTERNAL resource requirements for planned events? (n = 39)

Frequency Percent

Yes 30 76.9%

No 9 23.1%

a. If “yes,” did you use what you learned in this course when you identified INTERNAL resource requirements for the planned events? (n = 30)

Frequency Percent

Yes 29 96.7%

No 1 3.3% 10. Since taking this course, have you had an opportunity to identify EXTERNAL resource

requirements for planned events? (n = 39)

Frequency Percent

Yes 29 74.4%

No 10 25.6%

a. If “yes,” did you use what you learned in this course when you identified EXTERNAL resource requirements for the planned events? (n = 29)

Frequency Percent

Yes 29 100.0%

No 0 0.0% 11. Since taking this course, have you prepared a formal agreement among organizations

and/or jurisdictions that facilitates the sharing of resources necessary to conduct security during an event? (n = 39)

Frequency Percent

Yes 10 25.6%

No 29 74.4%


a. If “yes,” did you use what you learned in this course when you prepared the agreement? (n = 10)

Frequency Percent

Yes 9 90.0%

No 1 10.0%

12. Since taking this course, have you developed effective command and control plans for planned events? (n = 39)

Frequency Percent

Yes 26 66.7%

No 13 33.3%

a. If “yes,” did you use what you learned in this course when you developed the plans? (n = 26)

Frequency Percent

Yes 24 92.3%

No 2 7.7% 13. Since taking this course, do you now have a better understanding regarding the role of a

security committee when planning for events? (n = 39)

Frequency Percent

Yes 38 97.4%

No 1 2.6% 14. Since taking this course, have you identified ways to maximize department capabilities

when planning security for major events? (n = 39)

Frequency Percent

Yes 32 82.1%

No 7 17.9%

| 36

a. If “yes,” did you use what you learned in this course when you identified ways to maximize department capabilities to the planning of security for major events? (n = 31)

Frequency Percent

Yes 32 100.0%

No 0 0.0% 15. Please explain how you and your department have employed Event Security Planning

tactics as relayed in “MGT 335: Event Security Planning for Public Safety Professionals.” • “Better coordination of Department resources and non‐Department resources and

identification of sector responsibilities. Having officers use the highly reflective traffic vests at all times to give the illusion that more officers are present and make them more identifiable to each other. Using elevated platforms that officers can stand on giving patrons an unobstructed view of the officers and vice versa.”

• “Our department uses the planning tactics on all events sponsored by the city.”

• “I would like to say that I was able to plan a major event but I have recurring events that I was able to review what I have done in the past and make minor corrections. I am looking forward to preparing a new event that will tax me/my agency and force me to make fuller use of what I was taught.”

• “In developing the plan for securing the National Strategic Stockpile and Dispensing locations, we worked extensively with the Health Dept, Emergency Management, and KHP in coordinating activities. Utilizing the 4 types of Intelligence Resources, we were able to develop a plan that has been tested in a scenario.”

• “Local county fair security planning, and search warrant activities.”

• “Most recently with high profile trials. We also conduct a camp operation every year for about 150 youth for a week’s time. We are using the ICS model.”

• “My participation has been from a tactical preparation, as well as a training level.”

• “Numerous events planned for over the past year.”

• “Once again, while our local events are small and the identified risk levels low I feel much better about how we approach our events. In our upcoming 4th of July Fireworks (2009) we were able to determine the probability of an increase in crowd size over past tears due to several area communities not having any this year due to budget constraints. This could potentially impact our event and we could have been caught off guard had we not had a process in place for security planning. Time will tell how it all plays out as our event is scheduled for July 3rd (next Friday).”

• “Participated in a multi‐agency team to plan for multiple local events, such as the fair, marathon, July 4th, etc.”

• “Placement of personnel.”

• “Reassignment of officers to better fill needed positions.”


• “Security deployment of available resources.”

• “Though our event is yearly it keeps growing many new partnerships were forged and that there were issues all were worked thru and in the long run the whole event went smoother than in the past.”

• “We coordinate better internally and in a more organized manner.”

• “We had a high profile court trial that brought in media and spectators for which I used what was learned in MGT 335 Event Security planning.”

• “We have a large fairground in our District. They host Islamic, Chinese, Hispanic and Indian ethnic events each summer. Strategic planning of our resources in the event of an emergency has been most helpful.”

• “We have developed minimum staffing levels in every unit in the organization. We then took the information learned from that exercise to develop operational plans that can be used to handle large scale planned and unplanned events. We already have MOU's, MOA"s and mutual aide agreements in place.”

• “We have had a number of events that included internal VIP's as well as external VIP's. Used the Event Security Planning training to formulate effective security planning as well as effective use of internal and external resources.”

• “We have included representatives of other lead and support agencies in our planning.”

• “We have used the information to all aspects of planning for large scales events. I have even passed this information along to other departments within my jurisdiction to also use.”

• “We oversee police operations at an outdoor concert venue. Since taking this course I was able to view each concert as a separate incident and carefully analyze the various risks.”

• “We planned the security for the upcoming County Fair including site security while event is taking place, overnight security when the event is not open to the public, traffic control, what to do in case of inclement weather such as a tornado or even possibly flooding in the area, special event control such as the Fair parade and positioning of units for most effective response.”

• “We use the Event Security Planning to Develop plans for large scale events, large school events, and public gatherings of size. It gives our supervisors an idea of how to organize their resources and deploy assets.”

• “We used a basic outline of the course to help prepare for an event.”

• “We used your basic outline to improve the way we had been completing security plans.”

• “We utilized ESP for our annual 4th of July celebration in conjunction with multiple other agencies.”

| 38

• “While serving with the NCR‐IMT during several deployments I was able to use some of the information learned during the course. While not ‘planned events’ per say there were issues that I was able to deal with that I learned in the course.”

16. Because of what you learned in “MGT 335: Event Security Planning for Public Safety

Professionals,” have you been responsible for outlining the key components of event security planning and/or developing security plans for events supported by your department? If so, please explain.

• “A team has been formed to develop a County Special Events Ordinance.”

• “Contributed info to assist the Mgt. in a safety plan.”

• “I am very happy to say we have started and now have a much better rapport between agencies. Though there are still some minor issues, they are much easier to work through.”

• “I bring a more formalized approach to event planning to the table since taking the training. Others are allowing me to take the lead.”

• “I plan all small and large scale events like Halloween, Festifall, and NCAA celebrations. All this information has been very helpful.”

• “Mass dispensing medications and threat assessments of facilities.”

• “On a limited basis with our planning chief.”

• “Local co fair security planning, and search warrant services.”

• “This was part of my job assignment prior to taking the course. It allowed me some additional approaches to consider in the development of strategies.”

• “We have multiple events requiring advanced planning and we use many of the features offered by MGT 335.”

• “We have planned for several dignitary visits and have used the Basic Event Planning guide.”

• “Yes and no. I use the process I learned but as Chief I was responsible for this function either way. I will tell you however I feel much more comfortable using this process and feeling assured we have done our best to completely identify, assess and address the risks involved in our events.”

• “Yes but most of these occur yearly ‐ Fireworks, Taste of Wheaton, World of Montgomery County, etc, that are down pat. I used the information to review how I prepared for these and was able to see that I could make corrections on several areas but there was nothing that required amazing changes. I look forward to a new event to test my new‐taught abilities.”

• “Yes I have assisted the Sheriff and Undersheriff in developing a plan for the County Fair this year.”

• “Yes, I am involved with a Special Events Response Unit that prepares for large scale events. Planning is seen as a team approach.”


• “In developing the plan for securing the National Strategic Stockpile and Dispensing locations, we worked extensively with the Health Dept, Emergency Management, and KHP in coordinating activities. Utilizing the 4 types of Intelligence Resources, we were able to develop a plan that has been tested in a scenario.”

• “Yes, the concert venue and other special events we handle throughout the year. The weekend after the event we planned for then senator Obama's visit our county the night before the election. One of the largest events in the county history.”

• “Yes. I am responsible for event planning to include the development of operational plans before the event and completing an after‐action report afterwards. Included in the plan are internal and external security, traffic control, crowd management, fire and EMS control, communications, Incident Command components, etc.”

rdpc level 3 evaluation program study...rdpc level 3 evaluation program study: mgt 335, 2009 | 9 1.0...

Documents