level 3 course evaluation program awr 147 rail …...level 3 course evaluation program awr 147 rail...

Level 3 Course Evaluation Program AWR 147 Rail Car Incident Response

Prepare For The Worst, Train To Be The Best

This report was supported under award #2008-GD-T8-K015 from the U.S. Department of Homeland Security, Federal Emergency Management Agency, National Training and Education Division. Points of view or opinions in this report are those of the authors and do not represent the official position or policies of the U.S. Department of Homeland Security.

Research Team

Brian Simpkins Associate Director - Research and Evaluation [email protected]

Erin Henry Research Associate [email protected]

Suggested Citation Simpkins, B., and Henry, E. (2013). RDPC Level 3 Course Evaluation Program: AWR 147 Rail Car Incident Response. Richmond, KY: Rural Domestic Preparedness Consortium.

1.0 Introduction and Program Purpose ............................................................................................................................................1

2.0 Course Overview ................................................................................................................................................................................2

3.0 Methodology .......................................................................................................................................................................................3

4.0 Results ...................................................................................................................................................................................................4

5.0 Discussion ............................................................................................................................................................................................7

6.0 Conclusion ...........................................................................................................................................................................................8

7.0 Appendix A: Modified Version of First Invitation Letter ......................................................................................................9

8.0 Appendix B: Survey Instrument ...................................................................................................................................................11

Figures

Figure 2-1: Modules and Corresponding TLOs ..........................................................................................................................................................3

Figure 4-1: Question 1 Results ..........................................................................................................................................................................................5









Figure 4-10: Question 9a Results .....................................................................................................................................................................................6

Figure 4-11: Question 10 Results .....................................................................................................................................................................................6




Figure 4-15: Question 13a Results ..................................................................................................................................................................................6



Figure 4-18: Actions Taken Subsequent to AWR 147 Completion ....................................................................................................................6

Table of Contents

This page intentionally left blank

Executive Summary

In 2010, the Rural Domestic Preparedness Consortium (RDPC) implemented a post-training evaluation program to evaluate the effectiveness of its courses. This report details the results of the course evaluation for AWR 147 Rail Car Incident Response, which is one of several awareness-level courses that RDPC currently offers to rural communities. AWR 147 Rail Car Incident Response course was developed to educate emergency responders on freight rail car incidents involving hazardous materials.

The sample drawn for the evaluation consisted of participants who had completed AWR 147 between February 2009 and December 2010. The adjusted sample size was 1,876 and a total of 390 completed surveys were received, resulting in an adjusted response rate of 20.7%.

Overall, the data indicates that the course helped the majority of the respondents to better understand the response elements of rail car incidents, including:

• Importance of U.S. railroad freight industry (96%);

• Relevant federal laws, regulations, and industry associations (95%);

• Recognition of rail car designs, construction, tank car fittings, and accessories (98%);

• Ability to read and understand CONSIST shipping papers (87%); and

• Understanding of mitigation techniques and strategies (94%).

Despite the low numbers in terms of the overall opportunities for respondents to implement knowledge, skills, and abilities(KSAs) acquired via AWR 147, a vast majority of the respondents indicated that they did use their acquired KSAs in the opportunities they were offered. This indicates that one of the main goals of this course, which is to prepare the participants to manage and/or respond to a rail car incident without endangering the health and safety of the responders, has been met.

The RDPC will use the information obtained from this study to refine the AWR 147 course curriculum during its upcoming three-year review process in order to ensure the RDPC continues to offer high-quality training courses.

AWR 147 Level 3 Course Evaluation I 1

1.0 Introduction and Program Purpose

The Rural Domestic Preparedness Consortium (RDPC) was established in 2005 by Congress to develop and deliver all-hazards preparedness training to rural communities across America. The mission of RDPC is to coordinate the develop-ment and delivery of preparedness training in support of rural homeland security requirements and facilitate relevant information sharing. It is essential that emergency responders in small, rural, and remote communities are properly trained to deal with all-hazards events. It is also important that the training delivered to rural emergency responders be effective in meeting its goals and objectives. For more information on the RDPC, please visit http://www.ruraltraining.org/.

In 2010, the RDPC established a Level 3 Course Evaluation Program to evaluate the training effectiveness of its courses. This program is based on Level 3 of Donald Kirkpatrick’s Four Levels of evaluating training programs — behavior.¹ The purpose of the program is to measure the transfer in behavior that has occurred in the participant due to his/her comple-tion of the training course. Therefore, the program assesses whether the knowledge, skills, and abilities (KSAs) that each participant acquires via the training course are being applied in the daily work setting of the participant.

Kirkpatrick claims that four conditions are necessary for change to occur in a participant’s behavior once he/she has attended a training course. The first two conditions — the person must have a desire to change and the person must know what to do and how to do it — can be accomplished through a training course by “…creating a positive attitude toward the desired change and by teaching the necessary knowledge and skills.” Furthermore, these two conditions are bestowed upon on the participants and their willingness to learn the train-ing curriculum, as well as the training instructor and his/her ability to educate the participants to meet the learning objec-tives. The third condition — the person must work in the right climate— is outside of the training program’s (e.g., RDPC) con-trol, as this condition pertains to the participant’s immediate supervisor or work environment as a whole. Kirkpatrick lists five different kinds of climate, which range from a supervisor intentionally preventing a participant from implementing the KSA that he/she acquired from the training course to a kind of climate in which a supervisor requires the participant’s learning transfer courses. It is likely that participants in RDPC training courses will work in climates more like the latter, since the participants of such courses are middle-to-senior management level and the training courses are essential to participants’ job duties. The final condition — the person must be rewarded for changing — can be either intrinsic or extrin-sic, according to Kirkpatrick. Kirkpatrick explains that intrin-sic rewards may include the feelings of satisfaction, pride, and achievement that can occur when change in behavior has positive results, while extrinsic rewards include praise from the boss, recognition by others, and monetary rewards, such as merit pay increases and bonuses. The RDPC may contribute to either type of rewards by simply encouraging participants throughout the training process and by providing an incentive to participants, such as continuing education units.

The RDPC used these conditions as a framework in develop-ing the four criteria that courses must meet in the Level 3 Course Evaluation Program.

1. The Terminal Learning Objective (TLO) for each train-ing module must be observable and measurable for research and training purposes.

2. A process and the needed tools must be in place to be able to evaluate the transfer in behavior from the classroom to the workplace (e.g., RDPC’s Level 3 Course Evaluation Program).

3. Participants can and must use the tools that are in place to fulfill their responsibilities in the evaluation of the transfer in behavior.

4. Participants must be provided with on-the job op-portunities to demonstrate the TLO for each module learned in the training course.

Based on the criteria above, AWR 147 Rail Car Incident Response was selected as a suitable candidate for evaluation. Further, the evaluation was conducted in parallel with the required U.S. Department of Homeland Security (DHS) three-year course review and update of AWR 147. The remainder of this report details the results of the course evaluation.

¹ See Kirkpatrick, D. (1998). Evaluating training programs: The four levels (2nd ed.). San Francisco, CA: Berrett-Koehler Publishers, Inc.

2 I AWR 147 Level 3 Course Evaluation

2.0 Course Overview

Railroads remain the backbone of America’s freight trans-portation network and are vital to the economic health of the United States. America’s freight railroads today serve nearly every industrial, wholesale, retail, and resource-based sector of the economy, operating over a network of nearly 140,000 miles. Railroads carry 43 percent of intercity domestic freight – more than any other mode of transportation. Together with their counterparts in Canada and Mexico, America’s freight railroads form the world’s most efficient, lowest-cost freight rail system. In addition to providing shippers with an afford-able and efficient way to move their products, freight rail-roads provide enormous public benefits, including increased fuel efficiency, lower greenhouse gas emissions, and less highway congestion.²

More than 560 freight railroads operate in the United States. The seven “Class I” railroads account for approximately 69 percent of U.S. freight rail mileage, 90 percent of employees, and 94 percent of revenue. Class I railroads typically oper-ate in many different states over thousands of miles of track. Non-Class I railroads — also known as short line railroads — range in size from small operations handling a few carloads a month to multi-state operators that are similar to Class I railroads. Together, America’s freight railroads form an inte-grated system that provides very productive and lowest-cost freight service.³

AWR 147 Rail Car Incident Response course was developed to educate emergency responders on freight rail car incidents involving hazardous materials. As more and more rail traffic transports cargo, including hazardous materials, it is critical that emergency responders are well trained. The increase in traffic adds to the already present risk of transportation inci-dents involving hazardous cargo. In addition, since a majority of the rail traffic travels through rural America, it is impor-tant that the often times resource limited rural responder community become educated about the dangers and unique hazards presented with rail cars. The information covered in this course will enhance the ability of emergency respond-ers, especially rural emergency responders, to manage rail car incidents. Some of the key elements as well as the topics addressed through the course include the following:

• Recognizing the chemical being transported, which has physical, chemical, and toxicological properties that dictate the transportation requirements and the manner in which the chemical must be considered in an emergency situation.

• Identifying the packaging system, which is reflected in the design and construction of the rail tank car, and the components of the rail car as well as all the information conveyed in the car stenciling.

• Assessing the incident to determine the damage to the car and the potential hazard to people, property, and the environment from the release or reaction of the chemical and to take appropriate protective action.

• Recognizing the appropriate and safest handling method of the chemical in the damaged car to mitigate the situation, whether by transfer, neutralization, venting, flaring, etc., and understanding these options.

• Managing the incident and the many conflicting interests represented by all the potential participants at an incident.

• Identifying federal, state and private sector resources available to assist in the response.

This course also covers additional information including topics such as the effects of Boiling Liquid Expanding Vapor Explosion (B.L.E.V.E.), the Detect, Estimate, Choose, Identify, Do, and Evaluate (D.E.C.I.D.E.) analysis system, basic rail car design, damage assessment, product transfers, and a glossary of railroad terms. Upon completion of this course, the par-ticipant should be well prepared to manage and/or respond to a rail car incident without endangering the health and safety of the responders.

This course supports the strategic goals of Presidential Policy Directive 8: National Preparedness (PPD-8), National Pre-paredness Goal, and the Core Capabilities in the areas of:

• Planning

• Operational Coordination

• Situational Assessment

• Threat and Hazard Identification

• Physical Protective Measures

• Response/Health and Safety

• Public and Private Services and Resources

• Public Information and Warning

To cover the items above, AWR 147 is comprised of ten modules each with individual Terminal Learning Objectives (TLOs). Figure 2-1 presents the modules and their corre-sponding TLOs, which were utilized in the research design.

² Association of American Railroads (n.d.). About Us. Retrieved October 2012 from http://www.aar.org/About-Us/Industry-Information.aspx

³ Association of American Railroads (2012, July). An Overview of America’s Freight Railroads. Retrieved October 1012 from http://www.aar.org/About-Us/`/Media/aar/Background-Papers/Overview%20of%20US%20Freight%20RRs%20October%2019%202011.ashx


Figure 2-1: Modules and Corresponding TLOs

Module TLOs

Module 1: Introduction, Administration, and Pre-test To explain the course purpose, goals and objectives, and summarize how course materials will give the participants and understanding of maritime security basics

Module 2: Introduction to Rail Car Emergency Response To discuss potential hazards at a train derailment and systematic process for making emergency response decisions

Module 3: Overview of the Rail/Freight Industry To discuss rail/freight industry characteristics as well as identify the regulatory requirements pertinent to a rail car emergency.

Module 4: Basic Rail Car Design To be able to identify and interpret key rail car design features to aid in the assessment of a rail car incident

Module 5: Characteristics of Chemicals and Toxicology To be able to define basic chemical terminology and will be able to differentiate the behaviors of chemicals based on physical and chemical properties. In addition, participants will be able to describe toxic effects of hazardous materials especially the impact on the human body.

Module 6: Hazardous Materials Recognition and Identification

To recognize and identify hazardous materials by evaluating available information present at a rail car incident such as occupancy/location, container/vehicle shape, markings and colors, placards and labels, documentation/shipping papers, as well as senses.

Module 7: Rail Tank Car Damage Assessment and Mitigation Techniques

To recognize important tank car features to assess during an emergency response. In addition, participants will perform simulated damaged tank car inspections utilizing assessment techniques discussed during the module. The simulated assessment is not conducted in a real world hazardous environment (i.e., cold zone) using pictures of damaged tank cars. Participants will also be able to recognize various incident control, confinement, and containment techniques that might be used by trained specialists during an incident.

Module 8: Application of the Incident Command System in Rail Car Incidents

To explain the important elements regarding planning, organizing, locating resources, and responding to a rail car incident as part of the overall incident command effort to ensure a safe conclusion of the incident and also to understand the roles and responsibilities of each responder as they apply to the Incident Command System (ICS) organization that is consistent with NIMS.

Module 9: Review of Case Studies To apply the knowledge learned throughout the course to a real life rail car incident case study, specifically elements regarding recognition, evaluation, control, information, planning, and safety.

Module 10: Post-test and Course Evaluation To complete a comprehensive post-test and course evaluation.

3.0 MethodologyAs previously mentioned, an evaluation survey was used for the AWR 147 Level 3 course evaluation to assess whether the KSAs that each participant acquired via the training course have been applied in the daily work setting. The sample drawn for this study was comprised of all participants who had completed the course from February 2009 to December 2010. Most of the participants selected for the Level 3 survey had completed the AWR 147 training within the previous 1.5 years resulting in a minimum amount of time between completing training and receiving the survey of 2.5 months. This timeframe was determined to be an adequate amount of time for participants to have possibly utilized the KSAs acquired from the training. A total of 2,879 participants were invited to participate in the survey. The postal mailing address or e-mail address that a course participant provided during course registration was used to send the evaluation invitation letter.

Multiple methods were implemented to collect data for the study. The majority of respondents (78%; n=2,244) were initially contacted by e-mail with a letter inviting them to participate in the study by completing it online (see Ap-pendix A for a copy of the invitation letter). The remaining respondents (22%; n=635) were contacted via postal as no

e-mail address was available. The date for the first dissemi-nation of invitations (e-mail and postal mail) was March 8, 2011. On June 3, 2011, as a courtesy reminder of the RDPC’s invitation to participate in the survey, postcards were e-mailed and postal mailed to all subjects who did not respond to the initial mailing. Lastly, in a final attempt to solicit a response from those subjects who had not yet participated in the study, the RDPC e-mail and postal mailed reminder post-cards on August 11, 2011 and October 22, 2011. The RDPC officially ended data collection on January 4, 2012.

The course evaluation was completed via a survey instru-ment, which included fifteen single-choice and two open-ended questions. Eleven questions were directly aligned with the TLOs listed in Section 2.0. The remaining four questions focused on obtaining details of actions taken after completion of AWR 147. Please see Appendix B for a copy of the survey instrument.

The RDPC utilized two survey formats to collect data for this study. The first format was a pen or pencil self-administered survey, which respondents returned via postal mail. The other format was an online self-administered survey which enabled the RDPC to download the data from a central server


via the Internet. The software used to create the instrument and collect the data was the Statistical Package for the Social Sciences® (SPSS) Dimensions’ mrInterview™ program. Respondents were provided the online link to the survey in the cover letter that accompanied each mailing. Additionally, each subject was assigned a unique three-digit survey code as an identifier to track his/her completion of the survey, which was also provided within the invitation letters. Sub-jects had to enter their survey codes in order to access the online survey. When participants preferred to mail the hard copy of the survey back the research personnel at the RDPC, the data was manually entered into a database. After data collection was complete, both databases (on-line and hard copies) were combined and analyzed with Statistical Package for the Social Sciences® (SPSS) 19.0.

4.0 Results

The adjusted sample size was 1,876 due to some subjects having an insufficient e-mail or postal mail addresses. Overall, a total of 390 completed surveys were received, resulting in an adjusted response rate of 20.7%. Survey completion method favored online completion (92%; n=359) versus pen/pencil completion (8%; n=31).

The data from each of the 390 surveys was analyzed using the SPSS® 19.0. Statistical analyses including frequencies and percentages were conducted to analyze the data. It was determined that these methods of univariate analysis were the most appropriate given the research objective. In regard to the single-response questions, subjects were asked whether they have utilized the skills learned via AWR 147.

The objective of this study was to determine if participants who successfully completed AWR 147 have a better under-standing of concepts related to rail car incident response as well as if they have utilized the KSAs acquired from the course in their work setting. Overall, the data indicates that the course helped the majority of the participants to better understand the response elements of rail car incidents. For example, more than 80% of the respondents indicated “Yes” on eleven specific questions (1-8, 12, 14, and 15) designed to capture whether the participants felt that the course was helpful in learning the concepts. These questions captured concepts such as:

• Better understanding of the importance of the U.S. railroad freight industry to the Nations’ overall economy and the need for protection (96%; n=376);

• Better understanding of relevant federal laws, regulation, and industry associations important to the safety and security of the U.S. railroad freight industry (95%; n=370);

• Ability to recognize various rail car designs and aspects of their construction, including tank car fitting and accessories (98%; n=381);

• Ability to recognize the different rail car classes based on the specification markings that appear on tank cars (94%; n=367);

• Better understanding of the chemical characteristics and the toxicology of hazardous materials often transported by rail cars (94%; n=365);

• Proficiency in ability to read and understand U.S. Department of Transportation (DOT) placards and labels in order to recognize the presence of hazardous materials in a rail car (96%; n= 375);

• Proficiency in ability to read and understand CONSIST shipping papers in order to gain details regarding hazardous materials in a rail car (87%; n=336);

• Better understanding of the appropriate inspection guidelines, procedures, and safety measures associated with performing a rail tank care damage assessment (87%; n=337);

• Better understanding of the various mitigation techniques and strategies associated with responding to a rail car incident (94%; n=367);

• Better understanding of the use of the Incident Command System (ICS) at rail car incidents (94%; n=365); and

• More familiar with the duties and responsibilities of the various ICS positions that would be utilized at a rail car incident scene (93%; n=363).


Figure 4-4: Question 4 ResultsQuestion 4: Since taking this course, are you able to recognize and understand the different rail car classes based on the specifi-cation markings that appear on tank cars? (n=389)

Answers Frequency %

Yes 367 94

No 22 6

Figure 4-5: Question 5 ResultsQuestion 5: Since taking this course, do you have a better under-standing of the chemical characteristics and the toxicology of hazardous materials often transported by rail cars? (n=390)

Answers Frequency %

Yes 365 93

No 25 7

Figure 4-6: Question 6 ResultsQuestion 6: Since taking this course, do you feel proficient in your ability to read and understand U.S. Department of Transportation (DOT) placards and labels in order to recognize the presence of hazardous materials in a rail car? (n=390)

Answers Frequency %

Yes 375 96

No 15 4

Figure 4-7: Question 7 ResultsQuestion 7: Since taking this course, do you feel proficient in your ability to read and understand CONSIST shipping papers in order to gain details regarding hazardous materials in a rail car? (n=388)

Answers Frequency %

Yes 336 87

No 52 13

Figure 4-8: Question 8 ResultsQuestion 8: Since taking this course, do you understand the ap-propriate inspection guidelines, procedures, and safety measures associated with performing a rail tank car damage assessment? (n=389)

Answers Frequency %

Yes 337 87

No 52 13

Figure 4-9: Question 9 ResultsQuestion 9: Since taking this course, have you performed a rail car damage assessment? (n=390)

Answers Frequency %

Yes 30 8

No 360 92

Additionally, there were four questions (9, 10, 11, and 13) developed to evaluate whether the KSAs acquired via AWR 147 have been applied in the respondents’ daily work setting, which are noted below:

• 8% (n=30) have conducted a rail car damage assessment

• 10% (n=39) have responded to a rail car incident

• 27% (n=97) have utilized the Detect, Estimate, Choose, Identify, Do, and Evaluate (D.E.C.I.D.E) process for making emergency response decisions

• 12% (n=45) have witnessed or assisted with mitigation techniques during a rail car incident (e.g., product transfer, grounding and bonding, hot tapping, and/or flaring)

Despite the low numbers in terms of the overall opportuni-ties for respondents to implement KSAs acquired via AWR 147, a vast majority of the respondents indicated that they did use their acquired KSAs in the opportunities they were offered. For example, over two thirds of the respondents stated they used KSAs acquired through AWR 147 when performing rail car damage assessments (85%; n=22) and mitigation techniques (68%; n=30).

Figures 4-1 through 4-17 present the detailed results of each close-ended survey question (questions 1-15).

Figure 4-1: Question 1 ResultsQuestion 1: Did the course help you to better understand the im-portance of the U.S. Rail/Freight Industry to the nation’s freight transportation network? (n=390)

Answers Frequency %

Yes 376 96

No 14 4

Figure 4-2: Question 2 ResultsQuestion 2: Did the course provide you with a better understand-ing of the federal laws, regulations, and industry associations important to the safety and security of the freight transportation network in your jurisdiction: (n=389)

Answers Frequency %

Yes 370 95

No 19 5

Figure 4-3: Question 3 ResultsQuestion 3: Since taking this course, are you able to recognize various rail car designs and aspects of their construction includ-ing tank car fittings and accessories? (n=390)

Answers Frequency %

Yes 381 98

No 9 2


Figure 4-16: Question 14 ResultsQuestion 14: Since taking the course, do you have a better understanding of the use of the Incident Command System (ICS) at rail car incidents that is consistent with the National Incident Management System (NIMS)? (n=390)

Answers Frequency %

Yes 365 94

No 25 6

Figure 4-17: Question 15 ResultsQuestion 15: Since taking the course, are you more familiar with the duties and responsibilities of the various positions that are used in the Incident Command System (ICS) at the scene of a rail car incident? (n=390)

Answers Frequency %

Yes 363 93

No 27 7

In addition to the questions above, respondents were asked to answer two open-ended questions (16 and 17). The first open ended question focused on actions taken as a direct result of attending the class (Please list any actions your organization has taken as a direct result of attending AWR 147 Rail Car Incident Response [e.g., pre-incident planning has been conducted and documented, performed a rail car dam-age assessment, etc.]?). Coding of the 149 responses resulted in seven main categories, which are provided in Figure 4-18 below.

Figure 4-18: Actions Taken Subsequent to AWR 147 Completion

Answers Frequency %

Updated policies, procedures, plans, etc.

38 25

Completed additional training (in-house and/or external)

30 20

Began closer relationships with local rail road companies

7 4

Increased general awareness 7 4

Participated in drills and exercises related to rail car incidents

7 4

Successfully responded to a rail car incident

5 3

No action taken 55 36

The second open-ended question asked if the participants have been involved in rail car incident response and what skills they used (Because of what you learned in AWR 147 Rail Car Incident Response, have you responded to or as-

Figure 4-10: Question 9a ResultsQuestion 9a: If “yes,” did you use the procedures you learned in this course when assessing the damage? (n=26)

Answers Frequency %

Yes 22 85

No 4 15

Figure 4-11: Question 10 ResultsQuestion 10: Since taking this course, have you responded to a rail car incident? (n=390)

Answers Frequency %

Yes 39 10

No 351 90

Figure 4-12: Question 11 ResultsQuestion 11: If you have responded to a rail car incident since taking this course, did you utilize the Detect, Estimate, Choose, Identify, Do, and Evaluate (D.E.C.I.D.E) process for making emer-gency response decisions? (n=364)

Answers Frequency %

Yes 97 27

No 267 73

Figure 4-13: Question 12 ResultsQuestion 12: Since taking this course, do you understand the various mitigation techniques and strategies associated with responding to a rail car incident? (n=390)

Answers Frequency %

Yes 367 94

No 23 6

Figure 4-14: Question 13 ResultsQuestion 13: Since taking this course, have you witnessed or assisted with any mitigation techniques such as product transfer, grounding and bonding, hot tapping, and/or flaring? (n=388)

Answers Frequency %

Yes 45 12

No 343 88

Figure 4-15: Question 13a ResultsQuestion 13a: If “yes,” did you use the procedures you learned in this course when conducting the techniques? (n=44)

Answers Frequency %

Yes 30 68

No 14 32


sisted with a rail car incident in your jurisdiction? If so, please explain the incident and how you applied the knowledge you learned through the course). While a majority of respondents indicated that they have not responded to an incident, the comments below are from those respondents who have responded to an incident since taking AWR 147.

• “We got a call that a rail car was leaking sulfuric acid in the rail yard downtown. I responded for our emergency response team. An ICS was set up and procedures followed. The local fire department and police here do an excellent job in emergencies. Turned out to be just a venting car. This class helped me to understand everything going on there and will help in future derailments. We have had two in the last five years here. Great class!!”

• “February 8, 2010, 21 rail car derailment; Incident Commander. I was able to have an informed discussion with railroad personnel; provided verbal command to fire personnel much closer to the scene and helped them interpret what they and not I could observe; I was able to read the CONSIST.”

• “We used DECIDE to get the ball rolling on discovering the magnitude of the hazardous materials spill as well as the danger associated with it and the community surrounding it.”

• “We have responded to several incidents from derailments into rivers to small leaks in non pressure cars. Our team has progressed in our knowledge of rail car safety and container design and construction. This course has allowed us to build many relationships with our local rail yard, which has in turn allowed us to obtain several training classes with local engineers and repairmen.”

5.0 DiscussionIn The Kirkpatrick Four Levels: A Fresh Look After Fifty Years, 1959-2009, authors Dr. Jim Kirkpatrick and Wendy Kirkpat-rick state that the actual execution of learning programs and overall corporate strategy occurs primarily at Level 3. In this article, the authors also quote 2008 research by Bersin and Associates that indicates that as much as 70% of learning may take place when a trainee takes the learning material back onto the job. The course designers of AWR 147 wanted to ensure that the learning objectives of the course focused on providing information that would make the rural emergency responder’s job more efficient and effective while ensuring safety for personnel and environment when responding to a rail car incident in their jurisdiction. In addition, since the course was designed at the awareness level, it was hoped that participant’s interest would be sparked to obtain additional specific training in topics that were of interest to the partici-pant. A closer examination of the survey data provides some interesting insight about the respondents’ opinions of the course’s effectiveness.

An overwhelming majority of participants responded posi-tively to gaining a better understanding of major response elements of rail car incidents as noted below.

• Importance of U.S. railroad freight industry (96%): This is a critical element that rural emergency responders need to comprehend because the U.S. railroad freight industry is a major player in the event of a rail car incident.

• Relevant federal laws, regulations, and industry association (95%): It is important for rural emergency responders to understand the applicable laws, regulations, and industry association standards that impact safety and security of the U.S. railroad freight industry.

• Recognition of rail car designs, construction, tank car fittings, and accessories (98%): One of the key goals of AWR 147 was to ensure the emergency responders were able to identify rail cars by just looking at key features. It is evident by the 98% response that this goal was accomplished. Being able to identify what types of rail cars are involved in an incident is paramount in determining response options.

• Ability to read and understand CONSIST shipping papers (87%): One of the key documents that can be found at the scene of a rail car incident is the CONSIST shipping papers. The CONSIST documents every rail car that is part of the train as well as its contents and quantity. The CONSIST identifies any hazardous materials that are stored in any of the rail cars along with emergency contact information. Being able to read a CONSIST can be very helpful in a crisis incident, especially when it comes to communicating with railroad personnel.

• Understanding of mitigation techniques and strategies (94%): Although mitigation techniques and strategies are discussed in AWR 147, the performance of these techniques is not covered in detail. Therefore, the results of Questions 13 and 13a are a somewhat misleading in that the actual procedures to conduct mitigation strategies are not specifically taught in AWR 147, but rather only reviewed at a high level.

Emergency or crisis response training is conducted to prepare responders in the event an incident happens; however, no one ever knows if responders will be able to put into action the training experienced. Given the rail car incident focus of AWR 147 and the relatively few rail car incidents that occur nationwide, it was surprising that 10% of the respondents had an opportunity to utilize the KSAs learned through AWR 147. Of particular interest was of those who responded that they performed a rail car damage assessment in which 85% used procedures learned during the training, which is a testament to the value of the assessment discussion included in the training. Participants may not have had many opportunities


What effect does successful completion of AWR 147 have on job performance and what value does it bring to the cadre of courses that the RDPC provides to rural jurisdic-tions? Questions 16 and 17 directly addresses this by asking respondents to identify action taken as a result of attending the course (List any actions your organization has taken as a direct result of attending AWR 147; Because of what you learned in AWR 147, have you responded or assisted in a rail car incident). Although there are many valuable responses to these open-ended questions (see all data included in Appen-dix C), those listed below provide a representative sample of the responses:

• “Knowing rail car design, specifically the types/locations of valves within the protective dome, helped us safety control the leak.”

• “The knowledge I learned from this class helped me in my assessment, and relay information and findings to Incident Command.”

• “The information from your course did help with our using the correct names for the rail equipment involved in the incident. This made it much easier to work with rail crews as the train was removed from the scene.”

• “I work for the only paid municipal department in my county. Your course was what prompted me to further increase my knowledge by seeking specialized training.”

It is evident that AWR 147 was a positive learning experience for the respondents and the primary learning objectives of the course had been met. Since a majority of the respondents obtained a better understanding of major response elements of rail car incidents, it is likely that in the event of an actual rail car incident the participants will be better prepared to respond and assist as a result of their completion of AWR 147.

In summary, the findings from this study suggest that the course development and evaluation processes developed and administered adopted by the RDPC is effective at producing training courses that achieve the goal of increasing the KSAs of participants. In any event, the RDPC will use the informa-tion obtained from this level three course evaluation to refine the AWR 147 course curriculum during its scheduled three-year review and update and integrate the lessons learned from this evaluation into future evaluations.

to respond to a rail car incident, but when they did, an over-whelming majority utilized techniques covered in the training.

This was evident on June 10, 2010 in the rural community of Liberty in Pickens County, South Carolina when a 24-car train derailment occurred resulting in a highly volatile situation of rail cars leaking hazardous materials. Just three months prior, members of the response team attended at delivery of AWR 147 along with other first responders in and around the area. Director Lynn Fisher, Pickens County Emergency Management Agency, stated that a value a mount cannot be assignment to the training the emergency responders received prior to the incident. Further, at the scene of the derailment, leftover field guides from the AWR 147 delivery were distributed to first responders thereby putting critical information in the hands of those who needed it, exactly when they needed it. Please see the Fall 2010 issue of the Rural Preparedness Quarterly, the official newsletter of the RDPC, for an in-depth article on the Liberty train derailment and response. 4

Another important finding is that 27% of respondents indi-cated they used the Detect, Estimate, Choose, Identify, Do, and Evaluate (D.E.C.I.D.E) process for making emergency response decisions which is a process covered in AWR 147. D.E.C.I.D.E. is a decision-making process, developed by Ludwig Benner that guides responders through a thorough and systematic approach that minimizes risk and creates continuity and ac-countability.5 Responders who follow it have the right mind-set to deal with incidents thoroughly and safely.

It is evident from some of the responses that the “awareness” level of the course may not be understood by all participants. This emphasizes the criticality of making sure all marketing materials for the course are clear on the “awareness” level focus. This course is not designed for the hazardous materials technicians who are proficient with rail cars and may be ex-pecting a performance/operational level course. Emphasizing this fact in the marketing materials as well as with the spon-soring agency will help to ensure the participants registered to attend have the correct expectation of the level of details covered in the course.

6.0 Conclusion

4 RDPC (2010, Fall). Derailment: South Carolina Puts Rail Car Training Into Action. Rural Preparedness Quarterly: Training and Information for America’s Rural Communities, pp. 1-3. Retrieved October 2012 from http://www.rural-training.org/sites/default/files/RDPC-Newsletter-Fall-2010.pdf 5 Benner, L., Jr., (1973) D.E.C.I.D.E in Hazardous Materials Emergencies. Re-trieved from http://www.bjr05.net/papershm/DECIDE.htm#Heading1


7.0 Appendix A: Modified Version of First Invitation Letter

Dear AWR 147 Participant:

You have been selected to participate in an evaluation research study regarding your completion of the course AWR 147 Rail Car Incident Response, a training sponsored by the Rural Domestic Preparedness Consortium (RDPC). This study is being conducted by the Justice and Safety Center (JSC) at Eastern Kentucky University (EKU) and is funded by the U.S. Department of Homeland Security, Federal Emergency Management Agency, National Training and Education Division. In this study, research-ers are assessing the transfer of knowledge from the classroom to the job in order to determine the success of the training, as well as to help guide the development and delivery of future training.

Your participation in this study is completely voluntary. There will not be any consequences for refusal to participate, nor will we identify those who refuse to participate. Your willingness to participate, however, will result in highly beneficial infor-mation for RDPC. It is important that your unique perspective is represented, so we ask that you not transfer the survey to another individual to complete without first consulting us. The information that you provide us will be kept confidential and you will not be identified in any way. Your information will only be combined with information from other respondents taking part in the study.

This project was reviewed and approved by EKU’s Institutional Review Board (IRB) for the Protection of Human Research Subjects. If you have questions about your rights as a participant in this study, please contact the EKU IRB Administration at 859-622-3636. You may access the survey online at the following link: http://surveys.jsc.eku.edu/AWR147survey.html. Once you access the survey, you will be required to enter a three or four-digit survey code number. Please locate your survey code number in the top-right corner of the first page of the paper version of the survey. The survey will take approximately 20 minutes to com-plete. Please complete your survey by May 15, 2011.

If you have any questions, please contact Erin Henry at (859) 622-6763 or [email protected]. Thank you for your willingness to share your experiences with us. We appreciate your participation.

Sincerely,

Dr. Pam CollinsPrincipal Investigator, RDPC / Executive Director, JSC – EKU


8.0 Appendix B: Survey Instrument

Rural Domestic Preparedness ConsortiumAWR 147 Rail Car Incident ResponseLevel 3 Course Evaluation

Instructions: Please answer the questions based on your experiences since taking AWR 147 Rail Car Incident Response. If you encounter any problems while taking the survey, or have any questions or comments in general, please contact Erin Henry at (859) 622-6763 or [email protected]. Thank you in advance for your participation.

Click “Next” to proceed in taking the survey.

Survey Code:

Please enter the survey code number that can be found at the top right corner of the first page of the paper version of the sur-vey instrument, above your mailing address on the reminder postcard or highlighted in your course evaluation email. You must enter your assigned number before you may proceed in taking the survey. (Please set number range 1 – 5,000)

1. Did the course help you to better understand the importance of the U.S. Rail/Freight Industry to the Nation’s overall economy and the need to protect the Nation’s freight transportation network?

Yes No

2. Did the course provide you with a better understanding of the federal laws, regulations, and industry associations impor-tant to the safety and security of the freight transportation network in your jurisdiction?

Yes No

3. Since taking this course, are you able to recognize various rail car designs and aspects of their construction including tank car fittings and accessories?

Yes No

4. Since taking this course, are you able to recognize and understand the different rail car classes based on the specification markings that appear on tank cars?

Yes No

5. Since taking this course, do you have a better understanding of the chemical characteristics and the toxicology of hazardous materials often transported by rail cars?

Yes No

6. Since taking this course, do you feel proficient in your ability to read and understand U.S. Department of Transportation (DOT) placards and labels in order to recognize the presence of hazardous materials in a rail car?

Yes No

7. Since taking this course, do you feel proficient in your ability to read and understand CONSIST shipping papers in order to gain details regarding hazardous materials in a rail car?

Yes No


8. Since taking this course, do you understand the appropriate inspection guidelines, procedures, and safety measures associ-ated with performing a rail tank car damage assessment?

Yes No

9. Since taking this course, have you performed a rail car damage assessment?

Yes No

(a) If “yes,” did you use the procedures you learned in this course when assessing the damage? Yes No

10. Since taking this course, have you responded to a rail car incident?

Yes No

(b) If “yes,” please explain your role in the incident response and how you applied what you learned in this course.

(Open-Ended Response)

11. If you have responded to a rail car incident since taking this course, did you utilize the Detect, Estimate, Choose, Identify, Do, and Evaluate (D.E.C.I.D.E) process for making emergency response decisions?

Yes No

(a) If “yes”, please give a brief explanation of the incident and how the D.E.C.I.D.E process was utilized.


12. Since taking this course, do you understand the various mitigation techniques and strategies associated with responding to a rail car incident?

Yes No

13. Since taking this course, have you witnessed or assisted with any mitigation techniques such as product transfer, grounding and bonding, hot tapping, and/or flaring?

Yes No

(a) If “yes,” did you use the procedures you learned in this course when conducting the techniques?

Yes No

14. Since taking the course, do you have a better understanding of the use of the Incident Command System (ICS) at rail car incidents that is consistent with the National Incident Management System (NIMS)?

Yes No


15. Since taking the course, are you more familiar with the duties and responsibilities of the various positions that are used in the Incident Command System (ICS) at the scene of a rail car incident?

Yes No

16. Please list any actions your organization has taken as a direct result of attending AWR 147 Rail Car Incident Response (e.g., pre-incident planning has been conducted and documented, performed a rail car damage assessment, etc.)?


17. Because of what you learned in AWR 147 Rail Car Incident Response, have you responded to or assisted with a rail car incident in your jurisdiction? If so, please explain the incident and how you applied the knowledge you learned through the course.


Closing Text:

You have now completed the survey. Please look back over the survey and make sure you have answered all of the questions. If you have questions or comments about the survey, please contact Erin Henry at (859) 622-6763 or [email protected].

In addition to AWR 147, the RDPC offers a wide variety of training courses for rural first responders. Please visit the RDPC web-site (http://www.ruraltraining.org) for more information on additional course offerings.

Thank you for your time and participation.


Prepare For The Worst, Train To Be The Best

Rural Domestic Preparedness ConsortiumJustice & Safety Center

Eastern Kentucky University521 Lancaster Avenue/50 Stratton

Richmond, KY 40475

Ph: (859) 622-8106 Fax: (859) 622-8038

www.ruraltraining.org

level 3 course evaluation program awr 147 rail …...level 3 course evaluation program awr 147 rail...

Documents