Slide 1

INTRODUCTION TO EXERCISE DESIGN & EVALUATION

Slide 2

INTRODUCTIONS Name Agency Exercise Experience Course Goals

Slide 3

OBJECTIVES Familiarize participants with and/or review the exercise design and evaluation process Introduce MTDES evaluation documentation Examine and analyze HSEEP Toolkit documents and systems

Slide 4

REASONS TO EXERCISE Establish a comprehensive exercise program Reveal planning weaknesses Reveal gaps in resources Satisfy regulatory requirements Clarify roles and responsibilities Test and evaluate plans, policies, and procedures

Slide 5

BENEFITS OF EXERCISING Individual Training System Improvement

Slide 6

REASONS TO USE HSEEP Common exercise policy and program guidance Consistent terminology Provides tools for exercise design, conduct, and evaluation

Slide 7

PREPAREDNESS CYCLE

Slide 8

CAPABILITIES-BASED PLANNING Reviewing current plan Conducting needs assessment Assessing jurisdictions capability to conduct an exercise

Slide 9

EXERCISE PROGRAM MANAGEMENT Project Management Multi-Year Planning Budgeting Grant Management Staff Hiring Funding Allocation Expenditure Tracking

Slide 10

MULTI-YEAR TRAINING & EXERCISE PLAN Goals Components Capabilities-Based

Slide 11

PROGRESSIVE EXERCISES Component of a comprehensive exercise program Goal: Large-scale functional or full-scale exercise Small-scale discussion-based and operations-based exercises in preparation for large-scale exercise

Slide 12

EXERCISE TYPES

Slide 13

DISCUSSION-BASED EXERCISES

Slide 14

SEMINAR Overview/Introduction Purpose Leadership Participants Applications

Slide 15

WORKSHOP Purpose Characteristics Participants Applications

Slide 16

TABLETOP Purpose Characteristics Participants Applications

Slide 17

GAME Purpose Characteristics Applications

Slide 18

OPERATIONS-BASED EXERCISES

Slide 19

DRILL Purpose Characteristics Applications

Slide 20

FUNCTIONAL EXERCISE Purpose Characteristics Applications

Slide 21

FULL-SCALE Purpose Characteristics Participants Applications

Slide 22

EXERCISE PROJECT MANAGEMENT Foundation Design and Development Conduct Evaluation Improvement Planning

Slide 23

FOUNDATION

Slide 24

Base of Support Exercise Planning Timelines Exercise Planning Team Exercise Planning Conferences

Slide 25

BASE OF SUPPORT Senior Officials Buy-In Purpose

Slide 26

EXERCISE PLANNING TIMELINES Target timeframes Multi-Year Training and Exercise Plan Origin & Composition

Slide 27

EXERCISE PLANNING TEAM Purpose Responsibilities Composition ICS Exercise Planning Team Leader Operations Safety (Ops only) Planning Admin Finance Logistics Site Liaison Resources Exercise Documentation Evaluation Props Actors Reporting Budgeting

Slide 28

EXERCISE PLANNING CONFERENCES Concept and Objectives Meeting Initial Planning Conference Mid-Term Planning Conference Master Scenario Events List Conference Final Planning Conference

Slide 29

DESIGN & DEVELOPMENT

Slide 30

EXERCISE DESIGN STEPS Assess Needs Define Scope Write a Statement of Purpose Define Objectives Compose a Narrative Write Major and Detailed Events List Expected Actions Prepare Injects Assess Needs Define Scope Statement of Purpose Define Objectives Narrative Major & Detailed Events Expected Actions Injects

Slide 31

ASSESS NEEDS Purpose Needs Assessment Process Evaluations from Past Exercises Needs Assessment Product

Slide 32

TARGET CAPABILITIES Capabilities-based planning Capabilities-based exercises Tasks & Objectives

Slide 33

DEFINE THE SCOPE Purpose Determining Functions Key Elements

Slide 34

WRITE A STATEMENT OF PURPOSE Purpose Goal Content

Slide 35

DEFINE OBJECTIVES Purpose Origin Number Good Objectives SMART Objectives S imple M easurable A chievable R ealistic T ask Oriented SMART System

Slide 36

SCENARIO Purpose Characteristics Narrative Outline

Slide 37

WRITE MAJOR AND DETAILED EVENTS Major EventsSample Purpose Goal Strategy Origin Sample 1. Fuselage breaks apart and hits buildings below 2. Jet fuel ignites several homes in the area 3. About 60 survivors are thought to be trapped in the front section of the aircraft 4. Estimates of passenger casualties rise between 200 & 220 deaths and at least 70 severe burn victims

Slide 38

WRITE MAJOR AND DETAILED EVENTS (CONT.) Detailed EventsSample Purpose Goal Strategy Origin Sample 1. The mortuary is unable to accept the large numbers of remains resulting from the crash. 2. Local hospitals lack specialized facilities and personnel to treat large numbers of severe burn victims. 3. The American Red Cross has agreed to set up a family information center to link victims and their families

Slide 39

LIST EXPECTED ACTIONS Purpose Goals Types of Actions Guidelines Relationship to Objectives Example

Slide 40

DOCUMENTATION Situation Manual Exercise Plan Controller & Evaluator Handbook Master Scenario Events List Player Handout Exercise Evaluation Guides Exercise Policies

Slide 41

LOGISTICS Discussion-based Exercises Operations-based Exercises

Slide 42

EXERCISE CONDUCT

Slide 43

Setup Briefings Management of Facilitators, Controllers, Evaluators, Players, and Actors Wrap-Up Activities

Slide 44

SETUP Site Visits Pre-Exercise Check Discussion-Based Exercises Operations-Based Exercises

Slide 45

PRESENTATIONS/BRIEFINGS Discussion-Based Exercises Operations-Based Exercises

Slide 46

PERSONNEL Facilitators Controllers Evaluators Players Actors Simulators

Slide 47

EXERCISE EVALUATION

Slide 48

EVALUATION GOALS Identify successes and shortfalls of: Response organizations Planning documents Facilities Emergency Management System Training/Staffing deficiencies Equipment Improve Emergency Management Capabilities

Slide 49

EVALUATION STEPS Plan and Organize the Evaluation Observe the Exercise and Collect Data Analyze Data Develop the Draft After Action Report & Improvement Plan

Slide 50

PLAN & ORGANIZE THE EVALUATION

Slide 51

EVALUATION TEAM LEADER Qualifications Responsibilities Designation

Slide 52

EVALUATION REQUIREMENTS & METHODOLOGY Purpose Defining the Team Structure Criteria Strategies Evaluation Plan

Slide 53

EVALUATION TEAM Team Structure Responsibilities Strategies

Slide 54

RECRUIT, ASSIGN, AND TRAIN EVALUATORS Recruiting Experience & Expertise Time Commitment Establishing Base of Evaluators Assigning Strategy When to Assign Training Purpose Materials Instructions

Slide 55

FINALIZE EVALUATION PLAN Exercise-Specific Information Evaluator Team Organization, Assignments, and Location Evaluator Instructions Evaluation Tools

Slide 56

CONTROLLER AND EVALUATOR BRIEFING Purpose Composition

Slide 57

OBSERVE THE EXERCISE AND COLLECT DATA

Slide 58

EXERCISE EVALUATION GUIDES Purpose Use Composition

Slide 59

DISCUSSION-BASED EXERCISES Higher Level Capability Issues Types of Information Collected Debrief

Slide 60

OPERATIONS BASED EXERCISES Task & Activity Level Observations Types of Information Collected Use Events to Gauge Success Evaluator Guidelines

Slide 61

HOT WASH Purpose Roles and Responsibilities Guidelines

Slide 62

COLLECTING SUPPLEMENTAL DATA Delegation Purpose of Data Examples

Slide 63

ANALYZE DATA

Slide 64

IDENTIFYING ROOT CAUSE AND DEVELOPING RECOMMENDATIONS What Happened Why It Happened Root Cause Analysis

Slide 65

DATA ANALYSIS Discussion-Based ExercisesOperations-Based Exercises Controller & Evaluator Debrief Draft AAR Content Preliminary Analysis Content Controller & Evaluator Debrief Draft AAR Content Exercise Event Timeline Preliminary Analysis Content

Slide 66

DEVELOP THE DRAFT AFTER ACTION REPORT & IMPROVEMENT PLAN

Slide 67

AFTER ACTION REPORT Purpose Who is responsible for it? Form Format

Slide 68

IMPROVEMENT PLANNING

Slide 69

CONDUCT AFTER ACTION CONFERENCE Purpose Content Format

Slide 70

IDENTIFY CORRECTIVE ACTION TO BE IMPLEMENTED Corrective Actions Purpose Content Realistic Goals

Slide 71

IMPROVEMENT PLAN Purpose Components Capability Observation Recommendations Corrective Action Description Responsible Party/Agency Agency POC Start Date End Date

Slide 72

CAPABILITY ELEMENTS Planning Organization and Leadership Personnel Equipment and Systems Training Exercises, Evaluations, and Corrective Actions

Slide 73

FINALIZE AAR/IP Purpose Composition Goals

Slide 74

TRACK IMPLEMENTATION Event Points of Contact Participating Entity Points of Contact Action Officers Continual Improvement

Slide 75

QUESTIONS?