General Membership MeetingApril 15, 2004

St. Louis County Emergency Operations Center

PLANNING AND EXECUTING

CONTINGENCY EXERCISESWORKAREA, SYSTEMS, AREA-WIDE, REGIONAL

Anna M. Bathon, CBCP

Bank of America1

Why test recovery plans?

Recovery Strategy Considerations

Types of Exercises

Establish A Testing Strategy

Exercise Phases Planning the Exercise Preparing for the Exercise Executing/Conducting the Exercise Follow-up / Issues Resolution Closure / Next Exercise Date

Questions

Agenda

2

Why Test Recovery Plans?

1. The confluence of five major trends are driving acceptance and adoption of more aggressive recovery solutions: Businesses’ increased reliance on IT and data Availability of solutions Economics – impact of downtime and declining cost of solutions IT data management challenge

2. Gartner Group comments: “… Enterprises that today tolerate two-day recovery time objectives will see that horizon diminish to one day or less.”

3. Key disaster-related statistics: 43% of companies having a disaster never reopen. An additional 29%

close within two years. 68% of businesses that lose their computers for more than 7 days never

reopen. Within 2 weeks of the loss of computer support, 75% of those

organizations affected reach critical or total loss of business functions. Average hourly revenue lost from downtime is $78,000.

4. Businesses’ availability requirements being measured in hours.

3

5. Demonstrates to Management ability of critical business processes to continue functionality within required timeframes following a disruption.

6. Recognizing a workable plan and making a plan work are two different things.

7. Regular testing and maintaining the plan accordingly will ensure optimum performance.

8. Exercising a plan is not a PASS or FAIL situation, but an opportunity to identify plan deficiencies and improve the recovery processes.

9. Testing is a dynamic process.

10. Provides an opportunity to stress test plans already reviewed as good; exercise strenuously to identify flaws.

11. Environments – workarea, systems – change and should be monitored continuously to assess the impact of changes to recovery strategies.

12. Major revisions to recovery plans require testing and appropriate documentation updated.

Why Test Recovery Plans?

4

Recovery Strategy Considerations Workarea – physical workspace of business units, including

critical components, to ensure functionality can be resumed appropriately: Equipment / hardware Software Telecom Vital records Compliance Associate support / Intellectual Capital – What if most or all

associates or lost in a disaster situation? Support partners Regional impacts

Applications – systems, infrastructure: File-and-print servers Application components / locations:

Simple configurations Complex configurations

Infrastructure dependencies (firewalls, shared components) External dependencies

5

Recovery Strategy Considerations

Third-Party Service Providers – Dependencies on vendors increasing, thus creating a greater impact when vendors encounter disruptions. Who are the major strategic suppliers? What is the product flow throughout your company? Contingency plan options if vendor suffers a disruption? Specialized equipment or processes? Maximum potential for lost income if disruption encountered? Does an interdependency chart exist?

Regional scenarios: Natural

Weather (hurricane, earthquake, tornado, ice/snow) Man-made

Fire Terrorism Disgruntled associate reactions Accidental construction disruptions

6

Recovery Strategy Considerations

Crisis Management: Call tree notification processes Associate impacts Decision-making process to diminish roadblocks in

recovery process

7

Types of Exercises

1. Talk-Through / Table Top

2. Simulation / Connectivity

3. Integrated

4. Live

8

Types of Exercises

Talk-Through / Table Top

Generally considered first test of a plan

Cost-effective method of exercising plans

Minimal disruption to business

Raise level of awareness of the actual state of readiness

Identify major weaknesses or steps requiring further documentation

9

Types of Exercises

Simulation / Connectivity

Validates the facility, supplies, and equipment at the alternate site.

Should include connectivity testing, including voice and/or data connectivity.

Alternate site testing must include network connectivity testing, as appropriate.

Technical support participation dependent on extent of testing as defined by exercise objectives.

10

Types of Exercises

Integrated

Exercises multiple components of a plan, in conjunction with each other, typically under simulated operating conditions.

Workarea involves recovery of multiple critical business functions and related onsite systems that would be lost in the event of a site disaster.

Systems involves testing of recovery of multiple applications running on a single component or within a single site, i.e., data center environment.

Where appropriate, upstream/downstream interfaces should be exercised.

11

Types of Exercises

Live

Senior Management approval should be required for this type of exercise.

Perform production work at alternate recovery site.

High level of risk involved.

Selected associates, clients, vendors, technical support personnel, business continuity support personnel, and other dependent business units should participate.

12

Establish A Testing Strategy

1. Identify critical components of the recovery plan.

2. Identify frequency of testing based on risk rating determined through completion of BIA, i.e. quarterly, annually, bi-annually.

3. Select test type to most adequately validate all critical components. Several different test types may need to be conducted to

address all critical components to remain compliant.

4. When possible, conduct fully integrated exercises, requiring testing of all critical components.

13

Exercise Phases

Planning

Preparing

Executing / Conducting

Follow-up / Resolution

Closure / Next Exercise Date

14

Planning the Exercise

1. Identify resources

2. Select a test coordinator

3. Select the type of test

4. Define the test scope

5. Develop test goals and objectives

6. Define the disaster scenario

7. Document test assumptions

8. Set test date and duration

9. Define test team and participants

10. Schedule meetings15

Preparing for the Exercise

1. Conduct preparatory meetings with participants

2. Develop tasks and issues lists

3. Identify equipment and site requirements

4. Document high-level test scripts

5. Develop exercise packet

6. Obtain approvals

16

Executing / Conducting the Exercise1. Facilitate communication among test teams/participants.

2. Ensure activities occur in order published in exercise packet / scripts. Document deviations.

3. Ensure appropriate participants in the command center or appropriate alternate sites.

4. Work with sequence of events to log timeframes, issues, and any pertinent notations regarding activities.

5. Ensure issues documented and turned into test coordinator.

6. Compile issues into Issues List Report for tracking/resolution purposes.

7. Issues resolved during the test should be noted so.

8. Unresolved issues documented, assigned and tracked to resolution following the exercise.

9. Conduct periodic executive and test team status meetings and issue status updates throughout the exercise.

10. Document all costs associated with conducting the exercise.

11. Update appropriate telephone status resources. 17

Follow-up / Resolution

1. Schedule and conduct post-test review meeting shortly after concluding exercise.

2. Assign appropriate associates to work on resolving outstanding issues.

3. Follow up on resolution status.

4. Distribute test results and outstanding issues list report to Management, appropriate personnel.

5. Obtain validation sign-off forms from participant groups.

6. Retain exercise packets and test results for audit and regulatory reviews.

7. Follow up with participant groups to ensure recovery plans are updated based on test results / observations.

18

Closure / Next Exercise Date

1. Draft Final Summary Report and review with team in preparation for submission to Management: Final Report is a summary of actual date, time, and results of

the exercise. Include recent upgrades or changes to the workarea/units,

systems, or equipment. List exercise objectives Briefly note outstanding issues with resolution status and

target final resolution date.

2. Finalize Final Summary Report.

3. Submit Summary Report to Management.

4. Ensure all issues are resolved prior to next test.

5. Determine and communicate next exercise date.

19

Future Testing Considerations

1. End-to-end process testing.

2. Integration of different types of plans: Regional with workarea implications

Regional impacting numerous systems, workareas, vendors

3. Inclusion of new associates in process.

4. Participation in vendor contingency testing.

5. New regulatory concerns impacting recovery strategies.

6. Cyber-threat scenarios.

7. Others???

20

???????

Questions

???????

21