Update on Disaster RecoveryTask Force Efforts

June 4, 2007

Contributors: CNIC Disaster Recovery Task Force State Data Center

All SDC-based Agencies will have developed, maintained,

and tested plans by June 2009 to restore and recover their

agency critical business functions/applications in the event of

a disaster. (per DAS Statewide Business Continuity Planning Policy

107- 001-010 -- http://egov.oregon.gov/DAS/OP/docs/pdf/107001010.pdf )

CNIC Disaster Recovery Task Force 2

Background

The CNIC CIO’s formed a CNIC Disaster Recovery Task force to examine and propose DR solutions for systems and services managed by the State Data Center.

The Sungard DR service contract was extended through October 2007, while requirements for a new RFP and service contract can be gathered and awarded.

CNIC Disaster Recovery Task Force 3

Disaster Recovery Plan: Steps taken to restore specified infrastructure requirements in support of key Business Functions such as Information Systems, business equipment environments, internal and external network connections, and data structures utilizing alternate resources for hardware, software, data, and networks. - Hardware - System Software

- Data and Data Structures - Applications - Network Connectivity - Desktop Services - Production Support

DRP

Business Continuity Plan: Steps taken to restore alternate business processes in the event that automated processes or business infrastructures are unavailable, employing documented workaround and/or manual procedures and alternate resources.

- Relocation of Personnel

- Availability of remote support services - Contingency office space

- Process workarounds

BCP

Components of the Business Continuity Plan

CNIC Disaster Recovery Task Force 4

DR Task Force – Objectives

Complete DR plans for IT infrastructure that support the Agency critical business functions.

Track and manage the BCP/DR planning effort by the use of a Scorecard which includes the DR tasks, milestone dates, and detailed template worksheets.

Assure all IT application and systems information is gathered and loaded in the eBRP Agency instance tool. Assumption: For disaster recovery, the SDC is to be given access to each Agency’s eBRP instance.

Develop DR operational guidelines, ala a Concept of Operations.

Collaborate with the SDC to issue an RFP for recovery service contract pricing and selection of a recovery vendor.

Issue a disaster recovery service contract that: Will provide recovery for SDC-managed critical business services. Can adopt recovery services for other SDC customers and/or out-of-scope services.

CNIC Disaster Recovery Task Force 5

DR Task Force Issues and Proposed Mitigation

Action All SDC-Based Agencies to validate their Business Critical Functions annually!Action: - Agencies to commit and actively support resources and time to identify Agency’s key Business Critical Functions.

- CIO’s to identify which of these functions need to be recovered in the SDC during a disaster.

State to hire limited duration DR ExpertiseAction: - Using existing SunGard Contract or an RFP process, DAS to hire full time DR expertise. - Assign DR expert to SDC or State CIO Office.

State to hire ongoing full time DR Program/Project ManagerAction: - Assign DR Program/Project Manager to SDC or State CIO Office.

Establish Multi-Agency Workgroups to address key DR servicesAction: - Establish recovery teams for Roles & Responsibilities, Concept of Operations Declaration, network services, etc.

- Decide what work can be performed prior to the introduction of a DR Expert.

CNIC Disaster Recovery Task Force 6

Combine BCP information in eBRP with Agency’s DR detailed template file.Action: - For disaster recovery, propose Agencies to give SDC access to their eBRP instance. - No need to create a program for a single combined SDC instance of Agencies DR requirements.

Expand Disaster Recovery Scorecard TimelineAction: - Modify timeline to accommodate agency-specific time requirements. - Extend the SunGard contract as necessary until new DR Services RFP is ready. - Ensure Agencies still meet overall deadline of June 30, 2009 BCP deadline, per DAS Policy 107- 001-010.

Ensure DR resources to complete the DR Scorecard and Template. Action: - Agencies have already designated the BCP coordinator, but CIO’s need to ensure they have dedicated DR technical resources to complete the DR Scorecard.

DR Task Force Issues and Proposed Mitigation

Action (cont’d)

CNIC Disaster Recovery Task Force 7

DR Task Force Efforts – The Scorecard

CNIC Disaster Recovery Task Force 8

Disaster Recovery Declaration Process(Sample Only)

DR

Dec

lara

tio

nS

tag

eD

R M

ob

iliza

tio

nS

tag

eD

R A

ctio

n/R

eco

very

Sta

ge

State Data Center Disaster Recovery Plan Team Structure (CNIC Governing Board Management Communication)

Disaster Recovery Plan Team Structure

Disaster Recovery Commander

Facility Recovery Team Administrative Support

Functions 1. Sustain Command Center 2. IS Recovery Center Help Desk 3. Problem Reporting Log 4. DR Documentation 5. Team/Employee Contacts 6. Vendor Support Agreements 7. Order Tracking Log

Functions 1. Facility Evaluation 2. Damage Assessment 3. Safety and Security 4. Vendor Support 5. Voice communications

Alternate Site Recovery Team Applications Support Team SDC Technology Recovery Team

DR Functions 1. Perform Host Recovery 2. Perform Server Recovery 3. Perform Network Recovery 4. Assist system Operations 5. Assist Network Operations 6. Assist Server Operations 7. Technical Support 8. Coordinate Vendor Support

DR Functions 1. Downtime Transaction Management 2. User Application Support 3. Coordinate Vendor Support 4. Data Integrity Balancing

DR Functions 1. Establish Data Center Operations 2. Perform daily Operations Schedule 3. Perform Network Monitoring 4. Establish Telephone Connections 5. Coordinate with agency Help Desk 6. Assist with Vendor Support

Command Center Manager

Property Maintenance Management Vendor Support Representatives SDC Telecom Mgt.

Support Center Ops. Mgt.

Predefined SunGard Site

Existing State Contract 4 Agencies

Alternate Site Data Center

Options: State, Business Partner

Applications Management Recovery Technical Teams Prod Sys Mgr

Network Svcs Mgr Telecom Mgr.

Executive Advisory Group

CIO’s & Directors

IS Disaster Recovery Management Team

Directors: DAS & Agency PIO Communications,

Customer Relationship Managers

Technology Mgt. Telecom

Prod Sys. & Networks

Safety Officer Liaison Officer

Repair Existing Data Center

SDC Management

Critical Call Management Process

CNIC Disaster Recovery Task Force 9

BCP and DR Framework Chart

CNIC Disaster Recovery Task Force 10

Critical Business

Functions (ODOT Sample)

SDC-based Service(11 In-Scope)

SDC Out-of-Scope

CNIC Disaster Recovery Task Force 11

Disaster Recovery Taskforce Update CNIC Steering Committee