understanding disaster recovery in california...ocha & ibm confidential © 2003 ibm corporation...

Disaster Recovery Planning

© 2008 Cyber Communication

Understanding Disaster Recovery in California

Protecting your EnterpriseDisaster Recovery Planning

OCHA & IBM Confidential © 2003 IBM Corporation

OCHA: COOP Disaster Recovery Planning


Session Overview

Why do we PrepareWhat is Disaster Recovery Planning?How do I analyze (measure) it?What to do with it?How do I communicate it?What does it mean to management?




1996-2006: A Decade of Natural Disasters

1 million thunderstorms1 million thunderstorms100,000 floods100,000 floodsTens of thousands of landslides, earthquakes, Tens of thousands of landslides, earthquakes, wildfires & tornadoeswildfires & tornadoesSeveral thousand hurricanes, tropical cyclones, Several thousand hurricanes, tropical cyclones, tsunamis & volcanoestsunamis & volcanoes

Sources: CDC & EK Noji, Sources: CDC & EK Noji, The Public Health Consequences of DisasterThe Public Health Consequences of Disaster




Executives and Management are being held to a higher level of performance or

Governance

Governance and Culture – Rising ExpectationsGovernance and Culture – Rising ExpectationsThe Regulatory Environment (HIPAA, PCI, SAM, BL)Control Framework – manage risk (ITIL, ISO, COBIT) Aligning business with ITHaving a resilient business model

Processes and Procedures Processes and Procedures

Efficiency addressing mandatesDelivering valueTools and technologyImprove the management of Trust




Identifying exposures and managing associated risks increases your appeal to customers, stakeholders, business partners, and regulators.

A stable and prepared business builds trust with its:A stable and prepared business builds trust with its:

The PublicRegulatorsStakeholdersBusiness partners

Increased customer satisfaction and….Increased customer satisfaction and….

Lower total operating expensesOptimized expenditures Enhanced public value




Video??

Video




Three Phases of Continuity

Emergency Response - Life Safety

First 72 Hours

Damage Assessment

First 72 hours

Restoration

Business back

back to

normal

IT Disaster Recovery

up to 30 daysBusinessRecoveryup to 30

days

Planning, Documenting, Testing, and Training

Departments

Phase I Phase II Phase III




Definitions

Life Safety – ERPEssential State Government Functions – COGEssential Department Functions – COOPCommunications Functions – CCPBusiness Recovery Functions – BCPIT Recovery Functions – DRP




Emergency Response

The immediate reaction and response to an emergency situation commonly focusing on ensuring life safety and reducing the severity of the incident. – typically the first 24 hours & up to ........

• Definition from Disaster Recovery Journal (DRI) website at: http://www.drj.com/glossary/




Continuity of Operations & Continuity of Government (COOP/COG)

(Also known as Business Continuity)

Continuity of Operations (COOP) – The activities of individual departments and agencies and their sub-components to ensure that their essential functions are continued under all circumstances. This includes plans and procedures that delineate essential functions; specify succession to office and the emergency delegation of authority; provide for the safekeeping of vital records and databases; identify alternate operating facilities; provide for interoperable communications; and validate the capability through tests, training, and exercises.

• Office of Emergency Services (OES)




Disaster Recovery Planning (DRP)

Disaster Recovery Plan (formally known as - Operational Recovery Plan):

The management approved document that defines the resources, actions, tasks and data required to manage the technology recovery effort. It provides for owners to define the Maximum Allowable Outage (MAO) requirements for the “essential” applications. This is a component of the Business Continuity Management Program.

• Definition from Disaster Recovery Journal (DRI) website at: http://www.drj.com/glossary/




Relationship of Plans

Continuity of OperationsContinuity of Government

OperationalRecovery

EmergencyResponse

Business Continuity




Costs of Recovery ControlsCosts of Recovery ControlsCosts of Recovery Controls

TotalCost of

Recovery

TotalTotalCost of Cost of

RecoveryRecovery

Level of Protection ProvidedLevel of Protection ProvidedLevel of Protection Provided

Costs of Disaster EventsCosts of Disaster EventsCosts of Disaster Events

Minimized Total CostsMinimized Total Costs




Risk Assessment

A Risk Assessment is the analysis of possible disasters, including natural, technical, social and human threats that can result in short or long term downtime. Each functional area of the organization should be analyzed to determine the potential negative consequences and impact associated with various disaster scenarios. During the risk assessment process consideration should be given to evaluate the safety of critical documents and vital records related to the continuance of business operations.




Risk Assessment

Items to consider in determining the probability of a specific disaster should include, but not be limited to:

• Proximity to power sources, water bodies, and airports

• History of the area’s susceptibility to natural threats

• Proximity to major highways which transport hazardous waste and combustible products

• Business climate and cultural risks• Other factors




Business and Operation Impact Assessment

A Business Impact Assessment (BIA) is the foundation for business and patient care continuity planning. A detailed BOIA should identify the business, financial and clinical operational impacts that may result from a disruption of operations. Negative impacts may results in:

• Cost of downtime

• Loss of Revenue

• Inability to continue with patient care

• Loss of automated processes




Awareness – Financial Impact

High Availability Cannot Be Acquired Out-Of-The-Box; It Is Built Into the Architecture and Preserved by Effective Processe

Lost Revenue• Direct Loss• Compensatory Payments• Lost Future Revenues• Investment Loss

Productivity Loss• Number of Fully Burdened

Employee impacted

Damaged Reputation• Patient, Suppliers, Partners,

Banks, Financial Markets• Credit Ratings

Delayed Collections• Billing Losses• Missed DiscountsExtra Expense

• Cost to Recover• Overtime Expense• Increased Fraud Risk• Increased Error Rate• Travel Expenses• Temporary Employees

Penalties • Contractual• Regulatory• Legal




• The disaster recovery plan should include a descriptive list of the organization's major business areas. This list should rank the areas in order of importance to the overall organization.

• Each item should include a brief description of the business and processes and main dependencies on systems, communications, personnel, information systems and data.

Assessing Key Business Areas




The Process – Getting Started

• AssessAssessments are critical to the planning of healthcare disaster recovery. They can provide detail information that can be crucial when making a decision. Accurate Disaster Recovery Planning can be accomplished by having information before hand regarding risk factors and the impact of operations interruption.

• Determine what the Recovery Plan and Time Recovery Objectives.

Determine what the objectives are for planning and recovery time. • Determine the requirements for planning.

These are the planning requirements that need to be met in order to accomplish your recovery plan and time objectives (RPO & RTO).Infrastructure




The Office space, phones, intranets, LAN/WAN access, internet/intranet, security etc.

• Systems RestoreIncludes both Hardware and Operating System

• Critical Applications Includes programs that are critical to the continuity of the business and patient care.

• DataLive records containing business and clinical transactions as well as specific procedures and business rules.

• Operations ContinuityDaily operations and tasks to secure the continuance business and patient care processes.




Who Owns It?

DepartmentsITSDDTSThird-party vendors ???????????????




What’s It Worth?

States ImageReplacement Branding Daily OperationsCompetitive Advantage




12% 20%

26%42%

Employee Brains

Sharable ElectronicKnowledge

Base

ElectronicDocuments

Paper Documents

Assessing our Knowledge Assets

* 2005 disaster Resource * 2005 disaster Resource GuideGuide




Department Data Classification Matrix

Time Sensitive Nature

Category A(Highest, most essential)

Category B(Moderate, some level of

criticality)

Category C(Very low, but still

desirable)

Legal requirements

Protection of data is required by law (see attached list for specific HIPAA and FERPA data elements)

Department has a contractual obligation to protect the data

Reputation risk High Medium Low

Other Institutional Risks

Information which provides access to resources, physical or virtual

Smaller subsets of Category A data from a department

Data about very few people or other sensitive data assets




The Hamster Wheel of Pain (how management sees Disaster Recovery strategies)

Ignorance Am

I Hose

d

YesSheer Panic

It’s

“Fix

ed”

Management’s View

Disaster Management to most is “Risk Identification”

Captures a simple Risk Management message,

“Identifying and fixing things”Disaster Management (and the analysis and assessment of it’s performance) needs to be organizationally focused & using business domain knowledge

How do my strategies compare with my peers?




The Disaster Recovery Plan

The WorkflowIt is crucial to develop an effective workflow. The workflow can determine how your DR plan will be executed.It also provides a guide and road map to the decision making process.The response and recovery time frame will impact on overhead costs and loss of revenue.

Crisis Anticipation/Declaration

EmergencyResponse

MobilizeResources

RestoreApplication

RestoreFrom Backup

ResumeOperations

RemoteLocation

RemoteLocation

RemoteLocation

RemoteLocation

Overhead Costs and Loss of Revenue

RestoreTo

NormalOperations




Testing

Annual testing of the ORP is essential to:– Ensure for training the management and recovery teams.– Validate that the procedures have the appropriate level of

detail.– Verify Call Back lists are current.– Confirm that Recovery strategies are appropriate for your

environment.




IMPLEMENTATION OF PLANS

Disruption of business occurs and you are informed, next steps

Emergency Response – safety and security of staff.Securing the site.Activate COOP/COG Plan to ensure the continuation of essential functions.Implementation of the communication plan.After assessing incident, determine if implementation of BCP & ORP is required.Contact SISO to report incident.Implement BCP and ORP




OISPP Requirements

DRPs must describe:Agency Administrative InformationCritical Business Functions/ApplicationsRecovery StrategyBackup and Offsite Storage ProceduresOperational Recovery ProceduresData Center ServicesResource RequirementsAssignment of ResponsibilityContact InformationTesting




Disaster Recovery Lifecycle

Put all this in place with our business

partners

What can we add or change to improve our recoverability?

How well are we protected, now and

in the future?

Given what we have, how do we handle a

catastrophic Disaster?




What else do I need to consider?

• Several things, but first and foremost, make sure your critical data/vital records, as in tape files, mirrored disk, paper archives, etc., are stored in a safe location (off-site storage) and can be retrieved

• Without your data, your plan will not work

• Maintain the plan on a regular basis

• Think “out of the box”!




Conclusions

– Physical and IT security will become more tightly integrated

– BCP must encompass all aspects of an organization

– Security is a crucial component to BC and disaster prevention

– Proper identification, planning, and implementation will ensure not only success, but business survival




At a Personal Level

• Contact your Emergency management or civil defense office• Meet with your family and discuss how to prepare and respond• Plan how your family will stay in contact if separated• Complete these steps:

• Post emergency numbers on each phone• Show responsible family members where to shut off utilities• Install (and test) smoke detectors on each level of your home• Contact your local fire department and learn about in-home fire

hazards• Learn first aid and CPR

• Meet with your neighbors and plan how the neighborhood could work together after a disaster

• Know your neighbor’s skills (medical, technical)• Consider special needs such as elderly, disabled, child care




Resources

SISO web site: http://www.infosecurity.ca.gov/ORP/

Budget Letter 07-03 – ORP Policy Changeshttp://www.dof.ca.gov/OTROS/StatewideIT/IT_BdgtLttrs.asp

ORP Policy in the State Administrative Manual (SAM):Operational Recovery Planning: http://sam.dgs.ca.gov/TOC/4800/4843.htmOperational Recovery Plan http://sam.dgs.ca.gov/TOC/4800/4843.1.htm

ORP – SIMM 65A: http://www.infosecurity.ca.gov/Policy/

http://www.infosecurity.ca.gov/ORP/

http://www.dof.ca.gov/OTROS/StatewideIT/IT_BdgtLttrs.asp

http://sam.dgs.ca.gov/TOC/4800/4843.htm

http://sam.dgs.ca.gov/TOC/4800/4843.1.htm

http://www.infosecurity.ca.gov/Policy/




Resources

Web Sites:www.drj.comwww.contingencyplanning.comwww.globalcontinuity.comwww.recovery.sungard.comwww.disaster-resource.comwww.fema.gov

Professional Organizations

http://www.drj.com/

http://www.contingencyplanning.com/

http://www.globalcontinuity.com/

http://www.recovery.sungard.com/

http://www.disaster-resource.com/

http://www.fema.gov/




Thank You! Jack Orlove (916) 316-1375

www.cybercommunication.com

Business Continuity

Cyber SecurityBusiness Analysis

Disaster Recovery