disaster recovery and business continuity for your clinical and safety systems
TRANSCRIPT
Disaster Recovery and Business Continuity for Your
Clinical and Safety Systems
Sean Bernard, Lead Business Consultant, Life Sciences, Perficient
2
ABOUT PERFICIENT
Perficient is a leading information
technology consulting firm serving
clients throughout North America.
We help clients implement business-driven technology
solutions that integrate business processes, improve
worker productivity, increase customer loyalty and create
a more agile enterprise to better respond to new
business opportunities.
3
PERFICIENT PROFILEFounded in 1997
Public, NASDAQ: PRFT
2014 projected revenue ~$454 million
Major market locations:
Allentown, Atlanta, Ann Arbor, Boston, Charlotte,
Chicago, Cincinnati, Columbus, Dallas, Denver,
Detroit, Fairfax, Houston, Indianapolis, Lafayette,
Milwaukee, Minneapolis, New York City, Northern
California, Oxford (UK), Southern California,
St. Louis, Toronto
Global delivery centers in China and India
>2,600 colleagues
Dedicated solution practices
~90% repeat business rate
Alliance partnerships with major technology vendors
Multiple vendor/industry technology and growth awards
4
Business Process Management
Customer Relationship Management
Enterprise Performance Management
Enterprise Information Solutions
Enterprise Resource Planning
Experience Design
Portal / Collaboration
Content Management
Information Management
Mobile
BU
SIN
ES
S S
OL
UT
ION
S
50
+ P
AR
TN
ER
S
Safety / PV
Clinical Data Management
Electronic Data Capture
Medical Coding
Clinical Data Warehousing
Clinical Data Analytics
Clinical Trial Management
Healthcare Data Warehousing
Healthcare Analytics
CL
INIC
AL / H
EA
LT
HC
AR
E IT
Consulting
Implementation
Integration
Migration
Upgrade
Managed Services
Private Cloud Hosting
Validation
Study Setup
Project Management
Application Development
Software Licensing
Application Support
Staff Augmentation
Training
SE
RV
ICE
S
OUR SOLUTIONS PORTFOLIO
5
WELCOME AND INTRODUCTION
Sean Bernard, Lead Business Consultant, Computer Systems Validation
Life Sciences, Perficient
(603) 340-7857
6
AGENDA
• The Danger is Real
• Business Continuity and Disaster Recovery Overview
• Regulatory Requirements for BC and DR
• What a BC Program Is and Why You Need One
• BC Program Stages and Deliverables
• Conclusions
• Questions
7
THE DANGER IS REAL
The (Business) World is a Scary Place
8
THE DANGER IS REAL
Weather and Climate Disasters 1980 - 2014
From 1980–2014, there were:
• 22 drought events
• 20 flooding events
• 7 freeze events
• 70 severe storm events
• 34 tropical cyclones
• 12 wildfire events
• 13 winter storm events
EACH with losses exceeding
$1 billion (CPI-Adjusted)
“Billion Dollar Weather and Climate Disasters”, National Oceanic and Atmospheric Administration,
http://www.ncdc.noaa.gov/billions/mapping (accessed 23-Jan-2015)
9
THE DANGER IS REAL
Natural Disaster Losses in the US, 1980 - 2014
“Catastrophes: US”, Insurance Information Institute, http://www.iii.org/fact-statistic/catastrophes-us (accessed 23-Jan-2015)
10
THE DANGER IS REAL
“Cyberattacks account for up to $1 trillion in global losses”, Dana Kerr, CNET,
http://www.cnet.com/news/cyberattacks-account-for-up-to-1-trillion-in-global-losses/ (accessed 10-Feb-
2014)
11
THE DANGER IS REAL
• The most common generalized threats are:
– Computer systems outages
– Criminal activity/hostilities
– Fire
– Natural disasters
– Epidemic/pandemic
12
THE DANGER IS REAL
So BC and DR are taken
seriously…right?
13
THE DANGER IS REAL
“2013 AT&T Business Continuity Study Results - U.S. Trend Data, http://www.business.att.com/content/article/2013-ATT-
BCStudy-Comparison-Results.pdf (accessed 27-Jan-2015)
Business Continuity Trends
14
Business Continuity and Disaster Recovery Overview
15
BC & DR OVERVIEW
Business Continuity
• Can be defined as:
A proactive process which identifies the key functions of an organization and the likely threats to those
functions; from this information plans and procedures which ensure key functions can continue whatever
the circumstances can be developed.1
Disaster Recovery
• Can be defined as:
The process an organization uses to recover access to their software, data, and/or hardware that are
needed to resume the performance of normal, critical business functions after the event of either a natural
disaster or a disaster caused by humans.2
1. The Definitive Handbook of Business Continuity Management, ed. Andrew Hines and Peter Barnes
(West Sussex: John Wiley & Sons, Ltd., 2006), 379
2. “Information on Business Continuity and Disaster Recovery,” www.disasterrecovery.org (accessed 30-
Dec-2014)
16
BC & DR OVERVIEW
What’s the difference between BC and DR?
• Business continuity is about the survival of critical business functions.
• Disaster recovery is about the survival of critical IT systems.
• Since most critical business functions rely on systems, DR is frequently seen as a subset of BC,
which is how we’ll look at it in this presentation.
• BC is higher-level, and less technical. DR is lower-level, and more technical.
17
BC & DR OVERVIEW
Example:
XYZ PharmaCo, Inc. is a small pharmaceutical company located in Buffalo, NY. They have
defined Clinical Data Acquisition as one of their critical business functions. XYZ has a data
center in Buffalo equipped with IT systems which include Oracle Clinical and Oracle Remote
Data Capture systems. Site users from the Northern NY area log into the systems to enter
clinical study data for ongoing trials. So for XYZ, an interruption in the operation of these IT
systems would result in an interruption of the Clinical Data Acquisition function. So, since both
the systems and the function are interrupted the outage would encompass both DR and BC.
However, the Clinical Data Acquisition could also be interrupted by a strong winter storm. Site
personnel and patients would be unable to report to the site, and data acquisition would not be
possible. The IT systems are intact and working perfectly, but the function is still interrupted.
This interruption would require only BC.
18
BC & DR OVERVIEW
Common misconceptions about BC and DR:
• I’ve got a Disaster Recovery Plan, I’m good!
– Since DR only covers systems, BC is also required to cover
functions.
• Business continuity is only for large organizations.
– Every organization has critical functions which must be
protected.
• We have multiple locations, so we don’t have to worry.
– Multiple locations may help during recovery operations, but
they cannot replace BC/DR
• Regulations don’t require me to have BC/DR.
– 21 CFR Part 11 Subpart B 11.10c requires you to protect your
data, as does EU Annex 11.
19
Regulatory Requirements for DR / BC
20
REGULATORY REQUIREMENTS FOR DR/BC
21 CFR Part 11, Subpart B, Section 11.10c “Controls for Closed Systems” states:
Persons who use closed systems to create, modify, maintain, or transmit
electronic records shall employ procedures and controls designed to ensure the
authenticity, integrity, and, when appropriate, the confidentiality of
electronic records... Such procedures and controls shall include the
following:
...
(c) Protection of records to enable their accurate and ready retrieval
throughout the records retention period.
...
“CFR Code of Federal Regulations Title 21”, US Food and Drug Administration,
http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfcfr/CFRSearch.cfm?fr=11.10 (accessed 31-Dec-2014)
21
REGULATORY REQUIREMENTS FOR DR/BC
The FDA, through issued guidance, has affirmed that 21 CFR Part 11 applies to DR.
Section 5.2.5:
System level software testing addresses functional concerns and the following
elements of a device's software that are related to the intended use(s):
...
Effectiveness of recovery procedures, including disaster recovery;
...
“CFR Code of Federal Regulations Title 21”, US Food and Drug Administration,
http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfcfr/CFRSearch.cfm?fr=11.10 (accessed 31-Dec-2014)
22
REGULATORY REQUIREMENTS FOR DR/BC
EU Volume 4, Annex 11, Computerised Systems, Section 16 states:
For the availability of computerised systems supporting critical processes,
provisions should be made to ensure continuity of support for those processes
in the event of a system breakdown (e.g. a manual or alternative system). The
time required to bring the alternative arrangements into use should be based
on risk and appropriate for a particular system and the business process it
supports. These arrangements should be adequately documented and tested.
“Volume 4, Good Manufacturing Practice, Medical Products for Human and Veterinary Use, Annex 11,
Computerised Systems”, European Commission, Health and Consumers Directorate-General,
http://ec.europa.eu/health/files/eudralex/vol-4/annex11_01-2011_en.pdf (accessed 31-Dec-2014)
23
REGULATORY REQUIREMENTS FOR DR/BC
So…
Strictly speaking, the regs could be interpreted to mean that Disaster Recovery is a regulatory
requirement for your Clinical & Safety systems, and BC is not a regulatory requirement.
BUT…(Big But)
A BC Program should ALWAYS be a business requirement
24
What a BC Program Is and
Why You Need One
25
BC PROGRAM – WHAT AND WHY?
What is a BC Program?
• A BC Program is a comprehensive, ongoing process of developing, testing, and maintaining a Business
Continuity Plan (BCP) and recovery procedures for your organization.
• A BCP is a living document that must be continually refined in order to ensure it is kept accurate and up-
to-date as your business matures and changes.
• BC planning is not a one-and-done affair, it is a continuous process.
• A BC Program must be sponsored and approved by senior management. It is a high visibility core
business function, and should receive a high degree of attention.
• A good BC Program integrates with ongoing GxP validation and Quality Assurance activities.
• An outdated BCP and/or DRP can be more dangerous than none at all, as it will give obsolete
information at a time of great need.
26
BC PROGRAM – WHAT AND WHY?
You need a BC Program because…
• Systems alone do not operate your business
• Recovery takes much longer and is much
more expensive without a BC program
• Difficult to determine critical systems without
knowing your critical functions
• If you’re a service provider, consulting
company, or CRO, you’re clients will need to
know your business, and their data, can
survive.
27
BC PROGRAM – WHAT AND WHY?
• Planned and tested – reduces the likelihood and impact of the disaster
• Orderly – Critical functions and systems have been identified, so effort can be made to recover them
first, and then concentrate on non-critical functions/systems
• Fast – People know what to do and where to find information
Recovery With a BC Program
28
BC PROGRAM – WHAT AND WHY?
• Unplanned and untested – will increase the impact that any disaster will have
• Chaotic – What functions/systems should we bring up first? Using what resources? Using what
procedures?
• Slow – Since critical functions/systems haven’t been identified, this will need to be done during the
disaster or people will be working on non-critical functions/systems
Recovery Without a BC Program
29
BC PROGRAM – WHAT AND WHY?
• We’ve discussed:
– WHY you must have DR
– WHY you should have a BC Program
– WHAT a BC Program is
• Now we’ll look at HOW to implement it
30
BC Program Stages and Deliverables
31
BC PROGRAM – STAGES & DELIVERABLES
• Analysis
– Business Impact Analysis
• Development
– Business Continuity Plan (BCP)
– Disaster Recovery Plan (DRP)
– BC and DR policies, procedures, and guidelines
– Forms, templates, etc. to support BC and DR functions and testing
• Implementation
– Approve and release BCP, DRP, policies, procedures, guidelines, and forms
• Testing
– Approved and executed BC Test Plan and DR Test Plan, forms, test cases, etc.
– Approved summary reports for BC and DR testing
• Maintenance
– Lessons learned from testing feed back into Analysis phase
32
BC PROGRAM - ANALYSIS
• BC starts with a thorough Business Impact Analysis which identifies:
– Critical business functions for your organization
– Information systems which support critical business functions
– Maximum allowable downtime for these functions
– The objective ($) impact that offline functions will have on the organization
– Assets and resources needed for the continued operation/recovery of these systems & functions
• Must have approval AND buy-in from senior management
• Must be exhaustive so that no critical functions are overlooked
• Must identify the BC champion and other responsible parties
33
BC PROGRAM - DEVELOPMENT
From the knowledge gained from the Business Impact Analysis:
• Develop Policies, Procedures, and Guidelines
– State that a BCP and DRP will be created, and what they will contain
– Describe when and how they will be tested, testing constraints, and acceptance criteria
– How testing will be summarized and reported
– How the BCP and DRP will be reviewed, maintained, and updated
• Write the BCP
– Contains a risk assessment analyzing current threats
– Details any measures taken for disaster preparedness
– Lists critical business functions
– Details the procedures to be taken during a disaster to recover critical functions
– Lists personnel and other resources involved in BC activities
– Flexible enough to handle unforeseen threats
34
BC PROGRAM - DEVELOPMENT
From the knowledge gained from the Business Impact Analysis:
• Write the DRP
– Contains a risk assessment analyzing company systems to determine which should be
considered critical
– Lists critical information systems and services
– Describes measures taken to reduce data loss, and determines the maximum data loss
– Details the procedures to be taken during a disaster to recover critical information systems
– Lists personnel and other resources involved in DR activities
– Lists contact information for approved vendors providing DR materials/services
• Create any forms, templates, documents to support BC and DR testing
– Damage assessment forms
– Incident report forms
– Testing templates, etc.
35
BC PROGRAM - IMPLEMENTATION
After the development and review of governing documents:
• Approve and release the Policies, Procedures, and Guidelines
– Approved by senior management and Quality Assurance
– Release in accordance with your Quality Management System (QMS)
– Allow time for training before effective dates
– Retain training records for all personnel with BC/DR responsibilities
– Release any required forms, templates, testing docs, etc.
• Approve the BCP and DRP
– Same as above (approval, release, and training)
– Schedule testing in accordance with the approved policies, procedures, and guidelines
36
BC PROGRAM - TESTING
After the implementation of the governing documents:
• Schedule testing
– In accordance with policies, procedures, and guidelines
– Notify responsible personnel of testing schedule
• Write the BC and DR Test Plans
– Scope of testing (what critical functions and systems are to be recovered)
– Responsibilities
– Test methodology
– Deliverables
– Acceptance criteria
37
BC PROGRAM - TESTING
After the implementation of the governing documents:
• Execute testing
– Documented supporting evidence (completed forms, executed test cases, screenshots, etc.)
– Reviewed
• Write the BC and DR Summary Reports
– Summary of test results
– Detail any deviations or issues encountered during testing
– State whether the acceptance criteria has been met
38
BC PROGRAM – MAINTENANCE
After the testing has been completed:
• Lessons Learned
– Schedule meeting with personnel who conducted testing and those who have BC/DR
responsibilities
– Capture lessons learned and action items
• What worked? What didn’t?
• Were gaps identified during testing?
• What discrepancies were identified during testing?
– Assign action items and deadlines for completion, monitor progress.
• Completed action items feed back into the Analysis phase
39
BC PROGRAM - REVIEWS
Periodic Reviews
In accordance with the schedule specified in policies, procedures, and guidelines – restart the BC
Program:
• Review/revise the Business Impact Analysis (Analysis phase):
– Have the critical functions/systems changed? What is the impact?
– Have recovery timeframes increased or decreased?
• Review/revise policies, procedures, and guidelines (Development & Implementation phases)
– Have changes in the Business Impact Analysis affected BC/DR procedures?
• Review/revise BCP & DRP (Development & Implementation phases)
– Have recovery procedures changed?
– Have critical functions/systems changed? What are the recovery procedures?
– Have BC/DR personnel, vendors, contractors, etc. changed?
Ad Hoc Reviews
Same activities, but not scheduled. Predicated on audit findings, changes in threats, changes in
personnel, etc.
40
CONCLUSIONS
• Regulatory agencies require DR, but BC Program is necessary to fully protect your business
• A wide array of threats exist in the world, and without analyzing and preparing for these threats, your
business is unprepared to deal with them if they become a reality
• Everyone in the organization must recognize the value in protecting the business, from senior
management down
• A BC Program results in the creation of a BC Plan and a DR Plan which must be reviewed, tested,
and maintained in order to remain accurate, timely, and comprehensive
• The phases of the BC Program constitute a lifecycle approach that is a best practice for the creation,
implementation, testing, and maintenance of the BCP and DRP
41
QUESTIONS?
Please submit your questions in
the chat box.
42
www.facebook.com/perficient
www.perficient.com
www.twitter.com/perficient_LS
For more information, please contact:
[email protected] (Sales)
+1 877 654 0033 (U.S. Sales)
+1 303 570 8464 (U.S. Sales)
+44 (0) 1865 910200 (U.K. Sales)
THANK YOU
linkedin.com/company/perficient