disaster recover plan · web viewdisaster recovery plan 2020 submitted by: eqhealth solutions 8440...

27
Disaster Recovery Plan 2020 Submitted by: eQHealth Solutions 8440 Jefferson Highway, Suite 101 Baton Rouge, LA 70809

Upload: others

Post on 03-Feb-2021

4 views

Category:

Documents


0 download

TRANSCRIPT

Disaster Recover Plan

Disaster Recovery Plan

Disaster Recovery Plan

2020

In times of disaster, the business community looks to disaster recovery plans as business continuity tools. Basic elements include communication plans, risk assessment, business impact analysis, process plans, and offsite operations plans.

While our computer systems are well protected against catastrophic failure and business interruptions, these disaster recovery plans guide staff, protect data, enable speedy operational recovery, and minimize customer impact.

To facilitate the support role, the corporate Information Systems (IS) Team is trained in all aspects of the eQHealth Solutions disaster recovery plan. Cross training of personnel is a key part of our Information Systems Team’s mission, preparing staff to perform critical functions during times of crisis.

DISASTER Recovery Team

The Disaster Recovery Team is composed of the following eQHealth Solutions staff: Chief Executive Officer, Chief Administrative Officer, VP of IT Operations, Manager of Administrative Services, Director of Information Technology, Systems Administrator, and Chief of Staff. The eQHealth Solutions CEO is immediately notified of disaster declaration decisions.

DISASTER COMMUNICATIONS

The Disaster Recovery Team declares the disaster and initiates the staff-calling tree as appropriate. Members of the disaster recovery team are responsible for their areas’ roles and responsibilities during the declared disaster, leading to the successful conduct of business. The implementation of the disaster recovery plan results from staff participation in the implementation of the disaster recovery plan.

The impact to operations of a disaster is classified as minor, major and catastrophic. For each disaster, the impact to operations is determined through the Recovery Time Objectives (RTO) for these disasters. Multiple RTO take into consideration the role of external organizations, the severity of a disaster in the conduct of business service levels, and the continuity of operations.

When the Disaster Recovery Team institutes the staff-calling tree, eQHealth Solutions Management directs staff where and when they need to report. All managers have a list of current staff home telephone numbers. Information Systems advises the Disaster Recovery Team related to the following risks: computer viruses, network security, WAN outages, telephone outages, and equipment breakdown.

The Calling Tree works as follows:

eQHealth Solutions Chief Executive Officer, or designee, contacts the Centers for Medicare and Medicaid Services. The Chief Executive Officer also calls the Project Manager, Office Administrator, Education Manager and Information Systems Manager. The Office Administrator contacts respective staff. All Managers are responsible for contacting their respective staff. In addition, Information Systems Manager contacts Director Information Systems and System Administrator.

RISK ASSESSMENT

Disasters are classified as minor, major and catastrophic. Classifications are determined by evaluating the likelihood of an event and its potential impact. A minor disaster can disrupt customer service for up to 24 hours. A major disaster can disrupt customer service for 24 – 48 hours. A catastrophic event can disrupt customer service for a period greater than 48 hours. If an event is possibly a minor, major and catastrophic event, the event is categorized as “All”.

Risk Category

Risk Factor Event

Impact to Business Operations

Impact to Facility or Equipment

Recovery Time Objective

Risk Probability of Event Occurring

Weather

Tornado

Catastrophic

Catastrophic

Catastrophic

Minor

Weather

Hurricane

Catastrophic

Catastrophic

Catastrophic

Minor

Weather

Flood

Major

Minor - assuming no damage to voice/data/fax circuits

All

Minor

Weather

Snow

Minor

Minor

Minor

Minor

Biological

Pandemic

Minor

Not Applicable

Minor

Minor

External Destructive Forces

Terrorist Attack

Minor

Minor

All

Minor

Physical Plant

Facility location

Minor

Minor

Minor

Minor

Physical Plant

Power Outage

Major

Minor

Minor

Minor

Information Systems

Computer Virus Attack

Minor

Not applicable

Minor

Minor

Information Systems

Loss of Operation of WAN

Not Applicable

Not Applicable

Minor

Minor

Information Systems

Telephone Outage

Major

Not Applicable

Minor

All

Information Systems

Computer/Printer Breakdown

Major

Not Applicable

Minor

Minor

Security

Facility Security

Major

Major

Minor

Minor

Security

Access to Network and Computer Resources

Minor

Not Applicable

Minor

Minor

BUSINESS IMPACT ANALYSIS

Weather

· Tornado

· Central Louisiana averages approximately 24 storms per year with confirmed tornado development. (http://www.ncdc.noaa.gov). If a tornado directly hit the building and the facility is damaged, all information services are redirected to the Illinois location until the local facilities are restored.

· For Recovery Time Objective (RTO) – Less than 2 hours.

· For Recovery Point Objective (RPO) – No review data will be lost, since all data is being replicated to the SAN located in the Illinois data center. Applicable HITRUST / HIPAA requirements are maintained

· Hurricane

· Hurricanes can directly affect the current physical location. With current weather monitoring techniques and experience, the Baton Rouge office is notified in advance – transitioning necessary data operations till the weather subsides.

· For Recovery Time Objective (RTO) – Less than 2 hours.

· For Recovery Point Objective (RPO) – No review data will be lost, since all data is being replicated to the SAN located in the Illinois data center. Applicable HITRUST / HIPAA requirements are maintained

· Flood

· eQHealth Solutions office might be affected by a water main break or natural flooding. All the servers are physically located in a secure data center located in Baton Rouge which has been designed against the threat of floods. All computers and printers are at least six (6) inches off the floor. All services are redirected to another corporate office until the local facilities are restored

· For Recovery Time Objective (RTO) – Less than 2 hours.

· For Recovery Point Objective (RPO) – No review data will be lost, since all data is being replicated to the SAN located in the Illinois data center. Applicable HITRUST / HIPAA requirements are maintained

· Snow/Ice

· Since the facility is located on a major street, staff commute times are possibly increased from a significant snowfall or road icing, resulting in a potential late opening.

· RTO/RPO - no impact to operations. Applicable HITRUST / HIPAA requirements are maintained

Biological

· Pandemic

· The overall likelihood of a declared pandemic is rare. However, eQHealth Solutions has taken steps to insure that employees have the ability to perform their daily functions from the personal residence.

· RTO/RPO - no impact to operations. Applicable HITRUST / HIPAA requirements are maintained

External Destructive Forces

· Fire

· The building has a fire alarm system and fire sprinkler system installed throughout the building. The risk of fire affecting operations depends on when a fire might break out, where it might breakout, and how long it might go undetected. Operationally, the following procedures help in reducing the possibility and/or potential impact of fire:

· All computer equipment has been purchased new.

· Additional electrical equipment must be approved.

· All electrical equipment is plugged into surge protectors or uninterruptible power supplies.

· Several fire extinguishers are strategically located and available at eQHealth Solutions exits, major office space areas and the data center.

· A fire sprinkler system is installed throughout the office.

· The office suites are located on the first and second floors of the building. All offices and conference rooms have doors, telephones, and (with one exception) windows. eQHealth Solutions offices are located near one of two main building stairwells that exit in close proximity to the building’s three main exits.

· Operational impact from fire would be based on location:

· Data center fire – The data center utilizes both smoke and heat detection sensors throughout the facility along with early detection devices to eliminate the threat of a fire. The facility is equipped with a dry-fire suppression system, fire doors as well as handheld gas-based fire extinguishers to quickly contain and prevent the outbreak of a fire. The data center features a completely redundant network infrastructure with multiple data circuits from different providers. The degree of damage might require redirecting services to the Illinois data center. As soon as replacement servers are received, Information Systems staff will restore the servers from the full daily snapshots and archive logs. The only review data loss would be from those reviews that are in process. Email, network security IDs and file storage would be available through the Illinois data center. The only loss would be from the current day’s work. Applicable HITRUST / HIPAA requirements are maintained

· Within the staff work area, a potential fire would directly affect user workstations (computers and telephones). Since all computers have a standard software build, we could immediately relocate users to other machines and prioritize workflow needs with available equipment. Applicable HITRUST / HIPAA requirements are maintained.

· Whole building destroyed by fire - The toll-free lines would be immediately redirected to the Illinois office. Maximum down time is approximately one (1) hour to discover the issue and redirect telephone lines. The major recovery areas from a fire are paper records (work-in-progress), computer equipment, telephones, fax lines, printers and basic office supplies. Applicable HITRUST / HIPAA requirements are maintained

· Terrorist Attack

· Targets of terrorist attack could include physical damage to the capitol city’s infrastructure. The Baton Rouge Metro Airport is approximately twenty (20) miles away and would not directly affect business operations. Services would be redirected to another corporate office location until full impact of the situation is determined. Applicable HITRUST / HIPAA requirements are maintained

· RTO is less than 2 hours.

· RPO – no data loss

· Other areas of attack could include the Riverbend Nuclear Power Plant located approximately 38 miles away in St. Francisville, LA and the Exxon Refinery located approximately 15 miles away in Baton Rouge, LA. Damage to either of these locations could significantly impact operations depending on the extent of the damage. Services would be redirected to another corporate office location until full impact of the situation is determined. Applicable HITRUST / HIPAA requirements are maintained

· RTO is less than 2 hours.

· RPO – no data loss.

· Damage to the city’s water supply would require bottled water for staff usage, which eQHealth Solutions has readily available. Staff would not be able to use drinking fountains or the ice machine.

· RTO is less than 2 hours.

· RPO – no data loss.

· Chemical attacks would probably be targeted to areas with high concentrations of population gatherings i.e. downtown Baton Rouge, Louisiana State University, sports venues, or public festivities. eQHealth Solutions is located near the business center of United Plaza and an attack on the business center would severely impact or cripple operations. Files and data systems would be made available through VPN tunnels over the Internet. Applicable HITRUST / HIPAA requirements are maintained

· RTO is less than 2 hours.

· RPO – no data loss.

Physical Plant

· Facility Location

· The facility is located less than (1) mile from I-10 and I-12, and four (4) miles from Airline Highway. These roadways provide major access points for eQHealth Solutions staff, the destruction of these intersections could potentially result in a potential late opening.

· RTO/RPO - no impact to operations.

· Operations are also impacted by damage to city/communication services, i.e., water, power, telephone lines and T1 connectivity. In general, these types of outages are less than 24 hours.

· RTO less than 2 hours – Services will be redirected to the Illinois office until local facilities are restored. Applicable HITRUST / HIPAA requirements are maintained

· RPO – no data loss.

· Power Outage/Surge

· Utility supplied power is interrupted. The data center is engineered with uninterruptible power system (UPS), and backup generators to deliver seamless power. In the event of a commercial power failure, the UPS will provide immediate backup power until the generators take over the load. The telephone system vendor to redirect calls to another corporate office if needed. Staff workstations are immediately shut down by a power outage. If the projected power downtime is estimated to exceed 2 hours the impacted site will cutover systems to an alternate backup site. Applicable HITRUST / HIPAA requirements are maintained

· RTO is less than 2 hours.

· RPO – no data loss.

· When utility power is restored, equipment is protected from a power surge. We have surge protectors on all staff computer equipment and printers, in addition to all equipment located in the data center.

· RTO/RPO – no impact to operations.

Information Systems

· Computer Virus Attack

· Carbon Black and Palo Alto Networks firewalls serve as our protection to computer virus attacks. The Carbon Black server is updated hourly with the most recent virus protection files. Carbon Black has a mechanism for preventing all unauthorized applications from running on the computer which provides a higher level of protection. Supervisor approval is required for staff access to the Internet. The Internet and Email Policies are distributed to staff. During a computer virus attack, all Internet access and email are immediately shutdown until the virus is removed from the network. Telephonic reviews are continued.

· RTO – no impact on operations.

· RPO – no data loss.

· Loss of Operation of WAN – Wide Area Network

· No impact on operations.

· Service Level Agreement with carrier is 4-hour turnover on repairs

· Telephone/Fax Outage

· Telephone lines - The telephone vendor currently provides the failsafe routing system on a fully redundant nationwide network. If at any time the telephone lines (toll free numbers) do not return a signal to the vendor’s main switch, the lines will be re-routed to another switch to in the vendor’s network ensure high availability. eQHealth Solutions maintains two independent circuits from different providers both through different avenues reducing the chance of both being out simultaneously. The vendor operates a redundant Network Operations Center (NOC) with 24 hours a day 7 days a week monitoring of all circuits to ensure reliability

· RTO – Less than 2 hours.

· RPO – No data loss.

· Telephone System – The telephone vendor provides redundant systems which are monitored 24 hours a day 7 days a week by their NOC to ensure no downtime.

· RTO – Less than 2 hours.

· RPO – No data loss.

· For a fax outage, all inbound and outbound faxes are queued until connectivity has been restored to the cloud-based service. The fax vendor maintains multiple data center locations for redundancy which reduces the risk of an outage. The Help line is updated with the current status in order to assist providers.

· RTO – less than 2 hours.

· RPO – No data loss

· Computer/Printer Breakdown

· Computers – We have the HP 3-year maintenance agreement on all workstations. We maintain extra computers to provide an immediate replacement for any failed user workstation.

· RTO/RPO – no impact to operations.

· All Xerox printers have backup printers to redirect production output. eQHealth Solutions maintains a service contract for all of our Xerox copier/printers.

· RTO/RPO – no impact to operations.

Security

· Facility Security

· All staff is issued security key card to access the building. With all entrances locked, the only options for entry are staffs with a key card or guests that are granted access by the receptionist. Upon entry, all staff and guests are required to sign-in to log books. All guests are escorted by a staff member while in the building.

· All staff are required to sign-off their computer each night. Information System policies force compliance.

· Data center which hosts eQHealth Solutions server infrastructure is engineered with a minimum of five levels of security to provide the utmost level of protection. Proximity cards with personal identification number, biometric fingerprint readers, hardware is secured in a locked cage fitted with a combination lock, video surveillance cameras monitored 24x7x365 and motion/vibration detectors

· Access to Network and Computer Resources

· On their first day as an eQHealth Solutions employee, staff receives a network ID and a security badge issued by the Human Resources and Information Systems departments. Permissions for network IDs are based on Windows Active Directory security. Each account is given access levels directly related to job functions using the least privilege rule.

· Upon termination of employment with eQHealth Solutions, the network IDs are disabled and badges are returned and destroyed.

PROCESS PLANS

· Voice and Fax Circuit Issues

eQHealth Solutions operations depends upon our telephone system for conducting business. The telephone vendor currently provides the fail-safe routing system on a fully redundant nationwide network. If at any time the telephone lines (toll free numbers) do not return a signal to the vendor’s main switch, the lines will be re-routed to another switch to in the vendor’s network ensure high availability. eQHealth Solutions maintains two independent circuits from different providers both through different avenues reducing the chance of both being out simultaneously. The vendor operates a redundant Network Operations Center (NOC) with 24 hours a day 7 days a week monitoring of all circuits to ensure reliability.

For a fax outage, all inbound and outbound faxes are queued until connectivity has been restored to the cloud-based service. The fax vendor maintains multiple data center locations for redundancy which reduces the risk of an outage.

· Data Circuit Issues

Wide Area Network Circuit: eQHealth Solutions utilizes a private IP switched fully meshed national fiber network for the Wide Area Network connection. The service level agreement for the WAN circuit has a 4-hour turnaround on outage repairs. This redundancy reduces the possibility of corporate operations or critical data replication from going down for any length of time.

Internet Circuit: eQHealth Solutions utilizes multiple circuits on a switched fully meshed national fiber network for the Internet connection. The service level agreement for the WAN circuit has a 4-hour turnaround on outage repairs.

· Telephone Equipment Maintenance

eQHealth Solutions utilizes a hosted telephone system for our voice communication and voice recording. The telephone vender maintains the system and provides all upgrades as part of the service.

· Server(s) Maintenance and Equipment Contingency Plans

Storage Area Network: eQHealth Solutions utilizes two Nimble Storage Area Network (SAN) for all mission critical data which offers the latest in remote replication technology to insure the highest level of availability and business continuity. The SAN is fully redundant which has dual controllers, dual power supplies, and multiple fiber channel routes and uses triple-parity RAID to insure system uptime. If one component of the SAN unit fails, the redundant component picks up the load with no system downtime until a replacement part can be installed. If the primary SAN fails, services can be redirected to the backup SAN. If both storage area network units fail, services are redirected to the Illinois data center. Installation of the failed part can be installed HOT, meaning there is also no system downtime for repair. We maintain a service contract on the SAN unit with a 4-hour response time.

File, Database and Email Server: eQHealth Solutions utilizes Cisco UCS equipment for our server infrastructure. We maintain a maintenance agreement with Cisco for a 4-hour response time on all of our servers.

eQHealth Solutions uses the latest in virtualization technology, VMware vSphere for our virtualization platform. The virtual infrastructure is configured in a clustered mode meaning the platform is hosted on multiple servers. This allows for the high availability for all mission critical servers. If one of the servers of the virtual platform fails the servers can be moved to the other platform with no system downtime. Another key component is the use of a dynamic resource pool to ensure that all mission critical servers will always have sufficient memory and processors to operate with maximum efficiency.

Potential problems include disk drive failure, damaged files, and server failure.

· If a disk drive failure occurs, eQHealth Solutions protects operations from loss of data by using Nimble triple-parity RAID configuration with a no single point of failure Storage Area Network unit. In the event of a drive failure, the system automatically accesses the failover drives without system down time. The damaged drive is replaced HOT within 4 hours.

· Complete snapshots of the data on the volumes of these servers are created several times a day to facilitate retrieval of damaged files.

· eQHealth Solutions incorporates the latest in data replication technology which allows for asynchronous replication of mission critical data to a remote location.

· If the file, database or email server(s) goes down completely, a virtual instance or copy of the server can be put into production with little downtime. If one of the VMware host platforms, on which the file, database and email server resides on fails, the servers are automatically moved to a redundant host until repairs are made. If the entire host platform completely fails, backup instances of the servers can be brought online in the Illinois data center which maintains a current copy of the data. The servers in Illinois data center stay in active mode until the unit host platforms are repaired.

Network Switch: If a switch fails, all network traffic will automatically route over a redundant switch until the failed unit is repaired.

· Workstation Maintenance and Contingency Plans

eQHealth Solutions uses HP workstations with a three-year warranty period. HP’s maintenance agreement provides for next business day response on workstations. eQHealth Solutions budgets several extra workstations to be used as immediate replacements. If a user’s workstation, monitor, or mouse fails, it will be swapped with a replacement while awaiting the repair. Our complete inventory includes HP Service tags. All HP Service Tags have complete information on the machine configuration sold to eQHealth Solutions in order to facilitate repair service.

· Printer Maintenance and Contingency Plans

eQHealth Solutions uses Xerox printers for the day to day operations and maintains service contracts with Xerox.

· Database Issues Maintenance and Contingency Plans

eQHealth Solutions uses Microsoft SQL 2016, the industry standard relational database software. eQHealth Solutions maintains a universal subscription through Microsoft which includes support for immediate assistance with database issues.

If the database is down due to a SQL problem, the Database Administrator will work with Microsoft until the issue is resolved. If extended downtime is anticipated, database operations will be transferred to the failover server until the issue can be resolved.

Backups: eQHealth Solutions performs hourly automated system snapshots of the database. We maintain a one week set of snapshots. SQL transaction log files are also enabled and replicated to the Illinois location for full redundancy. If eQHealth Solutions experiences any data corruption, data will be reloaded from a snapshot residing on the SAN and log file transactions will be applied to the database where possible. By adhering to a weekly rotation, and replicating critical database information, eQHealth Solutions ensures systems can either be re-directed to other servers if necessary or restored to functionality in the least amount of time possible.

eQHealth Solutions utilizes the latest in data replication technology. All data from critical systems resides on the SAN. The data is constantly monitored for changes. Any changes that occur are then replicated to a SAN in the Illinois data center asynchronously

INFORMATION SYSTEMS MAJOR RESPONSIBILITIES

The following table highlights the major responsibilities of the Information Systems Team in supporting the Louisiana contract requirements.

Information Systems Tasks

Lead Accountability - Location

Cross-Trained Backup Personnel

Database Mgmt

Database Administrator - Louisiana

Director of IT Ops – Illinois

Systems Engineer – Louisiana

User Support

System Support Specialist – Louisiana #1

System Support Specialist – Louisiana #2

Corp. Systems Support Manager – Mississippi

System Backups and Replication

Systems Engineer – Illinois

Director of IT Ops – Illinois

Systems Engineer – Louisiana

Hardware Troubleshooting

Systems Engineer - Louisiana

Director of IT Ops – Illinois

Systems Engineer – Illinois

Hardware Installs and Fixes

Systems Engineer - Louisiana

Director of IT Ops – Illinois

Systems Engineer – Illinois

System Security and Access

Systems Engineer – Illinois

Director of IT Ops – Illinois

Systems Engineer - Louisiana

Technical Engineering Design

Director of IT Ops – Illinois

CIO – Louisiana

VP of IT Ops - Louisiana

Internet Services Admin and Security

Director of IT Ops – Illinois

Director of IT Ops – Illinois

Systems Engineer - Louisiana

Web Server Administration

Systems Engineer - Louisiana

Director of IT Ops – Illinois

Systems Engineer – Illinois

Information Systems Components and Recovery Plans Table

IS COMPONENTS AND RECOVERY PLANS

Section 1Infrastructure

Section 2

Database Server

Section 3

Server Backups and Replication

Section 4

File and Application Servers

Section 5

Exchange Email Server

Section 6

Storage Area Network

Section 7

User Access

Section 8

Web Server

Section 9

Data Circuits

Section 10

Printers and Switches

Section 11

Firewall

Section 12

Anti-Virus Server

Summary of Information Systems Components and Recovery PlansInfrastructure All eQHealth Solutions servers and workstations were purchased new for the contract. For server hardware failures, the company has Cisco UCS servers with the 4-hour replacement warranty. For workstation hardware failures, eQHealth Solutions maintains an inventory of extra workstations to immediately replace any failing workstations. HP is contacted for service on any defective workstation.

For SAN component, eQHealth Solutions has service agreements with the vendors for 4-hour replacement warranty.

For other equipment hardware failures, eQHealth Solutions maintains backup printers, switches, firewall, router, and telephones for immediate replacement of the failed equipment. Vendors are contacted for servicing or replacing the defective equipment.

eQHealth Solutions has developed a recovery procedure for each Information Systems Component. A copy of the Disaster Recovery Plan, in addition to the recovery procedure documents is located in the Information Systems Departments at Florida, Louisiana, Illinois, and Mississippi.

Database Server

Recovery Methodology

Physical Server:

a) Hardware Failure – The virtual instance of the server moved to backup host platform with little to no downtime.

b) File corruption – Restore and recover from previous backup\snapshot.

c) Catastrophic Loss (including Fire or Weather damage) – Bring database services up on backup server in the Illinois office and connect to the replicated database and apply log file transactions where possible.

SQL Database:

a) Data Corruption – Restore and recover data from previous hourly snapshot and apply log file transactions where possible.

Server Backups and Replication

Data Backup:

a) All mission-critical data is backed up hourly. Mission-essential data is backed up daily.

b) Hourly asynchronous replication of mission critical data. Nightly asynchronous replication of mission essential data.

c) Backups and replication are verified to ensure the process has been completed successfully. If they did not, a manual backup will be done at that time.

Server Backup:

a) All servers are backed up daily.

b) Server backups are asynchronously replicated to the IL data center weekly.

c) Backups and replication are verified to ensure the process has been completed successfully. If they did not, a manual backup will be done at that time.

File, Application and WebServersRecovery Methodology

a) Hardware Failure – The virtual instance of the server moved to backup host platform with little to no downtime.

b) File corruption – Restore and recover from previous backup\snapshot.

c) Catastrophic loss (including Fire or Weather damage) – Redirect services to a replicated server in the Illinois data center.

Exchange Email Server

Recovery Methodology

a) Hardware Failure – The Exchange environment is on a hosted platform which the vendor maintains in multiple data centers. No downtime

b) File corruption – The vendor maintains the Exchange platform.

c) Catastrophic Loss (including Fire or Weather damage) – The vendor is configured for high availability and all data is replicated to multiple data centers in their environment.

Storage Area Network

Recovery Methodology

Physical Server:

a) Hardware Failure – The Storage Area Network has dual controllers, dual power supplies, and multiple fiber channel routes and uses triple-parity RAID to insure system uptime. If one component of the SAN unit fails, the sister component picks up the load with no system downtime until a replacement part can be installed. Installation of the failed part can be installs HOT, meaning there is also no system downtime for repair.

b) File corruption – Restore and recover from previous backup\snapshot.

c) Catastrophic Loss (including Fire or Weather damage) – All mission critical and mission essential data is asynchronously replicated to the Illinois data center.

Data:

a) Data Corruption – Restore and recover data from previous snapshot.

User Access

Recovery Methodology

a) File Corruption – Transfer the Active Directory roles to a backup domain controller until the failed domain controller can be repaired.

b) Hardware Failure or Catastrophic Loss - Transfer the Active Directory roles to a backup domain controller until the failed domain controller can be restored from a snapshot.

Data Circuits

Recovery Methodology

a) Hardware Failure for Data Circuits – Internet service at the data center consists of a multi-carrier redundant configuration with no single point of failure. If the circuits are down for greater than 4-hours, the web services are redirected to the Illinois office until the circuits are restored.

b) Catastrophic Loss (including Fire or Weather damage) – Services are redirected to the Illinois office until services are restored.

Printers and Switches

Recovery Methodology

a) Printer Failure – Redirect to an alternate printer. Contact Xerox for repair.

b) Switch Failure - If a switch fails, all network traffic will automatically route over a redundant switch until the failed unit is repaired.

Firewall Recovery Methodology

a) Hardware Failure – Traffic will automatically be redirected through the redundant firewall. Repair or replace defective unit

b) Catastrophic Loss (including Fire or Weather damage) – Redirect services to the Illinois location.

Antivirus ServerRecovery Methodology

a) Hardware Failure – The virtual instance of the server moved to backup host platform with little to no downtime.

b) File corruption – Restore and recover from previous backup\snapshot.

c) Catastrophic loss (including Fire or Weather damage) – Redirect services to a replicated server in the Illinois office.

Disaster Recovery Plan Reviews and Updates

eQHealth Solutions will reevaluate the Disaster Recovery Plan annually to verify that the following targets have been tested. After Disaster Recovery Plan updates, copies of the Plan are sent to the Louisiana, Illinois and Mississippi offices.

Disaster Recovery Plan Testing Matrix

Test Target

Testing Process

Frequency

SQL Database Server

Bring a virtual copy of the server online, in an isolated environment, to verify access to database.

Daily

Data Snapshots and Replication

All mission critical data is being replicated to the SAN in the Illinois data center. In addition, snapshots of the same data are taken frequently throughout the day.

Business days check error log for process issues.

Server Recovery

Bring a virtual copy of the server online, in an isolated environment, to verify access.

Quarterly

Web Server

Tested daily by staff and providers. Troubleshoot all connectivity issues.

Business days check error log for process issues.

Data circuits

Vendor automated backup processes.

Designed as part of vendor relationship.

Antivirus Software

Carbon Black servers are updated.

Check alerts as they are generated.

Security – Network

Process set up to activate new employees, and deactivate terminated employees.

Process followed as employee statuses changed.

Update Workstation Images

Create image with new system updates and test functionality.

Within 6 months of major system updates, at a minimum test annually.

Equipment inventory

Verify the following replacement equipment is available: printer, network switch, workstation, and laptop.

Add equipment as needed. Complete formal inventory for DR Plan Annual Review.

Temporary Office Space Vendors

Contact to validate features and availability.

Annually

Verify, Test, and Update DR Plan

Review plan for validity. Gather test data. Update as needed and redistribute to Louisiana, Illinois and Mississippi offices.

Annually (or sooner if major infrastructure changes occur)

VoIP SLA Review

Review provider contingency plans to ensure that the plans meet organizational contingency requirements

Annually

Review/Revision History:

Wayne Wallace – Corp. IT Director. – 5/6/2016

Eric Gorczynski – Corp. Infrastructure Manager. – 6/20/2017

Wayne Wallace – Corp. IT Director – 6/20/2017

Eric Gorczynski – Corp. Infrastructure Manager. – 8/9/2018

Eric Gorczynski – Corp. Infrastructure Manager. – 8/30/2018

Eric Gorczynski – Director of IT Operations – 9/4/2019

Eric Gorczynski – Director of IT Operations – 3/2/2020

Allen Dixon – Service Delivery Manager – 3/16/2020

2