Disaster Recovery

Steve BeardenThe Prism Partnership

What We Will Cover

2Three Threads of Data Privacy & Security

Why Disaster Recovery Planning

Prevention: Key Design Strategies

Preparation: Contingency Planning

Recovery: Putting It Back Together

Why Disaster Planning?A good Disaster Recovery Plan (DRP) can save guest experience, revenue and resources. An effective DRP will:

• Minimize potential economic loss• Reduce disruptions to operations• Ensure organizational stability• Provide an orderly recovery• Protect the assets of the organization• Ensure the safety of personnel and guests• Minimizing decision-making during a

disastrous event

Who’s Doing It Right?

In a survey conducted by Comdisco, Inc., one of the largest disaster recovery companies, it is estimated that:

• 45% of companies maintain a formal DRP

• 12% have an effective DRP

• 6% are partially prepared

• 82% are ineffectively prepared for a technology

disaster

What Goes Wrong?

Data Center DR Design Location – avoid water and public areas Security – proper access policies Climate – independent HVAC on generator Fire – dry pipes Power – clean power on UPS and Generator

Operations DR Design

Power to strategic locations is key – Front Desk, Restaurants, Housekeeping, PBX….

Outsource??

Customer Environment

Primary ISP Route

Redundant Applications

(Failover)

Sample Outsource Environment

Failover & Testing ISP Route

CustomerLocal Area

Network

Customer Equipment

Production Applications

Failover Data

Center

Primary Data

Center

Production DataRedundancy

The Plan

Every hotel has different characteristics:

• Systems• Location• Flag Programs

Therefore: Every plan should be property specific

Plan Documentation

Plan ownership – IT/Operations

The Plan should be documented in written form, with hard-copy printed versions readily available to hotel management in strategic locations

Know your processes and have forms & supplies available to operate manually

Detailed systems design – document make and model of equipment and software for easy restoration or remote site conversion

Know Your EquipmentNetwork

Property Management System

Website

SQL Database

Customer Facing

CentralReservations

Guest Network

In-RoomEntertainment

Door Locking

Package Tracking

Minibar

Parking

YieldManagement

CallAccounting

Sales System

ACD

Ticket Tracking

Bay Networks

SD

Ba

y N

etw

ork

s

Bay

Sta

ckA

cc

es

s P

oin

t65

0 W

irele

ss

CeridianPayroll

Fax Server

Optical Storage

Health Club

Readerboard

Internet Facing

Email

ADACO F&BInventory

VPN

802.11Wireless

SD

8000HP J4110A

ProCurve

Hotel Domain

Timekeeping

Timeclock

InfoGenesis(Internal ASP)

VoiceMail

TelephoneSwitch

Point Of Sale

Remote PC's

Fire

wal

l

Tape Backup

Oracle GLBack Office(Internal ASP)

CampaignManagement

(External ASP)

Firewall

Hotel Systems Configuration

(Q4 2006 Replacement)

(Q4 2006 Replacement)

EGuest PC’s

Hotel Reservations

Communications

Notification list

• Contingency Team – Escalation List

• Key management / Alternates

• Home / Cell

Emergency contacts

Sample letters from Management

Vendor Management

Vendor contact escalation list

Have vendor contracts available

Maintenance contracts / SLA’s

Ensure appropriate severity levels in your

contracts to meet contingency expectations

Alternate Location – corp offices, sister property

or “Hot site” hosted location?

Backup Data! Data Backup

• Tape or Optical• Off-Site Storage• Remote backup across internet to hosted backup

service?• Real time?

Save at least two recent generations of back-up (i.e., last night and last week)

Store in a fireproof safe in a remote location

Back-ups should be periodically verified

Test restores periodically

Backup Reports Run PMS contingency reports hourly Spool them to a Laptop Sample chronological control reports:

• All guests in-house, checked-out or expected today with balance

• Every room in the hotel with the current occupancy status

• Expected arrivals/departures for next five days

• A room availability report

PMS Reports

Contingency SuppliesContingency operations require paper forms and supplies

• Laptop w/contingency forms• Pre-printed Registration cards• Room status log• Reg card / Folio “buckets”• Credit card imprinters• CC Authorization terminal

• Generator / Measured business line• Calculators• Flashlights

Forms and supplies should be stored in a secured location with ready access when needed

The Outage When an outage occurs, the Manager On Duty

should first alert the Contingency Team

The contingency team leaders meet ASAP to determine type of outage

• Brief Outage < 6 Hours• Short term - > 6 hours < 2 days• Long Term > 2 days

Brief Outage < 6 Hours Contingency team will alert the management

team and affected departments Department management will call in more staff –

front desk, housekeeping coordinators, night audit, accounting

Distribute copies of the most recent contingency reports – (copier on emergency generator)

Distribute radios to key personnel, especially if telephone service is affected

Brief Outage < 6 Hours Distribute contingency supplies Centralize the room inventory function with a

single individual located at or near the Front Desk, essentially a “rack clerk” to assign rooms for arrivals and coordinate with Housekeeping

Update the Contingency Reports with all arrivals, departures, etc as they occur

As guests check-in, advise PBX to enable outbound calls

Centralize the revenue/F&B posting function with a single individual

Short term - > 6 hours < 2 days

Implement brief outage procedures plus: Letter from the General Manager to all guests Alert staff to come in early and plan to stay late Transcribe beginning balances from Contingency

Reports and all room charges to hard copy guest folios

Use pre-printed folio forms for contingency Operations

Maintain room status sheet from the contingency reports

Short term - > 6 hours < 2 days

Use a laptop (on generator) to maintain an Excel worksheet to facilitate changing statuses – FD / Housekeeping

Maintain constant communications with Housekeeping

Distribute updated telephone lists with arrivals and departures for operators

Prepare for a manual Night Audit

Post room and tax manually to the paper folios

Short term - > 6 hours < 2 days

Manually create housekeeping assignment sheets during the Night Audit for housekeepers

As checkouts occur, settle the balances manually Advise guests that the system is down and that a

copy of their folio will be sent to them after the system is restored

Be certain to capture addresses and email to honor this commitment

Each cashier needs to balance their cash and credit card receipts to their cashier’s report at the end of their shift

Long Term > 2 days

Implement brief & short term procedures plus:

Continue updating revenue and status forms in

laptop or on manual forms

Establish a room revenue control sheet in Excel

to capture room revenue as it is posted by hand

to the hard copy folios

Use this sheet, plus the room charge control

sheets to generate a basic revenue report

Long Term > 2 days Secure current availability and expected arrivals

reports from the CRO

Store all control sheets (Arrival/Departure;

Posting; Cashier Control; Room Revenue) in an

accordion file by day to facilitate recovery

processes

POS / RestaurantIf just PMS is down, settle checks to “House”

If POS down: Issue a small supply of checks to servers

recording the serial numbers Consider going to a limit menu or buffet Make CC Authorization terminal available

(generator / analog line) Remind kitchen personnel to make meals only

from the soft copies Use cashiers rather than allowing servers to self-

cashier

POS / Restaurant Adjust both staffing and guest expectations

accordingly Have cashiers maintain a manual control sheet

(cashier’s report) for all closed checks by meal period or shift

After each shift, reconcile the used checks with kitchen dupes and unused checks

Beverage outlets should draw a line under each drinks order as they make it

Bartenders record immediate sales over the bar onto a single control sheet

Alternatively, instruct bartenders to do a “blind drop”

Locking System

Maintain an inventory of contingency

key cards, often called “fail-safe keys”

• Keep secure but available

• 3 generations

• Log in and out

Telephone System Outage

Power fail bypass trunks

Analog telephones – patch panel?

Letter to guests

Post floor monitors

Interfaces Down

Check guests into PBX

In-room Entertainment – Enable Movies

Call Accounting – Post only $$ calls?

• Watch the buffer – old calls to new guests?

Voice mail or manual?

• If VM, check out guests for next guest

Recovery

Restore backup tapes

Input all sequential activity and charges

• Enter data from backup-forward

• check in/out - run night audits

• Post charges

Continue to operate manually until data restored

Remote Site (Yikes!)

If the hotel must be closed then you need to get your remote site operations underway.

• Set Up “War Room” for contingency communications

• Set up telephone center and network• Staff remote site• Reroute hotel/reservation call traffic• Get equipment ordered as needed• Install PC’s and distribute supplies• Set up and load most recent PMS backup

Test Test Test

Your DRPAnnually

At a minimum!

Questions?