business continuity planning and execution - … planning for pension funds lessons...business...

18
Business Continuity Planning and Execution The Teachers’ Retirement System of the City of New York and Hurricane Sandy Presented by: Patricia Reilly Deputy Executive Director Teachers’ Retirement System of the City of New York

Upload: phamduong

Post on 06-Jul-2018

214 views

Category:

Documents


0 download

TRANSCRIPT

Business Continuity Planning and

Execution

The Teachers’ Retirement System of the City of New York

and Hurricane Sandy

Presented by: Patricia Reilly Deputy Executive Director Teachers’ Retirement System of the City of New York

Overview

• What We Planned • What Happened • What Worked / What Didn’t Work • Lessons Learned

TRS’ Experience on 9/11

• Temporary Office Space (cold site): Shared 50 Seats • TRS Declares Disaster • Cold Site Could Not Accommodate TRS

Lesson Learned: Need Guaranteed Disaster Recovery Site, NOT Shared Site

New Mandate from Executive Director: Develop New Business Continuity Plan

Onsite Production Data Center

Mirrored Offsite

TRS Business Continuity Plan Components

Dedicated Mobile Office

Secure Storage Facility for Critical Equipment & Supplies

Documented & Rehearsed Continuity Plan

Prior to Hurricane • In the month before the Hurricane, TRS Lost 3 Senior Staff

• 2 Senior Staff on Vacation

• Negotiations with Vendor for High-Volume Printing

• Check Printer from Iron Mountain in TRS Office

• TRS Email Functionality Not Redundant

• City Evacuated Zone A and Executed Controlled Citywide Transit Shutdown

Benefits Admin Deputy Director Member Services Director

CIO Executive Director Benefits Administration Director

NYC Hurricane Evacuation Zones

• 3 ½ Feet of Water in Lobby • 6 Levels Flooded Below Ground • Building Infrastructure Services Compromised

• Power, Telecom, Water, Sewage, HVAC • 71 Elevators in Building, Inoperable

• Some Damage to Walk-In Center Entrance • No Damage to 16th or 17th Floor Administrative

Offices • We Have No Access to Building

55 Water Street After the Hurricane

New York Metropolitan Area After the Hurricane

– Tragic Loss of Life – More than 1.5M Lose Power in Tri-state – Gas Shortages – Damage to Homes, Loss of Landlines, Internet – Public Transit Severely Limited – Cell Towers Powered by Battery Backup/Service

Disruption • TRS Staff

– At Home Dealing with Their Own Problems – Displaced

• TRS Members – May have Lost Their Home – Need Loans and Hardship Withdrawals

M 10/29 T 10/30 W 10/31 Th 11/1 F 11/2 Sa 11/3 Su 11/4

HURRICANE HITS

•Disaster Declared/Arranged for Delivery of Mobile Office

•Member Website & Phone Message Updated

•Daily Departmental Conference Calls Initiated

•NYC Employee Retirement System Provides Office Space in Queens

•Office of the Actuary Provides Space

•Board of Education Retirement System Provides Space for Walk-In Center

• Start Setup at Queens Facility for Call Center

•HR Accesses Employee Payroll via Space at Other City Agencies

•TRS IT Performs Mobile Office Preparations -Workstations -Printers -Fax -Communications

• Still No Dial Tone at Queens Location

•No Power and Gas

•Team Conference Calls Begin

•Call Tree Triggered – Outreach to Staff

•Employee Website & Emergency Phone Message Updated

•Mobile Office Arrives in Manhasset, Long Island

•Phones Are Down at Queens Location

What Happened – First Week

NO COMMUNICATION WITH 55 WATER ST.

NO TRS EMAIL – Personal Email Addresses Shared CONTINUING SIGNIFICANT TRANSIT ISSUES

NO COMMUNICATION WITH NYC GOV’T

SNOWSTORM

M 11/5 T 11/6 W 11/7 Th 11/8 F 11/9 Sa 11/10 Su 11/11

What Happened – Second Week • Mobile Office Site Up & Running • Chairman of the TRS Board, Mel

Aaronson, Personally Visits Staff in Trailer

• Only 4 Dedicated Call Center Phones Available at Trailer

• Website Configured for Member Inquiries via Email

• 1st Executive Team Face-to-Face Meeting

• Phones Finally Working at Queens Location

• Setup for Call Center Complete • Still no Access to Computer

System at Queens Location

• Additional Call Center Staff Deployed to Queens Location

• Member Data Requests Relayed to Mobile Office – Research Results Relayed Back

• Retirements Unit Starts Working from Actuary’s Office

Lack of Check Printer

EFT Campaign

Non-Critical Staff Assignments

• Walk-In Center Opens in BERS Offices

• Word on the Street: LONG Absence from 55 Water

• Expansion Planning at Queens Location: Equipment & Furniture Ordered

Business (Continuity) as Usual

Mobile Office • Retirement Payroll • A/R • General Accounting • Loans • Member Accounting

• TDA • Death Benefits • Member Service • IT Operations/Support • Help Desk

Office of the Actuary Retirements/Withdrawals

BERS Walk-In Center

Work at Home Non-Critical Staff

Queens Site • Call Center • All Other Units

X

November 2012

• Access Granted to 55 Water to Retrieve Essential Items

• Board and Investment Meetings Held at NYCERS Offices

• IRS Relaxes Hardship Withdrawal Rules for Hurricane Sandy Victims

• Cold Weather Begins to Take Hold • Police Pay a Visit to Mobile Agility Site

December 2012

• Business Continuity Fatigue Sets In • Non-critical Business Processes Become

Critical as Time Progresses • Backlogs Generate Increasing Call Volume • Building Sends Re-Occupy Notice • Business Resumption Planning Begins • Fire in the Building – Re-Occupation Delayed • It’s Getting Colder!

Business Resumption

Working to Ensure Adequate Building Services

Working to Assure Safety of Premises for Staff

Back to “Normal” by Early February

What Worked What Didn’t Work • Documented & Rehearsed

Business Continuity Procedures

• Communication • With Staff • With Members

• Mobile Office

• Retrieving Continuity Materials from Iron Mountain

• Technical Setup for Working from Business Continuity Sites

• Non-Critical Staff Not Aware of Business Continuity Plan

• Communication • With 55 Water St. • With NYC Government

• Mobile Office Alone

• Incomplete Set of Continuity Materials at Iron Mountain

• Technical Setup for Working from Home

What Worked What Didn’t Work • In-Process System Upgrades

(Email, Data Tools)

• Mobile Office Mail Room • Size • 2 Months’ Mail Volume

• Lack of Generic Business Resumption Planning Checklist

• Commute was a Hardship

• No Pre-Disaster Exit Plan

• System Redundancy

• Redirecting Incoming Mail to Mobile Office

• Reactive Business Resumption Planning

• Staff Able to Get to Business Continuity Locations

Lesson Learned

BE FLEXIBLE! Business Continuity Management Team

Must Be Able to Adapt and Respond