financial services technology consortium incident management an exchange of practices and...

FINANCIAL SERVICES TECHNOLOGY CONSORTIUM

www.aesrm.org

Incident ManagementAn Exchange of Practices and

Experiences

2008 Annual Meeting – Sonoma California June 18, 20088:15 to 10:15

Andrew McCruden, CitigroupRandall Till, MasterCard Worldwide

2


Agenda

I. Opening commentsII. Methodologies for managing incidentsIII. Building and managing external relationshipsIV. Conducting effective exercises of incident mgmt plansV. Communication strategies and case studies/experiences VI. Wrap-up


www.aesrm.org

Methodologies for Managing Incidents

Incident Command Systems (ICS)

4


Why Incident Command System (ICS)?

• Global events (e.g. Pandemic)• Promote emergency management plan• Management awareness• Reputation and shareholder value• US Presidential Directive (PD #5) - mandatory

for:– A US federal agencies for federal funding– US State governments– All hazardous material incidents– US law enforcement, government, and the military

5


What is Incident Command System (ICS)?

• It is modular and scalable – only use teams that you need• Provides for consistent and reliable communications using

common terminology• Ensures coordinated response among teams and locations

(horizontal & vertical)– Especially helpful for companies with multiple locations

• Employs standard and proven practices

ICS is a well organized team approach for managing critical incidents

Source: Emergency Management & Safety Solutions

6


ICS Organizational Structure

Command(manages)

Operations(does)

Logistics(care/gets)

Planning & Intelligence

(plans)

Financial(pays/records)

7


ICS Team Types

IAT = Initial Assessment Team

• Team for Small Regional Offices and sub-team of the CIRT/LIRT

LIRT = Local Incident Response Team

• Regional Headquarters and Select Offices

CIRT = Corporate Incident Response Team

• Corporate Headquarters ONLY

8


IAT Members

Initial Assessment Team (IAT)City name

Operations Planning &Intelligence Logistics

Facilities/Real Estate

Technology/Operations

Security

Financial

Last Revised

Group LeadAlternate

PrimaryAlternate

PrimaryAlternate

PrimaryAlternate

CommanderAlternate

Group LeadAlternate

Group LeadAlternate

Group LeadAlternate

9


CIRT/LIRT Members

Corporate Incident Response Team (CIRT)

Operations Planning &Intelligence Logistics

Facilities/Real Estate

Security

Technology/Operations

Key Linesof Business

Communications

HR BusinessPartner

Travel

Purchasing

InvestorRelations

Information Security Technical

Recovery

Business Recovery

Financial

BusinessContinuity

Legal/Regulatory

Meetings

Benefits

Global Finance

Accounting/Accounts Payable

Insurance

Payroll

= Initial Assessment Team= Initial Assessment Team

10


ICS Structure (example)

Asia PacificLIRT

Country IAT

Country IAT

Country IAT

Country IAT

Country IAT

Country IAT

Corporate HQCIRT

Middle East &Africa LIRT

Country IAT

Country IAT

Country IAT

Country IAT

Country IAT

CountryIAT

Country IAT

CountryIAT

Country IAT

CountryIAT

Country IAT

CountryIAT

Country IAT

CountryIAT

11


CIRT – Corporate Headquarters

ICS Escalation FlowEVENT

First Response process

STOP

Normal operating

procedures ?Yes

NoIAT only - Regional Offices

IAT activated

Assess (use Initial Assessment Form)

Incident Commander:IC notifies Regional LIRT Incident Commander

LIRT – Regional HQs & Select Offices

Yes

Security (if any), Incident Commander & Business Continuity

discusses IAT Activation?

No

Activate LIRT?(appropriate components)

IAT activated Assess (use Initial Assessment Form)

IAT continues monitoring

Yes

No

LIRT activatedConduct Action Planning ProcessIC notifies local Executive ManagementIC notifies CIRT Incident Commander

Activate CIRT?(appropriate components)

CIRT activated•Conduct Action Planning Process•IC notifies the Policy Committee

IAT continues monitoringYes

No

Notify GSCC

Global Security Control Center (GSCC) Process

Security, Incident Commander &

Business Continuity discusses IAT

Activation and Cross Office Notification?

NoCross Office Notification

Process

IAT activatedAssess (use Initial Assessment Form)Cross Office Notification Process

Yes

Cross Office Notification Process

Monitoring Continues


www.aesrm.org

External Relationships

13


Building and Managing External Relationships – Taking Incident Management “Beyond Your Four Walls”

The major events of this decade support the premise that an organization’s incident management planning should be externally as well as internally focused.

Pre-Event Coordination Strategies with: Financial Services Firms and Industry Associations Key Suppliers Public Sector – Governmental and Non Governmental Organizations Regulators

Discuss as a group what’s working, where more attention is needed, and what’s being done to close the gaps.


www.aesrm.org

Conducting Effective Exercises of Emergency Management Plans

15


Integration of ICS with existing Business Continuity Program

• Business Recovery and Technical Recovery activation– Planning & Intelligence on CIRT/LIRT– Problem Resolution Team (PRT) process

• Business Recovery Plan– Activation Flow

• Pandemic planning scenario• Business Continuity Manuals

ICS is an integral part of the Business Continuity Program

16


ICS Process

• Event occurs beyond normal operations• Initial Assessment Team (IAT) meets to determine impacts, incident level,

and necessity of LIRT activation

• LIRT activated -- Incident Commander (IC) and Group Leaders hold action planning meeting to determine objectives and operational period (OP)

• Group Leaders share objectives on Action Plan and functional areas begin work

• LIRT members of the functional areas complete Action Plan Objectives and provide status to Group Leader

• Incident Commander and Group Leaders meet to share status and if needed determine new objectives and new operational period

17


IAT Assessment

• Assess impacts of the incident• Determine incident level• Based on incident level, take

appropriate action• Offices with IAT only, continue to

address the event

Incident Levels:

Level 1: Compartmentalized or Minor• An emergency that is limited in scope

Level 2: Local or Minimum• An emergency that is moderate to severe

in scope

Level 3: Regional or Major• A catastrophic disaster that has severely

damaged a mission critical facility requiring relocation of staff and business processes and/or severe disruption of services at that facility

18


CIRT/LIRT Process

• Decision is made to activate virtually or physically

• An action planning meeting by the Incident Commander (IC) and the Group Leaders is held as soon as the decision is made to activate the CIRT/LIRT

• The IC coordinates the Action Plan to share with CIRT/LIRT members

• CIRT/LIRT members take steps to complete Action Plan Objectives

• Report status updates to the Group Leader

• If needed, Action Planning begins again

19


Steps to Complete

• Incident Commander and Group Leaders conduct an Action Planning Meeting– Determine strategic objectives– Assign objectives to Groups– Set Operational Period (OP)

• LIRT group members receives objectives and begin taking action– Work across all Groups if necessary– Record findings– Update Group Leader

20


Emergency Management Planning DeliverablesDeliverables Core Offices

CIRT/LIRTsLIRT’s IAT’s Due Date

C1, C2, C3 K1, K2, K3, K4 Remaining offices

CIRT/LIRT Notification Test (conducted by BC) 2 Same as Exercise

IAT Training (conducted by BC) 2C1 = Mar. & Sep.C2 = Mar. & Sep.C3 = Mar. & Oct.

CIRT/LIRT Functional Group Training (conducted by BC) 1C1 = Aug.C2 = Aug.C3 = Apr.

CIRT/LIRT Scenario Based Exercise (conducted by BC) 1C1 = Nov.C2 = Nov.C3 = May

LIRT Notification Test (conducted by BC) 1 Same as Exercise

IAT Training (conducted by BC) 1

K1 = MayK2 = Mar.K3 = Jul.K4 = Jul.

LIRT Scenario Based Exercise (conducted by BC) 1

K1 = Jun.K2 = MayK3 = Oct.K4 = Oct.

IAT Notification Test (conducted by BC) 1 29-Aug.

IAT Training (conducted by BC) 1 Dates through out year

IAT Self Exercise (conducted by your team) 1 29-Aug.

21


IAT Notification Tests and Self Exercise

Notification Test• Test SMS message on work mobile phones and devices • Execute Emergency Notification Tool sending a voice message to

Work Phone and Mobile, text message to Work Email, and SMS to Mobile.– Respond to each message as requested.

Self Exercise• Conduct an IAT emergency table top exercise led by Incident

Commander• Use the IAT Self Exercise Guidelines and ICS forms and tools• Complete BC survey to validate successful completion

22


Comprehensive ICS Exercise

Objectives

• Practice the use of ICS processes under simulated emergency conditions and identify any processes or policies that need improvement

• Practice the LIRT’s ability to coordinate their response and decision making under simulated emergency conditions

• Provide a learning environment to allow LIRT members to increase proficiency in executing their roles and responsibilities

23


Exercise Structure

• Exercise conducted in a physical command center.

• Business Continuity staff will facilitate and provide assistance with ICS processes when needed.

• A simulation (sim) team will act as the “outside world” for this exercise. All issues requiring the outside world must be solved by contacting the simulation team. Such as; gathering information, order equipment, etc.

• Distribute messages with questions and concerns throughout the exercise from numerous entities (internal employees, media, etc…).

24


ICS Team MemberCommitment and Empowerment

ICS team members must be:• Trained to clearly understand their roles

and responsibilities• Committed to fulfilling their responsibility• Engaged by participating in meetings and

exercises• Empowered to perform their roles in

accordance with practiced guidelines

Effective emergency response is dependent on qualified staff being trained to execute with proper authority


www.aesrm.org

Communications and Case Studies

26


Communications – Strategies Before, During and After

Focus Areas for Incident Communications: Awareness (before) Response (during and after)

What are Some Practical Challenges We Face? What are the benefits and limitations of various communication

tools and media? How to manage multiple threads of internal and external

communications, many of which are spontaneous during an incident? How do you (or should you) look to establish a “sole source of truth?”

How should plans factor in the unavailability of various media during an incident?

27


Case Studies

Communication and Coordination Strategies – Putting It All Together:

9/11 Atlantic Storms of 2005: Katrina, Rita, Wilma London Underground Bombings

What Experiences Can We Apply to the Incident Management Challenges Likely to Occur with Events of Uncertain or Lengthy Duration (e.g., Pandemic)?


www.aesrm.org

financial services technology consortium incident management an exchange of practices and...

Documents