draft_3 - copy

8/4/2019 Draft_3 - Copy

1/22

1

CERTIFICATE

This is to certify that the Project Work titled Building Disaster recovery

capabilitieshas been successfully completed at MphasiS Ltd.byDinesh Kumar,

under my guidance, in partial fulfillment of the Post Graduate Diploma inIndustrial Management at National Institute of Industrial Engineering (NITIE),

Mumbai. I wish him a bright and prosperous future.

Mr. V. B. Khanapuri

Assoc.Professor

Faculty Guide

NITIE, Mumbai

8/4/2019 Draft_3 - Copy

2/22

2

Acknowledgement ...................................................................................... ............................................. 3

Executive Summary ................................................................................................................................. 3

Company Overview.................................................................................................................................. 3

Applications ......................................................................................................................................... 4

BPO ..................................................................................................................................................... 4

Infrastructure Services ......................................................................................................................... 4

Payments Solutions.............................................................................................................................. 5

Project Overview ..................................................................................................................................... 6

Need for the project ............................................................................................................................ 6

Objectives of the project ...................................................................................................................... 6

Key Deliverables .................................................................................................................................. 6

Literature Survey ..................................................................................................................................... 6

BCP Planning Methodology .................................................................................................................. 6

Building Disaster recovery capability in Service desk business .................................................................. 7

Identification of critical requirements .................................................................................................. 7

Defining Maximum Tolerable Period of Disruption ...................................................... ......................... 7

Defining Recovery Time Objectives .................................................................... .................................. 7

Identification of Alternate Site ............................................................................................................. 8

Identification of resource and infrastructure requirements .................................................................. 8

BCM Response ..................................................................................................................................... 8

Training and testing .............................. ............................................................................................. 11

Maintenance and updating ................................................................................................................ 11

Identification and prioritization of possible risks in Disaster Recovery strategy ...................................... 11

Risk Identification .............................................................................................................................. 11

Risk Prioritization .................................................................................. ............................................. 16

Analysis of current recovery strategies .................................................................................................. 18

Account A .......................................................................................................................................... 18

8/4/2019 Draft_3 - Copy

3/22

3

Account B .......................................................................................................................................... 19

Account C ..........................................................................................................................................20

Account D ..........................................................................................................................................21

Miscellaneous ....................................................................................................................................22

Acknowledgement

This project is the result of two months of work whereby I have been accompanied and

supported by many people. It is a pleasant aspect that I have now the opportunity to express my

sincere thanks to all of them. I would like to express my gratitude to all my professors, my

institute, team members & the company. Their generous help and support enabled me to

complete this study within the stipulated time period.

I am very obliged to National Institute of Industrial Engineering (NITIE)& MphasiSLtd.for

giving me the opportunity to work on this project & would like to specially thank Dr. V. B.

Khanapuri (Associate Professor, NITIE) ,Mr. MakarandVaze ( Project Manager, MphasiS Ltd.)

and Mr. RitieshSethi (Associate Vice President,MphasiS Ltd.) who provided guidance, help and

cooperation at every stage of the project.

I would like to thank my colleagues and friends who have directly or indirectly provided

me assistance and helped to materialize this project. However I take the responsibility for all my

shortcomings. I, in all sincerity, hope that my efforts will be appreciated.

Executive Summary

The Mphasis India BPO Service Delivery Centre provides Service Desk support to various

corporate clients. In order to ensure that the Service Desk services are available at all times, the

BPO has developed a disaster recovery plan to provide basic Service Desk service in the event

of a disaster.

The objective of the project is to analyze the disaster recovery strategies for four accounts so as

to improve the current disaster recovery strategies. The project expected identification and

prioritization of possible risks and analysis of recovery strategies of four corporate accounts.

Company Overview

8/4/2019 Draft_3 - Copy

4/22

4

MphasiS (BSE: 526299, NSE: MPHASIS), a unit of Hewlett-Packard Co., is an information

technology services company based in Bangalore, India. MphasiS is certified with ISO

9001:2008, ISO/IEC 27001:2005 and is assessed at CMMI v 1.2 Level 5.It is the sixth largest IT

company in India with more than 38,000 employees as of 2010. The company has 29 offices in

14 countries with delivery centers in India, Sri Lanka, China, North America and Europe.

MphasiS is engaged in providing information technology services to its customers around the

world. MphasiS provides integrated solutions that include business process outsourcing,

infrastructure technology, and application services. The application service offered by the

company includes application development as well as applications maintenance and support

services. The markets served by the company are financial services & insurance, healthcare,

manufacturing, government, transportation, communications, and consumer & retail industries.

Applications

MphasiS is uniquely positioned to cater to the global strategic outsourcing demand and have

integrated changing environments into Applications service portfolio. Full range of application

services helps to invest more strategically in core business activities, while improving ROI from

tight IT budgets.

BPOMphasiS, a leader in providing end-to-end Business Process Outsourcing services, provides

high quality, value-added voice and transaction-based services to Fortune 500 companies

worldwide. Being the early entrant in the BPO space, they are experts in providing strategy,

solutions, and services to solve complex business issues and achieve results. By coupling theirBusiness Process Outsourcing services and in-depth industry specific knowledge, they provide

business focused solutions tailored to clients strategic goals.

Infrastructure Services

MphasiS Infrastructure services have the distinguished blend of classic HP pedigree clubbed

with skilled scalable workforce and firm focus on service delivery quality. Whether you are

looking to reduce costs, improve business-to-IT alignment, drive innovation, or reduce risks,

MphasiS offers a comprehensive suite of remote Infrastructure management services. MphasiS

helps optimize clients infrastructure through a mature global delivery model, standardized

processes, adoption of IT Infrastructure Library (ITIL) framework and global service network,

robust private network with built in redundancies. This network enables secure connection with

clients 24x7x365.

8/4/2019 Draft_3 - Copy

5/22

5

Payments SolutionsMphasiS offers a gamut of services such as consulting, application development,

modernization, integration, and testing in payments arena to clients worldwide. The array of

solutions from MphasiS for wholesale payments, retail payments, PCI DSS compliance, and

other value added services are well designed to help clients reduce processing costs, retain

investments, streamline processes, transform applications, identify and deploy new channelsthereby increasing their revenues streams.

Applications

ApplicationsDevelopment

ApplicationsManagement

EnterpriseApplications

Services

Business Practices

TechnologyPractices

ConsultingServices

BPO

Industry Offerings

Service Offerings

Infrastructure

Services

RemoteMonitoring

Center

Data CenterServices

WorkplaceServices

Network Services

Security Services

Solutions

Payments

Solutions

Retail &Wholesale

Payments

PCI & DSSCompliance

MobileTransactionProcessingSolution

Value AddedServices

8/4/2019 Draft_3 - Copy

6/22

6

Project Overview

Need for the project

Business continuity plans are not taken up very seriously and disaster recovery strategies were

not defined with proper rationale for many BPO accounts of the Mphasis. This is so because

disaster situation had not been faced since the inception of most of the accounts. This project

aims to identify the possible risks in the disaster recovery strategies and to recommend the

improvements in the disaster recovery strategies of the four accounts.

Objectives of the project To identify the possible risks inthe recovery strategies ofthe service desk business.

To suggest corrective actions for the identified risks.

Prioritization of the identified risks.

Analysis of the recovery strategies of the four accounts of the company.

Key Deliverables List of the possible risks with corrective actions

Prioritized list of the identified risks

Risks in the current recovery strategies of the four account

Recommendations to avoid the risks associated with current recovery strategies of the

four accounts

Literature Survey

BCP Planning Methodology

8/4/2019 Draft_3 - Copy

7/22

7

Building Disasterrecovery capability in

Service desk business

Building disaster recovery capabilities in Service Desk business involves the following steps:

Identification of critical requirementsFollowing are the critical requirements for the service desk business:

- Human resources: Since the business requires the calls to be handled by the agents

therefore, agents are the most critical requirement for the service desk business.

o Number of agents to provide expected/agreed level of service.

- Infrastructure Requirements: Following are the supporting infrastructure required for

an agent to perform his job

o Seats

o Desktops and laptops

o VOIPS (Voice over Internet Protocol)

o Network / LAN, power points

o Landline, mobile, fax, printer

- Application requirement

o DW software: It is the software used to maintain the status of the agents and

route the calls to the agents.

o EKMS: Enterprise Knowledge Management System contains all thesupporting documents for the agents to find the solutions to the problems

faced by caller.

Defining Maximum Tolerable Period of DisruptionThe Maximum Tolerable Period of Disruption (MTPoD) for the project/account is that

duration of time after which an organizations viability will be irrevocably threatened if

delivery of products/services cannot be resumed. It can either be defined in the

contract or can be calculated through business impact analysis.

Defining Recovery Time ObjectivesThe maximum period of time for which the project may remain non-operationalwithout incurring irrevocable damage is referred to as the Recovery Time Objective(RTO). It should be defined with the service level requirements. It should not exceedMTPoD.

8/4/2019 Draft_3 - Copy

8/22

8

Identification of Alternate SiteAlternate site should be defined considering the availability of infrastructure

requirements and the time and cost of transportation.

Identification of resource and infrastructure requirementsRequirements of resources and infrastructure should be quantified according to the

service level requirements.

BCM Response

Project BCP Team

The Project BCP team required to respond to an incident and subsequently initiate

recovery procedures are as follows:

y Project/Account Owners

Project owners are the personnel responsible for all the decisions made regarding theproject. These personnel are to be informed when the BCP is to be invoked, as theyare to take the key decisions during the incident and the recovery phases.

y Project/ Account BCM Coordinator(s)

BCM Coordinators are the personnel responsible for authorizing activities,communications and response teams that may be necessary for supporting the

incidents in consultation with IMT and Project Owners. They aid the Project Owners indetermining whether a disaster should bedeclared and which Teams should bemobilized. They also Report progress and problems as required.

y Recovery Team

The Recovery Team consists of resources that are critical to the delivery of the projectincluding team members, team leads and project managers. Their role is to initiate therecovery of critical activities identified during the Business Impact Analysis (BIA).

The details of this team are depicted the call tree in Table below and referenced inTable 2 : Recovery Team Call Tree

Project Team Detail

Role Name Contact DetailsBackup Team

Member Contact Details

Project OwnerName of theproject owner

Office: ContactNo.

Home: ContactNo.

Cell Phone:

Name of theBackup teamperson

Office: ContactNo.

Home: ContactNo.

Cell Phone:

8/4/2019 Draft_3 - Copy

9/22

9

Contact No. Contact No.

BCMS

CoordinatorName of BCMSCoordinator

Office: ContactNo.

Home: ContactNo.

Cell Phone:Contact No.

Office: ContactNo.

Home: ContactNo.

Cell Phone:Contact No.

Recovery

TeamRefer Table 2: Recovery Team Call Tree for more details

Table 1: Project/Account BCP Team Contact Details

Recovery Team Call Tree

RecoveryFunction

Name of Individual ContactNumbers

Alternate Name ContactDetails

Project/AccountDelivery Head

Name of Project/ccountDelivery Head

Home: ContactNo.

Office: ContactNo.

Cell: ContactNo.

Name ofProject/AccountDelivery HeadDelegate

Home: ContactNo.

Office: ContactNo.

Cell: ContactNo.

Project/AccountDelivery Manager

Name ofProject/AccountDelivery Manager

Home: ContactNo.

Office: ContactNo.

Cell: ContactNo.

Name ofProject/AccountDelivery ManagerDelegate

Home: ContactNo.

Office: ContactNo.

Cell: ContactNo..

BCMS Coordinator Name of BCMSCoordinator

Home: ContactNo.

Office: ContactNo.

Cell: ContactNo.

Name of AlternateBCMS Coordinator

Home: ContactNo.

Office: ContactNo.

Cell: ContactNo.

Member 1 Name of projectmember

Member 2 Name of project

8/4/2019 Draft_3 - Copy

10/22

10

member

Table 2: Recovery Team Call Tree

Notification and Escalation Process

In the event of a disaster, the steps to be followed from initial notification to the recovery ofthe identified business processes are detailed below in Figure 1:

Figure 1: Notification and Escalation Process

BCP Invocation Plan

Invoking this plan implies that a recovery operation has begun and will continue with toppriority until workable recovery support has been established. The critical systems to berecovered will depend on Maximum Tolerable Period of Disruption (MTPoD) defined bythe project/account team.

The specific project BCP will be invoked by the Project Manager. The recovery team willbe mobilized using the call tree presented in Table 2: Recovery Team Call Tree. Therecovery teams will meet at the alternate location as described in the Detailed RecoveryProcedures and Requirements.

8/4/2019 Draft_3 - Copy

11/22

11

Incident Management Plan

Each project or account is associated with a facility. Each facility owns a site level IncidentManagement Plan.

Training and testingTimely training and testing should be done for the Business continuity managementprocedures so that everyone involved in the recovery procedure knows their roles andresponsibilities.

Maintenance and updatingBCM (business continuity management) procedures and BCP should be updated regularly

based on the feedback of testing. BCP also needs to be updated based on any changes inthe organization like change in contact details, change in the service level agreements orchanges in the call volume.

Identification and prioritization of

possible risksin Disaster Recovery

strategy

Possible flaws or risksin the recovery strategy were identified and the prioritizationwasdone based on the response of the operations managers.

Risk IdentificationFollowing major risks with the recovery strategy were identified:

MTPoD is not defined

If MTPoD is not defined then there would be no target time for recovery and the

company may exceed the time after which the organizations viability will be

irrevocably threatened. It may lead to loss of business and loss of goodwill.

Corrective action:

MTPoD is either defined in the service level agreements and contract or it should be calculatedusing the business impact analysis. In former case, MTPoD is agreed upon with

8/4/2019 Draft_3 - Copy

12/22

12

the customer and it is defined in SLA. In later case, it is calculated by comparingthe revenue loss or penalty to the expected profit from the account.

Number of daysconsidered (foradjusting 24X7 ops)

13 5 10 15

Factors 0 - 4hours`

4 hours -24 hours

1 day - 3days

3 days - 7days

1 week - 2weeks

2 weeksand above

Number of billablehours lost

PrimaryOperationalimpact

Any other financialImpact

Total OperationalImpact

This operational impact could be compared with the expected profit from the account or theprofit above minimum profit margin required. MTPoD could be defined for eachservice separately.

RTO is not defined

If RTO is not defined recovery process could exceed MTPoD. This could result in

the irrevocable loss. It should be defined with service level requirements to be

resumed.

Corrective action:

To Determine the time to resume activity, some thought needs to be given to

defining the level of performance at resumption (e.g. number of personnel,

manufacturing throughput, invoices produced) and determining the time required

to return to normal levels of operation. It is believed that the standard is not

looking for a scientific calculation of the latter (BCM is not a science) but is just

looking for an indication from someone that understands the activity.It should begenerally less than 50% of the MTPoD.

Resource requirements are not quantified

It would lead to confusion at the time of disaster because there is no clear

requirement defined for the number of agents that needs to be present. And it

8/4/2019 Draft_3 - Copy

13/22

13

would be difficult to identify the infrastructure requirementsas it would depend on

the number of agents. If infrastructure requirements are not defined clearly there

are chance to raise request for more or less than requirement.

Corrective action:

Number of agents required to meet the service level requirements defined in

RTOshould be clearly defined. It could be derived from the peak level

requirements and the service level of the RTO. For example, if 100% service

level is required and the peak level agents requirement is 45 then 45 seats needs

to be booked. And if 33% service level is required then 15 seats needs to be

reserved and it should be scaled to 45 within MTPoD.

Seats Required = (No of agents in peak hours) x ( service level requirement)

One or two extra seats may need to be added for operations manager and

assistant operations manager.Other infrastructure requirements could be defined

using the number of agents as follows:

Desktops = No of agents

Landlines = 1 or 2 for operation manager and assistant operations manager

Printers = 1

No. of ports = Total number of seats

VOIP = Total number of seats

LAN = Total number of seats

Number of power points = 2 x (number of agents) + 2

(Each agent requires two power points while manager requires only one)

Applications: Internet explorer, DW, Avaya interaction center, EKMS

Alternate site is quite far

It would take lot of time to travel to the alternate location or transportation would

be quite expensive. This could lead to not meeting the RTO or could lead to high

expenditure on transportation. If it is defined in different country it would be

difficult to have enough resources there or it could be quite expensive.

Corrective action:

Recovery site should be reachable within RTO so that operations could be

resumed and it transportation should not be very expensive. And the facilities like

8/4/2019 Draft_3 - Copy

14/22

14

Mphasis network should be available at the alternate site. As the Mphasis has

offices at Mumbai and Pune, they can act as alternate site for each other.

Mumbai-Pune:

Distance = 160 kms

Travel time by road = 3 to 4 hrs

Travel time by train = 3hrs

Alternate site is in the same city

Usually natural or civil disasters affect the whole city so it would not be a nice

idea to have alternate site in the same city.

Corrective action:

It would be better not to have alternate site in the same city.

Alternate contact is not defined in the BCP team

If the primary contact is not available at the time of disaster due to any reason it

would lead to lot of chaos and mismanagement and it may lead to complete

failure of disaster recovery.

Corrective action:

Every BCP team member should have an alternate contact. It is very crucial and

cannot be neglected.

Unavailability of BCP copies at the time of disaster

If BCP copies are not accessible at the time of disaster it would become difficult

to carry out disaster recovery procedure.

Corrective action:

Usually soft copy of the BCP is kept at a shared location and local copy is kept

with recovery team members on their laptops. Hard copy is also kept at recovery

location. In addition to this hardcopy could be kept at the residence of therecovery team members.

Unawareness and inadequate training of BCP

8/4/2019 Draft_3 - Copy

15/22

15

If team members are not aware of the recovery process or not practiced the BCP

procedure they may find it difficult to execute it at the time of disaster.

Corrective action:

Regular exercise should be done for the BCP procedures and escalation process

should be practiced regularly. Currently it is practiced annually or semi-annually.

Quarterly tests could be taken to find out the understanding of the disaster

recovery procedures. If tests results are not satisfactory training needs to be

given for that member.

BCP is not updated with latest contacts

If BCP is not updated with latest contacts it would create trouble in contacting

theBCP team members. It may cause delay due to the time lost in finding latest

contacts for the members.

Corrective action:

Whenever a team member leaves or joins company the contact details should be

updated. And whenever contact information of a member is changed the BCP

plan needs to updated with latest contact information.

Availability of infrastructure requirements at alternate site is not verified

If all the infrastructure requirements are not verified it would lead to not meeting

service level requirements. Followings could be the causes:

Network Availability: Mphasis network availability is the most crucial

requirements. If it is not available no calls could be received.

Desktops, VOIPs, Power points, etc: All these requirements are crucial for

handling the call by an agent.

Applications ( DW, EKMS) : DW is necessary for routing the calls and EKMS is

necessary for the agents find the solutions of the customers problems.

Corrective action:

Network and application availability needs to be verified. Assignment of seats

and infrastructure requirements needs to be verified with the chief risk officer at

the alternate site.

8/4/2019 Draft_3 - Copy

16/22

16

Risk PrioritizationOperations Managers and assistant operations manager were asked to identify each risk as

high, medium or low. Based on the following description:

High Risk:Risks that could lead to failure of recovery or could lead to exceed the

MTPoD.These risks could lead to serious business loss.

Medium Risk : Risks that causes large delays and lead to exceed RTO and

sometimes MTPoD. These could cause the business loss and could affect the

service levels.

Low Risk: Risks that could cause operational problems and could lead to delays

in RTO. Business is not affected but could cause delays and interruption in

recovery.

Following values were assigned to the three levels:

High : 3

Medium : 2

Low : 1

Averages are calculated for each risks and categorization is done as follows:

1 to 1.5 : Low

1.5 to 2.5 : Medium

2.5 to 3 : High

Sl.No.

Risks R1 R2 R3 R4 R5 R6 R7 R8 AverageRisk

category

1MTPoD is not

definedH H H H H H M H 2.9 High

2RTO is not

definedH M M M M H M M 2.3 Medium

8/4/2019 Draft_3 - Copy

17/22

17

3

Resourcerequirements

are notquantified

H M M H M M M H 2.4 Medium

4

Alternate site

is quite far M L L M L L L M 1.4 Low

5Alternate site

is in thesame city

M M M M L L L M 1.6 Medium

6

Alternatecontact is notdefined in the

BCP team

M M M L M M M M 1.9 Medium

7

Unavailabilityof BCP

copies at thetime ofdisaster

M M M M M L M L 1.8 Medium

8

Unawarenessand

inadequatetraining of

BCP

H H M M M H M M 2.4 Medium

9

BCP is notupdated with

latestcontacts

M L L L M M M M 1.6 Medium

10

Availability ofinfrastructurerequirementsat alternatesite has not

been verified

H H M H H H M H 2.8 High

Following list of prioritization is created from the average score:

1) MTPoD is not defined

2) Availability of infrastructure requirements at alternate site is not verified

3) Resource requirements are not quantified4) Unawareness and inadequate training of BCP

5) RTO is not defined

6) Alternate contact is not defined in the BCP team

7) Unavailability of BCP copies at the time of disaster

8) Alternate site is in the same city

9) BCP is not updated with latest contacts

10) Alternate site is quite far

8/4/2019 Draft_3 - Copy

18/22

18

Analysis of current recovery strategies

Four accounts had to be analyzed to find the risks in their currentrecovery strategy. Corrective

actions also need to be suggested for the accounts.

Account A

Account Details:

Following are the important details about the account:

Number of Agents= 96 approx.

Approx. no of calls = 22,000 per month

Contract is with HP and HP deals with customer

Current Recovery strategy:

ForUS/Canada and EMEA, the alternative recovery site is Tower 4, Magarpatta, Pune.

Strategy is to have the calls routed through a DR skill to a specific defined set of

employee ids.

3 agents would be creating the ticket number with the customer details and problems.

8/4/2019 Draft_3 - Copy

19/22

19

Tickets would be sent to the Mumbai site when it is up.

Risks with the strategy:

No business impact analysis is done and MTPoD and RTO are not defined:

No plans if the disaster effects the functioning of Mumbai site for more than 24 hrs.

Plan has been left to discussion at the time of disaster.

Three agents would not be enough to handle peak hour volumes which requirearound

40 agents.

Recommendations:

Business impact analysis must be done to define MTPoD and RTO or it shouldbe

agreed upon with HP.

Based on RTO and service level agreements alternate site and recovery requirementsshould be set.

Account B

Account Details:


Approx no of calls = 24,000 per month

New account, started in Oct, 2010

BCP plan is yet to be signed off


MTPoD is 120hr according to contract

RTO is set as 90 hrs

30 % of services need to be rendered within 24 hrs

Alternate site 1: Tower CC4, Magarpatta, Pune

Alternate site 2: Infinity Park, Mumbai

Number of seats required = 30

8/4/2019 Draft_3 - Copy

20/22

20

LAN (30), VOIP (30) , Power Points (58), Laptops (2), printer (1),etc are required

Risks with current strategy:

RTO of 90 hrs is not defined with the service level requirements. 90 hrs that is 75% of

MTPoD is quite large.

No rationale behind 30 seats requirements. It would not be possible to handle peak hour

call volume which requires 45 agents.

Recommendations:

RTO must be defined with the service level requirements. RTO is generally less than

50% of the MTPoD.

45 seats are required if 100% service level needs to be met because peak hour

requirements is 45 agents.

If 30 seats are used then what service level could be met should be defined and agreedupon.

Account C

Account Details:



BCP plan is yet to be signed off although the account is quite old


RTO = 9 hrs

Call transferred to HP, Budapest and HP would handle the calls



Alternate site is not defined in India

Recovery requirements have not been identified

Recommendations:

8/4/2019 Draft_3 - Copy

21/22

21

BIA must be done to define the MTPoD

Alternate site could be Mumbai. Mangalore agents who handles only mails and chats

can be trained to manage the voice calls in case of disaster

Service level should be defined with RTO

Account D

Account Details:



Standard format is not used for BCP


Calls will be routed to Pune EON Kharadi and as many agents as possible will be moved

to Pune EON Kharadi.

Critical Service Desk support is to be provided within 1 hour

Basic Service Desk services would be available within 4 hours



Alternate site is in the same city

Recovery requirements have not been identified

Recommendations:

BIA must be done to define the MTPoD

Alternate site could be Mumbai

Requirements should be defined

8/4/2019 Draft_3 - Copy

22/22

22

Miscellaneous

Following observations are common to all the accounts:

Total number seats required at a recovery site which is an alternate site to the accountswhich are located at same site should be compared with available seats.If there is

significant difference in the requirements and availability, alternate sites should be

adjusted.

Tests for the awareness of the BCP should be conducted quarterly and training should

be conducted if the results are not satisfactory

draft_3 - copy

Documents