_la force de l’engagement mc an informal report it performance management in a financial crisis

_la force de l’engagement MC

An informal Report

IT Performance Management in a Financial Crisis

2

Index

• Historical causes of the Argentinean financial crisis• IT situation at the Bank• The crisis starts• Impact of the crisis on the transactional channels• The “solution” to the crisis• Financial impact of the “solution”• Impact of the “solution” on software development• Long duration impact on the country and the bank• Lessons learned

33

CGI – Some facts

Founded in 1976

31,000 professionals

125+ offices worldwide

2010 revenue $3.7 billion

Backlog $13.3 billion

Founded in 1976

31,000 professionals

125+ offices worldwide

2010 revenue $3.7 billion

Backlog $13.3 billion

One of the largest pure play IT/BPO services firms

• 1st in Canada• 4th in North America• 7th in the world among firms with client proximity

presence

One of the top global outsourcing firms• Among the top 10 IT/BPO providers within the Global

Outsourcing 100 ranking

44

CGI’s - Offices

•

•

•

• •

•

•

•

• •

•

•

•

•

•

• •

• •

• •

•

•

• • •

•

• •

• • •

•

•

• •

•

•

•

•

•

•

• •

Alaska

• •

•

•

• •

• • • •

•

• • • • • • • •

•

•

•

•

•

• •

•

•

•

• • •

• •

•

• •

•

•

•

•

•

• •

•

•

•

• •

•

•

•

•

•

•

•

Hawaii

Europe

Asia - Pacific

•

•

• • • •

•

5

Argentina in the world

Argentina USA Canada

Area 1M 3.8M 3.9M

Pop. 36M 310M 31M

6

Economic evolution of Argentina

7

Causes of the poor economic growth

• Protectionist policies• Populist governments

• Commercial deficit• Fiscal deficit

• Inflation• Public debt

Commercial deficit Fiscal deficit

Inflation&

Public debt

Protectionist policies&

Populist governments

Political

Instability

8

Inflation & hyperinflation

9

Growth & Unemployment in the 90s

-5

0

5

10

15

20

1980 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000

% GDP per capita % Unemployment

10

Situation at the Bank (2001)

• From 40 to 130 branches• From 1500 to 4,000 employees• From 300,000 to 900,000 accounts

• Mainframe• 30 AIX servers• 300 Windows servers

• Transactional channels• Human tellers• ATMs• Call Center• Internet

• A Capacity planning and performance process in place and working

11

Chronology of the crisis

• 30/11/01 : A run on the banks begins, with the central bank reserves falling by $2 billion.

• 01/12/01 : The government imposes a $1,000 per month limitation on personal bank withdrawals in cash. Checks and electronic transfers go up.

• New Year’s Eve party, the president told us that “this will be a hard year”.

• The Internet traffic and the call center calls suddenly skyrocketed. ATMs became empty in a matter of hours.

12

Internet transactions

•The number of concurrent user increased 5 to 6 times from one day to the other.

•We proceeded to change the architecture of the main page of the bank to reduce the workload in the system, but it was only a partial solution.

13

Internet infrastructure

Internet

14

Call center

•The talk time increased

•People don’t like to talk with machines when in a crisis

•Having young call center operators is cheap, but it could backfire

•Having call center operators abroad is probably inconvenient

15

The trick of the withdrawal limit

Bank 1

Bank 2

Bank 3

Bank 1

Bank 2

Bank 3

Interchange bank

system

The system became saturated

The checks clearing took more time than the allowed legal period

The number of bank accounts exploded, branches started to open Saturdays to cope

with the demand

16

Mainframe – The perfect Storm

There weren’t enough resources for the on-line

The branches workedin extended mode

The nocturnal batch process took more

Than 24hs

The Perfect

Storm

There wasn’t an easy way to increase the capacity

There wasn’t hardware available to increase it

All the banks had the same problem

17

Chronology of the crisis

• 30/11/01 : A run on the banks begins, with the central bank reserves falling by $2 billion.

• 01/12/01 : The government imposes a $1,000 per month limitation on personal bank withdrawals in cash. Checks and electronic transfers go up.

• 20/12/01 : A riot leaves 28 people dead, the president has to resign. The two subsequent presidents last only days. Several supermarkets are looted. Argentina declares itself in default.

• 02/01/02 : President Duhalde takes office. • 06/01/02 : The “solution” is approved by the

Congress.

• New Year’s Eve party, the president told us that “this will be a hard year”.

• The Internet traffic and the call center calls suddenly skyrocketed. ATMs became empty in a matter of hours. The IT infrastructure collapses under the workload.

• The software development team starts to work to adapt the systems to the “solution”

18

The solution

• Default on the national debt. The country declares itself in bankruptcy.

• End of the dollar as a legal currency in Argentina.

• The old parity 1 dollar = 1 peso became 1 dollar = 3 pesos.

• The mortgages are converted to a 1 = 1 parity.

• All the bonds that the banks and the persons held became “junk bonds” with 0 value.

• All the bank accounts in dollars had to be converted to pesos, the same with mortgages and investments.

• All dollar investments lose 66% of their value.

• Most of the bank assets are reduced to 33%, all the banks are technically broken.

But, didn’t all this mess help protect the financial system from becoming insolvent?

19

The people were not happy with the solution…

20

Working in the Solution

• Modify or replace almost all the core programs of the bank.• Checks, mortgages, savings accounts.• General Ledger, financial reports• Transactional channels (web, IVRs)

• New programs like the system to manage the legal claims of those people which were on “especial situations” were made to extract more money that the central bank allowed

• Everything had to be done in a question of days.

21

How they worked in the Solution

• There were only 3 environments :• unitary test • integration test• production

• The tests were reduced to their bare bones.• The emphasis was on the communication not in the

documentation.• The development teams were all on the same floor.• The technical support team was 2 floors below.• The operations people that had to interpret the Central Bank measures

were there. • Everybody understood the importance of what they were doing.

22

Long duration impact in the bank

1992 2001 2003

Branches 40 130 60

Empl. 1500 4000 2000

Accounts 300,000 900,000 700,000

Hw = ↑ Outsourced

Sw = ↑ Opportunity

23

Long duration impact in the country

• -11% GDP in 2002 alone (the 2008 financial crisis in US was a - 3.8%)

• 40% inflation in 2002

• 20% of unemployment

• 63% of families below the line of poverty

• 20% of families without enough money to buy food

• The emigration rate multiplied by 6 in 2 years• Larger between university graduates and post-graduates

24

Can it happen to you? Can it happen now?

YES

The IQ of the governments doesn’t look like it has improved in the last years

The financial system is more and more dependent on IT

25

But in 10 years it should have new solutions…

YES, and new problems…

The mainframe has reserve capacity like CPUs that can be activated on demand

Cloud computing could also help…

But the providers of these services could have an attitude of “pay now, use later”

If the cloud is shared by companies of the same market (e.g. banks), and the cloud is in the same geographical place it could

become saturated

What will be the impact on new technologies like smartphones?

26

Not two financial crisis are equal but…

• Liquidity crisis (bank run)• A large number of customers withdraw their deposit

believing that the bank will fail, and in that way causes the failure.

• Nationalisations and quasi-nationalisations of banks, but banks don’t fail

• Short lived but source of “recessions”• More time to change the system

• Banking panics• The crisis affects many banks• The run take some time to settle but the recession takes

years• Generates changes in the banking system

• Systemic banking crises• Great Depression & Argentine economic crisis• Generalized failure of banks; all the capital of the financial

system is wiped out• The crisis take months to develop

• Constant change of regulations.

27

Lessons learned

The possibility of a financial crisis has to be part of the IT contingency plan

28

Always have a plan “B”

“Plan B” for the rescue effort of the Chilean trapped miners

29

Document the way that performance can be improved

30

Design infrastructures as flexible as possible

31

Never underestimate “ad-hoc” solutions

32

Keep an eye on your team’s morale

33

Be prepared to throw everything you know out of the nearest window

34

Don’t worry about what top management thinks

35

SCCMG 2011