The need for contingency planning After reading this article on real disasters which have befallen computer installations, you may decide advance planning is a sensible idea

by KEN WONG

H ave you ever experienced a computer disaster in your in- stallation, or a serious disrup-

tion to data processing through some natural or manmade cause? How did you recover? Do you have a contin- gency plan for major disruptions? What lessons have you learnt from your disaster? This article reviews the possible answers to these questions and offers a plan of action to minimize the effect of a computer disaster.

Over the last two to three years, many organizations have been in- creasingly concerned about the im- pact of a major computer disaster. As

Abstract: The article reviews th effects of various computer disasters upon the data processing function. The effects of fire, explosion, water andjlood damage, building defects, lightning, radar interference and industrial action are all considered. The various types of standby facilities, as well as the top-down approach to contingency planning are also discussed.

Keywords: data processing, computer disaster, contingengy planning, backuplstandby facilities.

Ken Wong is manager of the Security and Privacy Division of BIS Applied Systems Limited.

a result, many have subscribed to cold standby or portable recovery centres and some have even protected them- selves with guaranteed immediate backup processing through warm standby options offered by several commercial concerns.

A recent BIS survey of some 50 medium to large computer installa- tions in the UK revealed that over 70% of the sample have invested in some form of disaster backup and standby. Many have produced in- house contingency plans, in some cases with the help of consultams, to speed up recovery from major com- puter disruptions.

Sadly, no government or official

Figure 1. Distnbution of computer dis-

aster cases. (Source: ref. I .)

institution in the UK collects statistics on computer disasters. As a result, any estimates on the probabilities of various forms of disaster and their likely impact on businesses have been purely guesswork and can rarely be substantiated by official figures from actuarial or government sources. Nevertheless, the results of an inde- pendent survey are shown in Fig- ure 1’.

Over 50% of the total reported cases occurred in the last two and a

half years and a significant number

incurred serious losses to the organi-

zations concerned. Unlike computer fraud and com-

puter crime where victim companies are reluctant to reveal details, victims of computer disasters have been more forthcoming and have related their personal experience in the salvage and recovery operations as well as the lessons learnt. Some would only re- lease details provided the anonymity of their organization was preserved. We are hopeful that in future many more cases will be reported as they come to light. This would allow a more meaningful analysis and would enable us to establish risk patterns and to provide useful statistics on loss figures. In time, the statistics being

6 0011-684x/84/080006-07$03.00 @ 1984 Butterworth 81 Co (Publishers) Ltd. data processing

policy

m Arson

0 Fire explosion

Loss in f'000

Figure 2. Computer fire incidents in the

UK.

compiled now should offer valuable input for management to budget for contingency planning and to provide cost-effective measures to deal with various forms of computer disasters.

Fire and explosion

Approximately 10 cases a year of fire and explosion involving computer in- stallations in the UK came to our attention in the last few years, al- though only 50% of the reported cases

Figure 3. Where was jire started. Figure 4. Cause of fire.

~0126 no 8 October 1984

gave enough details for followup and inclusion in the casebook.

Figure 2 illustrates the distribution of fire losses out of a total of 47 cases of fire, explosion and arson in our files. Thirty per cent of the total were attributed to arson or bombing, rising to over 50% for losses in excess of &500000.

We have to emphasize that reported details of business interruption losses remained sketchy in most cases. Al- though information on the value of damaged equipment was readily avail- able, the costs of refurbishing build- ing and contents, replenishment of stock, loss of profits, as well as addi- tional processing expenditure during the recovery period, were difficult to ascertain and have probably been understated in the documented cases. In some cases, we were told the full extent of business losses could only be obtained some 18 months to two years after the major disaster, and only with the full cooperation of all business functions.

Much of the electronic equipment superficially contaminated by the pro- ducts of combustion, such as smoke, soot and hydrochloric acid released from burning PVC, was successfully cleaned and recommissioned. This

substantially reduced the total losses attributed to equipment damage. However, in some cases, doubts were expressed concerning the long-term reliability of such refurbished equip- ment. Suppliers have generally been in favour of replacing the damaged equipment with new stock and have refused to provide maintenance con- tracts for the recommissioned equip- ment.

The majority of the arson cases’ and 50% of fire and explosion cases occurred at night, usually between 9.00 pm and 6.00 am when the instal- lation was either unmanned or left in the hands of nightwatchmen or secu- rity guards.

Out of the 47 cases of arson re- ported:

47% of the fires were started out- side the computer installation either in the same office block or in adjoining warehouses and then spread into the installation. 30% were started in the computer room. 23% were started in the installation but outside the computer room, such as in terminal areas. In several cases the fire did not penetrate into the computer room but the equip-

Outside causes

ment was nevertheless damaged by smoke and soot which became deposited on computing mach- inery.

l 26% of the fires were attributed to

electrical causes such as overheat- ing, short-circuiting or malfunc- tioning of electrical components.

l 11% originated from the air condi- tioning units.

l 8% were due to poor housekeeping

in the installation. l 4% were due to outside causes. l 32% were the results of arsonists

and IRA bombers. l 19% did not have enough support-

ing evidence to establish the true cause of fire, but it could be safely assumed that a good proportion were arson cases. In most com- puter fires there was only limited direct equipment damage by heat, burn out or explosion. Real dam- age was caused by widespread con- tamination of equipment by pro- ducts of combustion, hydrochloric

acid resulting from burning PVC, and water used for fire fighting.

The above statistics emphasize the importance of partitioning off com- puter rooms from adjoining areas by using effective fire resistant insulation material. Regular preventive main- tenance of electrical equipment and automatic fire protection equipment

coupled with good fire procedures enforced in the computer room would serve to reduce both the likelihood and severity of electrical fires.

Staff morale could rapidly deterior- ate if computer staff were required to operate in remote backup installations for long periods while awaiting the delivery of computer equipment

The traumatic experience, as re- lated by fire victims, of overcoming system recovery problems after the disaster, processing insurance claims with loss adjusters, and then slowly returning to normal business opera- tion, all demonstrate that fire preven- tion is much cheaper and less painful than disaster recovery.

which may have fallen into obsoles- cence at the time of the disaster. Some companies were able to make rapid

recovery because they had adequate offsite storage of files and stationery

and effective backup arrangements to maintain the processing of vital busi- ness systems. Staff were mobilized

quickly to respond to the disaster according to laid-down procedures and helped to cut losses through speedy recovery.

Most equipment suppliers were sympathetic to disaster victims and, in many cases, were able to deliver

replacement equipment within a short time. With an increasing number of

vendors offering warm and cold standby services, as well as ready replacement minis on lorries, the problems of seeking out backup in- stallations with adequate resource capacity and compatible software, as well as providing temporary accom- modation to rehouse computing equipment, can now be tackled with outside help.

Portable recovery centres have the added advantage of providing tem- porary premises nearby to minimize the time and cost of rerouting com- munication cables.

Water and flood damage

Thirteen cases of equipment damage

For some installations located in

by water and flooding were docu-

low lying areas or below flood levels in

mented in the casebook, although many more have been reported. Each

the vicinity of rivers, a burst pipe or

severe winter has brought with it a number of cases of burst pipes, result- ing in water damage or flooding of

torrential rains which swelled the

computer equipment located either on the floor below or immediately under

river banks have separately resulted in

the fractured piping. In the majority of cases individual losses were in the range of &10000-&40000, this being the cost of replacing the damaged equipment.

extensive water damage to computing equipment. In one instance $5 million

worth of point-of-sale equipment was found drenched in sewage water.

In another instance, because the installation was in the heart of the building, immediate access was im- possible. As a result, the salvage operation was seriously delayed and all the equipment had to be written off. Also, all the system documenta- tion was destroyed and staff had to work on night shifts in the remote backup installation for three months after the disaster.

It is worth noting that even though an installation may be located above the flood level in a low lying area and is immune from water damage through flooding, serious disruption to processing could still result if staff cannot get to the installation through disruption to local transport or vehi- cular access in the area.

Two cases were reported of water seeping through the ceilings from flat roofs following heavy storms, one

occasioned by sleet blocking the gut- ters and drainage, and the other via cracks on the roofing material de- veloped through drastic temperature changes from a hot dry spell to a period of heavy storms.

Five cases were recorded of water damage resulting from fire-fighting either on the floor above the installa- tion or in the adjacent area. In one instance, the ceiling of the installation was treated with a water sealant. The

damp equipment was dried out after- wards and no damage was done. The same treatment was not, however, applied to the telephone exchange in the next room and the equipment was completely corroded by the dirty water seeping through the ceiling. In another case it took several days to retrieve the fire safe located below the computer room after a major fire in the building. The door to the safe was warped and when it was eventually opened all the paper contents were found to be soaked in water from fire-

fighting. Three of the six backup

8 data processing

discs were unusable and 5% of the source programs in the form of punched cards could not be used. There was no backup for system documentation and the originals were

destroyed in the fire, causing serious problems in system recovery.

Several installations have expressed concern that over a number of years, through several major upgrades of

hardware and additions to disc and tape storage, their computer area has now approached the limit of critical floor loading. Further aggravation in weight resulting from water accumu- lating on the floor void either through leakage from air conditioning pipe- work malfunction or fire-fighting from higher floors could cause struc- tural damage to the building.

Early detection of flooding in the floor void could be helped by instal- ling moisture detectors in strategic locations. The risk can also be pre-

vented by incorporating drainage faci- lities in the concrete flooring in the building design.

Building structural defects

Four cases of ceiling collapse were reported. In one case, a pipe burst above an office building caused water to escape and seep through the floor to cause a build-up in the ceiling area. Eventually the weight of water proved too much and the ceiling collapsed, severely damaging two VDUs and covering the computer with debris.

In another case, the cause of the collapse was due to a fault discovered in the building material and the struc- tural design of the building. All the computer equipment had to be re- located in another building. Two other cases involved the collapse of the false ceiling above the computer room. Although the computer equip- ment did not sustain damage, the computer room was covered in rubble and processing could not continue.

In one company the installation was visited by the computer security offi- cer who pointed out the low hanging

~0126 no 8 October 1984

false ceiling and the risk of imminent collapse. The warning was dismissed by the computer manager, claiming

that the building had been in that state for several years. Late that night the security officer received a panic telephone call from the manager - the false ceiling had just collapsed.

Cracks were discovered this sum-

mer in a 20 year old building housing a large computer installation. This

was thought to be due to a combina- tion of ground shrinkage and heavy traffic. Fortunately only the terminal room was affected. The installation’s

contingency plan was invoked. Staff worked overnight to relocate the ter-

minals and users to other offices. The network software had to be modified

to accommodate the changes. Drilling by workmen as part of a

refurbishment programme acceler- ated the collapse of a floor-supporting

pillar in a 30 year old building hous- ing another computer installation. The pillar was already eroded by acid and the weakened support gave way

through drilling. As a result part of the building dropped by four inches. The installation was declared unsafe and staff were denied access for six weeks.

Lightning strike, power surge and failure

Fifteen cases of lightning strikes on computer installations have been re- ported in the UK, all within the last

two to three years. One installation was hit five times in one year and another suffered the same fate in the space of two years. In each case the lightning struck after a violent thun- derstorm and caused a strong power surge which ran down the cables and affected the communications control- ler and terminals. In two cases, the power surge was carried to the central processors and damaged some of the printed circuit cards.

Some of the installations affected had lightning conductors installed and others did not. One organization

suffered a total loss of f400000 worth of equipment - two-thirds of the value of the entire communications

and computer system. The cause of damage to communi-

cations equipment from a lightning strike has so far remained a mystery, although a number of theories have been put forward and counter-

measures proposed. Currently, the Insurance Technical Bureau in Lon-

don is setting up a project to study the causes and effects of lightning on communications and computing equipment. The Bureau will recom- mend effective protection against future occurrences.

Power cuts and power failures have also caused disruption to processing. A general power failure in London caused one installation over 100 hours of downtime in two months. The original power failure had caused

some parts of the computer to fail. When these elements were replaced with new ones, these in turn caused other, older elements to blow. It was

only when all the elements were replaced that the computer perform- ance was stabilized.

Radar interference

At least two cases of electromagnetic interference due to radar transmission

from ships nearby have been re- ported. In one case, an installation experienced random malfunctions of its disc subsystem which were mani- fested in the form or erroneous disc transfers, corruption of files and mis- reading of data. The faults occurred only sporadically and would suddenly stop and not reappear for some six to eight weeks. One day, managers noticed that a ship docked in a wharf nearby was testing its radar and at the same time the computer was affected by the mysterious data corruption. They observed that the corruption ceased when the testing stopped.

Further investigation revealed that the dates and times recorded in the ship’s log for radar testing coincided

9

exactly with the dates and times of the computer corruptions.

The risk of electromagnetic inter- ference is high for installations located in the vicinity of airports, docks and defence establishments. The only pro- tection against this form of inter- ference is through shieldmg the entire building or computer room with earthing material, known as a Faraday cage, which inhibits the flow of elec- tromagnetic radiation into the opera- ting equipment.

Industrial action

VAT revenue and repayment of El00 M a week to traders. Payments of grants and subsidies to industry of between f E&f200 M a week were halted. The strike helped cause a rise in the UK Bank Minimum Lending Rate to 17%. Government public borrowing was increased to f 1 B. At its peak, f 1.4 B in Government revenue and payments were held up.

This type of selective strike demon- strates how vulnerable the Civil Ser- vice computer centres are to industrial action. After the strike was settled,

major loss in an area where such equipment is located, or a serious disruption to the power supply could paralyse the entire nerve centre for voice and data communication in the company. Many of the business acti- vities may have to be held in limbo until management successfully re- stores the automated facilities.

As more personal computers and word processing equipment are intro- duced into an organization, the un- suspecting first-time user may not attend to such needs as offsite backup

the Civil Service formulated various of floppy discs and other magnetic contingencv olans to distribute nro- files. Thus a fire in the office could

A number of cases in which industrial cessi& and to give preferential &eat- create havoc to the business operation action has disrupted computer proces- ment to computer staff in future pay and system recovery may be ex- sing have been recorded in the last negotiations. This policy directly tremely difficult and time consuming, few years. In 1979 two strikes affected caused the collapse of the 1982 Civil if it is possible at all. the Post Office and the UK Govern- Service strikes. ment. The selective closure over a period of 18 weeks of computer centres for telephone billing, cargo handling and System X development work virtually stopped the UK Post Office from processing and despatch- ing one million telephone bills per week. Processing of payroll was not disrupted, however, which caused the cash flow problem to deteriorate fur- ther. At the end of the strike the backlog in revenue had escalated to f 10 M per day and reached a total of f 1 B. Loss of profits came to f 130 M.

The Civil Service Union used the 1974 Post Office strike as a model to formulate its strategy for selective strikes. It conducted a systematic checking of all important work car- ried out by various Government com- puter centres and picked its targets so that the minimum number of key staff needed to be called out to inflict maximum damage on the Govern- ment.

In the end 1300 computer staff at selected installations were called out on strike in support of a wage claim for 340000 union members. Dues were levied from all union members to make up the full pay to compensate the strikers. The strikes caused delays in the collection of &500 M a week in

Standby planning Future implications

As more organizations move into dis- tributed processing, with intelligent terminals or minis linked via a corpor- ate network, the total cost of security and protection is likely to be thinly spread over all sites. As a result, the risk of a local disaster such as fire or arson is likely to be higher than that in a central mainframe installation. However, the severity of corporate loss arising from such a disaster is unlikely to be of the same scale as that from a disaster in the central installa- tion. Also, the prudent corporate DP executive can direct individual sites to procure mutually compatible hard- ware, software and storage devices to provide effective mutual backup be- tween sites in the event of both major and minor disruptions.

Over the last few years, many organi- zations have felt increasingly uneasy about their growing dependence on computer processing in their various day-to-day business and administra- tive functions. As a result the com- puter service manager or data centre manager has been charged with the responsibility of obtaining some form of disaster standby provision to cater for possible disruptions in data pro- cessing. At the very least, certain critical systems such as payroll can be run at another installation for a short period. Some have gone as far as duplicating entire computer facilities at a second site.

By and large, there are three types of standby facilities available for con- sideration:

The electronic office is a mixed l Hot standby: Twin computers are blessing. Operational backup through used with full switching capability swapping or sharing the use of work- to share common peripherals and stations should not pose any serious mass storage. If one computer is problems in local area networks. But, down, processing can continue by unless adequate redundancy and switching jobs over to the second backup of communications control computer. Unfortunately the com- equipment is provided, and kept on puters tend to be located back-to- split sites, and alternative power sup- back and housed in the same room, ply is available to the network, a together with the communications

10 data processing

control equipment, to facilitate swift changeover. If a disaster such as a fire strikes in that room, all the equipment could be affected by the fire and additional remote disaster standby would need to be con-

sidered. l Warm standby: A remote installa-

tion is available to take on the processing of systems from the disrupted installation. A minimum

delay of 24 hours is normally ex- pected in the changeover of instal- lations. The standby processing capability is normally restricted and would not handle the full processing requirements of the dis- rupted installation for any length of time. To obtain standby processing

from a number of backup installa- tions would mean reproducing files, job control and careful plan- ning of staff transport and move-

ments to ensure the processing load can be split up and processed at different locations, probably all with slightly incompatible hard-

ware, operating software and mass storage media. Planning for stand- by processing of online systems poses yet another major headache for the computer service manager, especially if these systems serve a large number of remote users scat- tered across a wide geographical area. Some service vendors offer the temporary replacement of a full minicomputer installation within 24 hours, with the equipment transported to a designated site.

l Cold standby: An empty installation is available at a remote site to receive replacement equipment for the destroyed site. The cold stand- by site provides temporary accom- modation for the new computer installation for up to six months.

Portable computer centres are also available to provide temporary accommodation for replacement equipment and are erected near the destroyed computer installation. Proximity to the destroyed site would minimize the delay and cost

of rerouting existing communica- tion cable links to the temporary location.

In BIS’s experience, computer disas- ters are rare. Every year 20-30 fires

involving computers are reported in the UK among a total population of 50 000-60 000 installations. Thus, the chance of a company suffering from a

computer fire is roughly one in a thousand. Also standby provisions

cost money, more so for hot and warm standby than others. While many computer service managers are con- stantly reminded by senior executives that reasonable steps should be taken to ensure that the company does not

go under or suffer a major setback, at the same time they have experienced great difficulty in obtaining a realistic budget for standby provisions.

In practice, many installations have resorted to a gut feeling of what systems the operations manager con-

siders to be critical to the company, which user shouts the loudest and what is currently in vogue for standby offerings. He/she then makes some provisions on whatever resource can

be squeezed out of an already stretched DP budget.

There has rarely been a concerted effort to involve senior business executives in discussions on standby considerations. Nor have we come across many organizations which in-

sist on incorporating standby require- ments and backup options in the design of new business systems, al- though this would ultimately simplify the task of computer operations in any subsequent standby planning. Fur- thermore, very few business execu- tives appreciate that proper contin- gency planning requires a consider- able amount of time and resources from both computer management and business users.

This is one area where outside consultancy expertise can help to guide computer users to define their standby requirements, identify a realistic budget, the necessary re-

source and schedule for contingency planning, assist DP management to review the various standby options, prepare contingency action plans and design a series of tests to ensure the

standby provisions are practical and the action plans for the disaster re- covery team are properly coordinated and proven to work.

Top-down approach

A standard top-down approach offers a suitable methodology for contin- gency planning. The top-down ap- proach comprises four stages:

l business impact review

l contingency planning strategy re- view

l preparation of action plans l testing of standby facilities and

action plans

Without a proper appreciation of the extent of damage to a business arising from an extended computer disrup- tion, it is futile to plan and allocate

appropriate resource for standby pro- visions to mitigate that loss.

An objective business impact re- view, conducted with senior com- puter users, would serve to establish the extent of business dependence on data processing. It would also high- light the financial and business impli- cations of both short- and long-term interruption of computer service. The following are highlights of the find- ings of Rusiness Impact Reviews con- ducted by BIS:

l &I M of new business per week for a financial institution would be lost throughout the disruption period.

l &2-3 M of business per week in a manufacturing company would be at risk. If the products being manufactured are seasonal, a serious disruption in the peak sea- son could cause permanent damage to the company’s reputation in the market place.

l Invoicing and stock control from a

~0126 no 8 October 1984 11

commodity company would run completely adrift. Besides having to finance a cash flow delay of f21 M per month, customer dis- satisfaction would result in some profitable outlets being lost per- manently to competitors. Calculations of wages to hourly paid employees in a brewery would be held up. If disruption lasts for more than two weeks, unless agree- ment can be reached on alternative pay procedures, union withdrawal of labour would be almost certain. Monetary losses through opera- tional inefficiency and lost business opportunities in a large multina- tional would come to &19 M for the first week, rising to &50 M a month, with a further &270 M at risk. Cash-flow delays of &40 M would be incurred in the first week, rising to El70 M for a month, with a further &270 M at risk.

In addition to assessing the business impact of key application systems, and where appropriate quantifying business interruption losses, the re- view also identifies the operational requirements of various business sys- tems and possible knock-on effects from related systems or applications sharing common databases.

Priorities for standby and recovery can then be established based on the relative ‘criticality’ of systems in terms of the extent of potential losses incurred through business interrup- tion. The standby and recovery con- siderations would also take into account the critical response time requirements of each business system and offer an appraisal of the manual backup and system recovery problems for backlog clearance and the recrea- tion of lost data.

For those systems which the board deems can only tolerate a disruption period of a few hours - by virtue of their critical nature to business opera- tions - hot standby provisions should be made available to duplicate the necessary computing and com- munications facilities. For those which can only be allowed to be disrupted for a few days at most, the total operational resource require- ments can be worked out to provide warm standby inhouse or to negotiate with outside vendors for a minimum configuration of hardware, mass stor- age, software and communication facilities - to be made available on demand. In addition, cold standby provisions should be sought to accom- modate the replacement equipment.

Various service companies offer dif- ferent warm and cold standby options to cater for various computing and communications needs. It would be in the interest of the installation to shop around to obtain the best value for money whilst catering for the installa- tion’s requirements, e.g. proximity of the standby location and availabilty of the necessary communication links to service remote terminal users throughout the disruption period.

BIS regards the business impact In arriving at the total cost of review as the most suitable vehicle to standby provisions through the above highlight to senior management the process, budgeting for contingency vulnerability of various business sys- planning has involved the close colla- tems. The review can only be done by boration of senior executives, DP working closely with users, instead of management and business users. If

‘going it alone’ within DP and pre- tending that everything will be all right in a disaster.

The board of directors is ultimately responsible for the interests of the company. By right, it should be told of the real situation regarding the effects of computer disruption and the losses that might be incurred. The board is then in a position to decide on the corporate policy for contin- gency planning and to determine and allocate standby and recovery priori- ties to various business systems, with the aim of minimizing corporate losses and business impact.

the cost for hot or warm standby turns out to be too high and is unacceptable to the board, other options may be considered, e.g. reduce the number of critical systems on warm standby, thus exposing the company to a higher level of potential business in- terruption loss from a disaster. The final decision should be capable of withstanding close scrutiny from a business point of view.

Speeding up recovery requires pro- per coordination of action plans for all key DP areas, as well as making necessary provisions to meet all the operational requirements of such plans, e.g. offsite storage of backup files and documentation. In addition, computer users must be involved in the contingency plan preparation, enabling them to decide on any addi- tional clerical support which might be required to cope with an extended computer disruption.

Regular review and testing of standby facilities becomes more urgent if the installation requires immediate backup at short notice. Appropriate job controls and opera- ting instructions for the standby site should be prepared and checked for accuracy before the event.

Discrepancies in accounting and file-naming conventions should be resolved when the standby arrange- ments are first contemplated. There is little room for experimentation or improvization at a time of extreme commotion. Valuable time and re- sources could otherwise be lost un- necessarily.

References

1 Wong, Ken and Farquhar, Bill, Computer Disaster Casebook, BIS, London (1983).

2 Computer Crime Casebook, BIS, London. q

Security and Privacy Division, BIS Applied Systems Limited, Maybrook House, Black- friars Street, Manchester M3 2EG, UK. Tel: 061-831 7031.

12 data processing