cnic audit 2008

7/28/2019 Cnic Audit 2008

1/11

Secre

taryofState

Audit

Report

BillBradbury,

SecretaryofState

CharlesA.

Hibner,Director,AuditsDivision

Report No. 2008-21

July 15, 2008Department ofAdministrative Services:State Data Center Review

SummaryPURPOSE

The Department of Administrative Services(department) is responsible for providingcentralized services to state agencies, includingcomputer networks and processinginfrastructure. During 2005, the Oregon StateLegislature approved funding for theComputing and Networking Infrastructure

Consolidation (CNIC) project to consolidate 12state agency data centers into one facility.

The primary purpose of this audit was toevaluate the status of the departments efforts toreengineer the State Data Center (SDC)environment to achieve CNIC projectobjectives. In addition, because of the criticalityof SDC operations, we also evaluated controlsgoverning the current SDC computingenvironment.

RESULTS IN BRIEFBased on our audit work we found:

Important data center consolidationobjectives have not yet been achieved. As aresult, it is unlikely that the anticipatedsavings or operational benefits associatedwith the CNIC project, such as enhancedenterprise disaster recovery and securitysolutions, will occur.

Operational controls did not sufficientlyaddress service level agreements withcustomers, performance and capacitymanagement, standard operatingprocedures, configuration management, or

software licensing requirements. The department was ill-prepared to timely

resume data center operations or assistagencies in their efforts to restore criticalcomputer applications after a majordisruption.

The department had not provided a securecomputing environment for SDC clients.

Because of the sensitive nature of systemsecurity, we issued a separate report tocommunicate findings and recommendations inaccordance with ORS 192.501 (23), whichexempts such information from publicdisclosure.

RECOMMENDATIONSWe recommend that the department:

establish an appropriate projectmanagement framework and allocateresources to ensure data centerconsolidation objectives occur, includingdetailed plans directing how, when, and towhat degree it will consolidate networkservers, system tools, mainframe operationsand operating system platforms;

allocate resources to ensure the SDC gainsfull operational control, formalizes servicelevel agreements with agencies, establishesstandard operating procedures, providesperformance and capacity management,implements a centralized configurationmanagement system, and ensures controlsto track system software licenses;

create and test disaster recovery plans toensure timely restoration of SDCinfrastructure and systems, and coordinateand formalize disaster recovery plans formission critical applications; and

implement recommendations included inour confidential security report.

AGENCYS RESPONSEThe Department of Administrative Servicespartially agrees with the recommendations. Thedepartments response is attached to this report,beginning on page 6.

1

7/28/2019 Cnic Audit 2008

2/11

S e c r e t a r y o f S t a t e Audit Report No. 2008-21 July 15, 2008

Background

The Department ofAdministrative Services(department) is responsible forproviding centralized services tostate agencies, including computernetworks and processing

infrastructure. State statutespecifically directs the departmentto coordinate statewide planningand activities related to theacquisition, installation and use ofall information andtelecommunications technology forthe state.

During 2005, the Oregon StateLegislature approved funding forthe Computing and NetworkingInfrastructure Consolidation(CNIC) project to consolidate

twelve state agency data centersinto one facility. The total cost ofthis project was projected to beapproximately $63.6 million,consisting of approximately$20 million to construct a new datacenter building and $43.6 million toequip and configure theconsolidated operatingenvironment. The project wasintended to reduce future costswhile maintaining or improvingservice levels.

In September 2006, the OregonAudits Division completed aninitial risk assessment of the CNICproject. The resulting report,Department of AdministrativeServices: Computing andNetworking InfrastructureConsolidation (CNIC) RiskAssessment, identified severalweaknesses in the departmentsproject planning and managementprocesses that adversely affectedthe integrity and viability of the

project.

In response, department managersagreed that initial project planningand management was inadequate.

They further indicated they wouldcorrect the deficiencies identified inthe report by generally improvingidentified weaknesses and byreengineering the environment after

agencies relocated to the State DataCenter (SDC).

By the beginning of 2007, 11agencies had transferred their datacenter operations to thedepartments SDC. Thoseoperations include statewideenterprise applications and criticalagency applications.

The primary purpose of this auditwas to evaluate the status of thedepartments efforts to reengineerthe SDC environment to achieveoriginal CNIC project objectives.In addition, because of thecriticality of SDC operations,another purpose was to evaluate thecontrols governing the SDCcomputing environment.Specifically, we chose to evaluate

established controls over datacenter operations, disasterrecovery, and security.

Audit Results

Significant Data Center Consolidation Objectives

Have Not Yet Been Achieved

As outlined in our previous auditreport, an effective project

management framework isnecessary to provide clear directionregarding project scope andboundaries. It also provides aroadmap for successful projectcompletion and closure. Inaddition, project plans should be inplace that detail how majorobjectives will be achieved.

At the conclusion of our previousaudit of the CNIC project, agencymanagers were in process ofmoving their data centerinfrastructure and operations to theSDC. At that time, departmentmanagers had not yet developedcomprehensive plans to achieveproject objectives. Instead, theyopted to relocate agency datacenters to the SDC in their as-isstate, stabilize operations, and thenproceed with projects to reengineerthe environment.

During this audit, we evaluatedthe departments efforts to resolvedata center consolidation issues andto reengineer the SDCenvironment. Based on the resultsof this work, we concluded thatimportant data center consolidationobjectives had not yet been

achieved. Specifically, thedepartment had not madesignificant progress toward:

y defining the detailed end-statearchitecture of the SDC;

y reducing the number of networkservers or operating systemplatforms;

y providing additional enhancedenterprise disaster recovery orsecurity services for SDCclients;

y reducing SDC staffing levels;and

y consolidating data centeroperating procedures.

In addition, some agencies wereunable to successfully relocate theiroperations to the SDC. In fact, atthe conclusion of this audit, SDCmanagers indicated they will bemoving approximately 200 networkservers out of the SDC because itcurrently does not have sufficientpower capacity to safely host thoseservers. The department indicatedthe above power issue was atransitory condition that would beresolved by consolidating theserver environment throughvirtualization. However, we notedthe department did not have adefinitive plan to achieve this goal.

Furthermore, the Department ofEducations data center did notparticipate in the data center

migration as originally planned. Itsmove was delayed pendingresolution of potential legalquestions regarding confidentialityof student records. In light of theSDC power capacity problemsmentioned above, we concludedmovement of the Department ofEducations data center to the SDCwould likely be infeasible.

2

7/28/2019 Cnic Audit 2008

3/11


The above consolidation issuesexisted because the departmentcontinued to lack an appropriateproject management framework.Components of that framework thatwere noticeably absent included:

y a dedicated managementstructure that was responsiblefor governing overallconsolidation efforts;

y project plans defining how,when, or to what degree CNICproject objectives would, orcould, be achieved; and

y sufficient dedicated resources,including the necessary staffing,to resolve the above issues orachieve intended results.

The overall effect of not

achieving CNIC objectives issignificant. Justification for theproject centered on anticipated costsavings and operating efficienciesto be achieved by consolidatingdata center infrastructure,operations and human resources. Ifconsolidation does not occur, it isunlikely that actual cost savingscan be achieved to allow thedepartment to recoup its CNICinvestment of approximately$63.6 million. In addition, promisesof increased operational benefitssuch as enterprise disaster recoveryand security solutions will likelynot materialize.

We recommend that departmentmanagement establish anappropriate project managementframework, and allocate resources,to ensure that data centerconsolidation objectives occurwithin the current SDCenvironment. That frameworkshould include detailed plans

directing how, when, and to whatdegree the SDC will consolidatenetwork servers, system tools,mainframe operations andoperating system platforms.

Agencys Response:

The departments response isattached to this report, beginningon page 6.

Some State Data Center Operations Were Not

Uniformly or Effectively Controlled

Providing a controlled and stableoperating environment for anenterprise data center includes

managing basic InformationTechnology (IT) support functionssuch as:

y responding toand requests,

customer needs

y resolvingproblems,

incidents and

y establishing service-levelagreements with customers,

y scheduling and prioritizing jobsand processes,

y managing and monitoringperformance and capacity,

y managingand

the configuration,

y ensuring compliance withoutside requirements.

We found that the SDC hadvarious processes for responding tocustomer needs. These controlsincluded establishing a service deskto provide customer support,

manage incidents, and receiveservice requests. The SDC alsomanned a command center tomonitor the status of operatingsystems, jobs in progress, andnetwork device status. In addition,it had processes to track the statusof service tickets, change orders,and support requests. However, theSDC had not:

y established comprehensiveservice-level agreements withits agency customers;

y developed standard operatingprocedures for job scheduling,backup, and tape management;

y established a planning processfor review or resolution ofperformance and capacitymanagement issues;

y developed processes to managethe configuration of SDCinfrastructure; and

y implemented controls to ensurecompliance with softwarelicensing requirements.

These issues existed primarily

because the department did nothave sufficient resources or themanagerial structure toappropriately resolve them. As wepreviously discussed, thedepartment chose to move agenciesinto the data center in their as isstate, with the intent that SDC staffwould subsequently reengineer theenvironment. However, sincemigration, SDC management hasfocused resources on providingongoing services to customers. As

such, staff has not been available toestablish new data center controls,such as developing acomprehensive configurationmanagement system. In addition,some operational requirements,such as establishing service-levelagreements and standard operatingprocedures, remained undevelopedbecause SDC management had notassumed operational control ofsome agency platforms orestablished consensus withapplication owners regardingoperating requirements andexpectations.

The weaknesses noted abovedirectly affected the SDCs abilityto provide necessary and cost-effective services to its clients. Forexample, configurationmanagement weaknesses affectedthe SDCs ability to develop andimplement effective disasterrecovery plans. In addition,configuration and capacity

management issues inhibitedefforts to consolidate SDC systemsand resources. Furthermore,without formal service levelagreements, the department and itscustomers remained uncertainregarding how critical operatingrequirements would be fulfilled.

We recommend departmentmanagement allocate appropriate

3

7/28/2019 Cnic Audit 2008

4/11


resources to develop andimplement controls to:

y ensure full control of the SDCoperating environment andestablish consensus with systemowners, via formal service levelagreements, regarding operatingrequirements and expectations;

y establish standard operatingprocedures for job scheduling,backup, and tape management;

y plan for and resolveperformance and capacitymanagement issues;

y maintain a centralizedconfiguration managementsystem to document importantinformation regarding theoperating environment; and

y track software licenses.Agencys Response:


The SDC Lacked Appropriate Disaster

Recovery Plans The SDC hosts numerous mission

critical computer applications, as

well as enterprise InformationTechnology (IT) infrastructure.Therefore, department managementis responsible for ensuring that datacenter infrastructure, includingnetworks, operating systemenvironments, and data storagefacilities, can be timely restored inthe event of a disaster or othermajor incident. In addition, thedepartment shares theresponsibility for coordinating andprioritizing the restoration efforts

for agency computer applicationshosted at the SDC with businessowners.

Generally accepted controlssuggest that organizations haveformal continuity plans thatmitigate the business risksassociated with a major disruptionor loss of IT services. Those plansshould contain detailed responseand recovery procedures to timely

bring the business back to itsbefore-incident state. Continuityplans should also be periodicallyupdated and tested to ensure theirviability. Because the SDC is aservice provider, it should establishformal service level agreementswith its customers to clarify and

coordinate disaster recoveryresponsibilities and expectations.

Those agreements should defineeach partys specific expectationsduring a recovery effort, and shouldaddress critical issues such asstaffing, required recoverytimelines and resource allocation.

We evaluated the SDCs controlsover disaster recovery. Based onthat work, we concluded the SDCwas ill-prepared to timely resumedata center operations or assist

agencies in restoring their criticalcomputer applications after a majordisruption.

Items of most concern includedthe following:

y The SDC did not have formal orcomplete business continuity ordisaster recovery plans for itsoperating systems, networks,data storage systems, or systemutilities.

y SDC staff had not testedexisting disaster recoverystrategies.

y The department had no formalservice level agreements withagencies addressing theirdisaster recovery needs,requirements or expectations.

Our prior audits of the statesmajor data centers and criticalcomputer applications identifiedinsufficient disaster recoveryplanning as a weakness. CNIC

project planners also identifiedinsufficient disaster recoveryplanning as a significant projectrisk. Based on the results of thisaudit, these conditions have notsignificantly changed. Weconcluded that these findingscontinued to exist because thedepartment had not placedsufficient priority, or allocated

sufficient resources, to resolvethem.

We recommend that departmentmanagement assign a higherpriority to disaster recovery byallocating sufficient resources tocreate and test disaster recoveryplans to ensure timely restorationof the SDC operating environment.

Those plans should also ensure thatSDC efforts are coordinated withagency expectations andrequirements to recover missioncritical computer applicationshosted at the SDC. The plansshould be formalized throughservice-level agreements.

Agencys Response:

The departments response isattached to this report, beginning

on page 6.

The Department Did NotProvide For a Secure

Computing Environment

The department is responsible foroverall security of the SDC and forproviding various other securityservices at the enterprise level.

These responsibilities include butare not limited to:

yensuring physical and logicalsecurity of SDC resources;

y monitoring state network trafficto identify, and react to,security threats;

y conducting vulnerabilityassessments of agencyinformation systems; and

y establishing a state informationsystems security plan andassociated standards, policiesand procedures.

We evaluated the departmentsefforts to address theseresponsibilities and concluded thatthe department had not provided asecure computing environment forSDC clients.

Because of the sensitive nature ofsystem security, we have issued aseparate report outlining specificdetails of our findings, as well as

4

7/28/2019 Cnic Audit 2008

5/11


recommendations to improvesecurity. That confidential reportwas prepared in accordance withORS 192.501 (23), which exemptssuch information from publicdisclosure.

We recommend that departmentmanagement implement therecommendations included in ourconfidential report.

Agencys Response:


Objectives, Scope andMethodology

The purpose of our audit was toevaluate the status of consolidationand the general computing controlsat the State Data Center. Ourspecific audit objectives were:

1. Determine the status ofconsolidation.

2. Determine whether thedepartment had implementedgeneral computing controls toensure continuous service bythe State Data Center asrequired in the event of adisruption.

3. Determine whether thedepartment had implementedgeneral computing controls toensure the State Data Centerprovided a stable andcontrolled operatingenvironment.

4. Determine whether thedepartment had ensured systemsecurity of State Data Centeroperations by maintaining theintegrity of information and

processing infrastructure, andminimizing the impact ofsecurity vulnerabilities andincidents.

To achieve these objectives, weinterviewed various departmentpersonnel, observed operationsprocesses, reviewed departmentdocumentation, and conductedtests. Tests included review oflogical access, evaluation of project

planning documents, andverification of the existence ofsupporting documentation.

We also reviewed the status ofthe findings and recommendationsfrom our prior risk assessment ofCNIC that were relevant to ourcurrent audit objectives.

We used the IT GovernanceInstitutes (ITGI) publication,Control Objectives forInformation and Related

Technology, (CobiT) to identifygenerally accepted and applicableinterim control objectives andpractices for information systems.

We conducted our auditaccording to generally acceptedgovernment auditing standards.

5

7/28/2019 Cnic Audit 2008

6/11

Department of Administrat ive Services Response to Audit Report No. 2008-21

6

7/28/2019 Cnic Audit 2008

7/11


7

7/28/2019 Cnic Audit 2008

8/11


8

7/28/2019 Cnic Audit 2008

9/11


9

7/28/2019 Cnic Audit 2008

10/11


10

7/28/2019 Cnic Audit 2008

11/11

Secretary of StateAudits Division

255 Capitol St. NE, Suite 500

Salem, OR 97310

Auditing to Protect the Public Interest and Improve

Oregon Government

AUDIT MANAGER: Neal E. Weatherspoon, CPA, CISA, CISSPAUDIT STAFF: Erika A. Ungern, CISA

Constance S. Bailey DEPUTY DIRECTOR: William K. Garber, CGFM

Courtesies and cooperation extended by officials and staff of theDepartment of Administrative Services were commendable and muchappreciated.

This report, a public record, is intended to promote the best possiblemanagement of public resources. Copies may be obtained:

Internet: http://www.sos.state.or.us/audits/index.htmlPhone: at 503-986-2255

Mail: Oregon Audits Division255 Capitol Street NE, Suite 500Salem, OR 97310

11
http://www.sos.state.or.us/audits/index.htmlhttp://www.sos.state.or.us/audits/index.html

cnic audit 2008

Documents