Doc No. 4

NSW HealthIndependent Review of Lerner FirstNet

Contents

ContentsI

Purpose of this report

32

Executive Summary

43

Introduction

144

Observations

215

Recommendations

39Appendix A - Interview List

44Appendix B - Acronyms

46Appendix C - Source Documents

48Appendix D - Chronology of Events

54

2

Purpose of this report

1 Purpose of this reportCerner FirstNet is the IT system selected by the NSW Department of Health to support the operation ofemergency departments within the majority of public hospitals. Planning of this initiative dates back to2002 with the first implementation commencing in 2008. The Cerner FirstNet system has now beenimplemented in over 50 hospitals throughout NSW, and is a core component of the broader NSW Healthintegrated electronic medical record ("eMR") system.

Following media reports in early March criticizing the NSW Health Cerner FirstNet EmergencyDepartment ("ED") system, the Minister for Health committed to undertake an independent review of theimplementation of the Cerner FirstNet Emergency Department system in NSW hospitals. The NSWDepartment of Health commissioned Deloitte to conduct this review of the Cerner FirstNet system andthe effectiveness of its implementation. Specifically, the review has assessed the suitability of the FirstNetsystem to meet the clinical, functional and useability requirements of the users, and whether its continueduse posed any risk to clinical safety.

DisclaimerIt should also be noted that the findings and recommendations presented in this report are based on

information and documentation provided by representatives of NSW Health and various externalstakeholders. We have not audited or independently verified that information. If the information isincorrect, incomplete or out of date (i) our report and its contents may be inaccurate and/or not meet yourrequirements and (ii) we will not be responsible for any loss suffered as a result of our reliance on theinformation. Also, while this report contains advice and recommendations, all decisions in connection withthe implementation of such advice and recommendations will be the responsibility of NSW Health.

3

Executive Summary

2 Executive Summary

.1 eMR. and F

In March 2000 the Report of the NSW Health Council (Menadue Report)' gave renewed focus to theimplementation of a number of information management and technology initiatives to better supportclinical practice at the point of patient care. In part, as a response to the Report, NSW Health submitteda Point of Care Clinical system ("PoCCS") business case in 2002, the objective of which was to obtainapproved funding to implement a common State wide system, to provide an interactive clinical decisionsupport facility to clinicians, supporting the day-to-day clinical management of their patients within thehospital setting. The business case identified Cerner as the preferred Electronic Medical Record (eMR)system and a contract (head agreement) was signed between NSW Health and Cerner on 27 September2002.

Subsequent to the original contract, orders for the supply of eMR modules were raised by the variousArea Health Services'. The first instance of the eMR, including the FirstNet3 Emergency Department (ED)component, under the head agreement was implemented at St George Hospital in the South EastSydney and Illawarra region in September 2008. Since then the eMR including FirstNet has beenimplemented at 594 sites. The initial rollout program is due to complete by late 2011.

2.1.2 F

-:J_nd ,

.view

A number of issues with FirstNet have been raised by ED clinicians, other hospital staff, professional

bodies and academics. Issues raised include FirstNet's potential to negatively impact on efficient hospitaloperations and more critically, patient care.

Following reports in the media in March of criticisms of the NSW Health Cerner FirstNet ED system madeby Prof Jon Patrick at the University of Sydney, the Minister for Health committed to undertake an

independent review of the Cerner FirstNet emergency department system implemented in NSW hospitalemergency departments (The Review). In May 2011 the NSW Department of Health commissionedDeloitte to conduct an independent review of the Cerner FirstNet system and the effectiveness of itsimplementation.

The purpose of this review is to assess the criticisms raised and advise the NSW Minister for Health andthe Director-General of NSW Health on the appropriateness of continued use of the Cerner FirstNetsystem as a core component of the electronic medical record.

' Report of the NSW Health Council (Menadue Report) , 01 March 2000,2 When referring to events prior to 1 Jan 2011, this report uses the term `Area or Area Health Services'. Post IJanuary 2011, the term Local Health District or District, is used.3 FirstNet refers to the Emergency Department component of the Cerner eMR product suite. It is also the name

given to the NSW Health program of work for the implementation of this system across the NSW Healthhospitals. Unless specifically noted, reference to FirstNet in this report refers to the NSW Health Cerner EDsystem and its implementation.

d As at 1 June 2011

4

Executive Summary

2.2.1 Scope of this review

The scope of this review as set out in the terms of reference was:

1) The suitability of the FirstNet system to meet the clinical, functional and useability requirementsof the users, and whether it's continued use poses any risk to clinical safety

2) The effectiveness and integrity of FirstNet integration with other clinical systems, and FirstNet'scapacity to ensure accurate and timely exchange of data.

3) The level of engagement, consultation and ongoing communication with ED stakeholders andusers, including the effectiveness of implementation change management and end user trainingprocesses.

4) The ability of the system to support the key clinical processes of end users.

5) The response to user concerns raised through the Application Advisory Group (AAG)

6) The responsiveness of Cerner to change requests and vendor management by Health SupportServices ICT (HSS) and NSW Health (the Department).

2

c exclusions

The terms of reference for the review specifically excluded the following:

1) A detailed technical evaluation of the FirstNet system architecture and data model

2) The procurement processes related to the selection of FirstNet

3) The FirstNet implementation project management

4) A review of the emergency department clinical processes

2

To provide a structured framework for conducting the review a clear set of evaluation criteria across arange of dimensions was defined and then applied against an assessment of FirstNet and itsimplementation. The evaluation criteria, agreed to by the review's steering committee, are detailed below:

Fit for purpose

• Whether FirstNet provides clinicians and administrators with the capabilities required to runan ED and deliver efficient and effective care.

• The availability and reliability of the system in supporting clinicians to ensure the highestpossible quality of care can be delivered.

• The appropriateness of the operating environment within which FirstNet is managed andsupported.

• The integration capabilities of the system with regard to reliability, security and adaptability toensure accurate and timely exchange of data.

5

Executive Summary

.5 Vendor

• The performance of the vendor with regard to meeting contracted performance and reportingobligations during system development, rollout and operations.

• The responsiveness of the vendor to requests for product changes and support.

.6 Change Management and Training

• The effectiveness of change management and communications processes to build buy-in,commitment and capacity for change.

• The effectiveness of the training curriculum and delivery approach in addressing the gapbetween current skills of clinicians and future state skills required to operate the systemeffectively and leverage its capability.

2.3.7 Governance

• The alignment of the FirstNet program with the ICT Vision, ICT Strategy and theeffectiveness of governance processes with regard to monitoring and reporting on progressand decision making.

2.4The evaluation criteria map to the Terms of Reference as detailed in the following table: .

Clinical,functional &useabilityrequirements &clinical risk

Effectiveness &integrity ofFirstNetintegration with

other clinicalsystems

Engagement,consultationcommunication,changemanagement &end usertrainingprocesses

Ability of thesystem tosupport the keyclinicalprocesses ofend users

Response touser concernsraised throughthe ApplicationAdvisoryGroup

Responsivenessof Cerner tochange requests& vendormanagement byHSS ICT & theDept

integration

6

Executive Summary

The review commenced on Wednesday 18 May 2011. In carrying out the review Deloitte:• Conducted 37 interviews across a representative range of stakeholders, as listed in Appendix

A - Interview List

• Attended demonstrations of the system issues being experienced by some stakeholders.Four demonstrations were conducted.

• Obtained confidential feedback from LHDs, submitted via a confidential email inbox. Over 50submissions were received.

• Undertook a desktop based review of a set of documents requested from variousstakeholders - refer Appendix C - Source Documents for a list of the documents examined.

As a result of the work performed Deloitte made the following key observations:

2.6.1 Fit for purpose

Subject to the provision of an improved reporting capability and rectification of the faults identified in thisreport, the evidence suggests that FirstNet is broadly `fit for purpose' as an ED system when properlyconfigured.

Many of the criticisms of poor usability can be addressed by:•

Stabilising the infrastructure.

• Upgrading to the current FirstNet release.

• Adopting the most useful and proven configuration changes from sites that have alreadyinvested in useability improvements.

Future planned eMR project phases will reduce the issues associated with a 'hybrid" patient recordthereby enhancing FirstNet's utility and value.

Several ED Directors and other ED staff reported a reduction in ED efficiency as a result of theintroduction of FirstNet, with one ED Director providing evidence of an approximately 20% reduction intriage performance for categories 2,3 and 4 patients following the introduction of FirstNet.

Many users complained of excessive time spent in front of a data entry screen. Whilst FirstNet requiresmore data entry by ED staff than the systems it replaced and this can negatively impact ED performanceat some hospitals, the return on the increased data input work effort will be realised in a number of waysas the full eMR is progressively implemented. Problems with the FirstNet user interface should beresolved and the interface itself improved to make patient data entry and data management moreefficient.

2.6.2

'

The introduction of FirstNet has changed the nature of clinical risks that exist in the ED environment.Claims of increased clinical risk arising from FirstNet's deployment, mainly ascribed to excessive staff`screen time' or an increased likelihood of mistakes must be evaluated against risk reduction achieveddue to the elimination of lost or illegible patient notes, easier access to a more complete andcomprehensive patient history, and reliable orders and test results.

' A 'hybrid' patient record refers to a patient record that is a combination of an electronic and paper patient record.

7

Executive Summary

No evidence was obtained to support the claim that FirstNet has resulted in harm to a patient. FirstNethas changed work practices and has reportedly increased `screen time' for ED staff. However, timespent in capturing patient data in the ED is essential to realise the eventual benefits of an eMR, andmany clinicians reported to us the benefits of the electronic patient record as currently implemented.Some risk scenarios (such as mixing up labels or losing paper test results) arise from the use of a hybridmedical record but these will be progressively reduced as the patient record migrates to an electronicform.

Many of the reported risks arising due to poor usability (such as the absence of composite views ofcertain test results, ordering tests for the wrong patient or authorisation anomalies) have been addressedby FirstNet configuration and work practice changes at some sites. These and other improvements will ifmore broadly implemented address many of the concerns raised by clinicians and users.

2.6.3 Sy:

_ '`ins and support

FirstNet system performance (response times), system availability (up-time) and the effectiveness andefficiency of support"provided to users varies considerably across the sites for the following reasons:

• There are significant variations in the system end-to-end application architecture (includingPAS, RIS and laboratory systems), build configuration, integration approach and hardwareinfrastructure with distributed ownership of each component

• There are multiple parties involved in managing the FirstNet system and providing usersupport services, namely HSS, the LHDs, hospital IT services and various external vendors.The specific boundaries of responsibility of each party are not clearly defined or understoodby those responsible and users.

• There is no consistent and coordinated approach to the provision of support and issueresolution.

As a result FirstNet users are often repeatedly referred from one group to the other as they seek support,attempt to escalate performance or address access issues. Some hospitals or LHDs have invested inlocal additional support capabilities to help overcome these issues.

2.6.4 Integration

We identified three distinct approaches for integrating FirstNet with other clinical applications withvariations at different sites. This variety and complexity has added significantly to the overallmanagement effort required to maintain and support application integration. We also noted that theresponsibility for interface management varies and is often not well defined. As a result, performanceand reliability of interfaces varies widely across sites.

In addition, at some sites FirstNet went live without a number of core interfaces being implementedresulting in a need for workarounds - for example, ED staff manually printing orders to pass to pathologystaff for re -keying into the pathology system.

Integration relies on well defined messages successfully passing between systems. Public criticism ofFirstNet suggested that not all interfaces properly supported all the required messaging. Our reviewdetermined that in the majority of cases this was associated with integration to the PAS. There are nocurrent reports of message loss or adverse clinical outcomes as a direct result of integration issues.

Cerner's use of HL7 for message definitions and SNOMED for message content appears to be generallywell applied, although the support of a restricted set of HL7 messages requires workarounds.

In addition, system users are provided limited visibility of messages that contain errors or have failed.While this is a weakness, it does not appear to have had any adverse clinical consequences.

2.6.5

The scope of Cerner's responsibility is comprehensively established via three related contractingmechanisms (the original head agreement IT-135 dated 27 September 2002, GITC Official Orders, andsubsequent Change Requests). While the Cerner relationship has not been formally managed by HSS

8

Executive Summary

as specified in these agreements, our review concludes that HSS facilitated an effective and productiveworking relationship with the vendor.

No significant performance or delivery gaps by Cerner were identified.

?.6.6 B

The implementation approach adopted by the Department was that of a sequential rolling model wherebya team would implement the system, provide training and cutover support, before moving onto the nextsite. The approach and templates (base documents) for change management, communications andtraining were developed centrally by HSS. Each of the LHDs then localised the approach and templatesto suit their specific needs and were responsible for delivery.

While clinician feedback on capability and quality of work of the implementation teams was consistentlygood, with positive comment on the team's knowledge of the system, clinical processes andimplementation practices, a number of factors negatively impacted the effectiveness of the changemanagement and training delivered as part of the site implementation process:

• At some sites the size of the on-site implementation team and the team's duration of staywas not sufficient

• The implementation approach did not allow for follow up reinforcement of training• The impact on local work practices was not always properly addressed.

• Evidence was provided to our review of inconsistent support by senior management at somesites for the implementation process, specifically in ensuring training attendance andimplementation of changes to local work practices.

As a result, clinicians at some sites were not adequately prepared for using the system and the changesto clinical work practices imposed by the system (for example, 'clerking' before triage).

2.6.7 Gow

Governance applies to both the program established for the implementation of FirstNet and the operationand use of the FirstNet system itself.

This review identified the following issues which negatively impacted the effective governance of FirstNet:• FirstNet is a core component of the broader EMR program. A high-level planning roadmap

for the program and other initiatives is maintained by NSW Health for the purposes ofplanning business cases and budget submissions over a five year period. The program lacksa more formally documented vision for the use of technology to enable clinical practices.

• The effectiveness of the Application Advisory Group (AAG), a core component of the FirstNetgovernance model has eroded significantly over time as clinician participation in this forumhas declined.

• With some exception, FirstNet reporting is inadequate for effective governance of EDoperations

• At some sites users reported shared user sessions to avoid delays associated with loggingout and then logging back in as a new user.

• The requirement for clinicians to enter a user identifying pin number to authorise atransaction has been disabled at some sites. This means that it is not possible to reliablydetermine the identity of the clinician processing transactions. This issue is exacerbatedwhere clinicians share computers or user sessions.

9

Executive Summary

17.1 D_

. iNS T F

S f•1 '

'd c,

i

i1 4 and

ANet

initiate a r

af,tion p

--m to addressin this repor

NSW Health has implemented FirstNet as part of the overall eMR program to 59 hospitals across NewSouth Wales. While there have been a number of reported issues associated with the deployment, useand operation of the system, many clinicians have reported that it now adds considerable value to thedelivery of care and the operation of their Emergency Departments.

Many of the issues that have been reported and observed relate to system operations, governance,implementation, leadership and training, and not specifically to the capability of the FirstNet system.While this review confirmed that there are a number of issues with the use, implementation, support andconfiguration of FirstNet, we believe these can be remediated via an appropriate program of work.

2.7.2 Rec-

cl t' _ Define an(' ' iiplerent site specific -

"__1 plans tonuance and t.~. ` 1 % ssdes

.ng *rstNet it-1

A range of issues and user frustrations were identified across the FirstNet sites we examined. These areimpacting user satisfaction and the effective use of the system, highlighting the urgent need for a formalprogram of site specific remediation activities.

These remediation activities include remedial training, configuration and in some cases, upgrades to thecurrently implemented system. A well defined plan and program. of work needs to be put in place to bringall sites to a base level of acceptable infrastructure, functionality, useability and user training.

We further recommend that this remediation program be delivered by resources other than those alreadyresponsible for the current implementations or the planned eMR phase 2 to avoid resource conflicts withthese activities.

o

standardisatio;, ;- nd eff c e-ncie, c

While NSW Health has adopted the concept of a State Base Build (SBB) - many of the implementationsare deployed on different operating system versions; different Cerner versions; different SBB versions;different local configurations; and in some cases, local additions that are outside of the scope of the eMRprogram. This is driving inefficiency and complexity in the use, operation, support and management ofthese systems.

NSW Health should expand the scope of the SBB to include elements added in local implementations,

and then standardise where appropriate the use of this expanded SBB across the LHDs. This will

improve system operation and support efficiency and share the benefits derived through successful localconfiguration changes across all FirstNet users. The current plans for the implementation of EMR phase2 and medications management capability should build on the SBB model proposed above.

Reporting warrants a specific remediation focus given the high level of frustration expressed during theuser interviews and should be considered for special attention when upgrading the SBB.

NSW Health has made substantial steps toward the implementation of an eMR for all patients presentingat hospitals across the State. The eMR will provide a basis for participating in the broader national

10

Executive Summary

Personally Controlled Electronic Health Record (PCEHR) program, while providing the longitudinal recordof patient care essential for addressing the ever increasing demands on the health system.

As IT systems are increasingly used to support the delivery and improvement of health care, it isimportant that senior clinical input drive the vision and requirements for the use and future developmentof clinical systems. We recommend the creation of a Chief Medical Information Officer (CMIO) in NSWHealth to provide this direction.

A primary responsibility of this senior executive will be the development of a vision for the capabilitiesnecessary to enable the care delivery strategies of the Department. This vision should provide the basisfor all systems implementation strategies and will be fundamental to the successful use of existing andfuture clinical systems.

2.7.5 '

future funding requests i 1 W provisions for boththe initH c:

tion as weII as the on-going c.,,,

costThe review identified that the funding model for the FirstNet program has been very `project' driven. Whilethis approach has successfully supported the initial implementation of the system, it has not expresslyprovided for the iterative or retrospective application of enhancements resulting from lessons learned andthe ongoing clinical process improvement that typically takes place in an ED. No funding is providedunder the current funding model for on-going system development and training support after the initialimplementation.

This has a negative impact on user satisfaction and adoption and the harvesting of benefits expectedfrom the system. It is therefore important that funding for the future phases of the eMR program includesboth implementation and on-going operating costs.

2.7.E

Tin: AsseE

go

gee and organisational structuresy ofciinicl.

There are multiple parties involved in the delivery of FirstNet and the broader eMR program, includingHSS, Local Health Districts, Cerner, other vendors and the Department. The specific responsibilities ofeach organisation are not clearly defined or understood by all stakeholders. This uncertainty has driveninefficiency and stakeholder discontent.

An improvement to the governance and organisational structures required to effectively deliver acomputer enabled clinical care system within NSW Health is needed. Any changes to organisationstructure, delivery and support model, and governance processes must be clearly defined andimplemented.

2.7.7 R - c

Enhari _ _

te, to

The success of FirstNet and the broader eMR program requires highly reliable system availability andresponsive user support. This is currently not the case at many sites, resulting in significant userfrustration and an inefficient use of valuable resources.

It is important that the Department undertake a review of the existing system operation and user supportresources and processes with the aim of establishing a new transparent and effective system and usersupport capability that is appropriate for clinical applications.

As the hospitals across the state move more towards a paperless care delivery model, the reliance on theavailability of the clinical systems and the patient information in those systems becomes critical to aclinician's ability to deliver effective care.

It is essential that highly available and high performance

I1

Executive Summary

infrastructure be provided to support the delivery of this critical IT service. The review established thatthe existing IT infrastructure for FirstNet does not universally meet this minimum standard. Additionally,the responsibilities of the multiple parties involved in delivering support are not clearly defined.

A review should be undertaken of the infrastructure delivery approach, and an assessment made as tothe suitability and capability of each contributing party. This review should also include an assessment ofalternate sourcing approaches. Many international users of Cerner do not run their own data centres,and outsource the hosting and management of the core infrastructure required to run the application tospecialist third parties.

12

Executive Summary

The recommendations map to the Terms of Reference as follows:

Clinical,functional &useabilityrequirements &clinical risk

Effectiveness &integrity ofFirstNetintegration with

other clinicalsystems

Engagement,consultationcommunication,changemanagement &end usertrainingprocesses

Ability of thesystem tosupport the keyclinicalprocesses of

end users

Response touser concernsraised throughthe ApplicationAdvisoryGroup

Responsivenessof Cerner tochange requests& vendormanagement byHSS ICT & theDept

etain FirstNet

ite specificemediation

plansExpand &

onsolidateBB

Funding modelreview

linical programovernance

review

upport modelreview

nfrastructure"liverypproach review

13

Introduction

3 Introduction

3.1. I

E V Li. to t he NSW Health e MR program and FirstNetThe FirstNet` emergency department system is part of the NSW Health eMR journey which commencedsome 10 years ago. In March 2000 the Report of the NSW Health Council (Menadue Report)' gaverenewed focus to information management and technology initiatives that better support clinical practiceat the point of patient care.

As part of the response to the Report in 2002 NSW Health submitted a Point of Care Clinical (PoCCS)business case the objective of which was to obtain approved funding to implement a common State widesystem to provide an interactive clinical decision support facility to assist clinicians in the day-to-dayclinical management of their patients within the hospital setting. The business case identified Cerner asthe preferred Electronic Medical Record (eMR) solution and a contract (head agreement) was signedbetween NSW Health and Cerner on 27 September 2002. This contract forms the basis of the State'srelationship with Cerner.

PoCCS, which was later renamed the Electronic Medical Record (eMR), was identified as a keycomponent of NSW Health's Information Management and Technology Strategy. NSW Treasuryapproved, but only partially funded, the PoCCS business case in 2002. Following approval, limitedprogress was made in implementing the strategy.

At the end of 2005 a tender was released to explore options for an alternative eMR solution. Theevaluation confirmed the Cerner Millennium" suite as the appropriate solution for the NSW Health eMRprogram. Additional funding was sought and approved to significantly accelerate the implementation ofcomponents of the eMR including Results Reporting, Order Management, Electronic Discharge Referral,Operating Rooms and Enterprise Scheduling, and Emergency Department (FirstNet).

Following the signing of the original contract, "Official Orders" for the supply of eMR component moduleswere entered into by the Local Health Districts (LHDs). The first instance of the eMR including FirstNet foremergency department management and tracking was implemented under this agreement at St GeorgeHospital in the South East Sydney and Illawarra region in September 2008.

Since then FirstNet has been implemented at 59 sites and the rollout is due for completion by late 2011.

FirstNet refers to the Emergency Department component of the Cerner eMR product suite. It is also the namegiven to the NSW Health program of work for the implementation of this system across the NSW Healthhospitals. Unless specifically noted, reference to FirstNet in this report refers to the NSW Health Cerner EDsystem and its implementation.Report of the NSW Health Council (Menadue Report) , 01 March 2000,

8 Cerner MillenniurTM is a comprehensive suite of solutions supporting personal and community healthmanagement

14

Introduction

For the month March 2011' the following system use data was reported.

Total transactions 86m

Total Orders

Average Daily Users

Average Daily orders

2.84m

17,188

91,741

x.1.2 1

The State Base Build (SBB) for the eMR is a framework of content and system configuration parameterswhich has been used as the baseline for NSW Health implementations of the Cerner Millennium suite ofproducts. The St George Hospital site implementation formed the basis of the SBB.

The development of the current SBB has been an ongoing process and the SBB continues to bedeveloped. Approved change requests have been incorporated into subsequent versions of the SBB.Sites are required to use the SBB as the basis for their implementation, but are allowed flexibility in theway the system is configured at the site level. In practice this flexibility, coupled with the implementationof different versions of the SBB has resulted in significant variation in configurations across the State,LHDs and sites. This variation is one of the drivers behind the different levels of user satisfaction withFirstNet.

There are currently 4 versions of the SBB implemented across the 8 former Local Area Heath services(or 15 Local Area Health Districts).

3ound

A number of issues with FirstNet have been raised by ED clinicians, other hospital staff, professional

bodies and academics. Issues raised include FirstNet's potential to negatively impact on efficient hospitaloperations and more critically, patient care.

Following reports in the media in March of criticisms of the NSW Health Cerner FirstNet ED system madeby Prof Jon Patrick at the University of Sydney, the Minister for Health committed to undertake an

independent review of the Cerner FirstNet emergency department system implemented in NSW hospitalemergency departments (The Review). In May 2011 the NSW Department of Health commissionedDeloitte to conduct an independent review of the Cerner FirstNet system and the effectiveness of itsimplementation.

The purpose of this review is to assess the criticisms raised and advise the NSW Minister for Health andthe Director-General of NSW Health on the appropriateness of continued use of the Cerner FirstNetsystem as a core component of the electronic medical record.

The terms of reference for the review are as follows:1) The suitability of the FirstNet system to meet the clinical, functional and useability requirements

of the users, and whether it's continued use poses any risk to clinical safety2) The effectiveness and integrity of FirstNet integration with other clinical systems, and FirstNet's

capacity to ensure accurate and timely exchange of data.

Lights On Report for March 2011

15

Introduction

3) The level of engagement, consultation and ongoing communication with ED stakeholders andusers, including the effectiveness of implementation change management and end user trainingprocesses.

4) The ability of the system to support the key clinical processes of end users.5) The response to user concerns raised through the Application Advisory Group (AAG)6) The responsiveness of Cerner to change requests and vendor management by Health Support

Services ICT (HSS) and the NSW Health Department.

The Review's scope specifically excluded an examination of the following:1) A detailed technical evaluation of the FirstNet system architecture and data model2) The procurement processes related to the selection of FirstNet3) The FirstNet implementation project management4) A review of the emergency department clinical processes.

To provide a structured framework for conducting the review a clear set of evaluation criteria across arange of dimensions were defined and then applied against an assessment of FirstNet and itsimplementation. The evaluation criteria, agreed to by the review's steering committee, are detailed below:

Evaluation Criteria

Description

Fit for purpose Useability

Is the user interface appropriate to support the operational, time andcare delivery demands of an ED clinician?

Does the system provide a user interface that is appropriate for eachof the administration and care provider staff using FirstNet?Accessibility

Does the system provide an effective and secure mechanism forclinicians to access the required information and services?

Does the system provide access to information through multiple andappropriate devices (e.g. mobile, touch screen, real-time monitoring)?Functional Fit

Does the system provide appropriate support for all of the functionsrequired to both manage an ED and deliver effective patient care?

Does the system effectively support other reporting, administrationand care delivery processes within a hospital?

Clinical Process Support

Can the system be flexibly configured to support the local workpractices at each location?

Does the system enable the rapid implementation of new processesand models of care?

16

Introduction

Evaluation Criteria Description

Clinical Risk Information Integrity

Is the information provided by FirstNet always reliable, and do theclinicians have confidence in the information provided?

Is information reliably transferred between systems?System Responsiveness

Does the system provide information in an adequately responsivetimeframe?

Identity and Access Management

Does the system facilitate user identification and enforce user trackingof all transactions entered?

Information and System Availability

Is the system availability appropriate for clinical environments?System Operation and Service Delivery and SupportSupport

Is the FirstNet and supporting technology environment appropriatelymanaged to meet the availability, capacity, access and performancerequirements of the hospitals?

Are users

of the system well

properly supported when

raisingrequests for support or system changes?Infrastructure Management

Are appropriate strategies and plans in place to ensure that thetechnology infrastructure adequately supports the current and futureoperating requirements of FirstNet?Application Management

Are appropriate processes in place to ensure the effective planning,development,

implementation,

maintenance

and

support

of

theFirstNet application?

Are appropriate application change management and configurationmanagement processes in place?

Integration Integration Capability

Is the integration between FirstNet and other systems reliable?

Is the integration architecture sufficiently open and adaptable?Standards and Compliance

Are the system messages compliant with agreed standards?

Does the system

use code-sets that are compliant with agreedstandards?

Error Handling

Does the system provide adequate detection, logging and notificationof errors and exceptions?

Does the system facilitate appropriate response and rectification oferrors?

17

Introduction

Evaluation Criteria Description

Vendor Management &Support

Contract and Vendor Relationship Management

Has

the

vendor

met

contracted

performance

and

reportingobligations?

System Support Effectiveness

Has Cerner responded to requests for product changes and support ina sufficiently timely and satisfactory manner?

Change Management Change management and communicationsand Training

Do management processes exist to effectively assess the supportrequired by stakeholder groups, identify potential risks and barriersand design strategies to address these and build buy-in, commitmentand capacity for change?

Are stakeholders communicated with effectively to inform them of thesystem changes, foster buy-in and enable feedback mechanisms toenable dialogue with users?

Training effectiveness

Does the training curriculum and training delivery approach addressthe gap between current skills of clinicians and future skills required touse the system effectively?

Governance ICT Vision and Strategy

Does FirstNet align with the ICT Vision and Strategy?Business Operations Governance

Does the system enable effective reporting to monitor performance ofthe organisation?

Does the system adequately support the operational and clinicalgovernance processes within the hospitals?System Governance

Are the processes for managing the FirstNet implementation programeffective?

18

Introduction

The evaluation criteria described above can be cross referenced to the Terms of Reference as follows:

Clinical,functional &useabilityrequirements &clinical risk

Effectiveness &integrity ofFirstNetintegration with

other clinicalsystems

Engagement,consultationcommunication,changemanagement &end usertrainingprocesses

Ability of thesystem tosupport the keyclinicalprocesses ofend users

Response touser concernsraised throughthe ApplicationAdvisoryGroup

Responsivenessof Cerner tochange requests& vendormanagement byHSS ICT & theDept

Fit for purpose

linical Risk

ystem,perations &upport

Integration

endor

Management &Support

hangeanagement &raining

overnance

19

Introduction

The review commenced on Wednesday 18 May 2011. In carrying out our review Deloitte:

• Conducted 37 interviews across a representative range of stakeholders, as listed in AppendixA - Interview List

• Attended demonstrations of the system issues being experienced by some stakeholders.Four system demonstrations were conducted.

• Obtained confidential feedback from LHDs, submitted via a confidential email inbox. Over 50submissions were received.

• Undertook a desktop based review of a set of documents requested from variousstakeholders - refer Appendix C - Source Documents for a list of the documents examined.

20

Observations

4 ObservationsThis section details the observations gathered during the review, categorised by evaluation criteria

4`Fit for purpose' is a characteristic of product quality. Quality is not an absolute characteristic of a productbut is determined by the product's use. A product can be said to be `fit for purpose' if it is sufficientlysuitable for the intended purpose or use. Themes for this criterion that are common to multiple sites andaccounts fall into four groups: usability, accessibility, functional fit and clinical process support.

4.1.1 Usability

Usability can be defined as the ease of use and `learn-ability' of a product. FirstNet presents to users viaa 'thick client' window and requires both keyboard and mouse interaction for data entry. The userinterface broadly employs familiar Microsoft Windows user interface (UI) conventions but exhibits someinconsistencies between views and data entry forms. Basic activities such as searching and viewingpatient details, status checking and status monitoring, and navigating between functions typically requiresexplicit mouse -clicks, keystrokes, and usually a combination of both.

User interface is generally considered non-intuitive and difficult to useMany users reported that they did not find the FirstNet user interface useable or intuitive, based on itsscreen and menu design, and the way commonplace activities are supported. Users consistently raisedthe following criticisms:

• Some screens are crowded and present unrelated or irrelevant information or fields• The patient tracking screen was consistently criticised for presenting too much information

• There are too many icons for even regular users to remember, and the icons themselves areambiguous

• Many basic and frequently used actions take too many clicks to complete• The same key provides different functions in different fields or screens• The application is inconsistent with basic Windows conventions (for example, the Enter key

cannot consistently be used as a substitute for completing a form or action)

• The application does not support keyboard alternatives to mouse-clicks, which necessitatescontinuous alternating between the keyboard and mouse to navigate or complete a form.

Some of these usability criticisms have been addressed to varying degrees through local changing ofconfiguration parameters, both at the user and product level. For example, user-defined patient lists canbe created by users, with each list presenting on its own separate tab. If the number and names ofpatient lists is not constrained or made consistent, a large number of ambiguous tabs appear on theFirstNet landing screen.

In addition, a number of small but significant usability improvements have been made at some sites byCerner. Two examples are more readable discharge summaries and the addition of an asterisk in thepatient row on the `All Patients' table to indicate a completed theatre booking.

The design of some high-use screens negatively impacts work efficiencyA number of ED directors commented that the introduction of FirstNet had negatively impacted EDproductivity, necessitating the scheduling of additional resources to compensate. One ED Directorprovided evidence to substantiate this claim.

21

Observations

Examples of issues relating to screen design that contributes to lowered work efficiency raised during theinterviews include:

Users spend too much time navigating around multiple poorly designed screens. Forexample, the `All patients' table (one of the most frequently used screens) must be scrolledhorizontally and vertically to access rows (patients) and columns (data on each patient).While some reported this as merely inconvenient and time-wasting, others claimed that itrepresented a risk as it was possible to click on the wrong patient's row to launch a form toorder tests. It should be noted that this specific issue can be fixed through local configurationas was successfully done at one site by freezing the first two columns of the screencontaining the patient number and name so that the patient's identity remained visible as theremainder of the window scrolled.

• The same information sometimes needs to be entered multiple times when ordering tests.The system also prompts users for inappropriate or irrelevant information, or informationwhich wastes time.

There are reports of anomalous or apparently incorrect FirstNet behaviour, such as loss ofcomment data from the `Pre-Arrival' form when the 'Arrival source' drop-down selection ischanged. (It was pointed out that the lost data can be recovered by simultaneously pressingthe Control and Enter keys). That said, some sites do not use the FirstNet `Pre-Arrival'screen while others do not consider the potential for loss of this data to be important.

Configuration of certain labels and codes is inconsistent across the sitesThe decision to adopt the SNOMED taxonomy was made as part of the transition to FirstNet. At some of

the earlier implementation sites SNOMED code-sets were not edited or reduced to meet localrequirements or practice.

This resulted in practitioners being frustrated by the large number of irrelevant codes and unfamiliar(predominantly American) terminology in code names and labels. Some sites have addressed thisthrough local configuration to reduce the code set size - by removing irrelevant codes, or in one case bydefining 'Favourites' or categories of frequently used codes.

A similar situation exists for orders. Users agreed that the top 20 orders account for approximately 98%of all orders placed. One site implemented the `top 20 orders' as 'favorites', however no mechanismexists to share this configuration with other sites.

Visibility of the patient record varies across sites

A clinician's visibility of a patient's medical record in FirstNet varies across sites, depending on theprogress of the eMR program implementation at that site. At hospitals where Cerner PowerChart is notused or is not widely used, a patient's medical record will include both paper and electronic forms". Atsome sites, the handover from the ED to a ward involves printing the electronic ED records to create thecomplete patient record for the ward.

Clinicians at hospitals where PowerChart is used in both the ED and the wards are often strongadvocates of the electronic medical record, citing significant benefits relating to the visibility to patientrecords both within and between episodes at their hospital and within their area over time. Otherreported benefits include time savings from not having to search for paper-based notes, the elimination oflost and unreadable notes, and being able to always see patient arrival and discharge times as well asthe time they were seen by a doctor.

'0 Referred to as a 'hybrid' patient record.

22

Observations

System response times are variable and depend on location and time of day

System responsiveness is perceived as a usability issue. The review identified a number of sites whereusers reported unacceptable and frustrating response times. Clinicians do not distinguish betweenFirstNet and the infrastructure or hosting services, so a problem with Citrix or a network connection isperceived as a problem with FirstNet. Unreliable or poor system responsiveness erodes confidence inFirstNet and the eMR more generally and is viewed as a serious and unacceptable impediment to workby ED staff

Reports of system response times vary widely by area. Some practitioners report systemresponsiveness that is consistently slow to the point of being unusable. Comments such as the systemis painfully slow three out of seven shifts' and `screens are slow to refresh' were common from thesesites. Reports of EDs reverting to whiteboards for patient tracking during periods of unacceptableresponse times or unscheduled system downtime were made by several sites. It was also claimed that inmany cases information collected on whiteboards was later never entered into FirstNet or captured inother patient files.

Conversely, other sites (and areas) reported system responsiveness as being acceptable and relativelyconsistent. The differences are likely to be a function of hosting and infrastructure capacity and reliability,particularly the Citrix infrastructure capacity and network bandwidth.

4.1.2 Accessibility

Accessibility refers to the ability of ED staff to freely access and use FirstNet during their shifts.Accessibility is a function of the number, physical placement and hardware configuration of workstations,as well as the design of the FirstNet user interface.

Too few computers in EDs

It was claimed that the number, position and configuration of computers in EDs is not always conduciveto effective work practices. In most cases, clinicians and ED staff `share' a computer, sometimes creatingcontentions during busy periods. ED staff may or may not share a user session - at some sites doctorsand nurses share a logged-on user session for viewing results and doctors only change users whenelectronically signing an order or discharge.

Some clinicians stated that effective use of FirstNet requires large monitors and that these are either notinstalled or there are too few.

Logging in and changing users for authorisations

While most clinicians reported acceptable logon times at their hospitals, a limited number of cliniciansreported frustration at having to wait for up to 3 minutes to logon to FirstNet. The logon process can takeup to 6 mouse-clicks and presents unnecessary confirmation and `splash' screens. Improvements (suchas a single-click logon desktop icon) have been implemented more recently at some sites.

FirstNet supports the option for enforcement of an authorising user (a doctor) to electronically sign anorder or discharge summary. This ensures the appropriate authorisations are captured against patientrecords and orders. At sites where this feature is enabled, a doctor must log in or change user (byentering their username and password) to 'sign' and complete an order or discharge. At some sites, thisfeature is not turned on, and some clinicians reported that doctors had submitted orders and dischargeson another staff member's session.

System availability varies significantly across sites

System availability is affected by planned and unplanned outages, where an outage is defined as theunavailability of the system or any of its essential parts for normal use.

On planned outages, several sites reported the system initially being taken down for periodicmaintenance or upgrades with little regard for the impact on EDs (for example, on Saturday nights whenEDs are busy). These same hospitals reported recent improvements with consultation and agreement onthe times and the likely duration of planned outages.

23

Observations

On unplanned outages, site representatives gave very different accounts. Several sites reported anunacceptable degree of system unreliability for the first 12 months after cutover, as well as unacceptablylong downtimes when these unplanned outages occurred. In most cases, improvements in availabilityand restoration times have been made over the last 6 to 12 months.

Another accessibility issue arises where FirstNet prevents access to a patient record that is current (i.e.not discharged) at another hospital in the area. ED staff work around the problem by either calling theother hospital to request a staff member to discharge the patient, or they create a duplicate patient recordand merge the records at a later time.

4. L3 Functional fit

Functional fit refers to the degree to which FirstNet provides the right functions to support the activitiesand workflows of ED staff.

Reporting capability and access to data is inadequate

Reporting, access to ED data and ad hoc (or on-demand) reporting was widely criticised. The criticismsfall into three categories - too few reports were implemented; FirstNet's support for one-off access to EDdata is inadequate; and data quality in extracts or reports is often poor.

The criticism that too few reports were implemented in FirstNet derives from a comparison with itspredecessor EDIS, which provided reports which were not replaced in FirstNet, particularly reports on thenumbers of cases, services or diagnoses in a given period. This is a source of significant frustration forsome clinicians and has created additional work for data managers at some sites.

On the system's support for access to ED data and ad hoc reporting, a number of ED Directors reportedsignificant difficulties in getting access to one-off queries or reports providing data over a period of time tosupport departmental management, research or trend analysis. This kind of reporting is distinct fromFirstNet's support for user-defined lists of current ED patients. The combination of weak reportingcapability and lack of visibility of the data within the system leads clinicians to think `there is far more datagoing into the system but we are able to get far less out'.

On poor data quality, some sites reported little confidence in the integrity of FirstNet reports. One EDDirector provided a sample report reflecting a total of only 25 patients having visited the ED on a givenday when the actual number was known to be 125.

red to a lost not,-,,ED Director

ED performance data requires manual remediation

NSW Health's `Demand and Performance Evaluation Branch' routinely collects data from the State's EDsfor a range of purposes including performance management, customer satisfaction, cost effectiveness,capacity, workforce and service planning and health research.

Since the commencement of the initial FirstNet rollouts, the Demand and Performance Evaluation Branchhas reported a significant increase in data that fails basic validation criteria. Examples are missing data orpoor data quality including missing fields (such as treatment time, preferred language and insurancestatus), treatment time before triage time, and departure before triage time. The Demand andPerformance Evaluation Branch has implemented workarounds to partially remediate FirstNet datarequiring manual intervention by both the hospitals and the Branch. Despite these workarounds, theintegrity of some data used by the Branch continues to be compromised.

Poor discharge summaries

Inadequate discharge summary capability is one of the most common criticisms raised against FirstNet.The standard templates provided are clumsy and the editing and formatting capability provided is limited.Some ED doctors reported being 'embarrassed' by the quality of the discharge summaries produced byFirstNet, resorting to external word processors to manually format the report and then copying the

24

Observations

formatted document back into FirstNet. Some sites reported having developed local templates andformatting and in this way have addressed this issue.

Functionality is generally adequate

Apart from the usability problems described above, clinicians were broadly satisfied with the functionalcapabilities provided by FirstNet, with the noted exception of reporting. Benefits cited by a number ofclinicians include the value of electronic records and notes, increased transparency of patient flowthrough the ED, longitudinal visibility of patients over multiple visits both at the clinician's hospital and toother hospitals in the area, integrated ordering and results, triage tracking and support, and support forimproved decision-making.

4.1.4 Clinical process support

Clinical process support refers to the degree to which FirstNet supports the clinical activities andprocesses of the ED.

System introduces discipline around processes

Clinicians reported the need for changes to work practices following the implementation of FirstNet. While

some appreciated the enhanced enforcement of disciplines, others cited unwelcome disruption of localpractices. One site reported re-numbering their ED rooms because they found it easier to do thanchanging the room designators in FirstNet.

A significant process change introduced by FirstNet involves `clerking' patients (which includesestablishing their identity in the PAS) before triaging. The need to access the PAS to establish thepatient's identity before triage was cited as an `unnecessary administrative burden on clinical staff and acontributor to increased clinical risk'. One hospital claimed it could take up to 10 minutes to get a patientrecord from the hospital's PAS while others described it as a 'minor adjustment of work practices' that didnot significantly impact staff or increase risk because it simply required re-sequencing of existing tasksand was easily accomplished by accessing the PAS via a single button added to the screen.

Insufficient user support for local variations in clinical processes

Many ED clinicians criticised FirstNet for its apparent rigidity and enforcement of uniform clinicalprocesses. Some of this criticism derives from counter-intuitive aspects of the user interface and otherusability concerns discussed earlier. Users also stated that FirstNet introduced too much clinical processchange at one time and that post-implementation support for users was insufficient to ease the transition.

Users reported the need for greater support pre and post go-live in order to properly understand theiroptions to customise FirstNet, both at a user and site level. A lack of user awareness of localconfiguration options was evidenced, especially at sites with limited local user support capability. Thisincludes aspects such as user-defined patient lists, customised diagnostic and allergy code sets,changing American terminology, and the use of `favourites'.

4.2

Allegations have been made that the introduction of FirstNet has significantly raised clinical risk forpatients. During the review, we assessed aspects of system functionality, performance and integrity thatpresented the potential to contribute to a change in the nature of clinical risk. Our assessment ispresented using the following themes: system data and integration integrity, clinical process change, dataquality, system availability, and enforcement of authorisations.

Lack of system data and integration integrity increases clinical riskIt has been claimed that the system lacks data integrity and as a result has been known to `lose' patientrecords or data. There are two potential scenarios leading to loss of data - loss of a test order or results

25

Observations

and loss of a patient record while in the ED. In general, integration'' was reported as reliable,notwithstanding some initial field-level mapping anomalies which have been subsequently corrected. Noevidence was found of repeatable or systematic message loss at interfaces.

Similarly, no evidence was found to support concerns about the loss of an ED patient's record fromFirstNet. It appears that it is possible for a patient's record to become assigned to a clinician without thatclinician being aware or present and this scenario may have been interpreted as the loss of a patient'srecord. The patient would, however, still appear on the `All Patients' list.

Low data quality increases clinical risk

It has been claimed that the introduction of FirstNet compromised ED performance data collected byNSW Health. This is because field validation in FirstNet is currently turned off in the State Base Build'',so it is possible for incorrect dates, codes and other structured data to be saved or not entered at all. It isalso possible to save inaccurate treatment start and end times.

Poor data quality is having significant downstream consequences and this problem needs to beaddressed. However, because performance data is not used by clinicians to directly support patient care,this deterioration of data quality cannot be claimed to have significantly increased clinical risk.

4.2.2 System availability and responsiveness

Insufficient system availability and responsiveness increases clinical risk

An ED system must have high availability. There are many accounts of unacceptable or unscheduledFirstNet downtime at some sites, although in most cases reports indicate that this has improved overtime. When FirstNet is unavailable, the ED loses visibility of its patients, their assignments, statuses,orders and results. Typically, ED clinicians revert to a whiteboard for basic patient tracking. A fallbackand subsequent recovery from a period of unavailability arguably introduces some clinical risk on thebasis that information must be reconstituted on a whiteboard and then transferred back to FirstNet whenit becomes available.

4.2.3 Identity and access t

Lack of enforcement of authorisation increases clinical risk

The system requirement for the authorisation of orders, diagnoses, test results and discharges by adoctor is disabled at some sites, presumably because the task of changing users is cumbersome andslow. As a consequence, any ED staff member at these sites can order or authorise tests or discharge apatient from any `logged-in' computer. Some interviewees reported that it is also possible for a patient tobe discharged without all assigned staff having completed their notes and that a staff member can signoff an ECG on the current (potentially different) user's login. This compromises the integrity ofauthorisation processes and the ability to audit information in the system.

Authorisations should be enforced and made to work effectively to mitigate this risk. The futureintroduction of Medications Management will further raise the importance of the capability to record andenforce authorisations for prescribing and medicine administration.

4.2._

Changes to clinical processes increase clinical risk

Some ED Directors claimed that clinical process changes forced by FirstNet (primarily data entry and`clerking' before triaging) have raised clinical risk due to the time taken for patient lookup and admissionduring the triage process. Other ED Directors stated that risk has not increased because patient lookup

" Refer section 4.4 for detailed observations on the integration between FirstNet and interfacing systems (primarilyPAS, pathology and radiology).

z This was presumably done to remove a clinician's obligation to `fix' erroneous data when handling an emergencysituation.

26

Observations

or administration is handled separately from triage, and patient admittance processes allow judgement tobe exercised at all times. At most sites, FirstNet allows a patient to be entered by ED staff as unidentified(with name and basic identifying fields) and then linked to the PAS patient record at a later time.

Confusing UI design increases clinical risk

A number of clinicians and respondents argued that poor user interface design makes interactions withFirstNet unnecessarily time consuming, and as a result ED staff spent more time doing data entry thancaring for patients, thereby decreasing productivity and increasing clinical risk. The following usabilityissues were identified by interviewees as potential causes of increased clinical risk:

• Note-taking in FirstNet is considered by some ED staff as so cumbersome that they do notbother, resulting in reduced clinical documentation

• Navigation is found by many staff as counter-intuitive and multi-layered to the point where itslows down their work

• Searching and code selection problems make selecting diagnostic, allergy and test codestime-consuming, particularly the SNOMED diagnosis code set which is considered by manyto be too large, difficult to navigate and cluttered with irrelevant codes13 and obscure names

• Tests may be ordered for the wrong patient as a result of the patient name disappearingwhen scrolling in the patient tracking screen

• Some test specifications are poor (for example, it is reported that it is possible to order an X-ray or image without specifying the part of the body to be imaged)

• The way that FirstNet reportedly presents certain test results could obscure recognition of apotentially life-threatening condition.

While these usability issues can potentially introduce clinical risk, it should be noted that some of theseissues have already been partially or fully addressed at some of the sites that have carried out localconfiguration.

4.3.1 Service delivery ar .

- rt

Responsibility for service delivery and support of the FirstNet system is spread across multipleorganisations:

• With the exception of non SBB sites, HSS is responsible for management and support of thecore FirstNet system including the hardware and delivery services. HSS also manage thedevelopment and maintenance of the SBB and the associated governance process. HSSprovide a first level telephone support service to record and respond to FirstNet servicerequests.

• Local health services provide local training, support and configuration services, as well as thefirst level of governance for changes to the State Base Build.

• Cerner is responsible for FirstNet third level support and the development andimplementation of software changes based on direction from both the Application AdvisoryGroup (AAG) and local area or site requests in some cases.

Clarity of process for management of system support varies by site and area

There was evidence of considerable variance in users' understanding of issue resolution and supportprocesses across the areas and sites. Given the number of parties involved in support roles and thevarying investment in local support, the processes for an individual to seek help with a system issue andreceive follow-up resolutions and closure are not always clear.

1 3 Examples include `San Miguel Sea Lion virus' and `Mexican hairless doallergy'.g

27

Observations

Responsiveness and effectiveness of system support varies by site and areaThe varying effectiveness of local and centralised support is driving considerable discontent in a number

of sites. The rolling implementation team approach did not leave every hospital with an equivalent levelof system support capability. Each site has responded differently with some sites recruiting dedicatedlocal support resources, some training `super-users', some relying on scarce regional support, and othersrelying on the HSS provided centralised support. This has in turn led to considerable variance in userskills, with some clinicians being relatively expert and able to establish specific lists and shortcuts, whileothers have only basic FirstNet skills.

The communication and management of planned and unplanned outages varies by site and area

While planned outages were reported to be generally adequately communicated, their timing was often acause of frustration. The incidence of unplanned outages appears to vary significantly by site and area.Some sites report that system performance (i.e. response time) often deteriorates to the point where thesystem freezes completely. When this occurs, feedback on expected resolution timeframes is oftenminimal. For example, some users reported being given no direction other than being told to `keep ontrying until you can log in'.

Change request process lacks transparency and can be unacceptably lengthyWhile the change request processes are reasonably well defined, the visibility of the status of requests isoften inadequate. Many users stated that when requests were submitted for review there was often nofeedback on status, and in some cases change requests took a year or more to be processed.

4.3.2 Infrastructure management

There are multiple hardware and software components required for the delivery of end -to-end IT supportof an ED. At most hospitals FirstNet integrates with a local PAS, RIS, PACS, and laboratory systems.This integration is supported by local networks and in some cases local integration engines (e.g. eGate,

JCAPS), so that the centralised integration infrastructure provided by HSS may integrate directly to localsystems, or with a local integration infrastructure including other systems.

Also, in some areas there are multiple implementations of each type of local system that are integrated toa single Cerner instance. This infrastructure complexity, design variance, and distributed componentownership presents a difficult infrastructure management scenario.

Infrastructure is managed by various parties (HSS, area and /or site)

HSS has responsibility for managing the Cerner environments and the integration infrastructure requiredto interface to Cerner. The local areas are responsible for the delivery and management of local systemsand integration with these systems, to enable a complete business or clinical process to be supported.

While management responsibilities are well defined, there is no apparent single point of accountability forend-to-end system operation, which makes it difficult to assess system problems, and diagnose whatremediation is required.

Various levels of resource and capability available by site

We observed considerable variances in system performance across implementations. At some sites thesystem performs well with screen refresh times being well within what most users consider acceptable.At other sites it was reported that the system often becomes so slow that it becomes unusable. Forexample, one site reported that during the initial stages of FirstNet, there were periods when the system

regularly `almost stopped' driving clinicians to revert to manual processes for an hour or two untilperformance gradually improved. This was reported as occurring approximately three days in everyseven day roster.

Non-SBB sites generally reported a consistently higher level of support satisfaction, as did sites thatappear to have generally benefitted from higher levels of investment to ensure the successful introductionof FirstNet and the broader eMR program.

28

Observations

Effective use of FirstNet requires access to appropriately specified and located PCs and monitors

A number of Cerner screens present information that is wider than the display width of a standardresolution monitor. This makes it necessary for users to constantly scroll both horizontally and vertically,and leads to considerable user discontent. Some sites have addressed this concern by installing widemonitors on PCs where these screens are accessed. Some Emergency Departments have paid attentionto placing PCs in locations that give clinicians most efficient access to information while treating patients.

4.3.3 Application management

There are five Cerner software domains implemented under the eMR project, and two additionalinstallations of FirstNet (SSWAHS, CHW) which have been independently implemented. A State BaseBuild (SBB) domain exists for development and demonstration of the SBB. These FirstNet domains aredescribed below.

29

SSWAHS*

CHW*

NSCCH AU

SEAHS_AU

SWAHS_AU

Observations

North Sydney LHD

Central Coast LHD

South East Sydney LHD

Illawarra LHD

Western Sydney LHD

Nepean Blue Mountains LHD

Sydney LHD

South West Sydney LHD

Children's Hospital at WestmeadHave own Cerner implementation - not part of eMRproject

Far West LHD

Western LHD

Murrumbidgee LHD

Southern LHD

Northern NSW LHD

North Coast LHD

Hunter New England LHD

GSGW AU

*Not part of eMR project - does not run Cernersoftware.

NCAH_AU

HNELHD*

There is considerable variance in each of these implementations, including variance in operatingsystems, Cerner versions, SBB versions, local configuration, and local implementation of capabilityoutside of the eMR scope. Current implementations are summarised in the following table:

30

Observations

NSCCH_AU

SEAHS_AU

SWAHS_AU

GSGW_AU

NCAH_AU

SSWAHS +

CHW +

SBB*

+ Not part of eMR project - run and manage own software and technology* SBB instance is used for SBB development and demonstration only.

This complex and varied environment presents a difficult and expensive application managementchallenge.

Application governance model and processes exist, both locally and centralised

The processes to request a change to the State Base Build are well defined and documented. Theseprocesses include both local recording and triage processes which then feed into the central triage,

recording, assessment and management processes. While these processes are well defined, the localleadership's focus and representation on the governance committees impacts the overall effectiveness ofthe process. It was reported that ED doctors often find it difficult to attend governance meetings, and thishas at times skewed prioritisation and progress toward the interests of those who attend.

Multiple versions of infrastructure, Cerner and build across the State adds to managementcomplexity

There is a plan and strategy to standardise core FirstNet configurations across the State, however,system changes during the rolling implementation process, and the capacity of sites to localise theirsystems has driven variances in the implementations at each site. This diversity drives both complexityand operational cost. At some sites new capability cannot be implemented as the base build at that sitedoes not support the required changes.

This variance in both the central FirstNet implementation and local integration architecture makes itdifficult to get an end-to-end view of any process across the State. This complexity leads to userfrustration about the responsiveness of support services, such as the time taken for issue remediation orthe effort required to diagnose and respond to a reported issue.

Plan in place to gradually migrate regions to a consistent build

HSS have developed a plan to migrate the required infrastructure and Cerner versions toward theprerequisites for the delivery of a more standardised State Base Build. This plan includes the upgradingof hardware, operating systems and Cerner versions. Successful execution of this plan should lead to amore uniform set of environments that can accept and support new standardised configurations andcapability.

2010, 02

i

HPUX HPUX 2010.02. V4

XYesHPUX

HPU {

HPUX

HPU'.

HPUX

HPUX

H P U X

HPUX

HPUX

c

31

Observations

An ED system requires the integration of a number of systems, including FirstNet, the hospital's PatientAdministration System, and various order receiving and laboratory management systems. Ownership ofthese integrating systems is also shared by HSS and the Local Health Districts. The method by whichthis integration has been implemented varies, and is dependent on what systems and integrationtechnology exists at each site.

We observed a number of integration models at different sites:

Centralised integration

Cerner PAS

Local integration engine

Local systems integrate directly to the eMR SBBintegration engine. This has the advantage thatthere is only one integration infrastructure tomaintain, however, it makes the centralisedintegration engine more complex to manage.

Also, there are many more interfaces to manage atthe intersection of responsibility between HSS andthe Local Health District. The central integrationenvironment must know about all of theimplementation details of the local systems.

In sites where the Cerner Patient AdministrationSystem (PAS) is deployed, the Cerner software

manages all interactions between the clinicalmodules and the PAS. This considerably simplifiesthe integration architecture. If additional Cernermodules are deployed, the integration architectureis further simplified.

Where a local integration engine is deployed theintegration between the centralised and locallymanaged environments is much simpler andmore easily managed. Local Health Districts aremore clearly responsible for integration of localsystems to the local integration engine.

This variety of implementation models increases the overall management complexity of the end-to-end

solution and contributes to a general lack of clarity about the management responsibility of the overallarchitecture.

32

Observations

4.4.'

Variety of integration models

As described above, we found a variety of integration models deployed across the State, as well asvariances and combinations of these models. This variety and complexity has added significantly to theoverall management effort required to operate and support the end-to-end solution. This variance inmodels makes it harder to maintain and diagnose issues, since the responsibility for interfacemanagement is different for each model and is often not well defined.

Varying integration reliability and performance

We observed a range of reports on the performance and reliability of interfaces with the eMR solution.Some sites reported reliable interfaces and little awareness or concern about failed or lost messages.Elsewhere it was reported that while the integration was reliable the performance was often too slow.And in some sites FirstNet went live without some core interfaces being implemented, and as a result,staff resorted to printing out orders for sending to Pathology for manual re-entry into the pathologysystem.

There were also some reports of interfaces that did not reliably translate and deliver the completemessage set. However, in most cases, this was associated with PAS integration and was beingaddressed and rectified. We did not receive any reports of there being specific adverse clinical outcomesas a result of integration issues.

Cerner supports `publish-subscribe' model

The FirstNet solution supports a `publish' model for message integration in which messages are`published' to an integration engine for translation and transmission. In this architecture, FirstNeteffectively passes all responsibility for the delivery of the message to the integration engine. As aconsequence, FirstNet does not necessarily find out if a message fails or contained errors. Equally, thereis no feedback to the user if a message has errors or fails. While this is not un-common in systemsintegration architectures, it places considerably more responsibility on the integration environment andmanual processes needed to review and action any interface problems.

4.4.2

_._ -As and compliance

Code and message standards have been generally well applied

The use of HL7 for defining messages and SNOMED for the definition of message content (codes) hasbeen generally well applied, and has resulted in a defined and standardised integration architecture.

NSW Health has only implemented a restricted set of messages which has aided with standardisationand management of integration, but has in some cases resulted in restrictions on the use of the system.For example, the system only allows `new order' or `cancel order' messages which makes it difficult forusers if they wish to make a change to an existing order. A number of clinicians expressed the need tochange an order. The maintenance of message and code standards across systems is manuallyundertaken, which contributes to the effort required to manage overall system integration.

Limited visibility of failed messages

It was reported that there is little to no user visibility of any messages that contain errors or have failed.In most cases this was not a significant problem for users, who generally found integration to be reliable.In the interface design documentation we observed scant consideration for error processing. In mostcases specifications defined error handling functionality with the statement `Use standard errorprocessing'. We expected to find greater detail on what technology, reporting and processes arerequired to identify and remediate messages that report errors.

33

Observations

4.4.4-1

Cerner provide support for outbound messages

Cerner is responsible for defining and delivering interfaces into and out of the eMR SBB interface engine.The interfaces and message content were designed, documented, built and deployed by Cerner. While

Cerner was engaged to develop the message set, the Local Health Districts maintain overallresponsibility for reliable message delivery.

Local Health Districts provide support for local system integration

Any required message translation and integration with local systems was the responsibility of the LocalHealth Districts, and any relationship they had with their local system suppliers. These interfaces weredeveloped at each site with little coordination across sites. It was observed that most interfacing systemssupport the required interfacing capability and standards, or have interfaces to local integration enginesthat enable appropriate integration to the eMR SBB integration engine.It was reported that in one example the system used by the Pathology laboratory operator could notmatch the laboratory results with the corresponding order and encounter within FirstNet (the systemcould track at a patient identity level only). FirstNet was enhanced to match the result to an encounter;however, the result of this matching is not completely reliable. To compensate, staff check results at theuser level, rather than just at the encounter level.

4.5.1 c:i

v°< v G.

p1 s aki1^ C 111

The primary vendor for the supply and implementation of FirstNet is the Cerner Corporation Pty Limited.The scope of the vendor's responsibility is comprehensively established via three related contractingdocument types:

• The original deed of agreement IT-135 between Cerner Corporation Pty Limited "theContractor" and Health Administration Corporation (i.e. the New South Wales Department ofHealth), dated 27 September 2002. The original deed of agreement provides for hardwareacquisition and installation, hardware maintenance, software licensing, IT consultancy,software development and modification, software support, systems integration, dataconversion and migration and transition out services. In addition, the agreement specifies arange of mechanisms and measures to govern the relationship with Cerner.

• GITC Official Orders lodged under the period purchasing agreement IT-135, detailing thescope of services to be provided relating to a specific site implementation.

• Change Requests covering the provision of software development services or support relatedto system enhancements or fixes.

The vendor's responsibilities appear to be adequately specified via the contracting mechanisms

While the requirements for system development and support are formally documented (as describedabove) the approach taken by NSW Health for managing the relationship with Cerner has generally beenless formal. While Cerner provides many of the performance reports as specified in the deed ofagreement (IT-35), Cerner has not been formally held to account by HSS in line with the performancegovernance model specified in the agreements.

That said, by most measures it would appear that HSS established an effective working relationship withthe vendor. Evidence of this may be found in the measures taken by Cerner to invest in the success ofthe program, such as providing capability and services beyond that specified in the agreements, andconducting annual reviews by international specialists.

34

Observations

4.5.2

'eness

A layered governance model operates to identify, assess and escalate to the vendor support and changerequests received from SBB sites. User support and the handling of change requests are facilitated vialocal site representatives. The local user group escalates to the State AAG and ultimately Cerner ifnecessary. Under the support model, Cerner's responsibility includes third level FirstNet support and thedevelopment and implementation of software changes based on direction from both AAG and at times,local areas or sites.

Evidence of varying support effectiveness, change management and inadequate codeconfiguration

Feedback on overall support effectiveness varied by area and site, and some reports of inadequatesystem change management and code configuration management were received. For example, usersexpressed frustration that software changes previously implemented would be lost after subsequentversion updates.

The implementation approach adopted was that of a rolling model whereby a team would implement thesystem at a site, including the delivery of initial training, then move on to the next site. The framework forthe Change Management and Communications approach was developed centrally by HSS, including thedevelopment of tools and templates. Each of the LHDs then localised the approach and materials to suittheir specific requirements and took responsibility for delivery.

Implementation approach did not allow for follow up reinforcement

There is evidence that the time spent by the implementation teams implementing the system andsupporting users to learn and become familiar with the system varied by site, and varied according to thesize of the site and the number of users. The ED Director of a small rural hospital reported that theimplementation team was on site for a total of 5 days. At some of the larger urban hospitals theimplementation team was reported as being onsite for weeks prior to and following implementation.

There was general agreement that there was insufficient support during the implementations andinsufficient follow up support after the implementation team moved to the next site. Some cliniciansreported that their hospitals had invested in dedicated support technicians to trouble-shoot and providesupport.

Consensus on the value and quality of the implementation team

Feedback received throughout the review on the value delivered by the implementation teams wasconsistently high. Clinicians consistently reported that the implementers possessed the requisiteknowledge of the system, clinical processes and implementation practices to effectively implement thesystem.

Varying levels of local leadership and investment to support change management andcommunications

Evidence was provided to our review of inconsistent support by senior management at some sites for the

implementation process, specifically in ensuring training attendance, implementation of changes to localwork practices and investment on local support capabilities. This variation generally correlates with thelevel of user acceptance of the new system at a site.

Transparency of change management lacking

When functional changes are made to the system, (either as a result of change requests or versionupgrades that have impacted the way clinicians use the system), the process has not always been

35

Observations

sufficiently transparent. Clinician's `discovery' of unexpected, unannounced or unsupported changes hasresulted in varying degrees of frustration. .

Change impact of the system on local clinical processes was not adequately addressed

There is some evidence that the system was not designed to optimise business processes effectively.Some clinicians reported that the system imposed changes to clinical processes which they regard assuboptimal (for instance, requiring patients to be `clerked' prior to triaging). In contrast to this, someclinicians felt that the system has improved discipline and enforced best practice, albeit with anaccompanying increase in the number of steps in some tasks or activities.

Limited communication or formal process to support sharing of [earnings

Across various sites, as users have become more familiar with the system, local configurations havebeen made to address many of these usability and process-related problems and enable more effectivework practices. Typically, these improved techniques have only benefited the site where they weredeveloped, as there is no formal process or assigned resources to enable improvements that could beshared to be communicated and leveraged by other sites.

4.6.2 Training effectivenessConsistent with the Change Management and Communications Framework, the framework for trainingwas developed centrally by HSS, including the development of tools and templates. A combination of a`train the trainer' approach was used, with Cerner developing supporting web based training tools. Eachof the LHDs then localised the approach and materials to suit their specific requirements. The approachand timeframes for training at each site was agreed between the LHD, implementation team and theclinical management team. This caused a degree of variation amongst sites both within LHDs andacross the State. There was no formal user testing or minimum requirement before clinicians adopted thesystem. Ongoing training and user support continues to be provided by local health services.

Varying views on effectiveness of training

There were mixed views reported from the clinician community regarding the effectiveness of the trainingdelivered to enable them to adopt and use the system. Some were of the view the content, amount anddelivery mechanism were appropriate, while others considered the training to be insufficient, consideringthe complexity of the system.

No formal follow up training program was provided after the implementation to re-enforce learnings andimprove the ability of users to leverage the system effectively.

Variable ongoing training support based on quality and investment in local training support

There are varying levels of investment in training support amongst the LHDs and sites. Some haveinvested in dedicated local support resources while others relied on services provided by the areas orState. This variation correlated strongly with the perception of quality and availability of training.

Attendance at training varied

Reports on the levels of attendance at training sessions varied. Reports suggested that nurses and otherclinical staff had a higher training attendance rate than doctors. While this was not supported by trainingattendance data, a number of clinicians commented that it was difficult to get doctors to attend training.This was thought to be due to a number of factors, including the fact that at some sites ED managersmade training a mandatory requirement for their nursing staff.

4.6.3 U

Level and clarity of support processes vary by site

As previously mentioned, a considerable variance in the user support processes across areas and siteswas found. Users reported that it is not always clear who should be contacted to provide support. Some

36

Observations

sites have recruited dedicated local support resources, some have trained `super-users', while others relyon the regional or HSS centralised support services only.

As with ongoing training, support organisations that invested in local 'full-time' support effectivelyinsulated their users from the support processes of the State. This has led to a range of assessments ofsupport from users, from those who are very satisfied with their support processes to those who areconsiderably frustrated.

This variance in support effectiveness has led to considerable variance in user skills, with some doctorsbeing relatively `expert' and able to establish specific lists and shortcuts, while others have only basicuser skills. The variation in support also correlates with the degree of local configuration undertaken toovercome usability shortcomings of the system.

4.7.1 ICT visio - -

^- -ategy

FirstNet as a building block in the broader eMR program

NSW Health is engaged in an eMR program that consists of a number of project streams, includingFirstNet eMR, eMR Phase 2, and Medications Management. FirstNet eMR (the subject of this review) iscompleting the implementation of an eMR for EDs using components of the Cerner Millennium suiteincluding Cerner FirstNet, Cerner PowerNote, PowerChart, and integration with PAS' s and laboratorysystems. A subsequent project stream, eMR Phase 2, will build out additional clinical functionalityincluding further enhancements and change requests to Cerner FirstNet, device integration, and furtherdeployment of PowerChart. Another stream is planning to deploy the Medications Management module.A high-level planning roadmap for these and other initiatives is maintained by NSW Health for thepurposes of planning business cases and budget submissions over a five year period.

Other than this roadmap and the associated business case for funding purposes, we are not aware ofany documented vision or strategy that articulates the overall objectives of the eMR program, relativepriorities or dependencies, or a roadmap to provide more granular planning of the overall program. As aconsequence, the program cannot refer to a 'big picture' view that would guide investment decision-making, and questions remain about whether the program is pursuing a State-wide eMR, multiple eMRs,hospital automation, or a combination of these end- states and objectives.

4.7.2 Business cdz governance

As part of the broader eMR program, FirstNet must support the State's established governance approachto enable the effective monitoring and utilisation of the resource base. This will require FirstNet to supportstatutory reporting, and reporting on clinical service delivery and hospital operations.

Effective governance requires FirstNet to support adherence to security, user identification, authorisationand audit policies and protocols.

Reporting does not enable clinical governance improvements to be measured

With a few exceptions, reporting has been flagged as a common area of frustration. Managers reportedsignificant difficulty and frustration in accessing information to assist with managing their ED and otherfunctions, as well as enabling a view of clinical trends. Some clinicians also reported a distrust of the dataquality provided in reports, as noted previously.

Inadequate user identification and authorisation practices

At some sites users reported shared user sessions to avoid delays associated with logging out and thenlogging back in as a new user. Also, the requirement for clinicians to enter a user identifying PIN numberto authorise a transaction has been disabled at some sites. This means that it is not possible to reliablydetermine the identity of the clinician processing transactions. This issue is exacerbated where cliniciansshare computers or user sessions.

37

Observations

4.7.3 Systen- `

System governance processes for managing change requests can be described as follows:

• Upon identifying a change request the end user logs this with the local user group,sometimes referred to as the local Application Advisory Group (AAG) for review.

• The local user group reviews each change request to determine whether to put it forward tothe FirstNet State AAG. If the submission proceeds, the change request is logged by theState AAG for review and endorsement

• The FirstNet AAG Triage Group review changes and endorse, delegate to the AAG, ordecline each one. All decisions made at Triage are then attached to the following AAGAgenda available for members to review, raise and revisit the decision if necessary

• If endorsed (as either mandatory or optional) each change request will be placed into scopefor the next version of SBB conceptual design. LHDs are able to implement a change requestprior to it appearing in the next SBB version. Actual configuration into the LHD is currently theresponsibility of the Local Support team.

• The SBB Team tracks these changes and works with the LHDs to update the ChangeControl Tracker

• If a code change is required (i.e. the change request is an enhancement) this is raised withCerner, which liaises with HSS to ensure the change is made.

The system governance process is in place, however, there was evidence this was not effective

There is a clearly defined and established system governance process in place (as summarised above),however, a number of clinicians reported frustrations with the effectiveness of this process. Commonly,clinicians reported a perceived slowness and delays in change request implementation. One clinicianreported a change request on the FirstNet discharge summary taking 18 months to be implemented.

There was evidence that one of the factors contributing to delays of this sort was that the change requestprocess is not adequately robust. In some cases, requirements are not always sufficiently well articulatedat the outset leading to rework and delays. Another potential factor is that some change requests areapproved by the State AAG but not immediately implemented at the LHD level.

Other themes commonly reported include a lack of visibility of progress or status, and the need for manyseemingly simple requests to require product enhancements.

The imbalance of clinicians and administrators on local AAGs influences change requestdecisions and priorities

Given these and other frustrations with the change request process, some clinicians reported that theyhad chosen not to participate in the local user groups. Also, some clinicians noted that local user groups

were not held at convenient times for clinicians. These factors have contributed to the waning of clinicalrepresentation on local user groups. This in turn has influenced change request priorities and decisions.

38

Recommendations

5 Recommendations

NSW Health has implemented FirstNet as part of the overall eMR program to 59 hospitals across NewSouth Wales. While there have been a number of reported issues associated with the deployment, useand operation of the system, many clinicians have reported that it now adds considerable value to thedelivery of care and the operation of their Emergency Departments.

In the month of March 2011, 86 million transactions were processed; 2.84 million orders were placed and17,188 users accessed the system on a daily basis.

Many of the issues that have been reported and observed relate to system operations, governance,implementation, leadership and training, and not specifically to the capability of the FirstNet system.While the review confirmed that there are a number of issues with the use, implementation, support andconfiguration of FirstNet, we believe these can be remediated.

In deciding the future of the FirstNet solution, it is important to recognise that it comprises an importantfoundational component of a broader eMR architecture, where a patient's medical record may be storedand accessed from within a number of care settings across a continuum of care. Replacing FirstNetwould require a new solution to be integrated with the broader Cerner based eMR solution set. While thisintegration is possible, it would add an additional level of complexity into what is already a very complexenvironment.

A range of issues and frustrations were identified across the FirstNet sites examined, which are notablyimpacting user satisfaction and the effective use of the system.

These issues highlight the urgent need for a formal program of site specific remediation activities. Theseneeds might include remedial training, configuration and in some cases, upgrades to the currentlyimplemented system. A well defined plan and program of work needs to be urgently put in place to bringall sites to a base level of acceptable infrastructure, functionality, useability and user training.

From this context, we strongly recommend that the following activities be performed:

[an

A detailed independent review should be undertaken of each site to identify what issues exist and whatremediation activities are required to elevate the system and its use to a defined minimum standard. Thisminimum standard should be based on the capabilities delivered in the latest version of the SBB.

This review should be executed as a matter of urgency.

We further recommend that this remediation program be delivered by resources other than those alreadyresponsible for the current implementations or the planned eMR phase 2 to avoid resource conflicts withthese activities.

39

Recommendations

plan

A formal project should be established and funded to implement the recommendations set out in theremediation plan.

There is currently considerable variance across each of the implementations of FirstNet in the LHDs.While NSW Health has adopted the concept of a State Base Build (SBB) - many of the implementationsare deployed on different operating system versions; different Cerner versions; different SBB versions;different local configurations; and in some cases, local additions that are outside of the scope of the eMRprogram. This is driving inefficiency and complexity in the use, operation, support and management ofthese systems.

To address this issue, we propose the following recommendations:

5.3.1 Define and expand tl^ _ ` - ^ cope

To the extent that is possible, NSW Health should expand the scope of the SBB to include many of thecurrently variable components of the local implementations, and then standardise where appropriate theimplementation of the SBB across the LHDs. This will drive greater efficiencies of operation and enablethe sharing of many of the local configurations that are not currently available to all.

5.3.2 Develop a plan for expansion

A plan must be developed for the definition and implementation of all the components required for thenew SBB. This plan must take into account any remediation work that is required to address as many aspossible of the issues associated with the current implementations while taking into account any plannedimplementation and enhancement activities, including the implementation of eMR Phase 2 andMedications Management.

Analysis should also be undertaken to determine how those sites that are currently not part of the State-Base Build program i.e. non-SBB sites, would be managed going forward.

5.3.3 C-

to implement eMR 2 and medications ionalityThere are currently plans and budget submissions for the implementation of the eMR phase 2 andmedications management capability. This includes the addition of new clinical support functions as wellas the integration of `real time' monitoring devices into the Cerner system. It is our view that NSW Healthshould progress with these plans to implement this new capability in alignment with the approachrecommended for the new and expanded SBB.

5.3.4 E

Considerable dissent was expressed during the interviews regarding the reporting capability that isprovided by the current implementation of FirstNet. ED Directors and hospital managers need access tothe rich information that exists within FirstNet to better manage the day-to-day activities under theirresponsibility and to understand how their clinical and administrative strategies are performing.

NSW Health must establish a reporting capability that readily enables the development of new reportsand enables managers to personally interrogate the available information. Our initial investigationssuggest that the resolution of this issue will include a review of the use of data input validation rules, thereporting tools available and training of users in the use of these tools.

40

Recommendations

t SBA

licE

In developing a more standard approach to the definition, implementation and management of the SBB, aclear and effective communication plan must be developed and put in place. It will become increasinglyimportant that all SBB stakeholders are well informed of strategies, new capability, and current activitiesto improve the FirstNet system.

ovation

` C

ins to

A number of situations were observed where useability problems experienced by one or more users hadbeen solved by local configuration or modified practices at another site.

An exercise should be undertaken to establish where such changes have taken place and how thesemay be incorporated into the next release of the SBB and shared across the sites.

A longer term strategy and process should be put place to enable all stakeholders to understand andshare any local innovations and configurations.

NSW Health has made substantial steps toward the implementation of an eMR for all patients presenting

at hospitals across the State. The eMR will provide a basis for participating in the broader nationalPersonally Controlled Electronic Health Record (PCEHR) program, while providing the longitudinal recordof patient care essential for addressing the ever increasing demands on the health system.

The FirstNet solution and the broader eMR program provides key tools for NSW Health and the LocalHealth Districts to enable the delivery of improved care across the full continuum of care, which oftenstarts at the Emergency Department. These tools provide clinicians with improved access to informationto support the treating of patients, and to understand how clinical processes can be changed to improvepatient treatment outcomes and the utilisation of resources.

With this is mind, we would recommend the following:

5.4.1 a Chief Medical Information Officer (C_MIO) role

As IT systems are increasingly used to support the delivery and improvement of health care, it isimportant that senior clinical input drive the vision and requirements for the use and future developmentof clinical systems. We recommend the creation of a Chief Medical Information Officer (CMIO) in NSWHealth to provide this direction.

The Chief Medical Information Officer (CMIO) role should be established to provide the additional

leadership required to ensure that the future implementations and operation of FirstNet and other clinicalsystems are successful. This is an executive role which is responsible for the development of a vision forthe capabilities necessary to enable the care delivery strategies of the Department.

5.4-- Oil

Establish a vision for the clinical processes and necessary systems capability required to enable the care

delivery strategies of the Department. This will become increasingly important as patients' medicalrecords become more electronic, and start to be shared outside of the department, including with thePersonally Controlled Electronic Health Record (PCEHR). This vision should provide the basis fromwhich all systems implementation strategies will be defined and developed into implementationroadmaps. This vision will be fundamental to the successful delivery of future clinical based systems.

41

Recommendations

The success of systems introduced to support clinical functions requires appropriate funding for thesystem's initial implementation and on-going operation. The review identified that the funding modelestablished for the FirstNet program have been very project driven. While this approach has successfullysupported the initial implementation of the system, it has not expressly provided for the iterativeretrospective application of enhancements resulting from lessons learned and ongoing clinical processimprovement. More specifically, the on-going system development and training support postimplementation appears not to be expressly provided for under the current funding model. This negativelyimpacts user satisfaction and adoption and the harvesting of benefits expected from the system.

5.5.1 Review program f

modelsIt is therefore important that funding for the future phases of the eMR program includes bothimplementation and on-going operating costs.

There are multiple parties involved in the implementation and delivery of FirstNet. HSS has responsibilityfor delivery of the core application, integration and infrastructure; the LHDs have responsibility for localsystems, local infrastructure and systems integration; Cerner provides application implementation andsupport services; and the Department provides governance, funding and leadership support.

Evidence suggests that the specific responsibilities of each organisation involved in the implementation ofFirstNet and the broader eMR program are not well defined and understood by all stakeholders. Thisuncertainty has driven inefficiency and stakeholder discontent.

5.6.1 Establish

and t_*-._

nt program governance and organisationalstructures

An assessment of the governance and organisational structures required to effectively deliver a computerenabled clinical care system must be undertaken. Any changes to organisation structure, delivery andsupport model, and governance processes must be clearly defined and implemented.

There was considerable discontent expressed regarding the visibility and effectiveness of the systemoperation and user support processes. Users are often uncertain as to who is responsible for addressingsupport issues and what the proper process is for raising requests for change. The level of local supportavailable also varied greatly from site to site; in some cases the local organisations had clearly investedsignificantly to establish local expertise and support capability, while others relied mainly on the centrallydelivered support services.

42

Recommendations

Visibility and feedback on the status of requests for support or change is also an issue. It was reportedthat in many cases, requests were made and little or no feedback was received, and some cases noaction was ever taken.

5.°7.1 EstabIis'

and transparent

operation and uses

tprocesses

It is important that the Department establish the system operation and user support process required toensure that users of FirstNet achieve the maximum benefit from the use of the system. In addition, usersshould have access to appropriate ongoing training to ensure they have the skills to effectively use thesystem.

These redefined processes must ensure that the users of the system have clarity, transparency andcertainty about any requests they may make.

As the hospitals across the state move more towards a paperless care delivery model, the reliance on theavailability of the clinical systems and the patient information in those systems becomes critical to theclinicians ability to deliver effective care.

It is essential that highly available and high performance infrastructure be provided to support the deliveryof this increasingly critical IT service. Evidence suggests that the existing infrastructure does notuniversally meet this minimum standard.

We therefore recommend that the following activities be undertaken.

5.8.1 Define infrastructure delivery approach

A review should be undertaken of the infrastructure delivery approach, and an assessment made as tothe suitability and capability of each contributing party. This review should also include an assessment ofalternate sourcing approaches. Many international users of Cerner do not run their own data centres,and outsource the hosting and management of the core infrastructure required to run the application tospecialist third parties.

5.8.2 P..'

A'_n- .. °t_

)rt model

Evidence suggests that the support processes and responsibilities for the delivery of infrastructuresupport services are not clearly defined and understood by many users. The infrastructure supportmodel and the responsibilities of all parties involved in the delivering support should be clearly definedand communicated.

5.83 Define "

-` -f u 1

There is a broad variety of technologies deployed to enable the delivery of FirstNet. The size andlocation of PCs varies widely; there are multiple network and application integration designs; and a rangeof server environments and operating systems. A Standard Operating Environment (SOE) should bedefined and included in all future upgrade and implementations plans.

43

Appendix A

Appendix A - Interview List

We interviewed and/or conducted meetings with the following people in the conduct of thisengagement:

Name Position Interview date(s)

Tony Azzam eMR SBB Program Manager 24-May & 14-Jun

Ronan Herlihy Benefits Realisation Mgr - HSS 24-May

John Frisken CTO - Unitech Solutions Group 25-May

Dr Michael Barnet Director - Meridian Health Informatics 25-May

Prof Jon Patrick Professor - University of Sydney 26-May

Dr Tim Smyth Deputy Director-General, NSW Health 26-May

Howard Dawson Education co-ordinator - HSS 30-May

Cameron Burt General Manager - Cerner 31-May & 27-Jun

Prof Joanna Westbrook Director, Centre for Health Systems and SafetyResearch - UNSW 1-Jun

Joanne Callan Senior Research Fellow - Centre for HealthSystems and Safety Research - UNSW 1 -Jun

Greg Wells Chief Information Officer - HSS 2-Jun & 28-Jun

Tim Hume Director, Strategy and Architecture - HSS 2-Jun

Dr Richard Paolini Head of Emergency Department, ConcordRepatriation Hospital 2-Jun

Anthony Futia Data Integrity liaison Officer and ED DataCollection Co-ordinator 3-Jun

Barbara Howell Clinical Manager - Emergency Department -Wollongong 7-Jun

Dr Tom Corrigan Staff Specialist- Emergency Department -Wollongong 7-Jun

Kelly Peterson eMR Project Director - Northern Sydney LocalHealth District 8-Jun

44

Appendix A

Dr David Rivett GP Proceduralist - Batemans Bay 8-Jun

Dr Alan Forrester Chair FirstNet AAG/ Network Director Emergency 8-JunServices

Karen Braid Clinical Nurse Consultant Clinical Projects & 9-JunInformatics - Nepean Blue Mountains Hospital

Dr Rob Davies Staff Specialist, Emergency Dept - Tweed 9-JunHospital

Dr Rod Bishop Co-chair MTEC / Staff Specialist, Dep't of 9-JunEmergency, Nepean Blue Mountains Hospital

Dr Trevor Chan Staff Specialist, Emergency Dept, St George 9 -JunHospital

Leanne Ovington ED/HDU Nurse Unit Manager, Moruya / Batemans 10-JunBay Hospitals

Dr Linda Dann Director Emergency Department, Bankstown 10 JunHospitalDirector of Medical Service & Director, Medical

Dr Colin MacArthur Assessment Unit & Staff Geriatrician, Liverpool 10-Jun & 23-JunHospital

Dr Randall Greenberg Director emergency Department, Dubbo Hospital 14-Jun

Dr Matthew Vokasovic Emergency Department Director, Westmead 15 JunHospital

Dr Sally McCarthy Medical Director, Emergency Care Institute 15-Jun

Dr Mathew O'Meara Director, Emergency Department, Sydney 16-JunChildren's Hospital Network, Randwick

Dr Michael Hession Staff Specialist, Emergency Dept, Blacktown 16-JunHospital

Rosemary Beenie Nurse Manager Clinical Informatics, Liverpool 17-JunHospital

Dr Adam Chan Director, Emergency Department, St George 23-JunHospital

Dr Roger Trail Anaesthetist, Department of Anaesthesia, Royal 24-JunPrince Alfred Hospital, Camperdown

45

Appendix B

Appendix B - Acronyms

Health Support Services

State Base Build

NSW Health program to implement Cerner Millennium suite of products toenable support for clinical functions

EMR

Electronic Medical Record

AAG

Application Advisory Group

LHD

Local Health District

SSW

Sydney South West

CHW

Children's Hospital Westmead

SSWAH

Sydney South West Area Health

NSCCH

North Sydney Central Cost Health

SWANS

South West Area Health Service

GSGW

Greater Southern Greater Western

NCAH

Northern Coast Area Health

HNELHD

Hunter New England Local Health District

SEAH

South East Area Health

PAS

Patient Administration System

ECG

Electro Cardio Gram

RIS

Radiography Information System

PACS

Picture Archiving & Communication System

JCAPS

Java Composite Application Platform Suite

SNOMED

Systemised Nomenclature of Medicine

CMIO

Chief Medical Information Officer

ED

Emergency Department

AHS

Area Health Service

UI

User Interface

HSS

SBB

eMR

46

Appendix B

PCEHR

Personally Controlled Electronic Health Record

SOE

Standard Operating Environment

ICT

Information and Communication Technology

GITC

Government Information Technology Condition

47

Appendix C

Appendix C - SourceDocuments

1 Cerner NSW Health Contract HSS

2 State Baseline Build - Emergency Department Design Document HSS

3 EMR Architecture Strategy HSS

4 Contractor Specifications HSS

A Critical Essay on the Deployment of an ED Clinical Information System - Systemic5Failure or Bad Luck? Internet

6 Electronic Medical Record - State Baseline Build - Approach to Content Development HSS

7 Final Report of the Special Commission of Inquiry - Acute Care Services in NSW PublicHospitals Internet

8 Electronic Medical Record - Post Implementation Benefits Tracking Summary Report HSSDecember 2010

9 "Lights On" Information - Northern NSW & Mid North Coast HSS

Orders for all sites1. GSAHS Cerner_official order_Cerner-2909062. GWAHS Cerner_official order_Cerner3. North Coast Area Health Service10 4. South Eastern Sydney Illawarra Area Health Service HSS

5. Sydney South West Area Health Service6. Sydney West Area Health Service7. The Children's Hospital at Westmead

11 2010 NSW HEALTH VALUE REVIEW HSS

12 EMR PACSRIS ROLLOUT (as at 4 May 2011) HSS

13 Cerner Millennium Support Model HSS

14 Client Profile Metrics Explanations HSS

15 2010Q4 SolutionWorks Millennium Applications Client Dashboard HSS

16 Tender Evaluation Document 191205 HSS

EMR Master File All Finalised RFT documents HSS1. Appendix E_H_I_J_K_EMR_IT 190

17 2. Appendix A - Tenders Response Forms3. Appendix B Master HL7 Ref Guide4. Appendix C_IPS IT 189_EMR_IT 1905. Appendix D GITC RFT it - 190

48

Appendix C

6. Appendix E_Area Configurations_EMR_IT 1907. Appendix F_Demonstration Scripts_EMR_IT - 1908. Appendix G HL Compliance Resolutions_EMR_IT 1909. Appendix H - Software Acceptance Testing_EMR_IT 19010. Appendix I _I PS Methodology_EMR_IT_19011. Appendix J- Definition of Terms_EMR IT 19012. Appendix K_Non negotiable_EMR_IT 19013. EMR RFT 190 Final Version

18 Certificate of Registration - Quality Management System - ISO 13485: 2003 Cerner19 Certificate of Registration - Quality Management System - ISO 9001: 2008 Cerner20 How does information technology impact on quality of care? Johanna Westbrook

21What impact do emergency department information systems have on nurses' access toinformation? A qualitative analysis of nurses' use and perceptions of a fully integratedclinical information system

Johanna Westbrook

22 Use of information and communication technologies to support effective work practiceinnovation in the health sector: a multi-site study Johanna Westbrook

23 Cerner consulting and managed services HSS24 EMR2 Final Business Case HSS25 EMM Business Case 141010 Final HSS26 Cross Client Reporting: User Experience All Positions Cerner27 Cross Client Reporting: Client Profile Report Cerner28 Lights on Network Usability Dashboard User Guide (PowerOrders) Cerner29 Australian IP Overview Cerner30 Meetings Overview Cerner31 2010Q4 SolutionWorks Millennium Applications Client Dashboards Cerner32 Milestone Management Updated Cerner33 NSW_AU Milestone Management Cerner34 EMR Domain Status HSS35 Implementation Status of Strategic Products or Applications at each local Health Network HSS

36

2010Q4 SolutionWorks Millennium Applications Client DashboardClient # 45155 / CernerWorks Client / 2007.19Greater Southern Area Health Service (Queanbeyan, NSW)

Cerner

37

2010Q4 SolutionWorks Millennium Applications Client DashboardClient # 64968 / CernerWorks Client / 2007.19Greater Southern Greater Western (Chatswood, NSW)

Cerner

38

2010Q4 SolutionWorks Millennium Applications Client DashboardClient # 39130 / Not CernerWorks Client /Health Support Services (NSW) (North Sydney, NSW)

Cerner

39 "Lights On" Data - NSW Health - User Experience -April Cerner40 Lights on Data June 10 Cerner41 Lights on Useability April Cerner

49

Appendix C

42 NSW_AU Container Membership Report Cerner

43

2010Q4 SolutionWorks Millennium Applications Client DashboardClient # 21164 / CernerWorks Client / 2007.19North Coast Area Health Service (Lismore, NSW)

Cerner

44

2010Q4 SolutionWorks Millennium Applications Client DashboardClient # 20732 / CernerWorks Client / 2007.18Northern Sydney Central Coast (Gosford, NSW)

Cerner

5

2010Q4 SolutionWorks Millennium Applications Client DashboardGreater Southern Area Health Service (Queanbeyan, NSW)Greater Southern Greater Western (Chatswood, NSW)Health Support Services (NSW) (North Sydney, NSW)North Coast Area Health Service (Lismore, NSW)Northern Sydney Central Coast (Gosford, NSW)Health Administration Corporation NSW (North Sydney, NSW)

erner

46

2010Q4 SolutionWorks Millennium Applications Client DashboardClient # 855 / CernerWorks Client /Health Administration Corporation NSW (North Sydney, NSW)

Cerner

47 NSW eMR Learning Strategy HSS48 eMR Lessons Learned HSS49 Final Training Numbers Discipline NCAHS HSS50 elearning.htech.health.nsw.gov.au (2009 - 05) HSS51 elearning.htech.health.nsw.gov.au (2010 - 05) HSS52 elearning.htech.health.nsw.gov.au (2011 - 05) HSS53 elearning.htech.health.nsw.gov.au (2008 - 05) HSS54 Statistical Data on Class Attendance - Sector HSS55 SESI eMR Training Evaluation HSS56 Electronic Medical Record Training Evaluation HSS57 South Eastern Sydney Illawarra health Learning Plan HSS58 Sydney West Area Health Service Learning Plan HSS59 Greater Southern Health Learning Plan HSS60 Greater Western Area Health Service eMR Learning Plan HSS61 North Coast Area Health Service Learning Plan HSS62 Learning Plan Development Session (LPDS) Agenda HSS63 LPDS Gap Analysis HSS64 Risk Assessment Tool HSS65 The Learning Plan Development Session HSS

66 Emergency Dept Discharge Summary Children's HospitalRandwick

67 SCh Triage 4 Performance 2007 - 2010 Children's HospitalRandwick

50

Appendix C

68 What attributes did Cerner have in Oct 2002 that gave it Value? HSS69 Greater Southern and Greater Western Area Health Service eMR Business Case HSS70 GWAHS and GSAHS Project Charter eMR Implementation HSS73 eMR Post Implementation Action Plan_Goulburn HSS74 eMR Post Implementation Action Plan_Queanbeyan HSS75 Greater Southern Area Health Service (GSAHS) eMR Post Conversion Assessment HSS76 GSGW - Bathurst Base Hospital Deliverables HSS77 Greater Western Area Health Service (GWAHS) eMR Post Conversion Assessment HSS78 Cover Letter - PIR for Goulburn Base Hospital HSS79 Attendance list for User Acceptance Testing HSS80 High level timeline GSGW Baseline HSS81 Attendees - NCAHS FirstNet Advisory Group HSS82 North Coast Area Health Service eMR Business Case (Part A) HSS83 NCAH PCA FirstNet Deliverables HSS84 North Coast Area Health Service NCAHS eMR Post Conversion Assessment HSS85 NCAHS Event Evaluation HSS86 Post conversions review trip report_NCAHS HSS87 North Coast Area Health Service Project Charter HSS88 FirstNet UAT Attendees HSS89 Northern Sydney Central Coast Area Health Service Project Charter HSS90 Steering Committee Meeting Papers 20100527 - 29119526 HSS91 South Eastern Sydney Illawarra Area Health Service eMR Business Case HSS92 Feedback - SESI HSS93 SESIAHS - Project Charter Comments HSS94 SESIAHS End System Validation Assurance Report Sign Off HSS

95Post Conversion Report - Implementation of the Electronic Medical Record at St GeorgeHospital HSS

96 South Easstern Sydney Illawarra eMR Readiness Assessment Sign Off HSS97 Post Conversion Report - The Sutherland Hospital HSS

98The Wollongong Hospital - Electronic Medical Record Post Conversion AsssessmentReport HSS

99 SESIAHS Event Attendance Sheet HSS100 SESAHS - AU Event Attendance Sheet HSS101 Terms of Reference - SESIAHS FirstNet Clinical Advisory Group HSS102 SWAHS eMR Stack I Project Summary HSS103 Approved Plan A HSS

51

Appendix C

103 Attendance FirstNet SME Group HSS104 Chang Register FN Mad Nurse Clerical HSS105 Cerner Event Activity Report HSS106 Area Health Service FirstNet Application Advisory Group HSS107 Sydney West Area Health Service Project Charter HSS108 SWAHS Stack 2 Rollout Plan HSS109 Test Level Completion Report HSS110 SWAHS Event Activity Report HSS111 Health Technology DietOrders_Integration_Components HSS112 EMR Diet Order Interface HSS113 RIL Interface High Level Architecture HSS114 Health Support Services EMR Integration Plan HSS115 GW/GW iPM - EMR ADT Interface HSS116 EMR QR Interface HSS117 Health Support Services Integration Design - S.E.R Orders and Results HSS118 Health Technology Interface Design -ADT Messages HSS119 Interface Design -ORM Messages HSS120 GW AHS - GW AHS Radiology Interface Plan HSS121 EMR eGate NC Network Diagram HSS122 EMR NC AHS Order Interfaces HSS123 Health Technology Integration Design - NC AHS Orders HSS124 Health Support Services - Integration Design CRIS Orders and Results HSS125 NSCC Path Net Orders Interface Design HSS126 NSCC Diet Order Interface Architecture Diagram HSS127 Interface Project Plan - NSCC EMR HSS128 NSCC Millennium (EMR) Ward/ED/PAC HSS129 NSCC Millennium (EMR) Ward / ED / PAC HSS130 Millennium Foreign System Interface: GSAHS and GWAHS HSS131 Vision Software Technologies HL7 Interface Specifications HSS132 Vision Software Technologies Royal North Shore (RNS) HL7 Interface Gap Analysis HSS133 EMR QR Interface HSS134 EMR SBB Integration Engine HSS135 The PRB Problem Detail Segment HSS136 Millennium Foreign System Interface: SESIAHS Version 1.1 HSS137 Millennium Foreign System Interface: SESIAHS Version 1.4 HSS

52

Appendix C

138 Millennium Foreign System Interface, Foreign System i.PM Version 1.1 HSS139 Millennium Foreign System Interface, Foreign System Omnilab Version 1.1 HSS140 Millennium Foreign System Interface, Foreign System PMI/ADT Upload Version 1.0 HSS141 Millennium Foreign System Interface, Foreign System e*Index and i.PM Version 1.2 HSS

142FSI HL7 Universal Interface Specifications Unit20BBlood Bank Permanent Patient Dataand Blood Bank Product and Transfusion History Upload to PathNet Millennium

HSS

143 Foreign System Interfaces HL7 Universal Interface Specifications HSS144 Foreign System Interfaces HL7 Universal Interface Specifications Unit 8i; ADT HSS

145Foreign System Interfaces HL7 Universal Interface Specifications Unit 90: Order MessageProcessing Outbound HSS

146 Health Technology SWAHS EMR Interface Plan HSS147 List of External Systems Interested in the HL7 message testing HSS148 SWAHS Millennium Prod Interface Audit HSS149 Health Technology TSS Design - AD Design for Application Delivery by Citrix HSS150 EMR Domains and Servers - July 23 HSS151 EMR High Level Infrastructure Design V1.1 HSS152 eMR Summary Transaction Statistics HSS153 North Coast Area Health Services - HNAM High Availability Operational Guide HSS154 Sydney West Area Health Services - HNAM High Availability Operational Guide HSS155 Northern Sydney Central Coast - HNAM High Availability Operational Guide HSS156 SESIAHS EMR Performance Report Post St George Go Live HSS157 AHS eMR Quality Plan HSS158 FirstNet AAG Minutes (2009-11) HSS159 What is SBB? HSS160 SBB FirstNet Change Request Tracker HSS161 SCh Triage 4 Performance 2007 - 2010 HSS

53

Appendix D

Appendix D Chronology ofEvents

March 2000 Report of the NSW Health Council issued

2002 Point of Care Clinical Business Case issued and approved, partially funded

September 2002

Original deed of agreement with Cerner signed

May 2005

eMR tender released to explore options for an alternative eMR solution

Dec 2005

Evaluation report complete

May 2007

Former Sydney South West Area Health Service: First site goes live - Non State Base Build -Liverpool

14 May 2008

Former Sydney South West Area Health Service: First site goes live - Non State Base Build -Bankstown

30 September 2008

Former South East Sydney & Illawarra Area Health Service: First site goes live for the State BaseBuild eMR implementation (including FirstNet) - St George

24 November 2008

Former North Coast Area Health Service : First site goes live for the eMR implementation(including FirstNet) - Lismore

18 March 2009

Former Sydney West Area Health Service : First site goes live for the eMR implementation(including FirstNet) - Westmead

24 March 2009

Former Sydney West Area Health Service: Last site goes live for the eMR implementation(including FirstNet) - Nepean

26 October 2009

Former South East Sydney & Illawarra Area Health Service: Last site goes live for the eMRimplementation (including FirstNet)

26 October 2009

Former North Coast Area Health Service: Last site goes live for the eMR implementation(including FirstNet) - Port Macquarie

12 May 2010

Former Greater Southern Area Health Service: First site goes live for the eMR implementation(including FirstNet) - Goulburn

15 June 2010

Former Greater Western Area Health Service: First site goes live for the eMR implementation(including FirstNet) - Bathurst

6 July 2010

Former Greater Western Area Health Service: Last site goes live for the eMR implementation(including FirstNet) - Dubbo

28 July 2010

Former Greater Southern Area Health Service: Last site goes live for the eMR implementation(including FirstNet) - Wagga Wagga

54

Appendix D

12 April 2011

Former Northern Sydney Central Coast Area Health Service: First site goes live for the eMRimplementation (including FirstNet) - Gosford

About beloitte

Deloitte refers to one or more of Deloitte Touche Tohmatsu Limited, a UK private company limited by guarantee, and its network of member firms,each of which is a legally separate and independent entity. Please see www.deloitte.com/about for a detailed description of the legal structure ofDeloitte Touche Tohmatsu Limited and its member firms.

Confidential - this document and the information contained in it are confidential and should not be used or disclosed in any way without our priorconsent.

© 2011 Deloitte Touche Tohmatsu. All rights reserved.

55