PRESENTATION

Keynote Address: United KingdomExperiences of Evaluating Performance andQuality in Emergency MedicineSuzanne Mason, FRCS, FFAEM, MD

AbstractDemand for emergency care is rising throughout the western world and represents a major publichealth problem. Increased reliance on professionalized health care by the public means that strategiesneed to be developed to manage the demand safely and in a way that is achievable and acceptable toboth consumers of emergency care, but also to service providers. In the United Kingdom, strategieshave previously been aimed at managing demand better and included introducing new emergency ser-vices for patients to access, extending the skills within the existing workforce, and more recently, intro-ducing time targets for emergency departments (EDs). This article will review the effect of thesestrategies on demand for care and discuss the successes and failures with reference to future plans fortackling this increasingly difficult problem in health care.

ACADEMIC EMERGENCY MEDICINE 2011; 18:1234–1238 ª 2011 by the Society for AcademicEmergency Medicine

C rowding in emergency departments (ED) is aninternational problem for which we have, todate, found no solutions. The majority of inter-

ventions that have been tried have not been evidence-based and have been locally implemented rather thanaimed at delivering a national or international agenda toencourage change. Change may occur through the modi-fication of help-seeking behavior of patients, the avail-ability of alternative pathways of care for patients, theprocesses of care once patients are in the ED, or the easewith which patients are able to leave the ED after

receiving care. This article will review some of the U.K.evidence and experiences of managing crowding whilealso trying to deliver high quality patient care in the ED.

DEMAND FOR CARE

Demand for emergency care is rising and represents amajor public health problem. Table 1 indicates theincreasing scale of the problem worldwide.1 Risingdemand is being driven by public behavior and alsochanges to health care policy. In the United Kingdom,as in most other westernized countries, the key issuesdriving increasing demand for emergency care by thepublic include accessing a higher level of care than isactually required,2,3 partly due to an increased relianceon professionalized health care, and higher publicexpectations of health care. In addition, increased socialmobility and a lack of robust social care structures,along with an aging population, make patients morelikely to access emergency care rather than rely ontheir social networks. Coupled with this, there is anincreased ability to deliver efficient and timelyemergency care, which may partially drive increasingexpectations from the public.

In the United Kingdom, health care policy change hasalso presented challenges in the ability to deliver timelyand high-quality emergency care. The government hasrepeatedly asked for more convenience in health caredelivery, with care being taken to the patient, andincreased choice in how they access care. This hassometimes had the opposite effect of driving ED

ISSN 1069-6563 ª 2011 by the Society for Academic Emergency Medicine1234 PII ISSN 1069-6563583 doi: 10.1111/j.1553-2712.2011.01237.x

From the Director of Health Services Research, School ofHealth and Related Research, University of Sheffield, Sheffield,UK.Received July 13, 2011; revision received August 1, 2011;accepted August 2, 2011.Funding for this conference was made possible (in part) by1R13HS020139-01 from the Agency for Healthcare Researchand Quality (AHRQ). The views expressed in written conferencematerials or publications and by speakers and moderators donot necessarily reflect the official policies of the Department ofHealth and Human Services, nor does mention of trade names,commercial practices, or organizations imply endorsement bythe U.S. Government. This issue of Academic Emergency Medi-cine is funded by the Robert Wood Johnson Foundation.Supervising Editor: James Miner, MD.The authors have no relevant financial information or potentialconflicts of interest to disclose.Address for correspondence and reprints: Suzanne Mason,FRCS, FFAEM, MD; e-mail: s.mason@sheffield.ac.uk.

attendances up.4 Emergency services, while expandingthe role of allied health professionals, have seen juniordoctors’ hours of work reduce to a maximum of 48 perweek as a result of the European Working Time Direc-tive.5 This creates difficulties in staffing EDs consis-tently, leading to more challenging workingenvironments and difficulties in delivering high-qualityand timely patient care.

CHANGING WORKFORCE TO MANAGE DEMANDBETTER

As mentioned, a number of strategies in the UnitedKingdom were tried and have failed to manage the ris-ing demand for emergency care. The creation of alter-native services that patients could access for advice andmanagement of their urgent health care problemincluded National Health Service (NHS) Direct (a nurse-led 24-hour telephone helpline) aimed at providing tele-phone advice for patients, which it was hoped wouldincrease their ability to self-manage minor illness andinjury conditions and allow the direction of patients tothe most appropriate source of care. Studies haveshown that there was no discernable effect of NHSDirect on demand for ED or ambulance service care.4

Similarly, walk-in centers were set up to offset demandfor emergency care by giving patients an alternativeservice they could walk into for emergency care, but noevidence has been found to support this hypothesis.6 Inaddition, the general practitioners in the United Kingdomrenegotiated their contract and were given the optionto stop providing 24-hour care for their patients. Thiscreated a new gap in services in some parts of thecountry, where there were insufficient GPs to cover theemergency out-of-hours workload. Patients’ percep-tions of their GPs’ availability changed and their defaultposition was to attend the ED.7,8

Other emergency health care services have expandedthe remit of health care they provide to try to manageincreasing demand better and deliver care closer to thepatient. This is especially true of the nation’s ambulanceservices, which have seen that they can provide a cru-cial role in this respect, especially for the more vulnera-ble patients in society, such as the frail elderly.Extended roles for paramedics have evolved wherebypatients can be assessed in the home, treated wherepossible, or sign-posted to alternative sources ofcare.9,10 However, despite evidence showing the clearbenefits of this for patients, the spin-off benefits inreducing ED attendances, and some potential costsavings, these initiatives are only available in patchy

geographical areas of the United Kingdom and havenot been taken up in a more comprehensive fashion.

CHANGING PROCESSES OF CARE

A large study in 200411 tried to identify which organiza-tional factors had deleterious effects on the process ofcare that patients experienced and were therefore morelikely to lead to longer waiting times in U.K. EDs. Thisvery large study used mixed methods, recruited 137U.K. EDs for the study (representing 65% of U.K. EDs),and found that patients were likely to wait longer in theED if it had high visit numbers and a caseload skewedtoward more serious cases (accounting for 14% varia-tion in mean waiting times). Second, EDs that spentmore money on non–pay-related items, such as testsand investigations, had more staff sickness and a lessdemocratic clinical leader and had longer mean waitingtimes (accounting for an additional 33% of the variationin mean waiting times). Further in-depth qualitativework also demonstrated that waiting times worsened inEDs that lacked boundary spanning behavior, that is,where EDs failed to be proactive in developing workingrelationships with other agencies on the boundary(such as radiology, laboratories, community care, andthe acute ward setting), and thus enhance patient careacross the whole journey. In addition, where staffshowed higher levels of psychological strain and wherethere were higher levels of staff autonomy and control,this probably indicates a lack of departmental strategicdirection and team working. These factors clearly showthat the organization and leadership within an ED caninfluence factors that affect the level of crowding andpatient flow. Further research is needed around inter-ventions to identify the effect of organizational changeon crowding and patient care.

TARGETS IN THE UNITED KINGDOM

One other way of dealing with high levels of demandfor services and overcrowded EDs while ensuring effi-ciency and high quality is to performance manage aservice. For many years the U.K. ambulance service hashad time-based targets associated with the delivery ofcare, and this pressure has been brought to bear onEDs in England after several high-profile cases in themedia and also with health care policy initiatives by thegovernment. The government mandate for EDs basi-cally stated that ‘‘By 2005, 98% of all patients must bein and out of the ED within four hours.’’12 The 98%4-hour cutoff was not based on any good research evi-dence, and this target had not previously beendemanded anywhere else in the world. As a result ofthe introduction of the target in 2004, overall perfor-mance improved as shown in Figure 1. Although theUnited Kingdom was the first to set a throughput targetfor ED visits, New Zealand and parts of Australia andCanada are trialing a similar target for their EDpatients.13–15

To reach this target, EDs undertook a number ofmeasures that were not cost-neutral. These were docu-mented through a survey of English EDs in 2006, towhich 111 of 198 EDs responded (56%).16 The

Table 1Mean Annual Rise in ED Attendances by Country

Years CountryMean Annual

Rise (%)

1999–2006 Australia 5.31996–2006 United States 3.21996–1999 Switzerland 5.92002–2009 United Kingdom 5.9

ACADEMIC EMERGENCY MEDICINE • December 2011, Vol. 18, No. 12 • www.aemj.org 1235

commonest measures taken were additional seniordoctor hours (39% of EDs respondents), creation of a‘‘4-hour monitor’’ role (37%), improved access to emer-gency beds (36%), additional nonclinical staff hours(33%), additional nursing hours (29%), and triage bysenior staff (28%). However, in 32% of EDs responding,no changes were made to usual practice. The biggestinfluence on improved performance during monitoringweek was the number of measures that a departmenttook, rather than any specific measure.

The 4-hour target was designed to improve crowdingand patient outcomes by reducing the time patientsspend in EDs. No one denied that many patients spentlonger than needed in our EDs and often because of alack of an inpatient bed, not because they were receiv-ing essential treatment. However, it was not clearwhether putting a cap on time would improve patientoutcomes. To explore the effect that the 4-hour targetwas having on patients, data were analyzed to examinethe distribution of total time spent in the ED. Data wereavailable from 83 EDs for the month of April 2004(n = 428,953 patient episodes). Figure 2 shows the dis-tribution of time in the ED by disposition category ofadmitted and discharged. The median total time in theED for discharged patients was 96 minutes (98th per-centile = 341 minutes); 91.0% of these patients spentunder 220 minutes in the department, with a further3.6% spending 220 to 239 minutes.

Patients admitted from the ED had a median totaltime of 183 minutes (98th percentile = 625 minutes). Thedistribution of total time in the department for admittedpatients shows the most striking anomaly, with 64.0%of patients spending under 220 minutes in the depart-ment and a further 12.3% spending 220 to 239 minutes.Patients spending 220 to 239 minutes in the departmentwere also significantly older than those spending under220 minutes there.17

Further in-depth work with 15 EDs identified that the4-hour target did encourage change, and a number ofnew processes were introduced that were felt to bebeneficial to patient care. These included introducingmeasures such as streaming of patients according totheir acuity level on arrival, early ordering of laboratory

tests, increased senior doctor triage of patients, and theintroduction of clinical decision units where patientscould continue their investigations without being admit-ted to a hospital bed while also ‘‘stopping the clock.’’Key lessons from the implementation of the target werelearned,18 and these included emphasizing the responsi-bility of the whole organization, maintaining the focuson improving patient care, embracing opportunity, andinvolving all stakeholders.

A further study analyzed 12.2 million new patient epi-sodes at English EDs from Hospital Episode Statisticsdata for 2008 and 2009 to see whether the distributionof time in ED had changed. Figure 3 shows that the last20-minute ‘‘spike’’ in activity was still present and muchlarger than in 2004, with 30.7% of admitted patientsleaving the ED in the 20 minutes before the 4-hour tar-get is breached and 10.5% of discharged patients.

While many in the U.K. specialty of emergency medi-cine supported the benefits that the 4-hour target pro-duced, it was clear that they were not being

70

75

80

85

90

95

100

2002/3 2003/4 2004/5 2005/6 2006/7 2007/8 2008/9

%seen

within4ho

urs

year

Figure 1. England’s 4-hour ED performance, 2002–2008.

0

1

2

3

4

5

6

7

8

0-9

19-2

9

40-4

9

60-6

9

80-8

9

100-

109

120-

129

140-

149

160-

169

180-

179

200-

209

220-

229

240-

249

260-

269

280-

289

Total time in department (mins)

Perc

enta

ge o

f atte

ndan

ces

DischargedAdmitted

Figure 2. Distribution of total time in ED for episodes resultingin admission or discharge, 2004.

0

5

10

15

20

25

30

35

0-20

21-4

0

41-6

0

61-8

0

81-1

00

101-

120

121-

140

141-

160

161-

180

181-

200

201-

220

221-

240

241-

260

261-

280

281-

300

301-

320

321-

340

341-

360

perc

enta

ge o

f pat

ient

s (%

)

time in emergency department (mins)

admitted

discharged

Figure 3. Distribution of patient time in ED by admitted anddischarged in 2008–2009.

1236 Mason • U.K. EXPERIENCES OF EVALUATING PERFORMANCE AND QUALITY IN EM

experienced by all patients and that processes through-out the hospital and wider health care system may nothave improved sufficiently to accommodate it.19 Targetsand performance measures in health care are proliferat-ing worldwide as a means to improve the quality andvalue of care delivered to patients.20–23 However, theydo not always hit their mark in improving patient careand can have unintended consequences.24–34 To date,there is no evidence that the 4-hour target benefittedpatient care, and indeed the findings would suggestthat it has encouraged target-led care rather thanneeds-led care.

While it is clear that the 4-hour target has resulted infewer patients spending many hours in the ED, the ris-ing proportion of patients, particularly admissions, whohave dispositions in the last 20 minutes of the 4-hourinterval strongly suggests that a stringent absolute cut-off may not be the best way to manage ED crowding.The introduction of a stringent cutoff as a target forEDs is a matter of some controversy, and indeed, someinstitutions have warned against this practice, high-lighting that as soon as the target is breached, theincentive is lost, and that crowding near to the cutofftime appears.35

MEASURING QUALITY IN PATIENT OUTCOMES

The ‘‘targets and terror’’ regimes that have emergedover the past 20 years in the U.K. health service havefocused on immediate, easily measurable process inter-ventions to provide quantifiable evidence of perfor-mance. This has forced organizations to divert theirprofessional, largely value-driven staff away from‘‘doing the right thing’’ toward achieving externallyimposed goals instead. The Department of Health hasrecently announced a relaxation of the 98% 4-hourstandard, along with the introduction of a dashboard ofclinical quality indicators.36 It is likely that additionalquality and safety measures will be required to ensurean optimum balance of safety, quality, and timeliness inemergency care.

Measuring outcomes still remains a huge challengefor emergency medicine. This is because outcomes forpatients are based on the whole journey, from callingfor help to leaving the health care system. This mayinvolve a number of services and health care staff, andto focus attention on one part of that journey, the ED,does not necessarily mean that overall care, and there-fore outcomes, is improved. Having reliable data thatwe can interrogate and manipulate from our EDs isprobably the single biggest challenge. Whatever wechoose as our outcome or process measures, we needto be able to record and benchmark our activity to mea-sure current activity and inform future improvements.The U.K. experiences with the 4-hour target have led toa better understanding of the challenges that measuringperformance can present.

Finally, what outcomes should we measure? We haveseen a demonstration of some of the difficulties anddangers associated with having a target or indicator toperform to. The forthcoming challenges for the UnitedKingdom presented by the new quality indicators pre-sents us with several more opportunities. Change needs

to lead to the adoption of a culture that leads to excel-lent care for all patients, and this should be the casewhether our EDs are crowded or not.

References

1. Cameron PA, Schull MJ, Cooke MW. A frameworkfor measuring quality in the emergency department.Emerg Med J. 2011; 28:735–40.

2. Victor CR, Peacock JL, Chazot C. Who calls 999 andwhy? A survey of the emergency workload of theLondon Ambulance Service. Accid Emerg Med.1999; 16:174–8.

3. Pennycook AG, Makower RM, Morrison WG. Useof the emergency ambulance service to an inner cityaccident and emergency department–a comparisonof general practitioner and ‘999’ calls. J R Soc Med.1991; 84:726–7.

4. Munro J, Nicholl J, O’Cathain A, Knowles E. NHSDirect; Impact of NHS Direct on demand for immedi-ate care: observational study. BMJ. 2000; 32:150–3.

5. The National Health Service. About Modernising Med-ical Careers. Available at: http://www.mmc.nhs.uk/medical_education/about_modernising_medical_care.aspx. Accessed Sep 6, 2011.

6. Coleman P, Irons R, Nicholl J. Will alternativeimmediate care services reduce demands for non-urgent treatment at accident and emergency?Emerg Med J. 2001; 18:482–7.

7. Penson R, Coleman P, Mason S, Nicholl J. Why dopatients with minor or moderate conditions thatcould be managed in other settings attend theemergency department? Emerg Med J. 2011;doi:10.1136/emj.2011.107276.

8. Thompson C, Hayhurst C, Boyle A. How havechanges to out-of-hours primary care services since2004 affected emergency department attendances ata UK District General Hospital? A longitudinalstudy. Emerg Med J. 2010; 27:22–5.

9. Mason S, Knowles E, Colwell B, et al. Paramedicpractitioner older people’s support trial (PPOPS): acluster randomised controlled trial. Br Med J. 2007;335:919.

10. Mason S, O’Keeffe C, Knowles E, et al. A pragmaticquasi-experimental multi-site community interven-tion trial (NEECaP) evaluating the impact of emer-gency care practitioners in different UK healthsettings on patient pathways. Emerg Med J. Inpress (October 2010).

11. Mason S, Locker T, Carter A, Walters S, Stride C.What Are the Organisational Factors That InfluenceWaiting Times in Emergency Department? For theNHS Service Delivery and Organisation R&DProgramme. Available at: http://www.sdo.nihr.ac.uk/files/project/49-final-report.pdf. Accessed Sep 6,2011.

12. Department of Health. The NHS plan. A Plan forInvestment. A Plan for Reform. London: StationeryOffice, National Health Service, 2000.

13. Ontario Ministry of Health and Long-Term Care.Emergency Room Targets. Available at: http://www.health.gov.on.ca/en/pro/programs/waittimes/edrs/targets.aspx. Accessed Sep 7, 2011.

ACADEMIC EMERGENCY MEDICINE • December 2011, Vol. 18, No. 12 • www.aemj.org 1237

14. Australian Government. A national health and hospi-tals network: further investments in Australia’s health.Available at: http://www.yourhealth.gov.au/internet/yourhealth/publishing.nsf/Content/nhhn2-report2-toc. Accessed Sep 7, 2011.

15. New Zealand Ministry of Health. Shorter Stays inED Health Target. November 2009. Available at:http://www.moh.govt.nz/moh.nsf/indexmh/ed-target.Accessed Sep 7, 2011.

16. Munro J, Mason S, Nicholl J. Effectiveness of mea-sures to reduce emergency department waitingtimes: a natural experiment. Emerg Med J. 2006;23:35–9.

17. Locker T, Mason S. Analysis of the distribution oftime that patients spend in emergency departments.Br Med J. 2005; 330:1188–9.

18. Weber EJ, Mason S, Hew R, Carter A. Emptyingthe corridors of shame: organizational lessons fromEngland’s four-hour emergency throughput target.Ann Emerg Med. 2011; 57:79–88.

19. Mason S, Nicholl J, Locker T. Targets still lead carein emergency departments [letter]. BMJ. 2010;341:c3579.

20. Willcox S, Seddon M, Dunn S, et al. Measuring andreducing waiting times: a cross-national compari-son of strategies. Health Aff (Millwood). 2007;26:1078–87.

21. Pink GH, Brown AD, Studer ML, et al. Pay-for-performance in publicly financed healthcare: someinternational experience and considerations forCanada. Healthc Pap. 2006; 6:8–26.

22. Lilford RJ, Brown CA, Nicholl J. Use of processmeasures to monitor the quality of clinical practice.BMJ. 2007; 335:648–50.

23. Kerr EA, Fleming B. Making performance indica-tors work: experiences of US veteran’s healthadministration. BMJ. 2007; 335:971–3.

24. Bevan G. Have targets done more harm than goodin the English NHS? No. BMJ. 2009; 338:a3129.

25. Gubb J. Have targets done more harm than good inthe English NHS? Yes. BMJ. 2009; 338:a3130.

26. Kelman S, Friedman JN. Performance improvementand performance dysfunction: an empirical exami-nation of distortionary impacts of the emergencyroom wait-time target in the English National

Health Service. J Public Adm Res Theory. 2009;19:917–46.

27. Locker T, Mason S, Wardrope J, et al. Targets andmoving goal posts: changes in waiting times in aUK emergency department. Emerg Med J. 2005;22:710–4.

28. Locker TE, Mason SM. Are these emergencydepartment performance data real? Emerg Med J.2006; 23:558–9.

29. Wachter RM, Flanders SA, Fee C, et al. Publicreporting of antibiotic timing in patients with pneu-monia: lessons from a flawed performance mea-sure. Ann Intern Med. 2008; 149:29–32.

30. Appleby J, Boyle S, Devlin N, et al. Do English NHSwaiting time targets distort treatment priorities inorthopaedic surgery? J Health Serv Res Policy.2005; 10:167–72.

31. Propper C, Sutton M, Whithall C, Windmeijer F.Incentives and targets in hospital care: evidencefrom a natural experiment. J Pub Econ. 2010;94:318–35.

32. Pink GH, Brown AD, Studer ML, et al. Pay-for-performance in publicly financed healthcare: someinternational experience and considerations forCanada. Healthc Pap. 2006; 6:8–26.

33. Lilford RJ, Brown CA, Nicholl J. Use of processmeasures to monitor the quality of clinical practice.BMJ. 2007; 335:648–50.

34. Propper C, Sutton M, Whitnall C, Windmeijer F. Did‘‘Targets and Terror’’ Reduce Waiting Times in Eng-land for Hospital Care? Available at: http://ideas.repec.org/p/bri/cmpowp/07–179.html. Accessed Sep 7,2011.

35. Royal Statistical Society. Royal Statistical SocietyWorking Party on Performance Monitoring in thePublic Services. Performance Indicators. Good, Badand Ugly. Available at: http://membership.rss.org.uk/pdf/PerformanceMonitoringReport.pdf. AccessedSep 7, 2011.

36. Department of Health. A&E Quality Indicators: DataDefinitions. London 2010. Available at: http://www.dh.gov.uk/prod_consum_dh/groups/dh_digitalassets/@dh/@en/@ps/documents/digitalasset/dh_122892.pdf.Accessed Sep 7, 2011.

1238 Mason • U.K. EXPERIENCES OF EVALUATING PERFORMANCE AND QUALITY IN EM