From information to improvement: evidence from UK & US

27 April 2010Gwyn Bevan

Information improvement Three pathways &

evidence from US Change Selection Reputation

Evidence from UK Questions

Selection & Change

ResultsAware of differences

Knowledge about performance

Consumers able to choose

Knowledge about process & results

Management &Professionals Implement change

Motivation

Adapted from Berwick et al. 2003

Evidence from two systematic reviews Limited evidence

Industry of performance assessment Rigorous evalution

Cardiac Surgery Reporting System (CSRS) 6 other US systems

Change Weak evidence

Selection No evidence providers respond to threat

of patients using information as consumers

Ineffective in US likely to work in UK?Sources: Marshall et al (2000) & Fung et al (2008)

CSRS (2001): CABG outliers good & poor outcomes

Source: http://www.health.state.ny.us/statistics/diseases/cardiovascular/

RAMR Rates

The Clinton’s selection pathway (2001 CSRS data available)

sudden onset of chest pains & shortness of breath (September 2004) small hospital near home cardiologists @

Westchester Medical Center

quadruple bypass @ Columbia-Presbyterian

Source:http://www.nytimes.com/2004/09/06/health/06hosp.html?scp=1&sq=Clinton%20Surgery%20Puts%20Attention%20on%20Death%20Rate&st=cse

CSRS (2001): CABG outliers good & poor outcomes

Source: http://www.health.state.ny.us/statistics/diseases/cardiovascular/

RAMR Rates

Columbia Presbyterian

Westchester Medical Center

CSRS: pathways Change

mediocre / below-average performance: failed to use rich performance data

Selection outliers with good / poor performance: no

changes in market share Reputation

outliers with poor performance: ‘naming & shaming’ galvanised to improve

Source: Chassin (2002)

3rd Pathway: Reputation Characteristics

i. rankingii. published: widely

disseminatediii. easily understood:

performance good or poor?

iv. future report: performance improved?

Paradox information not

used by patients to switch from poor to good hospitals

managers of poor hospitals respond to repair perceived damage to public reputation not market share

Sources: Hibbard et al. (2003, 2005)

Controlled experiment in Wisconsin

Change Selection Reputation

public-report

private-report

no-report

Summary indices adverse events Deaths & complications) General: surgery / nonsurgery Specific: cardiac, maternity, & hip/knee

System characteristics Ranking, easily understood, followed up

Source: Hibbard et al. (2003, 2005)

Wisconsin: pathways Change: Private report

little effort to improve quality Selection: Public report

no anticipated changes in market share no actual changes in market share

Reputation: Public report significantly greater efforts to improve

quality because of concerns over reputational damage

Source: Hibbard et al. (2003, 2005)

Information improvement Three pathways & evidence from the

US Evidence from UK

Changing policy mix in England Natural experiment Evidence

Questions

Changing policy mix in England 1997-99: Selection Change

‘internal market’ ‘third way’ 2000-02: Reputation

star ratings (2001 – 05) 2002 - : Reputation & Selection

Reputation: star ratings healthcheck (2006)

Selection: FTs & ISTCs, PbR, patient choice, & World Class Commissioning

UK’s natural experimentChange Reputation Selection

England•1991-97•1997-99•2000-02•2002 -

Devolved countries•1999 -

Reward failure

Abolish p/p split

England: Selection Change1997 manifesto pledge waiting listsNumbers waiting elective admissions (England) (‘000s)

Star rating: Reputation ‘naming & shaming’ zero stars

‘devastating …hit right down to the workforce – whereas bad reports usually hit senior management upwards …nurses demanding changing rooms .. because being accosted in streets’

Source: Mannion et al (2005)

2001:the dirty dozen

England: Change ReputationNumbers waiting elective admissions (England) (‘000s)

Star ratings published

Funding of UK’s natural experiment

Natural experiment: Change & Reputation

No/‘000 waiting > 6 months for elective hospital admission

Natural experiment: Change & Reputation

Source: Connolly et al (2010) Funding and Performance of Healthcare Systems in the Four Countries of the UK before and after Devolution. The Nuffield Trust.

% waiting list < 13 weeks for hospital admission (March 2008)

Natural experiment: Change & Reputation

No/‘000 waiting > 3 months GP 1st outpatient appointment

England: Change Reputation

Change Reputation

% Ambulance response times to life-threatening emergencies < 8 minutes (Target 75%)

Source: Bevan & Hamblin (2009)

Natural experiment: Change & Reputation

Star ratings published

Target

Source: Bevan & Hamblin (2009)

% Ambulance response times to life-threatening emergencies < 8 minutes

Barber (2007) Instruction to Deliver Awful adequate

Command & control public not satisfied have to keep flogging the system

Adequate good / great quasi market & consumer choice

innovation from self-sustaining systems

The impact of the NHS market: An overview of the literature* No good evidence reforms produced

beneficial outcomes classical economic theory predicts of markets provider responsiveness to patients &

purchasers large-scale cost reduction innovation in service provision

NHS incurs transaction costs of market without benefits

* Brereton & Vasoodaven (2010) http://www.civitas.org.uk/nhs/download/Civitas_LiteratureReview_NHS_market_Feb10.pdf

System reform package & commmissioning

No

Questions Change

Devolution worse performance in devolved countries?

Selection Purchaser / provider split? ‘World Class Commissioning’? Patient choice?

Reputation Executive democratic accountabilty?

References Audit Commission & Healthcare Commission. Is the

treatment is working? <http://www.audit-commission.gov.uk/reports/NATIONAL-REPORT.asp?CategoryID=&ProdID=9F8B7F6A-214D-4165-BE65-716315270A82&fromREPORTSANDDATA=NATIONAL-REPORT>

Auditor General for Wales (2005) NHS waiting times in Wales. Volume 2 - Tackling the problem. <http://www.wao.gov.uk/reportsandpublications/2005.asp>

Bevan (2006) Setting Targets for Health Care Performance: lessons from a case study of the English NHS. National Institute Economic Review, 197, 67-79.

References Bevan (2010) Approaches and impacts of different systems of

assessing hospital performance. Journal of Comparative Policy Analysis, 2010, 12(1 & 2): 33 – 56

Bevan & Hamblin (2009) Hitting and missing targets by ambulance services for emergency calls: impacts of different systems of performance measurement within the UK. Journal of the Royal Statistical Society (A), 172(1):1-30.

Bevan & Hood (2006) Have targets improved performance in the English NHS? British Medical Journal, 332, 419-422.

Bevan & Hood (2006) What’s Measured is What Matters: Targets and Gaming in the English Public Health Care System. Public Administration 84, 3, 517-38.

Chassin (2002) Achieving and sustaining improved quality: Lessons from New York State and cardiac surgery. Health Affairs, 21(4), 40-51.

References Fung et al (2008) Systematic review: The evidence that

publishing patient care performance data improves quality of care. Annals of Internal Medicine, 148, 111-123.

Hibbard (2008) What can we say about the impact of public reporting? Inconsistent execution yields variable results. Annals of Internal Medicine, 148, 160 - 161.

Hibbard et al (2003) Does publicizing hospital performance stimulate quality improvement efforts? Health Affairs, 22 (2), 84-94.

Hibbard et al (2005) Hospital performance reports: impact on quality, market share, and reputation. Health Affairs, 24(4), 1150-60.

References House of Commons Health Committee (2010)

Commissioning. (Fourth Report of Session 2009–10, Volume I) HC 268-I. London: The Stationery Office Limited. < http://www.parliament.the-stationery-office.co.uk/pa/cm200910/cmselect/cmhealth/268/26802.htm>

Mannion et al (2005) Impact of star performance ratings in English acute hospital organisations. Journal of Health Services Research & Policy, 10 (1), 18–24.

Mannion & Goddard (2001) Impact of published clinical outcomes data: case study in NHS hospital organisations. BMJ, 323, 260 – 263.

References Mannion & Goddard (2003) Public disclosure of

comparative clinical performance data: lessons from the Scottish experience. Journal of Evaluation in Clinical Practice, 9 (2), 277–286

Marshall et al (2000) The Public release of performance data: What do we expect to gain? A review of the evidence. Journal of the American Medical Association, 283(14), 1866-1874.