Measuring health system performance: Problems and Opportunities in the Era of Assessment and Accountability

C. David Naylor*,Karey Iron† and Kiren Handa†

*Faculty of Medicine, University of Torontoand

† Institute for Clinical Evaluative Sciences

Health Care Reform: No Magic BulletsHealth Care Reform: No Magic Bullets

• Incrementalism rules

• Bottom up : improvements in services, measurement–driven

• Top down : policy adjustments, experience/evidence-based

The Health Care Decision-Making TriadThe Health Care Decision-Making Triad

EVIDENCEEVIDENCEEVIDENCE

VALUES or CULTUREVALUES or CULTUREVALUES or CULTURE

CIRCUMSTANCES or CONTEXTCIRCUMSTANCES or CONTEXTCIRCUMSTANCES or CONTEXT

Some Drivers of the MeasurementSome Drivers of the MeasurementMovementMovement

• Fiscal/Management Imperatives• Public Expectations• Ubiquity of Variations and Errors• Rising Stakes: Post-Genomic Technology

Average life expectancy at birth in 13 OECDAverage life expectancy at birth in 13 OECDcountries, 1960-96countries, 1960-96

60

65

70

75

80

85

19601964

19681972

19761980

19841988

19921996

Life

exp

ecta

ncy

at b

irth

Males

Females

WHO 00193

Hurst J. Bulletin of the World Health Organization, 2000, 78 (6)

EXAM QUESTION:EXAM QUESTION:

Health care has only modest effects onHealth care has only modest effects onpopulation health status.population health status.

Should measures of health systemShould measures of health systemperformance include population healthperformance include population healthIndicators as outcomes or as confounders?Indicators as outcomes or as confounders?

PLEASE ANSWER IN 200 WORDS OR LESS…PLEASE ANSWER IN 200 WORDS OR LESS…

Population Health: Major DeterminantsPopulation Health: Major Determinants

Health StatusHealth StatusHealth Status

Health ServicesSystem

Health ServicesHealth ServicesSystemSystem

EnvironmentalEnvironmentalEnvironmental

LifestyleLifestyleLifestyle

GeneticEndowment

GeneticGeneticEndowmentEndowment

Productivity& Wealth

ProductivityProductivity& Wealth& WealthSocioeconomicSocioeconomicSocioeconomic

Measurement Priorities for ProgramsMeasurement Priorities for Programsand Interventionsand Interventions

• Strong evidence for impacts on healthstatus

• High Costs or high risks

• High variation

AMI Patients across income quintileAMI Patients across income quintilein Ontario, Canadain Ontario, Canada

1 2 3 4 5Neighborhood median income quintile

base

line

fact

ors

Nagemale

P < 0.001

Male = 63.1%

Source: Alter et al. 1999

0.75

0.8

0.85

0.9

0.95

1

0 91 182 273 364

Time from Acute Myocardial Infarction (days)

Prop

ortio

n of

pat

ient

s su

rviv

ing

0.75

0.8

0.85

0.9

0.95

1

0 91 182 273 364

Time from Acute Myocardial Infarction (days)

Prop

ortio

n of

pat

ient

s su

rviv

ing

LowestLowestquintilequintile

HighestHighestquintilequintile

Log-rank test; p<0.001Log-rank test; p<0.001

Survival post-AMI by SESSurvival post-AMI by SES

Source: Alter et al. 1999

Accept complexity

Make selective use of composite measures

Different measures to inform differentstakeholders

�

�

��

��

�

92

94

9593

96

97

98

Number alive

Num

ber d

i ed

Example of Shewhart’s control chart to describe process variation

Mortality ofwomen aged 65 + in2 UK towns,1992-98

� Mortality by year ofDr. Harold Shipman’swomen patients

Source: Mohammed MA et al. Lancet 2001

Statistical Significance versusStatistical Significance versusPolicy RelevancePolicy Relevance

• Population health impact

• Budgetary impact

• ‘Identifiable victims’

• Font-size of headlines!

Focus on assembling accurate,reliable, and relevant data

Talk with different audiencesabout their data needs and

concerns before(re-) designing information

systems

Standardize and fill the ‘black holes’ inStandardize and fill the ‘black holes’ inhealth info-structurehealth info-structure

• Standardize definitions of inputs & outputs

• Better characterization of providers, patients,processes, and outcomes

• Address ‘black holes’: I.e. waiting lists, primary/ambulatory care, long-term and rehabservices

Supplement existing data on aproject-specific and time-limitedbasis with clear analytical goals.

Acquire data to follow patients throughepisodes of illness and

across the care continuum

Integrated data can help reducedis-integration of health services

Take advantage of technology andupgrade info-structure

as necessary

HOW…to achieve IMPROVEMENTHOW…to achieve IMPROVEMENT

ALIGN:• Information systems• Performance measures• Reporting systems• Organizational/Professional culture• Implementation mechanisms

0%

20%

40%

60%

80%

100%

Less than 10 minutes Longer than 20 minutes

Australia Canada New Zealand United Kingdom United States

Length of Most Recent Doctor Visit

43%33%

43%

65%

30%

13%23%

18% 15%

33%

1998 Commonwealth Fund International Policy Health Survey

Respondents reporting time theirdoctor spent with them was too short

23%

14%

14%

15%

13%

0% 20% 40% 60% 80% 100%

United States

UnitedKingdom

New Zealand

Canada

Australia

23%

14%

14%

15%

13%

0% 20% 40% 60% 80% 100%

United States

UnitedKingdom

New Zealand

Canada

Australia

1998 Commonwealth Fund International Policy Health Survey

0

5

10

15

20

Limited Moderate Severe

Threat to life from Coronary Blockages

Age 20-64Age 65-74Age ≥≥≥≥ 75

Rel

ativ

e C

AB

G ra

te (N

Y to

ON

)Coronary surgery: OntarioCoronary surgery: Ontario vs vs New York New York

7.3

10.8

16.8

0.8 1.22.2 2.0 2.5

4.5

Percentages of Senior Citizens receivingCardiac Procedures after Heart Attacks in

the United States and Ontario, Canada,1991

* The coronary angiography rates do not include procedures performedon an outpatient basis in either country

ProcedureUnited States

Ontario, Canada Relative Rate

United States

Ontario, Canada Relative Rate

Coronary angiography* 35 6 5 37 10 4PTCA 12 1 8 13 3 5CABG 10 1 8 12 3 4CABG + PTCA 22 3 8 25 6 4

ProcedureUnited States

Ontario, Canada Relative Rate

United States

Ontario, Canada Relative Rate

Coronary angiography* 35 6 5 37 10 4PTCA 12 1 8 13 3 5CABG 10 1 8 12 3 4CABG + PTCA 22 3 8 25 6 4

Tu et al. 1997. NEJM 336 (21):1500-1505

EQUOL GUSTOEQUOL GUSTO

Domains AssessedQuality of LifeFunctional Status

Employment Status/ role functioningSymptoms (chest pain, dyspnea)

Psychological well-beingTime trade-off (QALY)General Health Status

Economics/Resource ConsumptionHospitalizations

Cardiac catheterizationRevascularizationOutpatient care

Source: Mark et al NEJM 1994;331:1130-5

Diminishing Marginal Returnsof Health Care

Resources for Health Care

Hea

lth S

tatu

s

AA

BB

CC

Rating of zerostars for a dozenBritish hospitals

Medical PostOctober 16, 2001

“There has always been a convenient excuse when there is aproblem, but these tables explode the myth once and for all.”

Alan MilburnHealth Secretary

“I worry that strident reporting of a hospital’s weaknesses candent the confidence of the community in its local hospital,adversely affect recruitment and further damage the morale ofclinical staff.”

Peter HawkerChairman, BMA Consultants’ Committee

Medical Post, October 16, 2001

HOW…to achieve IMPROVEMENTHOW…to achieve IMPROVEMENT

ALIGN:• Information systems• Performance measures• Reporting systems• Organizational/Professional culture• Implementation mechanisms

Regulatory Admin/Professional

Market-based

EconomicNon-economic

IncentivesIncentives

MechanismsMechanisms

ActorsActorsConsumersPurchasers/FundersProfessionals/Managers

Measurement and ManagementAssessment and AccountabilityOnwards and Upwards!