from measurement and reporting to actual improvement and … · 2016-03-24 · the balanced...

KEY PERFORMANCE INDICATORS

IN HOSPITALS

From Measurement and Reporting to Actual Improvement and Action

DAY 2

Fadi El-Jardali, MPH, PhD.

March 2016

DEVELOPMENT AND

APPLICATIONS OF

BALANCED SCORECARD

SYSTEM IN HOSPITALS

The Balanced Scorecard Concept

The Balanced Scorecard

• Tracks indicators across different perspectives

• Provides a balanced view of performance

• Guides strategic decisions at both the provider- and health system- levels

• Tells a brief yet comprehensive story about the achievement and

performance of the organization toward goals.

Rationale

• No single aspect of the organization/ system causes poor or excellent

performance

• For this reason, performance-measurement activities must include measures

that provide insights into multiple dimensions of performance

El-Jardali F, Saleh S, Ataya N, Jamal D. Design, implementation and scaling up of the balanced scorecard for hospitals in Lebanon: policy coherence and application lessons for low and middle income countries. Health Policy. 2011 Dec;103(2-3):305-14. Zelman WN, Pink GH, Matthias CB. Use of the balanced scorecard in 686 health care. Journal of Health Care Finance 2003;29:1–16. Pink GH, McKillop I, Schraa EG, Preyra C, Montgomery C, Baker GR. 668 Creating a balanced scorecard for a hospital system. Journal of Health 669 Care Finance 2001;27:1–20.

What’s the difference between Dashboards

and the Balanced Scorecard?

http://www.bscdesigner.com/dashboard-vs-balanced-scorecard.htm

Dashboard Balanced Scorecard

Used for Performance measurement /

monitoring

Performance management and

progress

Focus Operational (short-term)

goals

Strategic (long-term) goals

It helps Visualize the performance to

understand the current state

Align KPI, objectives, and

actions to see the connection

between them

Example Automobile dashboard

(shows how your car is

operating)

GPS (shows when and how

you will arrive?)

International Applications of the BSC to Healthcare Sector

Ontario Acute Care

Hospitals

Hospital Flexibility Tracking

Project

National Health Service Trust

Hospitals

Participation Voluntary

Acute care hospitals in

Canada

Voluntary

Critical access hospitals in the

US

Compulsory

All NHS Trust hospitals

Primary

Audience

Providers & Consumers Providers & Policymakers Political and social agenda of

labour government

Purpose • Increase awareness of

performance variation

• Stimulate improvement

• Ensure accountability

• Improve quality of data

• Inform policymakers • Provide a balanced view of

NHS performance

Data

reported

• In Aggregate

• For Individual Hospitals

• In Aggregate • In Aggregate

• For Individual Hospitals

Domains • Financial Performance

Condition

• Patient Satisfaction

• Clinical Utilization and

Outcomes

• System Integration and

Change

• Financial

• Customer

• Process

• Infrastructure,

Governance, and

Community Relations

• Health improvement

• Fair access

• Effective delivery of

appropriate care

• Efficiency

• Patient experience

• Health outcomes of NHS care

Data 38 indicators for 89

organizations

47 indicators for 538 hospitals 36 indicators for All NHS Trust

hospitals

• First application of the BSC

in a developing country

• The BSC was used to

regularly monitor the

progress of the strategy of

the Ministry of Public Health

in Afghanistan to deliver a

basic package of health

services

• Facilities included: basic

health centres,

comprehensive health

centres, and the outpatient

and maternity wings at

district hospitals

• 29 core indicators and

benchmarks representing six

different domains of health

services.

Peters DH, Noor AA, Singh LP, Kakar FK, Hansen PM, Burnham G. A 696 balanced scorecard for health services in Afghanistan. Bulletin of the 697 World Health Organization 2007;85:146–51.

Ontario Hospital Association. Hospital Report 2007.

BSC used to

present

results for the

Ontario

Hospital

Association

Lessons learned

Lessons learned from international applications of the BSC

• The BSC allows comparability across organizations and the

relevant performance comparison is healthcare sector

benchmarks (which may be averages of the values for the

included organizations)

• Indicators have to be relevant for a large number of providers and

thus tend to be general, reflecting common services and

programs

• Reporting data in a standardized way from multiple organizations

presents data quality challenges

• The results from a BSC may serve as input for policy formulation

or change

Focus on Kuwait

• MOH reports indicators back to providers in a “dashboard”

specific to each Clinical Area.

Example: Appendectomy Indicators, Hospital X

Current Results

Range in 6 MOH Hospitals (Min- Max)

National Result

Compared to National Performance

Less than 12-Hours Interval between Admission & Appendectomy (H)

85% 4 - 90% 84%

Normal Pathology (L)

20% 4- 30% 9.8%

Perforated Appendicitis (L)

6% 4- 30% 13.9%

Missed Appendicitis (L)

3% 0- 14% 4.1%

Antibiotic Prophylaxis (H)

40% 14- 50% 47.1%

Post Appendectomy Complications (L)

3% 0- 11% 4.5%

Focus on Kuwait: Challenges

• Better reporting on a set of balanced indicators is needed to

provide “overall picture” on performance of healthcare

organization/ health system Need for a Balanced Scorecard

• Limited comparisons to international/ regional benchmarks or

targets

• Limited information on context to better understand

performance of healthcare providers

CONCEPT AND USE OF

BENCHMARKS IN

MONITORING

PERFORMANCE

Benchmarks: Definition and Sources

What is a Benchmark?

• An externally-agreed comparator to compare performance between

similar organisations or systems.

• International comparisons can help identify health system

performance issues and determine the extent to which other

countries also experience these.

• Benchmarking is an important tool to motivate improvement and

help healthcare providers understand where their performance falls in

comparison to others.

• Benchmarking can stimulate healthy competition.

Setting references for benchmarking

How high to set the bar?

• The identification of international, regional, or national reference

points is a critical exercise.

• It is helpful to consider which countries (or regions) might offer the

most relevant experience rather then focusing exclusively on the

countries with the greatest similarity to one’s own.

• Combining elements from the different reference groups is the

best way to make a sound comparison and, more importantly, to

identify opportunities for improving the health system.

Benchmarks: Definition and Sources

How to use benchmarks?

• Search for updated International Benchmarks in the references of the

Procedures Manuals and in accessible data that is clearly defined, regularly

collected and publicly reported on International databases (e.g., CIHI,

AHRQ, CDC, ACHS)

• Make sure measurement protocol (i.e., Formula/ inclusion & exclusion criteria,

reporting periods, risk adjustments, etc.) is consistent with protocol in Kuwait.

• Take note of these differences in measurement protocol in the report.

Differences may be expected since measurement protocol may have been

refined to the context of Kuwait.

• Compare averages to benchmarks. Benchmarking should act as a signal for

improvement.

• Compare results with targets, once targets are assigned.

SAMPLE BALANCED SCORECARD

Briefing Note Health Service Performance Agreements (HSPAs) Qatar 9

! Indicators Results National

Average

Benchmark Benchmark

source

Q1 Q2 Q3

4 Health

Workforce Turnover rate- PHC 0.4620

0.3721

0.6822

0.50 15.60

US Kansas Hospital

Association (2013)

"

5 Satisfaction /

Experience Patients and Staff

Patient/ customer experience 98.2023 93.37

24 87.6025 93.06 i. 80.81 Canada (2005) "

Staff satisfaction—PHC Annual- will be reported

after Q4

NHS England

(2013)

!

!

! !

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!20

Qatar Armed Forces and Qatar Red Crescent did not submit data. Qatar Petroleum Medical Services reported 8 terminations and no denominator, as such it was not included in Q1. 21

Qatar Petroleum Medical Services reported 6 terminations and Qatar Red Crescent Industrial reported 3 terminations and no denominator, as such it was not included in Q2. 22

Qatar Armed Forces and Qatar Red Crescent did not submit data. Qatar Petroleum Medical Services reported 1 termination and no denominator, as such it was not included in Q1. 23

ASPETAR and Al Emadi Hospital submitted data. There are concerns about data submitted from Al Emadi Hospital, Interpret results with caution. 24

ASPETAR, Doha Clinic Hospital, and Al Emadi Hospital submitted data. There are concerns about data submitted from Al Emadi Hospital, Interpret results with caution. 25

ASPETAR, Doha Clinic Hospital, and Al Emadi Hospital submitted data.


Indicators Results National

Average

Benchmark Benchmark

source

Q1 Q2 Q3

6 Performance and

Efficiency Percent Medication

reconciliation at admission 96.60

26 66.41

27 88.80

28 83.93 60

CIHI Canada

National compliance rates/

Accreditation Canada (2011)

"!

Blood culture contamination rate (%)29

0.6630

0.54 0.58 0.59 2.89 US CAP Q-Tracks study (1999-2003) "!

! !

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!26

ASPETAR and Al Emadi Hospital submitted data. Doha Clinic Hospital may have misinterpreted one of the criteria for this quarter, which could have affected overall results. 27

ASPETAR, Doha Clinic Hospital and Al Emadi Hospital submitted data. 28

ASPETAR, Doha Clinic Hospital and Al Emadi Hospital submitted data. 29

Not Applicable to American Hospital. 30

Doha Clinic Hospital did not submit data. Not Applicable to American Hospital.


Indicators Results National Average

Benchmark Benchmark source

Q1 Q2 Q3

2 Access and Responsiveness

Waiting time in clinic/ outpatient

department11

84.63 86.21 77.60 82.81 Internal Internal !

Percent Referrals to hospitals 1.8312

1.4913

1.2514

1.52 Internal Internal !

3 Safety

Patients and Staff

Percent compliance with hand

hygiene

Annual- will be reported

after Q4

Australia National Data (2014)

Blood and body fluid staff

exposure rate (%) 0.10

15 0.21 0.25

16 0.19% Internal Internal !

Medication errors

7.7517

; 0

caused harm

1.518

; 0

caused harm

3019

; 0

caused harm

13.08 Internal Internal !

!! !

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!11

% Patients seen within 30 minutes of their appointment time in clinic/ outpatient department in Hospitals. Interpret with caution as major corrections to data required. Not applicable to Doha Clinic Hospital. Only ASPETAR submitted data for Q1, Q2, and Q3 12

Ministry of Interior and QRC Al- Mamoura and Al Wafedeen Freej Abdel Aziz submitted data. Qatar Armed Forces reported 36 referrals and no denominator, as such it was not included in Q1. 13

Ministry of Interior, QRC Zikrit and Al Wafedeen Freej Abdel Aziz, Qatar Armed Forces, and Primary Healthcare Corporation submitted data. Qatar Petroleum Medical Services, reported 2,728 referrals and no denominator, as

such it was not included in Q2. 14

Ministry of Interior, QRC Al Mamoura, Indutrial and Al Wafedeen Freej Abdel Aziz, Qatar Armed Forces, and Primary Healthcare Corporation submitted data. Qatar Petroleum Medical Services, reported 3,636 referrals and no

denominator, as such it was not included in Q3. 15

ASPETAR, Al Emadi Hospital, and HMC submitted data. American Hospital reported 2 events and no denominator, as such it was not included in Q1. 16

American Hospital did not submit data. 17

HMC and Doha Clinic Hospital did not submit data. 18

HMC and Doha Clinic Hospital did not submit data. 19

HMC and American Hospital did not submit data.


Results for Quarters 1, 2 & 3 (April 2014 and December 2014)

Legend

! Major deviation from benchmark " Meets or better than

benchmark

# Deviates slightly

from benchmark ! Benchmarking

requires internal target


Average

Benchmark Benchmark

source

Q1 Q2 Q3

1 Processes,

Appropriateness, and Outcomes

In-hospital deaths rate (per 1000)

1.66 1.28 1.501 1.48 38

AHRQ US Nationwide (2010) "

Percent unplanned

readmissions within 28 days of discharge related to the

primary admission

1.562 0.77

3 1.40

4 1.25 1

ACHS National

Australian data (2013)

"

Percent unscheduled returns to the Emergency Department within 48 hours related to

primary visit

2.385 0

6 8.03

7 3.47 0.1

Government of

Western Australia: Lowest reported

rate (2014)

!

Postoperative pulmonary embolism or deep vein

thrombosis rate (per 1000)

0 0.02 0.05 0.02 8.27 AHRQ US

Nationwide (2012) "

Percent Women undergoing

general anesthetic for

Cesarean Section

41.018 46.11

9 53.49

10 46.87 6.40

ACHS Australia

nationwide (2013) !

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!1 American Hospital did not submit data for Q3

2 Al Ahli Hospital did not submit data for Q1


4 Al Ahli Hospital and American Hospital did not submit data for Q3

5 Not applicable to ASPETAR and Al Emadi Hospital. Doha Clinic Hospital and Al Ahli Hospital did not submit data for Q1.

6 Not applicable to ASPETAR and Al Emadi Hospital. Al Ahli Hospital and HMC did not submit data for Q2.


8 Not applicable to ASPETAR and American Hospital. Doha Clinic Hospital did not submit data for Q1.

9 Not applicable to ASPETAR and American Hospital.

10 Not applicable to ASPETAR and American Hospital. HMC did not submit data for Q3.



Legend


benchmark

# Deviates slightly




Average Benchmark Benchmark

source

Q1 Q2 Q3

1 Processes, Appropriateness,

and Outcomes

In-hospital deaths rate (per

1000) 1.66 1.28 1.50

1 1.48 38


Percent unplanned


primary admission

1.562 0.77

3 1.40

4 1.25 1

ACHS National


"

Percent unscheduled returns to the Emergency Department

within 48 hours related to

primary visit

2.385 0

6 8.03

7 3.47 0.1

Government of


rate (2014)

!



0 0.02 0.05 0.02 8.27 AHRQ US

Nationwide (2012) "



Cesarean Section

41.018 46.11

9 53.49

10 46.87 6.40

ACHS Australia

nationwide (2013) !













Legend


benchmark

# Deviates slightly




Average Benchmark Benchmark

source

Q1 Q2 Q3

1 Processes, Appropriateness,

and Outcomes


1.66 1.28 1.501 1.48 38


Percent unplanned


primary admission

1.562 0.77

3 1.40

4 1.25 1

ACHS National


"


primary visit

2.385 0

6 8.03

7 3.47 0.1

Government of


rate (2014)

!



0 0.02 0.05 0.02 8.27 AHRQ US

Nationwide (2012) "



Cesarean Section

41.018 46.11

9 53.49

10 46.87 6.40

ACHS Australia

nationwide (2013) !













Legend


benchmark

# Deviates slightly




Average

Benchmark Benchmark

source

Q1 Q2 Q3

1 Processes,



1.66 1.28 1.501 1.48 38


Percent unplanned


primary admission

1.562 0.77

3 1.40

4 1.25 1

ACHS National


"


primary visit

2.385 0

6 8.03

7 3.47 0.1

Government of


rate (2014)

!



0 0.02 0.05 0.02 8.27 AHRQ US

Nationwide (2012) "



Cesarean Section

41.018 46.11

9 53.49

10 46.87 6.40

ACHS Australia

nationwide (2013) !













Legend


benchmark

# Deviates slightly




Average

Benchmark Benchmark

source

Q1 Q2 Q3

1 Processes,



1.66 1.28 1.501 1.48 38


Percent unplanned


primary admission

1.562 0.77

3 1.40

4 1.25 1

ACHS National


"


primary visit

2.385 0

6 8.03

7 3.47 0.1

Government of


rate (2014)

!



0 0.02 0.05 0.02 8.27 AHRQ US

Nationwide (2012) "



Cesarean Section

41.018 46.11

9 53.49

10 46.87 6.40

ACHS Australia

nationwide (2013) !













Legend


benchmark

# Deviates slightly




Average

Benchmark Benchmark

source

Q1 Q2 Q3

1 Processes,



1.66 1.28 1.501 1.48 38


Percent unplanned


primary admission

1.562 0.77

3 1.40

4 1.25 1

ACHS National


"


primary visit

2.385 0

6 8.03

7 3.47 0.1

Government of


rate (2014)

!



0 0.02 0.05 0.02 8.27 AHRQ US

Nationwide (2012) "



Cesarean Section

41.018 46.11

9 53.49

10 46.87 6.40

ACHS Australia

nationwide (2013) !











How to interpret the Balanced Scorecard with

benchmarks?

• Identify and prioritize improvement goals

• Track progress toward those goals

• Monitor maintenance of progress over time

• Report and Disseminate data using KT reporting tools: Briefing notes,

policy briefs, and dialogues

MONITORING &

EVALUATION (M&E) Key to Evidence-based management in

making decisions and improvement plans

Monitoring & Evaluation (M&E)

M&E integral components of measuring performance:

Monitoring

• Systematic follow-up and reporting of achievements in relation to

goals and objectives set out in the strategic plan.

• Includes follow-up of inputs, activities and outputs, outcomes and

impacts.

• Continuous function that uses the systematic collection of data on

specified indicators to provide the management and main

stakeholders with indications of the extent of progress and

achievement of objectives and progress.

• Results-based monitoring compares the performance of a project,

program or policy with expected results.

Monitoring & Evaluation (M&E) Evaluation

• Measurement of results compared to strategic goals and

objectives and analysis of consistency and adequacy of such goals

and objectives.

• Builds upon monitoring data but the analysis goes deeper, taking

into account contextual changes and addressing attribution.

• Systematic and objective assessment of an on-going or completed

project, program or policy, including its design, implementation and

results.

• Aim of evaluation is to determine the relevance and fulfilment of

objects, development efficiency, effectiveness, impact, and

sustainability.

Critical Success Factors • Competence and capacity of providers to collect data on performance indicators

• Competence and institutional capacity for data collection, analysis, performance review,

quality assessment, conducting audits on the process of data collection, managing M&E

work effectively

• Clear institutional mechanisms for conducting M&E processes:

• Leading monitoring and evaluation

• Producing M&E reports within the scope of the strategic plan

• Proposing remedial actions

• Build context into the analysis to empower decision-makers to take action

• Dialogue-based approach with stakeholders to overcome resistance

• Set priorities for action without taking drastic positions that could alienate actors

Framework for M&E

Monitoring & performance evaluation

1. Monitor indicators of inputs, activities, outputs, outcomes & impacts

2. Evaluate data for efficiency, effectiveness, impact & sustainability

3. Analyze and compare performance with expected results

4. Provide data collected to providers, management and main stakeholders with indications of the extent of progress and

achievement of objectives and collect their feedback

5. Provide suggestions for improving implementation

6. Assess monitoring & evaluation procedures yearly to ensure quality & validity of

indicators

Reporting, Dissemination, and Communication Framework

Ensuring continuous monitoring and effective

evaluation of healthcare performance

1. Monitor healthcare performance through continuous follow-up on

indicators.

2. Evaluate measurement of results compared to strategic goals &

objectives:

• Analyze consistency and adequacy of such goals and objectives

through systematic and objective assessment of KPIs.

• Build upon monitoring data but take into account contextual

changes.

Framework for M&E

3. Analyze data and compare performance with expected results and targets

4. Provide performance reports to providers, management and key stakeholders

5. Recommend improvements for measuring KPIs including recommendations for

improving data gaps, standards and quality issues, consistency of measurement of

indicators

Framework for M&E

6. Conduct regular planned assessments of the M&E system on a yearly basis with

providers to ensure:

• Indicators are measuring what they are meant to and that they are relevant to

the strategic goals and objectives. Indicators are responsive to the information

needs for monitoring progress and performance.

• Indicators cover each domain along the results chain: inputs/ structure &

processes, outputs, outcomes, and impact.

• Data generated according to standards, and critical data gaps are identified

and addressed.

EXERCISES & GROUP

WORK

from measurement and reporting to actual improvement and … · 2016-03-24 · the balanced...

Documents