1

Measuring and Improving

Quality in Medical Imaging

John Mathieson MDBob Clark

VIHA

2

Measuring and Improving Quality

in Medical Imaging

Current Areas of Interest in VIHA

and

Overall Perspective

3

Measuring and Improving Quality in Medical Imaging

• Huge potential gains• Many current problems• Hard to Measure, Hard to Improve• Expensive

4

Current areas of interest

1. Typical Report Accuracy analysis

– CT Virtual Colonoscopy Project

2. Novel Electronic Systems

3. Report Turn-around Time - Productivity

5

Current areas of interest

• Report accuracy – how to measure?• Manual method

–Expensive–Time consuming –Not done routinely

• Current project – CT Virtual Colonoscopy – Endoscopic Pathologic correlation

6

Measuring Work Quality

Polyps called at CT VC – – What is found at Colonoscopy / Pathology?

Hire someone to track down clinical follow-up and correlate

Traditional statistics – PPV NPV etc

Not ordinary part of workSpecial Project

7

Current areas of interest

• Both the Imaging reports and the final diagnoses end up computerized –

–BUT – no method of automatic linking and feedback

• Ideally – all reports cases with some kind of proof would feed back to original reports

8

Other Questions – How many cases do new readers need to

be qualified to read CT VC?

• Wild guess• Nice sounding round number

• Actual Data– Measure accuracy vs experience– Subjective self assessment

9

CT VC Reader Assessment

• Testing on unknown cases at various points in experience

• Subjective – ask all readers to describe their own experience with retrospective recommendations

10

Potential for Electronic Systems

• Commissure – voice recognition for Intelligent text analysis

• Categorize reports automatically – positive / negative, other

• Correlate with – Indications / History

- Referring MD

11

12

Industry Overview

• Radiology is wrestling with optimizing the appropriate use of imaging, spiraling costs, decreasing reimbursements, and its role in improving patient outcomes.

– Over 1 billion radiology exams performed each year in US

– Fastest growing component of medical costs

– Compound annual growth rate (CAGR) of 20%

– Over $100 billion in annual US diagnostic imaging costs

13

Overview: Technology Background

• Appropriateness database consisting of over 11,000 rules based on patient demographics and covering MRI, MRA, Breast MR, CT, CTA, PET/CT and Cardiac Stress Testing

– Foundation based on ACR Appropriateness Criteria® – expanded to cover broader range of imaging procedures – with input from over 1500 clinicians at MGH/Harvard

– Exclusive license agreement for rules database

– Utility score (1-9) appropriateness ratings

14

Overview: Decision Support Utility Score

• The appropriateness scores range from 1-9 and are associated with the following relevance:

Indicated (7-9): indicates the desired exam is appropriate given the indications

Marginal (4-6): while the desired exam may yield results, a more appropriate exam may exist

Low (1-3): indicates the exam is less than optimal and more appropriate imaging techniques should be considered

15

16

Real World Case: Massachusetts General Hospital

• Low-utility (inappropriate) exams decreased significantly, from 11% of the total CT volume before implementation to 4% by the end of the study period.

17

Real World Case: Massachusetts General Hospital

• The portion of high-utility (appropriate) CT exams rose significantly, from 86% before implementation to 93% after referrers learned to use the system. The trend was the same for MR.

18

Real World Case: Massachusetts General Hospital

• Overall CT and MR utilization was also affected. CT use rose at an average 4% in each quarter from 2001 to 2003. The curve flattened after implementation, reflecting slowed growth. Again, a similar trend was seen for MR volume.

• Positive findings in radiology reports increased from 74% to 84% for CT and 73% to 85% for MR.

19

Total Outpatient High Cost Imaging Volume Trends

Radiology DS Implementation

20

MR Spine Positivity by Specialty

21

Fully integrated from Order Entry to Results Analysis –Results – feed back on ordering criteria

22

Possibilites for Data Analysis / Quality Measurement

• Front end • Back End• Linking Front End with Back End

• Ordering physician audit• Audit by Indication• Audit by radiologist

23

Turn Around Time – Productivity

24

Turn Around Time – Productivity

• Many steps involved – one of which is - Once study completed

– how fast to dictation and sign off?• Extremely variable

Under 24 hours to Over 1 week

• Problems with slow turn around• Delayed treatment decisions• Longer hospital stays• Extra work created – phone reports etc

25

3 Kinds of Workers

Turtles

Racehorses

Everyone Else

26

Turtles

Slow, steady, very attentive to detail, unhappy with change and pressure, miss very little

27

Racehorsesaka

Vacuums

28

Everybody Else

Sometimes fast, sometimes slowDistractible, curious, intelligent

Easily bored - “Focus-able”

29

What can you modify ?

Speed of reportingQuality of reportingTime spent reporting vs other thingsWork hoursDistribution of work

Easy thingsHard things

30

Everyone

Good, conscientious peopleProud of their workAt least some degree of :

people pleasing need egoinsecuritycompetitiveness

31

Versions of the TRUTH

People WILL shirk work they don’t like

People WILL get away with things

What you don’t count and measure will hurt you

32

Your co workers are extremely good people

You are lucky to work with them

Collegial competitiveness is better than cut-throat aggression

Versions of the TRUTH

33

Observer Effect

• It is impossible to accurately measure anything, because the act of measuring affects the answer

• Thermometer to measure absolute zero- the thermometer warms up the room

BAD THING – or GOOD THING ?

Why not try to MAXIMIXE the observer effect to get the Maximum change in the answer?

34

Count and

Measure

Study Report Status - Statistics GeneratorReport from 01/12/2007 to 01/13/2007+=+=+=+=+=+=+=+=+=+=+=+=+=

+=+=+=Date: Sat Jan 13 03:45:03 2007

Total Results Dictated: 1001Total Results Transcribed: 959

Radiologist Results Dictated ----------- ---------------- rjsmith 46 dshea 15 vvanraalte 56 nfinn 110 forkheim 126 dzacks 114 brlee 55 jmathies 123 dconnell 91 cvwinc 11 dchu 61 jwrinch 67 iweir 42 goodacre 24 whodgins 60

35

Problems

Racehorses vacuumed up everything

Others began to relax

Racehorses started to get annoyed

36

Basic Minimum - Quota

Consensus on a reasonable amount of work for each rotation

Background vs Variable Work

Example – US and GeneralDo all the US at that locationPlus – X number of

Radiographs

37

Quota Counter

38

Plus / minus scores – like hockey

39

Results

• Dramatic reduction in turnaround time– Actual measurement VGH – 67%

• Dramatic shift in time of day work is done

• Feelings of fairness, equity and group harmony

Unexpected ResultSpeed with which expectations changed

40

Quality in Medical Imaging

Areas of Concern• Access for Patients• Access to

Information• Image Quality• Patient Safety• Report Accuracy• Report Delivery

41

Access for Patients• Lack of access – wrong dx, unnecessary

surgery, wrong surgery, untreated conditions• Wrong test – right test hard to get – do inferior

test• Economic models – Activity based funding vs

Block Funding• Spend budget wisely – justify expenditures

– $100,000 is equivalent to 12,500 extra CT scans !

• A BIG Problem

42

Access to Information

• Integrated PACS / RIS / HIS systems• Integrated into community offices• “Middleware” – functionality

Host of benefits – accurate timely info- appropriate tests, no uneccessary repeats, right test first time, timely delivery important results

43

Access to InformationProblems

• Slow implementation of systems

• Expen$ive

• Privacy / Security Concerns– Often the balance between Access and Security is Skewed by Paranoia over security Access Security

44

Image Quality

• Equipment replacement – inventory maintenance – no financial model

• Single year purchases with wildly fluctuating amounts – Chronic inability to replace worn out

equipment– “Normal” to have some equipment

running that is not safe or diagnostic

45

Traditional DAP role –

Radiation dose vs Image Quality

• Sad truth – long history of operating poor quality equipment due to lack of funding

• What should be done? Put some teeth into DAP

– close down unsafe equipment

» change funding model

46

Summary• Many areas to focus on – beyond

traditional scope

• Clever use of electronic systems can make quality improvement more practical and routine

Access for PatientsAccess to InformationImage QualityPatient SafetyReport AccuracyReport Delivery

47