1

Audits, Benchmarks and Performance: What You Need to Know

Beth Burnside, MD, MPH

Outline • Auditing our practice: WHY? • History

• Evolution from technical to interpretive • Quality Assurance and Quality Control

• Accreditation • Audit

• New modalities included • What is on the horizon?

2

Accreditation and Audit–Why?

15-45% breast cancer mortality reduction

QUALITY WAS/IS CRITICAL Lancet 2012; 380:1778-1786

Timeline ACR

accreditation

ACR BI-RADS

ACR NMD

BCSC

Technical Interpretive/Outcomes

MQSA

IOM Report

ACR BCOE

IOM NCPF

Quality Assurance and Control

• Accreditation (technical) • ACR

– National Mammography Accreditation Program • established in 1987 as a voluntary program. • qualified personnel perform & interpret • dedicated mammography equipment

• MQSA – 1992 Congress passed – Mandated accreditation – Nationally based, nonprofit accreditation program

6

Compression

• Why? • Contrast • Decreases dose • Reduce scatter • Reduce geometric blur • Decreased motion blur

ACR Accreditation

Submitting Clinical Images • Examples of facility’s best work • Supervising radiologist review & approve

images • Submit “negative” images

– BI-RADS 1 or 2

Ultrasound Accreditation • Personnel • Equipment • Quality Control tests • Peer Review • Reporting • Clinical images (Exam ID) • Tracking non-definitive core biopsies

– Atypical/Radial scar (ARS) – Discordant – Insufficient

• Tracking biopsy complications

Stereotactic Accreditation • Personnel • Equipment • Quality Control tests • Clinical images (Exam ID) • Tracking non-definitive core biopsies

– Atypical/Radial scar (ARS) – Discordant – Insufficient

• Tracking biopsy complications

MRI Accreditation • Personnel • Equipment • Safety • Quality Control tests • Clinical images

Breast Imaging Center of Excellence

• All accreditation standards met: – Mammography – Stereotactic breast biopsy – Breast ultrasound – MRI

MQSA

MQSA • Last reauthorized in 2004 • In preparation for reauthorization in 2007 • IOM report

IOM Reports

2005 2015

Components of MQSA • Protocols • Accreditation • Quality Control • Audit Data

The MQSA Audit

Heavy Lifting of Interpretive Quality

• Audit metrics: : ACR BI-RADS • Benchmarks: BCSC - NMD • Guidelines: Experts and the Literature

Audit: ACR BI-RADS

Audit • Purpose: To demonstrate success in detecting

clinically occult breast cancer – False Positives – True Positives – Cancer Characteristics – True Negatives – False Negatives

Audit • Purpose: To demonstrate success in detecting

clinically occult breast cancer – False Positives – True Positives – Cancer Characteristics – True Negatives – False Negatives

Positives • PPV

– TP/(TP+FP) • PPV 1 = # cancer / all recalls • PPV2 = # cancer / biopsy recommendations • PPV3 = # cancer / biopsies performed

• Recall Rate – TP+FP / all screens

• Cancer Detection Rate – TP / all screens

Cancer Characteristics • Specifics about positives

– Size – Stage – Nodal status

Cancer Characteristics • Stage 0+1 • Minimal cancers • Negative nodes

Audit • Purpose: To demonstrate success in detecting

clinically occult breast cancer – False Positives – True Positives – Cancer Characteristics – True Negatives – False Negatives

Negatives • Much more difficult to evaluate • Direct tracking is costly • Estimation using reliably tracked patients • How can it be done?

– Retrospective review of cases – Registry match

Sensitivity and Specificity • Sensitivity

– TP/(TP+FN) – Of all cancer, how many detected

• Specificity – TN/(TN+FP) – Of all negatives, how many correctly assessed

Sensitivity and Specificity • Sensitivity

– TP/(TP+FN) – Of all cancer, how many detected

• Specificity – TN/(TN+FP) – Of all negatives, how many correctly assessed

Benchmarks vs. Guidelines • Benchmarks show existing performance • Guidelines convey desired/mandated

performance

Benchmarks

Rosenberg RD, et al. Performance benchmarks for screening mammography. Radiology. 2006;241(1):55-66.t

Benchmarks-Screening Mammography

Guidelines

Carney PA, et al. Identifying minimally acceptable interpretive performance criteria for screening mammography. Radiology. 2010;255(2):354-61.

Guidelines-Screening Mammography

IOM Reports

2005 2015

Databases

BCSC

NMD

1996

1997 2009

Breast Cancer Surveillance Consortium

ACR National Mammmography Database

BCSC sites

• 5 participating sites

• 9.5 million mammograms • 180,000 biopsies • 113,000 breast cancers

http://breastscreening.cancer.gov/about/bcsc_fact_sheet.pdf

National Mammography Database (NMD) • Gives a practice the ability to track performance and

compare to national and regional benchmarks • Participants receive semi-annual feedback reports that

include important benchmark data such as cancer detection rates, positive predictive value rates and recall rates. Comparison measures are calculated by Physician, Facility, and Physician group

• Participation in the NMD is free to Breast Imaging Center of Excellence (BICOE) facilities.

http://www.acr.org/Quality-Safety/National-Radiology-Data-Registry/National-Mammography-DB

NMD Participation Process

Participation in NMD is easy! 1. Complete a new facility registration 2. Send a signed copy of participation agreement to

NRDR 3. Submit data

NMD Participation • 383 registered facilities • 166 actively contributed To date: • 8,987,625 million

mammograms • 1,872,724 in the last year!

NMD Screening Exams Annual Accrual Cumulative Accrual

2008 84,238 84,238 2009 319,555 403,793 2010 680,873 1,084,666 2011 1,105,268 2,189,934 2012 1,280,292 3,470,226 2013 1,728,815 5,199,041 2014 1,915,860 7,114,901 2015 1,872,724 8,987,625

Mammography Screening Outcomes

PQRS: Physician Quality Reporting System • American Taxpayer Relief Act of 2012: Participation in a “clinical data registry”

meets the requirement for satisfactorily reporting quality measure data • The ACR NRDR has been approved as a Qualified Clinical Data Registry (QCDR)

for the CMS PQRS in the 2016 program year. • Under the Medicare Physician Fee Schedule Final Rule, eligible professionals

and group practices will be able to meet PQRS quality reporting requirements by participating in a QCDR.

• There are 6 measures from NMD for physicians to be used in PQRS reporting • http://www.acr.org/~/media/ACR/Documents/PDF/QualitySafety/NRDR/QCDR/

ACR%20QCDR%20Measures%20Supported%202016_NonPQRS.pdf

Registry Participation is Important to Deliver Quality Care Participate in NMD!!!

Benchmarks-Diagnostic Mammography

Guidelines-Diagnostic Mammography

US Audit • Breast US

– Unique: Standard views should be determined – Recall defined as additional images taken

• Either same day—aka online (~tech performed) • Or subsequent day (~automated full breast US)

• Screening US audit – Newly established based on:

• Limited research trials in high risk populations • May not reflect general practice

– Documented in BI-RADS V5

Benchmarks-US Screening

MRI Audit • Breast MRI

– Unique: screening = diagnostic examination – Audit will be based on rigorous collection of:

• clinical “indication” • Screening MRI audit

– Newly established based on: • Data from trials (high risk populations) • May not reflect general practice

– Documented in BI-RADS 5th Edition

Benchmarks-MRI Screening

MRI & US Diagnostic

Conclusions • Quality is critically important • Accreditation is a lot of work

Is there a silver lining to accreditation?

YES!

The Silver Lining • Defines the bar for all • Decreases variability • Combats turf battles

ENSURES BENEFITS!

Saves lives!

Thanks for you attention