-
Population and Patient Centered Care:Unintended Consequences of Measurement
11/11/19
David C. Aron, MD, MS
Louis Stokes Cleveland Dept. of Veterans Affairs Medical CenterDept. of Medicine, Div. of Clinical and Molecular Endocrinology,
School of Medicine andDept. of Organizational Behavior, Weatherhead School of Management,
Case Western Reserve University Cleveland, OH
1
-
The following presentation is rated R (for opining and sarcasm which may “inadvertently” seep in). Viewer discretion is advised, but feel free to challenge everything I say.
2
-
3
Disclaimer: The opinions expressed in the following presentation are solely those of the presenter, and do not represent those of any agency or organization
Disclosures: • Grant funding from Dept. of Veterans Affairs• Endocrine Society Representative to NCQA/AMA on
Performance Measures• NO money from Big Pharma, medical device companies,
etc.
-
Outline
• A story of a performance measure gone wrong• Unintended or unanticipated consequences• Causes of unintended consequences• Dealing with unintended consequences
• Identify possible consequences beforehand• Prevention• Detection• Mitigation
4
-
• Two new Comprehensive Diabetes Care measures assess whether members with diabetes have their blood pressure controlled to 135/85, and whether their HbA1c levels are controlled to less than 7 percent, the nationally accepted standard of adequate HbA1c control.
• Piloted in 2005, initiated in 2006
5
A brief history of the A1c
-
Age- and sex-standardized rate of severe hypoglycemia resulting in hospital
admission, observation stay, or emergency department visit per 100
person-years, 2006–2013.
Kasia J. Lipska et al. Dia Care 2017;40:468-475
©2017 by American Diabetes Association
What happened? Frequency of ED visits and hospitalizations for hypoglycemia increased.
6
-
What was behind the rise in rates of hypoglycemia rise?
7
-
InputsOutputsProcesses
Sensor
Other influences on the Inputs Feedback
Cybernetic System
Effector
8
-
Sensor Comparator Glucose Level
Effector
Feedback
Target Organs
Other Factors,
e.g., meals
Set Point
β-Cell
Cybernetic View of Glucose Control by the β-Cell
9
-
Sensor Comparator A1cEffector
Rx Adjust-
ment in a patient
with diabetes
Set Point
Physician
Feedback System (Self-Performance Management)
1. How effective is the feedback?
2. What about the set point?
3. What other actions might the physician take (or not take)?
10
-
If Set Point is set too LOW
If Set Point is set too HIGH
Patient’s A1c > set point
DIAGNOSIS: Inadequate
Glycemic Control
OVER-Rx
UNDER-Rx
Where you set the set point, matters.
What influences the choice of set point?
11
-
Sensor Comparator A1cEffector
Rx Adjust-
ment in a patient
with diabetes
Set Point
Physician
Feedback System (Self-Performance Management)
1. How effective is the feedback?
2. What about the set point?
3. What other actions might the physician take (or not take)?
12
-
Sensor Comparator Effector
Set Point
Physician
Health Care System
Desired Performance
Pay for Performance
Influence
Feedback
A1c
Rx Adjust-
ment in a patient
with diabetes
Patient Advocacy
GroupsPayersProfessional
SocietiesPharmaceutical/ Medical Device
Industry
Quality Measurement
Industry
Stakeholders
13
-
14
• A national PSA campaign sponsored by the Conference of Mayors in partnership with Aim. Believe. Achieve. The Diabetes A1C Initiative(tm).
http://usmayors.org/uscm/home.asp
-
New NCQA Quality Diabetes Measure was promoted by NDEP (4/2004)
• “An A1C
-
16
-
WHO RAN THE CAMPAIGN?
17
-
What happened to the NCQA A1c
-
Assessing potential glycemic overtreatment in persons on insulin and/or sulfonylureas at high hypoglycemic risk. Tseng et al. JAMA Int Med 2014: 174; 259-268
a Study population: patients with diabetes, on insulin or sulfonylureas, having HbA1c in FY 2009.bCI: Cognitive Impairments. cD: Dementia. dAdvanced diabetes complications: end-stage renal disease, amputations, advanced retinopathy. eDiminished life expectancy: cancer, end-stage hepatic disease.fMajor neurologic disorders: gastro paresis, Parkinsons, aphasia, dysphagia, hemiplegia, apraxia, epilepsy, transient ischemic attack. gCardiovascular diseases: myocardial infarction, chronic heart failure, ischemic vascular disease.
Group
High risk patientsIncreased number of patients
Number of patients in denominator
% of study population a(n=652,738)
Overtreatment measures (in %). % with A1c
-
Overtreatment in groups with high risk of hypoglycemia (an unintended consequence of focus on undertreatment)
Target measure A1c
-
“There are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns – the ones we don't know we don't know.” Donald Rumsfeld
21
If not an intended consequence, then what?
-
Unintended versus Unanticipated consequences
De Zwart F. Unintended but not unanticipated consequences. Theory and Society. 2015 May 1;44(3):283-97.
http://www.katinamichael.com/research/tag/users
Consequences Anticipated Unanticipated
Intended Known knowns Unknown knowns i.e., ignorance, denial
Unintended Known unknowns Unknown unknowns
(Hanlon’s Razor)
22
-
L. Gregory Pawlson, MD, MPH; and Thomas H. Lee Jr, MD Am J Mgd Care 2010; 16: 16-17 Letter in reponse to paper by Choe et al. re: hypertension overtreatment
23
“It is important to note that the National Committee for Quality Assurance (NCQA) includes a strong advisory that 100% performance is not the goal and that clinical judgment should be used in applying a measure…Although physicians should be aware of guidelines and measures, they need also to apply more nuanced approaches when seeing individual patients. To imply that clinicians would knowingly put patients in harm so they could perform marginally better than other physicians on a clinical performance measure provides a rather dim view of medical practice.”
http://www.ajmc.com/publications/issue/2010/2010-01-vol16-n01http://www.ajmc.com/publications/issue/2006/2006-06-vol12-n6/Jun06-2318p309-310
-
knowingly put patients in harm?
• In response to a vignette of a 77y/o with serum creatinine of 2.4 mg/dl on glipizide 5 mg day with A1c 6.5 :
• 55% state that patients would be harmed by relaxing control• 42% state that relaxing control would fall outside measures • 27% state that relaxing control would be difficult to recommend, 22% believe patients would find it difficult to accept
24
Caverly TJ, Fagerlin A, Zikmund-Fisher BJ, Kirsh S, Kullgren JT, Prenovost K, Kerr EA. Appropriate Prescribing for Patients With Diabetes at High Risk for Hypoglycemia: National Survey of Veterans Affairs Health Care Professionals. JAMA Intern Med. 2015 Dec;175(12):1994-6
-
• To the Editor: Drs Pogach and Aron used the Toyota example of failure to quickly “pull the cord” at the first sign of a safety or quality problem to suggest leading guideline and measure development organizations did not respond to changing evidence on control of glycated hemoglobin (HbA1c) levels in patients with diabetes.
25
Margaret E. O’Kane, National Committee for Quality Assurance Washington, DC; Greg Pawlson, Department of Medicine George Washington University School of MedicineWashington, DC JAMA May 18, 2011, Vol 305, No. 19, pp96-7
In response to: Pogach L, Aron DC. Sudden acceleration of diabetes quality measures. JAMA 2011 Feb 16;305(7):709-710.
https://blog.toyota.co.uk/andon-toyota-production-system
-
• In fact, the National Committee for Quality Assurance (NCQA) did respond quickly when the National Institutes of Health suspended the ACCORD (Action to Control Cardiovascular Risk in Diabetes) study intervention group in February 2008 and the ACCORD and ADVANCE (Action in Diabetes and Vascular Disease: Preterax and Diamicron Modified Release Controlled Evaluation) studies were published later that year. Within 2 weeks of the suspension, NCQA suspended use of the measure and launched an expedited review of the HbA1c measures…
26
https://blog.toyota.co.uk/andon-toyota-production-system
-
27
In Reply: In the mid 2000s, several measurement developers endorsed optimal measures for all persons with diabetes aged 18 to 74 years, even though the ACCORD and other studies were ongoing and, consequently, the outcomes still unknown. The strength of evidence supporting recommendations varied widely…The andon cord analogy highlights the fact that the developers ignored concerns voiced by the National Diabetes Quality Improvement Alliance and a subsequent Agency for Healthcare Research and Quality–funded conference years before ACCORD and other trials reported their findings—early warnings.
Although NCQA re-evaluated the scientific validity of their measures, it did so only subsequent to termination of the glycemic treatment group of ACCORD.
Pogach and Aron
https://blog.toyota.co.uk/andon-toyota-production-system
-
Choosing Wisely. An initiative of the ABIM. American Geriatrics Society
• Avoid using medications to achieve hemoglobin A1c
-
In response to all of this, VA launched the Choosing Wisely/Hypoglycemia Safety Initiative (CW/HSI) in 2014
• National voluntary program• Lists
– High risk: A1c < 7 and on Insulin or Sulfonylurea who:• Are age 75 or greater• OR renal impairment (creatinine >2.0)
– Ultra high risk: A1c
-
Results: Facility Rates of Over and Under Rx
measure 2013 2016mean±1SD range mean±1SD range
A1c
-
Change in Overtreatment Rate (A1c9%)
Change (%) in Undertreatment RateDecreasing Under RxIncreasing Under Rx
Chan
ge (%
) in
Ove
rtre
atm
ent R
ate
Decr
easin
g O
ver R
xIn
crea
sing
Ove
r Rx
R= -0.653
P
-
Ignore UnderRx
OverRx UnderRx
Focus on OverRx
Ignore OverRx
Focus on UnderRx
• Promotion of overtreatment reduction may be associated with an increase in undertreatment in patients with diabetes.
• Systems should include balancing measures to identify potential unintended harms.
Aron DC, Wilson B, Tseng CL, Soroka O, Pogach LM. Positive Deviance in Health Care: Beware of Pseudo-Equifinality. In: Putting Systems and Complexity Sciences Into Practice 2018 (pp. 189-198). Springer, Cham.32
-
Undertreatment rates over time among facilities with lowest rates of overtreatment
Aron DC, Wilson B, Tseng CL, Soroka O, Pogach LM. Positive Deviance in Health Care: Beware of Pseudo-Equifinality. In: Putting Systems and Complexity Sciences Into Practice 2018 (pp. 189-198). Springer, Cham.
average rate among all VA facilities.
33
-
Outline
• A story of a performance measure gone wrong• Unintended or unanticipated consequences• Causes of unintended consequences• Dealing with unintended consequences
• Identify possible consequences beforehand• Prevention• Detection• Mitigation
34
-
Goodhart’s Law
•"When a measure becomes a target, it ceases to be a good measure."
35
-
A taxonomy of unintended consequences, • Tunnel vision • Measure fixation • Acontextual actions or adverse selection • Misrepresentation• Gaming • Myopia • Suboptimization• Ossification is the inhibition of innovation.
36
• Smith P. On the unintended consequences of publishing performance data in the public sector. International journal of public administration. 1995 Jan 1;18(2-3):277-310.
• Rambur B, Vallett C, Cohen JA, Tarule JM. Metric-driven harm: an exploration of unintended consequences of performance measurement. Appl Nurs Res. 2013;26(4):269-72.
-
Adam A. Powell, PhD, Katie M. White, Melissa R. Partin, Ph , Krysten Halek, MA , Jon B. Christianson, PhD , Brian Neil, MD , Sylvia J. Hysong, PhD , Edwin J. Zarling, MD , and Hanna E. Bloomfield, MD J Gen Intern Med 27(4):405–12
37
Unintended Consequences of Implementing a National Performance Measurement System into Local Practice
-
• Bevan G, Hamblin R. Hitting and missing targets by ambulance services for emergency calls: effects of different systems of performance measurement within the UK. Journal of the Royal Statistical Society: Series A (Statistics in Society). 2009 Jan;172(1):161-90.
NHS Ambulance resonse times
38
-
39
Bloomrosen M, Starren J, Lorenzi NM, Ash JS, Patel VL, Shortliffe EH. Anticipating and addressing the unintended consequences of health IT and policy: a report from the AMIA 2009 Health Policy Meeting. Journal of the American Medical Informatics Association. 2011 Jan 1;18(1):82-90.
-
Campbell's law
"The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.”
By Source, Fair use, https://en.wikipedia.org/w/index.php?curid=4155776
40
-
Outline
• A story of a performance measure gone wrong• Unintended or unanticipated consequences• Causes of unintended consequences• Dealing with unintended consequences
• Identify possible consequences beforehand• Prevention• Detection• Mitigation
41
-
Sterman J. Business dynamics: systems thinking and modeling for a complex world. Boston, MA: Irwin McGraw-Hill, 2000.
Single-Decision “Open Loop” View
Problem Results
Goals
Situation
Decision
“SideEffects”
Feedback ViewGoals
Environment
Actions
Goals ofOthers
Actions ofOthers
“SideEffects”
Delay Delay
Delay
Delay
DelayDelay
Delay
DelayDelay
Delay
Delay
Delay
42
-
A System…
• An interconnected set of elements that is coherently organized in a way that achieves something.
• Elements• Interconnectedness• Function or purpose
-43-
Meadows, D. Thinking in Systems: A Primer.
-
Dividing a cow in half does not give you two smaller cows
44
The picture can't be displayed.
Todd Little
-
Properties of Complex Systems • Large collections of relatively simple components• Components interact locally and nonlinearly• Limited centralized control• Components have limited access to global information• Components have ‘bounded rationality’• System as a whole has emergent, collective behavior
• Complex dynamics• Information processing and computation• Adaptation and learning
45Mitchell M. Complexity: A guided tour. Oxford University Press; 2009.
-
Complex Environment
46
Thank you to whoever came up with this series of figures. Please let me know who you are so I can give you proper attribution.
-
Intervention
47Insert action into the system
-
Unpredictable Outcomes
48
EMERGENCE
-
I need some volunteers
49
-
Schelling Model
50
https://www.youtube.com/watch?v=JjfihtGefxk
https://www.youtube.com/watch?v=JjfihtGefxk
-
51
• Complex systems are inherently uncontrollable
• They cannot be understood sufficiently even while they are unfolding
• So prediction and control of complex systems is very hard, if not impossible, to do
-
52http://static.guim.co.uk/sys-images/Guardian/Pix/site_furniture/2010/4/28/1272477292847/Afghanistan-powerpoint-gr-008.jpg
-
Outline
• A story of a performance measure gone wrong• Unintended or unanticipated consequences• Causes of unintended consequences• Dealing with unintended consequences
• Identify possible consequences beforehand• Prevention• Detection• Mitigation
53
-
THUMB PINNING EXERCISE
Julie Johnson, PhD
• Goal: pin your partner as many times as possible in 15 seconds. I will tell you when to start.
54
-
Question your assumptions.
55https://thegreatoutdoorsman.net/2019/07/09/the-far-side/
-
P4P
Rx of DM Structure, process, outcomes, cost
“Quality” of Care“Value” (Q/cost)
“Set Points”
Assumption: Individual variation is caused by variation in motivation - i.e. that poor performance is a matter of intention, and that change in motivation is therefore likely to improve performance.
Identify possible consequences beforehand: 1. Question your assumptions, i.e. your mental model.
-
Identify possible consequences beforehand: 2. Systems thinking
• Meadows, D. Thinking in Systems: A Primer. 2008• De Savigny D, Adam T, editors. Systems thinking for health systems
strengthening. World Health Organization; 2009. • Systems Sciences and Population Health; El-Sayed and Galea, editors; 2017 57
-
System DynamicsSimulating Dynamic Complexity
Good at Capturing• Differences between short- and long-term consequences of an action• Time delays (e.g., incubation period, time to detect, time to respond)• Accumulations (e.g., prevalences, resources, attitudes)• Behavioral feedback (reactions by various actors)• Nonlinear causal relationships (e.g., threshold effects, saturation effects)• Differences or inconsistencies in goals/values among stakeholders
Forrester JW. Industrial Dynamics. Cambridge, MA: MIT Press; 1961.
Sterman JD. Business Dynamics: Systems Thinking and Modeling for a Complex World. Boston, MA: Irwin/McGraw-Hill; 2000.
-
Outline
• A story of a performance measure gone wrong• Unintended or unanticipated consequences• Causes of unintended consequences• Dealing with unintended consequences
• Identify possible consequences beforehand• Prevention• Detection• Mitigation
59
-
Prevention 1. Use good measuresWhat Makes a Good Quality Measure (in addition to accuracy and validity)?
• Target patients most likely to benefit• Help providers do the “right” thing• Incorporate (or at least don’t ignore) patient
preferences• Acknowledge limitations of current data sources
and resulting measures (and motivate collection and use of clinically detailed data)
• Avoid unintended consequences
http://www.ahrq.gov/QUAL/diabetescare/ 60
-
Outline
• A story of a performance measure gone wrong• Unintended or unanticipated consequences• Causes of unintended consequences• Dealing with unintended consequences
• Identify possible consequences beforehand• Prevention• Detection• Mitigation
61
-
In addition to making sure you are measuring what is important:
Avoiding and mitigating unintended consequences.
• Balancing measures• Indirect – What might doctors NOT do in order for them to do what you
want them to do.• Direct – Reciprocal; If you are assessing undertreatment, then also
asses overtreatment• Continuous rather than dichotomous measures
62
-
8
9
10
7.5
7
6
CHOOSING WISELY : A1c ≥7.5%Ages ≥65 on medication other than metformin alone
NQF MEASURE: A1c 65-75; no exclusion criteria
NQF MEASURE: A1c>9%UNDER TREATMENT (UT) Applies to ages >65-75; no exclusion criteria
In Range (IR)Out of Range (OOR)
DHHS PROPOSED OVER TREATMENT (OT) : A1c65 on hypoglycemic agents
8.5GUIDELINES: Limited life expectancy, significant co-morbid conditions
Pogach L, Tseng CL, Soroka O, Maney M, Aron D. A Proposal for an Out-of-Range Glycemic Population Health Safety Measure for Older Adults With Diabetes. Diabetes Care. 2017 Apr;40(4):518-525.
Out of Range Measure
63
-
Actions have consequences, both intended and unintended. The implementation of an improvement initiative or a performance measure are actions. When adverse unintended consequences can be anticipated, it is incumbent upon systems to include mitigating actions such as counterbalancing measures to ensure that unintended harms are avoided.
Summary
Aron DC. No "Black swan": unintended but not unanticipated consequences of diabetes performance measurement. Jt. Comm J Qual. Patient. Saf. 2013;39(3):106-8. 64
-
“It is easy to dodge our responsibilities, but we cannot dodge the consequences of dodging our responsibilities.”
Josiah Stamp
65
-
Additional References• Mannion R, Braithwaite J. Unintended consequences of performance
measurement in healthcare: 20 salutary lessons from the English National Health Service. Intern Med J. 2012;42(5):569-74.
• Hysong SJ, SoRelle R, Broussard Smitham K, Petersen LA Reports of unintended consequences of financial incentives to improve management of hypertension. PLoS ONE 2017; 12(9): e0184856.
• Nescolarde-Selva JA, Gash H, Usó-Domenech JL. What are unintended and adverse consequences?. Kybernetes. 2019 Feb 4;48(2):226-37.
• Franco-Santos M, Otley D. Reviewing and theorizing the unintended consequences of performance management systems. International Journal of Management Reviews. 2018 Jul;20(3):696-730.
• Mica, A., Sociology as Analysis of the Unintended From the Problem of Ignorance to the Discovery of the Possible. Routledge. 2019.
66
Population and Patient Centered Care:�Unintended Consequences of Measurement�11/11/19��David C. Aron, MD, MS��Louis Stokes Cleveland Dept. of Veterans Affairs Medical Center�Dept. of Medicine, Div. of Clinical and Molecular Endocrinology,�School of Medicine and�Dept. of Organizational Behavior, Weatherhead School of Management,�Case Western Reserve University Cleveland, OH��[email protected]�Slide Number 2Slide Number 3OutlineSlide Number 5Slide Number 6What was behind the rise in rates of hypoglycemia rise?Slide Number 8Slide Number 9Slide Number 10Slide Number 11Slide Number 12Slide Number 13Slide Number 14New NCQA Quality Diabetes Measure was promoted by NDEP (4/2004)Slide Number 16WHO RAN THE CAMPAIGN?What happened to the NCQA A1c