Building the Science of Health Care Quality Improvement

Intervention Denise Dougherty, Ph.D.

Senior Advisor, Child Health and Quality Improvement AHRQ Annual Conference 2010

Coordinator and ModeratorImplementation, Change, and Improving Health Care Quality and Safety: Lessons Learned From AHRQ’s

Implementation Science AwardsSeptember 28, 2010

Overview (15 minutes)

Introductory Comments: Why and how does AHRQ Focus on Implementation Science?

An emerging framework Implementation Science grantees work

– Rita Mangione-Smith– Carrie Byington

Interactive Discussion

The “3T’s” Road Map to Transforming U.S. Health Care

Key T1 activity to test what care works

Clinical efficacy research

Key T2 activities to test who benefits from

promising care

Outcomes research

Comparative effectivenessresearch

Health services research

Key T3 activities to test how to deliver high-quality

care reliably and in all settings

Measurement and accountability of health care quality and cost

Implementation of Interventions and health care system redesign

Scaling and spread of effective interventions

Research in above domains

T1 T2 T3Basic biomedical

scienceClinical efficacy

knowledgeClinical effectiveness

knowledge

Improved health care quality and

value andpopulation health

Source: JAMA, May 21, 2008: D. Dougherty and P.H. Conway, pp. 2319-2321. The “3T’s Roadmap to Transform U.S. Health Care: The ‘How’ of High-Quality Care.”

Relevant Provisions in New Laws: American Recovery and Revitalization Act (ARRA)

“Meaningful Use” of Health IT = Beyond getting the electrons in place Using health IT to improve quality and safety of health

care

Funding for Comparative Effectiveness Research (CER) beyond purely clinical interventions

System redesign Enhanced registries for QI and CER Accelerating Implementation of Comparative

Effectiveness Findings on Clinical and Delivery System Interventions by Leveraging AHRQ Networks

Other (http://www.ahrq.gov/fund/granarch.htm#RFA)

New Laws: Patient Protection and Affordable Care Act

Demonstration Projects for Quality Improvement

“demonstration” — 312 mentions “pilot” — 80 mentions#

Creation of the Center for Medicare and Medicaid Innovation (CMI)

National Strategy for Quality Improvement More (see CRS report)

# http://e-caremanagement.com/pilots-demonstrations-innovation-in-the-ppaca-healthcare-reform-legislation/http://www.aamc.org/reform/summary/ph.pdf

CHIPRA – CMS Quality Demonstration

Grants to States Aims:A) Experiment with, and evaluate the use of new measures for

quality of Medicaid/CHIP children's health care;B) Promote the use of HIT for the delivery of care for children

covered by Medicaid/CHIP;C) Evaluate provider-based models which improve the delivery

of Medicaid/CHIP children's health care services; orD) Demonstrate the impact of the model Electronic Health

Record format for children (developed and disseminated under section 401(f)) on improving pediatric health, and pediatric health care quality, as well as reducing health care costs.

E) Broad systems approaches/medical home 10 awards made Feb. 2010 to individual States and

consortia of States National Evaluation (in planning stage –AHRQ has lead) http://www.cms.gov/CHIPRA/15_StateDemo.asp

AHRQ Funding to Test and Disseminate

Strategies to Improve Quality and Patient Safety

ACTION II Program Announcements Evaluation of Spread of the Keystone projects (health care

associated infections) Assessing the Evidence for Context-Sensitive Effectiveness

and Safety of Patient Safety Practices: Developing Criteria (forthcoming report)

Co-sponsorship of the NIH Science of D & I conference (2011)

Innovations Clearinghouse Knowledge Transfer Projects Value Exchanges PAR 08-136

What are We Trying to Learn from all This Work?

Answer: Not Only the What, but the How and the Why of Healthcare Quality Improvement

Current State of the Science: QII and

Evaluation Designs—A Personal View

Problem identification (vaguely defined) Theory of action to solve the problem (often omitted, vague or in-name-only) Interventions

– vaguely described; – not replicable; – conceptual confusion between “intervention” and “implementation”

Focus on internal validity and related designs – Context of intervention/ implementation processes:

not considered or considered post hoc and descriptive/idiosyncratic effects of context/variation in context not considered in assessing results and variation in

results “qualitative” research does not mean standards of the qualitative research field Lack of validated measures of contextual variables (leadership, culture, teamwork,

resources)– For publication: design driven by clinical hierarchy of evidence standards (RCTs at

patient level) – If not for publication:

threats to internal validity rarely considered; post only studies or simple pre-post without comparisons; implications for knowledge base not widely recognized.

Few comparison studies (one QI intervention to another; multiple settings)

Specific Example: Context– Multiple

potential influences on QII Results

External factors – e.g.: Regulatory requirement Payments or penalties Local sentinel event

Structural/organizational characteristics (organization site)

Culture, Teamwork, Leadership Implementation Processes and Tools

– Staff education and training– Audit and feedback

Source: Shekelle, Pronovost, and Wachter, Contract Report to AHRQ, Contract #HHSA-290-2009-10001C, forthcoming.

Specific Example: Quantitative Approaches

to Context Heterogeneity - Progress Premise: Context often moderates intervention effectiveness

This moderation effect can be represented statistically through the “intervention x context” interaction:

– Yi = b0 + b1 × Ti + b2 × Ci + b12 × Ti × Ci + εi, where i denotes the unit of analysis (usually the various sites in the study, but can also be dyads of sites in matched comparisons), Yi

denotes the outcome measure, Ti denotes the intervention status (Ti=1 for intervention, Ti=0 for control), Ci denotes the contextual factor, Ti × Ci denotes the “intervention × context” interaction, εi denotes random error, b0 denotes the intercept for the model, b1 denotes the main effect for the intervention, b2 denotes the main effect for the contextual factor, and b12 denotes the moderation effect for the contextual factor, i.e., the influence of the contextual factor on intervention effectiveness.

Looks like progress, assuming we can quantify contextual variables

Source: Shekelle, Pronovost, and Wachter, Contract Report to AHRQ, Contract #HHSA-290-2009-10001C, Chapter 12, Special contribution from Naihua Duan, Columbia University, New York, New York forthcoming

Meta-Science Issues

IRBs Study sections Promotion and Tenure Little collaborative research (understanding

effects of variations in context) Conflict between research and evaluation Limited knowledge of evaluation “how to”?

Research in Implementation and Change While

Improving Quality – The Answer?

PAR -08-136– A relatively small attempt to specifically try to understand

the how and why in a rigorous way $300K/year

– No dedicated pot of funds at AHRQ – highly competitive This session:

– Two examples – in process – methods and interim results, not definitive findings

– Interactive discussion – What would you add? What else do you need to know?