the use and evaluation of experiments in health care delivery amanda kowalski associate professor of...
TRANSCRIPT
![Page 1: The Use and Evaluation of Experiments in Health Care Delivery Amanda Kowalski Associate Professor of Economics Department of Economics, Yale University](https://reader036.vdocument.in/reader036/viewer/2022062517/56649e925503460f94b98658/html5/thumbnails/1.jpg)
The Use and Evaluation of Experiments in Health Care DeliveryAmanda KowalskiAssociate Professor of EconomicsDepartment of Economics, Yale University
September 26, 2015
![Page 2: The Use and Evaluation of Experiments in Health Care Delivery Amanda Kowalski Associate Professor of Economics Department of Economics, Yale University](https://reader036.vdocument.in/reader036/viewer/2022062517/56649e925503460f94b98658/html5/thumbnails/2.jpg)
What are the questions that YOU want to answer?
![Page 3: The Use and Evaluation of Experiments in Health Care Delivery Amanda Kowalski Associate Professor of Economics Department of Economics, Yale University](https://reader036.vdocument.in/reader036/viewer/2022062517/56649e925503460f94b98658/html5/thumbnails/3.jpg)
How can I help you to answer them?
![Page 4: The Use and Evaluation of Experiments in Health Care Delivery Amanda Kowalski Associate Professor of Economics Department of Economics, Yale University](https://reader036.vdocument.in/reader036/viewer/2022062517/56649e925503460f94b98658/html5/thumbnails/4.jpg)
A Call to Action
http://www.sciencemag.org/content/347/6223/720.full
![Page 5: The Use and Evaluation of Experiments in Health Care Delivery Amanda Kowalski Associate Professor of Economics Department of Economics, Yale University](https://reader036.vdocument.in/reader036/viewer/2022062517/56649e925503460f94b98658/html5/thumbnails/5.jpg)
![Page 6: The Use and Evaluation of Experiments in Health Care Delivery Amanda Kowalski Associate Professor of Economics Department of Economics, Yale University](https://reader036.vdocument.in/reader036/viewer/2022062517/56649e925503460f94b98658/html5/thumbnails/6.jpg)
Heard in the Trenches:Barriers to Randomization
![Page 7: The Use and Evaluation of Experiments in Health Care Delivery Amanda Kowalski Associate Professor of Economics Department of Economics, Yale University](https://reader036.vdocument.in/reader036/viewer/2022062517/56649e925503460f94b98658/html5/thumbnails/7.jpg)
“Why randomize?”
Way to mitigate influence of confounding factors If you just compare treated group to a non-treated
group, there could be other factors that produce different outcomes
If you just compare treated group after intervention to treated group before intervention, there could be other factors that changed over time
![Page 8: The Use and Evaluation of Experiments in Health Care Delivery Amanda Kowalski Associate Professor of Economics Department of Economics, Yale University](https://reader036.vdocument.in/reader036/viewer/2022062517/56649e925503460f94b98658/html5/thumbnails/8.jpg)
“Can’t you just analyze data collected after the intervention?” It IS important to examine existing
administrative data
Hard to learn about causality if you only have data on the treatment group after the intervention
![Page 9: The Use and Evaluation of Experiments in Health Care Delivery Amanda Kowalski Associate Professor of Economics Department of Economics, Yale University](https://reader036.vdocument.in/reader036/viewer/2022062517/56649e925503460f94b98658/html5/thumbnails/9.jpg)
“Is randomization fair?”We only randomize if we don’t know if the
intervention will work – “equipoise”
Randomization standard in clinical trials for medical interventions
When resources are scarce, randomization can be a fair way to allocate them
![Page 10: The Use and Evaluation of Experiments in Health Care Delivery Amanda Kowalski Associate Professor of Economics Department of Economics, Yale University](https://reader036.vdocument.in/reader036/viewer/2022062517/56649e925503460f94b98658/html5/thumbnails/10.jpg)
“Is randomization fair?”Example: Oregon Health Insurance experimentSome funds to expand Medicaid coverage, but
not enough to expand coverage to all interested parties
Held a lottery in 2008
![Page 11: The Use and Evaluation of Experiments in Health Care Delivery Amanda Kowalski Associate Professor of Economics Department of Economics, Yale University](https://reader036.vdocument.in/reader036/viewer/2022062517/56649e925503460f94b98658/html5/thumbnails/11.jpg)
“Is randomization fair?”Example: Oregon Health Insurance experiment
![Page 12: The Use and Evaluation of Experiments in Health Care Delivery Amanda Kowalski Associate Professor of Economics Department of Economics, Yale University](https://reader036.vdocument.in/reader036/viewer/2022062517/56649e925503460f94b98658/html5/thumbnails/12.jpg)
“Is randomization fair?”Example: Oregon Health Insurance experimentKey findings:
Increased health care utilization Emergency room utilization increased
Decrease in depression No changes to physical health
Reduced financial strain No discernable impact on labor market outcomes
![Page 13: The Use and Evaluation of Experiments in Health Care Delivery Amanda Kowalski Associate Professor of Economics Department of Economics, Yale University](https://reader036.vdocument.in/reader036/viewer/2022062517/56649e925503460f94b98658/html5/thumbnails/13.jpg)
“But some large-scale questions can’t be studied with randomization!”For example, some initiatives like bundled
payments might have larger impacts if they are implemented more broadly
Could also randomize fraction of population affected across different sites
![Page 14: The Use and Evaluation of Experiments in Health Care Delivery Amanda Kowalski Associate Professor of Economics Department of Economics, Yale University](https://reader036.vdocument.in/reader036/viewer/2022062517/56649e925503460f94b98658/html5/thumbnails/14.jpg)
“I’m too busy implementing this initiative to think about anything else!”Randomization can be a seamless part of
implementation
After implementation is too late
![Page 15: The Use and Evaluation of Experiments in Health Care Delivery Amanda Kowalski Associate Professor of Economics Department of Economics, Yale University](https://reader036.vdocument.in/reader036/viewer/2022062517/56649e925503460f94b98658/html5/thumbnails/15.jpg)
“I’m afraid to find out that what I have done does not work!”Health care industry full of altruistic people
who want to improve quality, increase access, and decrease cost
If your intervention does not work, can try something else next
If your intervention does work, evidence indicating so can be useful to others, broadening the impact of your work
![Page 16: The Use and Evaluation of Experiments in Health Care Delivery Amanda Kowalski Associate Professor of Economics Department of Economics, Yale University](https://reader036.vdocument.in/reader036/viewer/2022062517/56649e925503460f94b98658/html5/thumbnails/16.jpg)
“Will the implementation be costly?” Clinical trials often costly
Subject recruitment Informed consent
Randomized experiments in health care delivery Subjects already in system Consent waived
Costs of designing experiment
Costs of implementing randomization
Costs of collecting data – could focus on existing
Costs of analyzing data
![Page 17: The Use and Evaluation of Experiments in Health Care Delivery Amanda Kowalski Associate Professor of Economics Department of Economics, Yale University](https://reader036.vdocument.in/reader036/viewer/2022062517/56649e925503460f94b98658/html5/thumbnails/17.jpg)
“How will results be disseminated?”Results will be published regardless of outcome
Institutional partner can opt for anonymity before publication
![Page 18: The Use and Evaluation of Experiments in Health Care Delivery Amanda Kowalski Associate Professor of Economics Department of Economics, Yale University](https://reader036.vdocument.in/reader036/viewer/2022062517/56649e925503460f94b98658/html5/thumbnails/18.jpg)
“When is a good time to get started?”Before implementing a new intervention
Baseline data can be collected Enriches comparison of treatment to control Ensures that outcomes can be measured
Program probably not rolled out to everyone at the same time anyway
![Page 19: The Use and Evaluation of Experiments in Health Care Delivery Amanda Kowalski Associate Professor of Economics Department of Economics, Yale University](https://reader036.vdocument.in/reader036/viewer/2022062517/56649e925503460f94b98658/html5/thumbnails/19.jpg)
“Which interventions are best studied with randomization?”Potential for large, detectable outcomes
But outcomes are not known
Large potential number of subjects Increases statistical power
Interventions that would be implemented anyway Increases real-world applicability Successful implementation paves the way for
future randomization of other initiatives
![Page 20: The Use and Evaluation of Experiments in Health Care Delivery Amanda Kowalski Associate Professor of Economics Department of Economics, Yale University](https://reader036.vdocument.in/reader036/viewer/2022062517/56649e925503460f94b98658/html5/thumbnails/20.jpg)
“What are the advantages of partnering with an economist?”Statistical techniques
More subtle than comparing treatment group to control group
Dissemination Potential to reach different audience
Results prepared by an independent entity potentially more impartial
![Page 21: The Use and Evaluation of Experiments in Health Care Delivery Amanda Kowalski Associate Professor of Economics Department of Economics, Yale University](https://reader036.vdocument.in/reader036/viewer/2022062517/56649e925503460f94b98658/html5/thumbnails/21.jpg)
“I’m on board with randomization. What’s next?”What is the problem to be addressed?
What administrative data are available and how can they be accessed?
Who will be the implementing partners? C-level advocate to push project through Administrative contact for day-to-day
![Page 22: The Use and Evaluation of Experiments in Health Care Delivery Amanda Kowalski Associate Professor of Economics Department of Economics, Yale University](https://reader036.vdocument.in/reader036/viewer/2022062517/56649e925503460f94b98658/html5/thumbnails/22.jpg)
Let’s talk further!
![Page 23: The Use and Evaluation of Experiments in Health Care Delivery Amanda Kowalski Associate Professor of Economics Department of Economics, Yale University](https://reader036.vdocument.in/reader036/viewer/2022062517/56649e925503460f94b98658/html5/thumbnails/23.jpg)
Let’s talk further!http://www.econ.yale.edu/~ak669/
research.html
Slides on my website, video will be posted
…Tell your friends to examine slides, watch this presentation, propose an idea, and contact me!
![Page 24: The Use and Evaluation of Experiments in Health Care Delivery Amanda Kowalski Associate Professor of Economics Department of Economics, Yale University](https://reader036.vdocument.in/reader036/viewer/2022062517/56649e925503460f94b98658/html5/thumbnails/24.jpg)
What are the questions that YOU want to answer?