documenting and reporting implementation research: new expectations, new opportunities brian s....
TRANSCRIPT
Documenting and reportingimplementation research:
New expectations, new opportunities
Brian S. Mittman, PhDVA Center for Implementation Practice
and Research Support (CIPRS)
January 6, 2009
Quality Enhancement Research Initiative
1. Background, motivation and goals for improved reporting
2. Ongoing initiatives to strengthen reporting
3. Recommendations and future possibilities
Outline
Research documentation and reporting:Current practices and problems
• Increasing focus on shortcomings in health research documentation and reporting– Variable reporting: different measures, design elements,
analysis standards– Incomplete reporting: publication bias, selective reporting
of outcomes (to favor evidence of efficacy)• Challenges for meta-analyses, replication, cumulative
knowledge
• Concerns over return on societal (public) investment in research, validity of conclusions (policy/practice implications)
Reporting problems in implementation science
• Each of these problems exists in implementation research, and many are more severe than in mainstream clinical research– Selective reporting and publication bias– Variable, inconsistent reporting– Emphasis on impact reporting only– Limited “return on investment” and value of research
1. Background, motivation and goals for improved reporting
2. Ongoing initiatives to strengthen reporting
3. Recommendations and future possibilities
Outline
Major initiatives to strengthen reporting
• CONSORT Statement, extensions and similar statements (e.g., SQUIRE guidelines; EQUATOR Network)
• ClinicalTrials.gov and other registries
• Ottawa statements and protocol publication
• External validity reporting
• Intervention reporting
• Trial bank initiative
CONSORT; EQUATOR Network
• Standardized format, structure, contents of reports of specific types of studies (experimental, observational, reviews, economic)
• Consistent reporting of core features (design, methods, results)
• “The EQUATOR Network is an international initiative that seeks to improve the quality of scientific publications by promoting transparent and accurate reporting of health research.” (www.equator-network.org)
• Updated SQUIRE guidelines are a recent addition
Trial registries
• Advance registration of clinical trials in a public registry reduces the likelihood of (un-detected) selective publication
• The International Committee of Medical Journal Editors (ICMJE) (and most journals) require a priori trial registration as a condition for publication
• Public Law 110-85, Title VIII expands the types of clinical trials that must be registered in ClinicalTrials.gov, increases the number of data elements that must be submitted, and requires submission of results data
Protocol publication
• The Ottawa Statement on trial registration is a consensus document that aims to guide the implementation of global trial registration (http://ottawagroup.ohri.ca/)
• The Statement addressed fundamental principles for trial registration (Part 1) and operationalization of protocol registration (Part 2) and results reporting (Part 3)
External validity; intervention details
• Several recent articles and editorials promote and guide enhanced reporting of external validity to increase the “relevance, generalizability, and applicability” (i.e., value in policy and practice) of health research (Green, Glasgow)
• Other initiatives include ongoing efforts to stimulate more complete, consistent reporting of implementation programs and interventions; contextual factors; etc.
The Trial Bank Project
• “The prose reporting of trial information could be powerfully augmented by a computable repository of trial information --a global trial bank. Unlike prose, computable information is structured and coded for computation and allows the use of advanced information technologies for knowledge management.”
• “The Global Trial Bank seeks to make available open-access and computable peer-reviewed results from all clinical trials conducted worldwide, regardless of whether the results are favorable or not, to provide an unbiased knowledge base for improving human health. “
1. Background, motivation and goals for improved reporting
2. Ongoing initiatives to strengthen reporting
3. Recommendations and future possibilities
Outline
Comprehensive documentation plan
• Systematic, comprehensive documentation plan– Protocol– Baseline/interim results– Main findings– Supporting findings– Secondary analyses
• Required elements of each component– Protocol includes current systematic review(s), power
calculations for contribution to existing evidence base– Main findings linked to protocol; report analyses for all (vs.
selected) outcomes– Supporting analyses explaining findings (mechanisms of action)
Rationalizing publication
• Distinguish types of evidence/knowledge– Policy/practice relevant (systematic reviews)– Primary evidence (main findings)– Explanatory evidence (process evaluations)
• Classify types of evidence, knowledge– Clinical domains– Health services domains
• Allocate types of evidence and knowledge to distinct publication outlets (review journals, researcher journals/on-line resources, trial banks, etc.)
• Expand registries to improve access to evidence, research output
• Link funding programs to registries, trial banks
Successful Implementation Manuscripts: Beyond the Basics
Jeroan J. Allison, MD, MS
Catarina I. Kiefe, PhD, MD
Co-Editors-in-Chief, Medical Care
January 6, 2008
VA Implementation Research Seminar
Conceptual Model for Implementation Research
Salanitro A. et al. In Essentials of Clinical Research. New York: Springer, 2008.
After carefully studying your manuscript, I am sorry to report that we will not be able to
publish it in Medical Care.
Medical Care currently receives over 800 original submissions each year, and we
eventually accept less than 20% for publication.
Because of the increasing competition for journal space, we are not able to assign a
high priority to all worthy manuscripts.
Current Publication Practices May Distort Science
• Problems– Preferential publication of extreme results– Artificial scarcity of publication outlets
• Potential Solutions– Increase publication preference for negative studies– Base acceptance on methodological quality– Enhance post-publication review– Dissociate academic advancement from journal
branding
Young N., et al. PLoS Medicine. 2008; 5(10): e201
“Why, sometimes I’ve believed as many as six impossible things before breakfast.”
L. Carol, Alice in Wonderland
Fundamental Assumption
• Interesting question• High-quality data• Appropriate analysis• Crisp presentation• Important conclusions
Necessary but not sufficient!
Potential Publication Barriers for Implementation Research
• Small sample size
• Local population
• Lack of “strong” study design
• Failure to distinguish care improvement from generalizable knowledge
• Competition with other types of research
Beyond the Basics
• Establish dialogue with readership
• Frame manuscript to show substance
• Demonstrate impact and relevance
• Be persistent
Dialogue with Readership
• Understand the target journal• Establish continuity with readership• Review relevant articles from target journal
– Journal self-citation– Inappropriate impact factor padding
• World Association of Medical Editors rich discussion
www.wame.org
Medical Care Vision Statement
A Message from the New Medical Care Co-Editors-in-Chief: Building Upon a Legacy of Scholarship and Impact. Medical Care. 2007; 45(1): 1-2.
Health Care Delivery*•Safe•Effective•Patient-centered •Timely•Efficient •Equitable
Health Care Populations
Methods
*From the Institute of Medicine in Crossing the Quality Chasm: A New Health System for the 21st Century, 2001.
Substance: Avoid the Least Publishable Unit
Exceptions• Same message multiple audiences• Repeated methods from large studies
OriginalSubmission
Plagiarism
Duplicate Publication
www.famousplagiarists.com
Substance: Avoid the Least Publishable Unit
Recently submitted manuscript• Concluded intervention equally effective
by race/ethnicity• Reviewer comment: focus on “broader
perspective of the impact of the intervention”
• MedLine search by editor: request fulfilled in another journal
Impact: Advance the Paradigm
• Address the “so what” question– Varies from journal to journal– Policy relevance for Medical Care
• Understand the cutting edge– Health Disparities: from understanding to
change– Risk Adjustment: statistical vs. tangible
meaningT. Kuhn. The Structure of Scientific Revolutions. Chicago: University of Chicago Press, 1962.
“So-What” Issues for Implementation Research
• Linking “basic” human/social sciences with process, structure, and outcomes
• Detangling the confusing picture of practice-change interventions
• Exploring unexpected consequences of emerging paradigms (e.g., P4P)
• Demonstrating value by mixed-methods
“So-What” Issues for Implementation Research
• Measuring the right outcomes
• Testing interventions at interface of patient, doctor, system, and community
• Promoting bi-directional information flow along implementation spectrum
Impact: Advance the Paradigm
• Use bounded speculation
• Focus on the core message
Watson, J. D. and Crick, F. H. 1953. Molecular structure of nucleic acids. Nature 171: 737738. Quoted in Sand-Jensen, 2005. Oikos 116: 723-727
It has not escaped our notice that the specific pairing we have postulated immediately suggests a possible copying mechanism for the genetic material *
Why Some Ideas Survive and Others Die
• Simple: focus on core messagePersonal lesson learned repeatedly
• Unexpected: unfold with mystery• Credible• ...
C. Heath, D. Heath. Made to Stick: Why some ideas survive and others die. New York: Random House, 2007.
Why Some Papers are Published and Others Die
• Simple: focus on core message– Personal lesson learned repeatedly
• Unexpected: unfold with mystery• Credible• ...
C. Heath, D. Heath. Made to Stick: Why some ideas survive and others die. New York: Random House, 2007.
The Value of Persistence
• Rejected manuscript– Poor quality– Poor fit– Bad luck
• Peer-review system– Best available– Not Perfect– Notable shortcomings
• Moniz won Nobel Prize in 1949• Respected peer-review publications touted lobotomy for headaches,
anxiety, delusions, confusion
• 0.20 x 5 = 1• No. journals in Medline, March 2007: 5,197
Take Home Points
• Increasing competition for top journal space
• More than science and traditional aspects of quality– Establish dialogue with readership– Address “cutting-edge” questions– Frame manuscript to show substance– Demonstrate impact and relevance– Be persistent
The real trouble with this world of ours is not that it is an unreasonable world, nor even that it is a reasonable one. The commonest kind of trouble is that is it nearly reasonable, but not quite.
G. Chesterton
I am pleased to inform you that your work has now been accepted for
publication in Medical Care.
All manuscript materials will be forwarded immediately to the
production staff for placement in an upcoming issue.