If We Want More EvidenceIf We Want More Evidence--Based Practice, We Based Practice, We Need More PracticeNeed More Practice--Based EvidenceBased Evidence

CDC Oral Health WorkshopCDC Oral Health WorkshopAtlantaAtlanta

October 24, 2007October 24, 2007

Lawrence W. GreenLawrence W. GreenUniversity of California at San FranciscoUniversity of California at San Francisco

PRECEDEPRECEDE--PROCEED & REPROCEED & RE--AIM AIM as Frameworks foras Frameworks for

PracticePractice--Based Planning and EvaluationBased Planning and Evaluation

The Need to Expand Upon GuidelinesThe Need to Expand Upon Guidelines

EvidenceEvidence--based public health and medicine based public health and medicine highlight gaps between science and practicehighlight gaps between science and practice

MetaMeta--analyses and systematic reviews that analyses and systematic reviews that produce practice guidelines typically:produce practice guidelines typically:

concentrate on concentrate on strength of evidencestrength of evidence::typically from randomized controlled trialstypically from randomized controlled trials

discount discount weight of evidenceweight of evidence -- indirect evidence from:indirect evidence from:Quasi or nonQuasi or non--experimental studies/dataexperimental studies/dataPractitioner experiences and evaluationsPractitioner experiences and evaluationsCumulative wisdom from systematic analysis of these Cumulative wisdom from systematic analysis of these and understanding situations in which they would be and understanding situations in which they would be appliedapplied

The World View of PractitionersThe World View of Practitioners

Many nonMany non--scientists give greater credence scientists give greater credence to the weight of evidence:to the weight of evidence:

Judicial and regulatory agencies Judicial and regulatory agencies No single study of behavior or social change can No single study of behavior or social change can establish causationestablish causation

Practitioners Practitioners Learn from each other, build on previous Learn from each other, build on previous experience, look for similarities in contextexperience, look for similarities in contextOral health: 16 objectives Oral health: 16 objectives vsvs 3 TFCPS guidelines3 TFCPS guidelines

What fuels these differences in outlook?What fuels these differences in outlook?

The Fuel for Two PerspectivesThe Fuel for Two PerspectivesScientists typically emphasize internal validity over Scientists typically emphasize internal validity over external validity; practitioners need both:external validity; practitioners need both:

Internal validityInternal validity (the essence of rigor)(the essence of rigor)Are we measuring what we purport to measure?Are we measuring what we purport to measure?Essential for knowing what is responsible for our resultsEssential for knowing what is responsible for our resultsWithout internal validity, external validity is irrelevant Without internal validity, external validity is irrelevant

External validityExternal validity (the essence of relevance)(the essence of relevance)GeneralizabilityGeneralizability (important to policy(important to policy--/decision/decision--makers):makers):

How applicable is this to the range of realHow applicable is this to the range of real--world settings and world settings and situations?situations?

SpecificitySpecificity (important to practitioners, communities):(important to practitioners, communities):Will following guidelines based on highly controlled studies, wiWill following guidelines based on highly controlled studies, with th select populations, work in select populations, work in mymy situation? situation?

Green LW & Glasgow RE. Evaluating the relevance, generalization, and applicability of research: Issues in Translation methodology. Eval Health Professions. 2006.

The ChallengesThe Challenges……1.1. The evidenceThe evidence--based practice movement needs based practice movement needs

to direct some energy and resources to to direct some energy and resources to developing and applying criteria and measures developing and applying criteria and measures of external validity.of external validity.

2.2. ““Where did the field get the idea that evidence Where did the field get the idea that evidence of an interventionof an intervention’’s efficacy from carefully s efficacy from carefully controlled trials could be generalized as controlled trials could be generalized as thethe ““best best practicepractice”” for widely varied populations and for widely varied populations and settings?settings?””----Green LW. From research to Green LW. From research to ““best practicesbest practices”” in other settings in other settings and populations. and populations. Amer J Health Amer J Health BehavBehav 2001;25:1652001;25:165--178.178.www.ajhb.org/2001/25www.ajhb.org/2001/25--33--22. .

““Law of halvesLaw of halves””e.g., e.g., ULTIMATE IMPACT OF MAGIC PILLULTIMATE IMPACT OF MAGIC PILL

50% of Clinics Use Adoption 50%

50% of Clinicians Prescribe Adoption 25%

50% of Patients Accept Medication Reach 12.5%

50% Follow Regimen Correctly Implementation 6.2%

50% of Those Taking Correctly Benefit Effectiveness 3.2%

50% Continue to Benefit After 6 Months Maintenance 1.6%

Dissemination Step Concept % Impacted

Suggested ReferencesSuggested ReferencesGlasgow RE, Vogt TM, Boles SM. Evaluating the Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: public health impact of health promotion interventions: The REThe RE--AIM framework. AIM framework. Am J Public HealthAm J Public Health. 1999; . 1999; 89:132289:1322--1327.1327.Green LW. From research to Green LW. From research to ““best practicesbest practices”” in other in other settings and populations. settings and populations. Am J Health Am J Health BehavBehav. 2001; . 2001; 25:16525:165--178.178.Green LW & Glasgow RE. Evaluating the relevance, Green LW & Glasgow RE. Evaluating the relevance, generalization, and applicability of research: Issues in generalization, and applicability of research: Issues in Translation methodology. Translation methodology. EvalEval Health ProfessionsHealth Professions. 2006. . 2006. Green LW & Kreuter MW. Green LW & Kreuter MW. Health Program Planning: An Health Program Planning: An Educational and Ecological Approach Educational and Ecological Approach (4(4thth edition). Boston: edition). Boston: McGraw Hill, 2005. McGraw Hill, 2005.

Design Issues for Translating Research Design Issues for Translating Research to Practice & Policy (& vice versa)to Practice & Policy (& vice versa)

Making practice more evidenceMaking practice more evidence--basedbasedSetting research & evaluation prioritiesSetting research & evaluation prioritiesMaking research findings actionable, Making research findings actionable, usable, relevant (to whom?)usable, relevant (to whom?)Disseminating & translating research to Disseminating & translating research to local circumstances, cultures, and local circumstances, cultures, and personnelpersonnelMaking evidence more practiceMaking evidence more practice--basedbased

R

The “Pipeline” Conceptualizationof Research to Practice*

Peer ReviewOf Grants

PublicationPriorities &Peer Review

ResearchSynthesis

Guidelines forEvidence-BasedPractice

Academic appointments,promotion, & tenure criteria

Funding; Consumerneeds, demands;Local practicecircumstances;Professionaldiscretion; Credibility & fit ofthe evidence.

-

Evidence-basedmedicine movement

Practice

Priorities for Research Funding

*Green, L.W. From research to “best practices” in other settings and populations. Am J Health Behavior 25:165-178, April-May 2001. Full text: www.ajhb.org/25-3.htm

Publication

Bibliographic databases

Submission

Reviews, guidelines, textbook

Negative results

variable

0.3 year

6. 0 - 13.0 years50%

46%

18%

35%

0.6 year

0.5 year

9.3 years

Dickersin, 1987

Koren, 1989

Balas, 1995

Poynard, 1985

Kumar, 1992

Kumar, 1992

Poyer, 1982

Antman, 1992

Negative results

Lack of numbers

Expertopinion

Inconsistentindexing

17:14

Original research

Acceptance

Implementation

““It takes 17 years to turn 14 per cent of original researchIt takes 17 years to turn 14 per cent of original researchto the benefit of patient careto the benefit of patient care””

Bridging the Chasm as Conceived Bridging the Chasm as Conceived by the U.S. Translation Agency* by the U.S. Translation Agency*

Practice is Practice is We want it toWe want it toherehere be herebe here

TRIPTRIP

ImplementationImplementation

DiffusionDiffusion

AdoptionAdoptionEducationEducation

InnovationInnovation

*Carolyn Clancy. Agency for Healthcare Research & Quality2003.

Reminiscent of the “Fallacy of the EmptyVessel” from earlyhealth education

EvidenceEvidence--Based Public HealthBased Public Health

Information of importance to communitychoice that is not even potentially of “evidence-based type.”

C A

B

Area where there is currently “good evidence-based” informationof importance to communitiesin making choices.

Information of importance to communitychoice that is potentially of “evidence-based” type.

A “Good evidence”

B Potential for “good evidence”

C Information of potential importance to communities in making health choices

3 Conceptualizations of the Gap 3 Conceptualizations of the Gap Between Research & PracticeBetween Research & PracticePractitioners need to receive the lessons of Practitioners need to receive the lessons of research and put them into practice.research and put them into practice.Research and practice are entirely separate Research and practice are entirely separate disciplines and each must develop their own disciplines and each must develop their own answers to their own problemsanswers to their own problemsResearch and practice have complementary Research and practice have complementary perspectives and skills that need to be used perspectives and skills that need to be used together to address the real need, together to address the real need, collaborative knowledge production. collaborative knowledge production. Add to this the need to include the patientAdd to this the need to include the patient’’s s perspective. Whose perspective prevails?perspective. Whose perspective prevails?

Van De Ven A, Johnson P. Knowledge for theory and practice. Academy of Management Review. 2006;31(4).

The Internal Validity Drift of Health Sciences The Internal Validity Drift of Health Sciences Evidence Evidence ““Lost in TranslationLost in Translation””

EvidenceEvidence--based medicine movement taken to based medicine movement taken to scale in community healthscale in community healthThe peer review preferences for experimental The peer review preferences for experimental control and certainty of causationcontrol and certainty of causationThe publishing preferences for The publishing preferences for RCTsRCTs and and positive resultspositive resultsThe limitations of print space driving out The limitations of print space driving out richer description of interventions, protocols, richer description of interventions, protocols, procedural lessons, subgroup variationsprocedural lessons, subgroup variationsBut a more But a more ““naturalnatural”” type of public health type of public health evidence has greater influence on multievidence has greater influence on multi--level level program planning, practice & policyprogram planning, practice & policy……

Issues for EvidenceIssues for Evidence--Based Practice Based Practice and Translating Research to Practiceand Translating Research to Practice

Making practice more evidenceMaking practice more evidence--basedbasedSetting research prioritiesSetting research prioritiesMaking research findings actionable, Making research findings actionable, usable, relevant within settingsusable, relevant within settingsTranslating research Translating research from outsidefrom outside to local to local circumstances, cultures, personnelcircumstances, cultures, personnelMaking evidence more practiceMaking evidence more practice--basedbased

Some Benefits of Participatory Some Benefits of Participatory Research in PracticeResearch in Practice--Based EvidenceBased Evidence

Results are relevant to interests, Results are relevant to interests, circumstances, and needs of those who circumstances, and needs of those who would apply themwould apply themResults are more immediately actionable in Results are more immediately actionable in local situations for people and/or practitionerslocal situations for people and/or practitionersGeneralizable findings more credible to Generalizable findings more credible to people, practitioners and policy makers people, practitioners and policy makers elsewhere because they were generated in elsewhere because they were generated in partnership with people like themselvespartnership with people like themselvesHelps to reframe issues from health behavior Helps to reframe issues from health behavior of individuals to encompass system and of individuals to encompass system and structural issues.structural issues.

Green LW, Mercer SL. Am J Public Health Dec. 2001.

Definition and Standards of Definition and Standards of Participatory Research for Health*Participatory Research for Health*

Systematic investigationSystematic investigation……Actively involving people in a coActively involving people in a co--learning learning

processprocess……For the purpose of action conducive to For the purpose of action conducive to

health**health**

----not just involving people more intensively not just involving people more intensively as as subjectssubjects of research or evaluationof research or evaluation

*Green, George, Daniel, et al., Participatory Research…Ottawa: Royal Society of Canada, 1997. www.lgreen.net/guidelines.html

Layers of Collaboration Layers of Collaboration in Participatory Researchin Participatory Research

Basic & appliedBasic & applied---------->>

Action researchAction research---------------------->>

PAR in practicePAR in practice---------------------------------------------------->>

PAR on health needs of PAR on health needs of

patients, familiespatients, families---------------------------------------------------------->>

PAR on other needsPAR on other needs---------------------------------------------------->>

Traditionallydefinedresearchers

Practitioners,Practitioners,Service Service providersproviders

Patients,Families,Community

*Green et al., Participatory Research…Ottawa: Royal Society of Canada, 1997. www.lgreen.net/guidelines.html

Closing the Gaps Between Population & Closing the Gaps Between Population & ScientistsScientists’’ Perception of Needs, and Perception of Needs, and Policy MakersPolicy Makers’’ AssessmentsAssessments

AA

*Green & Kreuter, Health Program Planning: An Educational and EcologicalApproach, 4th ed., New York: McGraw-Hill, 2005.

“Actualneeds”

Resources,Resources,feasibilities,feasibilities,

policypolicy

People’sperceived needs,

prioritiesA

Reconciling Perceived Needs, Reconciling Perceived Needs, ““Actual Needs,Actual Needs,”” & Resources& Resources

Participatory ResearchAction

Community organization& capacity development(e.g., Chris Economos & Boyd Swinburn trials)

Health EducationPeople’s

perceived needs,priorities

“Actualneeds”

Resources,feasibilities,

policy

Issues for EvidenceIssues for Evidence--Based Practice Based Practice and Translating Research to Practiceand Translating Research to Practice

Making practice more evidenceMaking practice more evidence--basedbasedSetting research prioritiesSetting research prioritiesMaking research findings actionable, Making research findings actionable, usable, relevant: participatory researchusable, relevant: participatory researchTranslating research to local Translating research to local circumstances: External validity & circumstances: External validity & ““fidelityfidelity””vsvs adaptationadaptationMaking evidence more practiceMaking evidence more practice--basedbased

Building Policy and Practice from Building Policy and Practice from Evidence + TheoryEvidence + Theory

Not starting with theory and looking for problems Not starting with theory and looking for problems on which to test them, but starting with problems on which to test them, but starting with problems and looking for theories to help us solve them*and looking for theories to help us solve them*Evidence on solutions generalizes to other Evidence on solutions generalizes to other circumstances, settings, & populations in the circumstances, settings, & populations in the form of either replication or theoryform of either replication or theoryReplication is limited by the infinite number of Replication is limited by the infinite number of contextcontext--population combinationspopulation combinations"In theory, theory and practice are the same "In theory, theory and practice are the same thing. In practice they're not..thing. In practice they're not..““ --Jan L.A. van de Jan L.A. van de SnepscheutSnepscheut““All models are wrong. Some are usefulAll models are wrong. Some are useful”” ----BoxBox

*Green LW. Public health asks of systems science… Amer J Public Health 96, March 2006.

““FidelityFidelity”” vsvs AdaptationAdaptation

Researchers test an intervention for its efficacyResearchers test an intervention for its efficacyRigorous test qualifies the study for official lists Rigorous test qualifies the study for official lists of of ““evidenceevidence--based practicesbased practices”” and guidelinesand guidelinesPractitioners try to incorporate it into their Practitioners try to incorporate it into their programsprogramsPoor fit produces failure of programPoor fit produces failure of programPractitioners are blamed for not implementing Practitioners are blamed for not implementing with with ““fidelityfidelity””Now buy the producersNow buy the producers’’ training programtraining program

Issues for EvidenceIssues for Evidence--Based Practice and Based Practice and Translating Research to PracticeTranslating Research to Practice

Making practice more evidenceMaking practice more evidence--basedbasedSetting research prioritiesSetting research prioritiesMaking research findings actionable, usable, Making research findings actionable, usable, relevant: participatory researchrelevant: participatory researchTranslating outside research to local Translating outside research to local circumstancescircumstancesMaking evidence more practiceMaking evidence more practice--basedbased

Efficacy vs. EffectivenessEfficacy vs. Effectiveness: :

EfficacyEfficacy. The tested impact of an intervention . The tested impact of an intervention under highly controlled circumstances.under highly controlled circumstances.EffectivenessEffectiveness. The tested impact of an . The tested impact of an intervention under more normal circumstances intervention under more normal circumstances ((relativelyrelatively less controlled, realless controlled, real--time, time, ““typicaltypical””setting, population, and conditions).setting, population, and conditions).Broad Program Evaluation.Broad Program Evaluation. The tested impact of The tested impact of a blended set of interventions on larger systems a blended set of interventions on larger systems and populations. and populations. ““Natural ExperimentsNatural Experiments”” with with minimal control, maximum variability.minimal control, maximum variability.

The TradeThe Trade--offsoffs

Efficacy. Efficacy. Maximizes Maximizes internal validityinternal validity, i.e., the , i.e., the degree to which one can conclude with degree to which one can conclude with confidence that the intervention caused the confidence that the intervention caused the result.result.Effectiveness.Effectiveness. Maximizes Maximizes external validityexternal validity,* ,* i.e., the degree to which one can generalize i.e., the degree to which one can generalize from the test to other times, places, or from the test to other times, places, or populations. populations. Program Evaluation. Program Evaluation. Maximizes reality Maximizes reality testing in particular settings, & with the testing in particular settings, & with the combination of interventions at multiple combination of interventions at multiple levels required for public health effectlevels required for public health effect..

* Green & Glasgow, Evaluation & the Health Professions, Mar. 2006.

PracticePractice--Based Research Networks Based Research Networks (PBRNs)(PBRNs) 36 new PBRN grants awarded in 200236 new PBRN grants awarded in 2002

19 PBRN grants awarded in 200019 PBRN grants awarded in 2000

Aligning Evidence with (and deriving it from) Aligning Evidence with (and deriving it from) Practice: Matching, Mapping, Pooling and PatchingPractice: Matching, Mapping, Pooling and Patching

MatchingMatching ecological levels of a system or community ecological levels of a system or community with evidence of with evidence of efficacyefficacy for interventions at those for interventions at those levelslevelsMappingMapping theory to the causal chain to fill gaps in the theory to the causal chain to fill gaps in the evidence for evidence for effectivenesseffectiveness of interventionsof interventionsPoolingPooling experience to blend interventions to fill gaps experience to blend interventions to fill gaps in evidence for the effectiveness of programs in in evidence for the effectiveness of programs in similar situationssimilar situationsPatchingPatching pooled interventions with indigenous pooled interventions with indigenous wisdom and professional judgment about plausible wisdom and professional judgment about plausible interventions to fill gaps in the interventions to fill gaps in the programprogram for the for the specific populationspecific population

*Green & Kreuter, Health Program Planning: An Educational and EcologicalApproach. 4th ed. NY: McGraw-Hill, 2005, Chapter 5. Green & Glasgow, 2006.

From the AllFrom the All--Time Transdisciplinary Time Transdisciplinary Scientist and Innovator:Scientist and Innovator:

““I have been impressed with the urgency of doing. I have been impressed with the urgency of doing. Knowing is not enough, we must apply.Knowing is not enough, we must apply.””----Leonardo Leonardo dada VinciVinci

To generate usable knowledge for multiTo generate usable knowledge for multi--component behavioral change interventions, we component behavioral change interventions, we can either build it componentcan either build it component--byby--component with component with RCTsRCTs ..Or we can draw it from the natural experiments Or we can draw it from the natural experiments offered by practice and policy innovations.offered by practice and policy innovations.Or bothOr both

The Bridge (not the Pipeline) from The Bridge (not the Pipeline) from Research to Practice and BackResearch to Practice and Back

If we want more evidenceIf we want more evidence--based practice, we based practice, we need more practiceneed more practice--based evidence. based evidence. The importance of practitioners and policyThe importance of practitioners and policy--makers in shaping the research questions.makers in shaping the research questions.Practitioners and their organizations Practitioners and their organizations represent the structural links (and barriers) represent the structural links (and barriers) to addressing the important determinants at to addressing the important determinants at each level. Engage them.each level. Engage them.

*Green, L.W. From research to “best practices” in other settings and populations. Am J Health Behavior 25:165-178, April-May 2001. Full text: www.ajhb.org/25-3.htm or via www.lgreen.net