logic models and qualitative synthesis ideas and issues
TRANSCRIPT
Logic models and qualitative synthesis
Ideas and issues
18/04/23 © The University of Sheffield
2
• Why logic models?
• What are logic models?
• How do I construct a logic model?
• Qualitative synthesis
• Where does and does it not get you?
Outline
18/04/23 © The University of Sheffield
3
Link practice to outcomes
Explore complex relationships
Examine processes and systems
Include multiple sources of evidence
Identify further
research needs
Healthcare research
Logic models/conceptual models/impact models
A systematic and visual way to present relationships and outcomes
Read from left to right following a chain of reasoning
18/04/23 © The University of Sheffield
4
Health promotion literature
18/04/23 © The University of Sheffield
5
Conveying connections between a programme activities and outcomes
(the logic of the programme)
Lando et al., 2006
18/04/23 © The University of Sheffield
6
Inputs Short-term outcomes
Mid-term outcomes
Long-term outcomesActivitie
s
Lando et al., 2006
Programme evaluation
18/04/23 © The University of Sheffield
7
Relationships among programme resources planned activities, and anticipated changes or
results
The Kellog Foundation, 2004
18/04/23 © The University of Sheffield
8
Resources/inputs
Activities
Outputs
Outcomes
Impact
Planned work Intended results
W.K. Kellog Foundation, 2004
Public health
18/04/23 © The University of Sheffield
9
Explore ways in which intervention impacts on different levels of outcomesJoly et al., 2007
18/04/23 © The University of Sheffield
10
Inputs
Outputs
Long-term outcomes
Short-term out-comes
Strategies
Inter-mediate outcomes
Joly et al., 2007
Worked example• We are examining interventions to improve
the happiness of systematic reviewers
• We can find little published evidence reporting happiness outcomes
• We are wanting to model how an intervention might help reviewers
18/04/23 © The University of Sheffield
11
18/04/23 © The University of Sheffield
12
Intervention
Short term outcomes
Medium term Longer term impacts
Social support
Increased happiness
Reduce adverse stress
Improvedmental healthProg-
ramme of lunch time group walks Improved
physical health
Improved fitness
Reduction of pain
Improved diet
Heart rate
Biscuit intake
Caffeine
Headaches/back/neck
Cortisol
Personal contact
Building on the method
18/04/23 © The University of Sheffield
13
NICE Centre for Public Health Excellence
What interventions are effective in enhancing mental wellbeing in the workplace?
18/04/23 © The University of Sheffield
14
• Previous systematic review (Graveling et al. 2008) lack of evidence for effectiveness of interventions
• BUT was this lack of evidence or lack of study designs meeting inclusion criteria?
18/04/23 © The University of Sheffield
15
18/04/23 © The University of Sheffield
16
2883 de-duplicated retrieved citations
Full paper reviewed299
Reject title/abstract 2584
Reject 297Language 2, population 20, not relevant 43, Not peer reviewed 4, discussion/review 102, study design 101
Included papers 2
Are we missing something?
18/04/23 © The University of Sheffield
17
Expert reference group identified literature in addition to papers from systematic review
Proposal to use logic modelling to explore relationships
18/04/23 © The University of Sheffield
18Review paper 45
Discussion paper 35
Survey reporting associations 31
Survey reporting prevalence 30
Policy document/report 29
Cohort study 19
Book 10
Meta analysis 8
Cluster randomised controlled trial 5
Case study 4
Book chapter 3
Qualitative study 2
Randomised controlled trial 1
Controlled before and after study 1
Case control study 1
Workproductivity/competitiveness
Sickness absence behaviour
Work (stress) related Conditions
Health-relatedbehaviours& disease sequela
Health inequalities
Individual psychological‘adjustment’/wellbeing
•Job satisfaction•Commitment•Empowerment•Perceived jobstress
National policies &practices
•Economic policy•Employment -rights/ H&S
•Education /training•Social welfare
Economic & social Economic & social trendstrends•• Economic Economic --employment employment levels, wages, securitylevels, wages, security
••Changes in nature of Changes in nature of work work
••Labour marketLabour market
••Distribution of Distribution of incomeincome
Competitivenesslinked of trends inheath & wellbeing
of workforce(burden of stress-related absence) Job design:
•Workload•Control/autonomy•Role clarity
Job development:•Security•Pay/promotion equity•Learning•Flexibility
Individual characteristics•Gender, ethnicityage•Social circumstances
OrganisationalAttributes•Values & beliefs•Policies & Practices•Health & safety
Social relationships•Org support•Co-worker support•Participation•commitment
Social stratification
Life time workexperience
(risk/resilience)
EnvironmentalHazards& assets
18/04/23 © The University of Sheffield
19
Logic model creation
Little detail regarding the method
• “was drafted by the authors”
• “was developed in conjunction with an expert panel”
• “A group with a facilitator identified activities, outcomes and resources needed, initial model circulated, then refined and clarified”
• A group reviewed existing models, identified principles of new model, the model was refined following feedback
18/04/23 © The University of Sheffield
20
A systematic way to build a model?
18/04/23 © The University of Sheffield
21
18/04/23 © The University of Sheffield
22
Method
Data extractionAnalysis
18/04/23 © The University of Sheffield
23
Synthesis
Method
Extraction
Description of elements impacting on wellbeing and associations between elements extracted on developed form
Analysis
18/04/23 © The University of Sheffield
24
Synthesis
18/04/23 © The University of Sheffield
25
WARNINGReviewers of a
nervous disposition look
away now
Treating all data equallyTreating all data as qualitative
Method
Extraction
Analysis
Line by line reading of extractions,
coding of data, use of
CAQDA software (Nvivo) for storage and retrieval
18/04/23 © The University of Sheffield
26
Synthesis
Qualitative synthesis
• Combining of primary qualitative data
• Emerging approaches (Meta-synthesis, thematic synthesis, meta-ethnography etc)
• Seeking and developing recurring patterns or themes across the data
18/04/23 © The University of Sheffield
27
Thematic synthesis (Thomas & Harden 2008), examine themes across set, and by process of comparison and contrast establish meta-themes (integrate and/or create)
18/04/23 © The University of Sheffield
28
For example
Primary analysis – a paper describes outcomes following a buddy system intervention on employee wellbeing, elements of data coded as peer interaction. Another paper describes the link between working at home and reduced wellbeing, elements of data coded as social interaction.
Secondary analysis – peer interaction and social interaction data combined into social support theme, later when other data added a category of social resources is created.
18/04/23 © The University of Sheffield
29
18/04/23 © The University of Sheffield
30
Health
Well being
Job satisfaction
Work contextHealth and safetyManagement priorities/valuesSupervisor behaviourFeedback & appraisalOrganisational climateOrganisational justiceWork-place supportEmployee participationCommunication systemsIndustry type
Individual factorsIndividual response to workPersonality traitsSelf esteemSelf efficacyPsychological flexibilityPerson-environment fitValuesSocial resourcesIndividual responses to management style or working practicesOptimal workload thresholdCoping response
EconomicAnd social trends
Fixed term contractsFlexible employmentHealth inequalitiesCultural diversityDemographic changesNew technologyICTGlobalisationChange from production to service economy
Individual characteristicsGenderAge Social circumstancesEducationEthnicity Marital statusPredispositions
National polices and practicesEqual opportunities Anti-discrimination policiesFamily-friendly policiesMaximum working hours legislation
Behavioural outcomesHealth-related behaviourSickness absenceStaff turnoverPresenteeismBurnoutCitizenship
Attitudinal outcomesCommitmentMotivationEngagementEmployee expectationsPerceptions of fair treatment
Business outcomesAbsence/turnover costsPerformanceProductivityCustomer satisfactionProfitabilityOptimal staffingSatisfy statutory regulationsSafety/avoidance of litigationCorporate social responsibilityCorporate imageRecruitment cost savingsAttracting and retaining skilled workforceCompetitivenessInsurance premiums/healthcare costs
Work contentWork demandsJob control/decision latitudeEffort requiredRewardsRole Working schedulesOpportunity for learning/developmentMonotonySkills utilisationWorthwhile/fulfillingStability/job futureOccupational group
Intervention points OutcomesWider influences
Where could logic modelling get you?
• May act as a common frame of reference, defining poorly developed concepts
• Used as a participatory and communication tool,
user involvement?
18/04/23 © The University of Sheffield
31
Provides a systems perspective
• Increasing recognition of systems perspective in healthcare practice eg pt safety
• Recognition of complexity eg MRC Developing and evaluating complex interventions
18/04/23 © The University of Sheffield
32
Implementation Research
eg Eccles et al. 2009, Cooksey Report
Transfer of research findings into practice
Complexity of context
Fidelity of interventions
18/04/23 © The University of Sheffield
33
• Used as a research development tool, consider issues that a research agenda needs to address
• Takes an inclusive approach to evidence, areas where little trial work
• Pilot work, Mapping review?
18/04/23 © The University of Sheffield
34
Links between advice services for patients, and health outcomes
18/04/23 © The University of Sheffield
35
Financial outcomes
Health outcomes
Advice
Where does it not get you?• Effectiveness
• Issues with identification of material – selection bias
• Issues with hierarchy of evidence – representation of strength?
• Hierarchy of qual designs? Frequency? Shows where research has been done rather than evidence?
18/04/23 © The University of Sheffield
36
More issues• Critical appraisal especially qualitative
sources
• Issues with integration of diff types of data
• Removal of context
• Use of primary material in secondary synthesis
• Integration versus interpretation
18/04/23 © The University of Sheffield
37
Logic is the
beginning of wisdom,
not the end.
Leonard Nimoy
Logic will get you from A to
B.
Imagination will take you
everywhere.
Albert Einstein
18/04/23 © The University of Sheffield
38
18/04/23 © The University of Sheffield
39
39
Promise improvement,Exact retribution
Check with Clients
Logical Model for Creating Achievable and Sustainable Change Modalities over Time
Put paper in printer
Engage 1st gear
Realize that you look important if it’s really really
confusing.
Sta
rt t
hin
kin
g a
bout
creati
ng a
really
com
fusi
ng c
hart
.
Realize that even you don’t understand any
of this.
Roll down hill
Capture Vision
Engage bullshit generator
ReportOutcomes
WaxEloquent
The world is a
better place
because they
actually paid you
to do this.
Analyze everything you can think of in
depth
Deta
iled
cau
se a
nd
eff
ect
follo
wsBuild
conceptual engine
Engage graphics generator
Realize that you look important if you’re the only
one who understands it.
InternalizeConceptualize
Target audience Create message Engage audienceRealize/Engage Actualize/Energize Promote/Enhance Expose/Repose
Hirechimp Let Vision
Go
Re-captureVision
Jump start engine
AchieveGoals
FinalizeResults
WaxCar
Firechimp
Put orange box here.
Keep close watch on vision.
Unknown author
References
Baxter R, Killoran, A, Kelly MP, Goyder E. Synthesizing diverse evidence: the use of primary qualitative data analysis methods and logic models in public health reviews. Public Health, 2010; 124: 99-106.
Barbour R, Barbour M. Evaluating and synthesising qualitative research: the need to develop a distinctive approach. J Eval Clin Prac 2003;9:179–86.
Bunton R, Macdonald G, editors. Health promotion: disciplines, diversity and developments. 2nd edn. London: Routledge; 2002
Cooksey D, A Review of UK health research funding. Norwich: HMSO; 2006.
Eccles M, Armstrong D, Baker R et al. An Implementation Research Agenda. Implementation Science, 2009: 4: 18 doi: 10.1186/1748-590804018.
Graveling R, Crawford J, Cowie H, Amati C, Vohra S. A review of workplace interventions that promote mental wellbeing in the workplace. Edinburgh: Institute of Occupational Medicine; 2008.
Joly B, Polyak G, Davis M, Brewster J, Tremain B, Raevsky C, et al. Linking accreditation and public health outcomes: a logic model approach. J Public Health Manag Prac 2007;13:349–56.
Kelly M, Stewart E, Morgan A, Killoran A, Fischer A, Threlfall A, Bonnefoy J. A conceptual framework for public health: NICE’s emerging approach. Public Health 2009;123:14–20.
Kellog. Foundation, The logic model development guide. Michigan: Kellog Foundation; 2004.
Lando J, Williams S, Williams B, Sturgis S. A logic model for the integration of mental health into chronic disease prevention and health promotion. Prev Chron Disease Pub Health Res Prac t Policy 2006: 3; 1-4.
Schmitz C. Everything you wanted to know about logic models but were afraid to ask. Michigan: Kellog Foundation; 1999.
18/04/23 © The University of Sheffield
40
18/04/23 © The University of Sheffield
41
Tannihill A. Beyond evidence to ethics: a decision-making framework for health promotion, public health and health improvement. Health Promot Int 2008;23:380–90.Thomas J, Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Med Res Meth 2008;8:45.Tones K, Green J. Health promotion: planning and strategies. London: Sage; 2004.
Further reference lists/resources:http://www.cdc.gov/eval/resources.htm#logic model