![Page 1: Setting the Stage: Workshop Framing and Crosscutting Issues](https://reader036.vdocument.in/reader036/viewer/2022062310/568165bb550346895dd8b8c1/html5/thumbnails/1.jpg)
Setting the Stage: Workshop Framing
and Crosscutting Issues
Simon Hearn, ODI
Evaluation Methods for Large-Scale, Complex, Multi-National Global Health Initiatives, Institute of Medicine
7-8 January, London, UK
![Page 2: Setting the Stage: Workshop Framing and Crosscutting Issues](https://reader036.vdocument.in/reader036/viewer/2022062310/568165bb550346895dd8b8c1/html5/thumbnails/2.jpg)
The nature of the intervention:1. Focus of objectives2. Governance3. Consistency of implementation
What do we mean by complex interventions?
How it works:4. Necessariness5. Sufficiency6. Change trajectory
Photo: Les Chatfield - http://www.flickr.com/photos/elsie/
![Page 3: Setting the Stage: Workshop Framing and Crosscutting Issues](https://reader036.vdocument.in/reader036/viewer/2022062310/568165bb550346895dd8b8c1/html5/thumbnails/3.jpg)
What are the challenges of evaluating complex interventions?
Describing what is being implementedGetting data about impactsAttributing impacts to a particular programme
Photo: Les Chatfield - http://www.flickr.com/photos/elsie/
![Page 4: Setting the Stage: Workshop Framing and Crosscutting Issues](https://reader036.vdocument.in/reader036/viewer/2022062310/568165bb550346895dd8b8c1/html5/thumbnails/4.jpg)
Wikipedia: Evaluation Methods
Why a framework is needed
![Page 5: Setting the Stage: Workshop Framing and Crosscutting Issues](https://reader036.vdocument.in/reader036/viewer/2022062310/568165bb550346895dd8b8c1/html5/thumbnails/5.jpg)
Image: Simon Kneebone. http://simonkneebone.com/
Why a framework is needed
![Page 6: Setting the Stage: Workshop Framing and Crosscutting Issues](https://reader036.vdocument.in/reader036/viewer/2022062310/568165bb550346895dd8b8c1/html5/thumbnails/6.jpg)
The Rainbow Framework
![Page 7: Setting the Stage: Workshop Framing and Crosscutting Issues](https://reader036.vdocument.in/reader036/viewer/2022062310/568165bb550346895dd8b8c1/html5/thumbnails/7.jpg)
DEFINE what is to be evaluated
![Page 8: Setting the Stage: Workshop Framing and Crosscutting Issues](https://reader036.vdocument.in/reader036/viewer/2022062310/568165bb550346895dd8b8c1/html5/thumbnails/8.jpg)
Why do we need to start with a clear definition?
Photo: Hobbies on a Budget / Flickr
![Page 9: Setting the Stage: Workshop Framing and Crosscutting Issues](https://reader036.vdocument.in/reader036/viewer/2022062310/568165bb550346895dd8b8c1/html5/thumbnails/9.jpg)
1. Develop initial description
2. Develop program theory or logic model
3. Identify potential unintended results
![Page 10: Setting the Stage: Workshop Framing and Crosscutting Issues](https://reader036.vdocument.in/reader036/viewer/2022062310/568165bb550346895dd8b8c1/html5/thumbnails/10.jpg)
Options for representing logic models
Pipeline / results chain
Logical framework
Outcomes hierarchy / theory of change
Realist Matrix
![Page 11: Setting the Stage: Workshop Framing and Crosscutting Issues](https://reader036.vdocument.in/reader036/viewer/2022062310/568165bb550346895dd8b8c1/html5/thumbnails/11.jpg)
FRAME what is to be evaluated
![Page 12: Setting the Stage: Workshop Framing and Crosscutting Issues](https://reader036.vdocument.in/reader036/viewer/2022062310/568165bb550346895dd8b8c1/html5/thumbnails/12.jpg)
Source: Hobbies on a Budget / Flickr
Frame Decision Make Decision
Frame Evaluation Design Evaluation
![Page 13: Setting the Stage: Workshop Framing and Crosscutting Issues](https://reader036.vdocument.in/reader036/viewer/2022062310/568165bb550346895dd8b8c1/html5/thumbnails/13.jpg)
1. Identify primary intended users
2. Decide purpose(s)3. Specify key evaluation
questions4. Determine what
‘success’ looks like
![Page 14: Setting the Stage: Workshop Framing and Crosscutting Issues](https://reader036.vdocument.in/reader036/viewer/2022062310/568165bb550346895dd8b8c1/html5/thumbnails/14.jpg)
DESCRIBE what happened
![Page 15: Setting the Stage: Workshop Framing and Crosscutting Issues](https://reader036.vdocument.in/reader036/viewer/2022062310/568165bb550346895dd8b8c1/html5/thumbnails/15.jpg)
1. Sample2. Use measures,
indicators or metrics3. Collect and/or retrieve
data4. Manage data5. Combine qualitative
and quantitative data6. Analyze data7. Visualize data
![Page 16: Setting the Stage: Workshop Framing and Crosscutting Issues](https://reader036.vdocument.in/reader036/viewer/2022062310/568165bb550346895dd8b8c1/html5/thumbnails/16.jpg)
Combine qualitative and quantitative data
EnrichExamineExplainTriangulate
ParallelSequential
ComponentIntegrated
![Page 17: Setting the Stage: Workshop Framing and Crosscutting Issues](https://reader036.vdocument.in/reader036/viewer/2022062310/568165bb550346895dd8b8c1/html5/thumbnails/17.jpg)
UNDERSTAND CAUSES of outcomes and impacts
![Page 18: Setting the Stage: Workshop Framing and Crosscutting Issues](https://reader036.vdocument.in/reader036/viewer/2022062310/568165bb550346895dd8b8c1/html5/thumbnails/18.jpg)
Outcomes Impacts
![Page 19: Setting the Stage: Workshop Framing and Crosscutting Issues](https://reader036.vdocument.in/reader036/viewer/2022062310/568165bb550346895dd8b8c1/html5/thumbnails/19.jpg)
As a profession, we often either oversimplify causationor we overcomplicate it!
![Page 20: Setting the Stage: Workshop Framing and Crosscutting Issues](https://reader036.vdocument.in/reader036/viewer/2022062310/568165bb550346895dd8b8c1/html5/thumbnails/20.jpg)
“In my opinion, measuring attribution is critical, and we can't do that unless we use control groups to compare them to.”
Comment in an expert discussion on The Guardian online, May 2013
![Page 21: Setting the Stage: Workshop Framing and Crosscutting Issues](https://reader036.vdocument.in/reader036/viewer/2022062310/568165bb550346895dd8b8c1/html5/thumbnails/21.jpg)
1. Check that the results support causal attribution
2. Compare results to the counterfactual
3. Investigate possible alternative explanations
![Page 22: Setting the Stage: Workshop Framing and Crosscutting Issues](https://reader036.vdocument.in/reader036/viewer/2022062310/568165bb550346895dd8b8c1/html5/thumbnails/22.jpg)
SYNTHESIZE data from one or more evaluations
![Page 23: Setting the Stage: Workshop Framing and Crosscutting Issues](https://reader036.vdocument.in/reader036/viewer/2022062310/568165bb550346895dd8b8c1/html5/thumbnails/23.jpg)
Was it good?Did it work?
Was it effective?
For whom did it work?
In what ways did it work?
Was it value for money?Was it cost-effective?
Did it succeed in terms of the Triple Bottom Line?
![Page 24: Setting the Stage: Workshop Framing and Crosscutting Issues](https://reader036.vdocument.in/reader036/viewer/2022062310/568165bb550346895dd8b8c1/html5/thumbnails/24.jpg)
24
How do we synthesize diverse evidence about performance?
All intended impacts achieved
Some intended impacts achieved
No negative impacts
Overall synthesis GOOD ?? ?? BAD
![Page 25: Setting the Stage: Workshop Framing and Crosscutting Issues](https://reader036.vdocument.in/reader036/viewer/2022062310/568165bb550346895dd8b8c1/html5/thumbnails/25.jpg)
1. Synthesize data from a single evaluation
2. Synthesize data across evaluations
3. Generalize findings
![Page 26: Setting the Stage: Workshop Framing and Crosscutting Issues](https://reader036.vdocument.in/reader036/viewer/2022062310/568165bb550346895dd8b8c1/html5/thumbnails/26.jpg)
REPORT and SUPPORT USE of findings
![Page 27: Setting the Stage: Workshop Framing and Crosscutting Issues](https://reader036.vdocument.in/reader036/viewer/2022062310/568165bb550346895dd8b8c1/html5/thumbnails/27.jpg)
I can honestly say that not a day goes by when we don’t
use those evaluations in one
way or another
![Page 28: Setting the Stage: Workshop Framing and Crosscutting Issues](https://reader036.vdocument.in/reader036/viewer/2022062310/568165bb550346895dd8b8c1/html5/thumbnails/28.jpg)
1. Identify reporting requirements
2. Develop reporting media
3. Ensure accessibility4. Develop
recommendations5. Support use
![Page 29: Setting the Stage: Workshop Framing and Crosscutting Issues](https://reader036.vdocument.in/reader036/viewer/2022062310/568165bb550346895dd8b8c1/html5/thumbnails/29.jpg)
![Page 30: Setting the Stage: Workshop Framing and Crosscutting Issues](https://reader036.vdocument.in/reader036/viewer/2022062310/568165bb550346895dd8b8c1/html5/thumbnails/30.jpg)
MANAGE your evaluation
![Page 31: Setting the Stage: Workshop Framing and Crosscutting Issues](https://reader036.vdocument.in/reader036/viewer/2022062310/568165bb550346895dd8b8c1/html5/thumbnails/31.jpg)
1. Understand and engage with stakeholders
2. Establish decision making processes
3. Decide who will conduct the evaluation
4. Determine and secure resources5. Define ethical and quality
evaluation standards 6. Document management
processes and agreements7. Develop evaluation plan or
framework8. Review evaluation 9. Develop evaluation capacity
![Page 32: Setting the Stage: Workshop Framing and Crosscutting Issues](https://reader036.vdocument.in/reader036/viewer/2022062310/568165bb550346895dd8b8c1/html5/thumbnails/32.jpg)
![Page 33: Setting the Stage: Workshop Framing and Crosscutting Issues](https://reader036.vdocument.in/reader036/viewer/2022062310/568165bb550346895dd8b8c1/html5/thumbnails/33.jpg)
DESCRIBEUNDERSTAND CAUSESSYNTHESIZEREPORT & SUPPORT USE
Descriptive Questions – Was the policy implemented as planned?
Causal questions – Did the policy change contribute to improved health outcomes?
Synthesis questions – Was the policy overall a success?
Action questions – What should we do?
Making decisionsLook at type of questions
![Page 34: Setting the Stage: Workshop Framing and Crosscutting Issues](https://reader036.vdocument.in/reader036/viewer/2022062310/568165bb550346895dd8b8c1/html5/thumbnails/34.jpg)
Making decisionsCompare pros and cons
![Page 35: Setting the Stage: Workshop Framing and Crosscutting Issues](https://reader036.vdocument.in/reader036/viewer/2022062310/568165bb550346895dd8b8c1/html5/thumbnails/35.jpg)
Participant Questionnaire
Key Informant Interviews Project Records
Observation of program implementation
KEQ1 What was the quality of implementation?
✔ ✔ ✔ ✔
KEQ2 To what extent were the program objectives met?
✔ ✔ ✔
KEQ3 What other impacts did the program have?
✔ ✔
KEQ4 How could the program be improved?
✔ ✔ ✔
Making decisionsCreate an evaluation matrix
![Page 36: Setting the Stage: Workshop Framing and Crosscutting Issues](https://reader036.vdocument.in/reader036/viewer/2022062310/568165bb550346895dd8b8c1/html5/thumbnails/36.jpg)
www.betterevaluation.org
![Page 37: Setting the Stage: Workshop Framing and Crosscutting Issues](https://reader036.vdocument.in/reader036/viewer/2022062310/568165bb550346895dd8b8c1/html5/thumbnails/37.jpg)
Examples
Descriptions
ToolsGuides
Comments
R & D
Documenting
Sharing
Events ACTIVITIES
WEBSITE
![Page 38: Setting the Stage: Workshop Framing and Crosscutting Issues](https://reader036.vdocument.in/reader036/viewer/2022062310/568165bb550346895dd8b8c1/html5/thumbnails/38.jpg)
Founding Partners
Financial Supporters
![Page 39: Setting the Stage: Workshop Framing and Crosscutting Issues](https://reader036.vdocument.in/reader036/viewer/2022062310/568165bb550346895dd8b8c1/html5/thumbnails/39.jpg)
For more information:www.betterevaluation.org/start_here