evaluation approaches heather aquilina 24 march 2015
TRANSCRIPT
![Page 1: EVALUATION APPROACHES Heather Aquilina 24 March 2015](https://reader036.vdocument.in/reader036/viewer/2022062321/56649eb35503460f94bbb54f/html5/thumbnails/1.jpg)
EVALUATION APPROACHESHeather Aquilina
24 March 2015
![Page 2: EVALUATION APPROACHES Heather Aquilina 24 March 2015](https://reader036.vdocument.in/reader036/viewer/2022062321/56649eb35503460f94bbb54f/html5/thumbnails/2.jpg)
2
Aim of Today
• Different frameworks for viewing evaluation approaches (M&E, formative/summative, Owen’s forms, belief system), data collection type
• Different approaches: what are they, when to use them and why
• Questions
![Page 3: EVALUATION APPROACHES Heather Aquilina 24 March 2015](https://reader036.vdocument.in/reader036/viewer/2022062321/56649eb35503460f94bbb54f/html5/thumbnails/3.jpg)
3
Evaluation and Monitoring
Evaluation – a structured process of assessing the success of a project or program in meeting its goals and on lessons learnt.
Monitoring – setting targets and milestones to measure progress and achievements and whether inputs are producing planned outputs.
![Page 4: EVALUATION APPROACHES Heather Aquilina 24 March 2015](https://reader036.vdocument.in/reader036/viewer/2022062321/56649eb35503460f94bbb54f/html5/thumbnails/4.jpg)
4
Formative and Summative Evaluation
Formative (process) • Assesses the value of a project as it is being
implemented, to check its implementation and to improve its design and performance.
• Uses both qualitative data (how participants felt about the process) as well as quantitative data.
Summative (impact/outcome) • Focuses on whether a project's goals have
been achieved. • More concerned with quantitative data.
![Page 5: EVALUATION APPROACHES Heather Aquilina 24 March 2015](https://reader036.vdocument.in/reader036/viewer/2022062321/56649eb35503460f94bbb54f/html5/thumbnails/5.jpg)
5
Form Use ExamplesProactive (Design)
Synthesis Needs assessment, best practice review
Clarificative Clarification Program logic
Interactive Improvement, implementation
Action research, developmental
Monitoring Justification, fine tuning
PI, systems analysis
Impact Accountability Performance audit
Evaluation Form (Owen)
John M. Owen, Program evaluation – forms and approaches
![Page 6: EVALUATION APPROACHES Heather Aquilina 24 March 2015](https://reader036.vdocument.in/reader036/viewer/2022062321/56649eb35503460f94bbb54f/html5/thumbnails/6.jpg)
6
Positivism Realism Interpretivism / Constructivism
Random trials
SMART
Program Logic
Realist Participatory evaluation
Action research
Needs assessment
Focus Groups
Belief About The World
![Page 7: EVALUATION APPROACHES Heather Aquilina 24 March 2015](https://reader036.vdocument.in/reader036/viewer/2022062321/56649eb35503460f94bbb54f/html5/thumbnails/7.jpg)
7
Methodology
By methodology:
• Document analysis
• Program data
• Interviews
• Focus groups
• Surveys
• Descriptive and inferential statistics
• Observation
![Page 8: EVALUATION APPROACHES Heather Aquilina 24 March 2015](https://reader036.vdocument.in/reader036/viewer/2022062321/56649eb35503460f94bbb54f/html5/thumbnails/8.jpg)
8
Data Collection - Qualitative
Descriptions of problems, behaviours or events, people’s thoughts and opinions about their experiences, attitudes, and beliefs.
• Pros – more depth and insights as to the “why” and “how” of attitudes and behaviours, clarifies quantitative data.
• Cons - time consuming to capture and analyse, more subjective and difficult to summarize and compare systematically. Generally viewed as less reliable because it is more subjective than quantitative methods, and may yield smaller sample sizes.
![Page 9: EVALUATION APPROACHES Heather Aquilina 24 March 2015](https://reader036.vdocument.in/reader036/viewer/2022062321/56649eb35503460f94bbb54f/html5/thumbnails/9.jpg)
9
Data Collection - Quantitative
Counts or frequencies, rates or percentages, or other statistics.
• Pros - easy to administer, can have large number of questions, can yield large samples, easier to summarize, more widely accepted as a form of evidence.
• Cons - not as rich or as detailed as qualitative methods, survey/written questionnaires may be difficult for some participants, may not be able to be interpreted, and large amounts of data may require more sophisticated analysis approaches.
![Page 10: EVALUATION APPROACHES Heather Aquilina 24 March 2015](https://reader036.vdocument.in/reader036/viewer/2022062321/56649eb35503460f94bbb54f/html5/thumbnails/10.jpg)
10
Data collection – Mixed Methods
Combines quantitative and qualitative data. Maximizes the strengths and minimizes the weaknesses of both.
• Pros – Can improve research results. Complement each other and addresses a broader range of questions. Provides insights that don't appear when using a single method.
• Cons – Need to be experienced in both types. Different methods need to be combined appropriately, which requires more time and money.
![Page 11: EVALUATION APPROACHES Heather Aquilina 24 March 2015](https://reader036.vdocument.in/reader036/viewer/2022062321/56649eb35503460f94bbb54f/html5/thumbnails/11.jpg)
11
Approaches
• Literature review
• Program Logic
• Case Studies
• SMART
• Outcome Mapping (OM)
• Action Learning/Research
• Realist
• Mixed Methods
• Appreciative Enquiry
• Most Significant Change (MSC)
• Participatory Evaluation
• Performance Indicators
• Social Return on Investment (SROI)
![Page 12: EVALUATION APPROACHES Heather Aquilina 24 March 2015](https://reader036.vdocument.in/reader036/viewer/2022062321/56649eb35503460f94bbb54f/html5/thumbnails/12.jpg)
12
Literature Review
Overview of the theory and the research literature.
• Seeks to describe, summarise, evaluate, clarify and/or integrate the content of primary reports.
• Useful for:
– design
– identifying key issues/factors
– justifying questions or framework & method
– understanding the study
![Page 13: EVALUATION APPROACHES Heather Aquilina 24 March 2015](https://reader036.vdocument.in/reader036/viewer/2022062321/56649eb35503460f94bbb54f/html5/thumbnails/13.jpg)
Program Logic
Describes how a program is intended to work.
• Pros - Clarifies the theory of action, assumptions, builds
• Cons – May not be appropriate for complex cases, may restrict improvement
![Page 14: EVALUATION APPROACHES Heather Aquilina 24 March 2015](https://reader036.vdocument.in/reader036/viewer/2022062321/56649eb35503460f94bbb54f/html5/thumbnails/14.jpg)
14
Case Studies
Focuses on a particular unit - a person, a site, a project.
• Often uses a combination of quantitative and qualitative data.
• Particularly useful for: – understanding how different elements fit
together and – how different elements (implementation,
context and other factors) have produced the observed impacts.
![Page 15: EVALUATION APPROACHES Heather Aquilina 24 March 2015](https://reader036.vdocument.in/reader036/viewer/2022062321/56649eb35503460f94bbb54f/html5/thumbnails/15.jpg)
SMART
Defined as Specific, Measurable, Attainable, Relevant and Time-bound
• Outcome or Summative approach and useful in program evaluations.
• Records the achieved outcomes or consequences of a program.
• Measures program effectiveness, cost effectiveness, appropriateness and efficiency.
![Page 16: EVALUATION APPROACHES Heather Aquilina 24 March 2015](https://reader036.vdocument.in/reader036/viewer/2022062321/56649eb35503460f94bbb54f/html5/thumbnails/16.jpg)
16
Outcome Mapping - OM
Measures behavioural change of stakeholders.
• Collects data on immediate, basic changes that lead to longer, more transformative change.
• Useful for programs which use a development/influence rather than direct service model
![Page 17: EVALUATION APPROACHES Heather Aquilina 24 March 2015](https://reader036.vdocument.in/reader036/viewer/2022062321/56649eb35503460f94bbb54f/html5/thumbnails/17.jpg)
17
Action Learning/Research
A learning-by-doing process. Analysis of actions feeds into a reflective process leading to improvements.
• Gathers and shares information about what works and what doesn’t work as the project/program operates.
• Makes improvements along the way rather than waiting until the end.
![Page 18: EVALUATION APPROACHES Heather Aquilina 24 March 2015](https://reader036.vdocument.in/reader036/viewer/2022062321/56649eb35503460f94bbb54f/html5/thumbnails/18.jpg)
18
Realist
What works, for whom, in what respects, to what
extent, in what contexts, and how?
• Four key linked concepts - mechanism,
context, outcome pattern, and context-
mechanism-outcome pattern configuration.
• Data collection includes interviews, focus
groups, questionnaires.
![Page 19: EVALUATION APPROACHES Heather Aquilina 24 March 2015](https://reader036.vdocument.in/reader036/viewer/2022062321/56649eb35503460f94bbb54f/html5/thumbnails/19.jpg)
Mixed Methods
Combines a number of approaches in one evaluation study.
• Pros - Different types of data complement and support each other
• Cons – Carried out ad hoc, not matched to rationale
![Page 20: EVALUATION APPROACHES Heather Aquilina 24 March 2015](https://reader036.vdocument.in/reader036/viewer/2022062321/56649eb35503460f94bbb54f/html5/thumbnails/20.jpg)
Appreciative Enquiry
Focuses on strengths rather than on weaknesses. Identifies what is working well and seeks to improve performance and conditions.
• Pros – encourages people to produce their best
• Cons – may lead to ignoring problems
![Page 21: EVALUATION APPROACHES Heather Aquilina 24 March 2015](https://reader036.vdocument.in/reader036/viewer/2022062321/56649eb35503460f94bbb54f/html5/thumbnails/21.jpg)
21
Most Significant Change - MSC
Generating and analysing personal accounts of change and deciding which of these accounts is the most significant – and why.
• Pros - Useful for HOW change comes about & WHEN – in what situations & context. Compelling stories.
• Cons – Expensive in resourcing and time, only answers contribution to outcomes, not cost effectiveness or appropriateness.
![Page 22: EVALUATION APPROACHES Heather Aquilina 24 March 2015](https://reader036.vdocument.in/reader036/viewer/2022062321/56649eb35503460f94bbb54f/html5/thumbnails/22.jpg)
Participatory Evaluation
Why is evaluation being done, how evaluation is done, who evaluates, what is being evaluated, and for whom is it being done.
• Pros – more useful, more fair, more valid
• Cons – expensive in time and resources, difficult, open to bias, intervention rather than evaluation
![Page 23: EVALUATION APPROACHES Heather Aquilina 24 March 2015](https://reader036.vdocument.in/reader036/viewer/2022062321/56649eb35503460f94bbb54f/html5/thumbnails/23.jpg)
23
What to choose when
• Evaluation question
• Program stage
• Resources available
• Skills of the evaluator
• Personal philosophical view
![Page 24: EVALUATION APPROACHES Heather Aquilina 24 March 2015](https://reader036.vdocument.in/reader036/viewer/2022062321/56649eb35503460f94bbb54f/html5/thumbnails/24.jpg)
24
Thank you
• Any Questions?