international engagement: m & e meeting

International engagement: m & e meeting

Monitoring & Evaluation: an introduction for practitioners

Liz Allen

Source: Subsetum; http://www.flickr.com/photos/subsetsum/

monitoring & evaluation can play a monitoring & evaluation can play a vital role in the management & vital role in the management & improvement of an activity, improvement of an activity, organisation or process organisation or process

Monitoring & evaluation: why do it?

Theory & practice

M & E: challenges

Keeping it real: practical & proportionality

my talkMonitoring & evaluation: an introduction for practitioners

o accountability & validation

o management of funding initiatives

o strategy & planning

o policy & advocacy

o learning

my funder requires it …

I want my programme to

work well …

I want to do more in the

future …

I want to tell people about what we do … & show the benefits …

I want to find out what works …

rationaleMonitoring & evaluation: rationale

Theory

Source: Subsetum; http://www.flickr.com/photos/subsetsum/

o M & E frameworko Monitoringo Set up reviewo Process reviewo Formative reviewo Summative review

Source: Subsetum; http://www.flickr.com/photos/subsetsum/

o M & E frameworko Monitoringo Set up reviewo Process reviewo Formative reviewo Summative review o Inputs/activities

o outputso outcomeso … impact?o (longer term impact)

Source: Subsetum; http://www.flickr.com/photos/subsetsum/

o M & E frameworko Monitoringo Set up reviewo Process reviewo Formative reviewo Summative review o inputs/activities

o outputso outcomeso … impact?

(longer term impact)

Things to do & things to ‘measure/track’

Theory & formal frameworks

o Logical framework

o Outcome mapping

o Payback framework

o Theories of change

o Results-based management

Monitoring & evaluation: rationale M & E in project cycle

Monitoring & review pointsINPUTS & OUTPUTS

‘ M & E Framework’ design

Set-up/process review

INPUTS

Project start

Taking stockOUTPUTS/OUTCOMES

Learning & lessons for next time

Frameworks good for

o keeping outcomes in mind

o ‘forces’ definition of: o project objectiveso indicatorso appropriate data

collection method & timings

o organisationo simplification

Monitoring & evaluation: rationale

Frameworks less good

o keeping outcomes in mind

o ‘forces’ definition of: o project objectiveso indicatorso appropriate data

collection method & timings

o organisationo simplification

Monitoring & evaluation: rationale

o accommodating externalities & context

o dealing with the unexpected

o understanding process

Practicedoing m & e

(my) basic principles

o involvement of stakeholders

o definition of objectives/outcomes & associated indicators & method

o integrated into project plan from start

o properly resourced

o practical (usable) & proportionate

Monitoring & evaluation: doing m & e

(my) basic principles

o involvement of stakeholders

o definition of objectives/outcomes & associated indicators & method

o integrated into project plan from start

o properly resourced

o practical (usable) & proportionate

Monitoring & evaluation: doing m & e

an indicator“quantitative or qualitative factor or variable that provides a simple and reliable means to measure achievement.”

“ something that helps us to understand where we are, where we are going and how far we are from the goal ... They are bits of information that summarize the characteristics of systems or highlight what is happening in a system. “

Introduction to LFA: defining indicators

SMART & SPICED indicators

SMART

SpecificMeasurableAchievableRelevantTimebound

SPICED

SubjectiveParticipatory

Interpreted/communicableCross-checked

EmpoweringDiverse & disaggregated

Introduction to LFA: defining indicators

“Everything that can be counted does not necessarily count; everything that counts cannot necessarily be counted.”attributed to Albert Einsteintheoretical physicist, philosopher & author 1879-1955

Quantitative

o access & participation data (‘penetration’)o attendance & visitor data (‘reach’)o ‘audience’ characteristics (e.g. demographics)o consumer characteristicso scientific output assessment (e.g. publications)o ‘alt metrics’o amount raised/ follow-on fundingo structured opinion & feedback

Qualitative

o interviewso open questions (in questionnaire) o focus groupso ethnography o participatory researcho observationo comment/bulletin boardso visitor books

Monitoring & evaluation: rationale & overview

Theory & practice

M & E: challenges

Keeping it real: practical & proportionality

my talkMonitoring & evaluation: an introduction for practitioners

the elusive ‘impact’time frame involved

serendipity

attribution & contribution

‘ripple effects’

counter-factual

value of ‘negative’ findings

and closer to home …

getting agreement on goals, objectives, indicators & method

and resourcing m & e properly …

adequately, systematically, proportionately…

Monitoring & evaluation: rationale & overview

Theory & practice

M & E: challenges

Keeping it real: practical & proportionality

my talkMonitoring & evaluation: an introduction for practitioners

Understand stakeholder & audience requirements Be prospective: build m & e in from the start Choose right method & tailor Resourcing: ensure access to key data & information &

someone to manage this Consider option for trends & benchmarks Keep it real: be proportionate & practical – measures

can evolve Be flexible – learning is iterative & should be part of

the process Beware over-monitoring & evaluation!

SummaryMonitoring & evaluation: keeping it real

Wellcome Trust’s Indicators of ProgressOutcomes Key indicators of progress

Discoveries

Applications

Engagement

Research leaders

Research environment

Influence

1. significant advances in the generation of new knowledge2. contribute to discoveries with tangible impacts on health

3. contribute to the development of enabling technologies, products and devices

4. uptake of research into policy and practice

5. enhanced level of informed debate in biomedicine6. significant engagement of key audiences & increased reach

7. develop a cadre of research leaders8. evidence of significant career progression among those we

support

9. key contributions to the creation, development and maintenance of major research resources

10. contributions to the growth of centres of excellence

11. significant impact on science funding & policy developments12. significant impact on global research priorities and processes

Wellcome Trust’s Indicators of ProgressOutcomes Key indicators of progress

Discoveries

Applications

Engagement

Research leaders

Research environment

Influence

1. significant advances in the generation of new knowledge2. contribute to discoveries with tangible impacts on health

3. contribute to the development of enabling technologies, products and devices

4. uptake of research into policy and practice

5. enhanced level of informed debate in biomedicine6. significant engagement of key audiences & increased reach

7. develop a cadre of research leaders8. evidence of significant career progression among those we

support

9. key contributions to the creation, development and maintenance of major research resources

10. contributions to the growth of centres of excellence

11. significant impact on science funding & policy developments12. significant impact on global research priorities and processes

Wellcome Trust’s Indicators of ProgressOutcomes Key indicators of progress

Discoveries

Applications

Engagement

Research leaders

Research environment

Influence

1. significant advances in the generation of new knowledge2. contribute to discoveries with tangible impacts on health

3. contribute to the development of enabling technologies, products and devices

4. uptake of research into policy and practice

5. enhanced level of informed debate in biomedicine6. significant engagement of key audiences & increased reach

7. develop a cadre of research leaders8. evidence of significant career progression among those we

support

9. key contributions to the creation, development and maintenance of major research resources

10. contributions to the growth of centres of excellence

11. significant impact on science funding & policy developments12. significant impact on global research priorities and processes

l.allen@wellcome.ac.uk

Questions

Suggested resources

Ahmed, S. & Palermo, A-G (2010) Community Engagement in Research: Frameworks for Education and Peer Review. American Journal of Public Health, 100, 8: 1380-1387

Matthew, B. & Ross, L. (2010) Research Methods: a practical guide for the social sciences. Pearson Education Limited

NORAD (2008) Results Management in Norwegian Development Cooperation: a practical guide. http://www.norad.no/en/tools-and-publications/publications/publication?key=109837

Stein, D. & Valters, C. (2012) Understanding ‘Theory of Change’ in international al development : a review of existing knowledgehttp://www2.lse.ac.uk/internationalDevelopment/research/JSRP/downloads/ToC_Lit_Review.pdf

Vogel, I. (2012) Review of the use of ‘Theory of Change’ in international development. For DfID UK

Web Center for Social Research methods http://socialresearchmethods.net