ttipec: monitoring and evaluation (session 2)
DESCRIPTION
Powerpoint on monitoring and evaluation for sessions 2 of the TTIPEC Ghana workshopTRANSCRIPT
![Page 1: TTIPEC: Monitoring and Evaluation (Session 2)](https://reader034.vdocument.in/reader034/viewer/2022051817/54832c55b4af9f910d8b4934/html5/thumbnails/1.jpg)
Monitoring & Evaluation
IN PRACTICE
TTI PEC Anglophone Africa Regional Peer Learning Workshop
Coconut Grove Regency Hotel | Accra, Ghana
Wednesday, July 16th, 2014 !!Mo Adefeso-OlatejuRebecca Oluwatoyin Doherty
![Page 2: TTIPEC: Monitoring and Evaluation (Session 2)](https://reader034.vdocument.in/reader034/viewer/2022051817/54832c55b4af9f910d8b4934/html5/thumbnails/2.jpg)
Linking Monitoring, Evaluation, & Measurement
SHORT VIDEOS
Importance of monitoring & evaluation
https://www.youtube.com/watch?v=2mBcb-AQ8xQ 4:00minutes - end !Indicators https://www.youtube.com/watch?v=SZqOKW2_jNU !Indicators: Basic Concepts https://www.youtube.com/watch?v=qmrDNuJzRv4
![Page 3: TTIPEC: Monitoring and Evaluation (Session 2)](https://reader034.vdocument.in/reader034/viewer/2022051817/54832c55b4af9f910d8b4934/html5/thumbnails/3.jpg)
1ST THINGS FIRST
![Page 4: TTIPEC: Monitoring and Evaluation (Session 2)](https://reader034.vdocument.in/reader034/viewer/2022051817/54832c55b4af9f910d8b4934/html5/thumbnails/4.jpg)
4 BASIC M&E POINTS MUST BE FOLLOWED
Who is our population? What data is to be collected from them? Or about them?
How will the data captured enhance communications? And also fit within the rest of the M&E analysis?
How will the data be analyzed? Who will be doing the analysis? What do we know about our baseline data?
How will the process be managed? And by whom?
An organizational assessment is needed…in order to establish a process for M&E!
Source: https://sites.google.com/site/newmediacommunications2010/mobiles-and-m-e/2-case-studies
![Page 5: TTIPEC: Monitoring and Evaluation (Session 2)](https://reader034.vdocument.in/reader034/viewer/2022051817/54832c55b4af9f910d8b4934/html5/thumbnails/5.jpg)
ASSESSMENT OF READINESSIn order for the M&E Function to succeed, it is necessary to meet two major conditions: !▪ Organizational readiness
– Is there a clear M&E framework with targets?
– Is the planning process evidence-based? – Is the TT focused on achieving results (leadership, communications) !▪ Technical readiness - Do the staff have the relevant experience and skills?
WORKSHEET
![Page 6: TTIPEC: Monitoring and Evaluation (Session 2)](https://reader034.vdocument.in/reader034/viewer/2022051817/54832c55b4af9f910d8b4934/html5/thumbnails/6.jpg)
What do we monitor? Inputs. Activities. RESULTS.
WORKSHEET
![Page 7: TTIPEC: Monitoring and Evaluation (Session 2)](https://reader034.vdocument.in/reader034/viewer/2022051817/54832c55b4af9f910d8b4934/html5/thumbnails/7.jpg)
Results chain
Resources Processes
Within control of the organization Partial Org. Control No Org. Control
Sphere of Control: Advocacy, Tactics, Inputs, Activities, Outputs
Sphere of Influence: Changes in behavior,
relationships, attitudes, agenda
Sphere of Concern: Policy
& Political Context
WORKSHEET
WORKSHEET
![Page 8: TTIPEC: Monitoring and Evaluation (Session 2)](https://reader034.vdocument.in/reader034/viewer/2022051817/54832c55b4af9f910d8b4934/html5/thumbnails/8.jpg)
Results chain – ExampleWORKSHEET
![Page 9: TTIPEC: Monitoring and Evaluation (Session 2)](https://reader034.vdocument.in/reader034/viewer/2022051817/54832c55b4af9f910d8b4934/html5/thumbnails/9.jpg)
Indicators, baseline and targets (& Planning)WORKSHEET
![Page 10: TTIPEC: Monitoring and Evaluation (Session 2)](https://reader034.vdocument.in/reader034/viewer/2022051817/54832c55b4af9f910d8b4934/html5/thumbnails/10.jpg)
Strategies & approaches for identifying evidence of change
• Signposts
– output log, web analytics, media tracking, citation tracking
– discourse analysis, social network analysis
• Outcome (or influence) log
– record of interactions with decision-makers, anecdotes shared by stakeholders (e.g. with the director)
• Feedback mechanisms
– stakeholder surveys
– interviews
– focus group discussions
WORKSHEET
![Page 11: TTIPEC: Monitoring and Evaluation (Session 2)](https://reader034.vdocument.in/reader034/viewer/2022051817/54832c55b4af9f910d8b4934/html5/thumbnails/11.jpg)
Integrated Learning Processes in M&E - Illustration
jan marfeb apr sep novoct decmay julyjune aug
Quarterly report
Half-yearly & yearly report
WORKSHEET
![Page 12: TTIPEC: Monitoring and Evaluation (Session 2)](https://reader034.vdocument.in/reader034/viewer/2022051817/54832c55b4af9f910d8b4934/html5/thumbnails/12.jpg)
BREAK Questions, Answers, Comments
![Page 13: TTIPEC: Monitoring and Evaluation (Session 2)](https://reader034.vdocument.in/reader034/viewer/2022051817/54832c55b4af9f910d8b4934/html5/thumbnails/13.jpg)
WEBSITES: www.raguide.org www.resultsaccountability.com
BOOK ORDERS: www.trafford.com www.amazon.com
Results Accountability Decision-makingand Budgeting
Fiscal Policy Studies Institute Santa Fe, New Mexico
![Page 14: TTIPEC: Monitoring and Evaluation (Session 2)](https://reader034.vdocument.in/reader034/viewer/2022051817/54832c55b4af9f910d8b4934/html5/thumbnails/14.jpg)
“If you do what you always did,
you will get what you always got.”
Kenneth W. JenkinsPresident, Yonkers NY NAACP
TO THINK OR NOT TO THINK ABOUT M&E? FOOD FOR THOUGHT…
![Page 15: TTIPEC: Monitoring and Evaluation (Session 2)](https://reader034.vdocument.in/reader034/viewer/2022051817/54832c55b4af9f910d8b4934/html5/thumbnails/15.jpg)
The first step in performance accountability is toDRAW A FENCE
Around something that has ORGANIZATIONAL OR FUNCTIONAL IDENTITY
The Whole Organization
Division ADivision B
Unit
Division C
FunctionUnit 1
WORKSHEET
![Page 16: TTIPEC: Monitoring and Evaluation (Session 2)](https://reader034.vdocument.in/reader034/viewer/2022051817/54832c55b4af9f910d8b4934/html5/thumbnails/16.jpg)
HOW DO WE THINK ABOUT WHO, WHAT & WHY WE
MEASURE?
![Page 17: TTIPEC: Monitoring and Evaluation (Session 2)](https://reader034.vdocument.in/reader034/viewer/2022051817/54832c55b4af9f910d8b4934/html5/thumbnails/17.jpg)
Different Kinds of Progress1. Data
a. Population indicators Actual turned curves: movement for the better away from the baseline.
b. Program performance measures: customer progress and better service: How much did we do? How well did we do it? Is anyone better off?
2. Accomplishments: Positive activities, not included above.
3. Stories behind the statistics that show how individuals are better off.
![Page 18: TTIPEC: Monitoring and Evaluation (Session 2)](https://reader034.vdocument.in/reader034/viewer/2022051817/54832c55b4af9f910d8b4934/html5/thumbnails/18.jpg)
What’s Next?A Basic Action Plan for Results Accountability
TRACK 1: POPULATION ACCOUNTABILITY● Establish results ● Establish indicators, baselines and charts on the wall● Create an indicators report card● Set tables (action groups) to turn curves
TRACK 2: PERFORMANCE ACCOUNTABILITY● Performance measures, and charts on the wall for programs, agencies and service systems● Use 7 Questions supervisor by supervisor and program by program in management, budgeting and strategic planning
![Page 19: TTIPEC: Monitoring and Evaluation (Session 2)](https://reader034.vdocument.in/reader034/viewer/2022051817/54832c55b4af9f910d8b4934/html5/thumbnails/19.jpg)
Input
Activity
OutputOutcome
Goal
Logic Model
WORKSHEET
![Page 20: TTIPEC: Monitoring and Evaluation (Session 2)](https://reader034.vdocument.in/reader034/viewer/2022051817/54832c55b4af9f910d8b4934/html5/thumbnails/20.jpg)
If I can…. Measure, Track, Improve & Show Performance
Education Performance Accountability
What’s in it for me?
Then…. here’s what’s in it for me:
! Improvement = more money
! Success at the State’s Accountability Game
! Student and Parent Satisfaction
! Tool vs. fingers crossed
! Knowing where your students are between achievement tests
! Knowing that you’ve made a difference
![Page 21: TTIPEC: Monitoring and Evaluation (Session 2)](https://reader034.vdocument.in/reader034/viewer/2022051817/54832c55b4af9f910d8b4934/html5/thumbnails/21.jpg)
WORKSHEET
![Page 22: TTIPEC: Monitoring and Evaluation (Session 2)](https://reader034.vdocument.in/reader034/viewer/2022051817/54832c55b4af9f910d8b4934/html5/thumbnails/22.jpg)
WORKSHEET
![Page 23: TTIPEC: Monitoring and Evaluation (Session 2)](https://reader034.vdocument.in/reader034/viewer/2022051817/54832c55b4af9f910d8b4934/html5/thumbnails/23.jpg)
WORKSHEET
![Page 24: TTIPEC: Monitoring and Evaluation (Session 2)](https://reader034.vdocument.in/reader034/viewer/2022051817/54832c55b4af9f910d8b4934/html5/thumbnails/24.jpg)
Turn the Curve Exercise: Program Performance5 min: Starting Points - timekeeper and reporter - identify a program to work on - two hats (yours plus partner’s)10 min: Performance measure baseline - choose 1 measure to work on – from the lower right quadrant - forecast – OK or not OK?15 min: Story behind the baseline - causes/forces at work - information & research agenda part 1 - causes15 min: What works? (What would it take?) - what could work to do better? - each partners contribution - no-cost / low-cost ideas - information & research agenda part 2 – what works
10 min: Report Convert notes to one page
Two pointers to action
WORKSHEET
![Page 25: TTIPEC: Monitoring and Evaluation (Session 2)](https://reader034.vdocument.in/reader034/viewer/2022051817/54832c55b4af9f910d8b4934/html5/thumbnails/25.jpg)
Program: _______________Performance Measure
(Lay definition)PerformanceMeasureBaseline
Story behind the baseline --------------------------- --------------------------- (List as many as needed)
Partners --------------------------- --------------------------- (List as many as needed)
Three Best Ideas – What Works 1. --------------------------- 2. --------------------------- 3. ---------No-cost / low-cost
ONE PAGE Turn the Curve Report: Performance
Sharp Edges
4. --------- Off the Wall
4. --------- Off the Wall
WORKSHEET
![Page 26: TTIPEC: Monitoring and Evaluation (Session 2)](https://reader034.vdocument.in/reader034/viewer/2022051817/54832c55b4af9f910d8b4934/html5/thumbnails/26.jpg)
2 customers
1 measure
1 measure
Discuss how we’re doing
2 partners
2 what works ideas (1 nc/lc)
Discuss how to implement
20 MinuteExercise
40
WORKSHEET
![Page 27: TTIPEC: Monitoring and Evaluation (Session 2)](https://reader034.vdocument.in/reader034/viewer/2022051817/54832c55b4af9f910d8b4934/html5/thumbnails/27.jpg)
1 population
1 result
1 indicator
Discuss how we’re doing
2 partners
2 what works ideas (1 nc/lc)
Discuss how to implement
20 MinuteExercise
1. What population are we concerned about? (e.g. U.S. population)
2. What condition(s) do we want for this population? (Americans are free of cancer.)
3. How could we measure these conditions? ( (e.g. cancer rates)
4. How are we doing on the most important of these measures? (baseline history)
5. Who are the partners with a role to play in doing better?6. What works – what would it take – to do better? (What is our role?)7. What do we propose to do?
POPULATION ACCOUNTABILITYWORKSHEET
![Page 28: TTIPEC: Monitoring and Evaluation (Session 2)](https://reader034.vdocument.in/reader034/viewer/2022051817/54832c55b4af9f910d8b4934/html5/thumbnails/28.jpg)
Results Based Grantmaking ExerciseA. Result _________________________B. Indicators 1. _______________ 2. _______________ 3. _______________
For each: Getting Better Getting Worse or About the Same?
C. Story behind the baselines (Why better or worse?) __________ __________ __________ __________
D. Partners with a role to play __________ __________ __________ __________
E. What would it take? (5 most important strategies) 1. _____________ 4. ______________ 2. _____________ 5. ______________ 3. _____________F. Our Roles (3 best ideas) 1. __________________ 3. _________________ 2. __________________ Off the wall:. _________
10 min:
10 min:
10 min:
15 min:
10 min:
WORKSHEET
![Page 29: TTIPEC: Monitoring and Evaluation (Session 2)](https://reader034.vdocument.in/reader034/viewer/2022051817/54832c55b4af9f910d8b4934/html5/thumbnails/29.jpg)
Performance Measurement: How to Develop Them Sectoral Examples
See Also: raguide.org Questions 3.10 and 3.11
![Page 30: TTIPEC: Monitoring and Evaluation (Session 2)](https://reader034.vdocument.in/reader034/viewer/2022051817/54832c55b4af9f910d8b4934/html5/thumbnails/30.jpg)
● # of people served
● % participants who got jobs
● staff turnover rate
● # participants who got jobs
●% of children reading at grade level ● cost per unit of service
● # applications processed
● % patients who fully recover
What Kind of PERFORMANCE MEASURE?
Upper Left
Lower Right
Upper Right
Lower Left
Lower Right
Upper Left
Upper Left
Lower Right
QUANTITY !!
How much did we do?
QUALITY !!
How well did we do?
E F O R T
E F E C T
Is anyone better off?
WORKSHEET
![Page 31: TTIPEC: Monitoring and Evaluation (Session 2)](https://reader034.vdocument.in/reader034/viewer/2022051817/54832c55b4af9f910d8b4934/html5/thumbnails/31.jpg)
How much did we do?
Professional Development
How well did we do it?
Is anyone better off?
Quantity Quality
E
ffect
Effo
rt
Customers # teachers served# students# parents Activities # training sessions# hours of training # study groups# TA contacts # mentoring relationships
Common measures% teachers certified in subjects they teach% customers satisfied with how service is delivered % staff with good morale Unit cost per training unit (HL #3) Activity specific measures% positive evaluations on training% attendance in study groups% of faculty participating in study groups % mentoring pairs who complete mo. rept. % mentees satisfied with match
# # !# # !# !# # #
% teachers satisfied that prof dev helped them be a better teacher (DDA #2) % teacher retention rate - total - “quality” teachers (DDA #1)% mentees who report relationship helped build their skills % teachers who demonstrate new skills - by observation (HL #2) - by self report % students in “trained” classrooms with improved test scores (HL #1)% students with reduced behavior problems % students with good school attendance% teachers who report better use of time
SAMPLE WORKSHEET
![Page 32: TTIPEC: Monitoring and Evaluation (Session 2)](https://reader034.vdocument.in/reader034/viewer/2022051817/54832c55b4af9f910d8b4934/html5/thumbnails/32.jpg)
How much did we do?
Professional Development
How well did we do it?
Is anyone better off?
Quantity Quality
E
ffect
Effo
rtNumber of students &
teachers served !
Number of workshops
Unit cost !
% of teachers with PD plan
# teachers who used training (by self rept or observation) !# Teachers retained !# Achievement in classroom with trained teachers
% teachers who used training (by self rept or observation) Teacher retention rate !Achievement rates in classroom with trained teachers
![Page 33: TTIPEC: Monitoring and Evaluation (Session 2)](https://reader034.vdocument.in/reader034/viewer/2022051817/54832c55b4af9f910d8b4934/html5/thumbnails/33.jpg)
How much did we do?
Environment: Water Quality
How well did we do it?
Is anyone better off?
Quantity Quality
E
ffect
Effo
rtNumber of
stream stations monitored
Average sitesper monitorper month
Number miles of healthy streams
Percent miles of healthy streams
![Page 34: TTIPEC: Monitoring and Evaluation (Session 2)](https://reader034.vdocument.in/reader034/viewer/2022051817/54832c55b4af9f910d8b4934/html5/thumbnails/34.jpg)
How much did we do?
Commerce Tourism
How well did we do it?
Is anyone better off?
Quantity Quality
E
ffect
Effo
rtNumber of
books sent to interested
consumers
Cost perinquiry
delivered
Number oftourists
Number touristbusinesses
Tourism market share
Growth in tourist industry
![Page 35: TTIPEC: Monitoring and Evaluation (Session 2)](https://reader034.vdocument.in/reader034/viewer/2022051817/54832c55b4af9f910d8b4934/html5/thumbnails/35.jpg)
How much did we do?
Personnel Department
How well did we do it?
Is anyone better off?
Quantity Quality
E
ffect
Effo
rtNumber ofApplicantsProcessed
AverageRecruitment
Period
Workforce New Hires
Workforce Turnover Rate
(non-promotions)
CustomerSatisfaction
![Page 36: TTIPEC: Monitoring and Evaluation (Session 2)](https://reader034.vdocument.in/reader034/viewer/2022051817/54832c55b4af9f910d8b4934/html5/thumbnails/36.jpg)
How much did we do?
Every Program(Financial Performance Measures)
How well did we do it?
Is anyone better off?
Quantity Quality
E
ffect
Effo
rt
Cost(total)
UnitCost
Benefit Value (total)
Cost-Benefit(ratio)
![Page 37: TTIPEC: Monitoring and Evaluation (Session 2)](https://reader034.vdocument.in/reader034/viewer/2022051817/54832c55b4af9f910d8b4934/html5/thumbnails/37.jpg)
How much did we do?
Corrections
How well did we do it?
Is anyone better off?
Quantity Quality
E
ffect
Effo
rt # Inmates Rate ofovercrowding
# Recidivism
% Recidivism
![Page 38: TTIPEC: Monitoring and Evaluation (Session 2)](https://reader034.vdocument.in/reader034/viewer/2022051817/54832c55b4af9f910d8b4934/html5/thumbnails/38.jpg)
How much did we do?
Blank
How well did we do it?
Is anyone better off?
Quantity Quality
E
ffect
Effo
rtX X
X X
![Page 39: TTIPEC: Monitoring and Evaluation (Session 2)](https://reader034.vdocument.in/reader034/viewer/2022051817/54832c55b4af9f910d8b4934/html5/thumbnails/39.jpg)
PLANNING ALONG M&E
OBJECTIVES
PERFORMANCE
INDICATORS
INPUTSACTIVIT
IESOUTPUTS
OUTCOMES
IMPACT TIMELINERESPONSIBILITY
WORKSHEET
![Page 40: TTIPEC: Monitoring and Evaluation (Session 2)](https://reader034.vdocument.in/reader034/viewer/2022051817/54832c55b4af9f910d8b4934/html5/thumbnails/40.jpg)
More ResourcesONLINE RESOURCES http://resultsaccountability.com/ !http://researchtoaction.org !http://www.dgmt-growingconfidence.co.za/ !YOUTUBE TRAINING VIDEOS !https://www.youtube.com/watch?v=MUSzePBTpAY !https://www.youtube.com/watch?v=2mBcb-AQ8xQ !https://www.youtube.com/watch?v=SZqOKW2_jNU !https://www.youtube.com/watch?v=qmrDNuJzRv4
![Page 41: TTIPEC: Monitoring and Evaluation (Session 2)](https://reader034.vdocument.in/reader034/viewer/2022051817/54832c55b4af9f910d8b4934/html5/thumbnails/41.jpg)
More Resources M&E Online Tools
!ALT METRICS http://www.researchtoaction.org/2013/09/how-to-harness-the-power-of-altmetrics-a-researchers-guide/ http://www.researchtoaction.org/2013/08/altmetrics-and-the-global-south-increasing-research-visibility/ !USING GOOGLE TO TRACK AND IMPROVE YOUR RESEARCH IMPACT http://www.researchtoaction.org/2013/08/using-google-to-track-and-improve-your-research-impact/ http://opendocs.ids.ac.uk/opendocs/handle/123456789/2878#.U8VcXl7yDwJ !GOOGLE ANALYTICS http://www.researchtoaction.org/2013/11/key-indicators-for-google-analytics/ http://www.google.com/analytics/ !HOW TO GET STARTED WITH GOOGLE ANALYTICS http://mashable.com/2011/05/23/how-to-use-google-analytics/ !
M&E in Theory and Action
!MONITORING & EVALUATION: SOME TOOLS, METHODS & APPROACHES http://gametlibrary.worldbank.org/FILES/215_M&E%20Guidelines%20on%20Indicators%20and%20Data%20Sources.pdf !THEORY OF CHANGE VS. LOG FRAMES http://www.researchtoaction.org/2012/05/theory-of-change-vs-logframes/
http://www.knowledgebrokersforum.org/wiki/item/theory-of-change-vs-logframes
http://mande4mfi.wordpress.com/category/theory-of-change/
!CASE STUDIES ON M&E https://sites.google.com/site/newmediacommunications2010/mobiles-and-m-e/2-case-studies !EXAMPLES OF PROGRAM OUTCOMES www.whatworks.org
EVALUATION PLANS
http://nonprofitanswerguide.org/faq/evaluation/what-is-an-evaluation-plan-and-why-create-one/
![Page 42: TTIPEC: Monitoring and Evaluation (Session 2)](https://reader034.vdocument.in/reader034/viewer/2022051817/54832c55b4af9f910d8b4934/html5/thumbnails/42.jpg)
Questions/Comments/Feedback
![Page 43: TTIPEC: Monitoring and Evaluation (Session 2)](https://reader034.vdocument.in/reader034/viewer/2022051817/54832c55b4af9f910d8b4934/html5/thumbnails/43.jpg)
For further inquiries, please contact:
Dr. Modupe Adefeso-Olateju, [email protected] Rebecca O. Doherty, [email protected]
THANK YOU