how to plan and manage an environmental education program – and evaluate its success philip cox,...
Post on 25-Dec-2015
218 Views
Preview:
TRANSCRIPT
How to plan and manage an
environmental education program –
AND evaluate its success
Philip Cox, Plan:Net Limited (www.plannet.ca)
Alberta Council for Environmental Education (ACEE)
October 18th Webinar
A craft that comes by different names…
Outcomes MeasurementPerformance Measurement and EvaluationOutcomes MappingManaging by ResultsResults Based Management
A craft that comes by different names…
Outcomes MeasurementPerformance Measurement and EvaluationOutcomes MappingManaging by ResultsResults Based Management
Introducing Three Outcomes Measurement Tools…
Logic Model
Risk Analysis and Management Table
Monitoring and Evaluation Plan
Project Initiation
Proposal Development
Donor Appraisal
Start Up
Implementation
Monitoring / Evaluation
Phase Out
1.The basis for an introductory conversation with stakeholders
2. The centrepiece of your proposal or plan
3.The basis for workplans and job descriptions
4. A management reference during team meetings
5.A guide for monitoring and reporting on progress & adjusting strategies
6.An aid in designing an evaluation
Logic Model, Risk Analysis and Management Table, and Monitoring and Evaluation Plan are relevant to all parts of the program cycle…
What is Outcomes Measurement?
It is a way of thinking and a set of tools to help us:1. Convert a project idea into a set of activities and
anticipated changes in a way that is:• Mindful of context• In keeping with the time and resources available
2. Keep track of progress against the plan and use this information to:• Make management decisions, • Service accountability relationships and• To be a learning organization.
Outcome Measurement is Part of a Global Trend
• A response to new realities...
– Need to demonstrate value to citizens and taxpayers…
– A response by funding bodies around the globe who are under pressure to provide evidence that their programs achieve results
– EEC, World Bank, USAID, UN, WFP and others need to provide credible evidence of results they achieve
– By mid 90’s, almost all OECD governments had adopted RBM approaches
– In Canada - federal government and all provinces
– Increasingly philanthropic organizations such as Alberta Ecotrust Foundation, EcoAction, etc.
–
What does this mean in practical terms?
• When making a plan, or reporting on progress, we are challenged like never before to:
• Describe the connection between the work we do and the difference we make
If you are planning…• A new web-based information portal for educators on
climate change• You might focus attention on…
– The quantity and quality of information loaded on to the portal– The ‘user-friendliness’ of the web site– The number of users and how they are accessing the portal– And also…– What teachers are looking for on the portal– How they are using the information to inform their teaching– How the teaching-learning experience in the classroom is
altered
Exercise
• Think about a project that you know well
• Craft two outcome statements that reflect the change you want to see
• Share your statement with your neighbour and then in plenary
Where can Outcomes Measurement be used in your
organization?
• Organization as a whole –Donor funded programs
• Partner organizations– Individual Partner Initiatives/Projects
Three Outcomes Measurement Tools
Logic Model
- “Measuring Success…” pages 19 - 23
Risk Analysis and Management Table
Monitoring and Evaluation Plan
The Logic Model• Outcomes Measurement practice revolves
around the logic model
Splash and Ripple!
Splash and Ripple !Another way of thinking about
Activities and Outcomes
Situation Assessment
Before beginning a results based plan..
1. Looking Inward
- Mandate- Competencies- Experience
2. Looking Outward- Issues & Opportunities
3. Clarifying Priorities
Situation Assessment - Stakeholders
Influence
Inte
rest
or
Impo
rtan
ce High Interest/ Importance, High Influence
Low Interest/ Importance, High Influence
Low Interest/ Importance, Low Influence
High Interest/ Importance, Low Influence
Build coalition of support
Spend more time on the other Stakeholders
Consider ways to strengthen or protect interests
Caution - be aware of risks or obstacles
HIGH
LOW
HIG
H
LOW
Discussion
• Generate a list of groups with a stake in the success of your project
• Place them on the stakeholder matrix according to their current– Interest/importance– Influence
Logic Model Format - One VersionInputs
Inputs
Inputs
Inputs
Inputs
Activities
Activities
Activities
Activities
Activities
Impact
Outcomes
Outcomes
Outcomes
Outputs
Outputs
Outputs
Outputs
Outputs
Logic Model Format - Another VersionUltimate
Outcomes
How
What we want
Why
Inputs
ImmediateOutcomes
ImmediateOutcomes
ImmediateOutcomes
ImmediateOutcomes
Activities Activities Activities Activities Activities
Outputs Outputs Outputs OutputsOutputs
Intermediate Outcomes
Intermediate Outcomes
Intermediate Outcomes
Inputs…
• Are the human and physical resources that make a program or project possible.
• Links outcomes logic to budgeting
Example: Train and Mentor Environmental Educators in Outcomes Measurement
Ultimate Outcomes
Intermediate Outcomes
Immediate Outcomes
Outputs
Activities
Inputs Trainers, Materials, Equipment, Communications…
Activities
• Describe the essential work of the program or project – what you DO
• Should be summarized in no more than 5 statements.
Example: Train and Mentor Environmental Educators in Outcomes Measurement
Ultimate Outcomes
Intermediate Outcomes
Immediate Outcomes
Outputs
Activities Assess training needs
Design training materials
Deliver training
Provide follow-up coaching
Inputs Trainers, Materials, Equipment, Communications…
Outputs…
• Short term…a change that is observable as you
complete the activity.
• Each output relates to one activity.• You have substantial control.• Involves
– a specific group of people - those directly involved in the activity, or
– the creation of a product
• The change helps make outcomes possible.
Example: Train and Mentor Environmental Educators in Outcomes Measurement
Ultimate Outcomes
Intermediate Outcomes
Immediate Outcomes
Outputs Defensible report issued
Relevant, user friendly training materials in place
Participants complete training
On-the-job queries addressed
Activities Assess training needs
Design training materials
Deliver training Provide follow-up coaching
Inputs Trainers, Materials, Equipment, Communications…
Immediate Outcomes…
• Observable within the project cycle, following activities
• Describes the potential created by the outputs
• You still have considerable control
• Centres on a very specific group of people - those directly involved in the activity or using the product.
Example: Train and Mentor Environmental Educators in Outcomes Measurement
Ultimate Outcomes
Intermediate Outcomes
Immediate Outcomes
Stakeholder commitment to a training strategy
Trained managers apply Outcomes Measurement approaches and tools in the design and implementation of the projects to which they are assigned
Outputs Defensible report issued
Relevant, user friendly training materials in place
Participants complete training
On-the-job queries addressed
Activities Assess training needs
Design training materials
Deliver training Provide follow-up coaching
Inputs Trainers, Materials, Equipment, Communications…
Intermediate Outcomes…
• Describe the key changes you want to make in the program/project
• Observable at or shortly after the completion of the program/project
• You have influence, but not control
• Usually involves a larger group of people than those directly involved in the program/project
• Are fewer in number; each flows naturally from several Immediate Outcomes
Example: Train and Mentor Environmental Educators in Outcomes Measurement
Ultimate Outcome
Intermediate Outcomes
Agency leaders make evidence- based, results- focused planning and management decisions
Donors and community stakeholders recognize the ‘value-added’ by the agency’s work
Immediate Outcomes
Stakeholder commitment to a training strategy
Trained managers apply Outcomes Measurement approaches and tools in the design and implementation of the projects to which they are assigned
Outputs Defensible report issued
Relevant, user friendly training materials in place
Participants complete training
On-the-job queries addressed
Activities Assess training needs
Design training materials
Deliver training Provide follow-up coaching
Inputs Trainers, Materials, Equipment, Communications…
Ultimate Outcome• Long term
…a picture of a preferred future; the reason why the program or project is important.
• You can only make a contribution and have an indirect influence.
• The change usually involves an even wider group of people
• Is observable well after the program or project is complete.
Example: Train and Mentor Environmental Educators in Outcomes Measurement
Ultimate Outcomes
People/groups identified in environmental education projects engage with the subject matter in a way that helps them become active stewards of the environment
Intermediate Outcomes
Group leaders make evidence- based, results- focused planning and management decisions
Donors and community stakeholders recognize the ‘value-added’ by the group’s work
Immediate Outcomes
Stakeholder commitment to a training strategy
Trained educators apply Outcomes Measurement approaches and tools in the design and implementation of the projects to which they are assigned
Outputs Defensible report issued
Relevant, user friendly training materials in place
Participants complete training
On-the-job queries addressed
Activities Assess training needs
Design training materials
Deliver training Provide follow-up coaching
Inputs Trainers, Materials, Equipment, Communications…
A Progression of Results - Typical Terms
What we Need - Staff Equipment Supplies Office Space Services
Inputs
What we Do - Research Build Train/Mentor Promote Procure Convene
Activities
What we Produce - Networks Courses Trained participants Assessments Publications Infrastructure
Outputs
Capacities Developed - Institutional exchanges/collaborations
Applied knowledge Functioning spaces/infrastructure
System improvements/adaptations New/Improved leadership
Aligned Decision-making
Immediate Outcomes
Performance Improved - Lifestyle shifts
Institutional behaviour changes - policy & programming
Social mobilization
Intermediate Outcomes
Conditions - Social Economic
Civic Cultural Environmental
Ultimate Outcomes
Logic Model Quiz
“Organize a Provincial Climate Change Summit”
Making a Logic Model - Some Tips
1. Vantage Point• Logic Models can be written from many
different points of view - • it is important to be clear who is doing the
activities - spending implementation funds • there is normally only one vantage point in a
framework, it could be• a single entity, or • shared across two or more entities that are
working in partnership
2. When building an logic model consider three variables…
a. Time• Outputs - Immediate
• Immediate Outcomes - within time-frame following related activities
• Intermediate Outcomes - by the end of the project or shortly afterward
• Ultimate Outcomes - beyond the project
b. Control• Outputs - substantial control
• Immediate Outcomes - still considerable control
• Intermediate Outcomes - direct influence
• Ultimate Outcome - indirect influence
• Diffusion• Outputs - persons directly involved in the activity
• Immediate Outcomes - same, possibly early interaction with additional people closely connected (e.g. users, colleagues, family members)
• Intermediate Outcomes - larger teams, organizations, population segments
• Ultimate Outcome - systems (communities, networks, societies)
3. Proportion…• Outputs, outcomes and impacts vary in their
magnitude according to how big (in money terms) and how long the project is.
• If you have a three week, $5,000 project, your outcomes should reflect what is reasonable to expect at three weeks and after $5,000 has been spent
• If you have a three year, $500,000 project, your outcomes should reflect what is reasonable to expect at three years and after $500,000 has been spent
4. Scale - macro to micro
Outcome Measurement can be used at any scale - but which is best for you?
Writing Activity and Results Statements - Tips
5. Separating the Action from the Change– Activities - where you describe what you, as the implementing
group, are DOING
– Results (output and outcome - immediate, intermediate and ultimate) - where you describe what is DIFFERENT
– For example… • Activity: Train 25 participants in Outcomes Measurement• Result: Participants apply Outcomes Measurement knowledge in
when making project plans
6. Avoid ‘run-on’ statements - for example:“Rural population retained in rural areas THROUGH
improving their living conditions BY MEANS OF enhancing rural income, better access to health, social services and
infrastructure.”-An impact statement for a rural development project
“Improved political freedoms of local communities TO access, participate and CONTRIBUTE TOWARDS sustainable
regional development.”- An impact statement for a governance related project
Difficult to locate the actual change you seek
7. Do not lose sight of the people in your outcomes statements.
– Sometimes we write statements without naming the people or groups who are experiencing the change. For example…
“increased awareness”
- Try to include the subject in the statement at all times
8. Avoid tentative wording like: “able to”, “have increased capacity to…”, “contribute toward”. These are vague in meaning and very hard to measure.
Now your turn…..• Describe your group project in
a results logic model?• 1 Ultimate Outcome • 2 Intermediate Outcomes• 3-4 Immediate Outcomes• 4-6 Activities and Outputs
Sample Logic Model WorksheetSITUATION (I.e. context, problem, identified needs, mandate, objectives)
Organization :
Name of project :
Duration :
How ? What do we want ? Why ?
Inputs Activities Outputs Immediate outcomes
Intermediate outcomes
Ultimate outcome
Three Outcomes Measurement Tools
Results Logic Model
Risk Analysis and Management Table Monitoring and Evaluation Plan
Enablers and Constraints…
• Are forces affecting any part of your project both positively and negatively
Risk Analysis MatrixEffects
Significant (3)
Moderate (2)
Minor (1)
Low (1) Medium (2) High (3)
Likelihood
Monitor Risks
Acceptable Risks
Acceptable Risks
Monitor Risks
Monitor Risks
Risk Mitigation Required
Risk Mitigation Required
Risk Mitigation Required
Acceptable Risks
Project Example…
Climate Change Conference
Immediate Outcome:Conference participants deliberate across stakeholder lines on conference topics; they identify climate change strategies around which there is common agreement, as well as strategy areas requiring further discussion
1. What could hamper progress?
2. What is the likelihood of a problem?
3. What effect would the problem have on the Outcome?
• Risk mitigation strategies should reduce the likelihood of an undesirable event, or minimize the effect the event could have on the project.
• Risk areas that remain above the level of acceptable risk should be monitored using risk indicators.
Risk Mitigation & Monitoring
A Typical Risk Analysis and Management Table
Description of Risk
Likelihood of Occurrence
Effect on Planned Results (outputs/
outcomes/ impact)
Rank in Importance
1 = highest
Owner of the Risk
Risk Mitigation Strategy
Risk Indicators
Intermediate outcome level
Immediate Outcome level
Output level
Three Outcomes Measurement Tools
Logic Model
Risk Analysis and Management Table
Monitoring and Evaluation Plan - “Measuring Success…” Appendices, pages 44 - 66
Monitoring and Evaluation - The Distinction
• MonitoringA continuous (or regular) self-assessment of progress -collecting data and comparing current performance with planned activities and outcomes.
• EvaluationIndependent (often external), periodic, strategically focused assessment of a program/project’s continuing relevance, management, governance, results, coverage, external relationships, sustainability, etc.
Conventional vs Participatory Monitoring & Evaluation - Contrasts
Empowerment - to help people initiate, navigate, control
Accountability - to address donor requirements
More frequent, small scale evaluations
Usually Mid and End of Project
Self-evaluation, simple methods adapted to local conditions; open, immediate sharing of results through local involvement
Focus on scientific objectivity, evaluator distanced from other participants; delayed and limited access to results
People identify their own indicators of success
Predetermined indicators of success
Community members, project staff, facilitator(s)
External experts
ParticipatoryConventional
Why
When
How
What
Who
Adapted from Narayan-Parker, 1993: 12 (taken from Estrella and Gaventa, 1998)
Different kinds of Evaluation
• Formative - mid stream, where inquiry focused on areas where the project can improve
• Summative - at the end, where inquiry focused on end results and lessons for future practice
• What is your experience with these two kinds of evaluation?
Indicators…• Evidence that the
program is producing expected results at the output and outcome levels.
• They can be expressed quantitatively and qualitatively.
• Information collected should help managers make wise decisions.
What indicators tell us about the wind…
Who has seen the wind?
Neither you nor I.
But where the trees bow down their heads,
The wind is passing by.-The Wind, Christina Rosetti, 1830-1894
Quantitative IndicatorsExpressed as a percentage, ratio, proportion, or number that can be analyzed statisticallyExamples?
– Number of conferences/public engagement activities held
– Ratio of males to females attending public meetings
– Number and type of research grants distributed
– Proportion of targeted schools making requests for assistance with educational resources
Quantitative Indicators -Your Turn
?• Climate Change
Conference Project
• Immediate Outcome
• Major media outlets cover the Climate Change Summit with ample in-depth news analysis pieces
Qualitative IndicatorsExpressed as a change or comparison between two states of understanding or experience that can be analyzed for patternsRelies on people’s judgment or perceptionsExamples:
– Trends in the way community newspapers cover citizen actions to reduce environmental impacts
– Before-after comparison in the way the largest five land developers are handling wetland areas zoned for development
Qualitative Indicators -Your Turn
?
• Climate Change Conference Project
• Immediate Outcome
• Major media outlets cover the Climate Change Summit with ample in-depth news analysis pieces
Qualitative Indicators - Setting out Markers of Progress
• Sometimes you cannot meaningfully track progress by counting “units” at the beginning (baseline), middle and end – Indicators are not always as easy as “number of people
trained”…
• You may need to set out…– Stages of completion, or– Markers of progress
Qualitative Indicators - Setting out Markers of Progress
• Consider “policy formulation”…– Indicator is ‘Progress in updating ‘x’ policy’
• You could say…– Baseline - ‘0’ updated policy – Target - ‘1’ updated policy
• But how useful is that from a management point of view?
Qualitative Indicators - Setting out Markers of Progress
• Would be more useful to identify progress markers. For example:
a) Agreed need for updated policyb) Policy research underwayc) Policy options formulatedd) Discussion and approvals underwaye) Updated policy approvedf) Updated policy enacted
• Baseline might be ‘a’• Yr 1 target might be ‘c’• Yr 3 target might ‘e’
Indicator ChecklistClimate Change Conference -
Key stakeholder groups commit financial and human resources to implement the Action Plan
Possible Indicators - How well do they score?
a. Trends in the level of financial commitment, by action plan strategy, by source
b. Number and type of organizations publicly endorsing the Climate Change Action Plan
c. Number and type of initiatives underway in the Province, by action plan strategy
Validity - Does it measure the result? Yes No
Is it cost-effective to collect the information?
Yes No
Does it help understand how the program or project is affecting men and women, and/or specific sub-groups of people differently?
Yes No
Does it provide useful information for management decisions?
Yes No
Does it communicate well to stakeholders (public, policy makers, donors)?
Yes No
Indicators, Baseline, and Target
Indicator
Baseline
Target
Meaning
The information to be collected
What the indicator would tell you at the beginning
What you want the indicator to tell you at a specific point in the future
Example
The number of visits to the organization’s new website
720
2,000
Sources of Information
• Data Sources are the people or documents that provide data, not the method of collecting data
• Sources of performance information include: people, individual beneficiaries, groups of beneficiaries, organizations, partners, and documents, etc.
• To ensure reliability, try not to change data sources over time.
Methods of Gathering Data
See “Measuring Success…” Appendix I, Pros and Cons, Pages 49 - 51; Appendix IV, Samples ,Pages 55 - 66
Collecting the Data
• Conventional Methods - surveys, focus groups, observation, document reviews, counts, etc.
Collecting the Data
• Participatory Reflection and Action Methods - modeling/mapping, ranking exercises, calendars, walks, historical profiles
Frequency of Collecting Data
• For each indicator, determine how often data will be collected.
• Indicators for outputs and immediate outcomes are more likely to be collected on a quarterly or six-monthly time period, for intermediate outcomes - less frequently
Responsibility - who does it?
• Spell out clearly who will do the collection and analysis of the data for each indicator
• Ensure local capacity is built and is responsible to ensure sustainability.
• Decentralize responsibility as much as possible.
A Typical Monitoring and Evaluation Plan- Completed example on page 30 “Measuring Success…”
Results Levels
Measurement
IndicatorsBase-Line
Target Data Source/ Method
Report Frequency
& Resp.
Intermediate Outcome
Immediate
Outcome
Output
Now your turn…..
1. Take one outcome statement, brainstorm indicators
2. Use the indicator checklist to refine your best indicators
3. Identify the source, means of data gathering (a.k.a. ‘instrument’), and frequency
4. Repeat, if time permits
5. Share with a neighbour
top related