combining utilization focused and developmental evaluation: … · 2014. 3. 26. · • utilization...
TRANSCRIPT
Combining Utilization Focused and Developmental Evaluation: The Case
of the Aflateen Social and Financial Education Program for Youth
Daniel Shephard & Simon Bailey
Aflatoun, Child Social & Financial Education
The Netherlands
Background: Context
• Aflatoun
• Provides Social and Financial Education to Other Organizations (Social Franchise)
• History
• Started as a small education programme in Mumbai in early 90s
• Secretariat for Global Expansion started in Amsterdam in 2005
• Focus on primary education
• Has partners in over 100 countries and work with over 2 million children per year
• Aflateen
• New curriculum and first experience with age group
• Five year partnership with MasterCard Foundation to roll out to 50 countries
• Period of curriculum development, then piloting, then re development, then roll out
• Mid Term Evaluation
• MC requested a mid term evaluation, not part of original contract
• Provided opportunity for Secretariat to co design evaluation
• Secretariat Perspective
• Wanted approach that reflected the stage of programme we were in (re development)
• Interested in how the process of supporting organizations had worked
• Settled on utilization focused evaluation as a potential option
1
Background: Process and Level of Assessment
• Selected Partnership for Evaluation • External
• Partnership of Utilization Focused External Evaluation team of two
• Internal
• Research Manager
• Partners & Consultants from 8 countries
• Secretariat level: • Assessment of the Aflateen curriculum development and pilot process to help the
Secretariat review and improve its rollout and support of the program
• 3 visits by Evaluator(s)
• (Co)writing
• 2 field visits by evaluators (focus groups with country/regional partners)*
• Surveys and Phone Interviews
• Implementation level (not covered in presentation) • Generating lessons for current and future partners.
• 2 field visits by evaluators (focus groups with country/regional partners)*
• 8 commissioned research projects (2 to 5K projects)
*This element provided data on both the secretariat functioning and program implementation
2
Background: Methodology Combination
• Utilization focused evaluation – Focus on Users, uses, key evaluation questions, and
facilitating use
• Developmental evaluation – Documenting a model in the making and learning from it
• UDFE instead of UFE
3
Simplified Model of Secretariat Social Franchise
4
Brainstorming & Clustering
5
Intended Uses: Course Correction and Improvement
Secretariat (By Topic)
1. The curriculum development process
• Define our desired curriculum development process for the future
2. E-learning
• Create a plan to scale up and broaden e-learning.
3. Partners / dissemination
• Align Aflateen partner support practices with strategic goals
4. Training
• Differentiate training process for Aflateen vs. Aflatoun as needed
5. Evaluation
• A methodology for research calls that improves their program
6
What was the Model Used?
7
Intended Users
Primary Intended USERS
• Curriculum Manager* and Programme Head
• Specialized Staff, Department Head, and Research manager*
• Programme Head, 2 Programme Staff, and Research Manager*
• Training Manager, 2 Programme Staff and Research Manager*
• Department Head, and Research manager*
Other USERS
• Programme Staff
• Are what we doing effective? How to improve?
• Management
• How do we manage programme expansion? How do we change? How do we
secure more funding?
• MasterCard Foundation
• Is this approach applicable to our portfolio of grantees?
*This is the same staff member
8
Actual Users
Primary Intended Users
• Not all primary Users available or full participants, traveling challenges
• Research Manager often mediator or primary participant
• Time pressure reduced input beyond research team
• Evaluators did think contribution was better than most cases!
Programme Staff
• Importance of in person discussions
• Team contributed (even beyond primary Users) and were affected
• Data collection and use process merged
• Often brought out and document tacit processes, perceptions, and practices
Management
• Process Ownership
• Reputation Management
• Follow Up on Recommendations
Mastercard
• Learning about methodology
• Interest in how organization used the approach and changed
9
Finalization of Uses: Process
• External researcher provided recommendation domains based on findings
which were then adapted, replaced or expanded by the Users.
• The final recommendations for use were drafted by Users prior to a final visit by
the researcher
• Recommendations were finalized during the last visit with involvement of the
management and full staff.
• Management determined much of the wording of the final recommendations.
• Donor asked for formal timelines for implementation
10
Finalization of Uses:
Curriculum Development
• Develop a Secretariat strategy to support country-level curriculum integration, including
national events and a guide for partners
• Modify and simplify the curriculum Contextualization Manual through the creation of simple
checklists, a list of most frequent adaptations, and FAQs, to increase use
• Document the process for the development of new curriculum based on experience from
Aflateen, Aflatoun and Aflatot as part of the Quality Assurance process
E-learning
• Disseminate Aflateen Digital learning to existing Aflateen partners
• Disseminate Aflateen Digital learning to youth organizations and other stakeholders who
are not currently partners
• Identify ‘Aflateen Digital Champions’ from amongst the youth who are actively using and
promoting the platform
• Optimize the platform on an ongoing basis via usability testing
Programme Support
• Strengthen national networks and partners that function as country level secretariats for
scale-up of Aflateen
• Strengthen direct linkages between partners regionally and globally by leveraging partner
skills and experience more strategically
• Ensure that Aflateen partners get more differentiated support and Aflateen specific partner
communication
11
Cont’d
Training
• Update the Aflateen training to strengthen content related to facilitation skills for training
youth and then conduct new regional master trainings in 2014
• Expand and strengthen the pool of Aflateen specific trainers
• Revise monitoring and evaluation tools for both Secretariat and Partner led trainings to
facilitate learning
Evaluation (Changed question)
• To prepare and implement a research plan that privileges proof of concept, scaling and
performance while linking partners to external capacity development opportunities
• Complete the creation of the Implementation & Monitoring Guide for monitoring
programme numbers. This should also capture variability of programme implementation
• Produce user-friendly guides and integrate them into programmatic support for
conducting research of various types
12
Actual Implementation of Uses
General Findings
• Some automatic or based on past understanding of issues
• Some instigated via the process itself
• Some came late in process after dealing with new information or having it presented to us
by the evaluator in challenging ways
Process
• Responsibility for driving the process shifted to Management level from research level
• Transition from recommendations with clear timelines to non-timelines in the final report
affected the actualization of recommendations
• To date, more likely to be completed if started in the evaluation or shortly thereafter than if
it was to be started at some time in the future
13
Contributing Factors to Use
Supporting
• Willingness of funder to consider learning as end goal of evaluation
• Allow Secretariat to define its own key questions
• Ability of Lead Evaluator to communicate with both team and funder (in person)
• Dedicated time for reflection for staff to consider what they were doing as a group
• Facilitated time as well as internal discussions
• Length of time allowed for continued progress and incorporation of findings
• Ability to modify evaluation questions to use (ex. evaluation) while maintaining evaluation’s
focus on uses
• Ability to have internal evaluation staff as part of process helped mediate concerns on
both sides and keep process going
Challenging
• Balancing demands of outcome evaluation with aim of UFD evaluation
• Transition from evaluation to use after end of report
• Key responsibilities had moved within the organization
• Keeping focus on the programme vs expansion to the whole organization
Point of Discussion
• Developmental approach as a key requirement for Utilization Focused Evaluation to
achieve change?
14
Aflatoun, Child Social & Financial Education
Stichting Child Savings International | Po Box 15991 | Spaklerweg 14 | 1096 BA Amsterdam
The Netherlands
P: +31 20 760 1340 | F: +31 20 626 2118 | www.Aflatoun.org
Thanks…. Questions, Clarifications, Discussions
For more information on all Aflatoun evaluation work visit
www.aflatoun.org/evaluation