ECD in the 21st Century: What’s The Best Way Forward?

Karen Russon and Michele TarsillaEvaluation Capacity Development Group (ECDG)

AEA Presentation - October 18, 2013

ECDG organized an Int’l Workshop on ECD 17-21 October 2011

Today is a sort of anniversary ..Two years ago today, evaluation practitioners, commissioners and members of Standards organizations gathered to discuss the current situation and the future of ECD.

As a Result of the Workshop, Four Key ECD Principles were Identified

Ownership

Integration

Usefulness

Relevance

1. “What does ECD look like in practice?”

2. “How could ECD processes be enhanced and evaluated in the future?”

Today’s Questions

Today’s ECD Stories Are Global

A Global ECD was Conducted to Understand ECD better

• Collection and Analysis of stories from multiple perspectives: actors, levels and contexts• Focus on:

-ECD partnerships -Infrastructure changes-Organizational processes improvements

Key ECD Trends

• Tri-partite ECD collaboration• Indigenous-to-indigenous ECD cooperation

• Creative use of action learning/research to enhance ECD buy-in

• ECD as a tool for social emancipation

A Few Selected ECD Stories

The Government of Papua New Guinea

Actors/Location: Gov’t Agencies, Papua New Guinea

Level: National, Provincial, Local

Sector: Government

Time period: 2003-present

Documentation: Paper presented at AES 2013

The Government of Papua New Guinea

Lessons Learned• Developing a “fit-for-use M&E system” was conducive to

enhancing coordination and harmonization among stakeholders in the public sector• The ECD participatory process eased the tensions among a

variety of ECD stakeholders at different levels• The availability of better data translated into stronger

district planning and decision-making.

The Papua New Guinea Assn. of Professional Evaluators (PNGAoPE)

Actors/Location: Australian Evaluation Society, PNGAoPE, Coffey Int’l, Expat evaluators, AusAID, Gov’t of Papua New Guinea

Level: Organizational

Sector: Evaluation, multi-sectors

Time period: 2006 – present

Documentation: Guest blog posted on www.ecdg.net

The Papua New Guinea Assn. Of Professional Evaluators (PNGAoPE)

Lessons Learned• The sustained support provided by international development

partners is often key to ECD success but it is not sufficient.• The official government recognition of a national evaluation

association was critical• The critical friend and support role played by expats could be

effective during the development of a national evaluation association

• The selection of topics to be discussed during monthly meetings was a sign of ownership and the first necessary step towards sustainability

Innovation NetworkActors/Location: non-profit providing consulting services in research and evaluation to other non-profits and funders, three funders and their respective grantees; Washington, D.C.

Level: Organizational

Sector: women’s economic empowerment, social justice, corporate philanthropy

Time Period: 2006-2011

Documentation: Personal communication with Innovation Network staff.

Innovation Network

Lessons Learned:• Provision of “default” services is better, especially if within the scope of an existing evaluation assignment

• ECD often consists of a Gradual Approach (shifting from funder’s interests to organizational needs and strategies)

• It is important to respond to strategic grantees’ demand (that is, demand for critical evaluation services) and not to confused or disjointed requests;

• The service provider should not be the one “running” the ECD program: leadership/management and internal staff or the funders themselves should play such role

DECI-2 Actors/Location: Funder, organizations, regional consultants

Level: Organizational and Regional

Sector: Information and Communication Technology (ICT) paired with UFE curriculum development

Time Period: (DECI-1: 2009-2011; DECi-2:2012-2015)

Documentation: E-mail Exchange with ECDG

DECI-2

Lessons Learned• Pairing ECD with capacity building in other technical sectors

reinforces learning and use;• Participatory curriculum development is lengthy but ensures

training uptake;• Two-level mentoring was effective: mentoring to partners and

mentoring to regional mentors (“at their own pace” without a classic workshop approach on pre-established dates );

• Mostly virtual + 2 face-to-face mentoring sessions;• Timeliness of support enhanced and upfront investment to ensure

organizational readiness;• Presented as a research project: joint learning- facilitator of

learning process NOT external expert;

Habitat For Humanity - LAC

• Actors/Location: HQ staff, regional managers, national technical specialists (planning and implementation officers) in Central and South America

• Level: HQ, Regional and National

• Sector: Housing Solutions, Community Development

• Time Period: 2009-2012

• Documentation: Direct Exchange with LAC Regional Office Coordinator

Lessons learned• The development of a qualitative instrument capturing outcome-

related information could become the entry point for a learning and collaborative ECD effort across levels;

• Getting managers and Implementers (not just researchers or evaluation specialists) involved in the development of a data collection tool made it possible to emphasize the need for timely and adequate resources as a prerequisite for better evidence;

• Balance between technical rigor (19 staff and 10 experts) and face validity is critical for capacity development within and outside an organization (the HFH tool was also validated through focus groups with target populations in four countries)

• Sustainability and scaling-up of the approach: this process will be replicated for the development of indicators on risk mitigation associated with HPC disaster response programs globally

Habitat For Humanity - LAC

Actors/Location: UN (IFAD), regional capacity building platform, indigenous population groups, evaluation professionals in the Latin American region

Level: national, regional, local

Sector: Agriculture, Rural Community Development

Time period: Participatory Research on the state of 15 projects M&E Systems (2004-2007) + Training on Innovative PME and Image-based M&E (2008-2010)

Documentation: Reflection Paper submitted by the Program Coordinator

PREVAL (Latin America)

PREVAL (Latin America)Lessons Learned• ECD could be a remarkable tool for social emancipation and an

effective enabler of transformational learning in marginalized communities• Joint Regional training aimed at both rural community leaders

and evaluation specialists enhanced the ECD process• The use of several communication means (talking maps,

thematic maps, photographs, info-graphs) and visits to indigenous communities enhanced inter-learning among the trainers’ workshop participants• The organization of peasant contests on who (within the

community) had used the evaluation findings more effectively than others was a good entry point for transfer of evaluation knowledge and skills in the following months

Ukk'aa – Alaskan ECD

• Actors/Location: ANZEA; indigenous research group from Massey University, NZ; Director & students at Interior-Aleutians Campus, Univ of Alaska Fairbanks; Evaluation Research Associates, Native Elders, Fulbright-scholar-in-residence; Ukk'aa - the Alaska Native group

• Level: Individual - statewide; organizational

• Sector: various

• Time period: 2008 – present

• Documentation: AJE article, email correspondence

Ukk'aa – Alaskan ECD

Lessons Learned• Interactions and collaboration among indigenous groups

promotes an authentic horizontal ECD process and collaborative learning

• Buy-in by native Elders critical to project success• Critical friends are often key enablers of ECD processes: New

Zealand Evaluation Association members (Maori) mentored Alaskan Native population on indigenous evaluation theory & practice

• A research grant might bring ECD actors to the table for the first time and enable the development of an ECD plan

Civic Theater (Kalamazoo, MI)

Actors/Location: Community theater, Kalamazoo, MI

Level: Organizational

Sector: Arts

Time period: ongoing

Documentation: Key informant interviews

Civic Theater (Kalamazoo, MI)Lessons Learned• The systemic integration of evaluation throughout an

organization where all of management acts as evaluators is a remarkable ECD strategy;

• Identifying and maximizing opportunities for conducting and using evaluation is a truly ECD-savvy strategy. Activities included:--Seeking informal feedback as patrons exit theater lobby;- -Using evaluation data for marketing purposes;- -Establishing a mechanism where staff and supervisors could evaluate each other;

--Evaluating the quality of vendors and software;

Quick Observations on the Process

The way a story was told was sometimes as illuminating as the content of the story. Two contrasting storytellers:• New Zealand (Maori) and Alaskan (Native) began telling their

story by introducing themselves, providing backgrounds and personal perspectives from participating in the ECD activity. • A different story of the creation of a new evaluation

association began by describing political/economic background as one of deficit, needing to be developed and improved.

What’s the Best Way Forward?

• Develop long-term collaborative relationships with funders as partners in the development process.

• Don’t Develop Capacity, work with actors who are WILLING and MOTIVATED to develop it further

• Work on at all three levels (individual, organizational and contextual) within your system

• Step back and let ownership emerge: Home grown, not externally driven - imposed by funders, gov't or others "forcing" evaluation on an institution/gov't.

• Need internal champions who are leaders/decision-makers and external champions who can act as critical friends/coaches/mentors.

What’s the Best Way Forward?• Give ECD a chance to take hold - long-term. Nonetheless, don't be

averse to change: tactics/strategies, partnerships, actors, policies, budgeting, training methods, and infrastructural arrangements

An evaluation system design should be appropriate to the needs of the institution (considering size, complexity, political environment, culture, etc.). Consider organizational design, job descriptionsevaluation approaches that are unique to meet those needs.

• Seeing the value of evaluation enhances its use/usefulness:providing evidence for policy-makers can dramatically affectresource Allocation

• Acknowledge the power of peer-to-peer support: indigenous-to-indigenous

“It takes time for an organisation to find its way into evaluation; and for evaluation to find its way into organisational functioning. This involves overcoming resistance, seeking best approaches, trial and error, staff training and exposure and eventual 'absorption' of the practices as part of ordinary functioning.”

Community Development Resource Assn. (CDRA), South Africa

QUOTE

Thank You Very Much!

Michele Tarsilla, Ph.D.E-mail: mitarsi@hotmail.com

Karen Russonkrusson@ecdg.net

Please follow us on:www.ecdg.net