stone soup webinar series · 2019-03-08 · webinar series with expert group april – july 2012...

EVPA and its Knowledge Centre are supported by:

Stone Soup Webinar Series

The adventure of impact evaluation: is it really that complex?

Lisa Hehenberger, Research Director, EVPA

June 7th 2013


Agenda

EVPA’s Impact Measurement Initiative

Development of the Practical Guide

Target Audience of the Practical Guide

Key Definitions in Impact Measurement

The Impact Measurement Process

2

1

3

4

5


EVPA’s Impact Measurement Initiative (IMI) 1

EVPA focuses on providing its members resources and knowledge to assist them in the development of strategy and best practice in the VP approach (incl. performance measurement, high engagement, non financial support)

IMI conceived to address concerns raised by EVPA members about guidance on how to work with the maze of existing methodologies, databases, tools & metrics

The Practical Guide provides a road map on how to actually implement an impact measurement system for a VP/SI investor and their SPO investees

It is neutral in terms of methodologies, databases, tools and metrics and does not “reinvent the wheel”

It is grounded in practice as it is the result of a multi-stakeholder initiative


IMI / Expert group 1

Expert group formed of 27 representatives (14 countries) from prominent organisations:


Development of the Practical Guide 2

Webinar Series with expert group

April – July 2012

Guide and Case Study Development

April – August 2012

1st Draft Guide & Workshop

November 2012

Consultation period 1st Draft November 2012-February 2013

“Version 1.0” Practical Guide

April 2013

The Practical Guide to Measuring and Managing Impact can be downloaded, free of charge, at: http://evpa.eu.com/knowledge-centre/publications/evpa-publications/

http://evpa.eu.com/knowledge-centre/publications/evpa-publications/






Target Audience of the Practical Guide

Recommendations relevant at the level of the social investor (i.e. the funder) and the investee level (i.e. the organisations they support)

Useful for:

Beginners in impact measurement: how to get started

Advanced practitioners: how to better integrate impact focus into everyday investment management decisions

3


Key Definitions in Impact Measurement 4

Inputs Activities Outputs Outcomes Impact

Organisation’s Planned Work Organisation’s Intended Results

Resources (capital, human) invested in the activity

Concrete actions of the organisation

Tangible products from the activity

Changes, benefits, learnings, effects resulting from the activity

Attributions of an organisation’s activities to broader & longer-term outcomes

€, number of people etc.

Development & implementation of programs, building new infrastructure etc.

Number of people reached, items sold, etc.

Effects on target population e.g. increased level of education

Take account of actions of others (alternative programs e.g. open air classes), unintended consequences etc.

€50k invested, 5 people working on project

Land bought, school designed & built

New school built with 32 places

Places occupied by students: 8

New students with access to education: 2


The Impact Measurement Process

5

The five steps of social impact measurement

Source: EVPA


The IM Process/ Step 1: Setting Objectives

For VP/SI, objectives should be set at:

- level of VP/SI: scope of impact measurement and objectives in terms of impact & relationship with investee

- level of investee: what are specific impact objectives – social problem, proposed solution, expected outcomes

More specific objectives means better preparation for impact measurement

Wide range of methods & tools available: theory of change, logic model, parts of methodologies like SROI or balanced scorecard

5


Step 1: Worked Example

VPO/SI investing in early stage SPOs in Africa to improve the lives of people living under the poverty line through investments in the water, sanitation and health sectors

Rationale for impact measurement: investment selection, ongoing monitoring, reporting

Example SPO: building and scaling viable sanitation infrastructure in Kenyan slums

Setting objectives for SPO:

Social problem: 2.6 billion people globally do not have access to adequate sanitation. Resulting disease & water pollution cause 1.7 million deaths each year. In Kenya’s slums, 8 million people lack access to adequate sanitation causing disease & death

Activities: (i) building network of low cost sanitation centres; (ii) distributing them through franchising to local entrepreneurs; (iii) collecting waste produced; (iv) processing waste into fertilizer and electricity

Expected outcomes: (i) access to sanitation; (ii) employment; (iii) health (users & overall slum); (iv) income (toilet operators); (v) environment (less waste in open waterways)

Unintended consequences: (i) job displacement; (ii) zero job growth

5


The IM Process/ Step 2: Analysing Stakeholders

“Any party effecting and/or affected by the activities of the organisation”

Understand expectations of stakeholders, their contribution to, & potential impact our work will have on them.

Involves two parts:

- Stakeholder identification: mapping, selecting (materiality / relevance) & understanding expectations

- Stakeholder engagement: opening communication & later (step 4) verifying if expectations have been met

Co-operation of main stakeholders critical in process

Key aspects:

- Equally important for VP/SI and SPO

- Begin by focusing on a small number of stakeholders but beware of missing large positive or negative issues

- Learning process!

5


Step 2: Worked Example

*Denotes focus of analysis given assessment of relevance by VPO/SI

Expectations:

Toilet user: clean toilet & may expect fewer health problems

Toilet operators: customers (i.e. income) & necessary franchisor support

Slum dwellers: fewer health problems (may not necessarily have this expectation)

5

Direct Indirect

Contributors Toilet operators*, SPO’s employees, VPO/SI

Gov’t health & sanitation orgs, Microfinance orgs

Beneficiaries Toilet users*, Toilet operators, Waste collectors

Slum dwellers*, SPO’s employees, Gov’t, Farmers, Existing fertiliser companies, Existing power companies


The IM Process/ Step 3: Measuring Results

A VP/SI must consider impact achieved by SPO and assess the impact of the VP/SI on the SPO

Outcomes: selected based on importance, materiality, usefulness, feasibility (of achievement not measurement)

Indicators: used to show progress towards or away from outputs and outcomes

Factors of a “good” indicator:

- Aligned with purpose of organisation

- Clearly defined, comparable with others - various libraries of standardised indicators exist such as IRIS, WikiVOIS

Impact: often involves many theoretical assumptions (drop off, displacement, deadweight, attribution…)

Recommendation: calculate outcomes, acknowledge/adjust for factors increasing/decreasing impact*

5


Step 3: Worked Example 5

Stakeholder Outcome Indicators

Slum dwellers Increased access to sanitation facilities

No. toilet units installed by SPO No. individuals that were clients using the toilets Increase (vs. beginning of SPO operations) in no. toilet units installed (by SPO or

any other organisation)

Slum dwellers Increased employment

% of community with some form of regular income through full time and part time work

No. employees (toilet operators, waste collectors etc. of SPO incl. full time and part time (but not temporary) that reside in community where toilets are situated

Slum dwellers Improved health

No. days a slum dweller has not been able to be up and about during the reporting period due to some stomach related illness

No. outbreaks typhoid / cholera in the slum area

Toilet user Improved health

No. days a toilet user has not been able to be up and about during the reporting period due to some stomach related illness

Toilet operator

Increased income

Total earnings generated by micro-entrepreneurs from selling SPO’s services % toilet operators with all their children attending school % of toilet operators with their house’s outer walls made from strong materials

(e.g. iron, aluminium, tile, concrete, bricks, stone, wood)


The IM Process/ Step 4: Verifying & Valuing Impact

At 2 levels: VP/SI (non-financial support etc.) & SPO (triangulating information received)

Necessary to refine target outcomes & associated indicators and identify impacts with highest social value:

- Verify impact: Did the impact happen in the way it was expected?

- Value impact: Was the impact important i.e. valuable to the stakeholders?

Verifying impact: desk research, competitive analysis, interviews / focus groups

Measuring value created:

- Qualitative methods include storytelling, qualitative surveys, interviews, focus groups.

- Quantitative methods include quantitative surveys, revealed preference, perceived value and use of financial proxies (i.e. monetisation).

5



(i) Verifying & valuing technical assistance provided to SPO

Tracks all pro-bono assistance: type, hours, assigning € amount to how much would cost if purchased on the market

Annual survey to understand value SPOs see in technical assistance (firstly with help of external consultant, now on its own to keep costs down)

(ii) Verifying & valuing outcomes of SPO

Desk research: tracks gov’t & health NGO data on disease outbreaks in and around Nairobi (location of toilets)

Competitive analysis: monitors activities & results of similar companies working in similar settings (although not necessarily in the same country)

Focus group with sample of toilet users as well as with sample of broader slum dwellers to assess perceived value (using tools such as value game)


The IM Process/ Step 5: Monitoring & Reporting

Monitoring: tracking progress against (deviation from) indicators & objectives

Reporting: transforming data into presentable formats, relevant for key stakeholders (may need >1 report)

Iterative process: what is monitoring to one stakeholder is reporting to another

VP/SI: systemise data it tracks (overall, per SPO, non-financial support provided), analyse to identify problems & implement corrective actions

SPO: evaluate outcomes & impacts achieved, practical lessons learned, actions needed to increase impact

Tools and resources available: PULSE (numeric metric data collection and reporting tool), Social Reporting Standard (guidelines & framework), Global Reporting Initiative (principles of report quality)

5



VPO/SI

“Salesforce” style CRM system being integrated to facilitate later move from excel to PULSE

SPO

Early stage SPO: data collection begun using excel spread sheets

VPO/SI developed template for reporting with other investors & agreed with SPO to reduce burden

Use of pro bono consultants to help SPO develop more robust internal monitoring system


The IM Process/ Managing Impact 5

Impact measurement as learning process & management tool

Integrating impact approach into investment process is key to achieving social impact

Investment strategy

Investment process Deal screening Due diligence

(detailed screening)

Deal structuring Investment management

Exit

Decide on overarching social impact objectives of VPO/SI

Asking questions detailed in Setting objectives

Setting objectives - more detail Stakeholder analysis Verify and value expected results

Map outputs, outcomes and impacts and set indicators Decide on monitoring and reporting

Regularly assess impact results against key indicators Verify and value reported results Revise indicators if necessary

Analyse impact results against objectives – verify and value


Q&A

Thank you for your time!

The Practical Guide to Measuring and Managing Impact can be downloaded, free of charge, at: http://evpa.eu.com/knowledge-centre/publications/evpa-publications/








Dr. Lisa Hehenberger Research Director

[email protected]

For further questions, please contact

mailto:[email protected]

stone soup webinar series · 2019-03-08 · webinar series with expert group april – july 2012...

Documents