Measuring Impact in ImplementationMonday, April 21, 2014 @ 2.00pm

Jeffrey Swedberg (Senior M&E Associate, QED Group)

Purpose

“The success or otherwise of the implementation of security and human rights programs can often be difficult to measure. Regardless, interested parties demand that monitoring and evaluation provide demonstrable results of the effectiveness of implementation projects. This Roundtable will examine how the effectiveness of implementation can be measured.”

Overview

• USG Peace and Security Indicators – Program Level (Where is the impact?)

• Peace and Security Index – Country Level (Where is the attribution?)

• Need to focus on regions, or sub-regions of need• Two evaluations conducted for USAID in

unstable regions of Africa (2010-2013) provide some lessons for future exploration

USG Peace & Security Indicator List (Examples)

Program Area: Counter-Terrorism• ELEMENT: PS 1.1 – DENY TERRORIST SPONSORSHIP,

SUPPORT AND SANCTUARY• INDICATOR TITLE: NUMBER OF PEOPLE TRAINED ON

COUNTERTERRORISM FINANCE BY USG PROGRAMS

• ELEMENT: PS 1.2 – DE-LEGITIMIZE TERRORIST IDEOLOGY• INDICATOR TITLE: NUMBER OF PUBLIC INFORMATION

CAMPAIGNS COMPLETED BY USG PROGRAMS

An Index for Peace and Security Program Elements as Defined by U.S. State Department

Index Elements: State Department’s annual Country Reports on Terrorism…

THAILAND There was no direct evidence that international terrorist groups were directly involved in attacks within Thailand, nor was there any evidence of operational linkages between the southern Thai insurgent groups and international terrorist networks. ….

Countering Radicalization and Violent Extremism: A range of Thai government agencies, including the Ministries of Interior and of Social Development and Human Security, and the Thai military and police academies continued to organize outreach programs to ethnic Malay-Muslims to counter radicalization and violent extremism. A small group of international non-governmental organizations also reached out to communities in the southern provinces to provide services and to identify the underlying causes of the area's violence…..

State Department’s Annual Country Reports on Terrorism (excerpt)

Index Elements: Failed States Index

Failed States Index

Index Elements: World Bank Institute

Index Elements: Human Rights

Index Elements: Homicide Rate

International Crime - Trafficking

Peace and Security Issues Are Often Very

Localized…within a Country…

…or a larger region

Proposed Indicators from Two Evaluations

Data Collected in Five Countries (2010-12)

Based on USAID Analysis of Drivers of Violent Extremism

• Perception that the international system is unjust; a mechanism through which Muslims are oppressed and their culture de-valued

• Perceptions of social exclusion and marginality

• Real or perceived societal discrimination

• Frustrated expectations and relative deprivation

• Unmet social and economic needs

• Local conflicts

Output Indicators – From “F” Common Indicator List

Indicator Source Definition

Number of public information campaigns completed by USG programs

Implementing Partners

Public information campaigns include radio, public service announcements, print media, internet postings, etc. that provide information de-legitimizing terrorist ideology

Number of community development projects

Implementing Partners Measures the local government’s ability to execute/implement community-initiated development projects in what are considered to be ungoverned spaces.

Number of individuals from at-risk groups that have been reached Implementing Partners Number of people from ‘at-risk’ groups

who have participated in USAID-sponsored civic and/or social activities specifically conducted to benefit members of such groups Activities can include but are not limited to, conferences, dialogues, training courses, workshops, and/or seminars;

Acceptance of violence in name of religion

Indicator Source Definition

Perception of terrorism as permissible under Islamic Law.

Survey Question. Answer (on a 1 to 5 Scale) to question: Are Al Qaeda’s violent activities permitted under Islamic law?

Community AlienationIndicator Source DefinitionPerception of individual power within community

Survey Question. Answer (on a 1 to 5 Scale) to question:? How much can an ordinary person do to improve community problems?

Satisfaction level with community services

Survey Question. Answer (on a 1 to 5 Scale) to question: What is your level of satisfaction with your access to services and resources in your community?

Community Participation Indicator Source Definition

Participation in Community Decision Making

Survey Question.

Answer (on a 1 to 5 Scale) to question: Do you participate in decision making in your community?

Level of engagement with local government?

Survey Question.

Answer (on a 1 to 5 Scale) to question: Have you contacted local government?

Results: Mogadishu

1.01.52.02.53.03.54.04.55.0Attendance of a community meeting

Raised an issue with authorities

Participation in decision-making

Satisfaction with local governmentdecision making

How much can an ordinary person do?o?

Do youth associations make apositive contribution?

View of youth by leadersSupported by youth organizations?

Preparedness to enter the job market

Optimism about a better future

Importance of training vs familyconnections

Justification of using violence in thename of Islam

Violent activities permitted underIslamic law?

Full beneficiaries (100) Partial beneficiaries (80) Comparison group (100)

What Does this Tell Us?

• Indicators based on targeted surveying can assess attitudes at sub-national level - information not always available at country level.

• Provide information beyond output level• Can address attribution through use of

counterfactuals• Use of standard survey questions allows

comparability