monitoring & evaluation guidebook for the textbooks ...pdf.usaid.gov/pdf_docs/pnadg235.pdf ·...

FEBRUAR Y 2006

Prepared collaboratively with:

Monitoring & Evaluation Guidebookfor the Textbooks & Learning MaterialsProgram (TLMP)

African Education Initiative (AEI)United States Agency for International Development (USAID)

Monitoring & Evaluation Guidebook for the AEI TLMP

ContentsPreface

Acronyms

Why Monitor and Evaluate the Textbooks and Learning Materials Program?

How to Develop a Performance Monitoring Plan and Tracking System

Monitoring and Evaluating the TLMP

Conclusion

References

Appendix

Updated Matrix of Awardee Deliverables for USAID

Figures

1 The Combined M&E Approach2 The Program Cycle3 The Project Cycle4 AEI Flowchart

Tables

1 The PMP Template2 The PMP Example3 Incorporating TLMP Tasks into the Work Plan4 M&E Tracking Form

2

5

6

11

26

27

28

30

6101013

1617-222325

Monitoring & Evaluation Guidebook for the AEI TLMP — 2

The Africa Education Initiative (AEI) is a $200 millionpresidential initiative administered by the United StatesAgency for International Development (USAID) AfricaBureau/Office of Sustainable Development/EducationDivision. The Textbooks and Learning Materials Program(TLMP) is one of AEI’s central programs. AEI is currentlyactive in over 40 countries in sub-Saharan Africa, through100+ distinct activities in girls’ education, teacher training,textbook development, and innovations in education.The TLMP specifically focuses on challenges relating to the lack of sufficient textbooks and other learning materials in sub-Saharan Africa.

The six TLMP Awardees are responsible for providing (that is, identifying, selecting, developing/adapting, printing,and assisting with distribution) a minimum of 600,000copies of high quality, cost-effective education materials for use in primary schools in its host country. This programemphasizes strengthening the capacity of U.S. implement-ing partners to develop and sustain links with African institutions. Awardees include Alabama A&M University,Chicago State University, Elizabeth City State University,Mississippi Consortium for International Development,South Carolina State University, and the University

of Texas at San Antonio. African partners are fromEthiopia, Ghana, Senegal, South Africa, Tanzania,and Zambia.

TLMP began with a post-award workshop on November6–9, 2005. At the workshop,USAID facilitated a monitor-ing and evaluation (M&E) session, which highlighted thenuances between monitoring and evaluation; showed howM&E connects to other areas of the project cycle, such as management and implementation; and guidedAwardees in the development of individualized M&Eplans. Based on feedback from the Awardees, USAID has developed this practical guidebook to assist teams in the M&E process.

Africa’s progress also depends on the education of Africa’s children . . . If Africa is to meet its full potential, these children must have the chance to study and learn.

President George Bush June 26, 2003

1 White House, “Africa Education Initiative,”http://www.whitehouse.gov/news/releases/2005/06/20050630-7.html.

PREFACE


USAID developed this guidebook to:

Explain Why TLMP Must Monitor and Evaluate.

The guidebook explains the context of TLMP within USAID and the Africa Education Initiative and the policyimperative to monitor and evaluate programs.

Show How to Develop a Results-BasedMonitoring & Evaluation System to Assist with Project Management.

The guidebook explains how to develop a PerformanceMonitoring Plan (PMP) and tracking system. The PMP is a key management tool that, along with other documents,such as the work plan, will better ensure desired results andpositive formative and summative evaluations. The PMP template, definitions, and sample should be useful to theAwardees and clarify their roles as monitors and evaluators.

Identify Monitoring, Evaluation and Reporting Information Required by USAID.

USAID has identified the data submission requirements for the TLMP, including indicators of performance andachievement, and reporting templates and submissiontimeline.

Provide References and Other Resources.

Apart from resources provided at the TLMP workshop,the guidebook includes other useful resources.

Laura Bush with Ghana President John AgyekumKufuor at the launch of the Africa EducationInitiative Textbooks Program Jan. 17, 2006 in Accra, Ghana.

PH

OTO

GR

AP

H:

SH

EA

LAH

CR

AIG

HE

AD

PREFACE


ADS Automated Directives System

AEI Africa Education Initiative

AFR/SD/ED Africa Bureau/Office of SustainableDevelopment/Education Division

CTO Cognizant Technical Officer

IR Intermediate Result

Exegesis Exegesis Consulting

MOE Ministry of Education

MOU Memorandum of Understanding

MSI Minority Serving Institution

M&E Monitoring and Evaluation

PMP Performance Monitoring Plan

RFA Request for Application

SO Strategic Objective

TLM Textbooks and Learning Material

TLMP Textbooks and Learning MaterialProgram

USAID United States Agency forInternational Development

ACRONYMS


USAID Policy Is to “Manage for Results”In recent years, there has been a global change in govern-mental accountability in response to numerous internaland external forces. As a recent World Bank publicationstates:

Stakeholders are no longer solely interested in organizational activities and outputs; they are now morethan ever interested in actual outcomes. Have policies,programs, and projects led to the desired results and outcomes? How do we know we are on the right track?How do we know if there are problems along the way?How can we correct them at any given point in time?How do we measure progress?How can we tell success from failure?2

“Managing for Results” is one of the core values of USAID program policies and required procedures,spelled out in its Automated Directives System (ADS).In the TLMP, USAID and its partners—the TLMPAwardees and their country partners—should agree on the results that they are seeking to achieve and thenplan and implement activities accordingly.

USAID reports to policy makers on the results of the Africa Education Initiative programs,includingTLMP. Therefore, just as AEI staff must manage forresults and be prepared toreport if specifically statedresults have been achieved, the Awardees, as the AEIimplementing partners, must ensure that their work and reporting is in alignment with USAID’s systems and requirements.

USAID recommends that Awardees design an M&E system that:

Measures RESULTSConsiders UTILIZATIONEmploys PARTICIPATORY TECHNIQUES

Understanding the following three complemen-tary practical M&E approaches will guideAwardees in their design of an effective M&E system:

1. Results-Based Monitoring & Evaluation2. Utilization-Based Monitoring & Evaluation3. Participatory Evaluation

J.Z Kusek & R.C. Rist, Ten Steps to a Results-Based Monitoring and Evaluation System (Washington, DC: World Bank, 2004),

WHY MONITOR AND EVALUATE

THE TEXTBOOKS AND LEARNING

MATERIALS PROGRAM?

Figure 1: The Combined M&E Approach

Results-Based

Utilization-Based

M&E SYSTEM

Focuses on how and why the program/project

is successful

Focuses on full involvement of all

stakeholders

Focuses on full involvement of all

stakeholders


Monitoring

What Is Results-Based Monitoring & Evaluation?

A results-based M&E system helps to better manageresources. It is used along with annual plans and otherwork plans for a total results-oriented system.

Monitoring

Traditional monitoring focuses on implementation monitoring: tracking inputs (budget, human resources);activities; and outputs (products or services producedfrom inputs, such as the number of textbooks and learning materials [TLM] produced).

An output must be measured through performance indicators and monitored to ensure that it is helpingachieve the desired results. The number of TLM produced, for example, is an output; the output wouldhelp achieve AEI’s intended intermediate result, which is Strengthened Development and Distribution of Textbooks and Learning Materials.

Results-based monitoring focuses on how well a project, program, or policy is being implemented.In the TLMP Request for Application (RFA), thereare capacity-building and sustainability objectives alongwith the education objectives, indicating that it is essentialto monitor results in program implementation as well as TLM production and distribution. Ultimately,for example, textbooks and learning materials could be delivered without achieving capacity-building and sustainability objectives. Therefore, monitoring resultsalong the way is essential.



MATERIALS PROGRAM?

From page 38 of the TLMP RFA.

The output results to be achieved by eachAwardee are as follows:

• Strengthened capacity of the Awardee to …

• Production of printed materials…

• Selection of a printer/publisher in Africa…

•Increased knowledge and skills of host countrypartners, including the MOE, and the Awardeethrough the active involvement of the partners in each stage and sharing of knowledge with partners, other TLMP Awardees, and theAwardee’s institution.A stack of some of the books that are part of the

Africa Education Initiative Textbooks Program,as seen on stage Jan. 17, 2006 in Accra, Ghana.

PH

OTO

GR

AP

H:

SH

EA

LAH

CR

AIG

HE

AD


EvaluationWhile monitoring describes what a project or program is doing, evaluation addresses why and how.

Results-based evaluation considers the monitoring data that addresses how well a project, program, or policy is being implemented and then probes further and askswhy? For example, an evaluation examines how well each Awardee achieved its objectives and why the objec-tives were or were not met. Finding out why an objectivehas, or has not, been met is important. Also, by evaluatingimplementation across the six countries, comparativefindings should provide valuable lessons for AEI as it moves forward. For example, what conditions createthe best situation for successful TLM production andwhich create challenges?

Exegesis will collect monitoring data from all Awardees to evaluate TLMP; this will become part of the AEI evaluation. Monitoring data, therefore, is the foundationfor evaluation.

What Is Utilization-Focused Monitoring & Evaluation?In utilization-focused M&E, the monitoring and evalua-tion design and facilitation processes should take intoconsideration the intended use and the intended users of M&E products.

Utilization-focused M&E• Moves from the abstract to the real and specific• Allows intended users to do the monitoring and

evaluating• Is highly personal and situational• Promotes action, reaction, and adaptation

by evaluators and users• Incorporates all relevant stakeholders

For TLMP this means that M&E should be conducted—and the results disseminated and discussed—for the purposes of programmatic improvement.

What Is Participatory Evaluation?Experience shows that participatory evaluation improves program performance.

Participatory Evaluation• Provides for active involvement in the evaluation

process of those with a stake in the program(providers, partners, beneficiaries, and other stakeholders)

• Occurs throughout the evaluation • Includes planning and design; gathering and analyzing

data; identifying the evaluation findings, conclusions,and recommendations; disseminating results; andpreparing an action plan to improve program performance

Participatory evaluation should yield collaborationbetween stakeholders in the African host countries,USAID, and the Awardee institutions.

Adapted from M. Quinn Patton’s, Utilization-Focused Evaluation, (Thousand Oaks, CA: Sage, 1987).

USAID. “Conducting a Participatory Evaluation TIPS” Number 1 (1996).



MATERIALS PROGRAM?

Officials listen as schoolgirls read from brandnew locally-produced textbooks.

PH

OTO

GR

AP

H:

L.LA

RTI

GU

E


TLMP Monitoring and Evaluation

Level 1Project Monitoring and Evaluation at the Awardee level:

Each Awardee will monitor its own project according to the objectives/intermediate results, indicators,and activities that it has identified and designed.The Awardee will document its monitoring activities;this will serve as a record of activities during a particularperformance period. The AEI Monitoring andEvaluation Contractor, Exegesis Consulting, will not evaluate individual Awardees, but rather, the TLMP as a whole.

Level 2Overall TLMP Program Monitoring andEvaluation.

Exegesis Consulting will prepare an interim monitoringreport and a summative evaluation of the integratedTLMP to see how well the program has performed inachieving its intermediate results. According to USAID,the intermediate result (IR) is an essential step to achiev-ing a strategic objective (SO), which is the most ambitiousresult that the USAID initiative can materially affect.“The IR is a measurable result that may capture a numberof discrete and more specific results.”5 In conducting its comprehensive program evaluation, Exegesis willreview Awardee monitoring documents. While Exegesis is not evaluating each individual Awardee, Exegesis’TLMP monitoring and evaluation reports will highlightAwardees' projects using information from Awardees’monitoring reports, direct observations, and other information gathered by Exegesis.

By working collaboratively, USAID, Exegesis, and theAwardees will be able to track if activities are moving in the right direction, suggest potential adaptations forsubsequent stages, and provide data to monitor and evaluate if results are being achieved. This process will ultimately ensure that AEI’s Strategic Objective of Improved Education Programs in Africa is met.

M&E must be integrated into the program andproject cycles.

Monitoring and evaluation is part of both the individualproject and larger program management cycles; it is not a phase tacked on to the end. Figure 2, on the followingpage, illustrates this point. Furthermore, Figure 2 showsExegesis in the middle of the program cycle to convey a second point: while each Awardee is required to moni-tor and evaluate its individual project, Exegesis is requiredto monitor and evaluate TLMP overall;monitoring andevaluation is therefore occurring on parallel levels—at the individual project and overall program levels.For both parties, success at the initiative level is the ultimate goal.

USAID, The Performance Management Toolkit: A Guide to Developing

and Implementing Performance Management Plans, (April, 2003), 21-22

Program vs. Project Definitions

• A program is a series of projects with a common strategic objective.

• A project is a series of activities with set objectives, designed to produce a specificresult in a specified time.

Source: European Commission 2001.

TLMP monitoring and evaluation will take place on two levels:



MATERIALS PROGRAM?


IntroductionThis section defines, illustrates, and outlines the monitoring and reporting process. It is intended to guideeach team in the creation of a Performance Monitoring Plan (PMP) and tracking system.

There are four steps in the monitoring and reporting process:

1. Creating a PMP2. Incorporating the PMP tasks into

the work plan3. Recording and tracking data4. Reporting to USAID

Step One: Creating a PMPThe PMP Context and Narrative

Before creating the actual PMP, Awardees should write a narrative that summarizes the project and provides the PMP background. This information shouldbe consistent with information provided in the LearningMaterial Outline and Roadmap to Publication.6 If alreadycompleted, the Design/Plan information can be utilizedto write the narrative or vice versa. Writing this narrativemay help you become aware of areas of design that needclarifying or strengthening.

Please provide a brief, two paragraphs or lessdescription of each of the following:

• TLM problem or gap that this project addresses

• Approach to addressing cross-cutting themes(HIV/AIDS and community participation)

• All partners involved and brief description of their involvement

• Approach for production and distribution • Strategy for orienting educators to the new

materials• Approach to sustainability and innovation• Strategy for assessing utilization• Other project elements

The narrative should also include the following information on each TLM:

• Type(s) of TLM (math textbook, conceptbook, science workbook, etc.)

• Grade level(s) of each TLM• Intended beneficiaries (i.e. All 6th grader,

5th grader in regions A, B, and C.,specificschools, out-of-school youth, etc.)

• Language(s) of each TLM• Number of copies of each TLM

to be produced• Percentage of total Awardee budget allocated

to each TLM(s)

Details can be found on page 35 of the TLMP RFA.

HOW TO DEVELOP A

PERFORMANCE MONITORING PLAN

AND TRACKING SYSTEM


The PMPThe PMP is a tool that USAID implementing partnersuse to plan, manage, and document performance datacollection. The PMP tool “contributes to the effective-ness of the performance monitoring system by assuringthat comparable data will be collected on a regular andtimely basis.” 7

The purpose of developing a PMP is to:

• Ensure the TLMP’s success by providing a meansof measuring and tracking program achievements,as well as problems encountered and solutions developed

• Meet the RFA requirements (Monitoring and evaluatingprogress is Task 6 in the RFA, which states, “monitoringwill focus on progress based upon: annual work plan;achievement indicators of programs outputs; and implemented/completed milestones”)

• Serve as a guide by which to monitor program activities over time

• Synchronize each Awardee’s monitoring and evaluationactivities with the monitoring and evaluation activitiesconducted by Exegesis Consulting

At a minimum, the PMP should include:

• A detailed definition of each performance indicator• The source, method, frequency, and schedule

of data collection• The team or individual responsible for ensuring

data are available on schedule

Other possible elements to incorporate are:

• How the performance data will be analyzed• How it will be reported, reviewed, and used

to inform decisions

The PMP Template and Supporting Documents

During the post-award TLMP workshop, Exegesis sharedwith Awardees the TLMP M&E framework document.Per the request of several teams at the workshop,Exegesis has developed a more utilization-focused

template and supporting documents to guide the teams through the M&E process. The concepts from the M&E framework are incorporated into the following documents so that the PMP template and sample are comprehensive and functional.

These documents are as follows:

A An Intermediate Results Flowchart page 13B PMP Definitions page 14C PMP Template page 16D PMP Example pages 17 -22E Sample M&E Tasks in Work Plan page 19F M&E Tracking Form page 25

A. The TLMP Intermediate Result Flowchart

Figure 4, on the following page, shows the links between the results to be attained at the overall program level and those at the six Awardees’project level. The activities that the Awardees manage enable the TLMP to achieve its results.In turn, the TLMP results, coupled with the results that the other AEI programs achieve, will result in the achievement of the highest level AEI Strategic Objective: Improved education programs in Africa.

The flowchart’s purpose is twofold:

1) To illustrate how the results sought through the individual project of each Awardee fit within the AEI; and,

2) To break down the two main intermediate results as stated in the RFA (shaded in yellow) into smallersub-intermediate results (shaded in green), each of which requires a set of indicators. In other words, performance indicators should be established to measure progress on each sub-intermediate result(although not simultaneously). The flowchart demonstrates how meeting each sub-IR contributes to the success of each TLMP project, the overallTLMP, and the AEI.

USAID, “Performance Monitoring Evaluation TIPS” Numbers 6 & 7 (1996);

USAID, The Performance Management Toolkit (April 2003).

HOW TO DEVELOP A


AND TRACKING SYSTEM


HOW TO DEVELOP A


AND TRACKING SYSTEM


❖ Responsible Person(s): the individual(s), office, and/or department responsible for conducting the timelyacquisition of data.

Category 3: Analysis, Use, and Reporting

❖ Method of Analysis and Data Use: methods of analysis include comparing disaggregated data;comparing current performance against multiple criteria (planned performance, milestones, and so on);analyzing relationships among performance indicators;and analyzing cost effectiveness. Data use refers toplans for how the data will be used.

❖ Reporting Format & Deadline: reporting format and deadlines.

B. PMP Definitions❖ Strategic Objective: the most ambitious result

that a USAID initiative can materially affect.Intermediate Result: a measurable result that may capture a number of discrete and more specific results.

Category 1: Performance Indicators

❖ Performance Indicators: this measure describes how well a project is achieving success; it showsresults relative to the objective. Indicators are both quantitative and qualitative and define how performance will be measured along a scale or dimension.

❖ Indicator Definition and Unit of Measurement:the indicator definition clarifies terms and entitiesused in the performance indicator statement;it includes a unit of measurement when appropriate.It is detailed enough to ensure that different users at different times, given the task of collecting data for a given indicator, would collect identical types of data.

Category 2: Data Acquisition

❖ Data Source: the data source is the entity from which data are obtained. Data sources may includegovernment departments, international organizations,other donors, non-governmental organizations, privatefirms, contractors, and similar organizations.It can also specify the actual document or report.

❖ Method of Data Collection: this provides sufficientdetail on the data collection or calculation methodto enable it to be replicated. For primary datacollection,consider the unit of analysis, data desegregation needs, sampling techniques, and techniques or instruments for acquiring data on selected cases.

❖ Schedule/Frequency: this is how often the datais collected or aggregated. Depending on the indicator, it may be collected on a quarterly, annual,or less frequent basis.

HOW TO DEVELOP A


AND TRACKING SYSTEM

Adapted from USAID, “Performance Monitoring Evaluation TIPS” Numbers 6and 7 (1996); USAID, “The Performance Management Toolkit: A Guide toDeveloping and Implementing Performance Management Plans” (April 2003)http://www.dec.org/usaid_eval/; and Development Assistance Committee,Glossary Of Key Terms (Paris: OECD, 2002).


C. The PMP TemplateThe PMP template on the subsequent page followsUSAID guidelines and tools. It is a flexible document,similar to the work plan template that each team receivedat the workshop. The PMP template is designed to guideAwardees in developing PMPs that are applicable and relevant to the Awardees’ project design and work plan.Please keep the following in mind when reviewing andusing the following documents:

• This PMP template is designed to assist each team in addressing Task 6 in the RFA.

• Each team’s PMP will look different. The format and amount of detail should be adapted to a team’swork plan and project activities.

• The PMP is dynamic and will change as each team progresses through the project cycle.

• Exegesis adapted these definitions from the USAID“TIPS” series; these can be accessed athttp://www.dec.org/usaid_eval/ (see References).

D. The PMP ExampleThe PMP example includes both required and suggested performance indicators for measuring project success. Awardees are expected to incorporate all of the required indicators and one or more suggested indicators (or illustrative indicators) listed in the following PMP example. Examples of quantitative and qualitative indicators are included to illustrate the importance of using both approaches in monitoring and evaluation.

Please note that the PMP should be as comprehensive as possible. While some indicators may not be measurablein the project cycle, their inclusion shows how the hostcountry partner institutions will continue program effortsfollowing the Awardee’s participation in the TLMP.

The First Lady receives a "Gift for Madam Laura"from a young student of Al-Rahma madrasa.

PH

OTO

GR

AP

H:

FAYA

GE

RW

IN

HOW TO DEVELOP A


AND TRACKING SYSTEM


Per

form

ance

Ind

icat

or

(Wh

at is

bei

ng

m

easu

red

)

Def

init

ion

& U

nit

of

Mea

sure

men

tD

ata

So

urc

e(D

ocu

men

t, D

atab

ase

and

/or

info

rman

t)

Met

ho

d o

f D

ata

Co

llect

ion

(Ho

w d

ata

are

colle

cted

)

Sch

edu

le/

Fre

qu

ency

(W

hen

)R

esp

on

sib

le P

erso

n(s

) (W

ho

man

ages

th

e p

roce

ss)

Met

ho

d o

f A

nal

ysis

& D

ata

Use

Aw

ard

ee S

ub

Inte

rmed

iate

Res

ult

1.2

: S

ucc

essf

ully

pro

du

ced

an

d d

istr

ibu

ted

lear

nin

g m

ater

ials

to in

tend

ed r

ecip

ien

ts

Aw

ard

ee S

ub In

term

edia

te R

esu

lt 1

.3:

Des

ign

ed c

ost

-eff

ecti

ve le

arn

ing

mat

eria

ls

Aw

ard

ee S

ub

Inte

rmed

iate

Res

ult

2.1

: E

stab

lish

ed f

unct

ion

al o

per

ativ

e p

artn

ersh

ips

Aw

ard

ee S

ub

Inte

rmed

iate

Res

ult

2.2:

Bu

ilt in

stit

uti

on

al c

apac

ity

Aw

ard

ee In

term

edia

te R

esu

lt 2

: B

uilt

inst

itu

tio

nal

cap

acit

y an

d c

reat

ed s

ust

ain

able

lin

kag

es b

etw

een

Aw

ard

ees

and

Afr

ican

inst

itu

tion

s

Aw

ard

ee S

ub

Inte

rmed

iate

Res

ult

1.4

: In

corp

ora

ted

th

e in

tend

ed u

tiliz

atio

n o

f le

arni

ng

mat

eria

ls in

to p

roje

ct d

esig

n

AE

I R

esu

lt 3

(T

LM

P):

Str

eng

then

ed D

evel

op

men

t &

Dis

trib

utio

n o

f T

extb

oo

ks &

Lea

rnin

g M

ater

ials

AN

AL

YS

IS, U

SE

& R

EP

OR

TIN

G

Rep

ort

ing

Fo

rmat

&

Dea

dlin

e

AE

I Str

ateg

ic O

bje

ctiv

e: Im

pro

ved

Ed

ucat

ion

Pro

gra

ms

in A

fric

a

PE

RF

OR

MA

NC

E IN

DIC

AT

OR

DA

TA

AC

QU

ISIT

ION

Aw

ard

ee In

term

edia

te R

esu

lt 1:

Pro

vid

ed a

min

imu

m o

f 60

0,00

0 th

ou

san

d c

op

ies

of h

igh

qua

lity,

co

st e

ffec

tive

tex

tbo

oks

an

d o

ther

lear

nin

g m

ater

ials

Aw

ard

ee S

ub

Inte

rmed

iate

Res

ult

1.1

: D

evel

ope

d h

igh

qu

alit

y le

arn

ing

mat

eria

ls

TA

BL

E 1

: T

HE

PM

P T

EM

PL

AT

E


Per

form

ance

Indi

cato

r(W

hat

is b

ein

g m

easu

red

)D

efin

ition

& U

nit

of

Mea

sure

men

tD

ata

So

urce

(Doc

um

ent,

data

bas

e an

d/o

r in

form

ant)

Met

ho

d of

Dat

a C

olle

ctio

n(H

ow

dat

a ar

e co

llect

ed)

Freq

uen

cy o

f th

e D

ata

Co

llect

ion

(Wh

en)

Res

pon

sib

le

Per

son

(s)

(Wh

o m

anag

es

the

pro

cess

)

Met

ho

d o

f A

nal

ysis

& D

ata

Use

Rep

ort

ing

For

mat

& D

ead

line

(Req

uir

ed) T

LM s

tand

ards

of

qual

ity d

evel

oped

join

tly w

ith

MO

E a

nd P

AC

(at

a m

inim

um

stan

dard

s sh

ould

incl

ude

age

appr

opria

tene

ss, d

urab

ility

, and

co

ntex

tual

rel

evan

ce)

TLM

: 5t

h gr

ade

mat

h te

xtbo

ok; S

tand

ards

: C

riter

ia fo

r as

sess

ing

qual

ity; U

nit:

Def

initi

ons

of q

ualit

y fo

r T

LM

Do

cs: M

OU

, Wor

k P

lan,

Doc

umen

tatio

n of

com

mun

icat

ion,

P

MP

; Inf

orm

ants

: M

OE

rep

s, A

war

dee

team

Dis

cuss

ions

dur

ing

Nee

ds

Ass

essm

ent V

isit;

Dia

logu

e an

d co

mm

unic

atio

n re

gard

ing

qual

ity s

tand

ards

and

PM

P

qual

ity in

dica

tors

Dur

ing

Nee

ds

Ass

essm

ent V

isit;

Prio

r to

w

ritin

g th

e R

oadm

ap to

P

ublic

atio

n

Pro

ject

C

oord

inat

orA

naly

sis:

Det

erm

inat

ion

of

whe

ther

an

acce

ptab

le

agre

emen

t on

qual

ity h

as

been

rea

ched

U

se: T

o de

term

ine

the

crite

ria

for

eval

uatin

g th

e T

LM

Inte

rnal

: Cou

ntry

A

sses

smen

t Rep

ort (

in fi

rst

quar

terly

rep

ort 1

/20/

06),

M

onito

ring

Tra

ckin

g F

orm

up

date

d qu

arte

rly (

4/20

/06,

et

c.);

Ext

erna

l (U

SA

ID):

P

MP

(4/

20/0

6), R

oadm

ap

(4/2

0/06

), M

onito

ring

Tra

ckin

g F

orm

with

A

nnua

l Rep

ort (

10/3

0/06

)

(Req

uir

ed) T

LM p

roto

type

(s)

has

been

vet

ted,

rev

ised

, and

ap

prov

ed b

y P

AC

, MO

E,

Aw

arde

e, a

nd e

duca

tors

(Req

uir

ed) T

he T

LM is

alig

ned

with

the

natio

nal c

urric

ulum

(Illu

stra

tive

) T

LM e

xhib

its

appr

opria

te tr

ansl

atio

n or

use

of

loca

l lan

guag

e(s)

(Illu

stra

tive

) T

LM in

clud

es th

e cr

oss-

cutti

ng th

eme

of

HIV

/AID

S

TLM

P In

term

edia

te R

esu

lt 1:

Pro

vide

d a

min

imum

of 6

00,0

00 c

opi

es o

f hi

gh

qual

ity,

cos

t eff

ecti

ve t

extb

ooks

an

d ot

her

lear

nin

g m

ater

ials

Aw

arde

e S

ub

Inte

rmed

iate

Res

ult

1.1:

Dev

elo

ped

high

qu

ality

lear

nin

g m

ater

ials

PE

RF

OR

MA

NC

E IN

DIC

ATO

RD

ATA

AC

QU

ISIT

ION

AN

AL

YS

IS, U

SE

& R

EP

OR

TIN

G

(Onl

y on

e ex

amp

le h

as b

een

prov

ided

per

sub

-inte

rmed

iate

res

ult.

For

Req

uire

d, Il

lust

rati

ve, a

nd A

war

dee

-ide

ntif

ied

indi

cato

rs, A

war

dees

sho

uld

fill i

n th

e re

st o

f the

ta

ble

acc

ordi

ng to

thei

r un

iqu

e ne

eds.

)

TA

BL

E 2

: T

HE

PM

P E

XA

MP

LE


Per

form

ance

Indi

cato

r(W

hat i

s be

ing

mea

sure

d)D

efin

ition

& U

nit o

f M

easu

rem

ent

Dat

a S

ourc

e(D

ocum

ent,

data

base

an

d/or

Info

rman

t)

Met

hod

of D

ata

Col

lect

ion

(How

dat

a ar

e co

llect

ed)

Freq

uenc

y of

the

Dat

a C

olle

ctio

n (W

hen)

Res

pons

ible

P

erso

n(s)

(W

ho m

anag

es

the

proc

ess)

Met

hod

of A

naly

sis

& D

ata

Use

Rep

ortin

g Fo

rmat

& D

eadl

ine

(Req

uire

d) T

he R

oadm

ap to

P

ublic

atio

n in

clud

es a

cle

ar

prod

uctio

n an

d di

strib

utio

n st

rate

gy a

s de

velo

ped

with

the

MO

E a

nd P

AC

(Req

uire

d) N

umbe

r of T

LM

prod

uced

TL

M: 5

th g

rade

mat

h te

xtbo

ok; U

nit:

One

cop

y of

TLM

Doc

s: P

rintin

g or

ders

an

d in

voic

es, d

eliv

ery

conf

irmat

ions

to M

OE

; In

form

ants

: Prin

ting

Com

pany

, PA

C

mem

bers

Col

lect

ion

and

revi

ew o

f pu

blic

atio

n or

ders

aga

inst

in

voic

es a

nd d

eliv

ery

conf

irmat

ions

; dire

ct e

xcha

nge

with

MO

E (t

elep

honi

c, in

-pe

rson

, ele

ctro

nic)

Upo

n re

ceip

t of T

LM fr

om

prin

ting

com

pany

MO

E R

epA

naly

sis:

Agg

rega

te n

umbe

r of

TLM

pro

duce

d U

se:

To

dem

onst

rate

con

tract

co

mpl

ianc

e on

del

iver

able

s

Inte

rnal

: Mon

itorin

g T

rack

ing

Form

upd

ated

qu

arte

rly (4

/20/

06, e

tc.)

; E

xter

nal (

US

AID

):

Ope

ratio

nal W

ork

Pla

n,

Roa

dmap

to P

ublic

atio

n (4

/20/

06),

Mon

itorin

g T

rack

ing

Form

sub

mitt

ed

with

Ann

ual R

epor

t (1

0/30

/06)

(Req

uire

d) N

umbe

r of T

LM

dist

ribut

ed p

er s

choo

l

(Req

uire

d) N

umbe

r of p

upils

w

ith a

cces

s to

TLM

(Illu

stra

tive)

Num

ber

of

teac

hers

with

acc

ess

to T

LM

Aw

arde

e S

ub In

term

edia

te R

esul

t 1.2

: Suc

cess

fully

pro

duce

d an

d di

stri

bute

d le

arni

ng m

ater

ials

to in

tend

ed r

ecip

ient

s

PE

RFO

RM

AN

CE

IND

ICA

TOR

DA

TA A

CQ

UIS

ITIO

NA

NA

LYS

IS, U

SE

& R

EP

OR

TIN

G

T

AB

LE

2 (c

onti

nued

): T

HE

PM

P E

XA

MP

LE


Perfo

rman

ce In

dica

tor

(Wha

t is

bein

g m

easu

red)

Defin

ition

& U

nit o

f M

easu

rem

ent

Data

Sou

rce

(Doc

umen

t, da

taba

se

and/

or In

form

ant)

Met

hod

of D

ata

Colle

ctio

n(H

ow d

ata

are

colle

cted

)Fr

eque

ncy

of th

e D

ata

Colle

ctio

n (W

hen)

Resp

onsi

ble

Pers

on(s

) (W

ho m

anag

es

the

proc

ess)

Met

hod

of A

naly

sis

& Da

ta U

seR

epor

ting

Form

at &

Dea

dlin

e

(Req

uire

d) E

xpen

ditu

re a

nd

budg

et tr

acki

ng s

yste

m h

as

been

dev

elop

ed a

nd

disa

ggre

gate

s by

eac

h ty

pe o

f TL

M(R

equi

red)

TLM

is d

esig

ned

to

be a

fford

able

for M

OEs

to

repr

oduc

e or

ada

pt in

the

futu

re

Affo

rdab

ility

: as

defin

ed

by c

ompa

rabl

e or

bu

dget

ed T

LM

prod

uctio

n/re

prod

uctio

n co

sts;

Uni

t: Es

timat

ed

cost

per

boo

k fo

r 1)

prod

uctio

n an

d 2)

re

prod

uctio

n

Doc

s: P

revi

ous

year

s'

MO

E bu

dget

, Aw

arde

e ex

pend

iture

s;

Info

rman

ts: P

AC

mem

bers

, MO

E re

ps,

Awar

dees

Col

lect

ion

and

revi

ew o

f MO

E bu

dget

aga

inst

Aw

arde

e ex

pend

iture

s; d

irect

exc

hang

e w

ith M

OE

staf

f and

pub

licat

ion

com

pani

es (t

elep

honi

c, in

-pe

rson

, ele

ctro

nic)

MO

E bu

dget

(s) c

olle

ctio

n du

ring

coun

try-

asse

ssm

ent;

Awar

dee

quar

terly

fina

ncia

l rep

orts

Proj

ect

Acco

unta

ntAn

alys

is: Q

uant

itativ

e co

mpa

rison

of p

rodu

ctio

n an

d re

prod

uctio

n co

sts

to c

urre

nt

MO

E bu

dget

; cos

t-ben

efit

anal

ysis

; U

se: T

o en

sure

su

stai

nabi

lity

Inte

rnal

: Mon

itorin

g Tr

acki

ng F

orm

upd

ated

qu

arte

rly (4

/20/

06, e

tc.);

Ex

tern

al (U

SAID

): Q

uarte

rly F

inan

cial

R

epor

ts (4

/30/

06, e

tc.),

M

onito

ring

Trac

king

For

m

subm

itted

with

Ann

ual

Rep

ort (

10/3

0/06

)(R

equi

red)

Bes

t-val

ue

publ

ishe

r was

iden

tifie

d th

roug

h a

com

petit

ive

proc

ess

(Illu

stra

tive)

Exp

endi

ture

and

bu

dget

trac

king

sys

tem

reco

rds

cost

-sha

ring

dolla

r val

ues

for a

ll pa

rtner

s

Awar

dee

Sub

Inte

rmed

iate

Res

ult 1

.3: D

esig

ned

cost

-effe

ctiv

e le

arni

ng m

ater

ials

PERF

ORM

ANCE

IND

ICAT

OR

DATA

AC

QUI

SITI

ON

ANAL

YSIS

, USE

& R

EPO

RTIN

G

T

AB

LE

2 (c

onti

nued

): T

HE

PM

P E

XA

MP

LE


Per

form

ance

Indi

cato

r(W

hat i

s be

ing

mea

sure

d)D

efin

ition

& U

nit o

f M

easu

rem

ent

Dat

a So

urce

(Doc

umen

t, da

taba

se

and/

or In

form

ant)

Met

hod

of D

ata

Col

lect

ion

(How

dat

a ar

e co

llect

ed)

Freq

uenc

y of

the

Dat

a C

olle

ctio

n (W

hen)

Res

pons

ible

Pe

rson

(s)

(Who

man

ages

th

e pr

oces

s)

Met

hod

of A

naly

sis

& D

ata

Use

Rep

ortin

g Fo

rmat

& D

eadl

ine

(Req

uire

d) A

vis

ion

stat

emen

t th

at d

escr

ibes

pup

il an

d te

ache

r use

of t

he T

LM is

de

velo

ped

join

tly b

y P

AC

, M

OE

, and

Aw

arde

e (R

equi

red)

Util

izat

ion

asse

ssm

ent s

tudy

is c

ompl

eted

an

d in

clud

es fe

edba

ck fr

om a

sa

mpl

e of

teac

hers

and

st

uden

ts o

n ut

iliza

tion

and

effe

ctiv

enes

s(Il

lust

rativ

e) N

umbe

r of

educ

ator

s tra

ined

to u

se T

LMTL

M: 5

th g

rade

mat

h te

xtbo

ok; U

nit:

educ

ator

Doc

s: D

ocum

ents

, re

ports

, and

rece

ipts

fro

m tr

aini

ngs,

sur

veys

of

teac

hers

and

sch

ool

adm

in.;

Info

rman

ts:

MO

E, A

war

dees

, ed

ucat

ors

and

scho

ol

adm

inis

trato

rs

Col

lect

ion

and

revi

ew o

f re

ports

, rec

eipt

s, a

nd

docu

men

ts fr

om th

e tra

inin

gs

(or i

nten

ded

train

ings

)

Afte

r eac

h tra

inin

g an

d ag

greg

ated

in A

nnua

l R

epor

t

Pro

gram

C

oord

inat

orA

naly

sis:

Agg

rega

te n

umbe

r of

edu

cato

rs tr

aine

d U

se: T

o en

sure

effe

ctiv

e ut

iliza

tion

of

the

TLM

Inte

rnal

: Mon

itorin

g Tr

acki

ng F

orm

upd

ated

qu

arte

rly (4

/20/

06, e

tc.);

Ex

tern

al (U

SAID

): O

pera

tiona

l Wor

k P

lan,

P

MP

Nar

rativ

e (4

/20/

06),

Mon

itorin

g Tr

acki

ng F

orm

su

bmitt

ed w

ith A

nnua

l R

epor

t (10

/30/

06)

PER

FOR

MAN

CE

IND

ICA

TOR

Aw

arde

e Su

b In

term

edia

te R

esul

t 1.4

: Inc

orpo

rate

d th

e in

tend

ed u

tiliz

atio

n of

lear

ning

mat

eria

ls in

to p

roje

ct d

esig

n

DA

TA A

CQ

UIS

ITIO

NA

NAL

YSIS

, USE

& R

EPO

RTI

NG

T

AB

LE

2 (c

onti

nued

): T

HE

PM

P E

XA

MP

LE


Perfo

rman

ce In

dica

tor

(Wha

t is

bein

g m

easu

red)

Defin

ition

& U

nit o

f M

easu

rem

ent

Data

Sou

rce

(Doc

umen

t, da

taba

se

and/

or In

form

ant)

Met

hod

of D

ata

Colle

ctio

n(H

ow d

ata

are

colle

cted

)Fr

eque

ncy

of th

e Da

ta

Colle

ctio

n (W

hen)

Resp

onsi

ble

Pers

on(s

) (W

ho m

anag

es

the

proc

ess)

Met

hod

of A

naly

sis

& Da

ta U

seRe

porti

ng F

orm

at &

Dea

dlin

e

(Req

uire

d) Id

entif

ied

appr

opria

te m

embe

rs o

f the

PA

C an

d pu

t a c

omm

unica

tion

stra

tegy

and

kno

wled

ge s

harin

g sy

stem

in p

lace

PAC:

Pro

gram

Adv

isory

Co

mm

ittee

; Uni

t: PA

C m

embe

r

Docs

: Cou

ntry

As

sess

men

t, TL

MP

prop

osal

; Inf

orm

ants

: PA

C m

embe

rs, M

OE,

Aw

arde

es

Com

mun

icatio

n wi

th M

OE

and

PAC

mem

bers

; mee

ting

note

s; s

urve

ys a

dmin

ister

ed

to P

AC m

embe

rs

PAC

and

role

s/re

spon

sibilit

ies

defin

ed in

firs

t qua

rter

Prog

ram

Di

rect

orAn

alys

is: Q

ualita

tive

anal

ysis

of re

spon

ses

and

feed

back

; Us

e: T

o en

sure

effe

ctive

ad

visor

y, m

anag

emen

t, an

d im

plem

enta

tion

of th

e pr

ojec

t

Inte

rnal

: Mon

itorin

g Tr

ackin

g Fo

rm u

pdat

ed

quar

terly

(4/2

0/06

, etc

.);

Exte

rnal

(USA

ID):

Qua

rterly

Rep

ort (

1/20

/06)

, M

onito

ring

Trac

king

For

m

subm

itted

with

Ann

ual

Repo

rt (1

0/30

/06)

(Req

uire

d) E

ffect

ive d

ivisio

n of

ro

les

and

resp

onsib

ilitie

s ar

e ev

iden

t in

the

MO

U(Il

lust

rativ

e) M

embe

rs o

f PA

C de

mon

stra

te th

at th

ey a

re

adeq

uate

ly in

form

ed a

bout

pr

ojec

t's p

rogr

ess

TLM

P In

term

edia

te R

esul

t 2: B

uilt

inst

itutio

nal c

apac

ity a

nd c

reat

ed s

usta

inab

le li

nkag

es b

etw

een

Awar

dees

and

Afri

can

inst

itutio

ns

Awar

dee

Sub

Inte

rmed

iate

Res

ult 2

.1: E

stab

lishe

d fu

nctio

nal o

pera

tive

partn

ersh

ips

PERF

ORM

ANCE

INDI

CATO

RDA

TA A

CQUI

SITI

ONAN

ALYS

IS, U

SE &

REP

ORT

ING

T

AB

LE

2 (c

onti

nued

): T

HE

PM

P E

XA

MP

LE

Per

form

ance

Indi

cato

r(W

hat i

s be

ing

mea

sure

d)D

efin

ition

& U

nit o

f M

easu

rem

ent

Dat

a S

ourc

e(D

ocum

ent,

data

base

an

d/or

Info

rman

t)

Met

hod

of D

ata

Col

lect

ion

(How

dat

a ar

e co

llect

ed)

Freq

uenc

y of

the

Dat

a C

olle

ctio

n (W

hen)

Res

pons

ible

P

erso

n(s)

(W

ho m

anag

es

the

proc

ess)

Met

hod

of A

naly

sis

& D

ata

Use

Rep

ortin

g Fo

rmat

& D

eadl

ine

(Req

uire

d) S

trate

gy fo

r st

reng

then

ing

each

inst

itutio

n (A

war

dee

and

Afri

can)

de

velo

ped

and

is a

ligne

d w

ith

coun

try a

nd o

rgan

izat

iona

l as

sess

men

t fin

ding

s

Aw

arde

e: T

he O

ffici

al

Uni

vers

ity; A

fric

an

Inst

itutio

n: M

OE

, The

M

ath

Wiz

ards

O

rgan

izat

ion;

Uni

t : S

trate

gy

Doc

s: W

ork

Pla

n,

Roa

dmap

, or M

OU

; In

form

ants

: PA

C

mem

bers

, MO

E re

ps,

Aw

arde

e re

ps

Rev

iew

of W

ork

Pla

n,

Roa

dmap

or M

OU

for

stra

tegi

c el

emen

ts re

latin

g to

in

stitu

tiona

l cap

acity

Dur

ing

Nee

ds

Ass

essm

ent a

nd

Org

aniz

atio

nal C

apac

ity

Ass

essm

ent;

Ann

ually

Pro

gram

D

irect

orA

naly

sis:

Qua

litat

ive

anal

ysis

of

cou

ntry

and

org

aniz

atio

nal

asse

ssm

ent c

ompa

red

to

stra

tegy

for s

treng

then

ing

skill

s an

d ca

paci

ty; U

se: T

o de

mon

stra

te th

at c

apac

ity-

build

ing

activ

ities

are

wel

l th

ough

t out

and

hav

e at

tain

able

goa

ls

Inte

rnal

: Mon

itorin

g Tr

acki

ng F

orm

upd

ated

qu

arte

rly (4

/20/

06, e

tc.);

E

xter

nal (

US

AID

): M

OU

(3

/1/0

6), R

oadm

ap to

P

ublic

atio

n (4

/20/

06),

Mon

itorin

g Tr

acki

ng F

orm

su

bmitt

ed w

ith A

nnua

l R

epor

t (10

/30/

06)

(Req

uire

d) A

war

dee

and

Afri

can

inst

itutio

ns d

emon

stra

te

area

s of

incr

ease

d ca

paci

ty (i

.e.

finan

cial

man

agem

ent,

text

book

des

ign,

etc

.) as

a

resu

lt of

pro

ject

(Req

uire

d) A

war

dees

and

A

frica

n in

stitu

tions

app

ly n

ew

skill

s(Il

lust

rativ

e) U

se o

f inn

ovat

ion

and

tech

nolo

gy to

incr

ease

ef

ficie

ncy

of te

xtbo

ok d

eliv

ery

or c

olla

bora

tive

proc

ess

Aw

arde

e S

ub In

term

edia

te R

esul

t 2.2

: Bui

lt in

stitu

tiona

l cap

acity

PE

RFO

RM

AN

CE

IND

ICA

TOR

DA

TA A

CQ

UIS

ITIO

NA

NA

LYS

IS, U

SE

& R

EP

OR

TIN

G

T

AB

LE

2 (c

onti

nued

): T

HE

PM

P E

XA

MP

LE



Task Dead-

lines Baseline Milestones Outcomes Person/

Entity Deliver-ables

Host Country Partner Role

1 Survey of representatives in region for verification of distribution

1.1 Design survey 6/15/08 Survey examples

Survey draft Survey ready for pilot

AC, PD, PC

MOE provides feedback

1.2 Identify all survey informants 7/1/08 0 Final list MOE leads

identification

1.3 Conduct surveys 8/30/08 Survey form Completed surveys

AC, PC, MOE

MOE helps conduct surveys

1.4 Collate & analyze surveys

9/15/08 Complete surveys

Analysis of findings

AC, PC MOE is given analysis for review

1.5 Analyze & report findings

9/30/08 Data & info. Draft of report Final Report PC Final Report

MOE receives copy of report

2 Sample study for on-sight verification of distribution

2.1 Identify Sample Group and location

6/30/08 Map of regions

Sample selected

MOE, PC, AC

MOE will lead identification

2.3 Design field instruments to measure distribution

7/15/08 Examples of instruments

Draft of instrument

Final instruments

AC, PC MOE will review instrument before finalization

2.4 Travel to field 8/15/08 Travel arranged

Study completed

MOE, PC, AC

MOE travels with Awardee

2.6 Analyze & report findings

9/30/08 Findings Draft of report Final Report PC, PD Final Report

MOE receives copy of final report

After completing the PMP, Awardees should incorporate all monitoring tasks into the Operational Work Plan.Integrating PMP tasks into the Work Plan will allow Awardees to track their monitoring tasks, deadlines, and person(s) responsible for implementation. The number of tasks will depend on the indicator, and some tasks may encompass more than one indicator. The following table provides an example of how to incorporate the required indicator, number of learning materials distributed, into the work plan.

Step Two: Incorporating PMP Tasks Into The Work Plan

TABLE 3: INCORPORATING PMP TASKS INTO THE WORK PLAN (EXAMPLE)

HOW TO DEVELOP A


AND TRACKING SYSTEM


Step Three: Recording and Tracking DataAfter creating a Performance Monitoring Plan and incorporating M&E tasks into the work plan, the nextstep is to record and track data, findings and informationas the project progresses. Keeping good records is essential in identifying and monitoring trends andissues that emerge as well as informing transparent andaccurate evaluations.

Awardees are expected by USAID to use the followingtemplate to record data and findings from their project.It should be included as an appendix in the AnnualReport. Qualitative and quantitative examples of findings/results have been provided. Please note that indicators should describe qualitative or quantitative finding(s), or both, if available. However, in the casethat there are no findings, please use the “additional comments” column to provide an explanation of anyprogress.

The purpose of the M&E Tracking Form is to provide a format for Awardees to 1) monitor their projects to efficiently report to USAID, and 2) identify and resolvechallenges to reaching work plan goals and timelines.Findings should be utilization-focused and should reflectauthentic progress made towards the sub-intermediateresults. This Form is a data transmission tool formanagers and is not used for punitive purposes.An absence of data can be a helpful finding, especiallywhen combined with meaningful clarification and explanation in the Additional Comments column.The following page includes an example for filling out the M&E Tracking Form.

HOW TO DEVELOP A


AND TRACKING SYSTEM

Step Four: Reporting To USAIDThe final step involves reporting data findings as recorded in Step 3, as well as other pertinent information, to USAID. As shown on the previous page, updated PMPs and M&E Tracking Forms are required each year in addition to other deliverables.They should accompany the Annual Reports.(See Appendix A for a matrix of all Awardee deliverable deadlines).

1. Initial Performance Monitoring Plan & Narrative of Activities and Project Targets

(submitted to USAID on April 20th, 2006 as an appendix to the second Quarterly Technical Report)

2. Updated Performance Monitoring Plan &Narrative of Activities and Project Targets

(submitted to USAID annually on October 30th as an appendix to the Annual Reports)

3. Annual M&E Tracking Form(updated quarterly for internal purposes; submitted to USAID annually on October 30th as an appendix to the Annual Reports)


TABLE 4: M&E TRACKING FORM (EXAMPLE)

PERFORMANCE INDICATOR

Performance Indicator (What is being

measured)

Data Source(Document, database

and/or informant)

Quantitative Qualitative

(Directly from PMP) (Actual vs. planned) (Actual vs. planned) (Finding) (Finding)(Any comments needed to clarify findings)

# of TLM produced Docs: Printing orders and invoices, delivery confirmations to MOE; Informants: Printing Company, PAC members

Collection and review of publication orders against invoices and delivery confirmations; direct exchange with MOE (telephonic, in-person, electronic)

200,000 Target was 350,000 TLM. Reported TLM produced as of 9/06. Produced with Uhuru publishers for regions x, y, z. An additional 150,000 TLM will be completed by 12/06. MOE requested a delay in producing last 150,000 copies, while they completed a new census of schools/students and considered shifting distribution to other regions, which might require adaptation of TLM for regional cultural considerations. The 150,000 remaining copies for Year 1 are expected to be completed by 12/06.

Identified appropriate members of the PAC and put a communication strategy and knowledge sharing system in place

Docs: Country Assessment, TLMP proposal; Informants: PAC members, MOE, Awardees

Communication with MOE and PAC members; meeting notes; surveys administered to PAC members

PAC members meet the needs and skills as identified during the Country Assessment. All PAC members have signed an agreement to advise the TLMP staff and uphold the objectives of the project to the best of their ability. The MOE selected 5 of the 10 PAC members and the remaining were recommended by Awardees and USAID mission staff. A formal communication flow was designed but has not yet been finalized with PAC members. The PAC team will use eTeaching to share their documents and materials and will follow the established meeting schedule.

See attachment for PAC meeting schedule and formal list of members.

DATA ACQUISITION

Method of Data Collection

(How data are collected)

FINDINGS/RESULTS

AdditionalComments

HOW TO DEVELOP A


AND TRACKING SYSTEM


At the post-award workshop, each team met withUSAID’s M&E contractor, Exegesis Consulting,to discuss progress, challenges, and possible consultingservice needs over the next year. This meeting was the first step in establishing a collaborative and effectivepartnership. M&E activities for the Africa EducationInitiative were designed along several principles:

1. That which gets measured, gets done!2. Utilization-focused M&E aims to improve programs,

not to “audit” success or failure.3. M&E efforts and products should benefit project

implementers as well as provide standard data requiredfor USAID accountability.

4. Quality M&E starts with implementer’s self-assessment/self-reporting, all other M&E productsand reports should build on that.

Given these principles, it is in USAID’s best interest to ensure effective monitoring and evaluation systems are established and maintained for each TLMP Awardee.To support effective monitoring and evaluation, M&Econsulting services may be available to Awardees,as requested and approved by USAID.

MONITORING AND EVALUATINGTHE TLMP

PH

OTO

GR

AP

H:

R.N

YB

ER

G


Monitoring and Evaluation is a critical part of the TLMP program and project cycles. At the project level,M&E is important to Awardees because it helps monitorand ensure the project’s progress with illustrative data.At the program level, the M&E contractor, ExegesisConsulting, will prepare an interim monitoring report and summative evaluation of the TLMP to be used by USAID to analyze and assess program progress and success, and to report results internally and externally.In order to conduct evaluations, USAID and Exegesisrely greatly on the performance indicators monitored,and the data collected, by Awardees at the project level.Therefore it is important that M&E plans and systems at the program and project levels are in alignment.

USAID developed this Guidebook to reiterate the importance of M&E, outline the TLMP M&Eprocess and requirements, provide references and otherresources, and to present a results based, and utilization-focused, guide that will help Awardees develop their own M&E plan, or PMP. The PMP is a tool used byUSAID implementing partners to plan, manage, and document the collection of performance data. A PMPwill help Awardees monitor their project’s progressand make changes and adaptations as needed. EachAwardee’s PMP will be unique, however, all Awardeesshould monitor and report the same essential projectinformation to allow USAID to conduct formative and summative evaluations of the program. Requiredinformation has been outlined in Chapter 2.

PMPs can also be shared between Awardees. Awardeesexpressed the desire to share information with each other at the Post-Award Workshop. Exchanging PMPs is an excellent way to facilitate this information sharingand to work together to realize the TLMP’s IntermediateResult—Strengthened Development and Distribution of Textbooks and Learning Materials. ExegesisConsulting is a resource provided by AEI and is committed to providing Awardees with practical toolsthat will allow them to realize project success.Collaboration between USAID’s CTOs and Backstops,the Exegesis team, and Awardees is intended to optimize successful planning, implementation,

and monitoring and evaluation of each project,and the program as a whole. Project success generatesprogram success. Developing PMPs and keeping comprehensive records will help ensure that Awardees are contributing to TLMP’s success and thus contributingto AEI’s Strategic Objective of Improved EducationPrograms in Africa.

Good luck and please keep up the importantwork that your teams are doing!

CONCLUSION


REFERENCES

DAC (Development Assistance Committee) Working Party on Aid Evaluation. 2002.“Glossary of Key Terms in Evaluation and Results-Based Management.”In Evaluation and Effectiveness, No. 6. Paris: OECD Publications.

European Commission. 2001. Manual: Project Cycle Management. European Aid Cooperation Office.

Kusek, J. Z., and R. C. Rist. 2004. Ten Steps to a Results-Based Monitoring and EvaluationSystem.Washington, DC: The World Bank.

Patton, Michael Quinn. 1997. Utilization-Focused Evaluation. Third Edition.Thousand Oaks, CA: Sage.

USAID (United States Agency for International Development).1996a.“Conducting a Participatory Evaluation TIPS,” Number 1. USAID, Washington, DC.

———. 1996b. “Performance Monitoring Evaluation TIPS,” Numbers 6 and 7. USAID,Washington, DC.

———. April 2003. “The Performance Management Toolkit: A Guide to Developing and Implementing Performance Management Plans.”http://www.dec.org/usaid_eval/(accessed December 12, 2005).

The White House. “African Education Initiative.”http://www.whitehouse.gov/news/releases/2005/06/20050630-7.html(accessed December 27, 2005).

USAID and Other ReferencesUSAID References:ADS 200, http://www.usaid.gov/pubs/ads/200/200.pdf.ADS 201, http://www.usaid.gov/pubs/ads/200/201.pdf.ADS 202, http://www.usaid.gov/pubs/ads/200/202.pdf.ADS 203, http://www.usaid.gov/pubs/ads/200/203.pdf.

Dialogue about ADS Programming Policies,http://www.USAIDResults.org (click on Town Hall).

Economic and Social Data Services,http://cdie.usaid.gov (internal USAID only).

Research and Reference Services,http://cdie.usaid.gov (internal USAID only).

“Legal and Policy Considerations when involving partners and customers on Strategic Objective Teams and other consultations,”http://www.usaid.gov/pubs/ads/200/2016s1.doc.


REFERENCES

Database of R4s and Annual Reports,http://www.dec.org/partners/pmdb.

Database of “Development Experience,”http://www.dec.org.

TIPS No. 1: Conducting a Participatory Evaluation (1996),http://www.dec.org/pdf_docs/PNABS539.pdf.

TIPS No. 2: Conducting Key Informant Interviews,http://www.dec.org/pdf_docs/PNABS541.pdf.

TIPS No. 3: Preparing an Evaluation Scope of Work (1996),http://www.dec.org/pdf_docs/PNABY207.pdf.

TIPS No. 4: Using Direct Observation Techniques,http://www.dec.org/pdf_docs/PNABY208.pdf.

TIPS No. 5: Using Rapid Appraisal Methods (1996),http://www.dec.org/pdf_docs/PNABY209.pdf.

TIPS No. 6: Selecting Performance Indicators (1996),http://www.dec.org/pdf_docs/PNABY214.pdf.

TIPS No. 7: Preparing a Performance Monitoring Plan (1996),http://www.dec.org/pdf_docs/PNABY215.pdf.

TIPS No. 8: Establishing Performance Targets (1996),http://www.dec.org/pdf_docs/PNABY226.pdf.

TIPS No. 9: Conducting Customer Service Assessments (1996),http://www.dec.org/pdf_docs/PNABY227.pdf.

TIPS No. 10: Conducting Focus Group Interviews (1996),http://www.dec.org/pdf_docs/PNABY233.pdf.

TIPS No. 11: Role of Evaluation in USAID (1997),http://www.dec.org/pdf_docs/PNABY239.pdf.

TIPS No. 12: Guidelines for Indicator and Data Quality (1998),http://www.dec.org/pdf_docs/PNACA927.pdf.

TIPS No. 13: Building a Results Framework (2000),http://www.dec.org/pdf_docs/PNACA947.pdf.

TIPS No. 14: Monitoring the Policy Reform Process (2000),http://www.dec.org/pdf_docs/PNACA949.pdf.

TIPS No. 15: Measuring Institutional Capacity (2000),http://www.dec.org/pdf_docs/PNACG612.pdf;the annex is available athttp://www.dec.org/pdf_docs/PNACG624.pdf.

Other Organizational References:

ADB (Asian Development Bank). Handbook for the EconomicAnalsis of Helath Sector Projects.http://www.adb.org/Documents/Handbooks/Health_Sector_Projects/default.asp.

CIDA (Canadian International Development Agency).Results-Based Management Handbook.http://www.acdicida.gc.ca/cida_ind.nsf/vLUallDocByIDEn/98D3AA0A746EECF0525698400 FA4D?

OECD (Organisation for Economic Co-operation and Development). Public Management and Governance.http://www.oecd.org/puma/.

World Bank. 1996. Performance Monitoring Indicators (based on logical frameworks).http://www.worldbank.org/html/opr/pmi/pmi.pdf.


AP

PE

ND

IX A

: A

war

dee

Del

iver

able

s fo

r U

SA

ID D

uri

ng

Yea

r 1

Aw

arde

e D

eliv

erab

le

Dea

dlin

e D

escr

ipti

on o

f Del

iver

able

O

pera

tiona

l Wor

k P

lan

Pos

t cou

ntry

vis

it, a

s ne

goti

ated

wit

h C

TO

D

etai

led

plan

wit

h ta

sks,

dea

dlin

es, b

asel

ines

, mile

ston

es, o

utco

mes

, res

pons

ible

per

sons

/ent

ity,

de

live

rabl

es, h

ost c

ount

ry p

artn

er r

oles

. F

irst

Qua

rter

ly T

echn

ical

Rep

ort,

inco

rpor

atin

g C

ount

ry A

sses

smen

t &

Org

aniz

atio

nal A

sses

smen

t

1/20

/06

No

mor

e th

an 1

5 pa

ges.

Inc

lude

s su

mm

ary

of a

ctiv

itie

s un

dert

aken

dur

ing

repo

rtin

g pe

riod

, pr

ojec

t m

anag

emen

t, an

d pa

rtne

rshi

p in

tera

ctio

n; n

ext q

uart

er p

lann

ed a

ctiv

ities

; pro

blem

s an

d so

luti

ons

rela

ted

to w

ork

plan

mile

ston

es; C

ount

ry A

sses

smen

t Rep

ort;

rep

ort o

n O

rgan

izat

iona

l A

sses

smen

t. F

irst

Qua

rter

ly F

inan

cial

Rep

ort

1/30

/06

Incl

udes

: Fin

anci

al s

tatu

s of

wor

k pl

an b

udge

t cat

egor

ies

usin

g a

tabl

e of

exp

endi

ture

s; q

uart

erly

ac

tual

s; c

umul

ativ

e ac

tual

s; c

omm

itm

ents

.

Sec

ond

Qua

rter

ly T

echn

ical

R

epor

t 4/

20/0

6 N

o m

ore

than

15

page

s. I

nclu

des

sum

mar

y of

act

ivit

ies

unde

rtak

en d

urin

g re

port

ing

peri

od,

proj

ect

man

agem

ent,

and

part

ners

hip

inte

ract

ion;

nex

t qua

rter

pla

nned

act

iviti

es; p

robl

ems

and

solu

tion

s re

late

d to

wor

k pl

an m

ilest

ones

; Cou

ntry

Ass

essm

ent R

epor

t; r

epor

t on

Org

aniz

atio

nal

Ass

essm

ent (

if n

ot in

clud

ed in

Fir

st Q

uart

erly

Tec

hnic

al R

epor

t).

A

ttac

h P

erfo

rman

ce M

onit

orin

g P

lan

as d

escr

ibed

in

the

M&

E G

uide

book

. (P

MP

upd

ates

are

su

bmitt

ed w

ith

Ann

ual R

epor

t. P

leas

e no

te t

hat t

he P

MP

is r

efer

red

to a

s th

e M

&E

Pla

n in

the

RF

A).

S

econ

d Q

uart

erly

Fin

anci

al R

epor

t 4/

30/0

6 S

ame

as 1

/30/

06.

Lea

rnin

g M

ater

ial O

utli

ne

and

R

oadm

ap to

Pub

licat

ion

Dep

ends

on

Aw

arde

e’s

wor

k pl

an, a

s ne

goti

ated

wit

h C

TO

Sho

rt d

ocum

ent

with

an

outl

ine

of T

LM

and

a r

oadm

ap (

with

aut

hors

, edi

tors

, ven

dors

, vet

ting

pl

an, d

istr

ibut

ion

plan

, etc

).

Thi

rd Q

uart

erly

Tec

hnic

al R

epor

t 7/

20/0

6 S

ame

as 4

/20/

06.

Thi

rd Q

uart

erly

Fin

anci

al R

epor

t 7/

30/0

6 S

ame

as 1

/30/

06.

Ope

ratio

nal W

ork

Pla

n Y

ear

2 U

pdat

e 8/

30/0

6 U

pdat

e of

wor

k pl

an w

ith

tech

nica

l and

fin

anci

al p

lan

(w/c

ost s

hari

ng).

Suc

cess

Sto

ries

8/

30/0

6 an

d as

re

ques

ted

by U

SA

ID

Inco

rpor

ate

Suc

cess

Sto

ries

into

Qua

rter

ly R

epor

ts. 1

pag

e te

xt o

nly,

1.5

text

and

imag

e. I

nclu

des:

pr

ojec

t tit

le; a

ctiv

ity

fund

ed b

y A

EI;

wha

t Suc

cess

Sto

ry d

emon

stra

tes;

nar

rativ

e (w

ith p

urpo

se

and

resu

lts).

F

ourt

h Q

uart

erly

Tec

hnic

al R

epor

t 10

/20/

06

Sam

e as

4/2

0/06

F

ourt

h Q

uart

erly

Fin

anci

al R

epor

t 10

/30/

06

Sam

e as

1/3

0/06

A

nnua

l Rep

ort (

2005

/200

6)

10/3

0/06

N

o m

ore

than

25

page

s. I

nclu

des

sum

mar

y of

act

ivit

ies

unde

rtak

en d

urin

g th

e ye

ar, p

roje

ct

man

agem

ent,

and

part

ners

hip

inte

ract

ion;

nex

t yea

r pl

anne

d ac

tivi

ties

; pro

blem

s an

d so

luti

ons

rela

ted

to w

ork

plan

mile

ston

es;

In

clud

es M

onit

orin

g T

rack

ing

For

m f

illed

in w

ith a

ppro

pria

te a

nd a

vaila

ble

data

/fin

ding

s.

(Upd

ated

qua

rter

ly f

or A

war

dee

inte

rnal

use

), a

nd a

n up

date

d P

erfo

rman

ce M

onit

orin

g P

lan

as

atta

chm

ents

.