national aids councils - fsn network · the national aids councils: monitoring and evaluation...

The National AIDS Councils: Monitoring and Evaluation Operations Manual is designed as a practical toolkit and road map for practitioners to use in designing and implementing programme monitoring and evaluation (M&E). The manual introduces key concepts; presents simple, clear procedures, with a checklist of the process, timing and costs of building participatory programme M&E for National AIDS Councils (NACs); and offers key tools that implementing partners need for M&E. The manual emphasizes the development of the overall M&E system, in relation to the National Strategic Plan, and the monitoring of services provided through NACs and their implementing partners.

The World Bank1818 H Street, NW

Washington, DC, 20433 USA Telephone: (202) 473 1000

Fax: (202) 477 6391E-mail: [email protected]

Internet: http://www.worldbank.org

UNAIDS 20 avenue Appia

1211 Geneva 27, SwitzerlandTelephone: (+41) 22 791 36 66

Fax: (+41) 22 791 41 87E-mail: [email protected]

Internet: http://www.unaids.org

NATIONAL AIDS COUNCILS

MONITORING AND EVALUATION OPERATIONS MANUAL

21932_OMS_CouvCor 09.09.2002 16:55

© Joint United Nations Programme on HIV/AIDS (UNAIDS) 2002.

All rights reserved. This document may be freely reviewed, quoted, reproduced or translated, in part or in full, provided the source is acknowledged. The document may not be sold or used in conjunction with commercial purposes without prior written approval from UNAIDS (contact: UNAIDS Information Centre).

The views expressed in documents by named authors are solely the responsibility of those authors.

The designations employed and the presentation of the material in this work do not imply the expression of any opinion whatsoever on the part of UNAIDS concerning the legal status of any country, territory, city or area or of its authorities, or concerning the delimitation of its frontiers and boundaries.

The mention of specific companies or of certain manufacturers products does not imply that they are endorsed or recommended by UNAIDS in preference to others of a similar nature that are not mentioned. Errors and omissions excepted, the names of proprietary products are distinguished by initial capital letters.

UNAIDS - 20 avenue Appia - 1211 Geneva 27 - SwitzerlandTel. (+41) 22 791 36 66 - Fax (+41) 22 791 41 87

E-mail: [email protected] - Internet: http:/www.unaids.org

UNAIDS/02.47E (English original, August 2002)ISBN: 92-9173-227-3

The Joint United Nations Programme on HIV/AIDS (UNAIDS) is the leading advocate for global action on HIV/AIDS. It brings together eight UN agencies in a common effort to fi ght the epidemic: the United Nations Children’s Fund (UNICEF), the United Nations Development Programme (UNDP), the United Nations Population Fund (UNFPA), the United Nations International Drug Control Programme (UNDCP), the United Nations Educational, Scientifi c and Cultural Organization (UNESCO), the World Health Organization (WHO), the International Labour Organization (ILO) and the World Bank.

UNAIDS both mobilizes the responses to the epidemic of its eight cosponsoring organizations and supplements these efforts with special initiatives. Its purpose is to lead and assist an expansion of the international response to HIV on all fronts: medical, public health, social, economic, cultural, political and human rights. UNAIDS works with a broad range of partners—governmental and NGO, business, scientifi c and lay—to share knowledge, skills and best practice across boundaries.

Produced with environment-friendly materials

21932_OMS_CouvCor 09.09.2002 16:55



21932_OMS_brCor 09.09.2002 17:21 Page i

ii

Acknowledgements

We wish to express our gratitude to the numerous National AIDS Councils’ members, theorganizations and the individuals that contributed to this manual. The manual was prepared byProfessor David Wilson, PhD, and reviewed by practitioners and donors. The manual was alsoofficially reviewed by the International Partnership against AIDS in Africa (IPAA) in Dakar inOctober 2001 and the UNAIDS Monitoring & Evaluation Reference Group (MERG) inGeneva in November 2001. A final draft was reviewed by a UNAIDS, World Bank and Centersfor Disease Control and Prevention (CDC) steering group in Geneva in March 2002.

Acknowledgements

21932_OMS_brCor 09.09.2002 17:21 Page ii

iii

Acronyms

BSS Behavioural surveillance survey

CBO Community-based organization

CDC Centers for Disease Control and Prevention

CPA Country Programme Adviser

DHS Demographic and Health Survey

FHI Family Health International

IPAA International Partnership against AIDS in Africa

M&E Monitoring and evaluation

MAP World Bank Multi-Country AIDS Program

MEASURE Monitoring and Evaluation to Assess and Use Results

MERG UNAIDS Monitoring and Evaluation Reference Group

MICS UNICEF Multiple Indicator Cluster Survey

NAC National AIDS Council

NAP National AIDS Programme

NGO Nongovernmental organization

OVC Orphans and other vulnerable children

PMTCT Prevention of mother-to-child transmission

PLWHA People living with HIV/AIDS

STI Sexually transmitted infection

UNAIDS Joint United Nations Programme on HIV/AIDS

VCT Voluntary counselling and testing

WHO World Health Organization

Acronyms

21932_OMS_brCor 09.09.2002 17:21 Page iii

21932_OMS_brCor 09.09.2002 17:21 Page iv

TABLE OF CONTENTS

A. INTRODUCTION 1Who is this manual intended for? 2

What does this manual attempt to do? 2

What does this manual not attempt to do? 2

What is M&E? 3

What key M&E lessons have been learned? 3

What framework is suggested for M&E? 5

What M&E components exist and how well developed are they? 6

Who should do what? 7

NAC indicators and data sources 7

B. OPERATIONS PROCEDURES 9Overview 9

Clarifying NAC's role and increasing its capacity to coordinate 9

Contracting an M&E consultant 9

Recruiting specialized M&E units and agencies 10

Developing an M&E manual with participatory approaches 10

Implementing the agreed M&E system 13

APPENDICES 1 TO 9: TOOLS 171. Illustrative indicators 18

2. Summary terms of reference for NAC M&E staff 20

3. Detailed terms of reference for NAC M&E consultant 21

3.1. Draft advertisement for NAC M&E consultant 24

3.2. Proposed NAC consultant selection scoring criteria 25

4. Summary terms of reference for specialized programme 26activity monitoring entity

5. Planning, monitoring and evaluation form 27

5.1. Service delivery sub-section 29

6. Programme areas requiring quality-assurance checklists 30

7. Illustrative quality-assurance checklists for 31interpersonal communication

8. Checklist for project managers in NACs and donor 32 agencies

9. Indicative budget for design of overall M&E system 33

and programme activity monitoring

APPENDICES 10 TO 12: FURTHER INFORMATION 3510. Key sources for further information on M&E 36

11. Illustrative time frame for participatory planning process 37

12. Illustrative budget for participatory planning process 38

Table of contents v

21932_OMS_brCor 09.09.2002 17:21 Page v

21932_OMS_brCor 09.09.2002 17:21 Page vi

1

A. INTRODUCTION

1. HIV/AIDS is the leading cause ofdeath in sub-Saharan Africa. More than 18million Africans have died, more than 12million African children have beenorphaned because of AIDS, and another 28million Africans are living with the virustoday, the vast majority of them in theprime of their lives as workers and parents.Life expectancy is dropping, familyincomes are being decimated, and agricul-tural and industrial efficiency is decliningbecause of the epidemic. African nationsand the international community have rec-ognized how disastrous the epidemic is tothe African continent, and have concludedthat past efforts to wage war against thevirus have failed because: (i) there wasinsufficient commitment and leadership tofight the epidemic among nations bothinside and outside the continent; (ii) thewar was being waged with too few humanand financial resources; (iii) those pro-grammes that were effective, often under-taken by civil society organizations, wererarely scaled up; (iv) resources were notreaching communities; and (v) pro-grammes were too narrowly focused on thehealth sector. A new strategy has beendeveloped by African countries and thedonor community to wage war more effec-tively. It is based on:

• defining national HIV/AIDS prevention,care, treatment and mitigation strategiesand implementation plans through a par-ticipatory process;

• establishing National AIDS Councils(NACs) at the highest level of govern-ment, with broad stakeholder representa-tion from the public and private sectorand civil society;

• empowering stakeholders from the vil-lage to national level with money and

The new approach relies on immediateM&E of programmes to determine whichactivities are efficient and effective andshould be expanded further, and which arenot and should be stopped or would benefitfrom capacity-building.

decision-making authority within amultisectoral framework; and

• using exceptional implementationarrangements such as channellingmoney directly to communities andcivil society organizations and con-tracting out many administrative func-tions.

2. The new approach emphasizes speed(due to the nature of the epidemic), scalingup existing programmes and capacity-building, 'learning by doing' and continu-ous project rework, rather than exhaustiveup-front technical analysis of individualprojects. The new approach relies onimmediate monitoring and evaluation(M&E) of programmes to determine whichactivities are efficient and effective andshould be expanded further, and which arenot and should be stopped or would benefitfrom capacity-building.

3. This new approach is being supportedby a number of donors, including bilateralagencies, the Global Fund to Fight AIDS,Tuberculosis and Malaria1 and the WorldBank, which is committing US$1 billionthrough the Multi-Country HIV/AIDSPrograms (MAP) for Africa2. In an erawhen those inside Africa as well as aroundthe world demand performance and trans-parency in the use of funding, programmeM&E is essential in order to:

• establish performance incentives forprogramme implementers in both thepublic sector and civil society;

Introduction

1 www.globalfundatm.org 2 www.worldbank.org/afr/aids/map.htm

21932_OMS_brCor 09.09.2002 17:21 Page 1

2

• detect and address problems so thatproject redesign and improvementbecome standard operating procedures;

• provide early evidence of programmeeffectiveness; and

• communicate to those infected andaffected by HIV/AIDS, in transparentand objective ways, the effort beingmade to improve prevention, care, treat-ment and mitigation programmes.

4. M&E thus becomes a core part of thefiduciary architecture of financial manage-ment, disbursement and procurement,which is the basis for the performance con-tract on which the war against HIV/AIDSis being waged. The design of programmeM&E has to change with this newapproach. M&E must be relevant, objec-tive, transparent and, most importantly,available as: (i) a source of information onperformance for the public and for donors;and (ii) a management tool for implemen-tation agencies in the public and privatesector, in civil society and for countrycoordination mechanisms such as NACs.M&E systems must also be fully fundedfor at least 5-10 years, including local costsand incremental operating costs, since lackof sustained funding is a major reason whymonitoring and evaluation fail.

5. This manual is designed as a practicaltoolkit and road map for practitioners touse in designing and implementing pro-gramme M&E. While published jointly byUNAIDS and the World Bank, it has bene-fited from extensive consultation and pilot-ing among African countries and otherstakeholders who recognize the importanceof M&E as an essential weapon in the waragainst HIV/AIDS.

Who is this manual intended for?

6. This manual is intended for:

• NACs, particularly those that are tak-ing on the role of grant-provider, andtheir public sector and civil society

implementing partners in sub-SaharanAfrica; and

• donor institutions that are involved inthe preparation, implementation andM&E of HIV/AIDS programmes inpartnership with NACs.

What does this manual attemptto do?

7. This manual attempts to:

• introduce key concepts;

• present simple, clear procedures, witha checklist of the process, timing andcosts of building participatory pro-gramme M&E for NACs;

• offer key tools that implementing part-ners need for M&E; and

• provide examples of terms of referenceand other M&E management andadministration materials.

8. This manual emphasizes:

• the development of the overall M&Esystem, in relation to the NationalStrategic Plan; and

• programme M&E or the monitoring ofservices provided through NACs andtheir implementing partners.

9. It focuses on these componentsbecause they are essential in order forNACs to be accountable and they are theleast developed components of M&E sys-tems.

What does this manual notattempt to do?

10. This manual does not attempt to coverall aspects of M&E. In particular, it doesnot attempt to deal with topics that are well

This manual is designed as a practicaltoolkit and road map for practitioners touse in designing and implementing pro-gramme M&E.

Introduction

21932_OMS_brCor 09.09.2002 17:21 Page 2

3

• Condomavailability

• Trained staff• Quality of services

(e.g., STI, VCT, care)• Knowledge of HIV

transmission

Short-term and inter-mediate effects:

• Behavior change• Attitude change• Change in STI

trends• Increase in social

support

Long-term effects/changes in:

• HIV/AIDS trends• AIDS-related

mortality• Social norms• Coping capacity in

community• Economic impact

• Resources• Staff• Funds• Facilities• Supplies• Training

ALL MOST SOME FEW

INPUTSOUTPUTS

OUTCOMESIMPACT

Monitoring and evaluation results pyramid

MONITORINGProcess evaluation

EVALUATIONEffectiveness evaluation

LEVELS OF EVALUATION EFFORTS

NU

MB

ER

OF

PR

OJE

CT

S

Introduction

covered elsewhere. Thus, it does not coverin any detail surveillance, essentialresearch or financial management. Therole of these elements within an overallM&E programme is discussed, but usersare referred to sources in Appendix 10 fordetailed information. Thus, the manualfocuses more on tracking inputs and out-puts than outcomes and impacts.

What is M&E?

11. Confusion between monitoring andevaluation is common. There is a simpledistinction between monitoring and evalua-tion that may be helpful. Monitoring is theroutine, daily assessment of ongoing activ-ities and progress. In contrast, evaluation isthe episodic assessment of overall achieve-ments. Monitoring looks at what is being

done, whereas evaluation examines whathas been achieved or what impact has beenmade.

What key M&E lessons have beenlearned?

12. We have learned the following keyM&E lessons:

(i) The results pathway or cycle, shown inthe figure below, may be likened to apyramid. The higher up the resultscycle we go, the fewer organizations,projects and studies are involved inM&E. Thus, all implementing part-ners should collect complete input andoutput data. Many implementing part-ners should collect some process data.Far fewer implementing partners will

21932_OMS_brCor 09.09.2002 17:21 Page 3

4

assess outcomes. Even fewer imple-menting partners and studies arerequired to assess impact.

(ii) Good M&E requires both internalself-assessment and external verifica-tion. Thus, implementing partners col-lect their own internal data and anexternal agency verifies the complete-ness and accuracy of the data collect-ed by those implementing partners.Supervisory visits should be based onthe analysis of internal self-assessmentand externally verified primary data.

(iii) M&E systems must be as simple aspossible. Most programmes collect farmore data than they use. The morecomplex an M&E system, the morelikely it is to fail.

(iv) M&E systems must include a stan-dardized core. If each implementingpartner uses different systems or tools,the data cannot be analysed or sum-marized effectively. The need for astandardized core does not precludeindividual implementing partnersfrom collecting additional, situation-specific M&E data.

(v) A specialized entity is required to col-lect, verify, enter and analyse primaryM&E data from each partner. Withoutsuch an entity, data collection, verifi-cation and analysis are unlikely tohappen. Ministries and other publicagencies are seldom equipped to man-age such a process.

(vi) Contracting a single specialized entityto manage both financial and pro-gramme M&E should be seriouslyconsidered. There are several reasonswhy this is desirable, which are listedbelow.

• Since the same personnel and oper-ations can often be used for bothfinancial and programme monitor-ing, combining financial and pro-gramme M&E makes proper moni-

toring economical and affordable.It may be added to financial man-agement at a marginal cost.

• Combining financial and pro-gramme monitoring provides abasis for cross-checking financialand activity data and ensuringsound finance-programme datacross-verification.

• Given the sensitivities concerningdata verification, it may be prefer-able to delegate the task to account-ing/consulting firms that are usedto handling such sensitive opera-tions.

• Programme M&E entails complexdata-gathering-management capaci-ties, and it is usually easier to pur-chase established capacity than toestablish it afresh.

• Financial management monitoringsystems and procedures are almostalways the best developed subcom-ponent of M&E. It makes sense tolink programme monitoring to thestronger process of financial moni-toring.

• Linking programme reporting tofinancial management and furtherdisbursements will improve pro-gramme reporting.

• Outsourcing financial and pro-gramme activity monitoring to asingle entity ensures that financialand programme reports are linkedand provide a more comprehensivepicture. Equipped with comprehen-sive, verified data, NAC M&E staffmembers are free to focus on thestrategic programme implicationsof monitoring.

Introduction

Contracting a single specialized entity tomanage both financial and programmeM&E should be seriously considered.

21932_OMS_brCor 09.09.2002 17:21 Page 4

5Introduction

(vii) M&E must be built into the design ofa programme, and should be opera-tional before grants are provided,rather than being added later. It ismuch harder and less effective to'retrofit' M&E after grants havealready been given.

(viii)No matter how sound an M&E systemmay be, it will fail without wide-spread stakeholder 'buy-in'. Thus, alarge-scale, participatory process isessential to build ownership and buy-in from the start.

(ix) NACs lack comprehensive, long-termfunding for all major M&E compo-nents, including local costs and incre-mental operating costs. The WorldBank, through MAP credits, may pro-vide comprehensive, long-term M&Efunding in areas where grant funding isunavailable. The World Bank recom-mends that up to 10% of MAP creditsbe used for local and operating costs ofa long-term M&E system.

(x) It may be more helpful to move frombaseline and follow-up analysis totrend-tracking, following trends overseveral time periods.

What framework is suggested forM&E?

13. Effective M&E is based on a clear,logical pathway of results, in which resultsat one level are expected to lead to resultsat the next level, leading to the achieve-ment of the overall goal. Consequently, ifthere are gaps in the logic, the pathway willnot lead to the required results.

14. The major levels are:

• inputs• outputs• outcomes• impacts

15. These levels are introduced in Panel 1below. Each level is connected to the next,in a clear, logical way.

Level DescriptionInputs Inputs are the people, training, equipment and resources that we put into a project, in

order to achieve outputs.

Outputs Outputs are the activities or services we deliver, including HIV/AIDS prevention,care and support services, in order to achieve outcomes.

The processes associated with service delivery are very important and involve qual-ity, unit costs, access and coverage.

Outcomes Through the provision of good-quality, economical, accessible, and widespread serv-ices, key outcomes should occur. Outcomes are changes in behaviour or skills, espe-cially safer HIV prevention practices and increased ability to cope with AIDS.

Impacts The above-mentioned outcomes are intended to lead to major measurable healthimpacts, particularly reduced STI/HIV transmission and reduced AIDS impact.

Panel 1: M&E levels

No matter how sound an M&E system maybe, it will fail without widespread stake-holder ‘buy-in’.

21932_OMS_brCor 09.09.2002 17:21 Page 5

6 Introduction

ComponentOverall system

Surveillance

Research

NAC, public sector and civilsociety financial managementmonitoring

NAC, public sector and civilsociety programme activitymonitoring

DescriptionOverall flowchart and database

National biological and behavioural surveillance ofSTI/HIV/AIDS/TB, and sexual behaviour trends.

Essential research to complement national surveillance.

National financial management monitoring of NAC, thepublic sector and civil society's utilization of resources.

National programme activity monitoring of NAC grants pro-vided to implementing agencies, and the relevance, quantityand quality of services delivered by those agencies.

Panel 2: M&E components

What M&E components exist andhow well developed are they?

16. We suggest the following M&E com-ponents (see Panel 2 above).

17. M&E strengths in each of the abovecomponents vary widely.

Overall system

18. The overall system comprises a gov-erning flowchart and database, whichdescribe precisely how data are collectedand flow.

Surveillance

19. Surveillance comprises biological,behavioural and social impact surveillance.Both are well developed globally. WHO,UNAIDS and CDC support ensures soundantenatal biological surveillance in mostcountries and a proven procedure for estab-lishing it in other countries. Behaviouraland social impact surveillance, supportedby UNAIDS/FHI and social assessmentguidelines, is widespread and readily applicable. This surveillance should beaccompanied by second generation surveil-lance (see Box 1). With such support,NACs can quickly initiate sound surveil-lance. In mature epidemics, NACs willusually use existing surveillance; in nas-cent epidemics, they may use improved

surveillance. NACs should provide ade-quate resources and support to NationalAIDS Programmes (NAPs) within min-istries of health to ensure sound surveil-lance and health-related M&E.

Research

20. Surveillance should be complementedby essential research, including epidemio-logical, evaluation and social impact res-earch. NACs have a strategic role in collat-ing, interpreting and disseminating res-earch findings.

Financial management monitoring

21. NAC, public sector and civil societyfinancial management monitoring is wellsupported. The World Bank, for example,has substantial in-house financial manage-ment capacity and experience. Social pro-tection funds have demonstrated the feasi-bility of outsourcing financial managementto accounting firms/banks.

Programme activity monitoring

22. NAC, public sector and civil societyprogramme activity monitoring representsthe greatest challenge facing NACs. It isaddressed partly through draft operationsmanuals, but significant challenges remain.NACs will assume a major role as grant-provider, supporting literally hundreds of

21932_OMS_brCor 09.09.2002 17:21 Page 6

7Introduction

BOX 1: What is second generation surveillance?

Traditional surveillance systems typically tracked HIV or sexually transmitted infections(STIs). However, they did not concurrently track the sexual practices that lead to HIV/STItransmission. This made it difficult to corroborate and explain HIV/STI trends. To addressthese limitations, second generation surveillance evolved. This form of surveillance seeksto combine biological and behavioural data, to increase explanatory power. The concor-dance of diverse biological, behavioural and qualitative insights not only enhances confi-dence in trends, but it allows for meaningful explanations of these trends. Examples froma wide range of countries, including Senegal, Thailand and Uganda, show how second gen-eration surveillance can identify HIV trends through biological surveillance, and then con-vincingly explain these trends through behavioural surveillance. Such examples underscorethe vital importance of second generation surveillance.

HIV/AIDS prevention, care and mitigationactivities. However, they lack essential sys-tems and procedures, particularly thoserequired for:

• maintaining an overall integratedM&E flowchart and database;

• identifying epidemiological prioritiesand soliciting compliant applications;

• publicizing the availability of fundingfor public sector and civil society ini-tiatives and application mechanisms;

• developing and publicizing structured,transparent selection criteria andapproval procedures;

• publicizing grant recipients;

• monitoring programme progress ofrecipients and communicating achie-vements; and

• reviewing overall national programmeprogress, with particular reference togeographic focus, coverage and equi-ty, interventions and service to vulner-able groups.

23. Programme activity monitoring isleast developed and requires the greatestemphasis. As with financial monitoring,this component should be contracted out toan independent firm. It is recommendedthat financial and programme activity

monitoring be combined and contractedout to one firm, for economy and finance-programme cross-verification.

24. The best developed components are:

• surveillance (especially biologicalsurveillance);

• research; and

• financial monitoring.

25. The least developed components are:

• overall M&E system, with flowchartsand a unified database; and

• programme monitoring.

Who should do what?

26. Building on the first lesson learned (seethe monitoring and evaluation results pyramidon page 3), data sources, implementing part-ners' responsibilities for M&E and suggestedtime frames for progress are summarized inPanel 3 (see page 8).

NAC indicators and data sources

27. We suggest an illustrative indicator set,with data sources, in Appendix 1. This indica-tor set is neither definitive nor exhaustive.Individual programmes will both change andadd to this indicator set. However, it does pro-vide examples of possible indicators and datasources at each level of M&E.

21932_OMS_brCor 09.09.2002 17:21 Page 7

8 Introduction

Level

Inputs

OutputsQuantity

Quality

Unitcosts

Accessand coverage

Outcomes

Impacts

Overallsystem

Data

Finance and programmemonitoring


Programme monitoringusing quality checklists


Modules of behaviouralsurveillance and facilitysurveys

Behavioural surveillanceand epidemiologicalresearch

Biological surveillanceand epidemiologicalresearch

Flowchart and database

Partner role

All implementing partners submit month-ly data

Specialized external agency routinelyanalyses and verifies data

All implementing partners submit month-ly data

Specialized external agency routinelyanalyses and verifies data

All implementing partners do internalquality assurance

Specialized external agency routinelydoes external quality verification

Specialized external agency will use veri-fied financial and programme output datato estimate unit costs for selected imple-menting partners

Access to prevention, care, mitigationservices and coverage will be included asa subset of behavioural surveillance,social impact surveys and facility surveys,and assessed when behavioural or facilitysurveys are used

Behavioural surveys to assess outcomesare encouraged in 5-10 sites per countryevery 1-2 years. Behavioural surveys mayalso be conducted in selected large-scalepublic sector or civil society programmes.Examples include public sector pro-grammes for transport workers or soldiersand civil society programmes for refugees

Behavioural surveys should be contractedto specialized agencies and use UNAIDSand FHI guidelines

The ministry of health, often assisted byWHO, UNAIDS and CDC, is responsiblefor national STI and HIV surveillance

Selected epidemiological STI/HIV preva-lence/incidence studies may also be con-ducted and may illustrate impacts in spe-cific areas/populations

NAC maintains overall flowchart anddatabase

Time frame

Progress within 6 months

Progress within 1 year

Progress within 1-2 years

Progress within2-3 years

Progress within 3-5 years inmature epidemicsand 7-10 years innascent epidemics

To be designedbefore NAC begins providinggrants

Panel 3: Data sources, partners' roles and time frame by M&E level

21932_OMS_brCor 09.09.2002 17:21 Page 8

9

Overview

28. We propose the following steps forputting M&E into practice:

(i) NAC clarifies its coordination roleand increases its capacity to coordi-nate, but not implement, M&E.

(ii) NAC contracts out the implementa-tion of M&E to specialized entities.Thus, surveillance, research, andfinancial and programme monitoringshould ideally be contracted out to arange of entities. The specialized pro-gramme-activity-monitoring entity isresponsible for training implement-ing partners and verifying, collating,analysing and reporting data.

(iii) NAC and stakeholders engage in anintensive participatory process tobuild ownership and buy-in, particu-larly for the overall M&E system andprogramme monitoring.

(iv) Each implementing partner agrees onits key targets with NAC, using asimple, structured planning, monitor-ing and evaluation form.

(v) Each implementing partner reportsresults every month using the plan-ning, monitoring and evaluation form.

(vi) These results are checked and veri-fied at least every six months by thespecialized entity.

(vii) The specialized unit/agency assesseseach implementing partner's progresstowards targets every six months andrates their progress using the plan-ning, monitoring and evaluation form.

(viii) The specialized entity collates, analy-ses and submits to NAC summaryreports of aggregate activities everysix months, using a simple, struc-tured progress report form.

(ix) NAC and key stakeholders, includingdonors, meet every six months toreview M&E reports, to identify keylessons learned and to make strategicrecommendations and decisions.

(x) NAC and key stakeholders updatetheir M&E manuals and proceduresbased on lessons learned.

Clarifying NAC's role and increasingits capacity to coordinate

29. A small NAC M&E structure is sug-gested, with one or two staff (no more arerequired for what is purely a coordinationand facilitation role), depending on the sizeof the country and the complexity of the pro-gramme. The M&E staff's proposed terms ofreference are outlined in Appendix 2.

30. A major element of NAC's coordina-tion role is effective coordination with thehealth sector, which is historically therepository of M&E capacity in AIDS pro-grammes.

31. There is an urgent need to develop ashared NAC and donor vision of NACM&E philosophy and priorities and toensure coordinated and complementarydonor inputs to NAC M&E plans.

Contracting an M&E consultant

32. NAC's role is to ensure that the resultsof M&E are used at the appropriate level.NACs may contract an M&E consultant toassist with the design and management ofthe steps outlined below. Such a consultantshould, above all, be interdisciplinary andable to combine insights from research, pro-gramme management and information sys-tems. Further guidance on suitable M&Econsultants is presented in Box 2.

Operations procedures

B. OPERATIONS PROCEDURES

NAC clarifies its coordination role andincreases its capacity to coordinate, but notimplement, M&E.

21932_OMS_brCor 09.09.2002 17:21 Page 9

10 Operations procedures

Box 2: What is an M&E consultant?

The essence of an M&E consultant is interdisciplinary experience, understanding,insights and instincts. Above all, the consultant must fully grasp that M&E consists ofseveral discrete components, all of which must be addressed. Thus, a consultant shouldhave some epidemiological literacy. However, this may be the least important overallskill, since epidemiological surveillance is typically better developed than other compo-nents. Very detailed surveillance procedures exist, and numerous national and interna-tional organizations can assist with surveillance.

The consultant should, above all, have a firm grounding in programme activity M&E,because it is the weakest component of the overall M&E system with the least developedprocedures and relatively few experienced practitioners. The consultant should havemanaged a project wherein systematic input, output and process data were collected,summarized and used to guide programming. The consultant should understand the flowof data, level by level, from a single youth peer educator or grandmother supportingorphans, to a national information system. Ideally, the consultant should have enough lit-eracy in management information systems to specify the systems needed and to reviewsystems commissioned. Combining epidemiological, evaluation, programme and infor-mation system skills, the consultant should be able to devise an integrated system thatgenerates and links verified primary biological, behavioural, programme and financialdata. The M&E consultant should be committed to simple, low-cost, low-technologyapproaches that may be widely applied in the field.

It is also helpful to emphasize what an M&E consultant should not be. The consultantshould not be an academic researcher concerned with indisputable proof; an epidemiol-ogist who equates M&E with surveillance; an evaluation expert who equates M&E withimpressionistic external evaluations, often conducted without primary data; or a man-agement information systems expert who equates M&E with information technology.

33. NACs are strongly encouraged torecruit an M&E consultant to help build anoverall M&E system. Detailed terms ofreference, an illustrative job advertisementand scoring criteria are presented inAppendix 3.

Recruiting specialized M&E unitsand agencies

34. In keeping with its coordination role,it is suggested that NACs outsource M&Ecomponents, as shown in Panel 4 (see page 11).

35. The suggested terms of reference forthe specialized programme activity moni-toring unit/agency are presented inAppendix 4.

Developing an M&E manual withparticipatory approaches

36. NAC experience highlights the impor-tance of participatory approaches to buildownership and buy-in for M&E. The fol-lowing participatory process is suggested:

• undertake preparatory research;

• determine interim M&E ReferenceGroup and indicators;

• organize district and national stake-holder consultative meetings;

• produce a draft M&E manual;

• organize district and national stake-holder meetings to review draft M&Emanual;

21932_OMS_brCor 09.09.2002 17:21 Page 10

11Operations procedures

Component

Overall system

Surveillance

Research

National NAC, public sector and civilsociety financial management monitoring

National NAC, public sector and civilsociety programme activity monitoring

Contracted out to

NAC/Consultant

Biological: National AIDS or epidemiologyprogramme, supported by surveillanceexpert committee

Behavioural: Universities, research agenciesor consulting firms

Universities/institutions

Major accounting/consulting firms

Major accounting/consulting firms

Panel 4: Recommended M&E agencies by M&E component

• finalize M&E manual; and

• organize a national M&E launchmeeting.

Preparatory research

37. Preparatory research to identify exist-ing M&E approaches, opportunities andconstraints and to identify key issues forfurther analysis should be undertaken inthe first month of the exercise. Preparatoryresearch includes a document review, inter-views and field visits.

38. The document review should includethe following country-specific documents:

• the National Strategic Plan;

• NAC strategic plans, workplans anddraft operations manuals; and

• ministry of health surveillance andM&E reports.

39. Key stakeholder interviews should beundertaken to solicit stakeholders' adviceand concerns regarding M&E. The stake-holders should include:

• the ministry of health, other key min-istries and key implementing partners;

• NAC staff and consultants;

• the expanded UN Theme Group mem-bers, including the UNAIDS CountryProgramme Adviser (CPA);

• major bilateral/multilateral donors;

• major NGOs;

• major academic/research institutions;and

• major groups of people living withHIV/AIDS (PLWHA).

40. Field visits to a broad cross-section ofstakeholders should be undertaken to inter-view field staff and beneficiaries andreview existing M&E systems, procedures,manuals, forms, checklists and reports.These field visits will provide a communi-ty perspective and will also yield fieldinformation on what has and has notworked in M&E. They will also generatepractical examples of effective M&E sys-tems and tools.

41. Based on the document review, inter-views and field visits, a synthesis of exist-ing M&E strengths and gaps and existingand potential M&E resources should beprepared.

21932_OMS_brCor 09.09.2002 17:21 Page 11

12 Operations procedures

Interim M&E Reference Group andindicators

42. An Interim M&E Reference Groupshould be formed by NAC within the firstmonth. This group will meet monthly andon an ad hoc basis, as required, to provideadvice and to review draft outputs. Theinterim group may include the followingrepresentatives:

• ministry of health and other key min-istries;

• NAC;

• expanded UN Theme Group members,including the UNAIDS CPA;

• key bilateral/multilateral donors;

• key NGOs;

• academic/research institutions; and

• PLWHA groups.

43. By the second month, NAC, advisedby the Interim M&E Reference Group,should prepare a set of interim indicatorsand instruments to enable NAC to monitorand evaluate existing projects while a sub-stantive M&E system is developed.Without interim indicators and instru-ments, there is a risk that M&E will not getunder way for several months. The InterimM&E Reference Group will undertake todo the following:

• develop a set of interim indicators;

• develop interim data collection tools;

• develop interim data collection proce-dures;

• institute interim data collection;

• supervise interim data collection; and

• incorporate interim data collection les-sons into substantive M&E plans.

District and national stakeholder consul-tative meetings

44. By the third month, a two-day district-level stakeholder consultative meetingshould be convened to develop a detailed

district M&E strategy, including districtindicators and collection mechanisms.NACs will ensure balanced representationfrom the following public and civil societystakeholders:

• national and provincial NAC mem-bers;

• the ministry of health;

• other key ministries, including Educa-tion and Social Welfare;

• district NGOs and community-basedorganizations (CBOs);

• academic/research/consulting groupswith local activities; and

• local PLWHA groups.

45. Also by the third month, a nationalstakeholder meeting should be convened todevelop a national M&E strategy, includ-ing national indicators and collectionmechanisms. NAC will ensure balancedrepresentation from the following publicand civil society stakeholders:

• the ministry of health and other keyministries;

• NAC staff and consultants;

• the expanded UN Theme Group mem-bers, including the UNAIDS CPA;

• major bilateral/multilateral donors;

• major NGOs;

• academic and research institutions ofnational relevance; and

• major PLWHA groups.

46. The meeting will develop a draftM&E strategy to provide a framework forthe development of a draft M&E system.

Draft M&E manual

47. By the fourth month, and based on theabove-mentioned consultative steps, asmall nucleus of NAC members and con-sultants should develop a draft M&E man-ual, including:

21932_OMS_brCor 09.09.2002 17:21 Page 12

13

• the overall M&E framework;

• the overall governing flowchart;

• the overall M&E database;

• M&E systems and data collectioninstruments; and

• a detailed M&E workplan and budget.

District and national stakeholder consul-tative meetings to review draft M&Emanual

48. Also by the fourth month, a secondseries of (a) district and (b) national stake-holder consultative meetings should beconvened to carefully review and revise thedraft M&E manual. It will be attended bythe same broad range of constituents iden-tified above.

Finalization of M&E manual

49. Based on the feedback and revisionsfrom the district and national meetings andother sources, a final M&E manual will beprepared by the fifth month.

National launch meeting

50. By the sixth month, a half-day nation-al meeting should be held in a major city(or major cities) to launch the M&E manu-al. The launch may be attended by up to200 stakeholders from the public and pri-vate sectors.

51. An illustrative time frame and budgetfor the above-mentioned participatoryprocess are presented in Appendices 11 and12, respectively.

Implementing the agreed M&Esystem

Monitoring and reporting mechanisms

52. The next stage involves implementingthe agreed M&E system, using the follow-ing steps:

(i) Each implementing partner refers tothe planning, monitoring and evalua-tion form in Appendix 5. The reviewedindicators should be listed in column 1and targets agreed with NAC for eachrelevant indicator in column 2. (Not allindicators apply to all implementingpartners.)

(ii) Each implementing partner recordstheir ongoing services on the servicedelivery section of the planning, mon-itoring and evaluation form.

(iii) Every six months, each implementingpartner uses the planning, monitoringand evaluation form to report theirprogress towards agreed targets in col-umn 2. The specialized agency veri-fies data every six months.

(iv) During the six-monthly data-verificationvisits, the specialized unit/agencyassesses the quality of services deliv-ered, using quality-assurance check-lists adapted by NACs for each coun-try context. Examples of 'programmeareas requiring quality-assurancechecklists' and 'an illustrative quality-assurance checklist for interpersonalcommunication' are presented inAppendices 6 and 7, respectively.

(v) The specialized agency uses the plan-ning, monitoring and evaluation formto rate progress towards targets everysix months, according to the followingsimple rating scale:

• Targets largely/completely attained

• Targets partially attained

• Targets largely/completely unattained

(vi) The specialized agency collates,analyses and submits to NAC summa-ry reports of aggregate activity everysix months, using the planning, moni-toring and evaluation form and a struc-tured progress report form.


21932_OMS_brCor 09.09.2002 17:21 Page 13

14

Programme activity monitoring throughintermediate structures

53. The above-mentioned operational pro-cedures refer to situations in which the spe-cialized monitoring unit/agency directlymonitors implementing partners. There areseveral situations in which the specializedentity will work through intermediaries tomonitor implementing partners. Threeexamples are presented below:

• provincial intermediaries, in contextswhere NACs devolve major grant-provision and monitoring responsibili-ties to provincial bodies;

• district intermediaries, in contextswhere NACs devolve major grant-provision and monitoring responsibili-ties to district bodies; and

• NGO intermediaries, in contextswhere NACs devolve major grant-provision and monitoring responsibili-ties to NGOs.

54. How is programme activity monitor-ing undertaken when intermediary bodiesare involved? The steps listed below aresuggested:

(i) A specialized monitoring entity is stillrequired, even when elements of mon-itoring are devolved to districts.However, the entity's role changesfrom direct implementation to capacity-building and supervision.

(ii) The specialized entity trains and sup-ports intermediary bodies to equipthem with the skills to undertake pro-gramme activity monitoring.

(iii) The specialized entity supervises andmonitors intermediary bodies, to ensurethat their monitoring is timely, method-ical, complete, accurate and in accor-dance with the procedures manual.

(iv) The specialized entity is still responsi-ble for collecting, verifying, entering

and analysing primary M&E data fromimplementing partners and for prepar-ing and submitting regular M&Ereports to NAC.

Access database

55. For the overall M&E system, an M&Edatabase has been written in MicrosoftAccess and may be obtained from theWorld Bank. The database is written inAccess because most countries haveAccess packaged with Microsoft Office,most consulting firms use it and most localIT specialists support it.

Checklist for programme managers

56. A checklist to assist programme man-agers in managing the M&E function ispresented in Appendix 8.

Indicative M&E budget

57. The UNAIDS Monitoring andEvaluation Reference Group (MERG) isworking on a costing model for an overallprogramme. Appendix 9 offers an indica-tive budget for the M&E componentsemphasized in this manual, specifically thedesign of an overall M&E system and theprogramme activity monitoring compo-nent. The budget in Appendix 9 does notinclude cost estimates for the surveillance,research or financial management compo-nents.

58. The indicative budget includes thecosts of designing the overall M&E systemand the programme activity monitoringsystem, but it does not include the costs ofactually operating the system. It is possibleto offer tentative guidelines concerningthese costs. The key factor influencing costwill be whether financial monitoring andprogramme monitoring are combined anddelegated to a single entity. Delegating thisactivity to a single entity is significantlycheaper. If this is done, it is estimated thatprogramme activity monitoring may beconducted for 2-3% of the total cost of


21932_OMS_brCor 09.09.2002 17:21 Page 14

15

grant funds transferred to partners. Iffinancial and programme activity monitor-ing are not combined, but done separately,it is estimated that programme activitymonitoring will require 5-6% of the total

costs of grant funds transferred to partners.It should be noted that these percentagesapply only to grant funds transferred toimplementing partners, not to the overallNAC budget.


21932_OMS_brCor 09.09.2002 17:21 Page 15

21932_OMS_brCor 09.09.2002 17:21 Page 16

APPENDICES 1 TO 9: TOOLS

21932_OMS_brCor 09.09.2002 17:21 Page 17

Indicator

Prevention

1. HIV and syphilis prevalence among (a) all antenatal women; (b) women aged15-19; and (c) women aged 20-24

Mitigation

2. Increased quality of life for PLWHA and orphans and other vulnerable chil-dren (OVC)

Prevention

3. Percentage of respondents who both correctly identify ways of preventing thesexual transmission of HIV and reject major misconceptions about HIV trans-mission and prevention

4. Safer sexual practices: youth (15-19) The (a) increased age of sexual inception and (b) reduced occurrence of unpro-tected sexual intercourse

5. Safer sexual practices: adults (20-49) Reduced occurrence of unprotected sexual intercourse with non-regular partner

Mitigation

6. Increased PLWHA/OVC household-coping capacities

Increase NAC capacity

7. NAC board and staff appointed and functional

8. NAC workplans and budgets developed

9. NAC financial, procurement, implementation, technical support and M&E sys-tems established

10. NAC fund disbursement ratios

11. Number and percentage of districts with HIV/AIDS workplans and budgetsapproved and funded

Increase public sector services

12. The (a) number and (b) percentage of line ministries with HIV/AIDS workplansand budgets for employees

13. The (a) number and (b) percentage of health facilities providing HIV/AIDS careappropriate for level of facility

14. The (a) number and (b) percentage of primary/secondary/tertiary educationinstitutions with HIV/AIDS programmes for their students

15. The (a) number and (b) percentage of districts with functioning social welfaredepartments providing grants to orphans and other vulnerable children

16. Total AIDS services delivered by public sector

18 Appendices 1 to 9: Tools

APPENDIX 1. Illustrative indicators3

3Although the text presents the results chain in causal order, from inputs to impact, this appendix presents theresults chain in conventional logical framework order, from impact to inputs.

Data source

Antenatal surveillance

Household surveys

DHS, MICS and BSS

Behavioural surveillanceand social impact surveys

Behavioural surveillanceand social impact surveys

Household surveys

NAC reports

NAC workplans and budgets

NAC reports

NAC reports

NAC reports

NAC reports

Health facility surveys

NAC reports

NAC reports

NAC reports

Outcome level (behavioural outcomes)

Impact level (health impact)

Output level (activities)

21932_OMS_brCor 09.09.2002 17:21 Page 18

19Appendices 1 to 9: Tools

Increase civil society services

17. Number of civil society organizations receiving NAC funding

18. Percentage of overall funding granted to civil society services

19. Number of new civil society partners introduced to HIV/AIDS programmingwith NAC support

20. Total HIV/AIDS services delivered by civil society

HIV/AIDS services: prevention

21. The (a) number of HIV/AIDS radio/television programmes produced and (b)number of hours aired

22. The number of HIV/AIDS prevention brochures/booklets (a) developed and(b) distributed

23. The number of (a) HIV prevention staff and (b) volunteers trained

24. The (a) number of HIV prevention meetings held and (b) number ofmen/women reached

25. The number of condoms sold/given

26. The number of men/women receiving STI care from health facilities withtrained staff and uninterrupted supply of drugs

27. The (a) number and (b) percentage of men/women receiving HIV testing andcounselling

28. The (a) number and (b) percentage of women tested and receiving PMTCT ifHIV-positive

HIV/AIDS services: care

29. Number of care (a) staff and (b) volunteers trained

30. The (a) number of PLWHA support groups; (b) number of men/womenenrolled; and (c) percentage of men/women enrolled

31. The (a) number of community HIV/AIDS care projects; (b) number ofmen/women enrolled; and (c) percentage of men/women enrolled

32. The (a) number of community orphan support projects; (b) number of orphanboys/girls enrolled; and (c) estimated percentage of orphan boys/girls enrolled

33. The (a) number and (b) estimated percentage of orphan boys/girls receivingsupport for school fees

34. Paid staff, volunteers recruited, training conducted, equipment and resourcesprovided

NAC reports

NAC reports

NAC reports

NAC reports

NAC reports

NAC reports

NAC reports

NAC reports

NAC reports

NAC reports

NAC reports

NAC reports

NAC reports

NAC reports

NAC reports

NAC reports

NAC reports

NAC records

Input level (personnel, training, equipment and funds)

21932_OMS_brCor 09.09.2002 17:21 Page 19


APPENDIX 2. Summary terms of reference for NAC M&E staff

• Prepare an overall and annual M&E plan

• Prepare semi-annual and annual M&E reports

• Prepare technical specifications for each M&E component and contract external agencies to manage each component

• Supervise the quality and timeliness of M&E products contracted out

• Review contracted M&E products and distil and communicate their implications forprogramme implementation, including modifications in geographic priorities, targetgroups, interventions and implementing partners. M&E must play a central role in shaping programme direction

21932_OMS_brCor 09.09.2002 17:21 Page 20


Background

The National AIDS Council (NAC) wasestablished in [date] and is recruiting staff.It is also developing its management sys-tems, including financial management,procurement, implementation managementand M&E systems.

A consultant is required for [number ofmonths], from [date] to [date], to helpNAC develop its M&E system.

Objectives

The key objectives of the consultancy are:

• to help NAC develop an overall M&Ecoordination plan, with manuals, sys-tems, procedures, tools, a database,flowcharts for data and clearly speci-fied institutional roles and responsi-bilities and an implementation planand budget; and

• to strengthen NAC's monitoring sys-tems, to ensure sound output andprocess monitoring.

There are existing M&E systems, manualsand tools prepared for HIV/AIDS pro-grammes elsewhere in Africa, but thesesystems, manuals and tools must be adapt-ed to the country context, in consultationwith a broad range of country stakeholders.

Selection

The consultant will be selected by NAC, inconsultation with its major partners, andhoused and supported by NAC. The con-sultant will receive technical support fromNAC and other technical partners.

Skills and experience

The key skills required for the consultancyare:

• communication and facilitation skills,to ensure that the M&E system, man-ual and tools are developed with the

full participation of NAC, govern-ment, and corporate and civil societystakeholders;

• coordination skills, to ensure that thecontributions of diverse stakeholdersare effectively harmonized into oneNAC M&E system, with supportingmanual and tools;

• analytical skills, to ensure that theNAC's M&E system, with supportingmanual and tools, is detailed and logi-cal; and

• system-building skills, to ensure thatthe NAC's M&E system is compre-hensive, internally consistent, specificand self-contained, and that it can beimplemented.

Tasks

The consultant's specific tasks are to:

• identify data/reporting needs;

• design the overall M&E system;

• prepare an implementation plan;

• train coordinating and implementingpartners;

• ensure that the system is tested,refined and fully implemented byNAC; and

• specify further steps that NAC musttake.

These tasks are outlined in detail below.

1. Identify data/reporting needs

The consultant will build on major activi-ties already undertaken by NAC. The con-sultant will rapidly review NAC's existinglogical framework matrix and indicator sets and propose a circumscribed set ofoperational, measurable indicators, withdetailed input, output, process, outcomeand impact measures. The consultant may

APPENDIX 3. Detailed terms of reference for NAC M&E consultant

21932_OMS_brCor 09.09.2002 17:21 Page 21


help NAC streamline its logical frame-work. Ideally, the logical framework willnot contain more than 40 indicators.

This task will begin on [date] and be com-pleted, with agreement from all stakehold-ers, by [date].

2. Design the overall M&E system

The consultant will then help NAC to designan M&E system and to prepare an M&Emanual. This M&E system should, as far aspossible, reflect the National Strategic Plan.The consultant will use the current NationalAIDS Councils: Monitoring and evaluationoperations manual as a point of departurewhen developing NAC's M&E manual.

During preparation of NAC's M&E systemand manual, the consultant will do the fol-lowing:

• Review documents for the followingcomponents of a comprehensive M&Esystem: (i) biological surveillance; (ii)behavioural surveillance; (iii) research,especially prevalence, incidence andintervention impact studies; (iv)financial management; and (v) pro-gramme activity.

• Assess how well developed each ofthe above M&E components is in thecountry context and identify compo-nents requiring concerted action.

• Collect and include as appendices orseparate volumes all existing proce-dures manuals that exist for specificcomponents of NAC's M&E system,including biological and behaviouralsurveillance and clearly defined sub-components, including blood safety,PMTCT and VCT.

• Outline a strategic vision and specificscope of work for M&E in NAC,which clarifies NAC's coordinationrole and increases its capacity to coor-dinate, rather than implement, M&E.

• Describe precisely how, to whom andwithin what period NAC will dele-gate/contract out each of the compo-nents and subcomponents of thenational M&E system.

• Present detailed technical specifica-tions for each component of the over-all M&E system.

• Outline a participatory process thatNAC may undertake to ensure thatstakeholders are fully involved in thedevelopment of NAC's M&E system,to ensure ownership and buy-in. Theparticipatory process will include thefollowing steps: document review,stakeholder interviews and field vis-its; formation of an Interim M&EReference Group and indicators; dis-trict and national stakeholder consul-tative meetings; preparation of a draftNAC M&E manual; district andnational stakeholder meetings toreview the draft manual; finalizationof NAC's M&E manual; and a nation-al meeting to launch NAC's M&E planand manual.

• Undertake the participatory processoutlined above to ensure that NAC'sM&E plan is developed in a consulta-tive manner.

• Prepare a draft NAC M&E manual,based partly on the current NationalAIDS Councils: M&E operationsmanual. This manual will reflect theoperational indicators developed forthe logical framework, outline anoverall data collection, collation,analysis and reporting system andspecify the reporting responsibilitiesand interrelationships of all the insti-tutions involved in M&E at all levels.The manual will include detailed datacollection procedures, specific flow-charts to illustrate the precise dataflow from tier to tier, and prototypesof each of the data collection instru-

21932_OMS_brCor 09.09.2002 17:21 Page 22


ments required. It will be prepared sothat it is clear what is required in eachcomponent and at each tier from dis-trict to national level. It may includeseparate modules, which specify assuccinctly as possible what is requiredin each component and at each tier.

• Subject NAC's draft manual to inten-sive review by a broad range of stake-holders and revise the manual inaccordance with guidance received.

• Finalize NAC's draft M&E manualbased on the above guidance.

• Prepare a flowchart and template forthe development of an Access data-base to capture all the data required byNAC's M&E system.

• Liaise with an IT consultant to ensurethat the Access database captures allthe data required by NAC.

• Prepare flowcharts, guidelines andprototypes for M&E reports from eachof the implementing partners to NACand from NAC to parliament.

This component will begin on [date] andbe completed by [date].

3. Prepare an implementation plan

The implementation plan will include adetailed workplan, time frame, key mile-stones and budget for each of the M&Ecomponents proposed in NAC's M&Emanual.


4. Train coordinating and implement-ing partners

The consultant will also train key monitor-ing partners, particularly NAC, line min-istries, NGOs and districts, to use the man-ual to implement effective M&E.


5. Ensure that the system is tested,refined and fully implemented byNAC

The consultant will work with NAC tofield-test the entire M&E system. The con-sultant will revise NAC's M&E system inaccordance with feedback during the fieldtest. The consultant will then work withNAC to ensure that the systems are fullyimplemented for at least three monthsbefore NAC assumes sole responsibilityfor coordinating M&E.


6. Specify further steps that NAC musttake

The consultant will then present a detailedplan outlining further steps that NAC musttake in order to continue to consolidateM&E, including the use of mechanismswhereby NAC can ensure that the manualis updated through regular feedback andconsultative processes, as required.

NAC may choose to extend the consultancyto continue implementation beyond theinitial period envisaged in the above-mentioned points.


21932_OMS_brCor 09.09.2002 17:21 Page 23


Background

A consultant is required for an initial peri-od of [number of months], with a possibleextension, to develop and test an M&E sys-tem for a development programme. TheM&E system will be developed in consul-tation with stakeholders and documentedin a detailed M&E operations manual. Theconsultant's major responsibilities are tofacilitate the participatory planningprocess, design the overall M&E systemand prepare the M&E operations manual.The consultant should have good commu-nication, facilitation, planning and analyti-cal skills. Applicants should have the fol-lowing qualifications:

Qualifications

• A master's or doctoral degree or equiv-alent in a health, social science, man-agement or engineering discipline andthree years' relevant work experience

OR• A bachelor's degree in a health, social,

management or engineering sector andsix years' relevant work experience

• Specific experience in facilitating aparticipatory planning process foran activity involving at least 100staff members or a budget of overUS$1 million

ANDSpecific experience in developingmanagement or M&E systems for anorganization/programme employing at

least 100 staff members or with abudget of over US$1 million

ANDSpecific experience implementingmanagement or M&E systems for anorganization/programme employing atleast 100 staff members or with abudget of over US$1 million

• A high level of computer literacy, par-ticularly in the use of Word, theInternet and e-mail. Knowledge ofPowerPoint, Excel and Access is aplus.

Compensation

The position offers attractive remuneration,working conditions and professional devel-opment opportunities.

Applications

Candidates are requested to submit a letteroutlining why they are qualified for theposition, and a CV that includes the appli-cant's telephone number and the telephonenumbers of at least two referees who havesupervised the applicant's professionalwork. They should also attach or mailexamples of management or M&E sys-tems and manuals they have prepared,indicating their specific contribution to theproduct.

Applications may be mailed or submittedelectronically to the address below[address to be inserted]. Candidates areencouraged to apply electronically.

APPENDIX 3.1. Draft advertisement for NAC M&E consultant

21932_OMS_brCor 09.09.2002 17:21 Page 24

25

APPENDIX 3.2. Proposed NAC consultant selection scoring criteria

Criteria

Academic qualifications

Relevant professional experience

Computer skills

Total

Weighting

20

60

20

100

Appendices 1 to 9: Tools

21932_OMS_brCor 09.09.2002 17:21 Page 25


APPENDIX 4. Summary terms of reference for specialized programme activity monitoring entity

• To verify the internal consistency and validity of service delivery data reported byNAC implementing partners, through at least six-monthly visits

• To cross-validate programme and financial data, and to increase confidence in bothdata sources

• To assess the quality of implementing partners' services, using agreed quality-assurance checklists, through at least six-monthly visits

• To develop a simple management Access database that can be shared widely and usedfor further analyses

• To collect, enter and analyse implementing partner programme monitoring datamonthly

• To assist NAC in identifying implementing partners whose performance is exempla-ry and may serve as a positive example, and implementing partners who are under-performing, for whom corrective action will be suggested

• To prepare the six-monthly programme monitoring reports, containing summary data,reviewing overall performance against targets and making overall programme recom-mendations, including recommendations to improve both programme performanceand M&E

21932_OMS_brCor 09.09.2002 17:21 Page 26

27

Indicator

Prevention

1. HIV and syphilis prevalence among (a) all antenatal women; (b)women aged 15-19; and (c) women aged 20-24

Mitigation

2. Increased quality of life for PLWHA and OVC

Prevention

3. Percentage of respondents who both correctly identify ways of pre-venting the sexual transmission of HIV and reject major misconceptionsabout HIV transmission and prevention

4. Safer sexual practices: youth (15-19) The (a) increased age of sexual inception and (b) reduced occurrence ofunprotected sexual intercourse

5. Safer sexual practices: adults (20-49) Reduced occurrence of unprotected sexual intercourse with non-regularpartner

Mitigation

6. Increased PLWHA/OVC household-coping capacities

Increase NAC capacity

7. NAC board and staff appointed and functional

8. NAC workplans and budgets developed

9. NAC financial, procurement, implementation, technical support andM&E systems established

10.NAC fund disbursement ratios

11.The number and percentage of districts with HIV/AIDS workplansand budgets approved and funded

Increase public sector services

12.The (a) number and (b) percentage of line ministries with HIV/AIDSworkplans and budgets for employees

13.The (a) number and (b) percentage of health facilities providingHIV/AIDS care appropriate for level of facility

14.The (a) number and (b) percentage of primary/secondary/tertiaryeducation institutions with HIV/AIDS programme for their students

15.The (a) number and (b) percentage of districts with functioning socialwelfare departments providing grants to OVC

16.Total HIV/AIDS services delivered by public sector


Ratingof

progress

Progresstowardstargets

Agreedtargets

APPENDIX 5. Planning, monitoring and evaluation form

Impact level (health impact)

Output level (activities)

Outcome level (behavioural outcomes)

21932_OMS_brCor 09.09.2002 17:21 Page 27


Increase civil society services

17.The number of civil society organizations receiving NAC funding

18.The percentage of overall funding granted to civil society services

19.The number of new civil society partners introduced to HIV/AIDSprogramming with NAC support

20.Total HIV/AIDS services delivered by civil society


21.The (a) number of HIV/AIDS radio/television programmes producedand (b) number of hours aired

22.The number of HIV/AIDS prevention brochures/booklets (a) devel-oped and (b) distributed

23.The number of (a) HIV prevention staff and (b) volunteers trained

24.The number of (a) HIV prevention meetings held and (b) men/womenreached

25.The number of condoms sold/given

26.The number of men/women receiving STI care from health facilitieswith trained staff and uninterrupted supply of drugs

27.The (a) number and (b) percentage of men/women receiving HIV test-ing and counselling

28.The (a) number and (b) percentage of women tested and receivingPMTCT if HIV-positive


29.Number of care (a) staff and (b) volunteers trained

30.The (a) number of PLWHA support groups; (b) number ofmen/women enrolled; and (c) percentage of men/women enrolled

31.The (a) number of community HIV/AIDS care projects; (b) numberof men/women enrolled; and (c) percentage of men/women enrolled

32.The (a) number of community orphan support projects; (b) number oforphan girls/boys enrolled; and (c) estimated percentage of orphanboys/girls enrolled

33.The (a) number and (b) estimated percentage of orphan boys/girlsreceiving support for school fees

34.Paid staff, volunteers recruited, training conducted, equipment andresources provided

Input level (deliver personnel, training, equipment and funds)

21932_OMS_brCor 09.09.2002 17:21 Page 28



21. Number of HIV/AIDS radio/television programmesproduced and number of hours aired

22. Number of HIV/AIDS prevention brochures/bookletsdeveloped and numbers distributed

23. Number of HIV prevention staff and volunteers trained

24. Number of HIV prevention meetings held andmen/women reached

25. Number of condoms sold/given

26. Number of health facilities providing STI care withboth trained staff and uninterrupted supply of drugs

27. Number and percentage of men/women receiving HIVtesting and counselling

28. Number and percentage of women tested and receivingPMTCT if HIV-positive


29. Number of care staff and volunteers trained

30. Number of PLWHA support groups and number andpercentage of men/women enrolled

31. Number of community HIV/AIDS care projects andnumber and percentage of men/women enrolled

32. Number of community orphan support projects andnumber and percentage of boys/girls enrolled

33. Number and percentage of boys/girls receiving supportfor school fees

Radio programmes produced:Television programmes produced:Number of hours radio programmes aired:Number of hours television programmes aired:

Number of HIV/AIDS prevention brochuresdeveloped:Number of HIV/AIDS prevention brochure distributed:

Staff trained:Volunteers trained:

Meetings held:Men reached:Women reached:

Condoms sold:Condoms given:

Number:

Number men:Number women:Percentage men:Percentage women:

Number:Percentage:

Staff:Volunteers:

PLWHA groups:Number men enrolled:Number women enrolled:Percentage men enrolled:Percentage women enrolled:

Care projects:Number men enrolled:Number women enrolled:Percentage men enrolled:Percentage women enrolled:

Orphan projects:Number boys enrolled:Number girls enrolled:Percentage boys enrolled:Percentage girls enrolled:

Number boys:Number girls:Percentage boys:Percentage girls:

APPENDIX 5.1. Service delivery sub-section

21932_OMS_brCor 09.09.2002 17:21 Page 29

30

Programme areaPrevention:Mass communicationInterpersonal communicationCondom distribution and promotionSTI careHIV counselling and testingBlood safetyPMTCT Care:PLWHA supportClinical AIDS careCommunity HIV/AIDS careOrphans and other vulnerable children


APPENDIX 6. Programme areas requiring quality-assurance checklists

21932_OMS_brCor 09.09.2002 17:21 Page 30


No.

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

NoQuality-assurance questions

Was the meeting place as cool and airy as possible?

Was the audience sitting comfortably on seats or mats, for indoor meetings,or sitting or standing under shade, for outdoor meetings?

Was the audience arranged in a horseshoe, within 5 metres of the presen-ter(s) for lectures or discussion and within 7 metres for participatory activi-ties?

Were there at least 10 people in the audience, excluding the presenter/facil-itator(s)?

Did the presenter/facilitator(s) talk loudly enough for the audience to hear?

Did the audience listen quietly, when the presenter/facilitator(s) spoke, andwere disruptive or drunk people silenced?

If there was a lecture, was it no longer than 10 minutes?

Was there at least one participatory activity, followed by a discussion?

Did the audience show enthusiasm during the participatory activity?

Was all the factual information that was presented in the lecture, participa-tory activity or discussion, accurate and up to date?

Did the lecture and/or participatory activity avoid blaming women for thespread of HIV/STIs?

Were there at least 20 minutes, preferably 30 minutes, for discussion?

Did at least 5, preferably 10, members of the audience join in the discus-sion?

Was the number of women contributing to the discussion proportionate tothe number of women in the audience?

During the discussion, did the presenter/facilitator(s) listen to each commentwithout showing facial disapproval or interrupting (except where the speak-er was drunk or deliberately disruptive)?

Did the presenter/facilitator(s) respond very briefly to each comment, ask-ing the audience to comment further, without answering the comment per-sonally?

Did the presenter/facilitator(s) lead the discussion away from basic factsabout HIV/AIDS to attitudes, values and personal concerns?

When women and HIV/AIDS/STIs were discussed, did the presenter/facili-tator(s) guide the audience to focus on men's responsibility?

Did the presenter/facilitator(s) offer condoms at the end of the meeting?

Did the presenter/facilitator(s) end by telling the audience where and whenthey could contact the project officials for further information and refer theaudience for other requested services, such as STI care or VCT?

Yes

APPENDIX 7. Illustrative quality-assurance checklist for interpersonalcommunication

21932_OMS_brCor 09.09.2002 17:21 Page 31


NoYesStep

Does the project manager possess, or has the project manager contractedout, sufficient M&E expertise to provide effective oversight?

Has the project manager reviewed the Programme Description Summaryto ensure that its logic is consistent with the M&E manual?

Has the project manager reviewed existing national M&E activities, iden-tifying which components are already addressed, and which need to be fur-ther addressed?

Has the project manager ensured that NAC fully understands, and isunequivocally committed to, coordination and not implementation?

Does NAC have an M&E plan and is it sufficiently detailed, operationaland time-bound to ensure that comprehensive and timely M&E will occur?

Does NAC have sufficient internal, or contracted, capacity to coordinateM&E?

Has NAC contracted out each M&E component to specialized agencies?

Do contracted agencies have the capacity to conduct the required M&Eactivities on time?

Are terms of reference and technical specifications for each monitoringcomponent clear and comprehensive?

Is there an adequate participatory process to ensure national engagementand ownership of the overall M&E plan?

Is each specialized agency submitting high-quality M&E inputs, on time,as specified in the M&E plan and contracts?

If not, is NAC taking prompt and adequate corrective action?

Are NAC and partners meeting as agreed in the workplan to review per-formance?

Is the Planning, monitoring and evaluation form being completed com-prehensively and on time?

Is M&E being used to make prompt and appropriate project adjustments?

Are M&E manuals and procedures being updated on the basis of pro-gramme learning?

APPENDIX 8. Checklist for project managers in NACs and donor agencies

No.

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

21932_OMS_brCor 09.09.2002 17:21 Page 32


Activity

International consultant to develop frame-work and assist in identifying and support-ing local M&E consultant

National consultant to design overall M&Esystem and programme activity monitoringmanual and system

National IT specialist to adapt Access data-base for national use

Participatory process to ensure stakeholdercommitment and understanding

Total

Budget

20 days x US$700 daily for internationaltravel, accommodation, meals, local trans-port and communication and fees= US$14,000Subtotal = US$14,000

150 days x US$350 daily for accommoda-tion, meals, local transport and communi-cation and fees = US$52,500Subtotal = US$52,500

15 days x US$300 daily for accommoda-tion, meals, local transport and communi-cation and fees = US$4,500Subtotal = US$4,500

See detailed budget in Appendix 12, whichdoes not include national consultant feesbudgeted within 150 days aboveSubtotal = US$60,000

US$131,000

APPENDIX 9. Indicative budget for design of overall M&E system andprogramme activity monitoring

Note: This budget was prepared in 2001 with US$ and is an average figure. It needs to beadjusted for individual contexts.

21932_OMS_brCor 09.09.2002 17:21 Page 33

21932_OMS_brCor 09.09.2002 17:21 Page 34

APPENDICES 10 TO 12:FURTHER INFORMATION

21932_OMS_brCor 09.09.2002 17:21 Page 35

36 Appendices 10-12: Further information

APPENDIX 10. Key sources for further information on M&E

The major sources for guidelines cited below are UNAIDS, WHO, MEASURE and FHI.The latest versions of these guidelines may be found on the Internet at:

http://www.unaids.orghttp://www.who.inthttp://www.cpc.unc.edu/measurehttp://www.fhi.orghttp://www.cdc.govhttp://www.usaid.gov

Family Health International (2002) Evaluating Programs for HIV/AIDS Prevention and Carein Developing Countries: A Handbook for Program Managers and Decision Makers.Washington: Family Health International.(http://www.fhi.org/en/aids/impact/impactpdfs/evaluationhandbook.pdf)

UNAIDS/MEASURE (2000) National AIDS Programmes: A Guide to Monitoring andEvaluation. Geneva: UNAIDS.(http://www.cpc.unc.edu/measure/guide/guide.html)

Family Health International (2000) Behavioural Surveillance Surveys (BSS): Guidelines forRepeated Behavioural Surveys in Populations at Risk for HIV. Arlington: Family HealthInternational.(http://www.fhi.org/en/aids/wwdo/wwd12a.html#anchor545312)

Centers for Disease Control and Prevention (2002) Strategic Monitoring and Evaluation: ADraft Planning Guide and Related Tools for CDC GAP Country Programs. Atlanta: Centers forDisease Control and Prevention.

UNAIDS/Family Health International (2000) Second Generation Surveillance for HIV: TheNext Decade. Geneva: UNAIDS.(http://www.who.int/emc-documents/aids_hiv/docs/whocdscsredc2005.PDF)

UNAIDS/WHO (1999) Guidelines for Sexually Transmitted Infections Surveillance.UNAIDS/WHO Working Group on Global HIV/AIDS/STI Surveillance. Geneva: UNAIDS.(http://www.unaids.org/publications/documents/impact/std/JC240-SexTransmInfSurv-E.pdf)

UNAIDS (1999) Acting Early to Prevent AIDS: The Case of Senegal. Geneva: UNAIDS.(http://www.unaids.org/publications/documents/epidemiology/determinants/una99e34.pdf)

UNAIDS (1998) The Relationship of HIV and STD Declines in Thailand to BehaviouralChange. Geneva: UNAIDS.(http://www.unaids.org/publications/documents/epidemiology/determinants/una98e2.pdf)

UNAIDS (1998) A Measure of Success in Uganda. Geneva: UNAIDS.(http://www.unaids.org/publications/documents/epidemiology/determinants/una98e8.pdf)

UNAIDS/Family Health International (1998) Meeting the Behavioural Data Collection Needs of National HIV/AIDS and STD Programmes. Geneva: UNAIDS.(http://www.fhi.org/en/aids/impact/imppub/bdcbiback.html#anchor1086792)

21932_OMS_brCor 09.09.2002 17:21 Page 36

37

APPENDIX 11. Illustrative time frame for participatory planning process

Month6

Month 5

Month 4

Month 3

Month 2

Month1

Action


Interim M&E Reference Group andindicators

District and national stakeholder meet-ing to develop draft M&E manual

Development of draft M&E manual

District and national stakeholdermeeting to review draft M&E manual

Finalization of M&E manual

National meeting to launch M&Emanual

Appendices 10-12: Further information

X

X X

X

X

X

X

X

21932_OMS_brCor 09.09.2002 17:21 Page 37

38

Activity


Interim M&E Reference Group and indicators

District and national stakeholder consultativemeeting to develop manual

Development of draft manual

District and national stakeholder consultativemeeting to review draft manual

Finalization of manual

National meeting to launch manual

Consultancy support

Total

Appendices 10-12: Further information

APPENDIX 12. Illustrative budget for participatory planning process

Budget

Monthly meetings: 6 meetings @ US$1,000per meeting = US$6,000Subtotal = US$6,000

Facilitation: US$3,000Consumables: US$2,000Transport and 2 days’ accommodation: 20people @ US$400 = US$8,000 x 2 (districtand national)Subtotal = US$21,000

Facilitation: US$3,000Consumables: US$2,000Transport and 2 days’ accommodation: 20people @ US$400 = US$8,000 x 2 (districtand national)Subtotal = US$21,000

M&E guide: US$7,000Transport and refreshments for 200 partici-pants: 200 people @ US$25 = US$5,000Subtotal = US$12,000

100 person days @ US$300 = US$30,000Subtotal = US$30,000

US$90,000

Note: This budget was prepared in 2001 with US$ and is an average figure. It needs to be adjust-ed for individual contexts. The budget is only for the participatory component of the over-all M&E planning process.

21932_OMS_brCor 09.09.2002 17:21 Page 38

39

Notes

21932_OMS_brCor 09.09.2002 17:21 Page 39

40

Notes

21932_OMS_brCor 09.09.2002 17:21 Page 40

41

Notes

21932_OMS_brCor 09.09.2002 17:21 Page 41

42

Notes

21932_OMS_brCor 09.09.2002 17:21 Page 42

© Joint United Nations Programme on HIV/AIDS (UNAIDS) 2002.

All rights reserved. This document may be freely reviewed, quoted, reproduced or translated, in part or in full, provided the source is acknowledged. The document may not be sold or used in conjunction with commercial purposes without prior written approval from UNAIDS (contact: UNAIDS Information Centre).

The views expressed in documents by named authors are solely the responsibility of those authors.

The designations employed and the presentation of the material in this work do not imply the expression of any opinion whatsoever on the part of UNAIDS concerning the legal status of any country, territory, city or area or of its authorities, or concerning the delimitation of its frontiers and boundaries.

The mention of specific companies or of certain manufacturers products does not imply that they are endorsed or recommended by UNAIDS in preference to others of a similar nature that are not mentioned. Errors and omissions excepted, the names of proprietary products are distinguished by initial capital letters.

UNAIDS - 20 avenue Appia - 1211 Geneva 27 - SwitzerlandTel. (+41) 22 791 36 66 - Fax (+41) 22 791 41 87

E-mail: [email protected] - Internet: http:/www.unaids.org

UNAIDS/02.47E (English original, August 2002)ISBN: 92-9173-227-3

The Joint United Nations Programme on HIV/AIDS (UNAIDS) is the leading advocate for global action on HIV/AIDS. It brings together eight UN agencies in a common effort to fi ght the epidemic: the United Nations Children’s Fund (UNICEF), the United Nations Development Programme (UNDP), the United Nations Population Fund (UNFPA), the United Nations International Drug Control Programme (UNDCP), the United Nations Educational, Scientifi c and Cultural Organization (UNESCO), the World Health Organization (WHO), the International Labour Organization (ILO) and the World Bank.

UNAIDS both mobilizes the responses to the epidemic of its eight cosponsoring organizations and supplements these efforts with special initiatives. Its purpose is to lead and assist an expansion of the international response to HIV on all fronts: medical, public health, social, economic, cultural, political and human rights. UNAIDS works with a broad range of partners—governmental and NGO, business, scientifi c and lay—to share knowledge, skills and best practice across boundaries.

Produced with environment-friendly materials

21932_OMS_CouvCor 09.09.2002 16:55 Page 2

The National AIDS Councils: Monitoring and Evaluation Operations Manual is designed as a practical toolkit and road map for practitioners to use in designing and implementing programme monitoring and evaluation (M&E). The manual introduces key concepts; presents simple, clear procedures, with a checklist of the process, timing and costs of building participatory programme M&E for National AIDS Councils (NACs); and offers key tools that implementing partners need for M&E. The manual emphasizes the development of the overall M&E system, in relation to the National Strategic Plan, and the monitoring of services provided through NACs and their implementing partners.

The World Bank1818 H Street, NW

Washington, DC, 20433 USA Telephone: (202) 473 1000

Fax: (202) 477 6391E-mail: [email protected]

Internet: http://www.worldbank.org

UNAIDS 20 avenue Appia

1211 Geneva 27, SwitzerlandTelephone: (+41) 22 791 36 66

Fax: (+41) 22 791 41 87E-mail: [email protected]

Internet: http://www.unaids.org



21932_OMS_CouvCor 09.09.2002 16:55 Page 1

national aids councils - fsn network · the national aids councils: monitoring and evaluation...

Documents