handbook of democracy and governance ...democracy objectives. the indicators included in this...

CENTER FOR DEMOCRACY AND GOVERNANCE“...promoting the transition to and consolidation of democratic regimes throughout the world.”

HANDBOOK OFDEMOCRACY AND GOVERNANCE

PROGRAM INDICATORS

August 1998

Technical Publication Series

Center for Democracy and GovernanceBureau for Global Programs, Field Support, and Research

U.S. Agency for International DevelopmentWashington, D.C. 20523-3100

TO ORDER THIS DOCUMENT FROM THE DEVELOPMENT EXPERIENCE CLEARINGHOUSE:

C Please reference the document title (Handbook of Democracy and Governance Program Indicators)and document identification number (PN-ACC-390).

C USAID employees, USAID contractors overseas, and USAID sponsored organizations overseas mayorder documents at no charge.

C Universities, research centers, government offices, and other institutions located in developingcountries may order up to five titles at no charge.

C All other institutions and individuals may purchase documents. Do not send payment. Whenapplicable, reproduction and postage costs will be billed.

Fax orders to: (703) 351-4039 Attn: USAID Development Experience Clearinghouse (DEC)E-mail orders to: [email protected]

ABOUT THE TECHNICAL PUBLICATION SERIESThe USAID Center for Democracy and Governance Technical Publication Series was launched in March1998. The series includes publications intended principally for USAID personnel; however, all personsinterested in the sector may benefit from the series. Authors of individual publications may be USAIDofficials and/or other individuals from the public and private sector. The Center for Democracy andGovernance reserves the right to review and edit all publications for content and format and all are subjectto a broad USAID review process. The series is intended in part to indicate best practices, lessons learned,and guidelines for practitioner consideration. The series also includes publications that are intended tostimulate debate and discussion.

A list of other relevant publications and ordering information are included at the back of this document.

ABOUT THIS PUBLICATIONThis handbook was developed to enhance the ability of strategic objective teams—those USAID officers andtheir partners responsible for program management—to monitor progress in achieving planned results anduse performance information to guide program implementation.

Comments regarding this handbook and inquiries regarding USAID’s ongoing work in the area of DGprogram indicators should be directed to

Robin Silver, Strategies & Field SupportTel: (202) 712-0567Fax: (202) [email protected]

Jerry Hyman, Team Leader, Strategies & FieldSupportTel: (202) 712-1501Fax: (202) [email protected]

Center for Democracy and GovernanceBureau for Global Programs, Field Support, and ResearchU.S. Agency for International DevelopmentWashington, DC 20523-3100

ABOUT THE CENTERThe Center for Democracy and Governance is the U.S. Agency for International Development’s focalpoint for democracy and governance programming. The Center’s role is to provide USAID and otherdevelopment practitioners with the technical and intellectual expertise needed to support democraticdevelopment. It provides this expertise in the following areas:

CC Rule of LawCC Elections and Political ProcessesCC Civil SocietyCC Governance

For further information regarding the Center or the Technical Publication Series, please contact theCenter for Democracy and Governance Information Unit at (202) 661-5847.

The Center would appreciate your comments as to the appropriateness and utility of this handbook.Please contact the Center with any comments or suggestions.

OTHER DEMOCRACY AND GOVERNANCEPUBLICATIONS FROM USAID

PN-ACB-895Alternative Dispute Resolution Practitioners Guide

PN-ACC-887Civil-Military Relations: USAID’s Role

PN-ABS-534 Special Evaluation Constituencies for Reform: Strategic Approaches for Donor-Supported Civic AdvocacyPrograms

PN-ACD-395Democracy and Governance: A Conceptual Framework

PN-ACC-390Handbook of Democracy and Governance Indicators

PN-ACE-070A Handbook on Fighting Corruption

PN-ACE-500USAID Political Party Development Assistance

PN-AAX-280Weighing in on the Scales of Justice

TO ORDER FROM THE DEVELOPMENT EXPERIENCE CLEARINGHOUSE:

C Please reference the document title and document identification number (listed above thedocument titles on this page and on the back cover of this publication).

C USAID employees, USAID contractors overseas, and USAID sponsored organizationsoverseas may order documents at no charge.

C Universities, research centers, government offices, and other institutions located indeveloping countries may order up to five titles at no charge.

C All other institutions and individuals may purchase documents. Do not send payment.When applicable, reproduction and postage costs will be billed.

Fax orders to: (703) 351-4039 Attn: USAID Development Experience Clearinghouse (DEC)E-mail orders to: [email protected]

PREFACE

This handbook was developed to enhance the ability of strategic objective teams—those USAID officersand their partners responsible for program management—to monitor progress in achieving plannedresults and use performance information to guide program implementation. USAID’s Center forDemocracy and Governance undertook this effort, recognizing that defining objectives, establishingbenchmarks, and assessing progress in the democracy and governance arena(s) present unique challengesto the Agency. The Center believed that real gains in our understanding of performance measurementcould only come from a concerted effort, drawing together practitioners and experts of different kinds.The result is this handbook, which offers advice on the collaborative process of developing indicatorsand gives a broad selection of candidate (not mandated) indicators for monitoring each of the Agency’sdemocracy objectives.

The indicators included in this handbook were developed by four working groups, each responsible forone of the Agency’s four democracy objectives: 1) strengthened rule of law and respect for humanrights; 2) more genuine and competitive political processes; 3) increased development of politicallyactive civil society; and 4) more accountable and transparent government institutions. The groupsincluded USAID officers, NGO staff members, performance measurement experts, InterAmericanDevelopment Bank and World Bank staff, and others interested in improving monitoring systems fordemocracy programs. Starting with the democracy-governance portion of the Agency’s strategicframework, each working group developed or refined a hierarchy of objectives against which progresscan be measured. Then indicators were developed to capture progress in meeting each objective in theframework.

Once draft indicators were available from each group, key indicators were selected for field testing infour countries: Guatemala, the Philippines, Uganda, and Ukraine. The countries represent different pointson the democratic continuum and different circumstances regarding data availability and data collectionexpertise. Field test teams set out to assess the appropriateness and validity of selected indicators as wellas data availability, the cost of collecting data, and data quality. The teams also looked at issues involvedin setting targets and interpreting trendlines. While these four countries do not make a representativesample, the experience of testing the indicators in real-world situations gave the working groups valuableinformation about how institutional design issues affect the validity of given indicators, how trendlinesbehave, and how difficult it is to set targets. The information from the field tests was used to refine themenu of indicators.

ACKNOWLEDGMENTS

The Center would like to thank the many individuals who contributed to the handbook. They workedwith great diligence over the span of eighteen months. We believe the handbook benefits greatly from thevariety of skills, experience, and regional expertise they brought to the process. They are identifiedaccording to the team or working group to which they were assigned. In addition, the Center would liketo thank the USAID staff members in the “field test” countries and in the Bureau for Program and PolicyCoordination (PPC).

PARTICIPANTS

Core Team:

Erin Soto, USAID Center for Democracy andGovernance (G/DG), Team Leader

Steffi Meyer, G/DG, Team LeaderLynn Carter, Management Systems International

(MSI), Team LeaderRob Barr, G/DG, Democracy FellowMary Ann Scheirer, MSI MethodologistMitchell Seligson, MSI Methodologist Rule of Law Working Group:

USAIDDebra McFarland, G/DG, Sr. Advisor, Working

Group ChairMichael Miklaucic, G/DGLinn Hammergren, G/DG, Democracy FellowKeith Henderson, ENI BureauMichael Henning, G/DGCate Johnson, G/WID

Other Donors/NGOsMadeleine Crohn, National Center for State CourtsRoberto Lavier, World BankFay Armstrong, State DepartmentPam Swain, DOJ/ICITAPChas. Cadwell, U. of MD, IRIS

MSISheryl Stumbras, Working Group CoordinatorMary Said, ConsultantSteven Golub, Consultant

Elections and Political Processes Group:

USAIDAmy Young, G/DG, Sr. Advisor, Working Group

ChairKatherine Nichols, G/DG, Working Group ChairMark Feierstein, G/DG, Senior AdvisorRon Shaiko, G/DG, Democracy FellowCate Johnson, G/WIDNadereh Chahmirzadi, G/DGKatherine Stratos, ENI Bureau

Other Donors/NGOsKeith Klein, IFESPamela Reeves, IFES Working Group CoordinatorRudi Jeung, The Asia FoundationTom Melia, NDIPat Merloe, NDICathy Westley, NDILisa McLean, NDIMarissa Brown, NDIDave Merkel, IRIFrances Chappardi, IRI

Civil Society Working Group:

USAIDGary Hansen, G/DG, Sr. Advisor, Working Group

ChairChris Sabatini, G/DG, AAAS FellowHannah Baldwin, DG Advisor, G/WIDTodd Amani, G/DG, Senior Advisor

Other Donors/NGOsBob Asselin, Partners of the AmericasRudi Jeung, The Asia FoundationJorge F. Landívar, IDB

MSIJoan M. Goodin, Working Group CoordinatorMichael Bratton, Consultant

Governance Working Group:

USAIDPatrick Fn’Piere, G/DG, Sr. Advisor, Working

Group ChairTodd Amani, G/DG, Sr. Advisor, Working Group

ChairPat Isman, G/DGStephen Brager, G/DG, Democracy FellowPhyllis Dininio, G/DG, AAAS FellowAndrea Allen, G/WIDLois Godiksen, CDIEJohanna Mendelson, OTICynthia McCaffrey, G/DG

Other Donors/NGOsHelen des Fosses, SUNY/AlbanyJohn Johnson, SUNY/AlbanyRobert Nakamura, SUNY/AlbanyCamille Barnett, Research Triangle InstituteLou Goodman, American UniversitySusan Rose Ackerman, Transparency InternationalWendy Raymont, Transparency InternationalSusan Benda, NDI

MSIDavid Hirschmann, Working Group CoordinatorJoel Barkan, Consultant

Field Test Working Group:

Rob Barr, G/DG, Democracy FellowLinn Hammergren, G/DG, Democracy FellowLynn Carter, MSIMary Ann Scheirer, MSIJim Fremming, MSISheryl Stumbras, MSI

HANDBOOK OF DEMOCRACY AND GOVERNANCEPROGRAM INDICATORS

CONTENTS

PART 1 INDICATOR DEVELOPMENT

I. INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

II. STRATEGIC PLANNING AND PERFORMANCE MONITORING . . . . . . . . . . . . . . . . . . . . . 5A. Step 1: Strategic Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

1. Strategic Objective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52. Intermediate Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

B. Step 2: Developing and Selecting Indicators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61. Propose potential indicators for each level of results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62. Explore potential data sources for candidate indicators . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63. Refine the indicators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

C. Step 3: Using Indicators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91. Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92. Managing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103. Reporting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

PART 2 CANDIDATE INDICATORS

I. INTRODUCTION TO THE RESULTS FRAMEWORKS AND TABLES . . . . . . . . . . . . . . . . . 13

Section A: Rule of LawResults Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17Definitions—Rule of Law . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19Notes on Reading the Indicators Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24Indicators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

Section B: Elections and Political ProcessesResults Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57Definitions—Elections and Political Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59Notes on Reading the Indicators Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65Indicators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

Section C: Civil SocietyResults Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115Definitions—Civil Society . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117Notes on Reading the Indicators Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122Indicators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123

Section D: GovernanceResults Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151

Definitions—Governance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153Notes on Reading the Indicators Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159Indicators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160

PART 3 APPENDIXES

APPENDIX A: CRITERIA FOR DEVELOPING AND ADAPTING PERFORMANCE INDICATORS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227

A. Selecting Indicators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227B. Establishing Baseline Data and Targets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233C. Interpreting Data Points and Trendlines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234

APPENDIX B: ASSESSING DATA COLLECTION APPROACHES AND COSTS . . . . . . . . . 237A. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237B. Using “Secondary Source” Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239C. Using Judgmental or Narrative Sources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242D. Cost Factors for Sample Surveys . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245E. Rapid Appraisal and Other Qualitative Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249

APPENDIX C: USING SCALES, INDEXES, AND SCORECARDS FOR PERFORMANCEMEASUREMENT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255

A. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255B. Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256C. Strategies for Increasing the Reliability of Scoring for Scales, Indexes, and Scorecards . . . . 257D. Examples of Milestone Scales, Indexes, and Scorecards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258

PART 1 INDICATORDEVELOPMENT

I. Introduction 1

I. INTRODUCTION

The purpose of this handbook is to help strategicobjective teams—those responsible for programmanagement—develop indicators that are usefulfor management decisions. Although it isprimarily directed towards USAID democracyand governance (DG) officers and their partners,it may be of use to others who have similardevelopment programs. Performance monitoringis a necessary and integral part of good programmanagement. It is also related to the 1994Government Performance and Results Act (theResults Act), the intent of which is to improveFederal program effectiveness and publicaccountability. To this end, performanceinformation from each operating unit iscombined with broader trend information withinthe Agency’s goal areas to report results to theOffice of Management and Budget, Congressand the public in compliance with the ResultsAct. The same basis informs the Agency’sbudgeting process. Nevertheless, the primarypurpose of performance indicators is at theoperational level: to assist in makingprogrammatic decisions and learning from pastexperience.

This handbook by no means represents the lastword on good indicators for democracy andgovernance programs. The promotion ofdemocracy is a complex, dynamic process onlypartially understood. These factors need to berecognized and accommodated in thedevelopment of performance indicators andtargets. Because there remains much to belearned about how to capture changes indemocratization, the handbook should beconsidered a work-in-progress that reflectscurrent thinking about measurement under thepresent Results Act/AID monitoring system.

What the handbook contains

Part I of this handbook places indicators in thecontext of strategic planning and performance

monitoring. This context is presented as a seriesof steps: the first is developing a strategy, thesecond is developing indicators, and the last isusing the indicators. Although the primarypurpose of the handbook is to provide guidanceon the second of these steps, the others areincluded to provide the proper perspective.Sequence and purpose are important to keep inmind so that indicators do not driveprogramming or absorb too much time andresources.

This section includes some key definitions forthose unfamiliar with reengineering terminologyand briefly describes the criteria for goodindicators. The appendixes provide greaterdetail: Appendix A discusses the criteria forselecting indicators, gives tips for selectingtargets and interpreting trend lines; Appendix Baddresses different methodologies that may beused in collecting data; and Appendix C dealswith the use of scales and indices, which aretypes of qualitative measures to trackperformance. Another useful source ofinformation on indicator development is CDIE’s“Tips” series.

Part II contains candidate indicators. These areorganized by two levels of results—labeledintermediate results and sub-intermediateresults—under each Agency DG objective (Ruleof Law, Civil Society, Governance, andElections & Political Processes). There are avariety of results included, so objective teamsshould find some that closely match their ownintermediate results and possibly their strategicobjectives as well.

The indicators associated with these results,likewise, should be most useful for missions’intermediate results, but hopefully also forstrategic objectives. Agency objective levelindicators have not been included, since theseare not the responsibility of missions; nor haveactivity level indicators, since these are typicallyeasier to devise and are not required inperformance reporting to Washington. At this

2 I. Introduction

Strategic Objective indicatorsUsed by a mission to manage for resultsby tracking performance toward the mostambitious objective upon which it expectsto have a material effect. Required inperformance reporting, and used in theR4 process. Candidate indicators mightbe found in Part 2.e.g., S.O.: “Effective Justice SectorInstitutions”Indicator: Average time for casedisposition

Intermediate Result indicatorsUsed by a mission to manage for resultsby tracking performance toward lowerlevel objectives. Required in performancereporting, and may be used in the R4process. Candidate indicators can befound in Part 2.e.g., I.R.: “Increased Transparency inJustice Sector Institutions”Indicator: Percentage of court cases opento the public

Activity indicators Used by a mission to trackimplementation of a specific program’sactivities.e.g., Activity: Training of legal assistantsIndicator: Number of assistants trained

time, although the Freedom House Index is usedas a gauge of overall country democratization,there are no agreed-upon measures for each ofthe Agency’s four DG objectives. Agencyindicators can be divided into the followingcategories:

Caveats

1. The indicators in Part II are “candidate”indicators. The working groups tried to developindicators that would work in a variety ofsettings. However, the inclusion of a givenindicator in this handbook does not mean that itwill always be appropriate for a desiredobjective. An indicator is a good choice when itfits a specific objective, program and country

setting. Strategic objective teams may need toadapt many of these to make them applicable forthe local context. Nevertheless, the handbookcontains a variety of indicators which should be,at a minimum, suggestive of good indicators tomeasure progress toward objectives.

2. These indicators are not “common” indicatorsas originally intended in USAID’s “commonindicators exercise”—i.e., indicators in use byseveral missions (hence the term) which areworking in practice and which could be sharedwith other missions. While many are being usedby missions, the primary basis of developingthem was not current usage. Instead, theworking groups tried to determine what wouldbe good indicators for various programs. This isa learning process—we hope to find out whichones are good indicators, and possibly whichones could be adopted as “common.”

3. The following pages contain criteria for goodindicators, the same criteria which guided theselection of the indicators in Part II. However,because context is an important element indeveloping appropriate indicators, it was notalways possible to meet all of these criteria.Sometimes the indicators do not have asufficient level of specificity to be operationalbecause it was not possible to define them in away applicable to all country settings. This ismore often the case with the multicomponentindicators, such as indexes. Similarly, someindicators call for complex data collectiontechniques which may be too costly to befeasible, unless multiple indicators use the samemethodology.

4. Although there are multiple indicatorsprovided for each objective, this does not meanthat every one must be used if the mission has asimilar program. Instead, it is a menu of optionsfrom which objective teams can pick andchoose.

5. The framework used in Part II includes anintermediate result level and a sub-intermediateresult level to provide a consistent hierarchy of

I. Introduction 3

results. We are not suggesting that theseobjectives take the same position in a mission’sresults framework. For example, what we call anintermediate result may be perfectly suitable asa mission’s strategic objective.

6. Results frameworks are supposed to reflect acausal hypothesis, but this is not always the casewith the framework used here. A causalframework, that is, should illustrate the rationalebehind a program, so that reading down theresults “tree” answers the question of how youare going to accomplish something, and readingup the tree answers the question of why you areengaged in a particular program or seekingparticular intermediate results. Perhaps becausea democracy, even a stable one, is composed ofdynamic processes which are highly variable, itis a difficult concept to dissect and determinethe causes behind it. As a result, some sectionsof the framework are more definitional thancausal. In other words, lower level results maysimply reflect aspects of the higher result, ratherthan the sub-results necessary to reach thathigher result. For example, is improvedtimeliness of court processes a causecontributing to a fairer and more impartialjustice system or is it merely one aspect ordimension of a better justice system?

7. The complex and dynamic nature ofdemocratization also makes target settingchallenging. Often it is not clear how muchchange should be expected from a certain levelof activity, nor the rate of change. The specificadvice provided for each indicator sometimesreflects this level of knowledge, althoughAppendix A contains some general pointers.Keep in mind, however, that a good target is setby using our best informed judgment. If thetarget is not met, an analysis should beundertaken, from which we could conclude thatthe target was too ambitious and thereforerequires modification.

How to use the handbook

This handbook is designed to be a tool in

assisting mission personnel and their partnersdevelop useful and effective indicators formeasuring program performance. The text andthe candidate indicators should serve asguideposts—points to consider and examples touse and/or adapt when determining whichmeasures best facilitate program management. Itis important to keep in mind that this handbookis not the last word on meaningful indicators;that the indicators included are not necessarilythe best ones for given results; and that strategicobjective teams do not have to draw from it. DGofficers and their partners still need to workcollaboratively in developing appropriatemeasures. To reiterate, this handbook is a basisfor learning the best ways of monitoring DGprograms.

II. Strategic Planning and Performance Monitoring 5

II. STRATEGIC PLANNING AND PERFORMANCE MONITORING

The following explains the general process ofdeveloping a performance monitoring system,which should help place indicators in theappropriate context. The process begins with acountry analysis which helps identifydevelopment problems, as well as priorities andpossibilities for assistance (both USAID andother donor assistance). Because this is thesubject of the Center’s Strategic AssessmentFramework for Democracy and GovernanceProgramming (forthcoming), it is not coveredbelow. Following the analysis is thedevelopment of a strategy based on theassistance goals, which is presented here as Step1. This step involves the design of achievableand measurable program objectives (the termsobjective and result are used interchangeably).The next step is the development and selectionof appropriate performance measures for theobjectives, which is where the handbook shouldbe most helpful with its advice in theappendixes to Part 1 and the candidateindicators in Part 2. Step 3 addresses the use ofindicators, including the performancemonitoring plan and the R4.

A. Step 1: Strategic Planning

Strategic planning—the development of theassistance goal, program objectives, activities,and workplans—is the first step in developing aperformance monitoring plan, and goodobjectives are necessary for good performancemonitoring and measurement. A good objectiveis one with the following criteria:

CC impact-oriented, so that it represents aprogram objective rather than the outputof particular activities;

CC manageable, in that it can be materiallyaffected by USAID assistance;

CC time-limited, in that it is achievablewithin the time frame of the strategy;

CC uni-dimensional, so that it targets asingle development problem; and

CC specific, so that it cannot be interpretedin different ways.

Taken together, these characteristics mean thatthe objectives will be clear and specific,understandable, and measurable. The wording ofthe objective defines that which is to beachieved and, therefore, to be measured. If theobjectives are vague or unrealistic, accurate andmeaningful measurement will be difficult atbest. Even a seemingly simple objective such as“Increased Independence of the Justice System”can be vague. What does independence mean? Isindependence synonymous with impartiality, ordoes it merely imply a set of structuralconditions which might permit neutrality? Doesit mean independence from central and localgovernment intervention, or from private sectorcorruption and bribery? The indicators maydiffer depending on the meaning, so it isimportant that each objective has the abovecharacteristics to the extent possible.

The two levels of objectives in a strategic planare the strategic objectives and intermediateresults1:

1. Strategic Objective

A strategic objective is the most ambitious result(intended measurable change) in a particularprogram area that a USAID operational unit,along with its partners, can materially affectand for which it is willing to be heldresponsible. This is the highest goal that amission hopes to achieve within the 5 to 8 yeartime frame of the strategy. It is also the highest

1For details on these objectives, see the Agencyguidance found in the Automated Directive System(ADS), section 201.5.10.

6 II. Strategic Planning and Performance Monitoring

level for which missions must reportperformance, and PPC has identifiedperformance at this level as a focal point in theR4 review.

Sometimes SOs are written at levels equivalentto Agency objectives or Agency goals, like“Sustainable Democracy Built,” and/or theybundle together multiple objectives, like“Improved Responsiveness of DemocraticInstitutions with Greater Citizen Participation.”A high or bundled objective can makeperformance measurement problematic, becauseprecisely defining the specific objective—thething to be measured—is difficult. Such an SOmay also call into question manageable controland plausible attribution: for example, can anysingle donor really build, and is it willing to beheld accountable for building, a sustainabledemocracy?

Agency guidance permits bundled objectives,but only when the components a) areimplemented in an integrated manner (e.g., thetwo components are part of the same activity);b) are achievable by a common set ofintermediate results and causal linkages in theresults framework; and c) the component resultsare inseparable and mutually reinforcing (i.e.,achievement of one facilitates the achievementof the other) (see ADS 201.5.10a).

If these conditions are not met, it would bebest—from a performance monitoring point ofview— to separate the objective into itscomponent parts (while it is recognized thatguidance from different bureaus on this pointvaries, from the perspective of overall Agencyguidance, there are no restrictions on thenumber of SOs a mission can have). In the firstexample above, the SO could be “lowered” bybreaking it apart to reflect the components of asustainable democracy that the mission isworking on. Likewise, the SO in the secondexample above might be broken into two SOs:one regarding responsiveness and the other

regarding citizen participation. Although theremay be reasons to limit the number of SOs andkeep them at high levels, strategic objectiveteams may need to weigh the costs and benefitsof doing so.

2. Intermediate Results

An intermediate result is a key result which mustoccur in order to achieve a strategic objective.Like an SO, it reflects a reason a program wasundertaken. The difference between the twolevels is simply that one must achieve theintermediate results before one can achieve thehigher level strategic objective.

Although these results are at a lower level thanthe strategic objective and are essentially stepsleading to the strategic objective, they are notactivities. An IR is not, for example, a trainingsession that must be completed in order toincrease the effectiveness of an administrativestaff. In this example the IR would be theincrease in effectiveness; the activity would bethe training session. Or, if the strategic objectiveis “Increased Government Responsiveness toCitizens at the Local Level,” an IR might be“Increased Local Government Capacity to Act.”In other words, the relationship between the SOand the IRs should reflect the developmenthypothesis. The more clearly this is articulated,the easier the task of developing appropriateindicators becomes.

B. Step 2: Developing and SelectingIndicators

Once each objective has been clearly articulatedand defined, DG officers and their partnersshould ask: “how will we know if that result isoccurring?” Determining what information isnecessary to answer this question and how toprovide the necessary information is the processof developing performance measurementindicators.


Indicators: An indicator is one of a varietyof mechanisms that can answer thequestion of how much (or whether)progress is being made toward a certainobjective. It measures the performance of aspecific program by comparing actualresults with expected results. It does notanswer the question of why progress is oris not being made.

The process of selecting indicators, like definingobjectives, is iterative. The following steps willnot necessarily occur in sequence, but they doillustrate the process that should take place afterthe objectives have been defined, as discussed inStep 1 above.

1. Propose potential indicators for eachlevel of results

The group that discusses the logic underlyingthe program should begin by brainstormingpossible answers to the question above—howwould one know if the result is occurring. Part 2of the handbook can be used at this step byproviding initial, candidate answers and perhapsclues for more appropriate alternatives. Avariety of indicators should be considered tobest determine the most applicable choices foreach result. If there are no possible directmeasures, then consider proxies—measureswhich can indirectly inform managers ofperformance.

2. Explore potential data sources forcandidate indicators

For each of the applicable indicators, explorewhat data sources are available (or might beavailable if the indicators are conceptualized indifferent ways). Only indicators for which it isfeasible to collect data in a given country shouldbe used. Determining feasibility will requireconversations with people knowledgeable aboutvarious data sources (e.g., partners, governmentstatistical or service agencies, public opinionsurvey organizations, university social scienceresearch centers, etc.). These contacts will help

to understand what data are already beingcollected, whether existing data would beappropriate for a candidate indicator, whetherthe candidate indicators are relevant andfeasible for the situation, and possibly whatalternatives may work. Further, grantee andcontractor programs often include datacollection to monitor their activities, which mayprovide potential data sources for the result’sindicators. If there are no feasible or reliabledata sources available, then consider proxyindicators for which good data will be available.

3. Refine the indicators

The last step is putting in place indicator detailsto make them fully operational and conform tothe criteria below to the extent possible. Doesthe indicator specify the “operations” or actionsnecessary to provide the information? Is itcrafted so that gender or minority-specificinformation can be collected? Will it reflectincremental change over time?

When determining which indicators to use,consider the following criteria of good resultsindicators. Each indicator may not have all ofthese characteristics in practice, but to the extentpossible the following should be found in eachindicator:

C useful for program management, inthat only indicators which can helpstrategic objective teams makemanagement decisions should be used;

C appropriate for the result, in that themeasurement tool fits the task (just asone wouldn’t use a thermometer tomeasure weight, don’t use a populationstatistic to measure legislative quality);

CC direct, in that it measures the result asstated (don’t measure citizen awarenessif the result is legislative effectiveness),and at the correct level (don’t measurecapacity for effectiveness if the result iseffectiveness itself);


CC cost-effective, in that the costs ofmeasurement should be proportional tothe benefits (the mission should notspend more than 10 percent of its SObudget on performance monitoring);

CC based on reliable and valid data, sothat the measures are accurate andcomparable over time (decisions basedon incorrect data are unlikely to bebetter than decisions based on no data atall);

CC operational, so that data collectorsunderstand what data are needed andhow to collect them;

CC objective, in that everyone reviewingthe data would reach the sameconclusion about progress;

CC sensitive to the size of the problem(for example, including the numeratorand denominator is more telling than theformer by itself);

CC sensitive to change, so that the datareveal incremental change over time (ayes or no question, for example, doesnot show incremental change); and

CC disaggregated by gender or otherpopulation characteristics whereappropriate.

See Appendix A for further detail and examplesof these characteristics (also see CDIE’s “Tips”series).

Quantitative Indicators and QualitativeCharacteristics

Although Agency guidance notes a “preference”for quantitative indicators, their use is notrequired (see ADS 203.5.5); indicators thatmeasure qualitative aspects are perfectly

acceptable. Simple quantitative indicators, or“counts,” are noted by the ADS for theirobjectivity , meaning that the data would beinterpreted in the same way by different people.However, often these measures capture only athin slice of something larger and morecomplex. For example, the number of citizensattending town meetings is a straightforwardcount but it may not fully capture “IncreasedParticipation in Local Government.” In order tofully assess this objective, one might also needto know the type or character of the interactionbetween citizens and local government officials.This is not an isolated example. Many attributesof democratization, moreover, are considered tobe inherently complex and qualitative in nature.The working groups considered both types ofmeasures; where appropriate, they offeredsimple counts as candidate indicators, but inother cases they decided that no relevant simplequantitative indicators existed. The handbooktherefore includes many candidate indicatorsthat attempt to capture the qualitative nature ofDG programs.

Measures of qualitative aspects can take avariety of forms, and do not require a narrative.This handbook includes a number of measuresbased on judgmental assessments, such asquality scales, multi-component indexes andpublic surveys. For example, an indicator mightrely on the assessment of a panel of expertsregarding the quality of legislative processes,where one could report the combined score onan index of quality scales, deciding for eachscale whether to award one to five points andapplying clearly defined criteria to each scale.Or, where a single concept is being measured,the indicator can use a simple scale along aclearly defined range of scores from poor toexcellent, for example. The “comments” sectionin the performance data tables, or the spaceprovided in the qualitative measures form in theNMS, should be used to explain how the scoresare determined. While the working groups madeconsiderable use of such indicators, they


sometimes felt that the range of countrysituations precluded establishing firm criteria.Decisions about the criteria were in these casesleft to the user. Quality indexes can be valuableand useful tools, but they can also be quitecomplex. So that we collectively learn how tobest construct these types of measures, pleaseconsult PPC/CDIE/PME and PPC/DG, andadvise G/DG, when developing them. SeeAppendix C for greater detail on theseindicators.

The key requirement for any indicator—whetherbased on simple counts or informedjudgments—is that they be defined so as topermit regular, systematic and relativelyobjective interpretation regarding change in the“value” or status of the indicator. In order tomake judgmental indicators more objective, theyshould be very clearly defined and narrowlyspecified, such as listing which aspects toconsider when assessing legislative quality. Forthis reason, scales and indexes are moreappropriate measures of qualitative aspects thansimple descriptions in meeting the standard setby the Results Act. Nevertheless, there are noperfect indicators—just as simple quantitativeindicators can be misleading because they donot tell the whole story, indicators about qualitycan be complex and are more subjective. Andboth types are subject to the problemsassociated with poor data. The trick is to findthe best indicator or mix of indicators for thepurpose of program management.

C. Step 3: Using Indicators

As mentioned in the introduction, indicators areused for the following purposes: first (andforemost) is program management at theoperational level: managers need to knowwhether their programs are having the desiredeffects, and whether the activities need to beadjusted in order to achieve the result. If theindicators cannot provide this information, thenreconsider their use. A mission’s performancemonitoring plan keeps track of the data for thispurpose. The next use of the indicators is to

inform bureau and Agency budget decisions,although performance is only one of severalfactors used for these decisions. This data ispresented in the R4 submission, which describesthe progress of the mission’s program and laysout future needs. Finally, the same performanceinformation is used for the Agency’s reportingon results in compliance with Results Act andon what it has achieved with public resources.Washington uses performance data from the R4sto supplement and complement otherinformation for this purpose.

For strategic objective teams, using indicatorsinvolves monitoring performance in order tomanage for results, and then reporting onachievements.

1. Monitoring

Performance Monitoring Plan: In theperformance monitoring plans, missionsprovide detailed definitions of theperformance indicators that will be tracked;and specify the source, method, andschedule of data collection.

The purpose of the plan is to enable comparableperformance data to be collected over time, evenin the event of staff turnover, and to clearlyarticulate expectations in terms of schedule andresponsibility. It details, in other words, themeans for gathering the data required for theresults indicators. By following the plan, andcomparing the periodic data with the baselinedata, managers should have in place theinformation needed to monitor their programs inan on-going fashion and thus make informedmanagement decisions. It also provides theinformation used in the R4 submission.

2. Managing

Managing for results: Managing forresults is the use of performance data toinform management decisions regarding thebest use of resources to achieve the desiredobjectives, as opposed to fulfilling specific


activities. Strategic objective teams should use theinformation generated by the performancemonitoring plan to inform their decisions. Thisinformation should help managers make basicprogramming choices, such as which activitiesto continue, to stop or to modify. In addition, itshould contribute to the understanding ofdemocratic development by providing answersto questions like: Which programs work inwhich circumstances? What are reasonable ratesof change? Which assumptions hold and whichdo not? These answers will contribute toprogram development and success.

3. Reporting

Results Review and Resources Request:The R4 is the document which is reviewedinternally and submitted to USAID/W by theoperating unit on an annual basis.Judgment of progress is based on acombination of data and analysis and isused to inform budget decisions.

For the R4, performance is judged according totargets—whether programs met, exceeded or fellshort of their targets for the year—for thestrategic objectives and intermediate results.Although missions must report on both of theselevels, priority in the review is given to thestrategic objective level. Each indicator reportedmust identify baseline data and periodic targets.These targets should be annual, unless theprogram does not permit annual data collection(as might be the case for elections-relatedprograms). See Appendix A for some commonsense tips on setting targets. Also, the baselinedata should reflect, as near as possible, the valueof each performance indicator at the beginningof the program. For specifics on reporting, seePPC’s R4 Guidance Cable, State 010280,1/20/98, and the USAID General Notice onperformance measurement of 3/26/98.

When considering the suggestions provided inthis handbook, keep in mind that performancemeasurement should support, not detract from,program implementation. Prudence andreasonableness should guide how much andwhat information is collected and used fordecisions, while keeping in mind that bad dataare not better than no data at all. Ultimately,professional judgment is required to establishwhat results are possible, and what measuresand evaluations provide the best evidence ofperformance. So that we can determine whichare better and worse indicators, readers areencouraged to provide feedback about theirperformance monitoring efforts.

PART 2 CANDIDATEINDICATORS

Introduction to the “Results Frameworks” and Tables 13

I. INTRODUCTION TO THE RESULTS FRAMEWORKS AND TABLES

As a way of organizing the development andpresentation of indicators, the four workinggroups first elaborated results frameworks in theirrespective areas. A results framework is amanagement tool commonly used by USAID toarticulate the development hypotheses at work ina given strategy. It spells out inter-lockingprogram objectives. These objectives or resultsare organized in a hierarchical pattern thatattempts to specify cause-and-effect relationships.For example, if (A) the legal and regulatoryenvironment for civil society organizationsimproves: and (B) NGO financial andmanagement practices improve; and (c) NGOslearn how to solicit and represent constituentinterests to the government and the public-at-large; then civil society may have more directimpact on governmental policies.

Spelling out the causal or influential relationshipsat work in democratization efforts is currentlyvery difficult for two reasons. Often we do not yethave adequate empirical evidence to be clearabout causal chains. In addition, the complexity ofthe results at each level makes it problematic tobreak apart the various components. For thesereasons, the frameworks used to organize theindicators may sometimes appear definitionalrather than causal. For example, is Improving

Access to Justice a lower-level result which(along with other accomplishments) contributes toa Strengthened Rule of Law or is it one aspect ordimension of a strengthened rule of law?

The four results frameworks are presented in agraphic form in Sections A through D. Each onebegins with an Agency democracy andgovernance objective: 1) Strengthened Rule ofLaw and Respect for Human Rights; 2) MoreGenuine and Competitive Political Processes; 3)Increased Development of Politically Active CivilSociety; and 4) More Accountable andTransparent Government Institutions.

For each of these four objectives, the workinggroups developed two additional levels of results,intermediate results (IRs) and sub-intermediateresults (sub-IRs). As one example, the AgencyObjective of Increased Development of PoliticallyActive Civil Society has five Intermediate Results:

C A Legal Framework to Protect and PromoteCivil Society Ensured

C Increased Citizen Participation in the PolicyProcess and Oversight of Public Institutions

C Increased Institutional and FinancialViability of CSOs

C Enhanced Free Flow of InformationC Strengthened Democratic Political Culture

These five Intermediate Results were believed tobe the elements needed to constitute a strong andpolitical active civil society. Each of these IRsthen has a series of sub-IRs which contribute to

the achievement of the principal IR. If, forexample, we take the IR for Enhanced Free Flowof Information, it has four sub-IRs:

C Plural Array of Independent Sources ofInformation Encouraged

C Improved Investigative ReportingC Increased Use of New Information

TechnologiesC Improved Financial and Management

Systems in Media Entities

Working groups defined each of these IRs andthen developed indicators that would adequatelymeasure the two levels of results. Indicators werenot developed for the four Agency Objectivesthemselves. Currently, the Agency is reviewingthe possible use of the Freedom House Index tomeasure the four objectives.

Following the results framework for eachobjective area, readers will find definitions of theresults laid out in each framework. These willassist readers in understanding the frameworksand why they are constructed as they are. Thedefinitions are then followed by the relevantindicators tables. In searching for relevantindicators, the starting point for strategicobjective teams should be in trying to match theirresults or objectives with particular objectives inthe results framework. Then they can locate thoseobjectives on the indicator tables and review theassociated indicators.

The tables are organized by objective, indescending order. Results or objectives are givenat the top of the tables. In addition to presentingthe indicator and its definition, the tables alsoprovide recommendations on data collectionapproaches, tips on conditions or circumstancesthat might limit the utility of a given indicator,and advice on how trendlines tend to move for agiven indicator.

Section A: Rule of Law 15

Section A: Rule of Law


Results Framework


Definitions — Rule of Law

Agency Objective 2.1. Strengthened Rule of Law and Respect for Human Rights

Laws establish the terms of reference for the social contract under which citizens live together and aregoverned by a state authority. The rule of law prevails when the terms of the social contract are observedby both citizens and the state authority, when the terms are enforced either by voluntary cooperation orby legal processes and institutions, and when violations of the terms of the contract are punishedaccording to the law. The Rule of Law ensures that individuals are subject to, and treated equallyaccording to the law, and that no one is subject to arbitrary treatment by the state. A rule of law thatcontributes to the building of sustainable democracy is one that protects basic human rights (asenumerated in the Charter of the United Nations, Universal Declaration of Human Rights, InternationalCovenant on Civil and Political Rights, the Convention on the Elimination of Discrimination AgainstWomen, and the U.S. Constitution, among others). It is one in which market based economic activity isenabled, and freely operates. It is one in which the processes and institutions of justice are available to allindividuals without prejudice to their origins, religion, political persuasion, race, gender, or creed. Ademocratic Rule of Law is also one in which the processes and institutions of justice work efficiently andeffectively to establish justice and resolve disputes. To promote the rule of law, USAID pursues thefollowing approaches;

Intermediate Results 2.1.1. Foundation for Protection of Human Rights and Gender Equity is inConformity with International Commitments

Under this approach USAID seeks to ensure that the constitutions and laws in force in the host countryare in compliance with basic international human rights norms. By international human rights norms wemean specifically, the rights to life; freedom from torture or cruel, inhuman, or degrading punishment ortreatment; freedom from discrimination on the basis of race, color, language, social status, or sex;freedom from incarceration solely on the ground of inability to fulfill a contractual obligation; freedomfrom retroactive criminal laws; the right to recognition as a person before the law; the right to freedom ofthought, conscience, and religion; the right to participate in government; and the right to change ofgovernment. Protection implies defense of those rights against derogations on the part of the state as wellas non-governmental social actors. Violations of these rights must be criminalized and/or sanctioned,citizens must have the right to protest violations, and be provided with means and mechanisms for doingdo. USAID interventions might include assistance with drafting legislation or regulations protective ofhuman rights, helping governments to establish and develop official institutions for the protection ofhuman rights, or helping to build consensus and coalitions to advocate greater state adherence to bothlegal/constitutional and international human rights commitments.

The following sub-intermediate results represent key elements which need to be in place to bring aboutthe principal intermediate result:

(Sub) Intermediate Result 2.1.1.1. Legislation Promoting Human Rights Enacted

The first step a state must take toward establishing the foundations for protection of human rights isto enact legislation that codifies the state*s commitment to those standards. Enactment might bethrough ratification of appropriate international covenants or treaties, or through promulgation ofenabling legislation if those treaties and covenants are not considered self-executing. To meet thisintermediate result the legislation must provide ordinary citizens with a means of redress in law

20 Section A: Rule of Law

against derogations of their rights. This requires the enactment of whatever statutes, regulations, andrules of court and procedure are necessary to bring state authorities to account for human rightsabuses. The framework must also enable ordinary citizens to appeal to international procedures forwhich they might be eligible by international law.

(Sub) Intermediate Result 2.1.1.2. Effective Advocacy for Adherence to International HumanRights Commitments Increased

Effective advocacy is defined as organized, non-governmental citizens* groups operating for thepurpose of influencing state policy toward greater respect for human rights. Such groups must not bedependent on government funding. Influencing state policy might be through mobilizing popularinterest and action, or direct appeal to state authorities. Operations, whether including paid staff ornot, must be continual and on-going, and must result in demand that is visible to state authorities.

(Sub) Intermediate Result 2.1.1.3. Government Mechanisms Protecting Human RightsEstablished

The legal foundation for the protection of human rights must include provision for whatever specialmechanisms are necessary to enforce the framework. The human rights ombudsman is one suchmechanism. Specialized human rights courts, government human rights offices or human rightspolice units might also be established to fulfill this need. The official authorization of the operationsof such units, and their operating procedures, must be clear and unequivocal.

Intermediate Result 2.1.2. Laws, Regulations, and Policies Promote a Market-based Economy

A market-based economy is based on the principles of private property ownership, the free purchase andsale of goods and services at prices established through supply and demand, with the minimum regulationnecessary to protect the public interest, and minimum state intervention. Paramount to the sustainabilityof a market-based economy is the sanctity of contracts, and the existence of laws, regulations, andmechanisms (public or private) to resolve economic disputes in a fair and timely fashion. Equally criticalis respect for property, both publicly or privately owned, and the ability to transfer ownership of thatproperty according to internationally accepted business norms. A market-based economy that contributesto sustainable democracy also ensures the equal treatment of all economic actors under the law. USAIDactivities under this approach include assistance with drafting legislation and regulations that conformwith sound economic norms, as well as shaping policies to promote those norms. In addition USAIDassistance to advocacy groups supporting such laws, regulations, and policies is a component of thisapproach.


(Sub) Intermediate Result 2.1.2.1. Legislation, Regulations, and Policies Conforming withSound Commercial Practices Enacted

A legal framework supporting a market-based economy includes laws, statutes, and regulations thatconsistently protect the sanctity of property and contracts and the rights and obligations associatedwith them. The framework must include provisions for transparency of commercial transactionswhether conducted by the state or private entities, as well as equal access to and treatment for all


before the courts in the case of commercial disputes. The enactment of such laws, statutes, andregulations may be through administrative order, legislation or ratification, but must provideactionable means in domestic judicial fora for the resolution of disputes. The framework mustprovide private citizens with accessible means to challenge official state actions and be treated fairlyin such disputes.

(Sub) Intermediate Result 2.1.2.2. Effective Advocacy for the Promotion of a Market-basedEconomy Increased

Advocacy groups are non-governmental organizations which serve the commercial and economicinterests of citizens. Such groups must not depend on government funding and must articulate,represent, and advance the interests of citizens engaged in private commercial and economic activity.The strengthening of such groups entails either or both of the following: increasing their capacity tomobilize popular opinion and action; enhancing their capacity to petition state authorities forimprovement in the market environment.

(Sub) Intermediate Result 2.1.2.3. Government Mechanisms that Promote Market-basedEconomies Established

The legal framework for a market-based economy must include provision for whatever specializedagencies are necessary to promote the implementation and enforcement of the framework.Commercial courts or arbitration centers may fulfill this need. Given the specifics of the countrycontext, these might also include specialized business service units within state ministries or regionaland local authorities.

Intermediate Result 2.1.3. Equal Access to Justice

Access to justice refers to citizens’ ability to use various public and private services on demand. Equalaccess means that such ability (and the treatment accorded) is not restricted to certain classes or groupsof citizens. Since access will in some sense always be restricted (e.g., one must have standing to bring acase, court’s jurisdiction will be limited to certain kinds of cases, or individuals may be required to havea lawyer or submit to mediation before taking their case to court), the concern here is the basis for therestriction — and particularly that it not work to decrease the chances of already disadvantaged groups.Which groups are relevant is system dependent, but the usual concern is that access not be inequitablyavailable according to gender, ethnic, or racial group, political or religious category, social or economicclass, or physical incapacity. Where apparently equitable rules may in fact bar access to such groups,special measures may be introduced (e.g., subsidized legal services, interpreters for those who cannotcommunicate in the court language) in compensation.

The following sub-intermediate results represent key elements which need to be in place to bring aboutthe principal intermediate result.

(Sub) Intermediate Result 2.1.3.1. Increased Availability of Legal Services

To increase access it is necessary to increase the supply of services and to target that increase togroups for whom the unmet demand is greatest. This is most often because of their own resourcerestrictions, but it may also be because of special needs even for resource rich groups (e.g.,entrepreneurs who require timely and more specialized handling of a far greater number of cases).


(Sub) Intermediate Result 2.1.3.2. Increased Availability of Information

Another necessary result to increase access is to provide more information, for normal users and forthose normally excluded. Relevant information may be on system workings or relevant to a case inwhich the actor has an interest.

(Sub) Intermediate Result 2.1.3.3. Decreased Barriers

Finally access will also be limited if there are barriers, including formal and informal institutionaland legal biases. Legal requirements may impose exceptional hardships on some groups (e.g., arequirement that a woman get permission from a male relative before appearing in court, or unusuallylarge fees), services may be located so that some populations must travel long distances to tap them,or institutional members may work with certain conscious or unconscious prejudices in carrying outtheir duties.

Intermediate Result 2.1.4. Effective and Fair Legal Sector Institutions

The legal system includes (but is not necessarily limited to) such institutions as trial and appellate courts,the judiciary, other court officials, administrative courts, the prosecution, public defense, police and otherenforcement agencies, prisons, as well as non-governmental entities such as bar associations, advocacyorganizations, legal aid service providers, law schools, and other private organizations. The bindingcharacteristic is their involvement in legal dispute resolution. Effectiveness is the degree to which thelegal system (and the justice sector institutions in particular) resolve disputes in a timely, predictable, andreliable manner; fairness is the degree to which the justice sector institutions uphold the principles of thelaw in a manner that serves the public interest while treating all individuals according to the samestandards. Fairness would also prohibit any arbitrary favors to any individual or group, unless prescribedby law.


(Sub) Intermediate Result 2.1.4.1. Increased Transparency

Because it is a public service, and ultimately responsible to the people, justice (the authoritativemeans for resolving conflicts and imposing legally based social control) will only work if itsoperations and operating principles are widely known. Otherwise there is too much room forpartisan, skewed, or arbitrary decisions.

(Sub) Intermediate Result 2.1.4.2. Increased Independence

Independence is relative and is both individual and institutional. What it means in both cases is thatinstitutions and their members must be able to do their jobs according to the rules and principlesestablished for them. Their actions and their tenure should not be dependent on the extraneouspreferences of higher ranking officials or those outside their institutions. While there is always atension between individual and institutional independence, ideally the institution should beresponsible for and capable of enforcing individual compliance with legitimate institutional norms.

(Sub) Intermediate Result 2.1.4.3. Improved Management and Administrative Capacity


Since most justice system actors operate within organizations, their performance hinges on the abilityof that organization to monitor and facilitate their activities. An organization which does not knowwhat its parts are doing or cannot provide them with essential resources is not doing this.Management and administrative systems are critical to that ability and become more so asorganizations grow in size and become more complex.

(Sub) Intermediate Result 2.1.4.4. Improved Functional Organization

Growth and complexity also require changes in organizational forms and operating rules for purelyfunctional tasks. Indicators focus both on organizational characteristics assumed to aid performanceand on outcomes indicating that the latter is adequate.

(Sub) Intermediate Result 2.1.4.5. Professionalization of Technical Personnel

Modern organizations are increasingly staffed by professionals, individuals with specialized, job-relevant training and skills. Personnel systems should be based on recognizing and rewardingprofessionalism.


Notes on Reading the Indicators Tables

The following categories of information are provided in the tables:

1. The statement of the indicator.

2. A definition of the indicator and its unit of measure.

3. The relevance of the indicator or why it was selected as a measure of that particular result.

4. Approaches to collecting data for the indicator and the approximate cost of collection. Threecategories were used for cost: low (under $500); medium/moderate ($500 - $1500); and high (over$1500). The cost specifications are rough and should be treated cautiously, because costs may varygreatly from one country to the next. The cost of data collection for questions requiring surveys ofthe population (for example) will depend upon 1) the existence of periodic surveys which address thedesired topics conducted by someone else; 2) the existence of surveys which do not address thedesired topic but to which strategic objective teams could add questions; and 3) the need to conductone’s own surveys and the number of indicators for which that survey can serve as the data collectionapproach. Similarly, the cost for monitoring broadcasts or periodicals for specific content willdepend on whether there is 1) a commercial monitoring service, which conducts monitoring foradvertisers anyway, so fees can be very modest; 2) an NGO that uses this information for its ownpurposes anyway; or 3) a need to cover the entire cost of monitoring several periodicals and/orbroadcast stations.

5. Issues related to target setting and the interpretation of trendlines. In this final column, we share whatwe have learned about how much progress might be made over what period of time. In some cases,such as changes in the level of political tolerance, we know very little. In other cases, it seems clearthat an indicator should change sharply and immediately in the wake of particular interventions. Wealso discuss issues to be aware of in setting performance targets or in interpreting trendlines.

Not all categories will be discussed for every indicator. In some cases, the relevance of the indicator isclear and does not require any discussion. In other cases, we have little at this point to contribute to ideasabout how to set targets.


Indicators

Agency Objective 2.1: Strengthened Rule of Law and Respect for Human Rights

Intermediate Result 2.1.1: Foundations for Protection of Human Rights and Gender Equity Conform to International Commitments

Indicator Definition and Unit of Measurement Relevance of Indicator Data Collection Methods/Approx. Costs Target Setting/Trendline Interpret. Issues1) Number ofhuman rightsviolations filedagainstgovernmentsecurity forces per100,000.

Number of human rights abuses filedagainst security forces divided by totalpopulation/100,000.Filed means in a domestic court, a regionalhuman rights tribunal, like the EuropeanCourt of Human Rights or the Inter-America Court appropriate in the country.*

Direct measure of citizenconfidence in the foundations & thewillingness & ability of thegovernment to address violations.

National Stats; International Organizations; StateDept. Reports.Cost: Low if there are case tracking or secondarysources. But possibly cost prohibitive if theinformation has to be collected.

It would depend on the specific problem asarticulated by the mission. However, thisindicator is most likely to be meaningful wheneffective means/mechanisms for filingcomplaints have recently been put in place. Inthis case, the number of violations filed shouldincrease over the life of the strategy. It wassuggested in Guatemala that this might be agood indicator only for a limited period of time.Once a large number of cases are filed, it wouldbe impossible to track them (unless they allbegin with the ombudsman).

*Comments: The indicator seeks to measure the violations that are getting beyond the ombudsman’s office, thus that office would not be included in the definition of “filed.” The only exception might bein a country where the ombudsman’s office is not only vested with the authority to investigate but also to adjudicate human rights violations.2) % of detaineesheld in pre-trialdetention.

Pre-trial detention includes the time frominitial arrest to the time of trial or release,whichever comes first.

Direct measure of whether thefoundations are providingprotections.

Court records; national stats; if there are no reliablestats, then we suggest a sample survey.Organizations, such as Amnesty International andlocal human resource organizations may collect thisdata.Cost: Low if records are available.

% should decrease with time.

Comments: An initial question will need to be asked if there is a system for recording detainees. If not, this would not be an appropriate indicator. In many Latin American countries, it is more practical tolook at pretrial detainees as a % of prisoners. Most countries in that region do appear to keep such statistics. In Uganda, however, we found that there were not reliable comprehensive statistics on thisindicator. Nevertheless, there were reliable statistics on the % of prisoners on remand for a period exceeding that provided for by law. This indicator will not capture detainees until they have theirfirst court appearance. It will also not capture people who are detained outside of formal detention centers, for example by military/security forces. However, it was the most reliable indicator we couldidentify. If “informal” detention is a concern then you may want to couple this indicator with human rights reports that report on the number of informal detainees. If you are in a country that has verygood statistics, you may also want to accompany this indicator with an indicator measuring the average length of pretrial detention.3) Average timetaken to processhuman rightscomplaints.

From the time a complaint is filed (with thepolice or courts, whichever is first) to theinitial hearing before a judge.Average of a sample of complaints.

If cases are not processed within areasonable time, then HR are notadequately protected.

Police and court records.Cost: Low if the records are good and areaggregated.

A downward trend would be positive. However,one may also want to look at the results,especially in the beginning and if there are fewcases.

Comments: In Guatemala, interviewees suggested this might be difficult to track because very few cases are actually filed and no existing statistical system captures them from beginning to end, or evenidentifies them adequately.

Indicator Definition and Unit of Measurement Relevance of Indicator Data Collection Methods/Approx. Costs Target Setting/Trendline Interpret. Issues


4) % of peoplewho believe thatthey could file ahuman rightscomplaint withoutfear of reprisal.

% of population (males/females, by ethnicgroup, etc.).

Indicates whether people are willingto come forth and use the systemsthat are in place to protect theirrights.

Survey.Cost: High unless a survey is being done anyway.However, if Gallup or some other agency is alreadydoing any kind of survey, it may be fairly costeffective to include this as an additional question.

This may be the best proxy to use since it wouldbe difficult to find a measure that captures thenumber of people that had their human rightsviolated but did not file a complaint.

Comments: Alternative suggestion, % of people who believe that they will obtain a fair hearing or % of people that believe that the government has a commitment to pursuing human rights cases. In theGuatemala field test it was suggested that tracking responses among different groups would also be informative.



Intermediate Result 2.1.1: Foundations for Protection of Human Rights and Gender Equity Conform to International CommitmentsIntermediate Result 2.1.1.1: Legislation Promoting Human Rights Enacted

Indicator Definition and Unit of Measurement Relevance of Indicator Data Collection Methods/Approx. Costs Target Setting/Trendline Interpret. Issues1) Ratification/accession tomajor internationalhuman rightsinstruments.

These include the International Covenantson Civil and Political Rights and Economic,Social, and Cultural Rights, as well as theOptional Protocol to the former, theCEDAW, and appropriate regionalinstruments (i.e., European Convention,Inter-American Convention, AfricanConvention, etc.).Unit could be yes/no by instrument.

Demonstrates legal commitment toprinciples of human rightsarticulated in the internationalinstruments.

U.N. and regional organization documents.Each country could take a set of 6-10 internationaland regional instruments as a sample.

Cost: Low.

Comments: Note, this is only a threshold indicator and is not meant to reveal information about the real human rights situation. In fact, it would probably only be used in cases where the USAID interventionswere specifically aimed at ratification/accession.2) Internationalhuman rightsinstrumentsbecome nationallaw.

This indicator requires two elements: A.International instruments must be eitherself-executing within domestic law, orimplementing legislation providing forenforcement must be passed. -AND-B. Anyadditional enabling legislation necessary tooperationalize these instruments isenacted.Unit: Yes/No.

Without such provisions terms ofinternational/regional instrumentscannot be recognized in domesticcourts.

Legal review.

Cost: Low.

Note that this indicator would only be used withthe above indicator, or in a country where theabove indicator has already been achieved.

Comments:



3) Human Rightsenumerated andexplicitly providedfor in Constitutionor Basic Law.

Rights might include: the rights to life;freedom from torture or cruel, inhuman, ordegrading punishment or treatment;freedom from slavery; freedom fromdiscrimination on the basis of race, color,language, social status, or sex; freedomfrom incarceration solely on the ground ofinability to fulfill a contractual obligation;freedom from retroactive criminal laws; theright to recognition as a person before thelaw; the right to freedom of thought,conscience, and religion; the right toparticipate in government; and the right tochange government. The unit of measurewould be a checklist.

While the constitutional enumerationof rights does not guarantee theirimplementation, it neverthelesssignifies the terms of reference bywhich an individual is to take thestate to task for derogations fromthose rights.

Legal review.

Cost: Medium.

Comments: As with the two previous indicators, this is not very meaningful in Latin America.



Intermediate Result 2.1.1: Foundations for Protection of Human Rights and Gender Equity Conform to International CommitmentsIntermediate Result 2.1.1.2: Effective Advocacy for Adherence to International Human Rights Commitments Increased


1) % of NGOsrated as effectiveon the advocacyindex or showingimprovements onthat index.

See Appendix C, part D. “Effective” can bedefined as reaching a particular number ofpoints on the scale.

Direct measure of whether theNGOs are becoming more effectiveadvocates.

Cost: Medium but this depends on the # of NGOs. NGOs should rate higher with the passage oftime. Note, exercise caution when adding newNGOs to the target group. Depending on wherethe NGOs begin on the scale, your targets willbe effected.

Comments: The index can be adapted to fit local condition or particular program emphasis.

2) Number ofNGOs that identifyhuman rightspromotion withintheir mandate.

Number of active non-governmentalorganizations that specifically identify thepromotion of human rights as part of theircharter or mandate

This is a proxy indicator that gets ateffective advocacy via moreadvocacy. It should only be used if amore direct indicator is unavailable.

Interviews with NGOs.

Cost: Low.

This should increase with time. It may be morerelevant in country where NGOs are justbeginning to have a voice and are too nascentto see a measurable change in their ability toadvocate—which would still be the long termobjective. Few very strong NGOs may havemore impact than several weak ones.

Comments: Might be difficult to track unless there is a governmental registry or as in Guatemala, a donor is doing an inventory of NGOs.

3) % ofInternational andlocal human rightsgroups thatproduce anddistributeuncensoredreports.

Groups are unrestricted with regard tofreedom of speech and the press, and todistribute their materials withoutinterference or coercion

Again, this is a proxy indicator thatmeasures the environment foradvocacy more than whether theadvocacy is effective.

Survey of Groups.

Cost: Low - High, depends on # of groups involved.

This should increase with time. Since this is aproxy indicator, we would not recommend usingit if you were able to use the advocacy scale,mentioned in number 1 above, which is a moredirect measure of the result.

Comments: Alternative: # or % of human rights groups who say they can publish and distribute their findings and analysis openly and without self- or government imposed censorship. There was somequestion in Guatemala as to whether this described a real problem, i.e., are there human rights organizations which produce censored reports anywhere? A better question might have to do with instancesof repression of NGOs for the reports they published or a denial of access to information.



Intermediate Result 2.1.1: Foundations for Protection of Human Rights and Gender Equity Conform to International CommitmentsIntermediate Result 2.1.1.2: Government Mechanisms Protecting Human Rights Established


1) An independenthuman rightscommission,human rightscourt, orombudsman isoperating.

To be considered established thecommission must:1. Have the legal authority to investigateviolations.2. The commission must be funded by thegovernment (perhaps the commissionreceived funding from the governmentmore or less in accordance with its budgetrequest).3. The commission is actively investigatingcases.Unit: Yes/No.

The important issue here is theacceptance by the government ofthe private right to file complaints indomestic institutions againstgovernmental abuses.

Informal survey; document review.

Cost: Low.

Establishment is a one time achievement.

Comments: If this is the focus of the USAID program, then the proposed indicator under APA 2.1.3 (effective and fair legal sector institutions) is likely to focus on human rights institutions.

2) An entity andprocedure forhearing humanrights cases is inplace.

This requires jurisdiction to hear humanrights cases as well as power to adjudicateand impose sanctions.

Unit: Yes/No.

A legislative directive to the judiciaryconferring jurisdiction on humanrights matters is an explicit politicalstatement of intent to treat humanrights issues as legitimate issues ofcontention between individuals andthe state.

Informal interview, review of legislative directives.

Cost: Low.

Comments:



Intermediate Result 2.1.2: Laws, Regulations, and Policies Promote a Market-based Economy


1) % change inthe number ofcommercial casesfiled in the courtsystem.

Commercial needs to be defined in thecontext of the mission program.

People are more likely to bringcomplaints to the court system ifthere are good laws and regulationsconsistently applied.

Court records.

Cost: Low, if records exist.

Should increase over time. There may be timelag issue. Change in this indicator may takesome time to follow system improvements. Theprovision of alternatives, such as arbitration,may affect the trendline.

Note when determining whether to use thisindicator, you should consider whether there arefactors such as broad-based corruption that willprevent people from using the system. Thisindicator may be most relevant in the initialstages of putting a legal framework in place.

Comments:

2) # of businessesregistered insample year.

Registration means official registration witha ministry or board of trade, commerce, ormembership in a chamber of commerce.

Shows increased participation/confidence in the formal economicand legal sector. Easier registration,fewer disincentives or transactioncosts for joining the formal sector(i.e., taxes, restrictions), betterimplementation of the law.

Review of business registration records.

Cost: Costs are low if there is existing data/records.

Within the life of a CSP, an upward trend wouldbe positive.

Comments: Some caution should be exercised in using this indicator. An increase in the number of businesses registered could also reflect economic growth or some other factor not associatedwith trust in the legal system.

3) Change in thepercentage ofmonetary assetssecured bycontracts.(Country IntensiveMoney).

Definition: the ratio of non-currency moneyto the total money supply or M2-C/M2 -where M is a broad definition of the moneysupply and C is currency held outsidebanks.

Shows that people are more likely toenter into contract, reflecting theirconfidence in the protection andenforceability of contract rights. Thisis a very high level indicator,capturing the results of broadsystemic improvement in the justicesystem but also in some other areassuch as bank supervision.

Contract Intensive Money (CIM) rate; Central Bank;IMF.

Cost: Low.There are data on both the quantities of currencyand the quantity of M subscript 2 for almost allcountries. However, the costs should be high forascertaining assets secured by contracts.

Rising trend indicates growing confidence incontract rights and the willingness and ability ofthe government to enforce contracts. May be atime lag problem, with CIM changing only afterpeople recognize broad systemic improvement.

Comments: Allows international comparison, observation of changes over time as data is available every year.



Intermediate Result 2.1.2: Foundations to Promote and Support a Market-based EconomyIntermediate Result 2.1.2.1: Legislation, Regulations, and Policies in Conformity with Sound Commercial Practices Enacted


1) Legislation/Regulations areconsistent withWTO standards

Countries that accede to World TradeOrganization treaty.Unit: Yes/No

Direct measure of whether a countryhas a set of laws that arerecognized as conforming tointernational standards ofcommercial soundness.

WTO records.

Cost: Low-Medium

Obviously this indicator is a one-time indicator.Once a country accedes to WTO treaty, thisindicator will no longer be relevant.

Comments: Alternate: % of WTO steps or criteria met. WTO membership implies a complex set of market-supportive arrangements as determined by the international community.


Agency Objective 2.1: Strengthened Rule of Law and Respect for Human RightsIntermediate Result 2.1.2: Foundations to Promote and Support a Market-based EconomyIntermediate Result 2.1.2.2: Effective Advocacy for the Promotion of a Market-based Economy Increased

Indicator Definition and Unit of Measurement Relevance of Indicator Data Collection Methods/Approx. Costs Target Setting/Trendline Interpret. Issues1) # or % ofadvocacy NGOsrated as effectiveor showing animprovement onthe advocacyindex.

See Appendix C, part D for the advocacyindex. “Effective” can be defined asmeeting a set number of points on theindex.

Direct indicator of their strength inadvocacy

Survey/interview.

Cost: Medium, but this depends on the number ofNGOs being rated.

Increase over time in relation to TA and trainingin elements included in the index.

Comments: The index can be adapted to fit local conditions or program emphasis.2) Number ofNGOs that identifybusinessadvocacypromotion withintheir mandate.

There will need to be some standarddefining what constitutes “identifying” onthe part of the NGOs

Unit: Number

Relevance is based on the assump-tion that proliferation of businessadvocacy groups indicates astrengthening of such groups withinthe sector. This is a less directmeasure then the advocacy scale.

Review of NGO materials (equivalent to corporatecapabilities).

Cost: Low, depends on the number of NGOs intarget group.

Comments: More, weak groups may not be better than few strong groups but would like to see diverse interests represented.3) Number ofcommercial lawssubmitted to leg-islature that weredrafted or review-ed by advocacygroups.*

This would require identifying target lawsthat NGOs were lobbying for out of sometotal that came before the legislature.

Individual missions will need to define“submitted” and “reviewed”Unit: Number.

Measures actual impact of theadvocacy.

Survey of NGOs; Committee Hearing Minutes.

Cost: Low.

Should increase.

Comments: Alternative: % of all commercial laws submitted that were drafted or reviewed. *Perhaps not just advocacy groups, but by interested economic parties. Commercial advocacy groups may notrepresent most important economic interests and the views of the latter may be made known by other means in Latin America.4) % of targetNGOs whobelieve theiradvocacy haslead to policy andlegal changes andcan providespecific examplesof change.

Unit: Percent. Measures actual impact of theadvocacy.Higher level of measuring theiradvocacy abilities than #3, since itmeasures whether the advocacyhad impact proxy to 1) above.

Survey of NGOs.

Cost: Medium, depends on # of NGOs.

Should increase.

This would require identifying target laws thatNGOs were lobbying for.

Comments:Agency Objective 2.1: Strengthened Rule of Law and Respect for Human Rights


Intermediate Result 2.1.2: Foundations to Promote and Support a Market-Based EconomyIntermediate Result 2.1.2.3: Government Mechanisms Established Which Promote Competitive Open Markets

Indicator Definition and Unit ofMeasurement

Relevance of Indicator Data CollectionMethods/Approx. Costs

Target Setting/TrendlineInterpret. Issues

1) State anti-monopoly regulatoryunit is established.

Such a unit would be an officialstate mechanism for promotingcompetition and regulatingmonopoly practices.Unit: Yes/No.

Market-based economies dependon an environment conducive tofair competition amongenterprises.

Informal interviews, documentingevidence.

Cost: Low.

Creation is first step. May needthem to look separately at howwell it functions.

Comments:

2) Specialized commercial courtsare established.

Unit: Yes/No. Court specialization is a measureof increased ability to adequatelydeal with the subject matter. Thisis more relevant in country with nohistory with commercial/ businesslitigation.

Review of the courts.

Cost: Low.

Comments:



Intermediate Result 2.1.3: Equal Access to Justice


1) # of new courtsopened in ruraland urban areaswithconcentrations ofmarginalizedpopulations.

Areas or districts with concentrationsshould be those where 60% (or TBD %) ofthe population is poor, ethnicallymarginalized, etc.

Measure per annum.

Local availability of courts, and thegeographic distribution of courts isan indicator of access.

Court records and census data.

Cost: Should be fairly low once areas aredetermined.

This would require an assessment of how manycourts would be optimal. Obviously, more is notalways better.

Comments: Alternative measures might include numbers of courts available for use, against a plan; and average daily utilization per court, in hours, number of court staff in posts. These indicatorsshould be disaggregated by court type. If this indicator is very costly, reporting on it may be biannual, or at some other longer interval.

2) # of courts,police posts per100,000population.

Disaggregate by court and police post. Depending on the programemphasis, police posts might bedeleted.

Court statistics.

Cost: Low.

Should increase up to a point. Depends also onalternatives, level of demand.

Comments:

3) % of citizenswho say that theyhave access tocourt systems toresolve disputes.

% of those surveyed. Might do series ofquestions, one on distance, one on costs,one on other barriers (knowledge).

While this is not the same as havingaccess, it is more than belief —question can be phrased lessabstractly to increase validity

Survey.

Cost: High, unless survey was already beingconducted.

The survey questions would have to be wordedin such a way that they only measured accessand not fair justice at this level.

Comments:

4) % of accusednot represented attrial.

Unit: Percent. If more individuals are representedat trial, arguably they are moreaware of their rights torepresentation, the services areavailable, and there are noadministrative or other barriers togetting the representation.

Court Records, aggregate stats if they exist;otherwise sample survey of criminal cases.

Cost: Medium, but depends on how difficult it mightbe to compose a sample frame.

Should decline over time.

Comments: Some care needs to be used in determining whether to use this indicator. In Uganda, for example, the criminally accused have a right to representation. However, given the lack of publicdefenders, the result is that accused end up on remand for a longer period of time awaiting representation. Thus, the % represented may always be 100%. However, the time on remand awaitingrepresentation would be more telling. Guatemala suggests a similar problem, and notes that it may be more critical to provide a defender when the person is arrested.Case loads of public defenders may be so heavy that the representation is very poor. Thus, it might also make sense to look at average case load per defender in addition to this indicator.



Intermediate Result 2.1.3: Equal Access to JusticeIntermediate Result 2.1.3.1 Increased Availability of Effective Legal Services


1) Number ofpublic defenders,and defendersprovided throughlegal aid, and lawclinics defendersper 100,000population.

Definition:Public Defenders: attorneys the state paysto defend the poor;Legal Aid: NGOs or other private orprivate/state mixed agencies existing toprovide legal representation to the poor.Law Clinics: Groups established by the barand law schools where students, underclose supervision, give legal assistance tothe poor.Unit: No. of defenders per 100,000 pop.

It is difficult to get into court withoutsome type of legal guidance and/orrepresentation.

Statistics and/or surveys; personnel records, MOJ,NGOs, etc.

Cost: Most governments keep such figures. ManyUSAID programs are installing the informationsystems that allow it to happen. Therefore costsshould be low.

The number should increase over time to meetgrowing population and growing need forconflict resolution. If numbers decrease, there is a problem. Mostlikely funding is difficult to come by. (May wantto separate by rural and urban.)There is also a problem (Guatemala) of how tocount pro bono or part time defenders. In somecountries, every member of the bar istheoretically available to provide pro bonoservices.

Comments:

2) % change inthe cases handledby publicdefender, legalaid, or law clinics

Definition:Public Defenders: attorneys the state paysto defend the poor;Legal Aid: NGOs or other private orprivate/state mixed agencies existing toprovide legal representation to the poor.Law Clinics: Groups established by the barand law schools where students, underclose supervision, give legal assistance tothe poor.

Unit: Percent change from a baseline year.

While percent increase does notindicate quality of service, it at leastshows services are available (theiruse may also indicate perceptions ofquality — if they were ineffectual,who would use them?).

Court or service statistics

Varies by size of country, shape of statistics. IfUSAID is funding only services it will have numbers,but if not this could end up being quite costly. Courtstatistics may be more easily obtained, although theywould only show cases represented in court, notthose where only advice is given or resolutions arereached out of court.

Generally, number would be expected toincrease for most of countries where USAID isworking. However, at some point subsidizedlegal service does become anti-economic. Itcould also encourage conflict.

Need to compare increasing number with caseloads per defender. Case loads could simply begrowing.There is a problem with part timers here as well,although it may be less critical in this indicator(numbers should go up, just more slowly).

Comments: Alternative: # of cases handled.

3) Number ofcases usingalternativesystems.

Case is defined as a conflict between twoor more parties.An alternative system is one that is legallyrecognized.

Unit: Number of cases handled byalternative systems.

Indicates accessibility if not quality. Statistics for legallyrecognized systemsmay be available;however, for tribalcouncils, etc, could bedifficult.

Varies with source, could berather high if there are alarge number of alternativesand most of them are prettyinformal.

If under utilized there is a problem. Guatemalaand other countries which recognize customarylaw add some difficulties — traditional “courts”probably don’t keep records, may not evenconsider “cases” in the same way. Here onemay have to work with new ADR systems, courtannexed, stand alone, and linked to traditionalcommunities and use a sampling technique.



Comments: Alternate indicators include: % or number of ADR entities showing an increase in case loads of X% in the past year with X to be defined by program managers. In countries where nationalstatistics are not very good you may want to consider whether partner NGOs are collecting information such as: % of villages where mediation services are provided.

4) Percent ofplaintiffsappearing pro sewho report thatcourt documentsand procedureswere madeavailable to themupon request.

One appears “pro se” in a legal actionwhen one represents oneself without theaid of counsel.

Unit: Percent

No country can provide 100%assistance to everyone, and part ofthe ideal may be reducing the needfor lawyer services —making someactions simple enough so thatcitizens can handle their own legalwork. To be effective, the plaintiffswould have to be able to access theinformation essential for them toeffectively represent themselves

Sample survey of prose plaintiffs.

Med to High—you wouldneed to put together a list ofpro se plaintiffs to draw asample from a group ofselected courts.

Should increase. Note, this in only a relevantindicator in country where pro se representationis allowed.

Comments:

5) ArbitrationCenters forcommercialdispute resolutionare established.

Unit: Yes/No This may be redundant, however wewant to make it clear that we areproviding services for economic aswell as criminal cases.

Internal interviews;document reviews.

Comments: Note, this is a cross over with APA 2.1.2.3. Y/N is really not adequate. After the center is established, then the indicators should change. One potential may be the number of caseshandled. In many Latin American countries, centers are started and then do nothing for some time.



Intermediate Result 2.1.3: Equal Access to JusticeIntermediate Result 2.1.3.2: Increased Availability of Information


1) % of caseswhere notice wassent and received.

Notice = notification to parties of a pendingcase.

Notification should be in compliance withthe country’s laws.

Unit: % of total cases.

There is no access if you don’t knowthere is a law suit/action.

Court records & sample survey.

Cost: Probably moderate to high since where this isa problem, records are unlikely to exist.

If notice is not routinely sent, may be a matter oftraining and enforcement. Also logisticalproblems especially in countries with ruralpopulations, large urban slums, lack ofcommunication, transportation infrastructure. InColombia, prosecutors frequently post notice inthe court if they don’t have the address of thedefendant.

Comments: In a criminal case you may also want to measure % of cases where the defendants were appraised of their rights. You would measure this through a sample survey.

2) % of caseswhere records areprovided toparties.

% of cases (criminal, civil, etc.) whererecords provided to requesting parties.

These are public record and areneeded for defense and prosecutionof cases both civil and criminal.

Court records, statistics. Probably will not exist thena sample survey could be conducted.

This should go up and remain at a high level.

Comments: Alternative indicators include: % of lawyers/prosecutors who say the provision of records is rarely, sometimes, or often a problem. Guatemala liked this better than notifications (very important,but very difficult to track).

3) Number ofmedia stories,articles, and/orbroadcastscovering changesin law/procedureor court case.

Unit: a) number of articles appearing as amonthly average; b) number of TV andradio programs as a monthly average.

Media records.

Cost: Medium, time consuming requires absolutecount over several months.

Trendline may be uneven if sensational caseshit the press.

Comments: In Guatemala, and most of Latin America, this is increasing by leaps and bounds, but interviewees were unsure as to how to measure it meaningfully, especially since quality and length ofarticles has also risen.

4) Number ofcitizens by keypopulationcategoriesreceivinginformation onlegal rights.

Citizens receiving information via training,mass media, etc.

Unit: number, diaggregated.

Survey, NGO records.

Cost: Depends on size of country, number of NGOsin all.

Comments:



5) % of populationwho know how toaccess the legalsystem.

Disaggregate by gender, ethnic group. Willhave to develop basic set of questions, forexample, where would you go if youwanted to report a crime, if a friend orfamily member were arrested, if someonerefused to honor a contract, if you wereserved with notice of a civil complaint.Unit: % correct responses to 5 to 10 basicquestions.

Ignorance of system is one ofprimary barriers to access.

Measures above the stated IR ofavailability of information andexamines whether citizens havelearned anything by virtue of havingaccess to more information.

Sample survey.

Cost: Moderate to high, depending on size ofsample, number of questions, how representative agroup.

Should increase over time. May also want totarget certain at risk groups, or specific kinds ofknowledge.

Comments: This may require some rethinking. What kind of an answer would the average person give in the States or Europe? How would one distinguish the level of legal literacy of a U.S. citizenfrom that of a Guatemalan villager? Part of the difference may be in how one uses often very minimal knowledge and their sense of efficacy. For example, I know if someone threatens a law suit Ishould find a lawyer, and I can guess at various ways of finding a good one, but beyond that I expect the lawyer to guide me. The Guatemalan villager might not have that different an answer.



Intermediate Result 2.1.3: Equal Access to JusticeIntermediate Result 2.1.3.3: Decreased Barriers


1 ) % of pop-ulation at leasthalf day removed(by normal form oftravel) fromnearest court orpolice post.

Unit: %. This measures physical access, itaddresses barriers created by longdistance travel and people’s abilityto afford it.

Court records, statistics, census data, survey.

Cost: Will have to be on estimate; initial cost may notbe that high, but may need to hire someone to dorough analysis.

Should decrease over time. Unlikely to changevery rapidly unless many courts and policeposts are being built.

Comments:

2) Reform lawswhich directly, orthroughinterpretation, limitaccess.

Limit access by imposing costs, definingwho can be represented, establishinglanguage requirements.

Unit: Laws. Comparison of laws reformedwith changes supported.

Analysis of laws, probably by informed observers.

Cost: Probably moderate (depending on how muchanalysis and interpretation is necessary to define theuniverse of laws) one time analysis and then trackthereafter.

This should decrease over time.

Comments: In order to use this indicator you would probably need to identify the universe of laws that were limiting access and work on amending them. In Guatemala, observers noted that all laws limitaccess— i.e., commercial cases go to commercial courts, justices of the peace can only see cases up to a certain amount, statute of limitations, provisions on standing, etc. Consequently, as phrased thiswas not found to be very useful. The suggestion was that in systems where specific laws limited specific kinds of access, the question should be rephrased to handle just that.

3) User/filing feeseither absent,nominal, or linkedto ability to pay.

Unit: Scale: 1 (absent); 2 (nominal,affordable by most); 3 (nominal butinformal payments add significantly to thecosts); 4 (medium, poorest discouraged); 5(high, discourage many).

Fees often pose a barrier; if they arein force, they should be lowered oreliminated for the poor.

Public record, observation, interview.

Cost: Low.

Trend should be to eliminate fees that poseonerous barriers to poor, keep those in forcethat may help finance costs, perhapsdiscourage frivolous law suits.

Comments: Guatemalan interviewees were not enthusiastic about this indicator. mission may want to identify benchmarks that are appropriate for the country context/tailored to specific system. Examplesmight include: average filing fee for different kinds of cases; filing fees as a portion of monthly per capita income; filing fee as proportion of monthly per capita income for those below thepoverty line. Also, illegal fees are usually a large barrier, and indicator might reflect efforts to eliminate them.



4) % of casesdropped due toinability to affordthe costs.

Unit: % of total civil cases that are droppedprior to resolution.

Would indicate discriminationagainst poor.

Court records, if they exist. Sample survey but maybe difficult to draw a sample.Cost: Moderate to high.

Should decrease over time if this is an issue.

Comments: Alternative indicator classification of fees as high, medium, or low based on an analysis or per capita income (or per capita income for the lowest 50% of the population) with fees?Guatemala found this impossible to measure.

5) Number ofinterpreters per100,000 minoritylanguagepopulation.

Unit: number per 100,000 minoritypopulation. May help to look at overallaverage as well as monitoring where theyare located.

Court records.

Cost: Low.

Will be more important in some contexts thanothers; also by region of country, rural/urban.

Comments: Alternative indicators: % of courts that should offer interpretation that actually do so— “should offer” would need to be defined locally. In testing these indicators, it was also suggestedthat we measure the # of courts with professional interpreters since in many courts, they use court staff that happen to speak the relevant language, but may not have the proper training to accuratelytranslate information to the parties. In Guatemala, there is a program to provide interpreters, but the trial is not the most important place. Consultations in prisons and with police may be moreimportant. By the time a case gets to trial, if it does, much of the damage may already be done.



Intermediate Result 2.1.4: Effective and Fair Legal Sector Institutions


1) Average timefor casedisposition in newcases.

Average time from filing to disposition forcriminal and/or civil cases.

Unit: # of Months.

Timeliness is one of indicators ofbetter service. Generally averagetime should drop.

Court statistics if available; otherwise sample survey;at pilot stage, will include only changes in pilot.

Cost: If statistics exist or are created as part ofproject, costs will be minimal. Any pilot activity willalso generate its own statistics.

In most cases, we would expect the averagetime to decline since the usual problem isexcessive delays. However, in verydisorganized systems, just completing a largernumber of cases might be a goal. In someinstances, this would not be an appropriategoal, especially if the problem is perfunctoryhandling of certain kinds of cases. This is likelyfor political cases, for example. In verydisorganized systems, this can be changed tonumber of cases completed and the desiredtrend would be an absolute increase in thenumber. Where delay reduction is not animmediate goal, the average time should atleast stay stable.

Comments: In countries where there are not aggregate statistics, the cost of this indicator would seem to be very high. Proxies such as percent of cases in backlog, percent change in backlog cases, orpercent of cases that it takes longer than two years to resolve may be used. These latter indicators are not as sensitive to change as average case processing time but it may be more feasible in manyenvironments. Also probably should be for new cases — if there is a big backlog, they will continue to skew the results.

2) # of criminalcases involvingpolitical,economic, andinstitutional elitestaken to trial.

Political elites are elected office holders orare political appointees.Economic elites are top 10% of economicladder.Institutional elites include high members ofkey institutions. This will vary depending onthe country. Examples may include church,military, ethnic groups, and political parties.Unit is number of cases.

Willingness to pursue “important”suspects is an important indicator ofthe efficacy and impartiality of thesystem.

Media reports,informed observers.

Cost: Low.

This is difficult as we have no way of knowingthe number of crimes or potential suspectsinvolved, nor whether the final disposition is fairor not. Also the universe depends on suchthings as the level of corruption. Conceivably, ina very honest system, the number would startand remain low, as it would in a very corruptone with a dependent judiciary. Thus the targetsetting and relevance of the indicator willdepend on the specific situation.

Comments: We contemplated deleting this indicator because it is so difficult to define, however we would like to retain it. How it is measured might be changed. We may want to measure “patterns”and report qualitatively rather than quantitatively. If this indicator is too advanced for a country, and therefore is not sensitive to change, a preliminary indication of change might be that these eliteare investigated or charged. Note that in some countries, when a new regime comes to power it may bring previous elites to trial, but it’s motives are often not consistent with this I.R.



3) % of citizensresponding thatthey will be fairlytreated if arrested,or if they file acomplaint with thecourt system.

Response to survey question.

Unit: %, disaggregate by gender, ethnicgroup where important.

While perceptions may not matchreality, we assume there is someconnection. Furthermore, onerationale for rule of law work is toincrease citizen faith in democraticinstitutions. See comments in datacollection column.

Surveys.Cost: High to moderate. Can be held down bysample size and by including as a question in alarger survey. Guatemala is already collecting verysimilar information, with some interesting results.Faith in the fairness of the system seems to beinversely related to education (although the educatedmay believe they can manipulate the system to win;one may want to be careful about that possibility).

In all cases, the desired direction is an increasein positive responses. However, responses maylag behind “real” improvements and will also beinfluenced by salient events. Thus, changesfrom one year to another are less important thana trend over several years.

Comments: Alternative: % of population saying they would take their case to court if they were the victim of a crime.

4) Convictionsand settlementsas % of reportedcrimes [or as % ofreal crime].

Settlements are crimes “mediated out”* byagreements between victim and accused.[Real crime would have to be based on asurvey.]*Note: In Latin America there is a tendencyto substitute mediation for property crimeswhich allows the accused to pay the victimfor the damage without admitting guilt.

Presumably every reported crimehas a perpetrator, and one purposeof a criminal justice system is toidentify him and bring him to justice.For lesser crimes, some sort ofreconciliation may also be possible.

Police and court records otherwise direct datacollection at offices, or surveys.

Cost: Low if records exist; if not, surveys will have tobe done and these could be high cost.You are more likely to be able to get the % of realcrime where surveys are completed which inquirewhether people have been a victim of a crime andwhether they reported that crime.

Generally one would expect the percentage toincrease as the system improves. There areseveral caveats, however. First where there isno faith in the system, the denominator may noteven come close to reflecting the real rate ofcriminal activity. As faith in the systemincreases, the rate of reporting may also risefaster than the ability to produce results.Second, in very repressive systems there isalways the fear that every crime will have aguilty party regardless of whom the perpetratoractually is. A related indicator used inGuatemala to cover this last problem is the % ofcases for which an investigation is done beforea suspect is arrested. Also % of convictionsbased on evidence! Both obviously require asample.

Comments: The ideal indicator would have “real incidence of crime” as the denominator. However, at best there will be statistics on reports which then becomes a proxy. This really tries to get at the heartof the question of efficacy for the criminal justice system. In countries that do not have solid statistics this may be very difficult. In Uganda, for example, the only information you could probablyget is total number of police files opened during the calendar year and the number of cases that came to conclusion (acquittal, convictions). There is currently no systematic way to track the policenumber with the court number; nor does there appear to be reliable statistics on the real crime rate.



5) % or # of felonycases involvinggovernmentofficials that aretried and resolved.

Unit: Annual # or % depends on situation. An independent court must be ableto take action against the powerful.While the ideal indicator is apercentage, in some countries justhaving a few cases processed willbe an advance.

News media, word of mouth.

Cost: Fairly inexpensive.

This is difficult to measure or to target becauseof the unknown size of the “universe”. In mostcases, a number will be sufficient, althoughincreases could also be a sign (unlikely) ofincreasing corruption.

Comments: This indicator, like the above indicator on trial of elites, may be so difficult to define and measure as to not merit retention. However, it is a very important concept. Perhaps it is somethingthat missions can report in their narrative, but not try to use as an indicator — except for very special circumstances which might make it easier to determine. In some circumstances it may alsobe necessary to further qualify that you are only looking at the number of government officials of the government currently in office.



Intermediate Result 2.1.4: Effective and Fair Legal Sector InstitutionsIntermediate Result 2.1.4.1: Increased Transparency


1) % of casesholding hearingsthat are open tothe public.

Hearing may be a trial or may simply beone of a series of audiences. However,should be open to public.[May not be appropriate to include juvenileor other culturally sensitive cases in thisindicator.]

Unit: % cases with hearings over total ofcases.

While there will always be somehearings that are closed for reasonsof security or to protect the parties,most cases should involve somesort of hearing open to the public ingeneral.

Court records, or sample surveys; in person visits.[For in person visits, the indicator should be % ofhearings visited that were open.]

Costs: Moderate if records or small surveys areused. May be difficult to get data for entire courtsystem.

Targeting and predicted change depends onexisting situation. Relatively transparentsystems will not improve much; those that areclosed should show an increase. Will bedesirable to fix some ideal level, say 90% ofcases, so as not to punish the already goodperformers. Lawyers working in Guatemalaseemed to feel this should be limited to criminalcases — and that the law always defines moreopenness than exists in fact. One elementaccounting for this view is the US system’sgreater reliance (up to 80% of cases) on postverdict bargaining in civil cases. US attorneysapparently don’t hold civil cases to the samelevel of transparency, in theory or in fact.

Comments:

2) Degree ofaccess to casesas reported byjournalists.

Access to cases means the media can a)sit in on hearings and interview participantsafter the decisions are made and b) thatthey can publish on pending trials and ondecisions within the confines of the law.

Unit: Scale consisting of: (1-5)no access; limited access; occasionalaccess; frequent access; or unlimitedaccess.

Transparency denotes informationon system workings and an ability tocriticize them.

Interviews with journalists. This will involve somesubjective judgments, but they will be unavoidable inany but the most extreme cases (absolute control ofthe press or absolute freedom).

Cost: Moderate.

Targets will be set by the context, and probablyinvolve a progression through the two steps. Atthe extremes, movement will be slight or nil.There is also the problem however, of a free butirresponsible press, one which editorializeswithout bothering to get information. Oneproblem here will be restrictions imposed toprotect privacy of parties and prevent pressfrom influencing outcome. Several rights comeinto conflict here and are handled differently byvarious countries. At the very least, once adecision is reached, the press should be able tocover the results; interviews with officials duringa case may be subject to reasonablerestrictions.

Comments:



3) % of informedpeople who reportthat there is set ofwidely availablewritten rules andregulationsestablishingprocedures orinternalregulations forcarrying outdecisions, actions,and majorfunctions in thevarious sectorinstitutions?

Decisions, actions are major functionalevents, i.e., arrests, investigations, judicialfindings, etc.

Sector institutions are courts prosecutionand police. Informed people would be thetechnical people working in thoseorganizations as well as the professionalthat work with the organizations (such aslawyers, law academics, etc.). Unit: %, by major institutions.

Transparency depends on existenceof predetermined rules or criteria forinstitutional actions. These areusually written but in any caseshould be widely known.

Public records/laws and internal regulations.

Cost: Low.

The existence of rules is a precondition, butdoes not determine whether agencies actualfollow them. However, without such rulesdecisions can be arbitrary or governed by somehidden agenda. Individual countries may want topick out the areas or institutions of mostconcern, e.g., police and detentions, rules aboutgathering evidence, prosecutorial decisions asto whether to indict, etc.

Comments: This indicator will not work in a country if there is a “gap” in the legislation. In that case, a more appropriate indicator might identify the gaps and measure the closure of those gaps. Afterthe gaps are closed then this indicator could be substituted. Guatemalan observers had problems with this indicator. Latin America always has procedural rules and an “informed person” will knowthem. The problem is whether they are used, and in some cases this is as much a question of whether officials know or understand them, as one of corruption. The concept is a good one, but itneeds more refinement. In Guatemala and other Latin American countries, projects have focused on specific procedural rules (e.g., time lapse before a detainee brought before a judge, respect fortime limits on stages, required notifications, etc. This does not ease the tracking problem, but it does make the indicator more intelligible.



Intermediate Result 2.1.4: Effective and Fair Legal Sector InstitutionsIntermediate Result 2.1.4.2: Increased Independence


1) Judges mayquestionconstitutionality oflaws and defertheir applicationpending someauthoritativedecision.

Authoritative decision may be by SupremeCourt, Constitutional tribunal, or someother mechanism.

Unit: Y/N.

While full blown judicial review is stillnot universally accepted, there isemerging agreement that anindependent judiciary should havesome ability to question thelegality/constitutionality of the laws itis expected to apply.

Laws, public documents.

Cost: Low.

This still does not answer the very importantquestion of whether the powers are exercised;one may want to add some measure assuringthat they are more than theoretical. However,beyond that, factors like the quality of legislationmake attempts to measure its use extremelydifficult and extremely context relevant.

Comments: Guatemalan interviewees agreed that the abstract possibility is less important than the real use. Suggested number of constitutional challenges might be more important. However, in manyLatin American countries where every other appeal seems to be based on a violation of one’s constitutional rights, this could be difficult. Perhaps here it should be rulings in favor of the plaintiff or the actualderogation of a law either by the judiciary or as the result of their constant ruling against it in individual cases.

2) % of a) newappointments andb) promotions inaccord withobjective, merit-based criteria.

Objective merit-based criteria meansstandards having to do with educationalbackground, experience, knowledge, etc.Even if political criteria also come into play,the merit standards should set a minimumbelow which no contacts will help.Depending on the focus of the mission, thiscould be measured against Judges, Police,Prosecutors, or Defenders.

Independence requires thatpersonnel policies not be dictated byinternal or external contacts.

First is simple observation; second is probably aresult of interviews with informed observers.

Cost: Low if sole requirement is existence ofstandards. Will go up if one attempts to determinewhether standards are applied.

The US would not get 100% on this, althoughthe point is not an elimination of connections,but simply the application of some job-relevantcriteria as well.

Comments: Problem, there is merit and there is merit. Does a law degree and several years experience count? How about a comparison of curricula? Is a test better? As countries begin to adopt meritappointments (in itself a big jump) both the many ways of getting around them and the difficulty of establishing relevant criteria have begun to cause difficulties. The concept is important, but each case willrequire a refined definition and some major judgment calls.



3) % of sectorprofessionals withsecurity of tenurewhile in goodstanding.

Security of tenure means removal forcause or objectively determined poorperformance.Tenure is an appointment for a fixedduration or until retirement age.

Whether career appointments arethe best system or not is debatable,but judges and other professionalsshould have relative security oftenure (periodic “ratifications” areone solutions so long as the criteriaused are transparent).

Court records, key interviews; survey.

Cost: Low.

Institutional independence is compromised byexternal control over appointments; individualindependence can also be compromised ifpersonnel decisions are based on whim. In bothcases, tenure should be contingent onprofessional deportment, not on contacts, partyidentification, or just which administration is inpower.Note, the value of this indicator may bemitigated if there are “drummed up” chargesthat have no basis in fact.

Comments: In countries under reform, this may also require some judgment calls. In many cases, it is very difficult to determine whether a dismissal for cause is really for cause. Perhaps this mightwork best as the percentage of professionals legally covered. Then the narrative could discuss any problems with the seriousness of the legal provisions.

4) Judicial salaryas a % of what acomparableprofessionalmakes in privatepractice.

Salary is wages plus benefits like housing,car, etc. costed out. Select 2 or 3comparable professions and establishaverage salary range. Compare judicialsalaries to this range.Salary in private practice is average.Unit: %.

Judges need living wages, as dopolice, prosecutors, etc.

Court and other institutional records.

Cost: Low.

Often wages are lower for some kinds of judicialprofessional than for others. This can becompensated by averaging of the differentcategories covered. Once the level is about 80-90% of average private practice, it should peak.

Comments: Alternate: % change in salary/monetized benefits

5) Is there afunctioninginternaldisciplinary/internal affairsoffice whichmonitors forviolations in thevarious keyinstitutions?

Key institutions may be the Judiciary,Police, Public Defenders, or ProsecutorsOffice.Violations may include bribery, corruption,susceptibility to political pressures, etc.Both of these definitions need to beclarified by the mission.This indicator calls for a two part analysis.First is the institution established, andsecond is it working.Unit of Measure: Yes/No.

Establishing internal offices forviolations will increase the ability ofthe branches to operateindependent of these corruptinginfluences.

Institutions.

Cost: Low - we are not asking questions aboutwhether they are working effectively.

One time response.

Comments: While it is probably best to leave this as a question of presence not efficacy, changes in the latter may be a goal of a project, and thus some means will have to be found to measure them.Honduras and Colombia (not surveyed with this exercise) are both attempting to do this simply by counting cases reviewed and their results. This obviously poses all kinds of risks, but it is worthcontemplating in countries where such offices have been ineffectual for years (most of Latin America).



Intermediate Result 2.1.4: Effective and Fair Legal Sector InstitutionsIntermediate Result 2.1.4.3: Improved Management and Administrative Capacity

Indicator Definition and Unit of Measurement Relevance of Indicator Data Collection methods/Approx. costs Target setting/Trendline Interpret. Issues

1) Budgetsubmissioncorresponds toreal expenditures.

The budget submission is the budgetprepared by the institution.Unit: % correspondence; may also bebroken down by categories, salaries, otheroperating expenses, capital budget (if itexists).

While many other factors may enterin, submitting a budget thatapproximates what will be approvedand what will actually be spent is apretty good reflection of the qualityof institutional administration.

Official records.

Cost: Low assuming records are public.

This will go from a situation where noinstitutional budget is submitted (the govern-ment just gives them money) to one where thesubmission, while probably lowered, will matchreal expenditures about 90%. We would ex-pectthe correspondence to increase over time.

Comments:

2) % of time highlevel functionalofficials spend onadministrativedetails.

Administrative details: decisions on hiringof individuals (not general policy),individual procurement, etc.

High level officials: Judges and functionalequivalents in the Attorney General’soffice.

While one or two of them maydevote full time to administrativedetails, most should spend little orno time on such matters as howmuch officials are paid, vacationschedules, or approvingappointments of individual staff notin their immediate courtroom.

Interviews with officials.

Cost: Low [Medium if it requires a survey].

Estimates will be rough, but generally oncechanges begin, interviewees will report a one-time dramatic decrease in time. If the changesdon’t work, they may reverse that initialdecision.

Comments: It may be best to limit this to the courts, where we seem to have a clear trend (at least in Latin America, to less time being spent on these issues). In the AG’s office people were less clearas to what the trend is or should be. In countries where the Ministry of Justice “runs” the judiciary, there is sometimes a move to give more not less administrative authority to justices. That a judiciarycontrols its own administrative offices should probably mean that its professional administrative officers are accountable to the Court or its president, not that all justices intervene all the time. A moresensitive indicator might tap this division of labor (perhaps noting an effort to divide administrative decisions into policy, seen by justices, and routine decisions, handled by administrators.

3) Existence of anMIS.

Scorecard: MIS exists which providesaccurate, timely (to within last quarter):1. Info on number of officials and wherelocated; 2. Info on number of caseshandled in courts in capital; 3. Info oncases filed in district courts; 4. Data whichare aggregated and analyzed.Unit: Scale 0 (no), 1 (yes) per elementtotaled.

Having even a rudimentary MIS isessential to running an institution.

Observation, court records.

Cost: Low.

Comments: The only problem will be refining this beyond the most basic levels. There may have to be categories for more sophisticated systems, should program managers be working on these. A fifth stepwould be the presentation of data in a user friendly form to high level officials, and their report that they actually read and use it to make decisions.

Indicator Definition and Unit of Measurement Relevance of Indicator Data Collection methods/Approx. costs Target setting/Trendline Interpret. Issues


4) Time it takes aparty to receiveinformation on thestate of a case.

Party is complainant or defendant or legalcounsel.

Unit: Average time in minutes, hours, ordays.

May indicate other factors, butshould also indicate presence of anadequate case filing andmanagement system.

Interviews with users or random visits to courts torequest info.

Cost: Low - medium unless desire survey rigor.

In the beginning it is very likely that noinformation will be available for most cases; thenext step is days. The goal is immediate tounder an hour, depending on the demand.There may be urban/rural differences which willhave to be factored in.

Comments:

5) Inventory andmaintenancesystem exists forequipment andinfrastructure.

System must include means for trackingexistence of holdings and schedulingrepairs.

Unit: Y/N.

One of frequent impediments toefficacy is lack of equipment andloss of that which is provided.

Interviews, observation.

Cost: Low.

This is very basic, but for many systems is acritical impediment to efficiency. Probably donorefforts will go into creating a rudimentarysystem, especially if they are also donatingequipment.

Comments: In Guatemala, observers suggested an addition or modification to test whether repairs actually were done and equipment supplied in a timely fashion. The problem of course is that manysystems will report that they comply, but whatever exists is nonfunctional.If this does not lend itself to easy measure, one might still do more qualitative reports on what really happens to accompany theindicator itself.

6) Meritappointment ofadministrativestaff.

Staff must meet minimal skill requirementswhich are set out in appointment system.

Unit: Y/N.

Untrained, incompetentadministrative staff arecharacteristic of many inefficientsystems. Jobs are given to friendsor family with no regard tocompetence.

Legal provisions (interviews, observation, for tappingextent to which it works).Cost: Low to determine formal presence — could goup considerably if one is trying to assess the qualityof the legal system and whether it is actually applied.

Over time both percentage of appointmentscovered and rigor and job specificity of testsshould increase.

Comments: This may lend itself to percentages (if only certain institutions adopt it, or adopt it for only certain categories of employees). In Latin America, and we suspect more broadly, while the formalacceptance of merit appointments is an important leap, it remains honored in the breach. Also the definition of merit may become more rigorous over time — shifting from a minimal list of qualifications (anda “comparison of curricula”) to more job specific criteria and more rigorous and transparent means of applying them. This may lend itself to scaling: 1. Any job requirements at all; 2. Definition of job relatedskills; 3. Ranking and selection of candidates on the basis of skills testing).



Intermediate Result 2.1.4: Effective and Fair Legal Sector InstitutionsIntermediate Result 2.1.4.4: Improved Functional Organization


1) a. Are therelegal deadlines/guidelines forcase processingthat are realistic-AND-b. % of casesprocessed withinthe timeestablished by theguidelines.*

Mission may want to disaggregate thisindicator by various stages, for examplepolice investigation, prosecutorialinvestigations, actual trial, etc., dependingon the specifics of the country.

a) Yes/No

b) % of cases (by type) processed withinthe time established by the guidelines.

This will indicate both efficiency andrealism of deadlines.

A. Need to be assessed through as expertassessment.

B. Court statistics, sample.

Cost: Low to high, depending on whether statisticsexist, survey if no statistics.

Generally, positive change should come fromboth directions: more realist guidelines shouldbe developed, and more cases should meetthem. There may be differences by regions ortype of case, or projects may only focus oncertain of them (land disputes, serious felonies,etc). This will alter the target.Where deadlines don’t exist, average time toresolution can also be used, generally shoulddecrease. It should be noted that a carefulassessment needs to be done. The deadlinesmay be too short as well as too long.Note, in some countries there may be guidelinesrather than laws and in other countries neithermay exist. In those countries the precedentestablished on cases appealed on the issue oftimeliness may guide the determination of“reasonable.”

Comments: Guatemala informants liked this indicator, although they would have to base results on the pilot courts where they are working. Otherwise statistics are not very good.

2) Averagecaseload fordifferent types ofofficials.

Choice of officials depends on whereprogram is working and will probably betrial judges, persecutors, defenders andpolice investigators.

Indicator of greater efficiency,efficacy.

Court statistics, samples.

Cost: Low to high, depending on whether statisticsexist; survey if no statistics.

In most cases, productivity is low due tooutdated, unnecessary procedures; thusgenerally would expect to increase. However,where there are too few officials, may decrease.May want to add another indicator of variancearound the mean, since in some systems theproblem is the inequitable distribution.

Comments: A large backlog reduction program or the introduction of filtering mechanisms may cause an enormous change in the averages. This will have to be explained in the accompanying text.



3) % of casesannually cleared,disaggregated bycriminal and civilcases.

This measure from the time a case isofficial registered by any judicial official(judge or judge-like person) to the time it isresolved or dismissed with prejudice.

Unit: %.

Ideal is 100% which means they arekeeping up with demand.

Official statistics or samples.

Cost: Low to moderate depending on whetherstatistics are available.

Target is 100% over long haul; might be higherat beginning if there is a substantial backlogproblem. Backlog reduction might also confusethings if they work on getting rid of old casesfirst. This means they could have 200% but thatno new cases were being resolved.

Comments:

4) % of reportedcrimes for whichsuspect isidentified, and theinvestigation iscompleted.

Depending on program focus this could beused to measure the police or prosecutors’investigatory processes.

Unit: %.

Indicates investigative capability ofsystem.

Official statistics or samples.

Cost: Moderate.

This should increase over time. Caveat is that“any old suspect” could be fingered. Thus otherindicators needed as well if this is a concern(Note: Guatemala uses % of cases in which aninvestigation precedes arrest, just for thisreason).

Comments: In two of the field tests this indicator was not feasible to measure. In Uganda, potential proxies included % of cases which were adjourned because of incomplete investigation; and numberof adjournments due to incomplete investigations. average length of time to complete an investigation. In Ukraine average length of time to complete an investigation was seen as superior.

5) % of reversalson appeal.

Unit: percentage of cases in which thedecisions are reversed upon appeal.

Indicates “juridical security”— oneaspect of Rule of Law.

Official statistics.Cost: Moderate.

Will have to make determination depending onanalysis of system problems. Also depends onnumber of appeals. In Latin America and Egyptwhere “everything is appealed” % of reversalsmight be higher than in more stabilized systemin which past history suggests that only certainkinds of cases are worth appealing. Timeframemay also be a problem. You may not showprogress over the life of a 5 year strategy.

Comments: Depending on the constraints the mission is addressing, they may also want to consider measuring the % of cases are appealed. However, this is not a good indicator where there are nobarriers to appealing a case and appeals are often used just to delay the outcome. Average # of appeals per case; and average length of time to complete the appeals process. Guatemala notes thatthis may be best in criminal cases, especially if post verdict bargaining ever becomes popular.

6) Where thereare conflicts oflaw, there is anauthoritativemeans fordeciding whichlaw is applicableto a givensituation.

Authoritative means would be an entitycharged with making the decision and/or alogical system for prioritizing applications(i.e., hierarchy of laws).

Unit: Y/N(Users may want to disaggregated byconflicts of substantive law andjurisdiction).

Such conflicts can be a source ofcorruption or instability andfrequently characterize systems inthe process of change.

Laws, interviews, observation of cases.

Cost: Low.

Target is the existence of a system. A moresensitive measure would look at actual conflictsand the manner of their resolution, to check foruniformity of applications of mechanisms. Alsothere are likely to be areas where anauthoritative mechanism does not work — forexample certain kinds of conflict of jurisdiction.Some of these may also emerge over time asnew types of cases develop.

Comments:Agency Objective 2.1: Strengthened Rule of Law and Respect for Human Rights


Intermediate Result 2.1.4: Effective and Fair Legal Sector InstitutionsIntermediate Result 2.1.4.5: Professionalization of Technical Personnel


1) Career systembased on skills,job performanceexists.

Career system defines jobs, stages foradvancement, predictable trajectory.

Unit: Y/N.

Links professional performance toadvancement in organization.

Interviews, observation, official documents.

Cost: Low.

May be less relevant for systems (US, Britain)where some officials (judges) are recruited onthe basis of outside experience. Still for mostofficials career system is important to ensureprofessionalism.

Comments: This is an important indicator and could be refined further. That will make collecting data more costly, especially if one is attempting to determine the quality of the system and the extent towhich it is followed. However, if this is a goal of the program, it is worth further work.

2) % of officialsgiven formalannualperformancereviews.

To be considered a formal review 1) it mustbe written; 2) it must be included in apersonal file; 3) the person evaluated musthave be given an opportunity to respond inwriting to the evaluation. Unit:%

Even for officials with permanenttenure, some sort of evaluation feedback is beneficial.

Interviews, observation, official documents.

Cost: Low.

Comments:

3) % of newprofessionalsgiven entry-leveltraining.

Entry level course can be of varyinglengths, content — may be classroom ormentoring.Unit: %.

Shows degree of interest inupgrading skills — proxy.

Personnel or training records.

Cost: Low.

Should increase over time. May also want tolook at quality if this is part of program.

Comments:

4) Professionalranking of sectorpersonnel.

Professional ranking is professional imageas compared to all public and privateprofessionals in same category.Unit: High, medium, low.

Jurists at least belong to a largerprofessional community; thus howthey are ranked professionallycompared to that community is anindicator of their professionalquality.

Focus groups of opinion leaders.

Cost: Low.

Target is that they are least ranked average.One expects that public sector professionalsmight not be the absolute leaders in theprofession, but they should be ranked at leastaverage. There may be differences with levelswithin the public sector. Often the highest level,in part because of their position, receive ahigher than average ranking, whereas those atlower levels do not. [Although we are leavingthis as a proposed indicator there is somequestion whether the return is too low for theinvestment.]

Comments: Indicator probably not sensitive to change.



5) A continuinglegal educationrequirement isincorporated intomerit criteria orconsiderations onpromotions/transfers or is anelement in theperformanceevaluation.

Continuing legal education could includeformal and informal education, to bedefined in country.

Unit: Yes/No.

There is a great need for continuallyeducating professional staff tomaintain their knowledge andreputation as professionals.

Review evaluation. Institutional standards. Personnelinterviews.

Low if only want to know whether exists. If need toassess quality or compliance, cost will go up.

Comments:

6) % of judges,prosecutors,police officers,and prison anddetention officerswho are membersof disadvantagedgroups (might begender, ethnic,racial, class,religious, etc.)

Percentage of officials disaggregated bytype of employee.

This demonstrates integration withinthe generalized justice system, andextends beyond the establishmentof specific institutions. It may not bea sufficient measure by itself butcould be effectively coupled withanother of the above proposedindicators.

Survey of personnel records (if available) or samplesurvey of court personnel.

Cost: High if information not readily accessible ingovernment records.

Comments: For Latin America, ethnic, racial, religious representation may be more important than gender for fair justice. Also since most prisoners are men, why should large percentage of prisonstaff be women?

Section B: Elections andPolitical Processes

Section B: Elections and Political Processes 57

Results Framework


Definitions — Elections and Political Processes

Agency Objective 2.2. More Genuine and Competitive Political Processes

A competitive political process includes, but is not limited to, an election in which political parties orindependent candidates have the right to compete openly. The elections should be administered impartially,and all citizens should have the right to vote without unreasonable impediments. Political processes also referto the role of institutions, such as political parties, and the environment within which they operate. Politicalparties should be representative and responsive bodies. Citizens should be knowledgeable about the politicalprocess and contemporary issues and have the right to shape party platforms and influence politicaldecision-making. And elected officials need to be able to take office and govern effectively.

Intermediate Results 2.2.1 Impartial Electoral Framework

The electoral framework refers to those constitutional provisions, laws, regulations, and institutions whichgovern electoral processes. Although not a sufficient condition, an impartial framework is a necessarycondition for free and fair elections and electoral processes. Impartiality can take a number of forms, but willgenerally be recognized by a broad acceptance of the framework. Impartiality is therefore best served byfostering a substantive and inclusive debate on the framework.

(Sub) Intermediate Result 2.2.1.1. Substantive, Inclusive Debate on New Electoral Laws and/orChanges to Laws and Regulations

This result refers to those fora in which the creation of, and any revisions to, electoral laws andregulations are openly and publicly discussed. These laws and provisions are more likely to beimpartial when the debate surrounding them includes a variety of perspectives from inside andoutside the government. The specific fora of these debates will vary across cases.

Intermediate Results 2.2.2. Credible Electoral Administration

This refers to the process and system of electoral administration. In many countries, the administration ofelections and related matters (e.g., voter registration) is largely delegated to an elections authority within agovernment ministry, or to an electoral commission. “Credible” means that the electoral administration istrusted and deemed worthy of its authority by the public.

(Sub) Intermediate Result 2.2.2.1. Impartial and Transparent Electoral Authority

“Electoral authority” refers to that body responsible for managing and conducting elections andelectoral processes. While the electoral framework provides the legal basis for an election system, anelectoral authority is the primary agency responsible for overseeing the law. An impartial electoralauthority is one which carries out its duties in a neutral manner; one which does not manipulate theelectoral framework to favor certain individuals/groups; and one which has sufficient resources topermit impartial administration. This is distinct from an impartial electoral framework since it dealswith the authority’s performance under the framework.

(Sub) Intermediate Result 2.2.2.2. Effective Administration of the Electoral Process

An electoral administration, in order to be credible, must carry out its functions in a competent

60 Section B: Elections and Political Processes

manner. Such an authority has sufficient professional staff and resources and the willingness andability to make decisions autonomously. It operates effectively when it conducts administrative andlogistical aspects of the election, such as registering, designating polling sites, drawing up voters’lists, tabulating votes, providing security, educating voters and adjudicating complaints.

Intermediate Results 2.2.3. An Informed and Active Citizenry

Citizenry, under this IR, includes those eligible to vote, whether they are registered or not. The “citizenry”could also include adolescents (future voters whose attitudes toward government and politics may beinfluenced) or noncitizen residents of a country (whether or not they are future voters, they may placedemands on the public institutions of a country for services or status). An “informed citizenry” is seen as thegeneral objective of “civic education” programs, which inform citizens on key aspects of democratic politicalprocesses. “Voter education” programs have as their goal “an informed electorate” and generally are morenarrowly focused on an upcoming electoral event. “Voter education” and “an informed electorate” are time-specific dimensions of “civic education” and “an informed citizenry.” The framework presented below couldequally apply to civic education programs not addressing election specific issues, as well as to votereducation programs.

(Sub) Intermediate Result 2.2.3.1. Increased Understanding of the Political System AmongTargeted Citizens

This result involves citizen understanding of basic elements of the electoral system and of politicalchoices. Information presented to citizens regarding these issues generally come from three sources:a) official bodies, which include government agencies, election commissions, state-owned ormanaged news media (print, broadcast, or on-line), and state-directed schools/universities; b)independent, neutral, or disinterested sources, which could include independent and crediblejournalism, whether print, broadcast, or on-line/private schools/universities; civic groups, NGOs orreligious bodies; and c) partisan or interest-group sources that explicitly or implicitly advocate apolicy or electoral outcome, such as a political party newspaper or other publicity in the service of acandidate or cause. The source of information leading to greater understanding and confidence is notexplicitly captured in the indicators below. If the purpose is to evaluate a particular voter or civiceducation program, investigators will need to discern the source of the information/knowledge aswell as measure its existence.

(Sub) Intermediate Result 2.2.3.2. Increased Consumption of Political Information amongTargeted Citizens

Consumption can take the form of listening to radio broadcasts, watching television news programs,reading newspapers or political literature. Political information includes, but is not limited to, currentpolitical issues, party platforms, and candidate positions. This, along with increased understanding ofthe political system, is a necessary part of an informed citizenry, recognizing that what citizens dowith this information is what should ultimately be evaluated (i.e., does understanding of democraticprocesses actually increase? Do citizens engage in political processes more as a result of increasedconsumption?).

(Sub) Intermediate Result 2.2.3.3. Increased Political Participation among Targeted Citizens

This refers to citizens’ engagement and participation in the political process. It is comprehensive,


including party membership, volunteer service, voting, talking about politics and membership inNGOs which advocate for various policies. It represents the notion that political involvement goesfar beyond the act of voting.

Intermediate Results 2.2.4. Effective Oversight of Electoral Processes

A key element in effective oversight of electoral processes is election monitoring. Election monitoring iscarried out within a country by three types of actors: 1) political contestants, who monitor violations oftheir supporters’ political rights and seek redress of such violations through complaint mechanismsand/or through public opinion; 2) nonpartisan citizen organizations, which mobilize election monitorsand may also make use of complaint mechanisms where they have the standing to do so; and 3)international sources, which evaluate a country’s electoral framework and functions compared tointernational standards and practices. Each of these three types of monitoring operate in the pre-electionperiod, on election day and during the immediate post-election period. The purposes of such monitoringinclude: reducing the opportunities and incentives for electoral fraud, identifying shortcomings/problemsin the electoral process with the intention of facilitating free and fair elections, and legitimizing apeaceful transfer of power.

(Sub) Intermediate Result 2.2.4.1. Effective Electoral Process Monitoring

This result refers to the activity of election monitors, both domestic and international. Itaddresses the monitors’ ability to observe and evaluate all aspects of the political process andreport the findings.

(Sub) Intermediate Result 2.2.4.2. Media Fulfills Role as Watchdog in the Electoral Process

The media plays a crucial role in the oversight of electoral processes through their reports. Thisresult addresses the media’s capacity to produce credible, and accurate reports about thepreparations for and the conduct of elections.

Intermediate Results 2.2.5. Representative and Competitive Multiparty System

A representative and competitive multiparty system includes the following: 1) parties (through theirstatements, structure, and leadership) that demonstrate a commitment to transparent, inclusive, andaccountable democratic political processes; 2) parties that adopt institutional structures that enable themto reflect the interests of those they choose to represent in government or in the opposition, and tocompete effectively in periodic elections at all levels; and 3) political parties that enjoy the confidence ofcitizens, encourage citizen participation, and reinforce the legitimacy of democracy as a governingapproach.

(Sub) Intermediate Result 2.2.5.1. Political Parties Have Institutional Structures WhichReflect Internal Democratic Procedures, That Are Judged to be Transparent, Inclusive,and Accountable and That Are Accepted by the Party Leaders

This result addresses a key role of political parties: as intermediary between the electorate andthe elected. It therefore considers parties’ ability to consider and respond to constituents’interests, which can be enhanced when a party, inter alia, holds periodic, democratic electionsfor party offices, nominating candidates who reflect party membership; establishes bylaws;


holds/reports on open, scheduled meetings; ensures that the membership debates and approvesthe party platform; and has effective financial controls.

(Sub) Intermediate Result 2.2.5.2. Political Parties Have Established Functioning PoliticalParty Administrative Structures That Advance Institutional Stability in the Long-term

Political party stability encourages citizens’ political participation and promotes faith in thesystem. The specific administrative structures which can advance institutional stability will vary,but consistent elements will include the existence of and reliance upon bylaws (rather thanpersonalistic control) and effective communication systems.

(Sub) Intermediate Result 2.2.5.3. Increased Institutional Capacity of Each Political Partyto Identify, Represent, and Expand its Defined Constituency in the Electorate

In order to fulfill its role as intermediary between the electorate and the elected, a political partymust have the ability to reach out to its constituency. A political party is representative (and non-discriminatory) when it, inter alia, excludes no member based on ethnicity, religious beliefs orgender; establishes a mechanism to identify and expand a defined constituency; is organizedgeographically at several levels; maintains accurate membership lists; and has the capacity toresearch demographic characteristics.

Intermediate Results 2.2.6. Inclusion of Women and Disadvantaged Groups

A disadvantaged group is defined as any group which, in a given country, has historically been excludedfrom fair participation in political processes. The implication is that when this result is realized, theinterests advocated by women and disadvantaged groups will fully be taken into account in politicalprocesses.

(Sub) Intermediate Result 2.2.6.1. Laws Pertaining to Elections and Political ProcessesProvide for Non-discrimination Against Women and Disadvantaged Groups

Discrimination refers to anything that deters or restricts participation in the political process.“Laws” includes the laws, regulations, and enforcement provisions guiding elections andpolitical participation.

(Sub) Intermediate Result 2.2.6.2. Women’s and Disadvantaged Groups’ Legal Rights AreProtected Through Effective Enforcement of Non-discriminatory Laws Pertaining toElectoral and Political Processes

The relevance of this result depends on non-discriminatory principles being incorporated in legaland regulatory form. The result refers to the extent to which these laws and regulations areenforced in practice.

(Sub) Intermediate Result 2.2.6.3. Increased Participation by Targeted Women andDisadvantaged Groups on Election Day

Participation is defined here as voting on election day.


(Sub) Intermediate Result 2.2.6.4. Electoral Administration Is Free from Bias, Impartial inits Oversight, and Devoid of Discrimination Against Women and Disadvantaged Groups

This refers to the conduct of the electoral authority in its role as administrator of elections and inits corporate operations.

(Sub) Intermediate Result 2.2.6.5. Effective Voter Education Provided to FacilitateWomen’s and Disadvantaged Groups’ Understanding of and Ability for PoliticalParticipation

This result overlaps with those results concerning voter education, but it specifically addressesthe extent to which voter education reaches women and disadvantaged groups, as well as thelevel of education on issues concerning the rights and participation of these groups.

(Sub) Intermediate Result 2.2.6.6. Political Parties Are Supportive of the Participation ofWomen and Disadvantaged Groups in Political Processes.

This refers to the inclusion of women and disadvantaged groups in the decision-making structureof political parties, the parties’ outreach thereto, and their willingness to back members of thesegroups as political candidates. At a minimum, parties should exclude no member based onethnicity, religious beliefs or gender in the equal participation of party activities, both in terms ofthe written regulations and the behavior of the membership.

Intermediate Results 2.2.7. Effective Transfer of Political Power

This essential element is applicable to the cases where new individuals, groups, or political parties arecoming to power in the legislative or executive branches of government, or at the national, regional, andlocal levels.

(Sub) Intermediate Result 2.2.7.1. Procedures for the Transfer of Power Are Establishedand Followed

This addresses the extent to which the laws regarding the process are followed.

(Sub) Intermediate Result 2.2.7.2. Newly Installed Officials Are Prepared to Fulfill TheirResponsibilities

“Prepared” in this result refers to whether the officials are oriented to their new positions throughtraining and ready to fulfill their responsibilities.

(Sub) Intermediate Result 2.2.7.3. Agencies of Government, Including Military andSecurity and Opposition Groups, Accept the Authority of the Newly Installed Officials

This refers to the extent to which government agencies and the congressional opposition followthe orders and/or cooperate with the new officials in their legally sanctioned realms ofresponsibility.

(Sub) Intermediate Result 2.2.7.4. The Public Recognizes the Legitimacy of the Process by


Which New Officials Are Chosen

This refers specifically to the process, not to the officials themselves, and is evidenced by mediastatements, lack of public demonstrations, etc.

Section B: Political Processes 66

Indicators

Agency Objective 2.2: More Genuine and Competitive Political Processes

Intermediate Result 2.2.1: Impartial Electoral Framework

Indicator Definition and Unit of Measurement Relevance of Indicator Data Collection Methods/Approx. Costs Target Setting/Trendline Interpret. Issues1) Degree towhich electorallaw/rules conformwith internationalstandards (couldbe a scorecard orindex, seeAppendix C foradvice on thesetypes ofmeasures).

International standards: a) internal consistency and clarity;b) comprehensiveness;c) degree to which transparency isrequired; andd) degree of avenue of redress available toelectorate.

There is a broad consensus oninternational standards forelectoral/political legal framework.

Interviews with domestic and internationalmonitors/panel of experts.

Cost: Collateral with mission activities/moderate(monitoring).

Should demonstrate progress toward conformitywith international standards in successiveelection cycles.

Comments: Assumes that electoral law exists rather than appointed commission that determines electoral policy.2) a) Degree ofacceptance by allpolitical parties.

Significance of the political party anddegree of (non)acceptance should both beweighed in this indicator.Scale of acceptance according to keyrules/laws by each political party.

Acceptance of electoral law/rules bycompeting parties legitimizes theelectoral process.

Interviewing of party leaders/evaluation of publicpronouncements by party leaders regardingacceptance of electoral rules.Cost: Interviewing/high (although depends on whoconducts interviews), public record/low

General trend toward broad acceptance.Less than complete acceptance by someparties is not unusual in a polarized politicalsetting. The opposition is likely to demonstrateless acceptance than the ruling party.

Comments: Mission should determine specific focus of acceptance, such as electoral rules regarding vote thresholds for representation, choice of proportional representation or plurality system, delineationof electoral boundaries, etc. In many electoral situations, opposition party complaints about the election framework are unfounded/based on self interest. This should be taken into account when utilizing thisindicator and addressed in any text when reporting results. 2) b) Degree towhich multipleparties arerecognized andsanctioned inelectorallaw/rules.

Political perceptions/actual languagereflecting openness of party system. 5-point scale: no recognition to fullrecognition and sanction: 1=no languageregarding multiple parties to 5=cleardefinition of rights/recognition of multipleparties in ballots.

Recognition and sanctioning ofmultiple parties provides the varietyof choices for citizens and reflectsthe impartiality of the politicalsystem.

Content analysis of electoral law/rules.Survey or focus group to determine actors’interpretation and implementation of electorallaw/rules.

Cost: Content analysis/low, perceptions, moderate.

Trend toward increased recognition of multipleparties.

Comments:



3) Degree towhichadministrative orpolitical problemsoccurred duringan electionprocess as aresult ofweaknesses in thelaw. (Scale ORqualitativeassessment.)

Definition:Significant administrative or politicalproblem: structural/managementbreakdowns in electoral administration dueto vagueness or lack of direction inelectoral law.

Clear guidance from electoralframework is vital to effectiveadministration of electoral process.

Independent monitors’ reports.

Cost: Collateral with mission activity(monitoring)/moderate.

Decrease in administrative problems aselectoral law is reformed.

Comments: Weaknesses relative to international standards, as identified in 1 above. 4) Degree ofindependence ofelectoral authority(ies). (Scale orqualitativeassessment.)

Definition:Independent body: Electoral authority isindependent from influence/control byexecutive, ruling party, or other politicalforces.Survey/interview questions: assessingdegree of independence.

An independent electoral authorityserves as an impartial, independentinstitution in democratic societies.

Evaluations by independent monitors, domesticmonitors/surveys of public acceptance of electoralauthority as independent institution.

Cost: Moderate/high. Interview/survey.

In many countries, including some developeddemocracies, elections are administered by aministry of the government. This may or maynot impinge on the perceived neutrality of theelectoral authority. Also, public opinion may nottrack closely with improvements inindependence.

Comments: Independence could involve different bodies or organizations, depending on the context. Mission should determine which is the appropriate focus (i.e., electoral authority’s independencefrom ruling party or from the executive, etc.). 5) Existingelectoral lawsmodified inkeeping withimprovementssought.

Modification of electoral laws: changesmade in existing law that improve theelectoral process; through greateropenness; cleaner delineation of voter,party; and candidate rights andresponsibilities.Comparative assessment of existing lawand reformed law against proposals forchange by domestic/internationalobservers (specific modifications andidentification of those interests seekingmodifications to be determined bymissions, depending on country context).

Adaptability of electoral law toreforms articulated by the public isan important element indemonstrating impartiality inframework.

Comparative content analysis of existing laws/publicproposals for change/reformed electoral laws.Evaluations undertaken by observers/monitors.

Cost: Low/Moderate

Short term reform efforts followed by stability inelectoral law.

Comments: See the legal reform scale in Appendix C as an alternative measure.



6) Breadth ofdissemination ofelection provisionsand frameworks.(Index ORqualitativeassessment.)

Breadth of dissemination by any means orformat: Index of media attributes scaledfrom non-existent to high (1 to 5). Levels ofdissemination/reach (see note below) orqualitative assessment by observers.

Importance of reaching citizens withmessage of electoral rules.

Comprehensive assessment ofdistribution/dissemination by mediamonitors/observers.

Cost: Moderate/high.

Improvements would reflect more clearlytargeted dissemination efforts.

Comments: Index of media attributes: a. print media (public/private); b. television broadcast; c. languages; d. reach illiterate, semi-literate; e. adequacy of distribution; f. government; g. parties; h. NGOs.



Intermediate Result 2.2.1: Impartial Electoral FrameworkIntermediate Result 2.2.1.1: Substantive, Inclusive Debate on New Electoral Laws and/or Changes to Laws and Regulations

Indicator Definition and Unit of Measurement Relevance of Indicator Data Collection Methods/Approx. Costs Target Setting/Trendline Interpret. Issues1) Nature ofcoverage ofdebate by publicand private media.(Scale ORqualitativeassessment.)

Volume/quality of media coverage. Columninches/minutes of coverage.Qualitative assessment of debatecoverage.

Openness of debate is importantmeasure of publicsupport/acceptance of electoralprocess/rules.

Content analysis of media coverage of debate onelectoral laws/reforms.

Cost: Moderate/high, quite costly if extensive debateperiod. Could be modest if commercial monitoringservices exist.

Content specific, increased coverage of debateby media.

Comments: “Nature” includes both extent/volume of coverage and quality of coverage. Coverage of high quality means, in part, that the media has fairly presented the choices (opposing sides of thedebate) to the public. This is an indirect measure of inclusiveness; needs to be combined with other indicators if selected.2) Number ofparticipants indebate fora.

Number of debates/average number ofparticipants in each debate.

Level of participation reflects publicinterest in and attentiveness tochange in electoral law.

Attendance lists or counts from debate fora/mediaaccounts of attendance at debate fora.

Cost: Low/moderate.

Context specific, increased attendance, andmore opportunities for debate.

Comments:3) Diversity ofparticipants inpublic debate fora.

Diversity of participants: multiple points ofview represented in debate fora.Initial measure: presence/absence ofmultiple NGOs participating. Scale includesparticipation by only a few NGOrepresentatives and individual citizens, tobroad representation of NGOs and citizenopinions.

Diversity of participation reflectsbroad public interest in andattentiveness to change in electorallaw.

Monitors in attendance at debate fora/mediaaccounts of representativeness of debateparticipants.

Cost: Low/moderate, depending upon number of foraand records of organizers.

Increased diversity, context specific.

Comments: Can be measured by level of NGO participation (scale) or level of participation by any other civil society groups.4) Diversity ofparticipants inlegislative debate.(Two-level coding:Y/N; if Y, thenscale fordiversity.)

Diversity of participants: multiple points ofview represented in legislative debate.Initial measure = presence/absence ofmultiple points of view.Scale: includes participation by only a fewmembers of the governance party/coalitionto broad participation by members of bothgovernment and opposition parties.

Legislative debate should includediversity of view points in debateover electoral rules.

Observer accounts of legislative debates anddiversity of participation.

Cost: Low/moderates.

Increased diversity of participation in legislativedebate, context specific.

Comments: See also legislative indicators.



5) Degree towhich public foradebate involves(interactive)dialogue.

Interactive dialogue: full exchange of ideas;ordered, rule-governed debate; andfreedom of participation.Scale: limited debate/noexchange/disorderly to ordered debate withfull exchange of ideas among participants.

Exchange of ideas and full, opendebate is preferred overmonologues without discussion.

Content analysis of public debate.


Increasingly interactive debate in public fora.

Comments:6) Degree towhich debate inlegislatureinvolves(interactive)dialogue.

Interactive dialogue: full exchange of ideas;ordered, rule governed debate, andfreedom of participation.Scale: limited debate/noexchange/disorderly to ordered debate withfull exchange of ideas among participants.

Exchange of ideas and full, opendebate is preferred over disorderlymonologues without discussion.

Content analysis of debate in legislature, focusing onkey issues.


Increasingly interactive debate in legislature.

Comments: 7) Degree towhich the newlaw, or changes tolaw, reflectoutcome ofdebate (eitherdebate inlegislature or inpublic fora).

(See comments.)Qualitative judgment of impact of debate ofelectoral reform.

Debate results in change inelectoral law.

Content analysis of existing law/reformed law andcontent analysis of debate.

Cost: Depends on number of laws covered.

Increasing impact of public/legislative debate onelectoral reforms.

Comments: The resultant changes to the law may reflect the consensus reached through legislative or public debates. It is possible to have an excellent debate without affecting the law. See also the legalreform scale in Appendix C.



Intermediate Result 2.2.2: Credible Electoral Administration


1) Degree towhich significantpolitical actors(parties andcandidates)accept thelegitimacy of theelectoralauthority asmanager andarbiter of theelectoralprocess.

Definition: Significant parties: those withthe ability to command 5% of the nationalvote and/or with the ability to disrupt thepeace with violent demonstrations or riots.

Unit of measurement: A matrix utilizing lowto high significance (on a 1-5 scale) andlow to high acceptance (also on a 1-5scale).

Acceptance of legitimacy ofelectoral authority by key politicalactors is crucial to the credibility ofelection administration.

Interviews with party leaders- leaders’ assessmentsof election administration.

Cost: Low/moderate (access to party leaders).

Increasing acceptance of legitimacy of electoralauthority.

Comments:

2) Level of voterconfidence inelectoralauthority(Scale).

Voter confidence in electoral authorityscale: no confidence to full confidence (1-5).

Voter confidence is also a vitalelement in gauging the credibility ofelection administration.

Survey research/focus groups (registered voters).

Cost: Moderate/high. Cost depends on content ofsurvey.

Expect increase in level of voter confidence inelectoral authority. Confidence may not keeppace with actual improvements.

Comments:3) Degree towhich inputand/or feedbackby parties andcandidates isconsidered byelectionsauthority.

Each element on a 1-5 scale, forming anindex. Index could be based on:announcement of call for elections;selection of electoral authority; debate overformation of electoral authority;implementation of electoral policy; andinstitutionalization of electoral authority.

Receptivity of electoral authority tocomments/feedback from politicalactors demonstrates openness ofelectoral process.

Interviews with party leaders, interviews withmembers of electoral authority.

Cost: Access to subjects; Low/moderate.

Increased consideration of views of politicalactors by electoral authority.

Comments:



4) Degree towhich electionauthorityprotectsindividualelection-relatedrights andfreedoms.

Protection of voter rights and freedoms: (tothe degree that such responsibilities aremandated to the election authority)elements determined by election law itself.

Electoral authority mustdemonstrate that it is able to protectthe rights of citizens in the electoralprocess.

Evidence collected regarding the lack of protection ofrights and freedoms by citizens.

Cost: Low/moderate.

Increasing protection of voter rights andfreedoms, as delineated by electoral laws.Target setting should include only 1-2 yearspreceding elections.

Comments:5) Degree towhich electionauthorityprotects rightsand freedoms ofcandidates,parties, andorganizedinterests.

Protection of rights and freedoms asspecified in electoral law.

Electoral authority mustdemonstrate ability to protect therights and freedoms of parties andcandidates in electoral process.

Evidence collected regarding the lack of protection ofrights and freedoms by citizens.

Cost: Low/moderate.

Increasing protection of candidate and partyrights and freedoms, as delineated by electionlaw.Target setting should include only 1-2 yearspreceding elections.

Comments:



Intermediate Result 2.2.2: Credible Electoral AdministrationIntermediate Result 2.2.2.1: Impartial and Transparent Electoral Authority


1) Degree towhich electionauthority isperceived to beconstituted in amanner whichallows it to beneutral. (Scale).

National election authority: the governmentministry, agency, office, or commissioncharged with administering elections.

Unit: 1-5 scale, from not at all neutral tototal neutrality.

Perception of neutrality is not thesame thing as true neutrality, butgiven the importance of perceptionin election politics, the perception ofmajor political actors is a relevantindicator of true neutrality.

Focus groups or observer reports.

Panel of experts.

Costs: Medium

It is unlikely that any data sample will result inan “entirely neutral" judgment. In polarizedsocieties, any “score" above 1.5 may beconsidered satisfactory.

Comments:

2) a) Adequacy ofresourcesprovided toelectoral authority.(Scale ORqualitativeassessment.)

Minimum amount of resources necessaryto carry out election law mandate: expertpanel—evaluation of existing resources.

Scale: 1-5, totally inadequate-sufficient.

Electoral authority must have thecapacity to carry out its mandate inan independent fashion withadequate resources.

1. Determination of costs by experts.2. Analysis of resources made available.

Cost: Moderate.

Increased adequacy of resources.

Comments: Lack of sufficient financial resources is a structural constraint that can keep the electoral authority tied to government demands. This indicator also assumes the electoral authority may usevarious sources of funding including governmental, nongovernmental, and international.

2) b) Degree towhich adequateresources areprovided in atimely fashion toelection authority.

Timely provision: funding and materialsprovided to authority in order to carry outits mandate without delay or detriment tocredibility of electoral process.

Lack of sufficient financial resourcesis a structural constraint that cankeep the electoral authority tied togovernment demands.

Evaluation of timely provision of adequate resourcesby international monitors/panel of experts.

Cost: Moderate.(Should be linked with 2) a).

Increasingly timely delivery of resources toelection authority.

Comments: Assumes mission does not have access to Election Authority to gather information on this indicator directly.



3) Publicperception ofelection authorityimpartiality andtransparency.

Survey/focus group questions to publicregarding impartiality and transparency ofelection authority. Citizen perceptions onan indexed scale of 0-100 with 100meaning perception of absolute impartialityand transparency. Scale: Low/highimpartiality. Low/high transparency.

Important to have public support forelection authority as impartial andtransparent institution-legitimacy.

Focus groups/survey research.

Cost: Moderate.

Increasing public support for election authorityas impartial and transparent institution. Publicperception may lag behind real improvement.

Comments:

4) Perception ofpolitical actorsregarding electionauthority’simpartiality andtransparency.

Survey/focus group questions to politicalactors regarding impartiality andtransparency of election authority.

Scale: Low/high impartiality.Low/high transparency.

Important to have support of politicalleaders for election authority asimpartial and transparent institution.

Focus groups/survey research.

Cost: Moderate/high, depending on number ofinterviews.

Increasing political leadership support forelection authority as impartial and transparentinstitution.

Comments: Indicators 3) and 4) should be analyzed in tandem to demonstrate elite and mass support for election authority.



Intermediate Result 2.2.2: Credible Electoral AdministrationIntermediate Result 2.2.2.2: Effective Administration of the Electoral Process

Indicator Definition and Unit of Measurement Relevance of Indicator Data collection methods/Approx. costs Target setting/Trendline interpret. issues1) Number ofyears sinceupdate of voterregistry completedand/or number ofyears since lastaudit completed.

Definitions: Voters registry: officialgovernment voter registration documents.Audit: thorough analysis of voter rolls.Comprehensive update: purging deceasedvoters, removing voters who have leftdistrict/region, adding voters who havebecome eligible since the last election,changes due to changes in address. Unit:Number of years.

An election authority’s ability andwillingness to administer credibleelections can be partially measuredby its ability and willingness to keepvoter registration information up todate.

Interviews of election authority and/or press reports.

Cost: Low.

At minimum, registry should be updated and asaccurate as possible prior to a national election.

Comments: This is only appropriate where audits are known to be reliable; otherwise, some measure of audit accuracy may be more appropriate.2) Percentage oferrors in voterregistry.

Errors: Incorrect information in the registry,either typographical or factual, andomissions.Unit: percentage of errors, out of totalentries.

Voter registry should be as free oferrors as possible: omissions ofeligible voters, inclusion of ineligiblevoters.

Results of audits undertaken by electionauthority/judgment of the comprehensiveness ofaudit undertaken by election authority. Cost: Low.Independent audit conducted by international/domestic monitoring source. Cost: High.

Fewer errors in voter registry over time.

Comments: Measurement constrained by external auditing mechanism; it may be limited to available alternative vote tabulation efforts.3) Degree towhich partieshave opportunityto review voterregistry andchallengeinaccuracies ifnecessary. (two-level coding: Y/N;if Y then level ofopportunity.)

A., Statutory language that allows partiesaccess to voter registry (Y/N).

B. Practical opportunities for parties tochallenge voter registry (Scale: minimal tomultiple opportunities).

Access to voter registry by politicalparties provides multiple externalchecks of the accuracy of electionauthority efforts to register eligiblevoters.

Analysis of statutory language regarding access tovoter registry/interviews with party leaders todetermine practical application of election law.

Cost: Moderate/low.

Greater access to voter registry by politicalparties; increased ability to challengeinaccuracies.

Comments: 4) a) Percentageof eligible votersregistered to vote.

Eligible population: legal voting agepopulation. Dissaggregate by gender,ethnicity, etc. where appropriate.Unit: percentages, disaggregated.

Voter registration percentages canbe a superior measure of electionauthorities administrativeeffectiveness than voter turnout.

Collect data on number of eligible voters, which isusually available from election authority. Number ofeligible voters is usually available only through (oftenunreliable) census estimates. Cost: Low.

While 100% registration of eligible voters isoptimal, 80% would be considered a goodparticipation rate in most countries, althoughthis should be determined by mission.

Comments: This is the first of two alternate indicators regarding voter registration participation. Mission should consider country context as well as the mandate of the electoral authority.

Indicator Definition and Unit of Measurement Relevance of Indicator Data collection methods/Approx. costs Target setting/Trendline interpret. issues


4) b) Percent ofvoters who haveofficiallysanctioned meansof verifying theireligibility to vote.

% of voters turning out to vote who carryon officially sanctioned means of verifyingtheir eligibility to vote.

Voters should be able to easilyidentify themselves at the pollingstation on election day. Often,voters are given a voter ID whenthey register to prove that they areeligible to vote at the polling station.Sometimes other ID documents areaccepted for proof of eligibility.

Election day monitoring (international and domestic).Monitors could, for example, interview every fifthvoter.Cost: Moderate/high.

Increasingly systematic voter registration andvoter identification processes over time.

Comments: It would also be interesting to try to capture whether these means are being routinely accepted by poll workers. Monitors might be able to report on the extent of problems.4) c) Percentageof voters withproof ofregistration whosenames do notappear on votinglists at pollingsites.

Rejected voters with proof of registration,disaggregated by region/district, ethnicity,etc. to identify particular areas withproblems.

Rejecting eligible voters at pollingsites delegitimizes election authorityand the electoral process moregenerally.

Election site monitors across nation. Reports frompolling sites recorded/reported in media.

Cost: Moderate/high, but it depends on quality ofrecords maintained by authority.

Decrease in percentage of voters with proof ofregistration rejected at polling sites.

Comments:5) Keybenchmarks inelectoral calendarare accomplished.

Key benchmarks: predetermined deadlinesfor voter registration, candidate/party filing,fulfillment of ballot access requirements,etc.

Adherence to a pre-establishedcalendar of election-preparationbenchmarks is an importantindication of the election authority’sability to effectively administer theprocess.

1. Obtain calendar from election authority atbeginning of process.

2. Monitor accomplishments against calendar.

Accomplishment of all benchmarks on electoralcalendar.

Comments: Accomplishment involves timeliness. The mission should choose which benchmarks are key, then determine the percentage of those which are accomplished.6) a) Degree towhich votetabulation andreporting votes iscarried outaccurately andtransparently.

Counting: Qualitative assessment of bothtabulation and reporting efforts(accuracy/transparency) conducted byinternational or domestic monitors.Tabulation: tallying of vote counts bycentral authority. Reporting: disseminationof election results.

Each step of the vote tabulationprocess should be carried out in anaccurate and transparent fashion.

Reports of international and domestic monitors.

Cost: Collateral with mission activities,moderate/high.

Increasingly accurate and transparent votetabulation process.

Comments: This is the first of three alternate indicators regarding election results reporting. In most cases the assessment of accuracy and transparency should be conducted by independent monitors.



6) b) Number ofdays required totabulate andannounce theofficial results.

Number of days. Timely reporting of election resultsserves to demonstrate the efficiencyand effectiveness of the electionauthority.

Monitoring of time elapsed between closing of pollsand reporting of results of election.

Cost: Minimal.

This can be highly variable across countries. Inthe U.S. and U.K., results are announced withinhours of the closing of the polls. In othercountries, a delay of several days is notunusual. The standard used should be priorhistory of multiparty elections in eachparticularly country.

Comments:

6) c) If parallelvote tabulation—discrepanciesbetween PVT andofficial results.

Parallel vote tabulation: alternative votecount conducted by independent monitor.Systematic methods of PVT are availablefor missions.

Significant discrepancies betweenthe results of a well-done PVT andthe official results can be highlyindicative of malfeasance on thepart of the electoral authority.

Domestic monitoring groups sometimes carry out aPVT, often with external assistance.


Expected deviation between a well-done PVTand the official results should be no more than2-3%.

Comments: PVT could be less accurate than official count.7) Percentage ofpolling places thatopened on timeon election day.

Number of polling sites opened ontime/total number of polling sites.

The ability of the election authorityto manage the logistics of gettingpeople and materials to the pollingplaces on time is a useful measureof the election authority’sadministrative effectiveness.

International or domestic observer reports.

Could be either a complete reporting or a sample.


Increase in percentage of polling placesopened on time.

Comments:8) Percentage ofpolling places thatran out ofmaterials duringelection day.

Number of polling sites that ran out ofvoting materials/total number of pollingsites.

Inability to provide necessaryresources to eligible voters reflectsnegatively on electoral authority.

International or domestic observer reports.

Cost: Moderate/collateral.

Decline in percentage of polling sites withinadequate voting materials.

Comments: Materials should be specified (ballots, supporting documents, etc.).



9) Nature ofcomplaints bypolitical partiesabout equitableaccess to publicprint andelectronic media.

International standards expect that allpolitical parties be provided access topublic media. Credibility of complaints to be judged bymonitors.Unit: number and substance of crediblecomplaints. Could convert to a scale.

Political party complaints aboutmedia access are often but notalways a valid indicator of theelection authority’s failure to ensureequitable access.

Informal survey of political party leaders, evaluationof complaints by monitors. Media content analysis.

Cost: Moderate/high, if there are a high number ofcomplaints.

Decline in complaints by political parties aboutaccess to media.

Comments: “Nature” includes both extent/volume of complaints and quality/legitimacy of complaints. Election monitors will often serve to assess the nature of complaints. 10) Nature ofcredible com-plaints by citizens,media, or politicalactors concerningelection authorityfavoritism orsuccumbing topolitical pressure.

Favoritism/succumbing to politicalpressure: subjective measures to beassessed by monitors.

Credibility of complaints to be judged bymonitors.

Unit: Number and substance of crediblecomplaints. Could convert to a scale.

Complaints regarding perceivedfavoritism or succumbing to politicalpressures by electoral authority bypublic, media, and/or political actorschallenge the impartiality andtransparency of the authority.

Evaluation of complaints by monitors.


There will always be complaints about the lackof neutrality of the electoral authority. In eachcountry, a threshold should be pre-established,perhaps based on previous experience, for anacceptable number of complaints.

Comments: “Nature” includes both extent/volume of complaints and quality/legitimacy of complaints. Election monitors will often serve to assess the nature of complaints. 11) Nature ofcomplaints fromsignificant politicalparties aboutelection-relatedsecurityarrangements.

Complaints related to incidents involvingbreakdown in the maintenance of safe andsecure polling places.

It is the responsibility of thegovernment to ensure the securityof candidates and voters to thedegree possible.

Evaluation of complaints by monitors.

Cost: Collateral/moderate.

Decline in complaints regarding securityproblems.

Comments: Mission should specify type of complaints, which could be too many or too few police officers at polling stations, for example. It may depend on the types of problems experienced in previouselection cycle. 12) Breadth ofdissemination andquality of votereducationmaterials byelection authority.

In most countries, it is the responsibility ofthe electoral authority to provide minimuminformation about election procedures tothe electorate.Unit: both number and quality should bemeasured.

Citizens need to be informed aboutthe mechanics of electionprocedures; voter educationprograms provide informationnecessary for citizens to participatein elections.

Analysis of voter education materials anddissemination process by monitors; assessmentreport.


Increased quality and breadth of disseminationof voter education materials by electionauthority.

Comments: Use only when electoral authority is responsible for voter education.



13) Degree towhich spoiledballots are due toinadequateunderstanding ofvoting process, asassessed byelection monitors.

Spoiled ballot: a ballot that is ruled to beunacceptable as a valid indication of voterchoice by election authority.

A large percentage of spoiledballots due to inadequateunderstanding of voting processcould reflect lack of sufficienteducation campaigns undertaken byauthority.

Assessment of spoiled ballots by monitors.Cost: Moderate/high.

A percentage of spoiled ballots greater than5%, for example, is often an indication thatvoters were insufficiently informed about voterprocedures. Exact figure should be determinedby mission.

Comments: If unable to distinguish those ballots spoiled because of inadequate understanding, consider using the % of total ballots that are spoiled, although spoiled ballots could also reflect fraud or citizenprotest.14) Percentage ofeligible votersunable to vote dueto irregularities atpolling stations.

See 2.2.2.2. 4) c) Eligible voters turnedaway at polling sites/eligible voters turnedaway + eligible voters who voted.

Irregularities at polling sites reflectinadequate training of pollingpersonnel, disregard for electionrules, lack of impartiality.

Monitoring reports of voters unable to vote due toadministrative irregularities.


Decline in election day irregularities over time.

Comments:15) Degree ofmalapportionmentbased onprovisions in theelectoral law, inelectoral districts.

Based on electoral law, voting districts thatsurpass limits of malapportionment by asignificant margin.

One-person-one-vote standard asgoal, taking into accountpolitical/ethnic/geographic factors incountry.

Analysis of voter boundaries and voter populationsfound in each district, voting unit.

Electoral systems approaching one-person-one-vote principle are most democraticallyrepresentative.

Comments:



Intermediate Result 2.2.3: An Informed and Active Citizenry


1) Level ofconfidence indemocraticprocesses amongcitizens.

Index of responses to questions regardingcitizen confidence in democraticinstitutions, media, political parties, andelection process; and measures of politicaltrust.

Public confidence in democraticprocesses is important element instabilizing the political system.Measures at a higher level than thestated result. Better planningindicator than program performanceindicator.

Survey research/focus groups: public/targetaudiences.

Cost: Moderate/high, unless a survey is being doneto collect data for several indicators.

Increased confidence in democratic processesover time, but this would be an indirect measureof a program and may not accurately reflectimprovements. Not worth measuring annually.

Comments:

2) Level ofefficacy regardingpolitical systemamong citizens.

Index of responses to questions regardingcitizens’ level of activism, support role inthe political system. Do citizen viewsmatter, does citizen vote matter?

It is important that citizens believethat they matter or that they have avoice in the political system. May bea better planning indicator thanprogram performance indicator.



Increased levels of efficacy among citizens,over time. Not worth measuring annually.Change likely to be slow. Movement may alsobe tied to levels of general education.

Comments:

3) Level ofpoliticalknowledge andunderstanding ofpolitical systemamong citizens.

Index of responses to questions regardingcitizen knowledge and understanding ofpolitical institutions, laws, rights, freedoms,responsibilities, constitution.

An informed electorate is anecessary element in a democraticsociety.



Increasingly better informed electorate.

Comments:

4) Level ofpartisan orelectoral activismamong citizens.

Index of political activism — number andfrequency of activities: voting, talking withfriends about politics, volunteering with aparty, etc.

A politically active citizenrydemonstrates a civic vitality that isimportant to a democratic society.



Increased levels of political activism.

Comments: Regarding IRs below: Missions undertaking programs that target specific constituencies or groups should make every effort to identify clearly the targeted sector as well as preexisting levels foreach of the indicators presented under each IR.



Intermediate Result 2.2.3: An Informed and Active CitizenryIntermediate Result 2.2.3.1: Increased Understanding of Political System Among Targeted Citizens


1) Targetedcitizens’ abilities toidentify andunderstand basicelements ofelectoral system.

Index of responses to questions of targetedcitizens regarding knowledge andunderstanding of major political actors,political institutions, and electoral laws,compared with baseline responses.

Program activities should result inmore informed target citizenry.

Survey research/targeted citizens. Pre/post activity.Cost: High. (Surveying targeted individuals over timein a panel survey is preferred but likely to be moreexpensive than comparable random samples amongthe target group but this depends on the size of thetarget group.

Increased levels of political knowledge andunderstanding of political system within targetedcitizenry.

Comments: Pre- and Post-Activity Assessments.

2) Targetedcitizens’ abilities todiscerndifferences inpolitical choices.

Index of responses to questions of targetedcitizens regarding political choicesavailable to citizens in political system,compared with baseline responses.

Recognition of differences amongpolitical candidates, partiesdemonstrates a discerningelectorate.

Survey research/targeted citizens. Pre/post activity.Cost: High. See comments in indicator 1) aboveabout panel surveys.

Increased ability of target citizens todifferentiate between political choices availableto them in electoral system.

Comments: Mission should specify which type of choice would be appropriate to follow. It could be, for example, differences among parties, between candidates, among policy positions, etc.



Intermediate Result 2.2.3: An Informed and Active CitizenryIntermediate Result 2.2.3.2: Increased Consumption of Political Information Among Targeted Citizens


1) Newspaperreading, radiolistening, andtelevision viewingfocused onpolitics, reading ofparty literature,reading of electionmaterials, andother politicalcommunicationsby targetedcitizens.

Index of information consumption; pre/postactivity.

Frequency of newspaper reading, forexample: 6) every day, 5) 4-5 times/week,4) 2-3 times/week, 3) once/week, 2) lessthan once/week, 1) never.

Increase in consumption of politicalinformation fosters betterunderstanding of political system.

Survey research/targeted citizens. Pre/post activity.

Cost: High. See comments in indicator 1) aboveabout panel surveys.

Increased consumption of political information.

Comments:

2) Existence ofinformationgatheringactivities such asattendance atpublic fora,political rallies,political partyevents andactivities, andeducationalprograms bytargeted citizens.

Index of political information gathering.

Pre/post activity.

Measure: Frequency of informationgathering activities: 1) attendance at publicfora; 2) attendance at political rallies; 3)attendance at political partyevents/activities; and 4) participation inpolitical education activities. Eachcomponent could be scored as 1) never; 2)a few times; 3) often; 4) all the time.

Beyond media resources and partydocuments, citizens may seekinformation through participatorymeans.


Cost: High. See comments in indicator 1) aboveabout panel surveys.

Increased efforts at information gathering.

Comments:



Intermediate Result 2.2.3: An Informed and Active CitizenryIntermediate Result 2.2.3.3: Increased Political Participation Among Targeted Citizens


1) Voter turnoutamong targetedcitizens.

Past turnout rates of targeted citizens,compared with post-activity turnout rates.Disaggregate by gender and other relevantcategories.

Reflects an informed and committedelectorate.

Baseline turnout levels/turnout in targeted election. Increase in turnout among targeted citizens.Since many factors affect voter turnout, dataneed to be handled carefully.

Comments: Increased voter turnout is not always a valid measure of an informed and committed electorate. As a country’s political situation stabilizes, turnout may decrease. 2) Political activityamong targetedcitizens.

Index of political activism among targetedcitizens. Pre/post activity.Index: joining a party, volunteering serviceto party, serving as party poll watcher,contributing to party, campaigning for party,attending party functions, running as partycandidate for office, talking to friends aboutpolitics, contacting public officials,attending political rallies, etc.

Increased political activism isindicative of an informed andcommitted electorate.


Cost: High. Conducting a panel survey of targetedindividuals over time is preferred but likely to be moreexpensive and complicated than conductingcomparable national samples.

Increase in levels of political activism amongtargeted citizens.

Comments: Increased voter turnout is not always a valid measure of an informed and committed electorate. As a country’s political situation stabilizes, turnout may decrease.

3) Party activityamong citizens asperceived bypolitical actors.

Index of party activism among targetedcitizens.

Pre/post activity.

Indicative of increased vitality ofpolitical parties as organizing agentsfor citizens.


Cost: High. See comment about panel surveys inindicator 2) above.

Increased levels of party activism amongtargeted citizens.

Comments:



Intermediate Result 2.2.4: Effective Oversight of Electoral Processes


1) Degree towhichassessments byelection monitorsare acknowledgedand received byelectoral authority.

a) International monitorsb) Domestic monitorsc) Party pollwatchers and political partyrepresentatives.Scale range: closed process/no access toopen feedback process facilitated byelection authority through public hearings,ombudsman, etc.

Openness and receptivity ofelection authority to monitoringassessments.

Oversight function served bymonitoring activities must be linkeddirectly to election authority.

Evaluation of openness and receptivity by panel ofexperts/monitors.

Cost: Low.

Increased openness and receptivity byelection authority to assessments of electoralprocess.

Comments: Missions may choose to focus on one or more of the monitoring efforts undertaken, as applicable.2) Degree towhich electionmonitors assessthe electoralprocesses overtime.a) Internationalmonitorsb) Domesticmonitorsc) Partypollwatchers andpolitical partyrepresentatives.

Length of commitment by monitoringoperations:1) One election cycle. a. Pre-election. b. Election. c. Post-election.2) Two election cycles. a. Pre-election. b. Election. c. Post-election.3) Three election cycles. a. Pre-election. b. Election. c. Post-election.4) Constant presence through more thanthree cycles.

Enduring presence of monitorsprovides continuity in oversight, asopposed to one shot approachwithout institutionalization.

Evaluation by mission/partners.

Cost: Minimal/collateral.

Replacement of international monitors withinstitutionalized domestic watchdog groups.

Comments:

3) Level of publicconfidence thatelection monitorsserve as effectiveoversightmechanisms.

a) International monitorsb) Domestic monitorsc) Party pollwatchers and political partyrepresentatives. Public response to statement:“election monitors are effective at makingsure that the election authority fulfills itsresponsibility to administer free and fairelections." Strongly agree, stronglydisagree (5 point scale).

Public support of and confidence inmonitors serve to legitimize theirrole in the electoral process.

Survey research/focus groups.


Increased public confidence in electionmonitors as effective oversight mechanisms.

Comments:



4) Level ofacceptance ofelection monitorsas oversightmechanisms bypolitical actors.

a) International monitorsb) Domestic monitorsc) Party pollwatchers and political partyrepresentatives.Behavioral and attitudinal measures: 1)Assessment of the degree to whichpolitical actors seek to facilitate the workof election monitors;2) Responses to questions regardingpolitical actors’ acceptance of electionmonitoring.Scale: from 1 no acceptance, to 4 fullacceptance.

Political actors’ support of andconfidence in monitors serve tolegitimize their role in the electoralprocess.

Focus groups: Political actors’ assessments bypanel of experts (party leaders, candidates,government officials).

Cost: Moderate.

Increased acceptance of election monitors asoversight mechanisms by political actors.

Comments:

5) Level of publicconfidence thatmedia outletsserve as neutral,unbiasedwatchdogs of theelectoral process.

Public response to statement: “Mediaoutlets such as newspapers, radiostations, and television stations, serve asneutral, unbiased watchdogs of theelectoral process.”

An independent media serves as awatchdog in the electoral process.



Increased public confidence in the media asneutral, unbiased watchdogs.

Comments: Missions should make preliminary judgments regarding the capacity of the media in country to perform such a role.

6) Level ofacceptance ofmedia outlets asneutral, unbiasedwatchdogs of theelectoral processby political actors.

Responses of political actors regardingtheir acceptance of the media as neutral,unbiased watchdogs of the electoralprocess.

Acceptance of the media aswatchdog by political actors servesto legitimize their role inoverseeing the electoral process.

Focus groups - political actors.

Cost: Moderate.

Increased acceptance of media as neutral,unbiased watchdogs.

Comments: Missions should make preliminary judgments regarding the capacity of the media in country to perform such a role.



Intermediate Result 2.2.4: Effective Oversight of Electoral ProcessesIntermediate Result 2.2.4.1: Effective Electoral Process Monitoring

Indicator Definition and Unit of Measurement Relevance of Indicator Data Collection Methods/Approx. costs Target setting/Trendline interpret. issues

1) Access to allaspects of theelectoral process,from the earlieststages ofcandidaterecruitment byparties to post-election activities,by monitors.

a) Internationalb) Domesticc) Political party representatives.

Areas where “monitor access” is importantinclude access to electoral processincluding election authority, political partyorganizations, polling sites, vote counting,and tabulation processes.

Measure of level of transparency,openness of electoral process.

Evaluations by monitors.

Cost: Low/collateral. Collateral costs: costs ofmonitoring are funded under separate mechanismsso that missions do not bear such costs. Therefore,costs associated with this measure should be low.

Increased access to all aspects of electoralprocess by monitors.

Comments:

2) Percentage ofpolling placescovered bymonitors.

a) Internationalb) Domesticc) Political party representatives.Percentage of polling places covered bymonitors.

Wider coverage allows for moresystematic appraisal of electionprocess.

Mission collection of monitors’ reports.

Cost: Minimal/collateral.

Increase in percentage of polling placesmonitored.

Comments:

3) Degree ofaccess totabulationprocesses bymonitors.

a) Internationalb) Domesticc) Political party representatives.

More focused version of 1) above.Evaluation of level of access to votetabulation processes by monitors.

Scaled measure: Little or no access— totalaccess.

Indication of level of transparency oftabulation process.

Mission collection of monitors’ reports.

Cost: Low/collateral.

Increased access to vote tabulation process bymonitors.

Comments:

4) Breadth ofpublication anddissemination ofmonitor findings.

A. Breadth of publication - Number ofmonitoring sources producing reports.B. Dissemination - Breadth of circulation ofeach of the reports (in country/externally,and in local languages.)

Degree to which results ofmonitoring efforts are presented toelection authority, political actors,general public, internationalaudiences.

Assessments by mission.

Cost: Low.

Multiple monitoring sources, widelydisseminated documents.

Indicator Definition and Unit of Measurement Relevance of Indicator Data Collection Methods/Approx. costs Target setting/Trendline interpret. issues


Comments:

5) Comprehen-siveness ofassessments ofelectoral systemby monitors.a) Internationalb) Domesticc) Political partyrepresentatives.

“Comprehensiveness.” Breadth ingeographical, temporal, and issue areacoverage.

Greater comprehensiveness reflectsbroader oversight of the electoralprocess.

Assessment by mission.

Cost: Low.

Increased comprehensiveness of monitoringreports.

Comments:



Intermediate Result 2.2.4: Effective Oversight of Electoral ProcessesIntermediate Result 2.2.4.2: Media Fulfills Role as Watchdog in the Electoral Process


1) Breadth andaccuracy of mediaassessments ofthe electoralprocess.

Number of media outlets providing criticalassessments.

Assessment of accuracy of media critiques.

Broad scope of media coverageincreases ability to serve aswatchdog.

Content analysis of media documents - newspaperarticles, radio/television newscasts by trained mediaanalysts, where possible.

Cost: Moderate.

Increased breadth and accuracy of mediaassessments.

Comments:



Intermediate Result 2.2.5: Representative and Competitive Multiparty System


1) Degree towhich electorallaw facilitatesmultiple politicalparties in theelectoral system.

Electoral law, governmental statutes,constitutional provisions, electoralcommission rules and practices.

Evaluation of legal framework.

Legal framework should not impedethe development of competitivemultiparty system.

Content analysis of legal documents relating toparties and elections by experts on electoral systemdesign.


Development of a legal system, free ofconstraints on party competition.

Comments:2) Degree towhich politicalopposition isinstitutionalized inthe form of a legalpolitical party orparties.

“Institutionalization:” formal partyorganization, leadership structure,membership base, candidates running foroffices, party platforms.

Assessment of development of organizedopposition parties.

Beyond legal framework, the actualinstitutionalization of oppositionparty (ies) is important indevelopment of multiparty system.

Mission assessment.

Cost: Low/moderate.

Increased institutionalization of oppositionparty(ies).

Comments:3) Degree towhich candidatesfor office runningunder politicalparty labelsgarner the supportof voters.

Percent of votes received by candidatesrunning for office under political partylabels.

Gauge of legitimacy of politicalparties by general public.

Analysis of voting data.

Cost: Moderate/low.

Increase in percent of voters choosingcandidates running under party labels.

Comments:

4) Degree towhich recognizedpolitical partiesfield candidates atall applicablelevels ofgovernment.

Percent of candidates in national,state/regional, and local levels running foroffice under party labels.

National %; state, regional %; local %.

Assesses the level of integration ofpolitical parties at all levels ofgovernment.

Analysis of candidate registration data.

Cost: Moderate.

Increased presence of candidates runningunder party labels at all levels of government.

Comments: Nonpartisan elections may occur at local levels.



5) Degree towhich there isparty competitionin all regions ofthe country.

By electoral region. Scale of 1-5, from nocompetition to extensive competition.Report on the average score acrossregions.

Party competition throughoutcountry is important.

Analysis of election data by region.

Cost: Moderate.

Increased party competition in all regions ofcountry. Only set targets for pre-electionperiods.

Comments:6) Degree towhich competingpolitical partiesarticulate distinctprogrammaticagendas thatprovide clearchoices for theelectorate.

Agendas: Public articulation of issues andpolicies that define and organize the party,often presented in the form of a partyplatform.

Policy differences defining politicalparties rather than personalities.

Content analysis of party platforms.

Cost: Moderate.

Growing clarity of issue agendas betweencompeting parties.

Comments:

7) Degree towhich citizensdiscerndifferencesbetweencompeting politicalparties sufficientto affect theirchoice ofcandidates.

Voter assessments of party issue agendas.Voter acknowledgment of issue differencesbetween party candidates is a significantfactor in vote choice.

Perceived issue differences amongcompeting parties demonstrateprogrammatic nature of parties.

Survey research (exit polling).

Cost: High.

Increased ability of citizens to discerndifferences in issues among competing partieswith increased impact of differences on voterchoices.

Comments:



Intermediate Result 2.2.5: Representative and Competitive Multiparty SystemIntermediate Result 2.2.5.1: Political Parties Have Institutional Structures That Reflect Internal Democratic Structures and Procedures That Are Judged to Be

Transparent, Inclusive, and Accountable and That Are Accepted by the Party Leaders


1) Political partieshave writtenbylaws thatpromote internaldemocratic partygovern-ance andare adhered to byparty leaders.

See comments, below.

Unit: Number of political parties out oftarget group.

Index with each element scaled 1-5 orscore card. See Appendix C.

Internal structures of political partiesreflect democratic principles oforganization and governance.

Assessment of party bylaws, organizationalstructures, and decision-making processes.

Cost: High, but depends on number of partiescovered.

Increasingly open and transparentorganizational structures and decision makingprocesses within political parties.

Comments: Criteria for evaluating adequate bylaws and appropriate internal party meetings include:a. The political party has clear functional (i.e., administration, finance, political, training, field) and geographic (i.e., local, regional, national) divisions that are defined on paper by an organizational

chart or functional structure, such that staff and officers understand responsibilities.b. The political party has bylaws, which:

i. establish rules for the party at national, regional, and local levels;ii. govern different organizational functions and decision-making processes;iii. are agreed upon by the membership or member representatives, registered (if necessary) and implemented by party leaders and members.

c. The political party has periodic, democratic elections of party officers at every level with an effort to give a voice to the membership.d. The political party selects party officers at every level with efforts made to nominate and elect candidates that reflect the diverse membership of the party.e. The political party holds regularly scheduled party congresses and/or conventions with standing committees elected or appointed at each level according to adopted bylaws.f. The political party announces political party meetings with venue, time, and date according to bylaws and makes available to party members reports of meeting results.g. At party meetings, congresses, and/or conventions, the political party discusses and debates issues, party positions, and/or platforms openly and tolerates differing opinions such that decisions

are adopted based on processes that are transparent, acceptable to members/leaders, and according to established party bylaws.h. The political party reflects in its bylaws, or through other internal procedures, mechanisms for periodic review of performance, whereby member recommendations for improvement are

incorporated.



2) Degree towhich politicalparties allow forbroadmembershipparticipation indrafting of partyplatforms.

Membership participation:a. The political party arranges for themembership to debate and approve theparty platform according to establishedparty rules and on a regular basis.b. The political party produces draftplatforms and suggested amendments thatreflect a degree of consultation withmembers, party constituent groups, and/orregional and local party leadership.

Assesses the degree to which eachpolitical party has a mass base thatcontributes to the policy debatewithin the party.

Assessment of internal decision-making processes ofindividual political parties.

Cost: Moderate/high. (Access to internal activities ofpolitical parties may be difficult.)

Increasingly participatory decision makingprocesses within political parties.

Comments:

3) Degree towhich politicalparties haveinternalcommunicationstructures thatpromote two-waycommunicationbetween partybranches andheadquarters andreflect acommitment totransparency andaccountability.

Internal communication structure:See Comments below.

(Scorecard or index: Measures for each offive components.)

Unit: Number of political parties out oftargeted group.

Assesses the degree to whichinternal party communicationsreflect the informational role ofleadership and the feedback role ofmembership.

Analysis of internal communication structure by panelof experts.

Cost: Moderate/high. (Access to internal partyactivities may be difficult.)

Increased communication between partyleadership and membership.

Comments: Expected internal communication structures would include:a. The political party demonstrates party activity that originates and is coordinated effectively from both the national and local levels, where appropriate.b. The political party has established systems for regular, structured communication between the organization at all levels (e.g., leadership tours, newsletters, regularly scheduled meetings).c. The political party has members that are informed of party initiatives and decisions and are incorporated into activities.d. The political party has an established system for regular coordination, interaction, and participation with its elected officials to advance the party’s and the elected officials’ goals.e. The political party has effective and enforced internal financial controls.



Intermediate Result 2.2.5: Representative and Competitive Multiparty SystemIntermediate Result 2.2.5.2: Political Parties Have Established Functioning Political Party Administrative Structures That Advance Institutional Stability in the

Long-term


1) Percentage oflocalities orregions in whichpolitical partieshave offices orrepresentatives.

Unit of measure: Number of localities orregions.

Measure of breadth ofinstitutionalization of political parties.

Analysis of party records/verification of local offices.

Cost: Low.

As parties develop, increased coverage ofcountry with party representation.

Comments:2) Degree towhich politicalparties maintainwritten docu-mentation oforganizationleadershippositions andresponsibilities aswell as a definedcom-mandstructure tofacilitate timely,well-informeddecision-making.

Permanent record of political partyorganization and activities (existence ofdocumentation).

Measure of permanence of politicalparty.

Analysis of party documentation.

Cost: Low.

As parties develop, increased presence offormal documentation of party organization,leadership, and activities.

Comments:

3) Degree towhich politicalparties have well-trained staffand/or volunteerstaff support.

Number of paid and volunteer staffers inparty headquarters/regional offices.

Level of permanent staffing: paidand volunteer.

Assessment of party work force.

Cost: Moderate.

Comments:



4) Degree towhich politicalparties haveestablishedleadershipdevelopmentprograms.

Presence/absence of leadershipdevelopment programs.

Presence = Number of future leaderstrained per year.

Sustainability of party leadershipthrough development programs.

Analysis of leadership development programs.

Cost: Low/moderate.

Increased efforts at leadership development bypolitical parties.

Comments:5) Political partieshave establishedinternalstaff/volunteertraining programs.

Presence/absence of staff/volunteertraining programs.

Presence = Number of staff/volunteerstrained per year.

Professionalization of workforcethrough training programs.

Analysis of staff/volunteer training programs.

Cost: Low/moderate.

Increased professionalization of partystaff/volunteers through training programs.

Comments:6) Political partiesestablish andmaintain planningdocuments suchas annual partyplans,organizationalbudgets, andresourceallocationmechanisms.

Presence/absence of planning documentswithin party organization.

Long-term planning/partydevelopment as indicator ofpersistence of parties.

Analysis of party planning documents.

Cost: Low/moderate.

Increased attention to long term planning byparties.

Comments:7) Political partiesmaintain internalpublic policyresearchinstitutions.

Presence/absence of research capacitywithin political party organization.

Ability to formulate and analyzepolicy positions, analyze electorateand party membership serves toenhance ability of party to planstrategies for elections.

Analysis of party organization structure for researchcapacities.

Cost: Low/moderate.

Increased ability of parties to plan strategicallythrough the use of research produced internally.

Comments:



Intermediate Result 2.2.5: Representative and Competitive Multiparty SystemIntermediate Result 2.2.5.3: Increased Institutional Capacity of Each Political Party to Identify, Represent, and Expand its Defined Constituency in the

Electorate


1) Percentage ofvoters who cancorrectly link agiven party with itsbasic principlesand key issues, aswell as identifyindividualcandidates (of thegiven party).

Survey responses: Party with issue agenda,Party with candidates.

Demonstration of degree to whichelectorate is informed regardingpolitical choices available.

Assumes parties have clear issueagendas or platforms.

Survey research/focus groups, general public.

Cost: High.

Increasingly informed electorate.

Comments:

2) Extent to whichpolitical partiesconductdemographicresearch and/orelectoral historyon each electoraldistrict and applyresults to allappropriatefunctions.

Demographic research/voting history:Assessment of parties’ abilities to conductresearch and analyze voting behavior.

Measurement of abilities of partiesto assess electoral behavior and toact on this analysis to solicitadditional voters to their parties.

Assessment of party organizations to discern abilitiesto conduct or pay for public opinion research foragenda-setting purposes.

Cost: Moderate.

Increased ability of parties to analyze electoralbehavior and to use information to further partygoals.

Comments:



3) Extent to whichpolitical partiesunder-take orcom-missionresearch toidentify andprioritize constit-uencies’ concernsand present policyoptions to partyleadership.

Opinion research by parties:Agenda setting through analysis ofconstituency preferences.

Measurement of abilities of partiesto assess voter opinions throughsurvey research/focus groups.

Assessment of party organizations to discern abilitiesto conduct or pay for public opinion research foragenda-setting purposes.

Cost: Moderate.

Increased ability to conduct or commissionpublic opinion research in order to formulateparty issue agendas.

Comments:

4) Extent to whichpolitical partiesestablish andmaintain accuratemembership lists(separatedaccording toadministrativeand/or electoraldivisions) that areupdated regularly.

Membership data bases: Information onparty members including voting records,organized by electoral district/voting units.

Assessment of party membership lists.

Measurement of parties’ abilities totrack members, to inform themthrough party communications, andto mobilize them at the time ofelections.

Assessment of party membership lists.

Cost: Low/moderate.

Increased ability of parties to maintaincomprehensive membership lists and to usethem for electoral advantage.

Comments:

5) Extent to whichpolitical partiesconduct periodicmembershipdrives or otherrecruitmentmeasures toincreasemembership.

Membership recruitment:any party efforts to build and maintain amembership base. This could includenumber of membership drives, number ofcitizens solicited.

Assessment of party recruitment efforts.

Measurement of parties’ abilities torecruit and maintain a membershipbase.

Assessment of party recruitment efforts.

Cost: Low/moderate.

Increased recruitment efforts by parties.

Comments:



6) Extent to whichpolitical partiesorganize andmaintainauxiliaries forwomen, youth,and/or othergroups at local,regional, andnational levels.

Auxiliaries: Segmented groupings withinparties organized to maintain and mobilizedistinct constituencies within parties. Thiscould include number of auxiliaries ornumber of members.

Assessment of party auxiliaries.

Measurement of parties’ abilities toidentify and support distinctconstituencies within partyorganization.

Assessment of party auxiliary structures.

Cost: Low/moderate.

Increased auxiliary structures within partyorganizations.

Comments:



Intermediate Result 2.2.6: Inclusion of Women and Disadvantaged Groups


1) Degree towhich electorallaws are free oflanguage thatlimits the abilitiesof women anddisadvantagedgroups toparticipate fully inelections, both asindividual votersand as membersof organizedpolitical parties.

Disadvantaged group:any group which, in a given country, hashistorically been excluded from fairparticipation in political processes.

Extent to which statutory frameworkprovides protection for or facilitatesthe inclusion of women and/ordisadvantaged groups in the politicalprocess.

Content analysis of electoral laws/regulations—assessment of degree of discriminatory languagerelating to women/disadvantaged groups.

Cost: Low.

Increased statutory reform to eliminatediscriminatory language that limits the abilities ofwomen and/or disadvantaged groups fromparticipating in society.

Comments:

2) Degree towhich politicalparticipation ofwomen andmembers ofdisadvantagedgroups mirrors thelevel of politicalparticipation of thegeneralelectorate.

Political participation may be defined toinclude voting, running for public office,serving in party leadership, contactingelected officials, volunteering in politicalorganizations.

Equity in participation by womenand disadvantaged groupsdemonstrates equal access topolitical system.

Assessment of participation levels: survey research;electoral administration; and party records.


Increased levels of participation by women andmembers of disadvantaged groups.

Comments:


Agency Objective 2.2: More Genuine and Competitive Political ProcessesIntermediate Result 2.2.6: Inclusion of Women and Disadvantaged GroupsIntermediate Result 2.2.6.1: Laws Pertaining to Elections and Political Processes Provide for Non-discrimination Against Women and Disadvantaged Groups

Indicator Definition and Unit ofMeasurement

Relevance of Indicator Data collection methods/Approx. costs Target setting/Trendline interpret. issues

1) Level ofimprovements in lawsand regula-tionspertaining to electionsand political processesthat openly disc-riminateagainst women anddisadvantaged groups,in com-parison withplanned improvements.

Discrimination: anything that detersor restricts participation in politicalprocesses.

Comparative assessment oforiginal statutory/regulatorylanguage adopted.

Degree to which government seeksto address and reform statutes andregulations that are discriminatory towomen and/or disadvantagedgroups.

Comparative content analysis is of existinglaws/regulations and reform proposals,laws/regulations adopted or introduced.

Cost: Low/moderate.

To the degree that discriminatorylaws/regulations exist, increased efforts atreforming existing legislation/regulations toeliminate discriminatory practices.

Comments:

2) Level ofimprovements inproviding penal-tiesagainst discrimination,in comparison withplanned improvements.

Penalties: Sanctions identified inanti-discriminatory laws/regulationsthat are created/improved uponand enforced.

Beyond existence of anti-discriminatory laws/regulations,extent to which such governmentprovisions include enforceablesanctions that are imposed.

Assessment of utility of anti-discriminatorylaws/regulations.

Cost: Moderate.

Increased enforcement of anti-discriminatorylaws/regulations.

Comments:

3) Degree to whichelectoral laws facilitatethe participation ofwomen and mem-bersof disadvan-tagedgroups as candidatesfor office and as voters.

Electoral law language thatidentifies women and/ordisadvantaged groups specificallyand offers mechanisms, rights, etc.that facilitate their participationand/or representation in electoralprocesses.

Assessment of degree to whichelectoral law not only does notdiscriminate, but also providesmechanisms to facilitate theparticipation and/or representationof women and/or disadvantagedgroups.

Content analysis of electoral laws—assessment oflanguage to discern degree to which women anddisadvantaged groups are aided in electoralprocesses.

Cost: Moderate.

Increased electoral law provisions that protestand facilitate women and disadvantagedgroups’ electoral behavior.

Comments:


Agency Objective 2.2: More Genuine and Competitive Political ProcessesIntermediate Result 2.2.6: Inclusion of Women and Disadvantaged GroupsIntermediate Result 2.2.6.2: Women’s and Disadvantaged Groups’ Legal Rights Are Protected Through Effective Enforcement of Non-discriminatory Laws

Pertaining to Electoral and Political ProcessesIndicator Definition and Unit of Measurement Relevance of Indicator Data collection methods/Approx. costs Target setting/Trendline interpret. issues

1) Nature ofcomplaints filedregardingdiscriminationagainst womenand/or disadvan-taged groups.

Complaints registered by or on behalf ofwomen and/or disadvantaged groupsagainst the government, electoralauthority, political parties, and privateorganizations for violations of existinglaws.

Measurement of noncompliancewith existing non-discriminatorylaws.

Assessment of formal complaints registered withproper legal authorities.

Cost: Moderate (access).

Decline in the number and severity of formalcomplaints as government/authorities moreeffectively enforce existing laws.

Comments: “Nature:” quality and quantity. Again it is important to establish a baseline regarding relative repression, ability to voice complaints on discrimination.

2) Percentage ofcomplaints regardingdiscrimination thatare remedied byproperlegal/regulatoryauthorities.

Complaints that are judged to have meritand subsequently remedied byappropriate authority.

Measurement of level ofenforcement of existing non-discriminatory laws.

Analysis of complaints/rulings by appropriateauthorities.

Cost: Moderate.

Greater compliance with laws should result in adecline in valid complaints.

Comments: Complaints against government, electoral authority, political parties, and private organizations.

3) Percentage ofcomplaints about theenforcement systemregistered byorganized interests(NGOs) representingwomen and/ordisadvantagedgroups.

Organizations filing complaints on behalfof women and/or disadvantaged groups.

Measurement of degree to whichenforcement of existing non-discriminatory laws is challenged byinstitutionalized politicalorganizations.

Assessment of complaints.

Cost: Moderate.

Increased organized political representation byNGOs in protecting rights of women and/ordisadvantaged groups.

Comments:

4) Percentage oftargeted groups whoacknowledge thatrights are protected.

Organizational/individual responses toquestions regarding satisfaction withenforcement of laws.

Level of acceptance of enforcementof existing non-discriminatory laws.

Interviews with organization leaders.Survey research: Women/disadvantaged groupmembers.


Increased satisfaction with protection of rights.

Comments: Mission identifies targeted organizations representing women and/or disadvantaged groups or targeted audiences representing women and/or disadvantaged groups.



Intermediate Result 2.2.6: Inclusion of Women and Disadvantaged GroupsIntermediate Result 2.2.6.3: Increased Participation by [Targeted] Women and Disadvantaged Groups on Election Day


1) Percentage ofeligiblewomen/eligiblepersons ofdisadvantagedgroups registeredto vote [comparedwith target goal].

Voting age women/members ofdisadvantaged groups registered tovote/voting age population ofwomen/members of disadvantaged groups.

Degree to which women/membersof disadvantaged groups aremobilized to register to vote.

Analysis of government data on population of countryand voter registration data.

Cost: Moderate/low.

Increased voter registration among targetedgroups of women/members of disadvantagedgroups.

Comments: Target goals for voter registration established by mission.

2) Percentage ofregisteredwomen/personsfromdisadvantagedgroups that vote,[compared withtarget goal].

Voter turnout among targetedgroups/registered voters in targetpopulation.

Degree to which targeted votersactually vote.

Analysis of overall turnout patterns/turnout levelsamong targeted groups.

Cost: Moderate/low.

Increased voter turnout among targeted groupsof women/members of disadvantaged groups.

Comments: Target goals for voter turnout established by mission.



Intermediate Result 2.2.6: Inclusion of Women and Disadvantaged GroupsIntermediate Result 2.2.6.4: Electoral Administration Is Free from Bias, Impartial in its Oversight, and Devoid of Discrimination Against Women and

Disadvantaged Groups


1) Level ofrepresentation ofwomen andmembers ofdisadvantagedgroups in electionadministrationworkforce,including voterregistrationworkers, pollwatchers,domestic electionmonitors, and theelectoralcommission itself.

Number or percent of women/number ofmembers of disadvantaged groups inelection administration workforce/totalnumber citizens in election administrationworkforce (could be divided by type ofwork).

Representation of women and/ormembers of disadvantaged groupsin administration of elections.

Employment data for election administration dividedby gender/disadvantaged groupings.

Cost: Moderate.

Increased representation of women andmembers of disadvantaged groups in electionadministration workforce.

Comments:

2) Degree towhich registrationand polling sitesare as accessibleto women anddisadvantagedgroups as theyare tomen/dominantgroups

Accessibility process of registration andvoting unconstrained by physical barriers,threats, intimidation targeted at womenand/or disadvantaged groups.Reports of inaccessibility bywomen/disadvantaged groups.

Identification of isolated andsystematic efforts to constrainaccess to electoral processes bywomen and members ofdisadvantaged groups.

Assessment of reports of access to registration andvoting by women and members of disadvantagedgroups being limited.

Cost: Moderate/low.

Increased access to registration and votingprocesses by women and members ofdisadvantaged groups.

Comments:



3) Degree towhich adjustmentsare made toinsurepolling/registrationsites are moreaccessible.

Number of substantively different types ofcomplaints registered.

Number of remedies presented by electoralauthority.

Assess response to reports ofinaccessibility in electoral process.

Assessment of actions taken by electoral authority toremedy complaints registered regardinginaccessibility.


Increased responsiveness of electoral authorityto complaints of inaccessibility by women andmembers of disadvantaged groups.

Comments:



Intermediate Result 2.2.6: Inclusion of Women and Disadvantaged GroupsIntermediate Result 2.2.6.5: Effective Voter Education Provided to Facilitate Women’s and Disadvantaged Groups’ Understanding of and Ability for Political

Participation


1) Number ofwomen/disad-vantaged reachedby voter educationmessages,compared totarget goal.

Total number of women/members ofdisadvantaged groups contacted/educatedby activity compared to target goal.

Breadth of coverage of activity intargeted audience groups.

Assessment of dissemination plans: documentsprinted; television/radio commercials purchased;viewership/listenership by target audiences.


Activity specific/target goal expectation thatfuture activities will reach additional numbers intarget audiences.

Comments:

2) Number ofmaterials spec-ifically gearedtoward inclusionof women anddisadvantagedproduced andwidelydisseminated,compared totarget goal.

Number of written documents, audio andvideo materials produced, trainingsessions; circulation ofdocuments/materials for targeted womenand disadvantaged groups.

Breadth of coverage of educationalmaterials in targeted audiencegroups.

Tracking of number of written documents and othermaterials.

Tracking of circulation of documents/materials.

Cost: Moderate.

Activity specific. Target goal expectation thatfuture activities will reach additional numbers intarget audiences.

Comments:

3) Level ofincrease inpercentage oftarget population’sunderstanding ofkey messages,compared totarget goal.

Political knowledge of targeted messageprior to activity among target audiencemembers/understanding of message bytarget audience members after activity.

Measurement of understandingamong targeted audiences followingactivity.

Survey research: pre-test/post-test.

Cost: High.

Activity specific/target goal expected increasesin future activities.

Comments:



4) Number ofwomen/disadvantaged whoreceived electioneducation training,compared totarget goal.

Number of participants in electioneducation training/target goal.

Breadth of training within targetedaudience groups.

Attendance records at training sessions.

Cost: Low.

Activity specific/target goal dependent uponfunding of activity.

Comments:



Intermediate Result 2.2.6: Inclusion of Women and Disadvantaged GroupsIntermediate Result 2.2.6.6: Political Parties Are Supportive of the Participation of Women and Disadvantaged Groups in Political Processes


1) Percentage ofpoliticalcandidates whoare women ormembers ofdisadvantagedgroups.

Number of candidates who arewomen/members of disadvantaged groupsas a proportion of the total number ofcandidates in particular election.

Political representation ofwomen/members of disadvantagedgroups among political candidates.

Analysis of candidate lists.

Cost: Low.

Increase in the percentage of women andmembers of disadvantaged groups running foroffice.

Comments:

2) Level of equityof financialsupport of womenand members ofdisadvantagedgroups ascandidates bypolitical parties.

Measurement = average party spending onfemale and disadvantaged candidatesdivided by average party spending on allcandidates. Scaled 0-25% of average forall candidates spent for women/marginalcandidates = very low; 25-50% = low; 50-75% middle; 75-100% = optimal.

Degree to which party allocations offinancial resources to candidates isbiased against women andmembers of disadvantaged groups.

Assessment of financial records of parties andcandidates (access and availability — potentialproblems).


Increasingly equitable funding for all candidatesby political parties.

Comments: Aggregated with all party spending combined or analysis of allocation within each party.

3)Women/disadvantaged candidatesuccess rates vis-a-vis allcandidates.

Number of successful women anddisadvantaged candidates/number of totalwomen and disadvantaged candidatescompared with the total number ofsuccessful candidates/total number of allcandidates.

Measurement of comparativesuccess of women and members ofdisadvantaged groups.

Assessment of election outcomes for all candidates.

Cost: Low/moderate.

The trendline used in this indicator should startfrom the record of the previous election(s).

Increased success of women and members ofdisadvantaged groups.

Comments:

4) Number ofmajor politicalparties with anti-discriminationregulations.

Language in party bylaws providingsafeguards against discrimination ofwomen and members of disadvantagedgroups.

Explicit statements againstdiscrimination reflect attentivenessto issue by parties.

Content analysis of party bylaws/governingdocuments.

Cost: Low.

Increasing number of political parties adoptinganti-discriminatory language in bylaws.

Comments:



5) Percentage ofwomen/disadvantagedwho are membersof major politicalparties, comparedwith thepercentage of allcitizens who aremembers of majorparties.

Party membership by gender; analysis byparty as well as aggregation: overall partymembership by gender and disadvantagedgroups.

Identification of presence orabsence of bias in partymembership.

a. Analysis of party membership lists.

Cost: Moderate (access problem).

b. Voter registration lists by party identification.

May be difficulty in identifying of members ofdisadvantaged groups.

Decline in gender/disadvantaged imbalances inparty membership.

Comments:

6) Number/percentage ofwomen/disadvantaged inparty leadershippositions.

Party leadership positions: toporganizational positions in each party.

Degree of representation ofwomen/disadvantaged groups inparty leadership.

Analysis of party leadership records.

Cost: Low.

Increased representation ofwomen/disadvantaged groups in partyleadership.

Comments:7) Number ofparties withplatforms that donot include issuesthat discriminateagainst womenanddisadvantaged.

Party platforms: Organizational and/orpolicy language in governing documentsthat discriminates againstwomen/disadvantaged groups.

Degree of non-discrimination bypolitical parties.

Content analysis of party platforms.

Cost: Low.

Decline in discriminatory language againstwomen and minorities.

Comments:



Intermediate Result 2.2.7: Effective Transfer of Political Power


1) Newly elected/appointed officialstake office,replacingincumbents.

Seating of new parliament/legislaturefollowing elections without incident.

Measurement of peaceful transferralof political power.

Analysis of the transfer process/media accounts.

Cost: Moderate.

Context specific: Each successive electionshould result in more effective transfer ofpower.

Comments:



Intermediate Result 2.2.7: Effective Transfer of Political PowerIntermediate Result 2.2.7.1: Procedures for the Transfer of Power Are Established and Followed


1) Steps fortransfer of powerare set forth inpublic document.

The procedures for handing overgovernment office or governmental powerare pre-established, written, and accessibleto the public.

A valid indication of whether power-transferral procedures areestablished with their existence inpublicly accessible documents.

Analysis of public documents relating to transferprocess.

Cost: Low

Establishment and maintenance of formalframework/structure in law for political transferof power.

Comments:

2) Degree ofconsensus amongsignificant politicaland/or militaryactors on theprocedures fortransfer of power.

Acceptance of transfer procedures byparty/NGO/military/bureaucracy leadership.Unit: numbers of actors.

Necessity for wide consensus onthe mechanisms for transfer ofpolitical power.

Interviews, press reports.


Context specific: increased consensus of themechanisms of the political transfer process.

Comments:



Intermediate Result 2.2.7: Effective Transfer of Political PowerIntermediate Result 2.2.7.2: Newly Installed Officials Are Prepared to Fulfill Their Responsibilities


1) Degree towhich a start-upagenda isdeveloped andimplemented bynewly elected (orappointive)deliberativebodies andexecutiveagencies.

Four-point scale:1 = not at all developed2 = developed3 = developed and communicated4 = implemented

Measurement of the preparednessof new government to governfollowing elections.

Interviews with new political leaders/assessment oftransition period and initial period of governance.


Context specific: successive transfers willproduce more transfer agendas.

Comments:2) Degree towhich credibledomestic sourcesfind that newofficials areprepared to fulfilltheir newresponsibilities.

Credible domestic sources: NGO leaders,press accounts, party leaders.

Four-point scale:1 = not at all2 = minimally3 = largely4 = completely

Degree to which transfer processjudged to be acceptable by majoropinion leaders.

Interviews with political leaders/NGO leaders.

Media assessments of preparedness of new officials.

Cost: Moderate.

Increased credibility of transfer process insuccessive post-election cycles, as judged bymajor opinion leaders.

Comments:3) Degree towhich orientationprograms exist fornewly installedofficials.

Orientation programs: Trainingseminars/briefing for new members toprepare them for their new positions ingovernment.

1 = no orientation2 = one or fewer training sessions3 = several training sessions4 = comprehensive orientation programs

Measurement of preparation ofnewly elected officials for thelegislative/governmentalresponsibilities.

Assessment of orientation programs.

Cost: Moderate.

Strengthened orientation processes insuccessive post-election cycles.

Comments:



Intermediate Result 2.2.7: Effective Transfer of Political PowerIntermediate Result 2.2.7.3: Agencies of Government, Including Military and Security and Opposition Groups, Accept the Authority of Newly Installed Officials


1) Degree towhich governmentagencies performtheir duties inaccordance withduly constitutedauthoritiesfollowing atransfer of power.

Four-point scale:1 = refusal to follow orders2 = passive resistance3 = partial accommodation4 = thorough accommodation

Acceptance of political transfer bypermanent bureaucracy is importantin legitimizing new government.

Analysis of bureaucratic responses to electionresults/new political leadership.



Increased acceptance of electoral outcomes bygovernment agencies/bureaucracies throughperformance of required governmentalactivities.

Comments:

2) Degree towhich oppositiongroups/parties(former govern-ment leaders)participatevoluntarily in newparliament/legislature/government.

Participation: Taking seats won inelections.

No boycotts of legislative proceedings/noextra-institutional means of blockinglegislative process.

1 = no cooperation/total boycottto4 = full cooperation with new government.

Acceptance of results of election byopposition forces is crucial to thetransfer of power.

Interviews/assessments of actions of oppositiongroups/parties and their leaders.

Cost: Moderate.

Increased cooperation by opposition forces innew governments.

Comments:

3) Degree towhich militaryaccepts theauthority of newlyinstalled officials.

Four-point scale:1 = refusal to accept electoral results2 = moderate resistance3 = only slight resistance4 = acceptance and accommodation.

Acceptance of electoral results bymilitary is important to peacefultransfer of power.

Analysis of military response to electoral results.

Interviews/media accounts.

Cost; Moderate/high.

Increased acceptance of authority of newlyinstalled officials by military.

Comments:



Intermediate Result 2.2.7: Effective Transfer of Political PowerIntermediate Result 2.2.7.4: The Public Recognizes the Legitimacy of the Process by Which New Officials Are Chosen


1) Degree towhich citizenryaccepts legitimacyof newly installedbodies and/orofficials.

Citizenry acceptance: Public support fornewly elected officials and new institutionsof governance.

Public acceptance of newgovernment through electoralprocess is a crucial element inpeaceful transfer of power.


Assessment at public response to new government

Increased public acceptance of electoraloutcomes and resulting new government.

Comments:

2) Degree towhich relevantpolitical actors,includingoppositionleaders, acceptthe outcome ofthe transferprocess.

Relevant political actors: Partyleaders/NGO leaders/military elite supportfor electoral outcomes and resulting newgovernment.

Elite support is also important tolegitimacy of new politicalinstitutions.

Press accounts/interviews/focus groups.


Increased support by major political actors ofelectoral outcomes.

Comments:

Section C: Civil Society

Section C: Civil Society 115

Results Framework


Definitions — Civil Society

Agency Objective 2.3. Increased Development of a Politically Active Civil Society

Citizens organizing to accomplish shared objectives constitute a vital force for the circulation of informationand the representation of interests. The collective nature of civic action helps ensure that the interests ofcitizens, including women, disadvantaged minorities, and the poor, are weighed by public institutions thatmake policy and allocate resources. Civil society organizations (CSOs), which include free, independent, anddemocratic trade unions, can monitor government performance and demand leadership accountability bothat national centers and in various localities. CSOs can inculcate democratic values, give people practice withdemocratic principles, and create opportunities for new leaders to arise. The following results contribute tothe achievement of this Agency objective.

Intermediate Results 2.3.1. A Legal Framework to Protect and Promote Civil Society Ensured

The emergence and growth of civil society require an enabling legal and regulatory environment. Among theprerequisites are a body of fundamental laws and rules that permit the right of association, including the rightfor workers to organize and bargain collectively, encourage citizens to join together for a variety of otherpurposes, promote volunteerism and charitable contributions, protect communications media and informationnetworks, and generally ensure autonomy from state interference.

(Sub) Intermediate Result 2.3.1.1. Strengthened Advocacy for Legal and Regulatory Reform

Strengthened advocacy can make an important contribution to achieving a better legal framework.CSOs may need to develop their capacity to analyze the policy environment and to advocateeffectively for new or amended laws and regulations to encourage civil society organization.Strengthened advocacy may involve the formation of advocacy coalitions in which CSOs cometogether to analyze problems, formulate solutions, and undertake joint reform efforts.

(Sub) Intermediate Result 2.3.1.2. Increased Public Support for Needed Reforms

The public must understand, support, and participate in CSO efforts to reform the system. This, inturn, implies a two-way exchange between the CSO community and the general public.

Intermediate Results 2.3.2. Increased Citizen Participation in the Policy Process and Oversight ofPublic Institutions

Democratic governance implies popular participation, including by disadvantaged social groups, in bothpublic policy-making and its implementation. In this process, civil society performs its classic role ofcomplementing, and sometimes countervailing, the state. By promoting and protecting civil rights, CSOsensure that citizens have the means to express their preferences, engage in dialogue with policy-makers, andaffect decisions in the public realm. After policies have been chosen, CSOs perform as watchdogs of stateperformance by demanding accountability in the allocation and management of public resources. Byperforming selected public governance functions, CSOs can also lighten the state’s burden and reduce theconcentration of resources.

118 Section C: Civil Society

(Sub) Intermediate Result 2.3.2.1. Improved CSO Advocacy

In order for CSOs to intervene effectively in the policy formulation or reform process, they must gainor strengthen the advocacy skills of their organizations. Such skills run the gamut from simplycollecting information on the subject at hand to such other tasks as obtaining or allocating humanand fiscal resources to advocacy functions, coalition and network building, taking action to influencepolicy, and monitoring implementation once a policy decision has been made.

(Sub) Intermediate Result 2.3.2.2. Increased Openness of Public Institutions to CSOInvolvement in the Policy Process

Public institutions must be open to CSO involvement in all phases of the policy process, includingformulation, implementation, and oversight. For this to occur, mechanisms must exist for theexchange of information and views on the issues targeted. Such mechanisms might include, forexample, public hearings, multi-sectoral boards or consultative committees, the appointment of CSOrepresentatives to official decision-making bodies, and so forth.

(Sub) Intermediate Result 2.3.2.3. Increased Political Participation of Groups RepresentingMarginalized Populations

The expression “marginalized populations” refers to women, ethnic, and other disadvantaged groups,who in many countries have not been part of the traditional mainstream that has benefitted fromdevelopment efforts. For this reason, these disenfranchised groups have tended not to participate inthe political process, nor have they learned the advocacy or monitoring skills needed to represent orsafeguard their own interests.

Intermediate Results 2.3.3. Increased Institutional and Financial Viability of CSOs

Whatever their specialized tasks, CSOs generally seek to improve their institutional effectiveness. To do so,they introduce organizational reforms and professional procedures into both internal management systems(e.g., for programs, personnel and finances) and external relations (e.g., with clients, governments, anddonors). They aim to supplement the value of commitments and voluntary efforts of their members withfund-raising plans that would reduce donor subventions and lead to long-term financial self-reliance. At thesame time, CSOs increasingly rely on participatory management techniques in a bid to embody the samedemocratic values and principles that they promote in the wider society.

(Sub) Intermediate Result 2.3.3.1. Improved Financial Management Systems

To build credibility and strengthen overall management capacity, CSOs must incorporate reliablesystems for the efficient management of their finances and for reporting in a transparent manner theirfiscal status to members, donors, and other interested parties. Indeed, accurate accounting recordsand favorable audit reports often become key tools for achieving and consolidating organizationalrecognition and influence.

(Sub) Intermediate Result 2.3.3.2. Improved Fund-raising Techniques

To achieve organizational independence from any given donor while solidifying their financial base,it is important for CSOs to develop comprehensive fund-raising plans and to assign to appropriate


staff or volunteers the responsibility of carrying out those plans. Depending on circumstances, suchplans might include, for example, individual contributions, institutional donations, or funds producedby the CSO’s own income-generating activities (i.e., sale of services or products).

(Sub) Intermediate Result 2.3.3.3. Increased Participatory Management

The operation and image of CSOs as credible, legitimate voices for the citizens they claim torepresent are directly linked to the manner in which they manage themselves. Increasingly, CSOsunderstand the wisdom of substituting participatory practices for traditional hierarchical orauthoritarian management styles if, indeed, they are to build constituencies and be seen to “practicewhat they preach.” Such practices may include, for instance, the establishment of mechanisms forsoliciting input from constituents on programming or operational matters, the involvement of staffin the decision-making process, and the election of board members.

(Sub) Intermediate Result 2.3.3.4. Improved Management Systems

Another facet of CSO institution-building efforts involves the incorporation of effective and efficientmanagement and administrative systems. This includes the process of strategic planning as an on-going and participatory task, the establishment of monitoring and evaluation systems withappropriate data-collection techniques, and improved personnel management systems.

(Sub) Intermediate Result 2.3.3.5. Improved External Relations

In addition to strengthening internal management practices, it is important for CSOs to developeffective outreach mechanisms. Such mechanisms involve a two-way flow of information andrelations with the public. On one hand, these organizations need to create ways to assess client needsas input for programming decisions, and on the other they need to report on their activities andaccomplishments through community meetings, media reports, and so on.

Intermediate Results 2.3.4. Enhanced Free Flow of Information

In order to formulate interests and participate in policy debates, citizens must have access to a wide rangeof information. A set of independent and competent media institutions is key to providing citizens withinformation and to revealing abuses of power. Beyond the formal media, citizens can engage in informalnetworks of communication, through simple technologies like telephones and photocopiers or the moresophisticated innovations available on the information superhighway. A plural array of private andindependent sources of information is extremely difficult for non-democratic governments to control.

(Sub) Intermediate Result 2.3.4.1. Plural Array of Independent Sources of InformationEncouraged

For citizens to participate effectively in matters that affect them, it is important that they have accessto a wide variety of diverse, independent information sources. These include both the electronic andprint media (i.e., television and newspapers), as well as newsletters or bulletins published by relevantCSOs. Where there are sub-populations speaking minority languages, steps are also needed to ensurethat those groups are reached by providing programming and print periodicals in those languages.Access to such other independent instruments as telephones, fax machines, and e-mail is alsoimportant for information-sharing and consensus-building.


(Sub) Intermediate Result 2.3.4.2. Improved Investigative Reporting

Information may flow freely but will be of limited value if it is of poor quality and can be harmfulif it is sensationalist. Increasing media attention to data collection, reporting of facts, and analysisand interpretation of data improves the quality of media information.

(Sub) Intermediate Result 2.3.4.3. Increased Use of New Information Technologies

New information technologies, which run the gamut from e-mail to use of the internet and thecreation of home pages on the internet, are vital instruments for enhancing the viability andeffectiveness of CSOs over time.

(Sub) Intermediate Result 2.3.4.4. Improved Financial and Management Systems in MediaEntities

To ensure their own financial viability, independent broadcast and print media cannot rely merelyon subscriber sales but must usually increase advertising revenues. In addition, improved accountingsystems and cost controls, along with increased capital investment in equipment, may contribute tofiscal stability and overall sustainability.

Intermediate Results 2.3.5. Strengthened Democratic Political Culture

The consolidation of democracy depends on the acceptance by both citizens and political elites of a sharedsystem of democratic norms and values. These include tolerance of political diversity, interpersonal trust,and a sense of political efficacy on the part of individuals. A democratic political culture also requires thatcitizens have knowledge about their system of government.

(Sub) Intermediate Result 2.3.5.1. Expanded Higher Quality Civic Education in Schools

In many countries, formal education remains rooted in outmoded or autocratic pedagogical modes.This is particularly true in newly-emerging or developing democracies, where repressive ordictatorial regimes may have used the education system as a tool for withholding information fromthe citizenry or as a way to assert control over the thoughts and actions of the population. In thosecases, official curricula did not include the study of democratic principles or other aspects of civiceducation.

(Sub) Intermediate Result 2.3.5.2. Expanded Higher Quality Informal Civic EducationInitiatives

In building democratic political culture, in addition to providing civic education in formal classroomsettings, it is important to educate the adult population about democratic practices.

(Sub) Intermediate Result 2.3.5.3. Community-based Civic Action ProgramsExpanded/Initiated

Experience has shown that, to develop and sustain a democratic political culture over time, it isusually necessary to work from the bottom up by initiating civic action programs at the community


level. Such programs are designed to engage the active participation of community members ininitiatives aimed at meeting their needs and which they themselves help identify, such as petitiondrives, cases brought before appropriate judicial authorities, public hearings, lobbying localgovernment officials, or regional regulatory agencies, etc.







4. Approaches to collecting data for the indicator and the approximate cost of collection. Threecategories were used for cost: low (under $500); medium/moderate ($500 - $1500); and high (over$1500). The cost specifications are rough and should be treated cautiously, because costs may varygreatly from one country to the next. The cost of data collection for questions requiring surveys of thepopulation (for example) will depend upon 1) the existence of periodic surveys which address the desiredtopics conducted by someone else; 2) the existence of surveys which do not address the desired topic butto which strategic objective teams could add questions; and 3) the need to conduct one’s own surveys andthe number of indicators for which that survey can serve as the data collection approach. Similarly, thecost for monitoring broadcasts or periodicals for specific content will depend on whether there is 1) acommercial monitoring service, which conducts monitoring for advertisers anyway, so fees can be verymodest; 2) an NGO that uses this information for its own purposes anyway; or 3) a need to cover theentire cost of monitoring several periodicals and/or broadcast stations.

5. Issues related to target setting and the interpretation of trendlines. In this final column, we share whatwe have learned about how much progress might be made over what period of time. In some cases, suchas changes in the level of political tolerance, we know very little. In other cases, it seems clear that anindicator should change sharply and immediately in the wake of particular interventions. We also discussissues to be aware of in setting performance targets or in interpreting trendlines.



Indicators

Agency Objective 2.3: Increased Development of Politically Active Civil Society

Intermediate Result 2.3.1: A Legal Framework to Protect and Promote Civil Society Ensured

Note: CSO stands for “civil society organization.”

Indicator Definition and Unit of Measurement Relevance of Indicator Data Collection Methods/Approx. Cost Target Setting/Trendline Interpret. Issues

1) Laws supportfreedom ofassociation andspeech.

Yes/No.Improvements made against a list ofproposed reforms; qualitative reviewagainst criteria.

Shows the degree to which the lawupholds the right of citizens toorganize.

Review of the Legal Code; data/studies by citizenorganizations & universities

Cost: Low, assuming data already exist.

Depending on country context, a list ofproposed reforms and monitoring of progresstoward their achievement

Comments: See Appendix C, Part D for example of a scale for measuring the progress of a legal reform.

2) a) % of targetCSOs that thinkthe registrationprocess is:1. timely (or hasbecome moretimely, etc.);2. transparent; &3. low-cost. ORb) general CSOsense thatprocess is timelyor untimely,transparent or not,low or high cost.

The degree to which a group of targetCSOs think/perceive the registrationprocess to be: 1) timely; 2) transparent;and 3) low-cost. Each element could belaid out as a scale and, if desired, acomposite index produced.

Shows the level of governmentacceptance of/support for citizens toorganize to represent their interests.

Data collection from existing CSOs (registered &unreg.) via a sample survey. If key informantinterviews or focus groups are used, then apercentage cannot be produced but only a sense ofhow difficult/easy registration is.

Cost: Can be low or high. Depends on whether thereis an existing survey instrument on which to tackthese questions, or whether informal methods areselected.

If the percentage is low to begin with, we wouldhope for improvement over time. If thepercentage is reasonable, it may stay relativelystable.

Comments:

3) $ cost ofregistration perCSO.

Financial cost of registering;application/other fees charged per CSO(includes informal payments).

(Same as above) Govt. data, plus cross check with key informants in arange of CSOs or sample survey of CSOs.Cost: Depends on existence and reliability ofgovernment data.

A simplified process should reduce costs. Theremay be a one-time reduction in costs and not agradual decline. Costs should be low enough toencourage registration of CSOs.

Comments:



4) Average lengthof time forregistration.

a) Average # months to register; OR b)length of time to register is high (over 1year), medium (6-11 months), low (under 6months).

Shows whether government hindersCSO organization.

Government data (sample or registrations) or samplesurvey of CSOs or key interviews with NGOs.

Cost: low to high depending on data collectionapproach.

A simplified process should reduce the numberof steps and therefore the time required toregister. There may be a one-time improvementrather than a gradual decline. A factor to beanalyzed is whether those groups with longestregistration time are the most political oroutspoken on human rights or other particularlysensitive issues.

Comments: b) may be better if data is collected via focus groups.

5) Laws permitCSOs to raisefunds/income.

a) Yes/No; or b) Laws improved compared to a list ofreforms promoted.

Shows level of support for citizenorganization & provides insight as toCSOs’ prospects for self-sustainability.

Qualitative review of laws or of legal reforms made.

Cost: Low but depends on the complexity of the lawor the reforms made.

We would hope to see an increasing number ofreforms made, but the potential impact of thereforms is more important than the sheernumber.

Comments: See Appendix C, Part D for legal reform milestone scale.

6) Tax lawsfavorable toCSOs.

a) Improvements made/constraintsremoved compared to a list of reforms; orb) % of CSOs that say tax laws are:1. unfavorable; 2. favorable to raisingmoney; or 3. effective rate of taxation.

Shows level of support for citizenorganization & provides insight as toCSOs’ prospects for self-sustainability.

Qualitative review of laws or of legal reforms made.

Cost: Low but depends on the complexity of the lawor the reforms made.

We would hope to see an increasing number ofreforms made, but the potential impact of thereforms is more important than the sheernumber.

Comments: See Appendix C, Part D for legal reform scale.



Intermediate Result 2.3.1: A Legal Framework to Protect and Promote Civil Society EnsuredIntermediate Result 2.3.1.1: Strengthened Advocacy for Legal and Regulatory Reform


1) # out of targetCSOs advocatingfor legal &regulatory reform

# out of target group of CSOs involved inadvocacy vs. service delivery; # out of totaltarget group.

A higher number of very weakCSOs advocating is obviously notnecessarily better than a lowernumber of strong ones. However, arising number may mean that morediverse constituencies are beginningto be heard. This needs to be paredwith a qualitative indicator looking atthe substance of the advocacy, suchas the index in Appendix C.

Gather info. from CSOs re type of activitiesundertaken. A CSO federation or association mayalready have that info. Depending on countrycontext, state of CSO and funds available, thecollection method can be anything from keyinformant interviews to focus groups to a more formalsurvey.

Cost: Low-moderate

Would hope to see an increase if very few areinvolved in advocating reform.

Comments:

2) # of advocacyinitiatives carriedout by CSOcoalitions for legalreform.

# of advocacy initiatives carried out jointlyby two or more organizations in CSOcoalitions to promote legal reform.User needs to define what constitutes an“initiative”.

Shows amount of advocacy activity. Gather info. from CSOs re type of activitiesundertaken. A CSO federation or association mayalready have that information. Depending on countrycontext, state of CSO and funds available, thecollection method can be anything from keyinformant interviews to focus groups to a more formalsurvey.

Cost: Low if involved CSOs keep adequate records.

Establish a baseline of the # of times two ormore organizations in a CSO coalition havejointly carried out advocacy initiatives; monitorincreases (shows more advocacy) anddecreases (may show weakened advocacycapacity by the CSO community as a whole, ordecreased need for advocacy) over specificperiods of time.

Comments:

3) # of targetCSOs active inadvocacycoalitions.

# of target CSOs active in advocacycoalition(s) for legal reform.

Shows breadth of support. Gather info. from CSOs re type of activitiesundertaken. A CSO federation or association mayalready have that info. Depending on countrycontext, state of CSO and funds available, thecollection method can be anything from keyinformant interviews to focus groups to a more formalsurvey.Cost: Low

Establish a baseline of the # of CSOs inadvocacy coalitions; monitor increases (showsmore advocacy) and decreases (may show“defeat" or “victory") over specific periods oftime.

Comments:



4) # of targetCSOs showingimprovement onthe advocacyindex or reachinga certain level ofexpertise on theindex.

Index — see Appendix C of the handbook,as applied to legal framework for civilsociety.

Shows improvement in advocacyskills of target CSOs.

See Appendix C. Cost depends on # of CSOs.

Comments: Index can be adapted to fit country conditions and particular program emphasis.



Intermediate Result 2.3.1: A Legal Framework to Protect and Promote Civil Society EnsuredIntermediate Result 2.3.1.2: Increased Public Support for Needed Reforms


1) % of pop.aware of CSOlegal reformconcerns.

% of population that has knowledge ofCSO messages re type of legal reformneeded.

Knowledge is a prerequisite tosupport, and informed support ismore useful than uninformedsupport.

Public opinion surveys conducted by professionalfirms.Cost: Depends on whether there is a pre-existingsurvey to which questions can be attached. If asurvey needs to be conducted for this (and related)indicators, cost will be high.

Targets should be set in accord with CSO plansfor public education on various legal reformissues.

Comments:

2) % of pop.supportive of CSOlegal reforminitiatives.

a) % of general population that agreeswith CSO reform agenda;

b) # responses CSOs have affirmingsupport; or

c) # of citizens participating in supportingsuch initiatives in some manner(volunteering, contrib. cash, etc.).

a) Surveyb) and c) Documents from CSOs showing affirmativeopinions/responses (including protests,demonstrations, strikes, etc.) and interviews/trackingof government officials & legislators.Cost: Low if can add questions to a pre-existingsurvey. If no survey is done and CSOs are trackingresponses and supporters, costs will be medium tohigh.

Should increase if CSO advocacy sparksinterest.

Comments:

3) % positivearticles/segmentsin key publicationsand broadcastprograms.

Articles or segments should discuss legalreform in positive terms.

Proxy for public support, when it istoo expensive to collect informationfor indicator 1) and 2).

Commercial monitoring service would be leastexpensive. Monitoring by involved CSOs alsopossible. Cost: Low-medium.

May be difficult to solicit much media interest inthis issue. Could be a flurry of articles/segmentsas the issue gains visibility, and then a drop offas media moves on to other topics.

Comments:



Intermediate Result 2.3.2: Increased Citizen Participation in the Policy Process and Oversight of Public Institutions


1) Public policieschangedconsistent withCSO advocacy.

#/type of policies/laws changed inaccordance with CSO advocacy agenda(or proposed negative changes defeated).

The connection between actualpolicy outcomes & CSO advocacy istricky. Compromises occur sorelating advocacy input to policyoutcome can be difficult. Only ifthere are policy changes of somekind is it possible to conclude thatcivil society is more influential andplaying a role in decision making.

CSO monitoring of legal/regulatory changes adoptedafter (and in keeping with their advocacy efforts)have begun. Data from CSO records, reports bylegislature & government admin. units, interviews

Cost: Low. One would assume that if CSOs have anadvocacy agenda, then they are monitoring progress.Other info. can be collected from officiallegislative/govt. records and interviews.

Some policy debates have very long life spanswith no resolution for years, so this may notalways be a good indicator for a limited strategyperiod. Would expect gradual rising trendline asgovernment becomes more accustomed toworking with CSOs and CSOs become moresophisticated.

Comments: The number is potentially deceptive. One change could have more significant impact than 15 changes. A qualitative review of the import of those changes must accompany this indicator.

2) % or # of targetCSOs that saythey can obtainneededinformation fromkey publicagencies.

# out of target group or % of target CSOsthat they can obtain needed informationfrom key counterpart government agencieson a scale of 1 (never); 2 (rarely);3 (sometimes); 4 (usually); 5 (always)could also present average score acrossCSOs.

Proxy indicator for idea thatinformation is necessary for CSOsto exercise oversight.

Sample survey required if % desired; otherwise,interviews with CSOs, back up documentation.

Cost: Low-moderate if using # out of target group orpre-existing survey of CSOs exists. High if need togenerate the survey, but cost depends on the totalnumber of CSOs to be interviewed.

Improvements may be so gradual that groupswill not easily notice change. Legal andregulatory changes dealing with informationrelease could have the biggest and mostnoticeable impact on this indicator.

Comments:

3) # out of targetgroup or % ofCSOsrepresentingmarginalizedgroups thatbelieve/candocument impacton thepolicy/oversightprocess.

# out of target group or proportion of CSOsadvocating issues related tomarginalization and have organizedmarginalized groups to be actively involvedin advocating for legal and regulatoryreforms. Standards for documenting impactcan be set according to localcircumstances.

Caution should be exercised indefining the universe understood tobe “marginalized." This will varyfrom one place to another; criteriaand a definition should be set beforeattempting to attributerepresentation or results.

Requires sample survey of groups represent themarginalized if proportion desired. Otherwise,interviews with target groups. Backup documentationof impact should be requested.

Cost: Low to high depending on number of groups tobe interviewed.

Hope to see growing number of NGOs havingimpact on policy. Growth may be slow.

Comments: This indicator is an alternative to counting the number of policies changed.



4) % of publicknowledgeableabout or aware ofan issue.

Proportion of the general public thatunderstand the various argumentsassociated with a particular issue.

Getting an issue on the publicscreen is an important contribution.

Public opinion surveys done periodically to measurechanges in awareness/knowledge of an issue.

Cost: High but depends on the number of indicatorsincluded in the survey.

Mass media outreach should lead to a muchhigher percentage.

Comments:

5) # of targetedissues which arereceivingheightened publicattention.

An issue is increasingly treated by thepress; policy makers address the issue;legal/regulatory policy is formulated toaddress the issue.

While it is difficult to delimit thesesigns, they might include a govt.commission to examine the issue,parliament holding hearings, aministry making a study orbeginning to talk about the issuepublicly.

Information collected on the frequency of reportageby media; #/type of meetings/speeches/policyinitiatives by policymakers; # of laws/reg. policiesdrafted or in discussion.

Data collected from CSOs, the mediagovt./legislative offices on frequency/perceivedimportance of issue; sample survey or focus groupswith CSOs, reporters, legislators, business sector,govt., etc.

Cost: Medium, but depends on the number of issuesbeing tracked.

Comments: Indicator could be constructed as a critical events agenda, scale, or index, or a quantitative indicator concerning media coverage, speeches, or public meetings.



Intermediate Result 2.3.2: Increased Citizen Participation in the Policy Process and Oversight of Public Institutions Intermediate Result 2.3.2.1: Improved CSO Advocacy


1) # of CSOs outof target groupundertakingadvocacyactivities for thefirst time.

Advocacy standard should be definedmodestly but to suit the local environment.

Proxy indicator. Advocacy indexpreferred. Good for places wherecivil society has been completelyflattened. Less useful where civilsociety is well developed.

CSO records and reporting, interviews with CSOstaff.

Cost: Low to medium, depends on number of CSOs.

Hope for steadily increasing trendline.

Comments:

2) # of CSOsshowingimprovement onthe advocacyindex or reachinga certain level ofexpertise on theindex.

See Appendix C, Part D-1 for the index. Index measures various aspects ofadvocacy.Where the index is simply too labor-intensive, other proxies could beused, such as: a) # of pressconferences/press releases bytarget CSOs; b) # of meetings withMPs/staff.

See Appendix C, Part D-1. Cost depends on numberof CSOs.

Should be very sensitive to improvementsgrowing out of training, TA, or experience.

Comments: Index can be adapted to fit local conditions or particular program emphasis.



Intermediate Result 2.3.2: Increased Citizen Participation in the Policy Process and Oversight of Public InstitutionsIntermediate Result 2.3.2.2: Increased Openness of Public Institutions to CSO Involvement in the Policy Process

Indicator Definition and Unit of Measurement Relevance of Indicator Data Collection Methods Target Setting/Trendline Interpret. Issues

1) Newmechanismsestablished.

#/type new mechanisms created, includingpublic hearings, appointment of CSO reps.on government bodies, etc.

Measures foundation for openness. Collect information from CSOs, govt. or legislativerecords

Cost: Minimal. CSOs with advocacy agendaspresumably collect this information.

Initial increase, can be expected to plateauafter a time.

Comments:

2) Frequency ofuse of newmechanisms, for aset of targetissues.

a) # of times CSOs use new mechanisms;b) # of opportunities to take advantage of

new mechanisms (# of hearings,commissions, etc.);

c) Total # of CSOs participating in theseopportunities. These could becalculated for a list of target issues.

b) Measures openness.a) & c) Measure extent to whichCSOs take advantage of openness.

Collect info. from CSOs on frequency/# of times theyhave participated in/appealed through newmechanisms.

Cost: Low - moderate since CSOs may need to trackthis info.

Comments:

3) CSO/otherperception/opin. ofwillingness ofpublic institutionsto engage indialogue.

% of CSOs; or # CSOs (& other organ.) outof target group whose leaders perceivethat public institutions (including admin. &regulatory agencies & legislature) are: a)always; b) usually; c) sometimes; d) rarelyor never open to dialogue.

Care is needed with this indicator,since perception may not alwaysmatch reality.

Sample opinions/perceptions of CSO leaders andkey observers (journalists, legislators, businesspeople, educators, etc.) through surveys, keyinformant interviews, or focus groups.

Cost: Cost can be moderate but depends on thenumber to be interviewed.

Increases in perception may lag behindimprovements in openness.

Comments:



Intermediate Result 2.3.2: Increased Citizen Participation in the Policy Process and Oversight of Public InstitutionsIntermediate Result 2.3.2.3: Increased Political Participation of Groups Representing Marginalized Populations

(Note: “Marginalized groups” includes women, ethnic, other disadvantaged sub-populations)


1) # of groupsrepresentingmarginalizedconstituenciestrying to affectgovernment policyor conductingoversight.

# of groups representing the “marginalized”active in advocacy/monitoring work. (Usersof this indicator will need to set someparameters regarding the amount & qualityof the advocacy.)

Survey a sample of the marginalized groupsidentified to see if they undertake political advocacy. Cost: Depending on # of “marginalized” groups anddata they collect, can be moderate to high.

We would not expect a constantly risingtrendline. Few strong groups may be better thanmany weak groups.

Comments:

2) % ofmainstream CSOleadershippositions held bymarginalizedgroups.

Average proportion of CSO officers andkey staff positions held by persons frommarginalized groups. (This should excludeCSOs created by/made up of persons frommarginalized groups.)

Proxy for how well integratedmarginalized groups are intomainstream CSO activity.

Survey of CSOs/review of their documentation to: a)identify # “leadership” positions (both volunteer &paid); and b) how many are held by persons fromgroups identified as “marginalized.”Cost: Moderate to high, depending on depth of CSOcommunity and # of groups identified as“marginalized.”

We would anticipate increases from a very lowbase but the trend may eventually level off.

May increase very slowly in ethnically dividedsocieties.

Comments:



Intermediate Result 2.3.3: Increased Institutional and Financial Viability of CSOs


1) % of targetCSOs that reducetheir dependencyon their largestsingle donor.

Proportion of target CSOs that either: a)decrease funds from major source w/outany loss of revenue; or b) increase othersources so that largest source provides areduced % of revenue.

This indicator is not relevant whenthere is not a single dominant donorcreating a potential dependency.Sign of financial viability. Ability toattract donors.

Survey of CSOs, annual reports.

Cost: Low to medium depending on the number ofCSOs.

In some very difficult environments,improvement may be slow and modest.Thresholds for a) and b) i.e., decrease fundsfrom major source by how much, can be setaccordingly.

Comments:

2) # of targetCSOs withincreased%/amount ofrevenues fromindigenoussources, includingself-generatedincome.

# of target CSOs receiving increasedproportion of revenues from local sources(user may wish to set a threshold percent,i.e., a 5% increase per annum).

Measures financial viability. Survey of CSOs, CSO documents.

Cost: Low to medium depending on number ofCSOs.

% of revenue should increase over time but if anNGO is successful in attracting new foreigndonor resources, this % could stay flat or evenfall. If this possibility is foreseen, replace the %of revenue with the total amount of revenue indollars from indigenous sources.

Comments:



Intermediate Result 2.3.3: Increased Institutional and Financial Viability of CSOsIntermediate Result 2.3.3.1: Improved Financial Management Systems


1) # of targetCSOs with nomajor auditfindings.

# of target CSOs that: a) have annual orbiennial audits in accordance with locallaw; and b) have no major criticisms(findings).

If financial management systemsare strong, audits should find onlyminor problems.

Target CSO financial records; auditor opinion letterscontained in audit reports.

Cost: Low to moderate—depends on # of targetCSOs & how frequently they are audited.

Set targets for years in which audits areexpected.

Comments:

2) # of targetCSOs thatimplement audit recommenda-tions.

# of target CSOs with audits thatimplement recommendations.

Shows commitment to goodfinancial management.

Same as above, plus review of auditrecommendations implemented.

Cost: Moderate to high, depends on number of targetCSOs and frequency of audits.

Set targets for the year after and are expected.

Comments:

3) # of targetCSOs withimproved financialaccountingpractices.

# of target CSOs that improve or introduce:a) adequate expense records;b) double-entry bookkeeping; c) internal controls, including approval of

expenditures;d) production of financial statements on a

regular basis.

Proxy indicator—small CSOs maynot be audited all that regularly.

Target CSO financial records, interviews. Should respond directly to TA and training inaccounting.

Comments: Could turn this into a scorecard, scoring each element as 0 (no) or 1 (yes) and showing the total score. Scores for all CSOs could be averaged, or the number of CSOs reaching a set scorecould be reported. Scorecard elements may need to be adjusted to local conditions.



Intermediate Result 2.3.3: Increased Institutional and Financial Viability of CSOsIntermediate Result 2.3.3.2: Improved Fund-raising Techniques

Indicator Definition and Unit of Measurement Relevance of Indicator Data Collection Methods Target Setting/Trendline Interpret. Issues

1) # of targetCSOs with a)increased # ofsuccessfulincome-producingactivities or b)increased incomefrom existingincome generatingactivities.

# of target CSOs witha) increased activities successfully

undertaken for purpose of raising funds(i.e., there is some net profit); or

b) increased income from existing incomegenerating activities. If desirable, canset a threshold for the increase — i.e.,5% per annum.

Result of improved techniques andthus a higher level proxy.b) could measure improvedmanagement rather than the IRspecified (i.e., better costcontainment, greater efficiency).

Examination of target CSO fund-raising activities andtheir balance sheets. Availability of data may be anissue; it may depend on whether CSO activities aretaxed.

Cost: Low to moderate, depends on # of targetCSOs and # of activities they undertake.

Increases will be gradual in very difficultenvironments.

Comments:

2) # of targetCSOs withincreased # ofindividualcontributions andinstitutionaldonations.

# of target CSOs with increasedcontributions and donations fromindividuals and institutions. Establish abaseline of: a) # of target CSOs; and b) #of individual contributors and institutionaldonors for each.

Shows decreased dependency onfew donors, increased membershipbase, and diversification of revenuesources. Result of improvedtechniques and thus a higher levelproxy.

Examination of target CSO financial data.

Cost: Low to moderate, depends on # of targetCSOs; # of individual contributors & institutionaldonors; and quality of their financial records

Should rise with concerted effort, but may leveloff after a period of time. Tax environment willaffect this indicator.

Comments:

3) # of targetCSOs with fund-raising plans inplace and beingimplemented.

Plans need to be realistic in expectationsfor raising revenues in that environment.Plans should exist in writing, and staff andvolunteers should understand and supportthe plan.

Partial indicator, measures onetechnique.

CSO plans. Should respond directly to TA and training onfund-raising.

Comments: May need to relax criteria if working with community associations.



Intermediate Result 2.3.3: Increased Institutional and Financial Viability of CSOsIntermediate Result 2.3.3.3: Increased Participatory Management


1) % of targetCSOs withmechanisms forstaff involvementin decision-making in use.

% of target CSOs that make regular use ofa variety of mechanisms (staff meetings,etc.) to involve staff in decision making.’Regular’ needs to be defined.

Involvement of staff is one sign ofpartic. management.

Document review (i.e., CSO personnel policies;procedures, etc., if they exist); interviews with CSOleaders and staff.

Cost: Moderate, depends of # of target CSOs,#/openness of leaders and staff.

Increases should result from training and TA butgood intentions could trail off after a while.

Comments:

2) % of targetCSOs solicitinginput fromconstituencies.

% of target groups holding annualmeetings, etc. to solicit feedback fromprimary constituencies on a regular basis.“Regular” needs to be defined.

Involvement of constituencies is animportant sign of participatorymanagement.

CSO bylaws & reports/materials; interviews withCSO representatives.

Cost: Moderate, depends on # of target CSOs.

# of target CSOs that solicit constituent inputshould increase over time. Care is needed insetting an appropriate standard for input.

Comments:


Agency Objective 2.3: Increased development of politically active civil societyIntermediate Result 2.3.3: Increased Institutional and Financial Viability of CSOsIntermediate Result 2.3.3.4: Improved management systems


1) # of targetCSOs withstrategic plansbeingimplemented.

# of target CSOs that:a) have strategic plans; and b) implement them (benchmarks are on

track).

Interviews with target CSO leaders and staff, reviewof target CSO reports, other documents.

Cost: Moderate; depends on # of target CSOs,openness of leaders & staff.

Some caution is needed in setting the standardfor performance. If the standard for strategicplanning is set very high, then targets should bemodest. If the standard is set low, then targetscan be higher. It is key to figure out what isreasonable in a given environment.

Comments: See also the index for strategic planning, which includes 3 factors: 1) CSO has a clear vision; 2) CSO has multi-year strategic plan; 3) CSO monitors plan’s progress and continuing applicability.This is in Appendix A, p.15.

2) # of targetCSOs that: a)have monitoring &evaluationsystems; and b)collect/useresulting data.

Standards or criteria need to be set forwhat constitutes an acceptable M&Esystem and what represents adequatedata collection and use.

Gather info. from target CSOs re existence or not ofM&E systems and how resulting data are used bythose that do collect it. M&E plans and interviews.

Cost: Moderate; depends on # of target CSOs andthose that have M&E systems.

Some caution is needed in setting the standardfor performance. If the standard for M&E is setvery high, then targets should be modest. If thestandard is set low, then targets can be higher.It is key to figure out what is reasonable in agiven environment. Use of the data will be themost difficult step, requiring the most TA.

Comments: a) and b) could be turned into indexes with criteria laid out, and 1-5 points awarded for each criterion.

3) # of targetCSOs withimprovedpersonnelmanagement.

Number of target CSOs with a) personnel policies and procedures in

writing,b) job descriptions,c) staff knowledgeable about policies and

procedures,d) staff doing what their job descriptions

say. This could be a scorecard with 0 (no) or 1(yes) point awarded for each element andthe score totaled. It could also be an indexwith scaled responses of 1-5 pointsawarded for each element and a totalproduced. Elements can be changed to fitlocal circumstances.

CSO records and interviews with select staff.

Cost: Medium but depends on number of groups.

Score card may show quicker change but beless sensitive to improvements in quality overtime than an index.

Comments:



Intermediate Result 2.3.3: Increased Institutional and Financial Viability of CSOsIntermediate Result 2.3.3.5: Improved External Relations

Indicator Definition and Unit of Measurement Relevance of Indicator Data collection Methods/Approx. Costs Target Setting/Trendline Interpret. Issues

1) # of targetCSOs that assesscommunity/clientneeds in aparticipatorymanner (involvingthe clients) in: a)planning activities(services andadvocacy); and b)monitoring.

# of target CSOs that seek community/client input to identify priorities and conductplanning and monitoring activities. Criteriafor input will need to be developed to fitlocal circumstances.

Shows whether target CSO plansand activities are based on theopinions of the few or on thepriorities identified by the clientpopulation.

Interviews of CSO leaders & staff, as well as with arange of clients (focus groups, key informantinterviews).

Cost: Moderate to high; depends on # of targetCSOs and area of outreach (i.e., how manycommunities; where they are located; # of peoplereached, etc.).

Some caution is needed in setting the standardfor performance. If the standard for involvementis set very high, then targets should be modest.If the standard is set low, then targets can behigher. It is key to figure out what is reasonablein a given environment.

Comments:

2) # of targetCSOs holdingcommunitymeetings on aregular basis toreport on planningdecisions andprogress.

# of target CSOs that convene/sponsormeetings with community members on anon-going basis to provide feedback andstatus reports on their activities.“Regular basis” should be defined to suitlocal conditions.

Variant to indicator 1); easier tocollect data. Measures one-way flowof information.

Review of target CSO documents; interview targetCSO leaders & staff; survey members ofcommunities where target CSOs hold meetings.

Cost: Low to moderate; depends on # of targetCSOs holding community meetings, where, howoften, etc.

Comments:

3) # of targetCSOs that initiatemedia reports ontheir activities/accomplishments.

# of target CSOs that disseminate info. ontheir activities/accomplishments throughthe media (news releases, radio/TVinterviews, etc.). The indicator shouldestablish a standard for how often thisshould be done.

Review target CSO documents; interview target CSOleaders and appropriate staff; interview key mediarepresentatives.

Cost: Low to moderate; depends on # of target CSOleaders & staff and media reps. to be contacted.

Some caution is needed in setting the standardfor performance.

Comments:



Intermediate Result 2.3.4: Enhanced Free Flow of InformationIndicator Definition and Unit of Measurement Relevance of Indicator Data collection Methods/Approx. Cost Target setting/Trendline Interpret. Issues

1) % ofpopulation: a)listening to radio;b) watching TV; c)reading the news.

% of population receiving news(disaggregated by political interest, gender,geog. & ethnic group). TV should bedisaggregated from radio and print media;often TV is controlled by government.Index of consumption a-c, scaled asfollows 1) never; 2) less than once perweek; 3) once per week; 4) 2-3 times perweek; 5) 4-5 times per week; or 6)everyday.

Shows extent of population reachedby different information channels.

Review any existing data from news outlets(newspapers, radio, TV, UNESCO, World Bank, localsurveys, etc.); if none available, survey a sample ofpopulation in diff. strategic areas, taking care toinclude targeted groups.

Cost: Moderate to high; depends on existence ofdata and expanse of population to be surveyed.

Increases in the % of the population reading thenews may depend on literacy and affordability ofpublications; access to TV may also be anissue; radio is usually the most accessible,utilized source of information. Rising educationlevels may affect readership size.

Comments:

2) % of populationthat trustsavailable newssources.

% of population (disaggregated by abovegroups) that say they trust news sources(disaggregate sources, as above).

Trust is thought to correlate with thequality of media reporting andanalysis.

Survey a sample of population in difficult strategicareas, taking care to include targeted groups (shouldbe part of data collected for above indicator).

Cost: This may be high; data likely to exist in veryfew countries. Lower cost if questions can be addedto existing surveys.

Trust may not increase at the same rate or tothe same degree as actual improvement in themedia. We know little about setting targets forthis indicator.

Comments:

3) # of law suitsfor slander or libelagainst mediaorganizations forcriticizinggovernment orthose with closeties to govt.

# of times media organizations are sued forcriticizing govt. or those close to it.

Proxy indicator — provides anindication of the severity of govt. orgovt.-linked attacks on mediaorganizations.

Records/interviews with key media organizations;court records.

Cost: Low to moderate, depending on # of mediaoutlets contacted.

Frequency of suits should decrease over time.Severity of suits also matters; one suit thatthreatens a media entity with near bankruptcyand jail time may serve to increase self-censorship by most media entities.

Comments:

4) # of incidents ofviolence targetingjournalists.

# of times journalists are the victims ofviolence or attempted violence.

Proxy indicator; high levels ofviolence will cause journalists andmedia entities to exerciseself-censorship.

Same as above; police reports.

Cost: Low to moderate, depending on # of journalistsand visibility of violent activities.

Frequency of violence against journalists shoulddecrease over time.

Comments:



Intermediate Result 2.3.4: Enhanced Free Flow of InformationIntermediate Result 2.3.4.1: Plural Array of Independent Sources of Information Encouraged


1) # of non-govt.news sources.

# of private sector news sources that exist(electronic media, press, etc.).

Care should be taken in using thisindicator. Non-governmentalsources may be very partisan andcould be extremist.

Collect info. on existing media outlets.Cost: Low; such data usually exist in govt. ministry(where media are regulated) or univ. If not, the # ofoutlets is not hard to establish.

Should increase but will respond to degree ofopenness of govt..

Comments: Newspaper circulation & radios per capita data available from State of the World Atlas.

2) # of targetCSOs publishingbulletins.

# of target CSOs that have publications fordissemination (i.e., newsletter, periodicreports, special letters/papers on theiractivities, etc.). A standard should be setfor the expected frequency ofdissemination: i.e., 2 times per year, etc.

Gather information from target CSOs

Cost: Low; this info. should be readily available fromtarget CSOs.

Linked to financial viability of NGOs.

Comments:

3) # of a)telephones; b) faxmachines; c) e-mail subscribersper capita forgiven level ofGNP.

# and type of independent sources of info.per capita by level of country’s GNP.

Telephone per capita data avail. in The Book ofWorld Rankings by George Thomas Kurlan; Faxmach. & mobile cellular phones per capita dataavailable in UNDP’s Human Development IndexReport, for 83 & 54 countries, respectively; Info. oncomputers and computing power per capita avail. inLester Brown’s State of the World report, or moredirectly from Computer Industry Almanac, Inc.Cost: Low depending on availability of books/ref.materials in country.

Comments:

4) a) # of hours ofminority languageprogramming onradio/TV; b) # minoritylanguage printperiodicals.

a) # of hours per week/month dedicatedby radio or TV stations to programmingin minority languages;

b) # of minority print periodicals appearingregularly (set a standard: i.e., at least 3times per annum).

a) may wish to disaggregate bygovt. and non-govt. sources.

Some countries have commercial monitoringservices, mostly oriented toward advertisers.Sometimes CSOs monitor the media. Also, TV andradio stations can be monitored. Cost: Low tomoderate; volunteer “listening” groups can be set upby target CSOs to track minority lang. programming,and CSOs can interview station managers.

b) increases will depend on minority populationsize and literacy, and possibly on communitywealth (which would attract advertisingsupport).

Comments:



Intermediate Result 2.3.4: Enhanced Free Flow of InformationIntermediate Result 2.3.4.2: Improved Investigative Reporting


1) % of mediaspace/timedevoted to newsanalysis.

% of total newspaper space & radio/TVtime dedicated to analysis of the news,rather than to entertainment programming.

This indicator may pose problems inhighly divided societies, where“news analysis” may beinflammatory and divisive. Does notspeak to the quality of the analysis.

Monitor newspapers and radio/TV broadcasts;interview newspaper editors/station managers.

Cost: Low to moderate; data from media outlets canbe supplemented by volunteer “monitoring” groupsset up by target CSOs. Commercial monitoringservices may also be able to produce data.

May not change rapidly.

Comments:

2) Improvedscore(s) from acontent analysisof a sample ofnews reports/articles publishedby targetedmedia.

Scoring protocols would need to bedeveloped, but might include such aspectsas: fairness to and inclusion of differentpoints of view, appropriate use offacts/figures, separation of fact fromopinion, proper attribution of facts tosources, and extent and placement ofcoverage.

Particularly relevant when a missionhas been focusing on improvedmedia or where control of media isan issue.

Sample coverage of key or controversial issues inthe country. See Appendix B.C.2. of this handbookon “Content Analysis of Press Coverage or OtherDocuments.”

Cost: Moderate - high, depending on availability ofvolunteer monitors/coders and number of mediaissues to be tracked.

Trendline should increase in response totraining of journalists, but could be retarded bylack of commitment by media owners toimproving the content of reporting.

Comments:



Intermediate Result 2.3.4: Enhanced Free Flow of InformationIntermediate Result 2.3.4.3: Increased Use of New Information Technologies


1) % of targetCSOs usinginternet.

% of target CSOs that have access to anduse internet.

Direct measure, for CSOs usingnew technologies.

Collect info. from target CSO leaders & staff.

Cost: Low; info. already exists in target CSOs.

Should increase rapidly among moresophisticated urban based CSOs.

Comments:

2) % of targetCSOs withinternethomepage.

% of target CSOs that have access tointernet & have put up homepages.

Collect info. from target CSO leaders & staff.

Cost: Low; info. already exists in target CSOs.

Comments:

3) % of targetCSOs using e-mail.

% of target CSOs that use e-mail. Shows ability of target CSO tonetwork and exchange information.

Collect info. from target CSO leaders & staff.Cost: Low; info. already exists in target CSOs.

Comments:



Intermediate Result 2.3.4: Enhanced Free Flow of InformationIntermediate Result 2.3.4.4: Improved Financial Sustainability of Independent Media Entities


1) Average % ofmedia revenuesderived fromadvertising bytarget independentmedia entities.

Level of media revenue that comes fromadvertising rather than sales or othersources in target media entities.

Advertising revenues are key tofinancial well-being.

Advertising revenues can be reliably estimated byusing a calculation of average amount of space/timedevoted to advertising times the unit cost (per lineper minute) of advertising. Commercial firms thatmonitor the media for advertising purposes makethese kinds of calculations all the time.

Cost: Moderate; depends on the existence ofcommercial monitoring firms and willingness of targetmedia entities to share financial data.

Should increase in response to training and TA.US standard for print media is that 60% ofcolumn inches is devoted to advertising.

Comments:

2) # of target mediaoutlets that show animproved balancebetween revenuesand expenditures.

# of newspapers and radio/TV stationsthat are financially self-reliant (i.e.,revenue meets or exceeds expenditures).

Direct measure of financial healthbut trendline is important. Need tolook also at relationship betweencurrent costs and capitalinvestment.

Annual reports, financial records. Access to thisinformation could be a problem.

Cost: Low, depends on number of media outlets.

Should respond to improved financialmanagement and increases in advertising.However, other factors such as spirallinginflation could affect also.

Comments:

3) % of weeks ormonths per annumwhen target mediaentities pay salarieson time.

a) % of times that target entities paysalaries on time;orb) number of target entities paying 90%salary payments on time.

Proxy indicator; reflects financialstatus of target media entities.

Collect information from target entities and staff.

Cost: Low to moderate, depending on # ofentities/staff targeted.

Salaries should be paid more punctually as theentity’s financial health improves.

Comments:

4) #/% of targetmedia entities thatmake new capitalinvestments.

#/% of target media entities that makenew investments in plant or equipment.

Proxy indicator; reflects capacity oftarget entities to afford new capitalinvestments. Capital investmentsare important to remainingcompetitive.

Target entity data, interviews.

Cost: Low

Should increase but may take considerable timein very difficult environments.

Comments:



Intermediate Result 2.3.5: Strengthened Democratic Political Culture


1) % of citizenswith civicknowledge.

% knowing selected rights andresponsibilities (freedom of speech, dueprocess, etc.); % knowing names of selected politicalreps.; % knowing functions of politicalinstitutions.

Shows whether citizens have abasic understanding of a democraticpolitical system and their own rolewithin it.

National sample of all citizens or a sub-sample ofparticipants in civic ed. programs. Also, tests shouldbe administered before and after civic ed. Anydifferences between groups must be controlled forextraneous influences like prior ed., exposure tomedia, interest in politics, etc.)Cost: High but depends on how many indicators areincluded in the survey. See Handbook Chapter III.

Targets depend on nature of intervention. Ifspecific groups are targeted for participation incivil society, increases in before-and-after testsare to be expected. But if the intervention isaimed at the gen. public, and there have beenno improvements in govt.. accountability tocitizens, then no change is expected, or theremay even be a decline. Little is known abouthow to set targets. These surveys should onlybe done at 2-3 year intervals; otherwise,changes may not exceed the margin of error.

Comments:

2) % of citizensexhibitingdemocraticvalues.

% exhibiting attitudes of political tolerance(e.g., toward ethnic, religious, and politicalpluralism); % exhibiting attitudes of political trust: a)inter-personal trust (of fellow citizens,minority groups, CSOs, media outlets); b)trust in govt. (e.g., police, magistracy, localauthorities, cent. govt. institutions); % supporting selected basic human rights.

Shows how citizens feel towardsone another and towardgovernment.Measures the level of social capital,which is held to be a prerequisite fordemocratization.

See indicator 1 for IR 2.3.5. Each country must conduct its own baseline inorder to target specific groups (those withespecially low democratic values); specificcampaigns can then be worked on for thosegroups. For example, in Nicaragua andGuatemala, small but statistically significantimprovements have been seen in recent yearsin tolerance, and in Nicaragua participation hasincreased. In both countries, democracy hasbeen consolidated somewhat over this sameperiod of time.Values are harder to change than knowledge.Little is known about how to set targets. Do notdo surveys at intervals less than 2-3 years.

Comments:



3) % of citizenswith civic skills.

% expressing a sense of political efficacy(e.g., find govt. policy processcomprehensible, feel able to influenceopinions of others, feel able to makepolitical reps. listen); % with positive self-assessment of civicskills (e.g., public speaking, letter writing,consensus building).

Shows whether citizens have thenecessary confidence andcompetence to be active citizens.

See indicator 1 for IR 2.3.5. Sense of efficacy may be difficult to affectthrough short training problems. Growingeducation levels should have an impact onovertime.

Comments:

4) % of citizensparticipating inpolitical activities.

% who discuss civic & political affairs (&how frequently); % who vote (whether registered as voter,whether voted in given elections); % involved in other forms of electoralparticipation (attend campaign rallies, workfor candidate, etc.); % who contact political reps. (e.g., localcouncillor, legislator, govt. officials; % who engage in direct political action(e.g., boycotts, demonstrations); % with experience as civic activists(belong to CSOs, attend CSO meetings,speak at meetings, organize CSOactivities).

Shows the extent of citizenparticipation in electoral and policyprocesses

See indicator 1 for IR 2.3.5. For countries that have undergone a recenttransition to democracy, voter turnout maydecline between founding and subsequentelections. The more secure the election processseems, the lower the stakes and the moreturnout may actually drop. Forms of politicalparticipation (other than voting) are known to betied to high levels of socio-economic status, sotargets in poor and illiterate countries shouldprobably be very modest.

Comments:



Intermediate Result 2.3.5: Strengthened Democratic Political CultureIntermediate Result 2.3.5.1: Expanded Higher Quality Civic Education in Schools


1) % of schoolsoffering civic ed.classes.

% of schools (primary, secondary, tertiary)offering at least one civic education class.

Shows extent of civic education inschool system.

Annual reports of Ministry of Education & privateschool association; NGO implementors.Cost: Low, assuming the Ministry of Education orimplementing NGOs are tracking.

When curriculum changes are made thatintroduce civic education, there should beconsiderable increases in the first few years.After that, there will be a leveling off.

Comments:

2) % of schoolswith electedstudent govt.

% of schools (primary, secondary, tertiary)that have elected student governments thatare active.

Indicates whether students arepracticing democratic skills in theschool environment.

Self-administered questionnaire mailed to schooladministrators or tracking by Ministry orimplementing CSOs.Cost: Low if Ministry of Education or implementingNGOs track.

Likely to increase only if civic education programincludes or encourages.

Comments:

3) % of schoolswith higher qualitycivic ed. classes.

% of schools (primary, secondary, tertiary)that:a) use participatory teaching methods;b) assign students to applied actionprojects.

Indicates whether civic ed.curriculum uses methods known tocorrelate with higher studentlearning.

Sample survey of school classes. Probably requiresobservation and not just interviews and reviews ofmaterials.Cost: Medium-high, depending on sample size.

May increase slowly in countries where rotelearning is the norm.

Comments:



Intermediate Result 2.3.5: Strengthened Democratic Political CultureIntermediate Result 2.3.5.2: Expanded Higher Quality Informal Civic Education Initiatives


1) # of citizensreached.

# of citizens by target groups (espec.women & disadvantaged minorities) whohave received informal civic ed. instruction.

Shows extent of outreach. Lists maintained by CSOs implementing civiceducation programs.

# will increase relative to funding available forcivic education.

Comments:

2) # of CSOsengaged in civiced. activities.

# of organizations (including bothregistered and informal groups) with civiced. activities.

Shows plurality of civic ed.initiatives.

Self-administered questionnaire mailed to CSOadministrators. Could probably do this more simplyby asking select CSOs to name all the groups theycan think of who do this.

Cost: Low to medium.

Number will respond at least partly to theavailability of donor funding for civic education.

Comments:

3) % of CSOs withhigher quality civiced. programs.

% of organizations that:a) use participatory teaching methods;b) assign participants to civic ed. tasks

Shows quality of civic ed. initiatives. Observation, interviews among sample of groups oramong target groups.

Cost: Medium

Increases can be interpreted as improvementsin civic ed. quality.

Comments:



Intermediate Result 2.3.5: Strengthened Democratic Political CultureIntermediate Result 2.3.5.3: Community-based Civic Action Programs Expanded


1) # of civic actionprograms takenby target CSOs.

Number brought by civic action. Shows type and variety of civicaction.

Target CSO reports and interviews

Cost: Low

May respond primarily to the availability ofdonor funding. A diversity of the types of actionmay be more important than the total number ofactions.

Comments: Programs are civic education programs which organize citizens around real problems in their community.

2) # of citizensreached by civicaction programsundertaken bytarget CSOs.

# of citizens (by target group, espec.women & disadvantaged).

Shows extent of outreach by targetCSOs

Target CSO records and interviews.

Cost: Low to moderate, depending on # of CSOs andactivities.

Increase = expanded outreach.

Comments:

3) # of targetCSOs engaged incivic actionprograms.

# of organizations with civic actionprograms.

Shows plurality of civic actionorganizations

Tracking by target groups.

Cost: Low to medium, depending on # of CSOstargeted.

Increase = more plural sector.

Comments:

4) % of targetedCSOs usinginnovativeapproaches topromote/carry outcivic actionprograms at thecommunity level.

# of targeted organizations:a) introducing new activities;b) testing untried methods

Shows innovation. Surveys or interviews at target CSOs, plusobservation/interviews/focus group sessions inparticipating communities.

Cost: Medium to high, depending on # of targetCSOs.

Increase = more innovation.

Comments:

Section D: Governance

Section D: Governance 151

Results Framework


Definitions — Governance

Agency Objective 2.4. More Transparent and Accountable Government Institutions

The behavior of formal state institutions is an essential determinant of the degree of success or failure ofdevelopmental and democratic processes. Transparency requires that governments consult broadly toascertain citizen interests, publicize plans and decisions, share information widely and in good time, andconsistently act in an open manner. Accountability depends on governments taking full cognizance of,responding to, and being monitored by, organized public opinion. Transparency and accountability asdefined here encompass the concept of responsiveness, and are served by sharing decision-making withlocal government entities (and with citizens by increasing the space for self-governance), respectingethical standards, creating a constructive relationship between civilian and military authorities, enhancingthe role of the legislature, and strengthening government performance in all stages of the policy process.Such behavior and institutional relationships support the long-term sustainability of democratic politicalprocesses and people’s confidence in democratic principles. It also makes a vital contribution topromoting development and providing an encouraging environment for economic and social investment.

Intermediate Results 2.4.1. Increased Government Responsiveness to Citizens at the Local Level

Decentralization of government authority and responsibility can increase the competence andresponsiveness of public agencies by reducing the burden on those at the center and allowing those mostaffected by an issue to make decisions about it. It enables citizens who are most directly concerned toinfluence decision-making by putting the source of the decision closer to them. The main focus of manyprograms to support democratic decentralization is on encouraging the devolution of authority to electedlocal governments that are directly accountable and therefore primarily responsible to local citizens;improving the effectiveness and openness of local governments; and increasing community involvementin local government decision-making and service delivery.

(Sub) Intermediate Result 2.4.1.1. Constitutional and Legal Reforms Devolve Power

The essential requirement for progress with devolution is that either by way of the constitution orother legal enactments, laws effectively transferring authority to elected local officials for asignificant number of clearly defined governmental functions are passed and implemented, andrespected by central government.

(Sub) Intermediate Result 2.4.1.2. Local Government Capability to Act Increased

Enabling legislation and implementation by central government will not ensure devolution ofpower. Effective devolution will occur when elected local governments have the necessary level ofqualified personnel and financial resources to have the capability to formulate, implement, andenforce policy decisions, provide services efficiently and responsible to citizen needs, andcontribute significantly to their own recurrent and capital expenditures.

(Sub) Intermediate Result 2.4.1.3. Mechanisms of Participation Increased

For decentralization to be meaningful, local governments need to be democratic. Democracy at thelocal level includes regular competitive elections, opportunities for citizen participation throughmechanisms such as town meetings and joint commissions, and the dissemination of financial and

154 Section D: Governance

other kinds of information so that the public has the opportunity to study them before decisions aremade.

Intermediate Results 2.4.2. Increased Citizen Access to Improved Government Information

Access to information is important because it allows citizens to keep a watchful eye on governmentbehavior. Perhaps even more significantly, it permits citizens to learn about government plans or actionsthat may be critical to their interests. Therefore, governments need to improve the quantity, quality andtimeliness of the information that they make available to citizens and civil society organizations. Inparticular information about budgets, financial reports, bills, laws, tenders for contracts, recruitmentopportunities and government services must be available to the public.

(Sub) Intermediate Result 2.4.2.1. Laws and Regulations Reformed

Passage and implementation of laws and regulations that require government to share informationwith the public on key issues establishes the basic standards for openness and accessibility in localgovernment.

(Sub) Intermediate Result 2.4.2.2. Improved Dissemination of Information

Governments disseminate more, better, and more timely information to the public in various ways.These include advance information of public hearings to the public, the press, and civil society;full advertisement of jobs and contracts; information on public services; and by making budgets,financial reports, and other significant documents available in user friendly form to the public.

Intermediate Results 2.4.3. Ethical Practices in Government Strengthened

Strengthening those institutional mechanisms that exist to encourage ethical behavior and preventcorruption and abuse is also important. Improved transparency as discussed in 2.4.2 above is oneimportant way of doing this. Other checks on formal state actors include civil service reform (i.e.,restructuring incentives and punishments), limits on civil servants’ discretion, strengthened audits andinvestigative functions, more effective internal procedures for enhanced oversight, improved operatingsystems in government institutions, and building a public constituency against corruption.

(Sub) Intermediate Result 2.4.3.1. Laws and Regulations Reformed

Basic standards for ethical conduct and practice can be established through a code of conduct forelected and government officials, freedom of information, and financial disclosure laws, andregulations defining corrupt practices and penalties for those who engage in such practices.

(Sub) Intermediate Result 2.4.3.2. Oversight Mechanisms to Maintain Ethical StandardsStrengthened

To oversee the conduct of civil servants, governments maintain, and provide resources for anauditor-general’s office, an inspector-general’s office, an office of government ethics, and anindependent anti-corruption agency. Effective oversight is demonstrated by regular professionalauditing according to required standards of government agencies, in “findings” (indicating errors)of such audits that lead to full investigations and, if necessary, enforcement. Ethics in government


will be enhanced further by the capacity of the legislature, the media, and specialized NGOs tomonitor and publicize corruption.

(Sub) Intermediate Result 2.4.3.3. Professionalization of Staffing and Operations

Beyond laws and oversight mechanisms, high ethical standards and practices will be determinedby the quality of the personnel and procedures of the public service. High quality is achievedthrough recruiting and promoting personnel based on competitive exams, paying competitivesalaries, establishing sound financial management systems, and ensuring transparent and impartialservice in licensing, procurement, and tax collection. Ultimately, the purpose of this moreprofessional service is less corruption, less waste, and more efficient service to citizens.

Intermediate Results 2.4.4. Strengthened Civil-Military Relations Supportive of Democracy

The military has overturned or compromised democratic rule in many developing countries. It oftencontinues to control significant financial resources and productive assets during political transitions,enabling it to exercise power independently of civilian authorities. Therefore, changing the norms of themilitary and integrating these powerful players into a political process that operates according todifferent rules is an important requirement in ensuring the sustainability of democracy. Such changerequires constructive dialogue between the military and civilians over the appropriate role for the militaryin a democratic society, respect by the military for civilian authority and enhanced civilian expertise insecurity matters. In some situations, it also depends upon the demobilization and re-integration intosociety of soldiers, particularly in states emerging from violent conflict.

(Sub) Intermediate Result 2.4.4.1. Constitutional and Legal Reform Authorize Civilian Role

The basis of civilian authority over the military and of a constructive civilian-military relationshipis to be found in legislation clarifying responsibilities and relationships.

(Sub) Intermediate Result 2.4.4.2. More Accurate Information Publicly Available

Once formal legislation is in place, it is necessary for the government to make available to thepublic, the press, and civil society organizations accurate and useful information on the military.There will be some limitations on information that may threaten national security. The test ofgovernment willingness to provide good quality information will be observed not only in thedocuments that it issues, but also in public meetings, TV and radio/shows, and in newspaperarticles analyzing the military and its role.

(Sub) Intermediate Result 2.4.4.3. Increased Civilian Competence in Defense and SecurityAffairs

Without greater civilian competence, the laws and the information alone will be of no use inimproved oversight of the role of the military. For example, if the legislature is going to analyzethe defense budget and national security policy, it needs to be qualified to do so. Therefore, forcivilian authority to be respected by the military, responsive to the nation’s security and defenseconcerns, capable of maintaining a constructive public debate, and making informed budgetaryand policy choices, it is necessary to enhance competence in these areas among legislators, themedia, and civil society organizations.


(Sub) Intermediate Result 2.4.4.4. Increased Civilian & Civil-Military Networking

In addition to making essential documentary information available to the public, governmentsshould also share information and allow for public comment by holding open meetings for thepublic and joint commissions with civil society organizations.

(Sub) Intermediate Result 2.4.4.5. Reduced Size of Military-Security Forces

Reductions in the size of military and security forces, particularly in the wake of civil conflict,reduces the potential threat of these forces to civilian control, reduces cost, and makes it easier toreorient and professionalize the military. Reintegration into society is an important part ofdemobilization.

Intermediate Results 2.4.5. More Effective, Independent, and Representative Legislatures

In democratic systems, elected legislative bodies are fora where competing interests can be openlydiscussed and negotiated and important national decisions made about the use of public resources. Bystrengthening legislatures, citizens, through their representatives, are given greater access to the policyprocess and more influence over the behavior of the executive branch. In many countries, however,legislatures are relatively new and lack the competence or confidence to introduce legislation on theirown. Legislatures need to serve both as checks on executive branch behavior and effective arenas inwhich citizens’ interests can be raised and negotiated and conflict resolved. The role of the legislature indecision-making is enhanced by increasing its oversight of executive branch and military behavior, itsinfluence over policy-making and the budget, and its ability to shape appropriate legislation. Its role as arepresentative body is enhanced by encouraging greater public access, that allows citizens to provideinput and oversight.

(Sub) Intermediate Result 2.4.5.1. More Effective and Democratic Internal ManagementSystems

Improvement in internal management systems are both a requirement of an effective andautonomous legislature and a demonstration that it has the capacity to manage its own business.Such systems require enabling regulations and procedures which allow the legislature to operateeffectively and democratically. “Effective management” depends on administrative and secretarialsupport, and might be reflected in agendas being published and salaries paid on time; and libraryand reference services, and electronic and other information systems being available and used, asdemonstrated by better informed debate and legislative initiatives. “Democratic management”requires that all parties, majority and opposition, and where relevant, gender or community(minority ethnic or language or religious group) representatives receive equivalent resources andsupport. Efficient and democratic internal management of the legislature is an essentialprecondition to representation of the public, competent policy analysis and formulation, and toserving as an effective political counter-balance to the executive branch.

(Sub) Intermediate Results 2.4.5.2. Increased Legislative Capacity to Influence NationalPolicy and Budget Priorities

Increased capacity to exert influence — both through contributing to formulation and overseeing


implementation — of national policies and budgets requires regulations and procedures enablingthe legislature to operate in an independent and democratic manner, sufficient time andinformation for full consideration of proposals and records of implementation, professional staff toassist in analysis and drafting, resources and authority to investigate and question the executivebranch, and an effective system of committee meetings (or some equivalent mechanism forlegislative consideration or oversight). Such increased capacity will be observed in higher qualitydebates, more considered and constructive legislative initiatives and amendments, rigorousoversight hearings and effective action consequent upon those hearings.

(Sub) Intermediate Result 2.4.5.3. Increased Citizen Access to Legislative Processes

It is essential to transparency, accountability, and representation of the legislature that citizens,civil society organizations, and the media have substantial access to legislators and to thelegislative process. Access for purposes of facilitating citizen “input” to legislative action can berealized through mechanisms such as public hearings that include the contributions of citizens andCSOs, joint commissions, town meetings, and through meetings between legislators and theirconstituents. Access for purpose of citizen “oversight” can be improved by reforms such asallowing citizens, CSOs, and the media to observe committee and plenary sessions; makingavailable congressional records and records of joint meetings of legislators and citizens, and byallowing for in-depth analysis of legislative performance in periodicals; and by the development ofNGOs that monitor the activities of legislatures and their members.

(Sub) Intermediate Result 2.4.5.4. Improved Capacity to Reconcile Societal Conflict in anInstitutional Framework

Serious societal conflicts may result from ideological, historical, religious, or ethnic differencesand may have led in some cases to violent conflict and/or a legacy of deep bitterness and mistrust.By allowing the antagonists to be represented in a national forum, to participate fully in debates, tohave an input to policy-making and an opportunity to push their agenda within instructional rulesand limits, and in some circumstances to see the value of compromise and coalition-building, thelegislature framework has the potential to play a key role in helping to reduce or reconcile societalconflict, and begin to set precedents for peaceful rather than violent resolution of differences.

Intermediate Results 2.4.6. Enhanced Policy Processes in the Executive Branch

Whether it be through direct elections, as in the presidential system, or as leaders of the party that gains amajority in the legislature, as in the parliamentary system, the leadership of the executive branch,together with the cabinet and the bureaucracy over which it presides is a major actor in building andmaintaining democratic and effective government institutions. The executive branch role in relationshipto the legislature is key to effective governance — respecting and working with it as a full democraticpartner in policy making, and accepting, and cooperating with it, in its executive oversight role. In itspolicy implementation role, the executive branch is required to provide leadership, oversight, andincentives that assure constructive intra-agency communication and cooperation among governmentdepartments, and ensure that the bureaucracy carries out its functions effectively and impartially, remainsaccessible to public participation, and accountable and responsive to both the legislature and the public.Finally, given its prestige and relatively greater autonomy to act, there are critical times when theleadership has a vital role in taking the initiative, for example, when democracy is threatened or conflictsmay divide the nation.


(Sub) Intermediate Result 2.4.6.1. Rules Reformed for Making Policy Reform

Laws, rules, and regulations defining the division of responsibility for policy processes betweenthe executive and other branches of government need to be passed and implemented.

(Sub) Intermediate Result 2.4.6.2. Policy Formulation and Implementation CapacityStrengthened

Strengthened executive branch capacity in the various policy processes depends upon policymaking skills such as policy analysis methodologies, resources, and personnel of the quality and innumbers sufficient to implement and oversee policy implementation, and a supportive legal andinstitutional framework for implementation.

(Sub) Intermediate Result 2.4.6.3. Intra-Governmental Consultation and InformationImproved

Effective executive branch operations also require constructive intra-agency consultation andcoordination and more accurate, timely and helpful information.

(Sub) Intermediate Result 2.4.6.4. Opportunities for Public Participation Increased

As with the legislative branch, the executive also needs to provide access to the public, citizens,and civil society organizations, to allow for public initiative, comment on and oversight of policymaking and implementation.

(Sub) Intermediate Result 2.4.6.5. Executive Leadership in Enhancing Democracy

The executive branch, especially at the leadership level, tends in most countries to have verybroad formal authority and discretionary power. Therefore it is not the intention to strengthen theexecutive role yet further. However, there are critical moments — possibly a strategic opportunityto augment democratic momentum, possibly when democracy is under threat — that executiveleadership is critical.

(Sub) Intermediate Result 2.4.6.6. Executive Leadership in Resolving Divisive Conflicts

Following similar logic to 2.4.6.5 above, the capacity of the executive leadership to know whenand how to intervene to prevent or resolve potentially divisive conflicts is also key.









6. Not all categories will be discussed for every indicator. In some cases, the relevance of the indicatoris clear and does not require any discussion. In other cases, we have little at this point to contribute toideas about how to set targets.


Indicators

Agency Objective 2.4: More Transparent and Accountable Government Institutions

Intermediate Result 2.4.1: Increased Government Responsiveness to Citizens at the Local Level

Indicator Definition and Unit of Measurement Relevance of Indicator Data Collection methods/Approx. costs Target setting/Trendline interpret. Issues

1) a) % citizensin target localgovt. units whofeel that localgovt. isaddressing theirpriorityconcerns; or b) % citizensshowingconfidence intheir localgovernment.

Same as indicators, disaggregated bygender, ethnic group, urban-rural orwhatever divisions make sense.

These may be better contextual orplanning indicators than they areprogram performance indicators,because the resource levels mayneed to be very high to affectconfidence.

Sample survey of population in target local govt.units, stratified according to categories ofdisaggregation.

Cost: High, unless a survey is being done formultiple indicators or questions can be added toanother planned survey.

Both indicators should show gains as local gov-ernment responsiveness grows. However,trendlines need careful interpretation. Forexample, they may not always reflect actualchange, as there may be a lag time betweenactual change and citizen awareness of thatchange. Also, citizens may be influenced byextraneous factors such as changes in theeconomy or actions taken by the centralgovernment at the local level. There may be apossibility that higher levels of openness andtransparency in new democracies could bringabout a decline in confidence due to greaterknowledge about problems.

Comments:

2) % of eligiblevoters (male andfemale) voting inlocal elections.

Eligible refers to all adult citizens who arenot excluded by law by reason of criminalrecord or insanity. It includes bothregistered and unregistered voters.

% of eligible voters who vote. Recorded ateach election. Can be disaggregated byethnic group and gender if records arekept.

May be indirect measure of howcitizens feel about theresponsiveness of local govt.

Monitoring govt./local govt./records or/& NGOfindings on registration & voting.Possibly (on a selective basis) working with localorganizations to check on record keeping methods.

Costs relate to ease & availability of records. Maybe affected by whether records are manual orcomputerized.Should not be a demanding undertaking, butdepends on the credibility of local records.

Trend may be expected to jump at first, but itmay reach a plateau and may very well declineafter that. USAID should not be too ambitioussince it does not have influence over voterinterest & turnout; and a variety of factors maylead to lower turnouts. Voter turnout is difficultto interpret. Turnout could be high when anelection is critical or the stakes seem very highand low when people feel very comfortable orgood about the situation.

Comments:



3) # or % ofwomen orminority electedlocal councilors(or members oftask forces, jointcommissions orother non-elected bodiesset up to providecitizen input).

Women or minority councilors — whatconstitutes a “minority” will be situational.

# or % of all elected members of thecouncils.

After each election, unless some areappointed.

Diversity of representation oncouncils enhances democracy andlikely increases chances of moreequitable outcomes.This may be of general concern toUSAID or it may only be so whereUSAID supports a program aimed atincreasing women’s and minorityrepresentation.

Monitoring govt. or local govt. records.

Cost: Should be easy to get information on menand women. In case of other categories, such asreligion or ethnicity, definitions of identities andrecords may sometimes be less clear. If difficult,then it may be necessary to do interviews &therefore useful to work with a sample of localcouncils.

If programs to increase women’s representationand diversity are in place, then the trend shouldrise, but it is unlikely to be dramatic and couldeven go down after first successes; if there isno program in place, then it depends on localcustoms and on activism of local NGOs andnumbers may not increase.

Comments:

4) a) # or % ofmajor local govt.decisions inwhich input fromparticipationmechanismstaken intoaccount;orb) # or % oftarget localgovernmentunits (LGUs) inwhich themajority of keydecisions takeinto accountcitizen input.

Same as indicators. Key decisions can bein target issue areas identified in advance.Participation mechanisms can include townmeetings, joint commissions, etc.

Very important to assess whethercitizens & NGOs are not only invitedto meetings but that their views arehaving an influence.

Requires monitoring local council decisions,including the annual budget. If counting decisions,will need to be able to identify the base of totaldecisions in the year. Requires correlating citizeninput with the decisions, so may need to interviewNGOs, have minutes taken at town meetings,review joint commission reports, etc.

Cost: Requires detailed and sustained attention andtherefore could be fairly costly.

Should increase (probably rather slowly) asinput from public improves and local govts.demonstrate increased willingness to listen andadapt policies.

May be difficult to set an accurate target for a).

Comments: A qualitative assessment will be needed to support the interpretation of any absolute numbers or percentages. Could also construct this indicator as a scale: no decisions take input intoaccount; a few decisions take input into account; etc. A scale may be less labor-intensive than trying to track numbers of decisions.



5) # or % ofcitizens whomake use of, orare served by,programs,benefits,services of localcouncils.

Local citizens using or benefitting fromservices.

# per annum per category of service.

Disaggregation may be important here;depending on the community, this mayrefer to gender, class, ethnicity, or location.

Ultimately local govts. need toprovide services to the public at asatisfactory level; this indicatordemonstrates services arebecoming more available and thatcitizens are benefitting or makingthe effort to make use of them.

Monitoring records of local govt. agencies whichprovide services such as electricity, garbagedisposal, etc.

Calculating the percentage will require good censusdata. May make sense to use households as theunit of measure for some kinds of services, such aswater, sewage and garbage collection.

Cost depends on quality of records. If these arereasonable, this should not be too demanding.May be demanding if records are poor and if onetries to observe too many councils & too manyservices. Disaggregation will obviously add toexpense; how much will depend on the nature ofthe records. Suggest being selective.

If local govts. are serving communities better,this will increase over time. Initially, with thetransfer of services to local governments, it maybe ambitious enough to maintain servicedelivery at current levels. Wealthiercommunities will automatically do better at thisindicator. Makes sense to set annual targets.

Comments: Rather than total numbers, it could be more important to look at equity issues and how local governments are serving poorer citizens.

6) Servicedeliveryindicators.

These can range from percentage ofpopulation satisfied with particular servicesto specific service measures such asnumber and percent of collection routes notcompleted on schedule, number of citizencomplaints about a service per 1,000households served, percentage of streetslit, number of water quality characteristicsexceeding standards during the reportingperiod, etc.

Improvements in service orperception of quality of services.

Govt. records.

Cost: Will vary according to indicator, quality, anddegree of aggregation of records.

Will respond to attempts to improve efficiencyand effectiveness of services. Perception ofchange may lag behind actual improvements.

Comments:



7) Example(s) ofdecisions beingtaken at centralgovt. level as aconsequence ofpressure fromeither individuallocal govts. ororganizations orassociations oflocal govt.(See commentsbelow.)

Example involves a clear demonstrationthat a branch of the central govt. exec tooknote of a petition or a visit by a delegation,a resolution, or some mobilization ofopinion by local govts. and alteredlegislation or policy to conform to localrequest. The simplest unit of measurementwould be:yes = central govt. did so respond; no = central govt. did not respond or hadno occasion to respond.

Most local government measuresfocus on effectiveness of loc govt. atlocal level. This one seeks to assesswhether local govts. are able toexert pressure upwards on centralgovt. and so represent the interestsof citizens at the local level.

Interviews with key local govt. leaders, possibly tobe confirmed by interviews with members ofwhatever branch or agency of central govt. thatgave or was asked to give consideration to thepetitions.

Cost: This is not expected to be a frequentoccurrence; and should require interviews with onlyleaders of local govts. or associations of local govt.It should therefore not be too demanding in termsof time or money.

A trend will be difficult to anticipate and may notmake sense. In some years there may be moreopportunities, and some years fewer. Rather itis important to detect whether central govt.takes note of, and acts, when local govts. makeorganized and effectively prepared petitions to itto do so. Therefore one would look forward to aconsistent “yes” response as a target.

Comments: This indicator could be configured as the # or % of target LGUs that can name 2 or more decisions they have impacted at the central level in the previous 12 months. It might also bepossible to put together a critical events agenda based on known issues that have come up between local government and central government. It could be useful to disaggregate local governmentassociations from individual local govts.

8) # of targetlocal govt. unitsimplementinginvestment ordevelopmentplans a) withcitizen input; andb) that meetquality criteria.

a) Criteria will need to be develop as towhat constitutes adequate citizen inputb) Quality criteria can be developed tomeet local circumstances but might includeclear objectives; priorities meet majorneeds; marginalized communitiesincorporated; revenue expectations to fundplan are realistic; etc. Each criteria couldconstitute a yes/no indicator worth onepoint.

Measures both openness to citizeninput and dedication to qualityservice and planning. This is a highlevel indicator of meaningful citizenaccess because it focuses onimplementation of investment plans.

Review of investment plans; interviews with govt.officials, NGOs, media.

Cost: Medium, but depends on number of LGUs.

Would expect to see increases over a period oftime. b) If the indicator were scaled with 10criteria, each worth one point, could report onthe number of LGUs which achieved a passinggrade (i.e., 6 out of 10 points) or an excellentgrade (i.e., 8 out of 10 points).

Comments: For b), see Appendix C information on score cards.



Intermediate Result 2.4.1: Increasing Government Responsiveness to Citizens at the Local LevelIntermediate Result 2.4.1.1: Constitutional and Legal Reforms to Devolve Power


1) # of keyreforms passedin comparisonwith a list ofrecommendedor promotedreforms.

Reform agenda established at outset,progress then measured in terms ofachievement of items on agenda.

See also legal reform milestone scale inAppendix C.

First & usually basic requirement (&test of central govt. commitment).Also answers the question whetherthere is an enabling environment.Therefore, it is necessary to notewhether such laws are in place orwhich key aspects are in place. Ithas been observed that occasionallysome of these powers are in effectdevolved, or local govts. takeresponsibility, without or beforesuch laws are passed.

Review of govt. documents, legislation &/or theconstitution &/or an assessment of thesedocuments by appropriately qualified persons.

Cost: Low cost; requires annual monitoring of a fewkey documents and (if & when) in place should notrequire more attention.

Reforms could move forward all at once; or stepby step, the sequence of steps depending onthe local sensitivity of each and also on localcapacity (i.e., transfer of power needs to relateto local capacity to use power well). Pace ofchange will also depend on pressure by civilsociety and by extent of donor programs &interest in bringing about change. If local govts.fail, a downward trendline may be expected ascentral govt. may reverse some of thelegislation.

Comments: Necessary characteristics of reforms include: Scope of responsibilities clearly defined for each level; regular local elections with universal suffrage (there should be no provisions which havethe effect of excluding women or minority groups); substantial scope of decision-making including budgetary authority; financial authority to raise & spend money commensurate with capacity andresponsibility; appropriate number of tiers; equitable formula for allocation of revenues; & public access to local govt. information.

2) a) # or % oflaws which seekto devolvepower beingimplemented bythe central govt.;orb) level ofimplementationof key laws.

Laws which contain clauses which havethe effect of devolving authority to localgovt. “Implemented” means being carriedout; that is, depending on requirements,that central govt. draws up & publishesregulations, stops collecting certain taxesor hiring certain personnel, transfers funds,or transfers funds under new conditions,stops intervening in certain decisions, etc. b) scaled measure of implementation (high,medium, low, non-existent).

Once laws are passed, the moreimportant and more difficult stepand test is their implementation ingood faith by central govt.

Depending on the situation, this might require bothmonitoring of govt. records and communications(possibly in a ministry of interior or local govt., orfinance or the president’s office) & carefulinterviewing of key actors at all levels (both govt.and non-govt.). A scale could involve assessmentby an expert panel.

Cost: This might involve a medium to high level ofinvestment of time as one might be watching fairlydetailed changes in a number of locations. If localNGOs could be brought into the monitoringprocess, this might ease the responsibility. Anexpert panel would be comparatively lessexpensive but continuity in the panel would berequired over time.

Trendline should rise but it might be a slow andtedious process. It will depend on central govt.or incumbent party’s confidence & will todevolve authority, on past record of devolutionin the country (it may have failed before), onreligious and ethnic differences from region toregion, & on local capacity to take on newresponsibilities. While local capacity should notbe used as an excuse for slowing downdevolution, the pace of change must relate tolocal ability to use the new powers effectively &transparently.

Comments:



3) a) # or % oftarget localgovts. whichnote little or nointerference ininternallegislative affairsby the centralgovernment;orb) # or % of localcouncils’ lawspassed withouthindrance fromcentralgovernment.

a) a scaled measure that marks theconsensus of key local govt. officialsconcerning whether central govt.interference when the local govts.considers or passes a law is high, medium,low or non-existent. Would be useful tospecify criteria for what constitutes a highlevel of interference (i.e., more than 6times per year), etc.

b) This indicator could focus on key laws ordraft legislation. “Without hindrance”means without opposition from the centralgovt.

This is a test primarily of centralgovt.’s willingness to implementdecentralization; in this caseallowing local govts. to makedecisions for which they have beengiven authority. The indicator couldalso reflect on local govt.’s capacityto draft laws & undertake newresponsibilities.

a) Interviews with key local officials in target LGUs;interviewers would do the scaling;

b) Reviewing central govt. & local govt. records tosee how long decisions/confirmations of local govt.decisions took, or if there were examples ofobstacles, and any reasons given. It will benecessary to interview key actors in local govt.

Cost: a) may be less costly than b) If there are nocentral govt. impediments at all, data cost will below. But more likely, it will take a medium level ofeffort to observe this, particularly where financialceilings & checks have been built into the system.Help from partners or NGOs may be necessary.

As the “comments” section below notes, somelevel of central government oversight is merited.We would not expect oversight to fall to zero.We would hope for a declining trendline if thereis heavy central government interference at theoutset. It should be noted that if there isabsolutely no central government interferenceor oversight, this could simply be a sign of avery weak and negligent central governmentand not of complete decentralization or centralgovt. contentment with quality of local govt.decisions. Meaning of this indicator will becontextual.

Comments: Not all monitoring or reasonable delays to check on legislation should be seen as “hindrance”; the early process of decentralization might involve a certain amount of “paternalistic” monitoringof expenditures over a certain amount; or prevention of local discrimination against minorities might require this. It could be informative to look at the content of laws rejected to see if there is a pattern atthe local or central level of discrimination against women and/or other minority groups.



Intermediate Result 2.4.1: Increasing Government Responsiveness to Citizens at the Local LevelIntermediate Result 2.4.1.2: Local Government Capability to Act Increased

Indicator Definition and Unit of Measurement Relevance of Indicator Data Collection methods/Approx. Costs Target setting/Trendline Interpret. Issues

1) Degree towhich legislativeauthoritytransferred tolocal councils tolevy local taxesand fees.

Yes/no by category of taxes and fees.Could be measured on scale of none, verylimited insufficient, sufficient, etc. or couldset up a list of categories of taxes & feesw/points assigned to each category (eitherweighted or 1 pt. per category).

Although a subset of measures ofcentral govt. willingness, it is soessential to the process that it maybe worth separate focus.

Monitoring of legislative enactments & related rules& regulations. Could be by an NGO or independentpanel of experts.

Cost: Should be relatively easy, since this focuseson laws & rules and not on actual amounts raised.

Change may occur all at once in the initiatinglegislation; or trendline should rise but it might bea slow and tedious process. It will depend oncentral govt. or incumbent party’s confidence &will to devolve authority, on past record offinancial behavior by local councils (they mayhave failed before), & on local capacity to takeon new revenue raising responsibilities. Whilelocal capa-city should not be used as an excusefor slowing down transfer of authority to mobilizeresources, the pace of change must relate tolocal ability to use the new powers effectively &transparently.

Comments: Local expertise necessary to determine which are more important sources of revenues necessary to meaningful decentralization.

2) % of all localgovt. staffcompleting skilltraining courseswho say theyare using theirnew skills on thejob.

Elected officials and civil servants whohave been trained and who say that theyare using their new skills on the job. (Theyshould be required to provide examples ofhow they are using it.) (Disaggregation by gender & other socialcategories may be informative.)

A fairly low level indicator. Oneindication of local govt. capacity toeffectively serve the local public isthe skill level of its salaried staff.Direct measure of the outcomes oftraining.

Monitoring local govt. or training institutionpersonnel & training records. Survey should be noearlier than 3 - 6 months after training. Could do asample survey if number of trainees is largeenough. Directly relevant to trainers’ planning ofadditional training.

Cost: Medium but depends on the numbers trainedand whether a census of sample survey is done.

If the training is high quality, the percentageshould be high, unless there are institutionalbarriers to using skills.

Comments: Could expand this indicator to include recipients of technical assistance.



3) a) Average %of local govt.middle andsenior level civilservants hiredby local govts. orlocal govt. publicservicecommission;orb) % of localgovts. in whichthe majority ofmiddle andsenior level civilservants arehired by localgovt.

Personnel recruited by local govt. out oftotal number of such personnel(assumption that the rest are recruited bycentral govt.). Could be refined to includeonly more important posts or posts inparticular sectors (public finance, publicsecurity, etc.).

Indicates that central govt. hasgiven this authority to local govts.and has confidence that localauthorities can make the choicesbased on merit of individuals andprogrammatic need (it is fairlycommon for central govt. tocontinue appointing personnel,especially key persons); and showsthat in practice local govts. arehiring their own staff including theirsenior staff.

Monitoring central govt. PSC or local govt. PSC orlocal council’s personnel records, or govt. gazettesor newspapers in which posts are advertised. Lattermay not give complete information.

Cost: Should not be a heavy responsibility, but willdepend on accuracy & timeliness of records andwhether they are centrally maintained or not. Poorrecords will add to the burden.

As with many aspects of local government, it willdepend on central will and local resources andperformance. Should change all at once or insubstantial increments in response to legislationor policy changes.

Comments: There is a possibility of great difference between local govts. of the main urban city & the poorest rural district. Possibly a median might be more appropriate than an average or some otherrefinement might need to be added.

4) Incorporationof meritprinciples, basedon the following3 (illustrative)questions: a)Are there meritprinciples inplace b) arethere clear jobdescriptions inplace for keypositions; and c)# complaintsabout violationsof meritprinciples inhiring.

a) and b) yes/no. Posts covered may varybut should take into account: seniority,training, professional requirements;education; & decision-making power. Meritis to be distinguished from non-transparentrecruitment process which may allowpersonal whim, nepotism, old-boynetworks, discrimination based on gender,ethnicity, etc.

Indicates improvement in localgovernment capacity andtransparency in a number of ways:has the capacity to analyze anddefine job descriptions; to reducepersonal whim etc., in recruitmentprocess; and promises better resultsbecause of improved level of staff.

Document review (merit principles and jobdescriptions, and complaints); interviews. Indicatorc) requires that a system for registering complaintsbe in place.

Cost: Likely to require a medium level of effort.

Changes will depend partly on political will andpartly on capacity to develop comprehensive jobdescriptions, manage hiring process & availabilityat local level of qualified personnel, and localleaders’ acceptance of merit principles. Ifsystems are new, complaints may be high for aperiod of time but then should decline. There arealways likely to be some complaints. Whetherprospective job applicants choose to complain ornot will depend partly on whether they knowabout merit principles. No complaints may equallack of public knowledge, not perfect merit hiring.

Comments:



5) a) % local civilservice positionswhich areadvertised orposted;orb) % of localgovernment staffwho arereplaced afterlocal elections.

As noted in the indicator but out of allpositions filled during the year. May wish tolimit to target local govt. units. Positionscould be limited to certain sectors or topposts.

While the two indicators are verydifferent, they are both trying todemonstrate the extent of merithiring.

Local govt. records.

Cost: Medium, if records exist in some form.Depends to some extent on the number of targetlocal govts..

Would like to see a rising trendline on a) but willnever reach 100%. b) If starts high, would hopefor a decline but may not be steady trendline.Would hope to maintain at some minimal level.

Comments:

6) % of totallocal revenuegenerated bylocal govt.;or Dollar amount oflocal revenuegenerated bylocalgovernment.

Revenue refers to all the financialresources used for capital & recurrentexpenditure by local govts. Generatedmeans collected or raised by local govt.If using dollar amount, should compareincreases with inflation rate.

Another essential measure ofmeaningful decentralization andlocal govt. capacity to operate.Weak generation of local resourceshas been a major limitation on localgovts.’ autonomy & is sometimesused as a rationale for continuedcentral govt. control.

Review of financial records of Central Govt. (theremay be one ministry that keeps records on localgovts.) &/or of local govts.; budgets.

If there are many local govts., it may be necessaryto look at records of a representative sample.

Costs: Such records are probably available &therefore this should not be a demanding activity.

Poor records probably accompany weak localmobilization, and this will make the task moredemanding.

Trendline should rise in aggregate, but it willdepend on sources agreed to by central govt.,capacity of local govt. to collect, state of theeconomy; and there are likely to be majordiscrepancies between richer and poorer areas ofthe country.

“Dollar Amount” may be more appropriate thanthe % in cases in which local government isgenerating more revenues but the central govt. isalso transferring more revenues. If central govt.transfers go up at the same time, the percentagecould stay level.

Practical & useful to measure annually.

Comments:

7) % of locallygeneratedrevenue retainedlocally.

Revenue refers to all revenue raised locallythrough taxes, licenses, user charges, etc.& not sent to central govt.

Significant measure of centralgovt.’s willingness to allow localgovt.’s to take over substantialresponsibilities & reward them forcollection.

Review of financial records of central govt. (theremay be one ministry that keeps records on localgovts.) &/or of local govt.’s budgets. If there aremany local govts., it may be necessary to look atrecords of a representative sample. Cost: Suchrecords are probably available & therefore thisshould not be a demanding activity. Poor recordsprobably accompany weak local mobilization, andthis will make the task more demanding.

Trendline should rise as long as centralgovernment commitment to devolutioncontinues and loc govt. capacity to mobilizeresources improves. Increases maycorrespond to legal changes.

Practical & useful to collect annually.

Comments:



8) a) % ofcentrallycollectedrevenuedelivered to, foruse by, localgovts.; or b) $ amount.

Revenue refers to total financial resourcescollected by central govt.;

% refers to that proportion allocated/guaranteed and delivered to local govts.;

Almost everywhere even stronglocal govts. receive somesubventions from central govt. Thepercentage is an indicator of centralgovt. will to allow local govts. to beeffective; and also points to thelikelihood that local govts. willmanage their work effectively.

Review of financial records of Central Govt. (ifthere is a ministry that keeps records on localgovts.) &/or of local govts.; central govt. budget willprovide information on projected allocation to localgovts.; loc govt. budgets should provide infoexpected allocations from central govt.

Cost: Relatively easy.

Trend should be upwards. It is possible that if the% is satisfactory that the amount (controlling forinflation) may become more relevant. While thepercentage is a good measure in mostcircumstances, it is also possible, although notlikely, that the amount might rise but thepercentage go down because of a substantialincrease in the size of the total budget.Increases in the dollar amount should take intoconsideration the inflation rate.Practical & useful to collect annually.

Comments:

9) a) # times/#monthsgovernmenttransfers fall intoarrears;or b) % localgovt. salaries inarrears oraverage #months not paidon time or notpaid at all.

a) # times per annum transfers are late ornot made and/or (average) number ofmonths transfers fall behind.

b) by number of salaries and not $ amount,or average number of months that salariesare not paid within x days of the due date.

The timeliness of transfers can havea huge impact on local governmentfunctioning.

Government records, interviews with localgovernment staff.

Cost: Medium, depending on number of target localgovts. working with.

Would hope to see improvements in timelinessbut may reflect on national government capacity,something a local govt. strategy is likely to dolittle to improve. If the problem is political will,there could be a big improvement and then aleveling off. There may or may not be regionaldifferences in terms of when transfers are made.

Comments:

10) a) % of(target) localgovts. improvingcollection oftaxes by at leastx%; or b) taxcollection rate.

a) Out of total owed taxes. Thresholdimprovement % can be set in accord withlocal circumstances.

b) % of selected owed taxes that areactually collected.

Indicates local govt. more effectivein collecting taxes and probably thatit is providing more effectiveservices and therefore thattaxpayers may be more willing orused to paying.

Review of local govt. tax records. The less accuratethese are in terms of potential taxpayers andpayment, the less useful the records are for thispurpose.

Cost: If records are good, then this should not betime consuming. If they are not that good, it willtake more time. If they are poor, then it is probablynot feasible to collect.

Trend should be upwards, but usual list of factorswill affect the trend. Target should allow forimpact of inflation and actuals should beinterpreted in comparison with the inflation rate.In order to discern changes, should becollectable every two years.

Comments:



11) a) Ratio ofcapital torecurrentexpenditure;orb) # target LGUsimproving theratio.

Usually budgets divided into recurrentexpenditures (on-going, salaries,maintenance of services) & capitalinvestment (new plant, facilities,equipment).

Ratio of capital to recurrent.

Indicates that local govts. are ableto raise revenues that go beyondthe demands of recurrentexpenditure and also able to planfor larger projects on a multi-yearbasis. Also demonstrates efficiency.If salary rolls are padded, recurrentexpenditure will take up large chunkof budget, allowing few resourcesfor investment.

Review of recurrent and capital parts of the budgetsof local govts. provide good data. But annualexpenditure reports will be more accurate as theyprovide information on actual as opposed toprojected expenditure. May be an issue with thetimeliness of data.

Cost: Providing there are such records, this shouldnot be difficult. If information is consolidated by onegovt. agency, this would make it even easier. If it isnot and there are many local govts., then it may benecessary to focus on a sample only.

Trend should be upward in aggregate, but willvary between richer and poorer areas, and fromyear to year within the same local govt.

Collectable every year or two.

Comments:

12) a) % (target)local govt. unitsthat belong to alocal govt.association;b) level offinancial supportto local govt.associations by(target) localgovt. units.

a) As in indicator.

b) in USD.

Local govts. will have more impacton central govt. decision-making ifthey join together and work throughassociations.

Association records.

Cost: Should be nominal.

The association may only be effective ifmembership reaches a critical mass. Once it isperceived as successful, it should attractadditional members. If there are a number ofcompeting associations, then these indicatorsmay not be helpful.

b) Increases in the dollar amount should take intoconsideration the inflation rate.

Comments:


Agency Objective 2.4: More Transparent and Accountable Government InstitutionsIntermediate Result 2.4.1: Increasing Government Responsiveness to Citizens at the Local LevelIntermediate Result 2.4.1.3: Mechanisms of Participation Increased


1) a) % of localgovernmentelections that arefailed elections;or b) % of localgovernment unitsin whichcompetitiveelections are heldregularly.

a) Failed elections are those which arenot free or fair. The definition may beexpanded to include problems withviolence, and perhaps refusal by some toaccept the results.

b) “Regularly” needs to be defined ineach situation, depending on the electoralstructure.

Competitive elections in whichnumerous candidates sponsored bydifferent organizations (possibly butnot necessarily parties) compete, inwhich local issues are emphasized,and voters are free to choose, thereis no intimidation, are an essentialand obvious requirement ofdemocracy at local level.

Requires monitoring of nominating process, campaigns,& elections. If local credible monitoring organizationsare in place, they should be relied upon. If not,international organizations &/or donor(s) will need to bevery selective in number of places observed.In some conditions — possibly first time or verystrategic election — international organizations couldplay an active observation role. Cost: If local organizations are in place this should notbe a heavy responsibility; if they are not, then it may bea more serious undertaking in terms of effort andmoney. Local level manipulation of elections may be farmore difficult for outsiders to understand than nationalelections. It is suggested that use be made of localorganizations & donors augment their work in a few keyselected locales.

a) The trendline should decline, as morelocalities hold satisfactory elections.

b) The trendline should rise at first. It ispossible that over time a plateau will bereached.

Comments: For additional elections indicators, see Section B on political processes, Tab 7.

2) a) total # oraverage # ofpeople attendingtown meetingsorganized by localgovt.; or b) % oflocal govts.holding more thanx town meetingsin the last yearwith more than Ypeople attending.

a) Participants can be disaggregated bygender but may be difficult to do by ethnicgroup.

or

b) % of target local govts.; threshold fornumber of town meetings and average #participants annually can be setaccording to local conditions.

One important mechanism forallowing public participation in localdecisions.

Town meetings must be welladvertised in good time, held inconvenient place and time, andopen to whole community.

Observation of some or all meetings or interviews withnon-govt. & govt. informants; likely that local NGOsmay monitor and keep records. A systematic sample ofmeetings &/or of local govts. might be used.

Cost: Depends on how many meetings and how manycommunities are observed. Need not be too heavyespecially if reliable NGOs can act as informants.

If there is a legacy of hostility or distrust orlow expectations, the trend might move upslowly; if there is new excitement, it maymove more quickly and then might decline.Participation will also depend on theimportance of the issues and howcontroversial they might be.

Comments: Another possibility is to count the percentage of (target) local govts. making regular use of legally-provided participation mechanisms. “Regular” would need to be defined. Another option mightbe the % of citizens (male/female) who can name at least two mechanisms for participation. This measures knowledge of ways to participate rather than actual participation. It would be expensive tocollect this data unless a survey were being conducted to gather other data. If a survey is being done anyway, actual participation rates can be examined. Keep in mind that in heavily rural and largelyilliterate societies, participation rates (except in elections) may be low. Significant change could be difficult to bring about.



3) a) # ofmeetings of jointcitizen-localcouncilcommissions/boards; and b) #of joint boards.

Meetings of joint commissions (nameswill differ) in which citizens or NGOs havesignificant membership (m/f & ethnicgroup participation should be recorded). # of meetings per annum and number ofboards.

Another potentially usefulmechanism for citizen or NGOparticipation. Breadth ofrepresentation should beconsidered.

Observation of some or all meetings or interviews withnon-govt. & govt. informants; likely that local NGOsmay monitor and keep records. A systematic sample ofmeetings &/or of local govts. might be used.

Cost: Depends on how many meetings and how manycommunities involved.

Trend should be upwards, as this is usuallya novel mechanism and fewer people whohave a more focused interest are likely to beinvolved. It should be anticipated that aceiling will be reached after a while. Makessense to measure annually.

Comments:

5)Budgets/financialreports of localcouncilsavailable in goodtime tocouncillors, thepublic, the mediaand NGO;or b) # or % ofcouncils that bringout thesematerials in time.

1) a) Budgets are available x weeks priorto approval (yes/no);2) At least one fora for public discussionof the budget is held, with due noticegiven (yes/no); and3) Financial reports are made availableno later than x months after the close ofthe reporting period (yes/no).b) The # of target councils that satisfydefinitions of timely publication of thesematerials.

These are complex documents butamong the most important. It isessential that various parts of govt.and civil society have time to readand prepare for discussion.

Monitoring local govt. records and checking with NGOs& the media. In the case of many councils, it may benecessary to focus on a representative sample.

Cost: Depends on how comprehensive the coveragewill be. But these involve only a few key events a year,and it should not be too time consuming to follow this insome local councils.

As capacity of local councils improves andwillingness to open these materials forpublic viewing and participation, thissituation should improve with time, but it islikely to take time.

Comments:

6) a) Internal &external audits oflocal govt.expenditures a)takes place inaccordance withrequiredschedules and b)are published andavailableor c) # or % oflocal govts. inwhich both a) andb) above are met.

Audits accord withnational audit standards & full informationdisclosure.

a) and b) Yes/no.

or

c) % of local govts. that satisfy theserequirements, a) + b) out of all localgovts.

Key aspect of transparency.Citizens require information in orderto be able to participate.

Monitoring the records of local councils or central govt.ministry responsible for local govt. &/or auditing firms.

Cost: If this system is working at all, records will beavailable & the effort should not be heavy. If there areno records, the system is probably not working.

If there are programs in place which havegovt. support then the situation shouldimprove steadily (depending on govt. will,training of auditors, resources available topay external auditors). Eventually thisshould occur in all local govts. Makes senseto measure annually.

Comments:Agency Objective 2.4: More Transparent and Accountable Government Institutions


Intermediate Result 2.4.2: Increased Citizen Access to Improved Government Information

Indicator Definition and Unit of Measurement Relevance of Indicator Data Collection methods/Approx. cost Target setting/Trendline

1) Citizensbelieve theyhave adequateinformation onpolitical &economic issuesand on keyaspects ofgovernmentproceedings/activities.

% of citizens who answer surveyquestion(s) that they are satisfied that thegovt. is providing adequate information (interms of accuracy & sufficiency).Disaggregated by gender and otherrelevant categories.

Indicates level of public confidencein govt. information about its ownproceedings and public servants.

Sample survey of the public. Sample to include socialcategories relevant to disaggregation such as menand women.

Cost: Expensive if USAID has to carry out the surveyitself; less expensive if items can be included in asurvey carried out by a local organization or if USAIDitself is doing a survey for several DG indicators.

Findings should be treated with care & willneed expert interpretation, i.e., openinformation about mismanagement orcorruption will probably cause negative ratherthan positive feelings among the public.Useful to collect every two or three years.

Comments:

2) Journalistsbelieve thatgovt. isproviding themwith fullopportunity toobserve &pursue issuesor other key usergroups believe.

% or (possibly) panel of informed politicalcorrespondents and editors who feelconfident that they are able to cover &investigate most of the important publicconcerns.

If the press is reasonably satisfied,then it probably shows that govt.agencies are making a seriouseffort to open up records andproceedings to the public.

Annual interviews with key informed members of thepress or a survey. It is important to includerepresentatives of the independent and oppositionpress.

Cost: Selective interviews with key informants shouldnot be costly in time or money; a survey would bemore so but still not overly expensive.

Even if level of openness rises trendline mayrise and fall depending on the sensitivity ofissues, govt. confidence, and press trackrecord in dealing with investigations.Useful to collect every year or two.

Comments:



Intermediate Result 2.4.2: Increased Citizen Access to Improved Government InformationIntermediate Result 2.4.2.1: Laws and Regulations Reformed

Indicator Definition and Unit of Measurement Relevance of Indicator Data Collection methods/Approx. cost Target setting/Trendline Interpret. Issues

1a) Laws andregulationsrequiringgovernmentinformationsharing on keyissues are inplace;or b) improvementsmade in therequirements forinformationsharingcompared to alist of reformspromoted.

Key areas include requirements forfreedom of information, holding of publicmeetings, open records and reports,information on availability of services, andon-going administrative investigations.Could be yes/no; or there could be scoreagainst a checklist of desired laws.

Rules requiring govt. ministries tohold public meetings, allow forpublic input are an essential basisfor access to information.Increasing the number ofsituations which govt. has to openproceedings to the public & pressindicates improved environmentfor citizen access.

Monitoring of govt. documents, and/or laws and/orregulations.

Cost: Low.

General rules for all govt. departments couldbe established fairly rapidly; specific rules fordifferent ministries and situations mightincrease incrementally.

Comments: See also the legal reform scale in Appendix C.



Intermediate Result 2.4.2: Increased Citizen Access to Improved Government InformationIntermediate Result 2.4.2.2: Improved Dissemination of Information


1) # or % ofoccasions (forwhich it isrequired to doso) for whichgovt. agenciesprovideadequate noticeof publichearings.

“Adequate” will vary according to thecomplexity and length of materials to beread; “customers” should be consulted onwhat is reasonable & realistic takingaccount of normal constraints on govt.time.

Advance notice is first indication ofgovt. sincerity about opening upinvestigations, deliberations, etc. tothe public and to interest groups.Also indicates sufficient advanceplanning about informationdissemination by govt. agencies.

Monitoring of govt. agencies that are required to holdpublic hearings or select agencies. Interview key NGOleaders.

Cost: Might be costly if there are many opportunitiesfor public hearings and it is a large country; would berelatively inexpensive if select agencies only areobserved.

Should show positive trendline especially ifthere is a focus and programmatic support, butmay differ from ministry to ministry.

Comments:

2) # or % ofoccasions (forwhich they arerequired to doso) for whichgovt. agenciesprovideadequate noticeto the press ofpublic hearings

As above; “adequate” depends oncircumstances, but press needs less timethan organizations that need to respond.

Advance notice to the press is anindication of govt. sincerity aboutopening up investigations,deliberations, etc. and informing thebroad public.

Monitoring of govt. agencies that are required toinform the press or only select agencies. Interview keypress people.

Cost: Inexpensive if kept to select agencies and ifgood use is made of key informants in the press.

Should show positive trendline especially ifthere is a focus and programmatic support, butmay differ from ministry to ministry.

Comments:



3) # of targetagencies; (or %of agencies outof thoserequired to doso; or in aspecific locality;or functionalarea) providingfull informationto the publicabout theservices theyare required todeliver.

Information provided in good time andthrough a variety of channels accessible todifferent types of communities; i.e., throughmass media, including radio, govt. notice,extension officers, health officers, or torelevant NGOs and in relevant languagescould develop an index, if useful.

It is essential that the public knowswhat it has the right to expect froma govt. agency; it gives the publicsome basis to judge services; andputs pressure on agency to live upto promise.

Monitoring of govt. agencies (or a sample of selectones) that are required to inform the public; interviewswith key NGO leaders.It may be necessary to rely on a qualitative review ofthe information provided.

Cost: The more comprehensive, the more expensive;suggest focusing on select sample of agencies andfunctions.

Trend might rise at first and then possiblydecline if difficulties are caused or govt. isembarrassed by press informing public oferrors or wastage.

Comments:

4) # or % ofgovt. jobsadvertised in aset number andcategory ofnews channelsand in goodtime.

# or % of jobs fulfilling this requirement outof all jobs at these levels to which peopleare recruited. # and type of channels andadequacy of time should be worked outlocally in consultation with govt. publicservice commission and NGOs.

Full, timely advertisement ofrecruitment opportunities andprocedures is a first and key step inopen recruitment procedures.

Monitoring of govt. requirements concerningrecruitment, govt. documents, media and locales foradverts.

Cost: The more comprehensive the more expensive;possibly look at select sample of levels ofemployment; or check if other interested actors(NGOs, opposition parties) are monitoring.

If govt. is becoming more transparent, thetrendline should be up, but this will depend onpolitical will of govt., strength of patronagesystems, ethnic differences, alternativeemployment opportunities in the private sector,etc.

Comments:

5) # or % ofgovt. contractsor % ofmonetaryamount of govt.contractsadvertised in aset number andcategory ofmedia channels.

Procedures, channels of publicity, andrequired time should be worked out inconsultation with govt. contracts office andbusiness community.

Full, timely advertisement ofbids/tenders for govt. contracts is afirst and important step towardsopen govt. bidding and contractsystem; reduces opportunities forcorruption.

Monitoring of govt. requirements concerning contract,govt. documents, media and locales for adverts.

Cost: The more comprehensive the more expensive;possibly look at select sample of possibly largercontracts; or check if other interested actors,particularly business persons, but also NGOs &opposition parties, are monitoring.

If govt. is becoming more transparent, thetrendline should be up, but this will depend onpolitical will of govt., strength of patronagesystems, ethnic differences, alternativesources of contracts in the same lines ofbusiness.

Comments: Qualitative assessment by experts, NGOs or business persons of general trend.



6) Budgetsavailable ingood time tolegislators, thepublic, themedia, and civilsociety.

Actual time will vary, depends on local viewof what constitutes enough time to give fullconsideration. Once time set, then this could be a simpleyes/no, or it could measure how late it is byweeks or days.

Budget is possibly the mostimportant govt. policy document;the more information is availablethe better - key groups must havetime to consider it fully.

Monitor legislative documents & records. Concultstakeholders about what constitutes adequate time.

Cost: Relatively easy as this is a key document.

Trendline should show decrease in differencebetween satisfactory time and time of actualdelivery. Pace of difference will depend onincreased competence and political will of staffand legislators to open access of keydocuments to the public.

Comments: With reference to budget information, progress could also be measured by the type of budget produced, from line item (least informative) to program (more informative) to performance (mostinformative).

7) Financialreports on govt.expendituresavailable ingood time tolegislators, thepublic, themedia, & civilsociety.

Actual time will vary, depends on local viewof what “constitutes enough time to give fullconsideration.” Once time set, then this could be a simpleyes/no, or it could measure how late it is byweeks or days.

Financial reports are very importantgovt. policy documents; the moreinformation is available the better,key groups must have time toconsider it fully.

Monitor legislative documents & records. Consultstakeholders and experts about what constitutesenough time.

Cost: Relatively easy as these are key documents.

Trendline should show decrease in differencebetween satisfactory time and time of actualdelivery. Pace of difference will depend onincreased competence and political will of staffand legislators to open access of keydocuments to the public.

Comments:



Intermediate Result 2.4.2: Increased Citizen Access to Improved Government InformationIntermediate Result 2.4.2.2: Improved Dissemination of Information


1) # of wellpublicized govt.meetings opento citizens &citizen groups(functionalareas/ministriesof significantrelevance to beselected).

Govt. meetings might include one typesuch as town hall meetings, or mightinclude a variety of types of openmeetings. To be “open,” meetings shouldbe held at times and places which makethem accessible to, or do not excludewomen &/or potentially marginalizedgroups.

Indicates citizens have been madeaware of opportunity, opportunityis “real,” and they have sufficientinterest to participate.

Monitor records of govt. agency(ies) or of relevantNGOs or the press.

Cost: Depends on how many opportunities there arefor such meetings; possible to focus on moresignificant public issues.

Trendline should rise; it may decline ifpolitical tensions increase or the publicbecome apathetic or feel that it makes nodifference. May be difficult to set an accurate target for #meetings.

Comments:

2) # of jointcommissionmeetingsbetween govt.& civil society(functionalareas/ministriesof significantrelevance to beselected).

Number of public-private commissions ornumber of times that all suchcommissions meet to discuss substantialissues ex education, health.

Indicates govt. open to sharingplans, information, & ideas withand listening to civil society or keyinformed individuals.

Monitoring govt. records or through contacts withrelevant NGOs or key informants who participate injoint commissions.

Cost: It depends on number of meetings andcomplexity of political and govt. system; if no localorganization monitors, then possibly a select groupof issues should be followed.

Trend should increase although in themedium term the quality of the interchangeand the influence of the private citizens andof the commissions will be more importantthan the number of meetings.

Comments:



Intermediate Result 2.4.3: Ethical Practices in Government Strengthened

Indicator Definition and Unit of Measurement Relevance of Indicator Data Collection methods/Approx. costs Target setting/Trendline interpret. issues

1) Publicperceptions ofcorruption in thedelivery orprovision ofselected govt.services, asreported inopinion polls.

% of population (disaggregated if possibleand judged relevant) perceiving corruptionas a serious problem.

Relevant, because although thesemeasures are subjective, they arederived from surveys of either thegeneral public or most relevantclient groups; i.e., property taxpayers, water users, and businesspersons (whose assessments canindirectly affect flows of internationalprivate capital). However,perception may not always trackclosely with reality.

Survey.

Cost: High if data must be collected by USAID.Low to medium if data could be collected as part ofother surveys, particularly by local market researchorganizations.Very low to zero if surveys are undertaken by otherorganizations.

This will need to be set and interpreted withcare. It is possible with increased openness andmore media attention that public perception maybe more negative at first. Targets might need totake this into account at first and then look toimprovements. Targets could be set at intervalsof 3 to 5 years by specifying some desiredamount of change since first benchmark survey.

Comments:

2) Perceptionsof corruption bysurveys ofbusinesses orfirms attemptingto do businesswith the state.

% of owners or managers of registeredbusinesses (representing different sectorsand sizes, and if relevant ethnic/religious orregional populations) perceiving corruptionas a serious problem.

Highly relevant, because althoughthese measures are subjective, theyare derived from surveys of animportant and interested clientgroups, (whose assessments canindirectly affect flows of internationalprivate capital).

Survey or sample surveys.

Cost: Since sample is small and targeted, this wouldcost less than a public opinion survey. Would beeven less expensive if part of another survey.

Targets will need to be set and interpreted withcare. It is possible with increased openness andmore media attention that business people’sperceptions may be more negative at first.Targets might need to take this into account atfirst and then look to improvements. Targetscould be set at intervals of 3 to 5 years byspecifying some desired amount of changesince first benchmark survey.

Comments:

3) a) Time andb) real cost tocustomers ofgetting alicense(s) from aselected govt.regulatory orlicensing agency(ies).

Time would be from the moment ofapplication to the moment of receipt of thelicense.

Real cost would include cost of license,cost of “extra” payments such as bribes,time given a financial value andtransportation.

Disaggregation by social category might beinformative since marginalized or minoritygroups are often more vulnerable.

Viewed from the perspective of thecustomer, these two measures area very useful method ofdetecting improvements inmanagement and reduction incorrupt practices.

This will require a decision about which are keyareas of licenses, and then interviews (or a survey)with a cross section (i.e., rich/poor, men/women,ethnic groups) of the public involved in obtaininglicenses.

Cost: There may be records which, if reliable, wouldmake it relatively easy. But in most cases findingswould need to be confirmed though interviewing therelevant public, and this could certainly take time.

Trendline could be expected to decrease. Oneshould keep in mind that trends will alsodepend on good management and not only onreduction in corruption; and also that animprovement in services may lead to increaseddemand and that may once again increasedelays.

Comments:Agency Objective 2.4: More Transparent and Accountable Government Institutions


Intermediate Result 2.4.3: Ethical Practices in Government StrengthenedIntermediate Result 2.4.3.1 Laws and Regulations Reformed


1) Does countryhave a code(s)of conduct andother necessarysupportive andlegally bindingrules andregulationsrequiring ethicalbehavior ofelected andgovt. officials?orImprovementsmade in thelegalrequirementsaimed atstrengtheningethics comparedto a list ofrequirementspromoted.

Code and other regulations and lawsspecify and prohibit any forms of behaviorregarded as unethical, illegal, involving aconflict of interest, or otherwiseincompatible with holding office; stipulatesprocedures and agencies responsible forinvestigation, arbitration, and enforcement;provides protection for ’whistle-blowers;guarantees independence of the mainmonitoring agencies; and includes freedomof information act and financial disclosurelaws.

Could be simple binary (i.e., yes or no)indicators of a factual nature; or could beset up as some kind of quantitativeindicator comparing laws passed to lawsproposed.

Highly relevant, because withoutsuch codes there is no formalstatement of inappropriate or illegalbehavior and thus no standards ofethical behavior.

Review of country’s constitution and principalstatutes.

Cost: Low. Questions can easily be answered in ashort period through efforts of one or two missionstaff.

If country has no statutes at time of initialreview, targets could be set for the end of somespecified period when relevant codes ofconduct would be on books.

Comments: See also legal reform scale in Appendix C.

2) Does thecountry havefreedom ofinformation andfinancialdisclosure laws?

Refers to laws requiring govt. to provideand give public access to information,including a law requiring financialdisclosure of major transactions includingprocurement & expenditures.Could be yes/no; or it could be scoredagainst a number of desired reforms.

Laws requiring govt. to provideinformation to the public areessential legal instrument that givesthe public, the media, & CSOs theright to demand information; andlets govt. officers know that fulldisclosure will be required.

Perusal of government laws and regulations.

Cost: Low.

These are a test of govt. commitment todisclose information and reduce corruption. Ifgovt. is so committed, these should be passedrapidly; if not, they will not be passed, orpassed with limitations. The trend will thendepend on an interplay of ruling party/govt.determination to limit openness andopposition and/or civil society determinationto increase it.

Comments: See also legal reform scale in Appendix C.Agency Objective 2.4: More Transparent and Accountable Government Institutions

Intermediate Result 2.4.3: Ethical Practices in Government Strengthened


Intermediate Result 2.4.3.2: Oversight Mechanisms to Maintain Ethical Standards Strengthened


1) Does thecountry maintainan independent (a) auditor-general’s officeor equivalentorganization thatregularly (i.e.,annually) auditsgovt accounts;(b) inspector-general’s officethat regularlymonitors govt.contracting &procurementpractices; (c)office of govt.ethics to monitorand implementthe code ofethics in govt; d)anti-corruptionagency toprovide publiceducation about,preventbreaches of, andenforce all laws,codes, etc.regulatingcorruptpractices?

Simple binary (i.e., yes or no) indicators ofa factual nature for each of the four offices.Names of offices and division ofresponsibility will vary from country tocountry but accounting, procurement,ethics, and corruption all need to bemonitored.

Highly relevant, because without theexistence of independent institutionsfor the purpose of exercisingoversight over govt. departments,there is no check on potential illegalor unethical activity by publicofficials.

Review of country’s laws and structure of govt.

Cost: Low. Can be done quickly by USAID missionstaff.

If country does not have any of these threeoffices or equivalent at the of time of initialreview, targets could be set for the end of somespecified period when such institution could beestablished and functioning.

Comments:



2) Availability ofadequateresourcesproviding for theexistence &staffing of officesof an auditor-general, aninspector-general, ethicsor anti-corruption.

Simple binary (i.e., yes or no) indicators ofa factual nature for each of the four offices.Names of offices and division ofresponsibility will vary from country tocountry but accounting, procurement,ethics, and corruption all need to bemonitored. Obviously defining “adequate”will be problematic; but there should beresources for office space, professionalstaff, communications facilities,investigative/monitoring & arbitrationcapacity. (A money amount spent eachyear on these offices may be one simplerand only partial way of measuring this.)

Follow on to the above. Resourcesare essential to effective monitoringof ethics, and also indicate govt.seriousness of purpose.

As above and in addition govt. budget and annualexpenditure reports.

Cost: Not heavy commitment, but will require somecare to check on financial documents.

As above, but presumably might take longer.Also actual amounts will vary from year to yeardepending on overall govt. policies, revenues, &expenditures.

Comments:

3) Administrativecapacity of theauditor-general’s,inspector-general’s, ethics& anti-corruptionoffices’ equal tothe task.

Example: for auditor general a) Adequatenumber of appropriately trained staff toconduct audits?b) Adequate equipment & materials toperform work (office equipment, vehicles,financial resources for internal travel)?c) Existence of appropriate reportingmechanisms?d) Are audit reports regularly produced atprescribed frequencies?e) Are reports easily available to membersof the public or press?f) Are reports (or summaries) submitted forreview to an elected legislative body?Assessments would be either binary(yes/no) or an ordinal scale consisting of 3or 4 subjective data points. For example, d)and e) could be scaled: “most of the time,”some of the time," "rarely or never."

Highly relevant, because itsummarizes the level ofperformance of an institutioncharged with upholding ethicalstandards. If such an institutionexists, but cannot function it isessential to determine such.

Document review and interviews by experts.

Cost: Medium.

Comments:



4) % of govt.budget auditedaccording torequiredstandards in thelast financialyearand/or% of govt.contracts &procurementsreviewed by theinspectorgeneral’s office.

% which have been audited (according tostandards and requirements set oraccepted by a reputable agency private orpublic or training, domestic or international)out of a total amount of the budget or out ofthe total # of govt. agenciesIn the case of review of contracts, similardefinition but reference to ’contracts’ and“inspected”

Auditing of govt. accounts, andinspection of govt. contractingmechanisms and processes,especially when these conform tohigh standards, are essentialinstruments of monitoring anddiscouraging waste and corruption.

Review of govt. audit office & inspector general’soffice annual reports.

Cost: Assuming govt. agencies provide annualreports in good time, this should not be a demandingindicator to monitor. If govt. agencies do not do so,possibly firms employed for this purpose may havethe information. If not, interviews will be necessarybut this will probably not reflect on the wholeprocess.

Trendline should rise as training increases(which depends to some extent on theexistence of a pool of people qualified to betrained), resources for the auditor’s andinspector general’s offices grow or at leastremain adequate, and government acceptsnew practices of improved auditing andinspection.

Comments:

5) Either % or #of govt.departments &agencies withmajor auditfindings (i.e.,that find a majorerror) and/orequivalent forinspections ofcontracts &procurements.

% of those departments in which there is amajor finding (a term which means thatthere is a substantial error) of alldepartments in which audits are carriedout.

# refers to # of govt. departments in whichthere is a finding.

This can indicate two very differentbut both important processes, whichwill look to different trendlines. Firstit can indicate that the auditors areinvestigating more effectively.Second, it will indicate that theauditors are finding fewer errors andtherefore there is less evidence ofwaste or misuse of funds.

Review of auditors’ reports; augmented by interviewswith auditors.

Cost: As above.

Trendline here may be complex. If one is in asituation where there have been widespreadmisuse of funds and no serious auditing, & amajor focus of the program is on strengtheningauditing, then one should look to an upwardtrendline as the new system reveals problems.If one is in a situation where there has beenwidespread misuse of funds and auditing hasbeen systematic and the focus is on anti-corruption programs, then one might expect adownward trendline as misuse of funds isreduced. It is also possible that the firstexample will be relevant at first and thesecond will occur in time so that the trend-line should be expected to rise at first andthen fall.

Comments:



6) % or # ofexamples of fullinvestigation ofsignificantbreaches ofprocedures orethics at rela-tively high levelbeing pursuedfully and fairly totransparentoutcome, & ifnecessary toenforcement.

% of significant cases in which such fullinvestigations occur compared to all suchcases.

# would refer to the # of occasions onwhich this occurred.

Examples can be in the form of brief “casestudies” illustrating that the system isworking and at both high and low levels ofgovt.

Only way of observing whether therules, the agencies, and theresources are coming together andhaving the desired outcome.

Review of responsible govt. agency records; possiblyof newspapers; interviews with complainants orwatchdog NGOs.

Cost: Should not be onerous if govt. is relativelyopen and if only selected cases are taken up;obviously the more cases the more time.

If there have been no such examples, thepositive trend could be expected to rise to a fewsignificant examples which provide evidence ofgovt. will to reduce corruption at high levels.Positive conclusions may be tempered by thecontinuation of cases which are known to havebeen overlooked or only partially pursued.

Comments:

7) Effectivenessof legislativeoversight.

Overall, a subjective measure, but basedon concrete pointers which might include:a handbook of regulations and procedureshas been distributed to all legislators andmade available to the public; role ofopposition and minor parties is clear andunfettered in practice; the public and presshave access to the legislative process;main committees meet; the public andpress have access to individual MPs;voting records are public; the legislature isable to carry on its business withouthindrance from the executive.

Relevant to all settings. Rules mustbe transparent, available, andenforced for the legislature to beeffective.

Analysis of written handbooks, documents, andlegislative records, survey of legislators; a case studyof a specific legislative issue could be useful.

Cost: Medium cost except a case study would berelatively more expensive.

Initial resistance and spotty enforcement mightbe expected.

Comments: See also legislative development indicators under IR 2.4.5.



8) # of NGOswith specializedexpertise &capacity tomonitor,analyze, andpublicize govt.corruption.

NGOs that have a mission to focus oncorruption and have personnel with thenecessary specialized skills.

In addition to independent govt.agencies, it is necessary to havenon-govt. agencies to observe thereality of independence and theeffectiveness of these agencies, toact as consumer protectors andadvocates, to make the publicaware, and play a policy advocacyrole on this topic.

Observation of NGO activities and publications in thearea of corruption.

Cost: Relatively inexpensive. It is assumed that therewill be few of these types of NGOs.

If one begins with none, or one or two, then thetrendline could be expected to rise slowly. Itmay soon become static which is not aproblem; then the more important questionrelates to their effectiveness rather thantheir number.

Comments: See also civil society indicators.

9) Quality ofmediainvestigation andreporting ofcorruption.

Index might be appropriate; taking intoaccount a) numbers of papers, journals,radio and TV programs coveringcorruption; b) effort invested ininvestigation; c) use of documentary andinterview findings; d) a variety of views(including those of the investigated); and e)perseverance with an issue.

Media has an especially importantrole to play as a watchdog andpublicizer of corruption. Some biasis inevitable, but it is important thatthe media keeps some balanceinsofar as it listens to and reports allsides of the case.

If the media is not independent, thisindicator is irrelevant.

This will depend on a careful monitoring of themedia. There may be NGOs or the media itself maydo this. If not, then it may be possible to rely on apanel of experts; or it may be necessary to set oneup; and they should make their assessments on afixed set of criteria along the lines suggested in thesecond column. Suggest choosing selectedpublications and broadcast programs.Cost: If media performance is already beingmonitored by a credible and autonomousorganization, then this is easy. Establishing andconsulting with a panel possibly every year or twowill not be demanding.

There are many variables, including levels ofcorruption, govt. openness, public interest,editorial policy, editorial integrity, NGO interestand activism, training and experience ofreporters, etc. Donor programs can add apositive influence.

Every two years should suffice.

Comments: See also media indicators in Section C, Tab 8, IR 2.3.4.



Intermediate Result 2.4.3: Ethical Standards in Government StrengthenedIntermediate Result 2.4.3.3 Professionalization of Staff and Operations


1) % of newofficers hired bycompetitiveexam conductedin a "fair"manner.

% of all officers.Disaggregation by gender and other socialcategory might be relevant since minoritiesare often more vulnerable to unfairpractices and extortion.

Highly relevant. Relates to theprofessionalization of recruitmentinto the civil service which is asignificant step towards raisinglevels of ethical behavior.

Requires review of govt. requirements and interviewsabout practices with those who administer, thoseaffected and those who monitor them Cost:Depending on govt. records & ’openness’ should notbe too onerous on time and should be inexpensive.

Indicator value could change rapidly in responseto a new rule requiring competitive exams formore positions

Comments:

2) Averagesalaries ofselected levelsof civil servantas percentage ofsalaries ofroughly theequivalent levelsin the privatesector.

“Average” can be derived in a number ofways; i.e., select 5 categories/levels ofprofessional or technical personnel in thecivil service and determine an average oftheir actual salaries, or average for thoseposts; and then do the same for roughlyequivalent levels in the private sector.Another way might be by comparingsalaries of people with the same educationand experience.

Poor public service salaries areconsidered one cause of corruption,because people who are, or feel,seriously underpaid, feel greaterneed and even justification in takingmoney.

Govt. records, and records of the private sector,such as company records, or research by a chamberof commerce, or university or think tank, or byinterviewing some key informants. Establishingrough equivalence will take some time with keyinformants.Cost: Govt. records on salaries and benefits shouldbe readily available in many countries. Private sectormay be harder to track down; there are a number ofreasons why they might not want to disclose fullinformation. If a chamber of commerce orprofessional organizations keep records, then it willnot be too time consuming; if not, then a fewselected corporations who are willing will need to becontacted.

This is more of an indicator that one cannormally use to anticipate or explain corruptionthan one which is susceptible to direct donorinfluence or correction and usedfor purposes of program performance. Also itis difficult to generalize what the rightcomparative averages should be. If there is aspecific program to raise the salaries of keypublic servants, and if the govt. has theresources to do so, then one might anticipatesome reduction in the gap. Some discrepancybetween public and private salaries is to beanticipated. Job security in the public sector andother benefits may offer some balancewith higher private sector wages.

Comments:

3) # or % ofgovt. financial/accountingsystemsoperating underIntegratedFinancialManagementSystem (IFM).

Systems refers to those systems thatoperate in ministries, departments, stateowned enterprises, etc.

IFM is a key instrument for moreeffective management andmonitoring of revenues andexpenditures, intended to lead toenhanced information andmanagement and reducedopportunity for hidden transactions.

Reviewing with govt. officers’.budgetary/accounting/financial practices in variousagencies.

Cost: This should not be too time consuming. If thereis an attempt to measure qualitative improvementsover time in the implementation of the system thatwill take more time and advice of experts.

Trendline should be upwards if the govt. hascommitted itself to implementing the systemand training the personnel to manage it. Thepace of change will depend on resourcesdevoted to training and implementing andalso the number of people in govt. (who stayin govt. after the training) who have theprerequisite expertise to receive the training.

Comments: Could also construct as the % of government budget managed under IFM.



4) Time and realcost tocustomers of“connection” toutilities by agovt. utilitiescompany oragency.

Time would be from the moment ofapplication to the moment of receipt of thelicense.Real cost would include cost of license,cost of “extra” payments such as bribes,time given a financial value, andtransportation. It might be simpler to report on timeseparately and not give it a financial value.Disaggregation might be informativebecause of vulnerability of minorities.

Viewed from the perspective of thecustomer, these two measures area very useful method ofdetecting improvements inmanagement and reduction incorrupt practices.

This will require a decision about which are keytypes of connections, and then interviews (or asurvey) with a cross section (i.e., rich/poor,men/women, ethnic groups) of the public that hasgone through the process.Cost: There may be records which, if reliable, wouldmake it relatively easy. But in most cases findingswould need to be confirmed though interviewing therelevant public, and this could certainly take time.Records are unlikely to permit easy disaggregation.

Trendline could be expected to decrease. Oneshould keep in mind that trends will alsodepend on good management and not only onreduction in corruption; and also that animprovement in services may lead to increaseddemand and that may once again increasedelays.

Comments:

5) Transparentprocedures fora) licensing,b) procurement,c) privatization,d) tax collection,&e) customs.

(One “collective”measure or 5separatemeasures.)

Index might be appropriate taking intoaccount the following (these are illustrativeexamples): a) timely publication for all bids,tenders, auctions, applications, etc.; b)user-friendly information; c) publication ofdecisions; d) clear rules of appeal; e)examples of fair appeal processes; f)sense among stakeholders of impartiality indecisions; g) sense of impartiality amongthe informed public.

Panel of experts could use criteria such asthese as basis for discussion, or afterdiscussion, each member could score eachcriteria along a scale from 1 - 5, & thesescores could be aggregated and averaged.

Transparency of all stages of keytransactions in which govt. has thepower to select among privatepartners/contractors, to grant orwithhold permissions, licenses orexemptions, or facilitate or hinderprocesses, and involving either orboth large numbers of people andlarge an-mounts of money, isfundamental to reducingopportunities for corruption. Thisindicator seeks to measure how fargovt. has come in establishing andpracticing such open and fairprocesses.

A panel of experts — possibly a mix of lawyers withexpertise in these areas, business persons,academics, members of business associations, thinktanks, ex-senior govt. officials and would have to beestablished. Members of the panel would need toagree on appropriate criteria and how to score andweight the criteria. They would need to meet everyone or two years.

Cost: Setting up a panel will initially take time, asthere has to be a high level of confidence in themembers’ expertise, and willingness to participateregularly. Also while impartiality of members shouldbe sought, it may be necessary also to seekbalanced membership. Thereafter the proceedingsshould not be too costly in money or time.

Benchmark and trendline will obviously dependon how “bad” the situation is at the outset, howwidespread & longstanding the practices ofcorruption are. (Is there what might be referredto as a culture of corruption? Have public andprivate actors known any other way ofconducting business?) And how demanding arethe changes, the novelty and complexity of newundertakings such as privatization or of newprocedures ex property valuations, proceduresfor appeal etc. Therefore how much training isrequired, how strong a political will is needed,what incentives public servants need to abideby the rules, & how much public education willmake a difference. If there is a serious problemof corruption, the benchmark will be low andthen, assuming there is political will andrelatively effective programs aimed atenhancing transparency, some improvementshould be anticipated. But it should also beanticipated that stakeholders who benefit fromcorruption will consistently seek to circumventnew procedures and provide alternativeincentives. A smooth upward trendline thereforeshould not be expected.

Comments:



Intermediate Result 2.4.4: Strengthened Civil-Military Relations Supportive of Democracy


1) a) Evidenceof militaryrespect forconstitutionallimits;or b) # ofviolations by themilitary ofconstitutionallimits; andc) # of civilianorders to themilitary, whichviolateconstitutionallimits.

a) Whether each military activity has beenapproved by executive branch andcongress;

or

b) and c) # of significant violations ofconstitution and/or legislature/executiveoversight. (If human rights, women’s rights,protection of minorities are enshrined in theconstitution, violation of these rights shouldbe included here as demonstrating “lack ofrespect.”)

Proper formal process of oversightneeded in democracy.

Lack of violations or low number ofviolations does not necessarilymean that there is civilian control.

Following key events limiting/affecting military inpolitics. May not always be easy to track.

Cost: Moderate?

Comments: Some presidents declare states of emergency to preempt legislatures. Some military take that action in some areas.

2) % of govt.budget formilitary;orb) $ amount.

a) % of total budget devoted to the military;orb) Total military budget in US dollars maybe relevant in situations where total budgetchanges dramatically.

Military budget should be adequatebut not excessive, but theappropriate amount will differ, forexample, based on external andinternal security needs. Too low abudget could cause military alarmand threaten civilian control.

Calculate from all main budgeting instruments. Willneed advise from expert panel on what appears tobe appropriate and on extent of “security threats.”Cost: Little since much of this information should bein the budget. If some is kept secret, this informationmay be unobtainable.

Judgment needed as to proper size. This willvary between countries, and trendline may goup and down depending on security demands.

Should be tracked annually.

Comments: Military may receive extra-budgetary resources. In using this indicator, need to be clear about how defining the military, and whether in-budget resources represent the bulk of what the militaryreceives. a) May decline if there is good economic growth and the overall budget is growing. b) Should be adjusted for inflation.



3) a) % ofmilitaryexpenditurederived fromsources otherthan govt.appropriations;or b) $ amount.

% of military expenditures not covered bypublic budget.

Private sources of military budgetdecrease civilian control.

Data unlikely to be available.Knowledgeable Informant needed for this and thenthey may only be able to produce an estimate.

Cost: Complicated to identify sources and amounts.

% should go to zero, but will take time if militaryis well established in private sector. Not worthtracking annually.

Comments: There is a serious issue in countries with hyper inflation and when the military is involved in the private economy.

4) a) Ratio ofcivil/militaryexpenditure inkey functionalareas;orb) Ratio ofcivilian/militaryrun facilities/services

Civilian/mil expenditure (recurrent andcapital) non-defense related areas, i.e.,civic action, crime fighting, health,infrastructure, environmental protection,etc.

It can be risky to civilian control ofthe military if military involvement incivilian society is widespread anddeep. Military involvement in narco-trafficking in Latin America is onesuch example. Civilian control canbe safeguarded by building upcivilian capacity in these areas.

Monitoring military activity in those areas where theyare/have been active; if there is an open militarybudget, that will be informative. If cannot get budgetdata, will need to reconfigure as # key functionalareas in which military is very active.

Cost: Low to high, depending on availability ofbudget data.

In each functional area mil role should cease ascivilian capacity increases.Change may not be very dramatic nor worthtracking annually.

Comments: This is perhaps the major issue/indicator since civilian capacity is so low in many areas. The key point is to have a plan to increase civilian capacity and refocus military on its key role ofdefense. Even in long-term Western democracies, military may have non-military roles - vis. US Army Corps of Engineers. b) is a proxy when a) is unavailable.

5) # ofsubstantialchangesintroduced todefense budgetand legislationas a result oflegislativeinitiative.

Changes introduced as a result oflegislative committee and/or plenaryinitiative influence the outcome of defenselegislation and budget. “Substantial” isdifficult to define in advance; possibly itcould be agreed to post hoc when itbecomes clearer whether a decision wassignificant or not. Could limit to key areas.

Legislature should not be a rubberstamp. Assumption is that the maininitiative comes from the executivebranch possibly with large militaryinput.

Observe process of bills, hearings, debates, and finallegislation; or an expert panel could makeassessments.

Cost: Little, particularly if a panel can be called uponor if an NGO does the monitoring.

The trendline should be expected to be upward;but pattern of change will depend on theauthority given to, the information provided to,and the expertise of, the legislature. Also ondegree of problems with the military budget, etc.Trendline should eventually level off.

Comments:



6) a) NGO’sperception ofvalidengagementwith govt. ondefense policyissues.

a) Perception could be scale: frequent andsubstantial on all key issues; reasonableengagement on most key issues, limitedengagement and seldom on key issues,and no engagement. “Valid engagement”refers to a situation in which NGOs feelthat they were given the opportunity topresent their views, were listened to, andtaken into account in policy decisions.

Civilian society should have aninfluence over military policy andexpenditure. It might be preferableto measure changes that resultedfrom NGO input, but this will bedifficult to trace and NGOs may wellgive contradictory advice.

Observe bills, hearings, position papers andlegislation; interviews with key NGO informants.

Cost: Moderate depending on # of NGOs to beinterviewed. Could focus on key issues to limitinterviewing.

Movement in trendline will depend on NGOcapacity as well as civilian govt. and militarywillingness to countenance input.

Comments:



Intermediate Result 2.4.4: Strengthened Civil-Military Relations Supportive of DemocracyIntermediate Result 2.4.4.1: Constitutional and Legal Reforms Authorize Civilian Authority

Indicator Definition and Unit of Measurement Relevance of Indicator Data Collection Methods/Approx. Costs Target Setting/Trendline

1) # or % oflaws improvedin comparisonwith a list ofchangesproposed.

Yes/no against a list of promoted reforms. All military should be governed bymodern, laws; bad or outdatedlaws can produce structuralproblems.

Review of legislation with help of military legislationexpert.

Cost: Will require consultation with military andconstitutional experts.

Good laws reviewed by competent legislature.Legislative capacity problems need to bereviewed in setting targets.

Comments: See also Appendix C for Legal Reform Milestone Scale.


Agency Objective 2.4: More Transparent and Accountable Government InstitutionsIntermediate Result 2.4.4: Strengthened Civil-Military Relations Supportive of DemocracyIntermediate Result 2.4.4.2: More Accurate Information Publicly Available


1) # of publicmeetings onmilitary andsecurity issuesand civilian-military relations.

Public events. For example, conferences,roundtables & speeches on these issuesand on civilian-military relations.

Expresses the opportunity to openlydiscuss & exchange on civilian-military relations.

Reviewing notices in newspapers & announcementsby universities & think tanks.Cost: Time of officer(s) in USAID or NGO partners orgrantees. Since the number will probably not be thatlarge, it should not be too time consuming.

Steady increase from initial base should begoal. May level off over time. Will be eventdriven so may be very difficult to set targets for.

Comments:

2) # of TV &radio showsdiscussing themilitary andsecurity issues.

Public media events discussing civilian-military relations. Could also convert to ascale of frequency: (1) no coverage; (2) 1-4programs per year; and so on.Could include news programs that covercivilian-military issues as one segment.

Indicates informationis readily available to the public.

Reviewing TV & radio schedules as available.Cost: Medium amount of time of officer(s) in USAIDor NGO partners or grantees or commercial mediamonitoring service. The last is the cheapest when itexists.

Steady increase should be the goal. May leveloff over time. May also be event or crisis driven.

Comments: Data will not be complete, but an estimate will be worthwhile.

3) # ofnewspaper andjournal articles.

# of newspaper and journal articlesdiscussing military & civilian-militarysubjects in selected publications. If thereare many of these, could convert toaverage # per month.

Indicates availability of publicinformation on the subject.

Systematic and regular review of a constantrepresentative sample of newspapers. If money isavailable, then a content analysis might beappropriate.

Cost: Time of USAID officer(s); could be shared withUSIS and/or this work could be contracted to aninterested NGO. Commercial monitoring service mayalready cover main periodicals.

Steady increase should be the goal, at least inthe medium term. Then it may reach a ceiling ifmajor controversial issues are resolved. May beevent driven. May be difficult to set targets.

Comments: Choice of appropriate newspapers will change as papers fold and start up.

4) NGO rating ofavailability/quality of military-providedinformation.

Index could be constructed as follows: (1)no information; (2) some information, but oflimited significance or quality; (3) someinformation, quality and/or significancebetter; (4) more information, goodsignificance, reasonable quality; (5)considerable information of high qualityand significance.

Measures military openness. Survey of NGOs with interest in military matters.

Cost: Probably low to moderate. May not be manyNGOs to be interviewed, but scale needs to bedeveloped and pre-tested and then applied.

Would hope to see rising quantity, quality, andsignificance as military becomes more open.Obviously, there are matters of national securitythat should remain confidential.

Comments:



5) Itemization ofthe militarybudget.

Yes/no, annual budget itemizes resourcesfor the military into meaningful categories.

Demonstrates increasedtransparency; often correlates withcivilian control.

Budget.

Cost: Low.

May be a one-time change.

Comments: May be more interested in the degree of itemization. For example, in Guatemala, there is some degree of itemization but the categories are rather general and vague.

6) # publicationson military orsecurity affairspublished bythink tanks,NGOs,watchdoggroups.

As noted in the indicator. Increases availability of informationand analysis, and variety ofperspectives, about the military andsecurity topics.

Monitoring of relevant groups and their publications.

Cost: Low.

Should go up over time but may be event orcrisis driven. May be difficult to set targets forin later stages of development.Indicator more useful at earlier stages ofdevelopment.

Comments:



Intermediate Result 2.4.4: Strengthened Civil-Military Relations Supportive of DemocracyIntermediate Result 2.4.4.3: Increased Civilian Competence in Defense and Security Affairs


1) # of civilianlegislators,legislative staffand govt. stafftrained in military& security issueswho say that thetrainingimproved theirskills and theyare making useof the skills.

# of individuals out of total trained whohave been trained and who can say theyhave used their training and preferably cangive concrete examples of how they havedone so. Disaggregate by category ofpersonnel.

It is essential to meaningfuloversight that there exist a pool oftrained civilian advisers andoversight managers. Direct outcomeof training; trainers should have anduse this information.

Training records and survey of trainees. Informantsshould always be requested to back up positiveanswers with example(s).

Cost: Medium.

Number should increase depending on extentof training and education initiatives. Report onnumber out of total trained. In setting targets,need to think about whether barriers to the useof new skills other than knowledge exist.

Comments: Indicator can be applied to the expected outcome of TA as well if significant TA given.

2) Amount ofresourcesavailable for thefunctioning oflegislativeagenciesresponsible formilitaryoversight.

Size of budget of oversight agencies/# ofstaff/existence of offices.

In order to function, these legislativeand other govt. agencies needbudget, staff, and offices. Indicatoris relevant only if there is such alegislative agency.

Directly contact agencies for information onresources. Budget for legislature should also breakdown expenses for this purpose.

Cost: Telephone calls/visits — not heavy.

Should increase to levels allowing efficientfunctioning.

Assess increases in light of inflation.

Comments: Sometimes agencies are authorized but not funded.

3) # of NGOswhich areactively engagedin working onmilitary andsecurity issues.

# of NGOs wholly or partially dealing withsecurity & defense. If they are selfidentified, then it may need to be confirmedby independent observers. “Activeengagement” needs to be defined.

Competence needed in civil societyto provide oversight experts.

Record kept of NGOs involved in this area.

Cost: Maintaining records should not be timeconsuming as there are unlikely to be a largenumber.

# should increase for a period; then probablyreach a ceiling.

A large number is less important than having afew strong NGOs.

Comments:



4) # ofpublications byNGOs on civil-military &military/securityissues.

# of publications. “Quality” publicationsmight be insisted upon when NGOexpertise & development reach a relativelyadvanced stage, and when agreement canbe reached on how to define “quality.”

Shows capacity of NGOs/civiliansociety to participate in opendiscussion.

Counting publications (and possibly limiting this toquality publications once agreement is reached on adefinition).Cost: Interview NGOs that undertake this work.Keep records and find criteria to distinguish moreserious from less serious materials.

# should increase as public discussion of themilitary opens up; but a ceiling may be reached.

Comments: Could also develop a scale combining quantity (coverage of all major issues) and quality (use of facts and figures, analysis, exam). Of different points of view, projection of implications ofparticular policies for the country, etc.

5) # of civiliannon-govt.experts withspecializedtraining inmilitary &security affairswho say theymake use oftheir training.

# having completed formal training inspecial programs in defense policy (out oftotal trained) who can point to specificexamples of the use of their newknowledge and skills.

There needs to be some formal andspecialized training of civilians tooversee various aspects of militaryand security policy. Direct outcomeof training — trainers need thisinformation.

Training records and survey of trainees.

Cost: Medium but if few trained, burden should below.

Assess whether barriers other than lack ofknowledge are likely to keep some traineesfrom using skills.

Comments:

6) # of mediarepresentativeswith specializedtraining whoreport use oftraining.

# of reporters and editors with such trainingwho say they use it and can give examplesdemonstrating how the training has raisedthe quality of their work.

Good reporting on military/securityneeds knowledgeable reporters &editors. Direct outcome of training— trainers need this info.

Training records and survey of trainees.

Cost: Medium, although if few trained, burden shouldbe light.

Assess whether barriers other than knowledgemay preclude trainees from using newknowledge prior to setting targets.

Comments:

7) Assessmentof the quality oflegislativedebates onmilitary policy.

See index in Appendix C. Quality oflegislative debates on budgets and lawsaffecting the military, and defense policy,determined by a panel; and based on useof data, policy alternatives raised, timedevoted, different viewpoints. Possible useof scale from no debate, low quality,reasonably informed, well informed to highquality debate.

This is key indicator ofdemocratically elected legislature’scompetence and power of oversightof the military.

Observation of debates; but it will require a panel oflocal experts using criteria as suggested.

Cost: Probably not heavy, depends on attentiongiven to military, and availability of local experts.

Quality of debates should improve aslegislature (members and staff) becomesmore fully informed about main issues.

Comments:



Intermediate Result 2.4.4: Strengthened Civil-Military Relations Supportive of DemocracyIntermediate Result 2.4.4.4: Increased Civilian and Civilian-Military Networking

Indicator Definition and Unit of Measurement Relevance of Indicator Data Collection Methods/Approx. Costs Target Setting/trendline Interpret. Issues

1) # of publicfora onmilitary/securityorganized byNGOs.

# of public, open, publicized fora organizedby NGOs.

Shows NGO capacity to organizeand participate in open andinformed discussion.

NGO records.

Cost: Low.

# should increase. May be difficult to set targets.There could be sudden increases due to crisesor scandals.

Comments:

2) a) # ofoccasions NGOswork with govt.on public civil-military policy, orb) # key issueson which NGOswork withgovernment oncivil-militarypolicy.

a) # of times NGOs consulted on policy &legislation.

b) Requires quality criteria to beestablished. For amount and kind ofengagement.

Shows involvement of civiliansociety.

Survey NGO directors; NGO records.

Cost: Low if record keeping is put into place earlyon and maintained.

# should increase. May be difficult to settargets for a) but will show a more quicklyrising trendline than b).

Comments:

3) # universitycourses dealingwith military orsecurity issues.

Count of such courses at all universities. Indicates military and securityissues being dealt with seriously ata civilian institution and participantsinclude civilians and possiblymilitary personnel jointly.

University catalogues, interviews with universitypersonnel.

Cost: Low.

Would hope to see an increase but will level offat some point.

Comments:

4) # of trainingcourses thatinclude bothcivilians andmilitary.

# of training courses on civilian-militaryrelations or defense policy involving bothcivilians and military.

Training involving both groups ispreferable.

Have informants identify courses; also there shouldbe training records that could provide thisinformation.

Cost: Low.

# should increase and then plateau.

Comments: Courses may vary in length from a year to short-term.



Intermediate Result 2.4.4: Strengthened Civil-Military Relations Supportive of DemocracyIntermediate Result 2.4.4.5: Reduced Size of Military-Security Forces

Indicator Definition and Unit of Measurement Relevance of Indicator Data collection Methods/Approx. Costs Target Setting/Trendline Interpret. Issues

1) % of militaryforcesdemobilized.

Discharged and weapons turned in. Direct measures of reductions inforce.

Military personnel records.

Cost: Low, counts should be available. Accuracycould be a concern.

May decrease quickly (in wake of conflict orpeace accord) or gradually.

Comments:

2) % ofdemobilizedmilitary/securityforces who havebeen resettled.

% of all forces demobilized who have beenreabsorbed into civilian society (i.e., theyhave received land, agriculture inputs ortraining; they have jobs, etc.).

May set standard for reabsorptionmodestly, i.e., receiving training or, moreambitiously, jobs, agricultural landproducing crops.

Not a direct measure but rather auseful adjunct in the wake of largescale demobilizations whereconcerns about violation and largescale social disruptions loom large.

Military records, records of those assisting withresettlement, surveys of the resettled.

Cost: Low to medium depending on the availabilityof good records.

Would hope that forces would not bedemobilized until a resettlement program is inplace. Would hope that percentage would starthigh and stay high.

Comments:



Intermediate Result 2.4.5: More Effective, Independent, and Representative Legislatures


1) a) level ofconfidenceamong politicalactors (ormaybe justlegislators) thatlegislature hasthe capacity toperform itsfunction; b) levelof confidenceamonglegislators thatlegislature actsas anindependentbody; c) level ofconfidenceamong citizensthat legislatureacts as a checkagainst theexecutive (inpresidentialsystem); d) levelof confidenceamong citizensthat legislaturerepresents theirinterests.

% of legislators who agree/feel that .... or% of citizens who feel/agree that ...

Two sample categories are beingsuggested here: legislators & citizens; and4 different types of questions focusing on:legislature’s capacity, independence, abilityto check exec branch, & represent people.

Detailed definitions will depend on thesample and specific focus of survey.

Dissaggregation could be useful.

“Level of confidence,” particularlyamong citizens, may be animportant high level indicator: if thecitizens have confidence, this couldbe a very significant measure ofimprovement in the role of thelegislature. It is therefore moreappropriate in more mature latetransition/consolidation situations.It is also probably a better planningindicator than a programperformance indicator, given thehigh levels of investment and timerequired to improve performance tothe level that citizens could beexpected to pick up change.

Survey either of legislators or some other selectstakeholder group or of a representative sample ofcitizens.

Cost: Survey of legislators would be inexpensive;national survey would be expensive. However, thecosts would be reduced if the questions could beincluded in an already established survey instrument.

Interpretation here will always be difficult andrequire not only statistical expertise but politicalinsight. There may be lags in the time taken forimprovements to be appreciated by the public;the legislature may be doing a good job, andone scandal, or bad press publicity, coulddetract from the perception of the public; pooreconomic performance and other factors couldlead to a negative assessment even if thelegislature is being effective.

Comments:



1) b) Index forquality oflegislativeprocesses.

See Appendix C, Section D.2. for adescription of this index.

This is the purpose of much of thework on legislative strengthening;but to capture quality is difficult; thisindicator and accompanying scaleattempt to do so by taking a varietyof factors into account.

Measures effectiveness andrepresentation.

An expert panel might be convened (possiblyannually) at the end of the legislative session to ratethe quality of debate and legislative process. Eachmember should independently rate that session oneach factor using a scale such as suggested incolumn 2. Then the panel would convene to discusstheir ratings, perhaps with an opportunity to altertheir ratings. Scores of individual members would beadded, then averaged. Both total scores and extentof initial and final variability would be of interest. Cost: If qualified people are available, this requiresorganizing a panel, agreeing on factors to be used inthe scale and then calling a meeting once a year. Soafter initial effort this should not be too timeconsuming.

Targets should be modest. Given training,technical assistance, and new equipment thetrendline should be expected to go up, butpossibly slowly, since quality should not beexpected to change dramatically. It mightdecline temporarily if many inexperiencedmembers are elected, resources decline, or theexecutive branch reasserts its dominance of thelegislative process.

Comments:

2) Degree oflegislature’soversight ofExecutivebranchbehavior.

An index which could contain the followingelements: a) review of legislation drafted by the

executive; and b) review of executive branch

implementation of the law andexecutive branch behavior.

Scores of 1-5 could be totaled in eachelement. More elements may be needed tobetter specify what is involved inoverseeing executive branch behavior.

Partially captures independence,measures oversight.

Panel of experts.

Cost: Could be done annually or every 2 years.Selecting a panel at the outset may take time aspeople need to be qualified and prepared to serveover time.

Discerning clear trends will not be simple. Overthe medium to long term, the trend should goup, and then flatten out. In the short to mediumterm, the trend is likely to go up and downdepending on capacity, quality anddetermination of legislators, on the importanceand divisiveness of specific issues, on the partystructure in the legislature etc., as well as on thebehavior, record and attitude of the executive.

Comments:



3) a) % of total,substantiallegislative billsemanating fromthe legislature;b) the % of billsdrafted by theexecutive thatare substantiallyamended by thelegislature.

Can be quantitative in measuring number,but needs informed qualitative judgment asto what constitutes substantial“amendment;” and for the alternativewhether legislative initiatives other thanbills should be considered.

Relevance is that MPs are not onlyjoining the policy debates but areinfluencing outcomes. a) allows for cases where the

legislative body takes theinitiative;

b) assumes that the executiveinitiates most of policy agenda.

Monitoring & comparing bills at first submitted withthose finally passed. Interviews with legislators orother informed observers will be needed to check onattribution as who is responsible for what will notalways be clear.It might make sense to focus only on key legislativeprocesses, or on areas targeted for improvement.

Cost: Fairly low unless a large amount of legislationinitiated; in which a smaller sample should beselected for observation.

As legislature becomes more effective (as aconsequence of training, use of the committeesystem and improved research and informationsupport), the trendline should rise. But it shouldnot be expected to be steady and will depend tosome extent on issues that arise, the politicalenvironment and effectiveness and initiative ofthe executive branch. Elections and changes inthe composition of the legislature might changethe trendline.

Comments: It is possible that the number or percentage of bills initiated by the legislature will increase when the same party is in charge of both the executive and legislative branches. May be useful toexamine a) in conjunction with % of those bills passed.

4) % of newlaws which, asbills, were a)accompanied bya writtentechnicalanalysis, opinionpapers, and/orlegislative study;and b) were thesubject of acongressionalpublic hearing.

As stated in the indicator, a) and b) can beseparated or reported as one indicator withtwo standards, both of which must be met.Latter presents a more rigorous standard.

Proxy for quality, effectiveness, andrepresentation.

Identify key laws passed during the year, or take arandom sample. May be able to contract with NGOto collect data.

Cost: Low.

If treated as two indicators rather than one,trendline may increase more rapidly.

Comments:



Intermediate Result 2.4.5: Strengthened Representation, Effectiveness and Independence of LegislaturesIntermediate Result 2.4.5.1: More Effective and Democratic Internal Management Systems


1) Regulations &procedurespassed enablinglegislature tooperate moreefficiently &democratically incomparison witha list of reformsproposed.

Minimal requirements of such rules include1) clear and workable procedures relatingto roles & rules for debate &documentation in plenary & committeesessions; 2) a legitimate role for allmembers, including opposition members,to speak, vote, introduce legislation,participate in committees & share in theresources necessary to these activities; &3) full oversight and questioning of theexecutive.This could be a yes/no measure; or it couldbe a # of improvements compared to a listof improvements.

Relevant to all settings. Rules mustbe transparent, available, andenforced for the role of thelegislature to be democratic &effective.

Analysis of written handbooks and documents;selective reading of legislative minutes; survey oflegislators. Interviews with key observers would helpin understanding significance of various rules andpossible covert impediments.

Cost: Minimal; although a case study would involvehigher costs.

This should be expected to rise, but the contextwill determine the pattern. For example, much ofthe necessary legislation & rule making mighttake place at once, and refinements andimprovements follow; or reforms might be“squeezed” out of a reluctant incumbent party orexecutive branch one at a time.

Comments: See also the Legal Reform Milestone Scale in Appendix C.

2) Extent ofimplementationof regulations &proceduresenablinglegislature tooperate moreefficiently &democratically.

A subjective measure, but based onconcrete pointers which might include: ahandbook of regulations & procedures hasbeen distributed to all legislators and madeavailable to the public; role of oppositionand minor parties is clear & unfettered inpractice; the public & press have access tothe legislature process; main committeesmeet; the public & press have access toindividual MPs; voting records are public;the legislature able to carry on its businesswithout hindrance from the executive. A list of such key pointers to measureextent of implementation can be agreedupon and a multiple item scale worked outas a measure.

Relevant to all settings. Rules mustbe transparent, available, &enforced for the role of thelegislature to be effective.

Analysis of written handbooks, documents, &records; a case study of a specific legislature issuecould be useful.

Cost: Moderate cost, depending on number ofregulations and procedures examined.


Low enforcement may be due to low capacityrather than weak political will.

Comments: See also the Legal Reform Milestone Scale in Appendix C, which would allow combining this indicator and the previous one.



3) # of monthseach year thatMPs and staffreceivepromisedremuneration intimely manneror averagedelay inpayment ofsalaries to MPsand staff in # ofdays per month.

Unit would be # of months; or # of daysper month.

This assumes salaries paid monthly.

Relevant in all situations, but mostlyin cases where tardy paymentshave been problem for the effectiverunning of the legislature.

The legislature’s capacity forinternal management is critical to itsability to have an effective policyand oversight role, as well as tointeract constructively with civilsociety and constituents.

Review of records of legislature’s financialdisbursements; key informant interviews; annualmember and staff surveys of satisfaction.

Cost: State of record keeping will determine level ofeffort required; but in general it should not be costly.

Trendline will depend on how severe theproblem was; the seriousness with which thelegislature is taken by the executive; theavailability of govt resources; & the legislature’sinternal management system. With support for alegislature strengthening the trend should moveup rapidly, but it could “backslide” if resourcesdecline or legislature is unable to sustain theimprovements made with donor support.

Comments: This indicator could be converted into a score card. See Appendix C for more information.

4) % of plenarymeetings, withadvanceagendas, areconvened asscheduled.

% of plenary meetings (out of total numberof sessions) called in time, schedulefollowed, agendas for sessionsestablished, publicized in advance &followed.

Good indicator of improved planning& management & seriousness ofpurpose of legislative body.

Review of records of dates of public & legislaturenotice of meetings and dates of holding of meetings,and comparison. Interviews will be useful to confirmdates. Costs: Depends on record keeping of legislature &/orinterested NGOs; but should not be too demanding.

As internal management (and also will to betransparent and acceptance of full debate)improves so the trendline should be upwards.

Comments:

5) Number or %of laws bene-fitting duringdrafting from theuse of improvedinformationsystems.

Laws developed with benefit of electronicinternet access or other improvements ininformation system.

Indicates installed electronic systemand/or other improvement andaccompanying training being usedeffectively in legislative process.

Choose a sample of laws and interview key staffinvolved in the drafting.

Cost: If NGOs watch legislative process consistently,this should not be very demanding. If not,interviewing and also observing the process could befairly time consuming.

The trendline may move slowly depending oninstallation, training, and will of key staff.

Comments:

6) % of MPs &staff who saythey are able toobtaininformationwhen they needit.

% of MPs and staff who say this. Indicator of improved informationretrieval and management system.

This would require a survey of both staff and MPs.

Cost: A full survey would be relatively timeconsuming; a sample survey would be sufficient.

It could rise as MPs and staff are trained indelivery, demand, research and use ofinformation services. There is a possibility thatas the level of demand rises and becomes moresophisticated there could be a decline if thenumber and skills of those providing theinformation have not kept pace.

Comments:



7) a) Rulespermit equitableparticipation byopposition(yes/no); b) % oftime used byoppositionmembers in keydebates, or % ofspeakers inthose debateswho are fromthe opposition;c) oppositionmembers aregiven resourcescomparable toruling partymembers (e.g.,resources formeetings withconstituents); d)% of (key)committeemembers fromthe opposition inkey committees;or e) bills ofoppositionmembers aregiven dueconsideration.

b) - c) not useful unless answer to a) isyes. Can also convert elements to anindex; i.e., extent to which oppositionmembers express themselves in plenary.

Each criterion given a value on scale of 1-5with 1 indicating “no opportunity” and 5indicating “full and equal opportunity;” andthen the panel can record and aggregatetheir scores.

A democratically functioninglegislature requires that fullopportunity be given for participationby the opposition. This indicatorseeks to measure this by settingcriteria indicative of opportunity forparticipation, and use of thatopportunity, by the opposition.

This will require a person(s) to observe plenary andcommittee work so as to observe proceedings, andalso accounts to see how money and/or supportpersonnel resources are allocated between parties.It may be possible to focus on significant legislationonly. It may also be possible to achieve this end byperusing and doing content analysis of Hansards.

To establish whether formal rules allow foropposition participation will not be demanding.However, to find out if these rules are being followedmay be demanding. If a person or organization isrequired to do only this task, it will be expensive. Ifthe task can be added to the work of a full timeparliamentary watch organization, it will not beexpensive. The expense may be further reduced bylooking only at specific pieces of legislation of moredirect concern to the opposition.

The novelty of democratic and particularlyparliamentary practice, the depth of democraticculture, the heterogeneity of the society, thepolitical nature and sense of security of theincumbent party, the capacity and strength ofthe opposition party (ies) will determine both thebenchmark and the likely trend. The expectationwould be that as procedures are practiced,respected, and accepted, the trendline wouldrise.

The role played by the opposition is likely to betied to the number of seats it occupies. Targetsshould keep this in mind.

Comments:



Intermediate Result 2.4.5: Strengthened Representation, Effectiveness and Independence of LegislaturesIntermediate Result 2.4.5.2: Increased Legislative Capacity to Influence National Policy and Budget Priorities


1) a) Laws orconstitutionalreforms passedenablinglegislature (thrucontribution tolegislation oroversight) tooperate as anindependent &democratic body(yes/no); or b)Improvements inlaws orconstitutionalprovisions incomparison witha list of reformsproposed.

Minimal requirements of such rules include1) a legitimate role for all members,including opposition members, to speak,vote, introduce legislation, participate incommittees, & share in the resourcesnecessary to these activities; & 2) fulloversight and questioning of the executive.

This could be a yes/no measure; or as thesecond option implies, it could be a # or %of improvements made compared to a listof improvements proposed as suggestedabove in this column.

Relevant to all settings. Rules arean essential first requirement towarda democratically operatinglegislature.

Analysis of written handbooks and documents;survey of legislators. Interviews with key observerswould help in understanding significance of variousrules.

Cost: Minimal; although a case study would involvehigher costs.

This should be expected to rise, but the contextwill determine the pattern. Much of thenecessary legislation & rule making might takeplace at once, and refinements andimprovements follow; or reforms might be“squeezed” out of a reluctant incumbent party orexecutive branch one at a time.

Comments: See also the Legal Reform Scale in Appendix C.



2) a) Extent ofimplementationof laws orconstitutionalreforms &proceduresenablinglegislature (thrucontribution tolegislation oroversight) tooperate as anindependent &democratic body(yes/no); or b)Improvements intheimplementationof such lawsandconstitutionalprovision incomparison witha list of reformsproposed.

Overall a subjective measure, but based onconcrete pointers which might include: ahandbook of regulations & procedures hasbeen distributed to all legislators and madeavailable to the public; role of oppositionand minor parties is clear & unfettered inpractice; the public & press have access tothe legislature process; main committeesmeet; the public & press have access toindividual MPs; voting records are public;the legislature is able to carry on itsbusiness without hindrance from theexecutive.

Relevant to all settings. Rules mustbe transparent, available andenforced for the legislature to beeffective.

Analysis of written handbooks, documents &legislature records, survey of legislators; a casestudy of a specific legislature issue could be useful.

Cost: Low cost except a case study would berelatively more expensive.



3) % oflegislativesessions(plenary &committee)during whichtranslationservices areprovided.

% of such sessions out of all sessions.Translation should be provided throughoutthe session. There needs to be somediscussion locally of what isfeasible/realistic since some countries havehundreds of languages/dialects.

Only relevant in bilingual ormultilingual situations; but showsintention to overcome languagedivisions and to be inclusive ofgroups that do not speak mainlanguage. Also reflects good.management.

Survey of minority language MPs.

Cost: Low, unless there are many of these.

A scale may be more sensitive to change thanthe % of legislators.

Comments:



4) Adequacy oflegislativereviewprocesses: a)Executivesubmits budget;b) Congress hassufficient time toconsider budget;c) Appropriatecommittee re-views budget;d) Budgetanalysisprepared;e) Legislatureholds hearingson budget; andf) Legislatureamends budget.

Indicator is a scorecard. Each element a)to f) receives a score of 0 (no) or 1 (yes)and the scores are than totalled.

Measure of increased capacity rethe budget process. If quality ofelements b)-f) matters, thenconsider converting the scorecard toan index with 1-5 points awarded foreach element.

Legislature records; key informant interviews.

Cost: Low. NGOs may already track.

Collect annually.




5) Index ofcommitteecapacity: a)committeestructureappropriate forissue area (i.e.,an educationcommittee con-siders onlyeducativeissues);committeeshave sufficientresources (i.e.,professionalstaff) at theirdisposal; c)committeeshave theauthority toinitiate & amendlegislation; d)committeesexercise theirauthority.

Index would be based on criteria such asthese (or others which may be moreappropriate in local circumstances). Foreach of these criteria, there could be ascoring system on a scale from 1-5 with 1representing a complete negative and 5representing well established capacity &consistent practice. A panel of expertscould then discuss the topics and prepareindividual scores and then an average ofthe total scores could be used.

Some, but not all, legislativesystems rely to a great extent oncommittees to oversee research,conduct hearings, investigate,propose & formulate, sound outopinion, and negotiate & proposeamendments. They thus play a vitallegislative role before bills arebrought to plenary meetings. Thisindicator seeks to measure theextent to which the committeeshave the capacity to play this role.

This would require a panel of experts on politicalprocesses, including a mix of ex-MPs, academics,think tank or advocacy personnel, who would agreeto participate on annual or biennial basis.

Cost: Setting up the panel and agreeing on suitableelements would take time at the initial stage. Afterthat this should not be a time consuming affair,especially if the same panel can be used for some ofthe other qualitative/scale indicators.

Benchmark will probably be low at the outset ascommittees are usually not well developed inthe early years of new legislatures. Whereparliaments decide to use committees thetrendline should rise, but the rate would dependon resources given in support of, and time givento, committees. Also it may depend on awillingness to, and then experience with,delegating these roles to committees.

Comments:



6) Index ofcommitteeoversight: a) arethere oversighthearings? b) dooversightcommitteeshave sufficientresources toconductindependentinvestigations?c) do oversightcommitteeshave authority toquestionexecutivebranch?

Index would be based on criteria such asthese (or others which may be moreappropriate in local circumstances). Foreach of these criteria, there could be ascoring system on a scale from 1 to 5; with1 representing a complete negative (i.e.,they lack authority or resources) and 5representing well established authority, etc.A panel of experts could then discuss thetopics and prepare individual scores andthen an average of the total scores couldbe used.

Some, but not all, legislativesystems rely to a great extent oncommittees to conduct a variety ofroles (see # 5) before bills arebrought to plenary meetings.Committees also may play a keyrole in overseeing theimplementation of laws by theexecutive branch. This indicatorseeks to measure the extent towhich the committees carry out thatrole. This therefore relates to boththe effective functioning of thelegislature, but also, & morespecifically, to its constructive roleas a check on the executive branch.

This would require a panel of experts on politicalprocesses, including a mix of ex-MPs, academics,think tank or advocacy personnel, who would agreeto participate on annual or biennial basis.

Cost: Setting up the panel and agreeing on suitableindicators would take time at the initial stage. Afterthat this should not be a time consuming affair,especially if the same panel can be used for some ofthe other qualitative/scale indicators.

Benchmark will probably be low at the outset ascommittees are usually not well developed inthe early years of new legislatures, andexecutives tend to dominate. Where parliamentsdecide to use committees to oversee andchallenge (which does not necessarily mean ina confrontational manner) the performance ofthe executive branch, the trendline should rise;but the rate would depend on resources given insupport of, and time given to, committees. Alsoit may depend on a willingness to, and thenexperience with, delegating these roles tocommittees. Finally, it will depend on executive-legislature relations including the executive’swillingness to cooperate.

Comments:

7) % oflegislature’sbudget (or dollaramount)devoted to a)modernization;b) research &information.

As stated in the indicator. Dollar amountmay be more appropriate if legislature isstarved for resources.

Shows commitment to buildcapacity.

Leg. Budget.

Cost: Low.

We hope to see early and significant increases.

Increases should be assessed in light of theinflation rate.

Comments:

8) # of staff perlegislator or percommittee.

As stated in the indicator. Includestechnical, professional, administrative andclerical staff.

Inadequate staff support clearlyhampers legislative performance.

Personnel records.

Cost: Low.

Should increase slowly. Govt. budget deficitsand slow economic growth will affect theindicator’s performance.

Comments:



Intermediate Result 2.4.5: Strengthened Representation, Effectiveness and Independence of LegislaturesIntermediate Result 2.4.5.3: Increased Citizen Access to Legislative Process


1) # of publichearings open tocitizens, citizengroups & thepress: a) insidethe capital, & b)outside thecapital.

Count number of actual meetings open tothe public which take place, with legislatorsactively participating.

To satisfy definition of “open,” meetingsshould be held at times and places that donot exclude women and other socialgroups ex religious groups.

Relevant to any country, opennessof legislature to public participation,usually through NGOs, and themedia essential element ofdemocratic process. This attemptsto measure how many meetingswere actually held.

Observation, analysis of public meetings & hearings& legislative debates; key informant (legislators,media, NGOs, and citizens) interviews; monitoringpress coverage; case study of specific piece oflegislation in which the public has shown interest.

Cost: Low, especially if local watchdog organizationsare in place to observe legislature; and attend publicmeetings and hearings, etc; and there is monitoringby the press on a regular basis.

Should be a positive trend at first, then it mayreach a ceiling. Depends on openness oflegislators to the public, public interest andfocus, and importance of issues beingdiscussed. At a later stage, sense of publicefficacy will be relevant; that is, if the public orits organizations feel that their opinions aretaken into account, interest may rise and thetrendline will go up, or stay relatively high; if notthey will loose interest.Once hearings becomecommonplace, it will not be useful to count thenumber held.

Comments: Requires qualitative analysis of who is participating and in what manner.

2) a) average # ofmeetingslegislators holdw/constituents;or b) % of MPswho meet withNGOs & constit-uents more than xtimes a year.

Meetings involve discussion between twoparties in which citizens or NGOs havesubstantial input, i.e., puts a question,makes a request, suggests an option.

b) threshold for # meetings per year shouldbe set according to local circumstances.

Building a responsive relationshipbetween legislators & constituents isa significant element ofrepresentative democracy not oftenappreciated in the early days oftransition. This indicator attempts tomeasure progress in this area.

There would need to be reliance on records of MPsand of NGOs. It may be necessary to select andmonitor only a representative sample of MPs and/orNGOs.Cost: If MPs keep records and NGOs can confirm,then monitoring a few selected MPs should notprove too demanding. If no records are kept or theyare unreliable or there is no way of checking, thenthis would be difficult.

If a program is put in place to encourage MPsand to train them in building constituencyrelations, then the trendline should rise. Therate may be partially dependent on resourcesavailable for local travel for MPs.

Comments: Less relevant in party list systems where electoral areas are large, and there is less incentive for these meetings. Nevertheless, this kind of contact with constituencies remains significant to democratic process.

3) % legislatorswith functioningregional or localoffices of MPs.

% of legislators who have established inoffices in their constituencies which areopen for more than (for example) threemonths a year to deal with constituencyservices.

Building a responsive relationshipbetween legislators & constituents isa significant element ofrepresentative democracy not oftenappreciated in the early days oftransition. This indicator attempts tomeasure progress in this area with afocus on building constituencyrelations.

There would need to be reliance on records of MPs,with some cross-checking to see if the offices areopen.

Cost: Low.

Here the benchmark will be very low in mostplaces since “constituency relations” of this kindis both a new idea, and puts demands onscarce resources and time of legislators. In thebeginning it may be unclear to legislators whatthe purpose and incentives are for these kindsof practices. Trendline should be expected torise, but slowly.

Comments:



4) % of legislativecommitteemeetings inselected issueareas in whichcitizensparticipate. (Seelast columnsuggesting why insomecircumstances #of meetings or %change inmeetings mightbe moreappropriatemeasures than%.)

% of all committee meetings in whichcitizens are invited well in advance, givenrequired information, and reasonableopportunity to participate.

The importance and role ofcommittees has been described in anumber of places. By focusing oncitizens and committees, thisprovides one measure of theopenness of the legislative systemto participation by the public.

This requires on-going monitoring of committeeproceedings; or a combination of reading committeerecords backed up by some observation. It may bepossible to follow some selected issues rather thanobserving all of them.

Cost: Like most measures that require on-goingobservation, this would be costly if someone had tobe employed especially for this. If someone isalready observing a variety of activities of legislativecommittees then this would not add a great deal tothe task. Reading reports rather than observing allsessions would further reduce costs, as wouldselective rather than a comprehensive focus.

Benchmark and trendline would need to takeaccount of the fact that two interrelatedprocesses are involved: number of meetingsand number of meetings in which citizensparticipate. For example, it might occur thatthere are a higher total of meetings, and also ahigher number at which people are participating,and, even though this means that participationis increasing, the % may not go up. Therefore, itmight sometimes be preferable to use the # ofmeetings or % increase in number of meetingsas indicators. As both committee meetings andcitizen participation are usually novel practices,one should assume a low benchmark and thena rising trendline; the speed of change willdepend on both the legislature and the public,including political will, capacity and interestlevel.

Comments:



5) Scorecard ofcitizen access: a)do citizens haveaccess to recordsof meetings; b)are citizens ableto find out who orwhat group isresponsible forparticular areasand decisions; c)are citizens ableto obtain votingrecords of MPs;d) are citizensgranted access tomeetings of theparliament; e) arecitizens grantedaccess tocommitteemeetings; f) areplenary andcommitteemeetings open tothe press?

Scorecard would be based on criteria suchas these (or others which may be moreappropriate in local circumstances). Foreach of these criteria, there could bescoring of 0 (no) or 1 (yes). The scores forall elements would then be totaled.

Citizen & NGO access to meetingsand records of all aspects oflegislative proceedings is animportant measure of democraticlegislative process and willingnesson the part of the legislature &legislators to be transparent andaccountable for what they do.

This would require one or two experts on politicalprocesses.

Cost: This should not be a time consuming affair,especially if the experts can be used for some of theother qualitative/scale indicators.

Benchmark will probably be low as these arenew and fairly demanding undertakings. Theyrequire both political will and substantivelyenhanced management capacity on the part ofthe legislature. Therefore while there should bea rise in the trendline, it should not be expectedto be rapid.

Comments:



6) a) Legislaturepermits theoperation ofperiodicalsdevoted to thereporting oflegislativeactivities; and ifso, b) extent andquality of thecoverage.

a) Laws and regulations allow observers towitness proceedings, interview MPs, andpublish analyses; and freedom to publishremains undeterred even after articlescritical of the legislature. This could be yesor no or broken up into steps. b) Extentmay refer formally to numbers publishedbut it would be better to have an expertpanel to assess readership, influence, andquality (including relative accuracy, use ofinformation, rigor of analysis, clarity ofwriting & whether articles are picked up bythe regular press.)

Periodicals go further thannewspapers in that they providemore in-depth, carefully supported &considered assessments, and assuch may help to deepenunderstanding of practices andpossible recommendations amongscholars, MPs, politicians andinterested civil society.

Part a) requires first the perusal of relevant laws andregulations, observation of publishing of periodicals,and then checking with writers and publishers to seeif there is any harassment as a consequence.Questions of extent and quality should be put beforea panel of experts, and assessment should be basedon agreed criteria as suggested in column 2.Cost: Inquiry into rules should be inexpensive.Interviewing a few key people should also not bedemanding. The setting up of a panel will initiallytake time, but once in place, will need to choose asample of articles to review and criteria by whichcontent analysis will be performed.

Inquiry into the rules and regulations shouldsimply yield a yes or no response. Inquiry intoactual operation might be yes or no or mightlead to some gradations from “not allowed inpractice” to “as long as it is not too critical” to“full freedom,” and so a trendline might beanticipated to proceed along that line ifdemocracy makes progress. Extent and qualitywill probably commence at a low level giventhat these are probably new initiatives andthat they depend on both the willingness ofthe legislature and the skills, resources, &capacity of writers & publishers.

Comments:

7) % of bills whichincorporate theproposals madeby the opposition.

% of all bills in which the final form of thebill incorporated substantial proposalsmade by the opposition.

It is important in assessing theextent to which democratic practicesare taking hold in the legislature toenquire whether the opposition iscontributing to debates, andwhether its contribution is makingan impact on policy. This focus ofthis indicator is on the impact of theopposition.

Sample survey of bills. This would require a carefulreading of committee & plenary reports, and acomparison of what was said by members of variousparties in debates with the original and final drafts ofbills. Interviews with key actors either to point theway to more important issues, or after reading thematerial to get a more refined interpretation of therecords would be essential.

Cost: Medium but depends on sample size. Couldalso do only for key pieces of legislations.

It is difficult to generalize. If goodwill andopenness prevail, the target may not have to bethat low and improvements might dependmainly on the capacity of the opposition to takeadvantage of procedures and prepare them-selves (plan strategically, strengthenarguments, build coalitions, ) for substantivecontributions to debates. In situations ofparliamentary tension and conflict and/or ofinsecure dominant party, the benchmark may bevery low and improvements will depend on avariety of factors including opposition numbersand skill and the will of the incumbent party. Inthis case, change will probably not be rapid.

Comments:



8) Degree towhich legislaturemanages societalconflict, andattempts toreduce sources ofthe conflict.

Societal conflict refers to serious orpotentially serious conflicts of a racial,ethnic, class, religious, ideological, orregional nature.

Could establish a critical events agenda fora key conflict or conflicts in advance andmeasure progress in meeting agendaitems. Could also do an ex post factoanalysis that sets out criteria for conflictmanagement/resolution and measureslegislative performance against criteria.

While formally the role of thelegislature is to formulate legislationand check on the executive, it has abroader role of carrying out its rolein a manner that reduces or avoidssocietal conflicts; for example, byworking in a broadly inclusivemanner & considering the divisiveimpact of legislation and regulations.

This would require a panel of experts on politicalprocesses, including a mix of MPs, academics, thinktank or advocacy personnel, a representative sampleof incumbent & opposition MPs and reps ofstakeholder groups involved in conflicts, who wouldagree to participate on annual or biennial basis.

Cost: Setting up the panel and agreeing on suitablebases for discussion would take time at the initialstage. After that, this should not be a timeconsuming affair although the composition of thepanel may have to be added as the areas of conflictchange.

If there is one longstanding and/or fundamentalconflict dividing a society, it may be possible toprovide a qualitative benchmark based on thedepth & extent of the conflict and the extent andmanner in which parliament is dealing with it.Then, to the extent that it can be observed thatthe level of conflict is being reduced and thatparliament devised strategies, formulatedpolicies, included diverse opinions, etc. andgenerally acted in way that helped bring thisabout, it could be concluded by the panel thatthe body is making a positive contribution toreduction of societal conflict. If there are anumber of conflicts or the basis of conflictchanges then it becomes not impossiblebut certainly more complex to set even anotional benchmark or anticipate a trend.

Comments:



9) Perception ofa) political actorsor b) NGOsregarding thelegislature’s rolein managingsocietal conflict.

Since this will depend on a survey, the unitwould be % of pol. actors or NGOs whoagree that the legislature is effective inimproving its role... or who rate thelegislative performance as good...NGOs should be those who track theseconflicts.

While formally the role of thelegislature is to formulate legislationand check on the executive; it has abroader role of carrying out its rolein a manner that reduces or avoidssocietal conflicts; i.e., by working ina broadly inclusive manner &considering the divisive impact oflegislation and regulations.

Survey.

Cost: Medium to high depending on numbers to beinterviewed.

If interviewing political actors, opposition andparty-in-power could both have biases thatrender the indicator useless. Needs to be testedfirst. NGO perceptions could have greatervalidity.

Comments:

10) Content ofpublicparticipation incommitteesfactored into legislation &budget.

Will be subjective measure althoughresponses must provide concreteexamples demonstrating how ideas, etc.were taken into account or were ignored.It could be yes/no or be based on scalewhich could go from “many times” to “veryfew” to “none.” Do for key pieces oflegislation and budget.

This is a key indication that not onlyare legislators meeting with people,but that they are taking their inputinto account and including it as theydesign policies, laws, etc.

If govt. or NGOs keep records of meetings, thesecould be looked at. If not, it would requireobservations or interviews with NGOrepresentatives. To trace whether suggestions wereincluded or factored in would require monitoringlaws, budgets, and policy actions of all kinds. Cost: If selected areas and selected topics werefollowed, this is manageable. But it will take time andcare and conferring with local experts to determinethe extent to which legislation was influenced in thisway.

The will and procedures & capacity of thelegislature will determine whether this willimprove or not. But if there is such will andthere is a program to support this type ofparticipation and take it seriously, then thetrendline might rise, but it should not beexpected to be rapid.

Comments:



Intermediate Result 2.4.6: Enhanced Policy Processes in the Executive Branch


1) % of citizenswho believe thatthe executivebranch hastransparent,participatory,and thoughtfulpolicyprocesses.

As in the indicator, disaggregated bygender, ethnic group, etc. "Thoughtful" inthis instance means well-considered, usingfacts and figures and doing analysis andprojections of impact.

Demonstrates citizen confidence inthe policy process. Perception maynot line up with reality so somecaution is needed in using thisindicator. May be a better planningindicator than program performanceindicator.

Survey which should be part of a larger survey; thisis relatively expensive, but less so if a polling processis already in place in the country. If surveys are notaffordable, it may be possible to get a “sense” of themood of citizens through a series of group interviews.

Cost: Expensive if USAID has to carry out the surveyitself; less expensive if it can be included in a surveycarried out by a local organization.

Findings should be treated with care & will needexpert interpretation. Can citizens separate theprocess from the results of the policiesformulated during the process? The trendlineshould rise over time as policy processesbecome more democratic and more sound froma substantive angle. May be a time lag betweenimprovements in the process and popularperception of improvements. Targets should bemodest; change may not be dramatic. Not worthconducting annually.

Comments:

2) # keyExecutiveBranch policieswhich appear tobe having theirintended impactin comparisonwith a list ofpoliciessupported.

As noted in the indicator. Could be limitedto particular sectors. The assessment islikely to require special indicators formaking the judgment about impact for eachpolicy considered. These indicators couldbe substantive (small enterprise growth, #semployed by small enterprise) orperceptual (what do stakeholders,beneficiaries, and experts think about theimpact of a given policy?)

Indicates that application of varietyof policy analysis skills andbroadened consultation process areleading to more effective and moreimplementable policies. Note thatsome degree of unintendedconsequences may be anticipated.

Depends on the policies. One method could beannual detailed interviews with key informedmembers of relevant stakeholder groups. It would beimportant in initial interviews to set up key elementsto watch; i.e., effects of lifting of price controls onproduction and demand among certain categories ofpeople; also if there are ways of observing whetherthe consultative process made a difference to qualityand implementation.

Cost: Will vary depending on the policies. Selectiveinterviews with key informants should not be toocostly but it will take some time as this requiresdetailed and well supported evidence to demonstratehow a particular sector has been affected by aspecific policy.

Even if level of openness rises, trendline mayrise and fall depending on the sensitivity ofissues, govt. confidence and press track recordin dealing with investigations. Need to establisha reasonable time frame for when impact mightbegin to be seen, particularly if relying onperceptual measures.

Comments:



Intermediate Result 2.4.6: Enhanced Policy Processes in the Executive BranchIntermediate Result 2.4.6.1: Rules for Making Policy Reform


1) Improvementsin the laws andrules on divisionof policyresponsibilitiescompared to alist of reformspromoted.

Laws, etc., must make clear the roles,responsibilities and relationships of andbetween the different branches and levelsof govt. While there are no universal rules,there should be substantial roles for theexecutive and legislature branches, somebalance of powers, and forms of checksand balances.

Can be yes/no; or # or % out of keyreforms promoted.

For the product of policy processesto be effective and implementableand for the process to beconsultative in and out of govt.,requires for a start that laws andregulations clarify roles, etc., andways of checking powers and ofresolving conflicts of substance andprocedure.

Monitoring of govt. documents, and interviews withgovt. and non-govt. stakeholders to determinewhether the laws in place are being implementedand the text of the laws conforms to the titles andstated intentions of the laws.

Cost: Reasonably low; and once in place andoperating, may not need to be assessed again.However, it is possible for laws to be undermined bypractice and therefore this may need constant watch.

This will be very specific to different situations,traditions, and systems. Broad terms of divisionof responsibilities and balance/separation/checks of powers might be defined inconstitution; or it may require series of laws andregulations or even conventions; in some casesan independent court may come to determinesome of the details as disputes arise. At aformal level the outlines at least should be inplace soon; at a more refined and substantiallevel improvements will take time.

Comments: See also Appendix C for the legal reform scale, which represents an alternative measure.



Intermediate Result 2.4.6: Enhanced Policy Processes in the Executive BranchIntermediate Result 2.4.6.2 Policy Formulation and Implementation Capacity Strengthened


1) % of peoplereceivingTA/training whosay they haveused their newknowledge andskills and cangive examples.

As in the indicator, out of all those trainedor receiving TA on policy processes. It maybe important to disaggregate on basis ofgender and other bases.

This is a proxy measure for utility ofthe assistance and increasedcapacity in policy analysis. Itrequires that such people havebeen through a process of technicalassistance and are prepared to saythat they have made use of whatthey learned to be more effective.

Sample survey.Interviews with people who have undergone trainingor worked with advisers. It is important thatinterviewees be pressed for convincing examples.

Cost: Moderate, but depends on how many peoplewere to be included.

Should be expected to show positive trendlineespecially if there is focused, applicableassistance and programmatic support.First targets should be set no earlier than 6-12months after training.

Comments:

2) # or % ofusers applyingpolicy analysistools to policy.

Users could be individuals, govt. offices,NGOs or other entities. Tools includepolitical mapping, stakeholder analysis andothers. Could convert the indicator to ascale of frequent use, occasional use,limited use, or no use for a list of key policyissues and then average the scores acrossthe issues.

This indicator seeks to get at theactual use of the various methodsthat have been learned, both thevariety and level of skill andsophistication. The emphasis shouldbe on methods that both improvequality of formulation and feasibilityof implementation.

Detailed interviews with sample or panel ofobservers and govt. and non-govt. stakeholders. Indetermining improved quality, detailed interviewingwill need to take place.Cost: Inexpensive or moderate depending onnumber of users to be interviewed. Interviews willneed to be carried out by persons familiar with policyanalysis methods.

Should be expected to show positive trendlineespecially if there is focused, applicableassistance and programmatic support. Firsttargets should be set no earlier than 12 monthsafter the tools are taught.

Comments:

3) a) Budget andb) personnelallocations toimplementselectedpolicies.

Money and staff are allocated to carry outthe policy. This would be based on viewsof panel of govt. and non-govt.stakeholders who would be required toexplain their assessments. This could bebased on a scale indicating seriousness ofpurpose in implementing the policy: i.e.,no/minimal, insufficient, sufficient for amodest level of achievement, sufficient forsubstantive level of achievement.

By focusing on two essentialelements of effective policyimplementation, this indicator triesto measure the quality of the policy,and also the political will toimplement (which in turn indicatesthat political leaders are persuadedby it) and the likelihood that it will beimplemented.

Monitor annual budget and expenditure reports toobserve quantitative changes; in addition interviewrelevant and informed persons to get more subjectiveassessments of adequacy of resources allocated forimplementation of specified policies.

Cost: Not a major undertaking; the more policiesfocused upon, the more time consuming.

Indicators are trying to gauge adequacy ofmoney and personnel for a particular purposerather than simple quantitative increases overtime; policy implementation may require morepeople in the early period of implementationand fewer later or vica versa. Keeping“adequacy to achieve results” in mind, thetrendline should stay high or show amovement up the scale.

Comments:



4) Legal andinstitutionalframework toimplementselectedpolicies.

If necessary to facilitate implementation ofpolicy, new or adapted laws and rules andinstitutions are in place; or those whichmay have hindered implementation arealtered or rescinded. This could be yes/no;or based on a scale from serious obstacle;to unhelpful, to facilitative of a modest levelof achievement; to facilitative of asubstantive level of achievement.

Although this is a more qualitativeindicator than 3) above, it serves thesame purpose by looking at twoother factors essential to effectivepolicy implementation.

Monitoring of govt. laws, rules administrativeregulations; interviews with informed observers andgovt. and non-govt. stakeholders to assessappropriateness of laws and adequacy of design(and later) operation of institutions to facilitateimplementation.

Cost: Moderate, but careful assessment of adequacyof laws and institutions will take care and time.

As above in general terms. Laws should be inplace form the start or be adapted as lessons ofimplementation indicate need for change.Operation of relevant institutions should showimprovement over time.

Comments:



Intermediate Result 2.4.6: Enhanced Policy Processes in the Executive BranchIntermediate Result 2.4.6.3 Intra-Governmental Consultation and Information Improved


1) a) % of policymakers who saythey havesufficientinformation onwhich to makedecisions,orb) # offices orMinistries whosepolicymakersagree that theyhave sufficientinformation .....

Might organize by ministry of office. Policymakers are those in govt. who propose,formulate, or supervise implementation ofa selected policy or set of policies.’Sufficient’ will be based on their opinions,but they will be asked to back up theiropinion with supporting evidence.

One (admittedly subjective) methodof getting at whether improvedinformation is available for policymaking resulting from betterconsultation of various kinds andbetter information systems.

Interviews with govt. personnel involved in policyprocesses.

Cost: The more comprehensive or the larger thesample, the more expensive; but should beinexpensive given that not that many people areinvolved.

The trendline should be up as long there isclear programmatic and govt. departmentalsupport for the consultative process and useof new and better information.

Comments:

2) # ofmechanismsused for intra-and inter-ministerialconsultation.

Mechanisms include committees, semi orpermanent commissions, newsletters, e-mail and list serves and requirements forexchange of drafts. They should be usedregularly in order to qualify as a“mechanism used.”

A problem (sometimes theestablished practice) in many govts.has often been that policies havebeen developed in the isolation ofone unit. This indicator measuresefforts to establish new ways ofbroadening consultation. It becomesmore meaningful if one requiresregular use and defines ’regular.’

Interviews with govt. personnel in departments whichare involved; observation of meetings (if possible)and of selection of copies of newsletters or electroniccommunication.

Cost: Interviews will not be time consuming.Checking on few meetings and looking at a fewexamples of use of e-mail/list serves & newslettersalso should not take much time.

If policy processes are becoming moreconsultative and at more stages (includingmonitoring and evaluation of implementation),this trendline should increase for a while, but weare not talking about large numbers ofmechanisms. With time, it may be found thatsome are no longer required or do not work andthere may be a decrease in number. At thatstage, numbers of meetings and quality ofdeliberations becomes more important.

Comments: Frequency of use of mechanisms may be an important consideration.



Intermediate Result 2.4.6: Enhanced Policy Processes in the Executive BranchIntermediate Result 2.4.6.4: Opportunities of Public Participation Increased


1) # of wellpublicized policymeetings open tocitizens andcitizen groups.

“Publicized” includes publication inchannels that are easily accessible to thepublic at large and targeted efforts toinform citizens and citizen organizationswith specific interest in the topic; andnotice should be in “good time.” Thisdepends on local view of what "constitutesenough time to give full consideration." Once standard is set within a local context,the number of such meeting should berecorded. To satisfy the definition of“open,” meetings should be held at times &places that make them accessible to allsocial categories of society.

Indicates executive branch isopening up the policy process to thepublic and those organizations withparticular interest in a topic.

Interviews with private and NGO sector peopleconcerned; monitoring of govt. notices andnewspapers.

Cost: Relatively low if focused on selected policies,and if there are established channels for govt.notices.

Trendline should go up steadily at first and thenreach a plateau. If the trend declines, it might beindicator of govt. reducing opportunities for thepublic to participate or/and the public losinginterest. Will be difficult to set targets.

Comment: See also civil society indicators.

2) # of meetingsof joint policycommissionsbetween theexecutive branchand represent-atives of the for-profit and/or not-for-profit privatesectorson selectedpolicies.

# of meetings over a period of a year on allstages of policy processes.

As above; only more clearly focusedon the private and PVO sectorswhich should be actively involved inthe policy processes.

Interviews with private and NGO sector peopleconcerned; monitoring of govt. notices andnewspapers.

Cost: Relatively low if focused on selected policies,and if there are established channels for govt.notices.

As above. May be difficult to set targets.

Comments:



3) # or % ofprivate firmsand/or NGOs whosay they haveexperienced avalid engagementwith the executivebranch in theprocess of policymaking and/orimplementation.

“Valid engagement” refers to a situation inwhich these organizations have beeninvited in good time, give good information,have been listened to and have a sensethat their views have been taken intoaccount. The number could be related toone policy instrument, or a sector orselected policies or govt. policy making ingeneral. (Where relevant, an effort shouldbe made to include women’s & minorityorganizations as a test of inclusiveness ofthe process.)

Goes beyond 1) and 2) by focusingon whether the opinions,information, etc., of the private andNGO sectors that have chosen toparticipate are seriously being takeninto account.

Interviews with senior persons in private and NGOsectors. Quality of information will be helped bytalking to a few govt. officers.

Cost: Medium, depends on number of groups to beinterviewed.

As program takes effect, trendline should rise. Ifexecutive is committed to substantial increasein non-govt. input into policy processes, thisshould be reflected in sharp increase; ifcommitment and management is weak thetrend will of course be more gradual.

Comments: This indicator could be converted to a scale with criteria laid out for high level of engagement on all substantive issues, medium engagement on most key issues; limitedengagement and seldom on key issues; and virtually no engagement.



Intermediate Result 2.4.6: Enhanced Policy Processes in the Executive BranchIntermediate Result 2.4.6.5 Executive Leadership in Enhancing Democracy


1) Example(s) ofmajor initiativelaunched by theexecutive branchto enhance(broaden ordeepen)democracy.

Example involves a clear demonstrationthat the executive branch is willing to takethe lead in a major initiative instrengthening democracy, possibly in faceof indifference or even antagonism fromparts of the public: for example, in fightingcorruption, or for women’s rights, oragainst ethnic discrimination. Therequirements of a “major initiative” wouldinclude launching a public campaign,establishing task force with high levelappointments, devoting resources, settingan example, providing incentives andsanctions.

The executive branch, particularly athigher levels is often the agencymost able to push the democracyforward or save it from slipping backby (pro)actively taking majorinitiatives. While this cannot bemeasured by # or %, initiatives ofthis kind from time to timedemonstrate the executive branch’scommitment to and willingness toshow leadership in furtheringdemocracy.

Monitoring high level executive branch statementsand actions; at this level actions should be dealt within the mainstream media; so following TV, radio,newspapers should be sufficient.

Cost: Easy to follow; no special effort involved.However, to establish level and perseverance ofcommitment might require monitoring over time.

Not a question of trendlines. Only required incertain situations. If situation seems to require it(and that will need some consensus of informedobservers) then the test is whether theexecutive branch responds and the level of theresponse.

Comments:



Intermediate Result 2.4.6: Enhanced Policy Processes in the Executive BranchIntermediate Result 2.4.6.6 Executive Leadership in Resolving Divisive Conflicts


1) a)Example(s) ofuse of goodoffices of theexecutivebranch toresolve/reduce/preventsignificantdivisiveconflicts.

or

b) a criticalevents agendafor issuesknown to bedivisive.

Example involves a clear demonstrationthat the executive branch is willing to takethe lead in a major initiative to resolve,reduce, prevent a major divisive, orpotentially major conflict. This could beethnic, religious, ideological or resourcebased. Overtly this may include launchinga public campaign, establishing task forcewith high level appointments, devotingand promising resources, setting anexample, providing incentives andsanctions; but it may also be done in ancovert or low key manner. A criticalevents agenda would lay out certainsteps to diffuse or resolve issues andwould track progress in achieving thosesteps. The steps would include a timeframe.

The executive branch, particularlyat higher levels is often the agencymost able to push to resolve orprevent major conflicts which canlead to loss of life, tear the socialfabric, reduce political trust, andhinder economic progress all ofwhich are not only damaging inthemselves but will underminedemocracy. While this cannot bemeasured by # or %, initiatives ofthis kind demonstrate theexecutive branch’s commitment toand willingness to show leadershipin conflict reduction andavoidance.

Monitoring high level executive branch statementsand actions; at this level, actions should be dealt inthe mainstream media; so following TV, radio,newspapers should be sufficient.However, if initiative is covert or low key ornegotiations are secret which may all beappropriate. Then best information may observefrom high level contacts with senior exec officials.

Cost: Easy to follow; no special effort involved.However, to establish level and perseverance ofcommitment might require monitoring over time.

Not a question of trendlines. Only required incertain situations. If situation seems to requireit (and that will need some consensus ofinformed observers), then the test is whetherthe executive branch responds and the levelof the response.

Comments:

PART 3 APPENDIXES

Appendix A: Criteria for Developing and Adapting Performance Indicators 227

Example:

Result: Expanded citizen knowledge of humanrights

Weak Indicator: Numbers reached by civiceducation

Better Indicator: Percent of specified targetgroup understanding specific rights

APPENDIX A: CRITERIAFOR DEVELOPING ANDADAPTINGPERFORMANCEINDICATORS

A. Selecting Indicators

When selecting indicators, the first thing toconsider is whether the performance informationcollected will be useful in making programdecisions. If this is not the case, then reconsiderthe use of the indicator. To the extent possiblethe indicator should meet the following ninecharacteristics.

1. Match the indicator to the type of entity and result USAID is trying to achieve

When developing indicators, it is vital to matchthe type of measuring tool used for the indicatorto the type of entity and result that USAID’sactivities are intended to achieve. For example,just as one would not use a thermometer toindicate changes in “weight,” so one should notuse a population statistic to measure changes in“quality of legislative processes.” Each resultshould be carefully matched to an appropriatetype of measuring tool. Consider the followingexamples:

Number and/or percent of people, as well asdemographic ratios based on populationstatistics—appropriate when the objectiveinvolves changing individual attitudes, beliefs,or behaviors or when the objective involves theprovision of services to the population.

Numbers and/or percent of other entities, suchas service delivered, decisions, meetings, cases,etc.—appropriate when the objective involvesextending the scope of that entity or how it isprocessed.

Checklist, scorecard, milestone scale, or index

for a multi-dimensional process—appropriatewhen the objective is a complex developmentalprocess, such as “strengthened effectiveness ofthe legislature” or “improved NGOmanagement.” See Appendix C for additionaldetails on these types of indicators.

Quality ratings by experts or peers (usuallyusing a list of criteria)—appropriate whenquality improvement is the objective, such asthe quality of a piece of legislation.

C Comparison to a standard, such as a rulerfor measuring length or model legislationfor assessing legislative quality.

No one measuring tool is intrinsically betterthan others. Each must be considered in thecontext of what entity or result is intended to bemeasured, as well as the feasibility, reliability,and validity of the data from that measuringtool.

2. Measure the result as directly as possible

A basic principle is that indicators shouldmeasure the achievement of a result or anobjective as directly as possible. This includesmeasurement at the correct level—if anindicator measures the intermediate result, itshould not be used for the strategic objective.

The better indicator matches the result, whereasthe weak indicator is at a lower level. The pointis to know how well civic education is

228 Appendix A: Criteria for Developing and Adapting Performance Indicators

Example:

Result: Improved Application of the Law byJudges in Commercial Cases

Possible Indicators:

7) Expert review of a sample of cases to checkon whether the law was appropriatelyapplied;

2) percentage change in the number ofcommercial cases filed;

3) percent of judges knowledgeable aboutspecific aspects of the law;

4) sample of particular kinds of cases to gaugecorrection of common misapplications ofthe law;

5) percent of local/foreign businesses whoseCEOs say that they are willing to resolvedisputes in the courts; and

6) survey of commercial lawyers practicingbefore these courts concerning applicationof the law.

working—to what extent knowledge of rightshas expanded. The number reached by civiceducation programs is not necessarily the sameas the number of those with expandedunderstanding of rights. To assess theknowledge of rights conveyed through programinterventions, a sample survey is required. Theinformation gleaned from such a survey wouldprovide valuable information to managers,allowing them to adjust the content of theirprograms if the messages are not being heardand retained.

3. Plan for cost-effective measurement

Sometimes an indicator matches a resultperfectly but is too difficult to measure or toocostly (3-10 percent of the total budget is theexpected range of performance monitoringcosts). In these cases, strategic objective teamswill need to turn to proxy, or second-best,indicators. There can be trade-offs betweenmore expensive direct measures and lessexpensive proxy measures. Decisions aboutwhich to use should be based on the utility ofthe information that the indicator provides.

Indicator 1 would require a panel of legalspecialists and intense review of a sample ofcases. It might be very costly to develop asample frame, compile all the needed caserecords, and collect these data. There could wellbe reasonable differences among expert panelmembers concerning the judgment made in anygiven case, since legal specialists may differ onthe meaning of the law. Such differences couldbe difficult to reconcile.

Indicator 2 is a posible proxy with low datacollection costs. It assumes that citizens will bemore willing to file cases if they think that thelaw will be correctly and predictably applied.Strategic objective teams should, however, thinkabout whether other factors could be responsiblefor an increase or decrease in the number ofcases, prior to choosing this as an indicator. Forexample, market reforms and growing privateinvestment could lead to a rise in the number ofcommercial cases, even without substantial

court improvements. There may also be a timelag problem with this indicator, since it maytake some time before businesses or citizensrecognize that the system has improved and arewilling to bring cases.

Indicator 3 relies on testing the knowledge of asample of judges (not possible in somecountries). The indicator should be accompaniedby self-reporting on whether the judges thoughtthey were applying this knowledge. Any surveyof judges, which relies on self-reporting, shouldask about specific cases in which the newknowledge had been applied. However, if self-reporting is used, bias will be a concern. Thisindicator measures below the result and assumesthat improvements in knowledge will lead toimprovements in practice.

Indicator 4 could represent a clever response toa difficult measurement problem. If there are

common and identifiable misapplications of thelaw (i.e., in bankruptcy cases) and if judges are


trained to understand those aspects of the law,then one could develop and review a sample ofcases involving only those aspects to see if theapplication of the law was correct subsequent totraining. Data collection for this indicator mightbe less expensive than for indicator 1, althoughputting together the sample frame could stillpose problems.

Indicator 5 would require a sample survey oflocal and/or foreign-owned businesses. Thiswould probably be easier than reviewing asample of cases, and surveyors might pick up alot of useful information about what waslimiting or encouraging business use of thecourts. There may be a time lag problem herebut perhaps not one as serious as with countingcases in indicator 2. Also, if timeliness is anissue, judges applying the law more predictablymay not do much to persuade business ownersto settle serious disputes in court.

Indicator 6 would require either a sample surveyof practicing lawyers or an expert panel chosenfrom active lawyers of good reputation. Thereshould be no time lag problem here. Ifimprovements were made, lawyers withadequate commercial case loads shouldrecognize them. However, there could beproblems of bias if, for example, a lawyer on thepanel just lost an important case.

In this example, there is no single perfectindicator. Selecting one requires thinking aboutthe trade-offs. How much accuracy is needed?How can the information be used and by whom?

4. Consider data sources when selecting indicators

Indicators should be selected in conjunctionwith a review of data sources because datavalidity and reliability, as well as timeliness, arecritical. There needs to be consistency in thesource and data collection methodology acrosstime in order for the data to be truly comparable.In cases where the data are unreliable oruncollectible, proxy indicators will have to beselected. Whatever indicators are used, they

should balance the quality and utility of theinformation with practicality of data collection.

For example, as noted above, it may be possibleto collect very useful performance data relatingto court improvements from lawyers practicingbefore the courts. A sample frame could becomposed of Bar Association membership listsor from government lists of law licensesgranted. Lawyers in the sample could then beinterviewed about a range of court phenomena.The choice of data collection method will havean impact on the articulation of the indicator. Inthis example, a survey of lawyers cannotproduce data for an indicator such as “percentof cases in which the defendant’s counselobtained timely access to court records,” but itcan produce data for a similar indicator -“percent of active criminal lawyers who saythat they a) never have trouble obtaining accessto records in reasonable time; b) sometimeshave trouble; c) frequently have trouble; and d)never try because they believe they cannot getrecords.”

The important point here is that if indicators aredeveloped without a careful look at datacollection methodologies, a mission may end upwith a list of indicators requiring severaldifferent methodologies— aggregate statistics ofcourt case data, surveys of lawyers, surveys ofbusinessmen, and so on. Multiple methodologieswill make data collection expensive and time-consuming. Sometimes settling on one datacollection approach means that some of theindicators are less direct than might bedesirable, but the trade-offs between cost andquality issues are important to examine.


Example:

Result: Increased availability of legal services

Weak Indicator: Availability of legal services

Better indicators: Number of public defenders per100,000 population; or percent change in thetotal number of cases handled by publicdefenders.

5. Make indicators operational

Indicators are operational when the concept thatis being measured has been specified in terms ofthe actions used to measure it. In other words,data collectors understand exactly what data arerequired and how those data need to begenerated and organized.

The weak indicator is not fully operationalbecause it specifies neither the legal servicesinvolved (public defenders, alternative disputeresolution mechanisms, etc.) nor the unit ofmeasurement (total number of services, numberof services per district, number of districts withpublic defenders, etc.). As worded, this indicatordoes not provide sufficient information for thedata collector.

In the candidate indicators tables in Part 2,specific definitions are provided where neededto operationalize the indicators. However, someindicators will require further specification bythe strategic objective teams. Common amongthese are scales which should be scoredaccording to very specific criteria (see AppendixC for information of this type of indicator). Forexample, the “degree to which resources areprovided to the election authority in a timelymanner” may be scored on a 5 point scale, eachpoint of which has specific standards. Becausethe standards may vary from country to country,working groups were not always able to provideuniversal criteria. Strategic objective teams mayneed to do so in order to make the indicatorsoperational in the country context.

6. Make indicators objective

Indicators are objective when they will beinterpreted the same way by different people.This permits comparisons over time, even ifdifferent people collect the data from year toyear. So, an objective indicator is one whichleaves no confusion about which data to collect,how to organize the data, nor which criteria touse in judgmental indicators. If there isconfusion about these issues, then making year-to-year comparisons will be suspect, since thedata collectors may have collected different dataand/or used different standards.

However, for democracy and governanceprograms this will not always be fully feasible.As discussed earlier, some attributes ofdemocratization do not lend themselves tosimple count indicators which tend to be lesssubjective than indicators which try to capturecomplex qualitative change. Nevertheless, thelatter should be crafted to limit their subjectivityas much as possible. Making the indicatorsoperational should also help them be moreobjective. See the discussion in Part 1, ChapterII, Step 2, for more detail on various types ofindicators, and see Appendix C for advice onusing scales and indexes, which are ways ofmaking qualitative information more objective.

7. Show the size of the problem whenever possible

Quantitative measures should be expressed notjust in terms of a numerator (i.e., what has beenachieved) but should include the denominatorwhenever possible. It is the denominator thatindicates the size of the problem being tackled.


Example:

Result: Government deals more effectively withhuman rights violations

Weak Indicator: Number of reported humanrights violations investigated

Better Indicators: a) Percent of reported humanrights violations that are investigated; and b)percent of investigated violations that areprosecuted.

Example:

Result: Improved management of civil societyorganizations

Weak indicator: Percent of CSOs that have awritten strategic plan with clearly formulatedand realistic objectives.

The weak indicator leaves us with no sense ofthe magnitude of the problem, in terms of howmany violations are reported and perhaps notinvestigated. If 100 reported human rightsviolations are investigated, it makes a differencewhether this is 100 out of 1000 reportedviolations or 100 out of 125 reported violations.The better indicators give us the scale of theproblem.2

There is an exception to the preference for ratiosor percentages over absolute numbers. When thebase (i.e., the total population) is very small, itcould be misleading to formulate the indicatoras the percentage. For example, if there are 10partner cities in a municipal developmentprogram, a percentage of those cities adoptingpublic-private partnerships will not be a verytelling indicator. If the mission proposes toincrease the number from 2 to 4 partner cities inone year, the percentage would rise from 20percent to 40 percent, a percentage that thosenot knowledgeable about the progress couldinterpret as a grand achievement. In thisinstance, it is better to report the number ofpartner cities with partnerships out of the totalnumber of target cities.

Often, both the absolute number and thepercentage can be reported from the same dataand give a more complete picture of the changesoccurring.

8. Ensure that indicators are sensitive to change

Because of the need to gauge performance atregular intervals, it is important to selectindicators that are sensitive to change.Indicators which show target groups orinstitutions reaching a threshold of some kind,but which do not reflect progress below orabove the threshold, may not be sufficient togauge progress. The most common example isthe yes/no question. For example, one canrespond to “fund-raising plans established” withonly a yes or a no. This does not showincremental change over time or changes in thequality of the plans. How close is a plan to beingestablished? How extensive is the use of theplan?

Consider also the following:

This is a threshold indicator; CSOs either meetthe minimum threshold or they do not.Improvements below and above the thresholdwill be lost.

How one sets the threshold standard will alsoaffect the extent of progress that can be shown.For one mission using an indicator similar to theone in the example above, the definition of thethreshold was so modest that the 1997 baselinestudy showed that a very high percentage of

2 However, the better indicators do assume thatmost violations are reported. If this is not the case, dueto fear of reprisals, then the indicators do notadequately measure the objective.


Example:

Better indicator: percent of CSOs showingimprovement in strategic planning on an index.

target NGOs had already passed the threshold.USAID will only be able to demonstrate modestadditional progress from that high base. If 78percent of target NGOs have already crossed thethreshold, how much of an accomplishment is itover a 5 year strategy period to increase thatpercentage to 86 percent? Setting too high athreshold can be equally problematic becausethen it may be difficult to show any progress atall.

This indicator can show incremental change.The index (see Appendix C for information onconstructing indexes) could lay out key factorsbelieved to be related to high quality strategicplanning, each of which would be scored with arange of points. In this indicator, all CSOswhich show an increase of 3 or more points, forexample, out of a total of 15 possible pointscould be counted as “improved.” The indexcould be constructed as follows:

1) CSO has a clear vision (1-5 points)Factors to consider: clear written visionstatement; vision developed in participatorymanner; vision a realistic extension ofcurrent operations; board/staff/volunteersunderstand vision statement and support it

2) CSO has multi-year strategic plan (1-5points)Factors to consider: plan developed inparticipatory manner; plan has clearobjectives; plan exists in written form; planpremised on realistic revenue generation,staffing level, and feasible activities; keyplan elements understood by board, staff,and volunteers

3) CSO monitors plan’s progress andcontinuing applicability (1-5 points)

Factors to consider: plan has clearbenchmarks, progress monitored using data,and corrective action taken when needed,staff periodically assess externalenvironment and critical assumptions

The advantage of this indicator is that it issensitive to small gradations of change. A panelof experts or a small group of CSO leaders andproject managers would work through the indexannually, awarding anywhere between 1 and 5points for each of the three elements. It is,however, more labor intensive to collect data forthe index than it would be for the thresholdindicators. Missions working with largenumbers of groups might still opt for thethreshold indicator.

9. Disaggregate indicators by relevant population groups

It can be critical to disaggregate data by genderor other population characteristics (i.e., rural-urban, ethnic group) relevant to the country anda given strategy. Strategies can have verydifferent impacts on different population groupsdue to societal factors. These factors may bestructural or cultural and they can be subtle orblatant. Strategic objective teams need to knowif their activities are affecting all groups asintended. Disaggregation may increase the costsof data collection. The expected utility of thedata does need to be compared to the cost ofgathering those data.

Another point to consider is whether theindicator should refer to a specific populationwhich the mission is targeting or the wholepopulation. For example, do the localgovernance indicators apply to selectedmunicipalities or all municipalities? Do theindicators gauging democratic attitudes refer tothe entire adult population or only groupsreceiving civic education?

B. Establishing Baseline Data and Targets

1. Baseline data


Once indicators have been selected, plannersneed to choose the year (or years) that will serveas the baseline, or starting point, from whichprogress will be measured. It is preferable if thebaseline immediately precede the start of a newstrategy because we are trying to gauge theprogress of a particular strategy. It will notalways be possible, however, to secure baselinedata for the chosen year. In that instance, thebaseline should be the most recent past year forwhich the relevant information exists or can beacquired.

In selecting a baseline year or years, it isimportant to examine the trendline of pastperformance. There could be unexpected spikesor dips in the trend and a year in which one orthe other occurs would be a poor year to selectas the baseline year. Often, of course, we lacktrendlines in democracy indicators, and it iseither not possible or prohibitively expensive toconstruct them.

2. Setting targets

Setting targets for democracy measures is verydifficult. We know little at this point aboutprobable rates of change and what factors affectthose rates. What follows here are commonsense tips for setting targets.

a. Think about how easy it will be to settargets while selecting indicators. Forexample, an indicator that purports tomeasure improved interaction between localgovernment and civil society groups, suchas the number of meetings between the two,could pose difficulties for setting targets.

b. Think about what the trend has been inthe past for any given indicator. Historicaldata for some period prior to the start of thestrategy (perhaps for 5-10 years) is useful inassessing how rapid change has been in thepast and can be used to project the future.Some years ago, strategic objective teams inthe trade and investment sector in country xprojected export targets (as a result of their

trade and investment strategies) that showedgrowth over time but which actuallylowered the trendline. This was not becausethey anticipated that external circumstanceswould drive the trendline down but becausethey had not thought to look at the pattern ofgrowth over time.

c. Consider parallel experience from othercountries. For example, data onparticipation rates of various kinds exist forBritain. While those levels of participationare probably very high relative to what canbe expected in newer democracies, theycould be used as a ceiling. In other words, ina country with high illiteracy rates,considerable rural isolation and a history ofrepression, it would seem in most instancessensible to set targets for the proportion ofcitizens contacting local governmentofficials or writing letters to the editor wellbelow those of Great Britain. We arebeginning to obtain comparative data onparticipation from some newer democracies3

and can refer to these data for comparison.

d. Think through when program activities,and those of other key donors and partners,will have an impact on indicator values. Donot “straight line” targets (i.e., increasing avariable by 2 percent per annum over 5years because you project a total change of10 percent).

e. Think about external conditions whichmay affect indicator values over time. Forexample, in one country the number ofcitizens attending town meetings was usedas an indicator to demonstrate municipalgovernments becoming more responsive.The number attending town meetings didrise for a while, but then suddenly droppedoff rather precipitously. Reportedly,participation declined when a sum that

3 The work of Mitchell Seligson in Latin Americais one example.

234 Appendix B: Assessing Data Collection Approaches and Costs

municipal governments had been given byUSAID for small development projects wasexhausted. In this case, USAID fundsdirectly triggered the increased participationso the indicator did not tell us much aboutthe interaction of municipal governmentswith citizens.

f. Consider setting a target range rather thana single numerical target. For example, thenumber of local governments appointingcommunity organizations to specialcommissions will increase by 5-10 percentbetween 1998 and 1999 instead ofpinpointing a 6 percent change in thatperiod. The range should be kept narrow, sothat the level of accountability is clear.

g. Consider how clearly the target or theactual will communicate and how thetrendline will move when deciding upon anindicator’s unit of measurement.

h. When indicators are disaggregated,targets should be disaggregated as well.When relying on preexisting data, it isimportant to check whether the sourcedisaggregates the data. It is common, forexample, to see indicators for “proportion ofpopulation registered to vote” and“proportion of population voting”disaggregated by gender. It is not alwayscommon, however, for electionadministrations to record information bygender. Acquiring the data by gender wouldrequire a) persuading the electionadministration to obtain additionalinformation and set up its recordsdifferently; or b) conducting a nationalsample survey subsequent to an election.

C. Interpreting Data Points and Trendlines

It can be difficult to be certain about themeaning of data for particular indicators.Consider the possible meanings for an indicatorsuch as “the number of reported human rightsviolations per 100,000 population.” If the

number is increasing from one year to the next,it could mean a worsening situation with moreabuses. On the other hand, it could also be ahopeful sign, with the climate for reportingabuses improving. Citizens may now feel thatthey run fewer risks of revenge when they reportincidents and that the government is nowprepared to take such reports seriously.

In yet another example, declining or low voterturnout (number of adults voting as a proportionof the total voting age population) can signify anumber of things: 1) relative satisfaction withdemocratic government; 2) decliningsatisfaction with democratic government; 3) fearof election day violence or manipulation ofresults; 4) lack of knowledge about themechanics of voting or the issues andcandidates; or 5) lower stakes in the currentelections compared with past elections.Typically, transition elections draw very highvoter turnout. The percentage voting is likely todecline from its early high as people begin tofeel that democracy will last. In addition, insome countries such as the United States, thereare multiple opportunities for voting, reducingthe stakes (and therefore the turnout) in eachelection. By contrast, high or rising voter turnout could result from successful voter educationcampaigns, institutionalization of democraticnorms related to participation, higher stakes, ormanipulation by elites who use patronagenetworks to get out the vote.

The point of these examples is to suggest thatthe interpretation of indicator data must becountry specific. Similar patterns in indicatordata may signify different things in differentcountries.

Appendix B: Assessing Data Collection Approaches and Costs 235


APPENDIX B: ASSESSINGDATA COLLECTIONAPPROACHES AND COSTS

A. Introduction

This section provides guidance on the potentialadvantages/disadvantages and costs of severaldifferent data collection methods for democracyand governance indicators. These methodsinclude sample surveys, using data fromgovernmental or other agency sources, usingjudgmental sources, and using several types ofrapid appraisal methods (including fieldobservations, focus groups, and case studies). Inaddition to considerations of data reliability andvalidity, the costs of data collection are animportant factor when choosing or designingindicators. In addition to providing somepointers on weighing the pros and cons ofvarious data collection methods, this sectionalso gives guidance as to how to estimate thecosts.

Any discussion of factors to help estimate datacosts usually involves specifying the differenttasks or elements that produce costs. Most costsdiscussed here are for people with differentbackground capabilities doing various steps inan overall process to acquire the needed data.For this reason, outlining cost components alsorequires outlining and discussing the major tasksneeded to collect each type of data, and theskills needed for those tasks. Thus, this sectionshould also serve as a broad guide to major tasksand capabilities needed to implement the severaltypes of data collection. The specific affiliationof the staff members—DG officers, partners,contractors— carrying out each task is notdistinguished here, as this will vary for differentindicators and countries.

1. Generic data collection tasks

In general, producing indicator data requiresthree major types of tasks: a) planning theindicator(s) and developing data collectioninstruments, procedures, and criteria; b) actualdata collection; and c) managing the data,analyzing it, and producing reports on the data.These steps are amplified for each of the typesof data collection discussed below. Often thecosts for the first and third step - planning andanalysis - are as great as for acquiring the dataitself. However, often the work needed for thesesteps is underestimated, leading to poor data,cost overruns, staff “burnout,” and other majorproblems that might have been avoided ifrealistic cost (and time) estimates had beenused.

Over Time. If the same method is used overtime for the same indicator, the costs for secondand subsequent waves of data collection (forexample, annual surveys) should decrease.Planning and development will be substantiallyreduced, but not eliminated as each data waveshould reassess the continued appropriateness ofthe procedures, assumptions, and datadefinitions. Actual data acquisition costs will benearly the same for each data wave, but mayincrease with inflation. Data analysis steps maybe reduced somewhat with new waves of data,as procedures for analysis and presentation willhave been worked out using the first wave data.However, new findings will requireconsideration and discussion of their meaningand implications; these discussions are likely togenerate new questions from stakeholders andother users that will require more analysis. So,the budgets for data analysis should be retainedfor each wave of data.

2. Skills needed

For some of the methods described below, staffmay need to consult with those who havespecialized knowledge and skills for data


collection. Normally, this is acquired viagraduate level training, with course work inmeasurement/instrument design, researchmethods, and/or survey research. A necessary

skill is knowing how to assess data quality, suchas validity and reliability factors. Often, skillsfor less traditional methods, such as judgmentalratings and focus groups are taught in courses


labeled “Qualitative Methods.” The skillsneeded for data base construction, datamanagement, and manipulation of large datasets are usually associated with computersystems managers, who may or may not havebackgrounds in the data collection methodsneeded to acquire the data. In general, weassume that computer capabilities are availableat, or through, the mission (orcontractors/cooperating agencies) for dataprocessing and analysis. Finally, the statisticalskills needed for survey sample design or dataanalysis and presentation are a further set ofspecialized topics; these disciplines are widelytaught and persons with these skills should beavailable to missions, for example, on acontracted basis.

When considering the costs and technical detailsinvolved in setting up good quality datacollection methods for indicators, administratorsoften wonder if all costs are really necessary.Couldn’t they obtain nearly as accurate dataabout results by cutting down substantially onthe costs? The answer is usually “no,” becausepoor quality data are often worse than no data atall, if they seem to provide information aboutthe result being measured, but really reflectprimarily the characteristics and flaws of themeasurement methods. Particularly if the dataare for indicators to be compared over time, it isessential that data be consistently of high quality(reliable and valid), so that trends over timeshow real changes, and that users of the datahave confidence in the credibility of the resultsshown by the indicators.

In the more specific presentations below, thecost elements for conducting sample surveys arediscussed in some detail, as field experienceusing these methods for surveys of democraticvalues is available for deriving cost estimates.The remaining methods are outlined in lessdetail, but with descriptions of the differences intasks needed to conduct them. Many of theconcerns and considerations for sample surveys

will also apply to these other methods.

B. Using “Secondary Source” Data

1. Data from government or international agencies

a. Description/Definition. This refers tostatistical data already being collectedsystematically, often on an annual basis, by ahost country agency. For example, the ministryof justice may have statistics on the numbers ofhuman rights cases tried in its court system, oran election authority will have data on thenumber and percentages of citizens registered tovote. Sometimes, these statistics are kept forjurisdictions smaller than the whole country,such as provinces or court districts; if availablethese can provide a basis of comparison of“districts targeted by USAID versus those nottargeted” as well as looking at trends over time.In addition to government agencies, otherorganizations may collect and keep relevantdata, including other international or privateagencies, universities or other academic centersdoing research, or even commercial firms thatcollect data for business purposes.

b. Cost Elements. This type of data may seemalmost cost-free, but may have hidden costs, thatare necessary in order for staff members toconduct the following actions:

(1) Specifying precisely what data elementsare needed for the indicator, including thebases for percentage calculations and break-outs needed. (For example, as percent ofthe country’s population? percent brokenout by gender? percent of citizens?).Specification may require discussionbetween mission and government/agencystaff.

(2) Negotiating with the agency ororganization. Finding the office(s) and staffmembers (statisticians) that keep the data


within each agency; securing permission toaccess the data, if not published; arrangingfor its regular (e.g, annual) transmittal toUSAID.

(3) Learning about the original datacollection procedures and quality. How didthe agency acquire that data, whatdefinitions were used, and what procedureswere used to assure uniformity and accuracyof collection across various jurisdictions?How accurate do agency statistical staffconsider that data element to be? This stepmay require study of detailed technicalmanuals by staff members who havestatistical and data management skills, tounderstand fully the original datacharacteristics.

(4) (Sometimes) Cost sharing, if data arebeing collected by another agency; or aspayment for data from a commercialorganization.

(5) (Sometimes) Extracting data fromoriginal records. If data are available only inindividual records (for example, a paper filefor each court case), rather than compiled asaggregate statistics, considerable time maybe needed to extract data from paper records(or a sample of them), or for data processingto aggregate machine readable case data intothe needed statistics.

(6) (Sometimes) Acquiring data fromnumerous jurisdictions. If data are compiledseparately in several jurisdictions, such asprovincial governments, considerable effortmay be needed to acquire it from alljurisdictions, to assess the extent ofuniformity in the original data collection,and to compile it into aggregate statistics.

(7) Maintaining the data at USAID in formsthat are retrievable over time, usable byvarious staff members, and well

documented as to meaning and source. Thisis likely to involve setting up a spread sheetor MIS data file, particularly if dataelements are broken out by ethnic group,gender, or jurisdiction.

(8) Data analysis and presentation graphs, ifneeded. This may be as simple as showingtrends over time on a graph, but mightrequire analysis of data from severaljurisdictions or compiling and analyzingdata obtained from multiple sources.

c. Comments and Caveats. Secondary sourcesare often a very attractive data source forindicators. A major problem may be obtaininginformation on data quality - how accurate arethe statistics obtained? Can you connectconfidence intervals to the data — e.g., is thepercentage obtained from the data within plus orminus 10 percent of the “true” percentage?Were data elements fully defined for theoriginal collectors or coders, so they classifiedthings (such as type of court case, type of citizenrequest) in the same way across cases andjurisdictions? If data quality is low, indicatorswill be unreliable, and using such indicators totrack change over time may be problematic. It isvery important to assess early on how muchaggregation will be required. We found this tobe a major consideration during the indicatorfield tests.

d. Summary Cost Estimate. Low cost, if data arealready aggregated in needed form; about 2 to 5days of staff time per data element. For multiplejurisdictions, this amount of staff time may beneeded for each jurisdiction. Moderate to highcost, if data are extracted from individualrecords; depends on the number of sources andrecords, and the extent of data extracted fromeach record.

2. Data from contractor or granteerecords


a. Description/Definition. These are dataobtained from USAID’s partners. An examplemight be the records from partners working withthe management systems of civic organizations.This guidance assumes that the data are alreadybeing collected, and just need to be extracted forUSAID indicators.

If new data collection or monitoring systemsneed to be set up, costs would be moreextensive, including developingdefinitions/specifications for all data elements;developing data collection instruments andrecord keeping forms; training data collectors -often these are project staff members; pilotingthe procedures and checking the quality of theresulting data; and setting up data storage andanalysis systems. Usually, these costs will bepart of the grant or contract, but will be anoticeable part of its budget. Estimate about 5 to10 percent of the cost of activity delivery as thecost of data collection.

b. Cost Elements

(1) Specifying the data needed, in whatformats, with break-out groups andpercentage base(s) identified. Assessingdata quality characteristics.

(2) Negotiating with grantees to producethose data, if the specifying was not donejointly.

(3) Training grantees in monitoring andevaluation.

(4) Collecting data.

(5) Data transfer and processing. Are thedata already in computer-readable format?Is any further coding or transformation ofthe data needed to produce the statisticneeded for the indicator? Data cleaning ofcase level data may be needed to removeany duplicate cases, examine the extent of

missing data (if more than one data elementis obtained per case), to make sure that alldata elements are within the appropriaterange, etc.

(6) Data analysis to produce “user friendly”presentations for USAID and itsstakeholders, such as graphs showingtrends, breakouts for different locations orsubgroups, or other analyses to showinstances of higher or lower results, to beused for program improvement.

c. Comments and Caveats. Applicability fornational level data may be limited, unless thecontractor is working with the whole country, orcollecting data from representative samples.This could involve much higher costs if themonitoring system involves multiplecomponents, although these costs will usually bepart of the funding for that activity.

d. Summary Cost Estimate. Low-cost (e.g., 2 to5 days of staff time, per indicator), assumingthat the monitoring system already exists, but itmay be a bit higher the first time the data areacquired and analyzed at themission/contractor/grantee level, as proceduresand presentation graphs are developed. Costswill also be higher if the grantee requirestraining in data collection and analysis orsupervision in carrying out the work.


C. Using Judgmental or Narrative Sources

1. Expert panels and peer judgments

a. Description/Definition. This refers to usingthe judgments of people about a given topic asthe major source of data. These might be purelynarrative judgments of quality factors, orjudgments recorded on one or more ratingscale(s). The classic example is peer review ofscientific proposals and manuscripts forpublication. In the field of democracy andgovernance, an expert panel might be used forreviewing the qualities in written documents,such as rating election laws for their conformityto international guidance for conductingelections. People believed to be unbiased butinformed observers of political or governmentalprocesses might be called upon to rate thecharacteristics of a given process, using thecomponents of an index devised by the missionas the guideline criteria for their rating (seeAppendix C for examples). For example, localNGO leaders might be asked to periodically ratethe “democratic processes” of their localgovernmental bodies that are being assisted by aUSAID project, using rating forms provided byUSAID for the indicator. The “experts” shouldmake their ratings independently, but mightcome together as a panel to discuss theirobservations to see if they reach consensus.Written documentation of the ratings should beexpected, either as a narrative review, or using awritten rating form of specified criteria.

b. Cost Elements (see also Appendix C)

(1) Constructing the criteria or rating scale.This may take several weeks of effort, ifdone anew by the mission, as it requiresspecification of the sub-parts or dimensionsto be rated, clear definitions as to what thesemean, and written directions for what ratersshould do. Requires the involvement ofstakeholders, for reaching consensus on

what are the proper criteria. Examples ofscales and criteria are spelled out in theindicators tables.

(2) Selection of and invitations to raters.This involves identifying raters or panelmembers who: 1) have the needed expertiseand/or are in the location to observe on-going activities; 2) are believed to beunbiased (not a participant or beneficiary ofthe activity to be rated); 3) are literate in thelanguage to be used; and 4) are willing toparticipate as raters (whether on a voluntaryor paid basis). If the use of raters isextensive or will be done annually, a database of information about potential ratersmay be needed by staff member(s)managing the rating operations. Potentialraters must be contacted individually to seeif they are willing and available toparticipate in this task. Individual judgmentsare usually kept confidential, although theraters’ names in an overall panel may bemade public.

(3) Training raters. If this is a new activity,a technical assistance session is needed toorient raters to the nature of their task andwhy it is being done, the meaning and use ofthe specific criteria to be used, and theoperational procedures they will follow.Ideally, this would include rating examplematerials or “vignettes” to train raters in theuse of the criteria and to increase cross-raterreliability.

(4) Costs of the rating process. This mayinclude costs of one or more panel meetings,travel costs for both staff and raters for suchmeetings; potential payments to raters, costof reproducing rating forms and directions,also any costs involved in collecting therating forms, if the raters do not cometogether for a meeting with projectmanagers.


(5) Compilation and analysis of reviewers’ratings. Tasks and costs for this step willvary, depending on the extensiveness of theratings. For example, if 12 civic and legalexperts rate the characteristics of thenational laws that establish localgovernmental powers, the analysis task maysimply compare and compile their 12judgmental reports into a summary report.In contrast, if local “expert observers”periodically rate (e.g, quarterly) theperformance characteristics of their localgovernments, analysis of these reports islikely to require setting up a statistical dataanalysis system, similar to a survey analysis,for data maintenance and analysis over time.The extent of agreement among raters of thesame entity (reliability) should be examined.

(6) Summary and presentation of data. Thismay be included with data compilation (e.g.,when narrative expert ratings are used), ormay require a separate step for presentingthe findings as indicators, for example ineasy-to-understand graphs.

c. Comments and Caveats. This may be anappropriate data collection method when theindicator refers to the quality of a majordocument or process. The credibility of this typeof indicator depends heavily on the credibilityof the raters themselves, and their reputations asunbiased and well-informed about the relevanttopic, as well as how systematically the ratingcriteria are developed and carried out. If theratings are done more than once, withcomparisons across time, attention must be paidto across-time reliability, especially if differentraters are used at different time points.

d. Summary Cost Estimate. Specific costestimates will vary substantially, depending onthe extensiveness of the rating to be done.

2. Content analysis of press coverage orother documents

a. Description/Definition. This method refers tothe use of research staff for systematic contentcoding of publicly available documents, such asnewspaper accounts, political flyers, localgovernment budget reports, and so forth. Themethods might be used to construct indicatorsof, for example, the “fairness” of coverage of allpolitical parties by local media, media reportingon corruption, the types of issues covered bypolitical parties’ campaign literature, or theadequacy and transparency of publicly availablegovernment budgets.

b. Cost Elements

(1) Gathering documents to be contentanalyzed. Careful thought needs to be givento the selection of documents, if it is to be asample of some larger total population ofmedia, such as “press coverage.” Whichnewspapers and/or other press will beincluded, for what time period? Willregional and/or local publications beincluded, or only those from the capitalcity? Systematic sampling methods may beneeded, if the total is too large for completeanalysis. A person or agency (e.g., a pressclipping service?) must be designated tomonitor relevant publications and carefullyextract all relevant materials. If agencydocuments will be analyzed, an explicitdefinition is needed for which documents toinclude. For example, if local governmentbudgets are the target documents, thedefinition of “local governments” must beestablished, then a listing of them specifiedand possibly sampled. The unavailability ofrelevant budget documents from some localgovernments would itself be an importantdata element, for example, as the percent ofgovernments lacking required budgets.

(2) Establishing coding criteria. Unless theindicator measure is simply a count ofdocuments (or perhaps inches of press


coverage), the elements to be coded fromeach document, and the criteria for definingeach element, must be carefully specified inadvance. For example, in reviewing localgovernment budgets, what items should thecoder extract or rate? What are the criteriafor “fairness” if one is reviewing presscoverage of political parties? Often,establishing the criteria and their definitionsshould involve stakeholders and/or expertsin that content area to help establishcredible criteria. Pilot tests of the draftcoding criteria, the written forms to be used,and the coding conventions should beundertaken to check for clarity andagreement on definitions.

(3) Selecting and training coders.Sometimes coders will be staff members ofUSAID or from a contractor/grantee, ortemporary staff may be hired for thispurpose. Staff doing this task should havesome background in the content domain ofthe documents to be coded, for example, inaccounting for analyzing budgets, or inpolitical study for analyzing press coverageof political parties. A training session (e.g.,one to three days) will be needed to orientall coders in the nature of the task, to clarifywhy it is being done, to achieve a detailedunderstanding of each code category, topractice coding sample materials, to test forreliability among coders, and to discuss theprocedures that will be used. Severalpractice materials should be coded anddiscussed for resolution of any problems ordifferences in how to code items. Then,reliability among coders should beestablished by having all coders code thesame document independently, andchecking the percentage of agreementamong their coding data.

(4) Coding the documents. Coders use theestablished codes and other criteria tocomplete the rating form for each document.

Costs of this step can be established duringtraining, by observing the average timeneeded by coders for each document,multiplied by the number of documents. Tocontinue reliability checks, a sample ofdocuments should be coded twice, bydifferent coders, with their codes assessedfor agreement.

(5) Entering data from coding for analysis.A statistical analysis package will often beneeded for this step. Data are entered, thenchecked for accuracy and completeness,using the same methods as for other originaldata collections, such as survey data.

(6) Analyzing and presenting data. Usually,the first step in the analysis of content codeddata is producing simple frequencies foreach content category. Often, it is desirableto present these with examples of the typesof original material included in eachcategory, such as verbatim quotationsshowing “fairness” or the lack of it inpublications reviewed. Further analysisshould usually examine the coded material,broken down by sub-categories, such as“fairness” in capital city versus localnewspapers, or completeness of budgetpresentations in localities helped by USAIDversus those not receiving USAID’ssupport. This may require 5 to 20 days of astatistical analyst’s time.

c. Comments and Caveats. This method issomewhat similar to the use of expertjudgmental raters, but content coding placesgreatest emphasis on objectively rating thestimulus materials. If the code categories areclearly constructed, the ideal is that all trainedraters will reach the same coding decisions oneach document. In contrast, for the use of expertraters, the emphasis is on the competence of theraters and their capabilities to judge complexmaterials in an unbiased fashion. They will notnecessarily view the materials similarly, and


different judges might be chosen because theyrepresent different areas of expertise.

d. Summary Cost Estimate. This will varyextensively, depending on the extensiveness ofthe documents to be coded and the number andcomplexity of code categories to be rated.Again, most of the costs are for staff time to dothe specific tasks, although moderate cost mightbe involved in acquiring the documents to becoded. Calculate staff time needed for each costelement, as listed above, then add a generousamount for underestimation.

D. Cost Factors for Sample Surveys

1. Background

Modern survey research has become a complexfield with well established norms ofquestionnaire design, sample design, field work,and data analysis, which promises to producereliable and valid data if carefully conducted.Surveys have become popular in countries thatrecently emerged from long years ofauthoritarian rule, and local USAID missionshave available to them numerous organizations(NGOs, local universities, and for-profitenterprises) that claim expertise in this complexarea. Often these organizations do not possessthe full range of expertise that one associateswith highly professional survey work. Rather,one will find local organizations that might beexpert at sample design, but not at questionnairedesign. In rare instances when both of thesetalents are found locally, it would be even moreunusual to find competent data analystsassociated with the same local organization. Onemight find local organizations that have beenconducting demographic surveys, whichnormally means a high level of expertise insample design and field work, but no knowledgeas to how to structure or analyze a survey of (forexample) democratic attitudes and behaviors.Other local organizations have become verygood at election polls, which generally consist

of “quick and dirty” samples and questionnaires,since the only variable of real interest is thevote. In determining costs, then, it is veryimportant for strategic objective teams to keepin mind that unless the questionnaire, sample,field work, and analysis are competentlyperformed, the resulting data may be seriouslyflawed.

One alternative to designing and implementing asurvey on one’s own would be to add questionsto surveys (usually commercial in origin) thatare already being implemented on a periodicbasis. This is a low cost alternative, if data foronly a few straightforward indicators arerequired. There are some disadvantages to thisapproach, since multiple questions may berequired to explore complex concepts, such asattitudes or beliefs. “Buying in” may thereforework better for questions related to behavior. Itwill also limit opportunities to explore therelationships between various data elements. Forexample, a mission working on improving thejustice system might want to know why somecitizens think that justice is being administeredfairly while others do not. Only by being able torun an analysis on the data set relating justice tosocio-economic, demographic, political, andother variables will these patterns emerge.Finally, while national samples are important,missions often need special purpose samples onspecific geographic areas where USAIDfocuses, or on specific beneficiary groups orpotential beneficiary groups. General purposesamples will rarely if ever provide this level ofinformation. It is therefore very important tolook at the sample design when consideringbuying questions in a pre-existing survey.

2. Cost estimates

The cost of surveys is considerable, althoughthis will depend in part on whether citizens,NGOs, government departments, or some otherentity is being surveyed. The cost may be morethan reasonable if a given survey allows the


collection of data for a number of indicators.Obviously, no one will choose to conduct asurvey to collect data for one or two indicators.

Cost estimates are based on extensiveexperience in Central America. Costs should beadjusted somewhat based on GNP per capita forhigher or lower cost areas. The following costsare initially based on actual direct costs toacquire the services described. Final costs, asshown in the table at the end of this section, willattempt to factor in institutional overheadexpenses.

a. Questionnaire design. Cost estimate = $8,500. This aspect of the process is the only onefor which it is not easy to provide reliable costestimates. That is because it is difficult toprovide cost accounting for what often becomesa very diffuse process. Typically, questionnaireshave become “committee projects” in whicheach stakeholder inserts a list of “must-have”items. There are several problems with such aprocess. The first is the risk imposed when non-specialists write survey questions to measuredemocracy and create poor items, resulting inpoor quality data. Measuring democracy throughsurveys has occupied some of the best minds inthe political science field for the past thirtyyears. While there may be no single universallyagreed upon set of questions that measuredemocracy, there are countless ways to do itwrong, and committee-designed questionnairesby laymen are risky ventures. Committeeinvolvement should come at the outset, wheneach involved unit should specify what it istrying to gain from the survey, and the expertsurvey researcher should indicate if thoseobjectives can be achieved. The actual writingof survey questions is best done by surveyspecialists.

United States nationals, expert in questionnairedesign, familiar with the local culture, andfluent in the main language of the country beingstudied, normally receive a daily consulting fee

of $325 to $435/day, but could review andrevise an instrument in two (six-day) workweeks, including travel to/from the country to bestudied, for direct costs in the neighborhood of$5,220 plus travel (est. $1,000) and per diem (14days @ $150), for an estimated rounded total of$8,500. The proposed expert should haveextensive experience with the design of surveysand not be merely a consumer of surveys housedin national data banks. There is no substitute forextensive, on-the-ground field experience inquestionnaire design and implementation.

b. Sample design. Estimated cost = $4,500. Allsurveys must be based on a probability sample,otherwise known as a random sample. If theselection of respondents is not random, nothingcan be said about whether results meaningfullyrepresent the country, and follow-on surveys tomeasure change will be completely unreliable.Fortunately, this is an area in whichconsiderable expertise exists locally. Mostcountries have experts in sample designassociated with government agencies (e.g.,ministries of agriculture, labor, the censusbureau). These individuals have sometimesdeveloped satisfactory “sample frames,” i.e., the“list” from which the respondents are to bechosen. These lists are often detailed censusmaps in which each dwelling unit in the countryis shown. These maps are not without theirproblems, however, since they are sometimesvery out of date, and therefore do not reflectrecent population shifts resulting from rural-urban migration. Another even more seriousproblem is that sometimes access to these mapsis highly restricted by the military or byagencies that jealously guard their “property.”Often, however, USAID can use its contactswith the government to overcome problems ofaccess, especially when it is prepared to pay forthe cost of copying the maps.

Sample design, then, involves selecting a localexpert trained in the science of sampling andproviding him/her with the sample frame. Local


experts can often be hired for $100/day, and thedesign of a national sample should not involvemore than four weeks of work in the hands of acompetent technician, for an estimated cost of$2000. Obtaining and copying census mapscould add an additional $2,500 to this cost.Alternatively, a local firm might contract for thefield work and supply the sample design expert,but USAID would need to review the resume ofthat expert to be certain that the individual hastraining and experience in sampling.

Another factor the strategic objective teamsneed to consider is whether a national sample ora specialized sample is required. When there aretarget populations of special interest (perhapsbecause of the mission’s work with thesepopulations), specialized samples need to beused or perhaps added to the national sample.But those samples need to be “random.” Forexample, if USAID is working with members ofa group of local civil society organizations, thena list of members needs to be obtained and whatis known as a “systematic sample” would bedrawn. Then, comparisons can be made betweenthe changes in attitudes and behaviors of thespecialized groups compared to national trendsuncovered in the national sample.

Sample size always is subject to great debate,but it should not be. Sample size for nationalsamples is entirely independent of the size of thepopulation of the country; a sample of 1,200respondents from Costa Rica will have theidentical margin of error as a sample of 1,200respondents from China, given that each sampleis appropriately selected. Normally, a margin oferror4 of less than 3 percent is consideredsufficiently precise for most work in the socialsciences since 95 percent of the time the surveywill be able to pick up changes in nationalsentiment no greater than + or - 3 percent of the

actual value had everyone in the country beeninterviewed. A sample of 1,200 respondents hasa margin of error of 2.9 percent. If this samplewere increased to 2,000, the margin of errorwould only drop to 2.2 percent, so the addedcost is normally not justified. However, whenUSAID is interested in specialized subsets of thenation (e.g., the northern zone versus the south,or the indigenous population versus the non-indigenous population, or women versus men),the level of precision for subsets depends uponthe sample size of each subset. Thus, with sixethnic groups in the country, each group ofproportionately the same size, then a totalsample of 1200 would yield a sample size foreach group of 200, with a margin of error ofplus or minus 7.1 percent for ethnic groupbreakdowns. If one wishes to further subdivideeach ethnic group by gender or education or age,then the margin of error becomes very wide. Insum, it is important to establish sample size witha clear notion of the breakdowns one wishes tosee in the analysis.

c. Data collection. Estimated cost = $12,600.This is sometimes thought of as the mostexpensive part of the process but in fact it isoften less expensive than the analysis phase.Many local firms have expertise in conductingsurveys. Their costs are (or should be) very lowsince the interviewers are hired on a per-surveybasis, and normally paid an hourly wage with nofringe benefits (if interviewers are used fromlocal areas). Experience in Central Americafinds that labor costs to the survey firm rangefrom $2-4 per interview, based on wage rates ofabout $1/hour. This includes an interview ofbetween 30 and 45 minutes, travel time, locatingthe household and the pre-selected respondent.Added to those costs are the costs ofsupervision, with one supervisor for each 5 or 6interviewers. If we estimate $8 per hour forsupervisors (including per diem), and pro-ratethose costs for a team of six interviewers, thiswould add an estimated $64 per team-day. Thetotal direct labor cost per interview, including

4Technically, we are referring to the “confidenceinterval.”


supervisory cost, would come to about $5.50.

For a national sample, there are travel costs forbringing interviewers into a central location fortraining. It is difficult to estimate this cost sincemuch depends on the distances interviewershave to travel. In smaller countries, the capitalcity can be used to train all of the interviewers.In larger countries, trainers may have to travelfrom location to location. Additional travelcosts, normally requiring jeep rentals, are for thetransportation of supervisors to the various fieldsites. This transportation also includes the costof moving questionnaires to the field and backto the central office. Experience has shown thata budget of around $6,000 is reasonable for thetransportation costs (jeep rental, fuel, insurance)of interviewers and supervisors. Interviewersshould normally be experienced, and thereforetraining does not have to take more than a fewdays.

Summing up the costs of data collection, with aper-interview cost of $5.50, a sample of 1,200plus supervision and transportation should costabout $12,600. In higher cost areas, where wagerates are considerably higher than shown here,these costs would increase proportionately.

d. Cleaning, coding, and data entry. Estimatedcost = $1,800. Once the field work is complete,questionnaires need to be cleaned and coded. Ifthe survey has been done well, nearly all itemswill be closed-ended and therefore no codingwill have to take place. If many open-endeditems are used, costs can spiral out of control.Data entry is performed by highly experienceddata entry clerks, normally earning fairly lowwages. A standard questionnaire could normallybe entered in 15 minutes. Each survey has to beentered two times, the second time being forverification purposes, so total data entry timeshould come to 600 hours for a sample of 1,200.Total costs should amount to $1,800 when laborcosts are $3.00/hour for data entry clerks.

e. Analysis. Estimated cost = $26,000. This stepneeds to be carefully thought through bystrategic objective teams, as seldom will the“data speak for itself.” What we normally wantto know is how variables relate to each other.For example, we want to know if rural citizensare less supportive of their political system thanurban dwellers, or if males are more politicallytolerant than females, or what level of educationis associated with acceptably high levels ofpolitical tolerance. We might be interested incomparing regions or project areas to thenational norms. We almost certainly want toexamine indexes for various dimensions ofdemocracy (tolerance, opposition to militaryrule, etc.) rather than relying upon a singlequestion or number to tell us everything. Surveyfirms often generate masses of tables, but littleor no analysis. In this area, technical assistanceis needed so that the investment in the priorsteps will have real payoff for USAID. It couldtake an experienced researcher up to 60 days toprepare a statistical analysis and narrative for anational sample. If a specialized sample wereincluded as well, the cost could go higher. It isassumed that the cost per day would be $435 x60 = 26,100 or rounded off to $26,000.


Overall estimates for a sample survey (costs perindicator depends on how many indicators use

Task Rounded

1. Questionnaire design $8,500

$4,500

3. Data collection

4. Coding and data entry $1,800

$26,000

Sub-total

Indirect cost and fringes [forinternational consultants]

below. $32,040

$85,440

A note on indirect costs: For each of categories

vary widely depending upon how the work iscontracted. Normally, a very efficient method is

an IQC. Such a procedure reduces considerablythe contract negotiation efforts of USAID staff.

or an NGO. In both of these scenarios, sub-contracts would normally be used to obtain the

above. Indirect costs, then, would varydepending upon the indirect cost rate and fringe

E. Rapid Appraisal and Other Qualitative

These methods may or may not be useful forindicators of overall results. They are likely to

methods described above, as they dependheavily on the observer’s or interviewer’s skills,

collected. They are particularly valuable forquick information for planning purposes or on

learning about the reactions of participants tothe activities supported by USAID. They may be

over time, as the results may be substantiallyaffected by minor variations in how the methods

1. Direct observations of field activities

data collector visits sites where activities ofinterest are occurring, such as observing

“town meetings” conducted by localgovernments, or observing courtroom

observer should be prepared with a list of topicsto be observed and recorded, but this is usually a

recording of all actions or a pre-coded ratingform. This method is particularly appropriate for

they can be observed unobtrusively and forchecking whether program activities as

plans. The methods can become more rigorousand accurate by using sampling methods to

Sometimes data can be obtained on processesaffecting individuals (such as applying for a

information” processes, or treating all ethnicgroups equally in civil service proceedings) by


having data collectors “pose” as citizensrequesting those benefits, or by asking actualusers to serve as data collectors. Then, data arerecorded about that instance of governmentservice, such as: was clear and full informationprovided; was the official polite but efficient;was the process easy to follow and not too timeconsuming for citizens; are all citizens treatedequally or do some receive preferentialtreatment?

b. Cost Elements

(1) Planning for the field observation. Evenwith more open-ended methods, carefulpreparation is needed to clarify whatindicator topics will be assessed, whatactivities are to be observed, and whether theobservation sites will be a representativesample from a listing of all sites or, instead,an informal selection of those sites mostconvenient for observation. Usually, at leasta topic list is prepared to guide observations,and sometimes a more formal writteninstrument is used, e.g., with prepared ratingscales and questions for the observer toaddress. Sometimes permission forobservation or meeting attendance must beobtained. This planning step may requirethree to ten days of staff time.

(2) Selection and training of observers. Sincethe observer is the measurement tool in thismethod, the observers should be skilled inunbiased observational methods,knowledgeable about the content area andspecific topics to be observed, and welloriented in the specific procedures andrecording methods to use. Training inanthropological field methods is ideal. Ifmore than one observer is used, anorientation session will be needed tocoordinate their understanding of theunderlying concepts and procedures to beused. Allow one to two days per observer fortraining, plus time for selecting and hiring.

(3) Collecting the observations. This is theactual field observation with initial datarecording. To estimate time needed, multiplythe number of sites to be observed by theobservation span—perhaps a full day forelection observations, or two hours forattending a town meeting. Be sure to allowtravel time and costs if needed. The numberof sites needed will differ depending on thepurpose: a few sites (3 to 10) may besufficient for exploratory purposes, to “get afeel for how things are going,” but a moresystematic sample will be needed for arepresentative picture of that program.

(4) Processing the observations. Datacollected in field notes or observation formsmust be further processed by the observerafter leaving the field site, such as bytranscribing notes into full descriptions, orsummarizing several types of informationinto narrative, or writing full item responseson a standard format. This step can take asmuch time as the initial observation; forexample allow about one day for thisprocessing step for each day of observations.This step is needed to translate the observers’notes, observations, and insights intorecorded forms that are useable by otherpeople.

(5) Analysis and reporting. Sometimes theobservers’ report(s) are the final step if onlya few sites are being observed, but oftencross-site analysis is needed. This involvescomparing sites for similarities anddifferences on each topic observed, lookingfor the range of practices and activitiesobserved, discussing the major categorieswithin each topic, and assessing whetherdifferent types of sites differ in identifiableways. Unless the observations have beencoded into numeric formats such as scales orratings, the presentation will usually be in theform of a narrative report or “cross-caseanalysis.”


c. Comments and Caveats. This family ofmethods is very useful for project monitoring,particularly for assessing the extent andprocesses of implementation of USAID’sinterventions. Whether direct observations willbe useful for annual indicators is not entirelyclear. A key caveat for use of this type of datacollection for indicators: if data from fieldobservations will be used over time, then themethodology should be kept the same over time.This is difficult to maintain when the majormeasurement tool is individual observations andinsights.

d. Summary Cost Estimate. Costs will vary a lot,from about 15 days minimum for informalobservation of a few sites, to several months ofexpert staff time for a full observational study inmultiple sites.

2. Focus groups and key informantinterviews

a. Description/Definition. These are methods forobtaining target people’s opinions, observations,and understandings about the topics of interest.A focus group brings together individuals froman identified category (which may be allcitizens, or those who participated in an USAID-funded program, or those in particular roles).Discussions by small group discussions of 6 to12 people are facilitated by trained focus groupleaders to elicit the participants’ views in somedetail. Usually the range of opinions on thefocus questions are explored, with careful andgentle probing to understand why people holdthose views. The discussion is recorded as fullyas possible, by recording equipment or a skillednote taker. Each focus group should behomogeneous so that the more strata of societythat need to be queried, the higher the cost andtime involved.

Key informant interviews are often used withgovernment officials or leaders of a particularactivity to explore the interviewees’ roles,

priorities, actions on a particular issue, oropinions on specified areas of their expertise.Usually, the interview is guided by a set ofopen-ended questions, rather than closed-endedsurvey-type questions. The interviewer must beknowledgeable in the content areas beingdiscussed, skilled in devising probe questions toamplify the interviewees’ responses, diplomaticto keep up good rapport during the interview,and able to record quickly key elements ofanswers. These interviews are best done in-person, but can sometimes be done bytelephone, if the phone system works and theinterviewee is collaborative (not threatened orlikely to be defensive).

These are good methods for exploring a contentarea, prior to designing a more formalsystematic data collection tool such as a surveyquestionnaire. They are also used when thepriority is to understand the interviewee’sperspective, such as the meaning they attach togiven events, their priorities and value choices,or how they interpret a topic of interest, such asa law they are supposed to implement. It is lessclear whether and how these methods would beused for indicators to assess change over time,as the particular selection of focus groupparticipants or key informants is likely to affectthe data obtained.

b. Cost Elements

(1) Planning for topics to be covered. This isa key to obtaining useful data. It may requiredocument review, discussion withstakeholders, etc, to identify major issues andsub-topics to use as probes for fullunderstanding.

(2) Selecting focus group participants or “keyinformants” to be interviewed. This requiresspecifying clearly what types of people aredesired and why. How will they be selected?How will they be contact to arrange forparticipation or interview time and place?


How will locations be arranged for the focusgroups?

(3) Selecting and orienting focus groupleaders, interviewers and recorders. Trainingwill be needed, particularly if the people inthis role have a good content background, butlack the facilitation or data collection skills.

(4) Conducting the groups or interviews,with data recording.

(5) Processing the initial data recorded. Thisrequires transcribing tapes, if used, orsummarizing notes taken during the group orinterview.

(6) Data analysis and reports. Usually asummary report is organized by majorthemes or topics, preserving the “flavor” oforiginal material by using examples andquotations to illustrate the points. Analysis isqualitative, using words rather than numbers.

c. Comments and Caveats. Usually these datashould not be reported quantitatively, such aspercentages of those who participated, becauseaudiences may believe the percentages are froma sample survey and represent a largerpopulation’s opinions. Percentages may alsomislead.

d. Summary Cost Estimate. This is fairlyinexpensive for small scale projects; allow atleast 2 person-days for each focus group (forpreparing; selecting, and contacting potentialmembers, a facilitator, and a recorder duringeach group; and transcribing/summarizing aftereach group). Additional time is needed foroverall planning of topics and summarization.For open-ended interviewing, allow about 3hours of staff time for each hour of actualinterviewing; more if travel will be needed.

3. Case studies

a. Description/Definition. A case study bringstogether multiple sources of information toexamine a particular occurrence, site, or otherphenomenon within its natural context. Oftenthis method is used to examine complex socialsystems or institutions as they change over time.Traditionally this research method has beenused to understand how and why the casechanges over time, the processes needed toimplement a complex activity, or thecontributions of multiple actors andorganizations to broad scale development, suchas “systemic change.” Usually, diverse types ofinformation are brought into the case report,such as interviews, site visit observations,document review and analysis, and evenquantitative data about that site. Multiple casestudies can also be conducted, with cross caseanalysis to compare features and summarizefindings.

For indicators, the use of case studies isuncertain but could be helpful. For example,brief narrative case studies could help illuminatethe meaning behind a complex quantitativeindicator. Or a series of case studies might beconducted over time to assess progress incomplex processes such as institution building.For example, a set of 12 case studies of civicorganization development might document theirprogress, if done annually and compared overtime. Some aspect of national developmentmight be examined as a single case, such as thefactors leading to the passage of a major pieceof legislation affecting human rights.

b. Cost Elements

(1) Identifying the factors to be examined inthe case study. Usually this requiresbackground reading of relevant documents,consultation with stakeholders, and/orpreliminary interviews to specify why thecase study(ies) are being done, and what tolook for. A topic list for things to be doneand observed while on site should be


developed, often supplemented by open-endedquestions for each person to be interviewed.This is usually done by the person(s) who willdo the case study. However, additional casestudy researchers may need to be hired,oriented, and trained. Allow several days ofstaff time; more if complex backgrounddocuments must be reviewed, or multiple sitevisitors work together to develop procedures ormust be oriented/trained in the topics andprocedures developed by others.

(2) Selecting the sites for case examination.There should be a clear rationale as to whythe specific sites are selected, when severalsites are included. They might be sites thatmade particularly strong or weak progress,for example, with the case studies conductedto identify reasons for these results. Theymight be sites using different strategies ormodels to reach a program objective, withthe case studies done to illuminate how thesestrategies work in practice. Preliminary datamay be needed to characterize sites on thedimensions of interest, in order to select siteswisely. If the sites are individualorganizations, permission may be needed toconduct the case studies.

(3) Collecting the information. Often themajor sources of information are derivedfrom a site visit, which includes briefobservations of on-site activities andinterviews with several types of individuals,such as program administrators, “front line”staff, and participants, or local governmentofficials, politicians, and involved citizens.Focus groups might also be conducted. (Seemethods above in 2.a. for key informantinterviews and focus groups.) Documents areoften collected while on site, such as costrecords, local press reports, example caserecords, public notices or announcements,etc. It is preferable to have at least two sitevisitors make each visit, so they can compareobservations, contribute new probes for

interviewing, and increase the validity of thefindings. Other information might beobtained from external sources, such asrelevant government statistics, press reportsof events concerning the case, records ofUSAID’s program activities, and so forth. Inaddition to the time needed on site, allowtime for contacting those to be interviewedprior to the visit, setting up group meetings,securing permissions for observing activities,and making travel arrangements, if needed.Don’t forget actual travel costs.

(4) Compiling the case study. Usually, notesfrom on-site observations and interviewsneed to be completed and amplified afterleaving the site, while the material is fresh inthe observer’s memory. Then, all the sourcesof information are used in writing eachnarrative case study, generally following theoutline provided by the key topics developedin the planning stage. Charts and diagramsare often helpful to show the flow ofactivities, the network of factors involved indecisions, and so forth. Allow at least 5 to 10days of analysis and writing time per case.

(5) Cross-case analysis and summary reports.Case studies tend to generate lengthy sets ofnarrative materials, even if each case is confinedto 10 to 15 pages. A summary of findings isessential, even from a single case, and especiallyfrom multiple cases. Use cross-case analytictechniques for comparing multiple cases.

c. Comments and Caveats. The insightfulness ofcase studies depends on the skills of the peopleconducting them. Like the other methods in thissection, the person is the major instrument forcollecting data, and should be well prepared inthe use of qualitative methods.

d. Summary Cost Estimate. Each case studyinvolving about one week’s site visit by twopeople will require 4 to 6 weeks of time in totalfor preparation, travel arrangements, conducting

254 Appendix C: Using Scales, Indexes, and Scorecards for Performance Measurement

the visit, collecting and reviewing othermaterials, analysis of all information, andwriting the case. Allow additional time for theoverall planning for a series of case studies, andfor cross-site analysis and summarization.

Appendix C: Using Scales, Indexes, and Scorecards for Performance Measurement 255

APPENDIX C: USINGSCALES, INDEXES, ANDSCORECARDS FORPERFORMANCEMEASUREMENT

A. Introduction

As Part 1 of this Handbook has emphasized,developing and using indicators appropriatelycan be a complex process with manyconsiderations. Often, there are no clear-cutanswers to questions concerning which are thebest indicators for a given result. This appendixamplifies considerations for a family ofpotential indicators involving rating scales,indexes, and scorecards. Before turning to thesemeasures, however, missions should rememberthe principle of “parsimony”: do not search for acomplex tool or explanation if a simpler tool isavailable to do the job. As applied to indicators,if simple counts or statistics from secondarydata are applicable to your activities and theirresults, then a complex index is not likely to beworth the trouble and expense required for itsdevelopment and data collection.

In using measures involving scales, indexes, andscorecards, two concepts are of centralimportance:

C Validity - Does the indicator measure whatwe want to measure? Do various audiencesfor the indicator agree on its meaning?

C Reliability (or replicability) - If the indicatordata were collected again, by a differentperson or a short time later, would theyyield the same answer?

Measures which rely on qualitative datacollection may do well in meeting the validity

test, but can fall down on the reliabilitycriterion. When the data for democracyindicators are very subjective and not collectedin a standardized way, there is a risk that we endup measuring differences among data collectionteams rather than actual changes inperformance.

The three methods described in this appendixcan be used to develop indicators that help inquantifying complex results. Three suchapproaches are 1) rating scales; 2) indexes,which combine ratings into an overall score toserve as the indicator; and 3) scorecards orchecklists. These tools permit the quantitativeanalysis of answers that derive initially fromratings or assessments by people. While thesemethods are often based on the “subjectivity” ofindividual judgments, strategies are available forenhancing their reliability (see below). Suchjudgments are not necessarily inappropriatebases for measures. For example, progress inscientific publications and biomedical researchgrant awards rests heavily on the subjectivejudgments incurred during “peer review”concerning what is “good science.” Many otherfields—art, literature, some Olympic sports, andso forth—depend on the judgments of expertsfor determining what is good performance.

When the mission’s purpose is to facilitate thedevelopment of complex democracy-relatedinstitutions (such as “legislative capacity” or“organizational infrastructure”) it is unlikelythat simple measures will capture thecomplexity or magnitude of the changes beingattempted. Using a simple but invalid indicatorin such circumstances can often becounterproductive, if it encourages heavyemphasis on those activities that are counted, tothe neglect of other essential changes in thedevelopmental process. Further, indexes whichinclude the whole range of dimensions believednecessary to the result are often goodmanagement tools to aid program developmentand self-assessment. Using such indicators


fosters the twin goals of performancemeasurement: reporting to higher levels theresults achieved via USAID’s resources andusing the measures as feedback mechanisms tofoster actual achievement of desired results.

Another major concern in choosing indicators isthe resource burden for developing the indicatorand collecting the data. More complexindicators are likely to cost more; especially ifdata need to be collected from a large number oforganizations. Sometimes, systematic samplingcan reduce the data collection burden, but stillproduce reliable data; for example, use a sampleof NGO’s or local governments relevant to aprocess, or a sample of all the bills consideredby a legislature during a session. Alternately, ascorecard or checklist approach that is simplerthan an index may provide a “good enough”indicator, when data are needed from a largenumber of entities. Budgeting for data collectionfrom or about numerous organizations mayrequire similar resources to a population surveyfor collecting data from individuals. (SeeAppendix B of this handbook, on “AssessingData Collection Approaches and Costs”).

B. Definitions

A scale is a rating device that presents a rangeof responses. Terms like “excellent” and “high”define one end of the scale, while “poorly done”and “low” define the other.

1 2 3 4 5

Low High

Scales allow us to gather information on “soft”dimensions of democracy, such as the quality oflaws passed, the fairness with which laws areapplied, the extent to which a complex set ofprocesses have been completed, etc. A scaleenables the transformation of the complexhuman judgments on whatever dimension isbeing rated into numbers that can be combined,averaged, and otherwise processed statistically.

But the numbers and the derived statistics canalso be misused, if the underlying ratings are notvalid and reliable.

Scales for individual ratings can be combinedinto a multi-item index containing multiple bitsof information about the thing being measured.A common example of an index is a knowledgetest in education, which uses the percentage ofitems correctly answered as the overall measureof learning. An index may be ordered in termsof a set of steps or stages that are expected tooccur sequentially, called here a MilestoneIndex. An index can also be simply thecombined score from rating scales for a numberof characteristics of the process or object beingrated (see examples below).5

Scorecards and checklists are similar to indexesin that they also combine multiple bits ofinformation. Here, we use the term scorecard fora scoring system in which each item is a discrete“yes” or “no” response, not depending on arating scale. Sometimes each item must reach athreshold level before it is checked as “yes.”When appropriate, a scorecard can be easier todevelop and use than the more complexjudgments usually required for an index. Furtherexamples are provided at the end of thisappendix.

C. Strategies for Increasing the Reliabilityof Scoring for Scales, Indexes, andScorecards

5Unfortunately, the term “scale” is often used inboth everyday language and methodological literatureto mean both a single-dimension rating scale, as usedhere, and a combination of multiple items to assess theoverall concept, which we are calling an “index.” Thus,Quality of Life Scales in the health literature andcommonly used “attitude scales” are multi-itemindexes, in our terminology. We have tried toconsistently distinguish these terms, to avoid confusingreaders.


Scales. While it may be straightforward to thinkof the factors we need to measure using ratingscales, scales are not necessarily reliable toolswhen a number of different people will beapplying them. There are a number of concernswhich need to be addressed to ensure that scalespass the test of being both valid and reliablemeasures:

C Understand that different people tend to userating scales differently; i.e., some seldomuse the ends and show a tendency to ratemost items toward the center. Others tend touse the extreme values extensively. Oftenraters who have participated in an activitytend to rate it positively.

C For the examples included here, eachelement is scored or rated on a scale from 1to 5. Often people wonder how many valuessuch a scale should have. There is no setanswer to this, but the convention is usuallyto provide a large enough range for thoseusing the scale to show differences amongscale values, but not so many values thatthey get bogged down in fine distinctions.We suggest the 1 to 5 range, with “1”representing “no action,” or “never done”and “5” meaning “excellently” or “at everyopportunity.” We include the middle value“3” for raters to use when they are trulyneutral, or feel that the positive aspects onthe item are balanced by the negative or noprogress aspects.

C Introduce strategies to help raters achieveconsistency in ratings. One of the best waysto do this is to define what each point on thescale means, by defining the detectableincrements of change. For example, point 1,the NGO has no long term plan; point 2, theNGO has a rudimentary plan but objectivesare not clearly articulated and resourceexpectations are unrealistic; and so on upthe scale. This approach is called aMilestone Scale - see example D-3 below. If

this approach to defining the points on thescale is used, then the total score will equalthe number of changes that have occurred.

Indices. A number of additional strategies applyto both individual scales and their combinedindices:

C Train raters—even with well-defined scalesthat are to be combined into an index, ratersmust be trained. This gives ratersopportunities to practice, to ask questionsabout the meaning and interpretation of eachitem, and to compare or calibrate theirratings, so that to the extent possible,different raters will score each item with thesame number for a given situation. It isoften helpful in a training session to do“trial” ratings, for example of a writtenexample situation, then to compare anddiscuss any differences in ratings. Trainingreduces the differences in scores whichemerge as a function of differences in raterexperience or knowledge. The need fortraining exists even when raters are expertsin a field. It is also a good idea to pre-testthe index.

C Some “pointers” that might be included ininstructions to raters are the following: a)When using such rating scales, there are noconclusive “right or wrong” ratings,particularly for values in the middle of ascale; b) The preferred strategy is to usecomparative judgments, among the items inone index or among several similar entitiesthat are the focus of the index being rated(advocacy organizations, laws being passed,etc.). For example, when using the CSOAdvocacy Index (see below), it isappropriate for the raters to thinkcomparatively about the items—which onesare we doing better on, and which ones lesswell? Then score them accordingly. Use ofcomparative judgments is one reason why itis good to have external observers do the


rating, when possible, so they observe thevariations among CSO’s, for example, andrate them comparatively; c) It is alsoimportant that scales and indices used asindicators of USAID’s progress are notscored too highly on the initial rating. If thescores are already close to “excellent”during the baseline, there is no room toshow improvement later! This is called the“ceiling effect” in measurement. If theactivity is already being done excellently,why should USAID provide funding forimproving it?

C Whenever feasible, use the same team ofraters each time the indicator data arecollected, or a panel with “revolving”members, so that new raters are trained and“calibrated” by older members. For “peerreview,” it is important that the peers be inagreement concerning their standards andhow to assess them, i.e., the meaning of thescales in their ratings that will be used toconstruct an index.

C Using the average ratings among multipleraters will also tend to counter individualdifferences in perceptions, and will makethe overall aggregate indicator scores morereliable.

D. Examples of Milestone Scales, Indexes,and Scorecards

Often indicators working groups noted thatseveral key results which they were trying tomeasure fell into a pattern of stages orprocesses. For such situations, it appeared that aMilestone Index might be helpful, with datacollection designed to provide evidenceconcerning progress among the stages on eachscale. In other cases, the result to be measuredinvolves rating the qualities or characteristics ofa product or process. In this case, an index ofthese important qualities is a good indicator,composed of a set of rating scales.

With this type of measurement index, thespecific elements assess the amount or quality ofprogress for each step or characteristic, withinthe relevant units (for example, localgovernment legislative bodies, or the country asa whole). Diverse types of data would be used toprovide evidence for each component scale andfor different steps, while the score for the indexas a whole measures the aggregate assessmentof progress. For upward reporting purposes, themission should use the aggregate index score asthe indicator. For internal management andactivity improvement, the mission and itspartners may want to examine the ratings onindividual scales, to detect areas of strength andweakness.

The collection and use of data for the indicatorcould be viewed as using several “levels” ofevidence. For example:

Level 1 evidence: Local scoring of theprogress on each relevant component byUSAID and/or contractor/grantee staff,using narrative evidence and judgments,when “harder” data are not available. Staffmembers would compile the evidence used,such as news accounts, focus groups,secondary data, their own experience withinthe organization, etc., focusing on whether


or not the situation on that component haschanged since the previous annualassessment of that Index. The assessmentcould also be conducted jointly with thegroup whose progress is being evaluated.6

Level 2 evidence: Review panel of outsideexpert observers (such as local orinternational academics, newspaper editors,major foundation or NGO executives,prominent citizens, or performance measurecontractor staff) scores each component ofthe Index, using evidence supplied by theentity being evaluated, thecontractor/grantee and the USAID mission,as well as relying on their own knowledgeand judgments. Usually, the panel wouldmeet to discuss their independent ratings, aswell as to assess the types and consistencyof the evidence available. While level 2evidence is preferable to level 1 from thestandpoint of the neutrality of the raters, itmight not present the same opportunity forincreasing the understanding of thosedirectly involved in the program.

Level 3 evidence: Systematic data areavailable and used for scoring the extent ofchange on some or all components of anindex, for example from national statistics.An example here would be the elements thatmake up a Consumer Price Index. Ifmultiple sources of data are available for

individual components, they may need to beassessed and combined by a Level 1 orLevel 2 process.

Often the most appropriate type of evidencedepends on the entity being assessed, and thetypes of evidence available. Sometimes, acombination of evidence from several levelsmay be most appropriate. For example, whenmultiple NGO’s are each providing similaractivities, such as advocacy, then intensive selfscoring of the Advocacy Index may be mostfeasible, backed up by an external review ofevidence or spot checks of a sample of theNGO’s by outside experts, to ensure the overallvalidity of self ratings. When a single nationalentity is being assessed, such as the quality of anew law or the quality of legislative processes,then a single review panel of expert observers islikely to be most useful.

6The use of self assessment is rather controversialcurrently, with some evaluators advocating“empowerment” and other participatory methods whichuse self assessment extensively, and othersdiscouraging use of such methods, with concerns thata substantial bias may enter these ratings. We proposea middle ground for indicators, with self assessmentused when it appears to be the only feasible oradvantageous (from a training perspective) method, butwith cross-validation of such ratings by an externalobserver for at least a sample of the entities, assuggested above.


D-1 Example: Civil Society Organization (CSO) Advocacy Index

Because measuring CSO advocacy is a concern for all four Agency democracy objective areas, we haveused this as an example of a multi-component index. The index is intended to be scored concerning oneor more advocacy issues for a CSO. For example, given this issue, to what extent and how intensively didthe CSO perform each component? The elements following each component are given as examples of thetypes of evidence to think about when scoring that component. Not all elements are likely to be relevantto every situation. The total score needs to be accompanied by a narrative explaining progress orstrengths and weaknesses.

Each of the index components should be rated on a scale such as the following:

2) CSO collects input about the issue: (Circle one number)

None, not at all 1 2 3 4 5 Extensively

The scores for each component are added to form the score on the overall CSO Advocacy Index, whichwill range from 7 to 35 (given 7 components, as at present).

Components of the CSO Advocacy Index (scored for one or more issues):

Score:

1) Issue is timely, with the following possible elements:- Issue is of vital concern to the group’s constituents- Issue is critically important to the current or future well-being of the CSO and/or its clients,

but its importance is not yet broadly understood- New opportunities for effective action exist- At least a few key decision makers are receptive to the issue

2) CSO collects information and input about the issue, with the following elements and examples:- Relevant government agencies and their respective roles in the issue are identified at national

and local levels; knowledge and positions investigated- General public input is solicited (including from women and minorities) on the issue via

public meetings, focus groups, etc.- Representative input is collected on the issue via surveys (including from women and

minorities, where appropriate)- Existing information and data on the issue is collected, such as for summaries or positions

papers- Policy analyses, such as the legal, political, social justice, or health aspects of the issue, are

conducted


3) Formulating a policy position on the issue, with the following elements and characteristics:- Policy formulation done in participatory (and gender-sensitive) manner- Policy being advocated exists in writing, with formats and levels of detail that are appropriate

for various audiences and policy makers- Policy position is clearly and convincingly articulated- Rationale for policy is coherent, persuasive, and uses information collected in component 2- Presentation of policy position uses attractive and effective formats, such as graphs

4) Obtaining and/or allocating resources (especially time and money) for advocacy on the issue, with thefollowing elements and examples:

- Contributions collected from members, interested citizens, and/or from other organizations(businesses, foundations, religious groups, etc.)

- Financial or other resources assigned to the issue from within the CSO- Volunteer time to help advocate for the issue obtained and well managed- International agencies with interests in the issue area identified, and their procedures for

applying for financial support determined- (Other resources?)

5) Coalition and network building, to obtain cooperative efforts for joint action on the issue, with thefollowing elements and examples:

- Other groups and individuals with interests concerning the issue identified or persuaded totake an interest (may include govt. organizations which share concerns)

- Coalition formed (defined as any type of joint working group)- An existing or new coalition or network activated, such as by having informal contacts, joint

meetings, identifying common interests, sharing resources, etc.- Joint or coordinated actions planned (see #6 and #7 below, for carrying out the actions)

6) Taking actions to influence policy or other aspects of the issue, with the following elements andexamples:

- News releases generated or public meetings held- Members/citizens encouraged to take appropriate actions, such as writing letters to legislators- Active lobbying conducted for the policy position, such as by testifying in hearings, personal

visits to legislators, etc.- Model legislation drafted and circulated to legislators- Policy relevant position papers and recommendations disseminated, based on the input

collected and coalition’s joint interests


7) Follow up actions, after a policy decision is made, to foster implementation and/or to maintain publicinterest, with the following elements and examples:

- Monitoring the implementation of a newly passed law or policy, such as by making sure thatauthorized government funds are disbursed, implementing regulations are written anddisseminated, checking implementation in field sites, asking members for feedback on howwell it is working, etc.

- Some staff or volunteer time and resources are allocated to the issue or policy for monitoring- [If desired policy was not passed] At least a minimal level of advocacy methods maintained

to take advantage of next opportunity for pressing the issue, perhaps with a reformulatedapproach or different specifics

- [If desired policy was not passed] Public awareness and interest in issue monitored, to lookfor examples, incidents, opportunities to create or renew a sense of urgency on the issue

Data could be collected for this index by one or more of the methods laid out earlier in this section.


D-2 Example of an Index for Assessing the Quality of Legislative Processes (in General) or theQuality of Processes for Legislation on a Particular Issue

The index suggested on the next page could be more fully developed and used as an indicator to captureprogress in strengthening legislative processes. It can also be used to monitor the legislative process withrespect to one particular issue, such as reforming the criminal code or the laws on decentralization.

As shown in the draft below, a legislature would score between 8 and 40 points. Criteria for each of theeight components would need to be more fully fleshed out. The components could be changed or adaptedto fit local circumstances. An expert panel could be convened once each year toward the end orimmediately after the legislative session, with each panel member independently rating that session. Thenthe panel would convene to discuss their ratings, perhaps with an opportunity for panel members toadjust their original ratings. Scores of individual members would be added and divided by the totalnumber of panel members in order to obtain an average. Both the total scores and the extent of variabilityamong the raters would be of interest. Too much variability might suggest that the scale is insufficientlydetailed or that raters need more training in its application.

Over time, however, a panel of raters is likely to become more consistent, as they share understandingsabout the components, and become more observant of processes throughout the year. Data could bedrawn for making the assessment from the legislative record, documents produced by the researchservice, panel members’ contacts with MPs and staff, etc.


During the period from xxx to xxxx (or when working on the target legislation), how well did the legislature,in general, perform on each of the following quality characteristics?

Score:

1) Use of facts and figures, of data drawn from reference service, Internet, or other sources

2) Thoughtful use of legislative models and experience from other countries

3) Use of analysis from reference service or other sources (think tanks, NGOs, etc.), including projectionsof impact on the country from various provisions

4) Focus on constituent interests in considering legislation; MPs in frequent touch with constituents;information flows in both directions

5) Inclusion of NGO and expert testimony; openness to hearing outside testimony; reference to testimonyin considering legislation

6) Wide involvement in committees and debate of deputies from different factions and parties

7) Relevant government agencies and ministries work with legislature on drafting key items of legislation

8) Drafting competence, clarity of language, internal consistency, and consistency with other laws

Score each component on the following scale:

Never or Poorly 1 2 3 4 5 Excellently, at every opportunityModerately


Stage 1) - Interested groups propose that legislation is needed on issue.(Stakeholders, public interest groups, think tanks, key donors, and others are active in pressing fornew legislation, such as by sharing relevant legislation and models from other countries, solicitingpress coverage, sponsoring public fora or hearings, scheduling meetings with government officialsand elected officials, publishing papers, etc. Stakeholders may form networks or coalitions toadvocate.)

Stage 2) - Issue is introduced in the relevant legislative committee/executive ministry.(The issue is raised, discussions are being held, studies/research are being conducted, hearings arebeing conducted by committee)

Stage 3) - Legislation is drafted by relevant committee or executive ministry.(If drafted by the executive, it is submitted to the legislature.)

Stage 4) - Legislation is debated by the legislature. (Might include additional committee hearings, and/or consideration of alternative model laws,projecting likely impact of various provisions, and broad participation from delegates andstakeholders representing different factions and parties. This stage might extend over a considerabletime, if revised versions are needed before passage is scored.)

Stage 5) - Legislation is passed by full approval process needed in legislature.

Stage 6) - Legislation is approved by the executive branch (where necessary).

Stage 7) - Implementing actions are taken.(Such as executive agencies passing operating regulations, information disseminated to citizensabout the new law, administering agencies informed and provided with technical assistance to fillany new role required by the law, etc.)

Stage 8) - No immediate need identified for amendments to the law.(Shows law was well crafted) and lawmakers believe that given time it will have its intended effect.

D-3. Example of a Milestone Scale for Measuring Progress in Achieving a Legal Reform

Making progress in reforming the law is fundamental to several results in the democracy and governanceframework. The process of obtaining a reform is not, however, such a simple one. It involves a number ofsteps or stages and missions aiding the beginning of a difficult reform process may want to measureprogress short of the actual achievement of the legal reform.

For a given piece of legislation, this indicator could simply be scored by mission staff or an expertobserver, with the score being the highest stage (Milestone) passed during that year. If the processes arenot sequential, then the score could be the number of stages completed. (For assessing the quality ofthese processes, see example D-2 above.)

Stages 2 and 3 above may need some adjustment depending on whether the executive branch is theprimary drafter of legislation, which the legislature then reviews.D-4 Scorecards


Score WeightWeighted

Score

1. Has a double entry bookkeeping system X 2 =

2. Has an annual audit X 2 =

3. Produces accurate and timely quarterly financial statements X 1 =

4. Controller is a chartered accountant X 1 =

TOTAL

Yes = 1 point; No = 0 points

Note: The above is intended only to show the format for this type of scorecard. It does not include all thecomponents that might be needed to assess “financial management.”

Another type of index is a scorecard that more simply examines a law, a process, or an organization todetermine whether or not it has each of a set of desirable characteristics. As defined here, a scorecard isan index that uses a simple “yes” or “no” scoring, with a “yes” scored as 1 and “no” as 0 (binary scoring)for each of a number of characteristics or components. A scorecard is usually most appropriate when thejudgments to be made are straightforward (e.g., a court has a judge with a recognized law degree or itdoes not). An index with rating scales for individual components may be better when more complexqualitative judgments are needed.

A weighting system can also be used to assign a greater or lesser value to each characteristic. However,weights should be based on careful analysis, preferably empirical evidence, that those items weightedmore heavily are truly more influential in achieving the overall result being addressed.

The following is an example of the format of a scorecard system for assessing improvement in NGO’sfinancial management:

This very simple yes/no scorecard requires minimal training and inter-rater reliability testing, but it stillrequires some. The scorecard can be adjusted to adapt to NGOs at relatively early or late stages ofdevelopment.

PN-ACC-390

Center for Democracy and Governance

U.S. Agency for International DevelopmentWashington, D.C. 20523-3100

Fax: (202) 216-3232