external verification performance based financing in ... · pdf fileexternal verification...

EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING IN HEALTHCARE IN SIERRA LEONE

CORDAID NAAM BU » REPORT EXTERNAL VERIFICATION JUNI 2014

VOLUME 1 MAIN REPORT

EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 1


TABLE OF CONTENTS

Table of Contents .................................................................................................................................. 1

List of Tables ......................................................................................................................................... 3

List of Figures ........................................................................................................................................ 3

Acronyms ............................................................................................................................................... 5

Acknowledgement ................................................................................................................................. 7

Executive Summary .............................................................................................................................. 9

1 Introduction ................................................................................................................................... 13 1.1 Terms of Reference of the External Verification ...................................................................... 13 1.2 The External Verification team ................................................................................................. 13 1.3 Set up of the report ................................................................................................................... 13

2 Background ................................................................................................................................... 15 2.1 Health Sector in Sierra Leone .................................................................................................. 15

2.1.1 Health Status ..................................................................................................................... 15 2.1.2 Health system ................................................................................................................... 15

2.2 Free Health Care Initiative ....................................................................................................... 16 2.3 PBF in the Sierra Leone context .............................................................................................. 17 2.4 Trends in service delivery ........................................................................................................ 17

3 Methodology of the external verification .................................................................................... 19 3.1 Introduction ............................................................................................................................... 19 3.2 Organisation ............................................................................................................................. 19

3.2.1 Project team and verification teams .................................................................................. 19 3.2.2 Standard working procedure ............................................................................................. 19 3.2.3 Learning approach ............................................................................................................ 21

3.3 Timeframe ................................................................................................................................ 21 3.4 Sources of information ............................................................................................................. 22

3.4.1 Quantitative data ............................................................................................................... 22 3.4.2 Cross cutting issues .......................................................................................................... 24 3.4.3 Qualitative data ................................................................................................................. 24

3.5 Sampling .................................................................................................................................. 25 3.5.1 Facility sampling................................................................................................................ 25 3.5.2 Sampling for Patient Tracing and Patient Satisfaction survey .......................................... 27

3.6 Quality assurance ..................................................................................................................... 28 3.6.1 Prior to data collection ...................................................................................................... 28 3.6.2 During the verification ....................................................................................................... 29 3.6.3 After data collection .......................................................................................................... 29

3.7 Reliability and significance ....................................................................................................... 29 3.7.1 Reliability of data ............................................................................................................... 29 3.7.2 Equal distribution of facilities ............................................................................................. 29 3.7.3 Significance of PHU data .................................................................................................. 31 3.7.4 Significance of patient satisfaction data ............................................................................ 31 3.7.5 Completeness of data ....................................................................................................... 31

3.8 Data analysis ............................................................................................................................ 32 3.8.1 Systems used .................................................................................................................... 32 3.8.2 Primary Facility Output Data ............................................................................................. 33 3.8.3 Data from Patient and key informant interviews ............................................................... 33

3.9 External verification in the hospitals ......................................................................................... 33

4 External Verification Findings ..................................................................................................... 35 4.1 Introduction ............................................................................................................................... 35

4.1.1 Indicators ........................................................................................................................... 35 4.1.2 Hospital PBF ..................................................................................................................... 35

4.2 Output indicators in PHUs ........................................................................................................ 36



4.2.1 Variation between sources of information ......................................................................... 36 4.2.2 Influence of missing data on calculations ......................................................................... 38 4.2.3 Differences in data in facilities .......................................................................................... 39 4.2.4 Differences per level of facility .......................................................................................... 40 4.2.5 Differences per geographical area .................................................................................... 41 4.2.6 Satisfactory and unsatisfactory entries ............................................................................. 42

4.3 Crosscutting Issues .................................................................................................................. 45 4.3.1 General ............................................................................................................................. 45 4.3.2 Specific indicators ............................................................................................................. 47

4.4 Hospital external verification .................................................................................................... 48 4.4.1 Ola During Children Hospital ............................................................................................ 48 4.4.2 Princess Christian Maternity Hospital ............................................................................... 49 4.4.3 Non-PBF hospital .............................................................................................................. 50

5 Patient Tracing and Satisfaction Survey .................................................................................... 53 5.1 Patient tracing .......................................................................................................................... 53 5.2 Patient satisfaction ................................................................................................................... 54

5.2.1 Satisfaction scores ............................................................................................................ 54 5.2.2 Satisfaction scores per level of facility .............................................................................. 55 5.2.3 Payment for services ........................................................................................................ 56

5.3 Patient tracing by Councils and DHMTs .................................................................................. 57

6 Systems Assessment ................................................................................................................... 59 6.1 Introduction ............................................................................................................................... 59 6.2 Accessibility and equity ............................................................................................................ 59 6.3 Autonomy and accountability PHUs ......................................................................................... 60

6.3.1 Capacities ......................................................................................................................... 60 6.3.2 Planning and management of small projects .................................................................... 60 6.3.3 Financial management in practice .................................................................................... 61 6.3.4 Delays in payment ............................................................................................................ 62

6.4 Community involvement ........................................................................................................... 63 6.5 Separation of functions in the PBF programme ....................................................................... 63

6.5.1 Councils ............................................................................................................................ 64 6.5.2 DHMTs .............................................................................................................................. 64 6.5.3 MOHS ............................................................................................................................... 65 6.5.4 MOFED ............................................................................................................................. 65

6.6 Definition of indicators .............................................................................................................. 66 6.7 Contracts .................................................................................................................................. 66 6.8 Hospital PBF ............................................................................................................................ 66

6.8.1 Contracts ........................................................................................................................... 66 6.8.2 Implementation.................................................................................................................. 66 6.8.3 Indicators ........................................................................................................................... 66 6.8.4 Expenditure ....................................................................................................................... 67 6.8.5 Non-PBF hospitals ............................................................................................................ 67

7 Discussion ..................................................................................................................................... 69 7.1 Quality ...................................................................................................................................... 69 7.2 Data quality .............................................................................................................................. 69

7.2.1 Missing data ...................................................................................................................... 70 7.2.2 Data consistency ............................................................................................................... 70 7.2.3 Case definitions................................................................................................................. 71 7.2.4 Triangulation ..................................................................................................................... 71

7.3 PBF light ................................................................................................................................... 71 7.4 Financial management ............................................................................................................. 72

8 Conclusions and Recommendations .......................................................................................... 73 8.1 Conclusions .............................................................................................................................. 73

8.1.1 The general and specific objectives of the PBF programme ............................................ 73 8.1.2 The Terms of Reference .................................................................................................. 73

8.2 Recommendations ................................................................................................................... 74 8.2.1 Validation workshop .......................................................................................................... 74 8.2.2 Short-term recommendations ........................................................................................... 74 8.2.3 Clarification of the operational manual ............................................................................. 77



8.2.4 Longer-term recommendations ......................................................................................... 77 8.2.5 Future developments of PBF ............................................................................................ 77 8.2.6 Hospital PBF ..................................................................................................................... 78

LIST OF TABLES Table 1 Summary of activities and outputs ...................................................................................... 21 Table 2 Sources of information and tools used to capture this information ................................ 23 Table 3 Methods used for assessment of cross cutting issues ..................................................... 24 Table 4 Data sources and tools for qualitative information ............................................................ 25 Table 5 Number of PHUs in the PBF project in 2012 per district ................................................... 26 Table 6 Number of PHUs selected per Council (urban areas) ........................................................ 26 Table 7 Geographical areas selected in each rural Council ........................................................... 27 Table 8 Distribution of PHU type per district .................................................................................... 30 Table 9 Domains for assessment of hospital performance ............................................................ 36 Table 10 Totals per output indicator sampling 4 months 2012 ...................................................... 36 Table 11 Extrapolation of all service attendance based on missing data ..................................... 39 Table 12 Differences between data sources above 25% in PHUs .................................................. 40 Table 13 Differences between data sources FP per level PHU ....................................................... 41 Table 14 Percentage satisfactory entries per Council per indicator .............................................. 42 Table 15 Differences between average satisfactory scores (IV and EV) per district ................... 44 Table 16 Percentage satisfactory records per level health facility ................................................ 45 Table 17 Percentage of persons who could not be traced in EV ................................................... 53 Table 18 Reasons for not interviewing persons in EV .................................................................... 54 Table 19 Patient satisfaction scores and contributing factors ....................................................... 54 Table 20 Patient satisfaction per level of facility ............................................................................. 55 Table 21 Persons interviewed who were asked to pay for services .............................................. 56 Table 22 Average, minimum and maximum amounts paid ............................................................. 56 Table 23 Involvement in action panning of PHUs ............................................................................ 60 Table 24 PBF payments to PHUs requested by MOHS to MOFED ................................................. 61 Table 25 Percentage of quarterly payments, traced in PHUs during the EV ................................. 62 Table 26 Roles of HMC according to chairs HMC ............................................................................ 63 Table 27 Registers reported out of stock by PHUs .......................................................................... 70

LIST OF FIGURES Figure 1 Family planning 2011 - 2013 ................................................................................................ 17 Figure 2 ANC 4 2011 2014 .................................................................................................................. 18 Figure 3 PHU deliveries 2011 - 2013 .................................................................................................. 18 Figure 4 Children fully immunised 2011 - 2013 ................................................................................ 18 Figure 5 Selection process for patient sampling ............................................................................. 28 Figure 6 Number of PHU selected per district .................................................................................. 30 Figure 7 Formula for calculating sample size .................................................................................. 31 Figure 8 Percentage completeness of family planning records per districts ............................... 32 Figure 9 Reasons for unavailability of records per information source ........................................ 32 Figure 10 Comparison Output indicators per source of information ............................................ 37 Figure 11 Comparison sources of information per indicator ......................................................... 37 Figure 12 Extrapolation of service utilisation FP based on missing data ..................................... 38 Figure 13 Extrapolation of ANC service utilisation figures based on missing data..................... 38 Figure 14 Distribution of facilities by differences ............................................................................ 39 Figure 15 Geographical spread of facilities with differences data sources .................................. 41 Figure 16 Distribution of Absolute Differences between IV and EV .............................................. 43 Figure 17 Plot box differences satisfactory entries ......................................................................... 44 Figure 18 Crosscutting issues IV and EV ......................................................................................... 46 Figure 19 Distribution crosscutting issues scores in EV ................................................................ 46 Figure 20 Distribution of crosscutting scores in IV ......................................................................... 46 Figure 21 Kono District Comparing Crosscutting Indicators IV and EV ....................................... 47



Figure 22 Comparing crosscutting indicator Administration IV and EV ....................................... 47 Figure 23 Comparison crosscutting indicator Stock Outs IV and EV ............................................ 48 Figure 24 Ola During Children Hospital IV and EV .......................................................................... 49 Figure 25 Trend in IV scores in Ola During Children Hospital ........................................................ 49 Figure 26 Princess Christian Maternity Hospital IV and EV ............................................................ 50 Figure 27 Trend analysis Princess Cristian Maternity Hospital ..................................................... 50 Figure 28 Comparison PBF and non-PBF hospitals in EV .............................................................. 51 Figure 29: Building blocks of RBF ..................................................................................................... 59 Figure 30 Word cloud investments in PHUs ..................................................................................... 61 Figure 31 Roles in PBF in Sierra Leone ............................................................................................ 64 Figure 32 Relations in the quality system ......................................................................................... 69 Figure 33 PHU-F1 form box totals ..................................................................................................... 70 Figure 34 Step by step introduction of sampling IV ........................................................................ 76



ACRONYMS

AIDS Acquired Immunodeficiency Syndrome

ANC Antenatal Care

CHC Community Health Centre

CHP Community Health Post

CSOs Civil Society Organisations

DHIS District Health Information System

DHMT District Health Management Team

DHS Demographic and Health Survey

DMO District Medical Officer

DPPI Directorate of Policy, Planning and Information

EmONC Emergency Obstetric and Neonatal Care

EPI Expanded Programme on Immunisation

EV External Verification

FHC Free Health Care

GDP Gross Domestic Product

HMC Health Management Committee

HIV Human Immunodeficiency Virus

HMIS Health Management Information System

IMNCI Integrated Management of Neonatal and Child Illnesses

IV Internal Verification

IVT Internal Verification Team

LC Local Council

LGFD Local Government Finance Department

M&E Monitoring and Evaluation

MCH Maternal and Child Health

MCHP Maternal and Child Health Post

MDGs Millennium Development Goals

MOFED Ministry of Finance and Economic Development

MOHS Ministry of Health and Sanitation

NGO Non-Governmental Organisation

NHA National Health Account

OM Operational Manual

PAD Project Appraisal Document

PBF Performance-Based Financing

PHU Peripheral Health Unit

PMTCT Prevention of Mother to Child Transmission

PRSP Poverty Reduction Strategy Paper

RCH Reproductive and Child Health

RCHP Reproductive and Child Health Project

SDHSP Strengthening District Health Service Project

SDPS Service Delivery Perception Survey

SLDHS Sierra Leone Demographic and Health Survey

TBA Traditional Birth Attendant

TOT Training of Trainers

UN United Nation

UNICEF United Nations Children Fund

WB World Bank

WHO World Health Organisation



ACKNOWLEDGEMENT

To a great extent, this verification consisted of qualitative studies and included inputs from health care

staff and the population in service areas of the Peripheral Health Units (PHU). Only part of the data

was collected in a quantitative way, by looking at registers in health facilities and by using the existing

health information systems. The success of any External Verification exercise relies heavily on the

collaboration and openness of key people working in health facilities, policy makers and people

seeking health care. Having targeted 235 PHUs, their HMCs and 8 clients per PHU for structured

interviews, almost 2000 people were interviewed at local levels. Without exception, we received full

collaboration and dedication from all participants, not in the last place from people ‘on the road’ who

guided us to hard- to-reach areas in order to trace patients. Thanks to all these people it was possible

to collect a tremendous amount of valuable information for our analysis and recommendations. This

information will probably also be used for future comparison and trend analysis.

The Councils and District Health Management Teams (DHMTs) play a crucial role in steering,

management, monitoring and capacity building in relation to the performance based financing (PBF)

programme. For the External Verification we relied on them for planning and facilitation of the

fieldwork and as resource for data collection and interviews. All 13 DHMT’s and 18 Councils were

interviewed and shared valuable information. A special word of thanks goes to all the Monitoring and

Evaluation (M&E) officers and Planning Officers at the DHMTs and Councils who assisted us in the

collection of the required quantitative PBF data. We spent many hours in the M&E offices and always

met collaborative attitudes.

Without the patience and willingness of our counterparts at central MOHS and MOFED to share

information and additional data, the External Verification exercise would have been impossible.

The PBF programme in Sierra Leone does not operate in a vacuum as several international

Development Partners contribute to the progress and improvements of the country’s healthcare

sector. The External Verification team had extensive meetings with several of them. This enabled us

to put our findings in a broader perspective. We hope that this report provides useful inputs for further

strategising and alignment of all the interventions in the health sector.

Cordaid is grateful that this assignment was entrusted to us. Our activities not only focused on

performing the technical verification, but also on empowering different actors in the health sector and

on increasing knowledge on internal verification. Thus, it is our sincere hope that the insights and

knowledge that was built during the exercise will strengthen Sierra Leone’s Performance Based

Financing programme. We hope it may also set an example for efficient organisation of independent

external verification in other countries.

The EV team consisted of Dr Jaap Koot, team leader, Mrs Marjan Kruijzen, project manager, Mr Frank

van de Looij, PBF-expert, Mr Chenjerai Sisimayi, data analyst, and Mr David Yambasu, field

coordinator.

PBF External Verification Team



EXECUTIVE SUMMARY

Introduction

The health sector of Sierra Leone receives support from the World Bank through the Reproductive and

Child Health (RCH) Project, which has the objective to increase utilisation of a package of essential

health services by pregnant and lactating women and children under the age of five. Performance

Based Financing (PBF) is part of the RCH project.

The general objective of the PBF system is: to change the behaviour of health providers at facility level

for them to deliver more quality services under the free health care policy. Free health care is a

collaborative effort of the Government of Sierra Leone and Development Partners to achieve the

Millennium development Goals.

The PBF programme exists of payment for six output indicators, corrected for payment for crosscutting

issues and remoteness of facilities. District health management teams and Council officials perform

quarterly internal verification. Part of the programme is a Hospital PBF, piloted in two hospitals in

Freetown.

The Ministry of Finance and Economic Development - IPAU (Integrated Project Administration Unit) of

the Republic of Sierra Leone contracted the Dutch Non-Governmental Organisation Cordaid for the

External Verification (EV) of the PBF component of the RCH project in the health sector in Sierra

Leone, starting on 28 October 2013 and ending on 28 April 2014.

The Terms of Reference (TOR) for the external verification were:

1. To review the accuracy of the facility data from the registers and other records;

2. To analyse the data of the first full year of PBF implementation (2012);

3. To review the roles and responsibilities of the different PBF stakeholders and advise on the areas

of improvement if necessary;

4. To evaluate the benefits of the performance based financing in term of services delivery,

strengthening the health system information (verification of data and timely reporting), the

governance of health facilities (management of human resources, environmental health, financing,

etc.).

Cordaid developed a methodology for the External Verification and fielded teams to visit all 19

Councils, 13 Districts, 47 Community Health Centres (CHCs), 52 Community Health Posts (CHPs) and

130 MCH Posts, sampled. Furthermore four Hospitals were visited for an external verification of the

Hospital PBF.

Findings of the External Verification of Output Indicators The External Verification of Output Indicators in the PBF system showed:

Considerable, sometime significant differences exist between aggregated numbers in internal and

in external verification. With exception of deliveries, the aggregated Internal Verification (IV)

figures are 12% - 73% higher than the EV.

Recorded attendance in the IV is in the majority of indictors also higher than other sources of

information (Health Management Information System (HMIS), or F-forms1).

In general, the aggregated figures from various sources of information differ, whereby the EV

showed most concordance with F-reports.

The differences cannot be attributed to missing data.

At facility level for all indicators the differences between sources of information are large, often

more than 25% higher or lower.

The differences in recording are spread over the country and not related to specific districts.

Lower-level health facilities show larger error margins than higher-level facilities.

There is no statistically significant difference between IV and EV as regards percentage of

satisfactory or unsatisfactory entries in the records. There is no significant difference per level of

facility, and not per district.

Crosscutting issues

The external verification for crosscutting indicators showed:

The scores for the crosscutting issues in the external verification were consistently lower than in

the internal verification in nearly all districts, for nearly all the indicators.

1 F-forms or “returns” are standard reporting forms filled by PHUs and sent to the District Health Management Team for entry into the automated HMIS.



In the external verification standardised assessment criteria were applied, reducing the chances of

personal bias. Those criteria might have been stricter than applied in the internal verification.

Due to the time lapse between 2012 and 2014 differences may have been created, e.g. missing

registers, leading to lower scores.

Worsening supplies of medicines may have caused lower scores in availability of essential

medicines in 2014 compared with 2012.

The high percentage of maximum scores for all indicators in all facilities, as found in the internal

verification in some districts, could not be confirmed in the external verification.

Hospital PBF

The external verification of the Hospital PBF found that:

The EV team gave slightly higher scores in the EV to Ole During Child Hospital compared to the

latest IV (79% vs. 61%), but lower than in other IVs (85%-95%)

The EV team gave slightly lower scores in the EV to Princess Christian Maternal Hospital

compared to the latest IV (84% vs. 89%), in the range of other IVs (82%-89%)

Hospitals not receiving PBF scored lower than PBF hospitals, but one of those scored only slightly

lower, while the score of the other hospital was wide off range.

Patient tracing and satisfaction

The external verification of the patient tracing and satisfaction found that:

92.6% of the patient/clients earmarked for tracing could indeed be identified either by meeting the

clients in person, or by identification by a member of the community.

There is no reason to believe that PHUs recorded “ghost patients” to inflate the numbers of

attendance.

The average satisfaction score of clients was 7.3 (out of 10), with a variation between 4.1 and 9.8.

Client satisfaction was strongly related to short waiting times, friendly treatment, availability of

medicines and non-payment for services.

12% of patients interviewed had to pay for services, although they were supposed to benefit from

Free Health Care.

Systems analysis

Free Healthcare was introduced in 2010, and was supported by several donors an agencies, e.g.

through human resources management, provision of medicines, etc. Free health care has resulted in

considerable increase in service delivery in reproductive and child health services, although recently

there has been a levelling off of service utilisation at a higher level than before the introduction of Free

Healthcare.

The PBF programme works complementary to Free Healthcare, and offers to health facilities a

compensation for the loss of income through patient fees. The programme has been successful in this.

The programme has succeeded in providing more autonomy to health facilities to manage their own

small projects, which contribute to better work environment: more hygiene, better equipped buildings

and better supplies have been achieved.

Financial management is a weak area, with virtually no systems in place at grass root level. Late

payment during the period of review affected continuity of the PBF programme and had high

opportunity costs: PHUs were eager to implement health services improvements, but had to wait for

over one year to get their due payments. Health workers expressed fear that they would not receive

their bonuses after such a long period of delay, e.g. after transfer. Not receiving a performance bonus

created frustration, rather than motivation for better performance.

The programme has succeeded to some extent in improving community contribution to management

of health facilities, although the capacities are still limited.

In Sierra Leone a “light” PBF approach is applied, which means that not all theoretical concepts of

PBF with regard to separation of responsibilities (e.g. Health results Innovation Trust Fund2) are fully

implemented. The Local Council is officially responsible for the health services, but is at the same time

the contracting agency. The DHMT is the technical supervisor and at the same time the internal

verifier. In practice the collaboration between Councils and DHMTs often is not as envisaged in Sierra

Leone’s PBF plan. The DHMTs often operate independently, and Councils do not feel engaged in the

programme.

2 https://www.rbfhealth.org



Conclusions


for them to deliver more quality services under the free health care policy. The EV team concludes

that systems have been put in place and are operational to a reasonable extent in a number of health

facilities (see Chapter 7.1). Further strengthening of the system is possible within the present design

of PBF in Sierra Leone, when a number of implementation issues can be solved.

The EV team concludes that PBF provides cash at facility level to cover the local costs of delivering

services and removing the need for 'informal' fees. Only 12% of the patient paying for those services,

which supposedly are free. Late transfers of PBF funds may have forced PHUs to ask for contributions

for patient records, etc. when funds dried up. Payments by patients may reduce further if PBF

payment improves. Incidental misbehaviour by health workers cannot be ruled out.

The EV team concludes that to some extent PBF provides financial incentives to facilities in order to

increase productivity and quality of care, especially for the identified key indicators. There is an

increase in service utilisation, although that increase is levelling off. There are signs of improved

attention for quality. However, the relation between performance and payments is too weak for health

workers. The incentive system is not transparent enough and payments come so late, that they are no

longer seen as reward for good performance.

Equity of distribution of funds may have taken place using district-based payment formula, but was not

visible for grass root workers. The flow of funds in general was not regular enough to hire contract

workers (with exception of the two PBF hospitals). Outreach may have benefited from PBF funds, e.g.

by repair of motorcycles and purchase of fuel. In general, funds were used for repairs of the building,

furniture, equipment and supplies, water and sanitation, etc. These investments have contributed to

patient satisfaction and higher scores for crosscutting quality indicators.

Recommendations

The MOHS district visits planned for the month of April 2014 will offer an opportunity to confirm with

the Councils the roles and the responsibilities as laid-down in the PBF operational manual. The roles

of the Councils in contracting, in internal verification, and in financial management and reporting have

to be renegotiated per Council, as circumstances and conditions may vary. The roles of HMCs have to

be clarified. New Memoranda of Understanding can be signed to confirm commitments.

During the district visits the MOHS could provide an orientation workshop on quality of internal

verification. The quality of internal verification has to improve: uniform case definitions have to be

applied, and DHMT members involved should understand their tasks well. On the spot double check

of IV report, F-forms and HMIS form (brought from the DHMT’s M&E office) should be introduced to

identify data inconsistencies and resolve them, or explain them.

The validation workshop at the end of the external verification called for simplification of the Internal

Verification, while improving the quality. The idea was to introduce sampling, not only months (one

month per quarter), but also PHUs (e.g. 25% of PHUs). HMIS data would be guiding in payment for

performance, rather than the data from IV. This is possible, but only if certain criteria are met.

The first step in this process is to guarantee data quality of registers, F-forms and HMIS. Facilities

should have the required registers and forms. HMIS and F-forms should be filled completely and

should match. Districts, which cannot meet minimum criteria of HMIS quality, should first bring their

house in order.

The second step is to select PHUs, which meet criteria of data quality, with matching IV and HMIS.

Those with reasonable data quality are admitted to the pool. But they can be removed from the pool if

in a control they are found to be missing the quality standards. From there, step-by-step, more

facilities are added to the pool. NB: quarterly supervision and assessment crosscutting issues should

continue in all health facilities!



1 INTRODUCTION

1.1 TERMS OF REFERENCE OF THE EXTERNAL VERIFICATION

PBF project

The health sector of Sierra Leone receives support from the World Bank through the Reproductive and

Child Health (RCH) Project, which has the objective to increase utilisation of a package of essential

health services by pregnant and lactating women and children under the age of five3. Performance

Based Financing (PBF) is part of the RCH project.


for them to deliver more quality services under the free health care policy.

The specific objectives of the system are4:

1. Provide cash at facility level to cover the local costs of delivering services and removing the need

for 'informal' fees.

2. Provide financial incentives to facilities in order to increase productivity and quality of care,

especially for the identified key indicators.

3. Increase the equity of distribution of resources with funds from PBF allowing facilities to hire

contractual workers and finance outreach activities.

Terms of Reference

The Ministry of Finance and Economic Development - IPAU (Integrated Project Administration Unit) of

the Republic of Sierra Leone contracted the Dutch Non-Governmental Organisation Cordaid for the

External Verification (EV) of the PBF component of the RCH project in the health sector in Sierra

Leone, starting on 28 October 2013 and ending on 28 April 2014.

The Terms of Reference (TOR) for the external verification are:

1. To review the accuracy of the facility data from the registers and other records;

2. To analyse the data of the first full year of PBF implementation (2012);


of improvement if necessary;




etc.).

The assignment is therefore broader than an external verification per se; it encompasses an

assessment of elements of the project design and implementation. The summary of the Terms of

Reference is found in annex 1, in Volume II of the report.

1.2 THE EXTERNAL VERIFICATION TEAM

The EV team consisted of Dr Jaap Koot, team leader, Mrs Marjan Kruijzen, project manager, Mr Frank

van de Looij, PBF-expert, Mr Chenjerai Sisimayi, data analyst, and Mr David Yambasu, field

coordinator.

The field team for data collection consisted of five teams of local experts from the organisations

Christian Brothers, SEND and Njala University, School of Community Health Science. The names of

the team coordinators and enumerations are listed in annex 2, of Volume II of the report.

1.3 SET UP OF THE REPORT

The following chapters describe the background of the health sector in Sierra Leone (Chapter 2), the

methodology of the external verification and the sampling (chapter 3). Chapter 4 describes the

analysis of output indicators and crosscutting issues, as well the hospital PBF. Chapter 5 gives

3 RCHP project Grant Agreement, 2010

4 PBF Operational Manual, version edited July 2013



information on the patient satisfaction survey, and chapter 6 analyses the PBF system. Chapter 7

discusses some key issues and Chapter 8 gives conclusions and recommendations.

The annexes of the report are in a separate volume, and contain Terms of Reference, Team

members, List of Samples Facilities, Specific Council Reports and Case Definitions.



2 BACKGROUND

2.1 HEALTH SECTOR IN SIERRA LEONE

2.1.1 HEALTH STATUS

Since the end of the civil war in 2002, Sierra Leone has made progress in improving the health status

of the population1. Maternal Mortality Ratio (MMR) and Child Mortality Rate (CMR) decreased to MMR

890 per 100.000 live births in 2010 and CMR to 174 per 1,000 live births in 2010, but are at still far

from the MDG targets of 320 and 92 respectively5. Fertility rates dropped from 6.5 in 2004 to 5.0 in

2010, but only 17% of the demand for family planning was satisfied in 2010.

The 2010 document that lays the foundation for the Basic Package of Essential Health Services for

Sierra Leone6, stated “the Health Status of the population compared to other sub-Saharan countries is

critical”.

The disease burden of children under-five consists mainly of communicable diseases and poor

nutrition. Malaria (38%), acute respiratory infection (16.9%) and watery & bloody diarrhoea (9.7%)

together, account for about 65% of all diseases. The stunting percentage (36.4% in 2008) also

contributes to the high disease burden for under-fives.

Inequity in the health system also remains a problem, both in terms disparities between income

groups as well as disparities between geographical locations7. For example, the percentage of births

attended by skilled health personnel is around 28% for the poorest and around 75% for the richest

quintiles.

2.1.2 HEALTH SYSTEM

Infrastructure

Peripheral Health Units (PHUs), i.e. Community Health Centres (CHCs), Community Health Posts

(CHPs), and Maternal and Child Health Posts (MCHPs) deliver Primary Health Care. There are 40

hospitals in the country8. The number of government health facilities has increased to over 1,200 in

2012 compared to 843 in 2006. There are now five Basic Emergency Obstetric Care (BEmOC)

centres each in all the 13 districts. In total, 13 district medical stores have been constructed to enable

storage of medicines and medical products both at national and district levels. Blood Banks have been

established in all district hospitals to provide safe blood for transfusion. A school for training midwives

was established in Makeni9.

Since the start of the National Health Sector Strategic Plan (NHSSP) 2010 – 2015, Government has

embarked on a series of improvements in the health sector. It has increased the total workforce in the

public health sector from 7,164 in 2009 to 8,446 in 2011. Incentive allowances are provided to health

workers in remote communities. Many PHUs are still heavily understaffed, or do not dispose of

adequate infrastructure (electricity, water supply) or equipment7.

Organisation

The health system in Sierra Leone is decentralised, with devolution as the mode of operation. The

Ministry of Health and Sanitation (MOHS) is responsible for formulating government health policies

and for technical guidance to the Councils, who as Local Government Authorities are responsible for

the implementation of health services. The MOHS provides technical guidance in the area of health

5 Source : http://www.countdown2015mnch.org/http://www.countdown2015mnch.org/; Sierra Leone Health Data—2012 Profile

6 Basic Package of Essential Health Services for Sierra Leone, Government of Sierra Leone, Ministry of Health and Sanitation,

March 2010. 7 Koyejo Oyerinde, Yvonne Harding, Philip Amara, Rugiatu Kanu, Rumishael Shoo, Kizito Daoh, The status of maternal and

newborn care services in Sierra Leone 8 years after ceasefire, International Journal of Gynecology and Obstetrics, 114 (2011) 168–173. 8 National Health Sector Strategic Plan (NHSSP) 2010 – 2015, Government of Sierra Leone, Ministry of Health and Sanitation,

November 2009 9 MOHS, Health Sector Performance Report, Draft, July 2012

http://www.countdown2015mnch.org/

http://www.countdown2015mnch.org/



and has an important task in development of human resources for health and in logistics and supplies

of medicines and equipment for the health facilities.

The Ministry of Finance and Economic Development (MOFED) finances most of the health services

through the Councils and finances the human resources in health through the Human Resources

Management Office (HRMO).

In the districts the Local Councils are responsible for most of the service provision to the community,

including health. The District Health Management Teams (DHMTs) manage the primary healthcare

services on behalf of the Local Councils.

The NHSSP identifies the following major challenges in relation to health service delivery: (1) weak

M&E capacity, (2) weak mechanisms for public accountability, (3) minimal involvement of

communities, (4) low motivation of health workers, (5) high attrition rate, (6) weak health information

systems and (7) inadequate budgetary allocations.

Healthcare financing

Both MOHS and Development Partners have increased the funding for the health sector considerably

over the last years and have invested in human resources, supply of medicines, improvement of the

infrastructure, etc. The total health expenditure is estimated at US$ 78.7110

. Funding for the health

sector was estimated around US$ 85 million in 201111

.

The MOHS liaises with multilateral organisations such as WHO, UNICEF, UNAIDS, Global Fund,

GAVI, AfDB, EU, and World Bank and with bilateral organisations such as DFID, Ireland. The

Government of Sierra Leone (GOSL) agreed a Health Compact with most of the Development

Partners and NGOs aiming to make faster progress to achieve the ‘Agenda for Change’ in health and

the Millennium Development Goals12

.

2.2 FREE HEALTH CARE INITIATIVE

The President of Sierra Leone launched the Free Health Care Initiative in April 2010 with the aim to

increase access to health services by pregnant women and children. Removing patient fees would

take away the barriers for the poorest in society to seek institutional health care.

Measures undertaken to support this initiative included:

Undertaking a payroll cleaning exercise that removed around 1000 ghost workers, freeing up

resources, which enabled the Government of Sierra Leone (GoSL) to recruit 1000 legitimate

health workers. This resulted in a 15% increase in the health workforce which was cost neutral;

Increasing all health workers salaries by at least 90% with some highly skilled staff receiving five-

fold rises in their salaries;

Procuring over $10 million of pharmaceuticals and strengthening drug storage and supply

systems;

Initiating a new financial mechanism to provide cash grants to all health facilities for them to

purchase essential supplies;

Accelerating essential infrastructure repairs in hospitals, health centres and district drugs stores;

Conducting a mass communications campaign across the country to inform the target population

of their rights to free care.

Indeed, after the introduction the number of under-fives outpatient consultations increased with 250%

compared to the period before the launch of the Free Health Care Initiative, and this trend is

continuing. Immunisation coverage for children increased from 67% in 2006 to 82% in 2011.

Until today inputs from donors like DFID and UNICEF contribute to the Free Health Care Initiative for

e.g. medicines and salary costs. This collaborative effort is the backbone for improving healthcare in

Sierra Leone.

10

MOHS, National Health Accounts, draft 2012 11

MOHS, Health Sector Performance Report 2011, draft July 2012 12

GOSL, Health Compact, December 2011



2.3 PBF IN THE SIERRA LEONE CONTEXT

Performance Based Financing was launched in Sierra Leone in April 2011, to strengthen the Free

Health Care Initiative.

PBF is a systematic approach to health reforms, which provides incentives for health worker

performance to improve staff motivation and funds for additional investments at grass root level. It

leverages major paradigm shifts in terms of accountability, governance, information systems, planning

and the inclusion of communities in verification and providing feedback. This approach is expected to

have impact on performance of the healthcare system and to have a multiplier effect on efforts of all

partners involved in the Free Healthcare initiative.

Sierra Leone has chosen to implement a “light-PBF”, with a limited set of indicators and a highly

simplified, but well prioritised quality component. The PBF project team has not created new

structures for the different functions within PBF, but utilises existing institutions for contracting and

internal verification.13

2.4 TRENDS IN SERVICE DELIVERY

The Free Health Care initiative has resulted in increase of utilisation of health services. In general, a

positive trend in service delivery figures is visible from the HMIS statistics. Family planning is still

increasing, while Antenatal Care (ANC), institutional deliveries and children fully vaccinated are

levelling off at a substantial higher level than before the start of the Free Health Care initiative.

Unfortunately, during this external verification no quarterly statistics were available from before the

start of Free Health Care in 2010 to quantify the increase.

PBF is a countrywide system, to strengthen the impact of Free Health Care. There is no way to

disaggregate the contribution from PBF to improvement of healthcare services and the contribution

from other support activities, e.g. the support to salary increases, or the provision of essential

medicines. All districts were benefiting from PBF. As mentioned before, the increase of service

utilisation should be considered as the result of a collaborative effort of the MOHS, other Ministries,

Agencies and Departments and all Development Partners.

Figure 1 Family planning 2011 - 2013

Source: MOHS, HMIS

13

MOHS, Performance Based Financing, Operational Manual, Revised Version, October 2013



Figure 2 ANC 4 2011 2014

Source: MOHS, HMIS

Figure 3 PHU deliveries 2011 - 2013

Source: MOHS, HMIS

Figure 4 Children fully immunised 2011 - 2013

Source: MOHS, HMIS



3 METHODOLOGY OF THE EXTERNAL VERIFICATION

3.1 INTRODUCTION

Because service providers are paid according to their performance, verification of reported

performance is a crucial element in any performance-based financing (PBF) programme. Apart from

frequent internal verification, it is common practice to externally verify the program. Reasons for this

include fiduciary stipulations from donor organisations, limited capacities of organisations, which

perform internal verification and/or limited separation of functions.

External verification primarily answers the question whether payments in the program were indeed

valid and legitimate. As indicated in the terms of reference, this assignment also targets to review the

effectiveness of the programme. Therefore, the methodology used for external verification included

several stakeholder interviews, validation workshops and an extensive patient satisfaction survey. This

enabled the consultants to formulate clear recommendations for continuation of the programme.

This chapter explains the methodology that was used in more detail. It explains how data collection

was organized, which timeframe was applicable and which sources of information were used. An

important element of the methodology is the sampling technique that was used to select the PHUs and

patients that were visited to gather information. This technique is explained in a separate paragraph.

Thereafter the methods for quality control and data analysis are explained. The chapter ends with an

explanation the approach, which was used for external verification of the Hospital PBF component.

3.2 ORGANISATION

3.2.1 PROJECT TEAM AND VERIFICATION TEAMS

The final responsibility for the external verification lied with the international project team. This team

was responsible for overall planning, creating instruments and tools, development of training material,

organisation of workshops, quality assurance, data analysis and report writing.

Data collection was done by five verification teams each consisting of one coordinator and three

enumerators. The coordinators and enumerators were all employed by three partner organisations of

Cordaid: Christian Brothers, School of Community Health Science and SEND. Christian Brothers and

the School of Community Health Science provided each two coordinators and six enumerators. SEND

provided one coordinator and four enumerators.

Each verification team moved from Council to Council, covering a total of 18 councils, 235 PHU and

1196 patients in less than two months’ time. While the enumerators visited an average of two PHUs

and 16 patients per week, the coordinator mainly concentrated on the quality control, Local Council

and DHMT verification, data entry in the computer and a limited number of PHU verifications (one per

week). Both the international and national supervisors closely monitored the data collection by moving

from Council to Council, working alongside verification teams and executing spot checks.

3.2.2 STANDARD WORKING PROCEDURE

The verification in the districts encompassed meetings with the Local Councils, the DHMTs, the PHUs,

(traced) patients and community representatives (Health Management Committee, HMC). Each visit

followed a structure of introduction, interview, verification and preliminary feedback. The figure below

shows the approach.



Figure 1 Standard Working Procedure verification visits

Local Council The coordinator of the external verification team introduced the exercise to the Council in a courtesy

call to high-level officials. Thereafter the coordinator conducted meetings with relevant officers,

especially the M&E Officer and the Finance Officer. Topics for discussion were the involvement of the

Local Council in the internal verification, the financial management procedures and reporting. More in

general the place of PBF in decentralisation-by-devolution was discussed. The coordinator verified the

PBF financial reports, if available. All councils received a summary-sheet capturing the main findings

of the verification in the respective Council. These summary reports can also be found in annex 4, in

Volume II of this report.

DHMT The District Coordinator introduced the external verification to the DHMT and interviewed the District

Medical Officer (DMO) and other relevant officers, specifically the M&E Officer and the Finance

Officer. Topics for discussion were the overall progress in district healthcare, implementation of Free

Health Care and PBF systems, the internal verification process (e.g. role of the Councils), reporting

and financial management systems. The coordinator thereafter collected information from the HMIS

system (computers at district level) and verification reports. After completing the assignment the

coordinator presented major findings, completeness of the verification, availability of information and

progress made in the health facilities. The DMO also received the Council summary sheets.

PHU The enumerator introduced the assignment and interviewed the PHU staff. Important topics during

introduction were patient confidentiality and non-disclosure of medical records. Interview questions

focused on how PBF has helped the PHU and what has been done with the money received. Pictures

were taken of purchased equipment or improved infrastructure. Finally the method for patient tracing

will be explained.

Subsequently, the external verification took place and 8 patients were randomly selected from the

registers. At the end of the visit, the enumerator provided feedback on reporting, data quality, overall

progress, cross cutting issues and patient satisfaction. All information was captured in a PHU

summary sheet.

Health Management Committee The enumerator interviewed the Health Management Committee (HMC) and asked their views on the

performance of the PHU and the health system in general. Participation in the decision-making and

spending of PBF payments was an important topic for the discussion. HMC also assisted in tracing

patients.



Patients/clients The enumerator visited 8 randomly selected patients within the service area of the health facility

(maximum 10 KM). Opinion leaders, health staff and the HMC assisted in tracing patients. The

enumerators first sought permission from chiefs or opinion leaders in the Chiefdoms.

Clients receiving family planning were not traced, because of privacy issues. Children of less than five

years old, which visited the PHU for consultation, were also not selected because the recall time

between external verification and actual consultation in 2012 (between one and two years) is too long

for relatives to remember the treatment exactly. Other indicators like antenatal visits, delivery, post-

natal visits and vaccinations were for recall (often with evidence of ANC-cards or under-five cards).

The enumerator assessed whether the patient actually visited the PHU and received the service

indicated in the register. In addition, the enumerator also conducted a brief satisfaction survey (waiting

time, staff attitudes, etc.).

3.2.3 LEARNING APPROACH

During all verification visits and interviews, emphasis was put on mutual learning. The coordinators

and enumerators were trained in such a way that they were able to help all actors understand the

weaknesses and challenges in reporting and verification. The main purpose of feedback was to

explore possible interventions to improve the PBF system. Good examples of this approach were

experienced during both the inception workshop and the validation workshop. During the inception

workshop knowledge on indicator definition and sampling increased. The validation workshop led to

increased insight in the functioning of the free health care system, drug availability and challenges in

information systems.

During the external verification exercise 22 people from three independent organisations were trained

in external verification techniques. Also, several data collection tools were designed and the software

package EPI-info (already used in the DPPI) was introduced for PBF data entry and simple analysis.

The MoHS or other organisations can use these tools for future verification exercises.

3.3 TIMEFRAME

The external verification took place from November 2013 to April 2014, covering 6 months in a

sequence of activities, summarised as inception, verification, analysis and reporting. The table below

shows the output per phase in the verification.

Table 1 Summary of activities and outputs

Work package Main activities Timeframe Output

Scoping

Mission

Study documentation

Field visit two districts (DHMT,

council and facilities)

Assess HMIS reliability

Stakeholder interviews

(MOFED, MOHS, UNICEF,

DFID, World Bank, civil

society)

Stakeholder workshop

November /

December

2013

Adjusted methodology

Facility level verification sheets.

Tool for patient tracing

Interview questionnaires

Planning for interviews

Manual for enumerators

Sampling of facilities

Inception report

Verification of

facility

performance

Training of Council

Coordinators and enumerators.

Verify entry in registers

Consistency between registers

and HMIS

Verify score on quality

indicators

Trace patients.

Measure patient satisfaction

Interviews with facility staff,

DHMT and Councils

January /

February /

March 2014

Filled out verification sheets per

facility.

Filled out structured

questionnaires.

Brief verification reports at

council level.



Data entry (EPI info)

Data Analysis Final database cleaning and

cross checks on data validity

Standard sequential analysis

of HMIS data

Specific comparative analysis

of indicators and cross

sections

Regression analysis

Interviews at national and

council level

March 2014 Data analysis plan

Preliminary data analysis and

summary of interview findings

as part of the issues paper and

workshop presentations.

Validation of

findings and

reporting

One day workshop with

stakeholders and main

decision makers

Report writing

April 2014 Final report on verification

assignment

During the inception phase14

the scope and methodology of the external verification were determined

in detail. It allowed the MOHS to send out official letters to Local Councils and DHMTs, who in turn

communicated with the PHUs and opinion leaders in Chiefdoms about the process of external

verification and patient tracing. A crucial element in the inception phase was a workshop on 19 and 20

November 2013, in which the methodology was discussed, facilities were sampled and instruments for

external verification were aligned with existing instruments. An important example is the elaboration of

definitions used for the cross cutting issues as part of the quality checklist.

A six-day training of verification teams marked the start of the verification phase in the second week of

January 2014. This training focused on acquaintance with general healthcare knowledge, thorough

training in auditing facility registers, interview techniques and data entry. Immediately after the training,

the teams started the verification. The team started in the districts with meetings with the DMO to plan

the exact dates of visits to PHUs, Community Health Committees, DHMT and Council, thus assuring

the availability of staff and required records. The team did the sampling of months and patients to be

traced in the district together with the DMO. While the verification teams worked through districts in

batches and transferred data to the national supervisors and senior consultants, the analysis of district

and PHU data already started. This allowed for corrections and further investigations when questions

arose during the analysis. In some cases the verification teams and the supervisors returned to the

districts for further data collection.

The analysis phase15

covered the entire month of March. During this month, the international

consultants also performed the external verification in the two PBF hospitals. The preliminary results

and conclusions were discussed in a validation workshop on the 20th of March 2014. This resulted in

additional interpretations and recommendations that provided input for this final report.

3.4 SOURCES OF INFORMATION

3.4.1 QUANTITATIVE DATA

As indicated before, the core of any internal or external verification is to investigate the validity and

accuracy of reported patient numbers. In other words: is the information on which the payment was

based consistent with the actual patient numbers that visited the facility? If this is not the case, it is

necessary to understand the different stages of data processing and the challenges that exist during

each of these stages. If the entries in the registers are incorrect, then the facility staff should be

assisted to improve data entry. If differences are the result of data processing at district level, then

capacities of M&E officers or data entry clerks should be improved or errors in information systems

should be corrected. In general, five stages can be distinguished (see figure 2 below). The external

verification in Sierra Leone looked at facility registers as well as actual patient visits, by means of

random sampling of patients and asking for a confirmation of the reported visit.

14

The findings of the inception mission are reported in the PBF External Verification Inception Report, Cordaid, 18 Dec 2013 15

The preliminary findings of the analysis are reported in the PBF External Verification Issues Paper, Cordaid, 20 March 2014



Figure 2 Information Flow in the National PBF programme in Sierra Leone

At the start of the PBF program in Sierra Leone, facilities were to be paid based on the attendance as

recorded in the HMIS. However, later due to challenges in the completeness and accuracy of HMIS

information, it was decided to base payments on the internal verification data. For the purposes of this

external verification we distinguish five sources of data and checked their consistency: (1) patient

tracing, (2) facility registers, (3) PHU-F reports, (4) HMIS system and (5) Internal verification reports.

Table 2 summarises sources of information and tools used to collect the information during external

verification.

Table 2 Sources of information and tools used to capture this information

Source Detailed description Tools used

Patient tracing Interviews with 8 randomly sampled

patients

Excel sampling tool

Structured questionnaire

Data entered in EPI-Info

Facility registers

(referred to as

“external verification”)

Re-counting patients from the following

registers:

- Under five clinic register

- Immunisation register

- Family planning register

- Maternal and neonate health register

(ANC, PNC and delivery)

Excel sampling tool

Tally sheets

PHU verification sheet


PHUF report Information on all six indicators was taken

from the PHUF (1,2 and 3) reports which

were present at facility level.

Excel sampling tool



HMIS Data produced by computerised HMIS

system at district level. If data was not

present at district level, information from

national level was requested

Excel sampling tool



Internal verification Internal verification reports which were

present at the DHMT (district level)



Payment Cash books and ledgers present at facility

Payment requests from MoHS to MoFED

(national level)



Patient visit

•For a specific service, eg. OPD consultation or delivery

Facility registers

•For instance OPD under five or maternal and neonate register

PHU-F report

•Summary sheets that are filled out by PHU staff and sent to DHMT

HMIS

•Data from PHU-F reports entered in HMIS at DHMT level

Payment Request

•Based on HMIS information

Internal / External verification



3.4.2 CROSS CUTTING ISSUES

Apart from patient numbers (output indicators), payment to PHUs is also based on qualitative

characteristics of the facility. These are measured through a checklist consisting of 9 crosscutting

issues (see table 3). During the external verification, the coordinators and enumerators administered

exactly the same checklist and their scoring was compared with the quarter 4, 2012 scoring from the

internal verification. It has to be stressed though that some of the 9 cross cutting issues could not be

verified objectively for the year 2012. Cleanliness of the facility may for instance have improved or

deteriorated. Table 3 provides an overview of the cross cutting issues that were objectively verifiable

and which were not. It also indicates how scoring was done if indicators were not objectively verifiable.

Table 3 Methods used for assessment of cross cutting issues

Cross Cutting Issue Objectively

verifiable

Not

objectively

verifiable

How measured?

Facility attendance register is kept

up-to-date and accurate √

All reports submitted to DHMT by

5th of the following month. √

Monthly minutes of facility health

committee meetings are signed by

chair and securely retained at

facility.

√

A wall chart is displayed, with up-

to-date information on each of the

6 PBF interventions and financial

information.

√

Could not be verified for 2012.

Assessment looked at charts of

the month before external

verification.

All paperwork kept in good order √

The PHU and surrounding area is

clean and sanitary with no medical

waste exposed, no tall grass, etc. √

Actual situation during external

verification was assessed.

Drugs records are accurate and up-

to-date. √

Appropriate waste management

√

Assessment of actual situation.

If burning pit is found it is

assumed it was also present in

2012.

No stock-out of ACT, Amoxycillin

or ORS √

Physical in store or medicines

cupboard

To ensure that the clear definitions were used during external verification, the definitions of the cross-

cutting issues were discussed with the DMOs during the inception workshop on the 19th and 20

th of

November. Definitions and additional instructions were included in the checklist for enumerators (see

annex 5 in Volume II).

3.4.3 QUALITATIVE DATA

In order to review the effectiveness of the PBF program in Sierra Leone, qualitative information was

gathered at different levels of the health system. Table 4 summarises the different sources for

qualitative information and the tools used for data collection. It should be noted that most data at

district, council, PHU and patient level were captured with the use of nominal scales. Patient

satisfaction was measured through ordinal scales. This allowed for statistical analysis after data

collection.



Table 4 Data sources and tools for qualitative information

Level Sources Tools

National Interviews with:

- MOHS, MOFED

- Development partners, (e.g. DFID,

world bank, UNICEF)

- NGO’s (e.g. Save the Children and

Action for Health)

Structured questionnaires

Districts/Councils Interviews with DMO and officials at

Council level (often M&E or finance

officer)

Structured questionnaires

Data entered in EPI-Info to enable

statistical analysis

PHU’s Interviews with in-charge and facility

staff

Interviews with Health Management

Committee members

Structured questionnaires as part of

PHU verification sheets



Inventory sheet for investments and

pictures taken

Structured questionnaires for Health

Management Committee

Patients Interviews with 8 patients randomly

sampled from facility registers

Structured questionnaires for patient

satisfaction



3.5 SAMPLING

It is general practice that external counter verification of PBF programs is only performed in a sample

of the total number of health facilities which are in the programme. Verifying all facilities is time

consuming and costly, which might not be justified by the value of the extra information or payment

corrections resulting from such an exercise.

At the same time, the representativeness of the sample should be ensured. Conclusions from the

sample should be valid for all facilities and therefore samples should be taken at random.

Randomisation also serves another purpose. If each health facility believes it has an equal and

relatively large chance of being selected in the sample, the temptation to intentionally over-report

patient numbers will be reduced.

In PBF programs a sample of 20-25% is generally accepted as sufficient to do justice to

representativeness, whilst reducing intentional over-reporting. The TOR for the external verification in

Sierra Leone originally mentioned a 25% sample. This however, was based on an assumption of 900

facilities in total. As the number of facilities participating in PBF was estimated at 1,200 in 2012, this

percentage was reduced to 20%.

For this assignment, a weighed stratified randomisation was used. This further enhances efficiency

(grouping of facilities in geographical clusters) and representativeness (guaranteed geographical

spread and inclusion of all levels of facilities). The facilities and Chiefdoms to be visited were sampled

during the inception workshop on the 19th and 20

th of November 2013. Sampling of months to be

verified and patients to be traced took place during the actual verification in the districts.

It should be emphasised that, while developing the sampling methodology, sustainability of the

method was an important element. The method should be simple, robust and aligned with existing

methods or administrative boundaries, to ensure that the sampling could be repeated easily in the

future, with a minimum level of financial resources.

3.5.1 FACILITY SAMPLING

Determining the number of facilities in the PBF program in 2012 Starting point for determining the sample of facilities was an overview, which was acquired from the

national PBF team of the MOHS. This list contained all payments that were done to individual PHUs in

the 13 districts in 2012. DMOs were asked to check for double entries, incorrect entries and missing



PHUs. Corrections were received during the inception workshop or closely thereafter and resulted in a

total number of 1,163 PHUs that were in the program in 2012. Distribution of these facilities over the

13 districts can be found in table 5.

Table 5 Number of PHUs in the PBF project in 2012 per district

District # PHU

Bo 110 Moyamba 99

Bombali 96 Port Loko 103

Bonthe 55 Pujehun 68

Kailahun 81 Tonkolili 96

Kambia 65 Western Area 106

Kenema 121

Koinadugu 72

Kono 91

Total 1163

Determining location of facilities and characteristics. In order to ensure geographical spread and an equal representation of different levels of PHUs in the

sample, the list with PBF facilities was linked with a database with facility details from the 2011 WHO

Service Availability and Readiness Assessment. This database contained information on the village

and chiefdom in which the facility was located and the type of facility (CHC, CHP or MCHP). For

approximately 90% of all PHUs the required details could be found. During the inception workshop,

DMOs were requested to provide details regarding the remaining 10% of the PHUs.

Sampling in urban areas

PHUs in urban areas can be accessed easier; they are less geographically spread than rural areas.

Therefore a different sampling approach was used. City councils and the entire district of Western

Area were considered urban and for these areas a simple random sample of 20% of the total number

of facilities was selected. The following additional criteria were applied:

Minimal sample in a district is four (assuming that at least 1 CHC, 1 CHP and 2 MCHP’s are

covered).

If there are less than 4 facilities in the council, then all facilities will be verified.

The sample needs to contain at least one CHC, if present. If, after sampling, no CHC was

selected, an additional CHC was randomly chosen and added to the selected facilities.

Table 6 Number of PHUs selected per Council (urban areas)

Urban - selected #PHU

Bo city council 6

Makeni city council 4

Bonthe Municipal council 2

Kenema City Council 8

Koidu new Sembehun city council 4

Freetown an WA 21

Total 45

For each council, all facilities were listed and each PHU was given a unique identifier number. During

the inception workshop a computerised random series of numbers was generated and linked to the

list, thus resulting in a 20% sample in each city council. Table 6 shows the final results per council. A



detailed list with facility names can be found in Annex 5 of Volume II. All Council samples contained a

CHC.

Sampling in rural areas

Contrary to the approach in urban areas, in rural areas stratified samples were selected. Each rural

Council was divided into 10 geographical areas with a comparable number of facilities. These

geographical areas were primarily based on the existing Chiefdoms. However, if the number of PHUs

per Chiefdom was significantly below average, contiguous Chiefdoms with low PHU numbers were

clustered. If the number of PHUs per Chiefdom was significantly above average, the Chiefdom was

split in two parts. Annex 5 in Volume II provides insight in the number of facilities per geographical

area before and after the re-grouping mentioned above.

For each rural Council all 10 geographical areas were listed and each area was given a unique

number. During the inception workshop two numbers between 1 and 10 were randomly generated in

an Excel sheet for each Council and these numbers were linked to the geographical area. This

resulted in the selection of the following Chiefdoms per council (table 7).

Table 7 Geographical areas selected in each rural Council

Council Chiefdoms

Tonkolili Yoni A, Kunike Barina

Pujehun Soro Gbema, Yakemu Kpukumu Krim (YKK)

Porto Loko Kaffu Bullom, Koya (Porto Loko) A

Moyamba Bagruwa, Timdale, Kaiyamba

Kono Gbane, Gorama Kono, Gbane Kandor, Mafindor

Koinadugu Dembelia Sinkunia , Wara Wara Yagala, Wara Wara Bofadia

Kenema Small Bo, Kandu Leppiama, Dodo, Simbaru, Malegohun

Kailahun Malema, Penguia, Yawei

Kambia Magbema B, Masungbala

Bonthe Bendu Cha, Dema, Jong A

Bombali Libeisaygahun, Sanda Tendaran, Biriwa

Bo Komboya, Niawa Lenga, Badjia, Lugbu

The Chiefdoms of Koya, Magbema and Jong were split and the letter A or B refer to the subgroup. All

facilities in the mentioned Chiefdoms were verified. A full list of facilities verified can be found in Annex

5, Volume II of this report.

Additional Weighing

As mentioned above, the sample was weighed for PHU level. During the inception workshops, the

DMOs and other stakeholders were asked if additional weighing needed to be taken into account, for

instance because of ethnicity, economic status, political orientation of different areas or populations

within the councils. None of the workshop participants deemed this necessary and therefore no

additional weighing was done.

3.5.2 SAMPLING FOR PATIENT TRACING AND PATIENT SATISFACTION SURVEY

Similar to the sampling of facilities, it is not necessary to trace all the patients that visited a health

facility to draw conclusions about the validity of the reported patient numbers and the extent to which

patients are satisfied with the services rendered. In the independent verification of the Sierra Leone

PBF program, 8 patients were randomly chosen in each facility.



Figure 5 Selection process for patient sampling

For reasons of privacy, it was decided not to include family planning services in the patient tracing.

Due to expected difficulties in finding patients (see below) and challenges in relation to the reliability of

patient-based feedback about OPD visits that took place nearly two years before the interview, OPD

visits for under-fives were also not included. Therefore, patient tracing was done for:

Antenatal care 4th visits

Deliveries

Postnatal care 3 visits

Children fully immunised

For each of these four indicators 2 patients were randomly selected.

Working procedure to trace selected patients

As described earlier, the enumerators sought collaboration with opinion leaders in the Chiefdoms to

get permission and collaboration in tracing before actually visiting the patients. The community leaders

were asked to assist in the identification of the selected patient. The health committee also assisted in

the identification of the patients and, if applicable, referred the enumerator to a community health

worker in the residential area of the selected patients.

3.6 QUALITY ASSURANCE

3.6.1 PRIOR TO DATA COLLECTION

Before qualifying for the external verification, Cordaid selected local partner organisations, with proven

competence in field research, interviewing and quality assessment. Each of these organisations

provided members for the verification teams. Members were chosen after an elaborate selection

procedure, which looked at background, general understanding of the health sector, minimal

education levels, communication skills, commitment to fieldwork and willingness to travel long

distances.

To reduce the likelihood of errors and omissions during counting of patient numbers, assessment of

cross cutting issues and interviews, much attention was given to the development of standardised tally

sheets, reporting sheets, semi-structured interview sheets, electronic data entry forms, etc. Reporting

sheet and interview sheet also contained additional instructions to the enumerators. All developed

tools are available for use in subsequent external verifications.

Select Month

•Based on Simple Random Sampling (SRS) a number between 1 and 12 was selected, representing the month in which the patient is selected.

•Numbers were computer generated (Excel) and printed and handed over to the enumerator before his/her departure to the PHU.

Select patient

•Using (SRS) a number from 1 to 25 was generated (Excel) and the patient appearing on the corresponding entry in the register was selected. Numbers were computer generated (Excel) and printed and handed over to the enumerator before his/her departure to the PHU.

If entry is inadequate

• If name and address details of the patient were inadequate to trace the patient, the next patient in the register was taken, until a valid entry was found.

• If the patient was not from within the catchment area (e.g. from a neighbouring country), the next patient was selected.



Another important part of the preparation for data collection consisted of an 6-day training for

coordinators and enumerators. This training took place from 6 to 11 January 2014 in Bo and focused

on general information with regard to the health system and data collection, data entry in EPI info, re-

counting patients from the different registers, interpretation of PHUF forms, interpretation of internal

verification forms and interview techniques. At the end of the training was a simulation exercise was

done. This allowed for field-testing of all instruments and procedures by the enumerators and

coordinators.

3.6.2 DURING THE VERIFICATION

The actual external verification exercise start immediately after finalising the training. During the first

two weeks, enumerators and coordinators operated in teams of two. This enabled them to correct

each other and ask for a second opinion in case of doubt. After two weeks, enumerators and

coordinators visited PHU and patients individually. However, several procedures were developed to

allow the enumerators to consult their colleagues or supervisors. First of all, the verification team

coordinators could be contacted by phone for consultation. In addition, it was agreed that the

enumerators took pictures of registers and forms if they doubted the correctness of entries. These

pictures were then discussed with the coordinators, and, if necessary, forwarded for a second opinion

to the supervisors.

During the fieldwork the coordinators crosschecked enumerators, when receiving their reports. This

included an assessment of the completeness of forms and validation when entering data in computer.

In case of large deviations the coordinators contacted the enumerator and discussed whether re-

verification was necessary.

The national supervisors joined each verification team during a period of approximately one week.

This allowed them to apply corrective measures and provide additional instructions to the coordinators

and enumerators. In addition they validated data entry by analysing large deviations and discussing

inconsistencies with the team coordinators.

3.6.3 AFTER DATA COLLECTION

The final data that were entered in EPI-info were exported to Excel and STATA for further analysis.

However, before starting the analysis, the database was validated in two ways:

1. Outliers and irrational entries, e.g. blank entries for external verification or zero-scores for all

indicators in HMIS, were identified. For these cases, the original verification sheets were

examined and, if necessary, corrections were made. Eight cases were found and corrected.

2. A sample of five PHUs per district was taken (representing 25% of all PHUs) and all entries on

the original verification sheet of those PHUs were crosschecked with the data appearing in the

database. Less than 0.2% errors were found. If applicable, data were corrected.

3.7 RELIABILITY AND SIGNIFICANCE

3.7.1 RELIABILITY OF DATA

As described in the previous paragraph, all data were validated before analysis took place. During

validation, errors were found in approximately 0.2% of all data entries. In view of this low percentage

and the fact that found errors were corrected, the internal reliability of the database for the external

verification can be considered high.

3.7.2 EQUAL DISTRIBUTION OF FACILITIES

Figure 6 shows the final number of PHU’s selected for verification.



Figure 6 Number of PHU selected per district

The percentage of the total number of PHU’s in each district varies from 17% to 23%, with a national

average of 20,2%. The selected facilities may thus be considered equally distributed over the districts.

The final number of facilities per district that was externally verified deviated from the number that was

initially selected. This was the case in the following districts:

In Western Area one facility (Aberdeen Women) was part of the sample, but did not exist in 2012

and was therefore not included in the external verification;

In Bombali, both Fulla Town MCHP and Kayongoro MCHP had closed and could not be verified;

In Kono three facilities, which were initially not part of the sample, were found to be in the selected

Chiefdoms and part of the PBF programme in 2012 (Boroma MCHP, Kayongoro MCHP and Kania

MCHP). These were included.

Because three facilities were left out of the external verification, and three new ones were added, the

total number of facilities verified remained 235.

Table 8 shows that in each district, all types of facilities were included in the external verification. For

6 out of the 235 facilities no formal PHU-classification could be provided by the DHMT.

Table 8 Distribution of PHU type per district

Row Labels CHC CHP MCHP

Bo 6 2 17

Bombali 5 3 12

Bonthe 1 7 5

Kailahun 3 10 1

Kambia 1 1 8

Kenema 8 7 15

Koinadugu 2 3 7

Kono 3 1 16

Moyamba 3 4 12

Port Loko 3 4 10

Pujehun 4 2 6

Tonkolili 2 2 12

Western Rural 3 3 4

Western Urban 3 3 5

Grand Total 47 52 130



3.7.3 SIGNIFICANCE OF PHU DATA

The sampling exercise that took place during the inception workshop resulted in 235 PHUs selected

out of the total of 1.163 that received a payment as part of the PBF programme in 2012. The question

is how significant conclusions are, that can be drawn on the basis of the findings in the sample. For

this purpose, a sample size calculation can be used (figure 7). Such a calculation is generally applied

in statistical research16

.

Figure 7 Formula for calculating sample size

( ) ( )

The formula indicates that, with a confidence level of 95% and a sample of 235 PHUs, the outcomes

for the entire population (1,163 facilities) can be predicted with a 6% precision. In other words: If 50%

of the facilities in the sample reported stock-outs of registers, there is a 95% chance that the score will

be between 44% and 56% for all PHUs in the PBF programme. Therefore, conclusions from the

selected sample can be considered powerful. The conclusions of the external verification may be

considered valid for all PHUs in the PBF programme.

3.7.4 SIGNIFICANCE OF PATIENT SATISFACTION DATA

Similar to the calculation for statistical significance that was made for the facility sample (see figure 7),

a calculation can be made for the sample of patients. In total, 1,234 patients were interviewed. With

this number and a confidence level of 95%, the outcomes for the entire population (estimated at 6

million for 2011 according to the United Nations) can be predicted with a 2.5% precision. In other

words, it can be assumed that if the survey shows that 30% of the patients had to pay for services,

there is a 95% chance that between 27.5% and 32.5% of the entire population of Sierra Leone had to

pay for services. Therefore, conclusions from patient tracing and the satisfaction survey can be

considered powerful. The conclusions of the external verification may be considered valid for the

population of Sierra Leone.

3.7.5 COMPLETENESS OF DATA

In order to analyse the consistency between external verification, internal verification, HMIS and the

PHU F-reports, all information from mentioned sources has to be available and accessible. However,

the external verification encountered problems with completeness of information. Figure 8 provides an

example on the availability of data for family planning, one of the indicators in the PBF programme.

16

Naing L, Winn T and Rusli BN. Sample size calculator for prevalence studies, Version 1.0.01. Available at: http://www.kck.usm.my/ppsg/stats_resources.htm Daniel, WW (1999). Biostatistics: A Foundation for Analysis in the Health Sciences. Wiley & Sons, New York

http://www.kck.usm.my/ppsg/stats_resources.htm



Figure 8 Percentage completeness of family planning records per districts

As figure 8 shows, unavailability of records was an issue for all of the four data sources. However, the

availability of HMIS data (from automated systems at DHMT level) was found to be the most

challenging. In Freetown and Western Area only 39% and 32% of all HMIS data on Family Planning

was available. In Port Loko no HMIS records over 2012 were found. Similar patterns were found for all

other indicators. Table 9 summarises the reasons why records were not found.

Figure 9 Reasons for unavailability of records per information source

Source Reasons for unavailability

HMIS No data could be retrieved from servers at DHMT level and no back-up

data could be found at national level

Internal Verification No data (soft or hard copy) could be retrieved at DHMT level and no back-

up data could be found at national level

External Verification No registers or incomplete registers could be found at the PHU

PHU F-report A copy of the PHU F-forms could neither be found at facility level, nor at

the district level

If no records were found for an indicator in a specific source of information, these cases were not

included in further analysis. For that purpose a data cleaning exercise was carried out. In Chapter 4

the impact of missing data is calculated.

3.8 DATA ANALYSIS

3.8.1 SYSTEMS USED

All data in gathered during the external verification was entered into a relational EPI Info Database17

,

which was specifically developed for the purpose of this assignment. Apart from data entry, EPI Info

17

Epi Info is used in the MOPH, and various officers within the DPPI are conversant with this software programme. They will be able to continue to apply the software developed for this external verification in the future.

0

10

20

30

40

50

60

70

80

90

100

HMIS IV EV PHUF



was also used for spatial analysis with GIS Maps. For more sophisticated analysis, a CSV file was

created, that was exported to Microsoft Excel and STATA.

3.8.2 PRIMARY FACILITY OUTPUT DATA

Comparison of patient numbers for the six output indicators in the programme is the most crucial

element in this assignment. The computation of differences or deviations across different data sources

was an initial step in the analysis of reporting accuracies. These statistics were stratified according to

key study factors such as District, Council and Facility Type. A second step was the calculation of

deviations beyond 25% of reported values at different levels.

3.8.3 DATA FROM PATIENT AND KEY INFORMANT INTERVIEWS

Simple descriptive statistics were obtained from the analysis of the patient survey data and structured

interview data from key informants. In particular:

Percentage / number of patients positively identified and confirmed using the facility

Percentage / number of patients who paid for services at the facility during the visit

Average satisfaction scores

Percentage of patients satisfied with the services received during the visit

Number of joint internal verification sessions performed

Trainings received

PBF payments received

These descriptive statistics were stratified according to study factors such as facility type, service

received, region, district, age and sex of respondents. Furthermore, regression analysis was used to

check for the association between the above study factors (including whether patients paid for

services) and patient satisfaction scores. This was also done across different data sources, eg across

patient satisfaction data, cross cutting issues and key informant interviews.

3.9 EXTERNAL VERIFICATION IN THE HOSPITALS

Apart from the external verification at PHU-level, the external verification team also performed an

external verification in the two tertiary hospitals in the PBF programme. In order to assess the impact

of the PBF programme, the external verification was also done in two non-PBF hospitals. Three

international consultants performed the external verification in hospitals.

The hospitals that were visited include:

Macauley Hospital (non-PBF)

Rupoka Hospital (non-PBF)

Princess Christian Maternal Hospital (PBF)

Ola During Children’s Hospital (PBF)

Information for the external verification in hospitals was obtained in two ways. First, using exactly the

same instrument that was used during internal verification, a re- assessment was done. Secondly, key

staff, members of the management team and the in charge of the hospitals were interviewed. For the

interviews a structured questionnaire was created.

Some of the elements of the hospital checklist for PBF, like cleanliness, could not be re-assessed

retrospectively. Instead, the consultants looked at the present situation. For objectively verifiable

indicators, like the availability of patient records, the last quarter of 2013 was assessed. Data were

compared with previous assessments.

Because no thorough baseline was done, it is difficult to draw conclusions about the actual impact of

the PBF program at facility level. However, the in-depth interviews with the management teams in the

four hospitals provide some qualitative information regarding the impact.



4 EXTERNAL VERIFICATION FINDINGS

4.1 INTRODUCTION

The External Verification (EV) consisted of two components of checking the Internal Verification (IV):

Recalculating the output indicators based in recordings in the registers in health facilities;

Repeating the verification of crosscutting (quality) indicators in PHUs and repeating the verification

of hospital (quality) indicators

In addition the EV performed:

Patient tracing and satisfaction survey

Hospital verification of two non-PBF hospitals

The methodology is described in Chapter 3 of this report. This Chapter describes the findings, while in

chapter 7 these findings are put in a bigger context.

4.1.1 INDICATORS

Output indicators

The PBF project in Sierra Leone focuses on improving a number of indicators from the Basic Package

of Essential Health Services, which are part of the Free Healthcare Initiative. It pays a fee-for-service

for: 1. Women of reproductive age using modern family planning (BPEHS 7.2) 2. Pregnant women receiving four antenatal consultations (ANC-IV) (BPEHS 7.1.1) 3. Deliveries conducted under safe conditions (BPEHS 7.1.2) 4. Women receiving three postnatal consultations (PNC-III) (BPEHS 7.1.4) 5. Children under one year of age receiving full and timely course of immunizations (BPEHS 7.6) 6. Outpatient visits with curative services for children under five years old according to Integrated

Management of New-born and Childhood Illness (IMNCI) protocol (BPEHS 7.7)

Crosscutting Issues

In order to stimulate overall performance of the health facilities the PBF project provides additional

incentives for so-called cross cutting issues, which address quality. The following indicators provide a

multiplier for the total payment based on fee-for-service: 1. Recording of staff attendance. 2. Timely submission of DHIS, attendance and PBF reports. 3. A functioning Health Management Committee. 4. Display of up-to-date performance information at the facility. 5. All paperwork kept in good order at the facility. 6. Maintenance of appropriate standards of cleanliness. 7. Appropriate procedures for medical waste management in place and being observed. 8. Maintenance of up-to-date and accurate drugs records. 9. No stock-out of essential drugs for the three childhood diseases with highest mortality

4.1.2 HOSPITAL PBF

In 2012 a pilot was started with hospital PBF in two hospitals in Freetown, closely involved in maternal

and child health, i.e. the Ola During Children’s Hospital and the Princess Christian Maternity Hospital.

The primary focus of hospital PBF is on performance indicators with regard to quality of services,

measured by means of a composite performance score in eight domains as shown in the table below.

Each domain has 3 – 7 indicators, which are either objectively verifiable or more qualitative in nature.

For the performance assessment an extensive checklist is used.



Table 9 Domains for assessment of hospital performance

4.2 OUTPUT INDICATORS IN PHUS

4.2.1 VARIATION BETWEEN SOURCES OF INFORMATION

In principle the four sources of information traced in this external verification (F-report, HMIS, IV and

EV) should show no differences, as they all go back to the same basic source, i.e. the registers in the

health facilities. F-reports, IV and EV directly used the registers as source; HMIS is based on the F-

reports.

The EV found the following figures for the six output indicators in the PHUs (see table 10 below) in the

F-reports, the HMIS, the IV and the EV. The totals are for four sampled months in 2012 and are not

representative for the annual figures, as different months were sampled in the various districts.

Table 10 Totals per output indicator sampling 4 months 2012

Indicator F-reports HMIS IV EV

Family planning 10,817 11,328 17,497 10,105

Antenatal Care IV 12,689 10,477 12,672 11,361

Deliveries in PHUs 9,554 8,342 9,521 10,060

Post Natal Care III 10,613 8,282 9,644 7,063

Fully vaccinated before 1 year 10,434 10,829 13,801 9,511

Outpatient cases children under five 115,760 143,335 136,997 115,893

Figure 10 below shows the comparison between sources of information for indicators. The EV figures

for all indicators are put at 100%. It transpires that (with exception of deliveries) IV figures are 12% -

73% higher than the EV, while the F-reports show only one indicator (PNC) as outlier and others

closer to the EV. Also HMIS is closer with a variation between 83% and 124% the EV figures.

Financial management

Patient care

Human resources management

General organization

Pharmacy management and prevention of drugs stock out

Hygiene and sanitation

Health care services

Laboratory



Figure 10 Comparison Output indicators per source of information

Figure 11 below shows the same information, but now per indicator. This figure shows that the Family

Planning attendance recorded by IV and the Fully Vaccinated recorded by IV are (statistically

significant) outliers. ANC, deliveries and OPD indicators show less prominent differences among

sources of information. PNC-III shows differences of 17% - 50% in sources of information compared to

EV.

Figure 11 Comparison sources of information per indicator

It is relevant to disaggregate the differences between sources of indicators per district, to identify

whether some districts have consistently deviating figures, which could influence the national totals.

For example, in Bonthe, Kambia, Moyamba and Western Area the reported numbers for family

planning in the IV were considerably higher than the EV, but not in other Councils. In Freetown, Bo

Tonkalili and Koinadugu the reported numbers for PNC-III in the IV were much higher than the EV, but

less in other districts. The analysis does not prove that certain districts always score much higher for

all indicators in the IV compared with the EV. Or in other words: differences are not systematic and

can be found anywhere. The Council reports in the annex 4, Volume II of this report, provide detailed

information on deviations per indicator.



4.2.2 INFLUENCE OF MISSING DATA ON CALCULATIONS

As has been discussed in the chapter on methodology, missing data were a serious problem in some

areas. The verification team tried to assess whether missing data could be a cause for differences

between sources of information. Based on the missing data analysis (see Chapter 3) service utilisation

figures were extrapolated.

Figure 12 below shows that correction for missing data increases the differences in FP, while reducing

the differences in ANC (figure 13).

Table 11 below shows that for ANC, deliveries and PNC difference get smaller after correction for

missing data, but bigger for FP, OPD and EPI. The verification team therefore concluded that missing

data could be ruled out as general cause for differences between sources of information.

Figure 12 Extrapolation of service utilisation FP based on missing data

Figure 13 Extrapolation of ANC service utilisation figures based on missing data



Table 11 Extrapolation of all service attendance based on missing data

F -report HMIS IV EV

%

missing

Total

plus

missing

%

missing

Total

plus

missing

%

missing

Total

plus

missing

%

missing

Total

plus

missing

Family

Planning 4% 11,211 15% 13,394 9% 19,259 4% 10,507

ANC IV 3% 13,062 15% 12,310 9% 13,948 8% 12,303

Delivery 5% 10,079 17% 10,002 10% 10,585 6% 10,655

PNC III 5% 11,122 19% 10,244 11% 10,889 24% 9,273

Fully

Vaccinated 4% 10,862 15% 12,756 9% 15,226 8% 10,288

OPD under

5 5% 121,716 17% 171,856 8% 149,134 4% 121,044

4.2.3 DIFFERENCES IN DATA IN FACILITIES

In the previous section aggregated numbers have been discussed. Such figures are influenced by the

statistical phenomenon of regression towards the mean. Differences between one source of

information and another can be positive or negative, i.e. showing more or less client contacts. Adding

all numbers blurs the view on differences at the source. The verification team looked at the frequency

and size of deviations of records to assess accuracy of recording at the source.

Figure 14 below shows the distribution of facilities with deviating figures between sources of

information for family planning (14a) and OPD under 5 (14b). The green area shows differences of

10% and less; the yellow area differences between 10% and 25% and the red area shows differences

of over 25%. In principle, all bars in the figure should be green; the redder the bar, the more serious

disagreement between data sources at facility level.

Figure 14 Distribution of facilities by differences

Figure 14a Family Planning differences in data

14 23

11

37 33 17

24

17

17 24

68 53 71 46 43

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

HMIS and IV HMIS and

EV

IV and EV HMIS and F-

Report

EV and F-

Report

Distribution of Facilities By Differences - Family Planning

Above 25%

10%-25%

Below 10%



Figure 14b OPD under 5 differences in data

Table 12 below shows the differences over 25% (the red segments in the graphs above) for all

indicators.

Table 12 Differences between data sources above 25% in PHUs

HMIS and IV HMIS and EV IV and EV HMIS and F rep F-Report and EV

Family

Planning 68% 53% 71% 46 % 43%

ANC IV 60% 57% 61% 28% 51%

Delivery 42% 36% 33% 14% 27%

PNC III 65% 69% 67% 40% 61%

Fully

Vaccinat

ed

51% 62% 63% 71% 63%

OPD

under 5 60% 69% 49% 46% 59%

Average 58%

58% 57% 33% 51%

Internationally in external verifications plus or minus 10% difference between sources of information is

considered as big, given the fact that they all use the very same root source, i.e. the patient registers

in the facility. Differences of more than 25% are hardly seen. In Sierra Leone there are serious issues

of data consistency, as seen in the comparison between all sources of data: HMIS, F-rep, IV and EV.

On average over half of entries differ more than 25% between sources of information, with exception

of HMIS and F-reports (which should be simply retyping data from a paper sheet into a computerised

data system).

4.2.4 DIFFERENCES PER LEVEL OF FACILITY

In table 13 below the differences between data sources are shown for family planning, split out per

facility type. The association between facility type and the differences between HMIS and Internal

Verification is significant. CHCs more often have differences below 10% and MCHPs more often have

differences above 25%. The higher the level of the health facility the more concordance between

figures from the four data sources.

21 18

33

46

24

17 13

17 8

18

62 69 49 46 58

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

HMIS and IV HMIS and

EV

IV and EV HMIS and F-

Report

EV and F-

Report

Distribution of Facilities By Differences - OPD U5

Above 25%

10%-25%

Below 10%



This can to some extent be attributed to smaller numbers of clients in peripheral health facilities. For

example, if in one month there are 2 deliveries recorded on the F-form of an MCHP and 3 are reported

in the HMIS, the deviation is 50%, while the absolute number difference is only 1. However, in general

the conclusion has to be that the quality of data aggregation and reporting is not adequate in MCHPs

and only slightly better in higher-level facilities.

Table 13 Differences between data sources FP per level PHU

4.2.5 DIFFERENCES PER GEOGRAPHICAL AREA

Data analysis shows no major variation in geographical as regards facilities with over 25% deviation

between data sources. Figure 15 below shows differences between HMIS and external verification,

with no significant differences between areas. Data quality is an issue all over the country.

Figure 15 Geographical spread of facilities with differences data sources

Below 10% 10% to 25% Above 25% Total Below 10% 10% to 25% Above 25% Total

14 5 27 46 19 7 20 46

30.4 10.9 58.7 100.0 41.3 15.2 43.5 100.0

35.0 16.7 17.0 20.1 41.3 18.4 13.8 20.1

10 9 33 52 8 11 33 52

19.2 17.3 63.5 100.0 15.4 21.2 63.5 100.0

25.0 30.0 20.8 22.7 17.4 29.0 22.8 22.7

16 16 99 131 19 20 92 131

12.2 12.2 75.6 100.0 14.5 15.3 70.2 100.0

40.0 53.3 62.3 57.2 41.3 52.6 63.5 57.2

40 30 159 229 46 38 145 229

17.5 13.1 69.4 100.0 20.1 16.6 63.3 100.0

100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0

Total

HMI S and External Verification HMI S and I nternal Verification

CHC

CHP

MCHP



4.2.6 SATISFACTORY AND UNSATISFACTORY ENTRIES

General

In the internal and external verification a distinction is made between satisfactory and unsatisfactory

records in the PHU.18

In the PBF Operational Manual no exact criteria were given of these

qualifications, but during the inception workshop of the EV they were formulated with inputs from the

DMOs. For example, a delivery record is unsatisfactory when no partograph can be shown, or a fully

vaccinated record is unsatisfactory when vaccinations are marked with tick, instead of dates. The

external verifiers used these standard criteria, while internal verifiers used local criteria.

Satisfactory records

On average between 83% and 93% of the records in the IV are satisfactory, and between 92% and

96% of the EV are meeting the defined criteria for proper recording. Disaggregating scores per

council, as reflected in table 14, shows that incidentally there are variations. For example, Bo DHMT in

the internal verification consistently marked all entries as satisfactory, while Bonthe DHMT was very

critical. The EV showed a consistent picture for all districts. DHMTs often apply local criteria to

determine satisfactory or unsatisfactory entries.

Table 14 Percentage satisfactory entries per Council per indicator

19

Council

Fam

ily

Pla

nn

ing

AN

C

OP

D

Deliv

eri

es

PN

C

EP

I

Inte

rna

l

Exte

rna

l

Inte

rna

l

Exte

rna

l

Inte

rna

l

Exte

rna

l

Inte

rna

l

Exte

rna

l

Inte

rna

l

Exte

rna

l

Inte

rna

l

Exte

rna

l

Bo City Council 100 98 100 96 100 95 100 95 100 75 100 87

Bo District Coun 100 97 100 95 100 77 99 84 100 83 100 75

Bombali District . 100 . 99 . 97 . 100 100 88 . 94

Bonthe District 59 100 58 88 51 76 6 60 67 85 61 90

Bonthe

municipal

51 75 56 85 57 100 . 0 67 . 75 .

Freetown City

Co

99 94 98 95 100 84 100 89 100 100 96 96

Kailahun Distric 96 91 93 87 92 89 84 94 95 99 91 94

Kambia District 89 93 96 99 93 91 89 97 88 100 94 91

Kenema City

Coun

95 98 96 95 97 97 93 99 98 96 96 95

Kenema District 99 95 97 96 100 96 94 94 98 96 98 98

Koidu new

Sembeh

83 93 90 93 89 98 74 94 83 86 92 97

Koinadugu Distri . 100 . 94 . 86 . 98 . 100 . 98

Kono District Co 89 84 81 78 89 93 82 82 88 87 90 89

Makeni City

Coun

. 100 . 100 . 100 . 100 . 99 . 100

Moyamba

District

90 100 94 97 91 94 36 91 80 87 87 95

Port Loko Distri 100 99 98 100 99 88 88 98 91 100 99 94

Pujehun District 92 99 84 72 98 81 76 71 94 94 96 74

18

Satisfactory refers to completeness and correctness of data (e.g. adding address, age, weight, BP). Eligibility refers to the case definition, e.g. vaccination before

reaching age of one year. 19

Some fields are left blank, where data were missing



Council

Fam

ily

Pla

nn

ing

AN

C

OP

D

Deliv

eri

es

PN

C

EP

I

Inte

rna

l

Exte

rna

l

Inte

rna

l

Exte

rna

l

Inte

rna

l

Exte

rna

l

Inte

rna

l

Exte

rna

l

Inte

rna

l

Exte

rna

l

Inte

rna

l

Exte

rna

l

Tonkolili Distri 94 99 98 98 98 99 89 95 98 95 94 99

Western Area

Rur

99 88 99 88 100 90 99 95 100 74 100 93

Overall 93 96 92 93 93 90 83 90 92 90 93 92

Figure 16 below shows that overall there is no major difference between percentages of satisfactory

records in IV and EV. The main differences range between 2 and 3.5.

Figure 16 Distribution of Absolute Differences between IV and EV

The average differences between the internal verification and the external verification for the total

group of indicators ranged from -6% to 3.3% for the indicators with a negative difference indicating the

percentage in internal verification was lower than that in external verification. (See table 15 below.)

However, Bonthe District and Municipal Councils, Koidu New Sembehun and Western Area Rural

districts show consistently high differences above 10% across the six indicators. Other councils such

as Freetown City Council and Pujehun City Council fluctuate with some indicators having very minimal

differences and others having high values. The indicators that had relatively higher observed

differences were Deliveries and OPD with averages of 3.3% and 6.0% respectively.



Table 15 Differences between average satisfactory scores (IV and EV) per district

District Family

Planning

ANC OPD Deliveries PNC EPI

Bo 2.5 4.8 18.5 12.5 18.7 22.3

Kambia -2.4 -4.3 0.4 -9.2 -25.0 -1.6

Bonthe -40.1 -31.3 -37.1 -25.0 -17.2 -30.5

Kailahun 6.1 6.0 4.6 -10.5 -0.4 -3.1

Kenema 0.7 0.3 2.7 -1.2 2.3 -0.3

Kono -0.5 -2.3 -4.1 -3.0 2.9 -0.2

Moyamba -9.9 -2.4 -3.2 -55.7 -6.5 -7.9

Port Loko 0.6 -2.7 10.4 -10.8 -12.5 2.7

Pujehun -6.7 14.3 16.9 6.7 3.3 22.1

Tonkolili -4.7 0.1 -2.0 -10.6 5.3 -5.1

Western Rural 51.3 25.4 0.0 18.8 21.5 11.8

Western Urban 7.2 -1.1 23.2 11.7 - 4.5

Overall -2.7 0 3.3 -6 2.2 2.5

Overall, the differences are distributed around the averages between -6 and 3, although a number of

outlier facilities have significantly higher differences as shown by the box plot below, in figure 17.

Figure 17 Plot box differences satisfactory entries

Differences satisfactory entries per level of facility Table 16 below shows the percentages of satisfactory scores by level of health facility. From this analysis no conclusion can be drawn that recording at higher level is consistently better than at lower level. Nearly all levels are around 90% satisfactory entries.



Table 16 Percentage satisfactory records per level health facility

CHC CHP MCHP

IV EV IV EV IV EV

FP 94% 95% 93% 94% 93% 97%

ANC 93% 90% 90% 92% 93% 94%

OPD 95% 88% 91% 90% 94% 91%

PNC 92% 90% 88% 90% 92% 90%

Deliveries 89% 93% 79% 86% 83% 90%

EPI 96% 91% 92% 95% 92% 90%

THE EXTERNAL VERIFICATION OF OUTPUT INDICATORS IN THE PBF SYSTEM SHOWED: CONSIDERABLE, SOMETIME SIGNIFICANT DIFFERENCES EXIST BETWEEN

AGGREGATED NUMBERS IN INTERNAL AND IN EXTERNAL VERIFICATION. WITH

EXCEPTION OF DELIVERIES, THE AGGREGATED IV FIGURES ARE 12% - 73% HIGHER

THAN THE EV.

RECORDED ATTENDANCE IN THE IV IS IN THE MAJORITY OF INDICTORS ALSO HIGHER

THAN OTHER SOURCES OF INFORMATION (HMIS, F-FORM).

IN GENERAL, THE AGGREGATED FIGURES FROM VARIOUS SOURCES OF

INFORMATION DIFFER, WHEREBY THE EV SHOWED MOST CONCORDANCE WITH F-

REPORTS.

THE DIFFERENCES CANNOT BE ATTRIBUTED TO MISSING DATA.

AT FACILITY LEVEL FOR ALL INDICATORS THE DIFFERENCES BETWEEN SOURCES OF

INFORMATION ARE LARGE, OFTEN MORE THAN 25% HIGHER OR LOWER.

THE DIFFERENCES IN RECORDING ARE SPREAD OVER THE COUNTRY AND NOT

RELATED TO SPECIFIC DISTRICTS.

LOWER-LEVEL HEALTH FACILITIES SHOW LARGER ERROR MARGINS THAN HIGHER-

LEVEL FACILITIES.

THERE IS NO STATISTICALLY SIGNIFICANT DIFFERENCE BETWEEN IV AND EV AS

REGARDS PERCENTAGE OF SATISFACTORY OR UNSATISFACTORY RECORDS.

THERE IS NO SIGNIFICANT DIFFERENCE IN SATISFACTORY ENTRIES PER LEVEL OF

FACILITY, AND NOT PER DISTRICT. IT VARIES BETWEEN 90% AND 95%

SATISFACTORY ENTRIES, WITH EXCEPTION OF DELIVERIES, WHICH SHOW LOWER

PERCENTAGE OF SATISFIED RECORDS IN IV (83%).

4.3 CROSSCUTTING ISSUES

4.3.1 GENERAL

The external verification performed an assessment of crosscutting issues related to quality of care.

There are nine indicators, as explained in paragraph 4.1, scoring each either –3 or +3 respectively -4

or +4. In the inception phase the EV team together with representatives of the DHMTs formulated

standards for the indicators, to ensure that all enumerators would use similar assessment criteria.

Such standard criteria were not applied during the internal verification.

Part of the indicators was checked going back to 2012 data, e.g. attendance registers, report

submission, minutes of health management committees and administration. Part of the indicators was

checked during the EV, like cleanliness and waste management. (See explanation in Chapter 3.) The

average scores on crosscutting indicators were significantly higher during the internal verification (last

quarter of 2012) than during the external verification, as shown in figure 18 below.



Figure 18 Crosscutting issues IV and EV

Figure 19 Distribution crosscutting issues scores in EV

Statistical analysis shows a normal distribution in the external verification (in figure 19 above), which is

missing in the internal verification (figure 20 below). In the internal verification in some districts many

facilities received a maximum score, with some outliers.

Figure 20 Distribution of crosscutting scores in IV



All facilities in Kono district received the maximum score for all crosscutting indicators in the internal

verification of the last quarter of 2012, but not in the external verification, as shown in figure 21 below.

Figure 21 Kono District Comparing Crosscutting Indicators IV and EV

4.3.2 SPECIFIC INDICATORS

Scores on indicators, which used similar reference documents in IV and EV (e.g. attendance registers,

or timely submission HMIS reports), were mostly lower in EV. Especially the indicator on

administration scored very low (with exception of Port Loko), as in many health facilities

documentation and registers over 2012 went missing (figure 22).

Figure 22 Comparing crosscutting indicator Administration IV and EV

None of the other indicators assessed with present information in 2014 in the EV scored higher than

the IV in 2012 (e.g. cleanliness, drug records or waste management). Maybe the standardised

assessment in the EV applied stricter criteria than the less-structured assessment in the IV.



The scores for stock outs of medicines were considerably lower in EV compared with IV. This may

reflect a worsening supply of essential medicines in 2014 in the country compared with 2012.

Figure 23 Comparison crosscutting indicator Stock Outs IV and EV

THE EXTERNAL VERIFICATION FOR CROSSCUTTING INDICATORS SHOWED: THE SCORES FOR THE CROSSCUTTING ISSUES IN THE EXTERNAL VERIFICATION

WERE CONSISTENTLY LOWER THAN IN THE INTERNAL VERIFICATION IN NEARLY ALL

DISTRICTS, FOR NEARLY ALL THE INDICATORS.

IN THE EXTERNAL VERIFICATION STANDARDISED ASSESSMENT CRITERIA WERE

APPLIED, REDUCING THE CHANCES OF PERSONAL BIAS. THOSE CRITERIA MIGHT

HAVE BEEN STRICTER THAN APPLIED IN THE INTERNAL VERIFICATION.

DUE TO THE TIME LAPSE BETWEEN 2012 AND 2014 DIFFERENCES MAY HAVE BEEN

CREATED, E.G. MISSING REGISTERS, LEADING TO LOWER SCORES.

WORSENING SUPPLIES OF MEDICINES MAY HAVE CAUSED LOWER SCORES IN

AVAILABILITY OF ESSENTIAL MEDICINES IN 2014 COMPARED WITH 2012.

IN THE INTERNAL VERIFICATION SOME DHMTS GAVE PHUS MAXIMUM SCORES FOR

ALL INDICATORS. THIS NEVER OCCURRED IN THE EXTERNAL VERIFICATION.

4.4 HOSPITAL EXTERNAL VERIFICATION

The external verification team performed an external verification of PBF in the two hospitals in

Freetown, which are included in the pilot on hospital-based PBF and two hospitals not in the PBF. In

paragraph 4.1 the domains of inspection are explained. Per domain a series of 8 – 15 questions is be

answered, which each give a minus or plus score. The total maximum score is 1,000 points.

4.4.1 OLA DURING CHILDREN HOSPITAL

Figure 24 below shows the scores for Ola During Children Hospital (ODCH) comparing the latest IV in

2013 and the EV. The EV scored slightly higher in most domains and much higher in the general

domain. During the latest IV the administration was found not to be in order and scored very low.



Figure 24 Ola During Children Hospital IV and EV

Figure 25 Trend in IV scores in Ola During Children Hospital

Figure 25 shows the trend of scores. Quarter 2 and quarter 3 in 2013 were done in one assessment.

The ODCH hospital always scored 85% - 95% of the maximum, except in latest IV. In the EV ODCH

scored 79%.

4.4.2 PRINCESS CHRISTIAN MATERNITY HOSPITAL

Figure 26 below shows the comparison between the latest IV in 2013 and the EV for Princess

Christian Maternity Hospital (PCMH). Here scores in most domains were slightly lower in the EV

compared to the IV.



Figure 26 Princess Christian Maternity Hospital IV and EV

The figure 27 below shows the trend analysis for PCMH. The hospital scored in the past between 82%

and 90% of the maximum scores. In the EV the score was 84%.

Figure 27 Trend analysis Princess Cristian Maternity Hospital

4.4.3 NON-PBF HOSPITAL

The MOHS selected two secondary level hospitals in Freetown not in the PBF programme, to perform

a similar external verification. These hospitals provide general services at the primary referral level,

unlike the two PBF hospitals, which are specialised tertiary hospitals. Figure 28 below shows the

comparison of scores.



Figure 28 Comparison PBF and non-PBF hospitals in EV

One non-PBF hospital (Macauley) scored only 4% of the maximum scores and the other (Rokupa)

59%. In some areas Rokupa hospital scored even higher than one of the PBF hospitals. According to

information Rokupa has more income from patients, more support from the Council and other support

to make ends meet.

THE EXTERNAL VERIFICATION OF THE HOSPITAL PBF FOUND THAT: THE EV TEAM GAVE SLIGHTLY HIGHER SCORES TO ODCH COMPARED TO THE

LATEST IV (79% VS. 61%), BUT LOWER THAN IN OTHER IVS (85%-95%). THIS WAS

DUE TO THE INDICATOR ADMINISTRATION

THE EV TEAM GAVE SLIGHTLY LOWER SCORES TO PCMH COMPARED TO THE

LATEST IV (84% VS. 89%), BUT WITHIN THE RANGE OF OTHER IVS (82%-89%)

NON-PBF HOSPITALS SCORED LOWER THAN PBF HOSPITALS, BUT ONE OF

THOSE SCORED ONLY SLIGHTLY LOWER, WHILE THE SCORE OF THE OTHER

HOSPITAL WAS WIDE OFF RANGE.



5 PATIENT TRACING AND SATISFACTION SURVEY

5.1 PATIENT TRACING

The patient tracing and satisfaction survey served two purposes:

1. Confirming existence of patient or clients, to check on reliability of recording at facility level;

2. Getting feedback from patients with regard to their experiences in the health facilities.

The methodology for tracing patients is described in Chapter 3 of this report. Per PHU eight

patients/clients were randomly selected from registers to be traced and interviewed.

In total in all health facilities 1,680 persons were identified for tracing. Out of this total, community

leaders, neighbours or family members could not identify 124 persons as existing persons, which is

7.4% of the total persons identified for tracing. Table 17 below shows no clear pattern of non-

identifiable persons. Koinadugu District Council and Bo City Councils are outliers in this statistic.

Table 17 Percentage of persons who could not be traced in EV

Persons identified

for tracing

Number not

identified

Percentage not

identified

Bo City Council 48 13 26%

Bo District Council 134 18 14%

Bombali District Council 152 0 0%

Bonthe District Council 83 2 3%

Bonthe municipal Council 11 0 0%

Freetown City Council 58 10 17%

Kailahun District Council 96 9 9%

Kambia District Council 82 2 2%

Kenema City Council 56 1 2%

Kenema District Council 164 1 1%

Koidu new Sembehun City

Council

26 2 8%

Koinadugu District Council 110 27 25%

Kono District Council 110 2 2%

Makeni City Council 22 0 0%

Moyamba District Council 150 19 12%

Port Loko District Council 106 2 2%

Pujehun District Council 79 6 8%

Tonkolili District Council 122 4 4%

Western Area Rural District

Council

71 6 9%

Total 1,680 124 7.4%

Of the 455 persons who were not interviewed in the EV patient tracing survey, 4% had died, 64% had

moved, and 32% were unknown in the community, as shown in table 18 below. The long time between

the service delivery (13 months to two years) may have contributed to a larger number of patients who

could not be interviewed. The EV team concludes that it highly unlikely that PHUs recorded “ghost

patients” to inflate the numbers of attendance and therefore the payments through PBF.



Table 18 Reasons for not interviewing persons in EV

Districts Died Exists But

Relocated

Untraceable, No

Information

Bo 8.33% 60.42% 31.25%

Bombali 0.00% 100.00% 0.00%

Bonthe 11.11% 66.67% 22.22%

Kailahun 0.00% 61.11% 38.89%

Kambia 0.00% 83.33% 16.67%

Kenema 3.33% 90.00% 6.67%

Koinadugu 0.00% 12.00% 88.00%

Kono 11.11% 77.78% 11.11%

Moyamba 0.00% 73.85% 26.15%

Port Loko 0.00% 69.70% 30.30%

Pujehun 6.25% 75.00% 18.75%

Tonkolili 5.88% 70.59% 23.53%

Western Rural 4.00% 40.00% 56.00%

Western Urban 0.00% 36.67% 63.33%

Grand Total 4.18% 63.57% 32.25%

5.2 PATIENT SATISFACTION

5.2.1 SATISFACTION SCORES

The enumerators of the EV interviewed in total 1,233 persons after explaining the purpose of the

survey and confidentiality of obtained information. These persons were women who attended services,

or parents (generally mothers) of children who were vaccinated.

The average score of satisfaction was 7.3 (out of 10) with the lowest average score of 4.1 in Koidu

New Sembehun City Council and the highest average score of 9.9 in Port Loko.

Table 19 Patient satisfaction scores and contributing factors

Average

satisfaction

score

Percentage

which received

friendly

treatment

Percentage with

reasonable

waiting time

Percentage which

received

prescribed

medicines

Bo City Council 7.6 74 85 76

Bo District Council 5.3 54 52 75

Bombali District Council 7.7 85 79 100

Bonthe District Council 7.8 80 83 89

Bonthe municipal Council 8.0 95 75 100

Freetown City Council 9.7 92 83 92

Kailahun District Council 5.6 68 55 79

Kambia District Council 8.8 88 82 100

Kenema City Council 6.9 76 52 82

Kenema District Council 6.2 70 44 83

Koidu New Sembehun

City Council

4.1 65 18 74

Koinadugu District

Council

7.0 80 46 99

Kono District Council 6.6 68 51 86



Average

satisfaction

score

Percentage

which received

friendly

treatment

Percentage with

reasonable

waiting time

Percentage which

received

prescribed

medicines

Makeni City Council 8.8 94 74 100

Moyamba District Council 7.2 87 62 90

Port Loko District Council 9.9 98 99 100

Pujehun District Council 5.9 59 64 76

Tonkolili District Council 8.3 97 85 92

Western Area Rural

District Council

8.5 90 86 82

Grand Total 7.3 80 67 89

Higher satisfaction was correlated with short waiting times, kind attitudes of staff and availability of

medicines. There is no baseline survey available to compare the potential increase in patient

satisfaction. However, compared to international standards patient satisfaction is good.

5.2.2 SATISFACTION SCORES PER LEVEL OF FACILITY

Table 20 below shows the patient satisfaction per level of facility in the Councils. Contrary to the

expectation, patients are in general more satisfied with the services provided by lower level facilities

than provided by higher-level facilities. CHC score lower in staff attitude, waiting time and medicines

available. The average satisfaction scores were much higher for the MCHPs. The proportion of

facilities with average client satisfaction scores above 7.5 is 67% for MCHPs and 54% for CHPs.

Nearly two thirds of facilities with average scores above 7.5 are MCHPs.

Table 20 Patient satisfaction per level of facility

District CHC CHP MCHP Total

Bo City Council 8.00 7.31 7.61

Bo District Council 4.62 5.00 5.61 5.33

Bombali District Council 7.25 7.27 7.97 7.67

Bonthe District Council 3.33 8.64 7.71 7.77

Bonthe municipal Council 8.00 8.00

Freetown City Council 10.00 9.62 9.55 9.67

Kailahun District Council 5.00 5.61 8.00 5.62

Kambia District Council 6.67 8.13 9.06 8.79

Kenema City Council 6.43 5.00 7.37 6.92

Kenema District Council 7.39 4.38 6.54 6.20

Koidu New Sembehun City Council 5.00 6.00 2.78 4.12

Koinadugu District Council 7.50 7.78 6.93 7.03

Kono District Council 5.00 6.82 6.57

Makeni City Council 9.17 8.82

Moyamba District Council 4.23 7.50 7.69 7.20

Port Loko District Council 9.55 9.74 10.00 9.86

Pujehun District Council 5.91 5.50 6.00 5.85

Tonkolili District Council 3.89 7.14 9.06 8.35

Western Area Rural District Council 9.06 9.09 7.65 8.52

Total 6.64 6.96 7.67 7.30



5.2.3 PAYMENT FOR SERVICES

As free health care for reproductive and child health is important for Sierra Leone, the EV also asked

for details of payments made for services to pregnant mothers and children. Table 21 below shows

that 146 out of the 1,233 persons interviewed (12%), were asked to pay for services, with a variation

between 0% and 52%. The amounts paid are reflected in table 22. The average was 7,881 LE, with a

variation between 200 LE and 50,000 LE.

Table 21 Persons interviewed who were asked to pay for services

Number Percentage

Bo City Council 4 17%

Bo District Council 2 3%

Bombali District Council 1 1%

Bonthe District Council 6 8%

Bonthe municipal Council 0 0%

Freetown City Council 2 5%

Kailahun District Council 38 52%

Kambia District Council 1 1%

Kenema City Council 1 2%

Kenema District Council 29 21%

Koidu new Sembehun City Council 6 35%

Koinadugu District Council 2 3%

Kono District Council 19 22%

Makeni City Council 0 0%

Moyamba District Council 15 19%

Port Loko District Council 0 0%

Pujehun District Council 6 13%

Tonkolili District Council 1 1%

Western Area Rural District Council 3 5%

Total 146 12%

Table 22 Average, minimum and maximum amounts paid

District Average Amount Le Min Amount Le Max Amount Le

Bo 13,600 3,000 25,000

Bombali 15,000 15,000 15,000

Bonthe 10,833 5,000 30,000

Kailahun 6,592 500 40,000

Kambia 5,000 5,000 5,000

Kenema 5,590 200 20,000

Koinadugu 17,500 15,000 20,000

Kono 8,250 500 30,000

Moyamba 8,958 5,000 20,000

Pujehun 18,500 500 50,000

Tonkolili 5,875 1,000 10,000

Western Rural 1,000 1,000 1,000

Western Urban 10,000 10,000 10,000

Grand Total 7,881 200 50,000



Of the patients who paid, 37 paid for general consultation, 82 for MNCH services and 44 for

medicines. Payments were sometimes related to purchase of exercise books as patient records,

gloves, soap, etc.

5.3 PATIENT TRACING BY COUNCILS AND DHMTS

In the EV interviews 44% of the Councils and 46% of the DHMTs mentioned they did some type of

patient tracing, although not formalised. Some use HMCs to assist in this effort. Patient tracing is part

of the PBF Operational Manual, but no tools have been developed. None of the DHMTs could show

results of the tracing and therefore the EV team could not make a comparison with findings of previous

patient tracing activities.

THE EXTERNAL VERIFICATION OF THE PATIENT TRACING AND SATISFACTION FOUND THAT: 92.6% OF THE PATIENT/CLIENTS EARMARKED FOR TRACING COULD INDEED BE

IDENTIFIED EITHER BY MEETING THE CLIENTS IN PERSON, OR BY IDENTIFICATION BY

A MEMBER OF THE COMMUNITY.

THERE IS NO REASON TO BELIEVE THAT PHUS RECORDED “GHOST PATIENTS” TO

INFLATE THE NUMBERS OF ATTENDANCE.

THE AVERAGE SATISFACTION SCORE OF CLIENTS WAS 7.3 (OUT OF 10), WITH A

VARIATION BETWEEN 4.1 AND 9.8.

CLIENT SATISFACTION WAS STRONGLY RELATED TO SHORT WAITING TIMES,

FRIENDLY TREATMENT, AVAILABILITY OF MEDICINES AND NON-PAYMENT FOR

SERVICES (FREE HEALTH CARE.

12% OF PATIENTS INTERVIEWED HAD TO PAY FOR SERVICES, ALTHOUGH THEY

WERE SUPPOSED TO BENEFIT FROM FREE HEALTH CARE.



6 SYSTEMS ASSESSMENT

6.1 INTRODUCTION

The External Verification team performed a system analysis as per Terms of Reference. The EV

enumerators collected information from PHUs and interviewed in-charges and from health

management committees. The EV coordinators collected information from DHMTs and Councils and

interviewed relevant staff. The international consultants interviewed several parties at national level in

MOHS, MOFED, NGOs, civil society, etc.

Most topics were already discussed in the Inception Report of the EV and in the Issues Paper for the

Validation Workshop in March 2014. The inception report discussed issues of system design20

. Those

will not be repeated here. This final report analyses the system as it is implemented in practice.

PBF programmes in general consist of six building blocks, which together strengthen the health

system and produce better healthcare and better services utilisation. Figure 29 below shows the

building blocks. In this chapter topics are discussed per building block.

Figure 29: Building blocks of RBF

6.2 ACCESSIBILITY AND EQUITY

Free Healthcare was introduced in 2010, and was supported by several donors an agencies, e.g.

through human resources management, provision of medicines, etc. Free health care has resulted in

considerable increase in service delivery in reproductive and child health services, although recently

there has been a levelling off of service utilisation at a higher level than before the introduction of Free

Healthcare.

The PBF programme works complementary to Free Healthcare, and offers to health facilities a

compensation for the loss of income through patient fees. The programme has been successful in this,

as according to interviewed respondents the income from PBF is much higher than from patient fees.

However, late payment affects this element, as will be discussed below in paragraph 6.2.4. Bonuses

may contribute to improved staff attitudes for service delivery although there is no baseline study to

compare present patient satisfaction figures.

Free Healthcare is provided in the whole country and does not target specific vulnerable groups or

specific geographic areas. The PBF programme provides an equity bonus to health facilities and

personnel in remote districts. The equity bonus ranges from 0% (e.g. Freetown an Bo), 20% (e.g. Port

Loko and Pujehun), 30% (e.g. Kailahun), 40% (e.g. Kenema and Koinadugu) to maximum 50%

(Bonthe). This equity bonus could reduce staff turnover and motivate people to continue working in

20

Cordaid (2012), External verification Performance Based Financing in the health sector in Sierra Leone, Inception Report

Accessibility and Equity

Autonomy and Accountability of

Health Institutions

Contracting Indicators and

Monitoring

Community Involvement

Separation of functions



their duty station. However, no baseline exists to compare the present staffing levels. The EV could

not measure an impact of the remoteness bonus. In interviews the retention effect of extra bonuses for

health workers in remote areas was not clear, especially because of payment problems.

6.3 AUTONOMY AND ACCOUNTABILITY PHUS

6.3.1 CAPACITIES

In the PBF approach, decentralisation to the facility level of planning, budgeting implementation and

accounting for small projects, is at the heart of the system. This is new for the health system in Sierra

Leone, which is still building up after a long period of decline. Not only the numbers of staff are

insufficient (often only one qualified staff in lower level PHUs), but also the level of training is low (e.g.

MCH-aid). DHMTs rank human resources problems highest in the list of challenges in the health

sector. This creates challenges for the tasks decentralised to the PHUs. High turnover was noticed

both in health facilities and in DHMTs.

When the PBF programme started, training was provided to DHMTs and health staff through cascade

training. Most new staffs are dependent on on-the-job orientation in PBF, provided by the DHMT. In

77% of PHUs visited during the EV, one or more of staff members had been oriented in PBF in the

past. Of the PHUs visited 48% of present in-charges understood PBF system completely and 62% of

health workers understood bonus calculations applied in PBF. 82% of the staff understood the quality

aspects assessed in the crosscutting indicators. Capacity building therefore was mentioned as key

issue in the validation workshop in March 2014.

6.3.2 PLANNING AND MANAGEMENT OF SMALL PROJECTS

The PBF programme stimulates entrepreneurial capacities of health workers, who can actively

improve their working environment. Of the PHUs in-charges 87% indicated having sufficient autonomy

for small project management, but only 57% felt the staff had enough capacities to make a plan.

Of all PHUs 62% made action plans, at least a plan how to utilise the money paid from the PBF

programme. Sometimes PHUs just produced procurement lists. Often those plans were made shortly

before money was withdrawn from the bank accounts.

The action planning should be a collaborative effort between in-charge, staff and Health Management

Committee according to the operational manual. However, this was not always the case, as shown in

table 23 below. Community participation is discussed in the next paragraph.

Table 23 Involvement in action panning of PHUs

Involved in the action planning according to PHU No health facilities Percentage

DHMT 35 15%

Health Management Committee 134 58%

Facility staff 142 59%

Council 11 4%

In-charges manage the small projects which are paid from the PBF programme sometimes with

assistance from the DHMT or from the HMC.

As per instructions in the PBF operational manual PHUs has to spend 60% on incentives for all staff

members (formula based on positions) and 40% on investments.

Health facilities invested in: • Making the work environment more conducive for services (furniture, painting and repairs of

building, solar lights, repair of motorcycles, etc.) • Equipment and supplies (BP machines, weighing scales, registers, patient cards, stationery,

paper, kerosene for stoves, etc.) • Sanitation and hygiene in the PHU or hospital (improvement of water supply, cleaning materials,

utensils, waste management, etc.) • Medical supplies (small quantities of medicines)



Figure 30 Word cloud investments in PHUs

Most investments were small, not exceeding amounts of LE 500,000 (US$ 1,000).

Figure 30 above is a word cloud of recorded investments made with PBF funds. The Council reports in

annex 4 volume II provide details on the top 10 of investments per districts.

Health facilities considered the PBF funding a welcome addition to government funding. However,

they still struggle with problems of shortage of equipment and supplies, which was listed as the

number 1 constraint in interviews.

6.3.3 FINANCIAL MANAGEMENT IN PRACTICE

The PBF operational manual does not provide instructions on financial management. Most PHUs keep

very simple records of income and expenditure, often not even meeting the minimum standards of a

cashbook. This is not surprising as the in-charges were never instructed how to perform these duties.

During the EV only 62% of PHUs could show records in a cashbook or ledger book of any PBF

amount received. They could not produce payment slips from the bank, or other explanation of

transfers made to their bank accounts. One reason for missing records given was that the former in-

charge had taken the cashbook on transfer. In general there was no handover of finances on

replacement of the in-charge.

The tables 24 and 25 below show details of amounts, which should have been paid and amounts

identified by the enumerators. Kono, Moyamba and Tonkolili show very low percentages of amounts

identified.

Table 24 PBF payments to PHUs requested by MOHS to MOFED

District Payment request

Q2 2012

Payment request

Q3 2012

Payment request

Q4 2012

Bo 57.669.490 39.517.740 59.689.650

Bombali 32.969.115 41.159.835 43.226.223

Bonthe 64.197.268 7.490.038 29.423.563

Kailahun 21.894.331 25.282.491 31.859.685

Kambia 37.650.791 22.854.325 24.825.178

Kenema 64.933.974 79.009.452 68.304.576

Koinadugu 42.570.431 53.262.664 56.420.994



District Payment request

Q2 2012

Payment request

Q3 2012

Payment request

Q4 2012

Kono 59.958.146 2.653.578 33.086.118

Moyamba 18.648.839 11.992.702 20.820.651

Port Loko 24.372.522 26.597.352 36.303.282

Pujehun 41.017.270 18.283.765 22.506.204

Tonkolili 27.062.048 42.483.080 33.596.010

Western Area 74.188.790 116.564.140 79.050.090

Total 567.133.014 487.151.161 539.112.223

Table 25 Percentage of quarterly payments, traced in PHUs during the EV

2nd

quarter 2012 3rd

quarter 2012 4th

quarter 2012

Bo 54% 73% 30%

Bombali 79% 54% 53%

Bonthe 45% 117% 14%

Kailahun 63% 59% 38%

Kambia 51% 70% 79%

Kenema 72% 74% 56%

Koinadugu 79% 43% 6%

Kono 10% 0% 2%

Moyamba 2% 4% 59%

Port Loko 73% 24% 1%

Pujehun 36% 35% 25%

Tonkolili 2% 1% 67%

Western Area 71% 43% 5%

Total 41% 35% 27%

Investments of LE 621 million were traced in PHUs during External Verification, which amounts to

roughly 75% of the estimated LE 830 million, the available amount for investments in 201221

. Most

health facilities had not yet received full payment for 2012 when the EV took place. Therefore, most

PHUs could not have reached 100%. It may be concluded that in PHUs recording of expenditure was

more precise than recording of income.

Most in-charges kept receipts for expenses and all DHMTs checked expenditure during the IV (and

sometime took the receipts to their office). However only four DHMTs collected financial reports from

the PHUs and only two forwarded financial reports from the PHUs to the Councils.

6.3.4 DELAYS IN PAYMENT

The timelines as described in the PBF operational manual were not kept, due to delays in all stages of

the process: delayed internal verification reports by DHMTs, delayed processing of payment requests

by MOHS and delayed payments by MOFED. Payments for the fourth quarter of 2012 were processed

by the MOFED-LGFD in the first quarter of 2014. MOHS and MOFED now process all requests for

payments as they come in without first accumulating all claims. This will considerably shorten waiting

times for most facilities. However, the operational manual was too optimistic in its planning. Financial

procedures take their time and a six-moths’ procedure is more realistic than a 3-months’ procedure.

In PHUs there seems to be no insight in the relation between performance and payment. Some in-

charges interviewed claimed that they did receive payments for 2013, but not for 2012. They could not

see a relation between outputs, quality scores and amounts paid.

In the third quarter 2012 corrections were made over payments in the previous quarters: some

facilities received no or even “negative” payments in the third quarter of 2012. In-charges were not

21

MOHS could not provide exact amounts of disbursements for first quarter 2012. The EV team assumed it was more or less equal to other quarters.



informed, and some claimed they were denied payment over the third quarter. Due to insufficient

feedback from the IV, PHUs do not know the amount of PBF payment, which they could expect over

quarters under review. The payment into the bank accounts came with little information and therefore

PHUs often did not understand for which quarter the payment was. The complete lack of explanation

and transparency with regard to payments to PHUs was experienced as very frustrating by PHU in-

charges interviewed.

Late payment affects continuity of the PBF programme and had high opportunity costs: PHUs were

eager to implement health services improvements, but had to wait for over one year to get their due

payments. Health workers expressed fear that they would not receive their bonuses after such a long

period of delay, e.g. after transfer. Not receiving a performance bonus created frustration, rather than

motivation for better performance.

6.4 COMMUNITY INVOLVEMENT

The PBF programme aims at increased community participation and co-ownership. According to the

PBF manual Health Management Committees22

(HMCs) should contribute to planning, management

and monitoring of health services. The HMC chair was supposed to be the co-signatory of the

accounts.

The Health Sector Strategic Plan explicitly mentioned community involvement as priority in the health

sector.

The programme succeeded in some areas in enhancing this community participation. In 58% of the

PHUs the HMC was involved in planning and in 35% of the PHUs the HMC chair was co-signatory of

the PHU account.

However, in general HMCs do not see co-management of the health facility as their priority. Often

HMCs have a rather traditional perspective regarding their roles. Participating in planning and priority

setting is mentioned by only 49 PHUs (20%), as shown in table 26 below.

Table 26 Roles of HMC according to chairs HMC

Roles of the HMC No of HMCs Percentage

Sensitisation of community and health education 173 72%

Mobilisation of funds or voluntary labour 161 67%

Provide feedback to the community 158 66%

Prioritisation of activities in the health facility 49 20%

Other* 27 11%

*(e.g. monitoring of arrival of drugs and settling disputes)

In general, the element of strengthening community involvement did not get much priority in the

implementation of the PBF programme so far. According to the programme design – in line with the

PBF theory – the Councils represent the interests of the population. They aim for value for money on

behalf of the citizens. Therefore one of the roles of the Council was to strengthen community

participation and to stimulate involvement of the Health Management Committees in the management

of PHUs. 82% of the Councils indicated that the HMCs were not sufficiently involved in the PBF

programme.

6.5 SEPARATION OF FUNCTIONS IN THE PBF PROGRAMME

The PBF principle of separation is built on the theory of split of functions between provider, purchaser

and regulator, with good reasons. The regulator formulates national policies and sets the standards for

quality of care and defined “the rules of the game”. The purchaser procures at the best value for

money, taking into account the policies as defined by the regulator. The provider sells its products,

which meet standards as defined by the regulator, for a price negotiated with the purchaser. In the

Sierra Leone PBF this approach is followed as well, as shown in figure 31.

22

A variety of titles are used for this committee, e.g. village health committee, health facility monitoring committee.



MOH-Regulator

focuses on quality

PHU-Provider

focuses on accessibility

Council-Purchaser

focuses on costs

Figure 31 Roles in PBF in Sierra Leone

As described in Chapter 2 Sierra Leone has embarked on a decentralisation-by-devolution, whereby

the MOHS is the steward in the health sector, having only direct control over tertiary hospital. The

Councils are the owners of the health facilities and responsible for the health service provision.

Administratively the DMOs and the DHMTs are under the Council. Funds for managing the health

services are channelled via the Councils.

In Sierra Leone a “light” PBF approach is applied, which means that not all theoretical concepts of

PBF are fully implemented. The Council is officially responsible for the health services, but is at the

same time the contracting agency. The DHMT is the technical supervisor and at the same time the

internal verifier.

6.5.1 COUNCILS

In the present situation most Councils do not play their role of purchaser as envisaged in the

programme design. From the interviews it transpired that half of the Councils did not understand the

contents of the contract and 44% did not understand the criteria applied to PHUs for eligibility of

contracting. Only 28% of the Councils kept the contracts with the PHUs in their files. Of all Councils

30% indicated not to have capacities for the tasks attached to the PBF programme.

One third of all Councils was never involved in the quarterly internal verification, one third was rarely

involved, and only one third was regularly involved in IV. If the Council was involved it was mostly the

M&E officer (66%) and occasionally elected Council members (16%).

6.5.2 DHMTS

DHMTs have a triple role in the health sector and in the PBF programme:

Supervision and technical guidance of PHUs on behalf of the MOHS, which is the regulator in

PBF;

Management of the health services on behalf of the owner of the health services, which is the

Council according to the devolution policy;

In the PBF programme internal verification on behalf of the purchaser, which again is the Council.

In figure 31 above the DHMT is positioned in the middle, and sometimes it is caught in the middle,

serving two masters. Both the regulator and purchaser have to provide clear instructions to the DHMT

Technical guidance

and supervision

National policies for

healthcare

Value for money

DHMT



ensuring that all interests are covered adequately. The EV came across several challenges

concerning the DHMT’s role.

Internal verification is taking place systematically. 94% of the PHUs indicated that DHMTs performed

regular supervision and this was confirmed by the DHMTs. The IV puts a huge strain on the DHMTs,

as it is time consuming. With 50 to 100 PHUs in one district the process may take weeks every

quarter, claiming time from DHMT members and claiming means of transport from the office. More

than half of the DHMTs perform the IV within 4 hours; one-fourth spends between 4 and 6 hours and

few more than 6 hours. Based on the experience in the EV it is impossible to do a thorough verification

including review of registers and assessment of the crosscutting issues within 4 hours, certainly if it is

combined with supervision and if it includes travel time.

The results of the internal verification have been discussed extensively in Chapter 4. In general, in the

districts the results of internal verification of output indicators and crosscutting issues were higher (in

some cases significantly higher) that the external verification.

Financial management. DHMTs perform some supervision regarding financial management in PHUs;

One-third of the DHMTs receive a financial report from PHUs, but more often copies of receipts of

purchases. Only two of the 13 DHMTs forwarded them to the Council. DMOs in two districts are co-

signatory of the PHU accounts, and eight DMOs have to check the investment plan before the PHU

can withdraw money from the bank. None of the DHMTs introduced cashbooks in the PHUs.

Capacity building. Officially, the purchaser has the task to ensure that provider understands the rules

for payment for service delivery, i.e. how to register, how to report, how to manage funds, how to plan

for activities, etc. The Councils could delegate the task of capacity building to the DHMT. DHMTs have

indicated to perform on-the-job training, but concentrate more on technical issues than administrative

issues (like financial management). During the EV only half of the in-charges of PHUs indicated to

understand the PBF programme fully, and DHMTs indicated a huge need for capacity building.

6.5.3 MOHS

The MOHS has more functions than strictly according to the PBF theory. Besides producing payment

requests to MOFED, the PBF technical team is responsible for overseeing the verification process by

signing contracts with the internal verification teams, for the M&E systems (e.g. through HMIS) and

quarterly and annual reporting. The PBF programme is managed by a technical team consisting of

representatives from different departments in MOHS (Directorate of Policy, Planning and Information,

Directorate of Primary Health Care, Directorate of Reproductive and Child Health, Directorate of

Financial Management, Human Resources Division). The technical team provides training and

capacity building. The PBF technical team interacts with the Development Partners in the Steering

Committee.

Last year the PBF technical team was reshuffled. The new PBF technical team has not yet performed

a national verification; the previous team did perform verification, but did not produce a report. The EV

team could not check which issues they came across. The new PBF team did not perform data

triangulation of HMIS, IV and F-reports, which did show major data differences in the EV. It processed

payment requests to the MOFED based on IV reports only.

6.5.4 MOFED

The MOFED - Local Government Finance Department is responsible for disbursement of the PBF

fund after request by the MOHS and supervision of financial management by the Local Councils. The

MOFED completed financial reports on PBF for the World Bank, but in those reports only accounted

for money transferred to the PHUs, not how the money was spent in practice.



6.6 DEFINITION OF INDICATORS

The domains and indicators and described in Chapter 4.1. Indicators are the backbone for payment to

the PHUs and hospitals. Several indicators did not have SMART definitions in the operational manual

or other documentation. In practice internal verification teams applied no uniform definitions. Therefore

the consultants invited the stakeholders during the inception workshop to work jointly on the

formulation of definitions for indicators and certain terminology. These agreed definitions have been

applied in the external verification. Not having very clear indicators may have affected the internal

verification, leading to variation in scores between DHMTs.

6.7 CONTRACTS

PHUs were contracted for service delivery in tripartite contracts with DHMTs and Councils. As has

been explained above those contracts did not play any further role in the PBF implementation. The

procedures described, e.g. with regard to financial reporting were not followed. Never were any

sanctions applied in case when parties were not adhering to their duties.

Non-governmental healthcare providers would be eligible for contracting as well, according to the PBF

manual. In practice, none was contracted.

6.8 HOSPITAL PBF

6.8.1 CONTRACTS

Princess Christian Maternity Hospital (PCMH) and Ola During Children’s Hospital (ODCH) are the only

hospitals benefiting from the PBF programme. In the hospital PBF system there is strong emphasis on

learning, exchange of best practices, recommendations for improvement and assessment of the

follow-up of these recommendations during the next verification. Following the PBF operational

manual, in 2012 tripartite agreements were made between the Freetown Council, the MOHS and the

hospitals, in which roles and responsibilities were described. The Council acted as the purchaser.

However, the Council did not play a role in internal verification. Per January 2014 tertiary hospitals

were transferred back to the MOHS, but contracts have not been adjusted.

6.8.2 IMPLEMENTATION

ODCH has installed a Quality Assurance (QA) Committee; in PCMH the management team acts as

QA committee. These committees perform self-assessment on a monthly basis, using the PBF

checklist, as described in Chapter 4.1. The self-assessment is the starting point for setting priorities for

quality improvement.

The national PBF technical team, together with invited experts from other ministries, performed the

internal verification. It was noted that every time a different team did the internal verification, which

according to the hospitals resulted in inconsistent recommendations for improvement. The second and

third quarter 2012 internal verification was combined.

6.8.3 INDICATORS

The indicators in eight domains all contribute to a score for that domain. Not all indicators are SMART

(e.g. “are there functional toilets in the hospital”), and not all indicators are applicable (e.g. “clean

labour ward” in ODCH). The scoring may therefore be subjective. Nevertheless, scoring was often

high in the IV, as discussed in chapter 4. The IV forms often only have scores, with the remarks

column left blank. It is therefore not clear where the scoring is based on, making it difficult for a next

team to follow up.

The indicators only concentrate on quality and do not take output into account, with as argument that

tertiary hospitals are dependent on referrals and should not increase their output, taking patients away

from lower level facilities. However, the introduction of free health care has put a strain on quality of



care in hospitals. According to respondents, patient numbers increased 4 to 10-fold as a result of

abolishment of user fees, while financial compensation and inputs did not increase proportionally.

6.8.4 EXPENDITURE

Similar to the PBF project in PHUs, hospitals spend a maximum of 60% on staff incentives and 40%

on investments. The incentives are for all staffs and tied to their salary. For most staffs it means an

extra month’s salary every quarter. All workers are therefore very much engaged in PBF. Both

hospitals have introduced a performance element in bonus payment, first only presence and now also

commitment, to vary the height of performance bonus.

The 40% investment money is a lifeline for the hospitals, constituting 40% - 60% of their funds for

running costs. As Government funds are tied to strict procedures through the Freetown City Council,

and often delayed due to cumbersome bureaucracy, this free money offers an opportunity to solve

problems quickly, e.g. shortage of medicines, laboratory supplies, gloves, maintaining water supply,

etc. The funds are also used to procure equipment, stationery and bed linen. Hospitals make a

quarterly investment plan together with the heads of department.

Like PHUs, hospitals are affected by delayed verification and slow procedures of disbursement, which

affects continuity of the work. Recently, Government has informed that their regular budget would be

lowered, because the hospitals get PBF funding.

6.8.5 NON-PBF HOSPITALS

The two non-PBF hospitals verified, operate at very different levels. Rokupa Hospital is a very busy

well-equipped district hospital with many workers and patients. It has a busy theatre and labour ward.

Macaulay Medical Centre is a dilapidated health centre, with a labour ward and theatre under

construction. Their scores in the external verification cannot be compared.

The lesson learned form this comparison is that introduction of PBF requires a good programme of

training and introduction of systems and procedures. Verification forms should be more custom-made

as every level of hospital has other circumstances to be assessed.

SUMMARY OF SYSTEMS ANALYSIS

THE PBF PROGRAMME WORKS COMPLEMENTARY TO FREE HEALTHCARE, AND

OFFERS TO HEALTH FACILITIES A COMPENSATION FOR THE LOSS OF INCOME

THROUGH PATIENT FEES. THE PROGRAMME HAS BEEN SUCCESSFUL IN THIS.

THE PROGRAMME HAS SUCCEEDED IN PROVIDING MORE AUTONOMY TO

HEALTH FACILITIES TO MANAGE THEIR OWN SMALL PROJECTS, WHICH

CONTRIBUTE TO BETTER WORK ENVIRONMENT: MORE HYGIENE, BETTER

EQUIPPED BUILDINGS AND BETTER SUPPLIES HAVE BEEN ACHIEVED.

FINANCIAL MANAGEMENT IS A WEAK AREA, WITH VIRTUALLY NO SYSTEMS IN

PLACE AT GRASS ROOT LEVEL.

LATE PAYMENT DURING THE PERIOD OF REVIEW AFFECTED CONTINUITY OF THE

PBF PROGRAMME AND HAD HIGH OPPORTUNITY COSTS: PHUS WERE EAGER TO

IMPLEMENT HEALTH SERVICES IMPROVEMENTS, BUT HAD TO WAIT FOR OVER

ONE YEAR TO GET THEIR DUE PAYMENTS.

HEALTH WORKERS EXPRESSED FEAR THAT THEY WOULD NOT RECEIVE THEIR

BONUSES AFTER SUCH A LONG PERIOD OF DELAY, E.G. AFTER TRANSFER. NOT

RECEIVING A PERFORMANCE BONUS CREATED FRUSTRATION, RATHER THAN

MOTIVATION FOR BETTER PERFORMANCE.

THE PROGRAMME HAS SUCCEEDED TO SOME EXTENT IN IMPROVING

COMMUNITY CONTRIBUTION TO MANAGEMENT OF HEALTH FACILITIES,

ALTHOUGH THE CAPACITIES ARE STILL LIMITED.

IN SIERRA LEONE A “LIGHT” PBF APPROACH IS APPLIED, WHICH MEANS THAT



NOT ALL THEORETICAL CONCEPTS OF PBF WITH REGARD TO SEPARATION OF

RESPONSIBILITIES (E.G. HEALTH RESULTS INNOVATION TRUST FUND23

) ARE

FULLY IMPLEMENTED. THE LOCAL COUNCIL IS OFFICIALLY RESPONSIBLE FOR

THE HEALTH SERVICES, BUT IS AT THE SAME TIME THE CONTRACTING AGENCY.

THE DHMT IS THE TECHNICAL SUPERVISOR AND AT THE SAME TIME THE

INTERNAL VERIFIER. IN PRACTICE THE COLLABORATION BETWEEN COUNCILS

AND DHMTS OFTEN IS NOT AS ENVISAGED IN SIERRA LEONE’S PBF PLAN. THE

DHMTS OFTEN OPERATE INDEPENDENTLY, AND COUNCILS DO NOT FEEL

ENGAGED IN THE PROGRAMME.

THE HOSPITAL PBF HAS STIMULATED THE TWO INVOLVED HOSPITALS TO

IMPROVE PERFORMANCE IN MANY AREAS. HOSPITALS ARE BECOMING

DEPENDENT ON THESE FUNDS AS PART OF THEIR CORE FINANCING.

23

https://www.rbfhealth.org



7 DISCUSSION

7.1 QUALITY

The general objective of the PBF system is to change the behaviour of health providers at facility level


The External Verification assessed elements of improvement of quality of care. The patient

satisfaction survey was an instrument to assess the perceived quality of care and the crosscutting

issues quality assessment was an instrument to objectively score quality. The EV found a statistically

significant correlation between high total scores for crosscutting quality issues in EV (not in IV) and

high scores for patient satisfaction in facilities. Within the set of crosscutting topics cleanliness and

availability of medicines were most prominent in the correlation with patient satisfaction. Good

supervision by the DHMT and good feedback from internal verification were also in a statistically

significant way linked to higher scores on crosscutting issues. The regression analysis showed that

the involvement of the health management team in the action plans was significantly associated with

client satisfaction. The positive coefficient indicates that higher satisfaction scores are observed for

facilities where this is practiced. The EV also found a positive correlation between investments, patient

satisfaction and higher scores on crosscutting issues. A coherent system of enabling factors for quality

improvement is in place (figure 32).

Figure 32 Relations in the quality system

The quality system is dependent on more factors, like human resources, drug supply, etc. Quality in

the health system in Sierra Leone is a collaborative effort, and not exclusively linked to PBF. However,

the PBF programme was able to create leverage by well-directed triggers for quality improvement.

This can be enhanced when the link between performance and payment is strengthened, i.e. when the

system becomes more transparent, better understandable, and when payments of bonuses follows

shortly after provision of quality services.

7.2 DATA QUALITY

Data quality issues dominated this EV. Here four topics are discussed: missing data, data consistency,

case definitions and triangulation.



7.2.1 MISSING DATA

Of the F-forms 3%-5% were missing, of HMIS data 15%-17%, of IV reports 9% - 11%. The EV

encountered 4%-24% missing registers, as explained in chapter 4.

Missing hard copies of reports and soft copies of computer files indicates poor storage practices at all

levels. In 2012 the HMIS system crashed and no sufficient back-ups were available to restore data.

Missing IV reports at national level were caused by reshuffling of the PBF team and lost data on

computers, but at district level by poor practices of storing and backing-up computerised information.

Registers are frequently out of stock in PHUs (table 27). Especially the registers for maternal health

often are improvised registers in ledger books, notebooks, or loose papers with many shortcomings,

e.g. missing columns or data sets. In the EV many of these improved registers from 2012 were lost.

Table 27 Registers reported out of stock by PHUs

Register Books out of stock No of PHUs

OPD under 5 47

Immunization 33

Maternal Health Registers 118

Family Planning 21

None 79

The external verifiers occasionally came across situations where new maternal health registers were

present but not used, as the staff could not understand them. In few cases, even under-five cards

were missing. The erratic supply of stationery definitely affects the quality of the PBF programme.

7.2.2 DATA CONSISTENCY

F-forms. As explained in Chapter 4, between 90% and 95% of the registers are filled in satisfactory,

which means that information in the register is in general reliable, if the proper stationery is used.

Quite a number of staff do not understand the F-report forms fully and fill in wrong information, or copy

from wrong registers. At times, health workers have problems with mathematic skills.

One example of errors in reporting is the box for filling in daily attendance on form PHU-F1. See figure

33 below. It is not clear what is meant total head count (all services) and total OPD cases. Does head

count include all OPD services or all preventive and curative services in the facility? Does total OPD

cases mean number of patients or number of diagnoses? In the EV it was found that PHUs apply

different definitions. Clarity and consistency in use of report forms is needed.

Figure 33 PHU-F1 form box totals

HMIS. The deviations between HMIS, entered in the automated DHIS-2 system, and the F-forms may

be considerable. The PHUs with more than 25% data variation ranged from 14%-71% for output

indicators. Some districts mentioned in interviews making corrections in HMIS when figures in F-

reports were wrong, without correcting the F- forms. There may also be copy errors, when data entry

clerks exchange forms from facilities. Between 36%-69% of PHUs had a 25% data deviation between

HMIS and EV, which is very high. In the validation workshop it was concluded that capacities of M&E

units in the DHMTs are too low, and that quality control by DMOs was minimal. Poor quality of HMIS

not only affects the PBF system, but also the health sector as a whole.



Internal verification. The data in the internal verification deviated much from the EV. Between 33%-

100% of PHU had more than 25% data deviation. Discussions during the validation workshop pointed

at time constraints to complete the job in time or lack of capacities in the DHMT. The high scores in

internal verification cannot be contributed to deliberate over-scoring but rather to interpretation errors

of case definitions.

7.2.3 CASE DEFINITIONS

In previous chapters issues of clear case definitions were mentioned. The EV team noticed that

different interpretations of case definitions might have affected the IV, even for assessing the output

indicators:

Family planning: condom distribution was to be excluded from the count (more for prevention of

STD than for FP), but some DHMTs included it;

ANC: the fourth attendance was to be counted, but some included the fifth and later visit as well.

PNC: the third attendance was to be counted, but some included all post-natal controls after 6

weeks, even if it was the first after delivery.

Fully vaccinated: only children below the age of one were to be counted, but some included all,

even above the age of one.

Apparently, there was not sufficient guidance from the MOHS to come to uniform approach in IV.

7.2.4 TRIANGULATION

F-forms, IV and EV all base their information on the registers; HMIS is an electronic copy of the F-

forms. The deviations between F-forms, IV and HMIS could have been noted in 2012 if triangulation

had been done. Before sending the IV reports, DHMTs could have compared data with F-forms and

HMIS, and correct or explain deviations. Similarly, at national level comparison could have been done

between HMIS and IV district data.

Improving data quality should be one of the top priorities in advancing the PBF programme. This

requires agreement on definitions and procedures as well as capacity building at different levels to

ensure that at the grass root level in PHUs and DHMTs the right approach is followed. Incentives for

quality of information could be considered. Quality control, e.g. through triangulation, should be

considered.

At the same time the burden of work created by internal verification should be reduced.

7.3 PBF LIGHT

Sierra Leone has opted for a PBF light approach, which means that not all theoretical concepts of

separation of functions and systematic changes were taken on board. In the PBF operational manual

systems and procedures have been described.

However, the practice shows that PBF has become more a programme of MOHS, DHMTs and health

facilities, than an intersectoral programme, with involvement of the Councils, Ministry of Local

Government and Rural Development, and MOFED. Councils were in two third of the cases not or not

sufficiently involved and did not carry out (or were not enabled to carry out) their part of the PBF

programme. From the PBF perspective the value-for-money or purchasing perspective was not given

sufficient attention. The relation between payment and performance has not been sufficiently clear,

especially not at the PHU level. One of the reasons given during the validation workshop was that the

Councils did not have an official role in accounting for the money, as funding was bypassing

Government structures at Council level. There was no real incentive or obligation to take

responsibilities, while so many other programmes cried for more attention in an overburdened

administration.

The design of the PBF provides a bigger role for communities than actually implemented in many

PHUs. Capacities of HMCs to co-manage may be limited, or willingness on the side of the health

workers to share responsibilities may be limited. But for sure, it is an area, which requires further

attention. Patient tracing and satisfaction surveys are mentioned in the design, but not elaborated. It

can be a powerful instrument, if linked to incentives for health facilities.



The PHU design as described in the operational manual is sufficiently robust to guarantee a

performance based financing approach, and reconfirmation of that commitment and restart of that

system would bring more balance in the PBF programme. It requires new engagement of

stakeholders, maybe making adjustments in the procedures where needed.

7.4 FINANCIAL MANAGEMENT

Financial management is discussed in paragraph 6.3.4. The omission in the PBF manual to formulate

requirements and to provide recording and reporting instructions, did never get a follow up. It more or

less ‘fell through the cracks in the system’. Given the very limited financial management capacities of

grass root health workers, it is surprising that most expenditure could be explained and even justified

with documents and receipts in this EV (although income could not be explained, partly because

banks do not provide payment slips).

The fact that Councils did not feel obliged to control financial management, because money was

bypassing their system, is remarkable. Apparently in the contracting process no clear agreement was

reached on roles and responsibilities of partners.

The most important observation during this External Validation was that due to long delays in

payments, the lack of transparency in how amounts were reached, health workers at grass root level

completely missed the “emotion” of payment for performance”. Money was not seen anymore as a

reward for an effort during a given quarter, but as a (long-overdue) welcome addition to other sources

of income or supplies. MOHS and MOFED have started to catch up with clearing the backlog of

payments over 2013, and have changed to processing of payment requests. Hopefully it will go hand

in hand with an effort to reinstate the “payment for performance feeling” in the system.

The issue of payment for travel allowances and payment for incentives for DHMTs and Councils is

another issue to be clarified for involved stakeholders. Present confusion does not contribute to

motivation of DHMT members or Council officials to perform internal verification.



8 CONCLUSIONS AND RECOMMENDATIONS

8.1 CONCLUSIONS

8.1.1 THE GENERAL AND SPECIFIC OBJECTIVES OF THE PBF PROGRAMME



The EV team concludes that systems have been put in place and are operational to a reasonable

extent in a number of health facilities (see Chapter 7.1). Further strengthening of the system is

possible within the present design of PBF in Sierra Leone, when a number of implementation issues

can be solved.

The specific objectives of the system are:

1. Provide cash at facility level to cover the local costs of delivering services and removing the need

for 'informal' fees.

The EV team concludes that this has been largely achieved, with only 12% of the patients paying

for those services, which supposedly are free. (See Chapter 5.) Late transfers of PBF funds may

have forced PHUs to ask for contributions for patient records, etc. when funds dried up. Payments

by patients may reduce further if PBF payment timeliness and accuracy improves. Incidental

misbehaviour by health workers cannot be ruled out.

2. Provide financial incentives to facilities in order to increase productivity and quality of care,

especially for the identified key indicators.

The EV team concludes that this has been partially achieved. There is an increase in service

utilisation, although that increase is levelling off. There are signs of improved attention for quality.

However, the relation between performance and payments is too weak for health workers. The

incentive system is not transparent enough and payments so delayed, that they are no longer

seen as reward for good performance.

3. Increase the equity of distribution of resources with funds from PBF allowing facilities to hire

contractual workers and finance outreach activities.

Equity of distribution of funds may have taken place using district-based payment formula, but was

not visible for grass root workers. The flow of funds in general was not regular enough to hire

contract workers (with exception of the two PBF hospitals). Outreach may have benefited from

PBF funds, e.g. by repair of motorcycles and purchase of fuel. In general, funds were used for

repairs of the building, furniture, equipment and supplies, water and sanitation, etc. These

investments have contributed to patient satisfaction and higher scores for crosscutting quality

indicators.

8.1.2 THE TERMS OF REFERENCE

In response to the TOR for the external verification the EV team presents the following conclusions:

1. Review the accuracy of the facility data from the registers and other records

The registers at facility level were fairly accurate, with on average 90%-95% satisfactory entries.

Unfortunately, too many registers went missing over the period between 2012 and 2014. However,

the accuracy of F-form reports, HMIS reports and Internal Verification in too many cases is below

standard. Not only is the variation between the sources of information and the External Verification

too high, also between the sources of information amongst themselves there are too many

inconsistencies. This could have been avoided by triangulation of information from F-forms, HMIS

and Internal Verification.

2. To analyse the data of the first full year of PBF implementation (2012)

The methodology of analysis can be found in Chapter 3. Extensive analysis is reported in Chapter

4 of this document. Data quality and analysis of reasons for differences in numbers required much



attention. Data quality issues were caused by missing data, inaccurate data collection and transfer

of data, as well as variation in interpretation of case definitions.


of improvement if necessary

Chapter 6 analyses the PBF system and chapter 7 discusses some system successes and

failures. In general roles and responsibilities are sufficiently clear in the design of the PBF

programme (operational manual), but require new commitments and reconfirmation, especially to

increase the role of the Councils and communities in the PBF system. Councils should be

incentivised to actually pick up their purchasing role.




etc.).

Benefits of PBF are clearly visible, but more at micro-level than at meso- or macro-level. In

specific health facilities (and especially the two PBF hospitals) the impact can be seen and effects

of system strengthening can be proven: increased staff motivation, increased hygiene, better

supplies, etc. However, this is not the case across the board. The link between payment and

performance is not yet strong enough to motivate all health workers to go the extra mile to achieve

better. However, the EV has documented good practices and the potential effect has been shown

in the external validation.

There is no proof of improved performance of DHMTs and information systems. However, at the

district level the burden of work caused by the PBF programme might have been too much to cope

with. There is not enough clarity on payment for performance for IV and payment of allowances to

DHMT members and Council officials.

At the national level, discontinuity in the PBF team has caused major disruptions in further

progress of the programme, through loss of information, interruption of supervisory visits, loss

project implementation memory

8.2 RECOMMENDATIONS

8.2.1 VALIDATION WORKSHOP

The validation workshop on 20 March 2014 offered an opportunity for the EV team to discuss

preliminary findings with stakeholders from MOHS, MOFED, DHMTs, Councils, Civil Society,

Development Partners, etc. In the plenary and group discussions recommendations were formulated,

which found their way into this report.

One proposal by the Director of Reproductive Health Services in the MOHS should be mentioned

here, i.e. visiting all districts and discuss on the spot the way forward with stakeholders there, i.e.

DMOs, DHMTs, Councils and PHUs. The EV team supports this initiative. The specific district reports

annexed to this main report may guide this discussion, as well as the urgent recommendations.

8.2.2 SHORT-TERM RECOMMENDATIONS

Reconfirm roles and responsibilities

The MOHS district visits will offer an opportunity to confirm with the Councils the roles and the

responsibilities as laid-down in the PBF operational manual. The roles of the Councils in contracting,

in internal verification, and in financial management and reporting may have to be renegotiated per

Council, as circumstances and conditions may vary.

The roles of HMCs in the management of health facilities have to be clarified and their roles as

described in the operational manual have to be confirmed.

New Memoranda of Understanding can be signed to confirm commitments.



Clarify internal verification

During the district visits the MOHS could provide an orientation workshop on quality of internal

verification. The quality of internal verification has to improve: uniform case definitions have to be

applied, and DHMT members, Council officials and elected Councillors involved should understand

their tasks of verification of output indicators and crosscutting issues well.

On-the-job training in verification of cross-cutting issues may improve the consistency and quality of

this important part of the verification process.

On the spot double check of IV report, F-forms and HMIS form (brought from the DHMT’s M&E office)

should be introduced to identify data inconsistencies and resolve them, or explain them in case

mistakes are corrected. The by-product of the PBF programme should be more reliable HMIS.

Simplify internal verification

The validation workshop called for simplification of the Internal Verification, while improving the quality.

The idea was to introduce sampling, not only months (one month per quarter), but also PHUs (e.g.

25% of PHUs). HMIS data would be guiding in payment for performance, rather than the data from IV.

This is possible, but only if certain criteria are met. (See figure 34 below.)

The first step in this process is to guarantee data quality of registers, F-forms and HMIS. Facilities

should have the required registers and forms. HMIS and F-forms should be filled completely and

should match. Districts, which cannot meet minimum criteria of HMIS quality, should first bring

their house in order.

The second step is to select PHUs, which meet criteria of data quality, with matching IV and

HMIS. Those with reasonable data quality are admitted to the pool. But they can be removed from

the pool if in a control they are found to be missing the quality standards.

From there, step-by-step, more facilities are added to the pool introducing gradually a system of

sampling months and facilities. Throughout the time, random sampling should be used, which

even makes control in consecutive quarters possible.

NB: quarterly supervision and assessment of crosscutting issues should continue in all health

facilities! This has been shown to be a crucial element of quality improvement and cannot be done

through sampling.



Figure 34 Step by step introduction of sampling IV

Step 1

Select districts which

Have all data cells filled in

HMIS since January 2014 (no

missing data)

Have HMIS PHU report which

are similar to F-forms

Step 2

Select PHUs which

In Internal Verification have

for all six output indicators

less than 10% deviation from

HMIS (CHC and CHP)

In Internal Verification have

for all six output indicators

less than 25% deviation from

HMIS (MCHP)

District meets criteria

District does not meet

criteria

Continue full Internal Verification

Capacity building and control HMIS

until meeting criteria

PHU does not meet

criteria

Continue full Internal Verification


Step 3

Perform random sampling at

MOHS of PHUs in sample pool

Implement Internal

Verification of sampled PHUs

PHU has more than

allowed deviation for

one or more indicators

in Internal Verification

Remove from sampling pool

Perform Internal Verification in next

quarter; build capacity

When criteria are met during next

Internal Verification, move to sample

pool


PHU meets criteria

When criteria are met during next Internal

Verification, move to sample pool


Step 4

Continue process until all PHUs

meet criteria



8.2.3 CLARIFICATION OF THE OPERATIONAL MANUAL

Within six months MOHS could produce an update of the operational manual, including

communication or orientation of involved stakeholders, whereby the following suggestions can be

considered:

1. Create simple and targeted PBF manuals for different stakeholders at different levels in the

system. Also produce fact sheets and leaflets to inform patients and grass root health workers.

2. Inform PHUs, hospitals, Councils and DHMT directly when money is transferred and provide

details (e.g. period of payment) explain formula for calculating the amounts. Administrators in

Councils should also be informed.

3. Add to the operational manual instructions on financial management in PHUs, using a basic

cashbook as starting point;

4. Simplify the financial reporting procedures for PHUs, e.g. introducing a simple form to be collected

by the DHMT during supervisory visits.

5. Abolish the difference between satisfactory and unsatisfactory entries in the internal verification,

as this does not add value on the whole. Concentrate on overall data quality.24

6. Revise the scoring system for crosscutting issues (-3 or +3) to a simple score, (e.g. 1-5)

7. Introduce new indicators for crosscutting issues, which provide higher challenges of quality25

.

Adjust indicators annually.

8. Create more clarity with regard to incentives and bonuses, e.g. in fact sheets with calculation

tables. Distinguish for DHMT members and Council members clearly travel allowances from PBF

incentives for performing good verification.

9. Introduce an indicator of quality of information system for districts, to be implemented by MOHS

through triangulation of data sources.

10. Clarify the role of the HMC in more detail, and create more conditions for co-management of

health facilities (co-signing account, action plans, etc.).

11. Clarify the role of the national PBF team, the steering committee and institutionalise improved

supervision of DHMT and Council officials.

8.2.4 LONGER-TERM RECOMMENDATIONS

In the coming year the following recommendations could be considered, which require adaptations in

the operational manual, adjustment of agreements between stakeholders and training of persons at

the facility level.

1. Introduce patient satisfaction surveys to be part of payment to PHU (e.g. 20%). The Council might

contract independent Community Based Organisations for that purpose, and ask Civil Society

Organisations to train them. The formats and procedures as used in the EV might be used.

2. Introduce Annual Action Plans in the PHU, which are updated twice per year, to be signed by

PHU, the HMC, the Council and DHMT (which also serve as input for the Council Health Plan)

3. Establish a point at the Councils where patients can complain about healthcare in general (not

through 777) and where they can report when they had to pay for free services.

8.2.5 FUTURE DEVELOPMENTS OF PBF

The following could be considered when further developments of PBF are planned.

1. Increase payment for output indicators (now less than 0,5 USD per capita). This can be possible

when part of regular Government payments to the health system is channelled via PBF systems,

and by including funding from different international donor organisations

2. Include HIV and TB (and maybe other) performance indicators in the PBF system. Consider

channelling Global Fund funds via the PBF programme.

24

NB. Eligibility is another issue! When the indicator is children vaccinated under the age of one, there is no reason to include children over one in the count. Case definitions should be maintained. 25

For example: instead of statistics on wall posters, cumulative graphs of coverage vs. targets; move availability of delivery bed to crosscutting issues and add availability of delivery tray. Instead of minutes of HMC, co-signing HMC on action plan and chair HMC co-signing bank account.



8.2.6 HOSPITAL PBF

The following recommendations for hospital PBF could be considered, especially when new hospitals

are involved in the PBF system.

Within six months:

1. Align the checklists for IV with the hospitals, e.g. their level of service, the medical speciality;

create more flexible custom-made checklists, which recognise specific quality challenges for the

hospital.26

2. Organise a meeting to agree on the definition of certain indicators in checklists (e.g. what is meant

with M&E-plan) between hospital and verifiers before introducing revised checklists.

3. Announce visits and field a trained and qualified team, preferably a team, which performs regular

IVs in the same hospital.

4. Revise the scoring system for indicators (-3 or +3) to a simple score, (e.g. 1-5)

On a longer term:

1. Introduce some kind of flexible ceiling for certain output indicators, relevant for the hospitals27

2. Reconsider reduction of the 60% incentive for personnel and make those more tied to real

performance of individuals

26

For example: introduce hospital neonatal mortality as an indicator, or case fatality rate. 27

E.g. surgical interventions, treatment of cancer patients

contact

P.O. Box 164402500 BK The Hague

Lutherse Burgwal 102512 CB The HagueThe Netherlands

+31(0)70-31 36 [email protected]

more information

Frank van de LooijHealth [email protected] Marjan KruijzenCordaid Liaison Sierra [email protected] Remco van der VeenDirector [email protected]

about cordaid

Cordaid is the Catholic Organisation for Relief and Development Aid, with its headquarters in the Hague and country offices in 11 countries. It has been fighting poverty and exclusion in the world’s most fragile societies and conflict-stricken area’s for a century. It delivers innovative solutions to complex problems by emphasizing sustainability and performance in projects that tackle security and justice, health and economic opportunity. Cordaid is deeply rooted in the Dutch society with more than 300,000 private donors. Cordaid is a founding member of Caritas Internationalis and CIDSE.

external verification performance based financing in ... · pdf fileexternal verification...

Documents