tools for improving the quality of reported data and

Tools for improving the quality of reported data and information
A field manual for implementation
PREVENTIVE CHEMOTHERAPY
Tools for improving the quality of reported data and information
A field manual for implementation
Preventive chemotherapy: tools for improving the quality of reported data and information. A field manual for implementation ISBN 978-92-4-151646-4
© World Health Organization 2019 Some rights reserved. This work is available under the Creative Commons Attribution-NonCommercial-Sha- reAlike 3.0 IGO licence (CC BY-NC-SA 3.0 IGO; https://creativecommons.org/licenses/by-nc-sa/3.0/ igo). Under the terms of this licence, you may copy, redistribute and adapt the work for non-commercial purposes, provided the work is appropriately cited, as indicated below. In any use of this work, there should be no suggestion that WHO endorses any specific organization, products or services. The use of the WHO logo is not permitted. If you adapt the work, then you must license your work under the same or equivalent Creative Commons licence. If you create a translation of this work, you should add the following disclaimer along with the suggested citation: “This translation was not created by the World Health Organization (WHO). WHO is not responsible for the content or accuracy of this translation. The original English edition shall be the binding and authentic edition”. Any mediation relating to disputes arising under the licence shall be conducted in accordance with the mediation rules of the World Intellectual Property Organization. Suggested citation. Preventive chemotherapy: tools for improving the quality of reported data and information. A field manual for implementation. Geneva: World Health Organization; 2019. Licence: CC BY-NC- SA 3.0 IGO. Cataloguing-in-Publication (CIP) data. CIP data are available at http://apps.who.int/iris. Sales, rights and licensing. To purchase WHO publications, see http://apps.who.int/bookorders. To sub- mit requests for commercial use and queries on rights and licensing, see http://www.who.int/about/ licensing. Third-party materials. If you wish to reuse material from this work that is attributed to a third party, such as tables, figures or images, it is your responsibility to determine whether permission is needed for that reuse and to obtain permission from the copyright holder. The risk of claims resulting from infringement of any third- party-owned component in the work rests solely with the user. General disclaimers. The designations employed and the presentation of the material in this publication do not imply the expression of any opinion whatsoever on the part of WHO concerning the legal status of any country, territory, city or area or of its authorities, or concerning the delimitation of its frontiers or boundaries. Dotted and dashed lines on maps represent approximate border lines for which there may not yet be full agreement. The mention of specific companies or of certain manufacturers’ products does not imply that they are endorsed or recommended by WHO in preference to others of a similar nature that are not mentioned. Errors and omissions excepted, the names of proprietary products are distinguished by initial capital letters. All reasonable precautions have been taken by WHO to verify the information contained in this publication. However, the published material is being distributed without warranty of any kind, either expressed or implied. The responsibility for the interpretation and use of the material lies with the reader. In no event shall WHO be liable for damages arising from its use. Printed in France.
Design and layout: Patrick Tissot, WHO Neglected Tropical Diseases. Please consult the WHO Neglected Tropical Diseases website for the most up-to-date version of all documents (www.who.int/neglected_diseases/en)
A FIELD MANUAL FOR IMPLEMENTATION iii
Contents Preface vi
Introduction 1
1. Planning 3 1.1 Preparation and logistics 3 1.2 Selection of the survey area 7 1.3 Selection of the survey population 8 1.4 Calculation of the sample size 11 1.5 Systematic selection of subunits 12
2. Implementation 15 2.1 Creating segments 15 2.2 Selecting households within selected segments 22 2.3 Interviewing selected individuals 24
3. Interpretation and action 26 3.1 Analyse and interpret the results 26 3.2 Convert the results into programmatic action 29
SECTION 2: DATA QUALITY ASSESSMENTS 33
Introduction 33
1. Planning 39 1.1 Preparation and logistics 39 1.2 Review of national programme preventive chemotherapy data management system design 47
2. Implementation 50 2.1 Review of available historical data 50 2.2 Review of availability, completeness and timeliness of reporting 54 2.3 Data verification 56 2.4 Systems assessment through key informant interviews 58
3. Interpretation and action 61 3.1 Develop and operationalize an action plan 61
iv TOOLS FOR IMPROVING THE QUALITY OF REPORTED DATA AND INFORMATION
SECTION 3: SUPERVISORS’ COVERAGE TOOL 65
Introduction 65
1. Planning 66 1.1 Preparation and logistics 66 1.2 Methods 69
2. Implementation 70 2.1 Step 1: Identify the survey population 70 2.2 Step 2: Identify the supervision area to monitor 71 2.3 Step 3: Obtain a list of all households using registers or enumeration 73 2.4 Step 4: Randomly select 20 households 78 2.5 Step 5: Visit each selected household and randomly select one member of the survey population 78 2.6 Step 6: Interview the selected individuals to assess coverage 79
3. Interpretation and action 80 3.1 Step 7: Analyse and interpret the results 80 3.2 Step 8: Develop an action plan to improve or maintain MDA performance 81 3.3 Step 9: Implement the action plan 83
ANNEXES 84
Annex 1: Materials 84 Annex 2: CES statistical formulae and rationale 112 Annex 3: CES interviewing techniques 113 Annex 4: Sampling using PPES 116 Annex 5: Data flow diagram for compiling and reporting MDA data 120 Annex 6: How to use a random number table 121 Annex 7: SCT interview script 123
vi TOOLS FOR IMPROVING THE QUALITY OF REPORTED DATA AND INFORMATION
Preface Preventive chemotherapy is one of the main interventions used by national programmes to control and eliminate five neglected tropical diseases (NTDs): lymphatic filariasis, onchocerciasis, schistosomiasis, soil-transmitted helminthiases and trachoma. The intervention is distributed through mass drug administration (MDA) and school-based treatments, with the goal of treating populations at risk of infection at appropriate, regular intervals. Incorrect tallying or reporting of the numbers of individuals treated can bias routinely reported results, as can poorly documented shifts in population, reliance on outdated census data and treatment of individuals outside the targeted age group or geographical area.
A fundamental step in monitoring the success of programmes is knowing the coverage of preventive chemotherapy; that is, how many people who needed treatment swallowed the medicines. Without reliable information on treatment coverage, programme managers and their staff cannot monitor programmatic performance effectively. Routine monitoring is essential to track progress towards programmatic goals, identify communities in which coverage is low or insufficient and implement corrective actions to improve coverage. Coverage evaluation surveys are straightforward, population-based surveys designed to provide precise estimates of preventive chemotherapy coverage against targeted NTDs and provide a valuable tool for evaluating programmatic performance.
The quality of data on preventive chemotherapy for NTDs received by national programmes is at times incomplete, untimely and of questionable accuracy. As the goals for control and elimination of NTDs, as endorsed by the World Health Assembly and published in the WHO roadmap for implementation, approach, programmes must ensure that high-quality data are available and used effectively for decision-making. Data quality assessments are resources for national programmes to assess the quality of reported data on preventive chemotherapy and the quality of the data management system.
A recurring request from national programmes is the need for a rapid, inexpensive and simple tool to assess coverage of preventive chemotherapy as part of supervision activities. The objective of the Supervisors’ coverage tool is to improve coverage within a given supervision area within the current round of MDA. The advantage of this tool over other monitoring approaches, including independent monitoring and rapid coverage monitoring, is that it provides a classification of the coverage that was likely achieved during the current MDA round.
A FIELD MANUAL FOR IMPLEMENTATION vii
Tools for improving the quality of reported data and information on preventive chemotherapy of neglected tropical diseases
Coverage Evaluation Survey Data Quality Assessment Supervisors’ Coverage Tool
Purpose To validate reported coverage (obtain a statistical point estimate)
To verify reported data and assess capacity of data management and reporting systems
To classify coverage as above/below a threshold
Administrative level Implementation unit (district) National and/or district Supervision area (sub-district)
Sample size > 500 people N/A 20 people
Sites visited 30 villages 12 service delivery points 20 villages
Survey team External to programme Internal and external to programme Internal, self-assessment
Timing Within 6 months of MDA (longest within 12 months)
Every 3 years nationally Rotate every year in some districts
Within 2 weeks of MDA
Cost (US$) ~ US$ 2000–5000 per district US$ 12 000–15 000 nationally US$ 1000–2500 per district
~ US$ 250–1000 per supervison area
Duration 2–3 weeks ~ 2 weeks <1 week
MDA, mass drug administration; N/A, not applicable.
This field manual presents the three tools and their methodologies, with stepwise explanations on planning, implementation, and interpretation and action, to guide improvements in the quality of reported data and information on preventive chemotherapy for NTDs. Additional resources are available to accompany the guides.
Below is a tabulated summary description of the three tools.
viii TOOLS FOR IMPROVING THE QUALITY OF REPORTED DATA AND INFORMATION
Acknowledgements The World Health Organization (WHO) is grateful to all those who helped to prepare these tools for improving the quality of reported data and information on preventive chemotherapy of neglected tropical diseases (NTDs). Particular thanks are due to the following individuals and organizations for their valuable contributions to the methodology and field testing in national programmes.
Coverage Evaluation Surveys: Katherine Gass (Neglected Tropical Diseases Support Center, Task Force for Global Health, Decatur, GA, USA), Michael Deming (formerly United States Centers for Disease Control and Prevention; currently Neglected Tropical Diseases Support Center, Decatur, GA, USA), Ralph DiGaetano (WESTAT, Rockville, MD, USA), Pamela Sabina Mbabazi (WHO, Geneva, Switzerland), Kathryn Zoerhoff (RTI ENVISION Project, Washington, DC, USA); Ralph Henderson (formerly WHO, Atlanta, GA, USA and the RTI ENVISION Project, Washington, DC, USA); Square Mkwanda (Ministry of Health, Lilongwe, Malawi), Wintare Roland Bougma (Programme National de Lutte contre les Maladies Tropicales Négligées, Ministère de la Santé, Ouagadougou, Burkina Faso), Francois Drabo (Programme National de Lutte contre les Maladies Tropicales Négligées, Ministère de la Santé, Ouagadougou, Burkina Faso), Kristen Renneker (Neglected Tropical Diseases Support Center, Task Force for Global Health, Decatur, GA, USA), Abdel Direny (RTI ENVISION Project, Washington, DC, USA), Edridah Muheki Tukahebwa (Vector Control Division, Ministry of Health, Kampala, Uganda), Harriet Lwanga (RTI ENVISION Project, Kampala, Uganda), Rosa Elena Mejia (Pan American Health Organization, Tegucigalpa, Honduras), Reina Theresa Velasquez (Neglected Infectious Disease Division, Secretary of Health, Tegucigalpa, Honduras), Laura Catala (Neglected Infectious Diseases, Pan American Health Organization, Washington, DC, USA), Ana Morice (consultant for the Pan American Health Organization, San José, Costa Rica), Sitti Ganefa (Sub-Directorate of Filariasis and Helminthiasis Control, Ministry of Health, Jakarta, Indonesia), Wita Larasati (RTI ENVISION Project, Jakarta, Indonesia), Brian Fuller (RTI ENVISION Project, Washington, DC, USA) and Molly Brady (RTI ENVISION Project, Washington, DC, USA).
Data Quality Assessments: Kathryn Zoerhoff (RTI ENVISION Project, Washington, DC, USA); Timothy Wakabi (Consultant, Kampala, Uganda); Pamela Sabina Mbabazi (WHO, Geneva, Switzerland); Harriet Lwanga (formerly with RTI ENVISION Project, Kampala, Uganda); Kalpana Bhandari (RTI ENVISION Project, Washington, DC, USA); Hannah Frawley (RTI ENVISION Project, Washington DC, USA); Kaleigh Robinson (RTI ENVISION Project, Washington, DC, USA); the Ministries of Health in Benin, Cameroon, Cote d’Ivoire, Ethiopia, Guinea-Bissau, Haiti, Indonesia, Mozambique, Nepal, Nigeria, Philippines, Senegal, South Sudan, Uganda and United Republic of Tanzania; USAID ENVISION Project field teams in Benin (RTI International), Cameroon (Helen Keller International, Sightsavers, Perspective, IEF), Ethiopia (RTI International), Haiti (IMA World Health), Indonesia (RTI International), Mozambique (RTI International), Nepal (RTI International), Nigeria (RTI International and The Carter Center), Philippines (RTI International), Senegal (RTI International), United Republic of Tanzania (IMA World Health), Uganda (RTI International); Sightsavers field teams in Côte d’Ivoire, Guinea- Bissau, and South Sudan; and the WHO Regional Office for Africa, Brazzaville, Congo.
A FIELD MANUAL FOR IMPLEMENTATION ix
The preparation of this guide and the accompanying resources was informed by the DQA tools developed for programmes on HIV, tuberculosis and malaria as well as on immunization by partners including MEASURE Evaluation; the Global Fund to Fight AIDS, Tuberculosis and Malaria; the President’s Emergency Plan for AIDS Relief; the United States Agency for International Development; WHO; the Joint United Nations Programme on HIV/AIDS; the Global Alliance for Vaccines and Immunization; and WHO’s Immunization Information Systems and Data Assessment. Gratitude is also extended to Iota Ink for their contributions to updating the DQA tool. USAID funding for the development and field-testing of these materials in many countries through the RTI ENVISION project is gratefully acknowledged.
Supervisors’ Coverage Tool: Pamela Sabina Mbabazi (WHO, Geneva, Switzerland), Katherine Gass (Neglected Tropical Diseases Support Center, Task Force for Global Health, Decatur, GA, USA), Michael Deming (formerly United States Centers for Disease Control and Prevention; currently Neglected Tropical Diseases Support Center, Decatur, GA, USA), Ifeoma Anagbogu (Federal Ministry of Health, Abuja, Nigeria), Uzoma Nwankwo (Federal Ministry of Health, Abuja, Nigeria), Scott McPherson (RTI ENVISION Project, Addis Ababa, Ethiopia), Biruck Kebede (Federal Ministry of Health, Addis Ababa, Ethiopia), Asfaw Kejella (Benishangul-Gumuz Regional Health Bureau, Assosa, Ethiopia), Leda Hernandez (Department of Health, Manila, Philippines) and Winston Palasi (Department of Health, Manila, Philippines); Margaret Baker (RTI ENVISION Project, Washington, DC, USA), Hannah Betts (Liverpool School of Tropical Medicine, Liverpool, United Kingdom), Paul Byatta (Evidence Action, Washington, DC, USA), Michelle Clements (Schistosomiasis Control Initiative, Imperial College London, United Kingdom), Timothy Finn (Sightsavers, Chippenham, United Kingdom), Fiona Fleming (Schistosomiasis Control Initiative, Imperial College London, United Kingdom), Hannah Frawley (RTI ENVISION Project, Washington, DC, USA), Louise Kelly-Hope (Liverpool School of Tropical Medicine, Liverpool, United Kingdom), Hayley Mableson (Liverpool School of Tropical Medicine, Liverpool, United Kingdom), Egide Ndayishimye (FHI 360, Accra, Ghana), Jamie Tallant (The End Fund, New York City, NY, USA) and Kathryn Zoerhoff (RTI ENVISION Project, Washington, DC, USA).
x TOOLS FOR IMPROVING THE QUALITY OF REPORTED DATA AND INFORMATION
Abbreviations and acronyms CDD community drug distributor CES coverage evaluation survey CSAT coverage survey analysis tool CSB coverage survey builder DQA data quality assessment DEFF design effect EA census enumeration area HH household IAL intermediate data aggregation level IU implementation unit KAP knowledge, attitudes and practices MDA mass drug administration NTD neglected tropical disease
PPES probability proportionate to estimated size PSS probability sampling with segmentation PSU primary sampling unit SA supervision area SAC school-age children SCT supervisors’ coverage tool SDP service delivery point SOP standard operating procedure SSU secondary sampling unit TSU tertiary sampling unit USAID United States Agency for International Development VF verification factor WHO World Health Organization
A FIELD MANUAL FOR IMPLEMENTATION xi
Glossary The definitions given below apply to the terms as used in this field manual. They may have different meanings in other contexts.
assessment period The period of time in which the round of preventive chemotherapy under assessment was conducted; ideally, the assessment period should be determined by the ministry of health that commissions the data quality assessment (DQA).
census enumeration area (EA) The smallest geographical area for which census population results are available.
community drug distributor (CDD) A volunteer who is frequently utilized by neglected tropical disease programmes to deliver preventive chemotherapy to individuals in their community as a part of mass drug administration.
coverage evaluation survey (CES) A population-based probability survey designed to provide an estimate of preventive chemotherapy coverage, which meets precision needs and avoids the biases and some of the errors that can affect reported coverage. A CES differs from other tools used to monitor coverage (e.g. coverage supervision tool, rapid coverage monitoring, or independent monitoring).
coverage survey analysis tool (CSAT) A Microsoft CSV-based tool designed to analyse survey results and auto-generate a standardized report. It is downloadable at: https://coverage.securedatakit.com
coverage survey builder (CSB) A Microsoft Excel-based tool designed to aid survey coordinators with the planning and implementation of a coverage evaluation survey. It is downloadable at: http://www. ntdsupport.org/resources/coverage-survey-builder-coverage-evaluations
data verification A quantitative comparison of recounted to reported data that assesses whether data are being collected and reported accurately at every level in the data reporting system.
drug package A combination of preventive chemotherapy medicines (drugs), which are given out together to treat neglected tropical diseases.
eligible population The population targeted for treatment with mass drug administration, based on drug- specific eligibility criteria.
xii TOOLS FOR IMPROVING THE QUALITY OF REPORTED DATA AND INFORMATION
epidemiological coverage The proportion of individuals in the survey area who swallowed the medicine, or combination of medicines, out of the total population in the survey area, regardless of eligibility for treatment.
household (HH) A group of people who eat and live together.
implementation unit (IU) The administrative unit in a country that is used as the basis for making decisions about implementing mass drug administration; the IU is usually a district.
intermediate data aggregation level (IAL) Administrative levels that are lower than the national level but higher than the community drug distribution point, at which data on preventive chemotherapy are aggregated; the number of intermediate aggregation levels may vary between countries. The DQA tool provides for up to four intermediate levels: intermediate level 1 represents the next level after the community, followed by level 2, etc. For example, data may be sent from villages (service delivery points) to a health facility where the data are first aggregated (intermediate aggregation level [IAL 1)], then to a district (IAL 2), followed by a region (IAL 3); then data are sent to the national level.
interviewer The person in charge of questioning the respondents and filling out the questionnaire.
mass drug administration (MDA) A method of distributing preventive chemotherapy in which medicines are administered to the entire population of an area (e.g. state, region, province, district, sub-district, village) at regular intervals, regardless of individual infection status.
mop-up Localized mass administration of medicines that is repeated immediately after the original mass administration in areas where the coverage of preventive chemotherapy is found to be inadequate.
national level The highest administrative level at which data on treatment and stocks of medicine are aggregated for the entire country; at this level, programme managers, the national secretariat and other partners review the data and make decisions.
preventive chemotherapy The use of essential anthelminthic medicines (or, for trachoma, an antibiotic), alone or in combination, as a public health tool to control or eliminate neglected tropical diseases. In preventive chemotherapy, all individuals in endemic communities or areas are treated regardless of infection status. It is commonly delivered through mass community- and school-based distributions.
preventive chemotherapy coverage A general term encompassing the various types of coverage (geographical, national, epidemiological, reported) that programmes may calculate and report.
A FIELD MANUAL FOR IMPLEMENTATION xiii
programme reach The proportion of people in the survey area who were offered the opportunity to receive preventive chemotherapy, regardless of whether the medicine was ingested.
reported (or administrative) coverage The coverage calculated from the data reported by all community drug distributors; census figures or previous reports from drug distributors are used to estimate the population denominator.
report availability The percentage of source documents or reports that can be retrieved.
report completeness The percentage of source documents or reports that contain all the required data for indicators.
report timeliness The percentage of source documents or reports that were compiled or submitted by the due date.
sampling unit Administrative geographical areas in which service delivery points are located and where data are tabulated and aggregated. During the DQA, some of these areas are selected as part of a sample where the assessment will take place. Depending on the number of administrative levels in a country, sampling units may be divided into primary sampling units (PSU), secondary sampling units (SSU), and tertiary sampling units (TSU).
segment A grouping of households from within the initially sampled subunit; used for sampling efficiency.
service delivery point (SDP) The delivery of services to prevent neglected tropical diseases may include the following interventions, among others: preventive chemotherapy (the SAFE strategy for trachoma), morbidity management and disability prevention, and/or treatment of cases. The service delivery point is the lowest administrative level, school or fixed point where an intervention benefiting a population (i.e. service delivery) occurs. For diseases amenable to preventive chemotherapy, these points are typically communities, villages, or schools where the intervention has taken place and where treatment data are compiled from the treatment registers or tally sheets by community drug distributors, teachers or health workers.For the DQA, this is equivalent to “peripheral treatment” sites.
source documents Data collection tools where service delivery is first recorded. For neglected tropical diseases amenable to preventive chemotherapy, these tools may include treatment registers, tally sheets on preventive chemotherapy and inventory records at the distribution level. For other diseases, they may include patient records.
xiv TOOLS FOR IMPROVING THE QUALITY OF REPORTED DATA AND INFORMATION
sub-district data aggregation level
The administrative level directly below the district level for which data on preventive chemotherapy are aggregated. The number of sub-district aggregation levels may vary among countries; in very small districts there may be no aggregation of sub-district data.
subunit The smallest administrative unit for which population estimates are available; may correspond to a census enumeration area, village, hamlet or locality.
supervision area (SA) The geographical area in which the supervisors’ coverage tool is conducted. Typically, the supervision area corresponds to the smallest administrative or geographical unit for which a first-level supervisor is responsible.
supervisors’ coverage tool (SCT) implementer The individual in charge of conducting the supervisors’ coverage tool within a given supervision area; usually a supervisor of mass drug administration at sub-district level.
survey area The administrative geographical area in which mass drug administration is conducted and for which data on preventive chemotherapy coverage are tabulated and reported. For many neglected tropical diseases, this is the district or implementation unit.
survey coordinator The person, often from the central level, who plans and oversees a coverage evaluation survey.
surveyed coverage Coverage measured through the use of population-based survey sampling methods. It is calculated as a fraction. The denominator is the total number of individuals surveyed and the numerator is the total number of individuals surveyed who were identified as having ingested the medicine.
survey population The population for which a statistical estimate of preventive chemotherapy coverage is desired.
systems assessment A qualitative assessment of the strengths and weaknesses of the data management and reporting system, assessed through key informant interviews conducted at every level in the data reporting system.
target coverage threshold Disease-specific thresholds above which treatment levels are considered to be effective for achieving programme goals/public health goals.
target segment size The average number of households per segment.
verification factor (VF) The ratio of recounted value of the indicator to the reported value; the VF is used to measure the degree of accuracy of reported data.
xvi TOOLS FOR IMPROVING THE QUALITY OF REPORTED DATA AND INFORMATION
COVERAGE EVALUATION SURVEYS (CES)
Introduction
Typically, monitoring of coverage relies on routinely reported data which are aggregated from the records of community drug distributors. While reported coverage is an essential tool for monitoring programmes’ performance, it is prone to errors resulting from incorrect estimates of the target population, weak health information systems, underreporting and intentional inflation of the numbers of people treated2.
Uses of preventive chemotherapy coverage surveys
Coverage evaluation surveys are a valuable tool for evaluating programme performance. Such population-based surveys are designed to provide precise estimates of coverage while overcoming many of the biases and errors that can undermine reported coverage.
While typically regarded as a tool for estimating preventive chemotherapy coverage, the benefits of and uses for coverage evaluation surveys extend beyond the estimation of treatment levels attained. Other potential uses include:
To validate reported coverage rates: The results of coverage surveys can be used to check the accuracy of the system for recording and reporting data and take corrective actions where necessary.
To identify reasons for non-compliance: The common reasons for not swallowing the medicines can be identified, allowing country programmes to improve social mobilization before the next round of MDA.
To detect problems with the supply chain and distribution systems: Coverage surveys can identify groups of individuals for whom the medicines were never offered and where corrective action can be taken.
To measure coverage in specific populations: Survey tools can be used to measure coverage levels in sub-populations (e.g. by age, sex, and rural vs urban areas).
To provide an opportunity to measure other population attributes: An investigation of additional issues (e.g. knowledge, attitudes and practices towards the intervention; the prevalence of morbidity and the performance of community drug distributors) can generate valuable information for improving programme performance.
1. Accelerating work to overcome the global impact of neglected tropical diseases: a roadmap for implementation. Geneva: World Health Organization; 2012 (http://www.who.int/neglected_diseases/NTD_RoadMap_2012_ Fullversion.pdf, accessed february 2019).
2. Murray CJ, Shengelia B, Gupta N, Moussavi S, Tandon A, Thieren M. Validity of reported vaccination coverage in 45 countries. Lancet. 2003;362:1022–7.
2 TOOLS FOR IMPROVING THE QUALITY OF REPORTED DATA AND INFORMATION
Survey area
e.g. district
(30 chosen using PPES)
each chosen subunit
Individuals
selected households
PPES, probability proportionate to estimated size sampling
1. Multiple indicator cluster survey. New York (NY): United Nations Children’s Fund (http://mics.unicef.org/tools, accessed February 2019).
2. Turner AG, Magnani RJ, Shuaib M. A not quite as quick but much cleaner alternative to the Expanded Programme on Immunization (EPI) Cluster Survey design. Int J Epidemiol. 1996;25:198–203. doi.org/10.1093/ ije/25.1.198.
3. Report of the WHO Strategic and Technical Advisory Group for Neglected Tropical Diseases. Geneva, 12–13 April 2016. Geneva: World Health Organization; 2016 (http://www.who.int/neglected_diseases/ NTD_STAG_report_2016.pdf, accessed February 2019).
Throughout this guide an important distinction is made between two populations: the eligible population and the survey population.
The eligible population is the population targeted for MDA treatment, based on drug- specific eligibility criteria.
The survey population is the population for which an estimate of preventive chemotherapy coverage is desired. This may differ from the eligible population; for some NTDs the survey population will be larger than the eligible population.
For example, for lymphatic filariasis the eligible population to receive albendazole and diethylcarbamazine is all individuals aged ≥ 2 years, excluding women who are pregnant or in the first week of breastfeeding and the extremely ill. When evaluating coverage, the survey population for lymphatic filariasis is everyone who was living in the survey area at the time of the MDA, regardless of their eligibility for treatment.
Sampling overview
In this survey design, subunits (e.g. enumeration areas, villages) are chosen with probability proportionate to estimated size (PPES) in segments, where a segment represents a group of approximately 50 households. In the field, the subunits selected for the survey sample are divided into this predetermined number of segments, such that the segments are of approximately the same size in households. Then, one segment is selected at random and a fixed proportion of households is selected systematically from among the households in segments selected for the survey sample. This coverage evaluation survey design produces an equal probability of samples of the survey population (Annex 2 describes how to calculate selection probabilities using the coverage evaluation survey design). It is derived from the “modified segment design” option described in the manuals for UNICEF’s Multiple-Indicator Cluster Surveys 2–41 and by Turner et al. in 1996.2
The sampling methodology described in this guide was endorsed by WHO’s Strategic and Technical Advisory Group for Neglected Tropical Diseases in 2016.3
Implementing the survey methodology will result in an equal-probability sample of the survey population.
M O
D U
LE 1
Where and when should coverage evaluation surveys be conducted?
Coverage surveys are an integral part of evaluating the performance of NTD programmes. National programmes should be encouraged to conduct coverage surveys throughout the implementation of a programme, as part of routine monitoring of programmes’ performance, and to ensure that the target coverage thresholds are being reached and that the reporting system is functioning well. When coverage surveys are used as a part of routine programme performance monitoring, it is recommended that the survey areas (e.g. districts) be selected randomly, for example by drawing names from a hat. The number of coverage surveys conducted will be dictated by the resources available. Coverage surveys will be most informative to national programmes if they are conducted at various timepoints in the programme’s lifecycle and across different geographical areas.
In addition to routine integration of coverage surveys, there are certain circumstances in which coverage surveys may be indicated. For example, coverage surveys can be a useful tool to understand the reasons for low coverage or compliance with treatment, to estimate coverage when population figures (i.e. the denominator) are uncertain and to investigate any reasons for greater than expected levels of infection or morbidity. In such instances, the survey area (an implementation unit, such as a district) from which to sample should be selected purposefully, based on the areas experiencing the particular challenge. These issues are summarized in Table 1.1.
Table 1.1 Key considerations for determining when and where to conduct coverage surveys
Main purpose of the coverage survey Method used to select the survey area (e.g. district) Preferable timing in the programme
1. To check if the data reporting system is working well Random Early (first and second years of the programme), repeated in later years if early results show a large discrepancy
2. To better estimate coverage where there is reason to believe that routine reporting is incorrect: • in areas where the denominator (total population
figure) is suspected to be grossly incorrect or out of date
• where reported coverage is highly suspect • where incidence of morbidity is ongoing • where a transmission assessment survey or impact
assessment survey is failing
Purposeful or random, depending on the number of areas in which the problem is suspected
As need arises
3. To evaluate programmatic progress Random Mid-term evaluation of national action plan (third and fourth years of programme)
4. To integrate a KAP survey with a coverage survey in order to: • identify the reasons for low coverage
• test the effectiveness of social mobilization and communication strategies
Purposeful, if testing specific areas of low coverage
Random, if testing across the programme
Following MDA with reported low coverage
Following a new or change in social mobilization strategy
KAP, knowledge, attitudes and practices; MDA, mass drug administration
When should surveys be timed relative to the MDA?
It is recommended that coverage surveys be implemented as soon after the MDA as possible to minimize recall bias, ideally after the reported coverage from the last MDA has been calculated and before planning for the next MDA has begun; typically no later than 3–6 months after the MDA.
The following are suggested timelines that can be used to help with planning and budgeting, depending on the size of the country.
Planning for a coverage survey: 4–6 weeks
Implementing a coverage survey: 2–3 weeks
Analysis, interpreting and reporting: 1 week
Acting upon the coverage survey results to improve the next MDA: 4–6 weeks
M O
D U
LE 1
Who should conduct the survey?
A survey team should conduct the coverage evaluation survey. The teams may be comprised of staff at national, regional/provincial or district levels who are not directly engaged in the MDA. For the sake of transparency, it is often preferable to have multiple organizations or levels within the health system represented on the survey teams. Some countries choose to commission a local university or institute to implement the survey. Typically, each survey team has two members: at least one interviewer who speaks the local language and is in charge of questioning the respondents and filling out the questionnaire; one sampler, who ensures that the sampling plan is adhered to and determines which households are selected; and a driver who transports the team to each survey site. Sometimes the same driver may transport multiple teams. It is helpful if one or more of the team members is familiar with the local area. It is advisable to request the local community/village leader to provide a local guide to accompany the team while working in the area. A survey coordinator, typically from the central level, is needed to lead the survey planning and to supervise the teams in the field.
What materials are required?
The materials required for planning, implementing and actioning a CES are listed in Annex 1. National programmes should ensure that these materials are available in appropriate quantities before the planning phase.
What training is needed?
The survey teams should be trained immediately before the coverage survey is implemented to ensure that the information learnt is retained. A suggested outline for training is as follows:
Day 1 – Conduct in-class training on the segmentation approach, selection of households and practise with completing the questionnaire. Interviewing techniques (Annex 3) should be practised repeatedly in order to minimize errors and maximize efficiency of time spent in each household. Additional training should be provided for other optional questionnaires that may be included in the survey. Annex 1 (Appendix 4) provides an example of such a questionnaire for a knowledge, attitudes and practices (KAP) survey.
Day 2 – Practise the approach in one or two field sites (sites selected for field practice should not include any of the 30 sites selected for the actual survey).
Day 3 (full or half day) – Discuss experiences from Day 2, review the survey methods, and plan for the field work.
How much does a survey cost?
The cost of a typical coverage survey, including the cost of training, can vary substantially by country as well as within countries according to the size and density of the survey area. Previous surveys using other methods were reported to cost US$ 2000–10 000 per survey area or district to implement. Comparative operational research studies that compared three survey methods in four countries reported costs ranging from US$ 1867 to US$ 4816 (Table 1.2). Following consultations and review of several cost data sources during a technical meeting with monitoring and evaluation officers from NTD implementing agencies, it is suggested that coverage evaluation surveys using the recommended probability sampling with segmentation (PSS) method be budgeted at a cost of US$ 2500–3500 per district as part of planned programme activities.
How should the coverage survey builder be used?
To help survey planners design a coverage evaluation survey, an Excel-based tool called the Coverage Survey Builder (CSB) was developed. This tool can help survey planners to determine a sample size appropriate for the survey, select subunits, segment and systematically select households within the selected subunits and compile results. The CSB is available for download1 and is discussed further in sections 1.4, 1.5 and 2.1.
Table 1.2. Comparative cost estimates of three most common survey methods
WHO region
Survey method
Expanded Programme on Immunization (EPI)
Lot quality assurance sampling (LQAS)
Probability sampling with segmentation (PSS)
No. of days to complete Cost (US$) No. of days to
complete Cost (US$) No. of days to complete Cost (US$)
Country A (Africa) 18 4385 19 4816 17 4525
Country B1 (Americas) 22 1867 9 1167 18 1520
Country C (Africa) 14 4113 10 3247 16 4546
Country D (Africa) 23 4040 21 3835 26 4535
Average 19.25 3601 14.75 3266 19.25 3782
1 Surveys implemented using established ministry of health structures, which contributed to significantly reducing costs in this country.
M O
D U
LE 1
A FIELD MANUAL FOR IMPLEMENTATION 7
1.2 Selection of the survey area
Coverage evaluation surveys should be conducted at the administrative geographical area in which the MDA is conducted and for which data on the coverage of preventive chemotherapy are collated and reported. It is referred to in this manual as the survey area. For many NTDs, this is typically a district or implementation unit.
School-based distribution. School-age children (5–14 years) are frequently treated through school because the use of existing school infrastructure makes treatment more efficient and reduces distribution costs; however, it is important to recognize that the population of school-age children requiring treatment is not limited to children who attend school. Coverage evaluation surveys should include all school-age children – both in and out of school. To achieve this aim, coverage surveys evaluating school- based distributions should be conducted at the community level so that all children can be reached. The survey area for school-based distributions should correspond to the administrative level at which school-based treatments are aggregated and reported. This will enable programme managers to assess whether the target coverage threshold has been met and to validate the reported coverage. Note that there may be instances in which a school-based coverage survey is desirable. This guide does not apply in such instances.
1.3 Selection of the survey population
Before beginning a coverage survey the survey population that will be suitable for inclusion in the coverage survey must be specified.
Lymphatic filariasis, onchocerciasis and trachoma
For lymphatic filariasis, onchocerciasis and trachoma the survey population will be the same as the total population in the survey area. This corresponds to WHO’s definition of epidemiological coverage1, which reflects the proportion of the at-risk population that is covered by MDA. This means that everybody living in the survey area, regardless of their eligibility for treatment, is suitable for inclusion in the survey. For example, coverage surveys for onchocerciasis should include children aged < 5 years, even though they are ineligible to receive ivermectin.
Schistosomiasis and soil-transmitted helminthiases
For schistosomiasis and soil-transmitted helminthiases the survey population may vary, based on the treatment priorities and goals of the national programme. The population most commonly targeted for treatment with preventive chemotherapy is school-age children (aged 5–14 years); however, in some countries preschool-age children, women of childbearing age, or everyone living in high-risk areas may also be targeted. In areas where lymphatic filariasis is coendemic, the entire population is automatically treated for soil-transmitted helminthiases. Consequently, the decision about which survey population to use should be based on the population for which an estimate of coverage is desired. For example, if the national programme wishes to evaluate the coverage of a new deworming programme targeting preschool-age children, then the survey population would be preschool-age children (i.e. children aged 1–4 years).
Integrated coverage assessments
When a combination of drug packages is delivered to the population (either co- administered or independently administered during the same year) then an integrated coverage evaluation survey may be the most efficient way to assess the coverage of all relevant drug packages for treatment of NTDs amenable to preventive chemotherapy. For integrated assessments, it is important to clearly define the survey population for each drug package evaluated in the survey in advance of it. To avoid complications, it is recommended that no more than two survey populations be assessed through a single survey (although multiple drug packages may be assessed within the same survey population).
1. Monitoring and epidemiological assessment of mass drug administration in the global programme to eliminate lymphatic filariasis: a manual for national elimination programmes. Geneva: World Health Organization; 2011 (WHO/HTM/NTD/PCT/2013.9; http://www.who.int/lymphatic_filariasis/resources/9789241501484/ en/, accessed February 2019).
M O
D U
LE 1
Target coverage thresholds
To achieve WHO’s 2020 goals for control and elimination of NTDs treated through preventive chemotherapy, high levels of coverage must be sustained in the population where the diseases are endemic. For each of these five diseases, WHO has defined minimum target coverage thresholds to indicate sufficiently high coverage with preventive chemotherapy of the population at risk (Table 1.3).
Determining the expected survey population per household
Once the survey population has been defined, the average number of individuals in the survey population who are expected to be living in each household must be determined. If the survey population is equivalent to the entire population (e.g. for lymphatic filariasis, onchocerciasis and trachoma), this is the same as establishing the average household size. If the survey population is a subset of the total population (e.g. preschool-age children), information on the population structure by age group will be required, and is typically available from the most recent census or Demographic and Health Survey.
Table 1.3 Drug packages for preventive chemotherapy of neglected tropical diseases and WHO- defined target coverage thresholds for the survey population
Disease Drug package Survey populationa Target coverage thresholdb
Lymphatic filariasis Albendazole + ivermectin Albendazole + DEC
Everybody living in the survey area (e.g. district) ≥ 65%
Onchocerciasis Ivermectin Everybody living in the survey area (e.g. endemic focus, district)
≥ 80%c
Schistosomiasis Praziquantel The survey population may vary, based on national treatment priorities and could include one or more of
the following groups:e • school-age children (5–14 years) • high-risk adults
75%d
Soil-transmitted helminthiases
Albendazole or mebendazole School-age children (5–14 years) The survey population may vary, based on national
treatment priorities and could include one or more of the following groups:
• preschool-age children (1–4 years) • school-age children (5–14 years) • women of childbearing age • everybody living in the survey
area at the time of MDA
75%
Everybody living in the survey area (e.g. district) 80%
DEC, diethylcarbamazine; MDA, mass drug administration
a Corresponds to the population of interest for the coverage survey and that should be eligible for inclusion; the population for which an estimate of preventive chemotherapy coverage is desired. b Corresponds to the target “epidemiological coverage” threshold of the population. c Threshold applies when the goal is elimination of ocular morbidity caused by onchocerciasis. d WHO specifies a target coverage threshold of 75% for school-age children. e Preschool-age children are currently ineligible to receive praziquantel through MDA.
Additional calculations may be necessary if the age group reported in the census does not correspond to the coverage survey population. In general, the calculations for doing this are as follows:
Expected no. of people in the coverage population per
household = (% population in age group) x (fraction age group in coverage population)
x (average household size)
Example
Suppose the survey population for your survey is preschool-age children (defined as 1–4 years old). The census reports that 13.75% of the population is aged 0–4 years and that the average household size is 5.4 people. In this case, the survey population represents 80% of the census age group (that is, the survey population aged 1–4 spans 4 years, while the 0–4 census age group spans 5 years). The expected number of people in the survey population per household can be calculated as:
(13.75%) x (80%) x (5.4) = 0.6 people aged 1–4 years per household = Expected no. of people in the
coverage population per household
If you are conducting an integrated assessment with two survey populations you will need to calculate the expected number of people per household for each of the survey populations. When entering this information into the CSB tool, the population with the smaller expected number of people per household will be considered survey population 1. The population with the larger expected number of people per household will be survey population 2.
M O
D U
LE 1
1.4 Calculation of the sample size
Box 1.1 outlines the factors to consider when determining the sample size for the survey. These calculations can be performed automatically using the CSB.
Box 1.1 Sample size calculations
Step 1: Expected coverage (p). The proportion of the population that you expect will have swallowed the drug (expected coverage sample size will increase as the reported coverage approaches 50%). To help ensure that the sample size is sufficient to meet study objectives, it is recommended that at least 15 percentage points be subtracted from the reported programme coverage. For example, if reported coverage is 85%, it would be more conservative and ensure a greater sample size to subtract 15 percentage points and assume that the expected coverage figure is 70%. If after subtracting 15 percentage points the reported coverage is < 50%, then 50% should be used as the expected value. If you are conducting an integrated coverage assessment, the lowest of expected drug package coverage rates should be used. Suggested default: 50%
Step 2: Desired precision (δ). The desired precision measure considered here is half the width of a 95% confidence interval around the coverage estimate. For example, a measure of precision of 5 percentage points around a coverage estimate corresponds to a confidence interval of +/−5%. Suggested default: 5%
Step 3: Design effect (DEFF). The design effect is a measure that reflects the degree to which respondents in the same subunit are likely to be similar in terms of the information provided in response to a survey question. A design effect of 1.0 indicates that the use of cluster sampling (sampling people from select subunits) makes no contribution to the variability of the estimate. If possible, assumptions about the size of the design effect should be based on the experience of previous surveys. Otherwise, values between 2 and 4 are recommended. Suggested default: 4
Step 4: Alpha (α). An alpha value corresponds to the significance level associated with a confidence interval. When considering a single hypothesis test, choosing an alpha value of 0.05 means that, even if there is only random variation in the data, one is willing to mistakenly draw the conclusion that there is information in the data 5% of the time. Selecting alpha=0.05 corresponds to a 95% confidence interval (Zα/2=1.96). If the coverage survey were repeated multiple times and 95% confidence intervals calculated each time, then 95% of these intervals would be expected to contain the true coverage. Suggested default: 0.05
Step 5: Non-response rate (r). The percentage of members of the survey population sampled for the survey but for whom data were not obtained due to absenteeism, refusal, or other reason. Values of 10–20% are recommended. Suggested default: 10%
These responses can be used to generate the sample size to be targeted for your survey using the equation below (calculated automatically in the CSB):
n = (DEFF)(Z2
Example Using the default values the sample size would be:
1707 = (4)(1.962
(∝/2))(0.5) (0.5)
0.052 (1-0.1)
1.5 Systematic selection of subunits
For this protocol, 30 subunits should be randomly selected from among all those within the survey area. Subunits should be administrative areas for which population figures are available. The ideal subunits are census enumeration areas (EAs), although villages, wards, localities or any other small administrative unit may also be used. Although EAs may require greater effort initially than other choices of subunit, the use of EAs for survey sampling is advantageous because:
they account for all households in a country in a jointly exhaustive and mutually exclusive fashion (i.e. each household in the country will fall into one and only one EA);
they are among the smaller geographical units available, sometimes much smaller than villages and large towns, so they are easier to work in than larger units;
they can be used in urban areas as well as rural and semi-urban areas; and
outline maps for each EA are often available,1 which can help to create segments (see section 2.1).
The term “subunits” is used when referring to the first units of sample selection, be they EAs/villages/localities/etc. See the box below for a summary of this information.
Enumeration area Village/ward/locality/hamlet/etc.
Designed to be comprehensive: jointly exhaustive and mutually exclusive A household can belong to multiple villages or no village
Relatively consistent, small population size Population size can vary widely
Can be used in rural, semi-urban and urban areas Challenging to use in urban centres
Maps are available, typically from the same governmental office that conducts the census
Maps may not be available
To improve the efficiency of household sampling, a segment of households will be randomly selected within each selected subunit. Sampling of households will take place only within that segment. The probability that any one subunit is selected is proportionate to the estimated number of segments it contains. The default segment size is 50 households; however, sometimes larger segments may be necessary if the expected number of households required to meet the sample size exceeds 50 (segmentation is discussed in detail in section 2.1). Box 1.2 gives stepwise directions for selecting the subunits.
1. Maps (also referred to as “sketch maps”) are typically available from the same governmental office that conducts the national census. Both the Demographic and Health Survey and UNICEF’s Multiple Indicator Cluster Survey routinely use EAs as the primary sampling unit.
M O
D U
LE 1
Box 1.2 Systematic selection of subunits using PPES sampling
The following steps should be completed during the planning phase using the Coverage Survey Builder (CSB) tool (http:// www.ntdsupport.org/resources/coverage-survey-builder-coverage-evaluations) to select the subunits from which the 30 segments will be chosen.
Step 1: Obtain a list of all subunits in the survey area and their estimated size. This list should be exhaustive for the study area. A visit to the government office in charge of the census may be necessary to request a list of all subunits and their population (ideally in terms of the number of households [HH]). If projected growth rates are available, they should be taken into account when determining the projected population of each subunit. Make sure the subunits are listed according to geographical order, and not alphabetical order.
Step 2: Combine small subunits. Any subunit with < 25 households should be merged with an adjoining subunit on the list – which should also correspond to a subunit in close geographical proximity – to form one single subunit for sampling purposes. List these two combined subunits together on a single line in your spreadsheet (note: this is most easily performed using Microsoft Excel). For example, if Subunit A has 22 households, it should be combined with Subunit B, the next subunit on the list that is geographically contiguous. Supposing Subunit B has 128 households. The combined subunit should be listed in a single row as “Subunits A and B” with 150 households (128+22).
Step 3: Divide large subunits. Additionally, it is recommended, although not required, that any subunit with > 400 households be subdivided if possible and listed on separate lines, in order to make segmentation more manageable in the field. It is not necessary to have sub-subunit level population information at this stage. Approximate populations (e.g. “Subunit C part 1 of 2” (50% of the population in Subunit C) and “Subunit C part 2 of 2” (50% of the population in Subunit C) can be used and the exact boundaries of these sub-subunits can be determined upon arrival, based on well-defined neighbourhoods or other existing administrative units, if they are selected in Step 6 below. It is important to keep track of the number of parts into which a large subunit was divided (e.g. “part 1 of 3”) so that the team in the field knows the number of initial groups in which to split the large subunit.
Step 4: Enter the names of the subunits and the estimated number of households in the CSB. To ensure maximum geographical representation of the survey area, subunits should be listed in geographical order. This list of subunits and number of households may be copied and pasted directly from the census spreadsheet(s) used in Steps 1–3 to save time and reduce the potential for errors. If information on the number of households does not exist, it can be approximated by dividing the total population for each subunit by the average household size and rounding to the nearest whole number.
Step 5: Determine the target segment size and the number of segments per subunit. The CSB will automatically determine the target segment size, which is set at 50 households by default but may be larger if the sample size per segment is not expected to be met after visiting 50 households. For a more detailed explanation of how the target segment size is calculated, see Annex 2. The number of segments per subunit is equal to the projected subunit size divided by the target segment size and rounded to the nearest whole number (e.g. 131 HH / 50 HH per segment ≈ 3 segments).
Step 6: Select 30 subunits using PPES. The CSB will systematically select 30 subunits from the survey area with probability proportional to the estimated number of segments they contain. It is possible for larger subunits to be selected more than once. In such cases, the number of segments to be selected from the subunit is equal to the number of times it was selected. For example, if the sixth and seventh selected segments fall within subunit #28, then it will be necessary to randomly select two segments from subunit #28.
Sensitization of subunits The survey coordinator is in charge of making sure that the leaders of each subunit selected for the coverage survey are made aware of the survey in advance of the team’s visit. During this sensitization visit (or phone call) with the local leaders, the representative from the survey team should share the purpose of the coverage survey and also discuss the optimal day of the week and time of day for the survey team to visit in order to find members of the survey population at home.
M O
D U
LE 1
2. Implementation 2.1 Creating segments
A segment is the area to be surveyed within each selected subunit. Segments refer to groups of households and are used to reduce the time and work required for sampling in the field. Only households within the selected segment need to be enumerated. On average, the number of households in each segment is expected to be roughly the target segment size (Box 1.3 shows the method used to calculate this figure). For most surveys the target segment size will be 50 households. A total of 30 segments will be chosen from the subunits selected (at least one segment in each selected subunit) via PPES (see section 1.5).
Maps (or “sketch maps”)
Once the subunits have been selected it is worthwhile investigating whether maps of the selected subunits are available. If EAs were used as the subunits, the office of the census should have maps for each EA available upon request.
Because these maps outlining the boundaries of each EA may require a fee, it is advisable to solicit maps only for those EAs that are selected. Sometimes sketch maps are available in the field at the local health post or from a village leader. For this reason it is important that planning for a coverage survey takes place well in advance of the desired implementation.
While obtaining maps may require additional work during the planning phase, they often result in substantial time saving during survey implementation.
This survey method is still feasible to conduct in villages if maps are not available.
Box 1.3. Dividing a subunit into segments
The following steps should be followed by survey teams in the field to divide a subunit into segments and randomly select one segment in which to sample.
Step 1: Locate the outer boundaries of the subunit. If the subunit is a village or locality it may be easy for village leaders to describe where the boundaries lie; however, if the subunit is an EA, it will be necessary to use the maps to determine the boundaries of the EA because the EA may not necessarily coincide with a village or locality.
Step 2: Divide the subunit into segments with roughly the same number of households. Survey teams should carry with them a list of the selected subunits and the number of segments required in each (available from the CSB). Survey teams should work with local leaders to help divide the subunit into the predetermined number of segments such that each segment has approximately the same number of households. This means that the geographical size of the segments may vary considerably – densely populated areas will have geographically small segments and low-density segments will be large. It is recommended to use natural lines of division, such as roads, footpaths, streams or other distinguishable landmarks to form the boundaries of the segments so that it is clear into which segment each household falls. Maps and the assistance of community leaders will be essential in this process. Assign each segment a number.
Note: It is important that each subunit be divided into exactly the predetermined number of segments, based on the estimated number of households. The number of segments should not be revised in the field if the original estimate is found to be incorrect.
Example If a selected subunit is expected to have 131 households, then according to the CSB it will require 3 segments (131 HH / 50 HH per segment ≈ 3 segments). But suppose that upon reaching this subunit in the field the survey team discovers that there are only 106 households. It is very important that the survey team still divide this subunit into exactly 3 segments, as was originally planned, even though the correct number of households is quite different. Any deviation in the number of segments from that planned using the CSB will result in a non-equal probability sample.
Step 3: Randomly select one segment. Randomly select one segment by assigning a number to each segment and then drawing one number from a hat (or flipping a coin if there are only two segments from which to choose). When the entire village or EA is a single (1) segment there is no need for random selection.
Example If the subunit has been divided into four segments, assign each segment a number from 1 to 4. Write the numbers 1 through 4 on pieces of paper and put these pieces into a bowl or hat. Draw one paper. The number drawn corresponds to the number of segments that has been selected.
M O
D U
LE 1
Upon arrival in a selected subunit, members of the survey team should first familiarize themselves with the boundaries of the subunit, according to the map or local leadership. The team should then work with local leaders to divide the subunit into the predetermined number of segments. Follow the steps in Box 3 to create segments within each selected subunit.
Figure 1.1 provides a step-wise depiction of how to create segments and select households within each segment of an EA.
Large subunits If a selected subunit corresponds to a larger subunit that was subdivided in section 1.5 (e.g. “Subunit C part 1 of 2”) then the subunit should first be subdivided into the number of parts using any existing administrative structure (e.g. neighbourhoods, blocks, zones) and then one of these subdivisions selected at random. It is within this selected subdivision that the segmentation should occur. Note that larger towns are often more likely to have maps at the local level, which can help tremendously with the segmentation.
Panel C Panel D
Panel A Panel B
Figure 1.1. Depiction of the steps required to create segments and select households within each segment of the EA Panel A: Example of a rural EA with approximately 85 households. Panel B: The EA is segmented into two segments of approximately equal size using natural lines of division. Panel C: One segment is randomly selected (by tossing a coin or drawing pieces of paper from a hat or bowl). Panel D: A walking route through the selected segment is identified that passes by all households in the segment; households are selected for the survey according to the selected sampling list (either List A or List B).
Example The town of Lenbe is expected to have 1336 households and for that reason it was subdivided into four parts in section 5, with each part listed on a separate line in the CSB tool (see example below); “Lenbe part 3 of 4” was selected as a subunit by the CSB. Upon arrival in Lenbe the team learns that it contains five well-defined neighbourhoods (A, B, C, D and E). Because Lenbe was originally subdivided into four parts during the coverage survey planning phase, the team considers each of these neighbourhoods to be a part, pairing the two smallest neighborhoods (D and E), and then randomly selects one of these neighbourhoods to serve as “Lenbe part 3 of 4” – that is, the selected subunit. The team writes the neighbourhood names A, B, C and D+E onto slips of paper and places them into a hat. Neighbourhood A is randomly selected, which means that the part of Lenbe that corresponds with neighbourhood A will serve as the selected subunit. The team then visits neighbourhood A and divides it into the predetermined number of segments (seven in this example) of approximately equal size, from which one is randomly selected.
# Subunit names Estimated no. of households (from census)
No. of segments per subunit Cumulative segments Selected subunits
1 Lenbe part 1 of 4 334 7 7
2 Lenbe part 2 of 4 334 7 14
3 Lenbe part 3 of 4 334 7 21 1
4 Lenbe part 4 of 4 334 7 28
Figure 1.2 depicts an example of segmentation in a larger semi-urban EA.
M O
D U
LE 1
Panel A
Panel B
Figure 1.2. Example of segmentation in a larger semi-urban EA Panel A: A sketch map of a semi-urban EA with ~270 households. Panel B: The EA is divided into five segments with approximately equal numbers of households using roads as the primary line of division between segments. Panel C: Segment 5 is randomly selected. Panel D: A walking route through segment 5 is identified that bypasses all households in the segment and households are selected for the survey according to the selected sampling list.
Panel C
Panel D
M O
D U
LE 1
COMMON QUESTIONS
Does each segment need to have exactly the defined number of households? No, it is more important that each segment have approximately equal numbers of households than it is for a segment to contain exactly the target segment size. In fact, because of rounding and errors with the population estimation, it is unlikely for any one segment to have exactly the defined number of households. For example, if the target segment size is set at 50 households, in practice in some subunits the segments may end up having closer to 40 households, while in other subunits they may have 60.
What if the same subunit is selected more than once by the CSB? If a subunit is very large it may be selected more than once (see Box 1.2, step 5). In this case, the number of segments selected from the subunit should correspond to the number of times it was selected. Put a piece of paper into a hat or bowl for each segment in the subunit and then draw pieces of paper from the hat corresponding to the number of times the subunit was selected. The same segment may not be selected more than once.
Example Suppose that both the 21st and 22nd selected segments come from the same subunit, which has 390 households. As with the previous example, the first step is to divide the subunit into equal subgroups. The subunit requires 390/50 = 8 segments. Because 8 is a lot of segments, it may take some time, and even require the aid of rough maps, to divide the subunit into roughly 8 equal parts. The next task is to randomly select the two segments from the eight segments, which will be the 21st and 22nd selected segments; this can be done by drawing numbers from a hat or bowl. The same segment may not be selected more than once.
How can segmentation be used in urban areas? Segmenting urban areas can be easier than segmenting rural areas, as cities and towns are usually organized into blocks or some similar units. When using EAs, maps are usually available showing streets and blocks; if unavailable, these maps can be drawn by hand.
It is preferred to use EA rather than subunits in urban areas, because subunits may vary considerably in size. However, where reliable EA maps are not available for urban areas, hand drawn maps of subunits may be used.
2.2 Selecting households within selected segments
Once segments have been chosen among the sampled subunits, households will be selected within each sampled segment for inclusion in the coverage survey. A previously established sampling interval, automatically applied by the CSB, is used to determine which households in the segment are to be sampled in order to reach the expected sample size. The CSB will generate two lists (A and B) to facilitate the selection of households within the segment, according to the sampling interval. It is recommended that the Survey Coordinator generate these lists in advance using the CSB and give each survey team a laminated copy of lists A and B to carry with them in the field.
The survey teams should follow the steps outlined in Box 1.4 to identify the households from which individuals will be surveyed.
Box 1.4. Selecting households within each segment
Step 1: Identify a route through the segment. The survey team should work with a local guide to identify a walking route that will pass by every house in the segment and determine which household will serve as the initial household.
Step 2: Select list A or B. Flip a coin to determine if List A or List B will be used.
Step 3: Follow the route through the segment and survey households according to the selected list. Starting with the initial household, enumerate households as you follow the predetermined route through the segment (ignoring any structures that are not households). If culturally acceptable, it is often helpful to number the door of each household with chalk. For each enumerated household that corresponds to a number on the selected list, stop and survey all members of the survey population who were living in the household at the time of the MDA (see section 8). Continue until the next number on the selected list is greater than the total number of households in the segment.
Note: Lists A and B are intentionally made to be much longer than will be necessary for most segments. Since most segments are expected to have about 50 households, most survey teams will complete sampling in a given segment before they reach the end of the list. Once a segment is complete the teams should not attempt to enrol additional households in order to reach the end of List A or list B. Lists A and B are longer than necessary to account for the rare instances where the actual size of a selected segment is significantly greater than planned (e.g. has 80 households), which may be due to faulty census projections or a segmentation imbalance. In such instances it is important to have enough households in lists A and B so that the team can still apply the set sampling interval to all households in the segment even if it results in an unusually large sample size from that one segment.
.../...
.../...
What if people live in compounds instead of households?
In some settings, such as sub-Saharan Africa, households may be grouped into compounds, which are more visible and easier to enumerate than households.
Example 1 Instead of sampling 1 in 4 households according to List A, the survey team could use the same List A to sample 1 in 4 compounds throughout the segment (assuming that the segment size is still based on the number of households – such as 50 households). Within each selected compound all members of the survey population who were living there during MDA would be surveyed.
Example 2 Suppose the CSB generates lists A and B (shown on the previous page) for this coverage survey (right). In addition, suppose that the survey team has already randomly selected a single segment within the subunit (see section 6) and identified a route through the selected segment that will pass by each house (Step 1). At this point a team member flips a coin and selects List A. The survey team then walks the predetermined route through the selected segment, counting each household they pass. The team does nothing except to count (or label with chalk) the 1st, 2nd, 3rd and 4th households they pass. When they arrive at the 5th household, the team stops to interview all members of the survey population living in the household because this is the first household number on the selected List A (the starting “0” on List A can be ignored, as there is no “0th” house). Having finished all interviews in the 5th household, the team then continues counting each house along the predetermined route (e.g. 6th, 7th, 8th) until they arrive at the 10th household, the next on List A. Once again the team stops and interviews all members of the survey population living in the household. This process continues until the team reaches the last household in the segment; for the sake of example, suppose this corresponds with the 47th household. At this point the team stops because they have visited each house in the segment, even though List A has additional numbers. The sampling is considered complete for that segment and the team may travel on to the next selected subunit.
2.3 Interviewing selected individuals
All household members living in the selected house or compound who are part of the survey population (the predetermined age group that is the focus of the coverage evaluation survey) should be interviewed. Information for young children (aged < 10 years) may be collected from their primary caregivers or from the children themselves. In surveys where children were treated through schools, it may be appropriate to ask only the children directly.
Questionnaire A sample questionnaire is attached as Annex 1 (Appendix 1). This questionnaire contains the minimum information required to assess MDA coverage. It is important to ask whether people were offered the medicines (this provides a measure of the programme’s ability to reach the population) and whether they swallowed the medicines (this provides an estimate of the coverage of preventive chemotherapy). The survey coordinator may choose to add additional questions including: incidence of side-effects; knowledge, attitudes and practices of the population; socioeconomic status, NTD-related morbidity; household status of water, sanitation and hygiene; or other indicators for which a population-based statistically valid answer is desired. The questionnaire should be tested locally before use and back-translated from any local language to ensure consistency.
Electronic data collection Collection of electronic data via handheld mobile devices is a viable alternative to the paper-based questionnaire. Electronic data collection often incorporates built-in error- checking, which can help to minimize errors in data entry and tabulation. It is also helpful for instituting skip patterns and saving paper. A disadvantage is that it may be harder for supervisors to review questionnaires completed electronically, particularly if the data are sent to the cloud as soon as the interview is completed. Use of an electronic data capture device does not change the recommended survey sampling methodology described above.
Households with no members of the survey population If there is no member of the survey population in the selected household, or if the entire household is absent and not expected to return later in the day, proceed to the next selected household. A replacement household is not needed; the sample size was inflated to account for non-response (Box 1.1, step 5).
M O
D U
LE 1
Absent individuals If an adult (aged ≥ 10 years) survey respondent is absent but expected to return later in that day, the survey team should try to revisit the household. If the survey respondent is not expected to return, or is absent upon the revisit, then the survey team should try to reach the individual via mobile phone. If this is not possible, another adult in the household may serve as a proxy respondent and answer on behalf of the absent individual. If a child aged > 10 years is absent but expected to return later that day, the survey team should make an attempt to revisit the household, or to visit the school that the child attends; however, a primary caregiver can answer on the child’s behalf.
Integrated assessment For integrated coverage surveys with multiple survey populations, it is recommended that a separate form be used for each survey population. The sampler (person in charge of sampling) on the team should use the selected list (either List A or List B) to know in which households survey population 1 is interviewed (e.g. households with no asterisk) and in which households both survey populations 1 and 2 are interviewed (e.g. households with an asterisk). In households where both survey populations are to be interviewed, some individuals may fall into both populations; these individuals will be interviewed twice, once for each drug package.
3. Interpretation and action
3.1 Analyse and interpret the results
Once data collection is complete, the results should be compiled to assess surveyed coverage. While there are many reasons for conducting a coverage evaluation, the two main objectives are (i) to estimate the preventive chemotherapy survey coverage and determine whether it exceeds the target coverage threshold, and (ii) to validate the reported programme (or administrative) coverage.
Given the use of cluster sampling, as opposed to a simple random sample of population members (which is not operationally feasible), it is not possible to calculate the confidence intervals around each estimate by hand. Instead, a data entry form and simple website have been developed to facilitate the confidence interval calculations.
The CSB provides the user with a “Results Entry Form” (also shown in Annex 1 (Appendix 2)). Enter the summary information for each selected segment; that is, the total number of people interviewed from that segment, the number of people who reported being offered the medicines and the number of people who reported swallowing them. The information for each segment should be entered in the order in which it was selected from the CSB (that is, in geographical order). Complete a separate Results Entry Form for each medicine assessed. Do not modify this form by adding rows or columns. Once your information has been entered, save each Results Entry Form as a csv file (separate files should be created for each medicine). This information can then be uploaded directly into the online coverage survey analysis tool (CSAT).1 Headings should not deleted from the CSB files. Verify that the information has uploaded correctly into the web tool (alternatively, enter the information directly into the web tool by hand) and click, “calculate”. The programme will then return the estimate for the surveyed coverage and the programme reach along with the corresponding 95% confidence intervals and design effects.
1. http://coverage.securedatakit.com
M O
D U
LE 1
Estimating preventive chemotherapy survey coverage The data gathered in the survey can be used to estimate the true coverage of preventive chemotherapy. Because the survey methodology employed produces an equal probability sample, sample estimates can be calculated without the use of sample weights. The estimated coverage can be calculated as follows:
Surveyed coverage = [No. of “yes” responses to having swallowed the medicine]
[Total no. of people surveyed]
Interpretation: If the surveyed coverage falls below the target coverage threshold (Table 1.2), it is evidence that the MDA needs improvement. If the surveyed coverage exceeds the target coverage threshold (e.g. at least 10 percentage points greater than the threshold), then it is evidence that the MDA was successful and the programme is functioning well. When the surveyed coverage is near the target coverage threshold (e.g. < 10 percentage points above the threshold), visual inspection alone is not sufficient to conclude that the margin of error around the surveyed coverage is greater than the target coverage threshold. In such instances, the lower 95% confidence interval of the surveyed coverage should be compared with the target coverage threshold (Table 1.3) to determine whether the true coverage of preventive chemotherapy is likely to have exceeded the target coverage threshold. This lower 95% confidence interval will automatically be calculated by the CSAT.1 The surveyed coverage is a measure of both the reach of the programme and the compliance of individuals with MDA. If the lower 95% confidence interval of the surveyed coverage is below the target coverage threshold (see section 3.2 for next steps).
Example Suppose the surveyed coverage for azithromycin is 82%, while the target coverage threshold for trachoma is 80%. Because the surveyed coverage is only slightly greater than the target coverage threshold (< 10 percentage points), the data are entered into the online Coverage Analysis Tool to determine the lower confidence interval. Suppose the online tool returns a lower confidence interval (77%). This means it is likely that the true coverage could be as low as 77%. Because 77% is less than the target coverage threshold (80%), the MDA is not considered to have achieved effective coverage and improvements to the MDA are needed.
Programme reach By collecting information on whether or not an individual was offered the drug package(s), it is possible to determine how well the programme was able to reach the population. The coverage of programme reach can be calculated as follows:
Programme reach = [Number of “yes” responses to having been offered the medicine]
[Total number of people surveyed]
1. http://coverage.securedatakit.com
Interpretation: Programme reach is an indicator of the proportion of the population in the survey area that has the opportunity to participate in MDA. Low programme reach could be an indicator of supply chain difficulties, community drug distributor challenges or inadequate social mobilization. By comparing the surveyed coverage with the programme reach, it is possible to isolate the rate of individual compliance with MDA.
Validation of reported coverage The surveyed coverage is expected to be an unbiased estimate of the true coverage of preventive chemotherapy in the survey population.1 As a result, the surveyed coverage can be compared with the reported coverage: if the two figures are similar then the reported coverage can be considered validated; if the two figures are different then the reported coverage is not validated. However, what constitutes “similar” vs “different” can be subjective. A more objective way of validating the reported coverage is to calculate the 95% confidence interval around the surveyed coverage, based on the survey data, and determine whether the reported c

tools for improving the quality of reported data and

Documents