data quality review - who · • a desk review of the data that have been reported to national...

DATA QUALITY REVIEW

Users guide to the DQR

Desk review of data quality Excel tool user guide, v1.0

3

WHO DQR Desk Review Excel tool user guide – v1.0

Acknowledgements

This toolkit is the result of a multi-agency collaboration between the World Health Organization, The Global Fund, GAVI and USAID/MEASURE Evaluation. The toolkit proposes a unified approach to data quality and integrates and builds upon previous and current tools and methods designed to assess facility level data quality, taking into account best practices and lessons learned from many countries.

Kavitha Viswanathan (WHO Geneva Switzerland) oversaw the technical development of the toolkit, with overall guidance provided by Kathryn O’Neill (WHO Geneva) – with valuable technical inputs from Ties Boerma, David Boone, Ashley Sheffel, Claire Preaud and Marina Takane.

Important contributions were provided by technical experts from a broad range of public health programmes and include Marta Gacic-Dobo, Thomas Cherian, Jan Van Grevendonk (WHO IVB); Richard Cibulski, Michael Lynch; Katherine Floyd, Philippe Glaziou, HazimTimimi (WHO Tuberculosis); Txema Callejas, Isabelle Bergeri, Chika Hayashi (WHO HIV); Nathalie Zorzi, Annie Schwartz, Alka Singh, John Puvimanasinghe, Ryuichi Komatsu (The Global Fund); Peter Hansen, Chung Won (GAVI)

Particular thanks are due to the following country partners who tested and implemented different components of the approach and provided valuable feedback: Benin, Burkina Faso, Cambodia, Democratic Republic of Congo, Kenya, Mauritania, Sierra Leone, Togo, Uganda, and Zanzibar

Financial support was provided by GAVI, The Global Fund and the Norwegian Agency for Development Cooperation (Norad)

4


5


Table of Contents

Table of Contents ................................................................................................................................ 5 Table of Figures ................................................................................................................................... 6 1.1 Background.................................................................................................................................... 7 1.2 | Overview ....................................................................................................................................... 8 2 | Layout of Tool and Data Input ...................................................................................................... 9

3 | Analysis and Dashboards ...........................................................................................................20 Annex 1 | Data quality dimensions, metrics and standard benchmarks ...................................31

6


Table of Figures Figure 1: Opening Screen ............................................................................................................................................... 9 Figure 2: Basic Information Tab.................................................................................................................................... 10 Figure 3: Input Administrative Units tab........................................................................................................................ 11 Figure 4: Mapping Subnational Administrative Units to Survey Aggregation Units ..................................................... 12 Figure 5: Selecting Program Areas and Indicators ....................................................................................................... 13 Figure 6: Data Quality Thresholds ................................................................................................................................ 14 Figure 7: Data Quality Thresholds (continued)............................................................................................................. 15 Figure 8: Completeness of Reporting ........................................................................................................................... 16 Figure 9: Input data for program-specific reporting ...................................................................................................... 16 Figure 10: Program-specific data flow and frequency or reporting .............................................................................. 17 Figure 11: Program-specific quality thresholds for completeness and timeliness of reporting ................................... 17 Figure 12: Denominator data for the evaluation of the adequacy of population data - Domain 4 .............................. 17 Figure 13: Input indicator population for Domain 3 comparisons ................................................................................ 18 Figure 14: Service outputs for the evaluation of indicator trends - Domain 2: Consistency of reporting.................... 18 Figure 15: Input Indicator Data ..................................................................................................................................... 19 Figure 16: Color coding of outliers on data input tabs ................................................................................................. 19 Figure 17: Summary Dashboard.................................................................................................................................... 20 Figure 18: Domain 1: National district completeness ................................................................................................... 21 Figure 19: Domain 1: Completeness of indicator data .................................................................................................. 22 Figure 20: Consistency of reporting completeness ...................................................................................................... 23 Figure 21: Internal Consistency - Extreme outliers ...................................................................................................... 23 Figure 22: Internal Consistency - Consistency over time............................................................................................. 24 Figure 23: Internal Consistency - Consistency between related indicators ................................................................. 25 Figure 24: External Consistency - Comparison with survey values............................................................................. 27 Figure 25: Consistency of population data - Comparison with UN population estimate of live births ........................ 28 Figure 26: Consistency of population data - Consistency between Official Statistics Office and Health Program estimates ........................................................................................................................................................................ 28

User Guide to WHO DQR Desk Review tool in Excel – v1

1.1 Background Health data are widely used for a variety of purposes – including health sector reviews, planning, programme monitoring, quality improvement and reporting. For this reason, it is critical to have high-quality data on performance in the health sector available routinely.

The national health management information system (HMIS) and health and disease program-specific reporting systems (where they exist) collect data on routine health services and health problems that are reported from health facilities in the national health-care system. These health facility data are a primary source for assessing health sector performance – i.e. the Ministry of Health (MOH) compiles the data on a regular basis to report on achievements and trends in key health performance indicators. However, HMIS data often exhibit problems of quality, and many users do not trust these data.

All data are subject to quality limitations such as missing values, bias, measurement error, and human errors in data entry and computation. Data quality assessments should be undertaken to understand how much confidence can be placed in the health data that is used to assess health sector performance and to understand the relative strengths and weaknesses of the data sources. In particular, it is important to know the reliability of national coverage estimates and other results derived from health facility data.

Various data quality assessment mechanisms are used by national authorities and partner organizations to examine the quality of health facility data. In addition, computer software applications used by health information systems can build-in checks of data quality. However, the different tools and approaches have certain limitations, including:

• National health and disease-specific programmes carry out data quality assessments independently, making it difficult to assess the capacity of health facilities comprehensively. Data issues often cut across programmes and it is more efficient (and less burdensome to staff at the periphery) to examine them holistically.

• Data quality assessment efforts have often been ad hoc and uncoordinated so that results are not always available when needed, e.g. for a health sector review. Also, these assessments often use non-standardized methodologies, making results difficult to generalize or compare.

• The sample size of these assessments is often too small to be representative of all health facilities, thus making it difficult to reach broad conclusions about reporting accuracy. Small sample sizes can also reduce the precision of estimates derived from the sample.

• This toolkit represents a collaborative effort of WHO, the Global Fund, GAVI, and USAID/MEASURE Evaluation to promote a harmonized approach to assess the quality of data reported from the level of health facilities to the national level.

This Data Quality Review (DQR) methodology builds on existing data quality assurance mechanisms; the methodology and indicators have been developed and selected on the basis of broad consultation with international health programme experts from leading donor and technical assistance agencies. It is expected that individual health and disease programmes will use the findings of a completed DQR to inform their respective detailed assessments of data quality and programme-specific information systems. The ultimate goal of the DQR is to contribute to the improvement of the quality of data used by countries for reviews of progress and performance – such as annual health sector reviews, programme planning, and monitoring and evaluation – in order to facilitate decision-making.

8


1.2 | Overview The DQR methodology comprises two separate processes that can be used jointly or separately:

• a desk review of the data that have been reported to national level whereby the quality of aggregate reported data for recommended programme indicators is examined using standardized data quality metrics;

• health facility assessment to conduct data verification and an evaluation of the adequacy of the information system to produce quality data (system assessment).

The desk review examines data quality across four dimensions; completeness, internal consistency, external comparisons and external consistency of population data. Further, the desk review examines a core set of tracer indicators selected across program areas in relation to these dimensions. The desk review requires monthly or quarterly data by subnational administrative area for the most recent reporting year and annual aggregated data for the last three reporting years for the selected indicators.

This cross-cutting analysis of the recommended programme indicators across quality dimensions quantifies problems of data completeness, accuracy and consistency by individual programme areas but also provides valuable information on the overall adequacy of health facility data to support planning and annual monitoring. WHO recommends that the desk review component of the DQR be conducted annually.

The desk review compares country information system performance to recommended benchmarks for quality, and flags for further review subnational administrative units which fail to attain the benchmark. User-defined benchmarks can be established at the discretion of assessment planners.

The desk review has two levels of data quality assessment:

• an assessment of each indicator aggregated to the national level;

• the performance of subnational units (e.g. districts or provinces/regions) for the selected indicators.

This guide addresses the use of an automated tool in MS Excel to facilitate the desk review analysis. The desk review can also be conducted using the DQR ‘app’ on the local instance of the District Health Information System, version 2 (DHIS2) if that software system is available in country. This guide focuses exclusively on the Excel tool. Guidelines for the use of the DHIS2 DQR app can be found here: https://www.ssb.no/helse/artikler-og-publikasjoner/_attachment/291371?_ts=159abfc9bc0.

https://www.ssb.no/helse/artikler-og-publikasjoner/_attachment/291371?_ts=159abfc9bc0

9


2 | Layout of Tool and Data Input

Figure 1: Opening Screen

Overview of the DQR Tool

The DQR tool is an Excel-based analytical tool which calculates standard data quality metrics for program indicators selected by a national level technical working group (e.g. a DQR Coordinating Team). Although the standard methodology calls for the evaluation of specific indicators from specific program areas, any program area and indicators within program areas can be analyzed.

The DQR tool requires data from HMIS and other data sources to be input into the Excel workbook. Data can be copy-pasted into the workbook, typically in the form of monthly indicator values for selected subnational administrative levels of the health system (e.g. District). Data from these subnational units are then evaluated against predefined standard values and benchmarks. Subnational units that fail to achieve the benchmarks (or surpass critical values based on the benchmarks) are flagged for data quality problems. Subnational units flagged for potential data quality problems should be followed-up to determine whether the extreme data values are indeed related to data quality, or are perhaps related to changes in service delivery patterns.

Once the data have been entered into the DQR in the appropriate tabs, data quality metrics are automatically calculated, including the national score (derived from aggregating the data from subnational units), the number and percent of divergent subnational units, and the names of these units to facilitate follow-up on the data.

The DQR tool utilizes VBA code for certain functions and so macros should be 'enabled' in the Excel workbook for optimum functionality.

Data quality is evaluated through standard metrics within four different data quality domains;

▪ Domain 1 - Completeness and timeliness of reporting

▪ Domain 2 - Internal Consistency of Reported Data

▪ Domain 3 - External Consistency of Reported Data

10


▪ Domain 4 - External Consistency of Population Data

Each domain contains metrics calculated to evaluate routinely reported data in different ways. Each domain has a dashboard for results, as well as a 'Summary Dashboard' which compiles results from all domains.

On the domain-specific dashboards each metric has a text box in which to record interpretation of the findings. These interpretations are critical for determining the cause of the findings as well as the next steps for improving the quality of the data. The results should be interpreted by staff with knowledge of the indicators and program areas (e.g. changes in service delivery patterns) so that data quality problems can be identified.

Figure 2: Basic Information Tab

Enter Parameters for the Analysis

On the tab 'Input_basic_info', please provide the basic assessment information requested using the drop down lists provided. The information informs the tool about the parameters of the analysis, such as the year, country, data flow model, and periodicity of reporting. The information required includes:

1. Select Country: The Country selected will automatically be included in dashboards of results, as well as being used to calculate the UN population projection for Live Births.

2. Select year: This is the year of analysis, the year for which data will be obtained and analyzed.

3. Complete the data flow model for the Country HMIS (or Programme, depending on the scope of the DQR). Include all levels of the reporting system where data are collected, aggregated, and forwarded to the next higher level. The last box should indicate the 'National' level.

4. Select the level of the reporting system for which you are conducting your analysis, that is, the level for which metrics are calculated and compared. This is usually the level for which data are input, such as the district level.

5. Select the periodicity (i.e. how often the data are reported) for the level of analysis selected. This selection will configure the indicator data entry pages for the periodicity selected. Remember also to select the first period of the reporting year (input 10). The selection of the periodicity of reporting for the level of analysis will populate the drop down list in input 10.

11


6. Input the periodicity of reporting from health facilities. This is used in the evaluation of reporting performance from facilities (domain 1 - completeness and timeliness of reporting).

7. Input the periodicity of reporting from the 1st level of aggregation (usually the district). This is used in the evaluation of reporting performance from the 1st level of aggregation (domain 1 - completeness and timeliness of reporting).

8. Input the level of the reporting system for which you are inputting service output data. These are the indicator values by month or quarter. These data can be facility level (only rarely in the event that facility level data are entered into the computer), or district, or regional level depending on what aggregate level data are available at national level.

9. Input the level of the most recent population-based survey. In domain 3 - External comparison, routinely reported data values will be compared with survey values. The routine data will need to be aggregated to the level of the survey (typically the regional level) so that the values are comparable.

10. Enter the first period of the year of analysis. Depending on the periodicity of reporting from the level selected for analysis (in #5) the drop down list will provide the range of options. Select the first period (e.g. 1st quarter, the month of January etc.) from the drop down.

11. Enter the nature of facility reporting, either integrated (e.g. on the monthly form HMIS) or program-specific. Integrated reporting means the results from different health programs are all reported on the same form, and only that form is forwarded to the next level to satisfy reporting requirements for all health programs. Program-specific reporting means that health programs report to the next level separately, each program with its own set of reporting forms. Since it may be the case that reporting from health facilities is only partially integrated, selection of the type of reporting on the Input_basic_info tab will only hide or reveal the program-specific reporting data entry and results areas. The integrated reporting tab and result areas will always be available to enter information on reporting for the HMIS in general.

Figure 3: Input Administrative Units tab

Input Administrative Units

Input the relevant administrative units in column 2 on the Input_admin_units tab. These are the administrative units for the level of analysis selected on the Basic Information tab. It is recommended to 'copy and paste' the values from an electronic list (e.g. another Excel file). Only 'values' should be pasted - that is, do not paste formulas or formatting with the values. The DQR tool has background code to prevent pasting anything other than values; however this

12


makes the process of copying and pasting slower than normal. After pasting the values please allow Excel to run the code (automatically) which is designed to strip the values stored in memory of any formatting, or convert formula results to values. The column headings on the Input_admin_units tab are prepopulated with values derived from the data flow model specified on the Basic Information tab. For each unique administrative unit name in Column 2 (there should be NO duplicate names in Col 2), there should be a corresponding number in Column 1. Please DO NOT type any additional numbers beyond the total number of administrative units typed/pasted in Column 2. For example, if you have 100 districts in your country, please only type numbers from 1-100 in Column 1. Do not type in 101 and higher. Enter higher level administrative units in Cols 3-5 as necessary (according to the data flow model entered into the Basic Information tab. If the column heading is 'National' no values need to be pasted in the column. Figure 4: Mapping Subnational Administrative Units to Survey Aggregation Units

Map Subnational Administrative Units to Survey Aggregation Units

On the Survey_mapping tab you will indicate the administrative units from a recent population-based survey (e.g. DHS, MICS) that correspond to the administrative units identified as the level of analysis for the DQR. Typically, population-based surveys have domains of estimation at the regional level since it is expensive and time consuming to extend these surveys to lower administrative levels. To make the routine data comparable to the survey data the administrative units for the level of analysis need to be 'mapped' to the survey administrative unit. For example, if the district is the level of analysis for the DQR and the survey domain of estimation is the region, the districts within those regions need to be identified in the DQR.

Select Program Areas and Indicators

The DQR has a standard set of indicators from across program areas that are intended to provide a cross-cutting assessment of data quality. These are:

▪ Maternal health - Antenatal care 1st visit (ANC1)

13


▪ Immunization - DTP3/Penta3 ▪ HIV/AIDS - ART coverage ▪ TB - Notified cases of all forms of TB ▪ Malaria - Confirmed malaria cases However, the DQR is designed to accommodate any program areas and indicators. On the 'Program Areas and Indicators' tab select program areas and their associated indicators using the drop down lists provided. One primary indicator should be selected for each program area. The primary indicator is listed as #1 in the two spaces provided for each program area. The secondary indicator (#2) is only used for the Internal Consistency metric 'Comparison between related indicators'.

Figure 5: Selecting Program Areas and Indicators

Drop down lists for program areas and indicators include the standard indicators used for the recommended implementation of the DQR, as well as a supplemental list of alternative indicators for each program area. Information on the core and alternative indicators can be found in the DQR Technical Guide (Module 3: Review of data quality through a health facility survey; Annex 1 - Recommended Indicators).

It is also possible to include user-defined program areas and indicators by selecting 'Other_specify' from the drop down list. Another field will appear in which the user-defined program area and/or indicator can be entered. Once entered the program area and indicator names auto-populate the dashboards of results in the DQR.

Finally, a section for selecting the indicator type, either cumulative or current, is included. A cumulative indicator is one for which monthly values are added to the previous month's value to derive a running total (e.g. number counseled and tested for HIV). A current indictor is one where the current month's value updates or replaces the previous month's value (e.g. current on ART where lost, stopped, transferred out or died are all subtracted from the total, new patients are added, and those counted this month were most likely also counted last month). The default value is cumulative since most indicators are cumulative.

14


Review and/or Edit Data Quality Thresholds

To judge the quality of data using the metrics in the DQR it is necessary to define benchmarks of quality with which to compare the results. WHO has recommended thresholds for each metric which can be found on the 'Quality Thresholds' tab. Often, global standards are not relevant in a given country if the information system is immature, or is undergoing reform. In cases where the recommended thresholds are inappropriate, user-defined thresholds can be supplied by entering the values in column 2 on the 'Quality Thresholds' tab which will override the recommended thresholds. Figure 6: Data Quality Thresholds

Recommended User-definedDomain 1: Completeness and Consistency of Reporting/Indicator Data Col 1 Col 2

1a

1a1a 75%1a1b 75%

1a2a 75%1a2b 75%

1a3a 75%1a3b 75%

1a4a 75%1a4b 75%

1b

Program Area 1: Maternal_Health1b1 Indicator 1: ANC 1st Visit 90%

Program Area 2: Immunization1b2 Indicator 1: 3rd dose DPT-containing vaccine 67%

Program Area 3: HIV_AIDS1b3 Indicator 1: Number of HIV+ persons currently on ART 90%

Program Area 4: Malaria1b4 Indicator 1: Number of confirmed malaria cases reported 90%

Program Area 5: TB1b5 Indicator 1: Number of Notified TB cases (all forms of TB) 75%

Program Area 6: Multi-Program1b6 Indicator 1: DTP1 Specify in Col 2 90%

1c

1c1 10%

1c2 10%

1c3 10%

1c4 10%

Completeness of Region Level Reporting

Consistency of Province reporting completeness

Completeness of Indicator Reporting: % of data elements that are non-zero values; % of data elements that are non-missing values

Consistency of reporting completeness (each information system):

Timeliness of Region Level Reporting

Completeness of District Level Reporting

Threshold

Completeness and Timliness of Reporting from Health Facilities and Aggregation Levels: District, Region, Province

Completeness of Province Level ReportingTimeliness of Province Level Reporting

Timeliness of District Level Reporting

Completeness of Health Facility Level Reporting

Consistency of Region reporting completeness

Timeliness of Health Facility Level Reporting

Consistency of District reporting completeness

Consistency of Facility reporting completeness

Quality Thresholds: 'Quality thresholds' are the values that set the limits of acceptable error in data reporting. The analyses in the DQR compare results to these thresholds to judge the quality of the data. Recommended values are included for each metric in column 1. User-defined thresholds can be input into col 2 which will take precedence over the values in col 1.

15


Figure 7: Data Quality Thresholds (continued)

Domain 2: Internal consistency of reported data

2a

2a11

2a2;2a3

2

2bConsistency over time:

2b1 Program Area 1: Maternal_Health 33%

2b2 Program Area 2: Immunization 33%

2b3 Program Area 3: HIV_AIDS 33%

2b4 Program Area 4: Malaria 33%

2b5 Program Area 5: TB 33%

2b6 Program Area 6: Multi-Program 33%

2c

2c1 Program Area 1: Maternal_Health 10%

2c2 Program Area 2: Immunization 10%

2c3 Program Area 3: HIV_AIDS 10%

2c4 Program Area 4: Malaria 10%

2c5 Program Area 5: TB 10%

2c6 Program Area 6: Multi-Program 10%

Domain 3: External Comparison

3

3a Program Area 1: Maternal_Health 33%

3b Program Area 2: Immunization 33%

3c Program Area 3: HIV_AIDS 33%

Domain 4: External consistency of population data

4a4a1

5%

4b

4b1 10%

4b2 10%

4b3 10%

Identification of Outliers:

Percent difference between district ratio of current year indicator value and the average of the preceding three years and the same national level ratio (for indicators expected to remain constant), or the current year to the forcasted value (for indicators with a non-constant trend)

Extreme: Number of monthly district values over the course of one year that are extreme outliers, i.e. districts with >= 1 extreme outliers in the year are flagged for investigation

Moderate: Number of monthly district values over the course of one year that are moderate outliers, i.e. districts with >= 2 moderate outliers in the year are flagged for investigation

Consistency between related indicators (conduct one per program area): % departure from district the expected ratio of the two indicators

ANC 1st Visit : ANC 4th Visit

3rd dose DPT-containing vaccine : 1st dose DPT-containing vaccine

Number of HIV+ persons currently on ART : HIV Coverage

Number of confirmed malaria cases reported : Confirmed (or total) malaria cases receiving treatment Number of Notified TB cases (all forms of TB) : Number of TB cases successfully treated (all forms of TB)

DTP1 : ANC1

Ratio of DTP3 coverage rates from routine data to survey DTP3 coverage ratesRatio of HIV Counselling and Testing from facility to proportion of population HIV tested in C&T

Comparison of routine data with population-based survey values from the same period: % difference

Ratio of facility ANC1 coverage rates to ANC1 survey coverage rates

Consistency of population projectionsRatio of population projection of live births from the Country Census Bureau/Bureau of Statistics to a UN live births projection for the country (% difference)

Consistency of denominator between program data and official government population statistics% difference between district ratio of program denominator to official government denominator and the same national level ratio

Live births

Children under one year of age

Pregnant women

16


Figure 8: Completeness of Reporting

Input data on reporting completeness (integrated reporting)

On the 'Input_reports_received' tab enter the information required on completeness and timeliness of reporting from subnational units. Depending on the data flow model input in the Basic Information tab, you will need to enter data on the number of reports received for each level, and historically (3 prior years). Also required is information on the number of reports received by the deadline of reporting for the year of analysis. Please ensure to select the appropriate periodicity of reporting on the Basic Information tab for facility reporting and from the next higher level (#7) so the DQR tool will know the number of expected reports in the calculation of completeness of reporting. Figure 9: Input data for program-specific reporting

Input data on reporting completeness (program-specific reporting)

If Program-specific is selected in question #11 on the Input_basic_info tab, a different tab appears for entering information on program-specific reporting (‘Input_reports_program specific’). Specifications for the data flow model, frequency of reporting, and data quality thresholds will need to be entered for each health program under review in the appropriate cells (Figure 10-11).

17


Figure 10: Program-specific data flow and frequency or reporting

Figure 11: Program-specific quality thresholds for completeness and timeliness of reporting

Input Population Data

The DQR evaluates the adequacy of population data (i.e. denominators) used to calculate coverage rates for performance monitoring in 'Domain 4 - Consistency of population data'. Denominator data is also required to compute rates for comparisons of routine data to population-based survey data ('Domain 3 - External consistency'). There are two tabs in which input of population data are required, one for each domain. On the tab 'Input_Standard_Populations' (Figure 12) enter the populations from Official Government Statistics (e.g. from the National Statistics Bureau) by the level selected for analysis (e.g. district) for Live Births, Expected Pregnancies, and Children < 1 year of age (columns F-H). These denominators will be compared to the same populations used by health programs, if applicable. If health programs are using their own estimates of these populations enter the values by the level selected for analysis into the appropriate cells (columns I-K). Figure 12: Denominator data for the evaluation of the adequacy of population data - Domain 4

For the comparison of routine reporting results to the results of population based surveys (or other external data sources evaluating coverage using population-based rates) enter the standard populations by level selected for analysis on the tab Input_Indicator_Populations. These are the populations used to calculate rates for routine values that will be compared to survey values. As such, these values should be specific for the indicator and the year of the analogous survey value (Figure 13).

Program AreaMaternal_Health

Faci

lity

→

3rd level of aggregation

HIV_AIDS

Faci

lity

→

Multi-program District

Monthly

Monthly

Monthly

Monthly

Quarterly

DistrictDistrictDistrictDistrictRegion

MalariaTB

Immunization

Monthly

Data Flow by Progam Area Frequncy of reporting by Program Area and Level2nd to 3rd level of

aggregationFacility to 1st level

of aggregation1st to 2nd level of

aggregation1st level of aggregation

2nd level of aggregation

Default User defined




Maternal_Health 75% 90% 75% 100% 75% 90% 75% 100%Immunization 75% 90% 75% 100% 75% 90% 75% 100%HIV_AIDS 75% 75% 90% 75% 80% 75% 90%Malaria 75% 90% 75% 90% 75% 90% 75% 90%TB 75% 75% 100% 75% 100% 75% 100%Multi-program 75% 75% 100% 75% 90% 75% 100%

Completeness

Maternal_HealthImmunizationHIV_AIDSMalariaTBMulti-program

Quality Thresholds(enter percent value)

Facility to 1st level of aggregation

1st to 2nd level of aggregation

(enter percent

value)

Timeliness

Facility to 1st level of aggregation

1st to 2nd level of aggregation

18


Figure 13: Input indicator population for Domain 3 comparisons

Input data on trends

To evaluate 'Internal consistency - Consistency of indicator data over time' (Domain 2) you will need to enter annual values for the level selected for analysis for the DQR primary indicators (selected on the 'Program Areas and Indicators' tab). Annual values for the indicators are required for the three years prior to the analysis year. The annual values for the prior years need to be pasted into the appropriate columns for each of the indicators, while the values for the year of analysis are aggregated automatically by the DQR tool once the monthly values have been input into the indicator data tabs (e.g. 'Input_PA1_Ind1') (Figure 14). Figure 14: Service outputs for the evaluation of indicator trends - Domain 2: Consistency of reporting

19


Input indicator data

Paste monthly (or quarterly) data by level selected for analysis into the indicator data tabs. The indicator names should appear automatically at the top of each of the indicator data tabs once the indicators are selected on the 'Program Areas and Indicators' tab. The indicator data tabs are named according to the following logic: PA1 is Program Area #1, while Ind1 is the primary indicator for the program area. Each Program Area selected on the 'Program Areas and Indicators' tab has two indicators, a primary and a secondary indicator. The primary indicator is the indicator for which DQR metrics are calculated. The secondary indicator is only used for the 'Domain 2 - Internal consistency' evaluation of the consistency between related indicators. Furthermore, PA2 is Program Area #2, which has Ind1 and Ind2, etc. Figure 15: Input Indicator Data

Please ensure that the periodicity of reporting for the level of analysis is indicated in #5 on the Basic Information tab. This selection will configure the Indicator Data tabs for 12 columns for monthly reporting, and 4 columns for quarterly reporting. In Domain 2 – Internal Consistency of Reported Data extreme and moderate sub-national unit values are identified for monthly (or quarterly) reporting. These values are highlighted on the Input Indicator Data tabs by color coding as follows: outliers are noted by a stippling pattern and shaded gray for moderate outliers, and shaded pink for extreme outliers (Figure 16). These values are summarized and the sub-national units where they occur are identified in the summary tabs for Domain 2. Figure 16: Color coding of outliers on data input tabs

= moderate outlier = extreme outlier

20


3 | Analysis and Dashboards Figure 17: Summary Dashboard

No. Indicator DefinitionNational Score

(%)

# of districts not attaining quality

threhold

% of districts not attaining quality

threshold

1aCompleteness of District Reporting

National district reporting completeness rate and districts with poor completeness of reporting

99.1% 1 2.1%

1bTimeliness of District Reporting

National district reporting timeliness rate and districts with poor timeliness of reporting

90.3% 3 6.4%

1cCompleteness of Facility Reporting

National facility reporting completeness rate and districts with poor facility reporting completeness

96.1% 8 17.0%

1dTimeliness of Facility Reporting

National facility reporting timeliness rate and districts with poor facility reporting timeliness

89.6% 11 23.4%

Maternal_Health - ANC 1st Visit 98.9% 1 2.1%

Immunization - 3rd dose DPT-containing vaccine 99.3%

HIV_AIDS - Number of HIV+ persons in palliative care 99.8%

Malaria - Number of confirmed malaria cases reported 99.8%

Immunization - OPV3 98.9% 1 2.1%

Multi-program - Penta 1st doses 99.1%



HIV_AIDS - Number of HIV+ persons in palliative care 100.0%

Malaria - Number of confirmed malaria cases reported 99.5% 1 2.1%

Immunization - OPV3 100.0%


1f.1Consistency of Reporting Completeness - District Reporting

Consistency of district reporting completeness and districts deviating from the expected trend

105.8%

1f.2Consistency of Reporting Completeness - Facility Reporting

Consistency of facility reporting completeness and Districts deviating from the expected trend

103.5%

No. Indicator DefinitionNational DQ

Score (%)# of districts with

poor scores% of districts

with poor scores

Extreme Outliers (relative to the mean):



HIV_AIDS - Number of HIV+ persons in palliative care 0.5% 3 6.4%

Malaria - Number of confirmed malaria cases reported 0.0%

Immunization - OPV3 0.4% 2 4.3%


Total % of national values 0.2%

O li l i h

DOMAIN 1: COMPLETENESS OF REPORTING

BURUNDI - ANNUAL DATA QUALITY REVIEW: RESULTS, 2016

DOMAIN 2: INTERNAL CONSISTENCY OF REPORTED DATA

Completeness of indicator data (missing values)

1e.1

1e.2

Indicator 2a: Accuracy of event reporting - Identification of Outliers

2a.1

Percentage of national values that are extreme outliers relative to the mean (≥ 3 SD) and districts with extreme outliers

Indicator 1: Completeness and timeliness of reporting

Indicator 1e: Completeness of indicator data - presence of missing and zero values

Indicator 1f: Consistency of reporting completeness over time

Completeness of indicator data (zero values)

21


Summary Dashboard

The tab 'Summary_dashboard' (Figure 17) displays results for all DQR domains and metrics in summary form, without detail or graphics. The standard form for results is the value of the metric plus the number and percent of subnational units which do not attain the established benchmark for the metric. The subnational units which do not attain the standard are listed on the domain-specific dashboards. Figure 18: Domain 1: National district completeness

Domain 1 – Dashboard: Completeness & Timeliness of Reporting

Domain 1 includes the following metrics;

▪ Completeness of subnational unit reporting - the number of reports received over the number of reports expected from subnational units (Figure 18);

▪ Timeliness of subnational unit reporting - the number of reports received by the deadline over the number of reports received from subnational units;

▪ Completeness of facility reporting - the number of reports received from health facilities over the number of reports expected from health facilities;

▪ Timeliness of facility reporting - the number of reports received from health facilities by the deadline of reporting over the number of reports received from health facilities;

▪ Completeness of indicator data - measures the percentage of missing or zero values reported from subnational units (Figure 19);

22


Figure 19: Domain 1: Completeness of indicator data

▪ Consistency of subnational reporting completeness - compares the mean of reporting completeness of the three years immediately prior to the year of analysis to the reporting completeness of the year of analysis. If the trend in reporting completeness is non-constant (i.e. either increasing or decreasing) use the dropdown menu in cell C92 to select the trend in reporting (constant or increasing/decreasing). If the trend is non-constant and either increasing or decreasing is selected, the reporting completeness for the year of analysis is compared to a value predicted from the slope of the trend of the previous three years.

The actual trend in reporting completeness can be judged from the graphic 1f - Consistency of reporting completeness on the Domain 1 dashboard (Figure 20).

▪ Consistency of health facility reporting completeness - As above for subnational unit consistency of reporting completeness but for reporting from health facilities to the subnational units. Select the trend in reporting (constant or increasing/decreasing) from the dropdown list in cell C93. Again, the actual trend in reporting completeness can be judged from the graphic 1f on the Domain 1 dashboard.

National score

Program Area and Indicator Quality Threshold Type % No. % Name

Missing 98.9% 1 2.1%

Zero 98.9% 2 4.3%

Missing 99.3% 1 2.1%

Zero 99.5% 1 2.1%

Missing 99.8%

Zero 100.0%

Missing 99.8%

Zero 99.5% 1 2.1%

Missing 98.9% 1 2.1%

Zero 100.0%

Missing 99.1% 1 2.1%

Zero 100.0%

Missing 99.3% 4 8.5%

Zero 99.6% 4 8.5%

Malaria - Number of confirmed malaria cases reported

23


Figure 20: Consistency of reporting completeness

Domain 2 - Internal Consistency

Domain 2 includes the following metrics;

▪ Identification of extreme outliers - monthly (or quarterly) values entered for subnational units selected as the level of analysis are examined for the presence of extreme outliers, i.e. values that are ≥ 3 standard deviations from the mean of monthly (or quarterly) values entered for subnational units. For each primary indicator entered on the 'Program Areas and Indicators' tab, the number and percentage of values that are extreme outliers is calculated and the subnational units identified (Figure 21).

Figure 21: Internal Consistency - Extreme outliers

▪ Identification of moderate outliers - monthly (or quarterly) values entered for subnational units selected as level the of analysis are examined for the presence of moderate outliers, i.e. values that are

80%

85%

90%

95%

100%

105%

2013 2014 2015 2016

Facility Reporting Completeness

District Reporting Completeness

National score

% No. % Name

0.2% 1 2.1%

0.0%

0.5% 3 6.4%

0.0%

0.4% 2 4.3%

0.0%

0.2%

Indicator 2a.1: Extreme Outliers (>3 SD from the mean) 2016

Malaria - Number of confirmed malaria cases reported

Immunization - OPV3

Multi-program - Penta 1st doses

Total (all indicators combined)

DOMAIN 2: INTERNAL CONSISTENCY OF REPORTED DATA

-

Districts with extreme outliers relative to the mean

District 39

Indicator 2a: Identification of Outliers

-

District 31, District 39

-

Program Area and Indicator

District 9, District 38, District 41

Maternal_Health - ANC 1st Visit

Immunization - 3rd dose DPT-containing vaccine

HIV_AIDS - Number of HIV+ persons in pall iative care

Interpretation of results - Indicator 2a1: ••••••

24


between 2 and 3 standard deviations from the mean of monthly (or quarterly) values entered for subnational units. For each primary indicator entered on the 'Program Areas and Indicators' tab, the number and percentage of values that are moderate outliers is calculated and the subnational units identified. Moderate outliers are also identified based on the modified Z-score which evaluates monthly (or quarterly) values relative to the median of monthly (or quarterly) values entered for subnational units. The modified Z-score is preferable for routine data with large variability in monthly values, or when quarterly values are entered for subnational units.

▪ Consistency over time - The plausibility of reported results for selected programme indicators are examined in terms of the history of reporting of the indicators. Trends are evaluated to determine whether reported values are extreme in relation to other values reported during the year or over several years (Figure 22).

-For this metric the annual value of primary indicators for the year of analysis (aggregated from monthly or quarterly values entered for subnational units) is compared to the mean of annual values for the three years preceding the year of analysis. Subnational units with a ratio of the annual value for the year of analysis to the mean of the annual values from the 3 preceding years divergent from the expected ratio (or national ratio) more than the recommended (or user defined) quality threshold are identified, and the number and percent of such subnational units is calculated.

Figure 22: Internal Consistency - Consistency over time

There are two ways in which users can customize the evaluation of consistency over time;

Qual i ty threshold

National score (%)

Number of dis tricts with divergent scores

Percent of dis tricts with divergent scores

Names of dis tricts with divergent scores :

20%

Expected trend Increas ing

Compare dis tricts to: expected resul t

2b3: Consistency of 'General_Service_Statistics - OPD Total Visits' over time

Year 2014

100%

5

38.5%

District 6, District 7, District 8, District 9, District 11

0

100,000

200,000

300,000

400,000

500,000

600,000

0 100,000 200,000 300,000 400,000 500,000

Gen

eral

_Ser

vice

_Sta

tistic

s -O

PD T

otal

Vis

its

even

ts fo

r yea

r of a

naly

sis

Forcasted General_Service_Statistics - OPD Total Visits value for current year based on preceding years (3 years max)

0

1,000,000

2,000,000

3,000,000

2011 2012 2013 2014

Trend over time: General_Service_Statistics - OPD Total Visits

Int erpretation of results - Indicator 2c3: •This indicator is increasing over time (Outpatient visits are increasing - something we were expecting given social mobiliation for public health services.•Comparison of expected result (that the forecasted value is equal to the actual value for 2014) yeilds 5 districts with ratios that exceed the quality threhold of 20%. 3 are inferior of the quality threshold while 2 are greater.• Errors are not systematic (e.g. all in one direction) Review district

outpatient registers in affected districts to confirm reported values.

25


◦Users can select how subnational units are evaluated, either by comparing the subnational unit ratio (annual value for the year of analysis/mean of values for 3 preceding years) to the national ratio (aggregate of indicator for all subnational units for year of analysis/mean of aggregate annual values for 3 preceding years), or to the expected value. The value expected is the value when the trend in the data is consistent. If consistent then the ratio equals "1" since the annual value for the indicator equals the mean of 3 preceding years. If subnational units are expected to have a ratio more like the national ratio, (due for example to a variation or disruption in service delivery) comparison to the national ratio should be selected. The comparison to expected ratio or national ratio can be selected by using the dropdown list in column F for the line 'compare districts to:' line for each of the six indicator-specific dashboards for Consistency over time on the Domain 2 - Internal Consistency dashboard.

◦Users can select whether to compare the annual aggregate value from subnational units to the mean of the annual values for the preceding 3 years (for constant trend in the indicator), or to the value predicted (or forecast) from the slope of the trend line of the annual values from the preceding 3 years. The actual trend in the indicator values can be determined by evaluating the trend graphic for each of the primary indicators in the indicator-specific dashboards for Consistency over time on the Domain 2 - Internal Consistency dashboard. Select the trend in the indicator (constant, or increasing/decreasing) using the dropdown list in column F for the line 'Expected trend' in the indicator-specific dashboards on the Domain 2 -Internal Consistency dashboard.

Figure 23: Internal Consistency - Consistency between related indicators

▪ Consistency between related indicators - Programme indicators which have a predictable relationship are examined to determine whether, in fact, the expected relationship exists between those indicators. In

Percent of districts with divergent scores 15.4%

Names of districts with divergent scores:

District 5, District 6

Indicator 2c: Internal Consistency - Consistency Between Related IndicatorsConsistency between related indicators - Ratio of two related indicators and Districts with ratios significantly different from the national ratio *

2c1: Maternal Health Comparison: ANC 1st Visit : IPT 1st Dose

Year 2014

Expected relationship

National Score (%) 114%

Number of districts with divergent scores 2

equal

Quality Threshold 10%

Compare districts with: national rate

Interpretation of results - Indicator 2c1: • Data seem pretty good - only district 5 has a largely discrepant value• IPT seens consistently lower than ANC1 - more pregnant women should be receiving IPT• Stock out of fansidar in Region 2 could explain low number of IPT in Districts 5 . Call DHIO in these districts to investigate•National rate is 114% - most districts are close to this value. District 6 is performing well relative to the other districts but is 'discrepant' relative to the national rate. - no follow up needed.

0

5000

10000

15000

20000

25000

30000

35000

40000

45000

50000

ANC1

eve

nts f

or y

ear o

f ana

lysis

IPT 1st Dose events for year of analysis

Scatter Plot: ANC 1st Visit : IPT 1st Dose (Districts compared to national rate)

26


other words, this process examines whether the observed relationship between the indicators, as depicted in the reported data, is that which is expected (Figure 23).

For this metric, annual aggregate values for primary indicators are compared to annual aggregate values for secondary indicators input into the program area specific Indicator Data tabs. A ratio of the primary indicator to the secondary indicator is calculated and compared to the national ratio of the same two indicators, or to the expected value of the ratio of the two indicators. The expected value is the value of the ratio when the two indicators are equal, or for a ratio, the value of 1.

There are two ways in which users can customize the evaluation of consistency between related indicators;

◦Users can select the type of comparison of the two indicators; 1) that the two indicators are equal, 2) that the primary indicator (Ind 1) is greater than the secondary indicator (Ind 2), 3) the primary indicator is less than the secondary indicator, 4) the special case of a drop-out rate (a drop-out rate is a calculation of the loss of clients from one public health process to another associated process, for example, the loss in clients from the 1st dose of DTP to the 3rd dose). Selection of an equal relationship will enable a scatter plot of the two indicators and the possibility of choosing between comparing subnational units to the national level ratio between the two indicators, or the expected value of 1. Selection of any of the other types of comparisons between the two indicators yields a line graph of the ratio between the two indicators, with target thresholds in red. Type of comparison can be selected using the dropdown list in column F of the line reading "Expected relationship" in the indicator specific dashboards for '2c: Internal Consistency - Comparison between related indicators' on the Domain 2 - Internal Consistency dashboard. (Macros must be enabled for the Excel workbook for this functionality to work properly.)

◦If the expected relationship between the two indicators is selected as 'equal' then the user has the option of selecting how to evaluate subnational units; 1) comparison with the national rate (the ratio of the primary indicator to the secondary indicator aggregated over subnational units to derive a national value for each indicator, or 2) the expected result. As mentioned above, the expected result for indicators that are 'equal' is 1. Subnational units with a ratio between the two indicators greater than 1 + the recommended (or user-defined) quality threshold (or less than 1 - the recommended (or user-defined) quality threshold) are flagged for potential data quality problems.

Domain 3 - External Consistency

The level of agreement between two sources of data measuring the same health indicator is assessed. The two sources of data usually compared are data flowing through the HMIS or the programme-specific information system and a periodic population-based survey.

Data for recent population-based surveys are entered in the 'External_Data_Sources' tab. Routine data entered for primary indicators are aggregated to the administrative units of the survey as indicated on the 'Survey_Mapping' tab. The routine data value for the appropriate survey administrative units are then divided by the population value, also aggregated to the survey administrative unit to derive a rate comparable to the survey value for the same administrative unit. The ratio of the routine value to the survey value is then calculated. Subnational units with a ratio greater than 1 + the recommended (or user-defined) quality threshold (or less than 1 - the quality threshold) are flagged as potential data quality problems.

27


Figure 24: External Consistency - Comparison with survey values

In the graphics in the indicator-specific dashboards and the Domain 3 - External Consistency dashboard, the routine values are depicted as bars, while the survey values are depicted as points (a triangle) with error bars based on the standard error of the survey estimate (entered in the 'External_Data_Sources' tab) depicting the range of acceptable error between the survey and the routine values.

Note: this metric requires intensive calculation in Excel which can sometimes slow down navigation within the Domain 3 - External Consistency dashboard, or between the Domain 3 dashboard and the neighboring dashboards. Please allow Excel time to work through the calculations and complete the navigation.

Domain 4 - External Consistency of Population Data

This data quality metric helps determine the adequacy of the population data used in the calculation of health indicators. Population data serve as the denominator in the calculation of a rate or proportion and provide important information on coverage. The metric compares two different sources of population estimates (for which the values are potentially calculated differently) in order to ascertain the level of congruence between the two. If the two population estimates are discrepant, the coverage estimates for a given indicator can be very different even though the programmatic result (i.e. the number of events) is the same.


Number of Regions with divergent scores 3


Names of Regions with divergent scores: Region 2, Region 4, Region 8

Indicator 3a: Comparison of Routine Data with Population-based Survey Values from the Same Period

Percent of Regions with divergent scores 30.0%

3a1: 'ANC 1st Visit' consistency ratio (ratio between the facility rates and survey rates)Year 2014

0%

20%

40%

60%

80%

100%

120%

Mat

erna

l_He

alth

-AN

C 1s

t Visi

t

HMIS Survey

Interpretation of results - Indicator 3a1: •ANC HMIS value in region 4 looks too low - could result from missing source documents or a failure to record service delivery. Review report forms from districts in the region to verify the reported values. •ANC HMIS value in regions 2 and 8 seems too high - could be double counting or duplicate reporting. Call District Health Information Officers to investigate.•

28


Figure 25: Consistency of population data - Comparison with UN population estimate of live births

Indicator 4a – ‘Consistency with UN population projection’ compares an estimate of live births from official government sources to the United Nations estimate. A ratio statistic is used to measure discrepancies between the two estimates. Values of the ratio that exceed the established quality threshold should be investigated (Figure 25). Indicator 4b – ‘Consistency between official statistics office and health program estimates’ compares standard population estimates from an official government source, such as the National Bureau of Statistics, to the same population estimate used by Health Programs. Often these are the same, or are derived from the same source. If different, the two can be compared to determine the level of congruence. Population estimates used for comparison in the DQR are; ▪ Live births ▪ Expected pregnancies ▪ Children < 1 year of age Figure 26: Consistency of population data - Consistency between Official Statistics Office and Health Program estimates

Names of districts with divergent scores: District 1, District 5, District 7, District 12


Number of districts with divergent scores 4

Percent of districts with divergent scores 30.8%

Indicator 4b: Consistency of denominator between program data and official government population statistics

Indicator 4b1 - Comparing the official Live Births denominator to a program denominator, if applicable Year 2014


0.00

10000.00

20000.00

30000.00

40000.00

50000.00

60000.00

70000.00

Prog

ram

den

omin

ator

for L

ive

Birt

hs

Official government denominator for Live Births

Interpretation of results - Indicator 4b1: • the Program denominators in Districts 1, 7, and 12 seem too large - and too small in District 5. Review growth rates used by program to estimate intercensal yearly values for live births.•

29


The level of congruence between the denominators from official government sources and those used by Health Programs is evaluated by calculating the ratio between the two values for subnational units. Subnational units with a ratio greater than 1 + the recommended (or user-defined) quality threshold, or less than 1 - the quality threshold are flagged as potential data quality problems. The denominator-specific dashboards in the 'Domain 4 - External Consistency of Population Data' dashboard provide a scatter plot depicting the relationship between subnational unit values of the two denominators (Figure 26). Points falling outside the dashed gray lines indicate values that exceed the quality threshold.

30


Annex

31


Annex 1 | Data quality dimensions, metrics and standard benchmarks

DIMENSION 1: COMPLETENESS OF REPORTING

An assessment of each dimension should be conducted for each of the recommended core indicators: ANC on antenatal care, immunization, HIV, TB, and malaria. Additional indicators can be selected according to priority and focus of the data quality assessment.

Data Quality Metric Definition National Level Subnational Level

Completeness of district reporting

% of expected district monthly reports (previous 1 year) that are actually received

# and % of districts that submitted: 1) at least 9/12 expected monthly reports; 2) 100% of expected monthly reports

Timeliness of district reporting

% of submitted district monthly reports (previous 1 year) that are received on time (i.e. by the deadline of reporting)

# and % of districts that submitted on time at least 75% of the monthly reports received at national level from the district 1

Completeness of facility reporting

% of expected facility monthly reports (previous 1 year) that are actually received

# and % of districts with at least 9/12 monthly facility reports received # and % of facilities that submitted 100% of expected monthly reports

Timeliness of facility reporting

% of submitted facility monthly reports (previous 1 year) that are received on time (i.e. by the deadline of reporting)

# and % of districts that received on time at least 75% of monthly facility reports that were submitted

Completeness of indicator data

(% of data elements that are non-zero values, % of data elements that are non-missing values

–do each analysis separately)

ANC 1st Visit # and % of districts with < 90% 1) non-zero values, 2) non-missing values

3rd dose DPT-containing vaccine2 # and % of districts with < 67% 1) non-zero values, 2) non-missing values

ART coverage # and % of districts with < 90% 1) non-zero values, 2) non-missing values

Notified cases of all forms of TB3 # and % of districts with < 75% 1) non-zero values, 2) non-missing values

Suspected malaria cases tested # and % of districts with < 90% 1) non-zero values, 2) non-missing values

Consistency of reporting completeness

Each information system

Evaluate the trend in completeness of reporting over the past 3 years from district to national level

Evaluate the trend in completeness over the past 3 years from facility to district level.

1 Denominator is reports received (not expected) 2 Immunization Programs expect some months will have zero values for vaccination indicators 3 TB Reporting is generally quarterly

32


DIMENSION 2: INTERNAL CONSISTENCY OF REPORTED DATA

Data Quality Measure Definition

National Level Subnational Level Outliers4 Complete for each of 5 indicators: - ANC 1st Visit -3rd dose DPT-containing vaccine, - ART coverage, - Notified cases of all forms of TB, - Suspected malaria cases tested

Extreme: % of monthly subnational unit values that are extreme outliers (at least 3 standard deviations (SD) from the mean)

Number and % of subnational units in which one or more of the monthly subnational unit values over the course of 1 year is an extreme outlier

Moderate: % of subnational unit values that are moderate outliers (between ±2-3 SD from the mean or >3.5 on modified Z-score method).

Number and % of subnational units in which 2 or more of the monthly subnational unit values for the indicator over the course of one year are moderate outliers

Consistency over time Complete for each of 5 indicators: - ANC 1st Visit -3rd dose DPT-containing vaccine, - ART coverage, - Notified cases of all forms of TB, - Suspected malaria cases tested

Conduct one of the following based on expected trend of the indicator: • Comparison of current year to the value predicted from the

trend in the three preceding years (for indicators or programs with expected growth), or

• Comparison of current year to the average of preceding 3 years (for indicators or programs expected to remain constant)

No. and % of districts whose current year to predicted value ratio (or current year to the average of the preceding three years) is at least ±33% different from the national ratio

Graphic depiction of trend to determine plausibility based on programmatic knowledge

Consistency between related indicators

Maternal Health: ANC1 – IPT1 or TT1 (should be roughly equal)

No. and % of subnational units where there is an extreme difference (≥ ±10%)

Immunization: DTP3 dropout rate: (DTP1-DTP3)/DTP1 - should not be negative

Number and % of subnational units with the number of DTP3 immunizations higher than DTP1 immunizations (negative dropout)

HIV/AIDS: ART coverage - HIV care coverage (Ratio should be less than 1)5

No. and % of subnational units where there is an extreme difference (≥±10%)

TB: TB cases notified – TB cases put on treatment (in the past year) (should be roughly equal)

No. and % of subnational units where there is an extreme difference (≥ ±10%)

Malaria: Number of confirmed malaria cases reported - Cases testing positive (should be roughly equal)

No. and % of subnational units where there is an extreme difference (≥±10%)

Verification of reporting consistency through facility survey

-% agreement between verified counts for selected indicators in sampled facility records and reported values for the same facilities

Maternal Health: ANC 4th Visit

Immunization: Penta/DTP 1-3 in children < 1 year

HIV/AIDS: HIV coverage TB6: Notified cases of all forms of TB

Malaria: Suspected malaria cases tested

4 For Programs with inconsistent levels of service delivery and for which outliers are common (e.g. Immunization) a customized threshold can be set based on programmatic knowledge. Data that have high variability month to month can also be evaluated for outliers using the modified Z-score method (see Indicators section – internal consistency - outliers) which is based on the median and has higher tolerance for extreme values than the standard deviation method. 5 The extent of difference between the two indicators is dependent on the national treatment guidelines and when PLHIV are eligible for ART 6 Sampling of health facilities requires stratification by facility type to ensure an adequate number of facilities providing TB services

33


DIMENSION 3: EXTERNAL COMPARISON (Comparison of routine data with population-based survey values from the same period)7

Indicator Definition National Level Subnational Level

ANC 1st visit Ratio of facility ANC1 coverage rates to survey ANC1 survey coverage rates

Number and % of aggregation units used for the most recent population-based survey (such as a province/state/region) whose ANC1 facility-based coverage rates and survey coverage rates are at least 33% different.

3rd dose DPT-containing vaccine

Ratio of DTP3 coverage rates from routine data to survey DTP3 coverage rates

Number and % of aggregation units used for the most recent population-based survey (such as a province/state/region) whose DTP3 facility-based coverage rates and survey coverage rates are at least 33%.

HIV --- ---

TB8 --- ---

Malaria IPT

Comparison between program and HMIS values

For select indicators, compare the value aggregated for 12 months from HMIS to Program data

For select indicators, compare the subnational unit values aggregated over 12 months - # and % of districts with >10% difference in annual values between HMIS and Program data.

DIMENSION 4: EXTERNAL CONSISTENCY OF POPULATION DATA (Evaluation of adequacy of denominators used for calculating performance indicators)

Indicator Definition National Level Subnational Level

Consistency of population projections

Ratio of population projection of live births from the Country Census Bureau/Bureau of Statistics to a UN live births projection for the country

NA

Consistency of denominator between program data and official government population statistics

Ratio of population projection for select indicator(s) from the census to values used by programs

No. and % of subnational units where there is an extreme difference(e.g. ± 10%) between the two denominators

Consistency of population trend Ratio of population values for select indicator(s) from the current year to the predicted value from the trend in population values from up to 3 preceding years

No. and % of subnational units where there is an extreme difference(e.g. ± 10%) between the two denominators

7 Complete for each program area (if sufficient recent survey data are available): Administrative data should preferably be from the same year as the survey value. Denominators used for coverage estimates from administrative data may need adjustment to make them comparable to survey values (e.g. Women attending ANC at public facilities). 8 No viable survey indicator for TB

34


Table of ContentsTable of Figures1.1 Background1.2 | Overview2 | Layout of Tool and Data InputOverview of the DQR ToolEnter Parameters for the AnalysisInput Administrative UnitsMap Subnational Administrative Units to Survey Aggregation UnitsSelect Program Areas and IndicatorsReview and/or Edit Data Quality ThresholdsInput data on reporting completeness (integrated reporting)Input data on reporting completeness (program-specific reporting)Input Population DataInput data on trendsInput indicator data

3 | Analysis and DashboardsSummary DashboardDomain 1 – Dashboard: Completeness & Timeliness of ReportingDomain 2 - Internal ConsistencyDomain 3 - External ConsistencyDomain 4 - External Consistency of Population Data

Annex 1 | Data quality dimensions, metrics and standard benchmarks

data quality review - who · • a desk review of the data that have been reported to national...

Documents