a framework for assessing the quality of income poverty...

21
Draft February 3, 2005 A Framework for Assessing the Quality of Income Poverty Statistics WORLD BANK Development Data Group

Upload: dothien

Post on 10-Aug-2018

213 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: A Framework for Assessing the Quality of Income Poverty Statisticssiteresources.worldbank.org/INTPAME/Resources/Training-Materials/... · A Framework for Assessing the Quality of

Draft

February 3, 2005

A Framework for Assessing the Quality of

Income Poverty Statistics

WORLD BANK

Development Data Group

Page 2: A Framework for Assessing the Quality of Income Poverty Statisticssiteresources.worldbank.org/INTPAME/Resources/Training-Materials/... · A Framework for Assessing the Quality of

ii

Table of Contents

Page

Introduction... .................................................................................................................................................. i Prerequisites of quality. ...................................................................................................................................1 0.1 Legal and institutional environment…................................................................................................…1 0.2 Resources……………………………………………………………………..… ..................................2 0.3 Quality awareness… ...............................................................................................................................2 1. Integrity… ...................................................................................................................................................4 1.1 Professionalism…...................................................................................................................................4 1.2 Transparency… ......................................................................................................................................4 1.3 Ethical standards……………………………………………………………..… ...................................5 2. Methodological soundness… ......................................................................................................................6 2.1 Concepts and definitions… ...................................................................................................................6 2.2 Scope… .................................................................................................................................................6 2.3 Classification/sectorization…................................................................................................................6 2.4 Basis for recording….............................................................................................................................7 3. Accuracy and reliability …..........................................................................................................................8 3.1 Source data……………………………………………………………………..… ................................8 3.2 Statistical techniques… ......................................................................................................................…8

3.3 Assessment and validation of source data……………………..….........................................................9 3.4 Assessment and validation of intermediate data and statistical outputs ……………………..…...........9

3.5 Revision studies…...............................................................................................................................10 4. Serviceability….........................................................................................................................................11 4.1 Relevance… .........................................................................................................................................11 4.2 Timeliness and periodicity…...........................................................................................................… 11 4.3 Consistency….......................................................................................................................................11 4.4 Revision policy and practice………………………………………………………..… .......................12 5. Accessibility… ..........................................................................................................................................14 5.1 Data accessibility…..............................................................................................................................14 5.2 Metadata accessibility………………………………………………………..… .................................14 5.3 Assistance to users… ............................................................................................................................15

Page 3: A Framework for Assessing the Quality of Income Poverty Statisticssiteresources.worldbank.org/INTPAME/Resources/Training-Materials/... · A Framework for Assessing the Quality of

iii

A Framework for Assessing the Quality of Income Poverty Statistics 1

(Draft as of August 6, 2001)

Introduction A. Purpose of the Framework The purpose of the Framework is to provide a flexible structure for the qualitative assessment of income poverty statistics. The Framework covers all aspects of the statistical environment or infrastructure in which data are collected, processed, and disseminated. The Framework could be used in a variety of contexts, including: (i) compilation of the data module of the IMF Reports on Observance of Standards and Codes (ROSCs); (ii) reviews performed in the context of technical assistance programs; (iii) self-assessments performed by national statistical offices and other data producers; and (iv) assessments by other groups of data users. B. Organization of the Framework The Framework is organized in a cascading structure of levels that progress from the abstract/general to the more concrete/specific. The first-digit level defines the prerequisites of statistical quality and five dimensions of quality: integrity, methodological soundness, accuracy and reliability, serviceability, and accessibility. The first-digit level is sub-divided by elements (two-digit level) and indicators (three-digit level).2 At the next level (four-digit), focal issues that are specific to income poverty statistics are addressed. Below each focal issue, key points describe quality features that may be considered in assessing the focal issues. The key points are meant to be suggestive, not exhaustive. Box A provides a view of the cascading structure approach employed in the Framework.

1 Poverty statistics can be divided into two groups – monetary and non-monetary. Monetary poverty statistics, such as incidence of poverty (headcount indices) and poverty gaps, are based on income/expenditure statistics. Non-monetary poverty statistics are based on a different set of sources, including health and education statistics. Income poverty statistics refer to monetary poverty statistics.

2 The first three levels are common to other data quality assessment frameworks that have been developed to assess datasets such as national accounts, balance of payments, monetary and financial statistics, government finance statistics, and prices statistics, with the exception of 2.4. This design was developed to ensure a common and systematic assessment across datasets.

Page 4: A Framework for Assessing the Quality of Income Poverty Statisticssiteresources.worldbank.org/INTPAME/Resources/Training-Materials/... · A Framework for Assessing the Quality of

iv

C. Structure of the Framework The elements and indicators within their respective dimensions are described below. Prerequisites of quality: Although this is not itself a dimension of quality, it includes elements and indicators that have an overarching role as prerequisites, or institutional preconditions, for quality of statistics. These prerequisites cover the following elements: (i) legal and institutional environment, (ii) resources available for statistical work, and (iii) quality awareness informing statistical work. The elements are elaborated using indicators that cover the roles and responsibilities of statistical agencies, resources and training programs, and the focus on quality within the agencies involved in statistical work. Note that statistical agencies can be either specialized ministries/departments, or statistical units within ministries. Integrity: The three elements for this dimension of quality are: (i) professionalism, (ii) transparency, and (iii) ethical standards. These elements are elaborated using indicators relating to professional independence, transparency in statistical practices, and ethical standards governing the behavior of staff. Methodological soundness: This dimension has four elements, namely: (i) concepts and definitions, (ii) scope, (iii) classification/sectorization, and (iv) basis of recording. These elements are elaborated using indicators relating to adherence to internationally accepted standards, guidelines, or good practices for coverage, scope, classification, valuation, and recording of statistical data. Accuracy and reliability: The five elements of this dimension cover: (i) source data, (ii) statistical techniques, (iii) assessment and validation of source data, (iv) assessment and validation of intermediate data and statistical outputs, and (v) revision studies. These elements are elaborated using indicators that focus on the criteria underlying source data collection, the statistical techniques used in data processing, the methods used to assess and validate data, and the use of revision studies to inform statistical processes. Serviceability: The four elements for this dimension are: (i) relevance, (ii) timeliness and periodicity, (iii) consistency, and (iv) revision policy and practice. These elements are elaborated using indicators relating to the relevance of the statistical program to users needs, timeliness and periodicity of statistical publications, the consistency

Page 5: A Framework for Assessing the Quality of Income Poverty Statisticssiteresources.worldbank.org/INTPAME/Resources/Training-Materials/... · A Framework for Assessing the Quality of

v

within a dataset and over time and with other major datasets, and the transparence of the revision process.

Accessibility: This dimension has three elements, namely: (i) data accessibility, (ii) metadata accessibility, and (iii) assistance to users. These elements are elaborated using indicators that address issues relating to forms of dissemination and dissemination media, documentation, and assistance to users of statistics.

Page 6: A Framework for Assessing the Quality of Income Poverty Statisticssiteresources.worldbank.org/INTPAME/Resources/Training-Materials/... · A Framework for Assessing the Quality of

vi

Box A: An Example of the Cascading Structure of the Data Quality Assessment Framework of the Income Poverty Statistics:

Using serviceability as the example of a dimension of quality, the box below shows how the framework identifies four elements that point toward quality. Within consistency, one of those elements, the framework next identifies three indicators. Specifically, for one of these, internal consistency, quality is assessed by considering specific key points.

* Prerequisites of quality, like the dimensions, contain elements and indicators.

Indicators*

Focal Issues (Specific to the dataset)

Key Points (Specific to the dataset)

4. Serviceability

4.2 Timeliness and periodicity

4.3 Consistency

4.4 Revision policy and practice

4.3.1 Internal consistency

4.3.2Consistency over time

4.3.3 Consistency with other sources and/or other statistical frameworks (i) Internal consistency of income poverty statistics

4.1 Relevance

Dimension

Elements*

The following could be considered in an assessment of the focal issue Internal consistency of income poverty statistics: • Any discrepancy between the sum of the quarterly data and data from annual surveys is removed through benchmarking procedures or indicators. • Over the long run, the net errors and omissions item has not been large and has been stable over time.

Page 7: A Framework for Assessing the Quality of Income Poverty Statisticssiteresources.worldbank.org/INTPAME/Resources/Training-Materials/... · A Framework for Assessing the Quality of

1

Prerequisites of quality 0.1 Legal and institutional environment – The environment is supportive of statistics. 0.1.1 The responsibility for collecting, processing, and disseminating statistics is clearly specified. (i) The primary responsibility for collecting, processing, and disseminating income poverty statistics is clearly established. • A statistical law, or other arrangements specify the responsibility for producing and

disseminating income poverty statistics. • Practices are consistent with the statistical law or other arrangements. 0.1.2 Data sharing and coordination among data producing agencies are adequate. (i) There are arrangements or procedures to facilitate data sharing and cooperation between the agency with the primary responsibility for compiling income poverty statistics and other data producing agencies. • Arrangement are in place to ensure smooth and timely flow of data from other data

producing agencies. • Contacts (e.g., regular meetings, workshops) are maintained with other data producing

agencies to ensure proper understanding of data requirements, to avoid duplication of effort, and to take into account respondents’ burden.

0.1.3 Respondents’ data are to be kept confidential and used for statistical purposes only. (i) The legal provisions and/or other arrangements are adequate to ensure the confidentiality of individual data and such arrangements are widely known. • A statistical law or other formal provision clearly states that individual responses are to

be treated as confidential and shall not be disclosed or used for other than statistical purposes unless disclosure is agreed to in writing by the respondent.

(ii) Respondents are informed of how the reported data will be used. • Respondents are informed of the use of the information they provide. (iii) There are procedures to prevent disclosure of individual data. • There are rules and regulations to prevent disclosure including penalties against misuse of

confidential data. • Special aggregation rules have been developed to ensure residual disclosure does not

occur when aggregation of survey or other confidential data are disseminated. • Data storage is secured to prevent disclosure.

Page 8: A Framework for Assessing the Quality of Income Poverty Statisticssiteresources.worldbank.org/INTPAME/Resources/Training-Materials/... · A Framework for Assessing the Quality of

2

0.1.4 Statistical reporting is ensured through legal mandate or measures to encourage response. (i) The legal provisions and/or other arrangements are adequate to ensure reporting of information for compiling income poverty statistics. • The data producing agency has the legal authority to collect data required for compiling

income poverty statistics. • The rights and obligations of respondents are clearly defined and measures are taken to

reduce respondents’ burden. 0.2 Resources – Resources are commensurate with needs of statistical programs. 0.2.1 Staff, financial, and computing resources are commensurate with statistical programs. (i) The responsible statistical agencies have adequate permanent staff and temporary staff (e.g., enumerators) resources to do required tasks. • The number and the qualification of staff are adequate to perform existing and emerging

tasks. • The incentive structure is sufficient to retain trained staff. (ii) The responsible statistical agencies have adequate financial resources. • A budgeting system provides adequate financial resources for the sustained operation of

relevant data systems.

(iii) The responsible statistical agencies have adequate computing resources. • Sufficient resources are allocated: hardware (computers, networks, communication

facilities), software, maintenance. 0.2.2 Measures to ensure efficient use of resources are implemented. (i) Effective mechanisms are in place to monitor the cost-effectiveness of resource use. • Transparent and regular report on financial activities and outputs is available. • Cost-effective institutional organization, computing technology and data sources mix

(e.g., using alternative sources) are applied. 0.3 Quality awareness – Quality is a cornerstone of statistical work. 0.3.1 Processes are in place to focus on quality

Page 9: A Framework for Assessing the Quality of Income Poverty Statisticssiteresources.worldbank.org/INTPAME/Resources/Training-Materials/... · A Framework for Assessing the Quality of

3

(i) There is recognition throughout the data producing agency that quality builds trust and thus is a cornerstone of statistical work. • There is an expectation that managers are sensitive to the various dimensions of data

quality: integrity, methodological soundness, accuracy and reliability, serviceability, and accessibility.

• Processes or activities are implemented to focus on quality. • The data producing agency provides an organizational infrastructure for quality (e.g.,

mission statements emphasizing quality, data banks that permit cross-checking) in awareness of the economics of scale and interrelations of datasets.

0.3.2 Processes are in place to monitor the quality of the collection, processing, and dissemination of statistics. (i) There are measures in place to ensure quality review at the various statistical stages. • Procedures are in place to consult with key suppliers for reviewing arrangements and

sharing information. • There is statistics users’ council or an advisory council. • There are periodic users’ surveys or other systematic means of obtaining feedback from

users. 0.3.3 Processes are in place to deal with quality considerations, including tradeoffs within quality, and to guide planning for existing and emerging needs. (i) There are processes at the level of the data producing agency to deal with quality considerations, including implicit and explicit tradeoffs among the dimensions of quality, and are the reviews used to inform planning. • There is recognition of the tradeoffs among the dimensions of data quality (e.g., between

timeliness, and accuracy and reliability) and the significance of these tradeoffs is communicated to users of statistics.

• Measures are undertaken to address problems that were identified at the various stages of collection, processing and/or dissemination.

• Periodic users’ surveys are conducted. (ii) There are mechanisms aimed at addressing new and emerging data requirements. • Regular consultation between the data producing agency and users is undertaken.

Page 10: A Framework for Assessing the Quality of Income Poverty Statisticssiteresources.worldbank.org/INTPAME/Resources/Training-Materials/... · A Framework for Assessing the Quality of

4

1. Integrity – The principle of objectivity in the collection, processing, and dissemination of statistics is firmly adhered to.

1.1 Professionalism – Statistical policies and practices are guided by professional principles. 1.1.1 Statistics are compiled on an impartial basis. (i) The terms and conditions of compiling statistics guarantee the professional independence of the producing agency. • Legal provisions and/or other arrangements addresses the general need for the

professional independence of the data producing agency. (ii) Professionalism is actively promoted and supported within the agency. • There are training, professional meetings and publications to promote professionalism. 1.1.2 Choices of sources and statistical techniques are informed solely by statistical considerations. (i) The choices of data sources and statistical techniques are informed solely by statistical considerations. • The choice of source data (e.g., among surveys, between surveys and administrative

records) is based solely on statistical considerations such as quality, timeliness, costs, and the burden on respondents.

• The choice of statistical techniques (e.g., survey design, survey techniques) and definitions is based solely on statistical considerations.

1.1.3 The appropriate statistical entity is entitled to comment on erroneous interpretation and misuse of statistics. (i) The data producing agency comments when its statistics are misinterpreted or misused. • Immediate corrective actions are taken for misinterpretation or misuse of official

statistics. 1.2 Transparency – Statistical policies and practices are transparent. 1.2.1 The terms and conditions under which statistics are collected, processed, and disseminated are available to the public. (i) Information about the statistical law, about the obligation to produce and/or disseminate income poverty statistics, about the confidentiality of individual responses, and about other key features of the terms and conditions under which income poverty statistics are collected is available to the public.

Page 11: A Framework for Assessing the Quality of Income Poverty Statisticssiteresources.worldbank.org/INTPAME/Resources/Training-Materials/... · A Framework for Assessing the Quality of

5

• Agency publications and/or websites indicate material about the terms and conditions under which official statistics are compiled and disseminated.

• The basis of choice of statistical methodologies is made available on request. 1.2.2 Internal governmental access to statistics prior to their release is publicly identified. (i) If there is internal governmental access to income poverty statistics prior to their release to the public, the public is made aware of this access. • In the event of internal governmental access to statistics prior to release, the public is

made aware of who has access and at what point of the compilation process access is given.

1.2.3 Products of statistical agencies/units are clearly identified as such. (i) The products of statistical agencies are clearly identified. • Data released to the public are clearly identified as the data producing agency’s product. • Attribution to data producing agency is made when income poverty statistics are used or

reproduced. 1.2.4 Advance notice is given of major changes in methodology, source data, and statistical techniques. (i) Users of statistics are made aware in advance of major changes in methodology, source data, and statistical techniques. • Advance notice is given when major changes to the statistical methodology are to be

implemented. 1.3 Ethical standards – Policies and practices are guided by ethical standards. 1.3.1 Guidelines for staff behavior are in place and well known to the staff. (i) There are clear ethical guidelines for official statistics. • There are clear set of guidelines outlining correct behavior when the agency or its staff

are confronted with potential conflict of interest situations. (ii) The ethical standards are followed by staff of the statistical agencies • Management enforces guidelines. • Staff are reminded periodically of the guidelines.

Page 12: A Framework for Assessing the Quality of Income Poverty Statisticssiteresources.worldbank.org/INTPAME/Resources/Training-Materials/... · A Framework for Assessing the Quality of

6

2. Methodological soundness – The methodological basis for the statistics follows internationally accepted standards, guidelines, or good practices. 2.1 Concepts and definitions – Concepts and definitions used are in accord with internationally accepted statistical frameworks. 2.1.1 The overall structure in terms of concepts and definitions follows internationally accepted standards, guidelines, or good practices. (i) The concept and definition of income poverty follow international standards, guidelines and good practices. • Concept and definition follow methodologies used by the UN System or other

international organizations (e.g., “A Class of Decomposable Poverty Measures” by Foster, J.E., J. Greer and E. Thorbecke, “Poverty lines in Theory and Practice Vol. 1” by M. Ravallion, “Defining and Measuring Poverty” by M. Lipton, “Guidelines for Constructing Consumption Aggregates for Welfare Analysis” by A. Deaton and S. Zaidi, World Bank PRSP Sourcebook).

2.2 Scope – The scope is in accord with internationally accepted standards, guidelines, or good practices. 2.2.1 The scope is broadly consistent with internationally accepted standards, guidelines, or good practices. (i) The scope of income poverty statistics is consistent with internationally accepted standards, guidelines, or good practices. • Lateral coverage, in terms of geographical boundaries and/or socioeconomic groups is

adequate. • Temporal coverage (the time period for which estimates are required) is adequate. • Sufficiently detailed data on household income and consumption, certain demographic

and socioeconomic characteristics of the household, and prices are available. • Non-market transactions such as home consumption of food, non-market labor, gifts, in-

kind payments and owner-occupied dwelling services are measured. 2.3 Classification/sectorization – Classification and sectorization systems are in accord with internationally accepted standards, guidelines, or good practices. 2.3.1 Classification/sectorization systems used are broadly consistent with internationally accepted standards, guidelines, or good practices. (i) The classification of income/expenditure/consumption statistics complies with international standards or guidelines. • Classification is based on Classification of Individual Consumption by Purpose

(COICOP). (ii) The delimitation of person and household is consistent with international standards or guidelines.

Page 13: A Framework for Assessing the Quality of Income Poverty Statisticssiteresources.worldbank.org/INTPAME/Resources/Training-Materials/... · A Framework for Assessing the Quality of

7

• Delimitation is based on UN Census Guidelines. (iii) The classification of prices complies with international standards or guidelines. • Classification is based on “Consumer Price Indices: An ILO Manual.” • Consumption baskets used for price comparison take account of different consumption

patterns of different socioeconomic groups. 2.4 Basis for recording – Data are recorded according to internationally accepted standards, guidelines, or good practices. 2.4.1 Recording system follows internationally accepted standards, guidelines, or good practices. (i) Time frame of recording is appropriate. • The choice of recall period, reference period, survey period and recording techniques

(oral interviews, diaries, observation, etc) follows recommendations suggested by UN and other international agencies.

(ii) The value of non-market transactions approximates market prices. • Market prices are used to value final and intermediate consumption of own production

and other non-monetary transactions. (iii) Recording is done on an accrual basis. • Expenditures and consumption are recoded when goods and services are consumed rather

than acquired. • Incomes are recorded on an accrual rather than cash basis (e.g., adjustments for seasonal

income).

Page 14: A Framework for Assessing the Quality of Income Poverty Statisticssiteresources.worldbank.org/INTPAME/Resources/Training-Materials/... · A Framework for Assessing the Quality of

8

3. Accuracy and reliability – Source data and statistical techniques are sound, and output data sufficiently portray reality.

3.1 Source data – Source data available provide an adequate basis to compile statistics. 3.1.1 Source data are collected from comprehensive data collection programs that take into account country-specific conditions. (i) Income poverty statistics are collected through a regular household survey program. • Survey design is appropriate, i.e., modules include income/expenditure/consumption,

prices, household characteristics. • Survey frame employs appropriate survey units (e.g., individual, household, community)

according to the objective of the survey, minimizes undercoverage, overcoverage and conforms to the target population, and is updated regularly.

• Coverage is comprehensive, i.e., target population is adequately covered. • Sampling employs internationally accepted techniques (e.g., sampling design, estimation

method, sample size, stratification). • Frequency of the surveys is sufficient. (ii) Administrative data and census data are used. • Administrative data and census data are used to supplement source data and ensure data

consistency. 3.1.2 Source data reasonably approximate the definitions, scope, classifications, valuation, and time of recording required. (i) Specific procedures are used to improve the coverage, classification, valuation, and timing of information received by the data producing agency from various data sources. • Specific procedures have been developed to adjust data from various data sources to

improve coverage, classification, and valuation and conform to internationally accepted standards, guidelines or good practices.

3.1.3 Source data are timely. (i) Data collection system provides for the timely receipt of source data and detailed data. • Respondents are made aware of the deadlines set for reporting. • The compiling agency employs rigorous follow-up procedures to ensure the timely

receipt of respondents’ data. 3.2 Statistical techniques – Statistical techniques employed conform with sound statistical procedures. 3.2.1 Data compilation employs sound statistical techniques.

Page 15: A Framework for Assessing the Quality of Income Poverty Statisticssiteresources.worldbank.org/INTPAME/Resources/Training-Materials/... · A Framework for Assessing the Quality of

9

(i) Data compilation employ sound statistical techniques. • Data management procedures such as interviewer field checks, computer generated

warnings, relationships verification, elimination of outliers and data validation have been developed to minimize processing errors.

3.2.2 Other statistical procedures (e.g., data adjustments and transformations, and statistical analysis) employ sound statistical techniques. (i) Data adjustments and transformations employ sound statistical techniques. • Imputation methods, estimation techniques (e.g., sampling weights, calibration weights),

adjustments for inflation and spatial variations of prices and adjustments for seasonality employ sound statistical techniques.

(ii) Internationally accepted statistical methods are used to handle non-sampling errors. • Problems regarding non-responses, recall errors, reporting errors, respondents effects,

interviewer effects, inappropriate instrument design are addressed.

(iii) Appropriate adjustments are made for inadequate sample coverage. • If there is a sizeable part of the population that is not covered by sources used for regular

compilation of income poverty statistics, under-coverage adjustments are made. 3.3 Assessment and validation of source data– Source data are regularly assessed and validated. 3.3.1 Source data – including censuses, sample surveys and administrative records are routinely assessed, e.g., for coverage, sample error, response error, and non-sampling error; the results of the assessments are monitored and made available to guide planning. (i) Accuracy of the survey-based information is routinely assessed. • Source data - including sample surveys, censuses and administrative data – are routinely

assessed and their results are made available. • The information is available for coverage, sampling errors, non-sampling errors and non-

response of the surveys/censuses. • Sampling standard errors of survey estimates in order to form confidence intervals for

population values are provided, especially when the estimates are based on a small sample.

3.4 Assessment and validation of intermediate data and statistical outputs – Intermediate results and statistical outputs are regularly assessed and validated. 3.4.1 Main intermediate data are validated against other information where applicable.

Page 16: A Framework for Assessing the Quality of Income Poverty Statisticssiteresources.worldbank.org/INTPAME/Resources/Training-Materials/... · A Framework for Assessing the Quality of

10

(i) Aggregates from surveys are compared against independent data sources and statistical frameworks. • Survey results are compared against national accounts, census results, administrative

data, retail trade statistics, production statistics, reports of individual companies, international trade statistics, price statistics, socio-demographic data.

3.4.2 Statistical discrepancies in intermediate data are assessed and investigated. (i) Statistical discrepancies are routinely assessed and investigated. • Post-survey data analysis is conducted to monitor statistical discrepancies. • Provision is made for immediate follow-up to reconcile data inconsistencies. 3.4.3 Statistical discrepancies and other potential indicators of problems in statistical outputs are investigated (i) Errors and omissions are monitored. • Systematic processes are in place to monitor errors and omissions and address data

problems. (ii) Statistical discrepancies between first published and revised income poverty statistics are investigated. • Causes of statistical discrepancies are investigated and documented. 3.5 Revision studies – Revisions, as a gauge of reliability, are tracked and mined for the information they may provide. 3.5.1 Studies and analyses of revisions are carried out routinely and used to inform statistical processes. (i) Revisions to income poverty statistics are periodically assessed. • Studies are regularly conducted to determine the source and direction of revised

estimates. • Users of data are informed of the causes of revisions to the data.

Page 17: A Framework for Assessing the Quality of Income Poverty Statisticssiteresources.worldbank.org/INTPAME/Resources/Training-Materials/... · A Framework for Assessing the Quality of

11

4. Serviceability – Statistics are relevant, timely, consistent, and follow a predictable revisions policy. 4.1 Relevance – Statistics cover relevant information on the subject field. 4.1.1 The relevance and practical utility of existing statistics in meeting user's needs are monitored. (i) Specific actions are taken to ensure income poverty statistics collected adequately respond to users’ needs. • There is an established process of regular consultation with users (e.g., users’ surveys). • The data producing agency regularly participates in international statistical meetings and

seminars organized by international and regional organizations. (ii) Statistics cover relevant information in order to enable the estimation of incidence, magnitude and distribution of poverty. • In addition to income poverty data derived from aggregates (e.g., per capita

income/expenditure/consumption, prices, poverty line), information on household, demographic and socio-economic characteristics required for poverty analysis is available, preferably at the household level.

4.2 Timeliness and periodicity –Timeliness and periodicity follow internationally accepted dissemination standards. 4.2.1 Timeliness follows dissemination standards. (i) The timeliness of income poverty statistics follow international recommendations. • Income poverty statistics are disseminated with in 6-12 months after the reference period

as recommended as best practice. 4.2.2 Periodicity follows dissemination standards. (i) The periodicity of income poverty statistics follow international recommendations. • Income poverty statistics are disseminated at least every 3-5 years as recommended as

best practice. 4.3 Consistency – Statistics are consistent within a dataset and over time, and with other major data sets. 4.3.1 Statistics are internally consistent (e.g., accounting identities observed). (i) Income poverty statistics are internally consistent. • Cross-checking within the survey, across geographic areas and sub-groups of population

is in place.

Page 18: A Framework for Assessing the Quality of Income Poverty Statisticssiteresources.worldbank.org/INTPAME/Resources/Training-Materials/... · A Framework for Assessing the Quality of

12

4.3.2 Statistics are consistent or reconcilable over a reasonable period of time. (i) Income poverty statistics are consistent over time. • Statistical methodologies used to compile income poverty statistics are consistent over

time. • When methodological changes are introduced, information on their possible impact on

the comparability of data over time is provided. • When methodological changes are introduced, an attempt is made to revise the historical

series as far back as data permit. • Breaks in series are identified and explained. 4.3.3 Statistics are consistent or reconcilable with those obtained through other data sources and/or statistical frameworks. (i) Income poverty statistics are consistent or reconcilable with those obtained through other surveys, data sources and/or statistical frameworks. • Statistical methodologies between household income/expenditure surveys are consistent. • Classification and definition are consistent with censuses and other data sources. • Results are checked against national accounts, price statistics and other survey/census

results. 4.4 Revision policy and practice – Data revisions follow a regular and publicized procedure. 4.4.1 Revisions follow a regular, well established, and transparent schedule. (i) Revisions (of provisional estimates, weight updates, and methodology) follow a regular, well-established and transparent schedule. • Users are informed of the schedule of revisions of preliminary data and of the period to

which they relate. (ii) The policy and practice of revising income poverty statistics follows a publicly known process. • Revision policy is transparent and available to users. 4.4.2 Preliminary data are clearly identified. (i) Preliminary data or first estimates are clearly identified in statistical releases. • Users are made aware that the initially published data are preliminary and subject to

revision. • If possible, information on the likely direction of the revisions is provided. 4.4.3 Studies and analyses of revisions are made public. (i) The causes of revisions are made available to users.

Page 19: A Framework for Assessing the Quality of Income Poverty Statisticssiteresources.worldbank.org/INTPAME/Resources/Training-Materials/... · A Framework for Assessing the Quality of

13

• Documentation of data revision methodologies is made available showing directions and

magnitude of revisions, and reasons for revision.

Page 20: A Framework for Assessing the Quality of Income Poverty Statisticssiteresources.worldbank.org/INTPAME/Resources/Training-Materials/... · A Framework for Assessing the Quality of

14

5. Accessibility – Data and metadata are easily available and assistance to users is adequate. 5.1 Data accessibility – Statistics are presented in a clear and understandable manner, forms of dissemination are adequate, and statistics are made available on an impartial basis.

5.1.1 Statistics are presented in a way that facilitates proper interpretation and meaningful comparisons (layout and clarity of text, tables, and charts). (i) The dissemination of income poverty data is commensurate with users’ needs. • Income poverty data offer adequate detail and time series. • Analysis of current period income poverty estimates is available. • Disaggregated data (e.g., household level) are accessible. 5.1.2 Dissemination media and formats are adequate. (i) The dissemination media and formats for income poverty data are commensurate with user's needs. • There are mechanisms to assess user needs and the dissemination media and formats

reflect these needs. 5.1.3 Statistics are released on a pre-announced schedule. (i) Income poverty statistics are released according to an advance release calendar. • Income poverty statistics are released according to pre-announced release schedule. 5.1.4 Statistics are made available to all users at the same time. (i) Statistics are made available to all users at the same time. • The data are released simultaneously to all interested users on the date and/or time

specified in the release schedule. 5.1.5 Non-published (but non-confidential) sub-aggregates are made available upon request. (i) Non-published sub-aggregates are made available to statistics users. • Sub-aggregates of income, expenditure, consumption and price are made available upon

request. 5.2 Metadata accessibility – Up-to-date and pertinent metadata are made available. 5.2.1 Documentation on concepts, scope, classifications, basis for recording, data sources, and statistical techniques is available, and differences from internationally accepted standards, guidelines, or good practices are annotated.

Page 21: A Framework for Assessing the Quality of Income Poverty Statisticssiteresources.worldbank.org/INTPAME/Resources/Training-Materials/... · A Framework for Assessing the Quality of

15

(i) Income poverty statistics metadata provide users with information about concepts, definitions and classifications. • Comprehensive documentation of statistical methodologies (e.g., concepts, definitions,

classification, target population, frame, sample design, sample size, sample stratification, data collection, data processing), data analysis, data systems (e.g., data codes, organization, format), data quality is available.

5.2.2 Levels of detail are adapted to the needs of the intended audience. (i) Different levels of component detail are provided to meet users’ requirements. • Depending on the intended audience and purposes, data of different degree of

aggregation (e.g., household, community), sub-components (e.g., income/expenditure/consumption components, price components) and additional data (e.g., demographic, socioeconomic, geographic information) are made available.

5.3 Assistance to users – Prompt and knowledgeable support service is available. 5.3.1 Contact person for each subject field is publicized. (i) There are provisions to provide assistance to users. • Prompt and knowledgeable service and support are available to users. All statistical

releases identify specific individuals who may be contacted. 5.3.2 Catalogues of publications, documents, and other services, including information on any charges, are widely available. (i) Information on publication, documentation, and services, including on any charges, is made widely available and updated regularly. • Catalogues are made available to users and updated regularly.