1 surveys: collecting policy relevant data rachel govoni-smith kinnon scott, decrg january 17, 2006

1

Surveys: Collecting Policy Relevant Data

Rachel Govoni-SmithKinnon Scott, DECRGJanuary 17, 2006

2

Sources-

• The Impact of Economic Policies on Poverty and Income Distribution: Evaluation Techniques and Tools, eds. Francois Bourguignon and Luiz Al Pereira da Silva, World Bank, Washington, D.C., 2003.– Scott, Kinnon (2003) “Generating Relevant Household

Level Data: Multi-topic Household Surveys”

• Muñoz, Juan and Kinnon Scott (2005) “Household Surveys and the Millennium Development Goals”, report for Paris21 Task Force on Improved Statistical Support for Monitoring Development Goals

3

Household Surveys and the Impact of Economic Policies on Poverty and Income Distribution

• Estimating Incidence of Indirect Taxes• Analyzing the Incidence of Public Spending• Behavioral Incidence Analysis of Public Spending• Estimating Geographically Disaggregated

Welfare Levels and Changes• Assessing the Poverty Impact of an Assigned

Program• Ex Ante Evaluation of Policy Reforms

Micro Level

4

Household Surveys and the Impact of Economic Policies on Poverty and Income Distribution

• The Effect of Aggregate Growth on Poverty• Linking Macro-consistency Models to

Household Surveys• Partial Equilibrium; Multi-market Analysis• The 123PRSP Model• Social Accounting Matrices• Poverty and Inequality Analysis and CGE

models

Macro Level

5

Goals and Needs

Goals:

• Measure the poverty impact of economic policy

• Measure the distributional impact of economic policy

Needs:

• Rely heavily on household survey data

6

Household Surveys

• Single Topic

• In-between

• Multi-topic

7

Household Surveys

• Single Topic

• Labor Force Surveys( LFS) (ILO)

• Housing Surveys

• Census – national, UNFPA, 10 years

• In-between

• Multi-topic

8

Household Surveys

• Single Topic

• In-between

• Agricultural Surveys (FAO)

• Demographic and Health (DHS)

• Household Budget Surveys (HBS)• Multi-topic

9

Household Surveys• Single Topic• In-between

• Multi-topic• Multiple Indicator Cluster Surveys (MICS,

UNICEF)• Survey on Income and Living Conditions (SILC,

EU)• Core Welfare Indicator Surveys (CWIQ, WB)• Living Standards Measurement Study Surveys

(LSMS) and Integrated Surveys (IS) (WB)• Family Life Surveys (FLS, RAND)

10

What type of household data?

• Poverty measure: per capita or per adult equivalent consumption

• Government programs receipt, format, costs (formal and informal), use level

• Consumption of taxed goods

• Labor market participation (sector, hours, earnings)

• Income by sources

11

Census

• Accurate measure of the population of a country

• Geographic distribution of the population

• Basic demographic information

Purpose

12

Census

• Not a sample

• Universal coverage

• No sampling errors in estimates

• Some corrections for non-response may be needed

Sample

13

Census

• Short

• Trade-off between coverage and content

• Two types of errors: sampling and non-sampling

Content

DECRG: May 7 2004 Sample size

Sampling errorNon-sampling error

Sampling vs. non-sampling errors

Total error

15

Census

• Short

• Trade-off between coverage and content

• Two types of errors: sampling and non-sampling

Content

• Cost •Time•Non-response• Training

16

Census

• Demographic information: age, sex, race/ethnicity, family and household composition

• Housing information

• Others: basic education, labor, disability

Content

17

Census

• Basic needs– Subjective

– Limited monitoring use

Poverty Measurement

• Income: Panama example

•Albania: 2001 (1989)•BiH 1991 (1981)•Montenegro 2003 (1991)

– Limited use if looking at impact of policies affecting taxes, tariffs or pricing

18

Census

• Sample frame

• Link with household surveys for small area estimation

Uses

19

Poverty Indicator by Commun, Albania, 200

20

Labor Force Survey

• Direct measurement of unemployment

• General characteristics of the labor force

Purpose

21

Labor Force Surveys

• Relatively large samples

Need for precise estimates (change)

Desire to disaggregate to different geographic areas

• Individuals of working age

Sample

22

Labor Force Survey

• Characteristics of the labor force– Demographics

– Education

• Sectoral distribution of employment

• Degree of formality

• Seasonal

• Income

Content

23

Labor Force Survey

Three problems:

• LFS typically capture partial, not total, income– Under-estimate welfare (vs. NA)

– Mis-ranking of households by welfare level

Poverty Measurement

24

Venezuela: Income and Expend Survey

25

Venezuela: Social Survey

26

Labor Force Surveys, cont.

Three problems:

• LFS typically capture partial, not total, income

• Measurement Error– Labor income measurement error

– At both ends of the distribution

Poverty Measurement

27

LFS in Latin AmericaItem non-response

Salaried Self-employed

Employer All Indep-endent

Mean non- response rate

3.9% 10.2% 12.0 10.6%

Source: Feres, 1998

28

Labor Force Surveys, cont.

Three problems:• Partial vs total, income• Measurement error

• Income vs consumption measure– Potential vs actual welfare

– Smoothing

– Measurement Error

Poverty Measurement

29

Household Budget Surveys

• Inputs to national accounts on consumer expenditures

• Track changes in expenditures over time

• Track changes in the relative share of different expenditures

• Weights for the consumer price index

Purpose

30

Household Budget Surveys

• Medium size sample

• Sampling errors high at disaggregated level

• High non-response rates

• In some parts: only urban (capital city or group of large cities)

Sample

•Non response rates (Eurostat, 2003)•Bulgaria: 39.7%•Estonia, 44%•Hungary, 58.8% before replacement•Romania, 21.6 %

31

Household Budget Surveys

• Total Income

• Total Consumption

• Short Demographics

• In FSU and Central Europe: agriculture

Content

32

Household Budget Surveys

• Possible to construct both total consumption and total income

• Income may suffer from same measurement errors as LFS

Poverty Measurement

33

Household Budget Surveys

• Consumption based welfare measure

• Purpose of an HBS survey is NOT to measure welfare but to precisely measure mean expenditures on specific goods and services

• These are conflicting goals

Poverty Measurement

34

Household Budget Surveys

• Shortest possible reference periods

• Minimize number of omitted expenditures

• Good for precise measurement of regional or national means

• Because of lumpy nature of purchases, not good for comparisons among households

Need to adjust (lengthen) the reference periods used in HBS

Poverty Measurement

35

Household Budget Surveys

• Focus on expenditures– Not all expenditures are consumption

– Only purchases of durable goods and housing

Durable goods: list of items owned by household, age of items, current value

Housing: housing characteristics affecting value

Poverty Measurement

36

Household Budget Surveys

• Good for taxation issues

• Good for public (and private) transfers

• Sometimes has basic labor

• FSU and Central European countries: agriculture

• No health, education data

• Limited for other areas

Uses

37

Multi-topic Household Surveys

Those with a focus on measuring poverty

• National Socio-Economic Survey of Indonesia, SUSENAS

• Survey on Income and Living Conditions (SILC)

• Rand Family Life Surveys (FLS)

• Living Standards Measurement Study Surveys (LSMS)

38

Multi-topic Household Surveys

• Analysis of welfare levels and distribution

• Study links between welfare levels and individual and household characteristics, economic, human and social capital

• Social exclusion

• Causes of observed social outcomes

• Levels of access to, and use of, social services, government programs and spending

Purpose

39

Multi-topic Household Surveys

• Small sample sizes

• Trade-off issue: Quality and cost considerations

• Limits ability to assess programs or policies that affect small groups or small areas (over-sample)

• Infrequent in many countries (exceptions, inter alia, Indonesia, Panama, Jamaica, Peru, Ghana)

Sample

40

Multi-topic Household Surveys

ContentHousehold Demographics* Agricultural Activities*Housing* Non-farm household businesses*Education* Food consumption (purchase, produced, gift)*Health* Non-food consumption and durables*Labor* Other income (incl. public &private transfers)*Migration* Social capitalFertility* Shocks, vulnerabilityPrivatization Time UseCredit Subjective measures of welfareAnthropometrics

Note: Starred modules are those most often used.

41

Multi-topic Household Surveys

• Total consumption– Longer reference periods

– Able to calculate use value of durables and housing

• Total income– Suffers from standard measurement errors

Poverty Measurement

42

Multi-topic Household Surveys

• Poverty levels and distribution

• Social exclusion

• Public and private transfers

• Incidence analysis

• Tax policy

• Labor markets

• Education, health, social protections

• Changes in relative prices

• Monitoring (PRSP, MDGs), impact evaluation

Uses

43

Cross Section or Panel Cross Section or Panel Surveys?Surveys?

• Substantive applications

• Methodological issues

44

PPanelsanels1. Why do we need longitudinal data?

2. Designs for surveys across time

3. Advantages and uses of panels

4. Methodological issues

45

Understanding changeUnderstanding change

Longitudinal data are needed to understand the process of change, transitions between states, and the factors or events that are associated with those transitions

‘Longitudinal’ data is a catch-all phrase for a wide range of different types of studies

46

Designs for surveys across time Designs for surveys across time

Repeated cross sectional surveys (e.g. Household Budget Survey, Labour Force Survey)

• Common design for large government surveys

• New sample drawn for each survey• Carry similar questions each year• Used for trend analysis at aggregate level

47

Designs for surveys across time Designs for surveys across time Cohort Studies • Sample often based on an age group • Follow up same sample members at fairly long

intervals • Developmental data as well as social and

economic data• Data from parents, teachers associated with

cohort member

48

Designs for surveys across timeDesigns for surveys across timeRotating Panel Survey Survey of Income and Programme

Participation, USA (SIPP)

• Respondents stay in the panel for a set period of time and are rotated out systematically and replaced by new sample members.

• Used where the interviews are fairly close together (every 3 to 6 months) and respondent burden is high.

• Used where the collection of short spells e.g. a few weeks unemployed or in receipt of a particular benefit, is critical.

49

Designs for surveys across timeDesigns for surveys across time

Indefinite Life Panel Surveys e.g. Panel Study of Income Dynamics, USA – since 1968!

Living in BiH, LSMS Albania, LSMS Serbia

• Draw a sample at one point in time and follow those sample members indefinitely (or as long as the funding continues)

• Collect individual level data in household context• Repeated measures at fixed intervals (annual data

collection)

50

Panels from conference attendee Panels from conference attendee countriescountries

• Albania – 4 waves 2002 - 2005 • BiH – 4 waves 2001 - 2004• Serbia – 2 waves 2002 - 2003

51

Advantages of Panel DataAdvantages of Panel Data• Comparison of same individual over time - outcomes• Track of aspects of social change• Facilitates study of change and causal inference • Minimise the problem of inaccurate recall• Compare a person’s expectations with real change• Look at how changes in individuals’ behaviour affects their households

Identifies the co-variates of change and the relative risks of particular events for different types of people

52

Changes in Employment Changes in Employment StatusStatus

A: CROSS-SECTIONAL INFORMATION

Unemployed

Employed

2001 2007

Net change - 0.1% unemployed

53

Changes in Employment Changes in Employment StatusStatus

B: PANEL INFORMATION

Still Unemployed

Still Employed

Unemployed

Employed

2001 2007

Net change - 0.1% unemployed Actual change is 10.1

continuouslyemployed

86.7%

employed 2001but unemployed 2007

5%

continuouslyunemployed

3.2%

unemployed 2001 but employed 2007

5.1%

54

Balkan Examples

Albania - 15% of the unemployed in 2002 had made the transition to formal sector employment by 2004

BiH - About half who were poor in 2001 remained poor in 2004. Many individuals moved out of poverty.

(Cross section headcount 18% for both years)

55

Employment and the labour market Unemployment duration and exit rates Do the unemployed find stable employment? The effect of non-standard employment on

mental health Temporary jobs: who gets them, what are they

worth, and do they lead anywhere?

Family and Household Patterns of household formation and dissolution Breaking up - finances and well-being following

divorce or split The effect of parents’ employment on children's

educational attainment

56

Panel analysis

Mobility, poverty and well-being among the informally employed – Peter Sanfey European Bank for Reconstruction and Development

The origins of self employment, Leora Klapper et al, WB (soon to use Albania Panel also)

The impact of health shocks on employment, earnings and household consumption, Kinnon Scott et al

57

A SampleA Sample• Concept of ‘longitudinal household’

problematic for a panel - households change in composition over time or disappear altogether

• Individual level sample

58

Following rulesFollowing rules• All members of households interviewed at

Wave One• Children born to these original sample

members • Original members are followed as they

move house, and any new individuals who join with them are eligible to be interviewed

• New sample members are followed if they split from the original member

59

Questionnaire designQuestionnaire design

• Core content carried every wave• Rotating core questions • One-off variable components

– lifetime job history– marital and fertility history

• Variable questions to respond to new research and policy agendas

60

Attrition in panel surveysAttrition in panel surveys• Inevitable to some extent but can be

minimised• Multiple sources of attrition in a panel

– refusal to take part– respondents move and cannot be traced– non-contacts

• Worry is potential bias if people who drop out differ significantly from those who stay in

61

70

87.7 90.394.9 94.8 97.5 97.2 97

010

20304050

607080

90100

Wave 1 Wave 2 Wave 3 Wave 4 Wave 5 Wave 6 Wave 7 Wave 8

UK Panel Wave 1 RespondentsUK Panel Wave 1 RespondentsWave-on wave re-interview ratesWave-on wave re-interview rates

62

FieldworkFieldwork• respondent incentives as a ‘thank-you’• extended fieldwork period for ‘tail-enders’• refusal conversion programme• tracking procedures during fieldwork• panel maintenance between waves

– Change of Address cards to update addresses– mailing of Respondent Report– details of contacts with respondents between waves

63

Post-field checking and cleaningPost-field checking and cleaning

• Within wave consistency

• Cross wave consistency and longitudinal integrity

• Sample management– individuals within households correctly

identified across time

– issuing of sample for each wave

64

The user databaseThe user database

• Longitudinal data is complex• Provide users with database structure

which enhances usability• Consistent record structure over time• Key variables for matching and linking data

cross wave• Consistent variable naming conventions

65

Added valueAdded value

• ‘Added value’ to data set• Extensive set of derived variables • Production of weights

– household and individual levels – cross sectional and longitudinal

• Imputation of missing data• Flags to indicate imputed values

66

ConclusionsConclusions• Longitudinal panel data allows us to answer

research questions that cannot be answered with with cross-sectional data

• Provides a different view of the world - see process through the life-course not just a static picture

• Is complex (but so is the real world) - so needs to be well designed and conducted with sufficient resources to be successful

67

System of Household Surveys

• GOAL: System able to respond to evolving needs: not produce data X or survey Y– Determine data needs before they are

URGENT

– Identify appropriate instruments,

– Implement them properly, timely fashion,

– Analyze the resulting data

68

Improving the SHS• Linking Users and Producers• Providing adequate resources• Continuous Survey Program

– Not necessarily permanent survey– Benefits

• Avoid loss of capacity• Create greater levels of capacity (building on existing)• Economies of scale• Policy makers know when data will be available• Protects NSO from pressures for ad hoc surveys• Ongoing system actually allows more flexibility and

responsiveness

69

Final points

• Welfare: household surveys- always missing the homeless, street children, institutionalized population

• No one survey can meet all needs, review its purpose, coverage, content and quality before using

• Need a system of surveys that meets the needs of data users