measuring subjective expectations in surveys adeline delavande institute for social and economic...

Measuring Subjective Expectations in Surveys

Adeline Delavande

Institute for Social and Economic Research,University of Essex

2012 ESRC Research Methods FestivalSurvey Data Quality

Overview

1. Motivation

2. Measurement: how to elicit subjective expectations?

1. In developed countries

2. In developing countries

3. Data quality: Validation of answers1. Basic patterns of answers

2. Violation of properties of probabilities

3. Correlation of expectations with characteristics

4. Design issues

Motivation Social scientists and policy makers want to predict

choice behavior

Most decisions are made under uncertainty Deciding to retire Choosing a medical treatment

Typical assumption: individuals maximize subjective expected utility Basic identification problem with choice data alone: many

combinations of expectations and preferences generate same choices

Motivation

One possibility to mitigate the identification problem Elicit directly subjective expectations from survey

respondents Combine data on observed choice and subjective

expectations to make inference on preferences

Motivation

Other reasons to elicit subjective expectations from survey respondents

Find out whether there is systematic misperception in the population Do women at risk of unintended pregnancy have misperception

about effectiveness of birth control methods?

Ask expected choice behavior

Replace data on realizations Subjective expectations about survival instead of long panel data

on mortality to estimate differential mortality (Delavande and Rohwedder, forthcoming)

2. Measurement: how to elicit subjective expectations?

Eliciting subjective expectations

Attitudinal research and verbal expectations data

General Social Survey: Thinking about the next twelve months, how likely do you think it is that you will lose your job or be laid off—very likely, fairly likely, not too likely, or not at all likely?

Survey of Consumers: Now turning to business conditions in the country as a whole—do you think that during the next 12 months we’ll have good times financially, or bad times, or what?


Attitudinal research and verbal expectations data: various problems

Interpretation of answers Interpersonal comparability: Are the response

comparable across people? Very likely may mean different thing to different people

Intrapersonal comparability : Are the response comparable across event for the same individual?

Coarseness of response options Researchers gain only limited information


Eliciting probabilistic expectations Advantages:

well-defined numerical scale (comparability of answers)

Assessment of internal consistency of answers is possible Do answers respect basic properties of probabilities?

Answers can be compared with known events frequencies

Eliciting subjective probabilities in developed countries

Binary eventsTypical wording: percent chance format

Survey of Economic Expectations: What do you think is the percent chance that you will lose your job during the next 12 months?

Health and Retirement Study: What do you think is the percent chance that you will live to be 75?

My own survey: What do you think is the percentage chance that you would get pregnant during the next 12 months if you were using the birth control pill during that period?

Eliciting subjective probabilities in developed countries

Distribution of beliefs about a continuous variableTwo methods used so far:

Percent chance: elicit several points in the cumulative distribution

On a scale from 0 to 100, where 0 means no chance and 100 means you are absolutely certain, what do you think is the percent chance that your Social Security benefits will be more than $1,000 per month?

Visual format in Internet surveys (Delavande and Rohwedder, 2008)

Eliciting distribution of beliefs about a continuous variable

Eliciting subjective probabilities in developing countries

Binary events

Percent chance format (like in developed countries in a context of high literacy)“Frequentist” (consider 20 people like you)Visual format with physical objects (beans/stones)


Visual format (Delavande and Kohler, 2009)

“I will ask you several questions about the chance or likelihood that certain events are going to happen. There are 10 beans in the cup. I would like you to

choose some beans out of these 10 beans and put them in the plate to express what you think the

likelihood or chance is of a specific event happening.

One bean represents one chance out of 10.”


Distribution of beliefs about a continuous variable

physical objects used to allocate probability mass into intervals (e.g., Delavande, Gine and McKenzie, 2011)

3. Data quality: validations of expectations

Data quality: validations of expectations

Do people report their actual expectations?

Basic patterns of answers

Item non-response Tend to be low: People willing and able to answer

Respondents use the whole scale from 0 to 100 Some rounding to nearest 5 (see Manski and Molinari

2010)

Heterogeneity in answers

Often: focal answers at 0, 50, 100


0.0

5.1

.15

.2.2

5D

ens

ity

0 20 40 60 80 100r7liv75:w7 r probability to live 75+

Probability of being alive at age 75 HRS 2006


Heaping at 0, 50 and 100 may proxy for respondents’ burden

50-50: may reflect uncertainty rather than a true “1/2” probability In HRS 2006, respondents who answer 50% to the subjective

probability of survival are given a follow-up question asking whether they just do not know their survival probabilities or whether their belief is really that the chances are about 50 percent. It turns out that the fraction of 50s being simply uncertain is high – a little over 60 percent.

Rate of 50s tend to decrease with education and numeracy (Bruine de Bruin et al., 2000)

Rate of 50s varies with types of events

Violation of properties of probabilities

Do respondents understand concept of probabilities?

Delavande and Kohler (2008) ask about the probability of nested events in training questions in Malawi

Likelihood of going to the market within 2 days 2 weeks


99% of respondents provide answers coherent with probability theory:

P(market within 2 days)<=P(market within 2 weeks)

Elicited probabilities of going to the market

0

5

10

15

20

25

0 1 2 3 4 5 6 7 8 9 10

# beans

% r

es

po

nd

en

ts

Two days

Two weeks


Delavande, Gine and McKenzie (2011) ask a series of checks questions in a survey in India

Imagine I have 5 fishes, one of which is red and four of which are blue. If you pick one of these fishes without looking, how likely it is that you will pick the red fish?

Nested events Probability zero event: How likely do you think it is that you will not

catch any fish in the month of August if you go fishing 6 days a week?

Probability one event: How likely it is that you will eat fish at least once during the month of August?

Expectations and characteristics

Expectations vary with characteristics in the same way as actual outcomes

Likelihood of having to rely on family members in Malawi – by SES

Source: Delavande and Kohler (2009)

Design issues

Which design for better quality?


Comparing percent chance versus visual format (Delavande and Rohwedder, 2008)

Randomized design to ask about Social Security benefits expectations HRS Internet survey in 2007

Half sample was randomized into percent chance format (4 thresholds)

Half sample was randomized into visual format


How do the 2 methods compare? Non-response rate: less than 2% for both formats Similar survey time Respondents take only limited advantage of the

additional precision of their format Bins and balls: only 1 percent chose to use more than five

bins Percent chance: less than 3.5 percent of the answers per

bin are not multiple of 5


Unusable answers Percent chance format: respondents do not

respect the monotonicity across thresholds 20% of respondents who answer the 4 thresholds Those are less educated, less wealthy and in poorer

health Note: we did not inform respondents about the

violation (different from Dominitz and Manski, 2006) Visual format: respondents do not allocate all the

balls 3% of respondents


Central tendency and spread of the distribution The visual and percent chance format generate similar

central tendency. Respondents tend to allocate most probability mass in the

middle bin

Yet, the percent chance format generates more spread-out distributions

Average probability mass:

First bin Second bin Middle bin Fourth bin Fifth bin

Bins and balls 2.74 16.98 59.93 16.96 3.38

Percent chance 9.57 16.13 43.02 18.17 13.12


Does the visual format create anchoring towards the middle of the visual aid?

Experiment in the RAND American Life Panel Randomly allocate position of the bin containing

the point estimate (middle bin or third bin) Probability mass allocated in the bin containing

the point estimate is not sensitive to the position of that bin


How to evaluate which format yields higher quality of answers?

1. Check how uncertainty in Social Security benefits correlates with uncertainty about related outcomes (subjective probability of reform, distance to claiming age, etc…) Elicited uncertainty correlates with other sources of

uncertainty in the expected direction for both formats Both yield high-quality data from a substantive point of view


How to evaluate which format yield higher quality of answers?

2. Check whether uncertainty about future SS benefits predicts actual behavior related to retirement planning

Portfolio choice (Delavande and Rohwedder 2011) Find that higher levels of uncertainty about future benefits

hold a smaller share of their wealth in stocks in visual format only

Could also be due to smaller sample size or functional form assumptions that suit better visual format

Eliciting distribution of beliefs in developing countries Delavande, Gine and McKenzie (2011)

Experiment in India to test the sensitivity of elicited expectations to variations in three facets of the elicitation methodology: Number of beans (10 versus 20)

20 beans: more precise answers but may increase respondent’s burden

Design of the support (pre-determined with 20 intervals or self-anchored with 4 intervals based on elicited min/max) Self-anchor: more relevant to respondent but may increase

interviewer’s burden

Ordering of questions

Eliciting distribution of beliefs in developing countries

Subjective distribution elicited is robust to modifications in the elicitation design

When using a pre-determined support with many intervals, individuals with 20 beans use more intervals and have a distribution which is less uniform 20 beans does not seem to unduly increase respondent burden

and allows more precision

Compare individual-level realized distribution of daily catches during August to elicited distribution Greatest accuracy occurs when using 20 beans and the pre-determined support,

which results in a 38 percent improvement in accuracy relative to using 10 beans and a self-anchored support

Conclusions

Various ways can be used to assess expectations data quality Introduce training questions to check whether

respondents understand the concept of probabilities Correlate with characteristics Evaluate whether expectations predict actual

behavior

We may be able to improve data quality with more innovative elicitation formats

References Bruine de Bruin, W¨andi, Baruch Fischhoff, Susan G. Millstein, and Bonnie L. Halpern-Felsher (2000). “Verbal and

Numerical Expressions of Probability: ‘It’s a Fifty-Fifty Chance’.” Organizational Behavior and Human Decision Processes 81:115–31.

Delavande, Adeline, Xavier Giné and David McKenzie (2011) “Eliciting Probabilistic Expectations with Visual Aids in Developing Countries: How sensitive are answers to variations in elicitation design?”, Journal of Applied Econometrics.

Delavande, Adeline and Hans-Peter Kohler (2009) “Subjective Expectations in the Context of HIV/AIDS in Malawi”, Demographic Research.

Delavande, Adeline and Susann Rohwedder (2008). “Eliciting Subjective Expectations in Internet Surveys,” Public Opinion Quarterly.

Delavande, Adeline and Susann Rohwedder (2011). “Individuals’ Uncertainty about Future Social Security Benefits and Portfolio Choice ,” Journal of Applied Econometrics.

Delavande, Adeline and Susann Rohwedder (forthcoming). “Differential survival in Europe and the US: Estimates based on subjective probabilities of survival ,” Demography.

Manski, Charles F. and Francesca Molinari (2010). “Rounding Probabilistic Expectations in Surveys,” Journal of Business & Economic Statistics.

measuring subjective expectations in surveys adeline delavande institute for social and economic...

Documents