1. 2 the effects of survey design features and economic conditions on business survey response rates...

1

2

The Effects of The Effects of Survey Design Features and Survey Design Features and

Economic Conditions on Economic Conditions on Business Survey Response RatesBusiness Survey Response Rates

Diane K. WillimackU.S. Census Bureau

Economic Census

10

11

3

Outline

Motivation– Economic Census

Conceptual framework for nonresponse in establishment surveys

Data analysis, results, and conclusions– Number of pages– Economic conditions

4

U.S. Economic Census

Establishment level data Collected every 5 years

– Reference year ends in 2 or 7– Collection year is the following year

5

U.S. Economic Censuscontinued

Self-administered– Paper mail-out/mail-back– 2002: Electronic via downloadable software

Tailored by industry– 2002: 520 different questionnaire versions

6

Motivation

Changes in format affecting the 2002 Economic Census– Paper size: Legal Letter– Page layout: 2 columns 1 column

Number of pages increased

Response rates decline???

Thank you

for completing

the 2002

Econom

ic Census

8

The Data

1992, 1997, and 2002 Economic Censuses– Check-in rates

• Receipts + 2nd time Undeliverables Mail Counts

– 1st week of June, 1993, 1998, 2003

Unit of analysis = questionnaire versionquestionnaire version n = 1459 questionnaire versions

9

Check-In Rates by Year,1st week of June of the Collection Year

Census Year

Number of Pages

(Range)

Mean Check-in Rates Across

All Versions

1992 2 – 14 77.34%

1997 2 – 16 69.50%

2002 3 – 27 65.19%

14

Research Question

1) Was the effect of the number of pages on response rates different for the 2002 Economic Census than for previous censuses?

15

A Conceptual Framework forBusiness Survey Participation

Out of Survey Organization Control Under Survey Organization Control

External Environment Survey Design

The Business

ResponseBurden

BusinessGoals

Survey Participation

DecisionSource: Willimack, D.K., Nichols, E., and Sudman, S., (2002), “Understanding Unit and Item Nonresponse in Business Surveys,” in Survey Nonresponse, Groves, et al. (eds.) New York: Wiley.

The Respondent

16

Defining the Model

Concept

(External Environment;

Characteristics of the Business;

Characteristics of the Respondent;

Survey Design)

Measure

Economic Conditions %GDP, %Employment

Industrial Sector

----------------------

#pp for the form, availability of electronic mode

Response Rate = f

17

The Model

Concept

(External Environment;

Characteristics of the Business;

Characteristics of the Respondent;

Survey Design)

Measure

Economic Conditions %GDP, %Employment

Industrial Sector

----------------------

#pp for the form, availability of electronic mode

Response Rate = f

18

Main Effects ModelMain Effects Model:What is the relationship, regardless of

the census year?

Response Rate = a + b (#pages)

+ [effects of economic variables] + gi (Sector*) + h (Mode)

+ [error term]

* The omitted variable among the Sector dummies is Mining.

19

Care in Interpreting Results

Behavioral model– Explanatory, NOT predictiveNOT predictive– Estimated coefficients

• Relative magnitude• Direction (+ or -)• Significance

20

Main Effects Model

Estimated Coefficient

Significantly different from 0? p-value

Intercept 81.8181.81 YesYes <.0001<.0001

# Pages -1.13-1.13 YesYes <.0001<.0001

Availability of Electronic Mode -6.91-6.91 YesYes <.0001<.0001

21

Research Question

“Does 2002 differ from previous censuses?”

H0: 1992 = 1997 = 2002

Pairwise comparisons

22

Interaction Effects Model:Interaction Effects Model:Does the effect of pages differ

by census year?

= a + b92(#pp) + cc9292(#pp)(#pp)2 2 + d+ d9292 (#pp) (#pp)33

+ b97(#pp) + cc9797(#pp)(#pp)2 2 + d+ d9797 (#pp) (#pp)33

+ b02(#pp) + cc0202(#pp)(#pp)2 2 + d+ d0202 (#pp) (#pp)33

+ [effect of economic variables]

+ gi (Sector*)

+ h (Mode)

+ [error term]

Response Rate

* The omitted variable among the Sector dummies is Mining.

23

Does the effect of pages differ by census years?

Testing HTesting H00: 1992 = 1997 = 2002: 1992 = 1997 = 2002

Reject H0F-stat

p-value

#pp H0: b92 = b97 = b02 Yes 27.23p<.0001

(#pp)2 H0: c92 = c97 = c02 Yes 18.28p<.0001

(#pp)3 H0: d92 = d97 = d02 Yes 12.04p<.0001

24

Interaction Effects Model:Interaction Effects Model:Does the effect of pages differ

by census year?

Page Term Year

Estimated Coefficient

Differ from 1997?

Differ from 2002?

#pp 1992 -9.94

1997 -11.92

2002 -5.78

(#pp)2 1992 1.14

1997 1.23

2002 0.33

(#pp)3 1992 -0.04

1997 -0.04

2002 -0.01

25

Interaction Effects Model:Interaction Effects Model:Does the effect of pages differ

by census year?

Page Term Year

Estimated Coefficient

Differ from 1997?

Differ from 2002?

#pp 1992 -9.94 p=.0028 p<.0001

1997 -11.92 p<.0001

2002 -5.78

(#pp)2 1992 1.14

1997 1.23

2002 0.33

(#pp)3 1992 -0.04

1997 -0.04

2002 -0.01

26

Interaction Effects Model:Interaction Effects Model:Does the effect of pages differ

by census year?

Page Term Year

Estimated Coefficient

Differ from 1997?

Differ from 2002?

#pp 1992 -9.94 p=.0028 p<.0001

1997 -11.92 p<.0001

2002 -5.78

(#pp)2 1992 1.14 p=.5278 p<.0001

1997 1.23 p<.0001

2002 0.33

(#pp)3 1992 -0.04

1997 -0.04

2002 -0.01

27

Interaction Effects Model:Interaction Effects Model:Does the effect of pages differ

by census year?

Page Term Year

Estimated Coefficient

Differ from 1997?

Differ from 2002?

#pp 1992 -9.94 p=.0028 p<.0001

1997 -11.92 p<.0001

2002 -5.78

(#pp)2 1992 1.14 p=.5278 p<.0001

1997 1.23 p<.0001

2002 0.33

(#pp)3 1992 -0.04 p=.7564 p<.0001

1997 -0.04 p<.0001

2002 -0.01

28Number of Pages

Re

sp

on

se

Ra

te

Estimated ModelEstimated Model19921992

29Number of Pages

Re

sp

on

se

Ra

te

Estimated ModelEstimated Model19971997

30Number of Pages

Re

sp

on

se

Ra

te

Estimated ModelEstimated Model20022002

32

Research Questions

1) Was the effect of the number of pages on response rates different for the 2002 Economic Census than for previous censuses?

33

Research Question #1 Conclusion

Negative effect– #pages response rates – Caution: Behavioral model vs. Controlled

Experiment• Cannot identify causality• Cannot isolate factors

34

Research Question #1 Conclusion

continued

Reject H0: 1992 = 1997 = 2002

– 2002 differed from the previous censuses (per results from pairwise comparisons)

Negative effect of #pages on economic census response appears to have diminished over time

35

A Conceptual Framework forBusiness Survey Participation

Out of Survey Organization Control Under Survey Organization Control

External Environment Survey Design

The Business

ResponseBurden

BusinessGoals

Survey Participation

DecisionSource: Willimack, D.K., Nichols, E., and Sudman, S., (2002), “Understanding Unit and Item Nonresponse in Business Surveys,” in Survey Nonresponse, Groves, et al. (eds.) New York: Wiley.

The Respondent

36

Household SurveyLiterature

Better economic times refusal rates increased (in a household survey)– Harris-Kojetin and Tucker, 1999, “Exploring the

Relation of Economic and Political Conditions with Refusal Rates to a Government Survey,” JOS, 15:167-184

37

Research Questionscontinued

1) Was the effect of the number of pages on response rates different for the 2002 Economic Census than for previous censuses?

2) Did economic conditions affect economic census response rates?

38

The Model

Concept

(External Environment;

Characteristics of the Business;

Characteristics of the Respondent;

Survey Design)

Measure

Economic Conditions Economic Conditions %%GDP, %GDP, %EmploymentEmployment

Industrial Sector

----------------------

#pp for the form, availability of electronic mode

Response Rate = f

39

Main Effects ModelMain Effects Model:What is the relationship, regardless of

the census year?

Response Rate = a + b (#pages)

+ e (%GDP)

+ f (%Employment)

+ gi (Sector*) + h (Mode)

+ [error term]

* The omitted variable among the Sector dummies is Mining.

40

Main Effects Model

Estimated Coefficient

Significantly different from 0? p-value

%GDP -0.12-0.12 NoNo 0.41930.4193

%Employment -1.11-1.11 YesYes <.0001<.0001

41

Interaction Effects Model:Interaction Effects Model:Do the effects of economic conditions

differ by census year?

= a + [effect of page variables]

+ e+ e92 92 (%(%GDP)GDP) + f+ f9292 (% (%Employment)Employment)

+ e+ e97 97 (%(%GDP)GDP) + f+ f9797 (% (%Employment)Employment)

+ e+ e02 02 (%(%GDP)GDP) + f+ f0202 (% (%Employment)Employment)

+ gi (Sector*)

+ h (Mode)

+ [error term]

Response Rate

* The omitted variable among the Sector dummies is Mining.

42

Do the effects of economic conditions differ by census year?

Testing HTesting H00: 1992 = 1997 = 2002: 1992 = 1997 = 2002

H0: 1992 = 1997 = 2002 Reject H0F-stat

(p-value)

%%GDPGDP No 0.65(0.5209)

%%EmploymentEmployment No 1.25(0.2861)

43

Research Questions

1) Was the effect of the number of pages on response rates different for the 2002 Economic Census than for previous censuses?

2) Did economic conditions affect economic census response rates?

44

Research Question

3) Does the effect of number of pages on response rates vary with economic conditions?

45

Mixed Effects Models

+ j+ ji i (#pp) * (%(#pp) * (% GDP) GDP)

+ k+ kii (#pp) (#pp)22 * (% * (% GDP) GDP)

+ m+ mii (#pp) * (% (#pp) * (% Emply) Emply)

+ n+ nii (#pp) (#pp)22 * (% * (% Emply) Emply)

+

Response Rate

= a

+ bi (#pp)

+ ci (#pp)2

+ di(#pp)3

+ ei (% GDP)

+ fi (% Emply)

+ g (Sector*)

+ h (Mode)

* The omitted variable among the Sector dummies is Mining.

For each census year (i = 1992, 1997, 2002):

46

Summary of Results Under the Mixed Effects Model

Variables that become non-significant non-significant (with p-values)– (#pp)2 for 1992 (0.1159)– (#pp)3 for 2002 (0.3753)

47

Summary of Results Under the Mixed Effects Model

continued

%GDP (0.0282)(#pp) * (%GDP) (0.0103)(#pp)2 * (%GDP) (0.0087)

%Emply (0.0322)(#pp) * (%Emply ) (0.0117)(#pp)2 * (%Emply ) (0.0082)

Economic variables that are significantsignificant (all are for 2002)(all are for 2002) (with p-values)

48

What was happening in the U.S. economy in 2001-2003?

Economic recovery – beginning to emerge from a recession.

49

Economic Trends 1991-2003

-2.00

-1.00

0.00

1.00

2.00

3.00

4.00

5.00

1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003

Year

Per

cent

Cha

nge

GDP Employment

50

What was happening in the U.S. economy in 2001-2003?

Economic recovery – beginning to emerge from a recession. GDP increasing sharply Employment declining, “bottoming out”

Unstable economy– Sharp opposite movements in GDP and

Employment

51

What is the nature of the interaction between economic

conditions and #pages? GDP and Employment effects tend

to be offsetting. GDP moderates the page effect on

response rates. Emply exacerbates the page effect on

response rates. Magnitude of Employment effect is

larger.

53

What is the nature of the interaction between economic

conditions and #pages? continued

Employment effect

Number of pages

Re

spo

nse

Ra

tes

54

Economic Effects Postscript: Economic Theory of Economic Theory of Lagged ExpectationsLagged Expectations

In-depth examination of effects of quarterly employment change on response rates– Statistically significant impact of changes

4 quarters and 3 quarters prior to beginning of economic census data collection.

55

Interpreting the Economic Effect

Unstable economy– Employment decline and GDP growth

Uncertainty– Riskier to divert resources to non-

productive activities

Cost-benefit– Survey response results in a cost with no

associated productive benefit.

56

Interpreting the Economic Effect continued

The bottom line– In a risky economic environment…– Employees are too valuable to devote to

filling out forms.– Forms that appear longer (#pages) are

presumed to take longer and be more costly to complete…

– Response rates are in jeopardy!

57

Summary:Number of Pages

Negative effect– #pages response rates – Cubic functional relationship

Negative effect of #pages on economic census response appears to have diminished over time

58

Summary:Economic Conditions

Evidence that economic conditions interact with form length to affect response rates– Mainly through changes in employment

Survey response = non-productive cost– Corroborates qualitative research findings