1

Introductory Overview Lecture

Current Practices in Current Practices in Questionnaire Development, Questionnaire Development, Evaluation, and Testing for Evaluation, and Testing for

Establishment SurveysEstablishment Surveys

Diane K. WillimackU.S. Census Bureau

2

The Response Process in The Response Process in Establishment Surveys and Establishment Surveys and Implications for PretestingImplications for Pretesting

Diane K. WillimackU.S. Census Bureau

Survey

3

Outline

Establishments vs. households

The response process in establishment surveys

Questionnaire development, evaluation and testing methods for establishment surveys

Conclusions and challenges

4

Establishments vs. Households

What are establishments?– Economic units– Locations where business is conducted or

industrial operations performed (FCSM, 1988)

How are establishment surveys different?– Population, data, and survey design– Response process

5

Establishment Surveys Are Different!

Household Household SurveysSurveys

Establishment Establishment SurveysSurveys

Qs are about Self Business

Types of dataAutobiographical, Attitudes, Behaviors

Financial, Quantities

Data sources MemoryRecords & R’s knowledge of them

Respondent Identity Self or Proxy Informant

6

Establishment Surveys Are Different! Continued

Household Household SurveysSurveys

Establishment Establishment SurveysSurveys

R selection; R rule

Single R or Selected proxy

Multiple informants

ModeInterviewer administered

Self-administered

AuthorityVoluntary or Mandatory

Mandatory

Response Process

CognitiveCognitive and Organizational

7

The Response Process in Establishment Surveys

Cognitive response model

PLUS

Organizational processes

8

Tourangeau’s (1984)

Cognitive Response Model

1. Comprehension

2. Retrieval

3. Judgment

4. Communication Survey

9

Response Process Model for Establishment Surveys

Sudman et al., ICES-2

Organizational Organizational AspectsAspects

4. Comprehension

5. Retrieval

6. Judgment

7. Communication

1.1.

2.2.

3.3.

8.8.

Business Survey

10

Response Process Model for Establishment Surveys

Encoding in Memory / Record FormationEncoding in Memory / Record Formation

Selection / Identification of Respondent(s)Selection / Identification of Respondent(s)

Assessment of Priorities (Motivation)Assessment of Priorities (Motivation)

4. Comprehension

5. Retrieval

6. Judgment

7. Communication8.8.

1.1.

2.2.

3.3.

Release of the DataRelease of the Data

Business Survey

from Memory and / or Recordsfrom Memory and / or Records

11

The Response Process in The Response Process in Establishment Surveys Establishment Surveys

1.1. Encoding / record formationEncoding / record formation2. Respondent selection / identification

3. Assessment of priorities

4. Comprehension of the data request

5. Retrieval of data

6. Judging the adequacy of the response

7. Reporting the response

8. Release of the data

12

1. Encoding / Record Formation

Business Survey

Memory Records

13

1. Encoding / Record Formation, continued

Needs of company records– Management– Regulations– Standards –

• Generally Accepted Accounting Practices (GAAP)

14

1. Encoding / Record Formation, continued

Encoding includes – R’s knowledge of records encoded in memory

15

1. Encoding / Record Formation, continued

If R’s knowledge of records is – – Flawed– Incomplete– Nonexistent

Response errorsResponse errors

16

1. Encoding / Record Formation, continued

In establishment surveys, encodingencoding must include – Knowledge

of RecordsPersonal Knowledge

Business Survey

17

The Response Process in Establishment Surveys

1. Encoding / record formation

2.2. Respondent selection / identificationRespondent selection / identification3. Assessment of priorities

4. Comprehension of the data request

5. Retrieval of data

6. Judging the adequacy of the response

7. Reporting the response

8. Release of the data

18

2. Respondent Selection / Identification

Most knowledgeable R

19

2. Respondent Selection / Identification, continued

Distributed knowledge– Multiple respondents

Organizational hierarchies– Authority

Respondent selection is under the Respondent selection is under the control of the business / establishment.control of the business / establishment.

20

The Response Process in Establishment Surveys

1. Encoding / record formation

2. Respondent selection / identification

3.3. Assessment of prioritiesAssessment of priorities4. Comprehension of the data request

5. Retrieval of data

6. Judging the adequacy of the response

7. Reporting the response

8. Release of the data

21

3. Assessment of Priorities

Financial reporting priorities1. Requests from mgmt / investors2. Annual Reports, SEC & IRS filings3. Qtr & monthly financial reports4. Regulatory requirements

(govt & non-govt)5. Other govt data requests

(Census Bureau)6. Everybody else

(e.g., trade associations)

22

3. Assessment of Priorities, continued

Financial reporting priorities Job performance & evaluation criteria Professional standards Pride

R’s MotivationR’s Motivation

23

The Response Process in Establishment Surveys

1. Encoding / record formation

2. Respondent selection / identification

3. Assessment of priorities (Motivation)

4.4. Comprehension of the data requestComprehension of the data request5. Retrieval of data

6. Judging the adequacy of the response

7. Reporting the response

8. Release of the data

24

4. Comprehension

Interpreting the form– E.g., mailing label = reporting unit

Understanding the data request– Types and levels of data requested

Interpreting the meaning of the questions– Language, terminology, jargon

25

The Response Process in Establishment Surveys

1. Encoding / record formation

2. Respondent selection / identification

3. Assessment of priorities (Motivation)

4. Comprehension of the data request

5.5. Retrieval of dataRetrieval of data6. Judging the adequacy of the response

7. Reporting the response

8. Release of the data

26

5. Retrieval of Data

Data sourcesRecords

Personal Knowledge

Knowledge of Records

27

5. Retrieval of Data, continued

Retrieval incorporates –– Cognitive step – retrieving knowledge of

data sources– Access to records– Physical act of retrieving data

• Extracting information from files• Consulting multiple sources• Compiling information

28

5. Retrieval of Data, continued

Different Rs differ in their – – Knowledge of records– Access to records– Ability to retrieve data

29

The Response Process in Establishment Surveys

1. Encoding / record formation

2. Respondent selection / identification

3. Assessment of priorities (Motivation)

4. Comprehension of the data request

5. Retrieval of data

6.6. Judging the adequacy of the responseJudging the adequacy of the response7. Reporting the response

8. Release of the data

30

6. Judging the Adequacy of the Response

Does R’s answer retrieved from records or memory match the data being requested?

Business

Survey

RecordsMemory

31

6. Judging the Adequacy of the Response , continued

Does R’s answer retrieved from records or memory match the data being requested?

R doesn’t have the requested data in records. How does R estimate that figure?

Business Survey

32

6. Judging the Adequacy of the Response, continued

Does R’s answer retrieved from records or memory match the data being requested?

R doesn’t have the requested data in records. How does R estimate that figure?

R thinks, “My answer is good enough.”

Business Survey

33

The Response Process in Establishment Surveys

1. Encoding / record formation

2. Respondent selection / identification

3. Assessment of priorities (Motivation)

4. Comprehension of the data request

5. Retrieval of data

6. Judging the adequacy of the response

7.7. Reporting the responseReporting the response8. Release of the data

34

7. Reporting the Response

The act of writing the data onto the form or entering the data into an electronic instrument.

35

The Response Process in Establishment Surveys

1. Encoding / record formation

2. Respondent selection / identification

3. Assessment of priorities (Motivation)

4. Comprehension of the data request

5. Retrieval of data

6. Judging the adequacy of the response

7. Reporting the response

8.8. Release of the dataRelease of the data

36

8. Releasing the Data

Business relationships with the outside world– Review and verification of data– Reconciliation

37

8. Releasing the Data, continued

Business relationships with the outside world– Review and verification of data– Reconciliation

Consistency and Sensitivity Consistency and Sensitivity of Released Data of Released Data

– Confidentiality– Security

38

The Response Process in Establishment Surveys

1.1. Encoding / record formationEncoding / record formation

2.2. Respondent selection / identificationRespondent selection / identification

3.3. Assessment of priorities (Motivation)Assessment of priorities (Motivation)

4.4. Comprehension of the data requestComprehension of the data request

5.5. Retrieval of dataRetrieval of data

6.6. Judging the adequacy of the responseJudging the adequacy of the response

7.7. Reporting the responseReporting the response

8.8. Release of the dataRelease of the data

39

The Response Process in Establishment Surveys: Personal Cognitive StepsPersonal Cognitive Steps ++

Steps at the Organizational LevelSteps at the Organizational Level

1.1. EncodingEncoding / / record formationrecord formation

2.2. Respondent selection / identificationRespondent selection / identification

3.3. Assessment of priorities (Motivation)Assessment of priorities (Motivation)

4.4. Comprehension of the data requestComprehension of the data request

5.5. Retrieval of data Retrieval of data from memory / recordsfrom memory / records

6.6. Judging the adequacy of the responseJudging the adequacy of the response

7.7. Reporting the responseReporting the response

8.8. Release of the dataRelease of the data

40

Some General Characteristics of Establishment Surveys

Complex response process Request technical, records-based data Skewed target populations Estimate change vs. level Emphasis on timeliness of reporting

41

Emerging Themes That Affect Establishment Survey Pretesting

Nature of requested

data Survey response is

labor-intensive

Respondent burden

Uses of economic

data

42

Questionnaire Development, Evaluation and Testing (QDET)

in Establishment Setting

How QDET for establishment surveys has addressed –1) Nature of the requested data

2) Survey response is labor-intensive

3) Respondent burden

4) Uses of economic data

43

1) Nature of the Requested Data

44

1) Nature of the Requested Data: Issues for QDET

Technical concepts with precise definitions

Data found in business records

45

Technical Concepts with Precise Definitions

Personnel involved in questionnaire development– Subject area experts– Stakeholders and data users – Questionnaire design experts

1) Nature of the Requested Data

46

Technical Concepts with Precise Definitions (cont.)

Consultation methods– Expert / User / Advisory Groups– Iterative rounds of prioritizing data needs– Large-scale content reviews– Exploratory focus groups– Observers during pretesting

1) Nature of the Requested Data

47

Technical Concepts with Precise Definitions (cont.)

Standardization and harmonization of questions and concepts

Instructions

1) Nature of the Requested Data

48

Data in Records

In order to design effective questionnaires, information is needed about –

– Data availability– Degree of comparability– Estimation strategies– Ease of retrieval– Respondent identity

1) Nature of the Requested Data

49

Data in Records(cont.)

Methods – Exploratory / feasibility studies and

site visits• Small, purposive samples • Meetings with company staff• Topic protocol

– Record-Keeping Studies

1) Nature of the Requested Data

50

2) Labor-IntensiveSurvey Response Process

51

2) Labor-Intensive Response Process:

Issues for QDET

Tangible cost to businesses

Difficult to observe the response process– Multiple data sources– Multiple respondents– Elapsed response time

52

Tangible Cost to Businesses

Methods to minimize additional workload– Respondent focus groups

• Limitation: Reluctance of business participants to reveal information to competitors

• Solution: Teleconference

– Informal interactions with respondents

2) Labor-Intensive Response Process

53

Difficult to Observe the Response Process

Adapt cognitive research methods– Traditional methods– Hypothetical probes– Vignettes / mock records– Interview protocol

• Flexibility

2) Labor-Intensive Response Process

54

Difficult to Observe the Response Process (cont.)

Logistics for cognitive research– On-site interviews– Time constraints– Personnel

• Questionnaire design experts

• Subject matter observers

2) Labor-Intensive Response Process

55

Difficult to Observe the Response Process (cont.)

Usability testing of electronic instruments– Methods

• Prototypes: paper, non-functioning screen shots, instruments with partial functionality

• Cognitive-like methods• User evaluations

– Logistics• On-site testing

2) Labor-Intensive Response Process

56

3) Respondent Burden

57

3) Respondent Burden:

Issues for QDET

Avoid adding burden due to questionnaire testing

Make effective use of respondent contacts

58

Avoid Adding Burden

Respondent-free methods

3) Respondent Burden

•Expert reviews and cognitive appraisals

•Electronic instruments— Heuristic reviews — Style guide

59

Effective Use of Respondent Contacts

Feedback from survey personnel– Interviewers – Survey staff interactions with respondents

• “Help Desk” calls• Follow-up calls regarding edit failing data

– Methods for collecting feedback• Debriefings / focus groups• Integrated database

3) Respondent Burden

60

Effective Use of Respondent Contacts (cont.)

Pilot tests and pretests– New surveys or major redesigns– “Dress rehearsal” vs. limited scope

3) Respondent Burden

61

Effective Use of Respondent Contacts (cont.)

Development strategies for electronic instruments– Small-scale pilots– Carefully screened survey participants– Respondent evaluation and feedback

3) Respondent Burden

62

Effective Use of Respondent Contacts (cont.)

Split sample experiments– Empirical evaluations of

• Alternative questions• Positioning of instructions• Mode selection

– Pilot studies vs. production data collection

3) Respondent Burden

63

4) Uses of Economic Data

64

4) Uses of Economic Data:

Issues for QDET Precise definitions Emphasis on timeliness Preservation of time series Recurring surveys that require

longitudinal designs and stability in the measurements (i.e., questions)

Post-Collection Quality EvaluationsPost-Collection Quality Evaluations

65

Post-Collection Quality Evaluations

Empirical analyses– Item nonresponse– Imputation rates– Outliers– Edit-failing data– External consistency checks

4) Uses of Economic Data

66

Post-Collection Quality Evaluations (cont.)

Respondent debriefings or Response Analysis Surveys– Re-contact respondents

• Actual response strategies• Reported data vs. definitions• Use of records, estimation strategies

4) Uses of Economic Data

67

Post-Collection Quality Evaluations (cont.)

Respondent debriefings

– Unstructured interview using a protocol

– Qualitative summary

Response Analysis Surveys

– Structured questionnaire

– Quantitative summary

4) Uses of Economic Data

68

Post-Collection Quality Evaluations (cont.)

Reinterview studies and content evaluations– Empirically assess response error and

bias in summary statistics

4) Uses of Economic Data

69

Post-Collection Quality Evaluations (cont.)

Primary goals – Assess data quality– Correct errors

Feedback into questionnaire design is rare

4) Uses of Economic Data

70

Conclusions and Conclusions and Future Research Challenges Future Research Challenges

for Establishment Survey for Establishment Survey QDETQDET

71

Conclusions

Establishment surveys are different. The establishment survey response

process model appears to be robust. An alternative

QDET cultureculture for Establishment Surveys seems justified.

72

An Alternative QDET Culture in Establishment Surveys

Complex response process

High “loss-function” for incorrect data

Role of stakeholders and data users

Advent of electronic methods

Methods for studying data availability and respondent selection

73

An Alternative QDET Culture in Establishment Surveys (cont.)

Estimation of change in key variables

Respondent burden with tangible reporting costs

Constraints on QDET activities

Adaptation of cognitive methods

Questionnaire stability

Post-collection quality evaluations

74

Future Research Challenges

The respondent– Identifying the “right” respondent– Influencing respondent selection

Facilitate the response process– Task analysis– Multiple respondents– Multiple data sources

Respondent role vs. data quality

75

Future Research Challenges (cont.)

Data needs and question development– Concepts Measurements Records– Records Measurements Concepts

Empirical evaluation of response errors– Edit failures– Item nonresponse– Other evidence?

76

Future QDET Challenges

Make better use of current activities Strengthen feedback loops between

production and development

Questionnaire

development

Post-collection processes and evaluation of data

quality

77

Future QDET Challenges (cont.)

Embed research into production data collection for recurring surveys

Evaluate “value-added” of QDET activities

78

Measuring Value-Added

Following production data collection – – Respondent debriefings– Response analysis surveys – Re-interviews– Validation studies

Intensive follow-up of nonrespondents– Evaluate bias– Nonresponse adjustments

79

Measuring Value-Addedcontinued

Examine post-collection downstream processes before and after redesign– “Plausibility indices”– Edit failure rates– Imputation rates– Item nonresponse– Costs

80

Reference:

Willimack, D.K., L. Lyberg, J. Martin, L. Japec, & P. Whitridge (2004) “Evolution and Adaptation of Questionnaire Development, Evaluation, and Testing Methods for Establishment Surveys,” Ch. 19 in Methods for Testing and Evaluating Survey Questionnaires, S. Presser, et al., ed., New York: Wiley.

Diane K. WillimackDiane K. WillimackU.S. Census BureauU.S. Census Bureau

diane.k.willimack@census.govdiane.k.willimack@census.gov