Greg Griffithsfor:Rob BurnsideEmma Farrell

ICES 3 Montreal June 2007

A Coherent Approach to Questionnaire Design Standards, Trends, and Implementation at

the ABS

How ABS enterprise questionnaire design standards and guidelines are:

implemented organizationallycreated and maintained;

scope and content of the standards

trends - changes over time:role of form design methodologistsnew modes and questionnaire structures; andthe Total Respondent experience

Overview

Mandatory use of agreed standards for in-scope instruments detailed in Corporate policy manual.

Common conventions, look and feel. Bringing about use of best practice.

Organization arrangements give responsibility to areas for different aspects of final forms

Form development process is supported by:

1.Standards manual containing guidelines and examples available on the ABS intranet;

2.Organizational structures incl survey-specific meetings, an annual review of core questions; and

3.A forms image database with workflow/ sign-off process; Holds images and linked discussion for interim, current and historical forms back to 1998

Big Picture

Methodology Division Forms Consultancy Group (FCG):

maintain and extend standards and guidelines; understand mode effects and try to minimize; R&D into form design (all modes)

Operational sections - manage survey process and produce output

Economic Statistics Data Centre - includes a specialised area constructing forms to standards and coordinating testing

Conceptual sections - establish user needs

Economic Standards Section - consistency of core economic questions and classifications

National Accounts Branch - to ensure inputs for Australian National Accounts

Responsibilities of areas involved

FCG

Despatch and Collection Unit

(DCU)

Economic Standards and Classifications

Section(ESCS)

Collection Management Unit (CMU)

Provider Contact Unit (PCU)

Business Statistics Centres(BSCs)

Administrative Data Acquisition Unit

(ADAU)

Household Statistics Centres (HSCs)

Population Survey Operations (PSO)

Form standards cover:•format, structure and question wording principles, common components;

•all modes - paper mail out, fax, CATI, Excel, TDE/IVR •detailed specifications and examples for different types of questions and forms;

•contact letters and emails

Development and evaluation standards cover:observational studies, usability testing, cognitive testing ...suitability of study design for answering development questions.

Scope and content

Relies on maintaining knowledge of ABS and international theoretical and applied research and experience;

Maintenance and planned updating of standards based on:

• Advice on development methods and approaches• Emerging issues• Involvement in field testing• Organizing experiments to examine aspects of design

form length effects, scanning design, effectiveness etc.• Urgency of development need

Creation and maintenance of standards

Trends: 1 - Evolving role of ABS form design methodologists (FCG) ctd

Pre 2000 Policing layout standards

2000 Methodology standards established. Need to examine broader measurement error recognized.

2001 More modes and more development methods

2002/03 Total Approach Management and a wider view - every interaction with the respondent is significant - appropriate, consistent and effective contact

2007 Multimodal Data Collection project funded

2007 Move of household based collections into same branch as FCG. New structure under consideration.

More modes: structure, presentation and wording changes to keep

things the same conceptually implies more, different testing and development

techniques; increase testing and creation cost form design increasingly intertwined with IT capability

and applications

More question structures: Complex Subjective (i.e. non-economic) More like household questionnaire questions

Trends - 2: more modes and question types

More development techniques right technique to explore the right thing more use of cognitive and exploratory methods Post Enumeration Studies, behavior coding, card

sorting, interviewer debriefing

Expert review Classic usability testing for eform and CATI Classic experimental design where appropriate

Trends: 3 - More development techniques

Every interaction with a data provider is important. The form is more than the form.

pre approach and covering letters, brochures and explanatory material, Telephone, fax and e-mail contacts

The response experience is more than filling in the form:

Response enhancing techniques - social influence theoriesmode of choice, incentives ….

Trends: 4 - Total Approach Management

Strong senior management recognition of the importance of form design which is reflected in organizational structure.

Formal form development process with defined roles and sign-off.

Corporate systems ensuring communication of standards.

Visibility of deliberations & decisions.

Summary

Summary

A specialized group of methodologists exists outside operational areas

– Responsible for standards and development methods– Involved in day-to-day form development as well as

R&D into respondent related measurement error– Centralization of methodology for all modes and

methods ensuring consistency – Maintains critical mass of expertise – Maintains a strong commitment to research and the

communication of results / new developments to operational areas