MEADOW Questionnaire Validation

G. Sirilli & A. Arundel

MEADOW questionnaires

• Designed for computer assisted telephone interviews (CATI)

• No more than 20 to 30 minutes per interview

• Two separate questionnaires– Employers– Employees

Goals of question validation

1. Almost all respondents understand each question in the way that it is intended to be understood

• Includes consistency of understanding across countries

2. A large majority of respondents can answer each question

• Varies by the questionnaire method – mailed, by telephone, etc.

3. Minimize response error1. Identify appropriate response scales, remove

redundant questions, identify questions susceptible to ‘satisficing’ responses, etc.

Validation methods• Testing among MEADOW participants:

– Can identify many problems, but ‘expert’ familiarity often results in a failure to identify problems

• Cognitive testing: face-to-face interviews of a variety of types of respondents.– Random sampling of respondents is not needed– Ask respondents how they understand each question,

possible problems with answering, and probe for understanding of terms.

• Pilot survey: Random survey of target population under normal conditions.– Provides better data on ability of respondents to reply to each

question, checks for logical inconsistencies, etc.– Provides better data on response option categories

MEADOW testing

• Cognitive testing through 247 interviews (approximately 1 hr each)

• Employer questionnaire:– 118 interviews in 11 countries

• Employee questionnaire:– 129 interviews in 11 countries

• Questionnaires tested in Denmark, Finland, France, Hungary, Germany, Italy, Netherlands, Sweden, Ukraine, UK, and US.

Two rounds of interviews

• Phase 1: Approximately 2/3rds of interviews– Due to questionnaire length, each questionnaire

divided into two logically consistent versions (A and B) and tested separately

– Identified problem questions, questions revised

• Phase 2: 1/3 of interviews– Only retested one version consisting of revised

questions– Questions revised (slightly) again if needed or deleted

Examples of deletions

• Questions on the use of specific software systems (performance tracking software, etc).– most respondents were unfamiliar with the

software; can only be asked of IT specialists

• Question on policy of guaranteed job security for groups of employees– Respondents strongly believed that no jobs today

are guaranteed –question no longer relevant.

Question revisions

• Revisions extensive:– Employer questionnaire: approximately 25

questions revised (out of 90)– Employee questionnaire: approximately 20

questions revised (out of 115)

• In addition, many minor changes to response categories and wording.

Conclusions

• Extensive cognitive testing: 247 interviews instead of the norm of 20 – 30– partly required to ensure consistency in

responses across countries and to test two questionnaires

– Several questions deleted plus many revisions

• Result: much improved question reliability• Next step: Pilot survey