ifad vietnam rims training workshop 2011 (4 of 7)
DESCRIPTION
4. Questionnaire DesignIFAD Results and Impact Management System (RIMS) training materials for Monitoring and Evaluation (M&E), a culmination of the efforts of 2011 with the help of APMAS, and the direct support of Development and Policies Research Center (DEPOCEN) in Viet Nam. These materials were used in training courses in 2011 in Bac Kan and Tuyen Quang provinces. In total there are are 7 separate Powerpoint documents available in both Vietnamese and English language versions.TRANSCRIPT
Questionnaire Design Questionnaire Design MethodologyMethodology
1
ContentsContents
1. What are questionnaires? Why use them?
2. Properties of a questionnaire3. Procedures for questionnaire
design4. Principles of questionnaire design5. Questionnaire piloting6. Conclusions
2
Various definitions of Various definitions of questionnairequestionnaire• “ A set of questions arranged in a
certain structure to collect information from targeted group to answer one or several of research questions ”
• “A set of questions particularly used to address some statistically typical issues by gathering information in a survey.”
• And others…3
Why questionnaire?Why questionnaire?
Questionnaire is used to • Assess the need for investment and
development, and the differences among project targeted areas.
• To measure outcomes of project and intervention programmes
• Assess the beneficiaries’ satisfaction• Assess the project impacts
4
The importance of The importance of QuestionnaireQuestionnaireQuestionnaires are the most important
tool of a social investigationQuestionnaires reflect the research
issuesThe quality of a questionnaire
determines the quality of a surveyDesigning a high quality questionnaire
helps:◦Collect most accurate data◦ Increase response rate◦Reliable and valuable ◦Save time and money
Low quality Low quality questionnairesquestionnaires
Non response may affect the questionnaire validity just as same as missing data (or obtaining inaccurate data).
6
Low response ratesReduce the
effectiveness of sample
size Reduce the research power
Reduce the accuracy of final results
Procedures for questionnaire Procedures for questionnaire designdesign
3 major steps:1.Deciding on what to ask2.Defining types of question,
wording for each type of question3.Structuring and formatting the
questions
Deciding on what to askDeciding on what to ask
Define mid-level steps between the research question(s) and the final question(s)
Research question(s)Relevant mattersStay focusedQuestions
Defining types of questionDefining types of question
Yes/No questions >< Open questions
Single choice >< Multiple choiceCategorical questionsOrdinal questionsLinkert-scale questionsSingle-pole >< Double-pole
Wording for questionsWording for questions
Clear meaning, not vagueAvoid multi-meaning questionsAvoid negative questionsAvoided guiding questionsStay focusedQuestions
Structuring the questionsStructuring the questions
TitleIntroduction (enclosed with intro
letter)Contact Minimal types of questionAvoid horizontal & vertical
structureAvoid overuse of boxes, tables,
and lines
Structuring the questions Structuring the questions (cont.)(cont.)
Clear, complete and attractive structure
Two styles of question order:◦From easy to difficult, from general to
specific◦From most interesting to less interesting
No matter what style to be chosen, follow the logic, i.e. questions in each part are relevant
Keep the rhythm of the questionnaireAvoid overusing of skips and turns
The Art of asking The Art of asking questionsquestions• Ask the right question• Respondents must understand
the question• Respondents must know the
answer• Respondents must be willing and
able to give the answer
13
Types of Survey QuestionsTypes of Survey Questions
1. Questions about activities, facts/events
2. Questions about psychological status or attitudes
3. Questions about knowledge/perception
14
Questions with a specific Questions with a specific timetimeBad question:Over the past
6 months, what new household appliances have you purchased?
Better question:Now I’m going to read a
list of household devices. As I read each, please confirm whether you bought that appliance during the last 6 months.
How about…a refrigerator?
a kitchen device?a microwave?
15
Specifying…Specifying…Who, what, when, where and how?Ex. : Whose income? What are
included?Over what period of time? Ex.:
“In 2002, what was your total household income? Before tax? Please provide the incomes of each member of your family, including wages, social welfare, and public aids”
16
Questions with single-Questions with single-meaning wordsmeaning wordsNot clear:How do you
compare the distance between you and your family at the moment and in the past? Do you think you live closer to your family? Or still the same?
Clearer:Compared with
your previous house,
do you live closer now?
Or is it further? Or the same?
17
Open vs. Closed QuestionsOpen vs. Closed Questions
General rules: for quantitative research, closed questions are often better than open questions, because:◦Easier for respondents◦Easier for data entry, ◦Not necessary to recode data◦Easier for analysts to categorize
respondents
18
Responses to open & closed Responses to open & closed questionsquestions Open questions are those left open
for respondents to freely answer. “How do you feel about your health status today?”
Closed questions provide pre-determined answers“Do you feel well today?” Yes No
How much do you agree with the following statement: “Today my health is great”:
Agree Neither agree nor disagree Disagree
19
Closed formatClosed format
• Closed questions are often useful as it is…• Easy and quick to fill in• Easy to code, record and analyse
• Closed questions are often not useful as... • It does not cover all possible
responses (comprehensiveness) • It is not able to obtain deep
information
20
Disadvantages of Closed Disadvantages of Closed QuestionsQuestionsAnswers may lead the
respondentsIt may be too easy to answer
without much thinkingShould not be used when:
◦Asking for the frequency of sensitive behaviors
◦There are many different possible answers
21
Open formatOpen format
Open questions useful when…◦Answers are unknown◦Answers are too many/complicated to
be coded◦searching for further explanations of the
answers Open questions are not useful
when…◦There’s a need to code possible errors◦It is difficult to analyse and synthesize
22
Type of measuring scalesType of measuring scales Nominal scale
Ex. Male/female; Provinces: Bac Kan/Ha Tinh/Tra Vinh/Tuyen Quang…
Ordinal scale
Ex. Very poor – poor – average – rich – very rich; like – normal – dislike
Interval scale
Ex. range of income, range of age. Ratio scale
Ex. Percentage of male/female, ethnic groups
23
Format of response Format of response categories categories
Circling or tick off your answers
Skip & leave blank
24
Survey Intro/Cover LetterSurvey Intro/Cover Letter
Introduction should include:◦who is conducting the survey◦the topics covered in the survey◦Statement of confidentiality◦Statement that the time length of
the survey depends on the research mode, topics, population
25
Check the format of Check the format of questionnairequestionnaireNumber all the questions
sequentially Use large, clear font type;List groups of answers in vertical
sequenceAvoid double/triple-choice
answersBe consistent with guidelines and
orders of answer groups.
26
Check the format of Check the format of questionnairequestionnaireDon’t split question across pages. If necessary, move the question and
answers to next page.Put specific instructions of
questionnaire as needed, then the questions
Distinguish instructions from questions
Pre-code the questionnaire using check boxes
27
Principles for questionnaire Principles for questionnaire designdesignAsk what you want to attainAsk the way you want to gain the
answer: use of wordsUnderstand the mind of
respondents and the context of the interview
Other rules…
28
Use of wordsUse of wordsWording should be accurate: a
small change could make great differences
Words in a question affect the answer.
The more specific the question, the more it influences the answer.
29
Context of the interviewContext of the interviewMany cases are encountered:
◦The interviewer may even chitchat various topics with an enthusiastic respondent
◦In some cases, the interviewee may refuse to answer.
Notes for a new interviewer (see next page)
30
Rules for new interviewersRules for new interviewersAvoid jumping to the questions
until you really understand the research question
Once raising a question, ask yourself “why I need to know this matter?”
Use available questions from different sources.
31
Use available questionsUse available questionsReady-made questions from
previous successful investigations are encouraged to use, so that:◦To save time and money (short-cut)◦To compare with the previous results◦To improve the validity of the answers
of research with similar sample size, context and without changes
◦For investigations with different time and with changes, use the same questions to measure trend.
32
Use available questions Use available questions (cont.)(cont.)When using questions from other
surveys, be concerned with:◦Copyright◦Setting of question (as the answers
for some questions are particularly used for a certain situation)
33
Other principlesOther principlesGuarantee clear codes for main and
sub- questions (ex. 1, 2a, 3, 4b)Start with simpler questions (not
sensitive)Use proper logic. If the questionnaire is
long, start with the keysPut questions on anthropology at the
endPut questions for past experiences prior
to those for current status.Put general questions first, then specific
34
Other principles (cont.)Other principles (cont.)Apply the funnel-shaped rule
(from wide to narrow questions)Use the rule of filteringUse equivalent terms (avoid
technical ones)Avoid unclear terms/short termsAvoid multi-meaning words/termsKeep sentences short (if possible)
35
Other principles (cont.)Other principles (cont.)Avoid suggesting the answers (by
proposing a hypothesis)Avoid leading questionsBe cautious using hypothetical
questionsAvoid bias (see next)
36
Piloting: Testing the Piloting: Testing the QuestionnaireQuestionnairePreferable to test the
questionnaire with those similar to targeted subjects of the research population
Conduct pilot using the same mode for the major study
Consider cognitive pretesting
37
Piloting: Testing & Piloting: Testing & EvaluationEvaluation
38
Piloting 1. Develop &
test the research
tools
2. Assessing the feasibility of a survey
against sample size
3. Identify logistical problems of the questionnaire
Piloting: Testing & Piloting: Testing & EvaluationEvaluation
Prepare first draft and pre-testDistribute questionnaire to small sampleEvaluate responses & re-draft
Checklist items: (adapted by Bowling 2002) The questions measure what is intended wordings are understood by and means a
similar thing to all Instructions are easily understood/followed Do the responses cover all relevant issues? Any questions systematically/frequently
missed? Basically test for ‘bugs’
39
Complete the Questionnaire: Complete the Questionnaire: Questionnaire Design - Practical Questionnaire Design - Practical approachapproach
1. Decide what information is needed
2. Search for existing questions3. Focus groups4. Draft new questions/revise
existing ones5. Sequence the questions6. Get peer evaluation7. Revise and self-test or peer test
Complete the Questionnaire: Complete the Questionnaire: Questionnaire Design - Practical Questionnaire Design - Practical approachapproach
8. Think-aloud interviews9. Revise/eliminate questions10.Prepare interviewer instructions
for pilot test11.Pilot test (10-20 cases)12.Revise eliminate questions
based on respondent & interviewer comments
Complete the Questionnaire: Complete the Questionnaire: Questionnaire Design - Practical Questionnaire Design - Practical approachapproach
13.Pilot again, if necessary14.Prepare final interviewer instructions15.Modify questionnaires if interviewers
raise questions in training16.After the interview completes, debrief
interviewers for potential problems17.Use experience from a study for
planning a study in the future
ConclusionsConclusions
Not general recommendations, just issues to be aware of, consider and balance out
Consider the aims of study, population and resources available
Principal objective is to collect data that is RELIABLE, VALID & UNBIASED
43
Key References To Get Key References To Get StartedStartedDillman, Don. Mail and Internet Surveys:
The Tailored Design Method. New York: John, Wiley & Sons, Inc, 2000.
Bradburn, N, Sudman, S. and Wansink, B.. Asking Questions: The Definitive Guide to Questionnaire design – for Market Research, Political Polls, and Social and Health Questionnaires. San Francisco: Jossey Bass, 2004.
Oppenheim AN. Questionnaire design, interviewing and attitude measurement. 2nd ed. London: Pinter; 1992.See also questionnaires chapter in the handbook of survey
44