Questionnaire DesignQuestionnaire DesignA survey is only as good as A survey is only as good as

the questions it asksthe questions it asks

Questionnaire ModesQuestionnaire Modes► Face-to-face interviewFace-to-face interview

► Telephone surveyTelephone survey

► Self-administered survey (i.e. mail Self-administered survey (i.e. mail survey)survey)

Types of QuestionsTypes of Questions► The background question The background question

used to obtain demographic characteristics of the group being used to obtain demographic characteristics of the group being studied, such as age, sex, grade, level of assignment, and so forth. studied, such as age, sex, grade, level of assignment, and so forth. This information is used when you are categorizing your results by This information is used when you are categorizing your results by various subdivisions such as age or grade. Therefore, these various subdivisions such as age or grade. Therefore, these questions should be consistent with your data analysis plan.questions should be consistent with your data analysis plan.

► Multiple choice or closed-end question. Multiple choice or closed-end question. It is used to determine feelings or opinions on certain issues by It is used to determine feelings or opinions on certain issues by allowing the respondent to choose an answer from a list you have allowing the respondent to choose an answer from a list you have provided .provided .

► The intensity question (Attitude Measurement) The intensity question (Attitude Measurement) a special form of the multiple-choice question, is used to measure a special form of the multiple-choice question, is used to measure the intensity of the respondentthe intensity of the respondent’’s feelings on a subject. These s feelings on a subject. These questions provide answers that cover a range of feelings.questions provide answers that cover a range of feelings.

► The free response or open-end question.The free response or open-end question.This type requires respondents to answer the question in their own This type requires respondents to answer the question in their own words .It can be used to gather opinions or to measure the intensity words .It can be used to gather opinions or to measure the intensity of feelings.of feelings.

Questionnaire ConstructionQuestionnaire Construction► Keep the language simple.Keep the language simple.► Keep the number of questions to a minimum.Keep the number of questions to a minimum.► Limit each question to one idea or concept.Limit each question to one idea or concept.► Do not ask leading questions.Do not ask leading questions.► Allow for all possible answers.Allow for all possible answers.► Avoid emotional or morally charged questions.Avoid emotional or morally charged questions.► Include a few questions that can serve as checks on the accuracy Include a few questions that can serve as checks on the accuracy

and consistency of the answers as a whole.and consistency of the answers as a whole.► Organize the pattern of the questionsOrganize the pattern of the questions

Place demographic questions at the end of the questionnaire.Place demographic questions at the end of the questionnaire.· Have your opening questions arouse interest.· Have your opening questions arouse interest.· Ask easier questions first.· Ask easier questions first.· To minimize conditioning, have general questions precede specific · To minimize conditioning, have general questions precede specific

ones.ones.Group similar questions together.Group similar questions together.· If you must use personal or emotional questions, place them at the · If you must use personal or emotional questions, place them at the

end of the questionnaire.end of the questionnaire.

11%13%

33%

70%

21%

5% 3%

43%

To have a hotline For government to set up a hotline

Questionnaire designThe Effect of One Word

Kaiser Family Foundation/Harvard University National Survey of Americans’ Views on Consumer Protections in managed Care, January 1998

How important do you think it isto have a hotline telephone number people can call to get help when they have problems with their health plan?

How important do you think it is for the GOVERNMENT to set up a hotline telephone number people can call to get help when they have problems with their health plan?

Very Important Somewhat Important Not too important Not at all important

Leading QuestionsLeading Questions►Do you think the food in the hotel Do you think the food in the hotel

made your sickmade your sick►Did the hotel staff seem unhiegenic to Did the hotel staff seem unhiegenic to

you?you?►Do you agree that hospital staff were Do you agree that hospital staff were

close to exhaustionclose to exhaustion

Questionnaire designBalance in question wording

What’s wrong with these questions?

• Do you support the United States taking military action against Iraq?

• If you heard that Candidate X supported stricter gun control laws, would that make you more likely to vote for her?

• Did you happen to vote in the presidential election in 2000?

Questionnaire designBalance in question wording

Bad questions, improved:

• Do you support or oppose the United States taking military action against Iraq?

• If you heard that Candidate X supported stricter gun control laws, would that make you more likely to vote for her, less likely to vote for her, or wouldn’t it make much difference?

• Did you happen to vote in the presidential election in 2000, or not?

Questionnaire designOpen-ended or Closed-ended?

7%

8%

10%

12%

15%

15%

15%

Kaiser Family Foundation/Harvard School of Public Health Health News Index, April 2003 (conducted April 3-6, 2003)

What do you think is the most important problem in health or health care for the government to address? (top 7 answers)

Which of the following do you think is the most important health care issue for the President and Congress to deal with?

Cost of health care

Cost of Rx drugs

Coverage for uninsured

Senior citizens’ care

Other medical conditions

Rx coverage for seniors

Cost of insurance8%

11%

18%

27%

27%Helping seniors pay for prescription medicines

Increasing the number of Americans covered

by health insurance

Making Medicare more financially sound for

future generations

Lowering the cost of medical malpractice

insurance for physicians

Helping states with the cost of Medicaid

Questionnaire designDon’t Know Answers – offered or volunteered?

42%

54%

4%

NPR/Kaiser Family Foundation/Kennedy School of Government Education Survey, June 1999 (conducted June 25 – July 19, 1999)

Do you favor or oppose the government offering parents money or "vouchers" to send their children to private or religious schools, or public schools outside their district?

Do you favor or oppose the government offering parents money or "vouchers" to send their children to private or religious schools, or public schools outside their district, or haven’t you heard enough about that to have an opinion?

31%

36%

33%

Favor

Oppose

No opinion

Favor

Oppose

No opinion

Types of questionsTypes of questions

• Single dichotomy or dichotomous-alternative questions“Are you currently registered in a course at the

University of Lethbridge?Yes____ No____”

• Respondent chooses one of two alternatives (yes/no; male/female)

Types of fixed alternative questions…

►Multi-choice alternativeMulti-choice alternative Respondent chooses from several Respondent chooses from several

alternativesalternatives Many types…Many types…

Types of fixed alternative Types of fixed alternative questions…questions…

► Determinant choiceDeterminant choice Choose only one from several possible Choose only one from several possible

responsesresponses““Which faculty are you currently registered Which faculty are you currently registered

in at the University of Lethbridge?in at the University of Lethbridge?Management ___Management ___Education ____Education ____Arts/Science____Arts/Science____Health sciences____Health sciences____Combined degree____Combined degree____

Multi-choice alternative Multi-choice alternative questions…questions…

►Frequency determinationFrequency determination Asks for an answer about Asks for an answer about

frequency of occurrencefrequency of occurrenceIn a typical week, how often do In a typical week, how often do

you purchase chocolate chip you purchase chocolate chip cookies?cookies?

__never__never__ once__ once__ 2 or more times__ 2 or more times

►Check listCheck list Provide multiple answers to a single Provide multiple answers to a single

questionquestion Should be mutually exclusive and Should be mutually exclusive and

exhaustiveexhaustive““What brands of chocolate chip cookies What brands of chocolate chip cookies

have you, to the best of your have you, to the best of your memory, purchased in the past memory, purchased in the past month (check all that apply?)”month (check all that apply?)”

__ Dare__ Dare__ Chips A’hoy__ Chips A’hoy__ Presidents Choice Decadent etc. etc.__ Presidents Choice Decadent etc. etc.

►Attitude rating scalesAttitude rating scalesAttitude:Attitude:An enduring disposition to An enduring disposition to

consistently respond to various consistently respond to various aspect of the world, including aspect of the world, including persons, events and objectspersons, events and objects

Attitude Scales: Scaling Defined

The term scaling refers to procedures for attempting to determine quantitative measures of

subjective and sometimes abstract concepts.

Types of attitude scalesTypes of attitude scales► Simple attitude scalesSimple attitude scales► Most basic form – respondent responds to a Most basic form – respondent responds to a

single questionsingle question► Do not allow for fine distinctions or Do not allow for fine distinctions or

placement on continuaplacement on continua You are at a company party and are feeling You are at a company party and are feeling

nervous, but you are obligated to be there. Do nervous, but you are obligated to be there. Do you:you:

__ find someone you know to buddy up with__ find someone you know to buddy up with__ take it as an opportunity to meet new people__ take it as an opportunity to meet new people

► Category scalesCategory scales More sensitive; provides more informationMore sensitive; provides more information Overall, how satisfied are you with the high Overall, how satisfied are you with the high

speed performance of your Mercedes:speed performance of your Mercedes:__ very satisfied__ very satisfied__ somewhat satisfied__ somewhat satisfied__ neither satisfied nor dissatisfied__ neither satisfied nor dissatisfied__ somewhat dissatisfied__ somewhat dissatisfied__ very dissatisfied__ very dissatisfiedIf you could choose, how long would each term If you could choose, how long would each term

be?be?___26 weeks __ 13 weeks __ 6 weeks ___4 ___26 weeks __ 13 weeks __ 6 weeks ___4

weeksweeks

►Summated rating scales – the Likert Summated rating scales – the Likert scalescale Respondents indicate their attitudes by Respondents indicate their attitudes by

checking how strongly they agree or checking how strongly they agree or disagree with statementsdisagree with statements

Chocolate chip cookies are my Chocolate chip cookies are my preferred variety of cookiepreferred variety of cookie

Strongly disagree Disagree Uncertain AgreeStrongly disagree Disagree Uncertain AgreeStrongly AgreeStrongly Agree

(1) (2) (3) (4) (5)(1) (2) (3) (4) (5)

Likert ScaleLikert ScaleKids apparels provided by the retailers do Kids apparels provided by the retailers do not address the safety issues of babiesnot address the safety issues of babiesNo, I strongly disagreeNo, I strongly disagreeNo, I disagree quite a lotNo, I disagree quite a lotNo, I disagree just a littleNo, I disagree just a littleI am not sure about thisI am not sure about thisYes, I agree just a littleYes, I agree just a littleYes, I agree quite a lotYes, I agree quite a lotYes, I strongly agreeYes, I strongly agree

►Semantic Differential Rating scaleSemantic Differential Rating scale An attitude measure consisting of a An attitude measure consisting of a

series of seven-point bipolar rating series of seven-point bipolar rating scales allowing response to a scales allowing response to a “concept”“concept”

Think of your favorite type of cookie. Rate it Think of your favorite type of cookie. Rate it on each of the following continua:on each of the following continua:

Hard----------------------------------------------------SoftHard----------------------------------------------------SoftLots of chips----------------------------------Fewer Lots of chips----------------------------------Fewer

chipschipsCrispy------------------------------------------------Crispy------------------------------------------------

chewychewy

Example of Semantic Differential Scaling Form

Service is discourteous ___:___:___:___:___:___:___ Service is courteousLocation is convenient ___:___:___:___:___:___:___ Location is inconvenientHours are convenient ___:___:___:___:___:___:___ Hours are inconvenientLoan interest rates are high ___:___:___:___:___:___:___ interest rates are

low

Figure 5: Contrasting Profiles of Banks A and B

Service is discourteous ___:___:___:___:___:___:___ Service is courteous

Location is convenient ___:___:___:___:___:___:___ Location is inconvenient

Hours are convenient ___:___:___:___:___:___:___ Hours are inconvenient

Loan interest rates are high ___:___:___:___:___:___:___interest rates are low

Bank ABank B

►Numerical Rating scaleNumerical Rating scale Similar to a semantic differential Similar to a semantic differential

except that it uses numbers as except that it uses numbers as response options to identify response response options to identify response positions instead of verbal descriptionspositions instead of verbal descriptions

Think of your favorite type of cookie. Rate it on Think of your favorite type of cookie. Rate it on each of the following continua:each of the following continua:

Hard----------------------------------------------------SoftHard----------------------------------------------------Soft88 77 66 55 44 33 22 11

► Constant Sum ScalesConstant Sum Scales Attributes based on their importance to the Attributes based on their importance to the

person. Respondents are asked to divide a person. Respondents are asked to divide a constant sum to indicate the relative constant sum to indicate the relative importance of attributesimportance of attributes

Example: Suppose the photocopy budget per Example: Suppose the photocopy budget per professor was $100 per month. How much professor was $100 per month. How much should be allocated to the following. Divide should be allocated to the following. Divide the $100 according to your preference:the $100 according to your preference:

____ photocopying for student needs; ____ photocopying for student needs; ____ photocopying for research needs; ____ photocopying for research needs; ____ photocopying for committee needs.____ photocopying for committee needs.========$100 TOTAL$100 TOTAL

► Stapel ScalesStapel Scales An attitude measure that places a single An attitude measure that places a single

adjective in the center of an even-number adjective in the center of an even-number range of numerical valuesrange of numerical values

Example:Example:Research MethodologyResearch Methodology+3+3+2+2+1+1ExcitingExciting-1-1-2-2-3-3

Example of a Stapel Scale

-5 -4 -3 -2 -1 1 2 3 4 5

Service is courteous

Location is convenient

Hours are convenient

Loan interest rates are high

►Graphic Rating ScalesGraphic Rating Scales An attitude measure consisting of a An attitude measure consisting of a

graphic continuum that allows graphic continuum that allows respondents to rate an object by respondents to rate an object by choosing any point on the continuumchoosing any point on the continuum

►Rank-Order ScalesRank-Order Scales Scales in which the respondent Scales in which the respondent

compares one item with another or a compares one item with another or a group of items against each other and group of items against each other and ranks them.ranks them.

Example: next slideExample: next slide

Most important skillsMost important skills► Adaptability to changeAdaptability to change► Problem identificationProblem identification► Listening skillsListening skills► Written communicationWritten communication► LeadershipLeadership► Informal Oral communicationInformal Oral communication► Analytical thinking/problem solvingAnalytical thinking/problem solving► Time managementTime management► Coping with stress/job pressuresCoping with stress/job pressures► Interpersonal relationsInterpersonal relations► Formal oral presentationsFormal oral presentations

Most important skillsMost important skills

Adaptability to changeAdaptability to changeProblem identificationProblem identificationListening skillsListening skillsWritten communicationWritten communicationLeadershipLeadershipInformal Oral communicationInformal Oral communicationAnalytical thinking/problem Analytical thinking/problem

solvingsolvingTime managementTime managementCoping with stress/job pressuresCoping with stress/job pressuresInterpersonal relationsInterpersonal relationsFormal oral presentationsFormal oral presentations

8 96 61 12 44 23 35 57 1011 79 810 11

ManagersBusin

ess

grads

►Paired Comparison ScalesPaired Comparison Scales For each pair of sports car, place a For each pair of sports car, place a

check beside the one you most prefer if check beside the one you most prefer if you had to choose between the two.you had to choose between the two.

- BMW Z3- BMW Z3 - Chevrolet Corvette- Chevrolet Corvette- Porsche Boxter- Porsche Boxter - Porsche Boxter- Porsche Boxter

- Chevrolet CorvetteChevrolet Corvette- BMW Z3BMW Z3

►SortingSorting Respondent indicates their attitudes Respondent indicates their attitudes

or beliefs by arranging items. or beliefs by arranging items.

Example: Please sort the following cards with Example: Please sort the following cards with pictures of cookies into the following pictures of cookies into the following categoriescategories

LikeLikeDislikeDislikeNeither like nor dislikeNeither like nor dislike

Examples of Poor Survey Examples of Poor Survey QuestionsQuestions

Which of these services would you be interested in?Which of these services would you be interested in?____ loans ____ mentoring ____ business counselling ________ loans ____ mentoring ____ business counselling ____information referralinformation referral(This question should have an “other” category)(This question should have an “other” category)

What beverages do you drink?What beverages do you drink?____ Milk ____ coke ____ non-cola drink ____ coffee ____ ____ Milk ____ coke ____ non-cola drink ____ coffee ____ teatea____ juice____ juice(This question is too broad. Most of us will have drunk (This question is too broad. Most of us will have drunk some ofsome ofthese at some time. Is the respondent to check a these at some time. Is the respondent to check a number of boxes or only one?number of boxes or only one?

Sample QuestionnaireSample Questionnaire(developed by a woman who was interested in selling (developed by a woman who was interested in selling southwestern jewelry made by Native Indians)southwestern jewelry made by Native Indians)Q1. Have you ever purchased or received southwestern jewelry?Q1. Have you ever purchased or received southwestern jewelry?Yes ______ No ______Yes ______ No ______Q2. Have you ever purchased or received southwestern jewelry Q2. Have you ever purchased or received southwestern jewelry made by native Indians?made by native Indians?Yes ______ No ______Yes ______ No ______If Yes, what type of jewelry?If Yes, what type of jewelry?Necklace____ Ring _____ Bracelet _____ Earnings _____Other _____Necklace____ Ring _____ Bracelet _____ Earnings _____Other _____Q3. Would you be interested in purchasing the above mentioned Q3. Would you be interested in purchasing the above mentioned jewelry made by native Indians?jewelry made by native Indians?Yes ______ No ______Yes ______ No ______Q4. Do you know where to shop for such jewelry? Q4. Do you know where to shop for such jewelry? Yes ______Yes ______ No ______No ______Q5. When buying jewelry, what do you value the most? On a Q5. When buying jewelry, what do you value the most? On a scale of 1 through 5, list in order according to yourscale of 1 through 5, list in order according to yourpreference. One represents your most valued choice.preference. One represents your most valued choice.Craftsmanship_____ Cost _____ Uniqueness _____ Other_____Craftsmanship_____ Cost _____ Uniqueness _____ Other_____