effective questionnaire design

Effective Questionnaire

Design

Craig Parylo & Annette Lee

What is a questionnaire?

A series of questions.

Gathers information

from lots of people.

Focussed around a

single topic or area of

interest.

An effective questionnaire…

Asks relevant questions• Questions that will provide the answers to the

research topic in question

Gets valid responses• Responses given reflect the respondent’s actual

opinions

Gets a representative response• Makes it more reliable to infer the findings to

wider population

Survey

report

Food

Question 1

Question 2

Question 3

Introduction

Anyone can design a questionnaire.An effective questionnaire requires planning and an appreciation of the psychology of respondents:Filling in a questionnaire is a complex process.• The job of a questionnaire designer is to

make it easy to respond.

Complex cognitive process

Do you drink coffee? Yes No

Food Question 1

1. Identify subject

2. Phrase as question

3. Reads

4. Decodes 5. Collects info

6. Share?

7. Encode

I prefer tea

8. Option No9. Respond

Exercise: responding to questions

1. What is the colour of your car?2. What do you usually have for breakfast?3. What did you have for breakfast last Friday?4. Have you ever not left a tip at a restaurant?5. What is your sexuality?6. What is the biggest regret in your life?

Choosing the right approach

Phot

o by

Ron

it G

elle

r (htt

p://

ww

w.s

xc.h

u/pr

ofile

/Rog

el)

Types of questionnaireSelf-

completed

Postal

Electronic

Interviewer-led

Face to face

Telephone

No method consistently outperforms others

Considerations

Your approach should take consideration of:• Your topic• Who you want survey• The type of information

you want• Your budget• How quickly you need

the information

Sample size vs Depth of info

• Better response rates from interviewer-led surveys than from self-response surveys.

• Self-response surveys are usually cheaper.

• Interviewer-led surveys can produce richer information.

• Self-response surveys can reach more people.

Don’t confuse quantity for quality.

Children

Is it pitched at the right age group?

Timing and convenience

• Will it be completed right after the event, or sometime after?

• Will it be completed in front of other people or in private?

• Many people are poor at estimating numerical info about their lives.

• The more common the event the shorter a person’s window of recall is.

Question types

Question types

• Open / closed• Single / multiple

response• Attitudinal / opinion

questions• Filtering / routing

Attitude/opinion

• Can be difficult to get the phrasing of the question right so it means the same thing to all people.

• Can be tempting to put lots of explanation into the question – but sometimes this just confuses things further.

• Use clear, simple English/ language and avoid being too general.

• Try not to ask ‘Iceberg’ questions.

Mid-point / neutral opinions

Use mid-points in questions on attitude.

• Historically controversial as it was thought that respondents chose a neutral option to avoid thinking about a question.

• Now shown that including a mid-point increases reliability and validity

Strongly disagree

Strongly agree

‘How important?’

Following your visit to the clinic today, please rate the following – first based on your experience and second how important this is to you?

1 2 3 4 5 6 7Waiting time Very poor Excellent

How important is the length of waiting time to you?

Not important Very

important

Quality of magazines in waiting room

Very poor Excellent

How important is the quality of magazines in waiting room to you?

Not important Very

important

17

Common mistakes

What might be wrong with this question?

What was the first drink you had

today?

Tea

Coffee

Water

Juice

Haven’t had a drink today

Other (please state)

Need to cover all options

Common mistakes

What might be wrong with this question?

What is your age group?0-18 18-30 30-45 45+

0-18 19-30 31-45 46+

Options should be mutually exclusive.

Common mistakes

What might be wrong with this question?

What did you think about the waiting time?Excellent Very

goodGood Average Poor

‘Good’ responses ‘Bad’ response

Options should be balanced

Common mistakes

What might be wrong with this question?

How would you rate the appointment booking service using the:

Good Average PoorWebsitePhoneReception

Use opt-out responses appropriately

N/A

Increasing clarity

22

Question wording

• Keep questions as short as possible– Max 16 words.

• Consider providing medium length (30 word) introductions to groups of related questions (context is vital!)

• Use open-ended questions sparingly

23

Double questions

If a question can be broken down into parts then it should be?

Is space and storage adequate?

☐ Yes☐ No

Was the consent of the patient

obtained and recorded?

☐ Yes☐ No

Was the consent of the patient:

☐ Obtained?

☐ Recorded?

Double negatives

• If the respondent has to tick ‘No’ to give a positive response then this can be considered a double negative.

• Try and phrase questions with a positive slant so that the positive response is ‘Yes’ rather than ‘No’.

25

Do you think that it is an unwise policy to continue sending food aid to Africa. ☐ Yes ☐ No

Style, appearance and layout

Design

• Use booklet format with double-sided printing– A4 folded to A5, or, A3 folded to A4

• Front cover:– The title of the questionnaire– Identity of the organisation carrying it out– Clearly explain the purpose of the questionnaire– Keep graphics neutral

Aesthetics

• Arial font, size 12 point or larger• Questionnaire length– Shorter is generally better– Also keep plenty of ‘whitespace’

• Be consistent in the use of colour, shapes and location in order to guide the respondent through the questionnaire

Other elements to remember

• Make it look professional.• Give an estimated completion time.• Put the deadline for return at the beginning

and end of the questionnaire.• Provide a return address – on the

questionnaire. • Say ‘Thank you’.

Ordering your questionnaire

• Place easy, non-challenging questions first• Key filter questions should go in early.• Use a funnel structure – Go from basic to more in depth.– General questions before specific questions.– Go from the least sensitive to the most sensitive

questions.– But – try and put important questions in before the

end.

Ordering your questions

• Ask for suggestions for improvement at the end.

• Check for leading questions.• Add prompts where required.• Order of questions can impact heavily on the

amount of time it takes for the respondent to complete the questionnaire.

Filtering

• Filtering (or routing) helps identify which respondents should answer a detailed set of questions about a certain aspect of the topic.

• Can be difficult to achieve – should be VERY CLEARLY signposted

Filtering ExampleAbout our new website Yes No1. Did you visit our website today?(If yes please answer questions 2-4, if no go to question 5)

2. Did you like the new look?3. Did you find it easy to navigate?4. Would you recommend the site to others?5. Did you pick up a copy of our latest leaflet?

✓

✓✓

✓✓

Filtering ExampleAbout our new website Yes No1. Did you visit our website today?

Please continue with Q1a

Please go to Q2

1a. Did you like the new look?1b. Did you find it easy to navigate?1c. Would you recommend the site to others?

2. Did you pick up a copy of our latest leaflet?

Tips for increasing response rate

Phot

os b

y Sa

rah

Will

iam

s & D

avid

e G

uglie

lmo

(http

://w

ww

.sxc

.hu/

profi

le/M

eHer

e) (h

ttp:/

/ww

w.s

xc.h

u/pr

ofile

/bro

kena

rts)

Improving the response rate (1)

• Shorter questionnaires• Good communication– Contact the respondents beforehand– Follow-up contact with non-responders

• Monetary incentives• Personal touch– Handwritten address– Including respondent’s name in cover letter

Improving the response rate (2)

• Method of delivery– Include a stamped return envelope– Send by first class or recorded delivery– University sponsorship can add credibility

• Assure confidentiality

Key take-home message

Pilot your tool

• This is the MOST IMPORTANT part.• If a range of respondents from different

backgrounds will be using the questionnaire make sure you use as many of these different groups as possible in the trial.

• Go back to your objectives – does the data help you fulfil your objectives and measure against the standards you set?

39