questionnaire design 2012
TRANSCRIPT
-
8/10/2019 Questionnaire Design 2012
1/37
Research Ski l ls One
Oh, no!Please, no!
Not statistics!!
-
8/10/2019 Questionnaire Design 2012
2/37
A ims o f the two f i rst year Research Ski lls
courses:
Basic gro und ing in research sk i l ls -
designing q uest ionnai res and exper iments
data analysis us ing stat ist ics, Excel and SPSS
wri t ing up stud ies using s tandard presentat ion convent ions
-
8/10/2019 Questionnaire Design 2012
3/37
-
8/10/2019 Questionnaire Design 2012
4/37
-
8/10/2019 Questionnaire Design 2012
5/37
-
8/10/2019 Questionnaire Design 2012
6/37
Comparison of RM1 exam marks for 2004-2008
0
5
10
15
20
25
30
35
40
0-10 11-20 21-30 31-40 41-50 51-60 61-70 71-80 81-90 91-100
exam mark
frequency(%)
2004 (N = 249)
2005 (N = 246)2006 (N = 243)
2007 (N = 214)
2008 (N = 234)
-
8/10/2019 Questionnaire Design 2012
7/37
-
8/10/2019 Questionnaire Design 2012
8/37
Outl ine of this c ou rse (Research Ski l ls One):
Issues in designing questionnaires and experiments.
Descriptive statistics - effective summarising of data.
Statistical tests (correlation, Chi-Square, nonparametric tests for
comparing groups or conditions).
Using SPSS("Statistical Package for the Social Sciences") for data
analysis and Excelfor producing graphs and tables.
-
8/10/2019 Questionnaire Design 2012
9/37
Research Ski l ls Two:
More statistical theory.
More statistical tests.
Practice at running experiments and writing them up.
Ethics.
-
8/10/2019 Questionnaire Design 2012
10/37
Organisat ion for th is cou rse:
One lecture and one practical session every week.
(Lectures: Thursdays at 1pm in Chichester Lecture Theatre
until week 12December 6th).
Practical sessions - details on Sussex Direct (you are split into
lots of smaller groups).
-
8/10/2019 Questionnaire Design 2012
11/37
Course requi rements:
Assessment is by a mixture of coursework and
exam:
Coursework
SONA research participation (4 hours this term): 10%
Lab-report 1 10%
Lab-report 2 20%
Exam -
Unseen exam (in winter assessment period): 60%
Overal lpass mark (exam and coursework combined): 40%
-
8/10/2019 Questionnaire Design 2012
12/37
This term's requi rements:
1. Write a lab-report based on a questionnaire from the STARS fast
food project (10%).
2. Write a lab-report based on a questionnaire on maths learning
experiences (20%).
3. Tackle the weekly statistics problem sheets (not marked, but
essential practice).
4. Four hours of research participation (10%).
5. Unseen 2-hour exam at the end of the course
statistical concepts (multiple choice questions);
deciding which statistical test is most appropriate;
interpreting statistical output from SPSS.
-
8/10/2019 Questionnaire Design 2012
13/37
Recommended textbook s:
Field and Hole (2003). Sage publications.
Palgrave, 2006
Sage, 2006
BUY THIS ONE, I NEED THE MONEY !!!!!
http://www.amazon.co.uk/gp/product/images/0230362729/ref=dp_image_z_0?ie=UTF8&n=266239&s=bookshttp://www.amazon.co.uk/gp/product/images/0230362729/ref=dp_image_z_0?ie=UTF8&n=266239&s=books -
8/10/2019 Questionnaire Design 2012
14/37
-
8/10/2019 Questionnaire Design 2012
15/37
Perm it ted calculators:
Casio FX82, 83, 85, 115, 570 or 991, all with any suffix.
No oth er type of calculator is acceptable.
-
8/10/2019 Questionnaire Design 2012
16/37
Course documentat ion, hando uts, lecture sl ides:
Go to Sussex Direct - this course is
Research Skills One (C8511)
OR
Type Graham Hole into Google
-
8/10/2019 Questionnaire Design 2012
17/37
Click on
this link
-
8/10/2019 Questionnaire Design 2012
18/37
Everything
you need to
know for this
course
-
8/10/2019 Questionnaire Design 2012
19/37
Who to con tact :
In the first instance, the course tutors running your practical
session.
Graham Hole ([email protected])
-
8/10/2019 Questionnaire Design 2012
20/37
Quest ionnaire design :
Perhaps now you'll fill in
my questionnaire?
-
8/10/2019 Questionnaire Design 2012
21/37
How does the way you collect data affect the datacollected?
How does question wording affect peoples answers?
-
8/10/2019 Questionnaire Design 2012
22/37
Stages in quest ionnaire design :
5. Adm inister the quest ion naire, after i t has b een revisedin th e l igh t o f (4).
1. Formulate the research quest ion(s) clearly.
2. Ident i fy the popu lation and samp le.
3. Design the quest ionnai re: th ink about quest ion
wo rding ; quest ionn aire formatt ing; mode ofadm inistrat ion ; data analysis.
4. Pre-test the questionnaire.
-
8/10/2019 Questionnaire Design 2012
23/37
Samples and popu lat ions:
Sample: a subset f rom a
po pu lat ion (e.g. f irs t-year
psy cho logy students).
Populat ion:a complete set
of th ing s (e.g. al l of
humanity).
For val id inferences to be made abou t a populat ion 's
characterist ics, a sample mu st be representat ive of i ts
paren t pop ulat ion (e.g. sim ilar in age, SES, IQ, etc.)
-
8/10/2019 Questionnaire Design 2012
24/37
Methods of obtain ing quest ionnaire data:
Postal
questionnaire
Personal
interview
Phone
interview
Internet
questionnaire
Cost Low High Moderate Low
Data quality:
Respo nse r ate
Respondent
mot ivat ion
Interviewer bias
Low
Low
None
High
High
Moderate
Moderate
High
Low
Low
Low
None
Sample quality: Low, unless
high responserate
High Moderate to
high,especially
with random
digit dialling
Moderate, but
improving asInternet access
widens
-
8/10/2019 Questionnaire Design 2012
25/37
Postal
Questionnaire
Personal
interview
Phone
interview
Internet
questionnairePossible interview
length:
Short Very long Long Short
Ability to clarify
and probe:
None High High None
Anonymity: High Low Low Low
Dependence on
respondents
literacy:
High None None High
Control of context
and questionorder:
None High High Depends
-
8/10/2019 Questionnaire Design 2012
26/37
-
8/10/2019 Questionnaire Design 2012
27/37
Quest ionnaire wording :
1. Should be exact.
2. Should be simple.
3. Avoid biased or emotive words.
Schuman and Presser (1981): subtle changes of wording
may influence responses.
e.g. Should the Governmentallow publ ic speeches by a
Communis t? produced 25% fewer pro-free-speech
responses when allowwas replaced with forb id .
-
8/10/2019 Questionnaire Design 2012
28/37
4. Make all alternatives clear.
e.g. Payne (1951): "Do you th ink most manufactur ing
com panies that lay of f wo rkers dur ing slack per iods cou ldarrange thing s to avoid layof fs and give steady work
througho ut the year?"
63% - companies could avoid layoffs.
22% - couldnt avoid layoffs.
15% - no opinion.
Same question plus phrase "or do you think layof fs are
unavoidable?"
35% - companies could avoid layoffs.
41% - couldnt avoid layoffs.
24% - no opinion.
-
8/10/2019 Questionnaire Design 2012
29/37
5. Avoid the format: "Some people say x: do y ou agree ordisagree?"
6. Avoid unwarranted assumptions. e.g. "What is youroccupation?" assumes person has a job.
7. Avoid double-barrelled questions. e.g. "Should immigrants
be repatriated and their possessions confiscated?" is twoquestions.
8. Avoid double negatives. e.g. "Are you against a ban onsmoking?"
9 C id h l i i f d
-
8/10/2019 Questionnaire Design 2012
30/37
9. Consider the relative merits of open-endedand
closed-endedquestions.
Open-ended:allow unconstrained responses.
e.g. "How do you travel to the Univers i ty?".
May produce richly detailed responses, but hard and tediousto score.
Closed-ended:require choice from a limited range ofalternatives.
e.g. "Do you travel to the Univers i ty by
(a) bu s, (b) car, or (c) un icy cle (t ick one)".
Easy to code, but prone to bias.
-
8/10/2019 Questionnaire Design 2012
31/37
Closed-ended questions must have
(a) a balanced response scale;
(b) mutually exclusive categories;
(c) facilities for handling "don't know" and "other"
responses.
R ti l
-
8/10/2019 Questionnaire Design 2012
32/37
Rating scales:
The Likert Scale:
"Criminals should be flogged".Strongly agree Agree Neither agree
nor disagree
Disagree Strongly disagree
1 2 3 4 5
Can be 5-. 7- or 9-point scale (doesn't make muchdifference).
Visual Analogue Scale:
Strongly agree Strongly disagree
-
8/10/2019 Questionnaire Design 2012
33/37
Prob lems w i th quest ionnai res abou tatt i tudes:
1. May not have an attitude - "doorstep opinions".
2. Attitudes may be complex and multi-dimensional.
3. Attitudes vary in intensity.
4. Expressed attitudes may depend on question wording,
sequence and interviewer effects.
Prob lems w i th quest ionnaires abou t
-
8/10/2019 Questionnaire Design 2012
34/37
Prob lems w i th quest ionnaires abou tbehaviour:
1. Memory l imi tat ions-
e.g. Chapman and Underwood's (2000) study of drivers' memory for accidents
and near-misses.
Can be counteracted by
(a) asking specific questions;
(b) asking for birth date rather than age;
(c) using a chronological format;
(d) re-interviewing.
2. Response biasesdue to social desirability or suspicion,
especially for illegal or anti-social activities.
Can be counteracted by ensuring anonymity.
What are the prob lems w i th these quest ions?
-
8/10/2019 Questionnaire Design 2012
35/37
5. Do your children prefer to eat in KFC or Macdonalds?
1. Do you visit fast food emporia regularly?
Over-complex wording. Exactly what does "regularly" mean?
2. How many burgers do you eat per month?
Assumes you eat burgers.
3. Some people suggest that fast food is leading to increased tooth decay and an
increase in obesity amongst teenagers in many parts of the U.K. Do you agree?
Over-long. Includes two separate questions. Implies you should agree with the viewsexpressed.
4. Which of the following methods do you use to travel to your fast-
food outlet? (a) Bus (b) Car (c) Bicycle
Does the questioner want you to choose only one option, or can you
choose more? No option for responding "other" .
What are the prob lems w i th these quest ions?
Assumes you have children.
-
8/10/2019 Questionnaire Design 2012
36/37
Conclus ions:
It's hard to design questionnaires properly!
Always be sceptical of survey results -ask yourself
Who were they collected by?
Who were they collected from?
How were the questions worded, exactly?
Remember - 8 out of 10 cats prefer Whiskas" has
become "In tests, 8 out of 10 cats who expressed apreference, preferred Whiskas".
Useful references:
-
8/10/2019 Questionnaire Design 2012
37/37
Useful references:
Burgess, T.F. (2001). A general int roduct ion to the design of q uest ionnaires
for su rvey research.
http://iss.leeds.ac.uk/info/312/surveys/217/guide_to_the_design_of_questionnaires
Taylor-Powell, E. (1998). Quest ionnaire design: asking quest ions w ith a
purpose.http://learningstore.uwex.edu/pdf/G3658-02.pdf