how to get better results from a survey: meet the survey octopus

How to get better results from a survey

Caroline Jarrett@cjformsContent Strategy Summit 2015 #CSSummit

Caroline Jarrett @cjforms (CC) BY SA-4.0

I’m a forms specialist

2Image credit: Flickr, taxrebate.org.uk

https://creativecommons.org/licenses/by-sa/4.0/



Why do people answer questions?

3Caroline Jarrett @cjforms (CC) BY SA-4.0






A dollar bill with a mail survey worksbetter than $10 (guaranteed) later







Response relies on effort, reward, and trust

People will only respond if they trust you. After that, it's a balance between the perceived reward from filling in the survey compared to the perceived effort that's required. Strangely enough, if a reward seems 'too good to be true' that can also reduce the response.

Diagram from Jarrett, C, and Gaffney, G (2008) “Forms that work: Designing web forms for usability” inspired by Dillman, D.A. (2000) “Internet, Mail and Mixed Mode Surveys: The Tailored Design Method”

5




The aim of a survey is to get a number that helps you to make a decision




To get better results from your survey,think about the Survey Octopus




People ask me about surveys• “How many people do I need to ask?”• “How many questions can I have?”• “Please tell me whether this is a good question”• “I think it’s best to have 5 points in my rating scale, but

my boss wants to have 7. Who is right?”

9



Agenda “How many people do I need to ask?”

“How many questions can I have?”

“What makes a good question?”

“How many points in a rating scale?”





To find out how many people to ask, start at how many we need to answer


Fieldwork:Who answers?

11






Whether they’ll answer depends on effort

Questions:What are you asking about?How many questions?







And on the reward you’re offering

Goals:Why are you asking?Is helping you a reward in itself?Are you offering any other incentive?






Then there’s the ‘Justin Bieber North Korea’ problem

14

http://www.bbc.com/news/10506482


We don’t just want answers, we want answers from the right people

Response







So it matters where we get our sample from

Sample: the list you sample from







And now it’s easy to work out how many to ask


Sample: the number of people to ask






We thought about a lot of topics to work that outGoals

SampleQuestions

Fieldwork

Response18







To get really good results, we wantuseful answers from the right people

ResponseResponse







“You're using this site from outside the UK. Where are you answering from?We’d like answers like these

– Anchorage Alaska U.S.A.– Cameroon– Dubai, United Arab Emirates

Even better, extra-interesting answers like these:– Cote d'Ivoire / France / UK.– Canada, originally from the UK– Philippines, would like to work in UK with my husband if possible

But definitely not like these:– ä¸ å›½– its not important– nnnnnn– none of your business

20




Insights are the numbers that you use for decisions

Insights 21Caroline Jarrett @cjforms (CC) BY SA-4.0






We’ve thought about a lot of issuesGoals

QuestionsSample

Fieldwork

ResponseResponse

Insights 22Caroline Jarrett @cjforms (CC) BY SA-4.0






To get better results from your survey,think about the Survey Octopus

23





“What sorts of questions are best?”






“How many questions can I have?”Let’s work backwards from insights

25

Which answers do you need to make the decisions?




You need accurate answers to the questions

26

Will your respondents give you good answers to all of them?



27

In your last five days at work, what percentage of your work time do you estimate that you spent using publicly-available online services (not including email, instant messaging, and search) to do your work using a work computer or other device?

%




29

"Phone photography" by Petar Milošević - Own work. Licensed under CC BY-SA 3.0 via Commons - https://commons.wikimedia.org/wiki/File:Phone_photography.jpg#/media/File:Phone_photography.jpgModified by Caroline Jarrett

https://commons.wikimedia.org/wiki/File:Phone_photography.jpg#/media/File:Phone_photography.jpg




You need questions that your respondents can answer accurately

Which questions will get you the answers that you can use?

30




Now we get to fieldwork

31

How many of those questions are people willing to answer?




Fieldwork used to be expensiveso a survey was a rare event.

32Image credit: http://www.census.gov/history/www/genealogy/decennial_census_records/




1950s mindset: “Ask Everything”

33

Survey = Big Honkin’ Survey




2015 mindset: the Light Touch survey• Choose ONE question• Find ONE person• Ask the question, face-to-face• See if you can make ONE decision• Improve, iterate, increase

34



35

Tip Interview first




Time for new question


One way toiterate, improve,increase



Big Honkin’ Survey• Negotiate to get survey

down to 20 questions• Ask 10,000 people• Get 1,000 responses• Take a week (or more) to

analyze the responses• Have a big presentation

a month later

Light touch survey• ‘Question of the day’

(one question)• Ask 100 people• Get 50 responses• Analyze them same day• Present same day• Repeat 4 x 5 = 20 times





Do both!

Big Honkin’ Survey• Big numbers are

impressive• Can compare answers

from different segments• Easier ‘sell’ to

stakeholders

Light touch survey• Quick, useful results• Rapidly get better

at doing surveys• Wonderful way to test the

questions for the Big Honkin’ Survey

38



Caroline Jarrett @cjforms

40

"Phone photography" by Petar Milošević - Own work. Licensed under CC BY-SA 3.0 via Commons - https://commons.wikimedia.org/wiki/File:Phone_photography.jpg#/media/File:Phone_photography.jpgModified by Caroline Jarrett




41

TipAlways allow for ‘other’

Design by @RickyBuchanan; t-shirt from nopitycity.com or zazzle.co.uk

42

In your last five days at work, what percentage of your work time do you estimate that you spent using publicly-available online services (not including email, instant messaging, and search) to do your work using a work computer or other device?

%





Response relies on effort, reward, and trust

People will only respond if they trust you. After that, it's a balance between the perceived reward from filling in the survey compared to the perceived effort that's required. Strangely enough, if a reward seems 'too good to be true' that can also reduce the response.

Diagram from Jarrett, C, and Gaffney, G (2008) “Forms that work: Designing web forms for usability” inspired by Dillman, D.A. (2000) “Internet, Mail and Mixed Mode Surveys: The Tailored Design Method”

43




A good question works in three ways

Appropriate

Obvious Interesting




Why did you visit our website today?

Appropriate

Obvious Interesting




Would you recommend us to a friend or family member?

46

In a shop, buying a baby carriage

In a hospital,having a miscarriage

Obvious YesInteresting YesAppropriate Yes Cruelly

inappropriate



47

TipTest your questions by interviewing in context





Likert had several different types of question in his response formats

Likert, Rensis. (1932). A Technique for the Measurement of Attitudes. Archives of Psychology, 140, 1–55.

49




You can find an academic paper to support almost any number of points

• Krosnick and Presser refer to over 80 papers

50

Krosnick, J. A. and S. Presser (2009). Question and Questionnaire Design. Handbook of Survey Research (2nd Edition) J. D. Wright and P. V. Marsden, Elsevier.http://comm.stanford.edu/faculty/krosnick/docs/2010/2010 Handbook of Survey Research.pdf



http://comm.stanford.edu/faculty/krosnick/docs/2010/2010%20Handbook%20of%20Survey%20Research.pdf

51

Tip Don’t stress too much about the number of points in your rating scale

Picture credit: Flickr - Bill Soderman (BillsoPHOTO)


Well, OK, stress a little bit.

52

This scale is downright

peculiar. Avoid.



Summary





The aim is to get the best number you can, within the resources you have





What you want to ask about

The resources you have

The questions you ask

The answers you get

The answers you use

The number

Who you want to ask

The list that you sample from

The sample you ask

The ones who answer

The ones whose answers you can use

55








The answers you get

The answers you use

Who you want to ask

The list you use to sample from

The ones you ask

The ones who answer


The number 56




Survey statistic

Post-survey adjustments

Respondents

Sample

Sampling frame

Representation

Edited response

Response

Measurement

Construct


Resources




The answers you get

The answers you use

Who you want to ask

The list you use to sample from

The ones you ask

The ones who answer




58

Total Survey Error diagram as presented in Groves, R. M., F. J. Fowler, M. P. Couper, J. M. Lepkowski, E. Singer and R. Tourangeau (2009). Survey methodology. Hoboken, N.J., Wiley.


Caroline JarrettTwitter @cjforms

http://www.slideshare.net/cjforms

[email protected]

59

