introduction to ux research: designing surveys that don't suck!

Will Evans

Yana Kuchirko

"They've done studies, you know. 60% of the time, it works every time. ”

- Brian Fantana

04/07/23 2

04/07/23 3

Surveys Aren’t Objective

They are created by people who are biased and analyze their results based on their own (mis)perceptions.

Epistemological Dead End?

No. Awareness that the researcher plays an integral role in the process of measuring any given phenomena by deciding how to measure is key.

There are ways to minimize researcher bias by creating better questions.04/07/23 4

Methods are not “just methods”

04/07/23 5

How you measure what you are studying shapes what you find.

Let’s say you are studying…

04/07/23 6

How often teenagers use your website

You might ask teenagers…

“How often do you my website?”

a.Very rarelyb.Rarelyc.Occasionallyd.Frequentlye.Very frequently

04/07/23 7

Response options source: http://www.dataguru.org

You might ask teenagers…

“How often do you my website?”

a.Very rarelyb.Rarelyc.Occasionallyd.Frequentlye.Very frequently

04/07/23 8

Response options source: http://www.dataguru.org

And what’s wrong with these????

You might ask teenagers…

“How often do you my website?”

a.Very rarelyb.Rarelyc.Occasionallyd.Frequentlye.Very frequently

04/07/23 9

Response options source: http://www.dataguru.org

And what’s wrong with these????

SUBJECTIVITY!!!

“Hmmm, for me “frequently” really means….”

A few times a week A few times a day

So what are we really measuring? NOISE.

04/07/23 10

What is “noise”?

What we observe…

04/07/23 11

We WANT this!We WANT this!

We DON’T WANT this!We DON’T WANT this!

04/07/23 12

What is “noise”?

Let’s really stretch our thinking a bit here to provide a more concrete understanding of “noise” in your data.

Imagine you have AT&T phone service and you’re trying to make a call. The signal isn’t clearly going through and you end up hearing everything but the other person’s voice. Frustrating, right? Thank God this is only a pretend scenario.

04/07/23 13

An imaginary scenario

How is AT&T related to “Noise”?

When you develop bad questions, you don’t “hear” the message of your data clearly.

04/07/23 14

So…why is “noise” bad???

• Badly structured and poorly worded questions that obfuscate meaning for participants provide bad data.

• Bad data– Doesn’t answer your research question–Makes it difficult to interpret results– Is pretty much useless.

04/07/23 15

More bad examples of frequently used response

optionsa. Completely satisfiedb. Very satisfiedc. Fairly well satisfiedd. Somewhat dissatisfiede. Very dissatisfied

a. Totally likeb. Very much likec. Moderately liked. Somewhat likee. Not like

What’s the difference between “fairly well” and “somewhat”?

What’s the difference between “moderately like” and “somewhat like”?

04/07/23 16

a. Completely satisfiedb. Very satisfiedc. Fairly well satisfiedd. Somewhat dissatisfiede. Very dissatisfied

a. Totally likeb. Very much likec. Moderately liked. Somewhat likee. Not like

What’s the difference between “fairly well” and “somewhat”?

What’s the difference between “moderately like” and “somewhat like”?

Semantics04/07/23 17

More bad examples of frequently used response

options

Words like….

• Sometimes• Often• Moderately• Very• Not very much

Are subjective = they mean different things for different people

04/07/23 18

A better alternative?

Actually asking people how often they do certain activities.

04/07/23 19

In order to gather objective data…

…your questions must mean the same thing for everyone.

How often do you use my website?a.Neverb.A few times a yearc.Once a monthd.2-3 times a monthe.Once a weekf.A few times a weekg.Every day

“Never” means never for everyone!04/07/23 20

Importance of “Anchors”

Responses options serve as “anchors” for each question, determining the “location” of each responses as qualitatively distinct from the other.

04/07/23 21

Responses options serve as “anchors” for each question, determining the “location” of each responses as qualitatively distinct from the other.

04/07/23 22

Importance of “Anchors”

Strongly disagree Disagree Neutral Agree

StronglyAgree

(Ideal) Response Symmetry

• Good questions aim to have symmetrical quantitative/qualitative distance between anchors a. Neverb. A few times a yearc. Once a monthd. 2-3 times a monthe. Once a weekf. A few times a weekg. Every day

The distance between “never” and “a few times a year” is proportionate to “every day” and “a few times a week”.

04/07/23 23

Adherence to Logic & Linearity

People are accustomed to paradigms that are intuitive, and often “linear”.

Anchors should not be the exception.04/07/23 24

Intuitive Anchor Directions

Never Always

Disagree Agree

Very poor Very good

Not very important

Very Important

04/07/23 25

Value of Clarity

If questions are clear and concise, participants would spend less time analyzing the questions themselves and more time on answering them. Mystic Arts, LLC

04/07/23 26

So do good questions guarantee valid results?

• No. But good questions offer more assurance that you are listening to signal and not the noise.

• But answering your research question can be done in other ways

• Behavioral Observations• Open ended qualitative questions• Many more…

04/07/23 27

The End.

So was this presentation simply awesome?

a. Hell yeah!b. Definitely!c. Totally rocked!

04/07/23 28

Thanks!