Beyond Satisfaction

Questionnaires:“Hacking” the Online

Survey

Andrea Evansandrea.evans@oracle.com

HCII2013

“Hacks”: non-traditional uses of online surveysContent Hacks: not

satisfaction/rating/demographicSoftware Hack: off-label, un-advertised

functionalityAll examples are directly from user research

at Oracle

Simple ComparisonsTwo or more static screenshots on the same page.If you can’t collect NDAs, keep the screenshots

generic.Randomize screenshot order to prevent order

effects.Simple Preference: “Which do you prefer?”Comprehensibility: “Which is easier to

understand?”Look and Feel: “Which looks cleaner/more

modern?”Or the question can show how the user interprets

the meaning of the designs.

A B

Which of these pages contains an editable table?o Page Ao Page Bo Both pageso Neither pageo Not sure

Progressive ComparisonsDecide among designs which vary along a

single continuum by subtle degrees (e.g., range of colors)

Grayscales

Blues

Progressive ComparisonsThe survey tool must have Branch Logic.Show pairs of images on the same page.The first pair is the most dissimilar (1 & 5 of a

set of 5)If they choose 1, the next pair they see is 1 & 4If they choose 5, the next pair they see is 2 & 5Pairs get more similar every time,

until the user says a pair looks about the same,then the survey ends.

Takes account of users’ sensitivity to subtle differences.

Which background do you prefer?

o I prefer the first backgroundo I prefer the second backgroundo They both look about the same

Walkthrough: Selection PersistenceDo users expect their selections to persist

when they navigate to another tab,even when they don’t click OK or Apply first?

Rather than just asking users, walk them througha sequence of generic screenshots with callouts.

Imagine yourself interacting with the following screens,then choose which outcome you prefer.Initially, the first checkbox is not checked.

Now, you have checked the first checkbox.

Now, you have switched immediately to a different tab.

If you switched back to the previous tab, which would you prefer to see?

A B

Walkthrough: Preliminary DesignsShow static screenshots of early design

concepts,and ask users how they would perform simple tasks.

Be careful to include all relevant response options.

Always include these options as well:“Other:” with a text field. “Don’t Know” / “Unsure”

On this screen, how would you edit the message text to make some parts of it bold?

o Select the text to be bolded and press Control-B (PC) or Command-B (Mac)o Type <b> </b> tags around the text to be boldedo Type <strong> </strong> tags around the text to be boldedo I don’t knowo Other:

Software Hack: Chronological Order of ClicksNeeds entry-level JavaScript, and HTML Image

Maps.The survey tool must allow you to insert code,

e.g. by manually editing <img> tags after uploading images, or by adding HTML in text blocks.

Not all survey tools allow this code insertion.Vovici allows it. Demo at http://ow.ly/aYLOUSurveyGizmo allows it. Demo at http://ow.ly/g5dezSurveyMonkey does not allow it.

The <map> tag includes the <img> tag, and a set of <area> tags, one for each clickable area in the image.

Software Hack: Chronological Order of ClicksThe JavaScript includes an array to record each

area the user clicks in, in chronological order.An essay question captures the array’s output.This essay question is left visible in the online

demos,so you can see the contents of the array.

In a real survey, hide this essay question from users.Prefix the question’s label with<div style="display:none;">

Add a text block immediately after the question, containing only the closing </div> tag.

Vovici http://ow.ly/aYLOUSurveyGizmo http://ow.ly/g5dez

JavaScript & Image Map for Clickstream Demo<script type="text/javascript">var clickCount=0;var clickArray=new Array;function planets(correct,planet) {  clickCount++; clickArray.push(planet);  if (correct) {   document.getElementById("Q1").value=clickArray;   alert("Correct! That took you " + clickCount + " click(s).");  } else {   alert("Incorrect: this is " + planet + "!");  }}</script><map name="planets"><area href="#" coords="68,186,6" shape="circle" id="Mercury" onclick="planets(false,this.id)" /><area href="#" coords="101,180,12" shape="circle" id="Venus" onclick="planets(false,this.id)" /><area href="#" coords="136,178,10" shape="circle" id="Earth" onclick="planets(false,this.id)" /><area href="#" coords="175,175,8" shape="circle" id="Mars" onclick="planets(false,this.id)" /><area href="#" coords="223,169,27" shape="circle" id="Jupiter" onclick="planets(true,this.id)" /><area href="#" coords="285,170,14" shape="circle" id="Saturn" onclick="planets(false,this.id)" /><area href="#" coords="357,160,15" shape="circle" id="Uranus" onclick="planets(false,this.id)" /><area href="#" coords="449,154,11" shape="circle" id="Neptune" onclick="planets(false,this.id)" /><area href="#" coords="513,142,4" shape="circle" id="Pluto" onclick="planets(false,this.id)" /><!--hack to stop Vovici corrupting img tag is to break it over two lines--><img border="0" src="/AppData/1885224700/users/256801141/User%20Media/planets.png" usemap="#planets" /></map>

Thank you!