#ouruxflow webinar: finding team synchronicity in an iterative design process

Team Synchronicity in an Iterative Design Process

July 23, 2015 | 11:00 am PDTTryMyUI webinar

#ourUXflow@romanocog @trymyUI

Jen Romano-BergstromUX Researcher | FacebookMarketing Director | [email protected]

Ritvij GautamCEO & Co-Founder | [email protected]

mailto:[email protected]


Iterative Usability Testing

2

Iterative Process Throughout Development


Iterative Usability Testing

3

Iterative Process Throughout Development

Benchmark


Who is involved

4

SATQ Items

Other materials

Debriefing Reports

.

. .

.

. .

.

“Make it a spectator sport” @skrug

Project Managers

UX Designers

Executives

Engineers

Researchers


Why people do not conduct iterative testing

5

• “There is no time.”• “We don’t have a budget.”• “That’s way too many people to coordinate.”• “I can’t recruit representative users quickly.”• “We don’t have a lab.”


Benefits of Unmoderated Remote Testing

6

• No moderator bias• No coaching1,2

• Participants in natural environment

• Hard to reach participants• Lots of data quickly• Data from many cities,

states, countries• No travel costs

1. Olmsted-Hawala, E. L. & Romano Bergstrom, J. C. (2012). Think-aloud protocols. Does age make a difference? Proceedings from the Society for Technical Communication Summit, May 2012, Chicago, IL.

2. Olmsted-Hawala, E. L., Murphy, E. D., Hawala, S., & Ashenfelter, K. T. (2010). Think-aloud protocols: A comparison of three think-aloud protocols for use in testing data-dissemination web sites for usability. Proceedings from CHI, April 2010, Atlanta, GA.

Higher accuracy and satisfaction when moderators “coach.”


7

Qualitative QuantitativeSelf-ReportSatisfaction and knowledge questionnaires YES YES

Verbal think aloud YES NOModerator follow up NO NOObservationalTime on page/task NO YESSelection/click behavior YES YESSuccess/fail rate NO YESConversion rate YES YESImplicitVerbalization analysis YES YESEye tracking YES YES

#ourUXflow@romanocog @trymyUIUnmoderated Remote UX Data

Iterative Process – example timelineProduct Overhaul

8

Task development

Iteration 1 testing: 6 days

N=7

Iteration 1 findings and

recommendations presented to design

team: 13 days later

9.8.08 10.23.0811.10.08

Meeting with design

team: 13 days later

Iteration 2 testing: 1 month

N=14

12.12.08 6.23.09

Iteration 2findings and


team: 8 days later

7.31.09

Meeting with design

team: 2 weeks

later

8.12.09

Iteration 3: Novice users

testing: 8 days

N=9

9.09.09


findings and recommendations

presented to design team:

8 days later

Meeting with design

team: 2 weeks

later

10.08.099.25.09

Iteration 3: Expert users

testing: 3 days

N=4

10.14.09


Romano Bergstrom, J. C., Olmsted-Hawala, E. L., Chen, J. M., & Murphy, E. D. (2011). Conducting iterative usability testing on a Web site: Challenges and benefits. Journal of Usability Studies, 7, 9-30.

9

Task development

Iteration 1 testing: 6 days

N=7

Iteration 1 findings and


team: 13 days later

9.8.08 10.23.0811.10.08

Meeting with design

team: 13 days later

Iteration 2 testing: 1 month

N=14

12.12.08 6.23.09

Iteration 2findings and


team: 8 days later

7.31.09

Meeting with design

team: 2 weeks

later

8.12.09


testing: 8 days

N=9

9.09.09


findings and recommendations

presented to design team:

8 days later

Meeting with design

team: 2 weeks

later

10.08.099.25.09

Iteration 3: Expert users

testing: 3 days

N=4

10.14.09


Romano Bergstrom, J. C., Olmsted-Hawala, E. L., Chen, J. M., & Murphy, E. D. (2011). Conducting iterative usability testing on a Web site: Challenges and benefits. Journal of Usability Studies, 7, 9-30.


10

• One morning a month with 3 users (Krug)• 12 people in 2 days (Lebson & Romano Bergstrom)• 16 people in 3 days (Patel & Romano Bergstrom)



Iterative Process – example timelineUnmoderated Remote Testing

11

• Thousands of people in one day• From different cities• Test different designs• Team is free to work on different things and assess data as

it comes in


A Typical UX Research Process

12

Data Collection:

● Create structured user flows for your website and have users navigate through them

● Get video/audio recordings and written responses to questions (PRIMARY RAW DATA)

● Additional data like SUS & SEQ, click streams, & heat-maps (AUXILIARY RAW DATA)

Data Analytics

● User researchers watch the videos and create notes of pain points.

● Share notes with the team -- they too watch the videos and add notes of their own.

● Collate the data, present a list of actionables to the Stakeholder/Client, explain why the

changes are necessary.

● Relay the changes and desired impact to the Dev team with relevant data. ITERATE...


Challenges

13

● Data != Insights

● Covering the different takeaways

from the results for the different

team members.

● Getting people “in on the loop” &

“on the same page”, so that they can

“circle back around” to “re-interpret

data”

● Getting all of this done in time for the

next build!


Data Analytics

14

● In doing UX research, we often get carried away with

the data collection process. In fact there is a terrible

pre-conceived notion that usability testing is purely a

data collection activity.

● We can collect primary & auxiliary raw usability data,

but if we do not establish an analytical methodology

that drives the way we extract insights from them, we

are simply wasting time.

● The auxiliary data should help you identify patterns

and trends in the primary data, but what you need is a

tool that will help you analyze and interpret it all.


“Big data is not about the data.”-Gary King, Harvard University

Example Auxiliary Data

15


Time Sensitivity

16

● The more tools/vendors/platforms used to manage

the data, the more unwieldy the analytical process.

● The more isolated the team members’ respective

analysis, the more disjointed the analytical process.

● Analysis that could take a week often takes twice or

thrice the amount of time because of gaps in

communication and the time wasted in between

meetings and follow ups.


“The ultimate inspiration is the deadline.”

- Nolan Bushnell, Founder of Atari

Team Synchronicity

17

● Usability testing data is relevant to your team members in different ways.

● They all need something from it, they view it through different lenses, and they draw

different insights from it.

● Aggregation of these insights will lead us to the treasure we seek!


Team Synchronicity: UX Researcher

18


It falls to the UX Researcher to watch all of the results and

identify key findings, judge what is important and what isn’t,

and then persuasively communicate these to decision makers.

To achieve his or her goals, therefore, what the UX Researcher

really needs is a way to easily pinpoint and reference critical,

demonstrative moments in the results to back up their

arguments and justify making changes to the design.

UX Researcher

Decision Maker

UI/UX Designer

Developer

Team Synchronicity: Decision Maker

19


The Decision Maker’s objective, on the other hand, is to make

an informed decision about the direction of the product

roadmap and which issues will be prioritized over others, based

on the topline analysis of what the pain points are.

What the Decision Maker needs to achieve their goals is an

efficient means of knowing and understanding the issues, and

seeing the evidence needed to green-light changes.

UX Researcher

Decision Maker

UI/UX Designer

Developer

20


The goal of the Designer, for their part, is to see why particular

design elements work or do not work, and then use that

information to create new solutions based on real user behavior.

For the Designer to achieve his or her goals, they need to be able

to directly access the results and witness users interacting with

the product at key junctures and hear their thoughts and

reactions when they run into walls.

Team Synchronicity: UI/UX Designer

UX Researcher

Decision Maker

UI/UX Designer

Developer

Team Synchronicity: Developer

21


The Developer’s role is to determine how & where in the

usability test (and the production code) a functionality issue

arises. They need to see the actions leading up to it so that they

can replicate it, identify it and solve it.

For the Developer to achieve his or her goals, they need to be

able to see the user’s actions leading up to an error and also see

which aspects of recurring user behaviour will dictate the way

interactive elements are coded.

UX Researcher

Decision Maker

UI/UX Designer

Developer

Why do we need to collaborate?

22

It is all about bettering the UX of Usability Testing:● Collaboration is inevitable. The data/data-derivatives will have to reach all members of the team

at some point. So why not optimize our workflow for it?

● Valuable insights with less organziational inefficiency.

● Increase:o Relevance of datao How many insights each unit of data yields

● Decrease:o Time taken to deliver & implement the suggestions for changeo Entropy and miscommunication within the team



23

It is all about bettering the UX of Usability Testing:● Collaboration is inevitable. The data will have to reach all members of the team at some point. So

why not optimize our workflow for it?

● Valuable insights with less back and forth

● Increase:o Relevance of datao How many insights each unit of data yields

● Decrease:o Time taken to deliver & implement the suggestions for changeo Entropy and miscommunication within the team



24



25



26



27



28


Team Synchronicity in an Iterative Design Process

July 23, 2015 | 11:00 am PDTTryMyUI webinar


Jen Romano-BergstromUX Researcher | FacebookMarketing Director | [email protected] romanocog.com

Ritvij GautamCEO & Co-Founder | [email protected] Trymyui.com/webinar



#ouruxflow webinar: finding team synchronicity in an iterative design process

Technology