hypothesis testing for y e · 2020. 6. 5. · discovery testing ideas! hypothesis generation does...

everyoneHypothesis testing for

Nick FineBAD 2020

@doctorfine@Nick Fine

1996 BSc (Hons) Psychology | heavy MUDding | Worms Armageddon | C&C

2004 MSc Human Computer Interaction with Ergonomics2009 PhD Human Computer Interaction, Personalisation | Eve Online

2010 User Experience Executive, LV=

2011 Senior User Experience Consultant, Redweb

2013 Head of User Experience, SAS/MSLGROUP

2014 Lead User Experience Architect (contract), Virgin Media

2016 Lead User Experience Architect (contract), Camelot & VML London

2017 Lead User Researcher (contract), Home Office

2018 Lead User Researcher (contract), Eurostar & Greater London Authority

ACADEMIC STUPIDITY

COMMERCIAL UX

EXPERIENCE

2019- Lead User Researcher (contract), Shell, Emergn @ GSK

EARLY COMPUTING

INTERNET PIONEERING

1997 Senior Technical Consultant, Demon Internet

DOTCOM BOOM

1998 Head of Technical Pre-Sales, Internet Network Services

1999-03 Chief Technical Officer, EO plc

1999-03 Technical Advisor, New Media Spark plc

1985 First modem. Hacked UK car manuf., bank and florist on BBC Model B | Elite

1991 Active member on SFnet, coffee table network in San Francisco

1979 First computer, Tandy TRS-80. First console, Atari 2600

PSYCHOLOGY!

1990 Introduced first computing, Mac 2SE & Filemaker Pro at Marvel Comics UK

1996 BSc (Hons) Psychology | heavy MUDding | Worms Armageddon | C&C

2004 MSc Human Computer Interaction with Ergonomics2009 PhD Human Computer Interaction, Personalisation | Eve Online

2010 User Experience Executive, LV=

2011 Senior User Experience Consultant, Redweb

2013 Head of User Experience, SAS/MSLGROUP

2014 Lead User Experience Architect (contract), Virgin Media

2016 Lead User Experience Architect (contract), Camelot & VML London

2017 Lead User Researcher (contract), Home Office

2018 Lead User Researcher (contract), Eurostar & Greater London Authority

ACADEMIC STUPIDITY

COMMERCIAL UX

EXPERIENCE

2019- Lead User Researcher (contract), Shell

EARLY COMPUTING

INTERNET PIONEERING

1997 Senior Technical Consultant, Demon Internet

DOTCOM BOOM

1998 Head of Technical Pre-Sales, Internet Network Services

1999-03 Chief Technical Officer, EO plc

1999-03 Technical Advisor, New Media Spark plc

1985 First modem. Hacked UK car manuf., bank and florist on BBC Model B | Elite

1991 Active member on SFnet, coffee table network in San Francisco

1979 First computer, Tandy TRS-80. First console, Atari 2600

PSYCHOLOGY!

1990 Introduced first computing, Mac 2SE & Foxbase at Marvel Comics UK

• What is hypothesis testing? • Who is it for? • Working with Product • How do I do it? How do we do it? • Storing hypotheses • Sharing and socialising

What we’ll be covering

What's in it for you ?tried and testing way of learning about stuff you learn better stuff

raise your profile at work by leading the thinking

demonstrable value = job security, £££, builds confidence

build better product better experience, better folio

AUTHORS

INSIGHTS

TECHNIQUES

TOOLS

METHODS EFFECTS

??

??

??

??

?? ??

THINGS YOU LEARN FOR YOUR SELF

THINGS YOU PRACTICE AT WORK



??? ???



HYPOTHESIS TESTING

HYPOTHESIS

HYPOTHESIS =

IDEA

IDEATHIS IS ALL ABOUT

TESTING

IDEASEVERYONE HAS

you! your boss your bosses boss

product development deliverydesign contentmarketing

the client

your partner some dude David Icke

IDEAS USERS

IDEAS USERS

USE

R E

XPER

IEN

CE

IDEAS USERS

UX

MAR

KET

ING

R&D

IDEAS USERS

PRO

DU

CT

MAR

KET

ING

R&D

UX

AGEN

CY

AGEN

CY

AGEN

CY

AGEN

CY

AGEN

CY

AGEN

CY

MAR

KET

ING

R&D

AGEN

CY

AGEN

CY

AGEN

CY

AGEN

CY

AGEN

CY

AGEN

CYIDEAS USERS

PRO

DU

CT

UX HYPOTHESIS

TESTING

MAR

KET

ING

R&D

AGEN

CY

AGEN

CY

AGEN

CY

AGEN

CY

AGEN

CY

AGEN

CYIDEAS USERS

PRO

DU

CT

UX

TRYING OUT A COOL IDEA I LEARNED AT BAD2020 TO SEE IF IT WORKS FOR OUR USERS

ORGANISE YOUR IDEAS

MANAGE YOUR IDEAS

STRUCTURE YOUR

HYPOTHESES

What do I mean by

STRUCTURE ?

1. Structured HYPOTHESES2. Structured INQUIRY

What do I mean by

STRUCTURE ?

HYPOTHESIS STRUCTURE

If I do [ ] then [ ]

If I make the CTA green then more people buy more

If I add a chatbot to the product page then people will be able to get help quicker and buy more

CHANGEACTION

If I choose stock images with individuals not looking directly at the camera, then people will feel more comfortable on our website. I will know this to be true when time on page and/or pages consumed increases from repeat visitors by at least 20%

If I do [ACTION] then [CHANGE]I will know this to be true when [MEASURABLE OUTCOME]

1. Structured HYPOTHESES2. Structured INQUIRY

What do I mean by

STRUCTURE ?

STRUCTURED INQUIRY

Structuring your learning activity

i.e. how do you learn about stuff?

UNSTRUCTURED STRUCTURED

HYPOTHESIS #1

HYPOTHESIS #2

FAIL

FAIL

HYPOTHESIS #1

HYPOTHESIS #2

FAIL

FAIL

HYPOTHESIS #3

PASS

HYPOTHESIS #3

FAIL


HYPOTHESIS #1

HYPOTHESIS #2

HYPOTHESIS #3

FAIL

FAILFAIL

HYPOTHESIS #1

HYPOTHESIS #2

FAIL

FAIL

HYPOTHESIS #3

PASS

Guessing


HYPOTHESIS #1

HYPOTHESIS #2

HYPOTHESIS #3

FAIL

FAILFAIL

HYPOTHESIS #1

HYPOTHESIS #2

FAIL

FAIL

HYPOTHESIS #3

PASS

LearningGuessing

THE STRUCTURE THAT COMES FROM HYPOTHESIS TESTING CHANGES YOU, YOUR TEAM AND YOUR BUSINESS INTO A

LEARNING MACHINE

• Break your ideas down into manageable hypotheses

• investigate one thing at one time • don’t be overambitious • repeat hypotheses with different samples

LEARNING MACHINETop Tips:

VARYING YOUR HYPOTHESESaka trying something a little different to see if it

works any better

CALL TOACTION

COLOUR SHAPE POSITION

H S L ROUNDED SQUARE TOP BOTTOM BOTH

• Be agile about it • learn small • fail fast • iterate quickly

LEARNING MACHINETop Tips:

How do I fit this into my ways of working?How does this work on agile teams?

PRODUCT TEAM SPRINT 1

WEEK 1 WEEK 2DISCOVERY TESTING



IDEAS!

HYPOTHESIS GENERATION

DOES IT WORK?

HYPOTHESIS TESTING

PR

OTO

TY

PIN

G





IDEAS!


DOES IT WORK?

HYPOTHESIS TESTING

IDEAS!


PR

OTO

TY

PIN

G

PR

OTO

TY

PIN

G

DEV TEAM SPRINT 1

DEV TEAM SPRINT 2

DEV TEAM SPRINT 3





DEV TEAM SPRINT 2

DEV TEAM SPRINT 3




DEV TEAM SPRINT 1


VALIDATED IDEAS

DEV TEAM SPRINT 1

DEV TEAM SPRINT 2

DEV TEAM SPRINT 3





MANAGING YOUR RESEARCH ACTIVITY

IDEASHYPOTHESES PERSONA

DECKINSIGHT

SPORTS OBSESSED

DAD

47%of users

in May 20

IDEAS• Personalised landing• Learn primary sport• Effect of team badges?

SPORTS OBSESSED

DAD

47%of users

in May 20

HYPOTHESES

• If we show the user's favourite club badge on the CTA, we will see more sales, more pages consumed

SPORTS OBSESSED

DAD

47%of users

in May 20

HYPOTHESES

• If we show the user's favourite club badge on the CTA, we will see more sales, more pages consumed

FAILED Sprint 14

SPORTS OBSESSED

DAD

47%of users

in May 20

HYPOTHESES• If we show the user's favourite club badge on the order summary page, we will see more sales and less bounce SPRINT 18-19

SPORTS OBSESSED

DAD

47%of users

in May 20

INSIGHTS

• Showing the latest scores in the banner increased scroll depth by 25% for new users (Sprint 4)

Retired Repeat

BRIEF OVERVIEW IDEAS

USER NEEDS

PAIN POINTS

TECHNOLOGY USE

MOTIVATORS

HYPOTHESES

August 2019 Mary Person

INSIGHTS

Sports Obsessed Dad

PUTTING IT ALL TOGETHER

IDEA HYPOTHESIS STRUCTURE

IDEA

HYPOTHESIS

STRUCTURE

1

2

3

e.g. Be inspired at BAD conference

Write a few hypotheses to explore the idea

Think about how you might structure your investigation, one hypothesis at a time

VARY YOUR HYPOTHESIS

STORE IT ALL IN YOUR PERSONAS

SHARE SHARE SHARE!

4

5

6

Try variants to see if they perform any better

Store ideas, hypotheses, insights

Share successes and failures with your team and wider organisation so that they can benefit from your learning

IDEA

HYPOTHESIS

STRUCTURE

VARIATION

STORAGE

SHARING

UX PSYCHOLOGY

UX PSYCHOLOGYThe future of UX involves developing deep behavioural understanding of our users' needs from multiple sources, including direct observation and machine learning. UX Psychology seeks to develop psychology knowledge and practices within UX to support this behaviourally centric future.

UX PSYCHOLOGYUKAUSTRALIAGERMANY

Nick FineDavid Di SipioMark Drexler

Nick Fine

@[email protected]

Questionsand thanks :)

hypothesis testing for y e · 2020. 6. 5. · discovery testing ideas! hypothesis generation does...

Documents