a/b testing - how to break things and fail fast...without breaking things

64

Upload: magnolia-cms

Post on 10-May-2015

245 views

Category:

Software


0 download

DESCRIPTION

In this talk Mark Halvorson will explore how Atlassian, the creator of JIRA and Confluence, has used Magnolia CMS and other tools to operationalize A/B testing to continually optimize Atlassian's website.

TRANSCRIPT

Page 1: A/B Testing - How to Break Things and Fail Fast...without Breaking Things
Page 2: A/B Testing - How to Break Things and Fail Fast...without Breaking Things

M A R K H A LVO R S O N • H E A D O F I N T E R AC T I V E • AT L A S S I A N S O F T WA R E • @ h a l v 0 1 1 2

A/B TestingHOW TO BREAK THINGS AND FAIL FAST…

WITHOUT BREAKING THINGS

Page 3: A/B Testing - How to Break Things and Fail Fast...without Breaking Things

About Atlassian

• Products include: JIRA, Confluence, Bitbucket, HipChat…

• Founded in 2002

• Over 35,000 Customers

• Over 700 employees globally

• No sales people

!

!

!

W e h e l p g r e a t t e a m s b u i l d b e t t e r s o f t w a r e , t o g e t h e r.

Page 4: A/B Testing - How to Break Things and Fail Fast...without Breaking Things
Page 5: A/B Testing - How to Break Things and Fail Fast...without Breaking Things

Think big, act small, fail fast, learn rapidly.

" L E A N S O F T WA R E D E V E LO P M E N T ” BY M A RY A N D TO M P O P P E N D I E C K ( 2 0 0 3 )

”“

Page 6: A/B Testing - How to Break Things and Fail Fast...without Breaking Things

Atlassian A/B Testing is cross functional

• Email

• Product Management

• Customer Platform

• Customer Advocates

• Internal Systems

!

!

!

C e n t r a l G r o w t h H a c k i n g Te a m t o s u p p o r t :

• Leads

• Product Marketing

• Support

• Bitbucket

• HipChat

Page 7: A/B Testing - How to Break Things and Fail Fast...without Breaking Things
Page 8: A/B Testing - How to Break Things and Fail Fast...without Breaking Things
Page 9: A/B Testing - How to Break Things and Fail Fast...without Breaking Things

SCHEDULE TO AVOID CONFLICTS

BUILD OUT GOOD CANDIDATES

TRIAGE BASED ON POTENTIAL IMPACT

COLLECT GREAT IDEAS

Our Process

ANALYZE RESULTS

RUN EXPERIMENTS

Page 10: A/B Testing - How to Break Things and Fail Fast...without Breaking Things

SCHEDULE TO AVOID CONFLICTS

BUILD OUT GOOD CANDIDATES

TRIAGE BASED ON POTENTIAL IMPACT

COLLECT GREAT IDEAS

Our Process

ANALYZE RESULTS

RUN EXPERIMENTS

Page 11: A/B Testing - How to Break Things and Fail Fast...without Breaking Things

Where do experiment ideas come from?

Page 12: A/B Testing - How to Break Things and Fail Fast...without Breaking Things

Anywhere.

Page 13: A/B Testing - How to Break Things and Fail Fast...without Breaking Things

What are you are trying to improve?

Page 14: A/B Testing - How to Break Things and Fail Fast...without Breaking Things

Blimp Dashboard

Page 15: A/B Testing - How to Break Things and Fail Fast...without Breaking Things

JIRA – Ticket

Page 16: A/B Testing - How to Break Things and Fail Fast...without Breaking Things

• Pebbles

• Change in copy or CTA presentation

• Rocks

• New page (e.g. no Dev Tools family page)

• Boulders

• Simplified order form.

!

!

!

Types of tests

Page 17: A/B Testing - How to Break Things and Fail Fast...without Breaking Things

Examples

Page 18: A/B Testing - How to Break Things and Fail Fast...without Breaking Things

Reduce Friction

Page 19: A/B Testing - How to Break Things and Fail Fast...without Breaking Things

Increase Visibility

Page 20: A/B Testing - How to Break Things and Fail Fast...without Breaking Things

Test drive campaigns

Page 21: A/B Testing - How to Break Things and Fail Fast...without Breaking Things

Simplify

Page 22: A/B Testing - How to Break Things and Fail Fast...without Breaking Things

SCHEDULE TO AVOID CONFLICTS

BUILD OUT GOOD CANDIDATES

TRIAGE BASED ON POTENTIAL IMPACT

COLLECT GREAT IDEAS

Our Process

ANALYZE RESULTS

RUN EXPERIMENTS

Page 23: A/B Testing - How to Break Things and Fail Fast...without Breaking Things

What are you are trying to improve?

Page 24: A/B Testing - How to Break Things and Fail Fast...without Breaking Things

Blimp Dashboard

Page 25: A/B Testing - How to Break Things and Fail Fast...without Breaking Things

JIRA – Prioritized Backlog

Page 26: A/B Testing - How to Break Things and Fail Fast...without Breaking Things

JIRA – Ticket

Page 27: A/B Testing - How to Break Things and Fail Fast...without Breaking Things

JIRA – Ticket

Page 28: A/B Testing - How to Break Things and Fail Fast...without Breaking Things

Experiment Illuminati

Page 29: A/B Testing - How to Break Things and Fail Fast...without Breaking Things

SCHEDULE TO AVOID CONFLICTS

BUILD OUT GOOD CANDIDATES

TRIAGE BASED ON POTENTIAL IMPACT

COLLECT GREAT IDEAS

Our Process

ANALYZE RESULTS

RUN EXPERIMENTS

Page 30: A/B Testing - How to Break Things and Fail Fast...without Breaking Things

Optimizely – Build

Page 31: A/B Testing - How to Break Things and Fail Fast...without Breaking Things

Optimizely – Build

Page 32: A/B Testing - How to Break Things and Fail Fast...without Breaking Things

Prepare for results.

Page 33: A/B Testing - How to Break Things and Fail Fast...without Breaking Things

Confluence – Corresponding Test Page

Page 34: A/B Testing - How to Break Things and Fail Fast...without Breaking Things

Confluence – Corresponding Test Page

Page 35: A/B Testing - How to Break Things and Fail Fast...without Breaking Things

Confluence – Corresponding Test Page

Page 36: A/B Testing - How to Break Things and Fail Fast...without Breaking Things

Confluence – Corresponding Test Page

Page 37: A/B Testing - How to Break Things and Fail Fast...without Breaking Things

Confluence – Corresponding Test Page

Page 38: A/B Testing - How to Break Things and Fail Fast...without Breaking Things

Confluence – Corresponding Test Page

Page 39: A/B Testing - How to Break Things and Fail Fast...without Breaking Things

Confluence – Corresponding Test Page

Page 40: A/B Testing - How to Break Things and Fail Fast...without Breaking Things

Confluence – Corresponding Test Page

Page 41: A/B Testing - How to Break Things and Fail Fast...without Breaking Things

Confluence – Corresponding Test Page

Page 42: A/B Testing - How to Break Things and Fail Fast...without Breaking Things

Confluence – Corresponding Test Page

Page 43: A/B Testing - How to Break Things and Fail Fast...without Breaking Things

Confluence – Corresponding Test Page

Page 44: A/B Testing - How to Break Things and Fail Fast...without Breaking Things

Confluence – Corresponding Test Page

Page 45: A/B Testing - How to Break Things and Fail Fast...without Breaking Things

SCHEDULE TO AVOID CONFLICTS

BUILD OUT GOOD CANDIDATES

TRIAGE BASED ON POTENTIAL IMPACT

COLLECT GREAT IDEAS

Our Process

ANALYZE RESULTS

RUN EXPERIMENTS

Page 46: A/B Testing - How to Break Things and Fail Fast...without Breaking Things

JIRA – Prioritized Backlog

Page 47: A/B Testing - How to Break Things and Fail Fast...without Breaking Things

SCHEDULE TO AVOID CONFLICTS

BUILD OUT GOOD CANDIDATES

TRIAGE BASED ON POTENTIAL IMPACT

COLLECT GREAT IDEAS

Our Process

ANALYZE RESULTS

RUN EXPERIMENTS

Page 48: A/B Testing - How to Break Things and Fail Fast...without Breaking Things

Optimizely – Start Experiment

Page 49: A/B Testing - How to Break Things and Fail Fast...without Breaking Things

Optimizely – Project Code

Page 50: A/B Testing - How to Break Things and Fail Fast...without Breaking Things

Magnolia CMS – Include Optimizely Checkbox

Page 51: A/B Testing - How to Break Things and Fail Fast...without Breaking Things

Magnolia CMS – Include Optimizely Checkbox

Page 52: A/B Testing - How to Break Things and Fail Fast...without Breaking Things

Confluence – Corresponding Test Page

Page 53: A/B Testing - How to Break Things and Fail Fast...without Breaking Things

SCHEDULE TO AVOID CONFLICTS

BUILD OUT GOOD CANDIDATES

TRIAGE BASED ON POTENTIAL IMPACT

COLLECT GREAT IDEAS

Our Process

ANALYZE RESULTS

RUN EXPERIMENTS

Page 54: A/B Testing - How to Break Things and Fail Fast...without Breaking Things

Optimizely – Results

Page 55: A/B Testing - How to Break Things and Fail Fast...without Breaking Things

What are you are trying to improve?

Page 56: A/B Testing - How to Break Things and Fail Fast...without Breaking Things

Blimp Dashboard

Page 57: A/B Testing - How to Break Things and Fail Fast...without Breaking Things

Optimizely – Results

Page 58: A/B Testing - How to Break Things and Fail Fast...without Breaking Things

Confluence – Corresponding Test Page

Page 59: A/B Testing - How to Break Things and Fail Fast...without Breaking Things

Confluence – Results Archive

Page 60: A/B Testing - How to Break Things and Fail Fast...without Breaking Things

• Ideas are everywhere

• Everything affecting conversion is tested

• No traffic wasted

• Be aggressive

• Regular cadence

• Never run a test that you wouldn’t want to win

Our Test Philosophy

Page 61: A/B Testing - How to Break Things and Fail Fast...without Breaking Things

• Ideas are everywhere collect as many as you can, prioritize based on impact.

• Use ticket number everywhere so everyone is referring to the same test.

• Involve stakeholders and developers… and get approvals.

Key Take Aways

Page 62: A/B Testing - How to Break Things and Fail Fast...without Breaking Things
Page 63: A/B Testing - How to Break Things and Fail Fast...without Breaking Things

Thank you!

M A R K H A LVO R S O N • H E A D O F I N T E R AC T I V E • AT L A S S I A N S O F T WA R E • @ h a l v 0 1 1 2

Page 64: A/B Testing - How to Break Things and Fail Fast...without Breaking Things