a/b testing - how to break things and fail fast...without breaking things

M A R K H A LVO R S O N • H E A D O F I N T E R AC T I V E • AT L A S S I A N S O F T WA R E • @ h a l v 0 1 1 2

A/B TestingHOW TO BREAK THINGS AND FAIL FAST…

WITHOUT BREAKING THINGS

About Atlassian

• Products include: JIRA, Confluence, Bitbucket, HipChat…

• Founded in 2002

• Over 35,000 Customers

• Over 700 employees globally

• No sales people

!

!

!

W e h e l p g r e a t t e a m s b u i l d b e t t e r s o f t w a r e , t o g e t h e r.

Think big, act small, fail fast, learn rapidly.

" L E A N S O F T WA R E D E V E LO P M E N T ” BY M A RY A N D TO M P O P P E N D I E C K ( 2 0 0 3 )

”“

Atlassian A/B Testing is cross functional

• Email

• Product Management

• Customer Platform

• Customer Advocates

• Internal Systems

!

!

!

C e n t r a l G r o w t h H a c k i n g Te a m t o s u p p o r t :

• Leads

• Product Marketing

• Support

• Bitbucket

• HipChat

SCHEDULE TO AVOID CONFLICTS

BUILD OUT GOOD CANDIDATES

TRIAGE BASED ON POTENTIAL IMPACT

COLLECT GREAT IDEAS

Our Process

ANALYZE RESULTS

RUN EXPERIMENTS

SCHEDULE TO AVOID CONFLICTS

BUILD OUT GOOD CANDIDATES

TRIAGE BASED ON POTENTIAL IMPACT

COLLECT GREAT IDEAS

Our Process

ANALYZE RESULTS

RUN EXPERIMENTS

Where do experiment ideas come from?

Anywhere.

What are you are trying to improve?

Blimp Dashboard

JIRA – Ticket

• Pebbles

• Change in copy or CTA presentation

• Rocks

• New page (e.g. no Dev Tools family page)

• Boulders

• Simplified order form.

!

!

!

Types of tests

Examples

Reduce Friction

Increase Visibility

Test drive campaigns

Simplify

SCHEDULE TO AVOID CONFLICTS

BUILD OUT GOOD CANDIDATES

TRIAGE BASED ON POTENTIAL IMPACT

COLLECT GREAT IDEAS

Our Process

ANALYZE RESULTS

RUN EXPERIMENTS

What are you are trying to improve?

Blimp Dashboard

JIRA – Prioritized Backlog

JIRA – Ticket

JIRA – Ticket

Experiment Illuminati

SCHEDULE TO AVOID CONFLICTS

BUILD OUT GOOD CANDIDATES

TRIAGE BASED ON POTENTIAL IMPACT

COLLECT GREAT IDEAS

Our Process

ANALYZE RESULTS

RUN EXPERIMENTS

Optimizely – Build

Optimizely – Build

Prepare for results.

Confluence – Corresponding Test Page

Confluence – Corresponding Test Page

Confluence – Corresponding Test Page

Confluence – Corresponding Test Page

Confluence – Corresponding Test Page

Confluence – Corresponding Test Page

Confluence – Corresponding Test Page

Confluence – Corresponding Test Page

Confluence – Corresponding Test Page

Confluence – Corresponding Test Page

Confluence – Corresponding Test Page

Confluence – Corresponding Test Page

SCHEDULE TO AVOID CONFLICTS

BUILD OUT GOOD CANDIDATES

TRIAGE BASED ON POTENTIAL IMPACT

COLLECT GREAT IDEAS

Our Process

ANALYZE RESULTS

RUN EXPERIMENTS

JIRA – Prioritized Backlog

SCHEDULE TO AVOID CONFLICTS

BUILD OUT GOOD CANDIDATES

TRIAGE BASED ON POTENTIAL IMPACT

COLLECT GREAT IDEAS

Our Process

ANALYZE RESULTS

RUN EXPERIMENTS

Optimizely – Start Experiment

Optimizely – Project Code

Magnolia CMS – Include Optimizely Checkbox

Magnolia CMS – Include Optimizely Checkbox

Confluence – Corresponding Test Page

SCHEDULE TO AVOID CONFLICTS

BUILD OUT GOOD CANDIDATES

TRIAGE BASED ON POTENTIAL IMPACT

COLLECT GREAT IDEAS

Our Process

ANALYZE RESULTS

RUN EXPERIMENTS

Optimizely – Results

What are you are trying to improve?

Blimp Dashboard

Optimizely – Results

Confluence – Corresponding Test Page

Confluence – Results Archive

• Ideas are everywhere

• Everything affecting conversion is tested

• No traffic wasted

• Be aggressive

• Regular cadence

• Never run a test that you wouldn’t want to win

Our Test Philosophy

• Ideas are everywhere collect as many as you can, prioritize based on impact.

• Use ticket number everywhere so everyone is referring to the same test.

• Involve stakeholders and developers… and get approvals.

Key Take Aways

Thank you!

M A R K H A LVO R S O N • H E A D O F I N T E R AC T I V E • AT L A S S I A N S O F T WA R E • @ h a l v 0 1 1 2