a/b testing - how to break things and fail fast... without breaking things
DESCRIPTION
Everyone wants to optimize their web property, especially if that web property is their only sales channel. Lots of tools like Optimizely, allow marketers to go in and run tests on their own in an effort to "break things and fail fast", then take those learnings to make the site better. Unfortunately, the risk of severely breaking things can be significant, and it isn't the marketer who gets the call in the middle of the night when something is broken. In this talk Mark will explore how Atlassian, the creator of JIRA and Confluence, has used Magnolia CMS and other tools to operationalize A/B testing to continually optimize Atlassian’s website. Using this process we can balance failing fast with keeping things running smoothly.TRANSCRIPT
M A R K H A LVO R S O N • H E A D O F I N T E R AC T I V E • AT L A S S I A N S O F T WA R E • @ h a l v 0 1 1 2
A/B TestingHOW TO BREAK THINGS AND FAIL FAST…
WITHOUT BREAKING THINGS
About Atlassian
• Products include: JIRA, Confluence, Bitbucket, HipChat…
• Founded in 2002
• Over 35,000 Customers
• Over 700 employees globally
• No sales people
!
!
!
W e h e l p g r e a t t e a m s b u i l d b e t t e r s o f t w a r e , t o g e t h e r.
Think big, act small, fail fast, learn rapidly.
" L E A N S O F T WA R E D E V E LO P M E N T ” BY M A RY A N D TO M P O P P E N D I E C K ( 2 0 0 3 )
”“
Atlassian A/B Testing is cross functional
• Product Management
• Customer Platform
• Customer Advocates
• Internal Systems
!
!
!
C e n t r a l G r o w t h H a c k i n g Te a m t o s u p p o r t :
• Leads
• Product Marketing
• Support
• Bitbucket
• HipChat
SCHEDULE TO AVOID CONFLICTS
BUILD OUT GOOD CANDIDATES
TRIAGE BASED ON POTENTIAL IMPACT
COLLECT GREAT IDEAS
Our Process
ANALYZE RESULTS
RUN EXPERIMENTS
SCHEDULE TO AVOID CONFLICTS
BUILD OUT GOOD CANDIDATES
TRIAGE BASED ON POTENTIAL IMPACT
COLLECT GREAT IDEAS
Our Process
ANALYZE RESULTS
RUN EXPERIMENTS
Where do experiment ideas come from?
Anywhere.
What are you are trying to improve?
Blimp Dashboard
JIRA – Ticket
• Pebbles
• Change in copy or CTA presentation
• Rocks
• New page (e.g. no Dev Tools family page)
• Boulders
• Simplified order form.
!
!
!
Types of tests
Examples
Reduce Friction
Increase Visibility
Test drive campaigns
Simplify
SCHEDULE TO AVOID CONFLICTS
BUILD OUT GOOD CANDIDATES
TRIAGE BASED ON POTENTIAL IMPACT
COLLECT GREAT IDEAS
Our Process
ANALYZE RESULTS
RUN EXPERIMENTS
What are you are trying to improve?
Blimp Dashboard
JIRA – Prioritized Backlog
JIRA – Ticket
JIRA – Ticket
Experiment Illuminati
SCHEDULE TO AVOID CONFLICTS
BUILD OUT GOOD CANDIDATES
TRIAGE BASED ON POTENTIAL IMPACT
COLLECT GREAT IDEAS
Our Process
ANALYZE RESULTS
RUN EXPERIMENTS
Optimizely – Build
Optimizely – Build
Prepare for results.
Confluence – Corresponding Test Page
Confluence – Corresponding Test Page
Confluence – Corresponding Test Page
Confluence – Corresponding Test Page
Confluence – Corresponding Test Page
Confluence – Corresponding Test Page
Confluence – Corresponding Test Page
Confluence – Corresponding Test Page
Confluence – Corresponding Test Page
Confluence – Corresponding Test Page
Confluence – Corresponding Test Page
Confluence – Corresponding Test Page
SCHEDULE TO AVOID CONFLICTS
BUILD OUT GOOD CANDIDATES
TRIAGE BASED ON POTENTIAL IMPACT
COLLECT GREAT IDEAS
Our Process
ANALYZE RESULTS
RUN EXPERIMENTS
JIRA – Prioritized Backlog
SCHEDULE TO AVOID CONFLICTS
BUILD OUT GOOD CANDIDATES
TRIAGE BASED ON POTENTIAL IMPACT
COLLECT GREAT IDEAS
Our Process
ANALYZE RESULTS
RUN EXPERIMENTS
Optimizely – Start Experiment
Optimizely – Project Code
Magnolia CMS – Include Optimizely Checkbox
Magnolia CMS – Include Optimizely Checkbox
Confluence – Corresponding Test Page
SCHEDULE TO AVOID CONFLICTS
BUILD OUT GOOD CANDIDATES
TRIAGE BASED ON POTENTIAL IMPACT
COLLECT GREAT IDEAS
Our Process
ANALYZE RESULTS
RUN EXPERIMENTS
Optimizely – Results
What are you are trying to improve?
Blimp Dashboard
Optimizely – Results
Confluence – Corresponding Test Page
Confluence – Results Archive
• Ideas are everywhere
• Everything affecting conversion is tested
• No traffic wasted
• Be aggressive
• Regular cadence
• Never run a test that you wouldn’t want to win
Our Test Philosophy
• Ideas are everywhere collect as many as you can, prioritize based on impact.
• Use ticket number everywhere so everyone is referring to the same test.
• Involve stakeholders and developers… and get approvals.
Key Take Aways
Thank you!
M A R K H A LVO R S O N • H E A D O F I N T E R AC T I V E • AT L A S S I A N S O F T WA R E • @ h a l v 0 1 1 2