craig sullivan - keynote speaker summary & final thoughts - conversion hotel 2015
TRANSCRIPT
![Page 1: Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel 2015](https://reader033.vdocument.in/reader033/viewer/2022042708/58ad7b171a28ab2a768b5bc3/html5/thumbnails/1.jpg)
The AB Testing Hype Cycle
Budapest - Netbooster – 11 Nov 2015 @OptimiseOrDie
It’s a Wrap! #CH2015
![Page 2: Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel 2015](https://reader033.vdocument.in/reader033/viewer/2022042708/58ad7b171a28ab2a768b5bc3/html5/thumbnails/2.jpg)
How was your trip to Texel, dear?
@OptimiseOrDie
![Page 3: Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel 2015](https://reader033.vdocument.in/reader033/viewer/2022042708/58ad7b171a28ab2a768b5bc3/html5/thumbnails/3.jpg)
• Genres – Facebook Poll • 3-4 Sets • Spotify Playlist • Outline set • Feedback • Mix live!
Music Optimisation
@OptimiseOrDie
![Page 4: Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel 2015](https://reader033.vdocument.in/reader033/viewer/2022042708/58ad7b171a28ab2a768b5bc3/html5/thumbnails/4.jpg)
![Page 5: Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel 2015](https://reader033.vdocument.in/reader033/viewer/2022042708/58ad7b171a28ab2a768b5bc3/html5/thumbnails/5.jpg)
Peep Laja
@OptimiseOrDie
• Peeeeep - A car sound • Pap - Common blog problem • Peahp - Texan pipe • Peppie - Food for small dogs • Payp - Micro payment system
![Page 6: Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel 2015](https://reader033.vdocument.in/reader033/viewer/2022042708/58ad7b171a28ab2a768b5bc3/html5/thumbnails/6.jpg)
World CRO Dude Dickens Character
PEP PIP
![Page 7: Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel 2015](https://reader033.vdocument.in/reader033/viewer/2022042708/58ad7b171a28ab2a768b5bc3/html5/thumbnails/7.jpg)
Peep Laja – Points to note
@OptimiseOrDie
6 key pillars • Heuristic analysis & walkthrough
• Session Replay / Mouse Tracking / Scroll / Click • Technical (compatibility & performance)
• Web Analytics • User Testing • Qualitative Surveys
![Page 8: Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel 2015](https://reader033.vdocument.in/reader033/viewer/2022042708/58ad7b171a28ab2a768b5bc3/html5/thumbnails/8.jpg)
“Stop copying your competitors. They may not know what the f*** they are doing either.” “We need better data, not more data” “Use a process – not 100 tactics. Discover what actually matters and lift heavy ass weights.”
Peep Laja – Quotes
@OptimiseOrDie
![Page 9: Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel 2015](https://reader033.vdocument.in/reader033/viewer/2022042708/58ad7b171a28ab2a768b5bc3/html5/thumbnails/9.jpg)
![Page 10: Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel 2015](https://reader033.vdocument.in/reader033/viewer/2022042708/58ad7b171a28ab2a768b5bc3/html5/thumbnails/10.jpg)
Chris Stucchio
@OptimiseOrDie
![Page 11: Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel 2015](https://reader033.vdocument.in/reader033/viewer/2022042708/58ad7b171a28ab2a768b5bc3/html5/thumbnails/11.jpg)
@OptimiseOrDie
STATS FOR A/B
TESTING
![Page 12: Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel 2015](https://reader033.vdocument.in/reader033/viewer/2022042708/58ad7b171a28ab2a768b5bc3/html5/thumbnails/12.jpg)
Chris Stucchio – Quotes
@OptimiseOrDie
“Misinterpretations of Significance.” “Bayesian statisticians can never agree to disagree.” “CRO is more like trading – the goal is to get more conversions.”
![Page 13: Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel 2015](https://reader033.vdocument.in/reader033/viewer/2022042708/58ad7b171a28ab2a768b5bc3/html5/thumbnails/13.jpg)
• The unconference session feedback! • This stuff isn’t easy • It’s about decision making though • Do we keep control, deploy challenger or test
further? • We can’t educate the world easily but we can build
tools and process that avoids or minimises poor decision making.
Chris Stucchio – Points to note
@OptimiseOrDie
![Page 14: Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel 2015](https://reader033.vdocument.in/reader033/viewer/2022042708/58ad7b171a28ab2a768b5bc3/html5/thumbnails/14.jpg)
![Page 15: Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel 2015](https://reader033.vdocument.in/reader033/viewer/2022042708/58ad7b171a28ab2a768b5bc3/html5/thumbnails/15.jpg)
@OptimiseOrDie
4. If you have 4 hours
PLUS • Snap interviews (Sales, Customer Services, Tech
Support) • Run a quick poll or survey (See my tools slides)
Less Bullshit!
![Page 16: Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel 2015](https://reader033.vdocument.in/reader033/viewer/2022042708/58ad7b171a28ab2a768b5bc3/html5/thumbnails/16.jpg)
Oli Gardner – Quotes
@OptimiseOrDie
“Shitting marketing pisses me off…” “Delight should be the default, not the exception.” “Every page has one primary goal.” “Personalisation or targeting is still a hypothesis until it’s tested”
![Page 17: Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel 2015](https://reader033.vdocument.in/reader033/viewer/2022042708/58ad7b171a28ab2a768b5bc3/html5/thumbnails/17.jpg)
• NSAMCWADLP – Never Start a Marketing Campaign Without a Dedicated Landing Page
• 4 corners – Copy, Design, Interaction, Psychology • A multi-faceted approach is required
Oli Gardner – Points to note
@OptimiseOrDie
![Page 18: Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel 2015](https://reader033.vdocument.in/reader033/viewer/2022042708/58ad7b171a28ab2a768b5bc3/html5/thumbnails/18.jpg)
“ Affordance is the ability of a user to perceive that some action is possible”
– like a door handle or push plate!
Affordance isn’t about stuff that can’t be inferred (for example, hover states, feedback messages, hidden interactions)
http://www.jnd.org/dn.mss/affordances_and.html
Affordance
@OptimiseOrDie
![Page 19: Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel 2015](https://reader033.vdocument.in/reader033/viewer/2022042708/58ad7b171a28ab2a768b5bc3/html5/thumbnails/19.jpg)
![Page 20: Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel 2015](https://reader033.vdocument.in/reader033/viewer/2022042708/58ad7b171a28ab2a768b5bc3/html5/thumbnails/20.jpg)
“Best Practices don’t work” “The best optimisation champions integrate both yin and yang into their work.” “You need a qual and quant mindset” “Don’t start with tactics. Process is critical but framework thinking is the best way.” “Optimise the Optimisation” “It’s about understanding buyers through testing”
Chris Goward – Quotes
@OptimiseOrDie
![Page 21: Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel 2015](https://reader033.vdocument.in/reader033/viewer/2022042708/58ad7b171a28ab2a768b5bc3/html5/thumbnails/21.jpg)
• Marketing + Scientific process – CRO • UX, Analytics, Persuasion Principles,
Business Context and your Prior test results – are all part of your testing framework
• NPS is vital (Lovefilm example) • Measure NPS service metrics over time! • Continuous Improvement – not SLEDD*
*Single Large Expensive Doomed Development
Chris Goward – Points to note
@OptimiseOrDie
![Page 22: Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel 2015](https://reader033.vdocument.in/reader033/viewer/2022042708/58ad7b171a28ab2a768b5bc3/html5/thumbnails/22.jpg)
![Page 23: Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel 2015](https://reader033.vdocument.in/reader033/viewer/2022042708/58ad7b171a28ab2a768b5bc3/html5/thumbnails/23.jpg)
Brent Dykes
@OptimiseOrDie
![Page 24: Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel 2015](https://reader033.vdocument.in/reader033/viewer/2022042708/58ad7b171a28ab2a768b5bc3/html5/thumbnails/24.jpg)
Brent Dykes
@OptimiseOrDie
![Page 25: Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel 2015](https://reader033.vdocument.in/reader033/viewer/2022042708/58ad7b171a28ab2a768b5bc3/html5/thumbnails/25.jpg)
@OptimiseOrDie
![Page 26: Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel 2015](https://reader033.vdocument.in/reader033/viewer/2022042708/58ad7b171a28ab2a768b5bc3/html5/thumbnails/26.jpg)
• “Stories beat Statistics” • Storytelling is vital to our work “We saw this, so checked that and found this, which taught us that”
• We use persuasion on customers but not internally within our company?
• “Show their digital journey”
Brent Dykes – Points to note
@OptimiseOrDie
![Page 27: Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel 2015](https://reader033.vdocument.in/reader033/viewer/2022042708/58ad7b171a28ab2a768b5bc3/html5/thumbnails/27.jpg)
![Page 28: Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel 2015](https://reader033.vdocument.in/reader033/viewer/2022042708/58ad7b171a28ab2a768b5bc3/html5/thumbnails/28.jpg)
Jorden Lentze
@OptimiseOrDie
bit.ly/urinalux
![Page 29: Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel 2015](https://reader033.vdocument.in/reader033/viewer/2022042708/58ad7b171a28ab2a768b5bc3/html5/thumbnails/29.jpg)
Jorden Lentze
@OptimiseOrDie
![Page 30: Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel 2015](https://reader033.vdocument.in/reader033/viewer/2022042708/58ad7b171a28ab2a768b5bc3/html5/thumbnails/30.jpg)
“Use the collective brain” “The Mixing of ideas and unexpected encounters through unplanned meetings” “The most important thing is the insights that you learn”
Jorden Lentze – Quotes
@OptimiseOrDie
![Page 31: Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel 2015](https://reader033.vdocument.in/reader033/viewer/2022042708/58ad7b171a28ab2a768b5bc3/html5/thumbnails/31.jpg)
• Note the core team – CRO, Dev, Psychologist, Product, UX, Analyst, Copywriter
• Explore the problem domain, focus on the problem at hand, incubate, get insights, follow through!
• LISTEN to customers, MAKE incremental changes AND big ideas, AB Test and Measure impact
• Sliders can work but most implementations suck!
Jorden Lentze – Points to note
@OptimiseOrDie
![Page 32: Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel 2015](https://reader033.vdocument.in/reader033/viewer/2022042708/58ad7b171a28ab2a768b5bc3/html5/thumbnails/32.jpg)
Marketing?
Variation
Heritability of good ideas Selection
based on death
![Page 33: Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel 2015](https://reader033.vdocument.in/reader033/viewer/2022042708/58ad7b171a28ab2a768b5bc3/html5/thumbnails/33.jpg)
![Page 34: Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel 2015](https://reader033.vdocument.in/reader033/viewer/2022042708/58ad7b171a28ab2a768b5bc3/html5/thumbnails/34.jpg)
![Page 35: Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel 2015](https://reader033.vdocument.in/reader033/viewer/2022042708/58ad7b171a28ab2a768b5bc3/html5/thumbnails/35.jpg)
Harry Brignull
@OptimiseOrDie
![Page 36: Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel 2015](https://reader033.vdocument.in/reader033/viewer/2022042708/58ad7b171a28ab2a768b5bc3/html5/thumbnails/36.jpg)
• Ethics are a vital and ongoing debate • It’s all about intent for me • Read these articles – very good
www.measuringu.com/blog/ethical-treatment.php
www.twitter.com/search?q=optimiseordie%20ethics&src=typd
Harry Brignull – Points to note
@OptimiseOrDie
![Page 37: Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel 2015](https://reader033.vdocument.in/reader033/viewer/2022042708/58ad7b171a28ab2a768b5bc3/html5/thumbnails/37.jpg)
![Page 38: Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel 2015](https://reader033.vdocument.in/reader033/viewer/2022042708/58ad7b171a28ab2a768b5bc3/html5/thumbnails/38.jpg)
Talia Wolf
@OptimiseOrDie
![Page 39: Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel 2015](https://reader033.vdocument.in/reader033/viewer/2022042708/58ad7b171a28ab2a768b5bc3/html5/thumbnails/39.jpg)
“We like to think of ourselves as rational people”
“We’re basically buying better versions of ourselves”
“Don’t talk about yourself – make it about them”
“It’s about testing concepts, not elements or tactics. You can’t learn anything from these kind of tests”
“It’s about the strategy and knowledge you acquire”
Talia Wolf – Quotes
@OptimiseOrDie
![Page 40: Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel 2015](https://reader033.vdocument.in/reader033/viewer/2022042708/58ad7b171a28ab2a768b5bc3/html5/thumbnails/40.jpg)
The 3 pillars: • “Make it about the customer” • “Show it, don’t just say it” • “Test concepts – not elements or tactics”]
Cycle: Research, Strategy, Hypothesis, Prioritisation, Design, Align team, Launch, QA, Run, Analyse
Talia Wolf – Points to note
@OptimiseOrDie
![Page 41: Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel 2015](https://reader033.vdocument.in/reader033/viewer/2022042708/58ad7b171a28ab2a768b5bc3/html5/thumbnails/41.jpg)
The AB Testing Hype Cycle
Budapest - Netbooster – 11 Nov 2015 @OptimiseOrDie
It’s a Wrap! #CH2015
![Page 42: Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel 2015](https://reader033.vdocument.in/reader033/viewer/2022042708/58ad7b171a28ab2a768b5bc3/html5/thumbnails/42.jpg)
What About the Organisation?
@OptimiseOrDie
![Page 43: Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel 2015](https://reader033.vdocument.in/reader033/viewer/2022042708/58ad7b171a28ab2a768b5bc3/html5/thumbnails/43.jpg)
@OptimiseOrDie
Burn Down the Silos
• Non agile, non iterative design
• Silos work on product separately
• No ‘One Team’ per product/theme
• Large teams, unwieldy coordination
• Pass the product around
• More PMs and BAs than a conference
• Endless sucking signoff
• AB testing done the same way!
![Page 44: Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel 2015](https://reader033.vdocument.in/reader033/viewer/2022042708/58ad7b171a28ab2a768b5bc3/html5/thumbnails/44.jpg)
@OptimiseOrDie
FT, Booking, Skyscanner, ABN, Expedia + more
• Small teams (6-15) with direct access to publish • Ability to set and get metrics data directly • Tools, Autonomy, Lack of interference
• No Project Managers or Business Analysts
• Business defines ‘outcomes’ – teams deliver • No long signoff chain • No pesky meddling fools
• 18 Month projects over budget?
![Page 45: Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel 2015](https://reader033.vdocument.in/reader033/viewer/2022042708/58ad7b171a28ab2a768b5bc3/html5/thumbnails/45.jpg)
@OptimiseOrDie
FT Example
• 100s of releases a day!
• Iterative MVP approach
• Launch as alpha, beta, pilot, phased rollout
• Like getting in a shower
• Read more at labs.ft.com
![Page 46: Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel 2015](https://reader033.vdocument.in/reader033/viewer/2022042708/58ad7b171a28ab2a768b5bc3/html5/thumbnails/46.jpg)
CRO
46
![Page 47: Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel 2015](https://reader033.vdocument.in/reader033/viewer/2022042708/58ad7b171a28ab2a768b5bc3/html5/thumbnails/47.jpg)
@OptimiseOrDie
10. Positive Attributes
• Rapid, Iterative, User Centred & Agile Design. No Silos.
• Small empowered autonomous teams
• Polymaths and Overlap
• Toolkit & Analytics investment
• Persuasive copywriting & Psychology
• Great Testing & Optimisation Tools
![Page 48: Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel 2015](https://reader033.vdocument.in/reader033/viewer/2022042708/58ad7b171a28ab2a768b5bc3/html5/thumbnails/48.jpg)
@OptimiseOrDie
• Agile, Lean, Iterative x silo teams • Ability to get and set metrics • Autonomy, Control, Velocity • Iterative MVP approach • Work on outcomes, not features
! Burn Down the Silos!
![Page 49: Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel 2015](https://reader033.vdocument.in/reader033/viewer/2022042708/58ad7b171a28ab2a768b5bc3/html5/thumbnails/49.jpg)
49
Time
ROI
![Page 50: Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel 2015](https://reader033.vdocument.in/reader033/viewer/2022042708/58ad7b171a28ab2a768b5bc3/html5/thumbnails/50.jpg)
50
![Page 51: Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel 2015](https://reader033.vdocument.in/reader033/viewer/2022042708/58ad7b171a28ab2a768b5bc3/html5/thumbnails/51.jpg)
Summary Grid
@OptimiseOrDie
Do Your Research Prioritise Hypothesise
• Heuristic analysis • Usability testing • Heatmap tools • Session replay • Web Analytics • Qualitative surveys • Polls • Chat & Text analysis • Interviews • Device compatibility • Performance
• Use TIR or PIE • Prioritise around
opportunity vs. cost • Estimate political
difficulty or friction as well as resource costs
• There is no best practice on prioritisation
• Continually re-prioritise against new discoveries
• Use the collective brain • Assemble and circulate
your research • Set clear goals and
have a process for sketching
• Sketch the Hypothesis • Build W/F or mockup • Set ONE primary
(possibly one secondary) success KPI
![Page 52: Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel 2015](https://reader033.vdocument.in/reader033/viewer/2022042708/58ad7b171a28ab2a768b5bc3/html5/thumbnails/52.jpg)
Summary Grid
@OptimiseOrDie
Pre Flight Checks Run Analyse & Report
• QA testing • Device & Browser
compatibility • Analytics tracking • Ecommerce tracking • Launch
• Set your test length with a calculator
• Run it, stop it, analyse it • Don’t run over time
unless your sample is smaller than expected
• Learn about the statistics
• Watch tests closely!
• Wash up analysis • Share findings widely • Don’t share useless
stuff • What is useful across
the organisation? • Store in knowledgebase • Intellectual Property • Use as training!
![Page 53: Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel 2015](https://reader033.vdocument.in/reader033/viewer/2022042708/58ad7b171a28ab2a768b5bc3/html5/thumbnails/53.jpg)
DISCOVERY CYCLE
@OptimiseOrDie
Discover Ideas
Priori/se
Build & QA
Test or Live Measure
Data
Learn
![Page 54: Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel 2015](https://reader033.vdocument.in/reader033/viewer/2022042708/58ad7b171a28ab2a768b5bc3/html5/thumbnails/54.jpg)
TEST CYCLE
@OptimiseOrDie
Hypothesis Design
Sketch
Wire Frame
Mockup / Prototype
Signoff (mul/ple)
Build
QA
SoH and Hard Launch
Analyse & Publish
Learn
![Page 55: Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel 2015](https://reader033.vdocument.in/reader033/viewer/2022042708/58ad7b171a28ab2a768b5bc3/html5/thumbnails/55.jpg)
What did I learn personally?
@OptimiseOrDie
• That best practice is emerging • It’s about how you can do this properly! • Methodology & Frameworks – not tactics • Shared mindset across CRO practitioners • Stories beat statistics, Data beats opinion • I always have lots to learn • “Hold strong opinions, but loosely”
![Page 56: Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel 2015](https://reader033.vdocument.in/reader033/viewer/2022042708/58ad7b171a28ab2a768b5bc3/html5/thumbnails/56.jpg)
![Page 57: Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel 2015](https://reader033.vdocument.in/reader033/viewer/2022042708/58ad7b171a28ab2a768b5bc3/html5/thumbnails/57.jpg)
Go Forth and Optimise!
Email me [email protected]
Slides Will be mailed to you!
Linkedin linkd.in/pvrg14