conversion jam
TRANSCRIPT
How to run 200 tests a year & what do you learn?
Martijn Scheijbeler / Director of Marketing / [email protected] / @MartijnSch
Know why, you test and how you scale what matters:
@MartijnSch
WHAT YOU GET OUT OF THIS?
We’ll be sharing what tools + process we have at The Next Web to get started!
“So what would we have
to do to grow x10?”
Boris (CEO)
“I have no idea what I’m doing”
@MartijnSch / #ConversionJam
Testing Figures Data from 2015
2,5 CRO Team in 2015
38.000.000 Sessions with Tests
0 CRO Team in 2014
5-7 Tests per week
Priceless Amounts of fun
35% Traffic v.s. Tests
@MartijnSch
@MartijnSch
Our roadmap for getting to 200 tests a yearJan ‘15
Let’s do more testing!After running 20-30 tests in 2014 we knew we had to scale up.Resources were low, but with the addition of a web analist we could grow.
Apr ‘15
Hitting some limitsGoogle Analytics Content Experiments (JS) was our first tool to use. But soon we hit the limits of the tool.
May ‘15
Setting the visionAfter running 40 tests by May we needed a beter process that could scale and that aligned with our testing vision.
Dec ‘15
Reaching XXX testsWith the testing vision and updated testing program we were able to run XXX tests in total in 2015.
3 Months
1 Month
7 Months
Quantity v.s. Quality?
@MartijnSch
But why would you like to run at high velocity? Why not go for quality?
#ConversionJam
“If you double the number of experiments you’re going to double your inventiveness”
- Jeff Bezos
@MartijnSch
The opportunity is 350-400 tests a
year. As ROI per winning test is
lower we need to run a lot to have
a positive return.
Efficiency <> Costs
@MartijnSch / #ConversionJam
We needed to learn,Knowledge building?
@MartijnSch
You got to love working with people who are like minded.
#ConversionJam
The more we test the more
we innovate and the higher
inventiveness.
Find theperfect process
@MartijnSch / #ConversionJam
The opportunity is 350-400 tests a
year. As ROI per winning test is
lower we need to run a lot to have
a positive return.
Know your metrics?
@MartijnSch / #ConversionJam
@MartijnSch
1
2
Unique UsersGrowing our unique user base willincrease the potential to reach Generation T.
2
3 3
Our metrics/KPIs for success;
User Engagement: PV/SessionA higher engagement with our user base will increase the number of pageviews per session.
User Retention: Returning %Increasing the percentage or returning users will increase our brand awareness.
Newsletter SignupsTo get a relationship with our users we’d like to have their email address for the newsletter.
Social SharesMore shares will influence our reach amongst new and returning users.
The opportunity is 350-400 tests a
year. As ROI per winning test is
lower we need to run a lot to have
a positive return.
Do you have a swimming pool?
@MartijnSch / #ConversionJam
The opportunity is 350-400 tests a
year. As ROI per winning test is
lower we need to run a lot to have
a positive return.
Segment Users
@MartijnSch / #ConversionJam
Google Tag Manager (GTM) is making our lives easier + scalable.
What tools we use
@MartijnSch / #ConversionJam
So we need to improve most of our communication around testing.
Communicate more
@MartijnSch / #ConversionJam
We didn’t move to any ‘real’
CRO tool but decided to
iterate on our own: GTM <> JS.
GTM
Download:
Github.com/TheNextWeb/CRO
@MartijnSch / #ConversionJam
Building our own solutions for
saving testing documentation
+ analysis of tests.
Innovate
@MartijnSch / #ConversionJam
We moved most of our
communication towards
Slack. Making the teams more
involved in the whole process.
Always communicate
@MartijnSch / #ConversionJam
We hired two front-end
developers, to make sure in
the long-run we can test
bigger things.
Front-end Dev
@MartijnSch / #ConversionJam
5-7 Tests per week
2-4 Tests per week
30-40 Tests per year
20% More efficient
@MartijnSch
Improvement What does progress look like?
Who’s able to deploy winning
variants within minutes from their
codebase?
Continous Improvement
@MartijnSch / #ConversionJam
This will make segmentation +
automation even easier. In addition
it will decrease the flickering effect.
Back-end Testing
@MartijnSch / #ConversionJam
Testing small doesn’t work, so we
need to grow even bigger. On
average we grow traffic with around
10-20% yearly. At the same rate we
can increase our testing program.
Big > Biggest
@MartijnSch / #ConversionJam
TL;DR Process
1. Have the right company vision 2. Have the right team structure 3. Have access + support from C level 4. Create partnerships with like minded 5. You’ll have trouble as well, nobody’s perfect 6. Don’t be afraid to invent the wheel again 7. Always focus on innovation
@MartijnSch #ConversionJam
• Variant increase clicks with 16% • Had a significance + power level of > 99% • We ran the test for 1x 7+ days • Targetting over 200.000 users
Winner: Variant
Original:
Variant
Engagement
@MartijnSch #ConversionJam
More engagement increase loyalty which provides advertising revenue
Engagement is important,
focus on getting more clicks
on the related stories in the
sidebar.
Rankings!
• Variant increase clicks with 36% • Had a significance + power level of > 99% • We ran the test for 7+ days • Targetting over 300.000 users
Winner: Variant 1
Original: Variant:
Engagement is important,
focus on getting more clicks
on the related stories below
the article.
Related Stories
• Variant increase clicks with 36% • Had a significance + power level of > 99% • We ran the test for 2x 14+ days • Targetting over 500.000 users
Winner: Variant
Original:Variant
So what if we remove the
whole sidebar, wouldn’t people
be more focused on the stuff
that is out there?
Big, bigger
• Variant decreased clicks with 17% + 14% • Had a significance + power level of > 99% • We ran the test for 14+ days • Targetting over 420.000 users
Winner: Original
“Why didn’t we run 200 tests?”
“What went wrong?”
“Do I need to start crying?”
Boris (CEO)
@MartijnSch / #ConversionJam
“Well, if this guy wouldn’t have stuck
his fingers into a lawn mower, YES!”
Missing out on ±20 tests in 2015.
The Lawn Mower
@MartijnSch / #ConversionJam
Take Aways:
1. Focus on running X tests a year
2. Improve the process
3. Have the right (free) tools 4. Have the right people + support 5. Document/Communicate everything
6. Know what the future looks like
7. Have fun!
@MartijnSch #ConversionJam