The Finishing Line- when is a Multivariate Test finished?

By Joe Doveton

Oban Digital

When Is A Test Finished?

• One of the most commonly asked questions is “when is an MVT experiment or AB test finished?”

• Is it at 30 days...? 100 conversions...? 10,000 visitors...?

• The short answer is... it depends

Known and Unknown VariablesA test is made up of known and unknown

variables...

Things you know:– Likely visitors to

test page(s)

– Likely conversion rate for the original (control)

–Numbers of design variations in the test

Things you don’t:

– The improvement (or deterioration) in the conversion rate generated by the challenger designs

Understanding Test Data

Conversion rate for the original is 51.68%

The leading challenger is Combination 7with an 80% probability of beatingthe original

This is a 2.40% uplift in Conversion Rate vsthe original

Margin of erroris important because it effectively tells you if you have a big enough

sample. In an ideal world one would wait until the margin of error

was 0%

Margin of error

• In this test there is 0.7% margin of error in the sample

• This means that the original is converting at 51.68% plus or minus 0.7%

• The best case scenario for the original is therefore to add 0.7% to the conversion rate making 52.38%

• The worst case scenario for the test leader is to subtract 0.7% from that conversion rate, making a CR of 52.22%

• This means the test is not quite statistically valid-we could run the test a little longer to get a big enough sample

• Despite this, there is enough evidence to apply Combination 7 as a test winner- 80% confidence and 2.40% uplift over the original plus £10k in increased revenue

In some tests it is not practical to wait until you have 0% margin of

error, especially on low traffic pages.

Sometimes you just have to make a call. Here are some tips to help you decide when your test is finished.

Fit test type to visitor profile

• With low traffic pages make small changes and test via AB testing. You will get a quicker result and you can iterate

• With higher traffic pages you can use MVT. MVT has the advantage of allowing you to test many variations at once but still isolate the impact of individual on-page elements

Brainstorm

• A test is only as good as the ideas that go into it

• Manually go through funnels

• Use analytics

• Analyse competitors

• Group and ungroup elements

• Prioritise tests- do the low hanging fruit first

• Set yourself cut off points

Test goals

Match your test goals as closely as possible with your regular goals set up in analytics

Test goals should be consistent with the elements that you want to change- for example if you want to change CTR changing the footer navigation isn’t likely to contribute much to this

Keep Things Manageable

• If using MVT keep your test down to a manageable number of combinations

• Test only a few elements together concurrently

• Each time you add in a new element, you double the length of time it takes to get a statistically valid result (same with segments)

• No-one wants to read a test with 128 possible test combinations (sorry Mr. Taguchi)

Check The “Elements” View

• With MVT, one can check both the interaction between elements (the ‘combination’) and the individual elements in isolation

• Often, the elements view will give you a different view of the data

• In a test with no clear winners, there may be evidence of specific elements driving conversions that allows you to plan a follow-on experiment

Confidence Level

• 100% confidence results in test are rare. Very rare- even in AB tests.

• Set yourself an acceptable confidence level and go with it

• Anything over 70% is pretty good- 80% and above is exceptional

• PS this also works in reverse with losing tests

• Be prepared to call winners and losers

• Accept that inconclusive results are part of the game

Be Prepared To Iterate

• So your first test result was inconclusive. Great! Why great? It gives you an opportunity to re-evaluate the design

• You start to gain insight into what elements and what changes affect user behaviour

• Inconclusive test results give you the chance to try something else

• Each test is a stepping stone-iterate, test and move on

Lifetime Value of Test Result

• A small uplift over a two week test period might leave you feeling underwhelmed

• However, what is that uplift worth when extrapolated over a whole year?

• Even a modest test result could transform your business

There’s No Such Thing As Bad Data

• Everybody loses tests

• It’s part of the game

• Losing tests stop you from going off with design changes in the wrong direction

• Losing tests help to point you back in the right direction

Follow The Money

• ... and finally... follow the money

• Many platforms allow you to track revenue as well as conversions- if not, integrate with Google Analytics

• Even if you increase conversions it may not increase revenue- your design could make it easier for customers to buy your lower priced items

• Track the test impact on your bottom line either through your test tool or your analytics

Contact Oban Digital

Phone: +44 1273 936095

Email: sales@obandigital.com

Website: www.obandigital.com

Twitter: twitter.com/ObanDigital