culture of optimization

PRESENTED BY:

ERIC HANSER – DIGITAL OPERATIVE

Culture of OptimizationEcommerce A/B and Multivariate Testing Primer

digitaloperative.com Tweet us @digitaloperativ

THESE ARTICLES WILL NOTGIVE YOU A STRATEGY

Some of these tests may be effective, and some may not. Probably more of the latter.

Every site is unique and has unique users with unique problems.

Without more concrete hypotheses, a failed test leads to a dead end.


NUMBER 1Getting your site into shape is just like getting yourself into shape.

There is no magic pill or simple solution. It is a lifestyle change.


OK. SO WHERE DO YOU START?


LET’S TAKE ONE STEP BACK.

WHAT DO WE MEAN BY “TESTING”


SPLIT TESTING: SIMPLE IDEA, LOTS OF POSSIBILITIES

A split test is a controlled, randomized experiment run on your live site.

Users to your site are split up and shown different versions of the site.

The users’ progress is tracked to determine which version will be most successful.


SPLIT TESTING: PLATFORMS


CRO = CONVERSION RATE OPTIMIZATION


WHAT HAPPENS IF YOU IMPROVE CONVERSION RATE,

BUT AOV GOES DOWN?


THE TERM CRO CAN BE SOMEWHAT MISLEADING

A “conversion” technically refers to a a user completed any desired action.

In ecommerce, the word “conversion” is almost always used in reference to making a purchase.


THE TERM CRO CAN BE SOMEWHAT MISLEADING

Split testing can be used effectively to help with everything from branding to design to marketing, etc.

By using the term “CRO,” some departments or colleagues might not see immediately that these tools and processes are relevant and helpful to the work they do.


NUMBER 2In building a culture of optimization, it may be helpful to limit how often you use the term “CRO.”


TO AVOID CONFUSION, USE THE TERM

OPTIMIZATION & TESTING


WHAT ARE THE KINDS OF TESTS WE’RE TALKING ABOUT?


A/B TESTS: SIMPLEST (AND RAREST)

A/B Testing simply means testing one element on a page. Each option is called a version or a variation.

Not limited to 2 variations. Sometimes called A/B/C or A/B/N testing.

Only a single element is tested: color, placement, wording, picture, etc.

If more than one of these elements change, it is no longer an A/B test.


TYPES OF TESTS: A/B TESTING


MULTIVARIATE TESTING: A BIT MORE TIME

When more than one element on the page is being tested.

Each possible combination of elements should be tested.

With more variations, it can take longer to get statistically significant results (more on this later).

All variations are still on a single page.


TYPES OF TESTS: MULTIVARIATE TESTING


MULTI-PAGE TESTING: USERS SEE MORE THAN ONE PAGE

If an element appears on more than one page, you must test the change on each page.

Also often called “Flow Testing.”

Multi-page testing may be either A/B or Multivariate.


TYPES OF TESTS: MULTI-PAGE TESTING


SPLIT TESTING TYPES

A/B Tests – A single-page test with a single element changed

Multivariate Tests – A single-page test with more than one element changed.

Multi-page Tests – A test that shows a change across more than one page in the user flow.


ALL OF THESE TEST RANDOMLY SPLIT USERS INTO GROUPS.


TARGETING: ALL USERS ARE NOT THE SAME.

Optimizing your site for all users is important, but speaking directly to specific segments is even more effective.

Many of the same tools that are used to run split testing can also be used to target specific audiences and optimize the site for them.


NOT ALWAYS A TEST:SEGMENTING AND

TARGETING


NUMBER 3Results from testing can lead to targeting.

When looking at results of your test, you may find that certain segments should be targeted differently.


SO, WHERE DO YOU BEGIN?


IT IS TEMPTING TO JUMP RIGHT IN

It may seem that there are things “wrong” with your site or that you can make tests based on “best practice.”

Tests have “winners” and “losers” - there will be scrutiny.

The more information you have, the better chance you have of making wins early on.


NUMBER 4Don’t start with testing!

If your first few tests are flops, confidence in the whole idea of split testing could be compromised.


START WITH LEARNING THE “WHATS” AND “WHYS” OF

YOUR SITE.


FIND THE “WHAT”DATA, DATA, DATA!


FIND THE “WHAT”

What are users doing or not doing? What do they like and/or dislike?

Learn all you can from analytics. Proper set up and tracking could take time.

Ask questions like:What are the biggest drop-off and exit points?What pages are most successful?What are the most optimal site flows? What are

the least optimal?


FIND THE “WHY”USABILITY TESTING


FIND THE “WHY”

With data collected from analytics, develop paths or a series of tasks for users to complete in a usability test.

Several tools are available to either conduct in-person or remote usability testing. Usertesting.com is one of the most popular.

Often, unexpected and surprising results surface that you would have not necessarily seen in the analytics alone.


YOU KNOW THE PROBLEMS.

NOW, MAKE A HYPOTHESIS.


START WITH THE HYPOTHESIS – THEN DEVELOP THE TEST

When making a test, base your hypothesis on the problem.

“If I change X, then Y should improve.”

Never make a test and THEN ask the question, “how will I know which variation won?”

Simply looking at overall ecommerce conversion will likely be inconclusive.


WHAT & WHY30% of traffic direct to product pages

63% bounce rate

“Best Practice” recommendation to increase the add-to-cart button size and the free shipping message


WHAT & WHYUsers had no validation that the product was right for their specific condition.

Users had no clear way to find the right brace for them.


GETTING RESULTSSTATISTICS, BLEH!


GETTING RESULTS – DETERMINING THE WINNER!

Two primary metrics:Percentage Increase = How much a variation

improved over originalConfidence Interval = How confident we are

that the improvement is not due to chance

A 10% improvement with an 85% confidence interval basically means that there is an 85% chance that the test will result in a 10% improvement.

95% confidence interval = Statistically Significant


GETTING THE RESULTSSTATISTICAL SIGNIFICANCE

10% Improvement85% Confidence Interval



Original


WIN, LOSE, OR DRAW

Optimization is ongoing and iterative.

Ultimately, if a test is based on data and usability, we’re always learning and improving.


NUMBER 5If your testing is always based on data, every test will be a win.

If your tests aren’t based on data, even wins may be losses.


WINS & LOSSES – WHEN A WIN IS NOT A WIN.

If a test is implemented without a clear hypothesis and clear goals, a win can lead to a dead end.

If you don’t have a clear idea as to WHY a variation won, we’ve got nowhere to go.


WHEN A WIN IS NOT A WIN


WINS AND LOSSES – WHEN A LOSS IS NOT A LOSS

Similarly, we can learn a lot when a variation underperforms if the hypothesis was based on data.

With new information in mind, we can look again at the problem with additional data to find other solutions.


digitaloperative.com

WHEN A LOSS IS NOT A LOSS


WINS & LOSSES – WHEN A DRAW IS A WIN

Occasionally, a test goal will be something less “tangible” than an increase in conversions.

Some tests’ goals, for example, will be to bring consistency to a cluttered site, make copy more on brand, or to adapt to changing business requirements.


WINS AND LOSSESWHEN A DRAW IS A WIN


MAKE A PLAN – PRIORITIZE!

Not everything can be tested at once.

The P.I.E prioritization process helps put all possible tests in a digestible order.

P = Potential / How big is the problem?I = Importance / How valuable is the traffic?E = Ease / How difficult will the test be to

implement?


MAKE A PLANPRIORITIZE!

WEBSITE TESTING PrioritizationTest Name HYPOTHESIS Potential Importance Ease PIE AVG PIE Notes

Product Page - Short Product Description

By giving users feedback related to their path (filters, categories, search terms) they will be more confident in purchasing 8 9 7 8

High Traffic, High Bounce Rate, text-only implementation, but need use of audiences tool

Cart / Global Nav - View Cart

Design and placement of "view cart" CTA will bring more people back into the purchasing funnel if they leave the cart 6 8 9 7.666666667

Valuable traffic, but most users are directed to the shopping cart. Easy implementation

Category - Subcategory/Filter Design

By providing clarity about the differentiation between subcateory links and filters, potential UX issues will be mitigated 8 8 7 7.666666667

Noted problem/issue in usability. Use of filters shows intent, and therefor important.

Checkout - Security Messaging

Overstating Security may actually be harming conversion. Possibly finding a more optimal amount of messaging could improve it 5 9 9 7.666666667

No known issue, but hightly valuable traffic, and extremely easy test to implement.

Product Page - Find Other Products

Allowing users on a PDP to easily click off to other similar products if it not appropriate for their needs, we decrease bounce rates 8 7 7 7.333333333

Lots of traffic, high bounce rate, but the feature would be new, and therefore importancei is measured

Global Nav - Elements and Design

Users perform better when accessing injury categories as opposed to body type cateogirs 8 8 6 7.333333333

As a global element, importance and potential is high. Implementation will be slightly more than others.


NUMBER 6Prioritization not only keeps you organized, it helps your organization understand the breadth and importance of optimization and testing overall.


CULTURE OF OPTIMIZATIONWEBSITE FITNESS TIPS

There is no magic pill – It’s a lifestyle change.

The term “CRO” can be confusing – Use “Optimization & Testing.”

Test then target – Some segments may need to be targeted differently.

Don’t start with testing – Gather data (“whats” and “whys”) first.

Base tests on solid data and hypotheses – All tests can be wins.

Stay organized – Keep a clearly prioritized list of possible tests.


QUESTIONS?

[email protected]

WWW.DIGITALOPERATIVE.COM


PLEASE USE THE OR APP TO RATE THIS SESSION

Thank you for attending!

culture of optimization

Business

high bounce

results start

statistically

view cart

targeted differently

term cro

goal set

magic pill