the freemium model and a/b testing

Confidential & Proprietary

The Freemium Model & A/B Testing

HostingCon 2011

By Haroon MokhtarzadaCEO & Co-Founder, Webs.com


Introduction

2

Haroon MokhtarzadaCo-Founder and CEO

Entrepreneur from the age of 10. Co-founded Webs in 2001 with brothers Zeki and Idris while studying Economics at the University of Maryland. Obtained J.D. from Harvard Law School in 2005 and thereafter joined Webs full-time.

Founding Vision“Build a free website creation tool so easy that Mom can use it”

FundingRaised $12.1 million from Novak Biddle, Columbia Capital and prominent angels in August 2006.


About Webs

3

Webs is a leading provider of online “front door” solutions for micro businesses across web, mobile and social platforms.

40 million websites created since inception

Started as a tool for anyone to easily create a website for free and evolved to focus on online solutions for micro businesses

Recently acquired and relaunched Pagemodo.com (Freemium Facebook page creation tool)

We run an average of 3 different A/B tests at any given time


Freemium – A Success Story

4

As a disruptive business model, freemium has been the driving force behind many successful startups and $billions in value creation


Simple Fundamentals

5

Freemium is a business model that works well in scenarios where the marginal cost of users can be driven towards 0 over time or through economies of scale

$ Per Active User

Time (or # users)

Variable Cost

ARPU


What Is a Freemium Product/Service?

Freemium is NOT…

Free trial or sample – user forced to pay after period of time

Shareware – user forced to pay to realize actual benefit of product

6


What Is a Freemium Product/Service?

Freemium IS…

A free, usable product/service

That attracts users through its price tag and free user base

And monetizes with optional paid premium features(typically subscription based)

7


Freemium is merely one option:

Popular web business models:

Sell a product to customers (premium) Upsell a product to your users (freemium)

Sell other people’s products to your users (ads)

Sell your users’ data (ex: addthis)

8

Than

ks!


Why we “chose” Freemium 10 yrs ago

Broke - No marketing budget

Clueless - Uncertainty about subscription/pricing model

Frugal - Cost per user was low

9


Challenges for Freemium Players

Low Conversion - People who may have been willing to pay normally do not pay because the free product is sufficient

Low ARPU – makes it difficult to accelerate growth via traditional marketing methods

Infrastructure – millions of free users can create non-trivial and costly infrastructure problems

Customer Support – free users often still demand support

10


Benefits of the Freemium Model

Viral – Rapid adoption, opens new, affordable, channels for getting users

Disruptive – Compete with even the most entrenched premium competitors by giving it away for free

Superior UI/UX – Offer a better product through rapid testing / customer development 11


The Freemium Offering on Webs

12

Free Starter Enhanced Pro

Sitebuilder & Apps

Included Included w/ Prem. Themes

Included w/ Prem. Themes

Included w/ Prem. Themes

Pages 10 Unlimited Unlimited Unlimited

Domain Name Included Included

Email Addresses

5 15

Webstore Items

5 5 15 Unlimited

Storage 40Mb 400Mb 1Gb 5Gb

Customer Support

Community 24/7 Live Chat 24/7 Live Chat 24/7 Live Chat


Leveraging our Free Users

13

Direct Marketing from Free UsersLinks on bottom of sitesBranding - Logo on bottom of sitesSEO value from websites back to Webs.comURL has webs.com in it

Indirect Marketing from Free UsersWord of mouthSize / Credibility


How Premium Businesses Can Leverage Freemium

14

Offer a limited version for free (but be careful NOT to cannibalize)

Create new, related free or freemium product, and use as marketing funnel for your paid product

Partner with a freemium product, offer their premium product to your audience

Confidential & Proprietary 15

Part II: Testing… Testing… 1 2 3

HostingCon 2011


The Freemium Conversion Funnel

16

Visitors

Registrations

Engagements

Referrers

$

“What gets measured gets improved”


Conversion Timeline

17

Understanding When Your Users Convert Is Key

Our likelihood of conversion decreases over timeWhy? Users are publishing a new website and want all features at time of launchImplications: Focus on upselling EARLY in customer lifecycle

Other models:User more likely to convert as time goes on:

- Dropbox (space fills up)- Mailchimp (email list grows)

Implications: Deemphasize upsell up front, and focus on getting as much usage as possible


Simple A/B Testing

Define some goal event

Create test treatments

Test - randomly split visitors into test & control groups; measure success to goal

Good for conversion events with immediacyEx: Homepage, signup page, shopping cart


Simple A/B TestingWhat to test?

Calls-to-action (buttons)Copy, Color, Size, Placement, Hover states

Forms/FieldsDescriptors, Size, Length/number, etc.

TextHeadlines, Value Props, Links, Amount (long/short)

Imagery

Segment results by source, use landing pages* Bonus Points


AbTests.com

Great site for testing inspiration


Case Study

21

Shortened Signup Flow

Tested shorter 1-page flow against 2-page flow with more fields

Shortened version created 10,000 more signups over 3 weeks

Going 100% translates to over 25,000 more signups/month!

Control Test


A/B Test Reporting (GWO)

Confidence increases over timeLow variation is goodGoogle Website Optimizer is good because:

Provides visitors a consistent experienceMeasures & reports confidence / significanceOffers multivariate (combination) testing


Interpreting ResultsWhen A/B Testing is Not Enough

Freemium Problem: Signup != $$$

A/B signup test only told us which produced more USERS, but does NOT tell us which produced more $$

Example: Asking more questions in signup will reduce signups but may increase user success rate and make more $$


Database Driven Cohort Analysis

Solution:

A system that permanently fixes a user to a test group, allowing one to run results analysis on any variable that is also attached to that user (i.e. $ spent)

Offers the ability to compare users who received different treatments through your entire funnel over time


Cohort Funnel Analysis Example

Control Test 1 Test2

Homepage 10000 10000 10000

Signup 3000 (30%) 2900 (29%) 3300 (33%)

Control Test 1 Test2

Homepage 10000 10000 10000

Signup 3000 (30%) 2900 (29%) 3300 (33%)

Publish 1800 (60%) 1850 (64%) 1950 (59%)

Purchase 180 (10%) 203(11%) 195 (10%)

A/B testing would have shown Test2 as the winnerTest 1 makes more money. Tradeoff?

* Dummy data


How to Set Up Cohort Testing

Quick & Dirty Shortcut: Mod on userid(ex: show even users one treatment and odd users another)

Thanks Eric Ries!

In Cohort?

Coin Toss

DisplayTreatment

Assign to Cohort

User Enters

Yes

No


Visualizing Results

27

6/15/2

011

6/16/2

011

6/17/2

011

6/18/2

011

6/19/2

011

6/20/2

011

6/21/2

011

6/22/2

011

6/23/2

011

Cumulative $

Control TEST

Day to day reporting on metrics like Premium $ can often be lumpy.

Looking at a cumulative view helps visualize more smoothly

Diverging lines means one is superior

Parallel lines apart means one got lucky;)


Segmentation & Cohort AnalysisAdvantages

Integrated front- and back-end testing

Complete ownership of user data

Ability to track impact to user long after user-facing experience has gone away

Provide users with consistent test experience that integrates your back-end - Great for pricing & promotional tests

Ability to “slice” data by internal user variables


Advanced Segmentation

29

Total Personal Group Business

Conversion Rates by Segment

Control Test

Total/Aggregate results can hide the real story

Not all users are the same

Segmenting data and “slicing” can produce surprising gains

Treatments can be applied to different segments accordingly Total Personal Group Business

Conversion Rates by Segment

Control Test


Visualizing Advanced Segmentation

30

Cumulative Charts

6/15/2

011

6/16/2

011

6/17/2

011

6/18/2

011

6/19/2

011

6/20/2

011

6/21/2

011

6/22/2

011

6/23/2

011

Segment 1

Control TEST

6/15/2

011

6/16/2

011

6/17/2

011

6/18/2

011

6/19/2

011

6/20/2

011

6/21/2

011

6/22/2

011

6/23/2

011

Segment 3

Control TEST

6/15/2

011

6/16/2

011

6/17/2

011

6/18/2

011

6/19/2

011

6/20/2

011

6/21/2

011

6/22/2

011

6/23/2

011

Segment 2

Control TEST

Different reports for each segmentCan use end results to find interesting variable combinations, then check for significance


Why A/B Testing is Crucial

31

Lots of Leverage – since your free customers are often your largest referring source, improving your funnel can have compounding effects on both virality and conversion.

Users constantly surprise – its impossible to guess what users will do. That’s why all great companies use testing.

Validate ideas and time/energy spent on product efforts

Better Serve Your Users – build a better product and improve your intuition about your customers


Summary

32

The freemium model is here to stay

When implemented properly, freemium is a disruptive business model allowing rapid growth in new or existing markets

Improving conversion is crucial to success, and requires rapid iteration and A/B testing

Rapid iteration and testing cannot be done without large influx of free users, favoring those who have achieved scale

Successful freemium products are always-improving, and tend to have the superior metrics & UX/UI (out of sheer necessity!)

Premium players who do not want to disrupt legacy business can can partner with freemium players to leverage their assets


Q&AHaroon Mokhtarzada, CEO & Co-Founder of Webs@haroon, [email protected]

the freemium model and a/b testing

Business

freemium isa free

free product

freemium productservice

freemium offering

free userslinks

free user baseand

free users13direct marketing

users adssell