startup analytics

32
STARTUP ANALYTICS Getting Started Down the Path to Understanding Your Business and Your Users Dale Beermann Chief Technology and Analytics Officer [email protected]

Upload: dale-beermann

Post on 01-Nov-2014

523 views

Category:

Documents


2 download

DESCRIPTION

Presentation given to the Gener8tor portfolio companies on 1/31/2013.

TRANSCRIPT

Page 1: Startup analytics

STARTUP ANALYTICSGetting Started Down the Path to Understanding

Your Business and Your Users

Dale BeermannChief Technology and Analytics Officer

[email protected]

Page 2: Startup analytics

THE GOAL OF ANALYTICS:IMPROVING YOUR BUSINESS BY

ANSWERING AND ACTING ON QUESTIONS

With every question answered, ask yourself if it is the desired result. If not, determine what needs to be

done to improve it.

Page 3: Startup analytics

BUSINESS METRICSVERSUS

USAGE METRICSYou should always be reporting on your business metrics. Analytics is the way to understand what is driving them.

Effectively, business metrics are the aggregate result of your usage metrics.

Page 4: Startup analytics

BUSINESS METRICS

The ultimate goal of business metrics is to evaluate the health of your business. Examples:

How fast is your business growing?

What is your churn rate?

What is your cost per acquisition for each channel?

What is your Average Revenue per Active User?

Page 5: Startup analytics

USAGE METRICS

The ultimate goal of usage metrics is to evaluate the health of your product. Examples:

What percentage of your users are realizing your value propositions?

Is your new feature reaching the expected audience?

What percentage of users make it through the onboarding process?

What percentage of users are using social channels?

Page 6: Startup analytics

INFLUENCING BUSINESS METRICS

Know the answers to your high-level business metrics before digging into your usage.

Use your usage metrics to determine how you can influence your business metrics.

Page 7: Startup analytics

BEST PRACTICES

Page 8: Startup analytics

THE RIGHT TIME TO START

Have you found your product/market fit?

There may be some high level business metrics that help you get there, but don’t start your analysis on a product that is going through a massive amount of change.

Page 9: Startup analytics

WHERE TO START

Have you filled out a Business Model Canvas? What are your business’ most important metrics?

How well are each of your customer segments doing when it comes to realizing your value propositions?

Take your value propositions and work backwards through the paths that your users take to get there.

Page 10: Startup analytics

DEVELOP GOOD HABITS

Make Analytics a core part of your development workflow.

Ensure you are creating both good behavioral habits as well as good programming habits.

Page 11: Startup analytics

GOOD BEHAVIORALHABITS

Review your metrics on a regular basis

Continually log changes that are going into your product

There will inevitably be a point in the future where you ask yourself what happened six months ago to influence a particular metric.

Page 12: Startup analytics

GOOD PROGRAMMING HABITS

Create guidelines and tools that require you to implement metrics as you build out your software

E.g. Use abstract click handlers that can be easily refactored:

display.addClassesHandler(new SBClickHandler(SBAnalytic.HOME_FIND_CLICK) { @Override public void doOnClick(ClickEvent event) { ... }});

Page 13: Startup analytics

AVOID VANITY METRICS

Page views don't matter (impressions may).

Time On Site can be interesting, but doesn't necessarily convey usage.

It’s very difficult to influence metrics like Page Views or Time on Site. Attempting to do so will be a waste of your time.

Page 14: Startup analytics

FOCUS ON ACTIONABLE METRICS

These are going to be different for every business.

Again, you want to find the metrics that mean the most to your company and determine how you can influence them.

Page 15: Startup analytics

MAXIMIZING A METRIC CAN HAVE SIDE EFFECTS

Providing multiple options splits your usage between them. Similarly, forcing users down one particular path means they can’t take another. This can arise in subtle ways.

In some cases, such as with a payment page, you may be able to find the optimal solution without many side affects.

Ask yourself: What user segments are affected by this change? Will any side effects be worth it?

Page 16: Startup analytics

IMPLEMENTATION

Page 17: Startup analytics

CAVEAT: I DO NOT SUBSCRIBE TO THE IDEA THAT YOU SHOULD LIMIT WHAT YOU TRACK.

If you are smart about how you’re doing your analysis, you will not fall into the trap of “analysis paralysis.”

Page 18: Startup analytics

START WITH GOOGLE ANALYTICS

It’s free and you can throw everything at it without worrying about usage tiers.

We don’t use the high level (vanity) metrics for much. Rather, by sending our events through Google Analytics, we have the ability to answer a lot of questions.

Page 19: Startup analytics

GETTING THE MOST OUT OF GOOGLE ANALYTICS

Track all of your events (views, clicks, actions).

This isn’t limited to your click stream. Track final events for workflows (e.g. completed_onboarding). This allows you to create Advanced Segments for those events.

Set up profiles for each platform (web, iOS, Android, etc.).

You’re going to have very different usage patterns for each platform, and they should be analyzed separately.

Page 20: Startup analytics

GETTING THE MOST OUT OF GOOGLE ANALYTICS

Make use of custom variables.

At the very least, you should be setting your (non personally-identifiable) user ID as one of the variables.

This will let you find some per-user data that is otherwise difficult with Google Analytics.

If you have organizational data, or if your users are segmented in pre-defined ways, this can help look at those segments more closely.

Page 21: Startup analytics

I’M TRACKING MY EVENTS. NOW WHAT?

Funnel Analysis

The goal of a funnel analysis is to determine where your users are falling off.

Take one of your core metrics and walk through the steps it takes to get there.

Page 22: Startup analytics

FUNNEL ANALYSIS EXAMPLE

StudyBlue and Indexable Content

We want to maximize the amount of content created that is “paired” with a class. How does that happen?

Page 23: Startup analytics

HOW DO YOU IMPROVE YOUR FUNNELS?

Think about how can you change an experience to improve the end result.

Sometimes this is as simple as changing a button’s color or using a modal popup (while thinking about the side effects).

A/B Testing

A/B Testing can be a reliable way to evaluate multiple paths.

Caveat: Do your homework and understand statistical significance. Learn what a chi-squared test is.

Page 24: Startup analytics

TOOLS FOR FUNNEL ANALYSIS

Google Analytics does make it possible to do some of this.

Their goal conversions are annoying if you don’t use page views the way they expect.

Create advanced segments for users with particular events.

Other good for-pay tools are KissMetrics and Mixpanel.

Roll your own.

In all honesty, doing this stuff yourself isn’t that hard.

Page 25: Startup analytics

TANGENT: YOUR OWN IMPLEMENTATION

You’ll want to use partitioned tables under the hood (if your data store supports it).

In postgresql, we use triggers to write data to the correct table. Queries then only hit the necessary tables for the time span you’ve defined.

We got away with a table per week for about 5 years. Our table schema:

user_id, session_id, platform, activity_id, activity_timestamp, activity_detail

Page 26: Startup analytics

ADVANCED ANALYTICS

Page 27: Startup analytics

COHORT ANALYSIS

A cohort is a set of users grouped in a particular fashion.

Typical cohorts are time-based (week of registration). Cohorts can also be based on acquisition campaigns (e.g. Adwords vs. Direct vs. SEO).

The purpose of a cohort analysis is to understand user retention and if your changes are making an impact between cohorts.

Page 28: Startup analytics

WHY COHORT ANALYSIS

Most educated investors are going to ask for cohort analyses. Cohort analyses, and their corresponding retention rates help determine:

Engagement levels. Are you a one-and-done sort of site?

Churn rates. If users aren’t coming back to your site, or if churn is higher than acquisition, your site will not grow.

Quantifying the value of your existing userbase.

Page 29: Startup analytics

A COHORT ANALYSIS EXAMPLE

Sadly, I can’t provide some of our own data here. But this is what your cohort analyses will look like:

Page 30: Startup analytics

COHORT ANALYSIS QUERIES

In Posgtresql: crosstab.

In MySQL: Pivot Tables (still pretty manual).

In everything else: pull your data into one of the above. Or write a lot of code.

Page 31: Startup analytics

THE HOLY GRAIL

A full fledged Customer Relationship Management system driven from your analytics solutions:

Adaptive in-app user education

Drip email campaigns

Churn prediction

Re-engagement

Page 32: Startup analytics

QUESTIONS?