startup analytics

STARTUP ANALYTICSGetting Started Down the Path to Understanding

Your Business and Your Users

Dale BeermannChief Technology and Analytics Officer

dale@studyblue.com

THE GOAL OF ANALYTICS:IMPROVING YOUR BUSINESS BY

ANSWERING AND ACTING ON QUESTIONS

With every question answered, ask yourself if it is the desired result. If not, determine what needs to be

done to improve it.

BUSINESS METRICSVERSUS

USAGE METRICSYou should always be reporting on your business metrics. Analytics is the way to understand what is driving them.

Effectively, business metrics are the aggregate result of your usage metrics.

BUSINESS METRICS

The ultimate goal of business metrics is to evaluate the health of your business. Examples:

How fast is your business growing?

What is your churn rate?

What is your cost per acquisition for each channel?

What is your Average Revenue per Active User?

USAGE METRICS

The ultimate goal of usage metrics is to evaluate the health of your product. Examples:

What percentage of your users are realizing your value propositions?

Is your new feature reaching the expected audience?

What percentage of users make it through the onboarding process?

What percentage of users are using social channels?

INFLUENCING BUSINESS METRICS

Know the answers to your high-level business metrics before digging into your usage.

Use your usage metrics to determine how you can influence your business metrics.

BEST PRACTICES

THE RIGHT TIME TO START

Have you found your product/market fit?

There may be some high level business metrics that help you get there, but don’t start your analysis on a product that is going through a massive amount of change.

WHERE TO START

Have you filled out a Business Model Canvas? What are your business’ most important metrics?

How well are each of your customer segments doing when it comes to realizing your value propositions?

Take your value propositions and work backwards through the paths that your users take to get there.

DEVELOP GOOD HABITS

Make Analytics a core part of your development workflow.

Ensure you are creating both good behavioral habits as well as good programming habits.

GOOD BEHAVIORALHABITS

Review your metrics on a regular basis

Continually log changes that are going into your product

There will inevitably be a point in the future where you ask yourself what happened six months ago to influence a particular metric.

GOOD PROGRAMMING HABITS

Create guidelines and tools that require you to implement metrics as you build out your software

E.g. Use abstract click handlers that can be easily refactored:

display.addClassesHandler(new SBClickHandler(SBAnalytic.HOME_FIND_CLICK) { @Override public void doOnClick(ClickEvent event) { ... }});

AVOID VANITY METRICS

Page views don't matter (impressions may).

Time On Site can be interesting, but doesn't necessarily convey usage.

It’s very difficult to influence metrics like Page Views or Time on Site. Attempting to do so will be a waste of your time.

FOCUS ON ACTIONABLE METRICS

These are going to be different for every business.

Again, you want to find the metrics that mean the most to your company and determine how you can influence them.

MAXIMIZING A METRIC CAN HAVE SIDE EFFECTS

Providing multiple options splits your usage between them. Similarly, forcing users down one particular path means they can’t take another. This can arise in subtle ways.

In some cases, such as with a payment page, you may be able to find the optimal solution without many side affects.

Ask yourself: What user segments are affected by this change? Will any side effects be worth it?

IMPLEMENTATION

CAVEAT: I DO NOT SUBSCRIBE TO THE IDEA THAT YOU SHOULD LIMIT WHAT YOU TRACK.

If you are smart about how you’re doing your analysis, you will not fall into the trap of “analysis paralysis.”

START WITH GOOGLE ANALYTICS

It’s free and you can throw everything at it without worrying about usage tiers.

We don’t use the high level (vanity) metrics for much. Rather, by sending our events through Google Analytics, we have the ability to answer a lot of questions.

GETTING THE MOST OUT OF GOOGLE ANALYTICS

Track all of your events (views, clicks, actions).

This isn’t limited to your click stream. Track final events for workflows (e.g. completed_onboarding). This allows you to create Advanced Segments for those events.

Set up profiles for each platform (web, iOS, Android, etc.).

You’re going to have very different usage patterns for each platform, and they should be analyzed separately.

GETTING THE MOST OUT OF GOOGLE ANALYTICS

Make use of custom variables.

At the very least, you should be setting your (non personally-identifiable) user ID as one of the variables.

This will let you find some per-user data that is otherwise difficult with Google Analytics.

If you have organizational data, or if your users are segmented in pre-defined ways, this can help look at those segments more closely.

I’M TRACKING MY EVENTS. NOW WHAT?

Funnel Analysis

The goal of a funnel analysis is to determine where your users are falling off.

Take one of your core metrics and walk through the steps it takes to get there.

FUNNEL ANALYSIS EXAMPLE

StudyBlue and Indexable Content

We want to maximize the amount of content created that is “paired” with a class. How does that happen?

HOW DO YOU IMPROVE YOUR FUNNELS?

Think about how can you change an experience to improve the end result.

Sometimes this is as simple as changing a button’s color or using a modal popup (while thinking about the side effects).

A/B Testing

A/B Testing can be a reliable way to evaluate multiple paths.

Caveat: Do your homework and understand statistical significance. Learn what a chi-squared test is.

TOOLS FOR FUNNEL ANALYSIS

Google Analytics does make it possible to do some of this.

Their goal conversions are annoying if you don’t use page views the way they expect.

Create advanced segments for users with particular events.

Other good for-pay tools are KissMetrics and Mixpanel.

Roll your own.

In all honesty, doing this stuff yourself isn’t that hard.

TANGENT: YOUR OWN IMPLEMENTATION

You’ll want to use partitioned tables under the hood (if your data store supports it).

In postgresql, we use triggers to write data to the correct table. Queries then only hit the necessary tables for the time span you’ve defined.

We got away with a table per week for about 5 years. Our table schema:

user_id, session_id, platform, activity_id, activity_timestamp, activity_detail

ADVANCED ANALYTICS

COHORT ANALYSIS

A cohort is a set of users grouped in a particular fashion.

Typical cohorts are time-based (week of registration). Cohorts can also be based on acquisition campaigns (e.g. Adwords vs. Direct vs. SEO).

The purpose of a cohort analysis is to understand user retention and if your changes are making an impact between cohorts.

WHY COHORT ANALYSIS

Most educated investors are going to ask for cohort analyses. Cohort analyses, and their corresponding retention rates help determine:

Engagement levels. Are you a one-and-done sort of site?

Churn rates. If users aren’t coming back to your site, or if churn is higher than acquisition, your site will not grow.

Quantifying the value of your existing userbase.

A COHORT ANALYSIS EXAMPLE

Sadly, I can’t provide some of our own data here. But this is what your cohort analyses will look like:

COHORT ANALYSIS QUERIES

In Posgtresql: crosstab.

In MySQL: Pivot Tables (still pretty manual).

In everything else: pull your data into one of the above. Or write a lot of code.

THE HOLY GRAIL

A full fledged Customer Relationship Management system driven from your analytics solutions:

Adaptive in-app user education

Drip email campaigns

Churn prediction

Re-engagement

QUESTIONS?