metrics that matter
DESCRIPTION
Creating metrics to measure the success or failure of your business may seem simple, but there are many avoidable mistakes that business leaders make every day. Many leaders end up focusing on “vanity metrics”, which might make the business look healthy, but do not actually give you insights into what is going on. And worst case, they are masking larger issues that you need to know and should be addressing.TRANSCRIPT
Metrics That Matter
How do you measure success? Many online businesses focus on dashboard-friendly metrics that answer questions like: How many new views did we have today? How many new daily active users? How much revenue did we bring in last week? Good information to have—and it’s never a bad idea to get people thinking about data—but are you really getting the information you need to make the best-possible decisions?
Modern analytics is about so much more than defining high-level KPIs and building simplistic dashboards to amplify them. With very little additional work, you can create custom analytics that inform product design and business growth. It’s as straightforward as defining the behaviors you want to quantify, measuring them in a logical way, and then pushing them through product and process changes—ultimately making for happier customers, more efficient operations, and higher revenues. At the same time, you’ll learn what data you should be tracking, so you can refine your metrics and monitor them over the long haul.
Take a look at three different scenarios where custom analytics work to support business goals. By creating specialized metrics that explain user behaviors, each of these companies was able to quantify a negative customer experience, gain valuable insights, and take action to solve the problem.
Define
Act &Refine
Quantify
Metrics That MatterCreating custom analytics that quantify user behaviors and drive business practices.
2 Metrics The Matter
Improve user experience by tracking when it’s badToday’s online businesses now have huge quantities of data at their fingertips, allowing product teams to adapt their designs to actual user behaviors. When Venmo, which produces a peer-to-peer payment app, realized that the vast majority of their users were sending money to friends (rather than charging them), the product team redesigned the interface to make it faster and easier to send money. The result? More people sent more money, boosting engagement metrics—just as the product team had hoped.
But, oops, the redesign caused a problem for the small percentage of people who were trying to request payments from their friends. By making it easier to send money, the new interface had made it more confusing to request payment, which resulted in user errors, double payments, offsetting reversals, and all-around social awkwardness.
How big was the problem? To quantify it, the Venmo product team developed a custom metric called “pay-charge mixup,” which measured the number of users who made a payment and then immediately requested double payment.
% of iOS users that make a pay/charge mix up - app version
Pay-charge mixups afterthe redeigned interface
Thu 15 Sat 17 Mon 19 Wed 21 Fri 23 Mon 25 Tue 27 Thu 29 Sat 3 Tue 03 Thu 05 Sat 07 Mon 09 Wed 11 Fri 13 Jun 15 Tue 17 Thu 19 Sat 21 Mon 23 Wed 25 Fri 27 Jun 29
pre-V6 v6.0-v6.1
0
2
4
6
8
10
PAYM
ENT
CH
ARG
E SW
ITC
H P
erce
nt R
ecip
ient
s Ar
e C
onfi
With the new version they saw that the instance of this type of error was unacceptably high. This information gave the team the support they needed to launch another redesign, version 6.2, which fixed the user experience.
% of iOS users that make a pay/charge mix up - app version
Pay-charge mixups afterthe redeigned interface
Mon 14 Tue 15 Wed 15 Thu 17 Fri 18 Sat 19 Jul 21 Mon 21 Tue 22 Wed 23 Thu 24 Fri 25 Sat 26 Jul 27 Mon 21 Tue 29 Wed 30 Thu 31 August Sat 03 Aug 03 Mon 04 Tue 05 Wed 06 Thu
pre-V6 v6.0-v6.1 v6.2+
0
2
4
6
8
10
PAYM
ENT
CH
ARG
E SW
ITC
H P
erce
nt R
ecip
ient
s Ar
e C
onfi
With the fixed version, this bad experience was now back to manageable levels. Going forward, the Venmo product team has a useful metric to watch as they continue to iterate and improve their app.
After the first redesign, many users sent money to friends when they intended to charge them.
Messages Jason Contact
Wednesday 10:17 AM
OH GODDAMMIT
Okay i can deal with this
Lollll ok
Lol sorry I messed up. I paid you for the AC instead of charging you so I’m gonna charge you double now...
Yes I just paid you 3 times the amount for the AC
Now it says you paid me double...
iMessage Send
3 Metrics The Matter
Track user happiness to focus the teamUser ratings on app stores represent a crucial metric that can make or break many startups. But think about apps that deliver local services with local supply: innovators like Uber, Lyft, HotelTonight, Instacart, and Shyp. They can often get an unfair rap: If people download the app in a geography where the service doesn’t exist, the user has an overall bad experience—and may feel free to share it on the store site.
How can a business start small, refine both their product and their operations, and then use data to expand to the right geographies in stages? The solution lies in creating a custom metric that quantifies this bad user experience, starting with: How many users open the app for the first time more than N miles from a service area? N equals whatever distance constitutes a poor experience. Then, having quantified a bad user experience, we can set a goal of driving these down to zero.
In the example above, a service provider has quantified a poor experience as 40 miles from a service area (N= 40). The graph shows coverage across several geos: from South America (purple), where coverage is still lacking, down to the United States (green), where coverage continues to improve over time.
Asia
Asia July
Augus
t
Septem
ber
October
November
Decem
ber20
14
Februa
ryMarc
hApril
MayJu
ne July
Augus
t
Septem
ber
October
November
DEcember
2015
Februa
ryMarc
hApril
May
Europe
Location Data Date
South America United States
100
90
80
70
60
50
40
30
20
10
0
Good ratings on app stores are the lifeblood of continuing success. And reviews like these (even ones with bad spelling) can pull down an excellent product or service that’s not available in the user’s area.
Not worth it l downloaded this app several times hoping that it would be changed...still as it is, limited to cretin famous cities only. craaap.
Worthless Only a few cities...not worth my time.
* * * * *
* * * * *
Having quantified a bad user experience, we can set a goal of driving these down to zero.
4 Metrics The Matter
Notice the massive improvement in user experience in Puerto Rico when the product was introduced, which translated into a visible improvement in the U.S. overall. By tracking this custom metric, the business can continue to improve the user interface by focusing on reducing this bad user experience.
To understand more local effects, and what caused a particularly good improvement in the United States, it’s easy to drill down into one particular geo of the United States.
Asia
Asia July
Augus
t
Septem
ber
October
November
Decem
ber20
14
Februa
ryMarc
hApril
MayJu
ne July
Augus
t
Septem
ber
October
November
DEcember
2015
Februa
ryMarc
hApril
May
Europe
Location Data Date
South America United States
100
90
80
70
60
50
40
30
20
10
0
Asia July
Augus
t
Septem
ber
October
November
Decem
bebrua
ryMarc
hApril
MayJu
ne July
Augus
t
Septem
ber
October
November
DEcember
2015
Februa
ryMarc
hApril
May
Europe
Location Data Date
South America United States
100
90
80
70
60
50
40
30
20
10
0
Asia
ber20
14
FebFeru
Asia July
Augus
t
Septem
ber
October
November
Decem
ber20
14
Februa
ryMarc
hApril
MayJu
ne July
Augus
t
Septem
ber
October
November
DEcember
2015
Februa
ryMarc
hApril
May
100
90
80
70
60
50
40
30
20
10
0
Don’t let top line metrics mask the detailed experienceGaming companies that produce game apps like Candy Crush, FarmVille, and Angry Birds love to track Daily Active Users (DAU) and Monthly Active Users (MAU), and data collectors regularly put out rankings of top apps based on these metrics.
But DAU and MAU can easily mask more important metrics related to retention and growth. To ensure attention is paid to ongoing user engagement, producers of a mobile gaming app, much like Candy Crush, developed a quick cohort analysis and turned it into a custom metric that they could track for each level of the game.
Define What kinds of behaviors do we want to measure?
QuantifyHow can we use data to understand behaviors?
ActHow can we create value from the data and put insights to work in our business?
RefineMoving forward, how can we track the right data and drive more valuable insights?
5 Metrics The Matter
Fortunately, it’s easy for a mobile gaming company to quickly iterate and update their app. By completely retuning these three levels, the team was able to eliminate the roadblocks that users were experiencing—and even to encourage churned players to return.
This level-completion custom metric tells them how many users who start a given level complete that level. In the example below, they saw that early, less-engaged users were struggling to get past Level 1, but loyalty quickly jumped and increased steadily over time, with some mild variance. More importantly, though, they also noticed that there were three levels that seemed to be driving users away.
Top 25 Facebook Games by MAU
Rank
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
Candy Crush Saga
Farmville 2
Texas Hold’em Poker
Pet Rescue Saga4
Dragon City
Diamond Crush
Bubble Witch Saga
Coasterville
Bubble Safari
Criminal Case
Words With Friends
Chefville
Pool Live Tour
Farm Heroes Saga
8 Ball Pool
Top Eleven
Farmville
Village Life
Angry Birds Friends
Zynga Slingo
Bejeweled Blitz
SongPop
Bubble Safari Ocean
Tetris Battle
Bubble Island
43,469,305
38,437,500
30,218,750
24,717,139
23,343,750
22,731,250
18,818,750
17,562,500
16,343,750
15,862,500
14,262,500
12,762,500
12,050,000
12,050,000
11,837,500
10,934,750
10,725,000
10,087,500
9,781,250
9,712,500
9,350,000
8,950,000
8,593,750
8,287,500
8,156,250
Game MAU
6 Metrics The Matter
Custom metrics show the devil in the detailsEveryone loves dashboards. And many tech companies find it motivating to have screens on the walls of the office that show how the company is growing. But all too often, these at-a-glance metrics are too simplistic to give a true view of how a product or a business is performing. By adding custom analytics to your metrics mix, whether it’s quantifying a bad experience or tracking retention at a more granular level, you can truly illuminate your business challenges—and gain the information you need to address them.
As you evaluate your current analytics solution or begin to look for a new analytics tool, make sure you have the flexibility to create reusable custom metrics so you can employ these tactics. Your analytics solution should absolutely be tailored to the unique aspects of your business so that you can track and measure the specific custom information that is critical to your company’s success.
Ready to Love Your Analytics?
Come see a demo and schedule your free trial. Call 888-960-2331 or go to:
looker.com/demo
About LookerLooker is an inventive software company that’s pioneering the next generation of
business intelligence (BI). We believe in bringing better insights and data-driven
decision-making to businesses of all sizes. The company has fast become the catalyst
that is creating data-driven cultures at hundreds of industry-leading companies such
as Yahoo!, Gilt, Warby Parker and Sony.
Looker is based in Santa Cruz, CA Follow @lookerdata or visit www.looker.com
© 2015 Looker. All rights reserved. Looker and the Looker logo are trademarks of Looker Data Sciences, registered in the United States. Other trademarks are trademarks of their respective companies. All services are subject to change or discontinuance without notice.