Data Driven Growth

1

John Egan Engineering Manager - Engagement Team @ Pinterest

2

• Retail shopping rewards app • Led Growth Engineering team • Grew from 1MM to 8MM users in 3 years

• Acquired for $200MM

• Visual discovery & bookmarking app

• Eng manager for Engagement • Over 100 million MAUs • Engagement experiments have added millions of WAUs

3

New Users

MAUs

Dormant Users

4

New Users

MAUs

Dormant Users

5

New Users

MAUs

Dormant Users

6

New Users

Dormant Users

Core

Casual

Marginal

7

New Users

MAUs

Dormant Users

MAU Growth Accounting

8

Acquisition

9

Funnels

10

11

2013

12

2014

13

2015

Activation

14

Retained MAUs• Percentage of new signups that are still using the app a month later • Figure out what a good retention number is for your industry

15

1d7s• Percentage of new signups that use the app 1 or more times in the week

following signup • Quicker to run experiments with • At Pinterest this is highly correlated with user’s long-term retention

16

17

18

Cohort Heatmap

19

d0

User’s Signup Date

d30

d60D

ays

Sinc

e Jo

inin

g

Cohort Heatmap

20Source: Placeholder Numbers

Cohort Heatmap

21Source: Placeholder Numbers

Engagement

22

One Key Engagement MetricMedium: Total Time Spent Reading (TTR) Twitter: MAUs with 7+ visits a month (7d28s) Uber: Weekly Trips Facebook: WAUs with 6+ visits a week (6L7s) Pinterest: Weekly Active Repiners or Clickers (WARCs)

23

Finding Your Key Engagement Metric1) What are the actions a user has to take to get value from your product?

2) What is the frequency someone need to use your product

24

User States

Core: Active multiple times a week Casual: Active once or twice a week Marginal: Active a couple times a month New: Joined in the past 28 days Dormant: Not active for 28 days Resurrected: Was dormant, but became active again in the past 28 days

25

Badging Holdout Experiment

2:50 PM 100%

26

Measuring Effect on Engagement

27

*AU Ratios

28

29

DAU MAU DAU/MAU ratioKnighthood 30,014 389,583 7.70%

RockYou Pets 50,482 582,720 8.66%Tetris Friends 100,404 841,223 11.94%

Superpoke Pets 299,505 2,296,175 13.04%Mobsters 539,012 3,431,063 15.71%

Vampire Wars 687,067 3,794,869 18.11%Country Story 1,647,763 8,097,356 20.35%

Fish World 2,084,358 8,581,587 24.29%Pet Society 5,046,347 20,480,397 24.64%Mafia Wars 6,473,504 25,431,622 25.45%

Restaurant City 5,033,468 17,250,419 29.18%Cafe World 7,409,711 22,062,364 33.59%

Source: Social Times & Nabeel Hyatt

Retention & Resurrection

30

Marketing

Pros: • Coverage. Can reach the entire user base

Cons: • Not personalized, lower open rates • User’s have much lower tolerance

compared to emails

31

Transactional

Pros: • More personal than marketing

notifications

Cons: • Users need activity to generate

notifications • Need mechanisms to rate limit

32

Recommendations

Pros: • Coverage. Can reach most users • Personalized to the user

Cons: • Expensive and time consuming to build

out recommendation engine • Quality may not be good if user has low

amount of activity

33

2:50 PM 100%2:50 PM 100%

Measuring Engagement for Emails/Push

• Positive measures of quality - Must lift key engagement metric - Minimum required CTR rates

• Negative measures of quality - App deletions - Spam reports - Unsubscribes

34

Experiment Segmentation

35

All Users

36

Marginal Users (~1 month)

37

Core Users (multiple times a week)

38

Pre Product/Market Fit

Focus on Product First

40

Initial Traction

Do Things That Don’t Scale

42

Growth Stage

Prioritize Based on ROI

44

Wrap Up• Use funnels for analyzing acquisition flows

• 1d7s & cohort heatmaps to analyze on activation

• Use user states to understand how engaged users are

• Make sure emails/notifications for true engagement

• Use segmentation for experiments & metrics in general

• Product comes first

• Do things that don’t scale

• Prioritize projects based on return on investment

45

John EganGrowth Blog: jwegan.com

Email: me@jwegan.com

Twitter: @jwegan_com

Pinterest: pinterest.com/jwegan

46