why everything is an a/b test at pinterest
TRANSCRIPT
![Page 1: Why everything is an A/B Test at Pinterest](https://reader035.vdocument.in/reader035/viewer/2022062515/55cfbf68bb61eb0d3d8b4621/html5/thumbnails/1.jpg)
Krishna Gade
Why everything is an A/B test at Pinterest
![Page 2: Why everything is an A/B Test at Pinterest](https://reader035.vdocument.in/reader035/viewer/2022062515/55cfbf68bb61eb0d3d8b4621/html5/thumbnails/2.jpg)
Confidential
2
![Page 3: Why everything is an A/B Test at Pinterest](https://reader035.vdocument.in/reader035/viewer/2022062515/55cfbf68bb61eb0d3d8b4621/html5/thumbnails/3.jpg)
What is Pinterest?
A visual bookmarking tool
Discover an inspiring ideaSave it to a board
Go do it
![Page 4: Why everything is an A/B Test at Pinterest](https://reader035.vdocument.in/reader035/viewer/2022062515/55cfbf68bb61eb0d3d8b4621/html5/thumbnails/4.jpg)
Pinterest is a data product
![Page 5: Why everything is an A/B Test at Pinterest](https://reader035.vdocument.in/reader035/viewer/2022062515/55cfbf68bb61eb0d3d8b4621/html5/thumbnails/5.jpg)
![Page 6: Why everything is an A/B Test at Pinterest](https://reader035.vdocument.in/reader035/viewer/2022062515/55cfbf68bb61eb0d3d8b4621/html5/thumbnails/6.jpg)
Data Flow
App
events
Kafka
SecorSkyline
Pinball
Redshift
Pinalytics
Features
Qubole (Hadoop)
Singer
![Page 7: Why everything is an A/B Test at Pinterest](https://reader035.vdocument.in/reader035/viewer/2022062515/55cfbf68bb61eb0d3d8b4621/html5/thumbnails/7.jpg)
Confidential
Metrics
7
![Page 8: Why everything is an A/B Test at Pinterest](https://reader035.vdocument.in/reader035/viewer/2022062515/55cfbf68bb61eb0d3d8b4621/html5/thumbnails/8.jpg)
Confidential
User Lifecycle
New Users
Active Users
Dormant Users
Acquisition
Activation
Churn
Churn Resurrection
![Page 9: Why everything is an A/B Test at Pinterest](https://reader035.vdocument.in/reader035/viewer/2022062515/55cfbf68bb61eb0d3d8b4621/html5/thumbnails/9.jpg)
Confidential
Terminology
● MAU – Monthly Active User● WAU – Weekly Active User● DAU – Daily Active User● WAR – Weekly Active Repinner● WAC – Weekly Active Clickthrougher● WARC – Weekly Active Repinner or Clickthrougher● 1d7 – User who comes back at-least once in 7 days after
signup● 1rc7 - User who repins/clicks at-least once in 7 days
after signup
![Page 10: Why everything is an A/B Test at Pinterest](https://reader035.vdocument.in/reader035/viewer/2022062515/55cfbf68bb61eb0d3d8b4621/html5/thumbnails/10.jpg)
Confidential
MAUs
● Current progress to quarterly goal● MAU forecast● MAUs by app/gender/country● MAU Accounting
![Page 11: Why everything is an A/B Test at Pinterest](https://reader035.vdocument.in/reader035/viewer/2022062515/55cfbf68bb61eb0d3d8b4621/html5/thumbnails/11.jpg)
Confidential
MAU Accounting
![Page 12: Why everything is an A/B Test at Pinterest](https://reader035.vdocument.in/reader035/viewer/2022062515/55cfbf68bb61eb0d3d8b4621/html5/thumbnails/12.jpg)
Confidential
Acquisition Metrics
● Total Signups● Signups by App● Signups by Referrer● Invites Sent● Unique Invite Senders● Invite Signups
![Page 13: Why everything is an A/B Test at Pinterest](https://reader035.vdocument.in/reader035/viewer/2022062515/55cfbf68bb61eb0d3d8b4621/html5/thumbnails/13.jpg)
Confidential
Activation Metrics
Activation Rate = 1rc7/signups
● Activation Rate by App/Gender/Referrer● Overall signups to 1rc7● Signup engagement funnel● Signups to WAU 35 days after signup
![Page 14: Why everything is an A/B Test at Pinterest](https://reader035.vdocument.in/reader035/viewer/2022062515/55cfbf68bb61eb0d3d8b4621/html5/thumbnails/14.jpg)
Confidential
Engagement Metrics● *AU ratios
● # of Emails sent● # of Emails opened● # of Emails clicked● # of Notifications sent● # of Notifications clicked
![Page 15: Why everything is an A/B Test at Pinterest](https://reader035.vdocument.in/reader035/viewer/2022062515/55cfbf68bb61eb0d3d8b4621/html5/thumbnails/15.jpg)
Confidential
Resurrection Metrics
● # of Users who weredormant for 28+ daysbut came back, split byplatform, referrer etc
![Page 16: Why everything is an A/B Test at Pinterest](https://reader035.vdocument.in/reader035/viewer/2022062515/55cfbf68bb61eb0d3d8b4621/html5/thumbnails/16.jpg)
Confidential
Takeaways
● Emphasize on activation (process of converting a new user to MAU)
● Activation critical to long-term growth● Deeply understand how different
segments of users interact
![Page 17: Why everything is an A/B Test at Pinterest](https://reader035.vdocument.in/reader035/viewer/2022062515/55cfbf68bb61eb0d3d8b4621/html5/thumbnails/17.jpg)
Confidential
Automated Metrics
17
![Page 18: Why everything is an A/B Test at Pinterest](https://reader035.vdocument.in/reader035/viewer/2022062515/55cfbf68bb61eb0d3d8b4621/html5/thumbnails/18.jpg)
Confidential
![Page 19: Why everything is an A/B Test at Pinterest](https://reader035.vdocument.in/reader035/viewer/2022062515/55cfbf68bb61eb0d3d8b4621/html5/thumbnails/19.jpg)
Confidential
Chart creation
![Page 20: Why everything is an A/B Test at Pinterest](https://reader035.vdocument.in/reader035/viewer/2022062515/55cfbf68bb61eb0d3d8b4621/html5/thumbnails/20.jpg)
Confidential
Reuse existing query result
![Page 21: Why everything is an A/B Test at Pinterest](https://reader035.vdocument.in/reader035/viewer/2022062515/55cfbf68bb61eb0d3d8b4621/html5/thumbnails/21.jpg)
Confidential
Metrics creation
![Page 22: Why everything is an A/B Test at Pinterest](https://reader035.vdocument.in/reader035/viewer/2022062515/55cfbf68bb61eb0d3d8b4621/html5/thumbnails/22.jpg)
Confidential
A/B Experimentation
22
![Page 23: Why everything is an A/B Test at Pinterest](https://reader035.vdocument.in/reader035/viewer/2022062515/55cfbf68bb61eb0d3d8b4621/html5/thumbnails/23.jpg)
Confidential
Categories of Experiments
● A/B testing new features● Copy testing● SEO changes● Infrastructure updates
![Page 24: Why everything is an A/B Test at Pinterest](https://reader035.vdocument.in/reader035/viewer/2022062515/55cfbf68bb61eb0d3d8b4621/html5/thumbnails/24.jpg)
Confidential
A/B Testing Framework
![Page 25: Why everything is an A/B Test at Pinterest](https://reader035.vdocument.in/reader035/viewer/2022062515/55cfbf68bb61eb0d3d8b4621/html5/thumbnails/25.jpg)
Confidential
![Page 26: Why everything is an A/B Test at Pinterest](https://reader035.vdocument.in/reader035/viewer/2022062515/55cfbf68bb61eb0d3d8b4621/html5/thumbnails/26.jpg)
Confidential
![Page 27: Why everything is an A/B Test at Pinterest](https://reader035.vdocument.in/reader035/viewer/2022062515/55cfbf68bb61eb0d3d8b4621/html5/thumbnails/27.jpg)
Confidential
![Page 28: Why everything is an A/B Test at Pinterest](https://reader035.vdocument.in/reader035/viewer/2022062515/55cfbf68bb61eb0d3d8b4621/html5/thumbnails/28.jpg)
Confidential
A/B Testing Dashboard
![Page 29: Why everything is an A/B Test at Pinterest](https://reader035.vdocument.in/reader035/viewer/2022062515/55cfbf68bb61eb0d3d8b4621/html5/thumbnails/29.jpg)
Confidential
![Page 30: Why everything is an A/B Test at Pinterest](https://reader035.vdocument.in/reader035/viewer/2022062515/55cfbf68bb61eb0d3d8b4621/html5/thumbnails/30.jpg)
Confidential
p-value (via chi-squared test)
Statistical tests for validation
![Page 31: Why everything is an A/B Test at Pinterest](https://reader035.vdocument.in/reader035/viewer/2022062515/55cfbf68bb61eb0d3d8b4621/html5/thumbnails/31.jpg)
Confidential
Impact of Badging
![Page 32: Why everything is an A/B Test at Pinterest](https://reader035.vdocument.in/reader035/viewer/2022062515/55cfbf68bb61eb0d3d8b4621/html5/thumbnails/32.jpg)
Confidential
Impact of Badging
● 7% lift in DAU● 2.5% lift over long term● Effective at increasingengagement levels
![Page 33: Why everything is an A/B Test at Pinterest](https://reader035.vdocument.in/reader035/viewer/2022062515/55cfbf68bb61eb0d3d8b4621/html5/thumbnails/33.jpg)
Confidential
Copy Testing
Definition: Testing different variations of text we display to the user.
● A/B Testing copy was a tedious process● Even when we did this we were optimizing
only for English
![Page 34: Why everything is an A/B Test at Pinterest](https://reader035.vdocument.in/reader035/viewer/2022062515/55cfbf68bb61eb0d3d8b4621/html5/thumbnails/34.jpg)
Confidential
New Approach
● Ability to run a copy test w/o changing a line of code.
● Support for translating text dynamically● Use Multi-armed bandits to converge to showing
best performing text
![Page 35: Why everything is an A/B Test at Pinterest](https://reader035.vdocument.in/reader035/viewer/2022062515/55cfbf68bb61eb0d3d8b4621/html5/thumbnails/35.jpg)
Confidential
Multi Armed Bandits
![Page 36: Why everything is an A/B Test at Pinterest](https://reader035.vdocument.in/reader035/viewer/2022062515/55cfbf68bb61eb0d3d8b4621/html5/thumbnails/36.jpg)
Confidential
SEO
36
50+ Billion Pinscategorized by people
into more than
1 Billion Boards
![Page 37: Why everything is an A/B Test at Pinterest](https://reader035.vdocument.in/reader035/viewer/2022062515/55cfbf68bb61eb0d3d8b4621/html5/thumbnails/37.jpg)
Confidential
37
What makes SEO so challenging?
What is the Impact of the new SEO feature?
Will this new change affect SEO?
Uhhh...Dunno...
Uhhh...Ask Google.
![Page 38: Why everything is an A/B Test at Pinterest](https://reader035.vdocument.in/reader035/viewer/2022062515/55cfbf68bb61eb0d3d8b4621/html5/thumbnails/38.jpg)
Confidential
KEEPCALM
and
RUN ANSEO EXPERIMENT
![Page 39: Why everything is an A/B Test at Pinterest](https://reader035.vdocument.in/reader035/viewer/2022062515/55cfbf68bb61eb0d3d8b4621/html5/thumbnails/39.jpg)
Confidential
What is an SEO experiment?
39
Regular AB Experiment SEO Experiment
Metrics User behavior Search engine referred traffic
Unique id for hashing userid / sessionid page url
![Page 40: Why everything is an A/B Test at Pinterest](https://reader035.vdocument.in/reader035/viewer/2022062515/55cfbf68bb61eb0d3d8b4621/html5/thumbnails/40.jpg)
Confidential
What is an SEO experiment?
Launch
● Measure the gap ● Before vs After launch
![Page 41: Why everything is an A/B Test at Pinterest](https://reader035.vdocument.in/reader035/viewer/2022062515/55cfbf68bb61eb0d3d8b4621/html5/thumbnails/41.jpg)
Confidential
Dashboard
41
DIFF = enabled - control
Traffic Gain
![Page 42: Why everything is an A/B Test at Pinterest](https://reader035.vdocument.in/reader035/viewer/2022062515/55cfbf68bb61eb0d3d8b4621/html5/thumbnails/42.jpg)
Confidential
42
Decision Making 1 2
3
4
SHIP!
SHIP!
SHUT DOWN!VALIDATE
![Page 43: Why everything is an A/B Test at Pinterest](https://reader035.vdocument.in/reader035/viewer/2022062515/55cfbf68bb61eb0d3d8b4621/html5/thumbnails/43.jpg)
Confidential
43
Experiment Driven SEO
What is the Impact of the new SEO feature?
We got XX% of traffic growth from it! Can we use more resources to make bigger impact?
No, the experiment result is flat, so let’s ship it.
SEO experiment
SEO experiment
Is my UI change affecting SEO negatively?
![Page 44: Why everything is an A/B Test at Pinterest](https://reader035.vdocument.in/reader035/viewer/2022062515/55cfbf68bb61eb0d3d8b4621/html5/thumbnails/44.jpg)
Confidential
Why you should be A/B testing even your infrastructure
![Page 45: Why everything is an A/B Test at Pinterest](https://reader035.vdocument.in/reader035/viewer/2022062515/55cfbf68bb61eb0d3d8b4621/html5/thumbnails/45.jpg)
Confidential
Infrastructure Changes
● 2013: A new web framework● 2014: Pyapns● 2015: User service
![Page 46: Why everything is an A/B Test at Pinterest](https://reader035.vdocument.in/reader035/viewer/2022062515/55cfbf68bb61eb0d3d8b4621/html5/thumbnails/46.jpg)
Confidential
Experiment results for replacing PyAPNs
![Page 47: Why everything is an A/B Test at Pinterest](https://reader035.vdocument.in/reader035/viewer/2022062515/55cfbf68bb61eb0d3d8b4621/html5/thumbnails/47.jpg)
Confidential
Conclusions
● Figure out what success means● Define your metrics● Automate metric computation● A/B test everything!
![Page 48: Why everything is an A/B Test at Pinterest](https://reader035.vdocument.in/reader035/viewer/2022062515/55cfbf68bb61eb0d3d8b4621/html5/thumbnails/48.jpg)
Confidential
Thank You!