experimentation platforms: build or buy?
DESCRIPTION
Experimentation Platforms: Build or Buy?. RecSys 2014. Mike Lo, PayPal. Hello!. Product Test Ops Merchant Testing Knowledge Mgt and Insights Experimentation Analytics. Mike Lo, Sr. Manager - Experimentation, PayPal. PayPal. Experimentation Program. - PowerPoint PPT PresentationTRANSCRIPT
© 2014 PayPal Inc. All rights reserved. Confidential and proprietary.
Experimentation Platforms: Build or Buy?RecSys 2014Mike Lo, PayPal
© 2014 PayPal Inc. All rights reserved. Confidential and proprietary.
2
• Product
• Test Ops
• Merchant Testing
• Knowledge Mgt and Insights
• Experimentation Analytics
Hello!Mike Lo, Sr. Manager - Experimentation, PayPal
PayPal
Experimentation Program
• Manage PayPal’s Experimentation Program
• Formerly part of eBay’s Program
• Build or Buy?
• We did both at PayPal
Expectations for today’s talk
Focus on Experimentation Platform
• PayPal’s ‘Testing Story’
• Some comparison with eBay program.
© 2014 PayPal Inc. All rights reserved. Confidential and proprietary.
3
Words words words
Big DATA
AN
ALY
TIC
S
Statistics
Ag
ile Mobile
Beta
Ramp
InsightsR
and
om
Alp
ha
© 2014 PayPal Inc. All rights reserved. Confidential and proprietary.
4
Becoming Data-Driven is difficult!It doesn’t happen overnight. It took us years.
Product
Process
People
© 2014 PayPal Inc. All rights reserved. Confidential and proprietary.
5
We bought.
© 2014 PayPal Inc. All rights reserved. Confidential and proprietary.
6
Technology Adoption eBay-focused
PayPal = eBay right? What about the existing platform?Rationale why we didn’t implement the eBay Experimentation Platform
• PayPal’s tech stack
• Fix Tracking
• Can’t wait
• Quick wins
• Endorsement
• Program, not the Product.
• Built ground up for eBay
• Tightly coupled with eBay systems
• Need to decouple features
© 2014 PayPal Inc. All rights reserved. Confidential and proprietary.
7
Outcome?Ran experiments successfully for 12 months… then …
Couldn’t Scale
Had to cut the cord
Result: Implemented eBay Platform
© 2014 PayPal Inc. All rights reserved. Confidential and proprietary.
8
No.
It’s all about the TESTING NEEDS of your organization.
So are you advocating to build rather than buy?
© 2014 PayPal Inc. All rights reserved. Confidential and proprietary.
9
No Product Tests Keep it separate? PayPal’s Needs
What happened?Like many companies, the need started from Marketing but didn’t end there.
• Product Pages
Marketing Pages Product Pages
• Testing different content variants • Testing different feature variants
• Technical Limitations • Is there a clean distinction? • 20 experiments is nice. But how
do we get to 2,000?
© 2014 PayPal Inc. All rights reserved. Confidential and proprietary.
10
Why it made sense for PayPal to build instead of buy3 main reasons PayPal is investing to build
1 Analytical
2 Technical
3 Scale
© 2014 PayPal Inc. All rights reserved. Confidential and proprietary.
11
1 - High Fidelity AnalyticsOur site is an ecosystem
Single Enterprise Testing Solution
Scientific rigor in the results
Other sites have natural ‘fences’
© 2014 PayPal Inc. All rights reserved. Confidential and proprietary.
12
Agile Product Lifecycle Who does your dev? Look beyond
2 - Technical agilityIt’s all about going Agile!
• PayPal’s tech stack
• Master of your domain
• Data
• Site speed performance
• Rolling Full-Bright
Test Plan Design Code Validate
Smoke Test
• Coupled with PDLC
• Code once, push in minutes
• Allows MVT
Biz Req Product Req
Eng Req QA Release
Alpha Beta AB Test Collect Analyze Ramp
© 2014 PayPal Inc. All rights reserved. Confidential and proprietary.
13
Volume Sophistication Deeper Insights
3- Scaling to the next levelUtilizing the platform to its fullest potential
• Thousands of experiments
• Patented mechanism
• Utilizing PayPal’s platforms • Entire data pipeline
• Automated design
• Real-time analytics
• Data modeling
© 2014 PayPal Inc. All rights reserved. Confidential and proprietary.
14
Beware2 potential unforeseen effects of building a robust platform, if left un-checked…
Platform is reduced to a CMS tool
Ramping / Throttling / Release tool
© 2014 PayPal Inc. All rights reserved. Confidential and proprietary.
15
If you build it, will they come? Again, it is not just about the Product… follow through with Adoption!
Product
Process
People
© 2014 PayPal Inc. All rights reserved. Confidential and proprietary.
16
Questions?Thanks for listening!