Download - Experimentation Platform at Netflix
Experimentation Platform at Netflix
What is Experimentation?• It’s the process of randomly dividing users into groups– Control Group, existing experience/behavior– Variant 1…n, one or more new experiences– Gather behavior and core metrics for all variants– Analyze and evaluate hypothesis by p-value
• Also known as AB Testing• Examples– User interface changes, new product features, changes
personalization algorithms etc.
Random allocation to experiments
Experiment 1, Variant A Invariant: Control
Netflix Customers
Experiment 2, Variant E Experiment ‘n’, Variant M
………
………
Why?
• Data driven product innovation– http
://www.quora.com/What-types-of-things-does-Netflix-A-B-test-aside-from-member-sign-up
– http://www.hakkalabs.co/articles/hive-controlled-experimentation-2
• Iteration of ideas in a controlled way– Validate features and presentation based on data– Fail fast
• It’s the culture
User interface Experiment
Variant 2 Variant 3
New feature (Profiles) Experiment
Variant 2
Variant 3
Variant 4
Personalization Algorithm Experiment
No user interface change
Variant 1: Control - No change
Variant 2: Movies and TV shows
Variant 3: TV shows only
Cassandra
Magma UI
Stratification
Allocation engine
Segmentation rules
Event stream
Hive
Experimentation Platform
Terradata
HTTP
memcached
Clients
Allocation engine• Responsible for assigning customers to a variant in the
experiment• Randomly distribute customers across variants• Stratified Sampling– Bias or variance reduction– Varying sample size across platforms for e.g. Game console vs
Blu-ray players vs Laptops• Start, Stop and Track allocations in near real-time
Segmentation• Divide a broad target population into subsets with
similar properties for e.g. customers who have not used a Tablet to access Netflix in the last ‘n’ days
• Have enough data to execute real-time• Achieve scale and maintain low latency• Clean data for analysis
Analysis• Gather core metrics and behavior data
– Goal: Near-time (~15 mins) data from all data sources• Petabytes of multi-dimensional data
– Pre vs post compute/aggregate processing– Adjust for biases and/or incidents– Interactive analysis within given constraints
• Explore for patterns• Scale to support fast growing business and the big datasets
Magma UI
• Experiment lifecycle management– Hypothesis, experiment variants– Real-time rule-based segmentation
• Define rules and conditions to identify the right population for experiments• Rules are dynamic and applied real-time
– Scheduling and forecasting
• Insights– Near real-time insights
• Analysis– Interactive analysis of petabytes of data
• Dimension filters• Behavior filters
– Data visualization (trends, behaviors etc)
Some Challenges• How do we efficiently scale the system• How do we continue to operate at low latencies, given
– Customer segmentation is based on real-time activity– Real-time allocations have broad applications– Data distributed across multiple clusters
• Big data (In petabytes) processing– Billions of rows of data from various sources– Efficient management of huge datasets to support interactive analysis– Rich and flexible filtering to support interactive analysis
• Rich forecasting and insights• Resiliency• And more…
Large scale and growing
We’re hiring• Work on large scale, big data and distributed systems
– Distributed systems – Server-side engineers• Data Structures & Algorithms• Concurrency, Multi-threading, Caching
– Data systems – Server-side engineers• Data Structures & Algorithms• Distributed data stores, Experience processing very large (petabytes) datasets
– UI engineers – Data management and visualization• CSS/JavaScript• Previous experience in data visualization
• Contact Us – [email protected]– bit.ly/ExperimentationAtNetflix
– Netflix has unique culture. Read about it here.