netflix’s recommendation ml pipeline using apache spark · netflix’s recommendation ml pipeline...
TRANSCRIPT
![Page 1: Netflix’s Recommendation ML Pipeline using Apache Spark · Netflix’s Recommendation ML Pipeline using Apache Spark DB Tsai Spark Summit ... from our recommendations Recommendations](https://reader033.vdocument.in/reader033/viewer/2022053004/5f07f4f87e708231d41f9bb6/html5/thumbnails/1.jpg)
Netflix’s RecommendationML Pipeline using Apache Spark
DB TsaiSpark Summit East - Feb 8, 2017
![Page 2: Netflix’s Recommendation ML Pipeline using Apache Spark · Netflix’s Recommendation ML Pipeline using Apache Spark DB Tsai Spark Summit ... from our recommendations Recommendations](https://reader033.vdocument.in/reader033/viewer/2022053004/5f07f4f87e708231d41f9bb6/html5/thumbnails/2.jpg)
At Netflix, we use ML everywhere
![Page 3: Netflix’s Recommendation ML Pipeline using Apache Spark · Netflix’s Recommendation ML Pipeline using Apache Spark DB Tsai Spark Summit ... from our recommendations Recommendations](https://reader033.vdocument.in/reader033/viewer/2022053004/5f07f4f87e708231d41f9bb6/html5/thumbnails/3.jpg)
Everything is a Recommendation
![Page 4: Netflix’s Recommendation ML Pipeline using Apache Spark · Netflix’s Recommendation ML Pipeline using Apache Spark DB Tsai Spark Summit ... from our recommendations Recommendations](https://reader033.vdocument.in/reader033/viewer/2022053004/5f07f4f87e708231d41f9bb6/html5/thumbnails/4.jpg)
Over 80% of what members watch comes from our recommendations
Recommendations are driven by Machine Learning Algorithms
![Page 5: Netflix’s Recommendation ML Pipeline using Apache Spark · Netflix’s Recommendation ML Pipeline using Apache Spark DB Tsai Spark Summit ... from our recommendations Recommendations](https://reader033.vdocument.in/reader033/viewer/2022053004/5f07f4f87e708231d41f9bb6/html5/thumbnails/5.jpg)
Jan 6th, 2016
![Page 6: Netflix’s Recommendation ML Pipeline using Apache Spark · Netflix’s Recommendation ML Pipeline using Apache Spark DB Tsai Spark Summit ... from our recommendations Recommendations](https://reader033.vdocument.in/reader033/viewer/2022053004/5f07f4f87e708231d41f9bb6/html5/thumbnails/6.jpg)
#NetflixEverywhere
● 93+ Million Members
● 190+ Countries
● 125+ Million streaming hours / day
● 1000 hours of Original content in 2017
● ⅓ of US internet traffic during evenings
![Page 7: Netflix’s Recommendation ML Pipeline using Apache Spark · Netflix’s Recommendation ML Pipeline using Apache Spark DB Tsai Spark Summit ... from our recommendations Recommendations](https://reader033.vdocument.in/reader033/viewer/2022053004/5f07f4f87e708231d41f9bb6/html5/thumbnails/7.jpg)
Constantly Innovating through A/B tests
![Page 8: Netflix’s Recommendation ML Pipeline using Apache Spark · Netflix’s Recommendation ML Pipeline using Apache Spark DB Tsai Spark Summit ... from our recommendations Recommendations](https://reader033.vdocument.in/reader033/viewer/2022053004/5f07f4f87e708231d41f9bb6/html5/thumbnails/8.jpg)
Try an idea offline using historical data to see if they would have made better recommendations
If it does, deploy a live A/B test to see if it performs well in Production
Data Driven
![Page 9: Netflix’s Recommendation ML Pipeline using Apache Spark · Netflix’s Recommendation ML Pipeline using Apache Spark DB Tsai Spark Summit ... from our recommendations Recommendations](https://reader033.vdocument.in/reader033/viewer/2022053004/5f07f4f87e708231d41f9bb6/html5/thumbnails/9.jpg)
Running an Experiment
Design Experiment
Collect Label Dataset
DeLorean: Offline Feature Generation
Distributed Model Training
Parallel training of individual
models using different
executors
Compute Validation Metrics
Model Testing
Choose best model
Design a New Experiment to Test Out Different Ideas
GoodMetrics
Offline Experiment
OnlineA/B Test
Online AB Testing
Bad Metrics
Selected Contexts
![Page 10: Netflix’s Recommendation ML Pipeline using Apache Spark · Netflix’s Recommendation ML Pipeline using Apache Spark DB Tsai Spark Summit ... from our recommendations Recommendations](https://reader033.vdocument.in/reader033/viewer/2022053004/5f07f4f87e708231d41f9bb6/html5/thumbnails/10.jpg)
We use a standardized data format across multiple ranking pipelines
This standardized data format is used by common tooling, libraries, and algorithms
![Page 11: Netflix’s Recommendation ML Pipeline using Apache Spark · Netflix’s Recommendation ML Pipeline using Apache Spark DB Tsai Spark Summit ... from our recommendations Recommendations](https://reader033.vdocument.in/reader033/viewer/2022053004/5f07f4f87e708231d41f9bb6/html5/thumbnails/11.jpg)
Contexts: The setting for evaluating a set of items (e.g. tuples of member profiles, country, time, device, etc.)
Items: The elements to be trained on, scored, and/or ranked (e.g. videos, rows, search entities)
Labels: For supervised learning, this will be the label (target) for each item
Ranking problems
![Page 12: Netflix’s Recommendation ML Pipeline using Apache Spark · Netflix’s Recommendation ML Pipeline using Apache Spark DB Tsai Spark Summit ... from our recommendations Recommendations](https://reader033.vdocument.in/reader033/viewer/2022053004/5f07f4f87e708231d41f9bb6/html5/thumbnails/12.jpg)
root |-- profile_id: long (nullable = false) |-- country_iso_code: string (nullable = false) |-- items: array (nullable = true) | |-- element: struct (containsNull = false) | | |-- show_title_id: long (nullable = false) | | |-- label: double (nullable = false) | | |-- weight: double (nullable = false)| | |-- features: struct (nullable = false) | | | |-- feature1: double (nullable = false) | | | |-- feature2: double (nullable = false) | | | |-- feature3: double (nullable = false)
DeLorean Data Format a.k.a DMC-12
![Page 13: Netflix’s Recommendation ML Pipeline using Apache Spark · Netflix’s Recommendation ML Pipeline using Apache Spark DB Tsai Spark Summit ... from our recommendations Recommendations](https://reader033.vdocument.in/reader033/viewer/2022053004/5f07f4f87e708231d41f9bb6/html5/thumbnails/13.jpg)
The nested data structure avoids an expensive shuffle when ranking
The features are derived from Netflix data or the output of other trained models
The features are persisted in HIVE using Parquet
Ensemble methods are used to build rankers
![Page 14: Netflix’s Recommendation ML Pipeline using Apache Spark · Netflix’s Recommendation ML Pipeline using Apache Spark DB Tsai Spark Summit ... from our recommendations Recommendations](https://reader033.vdocument.in/reader033/viewer/2022053004/5f07f4f87e708231d41f9bb6/html5/thumbnails/14.jpg)
Transformer
https://en.wikipedia.org/wiki/Spark_(Transformers)
![Page 15: Netflix’s Recommendation ML Pipeline using Apache Spark · Netflix’s Recommendation ML Pipeline using Apache Spark DB Tsai Spark Summit ... from our recommendations Recommendations](https://reader033.vdocument.in/reader033/viewer/2022053004/5f07f4f87e708231d41f9bb6/html5/thumbnails/15.jpg)
Transformer takes an input DataFrame and “lazily” returns an output DataFrame
Item Transformer ● Extends Spark ML’s Transformer● Accepts DMC-12 DataFrame with contextual
information● Transforms DataFrame at the item level
![Page 16: Netflix’s Recommendation ML Pipeline using Apache Spark · Netflix’s Recommendation ML Pipeline using Apache Spark DB Tsai Spark Summit ... from our recommendations Recommendations](https://reader033.vdocument.in/reader033/viewer/2022053004/5f07f4f87e708231d41f9bb6/html5/thumbnails/16.jpg)
Why DataFrame?
Catalyst Optimizations
Up-front Schema Verification
We found a 4x speedup during feature generation by migrating from RDD-based implementation to DataFrame implementation
![Page 17: Netflix’s Recommendation ML Pipeline using Apache Spark · Netflix’s Recommendation ML Pipeline using Apache Spark DB Tsai Spark Summit ... from our recommendations Recommendations](https://reader033.vdocument.in/reader033/viewer/2022053004/5f07f4f87e708231d41f9bb6/html5/thumbnails/17.jpg)
Negative Generator
Facts
root |-- profile_id: long (nullable = false) |-- country_iso_code: string (nullable = false) |-- items: array (nullable = true) | |-- element: struct (containsNull = false) | | |-- show_title_id: long (nullable = false) | | |-- label: double (nullable = false) | | |-- weight: double (nullable = false)
Facts with synthetic negatives
root |-- profile_id: long (nullable = false) |-- country_iso_code: string (nullable = false) |-- items: array (nullable = true) | |-- element: struct (containsNull = false) | | |-- show_title_id: long (nullable = false) | | |-- label: double (nullable = false) | | |-- weight: double (nullable = false)
Creating negatives from what member plays for supervised learning
![Page 18: Netflix’s Recommendation ML Pipeline using Apache Spark · Netflix’s Recommendation ML Pipeline using Apache Spark DB Tsai Spark Summit ... from our recommendations Recommendations](https://reader033.vdocument.in/reader033/viewer/2022053004/5f07f4f87e708231d41f9bb6/html5/thumbnails/18.jpg)
DeLorean Feature Generator
root |-- profile_id: long (nullable = false) |-- country_iso_code: string (nullable = false) |-- items: array (nullable = true) | |-- element: struct (containsNull = false) | | |-- show_title_id: long (nullable = false) | | |-- label: double (nullable = false) | | |-- weight: double (nullable = false)
root |-- profile_id: long (nullable = false) |-- country_iso_code: string (nullable = false) |-- items: array (nullable = true) | |-- element: struct (containsNull = false) | | |-- show_title_id: long (nullable = false) | | |-- label: double (nullable = false) | | |-- weight: double (nullable = false)| | |-- features: struct (nullable = false)| | | |-- feature1: double (nullable = false) | | | |-- feature2: double (nullable = false) | | | |-- feature3: double (nullable = false)
Creating features based on common code base in offline and online system
http://techblog.netflix.com/2016/02/distributed-time-travel-for-feature.html
![Page 19: Netflix’s Recommendation ML Pipeline using Apache Spark · Netflix’s Recommendation ML Pipeline using Apache Spark DB Tsai Spark Summit ... from our recommendations Recommendations](https://reader033.vdocument.in/reader033/viewer/2022053004/5f07f4f87e708231d41f9bb6/html5/thumbnails/19.jpg)
Creating the Dataset for Algorithms
![Page 20: Netflix’s Recommendation ML Pipeline using Apache Spark · Netflix’s Recommendation ML Pipeline using Apache Spark DB Tsai Spark Summit ... from our recommendations Recommendations](https://reader033.vdocument.in/reader033/viewer/2022053004/5f07f4f87e708231d41f9bb6/html5/thumbnails/20.jpg)
Multithreading Model Training
For single machine multi-threading algorithms, we allocate one task to one machine. Multiple tasks are running in Spark for different parameters
Broadcast in Spark has datasize limitation, we write data into HDFS, and stream the data into the trainers in executors which run single-machine multi-threading algorithms
![Page 21: Netflix’s Recommendation ML Pipeline using Apache Spark · Netflix’s Recommendation ML Pipeline using Apache Spark DB Tsai Spark Summit ... from our recommendations Recommendations](https://reader033.vdocument.in/reader033/viewer/2022053004/5f07f4f87e708231d41f9bb6/html5/thumbnails/21.jpg)
Distributed Model Training
We use both Spark ML’s algorithms and in-house ML implementations
We keep the interface similar for both multi-threading and distributed algorithms, so experimenters can try different ideas easily
![Page 22: Netflix’s Recommendation ML Pipeline using Apache Spark · Netflix’s Recommendation ML Pipeline using Apache Spark DB Tsai Spark Summit ... from our recommendations Recommendations](https://reader033.vdocument.in/reader033/viewer/2022053004/5f07f4f87e708231d41f9bb6/html5/thumbnails/22.jpg)
Scoring and RankingScorer is also a Transformer returned from the Trainer
Multiple models can be scored at the same time in parallel
The ranks are derived from sorted scores
Together with labels, we can compute metrics, NMRR, NDCG, and Recall, etc
root |-- profile_id: long (nullable = false) |-- country_iso_code: string (nullable = false) |-- items: array (nullable = true) | |-- element: struct (containsNull = false) | | |-- show_title_id: long (nullable = false) | | |-- label: double (nullable = false) | | |-- weight: double (nullable = false)| | |-- features: struct (nullable = false)| | | |-- feature1: double (nullable = false) | | | |-- feature2: double (nullable = false) | | | |-- feature3: double (nullable = false)| | |-- scores: struct (nullable = false)| | | |-- model1: double false| | | |-- model2: double false
![Page 23: Netflix’s Recommendation ML Pipeline using Apache Spark · Netflix’s Recommendation ML Pipeline using Apache Spark DB Tsai Spark Summit ... from our recommendations Recommendations](https://reader033.vdocument.in/reader033/viewer/2022053004/5f07f4f87e708231d41f9bb6/html5/thumbnails/23.jpg)
Lessons Learned - Pipeline Abstraction
Pros
● Modularity + Tests● Plug-N-Play● Notebook Prototyping● Customizability● Schema Verification● Serializability (AC)
Cons
● Dependent on Spark platform● Not easy to bring to production● Default Metric Evaluator doesn’t
support ranking multiple models and type of metrics in one pass
![Page 24: Netflix’s Recommendation ML Pipeline using Apache Spark · Netflix’s Recommendation ML Pipeline using Apache Spark DB Tsai Spark Summit ... from our recommendations Recommendations](https://reader033.vdocument.in/reader033/viewer/2022053004/5f07f4f87e708231d41f9bb6/html5/thumbnails/24.jpg)