netflix: streaming entertainment to 200 million members around … · 2021. 2. 23. · netflix:...
TRANSCRIPT
![Page 1: Netflix: Streaming Entertainment to 200 Million Members Around … · 2021. 2. 23. · Netflix: Streaming Entertainment USENIX FAST ‘21 Open Connect is Netflix’s content delivery](https://reader036.vdocument.in/reader036/viewer/2022071511/6130f4611ecc515869446d4b/html5/thumbnails/1.jpg)
Netflix: Streaming Entertainment to200 Million MembersAround the World
Jonathan LooneyUSENIX FAST ‘21
![Page 2: Netflix: Streaming Entertainment to 200 Million Members Around … · 2021. 2. 23. · Netflix: Streaming Entertainment USENIX FAST ‘21 Open Connect is Netflix’s content delivery](https://reader036.vdocument.in/reader036/viewer/2022071511/6130f4611ecc515869446d4b/html5/thumbnails/2.jpg)
Netflix: Streaming EntertainmentUSENIX FAST ‘21
Open Connect
![Page 3: Netflix: Streaming Entertainment to 200 Million Members Around … · 2021. 2. 23. · Netflix: Streaming Entertainment USENIX FAST ‘21 Open Connect is Netflix’s content delivery](https://reader036.vdocument.in/reader036/viewer/2022071511/6130f4611ecc515869446d4b/html5/thumbnails/3.jpg)
Netflix: Streaming EntertainmentUSENIX FAST ‘21
Open Connect is Netflix’scontent delivery network.It is global, widely distributed, efficient, and purpose-built for distributing Netflix’s content.
![Page 4: Netflix: Streaming Entertainment to 200 Million Members Around … · 2021. 2. 23. · Netflix: Streaming Entertainment USENIX FAST ‘21 Open Connect is Netflix’s content delivery](https://reader036.vdocument.in/reader036/viewer/2022071511/6130f4611ecc515869446d4b/html5/thumbnails/4.jpg)
CDNCloud Compute
Static startup data
Box Art Video,Audio,
Subtitles
![Page 5: Netflix: Streaming Entertainment to 200 Million Members Around … · 2021. 2. 23. · Netflix: Streaming Entertainment USENIX FAST ‘21 Open Connect is Netflix’s content delivery](https://reader036.vdocument.in/reader036/viewer/2022071511/6130f4611ecc515869446d4b/html5/thumbnails/5.jpg)
Netflix: Streaming EntertainmentUSENIX FAST ‘21
Open Connect deliversstreaming video to over200 million members, deliveringover 100 Tb/s at peak.
![Page 6: Netflix: Streaming Entertainment to 200 Million Members Around … · 2021. 2. 23. · Netflix: Streaming Entertainment USENIX FAST ‘21 Open Connect is Netflix’s content delivery](https://reader036.vdocument.in/reader036/viewer/2022071511/6130f4611ecc515869446d4b/html5/thumbnails/6.jpg)
Netflix: Streaming EntertainmentUSENIX FAST ‘21
The Open Connect ApplianceThe OCA is the workhorse of the Open Connect network.
The OCA almost exclusively runs open-source software, including its OS (FreeBSD).
![Page 7: Netflix: Streaming Entertainment to 200 Million Members Around … · 2021. 2. 23. · Netflix: Streaming Entertainment USENIX FAST ‘21 Open Connect is Netflix’s content delivery](https://reader036.vdocument.in/reader036/viewer/2022071511/6130f4611ecc515869446d4b/html5/thumbnails/7.jpg)
Netflix: Streaming EntertainmentUSENIX FAST ‘21
Typical Netflix OCA Workload
CPU
RAM
Disks
Data Encrypted Data
Network Ports
![Page 8: Netflix: Streaming Entertainment to 200 Million Members Around … · 2021. 2. 23. · Netflix: Streaming Entertainment USENIX FAST ‘21 Open Connect is Netflix’s content delivery](https://reader036.vdocument.in/reader036/viewer/2022071511/6130f4611ecc515869446d4b/html5/thumbnails/8.jpg)
Netflix: Streaming EntertainmentUSENIX FAST ‘21
Using commodity parts, weachieve 180 Gb/s serving TLS-encrypted connectionswith less than 50% CPU on asingle 32-core 2.5-GHz CPU in2 RU.
![Page 9: Netflix: Streaming Entertainment to 200 Million Members Around … · 2021. 2. 23. · Netflix: Streaming Entertainment USENIX FAST ‘21 Open Connect is Netflix’s content delivery](https://reader036.vdocument.in/reader036/viewer/2022071511/6130f4611ecc515869446d4b/html5/thumbnails/9.jpg)
Netflix: Streaming EntertainmentUSENIX FAST ‘21
OCA StorageDetails
![Page 10: Netflix: Streaming Entertainment to 200 Million Members Around … · 2021. 2. 23. · Netflix: Streaming Entertainment USENIX FAST ‘21 Open Connect is Netflix’s content delivery](https://reader036.vdocument.in/reader036/viewer/2022071511/6130f4611ecc515869446d4b/html5/thumbnails/10.jpg)
Netflix: Streaming EntertainmentUSENIX FAST ‘21
Storage Technologies
● Flash○ Moved to NVME attachment
● Spinning drives○ Planning to move to dual actuator
![Page 11: Netflix: Streaming Entertainment to 200 Million Members Around … · 2021. 2. 23. · Netflix: Streaming Entertainment USENIX FAST ‘21 Open Connect is Netflix’s content delivery](https://reader036.vdocument.in/reader036/viewer/2022071511/6130f4611ecc515869446d4b/html5/thumbnails/11.jpg)
Netflix: Streaming EntertainmentUSENIX FAST ‘21
Three Different OCA Flavors forDifferent Workloads
● All-flash appliance (180+ Gb/s)● Large spinning disk/flash combination (up to 80 Gb/s)● Small spinning disk/flash combination (up to 7 Gb/s)
![Page 12: Netflix: Streaming Entertainment to 200 Million Members Around … · 2021. 2. 23. · Netflix: Streaming Entertainment USENIX FAST ‘21 Open Connect is Netflix’s content delivery](https://reader036.vdocument.in/reader036/viewer/2022071511/6130f4611ecc515869446d4b/html5/thumbnails/12.jpg)
Netflix: Streaming EntertainmentUSENIX FAST ‘21
Filesystem Technologies
● UFS○ Used for content drives
● ZFS○ Planning to deploy ZFS for our non-content drives
![Page 13: Netflix: Streaming Entertainment to 200 Million Members Around … · 2021. 2. 23. · Netflix: Streaming Entertainment USENIX FAST ‘21 Open Connect is Netflix’s content delivery](https://reader036.vdocument.in/reader036/viewer/2022071511/6130f4611ecc515869446d4b/html5/thumbnails/13.jpg)
Netflix: Streaming EntertainmentUSENIX FAST ‘21
Some interesting context on data...● Data is “disposable”
○ Multiple redundant copies throughoutthe CDN
● We pre-position almost all content● Content placement is incredibly important for
efficiency○ You need the right number of copies...○ ...in locations close enough to the members…○ ...spread across the right servers…○ ...and across the correct disks in each server
![Page 14: Netflix: Streaming Entertainment to 200 Million Members Around … · 2021. 2. 23. · Netflix: Streaming Entertainment USENIX FAST ‘21 Open Connect is Netflix’s content delivery](https://reader036.vdocument.in/reader036/viewer/2022071511/6130f4611ecc515869446d4b/html5/thumbnails/14.jpg)
Netflix: Streaming EntertainmentUSENIX FAST ‘21
Position on Disk
![Page 15: Netflix: Streaming Entertainment to 200 Million Members Around … · 2021. 2. 23. · Netflix: Streaming Entertainment USENIX FAST ‘21 Open Connect is Netflix’s content delivery](https://reader036.vdocument.in/reader036/viewer/2022071511/6130f4611ecc515869446d4b/html5/thumbnails/15.jpg)
Netflix: Streaming EntertainmentUSENIX FAST ‘21
We try to be smart about disk I/O
● Using readahead● Keeping heavily accessed files in memory● Being careful about mixing reads and writes● Pacing out very disruptive operations like trims
![Page 16: Netflix: Streaming Entertainment to 200 Million Members Around … · 2021. 2. 23. · Netflix: Streaming Entertainment USENIX FAST ‘21 Open Connect is Netflix’s content delivery](https://reader036.vdocument.in/reader036/viewer/2022071511/6130f4611ecc515869446d4b/html5/thumbnails/16.jpg)
Netflix: Streaming EntertainmentUSENIX FAST ‘21
OCA Operating System Optimizations
![Page 17: Netflix: Streaming Entertainment to 200 Million Members Around … · 2021. 2. 23. · Netflix: Streaming Entertainment USENIX FAST ‘21 Open Connect is Netflix’s content delivery](https://reader036.vdocument.in/reader036/viewer/2022071511/6130f4611ecc515869446d4b/html5/thumbnails/17.jpg)
Netflix: Streaming EntertainmentUSENIX FAST ‘21
● Reducing memory bandwidth and/or makingsure things are hot in the cache
● Using hardware offloads to reduce memory bandwidth and/or CPU usage
● Using PCIe bandwidth and/or I/O controller resources efficiently
● Enabling new platforms/designs
Driving efficiency
![Page 18: Netflix: Streaming Entertainment to 200 Million Members Around … · 2021. 2. 23. · Netflix: Streaming Entertainment USENIX FAST ‘21 Open Connect is Netflix’s content delivery](https://reader036.vdocument.in/reader036/viewer/2022071511/6130f4611ecc515869446d4b/html5/thumbnails/18.jpg)
Netflix: Streaming EntertainmentUSENIX FAST ‘21
Reducing memory bandwidth:Async Sendfile
![Page 19: Netflix: Streaming Entertainment to 200 Million Members Around … · 2021. 2. 23. · Netflix: Streaming Entertainment USENIX FAST ‘21 Open Connect is Netflix’s content delivery](https://reader036.vdocument.in/reader036/viewer/2022071511/6130f4611ecc515869446d4b/html5/thumbnails/19.jpg)
Netflix: Streaming EntertainmentUSENIX FAST ‘21
Reducing memory bandwidth:Async Sendfile + KTLS
![Page 20: Netflix: Streaming Entertainment to 200 Million Members Around … · 2021. 2. 23. · Netflix: Streaming Entertainment USENIX FAST ‘21 Open Connect is Netflix’s content delivery](https://reader036.vdocument.in/reader036/viewer/2022071511/6130f4611ecc515869446d4b/html5/thumbnails/20.jpg)
Netflix: Streaming EntertainmentUSENIX FAST ‘21
Reducing memory bandwidth:Optimized data structures
Flags Data Ptr Size Offset ... ... ... ...
State ... ... ... ... ... ... ...
... ... ... ... ... ... Next Ptr ...
Flags Data Ptr Size Offset State Next Ptr ... ...
... ... ... ... ... ... ... ...
... ... ... ... ... ... ... ...
![Page 21: Netflix: Streaming Entertainment to 200 Million Members Around … · 2021. 2. 23. · Netflix: Streaming Entertainment USENIX FAST ‘21 Open Connect is Netflix’s content delivery](https://reader036.vdocument.in/reader036/viewer/2022071511/6130f4611ecc515869446d4b/html5/thumbnails/21.jpg)
Netflix: Streaming EntertainmentUSENIX FAST ‘21
Reducing memory bandwidth:Coalesce data structures
![Page 22: Netflix: Streaming Entertainment to 200 Million Members Around … · 2021. 2. 23. · Netflix: Streaming Entertainment USENIX FAST ‘21 Open Connect is Netflix’s content delivery](https://reader036.vdocument.in/reader036/viewer/2022071511/6130f4611ecc515869446d4b/html5/thumbnails/22.jpg)
Netflix: Streaming EntertainmentUSENIX FAST ‘21
Reducing memory bandwidth:Sorted/RSS-Assisted LRO
![Page 23: Netflix: Streaming Entertainment to 200 Million Members Around … · 2021. 2. 23. · Netflix: Streaming Entertainment USENIX FAST ‘21 Open Connect is Netflix’s content delivery](https://reader036.vdocument.in/reader036/viewer/2022071511/6130f4611ecc515869446d4b/html5/thumbnails/23.jpg)
Netflix: Streaming EntertainmentUSENIX FAST ‘21
Reducing memory bandwidth:VM Optimizations from Upstream FreeBSD
● Per-CPU page caches● Batched frees
![Page 24: Netflix: Streaming Entertainment to 200 Million Members Around … · 2021. 2. 23. · Netflix: Streaming Entertainment USENIX FAST ‘21 Open Connect is Netflix’s content delivery](https://reader036.vdocument.in/reader036/viewer/2022071511/6130f4611ecc515869446d4b/html5/thumbnails/24.jpg)
Netflix: Streaming EntertainmentUSENIX FAST ‘21
Hardware offloads:TCP Segmentation Offload (TSO)
![Page 25: Netflix: Streaming Entertainment to 200 Million Members Around … · 2021. 2. 23. · Netflix: Streaming Entertainment USENIX FAST ‘21 Open Connect is Netflix’s content delivery](https://reader036.vdocument.in/reader036/viewer/2022071511/6130f4611ecc515869446d4b/html5/thumbnails/25.jpg)
Netflix: Streaming EntertainmentUSENIX FAST ‘21
Hardware offloads:Inline TLS Encryption
![Page 26: Netflix: Streaming Entertainment to 200 Million Members Around … · 2021. 2. 23. · Netflix: Streaming Entertainment USENIX FAST ‘21 Open Connect is Netflix’s content delivery](https://reader036.vdocument.in/reader036/viewer/2022071511/6130f4611ecc515869446d4b/html5/thumbnails/26.jpg)
Netflix: Streaming EntertainmentUSENIX FAST ‘21
PCIe Optimizations:Coalescing small memory segments
![Page 27: Netflix: Streaming Entertainment to 200 Million Members Around … · 2021. 2. 23. · Netflix: Streaming Entertainment USENIX FAST ‘21 Open Connect is Netflix’s content delivery](https://reader036.vdocument.in/reader036/viewer/2022071511/6130f4611ecc515869446d4b/html5/thumbnails/27.jpg)
Netflix: Streaming EntertainmentUSENIX FAST ‘21
Enabling new platforms/designs:NUMA enablement
![Page 28: Netflix: Streaming Entertainment to 200 Million Members Around … · 2021. 2. 23. · Netflix: Streaming Entertainment USENIX FAST ‘21 Open Connect is Netflix’s content delivery](https://reader036.vdocument.in/reader036/viewer/2022071511/6130f4611ecc515869446d4b/html5/thumbnails/28.jpg)
Netflix: Streaming EntertainmentUSENIX FAST ‘21
Typical Netflix OCA Workload
CPU
RAM
Disks
Data Encrypted Data
Network Ports/
Offloads