Download - Socialite, the Open Source Status Feed Part 1: Design Overview and Scaling for Infinite Content
![Page 1: Socialite, the Open Source Status Feed Part 1: Design Overview and Scaling for Infinite Content](https://reader033.vdocument.in/reader033/viewer/2022051411/540d910d8d7f728d7e8b49eb/html5/thumbnails/1.jpg)
Building a Social Platform
Part 1: Design Overview;Storing Infinite Content
![Page 2: Socialite, the Open Source Status Feed Part 1: Design Overview and Scaling for Infinite Content](https://reader033.vdocument.in/reader033/viewer/2022051411/540d910d8d7f728d7e8b49eb/html5/thumbnails/2.jpg)
Solutions Engineering
• Identify Popular Use Cases– Directly from MongoDB Users– Addressing "limitations"
• Go beyond documentation and blogs• Create open source project• Run it!
![Page 3: Socialite, the Open Source Status Feed Part 1: Design Overview and Scaling for Infinite Content](https://reader033.vdocument.in/reader033/viewer/2022051411/540d910d8d7f728d7e8b49eb/html5/thumbnails/3.jpg)
Social Status Feed
![Page 4: Socialite, the Open Source Status Feed Part 1: Design Overview and Scaling for Infinite Content](https://reader033.vdocument.in/reader033/viewer/2022051411/540d910d8d7f728d7e8b49eb/html5/thumbnails/4.jpg)
Agenda• What is a status feed and why build it w/MongoDB• Application overview (goals, non-goals) • Architecture overview (arch diagram)• Operational overview (benchmarks, automation)• Describe components – Describe options
• For each component– Options tried– Results– Option chosen
![Page 5: Socialite, the Open Source Status Feed Part 1: Design Overview and Scaling for Infinite Content](https://reader033.vdocument.in/reader033/viewer/2022051411/540d910d8d7f728d7e8b49eb/html5/thumbnails/5.jpg)
Socialite
• News/Social Status Feed: popular and common
• Appears misleadingly simple: turns out to have many tricky problems to solve to have good performance
• We created a reference implementation – Configurable models and options– Built-in benchmarking
• Used this implementation to test out different options.• This talk will summarize
![Page 6: Socialite, the Open Source Status Feed Part 1: Design Overview and Scaling for Infinite Content](https://reader033.vdocument.in/reader033/viewer/2022051411/540d910d8d7f728d7e8b49eb/html5/thumbnails/6.jpg)
Status Feed
![Page 7: Socialite, the Open Source Status Feed Part 1: Design Overview and Scaling for Infinite Content](https://reader033.vdocument.in/reader033/viewer/2022051411/540d910d8d7f728d7e8b49eb/html5/thumbnails/7.jpg)
Status Feed
![Page 8: Socialite, the Open Source Status Feed Part 1: Design Overview and Scaling for Infinite Content](https://reader033.vdocument.in/reader033/viewer/2022051411/540d910d8d7f728d7e8b49eb/html5/thumbnails/8.jpg)
Socialite
• Open Source• Reference Implementation – Various Fanout Feed Models– User Graph Implementation– Content storage
• Configurable models and options• REST API in Dropwizard (Yammer)– https://dropwizard.github.io/dropwizard/
• Built-in benchmarking
https://github.com/10gen-labs/socialite
![Page 9: Socialite, the Open Source Status Feed Part 1: Design Overview and Scaling for Infinite Content](https://reader033.vdocument.in/reader033/viewer/2022051411/540d910d8d7f728d7e8b49eb/html5/thumbnails/9.jpg)
Architecture
Graph Service
Proxy
Cont
ent
Prox
y
![Page 10: Socialite, the Open Source Status Feed Part 1: Design Overview and Scaling for Infinite Content](https://reader033.vdocument.in/reader033/viewer/2022051411/540d910d8d7f728d7e8b49eb/html5/thumbnails/10.jpg)
Pluggable Services
• Major components each have an interface– see com.mongodb.socialite.services
• Configuration selects implementation to use• ServiceManager organizes : – Default implementations– Lifecycle– Binding configuration– Wiring dependencies– see com.mongodb.socialite.ServiceManager
![Page 11: Socialite, the Open Source Status Feed Part 1: Design Overview and Scaling for Infinite Content](https://reader033.vdocument.in/reader033/viewer/2022051411/540d910d8d7f728d7e8b49eb/html5/thumbnails/11.jpg)
Simple Interface
GET /users/{user_id} Get a User by their ID DELETE /users/{user_id} Remove a user by their ID POST /users/{user_id}/posts Send a message from this user GET /users/{user_id}/followers Get a list of followers of a user GET /users/{user_id}/followers_count Get the number of followers of a user GET /users/{user_id}/following Get the list of users this user is following GET /users/{user_id}/following count Get the number of users this user follows GET /users/{user_id}/posts Get the messages sent by a user GET /users/{user_id}/timeline Get the timeline for this user PUT /users/{user_id} Create a new user PUT /users/{user_id}/following/{target} Follow a user DELETE /users/{user_id}/following/{target} Unfollow a user
https://github.com/10gen-labs/socialite
![Page 12: Socialite, the Open Source Status Feed Part 1: Design Overview and Scaling for Infinite Content](https://reader033.vdocument.in/reader033/viewer/2022051411/540d910d8d7f728d7e8b49eb/html5/thumbnails/12.jpg)
Technical Decisions
User
timeline cache
Schema
Indexing Horizontal Scaling
![Page 13: Socialite, the Open Source Status Feed Part 1: Design Overview and Scaling for Infinite Content](https://reader033.vdocument.in/reader033/viewer/2022051411/540d910d8d7f728d7e8b49eb/html5/thumbnails/13.jpg)
Operational Setup
![Page 14: Socialite, the Open Source Status Feed Part 1: Design Overview and Scaling for Infinite Content](https://reader033.vdocument.in/reader033/viewer/2022051411/540d910d8d7f728d7e8b49eb/html5/thumbnails/14.jpg)
Real life validation of our choices.
User facing latencyLinear scaling of resources
Most important criteria?
Operational Testing
![Page 15: Socialite, the Open Source Status Feed Part 1: Design Overview and Scaling for Infinite Content](https://reader033.vdocument.in/reader033/viewer/2022051411/540d910d8d7f728d7e8b49eb/html5/thumbnails/15.jpg)
Scaling Goals
• Realistic real-life-scale workload– compared to Twitter, etc.
• Understanding of HW required– containing costs
• Confirm architecture scales linearly– without loss of responsiveness
![Page 16: Socialite, the Open Source Status Feed Part 1: Design Overview and Scaling for Infinite Content](https://reader033.vdocument.in/reader033/viewer/2022051411/540d910d8d7f728d7e8b49eb/html5/thumbnails/16.jpg)
Architecture
Graph Service
Proxy
Cont
ent
Prox
y
![Page 17: Socialite, the Open Source Status Feed Part 1: Design Overview and Scaling for Infinite Content](https://reader033.vdocument.in/reader033/viewer/2022051411/540d910d8d7f728d7e8b49eb/html5/thumbnails/17.jpg)
DB Architecture
The storage layer is separate from Socialite services, and each service has its own URI – its own mongodb server or cluster that can be configured differently from others.
This allows us to physically optimize each services' DB for the workload we'll be running on it.
It also allows us to scale out the DB that's currently the limiting factor (the bottleneck) in our setup.
![Page 18: Socialite, the Open Source Status Feed Part 1: Design Overview and Scaling for Infinite Content](https://reader033.vdocument.in/reader033/viewer/2022051411/540d910d8d7f728d7e8b49eb/html5/thumbnails/18.jpg)
Operational Testing
![Page 19: Socialite, the Open Source Status Feed Part 1: Design Overview and Scaling for Infinite Content](https://reader033.vdocument.in/reader033/viewer/2022051411/540d910d8d7f728d7e8b49eb/html5/thumbnails/19.jpg)
Operational Testing
![Page 20: Socialite, the Open Source Status Feed Part 1: Design Overview and Scaling for Infinite Content](https://reader033.vdocument.in/reader033/viewer/2022051411/540d910d8d7f728d7e8b49eb/html5/thumbnails/20.jpg)
Operational Testing
![Page 21: Socialite, the Open Source Status Feed Part 1: Design Overview and Scaling for Infinite Content](https://reader033.vdocument.in/reader033/viewer/2022051411/540d910d8d7f728d7e8b49eb/html5/thumbnails/21.jpg)
Operational Testing
![Page 22: Socialite, the Open Source Status Feed Part 1: Design Overview and Scaling for Infinite Content](https://reader033.vdocument.in/reader033/viewer/2022051411/540d910d8d7f728d7e8b49eb/html5/thumbnails/22.jpg)
Operational Testing
![Page 23: Socialite, the Open Source Status Feed Part 1: Design Overview and Scaling for Infinite Content](https://reader033.vdocument.in/reader033/viewer/2022051411/540d910d8d7f728d7e8b49eb/html5/thumbnails/23.jpg)
Operational Testing
![Page 24: Socialite, the Open Source Status Feed Part 1: Design Overview and Scaling for Infinite Content](https://reader033.vdocument.in/reader033/viewer/2022051411/540d910d8d7f728d7e8b49eb/html5/thumbnails/24.jpg)
Operational Testing
![Page 25: Socialite, the Open Source Status Feed Part 1: Design Overview and Scaling for Infinite Content](https://reader033.vdocument.in/reader033/viewer/2022051411/540d910d8d7f728d7e8b49eb/html5/thumbnails/25.jpg)
Operational Testing
![Page 26: Socialite, the Open Source Status Feed Part 1: Design Overview and Scaling for Infinite Content](https://reader033.vdocument.in/reader033/viewer/2022051411/540d910d8d7f728d7e8b49eb/html5/thumbnails/26.jpg)
Operational Testing
![Page 27: Socialite, the Open Source Status Feed Part 1: Design Overview and Scaling for Infinite Content](https://reader033.vdocument.in/reader033/viewer/2022051411/540d910d8d7f728d7e8b49eb/html5/thumbnails/27.jpg)
Operational Testing
![Page 28: Socialite, the Open Source Status Feed Part 1: Design Overview and Scaling for Infinite Content](https://reader033.vdocument.in/reader033/viewer/2022051411/540d910d8d7f728d7e8b49eb/html5/thumbnails/28.jpg)
Operational Testing
![Page 29: Socialite, the Open Source Status Feed Part 1: Design Overview and Scaling for Infinite Content](https://reader033.vdocument.in/reader033/viewer/2022051411/540d910d8d7f728d7e8b49eb/html5/thumbnails/29.jpg)
Operational Testing
![Page 30: Socialite, the Open Source Status Feed Part 1: Design Overview and Scaling for Infinite Content](https://reader033.vdocument.in/reader033/viewer/2022051411/540d910d8d7f728d7e8b49eb/html5/thumbnails/30.jpg)
Operational Testing
![Page 31: Socialite, the Open Source Status Feed Part 1: Design Overview and Scaling for Infinite Content](https://reader033.vdocument.in/reader033/viewer/2022051411/540d910d8d7f728d7e8b49eb/html5/thumbnails/31.jpg)
Operational Testing
Built-in benchmark capability
![Page 32: Socialite, the Open Source Status Feed Part 1: Design Overview and Scaling for Infinite Content](https://reader033.vdocument.in/reader033/viewer/2022051411/540d910d8d7f728d7e8b49eb/html5/thumbnails/32.jpg)
Operational Testing
• All hosts in AWS• Each service used its own DB, cluster or shards• All benchmarks through `mongos` (sharded config)• Used MMS monitoring for measuring throughput• Used internal benchmarks for measuring latency• Based volume tested on real life social metrics
![Page 33: Socialite, the Open Source Status Feed Part 1: Design Overview and Scaling for Infinite Content](https://reader033.vdocument.in/reader033/viewer/2022051411/540d910d8d7f728d7e8b49eb/html5/thumbnails/33.jpg)
Scaling for Infinite Content
![Page 34: Socialite, the Open Source Status Feed Part 1: Design Overview and Scaling for Infinite Content](https://reader033.vdocument.in/reader033/viewer/2022051411/540d910d8d7f728d7e8b49eb/html5/thumbnails/34.jpg)
Architecture
Graph Service
Proxy
Cont
ent
Prox
y
![Page 35: Socialite, the Open Source Status Feed Part 1: Design Overview and Scaling for Infinite Content](https://reader033.vdocument.in/reader033/viewer/2022051411/540d910d8d7f728d7e8b49eb/html5/thumbnails/35.jpg)
Socialite Content Service
• System of record for all user content• Initially very simple (no search)• Mainly designed to support feed– Lookup/indexed by _id and userid– Time based anchors/pagination
![Page 36: Socialite, the Open Source Status Feed Part 1: Design Overview and Scaling for Infinite Content](https://reader033.vdocument.in/reader033/viewer/2022051411/540d910d8d7f728d7e8b49eb/html5/thumbnails/36.jpg)
• Half life of most content is 1 day !
• Popular content usually < 1 month
• Access to old data is rare
Social Data Ages Fast