scaling dubsmash's backend from 0 to 100+ million users
TRANSCRIPT
![Page 1: Scaling Dubsmash's backend from 0 to 100+ million users](https://reader035.vdocument.in/reader035/viewer/2022062306/58ed2a3b1a28ab3e458b469f/html5/thumbnails/1.jpg)
Scaling Dubsmash's backendfrom 0 to 100+ million users
PYCON.DE Munich - Daniel Taschik – 10/29/2016
![Page 2: Scaling Dubsmash's backend from 0 to 100+ million users](https://reader035.vdocument.in/reader035/viewer/2022062306/58ed2a3b1a28ab3e458b469f/html5/thumbnails/2.jpg)
We hit a nerve.
>100M Users
192 Countries
1.5BVideos
![Page 3: Scaling Dubsmash's backend from 0 to 100+ million users](https://reader035.vdocument.in/reader035/viewer/2022062306/58ed2a3b1a28ab3e458b469f/html5/thumbnails/3.jpg)
Dubsmash
Connect Create Communicate
![Page 4: Scaling Dubsmash's backend from 0 to 100+ million users](https://reader035.vdocument.in/reader035/viewer/2022062306/58ed2a3b1a28ab3e458b469f/html5/thumbnails/4.jpg)
The Start
![Page 5: Scaling Dubsmash's backend from 0 to 100+ million users](https://reader035.vdocument.in/reader035/viewer/2022062306/58ed2a3b1a28ab3e458b469f/html5/thumbnails/5.jpg)
The Start
Backend• Django-powered BE for content
management• web-based Dubloader to add
sounds• deployed on Heroku
Content Delivery• sound files in S3• meta information in JSON file in S3• files served via Cloudfront CDN
Metrics• Dubloader with < 100 req/min• >500 TB! of traffic in January 2015
![Page 6: Scaling Dubsmash's backend from 0 to 100+ million users](https://reader035.vdocument.in/reader035/viewer/2022062306/58ed2a3b1a28ab3e458b469f/html5/thumbnails/6.jpg)
Dubsmash Service Landscape
Backend
Router
S3 sound storage
Cloudfront CDN
![Page 7: Scaling Dubsmash's backend from 0 to 100+ million users](https://reader035.vdocument.in/reader035/viewer/2022062306/58ed2a3b1a28ab3e458b469f/html5/thumbnails/7.jpg)
![Page 8: Scaling Dubsmash's backend from 0 to 100+ million users](https://reader035.vdocument.in/reader035/viewer/2022062306/58ed2a3b1a28ab3e458b469f/html5/thumbnails/8.jpg)
New Features: Registration & Search
User registration • API based on REST framework• Django user model• store user’s most like sounds• push notifications for new content
Server-side Sound search• new Django-based service• search via ElasticSearch using Haystack• Celery-based indexing on RQ
Metrics• 100.000 registrations within first 24h• >20.000 requests per minute on search service
![Page 9: Scaling Dubsmash's backend from 0 to 100+ million users](https://reader035.vdocument.in/reader035/viewer/2022062306/58ed2a3b1a28ab3e458b469f/html5/thumbnails/9.jpg)
caching
Cloudfront CDN
S3 sound storage
Dubsmash Service Landscape
Searc
h
Router
main API
![Page 10: Scaling Dubsmash's backend from 0 to 100+ million users](https://reader035.vdocument.in/reader035/viewer/2022062306/58ed2a3b1a28ab3e458b469f/html5/thumbnails/10.jpg)
DubTalk
Social Graph Service• friend relations on platform• Django • TitanDB on Cassandra• later DynamoDB
DubTalk Service• group & video management• Django
Service Communication• Async via Celery on RabbitMQ• Sync via internal HTTPS API
Metrics• > 50.000 requests per min on both• > 150.000.000 videos stored
![Page 11: Scaling Dubsmash's backend from 0 to 100+ million users](https://reader035.vdocument.in/reader035/viewer/2022062306/58ed2a3b1a28ab3e458b469f/html5/thumbnails/11.jpg)
NoSQL
Cloudfront CDN
caching
S3 sound storage
Dubsmash Service Landscape
GraphDubTalk
Router
Monolith
relational DB
![Page 12: Scaling Dubsmash's backend from 0 to 100+ million users](https://reader035.vdocument.in/reader035/viewer/2022062306/58ed2a3b1a28ab3e458b469f/html5/thumbnails/12.jpg)
Large Scale Problems
favorited sounds outgrew our PostgreSQL
• > 1.000.000.000 favorited sounds• simple data model & access pattern• Premium-7 120GB RAM, 1TB disk instance
dtaschik@unic0rn:~/dubsmash$ heroku pg:table-size -a dubsmash
name | size-----------------------------------+------------ users_favs | 158 GB
dtaschik@unic0rn:~/dubsmash $ heroku pg:index-size -a dubsmashname | size-----------------------------------+------------ users_favs_username_key | 132 GB
ID username
sound_id
1 daniel3 Dzdcjc
2 sarah 3jGYzH
Let’s make it a new service!
![Page 13: Scaling Dubsmash's backend from 0 to 100+ million users](https://reader035.vdocument.in/reader035/viewer/2022062306/58ed2a3b1a28ab3e458b469f/html5/thumbnails/13.jpg)
Cloudfront CDNS3 sound storage
Dubsmash Service Landscape
Auth
Graph DubTalk
Favs
Router
Monolith
relational DB
caching
NoSQL
many more
![Page 14: Scaling Dubsmash's backend from 0 to 100+ million users](https://reader035.vdocument.in/reader035/viewer/2022062306/58ed2a3b1a28ab3e458b469f/html5/thumbnails/14.jpg)
Our Goal
Interested? Come and join! 😜
Building the largest mobile video communication platform!