![Page 1: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/1.jpg)
Zero to 1 Billion Records
Kiril Savino @holacrat
![Page 2: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/2.jpg)
2
GC.com/about/product-team
![Page 3: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/3.jpg)
3
• have a sense of humor
• know what use cases work best
• remember that databases are hard
• don’t understate the difficulty in scaling up
![Page 4: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/4.jpg)
4
• 1,480,808,857 events
• 8 terabytes of primary data
• 35 nodes
• 420GB RAM on primaries
• 21TB SSD storage
• 14TB EBS storage
• 120,000 ops/s
![Page 5: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/5.jpg)
• Model
• Scale
• Grow
• Extend
5
![Page 6: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/6.jpg)
6
Model
![Page 7: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/7.jpg)
November 2009 — MongoDB 1.2
• More indexes per collection
• Faster index creation
• Map/Reduce
• Stored JavaScript functions
• Configurable fsync time
• Several small features and fixes
7
{.}
![Page 8: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/8.jpg)
8
{.?!?.}
![Page 9: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/9.jpg)
9
Decoding/Unmarshalling
Django ORM
{.}
[---]business logic
REST
API
MySQL
![Page 10: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/10.jpg)
10
Decoding/Unmarshalling
Django ORM
REST
API{.}
[---]business logic
MySQL
![Page 11: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/11.jpg)
11
InningOutsBallsStrikesPitcherBatter
![Page 12: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/12.jpg)
12
InningOutsBallsStrikesPitcherBatter
PeriodMinuteLocationShooterRebounderAssist
![Page 13: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/13.jpg)
13
[play]
[participant][role]
[sport][play_property]
![Page 14: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/14.jpg)
14
[play]
[participant][role]
[sport][play_property]
![Page 15: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/15.jpg)
15
{_id: ObjectId(), code: “1B”, participants: [{player_id: ObjectId(), roles: [“batter”, “out”]}, {player_id: ObjectId(), roles: [“pitcher”]}], situation: {outs: 1, balls: 2, strikes: 0}, properties: {location: [0.45, 0.721]}}
![Page 16: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/16.jpg)
16
{_id: ObjectId(), code: “shot”, participants: [{player_id: ObjectId(), roles: [“shooter”]}, {player_id: ObjectId(), roles: [“rebounder”]}], situation: {period: 1, time: 5:29}, properties: {location: [0.45, 0.721]}}
![Page 17: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/17.jpg)
17
Decoding/Unmarshalling
Django ORM
REST
API{.}
business logic
{.}MongoDB
![Page 18: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/18.jpg)
18
Decoding/Unmarshalling
Django ORM
REST
API{.}
business logic
{.}MongoDB
👏
![Page 19: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/19.jpg)
19
![Page 20: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/20.jpg)
Modeling data in MongoDB
20
• JSON won the internet
• Don’t write your own JSON storage engine
• Flexible schemas promote app simplicity
• Validation is your responsibility
• Invest in schema design early
![Page 21: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/21.jpg)
21
Scale
![Page 22: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/22.jpg)
22
![Page 23: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/23.jpg)
23
![Page 24: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/24.jpg)
24
![Page 25: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/25.jpg)
25
$$$
![Page 26: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/26.jpg)
26
$$$
😱
![Page 27: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/27.jpg)
27
User Load
System Latency
![Page 28: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/28.jpg)
28
User Load
System Latency
![Page 29: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/29.jpg)
29
User Load
System Latency
![Page 30: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/30.jpg)
30
Scaling is the process of decoupling load from latency.
![Page 31: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/31.jpg)
Latency comes from
31
• Writing data to your database
• Reading data from your database
• Aggregating data from multiple locations
• Running complex calculations
![Page 32: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/32.jpg)
32
{.}
This is a document.
![Page 33: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/33.jpg)
33
{.} {.}{.}
{.}{.}
API MongoDB Browser
![Page 34: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/34.jpg)
34
{.} {.}{.}
{.}{.}
API MongoDB Browser
![Page 35: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/35.jpg)
35
{.} {.}{.}
{.}{.}
API MongoDB Browser
+/-*
![Page 36: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/36.jpg)
36
Read Load
System Latency
![Page 37: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/37.jpg)
37
{.} {.}{.}
{.}{.}
API MongoDB Browser
![Page 38: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/38.jpg)
38
{.} {.}{.}
{.}{.}
API MongoDB Browser
+/-*
![Page 39: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/39.jpg)
39
Write Load
System Latency
![Page 40: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/40.jpg)
40
{.} {.}{.}
{.}{.}
API MongoDB Browser
Background+/-*
![Page 41: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/41.jpg)
41
{.} {.}{.}
{.}{.}
API MongoDB Browser
Background+/-*
![Page 42: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/42.jpg)
42
User Load
System Latency
![Page 43: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/43.jpg)
43
{.}{.}{.}
![Page 44: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/44.jpg)
44
{.}{.}{.}
{.} }
![Page 45: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/45.jpg)
45
{.}{.}{.}
{.} }
![Page 46: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/46.jpg)
46
![Page 47: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/47.jpg)
Scaling data access
47
• Decouple load from latency
• Queries are expensive
• Aggregation is expensive
• Do calculation in the background
• Serve content from single* documents
![Page 48: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/48.jpg)
48
Grow
![Page 49: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/49.jpg)
49
![Page 50: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/50.jpg)
50
![Page 51: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/51.jpg)
51
![Page 52: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/52.jpg)
52
{.}
![Page 53: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/53.jpg)
53
{.}
![Page 54: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/54.jpg)
54
{.}
![Page 55: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/55.jpg)
55
{.}
![Page 56: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/56.jpg)
56
![Page 57: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/57.jpg)
57
{.} {$addToSet: {a: 2}}
![Page 58: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/58.jpg)
58
{.} {$addToSet: {a: 2}}
{.} {v: 2}, {$set: {v: 3}}
![Page 59: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/59.jpg)
59
{.}
![Page 60: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/60.jpg)
60
![Page 61: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/61.jpg)
61
{.} {.}
![Page 62: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/62.jpg)
62
{a}{abc}{b}
{c} }
![Page 63: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/63.jpg)
63
{.}
![Page 64: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/64.jpg)
64
{.}{.}
![Page 65: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/65.jpg)
65
{.} {.}{.}
![Page 66: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/66.jpg)
66
{.} {.}{.}
![Page 67: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/67.jpg)
67
{.} {.}{.}
![Page 68: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/68.jpg)
68
{.} {.}{.}
![Page 69: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/69.jpg)
69
{.} {.}{.}
![Page 70: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/70.jpg)
70
{.} {.} {.}
![Page 71: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/71.jpg)
71
<id><id><id><id><id><id><id>
To Propagate
![Page 72: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/72.jpg)
72
<id><id><id><id><id><id><id>
To Propagate Propagating…
![Page 73: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/73.jpg)
73
<id><id><id><id><id><id><id>
To Propagate Propagating…
<id> {.}{.}{.}
![Page 74: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/74.jpg)
74
{$} {$} {$} {$} {$}
![Page 75: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/75.jpg)
Growing load
75
• Denormalize for constant access time
• Use MongoDB atomic operators
• Check out optimistic locking and MVCC
• Leverage external concurrency control
• Watch your oplog
![Page 76: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/76.jpg)
76
Extend
![Page 77: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/77.jpg)
77
{.} +
![Page 78: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/78.jpg)
78
![Page 79: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/79.jpg)
79
![Page 80: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/80.jpg)
80
![Page 81: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/81.jpg)
So there we have it
• Design your schema to MongoDB’s strengths
• Use monolithic documents
• Don’t do (live) querying
• You can still do transactional things
• You may need to denormalize & propagate
• Think about your overall architecture
81
![Page 82: Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger](https://reader033.vdocument.in/reader033/viewer/2022060119/558d0b46d8b42a915a8b45f7/html5/thumbnails/82.jpg)
82
• have a sense of humor
• know what use cases work best
• remember that databases are hard
• don’t understate the difficulty in scaling up
@holacrat