![Page 1: Leveraging MongoDB as a Data Store for Security Data](https://reader038.vdocument.in/reader038/viewer/2022102823/54937f67b47959b25e8b46a2/html5/thumbnails/1.jpg)
MongoDB as a Data Store for Security DataScaling out the mongod node
Daniel Bauman
Sr. Cyber Intelligence Analyst
LM-CIRT
© 2012 Lockheed Martin Corporation. All Rights Reserved.
![Page 2: Leveraging MongoDB as a Data Store for Security Data](https://reader038.vdocument.in/reader038/viewer/2022102823/54937f67b47959b25e8b46a2/html5/thumbnails/2.jpg)
2
Contexts
Information01101100011011010110001101101111
Influence (Application)
Intelligence
© 2014 Lockheed Martin Corporation. All Rights Reserved.
![Page 3: Leveraging MongoDB as a Data Store for Security Data](https://reader038.vdocument.in/reader038/viewer/2022102823/54937f67b47959b25e8b46a2/html5/thumbnails/3.jpg)
3 Key Brick Walls
© 2014 Lockheed Martin Corporation. All Rights Reserved.3
1• Isolation
2• Retention
3• Access
![Page 4: Leveraging MongoDB as a Data Store for Security Data](https://reader038.vdocument.in/reader038/viewer/2022102823/54937f67b47959b25e8b46a2/html5/thumbnails/4.jpg)
4
Isolated Information
© 2014 Lockheed Martin Corporation. All Rights Reserved.
01101100011011010110001101101111
01101100011011010110001101101111
01101100011011010110001101101111
01101100011011010110001101101111
![Page 5: Leveraging MongoDB as a Data Store for Security Data](https://reader038.vdocument.in/reader038/viewer/2022102823/54937f67b47959b25e8b46a2/html5/thumbnails/5.jpg)
5
Isolated Information
© 2014 Lockheed Martin Corporation. All Rights Reserved.
01101100011011010110001101101111
01101100011011010110001101101111
01101100011011010110001101101111
01101100011011010110001101101111
![Page 6: Leveraging MongoDB as a Data Store for Security Data](https://reader038.vdocument.in/reader038/viewer/2022102823/54937f67b47959b25e8b46a2/html5/thumbnails/6.jpg)
6
Pizza Boxes
© 2014 Lockheed Martin Corporation. All Rights Reserved.
✔Action
![Page 7: Leveraging MongoDB as a Data Store for Security Data](https://reader038.vdocument.in/reader038/viewer/2022102823/54937f67b47959b25e8b46a2/html5/thumbnails/7.jpg)
7
Single Pizza Box Throughput
© 2014 Lockheed Martin Corporation. All Rights Reserved.
✔Action
![Page 8: Leveraging MongoDB as a Data Store for Security Data](https://reader038.vdocument.in/reader038/viewer/2022102823/54937f67b47959b25e8b46a2/html5/thumbnails/8.jpg)
8
Pizza Boxes
© 2014 Lockheed Martin Corporation. All Rights Reserved.
✔Action
![Page 9: Leveraging MongoDB as a Data Store for Security Data](https://reader038.vdocument.in/reader038/viewer/2022102823/54937f67b47959b25e8b46a2/html5/thumbnails/9.jpg)
9 © 2014 Lockheed Martin Corporation. All Rights Reserved.
2• Retention
![Page 10: Leveraging MongoDB as a Data Store for Security Data](https://reader038.vdocument.in/reader038/viewer/2022102823/54937f67b47959b25e8b46a2/html5/thumbnails/10.jpg)
10
The Dream – MongoD Standard Install
© 2014 Lockheed Martin Corporation. All Rights Reserved.
Documents per SecondData Size
Data Size vs Documents/sec
Size
time
Doc
umen
ts/s
ec
![Page 11: Leveraging MongoDB as a Data Store for Security Data](https://reader038.vdocument.in/reader038/viewer/2022102823/54937f67b47959b25e8b46a2/html5/thumbnails/11.jpg)
11
Data Size vs Documents/sec
The Reality – MongoD Standard Install
© 2014 Lockheed Martin Corporation. All Rights Reserved.
Documents per SecondData Size
File size vs Inserts
Size
time
Doc
umen
ts/s
ec
![Page 12: Leveraging MongoDB as a Data Store for Security Data](https://reader038.vdocument.in/reader038/viewer/2022102823/54937f67b47959b25e8b46a2/html5/thumbnails/12.jpg)
12
The Dream – Data Retention
© 2014 Lockheed Martin Corporation. All Rights Reserved.
Documents per SecondData Size
Data Size vs Documents/sec
Size
time
Doc
umen
ts/s
ec
![Page 13: Leveraging MongoDB as a Data Store for Security Data](https://reader038.vdocument.in/reader038/viewer/2022102823/54937f67b47959b25e8b46a2/html5/thumbnails/13.jpg)
13
Mongo DatabaseDisk Is FULL
Single Pizza Box Data Retention
© 2014 Lockheed Martin Corporation. All Rights Reserved.
Trash
![Page 14: Leveraging MongoDB as a Data Store for Security Data](https://reader038.vdocument.in/reader038/viewer/2022102823/54937f67b47959b25e8b46a2/html5/thumbnails/14.jpg)
14
The Reality – MongoD Capped Collection
© 2014 Lockheed Martin Corporation. All Rights Reserved.
Documents per SecondData Size
File size vs Inserts
Size
time
Doc
umen
ts/s
ec
![Page 15: Leveraging MongoDB as a Data Store for Security Data](https://reader038.vdocument.in/reader038/viewer/2022102823/54937f67b47959b25e8b46a2/html5/thumbnails/15.jpg)
15 © 2014 Lockheed Martin Corporation. All Rights Reserved.
3• Access
![Page 16: Leveraging MongoDB as a Data Store for Security Data](https://reader038.vdocument.in/reader038/viewer/2022102823/54937f67b47959b25e8b46a2/html5/thumbnails/16.jpg)
16
The Dream - Querying the Cloud
© 2014 Lockheed Martin Corporation. All Rights Reserved.
Query Response
0110110001101101011000110111000110110101100011010110011010110110001101101011000110101101011000110110001111000110101101100011011010
![Page 17: Leveraging MongoDB as a Data Store for Security Data](https://reader038.vdocument.in/reader038/viewer/2022102823/54937f67b47959b25e8b46a2/html5/thumbnails/17.jpg)
17
And now for something less technical
© 2014 Lockheed Martin Corporation. All Rights Reserved.
![Page 18: Leveraging MongoDB as a Data Store for Security Data](https://reader038.vdocument.in/reader038/viewer/2022102823/54937f67b47959b25e8b46a2/html5/thumbnails/18.jpg)
18
172.100.178.247
Information Retrieval
172.100.27.143 172.100.164.66 172.100.255.250 172.100.235.24 172.100.195.178 172.100.7.227 172.100.215.227 172.100.31.0 172.100.81.242 172.100.156.25 172.100.139.53 172.100.235.229 172.100.25.137 172.100.171.91 172.100.71.242 172.100.108.64 172.100.96.73 172.100.126.217 172.100.77.25 172.100.214.219 172.100.102.211 172.100.124.176 172.100.96.81 172.100.131.150 172.100.98.250 172.100.178.247 172.100.138.157 172.100.45.67 172.100.122.239 172.100.138.218 172.100.102.110 172.100.49.93 172.100.245.74 172.100.213.39 172.100.80.14 172.100.41.125 172.100.150.202 172.100.1.184 172.100.149.233 172.100.98.83 172.100.199.75 172.100.244.223 172.100.140.69 172.100.187.27 172.100.209.228 172.100.6.249 172.100.60.48 172.100.138.64 172.100.130.181 172.100.188.177 172.100.142.25 172.100.109.79 172.100.70.58 172.100.65.184 172.100.250.150 172.100.215.195 172.100.137.136 172.100.49.64 172.100.148.19 172.100.244.227 172.100.178.131 172.100.255.199 172.100.65.112 172.100.201.249 172.100.53.21 172.100.235.60 172.100.84.205 172.100.16.194 172.100.216.90 172.100.45.88 172.100.240.174 172.100.248.179 172.100.48.70 172.100.8.200 172.100.45.130 172.100.235.59 172.100.171.231 172.100.29.124 172.100.239.204 172.100.172.241 172.100.158.216 172.100.70.109 172.100.227.117 172.100.144.199 172.100.223.36 172.100.166.60 172.100.48.61 172.100.70.76 172.100.51.152 172.100.157.95 172.100.71.133 172.100.0.25 172.100.167.58 172.100.94.133 172.100.93.92 172.100.192.109 172.100.176.25 172.100.169.236 172.100.164.186
© 2014 Lockheed Martin Corporation. All Rights Reserved.
“1.0 second is about the limit for the user’s flow of thought to stay
uninterrupted” – Nielson (1993)
J. Nielsen, "Response times: the three important limits," 1993
![Page 19: Leveraging MongoDB as a Data Store for Security Data](https://reader038.vdocument.in/reader038/viewer/2022102823/54937f67b47959b25e8b46a2/html5/thumbnails/19.jpg)
19
Information Retrieval – 10 seconds
© 2014 Lockheed Martin Corporation. All Rights Reserved.
1968 R. Miller, "Response time in man-computer conversational transaction,"
“response delays of a standard ten seconds will not permit the kind of
thinking continuity essential to sustained problem solving”
– R. Miller(1968)
![Page 20: Leveraging MongoDB as a Data Store for Security Data](https://reader038.vdocument.in/reader038/viewer/2022102823/54937f67b47959b25e8b46a2/html5/thumbnails/20.jpg)
20
Diving Back In
© 2014 Lockheed Martin Corporation. All Rights Reserved.
![Page 21: Leveraging MongoDB as a Data Store for Security Data](https://reader038.vdocument.in/reader038/viewer/2022102823/54937f67b47959b25e8b46a2/html5/thumbnails/21.jpg)
21
Random Data Access
© 2014 Lockheed Martin Corporation. All Rights Reserved.
past recent
Documents
![Page 22: Leveraging MongoDB as a Data Store for Security Data](https://reader038.vdocument.in/reader038/viewer/2022102823/54937f67b47959b25e8b46a2/html5/thumbnails/22.jpg)
Python-MongoR (R for Retention)
Distributed database expansion to MongoDB designed to optimize scale-out, write intensive document storage
© 2014 Lockheed Martin Corporation. All Rights Reserved.
![Page 23: Leveraging MongoDB as a Data Store for Security Data](https://reader038.vdocument.in/reader038/viewer/2022102823/54937f67b47959b25e8b46a2/html5/thumbnails/23.jpg)
23
Data Buckets
© 2014 Lockheed Martin Corporation. All Rights Reserved.
past recent
Documents
![Page 24: Leveraging MongoDB as a Data Store for Security Data](https://reader038.vdocument.in/reader038/viewer/2022102823/54937f67b47959b25e8b46a2/html5/thumbnails/24.jpg)
24
MongoR Buckets
© 2014 Lockheed Martin Corporation. All Rights Reserved.
past recent
DB DB DB DB DB DB
![Page 25: Leveraging MongoDB as a Data Store for Security Data](https://reader038.vdocument.in/reader038/viewer/2022102823/54937f67b47959b25e8b46a2/html5/thumbnails/25.jpg)
25
MongoR Automated Segmenting
© 2014 Lockheed Martin Corporation. All Rights Reserved.
past recent
DB DB DB DB DBDB DB DB DB DBGenerator
![Page 26: Leveraging MongoDB as a Data Store for Security Data](https://reader038.vdocument.in/reader038/viewer/2022102823/54937f67b47959b25e8b46a2/html5/thumbnails/26.jpg)
26
Mongo
Disk Is Full
Mongo
MongoR Retention
© 2014 Lockheed Martin Corporation. All Rights Reserved.
Trash
Mongo
Mongo Mongo
![Page 27: Leveraging MongoDB as a Data Store for Security Data](https://reader038.vdocument.in/reader038/viewer/2022102823/54937f67b47959b25e8b46a2/html5/thumbnails/27.jpg)
27
MongoR
Mongo
MongoR “Capped Collection”
© 2014 Lockheed Martin Corporation. All Rights Reserved.
Mongo
Mongo Mongo
![Page 28: Leveraging MongoDB as a Data Store for Security Data](https://reader038.vdocument.in/reader038/viewer/2022102823/54937f67b47959b25e8b46a2/html5/thumbnails/28.jpg)
28
MongoR Destructor
© 2014 Lockheed Martin Corporation. All Rights Reserved.
past recent
DB DB DBDB GeneratorDestructor
![Page 29: Leveraging MongoDB as a Data Store for Security Data](https://reader038.vdocument.in/reader038/viewer/2022102823/54937f67b47959b25e8b46a2/html5/thumbnails/29.jpg)
29
MongoR Destructor
© 2014 Lockheed Martin Corporation. All Rights Reserved.
past recent
DB DB DB DB DB DB DB DBDB DB DB DB DB DB DB DB DB DB DBDB DB DB DBGenerator
![Page 30: Leveraging MongoDB as a Data Store for Security Data](https://reader038.vdocument.in/reader038/viewer/2022102823/54937f67b47959b25e8b46a2/html5/thumbnails/30.jpg)
30
The Real
© 2014 Lockheed Martin Corporation. All Rights Reserved.
Documents per SecondData Size
Data Size vs Documents/sec
Size
time
Doc
umen
ts/s
ec
![Page 31: Leveraging MongoDB as a Data Store for Security Data](https://reader038.vdocument.in/reader038/viewer/2022102823/54937f67b47959b25e8b46a2/html5/thumbnails/31.jpg)
31
MongoR Production Behavior.
© 2014 Lockheed Martin Corporation. All Rights Reserved.
![Page 32: Leveraging MongoDB as a Data Store for Security Data](https://reader038.vdocument.in/reader038/viewer/2022102823/54937f67b47959b25e8b46a2/html5/thumbnails/32.jpg)
32
Best Practices – Bucket Size
Bucket size = ¼ RAM size
© 2014 Lockheed Martin Corporation. All Rights Reserved.
System RAM Mongo Mongo
Mongo Mongo
![Page 33: Leveraging MongoDB as a Data Store for Security Data](https://reader038.vdocument.in/reader038/viewer/2022102823/54937f67b47959b25e8b46a2/html5/thumbnails/33.jpg)
33
Best Practices – Bucket Limit
Bucket Limit = 85-90% Capacity
© 2014 Lockheed Martin Corporation. All Rights Reserved.
System Drive Capacity
![Page 34: Leveraging MongoDB as a Data Store for Security Data](https://reader038.vdocument.in/reader038/viewer/2022102823/54937f67b47959b25e8b46a2/html5/thumbnails/34.jpg)
34
Python-mongor In Production
• MIT Licensed
– https://github.com/lmco/python-mongor
© 2014 Lockheed Martin Corporation. All Rights Reserved.
![Page 35: Leveraging MongoDB as a Data Store for Security Data](https://reader038.vdocument.in/reader038/viewer/2022102823/54937f67b47959b25e8b46a2/html5/thumbnails/35.jpg)
Questions
35 © 2014 Lockheed Martin Corporation. All Rights Reserved.
![Page 36: Leveraging MongoDB as a Data Store for Security Data](https://reader038.vdocument.in/reader038/viewer/2022102823/54937f67b47959b25e8b46a2/html5/thumbnails/36.jpg)