mongo for the mission critical enterprise
TRANSCRIPT
![Page 1: Mongo for the Mission Critical Enterprise](https://reader033.vdocument.in/reader033/viewer/2022052903/5579f4e6d8b42abc2e8b4b0f/html5/thumbnails/1.jpg)
© EnerNOC Inc.
Mongo for the Mission-Critical Enterprise
Thom NicholsPrincipal Software Engineer
Advanced Technology@thom_nic
![Page 2: Mongo for the Mission Critical Enterprise](https://reader033.vdocument.in/reader033/viewer/2022052903/5579f4e6d8b42abc2e8b4b0f/html5/thumbnails/2.jpg)
2
What am I talking about?
Integrating Mongo into a mission-critical system
• EnerNOC provides a 24/ 7 system to help ensure electric grid stability
• How do we introduce Mongo to help meet scalability needs?
• How do we do so in a way that ensures stability?
![Page 3: Mongo for the Mission Critical Enterprise](https://reader033.vdocument.in/reader033/viewer/2022052903/5579f4e6d8b42abc2e8b4b0f/html5/thumbnails/3.jpg)
3
Background
We provide stability to the electric grid.
How we do it:
1. Instability on the grid (peaking load, transmission line failure)
2. Grid operator sends a “demand response” signal to us
3. We send a signal to our customers (large electric consumers)
4. Customers reduce their energy usage
5. ???
6. Profit!
We monitor customer telemetry in near-realtime
![Page 4: Mongo for the Mission Critical Enterprise](https://reader033.vdocument.in/reader033/viewer/2022052903/5579f4e6d8b42abc2e8b4b0f/html5/thumbnails/4.jpg)
4
Background
• We’ve experienced tremendous growth
• Software (esp. database) struggles to keep up
• Hundreds of tables
• Hibernate Inefficient queries
• Our solution? Make everything a stored proc
• Better, except…
• A few tables take up 99% of our storage
• It’s still slow
• Looking at $$millions in hardware cost to scale
![Page 5: Mongo for the Mission Critical Enterprise](https://reader033.vdocument.in/reader033/viewer/2022052903/5579f4e6d8b42abc2e8b4b0f/html5/thumbnails/5.jpg)
5
What’s worse than a database that won’t scale?
Two databases that won’t scale.
![Page 6: Mongo for the Mission Critical Enterprise](https://reader033.vdocument.in/reader033/viewer/2022052903/5579f4e6d8b42abc2e8b4b0f/html5/thumbnails/6.jpg)
6
Planning for Mongo
We listened
• Replication from day 1
• Sharding shortly after
![Page 7: Mongo for the Mission Critical Enterprise](https://reader033.vdocument.in/reader033/viewer/2022052903/5579f4e6d8b42abc2e8b4b0f/html5/thumbnails/7.jpg)
7
Planning for Mongo
• Bucketing
• Pre-allocated stubbed out documents
• Hashed IDs for data locality
pid = 1234 # natural key
hash_pid = pid.to_s.rjust(15,'0').reverse!.to_i
_id = { :p => hash_pid, :s => Date.new }
![Page 8: Mongo for the Mission Critical Enterprise](https://reader033.vdocument.in/reader033/viewer/2022052903/5579f4e6d8b42abc2e8b4b0f/html5/thumbnails/8.jpg)
8
The Software Architect’s Haiku
Avoid vendor lock-in
Mongo might solve scaling problems
But increase dependencies
![Page 9: Mongo for the Mission Critical Enterprise](https://reader033.vdocument.in/reader033/viewer/2022052903/5579f4e6d8b42abc2e8b4b0f/html5/thumbnails/9.jpg)
9
What did we do?
Create a Data Management Service
• Common HTTP interface for data in, data out
• Hides ID hashing & bucketing
• Interface for on-the-fly aggregation
• Caching behind the service
• Native client libraries – Java, Ruby, R
![Page 10: Mongo for the Mission Critical Enterprise](https://reader033.vdocument.in/reader033/viewer/2022052903/5579f4e6d8b42abc2e8b4b0f/html5/thumbnails/10.jpg)
10
The Basic Idea
Woo! Problem solved!
![Page 11: Mongo for the Mission Critical Enterprise](https://reader033.vdocument.in/reader033/viewer/2022052903/5579f4e6d8b42abc2e8b4b0f/html5/thumbnails/11.jpg)
11
So What?
![Page 12: Mongo for the Mission Critical Enterprise](https://reader033.vdocument.in/reader033/viewer/2022052903/5579f4e6d8b42abc2e8b4b0f/html5/thumbnails/12.jpg)
12
Ok, it’s not quite that simple…
![Page 13: Mongo for the Mission Critical Enterprise](https://reader033.vdocument.in/reader033/viewer/2022052903/5579f4e6d8b42abc2e8b4b0f/html5/thumbnails/13.jpg)
13
What’s the point? Centralized Auth
![Page 14: Mongo for the Mission Critical Enterprise](https://reader033.vdocument.in/reader033/viewer/2022052903/5579f4e6d8b42abc2e8b4b0f/html5/thumbnails/14.jpg)
14
![Page 15: Mongo for the Mission Critical Enterprise](https://reader033.vdocument.in/reader033/viewer/2022052903/5579f4e6d8b42abc2e8b4b0f/html5/thumbnails/15.jpg)
15
API Documentation
![Page 16: Mongo for the Mission Critical Enterprise](https://reader033.vdocument.in/reader033/viewer/2022052903/5579f4e6d8b42abc2e8b4b0f/html5/thumbnails/16.jpg)
16
Java API Method Example
![Page 17: Mongo for the Mission Critical Enterprise](https://reader033.vdocument.in/reader033/viewer/2022052903/5579f4e6d8b42abc2e8b4b0f/html5/thumbnails/17.jpg)
17
What about the other services?
Independently scalable based on usage
![Page 18: Mongo for the Mission Critical Enterprise](https://reader033.vdocument.in/reader033/viewer/2022052903/5579f4e6d8b42abc2e8b4b0f/html5/thumbnails/18.jpg)
18
Aggregation – Long running service operations
![Page 19: Mongo for the Mission Critical Enterprise](https://reader033.vdocument.in/reader033/viewer/2022052903/5579f4e6d8b42abc2e8b4b0f/html5/thumbnails/19.jpg)
19
Scaling the Architecture
![Page 20: Mongo for the Mission Critical Enterprise](https://reader033.vdocument.in/reader033/viewer/2022052903/5579f4e6d8b42abc2e8b4b0f/html5/thumbnails/20.jpg)
20
Scaling the Architecture
![Page 21: Mongo for the Mission Critical Enterprise](https://reader033.vdocument.in/reader033/viewer/2022052903/5579f4e6d8b42abc2e8b4b0f/html5/thumbnails/21.jpg)
21
Questions?
Thom Nichols
@thom_nic