managing microservices with neo4j
TRANSCRIPT
Managing Microservices with Neo4j
Rob SchoeningAshley Sun
Overview
• Intro• MacGyver + Neo4j• Use Cases• Q&A
• World’s largest online credit marketplace– Peer-to-peer personal loans– Small business loans– Medical financing
Micro ServicesAll the cool kids are doing it…
…and it makes complex softwareeasier to write and maintain
A long, long time ago…2 years ago…We ran 5 production services…Now we run…
139
That’s a lot of stuff to manage!
Continuously!
How do we do it?
Excel
ExcelJust Kidding
Neo4j, of course!
So, like, seriously, how do we do it?
MacGyver
What is MacGyver?
“Is that like MacGruber?”
What is MacGyver?
“Is that like MacGruber?”
No, but it’s on GitHub!
MacGyver provides…• What we can’t easily accomplish with Jenkins• API Integration• A batch scheduler• Data aggregation• An extensible UI• Tools for engineering, operations, infosec, QA
and finance• Performs monitoring and alerting
Integrations
MacGyver+
Neo4j
MacGyver+
Neo4j
Jenkins
GitHub
Load Balancer
Pager Duty SAN Storage
MacGyver+
Neo4j
Jenkins
GitHub AWSJira
Confluence HipChat
Active Directory
Pager Duty
New RelicLoad Balancer
Pager Duty SAN Storage
Graphite
MacGyver+
Neo4j
Jenkins
GitHub AWS
Artifactory
Jira
Confluence HipChat
VMWare
Active Directory
Pager Duty
New RelicLoad Balancer
Pager Duty
Catchpoint
SAN Storage
Graphite
Splunk
Why Neo4j?
• Perfect for capturing/modeling interdependencies
• Cypher’s ad hoc query capability can’t be beat• Easy to extend, build more relationships and
layers incrementally• JSON native, FTW
How Does DevOps at Lending Club Use Neo4j?
1. App check-in/service discovery2. Deployment and release automation 3. Infrastructure mapping
How Does DevOps at Lending Club Use Neo4j?
1. App check-in/service discovery2. Deployment and release automation 3. Infrastructure mapping
App Check-In/Service Discovery
Problem: Keeping track of many rapidly-changing services
Solution: All app servers phone home to MacGyver and are stored in Neo4j as ‘App Instance’ nodes
App Check-In/Service Discovery
Neo4j
App Instance
App Instance
App Instance
MacGyver
App Check-In/Service Discovery
MacGyver
Neo4j
App Instance
App Instance
App Instance
<<REST>><<REST>>
App Check-In/Service Discovery
Real time database of deployed infrastructureNew services get auto-discovered by MacGyverLow maintenanceEasy scalabilityLow latency ad hoc query capability
App Check-In/Service Discovery
“Show me all the instances of app X in environment Y”
How Does DevOps at Lending Club Use Neo4j?
1. App check-in/service discovery2. Deployment and release automation 3. Infrastructure mapping
Deployment and Release Automation
Problem: Highly manual and tedious releases Difficult to answer questions like:– What pool should I deploy to?– Is the most recent revision ‘live’ right now?– Are live pool revisions in sync in different
environments?
Deployment and Release Automation
Solution: Utilize app check-ins and Neo4j to expose info about live and dark pools, enabling us to automate deployments in a release
Deployment and Release Automation
• Blue-green deployment
Server 1
Server 2
Server 3
Server 5
Server 4
Server 6
Server 7
Server 8
Service Group
“Live” Pool “Dark” Pool
Deployment and Release Automation
• Blue-green deployment
Server 1
Server 2
Server 3
Server 5
Server 4
Server 6
Server 7
Server 8
Service Group
DEPLOY
Deployment and Release Automation
• Blue-green deployment
Server 1
Server 2
Server 3
Server 5
Server 4
Server 6
Server 7
Server 8
Service Group
DEPLOY
“Draining” Pool “Live” Pool
Deployment and Release Automation
• Blue-green deployment
Server 1
Server 2
Server 3
Server 5
Server 4
Server 6
Server 7
Server 8
Service Group
DEPLOY
Pool Cut-over
App Instances
MacGyver
Neo4j
<<update app instance>>MacGyver
Virtual Servers
MacGyver
Neo4j
Load Balancer<<REST API Polling>>
Virtual Servers
MacGyver
Neo4j
<<update virtual server>>
Load Balancer<<REST API Polling>>
Application Pools
Virtual Service
Deployment and Release Automation
Problem: Highly manual and tedious releases Difficult to answer questions like:– What pool should I deploy to?– Is the most recent revision ‘live’ right now?– Are live pool revisions in sync in different
environments?
Deployment and Release Automation
“What pool should I deploy to?”
Deployment and Release Automation
“Are live pool revisions in sync in different environments?”
Deployment and Release Automation
“Do multiple revisions exist within a single pool?”
Deployment and Release Automation
“Do multiple revisions exist within a single pool?”
Think about the Knight Capital meltdown in 2012!
App Instance
App Instance
App Instance
App Instance
Deployment and Release Automation
MacGyver
Neo4j
Load Balancer<<PUT>>
<<REST API>>
<<REST API>>
How Does DevOps at Lending Club Use Neo4j?
1. App check-in/service discovery2. Deployment and release automation 3. Infrastructure mapping
“Are any servers in a live pool degraded?”
“Do we have a single point of failure among any of our services?”
“If storage volume #3 goes down, what services will be impacted?”
Other Use Cases
• “Are servers in the correct security zones?”• “What is the correct AWS VPC placement for
this application?”• Deliver abstraction across multiple Load
Balancer implementations• etc.
For More Info
• https://github.com/if6was9/macgyver• https://macgyver.readme.io• https://github.com/if6was9/neorx
• Ashley Sun – [email protected], @ashleycsun
• Rob Schoening - [email protected], @rschoening
Questions