managing micro-services with neo4j

10.21.15

Managing Microservices with Neo4j

World’s largest credit marketplacePeer-to-peer personal loans Small business loans Medical financing

@lendingclub @ashleycsun @rschoening asun@lendingclub.com rschoening@lendingclub.com

Microservices

all the cool kids are doing it

Microservices

2013 2015

Because no talk on microservices would be complete without a container analogy

Can you check the load balancer?

How do we do it?

That’s not a joke

Let’s explain

Separated at Birth

Integrations

MacGyver

MacGyver

GitHub

Pager Duty

Jira

AWS

HipChatSplunk

MacGyver

GitHub

Pager Duty

Jenkins

Jira

AWS

New RelicGraphite

HipChatConfluence

Splunk

MacGyver

GitHub

SAN Storage

Pager Duty

Jenkins

Jira

AWS

New Relic

Active Directory

Graphite

HipChatConfluence

VMWare

Splunk

CloudStack

MacGyver

GitHub

SAN Storage

Load Balancer

Pager Duty

Jenkins

Jira

AWS

New Relic

Active Directory

Graphite

HipChatConfluence

VMWare

SplunkCatchpoint

Artifactory

Firewall

CloudStack

Operational Data Is Distributed

Operational Data Is Distributed

but real-time access is costly

Get It Together

Desired Database Characteristics

Near Real Time Low Latency Read Join Across Data Sets Ease of Use Ad Hoc Query

Desired Database Characteristics

Near Real Time Low Latency Read Join Across Data Sets Ease of Use Ad Hoc Query

How do we use it?

App Check-In and

Service Discovery

Problem: Keeping track of many rapidly-changing services

Problem: Keeping track of many rapidly-changing services

Solution: All app servers phone home to MacGyver and are stored in Neo4j as ‘App Instance’ nodes

MacGyver

Neo4j

App Instance

App Instance

App Instance

MacGyver

Neo4j

App Instance

App Instance

App Instance

<<REST>> <<REST>>

Real-time database of deployed infrastructure

Real-time database of deployed infrastructure

New services get auto-discovered by MacGyver

Real-time database of deployed infrastructure

New services get auto-discovered by MacGyver

Low maintenance

Real-time database of deployed infrastructure

New services get auto-discovered by MacGyver

Low maintenance

Easy scalability

Real-time database of deployed infrastructure

New services get auto-discovered by MacGyver

Low maintenance

Easy scalability

Low latency ad hoc query capability

“Show me all the instances of app X in environment Y”

“Show me all the instances of app X in environment Y”

Deployment and Release Automation

Problem: Highly manual and tedious releases

Problem: Highly manual and tedious releases Limited service visibility

Problem: Highly manual and tedious releases Limited service visibility

• What pool should I deploy to? • Is the most recent revision ‘live’ right now? • Are live pool revisions in sync in different

environments?

Problem: Highly manual and tedious releases Limited service visibility

Solution: Utilize app check-ins and Neo4j to expose info about live

and dark pools, enabling us to automate deployments in a release

Blue-Green Deployment

“Live” Pool “Dark” Pool

Server 1

Server 2

Server 3

Server 5

Server 7

Server 4 Server 8

Server 6

Service Group

Blue-Green Deployment

“Live” Pool “Dark” Pool

Server 1

Server 2

Server 3

Server 5

Server 7

Server 4 Server 8

Server 6

DEPLOY

Service Group

Blue-Green Deployment

“Draining” Pool

Server 1

Server 2

Server 3

Server 5

Server 7

Server 4 Server 8

Server 6

DEPLOY

Service Group

Blue-Green Deployment

Pool Cut-Over

Server 1

Server 2

Server 3

Server 5

Server 7

Server 4 Server 8

Server 6

DEPLOY

Service Group

Blue-Green Deployment

Dark Pool Live Pool

Server 1

Server 2

Server 3

Server 5

Server 7

Server 4 Server 8

Server 6

Service Group

App Instance

MacGyver

Neo4j

<<update app instance>>

Virtual Server

MacGyver

Neo4j

<<REST API Polling>> Load Balancer

Virtual Server

MacGyver

Neo4j

<<REST API Polling>> Load Balancer

<<update virtual server>>

Application Pools

Virtual Service

Problem: Highly manual and tedious releases Limited service visibility

• What pool should I deploy to? • Is the most recent revision ‘live’ right now? • Are live pool revisions in sync in different

environments?

What pool should I deploy to?

What pool should I deploy to?

Are live pool revisions in sync in different environments?

Are live pool revisions in sync in different environments?

Do multiple revisions exist within a single pool?

Do multiple revisions exist within a single pool?

MacGyver

Neo4j

<<REST API>> Load Balancer

<<REST API>>

App Instance

App Instance

App Instance

MacGyver

Neo4j

<<REST API>> Load Balancer

<<REST API>>

App Instance

App Instance

App Instance

Infrastructure Mapping

Problem: Limited visibility into infrastructure interdependencies

Problem: Limited visibility into infrastructure interdependencies

Solution: Use Neo4j to connect different components of our infrastructure

Are any servers in a live pool degraded?

Are any servers in a live pool degraded?

Do we have a single point of failure among any of our services?

Do we have a single point of failure among any of our services?

If storage volume #3 goes down, what services will be impacted?

If storage volume #3 goes down, what services will be impacted?

Other Use Cases

Because no talk on microservices would be complete without a lego

analogy

https://github.com/if6was9/macgyver https://github.com/if6was9/neorx

For More Info…

Lending Club Ashley Sun Rob Schoening @lendingclub @ashleycsun @rschoening

asun@lendingclub.com rschoening@lendingclub.com