neo4j for cloud management at scale
TRANSCRIPT
David Brian WardCEO, Telegraph Hill Software
www.telegraphhillsoftware.com535 Mission Street
San Francisco CA 94105
Neo4j for Cloud Management at Scale
Headline text will go right here
A Renaissance Of Database Innovation
Headline text will go right here
Big ITIL Framework, Asset Heavy IT
A shared CMDB as the single system of record is a key ITIL best practice for good reason.
Headline text will go right here
Pre-DevOps, Pre-Cloud Ops Engineering
Would you build an IT edifice like a cathedral today?Origins of DevOps Culture:• Open Source• Cloud Computing• Need for Dev Speed
Headline text will go right here
Early Cloud, Early DevOps Days
Shopping at the Open Source Bazaar – A new high quality DevOps tool every week?
Headline text will go right here
On Sale: DevOps for 90% Off!
A Graph Database Can Make It Possible!
Headline text will go right here
Evolving our DevOps Approach
Cloud DevOps should combine an integration hub architecture with deployment automation enabling the evolution of infra, apps and tools and their migration.
Headline text will go right here
MacGyver and Neo4J for Business Critical DevOps
• API Integration with infra, tools and apps• A batch scheduler for near real time config• Data aggregation across services• Monitoring and alerting integration• Java script execution• An extensible UI and dashboards• Infrastructure mapping• Tools for building engineering, operations,
infosec, QA and finance use cases• High availability and multi-site
o right here
MacGyver Service Basics
• Perfect for capturing/modeling interdependencies• Cypher’s ad hoc query capability can’t be beat• Easy to extend, build more relationships and layers
incrementally• Great join/traversal capability• Flexible and scalable vs rigid; dataset can easily evolve
and grow in terms of complexity and structure• Easy to consume• Natural for infrastructure mapping and enterprise
architecture• JSON native, FTW
Neo4J is Perfect for MacGyver
“Do we have a single point of failure among any of our services?”
WHO ARE OUR CUSTOMERS?
Continuous integration Test automation Release packagingHigh-availability/failoverServer, network and environment provisioningMonitoringApp performance monitoringCloud cost managementAudit
Change management, Continuous deployment, Service discovery Micro-service cloud migration Self healing systems System and security maintenance Operational cost and capacity
management Blue/green deployment, API management and security And more every day
MacGyver Use Cases at Lending Club
WHO ARE OUR CUSTOMERS?
Network: • A10• ASA (adaptive security appliance)• DNS• F5Software Code and Deployment:• Artifactory• Github• Jenkins• SpringframeworkAuthentication and Access:• AACS (advanced access content
system)• Ldap• Microsoft active directory & ssoCloud Service Providers:• Aws• Cloudstack• Vsphere
•User Interface & Dashboards
• Leftronic• Grafana
(graphite)•Notification and Collaboration
• Hipchat• Pagerduty• Smtp
•Operational Documentation
• Jira/Confluence•Database Platforms
• Mongo• JDBC (oracle,
mysql)•Performance Monitoring
• Catchpoint• L7• Newrelic
MacGyver Integrations at Lending Club
System Config and Management• Nimble• Purestorage• Puppet• SaltstackMonitoring and logs• Signalfx• SplunkMacGyver tools and services• ‘health check’ / bootstrap• Micro-service registry
Headline text will go right here
DevOps Evolution with MacGyver
Business Critical DevOps must anticipate the evolution of infra, apps and tools. E.g.,• Physical -> Vsphere -> AWS• Monolithic -> Micro Services• Jenkins -> AWS Code Deploy• Bonus: Vendor Independence
WHO ARE OUR CUSTOMERS?MacGyver Micro Services – Service Discovery
Problem: Keeping track of many rapidly-changing servicesSolution: All app servers phone home to MacGyver and are stored in Neo4j as ‘App Instance’ nodes. Deployment and release automation assure a real time database of deployed services. New services get auto-discovered by MacGyver.• Low maintenance• Easy scalability• Low latency ad hoc query capability
WHO ARE OUR CUSTOMERS?MacGyver Micro Services – Deployment
Problem:
Highly manual and tedious releases
Difficult to answer questions like:–What pool should I deploy to?–Is the most recent revision ‘live’ right now?–Are live pool revisions in sync in different environments?
Solution:Utilize app check-ins and Neo4j to expose info about live and dark pools, enabling us to automate deployments, and build on our existing monitoring automation.
Application Pools
Deployment and Release Automation
•Blue-green deployment
Server 1
Server 2
Server 3
Server 5
Server 4
Server 6
Server 7
Server 8
Service Group
“Live” Pool “Dark” Pool
Deployment and Release Automation
•Blue-green deployment
Server 1
Server 2
Server 3
Server 5
Server 4
Server 6
Server 7
Server 8
Service Group
DEPLOY
Deployment and Release Automation
•Blue-green deployment
Server 1
Server 2
Server 3
Server 5
Server 4
Server 6
Server 7
Server 8
Service Group
DEPLOY
“Draining” Pool “Live” Pool
Deployment and Release Automation
•Blue-green deployment
Server 1
Server 2
Server 3
Server 5
Server 4
Server 6
Server 7
Server 8
Service Group
DEPLOY
Pool Cut-over
Virtual Service Deployed
WHO ARE OUR CUSTOMERS?
280 production (micro)services
4338 hosts (+501 AWS EC2 instances)
2 sites (LAS & SJC)
4 environments (dev, demo, stage, prod)
108,917 graph nodes
5% #Lines of code compared to Netflix OSS
MacGyver Size and Scale at Lending Club
WHO ARE OUR CUSTOMERS?MacGyver UI views
WHO ARE OUR CUSTOMERS?MacGyver UI views
WHO ARE OUR CUSTOMERS?MacGyver UI views
WHO ARE OUR CUSTOMERS?
Headline text will go right here
What’s Next? Reactive DevOps
As with the F35 cockpit, the sky is no longer the limit.
CLOSING THOUGHTS• MacGyver provides an Integration Architecture that enables
scalability, enrichment, evolution and migration• MacGyver and Neo4J enables you to evolve your infrastructure
using best of breed components, all the while running your business critical systems with integrity.
• MacGyver dramatically reduces the amount of software development required for even the most sophisticated DevOps use cases.
• Perfect for managing your hybrid infrastructure while staying ahead of Dev.
Headline text will go right here
What’s Next? Reactive DevOps
As in the F35 cockpit, the sky is no longer the limit.
CLOSING THOUGHTS• MacGyver provides an Integration Architecture that enables
scalability, enrichment, evolution and migration• MacGyver and Neo4J enables you to evolve your infrastructure
using best of breed components, all the while running your business critical systems with integrity.
• MacGyver dramatically reduces the amount of software development required for even the most sophisticated DevOps use cases.
• Perfect for managing your hybrid infrastructure while staying ahead of Dev? We know so.
QUESTIONS? FOR MORE INFO…https://github.com/if6was9/macgyver
https://github.com/if6was9/neorxAshley Sun – [email protected], @ashleycsun
Rob Schoening - [email protected], @rschoeningDavid Ward – [email protected]
Sarah Lewis – [email protected], [email protected]
Extra slides
www.telegraphhillsoftware.com535 Mission Street
San Francisco CA 94105
Using Neo4j for Cloud Management at Scale
App Instances
MacGyver
Neo4j
<<update app instance>>
MacGyver
Virtual Servers
MacGyver
Neo4j
Load Balancer<<REST API Polling>>
Virtual Servers
MacGyver
Neo4j
<<update virtual server>>
Load Balancer<<REST API Polling>>
App Instance
App Instance
App Instance
App Instance
Deployment and Release Automation
MacGyver
Neo4j
Load Balancer<<PUT>>
<<REST API>>
<<REST API>>
