extending cloud foundry - london cf user group sept 2014.pptx
DESCRIPTION
Extending Cloud Foundry with Open Source Toys. A user group talk covering reliably deploying and testing Cloud Foundry using Jenkins and seamlessly and scalably monitoring your Cloud Foundry Deployments using Sensu. Presented by Troy Astle and Matt Johnson at the September 2014 Cloud Foundry London MeetupTRANSCRIPT
© 2013 Cisco and/or its affiliates. All rights reserved.
Cisco ConfidentialCisco Confidential
© 2013 Cisco and/or its affiliates. All rights reserved.
Extending Cloud Foundry withOpen Source toys
Troy Astle Matt JohnsonCisco UK Cisco UK@buggalo @mattdashj
©Cisco System 2014 - This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
© 2013 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
© 2014 Cisco. Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Who are we?Cisco
Cloud Security Engineers
Unholy dev/ops combo
©Cisco System 2014 - This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
© 2013 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
© 2014 Cisco. Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Why are we using Cloud Foundry?
We needed better tools
Consistent dev, test & prod environments
Custom built tooling outdated
Speed up development
©Cisco System 2014 - This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
© 2013 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
© 2014 Cisco. Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Starting out with Cloud Foundry
Trial and Error
Learning by debugging
“Borrowing” chunks of BOSH yaml.
Manually building and verifying CF deployments
©Cisco System 2014 - This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
© 2013 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
“I’ve got a running CloudFoundry system, and if you’ve ever tried to run CloudFoundry and BOSH, you may never have got to this point…”
Dr Nic Williams, CEO Stark & Wayne, CF Platform 2013
© 2013 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
© 2014 Cisco. Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
The BOSH learning curve
Day 1: Why do I need BOSH?
Day 2: I don’t want to learn BOSH
Day 10: BOSH actually works...
Day 12: BOSH is Cool
Day 30: BOSH is Awesome
Day 50: I got to get me some more BOSH
©Cisco System 2014 - This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
© 2013 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
BOSH Deployed successfully...But does it work?
© 2013 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
© 2014 Cisco. Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
We needed to reliably upgrade and test Cloud Foundry
Deploy, Test and Upgrade CF using Jenkins & BOSH
Separate CF manifest from environment manifest, merge with Spiff
Create new deployments using the same CF manifest, guarantee consistency
Project named C3CIJoint work between Cisco and CloudCredoAvailable on Github
©Cisco System 2014 - This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
© 2013 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
© 2014 Cisco. Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Abundance of CF Tests Available
Several Cloud Foundry test suites already available online
Older:
● YETI
● NYET
Newer:
● Smoke Tests
● CATS
We used the NYET and Smoke tests as they were the most reliable
©Cisco System 2014 - This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
© 2013 Cisco and/or its affiliates. All rights reserved.
Cisco ConfidentialCisco Confidential
© 2013 Cisco and/or its affiliates. All rights reserved.
BOSH + Jenkins + Spiff + NYET + Smoke = CF
C3CI JenkinsBOSH Release
Base CFmanifest
Environment specific manifest
CF BOSH Release
©Cisco System 2014 - This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
© 2013 Cisco and/or its affiliates. All rights reserved.
Cisco ConfidentialCisco Confidential
© 2013 Cisco and/or its affiliates. All rights reserved.
© 2013 Cisco and/or its affiliates. All rights reserved.
Cisco ConfidentialCisco Confidential
© 2013 Cisco and/or its affiliates. All rights reserved.
Consistency without development lockdown
Environment specific manifest
Prod Branch - Consistent base manifestOnly Environment specifics change
Dev Branch A Dev Branch B Dev Branch X
Branch allows development and testing of anypart of the deployment manifest
Base CF manifest
©Cisco System 2014 - This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
© 2013 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
Like an engine… Only with more moving parts
© 2013 Cisco and/or its affiliates. All rights reserved.
Cisco ConfidentialCisco Confidential
© 2013 Cisco and/or its affiliates. All rights reserved.
We needed to know what was going on insideCloud Foundry…
Without spending the evening with Wireshark.
© 2013 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
© 2014 Cisco. Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Requirements
Monitor & graph the vitals of Cloud Foundry VMs deployed with BOSH.
Alert if a part of Cloud Foundry is degraded or offline.
Integrate with the tools used to monitor our other VMs.
- ELK
- Puppet-deployed Services
©Cisco System 2014 - This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
© 2013 Cisco and/or its affiliates. All rights reserved.
Cisco ConfidentialCisco Confidential
© 2013 Cisco and/or its affiliates. All rights reserved.
Enter Sensu!
©Cisco System 2014 - This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Subscription
Checks
Control Queue
(Per Subscription)
Results Queue
(Shared)
Handlers
TCP3030
© 2013 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
© 2014 Cisco. Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
“What! no Nagios?”
Scalability
Dynamic Provisioning / Registration
Resilience
Extensibility
!(Nagios Love)
©Cisco System 2014 - This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
© 2013 Cisco and/or its affiliates. All rights reserved.
Cisco ConfidentialCisco Confidential
© 2013 Cisco and/or its affiliates. All rights reserved.
Monitoring Cloud Foundry with Sensu
Sensu Client Job
CollectD Job
Cloud Foundry Job(s)
©Cisco System 2014 - This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Individual VMs
© 2013 Cisco and/or its affiliates. All rights reserved.
Cisco ConfidentialCisco Confidential
© 2013 Cisco and/or its affiliates. All rights reserved.
© 2013 Cisco and/or its affiliates. All rights reserved.
Cisco ConfidentialCisco Confidential
© 2013 Cisco and/or its affiliates. All rights reserved.
© 2013 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
© 2014 Cisco. Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
How the Sensu BOSH release works
Sensu Client BOSH release deployed along with CF-Release
Sensu Client, CollectD and Nagios checks installed on all BOSH VMs
Sensu checks and CollectD metrics pushed to Sensu Server over RabbitMQ
Small Python application runs as a Sensu Check to report Monit job health
Uchiwa GUI is used to display Sensu Alerts
Sensu routes metric data to Graphite for Graphing
©Cisco System 2014 - This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
© 2013 Cisco and/or its affiliates. All rights reserved.
Cisco ConfidentialCisco Confidential
© 2013 Cisco and/or its affiliates. All rights reserved.
Thanks for having us
Troy Astle@buggalo
github.com/FreightTrain/C3CIgithub.com/FreightTrain/sensu-client-boshrelease
©Cisco System 2014 - This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Matt Johnson@mattdashj