continuous deployment into the unknown with … deployment into the unknown with ... ... continuous...
TRANSCRIPT
![Page 1: Continuous Deployment into the Unknown with … Deployment into the Unknown with ... ... Continuous Deployment into the Unknown with Artifactory, Bintray, Docker](https://reader031.vdocument.in/reader031/viewer/2022022013/5b2ea0a07f8b9ae16e8c7c95/html5/thumbnails/1.jpg)
© 2015 VMware Inc. All rights reserved.
Continuous Deployment into the Unknown with Artifactory, Bintray, Docker and Mesos
Gilad GaronKiril Nesenko
![Page 2: Continuous Deployment into the Unknown with … Deployment into the Unknown with ... ... Continuous Deployment into the Unknown with Artifactory, Bintray, Docker](https://reader031.vdocument.in/reader031/viewer/2022022013/5b2ea0a07f8b9ae16e8c7c95/html5/thumbnails/2.jpg)
Agenda
• What is the Common SaaS Platform (CSP)
• CI/CD processes for CSP
• Upgrading CSP
• Xenon - Distributed Control Plane (If we have the time)
2
![Page 3: Continuous Deployment into the Unknown with … Deployment into the Unknown with ... ... Continuous Deployment into the Unknown with Artifactory, Bintray, Docker](https://reader031.vdocument.in/reader031/viewer/2022022013/5b2ea0a07f8b9ae16e8c7c95/html5/thumbnails/3.jpg)
Who are we ?
3
Kiril Nesenko
DevOps Lead
Gilad Garon
Architect
[email protected] , Twitter @giladgaron
![Page 4: Continuous Deployment into the Unknown with … Deployment into the Unknown with ... ... Continuous Deployment into the Unknown with Artifactory, Bintray, Docker](https://reader031.vdocument.in/reader031/viewer/2022022013/5b2ea0a07f8b9ae16e8c7c95/html5/thumbnails/4.jpg)
VMware’s SaaS Transition
• VMware is developing many SaaS offerings
• Many services have the same common requirements (Billing, Identity, etc.)
• Like other good engineers, we like to reuse code wherever possible
• VMware’s Common SaaS Platform (CSP) is platform that internal SaaS offerings are using to leverage existing internal components
4
![Page 5: Continuous Deployment into the Unknown with … Deployment into the Unknown with ... ... Continuous Deployment into the Unknown with Artifactory, Bintray, Docker](https://reader031.vdocument.in/reader031/viewer/2022022013/5b2ea0a07f8b9ae16e8c7c95/html5/thumbnails/5.jpg)
Designing a SaaS platform
Design Principles
5
Cloud Agnostic
Highly Available
Scalable
Great Public APIs
Modular
In Practice
Infrastructure needs to support containers
Dynamic, Stateful and Distributed cluster
Tunable consistency helps to achieve availability & scalability
No internal APIs
Capabilities as libraries, Coupling is done with APIs
Ease of operability /
development
Single JAR, limited classpath dependencies set
![Page 6: Continuous Deployment into the Unknown with … Deployment into the Unknown with ... ... Continuous Deployment into the Unknown with Artifactory, Bintray, Docker](https://reader031.vdocument.in/reader031/viewer/2022022013/5b2ea0a07f8b9ae16e8c7c95/html5/thumbnails/6.jpg)
Deployment Architecture. yep that’s it.
6
Xenon Host Jar
Container
Xenon Host Jar
Container
Xenon Host Jar
Container
Xenon Host Jar
Container
Some Cloud Provider Inc.
![Page 7: Continuous Deployment into the Unknown with … Deployment into the Unknown with ... ... Continuous Deployment into the Unknown with Artifactory, Bintray, Docker](https://reader031.vdocument.in/reader031/viewer/2022022013/5b2ea0a07f8b9ae16e8c7c95/html5/thumbnails/7.jpg)
Infrastructure and Patch Life Cycle
![Page 8: Continuous Deployment into the Unknown with … Deployment into the Unknown with ... ... Continuous Deployment into the Unknown with Artifactory, Bintray, Docker](https://reader031.vdocument.in/reader031/viewer/2022022013/5b2ea0a07f8b9ae16e8c7c95/html5/thumbnails/8.jpg)
CI/CD Overview
8
Customer 1 Customer N
Customer 2
automation R&D production
promotion
deploy&test
staging
![Page 9: Continuous Deployment into the Unknown with … Deployment into the Unknown with ... ... Continuous Deployment into the Unknown with Artifactory, Bintray, Docker](https://reader031.vdocument.in/reader031/viewer/2022022013/5b2ea0a07f8b9ae16e8c7c95/html5/thumbnails/9.jpg)
CSP Mesos Infrastructure
9
![Page 10: Continuous Deployment into the Unknown with … Deployment into the Unknown with ... ... Continuous Deployment into the Unknown with Artifactory, Bintray, Docker](https://reader031.vdocument.in/reader031/viewer/2022022013/5b2ea0a07f8b9ae16e8c7c95/html5/thumbnails/10.jpg)
CI/CD Tools
• Artifacts: Artifactory, Bintray
• CI: Jenkins
• Source Control: git
• Code review: gerrit
• Slaves: dockers
• Infrastructure: mesos, dockers
• Code Analysis: Sonar
• Build: gradle, Makefiles
• Languages: Java, JS, Python, Go
• Communication: Slack
10
![Page 11: Continuous Deployment into the Unknown with … Deployment into the Unknown with ... ... Continuous Deployment into the Unknown with Artifactory, Bintray, Docker](https://reader031.vdocument.in/reader031/viewer/2022022013/5b2ea0a07f8b9ae16e8c7c95/html5/thumbnails/11.jpg)
CI Infrastructure
• ~300 jenkins jobs
• 20 git repositories
• On the fly jenkins slaves
• Jenkins and Slack integration
• Mesos cluster (Marathon, marathon-lb, mesos-dns, Calico, chronos)
11
![Page 12: Continuous Deployment into the Unknown with … Deployment into the Unknown with ... ... Continuous Deployment into the Unknown with Artifactory, Bintray, Docker](https://reader031.vdocument.in/reader031/viewer/2022022013/5b2ea0a07f8b9ae16e8c7c95/html5/thumbnails/12.jpg)
Jenkins Jobs Management
![Page 13: Continuous Deployment into the Unknown with … Deployment into the Unknown with ... ... Continuous Deployment into the Unknown with Artifactory, Bintray, Docker](https://reader031.vdocument.in/reader031/viewer/2022022013/5b2ea0a07f8b9ae16e8c7c95/html5/thumbnails/13.jpg)
Jenkins Job Builder
13
Jenkins job builder to the rescue!
![Page 14: Continuous Deployment into the Unknown with … Deployment into the Unknown with ... ... Continuous Deployment into the Unknown with Artifactory, Bintray, Docker](https://reader031.vdocument.in/reader031/viewer/2022022013/5b2ea0a07f8b9ae16e8c7c95/html5/thumbnails/14.jpg)
Jenkins Job Builder
• Developed by OpenStack folks
• Configuration as code (yaml format)
• Easy to review changes
• Configuration de-duplication
• Include shell/groovy/python… scripts
• Test before deploying
• Easier to organize (per directory, per file)
• Serves as backup (easy to replicate to another jenkins)
14
![Page 15: Continuous Deployment into the Unknown with … Deployment into the Unknown with ... ... Continuous Deployment into the Unknown with Artifactory, Bintray, Docker](https://reader031.vdocument.in/reader031/viewer/2022022013/5b2ea0a07f8b9ae16e8c7c95/html5/thumbnails/15.jpg)
15
![Page 16: Continuous Deployment into the Unknown with … Deployment into the Unknown with ... ... Continuous Deployment into the Unknown with Artifactory, Bintray, Docker](https://reader031.vdocument.in/reader031/viewer/2022022013/5b2ea0a07f8b9ae16e8c7c95/html5/thumbnails/16.jpg)
16
![Page 17: Continuous Deployment into the Unknown with … Deployment into the Unknown with ... ... Continuous Deployment into the Unknown with Artifactory, Bintray, Docker](https://reader031.vdocument.in/reader031/viewer/2022022013/5b2ea0a07f8b9ae16e8c7c95/html5/thumbnails/17.jpg)
Templates
• For nearly identical jobs better to use templates
17
![Page 18: Continuous Deployment into the Unknown with … Deployment into the Unknown with ... ... Continuous Deployment into the Unknown with Artifactory, Bintray, Docker](https://reader031.vdocument.in/reader031/viewer/2022022013/5b2ea0a07f8b9ae16e8c7c95/html5/thumbnails/18.jpg)
Templates
18
![Page 19: Continuous Deployment into the Unknown with … Deployment into the Unknown with ... ... Continuous Deployment into the Unknown with Artifactory, Bintray, Docker](https://reader031.vdocument.in/reader031/viewer/2022022013/5b2ea0a07f8b9ae16e8c7c95/html5/thumbnails/19.jpg)
Jobs Update
19
![Page 20: Continuous Deployment into the Unknown with … Deployment into the Unknown with ... ... Continuous Deployment into the Unknown with Artifactory, Bintray, Docker](https://reader031.vdocument.in/reader031/viewer/2022022013/5b2ea0a07f8b9ae16e8c7c95/html5/thumbnails/20.jpg)
20
![Page 21: Continuous Deployment into the Unknown with … Deployment into the Unknown with ... ... Continuous Deployment into the Unknown with Artifactory, Bintray, Docker](https://reader031.vdocument.in/reader031/viewer/2022022013/5b2ea0a07f8b9ae16e8c7c95/html5/thumbnails/21.jpg)
Jenkins Jobs Types
• Gating – listens for patch-set-created events
• Build – for building purposes (gradle, docker etc)
• Listeners – listens for change-merged events on gerrit (orchestrators for the pipelines)
21
![Page 22: Continuous Deployment into the Unknown with … Deployment into the Unknown with ... ... Continuous Deployment into the Unknown with Artifactory, Bintray, Docker](https://reader031.vdocument.in/reader031/viewer/2022022013/5b2ea0a07f8b9ae16e8c7c95/html5/thumbnails/22.jpg)
Gating Jobs
• For each patch we run a gating job
• Each git project has its own gating job
• Build + test + post results to gerrit
22
![Page 23: Continuous Deployment into the Unknown with … Deployment into the Unknown with ... ... Continuous Deployment into the Unknown with Artifactory, Bintray, Docker](https://reader031.vdocument.in/reader031/viewer/2022022013/5b2ea0a07f8b9ae16e8c7c95/html5/thumbnails/23.jpg)
Gating Jobs
23
Developer sends a patch Run build and tests(gating)
Post results to gerritMerge ?
Start build pipeline(listener)
![Page 24: Continuous Deployment into the Unknown with … Deployment into the Unknown with ... ... Continuous Deployment into the Unknown with Artifactory, Bintray, Docker](https://reader031.vdocument.in/reader031/viewer/2022022013/5b2ea0a07f8b9ae16e8c7c95/html5/thumbnails/24.jpg)
Gerrit
• web-based code review tool built on top of the git
24
![Page 25: Continuous Deployment into the Unknown with … Deployment into the Unknown with ... ... Continuous Deployment into the Unknown with Artifactory, Bintray, Docker](https://reader031.vdocument.in/reader031/viewer/2022022013/5b2ea0a07f8b9ae16e8c7c95/html5/thumbnails/25.jpg)
Jenkins Failure
25
![Page 26: Continuous Deployment into the Unknown with … Deployment into the Unknown with ... ... Continuous Deployment into the Unknown with Artifactory, Bintray, Docker](https://reader031.vdocument.in/reader031/viewer/2022022013/5b2ea0a07f8b9ae16e8c7c95/html5/thumbnails/26.jpg)
Sonar Failure
26
![Page 27: Continuous Deployment into the Unknown with … Deployment into the Unknown with ... ... Continuous Deployment into the Unknown with Artifactory, Bintray, Docker](https://reader031.vdocument.in/reader031/viewer/2022022013/5b2ea0a07f8b9ae16e8c7c95/html5/thumbnails/27.jpg)
Gerrit Failure
Gerrit hooks
• Executed on the server side
• Execute per event type
• Various checks: commit message style, trailing white spaces, etc.
• Integrations with external systems: bugzilla, jira, etc.
27
![Page 28: Continuous Deployment into the Unknown with … Deployment into the Unknown with ... ... Continuous Deployment into the Unknown with Artifactory, Bintray, Docker](https://reader031.vdocument.in/reader031/viewer/2022022013/5b2ea0a07f8b9ae16e8c7c95/html5/thumbnails/28.jpg)
CONFIDENTIAL 28
![Page 29: Continuous Deployment into the Unknown with … Deployment into the Unknown with ... ... Continuous Deployment into the Unknown with Artifactory, Bintray, Docker](https://reader031.vdocument.in/reader031/viewer/2022022013/5b2ea0a07f8b9ae16e8c7c95/html5/thumbnails/29.jpg)
Dynamic Pipelines
![Page 30: Continuous Deployment into the Unknown with … Deployment into the Unknown with ... ... Continuous Deployment into the Unknown with Artifactory, Bintray, Docker](https://reader031.vdocument.in/reader031/viewer/2022022013/5b2ea0a07f8b9ae16e8c7c95/html5/thumbnails/30.jpg)
Listener Jobs
• Executed on patch-merged event
• Orchestrating the build and delivery pipeline dynamically
• Orchestration done via the BuildFlow plugin (groovy)
• All listeners run the same code base
• On failure, user is notified on slack channel
30
![Page 31: Continuous Deployment into the Unknown with … Deployment into the Unknown with ... ... Continuous Deployment into the Unknown with Artifactory, Bintray, Docker](https://reader031.vdocument.in/reader031/viewer/2022022013/5b2ea0a07f8b9ae16e8c7c95/html5/thumbnails/31.jpg)
31
![Page 32: Continuous Deployment into the Unknown with … Deployment into the Unknown with ... ... Continuous Deployment into the Unknown with Artifactory, Bintray, Docker](https://reader031.vdocument.in/reader031/viewer/2022022013/5b2ea0a07f8b9ae16e8c7c95/html5/thumbnails/32.jpg)
32
Dynamic Flows
CONFIDENTIAL 32
Listener - 1
Listener - 2
Listener - n
war
Jar
doc
docker
Test2
Mesos
2
cont Mesos
1
Listeners Build Deploy
…
Test
Test1
RPublish
Upload
LPublish
Bintray
Repo
![Page 33: Continuous Deployment into the Unknown with … Deployment into the Unknown with ... ... Continuous Deployment into the Unknown with Artifactory, Bintray, Docker](https://reader031.vdocument.in/reader031/viewer/2022022013/5b2ea0a07f8b9ae16e8c7c95/html5/thumbnails/33.jpg)
Parallel Deployments
33
Automation R&D Staging Production
![Page 34: Continuous Deployment into the Unknown with … Deployment into the Unknown with ... ... Continuous Deployment into the Unknown with Artifactory, Bintray, Docker](https://reader031.vdocument.in/reader031/viewer/2022022013/5b2ea0a07f8b9ae16e8c7c95/html5/thumbnails/34.jpg)
34
![Page 35: Continuous Deployment into the Unknown with … Deployment into the Unknown with ... ... Continuous Deployment into the Unknown with Artifactory, Bintray, Docker](https://reader031.vdocument.in/reader031/viewer/2022022013/5b2ea0a07f8b9ae16e8c7c95/html5/thumbnails/35.jpg)
35
![Page 36: Continuous Deployment into the Unknown with … Deployment into the Unknown with ... ... Continuous Deployment into the Unknown with Artifactory, Bintray, Docker](https://reader031.vdocument.in/reader031/viewer/2022022013/5b2ea0a07f8b9ae16e8c7c95/html5/thumbnails/36.jpg)
CONFIDENTIAL 36
![Page 37: Continuous Deployment into the Unknown with … Deployment into the Unknown with ... ... Continuous Deployment into the Unknown with Artifactory, Bintray, Docker](https://reader031.vdocument.in/reader031/viewer/2022022013/5b2ea0a07f8b9ae16e8c7c95/html5/thumbnails/37.jpg)
Upgrading a Stateful platform
Goals:
• Minimal service interruptions
• Support schema changes
Challenges:
• Symmetrical cluster: Can’t refactor / add API paths
• State & Business Logic in the same tier: can’t separate schema upgrade from BL changes
37
![Page 38: Continuous Deployment into the Unknown with … Deployment into the Unknown with ... ... Continuous Deployment into the Unknown with Artifactory, Bintray, Docker](https://reader031.vdocument.in/reader031/viewer/2022022013/5b2ea0a07f8b9ae16e8c7c95/html5/thumbnails/38.jpg)
Upgrading a Stateful platform
Design:
• Work in cycles, get meaningful metrics per cycle
• Each cycle migrates and transforms state
• Use a Threshold to determine progress and cutoff point
• Smartly queue external traffic
• Reroute traffic to new cluster
38
![Page 39: Continuous Deployment into the Unknown with … Deployment into the Unknown with ... ... Continuous Deployment into the Unknown with Artifactory, Bintray, Docker](https://reader031.vdocument.in/reader031/viewer/2022022013/5b2ea0a07f8b9ae16e8c7c95/html5/thumbnails/39.jpg)
39
Node
Node
Node
Node
Node
Node
Blue NodeGroup
Green NodeGroup
{
“documents”:”15M”,“duration”:”25S”
}
{
“documents”:”15M”,“duration”:”25S”
}
{
“documents”:”6M”,“duration”:”5S”
}
{
“documents”:”6M”,“duration”:”5S”
}
{
“documents”:”90K”,
“duration”:”0.5S”}
{
“documents”:”90K”,
“duration”:”0.5S”}
External
Clients
{
“documents”:”10K”,
“duration”:”0.2S”}
![Page 40: Continuous Deployment into the Unknown with … Deployment into the Unknown with ... ... Continuous Deployment into the Unknown with Artifactory, Bintray, Docker](https://reader031.vdocument.in/reader031/viewer/2022022013/5b2ea0a07f8b9ae16e8c7c95/html5/thumbnails/40.jpg)
Xenon – Distributed Control Plane
• A design pattern and runtime for scalable orchestration and management logic
• A runtime powering tiny REST services
• IO Pipeline integrates key building blocks within each service operation
• Production ready code with continuous integration tests, design documents
40
https://github.com/vmware/xenon
![Page 41: Continuous Deployment into the Unknown with … Deployment into the Unknown with ... ... Continuous Deployment into the Unknown with Artifactory, Bintray, Docker](https://reader031.vdocument.in/reader031/viewer/2022022013/5b2ea0a07f8b9ae16e8c7c95/html5/thumbnails/41.jpg)
The Popular WayStand up N nodes for each of:
• Orchestration code & container (Spring Boot)
• Your HA persistency layer (Cassandra, Mongo)
• Your translation layer (ORM)
• Your arbitration/leader election (ZK, etcd, consul)
• Your UI server (node.js, tomcat, apache)
• Your cache layer (Redis, memcached)
• Your message bus, event broker
![Page 42: Continuous Deployment into the Unknown with … Deployment into the Unknown with ... ... Continuous Deployment into the Unknown with Artifactory, Bintray, Docker](https://reader031.vdocument.in/reader031/viewer/2022022013/5b2ea0a07f8b9ae16e8c7c95/html5/thumbnails/42.jpg)
The Xenon WayStand up N nodes running Xenon services:
• Orchestration as stateless or stateful REST endpoints
• Persist, replicate state independently
• Manage concurrency with a single JVM and one thread per core across ALL services
• Provide per operation owner selection (leader)
• Pub / Sub
• Stats
• UI
• Tracing
![Page 43: Continuous Deployment into the Unknown with … Deployment into the Unknown with ... ... Continuous Deployment into the Unknown with Artifactory, Bintray, Docker](https://reader031.vdocument.in/reader031/viewer/2022022013/5b2ea0a07f8b9ae16e8c7c95/html5/thumbnails/43.jpg)
Links
• Jenkins Jobs Builder - http://docs.openstack.org/infra/jenkins-job-builder
• Xenon - https://github.com/vmware/xenon
43
![Page 44: Continuous Deployment into the Unknown with … Deployment into the Unknown with ... ... Continuous Deployment into the Unknown with Artifactory, Bintray, Docker](https://reader031.vdocument.in/reader031/viewer/2022022013/5b2ea0a07f8b9ae16e8c7c95/html5/thumbnails/44.jpg)
Thank you!
44
Q&A
![Page 45: Continuous Deployment into the Unknown with … Deployment into the Unknown with ... ... Continuous Deployment into the Unknown with Artifactory, Bintray, Docker](https://reader031.vdocument.in/reader031/viewer/2022022013/5b2ea0a07f8b9ae16e8c7c95/html5/thumbnails/45.jpg)
Decentralized Model
• Scalable to lots of nodes
– SWIM node discovery and maintenance
– Replication with Eventual OR Strong Consistency (choose!)
• Every node in a node group has the same core services
– Operational simplicity
![Page 46: Continuous Deployment into the Unknown with … Deployment into the Unknown with ... ... Continuous Deployment into the Unknown with Artifactory, Bintray, Docker](https://reader031.vdocument.in/reader031/viewer/2022022013/5b2ea0a07f8b9ae16e8c7c95/html5/thumbnails/46.jpg)
Indexing/Queries
• Multi version, fully indexed, replicated document store
– Lucene!
• Query services with rich document query support modeled as tasks
– Real time or historical
• Collections are just queries
![Page 47: Continuous Deployment into the Unknown with … Deployment into the Unknown with ... ... Continuous Deployment into the Unknown with Artifactory, Bintray, Docker](https://reader031.vdocument.in/reader031/viewer/2022022013/5b2ea0a07f8b9ae16e8c7c95/html5/thumbnails/47.jpg)
Programming Model• Isolated, asynchronous components listening on URIs
• Each service instance represents a “living” document
– All side effects happen through REST actions on document
– Replication, consensus, notifications all leveraging symmetric model
• Stateless handlers are offered latest state and request body
• Developer declares requirements through Service options
– Replication with Strong (Eager) or Eventual consistency
– Scale out (Owner selection)
– Instrumentation
– Persistence (with deep indexing)
– And more …