cna2080bu deep dive: how to deploy and operationalize or ...€¦ · deep dive: how to deploy and...
TRANSCRIPT
Cornelia Davis, Pivotal
Nathan Ness
Technical Product Manager, CNABU
@nvpnathan
CNA2080BU
#VMworld #CNA2080BU
Deep Dive: How to Deploy and Operationalize Kubernetes
VMworld 2017 Content: Not fo
r publication or distri
bution
• This presentation may contain product features that are currently under development.
• This overview of new technology represents no commitment from VMware to deliver these features in any generally available product.
• Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.
• Technical feasibility and market demand will affect final delivery.
• Pricing and packaging for any new technologies or features discussed or presented have not been determined.
Disclaimer
#CNA2080BU CONFIDENTIAL 2
VMworld 2017 Content: Not fo
r publication or distri
bution
Agenda
#CNA2080BU CONFIDENTIAL 3
1 What is the need?
2 Introducing the toolchain
3 Pivotal Container Service (PKS)
4 PKS Day 1
5 PKS Day 2VMworld 2017 Content: Not fo
r publication or distri
bution
The Need for Operationalizing Kubernetes
VMworld 2017 Content: Not fo
r publication or distri
bution
CONTAINERSEVENT-DRIVEN
FUNCTIONS
DATA SERVICES
MICROSERVICES
BATCHES
MONOLITHIC APPLICATIONS
Companies Have Many Ways to Package and Run Their Workloads in the Cloud
#CNA2080BU CONFIDENTIAL 5
VMworld 2017 Content: Not fo
r publication or distri
bution
Workloads that Might Be Suitable for Kubernetes
Those:
• Requiring Persistence
– MongoDB, CouchDB, Couchbase, Elastic Search, …
• Managed as a cluster
– nodes need to communicate with one another
– often with the help of service meshes such as Istio or Linkerd
– Spark, Elastic Search
• Needing new architectural primitives
• Misc things like multiple ports, etc.
#CNA2080BU CONFIDENTIAL 6
VMworld 2017 Content: Not fo
r publication or distri
bution
Workeretcd
etcd
Serving up Kubernetes Dial-tone
7
Kubernetes
etcd
kubectlRouting
MasterMaster
WorkerWorker
Responsible for the
workloads running
in K8s
Responsible for the
K8s cluster(s)
themselves
manage
#CNA2080BU CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
Operational Challenges with Any Platform
Patches Patching platform components with thousands of apps running should feel normal.
Scaling Seamlessly scale platform components to accommodate changing demand.
Upgrades How do you roll out new versions of the platform with the lights on?
Operating Effort Operating a platform should require very few resources and minimum manual intervention. Otherwise, is it really providing operational benefits?
Multi-cloud Provide a reliable and smooth experience for any cloud.
Open APIs Allow platform operations from different toolsets and the creation of CD pipelines.
Consistency Provide a consistent setup experience, across different cloud environment configurations.
Setup time How long does it take to setup a real world working environment? Think hours, not weeks.
Day 1 - Build Day 2 - Operate
#CNA2080BU CONFIDENTIAL 8
VMworld 2017 Content: Not fo
r publication or distri
bution
Kubernetes - Especially Hard to Operationalize
#CNA2080BU CONFIDENTIAL 9
High Availability. No out-of-the-box fault-tolerance for the cluster components themselves (masters, workers and etcd nodes).
Scaling. Kubernetes clusters handle scaling the pod/service within the Nodes, but doesn’t provide a mechanism to scale Workers, Masters & etcd VMs.
Health checks and healing. The Kubernetes cluster only does routine health checks for the health ofworkloads running on Nodes.
Upgrades. Rolling upgrades on a large fleet of clusters is hard. Who manages the system it runs on?
VMworld 2017 Content: Not fo
r publication or distri
bution
Introducing BOSH
VMworld 2017 Content: Not fo
r publication or distri
bution
Powered by BOSH
#CNA2080BU CONFIDENTIAL 11
Pivotal container service ops
BOSH is an open source tool
for release engineering,
deployment, lifecycle
management, and monitoring
of distributed systems.
BOSHPackaging w/ embedded OS
Server provisioning on any IaaS
Software deployment across availability zones
Health monitoring (server AND processes)
Self-healing w/ Resurrector
Storage management
Rolling upgrades via canaries
Easy scaling of clusters
VMworld 2017 Content: Not fo
r publication or distri
bution
Powered by BOSH
#CNA2080BU CONFIDENTIAL 12
Pivotal container service ops
Packaging w/ embedded OS
Server provisioning on any IaaS
Software deployment across availability zones
Health monitoring (server AND processes)
Self-healing w/ Resurrector
Storage management
Rolling upgrades via canaries
Easy scaling of clusters
Workeretcdetcd
Kubernetes
etcd
MasterMaster
WorkerWorker
BOSHVMworld 2017 Content: Not fo
r publication or distri
bution
Primary BOSH Entities
#CNA2080BU CONFIDENTIAL 13
Workeretcdetcd
etcd
MasterMaster
WorkerWorker
BOSH
The definition of each of the nodes in the
cluster, including:
• The bits installed on a node (packages)
• The processes started on a node (jobs)
The definition of each of the nodes in the
cluster, including:
• The bits installed on a node (packages)
• The processes started on a node (jobs)
The definition of each of the nodes in the
cluster, including:
• The bits installed on a node (packages)
• The processes started on a node (jobs)
• Parameterized
BOSH release
A declaration of the desired state of the
cluster:
• Assembly of the components from BOSH
releases (relationships, dependencies)
• Parameter values
BOSH deployment
Relationship to the underlying infrastructure
BOSH cloud config
Kubernetes
VMworld 2017 Content: Not fo
r publication or distri
bution
The Workflow
#CNA2080BU CONFIDENTIAL 14
Workeretcdetcd
etcd
MasterMaster
WorkerWorker
BOSH
The definition of each of the nodes in the
cluster, including:
• The bits installed on a node (packages)
• The processes started on a node (jobs)
The definition of each of the nodes in the
cluster, including:
• The bits installed on a node (packages)
• The processes started on a node (jobs)
The definition of each of the nodes in the
cluster, including:
• The bits installed on a node (packages)
• The processes started on a node (jobs)
• Parameterized
BOSH release
A declaration of the desired state of the
cluster:
• Assembly of the components from BOSH
releases (relationships, dependencies)
• Parameter values
BOSH deployment
Relationship to the underlying infrastructure
BOSH cloud configSTEP 1: Install and configure BOSH
STEP 2: Install and Manage Kubernetes
Kubernetes
VMworld 2017 Content: Not fo
r publication or distri
bution
Pivotal Container Service (PKS)
VMworld 2017 Content: Not fo
r publication or distri
bution
Project Kubo
#CNA2080BU CONFIDENTIAL 16
Uniform way to instantiate, deploy, and manage highly available Kubernetes clusters. On any cloud.
Launched by Pivotal & Google Feb 2017, Donated to Cloud Foundry Foundation June 2017
“Day 1” Build● Deploy Kubernetes cluster via BOSH
“Day 2” Operate● Self-healing VMs and monitoring via
BOSH● Elastic scaling for clusters● Rolling upgrades to latest Kubernetes
release● High-availability and multi-AZ supportVMworld 2017 Content: N
ot for publicatio
n or distribution
Workeretcdetcd
Kubernetes
etcd
MasterMaster
WorkerWorker
BOSH
This forms the
Open Core of
Pivotal Container Service(PKS)
Release
templates
Manifest
Kubo Release
bosh deploy
Kubo Provides Specification of K8S Components
#CNA2080BU CONFIDENTIAL
17
VMworld 2017 Content: Not fo
r publication or distri
bution
Provides the control plane for provisioning and managing Kubo releases
Joint development effort between Pivotal, VMWare and Google
Kubernetes Dial Tone:
• Health management
• Aggregated Metrics and Logging
• Autoscaling
• Persistence interface
Control Plane:
• Provisioning Engine
• Self-service Clusters
• Software Update Automation
• Load balancing
• Networking
• Multi-tenancy
#CNA2080BU CONFIDENTIAL 18
VMworld 2017 Content: Not fo
r publication or distri
bution
PKS Leverages the Power of BOSH
19
PKS
Release
templates
Manifest
Kubo Release
BOSHVMworld 2017 Content: N
ot for publicatio
n or distribution
Kubernetes Cluster – Day 1Deploy
VMworld 2017 Content: Not fo
r publication or distri
bution
Starting with a BOSH Deployment...
#CNA2080BU CONFIDENTIAL 21
Workeretcdetcd
etcd
MasterMaster
WorkerWorker
BOSH
The definition of each of the nodes in the
cluster, including:
• The bits installed on a node (packages)
• The processes started on a node (jobs)
The definition of each of the nodes in the
cluster, including:
• The bits installed on a node (packages)
• The processes started on a node (jobs)
The definition of each of the nodes in the
cluster, including:
• The bits installed on a node (packages)
• The processes started on a node (jobs)
• Parameterized
BOSH release
A declaration of the desired state of the
cluster:
• Assembly of the components from BOSH
releases (relationships, dependencies)
• Parameter values
BOSH deployment
Kubernetes
VMworld 2017 Content: Not fo
r publication or distri
bution
Message Bus
vSphereBOSH
DB
BOSH Director
Blobs
Health Monitor
Deployment
• Packages
• Blobs
• Source
• Jobs
• Manifest
Deploy my
K8sWorker VMs
etcd
Target VMMaster
Target VMWorker
Target VM
Deploying a Kubernetes Cluster with Cloud Foundry BOSH
#CNA2080BU CONFIDENTIAL 22
VMworld 2017 Content: Not fo
r publication or distri
bution
Kubernetes Cluster – Day 2Operationalize
VMworld 2017 Content: Not fo
r publication or distri
bution
1 Managing Health
2 Scaling
3 Upgrade
Day 2: Operationalize
VMworld 2017 Content: Not fo
r publication or distri
bution
K8s Cluster Health: Processes are Monitored
#CNA2080BU CONFIDENTIAL 25
vSphereBOSH
Master
AGENT
etcd
AGENT
Worker
AGENT
Message Bus
Health Monitor
Responses:
pager
monitoring
…
VMworld 2017 Content: Not fo
r publication or distri
bution
K8s Cluster Health: Processes are Monitored
#CNA2080BU CONFIDENTIAL 26
vSphereBOSH
Master
AGENT
etcd
AGENT
Worker
AGENT
Message Bus
Health Monitor
Responses:
pager
monitoring
…
VMworld 2017 Content: Not fo
r publication or distri
bution
K8s Cluster Health: Processes are Monitored
#CNA2080BU CONFIDENTIAL 27
vSphereBOSH
Master
AGENT
etcd
AGENT
Worker
AGENT
Message Bus
Health Monitor
Responses:
pager
monitoring
…
VMworld 2017 Content: Not fo
r publication or distri
bution
K8s Cluster Health: VMs are Monitored
#CNA2080BU CONFIDENTIAL 28
vSphereBOSH
Master
AGENT
etcd
AGENT
Worker
AGENT
Message Bus
Health Monitor
Responses:
pager
monitoring
ressurector
…
BOSH Director
Desired State Actual State
VMworld 2017 Content: Not fo
r publication or distri
bution
K8s Cluster Health: VMs are Monitored
#CNA2080BU CONFIDENTIAL 29
vSphereBOSH
Master
AGENT
etcd
AGENT
Worker
AGENT
Message Bus
Health Monitor
Responses:
pager
monitoring
ressurector
…
BOSH Director
Desired State Actual State
VMworld 2017 Content: Not fo
r publication or distri
bution
K8s Cluster Health: VMs are Monitored
#CNA2080BU CONFIDENTIAL 30
vSphereBOSH
Master
AGENT
etcd
AGENT
Worker
AGENT
Message Bus
Health Monitor
Responses:
pager
monitoring
ressurector
…
BOSH Director
Desired State Actual State
CPI
VMworld 2017 Content: Not fo
r publication or distri
bution
1 Managing Health
2 Scaling
3 Upgrade
#CNA2080BU CONFIDENTIAL 31
Day 2: Operationalize
VMworld 2017 Content: Not fo
r publication or distri
bution
Primary BOSH Entities
#CNA2080BU CONFIDENTIAL 32
Workeretcdetcd
etcd
MasterMaster
WorkerWorker
BOSH
The definition of each of the nodes in the
cluster, including:
• The bits installed on a node (packages)
• The processes started on a node (jobs)
The definition of each of the nodes in the
cluster, including:
• The bits installed on a node (packages)
• The processes started on a node (jobs)
The definition of each of the nodes in the
cluster, including:
• The bits installed on a node (packages)
• The processes started on a node (jobs)
• Parameterized
BOSH release
A declaration of the desired state of the
cluster:
• Assembly of the components from BOSH
releases (relationships, dependencies)
• Parameter values
BOSH deployment
Relationship to the underlying infrastructure
BOSH cloud config
Kubernetes
VMworld 2017 Content: Not fo
r publication or distri
bution
instance_groups:
- name: etcd
instances: 3
networks:
- name: &network-name ((deployments_network))
azs: [z1]
jobs:
- name: etcd
release: kubo-etcd
properties:
etcd:
require_ssl: false
peer_require_ssl: false
stemcell: trusty
vm_type: common
persistent_disk_type: 5120
- name: master
instances: 2
networks:
- name: *network-name
azs: [z1]
jobs:
- name: cloud-provider
release: kubo
properties: {}
- name: kubernetes-api
release: kubo
properties:
admin-username: admin
admin-password: ((kubo-admin-password))
...
- name: kubeconfig
release: kubo
properties:
...
...
stemcell: trusty
vm_type: master
- name: worker
instances: 3
networks:
- name: *network-name
azs: [z1]
jobs:
- name: docker
release: docker
properties:
...
- name: kubeconfig
release: kubo
properties:
...
- name: kubelet
release: kubo
properties:
...
- name: kubernetes-proxy
release: kubo
properties:
...
stemcell: trusty
vm_type: worker
persistent_disk_type: 10240
Manifest
33
VMworld 2017 Content: Not fo
r publication or distri
bution
instance_groups:
- name: etcd
instances: 3
networks:
- name: &network-name ((deployments_network))
azs: [z1]
jobs:
- name: etcd
release: kubo-etcd
properties:
etcd:
require_ssl: false
peer_require_ssl: false
stemcell: trusty
vm_type: common
persistent_disk_type: 5120
- name: master
instances: 2
networks:
- name: *network-name
azs: [z1]
jobs:
- name: cloud-provider
release: kubo
properties: {}
- name: kubernetes-api
release: kubo
properties:
admin-username: admin
admin-password: ((kubo-admin-password))
...
- name: kubeconfig
release: kubo
properties:
...
...
stemcell: trusty
vm_type: master
- name: worker
instances: 3
networks:
- name: *network-name
azs: [z1]
jobs:
- name: docker
release: docker
properties:
...
- name: kubeconfig
release: kubo
properties:
...
- name: kubelet
release: kubo
properties:
...
- name: kubernetes-proxy
release: kubo
properties:
...
stemcell: trusty
vm_type: worker
persistent_disk_type: 10240
Scaling is a matter of changing the number of
instances and telling BOSH to
“make it so”
Scaling is a matter of changing the number of
instances and telling BOSH to
“make it so”
Scaling is a matter of changing the number of
instances and telling BOSH to
“make it so”
34
VMworld 2017 Content: Not fo
r publication or distri
bution
1 Managing Health
2 Scaling
3 Upgrade
Day 2: Operationalize
VMworld 2017 Content: Not fo
r publication or distri
bution
update:
canaries: 1
max_in_flight: 1
serial: true
canary_watch_time: 10000-300000
update_watch_time: 10000-300000
K8s Cluster Upgrade: Canary Deployments
#CNA2080BU CONFIDENTIAL 36
Manifest VMworld 2017 Content: Not fo
r publication or distri
bution
K8s Cluster Upgrade: Canary Deployments
#CNA2080BU CONFIDENTIAL 37
V1.0 V1.1
# OF CANARIES: 2
MAX IN FLIGHT: 2
EXAMPLE:
CANARIES
VMworld 2017 Content: Not fo
r publication or distri
bution
K8s Cluster Upgrade: Canary Deployments
#CNA2080BU CONFIDENTIAL 38
# OF CANARIES: 2
MAX IN FLIGHT: 2
EXAMPLE:
V1.1 V1.2Once failed, Canary VMs are kept
for troubleshooting purposes.
VMworld 2017 Content: Not fo
r publication or distri
bution
Operationalizing at Scale
VMworld 2017 Content: Not fo
r publication or distri
bution
Supporting Kubernetes Needs at Scale
40
PKS Service Broker
Release
templates
Manifest
Kubo Release
BOSH
VMworld 2017 Content: Not fo
r publication or distri
bution
41
PKS Service Broker
Release
templates
Manifest
Kubo Release
BOSH
create cluster(with upgrade policy)
Supporting Kubernetes Needs at Scale
manage
Thousands
Ones
https://thenewstack.io/comcast-1500-developers-working-cloud-foundry
VMworld 2017 Content: Not fo
r publication or distri
bution
Let Us Show You…
VMworld 2017 Content: Not fo
r publication or distri
bution
43#CNA2080BU CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
PaaS Control Plane
etcd
API-Server
Scheduler
NCM
Infra
Kubernetes
Adapter
CloudFoundry
Adapter
Libnetwork
Adapter
NSX Container Plugin
Mesos
Adapter
NSX
Manager
API Client
Proj: foo Proj: bar
NSX topology for K8s / CF
• NSX Container Plugin (NCP) for integrating with Kubernetes
• NSX Features for K8s PODs
• IP address per container / POD
• Container Network – Routed (BGP) & NATed mode
• Microsegmentation – via K8s Network Policy or native NSX APIs (mapping
K8s labels to NSX tags)
• Network & Security automation – created as part of app deployment
• Multi-tenant network topologies
NSX-T Integration
VMworld 2017 Content: Not fo
r publication or distri
bution
Structured Data
Metrics Alerts Events
VMware vRealize
Operations
Capacity, Performance and
Configuration Management Events
Launch in Context
Unstructured Data
Logs Messages
VMware vRealize
Log Insight
Log analytics, aggregation,
and search
Virtual Applications
vRealize Ops, vRealize Log Insight For Comprehensive Visibility
VMworld 2017 Content: Not fo
r publication or distri
bution
46
K8S Summary –Nodes, Pods, etc.
K8S Topology -Health
K8S Pods - Health
vRealize Ops – Managing Kubernetes Clusters
VMworld 2017 Content: Not fo
r publication or distri
bution
47
K8S Pod Relationship to Components
K8S Alerts
K8S Alerts
vRealize Ops – Kubernetes Integration Details
VMworld 2017 Content: Not fo
r publication or distri
bution
4
8
UI and API Backend
Advanced Analytics Engine
Metrics Collection and Storage
Iterate & Troubleshoot Issues
Trend & Alert on Anomalies
Visualize Metrics at Scale
Self-Service Metrics Analytics for All
Engineering & Business
Introducing Wavefront By VMware SaaS-Based Metrics Monitoring and Analytics Platform
VMworld 2017 Content: Not fo
r publication or distri
bution
App Containers
Docker Host
Docker Swarm
Container Metric Collector
Docker Host
Docker Host
Docker Cluster
AmazonECS
Real-time insight into Docker containers and orchestration
systems Kubernetes, Pivotal Cloud Foundry, Amazon ECS
Wavefront – Container Monitoring Suite
VMworld 2017 Content: Not fo
r publication or distri
bution
50
Need Harbor screenshot
user management & access control
role-based access control
AD/LDAP integration
security
vulnerability scanning
content trust - image signing
policy based image replication
audit and logs
restful API
lightweight & easy deployment
open-source under Apache 2 license
Registry – Enterprise-grade Private Registry
VMworld 2017 Content: Not fo
r publication or distri
bution
51
Registry – Content Trust, When Enabled Un-signed Images Can’t Be Pulled
VMworld 2017 Content: Not fo
r publication or distri
bution
52
Registry – Image Vulnerability Scanning Details
VMworld 2017 Content: Not fo
r publication or distri
bution
VMware PKS
Analytics Automation
SecurityOperations
MonitoringLogging
Physical Infrastructure
Container Registry
vSphere vSAN
Kubernetes on BOSH (Kubo)
NSX
BOSH
GCP Service Broker
masteretcd workermasteretcd worker
VMworld 2017 Content: Not fo
r publication or distri
bution
VMworld 2017 Content: Not fo
r publication or distri
bution
VMworld 2017 Content: Not fo
r publication or distri
bution