building the future of distributed systems with kubernetes · about me go • using since 2011 •...
Post on 20-May-2020
1 Views
Preview:
TRANSCRIPT
Building the future of Distributed Systems with KubernetesErik St. Martin Sr. Cloud Developer Advocate @erikstmartin
About MeGo • Using since 2011• Co-Organizer GopherCon• Co-Author Go in Action• Co-Host Go Time
Kubernetes • Using since 2014• Contributed to Docker and Kubernetes• Created SkyDNS (predecessor and
library for kube-dns)• Contributed heavily to Virtual Kubelet
@erikstmartin #GOTOchgo
What is Kubernetes?• Container Orchestration Platform?
@erikstmartin #GOTOchgo
What is Kubernetes?• Container Orchestration Platform? • Infrastructure as a Service Platform?
@erikstmartin #GOTOchgo
What is Kubernetes?• Container Orchestration Platform? • Infrastructure as a Service Platform? • Framework / Library for building Distributed Systems?
@erikstmartin #GOTOchgo
“Computers aren’t the thing… They’re the thing that gets us to the thing.”
- Joe MacMillan (Halt and Catch Fire S1:E1)
@erikstmartin #GOTOchgo
“Kubernetes isn’t the thing… It’s the thing that gets us to the thing.”
Customizing Kubernetes
@erikstmartin #GOTOchgo
How does Kubernetes work?
@erikstmartin #GOTOchgo
Reconciliation
Desired State Actual State
@erikstmartin #GOTOchgo
Kubernetes Objects
• Node • Pod • Service • Ingress • Namespace
• ConfigMap • Secret • Volume • PersistentVolume • PersistentVolumeClaim
• ReplicaSet • StatefulSet • DaemonSet • Job • Deployment
@erikstmartin #GOTOchgo
Spec FilesapiVersion: apps/v1
kind: Deployment metadata: name: nginx-deployment labels: app: nginx spec: replicas: 2 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.7.9 ports: - containerPort: 80
@erikstmartin #GOTOchgo
ReconciliationNode 1
nginx
Node 2
Node 3
nginx
apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment labels: app: nginx spec: replicas: 2 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.7.9
@erikstmartin #GOTOchgo
Kubernetes Control PlaneMaster 1
etcd
Worker 1
kubelet
kube-proxy
Container Runtime
Pod PodPod
Worker 2
kubelet
kube-proxy
Container Runtime
Pod PodPod
Worker 3
kubelet
kube-proxy
Container Runtime
Pod PodPod
Scheduler
Controller Manager
Master 3
etcd
API Server
Master 2
etcd
API Server
Cloud Controller Manager
@erikstmartin #GOTOchgo
Etcd
• Distributed • Fault Tolerant • Versioning • Watches • Distributed Locks • Leadership Election
@erikstmartin #GOTOchgo
API Server
API Server
Kube Proxy
Kubelet
Kubectl
Scheduler
Controller Manager
@erikstmartin #GOTOchgo
API ServerWhy would you want to create your own? • API Server Aggregation (Service Catalog, etc) • Custom Resource Definitions
@erikstmartin #GOTOchgo
Open Service Broker• Connect applications to hosted services and
partner services using Kubernetes API • Provisioning, Deprovisioning, Binding,
Unbinding • Currently supports (ACI, Cosmos DB, MySQL,
Postgres, Event Hubs, Key Vault, Redis, SQL Database, Search, Service Bus, Storage)
@erikstmartin #GOTOchgo
Service Catalog & Service Broker
API Server
Service Catalog
Service Broker A
• List Services • Provision Instance • Bind Instance
Managed Service 1
Managed Service 2
Service Broker B
Managed Service 3
Managed Service 4
Application
Bind Instance
Secret
• Service Details • Connection Credentials
@erikstmartin #GOTOchgo
Open Service Broker (mysql instance)apiVersion:servicecatalog.k8s.io/v1beta1kind:ServiceInstancemetadata:name:example-mysql-instancenamespace:defaultspec:clusterServiceClassExternalName:azure-mysqlclusterServicePlanExternalName:basic50parameters:location:eastusresourceGroup:demofirewallRules:-startIPAddress:"0.0.0.0"endIPAddress:"255.255.255.255"name:"AllowAll"
@erikstmartin #GOTOchgo
Open Service Broker (mysql binding)apiVersion:servicecatalog.k8s.io/v1beta1kind:ServiceBindingmetadata:name:example-mysql-bindingnamespace:defaultspec:instanceRef:name:example-mysql-instancesecretName:example-mysql-secret
@erikstmartin #GOTOchgo
Controller Manager
Controller Manager
apiVersion: apps/v1 kind: ReplicaSet metadata: name: nginx-deployment spec: replicas: 2 template: spec: …
apiVersion: apps/v1 kind: Pod metadata: name: nginx spec: …
apiVersion: apps/v1 kind: Pod metadata: name: nginx spec: …
API Server
@erikstmartin #GOTOchgo
Controller ManagerNode Controller: Responsible for noticing and responding when nodes go down. Replication Controller: Responsible for maintaining the correct number of pods for every replication controller object in the system. Endpoints Controller: Populates the Endpoints object (that is, joins Services & Pods). Service Account & Token Controllers: Create default accounts and API access tokens for new namespaces.
@erikstmartin #GOTOchgo
Cloud Controller ManagerNode Controller: For checking the cloud provider to determine if a node has been deleted in the cloud after it stops responding Route Controller: For setting up routes in the underlying cloud infrastructure Service Controller: For creating, updating and deleting cloud provider load balancers Volume Controller: For creating, attaching, and mounting volumes, and interacting with the cloud provider to orchestrate volumes
@erikstmartin #GOTOchgo
Controllers / Operator PatternWhy would you want to create your own? • Implement support for a different cloud provider • Decouple applications and resource management from spec definitions • Provide additional self-service infrastructure to your organization (databases, • services, monitoring, etc), and can leverage things like RBAC out of the box • Abstracting operational knowledge (Operator Pattern)
https://github.com/kubernetes/sample-controller
@erikstmartin #GOTOchgo
Kube-legoAutomatically request LetsEncrypt certificates for your services
Service Spec
metadata:annotations:kubernetes.io/tls-acme:“true"
@erikstmartin #GOTOchgo
Kube-legoIngress Spec
spec:tls:-secretName:mysql-tlshosts:-phpmyadmin.example.com-mysql.example.com-secretName:postgres-tlshosts:-postgres.example.com
@erikstmartin #GOTOchgo
Prometheus Operator
Prometheus Operator
apiVersion: monitoring.coreos.com/v1 kind: Prometheus metadata: name: prometheus spec: replicas: 2 serviceAccountName: prometheus alerting: alertmanagers: …
apiVersion: apps/v1 kind: StatefulSet metadata: name: prometheus spec: …
API Server
@erikstmartin #GOTOchgo
Prometheus Operator
Prometheus Operator
apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: nginx spec: selector: matchLabels: prometheus app: nginx endpoints: - port: http
apiVersion: apps/v1 kind: ConfigMap metadata: name: prometheus …
API Server
Prometheus
@erikstmartin #GOTOchgo
CRDs / Custom Controllers
@erikstmartin #GOTOchgo
CRDs / Custom Controllers
@erikstmartin #GOTOchgo
CRDs / Custom Controllers
@erikstmartin #GOTOchgo
CRDs vs API AggregationCRD • Can use any programming language • No need to handle multiple API versions • Can do minimal validation (beta 1.9) • Supports Scale and Status sub-resources
API Aggregation • Must use Go • Must handle multiple API versions • Any validation you want • Implement any sub-resources you’d like,
including things like exec, attach, etc
@erikstmartin #GOTOchgo
Scheduler
API Server
Schedulerkind: Pod
metadata: name: nginx spec: nodeName: <empty> containers: - name: nginx image: nginx:1.7.9 ports: - containerPort: 80
kind: Pod metadata: name: nginx spec: nodeName: Worker 1 containers: - name: nginx image: nginx:1.7.9 ports: - containerPort: 80
@erikstmartin #GOTOchgo
SchedulerWhy would you want to create your own? • You need outside information for scheduling decisions • Network based scheduling • Running workloads close to the data it needs • Hardware Encryption modules that are pre-primed with data, etc
@erikstmartin #GOTOchgo
Kubelet • Calls the configured container runtime • Pulls images from the image registry associated with pods assigned to this node. • Creates and mounts volumes within a container • Injects environment variables and updates volumes with Secrets, ConfigMaps, and the
Downward API • Updates the API Server with the latest Pod statuses • Provides an API for the API Server to call for things like ◦ kubectl exec ◦ kubectl attach ◦ kubectl logs ◦ metrics used by the scheduler and dashboard • Configures Pod networking (CNI)
@erikstmartin #GOTOchgo
KubeletAPI Server
Worker 1
kind: Pod metadata: name: nginx spec: nodeName: Worker 1 containers: - name: nginx image: nginx:1.7.9 ports: - containerPort: 80
Kubelet
Nginx
Container Runtime
CNI
@erikstmartin #GOTOchgo
KubeletWhy would you want to create your own? • You probably don’t • but… we did anyway (Virtual Kubelet)
@erikstmartin #GOTOchgo
Virtual Kubelet
API Server
kind: Pod metadata: name: nginx spec: nodeName: vkubelet containers: - name: nginx image: nginx:1.7.9 ports: - containerPort: 80
Virtual Kubelet (vkubelet)
\Azure Container Instances
Nginx
kind: Node metadata: name: vkubelet spec: …
@erikstmartin #GOTOchgo
Virtual Kubelet - Why?• Batch Jobs • CI/CD • Serverless • Bursting
@erikstmartin #GOTOchgo
CNI
CNI
Kubelet (Worker 1)API Server
kind: Pod metadata: name: nginx spec: nodeName: Worker 1 containers: - name: nginx image: nginx:1.7.9 ports: - containerPort: 80 Plugin
(bridge, ipvlan, macvlan, ptp, vlan, custom)
IPAM
{ "cniVersion":"0.2.0", "name":"mynet", "type":"bridge", "bridge":"cni0", "isGateway":true, "ipMasq":true, "ipam":{ "type":"host-local", "subnet":"10.22.0.0/16", "routes":[ {"dst":"0.0.0.0/0"} ] }}
/etc/cni/net.d/10-mynet.conf
@erikstmartin #GOTOchgo
CNIWhy would you want to create your own? • Multiple interfaces (CNI-Genie, Multus) • Custom Routing (BGP, NetConf, host/container routes, etc) • SDN (OpenFlow, etc) • Firewall configuration
@erikstmartin #GOTOchgo
Kube ProxyAPI Server
Worker 1
kind: Service metadata: name: backend spec: clusterIP: 10.10.1.5 ports: - name: http port: 80 protocol: TCP
Kube Proxy
Frontend
IPTables
Backend
Backend
Backend
@erikstmartin #GOTOchgo
But wait, there’s more!• Extended Resources (Custom resource limits) • initContainers (run a container before the rest of the pod starts, can have access to things the
app containers don’t, like secrets) • Initializers and Finalizers (force constraints) • readiness and liveness probes • node affinity / anti-affinity • pod affinity / anti-affinity
@erikstmartin #GOTOchgo
Istio• Can modify headers and inject distributed tracing, etc • Policy Enforcement • Intelligent Request Routing A/B tests, etc • Timeouts • Bounded retries with timeout budgets and variable jitter between retries • Limit number of concurrent connections and requests to upstream services • Active (periodic) health checks on each member of the load balancing pool • Fine-grained circuit breakers (passive health checks) – applied per instance in the load
balancing pool
Abstracting Kubernetes
@erikstmartin #GOTOchgo
Helm• Package manager for Kubernetes • Abstracts multiple resources into an
“application” • Supports upgrades & rollbacks of the entire
application • Repository of pre-created charts (https://
hub.kubeapps.com/)
@erikstmartin #GOTOchgo
OpenFaaS• Functions as a Service - backed by Kubernetes
@erikstmartin #GOTOchgo
Draft• Draft makes it easy to build applications that
run on Kubernetes. • Draft packs consist of a Dockerfile and a Helm
Chart that demonstrates best practices for deploying applications of a given language
@erikstmartin #GOTOchgo
Pachyderm• Open Source data pipelining and data
management layer for Kubernetes • Orchestrates workloads • Manages input and output data to ensure it is
available to the write code at the write time • Ensures that jobs that require or could benefit
from GPU acceleration end up on nodes with GPU resources.
• Data Versioning and Provenance
@erikstmartin #GOTOchgo
Metaparticle• Metaparticle is a standard library for cloud
native applications on Kubernetes. • Democratizes the development of distributed
systems. • Collection of libraries that enable programmers
to build and deploy containers using code that feels familiar to them.
• Aims to use language level features to add new capabilities to existing programming languages.
@erikstmartin #GOTOchgo
Metaparticlemetaparticle.Containerize(
&metaparticle.Runtime{ Ports:[]int32{port}, Executor:"metaparticle", Replicas:3, }, &metaparticle.Package{ Name:“web-demo", Repository:"docker.io/myuser", Builder:"docker", Publish:true, }, func(){ http.HandleFunc("/",handler) err:=http.ListenAndServe(fmt.Sprintf(":%d",port),nil) iferr!=nil{ log.Fatal("Couldn'tstarttheserver:",err) } })
@erikstmartin #GOTOchgo
Metaparticlemetaparticle.Containerize(
&metaparticle.Runtime{ Ports:[]int32{port}, Executor:"metaparticle", Shards:3, URLShardPattern:"^\\/users\\/([^\\/]*)\\/.*", }, &metaparticle.Package{ ` Name:“shard-demo", Repository:"docker.io/myuser", Builder:"docker", }, func(){ http.HandleFunc("/",handler) err:=http.ListenAndServe(fmt.Sprintf(":%d",port),nil) iferr!=nil{ log.Fatal("Couldn'tstarttheserver:",err) } })
@erikstmartin #GOTOchgo
The Future
Thank You!
Twitter: @erikstmartin Blog: erikstmartin.com GitHub: github.com/erikstmartin
@erikstmartin #GOTOchgo
Questions?No question is a bad question.
top related