in kubernetes multitenancy - qconsp.com · use multitenancy for improved resource efficiency, cost,...
TRANSCRIPT
![Page 1: IN KUBERNETES MULTITENANCY - qconsp.com · Use multitenancy for improved resource efficiency, cost, and operations Different models of multitenancy Soft Hard Hard multitenancy is](https://reader030.vdocument.in/reader030/viewer/2022041011/5ebbb91fa84e107f5276e813/html5/thumbnails/1.jpg)
MULTITENANCY IN KUBERNETES
![Page 2: IN KUBERNETES MULTITENANCY - qconsp.com · Use multitenancy for improved resource efficiency, cost, and operations Different models of multitenancy Soft Hard Hard multitenancy is](https://reader030.vdocument.in/reader030/viewer/2022041011/5ebbb91fa84e107f5276e813/html5/thumbnails/2.jpg)
WHAT COMPANIES CARE ABOUT
Velocity
2
Cost
![Page 3: IN KUBERNETES MULTITENANCY - qconsp.com · Use multitenancy for improved resource efficiency, cost, and operations Different models of multitenancy Soft Hard Hard multitenancy is](https://reader030.vdocument.in/reader030/viewer/2022041011/5ebbb91fa84e107f5276e813/html5/thumbnails/3.jpg)
Hello!
I AM KATHARINA PROBST
I’m a Senior Engineering Manager at Google.You can find me at www.linkedin.com/in/katharina.probst
3
![Page 4: IN KUBERNETES MULTITENANCY - qconsp.com · Use multitenancy for improved resource efficiency, cost, and operations Different models of multitenancy Soft Hard Hard multitenancy is](https://reader030.vdocument.in/reader030/viewer/2022041011/5ebbb91fa84e107f5276e813/html5/thumbnails/4.jpg)
WHY MULTITENANCY
4
![Page 5: IN KUBERNETES MULTITENANCY - qconsp.com · Use multitenancy for improved resource efficiency, cost, and operations Different models of multitenancy Soft Hard Hard multitenancy is](https://reader030.vdocument.in/reader030/viewer/2022041011/5ebbb91fa84e107f5276e813/html5/thumbnails/5.jpg)
KUBERNETES AT A GLANCE
5
masteruser CLI/API/UI
node
node
node
NODES
![Page 6: IN KUBERNETES MULTITENANCY - qconsp.com · Use multitenancy for improved resource efficiency, cost, and operations Different models of multitenancy Soft Hard Hard multitenancy is](https://reader030.vdocument.in/reader030/viewer/2022041011/5ebbb91fa84e107f5276e813/html5/thumbnails/6.jpg)
ONE USER, ONE CLUSTER
6
masteruser CLI/API/UI
node
node
node
NODES
![Page 7: IN KUBERNETES MULTITENANCY - qconsp.com · Use multitenancy for improved resource efficiency, cost, and operations Different models of multitenancy Soft Hard Hard multitenancy is](https://reader030.vdocument.in/reader030/viewer/2022041011/5ebbb91fa84e107f5276e813/html5/thumbnails/7.jpg)
MULTIPLE USERS, MULTIPLE CLUSTERS
7
masteruser CLI/API/UI
node
node
node
NODES
masteruser CLI/API/UI
node
node
node
NODES
![Page 8: IN KUBERNETES MULTITENANCY - qconsp.com · Use multitenancy for improved resource efficiency, cost, and operations Different models of multitenancy Soft Hard Hard multitenancy is](https://reader030.vdocument.in/reader030/viewer/2022041011/5ebbb91fa84e107f5276e813/html5/thumbnails/8.jpg)
SPRAWL OF MANY CLUSTERS
8
masteruser CLI/API/UI
kubelet
kubelet
kubelet
NODES
masteruser CLI/API/UI
kubelet
kubelet
kubelet
NODES
masteruser CLI/API/UI
kubelet
kubelet
kubelet
NODES
masteruser CLI/API/UI
kubelet
kubelet
kubelet
NODES
masteruser CLI/API/UI
kubelet
kubelet
kubelet
NODES
masteruser CLI/API/UI
kubelet
kubelet
kubelet
NODES
masteruser CLI/API/UI
kubelet
kubelet
kubelet
NODES
masteruser CLI/API/UI
kubelet
kubelet
kubelet
NODES
![Page 9: IN KUBERNETES MULTITENANCY - qconsp.com · Use multitenancy for improved resource efficiency, cost, and operations Different models of multitenancy Soft Hard Hard multitenancy is](https://reader030.vdocument.in/reader030/viewer/2022041011/5ebbb91fa84e107f5276e813/html5/thumbnails/9.jpg)
HOW DOES THIS SCALE FINANCIALLY?
9
masteruser CLI/API/UI
kubelet
kubelet
kubelet
NODES
masteruser CLI/API/UI
kubelet
kubelet
kubelet
NODES
masteruser CLI/API/UI
kubelet
kubelet
kubelet
NODES
masteruser CLI/API/UI
kubelet
kubelet
kubelet
NODES
masteruser CLI/API/UI
kubelet
kubelet
kubelet
NODES
masteruser CLI/API/UI
kubelet
kubelet
kubelet
NODES
masteruser CLI/API/UI
kubelet
kubelet
kubelet
NODES
masteruser CLI/API/UI
kubelet
kubelet
kubelet
NODES
![Page 10: IN KUBERNETES MULTITENANCY - qconsp.com · Use multitenancy for improved resource efficiency, cost, and operations Different models of multitenancy Soft Hard Hard multitenancy is](https://reader030.vdocument.in/reader030/viewer/2022041011/5ebbb91fa84e107f5276e813/html5/thumbnails/10.jpg)
HOW DOES THIS SCALE OPERATIONALLY?
10
masteruser CLI/API/UI
kubelet
kubelet
kubelet
NODES
masteruser CLI/API/UI
kubelet
kubelet
kubelet
NODES
masteruser CLI/API/UI
kubelet
kubelet
kubelet
NODES
masteruser CLI/API/UI
kubelet
kubelet
kubelet
NODES
masteruser CLI/API/UI
kubelet
kubelet
kubelet
NODES
masteruser CLI/API/UI
kubelet
kubelet
kubelet
NODES
masteruser CLI/API/UI
kubelet
kubelet
kubelet
NODES
masteruser CLI/API/UI
kubelet
kubelet
kubelet
NODES
![Page 11: IN KUBERNETES MULTITENANCY - qconsp.com · Use multitenancy for improved resource efficiency, cost, and operations Different models of multitenancy Soft Hard Hard multitenancy is](https://reader030.vdocument.in/reader030/viewer/2022041011/5ebbb91fa84e107f5276e813/html5/thumbnails/11.jpg)
...Namespace 2
Namespace 1
...
MANY USERS, ONE CLUSTER
11
master
User 2
CLI/API/UIUser 3
User 1
User n
Namespace n
![Page 12: IN KUBERNETES MULTITENANCY - qconsp.com · Use multitenancy for improved resource efficiency, cost, and operations Different models of multitenancy Soft Hard Hard multitenancy is](https://reader030.vdocument.in/reader030/viewer/2022041011/5ebbb91fa84e107f5276e813/html5/thumbnails/12.jpg)
MULTITENANCY USERS
12
CLUSTER ADMIN
◦ Set up cluster and namespaces. Set up resource limits.
◦ Ensure consistency across namespaces in the cluster.
◦ Operate the clusters (e.g., respond to incidents).
CLUSTER USER
◦ Start/stop/manage their own app(s) in their own namespace(s).
◦ Understand their namespace(s)’ resource limits.
◦ Don’t trample on other tenants.
NAMESPACE ADMIN
Admin rights to specific namespace(s)
![Page 13: IN KUBERNETES MULTITENANCY - qconsp.com · Use multitenancy for improved resource efficiency, cost, and operations Different models of multitenancy Soft Hard Hard multitenancy is](https://reader030.vdocument.in/reader030/viewer/2022041011/5ebbb91fa84e107f5276e813/html5/thumbnails/13.jpg)
MODELS OF MULTITENANCY
13
HARD MULTITENANCY
◦ Zero-trust tenants
◦ Not yet widely used in production
◦ Ongoing work in kubernetes community to strengthen
![Page 14: IN KUBERNETES MULTITENANCY - qconsp.com · Use multitenancy for improved resource efficiency, cost, and operations Different models of multitenancy Soft Hard Hard multitenancy is](https://reader030.vdocument.in/reader030/viewer/2022041011/5ebbb91fa84e107f5276e813/html5/thumbnails/14.jpg)
14
WHAT WILL HARD MULTITENANCY TAKE?
![Page 15: IN KUBERNETES MULTITENANCY - qconsp.com · Use multitenancy for improved resource efficiency, cost, and operations Different models of multitenancy Soft Hard Hard multitenancy is](https://reader030.vdocument.in/reader030/viewer/2022041011/5ebbb91fa84e107f5276e813/html5/thumbnails/15.jpg)
15
WHAT WILL HARD MULTITENANCY TAKE?
◦ Tenants can’t
▫ DoS/impact access to others’ resources
▫ See each others’ stuff (e.g., by intercepting network traffic or
accessing stored data)
![Page 16: IN KUBERNETES MULTITENANCY - qconsp.com · Use multitenancy for improved resource efficiency, cost, and operations Different models of multitenancy Soft Hard Hard multitenancy is](https://reader030.vdocument.in/reader030/viewer/2022041011/5ebbb91fa84e107f5276e813/html5/thumbnails/16.jpg)
16
WHAT WILL HARD MULTITENANCY TAKE?
◦ Tenants can’t
▫ DoS/impact access to others’ resources
▫ See each others’ stuff (e.g., by intercepting network traffic or
accessing stored data)
◦ Resource objects don’t collide, e.g., custom controllers/CRDs
![Page 17: IN KUBERNETES MULTITENANCY - qconsp.com · Use multitenancy for improved resource efficiency, cost, and operations Different models of multitenancy Soft Hard Hard multitenancy is](https://reader030.vdocument.in/reader030/viewer/2022041011/5ebbb91fa84e107f5276e813/html5/thumbnails/17.jpg)
17
WHAT WILL HARD MULTITENANCY TAKE?
◦ Tenants can’t
▫ DoS/impact access to others’ resources
▫ See each others’ stuff (e.g., by intercepting network traffic or
accessing stored data)
◦ Resource objects don’t collide, e.g., custom controllers/CRDs
◦ Control plane (master) resources are shared fairly
![Page 18: IN KUBERNETES MULTITENANCY - qconsp.com · Use multitenancy for improved resource efficiency, cost, and operations Different models of multitenancy Soft Hard Hard multitenancy is](https://reader030.vdocument.in/reader030/viewer/2022041011/5ebbb91fa84e107f5276e813/html5/thumbnails/18.jpg)
MODELS OF MULTITENANCY
18
SOFT MULTITENANCY
◦ Tenants are more trusted◦ Often used within one enterprise◦ Different teams → different namespaces
◦ Used in practice and in production◦ Often used in combination with infrastructure/platform built on
top by centralized team for consistent security, networking, etc.
![Page 19: IN KUBERNETES MULTITENANCY - qconsp.com · Use multitenancy for improved resource efficiency, cost, and operations Different models of multitenancy Soft Hard Hard multitenancy is](https://reader030.vdocument.in/reader030/viewer/2022041011/5ebbb91fa84e107f5276e813/html5/thumbnails/19.jpg)
MULTITENANCY PRIMITIVES
19
ACCESS CONTROL
Use policies to ensure that tenants can access only what they should have access to
FAIR SHARING
Enforce limits per tenant[Better developed on data plane; control plane WIP]
ISOLATION
Ensure tenants cannot access each others’ workloads, secrets, etc. (security isolation)
![Page 20: IN KUBERNETES MULTITENANCY - qconsp.com · Use multitenancy for improved resource efficiency, cost, and operations Different models of multitenancy Soft Hard Hard multitenancy is](https://reader030.vdocument.in/reader030/viewer/2022041011/5ebbb91fa84e107f5276e813/html5/thumbnails/20.jpg)
ACCESS CONTROL - RBAC
20
ClusterRole A pre-set of capabilities, cluster-wide
Role Like ClusterRole, but namespace-scoped
ClusterRoleBinding Give permissions defined in a ClusterRole
RoleBinding Like ClusterRoleBinding, but namespace-scoped
![Page 21: IN KUBERNETES MULTITENANCY - qconsp.com · Use multitenancy for improved resource efficiency, cost, and operations Different models of multitenancy Soft Hard Hard multitenancy is](https://reader030.vdocument.in/reader030/viewer/2022041011/5ebbb91fa84e107f5276e813/html5/thumbnails/21.jpg)
◦ Use RBAC for controlling access to Secrets etc.
◦ Pod Security Policy (e.g., access to volume types, privileged): enable
fine-grained authorization of pod creation and update
◦ Network Policy (ingress, egress): control which pods can talk to each other
◦ Make Custom Resource Definitions namespace-scoped
◦ Sandboxes, e.g., gVisor
▫ Ensure security isolation of pods
▫ Ensure that information is not leaked between untrusted tenants
21
ISOLATION
![Page 22: IN KUBERNETES MULTITENANCY - qconsp.com · Use multitenancy for improved resource efficiency, cost, and operations Different models of multitenancy Soft Hard Hard multitenancy is](https://reader030.vdocument.in/reader030/viewer/2022041011/5ebbb91fa84e107f5276e813/html5/thumbnails/22.jpg)
◦ Resource Quotas (with Limit Range defaults) for CPU, memory, object
counts
◦ Pod Priority, Quality of Service Classes
◦ Node
▫ Taints & Tolerations: allow a node to repel a set of pods
▫ Labels and node selectors (less flexible than Affinity)
◦ Pod
▫ Affinity
▫ Pod Anti-affinity
22
FAIR SHARING - DATA PLANE (NODES)
![Page 23: IN KUBERNETES MULTITENANCY - qconsp.com · Use multitenancy for improved resource efficiency, cost, and operations Different models of multitenancy Soft Hard Hard multitenancy is](https://reader030.vdocument.in/reader030/viewer/2022041011/5ebbb91fa84e107f5276e813/html5/thumbnails/23.jpg)
23
DATA PLANE VS. CONTROL PLANE MULTITENANCY
![Page 24: IN KUBERNETES MULTITENANCY - qconsp.com · Use multitenancy for improved resource efficiency, cost, and operations Different models of multitenancy Soft Hard Hard multitenancy is](https://reader030.vdocument.in/reader030/viewer/2022041011/5ebbb91fa84e107f5276e813/html5/thumbnails/24.jpg)
Much of what we’ve talked about is sharing the data plane (nodes)
24
CONTROL PLANE MULTITENANCY
![Page 25: IN KUBERNETES MULTITENANCY - qconsp.com · Use multitenancy for improved resource efficiency, cost, and operations Different models of multitenancy Soft Hard Hard multitenancy is](https://reader030.vdocument.in/reader030/viewer/2022041011/5ebbb91fa84e107f5276e813/html5/thumbnails/25.jpg)
...Namespace 2
Namespace 1
...
MANY USERS, ONE CLUSTER
25
master
User 2
CLI/API/UIUser 3
User 1
User n
Namespace n
![Page 26: IN KUBERNETES MULTITENANCY - qconsp.com · Use multitenancy for improved resource efficiency, cost, and operations Different models of multitenancy Soft Hard Hard multitenancy is](https://reader030.vdocument.in/reader030/viewer/2022041011/5ebbb91fa84e107f5276e813/html5/thumbnails/26.jpg)
Not able to manage multiple clusters →
1:1 mapping between master and cluster, though cluster may have multiple
tenants
26
MULTITENANT API SERVER?
![Page 27: IN KUBERNETES MULTITENANCY - qconsp.com · Use multitenancy for improved resource efficiency, cost, and operations Different models of multitenancy Soft Hard Hard multitenancy is](https://reader030.vdocument.in/reader030/viewer/2022041011/5ebbb91fa84e107f5276e813/html5/thumbnails/27.jpg)
...Namespace 2
Namespace 1
...
MANY USERS, ONE CLUSTER
27
master
User 2
CLI/API/UIUser 3
User 1
User n
Namespace n
![Page 28: IN KUBERNETES MULTITENANCY - qconsp.com · Use multitenancy for improved resource efficiency, cost, and operations Different models of multitenancy Soft Hard Hard multitenancy is](https://reader030.vdocument.in/reader030/viewer/2022041011/5ebbb91fa84e107f5276e813/html5/thumbnails/28.jpg)
All tenants share master (incl. Secrets, ConfigMap), but RBAC helps
Little protection against individual tenants DoSing each other
28
MULTITENANT API SERVER?
![Page 29: IN KUBERNETES MULTITENANCY - qconsp.com · Use multitenancy for improved resource efficiency, cost, and operations Different models of multitenancy Soft Hard Hard multitenancy is](https://reader030.vdocument.in/reader030/viewer/2022041011/5ebbb91fa84e107f5276e813/html5/thumbnails/29.jpg)
Max inflight requests: mechanism for protecting API server against CPU and
memory overloads
Current problem to address: Tenants can crowd each other out (accidentally or
on purpose)
29
BETTER API SERVER FAIRNESS (UNDERWAY)
![Page 30: IN KUBERNETES MULTITENANCY - qconsp.com · Use multitenancy for improved resource efficiency, cost, and operations Different models of multitenancy Soft Hard Hard multitenancy is](https://reader030.vdocument.in/reader030/viewer/2022041011/5ebbb91fa84e107f5276e813/html5/thumbnails/30.jpg)
Proposal generalizes … max-in-flight request handler in the apiserver to make
more distinctions among requests and provide prioritization and fairness among
the categories of requests.”
30
BETTER API SERVER FAIRNESS (UNDERWAY)
![Page 31: IN KUBERNETES MULTITENANCY - qconsp.com · Use multitenancy for improved resource efficiency, cost, and operations Different models of multitenancy Soft Hard Hard multitenancy is](https://reader030.vdocument.in/reader030/viewer/2022041011/5ebbb91fa84e107f5276e813/html5/thumbnails/31.jpg)
BETTER API SERVER FAIRNESS (UNDERWAY)
Multiple priority levels, each has queues
Within each priority level, queues compete evenly
31
![Page 32: IN KUBERNETES MULTITENANCY - qconsp.com · Use multitenancy for improved resource efficiency, cost, and operations Different models of multitenancy Soft Hard Hard multitenancy is](https://reader030.vdocument.in/reader030/viewer/2022041011/5ebbb91fa84e107f5276e813/html5/thumbnails/32.jpg)
WHAT COMPANIES CARE ABOUT
Velocity
32
Cost
![Page 33: IN KUBERNETES MULTITENANCY - qconsp.com · Use multitenancy for improved resource efficiency, cost, and operations Different models of multitenancy Soft Hard Hard multitenancy is](https://reader030.vdocument.in/reader030/viewer/2022041011/5ebbb91fa84e107f5276e813/html5/thumbnails/33.jpg)
◦ Use multitenancy for improved resource efficiency, cost, and operations
◦ Different models of multitenancy
▫ Soft
▫ Hard
◦ Hard multitenancy is still work in progress
◦ Soft multitenancy is already used in production by various companies
▫ But is often coupled with a shared internal platform to gain
consistency of networking, security, etc. across teams
33
KEY TAKE-AWAYS
![Page 34: IN KUBERNETES MULTITENANCY - qconsp.com · Use multitenancy for improved resource efficiency, cost, and operations Different models of multitenancy Soft Hard Hard multitenancy is](https://reader030.vdocument.in/reader030/viewer/2022041011/5ebbb91fa84e107f5276e813/html5/thumbnails/34.jpg)
QUESTIONS?
![Page 35: IN KUBERNETES MULTITENANCY - qconsp.com · Use multitenancy for improved resource efficiency, cost, and operations Different models of multitenancy Soft Hard Hard multitenancy is](https://reader030.vdocument.in/reader030/viewer/2022041011/5ebbb91fa84e107f5276e813/html5/thumbnails/35.jpg)
◦ Project plan for multitenancy
▫ Building CRD for more automatic management of namespaces
◦ API Machinery KEP for improved resource sharing
◦ Cluster multi-tenancy for a good overview
35
Links for additional details