replacing iptables with ebpf in kubernetes with cilium · 2020-01-30 · tc bpf xdp cilium agent...
TRANSCRIPT
![Page 1: Replacing iptables with eBPF in Kubernetes with Cilium · 2020-01-30 · TC BPF XDP CILIUM AGENT DAEMON CILIUM CLI CILIUM MONITOR CILIUM HEALTH CILIUM HEALTH NAMESPACE PLUGIN Build](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec557b313b08355f20a9fb2/html5/thumbnails/1.jpg)
Replacing iptables with eBPF in Kubernetes with CiliumCilium, eBPF, Envoy, Istio, Hubble
Michal RosteckiSoftware [email protected]@opensuse.org
Swaminathan VasudevanSoftware [email protected]
![Page 2: Replacing iptables with eBPF in Kubernetes with Cilium · 2020-01-30 · TC BPF XDP CILIUM AGENT DAEMON CILIUM CLI CILIUM MONITOR CILIUM HEALTH CILIUM HEALTH NAMESPACE PLUGIN Build](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec557b313b08355f20a9fb2/html5/thumbnails/2.jpg)
22
What’s wrong with iptables?
![Page 3: Replacing iptables with eBPF in Kubernetes with Cilium · 2020-01-30 · TC BPF XDP CILIUM AGENT DAEMON CILIUM CLI CILIUM MONITOR CILIUM HEALTH CILIUM HEALTH NAMESPACE PLUGIN Build](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec557b313b08355f20a9fb2/html5/thumbnails/3.jpg)
3
IPtables runs into a couple of significant problems:
● Iptables updates must be made by recreating and updating all rules in a single transaction.● Implements chains of rules as a linked list, so all operations are O(n).● The standard practice of implementing access control lists (ACLs) as implemented by iptables was to use
sequential list of rules.● It’s based on matching IPs and ports, not aware about L7 protocols.● Every time you have a new IP or port to match, rules need to be added and the chain changed.● Has high consumption of resources on Kubernetes.
Based on the above mentioned issues under heavy traffic conditions or in a system that has a large number of changes to iptable rules the performance degrades.
Measurements show unpredictable latency and reduced performance as the number of services grows.
What’s wrong with legacy iptables?
![Page 4: Replacing iptables with eBPF in Kubernetes with Cilium · 2020-01-30 · TC BPF XDP CILIUM AGENT DAEMON CILIUM CLI CILIUM MONITOR CILIUM HEALTH CILIUM HEALTH NAMESPACE PLUGIN Build](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec557b313b08355f20a9fb2/html5/thumbnails/4.jpg)
4
Kubernetes uses iptables for...
● kube-proxy - the component which implements Services and load balancing by DNAT iptables rules
● the most of CNI plugins are using iptables for Network Policies
![Page 5: Replacing iptables with eBPF in Kubernetes with Cilium · 2020-01-30 · TC BPF XDP CILIUM AGENT DAEMON CILIUM CLI CILIUM MONITOR CILIUM HEALTH CILIUM HEALTH NAMESPACE PLUGIN Build](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec557b313b08355f20a9fb2/html5/thumbnails/5.jpg)
5
And it ends up like that
![Page 6: Replacing iptables with eBPF in Kubernetes with Cilium · 2020-01-30 · TC BPF XDP CILIUM AGENT DAEMON CILIUM CLI CILIUM MONITOR CILIUM HEALTH CILIUM HEALTH NAMESPACE PLUGIN Build](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec557b313b08355f20a9fb2/html5/thumbnails/6.jpg)
6
![Page 7: Replacing iptables with eBPF in Kubernetes with Cilium · 2020-01-30 · TC BPF XDP CILIUM AGENT DAEMON CILIUM CLI CILIUM MONITOR CILIUM HEALTH CILIUM HEALTH NAMESPACE PLUGIN Build](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec557b313b08355f20a9fb2/html5/thumbnails/7.jpg)
7
What is BPF?
![Page 8: Replacing iptables with eBPF in Kubernetes with Cilium · 2020-01-30 · TC BPF XDP CILIUM AGENT DAEMON CILIUM CLI CILIUM MONITOR CILIUM HEALTH CILIUM HEALTH NAMESPACE PLUGIN Build](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec557b313b08355f20a9fb2/html5/thumbnails/8.jpg)
8
HW Bridge OVS .
Netdevice / Drivers
Traffic Shaping
Ethernet
IPv4 IPv6
Netfilter
TCP UDP Raw
Sockets
System Call Interface
Process Process Process ● The Linux kernel stack is split into multiple abstraction layers.
● Strong userspace API compatibility in Linux for years.
● This shows how complex the linux kernel is and its years of evolution.
● This cannot be replaced in a short term.
● Very hard to bypass the layers.
● Netfilter module has been supported by linux for more than two decades and packet filtering has to applied to packets that moves up and down the stack.
Linux Network Stack
![Page 9: Replacing iptables with eBPF in Kubernetes with Cilium · 2020-01-30 · TC BPF XDP CILIUM AGENT DAEMON CILIUM CLI CILIUM MONITOR CILIUM HEALTH CILIUM HEALTH NAMESPACE PLUGIN Build](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec557b313b08355f20a9fb2/html5/thumbnails/9.jpg)
9
HW Bridge OVS .
Netdevice / Drivers
Traffic Shaping
Ethernet
IPv4 IPv6
Netfilter
TCP UDP Raw
Sockets
System Call Interface
Process Process Process
BPF System calls
BPF Sockmap and Sockops
BPF TC hooks
BPF XDP
BPF kernel hooks
BPF cGroups
![Page 10: Replacing iptables with eBPF in Kubernetes with Cilium · 2020-01-30 · TC BPF XDP CILIUM AGENT DAEMON CILIUM CLI CILIUM MONITOR CILIUM HEALTH CILIUM HEALTH NAMESPACE PLUGIN Build](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec557b313b08355f20a9fb2/html5/thumbnails/10.jpg)
10
Mpp
s
![Page 11: Replacing iptables with eBPF in Kubernetes with Cilium · 2020-01-30 · TC BPF XDP CILIUM AGENT DAEMON CILIUM CLI CILIUM MONITOR CILIUM HEALTH CILIUM HEALTH NAMESPACE PLUGIN Build](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec557b313b08355f20a9fb2/html5/thumbnails/11.jpg)
11
PREROUTING INPUT OUTPUTFORWARD POSTROUTING
FILTER
FILTER FILTER
NATNAT
RoutingDecision
NAT
RoutingDecision
RoutingDecision
Netdev(Physical or
virtual Device)
Netdev(Physical or
virtual Device)
Local Processes
eBPF Code
eBPF Code
IPTables netfilter hooks
eBPF TC hooks
XDP hooks
BPF replaces IPtables
![Page 12: Replacing iptables with eBPF in Kubernetes with Cilium · 2020-01-30 · TC BPF XDP CILIUM AGENT DAEMON CILIUM CLI CILIUM MONITOR CILIUM HEALTH CILIUM HEALTH NAMESPACE PLUGIN Build](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec557b313b08355f20a9fb2/html5/thumbnails/12.jpg)
12
NetFilter NetFilter
To Linux Stack
From Linux Stack
Netdev(Physical or
virtual Device)
Netdev(Physical or
virtual Device)
Ingress Chain
Selector
INGRESS CHAIN
FORWARD CHAIN
[local dst]
[remote dst]
TC/XDP Ingress hook
TC Egress hook
Egress Chain Selector
OUTPUT CHAIN
[local src]
[remote src]
Update session
Label Packet
Update session
Label Packet
Store session
Store session
Store session
Update session
Label Packet
Connection Tracking
BPF based filtering architecture
![Page 13: Replacing iptables with eBPF in Kubernetes with Cilium · 2020-01-30 · TC BPF XDP CILIUM AGENT DAEMON CILIUM CLI CILIUM MONITOR CILIUM HEALTH CILIUM HEALTH NAMESPACE PLUGIN Build](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec557b313b08355f20a9fb2/html5/thumbnails/13.jpg)
13
….
Headersparsing
IP.dstlookup
IP1 bitv1IP2 bitv2IP3 bitv3
eBPF Program #1 eBPF Program #2 eBPF Program #3
IP.protolookup
* bitv1udp bitv2tcp bitv3
BitwiseAND
bit-vectors
Search first
Matching rule
Updatecounters
ACTION(drop/
accept)
rule1 act1rule2 act2rule3 act3
rule1 cnt1rule2 cnt2
eBPF Program
eBPF Program #N
Packet inPacket out
From eBPF hookTo eBPF hook
Tail
call
Tail
call
Tail
call
Tail
call
Packet header offsets
Bitvector with temporary result
per cpu _array shared across the entire program chain
per cpu _array shared across the entire program chain
Each eBPF program can exploit a different matching algorithm (e.g., exact match, longest prefix match, etc).
Each eBPF program is injected only if there are rules operating on that field.
LBVS is implemented with a chain of eBPF programs, connected through tail calls.
Header parsing is done once and results are kept in a shared map for performance reasons
BPF based tail calls
![Page 14: Replacing iptables with eBPF in Kubernetes with Cilium · 2020-01-30 · TC BPF XDP CILIUM AGENT DAEMON CILIUM CLI CILIUM MONITOR CILIUM HEALTH CILIUM HEALTH NAMESPACE PLUGIN Build](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec557b313b08355f20a9fb2/html5/thumbnails/14.jpg)
14
BPF goes into...
● Load balancers - katran● perf● systemd● Suricata● Open vSwitch - AF_XDP● And many many others
![Page 15: Replacing iptables with eBPF in Kubernetes with Cilium · 2020-01-30 · TC BPF XDP CILIUM AGENT DAEMON CILIUM CLI CILIUM MONITOR CILIUM HEALTH CILIUM HEALTH NAMESPACE PLUGIN Build](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec557b313b08355f20a9fb2/html5/thumbnails/15.jpg)
15
BPF is used by...
![Page 16: Replacing iptables with eBPF in Kubernetes with Cilium · 2020-01-30 · TC BPF XDP CILIUM AGENT DAEMON CILIUM CLI CILIUM MONITOR CILIUM HEALTH CILIUM HEALTH NAMESPACE PLUGIN Build](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec557b313b08355f20a9fb2/html5/thumbnails/16.jpg)
1616
Cilium
![Page 17: Replacing iptables with eBPF in Kubernetes with Cilium · 2020-01-30 · TC BPF XDP CILIUM AGENT DAEMON CILIUM CLI CILIUM MONITOR CILIUM HEALTH CILIUM HEALTH NAMESPACE PLUGIN Build](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec557b313b08355f20a9fb2/html5/thumbnails/17.jpg)
17
What is Cilium?
![Page 18: Replacing iptables with eBPF in Kubernetes with Cilium · 2020-01-30 · TC BPF XDP CILIUM AGENT DAEMON CILIUM CLI CILIUM MONITOR CILIUM HEALTH CILIUM HEALTH NAMESPACE PLUGIN Build](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec557b313b08355f20a9fb2/html5/thumbnails/18.jpg)
18
CNI FunctionalityCNI is a CNCF ( Cloud Native Computing Foundation) project for Linux ContainersIt consists of specification and libraries for writing plugins.Only care about networking connectivity of containers
● ADD/DEL
General container runtime considerations for CNI:
The container runtime must
● create a new network namespace for the container before invoking any plugins● determine the network for the container and add the container to the each network by calling the corresponding plugins for each network● not invoke parallel operations for the same container.● order ADD and DEL operations for a container, such that ADD is always eventually followed by a corresponding DEL.● not call ADD twice ( without a corresponding DEL ) for the same ( network name, container id, name of the interface inside the container).
When CNI ADD call is invoked it tries to add the network to the container with respective veth pairs and assigning IP address from the respective IPAM Plugin or using the Host Scope.
When CNI DEL call is invoked it tries to remove the container network, release the IP Address to the IPAM Manager and cleans up the veth pairs.
![Page 19: Replacing iptables with eBPF in Kubernetes with Cilium · 2020-01-30 · TC BPF XDP CILIUM AGENT DAEMON CILIUM CLI CILIUM MONITOR CILIUM HEALTH CILIUM HEALTH NAMESPACE PLUGIN Build](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec557b313b08355f20a9fb2/html5/thumbnails/19.jpg)
19
Kubernetes API Server
Kubelet
CRI-Containerd
CNI-Plugin (Cilium)Cilium Agent
eth0
BPF Maps
Container2
Container1
Linux Kernel Network Stack 000 c1 FE 0A
001 54 45 31002 A1 B1 C1004 32 66 AA
cni-add()..
Kubectl
K8s PodUserspace
Kernel
bpf_syscall()
BPF Hook
Cilium CNI Plugin control Flow
![Page 20: Replacing iptables with eBPF in Kubernetes with Cilium · 2020-01-30 · TC BPF XDP CILIUM AGENT DAEMON CILIUM CLI CILIUM MONITOR CILIUM HEALTH CILIUM HEALTH NAMESPACE PLUGIN Build](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec557b313b08355f20a9fb2/html5/thumbnails/20.jpg)
20
VM1Cont
1Cont
2Cont
3 App
TC BPF
XDP
CILIUM AGENT DAEMON
CILIUM CLI CILIUM MONITOR
CILIUM HEALTH
CILIUM HEALTH NAMESPACE
PLUGIN
Build sk_buff
BPF
maps
Device Driver
Queueing and Forwarding
IP Layer
VirtualNet Devices
PHYSICAL LAYER ( NETWORK HARDWARE
TCP/UDP LayerAF-XDP
AF-INET AF-RAW
VM’s and Containers AppsCILIUM POD (Control Plane)
USER
SPACE
KERNEL
SPACE
NETWORK STACK with BPF hook points
Bpf_create_mapsSO_ATTACH_BPF
BPF(sockmap, sockopts
BPF-Cont3BPF-Cont2BPF-Cont1
BPF-Cilium Bp
f_loo
kup
_elemen
ts
CILIUM HOST_NET
CILIUMOPERATOR
Cilium Components with BPF hook points and BPF maps shown in Linux Stack Orchestrator
![Page 21: Replacing iptables with eBPF in Kubernetes with Cilium · 2020-01-30 · TC BPF XDP CILIUM AGENT DAEMON CILIUM CLI CILIUM MONITOR CILIUM HEALTH CILIUM HEALTH NAMESPACE PLUGIN Build](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec557b313b08355f20a9fb2/html5/thumbnails/21.jpg)
21
Cilium as CNI Plugin
container A container B container C
Cilium Networking CNI
K8s pod
K8s cluster
K8s node K8s node
K8s pod K8s pod
eth0 eth0 eth0
lxc0 lxc0 lxc1
eth0 eth0
![Page 22: Replacing iptables with eBPF in Kubernetes with Cilium · 2020-01-30 · TC BPF XDP CILIUM AGENT DAEMON CILIUM CLI CILIUM MONITOR CILIUM HEALTH CILIUM HEALTH NAMESPACE PLUGIN Build](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec557b313b08355f20a9fb2/html5/thumbnails/22.jpg)
22
Networking modes
Use case:Cilium handling routing between nodes
Encapsulation
Use case:Using cloud provider routers, using BGP routing daemon
Direct routing
Node A
Node B
Node C
VXLAN
VXLAN
VXLAN
Node A
Node B Node C
Cloud or BGProuting
![Page 23: Replacing iptables with eBPF in Kubernetes with Cilium · 2020-01-30 · TC BPF XDP CILIUM AGENT DAEMON CILIUM CLI CILIUM MONITOR CILIUM HEALTH CILIUM HEALTH NAMESPACE PLUGIN Build](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec557b313b08355f20a9fb2/html5/thumbnails/23.jpg)
23
Pod IP Routing - Overlay Routing ( Tunneling mode)
![Page 24: Replacing iptables with eBPF in Kubernetes with Cilium · 2020-01-30 · TC BPF XDP CILIUM AGENT DAEMON CILIUM CLI CILIUM MONITOR CILIUM HEALTH CILIUM HEALTH NAMESPACE PLUGIN Build](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec557b313b08355f20a9fb2/html5/thumbnails/24.jpg)
24
Pod IP Routing - Direct Routing Mode
![Page 25: Replacing iptables with eBPF in Kubernetes with Cilium · 2020-01-30 · TC BPF XDP CILIUM AGENT DAEMON CILIUM CLI CILIUM MONITOR CILIUM HEALTH CILIUM HEALTH NAMESPACE PLUGIN Build](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec557b313b08355f20a9fb2/html5/thumbnails/25.jpg)
25
L3 filtering – label based, ingress
PodLabels: role=frontend
IP: 10.0.0.1
PodLabels: role=frontend
IP: 10.0.0.2
PodIP: 10.0.0.5
PodLabels: role=backend
IP: 10.0.0.3
PodLabels: role=frontend
IP: 10.0.0.4
allow
deny
![Page 26: Replacing iptables with eBPF in Kubernetes with Cilium · 2020-01-30 · TC BPF XDP CILIUM AGENT DAEMON CILIUM CLI CILIUM MONITOR CILIUM HEALTH CILIUM HEALTH NAMESPACE PLUGIN Build](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec557b313b08355f20a9fb2/html5/thumbnails/26.jpg)
26
L3 filtering – label based, ingressapiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
description: "Allow frontends to access backends"
metadata:
name: "frontend-backend"
spec:
endpointSelector:
matchLabels:
role: backend
ingress:
- fromEndpoints:
- matchLabels:
class: frontend
![Page 27: Replacing iptables with eBPF in Kubernetes with Cilium · 2020-01-30 · TC BPF XDP CILIUM AGENT DAEMON CILIUM CLI CILIUM MONITOR CILIUM HEALTH CILIUM HEALTH NAMESPACE PLUGIN Build](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec557b313b08355f20a9fb2/html5/thumbnails/27.jpg)
27
L3 filtering – CIDR based, egress
IP: 10.0.1.1Subnet: 10.0.1.0/24
IP: 10.0.2.1Subnet: 10.0.2.0/24
allow
deny
Cluster A
PodLabels: role=backend
IP: 10.0.0.1
Any IP not belongingto 10.0.1.0/24
![Page 28: Replacing iptables with eBPF in Kubernetes with Cilium · 2020-01-30 · TC BPF XDP CILIUM AGENT DAEMON CILIUM CLI CILIUM MONITOR CILIUM HEALTH CILIUM HEALTH NAMESPACE PLUGIN Build](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec557b313b08355f20a9fb2/html5/thumbnails/28.jpg)
28
L3 filtering – CIDR based, egressapiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
description: "Allow backends to access 10.0.1.0/24"
metadata:
name: "frontend-backend"
spec:
endpointSelector:
matchLabels:
role: backend
egress:
- toCIDR:
- IP: “10.0.1.0/24”
![Page 29: Replacing iptables with eBPF in Kubernetes with Cilium · 2020-01-30 · TC BPF XDP CILIUM AGENT DAEMON CILIUM CLI CILIUM MONITOR CILIUM HEALTH CILIUM HEALTH NAMESPACE PLUGIN Build](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec557b313b08355f20a9fb2/html5/thumbnails/29.jpg)
29
L4 filteringapiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
description: "Allow to access backends only on TCP/80"
metadata:
name: "frontend-backend"
spec:
endpointSelector:
matchLabels:
role: backend
ingress:
- toPorts:
- ports:
- port: “80”
protocol: “TCP”
![Page 30: Replacing iptables with eBPF in Kubernetes with Cilium · 2020-01-30 · TC BPF XDP CILIUM AGENT DAEMON CILIUM CLI CILIUM MONITOR CILIUM HEALTH CILIUM HEALTH NAMESPACE PLUGIN Build](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec557b313b08355f20a9fb2/html5/thumbnails/30.jpg)
30
L4 filtering
PodLabels: role=backend
IP: 10.0.0.1
allow
deny
TCP/80
Any other port
![Page 31: Replacing iptables with eBPF in Kubernetes with Cilium · 2020-01-30 · TC BPF XDP CILIUM AGENT DAEMON CILIUM CLI CILIUM MONITOR CILIUM HEALTH CILIUM HEALTH NAMESPACE PLUGIN Build](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec557b313b08355f20a9fb2/html5/thumbnails/31.jpg)
31
L7 filtering – API Aware Security
PodLabels: role=api
IP: 10.0.0.1
GET /articles/{id}
GET /private
PodIP: 10.0.0.5
![Page 32: Replacing iptables with eBPF in Kubernetes with Cilium · 2020-01-30 · TC BPF XDP CILIUM AGENT DAEMON CILIUM CLI CILIUM MONITOR CILIUM HEALTH CILIUM HEALTH NAMESPACE PLUGIN Build](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec557b313b08355f20a9fb2/html5/thumbnails/32.jpg)
32
L7 filtering – API Aware SecurityapiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
description: "L7 policy to restict access to specific HTTP endpoints"
metadata:
name: "frontend-backend" endpointSelector:
matchLabels:
role: backend
ingress:
- toPorts:
- ports:
- port: “80”
protocol: “TCP”
rules:
http:
- method: "GET"
path: "/article/$"
![Page 33: Replacing iptables with eBPF in Kubernetes with Cilium · 2020-01-30 · TC BPF XDP CILIUM AGENT DAEMON CILIUM CLI CILIUM MONITOR CILIUM HEALTH CILIUM HEALTH NAMESPACE PLUGIN Build](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec557b313b08355f20a9fb2/html5/thumbnails/33.jpg)
33
Standalone proxy, L7 filtering
Node APod A
+ BPF
Envoy
Generating BPF programs forL7 filtering through libcilium.so
Node BPod B
+ BPF
Envoy
Generating BPF programs forL7 filtering through libcilium.so
Generating BPF programsfor L3/L4 filtering
Generating BPF programsfor L3/L4 filtering
VXLAN
![Page 34: Replacing iptables with eBPF in Kubernetes with Cilium · 2020-01-30 · TC BPF XDP CILIUM AGENT DAEMON CILIUM CLI CILIUM MONITOR CILIUM HEALTH CILIUM HEALTH NAMESPACE PLUGIN Build](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec557b313b08355f20a9fb2/html5/thumbnails/34.jpg)
34
Features
![Page 35: Replacing iptables with eBPF in Kubernetes with Cilium · 2020-01-30 · TC BPF XDP CILIUM AGENT DAEMON CILIUM CLI CILIUM MONITOR CILIUM HEALTH CILIUM HEALTH NAMESPACE PLUGIN Build](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec557b313b08355f20a9fb2/html5/thumbnails/35.jpg)
35
Cluster Mesh
Cluster A Cluster B
Node APod A
+ BPF
Node B
+ BPF
Container
eth0
Pod BContainer
eth0
Pod CContainer
eth0
External etcd
Node APod A
+ BPF
Container
eth0
![Page 36: Replacing iptables with eBPF in Kubernetes with Cilium · 2020-01-30 · TC BPF XDP CILIUM AGENT DAEMON CILIUM CLI CILIUM MONITOR CILIUM HEALTH CILIUM HEALTH NAMESPACE PLUGIN Build](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec557b313b08355f20a9fb2/html5/thumbnails/36.jpg)
36
Istio (Transparent Sidecar injection) without Cilium
Socket Socket Socket Socket
Service Service
Socket
TCP/IP
Ethernet
eth0
Socket
TCP/IP
Ethernet
eth0
Network
K8s Pod K8s Pod
K8s Node
TCP/IP
Ethernet
IPtablesIPtables
TCP/IP
Ethernet
IPtables
Loopback
IPtables IPtables IPtables
TCP/IP TCP/IP
Ethernet Ethernet
Loopback
![Page 37: Replacing iptables with eBPF in Kubernetes with Cilium · 2020-01-30 · TC BPF XDP CILIUM AGENT DAEMON CILIUM CLI CILIUM MONITOR CILIUM HEALTH CILIUM HEALTH NAMESPACE PLUGIN Build](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec557b313b08355f20a9fb2/html5/thumbnails/37.jpg)
37
Istio with cilium and sockmap
Cilium CNI Cilium CNI
Socket Socket Socket Socket
Service Service
Socket
TCP/IP
Ethernet
eth0
Socket
TCP/IP
Ethernet
eth0
Network
K8s Pod K8s Pod
K8s Node
![Page 38: Replacing iptables with eBPF in Kubernetes with Cilium · 2020-01-30 · TC BPF XDP CILIUM AGENT DAEMON CILIUM CLI CILIUM MONITOR CILIUM HEALTH CILIUM HEALTH NAMESPACE PLUGIN Build](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec557b313b08355f20a9fb2/html5/thumbnails/38.jpg)
38
Istio
Service A Service B Service C
Cilium Networking CNI
K8s pod
K8s cluster
K8s node K8s node
K8s pod K8s pod
IstioPilot/Mixer/Citadel
![Page 39: Replacing iptables with eBPF in Kubernetes with Cilium · 2020-01-30 · TC BPF XDP CILIUM AGENT DAEMON CILIUM CLI CILIUM MONITOR CILIUM HEALTH CILIUM HEALTH NAMESPACE PLUGIN Build](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec557b313b08355f20a9fb2/html5/thumbnails/39.jpg)
39
Istio - Mutual TLS
Service A Service B
Cilium Networking CNI
K8s pod
K8s cluster
K8s node K8s node
K8s pod
IstioPilot/Mixer/Citadel
Mutual TLS
![Page 40: Replacing iptables with eBPF in Kubernetes with Cilium · 2020-01-30 · TC BPF XDP CILIUM AGENT DAEMON CILIUM CLI CILIUM MONITOR CILIUM HEALTH CILIUM HEALTH NAMESPACE PLUGIN Build](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec557b313b08355f20a9fb2/html5/thumbnails/40.jpg)
40
Istio - Deferred kTLS
Service A Service B
Cilium Networking CNI
K8s pod
K8s cluster
K8s node K8s node
K8s pod
IstioPilot/Mixer/Citadel
Deferred kTLSencryption
External Github Service
ExternalCloud Network
![Page 41: Replacing iptables with eBPF in Kubernetes with Cilium · 2020-01-30 · TC BPF XDP CILIUM AGENT DAEMON CILIUM CLI CILIUM MONITOR CILIUM HEALTH CILIUM HEALTH NAMESPACE PLUGIN Build](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec557b313b08355f20a9fb2/html5/thumbnails/41.jpg)
41
Kubernetes Services
● Hash table.
BPF, Cilium
● Linear list.● All rules in the chain have to be replaced as a whole.
Iptables, kube-proxy
Key
Key
Key
Value
Value
Value
Rule 1
Rule 2
Rule n
...Search O(1)InsertO(1)Delete O(1)
Search O(n)InsertO(1)Delete O(n)
![Page 42: Replacing iptables with eBPF in Kubernetes with Cilium · 2020-01-30 · TC BPF XDP CILIUM AGENT DAEMON CILIUM CLI CILIUM MONITOR CILIUM HEALTH CILIUM HEALTH NAMESPACE PLUGIN Build](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec557b313b08355f20a9fb2/html5/thumbnails/42.jpg)
42
usec
number of services in cluster
![Page 43: Replacing iptables with eBPF in Kubernetes with Cilium · 2020-01-30 · TC BPF XDP CILIUM AGENT DAEMON CILIUM CLI CILIUM MONITOR CILIUM HEALTH CILIUM HEALTH NAMESPACE PLUGIN Build](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec557b313b08355f20a9fb2/html5/thumbnails/43.jpg)
43
CNI chaining
Policy enforcement, load balancing,multi-cluster connectivity
IP allocation, configuring networkinterface, encapsulation/routing inside the cluster
![Page 44: Replacing iptables with eBPF in Kubernetes with Cilium · 2020-01-30 · TC BPF XDP CILIUM AGENT DAEMON CILIUM CLI CILIUM MONITOR CILIUM HEALTH CILIUM HEALTH NAMESPACE PLUGIN Build](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec557b313b08355f20a9fb2/html5/thumbnails/44.jpg)
44
Native support for AWS ENI
![Page 45: Replacing iptables with eBPF in Kubernetes with Cilium · 2020-01-30 · TC BPF XDP CILIUM AGENT DAEMON CILIUM CLI CILIUM MONITOR CILIUM HEALTH CILIUM HEALTH NAMESPACE PLUGIN Build](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec557b313b08355f20a9fb2/html5/thumbnails/45.jpg)
45
HUBBLE
Hubble is a fully distributed networking and security observability platform for cloud native workloads.It is built on top of Cilium and eBPF to enable deep visibility in a transparent manner.
Hubble provides
● Service dependencies and communication map● Operational monitoring and alerting● Application monitoring● Secure observability
Known limitations of Hubble:
● Hubble is in beta● Not all components of Hubble are covered by automated testing.● Architecture is scalable but not all code paths have been optimized for efficiency and scalability yet
![Page 46: Replacing iptables with eBPF in Kubernetes with Cilium · 2020-01-30 · TC BPF XDP CILIUM AGENT DAEMON CILIUM CLI CILIUM MONITOR CILIUM HEALTH CILIUM HEALTH NAMESPACE PLUGIN Build](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec557b313b08355f20a9fb2/html5/thumbnails/46.jpg)
46
HUBBLE Components
The following components make up Hubble:
● Hubble Agent○ The Hubble Agent is what runs on each worker node. It interacts with the Cilium agent running on the
same node and serves the flow query API as well as the metrics.● Hubble Storage
○ Hubble storage layer consists of an in-memory storage able to store a fixed number of flows per node.● Hubble CLI
○ The CLI connects to the flow query API of a Hubble agent running on a node and allows to query the flows stored in the in-memory storage using server-side filtering.
● Hubble UI○ The Hubble UI uses the flow query API to provide a graphical service communication map based on the
observed flows.
![Page 47: Replacing iptables with eBPF in Kubernetes with Cilium · 2020-01-30 · TC BPF XDP CILIUM AGENT DAEMON CILIUM CLI CILIUM MONITOR CILIUM HEALTH CILIUM HEALTH NAMESPACE PLUGIN Build](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec557b313b08355f20a9fb2/html5/thumbnails/47.jpg)
47
Hubble running on top of Cilium and eBPF
![Page 48: Replacing iptables with eBPF in Kubernetes with Cilium · 2020-01-30 · TC BPF XDP CILIUM AGENT DAEMON CILIUM CLI CILIUM MONITOR CILIUM HEALTH CILIUM HEALTH NAMESPACE PLUGIN Build](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec557b313b08355f20a9fb2/html5/thumbnails/48.jpg)
48
Hubble Service Maps
![Page 49: Replacing iptables with eBPF in Kubernetes with Cilium · 2020-01-30 · TC BPF XDP CILIUM AGENT DAEMON CILIUM CLI CILIUM MONITOR CILIUM HEALTH CILIUM HEALTH NAMESPACE PLUGIN Build](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec557b313b08355f20a9fb2/html5/thumbnails/49.jpg)
49
Hubble HTTP metrics
![Page 50: Replacing iptables with eBPF in Kubernetes with Cilium · 2020-01-30 · TC BPF XDP CILIUM AGENT DAEMON CILIUM CLI CILIUM MONITOR CILIUM HEALTH CILIUM HEALTH NAMESPACE PLUGIN Build](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec557b313b08355f20a9fb2/html5/thumbnails/50.jpg)
5050
To sum it up
![Page 51: Replacing iptables with eBPF in Kubernetes with Cilium · 2020-01-30 · TC BPF XDP CILIUM AGENT DAEMON CILIUM CLI CILIUM MONITOR CILIUM HEALTH CILIUM HEALTH NAMESPACE PLUGIN Build](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec557b313b08355f20a9fb2/html5/thumbnails/51.jpg)
51
Why Cilium is awesome?
● It makes disadvantages of iptables disappear. And always gets the best from the Linux kernel.● Cluster Mesh / multi-cluster.● Makes Istio faster.● Offers L7 API Aware filtering as a Kubernetes resource. ● Integrates with the other popular CNI plugins – Calico, Flannel, Weave, Lyft, AWS CNI.
![Page 52: Replacing iptables with eBPF in Kubernetes with Cilium · 2020-01-30 · TC BPF XDP CILIUM AGENT DAEMON CILIUM CLI CILIUM MONITOR CILIUM HEALTH CILIUM HEALTH NAMESPACE PLUGIN Build](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec557b313b08355f20a9fb2/html5/thumbnails/52.jpg)