securing & enforcing network policy and encryption with weave net

Securing & Enforcing Network Policy with Weave Net

Luke Marsden, Developer Experience@lmarsden

+

Agenda1. What is container networking? 2. Weave Net – what it is and how it works 3. Encryption & performance, Weave Net case

study 4. What is Kubernetes Network Policy & why do I

need it? 5. Kubernetes Network Policy example & demo

What is container networking?

wait, back up…

What is networking?

What is networking?

Layer 2 – e.g. Ethernet

Layer 3 – e.g. IP

Layer 7 – e.g. HTTP

…

Layer 4 – e.g. TCP, UDP

What is networking?

VM10.0.0.3

VM10.0.0.4

VM10.0.0.1

VM10.0.0.2


Layer 3 – e.g. IP


…

Layer 4 – e.g. TCP, UDPL2 connectivity

Adding containers


Layer 3 – e.g. IP


…

VM10.0.0.3

VM10.0.0.4

VM10.0.0.1

VM10.0.0.2

C1

C2

C3

L2 connectivityLayer 4 – e.g. TCP, UDP

Original Docker model

VM10.0.0.1

C1

C2

docker run -p 80:8080 nginx

:80

:8080

VM10.0.0.2

C3


VM10.0.0.1

C1

C2

docker run -p 80:8080 nginx

:80

:8080

eth0 docker0vethX

NAT

VM10.0.0.2

C3


VM10.0.0.1

VM10.0.0.2

C2

docker run -p 3306:3306 mysql

:80C1:8080

eth0 docker0vethX

NATC3:3306

eth0 docker0vethX:3306

New model used by CNI (Kubernetes, etc) and

Docker Overlay (libnetwork)


Layer 3 – e.g. IP


…


elevates the baseline for platforms from L2 to L3

VM VM

C110.0.0.1

C210.0.0.2

C310.0.0.3

New model IP address per container or pod

VM

C110.0.0.1

C210.0.0.2

C310.0.0.3

New model IP addresses for “services” as well,

aggregation of pods

service VIP10.1.0.1

“api”

VM

C110.0.0.1

C210.0.0.2

C310.0.0.3

Problem

service VIP10.1.0.1

how do you provide connectivity

between hosts when each container

needs its own IP? in any environment?

Options for IP-per-pod connectivity• Underlay, or “programmable fabric”

• IaaS provides the routing • Kubernetes supports this with

GCE, AWS • “Send all traffic to 10.1.1.0/24 to this

VM” • Limit on the number of routes per host • Network typically limited to a region • Hard to do cross or hybrid-cloud


Layer 3 – e.g. IP


…


Options for IP-per-pod connectivity• Overlay, like Weave Net

• Encapsulates L2 Ethernet frames on top of L4 to provide a “flat virtual L2 network”

• Expose virtual L2 network back to Linux which then runs normally (ARP, etc)

• It can run anywhere, only requirement is UDP/TCP between hosts

• Can create virtual networks which span networks, even across NAT

• Automatically allocates IP addresses for containers with IPAM


Layer 3 – e.g. IP

Layer 7 – e.g. HTTP…



Layer 3 – e.g. IP


Options for IP-per-pod connectivity• Overlay, like Weave Net

• Encapsulates L2 Ethernet frames on top of L4 to provide a “flat virtual L2 network”

• Expose virtual L2 network back to Linux which then runs normally (ARP, etc)

• It can run anywhere, only requirement is UDP/TCP between hosts

• Can create virtual networks which span networks, even across NAT

• Automatically allocates IP addresses for containers with IPAM


Layer 2

Layer 3

Layer 4

Layer 2

Layer 3

Layer 4


Layer 3 – e.g. IP


AWS DC

Doesn’t that sound a bit slow?• Actually, no • Graph shows

fastdp overlay within 89% of native on two AWS c3.8xlarge instances

• We’ll come on to encryption later…

Weave Net – what it is and how it works• Control plane, gossip &

CRDTs• Forms a dynamic mesh of

peers with least cost routing • Favours availability &

partition tolerance, like the Internet

• Only needs consensus initially when allocating IPAM ranges

• Data path, a few options• Fastdp, uses VXLAN to form

tunnels which are handled entirely in kernel, fast

• Sleeve, fallback user-space mode which does packet sniffing like wireshark (libpcap)

• AWS VPC mode, directly “programs the fabric”

Fastdp in diagrams

Encryption & performance, Weave Net case study

• Prior to Weave Net 1.9, you had to choose between • fast • secure

• Now you can have both! • We added IPsec support to our fastdp data path

mode

Encryption & performance, Weave Net case study

• Why did we choose IPsec?

VXLAN and IPsec• VXLAN

• Encapsulates Ethernet traffic over UDP, forming a “tunnel”

• Implemented in Weave Net using OVS, Open vSwitch kernel module

• IPsec • Secures the VXLAN UDP packets which

contain the overlay’s L2 packets • Weave Net’s control plane sets this all up for

youLayer 2 – e.g. Ethernet

Layer 3 – e.g. IP




Layer 3 – e.g. IP


VXLAN / OVS

IPsec / ESP

Weave Network Policy ControllerBlocking unwanted network traffic in Kubernetes

What is Kubernetes Network Policy and why do I need it?

with thanks to @bboreham for the

weave-npc slides!

Threat Model

Traditional defence

Problem

Solution

Now make it dynamic

Example

Presentation Tier

Middle Tier

Data Tier

Kubernetes NetworkPolicykind: NetworkPolicy metadata: name: presentation-policy spec: podSelector: tier: presentation ingress: - ports: - protocol: tcp port: 80

Presentation Tier

Middle Tier

Data Tier

:80

Kubernetes NetworkPolicykind: NetworkPolicy metadata: name: middle-tier-policy spec: podSelector: tier: middle ingress: - from: - podSelector: matchLabels: tier: presentation

Presentation Tier

Middle Tier

Data Tier

So how do we implement this?

Controller

Kubernetes Master

host1weave-npc

host2weave-npc

iptables iptables

watch on policies, pods

Top-level iptables rulesFORWARD chain:-o weave -j WEAVE-NPC -o weave -j DROP

WEAVE_NPC chain:-m state --state RELATED,ESTABLISHED -j ACCEPT -m state --state NEW -j WEAVE-NPC-DEFAULT -m state --state NEW -j WEAVE-NPC-INGRESS

Overall flow

weave-npc

src bridge iptables

ipsetipsetipset

dst

Per-policy iptables rulesWEAVE-NPC-DEFAULT chain: -m set --match-set weave-v/q_G.;Q?uK]BuDs2 dst -j ACCEPT -m set --match-set weave-k?Z;25^M}|1s7P3|H dst -j ACCEPT ... WEAVE-NPC-INGRESS chain: -m set --match-set weave-LuMDZrBg:KsT9Xll[ src -m set --match-set weave-hR9K[Olp~d>@1wQu/ dst -j ACCEPT -m set --match-set weave-hR9K[Olp~d>@1wQu/ src -m set --match-set weave-hR9K[Olp~d>@1wQu/ dst -j ACCEPT ...

What could possibly go wrong?Back in the FORWARD chain: -o weave -m state --state NEW -j NFLOG --nflog-group 86

We subscribe to this via ulogd so we can print:TCP connection from 10.32.0.7:56648 to 10.32.0.11:80 blocked by Weave NPC.

Also exported as a Prometheus metric

Demo in 2 parts

Part 1 - Encryption: https://www.katacoda.com/courses/weave/net-security-

training

Demo in 2 parts

Part 2 - Network Policy: https://www.weave.works/guides/cloud-testdrive-part-4-secure-container-firewalls/

Join the Weave user group!

meetup.com/pro/Weave/

weave.works/help

Other topics

• Kubernetes 101 • Continuous delivery: hooking up my CI/CD

pipeline to Kubernetes • How do I monitor this stuff?

We have talks on all these topics in the Weave user group!

Thanks! Questions?

We are hiring!DX in San Francisco

Engineers in London & SF

weave.works/weave-company/hiring

See also:weave.works/docs/net/latest/how-it-works/github.com/weaveworks/go-odptools.ietf.org/html/rfc2406

Appendix

Weave Net

• Pod network • Container overlay network • IP per container

Weave Mesh = Gossip + CRDTs

• Golang library used by Weave Net & WeaveDNS

Gossip

1

Gossip

1 1

1 1

Conflict!

1 2

1 2

CRDTs• Merge operation ⊕ for data structure such that:

• Associative — A ⊕ (B ⊕ C) = (A ⊕ B) ⊕ C • Commutative — A ⊕ B = B ⊕ A • Idempotent — A ⊕ A = A

Examples of CRDTs• Add-only set • Latest-value-wins singleton (using timestamp) • Set with deletion: give every element an id,

maintain a set of “deleted elements” using add-only set

• Many more… http://arxiv.org/abs/1608.03960

Using Weave Mesh• General form of update: A ⊕ B ⇒ C

A = local state, B = incoming update, C = resulting state

• Golang interface: • mergeReceived = B • mergeDelta = difference from A to C • mergeComplete = C

Benefits• Weave Net • No centralised store needed for network • Carries on working during network partitions • Weave heals seamlessly when the network

heals • Particularly interesting in federated clusters…