nsx-t deep dive: kubernetes networking · nginx tcp/80 mgmt tcp/22 logging udp/514 ipc external ip...

#vmworld

CNET1270BU

NSX-T Deep Dive:Kubernetes Networking

Yasen Simeonov, VMware, Inc.

#CNET1270BU

©2019 VMware, Inc.

Disclaimer

This presentation may contain product features or functionality that are currently under development.

This overview of new technology represents no commitment from VMware to deliver these features in any generally available product.

Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.

Technical feasibility and market demand will affect final delivery.

Pricing and packaging for any new features/functionality/technology discussed or presented, have not been determined.

2

The information in this presentation is for informational purposes only and may not be incorporated into any contract. There is no commitment or obligation to deliver any items presented herein.

©2019 VMware, Inc. 3

Public

Users

VMs, Containers, Microservices

VMware partners(VMC)

Private DataCenters

Telco Networks

Things

Private Cloud(VCF)

Edge

VCN

10k customers

to date

82% of Fortune 100

enterprises

70%of all Fortune

global 500 telcos

Gartner

MQ LeaderWAN Edge

Infrastructure

Ties it all together.

Virtual Cloud Networking


ResourcesHow to get started

Design Guides Demos

Take a Hands-on Lab Join VMUG, VMware Communities (VMTN)

LEARN TRY

VMware.com/go/NSXtechzone

CONNECTTRY

@VMwareNSX#runNSX

©2019 VMware, Inc.

Agenda

5

Kubernetes Overview

NSX-T Integration with Kubernetes

Demo: The new topology

Summary

6©2019 VMware, Inc.

Kubernetes OverviewNetworking focused


Kubernetes Components

K8s Cluster Consists of Master(s) and Nodes

K8s Master Components

• API Server

• Scheduler

• Controller Manager

• Dashboard

K8s Node Components

• Kubelet

• Kube-Proxy

• Containers Runtime

K8s masterK8s master

K8s Master

Controller Manager

K8s APIServer

Key-Value Store

dashboard

Scheduler

K8s nodeK8s node

K8s nodeK8s node

K8s Nodes

kubelet c runtime

Kube-proxy

> _ Kubectl

CLI

K8s Master(s)


Kubernetes Namespace

Namespaces are a way to divide cluster resources amongst users and groups

They can be thought of as Tenants

They are a way to provide Resources Quotas, RBAC, Networking Multitenancy, and Name uniqueness

Namespace: fooBase URI: /api/v1/namespaces/foo

‚redis-master‘ Pod:/api/v1/namespaces/foo/pods/redis-master

‚redis‘ service:/api/v1/namespaces/foo/services/redis-master

Namespace: barBase URI: /api/v1/namespaces/bar

‚redis-master‘ Pod:/api/v1/namespaces/bar/pods/redis-master

‚redis‘ service:/api/v1/namespaces/bar/services/redis-master


Kubernetes Pod

A Pod is a group of one or more containers that shares an IP address and a Data Volume

Pod

pause container(‘owns’ the IP stack)

10.24.0.0/16

10.24.0.2

nginxtcp/80

mgmttcp/22

loggingudp/514

IPC

External IP Traffic


K8s Master

Replication Controller:

• The replication controller enforces the 'desired' state of a collection of Pods. E.g. it makes sure that 4 Pods are always running in the cluster

Replica Set:

• Replica Set is the next-generation Replication Controller. Replica Set use Set-Based selectors while replication controllers use Equity-Based selectors.

Kubernetes RC & RS

Kubernetes Replication Controller (rc) and Replica Set (rs)

Replication Controller / Replica Set

Pods


Daemon Sets:

• A DaemonSet ensures that all (or some) nodes run a copy of a Pod.

• As nodes are added to the cluster, Pods are added to them.

• As nodes are removed from the cluster, those Pods are garbage collected

• Deleting a Daemon Set will clean up the pods it created

• Daemon Sets are used to replace SystemdUnits in a lot of cases today

Kubernetes Daemon Set

Kubernetes Daemon Set

K8s Node

InfraPod

K8s Node

InfraPod

K8s Node

InfraPod

K8s Node

InfraPod

K8s Master

Deamon Set


Kubernetes Service

A Kubernetes Service defines a logical set of Pods, selected with matching labels

Serves multiple functions:

• Service Discovery / DNS

• East/West load balancing in the Cluster (Type: ClusterIP)

• External load balancing for L4 TCP/UDP (Type:

LoadBalancer)

• External access to the service through the nodes IPs (Type: NodePort)

Redis Slave Pods

redis-slave svc

10.24.0.5

ClusterIP172.30.0.24

Web Front-EndPods

10.24.2.7

▶ kubectl describe svc redis-slave

Name: redis-slave

Namespace: default

Labels: name=redis-slave

Selector: name=redis-slave

Type: LoadBalancer

IP: 172.30.0.24

LoadBalancer Ingress: 134.247.200.20

Port: <unnamed> 6379/TCP

Endpoints: 10.24.0.5:6379,

10.24.2.7:6379

DNS:

redis-slave.<ns>.cluster.local ➔ 172.30.0.24

ExternalIP134.247.200.20

DNS:

redis-slave.external.com ➔ 134.247.200.20


Kubernetes Ingress

A Kubernetes Ingress Object is a L7 LoadBalancing rule that binds a hostname and url to a Service

The LoadBalancer Datapath can be implemented as an external Load Balancer or as a K8s Pod

Web Front-EndPods (shop svc)

http://www.bikeshop.com/shop

Web Front-EndPods (special-offers svc)

http://www.bikeshop.com/special-offers

LoadBalancer Datapath

(External or K8s Pods)

▶ kubectl describe ingress bikeshop-ingress-shopName: bikeshop-shopNamespace: bikeshopAddress: 100.64.240.9,134.247.200.1Default backend: default-http-backend:80 (<none>)

Rules:Host Path Backends---- ---- --------www.bikeshop.com /shop

web-svc-1:80 (<none>)

External IP: 134.247.200.1

DNS: *.bikeshop.com ➔ 134.247.200.1


NSX-T Data Center &KubernetesHow NSX-T does Kubernetes Networking


Key Design Goals of the NSX-T Data Center Kubernetes Integration

Don't stand in the way of the developer

!

Provide solutions to map the Kubernetes

constructs to enterprise networking constructs

Secure Containers, VMs and any other

endpoints with overarching Firewall

Policies

Provide visibility & troubleshooting tools to ease the

container adoption in the enterprise


Dynamic per Namespace Topology

NSX-T K8s Integration – Namespaces & Pods

admin@k8s-master:~$ kubectl create namespace foonamespace ”foo" created

admin@k8s-master:~$ kubectl create namespace barnamespace ”bar" created

admin@k8s-master:~$ kubectl run nginx-foo --image=nginx -n foodeployment "nginx-foo" created

admin@k8s-master:~$ kubectl run nginx-bar --image=nginx -n bardeployment "nginx-bar" created

Namespace: foo

Namespace: bar

NSX / K8s topology

10.24.0.0/24 10.24.1.0/24

10.24.2.0/24

NAT boundary

NAT boundary

K8s nodesK8s Masters


NSX-T K8s Integration – Routed Namespaces

admin@k8s-master:~$ vim no-nat-namespace.yaml

apiVersion: v1kind: Namespacemetadata:

name: no-nat-namespaceannotations:

ncp/no_snat: "true“

admin@k8s-master:~$ kubectl create –f no-nat-namespace.yamlnamespace ”no-nat-namespace" created

admin@k8s-master:~$ kubectl run nginx-no-nat --image=nginx –n no-nat-namespacedeployment "nginx-k8s" created

Namespace: no-nat-namespace

NSX / K8s topology

114.4.10.0/26

Direct Routing

114.4.10.64/26



Shared T1 for all Namespaces in the Cluster - Both for Policy and MP

NSX-T K8s Integration – Shared T1 Topology

NSX / K8s topology

10.24.0.0/24 10.24.2.0/24


T1

admin@k8s-master:~$ kubectl create namespace foonamespace ”foo" created

admin@k8s-master:~$ kubectl create namespace barnamespace ”bar" created

admin@k8s-master:~$ kubectl run nginx-foo --image=nginx -n foodeployment "nginx-foo" created

admin@k8s-master:~$ kubectl run nginx-bar --image=nginx -n bardeployment "nginx-bar" created

Active/StandBy T0Active/Active T0

NAT boundary

NSX-T 2.5/NCP 2.5


Simplified UI

Policy Support NCP 2.5


K8s / NSX-T Data Center Components

NCP is a software component provided by VMware in form of a container image, e.g. to be run as a K8s Pod.

NCP is build in a modular way, so that individual adapters can be added for different CaaS and PaaS systems at some point

NSX Container Plugin (NCP)

NCM

Infra

K8s / OSAdapter

CloudFoundry

Adapter

NSX Container Plugin

More…

NSX Manager

API Client

NSX Manager

NS: foo NS: bar

NSX/ K8s topology

K8s master

etcd

API-Server

Scheduler


NodeVM

DFW

eth2

NodeVM

DFW

eth0

Minion Mgmt. IP Stack

eth0


mgmtnetwork

OVS

mgmtnetwork

Vla

n10

vla

n11

Sub VIF

eth2

vla

n10

vla

n11

OVS

NSX CNI Plugin &

Node Agent

Pods

PodsNSX CNI Plugin &

Node Agent

Sub VIF

Sub VIF Sub VIF

K8s Node VMs: Most customers are looking to deploy K8s Nodes as VMs today

Nested Network-Virtualization: Instead of terminating the overlay tunnels in the Node VM, we are extending the Hypervisor vSwitch into the Node VM using VLAN tagging. The Node VM vSwitch (OVS) is ‘standalone’, and only gets programed by the NSX CNI Plugin

Benefits:

• Enhanced security through strong isolation of the Node VM from the NSX Control-Plane

• Less transport-nodes in NSX which equates to higher scale

Container Interfaces in NSX-TNSX-T Sub-VIF Interfaces


Kubelet: calls the NSX CNI plugin.

NSX CNI Plugin: is a simple python script that mainly translates between Kubelet and NSX Node Agent. It uses a simple private protocol to talk to the Node Agent over a unixsocket

NSX Node Agent: runs as a DeamonSet in HostNetworkMode and is responsible for:

1. The retrieval of IP/MAC/VLAN information from Hyperbus (see next 2 slides) through an NSX-RPC TCP connection from the Hypervisor

2. The creation and configuration of the uplink VLAN interfaces to the Hypervisor vSwitch in OVS

3. The creation and configuration of downlink interfaces connecting OVS to the Pods 'pause container' IP Namespace

NSX Kube Proxy: Is responsible for the creation of flow rules and Load Balancing rules in OVS to do the K8s Service East/West LB (will be covered later again). It is also responsible to create the needed flow/NAT entries needed for the Node-Agent to Hyperbus communication to work

Node Agent & CNI PluginNode Agent & CNI Plugin

eth0


eth2

vla

n10

vla

n11

OVS

Pods

Hostnetwork-Mode

Node Agent & NSX Kube-Proxy

DeamonSet

NSX CNI Pluginkubelet


Pod attachment workflow

NCM

Infra

K8s / OSAdapter

CloudFoundry Adapter

Libnetwork Adapter


More…

NSX Manager

API Client

NSX Manager

K8s master

etcd

API-Server

Scheduler

NS: foo NS: bar

NSX/ K8s topology

1)2)

3)

4)

1. NCP creates a ‚watch‘ on K8s API for any Pod events

2. A user creates a new K8s Pod

3. The K8s API Server notifies NCP of the change (addition) of Pods

4. NCP creates a logical port:

a) Requests an IP from the Namespace/LS Subnet

b) Request a MAC from the container MAC pool in NSX

c) Assigns a VLAN for the Podd) Creates a logical port (Sub-VIF / CIF) on

the Namespace LS and assigns the IP, MAC and VLAN to the logical port

e) Adds all K8s Pod Labels as Tags to the logical port

Shared under NDA

K8s / NSX Workflows (1/2)Pod attachment workflow


K8s / NSX Workflows (2/2)Pod attachment workflow Pod attachment workflow

Hypervisor(ESXi &

KVM)

NodeVM

Vla

n4

09

4

Vla

n2

NSX Hyperbus

cif

kubelet

Node Agent / CNI Plugin

NSX Manager

NSX Controllers

NSX LCP

5)

6)

7)

8)

9)

5. NSX LCP will create the LP on the hypervisor

6. Hyperbus monitors LCP for new CIF interfaces and learns the CIF’s Id/IP/MAC/VLAN binding

7. Kubelet sees a new ‘PodSpec’ from the K8s Master and starts a new Pod. It executes the NSX cni plugin binary to do the ‘network wiring’ of the Pod – This call is proxy'd to the NSX Node Agent

8. The NSX Node Agent gets the CIF’s Id/IP/MAC/VLAN binding data from the Hyperbus over the isolated and secured channel (one-way connection establishment)

9. The Node Agent creates the OVS port with the right VLAN, and configures the Pods network interface to connect to OVS with the received IP/MAC. After this, Kubelet is un-blocked and the Pod creation succeeds


Tenancy / Topology MappingPersistent IPs for K8s Namespaces

With NSX-T each Tenant (Kubernetes Namespace) either gets its own SNAT IP (NAT Mode), or is directly identifiable by its source subnet (No NAT Mode)

Node VM

OpenvSwitch

10.12.5.5/2410.12.1.8/24

172.16.1.11/24

mgmt IP

vn

ic

Namesp. FooT1 router

PAS VMsT1 router

VLAN Trunk

NSX-T Logical Switch

Namesp. BarT1 router

172.16.1.1/24 10.12.1.1/24 10.12.5.1/24

Pods

Database (VM based or Physical)

Physical DC Firewall

A new SNAT IP is allocated on the T0 router for each Tenant for NAT Mode

In NAT Mode, the external DC Firewall and the DB can distinguish tenant 'foo' and tenant 'bar' using the source SNAT IP that is allocated to a specific Tenant.

Tenant: fooTenant: bar

In No-NAT Mode, the external DC Firewall and the DB can distinguish tenant 'foo' and tenant 'bar' using the source IP Subnet that is allocated to a specific Tenant.


Infrastructure Teams can pre-create Firewall rules in existing DC physical Firewalls to allow traffic from specific workloads in K8s

The K8s user / DevOps can deploy applications that are easily identifiable in the physical network

With this feature a set of Kubernetes Workloads (Pods) can be assigned to use a specific IP or group of SNAT IPs to source their traffic from

Feature

Benefits

Persistent SNAT IP per K8s ServiceSpecifying the source IP Kubernetes Workloads using the K8s service

Tier0 LR

Corporate network

DB

allow – from: 134.247.100.10 (App)

to: 134.247.200.9 (DB)

Tier1 LR

Kubernetes Namespace: Foo

Web-FrontendPods

App Logic Pods

K8s Svc for AppK8s Svc for Web

Namespace LS(s)

SNAT App Svc Pods to:

134.247.100.10For all other Pods

use namespace SNAT IP


Kubernetes Metadata / NSX Logical Port Mapping

▶ kubectl get pod nsx-demo-rc-c7x65 -o yaml

apiVersion: v1

kind: Pod

metadata:

creationTimestamp: 2018-07-25T12:05:56Z

generateName: nsx-demo-rc-

labels:

app: nsx-demo

name: nsx-demo-rc-c7x65

namespace: nsx-ujo

Metadata within Kubernetes like Namespace, Pod names, Labels all get copied to the NSX Logical Port as Port Tags


NSX can be configured to collect ports and switches in dynamic security groups based on Tags (Kubernetes Metadata) and apply Firewall rules on them

Pre-Created Security Groups / Firewall rules (admin rules)

Match on Port Tags

Matching Pods are part of the Group

Groups are used in Firewall sections as src and dst


Support of Kubernetes Network Policy

---apiVersion: networking.k8s.io/v1kind: NetworkPolicymetadata:name: ps-fe-to-app

spec:podSelector:matchLabels:app: planespotter-app

policyTypes:- Ingressingress:- from:- podSelector:

matchLabels:app: planespotter-frontend

ports:- protocol: TCPport: 80

---apiVersion: networking.k8s.io/v1kind: NetworkPolicymetadata:name: ps-ing-to-fe

spec:podSelector:matchLabels:app: planespotter-frontend

policyTypes:- Ingressingress:- from:- ipBlock:

cidr: 100.64.0.0/16ports:- protocol: TCPport: 80


Policy Support – Security per Category

CONFIDENTIAL

Environment

Health-checksAdmin Rules

Application

Kubernetes Network PolicyDefault rule:1. Allow Cluster2. Allow Namespace3. None

NCP 2.5


Built-in support for Ingress (L7 HTTP/HTTPS) and Svc Type LB (L4 TCP/UDP) in the NSX-K8s integration. Most other K8s networking choice don't support Svc Type LB (L4), and you need an additional technology like NGINX from Ingress (L7).

Built-in Load Balancing

NCM Infra

K8s / OSAdapter


Libnetwork Adapter


More…

NSX Manager

API Client

NSX Manager

K8s master

etcd

API-Server

Scheduler

Virtual Server10.114.209.209HTTP and/or

HTTPS traffic

Server Pool 1

Server Pool 2Rule 2/bar/

Rule 1/foo/

LB Service

NCM Infra

K8s / OSAdapter


Libnetwork Adapter


More…

NSX Manager

API Client

NSX Manager

K8s master

etcd

API-Server

Scheduler

Virtual Server10.114.209.212TCP and/or

UDP traffic

Server Pool

LB Service


---apiVersion: v1kind: Servicemetadata:name: planespotter-frontendlabels:app: planespotter-frontend

spec:loadBalancerIP: 78.11.24.19type: LoadBalancerports:# the port that this service should serve on- port: 80

selector:app: planespotter-frontend

[root@master1 ~]# oc describe service/planespotter-frontendName: planespotter-frontendNamespace: defaultLabels: app=planespotter-frontendAnnotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"app":"planespotter-frontend"},"name":"planespotter-frontend","namespace":"d...

ncp/internal_ip_for_policy=100.64.64.1Selector: app=planespotter-frontendType: LoadBalancerIP: 172.30.253.255IP: 78.11.24.19LoadBalancer Ingress: 78.11.24.19, 100.64.64.1Port: <unset> 80/TCPTargetPort: 80/TCPNodePort: <unset> 32688/TCPEndpoints: 10.4.3.4:80,10.4.3.6:80Session Affinity: NoneExternal Traffic Policy: ClusterEvents: <none>[root@master1 ~]#

Persistent IP for Service of type LB

K8s / NSX Workflows


Containerize OVS and NCP Bootstrap initContainer

Installation Improvement

root@master1:~# kubectl get pods -n nsx-system -o wideNAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATESnsx-ncp-bcf5c8778-q67wg 1/1 Running 0 120m 10.114.209.215 node2 <none> <none>nsx-ncp-bootstrap-4xq5f 1/1 Running 0 131m 10.114.209.214 node1 <none> <none>nsx-ncp-bootstrap-grdqs 1/1 Running 0 131m 10.114.209.211 master1 <none> <none>nsx-ncp-bootstrap-pmhcx 1/1 Running 0 131m 10.114.209.213 master3 <none> <none>nsx-ncp-bootstrap-xkfgm 1/1 Running 0 131m 10.114.209.215 node2 <none> <none>nsx-ncp-bootstrap-zrnq5 1/1 Running 0 131m 10.114.209.212 master2 <none> <none>nsx-node-agent-4zfrj 3/3 Running 0 131m 10.114.209.212 master2 <none> <none>nsx-node-agent-7gr6t 3/3 Running 0 131m 10.114.209.214 node1 <none> <none>nsx-node-agent-g25v5 3/3 Running 0 131m 10.114.209.213 master3 <none> <none>nsx-node-agent-n2z4p 3/3 Running 0 131m 10.114.209.211 master1 <none> <none>nsx-node-agent-z5q87 3/3 Running 0 131m 10.114.209.215 node2 <none> <none>root@master1:~#

Bootstrap POD

initContainer installs:1. Installs/Upgrades the NSX-CNI.2.Loads the ncp-apparmor on Ubuntu.3. Installs/Upgrades/Downgrades the OVS kernel module if required.4. Stops OVS user space processes if running on the host machine.

NSX node agent POD

Containers:1. nsx-node-agent2. nsx-kube-proxy3. nsx-ovs


Troubleshooting OVS


Troubleshooting OVS

K8s worker VMESXi host


Hipster ShopCloud-Native Microservices Demo Application

https://github.com/GoogleCloudPlatform/microservices-demo

https://github.com/GoogleCloudPlatform/microservices-demo


Hipster Shop User Interface

Home Page Checkout Screen


Hipster Shop Architecture


Why NSX for Kubernetes?

Muti-tenancy Load Balancing and

services

Secure Containers, VMs and any other

endpoints with overarching Firewall

Policies

Provide visibility & troubleshooting tools to ease the

container adoption in the enterprise


Join the NSX VMUG Communityvmug.com/nsxConnect with your Peerscommunities.vmware.com

Embrace the NSX Mindsetnsxmindset.comFind NSX Resourcesvmware.com/products/nsx

Read the Network Virtualization Blogblogs.vmware.com/networkvirtualization

Where to Get Started

Attend the Networking and Security SessionsShowcases, breakouts, quick talks & group discussions

Visit the VMware BoothProduct overviews, use-case demos

Visit Technical Partner BoothsIntegration demos – Infrastructure, security, operations, visibility, and more

Meet the ExpertsJoin our experts in an intimate roundtable discussion

Free Hands-on LabsTest drive NSX with expert-led or self-paces hands-on labslabs.hol.vmware.com

VMware Education - Training and Certificationvmware.com/go/nsxtraining

Free NSX Training on Courseravmware.com/go/coursera

Engage and Learn Experience

Try Take

http://vmug.com/nsx

https://communities.vmware.com/community/vmtn/nsx

http://www.vmwarensxmindset.com/

http://vmware.com/products/nsx

http://blogs.vmware.com/networkvirtualization

http://labs.hol.vmware.com/

http://www.vmware.com/go/nsxtraining

http://www.vmware.com/go/coursera

nsx-t deep dive: kubernetes networking · nginx tcp/80 mgmt tcp/22 logging udp/514 ipc external ip...

Documents