LigatoA Platform for Development of Cloud-Native VNFs

The way Applications are developed & deployed… has changed…..

Microservices & Containers have changed many things…

• Microservices allow you to split an application into many modular pieces, the network is how you stitch the pieces back together.

• The interconnection of the pieces results in a more complex application network which consumes lots of resources

• The performance of the cloud native network is crucial to the behavior of the overall application.

• Applications are being developed and deployed very differently today.

Pod

Pod

PodPod

Pod

Pod

Pod

It’s crucial we get ”Container Networking” right! Lets not get “Openstacked”

Microservices & Containers have changed many things…

• Microservices allow you to split an application into many modular pieces, the network is how you stitch the pieces back together.

• The interconnection of the pieces results in a more complex application network which consumes lots of resources

• The performance of the cloud native network is crucial to the behavior of the overall application.

• Applications are being developed and deployed very differently today.

Pod

Pod

PodPod

Pod

Pod

Pod

It’s crucial we get ”Container Networking” right! Lets not get “Openstacked”

Traffic

Traffic

Traffic

Pod

Agent

VPP

Pod

Agent

VPP

Pod

Agent

VPP

Pod

Agent

VPP

Pod

Agent

VPP

Pod

Agent

VPP

Pod

Agent

VPP

Data Plane Microservices

Solution #1Move Cloud Native Networking out of the Kernel to Userspace

Container Networking moving from Kernel to Userspace• Userspace enables rapid upgradability, highly available (doesn't

bring down node), no system call overhead, no dependency on linux kernel networking community for features, higher performance and scale

• FD.io (dataplane), DPDK (network), SPDK (Storage) are examples

• Cloud Native apps are all connected by the network – lots of network end points to be managed, userspace offers lower overhead and higher performance

• Meltdown/Spectre bugs add a new tax for kernel networking

Solution #1Kubernetes & Contiv-VPP

Contiv-VPP• Kubernetes assumes seamless connectivity between pods, wherever it decides to

place them. A networking plugin is needed to abstract the network • Contiv is a networking plugin for Kubernetes that:

• Allocates IP addresses to Pods (IPAM)• Programs the underlying infrastructure it uses (Linux TCP/IP stack, OVS, VPP, …) to connect the Pods to

other Pods in the cluster and/or to the external world.• Implements K8s network policies that define which pods can talk to each other. • Implements K8s services; a service exposes one or more (physical) service instances implemented as

K8s pods to the other pods in the cluster and/or to external clients as a virtual instance (e.g. as a virtual “service” IP address).

• Contiv is a user-space based, high-performance, high-density networking plugin for Kubernetes - leveraging FD.io/VPP as the industry’s highest performance data plane

Kubelet

CNI

tapv2/veth

Contiv-VPP vswitch

Agent

PodPodPod

VPP …

K8s MasterK8s Master

IPv4/IPv6/SRv6 Network

App

Kernel Host stack

Legacy AppsK8s State Reflector

Contiv-VPPEtcd

Kubelet

CNI

tapv2/veth

Contiv-VPP vswitch

Agent

PodPodPod

VPP

App

Kernel Host stack

High PerformanceApps

PodPodPod

Envoy Sidecar App

VPP TCPStack

PodPodPod

High PerformanceApps

Envoy SidecarApp

VPP TCPStack

memif

Legacy Apps

PodPodPod

VNF

memif

Cloud-Native VNFs

PodPodPod

VNF

Cloud-Native VNFs

K8s policy & state distribution

Contiv-VPP Architecture• Can deliver complete container networking solution entirely

from userspace

• Replace all eth/kernel interfaces with memif/userspaceinterfaces.

• Apps can add VCL library for Higher Performance (bypass Kernel host stack and use VPP TCP stack)

• Legacy apps can still use the kernel host stack in the same architecture

BVI

Data Plane Network

Contiv-VPP RenderingVXLAN Overlay

Node 1

Node 2

Node 3

BD

BD

BDVXLAN Mesh Single VNI

BVI

App1Kernel App2

Cloud (Overlay)Network

BVI

App1Kernel App2

App1Kernel App2tap-v2

Creating BD with BVICreate BD

Create VXLAN Tunnel(one per rmt node)

Create BVI

• NFV-specific policy APIs (e.g. QoS, placement considering network resources)• Networking:

• HTTP or NAT-based load balancing isn‘t suitable for NFV use-cases• No support for high-speed wiring of NFs:

• To the outside world• To application containers• Between NFV containers• Creation of Service Function Chains (mixed physical and virtual – virtual a mix of VM and container)

• Management/Control: • Containerised NFs not really in the data plane (except for the vSwitch)• No support for cloud-native, high-performance NFs

• Forwarding: • Kernel used for forwarding – not sufficiently performance orientated (except for Contiv-VPP!)

Solution #2Cloud-Native VNFs

What Container-Networking Lacks for NFV Use-Cases:

Kubelet

CNICRI

tapv2/veth

Contiv-VPP vswitch

Agent

PodPodPodPodPod

Pod

VPP

IPv4/IPv6/SRv6 Network

High PerformanceApps

Istio Sidecar App

VPP TCPStack

App

PodPodPod

VNF

Kernel Host stack

memif

Legacy Apps Cloud-Native VNFs

DefineTopology

• Kubernetes does not provide a way to stitch micro-services together today

• Ligato allows you to wire the data plane together into a service topology

• Network functions can now become part of the service topology

Contiv-VPP EtcdContiv-VPP Etcd

K8s MasterK8s Master

Contiv-VPPNetmaster

LigatoController

DefineServices

DefineTopology

Solution #2Cloud-Native VNFs

Host

VPP Vswitch

CNFVPP

10.1.0.127

…

CNF1

VPP

…

CNF2

VPP

…

… Server

Vswitch VPP

CNFVPP

…

CNFVPP

…

CNF3

VPP

…

…

NF1NF1 NF2NF2 NF3NF3

Overlay Tunnel

Logical Representation

Physical Representation

Ingress NetworkIngress Classifier

Egress NetworkEgress Classifier

TopologyTopologyPlacement

(K8s)Placement

(K8s) Rendering

IngressRouterIngressRouter

EgressRouterEgressRouter

Overlay Tunnel Overlay Tunnel

Ingress Classifier Egress Classifier

Service Function Chaining with Cloud-Native VNFs

• In VM case have to copy via the kernel• With containers we use a shared memory interface (memif)

• Key is to chain between NFs on the same server• Containers are “cheap” so can have dedicated chain per tenant service

Accelerating NFV Using Containers

vSwitch

VNF1 VNF2

Virtual Machines

socket socket

socket

vSwitch

CNF1 CNF2

memif memif

memif

Containers

Intra-Server RenderingPoint to Point – 2 options based on policy

vSwitch

CNF1 CNF2

Direct East/West Memif

CNF1 CNF2

Memif via vSwitch

memif

memif

policy

Data Plane Network

VXLAN RenderingPoint to Point

vSwitch

CNF1

Node 1

vSwitch

CNF2

Node 2

memif

policy

memif

policyVXLAN Tunnel

Dedicated VNI

Creating VXLAN xConnectCreate memif

Create VXLAN Tunnel

Create xConnect

Data Plane Network 1

Data Plane Network 2

…

Physical Device

Physical Device

Physical Device

Physical Device…

Cloud tools & services

Cloud-Native VNF

Cloud-Native VNF

Cloud-Native VNF

Cloud-Native VNF

Cloud-Native VNF

Kubernetes

ContivLigatoCloud

Cloud-Native Application

Cloud-Native Application

Cloud-Native Application

Cloud-Native Application

Cloud (Overlay) Network

A VNF Cloud

Host

VPP vswitch

CN AppCN App

CN App

Host

VPP vswitch

CN-VNFCN-VNFCN-VNF

Host

VPP vswitch

CN AppCN App

CN AppHost

VPP vswitch

CN AppCN App

CN App

Host

VPP vswitch

CN-VNFCN-VNFCN-VNF

Host

VPP vswitch

CN-VNFCN-VNFCN-VNF Cloud

Network

KubernetesKubernetes

Cloud tools & services

Cloud-Native Control Plane

Cloud-Native Data PlaneVPP VPP

LIGATO Controller

CPECPE

CPE CPECPE

CPE CPE

…

…

A VNF Cloud: Data and Control Planes

Etcd

Ligato Agent

VPP

VPP-Agent

Etcd Prometheus Logging DB/ResyncHTTP gRPC Health REST

L2 L3 L4 ACL Interface IPSec Linux

Contiv-VPP

Service Policy Contiv-core Stats

GoVPP

KSRSFC Controller

Ligato and Kubernetes Control and Data Planes

Kubelet

CNI

tapv2/veth

Contiv-VPP vswitch

Agent

VPP …

K8s MasterK8s Master

IPv4/IPv6/SRv6 Network

Kernel Host stack

K8s State Reflector

Contiv-VPPEtcd

Kubelet

CNI

tapv2/veth

Contiv-VPP vswitch

Agent

VPP

Kernel Host stack

High PerformanceApps

PodPodPod

EnvoySidecar App

VPP TCPStack

PodPodPod

High PerformanceApps

Envoy SidecarApp

VPP TCPStack

memifmemifK8s state distribution

(policy, services, pods, nodes,

VNFs & K8s Networking

PodPodPod

VNF

Cloud-Native VNFs

PodPodPod

App

Legacy Apps

PodPodPod

App

Legacy Apps

PodPodPod

VNF

Cloud-Native VNFs

IPv4/IPv6/SRv6 Network

VXLAN, SRv6, GRE, MPLS, …

VXLAN, SRv6, GRE, MPLS, …

K8s VRFVNF VRFK8s VRF VNF VRF

K8s MasterK8s Master

KSR

Contiv-VPPEtcd

ONS Demo

CNI

tapv2

Contiv-VPP vswitch

Hoststack

memif

Pod

App

Pod

App

VNF Pod

AgentAgentVPPVPP

memif

VPP

VNF Pod

AgentAgentVPPVPP

Kubelet

Agent

CNI

tapv2

Contiv-VPP vswitch

Hoststack

memif

Pod

App

Pod

App

VNF Pod

AgentAgentVPPVPP

memif

VPP

VNF Pod

AgentAgentVPPVPP

Kubelet

Agent

SFC

NodesPods

SFC CRD

NodesNamespaces

PodsServicesPolicies

State distribution:• K8s• L3-L4 Service Mesh

management

K8s Vxlan Overlay

VNF1 Overlay

VNF2 Overlay

x-connect x-connect

Kubelet

CNICRI

tapv2/veth

Contiv-VPP vswitch

Agent

PodPodPodPodPod

Pod

VPP

Contiv-VPP EtcdContiv-VPP Etcd

K8s MasterK8s Master

IPv4/IPv6/SRv6 Network

Putting it All Together…

High PerformanceService Mesh Apps

Envoy Sidecar App

VPP TCPStack

App

PodPod

Kernel Host stackmemif

Legacy Apps Cloud-Native VNFs

Contiv-VPPNetmaster

Kubelet

CNICRI

tapv2/veth

Contiv-VPP vswitch

Agent

PodPodPod PodPod

Pod

VPP

High PerformanceService Mesh Apps

Envoy SidecarApp

VPP TCPStack

App

PodPod

Kernel Host stackmemif

Legacy AppsCloud-Native VNFs

IstioIstio

LigatoController

VNF Pod

AgentAgentVPPVPP

VNF Pod

AgentAgentVPPVPP

Ligato on Github

Cisco Confidential

CN-Infra

Core

PluginLifecycle

ManagementProbes

Status

Kafka

Etcd

Redis

Cassandra

Logrus

LogManager

DBRPC Messaging Logs Health

App Plugin

LifecycleSPI

App PluginApp Plugin

LifecycleSPI

API API API API API

Ligato CN-Infra: a CNF Development Platform www.github.com/ligato/cn-infra

CN-InfraPlugins

AppPlugins

Health Monitor(e.g. K8s)

Log Aggregator(e.g. Logstash)

KafkaKV Data Store(Etcd, Cassandra, Redis)

App clients

Cisco Confidential

AppPlugins

LinuxVETH

L2L3ACLNet Net Interfaces

GoVpp

CN-Infra

Core

PluginLifecycle

Management

Probes

Status

Kafka

Etcd

Redis

Cassandra

Logrus

LogManager

DBRPC Messaging Logs Health

App Plugin

LifecycleSPI

App PluginAgent App/Extension Plugin

LifecycleSPI

API API

API API API

Ligato VPP Agent: a CNF Management Agentwww.github.com/ligato/vpp-agent

API

API

LifecycleSPI

CN-InfraPlugins

VPP-AgentPlugins

Health Monitor(e.g. K8s)

Log Aggregator(e.g. Logstash)

KafkaKV Data Store(Etcd, Cassandra, Redis)

App clients

VPP

Cisco Confidential

LigatoController

LinuxVETH

L2L3ACLNet Net Interfaces

CN-Infra

Core

PluginLifecycle

Management

Probes

Status

Kafka

Etcd

Redis

Cassandra

Logrus

LogManager

DBRPC Messaging Logs Health

LifecycleSPI

Controller App

LifecycleSPI

API API

API API API

Ligato Controller: a CNF Deployment Platformwww.github.com/ligato/sfc-controller

API

API

LifecycleSPI

CN-InfraPlugins

VPP-AgentPlugins

Health Monitor(e.g. K8s)

Log Aggregator(e.g. Logstash)

KafkaKV Data Store(Etcd, Cassandra, Redis)

App clients

API

Cisco Confidential

Backup

Network Micro-Service Use Case:Service Function Chaining with Cloud-Native NFs

Network

ApplicationContainer

ApplicationContainer

Contiv Policy

Network

ApplicationContainer

ApplicationContainerCNF1 CNFn…

Network

ApplicationContainer

Policy

CNF2 CNFn…

Network Policy

CNF2 CNFn-1…

Network

ApplicationContainer

ApplicationContainer

Policy

CNF1 CNFn…

Same Server

Server Server

PolicyPolicy

PolicyPolicyPolicy

Policy