deploying kubernetes in the enterprise with cisco aci · container orchestration basic features

Deploying Kubernetes in the Enterprise with Cisco ACI

Camillo Rossi – TME INSBU

BRKACI-2505

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public

Cisco Spark

Questions? Use Cisco Spark to communicate with the speaker after the session

1. Find this session in the Cisco Live Mobile App

2. Click “Join the Discussion”

3. Install Spark or go directly to the space

4. Enter messages/questions in the space

How

cs.co/ciscolivebot#BRKACI-2505


Session Objectives

At the end of the session, the participants should be able to:

• Have a general understanding of containers

• Have a general understanding of Kubernetes

• Understand how ACI and Kubernetes integration is deployed

Initial assumption:

• The audience already has a good knowledge of ACI main concepts (Tenant, BD, EPG, L2Out, L3Out, etc.)

4BRKACI-2505

• ACI-Kubernetes value proposition

• Introduction to Containers

• Container Management and Orchestration

• ACI and Kubernetes Solution Overview

• Demos

• Q&A

Agenda


ACI Kubernetes Integration – Value proposition

• Allow containers direct access to the ACI policy model, so that they can participate as first-class citizens within an ACI fabric

• Allow seamless integration of containers, VMs, and physical devices on an ACI fabric

• Support native policy semantics, so that a container application that is specified using Kubernetes NetworkPolicy will work correctly out of the box

• I.E. The same config works on Google Cloud, AWS and ACI

• Leverage fabric resources and Opflex to assist accelerating Kubernetes service load balancing

6BRKACI-2505


ACI Kubernetes Integration – Value proposition (cont.)

• Security Policies within and between pods are defined using Kubernetes Network Security API. Implemented on OVS via Host Protection Profiles, visible by APIC

• ACI admin can (optionally) define EPGs and contracts that are exposed to Kubernetes users

• EPG is selected based on annotation, which can also be used to dynamically define new EPGs

7BRKACI-2505





• Demos

• Q&A

Agenda

Linux fundamentalsUnderstanding Containers Origins


Lightweight Process Virtualization is not new

• Create multiple views of the root filesystem to isolate applications and processes

• Lightweight Process Virtualization is not a new concept:

• Solaris Zones

• BSD jails

• Linux chroot

• AIX WPARs (Workload Partitions)

10BRKACI-2505


What is new?

• Linux Kernel 3.8 (February 2013) added support for namespaces and cgroups

• Namespaces are like chroot for processes, but also applied to network, UTS (Unix Timesharing), mount, IPC and users (UIDs).

• Example: Network namespaces enable the creation of multiple, isolated routing tables that operate independently. Multiple hostnames could also be used: • #ip netns add myns1 -> Create a new Namespace “myns1”

BRKACI-2505 11


What is new?

• Cgroups provide resource management capabilities

• Processes can be grouped into user-defined group of tasks, for optimized system resource usage

• Cgroups move resource allocation from the process level to the application level by grouping and labeling processes into hierarchies

• Resource allocation includes CPU time, block IO, RAM and network bandwidth

BRKACI-2505 14


What is new?

• Cgroups architecture:

CPU Network Memory Storage I/O

Cgroup1

Cgroup2

BRKACI-2505 15

What are containers?


What is a container?

• A container is a binary executable, packaged with dependencies and intended for execution in a private namespace with optional resource constraints.

• This provides the containers multiple isolated operating system environments with their own file system, network, process and block I/O space on the same host

BRKACI-2505 17


Compute Virtualization != Containers

Host OS

Hypervisor

App App App

Server Hardware

Host OS

Container Engine (Linux)

Guest

OS

Guest

OS

Guest

OS

VMs

Containers

Bins/

Libs

Bins/

Libs

Bins/

Libs

Server Hardware

App App App

Bins/

Libs

Bins/

Libs

Bins/

Libs

Virtualization Containers

BRKACI-2505 18

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 19BRKACI-2505

Compute Virtualization & Containers

• They provide a way to abstract resources

• They define logical boundaries to the resources they consume

• They enable multiple OS instances to run on the same host

• They share the resources of the host

Similarities


Compute Virtualization & Containers

• Containers can only run the same OS as the host

• Containers share the same kernel as the host

• Containers are faster to provision and boot

• Containers have lower overhead as there is no need for the hypervisor layer

Differences


Containers Current Challenges

• Containers images management

• Orchestration of containers across multiple hosts

• Lack of standards

• Integration with virtualization and cloud tools

• Networking management

21BRKACI-2505

Addressed by ACI

Why Containers?


Microservice

Single Purpose

Stateless

Independently Scalable

Automated

Service

Autonomous

Loosely-coupled

Function

Single Action

Event Sourced

Ephemeral

Application Architectural Evolution

23BRKACI-2505

f()


An effective platform for micro-services

• Containers are ideal candidates to run micro-services:

• Micro-services define stateless, loosely coupled application components communicating over API’s, running in different runtime environments.

• Containers meet new application requirements as they provide:• Density

• Speed

• Portability

• Low overhead management

24BRKACI-2505

Containers runtimes


Containers runtimes

• A container runtime enables users to make effective use of containerization mechanisms by providing APIs and tooling that abstract the low level technical details

• LXC - Open Source Solutions (OSS)

• Docker – OSS and commercial

• Rkt – part of CoreOS, OSS and commercial

• VMware Integrated Container (aka Project Bonneville) - Proprietary

• RunC - OSS

• Garden – part of Pivotal Cloud Foundry, OSS and commercial

BRKACI-2505

Docker


Docker provides an integrated technology suite that enables development and IT operations teams to build, ship, and run distributed applications anywhere.

30BRKACI-2505


A Docker containers wrap a piece of software in a complete filesystem that contains everything needed to run: code, runtime, system tools, system libraries –anything that can be installed on a server.

This guarantees that the software will always run the same, regardless of its environment.

31BRKACI-2505


Docker consists of two main components:

33BRKACI-2505

Docker Hub – SaaS component for managing

and sharing containers.

Docker Engine – the

actual app running

on the host.

Containers


Dockerfile and Registry

• A Dockerfile is simply a text file containing instructions on how to build a Docker image

• It can add components on top of an existing image

• Images are available online on the Docker hub repository

• Local, private registry can be created.

• A registry is an instance of the registry container image

34BRKACI-2505


Dockerfile and Registry

• A Dockerfile is simply a text file containing instructions on how to build a Docker image

• It can add components on top of an existing image

• Images are available online on the Docker hub repository

• Local, private registry can be created.

• A registry is an instance of the registry container image

35BRKACI-2505

More about Docker Images…


Docker Images

• A Docker image is made up of filesystems layered over each other.

37BRKACI-2505


Docker Images

• The storage driver is responsible for presenting these layers as a single, unified file system.

38BRKACI-2505


Docker Images

• When you start a container, Docker creates an empty, read-write layer on top of the stack – all changes are made in this layer.

39BRKACI-2505


Docker Images

• Docker uses “copy-on-write” container layers.

• If a file needs to be modified, it is copied into the read-write layer first.

40BRKACI-2505


• This means that multiple containers can share a single copy of the image.

41BRKACI-2505

Docker Networking


Option 1 - None

• Doesn’t create any network interface for the container

43BRKACI-2505


Option 2 - Bridge Mode

• Default mode where Docker attaches containers to Docker0 bridge

• Containers in the same host can talk to each other

• Containers on different host can’t talk to each other (or anything else) easily

44BRKACI-2505

Host-2

Container 3

docker0 Bridge

172.17.42.1

Eth0172.17.0.12

veth994786d

Container 4

Eth0172.17.0.13

vethab4e22e

Host-1

Container 1

docker0 Bridge

172.17.42.1

Eth0172.17.0.12

veth774786d

Container 2

Eth0172.17.0.13

vethde4e22e

Same IPs for the containers is not a typo


Option 2 - Bridge Mode

• It’s possible to expose a container externally by publishing a port (NAT)

45BRKACI-2505

Host

Container 1

NGINX

docker0 Bridge


Option 3 - Host Mode

• Connect containers to Host network stack

• All the network interfaces defined on the host will be available to the container (every container will have the same IP address as the host)

46BRKACI-2505

Host-1

Container 1

Eth0 (host interface)

192.168.0.2

Eth0192.168.0.2

Container 2

Eth0192.168.0.2


Option 4 - Mapped Container Mode

• Container is mapped to another container network stack

• Filesystem, processes and other resources are kept separate

• They share network resources (IP, interfaces)

47BRKACI-2505

Host-1

Container 1

docker0 Bridge

172.17.42.1

Eth0172.17.0.12

veth774786d

Container 2

Eth0172.17.0.12


All these Docker Networking options are complex

• Iptables rules must be created manually to allow/NAT traffic to containers ports…

• Containers on different hosts can’t communicated to each other even if they are in the same L2 domain

• Needs to manually manage port-mappings

• Prone to errors

48BRKACI-2505


Docker network driver plugins

• Network plugins can be used to extended Docker networking support to a wide range of networking technologies, such as VXLAN, IPVLAN, MACVLAN or something completely different.

49BRKACI-2505





• Demos

• Q&A

Agenda


In a multi-host environment, containers need to

• Have network reachability

• Be fault-tolerant

• Easily scalable

• Use resources optimally

• Can discover other containers/application automatically

• Communicate with each other

• Can be update/rollback without any downtime

• Expose services in an easy and reliable way

53BRKACI-2505


Container Orchestration basic features

• Bring multiple hosts together and make them part of a cluster

• Schedule containers to run on different hosts

• Help containers running on one host reach out to containers running on other hosts in the cluster

• Bind containers and storage

• Bind containers of similar type to a higher-level construct, like services, so we don't have to deal with individual containers

• Keep resource usage in-check, and optimize it when necessary

• Allow secure access to applications running inside containers.

BRKACI-2505 54


Containers Orchestrators

• Docker Swarm is a Container Orchestrator provided by Docker, Inc. It is part of Docker Engine.

• Kubernetes started by Google, now part of the Cloud Native Computing Foundation project.

• Mesos Marathon is one of the frameworks to run containers at scale on Apache Mesos.

• Amazon EC2 Container Service (ECS) is a hosted service provided by AWS to run Docker containers at scale on its infrastructure.

• Hashicorp Nomad is the Container Orchestrator provided by HashiCorp.

BRKACI-2505 55

https://docs.docker.com/engine/swarm/

https://www.docker.com/

https://docs.docker.com/engine/

https://kubernetes.io/

https://www.cncf.io/

https://github.com/mesosphere/marathon

http://mesos.apache.org/

https://aws.amazon.com/ecs/

https://www.nomadproject.io/

https://www.hashicorp.com/


Kubernetes

• Kubernetes is an open source Container Orchestration system for automating deployment, scaling and management of containerized applications.

• It was inspired by the Google Borg System and with its v1.0 release in July 2015, Google donated it to the Cloud Native Computing Foundation (CNCF).

• Generally, Kubernetes has new releases every three months. The current stable version is 1.9 (as of Jan 2018).

BRKACI-2505 56

https://www.cncf.io/


Kubernetes & Docker

• Kubernetes uses Docker to execute/run the containers

• Kubernetes adds, on top of Docker, all the intelligence and features of an orchestrator

BRKACI-2505 57


Kubernetes Features

• Automatic binpackingKubernetes automatically schedules the containers based on resource usage and constraints, without sacrificing availability.

• Self-healingKubernetes automatically replaces and reschedules the containers from failed nodes. It also kills and restarts containers which do not respond to health checks, based on existing rules/policy.

• Horizontal scalingKubernetes can automatically scale applications based on resource usage like CPU and memory. It also supports dynamic scaling based on customer metrics

BRKACI-2505 58


Kubernetes Features (cont.)

• Service discovery and Load balancingKubernetes groups sets of containers and refers to them via a DNS name. This DNS name is also called a Kubernetes service. Kubernetes can discover these services automatically, and load-balance requests between containers of a given service.

• Automated rollouts and rollbacksKubernetes can roll out and roll back new versions/configurations of an application, without introducing any downtime.

• Secrets and configuration managementKubernetes can manage secrets and configuration details for an application without re-building the respective images. With secrets, we can share confidential information to our application without exposing it to the stack configuration, like on GitHub.

BRKACI-2505 59


Kubernetes Features (cont.)

• Storage orchestrationWith Kubernetes and its plugins, we can automatically mount local, external, and storage solutions to the containers in a seamless manner, based on Software Defined Storage (SDS).

• Batch executionBesides long running jobs, Kubernetes also supports batch execution.

BRKACI-2505 60


Kubernetes Architecture

• At a very high level, Kubernetes has the following main components:

• One or more Master Nodes

• One or more Worker Nodes

• Distributed key-value store, like etcd.

BRKACI-2505 61


Kubernetes Components – Master Node

• The Master Node is responsible for managing the Kubernetes cluster. Master node access methods are CLI, GUI or APIs.

• For fault tolerance, there can be more than one Master Node.

• To manage the cluster state, Kubernetes uses etcd, and all Master Nodes connect to it. etcd is a distributed key-value store. The key-value store can be part of the Master Node. It can also be configured externally, in which case, the Master Nodes connect to it.

BRKACI-2505 63


Kubernetes Components – Worker Node

• A Worker Node is a machine (VM, physical server, etc.) which runs the containers using pods and is controlled by the Master Node.

• pods are scheduled on the Worker Nodes

BRKACI-2505 64


Kubernetes - pod

• A pod is the scheduling unit in Kubernetes. It is a logical collection of one or more containers which are always scheduled together.

• The set of containers composed together in a pod share an IP.

[root@k8s-01-p1 ~]# kubectl get pod --namespace=kube-system

NAME READY STATUS RESTARTS AGE

aci-containers-controller-1201600828-qsw5g 1/1 Running 1 69d

aci-containers-host-lt9kl 3/3 Running 0 72d

aci-containers-host-xnwkr 3/3 Running 0 58d

aci-containers-openvswitch-0rjbw 1/1 Running 0 58d

aci-containers-openvswitch-7j1h5 1/1 Running 0 72d

BRKACI-2505 65


Kubernetes – Deployment

• Deployments are a collection of pods providing the same service

• You describe the desired state in a Deployment object, and the Deployment controller will change the actual state to the desired state at a controlled rate for you

• For example you can create a deployment that declare you need to have 2 copies of your front-end pod.

[root@k8s-01-p1 ~]# kubectl get deployment --namespace=kube-system

NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE

aci-containers-controller 1 1 1 1 72d

BRKACI-2505 66


Kubernetes – Services

• A service tells the rest of the Kubernetes environment (including other pods and Deployments) what services your application provides.

• While pods come and go, the service IP addresses and ports remain the same.

• Kubernetes automatically load balance the load across the replicas in the deployment that you expose through a Service

• Other applications can find your service through Kurbernetes service discovery.

• Every time a service is create a DNS entry is added to kube-dns

[root@k8s-01-p1 ~]# kubectl get svc --namespace=kube-system

NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE

kube-dns 11.96.0.10 <none> 53/UDP,53/TCP 72d

BRKACI-2505 67


Kubernetes – External Services

• If there are external IPs that route to one or more cluster nodes, Kubernetes services can be exposed on those external IPs.

• Traffic that ingresses into the cluster with the external IP (as destination IP), on the service port, will be routed to one of the service endpoints.

• External IPs are not managed by Kubernetes and are the responsibility of the cluster administrator.

[root@k8s-01-p1 ~]# kubectl get svc front-end --namespace=guest-book


front-end 11.96.0.33 11.3.0.2 80:30002/TCP 3m

BRKACI-2505 68


Kubernetes – Ingress

• An Ingress is a collection of rules that allow inbound connections to reach the cluster services.

• It can be configured to give services externally-reachable URLs, load balance traffic, terminate SSL, offer name based virtual hosting, and more

• Think of NGINX

[root@k8s-01-p1 ~]# kubectl get ingress

NAME HOSTS ADDRESS PORTS AGE

test-ingress * 80 7s

BRKACI-2505 69


Kubernetes - Labels

• Kubernetes uses labels as “nametags” to identify things.

• Can be used to indicate roles, stability, or other important attributes.

• You can query anything in Kubernetes via a label.

• i.e. Return all the pod that are running “PreProduction” workload

[root@k8s-01-p1 ~]# kubectl get pod --namespace=kube-system -l component=kube-apiserver

NAME READY STATUS RESTARTS AGE

kube-apiserver-k8s-01-p1 1/1 Running 0 72d

BRKACI-2505 70


Kubernetes - Annotations

• Similar to labels but are NOT used to identify and select object

• Used in ACI, yes soon we will be speaking about ACI and Kubernetes

[root@k8s-01-p1 ~]# kubectl describe node k8s-01-p1 | more

Name: k8s-01-p1

Role:

Labels: beta.kubernetes.io/arch=amd64

beta.kubernetes.io/os=linux

kubernetes.io/hostname=k8s-01-p1

node-role.kubernetes.io/master=

Annotations: node.alpha.kubernetes.io/ttl=0

opflex.cisco.com/pod-network-ranges={"V4":[{"start":"11.2.0.130","end":"11.2.1.1"}]}

opflex.cisco.com/service-endpoint={"mac":"66:85:9a:e9:ef:2f","ipv4":"11.5.0.3"}

volumes.kubernetes.io/controller-managed-attach-detach=true

BRKACI-2505 71


Kubernetes – Namespace

• Groups everything together:

• Pod

• Deployment

• Volumes

• Services

• Etc…

BRKACI-2505 72


All Together: A K8S Cluster

Container

Application

pod1

Deployment1

Container

Application

pod2

Container

Application

pod[n]

Node1

Node[N]

Node2Service

1.1.1.1:80

Namespace

A node can be part of

Several Namespaces

BRKACI-2505 73





• Demos

• Q&A

Agenda


Application Centric InfrastructureAny Application – Any hypervisor

• Policy consistency provides for

containers running reliably and

securely

• Build to and support open systems

and standards

• Common pervasive gateway and

policy based routing provide optimal

network connectivity

• Ease of deploying, scaling and

managing

BRKACI-2505 75


Cisco ACI and Container Integration

Node

OpFlex OVS

ACI and Containers

Unified networking: Containers, VMs, and

bare-metal

Micro-services load balancing integrated in

fabric for HA / performance

Secure multi-tenancy and seamless

integration of Kubernetes network policies

and ACI policies

Visibility: Live statistics in APIC per

container and health metricsNode

OpFlex OVS

BRKACI-2505 76


`

1

2 Deploy/Scale Clusters

Build containers

WEB APP WEB APP DB

Server 1 Server 2

Opflex/OVS

Host level Policy

Enforcement

ACI Fabric

Define BDs, Context and

AP

Fabric Bring Up

2

1

Get VLAN Pools Allocated

For Each EPG3

Full Infrastructure Visibility,

Telemetry4

Annotate policy3EPG

Developer Network Administrator

Infosec

Define Container

Network Policy

Infrastructure Policy

Enforcement

1

77BRKACI-2505

ACI Network Plugin for KubernetesNative Security Policy Support


ACI VMM Domain for Kubernetes

78BRKACI-2505

Node

OpFlex OVS

Kubernetes

ACI Policies

Technical DescriptionNetwork Policy

Node

OpFlex OVS

• Network policies of Kubernetes supported using standard

upstream format but enforced through OpFlex / OVS using

APIC Host Protection Profiles

• Kubernetes app configurations can be moved without

modification to/from ACI and non-ACI environments

• Embedded fabric and virtual switch load balancing

• PBR in fabric for external service load balancing

• OVS used for internal service load balancing

• VMM Domain for Kubernetes

• Stats per namespace, deployment, service, pod

• Physical to container correlation


ACI CNI Plugin Components

• aci-containers-controller

• Handle IPAM

• Management of endpoint state

• Policy Mapping (annotations)

• Controls Load Balancing

• Pushes configurations into the APIC

79BRKACI-2505



• aci-containers-host is a DaemonSet composed of 3 containers:

• mcast-daemon: • Handles Broadcast, unknown unicast and multicast replication

• aci-containers-host:• Endpoint metadata

• Pod IP Address management

• Container Interface Configuration

• opflex-agent:• Support for Stateful Security Groups

• Manage configuration of OVS

• Render policy to openflow rules to program OVS.

• Handles loadbalanced services (connection tracking, natting, etc…)

80BRKACI-2505



• aci-containers-openvswitch

• Bridge traffic from containers to physical interfaces

• Enforce policies

81BRKACI-2505


Flexible Definition of EPG Boundaries

Cluster Isolation Namespace Isolation Deployment Isolation

pod

pod

pod

pod

pod

pod

• Single EPG for entire cluster,

default behavior

• No need for any internal contracts

• Each namespace is mapped to its

own EPG

• Contracts for inter-namespace traffic

• Each deployment mapped to an EPG

• Contracts tightly control service traffic

pod

pod

pod

pod

pod

pod

pod

pod

pod

pod

pod

pod

pod

pod

pod

pod

pod

pod

Contra

ct

pod

pod

pod

pod

pod

pod

pod

pod

pod

pod

pod

pod

Contra

ct

Contract

Contract

Contra

ct

EPG

Network

Policy

ACI Basic Configuration


acc-provision

• ACI Container Controller Provision:

• Takes a YAML file containing the parameters of your configuration

• Generates and pushes most of the ACI config

• Generates Kubernetes ACI CNI containers configuration

acc-provision --flavor=kubernetes-1.7 -a -u admin -p pass –c config.yml –o cni_config.yml

84BRKACI-2505

Used to select if we are deploying

kubernetes 1.6, 1.7 or OpenShift 3.6

APIC user and

passwordConfiguration file

Output file for ACI CNI

config


acc-provision – configuration file (1)

aci_config:

system_id: KubeSpray # Tenant Name and Controller Domain Name

apic_hosts: # List of APIC hosts to connect for APIC API

- 10.67.185.102

vmm_domain: # Kubernetes VMM domain configuration

encap_type: vxlan # Encap mode: vxlan or vlan

mcast_range: # mcast range for BUM replication

start: 225.22.1.1

end: 225.22.255.255

mcast_fabric: 225.1.2.4

nested_inside: # (OPTIONAL) If running k8s node as VMs specify the VMM Type and Name.

type: vmware # Only vmware for now, ports groups created automatically with system_id name

name: ACI

# The following resources must already exist on the APIC,

# they are used, but not created by the provisioning tool.

aep: ACI_AttEntityP # The AEP for ports/VPCs used by this cluster

vrf: # The VRF can be placed in the same Tenant or in Common.

name: vrf1

tenant: KubeSpray # This can be the system-id or common

l3out:

name: l3out # Used to provision external IPs

external_networks:

- default_extepg # Default Ext EPG, used for PBR redirection

85BRKACI-2505


acc-provision – configuration file (2)

#

# Networks used by Kubernetes

#

net_config:

node_subnet: 10.32.0.1/16 # Subnet to use for nodes

pod_subnet: 10.33.0.1/16 # Subnet to use for Kubernetes Pods

extern_dynamic: 10.34.0.1/24 # Subnet to use for dynamic external IPs

extern_static: 10.35.0.1/24 # Subnet to use for static external IPs

node_svc_subnet: 10.36.0.1/24 # Subnet to use for service graph

kubeapi_vlan: 4011 # The VLAN used by for nodes to node API communications

service_vlan: 4013 # The VLAN used by LoadBalancer services

infra_vlan: 3456 # The ACI infra VLAN used to establish the OpFlex tunnel with the leaf

#

# Configuration for container registry

# Update if a custom container registry has been setup

#

registry:

image_prefix: noiro # DO NOT CHANGE

86BRKACI-2505


ACI Fabric Configuration - EPGs

• kube-nodes: node API communication• Physical Domain

• kube-system: k8s components running as pods• Container Domain*

• kube-default: All user created pods will be placed in this EPG by default• Container Domain*

* Equivalent of VMM Domain for Containers


ACI Fabric Configuration – BDs and VRF

• kube-nodes-bd:• Only used for kube-node EPG

• Maps to node_subnet

• kube-pod-bd: • Any pod will be assigned an IP from

this BD Subnet

• Used for kube-default, kube-system and any other user defined EPGs.

• Maps to pod_subnet

• KubeSpray…-service:• BD for PBR/SG services

• Created when ACI CNI plugin is deployed


ACI Fabric Configuration – L3OUT

• User must create and configure the L3OUT

• L3OUT and Default Networks names must match the acc-provision config


ACI Fabric Configuration – Contracts

• Minimum sets configured to ensure basic cluster functionality

• User can define additional contracts as usual


ACI Fabric Configuration – L4-L7 Devices

• Created once the ACI CNI plugin is deployed

• Dynamically updated if nodes are added or removed from the k8s cluster

• Service Graph Template: Specify a template for PBR redirection


ACI Fabric Configuration – Container Domain

APIC keeps inventory of pods

and their metadata (labels,

annotations), deployments,

replicasets, etc.

View pods per node, map to

encapsulation, physical point

in the fabric.

Fabric admin can search

APIC for k8s nodes, masters,

pods, services …

BRKACI-2505 92

Demo 1 Deploying an Application


Demo 1 – Spin up a guestbook application

• The guestbook application uses Redis to store its data. It writes its data to a Redis master instance and reads data from multiple Redis slave instances.

• The code can be found at: https://kubernetes.io/docs/tutorials/stateless-application/guestbook/

94BRKACI-2505

For your

reference

https://kubernetes.io/docs/tutorials/stateless-application/guestbook/


Demo 1 – Spin up a guestbook application

• Create a namespace for our application

cisco@k8s-01:~/demo/guestbook1$ kubectl create namespace guestbook

• Deploy all the componentscisco@k8s-01:~/demo/guestbook1$ kubectl --namespace=guestbook apply -f complete.yaml

deployment "frontend" created

service "frontend" created

deployment "redis-master" created

service "redis-master" created

deployment "redis-slave" created

service "redis-slave" created

• Check POD status cisco@k8s-01:~/demo/guestbook1$ kubectl --namespace=guestbook get pod -o wide

READY STATUS RESTARTS AGE IP NODE

frontend-1768566195-mj43h 1/1 Running 0 2m 10.33.1.11 k8s-02

frontend-1768566195-tpw75 1/1 Running 0 2m 10.33.0.153 k8s-03

frontend-1768566195-vljrh 1/1 Running 0 2m 10.33.0.155 k8s-03

redis-master-2365125485-8hg60 1/1 Running 0 2m 10.33.0.152 k8s-03

redis-slave-3837281623-p4fs7 1/1 Running 0 2m 10.33.1.12 k8s-02

redis-slave-3837281623-qw894 1/1 Running 0 2m 10.33.0.154 k8s-03

Note: All the commands are executed from the Kubernetes master node95BRKACI-2505

For your

reference


Demo 1 – Check APIC Controller Domain

96BRKACI-2505

APIC has complete visibility into

k8s objects

Visibility on where this specific

pod is running

For your

reference


Demo 1 – Check APIC EPG

97BRKACI-2505

POD Name

For your

reference


Demo 1 – Cluster Services

• By default every POD is exposed only to the k8s cluster via a Service IP. You can imagine this as a Virtual IP of a load balancer.

• With the ACI CNI plugin the LoadBalancing for the internal cluster services is performed by OVS cisco@k8s-01:~/demo/guestbook1$ kubectl --namespace=guestbook get service


frontend 10.37.0.124 <nodes> 80:32677/TCP 25m

redis-master 10.37.0.162 <none> 6379/TCP 25m

redis-slave 10.37.0.136 <none> 6379/TCP 25m

• Try to access the service from one of the nodescisco@k8s-01:~/demo/guestbook1$ curl 10.37.0.124

<html ng-app="redis">

<head>

<title>Guestbook</title>

!SNIP!

98BRKACI-2505

For your

reference

Demo 2 Placing PODs/Namespaces in EPGs


Demo 2 - APIC Steps

• Create an EPG under your application

• BD = your pod BD

• VMM Domain = Your Kubernetes Domain

• Every POD in an EPG needs to be able to communicate with:

• kube-system for cluster wide DNS resolution

• kube-node for health monitoring probes

• Top Tip: Use EPG contract masters and inherit contracts from kube-default!

100BRKACI-2505

For your

reference


Demo 2 - acikubectl

• Utility to manage and troubleshoot the k8s cluster

• Can be used to annotate Namespeces or Deployments with the Tenant/App/EPG names

cisco@k8s-01:~/demo/guestbook1$ acikubectl set default-eg namespace guestbook -t KubeSpray -a kubernetes -g

guestbook

Setting default endpoint group:

Endpoint Group:

Tenant: KubeSpray

App profile: kubernetes

Endpoint group: guestbook

BRKACI-2505

For your

reference


Check under your EPG

• All your PODs should now have moved from kube-default to guestbook

102BRKACI-2505

For your

reference

Exposing Services


The extern_dynamic subnet

• Defined in acc-provision configuration file

• An IP address will be automatically selected from this subnet to expose your service outside of the k8s cluster/fabric

• Expose the service as “LoadBalancer” (as per kubernetes standard)

• The extern_dynamic subnet is not associated to a BD: You need to configure your external router with static routes toward your L3OUT for this subnet

104BRKACI-2505

cisco@k8s-01:~/demo/guestbook1$ kubectl --namespace=guestbook get svc frontend


frontend 10.37.0.124 10.34.0.5 80:32677/TCP 5h

extern_dynamic


Service Graphs and PBR

105BRKACI-2505

* defined in the acc-provision config file

Every time a service is exposed the ACI CNI controller will deploy:

• An External EPG with a /32 match for the Service IP

• A new contract between the svc_ExtEPG and the default_ExtEPG*

• A Service Graph with PBR redirection containing every node where an exposed POD is running

L3Out

default_ExtEpg

0.0.0.0/0

Svc_x_ExtEPG

10.34.0.5/32

Contract PBR Service Graph

RTRClient

Cons

Prov

Node1

Node2

NodeN

Pod1

Pod3

Pod5

Pod2

Pod4

NodeN

OVS

OVS

OVS


Service Graphs and PBR – Packet walk

106BRKACI-2505

L3Out

default_extEpg

0.0.0.0/0

Svc_x_ExtEPG

10.34.0.5/32


RTRClient

Cons

Prov

1. Client send a request to 10.34.0.2, ACI performs Longest Prefix Match (LPM) on the SIP and classify the traffic in the default_extEPG

Node1

Node2

NodeN

Pod1

Pod3

Pod5

Pod2

Pod4

NodeN

OVS

OVS

OVS



107BRKACI-2505

L3Out

default_extEpg

0.0.0.0/0

Svc_x_ExtEPG

10.34.0.5/32


Node1

Node2

NodeN

RTRClient

Cons

Prov


2. ACI does a routing lookup for 10.34.0.5, IP does not exist in the fabric, we should route it out however

LPM places it in the Svc_x_ExtEPG

3. PBR redirection is triggered and the traffic is LoadBalanced by the fabric to one of the nodes

SIP DIP

192.168.1.100 10.34.0.5

Pod1

Pod3

Pod5

Pod2

Pod4

NodeN

OVS

OVS

OVS



108BRKACI-2505

L3Out

default_extEpg

0.0.0.0/0

Svc_x_ExtEPG

10.34.0.5/32


Node1

Node2

NodeN

RTRClient

Cons

Prov


2. ACI does a routing lookup for 10.34.0.2, IP does not exist in the fabric, we should route it out however

LPM places it in the Svc_x_ExtEPG

3. PBR redirection is triggered and the traffic is LoadBalanced by the fabric to one of the nodes

Pod1

Pod3

Pod5

Pod2

Pod4

NodeN

OVS

OVS

OVS

SIP DIP

192.168.1.100 10.34.0.5



109BRKACI-2505

L3Out

default_extEpg

0.0.0.0/0

Svc_x_ExtEPG

10.34.0.5/32


RTRClient

Cons

Prov

4. The K8S node is not expecting any traffic directed to the external service IP so OVS will perform NAT as required

5. If there are multiple POD on a single node OVS will perform a second stage LB to distribute the load between Pods running on the same node

Node1

Node2

NodeN

Pod1

Pod3

Pod5

Pod2

Pod4

NodeN

OVS

OVS

OVS

SIP DIP

192.168.1.100 PodX IP

DIP

192.168.1.100 10.34.0.5



110BRKACI-2505

L3Out

default_extEpg

0.0.0.0/0

Svc_x_ExtEPG

10.34.0.5/32


RTRClient

Cons

Prov

4. PodX replies to the client

5. OVS restore the original external Service IP

6. PBR redirection is not triggered since the source EPG is the Shadow EPG of the PBR node

7. Traffic is routed back to the client (and is permitted by the contract)

DIP SIP

192.168.1.100 10.34.0.5Node1

Node2

NodeN

Pod1

Pod3

Pod5

Pod2

Pod4

NodeN

OVS

OVS

OVS

DIP SIP

192.168.1.100 PodX IP

Demo 3 Exposing Services


Exposing a service

• Simply choose the LoadBalancer ”type” in the service definition

• The ACI CNI plug in will:

• Automatically pick a free IP from the extern_dynamic subnet

• Create the ExtEPG

• Create contracts

• Create PBR redirection rules

• Deploy the service graph

112BRKACI-2505


Scalability

• Currently the scalability of exposing external service with PBR is limited by the number of external EPGs per L3OUT.

• ACI 3.1 supports 250 external EPGs per L3 OUT per leaf *

• This is a soft limit and will increase with time

• But we want more! So?

113BRKACI-2505

*For details check:

https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/3-x/verified_scalabilty/b_Verified_Scalability_3_1_1x_and_13_1_1x.html

Scaling External Services with Ingress



• Composed of two parts:

• Ingress Resources: collection of rules that defines how inbound connections can reach the internal cluster services.

• Ingress controller: responsible for fulfilling the Ingress, usually with a virtual loadbalancer (nginx, ha-proxy)

• Ingress controller can be shared between multiple namespaces

• It can be configured to give services externally-reachable URLs, load balance traffic, terminate SSL, offer name based virtual hosting etc…

• Bonus Feature: Easy integration with DNS: configure a wildcard DNS record (*.cisco.com) pointing to the IP of the ingress controller

115BRKACI-2505


Ingress Cont 1

Ingress Cont 2

Ingress Cont N

Namespace

ingress


116BRKACI-2505

Pod1

Pod2

Namespace

ciscolive

Namespace

cisco

Service

Pod1

Pod2

Service

Ingress

Controller

Ingress Resource

(I am www.cisco.com)

Ingress Resource

(I am www.ciscolive.com)

Clientwww.cisco.com

www.ciscolive.com


ACI and Kubernetes Ingress

• Expose the Ingress Controller via Service Graph with PBR

• A single Service Graph/ExtEPG can now host as many services as we want

• Ingress Controller can be scaled (creating multiple instances) as needed

• If you create a dedicated EPG for ingress you need the following contracts:

• All the contracts used in kube-default (remember contract inheritance)

• Consume: Kube-API, Ingress need to be able to speak with the Kube API server

• Consume: any required ports between Ingress Controller and the service you wan to expose

117BRKACI-2505


ACI and Kubernetes Ingress

118BRKACI-2505

L3Out

default_extEpg

0.0.0.0/0

Ingress_ExtEPG

10.34.0.6/32


RTR

Client

Cons

Prov

Ingress Cont 1

Ingress Cont 2

Ingress Cont N

Pod1

Pod2

Namespace

ingress

Namespace

ciscolive

Namespace

cisco

Service

Pod1

Pod2

Service

Ingress

Controller

Ingress Resource

(I am www.cisco.com)

Ingress Resource

(I am www.ciscolive.com)


ACI and Kubernetes External Services - Summary

• Two options (can be used at the same time even for the same service)

• Exposing services via ingress

• Exposing up to 250 services directly with Service Graph with PBR

119BRKACI-2505

Demo 4 Exposing Services with Ingress

Kubernetes Cluster Node Failure


Kubernetes Cluster Node Failure Detection

• Kubernetes Monitors by default all the node in the clusters

• Depending on the configuration, node failure detection and container restart can take from ~50s to 5min. This will depend on your specific configuration.

• Once a node is detected as NotReady (failed) the aci-container-controller will update the ACI configuration as required i.e. a failed node will be removed from the PBR redirection policy

122BRKACI-2505


ACI CNI redundancy during node failure

• DataPlane of the CNI Plugin

• Start and Stop with the Node

• If isolated from the network they will try to reconnect to the leaf

aci-containers-host & aci-containers-openvswitch


ACI CNI redundancy during node failure

• Stateless

• Does not sit in the data-path

• In case of failure k8s will restart it on a different node

aci-containers-controller (acc)


ACI CNI redundancy during node failure – Corner case

• Node connects to OOB and ACI Fabric

• K8S Cluster communications are happening over the ACI Fabric

• acc communicates with APIC via OOB

aci-containers-controller (acc)

Node1

OOB

aci-containers-controller1

ACI

Fabric

Node2


ACI CNI redundancy during node failure – Corner case

• Node 1 losses connectivity with the ACI Fabric (interface down)

• Kubernetes master will detect node1 as lost and restart acc on Node2

• The old instance of acc1 is still running and will keep injecting the old config, overwriting the configuration changes pushed by acc2

• When designing your network ensure that acc communication with the APIC goes trough the fabric

aci-containers-controller

Node1

OOB

aci-containers-controller1

ACI

Fabric

Node2 aci-containers-controller2

Container to Non-Container Communications


Container to Non-Container Communications

• In production environments is preferred, for example, to run services like high performance databases as VMs or Bare Metal Servers

• This calls for the ability to easily provide communication between Kubernetes POD and VMs/Bare Metal

• Simply deploy a contract between your EPGs, ACI will do the rest!

• This works for any VMM domain and Physical Domains, for example you can have a Container Domain using VXLAN speaking with a Microsoft SCVMM Domain using VLAN.

128BRKACI-2505

How can I test this?


LABACI-2010 ACI Runs Everything

• WISP Lab

• Come to Build a WordPress application running on:

• Kubernetes

• Hyper-V

• Bare Metal

• ESXi

130BRKACI-2505

S_HAProxy

Percona 1Cluster

BareMetal

Percona 3Cluster

Persona 2 Cluster

L3OUT

NGINXWordPress01

NGINXWordPress02

SCVMM

ESXiBareMetal

Kubernetes

Client

Tenant-Common

Tenant-Common

Tenant-CL18-PODX

How can build my own lab?


kubespray_aci!

• Fork of kubespray

• Modified to deploy an high available cluster using ACI CNI plugin

• Open Source (not supported by TAC/Cisco etc…)

• Optionally can clone VM templates and configure everything providing a 1-Click deployment solution for your lab

• https://github.com/camrossi/kubespray_aci

132BRKACI-2505

Not officially

supported

Yes it is me… Did I

mentioned is not

officially supported?


Contiv 1.0 and Contiv-ACIContiv 1.0 in ACI Mode

Docker 1.12+

Support

OpenShift 3.4+

Integration

Physical Domain /

REST API

Contiv-ACI

Open Source at github.com/contiv

Kubernetes 1.6, 1.7

VMM Domain /

OpFlex

OpenShift Integration

Contiv Solutions

Support


Platform Summary for ACIPlatform Contiv-ACI (VMM Domain) Contiv 1.0 (Physical Domain)

GA ACI 3.0 Not recommended

GA ACI 3.1 Supported.

Version 3.4 / RHEL 7.x

Beta ACI 3.1 Not supported

Future Supported.

Engine 1.12.x / Swarm 1.2.5 / Centos 7.x

1.1.6 + Engine 1.12.x / RHEL 7.x

2.1.x + Engine 17.03-ee / RHEL 7.x

Future TBD

Future Future

Kubernetes

Nomad


Cisco Spark

Questions? Use Cisco Spark to communicate with the speaker after the session

1. Find this session in the Cisco Live Mobile App

2. Click “Join the Discussion”

3. Install Spark or go directly to the space

4. Enter messages/questions in the space

How

cs.co/ciscolivebot#BRKACI-2505


• Please complete your Online Session Evaluations after each session

• Complete 4 Session Evaluations & the Overall Conference Evaluation (available from Thursday) to receive your Cisco Live T-shirt

• All surveys can be completed via the Cisco Live Mobile App or the Communication Stations

Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at www.ciscolive.com/global/on-demand-library/.

Complete Your Online Session Evaluation

http://ciscolive.com/Online

http://www.ciscolive.com/global/on-demand-library/


Continue Your Education

• Demos in the Cisco campus

• Walk-in Self-Paced Labs

• Lunch & Learn

• Meet the Engineer 1:1 meetings

• Related sessions

137BRKACI-2505

Thank you

deploying kubernetes in the enterprise with cisco aci · container orchestration basic features

Documents