Network Virtualization, Overlays and Containers

Srini Seetharamansrini.seetharaman@gmail.com

May 2015

2

• Today’s DC networks

• Network Virtualization‒ Motivation, Requirements, Architecture

• Service Virtualization‒ Motivation, Requirements, Architecture

• OpenStack Networking

• Docker Networking

• Hands-on overlay networking

• Lessons learned

Agenda

Technology Trends

4

Application Rollout Today

• Poor automation for VLAN, Service contexts, and VRFs

• Poor legacy application design?

WebTier

ApplicationTier

DatabaseTier

5

Typical Data Center Design

Rack

Core

Aggregation

Edge

Application group A

Application group B

6

Problem: Network not ready for VMs

Over 70% of today’s servers are Virtual Machines, but VMs are not treated as first class citizens by the network

‒ East-west traffic poorly managed‒ Lack of prioritization and rate-limiting at VM level‒ Traffic between VMs on same server often unsupervised‒ IP/MAC overlap not allowed, and addressing limited by VLANs

VM VM

VM VM

VM VM

VMVM

Containers

Symptoms of a broader problem with lack of proper network abstractions and policy layering

7

Solution: SDN and NFV

Business potential How?Reduced time to revenue Speed up of service provisioning

OpEx saving Automated operations and easier management of resources

New revenue Through new business models centered around on-demand usage

Feature velocity Introduce changes quickly according to business logic needs

Improved policy compliance Ensure that cloud workload is compliant with enterprise policies (e.g., access control)

Reduced OpEx during upgrades Introduce new functions and service by replacing just software stack

Trend #1: Network Virtualization

Dynamic, Programmable, Automated8

Computing Infrastructure

SDN-based Virtualized Network Platform

Storage Infrastructure

9

Network Virtualization Requirements

Integration with legacy network

End-to-end visibility of VM traffic

Traffic isolation acrossvirtual networks

• Support bare metal servers, appliances and gateways

• VLAN, VxLAN, GRE support, allowing IP overlap across tenants

• Edge-based control of VM traffic and scalable host tracking

Troubleshooting support

Application policyOrchestrating virtual L4-L7 services

• End-to-end visibility that maps Virtual to Physical scalably

• Provisioning, and chaining of virtual services

• Application level policy across and within virtual networks

Trend #2: Service Virtualization

10

Internet Internet

NFV

Step 1. Virtualizing network functionsStep 2. Chaining/Stitching them

NFV in Data Centers

1. Virtualizing the L4-L7 network service appliance (e.g., Load-balancer)

2. Chaining services to ensure that the trafficis routed through virtual appliances

3. Optimizing service delivery for applications • Increasing number of virtual appliances• Increasing CPU or memory of each appliance• Placement of virtual appliances• Offloading certain tasks to NIC or switch

11

Compute Orchestration

SDN control

Open-source?

12

Trend #3: New Infrastructure Tools

Deploying Network Virtualization

14

Goal

Computing Infrastructure

SDN-based Virtualized Network Platform

15

Deployment mode #1: Underlay

VPN termination,L3 routing

VM VM VM VMVM VM IP 192.168.1.2, MAC 0x1VM VM VM VMVM VMVM VM VM VMVM VM

VM VM VM VMVM VM

Controllercluster

CLI, REST, GUI

IP 192.168.1.2, MAC 0x2IP 192.168.2.2, MAC 0x1

IP 192.168.1.2, MAC 0x3

IP 192.168.1.2, MAC 0x2IP 192.168.1.2, MAC 0x1

IP 192.168.2.1, MAC 0x2IP 192.168.1.3, MAC 0x4

Tenant membershipdecided based on

{switch-port, MAC, IP}tuple in each flow

VNet identifiedusing VLANs,VxLANs or GRE

Internet

Custom routingby controller

16

• Problem: SDN switches have resource limitations‒ Weak CPU incapable of doing traffic summarization, frequent

statistics reporting, and packet marking‒ Flow-table limitation in switches (e.g., 1500 exact match entries)‒ Switch-controller communication limits (e.g., 200 packet_in/sec)‒ Firmware does not always expose the full capabilities of the chipset

• Solution: ‒ Next generation of hardware customized for OpenFlow‒ New TCAMs with larger capacity‒ Intelligent traffic aggregation ‒ Minimal offloading to vSwitches

Performance Limitations

17

LegacyL3 routing

LegacyL2 switching

VM VM VM VMVM VM

10.1.1.0/24 10.1.2.0/24 10.2.1.0/24

10.1.1.1 10.1.1.2 10.1.2.1 10.1.2.2 10.2.1.1 10.2.1.2

VM VM VM VMVM VMVM VM VM VMVM VM

VM VM VM VMVM VM

vDP vDP vDP vDP vDP vDP

Controllercluster

InternetLogical link

v/p-GatewayCLI, REST, GUI

Deployment mode #2: Overlay

vDP: Virtual Data Plane

VM addressingmasked from fabric

Tunnels

Tenant membershipdecided by virtual

interface on the vSwitch

vDP

18

VxLAN Tunneling

• Between VxLAN Tunnel End Points (VTEP) in each host server

• UDP port numbers allows better ECMP hashing

• In absence of SDN control plane, IP multicast is used for layer-2 flooding (broadcasts, multicasts and unknown unicasts)

VTEP outerMAC header

Outer IP header

Outer UDPheader

VxLANheader Original L2 packet

VxLAN flagsReserved

24bit VN IDReserved

Source portVxLAN portUDP LengthChecksum

19

• Solution: ‒ Offload it to the top-of-

rack leaf switch‒ Use hardware gateway

• Problem: ‒ Overlay mode is CPU

hungry at high line rates and has anecdotally fared poorly in real world

Performance Limitations

Throughput Recv side cpu

Send side cpu

Linux Bridge: 9.3 Gbps 85% 75%

OVS Bridge: 9.4 Gbps 82% 70%

OVS-STT: 9.5 Gbps 70% 70%

OVS-GRE: 2.3 Gbps 75% 97%

Source: http://networkheresy.com/2012/06/08/the-overhead-of-software-tunneling/

20

• Combined overlay and underlay (fabric) to achieve:‒ end-to-end visibility‒ complete control ‒ best mix of both worlds

• Also called P+V or Overlay-Underlay‒ Vendors are converging towards this architecture

• The integration may need 1) link-local VLANs or 2) integration with VM manager to detect VM profile

Deployment mode #3: Hybrid

21

• Decoupling elements inside the overlay and converging with the underlay to make best of both worlds

• Current mode:

Deployment mode #3: Hybrid

Host A

VM 1

VM 2

VM 1

Host B

VM 2

Host C

VM 2

VxLAN overlay

VM 1

VM 3

VM 3

VM 4

VLAN VLAN VLAN

22

Controller

• Traffic leaving host has VLAN tag

• The VLAN + Source-MAC is mapped to a VxLAN

• Future Mode:

Deployment mode #3: Hybrid

Host A

VM 1

VM 2

VM 1

Host B

VLAN trunk

VM 2

Host C

VM 2

VxLAN

Distributed Virtual Switch or VLAN-based overlay

VM 1

VM 3

VM 3

VM 4

OpenStack or vCenter

VTEP manager

Deploying Network Service Virtualization

23

24

VM VM VM VMVM VMVM VM VM VMVM VM

vNF vNF vNF vNFvNF vNF

CLI, REST, GUI

Typical Deployment Mode is Overlay

vNF: Virtualized Network Function

Services can be single-tenanted and multi-tenanted

vNF vNFvNF vNF

Traffic to vFirewall

Traffic todst VMTraffic to

VIP

NetworkController

Service Controller

ComputeController

25

Service Type: Stateful and Stateless

OVS

VM 1VM 2VM 1

Host

OVS

VM 2

Host

Dst = VIP 1 Dst = VIP 2

Stateless service: No additional appliance needed

Stateful service: Virtual function deployed in VM or container

VM 3

Change header and Fwd to specific VM

Traffic proxiedto specific VM

Typically stateless load-balancing anddistributed access control

Typically stateful LB, Intrusiondetection and SSL termination

26

Service Scaling: Scale-out and Scale-up

• Scale-out: ‒ Deploy more network function instances‒ Scale-out of workload is also necessary

• Scale-up:‒ Give more resources to each network function instance‒ Offloading simple tasks to vSwitch, pSwitch or pAppliance

27

Combined Solution

OVS

VM 1VM 2VM 1

Host 1

OVS

VM 2

Host 2

OpenStack

Dst = VIP 1 Dst = VIP 2

Controller

Orchestration

NetworkPlumbing

VM 3

Service rolloutand chaining

L2-L7 Service orchestration

DC Network Virtualization

Policy/ QoS

Trou

ble-

shoo

ting

UI/

Ana

lytic

s

Compute

L3 Spine

VTEPLeaf

OpenStack Networking

29

• Most common platform for standardizing open API for networking, and vendors to innovate.

• Neutron: High-level abstractions for creating andmanaging tenant virtual network‒ Flat L2 connectivity across DC‒ DHCP enabled IP addressing‒ Floating-IP (for outside-in access)‒ L3 subnets and routers‒ Gateway and VPN‒ Load-balancer service‒ Security groups‒ ….

OpenStack Platform

Tenant BTenant A

30

OpenStack API

Typical workflow1. Create a network2. Associate a subnet

with the network3. Boot a VM and

attach it to the network

4. Delete the VM5. Delete any ports6. Delete the network

Network Virtualization App

SDN Controller

pSwitch

pSwitch

vSwitch

vSwitch

OVSDBOpenFlow

XYZ Custom API

XYZ Mech driver

ML2 Plugin

Neutron API

Orchestration

North-bound API

Application

Controller

South-bound API

Dataplane elements

31

Basic Technology for OpenStack Networking

Namespace Containerized networking at the process level managed at /proc. Primarily used to create

dnsmasq Open-source DNS/DHCP agent run on every host

Linux Bridge L2/MAC learning switch built into the Kernel to use for forwarding

OpenvSwitch Advanced bridge that is programmable and supports tunneling• ovs-vsctl used to configure the bridge• ovs-ofctl used to configure the forwarding flow rules

NAT Network address translators are intermediate entities that translate IP address + Ports (Types: SNAT, DNAT)

iptables Policy engine in kernel that is used for managing packet forwarding, firewall, NAT features

32

OpenStack Agents

Courtesy: Edgar Magana @ Workday

33

• Basic free OpenStack software includes:‒ OVS plugin that runs as mech driver in Neutron server, and ‒ OVS plugin that runs in both network and compute node‒ No OpenFlow. Just wrappers to ovs-vsctl and ovs-ofctl CLI

OpenStack OVS Networking Agents

RPC overmgmt. network

Controller Network node Compute node

OVS OVS

OVS Agent

OVS Agent

OVS Mech driver

Neutron server

VM VMVM VMHorizon UI

ovs-*ctl ovs-*ctl

Datanetwork

35

• Key feature that reduces bottlenecks in network

• View of 1 tenant routing through namespacesCompute node Network node

Distributed Virtual Routing

eth1

br-ex

snatnamespace

qr qg“VM2

br-tun

br-int

eth0

qrouternamespace

S1 S2

br-tun

br-int

eth0

Public accessPrivate access

eth1

br-ex

“VM1

Floating-IP Non-floating-IP

rfp

36

• Vendor-driven consortium (with Cisco, Brocade, and others) for developing open-source SDN controller platform

OpenDayLight Controller

• Overlay-based OpenStack Networking supported today

• All required features offered using Open vSwitch programming

OpenStack Networking in OpenDaylight

<#>

Docker Networking

38

39

• Over the past few years, LXC came up as an alternative to VM for running workload on hosts

• Each container is a clone of the host OS

• Docker brought Linux containers to prominence‒ Tracks application configuration and possibly archives to DockerHub

Linux Containers

Container 1

App X

Container 2 Container 3

Host OS

Guest rootApp Y

Guest rootApp Z

Guest root

40

Docker

• Excellent way to track application dependencies and configuration in a portable format.

• For instance the Dockerfile on the right can be used to spawn a container with nginx LB and accessed at a host port

$ docker build XYZ$ docker images$ docker run -i --name=nginx1

-d –i nginx $ docker ps$ docker inspect nginx1

# Pull base image.FROM dockerfile/ubuntu

# Install Nginx.RUN \add-apt-repository -y ppa:nginx/stable && \apt-get update && \apt-get install -y nginx && \rm -rf /var/lib/apt/lists/* && \echo "\ndaemon off;" >> /etc/nginx/nginx.conf && \chown -R www-data:www-data /var/lib/nginx

# Define mountable directories.VOLUME ["/etc/nginx/sites-enabled", "/etc/nginx/certs", "/etc/nginx/conf.d", "/var/log/nginx"]

# Define working directory.WORKDIR /etc/nginx

# Define default command.CMD ["nginx"]

# Expose ports.EXPOSE 80EXPOSE 443

41

Networking Still in Early Stages

Today Docker usage is predominantly within a single laptop or host. The default network on right is allocated to the nginx container we spawned.

But, folks are exploring connecting containers across hosts.

"NetworkSettings": { "Bridge": "docker0", "Gateway": "172.17.42.1", "IPAddress": "172.17.0.15", "IPPrefixLen": 16, "MacAddress": "02:42:ac:11:00:0f", "PortMapping": null, "Ports": { "443/tcp": [ { "HostIp": "0.0.0.0", "HostPort": "49157" } ], "80/tcp": [ { "HostIp": "0.0.0.0", "HostPort": "49158" } ] }

42

Many ways to network in Docker

• Many of these are similar to what we can do with VM(except the Unix-domain socket method of direct access)

Host

ContainerC

Container D

Container E Container FContainer A Container B

DirectHostnetwork

Unix-domainsockets andother IPC

Docker0Linux bridge

Docker proxy (using iptables)

Open vSwitch

Port mapping

43

Mechanisms for Multi-Host Networking

• Option 1: Flat IP space (at container level) with routing (and possibly NAT) done by host‒ Step 1: Assign /24 subnet CIDR to each host for its containers‒ Step 2: Setup ip route to ensure traffic to external subnets leave

from host interface (e.g., eth0)

• Option 2: Create overlay network‒ Step 1: Create a parallel network for cross-host communication‒ Step 2: Connect hosts in cluster using encapsulation tunnels‒ Step 3: Plug containers to appropriate virtual networks

44

Option 1: Flat IP space

Step 1: Choose CIDR wisely when starting Docker daemonStep 2: Add static routes to other containers’ subnets

Host 1

Nginx1172.17.42.18

Bash1172.17.42.19

172.17.42.1Docker0 bridge

eth0192.168.50.16

Host 2

Nginx2172.17.43.18

Bash2172.17.43.19

172.17.43.1Docker0 bridge

eth0192.168.50.17

Dockermanages

theseallocation

route add -net 172.17.43.0/24 \gw 192.168.50.17

route add -net 172.17.42.0/24 \gw 192.168.50.16

Quiz: What IP address do packets on the wire have?

NAT rules already in place to masquerade internal IP addresses

45

192.168.50.16 192.168.50.17

nginx1 ContainerX

Host 1

bash1 ContainerY

docker0

Open vSwitch

Host 2

Internet

Open vSwitch

docker0

vxlan vxlanvxlan vxlanOtherclusterhosts

Option 2: Open vSwitch based Overlay Suggest creating parallel network that decouples container networking from underlying infrastructure

46

Container and VM networking unified

• Edge-based overlays are even more important in container world.

• Open vSwitch already supports network namespaces

• VxLAN provides:‒ isolation, ‒ improves L2/L3 scalability, ‒ allows overlapping MAC/IP address

Docker Engine

OVS OVS OVS

Container

Container

Container

Container

Container

Container

VM V VM

Orchestration ?? OpenStack

VxLAN Tunneled network

Neutron OVS agent

Hands-on Exercise- Creating a neutron-like overlay

47

48

• In this tutorial exercise, we will use the LorisPack toolkit that allows easily creating the parallel network, and isolating container communication to its own pod/group

• Desired end goals: 1. Containers isolated into two virtual networks2. c1 cannot access container in different virtual network3. c1 can have overlapping IP address

• Inter-host communication uses VxLAN encapsulation

Host 2Host 1

Goal for Tutorial: Preview of Microsegmentation using VxLAN

c110.10.0.1

c210.10.0.1

c310.10.0.3

c410.10.0.4

Virtual Network 1

Virtual Network 2

X X

49

Setup 1: Installation

• Bring up two Linux VMs (preferably Ubuntu over Virtualbox) on your laptop with following installed:‒ Open vSwitch (version 2.1 +)‒ Docker (version 1.5 +)‒ LorisPack (git clone https://github.com/sdnhub/lorispack)

• The VMs should have host-only adapter added as a second interface eth1 so that they can communicate with each other.

• In my case,‒ VM1 IP is 192.168.56.101‒ VM2 IP is 192.168.56.102

Setup 2: Docker and networking On VM 192.168.56.101, we run:

# docker run --name c1 -dit ubuntu /bin/bash

# docker run --name c2 -dit ubuntu /bin/bash

# loris init

# loris cluster \ 192.168.56.102

# loris connect c1 10.10.0.1/24 1

# loris connect c2 10.10.0.1/24 2

On VM 192.168.56.102, we run:

# docker run --name c3 -dit ubuntu /bin/bash

# docker run --name c4 -dit ubuntu /bin/bash

# loris init

# loris cluster 192.168.56.101

# loris connect c3 10.10.0.3/24 1

# loris connect c4 10.10.0.4/24 2

51

• Verify the Open vSwitch configurations for connecting two nodes with VxLAN and connecting the two containers to the OVS.

Port Configuration

# sudo ovs-vsctl show873c293e-912d-4067-82ad-d1116d2ad39f Manager "pssl:6640" Bridge "br0" Port "br0" Interface "br0" type: internal Port patch-tun Interface patch-tun type: patch options: {peer=patch-int} Port "tap3392" tag: 1000 Interface "tap3392" Port "tap3483" tag: 1001 Interface "tap3483"

Bridge "br1" Controller "pssl:6634" Port "vxlanc0a83866" Interface "vxlanc0a83866" type: vxlan options: {in_key=flow, out_key=flow, remote_ip="192.168.56.102"} Port "br1" Interface "br1" type: internal Port patch-int Interface patch-int type: patch options: {peer=patch-tun} ovs_version: "2.3.90"

(Equivalent to br-int)

(Equivalent to br-tun)

VxLANtunnelport

c1 port

c2 port

52

• In our setup, we can verify reachability between containers using ping

• We observe that c1 is able to access c3, but not c4

• We observe that c4 is able to access c2 despite IP overlap

Microsegmentation in Effect

VM 2192.168.56.102

VM 1192.168.56.101

c110.10.0.1

c210.10.0.1

c310.10.0.3

c410.10.0.4

X

# docker attach c1root@c1:/# ping 10.10.0.3

Success !root@c1:/# ping 10.10.0.4

Fails!

# docker attach c4root@c1:/# ping 10.10.0.1

Success !

53

# ovs-ofctl dump-flows br1 -OOpenFlow13OFPST_FLOW reply (OF1.3) (xid=0x2): cookie=0x0, duration=941.134s, table=0, n_packets=128, n_bytes=11936, priority=0 actions=resubmit(,3) cookie=0x0, duration=941.146s, table=0, n_packets=106, n_bytes=10220, priority=1,in_port=1,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,20) cookie=0x0, duration=941.142s, table=0, n_packets=41, n_bytes=2214, priority=1,in_port=1,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,21) cookie=0x0, duration=941.131s, table=3, n_packets=0, n_bytes=0, priority=0 actions=drop cookie=0x0, duration=941.123s, table=3, n_packets=0, n_bytes=0, priority=1,tun_id=0x10ffd actions=push_vlan:0x8100,set_field:8189->vlan_vid,resubmit(,10) cookie=0x0, duration=313.581s, table=3, n_packets=14, n_bytes=1116, priority=1,tun_id=0x103e9 actions=push_vlan:0x8100,set_field:5097->vlan_vid,resubmit(,10) cookie=0x0, duration=305.662s, table=3, n_packets=114, n_bytes=10820, priority=1,tun_id=0x103e8 actions=push_vlan:0x8100,set_field:5096->vlan_vid,resubmit(,10) cookie=0x0, duration=941.139s, table=10, n_packets=128, n_bytes=11936, priority=1 actions=learn(table=20,hard_timeout=300,priority=1,NXM_OF_VLAN_TCI[0..11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:0->NXM_OF_VLAN_TCI[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:NXM_OF_IN_PORT[]),output:1 cookie=0x0, duration=941.137s, table=20, n_packets=0, n_bytes=0, priority=0 actions=resubmit(,21) cookie=0x0, duration=295.740s, table=20, n_packets=0, n_bytes=0, hard_timeout=300, priority=1,vlan_tci=0x03e9/0x0fff,dl_dst=7a:fd:84:90:33:23 actions=load:0->NXM_OF_VLAN_TCI[],load:0x103e9->NXM_NX_TUN_ID[],output:2 cookie=0x0, duration=291.662s, table=20, n_packets=0, n_bytes=0, hard_timeout=300, priority=1,vlan_tci=0x03e9/0x0fff,dl_dst=96:38:ce:87:e9:40 actions=load:0->NXM_OF_VLAN_TCI[],load:0x103e9->NXM_NX_TUN_ID[],output:2 cookie=0x0, duration=244.056s, table=20, n_packets=106, n_bytes=10220, hard_timeout=300, priority=1,vlan_tci=0x03e8/0x0fff,dl_dst=5e:fa:1a:ff:f7:53 actions=load:0->NXM_OF_VLAN_TCI[],load:0x103e8->NXM_NX_TUN_ID[],output:2 cookie=0x0, duration=941.127s, table=21, n_packets=0, n_bytes=0, priority=0 actions=drop cookie=0x0, duration=941.119s, table=21, n_packets=0, n_bytes=0, priority=1,dl_vlan=4093 actions=pop_vlan,set_field:0x10ffd->tun_id,resubmit(,22) cookie=0x0, duration=313.578s, table=21, n_packets=7, n_bytes=558, priority=1,dl_vlan=1001 actions=pop_vlan,set_field:0x103e9->tun_id,resubmit(,22) cookie=0x0, duration=305.659s, table=21, n_packets=34, n_bytes=1656, priority=1,dl_vlan=1000 actions=pop_vlan,set_field:0x103e8->tun_id,resubmit(,22) cookie=0x0, duration=619.312s, table=22, n_packets=41, n_bytes=2214, priority=1 actions=output:2 cookie=0x0, duration=941.110s, table=22, n_packets=0, n_bytes=0, priority=0 actions=drop

• 18 rules configured in a pipeline form to handle traffic using multiple match/action tables in Open vSwitch

• LorisPack rules are exactly same as the standard OVS Neutron plugin rules

• Potential debugging nightmare if using standard OVS neutron plugin!

OVS rules to achieve this

54

• While ping is running from c1 to c3, inspect vxlan traffic on the host

• Notice that traffic on wire between two hosts is encapsulated in VxLAN header

Inspect VxLAN traffic

# sudo tcpdump -enntti eth1tcpdump: verbose output suppressed, use -v or -vv for full protocol decodelistening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes

1431461413.331687 08:00:27:fd:35:5e > 08:00:27:46:8c:6f, ethertype IPv4 (0x0800), length 148: 192.168.56.101.57727 > 192.168.56.102.4789: VXLAN, flags [I] (0x08), vni 6653616:1d:ab:79:ad:f8 > 5e:fa:1a:ff:f7:53, ethertype IPv4 (0x0800), length 98: 10.10.10.3 > 10.10.10.1: ICMP echo request, id 23, seq 41, length 64

1431461413.332774 08:00:27:46:8c:6f > 08:00:27:fd:35:5e, ethertype IPv4 (0x0800), length 148: 192.168.56.102.57727 > 192.168.56.101.4789: VXLAN, flags [I] (0x08), vni 665365e:fa:1a:ff:f7:53 > 16:1d:ab:79:ad:f8, ethertype IPv4 (0x0800), length 98: 10.10.10.1 > 10.10.10.3: ICMP echo reply, id 23, seq 41, length 64

Lessons Learned

56

Debugging is a ChallengeSymptom Plausible reasons Things to checkTwo VMs unable to contact each other

• Improper subnet and access control policies

• Perform neutron client commands and verify config

• Check iptables -L -t NAT rules on both compute nodes

• Ping from VMs and check tcpdump

• VM networking not configured right

• Check neutron-debug ping-all, ssh and

Traffic from VM is not reaching outside

• DHCP failed because the subnet’s dnsmasq is not accessible or down

• Check IP assignment and gateway in the VM

• Check neutron-debug dhcping

• Network node inaccessible from compute node

• Check ovs-vsctl br-tun to verify VxLAN or GRE tunnels

• S-NAT router in network node misbehaving

• Check router configuration in OpenStack

• Check router namespace using ip netns exec <id> route –n

57

Debugging is a ChallengeSymptom Plausible reasons Things to checkTraffic from outside is not reaching VM

• Not adding floating-IP to the VM

• Check floating-IP assignment

• NAT rules lost from compute node

• Check NAT rules on each compute node

• DVR in compute node misbehaving

• Check router configuration in OpenStack

• Check router namespace using ip netns exec <id> route –n

• Check if ip netns is able to ping VM

• MTU is not correctly set in network

• Perform iperf -m between endpoints to check effective MTU and check all interfaces

ping, tcpdump, ip netns, iptables, ovs-vsctl, ovs-ofctl, neutron-debug, neutron client will haunt your dreams!

58

• Open-source version of OpenStack has challenges going to production without vendor support‒ Overlay and underlay integration not available‒ Lacks high availability for the agents‒ Analytics, metering and other operational tools are immature‒ Debugging is a tricky art

Production Challenges: OpenStack

59

• Similar challenges plagues Docker networking too. In addition,‒ Fast evolving, overwhelming ecosystem with cute-sounding DevOps

tools that is going through “natural selection”‒ Storage and Networking are second order problems.

Production Challenges: Docker

ClusterHQ'sapproachto migratingcontainersacross hosts

nginx nginx

Networking Redefined

61

Summary

• SDN brings in all operational goodness from computing world to networking world.

• Looking at service virtualization separately is not wise. Recommend a joint evaluation

• Varying architectures and networking policy being compiled down.

• VM and container networking work with similar network abstractions

‒ But at different scale and velocity

‒ Docker and OpenStack networking fairly similar

• Edge-based overlay intelligence using Open vSwitch is powerful.

Thank you.

Slidehare.net/sdnhub