Troubleshooting VXLAN and Network Services in a

Virtualized Environment

Vyenkatesh (Venky) Deshpande, VMware

Sachin Thakkar, VMware

NET5654

#NET5654

2

Objectives of the Session

Explain VXLAN Packet Flow in a Controller based environment

Introduce various tools and commands that help monitor and

troubleshoot network and network services.

3

Recommended Sessions & Labs

NET5790 – Operational Best Practices for NSX in VMware

Environments

SEC5894 - Deploying, Troubleshooting, and Monitoring VMware

NSX Distributed Firewall

NET5266 – Bringing Network Virtualization to VMware

Environments with NSX

NET5584 – Deploying VMware NSX Network Virtualization

You can actually test all the workflows in the lab HOL-SDC-1303

4

Agenda

VMware NSX Overview

VXLAN

Enhancements – Data and Control Plane

Configuration Demo

Packet Walk

Troubleshooting Demo

Dynamic Routing

Enhancements

Routing Demo

Network Virtualization Operations

Demo

5

VMware NSX – Networking & Security Capabilities

Any Application (without modification)

Virtual Networks

VMware NSX Network Virtualization Platform

Logical L2

Any Network Hardware

Any Cloud Management Platform

Logical

Firewall

Logical

Load Balancer

Logical L3

Logical

VPN

Any Hypervisor

Logical Switching – Layer 2 over Layer 3,

decoupled from the physical network

Logical Routing – Routing between virtual

networks without exiting the software

container

Logical Firewall – Distributed Firewall,

Kernel Integrated, High Performance

Logical Load Balancer – Application Load

Balancing in software

Logical VPN – Site-to-Site & Remote

Access VPN in software

NSX API – RESTful API for integration into

any Cloud Management Platform

Partner Eco-System

6

VMware NSX Functional System Overview

vSphere vSphere vSphere vSphere

vSwitch vSwitch vSwitch vSwitch

Hosts

Data Plane

Operations

UI

Logs/Stats

CMP

Consumption

Tenant UI

API

Control Plane Run-time state

Management Plane API

API, config, etc.

HA, scale-out

NSX Manager

NSX Controller

7

VXLAN

8

VXLAN Protocol Overview

Ethernet in IP overlay network

Entire L2 frame encapsulated in

UDP

50+ bytes of overhead

Decouples Physical network

from the Logical

VMs do NOT see VXLAN ID

Physical Network devices don’t see

VMs MAC and IP address

VTEP (VXLAN Tunnel End

Point)

VMkernel interface which serves as

the endpoint for encapsulation/de-

encapsulation of VXLAN traffic

VXLAN can cross Layer 3

network boundaries

Technology submitted to IETF

for standardization

• With Cisco, Citrix, Red Hat,

Broadcom, Arista and Others

9

Inner

Dest

MAC

Inner

Source

MAC

Optional

Ether

Type

Optional

Inner

802.1Q

Original

Ethernet

Payload

Inner Ethernet Frame

Outer

Dest

MAC

Outer

Source

MAC

Optional

VXLAN

Type

Optional

Outer

802.1Q

IP

Header

Data*

IP

Proto

-col

Header

Check

Sum

Outer

Source

IP

Source

Port

Dest Port

(8472)

UDP

Length

UDP

Check

Sum

VXLAN

Flags RSVD

VXLAN

NI

(VNI)

FCS

RSVD

VXLAN Encapsulated Frame

Outer

Ethernet

Header

14 bytes

Outer IP Header

20 bytes

Outer

UDP

Header

8 bytes

VXLAN

Header

8 bytes

Ether

Type

Outer

Dest

IP

*IP Header Data = Version, IHL, TOS, Length, ID

VXLAN Frame Format

10

NSX VXLAN Enhancements – Data Plane

Support for multiple VXLAN vmknics

per host to provide additional

options for uplink load balancing

DSCP & COS Tag from internal frame

copied to external VXLAN

encapsulated header

Support for Guest VLAN tagging

Dedicated TCP/IP stack for VXLAN

and Static IP addressing for VTEPs

vSphere Cluster B

UWA VTEP

UWA VTEP

UWA VTEP

vSphere Cluster A

UWA VTEP

UWA VTEP

UWA VTEP

11

NSX VXLAN Enhancements – Control Plane

A highly available and secure

control plane to distribute VXLAN

network information to vSphere

hosts

Removes dependency on multicast

routing/PIM in the physical network

Suppress broadcast traffic in VXLAN

networks

• ARP Directory Service & Cache

Controller

Cluster

Controller

VXLAN Directory

Service

MAC table

ARP table

VTEP table

12

vSphere Cluster B

UWA VTEP

UWA VTEP

UWA VTEP

NSX VXLAN – Management Plane

NSX Manager deploys Controllers

and prepares vSphere Clusters for

VXLAN

After the configuration any new

logical networks are created

through NSX manager

Platform requirements: vSphere

(vCenter and ESXi) 5.5 and VDS

5.5

NSX Manager

Controller

Cluster

vSphere Cluster A

UWA VTEP

UWA VTEP

UWA VTEP

13

Configuration Demo

Simplified configuration Prepare Cluster

Expand Transport zone

14

VXLAN NSX for vSphere – Multicast Mode

vSphere Host

VM1

vSphere Distributed Switch

VXLAN Transport Network

vSphere Host

VM2

vSphere Host

VXLAN 5001

VTEP1 10.20.10.10 VTEP2 10.20.10.11 VTEP3 10.20.11.10

vSphere Host

VTEP4 10.20.11.11

VM3 VM4

Multicast Traffic

L3 - PIM L2 - IGMP L2 - IGMP

VXLAN Transport Subnet A 10.20.10.0/24 VXLAN Transport Subnet B 10.20.11.0/24

15

VXLAN NSX for vSphere – Unicast Mode

vSphere Host

VM1

vSphere Distributed Switch

VXLAN Transport Network

vSphere Host

VM2

vSphere Host

VXLAN 5001

VTEP1 10.20.10.10

VTEP2 10.20.10.11

VTEP3 10.20.11.10

vSphere Host

VTEP4 10.20.11.11

VM3 VM4

Unicast Traffic

Controller

Cluster

VXLAN Transport Subnet A 10.20.10.0/24 VXLAN Transport Subnet B 10.20.11.0/24

16

VXLAN NSX for vSphere – Hybrid Mode

vSphere Host

VM1

vSphere Distributed Switch

VXLAN Transport Network

vSphere Host

VM2

vSphere Host

VXLAN 5001

VTEP1 10.20.10.10

VTEP2 10.20.10.11 VTEP3 10.20.11.10

vSphere Host

VTEP4 10.20.11.11

VM3 VM4

Unicast Traffic

L2 - IGMP L2 - IGMP

Controller

Cluster

Multicast Traffic

VXLAN Transport Subnet A 10.20.10.0/24 VXLAN Transport Subnet B 10.20.11.0/24

17

Consumption Demo

Create Logical Networks in few Clicks Logical network

Connect Virtual Machines

Ability to configure advanced services on the logical network

18

Unicast Mode Packet Walk

19

VXLAN Packet Flow – VTEP Report

vSphere Host

VM

vSphere Distributed Switch

MAC1

Management

Network

10.20.10.10

vSphere Host

VM

MAC2

vSphere Host

Send VNI,VTEP

Mapping to

Controller

1

2

5

10

VXLAN 5001

3

VNI VTEP IP

5001 10.20.10.10

9

IP1 IP2

Controller

VNI VTEP IP

5001 10.20.10.10

VNI VTEP IP

5001 10.20.10.11

10.20.10.11 10.20.10.12

11

6 7

VNI VTEP IP

5001 10.20.10.10

10.20.10.11

VNI VTEP IP

5001 10.20.10.10

10.20.10.11

4 8

10

VNI VTEP IP

5001 10.20.10.10

10.20.10.11

11

Report the new

VNI,VTEP

Mapping to the

Hosts

20

VXLAN Packet Flow – MAC Report

vSphere Host

VM

vSphere Distributed Switch

MAC1

Management

Network

10.20.10.10

vSphere Host

VM

MAC2

vSphere Host

Send VNI,VM

MAC Mapping

and VTEP IP to

Controller

1

2

5 VXLAN 5001

3

IP1 IP2

Controller

VNI VM MAC

5001 MAC1

VNI VM MAC

5001 MAC2

10.20.10.11 10.20.10.12

6 7

4 8

VNI VM MAC VTEP

5001 MAC1 10.20.10.10

VNI VM MAC VTEP

5001 MAC1 10.20.10.10

5001 MAC2 10.20.10.11

21

VXLAN Packet Flow – IP Report

vSphere Host

VM

vSphere Distributed Switch

MAC1

Management

Network

10.20.10.10

vSphere Host

VM

MAC2

vSphere Host

Send VM MAC,

IP Mapping and

VNI to Controller

1

2

5 VXLAN 5001

3

IP1 IP2

Controller

VNI VM

IP

VM

MAC

5001 IP1 MAC1

VNI VM

IP

VM

MAC

5001 IP2 MAC2

10.20.10.11 10.20.10.12

6 7

4 8

VNI VM IP VM MAC

5001 IP1 MAC1

VNI VM IP VM MAC

5001 IP1 MAC1

5001 IP2 MAC2

22

VXLAN Packet Flow – ARP Request

vSphere Host

VM

vSphere Distributed Switch

MAC1

Management

Network

10.20.10.10

vSphere Host

VNI VM

IP

VM

MAC VTEP

5001 IP1 MAC1 10.20.10.10

VM

MAC2

vSphere Host

1

2

VXLAN 5001

3

IP1 IP2

Controller

10.20.10.11 10.20.10.12

6 5

VNI VM

IP

VM

MAC VTEP

5001 IP1 MAC1 10.20.10.10

5001 IP2 MAC2 10.20.10.11

4

Payload L2 DA: Broadcast

SA: MAC1

ARP Request for

VM IP2 sent to

Controller

ARP Report for

VM IP2, MAC2

sent to VTEP

10.20.10.10

VNI VM

IP

VM

MAC VTEP

5001 IP1 MAC1 10.20.10.10

5001 IP2 MAC2 10.20.10.11

23

VXLAN Packet Flow – Communication After ARP Resolution

vSphere Host

VM

vSphere Distributed Switch

MAC1

VXLAN Transport

Network

10.20.10.10

vSphere Host

VM

MAC2

vSphere Host

7

8

VXLAN 5001

IP1 IP2

Controller

VNI VM

IP

VM

MAC VTEP

5001 IP2 MAC2 10.20.10.11

10.20.10.11 10.20.10.12

9

VNI

V

M

IP

VM

MAC VTEP

5001 IP1 MAC1 10.20.10.10

5001 IP2 MAC2 10.20.10.11

Payload L2 DA: MAC2

SA: MAC1

L2 IP UDP VXLAN Payload L2

DA: 10.20.10.11

SA: 10.20.10.10 5001

10

DA: MAC1

SA: MAC2

VNI VM

IP

VM

MAC VTEP

5001 IP1 MAC1 10.20.10.10

5001 IP2 MAC2 10.20.10.11

24

Troubleshooting Demo

NSX Manager – Visibility and Monitoring tools

Host level troubleshooting Monitor different tables

Packet capture

25

Dynamic Routing Details

26

Routing Enhancements

NSX Edge Pair

Active-Standby

172.16.10.0/24

172.16.20.10

192.168.100.3

172.16.10.10

192.168.10.1

192.168.10.2

External Network

VM VM VM VM

172.16.20.0/24 172.16.30.0/24

172.16.10.11 172.16.30.10

OSPF

eBGP/iBGP

IS-IS

Route Re-distribution/Filtering

Graceful Restart

HA

Dynamic Routing

Logical Distributed

Router

27

Routing Demo

28

Network Virtualization - Operations

Highlights

• Per VM flow visibility

• Traffic Analysis – Packet Capture

• Transport health

• Inventory & Fault Management

Aggregate Operational Views

• Statistics collections

• Alarms & Health Monitoring

• Network Performance & Resource Utilization

• Manage & Monitor through infrastructure management tool such as vCenter Operations Manager

29

Operations Demo

vCenter Operations Statistics

Health Monitoring

30

Key Takeaways

No Multicast required in the Physical Network for VXLAN

VMware NSX provides various tools to help you troubleshoot your

network and network services

Integration with next generation operations/monitoring tools for

analysis and alerting of NSX constructs.

31

Q&A

http://blogs.vmware.com/vsphere/networking

@VMWNetworking

@Sachin_t

32

Other VMware Activities Related to This Session

HOL:

HOL-SDC-1303

VMware NSX Network Virtualization Platform

Group Discussions:

NET1003-GD

VMware Network Services with Arun Goel

NET5654

THANK YOU

Troubleshooting VXLAN and Network Services in a

Virtualized Environment

Vyenkatesh Deshpande, VMware

Sachin Thakkar, VMware

NET5654

#NET5654