nutanix technote-vmware vsphere networking with nutanix

18
©2014 All Rights Reserved, Nutanix Corporation Nutanix Tech Note VMware vSphere Networking on Nutanix Nutanix Virtual Computing Platform is engineered from the ground up for virtualization and cloud environments. This Tech Note describes vSphere networking concepts and how they can be used in a Nutanix environment to ensure optimal performance and availability. It also covers the recommended configuration settings for different vSphere networking options.

Upload: kris

Post on 08-Nov-2015

129 views

Category:

Documents


6 download

DESCRIPTION

2014 In this tech note, you’ll learn:●Core networking concepts in VMware vSphere.●General best practices for networking for vSphere.●Recommended configurations for Nutanix systems.

TRANSCRIPT

  • 2014 All Rights Reserved, Nutanix Corporation

    Nutanix Tech Note VMware vSphere Networking on Nutanix

    Nutanix Virtual Computing Platform is engineered from the ground up for virtualization and cloud environments. This Tech Note describes vSphere networking concepts and how they can be used in a Nutanix environment to ensure optimal performance and availability. It also covers the recommended configuration settings for different vSphere networking options.

  • 2014 All Rights Reserved, Nutanix Corporation

    Table of Contents Executive Summary ................................................................................................................................................... 3

    VMware vSphere Networking Overview ........................................................................................................... 3

    Network Discovery Protocols ................................................................................................................................ 4

    Cisco Nexus 1000V / VMware NSX ..................................................................................................................... 5

    Network I/O Control (NIOC) .................................................................................................................................. 5 Management Traffic ............................................................................................................................................. 6 vMotion Traffic ....................................................................................................................................................... 6 Fault Tolerance Traffic ........................................................................................................................................ 6 Virtual Machine Traffic ........................................................................................................................................ 7 NFS Traffic ............................................................................................................................................................... 7 Nutanix CVM Traffic ............................................................................................................................................. 7 iSCSI Traffic ............................................................................................................................................................. 8 vSphere Replication Traffic ............................................................................................................................... 8 Other Defaults ........................................................................................................................................................ 8

    Load Balancing, Failover, and NIC Teaming .................................................................................................... 9 NIC Team Load Balancing .................................................................................................................................. 9 Recommendation for VSS - Route Based on Originating Virtual Port ............................................. 9 Recommendation for VDS - Route Based on Physical NIC Load (LBT) .......................................... 9 Network Failover Detection ............................................................................................................................ 10 Notify Switches .................................................................................................................................................... 10 Failover Order ....................................................................................................................................................... 10 Failback ..................................................................................................................................................................... 11 Summary ................................................................................................................................................................... 11

    Security .......................................................................................................................................................................... 11

    Virtual Networking Configuration Examples .................................................................................................. 12 Option 1 - Virtual Distributed Switch (VDS) with VSS for Nutanix Internal deployment ......... 13 Option 2 - Virtual Standard Switch (VSS) with VSS for Nutanix Internal Deployment ............ 14

    Network Performance Optimization with Jumbo Frames ........................................................................ 15 Sample Jumbo Frame Configuration ............................................................................................................ 16 Recommended MTU Sizes for Traffic Types using Jumbo Frames .................................................. 16 Jumbo Frames Recommendations ................................................................................................................ 17

    Conclusion .................................................................................................................................................................... 18

    Further Information .................................................................................................................................................. 18

  • 2014 All Rights Reserved, Nutanix Corporation

    Executive Summary The Nutanix Virtual Computing Platform is a highly resilient converged compute and storage platform, designed for supporting virtual environments such as VMware vSphere. The Nutanix architecture runs a storage controller in a VM, called the Nutanix Controller VM (CVM). This VM is run on every Nutanix server node in a Nutanix cluster to form a highly distributed, shared-nothing converged infrastructure. All CVMs actively work together to aggregate storage resources into a single global pool that can be leveraged by user virtual machines running on the Nutanix server nodes. The storage resources are managed by the Nutanix Distributed File System (NDFS) to ensure that data and system integrity is preserved in the event of node, disk, application, or hypervisor software failure. NDFS also delivers data protection and high availability functionality that keeps critical data and VMs protected.

    Networking and network design are critical parts of any distributed system. A resilient network design is important to ensure connectivity between Nutanix CVMs, for virtual machine traffic, and for vSphere management functions, such as ESXi management and vMotion. The current generation of Nutanix Virtual Computing Systems comes standard with redundant 10GbE and 1GbE NICs, which can be used by vSphere for resilient virtual networking.

    This Tech Note is intended to help the reader understand core networking concepts and configuration best practices for a Nutanix cluster running with VMware vSphere. Implementing the following best practices will enable Nutanix customers to get the most out of their storage, networking, and virtualization investments.

    VMware vSphere Networking Overview VMware vSphere supports two main types of virtual switches, the Virtual Standard Switch (VSS) and the Virtual Distributed Switch (VDS). The main differences between these virtual switch types are their functionality and how they are created and managed.

    VSS is available in all versions of VMware vSphere and is the default method to connect virtual machines on the same host to each other and to the external (or physical) network. VDS is available only with vSphere Enterprise Plus and has more advanced features and functionality.

    In general, VSS is simple to configure and maintain, but it lacks support for Network I/O Control (NIOC), has no automated network load balancing functionality, and it cant be centrally managed.

  • 2014 All Rights Reserved, Nutanix Corporation

    Bundled in vSphere Enterprise Plus, VDS supports NIOC with Network Resource Pools and can be centrally managed, allowing the VDS configuration to be applied to remote ESXi hosts easily. It also supports network discovery protocols, including Cisco Discovery Protocol (CDP) and Link Layer Discovery Protocol (LLDP).

    The following sections include a discussion of the various features of VSS and VDS, along with configuration recommendations on the Nutanix platform.

    Network Discovery Protocols Network discovery protocols give vSphere administrators visibility on connectivity between the virtual and physical network. This visibility makes troubleshooting easier in the event of issues such as cabling problems or MTU / packet fragmentation.

    VMware vSphere offers three configuration settings for switch discovery Listen, Advertise, or Both. These configuration settings are used to determine the information sent and/or received by the discovery protocol.

    Nutanix recommends using the "Both" setting, which ensures information is collected and displayed from the physical switch, and also ensures that ESXi sends information about the virtual distributed switch (VDS) to the physical switch.

    The following information is visible in the vSphere client for advertising switches: 1. The physical switch interface the dvUplink is connected to 2. MTU size (i.e., If Jumbo Frames are enabled or not). Some switches will report

    maximum MTU of 9,216 3. The switch management IP address(es) 4. The switch name, description, software version, and capabilities

    The following are general recommendations for Discovery Protocol Configuration, but Nutanix suggests all customers careful consider the advantages and disadvantages of discovery protocols for their specific security and discovery requirements.

    Recommendations for Discovery Protocol Configuration Type Depending on your switching infrastructure, use either:

    1) CDP - For Cisco-based environments 2) LLDP - For non-Cisco environments

    Operation Both - Generally acceptable in most environments and provides maximum operational benefits. This allows both vSphere and the physical network to openly advertise information.

  • 2014 All Rights Reserved, Nutanix Corporation

    Cisco Nexus 1000V / VMware NSX The Cisco Nexus 1000V and VMware NSX solutions can be used with Nutanix solutions, however this is out of the scope for this document. You can engage on the topic of SDN with Nutanix on our community site http://next.nutanix.com.

    Network I/O Control Network I/O control (NOIC) is a feature available since vSphere 4.1 with the Virtual Distributed Switch (VDS). NOIC uses network resource pools to determine the bandwidth that different network traffic types provide. When NIOC is enabled, distributed switch traffic is divided into custom and predefined network resource pools, including fault tolerance traffic, NFS traffic, iSCSI traffic, vMotion traffic, ESXi management traffic, vSphere replication (VR) traffic, and VM traffic.

    The physical adapter shares assigned to a network resource pool determine what share of the total available bandwidth will be guaranteed to the traffic associated with that network resource pool in the event of contention. It is important to understand that NIOC has no impact on the network traffic, unless there is contention. So during times where the network is less than 100% utilized, NIOC will have no advantage or disadvantage. Bandwidth made available to a network resource pool is determined by the share assigned to that pool, compared to other network resource pools. For further information about shares, review the following best practice guide from VMware: http://www.vmware.com/files/pdf/techpaper/VMW_Netioc_BestPractices.pdf

    Limits can also be applied to selected traffic types if required. Nutanix recommends not to set limits as they may unnecessarily impact performance for given workloads when there is available bandwidth. Using shares ensures burst workloads, such as vMotion, can complete their workload (in this example the migration of a VM to a different ESXi host) as fast as possible where bandwidth is available, and NIOC shares will prevent other workloads from being significantly impacted in the event available bandwidth is limited.

  • 2014 All Rights Reserved, Nutanix Corporation

    The following table shows recommended values for the Network Resource Pool Share:

    Network Resource Pool Share Value

    Physical Adapter Shares Host Limit - MB/s

    Management Traffic 25 Low Unlimited vMotion Traffic 50 Normal Unlimited Fault Tolerance (FT) Traffic 50 Normal Unlimited Virtual Machine Traffic 100 High Unlimited iSCSI Traffic2 100 High Unlimited NFS Traffic 100 High Unlimited Nutanix Traffic1 100 High Unlimited vSphere Replication (VR) Traffic2

    50 Normal Unlimited

    Other pools, including: vSphere/Virtual SAN Traffic2

    50 Normal Unlimited

    Notes: 1. This is a custom Network Resource Pool, which needs to be created manually. 2. These pools are generally not applicable or relevant in Nutanix deployments.

    Management Traffic

    Management traffic requires minimal bandwidth with a share value of 25 over two 10Gb interfaces. This configuration will ensure a minimum of approximately 1.5Gbps bandwidth. This is more than sufficient for ESXi management traffic and above the minimum requirement of 1Gbps.

    vMotion Traffic

    vMotion is a burst-type workload, which uses no bandwidth until DRS or a vSphere administrator starts a vMotion (or puts a host into maintenance mode). As such, it is unlikely to have any significant ongoing impact on the network traffic. Nutanix recommends a share value of 50 over two 10Gb interfaces. This will guarantee a minimum of approximately 3Gbps which is sufficient for vMotion activity to complete in a timely manner. This also ensures vMotion has well above the minimum bandwidth requirement of 1Gbps.

    Fault Tolerance Traffic

    Fault tolerance (FT) traffic is dependent on how many FT VMs per host (current maximum of 4 per host) and will generally be a sustained workload, as opposed to a burst workload such as vMotion. This is because FT needs to keep the primary and

  • 2014 All Rights Reserved, Nutanix Corporation

    secondary VMs in "lockstep". Generally, virtual machines using FT are critical and you need to ensure FT traffic (which is also sensitive to latency) is not impacted during periods of contention. Nutanix recommends using a share value of 50 and sharing two 10Gb interfaces. Based on this configuration, FT will be guaranteed a minimum of 3Gbps which is well above VMware's recommended minimum of 1Gbps.

    Virtual Machine Traffic

    VM traffic is why datacenters exist in the first place, so this traffic is always important if not critical. If the VMs network connectivity slows, it can quickly impact end users and reduce productivity. As such, it is important to ensure this traffic has a significant share of the available bandwidth during periods of contention. Therefore, Nutanix recommends a share value of 100 over two 10Gb interfaces. Based on this configuration, virtual machine traffic will be guaranteed a minimum of approximately 6Gbps. For most environments, this bandwidth will be more than what is required and ensure a good amount of headroom in case of unexpected burst activity.

    NFS Traffic

    NFS traffic is essential to the Nutanix Distributed File System and to virtual machine performance, so this traffic is always critical. If NFS performance is degraded, it will have an immediate impact on Nutanix CVM and VM performance. As such, it is important to ensure this traffic has a significant share of the available bandwidth during periods of contention.

    In normal operation, NFS traffic is serviced locally. So NFS traffic will not impact the physical network card unless the Nutanix Controller VM is offline in the event of maintenance or unavailability. Under normal circumstances, there will be no NFS traffic going across the physical NICs and the NIOC share value will have no impact on other traffic. For this reason, it is excluded from calculations.

    As a safety measure, to ensure that in the event of network contention, CVM maintenance, or unavailability, a share value of 100 is assigned to NFS traffic. This guarantees a minimum of 6Gbps bandwidth.

    Nutanix CVM Traffic

    For the Nutanix DFS to function, it requires connectivity to the other CVMs in the cluster. This connectivity is used for tasks such as write I/O synchronous replication and Nutanix cluster management. Under normal circumstances, there will be minimal to no read I/O traffic going across the physical NICs. This is because the Nutanix NDFS architecture was designed with the key concept of VM-data locality. However,

  • 2014 All Rights Reserved, Nutanix Corporation

    write I/O will always utilize the physical network due to synchronous replication for data fault tolerance and availability.

    For optimal NDFS performance, each CVM will be guaranteed a minimum of 6Gbps bandwidth. In most environments, this bandwidth will be more than what is required and ensure a good amount of headroom in case of unexpected burst activity.

    iSCSI Traffic

    iSCSI is not a normal protocol within a Nutanix environment. However, if iSCSI is used within the environment, this traffic is also given a share value of 100.

    Note: NIOC does not cover In Guest iSCSI traffic regulation by default. In the event In Guest iSCSI is used, it is recommended to create a dvPortGroup for In Guest iSCSI traffic and assign it to a custom network resource pool called "In Guest iSCSI" and give it a share value of 100 (High).

    vSphere Replication Traffic

    vSphere Replication (VR) traffic may be critical to your environment if you choose to use VR with or without VMware Site Recovery Manager (SRM). When using SRM, it is highly recommended to leverage the Nutanix Storage Replication Adaptor (SRA) as this is more efficient than using vSphere-based replication alone.

    If using VR without SRM, the default share value of 50 (Normal) should be suitable for most environments. This guarantees approximately 3Gbps of network bandwidth.

    Other Defaults

    Other default points are generally not relevant and can be disregarded with their share value of 50.

  • 2014 All Rights Reserved, Nutanix Corporation

    Load Balancing, Failover, and NIC Teaming vSphere provides a number of load balancing, failover, and NIC teaming options. Each option should be carefully understood and considered for a vSphere networking deployment.

    NIC Team Load Balancing

    The available options for NIC team load balancing include:

    1. Route based on originating virtual port 2. Route based on IP hash 3. Route based on source MAC hash 4. Route based on physical NIC load called load-based teaming or LBT (VDS

    only) 5. Explicit failover order

    Recommendation for VSS - Route Based on Originating Virtual Port

    The route based on originating virtual port option is the default load balancing policy and has no requirement for advanced switching configuration such as LACP, Cisco EtherChannel, or HP teaming. These attributes make it simple to implement, maintain, and troubleshoot. Route based on originating virtual port requires 802.1q VLAN tagging for secure separation of traffic types. The main disadvantages are no load balancing based on network load, which results in traffic from a single VM always being sent to the same physical NIC, unless there is a failover event caused by a NIC or upstream link failure. This is less of an issue with the high throughput 10GbE network interfaces of the Nutanix Virtual Computing platform.

    Recommendation for VDS Route Based on Physical NIC Load (LBT)

    For environments using VDS, Nutanix recommends using route based on physical NIC load. With this option, LBT has no requirement for advanced switching configuration such as LACP, Cisco EtherChannel, or HP teaming. This option provides fully automated load balancing which takes effect when the utilization of one or more NICs reach and sustain 75% for a period of 30 seconds or more based on egress and ingress traffic. LBT requires 802.1q VLAN tagging for secure separation of traffic types. As a result, LBT is a very simple and effective solution to implement and maintain, and works very well in Nutanix deployments.

  • 2014 All Rights Reserved, Nutanix Corporation

    Network Failover Detection

    VMware ESXi uses one of two methods of network failover detection: beacon probing and link status.

    Beacon probing works by sending out and listening for beacon probes which are made up of broadcast frames. Beacon probing is dependent on having three network connections. As a result of this requirement, it is not recommended for the current generation of Nutanix Virtual Computing Platforms which currently have two network ports.

    Link status is dependent on the link status provided by the physical NIC. Link status can detect failures, such as a cable disconnection and/or physical switch power failures. Link status cannot detect configuration errors or upstream failures.

    To avoid the limitations of link status relating to upstream failures, enable "link state tracking" on physical switches that support this option. This enables the switch to pass upstream link state information back to ESXi, which will allow link status to trigger a link down on ESXi where appropriate.

    Notify Switches

    The purpose of the notify switches policy setting is to enable or disable communication by ESXi with the physical switch in the event of a failover. If configured as "Yes", ESXi will send a notification to the physical switch to update its lookup tables on a failover event. Nutanix recommends enabling this option to ensure that failover occurs in a timely manner with minimal interruption to network connectivity.

    Failover Order

    Using failover order allows the vSphere administrator to specify the order in which NICs failover in the event of an issue. This is configured by assigning a physical NIC to one of three groups: active adapters; standby adapters; or unused adapters.

    In the event all active adapters lose connectivity, the highest priority standby adapter will be used. Failover order is only required in a Nutanix environment when using Multi-NIC vMotion.

    When configuring Multi-NIC vMotion, the first dvPortGroup used for vMotion must be configured to have one dvUplink active and the other standby, with the reverse configured for the second dvPortGroup used for vMotion. For more information see: Multiple-NIC vMotion in vSphere 5 (KB2007467)

  • 2014 All Rights Reserved, Nutanix Corporation

    Failback

    For customers not using the VDS and LBT, the failback feature can help rebalance network traffic across the original NIC. This can result in improved network performance. The only significant disadvantage of setting failback to Yes is in the unlikely event of network instability or network route flapping, since having network traffic fail back to the original NIC may result in intermittent or degraded network connectivity. Nutanix recommends setting failback to Yes when using VSS and No when using VDS.

    Summary

    The following table summarizes Nutanix recommendations for NIC

    Recommendation for Load Balancing, Failover, and NIC teaming Virtual Distributed Switch (VDS) Load Balancing Route based on physical NIC load (LBT) Network Failover Detection Link Status Only (Ensure "Link State Tracking" or

    equivalent is enabled on Physical switches)

    Notify Switches Yes Failback No Virtual Standard Switch (VSS) Load Balancing Route based on originating virtual port Network Failover Detection Link Status Only (ensure "Link State Tracking" or

    equivalent is enabled on physical switches) Notify Switches Yes Failback Yes

    Security When configuring a VSS or VDS, there are three configurable options under security: promiscuous mode; MAC address changes; and forged transmits. Each of these can be set to "Accept" or "Reject".

    In general, the most secure and appropriate setting for each of the three options is "Reject". There are several use cases which may require you to set a specific option to Accept. An example of use cases to consider configuring "Accept" on forged transmits and MAC address changes are:

    1. Microsoft load balancing in Unicast mode 2. iSCSI deployments on select storage types

  • 2014 All Rights Reserved, Nutanix Corporation

    For more information, see the Network Load Balancing Unicast Mode Configuration (KB1006778) in the VMware Knowledgebase. The following are general recommendations for Virtual Network Security settings, but Nutanix suggests all customers carefully consider their requirements for their specific applications.

    Recommendation for Virtual Networking Security Promiscuous Mode Reject MAC Address Changes Reject Forged Transmits Reject

    Virtual Networking Configuration Examples The following two virtual networking configuration examples cover the Nutanix recommended configurations for both VSS and VDS solutions. Each configuration discusses the advantages, disadvantages, and common use cases.

    All Nutanix deployments use an internal-only VSS for the NFS communication between the ESXi host and the Nutanix CVM. This VSS remains unmodified regardless of the virtual network configuration for ESXi management, virtual machine traffic, vMotion, and so on. Nutanix recommends that no changes be made to this internal-only VSS.

    In both of the following options, Nutanix recommends all vmNICs be set as "Active" on the Portgroup and/or dvPortgroup unless otherwise specified.

  • 2014 All Rights Reserved, Nutanix Corporation

    Option 1 - Virtual Distributed Switch (VDS) with VSS for Nutanix Internal deployment

    Option 1 is recommended for customers using VMware vSphere Enterprise Plus who would like to use Virtual Distributed Switches. Option 1 has a number of benefits, including:

    1. The ability to leverage advanced networking features, such as NIOC and LBT (route based on physical NIC load)

    2. It reduces cabling/switching requirements 3. It provides the ability for all traffic types to "burst" where required up to

    10Gbps 4. It is a simple solution which only requires 802.1q configured on the physical

    network 5. It can be centrally configured and managed

    The following diagram shows a sample configuration for a VDS in a Nutanix environment. Note how the Nutanix internal VSS is unmodified.

    IPMI Out of Band Management Interface

    10Gb Adapter 1 Port 1

    10Gb Adapter 1 Port 2

    10GB 802.1q Trunk

    10GB 802.1q Trunk

    VMKernel Port

    Management Network

    Onboard Dual Port 1GB NICUNUSED

    1GB 802.1q Trunk

    1GB 802.1q Trunk

    1GB Network (LAN) 802.1q VLAN 10Vmnic1 10000

    Distributed Virtual Switch: dvSwitchNutanix

    Physical Adapters

    Vmnic0 10000 Auto

    Auto

    VMKernel Port

    vMotion

    VLAN: 11

    Virtual Machine Port group

    Svm-iscsi-pg

    Virtual Switch: vSwitch-Nutanix

    Physical Adapters

    No adapters

    VMKernel Port

    Vmk-svm-iscsi-pg

    Vmk1 : 192.168.5.1

    1 virtual machine(s)

    NTNX-XXXXXXXXXXX-A-CVM

    Vmk0 : ESXi Management IP Address

    Vmk2 : vMotion IP Address

    VLAN: 10

    VMKernel Port

    Fault Tolerance

    VLAN: 12

    Vmk3 : Fault Tolerance IP Address

    Virtual Machine Port Group

    Virtual Machine Traffic VLAN 15

    VLAN: 15

    Virtual Machine Port Group

    Virtual Machine Traffic VLAN 16

    VLAN: 16

    Virtual Machine Port Group

    Nutanix VLAN 10

    VLAN: 10

    Figure 1: Virtual Networking Option 1 with Virtual Distributed Switch

  • 2014 All Rights Reserved, Nutanix Corporation

    Option 2 - Virtual Standard Switch (VSS) with VSS for Nutanix Internal Deployment

    Option 2 is for customers not using VMware vSphere Enterprise Plus, or those who do not wish to use the VDS. Option 2 and has a number of benefits, including:

    1. It reduces cabling/switching requirements (No requirement for 1Gb ports) 2. It is a simple solution which only requires 802.1q configured on the physical

    network.

    The following diagram shows a sample configuration for a VDS in a Nutanix environment:

    IPMI Out of Band Management Interface

    10Gb Adapter 1 Port 1

    10Gb Adapter 1 Port 2

    10GB 802.1q Trunkc

    10GB 802.1q Trunk

    Onboard Dual Port 1GB NICUNUSED

    1GB 802.1q Trunkc

    1GB 802.1q Trunkc

    1GB Network (LAN) 802.1q VLAN 10

    Virtual Machine Port group

    Svm-iscsi-pg

    Virtual Switch: vSwitch-Nutanix

    Physical Adapters

    No adapters

    VMKernel Port

    Vmk-svm-iscsi-pg

    Vmk1 : 192.168.5.1

    1 virtual machine(s)

    NTNX-XXXXXXXXXXX-A-CVM

    VMKernel Port

    Management Network

    Vmnic1 10000

    Virtual Switch: vSwitchNTNX

    Physical Adapters

    Vmnic0 10000 Auto

    Auto

    Vmk0 : ESXi Management IP Address

    VLAN: 10

    VMKernel Port

    vMotion

    VLAN: 11

    Vmk2 : vMotion IP Address

    VMKernel Port

    Fault Tolerance

    VLAN: 12

    Vmk3 : Fault Tolerance IP Address

    Virtual Machine Port Group

    Virtual Machine Traffic VLAN 15

    VLAN: 15

    Virtual Machine Port Group

    Virtual Machine Traffic VLAN 16

    VLAN: 16

    Virtual Machine Port Group

    Nutanix VLAN 10

    VLAN: 10

    Figure 2: Virtual Networking Option 2 with Virtual Standard Switch

  • 2014 All Rights Reserved, Nutanix Corporation

    Network Performance Optimization with Jumbo Frames Jumbo Frames is a term given to an Ethernet frame that has a data payload and MTU size greater than 1,500 bytes. When configuring Jumbo Frames, the MTU size will typically be configured to 9,000, which is near the maximum size for an Ethernet frame.

    The idea behind Jumbo Frames is as the speed of networks increase, a 1,500-byte frame is unnecessarily small. With solutions such as IP-based storage (iSCSI/NFS) leveraging converged networks, larger frames will assist with both throughput and reducing the overhead on the network. By default, the network will be moving large numbers of 1,500 byte frames. By configuring the network to use larger 9,000 byte frames, it will process six times fewer packets, therefore improving throughput by reducing the overhead on the network devices. Solutions such as VXLAN, require a payload greater than 1,500 bytes. As a result, VMware recommends using Jumbo Frames for VXLAN to avoid packet fragmentation.

    One of the most important factors when considering the use of Jumbo Frames is confirming that all network devices can provide end-to-end support for Jumbo Frames, and can be enabled on all switches globally. If this is the case, Jumbo Frames is recommended. It is also important to point out Jumbo Frames does not have to be enabled for all traffic types.

  • 2014 All Rights Reserved, Nutanix Corporation

    Sample Jumbo Frame Configuration

    The following diagram shows an MTU size of 9,216 bytes configured on the physical network with two ESXi hosts configured with Jumbo Frames where appropriate. This illustrates the recommended configuration in a Nutanix environment. A combination of Jumbo and non-Jumbo Frames can be supported on the same deployment.

    ESXi Host ESXi Host

    MTU SIZE9216

    VMKernelESXi Management

    MTU 1500VMKernelvMotion

    MTU 9000VMKernel

    Fault Tolerance

    MTU 9000

    (dv)PortGroupVirtual Machine Traffic

    MTU 1500

    (dv)PortGroupNutanix

    MTU 9000

    VMKernelNFS or iSCSI

    MTU 9000

    VMKernelESXi Management

    MTU 1500VMKernelvMotion

    MTU 9000VMKernel

    Fault Tolerance

    MTU 9000

    (dv)PortGroupVirtual Machine Traffic

    MTU 1500

    (dv)PortGroupNutanix

    MTU 9000

    VMKernelNFS or iSCSI

    MTU 9000

    Figure 3: Using Jumbo Frames in ESXi environments

    Recommended MTU Sizes for Traffic Types using Jumbo Frames

    Note that performance is still excellent with the standard MTU of 1,500 bytes.

    The following table shows the various traffic types in a VMware vSphere / Nutanix environment, and Nutanix recommended Maximum Transmission Unit (MTU).

    Traffic Types Jumbo Frames2 (MTU of 9,000 bytes)

    1) Local and remote Nutanix CVM traffic 2) vMotion / Multi NIC vMotion 3) Fault tolerance 4) iSCSI 5) NFS 6) VXLAN1

    Jumbo Frames (MTU of 1,500 bytes)

    1) ESXi management 2) Virtual machine traffic3

    Notes:

  • 2014 All Rights Reserved, Nutanix Corporation

    1 Minimum MTU supported in 1,524, but >=1,600 is recommended 2 Assumes traffic types are not routed 3 Jumbo Frames can be beneficial in selected use cases, but not always required.

    In summary, the benefits of Jumbo Frames include:

    1. A reduced number of interrupts for the switches to process 2. Lower CPU utilization on the switches and Nutanix CVMs 3. Increased performance for some workloads, such as Nutanix CVM and

    vMotion/FT 4. Performance is either the same or better than without Jumbo Frames as long

    as they are properly configured on the end-to-end network path

    Jumbo Frames Recommendations

    The following are general recommendations for the configuration of Jumbo frames, but Nutanix suggests all customers carefully consider requirements for their specific applications and abilities of their network switching equipment.

    Recommendations for Jumbo Frames Recommendations when supported by the entire network stack

    1) Configure MTU of 9,216 bytes on a) All switches and interfaces

    2) Configure MTU of 9,000 on a) VMK for NFS / iSCSI b) Nutanix CVM internal and external interfaces c) VMK(s) for vMotion / FT

    Note: VMware also recommends using Jumbo Frames for IP-based storage, as discussed in Performance Best Practices for vSphere 5.5 on the VMware Knowledge Base site.

    The Nutanix CVMs must be configured for Jumbo Frames on both the internal and external interfaces. The converged network also needs to be configured for Jumbo Frames. Most importantly, the configuration needs to be validated to ensure Jumbo Frames are properly implemented end-to-end.

    If Jumbo Frames are not properly implemented on the end-to-end network, packet fragmentation can occur. Packet fragmentation will result in degraded performance and higher overhead on the network.

    The following ping commands can help you to test that end-to-end communication can be achieved at Jumbo Frame MTU size without fragmentation occurring:

  • 2014 All Rights Reserved, Nutanix Corporation

    Windows: ping l 9000 -f

    Linux: ping s 9000 M do

    ESXi: vmkping d s 8972

    Note: In ESXi 5.1 and later, you can specify which vmkernel port to use for outgoing ICMP traffic with the I option.

    Conclusion The Nutanix Virtual Computing Platform is a highly resilient converged compute and storage platform designed for supporting virtual environments such as VMware vSphere. Understanding fundamental Nutanix and VMware networking configuration features and recommendations is key to designing a scalable and high performing solution which meets customer requirements. Leveraging the best practices outlined in this document will enable Nutanix and VMware customers to get the most out of their storage, compute, virtualization, and networking investments.

    Further Information You can continue the conversation on the Nutanix Next online community (next.nutanix.com). For more information relating to VMware vSphere or to review other Nutanix Tech Notes, please visit the Nutanix website at http://www.nutanix.com/resources/.