plexxi and vmware overview · white paper plexxi and vmware integration capabilities plexxi and...

White PaperPlexxi and VMware Integration Capabilities

Plexxi and VMware Integration Capabilities

Plexxi and VMware OverviewPlexxi’s focus with VMware is to fully enable the software-defined data center (SDDC) vision through tight integration of the software-based tools from VMware with the hardware, software, and data that Plexxi controls from the network. The goal is to create a first-class user ex-perience for the Cloud/DC infrastructure operator where all of their high-level workflows for configuration/provisioning, operations and automation are provided through VMware tools.

Out of the box, Plexxi offers a number of key capabilities to enable a highly agile data center network. These capabilities are easily integrated into the relevant tools from VMware such as vCenter, vRealize Opera-tions, vRealize LogInsight, and vRealize Orchestrator. If the user also wants to integrate NSX network virtualization on top, the Plexxi solu-tion integrates the network physical data path and control paths with the network virtualization layer to create a truly synchronous physical and virtual network experience.

In this document we’ll describe how Plexxi and VMware’s respective solutions are integrated across a 3-phase strategy to offer a truly soft-ware-defined data center solution by looking at the various capabilities that are enabled. The 3 phases of the strategy are:

1. Dynamic Auto-configured VMware Fabric2. Operations and Management Integration3. Network Virtualization Integration

This paper will examine these three phases and talk about the benefits each unlocks for the VMware centric customer.

Phase 1. Dynamic Auto-configured VMware FabricAuto ConfigurationPlexxi supports a complete auto configuration model with VMware that allows the network ports and grouping information to be automatically learned from VMware. This integration builds and synchronizes the vir-tual/network infrastructure for the operator instead of building it manu-ally, significantly reducing operational costs and time-to-provision.Plexxi supports the following auto-configuration capabilities:

of 17© Plexxi, Inc. 2016



• VLANs and Link aggregation groups (LAGs) (i.e. the network topology)• LAGs are automatically learned from vCenter’s LAC

configuration• Dynamic vMotion network migration (“nMotion”) with Dynamic

Affinities (discussed in Dynamic Fabric section)• Isolated VSAN network topology

Logical Workload PoliciesOne of the most basic aspects of running a data center is creating new workloads. These workloads need physical resources on which to exe-cute – namely compute and storage, and also need connectivity between those components and to other workloads or the external world.

Data center operators leverage VMware to deploy these workloads. As they define the various virtual machines, storage LUNs, VSANs, vVols, and other virtualized abstractions of the physical resources, they also need to define how these components are interconnected. The integration of Plexxi’s network solution with VMware’s vCenter allows this inter-connection and policies related to this interconnection to easily synchronize automatically without user intervention.

VMware also allows for the additional administrative scoping of metadata within vCenter, specifically in the form of notes and/or tags. Plexxi uses these administratively scope-able fields to define network policy (known as “Affinities” in Plexxi’s terminology).

The ability to administratively scope metadata with respect to objects that exist in a given VMware environment allows infrastructure administrators the ability to perform batch operations on specific pieces of metadata defined with the VMware environment. Depending on the depth of a given environment and integration model this metadata could simply create Affinity Groups that are later used in specific policies defined within the network domain, or it could be used for other portions of an integration environment, such as visualization, monitoring, or logging.

This allows a flexible and abstract way to logically group objects within the VMware system for consumption within the Plexxi system. Ultimately this allows for the user to identify, match on, and apply specific business logic to meta-data harvested from a VMware installation such that given workloads achieve specific treatment.

Example Use CasesMulti-tiered Applications

A common need for a data center operator or application architect is to define tiers within a given application hosted within a VMware environment. Using the tag metadata model we can tag VMs according to their function within a given application architecture housed within the VMware hypervisor. A generic example of these tiers is depicted in the image below. It represents



Using the identified logical groupings, above, we can now use the tag metadata within VMware to specify if a specific node belongs to a given tier within the application, assign the VM to a specific Affinity Group based on that tag metadata and build network policy accordingly. An example of what the Affinity Group logic would look like is shown here:

These tags could also easily communicate any specific needs of the given sub-workload components. For example, if all traffic from VMs within the “Web Servers” Affinity Group to all VMs within the “Application Servers” group required the least number of switch hops, a “MinHop” tag could be designated to signify this policy. Within Plexxi Control, this tag would match on, and deliver, a “Hop Affinity” attribute on the Affinity Link between Web Servers and Application Servers affinity groups. This same model could extend to any logical grouping of resources, where policies are associated defining dynamic forwarding parameters based on metrics such as minimal latency, maximum bandwidth, isolated paths, or any combination of those.Plexxi Control will use the set of administrator defined “Affinities” to create unique paths through the network fabric aligned to the needs of the workloads, As workloads are added / changed /deleted the fabric is constantly adjusting to match their new demands.

Dynamic Affinity Groups

Plexxi’s base integration with VMware allows for the network to be automatically fine tuned for the VMware infra-structure. This integration includes VMware data services, storage arrays, and end-user applications. Some of the capabilities included in this set of features includes:

• Dynamic Affinity Groups for VMware Data Services• vMotion – allowing for dynamic migration support, or “nMotion”. This integration automatically senses a

vMotion about to occur and creates dedicated network bandwidth for the vMotion. This keeps the vMotion from impacting other user traffic and ensures a high-quality, speedy migration.

• Fault Tolerance – VM FT is used to create highly resilient VMs, and includes VM to VM state information that is synchronized. The Plexxi integration can detect this configuration and create an isolated network for these critical VMs to ensure their state synchronization is congestion free.

• vSAN – Plexxi’s VMware integration automatically discovers VMKernel Adapters that participate in vSAN and constructs an Affinity Group to keep the storage traffic isolated. Storage typically creates a “noisy neigh-bor” problem when integrated onto a common network, which is why many people create dedicated net-works for storage traffic, creating unneeded cost and complexity. In Plexxi’s case, it is a single network, that is logically separated leveraging the unique photonic fabric capabilities.

• Integrated in-band management or failover – Management in a VMware environment typically requires a separate out-of-band network. While this is generally a good practice, it can also create a separate operation-al domain. Plexxi can integrate the VMware management onto a logically separate part of the network fabric



to ensure containment. For additional resiliency, this network could also be used as a back-up or failsafe for the external out-of-band management.

• Dynamic Affinity Groups for VM lifecycle events• In many environments, especially ones leveraging virtual

desktops, many VMs will boot at the same time causing “boot storms”. The Plexxi integration has the ability to define a threshold that indicates a boot storm is in progress and auto-matically set up an affinity grouping to ensure the VM boots don’t create congestion that could slow the booting process. This could also be created as a scheduled event (e.g. every day at 9am) versus relying on a reactive detection mechanism.

Phase 1 BenefitsPhase 1 Plexxi to VMware integration allows a customer to easily create a network for VMware ESX environments. The integrations are aimed at solving basic auto-configuration capabilities, as well as more advanced capabilities that help improve policy automation for workload performance, and automatic network services that apply to the VMware infrastructure. Overall, this phase creates an easy-to-use network that has out-of-the-box automation to save time, easy ways to ensure workload performance, with a fabric that is already aware of critical VMware infrastructure services and will ensure they are provided with optimized connectivity to ensure performance.

Phase 2. Operations and ManagementMonitoring Network UtilizationMaximizing Network UtilizationData Center operators prefer to “run the network hot” – that is, utilize as much of the network capacity as they have paid for. Unfortunately, this is very difficult in most data centers due to the inherent limitations of most data center network architectures.

The Current Data Center Network ApproachOne common best practice in data centers with heterogeneous workloads is to create a “folded Clos” topology, or more commonly called a “leaf and spine” whereby a number of access or leaf switches provide access to end devices such as services and storage, and a number of intermediate devices or spines connect the leaf’s to each other.

Traffic is generally distributed over this architecture using the Equal-cost Multipath (ECMP) algorithm, which at-tempts to distribute the load of traffic evenly across all of the available links by randomly choosing a link between the leaf and the spine for a given flow of traffic. Here is where the trouble starts.



Different workloads = Different Network NeedsThe random selection of links is completely autonomous and subse-quently ignorant to the needs of workloads. While in theory one might expect that a random selection would essentially “even itself out” given a suitably reasonable set of flows. However, history has shown the opposite, specifically that with a diverse set of workloads (some bursty, some bandwidth intensive, etc.), networks develop hot spots that are extremely difficult to diagnose, never mind to remediate. Empirically we see this with the classic analogy where application users consistently complain about network congestion while network engineers consistent-ly attest that they have plenty of capacity!

East-West traffic versus North-SouthLeaf/spine and other hierarchical designs were created for primarily north-south oriented data centers (client/server). In these environments, you could safely assume that most traffic stayed within the rack (contained to a leaf) or went through an uplink to the spine to the external world (client to server). Workloads were architected to stay within the same rack to avoid cross-rack bandwidth usage, which by design always has to go through a Spine.

The north-south orientation allowed data center designers to “oversubscribe” the uplinks. That is, in many cases there could be as little as 3 and often as much as 48 times as much “downlink” (leaf to server) bandwidth than leaf to spine. This is because the assumption was that most traffic would be within the rack, and some would leave but only go out of the data center, versus across to another rack.

As east-west traffic flows increase, the only choices have been to lower the oversubscription to 1:1 (which makes the costs unfeasible in most cases). Higher speed uplinks (e.g. 100 Gbps) can help mitigate the issue for some time (with a full re-architecture and re-cabling to go from 40 to 100), but the same issue will occur as Moore’s law continues to push the performance of processors and storage densities, which naturally drives workload bandwidth.

Aggregating traffic over higher speed links (i.e 100 GbE) is only a winning strategy if electrical switching performance curve is equal to or faster than CPU and disk perfor-

mance curves (and its not, by a long shot).

When we discuss network utilization in the data center context, it is important that we first start with the understanding that this dynamic does not have to exist. If your starting assumption is that different workloads have different needs, and therefore different traffic characteristics, then a logical follow-on is that they need a different set of links (or links of different characteristics) that form the topology they use. This is a very sharp contrast to the mindset that the net-work will randomly (and therefore somehow evenly) distribute traffic, blind to the differences in workloads, and ev-erything will be ok. Similarly, if you are designing a data center for primarily east to west traffic flows, you will make very different decisions if your starting point is north to south.

Creating path diversity and Understanding WorkloadsIn a Plexxi network, the network fabric is designed first and foremost with this in mind. That is, that:

• Blindly (or randomly) distributing traffic without regard to workloads will lead to congestion and hot spots, and• Aggregating all flows over a smaller set of higher speed links is ultimately a losing war in relation to the growth

in workload performance and generated.Therefore, we aim to solve those 2 limitations of traditional networks. Specifically, we build a model of workloads based on information from higher-level orchestration and infrastructure management tools, such as VMWare’s vCen-



A great way to understand this is by thinking about an analogous system in terms of roads and GPS routing technology. Imagine a system without central control/authority, whereby each type of traffic workload (think freight lines, commuters, local traffic, motorcades) was randomly assigned a path to get to its destina-tion, and that all small roads aggregated to a few superhighways through which everyone had to pass. In short order chaos would ensue as certain highways became highly congest-ed and others were wide open. In fact, this is representative of today’s road system whereby users determine there own paths creating “ran-domness” that ultimately leads to congestion every day if you live in a city of any decent size!

Contrast this to a centrally controlled system whereby all traffic workloads are declared to a global planning system, and all possible paths from any point A to B is known by this system. Furthermore, in this new system, instead of all roads aggregating into some small number of highways, this system includes a large num-ber of point-to-point roads. Even more, these point-to-point roads can also be moved as the planning system learns about the common paths needed by specific traffic workloads. In this system, it is quite possible, and in fact quite easy, for a computer to calculate the best way to use the available roads to meet the needs of the different workloads (a process Plexxi refers to as “fitting”). Of course, the idea of dynami-cally movable roads and declared traffic work-loads in the physical world is highly unlikely in our lifetimes, this notion is very much possible in a data center network.

ter, and we provide a multitude of granular paths that a central-ized controller (Plexxi Control) can leverage to provide each workload with the right set of paths that satisfy its individual needs for network treatment.

More Switches = More Paths = More Path DiversityMore paths provide more ways to get from A to B to solve work-load performance criteria. The ability to granularly manage paths on a per workload basis allows the network to meet workload performance needs even in the face of exponential growth in CPU performance and storage density which generates exponen-tial workload demands on the network.

Path diversity in a Plexxi networks comes by way of LightRail™ and the photonic switching capability. Instead of leveraging higher-speed uplinks and aggregating all traffic over a smaller set of high speed links (that have to transit through a spine switch to get re-distributed), we leverage more granular paths that provide direct connections (physical and logical) from switch to switch. Each one of these paths provides our SDN controller with the ability to intelligently solve specific workload issues to avoid the congestion that naturally occurs in leaf/spine approaches.

Similar to the way scale-out computing works (more lower speed multi-core CPUs versus single higher speed CPUs), the Plexxi network leverages these paths to effectively spread traffic out and avoid congestion. As the network needs grow, new switches can be added completely incrementally, avoiding the need to scale-up before scaling-out, a Plexxi network can simply scale-out.

These pictures represent the multitude of paths available to the Plexxi control plane to solve for workload constraints. The con-troller has full knowledge and control over these paths and what



traffic gets placed on each one, and can even stitch paths together or re-allocate paths that aren’t being used to different parts of the network.

What makes this level of control possible is having a large number of paths to work with. Fewer higher speed paths would give much less control, forcing most/all workload traffic across the same set of paths, which would naturally lead to congestion issues.

Complete Data Center Visualization with vRealize Operations InsightThe VMware vRealize suite of tools provides an advanced set of soft-ware for managing the software-defined data center. vRealize Oper-ations specifically focuses on providing a centralized dashboard for health monitoring, predictive analytics, alerting, along with other fea-tures for capacity planning, monitoring, and application awareness.

vRealize Operations also adds the ability for device-specific integrations via “Management Packs”. Plexxi’s vReal-ize Operations Management Pack adds in key visibility components for the network portion of a software-defined data center.



This includes representation of the Plexxi network into the vRealize Operations model. This allows Plexxi to create new Plexxi dashboards, and allows the user to see Plexxi in the existing Ops dashboards. The vRealize Operations architecture makes for integration into a users ex-isting ‘workflow’ seamless. End users interact with vRealize operations through a series of dashboards, which are made up of badges, objects, metrics and thresholds. Alerts are generated based on these primitives. This allows for centralized cloud infrastructure teams to view the entire infrastructure as a single system, versus as separate physical silos.

The vRealize Operations model is made up of a series of objects, that typically represent something like a virtual machine or a virtual network interface. In the Plexxi Management Pack, Plexxi metrics are attached

to existing objects, and are used to create Plexxi objects that make sense to existing badges. This also includes the creation of new badges to represent the overall Plexxi solution. There does exist a relationship between these VMware objects across the physical fabric, but VMware tools inherently have no knowledge of this, however, Plexxi builds an end to end model which is now found within vRealize Operations. This presents the user with unprecedented capabili-ties to manage and troubleshoot their IT and network infrastructure from a single pain of glass, vRealize Operations.

Example Use CasesPrior to integration with Plexxi, VMware customers used vRealize Operations for their virtualized environment oper-ational needs but lacked the ability to correlate network related data. While Plexxi Control is still an important part of the solution, users can leverage the vRealize dashboards as their primary interface. This is possible due to the fact that Control is managed by vRealize through the API-level integration. Plexxi Control is available for advanced trouble-shooting context and for initial bootstrap. Day-to-day tasks can now be done exclusively through vRealize Operations and vCenter.This enables the following use cases:

• Plexxi dashboard for day to day use with Plexxi in a VMware environment• Users can glance at the Plexxi vRealize Operations dashboard to view the current state of the Plexxi fabric

• Switch / ring state / fitting parameters / machines harvested• Replication of the Plexxi Control dashboard in vRealize Operations, for a consolidated data center view

• Adding context to existing vCenter and NSX dashboards• When the user has an alert, relevant network related data can be included for correlation

• If a Physical NIC is down, check the last known Plexxi port for immediate status • If vMotion fails, check the network for corroborating information on link status, congestion,

etc.



• Report congested fabric links to the overall vRealize Operations health badge, for a vhost / vm / storage • Provide a user-actions option to refit the network, redistrib-

uting traffic and mitigating congestion• Trace Connectivity issues

• Trace the fabric path for a pair of vhost / vms, and report VLAN configuration mismatches and offer to proactively fix it if the VLAN is not configured properly on the Switch Port

• For example, a vMotion will fail if the vmnic port group is not configured correctly with the vlan on the associated port

• Offer a VM / host storage object button to Increase network capacity to a specific location• Create optimized rules for VM / host• Detect (via sflow) who this machine is talking to and create the Affinity automatically

• Porting any Plexxi Error level alert in Control into the vRealize Operations dashboard• Use vRealize Operations data to predict network fabric traffic patterns and potential future points of congestion

and provide feedback to proactively Fit the network• When virtual machines need to be redeployed, Plexxi can make a suggestion as to the best ‘network location’ for the VM

Phase 2 BenefitsNetwork operations have historically been a time-consuming and labor-intensive effort. In modern cloud-centric data centers that leverage virtualization, there are many advanced software tools that provide visibility and cor-relation of the physical infrastructure to workload-related issues and problems. Plexxi’s integration with vRealize Operations allows customers to further benefit from those tools as now the network can provide highly relevant details, never before available to the dashboard. This simplifies the day-to-day operations of the data center. Trou-bleshooting across the physical and virtual environments becomes easier and every-day operational tasks can be viewed from a single dashboard.

Phase 3. Network Virtualization and Run-book AutomationCreating “Native” Network Virtualization Ready NetworksVMware’s NSX is this industry’s first and market-leading network virtualization solution. Network virtualization allows customers to define their networking constructs in software – i.e. in virtual switches that operate on the end de-vices themselves (the servers). The NSX software creates virtual networks that define specific communication patterns between groupings of devices.

When NSX runs on a non-Plexxi physical network, it runs as an “overlay” meaning there is little to no coordination between the virtual network and the physical network – the physical network is treated as a pure transport layer. This separation of the physical and virtual network requires significant manual engineering effort and forethought to ensure the physical network will meet the needs of the virtual layer. When the physical network is reduced to transport only, it loses the visibility and awareness of the traffic, rendering it unable to react to congestion issues or other problems. When NSX runs on a Plexxi infrastructure, the physical network and the virtual network can be natively and dynami-cally aligned providing unified visibility, traffic awareness and traffic path optimization.

Some of the key questions network designers need to understand when network virtualization is used and their impli-cations on Plexxi and non-Plexxi infrastructure include:



• How much oversubscription is acceptable between the host switching tier and the aggregation tier of the network? Network virtualization creates L3 encapsulated L2 segments (VXLAN) that are meant to be opaque to the physical network. The phys-ical network sees only the outer IP addresses of the VXLAN Tunnel Endpoints (VTEPs) and in most L3-centric designs will randomly hash that traffic across the available network fabric links. As inter-VM traffic increases, more VXLANs will end up sharing the same network paths, leading to unpredictable re-sults. The only way to ensure that this doesn’t cause un-fixable network problems is to build a fully subscribed (no oversub-scription) network, which can be costly to build and operate, but more importantly leads to a lot of wasted capacity that is only there to prevent congestion.

• A native network virtualization ready network allows for dynamic redistribution of fabric capacity to match the actual VTEP to VTEP traffic demand without requiring over-allocation of bandwidth that may never be used.

• How are per-VXLAN segment (VNI) performance and network service attributes handled? While VXLAN allows L2 hosts to communicate easily even over a L3-based infrastructure, as mentioned previously, it also cre-ates a layer of encapsulation that makes it difficult to manage physical network policies at the Virtual Network level. A VXLAN segment is referred to as a Virtual Network and is identified in the VXLAN header as the VX-LAN Network Identifier (VNID) field. As more and more VNIs traverse a given pair of VTEPs, the need arises to be able to differentiate each VNI for either policy or SLA purposes.

• A native network virtualization ready network allows the DC operator to create per-VNI policies for crit-ical applications either for performance SLAs or to adhere to specific traffic steering requirements, e.g. for security purposes.

• How are physical VTEP gateways managed? While virtualized hosts create tunnel endpoints (VTEPs) on the host itself, non-virtualized hosts (like physical appliances or applications running on bare metal servers) typically require an external VTEP gateway. For performance and manageability reasons, it is preferred to implement these gateways on devices with hardware-based VTEP capabilities like top of rack (ToR) switches. However, in order for these VTEPs to work in concert with the virtualization-host VTEPs, the control plane of the switch must be integrated with the overlay control plane (in this case the NSX Controller). If this integration does not exist, the network man-ager needs to manually create VTEPs on the switches and manage them in case of moves/adds/changes of the end hosts, creating unnecessary work and adding room for errors to be introduced into the network.

• A native network virtualization ready network creates a completely seamless VTEP experience for the user. This means VTEPs are orchestrated completely from the network virtualization layer and dynamically react to moves/adds/changes of the end systems they connect. In addition, it will allow for hosts on different L3 subnets to communicate across VXLANs.

• How are optimal forwarding architectures created for NSX Distributed Routing and Edge Routing? NSX creates a number of new logical components such as Edge VMs and Distributed Router instances that need to be understood before a physical network can be properly architected. There are a number of very technical



documents that outline the best practices for such designs (i.e. https://communities.vmware.com/servlet/JiveServlet/preview-Body/27683-102-7-37383/NSXvSphereDesignGuidev2.1.pdf), but very careful network engineering is still required. For ex-ample, if using a traditional L2 Cisco vPC style network design from the hosts to the switches and then L3 to routers, VMware recommends that customers create Edge VMs and a transit VX-LAN segment that creates a more efficient routing architecture (avoids unnecessary hairpinning) and has better failover charac-teristics. However, creating Edge VMs creates L2 and L3 impli-cations on the physical network that need to be designed.

• A native network virtualization ready network makes it extremely simple for the user to create a physical network architecture based on the virtual network architecture. This means if Edge VMs are used, they are automatically discovered and are given a very specific network treatment and configuration with no user intervention.

NSX running on a Plexxi network creates the industry’s first network virtualization solution that most closely mimics the relationship between server virtualization and VT-enabled multi-core CPUs. Intel’s VT technology allowed serv-er virtualization to effectively leverage the hardware, and multi-core processors allowed for the horizontal scaling of processing power, versus the scaling up approach of adding bigger (higher CPU clock cycle) CPUs when more perfor-mance was needed.

Creating a native network virtualization solution requires 2 key capabilities:

1) A tight integration between the software-based network virtualization and the hardware-based network virtual-ization, similar to the way the VT instruction set enabled tight integration between the hypervisor and the physi-cal CPU. This integration includes:a. The bi-directional sharing of information between the virtual network and the physical network.b. The ability of the virtual network to direct the physical network resources completely, including the ability to

re-arrange “the wires” as necessary.2) A true “hyper-converged” network physical model. Similar to the way multi-core processors allow more VMs on

a given server, albeit each at a lower overall clock speed, so too should the physical network provide the ability for virtual networks to spread horizontally versus aggregated vertically into higher bandwidth pipes.

Native Network Virtualization with NSX and PlexxiTight virtual to physical IntegrationIn Plexxi’s parlance, tight virtual to physical integration is called “Overlay Awareness”. This includes the following key capabilities

• The ability to interface with the VMWare NSX Controller • This allows the Plexxi network to discover VXLAN Tunnel Endpoints (VTEPs), Distributed Logical Routers

(DLRs), NSX Edge VMs, Virtual Network Identifiers (VNIs) and VM MACs • The ability to visualize the components in the UI and make them available for debug, search, etc.• The ability to allow affinities to be created with these components (for a detailed discussion on affinities, please

refer to the following white paper: http://www.plexxi.com/wp-content/uploads/2012/06/Plexxi-White-Paper-Af-finity-Driven-Networking.pdf):

• Regular endpoint affinities between VTEPs, NSX Edge VMs, and VMs using all available affinity attributes

http://www.plexxi.com/wp-content/uploads/2012/06/Plexxi-White-Paper-Affinity-Driven-Networking.pdf

http://www.plexxi.com/wp-content/uploads/2012/06/Plexxi-White-Paper-Affinity-Driven-Networking.pdf



allows the user to create specific groupings of virtual network components that have specific treatment requirements.

• Domain affinities allow for all endpoint members of a specified domain to receive the same treatment without specifying each endpoint individually. This could be useful for• All VTEPs• A specific set of VNIs• Traffic to/from NSX Edge

• Network auto-configuration based on the information discovered allows for the ability to track movement/expansion of ports, VLANs, and LAGs.

• A single stretched VLAN topology for NSX Edge VMs (to allow for the distribution of that function in an NSX network amongst the VMWare compute host versus dedicated NSX edge instances)

• Availability of learned information from NSX Controller and the ability to use this in the Switch command-line interface (CLI)• VTEPs• VMs• VNIs• Use of this knowledge in CLI show commands (“show vtep”, “show vm”, “show vni”, etc)

• Ability to gather tunnel-based statistics• VTEP health snooping and alarming• Event/Alarm integration with NSX Manager and vRealize Log Insight• VTEP-based Traffic visualization

These capabilities allow the physical and the virtual network to appear as a single network to the user, but also allow for debugging and troubleshooting to occur at each layer with the relevant information correlated between layers. In contrast to separate and disjointed over/underlay solutions, this type of “native virtualization” creates a much more cohesive network infrastructure and drastically reduces provisioning times and mean-time-to-repair.

A Hyper-converged Network Physical Model

Storage and Compute have rapidly evolved over the last decade, while network architectures have remained un-changed. Organizations have invested heavily in building hyper-converged and virtualized data centers, but legacy network architectures inhibit reaching the full potential of those investments. Since network virtualization is funda-mentally about enabling servers and storage devices to more easily communicate with each other in their respective virtual networks, adding hierarchy to the network takes away from that ability and causes a multitude of problems in east-west oriented data centers. A fundamentally new network approach is required - a hyper-converged network.

A hyper-converged network consists of three defining characteristics that set it apart from the decades-old tradition-al network.

1) A truly flat (single-tier) physical architecture that can be globally managed and dynamically provisioned

Traditional leaf-spine and other tree-oriented topologies were beautifully designed to support north/south traffic patterns. Modern, highly virtualized data centers, however, require support for massive and unpredictable east/west traffic patterns. For this, a truly flat, single tier physical network is required.



Data Center Networks have traditionally been built as “trees” or tiers, with access switches that connect to end hosts, and aggregation/distribu-tion and core switches that allow the access switches to connect to each other, and to reach the external world. Even in so-called collapsed leaf and spine architectures, the spine tier is typically a multi-stage tree of switching chips within a single chassis.

Trees provide a number of disadvantages in east-west oriented data centers. The simplest way to think about it is this – if traffic needs to go from server to server, and those servers are not physically connected to the same access switch, then that traffic must go “north” to the aggrega-tion/distribution tier and back “south” to the appropriate access switch to

ultimately get “east or west”. As mentioned previously, many times the aggregation/distribution switch is also a multi-stage tree within the chassis, so this process may be repeated several times.

A truly flat physical network is architected around a hierarchy-free model. In Plexxi’s case, access switches connect directly to other access switches by means of a high-density optical fabric we call the “LightRail”. This means that servers that need to communicate directly to others servers or storage devices can do so without intermediate switch-ing layers.

2) A truly scale-out network model

Traditional leaf-spine topologies required over-provisioning of spine capacity, typically in the form of multiple, large and expensive chassis systems. Even with the initial over-provisioning, the scale of these leaf-spine deployments was limited by port capacity and cabling complexity. In a truly scale-out network, capacity can be added only when need-ed and in small, consumable increments. The addition of new switches in a scale-out model should increase fabric capacity and resiliency as well as introduce more data path options.

The virtual network concept is similar to the virtual server concept in that the networks are divided by user or user group (or application / application group) so they can more effectively leverage the available hardware resources. In order to do that, the hardware resources need to be scaled out versus up. In this context, scaling out means adding more physical east-west paths as you add more virtual networks. In a Plexxi network, each access switch also comes with a large number of east/west fabric paths that connect to other switches in the networks.

When leveraged with NSX, these east/west paths can be dynamically and incrementally mapped to specific VNIs that represent groupings of devices that require specific connectivity. In contrast, in a tree-based network, new virtual networks would all get aggregated across a set of “uplinks” that would need to continually be upgraded to support new bandwidth requirements. In the server analogy, this would mean as you added more VMs to a system you would need to continually replace servers with higher speed CPUs, which very much becomes a limiting factor.

3) An application workload-defined orchestration model

Traditional leaf-spine topologies are only “next-hop” aware, and forward traffic based on L2 and L3 protocols. Propri-etary multi-path options are only available from the leaf to the spine (north/south) and balancing traffic across these multi-path links is based only on a simple random hashing algorithm that is not tied to the workload in any way. In an applica-tion workload-defined orchestration model, it all starts with the workload. Workflow integration between network and virtualization domains enables the binding (Affinity) between workloads and logical network topology. This orchestration



ensures workloads have the best, lowest latency and highest bandwidth path through the fabric in real time.

Ultimately the network provides interconnection of workload resources such as compute, storage, and network services; that is, the network serves the needs of the workloads. Current network infrastructure acts as its own isolated silo, creating an operational domain for the network itself. Plexxi’s goal is to make the network appear as a workload resource in all respects. This means that as workloads are created, orchestrated, and managed through advanced systems such as VMware, the network should just follow along. In addition, the network should provide data and context to these systems relevant to the interconnection task. For example, instead of just providing network packet counters at the port level, the network

should provide counters that are relevant to the workload. Similarly, error messaging and fault conditions should be made relevant to specific workloads. This type of integrations puts more power in the hands of the SDDC administrator to man-age and operate large-scale environments without unnecessary operational domain silos.

Automating OrchestrationThe software-defined data center is driven by the notion that resources can be easily orchestrated – or put into play to support a given workload. VMware’s vRealize Orchestrator solution is an extremely sophisticated and valuable set of tools to help data center operators automate the various workflows required to manage a data center. In this section we will look at how the behavior of the network can be automated for specific workload needs leveraging vRealize Orchestrator integration with Plexxi Control.

Single, Global View of the Network with AbstractionsAs a network solution, one of the key distinguishing factors of Plexxi’s technology is that the entire network is rep-resented globally from a central controller, with a set of high-level abstractions. These abstractions serve as building blocks that can be easily leveraged with tools such as VMWare’s vRealize Orchestrator to include the network into the automated workflows.

The main benefit of a Plexxi network is the ability to identify groups of endpoints (what are called “Affinity Groups”) and to define relationships within or between those groups that express a specific type of network need or constraint between the groups or the elements within a group. This allows data center operators to easily describe specific appli-cation workload needs to ensure that the network solves for those needs when distributing traffic or when organizing the physical topology.

This is shown in the simple screenshot below. In this example, there are 2 Affinity Groups (represented by the orange circles, labeled “TestVM4” and “MACA”). These groups each contain one endpoint each: TestVM2 is part of MACA and TestVM4 is a part of TestVM4.



In a software-defined data center, these endpoints represent workload components (e.g. virtual machines, storage LUNs, etc.), and the group-ings represent the workloads logically as something the network can recognize and provide specific treatment around.

From an orchestration perspective, it is important that these unique capabilities of the Plexxi network be easily driven from the same point where other orchestration activities are happening. In this case, we assume it is vRealize Orches-tration, which provides the ability to create simple or complex Run Book automation.In order to provide this type of automation, the system being automated needs to provide a number of primitive constructs that can be assembled to produce an overall action. These constructs typically have four component ac-tions – create, read, update, and delete (“CRUD”). In the case of Plexxi, we provide the following high-level CRUD constructs:

• Affinity• Hypervisor• VM• Configure-Ports• Configure-VLAN• Configure-LAG• Configure-Routing

Within each high-level control, comes a number of related capabilities. In Plexxi’s case, this looks like this:• CRUD-Affinity

• List, e.g. (groupA, [list of elements], [uni/bi-direction], [attribute], <optional><groupB>, [list of elements])• Resulting actions:

• Create groupA• Then add the list of elements (by name)• Specify the direction• Add an attribute• Optionally a second group and element list

• CRUD-Hypervisor• (name, [list of adapters], <optional> [list of vms])

• CRUD-VM• (name, [list of adapters])

• CRUD-Configure-Ports• (switch, port, state: enable/disable, spd:10/1)

• CRUD-Configure-VLAN• (switch, port, list of VLANs)



• CRUD-Configure-LAG• (list of [switch, port])

• CRUD-Configure-Routing• (VLAN, network, mask)

In addition to these higher-level constructs, some lower-level functions are also exposed:

Affinities and Fitting• AffinityGroup

• DomainAffinityGroup• EndPointAffinityGroup

• AffinityLink• UserDefinedAffinityAttribute

• IsbTypeUserDefinedAffinityAttribute• LmhsTypeUserDefinedAffinityAttribute

• FittingManager – (signal to run a “Fit” of the workload constraints and render a topology)Network Elements

• ComputeResource• PhysicalComputerResource• PhysicalMachine• StorageServer• VirtualizationHost• VirtualMachine• HypervVirtualMachine• KvmVirtualMachine• VmwareVirtualMachine

• NetworkInterface• PhysicalNetworkInterface• PhysicalPort• VirtualNetworkInterface

Network Configuration• LinkAggregationGroup• VlanGroup• VirtualRouter

Miscellaneous• PlexxiUser

• PlexxiInternalUser• PlexxiLdapUser

In addition the following objects are exposed read-only:• PlexxiRing

• PlexxiSwitch• SwitchFabricInPort

• LldpLearnedNetworkDevice• AttachedNetworkDevice• Alarm• Job• PlexxiSwitchStatistics



!"#$$$#%&'#()*+,-./&0123456789::2#;5<Nashua, NH 03062

!!!"#$%&&'"()*

Plexxi also has the industry exclusive ability to distribute and replicate market data information streams at Layer 1, reducing the per device latency by orders of magnitude. Instead of hundreds of nanoseconds latency per switch through a traditional or even purpose built low latency Ethernet switch, a network topology made up of Plexxi switches adds less than 25 nanoseconds of latency per switch as the data moves through the fabric.

Plexxi creates a single optically meshed network spanning metro and global footprints, allowing data to be replicated and distributed anywhere across the fabric with the only additional latency coming from propagation delay.

Specialized hardware inside each Plexxi switch allows 10GbE based market data feeds to be copied or replicated, without incurring the latency penalty of an Ethernet switching chip. Replication is done at Layer 1, therefore no packet copy time is created. This means that all participating switch ports receive the replicated data with the exact same latency, whether this stream is unicast or multicast.

Plexxi Market Data DistributionConnected via a highly reliable and scalable fabric, a Plexxi Market Data Distribution network creates the world’s first converged solution that delivers network services for scale-out applications as well as ultra low latency data distribution. Combined with the seamless Data Center Interconnect (DCI) capabilities of an optical fabric, Plexxi provides an industry leading end-to-end solution.

To Learn MoreContact your Plexxi sales

representative, or contact us using theinformation below.

Page 2 of 2© Plexxi, Inc. 2015

With the ability to expose these CRUD actions on specific constructs within vRealize Orchestrator, it becomes possible to completely auto-mate how workloads are given specific treatment in the network. For example, a data center operator could quite easily create run books for deploying new workloads on a network, having different run books for test, development, and production workloads. Or key mission critical workloads can be defined in a separate run book that create specific Af-finity policies that express workload constraints such as latency, priority access to bandwidth, or isolation across the network from other work-loads. There are many permutations of automated workflows that can be assembled with these primitives to achieve full network automation.

Phase 3 BenefitsBuilding a network that is “native virtualization” ready is a critical future-proofing task when considering a modern data center. Native virtualization provides a much closer integration of the virtual network and the physical network rather than viewing one as an overlay and the other as an underlay. The approach allows customers to create a single operational domain that unifies the virtual and the physical, rather than maintaining both as separate entities. Adding run-book automation allows customers to create highly predictable and executable automation capabilities to rapidly decrease the time for new services to be deployed and drastically reduces error rates in setting up the infrastructure to match the needs of the workloads.

SummaryAt Plexxi, we believe the time has come for disruptive innovation in the network. Our next-generation networks are optimized to unlock the true potential of 3rd Platform applications, enabling you to operate at the speed of business. Plexxi is building a simply better network for the next era of IT. Plexxi takes a system approach to networking, al-lowing you to achieve the full potential of your data and applications through a consolidated single-tier network and converged application workflow. Plexxi’s products and integrated ecosystem solutions create an optimized application environment for agile data centers, scale-out applications like Big Data, and distributed cloud environments. Based on the needs of individual data and application workloads, Plexxi networks dynamically change the network topology in real time, intelligently forwarding traffic and delivering needed network capacity.

For customers building clouds based on VMware’s market leading technology, a fully agile infrastructure is needed. This agility can be achieved with a fully integrated network that leverages and enhances the agility of VMware. A Plexxi network without NSX integration delivers an easy-to-scale-out network fabric that can be fully operationalized from the VMware cloud management tool. When you add NSX integration, the native network virtualization pre-serves the agility expected by the user, down to the physical network and creates a simplified engineering and architec-ture experience that saves time, money and reduces errors to create a fully optimized network.

This is, simply a better network for VMware.

www.plexxi.com 100 Innovative Way, Suite 3322Nashua, NH 03062

+1.888.630.PLEX (7539)[email protected]

The information contained herein is subject to change without notice. Plexxi, the Plexxi logo, LightRail, and Flexx Ports are trademarks of Plexxi Inc.. Other company, product or service names may be trademarks or service marks of their respective owners.

plexxi and vmware overview · white paper plexxi and vmware integration capabilities plexxi and...

Documents