ieee 1588 ptp on cisco nexus 3100 platform and 9000 … 1588 ptp on cisco nexus 3100 platform and...
TRANSCRIPT
© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 1 of 30
White Paper
IEEE 1588 PTP on Cisco Nexus 3100 Platform and 9000 Series
Switches
© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 2 of 30
Contents
What You Will Learn ................................................................................................................................................ 3
Introduction .............................................................................................................................................................. 3 The Challenge ....................................................................................................................................................... 3 The Solution .......................................................................................................................................................... 4
PTP Concepts .......................................................................................................................................................... 4 PTP Clock Types .................................................................................................................................................. 4 PTP Topology ....................................................................................................................................................... 4 PTP Algorithm ....................................................................................................................................................... 5 Hardware and Software Timestamping ................................................................................................................. 7
End-to-End Data Center PTP Deployment with Cisco Nexus 3100 Platform and 9000 Series .......................... 8 Topology and Components ................................................................................................................................... 8
Grandmaster Clock .......................................................................................................................................... 9
PTP-Enabled Cisco Nexus 3100 Platform and 9000 Series Data Center Switches ......................................... 9
Nexus 3100 Platform and 9000 Series PTP Architecture ................................................................................... 11 Cisco Nexus 3100 Platform and 9000 Series PTP Packet Walk ......................................................................... 12
Networking Configuration and Best Practices with Cisco Nexus 3100 Platform and 9000 Series ................. 13 Cisco Nexus 3100 Platform and 9000 Series PTP Configuration ....................................................................... 13
Mandatory Configuration ................................................................................................................................ 13
Optional Configuration .................................................................................................................................... 13
Cisco Nexus 3100 Platform and 9000 Series PTP Configuration Validation ....................................................... 14
Data Center PTP Performance with Cisco Nexus 3100 Platform and 9000 Series ........................................... 16 PTP Performance Measurement Definitions and Concepts ................................................................................ 16 Cisco Nexus 3100 Platform and 9000 Series Performance Measurement Methodology and Equipment ........... 18 Cisco Nexus 3164Q PTP Performance ............................................................................................................... 19 Cisco Nexus 9396PX PTP Performance ............................................................................................................. 20 Cisco Nexus 9332PQ PTP Performance ............................................................................................................ 21 Cisco Nexus 9504 with X9636PQ Line Card PTP Performance ......................................................................... 22
Cisco Nexus 9000 Series ERSPAN and PTP Timestamping .............................................................................. 24 Concept of ERSPAN ........................................................................................................................................... 24 ERSPAN Support on the Cisco Nexus 9000 Series ............................................................................................ 26 ERSPAN Packet Format on Cisco Nexus 9000 Series ....................................................................................... 26 ERSPAN with Time-Stamping Configuration on Cisco Nexus 9000 Series ........................................................ 28 ERSPAN Granularity and Marker Packet ............................................................................................................ 29
Conclusion ............................................................................................................................................................. 30
For More Information ............................................................................................................................................. 30
© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 3 of 30
What You Will Learn
This document describes how to enable a highly accurate timing solution that can provide sub-microsecond
accuracy for today’s data center network and financial trading applications by using IEEE 1588-2008 Precision
Time Protocol (PTP) Version 2 (PTPv2) on the Cisco Nexus® 3100 platform switches and Cisco Nexus 9000 Series
Switches. PTP is a distributed timing synchronization protocol with nanosecond accuracy for packet networks.
This document explains the challenges that today’s network and applications are facing and why PTP is needed to
provide sub-microsecond accuracy and how the protocol works. It explains PTP concepts and compares hardware
and software timestamping. It also presents the PTP features that the Cisco Nexus 3100 platform and 9000 Series
support, including PTP timestamping in Encapsulated Remote SPAN (ERSPAN). It provides PTP best practices
using the Cisco Nexus 3100 platform and 9000 Series. The document also describes the PTP accuracy that the
Cisco Nexus 3100 platform and 9000 Series can achieve and explains how the measurements are performed.
Introduction
Accurate and precise timing information is critical to today’s data center networks and financial trading applications.
Network and system administrators need the visibility to see exactly what is happening on the network and when
each event occurs. Application developers and administrators need to correlate various event logs with processes
and applications in a very large and complex computing environment. Compliance and digital forensics also require
that every data transaction be precisely timestamped. A fundamental requirement for today’s data network is a
reliable, accurate, and deployable time-synchronization protocol so that accurate timing information can be
provided to all the relevant elements of the data communications network, including routers, switches, servers, and
applications.
The Challenge
Traditionally, Network Time Protocol (NTP) has been used to provide millisecond-level timing in packet-based
networks. However, millisecond accuracy is no longer adequate because of the reasons mentioned in the previous
section.
Today, to gain visibility into exactly what happens in each process and in the server and switch, organizations need
a timing synchronization protocol that can provide microsecond-level details across the whole network.
The Global Positioning System (GPS) can provide accuracy of +/-100 nanoseconds (ns), but it needs a dedicated
medium to distribute the signal to the end user, which means that every device in the network needs either a BNC
with IRIG-B or some other serial interface to receive the GPS timing information from a separate network. This
requirement makes GPS impossible to deploy in a data center network, in which even the smallest server farm has
hundreds or thousands of servers, as well as routers, switches, and other network elements that don’t have
dedicated special timing protocol interfaces. Practically, organizations need a packet-based solution that is precise
and easy to implement and manage.
© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 4 of 30
The Solution
IEEE 1588-2008 PTPv2 is a timing solution for today’s data center and financial applications. It provides the
following benefits:
● Spatially localized systems with options for larger systems
● Packet-based timing distribution and synchronization
● Nanosecond to sub-microsecond accuracy
● Easy management and maintenance, with few administrative operations
● Capability to manage redundant and fault-tolerant systems
● Low-cost and low-resource-use solution that works well for both high-end and low-end devices
PTP’s accuracy comes from the hardware support for PTP in the switch and server network interface cards (NICs).
It allows the protocol to accurately compensate for message delays and variation across the network. This
hardware support for PTP enables the protocol to achieve nanosecond accuracy.
PTP Concepts
This section presents some of the main concepts of PTP.
PTP Clock Types
The main PTP clock types are summarized here.
● Grandmaster clock (GM): A grandmaster clock is the highest-ranking clock within its PTP domain and is the
primary reference source for all other PTP elements.
● Slave clock: A slave clock receives the time information from a master clock by synchronizing itself with the
master clock. It does not redistribute the time to another clock. In the data center, servers are typically PTP
slave clocks.
● Ordinary clock: An ordinary clock is a PTP clock with a single PTP port. It can be a master clock
(grandmaster) or a slave clock.
● Boundary clock (BC): A boundary clock is the intermediary device between a PTP grandmaster and its PTP
slave clients. It has multiple PTP ports in a domain and maintains the time scale used in the domain.
Different ports on the boundary clock can be master ports or slave ports. The boundary clock terminates the
PTP flow, recovers the clock and timestamp, and regenerates the PTP flow. A slave port recovers the clock
and master ports to regenerate the PTP packets.
Transparent clock (TC): A transparent clock measures the time needed for a PTP event message to transit
the device and then compensates for the packet delay.
PTP Topology
The Best Master Clock Algorithm (BMCA) is used to select the master clock on each link, and it ultimately selects
the grandmaster clock for the whole PTP domain. It runs locally on each port of the ordinary and boundary clocks
to compare the local data sets with the received data from Announce messages to select the best clock on the link.
Figure 1 shows an example of a PTP topology in a data center.
© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 5 of 30
Figure 1. PTP Topology in a Data Center
In an IEEE 1588-2008 PTPv2 network, the master-slave hierarchical clock topology needs to be established in a
PTP domain before clock synchronization occurs. This tree-like topology is similar to the spanning-tree topology,
and the grandmaster clock is the most accurate clock in this hierarchical system, so every PTP slave clock
synchronizes with it. In the PTP network, PTP-enabled ports of the ordinary and boundary clocks examine the
contents of all PTP Announce messages received on the port, and then each port runs an independent PTP-state
machine to determine the port status. Using BMCA, Announce messages, and the data sets associated with the
ordinary or boundary clock, the PTP port can be determined to be in one of the following three states:
● Master: The port is the source of time on the path served by the port. A clock having a master port becomes
the master clock (not the grandmaster) for its downstream PTP nodes.
● Slave: The port synchronizes with the device on the path on the port that is in the master state.
● Passive: The port is not the master on the path, nor does it synchronize with a master. This state prevents
timing loops at the PTP level.
Figure 1 shows that when a network has multiple master clocks - for example, because a new grandmaster clock is
added to the system - eventually only one is selected as the grandmaster clock, and it becomes the root of the
master-slave topology. The port on boundary clock 1 connected to master clock 2 will transit to a passive state,
and no master-slave relationship will be established between those two clocks. The dashed line in the figure
indicates that the master-slave relationship between two boundary clocks is not formed, and that one of the ports is
in a passive state as determined by the port-state machine
PTP Algorithm
At a high-level, PTP exchanges packets that contain a timestamp that represents the current time to which the
clock of the receiving device needs to be adjusted. For example, assume that the PTP source sends a PTP
message advertising a time of 1:00:00 p.m. However, this message takes time to reach its destination. For
example, if the PTP packet takes 1 second to reach its source, it will be 1:00:01 p.m. when the source receives a
PTP packet indicating that the time is 1:00:00 p.m. How does PTP compensate for the network latency?
The synchronization is achieved through a series of messages exchanged between master and slave clocks, as
shown in Figure 2.
1. The master clock sends a Sync message. The time that the Sync message leaves the master is timestamped
as t1, which can be embedded in the Sync message itself (one-step operation) or sent in the Follow_Up
message (two-step operation).
2. The slave receives the Sync message; t2 is the time that the slave receives the Sync message.
© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 6 of 30
3. The slave sends the Delay_Req message, which is timestamped as t3 when it leaves the slave and
timestamped as t4 when the master receives it.
4. The master responds with a Delay_Resp message that contains time-stamp t4.
Figure 2. Clock Synchronization Process
In the example in Figure 3, the master-time value is initially 100 seconds, and the slave-time value is 80 seconds.
The slave time needs to be adjusted to match the master time.
Figure 3. PTP Message Exchange Example
© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 7 of 30
The clock offset (the difference between the master and slave clocks) is calculated as follows:
Offset = t2 - t1 - meanPathDelay
The link delay from the master clock to the slave clock is t2 - t1.
The link delay from the slave clock to the master clock is t4 - t3.
IEEE 1588-2008 assumes that the path delay between the master and slave clocks is symmetrical, so the mean
path delay is calculated as follows:
meanPathDelay = ((t2 - t1) + (t4 - t3)) / 2
You can use this formula to calculate the offset between the master and slave clocks in the preceding example:
Mean Path Delay = ((t2 - t1) + (t4 - t3)) / 2
= (-18 + 22) / 2
= 2
Offset = t2 - t1 - Mean Path Delay
= 82 - 100 - 2
= -20
So in this case, the slave clock is 20 seconds behind the master clock and will adjust its clock to +20 seconds.
A PTP device that implements this algorithm is known as a two-step clock because the time information is provided
by Sync and Follow_Up messages.
Another version of the algorithm, the one-step operation, sends the timestamp in the Sync message instead of
using a separate Follow_Up message. The one-step PTP can be more precise, but can be more complicated to
implement in hardware. This complexity mainly derives from the need to update the timestamp in real time and to
modify the checksum.
The advantage of one-step PTP is that it prevents increased load on both the link and the CPU.
Hardware and Software Timestamping
The objective of PTP implementation on a switch is to provide PTP timing signals to the connected servers so that
the system clocks can be synchronized accurately.
In a software PTP implementation, the CPU timestamps the packets. This solution is easier to design and
implement and is also cost effective compared to hardware timestamping. But it is less precise: packets need to go
to the CPU to be timestamped, and the CPU is running the whole operating system, which includes many
processes in addition to the PTP process. Even if the PTP process is given the highest priority, software PTP
cannot be as precise as hardware PTP.
In a hardware PTP implementation, the capture and insertion of timestamps in PTP packets is performed by the
application-specific integrated circuit (ASIC) for a switch or by the NIC for a server. This process ideally is
performed at the physical (PHY) level. For ingress, the best time for this process is right after packets have been
received on the wire. For egress, the best time is right before packets are serialized on the wire, and after packets
leave the buffer, so that the timestamp precision is not affected by potential buffer use due to congestion or speed
mismatch.
© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 8 of 30
End-to-End Data Center PTP Deployment with Cisco Nexus 3100 Platform and 9000 Series
Topology and Components
Because PTP is based on Ethernet, it can run over the existing data center architecture and doesn’t usually require
a redesign of the data center. Figure 4 shows a typical leaf-and-spine data center design with PTP components.
Figure 4. Data Center PTP Deployment
The components for this end-to-end data center PTP deployment include the grandmaster clock and Cisco Nexus
switches.
© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 9 of 30
Grandmaster Clock
The grandmaster clock typically is dedicated, specific equipment. This device usually is connected to GPS, which
provides an accurate time source input. The master clock then generates PTP packets over the network.
The Symmetricom TimeProvider 5000 is an example of a PTP grandmaster clock (Figure 5).
Figure 5. Symmetricom TimeProvider 5000 PTP Grandmaster Clock
PTP-Enabled Cisco Nexus 3100 Platform and 9000 Series Data Center Switches
The Cisco Nexus 3100 platform switches are compact 1- and 2-rack-unit (1RU and 2RU) switches for top-of-rack
(ToR) data center deployments or end-of-row (EoR) deployments. They offer wire-rate Layer 2 and 3 switching with
the data center-class Cisco® NX-OS Software operating system.
The Cisco Nexus 9000 Series Switches consist of Cisco Nexus 9500 platform modular switches and Cisco Nexus
9300 platform fixed-configuration switches. Cisco Nexus 9000 Series Switches can run in Cisco Application Centric
Infrastructure (ACI) mode or Cisco NX-OS mode.
In ACI mode, Cisco Nexus 9000 Series Switches, when used in combination with the Cisco Application Policy
Infrastructure Controller (APIC), provide an application-centric infrastructure.
In NX-OS mode, Cisco Nexus 9000 Series Switches function as classic switches. Equipped with enhanced Cisco
NX-OS Software as the operating system, Cisco Nexus 9000 Series Switches provide network connectivity through
traditional means but with exceptional performance and enhanced network resiliency and programmatic
automation functions.
This document focuses on PTP on the Cisco Nexus 9000 Series in NX-OS mode.
As of Cisco NX-OS Release 7.0(3)I7(1), PTP is supported on all Nexus 3100 and 9000 Series Switches and Line
Cards, except the N9K-X9408PC-CFP2 line card, the N9K-M4PC-CFP2 uplink module, and the 9500-R chassis.
Figure 6. Cisco Nexus 3164Q
© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 10 of 30
Figure 7. Cisco Nexus 9396PX
Figure 8. Cisco Nexus 9332PQ
Figure 9. Cisco Nexus 9500 Chassis with X9636PQ Line Card
© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 11 of 30
PTP is supported on the Cisco Nexus 3100 platform and 9000 Series Ethernet switches starting from Cisco NX-OS
Release 7.0(3)I1(1). It includes the following features:
● IEEE 1588-2008 PTPv2 standard
● Two-step boundary clock
● User Datagram Protocol (UDP) over IPv4 multicast using multicast address 224.0.1.129 as defined in the
IEEE 1588 standard
● Hardware-assisted PTP implementation
● Effective handling of network congestion by processing and forwarding PTP messages with higher priority
by default; no need for another step to configure additional quality of service (QoS)
● Support for ERSPAN with ERSPAN Header Version 3, which includes a timestamp
● Support for PTP on both Layer 2 and Layer 3 interfaces
Nexus 3100 Platform and 9000 Series PTP Architecture
The Cisco Nexus 3100 platform and 9000 Series support PTP operations with hardware assistance. The
forwarding ASIC timestamps the PTP packets in both the ingress and egress directions in hardware.
Figure 10 shows the Cisco Nexus 3100 platform and 9000 Series PTP architecture.
Figure 10. Cisco Nexus 3100 Platform and 9000 Series PTP Architecture
© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 12 of 30
On the Cisco Nexus 9300 platform switches, the PTP hardware clock is in the Application Leaf Engine (ALE) and
ALE-2 ASICs. On the Cisco Nexus 3100 platform switches, it is in a separate field-programmable gate array
(FPGA). On the Cisco Nexus 9500 platform modular switches, it is in an FGPA present on the supervisor card that
all line cards use.
Cisco Nexus 3100 Platform and 9000 Series PTP Packet Walk
Two scenarios are possible: one in which a front-panel port of the Cisco Nexus 3100 platform or 9000 Series
Switch is a PTP slave, and one in which the port is a PTP master.
When a front-panel port of the Cisco Nexus 3100 platform or 9000 Series Switch is a PTP slave, the switch is
receiving time information from a PTP master and correcting its own clock. The packet flow is as follows:
1. The switch receives a PTP Sync message from the PTP master on the slave port. The forwarding ASIC has a
clock synchronized to the PTP hardware clock. The forwarding ASIC records the time of arrival of the Sync
packet. It adds this timestamp to the packet using an internal header and forwards it to the CPU. The PTP
software process running on the CPU receives the packet and stores the timestamp. The timestamp is t2 in
Figure 2.
2. The PTP master sends a PTP Follow_Up message to the switch. The message contains the t1 timestamp that
was recorded by the master when it sent the PTP Sync packet. The Cisco Nexus 3100 platform or 9000 Series
Switch PTP software process receives this packet and stores its timestamp. The PTP software process now
has t1 and t2.
3. The Cisco Nexus 3100 platform or 9000 Series Switch sends a Delay_Req message to the master. The
timestamp of departure is recorded when the Delay_Req messages leaves the forwarding ASIC. The PTP
software process stores this timestamp, which is t3. The PTP software process now has t1, t2, and t3.
4. The PTP master records when it received the Delay_Req message. This timestamp is t4. It sends a
Delay_Resp message containing t4 to the Cisco Nexus 3100 platform or 9000 Series Switch slave. The PTP
software process now has t1, t2, t3, and t4.
5. The PTP software on the Cisco Nexus 3100 platform or 9000 Series Switch computes the offset from t1, t2, t3,
and t4 based on the formula described in the “PTP Algorithm” section earlier in this document. The PTP
hardware clock of the Cisco Nexus 3100 platform or 9000 Series Switch is adjusted accordingly. This
adjustment is why a device operating as a PTP boundary clock performs what is called clock recovery.
This concludes the PTP message exchange on a PTP slave port of the Cisco Nexus 3100 platform or 9000 Series
Switch. This message exchange is continuously performed between the Cisco Nexus 3100 platform or 9000 Series
Switch slave port and the master port to which it is connected, so the clock of the switch is always synchronized
with the master.
When a front-panel port of the Cisco Nexus 3100 platform or 9000 Series Switch is a PTP master, the switch is
providing the time to a slave device that is synchronized to it. The packet flow is as follows:
1. The Cisco Nexus 3100 platform or 9000 Series Switch sends a PTP Sync message to the slave. Using its PTP
hardware clock, it records the timestamp t1, which indicates when the packet left the forwarding ASIC.
2. The t1 timestamp is sent by the Cisco Nexus 3100 platform or 9000 Series Switch to the slave in a PTP
Follow_Up packet.
3. The Cisco Nexus 3100 platform or 9000 Series Switch receives a PTP Delay_Req message from the slave. It
records the timestamp t4, which indicates when the packet was received.
© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 13 of 30
4. The Cisco Nexus 3100 platform or 9000 Series Switch sends t4 to the slave in a PTP Delay_Resp packet.
5. The slave device now can adjust its own clock based on the timestamps it received from the master and
recorded. At no point does the Cisco Nexus 3100 platform or 9000 Series Switch adjust its own clock, because
this PTP message exchange takes place on a master port.
The Cisco Nexus 3100 platform or 9000 Series Switch PTP implementation is not affected by congestion or
buffering. The switch adds the timestamps at the PHY level, when the packet is ready to be serialized and after it
has been buffered. This approach results in very high PTP precision, as discussed in the section “Data Center PTP
Performance with Cisco Nexus 3100 Platform and 9000 Series” later in this document.
Networking Configuration and Best Practices with Cisco Nexus 3100 Platform
and 9000 Series
Cisco Nexus 3100 Platform and 9000 Series PTP Configuration
This section describes the most important PTP configuration commands on the Cisco Nexus 3100 platform and
9000 Series. A comprehensive guide to configuration can be found in the system management configuration guide
available at http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/7-
x/system_management/configuration/guide/b_Cisco_Nexus_9000_Series_NX-
OS_System_Management_Configuration_Guide_7x/b_Cisco_Nexus_9000_Series_NX-
OS_System_Management_Configuration_Guide_7x_chapter_0100.html.
Mandatory Configuration
The commands shown here need to be present for PTP to operate on the Cisco Nexus 3100 platform and 9000
Series.
Global Configuration Commands
switch(config)# feature ptp
This command enables PTP on the switch.
switch(config)# ptp source <ip>
This command specifies the source IP address for the PTP packets generated by the switch.
Interface Configuration Commands
switch(config)# interface Ethernet slot/port
switch(config-if)# ptp
These commands enable PTP on a port. The Cisco Nexus 3100 platform or 9000 Series Switch is a boundary
clock, so it has both master and slave ports. There is no configuration difference between a master port and a
slave port. They are both configured with the ptp option, and BMCA determines whether the port is a PTP slave or
master port.
Optional Configuration
The commands listed here are optional.
switch(config)# clock protocol ptp
This command configures the switch so that it uses PTP to update the system calendar. This configuration keeps
the switch’s clock synchronized with PTP. If you don’t enable this command, the switch still will propagate the PTP
clock on its master ports. However, the time source will be the Cisco Nexus local clock.
© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 14 of 30
switch(config)# interface ethernet slot/port
switch(config-if)# sync interval value
These commands configure the PTP Sync message packet rate on an interface. If a value is not specified, the
default is 0.
The PTP logSyncInterval (sync interval) value represents the number of PTP Sync message packets sent on the
interface per second (Table 1).
Table 1. PTP logSyncInterval Values
logSyncInterval Value Number of PTP Sync Messages per Second
-3 8
-2 4
-1 2
0 1
1 1 every 2 seconds
You should leave the default sync interval value set to 0 unless you have a specific need for greater accuracy.
Cisco Nexus 3100 Platform and 9000 Series PTP Configuration Validation
The commands shown here can be used to validate PTP on the Cisco Nexus 3100 platform and 9000 Series
Switches.
switch# show clock
This command can be used to verify that the switch clock is synchronized with the grandmaster clock. You cannot
verify precise accuracy with this command-line interface (CLI) command, but you can at least verify that the time of
day matches the grandmaster, if clock protocol ptp was configured.
switch# show ptp clock
This command displays the properties of the local clock, including the clock identity.
This is an example of show ptp clock output:
switch# show ptp clock
PTP Device Type: Boundary clock
Clock Identity : 88:f0:31:ff:fe:2a:fa:e1
Clock Domain: 0
Number of PTP ports: 3
Priority1 : 196
Priority2 : 255
Clock Quality:
Class : 248
Accuracy : 254
Offset (log variance) : 65535
Offset From Master : 0
Mean Path Delay : 0
Steps removed : 0
Local clock time:Wed Jan 28 17:56:19 2015
9396px-nd1#
© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 15 of 30
switch# show ptp parent
This command displays the properties of the PTP parent. It is useful for verifying the parent clock’s identity.
This is an example of show ptp parent output:
switch# show ptp parent
PTP PARENT PROPERTIES
Parent Clock:
Parent Clock Identity: 00:14:01:ff:fe:00:00:01
Parent Port Number: 1
Observed Parent Offset (log variance): N/A
Observed Parent Clock Phase Change Rate: N/A
Grandmaster Clock:
Grandmaster Clock Identity: 00:14:01:ff:fe:00:00:01
Grandmaster Clock Quality:
Class: 6
Accuracy: 35
Offset (log variance): 0
Priority1: 1
Priority2: 1
switch# show ptp brief
This command displays the PTP state of all interfaces. A PTP port can be in one of the following three states:
● Master: The port is the source of time on the path served by the port.
● Slave: The port synchronizes with the device on the path on the port that is in the master state.
● Disabled: PTP is not enabled on this port.
● Passive: The port is not the master on the path, nor does it synchronize with a master.
Because the Cisco Nexus 3100 platform or 9000 Series Switch is a PTP boundary clock and supports only one
PTP domain, the switch can have only one slave port. If the switch has one slave port already, a second port
connected to a second grandmaster will be in the passive state. When the first grandmaster or the first slave port
fails, BMCA will move the previously passive port to a slave state. With this process, grandmaster redundancy can
be achieved.
This is an example of show ptp brief output:
switch# sh ptp brief
PTP port status
-----------------------
Port State
© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 16 of 30
------- --------------
Eth1/1 Slave
Eth1/2 Master
Eth1/3 Master
Eth1/4 Master
Eth1/5 Master
Eth1/6 Master
...
Switch#
switch# show ptp corrections
This CLI command displays the last few PTP corrections.
Here is an example of show ptp corrections output:
PTP past corrections
---------------------------------------------------------
Slave Port SUP Time Correction(ns) MeanPath Delay(ns)
---------- ----------------------------------------------
Eth1/46 Mon Dec 23 09:52:11 2013 48581 -1 293
Eth1/46 Mon Dec 23 09:52:12 2013 49318 3 297
Eth1/46 Mon Dec 23 09:52:13 2013 49193 -8 297
Eth1/46 Mon Dec 23 09:52:14 2013 49208 12 298
Eth1/46 Mon Dec 23 09:52:15 2013 48625 -3 298
Eth1/46 Mon Dec 23 09:52:16 2013 47607 -13 295
Eth1/46 Mon Dec 23 09:52:17 2013 49091 0 295
Eth1/46 Mon Dec 23 09:52:18 2013 47961 2 295
Eth1/46 Mon Dec 23 09:52:19 2013 48005 -1 295
Eth1/46 Mon Dec 23 09:52:20 2013 48350 0 296
Eth1/46 Mon Dec 23 09:52:21 2013 48507 -5 292
Eth1/46 Mon Dec 23 09:52:22 2013 48105 2 292
Eth1/46 Mon Dec 23 09:52:23 2013 48188 12 301
Eth1/46 Mon Dec 23 09:52:24 2013 48021 6 301
Eth1/46 Mon Dec 23 09:52:25 2013 48239 -12 296
Data Center PTP Performance with Cisco Nexus 3100 Platform and 9000 Series
PTP Performance Measurement Definitions and Concepts
This section describes essential definitions and concepts used to characterize PTP performance.
The offset is the difference between the times on two clocks. In the following test results, it is the measure of how
accurately the slave synchronizes with the master clock. This measurement indicates the amount of inaccuracy
brought by the Cisco Nexus 3100 platform or 9000 Series Switch as a boundary clock. Less is better.
© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 17 of 30
The mean path delay is the average time taken by PTP frames to travel between the master and the slave. This
measurement does not indicate the performance or accuracy of the switch or servers. A small mean path delay is
useful for obtaining baseline results. A large mean path delay with a lot of jitter is representative of a complex data
center with buffering and latency spikes, control protocols running, a high rate of traffic, etc. This measure can be
interesting in a real-world PTP performance test in a processing-intensive environment.
The standard deviation indicates the amount of variation or dispersion from the average. A low standard deviation
indicates that the data points tend to be very close to the mean (also called the expected value); a high standard
deviation indicates that the data points are spread out over a large range of values.
In the rest of this document, the following definitions are used to characterize PTP performance:
● Average offset
● Offset standard deviation
● Minimum and maximum offset peak
● Mean path delay
These values are measured by external tools using the same stable clock reference.
The average offset is often close to 0 because the positive and negative clock offsets equalize each other when
calculating an average.
Minimum and maximum offset peaks are useful; however, those values alone do not indicate how often those
peaks were reached. Therefore, the offset standard deviation is important in addition to the average offset and
offset peaks. Another way to see the offset standard deviation is through the jitter.
These four data points together with the mean path delay provide a very good view of a device’s PTP performance.
Figure 11 is an example of an offset graph showing the average offset, offset standard deviation, and minimum and
maximum offset peaks. The vertical axis is the offset value, and the horizontal axis is the set of PTP clients.
© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 18 of 30
Figure 11. Sample Graph of Offset Values
Cisco Nexus 3100 Platform and 9000 Series Performance Measurement Methodology
and Equipment
The test described here uses a traffic generator from Spirent for PTP message generation, master and slave
simulation, and background traffic. It uses a Spirent 9RU chassis (SPT-9000) with HyperMetrics CV 8 x 10 Gigabit
Ethernet Enhanced Small Form Factor (SFP+) cards. Spirent Release 4.33.0086 is used.
The Spirent 9RU chassis is connected to a Symmetricom TimeProvider 5000 grandmaster clock. Spirent receives
the clock from a timing interface. The Symmetricom TimeProvider 5000 grandmaster gets its time source from an
embedded GPS receiver.
The Spirent 9RU chassis has one master port connected to the Cisco Nexus switch being tested. The rest of ports
on the Spirent 9RU chassis are PTP slave ports getting the clock from the Cisco Nexus PTP master ports, which
redistribute the clock from the Spirent 9RU chassis. Therefore, Spirent can compute the offset between the clock
sent from its master port and the clock received on its slave ports. This result indicates the accuracy of the Cisco
Nexus switch under test.
© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 19 of 30
Figure 12 shows the performance measurement methodology.
Figure 12. Performance Measurement Methodology
The Cisco Nexus 3100 platform and 9000 Series use Cisco NX-OS Release 7.0(3)I1(1).
For all the tests, the PTP configuration is the default on the Cisco Nexus switches. In particular, the PTP sync
interval parameter is left at the default value of 0.
Cisco Nexus 3164Q PTP Performance
The Cisco Nexus 3164Q is configured in 4 x 10-Gbps breakout mode, with 44 physical 10-Gbps ports connected to
Spirent. This configuration results in 43 PTP slaves. In addition, 212 emulated PTP slaves are configured in
Spirent. This configuration amounts to a total of 255 PTP slaves to which the Cisco Nexus 3164Q needs to provide
the time.
© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 20 of 30
Figure 13 shows the Cisco Nexus 3164Q performance topology with Spirent.
Figure 13. Cisco Nexus 3164Q Performance Topology with Spirent
After 6 hours, the results as averages of all ports are:
● Average offset: 4 ns
● Offset standard deviation: 50 ns
● Minimum offset peak: -300 ns
● Maximum offset peak: 302 ns
● Average mean path delay: 110 ns
The average offset is very close to 0. The offset standard deviation of 50 ns shows that during the duration of the
test, the average offset values were clustered closely around the mean. Therefore, the Cisco Nexus 3164Q
introduces an extremely small offset in the PTP client clock synchronization process even with a larger number of
PTP slaves.
Cisco Nexus 9396PX PTP Performance
The Cisco Nexus 39396PX is configured with 44 physical 10-Gbps ports connected to Spirent. This configuration
results in 43 PTP slaves. In addition, 4 emulated PTP slaves are configured in Spirent. This configuration amounts
to a total of 47 PTP slaves to which the Cisco Nexus 9396PX needs to provide the time.
© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 21 of 30
Figure 14 shows the Cisco Nexus 9396PX performance topology with Spirent.
Figure 14. Cisco Nexus 9396PX Performance Topology with Spirent
After 6 hours, the results as averages of all ports are:
● Average offset: 2 ns
● Offset standard deviation: 50 ns
● Minimum offset peak: -200 ns
● Maximum offset peak: 217 ns
● Average mean path delay: 115 ns
Cisco Nexus 9332PQ PTP Performance
The Cisco Nexus 9332PQ is configured in 4 x 10-Gbps breakout mode, with 44 physical 10-Gbps ports connected
to Spirent. This configuration results in 43 PTP slaves. In addition, 84 emulated PTP slaves are configured in
Spirent. This configuration amounts to a total of 127 PTP slaves to which the cisco Nexus 9332PQ needs to
provide the time.
© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 22 of 30
Figure 15 shows the Cisco Nexus 9332PQ performance topology with Spirent.
Figure 15. Cisco Nexus 9332PQ Performance Topology with Spirent
After 6 hours, the results as averages of all ports are:
● Average offset: -5 ns
● Offset standard deviation: 11 ns
● Minimum offset peak: -101 ns
● Maximum offset peak: 98 ns
● Average mean path delay: 120 ns
Cisco Nexus 9504 with X9636PQ Line Card PTP Performance
The Cisco Nexus X9636PQ line card on a Cisco Nexus 9500 platform switch is configured in 4 x 10-Gbps breakout
mode, with 44 physical 10-Gbps ports connected to Spirent. This configuration results in 43 PTP slaves. In
addition, 99 emulated PTP slaves are configured in Spirent. This configuration amounts to a total of 143 PTP
slaves to which the Cisco Nexus X9636PQ line card needs to provide the time.
© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 23 of 30
Figure 16 shows the Cisco Nexus X9636PQ performance topology with Spirent. In the figure, the line card is
connected to a Cisco Nexus 9504 Switch.
Figure 16. Cisco Nexus X9636PQ Performance Topology with Spirent
After 6 hours, the results as averages of all ports are:
● Average offset: 3 ns
● Offset standard deviation: 9 ns
● Minimum offset peak: -95 ns
● Maximum offset peak: 81 ns
● Average mean path delay: 108 ns
The average offset is almost 0 ns. It always stayed close to this average (standard deviation is 9 ns). Even with a
large number of PTP slaves, the Cisco Nexus 9500 with a Cisco Nexus X9636PQ line card maintains a very high
PTP accuracy.
© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 24 of 30
Cisco Nexus 9000 Series ERSPAN and PTP Timestamping
Concept of ERSPAN
ERSPAN transports mirrored traffic over an IP network. The traffic is encapsulated at the source router and is
transferred across the network. The packet is decapsulated at the destination router and then sent to the
destination interface.
ERSPAN consists of an ERSPAN source session, routable ERSPAN generic routing encapsulation (GRE)-
encapsulated traffic, and an ERSPAN destination session. You separately configure ERSPAN source sessions and
destination sessions on different switches. Figure17 shows an ERSPAN topology.
Figure 17. ERSPAN Topology
The Cisco Nexus 9000 Series supports ERSPAN Type III, which contains timestamp information. It can be used to
calculate packet latency between different parts of the network.
Figure 18 shows a typical example of latency monitoring in a data center. The goal is to determine the latency
between switch A and switch B.
To monitor the latency, an ERSPAN source session is configured on each switch. The source port for this session
is the ingress port of the flow for which the latency is monitored. The destination IP address of the session is the IP
address of a server running a packet-capture analyzer such as Wireshark. The two switches have their clocks
synchronized using PTP, so the timestamps recorded in the ERSPAN header have the same time reference. On
the analyzer server, the packets from both switches are received (because the same ERSPAN destination IP
address was configured).
With this information, you can easily compute the delta between the ERSPAN timestamps of the packets received
to obtain the network latency.
© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 25 of 30
Figure 18 shows how to configure ERSPAN and PTP to perform latency monitoring in a typical network.
Figure 18. Latency Monitoring Using ERSPAN and PTP
The important point to note is that an ERSPAN destination session is not configured on any switch in this scenario.
An ERSPAN destination decapsulates the GRE and ERSPAN headers from the packet. Therefore, the timestamp
would be lost. Instead, Wireshark is used, and it can read the ERSPAN header and display the timestamp.
Figure 19 shows an example of a Wireshark capture of an ERSPAN packet. The timestamp is highlighted in red.
© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 26 of 30
Figure 19. Wireshark ERSPAN Capture
ERSPAN Support on the Cisco Nexus 9000 Series
Cisco Nexus 9300 platform switches equipped with ALE or ALE-2 ASICs support the ERSPAN Type III header. As
of Cisco NX-OS Release 7.0(3)I1(1), the switches are the Cisco Nexus 93128TX, 9396PX, 9396TX, 9372PX,
9372TX, and 9332PQ.
The Cisco Nexus 3100 and 9500 platforms do not support ERSPANv3 headers in spanned copy of the packets,
and therefore cannot carry a timestamp in monitored packets. On those switches, packets are encapsulated as
GRE-tunnel packets. The ERSPAN header is not added to the packet.
ERSPAN Packet Format on Cisco Nexus 9000 Series
The ERSPAN Type III packet format includes an additional 12- or 20-byte header on top of the GRE header.
Therefore, it adds a total of up to 62 bytes to the original frame length: 14 (MAC) + 20 (IPv4, or 40 for IPv6) + 8
(GRE) + 20 (ERSPAN).
The 20-byte ERSPAN header includes a 32-bit time-stamp field. The ERSPAN header starts at the 42nd
byte in the
ERSPAN packet. The time-stamp field is bytes 46 to 49 in the ERSPAN packet.
© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 27 of 30
Here is a summary of the ERSPAN Type III packet format:
ERSPAN Type III header (12 octets)
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Ver | VLAN | COS |BSO|T| Session ID |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Timestamp |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| SGT |P| FT | Hw ID |D|Gra|O|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Platform Specific SubHeader (8 octets, optional)
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Platf ID | Platform Specific Info |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Platform Specific Info |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
The composite header in the preceding format summary is immediately followed by the original mirrored frame and
then by the standard 4-octet Ethernet cyclic redundancy check (CRC).
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Original Mirrored Frame |
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| CRC |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Table 2 provides descriptions of each field in the ERSPAN Type III packet header.
Table 2. ERSPAN Type III Packet Header Fields
Field Length Definition
Header Fields in Common with ERSPAN Type II
Ver 4 ERSPAN encapsulation format version
Type II: 0x1. Type: III 0x2. Version 0x0 is obsolete.
VLAN 12 VLAN of the frame monitored by an ERSPAN source session. For the ingress monitor, this will be the original source VLAN, and for the egress monitor this will be the destination VLAN.
Cos 3 Class of service (CoS) of the monitored frame. Ingress or egress CoS value depending on the monitor type.
BSO 2 2-bit value indicating the integrity of the payload carried by ERSPAN.
00: Payload is a good frame with no error, or unknown integrity.
11: Payload is a bad frame with CRC or alignment error.
01: Payload is a short frame.
10: Payload is an oversized frame.
T 1 Indicates that encapsulated frame is truncated. Encapsulated packet exceeds the maximum transmission unit (MTU) settings for an ERSPAN source session.
Session# 10 Identification associated with each ERSPAN source session. Identifies the stream to be decapsulated at the destination.
© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 28 of 30
Field Length Definition
Fields Specific to ERSPAN Type III Header
Timestamp 32 32-bit timestamp. Wraps approximately 4 billion time units depending on the time-stamp granularity specified.
SGT 16 Security group tag of the monitored frame.
P 1 This bit indicates that the payload is a protocol frame, or Bridge Protocol Data Unit (BPDU) frame.
FrameType 5 00000 expected.
Hardware ID 6 Unique identifier of an ERSPAN engine.
Direction 1 Original frame subjected to SPAN in ingress or egress. Ingress = 0 and egress = 1.
Timestamp Granularity
2 00: Granularity = 100 ms (mandatory default)
01: Granularity = 100 ns
10: 1588 PTP 11 = 1 ns
Optional SubHeader
1 O flag indicates whether or not the optional subheader is present.
When O == 0b ERSPAN, payload starts after the O flag.
When O == 1b ERSPAN, payload starts 8 bytes after the O flag.
Supplemental Info 64 Optional information.
ERSPAN with Time-Stamping Configuration on Cisco Nexus 9000 Series
The ERSPAN configuration is described in the configuration guide available at
http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/7-
x/system_management/configuration/guide/b_Cisco_Nexus_9000_Series_NX-
OS_System_Management_Configuration_Guide_7x/b_Cisco_Nexus_9000_Series_NX-
OS_System_Management_Configuration_Guide_7x_chapter_010001.html.
The ERSPAN Type III header contains a 32-bit timestamp. This timestamp represents the time of day since
January 1, 1970. The CLI option required to enable the ERSPAN timestamp is header-type 3 in the ERSPAN
session configuration.
A sample ERSPAN configuration with header-type 3 is shown here:
switch(config)# monitor session 1 type erspan-source
switch(config-erspan-src)# vrf default
switch(config-erspan-src)# source interface Ethernet1/30 rx
switch(config-erspan-src)# destination ip 192.168.8.1
switch(config-erspan-src)# header-type 3
switch(config-erspan-src)# no shut
The ERSPAN Type III header is added to the packets only if the ERSPAN destination IP address route is resolved
through an uplink interface (that is, an interface connected to an ALE or ALE-2 ASIC). On the Cisco Nexus
93128TX, 9396PX, and 9396TX, the uplinks are all the interfaces on slot 2. On the Cisco Nexus 9372PX and
9372TX, the uplinks are e1/49 to e1/54. On the Cisco Nexus 9332PQ, the uplinks are e1/27 to e1/32.
© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 29 of 30
ERSPAN Granularity and Marker Packet
The resolution of the ERSPAN timestamp is 100 picoseconds (ps). Therefore, the 32-bit time-stamp field in the
ERSPAN header will wrap around every 0.43 second. This is not enough to represent the full time of day, which
requires 64 bits. Consequently, Cisco NX-OS offers the capability to periodically send a marker packet that
contains the full Coordinated Universal Time (UTC) timestamp. The ERSPAN timestamp and the marker packet
time value combined provide the capability to recover the full value of the ERSPAN timestamp. Cisco NX-OS
sends 10 marker packets per second.
The marker packet is carried in a UDP packet with destination port 8880. It uses the same source and destination
IP addresses as the ERSPAN session.
A sample ERSPAN configuration with marker-packet capabilities is shown here:
switch(config)# monitor session 1 type erspan-source
switch(config-erspan-src)# marker-packet
Table 3 shows the marker packet format.
Table 3. Marker Packet Format
Field Position (Byte: Bit)
Length (Bits) Definition
Align 16 The 2-byte align bit is inserted to align the rest of the packet with a 4-byte boundary. The value is 0xFF to indicate the beginning of the marker packet.
Version 4 Version number. Default is Version 1.
Type 4 The type of marker packet. Default is 0.
ssid 8 The session ID of the ERSPAN source session.
granularity 8 The granularity of the 3- bit hardware timestamp.
The value is 0x100, which represents a 100-ps granularity.
Utc_offset 8 The UTC offset between the ASIC clock and the CPU UTC clock. Default is 0; currently set to 0.
timestamp 32 The 32-bit ASIC hardware timestamp.
UTC sec 32 The upper 32 bits of the UTC timestamp from the CPU clock of the Cisco Nexus 9000 Series Switch. This value is the base for the UTC recovery of the ERSPAN header time-stamp field.
UTC usec 32 The lower 32 bits of the UTC timestamp from the CPU clock of the Cisco Nexus 9000 Series Switch.
Sequence 32 The sequence number of the marker packet.
Reserved 32 Reserved for future use.
Signature 32 A value of 0xA5A5A5A5.
The ERSPAN timestamping is performed in hardware. The timestamps are recorded when the ERSPAN-monitored
packets arrive at the ALE or ALE-2 ASIC.
Note that the PTP timestamp and the ERSPAN timestamp mean different things:
● The PTP timestamp is the timestamp used in PTP control packets that allow PTP to operate and
synchronize clocks over the network.
● The ERSPAN timestamp is the timestamp in the ERSPAN header. It is separate from the PTP timestamp.
ERSPAN timestamping can actually work without PTP enabled on the switch. In such a case, the ERSPAN
timestamp would be derived from the local oscillator of the switch and would not be synchronized with any
source. This is an acceptable solution for basic latency monitoring using just one switch because it still
shows the delta between the arrival times of packets in the switch.
© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 30 of 30
Conclusion
IEEE 1588-2008 PTPv2 provides a reliable, highly accurate distributed time synchronization solution for data
centers that require nanosecond or sub-microsecond accuracy. PTP is easy to implement with very little
administrative effort and can tolerate network and clock failure with built-in fault-tolerant mechanisms.
The Cisco Nexus 3100 platform and 9000 Series Switches have an extremely precise PTP implementation. With a
large number of simultaneous PTP slaves, the Cisco Nexus 3100 platform and 9000 Series PTP accuracy stays
under 50 nanoseconds.
This solution can be combined with a software PTP implementation on the server to achieve microsecond
accuracy. Hardware PTP on the server can also be used if sub-microsecond accuracy is required.
PTP in combination with ERSPAN on the Cisco Nexus 9000 Series provides a very powerful and easy-to-use
solution to perform latency monitoring in the data center.
For More Information
For more information about IEEE 1588-2008 Precision Time Protocol Version 2, see the IEEE Standard for a
Precision Clock Synchronization Protocol for Networked Measurement and Control Systems at
http://ieeexplore.ieee.org/xpl/mostRecentIssue.jsp?reload=true&punumber=4579757.
For more information about the Cisco Nexus 9000 Series Switches, see the detailed product information at the
product homepage at http://www.cisco.com/c/en/us/products/switches/nexus-9000-series-switches/index.html.
The configuration guides for PTP and ERSPAN on the Cisco Nexus 3100 platform and 9000 Series can be found
at:
● http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/7-
x/system_management/configuration/guide/b_Cisco_Nexus_9000_Series_NX-
OS_System_Management_Configuration_Guide_7x/b_Cisco_Nexus_9000_Series_NX-
OS_System_Management_Configuration_Guide_7x_chapter_0100.html
● http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/7-
x/system_management/configuration/guide/b_Cisco_Nexus_9000_Series_NX-
OS_System_Management_Configuration_Guide_7x/b_Cisco_Nexus_9000_Series_NX-
OS_System_Management_Configuration_Guide_7x_chapter_010001.html
Printed in USA C11-733921-02 09/17