traffic monitoring and engineering for...
TRANSCRIPT
Traffic Monitoring and Engineering for UCSBRKCOM-2004
Steve McQuerry, CCIE# 6108
Technical Marketing Engineer
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
Agenda
UCS Networking Overview Network Statistics in UCSM Understanding Collection Policies Hotspot Detection Engineering to Avoid Hotspots
3
UCS Networking Overview
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
System Components: High-level Overview
5
• 61XX• 62XXUP
• 2104XP• 2204XP• 2208XP
• Cisco (VIC1280, VIC1225)• 3rd party
FABRIC INTERCONNECT
CHASSIS IO MODULE (FEX)
INTERFACE CARDS
• 2232PP
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
UCS Networking Overview
Fabric Interconnects–(10GE ports)
Chassis–Up to 8 half width blades or 4 full width blades
Fabric Extender (FEX or I/O Module)–Host to uplink traffic engineering–Up to 160Gb Flexible bandwidth allocation
Adapters–Virtualized adapter for single OS and hypervisor systems–Dual connected
Blade or Stand Alone Server
6
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
UCS Networking Overview
1st Generation Hardware
7
vpc
Mezz
Fabric A Fabric B
Mezz Mezz
FEX AFEX A FEX BFEX B
ni
c ni
c hb
a hb
a ni
c ni
c hb
a hb
a ni
c ni
c hb
a hb
a
SAN A SAN B
Port-Channel Port-Channel
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
UCS 2104 (1st Gen FEX) — Server to Fabric Pinning
8
Fabric Interconnect1 link
2 links
4 links
Server slots pinned to uplink
Uplink: slots 1,2,3,4,5,6,7,8
Uplink 1: slots 1,3,5,7Uplink 2: slots 2,4,6,8
Uplink 1: slots 1,5Uplink 2: slots 2,6Uplink 3: slots 3,7Uplink 4: slots 4,8
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
NIF
NIF
NIF
FEX
FEX
Fabric Interconnect
Fabric Interconnect
FEX
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
UCS Networking Overview
1st Generation Hardware
9
Fabric A Fabric B
Mezz
IOM AIOM A IOM BIOM B
ni
c ni
c hb
a hb
a
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
UCS Networking Overview
2nd Generation Hardware
10
Mezz
Fabric A Fabric B
Mezz Mezz
IOM AIOM A IOM BIOM B
ni
c ni
c hb
a hb
a ni
c ni
c hb
a hb
a ni
c ni
c hb
a hb
a
SAN A SAN B
vpc
Port-Channel Port-Channel
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
UCS 2204 (2nd Gen FEX) — Sever to Fabric Pinning
11
Fabric Interconnect1 link
2 links
4 links
Server slots pinned to uplink
Uplink: slots 1,2,3,4,5,6,7,8
Uplink 1: slots 1,3,5,7Uplink 2: slots 2,4,6,8
Uplink 1: slots 1,5Uplink 2: slots 2,6Uplink 3: slots 3,7Uplink 4: slots 4,8
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
NIF
NIF
NIF
FEX
FEX
Fabric Interconnect
Fabric Interconnect
FEX
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
UCS 2204 (2nd Gen FEX) — Sever to Fabric Pinning
12
4 links
Server slots channeled acrossall uplinks
Uplink 1: slots 1-8Uplink 2: slots 1-8Uplink 3: slots 1-8Uplink 4: slots 1-8
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
NIF
FEX
Fabric Interconnect
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
UCS 2208 (2nd Gen FEX) — Server to Fabric Pinning
13
Fabric Interconnect1 link
2 links
4 links
Server slots pinned to uplink
Uplink: slots 1,2,3,4,5,6,7,8
Uplink 1: slots 1,3,5,7Uplink 2: slots 2,4,6,8
Uplink 1: slots 1,5Uplink 2: slots 2,6Uplink 3: slots 3,7Uplink 4: slots 4,8
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
NIF
NIF
NIF
FEX
FEX
Fabric Interconnect
Fabric Interconnect
FEX
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
UCS 2208 (2nd Gen FEX) — Server to Fabric PinningServer slots pinned to uplink
8 linksUplink 1: slot 1Uplink 2: slot 2Uplink 3: slot 3Uplink 4: slot 4Uplink 5: slot 5Uplink 6: slot 6Uplink 7: slot 7Uplink 8: slot 8
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
NIF
FEX
Fabric Interconnect
8 links Uplink 1: slots 1-8Uplink 2: slots 1-8Uplink 3: slots 1-8Uplink 4: slots 1-8Uplink 5: slots 1-8Uplink 6: slots 1-8Uplink 7: slots 1-8Uplink 8: slots 1-8
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
NIF
FEX
Fabric Interconnect
Server slots channeled acrossall uplinks
14
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
UCS Networking Overview
2nd Generation Fabric Interconnect
15
Fabric A Fabric B
Mezz
IOM AIOM A IOM BIOM B
ni
c ni
c hb
a hb
a
Network Statistics in UCSM
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
UCS CLI
DMEUCSM AG blade
Events {xml}
HTTP{xml}
DB
AGn
Polling/notification
XML API
CIM Object Translation
IBM Tivoli
Syslog
AG FI AG chassis
SNMPCIM-XML
HPNNM HPOM Misc. SNMP
Smash
Events{SNMPTraps}
Polls{SNMPGET}
UCSM Core
17
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
5 most recent stats history records are contained by the current stats instance.
Stats history are subclasses of the stats.
Root
topSyssys
equipmentChassissys/chassis-1
equipmentChassisStatssys/chassis-1/stats
equipmentChassisStatsHistsys/chassis-1/stats/1
... equipmentChassisStatsHistsys/chassis-1/stats/5
Statistics are contained by monitored object
Statistics are instances of statsCurr
Statistics dn is often deducible.
UCSM Statistics
18
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
Stats Collection Policy
A Systemwide Set of Stats Collection Policies (One per Application Domain) Allow Configuration of Collection and Reporting Intervals.Domains: adapter, chassis, host, port, serverWhen a Collection Interval Is Changed, the reportingInterval Is Restarted collectionInterval: frequency that endpoints will send stats updates to DME reportingInterval: frequency that DME will report stats updates to external
collectors and update stats history
19
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
Network Statistics in UCSM
Network statistics are collected by UCSM from the NX-OS software in the Fabric Interconnects. These are counters that are available for networking components Because of NIV technology the Fabric has visibility to the Cloud (LAN/SAN
uplinks), the IOM and the server NIC.
20
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
Network Statistics in UCSM
Access Statistics through:– LAN or SAN tab (port-group) – Devices tab ( server ports, network uplink, storage ports, and mezz ports,)– Server Tab (vNIC)
21
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
Network Statistics in UCSM
FI to LAN Network Uplink (Cloud)
22
Fabric A Fabric B
SAN A
vpc
Port-Channel Port-Channel
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
Network Statistics in UCSM
Port Channel is the Aggregate of all interfaces in the channel Statistic for the channel are the sum of the statistics of the members Individual member statistics are also visible in the system Network usage is measured against TX Total bytes and RX Total bytes
23
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
Network Statistics in UCSMFI to Uplink Port Channel Statistics
24
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
Network Statistics in UCSMFI to Uplink Individual Port Statistics
25
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
Network Statistics in UCSMFI to SAN Port Statistics
26
Fabric A Fabric B
SAN A SAN B
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
Network Statistics in UCSM
A FC Port Channel is the Aggregate of all interfaces in the channel
Statistic for the channel are the sum of the statistics of the members
Individual member statistics are also visible in the system
FC usage is measured against Bytes RX and Bytes TX
FI to SAN Uplink (Cloud) Port Statistics
27
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
Network Statistics in UCSMFI to SAN Port Statistics
28
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
Network Statistics in UCSMFI to IOM (Internal LAN)
29
Fabric A Fabric B
IOM AIOM A IOM BIOM B
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
Network Statistics in UCSMFI to IOM (Internal LAN)
30
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
UCS Networking OverviewServer to IOM
31
Mezz
ni
c ni
c hb
a hb
a
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
Network Statistics in UCSMServer to IOM vNIC Port Statistics
32
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
Using SNMP tools to Monitor Traffic
The network statistics in UCSM are pulled from the NX-OS statistics for the networking components.
It is possible to monitor the traffic for the networking components using an external SNMP collection tool such as MRTG.
The advantage is long term analysis, however most tools have no alerting capabilities.
33
Understanding Collection Policies
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
Understanding Collection Policies
A collection Policy consist of a collection interval and a reporting interval The collection policy is set under the admin tab in Stats Management ->
Collection Policies -> Collection Policy name A unique policy can be set for, Adapters, Chassis, FEX, Port, Server, and Host* Not all policies involve network components
35
*Host is an unused policy in UCSM at present
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
Understanding Collection Policies
36
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
Understanding Collection Policies
The collection interval is how often the system will query a device for statistics.
The default collection interval is 60 seconds
The more frequent the interval the more granular the data. We will use 30 seconds.
The timing of the collection interval is important because it will be used in BW calculations for hotspot detection
Collection Interval
37
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
Understanding Collection Policies
The reporting interval is internal to UCSM and determines how often UCSM will store data from the collection interval.
This data is stored in tables and the last 5 reporting intervals are available for inspection in the system
Reporting interval data is used to calculate minimum, maximum and average values shown in the statics view.
Reporting Interval
38
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
Understanding Collection Policies
Select the Policy you want to change. Make selections and press the save changes button
39
Hotspot Detection
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
Hotspot Areas
41
Fabric Interconnects to LAN/SAN
FEX to Host
FEX to FI
Mezz
Fabric A Fabric B
Mezz Mezz
FEX AFEX A FEX BFEX B
ni
c ni
c hb
a hb
a ni
c ni
c hb
a hb
a ni
c ni
c hb
a hb
a
SAN A SAN B
Port-Channel Port-Channel
There are three potential hotspot locations for UCS network connectivity.
1.) Send and Receive between Fabric Interconnect and LAN/SAN
2.) Send and Receive between FEX and Fabric Interconnect
3.) Send and Receive between Host and FEX
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
Hotspot Detection
To identify hot spots we will use Threshold policies in conjunction with collection policies to alert as we pass thresholds.
A threshold is calculated by measuring a statistic against a policy.
The policy measures change against a user defined normal value and turns on the alert between a users set high/low threshold and turns off the alert below the user set low threshold.
Threshold Policies
42
High - Up
Low - Up
Normal
High - Up
Low - Up
Normal
High - Up
Low - Up
Normal
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
Hotspot Detection
The statistic we use to calculate bandwidth is the delta in bytes.
This should be measured for both TX and RX
This delta is calculated in bytes changed over a period of time defined by the collection interval, for example 30 seconds
Calculating BW Threshold Limits for an Element
43
‟… if standardizing on one size was a good thing, standardizing on several sizes would be even better.”
• Radia Perlman• Interconnections: Bridges and Routers
44
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
Kilo, Mega, etc
What is a Kilobit?– “Once upon a time… Computer professionals noticed that 210 was very nearly equal to 1000
and started using the SI prefix “kilo” to mean 1024.– The rest of the world uses the term Kilo in the SI format to mean 1000.– The result is that no one seems to know what kilo and mega mean.
The IEEE standards board decided that the IEEE standards will use the conventional, internationally adopted, definitions of the SI prefixes. Therefore Mega will mean 1,000,000 and Kilo will mean 1,000.*
In Telecom and by extension networking Kilo means 1000 and NOT 1024
The SNMP reported speed for a 10Gig link (on any platform) will be reported by SNMP is 1,250,000,000 – Bytes (MaxBytes[10.29.148.6_436207616]: 1250000000)
*Sources: NIST and IEEE
IEEE Networking Standards for Prefix Values
45
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
Hotspot Detection
For Ethernet the BW of a single link is 10Gbps
First we determine our desired threshold for example 8Gbps
We need to calculate the expected change in bytes over the collection interval.
To calculate divide the desired BW by 8 bits per byte and then multiply by the collection interval by the time to get the expected delta in bytes for our collection period.
Example 8 Gbps over 30 seconds = 30,000,000,000 bytes– 8Gbps / 8bits per byte = 1,000,000,000 bytes per second – 1,000,000,000 bytes per second * 30 seconds = 30,000,000,000 bytes
Calculating BW Threshold Limits for an Element
46
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
Hotspot DetectionThreshold Calculations
47
Speed in Gbps Percentage of BW Conversion to BytesDelta expected over 30 second collection interval
10 100% 1,250,000,000 37,500,000,000
9 90% 1,125,000,000 33,750,000,000
8.5 85% 1,062,500,000 31,875,000,000
8 80% 1,000,000,000 30,000,000,000
7.5 75% 937,500,000 28,125,000,000
7 70% 875,000,000 26,250,000,000
6.5 65% 812,500,000 24,375,000,000
6 60% 750,000,000 22,500,000,000
5 50% 625,000,000 18,750,000,000
4 40% 500,000,000 15,000,000,000
3 30% 375,000,000 11,250,000,000
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
Hotspot DetectionThreshold Alerts
48
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
Hotspot Detection
Threshold policies can be configured for the following:– Internal LAN – IOM to FI– LAN Cloud – FI to Upstream Ethernet switches– SAN Cloud – FI to Upstream SAN switches– Server – Between the server NIC and the IOM
Threshold Policies Placement
49
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
Hotspot Detection
Navigate to admin->stats management and expand the fabric.
Select thr-policy-default and create a threshold class
Choose Ether Tx Stats from the stat class and click next.
Threshold Policies Internal LAN
50
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
Hotspot DetectionThreshold Policies Internal LAN
51
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
Hotspot DetectionThreshold Policies Internal LAN
52
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
Hotspot Detection
Click the add button and select Ether Tx Stats Total Bytes Delta as the property type enter 0.0 as the normal value
Select the Alarm triggers you want to get and enter your values and click OK
Click finish to be returned to the policy.
Click the classes tab to see your policy.
Threshold Policies
53
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
Hotspot DetectionThreshold Policies Internal LAN
54
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
Hotspot Detection
You will need to add another class for RX traffic.
Click the + bottom to the right and repeat the steps from the previous policy choosing Eter Rx Stats as the stats class this time
Click save changes once you have completed the steps
Threshold Policies
55
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
Hotspot DetectionThreshold Policies
56
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
Hotspot Detection
For Uplinks you can repeat this process for the LAN cloud.
For SAN use a single Stats class fcstats and create a definition for rx and tx stas under the same stats class
Threshold Policies
57
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
Hotspot DetectionThreshold Policies
58
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
Hotspot Detection
For the vNIC port you will need to create a threshold policy to be used with a service profile.
Go the the appropriate organization level and select create threshold policy
Threshold Policies
59
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
Hotspot DetectionThreshold Policies
60
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
Hotspot Detection
Give the policy an applicable name and description and press next
Choose the vnic stats class and create a single threshold with the rx bytes delta and the tx bytes delta.
Threshold Policies
61
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
Hotspot DetectionThreshold Policies
62
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
Hotspot Detection
Apply the threshold policy to the Service profile of the servers you want to monitor
Threshold Policies
63
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
Hotspot Detection
When a server reaches a threshold you will receive an alert on UCSM.
This will exist while threshold is exceeded once it drops below the definition the alert will disappear
Alerts show as system faults as defined by the threshold policy
UCSM will send a TRAP when an alert is generated.
The TRAP will be sent regardless of what level of alert has been set.
Threshold Policies
64
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
Hotspot DetectionThreshold Policies
65
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
Hotspot Detection
When a threshold policy has been crossed UCSM will send out an SNMP Trap alert
This assumes that UCSM has been configured for SNMP operation and the Trap Receiver has been defined
This feature was added in UCSM version 2.x.x
66
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
Hotspot Detection
UCSM has a limited amount of space to store statistics and can only provide the last 5 instances of collection intervals. It provides a good tool for monitoring thresholds, but does not provide a long term monitoring solution.
Tools like MRTG are great for Long Term analysis of BW usage, but may not provide the required alerting and usually require a great deal of customization to be usable
UCS Central version (TBD) will provide tools for both alerting as well as long term analysis of traffic within systems it manages.
Long Term Monitoring vs. Point in Time
67
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
Automating Threshold Policies
Creating threshold policies for the systems require time and effort. Isn’t there some way to push standard configurations to the system?
GoUCS is an automation tool available from the Cisco Developer Network (CDN) that allows users to build XML configuration scripts that can be sent to UCS.
PowerTool can also be used to automate this process
http://developer.cisco.com/
GoUCS and CDN
69
Engineering to Avoid Hotspots
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
Traffic Engineering
71
FI-1
Class-A Class-B Class-C
vNIC-1
vNIC-2
vNIC-3
FEX-1
FI-2
Blade-2, VIC-1vNIC-1
vNIC-2
vNIC-3
FEX-2
VIC with 3 vNICs
2 Fabric Extenders in chassis, each with 1 link to the FI.
2 FI, both with 1 connection to each FEX
Blade-1, VIC-1
vNICs can be pinned to a specific FI when created (with configurable failover to other switch)
Depending on requirements, vNICs could be pinned to one interconnect or distributed evenly
vNICs in System Class C pinned to one interconnectvNICs in System Class C pinned to one interconnect
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
Controlling Pinning in Profile
72
From the each server a fabric interconnect can be chosen to balance the traffic
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
Traffic Engineering
73
FI-1
Class-A Class-B Class-C
vNIC-1
vNIC-2
vNIC-3
FEX-1
FI-2
Blade-2, VIC-1vNIC-1
vNIC-2
vNIC-3
FEX-2
Blade-1, VIC-1
vNICs can be pinned to a specific FI when created (with configurable failover to other switch)
Depending on requirements, vNICs could be pinned to one interconnect or distributed evenly
vNICs in System Class C distributed across interconnectsvNICs in System Class C distributed across interconnects
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
QoS Architecture
No packet drops within the array– Largest buffers are on switch and
host memory, so congestion pushed to edges
– Priority Flow Control (PFC) used to ensure packet drops are at vNIC or Switch
All traffic in a CA system belongs to 1 of 6 System Classes
– Four are user configurable while the other two are for FCoE and standard Ethernet
QoS parameters can be configured at a per system class level, or a per vNIClevel.
74
Compute ChassisCompute Chassis
erx86 Computer x86 ComputerX
IIx8x8x8x8
B
MGMT
SS
B
X X X X X
CC
AAGG GG
GG GG
SAN
GG
RR
AAGG
GG GG
GG
RR
GG
PPMM PP
SANLAN
FabricSwitchFabricSwitch
FabricSwitchFabricSwitch
FabricExtender
FabricExtender
FabricExtenderFabricExtender
Compute Blade(Half slot)
Compute Blade(Half slot)
AdapterAdapter
Compute Blade(Full slot)
Compute Blade(Full slot)
AdapterAdapterAdapterAdapter
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
System Buffering/Queuing
75
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
User Configuration
Globally for each System Class
Globally for each System Class
COS value for packets in this class
COS value for packets in this class
Drop/No-drop behavior
Drop/No-drop behavior
Strict PriorityStrict Priority
Bandwidth/WeightBandwidth/Weight
Users configure QoS parameters at two levelsUsers configure QoS parameters at two levels
Class Name FC Bronze
COSValue 3 0
Drop/No-Drop No-Drop Drop
Strict Priority No No
Bandwidth/Weight 20% 30%
Example
76
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
User Configuration
77
For each vNIC (Egress properties)
For each vNIC (Egress properties)
System Class for traffic from this vNIC
System Class for traffic from this vNIC
Rate limit (Mbps)Rate limit (Mbps)
Burst Size (Kbytes)Burst Size (Kbytes)
Users configure QoS parameters at two levelsUsers configure QoS parameters at two levels
vNIC1 vNIC2 vNIC3
Class FC FC Bronze
Rate 4000 4000 5000
Burst 300 400 100
Example: Logical Server A
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
User Configuration – Example
78
Class Name FC Gold Ethernet BE
COS Value 3 1 0
Drop/No-Drop No-Drop Drop Drop
Strict Priority No No No
Bandwidth/Weight 1 (20%) 3 (60%) 1 (20%)
vNIC1 vNIC2 vNIC3
Class FC FC Eth. BE
Rate 4000 4000 5000
Burst 300 400 100
Logical Server A
Global System Class Definitions
vNIC1 vNIC2
Class Gold Eth. BE
Rate 600 4000
Burst 100 300
Logical Server B
FC TrafficHigh PriorityEthernet
Best EffortEthernet
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
QoS Tools
79
Priority Flow ControlPriority Flow Control
• Enables lossless Fabrics for each class of service
• PAUSE sent per virtual lane when buffers limit exceeded
Transmit Queues Ethernet Link Receive Buffers
EightVirtualLanes
OneOne OneOneTwoTwo TwoTwo
ThreeThree ThreeThreeFourFour FourFourFiveFive FiveFive
SevenSeven SevenSevenEightEight EightEight
SixSix SixSixSTOP PAUSE
COS based Bandwidth ManagementCOS based Bandwidth Management
• Enables Intelligent sharing of bandwidth between traffic classes control of bandwidth
•802.1Qaz Enhanced Transmission
10 GE Link Realized Traffic Utilization
3G/s HPC Traffic3G/s
2G/s
3G/sStorage Traffic3G/s
3G/s
LAN Traffic4G/s
5G/s3G/s
t1 t2 t3
Offered Traffic
t1 t2 t3
3G/s 3G/s
3G/s 3G/s 3G/s
2G/s
3G/s 4G/s 6G/s
Among the tools used are aggregate shapers at the vNICs (VIC Adapter), ETS, Policers at the switch for each vNIC.
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
QoS Configuration in UCSMEnable QoS Classes in UCSM
80
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
QoS Configuration in UCSMCreate a QoS Policy
81
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
Applying QoS to a PolicyApply Policy to Adapter
82
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
Summary
UCSM is designed for optimized traffic flow Stats Management and Threshold policies allow you to monitor traffic levels QoS and Traffic engineering tools allow you to manage potential bottlenecks in
UCS
83
© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public
Maximize your Cisco Live experience with your free Cisco Live 365 account. Download session PDFs, view sessions on-demand and participate in live activities throughout the year. Click the Enter Cisco Live 365 button in your Cisco Live portal to log in.
Complete Your Online Session Evaluation
Give us your feedback and you could win fabulous prizes. Winners announced daily. Receive 20 Cisco Daily Challenge
points for each session evaluation you complete. Complete your session evaluation
online now through either the mobile app or internet kiosk stations.
84