cisco support community expert series webcast
TRANSCRIPT
Avinash Shukla, Manas Kumar Nath, and Sandeep Kumar
Customer Support Engineers
May 12, 2015
Cisco Unified Computing System (UCS) Mini Architecture, Troubleshooting and Upgrade
Cisco Support Community
Expert Series Webcast
Upcoming Expert Series Webcast
June 23, 2015
To help you more effectively deal with the inevitable troubleshooting scenarios you’ll encounter, this webcast equips you with a 7-step troubleshooting model that allows you to confidently approach and deal with any troubleshooting situation. You’ll also be equipped with efficiency tips for using Cisco IOS “show” commands, and the webcast concludes with a group exercise where you’ll help troubleshoot an issue being experienced on live gear.
Troubleshooting with Speed and
Confidence for Network Engineers
https://supportforums.cisco.com/event/12486661/expert-webcast-troubleshooting-speed-and-confidence-network-
engineers
Kevin Wallace
ACS 5 Installation, configuration, and
troubleshooting, configuration and troubleshooting
with Cisco Expert, Javier Henderson
Now through May 22nd
Ask the Expert Events – Active
Join the discussion for these Ask The Expert Events:
http://bit.ly/events-webinar
High CPU on IOS. Learn and ask questions about
high CPU condition that you might be facing your
environment and troubleshooting the same with the
tools and techniques available within the platform
with Cisco expert Vinit Jain
https://supportforums.cisco.com/expert-corner/top-contributors
Participate in Live
Interactive
Technical Events
and much more
http://bit.ly/1jlI93B
Become an Event Top Contributor
Rate Content Now your ratings on documents, videos, and blogs count give points to the authors!!!
So, when you contribute and receive ratings you now get the points in your profile.
Help us to recognize the quality content in the community and make your searches easier. Rate content in the community.
https://supportforums.cisco.com/blog/154746
Encourage and acknowledge people who generously share
their time and expertise
Cisco Support Community Expert Series Webcast
• Today’s first featured expert is Manas Nath, a customer support engineer from the Server Virtualization team at the Cisco Technical Assistance Center in Bangalore, India.
• Ask your questions now in the Q&A window
Manas Nath Customer Support Engineer
Cisco TAC, Server Virtualization
Tuesday, May 12, 2015
Cisco UCS Mini Architecture,
Troubleshooting and Upgrade
Cisco Support Community Expert Series Webcast
• Today’s second featured expert is Sandeep Kumar, a customer support engineer from the Server Virtualization team at the Cisco Technical Assistance Center in Bangalore, India.
• Ask your questions now in the Q&A window
Sandeep Kumar Customer Support Engineer
Cisco TAC, Server Virtualization
Tuesday, May 12, 2015
Cisco UCS Mini Architecture,
Troubleshooting and Upgrade
Expert Series Webcast: Cisco Unified Computing System (UCS) Mini Architecture, Troubleshooting and Upgrade
Technical Expert, Question Managers
Avinash Shukla
Now through May 22nd, 2015
Ask the Expert Event following the Webcast
Join the discussion for these Ask The Expert Events:
http://bit.ly/events-webinar
Avinash Shukla, Manas Kumar Nath, and Sandeep
Kumar
will be continuing the discussion in an Ask the Expert
event. So if you have more questions, please visit the
Knowledge Center on the Cisco Support Community
https://supportforums.cisco.com/discussion/12500391/ask -expert-cisco-unified-computing-system-ucs-mini-architecture-troubleshooting
If you would like a copy of the presentation slides, click the PDF file link in the chat box on the right or go to:
https://supportforums.cisco.com/document/12500371/expert-webcast-slides-cisco-ucs-mini-architecture-troubleshooting-and-upgrade
Thank You For Joining Us Today!
Submit Your Questions Now! Use the Q & A panel to submit your questions
and the panel of experts will respond.
Please take a moment to
complete the survey at
the end of the webcast
Avinash Shukla, Manas Kumar Nath, and Sandeep Kumar
Customer Support Engineers
May 12, 2015
Cisco Support Community Expert Series Webcast
Cisco Unified Computing System (UCS) Mini Architecture, Troubleshooting and Upgrade
Polling Question 1
Have you performed a Unified Computing System Upgrade?
1. I have done using auto install.
2. I have performed using manual upgrade method.
3. I have never performed a UCS upgrade
4. I want to perform UCS upgrade myself.
• Platform Overview & Software Architecture
• System Functionality
• Feature Specific Details (Licensing, Scalability Ports, USB, Firmware)
• Licensing
• UCS Mini Firmware & Upgrade
• Upgrade Demo
• FC Direct Connect & BFS
• Troubleshooting
• Backup and Restore
• Tips and Tricks
Agenda
Platform Overview & Software Architecture
Interconnect A
IO Modules
Servers
IO Modules
Classic UCS Architecture
Interconnect B
UCS Mini Platform Components
New Chassis UCS-5108-AC2 & UCS-5108-DC2
In-Chassis Fabric Interconnects UCS-FI-M-6324 (With Version 3.0)
Rack Servers C220/240 M3 SB + UCSC-PCIE-CSC-02 (Support for direct attach only; can be connected to scalability port via octopus cable)
Appliance
• In-Chassis Fabric Interconnect – UCS-FI-M-6324.
• Integrated FI and IOM
• 4 x 10G SFP+ (Unified Ports)
• 1 x 40G QSFP+ in 4x10G breakout (Eth/FCoE only).
• 16 x 10G Backplane KR links connected to blades.
• 1x1G Backplane Cluster Link
• 1x10G Backplane Peer Link for data
• USB Port
• 1G Management Port
• Console Port
UCS-Mini Fabric Interconnect
Management
Network port
USB
4 x SFP+
1 x QSFP +
Console Port
Supported Port Roles
Unified Ports - 4 x SFP+ Supported Roles:
Eth/FC Uplink, Server, Appliance, FCoE uplink & FCoE storage
Scalability Port: 1 x QSFP+ (Eth/FCoE only) in 4x10G Breakout at FCS Supported Roles:
Server, Appliance, FCoE Storage (i.e. only south-facing roles)
Management Use Cases
SMB / Branch Office
Cisco ISR Branch Office Router
UCS-Mini
Local Admin
SMB / Branch Office
Cisco ISR Branch Office Router
UCS-Mini
Remote Admin
HQ / Main Office Main
Office Router
Internet (VPN) / WAN
Mini-UCS
SMB / Branch Office
Cisco ISR Branch Office Router
UCS-Mini
Remote Admin
HQ / Main Office
Main Office Router
Internet (VPN) / WAN
UCS-Central (VM)
Autonomous (Local Admin) Autonomous (Remote Admin)
Centralized Management
• Baseline: All features supported in (UCS equivalent) 2.2.1 are supported in 3.0 with some exceptions.
• Unsupported features:
• Private VLANs
• Port Security
• New features:
• Dual Line PSU
• USB support
• QSFP+ (Scalability Port) Breakout Support
• Licensing Support for QSFP+
• HTML5 GUI (Java GUI will continue to be supported)
Supported Features
Software Components
SPRINGFIELD
MEZZ_CARD
PROCESSING_NODE
IO_MODULE IO_MODULE
SPRINGFIELD
BA
SAM
CMC
IBMC
BIOS
Palo/
Menlo
Host
Agent
DCOS
Utilities
CMC
DCOS
UCSM
Switch/N
XOS
Switch/N
XOS
FI-IOM FI-IOM
Adaptor
Firmware
FI-IOM FI-IOM
Fabric-Interconnect UCS-FI-M-6324
Blade
UCSM + Switch Software (NXOS) + CMC(Chassis Management Controller) are co-located on FI
UCSM Central 1.2 supports 100 UCS-Mini domains.
UCS-Central
• 3 Racks directly connected to the primary chassis via Scalability Port
Equipment – Main Topology View
FI-IOM Management Model
• UCS-FI-M-6324 is both an FI and an IO Module • IOM – Blade Connectivity & Chassis Mgmt
Functionality
• FI – Switching Functionality
• Management Model captures both aspects
HIF PC Chassis Backplane Ports, a.k.a Host Interface (HIF) Ports
Front Panel Ports including Scalability Port
FI-IOM Management Model – Port Mapping
NXOS Mapping: Blade 1: Eth1/6-7 Blade 2: Eth1/8-9
.. Blade 8: Eth1/20-21
NXOS Mapping:
Eth1/1-4 Scalability Port: Br-Eth1/5/1-4
NXOS Mapping: PC consists of one or more HIF ports
Internal Ports: eth 1/22 eth 1/23 eth 1/24
FI-IOM Management Model – Thermal Stats
Two new sensors added for UCS-Mini
FI-IOM Management Model – System/Port Stats
• Primary chassis discovery happens automatically upon system reboot. Chassis Discovery policy is ignored.
FI-IOM – Chassis Discovery
Acknowledge Chassis on primary chassis will retrigger chassis discovery.
FI-IOM Hybrid Display
Discovered Backplane Ports
• Background:
• Primary Heartbeat (HB) & Election Messages run over peer link (bond0).
• Secondary Heartbeat runs over chassis SEEPROM when primary HB fails.
• SEEPROM also maintains cluster, chassis, fabric & DB version info.
• Functionally same as classic UCS with the exception of following differences:
• Peer link (bond0) consists of a single 1G link (eth2) on chassis backplane.
• Only Primary Chassis SEEPROM is used for secondary heartbeat. (Racks will not be used for secondary heartbeat.)
High Availability (HA)
HA – Troubleshooting Tip
Upon reboot, this error (-36) appears transiently and should clear after chassis discovery is complete.
If the issue persists, check chassis discovery FSM. ‘Acknowledge’ Chassis may resolve
the issue.
In progress case
success case
• FI-IOM can be hot inserted or removed.
• Identical to classic UCS, during first setup, user must choose either standalone or cluster configuration.
• To fully setup a cluster configuration, user must setup both FI-IOMs. If only a single FI-IOM is setup, it is an incomplete configuration and UCSM will not be accessible.
• Once fully setup, an FI-IOM can be pulled out. This would cause the existing FI-IOM to become primary if it was secondary.
• Note: This is meant to be only for transient condition till removed FI-IOM is replaced.
Online Insertion and Removal (OIR)
OIR – Valid Operations A B
SWAP SWAP SWAP
Cluster Instance IDs match in each of these cases.
OIR – Invalid Operations
Ways to Recover (same as
classic UCS):
• Undo FI-IOM swap
• Erase config on one side and re-cluster
SWAP SWAP
OIR – Invalid Configurations - Standalone A-Standalone B-Standalone A-Cluster B-Standalone
• Standalone + Standalone case – Chassis Discovery will fail.
• Cluster + Standalone case – Chassis Discovery will fail, cluster
mismatch reported on the cluster setup. Ways to Recover (same as classic UCS):
• Enable cluster
• Erase configuration and re-cluster
• Remove one FI-IOM
• UCS-Mini [UCS-FI-M-6324] Supports
• 4 10G SFP Ports (Unified Ports)
• 1 40G Scalability Port
• Scalability Port :
• Octopus-Cable can be used to breakout the 40G Port into 4 x 10G QSFP Ports
Scalability Ports
Equipment-View
• During Chassis/FI-M Discovery,
Ports are discovered.
• 4 SFP Ports are numbered
1/1,1/2,1/3,1/4
• Breakout Scalability Ports are
number 1/5/1,1/5/2,1/5/3,1/5/4
Supported Roles
Scalability Ports Server Port
FCoE Storage Port
Appliance Port
SFP Unified Ports
Server Port
Uplink Port Fcoe Uplink Port
FCoE Storage Port
Appliance Port
Scalability Port Display
Scalability Port Inventory : CLI
Mini-B# scope fabric-interconnect b
Mini-B# show port
Scalability Ports 1/5/1,1/5/2,1/5/3,1/5/4
• 4 SFP 10G Unified Ports
• Can be configured as FC or Ethernet Ports
• Constraints :
• All FC ports must be contiguous.
• All Ethernet ports must be contiguous.
• FC Ports must come before the Ethernet ports in
• the order [1..to..4]
Unified Port Configuration
1 2 3 4
1 2 3 4
1 2 3 4
1 2 3 4
1 2 3 4
Ethernet
FC This behavior is different from UCS where Ethernet Ports come before FC Ports.
Unified Ports Configuration
Unified Port Configuration [ CLI ]
• FC Ports must be configured so that they are before Ethernet
Ports.
Error
Success
• USB can be hot inserted & removed. USB drive is auto-mounted at /mnt/usbslot1.
• USB is supported only for firmware downloads.
• Boot from USB is not supported.
• Export/Import to/from USB is not supported.
• CLI supports the ability to browse USB while GUI does not.
• Only Cisco USB key is officially supported.
USB Support
• It can take up to 1 min to detect insertion/ removal of the USB key.
• Download will fail if user attempts to download immediately after removing the USB key.
USB Support - GUI
When USB is inserted
• Local-mgmt supports all file operations including browsing on usbdrive:/
• Can download image from either side usbA or usbB.
USB Support - CLI
When USB is
not inserted.
Licensing
• Uses existing UCS License Mechanism
• Two Applicable Licenses
• ETH_PORT_ACTIVATION_PKG
• SCAL_PORT_ACTIVATION_PKG
• Includes a 120 Day Grace Period
• Upon Grace Period completion license is expired.
UCS-Mini License Scheme
• Default License (Factory-Installed)
• [ ETH_PORT_ACTIVATION_PKG]
• Cisco UCS 6324, 4 10G Unified Ports, Counted License.
• These can be used for Uplink, Server, Storage Appliance etc.
Default License
• Scalability Port : Additional License
• [ SCAL_ETH_PORT_ACTIVATION_PKG]
• Cisco UCS 6324, 1 40G Eth Port, Counted License.
• Break-out ports can be used for Server (Rack Server), Storage Appliance role
Scalability Port License
• UCS-Mini does support Scalability Port in 40G mode.
• It operates in breakout mode and provides 4x10G Breakout Ports
• SCAL_ETH_PORT_ACTIVATION_PKG gives 1 absolute port count.
• If any of the four breakout ports is configured and enabled, “1” license is used. If all the four ports are un-configured /disabled “1” license is released.
Scalability Port License Handling
Port 1/5/1 configured as Server Port
Scalability Port License : Used
• Admin License Management [SCAL_ETH_PORT_ACTIVATION_PKG]
Scalability Port License : Used
UCS Mini Firmware & Upgrade
• UCS-Mini has 5 images bundles:
• INFRA: ucs-mini-k9-bundle-infra.<version>.A.bin – new
• ucs-6324-k9-kickstart.<version>.bin
• ucs-6324-k9-system.<version>.bin
• ucs-manager-k9.<version>.bin
• BLADES: ucs-k9-bundle-b-series.<version>.B.bin – same as classic UCS
• RACKS: ucs-k9-bundle-c-series.<version>.C.bin – same as classic UCS
• CATALOG: ucs-catalog.<version>.T.bin
• UCS-Central: Release 1.2
• Version 3.0 is not available for classic UCS Infra bundle.
Firmware Management
• UCSM version 3.0 is not supported on classic UCS.
• UCSM can run on classic UCS as well as UCS-Mini.
• UCSM version pre-3.0 cannot be activated on UCS-Mini.
Firmware Management – UCSM Validation
Upgrade Process Demo
UCS Firmware Manual Upgrade Downloading the firmware bundle
Activate UCS Manager
Activate the Sub-ordinate FI
• UCS-A# show cluster state
• UCS-A# connect local-mgmt
• UCS-A (local-mgmt) # cluster {force primary | lead {a | b}}
At this step, do a manual failover to avoid control plane disruption.
Activate the Primary FI
Polling Question 2
Do you use SAN boot for your UCS servers?
1. I use local boot.
2. I use SAN boot for all UCS servers
3. I use local boot but SAN storage for my application Virtual Machines.
4. I use local storage for both boot and application Virtual Machine.
Upgrading Server Firmware
Creating a Host Firmware Package
Simple Configuration
Advanced Configuration
Assign Host Firmware Package to Service Profile
After you save Changes
After the upgrade is successful
Firmware Auto Install
Initiate the Infrastructure Firmware Upgrade
Pending User Ack
Before user ack to reboot the Primary FI
Pending Activities
After the upgrade is complete
Status after the upgrade
Host firmware upgrade using auto install
Select the Firmware Package
Select the Host Firmware Package
Firmware Package Dependencies
Impacted Endpoints
Configuration & Troubleshooting
FC Direct Connect Boot from SAN
Troubleshooting Tools
FC Direct Connect
Enable FC Zoning for each VSANs
Create Storage Connection Policy
Create HBA Initiator Groups
vHBA Initiator Groups
Boot From SAN
Add SAN Boot Target
Associate the boot policy with the Service Profile
Verify that he host can see the target
• UCS-Mini-A# connect adapter 1/3/1
• adapter 1/3/1 # connect
• No entry for terminal type "dumb";
• using dumb terminal settings.
• adapter 1/3/1 (top):1# attach-fls
• No entry for terminal type "dumb";
• using dumb terminal settings.
• adapter 1/3/1 (fls):1# lunlist
• vnic : 14 lifid: 3
• - FLOGI State : flogi est (fc_id 0x470040)
• - PLOGI Sessions
• - WWNN 50:0a:09:83:88:f8:79:f5 WWPN 50:0a:09:83:88:f8:79:f5 fc_id 0x470001
• - LUN's configured (SCSI Type, Version, Vendor, Serial No.)
• LUN ID : 0x0000000000000000 (0x0, 0x4, NETAPP , D1k-Y?E4VkJR) <<<<<<<<<<<<<<
• - REPORT LUNs Query Response
• LUN ID : 0x0000000000000000 <<<<<<<<<<<<<<<<<<
• - Nameserver Query Response
• - WWPN : 50:0a:09:83:88:f8:79:f5
• adapter 1/3/1 (fls):2#
Verify that the LUN is accessible by host
Troubleshooting
• Same tools as classic UCS
• Techsupport
• NXOS (connect nxos)
• Adaptor (connect adaptor)
• Blade (connect cimc)
• All CMC related debug commands will be under NXOS.
• All of the 3.0 UCS-Mini functionality is built upon 2.2.1 UCS Release. For documentation, please refer to:
• http://www.cisco.com/c/en/us/td/docs/unifi
ed_computing/ucs/sw/cli/config/guide/2-2/b_UCSM_CLI_Configuration_Guide_2_2.pdf
• http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/sw/gui/config/guide/2-2/b_UCSM_GUI_Configuration_Guide_2_2.pdf
Troubleshooting Tools
Monitoring Session
Span Destination
Select a FI
Create Monitoring Session
Select Scalability Port
Span Source
Select monitoring session
Select Direction Transmit | Recv | Both
Select Scalability Port
Create : eth-traffic-monitoring session
Monitoring Destination
Monitoring Source
eth-traffic-mon session details
Inapplicable Features
Primary Chassis cannot be
decommissioned or removed. (greyed out)
Chassis/FEX Discovery, RACK
server discovery and management connection policy are not applicable. (ignored)
Backup and Restore
• Functionality is identical to classic UCS.
• Import/export files are not UCS-Mini-specific, i.e. file format is platform-independent.
• Files can be imported/exported across UCS-Mini and classic UCS as long as file content (data) is compatible. If incompatible, import operation will fail.
• Full-state backup/restore should not be done across two platforms. But there is no explicit validation for this.
• Recommendation – Do not use import/export across UCS-Mini and classic UCS platforms.
Import / Export
Polling Question 3
Do you use UCS C series server as standalone or integrated to UCSM?
1. I use UCS C series servers in standalone mode.
2. I use UCS C series integrated with UCSM.
3. I don’t use UCS C series server.
Submit Your Questions Now! Use the Q & A panel to submit your questions and our expert will respond
Collaborate within our Social Media
Facebook- http://bit.ly/csc-facebook
Twitter- http://bit.ly/csc-twitter
You Tube http://bit.ly/csc-youtube
Google+ http://bit.ly/csc-googleplus
LinkedIn http://bit.ly/csc-linked-in
Instgram http://bit.ly/csc-instagram
Newsletter Subscription http://bit.ly/csc-newsletter
Learn About Upcoming Events
Cisco has support communities in other languages!
Spanish https://supportforums.cisco.com/community/spanish
Portuguese https://supportforums.cisco.com/community/portuguese
Japanese https://supportforums.cisco.com/community/csc-japan
Russian https://supportforums.cisco.com/community/russian
Chinese
http://www.csc-china.com.cn
If you speak Spanish, Portuguese, Japanese, Russian or Chinese we invite you to participate and collaborate in your language
More IT Training Videos and Technical Seminars on the Cisco Learning Network
View Upcoming Sessions Schedule
https://cisco.com/go/techseminars
Please take a moment to complete the survey
Thank you for Your Time!