release notes for cisco hx data platform, release 4€¦ · •cisco hyperflex m5 converged...

66
Release Notes for Cisco HX Data Platform, Release 4.0 First Published: 2019-05-02 Last Modified: 2020-05-11 Introduction Cisco HyperFlex Systems unlock the full potential of hyperconvergence. The systems are based on an end-to-end software-defined infrastructure, combining software-defined computing in the form of Cisco Unified Computing System (Cisco UCS) servers, software-defined storage with the powerful Cisco HX Data Platform, and software-defined networking with the Cisco UCS fabric that integrates smoothly with Cisco Application Centric Infrastructure (Cisco ACI). Together with a single point of connectivity and hardware management, these technologies deliver a pre-integrated and adaptable cluster that is ready to provide a unified pool of resources to power applications as your business needs dictate. These release notes pertain to the Cisco HX Data Platform, Release 4.0, and describe the features, limitations and caveats for the Cisco HX Data Platform. Revision History Description Date Release Added OTV for Stretched Cluster update in New Features for the 4.0(2b) release. May 11, 2020 4.0(2b) Updated HX 4.0(2a) New Feature section to show support for UCSM hardware and software enhancements and features from Cisco UCSM 4.1(1c) release. Updated Host Upgrade Utility (HUU) for M5 to UCS 4.0(4k) for HX 4.0(2a). May 5, 2020 4.0(2a) Created release notes for Cisco HX Data Platform Software, Release 4.0(2b). April 22, 2020 4.0(2b) Updated M4 and M5 Recommended FI/Server Firmware to UCS 4.0(4h) for HX 4.0(1b). March 30, 2020 4.0(1b) Release Notes for Cisco HX Data Platform, Release 4.0 1

Upload: others

Post on 20-May-2020

138 views

Category:

Documents


0 download

TRANSCRIPT

Release Notes for Cisco HX Data PlatformRelease 40

First Published 2019-05-02

Last Modified 2020-05-11

IntroductionCisco HyperFlextrade Systems unlock the full potential of hyperconvergence The systems are based on anend-to-end software-defined infrastructure combining software-defined computing in the form of CiscoUnified Computing System (Cisco UCS) servers software-defined storage with the powerful Cisco HX DataPlatform and software-defined networking with the Cisco UCS fabric that integrates smoothly with CiscoApplication Centric Infrastructure (Cisco ACI) Together with a single point of connectivity and hardwaremanagement these technologies deliver a pre-integrated and adaptable cluster that is ready to provide a unifiedpool of resources to power applications as your business needs dictate

These release notes pertain to the Cisco HX Data Platform Release 40 and describe the features limitationsand caveats for the Cisco HX Data Platform

Revision History

DescriptionDateRelease

Added OTV for Stretched Clusterupdate in New Features for the40(2b) release

May 11 202040(2b)

Updated HX 40(2a) New Featuresection to show support for UCSMhardware and softwareenhancements and features fromCisco UCSM 41(1c) release

Updated Host Upgrade Utility(HUU) for M5 to UCS 40(4k) forHX 40(2a)

May 5 202040(2a)

Created release notes for Cisco HXData Platform Software Release40(2b)

April 22 202040(2b)

Updated M4 and M5Recommended FIServer Firmwareto UCS 40(4h) for HX 40(1b)

March 30 202040(1b)

Release Notes for Cisco HX Data Platform Release 401

DescriptionDateRelease

Updated M4 and M5Recommended FIServer Firmwareto UCS 40(4h) for HX 40(2a)

March 24 202040(2a)

Created release notes for Cisco HXData Platform Software Release40(2a)

February 11 202040(2a)

Updated release notes for deferredCisco HyperFlex Release HX35(2c)

January 15 202035(2c)

Updated M4 and M5Recommended FIServer Firmwareto UCS 40(4e) for HX 40(1b)40(1a) 35(2f) 35(2e) and35(2d)

December 23 201940(1b)

Added CSCvs28167 to the list ofResolved Caveats for HX 40(1b)and HX 40(1a)

December 13 201940(1b)

Added CSCvs02466 to the list ofOpen Caveats

November 25 201940(1b)

Updated info in the HyperFlexEdge and Firmware CompatibilityMatrix for 3x Deployments

Updated info in the Storage ClusterSpecifications

November 7 201940(1b)

Added CSCvj95606 andCSCvq24176 to the list of SecurityFixes

October 25 201940(1b)

Updated Recommended FIServerFirmware versions

October 8 201940(1b)

Updated HUUCIMC info in theHyperFlex Edge and FirmwareCompatibility Matrix for 4xDeployments

September 30 201940(1b)

AddedCSCvq41985 to new sectionfor Related Caveats

September 17 201940(1b)

Updated HUUCIMC info in theHyperFlex Edge and FirmwareCompatibility Matrix for 4xDeployments

September 16 201940(1b)

Release Notes for Cisco HX Data Platform Release 402

Revision History

DescriptionDateRelease

Updated HUUCIMC info in theHyperFlex Edge and FirmwareCompatibility Matrix for 3x and4x Deployments

September 10 201940(1b)

Updated HUUCIMCrecommended firmware versionsfor HyperFlex Releases 40(1b)35(2e) and 35(2d)

August 28 201940(1b)

Updated Recommended FIServerFirmware versions for HyperFlexReleases 35(2e) and 35(2d)

August 23 201940(1b)

Added Cisco IMC version supportinfo in the HyperFlex Edge andFirmware CompatibilityMatrix for4x Deployments

August 21 201940(1b)

Created release notes for Cisco HXData Platform Software Release40(1b)

August 19 201940(1b)

Added bullet describing the CiscoHyperFlex SystemsUpgradeGuidefor Unsupported Cisco HXReleases in the UpgradeGuidelines section

August 8 201940(1a)

Added important note indicatingHyperFlex does not support UCSserver firmware 40(4a) 40(4b)and 40(4c)

August 5 201940(1a)

Updated component info forHX220c M5HXAF220c M5Cluster to VIC 1457 in HyperFlexEdge and Firmware CompatibilityMatrix for 4x Deploymentssection

July 25 201940(1a)

bull AddedRelease 35(2e) supportfor ESXi 67 U2 and updatednotes for 35(2d) 35(2c) and35(2b) support for ESXi 67U2 in Supported VMwarevSphere Versions andEditions

bull Added Witness Node Versionfor Release 35(2e)

July 22 201940(1a)

Release Notes for Cisco HX Data Platform Release 403

Revision History

DescriptionDateRelease

Added CSCvq39523 to the list ofOpen Caveats for Release 40(1a)

July 5 201940(1a)

bull Updated Important Note forSED-based HyperFlexsystems in SupportedVersions and SystemRequirements section

bull Updated Release 35(2b)support for ESXi 67 U2 inSupported VMware vSphereVersions and Editions

July 1 201940(1a)

bull Added CSCvp64140 andCSCvp98910 to the list ofOpen Caveats for Release40(1a)

bull Updated HyperFlex Edge andFirmware CompatibilityMatrix tables

June 20 201940(1a)

Updated BrowserRecommendations

June 17 201940(1a)

Added information indicating40(1a) features are supported onthe Intersight Virtual Appliance andon Intersightcom

May 31 201940(1a)

bull Updated M4M5Recommended FIServerFirmware for 35(2b) and40(1a)

bull Added bullet describingrecommended use of theHypercheck Health CheckUtility in the UpgradeGuidelines section

May 21 201940(1a)

bull Added New Featuredescription for DISA STIGCompliance

bull Updated Storage ClusterSpecifications for Hyper-V

bull Added CSCvp66277 to list ofOpen Caveats

May 14 201940(1a)

Release Notes for Cisco HX Data Platform Release 404

Revision History

DescriptionDateRelease

bull Updated VMware vCenterVersions for 40(1a)

bull Updated SupportedMicrosoftSoftware versions

May 8 201940(1a)

bull Added CSCvo36198 andCSCvk38003 to the list ofResolved Caveats

bull Added CSCvp21417 to the listof Open Caveats

May 3 201940(1a)

Created release notes for Cisco HXData Platform Software Release40(1a)

April 29 201940(1a)

New FeaturesCisco HX Data Platform Release 40 provides the following features These features including the InvisibleCloud Witness for HyperFlex Edge clusters are supported both on the Intersight Virtual Appliance and onIntersightcom

New Features in Release 40(2b)

bull 76TB SSD data drivemdashSupport on HX Edge configurations

bull All NVMe and All Flash limits increasemdashLimits increased on All NVMe (1TB 4TB and 8TB) andAll Flash (76TB SSD)

bull Cluster Scale Limits increasemdashSupport for maximum scale limit increase on a cluster See Cisco HXData Platform Storage Cluster Specifications on page 16

bull HW Offload optionmdashSupport for Hardware Offload option with Stretched cluster configurations

bull Cisco Overlay Transport Virtualization for Stretched ClustermdashSupport for OTV as an overlay forStretched Cluster

New Features in Release 40(2a)

bull Cisco UCS Manager 41(1c)mdashSupport for UCSM hardware and software enhancements and featuresfrom Cisco UCSM 41(1c) release

bull Boost ModemdashThis release introduces Boost Mode for the following configurations All NVMe AllFlash C240 All Flash C220 and Hypervisor ESX Boost Mode allows the Cisco HyperFlex cluster todeliver higher IOPs by increasing the storage controller VMCPU resources by 4 vCPU For configurationinformation see the Cisco HyperFlex Data Platform Administration Guide

bull Cisco HyperFlex HTML plug-in for VMWare vCentermdashEnables virtualization administrator tomanage and monitor the Cisco HyperFlex physical infrastructure by cross launching HyperFlex Connectfrom the vSphere Client UI and perform management actions in the HyperFlex Connect UI

Release Notes for Cisco HX Data Platform Release 405

New Features

bull 25GE networking for HX EdgemdashSupport for 25GE networking for HX Edge

bull All NVMe with Stretched ClustermdashSupport for All NVMe with Stretched Cluster (ESX only)

bull Cluster Upgrade Eligibility TestmdashThis release adds the capability to perform a pre-upgrade test whichchecks for cluster readiness before upgrading Example checks include validating cluster state rebalancestatus controllerVM Free Space ESXi version and much more The Elibility test is intended to helpavoid unexpected problems that may arise during the upgrade process It is highly recommended to runthe test before performing the Hyperflex upgrade

bull Registering Smart Software LicensingmdashThis release adds support for software that allows easy trackingof the status of license and software usage trends and simplifies the three core licensing functionsPurchasing Management and Reporting

bull Dynamic self-signed certificate generation enhancementsmdashThis release adds support for Self-signedSSL certificates on the Controller VMs which were static in prior releases The static certificates arereplaced with dynamically generated self-signed certificates upon upgrading to HXDP 40(2a) so thatthe certificates are unique per cluster The new clusters installed with HXDP 40(2a) have dynamicallygenerated self-signed certificates

bull Test Upgrade EligibilitymdashThis release adds support for testing your cluster readiness and infrastructurecompatibility for an upgrade For more information see the Test Upgrade Eligibility sections in theCiscoHyperFlex SystemsUpgrade Guide for VMware ESXi Release 40 or the CiscoHyperFlex UpgradeGuide for Microsoft Hyper-V Release 40

Disaster Recovery

bull Recovery Settings ConfigurationmdashThis release supports configuration of recovery settings to defineglobal recovery parameters and mapping for resources across recovery sites These parameters are usedduring recovery test recovery and migrate operations

bull HyperFlex DR Powershell RunbookmdashPowerShell runbook functionality is extended to support therecovery configuration settings in the runbooks for all recovery scenarios New-HXrunbook cmdlet cannow be used to generate runbook for a single or multiple protection groups In addition two new cmdletsWait-HXTask and Get-HXTaskStatus are introduced

bull Protected Virtual Machine ScalabilitymdashThis release adds support for 1500 VMs across both clustersand 750 VMs per cluster in a bi-direction or any split between the two clusters without exceeding thelimit of 1500 VMs For more information see Cisco HyperFlex Data Platform Administration Guide

bull System Management REST API enhancementsmdashPause data replication actions briefly using RESTAPI to explicitly inform users on the current status of replication actions

Cisco HyperFlex with Data Platform for Hyper-V

bull Cluster-Aware Updating (CAU)mdashThis is an automated feature that allows you to perform updates onwindows servers in a failover cluster with little or no loss in availability during the upgrade process

Support for new drivesmdashNew drives are qualified for the 40(2a) release The new drives include newcapacity points and new cache drive options Several of the new drives are alternate drives to already qualifiedexisting drives in function which are qualified in 40(2a) These drives are functionally compatible with theexisting drives and are available as alternates in case of lack of availability of existing drives For expansionof existing clusters or general information about interoperability of different drives see Cisco HyperFlexDrive Compatibility

Release Notes for Cisco HX Data Platform Release 406

New Features

NVMe Caching SSDs slot information is unavailable from HX-Connect for all AF server PIDs except forthe All-NVMe server PIDs Please refer to UCSM management console for NVMe SSD slot information

Note

Applicable PlatformsDrive PIDDrive Function

All existing HX M5 serversHX-M2-960GBAlternate boot drive

All existing HX M5 servers except AllNVMe

HX-SD480G6I1X-EVAlternate system (or housekeeping) drive

All existing HX M5 servers except AllNVMe

HX-SD480GM1X-EVAlternate system (or housekeeping) drive

The following HX M5 serversHXAF220C-M5SX HXAF240C-M5SXHXAF-E-220M5SX

HX-SD800G123X-EP800G 12G SAS Cache drive option forAll Flash

All NVMe HXAF220C-M5SNHX-NVME2H-I1000All NVMe 1TB Capacity drive

All NVMe HXAF220C-M5SNHX-NVME2H-I4000All NVMe 4TB Capacity drive

All NVMe HXAF220C-M5SNHX-NVMEHW-I8000New high density All NVMe 8TBCapacity drive

All Flash Configuration ndash namelyHXAF220C-M5SX HXAF240C-M5SXHXAF-E-220M5SX

ESX support only

HX-SD76T61X-EVNew high density All Flash 76TBCapacity drive

Maximum number of drives per node is12 drives on HX AF C240

HX240C-M5LHX-HD8T7K4KANAlternate drive for 8TB LFF capacity

HXAF220C-M5SX HXAF240C-M5SXHX-SD38T2HTNK9New 38TB FIPS compliant SED SSDdata drives

HXAF220C-M5SX HXAF240C-M5SXHX-SD960G2HTNK9New 960GFIPS compliant SEDSSD datadrives

New Features in Release 40(1b)

bull Support for Second Generation Intelreg Xeonreg Scalable Processor RefreshmdashThis release includessupport for the Second Generation Intelreg Xeonreg Scalable processor refresh (formerly Cascade Lake)

New Features in Release 40(1a)

The following new features are in Release 40(1a)

bull Ultra-Light HyperFlex Edge ClustersmdashThis release introduces support for two-node HyperFlex Edgeclusters enabling HyperFlex to run in environments requiring a small footprint Cisco Intersight providescomprehensive lifecycle management and includes remote cloud-based installation centralized upgradesand invisible witnessing Both 1GE and 10GE networking topology options are available

Release Notes for Cisco HX Data Platform Release 407

New Features

bull Scaled-Up HyperFlex Edge ClustersmdashThis release adds support for four node HyperFlex Edge clustersenabling a full range of size options for remote and branch offices Size the branch office environmentsto suit current needs with a two three or four node HyperFlex Edge cluster Cisco Intersight providesfull-lifecycle management and 1GE and 10GE networking options are available

bull Cisco Intersight Invisible Cloud WitnessmdashFor two node clusters this feature eliminates the need forwitness VMs the infrastructure to run those VMs and the management overhead to deploy scale andpatch witnessing software The Invisible Cloud Witness is responsible for maintaining cluster HA in theevent of failure scenarios This feature is included at no additional cost and is automatically deployedand managed by Cisco Intersight

bull Cloud-delivered HyperFlex Edge UpgradesmdashPowered by Cisco Intersight this feature adds supportfor multi-site orchestrated remote upgrades of the HyperFlex Data Platform This feature will be enabledwith the next 40 patch release and will allowHyperFlex Edge clusters deployed via Intersight to performorchestrated upgrades across one or many sites in parallel

bull All-NVMe HyperFlexmdashStarting with this release a new high-end performance node powered by allNVMe drives is available for HyperFlex clusters Co-engineered with Intel to support Intel VMD forhot-plug and surprise removal this offering represents an industry first an enterprise-ready and fullyvalidated all-NVMe HCI appliance The all-NVMe offering is available in the 220 form factor (1RU)and is powered by Intel Optane cache drives for maximum performance and highest endurance

bull VMware Site Recovery Manager (SRM) IntegrationmdashThis release brings support for a Cisco developedStorage Replication Adapter (SRA) for SRM The SRA provides the ability to leverage HyperFlex nativeasync replication with the powerful orchestration and runbook capabilities of SRM The SRA includesthe ability to perform test recoveries planned migrations and full disaster recovery

HX SRA is certified by VMware and is available for download from VMWareSRM site

Note

bull HyperFlex DR Powershell RunbooksmdashNew Powershell cmdlets are included for automated runbookgeneration when using HyperFlex native disaster recovery The New-HXRunbook cmdlet supports thefollowing workflows Test Recovery Planned Migration and Disaster Recovery These runbooks canbe used to orchestrate DR workflows without the requirement for any third-party software

bull Windows Server 2019 with Hyper-VmdashSupport has been added in this release for the Windows Server2019 operating system for Hyper-V based HyperFlex deployments

bull Kubernetes CSI PluginmdashThis release adds support for the HyperFlex CSI (HX-CSI) plugin based onthe Kubernetes Container Storage Interface (CSI) specification Customers can now use the HX-CSIplugin to provision andmanage persistent volumes in Kubernetes version 113 and later Note Kubernetes113 support for Cisco Container Platform amp Openshift Container Platform is forthcoming in theirrespective future releases

bull C480 ML Compute only nodemdashThis release introduces support for C480 ML as a new compute-onlynode for Deep LearningMachine Learning Workloads Data scientists can now use the power of up toeight NVidia SXM2 V100 GPUs to accelerate deep learning workloads VMs running deep-learningworkloads will need to use PCIe pass-through for access to GPUs

bull Higher Capacity DrivesmdashAnew 24TB 10k rpm SAS HDD option for SFF Hybrid HyperFlex clustersand a 12TB 72K rpmSASHDDoption for LFFHybrid are now available Both HyperFlex andHyperFlexEdge support the 24TB capacity point for maximum density in this form factor Note that HyperFlex

Release Notes for Cisco HX Data Platform Release 408

New Features

Edge does not support LFF-drives HyperFlex HyperV version does not yet support the new 12TB driveoption See the HyperFlex spec sheets for a full list of configurable options

bull New Cache and increased scale for Hyper-V mdashNVMe amp Optane SSDs are now supported as cachedrives for Hyper-V deployments Furthermore scale limits have been increased to 16+16(Converged+Compute-only) for both SFF (AF and Hybrid) amp LFF (Hybrid) clusters

bull Centralized Audit Log ExportmdashThis release adds support for audit logging via a remote syslog serverThis capability enables customers to retain audit logs from all HyperFlex nodes in a centralized remotesyslog server to meet retention and compliance requirements

bull DISA STIG CompliancemdashThis release adds new HX REST APIs for setting removing and checkingstatus of DISA STIGs for Controller VMs ESXi hosts and vCenter These APIs enable customers tomeet DISA security requirements by centrally and securely applying STIGs detecting and correctingfor drifts in any STIG settings

Supported Versions and System RequirementsCisco HX Data Platform requires specific software and hardware versions and networking settings forsuccessful installation

For a complete list of requirements see

bull Cisco HyperFlex Systems Installation Guide for VMware ESXi or

bull Cisco HyperFlex Systems Installation Guide for Microsoft Hyper-V

Hardware and Software Interoperability

For a complete list of hardware and software inter-dependencies refer to respective Cisco UCS Managerrelease version of Hardware and Software Interoperability for Cisco HyperFlex HX-Series

HyperFlex Software Versions

The HX componentsmdashCisco HX Data Platform Installer Cisco HX Data Platform and Cisco UCSfirmwaremdashare installed on different servers Verify that each component on each server used with and withinan HX Storage Cluster are compatible

bull HyperFlex does not support UCS Manager and UCS Server Firmware versions 40(4a) 40(4b)and 40(4c)

Do not upgrade to these versions of firmware

Do not upgrade to these versions of UCS Manager

Important

bull Verify that the preconfigured HX servers have the same version of Cisco UCS server firmware installedIf the Cisco UCS Fabric Interconnects (FI) firmware versions are different see the Cisco HyperFlexSystems Upgrade Guide for VMware ESXi Release 40 for steps to align the firmware versions

bull M4 For NEW hybrid or All Flash (Cisco HyperFlex HX240c M4 or HX220c M4) deploymentsverify that Cisco UCS Manager 31(3k) 32(3i) or 40(2b) or higher is installed For moreinformation see Recommended Cisco HyperFlex HX Data Platform Software Releases

Release Notes for Cisco HX Data Platform Release 409

Supported Versions and System Requirements

bull M5 For NEW hybrid or All Flash (Cisco HyperFlex HX240c M5 or HX220c M5) deploymentsverify that the recommended UCS firmware version is installed

For SED-based HyperFlex systems ensure that the A (Infrastructure) B (Bladeserver) and C (Rack server) bundles are at Cisco UCS Manager version 40(2b)or later for all SED M4M5 systems For more details see CSCvh04307 ForSED-based HyperFlex systems also ensure that all clusters are at HyperFlexRelease 35(2b) or later For more information see Field Notice (70234) andCSCvk17250

Important

bull To reinstall an HX server download supported and compatible versions of the software See theCisco HyperFlex Systems Installation Guide for VMware ESXi for the requirements and steps

bull Important For Intersight edge servers running older than 40(1a) CIMC version HUU is the suggestedmechanism to update the firmware

Table 1 HyperFlex Software Versions for M4M5 Servers

M5 Recommended FIServer Firmware

(be sure to review important notes above)

M4 Recommended FIServer Firmware

(be sure to review important notes above)

HyperFlexRelease

40(4h)40(4h)40(2b)

40(4h)40(4h)40(2a)

40(4h)40(4h)40(1b)

40(4e)40(4e)40(1a)

HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments

Cisco HX Data Platform Release 4x based Deployments

Confirm the component firmware on the server meets the minimum versions listed in the following tables

HyperFlex Edge does not support Cisco IMC versions 40(4a) 40(4b) 40(4c) 40(4d) and 40(4e)Important

Table 2 HX220c M4 HXAF220c M4 Cluster

Recommended Firmware Version - HXDP 4x

(be sure to review important note(s) above)

Component

402d0BIOS

40(2h)Cisco Integrated Management Controller(CIMC)

Release Notes for Cisco HX Data Platform Release 4010

HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments

Recommended Firmware Version - HXDP 4x

(be sure to review important note(s) above)

Component

13000012SAS HBA

43(2b)VIC 1227

0103 (Toshiba)

CS07 (Intel 480)

SSD

CN03 (Seagate)

CK03 (Seagate 18)

CK03 (Seagate 24)

sF3Q (Samsung)

3F3Q (Samsung)

HDD

40(2h)

Download Software

Host Upgrade Utility (HUU) Download Link

Table 3 HX220c M5 HXAF220c M5 Cluster

Recommended Firmware Version - HXDP 4x

(be sure to review important notes above)

Component

C220M5404p00224200755BIOS

40(4i)Cisco Integrated Management Controller(CIMC)

09000008SAS HBA

43(3c)VIC 1387

50(3e)VIC 1457

G201CS01 (Intel)

0104 (Toshiba)

0107 (Toshiba)

SSD

2F3Q (Samsung)

3F3Q (Samsung)

NOA4 (Seagate)

CK03 (Seagate 18tb)

CK03 (Seagate 24)

HDD

Release Notes for Cisco HX Data Platform Release 4011

HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments

Recommended Firmware Version - HXDP 4x

(be sure to review important notes above)

Component

40(4k)

Download Software

Host Upgrade Utility (HUU) Download Link

HyperFlex Licensing

As of version 26(1a) HyperFlex supports VMware PAC licensing Existing VMware embedded licenses willcontinue to be supported

As of version 25(1a) HyperFlex uses a smart licensing mechanism to apply your licenses See the CiscoHyperFlex Systems Installation Guide for VMware ESXi for details and steps

VMware vSphere Licensing Requirements

How you purchase your vSphere license determines how your license applies to your HyperFlex system

bull If you purchased your vSphere license with HyperFlex

Each HyperFlex server either has the Enterprise or Enterprise Plus edition preinstalled at the factory

bull HX Nodes have OEM licenses preinstalled If you delete or overwrite thecontent of the boot drives after receiving the HX servers you also delete thefactory-installed licenses

bull OEM license keys is a newVMware vCenter 60 U1b feature Earlier versionsdo not support OEM licenses

bull All factory-installed HX nodes share the same OEM license key WithvSphere OEM keys the Usage count can exceed the Capacity value

bull When you add an HX host to vCenter through the Add Host wizard in theAssign license section select the OEM license

We obfuscate the actual vSphere OEM license key for example0N085-XXXXX-XXXXX-XXXXX-10LHH

bull Standard Essentials Plus and ROBO editions are not available preinstalledon HX servers

Note

bull If you did NOT purchase your vSphere license with HyperFlex

The HX nodes have a vSphere Foundation license preinstalled After initial setup you can apply thelicense to a supported version of vSphere

bull If you purchased a vSphere PAC license

Follow the instructions in your PAC license letter from VMware to add the license to your MY VMwareaccount then follow the instructions to add your HX host to vCenter and assign the PAC license

Release Notes for Cisco HX Data Platform Release 4012

HyperFlex Licensing

HX Data Platform Software Versions for HyperFlex Witness Node

Witness Node VersionHyperFlex Release

10840(2b)

10840(2a)

10440(1b)

10440(1a)

Software Requirements for VMware ESXi

The software requirements include verification that you are using compatible versions of Cisco HyperFlexSystems (HX) components and VMware vSphere VMware vCenter and VMware ESXi

bull Verify that all HX servers have a compatible version of vSphere preinstalled

bull Verify that the vCenter version is the same or later than the ESXi version

bull Verify that the vCenter and ESXi versions are compatible by consulting the VMware ProductInteroperability Matrix Newer vCenter versions may be used with older ESXi versions so long as bothESXi and vCenter are supported in the table below

bull Verify that you have a vCenter administrator account with root-level privileges and the associatedpassword

For VIC1457 there is no support for ESXi 60Note

The below table applies for all of the following VMware vSphere Editions Enterprise Enterprise PlusStandard Essentials Plus ROBO

VMware vCenter VersionsVMware ESXi VersionsHyperFlex Version

60 U3 65 U3 67 U360 U3 65 U3 67 U340(2b)1

60 U3 65 U3 67 U360 U3 65 U3 67 U340(2a)2

60 U3 65 U3 67 U260 U3 65 U3 67 U2340(1b)

60 U3 65 U2 67 U260 U3 65 U2 67 U2440(1a)

1 Cisco HyperFlex Release 40(2) is the last major HyperFlex release that will support vSphere 60 (ESXiand vCenter) due to those versions reaching end of VMware general support on March 12 2020

2 Cisco HyperFlex Release 40(2) is the last major HyperFlex release that will support vSphere 60 (ESXiand vCenter) due to those versions reaching end of VMware general support on March 12 2020

3 Use of 67 U2 is not recommended see software advisory for CSCvq06952 and CSCvp88515 for furtherdetails

4 Use of 67 U2 is not recommended see software advisory for CSCvq06952 and CSCvp88515 for furtherdetails

Release Notes for Cisco HX Data Platform Release 4013

HX Data Platform Software Versions for HyperFlex Witness Node

Software Requirements for Microsoft Hyper-V

The software requirements include verification that you are using compatible versions of Cisco HyperFlexSystems (HX) components and Microsoft Hyper-V (Hyper-V) components

HyperFlex Software versions

The HX componentsmdashCisco HX Data Platform Installer Cisco HX Data Platform and Cisco UCSfirmwaremdashare installed on different servers Verify that each component on each server used with and withinthe HX Storage Cluster are compatible

bull Cisco HyperFlex M5 Converged nodesmdash For Hybrid (Cisco HyperFlex HX240c M5 HX220c M5)and All Flash (Cisco HyperFlex HXAF240cM5 HXAF220cM5) verify that Cisco UCSManager 40(2b)is installed HX 40 (1a) does not support Hyper-V on the All NVMe (HXAF220C-M5SN) nodes Fordetailed information on installation requirements and steps see theCisco HyperFlex Systems InstallationGuide on Microsoft Hyper-V

Table 4 Supported HyperFlex Software versions for M5 Servers on Hyper-V

M5 Recommended Server FirmwareHyperFlex Release

40(4h)40(2b)

40(4g)40(2a)

40(4e)40(1b)

40(4e)40(1a)

Release Notes for Cisco HX Data Platform Release 4014

Software Requirements for Microsoft Hyper-V

Table 5 Supported Microsoft Software versions

VersionMicrosoft Component

Windows Server 2016 Datacenter Core amp DesktopExperience

ForWindows Server 2016Datacenter Coreand Desktop Experience the Windows2016 ISO image should be Update BuildRevision (UBR) 1884 at a minimum

Note

Windows Server 2019Datacenter-DesktopExperienceis supported starting from HXDP 401(a) onwards

For Windows Server 2019 DesktopExperience theWindows 2019 ISO imageshould be Update Build Revision (UBR)107 at a minimum

Note

Windows Server 2019 DatacenterndashCore is notsupported currently

Also note that the following are currently notsupported

OEM activated ISOs and Retail ISOs are notsupported

Earlier versions ofWindows Server such asWindows2012r2 are not supported

Non-English versions of the ISO are not supported

Windows Operating System (Windows OS)

A Windows 2012 or later domain and forestfunctionality level

Active Directory

Supported Microsoft License Editions

The Microsoft Windows Server version that is installed on one or more HyperFlex hosts must be licensed asper Microsoft licensing requirements listed on Microsoft Licensing

Browser Recommendations

Use one of the following browsers to run the listed HyperFlex components These browsers have been testedand approved Other browsers might work but full functionality has not been tested and confirmed

Table 6 Supported Browsers

HX ConnectHX Data PlatformInstaller

Cisco UCS ManagerCisco IntersightBrowser

11 or higher11 or higher11 or higherNAMicrosoft InternetExplorer

70 or higher70 or higher57 or higher62 or higherGoogle Chrome

Release Notes for Cisco HX Data Platform Release 4015

Browser Recommendations

HX ConnectHX Data PlatformInstaller

Cisco UCS ManagerCisco IntersightBrowser

60 or higher60 or higher45 or higher57 or higherMozilla Firefox

NANA9 or higher10 or higherApple Safari

NANA35 or higherNAOpera

Notes

bull Cisco HyperFlex Connect

The minimum recommended resolution is 1024 X 768

bull Cisco HX Data Platform Plug-in

The Cisco HX Data Platform Plug-in runs in vSphere For VMware Host Client System browserrequirements see the VMware documentation at httpswwwvmwarecomsupportpubs

The HX Data Platform Plug-in is not displayed in the vCenter HTML client You must use the vCenterflash client

bull Cisco UCS Manager

The browser must support the following

bull Java Runtime Environment 16 or later

bull Adobe Flash Player 10 or higher is required for some features

For the latest browser information about Cisco UCSManager refer to themost recent Cisco UCSManagerGetting Started Guide

Cisco HX Data Platform Storage Cluster SpecificationsCluster Limits

bull Cisco HX Data Platform supports up to 100 clusters managed per vCenter as per VMware configurationmaximums

bull Cisco HX Data Platform supports any number of clusters on a single FI domain Each HX convergednode must be directly connected to a dedicated FI port on fabric A and fabric B without the use of a FEXC-series compute only nodes must also directly connect to both FIs B-series compute only nodes willconnect through a chassis IO module to both fabrics In the end the number of physical ports on the FIwill dictate the maximum cluster size and maximum number of individual clusters supported in a UCSdomain

The following table provides Cisco HX Data Platform Storage Cluster Specifications

Release Notes for Cisco HX Data Platform Release 4016

Cisco HX Data Platform Storage Cluster Specifications

Stretched Cluster (Availableon ESX Only)

Microsoft Hyper-VVMware ESXiNode

AllNVMe -HXAF220c-M5SN

HX240c-M5LHX220cM5

HX220cAF M5

HX240cM5

HX240cAF M5

HX240c-M5LHX220cM5

HX220cAF M5

HX240cM5

HX240cAF M5

HXAF220c-M5SN

AllNVMe -HXAF220c-M5SN

ESXi 65or 67only NotsupportedwithESXi 60orHyper-V

HX220cM5 Edge

HXAF220cM5 Edge

HX220cM4 Edge

HXAF220cM4 Edge

HX240c-M5LHX220cM5

HX220cAF M5

HX240cM5

HX240cAF M5

HX220cM4

HX220cAF M4

HX240cM4

HX240cAF M4

HXServers

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

C220M5C240M5B200M4B200M5

C240M5C220M5B200M4B200M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

mdashB200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

Compute-OnlyUCSB-SeriesC-SeriesServers

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

Convergednodesonly

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

SupportedNodes

Release Notes for Cisco HX Data Platform Release 4017

Cisco HX Data Platform Storage Cluster Specifications

Stretched Cluster (Availableon ESX Only)

Microsoft Hyper-VVMware ESXiNode

NA(requiresEnterpriseHXDP-PLicense)

NA(requiresEnterpriseHXDP-PLicense)

NA(requiresEnterpriseHXDP-PLicense)

Convergednodes3-16(12TBHDDoption isnotsupportedforHyperV)

Compute-onlynodes0-16

Convergednodes3-16

Compute-onlynodes0-16

NA(requiresEnterpriseHXDP-PLicense)

M4Convergednodes 3

M5Convergednodes23or 4

RequiresHXDP-ELicense

Convergednodes3-16(12TBdrive islimited toamaximumof 8nodes)

Compute-onlynodes0-16(12TBdrive islimited toamaximumof 8nodes)

Convergednodes3-32(76TBdriveconfigsarelimited tomaximumof 12drivesnodeon AF240)

Compute-onlynodes0-32

HXDP-SLicensedNodeLimits

11 ratioofHXDP-StoComputeonlynodes

(MinmdashMax)

Convergednodes2-16 perSite

Compute-onlynodes0-21 perSite

(up tomaxclustersize)

Convergednodes2-8 perSite

Compute-onlynodes0-16 perSite

(up tomaxclustersize)

Convergednodes2-16 perSite

Compute-onlynodes0-21 perSite

(up tomaxclustersize)

Convergednodes3-16(12TBHDDoption isnotsupportedforHyperV)

Compute-onlynodes0-16

Convergednodes3-16

Compute-onlynodes0-16

Convergednodes3-32

ComputeOnlynodes0-32

(up tomaxclustersize)

Convergednodes 3

(requiresHXDP-ELicense)

Convergednodes3-16(12TBdrive islimited toamaximumof 8nodes)

Compute-onlynodes0-32(12TBdrive islimited toamaximumof 16nodes)

Convergednodes3-32(76TBdriveconfigsarelimited tomaximumof 12drivesnodeon AF240)

Compute-onlynodes0-32

(up tomaxclustersize)

HXDP-PLicensedNodeLimits

12 ratioofHXDP-PtoComputeonlynodes

(MinmdashMax)

24 perSite 64percluster

24 perSite 48percluster

32 perSite 64percluster

32326434864MaxClusterSize

Release Notes for Cisco HX Data Platform Release 4018

Cisco HX Data Platform Storage Cluster Specifications

Stretched Cluster (Availableon ESX Only)

Microsoft Hyper-VVMware ESXiNode

212121111121mdash2121MaxComputetoConvergedratio

NoExpansion

Requires Enterprise license

Requires uniform expansion across both sites

Guidelines and Limitationsbull Starting with release 40(2a) SCVM is no longer needed on a Compute node

bull HX REST API Access Token Management ndash Applications leveraging HX REST APIs should re-useaccess tokens when making API calls Once obtained using the AAA Obtain Access Token API accesstokens are valid for 18 days (1555200 seconds) In addition AAA enforces rate limiting on ObtainAccess Token API requests in a 15 minute window auth can be invoked (successfully) a maximum of5 times A user is allowed to create a maximum of 8 unrevoked tokens Subsequent call to auth willautomatically revoke the oldest issued token tomake room for the new token Amaximum of 16 unrevokedtokens can be present in system In order to prevent brute-force attacks after 10 consecutive failedauthentication attempts a user account is locked for a period of 120 secondsFor more information seeCisco HyperFlex Systems REST API Reference guide

HxConnect makes use of AAA Authentication REST API for login and the above rate limit applies toHxConnect also

Upgrade Guidelines

The following list is a highlight of critical criteria for performing an upgrade of your HyperFlex system

bull Unsupported HX Data Platform 17x 18x 20 21x 25x and 26x clustersmdashUsers from anyversion prior to 26(1a) must step through an intermediate version before upgrading to 40 or later releasesIf you need to upgrade your environment from a Cisco HyperFlex HX Data Platform software releasethat is past the last date of support to the latest suggested release on the Cisco Software Download sitesee Cisco HyperFlex Systems Upgrade Guide for Unsupported Cisco HXReleases For more informationsee the Software Advisory for CSCvq66867 WARNING Only Use HXDP 26(1e) Upgrade PackageWhen Upgrading From HXDP 18(1a)-18(1e)

bull Hypercheck Health Check UtilitymdashCisco recommends running this proactive health check utility onyour HyperFlex cluster prior to upgrade These checks provide early visibility into any areas that mayneed attention andwill help ensure a seamless upgrade experience For more information see the HyperFlexHealth amp Pre-Upgrade Check Tool TechNote for full instructions on how to install and run Hypercheck

bull vSphere 67 Software AdvisorymdashDo not upgrade to Cisco HX Data Platform Release 40(1a) whenrunning ESXi 67U1 EP06 (build 11675023) Do not upgrade to 67U1 EP06 (build 11675023) ifrunning Cisco HX Data Platform Release 40(1a) See the Software Advisory CSCvo56350 for furtherdetails

Release Notes for Cisco HX Data Platform Release 4019

Guidelines and Limitations

The software build version posted at release will override any other local versions

bull Required vCenter upgrademdashFor enhanced security Cisco HX Data Platform Release 35(1a) or laterrequires the use of TLS 12 Therefore vCenter must be upgraded to 60 U3f or later before upgradingto Cisco HX Data Platform Release 35 or later In addition ESXi should be upgraded as required tomeet HX Data Platform compatibility requirements

bull Minimum HXDP version for upgrademdashHXData Platform clusters running 26(1a) or later may upgradedirectly to 40 using the HX Connect UI

bull Cluster ReadinessmdashEnsure that the cluster is properly bootstrapped and the updated plug-in is loadedbefore proceeding Manual cluster bootstrap is required for upgrade from a pre-35 release

bull Cluster ReadinessmdashEnsure that the cluster is properly bootstrapped and the updated plug-in is loadedbefore proceeding Manual cluster bootstrap is required for HX releases earlier than 35(1a) For moreinformation see theManual Bootstrap Upgrade Process in the Cisco HyperFlex Systems Upgrade Guidefor VMware ESXi Release 40 Do not skip this cluster bootstrap step it is required for all upgradesuntil HX Release 35(1a) Auto bootstrap is supported beginning with HX release 35(1a) For moreinformation see the Auto Bootstrap Upgrade Process from HX Connect UI in the Cisco HyperFlexSystems Upgrade Guide for VMware ESXi Release 40

Manual bootstrap is not supported on Intersight clusters

bull Initiating Upgrade―Use the HX Connect UI or CLI stcli commands when upgrading from 25(1a)or later releases Use either the CLI stcli commands or the HX Data Platform Plug-in to the vSphereWeb Client when upgrading from a pre-25(1a) release The vCenter plug-in should not be used forupgrades starting with the 25(1a) release The vCenter plug-in should not be used for upgrades startingwith the 25(1a) release

If the current cluster version is at 35(1a) or above you do not need to use the stcli command Directupgrade to 40 is possible

bull Complete your Upgrade―The self-healing (or rebalance) capability is disabled temporarily during theupgrade window If the upgrade fails you should complete the upgrade as soon as possible

bull ESXi and HXDP Compatibility―Ensure your cluster is running a compatible version of ESXi basedon the running the HXData Platform version (see the section Software Requirements for VMware ESXi)ESXi compatibility is determined by the major version and update release of ESXi It is generally bestto upgrade HXDP and ESXi together if combining the upgrade operations into a single optimized rebootWhen running a split upgrade first upgrade the HX Data Platform then proceed to upgrade ESXi

bull If Upgrading to vSphere 65

bull Certain cluster functions such as native and scheduled snapshotsReadyClones and Enter or Exit HX Maintenance Mode will not operatefrom the time the upgrade is started until the HX Data Platform upgrade iscomplete

bull After upgrading ESXi using the offline zip bundle use the ESX ExitMaintenance Mode option The HX Exit Maintenance Mode option doesnot operate in the vSphere Web Client until the HX Data Platform upgradeis complete

Note

Release Notes for Cisco HX Data Platform Release 4020

Guidelines and Limitations

bull vSphere 60 UpgradesmdashUsers on vSphere 60 migrating to 65 upgrade components in the followingorder

1 Upgrade HX Data Platform and UCS firmware

2 Upgrade HX Data Platform and ESXi

3 Upgrade HX Data Platform only first then upgrade ESXi or UCS firmware or both

bull M4 Server Firmware UpgradesmdashUpgrade server firmware to ensure smooth operation and to correctknown issues Specifically newer SAS HBA firmware is available in this release and is recommendedfor long-term stability

bull Users are encouraged to upgrade to 31(3c) C-bundle or later whenever possible

bull Users running C-bundle versions before 31(2f) must upgrade server firmware by performing acombined upgrade of UCS server firmware (C-bundle) to 31(3c) or later and HX Data Platform to25 Do not split the upgrade into two separate operations

bull If the cluster is already on 31(2f) C-bundle or later you may perform an HX Data Platform onlyor combined upgrade as required

bull M5 Server Firmware UpgradesmdashM5 generation servers must run firmware version 32(2d) or later

bull Firmware Downgrades mdashDowngrading UCSM from the HX-installer is not supported

bull M4M5 Mixed DomainsmdashA mixed domain occurs when a new separate M5 cluster is installed underthe same UCS domain that contains existing M4 clusters Under these conditions orchestrated UCSserver firmware upgrade will not operate until Cisco HX Data Platform Release 26 or later is installedon the M4 clusters Therefore it is best practice to first upgrade UCS server firmware to the latest 31(3)or 32(2) patch release before adding a new M5 cluster to the existing UCS domain Additionally any17 HX Data Platform clusters must first be upgraded before adding any new M5 clusters to the samedomain

bull Maintenance WindowmdashIf upgrading both HX Data Platform and UCS firmware you can select eithera combined or split upgrade through the vSphere HX Data Platform Plug-in depending on the length ofthe maintenance window Cisco UCSManager infrastructure upgrade is only supported using AutoInstalland the direct server firmware upgrade should be performed only through the upgrade orchestrationframework provided by the HX Data Platform Plug-in

bull Unsupported Self-Encrypting Drives (SEDs)mdashIf adding or replacing self-encrypting drives (SEDs)that have been recently qualified in newer versions of HX Data Platform insert the new drives only afterupgrading HX Data Platform to a compatible version All drives must be SED drives mixing SED andnon-SED is not supported

bullbull Enabling External Host AccessmdashWith Cisco HX Data Platform Release 40(1a) port 445 on themanagement network is blocked for enhanced security Note that prior to 40 port 445 port was openenabling external host access If you are upgrading to 40(1a) from a prior release and would like tocontinue external host access you can use a utility to open select hosts For more information aboutenabling external host access see the ConfiguringHyperFlex Share to SCVMM section in the InstallationGuide for Microsoft Hyper-V

Release Notes for Cisco HX Data Platform Release 4021

Guidelines and Limitations

Mixed Cluster Expansion Guidelinesbull Expanding existing M4 cluster with M5 converged nodes is supported

bull Expanding existing M5 cluster with M4 converged nodes is not supported

bull Expanding existing mixed M4M5 cluster with M4 or M5 converged nodes is supported

bull Adding any supported compute-only nodes is permitted with all M4 M5 and mixed M4M5 clustersusing the HX Data Platform 26 or later Installer Some example combinations are listed here manyother combinations are possibleExample combinationsExpand mixed M4M5 cluster with compute-only B200 C220 C240 M4M5Expand M4 cluster with compute-only B200 M5 C220 M5 C240M5

bull Only expansion workflow is supported to create a mixed cluster Initial cluster creation with mixedM4M5 servers is not supported

bull All M5 servers must match the form factor (220240) type (HybridAF) security capability (Non-SEDonly) amp disk configuration (QTY capacity and non-SED) of the existing M4 servers

bull HX220-M5 will use a maximum of 6 capacity disks (2 disk slots to remain empty) when mixedwith HX220-M4

bull HX Edge SED LFF Hyper-V and Stretched Clusters do not support mixed M4 and M5 clusters

Security FixesThe following security issues are resolved

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-13132 CVE-2019-9924CSCvq6313840(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-11719 CVE-2019-11727CVE-2019-11729

CSCvq7124040(2a)

Release Notes for Cisco HX Data Platform Release 4022

Mixed Cluster Expansion Guidelines

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-1125CSCvr0633940(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2015-9383 CVE-2018-14498CVE-2018-20406 CVE-2018-20852CVE-2019-10160 CVE-2019-13117CVE-2019-13118 CVE-2019-14287CVE-2019-14973 CVE-2019-15903CVE-2019-17546 CVE-2019-18197CVE-2019-18218 CVE-2019-5010CVE-2019-5094 CVE-2019-5481CVE-2019-5482 CVE-2019-9636CVE-2019-9740 CVE-2019-9947CVE-2019-9948

CSCvs0609440(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2017-13168 CVE-2017-18174CVE-2017-18216 CVE-2018-10876CVE-2018-10877 CVE-2018-10878CVE-2018-10879 CVE-2018-10880CVE-2018-10881 CVE-2018-10882CVE-2018-10902 CVE-2018-10938CVE-2018-12233 CVE-2018-12896CVE-2018-13053 CVE-2018-13094CVE-2018-13096 CVE-2018-13405CVE-2018-13406 CVE-2018-14609CVE-2018-14617 CVE-2018-14633CVE-2018-14734 CVE-2018-15572CVE-2018-15594 CVE-2018-16276CVE-2018-16658 CVE-2018-17182CVE-2018-17972 CVE-2018-18021CVE-2018-18690 CVE-2018-18710CVE-2018-6554 CVE-2018-6555CVE-2018-9363

CSCvp6501940(2a)

This vulnerability is due to insufficientHTML iframe protection An attackercould exploit this vulnerability bydirecting a user to an attacker-controlledweb page that contains amaliciousHTMLiframe A successful exploit could allowthe attacker to conduct clickjacking orother clientside browser attacks

NACSCvo9851640(2a)

Release Notes for Cisco HX Data Platform Release 4023

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerability is due to insufficientauthentication for the statistics collectionservice An attacker could exploit thisvulnerability by sending properlyformatted data values to the statisticscollection service of an affected deviceA successful exploit could allow theattacker to cause the web interfacestatistics view to present invalid data tousers

NACSCvj9558440(2a)

The vulnerability is due to insufficientCSRF protections for the web UI on anaffected device An attacker could exploitthis vulnerability by persuading a user ofthe interface to follow a malicious linkA successful exploit could allow theattacker to perform arbitrary actions withthe privilege level of the affected user

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvp2434340(2a)

Vulnerabilities with a version of cJSONidentified by CVE-2019-11834 andCVE-2019-11835

For more information see the relatedCisco Security Advisory

CVE-2019-11834 CVE-2019-11835CSCvq1994940(2a)

HX ESXi image patches for MCEPSCand TAA vulnerabilities with VMwareESXi identified by CVE-2018-12207 andCVE-20190-11135

For more information see the relatedCisco Security Advisory

CVE-2018-12207 CVE-2019-11135CSCvr5439840(2a)

CP networking vulnerabilities affectingthe Linux kernel identified byCVE-2019-11477 CVE-2019-11478CVE-2019-11479

For more information see the relatedCisco Security Advisory

CVE-2019-11477 CVE-2019-11478CVE-2019-11479

CSCvq1954640(2a)

Qualification of Microsoft securitypatches for vulnerabilities withMicrosoftHyper-V hypervisor identified withCVE-2018-12207 CVE-2019-11135

For more information see the relatedCisco Security Advisory

CVE-2018-12207 CVE-2019-11135CSCvr5439940(2a)

Release Notes for Cisco HX Data Platform Release 4024

Security Fixes

DescriptionCVEDefect IDRelease

OpenSSH vulnerabilities identified byCVE-2019-6111

For more information see the relatedCisco Security Advisory

CVE-2016-10708 CVE-2018-15473CVE-2018-20685 CVE-2019-6109CVE-2019-6111

CSCvp7646340(2a)

Multiple vulnerabilities associated withOpenSSL and LibNSS

For more information see the relatedCisco Security Advisory

CVE-2016-2105 CVE-2016-2106CVE-2016-2107 CVE-2016-2109CVE-2016-2176 CVE-2016-2177CVE-2016-2178 CVE-2016-2179CVE-2016-2180 CVE-2016-2181CVE-2016-2182 CVE-2016-2183CVE-2016-6302 CVE-2016-6303CVE-2016-6304 CVE-2016-6305CVE-2016-6306 CVE-2016-6307CVE-2016-6308 CVE-2016-7055CVE-2016-8610 CVE-2017-3731CVE-2017-3732 CVE-2017-3735CVE-2017-3736 CVE-2017-3737CVE-2017-3738 CVE-2018-0495CVE-2018-0732 CVE-2018-0734CVE-2018-0735 CVE-2018-0737CVE-2018-0739 CVE-2018-12384CVE-2018-12404 CVE-2018-5407CVE-2019-1559

CSCvp6667940(2a)

libpng vulnerability identified byCVE-2016-10087

For more information see the relatedCisco Security Advisory

CVE-2016-10087CSCvp6655540(2a)

Vulnerabilities associated with libjpg andlibpng

For more information see the relatedCisco Security Advisory

CVE-2014-9092 CVE-2016-3616CVE-2017-15232 CVE-2018-11212CVE-2018-11213 CVE-2018-11214CVE-2018-1152 CVE-2018-13785

CSCvp3458640(2a)

Vulnerabilities associated with Glibidentified by CVE-2018-16428 andCVE-2018-16429

For more information see the relatedCisco Security Advisory

CVE-2018-16428 CVE-2018-16429CSCvp3120740(2a)

Multiple vulnerabilities associated withcurl expat python 27 python 35 36and 37 freetype and giflib

For more information see the relatedCisco Security Advisory

CVE-2019-15133 CVE-2019-15903CVE-2019-5010 CVE-2019-5481CVE-2019-5482 CVE-2019-9636CVE-2019-9740 CVE-2019-9947CVE-2019-9948

CSCvr3690340(2a)

Release Notes for Cisco HX Data Platform Release 4025

Security Fixes

DescriptionCVEDefect IDRelease

Multiple third party softwarevulnerabilities

For more information see the relatedCisco Security Advisory

CVE-2019-14379 CVE-2019-12384CVE-2019-14439

CSCvq9203240(2a)

Vulnerabilities associated with elfutils

For more information see the relatedCisco Security Advisory

CVE-2018-16062 CVE-2018-16402CVE-2018-16403 CVE-2018-18310CVE-2018-18520 CVE-2018-18521CVE-2019-7149 CVE-2019-7150CVE-2019-7665

CSCvq4325040(2a)

Vulnerabilities associated with vim

For more information see the relatedCisco Security Advisory

CVE-2017-5953 CVE-2019-12735CSCvq4323040(2a)

Vulnerabilities associated with sqlite3

For more information see the relatedCisco Security Advisory

CVE-2016-6153 CVE-2017-10989CVE-2017-13685 CVE-2017-2518CVE-2017-2519 CVE-2017-2520CVE-2018-20346 CVE-2018-20505CVE-2018-20506 CVE-2019-8457CVE-2019-9936 CVE-2019-9937

CSCvq4321340(2a)

Vulnerabilities associated with glib20

For more information see the relatedCisco Security Advisory

CVE-2018-20843CSCvq4320940(2a)

Vulnerabilities associated with expat

For more information see the relatedCisco Security Advisory

CVE-2018-20843CSCvq4320540(2a)

Vulnerabilities associated with bzip2

For more information see the relatedCisco Security Advisory

CVE-2016-3189 CVE-2019-12900CSCvq4319440(2a)

Mulitple Vulnerabilities associated wtihNodeJS

For more information see the relatedCisco Security Advisory

CVE-2018-12115 CVE-2018-0734CVE-2018-5407 CVE-2018-12120CVE-2018-12121 CVE-2018-12122CVE-2018-12123 CVE-2018-12116CVE-2019-5737 CVE-2019-5739

CSCvq1069440(2a)

Vulnerabilities associated with PalletsJinja strformat_map and identified byCVE-2019-10906 and CVE-2016-10745

For more information see the relatedCisco Security Advisory

CVE-2019-10906 CVE-2016-10745CSCvq1038840(2a)

Release Notes for Cisco HX Data Platform Release 4026

Security Fixes

DescriptionCVEDefect IDRelease

Multiple Security Vulnerabilities

For more information see the relatedCisco Security Advisory

CVE-2014-8501 CVE-2014-9939CVE-2015-9262 CVE-2016-10087CVE-2016-2226 CVE-2016-4487CVE-2016-4488 CVE-2016-4489CVE-2016-4490 CVE-2016-4491CVE-2016-4492 CVE-2016-4493CVE-2016-6131 CVE-2018-10963CVE-2018-13785 CVE-2018-17100CVE-2018-17101 CVE-2018-18557CVE-2018-18661 CVE-2018-7456CVE-2018-8905

CSCvq0917840(2a)

Vulnerabilities associated withlibseccomp

For more information see the relatedCisco Security Advisory

CVE-2019-9893CSCvq0756840(2a)

Vulnerabilities associated withgdk-pixbuf

For more information see the relatedCisco Security Advisory

CVE-2017-12447CSCvq0675540(2a)

Vulnerabilities associated withpython-crypto

For more information see the relatedCisco Security Advisory

CVE-2018-6594CSCvp9381740(2a)

Vulnerabilities associated with wget

For more information see the relatedCisco Security Advisory

CVE-2018-20483 CVE-2019-5953CSCvp8672140(2a)

Multiple vulnerabilities associated withPython Crypto

For more information see the relatedCisco Security Advisory

CVE-2018-6594CSCvp6674840(2a)

Vulnerabilities associated withsensible-utils

For more information see the relatedCisco Security Advisory

CVE-2017-17512CSCvp6673440(2a)

Multiple vulnerabilities associated withPython

For more information see the relatedCisco Security Advisory

CVE-2018-1000030CSCvp6668940(2a)

Release Notes for Cisco HX Data Platform Release 4027

Security Fixes

DescriptionCVEDefect IDRelease

Multiple patch vulnerabilities identifiedby CVE-2016-10713

For more information see the relatedCisco Security Advisory

CVE-2016-10713CVE-2018-1000156 CVE-2018-6951

CSCvp6667240(2a)

Vulnerabilities associatedwith Little CMSidentified by CVE-2016-10165

For more information see the relatedCisco Security Advisory

CVE-2016-10165 CVE-2018-16435CSCvp6666440(2a)

Vulnerabilities associated with TenableScan identified byCVE-2011-5325CVE-2019-11068CVE-2016-7076CVE-2017-1000368

For more information see the relatedCisco Security Advisory

CVE-2011-5325 CVE-2016-7076CVE-2017-1000368CVE-2019-11068

CSCvp6474640(2a)

Vulnerabilities associated with libxml2294 and earlier as used in XMLSec1223 and earlier and other products andidentified by CVE-2016-9318

For more information see the relatedCisco Security Advisory

CVE-2016-9318 CVE-2017-16932CVE-2017-18258 CVE-2018-14404CVE-2018-14567

CSCvp3479240(2a)

Vulnerabilities associated with LFTPRemote File Names Unauthorized AccessVulnerability and identified byCVE-2018-10916

For more information see the relatedCisco Security Advisory

CVE-2018-10916CSCvp2926640(2a)

Vulnerabilities associated withCryptographyDeprecationWarning signerand verifier have been deprecated

For more information see the relatedCisco Security Advisory

CVE-2018-7750CSCvo3409740(2a)

Multiple Vulnerabilities in python python27 35

For more information see the relatedCisco Security Advisory

CVE-2019-16056CSCvr3979340(2a)

Release Notes for Cisco HX Data Platform Release 4028

Security Fixes

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toexecute commands as the root user

The vulnerability is due to insufficientinput validation An attacker could exploitthis vulnerability by connecting to thecluster service manager and injectingcommands into the bound process Asuccessful exploit could allow the attackerto run commands on the affected host asthe root user

Cisco has released software updates thataddress this vulnerability There areworkarounds that address thisvulnerability

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvq2417640(1b)35(2g)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

The vulnerability is due to an unprotectedlistening interface An attacker couldexploit this vulnerability by connectingto the listening interface and injectingcommands to the bound process Anexploit could allow the attacker to runcommands on the affected host as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvj9560640(1b)35(2g)

Release Notes for Cisco HX Data Platform Release 4029

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerabilities associated with theJVM18U121memory leak found duringVC alarms (concurrent 40 calls usingREST API)

CVE-2017-10053 CVE-2017-10067CVE-2017-10074 CVE-2017-10078CVE-2017-10081 CVE-2017-10087CVE-2017-10089 CVE-2017-10090CVE-2017-10096 CVE-2017-10101CVE-2017-10102 CVE-2017-10107CVE-2017-10108 CVE-2017-10109CVE-2017-10110 CVE-2017-10111CVE-2017-10115 CVE-2017-10116CVE-2017-10118 CVE-2017-10135CVE-2017-10176 CVE-2017-10193CVE-2017-10198 CVE-2017-10243CVE-2017-10274 CVE-2017-10281CVE-2017-10285 CVE-2017-10295CVE-2017-10345 CVE-2017-10346CVE-2017-10347 CVE-2017-10348CVE-2017-10349 CVE-2017-10350CVE-2017-10355 CVE-2017-10356CVE-2017-10357 CVE-2017-10388CVE-2017-3509 CVE-2017-3511CVE-2017-3526 CVE-2017-3533CVE-2017-3539 CVE-2017-3544CVE-2018-2579 CVE-2018-2582CVE-2018-2588 CVE-2018-2599CVE-2018-2602 CVE-2018-2603CVE-2018-2618 CVE-2018-2629CVE-2018-2633 CVE-2018-2634CVE-2018-2637 CVE-2018-2641CVE-2018-2663 CVE-2018-2677CVE-2018-2678 CVE-2018-2783CVE-2018-2790 CVE-2018-2794CVE-2018-2795 CVE-2018-2796CVE-2018-2797 CVE-2018-2798CVE-2018-2799 CVE-2018-2800CVE-2018-2814 CVE-2018-2815CVE-2018-2952 CVE-2018-3136CVE-2018-3139 CVE-2018-3149CVE-2018-3150 CVE-2018-3169CVE-2018-3180 CVE-2018-3183CVE-2018-3214 CVE-2019-2422

CSCvo8899740(1b)

Tomcat and Nginx logs are not beingcollected in the support bundle generatedthrough HX Connect in Release 35

NACSCvm5803140(1b)

The vulnerabilities associated with thelibmspack software package versionincluded in Cisco HX Data Platform

CVE-2018-18584

CVE-2018-18585

CSCvn3511940(1a)

Release Notes for Cisco HX Data Platform Release 4030

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerabilities associated withFasterXML Jackson-Databind TimeValueField Denial of Service

CVE-2018-14719

CVE-2018-14720

CVE-2018-1000873

CVE-2018-14721

CVE-2018-19360

CVE-2018-19362

CVE-2018-19361

CVE-2018-14718

CSCvn8228240(1a)

The vulnerabilities associated with theOpenSSL Protocol software packageversion included in Cisco HX DataPlatform

CVE-2013-3587CSCvo0505440(1a)

The vulnerabilities associated withThird-Party Software Denial of Service

CVE-2018-16487

CVE-2018-19361

CSCvo2781840(1a)

The vulnerabilities associated withLinuxkernel for Ubuntu 1710

CVE-2018-1092

CVE-2018-7492

CVE-2018-8087

CVE-2018-1068

CVE-2018-8781

CSCvm5314935(2a)

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)25(1d)stcli services dns remove should remove the DNS serverinfo from the interface files

CSCvs70967

40(2b)40(1b)HX 40(1b) new deployment from Intersight productioncloud

User receives following error while enabling CSI(Kubernetes integration)

Failure occurred during volume_access_enable Errorwas Nonzero exit code 1

CSCvt10522

40(2b)40(2a)Receive the following alertevent in HX Connect HXController VM HOSTNAME one or more configuredDNS servers not responding

CSCvt13947

Release Notes for Cisco HX Data Platform Release 4031

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)40(2a)API UI CLI show few Drive slots Empty after upgradeto 402a

The drives are seen in the sysmtool lsscsi and stclicommand outputs

The cluster is healthy with no errors

CSCvt14914

40(2b)35(2d)After successfully changing the DNS server on the HXcontroller we still can see the original DNS entry thatwas added during the deployment

CSCvs69154

40(2b)35(2e)HyperFlex Connect and VCenter show APD alarms forHX and non HX datastores This implies there is an issuewith HyperFlex when there may not be

CSCvs30080

40(2b)40(2a)An issue where all the disks in the node get locked afterrebooting the node

We successfully unlocked using sed-clientsh -Ucommand but wanted to test with another reboot anddrives locked again

CSCvs74286

40(2b)40(2a)When running stcli license commands on HyperFlexthe following error is seen

rootSpringpathControllertmp stcli license show all

Show smart licensing failed

Smart Agent is not ready please wait a minute and tryagain

CSCvt13929

40(2b)40(2a)After upgrading to HX 40(2a) the first RoboEdge nodethat gets rebooted then ignores persistent drivesPIDHX-HD24TB10K4KN

Cluster will be in degraded state and have reducedcapacity

CSCvt20144

40(2b)35(2b)Zookeeper fails to start while Exhibitor is runninghowever echo srvr returns nothing

CSCvs21562

40(2b)40(2a)When performing an HyperFlex upgrade a validationwarning may occur due to the host not having EnterprisePlus or Enterprise hypervisor licensing

Upgrade Validation Warning

ESXi host esx1labtest should be configured withVMware Enterprise license for upgrade to continue

CSCvs91787

Release Notes for Cisco HX Data Platform Release 4032

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)35(2g)Cluster expansion fails at Config Installer stage whenthe root and admin password for the storage controller(SCVM) are different

CSCvs69317

40(2b)35(2g)Panic while upgrading ESXiCSCvt06983

40(2b)40(2a)A cluster node running may hang in the Linux kernelThis is classified as an oops and a deviation from theexpected behavior

CSCvs54285

40(2b)35(2g)Rebalance failing on 10+10 node HX 35(2g) stretchcluster

CSCvs69007

40(2b)35(2d)The cluster becomes inaccessible to the IOVisorCSCvr54687

40(2b)40(2a)Panic on the storage controllerCSCvt61297

40(2b)40(2a)The size of support bundle is very large when collectedwith storfs-support command

CSCvt63306

40(2b)40(2a)storfs crash associated withFileSystemUsageWarningEvent orFileSystemUsageAlertEvent due to high usage ofvarstv above 80

CSCvt72807

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

40(2a)35(2d)In normal scenarios HX upgrade gets pendingACK afterupdating service profile In this case UCS only upgradewas stalled because after updating service profile HXupgrade was not getting pending ACK and it was alwayswaiting state

CSCvo39912

40(2a)35(1a)When using motherBoardReplace-12 to clean ZK fromold stNodepNodes resulted in unmounted datastore andmaking the size of the Hyperflex Datastores 0 resultingin all VMs in the cluster going offline

CSCvq39471

40(2a)26(1e)Upgrading cluster and getting error

Failed upgrade validations Checking vCenterconfiguration Reason Upgrade validations failed DRSFault Insufficient resources to satisfy configured failover

CSCvm77294

Release Notes for Cisco HX Data Platform Release 4033

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)The cluster expand fails on a node with DR replicationconfigured

When a HX cluster which has DR replication configuredis expanded we see the installer UI pulling in thereplicationVLAN information instead of themanagementVLAN information Even if we change that informationto the correct mgmt VLAN id and name it it does notseem to work as the node is configured with the VLANof the replicationVLAN in ESXi This leads to the failureof the node add with the host unreachable error

CSCvo70650

40(2a)35(1a)Customer reported that server didnt complete thefirmware upgrade automatically one by one

The user manually needs to put the host into maintenancemode and they manually acknowledged the pendingrequests to finish the UCS firmware upgrade

Maintenance policy is set to default (User-ack) as perthe design

CSCvo91624

40(2a)35(1i)When the cluster is in Failed state and stcli noderemove is attempted the output shows successful thoughthe node failed to get removed from the cluster

CSCvo93017

40(2a)35(1a)HyperFlex cluster upgrade may fail during validationwith a DRS Validation failed error

CSCvp31021

40(2a)35(2b)HX cluster expansion will fail with the following errormessage

MAC address pool configuration failure150[ErrorDescription] bad address block range definitioncollision

CSCvp58318

40(2a)35(2b)Memory usage by carbon cacheCSCvq34873

40(2a)35(2g)Upgrading ESXi cluster fails with error Nodemaintenance mode failed

CSCvr03240

40(2a)35(2e)storfs process does not automatically start on Exit HXMaintenance Mode or other tasks which restart StorageController

CSCvr88978

40(2a)35(2a)Post install script fails with message

Failed to execute ipmitool on HX node

CSCvp10707

40(2a)40(1a)HyperFlex expansionworkflow doesnt pull VLANnamecorrectly

CSCvp46539

Release Notes for Cisco HX Data Platform Release 4034

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)During HX cluster deployment - Validate ClusterCreation phase - HX Installer might fail with the error

fromvarlogspringpathstDeploylog

CSCvq45087

40(2a)35(2b)HX Installer fails to login to the SOL for configuringESX hosts

CSCvq91380

40(2a)35(2d)If DRS is enabled with configuration parameters otherthen fully automated ie manual or partially automatedthan this will change to fully automated during clusterexpansion

CSCvr44222

40(2a)35(1a)One of the validation task name has a typo The taskname is Validating node for mixed of SED and non-SEDdisks

It should be Validating node for mix of SED andnon-SED disks

CSCvp82175

40(2a)30(1i)This combination of using smaller NVMe drives is notsupported in expand operation For example if the drivesare only 375GB and they cannot be added to an existingcluster with larger caching SSD

CSCvo12359

40(2a)35(1a)HyperFlex includes a version of OpenSSL that is affectedby the vulnerabilities identified by the followingCommon Vulnerability and Exposures

CVE-2018-0495

CSCvp66679

40(2a)30(1e)SDK invoke exception when taking a native snapshot ona VM with GPU pci passthrough attached

CSCvo00511

40(2a)30(1i)Node replacement script fails due to EAM errorCSCvo62867

40(2a)35(1a)While pairing a cluster (HX release gt= 35) and cluster(HX release lt 35) remote replication network test failson the cluster (HX release lt 35)

CSCvo79760

40(2a)30(1b)Fixed the issue in the latest support-workflow bundleThis should not be hit anymore

CSCvo87061

40(2a)35(2a)On a 3 node cluster the MBreplace script fails as thescript looks for HEALTHY cluster status

CSCvo87080

40(2a)35(2a)MbReplace script (tar) hangs when running on 352b or40

CSCvp12359

Release Notes for Cisco HX Data Platform Release 4035

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)HX replication cleanup failing with error NA 35(2a) -INFODR state is not clean

CSCvp63958

40(2a)40(1a)The check_stig_parameters API incorrectly shows thecompliance state of the cluster as not STIG compliant

CSCvp66277

40(2a)35(2c)The snapshot creation on CBT enabled VM fails witherror Failed in vmreparent vmkfstools clone1

CSCvq06952

40(2a)35(2a)HyperFlex GUI when setting up remote DR-replicationpartnerpeer should show remote replication partner TCPport reachability test results

This will provide quick upfront connectivity results thatfrequently takes customer a while to manually check andconfirm

If reachability tests fail an info window and HX alertshould trigger that list the destination peer IP and thespecific port(s) that could not be reache

CSCvo60587

40(2a)35(2b)If you have more than one IP pool configured for areplication network and local replication network testsare failing you might be running into this issue

CSCvr67130

40(2a)35(2a)Skip task option in MBreplace and Software redeployscripts The scripts should have the option to skip tasksas user inputs and not by modifying the json file

CSCvp05204

40(2a)40(1a)Error noticed on SCVM Console print_req_error IOerror dev fd0 sector 0 on HyperV

It doesnt come up on SSH session

CSCvq34357

40(2a)30(1d)The VMNetwork switch is already created as part of theinstall process

Ideally the ip address assignment should be done duringthe initial install process as well No post install stepsshould be required altrnatively it could be done as partof a post install script

CSCvk36222

40(2a)25(1c)storfs service impacted during HX upgrade from HX21(1) to 25(1c)

HX node which is not being upgraded storfs servicestopped causing outage

CSCvg53223

40(2a)35(1b)HX Stretch Cluster Witness VM reverts to DHCP onreboot

CSCvp37536

40(2a)35(2b)HyperFlex Stretch Cluster Witness does not getprogrammed with an NTP server

CSCvp46578

Release Notes for Cisco HX Data Platform Release 4036

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2b)HyperFlex cluster does not heal after maintenance onnodeserver

CSCvr18528

40(2a)40(2a)High percentage of packet loss on the eth1 data interface9K MTU

CSCvr97089

40(2a)35(2f)The HyperFlex cluster healing state is stuck at 87 andis unable to progress

CSCvr16760

40(2a)40(2a)In some circumstances two replications can occur forthe same VM

From the GUI the replication status for this VM willshow stuck in an In Progress state but there are nojobs stuck in replication layer Subsequent Replicationsshould succeed

CSCvq22898

40(2a)35(2a)After an unexpected power outage the hyperflex clusteris unable to come up correctly

All processes are running time is synced and all nodescan vmkping each other as required

CSCvp13990

40(2a)40(1a)When specifying IP address instead of node ID stclinode maintenanceMode commands fail

CSCvp33657

40(2a)40(1a)2 node edge cluster if experiencing Intersightarbitratorconnectivity issues should NOT allowHXMaintenancemode to initiate as this will bring the cluster down

CSCvr31573

40(2a)35(2c)Old files in varsupportZKTxnlog are not purged withthe daily zklog-cleanup cron job

CSCvr89066

40(2a)35(2a)When a node is in MaintenanceMode any disk removalor replacement will be reflected in UI only after the nodeis brought back frommaintenancemode This is becausestorfs is not running on the node in maintenance andwill not be able to detect disk activities until it is broughtout of Maintenance Mode

CSCvo86431

40(2a)35(2c)Cluster might go offline and stop serving IO requestsrootcvm~ stcli cluster storage-summary--detailGet cluster storage summary failedjavanetConnectException Connection refused19216814210010207

CSCvr01645

40(2a)35(2d)A stretch cluster deployment fails with a messageFormatting nodes Node in Use on the Hyperflex installerand crmZoneType value is seen as 1 in storfscfg oroptspringpathconfigstretchtunes

CSCvq80340

Release Notes for Cisco HX Data Platform Release 4037

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)30(1c)During upgrade we upload the upgrade packagedownloaded from Ciscocom Upgrade packages are offile type tgz hxconnect acceptsstorfs-packages-401a-33028tar file a different type ofcompression and allows upgrade to start and fail whenit cannot decompress the file for bootstrap and scvmclientupgrade on ESXi

CSCvq63888

40(2a)40(2a)While attempting to install hyperflex via intersight thefollowing error is seen

Failed in Task Add hosts to vCenter Cluster with ErrorTry adding hosts manually to vCenter and retry Failedto get addressFailed to host by namelookupHostnamecompanycom on 000053 read udp12700145430-gt12700153 read connection refused

CSCvr52098

40(2a)35(2a)CoreAPI call may time out for cluster management APIwhile doing cluster create validation

CSCvo92952

40(2a)35(2d)Validation fails with Mixed mode expansion checkdue to Enhanced vMotion Compatibility Incompatibilitywhen the ESXi version is different

CSCvq95460

40(2a)30(1i)This failure is seen in installer when custom workflowis used hypervisor configuration + deploy (clean diskpartition=no) at Installing software packages on storagecontroller VM

CSCvr66309

40(2a)35(2d)The HX Installer apparently replaced a comma bysemicolon on the VLAN ID range while exporting theconfiguration to a JSON file

CSCvq93831

40(2a)40(1a)When using unsupported browsers HX Connect userswill not be able to login These users see an errorindicating unauthorized user

CSCvp22693

Release Notes for Cisco HX Data Platform Release 4038

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)40(1a)A vulnerability in the web-based management interfaceof Cisco HyperFlex Software could allow anunauthenticated remote attacker to conduct a cross-siterequest forgery (CSRF) attack on an affected system

The vulnerability is due to insufficient CSRF protectionsfor the web UI on an affected device An attacker couldexploit this vulnerability by persuading a user of theinterface to follow a malicious link A successful exploitcould allow the attacker to perform arbitrary actions withthe privilege level of the affected user

Cisco has released software updates that address thisvulnerability There are no workarounds that address thisvulnerability

This advisory is available at the following link

httpstoolsciscocomsecuritycentercontentCiscoSecurityAdvisorycisco-sa-20190807-hypflex-csrf

CSCvp24343

40(2a)35(2d)Add message to HX Connect Progress flow to notAcknowledge Pending Activities and reboot the serverson UCSM

HX Connect is doing a controlled rolling server upgradein the background

CSCvq22844

40(2a)40(1b)Compute Node expand operation fails due to incorrectcluster name

CSCvr43786

40(2a)26(1e)RF-2 cluster shutdown during data resynchronizationafter a node failure (non storfs) followed by multipledisk read failures and a HardBlacklist of a disk

CSCvp41241

40(2a)40(1a)HyperFlex Edge nodes do not properly set the ESXihostname during deployment

CSCvo13143

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4039

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4040

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1b) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401b-33133p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4041

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4042

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1b)40(1a)While running the HyperFlex installer with WindowsServer Hyper-V cluster creation process fails with theerror Failure occurred during ClusterCreation process Unable to post thecontent to the downstream This symptomis encountered while deploying HyperFlex cluster withHyperFlex nodes configuredordered with Cisco VIC1457 MLOM (PID HX-MLOM-C25Q-04) andWindows Server Datacenter or Core with Hyper-Vversion 2016 or 2019

CSCvp64140

Release Notes for Cisco HX Data Platform Release 4043

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)35(2b)Adding Micron 5200 drive to a cluster fails to increasecluster capacity

CSCvo69067

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4044

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4045

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1a) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401a-33028p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4046

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4047

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1a)30(1d)Cluster instability when disks of different sector sizeplaced in HX node

CSCvk17250

40(1a)

35(2c)

35(1a)When logged in using a local HX user account insteadof a Virtual Center account an error message appearsintermittently indicating Virtual Centerunreachable or Resource informationcannot be updated when VC is reachable

CSCvo36198

40(1a)

35(2b)

30(1d)HXDP does not work with EMC RecoverPoint - needsto support VMware API (FSS-Readdir)

CSCvk38003

Release Notes for Cisco HX Data Platform Release 4048

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)

35(2a)

30(1i)

25(1c)

25(1d)

Zookeeper (Exhibitor) process on HyperFlex StorageController VMs may fail to respond to requests whenvarzookeeper has low or no free space

CSCvm90352

40(1a)26(1c)Use Asynchronous consolidation for HX SnapshotsCSCvn02151

40(1a)35(2b)HX Quiesced Snapshot for Backup VendorsCSCvo90713

40(1a)35(2a)The cluster creationcluster expansion workflow stopswith the following error message at the validation step

FIRMWARE-Check UCSC-SAS-M5HD

FIRMWARE-Check UCSC-SAS-M5HD Required00000029000000320000003500000050 Found00000058

Action Needed Update the Controller Firmware toRequired Version

CSCvn17787

35(2b)30(1d)Kernel migration fails when a local datastore is searchedfor in the ESXi host

CSCvn73127

35(2a)26(1b)A node shuts down when two disks are replaced (acaching disk and another capacity disk) where thecapacity disk is inserted first and then the caching diskis inserted

CSCvk46364

Hyper-V

40(1a)35(2a)Cluster expansion may fail with an error code 500 -operation timed out

CSCvn28721

35(2b)35(2a)During upgrade remove the VLAN on team created foruser vSwitch During fresh installation only one VLANtag is set to the vSwitch and team although multipleVLANs were entered

CSCvn54300

35(2b)35(2a)While upgrading a Hyper-V cluster on account of a rarerace condition between the stUpgradeService andZookeeper servers the upgrade orchestration throws anupgrade validation error and the upgrade process isaborted

CSCvn60486

Release Notes for Cisco HX Data Platform Release 4049

Resolved Caveats in Release 40(1a)

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Startrestart EAM service invCenter

When EAM service is not runningvCenter registration is failing

CSCvt55712

35(2a)Contact Cisco TAC if you believeyou are seeing similar conditions

HyperFlex Stretch Cluster saw poorapplication performance due to writelatency cluster remained unhealthy andrebalance was stuck

This defect is currently being used forinvestigative purposes

CSCvt45344

35(2g)Remove log files fromvarlogspringpath folder when thelog files are not getting rotatedStop the relevant service removelarge log files fromvarlogspringpath folder andrestart the service

varstv folder in Controller VM maybecome full

CSCvt36374

40(2b)Restarting the stMgr will initializeit again

In a Cluster configured with DR networkStMgr may not get initialized and stuckin a deadlock while enabling the IPTablerules (can be verified from the stMgrlog)

CSCvt89709

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Click recover for the selectedVM again under the standalonemode This will move the VM torecovered state

When the user recovers a VM protectedin a group it along with other VMs in thegroup move to recovering state instandalonemode The selectedVM shouldmove to recovered state

CSCvs75553

35(2e)Clean up the fail over cluster andthen recreate the fail over clusterNo need to touch the HX storagecluster

Hyper-V When replicated DC was usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2d)40(1a)

Keep cluster utilization as low aspossible To avoid read-only statekeep below 92

HyperFlex clusters storage utilization isabove 92 usable capacity

CSCvr20922

Release Notes for Cisco HX Data Platform Release 4050

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1b)NAAll the disks in the node got locked afterrebooting the node

We successfully unlocked usingsed-clientsh -U command but wantedto test with another reboot and driveslocked again

CSCvs74286

35(2g)To recover the VM copy over thedata disks and attach them to thenew VM

After enable encryption HX connectencryption status showing Caution butall of the disks enable encryptionsuccessful

-Self Encrypting Drives Service isrunning on all of the nodes

-There is no error message of encryption

- USB0 interface is up

-All SED disks showing supported 1enabled 1 locked 0

CSCvs93245

40(1b)Enable kernelpanic_on_oops as apersistent configuration This willcause the node to panic and rebootimmediately

A cluster node running HX release40(1b) may hang in the Linux kernelThis is classified as an oops and adeviation from the expected behavior

CSCvs54285

Release Notes for Cisco HX Data Platform Release 4051

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Enabling K8iscsi stack in Hx clusterCSCvs41324

Release Notes for Cisco HX Data Platform Release 4052

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Steps to enable K8 on HX cluster

Perform following steps on ALLthe controller VMs

1 Updateetcinitscvmclientconf toenable tune iscsiEnable totrue

sed -iesiscsiEnable=falseiscsiEnable=trueetcinitscvmclientconf

2 Run following initctlcommand to reloadconfiguration

initctl reload-configuration

3 Restart scvmclient process

stop scvmclient startscvmclient

4 Run following initctlcommand to mount SYSTEMDS

initctl emit --no-waitsystem-datastore-created

5 Verify that iscsiEnable tuneis set to true using thefollowing command

ps -eaf | grep scvmclient |grep -v grep

root 6241 1 1 Dec06 014007optspringpathstorfs-corescvmclient-T logEnabled=false -TlogSyslogEnabled=true -TlogEchoToScreen=false -TstatLoggingToFile=false -TstatLoggingToSyslog=true-TlogDir=varlogspringpath-TnfsBackendServerList=1010748100-T iscsiEnable=true -TiscsiUseAsync=true -T

Release Notes for Cisco HX Data Platform Release 4053

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

iscsiConfigFilePath=nfsSYSTEMistgtconfps -eaf | grep scvmclient |grep -v grep

35(2e)Following procedure can be usedto check NFS Queue Depth andincrease if needed

rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth256 lt- Low value[rootHXESXI1] vsish -e setvmkModulesnfsclientmnt[DATASTORE]propertiesmaxQDepth 1024[rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth1024 lt- Optimalvalue

This requires an ESXi host rebootto take effect

Please place the node in HyperflexMaintenanceMode and gracefullyreboot the node for the changes tobe applied

HyperFlex Datastore NFS Queue Depthshows as 256 which can lead toperformance (including latency) issues

CSCvr83056

40(1b)This is cosmetic The error will goaway once the cluster returns tohealthy status

When a single node is offline in clusterstcli cluster storage-summary shows twonodes unavailable rootSCVM~ stclicluster storage-summary

CSCvs47735

35(2g)Modify both the root and adminpassword for the SCVMs to be thesame

stcli security password set -uadmin

Cluster expansion fails at Config Installerstage when the root and admin passwordfor the storage controller (SCVM) aredifferent

CSCvs69317

35(2d)NAAfter successfully changing the DNSserver on the HX controller we still cansee the original DNS entry that was addedduring the deployment

CSCvs69154

Release Notes for Cisco HX Data Platform Release 4054

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Customers are strongly encouragedto work with TAC in order toidentify and remediate this issue

1 Change getcluster from 2m to8m inoptspringpathstorfs-mgmtstMgr-10confapplicationconfon all nodes

2 Restart stMgr on all nodes

You may see this error message after afailed upgrade or other task such asattempting to enter a node into HXmaintenance mode

CSCvs53555

40(2a)Run the following command oneach storage controller VM as root

grep -i monitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp sed -iesmonitor_dns_serverstruemonitor_dns_serversfalseoptspringpathhx-diag-toolswatchdog_configjsonampamp grep -imonitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp restart watchdog

Receive the following alertevent in HXConnect HX Controller VMHOSTNAME one or more configuredDNS servers not responding

CSCvt13947

35(2b)Delete any empty (Size 0) log filesunder varlogzookeeperversion-2ORvarlogzookeeperstandaloneversion-2

Zookeeper fails to start while Exhibitor isrunning however echo srvr returnsnothing

CSCvs21562

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

40(4e)1 De-commission the server

2 Power drain the server -REMOVE BOTH POWERCORDS ON THE BACK OFTHE SERVER FOR 10SECONDS THENREINSERTPOWERCORDS

3 Re-commission the server

M2 boot disk is missing from serverinventory after upgrade to server firmware40(4e) As a result server fails to boot toOS installed in the M2 disk The issuepersists after re-acknowledgement as wellas de-commission andre-acknowledgement of the server

CSCvs02466

Release Notes for Cisco HX Data Platform Release 4055

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Re-start the flusher manuallyMany disks taken offline by a fault ormanually resulting in ENOSPC oncluster

CSCvq32721

35(2e)Clean up fail over cluster and thenrecreate the fail over cluster Thereis no need to touch the HX storagecluster

[Hyper-V]When replicated DCwas usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2b)Manually install upgradescvmclient VIB andmake sure thatit matches with HXDP version

Upgrade from release HX 35(2a) to35(2x) 4X does not upgrade scvmclientRecovery point may not work

CSCvq54992

Management

30(1b)To recover the VM copy over thedata disks and attach them to thenew VM

[HyperV]VMshows up onmultiple nodesCSCvj22992

35(1a)1 Do not change MTU at ESXlevel on running cluster

2 Reset back to the original value

ChangingMTU from 9000 to 1500 on oneESX node causes storfs process to restarton all nodes

CSCvm99150

Release Notes for Cisco HX Data Platform Release 4056

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)After the installation (for newdeployments) or after an upgradeto 401a (for existingdeployments) perform thefollowing steps on all ControllerVMs

1 Edit the following tune files (onall Controller VMs)

optspringpathconfiglfftunes

optspringpathconfigvsi_16tbtunes

2 SetcleanerEnableSegSummaryCleaning

to false

3 After editing the above tunefiles

a SSH in to all the ControllerVMs

b Run storfstool -- -Z

c Run the below command andcheck the values Tune valueshould be true

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=true

d Type below commands andapplied the tunes changesdynamically

echo false gttmpstprocfssystemtunecleanerEnableSegSummaryCleaning

e Verify that tune values arechanged Run below commandValue should be false

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=false

f umount tmpstprocfs

A cluster with 8TB or 12TB disk drivesmay experience IO stalls for severalminutes after another node fails in thecluster

CSCvp23718

Release Notes for Cisco HX Data Platform Release 4057

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)1 Update the conf file manuallyto change to the link local addressto

2 Restart scvmclient on thecontroller and ESX

HX 35(2b) FlexVol to HX 40 CSIupgrade does not work FlexVol continuesto work

CSCvp26319

35(2a)Disable rebalance until node healsthen reenable rebalance

7-node cluster healing very slowlybecause rebalance change AMS reqs areblocking healing process

CSCvp62512

35(2a)NAWith moderate cluster storage capacityusage under site failover cluster clustermay experience an all paths (APD) downcondition

CSCvq49412

40(1a)NAThis product includes Third-partySoftware that is affected by thevulnerabilities identified by the followingCommon Vulnerability and Exposures(CVE) IDs

CVE-2016-0762 CVE-2016-6797CVE-2016-6816 CVE-2016-8735CVE-2017-5647 CVE-2017-12615CVE-2017-12616 CVE-2017-12617CVE-2017-7674 CVE-2018-1304CVE-2018-8014 CVE-2018-1336CVE-2018-8034 CVE-2018-11784CVE-2019-0232

CSCvp36364

35(2d)NAstcli cluster info command needs toprovide UCSM VIP address Currently itshows ucsm-hostcom

CSCvq11456

35(2a)No workarounds currently Thestale entries do not affect the dayto day operations of HyperFlex

HyperFlex upgrade validation failedbecause cluster has extraneous stNodes ininternal database

CSCvq32530

35(2a)NAWhen not in maintenancemode all drivesappear in HX Connect and get updated asexpected

When in maintenance mode extra drivesare shown in HX Connect with no slotnumbers

CSCvq42378

35(2b)Contact TAC to cleanup duplicatenode

Nodes duplicated in HX Connect aftercluster recreate

CSCvq61825

Release Notes for Cisco HX Data Platform Release 4058

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)NACurrently HyperFlex installs on Hyper-Vdo not contain the stcli security whitelistcommandset

CSCvq66245

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

Release Notes for Cisco HX Data Platform Release 4059

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)[Hyper-V] Converged node expansionfails for incoming node with VIC 14xx

CSCvq39523

Release Notes for Cisco HX Data Platform Release 4060

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

1 Set Jumbo frame inWindowsHypervisor for incomingnode

Get-NetAdapter storage-data-a|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

Get-NetAdapter storage-data-b|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

2 Verify all interfaces are setcorrectly to support Jumbo frames

Get-NetAdapter storage |Get-NetAdapterAdvancedProperty| RegistryKeyword -matchjumbo | ft -auto

You should receive the followingmessage

Name DisplayName DisplayValue

RegistryKeyword RegistryValue

---- ----------- ------------

--------------- -------------

vswitch-hx-storage-data Jumbo

Packet 9014 Bytes

JumboPacket 9014

storage-data-b Jumbo Packet

Bytes 9014 JumboPacket

9014

storage-data-a Jumbo Packet

Bytes 9014 JumboPacket

9014

3 Reboot only the incoming nodeor the node being expanded

4 Retry Cluster Expansion in

Release Notes for Cisco HX Data Platform Release 4061

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

DescriptionDateRelease

Updated M4 and M5Recommended FIServer Firmwareto UCS 40(4h) for HX 40(2a)

March 24 202040(2a)

Created release notes for Cisco HXData Platform Software Release40(2a)

February 11 202040(2a)

Updated release notes for deferredCisco HyperFlex Release HX35(2c)

January 15 202035(2c)

Updated M4 and M5Recommended FIServer Firmwareto UCS 40(4e) for HX 40(1b)40(1a) 35(2f) 35(2e) and35(2d)

December 23 201940(1b)

Added CSCvs28167 to the list ofResolved Caveats for HX 40(1b)and HX 40(1a)

December 13 201940(1b)

Added CSCvs02466 to the list ofOpen Caveats

November 25 201940(1b)

Updated info in the HyperFlexEdge and Firmware CompatibilityMatrix for 3x Deployments

Updated info in the Storage ClusterSpecifications

November 7 201940(1b)

Added CSCvj95606 andCSCvq24176 to the list of SecurityFixes

October 25 201940(1b)

Updated Recommended FIServerFirmware versions

October 8 201940(1b)

Updated HUUCIMC info in theHyperFlex Edge and FirmwareCompatibility Matrix for 4xDeployments

September 30 201940(1b)

AddedCSCvq41985 to new sectionfor Related Caveats

September 17 201940(1b)

Updated HUUCIMC info in theHyperFlex Edge and FirmwareCompatibility Matrix for 4xDeployments

September 16 201940(1b)

Release Notes for Cisco HX Data Platform Release 402

Revision History

DescriptionDateRelease

Updated HUUCIMC info in theHyperFlex Edge and FirmwareCompatibility Matrix for 3x and4x Deployments

September 10 201940(1b)

Updated HUUCIMCrecommended firmware versionsfor HyperFlex Releases 40(1b)35(2e) and 35(2d)

August 28 201940(1b)

Updated Recommended FIServerFirmware versions for HyperFlexReleases 35(2e) and 35(2d)

August 23 201940(1b)

Added Cisco IMC version supportinfo in the HyperFlex Edge andFirmware CompatibilityMatrix for4x Deployments

August 21 201940(1b)

Created release notes for Cisco HXData Platform Software Release40(1b)

August 19 201940(1b)

Added bullet describing the CiscoHyperFlex SystemsUpgradeGuidefor Unsupported Cisco HXReleases in the UpgradeGuidelines section

August 8 201940(1a)

Added important note indicatingHyperFlex does not support UCSserver firmware 40(4a) 40(4b)and 40(4c)

August 5 201940(1a)

Updated component info forHX220c M5HXAF220c M5Cluster to VIC 1457 in HyperFlexEdge and Firmware CompatibilityMatrix for 4x Deploymentssection

July 25 201940(1a)

bull AddedRelease 35(2e) supportfor ESXi 67 U2 and updatednotes for 35(2d) 35(2c) and35(2b) support for ESXi 67U2 in Supported VMwarevSphere Versions andEditions

bull Added Witness Node Versionfor Release 35(2e)

July 22 201940(1a)

Release Notes for Cisco HX Data Platform Release 403

Revision History

DescriptionDateRelease

Added CSCvq39523 to the list ofOpen Caveats for Release 40(1a)

July 5 201940(1a)

bull Updated Important Note forSED-based HyperFlexsystems in SupportedVersions and SystemRequirements section

bull Updated Release 35(2b)support for ESXi 67 U2 inSupported VMware vSphereVersions and Editions

July 1 201940(1a)

bull Added CSCvp64140 andCSCvp98910 to the list ofOpen Caveats for Release40(1a)

bull Updated HyperFlex Edge andFirmware CompatibilityMatrix tables

June 20 201940(1a)

Updated BrowserRecommendations

June 17 201940(1a)

Added information indicating40(1a) features are supported onthe Intersight Virtual Appliance andon Intersightcom

May 31 201940(1a)

bull Updated M4M5Recommended FIServerFirmware for 35(2b) and40(1a)

bull Added bullet describingrecommended use of theHypercheck Health CheckUtility in the UpgradeGuidelines section

May 21 201940(1a)

bull Added New Featuredescription for DISA STIGCompliance

bull Updated Storage ClusterSpecifications for Hyper-V

bull Added CSCvp66277 to list ofOpen Caveats

May 14 201940(1a)

Release Notes for Cisco HX Data Platform Release 404

Revision History

DescriptionDateRelease

bull Updated VMware vCenterVersions for 40(1a)

bull Updated SupportedMicrosoftSoftware versions

May 8 201940(1a)

bull Added CSCvo36198 andCSCvk38003 to the list ofResolved Caveats

bull Added CSCvp21417 to the listof Open Caveats

May 3 201940(1a)

Created release notes for Cisco HXData Platform Software Release40(1a)

April 29 201940(1a)

New FeaturesCisco HX Data Platform Release 40 provides the following features These features including the InvisibleCloud Witness for HyperFlex Edge clusters are supported both on the Intersight Virtual Appliance and onIntersightcom

New Features in Release 40(2b)

bull 76TB SSD data drivemdashSupport on HX Edge configurations

bull All NVMe and All Flash limits increasemdashLimits increased on All NVMe (1TB 4TB and 8TB) andAll Flash (76TB SSD)

bull Cluster Scale Limits increasemdashSupport for maximum scale limit increase on a cluster See Cisco HXData Platform Storage Cluster Specifications on page 16

bull HW Offload optionmdashSupport for Hardware Offload option with Stretched cluster configurations

bull Cisco Overlay Transport Virtualization for Stretched ClustermdashSupport for OTV as an overlay forStretched Cluster

New Features in Release 40(2a)

bull Cisco UCS Manager 41(1c)mdashSupport for UCSM hardware and software enhancements and featuresfrom Cisco UCSM 41(1c) release

bull Boost ModemdashThis release introduces Boost Mode for the following configurations All NVMe AllFlash C240 All Flash C220 and Hypervisor ESX Boost Mode allows the Cisco HyperFlex cluster todeliver higher IOPs by increasing the storage controller VMCPU resources by 4 vCPU For configurationinformation see the Cisco HyperFlex Data Platform Administration Guide

bull Cisco HyperFlex HTML plug-in for VMWare vCentermdashEnables virtualization administrator tomanage and monitor the Cisco HyperFlex physical infrastructure by cross launching HyperFlex Connectfrom the vSphere Client UI and perform management actions in the HyperFlex Connect UI

Release Notes for Cisco HX Data Platform Release 405

New Features

bull 25GE networking for HX EdgemdashSupport for 25GE networking for HX Edge

bull All NVMe with Stretched ClustermdashSupport for All NVMe with Stretched Cluster (ESX only)

bull Cluster Upgrade Eligibility TestmdashThis release adds the capability to perform a pre-upgrade test whichchecks for cluster readiness before upgrading Example checks include validating cluster state rebalancestatus controllerVM Free Space ESXi version and much more The Elibility test is intended to helpavoid unexpected problems that may arise during the upgrade process It is highly recommended to runthe test before performing the Hyperflex upgrade

bull Registering Smart Software LicensingmdashThis release adds support for software that allows easy trackingof the status of license and software usage trends and simplifies the three core licensing functionsPurchasing Management and Reporting

bull Dynamic self-signed certificate generation enhancementsmdashThis release adds support for Self-signedSSL certificates on the Controller VMs which were static in prior releases The static certificates arereplaced with dynamically generated self-signed certificates upon upgrading to HXDP 40(2a) so thatthe certificates are unique per cluster The new clusters installed with HXDP 40(2a) have dynamicallygenerated self-signed certificates

bull Test Upgrade EligibilitymdashThis release adds support for testing your cluster readiness and infrastructurecompatibility for an upgrade For more information see the Test Upgrade Eligibility sections in theCiscoHyperFlex SystemsUpgrade Guide for VMware ESXi Release 40 or the CiscoHyperFlex UpgradeGuide for Microsoft Hyper-V Release 40

Disaster Recovery

bull Recovery Settings ConfigurationmdashThis release supports configuration of recovery settings to defineglobal recovery parameters and mapping for resources across recovery sites These parameters are usedduring recovery test recovery and migrate operations

bull HyperFlex DR Powershell RunbookmdashPowerShell runbook functionality is extended to support therecovery configuration settings in the runbooks for all recovery scenarios New-HXrunbook cmdlet cannow be used to generate runbook for a single or multiple protection groups In addition two new cmdletsWait-HXTask and Get-HXTaskStatus are introduced

bull Protected Virtual Machine ScalabilitymdashThis release adds support for 1500 VMs across both clustersand 750 VMs per cluster in a bi-direction or any split between the two clusters without exceeding thelimit of 1500 VMs For more information see Cisco HyperFlex Data Platform Administration Guide

bull System Management REST API enhancementsmdashPause data replication actions briefly using RESTAPI to explicitly inform users on the current status of replication actions

Cisco HyperFlex with Data Platform for Hyper-V

bull Cluster-Aware Updating (CAU)mdashThis is an automated feature that allows you to perform updates onwindows servers in a failover cluster with little or no loss in availability during the upgrade process

Support for new drivesmdashNew drives are qualified for the 40(2a) release The new drives include newcapacity points and new cache drive options Several of the new drives are alternate drives to already qualifiedexisting drives in function which are qualified in 40(2a) These drives are functionally compatible with theexisting drives and are available as alternates in case of lack of availability of existing drives For expansionof existing clusters or general information about interoperability of different drives see Cisco HyperFlexDrive Compatibility

Release Notes for Cisco HX Data Platform Release 406

New Features

NVMe Caching SSDs slot information is unavailable from HX-Connect for all AF server PIDs except forthe All-NVMe server PIDs Please refer to UCSM management console for NVMe SSD slot information

Note

Applicable PlatformsDrive PIDDrive Function

All existing HX M5 serversHX-M2-960GBAlternate boot drive

All existing HX M5 servers except AllNVMe

HX-SD480G6I1X-EVAlternate system (or housekeeping) drive

All existing HX M5 servers except AllNVMe

HX-SD480GM1X-EVAlternate system (or housekeeping) drive

The following HX M5 serversHXAF220C-M5SX HXAF240C-M5SXHXAF-E-220M5SX

HX-SD800G123X-EP800G 12G SAS Cache drive option forAll Flash

All NVMe HXAF220C-M5SNHX-NVME2H-I1000All NVMe 1TB Capacity drive

All NVMe HXAF220C-M5SNHX-NVME2H-I4000All NVMe 4TB Capacity drive

All NVMe HXAF220C-M5SNHX-NVMEHW-I8000New high density All NVMe 8TBCapacity drive

All Flash Configuration ndash namelyHXAF220C-M5SX HXAF240C-M5SXHXAF-E-220M5SX

ESX support only

HX-SD76T61X-EVNew high density All Flash 76TBCapacity drive

Maximum number of drives per node is12 drives on HX AF C240

HX240C-M5LHX-HD8T7K4KANAlternate drive for 8TB LFF capacity

HXAF220C-M5SX HXAF240C-M5SXHX-SD38T2HTNK9New 38TB FIPS compliant SED SSDdata drives

HXAF220C-M5SX HXAF240C-M5SXHX-SD960G2HTNK9New 960GFIPS compliant SEDSSD datadrives

New Features in Release 40(1b)

bull Support for Second Generation Intelreg Xeonreg Scalable Processor RefreshmdashThis release includessupport for the Second Generation Intelreg Xeonreg Scalable processor refresh (formerly Cascade Lake)

New Features in Release 40(1a)

The following new features are in Release 40(1a)

bull Ultra-Light HyperFlex Edge ClustersmdashThis release introduces support for two-node HyperFlex Edgeclusters enabling HyperFlex to run in environments requiring a small footprint Cisco Intersight providescomprehensive lifecycle management and includes remote cloud-based installation centralized upgradesand invisible witnessing Both 1GE and 10GE networking topology options are available

Release Notes for Cisco HX Data Platform Release 407

New Features

bull Scaled-Up HyperFlex Edge ClustersmdashThis release adds support for four node HyperFlex Edge clustersenabling a full range of size options for remote and branch offices Size the branch office environmentsto suit current needs with a two three or four node HyperFlex Edge cluster Cisco Intersight providesfull-lifecycle management and 1GE and 10GE networking options are available

bull Cisco Intersight Invisible Cloud WitnessmdashFor two node clusters this feature eliminates the need forwitness VMs the infrastructure to run those VMs and the management overhead to deploy scale andpatch witnessing software The Invisible Cloud Witness is responsible for maintaining cluster HA in theevent of failure scenarios This feature is included at no additional cost and is automatically deployedand managed by Cisco Intersight

bull Cloud-delivered HyperFlex Edge UpgradesmdashPowered by Cisco Intersight this feature adds supportfor multi-site orchestrated remote upgrades of the HyperFlex Data Platform This feature will be enabledwith the next 40 patch release and will allowHyperFlex Edge clusters deployed via Intersight to performorchestrated upgrades across one or many sites in parallel

bull All-NVMe HyperFlexmdashStarting with this release a new high-end performance node powered by allNVMe drives is available for HyperFlex clusters Co-engineered with Intel to support Intel VMD forhot-plug and surprise removal this offering represents an industry first an enterprise-ready and fullyvalidated all-NVMe HCI appliance The all-NVMe offering is available in the 220 form factor (1RU)and is powered by Intel Optane cache drives for maximum performance and highest endurance

bull VMware Site Recovery Manager (SRM) IntegrationmdashThis release brings support for a Cisco developedStorage Replication Adapter (SRA) for SRM The SRA provides the ability to leverage HyperFlex nativeasync replication with the powerful orchestration and runbook capabilities of SRM The SRA includesthe ability to perform test recoveries planned migrations and full disaster recovery

HX SRA is certified by VMware and is available for download from VMWareSRM site

Note

bull HyperFlex DR Powershell RunbooksmdashNew Powershell cmdlets are included for automated runbookgeneration when using HyperFlex native disaster recovery The New-HXRunbook cmdlet supports thefollowing workflows Test Recovery Planned Migration and Disaster Recovery These runbooks canbe used to orchestrate DR workflows without the requirement for any third-party software

bull Windows Server 2019 with Hyper-VmdashSupport has been added in this release for the Windows Server2019 operating system for Hyper-V based HyperFlex deployments

bull Kubernetes CSI PluginmdashThis release adds support for the HyperFlex CSI (HX-CSI) plugin based onthe Kubernetes Container Storage Interface (CSI) specification Customers can now use the HX-CSIplugin to provision andmanage persistent volumes in Kubernetes version 113 and later Note Kubernetes113 support for Cisco Container Platform amp Openshift Container Platform is forthcoming in theirrespective future releases

bull C480 ML Compute only nodemdashThis release introduces support for C480 ML as a new compute-onlynode for Deep LearningMachine Learning Workloads Data scientists can now use the power of up toeight NVidia SXM2 V100 GPUs to accelerate deep learning workloads VMs running deep-learningworkloads will need to use PCIe pass-through for access to GPUs

bull Higher Capacity DrivesmdashAnew 24TB 10k rpm SAS HDD option for SFF Hybrid HyperFlex clustersand a 12TB 72K rpmSASHDDoption for LFFHybrid are now available Both HyperFlex andHyperFlexEdge support the 24TB capacity point for maximum density in this form factor Note that HyperFlex

Release Notes for Cisco HX Data Platform Release 408

New Features

Edge does not support LFF-drives HyperFlex HyperV version does not yet support the new 12TB driveoption See the HyperFlex spec sheets for a full list of configurable options

bull New Cache and increased scale for Hyper-V mdashNVMe amp Optane SSDs are now supported as cachedrives for Hyper-V deployments Furthermore scale limits have been increased to 16+16(Converged+Compute-only) for both SFF (AF and Hybrid) amp LFF (Hybrid) clusters

bull Centralized Audit Log ExportmdashThis release adds support for audit logging via a remote syslog serverThis capability enables customers to retain audit logs from all HyperFlex nodes in a centralized remotesyslog server to meet retention and compliance requirements

bull DISA STIG CompliancemdashThis release adds new HX REST APIs for setting removing and checkingstatus of DISA STIGs for Controller VMs ESXi hosts and vCenter These APIs enable customers tomeet DISA security requirements by centrally and securely applying STIGs detecting and correctingfor drifts in any STIG settings

Supported Versions and System RequirementsCisco HX Data Platform requires specific software and hardware versions and networking settings forsuccessful installation

For a complete list of requirements see

bull Cisco HyperFlex Systems Installation Guide for VMware ESXi or

bull Cisco HyperFlex Systems Installation Guide for Microsoft Hyper-V

Hardware and Software Interoperability

For a complete list of hardware and software inter-dependencies refer to respective Cisco UCS Managerrelease version of Hardware and Software Interoperability for Cisco HyperFlex HX-Series

HyperFlex Software Versions

The HX componentsmdashCisco HX Data Platform Installer Cisco HX Data Platform and Cisco UCSfirmwaremdashare installed on different servers Verify that each component on each server used with and withinan HX Storage Cluster are compatible

bull HyperFlex does not support UCS Manager and UCS Server Firmware versions 40(4a) 40(4b)and 40(4c)

Do not upgrade to these versions of firmware

Do not upgrade to these versions of UCS Manager

Important

bull Verify that the preconfigured HX servers have the same version of Cisco UCS server firmware installedIf the Cisco UCS Fabric Interconnects (FI) firmware versions are different see the Cisco HyperFlexSystems Upgrade Guide for VMware ESXi Release 40 for steps to align the firmware versions

bull M4 For NEW hybrid or All Flash (Cisco HyperFlex HX240c M4 or HX220c M4) deploymentsverify that Cisco UCS Manager 31(3k) 32(3i) or 40(2b) or higher is installed For moreinformation see Recommended Cisco HyperFlex HX Data Platform Software Releases

Release Notes for Cisco HX Data Platform Release 409

Supported Versions and System Requirements

bull M5 For NEW hybrid or All Flash (Cisco HyperFlex HX240c M5 or HX220c M5) deploymentsverify that the recommended UCS firmware version is installed

For SED-based HyperFlex systems ensure that the A (Infrastructure) B (Bladeserver) and C (Rack server) bundles are at Cisco UCS Manager version 40(2b)or later for all SED M4M5 systems For more details see CSCvh04307 ForSED-based HyperFlex systems also ensure that all clusters are at HyperFlexRelease 35(2b) or later For more information see Field Notice (70234) andCSCvk17250

Important

bull To reinstall an HX server download supported and compatible versions of the software See theCisco HyperFlex Systems Installation Guide for VMware ESXi for the requirements and steps

bull Important For Intersight edge servers running older than 40(1a) CIMC version HUU is the suggestedmechanism to update the firmware

Table 1 HyperFlex Software Versions for M4M5 Servers

M5 Recommended FIServer Firmware

(be sure to review important notes above)

M4 Recommended FIServer Firmware

(be sure to review important notes above)

HyperFlexRelease

40(4h)40(4h)40(2b)

40(4h)40(4h)40(2a)

40(4h)40(4h)40(1b)

40(4e)40(4e)40(1a)

HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments

Cisco HX Data Platform Release 4x based Deployments

Confirm the component firmware on the server meets the minimum versions listed in the following tables

HyperFlex Edge does not support Cisco IMC versions 40(4a) 40(4b) 40(4c) 40(4d) and 40(4e)Important

Table 2 HX220c M4 HXAF220c M4 Cluster

Recommended Firmware Version - HXDP 4x

(be sure to review important note(s) above)

Component

402d0BIOS

40(2h)Cisco Integrated Management Controller(CIMC)

Release Notes for Cisco HX Data Platform Release 4010

HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments

Recommended Firmware Version - HXDP 4x

(be sure to review important note(s) above)

Component

13000012SAS HBA

43(2b)VIC 1227

0103 (Toshiba)

CS07 (Intel 480)

SSD

CN03 (Seagate)

CK03 (Seagate 18)

CK03 (Seagate 24)

sF3Q (Samsung)

3F3Q (Samsung)

HDD

40(2h)

Download Software

Host Upgrade Utility (HUU) Download Link

Table 3 HX220c M5 HXAF220c M5 Cluster

Recommended Firmware Version - HXDP 4x

(be sure to review important notes above)

Component

C220M5404p00224200755BIOS

40(4i)Cisco Integrated Management Controller(CIMC)

09000008SAS HBA

43(3c)VIC 1387

50(3e)VIC 1457

G201CS01 (Intel)

0104 (Toshiba)

0107 (Toshiba)

SSD

2F3Q (Samsung)

3F3Q (Samsung)

NOA4 (Seagate)

CK03 (Seagate 18tb)

CK03 (Seagate 24)

HDD

Release Notes for Cisco HX Data Platform Release 4011

HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments

Recommended Firmware Version - HXDP 4x

(be sure to review important notes above)

Component

40(4k)

Download Software

Host Upgrade Utility (HUU) Download Link

HyperFlex Licensing

As of version 26(1a) HyperFlex supports VMware PAC licensing Existing VMware embedded licenses willcontinue to be supported

As of version 25(1a) HyperFlex uses a smart licensing mechanism to apply your licenses See the CiscoHyperFlex Systems Installation Guide for VMware ESXi for details and steps

VMware vSphere Licensing Requirements

How you purchase your vSphere license determines how your license applies to your HyperFlex system

bull If you purchased your vSphere license with HyperFlex

Each HyperFlex server either has the Enterprise or Enterprise Plus edition preinstalled at the factory

bull HX Nodes have OEM licenses preinstalled If you delete or overwrite thecontent of the boot drives after receiving the HX servers you also delete thefactory-installed licenses

bull OEM license keys is a newVMware vCenter 60 U1b feature Earlier versionsdo not support OEM licenses

bull All factory-installed HX nodes share the same OEM license key WithvSphere OEM keys the Usage count can exceed the Capacity value

bull When you add an HX host to vCenter through the Add Host wizard in theAssign license section select the OEM license

We obfuscate the actual vSphere OEM license key for example0N085-XXXXX-XXXXX-XXXXX-10LHH

bull Standard Essentials Plus and ROBO editions are not available preinstalledon HX servers

Note

bull If you did NOT purchase your vSphere license with HyperFlex

The HX nodes have a vSphere Foundation license preinstalled After initial setup you can apply thelicense to a supported version of vSphere

bull If you purchased a vSphere PAC license

Follow the instructions in your PAC license letter from VMware to add the license to your MY VMwareaccount then follow the instructions to add your HX host to vCenter and assign the PAC license

Release Notes for Cisco HX Data Platform Release 4012

HyperFlex Licensing

HX Data Platform Software Versions for HyperFlex Witness Node

Witness Node VersionHyperFlex Release

10840(2b)

10840(2a)

10440(1b)

10440(1a)

Software Requirements for VMware ESXi

The software requirements include verification that you are using compatible versions of Cisco HyperFlexSystems (HX) components and VMware vSphere VMware vCenter and VMware ESXi

bull Verify that all HX servers have a compatible version of vSphere preinstalled

bull Verify that the vCenter version is the same or later than the ESXi version

bull Verify that the vCenter and ESXi versions are compatible by consulting the VMware ProductInteroperability Matrix Newer vCenter versions may be used with older ESXi versions so long as bothESXi and vCenter are supported in the table below

bull Verify that you have a vCenter administrator account with root-level privileges and the associatedpassword

For VIC1457 there is no support for ESXi 60Note

The below table applies for all of the following VMware vSphere Editions Enterprise Enterprise PlusStandard Essentials Plus ROBO

VMware vCenter VersionsVMware ESXi VersionsHyperFlex Version

60 U3 65 U3 67 U360 U3 65 U3 67 U340(2b)1

60 U3 65 U3 67 U360 U3 65 U3 67 U340(2a)2

60 U3 65 U3 67 U260 U3 65 U3 67 U2340(1b)

60 U3 65 U2 67 U260 U3 65 U2 67 U2440(1a)

1 Cisco HyperFlex Release 40(2) is the last major HyperFlex release that will support vSphere 60 (ESXiand vCenter) due to those versions reaching end of VMware general support on March 12 2020

2 Cisco HyperFlex Release 40(2) is the last major HyperFlex release that will support vSphere 60 (ESXiand vCenter) due to those versions reaching end of VMware general support on March 12 2020

3 Use of 67 U2 is not recommended see software advisory for CSCvq06952 and CSCvp88515 for furtherdetails

4 Use of 67 U2 is not recommended see software advisory for CSCvq06952 and CSCvp88515 for furtherdetails

Release Notes for Cisco HX Data Platform Release 4013

HX Data Platform Software Versions for HyperFlex Witness Node

Software Requirements for Microsoft Hyper-V

The software requirements include verification that you are using compatible versions of Cisco HyperFlexSystems (HX) components and Microsoft Hyper-V (Hyper-V) components

HyperFlex Software versions

The HX componentsmdashCisco HX Data Platform Installer Cisco HX Data Platform and Cisco UCSfirmwaremdashare installed on different servers Verify that each component on each server used with and withinthe HX Storage Cluster are compatible

bull Cisco HyperFlex M5 Converged nodesmdash For Hybrid (Cisco HyperFlex HX240c M5 HX220c M5)and All Flash (Cisco HyperFlex HXAF240cM5 HXAF220cM5) verify that Cisco UCSManager 40(2b)is installed HX 40 (1a) does not support Hyper-V on the All NVMe (HXAF220C-M5SN) nodes Fordetailed information on installation requirements and steps see theCisco HyperFlex Systems InstallationGuide on Microsoft Hyper-V

Table 4 Supported HyperFlex Software versions for M5 Servers on Hyper-V

M5 Recommended Server FirmwareHyperFlex Release

40(4h)40(2b)

40(4g)40(2a)

40(4e)40(1b)

40(4e)40(1a)

Release Notes for Cisco HX Data Platform Release 4014

Software Requirements for Microsoft Hyper-V

Table 5 Supported Microsoft Software versions

VersionMicrosoft Component

Windows Server 2016 Datacenter Core amp DesktopExperience

ForWindows Server 2016Datacenter Coreand Desktop Experience the Windows2016 ISO image should be Update BuildRevision (UBR) 1884 at a minimum

Note

Windows Server 2019Datacenter-DesktopExperienceis supported starting from HXDP 401(a) onwards

For Windows Server 2019 DesktopExperience theWindows 2019 ISO imageshould be Update Build Revision (UBR)107 at a minimum

Note

Windows Server 2019 DatacenterndashCore is notsupported currently

Also note that the following are currently notsupported

OEM activated ISOs and Retail ISOs are notsupported

Earlier versions ofWindows Server such asWindows2012r2 are not supported

Non-English versions of the ISO are not supported

Windows Operating System (Windows OS)

A Windows 2012 or later domain and forestfunctionality level

Active Directory

Supported Microsoft License Editions

The Microsoft Windows Server version that is installed on one or more HyperFlex hosts must be licensed asper Microsoft licensing requirements listed on Microsoft Licensing

Browser Recommendations

Use one of the following browsers to run the listed HyperFlex components These browsers have been testedand approved Other browsers might work but full functionality has not been tested and confirmed

Table 6 Supported Browsers

HX ConnectHX Data PlatformInstaller

Cisco UCS ManagerCisco IntersightBrowser

11 or higher11 or higher11 or higherNAMicrosoft InternetExplorer

70 or higher70 or higher57 or higher62 or higherGoogle Chrome

Release Notes for Cisco HX Data Platform Release 4015

Browser Recommendations

HX ConnectHX Data PlatformInstaller

Cisco UCS ManagerCisco IntersightBrowser

60 or higher60 or higher45 or higher57 or higherMozilla Firefox

NANA9 or higher10 or higherApple Safari

NANA35 or higherNAOpera

Notes

bull Cisco HyperFlex Connect

The minimum recommended resolution is 1024 X 768

bull Cisco HX Data Platform Plug-in

The Cisco HX Data Platform Plug-in runs in vSphere For VMware Host Client System browserrequirements see the VMware documentation at httpswwwvmwarecomsupportpubs

The HX Data Platform Plug-in is not displayed in the vCenter HTML client You must use the vCenterflash client

bull Cisco UCS Manager

The browser must support the following

bull Java Runtime Environment 16 or later

bull Adobe Flash Player 10 or higher is required for some features

For the latest browser information about Cisco UCSManager refer to themost recent Cisco UCSManagerGetting Started Guide

Cisco HX Data Platform Storage Cluster SpecificationsCluster Limits

bull Cisco HX Data Platform supports up to 100 clusters managed per vCenter as per VMware configurationmaximums

bull Cisco HX Data Platform supports any number of clusters on a single FI domain Each HX convergednode must be directly connected to a dedicated FI port on fabric A and fabric B without the use of a FEXC-series compute only nodes must also directly connect to both FIs B-series compute only nodes willconnect through a chassis IO module to both fabrics In the end the number of physical ports on the FIwill dictate the maximum cluster size and maximum number of individual clusters supported in a UCSdomain

The following table provides Cisco HX Data Platform Storage Cluster Specifications

Release Notes for Cisco HX Data Platform Release 4016

Cisco HX Data Platform Storage Cluster Specifications

Stretched Cluster (Availableon ESX Only)

Microsoft Hyper-VVMware ESXiNode

AllNVMe -HXAF220c-M5SN

HX240c-M5LHX220cM5

HX220cAF M5

HX240cM5

HX240cAF M5

HX240c-M5LHX220cM5

HX220cAF M5

HX240cM5

HX240cAF M5

HXAF220c-M5SN

AllNVMe -HXAF220c-M5SN

ESXi 65or 67only NotsupportedwithESXi 60orHyper-V

HX220cM5 Edge

HXAF220cM5 Edge

HX220cM4 Edge

HXAF220cM4 Edge

HX240c-M5LHX220cM5

HX220cAF M5

HX240cM5

HX240cAF M5

HX220cM4

HX220cAF M4

HX240cM4

HX240cAF M4

HXServers

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

C220M5C240M5B200M4B200M5

C240M5C220M5B200M4B200M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

mdashB200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

Compute-OnlyUCSB-SeriesC-SeriesServers

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

Convergednodesonly

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

SupportedNodes

Release Notes for Cisco HX Data Platform Release 4017

Cisco HX Data Platform Storage Cluster Specifications

Stretched Cluster (Availableon ESX Only)

Microsoft Hyper-VVMware ESXiNode

NA(requiresEnterpriseHXDP-PLicense)

NA(requiresEnterpriseHXDP-PLicense)

NA(requiresEnterpriseHXDP-PLicense)

Convergednodes3-16(12TBHDDoption isnotsupportedforHyperV)

Compute-onlynodes0-16

Convergednodes3-16

Compute-onlynodes0-16

NA(requiresEnterpriseHXDP-PLicense)

M4Convergednodes 3

M5Convergednodes23or 4

RequiresHXDP-ELicense

Convergednodes3-16(12TBdrive islimited toamaximumof 8nodes)

Compute-onlynodes0-16(12TBdrive islimited toamaximumof 8nodes)

Convergednodes3-32(76TBdriveconfigsarelimited tomaximumof 12drivesnodeon AF240)

Compute-onlynodes0-32

HXDP-SLicensedNodeLimits

11 ratioofHXDP-StoComputeonlynodes

(MinmdashMax)

Convergednodes2-16 perSite

Compute-onlynodes0-21 perSite

(up tomaxclustersize)

Convergednodes2-8 perSite

Compute-onlynodes0-16 perSite

(up tomaxclustersize)

Convergednodes2-16 perSite

Compute-onlynodes0-21 perSite

(up tomaxclustersize)

Convergednodes3-16(12TBHDDoption isnotsupportedforHyperV)

Compute-onlynodes0-16

Convergednodes3-16

Compute-onlynodes0-16

Convergednodes3-32

ComputeOnlynodes0-32

(up tomaxclustersize)

Convergednodes 3

(requiresHXDP-ELicense)

Convergednodes3-16(12TBdrive islimited toamaximumof 8nodes)

Compute-onlynodes0-32(12TBdrive islimited toamaximumof 16nodes)

Convergednodes3-32(76TBdriveconfigsarelimited tomaximumof 12drivesnodeon AF240)

Compute-onlynodes0-32

(up tomaxclustersize)

HXDP-PLicensedNodeLimits

12 ratioofHXDP-PtoComputeonlynodes

(MinmdashMax)

24 perSite 64percluster

24 perSite 48percluster

32 perSite 64percluster

32326434864MaxClusterSize

Release Notes for Cisco HX Data Platform Release 4018

Cisco HX Data Platform Storage Cluster Specifications

Stretched Cluster (Availableon ESX Only)

Microsoft Hyper-VVMware ESXiNode

212121111121mdash2121MaxComputetoConvergedratio

NoExpansion

Requires Enterprise license

Requires uniform expansion across both sites

Guidelines and Limitationsbull Starting with release 40(2a) SCVM is no longer needed on a Compute node

bull HX REST API Access Token Management ndash Applications leveraging HX REST APIs should re-useaccess tokens when making API calls Once obtained using the AAA Obtain Access Token API accesstokens are valid for 18 days (1555200 seconds) In addition AAA enforces rate limiting on ObtainAccess Token API requests in a 15 minute window auth can be invoked (successfully) a maximum of5 times A user is allowed to create a maximum of 8 unrevoked tokens Subsequent call to auth willautomatically revoke the oldest issued token tomake room for the new token Amaximum of 16 unrevokedtokens can be present in system In order to prevent brute-force attacks after 10 consecutive failedauthentication attempts a user account is locked for a period of 120 secondsFor more information seeCisco HyperFlex Systems REST API Reference guide

HxConnect makes use of AAA Authentication REST API for login and the above rate limit applies toHxConnect also

Upgrade Guidelines

The following list is a highlight of critical criteria for performing an upgrade of your HyperFlex system

bull Unsupported HX Data Platform 17x 18x 20 21x 25x and 26x clustersmdashUsers from anyversion prior to 26(1a) must step through an intermediate version before upgrading to 40 or later releasesIf you need to upgrade your environment from a Cisco HyperFlex HX Data Platform software releasethat is past the last date of support to the latest suggested release on the Cisco Software Download sitesee Cisco HyperFlex Systems Upgrade Guide for Unsupported Cisco HXReleases For more informationsee the Software Advisory for CSCvq66867 WARNING Only Use HXDP 26(1e) Upgrade PackageWhen Upgrading From HXDP 18(1a)-18(1e)

bull Hypercheck Health Check UtilitymdashCisco recommends running this proactive health check utility onyour HyperFlex cluster prior to upgrade These checks provide early visibility into any areas that mayneed attention andwill help ensure a seamless upgrade experience For more information see the HyperFlexHealth amp Pre-Upgrade Check Tool TechNote for full instructions on how to install and run Hypercheck

bull vSphere 67 Software AdvisorymdashDo not upgrade to Cisco HX Data Platform Release 40(1a) whenrunning ESXi 67U1 EP06 (build 11675023) Do not upgrade to 67U1 EP06 (build 11675023) ifrunning Cisco HX Data Platform Release 40(1a) See the Software Advisory CSCvo56350 for furtherdetails

Release Notes for Cisco HX Data Platform Release 4019

Guidelines and Limitations

The software build version posted at release will override any other local versions

bull Required vCenter upgrademdashFor enhanced security Cisco HX Data Platform Release 35(1a) or laterrequires the use of TLS 12 Therefore vCenter must be upgraded to 60 U3f or later before upgradingto Cisco HX Data Platform Release 35 or later In addition ESXi should be upgraded as required tomeet HX Data Platform compatibility requirements

bull Minimum HXDP version for upgrademdashHXData Platform clusters running 26(1a) or later may upgradedirectly to 40 using the HX Connect UI

bull Cluster ReadinessmdashEnsure that the cluster is properly bootstrapped and the updated plug-in is loadedbefore proceeding Manual cluster bootstrap is required for upgrade from a pre-35 release

bull Cluster ReadinessmdashEnsure that the cluster is properly bootstrapped and the updated plug-in is loadedbefore proceeding Manual cluster bootstrap is required for HX releases earlier than 35(1a) For moreinformation see theManual Bootstrap Upgrade Process in the Cisco HyperFlex Systems Upgrade Guidefor VMware ESXi Release 40 Do not skip this cluster bootstrap step it is required for all upgradesuntil HX Release 35(1a) Auto bootstrap is supported beginning with HX release 35(1a) For moreinformation see the Auto Bootstrap Upgrade Process from HX Connect UI in the Cisco HyperFlexSystems Upgrade Guide for VMware ESXi Release 40

Manual bootstrap is not supported on Intersight clusters

bull Initiating Upgrade―Use the HX Connect UI or CLI stcli commands when upgrading from 25(1a)or later releases Use either the CLI stcli commands or the HX Data Platform Plug-in to the vSphereWeb Client when upgrading from a pre-25(1a) release The vCenter plug-in should not be used forupgrades starting with the 25(1a) release The vCenter plug-in should not be used for upgrades startingwith the 25(1a) release

If the current cluster version is at 35(1a) or above you do not need to use the stcli command Directupgrade to 40 is possible

bull Complete your Upgrade―The self-healing (or rebalance) capability is disabled temporarily during theupgrade window If the upgrade fails you should complete the upgrade as soon as possible

bull ESXi and HXDP Compatibility―Ensure your cluster is running a compatible version of ESXi basedon the running the HXData Platform version (see the section Software Requirements for VMware ESXi)ESXi compatibility is determined by the major version and update release of ESXi It is generally bestto upgrade HXDP and ESXi together if combining the upgrade operations into a single optimized rebootWhen running a split upgrade first upgrade the HX Data Platform then proceed to upgrade ESXi

bull If Upgrading to vSphere 65

bull Certain cluster functions such as native and scheduled snapshotsReadyClones and Enter or Exit HX Maintenance Mode will not operatefrom the time the upgrade is started until the HX Data Platform upgrade iscomplete

bull After upgrading ESXi using the offline zip bundle use the ESX ExitMaintenance Mode option The HX Exit Maintenance Mode option doesnot operate in the vSphere Web Client until the HX Data Platform upgradeis complete

Note

Release Notes for Cisco HX Data Platform Release 4020

Guidelines and Limitations

bull vSphere 60 UpgradesmdashUsers on vSphere 60 migrating to 65 upgrade components in the followingorder

1 Upgrade HX Data Platform and UCS firmware

2 Upgrade HX Data Platform and ESXi

3 Upgrade HX Data Platform only first then upgrade ESXi or UCS firmware or both

bull M4 Server Firmware UpgradesmdashUpgrade server firmware to ensure smooth operation and to correctknown issues Specifically newer SAS HBA firmware is available in this release and is recommendedfor long-term stability

bull Users are encouraged to upgrade to 31(3c) C-bundle or later whenever possible

bull Users running C-bundle versions before 31(2f) must upgrade server firmware by performing acombined upgrade of UCS server firmware (C-bundle) to 31(3c) or later and HX Data Platform to25 Do not split the upgrade into two separate operations

bull If the cluster is already on 31(2f) C-bundle or later you may perform an HX Data Platform onlyor combined upgrade as required

bull M5 Server Firmware UpgradesmdashM5 generation servers must run firmware version 32(2d) or later

bull Firmware Downgrades mdashDowngrading UCSM from the HX-installer is not supported

bull M4M5 Mixed DomainsmdashA mixed domain occurs when a new separate M5 cluster is installed underthe same UCS domain that contains existing M4 clusters Under these conditions orchestrated UCSserver firmware upgrade will not operate until Cisco HX Data Platform Release 26 or later is installedon the M4 clusters Therefore it is best practice to first upgrade UCS server firmware to the latest 31(3)or 32(2) patch release before adding a new M5 cluster to the existing UCS domain Additionally any17 HX Data Platform clusters must first be upgraded before adding any new M5 clusters to the samedomain

bull Maintenance WindowmdashIf upgrading both HX Data Platform and UCS firmware you can select eithera combined or split upgrade through the vSphere HX Data Platform Plug-in depending on the length ofthe maintenance window Cisco UCSManager infrastructure upgrade is only supported using AutoInstalland the direct server firmware upgrade should be performed only through the upgrade orchestrationframework provided by the HX Data Platform Plug-in

bull Unsupported Self-Encrypting Drives (SEDs)mdashIf adding or replacing self-encrypting drives (SEDs)that have been recently qualified in newer versions of HX Data Platform insert the new drives only afterupgrading HX Data Platform to a compatible version All drives must be SED drives mixing SED andnon-SED is not supported

bullbull Enabling External Host AccessmdashWith Cisco HX Data Platform Release 40(1a) port 445 on themanagement network is blocked for enhanced security Note that prior to 40 port 445 port was openenabling external host access If you are upgrading to 40(1a) from a prior release and would like tocontinue external host access you can use a utility to open select hosts For more information aboutenabling external host access see the ConfiguringHyperFlex Share to SCVMM section in the InstallationGuide for Microsoft Hyper-V

Release Notes for Cisco HX Data Platform Release 4021

Guidelines and Limitations

Mixed Cluster Expansion Guidelinesbull Expanding existing M4 cluster with M5 converged nodes is supported

bull Expanding existing M5 cluster with M4 converged nodes is not supported

bull Expanding existing mixed M4M5 cluster with M4 or M5 converged nodes is supported

bull Adding any supported compute-only nodes is permitted with all M4 M5 and mixed M4M5 clustersusing the HX Data Platform 26 or later Installer Some example combinations are listed here manyother combinations are possibleExample combinationsExpand mixed M4M5 cluster with compute-only B200 C220 C240 M4M5Expand M4 cluster with compute-only B200 M5 C220 M5 C240M5

bull Only expansion workflow is supported to create a mixed cluster Initial cluster creation with mixedM4M5 servers is not supported

bull All M5 servers must match the form factor (220240) type (HybridAF) security capability (Non-SEDonly) amp disk configuration (QTY capacity and non-SED) of the existing M4 servers

bull HX220-M5 will use a maximum of 6 capacity disks (2 disk slots to remain empty) when mixedwith HX220-M4

bull HX Edge SED LFF Hyper-V and Stretched Clusters do not support mixed M4 and M5 clusters

Security FixesThe following security issues are resolved

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-13132 CVE-2019-9924CSCvq6313840(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-11719 CVE-2019-11727CVE-2019-11729

CSCvq7124040(2a)

Release Notes for Cisco HX Data Platform Release 4022

Mixed Cluster Expansion Guidelines

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-1125CSCvr0633940(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2015-9383 CVE-2018-14498CVE-2018-20406 CVE-2018-20852CVE-2019-10160 CVE-2019-13117CVE-2019-13118 CVE-2019-14287CVE-2019-14973 CVE-2019-15903CVE-2019-17546 CVE-2019-18197CVE-2019-18218 CVE-2019-5010CVE-2019-5094 CVE-2019-5481CVE-2019-5482 CVE-2019-9636CVE-2019-9740 CVE-2019-9947CVE-2019-9948

CSCvs0609440(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2017-13168 CVE-2017-18174CVE-2017-18216 CVE-2018-10876CVE-2018-10877 CVE-2018-10878CVE-2018-10879 CVE-2018-10880CVE-2018-10881 CVE-2018-10882CVE-2018-10902 CVE-2018-10938CVE-2018-12233 CVE-2018-12896CVE-2018-13053 CVE-2018-13094CVE-2018-13096 CVE-2018-13405CVE-2018-13406 CVE-2018-14609CVE-2018-14617 CVE-2018-14633CVE-2018-14734 CVE-2018-15572CVE-2018-15594 CVE-2018-16276CVE-2018-16658 CVE-2018-17182CVE-2018-17972 CVE-2018-18021CVE-2018-18690 CVE-2018-18710CVE-2018-6554 CVE-2018-6555CVE-2018-9363

CSCvp6501940(2a)

This vulnerability is due to insufficientHTML iframe protection An attackercould exploit this vulnerability bydirecting a user to an attacker-controlledweb page that contains amaliciousHTMLiframe A successful exploit could allowthe attacker to conduct clickjacking orother clientside browser attacks

NACSCvo9851640(2a)

Release Notes for Cisco HX Data Platform Release 4023

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerability is due to insufficientauthentication for the statistics collectionservice An attacker could exploit thisvulnerability by sending properlyformatted data values to the statisticscollection service of an affected deviceA successful exploit could allow theattacker to cause the web interfacestatistics view to present invalid data tousers

NACSCvj9558440(2a)

The vulnerability is due to insufficientCSRF protections for the web UI on anaffected device An attacker could exploitthis vulnerability by persuading a user ofthe interface to follow a malicious linkA successful exploit could allow theattacker to perform arbitrary actions withthe privilege level of the affected user

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvp2434340(2a)

Vulnerabilities with a version of cJSONidentified by CVE-2019-11834 andCVE-2019-11835

For more information see the relatedCisco Security Advisory

CVE-2019-11834 CVE-2019-11835CSCvq1994940(2a)

HX ESXi image patches for MCEPSCand TAA vulnerabilities with VMwareESXi identified by CVE-2018-12207 andCVE-20190-11135

For more information see the relatedCisco Security Advisory

CVE-2018-12207 CVE-2019-11135CSCvr5439840(2a)

CP networking vulnerabilities affectingthe Linux kernel identified byCVE-2019-11477 CVE-2019-11478CVE-2019-11479

For more information see the relatedCisco Security Advisory

CVE-2019-11477 CVE-2019-11478CVE-2019-11479

CSCvq1954640(2a)

Qualification of Microsoft securitypatches for vulnerabilities withMicrosoftHyper-V hypervisor identified withCVE-2018-12207 CVE-2019-11135

For more information see the relatedCisco Security Advisory

CVE-2018-12207 CVE-2019-11135CSCvr5439940(2a)

Release Notes for Cisco HX Data Platform Release 4024

Security Fixes

DescriptionCVEDefect IDRelease

OpenSSH vulnerabilities identified byCVE-2019-6111

For more information see the relatedCisco Security Advisory

CVE-2016-10708 CVE-2018-15473CVE-2018-20685 CVE-2019-6109CVE-2019-6111

CSCvp7646340(2a)

Multiple vulnerabilities associated withOpenSSL and LibNSS

For more information see the relatedCisco Security Advisory

CVE-2016-2105 CVE-2016-2106CVE-2016-2107 CVE-2016-2109CVE-2016-2176 CVE-2016-2177CVE-2016-2178 CVE-2016-2179CVE-2016-2180 CVE-2016-2181CVE-2016-2182 CVE-2016-2183CVE-2016-6302 CVE-2016-6303CVE-2016-6304 CVE-2016-6305CVE-2016-6306 CVE-2016-6307CVE-2016-6308 CVE-2016-7055CVE-2016-8610 CVE-2017-3731CVE-2017-3732 CVE-2017-3735CVE-2017-3736 CVE-2017-3737CVE-2017-3738 CVE-2018-0495CVE-2018-0732 CVE-2018-0734CVE-2018-0735 CVE-2018-0737CVE-2018-0739 CVE-2018-12384CVE-2018-12404 CVE-2018-5407CVE-2019-1559

CSCvp6667940(2a)

libpng vulnerability identified byCVE-2016-10087

For more information see the relatedCisco Security Advisory

CVE-2016-10087CSCvp6655540(2a)

Vulnerabilities associated with libjpg andlibpng

For more information see the relatedCisco Security Advisory

CVE-2014-9092 CVE-2016-3616CVE-2017-15232 CVE-2018-11212CVE-2018-11213 CVE-2018-11214CVE-2018-1152 CVE-2018-13785

CSCvp3458640(2a)

Vulnerabilities associated with Glibidentified by CVE-2018-16428 andCVE-2018-16429

For more information see the relatedCisco Security Advisory

CVE-2018-16428 CVE-2018-16429CSCvp3120740(2a)

Multiple vulnerabilities associated withcurl expat python 27 python 35 36and 37 freetype and giflib

For more information see the relatedCisco Security Advisory

CVE-2019-15133 CVE-2019-15903CVE-2019-5010 CVE-2019-5481CVE-2019-5482 CVE-2019-9636CVE-2019-9740 CVE-2019-9947CVE-2019-9948

CSCvr3690340(2a)

Release Notes for Cisco HX Data Platform Release 4025

Security Fixes

DescriptionCVEDefect IDRelease

Multiple third party softwarevulnerabilities

For more information see the relatedCisco Security Advisory

CVE-2019-14379 CVE-2019-12384CVE-2019-14439

CSCvq9203240(2a)

Vulnerabilities associated with elfutils

For more information see the relatedCisco Security Advisory

CVE-2018-16062 CVE-2018-16402CVE-2018-16403 CVE-2018-18310CVE-2018-18520 CVE-2018-18521CVE-2019-7149 CVE-2019-7150CVE-2019-7665

CSCvq4325040(2a)

Vulnerabilities associated with vim

For more information see the relatedCisco Security Advisory

CVE-2017-5953 CVE-2019-12735CSCvq4323040(2a)

Vulnerabilities associated with sqlite3

For more information see the relatedCisco Security Advisory

CVE-2016-6153 CVE-2017-10989CVE-2017-13685 CVE-2017-2518CVE-2017-2519 CVE-2017-2520CVE-2018-20346 CVE-2018-20505CVE-2018-20506 CVE-2019-8457CVE-2019-9936 CVE-2019-9937

CSCvq4321340(2a)

Vulnerabilities associated with glib20

For more information see the relatedCisco Security Advisory

CVE-2018-20843CSCvq4320940(2a)

Vulnerabilities associated with expat

For more information see the relatedCisco Security Advisory

CVE-2018-20843CSCvq4320540(2a)

Vulnerabilities associated with bzip2

For more information see the relatedCisco Security Advisory

CVE-2016-3189 CVE-2019-12900CSCvq4319440(2a)

Mulitple Vulnerabilities associated wtihNodeJS

For more information see the relatedCisco Security Advisory

CVE-2018-12115 CVE-2018-0734CVE-2018-5407 CVE-2018-12120CVE-2018-12121 CVE-2018-12122CVE-2018-12123 CVE-2018-12116CVE-2019-5737 CVE-2019-5739

CSCvq1069440(2a)

Vulnerabilities associated with PalletsJinja strformat_map and identified byCVE-2019-10906 and CVE-2016-10745

For more information see the relatedCisco Security Advisory

CVE-2019-10906 CVE-2016-10745CSCvq1038840(2a)

Release Notes for Cisco HX Data Platform Release 4026

Security Fixes

DescriptionCVEDefect IDRelease

Multiple Security Vulnerabilities

For more information see the relatedCisco Security Advisory

CVE-2014-8501 CVE-2014-9939CVE-2015-9262 CVE-2016-10087CVE-2016-2226 CVE-2016-4487CVE-2016-4488 CVE-2016-4489CVE-2016-4490 CVE-2016-4491CVE-2016-4492 CVE-2016-4493CVE-2016-6131 CVE-2018-10963CVE-2018-13785 CVE-2018-17100CVE-2018-17101 CVE-2018-18557CVE-2018-18661 CVE-2018-7456CVE-2018-8905

CSCvq0917840(2a)

Vulnerabilities associated withlibseccomp

For more information see the relatedCisco Security Advisory

CVE-2019-9893CSCvq0756840(2a)

Vulnerabilities associated withgdk-pixbuf

For more information see the relatedCisco Security Advisory

CVE-2017-12447CSCvq0675540(2a)

Vulnerabilities associated withpython-crypto

For more information see the relatedCisco Security Advisory

CVE-2018-6594CSCvp9381740(2a)

Vulnerabilities associated with wget

For more information see the relatedCisco Security Advisory

CVE-2018-20483 CVE-2019-5953CSCvp8672140(2a)

Multiple vulnerabilities associated withPython Crypto

For more information see the relatedCisco Security Advisory

CVE-2018-6594CSCvp6674840(2a)

Vulnerabilities associated withsensible-utils

For more information see the relatedCisco Security Advisory

CVE-2017-17512CSCvp6673440(2a)

Multiple vulnerabilities associated withPython

For more information see the relatedCisco Security Advisory

CVE-2018-1000030CSCvp6668940(2a)

Release Notes for Cisco HX Data Platform Release 4027

Security Fixes

DescriptionCVEDefect IDRelease

Multiple patch vulnerabilities identifiedby CVE-2016-10713

For more information see the relatedCisco Security Advisory

CVE-2016-10713CVE-2018-1000156 CVE-2018-6951

CSCvp6667240(2a)

Vulnerabilities associatedwith Little CMSidentified by CVE-2016-10165

For more information see the relatedCisco Security Advisory

CVE-2016-10165 CVE-2018-16435CSCvp6666440(2a)

Vulnerabilities associated with TenableScan identified byCVE-2011-5325CVE-2019-11068CVE-2016-7076CVE-2017-1000368

For more information see the relatedCisco Security Advisory

CVE-2011-5325 CVE-2016-7076CVE-2017-1000368CVE-2019-11068

CSCvp6474640(2a)

Vulnerabilities associated with libxml2294 and earlier as used in XMLSec1223 and earlier and other products andidentified by CVE-2016-9318

For more information see the relatedCisco Security Advisory

CVE-2016-9318 CVE-2017-16932CVE-2017-18258 CVE-2018-14404CVE-2018-14567

CSCvp3479240(2a)

Vulnerabilities associated with LFTPRemote File Names Unauthorized AccessVulnerability and identified byCVE-2018-10916

For more information see the relatedCisco Security Advisory

CVE-2018-10916CSCvp2926640(2a)

Vulnerabilities associated withCryptographyDeprecationWarning signerand verifier have been deprecated

For more information see the relatedCisco Security Advisory

CVE-2018-7750CSCvo3409740(2a)

Multiple Vulnerabilities in python python27 35

For more information see the relatedCisco Security Advisory

CVE-2019-16056CSCvr3979340(2a)

Release Notes for Cisco HX Data Platform Release 4028

Security Fixes

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toexecute commands as the root user

The vulnerability is due to insufficientinput validation An attacker could exploitthis vulnerability by connecting to thecluster service manager and injectingcommands into the bound process Asuccessful exploit could allow the attackerto run commands on the affected host asthe root user

Cisco has released software updates thataddress this vulnerability There areworkarounds that address thisvulnerability

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvq2417640(1b)35(2g)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

The vulnerability is due to an unprotectedlistening interface An attacker couldexploit this vulnerability by connectingto the listening interface and injectingcommands to the bound process Anexploit could allow the attacker to runcommands on the affected host as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvj9560640(1b)35(2g)

Release Notes for Cisco HX Data Platform Release 4029

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerabilities associated with theJVM18U121memory leak found duringVC alarms (concurrent 40 calls usingREST API)

CVE-2017-10053 CVE-2017-10067CVE-2017-10074 CVE-2017-10078CVE-2017-10081 CVE-2017-10087CVE-2017-10089 CVE-2017-10090CVE-2017-10096 CVE-2017-10101CVE-2017-10102 CVE-2017-10107CVE-2017-10108 CVE-2017-10109CVE-2017-10110 CVE-2017-10111CVE-2017-10115 CVE-2017-10116CVE-2017-10118 CVE-2017-10135CVE-2017-10176 CVE-2017-10193CVE-2017-10198 CVE-2017-10243CVE-2017-10274 CVE-2017-10281CVE-2017-10285 CVE-2017-10295CVE-2017-10345 CVE-2017-10346CVE-2017-10347 CVE-2017-10348CVE-2017-10349 CVE-2017-10350CVE-2017-10355 CVE-2017-10356CVE-2017-10357 CVE-2017-10388CVE-2017-3509 CVE-2017-3511CVE-2017-3526 CVE-2017-3533CVE-2017-3539 CVE-2017-3544CVE-2018-2579 CVE-2018-2582CVE-2018-2588 CVE-2018-2599CVE-2018-2602 CVE-2018-2603CVE-2018-2618 CVE-2018-2629CVE-2018-2633 CVE-2018-2634CVE-2018-2637 CVE-2018-2641CVE-2018-2663 CVE-2018-2677CVE-2018-2678 CVE-2018-2783CVE-2018-2790 CVE-2018-2794CVE-2018-2795 CVE-2018-2796CVE-2018-2797 CVE-2018-2798CVE-2018-2799 CVE-2018-2800CVE-2018-2814 CVE-2018-2815CVE-2018-2952 CVE-2018-3136CVE-2018-3139 CVE-2018-3149CVE-2018-3150 CVE-2018-3169CVE-2018-3180 CVE-2018-3183CVE-2018-3214 CVE-2019-2422

CSCvo8899740(1b)

Tomcat and Nginx logs are not beingcollected in the support bundle generatedthrough HX Connect in Release 35

NACSCvm5803140(1b)

The vulnerabilities associated with thelibmspack software package versionincluded in Cisco HX Data Platform

CVE-2018-18584

CVE-2018-18585

CSCvn3511940(1a)

Release Notes for Cisco HX Data Platform Release 4030

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerabilities associated withFasterXML Jackson-Databind TimeValueField Denial of Service

CVE-2018-14719

CVE-2018-14720

CVE-2018-1000873

CVE-2018-14721

CVE-2018-19360

CVE-2018-19362

CVE-2018-19361

CVE-2018-14718

CSCvn8228240(1a)

The vulnerabilities associated with theOpenSSL Protocol software packageversion included in Cisco HX DataPlatform

CVE-2013-3587CSCvo0505440(1a)

The vulnerabilities associated withThird-Party Software Denial of Service

CVE-2018-16487

CVE-2018-19361

CSCvo2781840(1a)

The vulnerabilities associated withLinuxkernel for Ubuntu 1710

CVE-2018-1092

CVE-2018-7492

CVE-2018-8087

CVE-2018-1068

CVE-2018-8781

CSCvm5314935(2a)

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)25(1d)stcli services dns remove should remove the DNS serverinfo from the interface files

CSCvs70967

40(2b)40(1b)HX 40(1b) new deployment from Intersight productioncloud

User receives following error while enabling CSI(Kubernetes integration)

Failure occurred during volume_access_enable Errorwas Nonzero exit code 1

CSCvt10522

40(2b)40(2a)Receive the following alertevent in HX Connect HXController VM HOSTNAME one or more configuredDNS servers not responding

CSCvt13947

Release Notes for Cisco HX Data Platform Release 4031

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)40(2a)API UI CLI show few Drive slots Empty after upgradeto 402a

The drives are seen in the sysmtool lsscsi and stclicommand outputs

The cluster is healthy with no errors

CSCvt14914

40(2b)35(2d)After successfully changing the DNS server on the HXcontroller we still can see the original DNS entry thatwas added during the deployment

CSCvs69154

40(2b)35(2e)HyperFlex Connect and VCenter show APD alarms forHX and non HX datastores This implies there is an issuewith HyperFlex when there may not be

CSCvs30080

40(2b)40(2a)An issue where all the disks in the node get locked afterrebooting the node

We successfully unlocked using sed-clientsh -Ucommand but wanted to test with another reboot anddrives locked again

CSCvs74286

40(2b)40(2a)When running stcli license commands on HyperFlexthe following error is seen

rootSpringpathControllertmp stcli license show all

Show smart licensing failed

Smart Agent is not ready please wait a minute and tryagain

CSCvt13929

40(2b)40(2a)After upgrading to HX 40(2a) the first RoboEdge nodethat gets rebooted then ignores persistent drivesPIDHX-HD24TB10K4KN

Cluster will be in degraded state and have reducedcapacity

CSCvt20144

40(2b)35(2b)Zookeeper fails to start while Exhibitor is runninghowever echo srvr returns nothing

CSCvs21562

40(2b)40(2a)When performing an HyperFlex upgrade a validationwarning may occur due to the host not having EnterprisePlus or Enterprise hypervisor licensing

Upgrade Validation Warning

ESXi host esx1labtest should be configured withVMware Enterprise license for upgrade to continue

CSCvs91787

Release Notes for Cisco HX Data Platform Release 4032

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)35(2g)Cluster expansion fails at Config Installer stage whenthe root and admin password for the storage controller(SCVM) are different

CSCvs69317

40(2b)35(2g)Panic while upgrading ESXiCSCvt06983

40(2b)40(2a)A cluster node running may hang in the Linux kernelThis is classified as an oops and a deviation from theexpected behavior

CSCvs54285

40(2b)35(2g)Rebalance failing on 10+10 node HX 35(2g) stretchcluster

CSCvs69007

40(2b)35(2d)The cluster becomes inaccessible to the IOVisorCSCvr54687

40(2b)40(2a)Panic on the storage controllerCSCvt61297

40(2b)40(2a)The size of support bundle is very large when collectedwith storfs-support command

CSCvt63306

40(2b)40(2a)storfs crash associated withFileSystemUsageWarningEvent orFileSystemUsageAlertEvent due to high usage ofvarstv above 80

CSCvt72807

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

40(2a)35(2d)In normal scenarios HX upgrade gets pendingACK afterupdating service profile In this case UCS only upgradewas stalled because after updating service profile HXupgrade was not getting pending ACK and it was alwayswaiting state

CSCvo39912

40(2a)35(1a)When using motherBoardReplace-12 to clean ZK fromold stNodepNodes resulted in unmounted datastore andmaking the size of the Hyperflex Datastores 0 resultingin all VMs in the cluster going offline

CSCvq39471

40(2a)26(1e)Upgrading cluster and getting error

Failed upgrade validations Checking vCenterconfiguration Reason Upgrade validations failed DRSFault Insufficient resources to satisfy configured failover

CSCvm77294

Release Notes for Cisco HX Data Platform Release 4033

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)The cluster expand fails on a node with DR replicationconfigured

When a HX cluster which has DR replication configuredis expanded we see the installer UI pulling in thereplicationVLAN information instead of themanagementVLAN information Even if we change that informationto the correct mgmt VLAN id and name it it does notseem to work as the node is configured with the VLANof the replicationVLAN in ESXi This leads to the failureof the node add with the host unreachable error

CSCvo70650

40(2a)35(1a)Customer reported that server didnt complete thefirmware upgrade automatically one by one

The user manually needs to put the host into maintenancemode and they manually acknowledged the pendingrequests to finish the UCS firmware upgrade

Maintenance policy is set to default (User-ack) as perthe design

CSCvo91624

40(2a)35(1i)When the cluster is in Failed state and stcli noderemove is attempted the output shows successful thoughthe node failed to get removed from the cluster

CSCvo93017

40(2a)35(1a)HyperFlex cluster upgrade may fail during validationwith a DRS Validation failed error

CSCvp31021

40(2a)35(2b)HX cluster expansion will fail with the following errormessage

MAC address pool configuration failure150[ErrorDescription] bad address block range definitioncollision

CSCvp58318

40(2a)35(2b)Memory usage by carbon cacheCSCvq34873

40(2a)35(2g)Upgrading ESXi cluster fails with error Nodemaintenance mode failed

CSCvr03240

40(2a)35(2e)storfs process does not automatically start on Exit HXMaintenance Mode or other tasks which restart StorageController

CSCvr88978

40(2a)35(2a)Post install script fails with message

Failed to execute ipmitool on HX node

CSCvp10707

40(2a)40(1a)HyperFlex expansionworkflow doesnt pull VLANnamecorrectly

CSCvp46539

Release Notes for Cisco HX Data Platform Release 4034

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)During HX cluster deployment - Validate ClusterCreation phase - HX Installer might fail with the error

fromvarlogspringpathstDeploylog

CSCvq45087

40(2a)35(2b)HX Installer fails to login to the SOL for configuringESX hosts

CSCvq91380

40(2a)35(2d)If DRS is enabled with configuration parameters otherthen fully automated ie manual or partially automatedthan this will change to fully automated during clusterexpansion

CSCvr44222

40(2a)35(1a)One of the validation task name has a typo The taskname is Validating node for mixed of SED and non-SEDdisks

It should be Validating node for mix of SED andnon-SED disks

CSCvp82175

40(2a)30(1i)This combination of using smaller NVMe drives is notsupported in expand operation For example if the drivesare only 375GB and they cannot be added to an existingcluster with larger caching SSD

CSCvo12359

40(2a)35(1a)HyperFlex includes a version of OpenSSL that is affectedby the vulnerabilities identified by the followingCommon Vulnerability and Exposures

CVE-2018-0495

CSCvp66679

40(2a)30(1e)SDK invoke exception when taking a native snapshot ona VM with GPU pci passthrough attached

CSCvo00511

40(2a)30(1i)Node replacement script fails due to EAM errorCSCvo62867

40(2a)35(1a)While pairing a cluster (HX release gt= 35) and cluster(HX release lt 35) remote replication network test failson the cluster (HX release lt 35)

CSCvo79760

40(2a)30(1b)Fixed the issue in the latest support-workflow bundleThis should not be hit anymore

CSCvo87061

40(2a)35(2a)On a 3 node cluster the MBreplace script fails as thescript looks for HEALTHY cluster status

CSCvo87080

40(2a)35(2a)MbReplace script (tar) hangs when running on 352b or40

CSCvp12359

Release Notes for Cisco HX Data Platform Release 4035

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)HX replication cleanup failing with error NA 35(2a) -INFODR state is not clean

CSCvp63958

40(2a)40(1a)The check_stig_parameters API incorrectly shows thecompliance state of the cluster as not STIG compliant

CSCvp66277

40(2a)35(2c)The snapshot creation on CBT enabled VM fails witherror Failed in vmreparent vmkfstools clone1

CSCvq06952

40(2a)35(2a)HyperFlex GUI when setting up remote DR-replicationpartnerpeer should show remote replication partner TCPport reachability test results

This will provide quick upfront connectivity results thatfrequently takes customer a while to manually check andconfirm

If reachability tests fail an info window and HX alertshould trigger that list the destination peer IP and thespecific port(s) that could not be reache

CSCvo60587

40(2a)35(2b)If you have more than one IP pool configured for areplication network and local replication network testsare failing you might be running into this issue

CSCvr67130

40(2a)35(2a)Skip task option in MBreplace and Software redeployscripts The scripts should have the option to skip tasksas user inputs and not by modifying the json file

CSCvp05204

40(2a)40(1a)Error noticed on SCVM Console print_req_error IOerror dev fd0 sector 0 on HyperV

It doesnt come up on SSH session

CSCvq34357

40(2a)30(1d)The VMNetwork switch is already created as part of theinstall process

Ideally the ip address assignment should be done duringthe initial install process as well No post install stepsshould be required altrnatively it could be done as partof a post install script

CSCvk36222

40(2a)25(1c)storfs service impacted during HX upgrade from HX21(1) to 25(1c)

HX node which is not being upgraded storfs servicestopped causing outage

CSCvg53223

40(2a)35(1b)HX Stretch Cluster Witness VM reverts to DHCP onreboot

CSCvp37536

40(2a)35(2b)HyperFlex Stretch Cluster Witness does not getprogrammed with an NTP server

CSCvp46578

Release Notes for Cisco HX Data Platform Release 4036

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2b)HyperFlex cluster does not heal after maintenance onnodeserver

CSCvr18528

40(2a)40(2a)High percentage of packet loss on the eth1 data interface9K MTU

CSCvr97089

40(2a)35(2f)The HyperFlex cluster healing state is stuck at 87 andis unable to progress

CSCvr16760

40(2a)40(2a)In some circumstances two replications can occur forthe same VM

From the GUI the replication status for this VM willshow stuck in an In Progress state but there are nojobs stuck in replication layer Subsequent Replicationsshould succeed

CSCvq22898

40(2a)35(2a)After an unexpected power outage the hyperflex clusteris unable to come up correctly

All processes are running time is synced and all nodescan vmkping each other as required

CSCvp13990

40(2a)40(1a)When specifying IP address instead of node ID stclinode maintenanceMode commands fail

CSCvp33657

40(2a)40(1a)2 node edge cluster if experiencing Intersightarbitratorconnectivity issues should NOT allowHXMaintenancemode to initiate as this will bring the cluster down

CSCvr31573

40(2a)35(2c)Old files in varsupportZKTxnlog are not purged withthe daily zklog-cleanup cron job

CSCvr89066

40(2a)35(2a)When a node is in MaintenanceMode any disk removalor replacement will be reflected in UI only after the nodeis brought back frommaintenancemode This is becausestorfs is not running on the node in maintenance andwill not be able to detect disk activities until it is broughtout of Maintenance Mode

CSCvo86431

40(2a)35(2c)Cluster might go offline and stop serving IO requestsrootcvm~ stcli cluster storage-summary--detailGet cluster storage summary failedjavanetConnectException Connection refused19216814210010207

CSCvr01645

40(2a)35(2d)A stretch cluster deployment fails with a messageFormatting nodes Node in Use on the Hyperflex installerand crmZoneType value is seen as 1 in storfscfg oroptspringpathconfigstretchtunes

CSCvq80340

Release Notes for Cisco HX Data Platform Release 4037

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)30(1c)During upgrade we upload the upgrade packagedownloaded from Ciscocom Upgrade packages are offile type tgz hxconnect acceptsstorfs-packages-401a-33028tar file a different type ofcompression and allows upgrade to start and fail whenit cannot decompress the file for bootstrap and scvmclientupgrade on ESXi

CSCvq63888

40(2a)40(2a)While attempting to install hyperflex via intersight thefollowing error is seen

Failed in Task Add hosts to vCenter Cluster with ErrorTry adding hosts manually to vCenter and retry Failedto get addressFailed to host by namelookupHostnamecompanycom on 000053 read udp12700145430-gt12700153 read connection refused

CSCvr52098

40(2a)35(2a)CoreAPI call may time out for cluster management APIwhile doing cluster create validation

CSCvo92952

40(2a)35(2d)Validation fails with Mixed mode expansion checkdue to Enhanced vMotion Compatibility Incompatibilitywhen the ESXi version is different

CSCvq95460

40(2a)30(1i)This failure is seen in installer when custom workflowis used hypervisor configuration + deploy (clean diskpartition=no) at Installing software packages on storagecontroller VM

CSCvr66309

40(2a)35(2d)The HX Installer apparently replaced a comma bysemicolon on the VLAN ID range while exporting theconfiguration to a JSON file

CSCvq93831

40(2a)40(1a)When using unsupported browsers HX Connect userswill not be able to login These users see an errorindicating unauthorized user

CSCvp22693

Release Notes for Cisco HX Data Platform Release 4038

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)40(1a)A vulnerability in the web-based management interfaceof Cisco HyperFlex Software could allow anunauthenticated remote attacker to conduct a cross-siterequest forgery (CSRF) attack on an affected system

The vulnerability is due to insufficient CSRF protectionsfor the web UI on an affected device An attacker couldexploit this vulnerability by persuading a user of theinterface to follow a malicious link A successful exploitcould allow the attacker to perform arbitrary actions withthe privilege level of the affected user

Cisco has released software updates that address thisvulnerability There are no workarounds that address thisvulnerability

This advisory is available at the following link

httpstoolsciscocomsecuritycentercontentCiscoSecurityAdvisorycisco-sa-20190807-hypflex-csrf

CSCvp24343

40(2a)35(2d)Add message to HX Connect Progress flow to notAcknowledge Pending Activities and reboot the serverson UCSM

HX Connect is doing a controlled rolling server upgradein the background

CSCvq22844

40(2a)40(1b)Compute Node expand operation fails due to incorrectcluster name

CSCvr43786

40(2a)26(1e)RF-2 cluster shutdown during data resynchronizationafter a node failure (non storfs) followed by multipledisk read failures and a HardBlacklist of a disk

CSCvp41241

40(2a)40(1a)HyperFlex Edge nodes do not properly set the ESXihostname during deployment

CSCvo13143

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4039

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4040

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1b) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401b-33133p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4041

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4042

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1b)40(1a)While running the HyperFlex installer with WindowsServer Hyper-V cluster creation process fails with theerror Failure occurred during ClusterCreation process Unable to post thecontent to the downstream This symptomis encountered while deploying HyperFlex cluster withHyperFlex nodes configuredordered with Cisco VIC1457 MLOM (PID HX-MLOM-C25Q-04) andWindows Server Datacenter or Core with Hyper-Vversion 2016 or 2019

CSCvp64140

Release Notes for Cisco HX Data Platform Release 4043

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)35(2b)Adding Micron 5200 drive to a cluster fails to increasecluster capacity

CSCvo69067

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4044

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4045

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1a) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401a-33028p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4046

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4047

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1a)30(1d)Cluster instability when disks of different sector sizeplaced in HX node

CSCvk17250

40(1a)

35(2c)

35(1a)When logged in using a local HX user account insteadof a Virtual Center account an error message appearsintermittently indicating Virtual Centerunreachable or Resource informationcannot be updated when VC is reachable

CSCvo36198

40(1a)

35(2b)

30(1d)HXDP does not work with EMC RecoverPoint - needsto support VMware API (FSS-Readdir)

CSCvk38003

Release Notes for Cisco HX Data Platform Release 4048

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)

35(2a)

30(1i)

25(1c)

25(1d)

Zookeeper (Exhibitor) process on HyperFlex StorageController VMs may fail to respond to requests whenvarzookeeper has low or no free space

CSCvm90352

40(1a)26(1c)Use Asynchronous consolidation for HX SnapshotsCSCvn02151

40(1a)35(2b)HX Quiesced Snapshot for Backup VendorsCSCvo90713

40(1a)35(2a)The cluster creationcluster expansion workflow stopswith the following error message at the validation step

FIRMWARE-Check UCSC-SAS-M5HD

FIRMWARE-Check UCSC-SAS-M5HD Required00000029000000320000003500000050 Found00000058

Action Needed Update the Controller Firmware toRequired Version

CSCvn17787

35(2b)30(1d)Kernel migration fails when a local datastore is searchedfor in the ESXi host

CSCvn73127

35(2a)26(1b)A node shuts down when two disks are replaced (acaching disk and another capacity disk) where thecapacity disk is inserted first and then the caching diskis inserted

CSCvk46364

Hyper-V

40(1a)35(2a)Cluster expansion may fail with an error code 500 -operation timed out

CSCvn28721

35(2b)35(2a)During upgrade remove the VLAN on team created foruser vSwitch During fresh installation only one VLANtag is set to the vSwitch and team although multipleVLANs were entered

CSCvn54300

35(2b)35(2a)While upgrading a Hyper-V cluster on account of a rarerace condition between the stUpgradeService andZookeeper servers the upgrade orchestration throws anupgrade validation error and the upgrade process isaborted

CSCvn60486

Release Notes for Cisco HX Data Platform Release 4049

Resolved Caveats in Release 40(1a)

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Startrestart EAM service invCenter

When EAM service is not runningvCenter registration is failing

CSCvt55712

35(2a)Contact Cisco TAC if you believeyou are seeing similar conditions

HyperFlex Stretch Cluster saw poorapplication performance due to writelatency cluster remained unhealthy andrebalance was stuck

This defect is currently being used forinvestigative purposes

CSCvt45344

35(2g)Remove log files fromvarlogspringpath folder when thelog files are not getting rotatedStop the relevant service removelarge log files fromvarlogspringpath folder andrestart the service

varstv folder in Controller VM maybecome full

CSCvt36374

40(2b)Restarting the stMgr will initializeit again

In a Cluster configured with DR networkStMgr may not get initialized and stuckin a deadlock while enabling the IPTablerules (can be verified from the stMgrlog)

CSCvt89709

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Click recover for the selectedVM again under the standalonemode This will move the VM torecovered state

When the user recovers a VM protectedin a group it along with other VMs in thegroup move to recovering state instandalonemode The selectedVM shouldmove to recovered state

CSCvs75553

35(2e)Clean up the fail over cluster andthen recreate the fail over clusterNo need to touch the HX storagecluster

Hyper-V When replicated DC was usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2d)40(1a)

Keep cluster utilization as low aspossible To avoid read-only statekeep below 92

HyperFlex clusters storage utilization isabove 92 usable capacity

CSCvr20922

Release Notes for Cisco HX Data Platform Release 4050

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1b)NAAll the disks in the node got locked afterrebooting the node

We successfully unlocked usingsed-clientsh -U command but wantedto test with another reboot and driveslocked again

CSCvs74286

35(2g)To recover the VM copy over thedata disks and attach them to thenew VM

After enable encryption HX connectencryption status showing Caution butall of the disks enable encryptionsuccessful

-Self Encrypting Drives Service isrunning on all of the nodes

-There is no error message of encryption

- USB0 interface is up

-All SED disks showing supported 1enabled 1 locked 0

CSCvs93245

40(1b)Enable kernelpanic_on_oops as apersistent configuration This willcause the node to panic and rebootimmediately

A cluster node running HX release40(1b) may hang in the Linux kernelThis is classified as an oops and adeviation from the expected behavior

CSCvs54285

Release Notes for Cisco HX Data Platform Release 4051

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Enabling K8iscsi stack in Hx clusterCSCvs41324

Release Notes for Cisco HX Data Platform Release 4052

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Steps to enable K8 on HX cluster

Perform following steps on ALLthe controller VMs

1 Updateetcinitscvmclientconf toenable tune iscsiEnable totrue

sed -iesiscsiEnable=falseiscsiEnable=trueetcinitscvmclientconf

2 Run following initctlcommand to reloadconfiguration

initctl reload-configuration

3 Restart scvmclient process

stop scvmclient startscvmclient

4 Run following initctlcommand to mount SYSTEMDS

initctl emit --no-waitsystem-datastore-created

5 Verify that iscsiEnable tuneis set to true using thefollowing command

ps -eaf | grep scvmclient |grep -v grep

root 6241 1 1 Dec06 014007optspringpathstorfs-corescvmclient-T logEnabled=false -TlogSyslogEnabled=true -TlogEchoToScreen=false -TstatLoggingToFile=false -TstatLoggingToSyslog=true-TlogDir=varlogspringpath-TnfsBackendServerList=1010748100-T iscsiEnable=true -TiscsiUseAsync=true -T

Release Notes for Cisco HX Data Platform Release 4053

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

iscsiConfigFilePath=nfsSYSTEMistgtconfps -eaf | grep scvmclient |grep -v grep

35(2e)Following procedure can be usedto check NFS Queue Depth andincrease if needed

rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth256 lt- Low value[rootHXESXI1] vsish -e setvmkModulesnfsclientmnt[DATASTORE]propertiesmaxQDepth 1024[rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth1024 lt- Optimalvalue

This requires an ESXi host rebootto take effect

Please place the node in HyperflexMaintenanceMode and gracefullyreboot the node for the changes tobe applied

HyperFlex Datastore NFS Queue Depthshows as 256 which can lead toperformance (including latency) issues

CSCvr83056

40(1b)This is cosmetic The error will goaway once the cluster returns tohealthy status

When a single node is offline in clusterstcli cluster storage-summary shows twonodes unavailable rootSCVM~ stclicluster storage-summary

CSCvs47735

35(2g)Modify both the root and adminpassword for the SCVMs to be thesame

stcli security password set -uadmin

Cluster expansion fails at Config Installerstage when the root and admin passwordfor the storage controller (SCVM) aredifferent

CSCvs69317

35(2d)NAAfter successfully changing the DNSserver on the HX controller we still cansee the original DNS entry that was addedduring the deployment

CSCvs69154

Release Notes for Cisco HX Data Platform Release 4054

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Customers are strongly encouragedto work with TAC in order toidentify and remediate this issue

1 Change getcluster from 2m to8m inoptspringpathstorfs-mgmtstMgr-10confapplicationconfon all nodes

2 Restart stMgr on all nodes

You may see this error message after afailed upgrade or other task such asattempting to enter a node into HXmaintenance mode

CSCvs53555

40(2a)Run the following command oneach storage controller VM as root

grep -i monitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp sed -iesmonitor_dns_serverstruemonitor_dns_serversfalseoptspringpathhx-diag-toolswatchdog_configjsonampamp grep -imonitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp restart watchdog

Receive the following alertevent in HXConnect HX Controller VMHOSTNAME one or more configuredDNS servers not responding

CSCvt13947

35(2b)Delete any empty (Size 0) log filesunder varlogzookeeperversion-2ORvarlogzookeeperstandaloneversion-2

Zookeeper fails to start while Exhibitor isrunning however echo srvr returnsnothing

CSCvs21562

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

40(4e)1 De-commission the server

2 Power drain the server -REMOVE BOTH POWERCORDS ON THE BACK OFTHE SERVER FOR 10SECONDS THENREINSERTPOWERCORDS

3 Re-commission the server

M2 boot disk is missing from serverinventory after upgrade to server firmware40(4e) As a result server fails to boot toOS installed in the M2 disk The issuepersists after re-acknowledgement as wellas de-commission andre-acknowledgement of the server

CSCvs02466

Release Notes for Cisco HX Data Platform Release 4055

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Re-start the flusher manuallyMany disks taken offline by a fault ormanually resulting in ENOSPC oncluster

CSCvq32721

35(2e)Clean up fail over cluster and thenrecreate the fail over cluster Thereis no need to touch the HX storagecluster

[Hyper-V]When replicated DCwas usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2b)Manually install upgradescvmclient VIB andmake sure thatit matches with HXDP version

Upgrade from release HX 35(2a) to35(2x) 4X does not upgrade scvmclientRecovery point may not work

CSCvq54992

Management

30(1b)To recover the VM copy over thedata disks and attach them to thenew VM

[HyperV]VMshows up onmultiple nodesCSCvj22992

35(1a)1 Do not change MTU at ESXlevel on running cluster

2 Reset back to the original value

ChangingMTU from 9000 to 1500 on oneESX node causes storfs process to restarton all nodes

CSCvm99150

Release Notes for Cisco HX Data Platform Release 4056

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)After the installation (for newdeployments) or after an upgradeto 401a (for existingdeployments) perform thefollowing steps on all ControllerVMs

1 Edit the following tune files (onall Controller VMs)

optspringpathconfiglfftunes

optspringpathconfigvsi_16tbtunes

2 SetcleanerEnableSegSummaryCleaning

to false

3 After editing the above tunefiles

a SSH in to all the ControllerVMs

b Run storfstool -- -Z

c Run the below command andcheck the values Tune valueshould be true

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=true

d Type below commands andapplied the tunes changesdynamically

echo false gttmpstprocfssystemtunecleanerEnableSegSummaryCleaning

e Verify that tune values arechanged Run below commandValue should be false

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=false

f umount tmpstprocfs

A cluster with 8TB or 12TB disk drivesmay experience IO stalls for severalminutes after another node fails in thecluster

CSCvp23718

Release Notes for Cisco HX Data Platform Release 4057

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)1 Update the conf file manuallyto change to the link local addressto

2 Restart scvmclient on thecontroller and ESX

HX 35(2b) FlexVol to HX 40 CSIupgrade does not work FlexVol continuesto work

CSCvp26319

35(2a)Disable rebalance until node healsthen reenable rebalance

7-node cluster healing very slowlybecause rebalance change AMS reqs areblocking healing process

CSCvp62512

35(2a)NAWith moderate cluster storage capacityusage under site failover cluster clustermay experience an all paths (APD) downcondition

CSCvq49412

40(1a)NAThis product includes Third-partySoftware that is affected by thevulnerabilities identified by the followingCommon Vulnerability and Exposures(CVE) IDs

CVE-2016-0762 CVE-2016-6797CVE-2016-6816 CVE-2016-8735CVE-2017-5647 CVE-2017-12615CVE-2017-12616 CVE-2017-12617CVE-2017-7674 CVE-2018-1304CVE-2018-8014 CVE-2018-1336CVE-2018-8034 CVE-2018-11784CVE-2019-0232

CSCvp36364

35(2d)NAstcli cluster info command needs toprovide UCSM VIP address Currently itshows ucsm-hostcom

CSCvq11456

35(2a)No workarounds currently Thestale entries do not affect the dayto day operations of HyperFlex

HyperFlex upgrade validation failedbecause cluster has extraneous stNodes ininternal database

CSCvq32530

35(2a)NAWhen not in maintenancemode all drivesappear in HX Connect and get updated asexpected

When in maintenance mode extra drivesare shown in HX Connect with no slotnumbers

CSCvq42378

35(2b)Contact TAC to cleanup duplicatenode

Nodes duplicated in HX Connect aftercluster recreate

CSCvq61825

Release Notes for Cisco HX Data Platform Release 4058

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)NACurrently HyperFlex installs on Hyper-Vdo not contain the stcli security whitelistcommandset

CSCvq66245

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

Release Notes for Cisco HX Data Platform Release 4059

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)[Hyper-V] Converged node expansionfails for incoming node with VIC 14xx

CSCvq39523

Release Notes for Cisco HX Data Platform Release 4060

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

1 Set Jumbo frame inWindowsHypervisor for incomingnode

Get-NetAdapter storage-data-a|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

Get-NetAdapter storage-data-b|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

2 Verify all interfaces are setcorrectly to support Jumbo frames

Get-NetAdapter storage |Get-NetAdapterAdvancedProperty| RegistryKeyword -matchjumbo | ft -auto

You should receive the followingmessage

Name DisplayName DisplayValue

RegistryKeyword RegistryValue

---- ----------- ------------

--------------- -------------

vswitch-hx-storage-data Jumbo

Packet 9014 Bytes

JumboPacket 9014

storage-data-b Jumbo Packet

Bytes 9014 JumboPacket

9014

storage-data-a Jumbo Packet

Bytes 9014 JumboPacket

9014

3 Reboot only the incoming nodeor the node being expanded

4 Retry Cluster Expansion in

Release Notes for Cisco HX Data Platform Release 4061

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

DescriptionDateRelease

Updated HUUCIMC info in theHyperFlex Edge and FirmwareCompatibility Matrix for 3x and4x Deployments

September 10 201940(1b)

Updated HUUCIMCrecommended firmware versionsfor HyperFlex Releases 40(1b)35(2e) and 35(2d)

August 28 201940(1b)

Updated Recommended FIServerFirmware versions for HyperFlexReleases 35(2e) and 35(2d)

August 23 201940(1b)

Added Cisco IMC version supportinfo in the HyperFlex Edge andFirmware CompatibilityMatrix for4x Deployments

August 21 201940(1b)

Created release notes for Cisco HXData Platform Software Release40(1b)

August 19 201940(1b)

Added bullet describing the CiscoHyperFlex SystemsUpgradeGuidefor Unsupported Cisco HXReleases in the UpgradeGuidelines section

August 8 201940(1a)

Added important note indicatingHyperFlex does not support UCSserver firmware 40(4a) 40(4b)and 40(4c)

August 5 201940(1a)

Updated component info forHX220c M5HXAF220c M5Cluster to VIC 1457 in HyperFlexEdge and Firmware CompatibilityMatrix for 4x Deploymentssection

July 25 201940(1a)

bull AddedRelease 35(2e) supportfor ESXi 67 U2 and updatednotes for 35(2d) 35(2c) and35(2b) support for ESXi 67U2 in Supported VMwarevSphere Versions andEditions

bull Added Witness Node Versionfor Release 35(2e)

July 22 201940(1a)

Release Notes for Cisco HX Data Platform Release 403

Revision History

DescriptionDateRelease

Added CSCvq39523 to the list ofOpen Caveats for Release 40(1a)

July 5 201940(1a)

bull Updated Important Note forSED-based HyperFlexsystems in SupportedVersions and SystemRequirements section

bull Updated Release 35(2b)support for ESXi 67 U2 inSupported VMware vSphereVersions and Editions

July 1 201940(1a)

bull Added CSCvp64140 andCSCvp98910 to the list ofOpen Caveats for Release40(1a)

bull Updated HyperFlex Edge andFirmware CompatibilityMatrix tables

June 20 201940(1a)

Updated BrowserRecommendations

June 17 201940(1a)

Added information indicating40(1a) features are supported onthe Intersight Virtual Appliance andon Intersightcom

May 31 201940(1a)

bull Updated M4M5Recommended FIServerFirmware for 35(2b) and40(1a)

bull Added bullet describingrecommended use of theHypercheck Health CheckUtility in the UpgradeGuidelines section

May 21 201940(1a)

bull Added New Featuredescription for DISA STIGCompliance

bull Updated Storage ClusterSpecifications for Hyper-V

bull Added CSCvp66277 to list ofOpen Caveats

May 14 201940(1a)

Release Notes for Cisco HX Data Platform Release 404

Revision History

DescriptionDateRelease

bull Updated VMware vCenterVersions for 40(1a)

bull Updated SupportedMicrosoftSoftware versions

May 8 201940(1a)

bull Added CSCvo36198 andCSCvk38003 to the list ofResolved Caveats

bull Added CSCvp21417 to the listof Open Caveats

May 3 201940(1a)

Created release notes for Cisco HXData Platform Software Release40(1a)

April 29 201940(1a)

New FeaturesCisco HX Data Platform Release 40 provides the following features These features including the InvisibleCloud Witness for HyperFlex Edge clusters are supported both on the Intersight Virtual Appliance and onIntersightcom

New Features in Release 40(2b)

bull 76TB SSD data drivemdashSupport on HX Edge configurations

bull All NVMe and All Flash limits increasemdashLimits increased on All NVMe (1TB 4TB and 8TB) andAll Flash (76TB SSD)

bull Cluster Scale Limits increasemdashSupport for maximum scale limit increase on a cluster See Cisco HXData Platform Storage Cluster Specifications on page 16

bull HW Offload optionmdashSupport for Hardware Offload option with Stretched cluster configurations

bull Cisco Overlay Transport Virtualization for Stretched ClustermdashSupport for OTV as an overlay forStretched Cluster

New Features in Release 40(2a)

bull Cisco UCS Manager 41(1c)mdashSupport for UCSM hardware and software enhancements and featuresfrom Cisco UCSM 41(1c) release

bull Boost ModemdashThis release introduces Boost Mode for the following configurations All NVMe AllFlash C240 All Flash C220 and Hypervisor ESX Boost Mode allows the Cisco HyperFlex cluster todeliver higher IOPs by increasing the storage controller VMCPU resources by 4 vCPU For configurationinformation see the Cisco HyperFlex Data Platform Administration Guide

bull Cisco HyperFlex HTML plug-in for VMWare vCentermdashEnables virtualization administrator tomanage and monitor the Cisco HyperFlex physical infrastructure by cross launching HyperFlex Connectfrom the vSphere Client UI and perform management actions in the HyperFlex Connect UI

Release Notes for Cisco HX Data Platform Release 405

New Features

bull 25GE networking for HX EdgemdashSupport for 25GE networking for HX Edge

bull All NVMe with Stretched ClustermdashSupport for All NVMe with Stretched Cluster (ESX only)

bull Cluster Upgrade Eligibility TestmdashThis release adds the capability to perform a pre-upgrade test whichchecks for cluster readiness before upgrading Example checks include validating cluster state rebalancestatus controllerVM Free Space ESXi version and much more The Elibility test is intended to helpavoid unexpected problems that may arise during the upgrade process It is highly recommended to runthe test before performing the Hyperflex upgrade

bull Registering Smart Software LicensingmdashThis release adds support for software that allows easy trackingof the status of license and software usage trends and simplifies the three core licensing functionsPurchasing Management and Reporting

bull Dynamic self-signed certificate generation enhancementsmdashThis release adds support for Self-signedSSL certificates on the Controller VMs which were static in prior releases The static certificates arereplaced with dynamically generated self-signed certificates upon upgrading to HXDP 40(2a) so thatthe certificates are unique per cluster The new clusters installed with HXDP 40(2a) have dynamicallygenerated self-signed certificates

bull Test Upgrade EligibilitymdashThis release adds support for testing your cluster readiness and infrastructurecompatibility for an upgrade For more information see the Test Upgrade Eligibility sections in theCiscoHyperFlex SystemsUpgrade Guide for VMware ESXi Release 40 or the CiscoHyperFlex UpgradeGuide for Microsoft Hyper-V Release 40

Disaster Recovery

bull Recovery Settings ConfigurationmdashThis release supports configuration of recovery settings to defineglobal recovery parameters and mapping for resources across recovery sites These parameters are usedduring recovery test recovery and migrate operations

bull HyperFlex DR Powershell RunbookmdashPowerShell runbook functionality is extended to support therecovery configuration settings in the runbooks for all recovery scenarios New-HXrunbook cmdlet cannow be used to generate runbook for a single or multiple protection groups In addition two new cmdletsWait-HXTask and Get-HXTaskStatus are introduced

bull Protected Virtual Machine ScalabilitymdashThis release adds support for 1500 VMs across both clustersand 750 VMs per cluster in a bi-direction or any split between the two clusters without exceeding thelimit of 1500 VMs For more information see Cisco HyperFlex Data Platform Administration Guide

bull System Management REST API enhancementsmdashPause data replication actions briefly using RESTAPI to explicitly inform users on the current status of replication actions

Cisco HyperFlex with Data Platform for Hyper-V

bull Cluster-Aware Updating (CAU)mdashThis is an automated feature that allows you to perform updates onwindows servers in a failover cluster with little or no loss in availability during the upgrade process

Support for new drivesmdashNew drives are qualified for the 40(2a) release The new drives include newcapacity points and new cache drive options Several of the new drives are alternate drives to already qualifiedexisting drives in function which are qualified in 40(2a) These drives are functionally compatible with theexisting drives and are available as alternates in case of lack of availability of existing drives For expansionof existing clusters or general information about interoperability of different drives see Cisco HyperFlexDrive Compatibility

Release Notes for Cisco HX Data Platform Release 406

New Features

NVMe Caching SSDs slot information is unavailable from HX-Connect for all AF server PIDs except forthe All-NVMe server PIDs Please refer to UCSM management console for NVMe SSD slot information

Note

Applicable PlatformsDrive PIDDrive Function

All existing HX M5 serversHX-M2-960GBAlternate boot drive

All existing HX M5 servers except AllNVMe

HX-SD480G6I1X-EVAlternate system (or housekeeping) drive

All existing HX M5 servers except AllNVMe

HX-SD480GM1X-EVAlternate system (or housekeeping) drive

The following HX M5 serversHXAF220C-M5SX HXAF240C-M5SXHXAF-E-220M5SX

HX-SD800G123X-EP800G 12G SAS Cache drive option forAll Flash

All NVMe HXAF220C-M5SNHX-NVME2H-I1000All NVMe 1TB Capacity drive

All NVMe HXAF220C-M5SNHX-NVME2H-I4000All NVMe 4TB Capacity drive

All NVMe HXAF220C-M5SNHX-NVMEHW-I8000New high density All NVMe 8TBCapacity drive

All Flash Configuration ndash namelyHXAF220C-M5SX HXAF240C-M5SXHXAF-E-220M5SX

ESX support only

HX-SD76T61X-EVNew high density All Flash 76TBCapacity drive

Maximum number of drives per node is12 drives on HX AF C240

HX240C-M5LHX-HD8T7K4KANAlternate drive for 8TB LFF capacity

HXAF220C-M5SX HXAF240C-M5SXHX-SD38T2HTNK9New 38TB FIPS compliant SED SSDdata drives

HXAF220C-M5SX HXAF240C-M5SXHX-SD960G2HTNK9New 960GFIPS compliant SEDSSD datadrives

New Features in Release 40(1b)

bull Support for Second Generation Intelreg Xeonreg Scalable Processor RefreshmdashThis release includessupport for the Second Generation Intelreg Xeonreg Scalable processor refresh (formerly Cascade Lake)

New Features in Release 40(1a)

The following new features are in Release 40(1a)

bull Ultra-Light HyperFlex Edge ClustersmdashThis release introduces support for two-node HyperFlex Edgeclusters enabling HyperFlex to run in environments requiring a small footprint Cisco Intersight providescomprehensive lifecycle management and includes remote cloud-based installation centralized upgradesand invisible witnessing Both 1GE and 10GE networking topology options are available

Release Notes for Cisco HX Data Platform Release 407

New Features

bull Scaled-Up HyperFlex Edge ClustersmdashThis release adds support for four node HyperFlex Edge clustersenabling a full range of size options for remote and branch offices Size the branch office environmentsto suit current needs with a two three or four node HyperFlex Edge cluster Cisco Intersight providesfull-lifecycle management and 1GE and 10GE networking options are available

bull Cisco Intersight Invisible Cloud WitnessmdashFor two node clusters this feature eliminates the need forwitness VMs the infrastructure to run those VMs and the management overhead to deploy scale andpatch witnessing software The Invisible Cloud Witness is responsible for maintaining cluster HA in theevent of failure scenarios This feature is included at no additional cost and is automatically deployedand managed by Cisco Intersight

bull Cloud-delivered HyperFlex Edge UpgradesmdashPowered by Cisco Intersight this feature adds supportfor multi-site orchestrated remote upgrades of the HyperFlex Data Platform This feature will be enabledwith the next 40 patch release and will allowHyperFlex Edge clusters deployed via Intersight to performorchestrated upgrades across one or many sites in parallel

bull All-NVMe HyperFlexmdashStarting with this release a new high-end performance node powered by allNVMe drives is available for HyperFlex clusters Co-engineered with Intel to support Intel VMD forhot-plug and surprise removal this offering represents an industry first an enterprise-ready and fullyvalidated all-NVMe HCI appliance The all-NVMe offering is available in the 220 form factor (1RU)and is powered by Intel Optane cache drives for maximum performance and highest endurance

bull VMware Site Recovery Manager (SRM) IntegrationmdashThis release brings support for a Cisco developedStorage Replication Adapter (SRA) for SRM The SRA provides the ability to leverage HyperFlex nativeasync replication with the powerful orchestration and runbook capabilities of SRM The SRA includesthe ability to perform test recoveries planned migrations and full disaster recovery

HX SRA is certified by VMware and is available for download from VMWareSRM site

Note

bull HyperFlex DR Powershell RunbooksmdashNew Powershell cmdlets are included for automated runbookgeneration when using HyperFlex native disaster recovery The New-HXRunbook cmdlet supports thefollowing workflows Test Recovery Planned Migration and Disaster Recovery These runbooks canbe used to orchestrate DR workflows without the requirement for any third-party software

bull Windows Server 2019 with Hyper-VmdashSupport has been added in this release for the Windows Server2019 operating system for Hyper-V based HyperFlex deployments

bull Kubernetes CSI PluginmdashThis release adds support for the HyperFlex CSI (HX-CSI) plugin based onthe Kubernetes Container Storage Interface (CSI) specification Customers can now use the HX-CSIplugin to provision andmanage persistent volumes in Kubernetes version 113 and later Note Kubernetes113 support for Cisco Container Platform amp Openshift Container Platform is forthcoming in theirrespective future releases

bull C480 ML Compute only nodemdashThis release introduces support for C480 ML as a new compute-onlynode for Deep LearningMachine Learning Workloads Data scientists can now use the power of up toeight NVidia SXM2 V100 GPUs to accelerate deep learning workloads VMs running deep-learningworkloads will need to use PCIe pass-through for access to GPUs

bull Higher Capacity DrivesmdashAnew 24TB 10k rpm SAS HDD option for SFF Hybrid HyperFlex clustersand a 12TB 72K rpmSASHDDoption for LFFHybrid are now available Both HyperFlex andHyperFlexEdge support the 24TB capacity point for maximum density in this form factor Note that HyperFlex

Release Notes for Cisco HX Data Platform Release 408

New Features

Edge does not support LFF-drives HyperFlex HyperV version does not yet support the new 12TB driveoption See the HyperFlex spec sheets for a full list of configurable options

bull New Cache and increased scale for Hyper-V mdashNVMe amp Optane SSDs are now supported as cachedrives for Hyper-V deployments Furthermore scale limits have been increased to 16+16(Converged+Compute-only) for both SFF (AF and Hybrid) amp LFF (Hybrid) clusters

bull Centralized Audit Log ExportmdashThis release adds support for audit logging via a remote syslog serverThis capability enables customers to retain audit logs from all HyperFlex nodes in a centralized remotesyslog server to meet retention and compliance requirements

bull DISA STIG CompliancemdashThis release adds new HX REST APIs for setting removing and checkingstatus of DISA STIGs for Controller VMs ESXi hosts and vCenter These APIs enable customers tomeet DISA security requirements by centrally and securely applying STIGs detecting and correctingfor drifts in any STIG settings

Supported Versions and System RequirementsCisco HX Data Platform requires specific software and hardware versions and networking settings forsuccessful installation

For a complete list of requirements see

bull Cisco HyperFlex Systems Installation Guide for VMware ESXi or

bull Cisco HyperFlex Systems Installation Guide for Microsoft Hyper-V

Hardware and Software Interoperability

For a complete list of hardware and software inter-dependencies refer to respective Cisco UCS Managerrelease version of Hardware and Software Interoperability for Cisco HyperFlex HX-Series

HyperFlex Software Versions

The HX componentsmdashCisco HX Data Platform Installer Cisco HX Data Platform and Cisco UCSfirmwaremdashare installed on different servers Verify that each component on each server used with and withinan HX Storage Cluster are compatible

bull HyperFlex does not support UCS Manager and UCS Server Firmware versions 40(4a) 40(4b)and 40(4c)

Do not upgrade to these versions of firmware

Do not upgrade to these versions of UCS Manager

Important

bull Verify that the preconfigured HX servers have the same version of Cisco UCS server firmware installedIf the Cisco UCS Fabric Interconnects (FI) firmware versions are different see the Cisco HyperFlexSystems Upgrade Guide for VMware ESXi Release 40 for steps to align the firmware versions

bull M4 For NEW hybrid or All Flash (Cisco HyperFlex HX240c M4 or HX220c M4) deploymentsverify that Cisco UCS Manager 31(3k) 32(3i) or 40(2b) or higher is installed For moreinformation see Recommended Cisco HyperFlex HX Data Platform Software Releases

Release Notes for Cisco HX Data Platform Release 409

Supported Versions and System Requirements

bull M5 For NEW hybrid or All Flash (Cisco HyperFlex HX240c M5 or HX220c M5) deploymentsverify that the recommended UCS firmware version is installed

For SED-based HyperFlex systems ensure that the A (Infrastructure) B (Bladeserver) and C (Rack server) bundles are at Cisco UCS Manager version 40(2b)or later for all SED M4M5 systems For more details see CSCvh04307 ForSED-based HyperFlex systems also ensure that all clusters are at HyperFlexRelease 35(2b) or later For more information see Field Notice (70234) andCSCvk17250

Important

bull To reinstall an HX server download supported and compatible versions of the software See theCisco HyperFlex Systems Installation Guide for VMware ESXi for the requirements and steps

bull Important For Intersight edge servers running older than 40(1a) CIMC version HUU is the suggestedmechanism to update the firmware

Table 1 HyperFlex Software Versions for M4M5 Servers

M5 Recommended FIServer Firmware

(be sure to review important notes above)

M4 Recommended FIServer Firmware

(be sure to review important notes above)

HyperFlexRelease

40(4h)40(4h)40(2b)

40(4h)40(4h)40(2a)

40(4h)40(4h)40(1b)

40(4e)40(4e)40(1a)

HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments

Cisco HX Data Platform Release 4x based Deployments

Confirm the component firmware on the server meets the minimum versions listed in the following tables

HyperFlex Edge does not support Cisco IMC versions 40(4a) 40(4b) 40(4c) 40(4d) and 40(4e)Important

Table 2 HX220c M4 HXAF220c M4 Cluster

Recommended Firmware Version - HXDP 4x

(be sure to review important note(s) above)

Component

402d0BIOS

40(2h)Cisco Integrated Management Controller(CIMC)

Release Notes for Cisco HX Data Platform Release 4010

HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments

Recommended Firmware Version - HXDP 4x

(be sure to review important note(s) above)

Component

13000012SAS HBA

43(2b)VIC 1227

0103 (Toshiba)

CS07 (Intel 480)

SSD

CN03 (Seagate)

CK03 (Seagate 18)

CK03 (Seagate 24)

sF3Q (Samsung)

3F3Q (Samsung)

HDD

40(2h)

Download Software

Host Upgrade Utility (HUU) Download Link

Table 3 HX220c M5 HXAF220c M5 Cluster

Recommended Firmware Version - HXDP 4x

(be sure to review important notes above)

Component

C220M5404p00224200755BIOS

40(4i)Cisco Integrated Management Controller(CIMC)

09000008SAS HBA

43(3c)VIC 1387

50(3e)VIC 1457

G201CS01 (Intel)

0104 (Toshiba)

0107 (Toshiba)

SSD

2F3Q (Samsung)

3F3Q (Samsung)

NOA4 (Seagate)

CK03 (Seagate 18tb)

CK03 (Seagate 24)

HDD

Release Notes for Cisco HX Data Platform Release 4011

HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments

Recommended Firmware Version - HXDP 4x

(be sure to review important notes above)

Component

40(4k)

Download Software

Host Upgrade Utility (HUU) Download Link

HyperFlex Licensing

As of version 26(1a) HyperFlex supports VMware PAC licensing Existing VMware embedded licenses willcontinue to be supported

As of version 25(1a) HyperFlex uses a smart licensing mechanism to apply your licenses See the CiscoHyperFlex Systems Installation Guide for VMware ESXi for details and steps

VMware vSphere Licensing Requirements

How you purchase your vSphere license determines how your license applies to your HyperFlex system

bull If you purchased your vSphere license with HyperFlex

Each HyperFlex server either has the Enterprise or Enterprise Plus edition preinstalled at the factory

bull HX Nodes have OEM licenses preinstalled If you delete or overwrite thecontent of the boot drives after receiving the HX servers you also delete thefactory-installed licenses

bull OEM license keys is a newVMware vCenter 60 U1b feature Earlier versionsdo not support OEM licenses

bull All factory-installed HX nodes share the same OEM license key WithvSphere OEM keys the Usage count can exceed the Capacity value

bull When you add an HX host to vCenter through the Add Host wizard in theAssign license section select the OEM license

We obfuscate the actual vSphere OEM license key for example0N085-XXXXX-XXXXX-XXXXX-10LHH

bull Standard Essentials Plus and ROBO editions are not available preinstalledon HX servers

Note

bull If you did NOT purchase your vSphere license with HyperFlex

The HX nodes have a vSphere Foundation license preinstalled After initial setup you can apply thelicense to a supported version of vSphere

bull If you purchased a vSphere PAC license

Follow the instructions in your PAC license letter from VMware to add the license to your MY VMwareaccount then follow the instructions to add your HX host to vCenter and assign the PAC license

Release Notes for Cisco HX Data Platform Release 4012

HyperFlex Licensing

HX Data Platform Software Versions for HyperFlex Witness Node

Witness Node VersionHyperFlex Release

10840(2b)

10840(2a)

10440(1b)

10440(1a)

Software Requirements for VMware ESXi

The software requirements include verification that you are using compatible versions of Cisco HyperFlexSystems (HX) components and VMware vSphere VMware vCenter and VMware ESXi

bull Verify that all HX servers have a compatible version of vSphere preinstalled

bull Verify that the vCenter version is the same or later than the ESXi version

bull Verify that the vCenter and ESXi versions are compatible by consulting the VMware ProductInteroperability Matrix Newer vCenter versions may be used with older ESXi versions so long as bothESXi and vCenter are supported in the table below

bull Verify that you have a vCenter administrator account with root-level privileges and the associatedpassword

For VIC1457 there is no support for ESXi 60Note

The below table applies for all of the following VMware vSphere Editions Enterprise Enterprise PlusStandard Essentials Plus ROBO

VMware vCenter VersionsVMware ESXi VersionsHyperFlex Version

60 U3 65 U3 67 U360 U3 65 U3 67 U340(2b)1

60 U3 65 U3 67 U360 U3 65 U3 67 U340(2a)2

60 U3 65 U3 67 U260 U3 65 U3 67 U2340(1b)

60 U3 65 U2 67 U260 U3 65 U2 67 U2440(1a)

1 Cisco HyperFlex Release 40(2) is the last major HyperFlex release that will support vSphere 60 (ESXiand vCenter) due to those versions reaching end of VMware general support on March 12 2020

2 Cisco HyperFlex Release 40(2) is the last major HyperFlex release that will support vSphere 60 (ESXiand vCenter) due to those versions reaching end of VMware general support on March 12 2020

3 Use of 67 U2 is not recommended see software advisory for CSCvq06952 and CSCvp88515 for furtherdetails

4 Use of 67 U2 is not recommended see software advisory for CSCvq06952 and CSCvp88515 for furtherdetails

Release Notes for Cisco HX Data Platform Release 4013

HX Data Platform Software Versions for HyperFlex Witness Node

Software Requirements for Microsoft Hyper-V

The software requirements include verification that you are using compatible versions of Cisco HyperFlexSystems (HX) components and Microsoft Hyper-V (Hyper-V) components

HyperFlex Software versions

The HX componentsmdashCisco HX Data Platform Installer Cisco HX Data Platform and Cisco UCSfirmwaremdashare installed on different servers Verify that each component on each server used with and withinthe HX Storage Cluster are compatible

bull Cisco HyperFlex M5 Converged nodesmdash For Hybrid (Cisco HyperFlex HX240c M5 HX220c M5)and All Flash (Cisco HyperFlex HXAF240cM5 HXAF220cM5) verify that Cisco UCSManager 40(2b)is installed HX 40 (1a) does not support Hyper-V on the All NVMe (HXAF220C-M5SN) nodes Fordetailed information on installation requirements and steps see theCisco HyperFlex Systems InstallationGuide on Microsoft Hyper-V

Table 4 Supported HyperFlex Software versions for M5 Servers on Hyper-V

M5 Recommended Server FirmwareHyperFlex Release

40(4h)40(2b)

40(4g)40(2a)

40(4e)40(1b)

40(4e)40(1a)

Release Notes for Cisco HX Data Platform Release 4014

Software Requirements for Microsoft Hyper-V

Table 5 Supported Microsoft Software versions

VersionMicrosoft Component

Windows Server 2016 Datacenter Core amp DesktopExperience

ForWindows Server 2016Datacenter Coreand Desktop Experience the Windows2016 ISO image should be Update BuildRevision (UBR) 1884 at a minimum

Note

Windows Server 2019Datacenter-DesktopExperienceis supported starting from HXDP 401(a) onwards

For Windows Server 2019 DesktopExperience theWindows 2019 ISO imageshould be Update Build Revision (UBR)107 at a minimum

Note

Windows Server 2019 DatacenterndashCore is notsupported currently

Also note that the following are currently notsupported

OEM activated ISOs and Retail ISOs are notsupported

Earlier versions ofWindows Server such asWindows2012r2 are not supported

Non-English versions of the ISO are not supported

Windows Operating System (Windows OS)

A Windows 2012 or later domain and forestfunctionality level

Active Directory

Supported Microsoft License Editions

The Microsoft Windows Server version that is installed on one or more HyperFlex hosts must be licensed asper Microsoft licensing requirements listed on Microsoft Licensing

Browser Recommendations

Use one of the following browsers to run the listed HyperFlex components These browsers have been testedand approved Other browsers might work but full functionality has not been tested and confirmed

Table 6 Supported Browsers

HX ConnectHX Data PlatformInstaller

Cisco UCS ManagerCisco IntersightBrowser

11 or higher11 or higher11 or higherNAMicrosoft InternetExplorer

70 or higher70 or higher57 or higher62 or higherGoogle Chrome

Release Notes for Cisco HX Data Platform Release 4015

Browser Recommendations

HX ConnectHX Data PlatformInstaller

Cisco UCS ManagerCisco IntersightBrowser

60 or higher60 or higher45 or higher57 or higherMozilla Firefox

NANA9 or higher10 or higherApple Safari

NANA35 or higherNAOpera

Notes

bull Cisco HyperFlex Connect

The minimum recommended resolution is 1024 X 768

bull Cisco HX Data Platform Plug-in

The Cisco HX Data Platform Plug-in runs in vSphere For VMware Host Client System browserrequirements see the VMware documentation at httpswwwvmwarecomsupportpubs

The HX Data Platform Plug-in is not displayed in the vCenter HTML client You must use the vCenterflash client

bull Cisco UCS Manager

The browser must support the following

bull Java Runtime Environment 16 or later

bull Adobe Flash Player 10 or higher is required for some features

For the latest browser information about Cisco UCSManager refer to themost recent Cisco UCSManagerGetting Started Guide

Cisco HX Data Platform Storage Cluster SpecificationsCluster Limits

bull Cisco HX Data Platform supports up to 100 clusters managed per vCenter as per VMware configurationmaximums

bull Cisco HX Data Platform supports any number of clusters on a single FI domain Each HX convergednode must be directly connected to a dedicated FI port on fabric A and fabric B without the use of a FEXC-series compute only nodes must also directly connect to both FIs B-series compute only nodes willconnect through a chassis IO module to both fabrics In the end the number of physical ports on the FIwill dictate the maximum cluster size and maximum number of individual clusters supported in a UCSdomain

The following table provides Cisco HX Data Platform Storage Cluster Specifications

Release Notes for Cisco HX Data Platform Release 4016

Cisco HX Data Platform Storage Cluster Specifications

Stretched Cluster (Availableon ESX Only)

Microsoft Hyper-VVMware ESXiNode

AllNVMe -HXAF220c-M5SN

HX240c-M5LHX220cM5

HX220cAF M5

HX240cM5

HX240cAF M5

HX240c-M5LHX220cM5

HX220cAF M5

HX240cM5

HX240cAF M5

HXAF220c-M5SN

AllNVMe -HXAF220c-M5SN

ESXi 65or 67only NotsupportedwithESXi 60orHyper-V

HX220cM5 Edge

HXAF220cM5 Edge

HX220cM4 Edge

HXAF220cM4 Edge

HX240c-M5LHX220cM5

HX220cAF M5

HX240cM5

HX240cAF M5

HX220cM4

HX220cAF M4

HX240cM4

HX240cAF M4

HXServers

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

C220M5C240M5B200M4B200M5

C240M5C220M5B200M4B200M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

mdashB200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

Compute-OnlyUCSB-SeriesC-SeriesServers

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

Convergednodesonly

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

SupportedNodes

Release Notes for Cisco HX Data Platform Release 4017

Cisco HX Data Platform Storage Cluster Specifications

Stretched Cluster (Availableon ESX Only)

Microsoft Hyper-VVMware ESXiNode

NA(requiresEnterpriseHXDP-PLicense)

NA(requiresEnterpriseHXDP-PLicense)

NA(requiresEnterpriseHXDP-PLicense)

Convergednodes3-16(12TBHDDoption isnotsupportedforHyperV)

Compute-onlynodes0-16

Convergednodes3-16

Compute-onlynodes0-16

NA(requiresEnterpriseHXDP-PLicense)

M4Convergednodes 3

M5Convergednodes23or 4

RequiresHXDP-ELicense

Convergednodes3-16(12TBdrive islimited toamaximumof 8nodes)

Compute-onlynodes0-16(12TBdrive islimited toamaximumof 8nodes)

Convergednodes3-32(76TBdriveconfigsarelimited tomaximumof 12drivesnodeon AF240)

Compute-onlynodes0-32

HXDP-SLicensedNodeLimits

11 ratioofHXDP-StoComputeonlynodes

(MinmdashMax)

Convergednodes2-16 perSite

Compute-onlynodes0-21 perSite

(up tomaxclustersize)

Convergednodes2-8 perSite

Compute-onlynodes0-16 perSite

(up tomaxclustersize)

Convergednodes2-16 perSite

Compute-onlynodes0-21 perSite

(up tomaxclustersize)

Convergednodes3-16(12TBHDDoption isnotsupportedforHyperV)

Compute-onlynodes0-16

Convergednodes3-16

Compute-onlynodes0-16

Convergednodes3-32

ComputeOnlynodes0-32

(up tomaxclustersize)

Convergednodes 3

(requiresHXDP-ELicense)

Convergednodes3-16(12TBdrive islimited toamaximumof 8nodes)

Compute-onlynodes0-32(12TBdrive islimited toamaximumof 16nodes)

Convergednodes3-32(76TBdriveconfigsarelimited tomaximumof 12drivesnodeon AF240)

Compute-onlynodes0-32

(up tomaxclustersize)

HXDP-PLicensedNodeLimits

12 ratioofHXDP-PtoComputeonlynodes

(MinmdashMax)

24 perSite 64percluster

24 perSite 48percluster

32 perSite 64percluster

32326434864MaxClusterSize

Release Notes for Cisco HX Data Platform Release 4018

Cisco HX Data Platform Storage Cluster Specifications

Stretched Cluster (Availableon ESX Only)

Microsoft Hyper-VVMware ESXiNode

212121111121mdash2121MaxComputetoConvergedratio

NoExpansion

Requires Enterprise license

Requires uniform expansion across both sites

Guidelines and Limitationsbull Starting with release 40(2a) SCVM is no longer needed on a Compute node

bull HX REST API Access Token Management ndash Applications leveraging HX REST APIs should re-useaccess tokens when making API calls Once obtained using the AAA Obtain Access Token API accesstokens are valid for 18 days (1555200 seconds) In addition AAA enforces rate limiting on ObtainAccess Token API requests in a 15 minute window auth can be invoked (successfully) a maximum of5 times A user is allowed to create a maximum of 8 unrevoked tokens Subsequent call to auth willautomatically revoke the oldest issued token tomake room for the new token Amaximum of 16 unrevokedtokens can be present in system In order to prevent brute-force attacks after 10 consecutive failedauthentication attempts a user account is locked for a period of 120 secondsFor more information seeCisco HyperFlex Systems REST API Reference guide

HxConnect makes use of AAA Authentication REST API for login and the above rate limit applies toHxConnect also

Upgrade Guidelines

The following list is a highlight of critical criteria for performing an upgrade of your HyperFlex system

bull Unsupported HX Data Platform 17x 18x 20 21x 25x and 26x clustersmdashUsers from anyversion prior to 26(1a) must step through an intermediate version before upgrading to 40 or later releasesIf you need to upgrade your environment from a Cisco HyperFlex HX Data Platform software releasethat is past the last date of support to the latest suggested release on the Cisco Software Download sitesee Cisco HyperFlex Systems Upgrade Guide for Unsupported Cisco HXReleases For more informationsee the Software Advisory for CSCvq66867 WARNING Only Use HXDP 26(1e) Upgrade PackageWhen Upgrading From HXDP 18(1a)-18(1e)

bull Hypercheck Health Check UtilitymdashCisco recommends running this proactive health check utility onyour HyperFlex cluster prior to upgrade These checks provide early visibility into any areas that mayneed attention andwill help ensure a seamless upgrade experience For more information see the HyperFlexHealth amp Pre-Upgrade Check Tool TechNote for full instructions on how to install and run Hypercheck

bull vSphere 67 Software AdvisorymdashDo not upgrade to Cisco HX Data Platform Release 40(1a) whenrunning ESXi 67U1 EP06 (build 11675023) Do not upgrade to 67U1 EP06 (build 11675023) ifrunning Cisco HX Data Platform Release 40(1a) See the Software Advisory CSCvo56350 for furtherdetails

Release Notes for Cisco HX Data Platform Release 4019

Guidelines and Limitations

The software build version posted at release will override any other local versions

bull Required vCenter upgrademdashFor enhanced security Cisco HX Data Platform Release 35(1a) or laterrequires the use of TLS 12 Therefore vCenter must be upgraded to 60 U3f or later before upgradingto Cisco HX Data Platform Release 35 or later In addition ESXi should be upgraded as required tomeet HX Data Platform compatibility requirements

bull Minimum HXDP version for upgrademdashHXData Platform clusters running 26(1a) or later may upgradedirectly to 40 using the HX Connect UI

bull Cluster ReadinessmdashEnsure that the cluster is properly bootstrapped and the updated plug-in is loadedbefore proceeding Manual cluster bootstrap is required for upgrade from a pre-35 release

bull Cluster ReadinessmdashEnsure that the cluster is properly bootstrapped and the updated plug-in is loadedbefore proceeding Manual cluster bootstrap is required for HX releases earlier than 35(1a) For moreinformation see theManual Bootstrap Upgrade Process in the Cisco HyperFlex Systems Upgrade Guidefor VMware ESXi Release 40 Do not skip this cluster bootstrap step it is required for all upgradesuntil HX Release 35(1a) Auto bootstrap is supported beginning with HX release 35(1a) For moreinformation see the Auto Bootstrap Upgrade Process from HX Connect UI in the Cisco HyperFlexSystems Upgrade Guide for VMware ESXi Release 40

Manual bootstrap is not supported on Intersight clusters

bull Initiating Upgrade―Use the HX Connect UI or CLI stcli commands when upgrading from 25(1a)or later releases Use either the CLI stcli commands or the HX Data Platform Plug-in to the vSphereWeb Client when upgrading from a pre-25(1a) release The vCenter plug-in should not be used forupgrades starting with the 25(1a) release The vCenter plug-in should not be used for upgrades startingwith the 25(1a) release

If the current cluster version is at 35(1a) or above you do not need to use the stcli command Directupgrade to 40 is possible

bull Complete your Upgrade―The self-healing (or rebalance) capability is disabled temporarily during theupgrade window If the upgrade fails you should complete the upgrade as soon as possible

bull ESXi and HXDP Compatibility―Ensure your cluster is running a compatible version of ESXi basedon the running the HXData Platform version (see the section Software Requirements for VMware ESXi)ESXi compatibility is determined by the major version and update release of ESXi It is generally bestto upgrade HXDP and ESXi together if combining the upgrade operations into a single optimized rebootWhen running a split upgrade first upgrade the HX Data Platform then proceed to upgrade ESXi

bull If Upgrading to vSphere 65

bull Certain cluster functions such as native and scheduled snapshotsReadyClones and Enter or Exit HX Maintenance Mode will not operatefrom the time the upgrade is started until the HX Data Platform upgrade iscomplete

bull After upgrading ESXi using the offline zip bundle use the ESX ExitMaintenance Mode option The HX Exit Maintenance Mode option doesnot operate in the vSphere Web Client until the HX Data Platform upgradeis complete

Note

Release Notes for Cisco HX Data Platform Release 4020

Guidelines and Limitations

bull vSphere 60 UpgradesmdashUsers on vSphere 60 migrating to 65 upgrade components in the followingorder

1 Upgrade HX Data Platform and UCS firmware

2 Upgrade HX Data Platform and ESXi

3 Upgrade HX Data Platform only first then upgrade ESXi or UCS firmware or both

bull M4 Server Firmware UpgradesmdashUpgrade server firmware to ensure smooth operation and to correctknown issues Specifically newer SAS HBA firmware is available in this release and is recommendedfor long-term stability

bull Users are encouraged to upgrade to 31(3c) C-bundle or later whenever possible

bull Users running C-bundle versions before 31(2f) must upgrade server firmware by performing acombined upgrade of UCS server firmware (C-bundle) to 31(3c) or later and HX Data Platform to25 Do not split the upgrade into two separate operations

bull If the cluster is already on 31(2f) C-bundle or later you may perform an HX Data Platform onlyor combined upgrade as required

bull M5 Server Firmware UpgradesmdashM5 generation servers must run firmware version 32(2d) or later

bull Firmware Downgrades mdashDowngrading UCSM from the HX-installer is not supported

bull M4M5 Mixed DomainsmdashA mixed domain occurs when a new separate M5 cluster is installed underthe same UCS domain that contains existing M4 clusters Under these conditions orchestrated UCSserver firmware upgrade will not operate until Cisco HX Data Platform Release 26 or later is installedon the M4 clusters Therefore it is best practice to first upgrade UCS server firmware to the latest 31(3)or 32(2) patch release before adding a new M5 cluster to the existing UCS domain Additionally any17 HX Data Platform clusters must first be upgraded before adding any new M5 clusters to the samedomain

bull Maintenance WindowmdashIf upgrading both HX Data Platform and UCS firmware you can select eithera combined or split upgrade through the vSphere HX Data Platform Plug-in depending on the length ofthe maintenance window Cisco UCSManager infrastructure upgrade is only supported using AutoInstalland the direct server firmware upgrade should be performed only through the upgrade orchestrationframework provided by the HX Data Platform Plug-in

bull Unsupported Self-Encrypting Drives (SEDs)mdashIf adding or replacing self-encrypting drives (SEDs)that have been recently qualified in newer versions of HX Data Platform insert the new drives only afterupgrading HX Data Platform to a compatible version All drives must be SED drives mixing SED andnon-SED is not supported

bullbull Enabling External Host AccessmdashWith Cisco HX Data Platform Release 40(1a) port 445 on themanagement network is blocked for enhanced security Note that prior to 40 port 445 port was openenabling external host access If you are upgrading to 40(1a) from a prior release and would like tocontinue external host access you can use a utility to open select hosts For more information aboutenabling external host access see the ConfiguringHyperFlex Share to SCVMM section in the InstallationGuide for Microsoft Hyper-V

Release Notes for Cisco HX Data Platform Release 4021

Guidelines and Limitations

Mixed Cluster Expansion Guidelinesbull Expanding existing M4 cluster with M5 converged nodes is supported

bull Expanding existing M5 cluster with M4 converged nodes is not supported

bull Expanding existing mixed M4M5 cluster with M4 or M5 converged nodes is supported

bull Adding any supported compute-only nodes is permitted with all M4 M5 and mixed M4M5 clustersusing the HX Data Platform 26 or later Installer Some example combinations are listed here manyother combinations are possibleExample combinationsExpand mixed M4M5 cluster with compute-only B200 C220 C240 M4M5Expand M4 cluster with compute-only B200 M5 C220 M5 C240M5

bull Only expansion workflow is supported to create a mixed cluster Initial cluster creation with mixedM4M5 servers is not supported

bull All M5 servers must match the form factor (220240) type (HybridAF) security capability (Non-SEDonly) amp disk configuration (QTY capacity and non-SED) of the existing M4 servers

bull HX220-M5 will use a maximum of 6 capacity disks (2 disk slots to remain empty) when mixedwith HX220-M4

bull HX Edge SED LFF Hyper-V and Stretched Clusters do not support mixed M4 and M5 clusters

Security FixesThe following security issues are resolved

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-13132 CVE-2019-9924CSCvq6313840(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-11719 CVE-2019-11727CVE-2019-11729

CSCvq7124040(2a)

Release Notes for Cisco HX Data Platform Release 4022

Mixed Cluster Expansion Guidelines

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-1125CSCvr0633940(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2015-9383 CVE-2018-14498CVE-2018-20406 CVE-2018-20852CVE-2019-10160 CVE-2019-13117CVE-2019-13118 CVE-2019-14287CVE-2019-14973 CVE-2019-15903CVE-2019-17546 CVE-2019-18197CVE-2019-18218 CVE-2019-5010CVE-2019-5094 CVE-2019-5481CVE-2019-5482 CVE-2019-9636CVE-2019-9740 CVE-2019-9947CVE-2019-9948

CSCvs0609440(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2017-13168 CVE-2017-18174CVE-2017-18216 CVE-2018-10876CVE-2018-10877 CVE-2018-10878CVE-2018-10879 CVE-2018-10880CVE-2018-10881 CVE-2018-10882CVE-2018-10902 CVE-2018-10938CVE-2018-12233 CVE-2018-12896CVE-2018-13053 CVE-2018-13094CVE-2018-13096 CVE-2018-13405CVE-2018-13406 CVE-2018-14609CVE-2018-14617 CVE-2018-14633CVE-2018-14734 CVE-2018-15572CVE-2018-15594 CVE-2018-16276CVE-2018-16658 CVE-2018-17182CVE-2018-17972 CVE-2018-18021CVE-2018-18690 CVE-2018-18710CVE-2018-6554 CVE-2018-6555CVE-2018-9363

CSCvp6501940(2a)

This vulnerability is due to insufficientHTML iframe protection An attackercould exploit this vulnerability bydirecting a user to an attacker-controlledweb page that contains amaliciousHTMLiframe A successful exploit could allowthe attacker to conduct clickjacking orother clientside browser attacks

NACSCvo9851640(2a)

Release Notes for Cisco HX Data Platform Release 4023

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerability is due to insufficientauthentication for the statistics collectionservice An attacker could exploit thisvulnerability by sending properlyformatted data values to the statisticscollection service of an affected deviceA successful exploit could allow theattacker to cause the web interfacestatistics view to present invalid data tousers

NACSCvj9558440(2a)

The vulnerability is due to insufficientCSRF protections for the web UI on anaffected device An attacker could exploitthis vulnerability by persuading a user ofthe interface to follow a malicious linkA successful exploit could allow theattacker to perform arbitrary actions withthe privilege level of the affected user

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvp2434340(2a)

Vulnerabilities with a version of cJSONidentified by CVE-2019-11834 andCVE-2019-11835

For more information see the relatedCisco Security Advisory

CVE-2019-11834 CVE-2019-11835CSCvq1994940(2a)

HX ESXi image patches for MCEPSCand TAA vulnerabilities with VMwareESXi identified by CVE-2018-12207 andCVE-20190-11135

For more information see the relatedCisco Security Advisory

CVE-2018-12207 CVE-2019-11135CSCvr5439840(2a)

CP networking vulnerabilities affectingthe Linux kernel identified byCVE-2019-11477 CVE-2019-11478CVE-2019-11479

For more information see the relatedCisco Security Advisory

CVE-2019-11477 CVE-2019-11478CVE-2019-11479

CSCvq1954640(2a)

Qualification of Microsoft securitypatches for vulnerabilities withMicrosoftHyper-V hypervisor identified withCVE-2018-12207 CVE-2019-11135

For more information see the relatedCisco Security Advisory

CVE-2018-12207 CVE-2019-11135CSCvr5439940(2a)

Release Notes for Cisco HX Data Platform Release 4024

Security Fixes

DescriptionCVEDefect IDRelease

OpenSSH vulnerabilities identified byCVE-2019-6111

For more information see the relatedCisco Security Advisory

CVE-2016-10708 CVE-2018-15473CVE-2018-20685 CVE-2019-6109CVE-2019-6111

CSCvp7646340(2a)

Multiple vulnerabilities associated withOpenSSL and LibNSS

For more information see the relatedCisco Security Advisory

CVE-2016-2105 CVE-2016-2106CVE-2016-2107 CVE-2016-2109CVE-2016-2176 CVE-2016-2177CVE-2016-2178 CVE-2016-2179CVE-2016-2180 CVE-2016-2181CVE-2016-2182 CVE-2016-2183CVE-2016-6302 CVE-2016-6303CVE-2016-6304 CVE-2016-6305CVE-2016-6306 CVE-2016-6307CVE-2016-6308 CVE-2016-7055CVE-2016-8610 CVE-2017-3731CVE-2017-3732 CVE-2017-3735CVE-2017-3736 CVE-2017-3737CVE-2017-3738 CVE-2018-0495CVE-2018-0732 CVE-2018-0734CVE-2018-0735 CVE-2018-0737CVE-2018-0739 CVE-2018-12384CVE-2018-12404 CVE-2018-5407CVE-2019-1559

CSCvp6667940(2a)

libpng vulnerability identified byCVE-2016-10087

For more information see the relatedCisco Security Advisory

CVE-2016-10087CSCvp6655540(2a)

Vulnerabilities associated with libjpg andlibpng

For more information see the relatedCisco Security Advisory

CVE-2014-9092 CVE-2016-3616CVE-2017-15232 CVE-2018-11212CVE-2018-11213 CVE-2018-11214CVE-2018-1152 CVE-2018-13785

CSCvp3458640(2a)

Vulnerabilities associated with Glibidentified by CVE-2018-16428 andCVE-2018-16429

For more information see the relatedCisco Security Advisory

CVE-2018-16428 CVE-2018-16429CSCvp3120740(2a)

Multiple vulnerabilities associated withcurl expat python 27 python 35 36and 37 freetype and giflib

For more information see the relatedCisco Security Advisory

CVE-2019-15133 CVE-2019-15903CVE-2019-5010 CVE-2019-5481CVE-2019-5482 CVE-2019-9636CVE-2019-9740 CVE-2019-9947CVE-2019-9948

CSCvr3690340(2a)

Release Notes for Cisco HX Data Platform Release 4025

Security Fixes

DescriptionCVEDefect IDRelease

Multiple third party softwarevulnerabilities

For more information see the relatedCisco Security Advisory

CVE-2019-14379 CVE-2019-12384CVE-2019-14439

CSCvq9203240(2a)

Vulnerabilities associated with elfutils

For more information see the relatedCisco Security Advisory

CVE-2018-16062 CVE-2018-16402CVE-2018-16403 CVE-2018-18310CVE-2018-18520 CVE-2018-18521CVE-2019-7149 CVE-2019-7150CVE-2019-7665

CSCvq4325040(2a)

Vulnerabilities associated with vim

For more information see the relatedCisco Security Advisory

CVE-2017-5953 CVE-2019-12735CSCvq4323040(2a)

Vulnerabilities associated with sqlite3

For more information see the relatedCisco Security Advisory

CVE-2016-6153 CVE-2017-10989CVE-2017-13685 CVE-2017-2518CVE-2017-2519 CVE-2017-2520CVE-2018-20346 CVE-2018-20505CVE-2018-20506 CVE-2019-8457CVE-2019-9936 CVE-2019-9937

CSCvq4321340(2a)

Vulnerabilities associated with glib20

For more information see the relatedCisco Security Advisory

CVE-2018-20843CSCvq4320940(2a)

Vulnerabilities associated with expat

For more information see the relatedCisco Security Advisory

CVE-2018-20843CSCvq4320540(2a)

Vulnerabilities associated with bzip2

For more information see the relatedCisco Security Advisory

CVE-2016-3189 CVE-2019-12900CSCvq4319440(2a)

Mulitple Vulnerabilities associated wtihNodeJS

For more information see the relatedCisco Security Advisory

CVE-2018-12115 CVE-2018-0734CVE-2018-5407 CVE-2018-12120CVE-2018-12121 CVE-2018-12122CVE-2018-12123 CVE-2018-12116CVE-2019-5737 CVE-2019-5739

CSCvq1069440(2a)

Vulnerabilities associated with PalletsJinja strformat_map and identified byCVE-2019-10906 and CVE-2016-10745

For more information see the relatedCisco Security Advisory

CVE-2019-10906 CVE-2016-10745CSCvq1038840(2a)

Release Notes for Cisco HX Data Platform Release 4026

Security Fixes

DescriptionCVEDefect IDRelease

Multiple Security Vulnerabilities

For more information see the relatedCisco Security Advisory

CVE-2014-8501 CVE-2014-9939CVE-2015-9262 CVE-2016-10087CVE-2016-2226 CVE-2016-4487CVE-2016-4488 CVE-2016-4489CVE-2016-4490 CVE-2016-4491CVE-2016-4492 CVE-2016-4493CVE-2016-6131 CVE-2018-10963CVE-2018-13785 CVE-2018-17100CVE-2018-17101 CVE-2018-18557CVE-2018-18661 CVE-2018-7456CVE-2018-8905

CSCvq0917840(2a)

Vulnerabilities associated withlibseccomp

For more information see the relatedCisco Security Advisory

CVE-2019-9893CSCvq0756840(2a)

Vulnerabilities associated withgdk-pixbuf

For more information see the relatedCisco Security Advisory

CVE-2017-12447CSCvq0675540(2a)

Vulnerabilities associated withpython-crypto

For more information see the relatedCisco Security Advisory

CVE-2018-6594CSCvp9381740(2a)

Vulnerabilities associated with wget

For more information see the relatedCisco Security Advisory

CVE-2018-20483 CVE-2019-5953CSCvp8672140(2a)

Multiple vulnerabilities associated withPython Crypto

For more information see the relatedCisco Security Advisory

CVE-2018-6594CSCvp6674840(2a)

Vulnerabilities associated withsensible-utils

For more information see the relatedCisco Security Advisory

CVE-2017-17512CSCvp6673440(2a)

Multiple vulnerabilities associated withPython

For more information see the relatedCisco Security Advisory

CVE-2018-1000030CSCvp6668940(2a)

Release Notes for Cisco HX Data Platform Release 4027

Security Fixes

DescriptionCVEDefect IDRelease

Multiple patch vulnerabilities identifiedby CVE-2016-10713

For more information see the relatedCisco Security Advisory

CVE-2016-10713CVE-2018-1000156 CVE-2018-6951

CSCvp6667240(2a)

Vulnerabilities associatedwith Little CMSidentified by CVE-2016-10165

For more information see the relatedCisco Security Advisory

CVE-2016-10165 CVE-2018-16435CSCvp6666440(2a)

Vulnerabilities associated with TenableScan identified byCVE-2011-5325CVE-2019-11068CVE-2016-7076CVE-2017-1000368

For more information see the relatedCisco Security Advisory

CVE-2011-5325 CVE-2016-7076CVE-2017-1000368CVE-2019-11068

CSCvp6474640(2a)

Vulnerabilities associated with libxml2294 and earlier as used in XMLSec1223 and earlier and other products andidentified by CVE-2016-9318

For more information see the relatedCisco Security Advisory

CVE-2016-9318 CVE-2017-16932CVE-2017-18258 CVE-2018-14404CVE-2018-14567

CSCvp3479240(2a)

Vulnerabilities associated with LFTPRemote File Names Unauthorized AccessVulnerability and identified byCVE-2018-10916

For more information see the relatedCisco Security Advisory

CVE-2018-10916CSCvp2926640(2a)

Vulnerabilities associated withCryptographyDeprecationWarning signerand verifier have been deprecated

For more information see the relatedCisco Security Advisory

CVE-2018-7750CSCvo3409740(2a)

Multiple Vulnerabilities in python python27 35

For more information see the relatedCisco Security Advisory

CVE-2019-16056CSCvr3979340(2a)

Release Notes for Cisco HX Data Platform Release 4028

Security Fixes

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toexecute commands as the root user

The vulnerability is due to insufficientinput validation An attacker could exploitthis vulnerability by connecting to thecluster service manager and injectingcommands into the bound process Asuccessful exploit could allow the attackerto run commands on the affected host asthe root user

Cisco has released software updates thataddress this vulnerability There areworkarounds that address thisvulnerability

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvq2417640(1b)35(2g)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

The vulnerability is due to an unprotectedlistening interface An attacker couldexploit this vulnerability by connectingto the listening interface and injectingcommands to the bound process Anexploit could allow the attacker to runcommands on the affected host as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvj9560640(1b)35(2g)

Release Notes for Cisco HX Data Platform Release 4029

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerabilities associated with theJVM18U121memory leak found duringVC alarms (concurrent 40 calls usingREST API)

CVE-2017-10053 CVE-2017-10067CVE-2017-10074 CVE-2017-10078CVE-2017-10081 CVE-2017-10087CVE-2017-10089 CVE-2017-10090CVE-2017-10096 CVE-2017-10101CVE-2017-10102 CVE-2017-10107CVE-2017-10108 CVE-2017-10109CVE-2017-10110 CVE-2017-10111CVE-2017-10115 CVE-2017-10116CVE-2017-10118 CVE-2017-10135CVE-2017-10176 CVE-2017-10193CVE-2017-10198 CVE-2017-10243CVE-2017-10274 CVE-2017-10281CVE-2017-10285 CVE-2017-10295CVE-2017-10345 CVE-2017-10346CVE-2017-10347 CVE-2017-10348CVE-2017-10349 CVE-2017-10350CVE-2017-10355 CVE-2017-10356CVE-2017-10357 CVE-2017-10388CVE-2017-3509 CVE-2017-3511CVE-2017-3526 CVE-2017-3533CVE-2017-3539 CVE-2017-3544CVE-2018-2579 CVE-2018-2582CVE-2018-2588 CVE-2018-2599CVE-2018-2602 CVE-2018-2603CVE-2018-2618 CVE-2018-2629CVE-2018-2633 CVE-2018-2634CVE-2018-2637 CVE-2018-2641CVE-2018-2663 CVE-2018-2677CVE-2018-2678 CVE-2018-2783CVE-2018-2790 CVE-2018-2794CVE-2018-2795 CVE-2018-2796CVE-2018-2797 CVE-2018-2798CVE-2018-2799 CVE-2018-2800CVE-2018-2814 CVE-2018-2815CVE-2018-2952 CVE-2018-3136CVE-2018-3139 CVE-2018-3149CVE-2018-3150 CVE-2018-3169CVE-2018-3180 CVE-2018-3183CVE-2018-3214 CVE-2019-2422

CSCvo8899740(1b)

Tomcat and Nginx logs are not beingcollected in the support bundle generatedthrough HX Connect in Release 35

NACSCvm5803140(1b)

The vulnerabilities associated with thelibmspack software package versionincluded in Cisco HX Data Platform

CVE-2018-18584

CVE-2018-18585

CSCvn3511940(1a)

Release Notes for Cisco HX Data Platform Release 4030

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerabilities associated withFasterXML Jackson-Databind TimeValueField Denial of Service

CVE-2018-14719

CVE-2018-14720

CVE-2018-1000873

CVE-2018-14721

CVE-2018-19360

CVE-2018-19362

CVE-2018-19361

CVE-2018-14718

CSCvn8228240(1a)

The vulnerabilities associated with theOpenSSL Protocol software packageversion included in Cisco HX DataPlatform

CVE-2013-3587CSCvo0505440(1a)

The vulnerabilities associated withThird-Party Software Denial of Service

CVE-2018-16487

CVE-2018-19361

CSCvo2781840(1a)

The vulnerabilities associated withLinuxkernel for Ubuntu 1710

CVE-2018-1092

CVE-2018-7492

CVE-2018-8087

CVE-2018-1068

CVE-2018-8781

CSCvm5314935(2a)

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)25(1d)stcli services dns remove should remove the DNS serverinfo from the interface files

CSCvs70967

40(2b)40(1b)HX 40(1b) new deployment from Intersight productioncloud

User receives following error while enabling CSI(Kubernetes integration)

Failure occurred during volume_access_enable Errorwas Nonzero exit code 1

CSCvt10522

40(2b)40(2a)Receive the following alertevent in HX Connect HXController VM HOSTNAME one or more configuredDNS servers not responding

CSCvt13947

Release Notes for Cisco HX Data Platform Release 4031

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)40(2a)API UI CLI show few Drive slots Empty after upgradeto 402a

The drives are seen in the sysmtool lsscsi and stclicommand outputs

The cluster is healthy with no errors

CSCvt14914

40(2b)35(2d)After successfully changing the DNS server on the HXcontroller we still can see the original DNS entry thatwas added during the deployment

CSCvs69154

40(2b)35(2e)HyperFlex Connect and VCenter show APD alarms forHX and non HX datastores This implies there is an issuewith HyperFlex when there may not be

CSCvs30080

40(2b)40(2a)An issue where all the disks in the node get locked afterrebooting the node

We successfully unlocked using sed-clientsh -Ucommand but wanted to test with another reboot anddrives locked again

CSCvs74286

40(2b)40(2a)When running stcli license commands on HyperFlexthe following error is seen

rootSpringpathControllertmp stcli license show all

Show smart licensing failed

Smart Agent is not ready please wait a minute and tryagain

CSCvt13929

40(2b)40(2a)After upgrading to HX 40(2a) the first RoboEdge nodethat gets rebooted then ignores persistent drivesPIDHX-HD24TB10K4KN

Cluster will be in degraded state and have reducedcapacity

CSCvt20144

40(2b)35(2b)Zookeeper fails to start while Exhibitor is runninghowever echo srvr returns nothing

CSCvs21562

40(2b)40(2a)When performing an HyperFlex upgrade a validationwarning may occur due to the host not having EnterprisePlus or Enterprise hypervisor licensing

Upgrade Validation Warning

ESXi host esx1labtest should be configured withVMware Enterprise license for upgrade to continue

CSCvs91787

Release Notes for Cisco HX Data Platform Release 4032

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)35(2g)Cluster expansion fails at Config Installer stage whenthe root and admin password for the storage controller(SCVM) are different

CSCvs69317

40(2b)35(2g)Panic while upgrading ESXiCSCvt06983

40(2b)40(2a)A cluster node running may hang in the Linux kernelThis is classified as an oops and a deviation from theexpected behavior

CSCvs54285

40(2b)35(2g)Rebalance failing on 10+10 node HX 35(2g) stretchcluster

CSCvs69007

40(2b)35(2d)The cluster becomes inaccessible to the IOVisorCSCvr54687

40(2b)40(2a)Panic on the storage controllerCSCvt61297

40(2b)40(2a)The size of support bundle is very large when collectedwith storfs-support command

CSCvt63306

40(2b)40(2a)storfs crash associated withFileSystemUsageWarningEvent orFileSystemUsageAlertEvent due to high usage ofvarstv above 80

CSCvt72807

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

40(2a)35(2d)In normal scenarios HX upgrade gets pendingACK afterupdating service profile In this case UCS only upgradewas stalled because after updating service profile HXupgrade was not getting pending ACK and it was alwayswaiting state

CSCvo39912

40(2a)35(1a)When using motherBoardReplace-12 to clean ZK fromold stNodepNodes resulted in unmounted datastore andmaking the size of the Hyperflex Datastores 0 resultingin all VMs in the cluster going offline

CSCvq39471

40(2a)26(1e)Upgrading cluster and getting error

Failed upgrade validations Checking vCenterconfiguration Reason Upgrade validations failed DRSFault Insufficient resources to satisfy configured failover

CSCvm77294

Release Notes for Cisco HX Data Platform Release 4033

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)The cluster expand fails on a node with DR replicationconfigured

When a HX cluster which has DR replication configuredis expanded we see the installer UI pulling in thereplicationVLAN information instead of themanagementVLAN information Even if we change that informationto the correct mgmt VLAN id and name it it does notseem to work as the node is configured with the VLANof the replicationVLAN in ESXi This leads to the failureof the node add with the host unreachable error

CSCvo70650

40(2a)35(1a)Customer reported that server didnt complete thefirmware upgrade automatically one by one

The user manually needs to put the host into maintenancemode and they manually acknowledged the pendingrequests to finish the UCS firmware upgrade

Maintenance policy is set to default (User-ack) as perthe design

CSCvo91624

40(2a)35(1i)When the cluster is in Failed state and stcli noderemove is attempted the output shows successful thoughthe node failed to get removed from the cluster

CSCvo93017

40(2a)35(1a)HyperFlex cluster upgrade may fail during validationwith a DRS Validation failed error

CSCvp31021

40(2a)35(2b)HX cluster expansion will fail with the following errormessage

MAC address pool configuration failure150[ErrorDescription] bad address block range definitioncollision

CSCvp58318

40(2a)35(2b)Memory usage by carbon cacheCSCvq34873

40(2a)35(2g)Upgrading ESXi cluster fails with error Nodemaintenance mode failed

CSCvr03240

40(2a)35(2e)storfs process does not automatically start on Exit HXMaintenance Mode or other tasks which restart StorageController

CSCvr88978

40(2a)35(2a)Post install script fails with message

Failed to execute ipmitool on HX node

CSCvp10707

40(2a)40(1a)HyperFlex expansionworkflow doesnt pull VLANnamecorrectly

CSCvp46539

Release Notes for Cisco HX Data Platform Release 4034

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)During HX cluster deployment - Validate ClusterCreation phase - HX Installer might fail with the error

fromvarlogspringpathstDeploylog

CSCvq45087

40(2a)35(2b)HX Installer fails to login to the SOL for configuringESX hosts

CSCvq91380

40(2a)35(2d)If DRS is enabled with configuration parameters otherthen fully automated ie manual or partially automatedthan this will change to fully automated during clusterexpansion

CSCvr44222

40(2a)35(1a)One of the validation task name has a typo The taskname is Validating node for mixed of SED and non-SEDdisks

It should be Validating node for mix of SED andnon-SED disks

CSCvp82175

40(2a)30(1i)This combination of using smaller NVMe drives is notsupported in expand operation For example if the drivesare only 375GB and they cannot be added to an existingcluster with larger caching SSD

CSCvo12359

40(2a)35(1a)HyperFlex includes a version of OpenSSL that is affectedby the vulnerabilities identified by the followingCommon Vulnerability and Exposures

CVE-2018-0495

CSCvp66679

40(2a)30(1e)SDK invoke exception when taking a native snapshot ona VM with GPU pci passthrough attached

CSCvo00511

40(2a)30(1i)Node replacement script fails due to EAM errorCSCvo62867

40(2a)35(1a)While pairing a cluster (HX release gt= 35) and cluster(HX release lt 35) remote replication network test failson the cluster (HX release lt 35)

CSCvo79760

40(2a)30(1b)Fixed the issue in the latest support-workflow bundleThis should not be hit anymore

CSCvo87061

40(2a)35(2a)On a 3 node cluster the MBreplace script fails as thescript looks for HEALTHY cluster status

CSCvo87080

40(2a)35(2a)MbReplace script (tar) hangs when running on 352b or40

CSCvp12359

Release Notes for Cisco HX Data Platform Release 4035

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)HX replication cleanup failing with error NA 35(2a) -INFODR state is not clean

CSCvp63958

40(2a)40(1a)The check_stig_parameters API incorrectly shows thecompliance state of the cluster as not STIG compliant

CSCvp66277

40(2a)35(2c)The snapshot creation on CBT enabled VM fails witherror Failed in vmreparent vmkfstools clone1

CSCvq06952

40(2a)35(2a)HyperFlex GUI when setting up remote DR-replicationpartnerpeer should show remote replication partner TCPport reachability test results

This will provide quick upfront connectivity results thatfrequently takes customer a while to manually check andconfirm

If reachability tests fail an info window and HX alertshould trigger that list the destination peer IP and thespecific port(s) that could not be reache

CSCvo60587

40(2a)35(2b)If you have more than one IP pool configured for areplication network and local replication network testsare failing you might be running into this issue

CSCvr67130

40(2a)35(2a)Skip task option in MBreplace and Software redeployscripts The scripts should have the option to skip tasksas user inputs and not by modifying the json file

CSCvp05204

40(2a)40(1a)Error noticed on SCVM Console print_req_error IOerror dev fd0 sector 0 on HyperV

It doesnt come up on SSH session

CSCvq34357

40(2a)30(1d)The VMNetwork switch is already created as part of theinstall process

Ideally the ip address assignment should be done duringthe initial install process as well No post install stepsshould be required altrnatively it could be done as partof a post install script

CSCvk36222

40(2a)25(1c)storfs service impacted during HX upgrade from HX21(1) to 25(1c)

HX node which is not being upgraded storfs servicestopped causing outage

CSCvg53223

40(2a)35(1b)HX Stretch Cluster Witness VM reverts to DHCP onreboot

CSCvp37536

40(2a)35(2b)HyperFlex Stretch Cluster Witness does not getprogrammed with an NTP server

CSCvp46578

Release Notes for Cisco HX Data Platform Release 4036

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2b)HyperFlex cluster does not heal after maintenance onnodeserver

CSCvr18528

40(2a)40(2a)High percentage of packet loss on the eth1 data interface9K MTU

CSCvr97089

40(2a)35(2f)The HyperFlex cluster healing state is stuck at 87 andis unable to progress

CSCvr16760

40(2a)40(2a)In some circumstances two replications can occur forthe same VM

From the GUI the replication status for this VM willshow stuck in an In Progress state but there are nojobs stuck in replication layer Subsequent Replicationsshould succeed

CSCvq22898

40(2a)35(2a)After an unexpected power outage the hyperflex clusteris unable to come up correctly

All processes are running time is synced and all nodescan vmkping each other as required

CSCvp13990

40(2a)40(1a)When specifying IP address instead of node ID stclinode maintenanceMode commands fail

CSCvp33657

40(2a)40(1a)2 node edge cluster if experiencing Intersightarbitratorconnectivity issues should NOT allowHXMaintenancemode to initiate as this will bring the cluster down

CSCvr31573

40(2a)35(2c)Old files in varsupportZKTxnlog are not purged withthe daily zklog-cleanup cron job

CSCvr89066

40(2a)35(2a)When a node is in MaintenanceMode any disk removalor replacement will be reflected in UI only after the nodeis brought back frommaintenancemode This is becausestorfs is not running on the node in maintenance andwill not be able to detect disk activities until it is broughtout of Maintenance Mode

CSCvo86431

40(2a)35(2c)Cluster might go offline and stop serving IO requestsrootcvm~ stcli cluster storage-summary--detailGet cluster storage summary failedjavanetConnectException Connection refused19216814210010207

CSCvr01645

40(2a)35(2d)A stretch cluster deployment fails with a messageFormatting nodes Node in Use on the Hyperflex installerand crmZoneType value is seen as 1 in storfscfg oroptspringpathconfigstretchtunes

CSCvq80340

Release Notes for Cisco HX Data Platform Release 4037

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)30(1c)During upgrade we upload the upgrade packagedownloaded from Ciscocom Upgrade packages are offile type tgz hxconnect acceptsstorfs-packages-401a-33028tar file a different type ofcompression and allows upgrade to start and fail whenit cannot decompress the file for bootstrap and scvmclientupgrade on ESXi

CSCvq63888

40(2a)40(2a)While attempting to install hyperflex via intersight thefollowing error is seen

Failed in Task Add hosts to vCenter Cluster with ErrorTry adding hosts manually to vCenter and retry Failedto get addressFailed to host by namelookupHostnamecompanycom on 000053 read udp12700145430-gt12700153 read connection refused

CSCvr52098

40(2a)35(2a)CoreAPI call may time out for cluster management APIwhile doing cluster create validation

CSCvo92952

40(2a)35(2d)Validation fails with Mixed mode expansion checkdue to Enhanced vMotion Compatibility Incompatibilitywhen the ESXi version is different

CSCvq95460

40(2a)30(1i)This failure is seen in installer when custom workflowis used hypervisor configuration + deploy (clean diskpartition=no) at Installing software packages on storagecontroller VM

CSCvr66309

40(2a)35(2d)The HX Installer apparently replaced a comma bysemicolon on the VLAN ID range while exporting theconfiguration to a JSON file

CSCvq93831

40(2a)40(1a)When using unsupported browsers HX Connect userswill not be able to login These users see an errorindicating unauthorized user

CSCvp22693

Release Notes for Cisco HX Data Platform Release 4038

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)40(1a)A vulnerability in the web-based management interfaceof Cisco HyperFlex Software could allow anunauthenticated remote attacker to conduct a cross-siterequest forgery (CSRF) attack on an affected system

The vulnerability is due to insufficient CSRF protectionsfor the web UI on an affected device An attacker couldexploit this vulnerability by persuading a user of theinterface to follow a malicious link A successful exploitcould allow the attacker to perform arbitrary actions withthe privilege level of the affected user

Cisco has released software updates that address thisvulnerability There are no workarounds that address thisvulnerability

This advisory is available at the following link

httpstoolsciscocomsecuritycentercontentCiscoSecurityAdvisorycisco-sa-20190807-hypflex-csrf

CSCvp24343

40(2a)35(2d)Add message to HX Connect Progress flow to notAcknowledge Pending Activities and reboot the serverson UCSM

HX Connect is doing a controlled rolling server upgradein the background

CSCvq22844

40(2a)40(1b)Compute Node expand operation fails due to incorrectcluster name

CSCvr43786

40(2a)26(1e)RF-2 cluster shutdown during data resynchronizationafter a node failure (non storfs) followed by multipledisk read failures and a HardBlacklist of a disk

CSCvp41241

40(2a)40(1a)HyperFlex Edge nodes do not properly set the ESXihostname during deployment

CSCvo13143

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4039

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4040

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1b) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401b-33133p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4041

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4042

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1b)40(1a)While running the HyperFlex installer with WindowsServer Hyper-V cluster creation process fails with theerror Failure occurred during ClusterCreation process Unable to post thecontent to the downstream This symptomis encountered while deploying HyperFlex cluster withHyperFlex nodes configuredordered with Cisco VIC1457 MLOM (PID HX-MLOM-C25Q-04) andWindows Server Datacenter or Core with Hyper-Vversion 2016 or 2019

CSCvp64140

Release Notes for Cisco HX Data Platform Release 4043

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)35(2b)Adding Micron 5200 drive to a cluster fails to increasecluster capacity

CSCvo69067

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4044

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4045

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1a) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401a-33028p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4046

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4047

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1a)30(1d)Cluster instability when disks of different sector sizeplaced in HX node

CSCvk17250

40(1a)

35(2c)

35(1a)When logged in using a local HX user account insteadof a Virtual Center account an error message appearsintermittently indicating Virtual Centerunreachable or Resource informationcannot be updated when VC is reachable

CSCvo36198

40(1a)

35(2b)

30(1d)HXDP does not work with EMC RecoverPoint - needsto support VMware API (FSS-Readdir)

CSCvk38003

Release Notes for Cisco HX Data Platform Release 4048

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)

35(2a)

30(1i)

25(1c)

25(1d)

Zookeeper (Exhibitor) process on HyperFlex StorageController VMs may fail to respond to requests whenvarzookeeper has low or no free space

CSCvm90352

40(1a)26(1c)Use Asynchronous consolidation for HX SnapshotsCSCvn02151

40(1a)35(2b)HX Quiesced Snapshot for Backup VendorsCSCvo90713

40(1a)35(2a)The cluster creationcluster expansion workflow stopswith the following error message at the validation step

FIRMWARE-Check UCSC-SAS-M5HD

FIRMWARE-Check UCSC-SAS-M5HD Required00000029000000320000003500000050 Found00000058

Action Needed Update the Controller Firmware toRequired Version

CSCvn17787

35(2b)30(1d)Kernel migration fails when a local datastore is searchedfor in the ESXi host

CSCvn73127

35(2a)26(1b)A node shuts down when two disks are replaced (acaching disk and another capacity disk) where thecapacity disk is inserted first and then the caching diskis inserted

CSCvk46364

Hyper-V

40(1a)35(2a)Cluster expansion may fail with an error code 500 -operation timed out

CSCvn28721

35(2b)35(2a)During upgrade remove the VLAN on team created foruser vSwitch During fresh installation only one VLANtag is set to the vSwitch and team although multipleVLANs were entered

CSCvn54300

35(2b)35(2a)While upgrading a Hyper-V cluster on account of a rarerace condition between the stUpgradeService andZookeeper servers the upgrade orchestration throws anupgrade validation error and the upgrade process isaborted

CSCvn60486

Release Notes for Cisco HX Data Platform Release 4049

Resolved Caveats in Release 40(1a)

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Startrestart EAM service invCenter

When EAM service is not runningvCenter registration is failing

CSCvt55712

35(2a)Contact Cisco TAC if you believeyou are seeing similar conditions

HyperFlex Stretch Cluster saw poorapplication performance due to writelatency cluster remained unhealthy andrebalance was stuck

This defect is currently being used forinvestigative purposes

CSCvt45344

35(2g)Remove log files fromvarlogspringpath folder when thelog files are not getting rotatedStop the relevant service removelarge log files fromvarlogspringpath folder andrestart the service

varstv folder in Controller VM maybecome full

CSCvt36374

40(2b)Restarting the stMgr will initializeit again

In a Cluster configured with DR networkStMgr may not get initialized and stuckin a deadlock while enabling the IPTablerules (can be verified from the stMgrlog)

CSCvt89709

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Click recover for the selectedVM again under the standalonemode This will move the VM torecovered state

When the user recovers a VM protectedin a group it along with other VMs in thegroup move to recovering state instandalonemode The selectedVM shouldmove to recovered state

CSCvs75553

35(2e)Clean up the fail over cluster andthen recreate the fail over clusterNo need to touch the HX storagecluster

Hyper-V When replicated DC was usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2d)40(1a)

Keep cluster utilization as low aspossible To avoid read-only statekeep below 92

HyperFlex clusters storage utilization isabove 92 usable capacity

CSCvr20922

Release Notes for Cisco HX Data Platform Release 4050

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1b)NAAll the disks in the node got locked afterrebooting the node

We successfully unlocked usingsed-clientsh -U command but wantedto test with another reboot and driveslocked again

CSCvs74286

35(2g)To recover the VM copy over thedata disks and attach them to thenew VM

After enable encryption HX connectencryption status showing Caution butall of the disks enable encryptionsuccessful

-Self Encrypting Drives Service isrunning on all of the nodes

-There is no error message of encryption

- USB0 interface is up

-All SED disks showing supported 1enabled 1 locked 0

CSCvs93245

40(1b)Enable kernelpanic_on_oops as apersistent configuration This willcause the node to panic and rebootimmediately

A cluster node running HX release40(1b) may hang in the Linux kernelThis is classified as an oops and adeviation from the expected behavior

CSCvs54285

Release Notes for Cisco HX Data Platform Release 4051

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Enabling K8iscsi stack in Hx clusterCSCvs41324

Release Notes for Cisco HX Data Platform Release 4052

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Steps to enable K8 on HX cluster

Perform following steps on ALLthe controller VMs

1 Updateetcinitscvmclientconf toenable tune iscsiEnable totrue

sed -iesiscsiEnable=falseiscsiEnable=trueetcinitscvmclientconf

2 Run following initctlcommand to reloadconfiguration

initctl reload-configuration

3 Restart scvmclient process

stop scvmclient startscvmclient

4 Run following initctlcommand to mount SYSTEMDS

initctl emit --no-waitsystem-datastore-created

5 Verify that iscsiEnable tuneis set to true using thefollowing command

ps -eaf | grep scvmclient |grep -v grep

root 6241 1 1 Dec06 014007optspringpathstorfs-corescvmclient-T logEnabled=false -TlogSyslogEnabled=true -TlogEchoToScreen=false -TstatLoggingToFile=false -TstatLoggingToSyslog=true-TlogDir=varlogspringpath-TnfsBackendServerList=1010748100-T iscsiEnable=true -TiscsiUseAsync=true -T

Release Notes for Cisco HX Data Platform Release 4053

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

iscsiConfigFilePath=nfsSYSTEMistgtconfps -eaf | grep scvmclient |grep -v grep

35(2e)Following procedure can be usedto check NFS Queue Depth andincrease if needed

rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth256 lt- Low value[rootHXESXI1] vsish -e setvmkModulesnfsclientmnt[DATASTORE]propertiesmaxQDepth 1024[rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth1024 lt- Optimalvalue

This requires an ESXi host rebootto take effect

Please place the node in HyperflexMaintenanceMode and gracefullyreboot the node for the changes tobe applied

HyperFlex Datastore NFS Queue Depthshows as 256 which can lead toperformance (including latency) issues

CSCvr83056

40(1b)This is cosmetic The error will goaway once the cluster returns tohealthy status

When a single node is offline in clusterstcli cluster storage-summary shows twonodes unavailable rootSCVM~ stclicluster storage-summary

CSCvs47735

35(2g)Modify both the root and adminpassword for the SCVMs to be thesame

stcli security password set -uadmin

Cluster expansion fails at Config Installerstage when the root and admin passwordfor the storage controller (SCVM) aredifferent

CSCvs69317

35(2d)NAAfter successfully changing the DNSserver on the HX controller we still cansee the original DNS entry that was addedduring the deployment

CSCvs69154

Release Notes for Cisco HX Data Platform Release 4054

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Customers are strongly encouragedto work with TAC in order toidentify and remediate this issue

1 Change getcluster from 2m to8m inoptspringpathstorfs-mgmtstMgr-10confapplicationconfon all nodes

2 Restart stMgr on all nodes

You may see this error message after afailed upgrade or other task such asattempting to enter a node into HXmaintenance mode

CSCvs53555

40(2a)Run the following command oneach storage controller VM as root

grep -i monitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp sed -iesmonitor_dns_serverstruemonitor_dns_serversfalseoptspringpathhx-diag-toolswatchdog_configjsonampamp grep -imonitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp restart watchdog

Receive the following alertevent in HXConnect HX Controller VMHOSTNAME one or more configuredDNS servers not responding

CSCvt13947

35(2b)Delete any empty (Size 0) log filesunder varlogzookeeperversion-2ORvarlogzookeeperstandaloneversion-2

Zookeeper fails to start while Exhibitor isrunning however echo srvr returnsnothing

CSCvs21562

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

40(4e)1 De-commission the server

2 Power drain the server -REMOVE BOTH POWERCORDS ON THE BACK OFTHE SERVER FOR 10SECONDS THENREINSERTPOWERCORDS

3 Re-commission the server

M2 boot disk is missing from serverinventory after upgrade to server firmware40(4e) As a result server fails to boot toOS installed in the M2 disk The issuepersists after re-acknowledgement as wellas de-commission andre-acknowledgement of the server

CSCvs02466

Release Notes for Cisco HX Data Platform Release 4055

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Re-start the flusher manuallyMany disks taken offline by a fault ormanually resulting in ENOSPC oncluster

CSCvq32721

35(2e)Clean up fail over cluster and thenrecreate the fail over cluster Thereis no need to touch the HX storagecluster

[Hyper-V]When replicated DCwas usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2b)Manually install upgradescvmclient VIB andmake sure thatit matches with HXDP version

Upgrade from release HX 35(2a) to35(2x) 4X does not upgrade scvmclientRecovery point may not work

CSCvq54992

Management

30(1b)To recover the VM copy over thedata disks and attach them to thenew VM

[HyperV]VMshows up onmultiple nodesCSCvj22992

35(1a)1 Do not change MTU at ESXlevel on running cluster

2 Reset back to the original value

ChangingMTU from 9000 to 1500 on oneESX node causes storfs process to restarton all nodes

CSCvm99150

Release Notes for Cisco HX Data Platform Release 4056

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)After the installation (for newdeployments) or after an upgradeto 401a (for existingdeployments) perform thefollowing steps on all ControllerVMs

1 Edit the following tune files (onall Controller VMs)

optspringpathconfiglfftunes

optspringpathconfigvsi_16tbtunes

2 SetcleanerEnableSegSummaryCleaning

to false

3 After editing the above tunefiles

a SSH in to all the ControllerVMs

b Run storfstool -- -Z

c Run the below command andcheck the values Tune valueshould be true

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=true

d Type below commands andapplied the tunes changesdynamically

echo false gttmpstprocfssystemtunecleanerEnableSegSummaryCleaning

e Verify that tune values arechanged Run below commandValue should be false

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=false

f umount tmpstprocfs

A cluster with 8TB or 12TB disk drivesmay experience IO stalls for severalminutes after another node fails in thecluster

CSCvp23718

Release Notes for Cisco HX Data Platform Release 4057

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)1 Update the conf file manuallyto change to the link local addressto

2 Restart scvmclient on thecontroller and ESX

HX 35(2b) FlexVol to HX 40 CSIupgrade does not work FlexVol continuesto work

CSCvp26319

35(2a)Disable rebalance until node healsthen reenable rebalance

7-node cluster healing very slowlybecause rebalance change AMS reqs areblocking healing process

CSCvp62512

35(2a)NAWith moderate cluster storage capacityusage under site failover cluster clustermay experience an all paths (APD) downcondition

CSCvq49412

40(1a)NAThis product includes Third-partySoftware that is affected by thevulnerabilities identified by the followingCommon Vulnerability and Exposures(CVE) IDs

CVE-2016-0762 CVE-2016-6797CVE-2016-6816 CVE-2016-8735CVE-2017-5647 CVE-2017-12615CVE-2017-12616 CVE-2017-12617CVE-2017-7674 CVE-2018-1304CVE-2018-8014 CVE-2018-1336CVE-2018-8034 CVE-2018-11784CVE-2019-0232

CSCvp36364

35(2d)NAstcli cluster info command needs toprovide UCSM VIP address Currently itshows ucsm-hostcom

CSCvq11456

35(2a)No workarounds currently Thestale entries do not affect the dayto day operations of HyperFlex

HyperFlex upgrade validation failedbecause cluster has extraneous stNodes ininternal database

CSCvq32530

35(2a)NAWhen not in maintenancemode all drivesappear in HX Connect and get updated asexpected

When in maintenance mode extra drivesare shown in HX Connect with no slotnumbers

CSCvq42378

35(2b)Contact TAC to cleanup duplicatenode

Nodes duplicated in HX Connect aftercluster recreate

CSCvq61825

Release Notes for Cisco HX Data Platform Release 4058

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)NACurrently HyperFlex installs on Hyper-Vdo not contain the stcli security whitelistcommandset

CSCvq66245

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

Release Notes for Cisco HX Data Platform Release 4059

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)[Hyper-V] Converged node expansionfails for incoming node with VIC 14xx

CSCvq39523

Release Notes for Cisco HX Data Platform Release 4060

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

1 Set Jumbo frame inWindowsHypervisor for incomingnode

Get-NetAdapter storage-data-a|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

Get-NetAdapter storage-data-b|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

2 Verify all interfaces are setcorrectly to support Jumbo frames

Get-NetAdapter storage |Get-NetAdapterAdvancedProperty| RegistryKeyword -matchjumbo | ft -auto

You should receive the followingmessage

Name DisplayName DisplayValue

RegistryKeyword RegistryValue

---- ----------- ------------

--------------- -------------

vswitch-hx-storage-data Jumbo

Packet 9014 Bytes

JumboPacket 9014

storage-data-b Jumbo Packet

Bytes 9014 JumboPacket

9014

storage-data-a Jumbo Packet

Bytes 9014 JumboPacket

9014

3 Reboot only the incoming nodeor the node being expanded

4 Retry Cluster Expansion in

Release Notes for Cisco HX Data Platform Release 4061

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

DescriptionDateRelease

Added CSCvq39523 to the list ofOpen Caveats for Release 40(1a)

July 5 201940(1a)

bull Updated Important Note forSED-based HyperFlexsystems in SupportedVersions and SystemRequirements section

bull Updated Release 35(2b)support for ESXi 67 U2 inSupported VMware vSphereVersions and Editions

July 1 201940(1a)

bull Added CSCvp64140 andCSCvp98910 to the list ofOpen Caveats for Release40(1a)

bull Updated HyperFlex Edge andFirmware CompatibilityMatrix tables

June 20 201940(1a)

Updated BrowserRecommendations

June 17 201940(1a)

Added information indicating40(1a) features are supported onthe Intersight Virtual Appliance andon Intersightcom

May 31 201940(1a)

bull Updated M4M5Recommended FIServerFirmware for 35(2b) and40(1a)

bull Added bullet describingrecommended use of theHypercheck Health CheckUtility in the UpgradeGuidelines section

May 21 201940(1a)

bull Added New Featuredescription for DISA STIGCompliance

bull Updated Storage ClusterSpecifications for Hyper-V

bull Added CSCvp66277 to list ofOpen Caveats

May 14 201940(1a)

Release Notes for Cisco HX Data Platform Release 404

Revision History

DescriptionDateRelease

bull Updated VMware vCenterVersions for 40(1a)

bull Updated SupportedMicrosoftSoftware versions

May 8 201940(1a)

bull Added CSCvo36198 andCSCvk38003 to the list ofResolved Caveats

bull Added CSCvp21417 to the listof Open Caveats

May 3 201940(1a)

Created release notes for Cisco HXData Platform Software Release40(1a)

April 29 201940(1a)

New FeaturesCisco HX Data Platform Release 40 provides the following features These features including the InvisibleCloud Witness for HyperFlex Edge clusters are supported both on the Intersight Virtual Appliance and onIntersightcom

New Features in Release 40(2b)

bull 76TB SSD data drivemdashSupport on HX Edge configurations

bull All NVMe and All Flash limits increasemdashLimits increased on All NVMe (1TB 4TB and 8TB) andAll Flash (76TB SSD)

bull Cluster Scale Limits increasemdashSupport for maximum scale limit increase on a cluster See Cisco HXData Platform Storage Cluster Specifications on page 16

bull HW Offload optionmdashSupport for Hardware Offload option with Stretched cluster configurations

bull Cisco Overlay Transport Virtualization for Stretched ClustermdashSupport for OTV as an overlay forStretched Cluster

New Features in Release 40(2a)

bull Cisco UCS Manager 41(1c)mdashSupport for UCSM hardware and software enhancements and featuresfrom Cisco UCSM 41(1c) release

bull Boost ModemdashThis release introduces Boost Mode for the following configurations All NVMe AllFlash C240 All Flash C220 and Hypervisor ESX Boost Mode allows the Cisco HyperFlex cluster todeliver higher IOPs by increasing the storage controller VMCPU resources by 4 vCPU For configurationinformation see the Cisco HyperFlex Data Platform Administration Guide

bull Cisco HyperFlex HTML plug-in for VMWare vCentermdashEnables virtualization administrator tomanage and monitor the Cisco HyperFlex physical infrastructure by cross launching HyperFlex Connectfrom the vSphere Client UI and perform management actions in the HyperFlex Connect UI

Release Notes for Cisco HX Data Platform Release 405

New Features

bull 25GE networking for HX EdgemdashSupport for 25GE networking for HX Edge

bull All NVMe with Stretched ClustermdashSupport for All NVMe with Stretched Cluster (ESX only)

bull Cluster Upgrade Eligibility TestmdashThis release adds the capability to perform a pre-upgrade test whichchecks for cluster readiness before upgrading Example checks include validating cluster state rebalancestatus controllerVM Free Space ESXi version and much more The Elibility test is intended to helpavoid unexpected problems that may arise during the upgrade process It is highly recommended to runthe test before performing the Hyperflex upgrade

bull Registering Smart Software LicensingmdashThis release adds support for software that allows easy trackingof the status of license and software usage trends and simplifies the three core licensing functionsPurchasing Management and Reporting

bull Dynamic self-signed certificate generation enhancementsmdashThis release adds support for Self-signedSSL certificates on the Controller VMs which were static in prior releases The static certificates arereplaced with dynamically generated self-signed certificates upon upgrading to HXDP 40(2a) so thatthe certificates are unique per cluster The new clusters installed with HXDP 40(2a) have dynamicallygenerated self-signed certificates

bull Test Upgrade EligibilitymdashThis release adds support for testing your cluster readiness and infrastructurecompatibility for an upgrade For more information see the Test Upgrade Eligibility sections in theCiscoHyperFlex SystemsUpgrade Guide for VMware ESXi Release 40 or the CiscoHyperFlex UpgradeGuide for Microsoft Hyper-V Release 40

Disaster Recovery

bull Recovery Settings ConfigurationmdashThis release supports configuration of recovery settings to defineglobal recovery parameters and mapping for resources across recovery sites These parameters are usedduring recovery test recovery and migrate operations

bull HyperFlex DR Powershell RunbookmdashPowerShell runbook functionality is extended to support therecovery configuration settings in the runbooks for all recovery scenarios New-HXrunbook cmdlet cannow be used to generate runbook for a single or multiple protection groups In addition two new cmdletsWait-HXTask and Get-HXTaskStatus are introduced

bull Protected Virtual Machine ScalabilitymdashThis release adds support for 1500 VMs across both clustersand 750 VMs per cluster in a bi-direction or any split between the two clusters without exceeding thelimit of 1500 VMs For more information see Cisco HyperFlex Data Platform Administration Guide

bull System Management REST API enhancementsmdashPause data replication actions briefly using RESTAPI to explicitly inform users on the current status of replication actions

Cisco HyperFlex with Data Platform for Hyper-V

bull Cluster-Aware Updating (CAU)mdashThis is an automated feature that allows you to perform updates onwindows servers in a failover cluster with little or no loss in availability during the upgrade process

Support for new drivesmdashNew drives are qualified for the 40(2a) release The new drives include newcapacity points and new cache drive options Several of the new drives are alternate drives to already qualifiedexisting drives in function which are qualified in 40(2a) These drives are functionally compatible with theexisting drives and are available as alternates in case of lack of availability of existing drives For expansionof existing clusters or general information about interoperability of different drives see Cisco HyperFlexDrive Compatibility

Release Notes for Cisco HX Data Platform Release 406

New Features

NVMe Caching SSDs slot information is unavailable from HX-Connect for all AF server PIDs except forthe All-NVMe server PIDs Please refer to UCSM management console for NVMe SSD slot information

Note

Applicable PlatformsDrive PIDDrive Function

All existing HX M5 serversHX-M2-960GBAlternate boot drive

All existing HX M5 servers except AllNVMe

HX-SD480G6I1X-EVAlternate system (or housekeeping) drive

All existing HX M5 servers except AllNVMe

HX-SD480GM1X-EVAlternate system (or housekeeping) drive

The following HX M5 serversHXAF220C-M5SX HXAF240C-M5SXHXAF-E-220M5SX

HX-SD800G123X-EP800G 12G SAS Cache drive option forAll Flash

All NVMe HXAF220C-M5SNHX-NVME2H-I1000All NVMe 1TB Capacity drive

All NVMe HXAF220C-M5SNHX-NVME2H-I4000All NVMe 4TB Capacity drive

All NVMe HXAF220C-M5SNHX-NVMEHW-I8000New high density All NVMe 8TBCapacity drive

All Flash Configuration ndash namelyHXAF220C-M5SX HXAF240C-M5SXHXAF-E-220M5SX

ESX support only

HX-SD76T61X-EVNew high density All Flash 76TBCapacity drive

Maximum number of drives per node is12 drives on HX AF C240

HX240C-M5LHX-HD8T7K4KANAlternate drive for 8TB LFF capacity

HXAF220C-M5SX HXAF240C-M5SXHX-SD38T2HTNK9New 38TB FIPS compliant SED SSDdata drives

HXAF220C-M5SX HXAF240C-M5SXHX-SD960G2HTNK9New 960GFIPS compliant SEDSSD datadrives

New Features in Release 40(1b)

bull Support for Second Generation Intelreg Xeonreg Scalable Processor RefreshmdashThis release includessupport for the Second Generation Intelreg Xeonreg Scalable processor refresh (formerly Cascade Lake)

New Features in Release 40(1a)

The following new features are in Release 40(1a)

bull Ultra-Light HyperFlex Edge ClustersmdashThis release introduces support for two-node HyperFlex Edgeclusters enabling HyperFlex to run in environments requiring a small footprint Cisco Intersight providescomprehensive lifecycle management and includes remote cloud-based installation centralized upgradesand invisible witnessing Both 1GE and 10GE networking topology options are available

Release Notes for Cisco HX Data Platform Release 407

New Features

bull Scaled-Up HyperFlex Edge ClustersmdashThis release adds support for four node HyperFlex Edge clustersenabling a full range of size options for remote and branch offices Size the branch office environmentsto suit current needs with a two three or four node HyperFlex Edge cluster Cisco Intersight providesfull-lifecycle management and 1GE and 10GE networking options are available

bull Cisco Intersight Invisible Cloud WitnessmdashFor two node clusters this feature eliminates the need forwitness VMs the infrastructure to run those VMs and the management overhead to deploy scale andpatch witnessing software The Invisible Cloud Witness is responsible for maintaining cluster HA in theevent of failure scenarios This feature is included at no additional cost and is automatically deployedand managed by Cisco Intersight

bull Cloud-delivered HyperFlex Edge UpgradesmdashPowered by Cisco Intersight this feature adds supportfor multi-site orchestrated remote upgrades of the HyperFlex Data Platform This feature will be enabledwith the next 40 patch release and will allowHyperFlex Edge clusters deployed via Intersight to performorchestrated upgrades across one or many sites in parallel

bull All-NVMe HyperFlexmdashStarting with this release a new high-end performance node powered by allNVMe drives is available for HyperFlex clusters Co-engineered with Intel to support Intel VMD forhot-plug and surprise removal this offering represents an industry first an enterprise-ready and fullyvalidated all-NVMe HCI appliance The all-NVMe offering is available in the 220 form factor (1RU)and is powered by Intel Optane cache drives for maximum performance and highest endurance

bull VMware Site Recovery Manager (SRM) IntegrationmdashThis release brings support for a Cisco developedStorage Replication Adapter (SRA) for SRM The SRA provides the ability to leverage HyperFlex nativeasync replication with the powerful orchestration and runbook capabilities of SRM The SRA includesthe ability to perform test recoveries planned migrations and full disaster recovery

HX SRA is certified by VMware and is available for download from VMWareSRM site

Note

bull HyperFlex DR Powershell RunbooksmdashNew Powershell cmdlets are included for automated runbookgeneration when using HyperFlex native disaster recovery The New-HXRunbook cmdlet supports thefollowing workflows Test Recovery Planned Migration and Disaster Recovery These runbooks canbe used to orchestrate DR workflows without the requirement for any third-party software

bull Windows Server 2019 with Hyper-VmdashSupport has been added in this release for the Windows Server2019 operating system for Hyper-V based HyperFlex deployments

bull Kubernetes CSI PluginmdashThis release adds support for the HyperFlex CSI (HX-CSI) plugin based onthe Kubernetes Container Storage Interface (CSI) specification Customers can now use the HX-CSIplugin to provision andmanage persistent volumes in Kubernetes version 113 and later Note Kubernetes113 support for Cisco Container Platform amp Openshift Container Platform is forthcoming in theirrespective future releases

bull C480 ML Compute only nodemdashThis release introduces support for C480 ML as a new compute-onlynode for Deep LearningMachine Learning Workloads Data scientists can now use the power of up toeight NVidia SXM2 V100 GPUs to accelerate deep learning workloads VMs running deep-learningworkloads will need to use PCIe pass-through for access to GPUs

bull Higher Capacity DrivesmdashAnew 24TB 10k rpm SAS HDD option for SFF Hybrid HyperFlex clustersand a 12TB 72K rpmSASHDDoption for LFFHybrid are now available Both HyperFlex andHyperFlexEdge support the 24TB capacity point for maximum density in this form factor Note that HyperFlex

Release Notes for Cisco HX Data Platform Release 408

New Features

Edge does not support LFF-drives HyperFlex HyperV version does not yet support the new 12TB driveoption See the HyperFlex spec sheets for a full list of configurable options

bull New Cache and increased scale for Hyper-V mdashNVMe amp Optane SSDs are now supported as cachedrives for Hyper-V deployments Furthermore scale limits have been increased to 16+16(Converged+Compute-only) for both SFF (AF and Hybrid) amp LFF (Hybrid) clusters

bull Centralized Audit Log ExportmdashThis release adds support for audit logging via a remote syslog serverThis capability enables customers to retain audit logs from all HyperFlex nodes in a centralized remotesyslog server to meet retention and compliance requirements

bull DISA STIG CompliancemdashThis release adds new HX REST APIs for setting removing and checkingstatus of DISA STIGs for Controller VMs ESXi hosts and vCenter These APIs enable customers tomeet DISA security requirements by centrally and securely applying STIGs detecting and correctingfor drifts in any STIG settings

Supported Versions and System RequirementsCisco HX Data Platform requires specific software and hardware versions and networking settings forsuccessful installation

For a complete list of requirements see

bull Cisco HyperFlex Systems Installation Guide for VMware ESXi or

bull Cisco HyperFlex Systems Installation Guide for Microsoft Hyper-V

Hardware and Software Interoperability

For a complete list of hardware and software inter-dependencies refer to respective Cisco UCS Managerrelease version of Hardware and Software Interoperability for Cisco HyperFlex HX-Series

HyperFlex Software Versions

The HX componentsmdashCisco HX Data Platform Installer Cisco HX Data Platform and Cisco UCSfirmwaremdashare installed on different servers Verify that each component on each server used with and withinan HX Storage Cluster are compatible

bull HyperFlex does not support UCS Manager and UCS Server Firmware versions 40(4a) 40(4b)and 40(4c)

Do not upgrade to these versions of firmware

Do not upgrade to these versions of UCS Manager

Important

bull Verify that the preconfigured HX servers have the same version of Cisco UCS server firmware installedIf the Cisco UCS Fabric Interconnects (FI) firmware versions are different see the Cisco HyperFlexSystems Upgrade Guide for VMware ESXi Release 40 for steps to align the firmware versions

bull M4 For NEW hybrid or All Flash (Cisco HyperFlex HX240c M4 or HX220c M4) deploymentsverify that Cisco UCS Manager 31(3k) 32(3i) or 40(2b) or higher is installed For moreinformation see Recommended Cisco HyperFlex HX Data Platform Software Releases

Release Notes for Cisco HX Data Platform Release 409

Supported Versions and System Requirements

bull M5 For NEW hybrid or All Flash (Cisco HyperFlex HX240c M5 or HX220c M5) deploymentsverify that the recommended UCS firmware version is installed

For SED-based HyperFlex systems ensure that the A (Infrastructure) B (Bladeserver) and C (Rack server) bundles are at Cisco UCS Manager version 40(2b)or later for all SED M4M5 systems For more details see CSCvh04307 ForSED-based HyperFlex systems also ensure that all clusters are at HyperFlexRelease 35(2b) or later For more information see Field Notice (70234) andCSCvk17250

Important

bull To reinstall an HX server download supported and compatible versions of the software See theCisco HyperFlex Systems Installation Guide for VMware ESXi for the requirements and steps

bull Important For Intersight edge servers running older than 40(1a) CIMC version HUU is the suggestedmechanism to update the firmware

Table 1 HyperFlex Software Versions for M4M5 Servers

M5 Recommended FIServer Firmware

(be sure to review important notes above)

M4 Recommended FIServer Firmware

(be sure to review important notes above)

HyperFlexRelease

40(4h)40(4h)40(2b)

40(4h)40(4h)40(2a)

40(4h)40(4h)40(1b)

40(4e)40(4e)40(1a)

HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments

Cisco HX Data Platform Release 4x based Deployments

Confirm the component firmware on the server meets the minimum versions listed in the following tables

HyperFlex Edge does not support Cisco IMC versions 40(4a) 40(4b) 40(4c) 40(4d) and 40(4e)Important

Table 2 HX220c M4 HXAF220c M4 Cluster

Recommended Firmware Version - HXDP 4x

(be sure to review important note(s) above)

Component

402d0BIOS

40(2h)Cisco Integrated Management Controller(CIMC)

Release Notes for Cisco HX Data Platform Release 4010

HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments

Recommended Firmware Version - HXDP 4x

(be sure to review important note(s) above)

Component

13000012SAS HBA

43(2b)VIC 1227

0103 (Toshiba)

CS07 (Intel 480)

SSD

CN03 (Seagate)

CK03 (Seagate 18)

CK03 (Seagate 24)

sF3Q (Samsung)

3F3Q (Samsung)

HDD

40(2h)

Download Software

Host Upgrade Utility (HUU) Download Link

Table 3 HX220c M5 HXAF220c M5 Cluster

Recommended Firmware Version - HXDP 4x

(be sure to review important notes above)

Component

C220M5404p00224200755BIOS

40(4i)Cisco Integrated Management Controller(CIMC)

09000008SAS HBA

43(3c)VIC 1387

50(3e)VIC 1457

G201CS01 (Intel)

0104 (Toshiba)

0107 (Toshiba)

SSD

2F3Q (Samsung)

3F3Q (Samsung)

NOA4 (Seagate)

CK03 (Seagate 18tb)

CK03 (Seagate 24)

HDD

Release Notes for Cisco HX Data Platform Release 4011

HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments

Recommended Firmware Version - HXDP 4x

(be sure to review important notes above)

Component

40(4k)

Download Software

Host Upgrade Utility (HUU) Download Link

HyperFlex Licensing

As of version 26(1a) HyperFlex supports VMware PAC licensing Existing VMware embedded licenses willcontinue to be supported

As of version 25(1a) HyperFlex uses a smart licensing mechanism to apply your licenses See the CiscoHyperFlex Systems Installation Guide for VMware ESXi for details and steps

VMware vSphere Licensing Requirements

How you purchase your vSphere license determines how your license applies to your HyperFlex system

bull If you purchased your vSphere license with HyperFlex

Each HyperFlex server either has the Enterprise or Enterprise Plus edition preinstalled at the factory

bull HX Nodes have OEM licenses preinstalled If you delete or overwrite thecontent of the boot drives after receiving the HX servers you also delete thefactory-installed licenses

bull OEM license keys is a newVMware vCenter 60 U1b feature Earlier versionsdo not support OEM licenses

bull All factory-installed HX nodes share the same OEM license key WithvSphere OEM keys the Usage count can exceed the Capacity value

bull When you add an HX host to vCenter through the Add Host wizard in theAssign license section select the OEM license

We obfuscate the actual vSphere OEM license key for example0N085-XXXXX-XXXXX-XXXXX-10LHH

bull Standard Essentials Plus and ROBO editions are not available preinstalledon HX servers

Note

bull If you did NOT purchase your vSphere license with HyperFlex

The HX nodes have a vSphere Foundation license preinstalled After initial setup you can apply thelicense to a supported version of vSphere

bull If you purchased a vSphere PAC license

Follow the instructions in your PAC license letter from VMware to add the license to your MY VMwareaccount then follow the instructions to add your HX host to vCenter and assign the PAC license

Release Notes for Cisco HX Data Platform Release 4012

HyperFlex Licensing

HX Data Platform Software Versions for HyperFlex Witness Node

Witness Node VersionHyperFlex Release

10840(2b)

10840(2a)

10440(1b)

10440(1a)

Software Requirements for VMware ESXi

The software requirements include verification that you are using compatible versions of Cisco HyperFlexSystems (HX) components and VMware vSphere VMware vCenter and VMware ESXi

bull Verify that all HX servers have a compatible version of vSphere preinstalled

bull Verify that the vCenter version is the same or later than the ESXi version

bull Verify that the vCenter and ESXi versions are compatible by consulting the VMware ProductInteroperability Matrix Newer vCenter versions may be used with older ESXi versions so long as bothESXi and vCenter are supported in the table below

bull Verify that you have a vCenter administrator account with root-level privileges and the associatedpassword

For VIC1457 there is no support for ESXi 60Note

The below table applies for all of the following VMware vSphere Editions Enterprise Enterprise PlusStandard Essentials Plus ROBO

VMware vCenter VersionsVMware ESXi VersionsHyperFlex Version

60 U3 65 U3 67 U360 U3 65 U3 67 U340(2b)1

60 U3 65 U3 67 U360 U3 65 U3 67 U340(2a)2

60 U3 65 U3 67 U260 U3 65 U3 67 U2340(1b)

60 U3 65 U2 67 U260 U3 65 U2 67 U2440(1a)

1 Cisco HyperFlex Release 40(2) is the last major HyperFlex release that will support vSphere 60 (ESXiand vCenter) due to those versions reaching end of VMware general support on March 12 2020

2 Cisco HyperFlex Release 40(2) is the last major HyperFlex release that will support vSphere 60 (ESXiand vCenter) due to those versions reaching end of VMware general support on March 12 2020

3 Use of 67 U2 is not recommended see software advisory for CSCvq06952 and CSCvp88515 for furtherdetails

4 Use of 67 U2 is not recommended see software advisory for CSCvq06952 and CSCvp88515 for furtherdetails

Release Notes for Cisco HX Data Platform Release 4013

HX Data Platform Software Versions for HyperFlex Witness Node

Software Requirements for Microsoft Hyper-V

The software requirements include verification that you are using compatible versions of Cisco HyperFlexSystems (HX) components and Microsoft Hyper-V (Hyper-V) components

HyperFlex Software versions

The HX componentsmdashCisco HX Data Platform Installer Cisco HX Data Platform and Cisco UCSfirmwaremdashare installed on different servers Verify that each component on each server used with and withinthe HX Storage Cluster are compatible

bull Cisco HyperFlex M5 Converged nodesmdash For Hybrid (Cisco HyperFlex HX240c M5 HX220c M5)and All Flash (Cisco HyperFlex HXAF240cM5 HXAF220cM5) verify that Cisco UCSManager 40(2b)is installed HX 40 (1a) does not support Hyper-V on the All NVMe (HXAF220C-M5SN) nodes Fordetailed information on installation requirements and steps see theCisco HyperFlex Systems InstallationGuide on Microsoft Hyper-V

Table 4 Supported HyperFlex Software versions for M5 Servers on Hyper-V

M5 Recommended Server FirmwareHyperFlex Release

40(4h)40(2b)

40(4g)40(2a)

40(4e)40(1b)

40(4e)40(1a)

Release Notes for Cisco HX Data Platform Release 4014

Software Requirements for Microsoft Hyper-V

Table 5 Supported Microsoft Software versions

VersionMicrosoft Component

Windows Server 2016 Datacenter Core amp DesktopExperience

ForWindows Server 2016Datacenter Coreand Desktop Experience the Windows2016 ISO image should be Update BuildRevision (UBR) 1884 at a minimum

Note

Windows Server 2019Datacenter-DesktopExperienceis supported starting from HXDP 401(a) onwards

For Windows Server 2019 DesktopExperience theWindows 2019 ISO imageshould be Update Build Revision (UBR)107 at a minimum

Note

Windows Server 2019 DatacenterndashCore is notsupported currently

Also note that the following are currently notsupported

OEM activated ISOs and Retail ISOs are notsupported

Earlier versions ofWindows Server such asWindows2012r2 are not supported

Non-English versions of the ISO are not supported

Windows Operating System (Windows OS)

A Windows 2012 or later domain and forestfunctionality level

Active Directory

Supported Microsoft License Editions

The Microsoft Windows Server version that is installed on one or more HyperFlex hosts must be licensed asper Microsoft licensing requirements listed on Microsoft Licensing

Browser Recommendations

Use one of the following browsers to run the listed HyperFlex components These browsers have been testedand approved Other browsers might work but full functionality has not been tested and confirmed

Table 6 Supported Browsers

HX ConnectHX Data PlatformInstaller

Cisco UCS ManagerCisco IntersightBrowser

11 or higher11 or higher11 or higherNAMicrosoft InternetExplorer

70 or higher70 or higher57 or higher62 or higherGoogle Chrome

Release Notes for Cisco HX Data Platform Release 4015

Browser Recommendations

HX ConnectHX Data PlatformInstaller

Cisco UCS ManagerCisco IntersightBrowser

60 or higher60 or higher45 or higher57 or higherMozilla Firefox

NANA9 or higher10 or higherApple Safari

NANA35 or higherNAOpera

Notes

bull Cisco HyperFlex Connect

The minimum recommended resolution is 1024 X 768

bull Cisco HX Data Platform Plug-in

The Cisco HX Data Platform Plug-in runs in vSphere For VMware Host Client System browserrequirements see the VMware documentation at httpswwwvmwarecomsupportpubs

The HX Data Platform Plug-in is not displayed in the vCenter HTML client You must use the vCenterflash client

bull Cisco UCS Manager

The browser must support the following

bull Java Runtime Environment 16 or later

bull Adobe Flash Player 10 or higher is required for some features

For the latest browser information about Cisco UCSManager refer to themost recent Cisco UCSManagerGetting Started Guide

Cisco HX Data Platform Storage Cluster SpecificationsCluster Limits

bull Cisco HX Data Platform supports up to 100 clusters managed per vCenter as per VMware configurationmaximums

bull Cisco HX Data Platform supports any number of clusters on a single FI domain Each HX convergednode must be directly connected to a dedicated FI port on fabric A and fabric B without the use of a FEXC-series compute only nodes must also directly connect to both FIs B-series compute only nodes willconnect through a chassis IO module to both fabrics In the end the number of physical ports on the FIwill dictate the maximum cluster size and maximum number of individual clusters supported in a UCSdomain

The following table provides Cisco HX Data Platform Storage Cluster Specifications

Release Notes for Cisco HX Data Platform Release 4016

Cisco HX Data Platform Storage Cluster Specifications

Stretched Cluster (Availableon ESX Only)

Microsoft Hyper-VVMware ESXiNode

AllNVMe -HXAF220c-M5SN

HX240c-M5LHX220cM5

HX220cAF M5

HX240cM5

HX240cAF M5

HX240c-M5LHX220cM5

HX220cAF M5

HX240cM5

HX240cAF M5

HXAF220c-M5SN

AllNVMe -HXAF220c-M5SN

ESXi 65or 67only NotsupportedwithESXi 60orHyper-V

HX220cM5 Edge

HXAF220cM5 Edge

HX220cM4 Edge

HXAF220cM4 Edge

HX240c-M5LHX220cM5

HX220cAF M5

HX240cM5

HX240cAF M5

HX220cM4

HX220cAF M4

HX240cM4

HX240cAF M4

HXServers

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

C220M5C240M5B200M4B200M5

C240M5C220M5B200M4B200M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

mdashB200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

Compute-OnlyUCSB-SeriesC-SeriesServers

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

Convergednodesonly

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

SupportedNodes

Release Notes for Cisco HX Data Platform Release 4017

Cisco HX Data Platform Storage Cluster Specifications

Stretched Cluster (Availableon ESX Only)

Microsoft Hyper-VVMware ESXiNode

NA(requiresEnterpriseHXDP-PLicense)

NA(requiresEnterpriseHXDP-PLicense)

NA(requiresEnterpriseHXDP-PLicense)

Convergednodes3-16(12TBHDDoption isnotsupportedforHyperV)

Compute-onlynodes0-16

Convergednodes3-16

Compute-onlynodes0-16

NA(requiresEnterpriseHXDP-PLicense)

M4Convergednodes 3

M5Convergednodes23or 4

RequiresHXDP-ELicense

Convergednodes3-16(12TBdrive islimited toamaximumof 8nodes)

Compute-onlynodes0-16(12TBdrive islimited toamaximumof 8nodes)

Convergednodes3-32(76TBdriveconfigsarelimited tomaximumof 12drivesnodeon AF240)

Compute-onlynodes0-32

HXDP-SLicensedNodeLimits

11 ratioofHXDP-StoComputeonlynodes

(MinmdashMax)

Convergednodes2-16 perSite

Compute-onlynodes0-21 perSite

(up tomaxclustersize)

Convergednodes2-8 perSite

Compute-onlynodes0-16 perSite

(up tomaxclustersize)

Convergednodes2-16 perSite

Compute-onlynodes0-21 perSite

(up tomaxclustersize)

Convergednodes3-16(12TBHDDoption isnotsupportedforHyperV)

Compute-onlynodes0-16

Convergednodes3-16

Compute-onlynodes0-16

Convergednodes3-32

ComputeOnlynodes0-32

(up tomaxclustersize)

Convergednodes 3

(requiresHXDP-ELicense)

Convergednodes3-16(12TBdrive islimited toamaximumof 8nodes)

Compute-onlynodes0-32(12TBdrive islimited toamaximumof 16nodes)

Convergednodes3-32(76TBdriveconfigsarelimited tomaximumof 12drivesnodeon AF240)

Compute-onlynodes0-32

(up tomaxclustersize)

HXDP-PLicensedNodeLimits

12 ratioofHXDP-PtoComputeonlynodes

(MinmdashMax)

24 perSite 64percluster

24 perSite 48percluster

32 perSite 64percluster

32326434864MaxClusterSize

Release Notes for Cisco HX Data Platform Release 4018

Cisco HX Data Platform Storage Cluster Specifications

Stretched Cluster (Availableon ESX Only)

Microsoft Hyper-VVMware ESXiNode

212121111121mdash2121MaxComputetoConvergedratio

NoExpansion

Requires Enterprise license

Requires uniform expansion across both sites

Guidelines and Limitationsbull Starting with release 40(2a) SCVM is no longer needed on a Compute node

bull HX REST API Access Token Management ndash Applications leveraging HX REST APIs should re-useaccess tokens when making API calls Once obtained using the AAA Obtain Access Token API accesstokens are valid for 18 days (1555200 seconds) In addition AAA enforces rate limiting on ObtainAccess Token API requests in a 15 minute window auth can be invoked (successfully) a maximum of5 times A user is allowed to create a maximum of 8 unrevoked tokens Subsequent call to auth willautomatically revoke the oldest issued token tomake room for the new token Amaximum of 16 unrevokedtokens can be present in system In order to prevent brute-force attacks after 10 consecutive failedauthentication attempts a user account is locked for a period of 120 secondsFor more information seeCisco HyperFlex Systems REST API Reference guide

HxConnect makes use of AAA Authentication REST API for login and the above rate limit applies toHxConnect also

Upgrade Guidelines

The following list is a highlight of critical criteria for performing an upgrade of your HyperFlex system

bull Unsupported HX Data Platform 17x 18x 20 21x 25x and 26x clustersmdashUsers from anyversion prior to 26(1a) must step through an intermediate version before upgrading to 40 or later releasesIf you need to upgrade your environment from a Cisco HyperFlex HX Data Platform software releasethat is past the last date of support to the latest suggested release on the Cisco Software Download sitesee Cisco HyperFlex Systems Upgrade Guide for Unsupported Cisco HXReleases For more informationsee the Software Advisory for CSCvq66867 WARNING Only Use HXDP 26(1e) Upgrade PackageWhen Upgrading From HXDP 18(1a)-18(1e)

bull Hypercheck Health Check UtilitymdashCisco recommends running this proactive health check utility onyour HyperFlex cluster prior to upgrade These checks provide early visibility into any areas that mayneed attention andwill help ensure a seamless upgrade experience For more information see the HyperFlexHealth amp Pre-Upgrade Check Tool TechNote for full instructions on how to install and run Hypercheck

bull vSphere 67 Software AdvisorymdashDo not upgrade to Cisco HX Data Platform Release 40(1a) whenrunning ESXi 67U1 EP06 (build 11675023) Do not upgrade to 67U1 EP06 (build 11675023) ifrunning Cisco HX Data Platform Release 40(1a) See the Software Advisory CSCvo56350 for furtherdetails

Release Notes for Cisco HX Data Platform Release 4019

Guidelines and Limitations

The software build version posted at release will override any other local versions

bull Required vCenter upgrademdashFor enhanced security Cisco HX Data Platform Release 35(1a) or laterrequires the use of TLS 12 Therefore vCenter must be upgraded to 60 U3f or later before upgradingto Cisco HX Data Platform Release 35 or later In addition ESXi should be upgraded as required tomeet HX Data Platform compatibility requirements

bull Minimum HXDP version for upgrademdashHXData Platform clusters running 26(1a) or later may upgradedirectly to 40 using the HX Connect UI

bull Cluster ReadinessmdashEnsure that the cluster is properly bootstrapped and the updated plug-in is loadedbefore proceeding Manual cluster bootstrap is required for upgrade from a pre-35 release

bull Cluster ReadinessmdashEnsure that the cluster is properly bootstrapped and the updated plug-in is loadedbefore proceeding Manual cluster bootstrap is required for HX releases earlier than 35(1a) For moreinformation see theManual Bootstrap Upgrade Process in the Cisco HyperFlex Systems Upgrade Guidefor VMware ESXi Release 40 Do not skip this cluster bootstrap step it is required for all upgradesuntil HX Release 35(1a) Auto bootstrap is supported beginning with HX release 35(1a) For moreinformation see the Auto Bootstrap Upgrade Process from HX Connect UI in the Cisco HyperFlexSystems Upgrade Guide for VMware ESXi Release 40

Manual bootstrap is not supported on Intersight clusters

bull Initiating Upgrade―Use the HX Connect UI or CLI stcli commands when upgrading from 25(1a)or later releases Use either the CLI stcli commands or the HX Data Platform Plug-in to the vSphereWeb Client when upgrading from a pre-25(1a) release The vCenter plug-in should not be used forupgrades starting with the 25(1a) release The vCenter plug-in should not be used for upgrades startingwith the 25(1a) release

If the current cluster version is at 35(1a) or above you do not need to use the stcli command Directupgrade to 40 is possible

bull Complete your Upgrade―The self-healing (or rebalance) capability is disabled temporarily during theupgrade window If the upgrade fails you should complete the upgrade as soon as possible

bull ESXi and HXDP Compatibility―Ensure your cluster is running a compatible version of ESXi basedon the running the HXData Platform version (see the section Software Requirements for VMware ESXi)ESXi compatibility is determined by the major version and update release of ESXi It is generally bestto upgrade HXDP and ESXi together if combining the upgrade operations into a single optimized rebootWhen running a split upgrade first upgrade the HX Data Platform then proceed to upgrade ESXi

bull If Upgrading to vSphere 65

bull Certain cluster functions such as native and scheduled snapshotsReadyClones and Enter or Exit HX Maintenance Mode will not operatefrom the time the upgrade is started until the HX Data Platform upgrade iscomplete

bull After upgrading ESXi using the offline zip bundle use the ESX ExitMaintenance Mode option The HX Exit Maintenance Mode option doesnot operate in the vSphere Web Client until the HX Data Platform upgradeis complete

Note

Release Notes for Cisco HX Data Platform Release 4020

Guidelines and Limitations

bull vSphere 60 UpgradesmdashUsers on vSphere 60 migrating to 65 upgrade components in the followingorder

1 Upgrade HX Data Platform and UCS firmware

2 Upgrade HX Data Platform and ESXi

3 Upgrade HX Data Platform only first then upgrade ESXi or UCS firmware or both

bull M4 Server Firmware UpgradesmdashUpgrade server firmware to ensure smooth operation and to correctknown issues Specifically newer SAS HBA firmware is available in this release and is recommendedfor long-term stability

bull Users are encouraged to upgrade to 31(3c) C-bundle or later whenever possible

bull Users running C-bundle versions before 31(2f) must upgrade server firmware by performing acombined upgrade of UCS server firmware (C-bundle) to 31(3c) or later and HX Data Platform to25 Do not split the upgrade into two separate operations

bull If the cluster is already on 31(2f) C-bundle or later you may perform an HX Data Platform onlyor combined upgrade as required

bull M5 Server Firmware UpgradesmdashM5 generation servers must run firmware version 32(2d) or later

bull Firmware Downgrades mdashDowngrading UCSM from the HX-installer is not supported

bull M4M5 Mixed DomainsmdashA mixed domain occurs when a new separate M5 cluster is installed underthe same UCS domain that contains existing M4 clusters Under these conditions orchestrated UCSserver firmware upgrade will not operate until Cisco HX Data Platform Release 26 or later is installedon the M4 clusters Therefore it is best practice to first upgrade UCS server firmware to the latest 31(3)or 32(2) patch release before adding a new M5 cluster to the existing UCS domain Additionally any17 HX Data Platform clusters must first be upgraded before adding any new M5 clusters to the samedomain

bull Maintenance WindowmdashIf upgrading both HX Data Platform and UCS firmware you can select eithera combined or split upgrade through the vSphere HX Data Platform Plug-in depending on the length ofthe maintenance window Cisco UCSManager infrastructure upgrade is only supported using AutoInstalland the direct server firmware upgrade should be performed only through the upgrade orchestrationframework provided by the HX Data Platform Plug-in

bull Unsupported Self-Encrypting Drives (SEDs)mdashIf adding or replacing self-encrypting drives (SEDs)that have been recently qualified in newer versions of HX Data Platform insert the new drives only afterupgrading HX Data Platform to a compatible version All drives must be SED drives mixing SED andnon-SED is not supported

bullbull Enabling External Host AccessmdashWith Cisco HX Data Platform Release 40(1a) port 445 on themanagement network is blocked for enhanced security Note that prior to 40 port 445 port was openenabling external host access If you are upgrading to 40(1a) from a prior release and would like tocontinue external host access you can use a utility to open select hosts For more information aboutenabling external host access see the ConfiguringHyperFlex Share to SCVMM section in the InstallationGuide for Microsoft Hyper-V

Release Notes for Cisco HX Data Platform Release 4021

Guidelines and Limitations

Mixed Cluster Expansion Guidelinesbull Expanding existing M4 cluster with M5 converged nodes is supported

bull Expanding existing M5 cluster with M4 converged nodes is not supported

bull Expanding existing mixed M4M5 cluster with M4 or M5 converged nodes is supported

bull Adding any supported compute-only nodes is permitted with all M4 M5 and mixed M4M5 clustersusing the HX Data Platform 26 or later Installer Some example combinations are listed here manyother combinations are possibleExample combinationsExpand mixed M4M5 cluster with compute-only B200 C220 C240 M4M5Expand M4 cluster with compute-only B200 M5 C220 M5 C240M5

bull Only expansion workflow is supported to create a mixed cluster Initial cluster creation with mixedM4M5 servers is not supported

bull All M5 servers must match the form factor (220240) type (HybridAF) security capability (Non-SEDonly) amp disk configuration (QTY capacity and non-SED) of the existing M4 servers

bull HX220-M5 will use a maximum of 6 capacity disks (2 disk slots to remain empty) when mixedwith HX220-M4

bull HX Edge SED LFF Hyper-V and Stretched Clusters do not support mixed M4 and M5 clusters

Security FixesThe following security issues are resolved

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-13132 CVE-2019-9924CSCvq6313840(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-11719 CVE-2019-11727CVE-2019-11729

CSCvq7124040(2a)

Release Notes for Cisco HX Data Platform Release 4022

Mixed Cluster Expansion Guidelines

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-1125CSCvr0633940(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2015-9383 CVE-2018-14498CVE-2018-20406 CVE-2018-20852CVE-2019-10160 CVE-2019-13117CVE-2019-13118 CVE-2019-14287CVE-2019-14973 CVE-2019-15903CVE-2019-17546 CVE-2019-18197CVE-2019-18218 CVE-2019-5010CVE-2019-5094 CVE-2019-5481CVE-2019-5482 CVE-2019-9636CVE-2019-9740 CVE-2019-9947CVE-2019-9948

CSCvs0609440(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2017-13168 CVE-2017-18174CVE-2017-18216 CVE-2018-10876CVE-2018-10877 CVE-2018-10878CVE-2018-10879 CVE-2018-10880CVE-2018-10881 CVE-2018-10882CVE-2018-10902 CVE-2018-10938CVE-2018-12233 CVE-2018-12896CVE-2018-13053 CVE-2018-13094CVE-2018-13096 CVE-2018-13405CVE-2018-13406 CVE-2018-14609CVE-2018-14617 CVE-2018-14633CVE-2018-14734 CVE-2018-15572CVE-2018-15594 CVE-2018-16276CVE-2018-16658 CVE-2018-17182CVE-2018-17972 CVE-2018-18021CVE-2018-18690 CVE-2018-18710CVE-2018-6554 CVE-2018-6555CVE-2018-9363

CSCvp6501940(2a)

This vulnerability is due to insufficientHTML iframe protection An attackercould exploit this vulnerability bydirecting a user to an attacker-controlledweb page that contains amaliciousHTMLiframe A successful exploit could allowthe attacker to conduct clickjacking orother clientside browser attacks

NACSCvo9851640(2a)

Release Notes for Cisco HX Data Platform Release 4023

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerability is due to insufficientauthentication for the statistics collectionservice An attacker could exploit thisvulnerability by sending properlyformatted data values to the statisticscollection service of an affected deviceA successful exploit could allow theattacker to cause the web interfacestatistics view to present invalid data tousers

NACSCvj9558440(2a)

The vulnerability is due to insufficientCSRF protections for the web UI on anaffected device An attacker could exploitthis vulnerability by persuading a user ofthe interface to follow a malicious linkA successful exploit could allow theattacker to perform arbitrary actions withthe privilege level of the affected user

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvp2434340(2a)

Vulnerabilities with a version of cJSONidentified by CVE-2019-11834 andCVE-2019-11835

For more information see the relatedCisco Security Advisory

CVE-2019-11834 CVE-2019-11835CSCvq1994940(2a)

HX ESXi image patches for MCEPSCand TAA vulnerabilities with VMwareESXi identified by CVE-2018-12207 andCVE-20190-11135

For more information see the relatedCisco Security Advisory

CVE-2018-12207 CVE-2019-11135CSCvr5439840(2a)

CP networking vulnerabilities affectingthe Linux kernel identified byCVE-2019-11477 CVE-2019-11478CVE-2019-11479

For more information see the relatedCisco Security Advisory

CVE-2019-11477 CVE-2019-11478CVE-2019-11479

CSCvq1954640(2a)

Qualification of Microsoft securitypatches for vulnerabilities withMicrosoftHyper-V hypervisor identified withCVE-2018-12207 CVE-2019-11135

For more information see the relatedCisco Security Advisory

CVE-2018-12207 CVE-2019-11135CSCvr5439940(2a)

Release Notes for Cisco HX Data Platform Release 4024

Security Fixes

DescriptionCVEDefect IDRelease

OpenSSH vulnerabilities identified byCVE-2019-6111

For more information see the relatedCisco Security Advisory

CVE-2016-10708 CVE-2018-15473CVE-2018-20685 CVE-2019-6109CVE-2019-6111

CSCvp7646340(2a)

Multiple vulnerabilities associated withOpenSSL and LibNSS

For more information see the relatedCisco Security Advisory

CVE-2016-2105 CVE-2016-2106CVE-2016-2107 CVE-2016-2109CVE-2016-2176 CVE-2016-2177CVE-2016-2178 CVE-2016-2179CVE-2016-2180 CVE-2016-2181CVE-2016-2182 CVE-2016-2183CVE-2016-6302 CVE-2016-6303CVE-2016-6304 CVE-2016-6305CVE-2016-6306 CVE-2016-6307CVE-2016-6308 CVE-2016-7055CVE-2016-8610 CVE-2017-3731CVE-2017-3732 CVE-2017-3735CVE-2017-3736 CVE-2017-3737CVE-2017-3738 CVE-2018-0495CVE-2018-0732 CVE-2018-0734CVE-2018-0735 CVE-2018-0737CVE-2018-0739 CVE-2018-12384CVE-2018-12404 CVE-2018-5407CVE-2019-1559

CSCvp6667940(2a)

libpng vulnerability identified byCVE-2016-10087

For more information see the relatedCisco Security Advisory

CVE-2016-10087CSCvp6655540(2a)

Vulnerabilities associated with libjpg andlibpng

For more information see the relatedCisco Security Advisory

CVE-2014-9092 CVE-2016-3616CVE-2017-15232 CVE-2018-11212CVE-2018-11213 CVE-2018-11214CVE-2018-1152 CVE-2018-13785

CSCvp3458640(2a)

Vulnerabilities associated with Glibidentified by CVE-2018-16428 andCVE-2018-16429

For more information see the relatedCisco Security Advisory

CVE-2018-16428 CVE-2018-16429CSCvp3120740(2a)

Multiple vulnerabilities associated withcurl expat python 27 python 35 36and 37 freetype and giflib

For more information see the relatedCisco Security Advisory

CVE-2019-15133 CVE-2019-15903CVE-2019-5010 CVE-2019-5481CVE-2019-5482 CVE-2019-9636CVE-2019-9740 CVE-2019-9947CVE-2019-9948

CSCvr3690340(2a)

Release Notes for Cisco HX Data Platform Release 4025

Security Fixes

DescriptionCVEDefect IDRelease

Multiple third party softwarevulnerabilities

For more information see the relatedCisco Security Advisory

CVE-2019-14379 CVE-2019-12384CVE-2019-14439

CSCvq9203240(2a)

Vulnerabilities associated with elfutils

For more information see the relatedCisco Security Advisory

CVE-2018-16062 CVE-2018-16402CVE-2018-16403 CVE-2018-18310CVE-2018-18520 CVE-2018-18521CVE-2019-7149 CVE-2019-7150CVE-2019-7665

CSCvq4325040(2a)

Vulnerabilities associated with vim

For more information see the relatedCisco Security Advisory

CVE-2017-5953 CVE-2019-12735CSCvq4323040(2a)

Vulnerabilities associated with sqlite3

For more information see the relatedCisco Security Advisory

CVE-2016-6153 CVE-2017-10989CVE-2017-13685 CVE-2017-2518CVE-2017-2519 CVE-2017-2520CVE-2018-20346 CVE-2018-20505CVE-2018-20506 CVE-2019-8457CVE-2019-9936 CVE-2019-9937

CSCvq4321340(2a)

Vulnerabilities associated with glib20

For more information see the relatedCisco Security Advisory

CVE-2018-20843CSCvq4320940(2a)

Vulnerabilities associated with expat

For more information see the relatedCisco Security Advisory

CVE-2018-20843CSCvq4320540(2a)

Vulnerabilities associated with bzip2

For more information see the relatedCisco Security Advisory

CVE-2016-3189 CVE-2019-12900CSCvq4319440(2a)

Mulitple Vulnerabilities associated wtihNodeJS

For more information see the relatedCisco Security Advisory

CVE-2018-12115 CVE-2018-0734CVE-2018-5407 CVE-2018-12120CVE-2018-12121 CVE-2018-12122CVE-2018-12123 CVE-2018-12116CVE-2019-5737 CVE-2019-5739

CSCvq1069440(2a)

Vulnerabilities associated with PalletsJinja strformat_map and identified byCVE-2019-10906 and CVE-2016-10745

For more information see the relatedCisco Security Advisory

CVE-2019-10906 CVE-2016-10745CSCvq1038840(2a)

Release Notes for Cisco HX Data Platform Release 4026

Security Fixes

DescriptionCVEDefect IDRelease

Multiple Security Vulnerabilities

For more information see the relatedCisco Security Advisory

CVE-2014-8501 CVE-2014-9939CVE-2015-9262 CVE-2016-10087CVE-2016-2226 CVE-2016-4487CVE-2016-4488 CVE-2016-4489CVE-2016-4490 CVE-2016-4491CVE-2016-4492 CVE-2016-4493CVE-2016-6131 CVE-2018-10963CVE-2018-13785 CVE-2018-17100CVE-2018-17101 CVE-2018-18557CVE-2018-18661 CVE-2018-7456CVE-2018-8905

CSCvq0917840(2a)

Vulnerabilities associated withlibseccomp

For more information see the relatedCisco Security Advisory

CVE-2019-9893CSCvq0756840(2a)

Vulnerabilities associated withgdk-pixbuf

For more information see the relatedCisco Security Advisory

CVE-2017-12447CSCvq0675540(2a)

Vulnerabilities associated withpython-crypto

For more information see the relatedCisco Security Advisory

CVE-2018-6594CSCvp9381740(2a)

Vulnerabilities associated with wget

For more information see the relatedCisco Security Advisory

CVE-2018-20483 CVE-2019-5953CSCvp8672140(2a)

Multiple vulnerabilities associated withPython Crypto

For more information see the relatedCisco Security Advisory

CVE-2018-6594CSCvp6674840(2a)

Vulnerabilities associated withsensible-utils

For more information see the relatedCisco Security Advisory

CVE-2017-17512CSCvp6673440(2a)

Multiple vulnerabilities associated withPython

For more information see the relatedCisco Security Advisory

CVE-2018-1000030CSCvp6668940(2a)

Release Notes for Cisco HX Data Platform Release 4027

Security Fixes

DescriptionCVEDefect IDRelease

Multiple patch vulnerabilities identifiedby CVE-2016-10713

For more information see the relatedCisco Security Advisory

CVE-2016-10713CVE-2018-1000156 CVE-2018-6951

CSCvp6667240(2a)

Vulnerabilities associatedwith Little CMSidentified by CVE-2016-10165

For more information see the relatedCisco Security Advisory

CVE-2016-10165 CVE-2018-16435CSCvp6666440(2a)

Vulnerabilities associated with TenableScan identified byCVE-2011-5325CVE-2019-11068CVE-2016-7076CVE-2017-1000368

For more information see the relatedCisco Security Advisory

CVE-2011-5325 CVE-2016-7076CVE-2017-1000368CVE-2019-11068

CSCvp6474640(2a)

Vulnerabilities associated with libxml2294 and earlier as used in XMLSec1223 and earlier and other products andidentified by CVE-2016-9318

For more information see the relatedCisco Security Advisory

CVE-2016-9318 CVE-2017-16932CVE-2017-18258 CVE-2018-14404CVE-2018-14567

CSCvp3479240(2a)

Vulnerabilities associated with LFTPRemote File Names Unauthorized AccessVulnerability and identified byCVE-2018-10916

For more information see the relatedCisco Security Advisory

CVE-2018-10916CSCvp2926640(2a)

Vulnerabilities associated withCryptographyDeprecationWarning signerand verifier have been deprecated

For more information see the relatedCisco Security Advisory

CVE-2018-7750CSCvo3409740(2a)

Multiple Vulnerabilities in python python27 35

For more information see the relatedCisco Security Advisory

CVE-2019-16056CSCvr3979340(2a)

Release Notes for Cisco HX Data Platform Release 4028

Security Fixes

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toexecute commands as the root user

The vulnerability is due to insufficientinput validation An attacker could exploitthis vulnerability by connecting to thecluster service manager and injectingcommands into the bound process Asuccessful exploit could allow the attackerto run commands on the affected host asthe root user

Cisco has released software updates thataddress this vulnerability There areworkarounds that address thisvulnerability

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvq2417640(1b)35(2g)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

The vulnerability is due to an unprotectedlistening interface An attacker couldexploit this vulnerability by connectingto the listening interface and injectingcommands to the bound process Anexploit could allow the attacker to runcommands on the affected host as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvj9560640(1b)35(2g)

Release Notes for Cisco HX Data Platform Release 4029

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerabilities associated with theJVM18U121memory leak found duringVC alarms (concurrent 40 calls usingREST API)

CVE-2017-10053 CVE-2017-10067CVE-2017-10074 CVE-2017-10078CVE-2017-10081 CVE-2017-10087CVE-2017-10089 CVE-2017-10090CVE-2017-10096 CVE-2017-10101CVE-2017-10102 CVE-2017-10107CVE-2017-10108 CVE-2017-10109CVE-2017-10110 CVE-2017-10111CVE-2017-10115 CVE-2017-10116CVE-2017-10118 CVE-2017-10135CVE-2017-10176 CVE-2017-10193CVE-2017-10198 CVE-2017-10243CVE-2017-10274 CVE-2017-10281CVE-2017-10285 CVE-2017-10295CVE-2017-10345 CVE-2017-10346CVE-2017-10347 CVE-2017-10348CVE-2017-10349 CVE-2017-10350CVE-2017-10355 CVE-2017-10356CVE-2017-10357 CVE-2017-10388CVE-2017-3509 CVE-2017-3511CVE-2017-3526 CVE-2017-3533CVE-2017-3539 CVE-2017-3544CVE-2018-2579 CVE-2018-2582CVE-2018-2588 CVE-2018-2599CVE-2018-2602 CVE-2018-2603CVE-2018-2618 CVE-2018-2629CVE-2018-2633 CVE-2018-2634CVE-2018-2637 CVE-2018-2641CVE-2018-2663 CVE-2018-2677CVE-2018-2678 CVE-2018-2783CVE-2018-2790 CVE-2018-2794CVE-2018-2795 CVE-2018-2796CVE-2018-2797 CVE-2018-2798CVE-2018-2799 CVE-2018-2800CVE-2018-2814 CVE-2018-2815CVE-2018-2952 CVE-2018-3136CVE-2018-3139 CVE-2018-3149CVE-2018-3150 CVE-2018-3169CVE-2018-3180 CVE-2018-3183CVE-2018-3214 CVE-2019-2422

CSCvo8899740(1b)

Tomcat and Nginx logs are not beingcollected in the support bundle generatedthrough HX Connect in Release 35

NACSCvm5803140(1b)

The vulnerabilities associated with thelibmspack software package versionincluded in Cisco HX Data Platform

CVE-2018-18584

CVE-2018-18585

CSCvn3511940(1a)

Release Notes for Cisco HX Data Platform Release 4030

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerabilities associated withFasterXML Jackson-Databind TimeValueField Denial of Service

CVE-2018-14719

CVE-2018-14720

CVE-2018-1000873

CVE-2018-14721

CVE-2018-19360

CVE-2018-19362

CVE-2018-19361

CVE-2018-14718

CSCvn8228240(1a)

The vulnerabilities associated with theOpenSSL Protocol software packageversion included in Cisco HX DataPlatform

CVE-2013-3587CSCvo0505440(1a)

The vulnerabilities associated withThird-Party Software Denial of Service

CVE-2018-16487

CVE-2018-19361

CSCvo2781840(1a)

The vulnerabilities associated withLinuxkernel for Ubuntu 1710

CVE-2018-1092

CVE-2018-7492

CVE-2018-8087

CVE-2018-1068

CVE-2018-8781

CSCvm5314935(2a)

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)25(1d)stcli services dns remove should remove the DNS serverinfo from the interface files

CSCvs70967

40(2b)40(1b)HX 40(1b) new deployment from Intersight productioncloud

User receives following error while enabling CSI(Kubernetes integration)

Failure occurred during volume_access_enable Errorwas Nonzero exit code 1

CSCvt10522

40(2b)40(2a)Receive the following alertevent in HX Connect HXController VM HOSTNAME one or more configuredDNS servers not responding

CSCvt13947

Release Notes for Cisco HX Data Platform Release 4031

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)40(2a)API UI CLI show few Drive slots Empty after upgradeto 402a

The drives are seen in the sysmtool lsscsi and stclicommand outputs

The cluster is healthy with no errors

CSCvt14914

40(2b)35(2d)After successfully changing the DNS server on the HXcontroller we still can see the original DNS entry thatwas added during the deployment

CSCvs69154

40(2b)35(2e)HyperFlex Connect and VCenter show APD alarms forHX and non HX datastores This implies there is an issuewith HyperFlex when there may not be

CSCvs30080

40(2b)40(2a)An issue where all the disks in the node get locked afterrebooting the node

We successfully unlocked using sed-clientsh -Ucommand but wanted to test with another reboot anddrives locked again

CSCvs74286

40(2b)40(2a)When running stcli license commands on HyperFlexthe following error is seen

rootSpringpathControllertmp stcli license show all

Show smart licensing failed

Smart Agent is not ready please wait a minute and tryagain

CSCvt13929

40(2b)40(2a)After upgrading to HX 40(2a) the first RoboEdge nodethat gets rebooted then ignores persistent drivesPIDHX-HD24TB10K4KN

Cluster will be in degraded state and have reducedcapacity

CSCvt20144

40(2b)35(2b)Zookeeper fails to start while Exhibitor is runninghowever echo srvr returns nothing

CSCvs21562

40(2b)40(2a)When performing an HyperFlex upgrade a validationwarning may occur due to the host not having EnterprisePlus or Enterprise hypervisor licensing

Upgrade Validation Warning

ESXi host esx1labtest should be configured withVMware Enterprise license for upgrade to continue

CSCvs91787

Release Notes for Cisco HX Data Platform Release 4032

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)35(2g)Cluster expansion fails at Config Installer stage whenthe root and admin password for the storage controller(SCVM) are different

CSCvs69317

40(2b)35(2g)Panic while upgrading ESXiCSCvt06983

40(2b)40(2a)A cluster node running may hang in the Linux kernelThis is classified as an oops and a deviation from theexpected behavior

CSCvs54285

40(2b)35(2g)Rebalance failing on 10+10 node HX 35(2g) stretchcluster

CSCvs69007

40(2b)35(2d)The cluster becomes inaccessible to the IOVisorCSCvr54687

40(2b)40(2a)Panic on the storage controllerCSCvt61297

40(2b)40(2a)The size of support bundle is very large when collectedwith storfs-support command

CSCvt63306

40(2b)40(2a)storfs crash associated withFileSystemUsageWarningEvent orFileSystemUsageAlertEvent due to high usage ofvarstv above 80

CSCvt72807

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

40(2a)35(2d)In normal scenarios HX upgrade gets pendingACK afterupdating service profile In this case UCS only upgradewas stalled because after updating service profile HXupgrade was not getting pending ACK and it was alwayswaiting state

CSCvo39912

40(2a)35(1a)When using motherBoardReplace-12 to clean ZK fromold stNodepNodes resulted in unmounted datastore andmaking the size of the Hyperflex Datastores 0 resultingin all VMs in the cluster going offline

CSCvq39471

40(2a)26(1e)Upgrading cluster and getting error

Failed upgrade validations Checking vCenterconfiguration Reason Upgrade validations failed DRSFault Insufficient resources to satisfy configured failover

CSCvm77294

Release Notes for Cisco HX Data Platform Release 4033

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)The cluster expand fails on a node with DR replicationconfigured

When a HX cluster which has DR replication configuredis expanded we see the installer UI pulling in thereplicationVLAN information instead of themanagementVLAN information Even if we change that informationto the correct mgmt VLAN id and name it it does notseem to work as the node is configured with the VLANof the replicationVLAN in ESXi This leads to the failureof the node add with the host unreachable error

CSCvo70650

40(2a)35(1a)Customer reported that server didnt complete thefirmware upgrade automatically one by one

The user manually needs to put the host into maintenancemode and they manually acknowledged the pendingrequests to finish the UCS firmware upgrade

Maintenance policy is set to default (User-ack) as perthe design

CSCvo91624

40(2a)35(1i)When the cluster is in Failed state and stcli noderemove is attempted the output shows successful thoughthe node failed to get removed from the cluster

CSCvo93017

40(2a)35(1a)HyperFlex cluster upgrade may fail during validationwith a DRS Validation failed error

CSCvp31021

40(2a)35(2b)HX cluster expansion will fail with the following errormessage

MAC address pool configuration failure150[ErrorDescription] bad address block range definitioncollision

CSCvp58318

40(2a)35(2b)Memory usage by carbon cacheCSCvq34873

40(2a)35(2g)Upgrading ESXi cluster fails with error Nodemaintenance mode failed

CSCvr03240

40(2a)35(2e)storfs process does not automatically start on Exit HXMaintenance Mode or other tasks which restart StorageController

CSCvr88978

40(2a)35(2a)Post install script fails with message

Failed to execute ipmitool on HX node

CSCvp10707

40(2a)40(1a)HyperFlex expansionworkflow doesnt pull VLANnamecorrectly

CSCvp46539

Release Notes for Cisco HX Data Platform Release 4034

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)During HX cluster deployment - Validate ClusterCreation phase - HX Installer might fail with the error

fromvarlogspringpathstDeploylog

CSCvq45087

40(2a)35(2b)HX Installer fails to login to the SOL for configuringESX hosts

CSCvq91380

40(2a)35(2d)If DRS is enabled with configuration parameters otherthen fully automated ie manual or partially automatedthan this will change to fully automated during clusterexpansion

CSCvr44222

40(2a)35(1a)One of the validation task name has a typo The taskname is Validating node for mixed of SED and non-SEDdisks

It should be Validating node for mix of SED andnon-SED disks

CSCvp82175

40(2a)30(1i)This combination of using smaller NVMe drives is notsupported in expand operation For example if the drivesare only 375GB and they cannot be added to an existingcluster with larger caching SSD

CSCvo12359

40(2a)35(1a)HyperFlex includes a version of OpenSSL that is affectedby the vulnerabilities identified by the followingCommon Vulnerability and Exposures

CVE-2018-0495

CSCvp66679

40(2a)30(1e)SDK invoke exception when taking a native snapshot ona VM with GPU pci passthrough attached

CSCvo00511

40(2a)30(1i)Node replacement script fails due to EAM errorCSCvo62867

40(2a)35(1a)While pairing a cluster (HX release gt= 35) and cluster(HX release lt 35) remote replication network test failson the cluster (HX release lt 35)

CSCvo79760

40(2a)30(1b)Fixed the issue in the latest support-workflow bundleThis should not be hit anymore

CSCvo87061

40(2a)35(2a)On a 3 node cluster the MBreplace script fails as thescript looks for HEALTHY cluster status

CSCvo87080

40(2a)35(2a)MbReplace script (tar) hangs when running on 352b or40

CSCvp12359

Release Notes for Cisco HX Data Platform Release 4035

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)HX replication cleanup failing with error NA 35(2a) -INFODR state is not clean

CSCvp63958

40(2a)40(1a)The check_stig_parameters API incorrectly shows thecompliance state of the cluster as not STIG compliant

CSCvp66277

40(2a)35(2c)The snapshot creation on CBT enabled VM fails witherror Failed in vmreparent vmkfstools clone1

CSCvq06952

40(2a)35(2a)HyperFlex GUI when setting up remote DR-replicationpartnerpeer should show remote replication partner TCPport reachability test results

This will provide quick upfront connectivity results thatfrequently takes customer a while to manually check andconfirm

If reachability tests fail an info window and HX alertshould trigger that list the destination peer IP and thespecific port(s) that could not be reache

CSCvo60587

40(2a)35(2b)If you have more than one IP pool configured for areplication network and local replication network testsare failing you might be running into this issue

CSCvr67130

40(2a)35(2a)Skip task option in MBreplace and Software redeployscripts The scripts should have the option to skip tasksas user inputs and not by modifying the json file

CSCvp05204

40(2a)40(1a)Error noticed on SCVM Console print_req_error IOerror dev fd0 sector 0 on HyperV

It doesnt come up on SSH session

CSCvq34357

40(2a)30(1d)The VMNetwork switch is already created as part of theinstall process

Ideally the ip address assignment should be done duringthe initial install process as well No post install stepsshould be required altrnatively it could be done as partof a post install script

CSCvk36222

40(2a)25(1c)storfs service impacted during HX upgrade from HX21(1) to 25(1c)

HX node which is not being upgraded storfs servicestopped causing outage

CSCvg53223

40(2a)35(1b)HX Stretch Cluster Witness VM reverts to DHCP onreboot

CSCvp37536

40(2a)35(2b)HyperFlex Stretch Cluster Witness does not getprogrammed with an NTP server

CSCvp46578

Release Notes for Cisco HX Data Platform Release 4036

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2b)HyperFlex cluster does not heal after maintenance onnodeserver

CSCvr18528

40(2a)40(2a)High percentage of packet loss on the eth1 data interface9K MTU

CSCvr97089

40(2a)35(2f)The HyperFlex cluster healing state is stuck at 87 andis unable to progress

CSCvr16760

40(2a)40(2a)In some circumstances two replications can occur forthe same VM

From the GUI the replication status for this VM willshow stuck in an In Progress state but there are nojobs stuck in replication layer Subsequent Replicationsshould succeed

CSCvq22898

40(2a)35(2a)After an unexpected power outage the hyperflex clusteris unable to come up correctly

All processes are running time is synced and all nodescan vmkping each other as required

CSCvp13990

40(2a)40(1a)When specifying IP address instead of node ID stclinode maintenanceMode commands fail

CSCvp33657

40(2a)40(1a)2 node edge cluster if experiencing Intersightarbitratorconnectivity issues should NOT allowHXMaintenancemode to initiate as this will bring the cluster down

CSCvr31573

40(2a)35(2c)Old files in varsupportZKTxnlog are not purged withthe daily zklog-cleanup cron job

CSCvr89066

40(2a)35(2a)When a node is in MaintenanceMode any disk removalor replacement will be reflected in UI only after the nodeis brought back frommaintenancemode This is becausestorfs is not running on the node in maintenance andwill not be able to detect disk activities until it is broughtout of Maintenance Mode

CSCvo86431

40(2a)35(2c)Cluster might go offline and stop serving IO requestsrootcvm~ stcli cluster storage-summary--detailGet cluster storage summary failedjavanetConnectException Connection refused19216814210010207

CSCvr01645

40(2a)35(2d)A stretch cluster deployment fails with a messageFormatting nodes Node in Use on the Hyperflex installerand crmZoneType value is seen as 1 in storfscfg oroptspringpathconfigstretchtunes

CSCvq80340

Release Notes for Cisco HX Data Platform Release 4037

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)30(1c)During upgrade we upload the upgrade packagedownloaded from Ciscocom Upgrade packages are offile type tgz hxconnect acceptsstorfs-packages-401a-33028tar file a different type ofcompression and allows upgrade to start and fail whenit cannot decompress the file for bootstrap and scvmclientupgrade on ESXi

CSCvq63888

40(2a)40(2a)While attempting to install hyperflex via intersight thefollowing error is seen

Failed in Task Add hosts to vCenter Cluster with ErrorTry adding hosts manually to vCenter and retry Failedto get addressFailed to host by namelookupHostnamecompanycom on 000053 read udp12700145430-gt12700153 read connection refused

CSCvr52098

40(2a)35(2a)CoreAPI call may time out for cluster management APIwhile doing cluster create validation

CSCvo92952

40(2a)35(2d)Validation fails with Mixed mode expansion checkdue to Enhanced vMotion Compatibility Incompatibilitywhen the ESXi version is different

CSCvq95460

40(2a)30(1i)This failure is seen in installer when custom workflowis used hypervisor configuration + deploy (clean diskpartition=no) at Installing software packages on storagecontroller VM

CSCvr66309

40(2a)35(2d)The HX Installer apparently replaced a comma bysemicolon on the VLAN ID range while exporting theconfiguration to a JSON file

CSCvq93831

40(2a)40(1a)When using unsupported browsers HX Connect userswill not be able to login These users see an errorindicating unauthorized user

CSCvp22693

Release Notes for Cisco HX Data Platform Release 4038

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)40(1a)A vulnerability in the web-based management interfaceof Cisco HyperFlex Software could allow anunauthenticated remote attacker to conduct a cross-siterequest forgery (CSRF) attack on an affected system

The vulnerability is due to insufficient CSRF protectionsfor the web UI on an affected device An attacker couldexploit this vulnerability by persuading a user of theinterface to follow a malicious link A successful exploitcould allow the attacker to perform arbitrary actions withthe privilege level of the affected user

Cisco has released software updates that address thisvulnerability There are no workarounds that address thisvulnerability

This advisory is available at the following link

httpstoolsciscocomsecuritycentercontentCiscoSecurityAdvisorycisco-sa-20190807-hypflex-csrf

CSCvp24343

40(2a)35(2d)Add message to HX Connect Progress flow to notAcknowledge Pending Activities and reboot the serverson UCSM

HX Connect is doing a controlled rolling server upgradein the background

CSCvq22844

40(2a)40(1b)Compute Node expand operation fails due to incorrectcluster name

CSCvr43786

40(2a)26(1e)RF-2 cluster shutdown during data resynchronizationafter a node failure (non storfs) followed by multipledisk read failures and a HardBlacklist of a disk

CSCvp41241

40(2a)40(1a)HyperFlex Edge nodes do not properly set the ESXihostname during deployment

CSCvo13143

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4039

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4040

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1b) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401b-33133p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4041

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4042

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1b)40(1a)While running the HyperFlex installer with WindowsServer Hyper-V cluster creation process fails with theerror Failure occurred during ClusterCreation process Unable to post thecontent to the downstream This symptomis encountered while deploying HyperFlex cluster withHyperFlex nodes configuredordered with Cisco VIC1457 MLOM (PID HX-MLOM-C25Q-04) andWindows Server Datacenter or Core with Hyper-Vversion 2016 or 2019

CSCvp64140

Release Notes for Cisco HX Data Platform Release 4043

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)35(2b)Adding Micron 5200 drive to a cluster fails to increasecluster capacity

CSCvo69067

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4044

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4045

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1a) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401a-33028p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4046

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4047

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1a)30(1d)Cluster instability when disks of different sector sizeplaced in HX node

CSCvk17250

40(1a)

35(2c)

35(1a)When logged in using a local HX user account insteadof a Virtual Center account an error message appearsintermittently indicating Virtual Centerunreachable or Resource informationcannot be updated when VC is reachable

CSCvo36198

40(1a)

35(2b)

30(1d)HXDP does not work with EMC RecoverPoint - needsto support VMware API (FSS-Readdir)

CSCvk38003

Release Notes for Cisco HX Data Platform Release 4048

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)

35(2a)

30(1i)

25(1c)

25(1d)

Zookeeper (Exhibitor) process on HyperFlex StorageController VMs may fail to respond to requests whenvarzookeeper has low or no free space

CSCvm90352

40(1a)26(1c)Use Asynchronous consolidation for HX SnapshotsCSCvn02151

40(1a)35(2b)HX Quiesced Snapshot for Backup VendorsCSCvo90713

40(1a)35(2a)The cluster creationcluster expansion workflow stopswith the following error message at the validation step

FIRMWARE-Check UCSC-SAS-M5HD

FIRMWARE-Check UCSC-SAS-M5HD Required00000029000000320000003500000050 Found00000058

Action Needed Update the Controller Firmware toRequired Version

CSCvn17787

35(2b)30(1d)Kernel migration fails when a local datastore is searchedfor in the ESXi host

CSCvn73127

35(2a)26(1b)A node shuts down when two disks are replaced (acaching disk and another capacity disk) where thecapacity disk is inserted first and then the caching diskis inserted

CSCvk46364

Hyper-V

40(1a)35(2a)Cluster expansion may fail with an error code 500 -operation timed out

CSCvn28721

35(2b)35(2a)During upgrade remove the VLAN on team created foruser vSwitch During fresh installation only one VLANtag is set to the vSwitch and team although multipleVLANs were entered

CSCvn54300

35(2b)35(2a)While upgrading a Hyper-V cluster on account of a rarerace condition between the stUpgradeService andZookeeper servers the upgrade orchestration throws anupgrade validation error and the upgrade process isaborted

CSCvn60486

Release Notes for Cisco HX Data Platform Release 4049

Resolved Caveats in Release 40(1a)

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Startrestart EAM service invCenter

When EAM service is not runningvCenter registration is failing

CSCvt55712

35(2a)Contact Cisco TAC if you believeyou are seeing similar conditions

HyperFlex Stretch Cluster saw poorapplication performance due to writelatency cluster remained unhealthy andrebalance was stuck

This defect is currently being used forinvestigative purposes

CSCvt45344

35(2g)Remove log files fromvarlogspringpath folder when thelog files are not getting rotatedStop the relevant service removelarge log files fromvarlogspringpath folder andrestart the service

varstv folder in Controller VM maybecome full

CSCvt36374

40(2b)Restarting the stMgr will initializeit again

In a Cluster configured with DR networkStMgr may not get initialized and stuckin a deadlock while enabling the IPTablerules (can be verified from the stMgrlog)

CSCvt89709

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Click recover for the selectedVM again under the standalonemode This will move the VM torecovered state

When the user recovers a VM protectedin a group it along with other VMs in thegroup move to recovering state instandalonemode The selectedVM shouldmove to recovered state

CSCvs75553

35(2e)Clean up the fail over cluster andthen recreate the fail over clusterNo need to touch the HX storagecluster

Hyper-V When replicated DC was usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2d)40(1a)

Keep cluster utilization as low aspossible To avoid read-only statekeep below 92

HyperFlex clusters storage utilization isabove 92 usable capacity

CSCvr20922

Release Notes for Cisco HX Data Platform Release 4050

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1b)NAAll the disks in the node got locked afterrebooting the node

We successfully unlocked usingsed-clientsh -U command but wantedto test with another reboot and driveslocked again

CSCvs74286

35(2g)To recover the VM copy over thedata disks and attach them to thenew VM

After enable encryption HX connectencryption status showing Caution butall of the disks enable encryptionsuccessful

-Self Encrypting Drives Service isrunning on all of the nodes

-There is no error message of encryption

- USB0 interface is up

-All SED disks showing supported 1enabled 1 locked 0

CSCvs93245

40(1b)Enable kernelpanic_on_oops as apersistent configuration This willcause the node to panic and rebootimmediately

A cluster node running HX release40(1b) may hang in the Linux kernelThis is classified as an oops and adeviation from the expected behavior

CSCvs54285

Release Notes for Cisco HX Data Platform Release 4051

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Enabling K8iscsi stack in Hx clusterCSCvs41324

Release Notes for Cisco HX Data Platform Release 4052

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Steps to enable K8 on HX cluster

Perform following steps on ALLthe controller VMs

1 Updateetcinitscvmclientconf toenable tune iscsiEnable totrue

sed -iesiscsiEnable=falseiscsiEnable=trueetcinitscvmclientconf

2 Run following initctlcommand to reloadconfiguration

initctl reload-configuration

3 Restart scvmclient process

stop scvmclient startscvmclient

4 Run following initctlcommand to mount SYSTEMDS

initctl emit --no-waitsystem-datastore-created

5 Verify that iscsiEnable tuneis set to true using thefollowing command

ps -eaf | grep scvmclient |grep -v grep

root 6241 1 1 Dec06 014007optspringpathstorfs-corescvmclient-T logEnabled=false -TlogSyslogEnabled=true -TlogEchoToScreen=false -TstatLoggingToFile=false -TstatLoggingToSyslog=true-TlogDir=varlogspringpath-TnfsBackendServerList=1010748100-T iscsiEnable=true -TiscsiUseAsync=true -T

Release Notes for Cisco HX Data Platform Release 4053

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

iscsiConfigFilePath=nfsSYSTEMistgtconfps -eaf | grep scvmclient |grep -v grep

35(2e)Following procedure can be usedto check NFS Queue Depth andincrease if needed

rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth256 lt- Low value[rootHXESXI1] vsish -e setvmkModulesnfsclientmnt[DATASTORE]propertiesmaxQDepth 1024[rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth1024 lt- Optimalvalue

This requires an ESXi host rebootto take effect

Please place the node in HyperflexMaintenanceMode and gracefullyreboot the node for the changes tobe applied

HyperFlex Datastore NFS Queue Depthshows as 256 which can lead toperformance (including latency) issues

CSCvr83056

40(1b)This is cosmetic The error will goaway once the cluster returns tohealthy status

When a single node is offline in clusterstcli cluster storage-summary shows twonodes unavailable rootSCVM~ stclicluster storage-summary

CSCvs47735

35(2g)Modify both the root and adminpassword for the SCVMs to be thesame

stcli security password set -uadmin

Cluster expansion fails at Config Installerstage when the root and admin passwordfor the storage controller (SCVM) aredifferent

CSCvs69317

35(2d)NAAfter successfully changing the DNSserver on the HX controller we still cansee the original DNS entry that was addedduring the deployment

CSCvs69154

Release Notes for Cisco HX Data Platform Release 4054

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Customers are strongly encouragedto work with TAC in order toidentify and remediate this issue

1 Change getcluster from 2m to8m inoptspringpathstorfs-mgmtstMgr-10confapplicationconfon all nodes

2 Restart stMgr on all nodes

You may see this error message after afailed upgrade or other task such asattempting to enter a node into HXmaintenance mode

CSCvs53555

40(2a)Run the following command oneach storage controller VM as root

grep -i monitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp sed -iesmonitor_dns_serverstruemonitor_dns_serversfalseoptspringpathhx-diag-toolswatchdog_configjsonampamp grep -imonitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp restart watchdog

Receive the following alertevent in HXConnect HX Controller VMHOSTNAME one or more configuredDNS servers not responding

CSCvt13947

35(2b)Delete any empty (Size 0) log filesunder varlogzookeeperversion-2ORvarlogzookeeperstandaloneversion-2

Zookeeper fails to start while Exhibitor isrunning however echo srvr returnsnothing

CSCvs21562

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

40(4e)1 De-commission the server

2 Power drain the server -REMOVE BOTH POWERCORDS ON THE BACK OFTHE SERVER FOR 10SECONDS THENREINSERTPOWERCORDS

3 Re-commission the server

M2 boot disk is missing from serverinventory after upgrade to server firmware40(4e) As a result server fails to boot toOS installed in the M2 disk The issuepersists after re-acknowledgement as wellas de-commission andre-acknowledgement of the server

CSCvs02466

Release Notes for Cisco HX Data Platform Release 4055

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Re-start the flusher manuallyMany disks taken offline by a fault ormanually resulting in ENOSPC oncluster

CSCvq32721

35(2e)Clean up fail over cluster and thenrecreate the fail over cluster Thereis no need to touch the HX storagecluster

[Hyper-V]When replicated DCwas usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2b)Manually install upgradescvmclient VIB andmake sure thatit matches with HXDP version

Upgrade from release HX 35(2a) to35(2x) 4X does not upgrade scvmclientRecovery point may not work

CSCvq54992

Management

30(1b)To recover the VM copy over thedata disks and attach them to thenew VM

[HyperV]VMshows up onmultiple nodesCSCvj22992

35(1a)1 Do not change MTU at ESXlevel on running cluster

2 Reset back to the original value

ChangingMTU from 9000 to 1500 on oneESX node causes storfs process to restarton all nodes

CSCvm99150

Release Notes for Cisco HX Data Platform Release 4056

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)After the installation (for newdeployments) or after an upgradeto 401a (for existingdeployments) perform thefollowing steps on all ControllerVMs

1 Edit the following tune files (onall Controller VMs)

optspringpathconfiglfftunes

optspringpathconfigvsi_16tbtunes

2 SetcleanerEnableSegSummaryCleaning

to false

3 After editing the above tunefiles

a SSH in to all the ControllerVMs

b Run storfstool -- -Z

c Run the below command andcheck the values Tune valueshould be true

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=true

d Type below commands andapplied the tunes changesdynamically

echo false gttmpstprocfssystemtunecleanerEnableSegSummaryCleaning

e Verify that tune values arechanged Run below commandValue should be false

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=false

f umount tmpstprocfs

A cluster with 8TB or 12TB disk drivesmay experience IO stalls for severalminutes after another node fails in thecluster

CSCvp23718

Release Notes for Cisco HX Data Platform Release 4057

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)1 Update the conf file manuallyto change to the link local addressto

2 Restart scvmclient on thecontroller and ESX

HX 35(2b) FlexVol to HX 40 CSIupgrade does not work FlexVol continuesto work

CSCvp26319

35(2a)Disable rebalance until node healsthen reenable rebalance

7-node cluster healing very slowlybecause rebalance change AMS reqs areblocking healing process

CSCvp62512

35(2a)NAWith moderate cluster storage capacityusage under site failover cluster clustermay experience an all paths (APD) downcondition

CSCvq49412

40(1a)NAThis product includes Third-partySoftware that is affected by thevulnerabilities identified by the followingCommon Vulnerability and Exposures(CVE) IDs

CVE-2016-0762 CVE-2016-6797CVE-2016-6816 CVE-2016-8735CVE-2017-5647 CVE-2017-12615CVE-2017-12616 CVE-2017-12617CVE-2017-7674 CVE-2018-1304CVE-2018-8014 CVE-2018-1336CVE-2018-8034 CVE-2018-11784CVE-2019-0232

CSCvp36364

35(2d)NAstcli cluster info command needs toprovide UCSM VIP address Currently itshows ucsm-hostcom

CSCvq11456

35(2a)No workarounds currently Thestale entries do not affect the dayto day operations of HyperFlex

HyperFlex upgrade validation failedbecause cluster has extraneous stNodes ininternal database

CSCvq32530

35(2a)NAWhen not in maintenancemode all drivesappear in HX Connect and get updated asexpected

When in maintenance mode extra drivesare shown in HX Connect with no slotnumbers

CSCvq42378

35(2b)Contact TAC to cleanup duplicatenode

Nodes duplicated in HX Connect aftercluster recreate

CSCvq61825

Release Notes for Cisco HX Data Platform Release 4058

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)NACurrently HyperFlex installs on Hyper-Vdo not contain the stcli security whitelistcommandset

CSCvq66245

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

Release Notes for Cisco HX Data Platform Release 4059

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)[Hyper-V] Converged node expansionfails for incoming node with VIC 14xx

CSCvq39523

Release Notes for Cisco HX Data Platform Release 4060

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

1 Set Jumbo frame inWindowsHypervisor for incomingnode

Get-NetAdapter storage-data-a|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

Get-NetAdapter storage-data-b|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

2 Verify all interfaces are setcorrectly to support Jumbo frames

Get-NetAdapter storage |Get-NetAdapterAdvancedProperty| RegistryKeyword -matchjumbo | ft -auto

You should receive the followingmessage

Name DisplayName DisplayValue

RegistryKeyword RegistryValue

---- ----------- ------------

--------------- -------------

vswitch-hx-storage-data Jumbo

Packet 9014 Bytes

JumboPacket 9014

storage-data-b Jumbo Packet

Bytes 9014 JumboPacket

9014

storage-data-a Jumbo Packet

Bytes 9014 JumboPacket

9014

3 Reboot only the incoming nodeor the node being expanded

4 Retry Cluster Expansion in

Release Notes for Cisco HX Data Platform Release 4061

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

DescriptionDateRelease

bull Updated VMware vCenterVersions for 40(1a)

bull Updated SupportedMicrosoftSoftware versions

May 8 201940(1a)

bull Added CSCvo36198 andCSCvk38003 to the list ofResolved Caveats

bull Added CSCvp21417 to the listof Open Caveats

May 3 201940(1a)

Created release notes for Cisco HXData Platform Software Release40(1a)

April 29 201940(1a)

New FeaturesCisco HX Data Platform Release 40 provides the following features These features including the InvisibleCloud Witness for HyperFlex Edge clusters are supported both on the Intersight Virtual Appliance and onIntersightcom

New Features in Release 40(2b)

bull 76TB SSD data drivemdashSupport on HX Edge configurations

bull All NVMe and All Flash limits increasemdashLimits increased on All NVMe (1TB 4TB and 8TB) andAll Flash (76TB SSD)

bull Cluster Scale Limits increasemdashSupport for maximum scale limit increase on a cluster See Cisco HXData Platform Storage Cluster Specifications on page 16

bull HW Offload optionmdashSupport for Hardware Offload option with Stretched cluster configurations

bull Cisco Overlay Transport Virtualization for Stretched ClustermdashSupport for OTV as an overlay forStretched Cluster

New Features in Release 40(2a)

bull Cisco UCS Manager 41(1c)mdashSupport for UCSM hardware and software enhancements and featuresfrom Cisco UCSM 41(1c) release

bull Boost ModemdashThis release introduces Boost Mode for the following configurations All NVMe AllFlash C240 All Flash C220 and Hypervisor ESX Boost Mode allows the Cisco HyperFlex cluster todeliver higher IOPs by increasing the storage controller VMCPU resources by 4 vCPU For configurationinformation see the Cisco HyperFlex Data Platform Administration Guide

bull Cisco HyperFlex HTML plug-in for VMWare vCentermdashEnables virtualization administrator tomanage and monitor the Cisco HyperFlex physical infrastructure by cross launching HyperFlex Connectfrom the vSphere Client UI and perform management actions in the HyperFlex Connect UI

Release Notes for Cisco HX Data Platform Release 405

New Features

bull 25GE networking for HX EdgemdashSupport for 25GE networking for HX Edge

bull All NVMe with Stretched ClustermdashSupport for All NVMe with Stretched Cluster (ESX only)

bull Cluster Upgrade Eligibility TestmdashThis release adds the capability to perform a pre-upgrade test whichchecks for cluster readiness before upgrading Example checks include validating cluster state rebalancestatus controllerVM Free Space ESXi version and much more The Elibility test is intended to helpavoid unexpected problems that may arise during the upgrade process It is highly recommended to runthe test before performing the Hyperflex upgrade

bull Registering Smart Software LicensingmdashThis release adds support for software that allows easy trackingof the status of license and software usage trends and simplifies the three core licensing functionsPurchasing Management and Reporting

bull Dynamic self-signed certificate generation enhancementsmdashThis release adds support for Self-signedSSL certificates on the Controller VMs which were static in prior releases The static certificates arereplaced with dynamically generated self-signed certificates upon upgrading to HXDP 40(2a) so thatthe certificates are unique per cluster The new clusters installed with HXDP 40(2a) have dynamicallygenerated self-signed certificates

bull Test Upgrade EligibilitymdashThis release adds support for testing your cluster readiness and infrastructurecompatibility for an upgrade For more information see the Test Upgrade Eligibility sections in theCiscoHyperFlex SystemsUpgrade Guide for VMware ESXi Release 40 or the CiscoHyperFlex UpgradeGuide for Microsoft Hyper-V Release 40

Disaster Recovery

bull Recovery Settings ConfigurationmdashThis release supports configuration of recovery settings to defineglobal recovery parameters and mapping for resources across recovery sites These parameters are usedduring recovery test recovery and migrate operations

bull HyperFlex DR Powershell RunbookmdashPowerShell runbook functionality is extended to support therecovery configuration settings in the runbooks for all recovery scenarios New-HXrunbook cmdlet cannow be used to generate runbook for a single or multiple protection groups In addition two new cmdletsWait-HXTask and Get-HXTaskStatus are introduced

bull Protected Virtual Machine ScalabilitymdashThis release adds support for 1500 VMs across both clustersand 750 VMs per cluster in a bi-direction or any split between the two clusters without exceeding thelimit of 1500 VMs For more information see Cisco HyperFlex Data Platform Administration Guide

bull System Management REST API enhancementsmdashPause data replication actions briefly using RESTAPI to explicitly inform users on the current status of replication actions

Cisco HyperFlex with Data Platform for Hyper-V

bull Cluster-Aware Updating (CAU)mdashThis is an automated feature that allows you to perform updates onwindows servers in a failover cluster with little or no loss in availability during the upgrade process

Support for new drivesmdashNew drives are qualified for the 40(2a) release The new drives include newcapacity points and new cache drive options Several of the new drives are alternate drives to already qualifiedexisting drives in function which are qualified in 40(2a) These drives are functionally compatible with theexisting drives and are available as alternates in case of lack of availability of existing drives For expansionof existing clusters or general information about interoperability of different drives see Cisco HyperFlexDrive Compatibility

Release Notes for Cisco HX Data Platform Release 406

New Features

NVMe Caching SSDs slot information is unavailable from HX-Connect for all AF server PIDs except forthe All-NVMe server PIDs Please refer to UCSM management console for NVMe SSD slot information

Note

Applicable PlatformsDrive PIDDrive Function

All existing HX M5 serversHX-M2-960GBAlternate boot drive

All existing HX M5 servers except AllNVMe

HX-SD480G6I1X-EVAlternate system (or housekeeping) drive

All existing HX M5 servers except AllNVMe

HX-SD480GM1X-EVAlternate system (or housekeeping) drive

The following HX M5 serversHXAF220C-M5SX HXAF240C-M5SXHXAF-E-220M5SX

HX-SD800G123X-EP800G 12G SAS Cache drive option forAll Flash

All NVMe HXAF220C-M5SNHX-NVME2H-I1000All NVMe 1TB Capacity drive

All NVMe HXAF220C-M5SNHX-NVME2H-I4000All NVMe 4TB Capacity drive

All NVMe HXAF220C-M5SNHX-NVMEHW-I8000New high density All NVMe 8TBCapacity drive

All Flash Configuration ndash namelyHXAF220C-M5SX HXAF240C-M5SXHXAF-E-220M5SX

ESX support only

HX-SD76T61X-EVNew high density All Flash 76TBCapacity drive

Maximum number of drives per node is12 drives on HX AF C240

HX240C-M5LHX-HD8T7K4KANAlternate drive for 8TB LFF capacity

HXAF220C-M5SX HXAF240C-M5SXHX-SD38T2HTNK9New 38TB FIPS compliant SED SSDdata drives

HXAF220C-M5SX HXAF240C-M5SXHX-SD960G2HTNK9New 960GFIPS compliant SEDSSD datadrives

New Features in Release 40(1b)

bull Support for Second Generation Intelreg Xeonreg Scalable Processor RefreshmdashThis release includessupport for the Second Generation Intelreg Xeonreg Scalable processor refresh (formerly Cascade Lake)

New Features in Release 40(1a)

The following new features are in Release 40(1a)

bull Ultra-Light HyperFlex Edge ClustersmdashThis release introduces support for two-node HyperFlex Edgeclusters enabling HyperFlex to run in environments requiring a small footprint Cisco Intersight providescomprehensive lifecycle management and includes remote cloud-based installation centralized upgradesand invisible witnessing Both 1GE and 10GE networking topology options are available

Release Notes for Cisco HX Data Platform Release 407

New Features

bull Scaled-Up HyperFlex Edge ClustersmdashThis release adds support for four node HyperFlex Edge clustersenabling a full range of size options for remote and branch offices Size the branch office environmentsto suit current needs with a two three or four node HyperFlex Edge cluster Cisco Intersight providesfull-lifecycle management and 1GE and 10GE networking options are available

bull Cisco Intersight Invisible Cloud WitnessmdashFor two node clusters this feature eliminates the need forwitness VMs the infrastructure to run those VMs and the management overhead to deploy scale andpatch witnessing software The Invisible Cloud Witness is responsible for maintaining cluster HA in theevent of failure scenarios This feature is included at no additional cost and is automatically deployedand managed by Cisco Intersight

bull Cloud-delivered HyperFlex Edge UpgradesmdashPowered by Cisco Intersight this feature adds supportfor multi-site orchestrated remote upgrades of the HyperFlex Data Platform This feature will be enabledwith the next 40 patch release and will allowHyperFlex Edge clusters deployed via Intersight to performorchestrated upgrades across one or many sites in parallel

bull All-NVMe HyperFlexmdashStarting with this release a new high-end performance node powered by allNVMe drives is available for HyperFlex clusters Co-engineered with Intel to support Intel VMD forhot-plug and surprise removal this offering represents an industry first an enterprise-ready and fullyvalidated all-NVMe HCI appliance The all-NVMe offering is available in the 220 form factor (1RU)and is powered by Intel Optane cache drives for maximum performance and highest endurance

bull VMware Site Recovery Manager (SRM) IntegrationmdashThis release brings support for a Cisco developedStorage Replication Adapter (SRA) for SRM The SRA provides the ability to leverage HyperFlex nativeasync replication with the powerful orchestration and runbook capabilities of SRM The SRA includesthe ability to perform test recoveries planned migrations and full disaster recovery

HX SRA is certified by VMware and is available for download from VMWareSRM site

Note

bull HyperFlex DR Powershell RunbooksmdashNew Powershell cmdlets are included for automated runbookgeneration when using HyperFlex native disaster recovery The New-HXRunbook cmdlet supports thefollowing workflows Test Recovery Planned Migration and Disaster Recovery These runbooks canbe used to orchestrate DR workflows without the requirement for any third-party software

bull Windows Server 2019 with Hyper-VmdashSupport has been added in this release for the Windows Server2019 operating system for Hyper-V based HyperFlex deployments

bull Kubernetes CSI PluginmdashThis release adds support for the HyperFlex CSI (HX-CSI) plugin based onthe Kubernetes Container Storage Interface (CSI) specification Customers can now use the HX-CSIplugin to provision andmanage persistent volumes in Kubernetes version 113 and later Note Kubernetes113 support for Cisco Container Platform amp Openshift Container Platform is forthcoming in theirrespective future releases

bull C480 ML Compute only nodemdashThis release introduces support for C480 ML as a new compute-onlynode for Deep LearningMachine Learning Workloads Data scientists can now use the power of up toeight NVidia SXM2 V100 GPUs to accelerate deep learning workloads VMs running deep-learningworkloads will need to use PCIe pass-through for access to GPUs

bull Higher Capacity DrivesmdashAnew 24TB 10k rpm SAS HDD option for SFF Hybrid HyperFlex clustersand a 12TB 72K rpmSASHDDoption for LFFHybrid are now available Both HyperFlex andHyperFlexEdge support the 24TB capacity point for maximum density in this form factor Note that HyperFlex

Release Notes for Cisco HX Data Platform Release 408

New Features

Edge does not support LFF-drives HyperFlex HyperV version does not yet support the new 12TB driveoption See the HyperFlex spec sheets for a full list of configurable options

bull New Cache and increased scale for Hyper-V mdashNVMe amp Optane SSDs are now supported as cachedrives for Hyper-V deployments Furthermore scale limits have been increased to 16+16(Converged+Compute-only) for both SFF (AF and Hybrid) amp LFF (Hybrid) clusters

bull Centralized Audit Log ExportmdashThis release adds support for audit logging via a remote syslog serverThis capability enables customers to retain audit logs from all HyperFlex nodes in a centralized remotesyslog server to meet retention and compliance requirements

bull DISA STIG CompliancemdashThis release adds new HX REST APIs for setting removing and checkingstatus of DISA STIGs for Controller VMs ESXi hosts and vCenter These APIs enable customers tomeet DISA security requirements by centrally and securely applying STIGs detecting and correctingfor drifts in any STIG settings

Supported Versions and System RequirementsCisco HX Data Platform requires specific software and hardware versions and networking settings forsuccessful installation

For a complete list of requirements see

bull Cisco HyperFlex Systems Installation Guide for VMware ESXi or

bull Cisco HyperFlex Systems Installation Guide for Microsoft Hyper-V

Hardware and Software Interoperability

For a complete list of hardware and software inter-dependencies refer to respective Cisco UCS Managerrelease version of Hardware and Software Interoperability for Cisco HyperFlex HX-Series

HyperFlex Software Versions

The HX componentsmdashCisco HX Data Platform Installer Cisco HX Data Platform and Cisco UCSfirmwaremdashare installed on different servers Verify that each component on each server used with and withinan HX Storage Cluster are compatible

bull HyperFlex does not support UCS Manager and UCS Server Firmware versions 40(4a) 40(4b)and 40(4c)

Do not upgrade to these versions of firmware

Do not upgrade to these versions of UCS Manager

Important

bull Verify that the preconfigured HX servers have the same version of Cisco UCS server firmware installedIf the Cisco UCS Fabric Interconnects (FI) firmware versions are different see the Cisco HyperFlexSystems Upgrade Guide for VMware ESXi Release 40 for steps to align the firmware versions

bull M4 For NEW hybrid or All Flash (Cisco HyperFlex HX240c M4 or HX220c M4) deploymentsverify that Cisco UCS Manager 31(3k) 32(3i) or 40(2b) or higher is installed For moreinformation see Recommended Cisco HyperFlex HX Data Platform Software Releases

Release Notes for Cisco HX Data Platform Release 409

Supported Versions and System Requirements

bull M5 For NEW hybrid or All Flash (Cisco HyperFlex HX240c M5 or HX220c M5) deploymentsverify that the recommended UCS firmware version is installed

For SED-based HyperFlex systems ensure that the A (Infrastructure) B (Bladeserver) and C (Rack server) bundles are at Cisco UCS Manager version 40(2b)or later for all SED M4M5 systems For more details see CSCvh04307 ForSED-based HyperFlex systems also ensure that all clusters are at HyperFlexRelease 35(2b) or later For more information see Field Notice (70234) andCSCvk17250

Important

bull To reinstall an HX server download supported and compatible versions of the software See theCisco HyperFlex Systems Installation Guide for VMware ESXi for the requirements and steps

bull Important For Intersight edge servers running older than 40(1a) CIMC version HUU is the suggestedmechanism to update the firmware

Table 1 HyperFlex Software Versions for M4M5 Servers

M5 Recommended FIServer Firmware

(be sure to review important notes above)

M4 Recommended FIServer Firmware

(be sure to review important notes above)

HyperFlexRelease

40(4h)40(4h)40(2b)

40(4h)40(4h)40(2a)

40(4h)40(4h)40(1b)

40(4e)40(4e)40(1a)

HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments

Cisco HX Data Platform Release 4x based Deployments

Confirm the component firmware on the server meets the minimum versions listed in the following tables

HyperFlex Edge does not support Cisco IMC versions 40(4a) 40(4b) 40(4c) 40(4d) and 40(4e)Important

Table 2 HX220c M4 HXAF220c M4 Cluster

Recommended Firmware Version - HXDP 4x

(be sure to review important note(s) above)

Component

402d0BIOS

40(2h)Cisco Integrated Management Controller(CIMC)

Release Notes for Cisco HX Data Platform Release 4010

HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments

Recommended Firmware Version - HXDP 4x

(be sure to review important note(s) above)

Component

13000012SAS HBA

43(2b)VIC 1227

0103 (Toshiba)

CS07 (Intel 480)

SSD

CN03 (Seagate)

CK03 (Seagate 18)

CK03 (Seagate 24)

sF3Q (Samsung)

3F3Q (Samsung)

HDD

40(2h)

Download Software

Host Upgrade Utility (HUU) Download Link

Table 3 HX220c M5 HXAF220c M5 Cluster

Recommended Firmware Version - HXDP 4x

(be sure to review important notes above)

Component

C220M5404p00224200755BIOS

40(4i)Cisco Integrated Management Controller(CIMC)

09000008SAS HBA

43(3c)VIC 1387

50(3e)VIC 1457

G201CS01 (Intel)

0104 (Toshiba)

0107 (Toshiba)

SSD

2F3Q (Samsung)

3F3Q (Samsung)

NOA4 (Seagate)

CK03 (Seagate 18tb)

CK03 (Seagate 24)

HDD

Release Notes for Cisco HX Data Platform Release 4011

HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments

Recommended Firmware Version - HXDP 4x

(be sure to review important notes above)

Component

40(4k)

Download Software

Host Upgrade Utility (HUU) Download Link

HyperFlex Licensing

As of version 26(1a) HyperFlex supports VMware PAC licensing Existing VMware embedded licenses willcontinue to be supported

As of version 25(1a) HyperFlex uses a smart licensing mechanism to apply your licenses See the CiscoHyperFlex Systems Installation Guide for VMware ESXi for details and steps

VMware vSphere Licensing Requirements

How you purchase your vSphere license determines how your license applies to your HyperFlex system

bull If you purchased your vSphere license with HyperFlex

Each HyperFlex server either has the Enterprise or Enterprise Plus edition preinstalled at the factory

bull HX Nodes have OEM licenses preinstalled If you delete or overwrite thecontent of the boot drives after receiving the HX servers you also delete thefactory-installed licenses

bull OEM license keys is a newVMware vCenter 60 U1b feature Earlier versionsdo not support OEM licenses

bull All factory-installed HX nodes share the same OEM license key WithvSphere OEM keys the Usage count can exceed the Capacity value

bull When you add an HX host to vCenter through the Add Host wizard in theAssign license section select the OEM license

We obfuscate the actual vSphere OEM license key for example0N085-XXXXX-XXXXX-XXXXX-10LHH

bull Standard Essentials Plus and ROBO editions are not available preinstalledon HX servers

Note

bull If you did NOT purchase your vSphere license with HyperFlex

The HX nodes have a vSphere Foundation license preinstalled After initial setup you can apply thelicense to a supported version of vSphere

bull If you purchased a vSphere PAC license

Follow the instructions in your PAC license letter from VMware to add the license to your MY VMwareaccount then follow the instructions to add your HX host to vCenter and assign the PAC license

Release Notes for Cisco HX Data Platform Release 4012

HyperFlex Licensing

HX Data Platform Software Versions for HyperFlex Witness Node

Witness Node VersionHyperFlex Release

10840(2b)

10840(2a)

10440(1b)

10440(1a)

Software Requirements for VMware ESXi

The software requirements include verification that you are using compatible versions of Cisco HyperFlexSystems (HX) components and VMware vSphere VMware vCenter and VMware ESXi

bull Verify that all HX servers have a compatible version of vSphere preinstalled

bull Verify that the vCenter version is the same or later than the ESXi version

bull Verify that the vCenter and ESXi versions are compatible by consulting the VMware ProductInteroperability Matrix Newer vCenter versions may be used with older ESXi versions so long as bothESXi and vCenter are supported in the table below

bull Verify that you have a vCenter administrator account with root-level privileges and the associatedpassword

For VIC1457 there is no support for ESXi 60Note

The below table applies for all of the following VMware vSphere Editions Enterprise Enterprise PlusStandard Essentials Plus ROBO

VMware vCenter VersionsVMware ESXi VersionsHyperFlex Version

60 U3 65 U3 67 U360 U3 65 U3 67 U340(2b)1

60 U3 65 U3 67 U360 U3 65 U3 67 U340(2a)2

60 U3 65 U3 67 U260 U3 65 U3 67 U2340(1b)

60 U3 65 U2 67 U260 U3 65 U2 67 U2440(1a)

1 Cisco HyperFlex Release 40(2) is the last major HyperFlex release that will support vSphere 60 (ESXiand vCenter) due to those versions reaching end of VMware general support on March 12 2020

2 Cisco HyperFlex Release 40(2) is the last major HyperFlex release that will support vSphere 60 (ESXiand vCenter) due to those versions reaching end of VMware general support on March 12 2020

3 Use of 67 U2 is not recommended see software advisory for CSCvq06952 and CSCvp88515 for furtherdetails

4 Use of 67 U2 is not recommended see software advisory for CSCvq06952 and CSCvp88515 for furtherdetails

Release Notes for Cisco HX Data Platform Release 4013

HX Data Platform Software Versions for HyperFlex Witness Node

Software Requirements for Microsoft Hyper-V

The software requirements include verification that you are using compatible versions of Cisco HyperFlexSystems (HX) components and Microsoft Hyper-V (Hyper-V) components

HyperFlex Software versions

The HX componentsmdashCisco HX Data Platform Installer Cisco HX Data Platform and Cisco UCSfirmwaremdashare installed on different servers Verify that each component on each server used with and withinthe HX Storage Cluster are compatible

bull Cisco HyperFlex M5 Converged nodesmdash For Hybrid (Cisco HyperFlex HX240c M5 HX220c M5)and All Flash (Cisco HyperFlex HXAF240cM5 HXAF220cM5) verify that Cisco UCSManager 40(2b)is installed HX 40 (1a) does not support Hyper-V on the All NVMe (HXAF220C-M5SN) nodes Fordetailed information on installation requirements and steps see theCisco HyperFlex Systems InstallationGuide on Microsoft Hyper-V

Table 4 Supported HyperFlex Software versions for M5 Servers on Hyper-V

M5 Recommended Server FirmwareHyperFlex Release

40(4h)40(2b)

40(4g)40(2a)

40(4e)40(1b)

40(4e)40(1a)

Release Notes for Cisco HX Data Platform Release 4014

Software Requirements for Microsoft Hyper-V

Table 5 Supported Microsoft Software versions

VersionMicrosoft Component

Windows Server 2016 Datacenter Core amp DesktopExperience

ForWindows Server 2016Datacenter Coreand Desktop Experience the Windows2016 ISO image should be Update BuildRevision (UBR) 1884 at a minimum

Note

Windows Server 2019Datacenter-DesktopExperienceis supported starting from HXDP 401(a) onwards

For Windows Server 2019 DesktopExperience theWindows 2019 ISO imageshould be Update Build Revision (UBR)107 at a minimum

Note

Windows Server 2019 DatacenterndashCore is notsupported currently

Also note that the following are currently notsupported

OEM activated ISOs and Retail ISOs are notsupported

Earlier versions ofWindows Server such asWindows2012r2 are not supported

Non-English versions of the ISO are not supported

Windows Operating System (Windows OS)

A Windows 2012 or later domain and forestfunctionality level

Active Directory

Supported Microsoft License Editions

The Microsoft Windows Server version that is installed on one or more HyperFlex hosts must be licensed asper Microsoft licensing requirements listed on Microsoft Licensing

Browser Recommendations

Use one of the following browsers to run the listed HyperFlex components These browsers have been testedand approved Other browsers might work but full functionality has not been tested and confirmed

Table 6 Supported Browsers

HX ConnectHX Data PlatformInstaller

Cisco UCS ManagerCisco IntersightBrowser

11 or higher11 or higher11 or higherNAMicrosoft InternetExplorer

70 or higher70 or higher57 or higher62 or higherGoogle Chrome

Release Notes for Cisco HX Data Platform Release 4015

Browser Recommendations

HX ConnectHX Data PlatformInstaller

Cisco UCS ManagerCisco IntersightBrowser

60 or higher60 or higher45 or higher57 or higherMozilla Firefox

NANA9 or higher10 or higherApple Safari

NANA35 or higherNAOpera

Notes

bull Cisco HyperFlex Connect

The minimum recommended resolution is 1024 X 768

bull Cisco HX Data Platform Plug-in

The Cisco HX Data Platform Plug-in runs in vSphere For VMware Host Client System browserrequirements see the VMware documentation at httpswwwvmwarecomsupportpubs

The HX Data Platform Plug-in is not displayed in the vCenter HTML client You must use the vCenterflash client

bull Cisco UCS Manager

The browser must support the following

bull Java Runtime Environment 16 or later

bull Adobe Flash Player 10 or higher is required for some features

For the latest browser information about Cisco UCSManager refer to themost recent Cisco UCSManagerGetting Started Guide

Cisco HX Data Platform Storage Cluster SpecificationsCluster Limits

bull Cisco HX Data Platform supports up to 100 clusters managed per vCenter as per VMware configurationmaximums

bull Cisco HX Data Platform supports any number of clusters on a single FI domain Each HX convergednode must be directly connected to a dedicated FI port on fabric A and fabric B without the use of a FEXC-series compute only nodes must also directly connect to both FIs B-series compute only nodes willconnect through a chassis IO module to both fabrics In the end the number of physical ports on the FIwill dictate the maximum cluster size and maximum number of individual clusters supported in a UCSdomain

The following table provides Cisco HX Data Platform Storage Cluster Specifications

Release Notes for Cisco HX Data Platform Release 4016

Cisco HX Data Platform Storage Cluster Specifications

Stretched Cluster (Availableon ESX Only)

Microsoft Hyper-VVMware ESXiNode

AllNVMe -HXAF220c-M5SN

HX240c-M5LHX220cM5

HX220cAF M5

HX240cM5

HX240cAF M5

HX240c-M5LHX220cM5

HX220cAF M5

HX240cM5

HX240cAF M5

HXAF220c-M5SN

AllNVMe -HXAF220c-M5SN

ESXi 65or 67only NotsupportedwithESXi 60orHyper-V

HX220cM5 Edge

HXAF220cM5 Edge

HX220cM4 Edge

HXAF220cM4 Edge

HX240c-M5LHX220cM5

HX220cAF M5

HX240cM5

HX240cAF M5

HX220cM4

HX220cAF M4

HX240cM4

HX240cAF M4

HXServers

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

C220M5C240M5B200M4B200M5

C240M5C220M5B200M4B200M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

mdashB200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

Compute-OnlyUCSB-SeriesC-SeriesServers

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

Convergednodesonly

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

SupportedNodes

Release Notes for Cisco HX Data Platform Release 4017

Cisco HX Data Platform Storage Cluster Specifications

Stretched Cluster (Availableon ESX Only)

Microsoft Hyper-VVMware ESXiNode

NA(requiresEnterpriseHXDP-PLicense)

NA(requiresEnterpriseHXDP-PLicense)

NA(requiresEnterpriseHXDP-PLicense)

Convergednodes3-16(12TBHDDoption isnotsupportedforHyperV)

Compute-onlynodes0-16

Convergednodes3-16

Compute-onlynodes0-16

NA(requiresEnterpriseHXDP-PLicense)

M4Convergednodes 3

M5Convergednodes23or 4

RequiresHXDP-ELicense

Convergednodes3-16(12TBdrive islimited toamaximumof 8nodes)

Compute-onlynodes0-16(12TBdrive islimited toamaximumof 8nodes)

Convergednodes3-32(76TBdriveconfigsarelimited tomaximumof 12drivesnodeon AF240)

Compute-onlynodes0-32

HXDP-SLicensedNodeLimits

11 ratioofHXDP-StoComputeonlynodes

(MinmdashMax)

Convergednodes2-16 perSite

Compute-onlynodes0-21 perSite

(up tomaxclustersize)

Convergednodes2-8 perSite

Compute-onlynodes0-16 perSite

(up tomaxclustersize)

Convergednodes2-16 perSite

Compute-onlynodes0-21 perSite

(up tomaxclustersize)

Convergednodes3-16(12TBHDDoption isnotsupportedforHyperV)

Compute-onlynodes0-16

Convergednodes3-16

Compute-onlynodes0-16

Convergednodes3-32

ComputeOnlynodes0-32

(up tomaxclustersize)

Convergednodes 3

(requiresHXDP-ELicense)

Convergednodes3-16(12TBdrive islimited toamaximumof 8nodes)

Compute-onlynodes0-32(12TBdrive islimited toamaximumof 16nodes)

Convergednodes3-32(76TBdriveconfigsarelimited tomaximumof 12drivesnodeon AF240)

Compute-onlynodes0-32

(up tomaxclustersize)

HXDP-PLicensedNodeLimits

12 ratioofHXDP-PtoComputeonlynodes

(MinmdashMax)

24 perSite 64percluster

24 perSite 48percluster

32 perSite 64percluster

32326434864MaxClusterSize

Release Notes for Cisco HX Data Platform Release 4018

Cisco HX Data Platform Storage Cluster Specifications

Stretched Cluster (Availableon ESX Only)

Microsoft Hyper-VVMware ESXiNode

212121111121mdash2121MaxComputetoConvergedratio

NoExpansion

Requires Enterprise license

Requires uniform expansion across both sites

Guidelines and Limitationsbull Starting with release 40(2a) SCVM is no longer needed on a Compute node

bull HX REST API Access Token Management ndash Applications leveraging HX REST APIs should re-useaccess tokens when making API calls Once obtained using the AAA Obtain Access Token API accesstokens are valid for 18 days (1555200 seconds) In addition AAA enforces rate limiting on ObtainAccess Token API requests in a 15 minute window auth can be invoked (successfully) a maximum of5 times A user is allowed to create a maximum of 8 unrevoked tokens Subsequent call to auth willautomatically revoke the oldest issued token tomake room for the new token Amaximum of 16 unrevokedtokens can be present in system In order to prevent brute-force attacks after 10 consecutive failedauthentication attempts a user account is locked for a period of 120 secondsFor more information seeCisco HyperFlex Systems REST API Reference guide

HxConnect makes use of AAA Authentication REST API for login and the above rate limit applies toHxConnect also

Upgrade Guidelines

The following list is a highlight of critical criteria for performing an upgrade of your HyperFlex system

bull Unsupported HX Data Platform 17x 18x 20 21x 25x and 26x clustersmdashUsers from anyversion prior to 26(1a) must step through an intermediate version before upgrading to 40 or later releasesIf you need to upgrade your environment from a Cisco HyperFlex HX Data Platform software releasethat is past the last date of support to the latest suggested release on the Cisco Software Download sitesee Cisco HyperFlex Systems Upgrade Guide for Unsupported Cisco HXReleases For more informationsee the Software Advisory for CSCvq66867 WARNING Only Use HXDP 26(1e) Upgrade PackageWhen Upgrading From HXDP 18(1a)-18(1e)

bull Hypercheck Health Check UtilitymdashCisco recommends running this proactive health check utility onyour HyperFlex cluster prior to upgrade These checks provide early visibility into any areas that mayneed attention andwill help ensure a seamless upgrade experience For more information see the HyperFlexHealth amp Pre-Upgrade Check Tool TechNote for full instructions on how to install and run Hypercheck

bull vSphere 67 Software AdvisorymdashDo not upgrade to Cisco HX Data Platform Release 40(1a) whenrunning ESXi 67U1 EP06 (build 11675023) Do not upgrade to 67U1 EP06 (build 11675023) ifrunning Cisco HX Data Platform Release 40(1a) See the Software Advisory CSCvo56350 for furtherdetails

Release Notes for Cisco HX Data Platform Release 4019

Guidelines and Limitations

The software build version posted at release will override any other local versions

bull Required vCenter upgrademdashFor enhanced security Cisco HX Data Platform Release 35(1a) or laterrequires the use of TLS 12 Therefore vCenter must be upgraded to 60 U3f or later before upgradingto Cisco HX Data Platform Release 35 or later In addition ESXi should be upgraded as required tomeet HX Data Platform compatibility requirements

bull Minimum HXDP version for upgrademdashHXData Platform clusters running 26(1a) or later may upgradedirectly to 40 using the HX Connect UI

bull Cluster ReadinessmdashEnsure that the cluster is properly bootstrapped and the updated plug-in is loadedbefore proceeding Manual cluster bootstrap is required for upgrade from a pre-35 release

bull Cluster ReadinessmdashEnsure that the cluster is properly bootstrapped and the updated plug-in is loadedbefore proceeding Manual cluster bootstrap is required for HX releases earlier than 35(1a) For moreinformation see theManual Bootstrap Upgrade Process in the Cisco HyperFlex Systems Upgrade Guidefor VMware ESXi Release 40 Do not skip this cluster bootstrap step it is required for all upgradesuntil HX Release 35(1a) Auto bootstrap is supported beginning with HX release 35(1a) For moreinformation see the Auto Bootstrap Upgrade Process from HX Connect UI in the Cisco HyperFlexSystems Upgrade Guide for VMware ESXi Release 40

Manual bootstrap is not supported on Intersight clusters

bull Initiating Upgrade―Use the HX Connect UI or CLI stcli commands when upgrading from 25(1a)or later releases Use either the CLI stcli commands or the HX Data Platform Plug-in to the vSphereWeb Client when upgrading from a pre-25(1a) release The vCenter plug-in should not be used forupgrades starting with the 25(1a) release The vCenter plug-in should not be used for upgrades startingwith the 25(1a) release

If the current cluster version is at 35(1a) or above you do not need to use the stcli command Directupgrade to 40 is possible

bull Complete your Upgrade―The self-healing (or rebalance) capability is disabled temporarily during theupgrade window If the upgrade fails you should complete the upgrade as soon as possible

bull ESXi and HXDP Compatibility―Ensure your cluster is running a compatible version of ESXi basedon the running the HXData Platform version (see the section Software Requirements for VMware ESXi)ESXi compatibility is determined by the major version and update release of ESXi It is generally bestto upgrade HXDP and ESXi together if combining the upgrade operations into a single optimized rebootWhen running a split upgrade first upgrade the HX Data Platform then proceed to upgrade ESXi

bull If Upgrading to vSphere 65

bull Certain cluster functions such as native and scheduled snapshotsReadyClones and Enter or Exit HX Maintenance Mode will not operatefrom the time the upgrade is started until the HX Data Platform upgrade iscomplete

bull After upgrading ESXi using the offline zip bundle use the ESX ExitMaintenance Mode option The HX Exit Maintenance Mode option doesnot operate in the vSphere Web Client until the HX Data Platform upgradeis complete

Note

Release Notes for Cisco HX Data Platform Release 4020

Guidelines and Limitations

bull vSphere 60 UpgradesmdashUsers on vSphere 60 migrating to 65 upgrade components in the followingorder

1 Upgrade HX Data Platform and UCS firmware

2 Upgrade HX Data Platform and ESXi

3 Upgrade HX Data Platform only first then upgrade ESXi or UCS firmware or both

bull M4 Server Firmware UpgradesmdashUpgrade server firmware to ensure smooth operation and to correctknown issues Specifically newer SAS HBA firmware is available in this release and is recommendedfor long-term stability

bull Users are encouraged to upgrade to 31(3c) C-bundle or later whenever possible

bull Users running C-bundle versions before 31(2f) must upgrade server firmware by performing acombined upgrade of UCS server firmware (C-bundle) to 31(3c) or later and HX Data Platform to25 Do not split the upgrade into two separate operations

bull If the cluster is already on 31(2f) C-bundle or later you may perform an HX Data Platform onlyor combined upgrade as required

bull M5 Server Firmware UpgradesmdashM5 generation servers must run firmware version 32(2d) or later

bull Firmware Downgrades mdashDowngrading UCSM from the HX-installer is not supported

bull M4M5 Mixed DomainsmdashA mixed domain occurs when a new separate M5 cluster is installed underthe same UCS domain that contains existing M4 clusters Under these conditions orchestrated UCSserver firmware upgrade will not operate until Cisco HX Data Platform Release 26 or later is installedon the M4 clusters Therefore it is best practice to first upgrade UCS server firmware to the latest 31(3)or 32(2) patch release before adding a new M5 cluster to the existing UCS domain Additionally any17 HX Data Platform clusters must first be upgraded before adding any new M5 clusters to the samedomain

bull Maintenance WindowmdashIf upgrading both HX Data Platform and UCS firmware you can select eithera combined or split upgrade through the vSphere HX Data Platform Plug-in depending on the length ofthe maintenance window Cisco UCSManager infrastructure upgrade is only supported using AutoInstalland the direct server firmware upgrade should be performed only through the upgrade orchestrationframework provided by the HX Data Platform Plug-in

bull Unsupported Self-Encrypting Drives (SEDs)mdashIf adding or replacing self-encrypting drives (SEDs)that have been recently qualified in newer versions of HX Data Platform insert the new drives only afterupgrading HX Data Platform to a compatible version All drives must be SED drives mixing SED andnon-SED is not supported

bullbull Enabling External Host AccessmdashWith Cisco HX Data Platform Release 40(1a) port 445 on themanagement network is blocked for enhanced security Note that prior to 40 port 445 port was openenabling external host access If you are upgrading to 40(1a) from a prior release and would like tocontinue external host access you can use a utility to open select hosts For more information aboutenabling external host access see the ConfiguringHyperFlex Share to SCVMM section in the InstallationGuide for Microsoft Hyper-V

Release Notes for Cisco HX Data Platform Release 4021

Guidelines and Limitations

Mixed Cluster Expansion Guidelinesbull Expanding existing M4 cluster with M5 converged nodes is supported

bull Expanding existing M5 cluster with M4 converged nodes is not supported

bull Expanding existing mixed M4M5 cluster with M4 or M5 converged nodes is supported

bull Adding any supported compute-only nodes is permitted with all M4 M5 and mixed M4M5 clustersusing the HX Data Platform 26 or later Installer Some example combinations are listed here manyother combinations are possibleExample combinationsExpand mixed M4M5 cluster with compute-only B200 C220 C240 M4M5Expand M4 cluster with compute-only B200 M5 C220 M5 C240M5

bull Only expansion workflow is supported to create a mixed cluster Initial cluster creation with mixedM4M5 servers is not supported

bull All M5 servers must match the form factor (220240) type (HybridAF) security capability (Non-SEDonly) amp disk configuration (QTY capacity and non-SED) of the existing M4 servers

bull HX220-M5 will use a maximum of 6 capacity disks (2 disk slots to remain empty) when mixedwith HX220-M4

bull HX Edge SED LFF Hyper-V and Stretched Clusters do not support mixed M4 and M5 clusters

Security FixesThe following security issues are resolved

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-13132 CVE-2019-9924CSCvq6313840(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-11719 CVE-2019-11727CVE-2019-11729

CSCvq7124040(2a)

Release Notes for Cisco HX Data Platform Release 4022

Mixed Cluster Expansion Guidelines

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-1125CSCvr0633940(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2015-9383 CVE-2018-14498CVE-2018-20406 CVE-2018-20852CVE-2019-10160 CVE-2019-13117CVE-2019-13118 CVE-2019-14287CVE-2019-14973 CVE-2019-15903CVE-2019-17546 CVE-2019-18197CVE-2019-18218 CVE-2019-5010CVE-2019-5094 CVE-2019-5481CVE-2019-5482 CVE-2019-9636CVE-2019-9740 CVE-2019-9947CVE-2019-9948

CSCvs0609440(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2017-13168 CVE-2017-18174CVE-2017-18216 CVE-2018-10876CVE-2018-10877 CVE-2018-10878CVE-2018-10879 CVE-2018-10880CVE-2018-10881 CVE-2018-10882CVE-2018-10902 CVE-2018-10938CVE-2018-12233 CVE-2018-12896CVE-2018-13053 CVE-2018-13094CVE-2018-13096 CVE-2018-13405CVE-2018-13406 CVE-2018-14609CVE-2018-14617 CVE-2018-14633CVE-2018-14734 CVE-2018-15572CVE-2018-15594 CVE-2018-16276CVE-2018-16658 CVE-2018-17182CVE-2018-17972 CVE-2018-18021CVE-2018-18690 CVE-2018-18710CVE-2018-6554 CVE-2018-6555CVE-2018-9363

CSCvp6501940(2a)

This vulnerability is due to insufficientHTML iframe protection An attackercould exploit this vulnerability bydirecting a user to an attacker-controlledweb page that contains amaliciousHTMLiframe A successful exploit could allowthe attacker to conduct clickjacking orother clientside browser attacks

NACSCvo9851640(2a)

Release Notes for Cisco HX Data Platform Release 4023

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerability is due to insufficientauthentication for the statistics collectionservice An attacker could exploit thisvulnerability by sending properlyformatted data values to the statisticscollection service of an affected deviceA successful exploit could allow theattacker to cause the web interfacestatistics view to present invalid data tousers

NACSCvj9558440(2a)

The vulnerability is due to insufficientCSRF protections for the web UI on anaffected device An attacker could exploitthis vulnerability by persuading a user ofthe interface to follow a malicious linkA successful exploit could allow theattacker to perform arbitrary actions withthe privilege level of the affected user

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvp2434340(2a)

Vulnerabilities with a version of cJSONidentified by CVE-2019-11834 andCVE-2019-11835

For more information see the relatedCisco Security Advisory

CVE-2019-11834 CVE-2019-11835CSCvq1994940(2a)

HX ESXi image patches for MCEPSCand TAA vulnerabilities with VMwareESXi identified by CVE-2018-12207 andCVE-20190-11135

For more information see the relatedCisco Security Advisory

CVE-2018-12207 CVE-2019-11135CSCvr5439840(2a)

CP networking vulnerabilities affectingthe Linux kernel identified byCVE-2019-11477 CVE-2019-11478CVE-2019-11479

For more information see the relatedCisco Security Advisory

CVE-2019-11477 CVE-2019-11478CVE-2019-11479

CSCvq1954640(2a)

Qualification of Microsoft securitypatches for vulnerabilities withMicrosoftHyper-V hypervisor identified withCVE-2018-12207 CVE-2019-11135

For more information see the relatedCisco Security Advisory

CVE-2018-12207 CVE-2019-11135CSCvr5439940(2a)

Release Notes for Cisco HX Data Platform Release 4024

Security Fixes

DescriptionCVEDefect IDRelease

OpenSSH vulnerabilities identified byCVE-2019-6111

For more information see the relatedCisco Security Advisory

CVE-2016-10708 CVE-2018-15473CVE-2018-20685 CVE-2019-6109CVE-2019-6111

CSCvp7646340(2a)

Multiple vulnerabilities associated withOpenSSL and LibNSS

For more information see the relatedCisco Security Advisory

CVE-2016-2105 CVE-2016-2106CVE-2016-2107 CVE-2016-2109CVE-2016-2176 CVE-2016-2177CVE-2016-2178 CVE-2016-2179CVE-2016-2180 CVE-2016-2181CVE-2016-2182 CVE-2016-2183CVE-2016-6302 CVE-2016-6303CVE-2016-6304 CVE-2016-6305CVE-2016-6306 CVE-2016-6307CVE-2016-6308 CVE-2016-7055CVE-2016-8610 CVE-2017-3731CVE-2017-3732 CVE-2017-3735CVE-2017-3736 CVE-2017-3737CVE-2017-3738 CVE-2018-0495CVE-2018-0732 CVE-2018-0734CVE-2018-0735 CVE-2018-0737CVE-2018-0739 CVE-2018-12384CVE-2018-12404 CVE-2018-5407CVE-2019-1559

CSCvp6667940(2a)

libpng vulnerability identified byCVE-2016-10087

For more information see the relatedCisco Security Advisory

CVE-2016-10087CSCvp6655540(2a)

Vulnerabilities associated with libjpg andlibpng

For more information see the relatedCisco Security Advisory

CVE-2014-9092 CVE-2016-3616CVE-2017-15232 CVE-2018-11212CVE-2018-11213 CVE-2018-11214CVE-2018-1152 CVE-2018-13785

CSCvp3458640(2a)

Vulnerabilities associated with Glibidentified by CVE-2018-16428 andCVE-2018-16429

For more information see the relatedCisco Security Advisory

CVE-2018-16428 CVE-2018-16429CSCvp3120740(2a)

Multiple vulnerabilities associated withcurl expat python 27 python 35 36and 37 freetype and giflib

For more information see the relatedCisco Security Advisory

CVE-2019-15133 CVE-2019-15903CVE-2019-5010 CVE-2019-5481CVE-2019-5482 CVE-2019-9636CVE-2019-9740 CVE-2019-9947CVE-2019-9948

CSCvr3690340(2a)

Release Notes for Cisco HX Data Platform Release 4025

Security Fixes

DescriptionCVEDefect IDRelease

Multiple third party softwarevulnerabilities

For more information see the relatedCisco Security Advisory

CVE-2019-14379 CVE-2019-12384CVE-2019-14439

CSCvq9203240(2a)

Vulnerabilities associated with elfutils

For more information see the relatedCisco Security Advisory

CVE-2018-16062 CVE-2018-16402CVE-2018-16403 CVE-2018-18310CVE-2018-18520 CVE-2018-18521CVE-2019-7149 CVE-2019-7150CVE-2019-7665

CSCvq4325040(2a)

Vulnerabilities associated with vim

For more information see the relatedCisco Security Advisory

CVE-2017-5953 CVE-2019-12735CSCvq4323040(2a)

Vulnerabilities associated with sqlite3

For more information see the relatedCisco Security Advisory

CVE-2016-6153 CVE-2017-10989CVE-2017-13685 CVE-2017-2518CVE-2017-2519 CVE-2017-2520CVE-2018-20346 CVE-2018-20505CVE-2018-20506 CVE-2019-8457CVE-2019-9936 CVE-2019-9937

CSCvq4321340(2a)

Vulnerabilities associated with glib20

For more information see the relatedCisco Security Advisory

CVE-2018-20843CSCvq4320940(2a)

Vulnerabilities associated with expat

For more information see the relatedCisco Security Advisory

CVE-2018-20843CSCvq4320540(2a)

Vulnerabilities associated with bzip2

For more information see the relatedCisco Security Advisory

CVE-2016-3189 CVE-2019-12900CSCvq4319440(2a)

Mulitple Vulnerabilities associated wtihNodeJS

For more information see the relatedCisco Security Advisory

CVE-2018-12115 CVE-2018-0734CVE-2018-5407 CVE-2018-12120CVE-2018-12121 CVE-2018-12122CVE-2018-12123 CVE-2018-12116CVE-2019-5737 CVE-2019-5739

CSCvq1069440(2a)

Vulnerabilities associated with PalletsJinja strformat_map and identified byCVE-2019-10906 and CVE-2016-10745

For more information see the relatedCisco Security Advisory

CVE-2019-10906 CVE-2016-10745CSCvq1038840(2a)

Release Notes for Cisco HX Data Platform Release 4026

Security Fixes

DescriptionCVEDefect IDRelease

Multiple Security Vulnerabilities

For more information see the relatedCisco Security Advisory

CVE-2014-8501 CVE-2014-9939CVE-2015-9262 CVE-2016-10087CVE-2016-2226 CVE-2016-4487CVE-2016-4488 CVE-2016-4489CVE-2016-4490 CVE-2016-4491CVE-2016-4492 CVE-2016-4493CVE-2016-6131 CVE-2018-10963CVE-2018-13785 CVE-2018-17100CVE-2018-17101 CVE-2018-18557CVE-2018-18661 CVE-2018-7456CVE-2018-8905

CSCvq0917840(2a)

Vulnerabilities associated withlibseccomp

For more information see the relatedCisco Security Advisory

CVE-2019-9893CSCvq0756840(2a)

Vulnerabilities associated withgdk-pixbuf

For more information see the relatedCisco Security Advisory

CVE-2017-12447CSCvq0675540(2a)

Vulnerabilities associated withpython-crypto

For more information see the relatedCisco Security Advisory

CVE-2018-6594CSCvp9381740(2a)

Vulnerabilities associated with wget

For more information see the relatedCisco Security Advisory

CVE-2018-20483 CVE-2019-5953CSCvp8672140(2a)

Multiple vulnerabilities associated withPython Crypto

For more information see the relatedCisco Security Advisory

CVE-2018-6594CSCvp6674840(2a)

Vulnerabilities associated withsensible-utils

For more information see the relatedCisco Security Advisory

CVE-2017-17512CSCvp6673440(2a)

Multiple vulnerabilities associated withPython

For more information see the relatedCisco Security Advisory

CVE-2018-1000030CSCvp6668940(2a)

Release Notes for Cisco HX Data Platform Release 4027

Security Fixes

DescriptionCVEDefect IDRelease

Multiple patch vulnerabilities identifiedby CVE-2016-10713

For more information see the relatedCisco Security Advisory

CVE-2016-10713CVE-2018-1000156 CVE-2018-6951

CSCvp6667240(2a)

Vulnerabilities associatedwith Little CMSidentified by CVE-2016-10165

For more information see the relatedCisco Security Advisory

CVE-2016-10165 CVE-2018-16435CSCvp6666440(2a)

Vulnerabilities associated with TenableScan identified byCVE-2011-5325CVE-2019-11068CVE-2016-7076CVE-2017-1000368

For more information see the relatedCisco Security Advisory

CVE-2011-5325 CVE-2016-7076CVE-2017-1000368CVE-2019-11068

CSCvp6474640(2a)

Vulnerabilities associated with libxml2294 and earlier as used in XMLSec1223 and earlier and other products andidentified by CVE-2016-9318

For more information see the relatedCisco Security Advisory

CVE-2016-9318 CVE-2017-16932CVE-2017-18258 CVE-2018-14404CVE-2018-14567

CSCvp3479240(2a)

Vulnerabilities associated with LFTPRemote File Names Unauthorized AccessVulnerability and identified byCVE-2018-10916

For more information see the relatedCisco Security Advisory

CVE-2018-10916CSCvp2926640(2a)

Vulnerabilities associated withCryptographyDeprecationWarning signerand verifier have been deprecated

For more information see the relatedCisco Security Advisory

CVE-2018-7750CSCvo3409740(2a)

Multiple Vulnerabilities in python python27 35

For more information see the relatedCisco Security Advisory

CVE-2019-16056CSCvr3979340(2a)

Release Notes for Cisco HX Data Platform Release 4028

Security Fixes

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toexecute commands as the root user

The vulnerability is due to insufficientinput validation An attacker could exploitthis vulnerability by connecting to thecluster service manager and injectingcommands into the bound process Asuccessful exploit could allow the attackerto run commands on the affected host asthe root user

Cisco has released software updates thataddress this vulnerability There areworkarounds that address thisvulnerability

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvq2417640(1b)35(2g)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

The vulnerability is due to an unprotectedlistening interface An attacker couldexploit this vulnerability by connectingto the listening interface and injectingcommands to the bound process Anexploit could allow the attacker to runcommands on the affected host as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvj9560640(1b)35(2g)

Release Notes for Cisco HX Data Platform Release 4029

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerabilities associated with theJVM18U121memory leak found duringVC alarms (concurrent 40 calls usingREST API)

CVE-2017-10053 CVE-2017-10067CVE-2017-10074 CVE-2017-10078CVE-2017-10081 CVE-2017-10087CVE-2017-10089 CVE-2017-10090CVE-2017-10096 CVE-2017-10101CVE-2017-10102 CVE-2017-10107CVE-2017-10108 CVE-2017-10109CVE-2017-10110 CVE-2017-10111CVE-2017-10115 CVE-2017-10116CVE-2017-10118 CVE-2017-10135CVE-2017-10176 CVE-2017-10193CVE-2017-10198 CVE-2017-10243CVE-2017-10274 CVE-2017-10281CVE-2017-10285 CVE-2017-10295CVE-2017-10345 CVE-2017-10346CVE-2017-10347 CVE-2017-10348CVE-2017-10349 CVE-2017-10350CVE-2017-10355 CVE-2017-10356CVE-2017-10357 CVE-2017-10388CVE-2017-3509 CVE-2017-3511CVE-2017-3526 CVE-2017-3533CVE-2017-3539 CVE-2017-3544CVE-2018-2579 CVE-2018-2582CVE-2018-2588 CVE-2018-2599CVE-2018-2602 CVE-2018-2603CVE-2018-2618 CVE-2018-2629CVE-2018-2633 CVE-2018-2634CVE-2018-2637 CVE-2018-2641CVE-2018-2663 CVE-2018-2677CVE-2018-2678 CVE-2018-2783CVE-2018-2790 CVE-2018-2794CVE-2018-2795 CVE-2018-2796CVE-2018-2797 CVE-2018-2798CVE-2018-2799 CVE-2018-2800CVE-2018-2814 CVE-2018-2815CVE-2018-2952 CVE-2018-3136CVE-2018-3139 CVE-2018-3149CVE-2018-3150 CVE-2018-3169CVE-2018-3180 CVE-2018-3183CVE-2018-3214 CVE-2019-2422

CSCvo8899740(1b)

Tomcat and Nginx logs are not beingcollected in the support bundle generatedthrough HX Connect in Release 35

NACSCvm5803140(1b)

The vulnerabilities associated with thelibmspack software package versionincluded in Cisco HX Data Platform

CVE-2018-18584

CVE-2018-18585

CSCvn3511940(1a)

Release Notes for Cisco HX Data Platform Release 4030

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerabilities associated withFasterXML Jackson-Databind TimeValueField Denial of Service

CVE-2018-14719

CVE-2018-14720

CVE-2018-1000873

CVE-2018-14721

CVE-2018-19360

CVE-2018-19362

CVE-2018-19361

CVE-2018-14718

CSCvn8228240(1a)

The vulnerabilities associated with theOpenSSL Protocol software packageversion included in Cisco HX DataPlatform

CVE-2013-3587CSCvo0505440(1a)

The vulnerabilities associated withThird-Party Software Denial of Service

CVE-2018-16487

CVE-2018-19361

CSCvo2781840(1a)

The vulnerabilities associated withLinuxkernel for Ubuntu 1710

CVE-2018-1092

CVE-2018-7492

CVE-2018-8087

CVE-2018-1068

CVE-2018-8781

CSCvm5314935(2a)

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)25(1d)stcli services dns remove should remove the DNS serverinfo from the interface files

CSCvs70967

40(2b)40(1b)HX 40(1b) new deployment from Intersight productioncloud

User receives following error while enabling CSI(Kubernetes integration)

Failure occurred during volume_access_enable Errorwas Nonzero exit code 1

CSCvt10522

40(2b)40(2a)Receive the following alertevent in HX Connect HXController VM HOSTNAME one or more configuredDNS servers not responding

CSCvt13947

Release Notes for Cisco HX Data Platform Release 4031

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)40(2a)API UI CLI show few Drive slots Empty after upgradeto 402a

The drives are seen in the sysmtool lsscsi and stclicommand outputs

The cluster is healthy with no errors

CSCvt14914

40(2b)35(2d)After successfully changing the DNS server on the HXcontroller we still can see the original DNS entry thatwas added during the deployment

CSCvs69154

40(2b)35(2e)HyperFlex Connect and VCenter show APD alarms forHX and non HX datastores This implies there is an issuewith HyperFlex when there may not be

CSCvs30080

40(2b)40(2a)An issue where all the disks in the node get locked afterrebooting the node

We successfully unlocked using sed-clientsh -Ucommand but wanted to test with another reboot anddrives locked again

CSCvs74286

40(2b)40(2a)When running stcli license commands on HyperFlexthe following error is seen

rootSpringpathControllertmp stcli license show all

Show smart licensing failed

Smart Agent is not ready please wait a minute and tryagain

CSCvt13929

40(2b)40(2a)After upgrading to HX 40(2a) the first RoboEdge nodethat gets rebooted then ignores persistent drivesPIDHX-HD24TB10K4KN

Cluster will be in degraded state and have reducedcapacity

CSCvt20144

40(2b)35(2b)Zookeeper fails to start while Exhibitor is runninghowever echo srvr returns nothing

CSCvs21562

40(2b)40(2a)When performing an HyperFlex upgrade a validationwarning may occur due to the host not having EnterprisePlus or Enterprise hypervisor licensing

Upgrade Validation Warning

ESXi host esx1labtest should be configured withVMware Enterprise license for upgrade to continue

CSCvs91787

Release Notes for Cisco HX Data Platform Release 4032

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)35(2g)Cluster expansion fails at Config Installer stage whenthe root and admin password for the storage controller(SCVM) are different

CSCvs69317

40(2b)35(2g)Panic while upgrading ESXiCSCvt06983

40(2b)40(2a)A cluster node running may hang in the Linux kernelThis is classified as an oops and a deviation from theexpected behavior

CSCvs54285

40(2b)35(2g)Rebalance failing on 10+10 node HX 35(2g) stretchcluster

CSCvs69007

40(2b)35(2d)The cluster becomes inaccessible to the IOVisorCSCvr54687

40(2b)40(2a)Panic on the storage controllerCSCvt61297

40(2b)40(2a)The size of support bundle is very large when collectedwith storfs-support command

CSCvt63306

40(2b)40(2a)storfs crash associated withFileSystemUsageWarningEvent orFileSystemUsageAlertEvent due to high usage ofvarstv above 80

CSCvt72807

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

40(2a)35(2d)In normal scenarios HX upgrade gets pendingACK afterupdating service profile In this case UCS only upgradewas stalled because after updating service profile HXupgrade was not getting pending ACK and it was alwayswaiting state

CSCvo39912

40(2a)35(1a)When using motherBoardReplace-12 to clean ZK fromold stNodepNodes resulted in unmounted datastore andmaking the size of the Hyperflex Datastores 0 resultingin all VMs in the cluster going offline

CSCvq39471

40(2a)26(1e)Upgrading cluster and getting error

Failed upgrade validations Checking vCenterconfiguration Reason Upgrade validations failed DRSFault Insufficient resources to satisfy configured failover

CSCvm77294

Release Notes for Cisco HX Data Platform Release 4033

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)The cluster expand fails on a node with DR replicationconfigured

When a HX cluster which has DR replication configuredis expanded we see the installer UI pulling in thereplicationVLAN information instead of themanagementVLAN information Even if we change that informationto the correct mgmt VLAN id and name it it does notseem to work as the node is configured with the VLANof the replicationVLAN in ESXi This leads to the failureof the node add with the host unreachable error

CSCvo70650

40(2a)35(1a)Customer reported that server didnt complete thefirmware upgrade automatically one by one

The user manually needs to put the host into maintenancemode and they manually acknowledged the pendingrequests to finish the UCS firmware upgrade

Maintenance policy is set to default (User-ack) as perthe design

CSCvo91624

40(2a)35(1i)When the cluster is in Failed state and stcli noderemove is attempted the output shows successful thoughthe node failed to get removed from the cluster

CSCvo93017

40(2a)35(1a)HyperFlex cluster upgrade may fail during validationwith a DRS Validation failed error

CSCvp31021

40(2a)35(2b)HX cluster expansion will fail with the following errormessage

MAC address pool configuration failure150[ErrorDescription] bad address block range definitioncollision

CSCvp58318

40(2a)35(2b)Memory usage by carbon cacheCSCvq34873

40(2a)35(2g)Upgrading ESXi cluster fails with error Nodemaintenance mode failed

CSCvr03240

40(2a)35(2e)storfs process does not automatically start on Exit HXMaintenance Mode or other tasks which restart StorageController

CSCvr88978

40(2a)35(2a)Post install script fails with message

Failed to execute ipmitool on HX node

CSCvp10707

40(2a)40(1a)HyperFlex expansionworkflow doesnt pull VLANnamecorrectly

CSCvp46539

Release Notes for Cisco HX Data Platform Release 4034

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)During HX cluster deployment - Validate ClusterCreation phase - HX Installer might fail with the error

fromvarlogspringpathstDeploylog

CSCvq45087

40(2a)35(2b)HX Installer fails to login to the SOL for configuringESX hosts

CSCvq91380

40(2a)35(2d)If DRS is enabled with configuration parameters otherthen fully automated ie manual or partially automatedthan this will change to fully automated during clusterexpansion

CSCvr44222

40(2a)35(1a)One of the validation task name has a typo The taskname is Validating node for mixed of SED and non-SEDdisks

It should be Validating node for mix of SED andnon-SED disks

CSCvp82175

40(2a)30(1i)This combination of using smaller NVMe drives is notsupported in expand operation For example if the drivesare only 375GB and they cannot be added to an existingcluster with larger caching SSD

CSCvo12359

40(2a)35(1a)HyperFlex includes a version of OpenSSL that is affectedby the vulnerabilities identified by the followingCommon Vulnerability and Exposures

CVE-2018-0495

CSCvp66679

40(2a)30(1e)SDK invoke exception when taking a native snapshot ona VM with GPU pci passthrough attached

CSCvo00511

40(2a)30(1i)Node replacement script fails due to EAM errorCSCvo62867

40(2a)35(1a)While pairing a cluster (HX release gt= 35) and cluster(HX release lt 35) remote replication network test failson the cluster (HX release lt 35)

CSCvo79760

40(2a)30(1b)Fixed the issue in the latest support-workflow bundleThis should not be hit anymore

CSCvo87061

40(2a)35(2a)On a 3 node cluster the MBreplace script fails as thescript looks for HEALTHY cluster status

CSCvo87080

40(2a)35(2a)MbReplace script (tar) hangs when running on 352b or40

CSCvp12359

Release Notes for Cisco HX Data Platform Release 4035

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)HX replication cleanup failing with error NA 35(2a) -INFODR state is not clean

CSCvp63958

40(2a)40(1a)The check_stig_parameters API incorrectly shows thecompliance state of the cluster as not STIG compliant

CSCvp66277

40(2a)35(2c)The snapshot creation on CBT enabled VM fails witherror Failed in vmreparent vmkfstools clone1

CSCvq06952

40(2a)35(2a)HyperFlex GUI when setting up remote DR-replicationpartnerpeer should show remote replication partner TCPport reachability test results

This will provide quick upfront connectivity results thatfrequently takes customer a while to manually check andconfirm

If reachability tests fail an info window and HX alertshould trigger that list the destination peer IP and thespecific port(s) that could not be reache

CSCvo60587

40(2a)35(2b)If you have more than one IP pool configured for areplication network and local replication network testsare failing you might be running into this issue

CSCvr67130

40(2a)35(2a)Skip task option in MBreplace and Software redeployscripts The scripts should have the option to skip tasksas user inputs and not by modifying the json file

CSCvp05204

40(2a)40(1a)Error noticed on SCVM Console print_req_error IOerror dev fd0 sector 0 on HyperV

It doesnt come up on SSH session

CSCvq34357

40(2a)30(1d)The VMNetwork switch is already created as part of theinstall process

Ideally the ip address assignment should be done duringthe initial install process as well No post install stepsshould be required altrnatively it could be done as partof a post install script

CSCvk36222

40(2a)25(1c)storfs service impacted during HX upgrade from HX21(1) to 25(1c)

HX node which is not being upgraded storfs servicestopped causing outage

CSCvg53223

40(2a)35(1b)HX Stretch Cluster Witness VM reverts to DHCP onreboot

CSCvp37536

40(2a)35(2b)HyperFlex Stretch Cluster Witness does not getprogrammed with an NTP server

CSCvp46578

Release Notes for Cisco HX Data Platform Release 4036

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2b)HyperFlex cluster does not heal after maintenance onnodeserver

CSCvr18528

40(2a)40(2a)High percentage of packet loss on the eth1 data interface9K MTU

CSCvr97089

40(2a)35(2f)The HyperFlex cluster healing state is stuck at 87 andis unable to progress

CSCvr16760

40(2a)40(2a)In some circumstances two replications can occur forthe same VM

From the GUI the replication status for this VM willshow stuck in an In Progress state but there are nojobs stuck in replication layer Subsequent Replicationsshould succeed

CSCvq22898

40(2a)35(2a)After an unexpected power outage the hyperflex clusteris unable to come up correctly

All processes are running time is synced and all nodescan vmkping each other as required

CSCvp13990

40(2a)40(1a)When specifying IP address instead of node ID stclinode maintenanceMode commands fail

CSCvp33657

40(2a)40(1a)2 node edge cluster if experiencing Intersightarbitratorconnectivity issues should NOT allowHXMaintenancemode to initiate as this will bring the cluster down

CSCvr31573

40(2a)35(2c)Old files in varsupportZKTxnlog are not purged withthe daily zklog-cleanup cron job

CSCvr89066

40(2a)35(2a)When a node is in MaintenanceMode any disk removalor replacement will be reflected in UI only after the nodeis brought back frommaintenancemode This is becausestorfs is not running on the node in maintenance andwill not be able to detect disk activities until it is broughtout of Maintenance Mode

CSCvo86431

40(2a)35(2c)Cluster might go offline and stop serving IO requestsrootcvm~ stcli cluster storage-summary--detailGet cluster storage summary failedjavanetConnectException Connection refused19216814210010207

CSCvr01645

40(2a)35(2d)A stretch cluster deployment fails with a messageFormatting nodes Node in Use on the Hyperflex installerand crmZoneType value is seen as 1 in storfscfg oroptspringpathconfigstretchtunes

CSCvq80340

Release Notes for Cisco HX Data Platform Release 4037

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)30(1c)During upgrade we upload the upgrade packagedownloaded from Ciscocom Upgrade packages are offile type tgz hxconnect acceptsstorfs-packages-401a-33028tar file a different type ofcompression and allows upgrade to start and fail whenit cannot decompress the file for bootstrap and scvmclientupgrade on ESXi

CSCvq63888

40(2a)40(2a)While attempting to install hyperflex via intersight thefollowing error is seen

Failed in Task Add hosts to vCenter Cluster with ErrorTry adding hosts manually to vCenter and retry Failedto get addressFailed to host by namelookupHostnamecompanycom on 000053 read udp12700145430-gt12700153 read connection refused

CSCvr52098

40(2a)35(2a)CoreAPI call may time out for cluster management APIwhile doing cluster create validation

CSCvo92952

40(2a)35(2d)Validation fails with Mixed mode expansion checkdue to Enhanced vMotion Compatibility Incompatibilitywhen the ESXi version is different

CSCvq95460

40(2a)30(1i)This failure is seen in installer when custom workflowis used hypervisor configuration + deploy (clean diskpartition=no) at Installing software packages on storagecontroller VM

CSCvr66309

40(2a)35(2d)The HX Installer apparently replaced a comma bysemicolon on the VLAN ID range while exporting theconfiguration to a JSON file

CSCvq93831

40(2a)40(1a)When using unsupported browsers HX Connect userswill not be able to login These users see an errorindicating unauthorized user

CSCvp22693

Release Notes for Cisco HX Data Platform Release 4038

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)40(1a)A vulnerability in the web-based management interfaceof Cisco HyperFlex Software could allow anunauthenticated remote attacker to conduct a cross-siterequest forgery (CSRF) attack on an affected system

The vulnerability is due to insufficient CSRF protectionsfor the web UI on an affected device An attacker couldexploit this vulnerability by persuading a user of theinterface to follow a malicious link A successful exploitcould allow the attacker to perform arbitrary actions withthe privilege level of the affected user

Cisco has released software updates that address thisvulnerability There are no workarounds that address thisvulnerability

This advisory is available at the following link

httpstoolsciscocomsecuritycentercontentCiscoSecurityAdvisorycisco-sa-20190807-hypflex-csrf

CSCvp24343

40(2a)35(2d)Add message to HX Connect Progress flow to notAcknowledge Pending Activities and reboot the serverson UCSM

HX Connect is doing a controlled rolling server upgradein the background

CSCvq22844

40(2a)40(1b)Compute Node expand operation fails due to incorrectcluster name

CSCvr43786

40(2a)26(1e)RF-2 cluster shutdown during data resynchronizationafter a node failure (non storfs) followed by multipledisk read failures and a HardBlacklist of a disk

CSCvp41241

40(2a)40(1a)HyperFlex Edge nodes do not properly set the ESXihostname during deployment

CSCvo13143

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4039

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4040

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1b) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401b-33133p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4041

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4042

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1b)40(1a)While running the HyperFlex installer with WindowsServer Hyper-V cluster creation process fails with theerror Failure occurred during ClusterCreation process Unable to post thecontent to the downstream This symptomis encountered while deploying HyperFlex cluster withHyperFlex nodes configuredordered with Cisco VIC1457 MLOM (PID HX-MLOM-C25Q-04) andWindows Server Datacenter or Core with Hyper-Vversion 2016 or 2019

CSCvp64140

Release Notes for Cisco HX Data Platform Release 4043

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)35(2b)Adding Micron 5200 drive to a cluster fails to increasecluster capacity

CSCvo69067

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4044

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4045

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1a) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401a-33028p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4046

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4047

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1a)30(1d)Cluster instability when disks of different sector sizeplaced in HX node

CSCvk17250

40(1a)

35(2c)

35(1a)When logged in using a local HX user account insteadof a Virtual Center account an error message appearsintermittently indicating Virtual Centerunreachable or Resource informationcannot be updated when VC is reachable

CSCvo36198

40(1a)

35(2b)

30(1d)HXDP does not work with EMC RecoverPoint - needsto support VMware API (FSS-Readdir)

CSCvk38003

Release Notes for Cisco HX Data Platform Release 4048

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)

35(2a)

30(1i)

25(1c)

25(1d)

Zookeeper (Exhibitor) process on HyperFlex StorageController VMs may fail to respond to requests whenvarzookeeper has low or no free space

CSCvm90352

40(1a)26(1c)Use Asynchronous consolidation for HX SnapshotsCSCvn02151

40(1a)35(2b)HX Quiesced Snapshot for Backup VendorsCSCvo90713

40(1a)35(2a)The cluster creationcluster expansion workflow stopswith the following error message at the validation step

FIRMWARE-Check UCSC-SAS-M5HD

FIRMWARE-Check UCSC-SAS-M5HD Required00000029000000320000003500000050 Found00000058

Action Needed Update the Controller Firmware toRequired Version

CSCvn17787

35(2b)30(1d)Kernel migration fails when a local datastore is searchedfor in the ESXi host

CSCvn73127

35(2a)26(1b)A node shuts down when two disks are replaced (acaching disk and another capacity disk) where thecapacity disk is inserted first and then the caching diskis inserted

CSCvk46364

Hyper-V

40(1a)35(2a)Cluster expansion may fail with an error code 500 -operation timed out

CSCvn28721

35(2b)35(2a)During upgrade remove the VLAN on team created foruser vSwitch During fresh installation only one VLANtag is set to the vSwitch and team although multipleVLANs were entered

CSCvn54300

35(2b)35(2a)While upgrading a Hyper-V cluster on account of a rarerace condition between the stUpgradeService andZookeeper servers the upgrade orchestration throws anupgrade validation error and the upgrade process isaborted

CSCvn60486

Release Notes for Cisco HX Data Platform Release 4049

Resolved Caveats in Release 40(1a)

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Startrestart EAM service invCenter

When EAM service is not runningvCenter registration is failing

CSCvt55712

35(2a)Contact Cisco TAC if you believeyou are seeing similar conditions

HyperFlex Stretch Cluster saw poorapplication performance due to writelatency cluster remained unhealthy andrebalance was stuck

This defect is currently being used forinvestigative purposes

CSCvt45344

35(2g)Remove log files fromvarlogspringpath folder when thelog files are not getting rotatedStop the relevant service removelarge log files fromvarlogspringpath folder andrestart the service

varstv folder in Controller VM maybecome full

CSCvt36374

40(2b)Restarting the stMgr will initializeit again

In a Cluster configured with DR networkStMgr may not get initialized and stuckin a deadlock while enabling the IPTablerules (can be verified from the stMgrlog)

CSCvt89709

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Click recover for the selectedVM again under the standalonemode This will move the VM torecovered state

When the user recovers a VM protectedin a group it along with other VMs in thegroup move to recovering state instandalonemode The selectedVM shouldmove to recovered state

CSCvs75553

35(2e)Clean up the fail over cluster andthen recreate the fail over clusterNo need to touch the HX storagecluster

Hyper-V When replicated DC was usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2d)40(1a)

Keep cluster utilization as low aspossible To avoid read-only statekeep below 92

HyperFlex clusters storage utilization isabove 92 usable capacity

CSCvr20922

Release Notes for Cisco HX Data Platform Release 4050

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1b)NAAll the disks in the node got locked afterrebooting the node

We successfully unlocked usingsed-clientsh -U command but wantedto test with another reboot and driveslocked again

CSCvs74286

35(2g)To recover the VM copy over thedata disks and attach them to thenew VM

After enable encryption HX connectencryption status showing Caution butall of the disks enable encryptionsuccessful

-Self Encrypting Drives Service isrunning on all of the nodes

-There is no error message of encryption

- USB0 interface is up

-All SED disks showing supported 1enabled 1 locked 0

CSCvs93245

40(1b)Enable kernelpanic_on_oops as apersistent configuration This willcause the node to panic and rebootimmediately

A cluster node running HX release40(1b) may hang in the Linux kernelThis is classified as an oops and adeviation from the expected behavior

CSCvs54285

Release Notes for Cisco HX Data Platform Release 4051

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Enabling K8iscsi stack in Hx clusterCSCvs41324

Release Notes for Cisco HX Data Platform Release 4052

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Steps to enable K8 on HX cluster

Perform following steps on ALLthe controller VMs

1 Updateetcinitscvmclientconf toenable tune iscsiEnable totrue

sed -iesiscsiEnable=falseiscsiEnable=trueetcinitscvmclientconf

2 Run following initctlcommand to reloadconfiguration

initctl reload-configuration

3 Restart scvmclient process

stop scvmclient startscvmclient

4 Run following initctlcommand to mount SYSTEMDS

initctl emit --no-waitsystem-datastore-created

5 Verify that iscsiEnable tuneis set to true using thefollowing command

ps -eaf | grep scvmclient |grep -v grep

root 6241 1 1 Dec06 014007optspringpathstorfs-corescvmclient-T logEnabled=false -TlogSyslogEnabled=true -TlogEchoToScreen=false -TstatLoggingToFile=false -TstatLoggingToSyslog=true-TlogDir=varlogspringpath-TnfsBackendServerList=1010748100-T iscsiEnable=true -TiscsiUseAsync=true -T

Release Notes for Cisco HX Data Platform Release 4053

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

iscsiConfigFilePath=nfsSYSTEMistgtconfps -eaf | grep scvmclient |grep -v grep

35(2e)Following procedure can be usedto check NFS Queue Depth andincrease if needed

rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth256 lt- Low value[rootHXESXI1] vsish -e setvmkModulesnfsclientmnt[DATASTORE]propertiesmaxQDepth 1024[rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth1024 lt- Optimalvalue

This requires an ESXi host rebootto take effect

Please place the node in HyperflexMaintenanceMode and gracefullyreboot the node for the changes tobe applied

HyperFlex Datastore NFS Queue Depthshows as 256 which can lead toperformance (including latency) issues

CSCvr83056

40(1b)This is cosmetic The error will goaway once the cluster returns tohealthy status

When a single node is offline in clusterstcli cluster storage-summary shows twonodes unavailable rootSCVM~ stclicluster storage-summary

CSCvs47735

35(2g)Modify both the root and adminpassword for the SCVMs to be thesame

stcli security password set -uadmin

Cluster expansion fails at Config Installerstage when the root and admin passwordfor the storage controller (SCVM) aredifferent

CSCvs69317

35(2d)NAAfter successfully changing the DNSserver on the HX controller we still cansee the original DNS entry that was addedduring the deployment

CSCvs69154

Release Notes for Cisco HX Data Platform Release 4054

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Customers are strongly encouragedto work with TAC in order toidentify and remediate this issue

1 Change getcluster from 2m to8m inoptspringpathstorfs-mgmtstMgr-10confapplicationconfon all nodes

2 Restart stMgr on all nodes

You may see this error message after afailed upgrade or other task such asattempting to enter a node into HXmaintenance mode

CSCvs53555

40(2a)Run the following command oneach storage controller VM as root

grep -i monitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp sed -iesmonitor_dns_serverstruemonitor_dns_serversfalseoptspringpathhx-diag-toolswatchdog_configjsonampamp grep -imonitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp restart watchdog

Receive the following alertevent in HXConnect HX Controller VMHOSTNAME one or more configuredDNS servers not responding

CSCvt13947

35(2b)Delete any empty (Size 0) log filesunder varlogzookeeperversion-2ORvarlogzookeeperstandaloneversion-2

Zookeeper fails to start while Exhibitor isrunning however echo srvr returnsnothing

CSCvs21562

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

40(4e)1 De-commission the server

2 Power drain the server -REMOVE BOTH POWERCORDS ON THE BACK OFTHE SERVER FOR 10SECONDS THENREINSERTPOWERCORDS

3 Re-commission the server

M2 boot disk is missing from serverinventory after upgrade to server firmware40(4e) As a result server fails to boot toOS installed in the M2 disk The issuepersists after re-acknowledgement as wellas de-commission andre-acknowledgement of the server

CSCvs02466

Release Notes for Cisco HX Data Platform Release 4055

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Re-start the flusher manuallyMany disks taken offline by a fault ormanually resulting in ENOSPC oncluster

CSCvq32721

35(2e)Clean up fail over cluster and thenrecreate the fail over cluster Thereis no need to touch the HX storagecluster

[Hyper-V]When replicated DCwas usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2b)Manually install upgradescvmclient VIB andmake sure thatit matches with HXDP version

Upgrade from release HX 35(2a) to35(2x) 4X does not upgrade scvmclientRecovery point may not work

CSCvq54992

Management

30(1b)To recover the VM copy over thedata disks and attach them to thenew VM

[HyperV]VMshows up onmultiple nodesCSCvj22992

35(1a)1 Do not change MTU at ESXlevel on running cluster

2 Reset back to the original value

ChangingMTU from 9000 to 1500 on oneESX node causes storfs process to restarton all nodes

CSCvm99150

Release Notes for Cisco HX Data Platform Release 4056

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)After the installation (for newdeployments) or after an upgradeto 401a (for existingdeployments) perform thefollowing steps on all ControllerVMs

1 Edit the following tune files (onall Controller VMs)

optspringpathconfiglfftunes

optspringpathconfigvsi_16tbtunes

2 SetcleanerEnableSegSummaryCleaning

to false

3 After editing the above tunefiles

a SSH in to all the ControllerVMs

b Run storfstool -- -Z

c Run the below command andcheck the values Tune valueshould be true

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=true

d Type below commands andapplied the tunes changesdynamically

echo false gttmpstprocfssystemtunecleanerEnableSegSummaryCleaning

e Verify that tune values arechanged Run below commandValue should be false

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=false

f umount tmpstprocfs

A cluster with 8TB or 12TB disk drivesmay experience IO stalls for severalminutes after another node fails in thecluster

CSCvp23718

Release Notes for Cisco HX Data Platform Release 4057

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)1 Update the conf file manuallyto change to the link local addressto

2 Restart scvmclient on thecontroller and ESX

HX 35(2b) FlexVol to HX 40 CSIupgrade does not work FlexVol continuesto work

CSCvp26319

35(2a)Disable rebalance until node healsthen reenable rebalance

7-node cluster healing very slowlybecause rebalance change AMS reqs areblocking healing process

CSCvp62512

35(2a)NAWith moderate cluster storage capacityusage under site failover cluster clustermay experience an all paths (APD) downcondition

CSCvq49412

40(1a)NAThis product includes Third-partySoftware that is affected by thevulnerabilities identified by the followingCommon Vulnerability and Exposures(CVE) IDs

CVE-2016-0762 CVE-2016-6797CVE-2016-6816 CVE-2016-8735CVE-2017-5647 CVE-2017-12615CVE-2017-12616 CVE-2017-12617CVE-2017-7674 CVE-2018-1304CVE-2018-8014 CVE-2018-1336CVE-2018-8034 CVE-2018-11784CVE-2019-0232

CSCvp36364

35(2d)NAstcli cluster info command needs toprovide UCSM VIP address Currently itshows ucsm-hostcom

CSCvq11456

35(2a)No workarounds currently Thestale entries do not affect the dayto day operations of HyperFlex

HyperFlex upgrade validation failedbecause cluster has extraneous stNodes ininternal database

CSCvq32530

35(2a)NAWhen not in maintenancemode all drivesappear in HX Connect and get updated asexpected

When in maintenance mode extra drivesare shown in HX Connect with no slotnumbers

CSCvq42378

35(2b)Contact TAC to cleanup duplicatenode

Nodes duplicated in HX Connect aftercluster recreate

CSCvq61825

Release Notes for Cisco HX Data Platform Release 4058

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)NACurrently HyperFlex installs on Hyper-Vdo not contain the stcli security whitelistcommandset

CSCvq66245

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

Release Notes for Cisco HX Data Platform Release 4059

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)[Hyper-V] Converged node expansionfails for incoming node with VIC 14xx

CSCvq39523

Release Notes for Cisco HX Data Platform Release 4060

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

1 Set Jumbo frame inWindowsHypervisor for incomingnode

Get-NetAdapter storage-data-a|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

Get-NetAdapter storage-data-b|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

2 Verify all interfaces are setcorrectly to support Jumbo frames

Get-NetAdapter storage |Get-NetAdapterAdvancedProperty| RegistryKeyword -matchjumbo | ft -auto

You should receive the followingmessage

Name DisplayName DisplayValue

RegistryKeyword RegistryValue

---- ----------- ------------

--------------- -------------

vswitch-hx-storage-data Jumbo

Packet 9014 Bytes

JumboPacket 9014

storage-data-b Jumbo Packet

Bytes 9014 JumboPacket

9014

storage-data-a Jumbo Packet

Bytes 9014 JumboPacket

9014

3 Reboot only the incoming nodeor the node being expanded

4 Retry Cluster Expansion in

Release Notes for Cisco HX Data Platform Release 4061

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

bull 25GE networking for HX EdgemdashSupport for 25GE networking for HX Edge

bull All NVMe with Stretched ClustermdashSupport for All NVMe with Stretched Cluster (ESX only)

bull Cluster Upgrade Eligibility TestmdashThis release adds the capability to perform a pre-upgrade test whichchecks for cluster readiness before upgrading Example checks include validating cluster state rebalancestatus controllerVM Free Space ESXi version and much more The Elibility test is intended to helpavoid unexpected problems that may arise during the upgrade process It is highly recommended to runthe test before performing the Hyperflex upgrade

bull Registering Smart Software LicensingmdashThis release adds support for software that allows easy trackingof the status of license and software usage trends and simplifies the three core licensing functionsPurchasing Management and Reporting

bull Dynamic self-signed certificate generation enhancementsmdashThis release adds support for Self-signedSSL certificates on the Controller VMs which were static in prior releases The static certificates arereplaced with dynamically generated self-signed certificates upon upgrading to HXDP 40(2a) so thatthe certificates are unique per cluster The new clusters installed with HXDP 40(2a) have dynamicallygenerated self-signed certificates

bull Test Upgrade EligibilitymdashThis release adds support for testing your cluster readiness and infrastructurecompatibility for an upgrade For more information see the Test Upgrade Eligibility sections in theCiscoHyperFlex SystemsUpgrade Guide for VMware ESXi Release 40 or the CiscoHyperFlex UpgradeGuide for Microsoft Hyper-V Release 40

Disaster Recovery

bull Recovery Settings ConfigurationmdashThis release supports configuration of recovery settings to defineglobal recovery parameters and mapping for resources across recovery sites These parameters are usedduring recovery test recovery and migrate operations

bull HyperFlex DR Powershell RunbookmdashPowerShell runbook functionality is extended to support therecovery configuration settings in the runbooks for all recovery scenarios New-HXrunbook cmdlet cannow be used to generate runbook for a single or multiple protection groups In addition two new cmdletsWait-HXTask and Get-HXTaskStatus are introduced

bull Protected Virtual Machine ScalabilitymdashThis release adds support for 1500 VMs across both clustersand 750 VMs per cluster in a bi-direction or any split between the two clusters without exceeding thelimit of 1500 VMs For more information see Cisco HyperFlex Data Platform Administration Guide

bull System Management REST API enhancementsmdashPause data replication actions briefly using RESTAPI to explicitly inform users on the current status of replication actions

Cisco HyperFlex with Data Platform for Hyper-V

bull Cluster-Aware Updating (CAU)mdashThis is an automated feature that allows you to perform updates onwindows servers in a failover cluster with little or no loss in availability during the upgrade process

Support for new drivesmdashNew drives are qualified for the 40(2a) release The new drives include newcapacity points and new cache drive options Several of the new drives are alternate drives to already qualifiedexisting drives in function which are qualified in 40(2a) These drives are functionally compatible with theexisting drives and are available as alternates in case of lack of availability of existing drives For expansionof existing clusters or general information about interoperability of different drives see Cisco HyperFlexDrive Compatibility

Release Notes for Cisco HX Data Platform Release 406

New Features

NVMe Caching SSDs slot information is unavailable from HX-Connect for all AF server PIDs except forthe All-NVMe server PIDs Please refer to UCSM management console for NVMe SSD slot information

Note

Applicable PlatformsDrive PIDDrive Function

All existing HX M5 serversHX-M2-960GBAlternate boot drive

All existing HX M5 servers except AllNVMe

HX-SD480G6I1X-EVAlternate system (or housekeeping) drive

All existing HX M5 servers except AllNVMe

HX-SD480GM1X-EVAlternate system (or housekeeping) drive

The following HX M5 serversHXAF220C-M5SX HXAF240C-M5SXHXAF-E-220M5SX

HX-SD800G123X-EP800G 12G SAS Cache drive option forAll Flash

All NVMe HXAF220C-M5SNHX-NVME2H-I1000All NVMe 1TB Capacity drive

All NVMe HXAF220C-M5SNHX-NVME2H-I4000All NVMe 4TB Capacity drive

All NVMe HXAF220C-M5SNHX-NVMEHW-I8000New high density All NVMe 8TBCapacity drive

All Flash Configuration ndash namelyHXAF220C-M5SX HXAF240C-M5SXHXAF-E-220M5SX

ESX support only

HX-SD76T61X-EVNew high density All Flash 76TBCapacity drive

Maximum number of drives per node is12 drives on HX AF C240

HX240C-M5LHX-HD8T7K4KANAlternate drive for 8TB LFF capacity

HXAF220C-M5SX HXAF240C-M5SXHX-SD38T2HTNK9New 38TB FIPS compliant SED SSDdata drives

HXAF220C-M5SX HXAF240C-M5SXHX-SD960G2HTNK9New 960GFIPS compliant SEDSSD datadrives

New Features in Release 40(1b)

bull Support for Second Generation Intelreg Xeonreg Scalable Processor RefreshmdashThis release includessupport for the Second Generation Intelreg Xeonreg Scalable processor refresh (formerly Cascade Lake)

New Features in Release 40(1a)

The following new features are in Release 40(1a)

bull Ultra-Light HyperFlex Edge ClustersmdashThis release introduces support for two-node HyperFlex Edgeclusters enabling HyperFlex to run in environments requiring a small footprint Cisco Intersight providescomprehensive lifecycle management and includes remote cloud-based installation centralized upgradesand invisible witnessing Both 1GE and 10GE networking topology options are available

Release Notes for Cisco HX Data Platform Release 407

New Features

bull Scaled-Up HyperFlex Edge ClustersmdashThis release adds support for four node HyperFlex Edge clustersenabling a full range of size options for remote and branch offices Size the branch office environmentsto suit current needs with a two three or four node HyperFlex Edge cluster Cisco Intersight providesfull-lifecycle management and 1GE and 10GE networking options are available

bull Cisco Intersight Invisible Cloud WitnessmdashFor two node clusters this feature eliminates the need forwitness VMs the infrastructure to run those VMs and the management overhead to deploy scale andpatch witnessing software The Invisible Cloud Witness is responsible for maintaining cluster HA in theevent of failure scenarios This feature is included at no additional cost and is automatically deployedand managed by Cisco Intersight

bull Cloud-delivered HyperFlex Edge UpgradesmdashPowered by Cisco Intersight this feature adds supportfor multi-site orchestrated remote upgrades of the HyperFlex Data Platform This feature will be enabledwith the next 40 patch release and will allowHyperFlex Edge clusters deployed via Intersight to performorchestrated upgrades across one or many sites in parallel

bull All-NVMe HyperFlexmdashStarting with this release a new high-end performance node powered by allNVMe drives is available for HyperFlex clusters Co-engineered with Intel to support Intel VMD forhot-plug and surprise removal this offering represents an industry first an enterprise-ready and fullyvalidated all-NVMe HCI appliance The all-NVMe offering is available in the 220 form factor (1RU)and is powered by Intel Optane cache drives for maximum performance and highest endurance

bull VMware Site Recovery Manager (SRM) IntegrationmdashThis release brings support for a Cisco developedStorage Replication Adapter (SRA) for SRM The SRA provides the ability to leverage HyperFlex nativeasync replication with the powerful orchestration and runbook capabilities of SRM The SRA includesthe ability to perform test recoveries planned migrations and full disaster recovery

HX SRA is certified by VMware and is available for download from VMWareSRM site

Note

bull HyperFlex DR Powershell RunbooksmdashNew Powershell cmdlets are included for automated runbookgeneration when using HyperFlex native disaster recovery The New-HXRunbook cmdlet supports thefollowing workflows Test Recovery Planned Migration and Disaster Recovery These runbooks canbe used to orchestrate DR workflows without the requirement for any third-party software

bull Windows Server 2019 with Hyper-VmdashSupport has been added in this release for the Windows Server2019 operating system for Hyper-V based HyperFlex deployments

bull Kubernetes CSI PluginmdashThis release adds support for the HyperFlex CSI (HX-CSI) plugin based onthe Kubernetes Container Storage Interface (CSI) specification Customers can now use the HX-CSIplugin to provision andmanage persistent volumes in Kubernetes version 113 and later Note Kubernetes113 support for Cisco Container Platform amp Openshift Container Platform is forthcoming in theirrespective future releases

bull C480 ML Compute only nodemdashThis release introduces support for C480 ML as a new compute-onlynode for Deep LearningMachine Learning Workloads Data scientists can now use the power of up toeight NVidia SXM2 V100 GPUs to accelerate deep learning workloads VMs running deep-learningworkloads will need to use PCIe pass-through for access to GPUs

bull Higher Capacity DrivesmdashAnew 24TB 10k rpm SAS HDD option for SFF Hybrid HyperFlex clustersand a 12TB 72K rpmSASHDDoption for LFFHybrid are now available Both HyperFlex andHyperFlexEdge support the 24TB capacity point for maximum density in this form factor Note that HyperFlex

Release Notes for Cisco HX Data Platform Release 408

New Features

Edge does not support LFF-drives HyperFlex HyperV version does not yet support the new 12TB driveoption See the HyperFlex spec sheets for a full list of configurable options

bull New Cache and increased scale for Hyper-V mdashNVMe amp Optane SSDs are now supported as cachedrives for Hyper-V deployments Furthermore scale limits have been increased to 16+16(Converged+Compute-only) for both SFF (AF and Hybrid) amp LFF (Hybrid) clusters

bull Centralized Audit Log ExportmdashThis release adds support for audit logging via a remote syslog serverThis capability enables customers to retain audit logs from all HyperFlex nodes in a centralized remotesyslog server to meet retention and compliance requirements

bull DISA STIG CompliancemdashThis release adds new HX REST APIs for setting removing and checkingstatus of DISA STIGs for Controller VMs ESXi hosts and vCenter These APIs enable customers tomeet DISA security requirements by centrally and securely applying STIGs detecting and correctingfor drifts in any STIG settings

Supported Versions and System RequirementsCisco HX Data Platform requires specific software and hardware versions and networking settings forsuccessful installation

For a complete list of requirements see

bull Cisco HyperFlex Systems Installation Guide for VMware ESXi or

bull Cisco HyperFlex Systems Installation Guide for Microsoft Hyper-V

Hardware and Software Interoperability

For a complete list of hardware and software inter-dependencies refer to respective Cisco UCS Managerrelease version of Hardware and Software Interoperability for Cisco HyperFlex HX-Series

HyperFlex Software Versions

The HX componentsmdashCisco HX Data Platform Installer Cisco HX Data Platform and Cisco UCSfirmwaremdashare installed on different servers Verify that each component on each server used with and withinan HX Storage Cluster are compatible

bull HyperFlex does not support UCS Manager and UCS Server Firmware versions 40(4a) 40(4b)and 40(4c)

Do not upgrade to these versions of firmware

Do not upgrade to these versions of UCS Manager

Important

bull Verify that the preconfigured HX servers have the same version of Cisco UCS server firmware installedIf the Cisco UCS Fabric Interconnects (FI) firmware versions are different see the Cisco HyperFlexSystems Upgrade Guide for VMware ESXi Release 40 for steps to align the firmware versions

bull M4 For NEW hybrid or All Flash (Cisco HyperFlex HX240c M4 or HX220c M4) deploymentsverify that Cisco UCS Manager 31(3k) 32(3i) or 40(2b) or higher is installed For moreinformation see Recommended Cisco HyperFlex HX Data Platform Software Releases

Release Notes for Cisco HX Data Platform Release 409

Supported Versions and System Requirements

bull M5 For NEW hybrid or All Flash (Cisco HyperFlex HX240c M5 or HX220c M5) deploymentsverify that the recommended UCS firmware version is installed

For SED-based HyperFlex systems ensure that the A (Infrastructure) B (Bladeserver) and C (Rack server) bundles are at Cisco UCS Manager version 40(2b)or later for all SED M4M5 systems For more details see CSCvh04307 ForSED-based HyperFlex systems also ensure that all clusters are at HyperFlexRelease 35(2b) or later For more information see Field Notice (70234) andCSCvk17250

Important

bull To reinstall an HX server download supported and compatible versions of the software See theCisco HyperFlex Systems Installation Guide for VMware ESXi for the requirements and steps

bull Important For Intersight edge servers running older than 40(1a) CIMC version HUU is the suggestedmechanism to update the firmware

Table 1 HyperFlex Software Versions for M4M5 Servers

M5 Recommended FIServer Firmware

(be sure to review important notes above)

M4 Recommended FIServer Firmware

(be sure to review important notes above)

HyperFlexRelease

40(4h)40(4h)40(2b)

40(4h)40(4h)40(2a)

40(4h)40(4h)40(1b)

40(4e)40(4e)40(1a)

HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments

Cisco HX Data Platform Release 4x based Deployments

Confirm the component firmware on the server meets the minimum versions listed in the following tables

HyperFlex Edge does not support Cisco IMC versions 40(4a) 40(4b) 40(4c) 40(4d) and 40(4e)Important

Table 2 HX220c M4 HXAF220c M4 Cluster

Recommended Firmware Version - HXDP 4x

(be sure to review important note(s) above)

Component

402d0BIOS

40(2h)Cisco Integrated Management Controller(CIMC)

Release Notes for Cisco HX Data Platform Release 4010

HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments

Recommended Firmware Version - HXDP 4x

(be sure to review important note(s) above)

Component

13000012SAS HBA

43(2b)VIC 1227

0103 (Toshiba)

CS07 (Intel 480)

SSD

CN03 (Seagate)

CK03 (Seagate 18)

CK03 (Seagate 24)

sF3Q (Samsung)

3F3Q (Samsung)

HDD

40(2h)

Download Software

Host Upgrade Utility (HUU) Download Link

Table 3 HX220c M5 HXAF220c M5 Cluster

Recommended Firmware Version - HXDP 4x

(be sure to review important notes above)

Component

C220M5404p00224200755BIOS

40(4i)Cisco Integrated Management Controller(CIMC)

09000008SAS HBA

43(3c)VIC 1387

50(3e)VIC 1457

G201CS01 (Intel)

0104 (Toshiba)

0107 (Toshiba)

SSD

2F3Q (Samsung)

3F3Q (Samsung)

NOA4 (Seagate)

CK03 (Seagate 18tb)

CK03 (Seagate 24)

HDD

Release Notes for Cisco HX Data Platform Release 4011

HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments

Recommended Firmware Version - HXDP 4x

(be sure to review important notes above)

Component

40(4k)

Download Software

Host Upgrade Utility (HUU) Download Link

HyperFlex Licensing

As of version 26(1a) HyperFlex supports VMware PAC licensing Existing VMware embedded licenses willcontinue to be supported

As of version 25(1a) HyperFlex uses a smart licensing mechanism to apply your licenses See the CiscoHyperFlex Systems Installation Guide for VMware ESXi for details and steps

VMware vSphere Licensing Requirements

How you purchase your vSphere license determines how your license applies to your HyperFlex system

bull If you purchased your vSphere license with HyperFlex

Each HyperFlex server either has the Enterprise or Enterprise Plus edition preinstalled at the factory

bull HX Nodes have OEM licenses preinstalled If you delete or overwrite thecontent of the boot drives after receiving the HX servers you also delete thefactory-installed licenses

bull OEM license keys is a newVMware vCenter 60 U1b feature Earlier versionsdo not support OEM licenses

bull All factory-installed HX nodes share the same OEM license key WithvSphere OEM keys the Usage count can exceed the Capacity value

bull When you add an HX host to vCenter through the Add Host wizard in theAssign license section select the OEM license

We obfuscate the actual vSphere OEM license key for example0N085-XXXXX-XXXXX-XXXXX-10LHH

bull Standard Essentials Plus and ROBO editions are not available preinstalledon HX servers

Note

bull If you did NOT purchase your vSphere license with HyperFlex

The HX nodes have a vSphere Foundation license preinstalled After initial setup you can apply thelicense to a supported version of vSphere

bull If you purchased a vSphere PAC license

Follow the instructions in your PAC license letter from VMware to add the license to your MY VMwareaccount then follow the instructions to add your HX host to vCenter and assign the PAC license

Release Notes for Cisco HX Data Platform Release 4012

HyperFlex Licensing

HX Data Platform Software Versions for HyperFlex Witness Node

Witness Node VersionHyperFlex Release

10840(2b)

10840(2a)

10440(1b)

10440(1a)

Software Requirements for VMware ESXi

The software requirements include verification that you are using compatible versions of Cisco HyperFlexSystems (HX) components and VMware vSphere VMware vCenter and VMware ESXi

bull Verify that all HX servers have a compatible version of vSphere preinstalled

bull Verify that the vCenter version is the same or later than the ESXi version

bull Verify that the vCenter and ESXi versions are compatible by consulting the VMware ProductInteroperability Matrix Newer vCenter versions may be used with older ESXi versions so long as bothESXi and vCenter are supported in the table below

bull Verify that you have a vCenter administrator account with root-level privileges and the associatedpassword

For VIC1457 there is no support for ESXi 60Note

The below table applies for all of the following VMware vSphere Editions Enterprise Enterprise PlusStandard Essentials Plus ROBO

VMware vCenter VersionsVMware ESXi VersionsHyperFlex Version

60 U3 65 U3 67 U360 U3 65 U3 67 U340(2b)1

60 U3 65 U3 67 U360 U3 65 U3 67 U340(2a)2

60 U3 65 U3 67 U260 U3 65 U3 67 U2340(1b)

60 U3 65 U2 67 U260 U3 65 U2 67 U2440(1a)

1 Cisco HyperFlex Release 40(2) is the last major HyperFlex release that will support vSphere 60 (ESXiand vCenter) due to those versions reaching end of VMware general support on March 12 2020

2 Cisco HyperFlex Release 40(2) is the last major HyperFlex release that will support vSphere 60 (ESXiand vCenter) due to those versions reaching end of VMware general support on March 12 2020

3 Use of 67 U2 is not recommended see software advisory for CSCvq06952 and CSCvp88515 for furtherdetails

4 Use of 67 U2 is not recommended see software advisory for CSCvq06952 and CSCvp88515 for furtherdetails

Release Notes for Cisco HX Data Platform Release 4013

HX Data Platform Software Versions for HyperFlex Witness Node

Software Requirements for Microsoft Hyper-V

The software requirements include verification that you are using compatible versions of Cisco HyperFlexSystems (HX) components and Microsoft Hyper-V (Hyper-V) components

HyperFlex Software versions

The HX componentsmdashCisco HX Data Platform Installer Cisco HX Data Platform and Cisco UCSfirmwaremdashare installed on different servers Verify that each component on each server used with and withinthe HX Storage Cluster are compatible

bull Cisco HyperFlex M5 Converged nodesmdash For Hybrid (Cisco HyperFlex HX240c M5 HX220c M5)and All Flash (Cisco HyperFlex HXAF240cM5 HXAF220cM5) verify that Cisco UCSManager 40(2b)is installed HX 40 (1a) does not support Hyper-V on the All NVMe (HXAF220C-M5SN) nodes Fordetailed information on installation requirements and steps see theCisco HyperFlex Systems InstallationGuide on Microsoft Hyper-V

Table 4 Supported HyperFlex Software versions for M5 Servers on Hyper-V

M5 Recommended Server FirmwareHyperFlex Release

40(4h)40(2b)

40(4g)40(2a)

40(4e)40(1b)

40(4e)40(1a)

Release Notes for Cisco HX Data Platform Release 4014

Software Requirements for Microsoft Hyper-V

Table 5 Supported Microsoft Software versions

VersionMicrosoft Component

Windows Server 2016 Datacenter Core amp DesktopExperience

ForWindows Server 2016Datacenter Coreand Desktop Experience the Windows2016 ISO image should be Update BuildRevision (UBR) 1884 at a minimum

Note

Windows Server 2019Datacenter-DesktopExperienceis supported starting from HXDP 401(a) onwards

For Windows Server 2019 DesktopExperience theWindows 2019 ISO imageshould be Update Build Revision (UBR)107 at a minimum

Note

Windows Server 2019 DatacenterndashCore is notsupported currently

Also note that the following are currently notsupported

OEM activated ISOs and Retail ISOs are notsupported

Earlier versions ofWindows Server such asWindows2012r2 are not supported

Non-English versions of the ISO are not supported

Windows Operating System (Windows OS)

A Windows 2012 or later domain and forestfunctionality level

Active Directory

Supported Microsoft License Editions

The Microsoft Windows Server version that is installed on one or more HyperFlex hosts must be licensed asper Microsoft licensing requirements listed on Microsoft Licensing

Browser Recommendations

Use one of the following browsers to run the listed HyperFlex components These browsers have been testedand approved Other browsers might work but full functionality has not been tested and confirmed

Table 6 Supported Browsers

HX ConnectHX Data PlatformInstaller

Cisco UCS ManagerCisco IntersightBrowser

11 or higher11 or higher11 or higherNAMicrosoft InternetExplorer

70 or higher70 or higher57 or higher62 or higherGoogle Chrome

Release Notes for Cisco HX Data Platform Release 4015

Browser Recommendations

HX ConnectHX Data PlatformInstaller

Cisco UCS ManagerCisco IntersightBrowser

60 or higher60 or higher45 or higher57 or higherMozilla Firefox

NANA9 or higher10 or higherApple Safari

NANA35 or higherNAOpera

Notes

bull Cisco HyperFlex Connect

The minimum recommended resolution is 1024 X 768

bull Cisco HX Data Platform Plug-in

The Cisco HX Data Platform Plug-in runs in vSphere For VMware Host Client System browserrequirements see the VMware documentation at httpswwwvmwarecomsupportpubs

The HX Data Platform Plug-in is not displayed in the vCenter HTML client You must use the vCenterflash client

bull Cisco UCS Manager

The browser must support the following

bull Java Runtime Environment 16 or later

bull Adobe Flash Player 10 or higher is required for some features

For the latest browser information about Cisco UCSManager refer to themost recent Cisco UCSManagerGetting Started Guide

Cisco HX Data Platform Storage Cluster SpecificationsCluster Limits

bull Cisco HX Data Platform supports up to 100 clusters managed per vCenter as per VMware configurationmaximums

bull Cisco HX Data Platform supports any number of clusters on a single FI domain Each HX convergednode must be directly connected to a dedicated FI port on fabric A and fabric B without the use of a FEXC-series compute only nodes must also directly connect to both FIs B-series compute only nodes willconnect through a chassis IO module to both fabrics In the end the number of physical ports on the FIwill dictate the maximum cluster size and maximum number of individual clusters supported in a UCSdomain

The following table provides Cisco HX Data Platform Storage Cluster Specifications

Release Notes for Cisco HX Data Platform Release 4016

Cisco HX Data Platform Storage Cluster Specifications

Stretched Cluster (Availableon ESX Only)

Microsoft Hyper-VVMware ESXiNode

AllNVMe -HXAF220c-M5SN

HX240c-M5LHX220cM5

HX220cAF M5

HX240cM5

HX240cAF M5

HX240c-M5LHX220cM5

HX220cAF M5

HX240cM5

HX240cAF M5

HXAF220c-M5SN

AllNVMe -HXAF220c-M5SN

ESXi 65or 67only NotsupportedwithESXi 60orHyper-V

HX220cM5 Edge

HXAF220cM5 Edge

HX220cM4 Edge

HXAF220cM4 Edge

HX240c-M5LHX220cM5

HX220cAF M5

HX240cM5

HX240cAF M5

HX220cM4

HX220cAF M4

HX240cM4

HX240cAF M4

HXServers

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

C220M5C240M5B200M4B200M5

C240M5C220M5B200M4B200M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

mdashB200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

Compute-OnlyUCSB-SeriesC-SeriesServers

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

Convergednodesonly

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

SupportedNodes

Release Notes for Cisco HX Data Platform Release 4017

Cisco HX Data Platform Storage Cluster Specifications

Stretched Cluster (Availableon ESX Only)

Microsoft Hyper-VVMware ESXiNode

NA(requiresEnterpriseHXDP-PLicense)

NA(requiresEnterpriseHXDP-PLicense)

NA(requiresEnterpriseHXDP-PLicense)

Convergednodes3-16(12TBHDDoption isnotsupportedforHyperV)

Compute-onlynodes0-16

Convergednodes3-16

Compute-onlynodes0-16

NA(requiresEnterpriseHXDP-PLicense)

M4Convergednodes 3

M5Convergednodes23or 4

RequiresHXDP-ELicense

Convergednodes3-16(12TBdrive islimited toamaximumof 8nodes)

Compute-onlynodes0-16(12TBdrive islimited toamaximumof 8nodes)

Convergednodes3-32(76TBdriveconfigsarelimited tomaximumof 12drivesnodeon AF240)

Compute-onlynodes0-32

HXDP-SLicensedNodeLimits

11 ratioofHXDP-StoComputeonlynodes

(MinmdashMax)

Convergednodes2-16 perSite

Compute-onlynodes0-21 perSite

(up tomaxclustersize)

Convergednodes2-8 perSite

Compute-onlynodes0-16 perSite

(up tomaxclustersize)

Convergednodes2-16 perSite

Compute-onlynodes0-21 perSite

(up tomaxclustersize)

Convergednodes3-16(12TBHDDoption isnotsupportedforHyperV)

Compute-onlynodes0-16

Convergednodes3-16

Compute-onlynodes0-16

Convergednodes3-32

ComputeOnlynodes0-32

(up tomaxclustersize)

Convergednodes 3

(requiresHXDP-ELicense)

Convergednodes3-16(12TBdrive islimited toamaximumof 8nodes)

Compute-onlynodes0-32(12TBdrive islimited toamaximumof 16nodes)

Convergednodes3-32(76TBdriveconfigsarelimited tomaximumof 12drivesnodeon AF240)

Compute-onlynodes0-32

(up tomaxclustersize)

HXDP-PLicensedNodeLimits

12 ratioofHXDP-PtoComputeonlynodes

(MinmdashMax)

24 perSite 64percluster

24 perSite 48percluster

32 perSite 64percluster

32326434864MaxClusterSize

Release Notes for Cisco HX Data Platform Release 4018

Cisco HX Data Platform Storage Cluster Specifications

Stretched Cluster (Availableon ESX Only)

Microsoft Hyper-VVMware ESXiNode

212121111121mdash2121MaxComputetoConvergedratio

NoExpansion

Requires Enterprise license

Requires uniform expansion across both sites

Guidelines and Limitationsbull Starting with release 40(2a) SCVM is no longer needed on a Compute node

bull HX REST API Access Token Management ndash Applications leveraging HX REST APIs should re-useaccess tokens when making API calls Once obtained using the AAA Obtain Access Token API accesstokens are valid for 18 days (1555200 seconds) In addition AAA enforces rate limiting on ObtainAccess Token API requests in a 15 minute window auth can be invoked (successfully) a maximum of5 times A user is allowed to create a maximum of 8 unrevoked tokens Subsequent call to auth willautomatically revoke the oldest issued token tomake room for the new token Amaximum of 16 unrevokedtokens can be present in system In order to prevent brute-force attacks after 10 consecutive failedauthentication attempts a user account is locked for a period of 120 secondsFor more information seeCisco HyperFlex Systems REST API Reference guide

HxConnect makes use of AAA Authentication REST API for login and the above rate limit applies toHxConnect also

Upgrade Guidelines

The following list is a highlight of critical criteria for performing an upgrade of your HyperFlex system

bull Unsupported HX Data Platform 17x 18x 20 21x 25x and 26x clustersmdashUsers from anyversion prior to 26(1a) must step through an intermediate version before upgrading to 40 or later releasesIf you need to upgrade your environment from a Cisco HyperFlex HX Data Platform software releasethat is past the last date of support to the latest suggested release on the Cisco Software Download sitesee Cisco HyperFlex Systems Upgrade Guide for Unsupported Cisco HXReleases For more informationsee the Software Advisory for CSCvq66867 WARNING Only Use HXDP 26(1e) Upgrade PackageWhen Upgrading From HXDP 18(1a)-18(1e)

bull Hypercheck Health Check UtilitymdashCisco recommends running this proactive health check utility onyour HyperFlex cluster prior to upgrade These checks provide early visibility into any areas that mayneed attention andwill help ensure a seamless upgrade experience For more information see the HyperFlexHealth amp Pre-Upgrade Check Tool TechNote for full instructions on how to install and run Hypercheck

bull vSphere 67 Software AdvisorymdashDo not upgrade to Cisco HX Data Platform Release 40(1a) whenrunning ESXi 67U1 EP06 (build 11675023) Do not upgrade to 67U1 EP06 (build 11675023) ifrunning Cisco HX Data Platform Release 40(1a) See the Software Advisory CSCvo56350 for furtherdetails

Release Notes for Cisco HX Data Platform Release 4019

Guidelines and Limitations

The software build version posted at release will override any other local versions

bull Required vCenter upgrademdashFor enhanced security Cisco HX Data Platform Release 35(1a) or laterrequires the use of TLS 12 Therefore vCenter must be upgraded to 60 U3f or later before upgradingto Cisco HX Data Platform Release 35 or later In addition ESXi should be upgraded as required tomeet HX Data Platform compatibility requirements

bull Minimum HXDP version for upgrademdashHXData Platform clusters running 26(1a) or later may upgradedirectly to 40 using the HX Connect UI

bull Cluster ReadinessmdashEnsure that the cluster is properly bootstrapped and the updated plug-in is loadedbefore proceeding Manual cluster bootstrap is required for upgrade from a pre-35 release

bull Cluster ReadinessmdashEnsure that the cluster is properly bootstrapped and the updated plug-in is loadedbefore proceeding Manual cluster bootstrap is required for HX releases earlier than 35(1a) For moreinformation see theManual Bootstrap Upgrade Process in the Cisco HyperFlex Systems Upgrade Guidefor VMware ESXi Release 40 Do not skip this cluster bootstrap step it is required for all upgradesuntil HX Release 35(1a) Auto bootstrap is supported beginning with HX release 35(1a) For moreinformation see the Auto Bootstrap Upgrade Process from HX Connect UI in the Cisco HyperFlexSystems Upgrade Guide for VMware ESXi Release 40

Manual bootstrap is not supported on Intersight clusters

bull Initiating Upgrade―Use the HX Connect UI or CLI stcli commands when upgrading from 25(1a)or later releases Use either the CLI stcli commands or the HX Data Platform Plug-in to the vSphereWeb Client when upgrading from a pre-25(1a) release The vCenter plug-in should not be used forupgrades starting with the 25(1a) release The vCenter plug-in should not be used for upgrades startingwith the 25(1a) release

If the current cluster version is at 35(1a) or above you do not need to use the stcli command Directupgrade to 40 is possible

bull Complete your Upgrade―The self-healing (or rebalance) capability is disabled temporarily during theupgrade window If the upgrade fails you should complete the upgrade as soon as possible

bull ESXi and HXDP Compatibility―Ensure your cluster is running a compatible version of ESXi basedon the running the HXData Platform version (see the section Software Requirements for VMware ESXi)ESXi compatibility is determined by the major version and update release of ESXi It is generally bestto upgrade HXDP and ESXi together if combining the upgrade operations into a single optimized rebootWhen running a split upgrade first upgrade the HX Data Platform then proceed to upgrade ESXi

bull If Upgrading to vSphere 65

bull Certain cluster functions such as native and scheduled snapshotsReadyClones and Enter or Exit HX Maintenance Mode will not operatefrom the time the upgrade is started until the HX Data Platform upgrade iscomplete

bull After upgrading ESXi using the offline zip bundle use the ESX ExitMaintenance Mode option The HX Exit Maintenance Mode option doesnot operate in the vSphere Web Client until the HX Data Platform upgradeis complete

Note

Release Notes for Cisco HX Data Platform Release 4020

Guidelines and Limitations

bull vSphere 60 UpgradesmdashUsers on vSphere 60 migrating to 65 upgrade components in the followingorder

1 Upgrade HX Data Platform and UCS firmware

2 Upgrade HX Data Platform and ESXi

3 Upgrade HX Data Platform only first then upgrade ESXi or UCS firmware or both

bull M4 Server Firmware UpgradesmdashUpgrade server firmware to ensure smooth operation and to correctknown issues Specifically newer SAS HBA firmware is available in this release and is recommendedfor long-term stability

bull Users are encouraged to upgrade to 31(3c) C-bundle or later whenever possible

bull Users running C-bundle versions before 31(2f) must upgrade server firmware by performing acombined upgrade of UCS server firmware (C-bundle) to 31(3c) or later and HX Data Platform to25 Do not split the upgrade into two separate operations

bull If the cluster is already on 31(2f) C-bundle or later you may perform an HX Data Platform onlyor combined upgrade as required

bull M5 Server Firmware UpgradesmdashM5 generation servers must run firmware version 32(2d) or later

bull Firmware Downgrades mdashDowngrading UCSM from the HX-installer is not supported

bull M4M5 Mixed DomainsmdashA mixed domain occurs when a new separate M5 cluster is installed underthe same UCS domain that contains existing M4 clusters Under these conditions orchestrated UCSserver firmware upgrade will not operate until Cisco HX Data Platform Release 26 or later is installedon the M4 clusters Therefore it is best practice to first upgrade UCS server firmware to the latest 31(3)or 32(2) patch release before adding a new M5 cluster to the existing UCS domain Additionally any17 HX Data Platform clusters must first be upgraded before adding any new M5 clusters to the samedomain

bull Maintenance WindowmdashIf upgrading both HX Data Platform and UCS firmware you can select eithera combined or split upgrade through the vSphere HX Data Platform Plug-in depending on the length ofthe maintenance window Cisco UCSManager infrastructure upgrade is only supported using AutoInstalland the direct server firmware upgrade should be performed only through the upgrade orchestrationframework provided by the HX Data Platform Plug-in

bull Unsupported Self-Encrypting Drives (SEDs)mdashIf adding or replacing self-encrypting drives (SEDs)that have been recently qualified in newer versions of HX Data Platform insert the new drives only afterupgrading HX Data Platform to a compatible version All drives must be SED drives mixing SED andnon-SED is not supported

bullbull Enabling External Host AccessmdashWith Cisco HX Data Platform Release 40(1a) port 445 on themanagement network is blocked for enhanced security Note that prior to 40 port 445 port was openenabling external host access If you are upgrading to 40(1a) from a prior release and would like tocontinue external host access you can use a utility to open select hosts For more information aboutenabling external host access see the ConfiguringHyperFlex Share to SCVMM section in the InstallationGuide for Microsoft Hyper-V

Release Notes for Cisco HX Data Platform Release 4021

Guidelines and Limitations

Mixed Cluster Expansion Guidelinesbull Expanding existing M4 cluster with M5 converged nodes is supported

bull Expanding existing M5 cluster with M4 converged nodes is not supported

bull Expanding existing mixed M4M5 cluster with M4 or M5 converged nodes is supported

bull Adding any supported compute-only nodes is permitted with all M4 M5 and mixed M4M5 clustersusing the HX Data Platform 26 or later Installer Some example combinations are listed here manyother combinations are possibleExample combinationsExpand mixed M4M5 cluster with compute-only B200 C220 C240 M4M5Expand M4 cluster with compute-only B200 M5 C220 M5 C240M5

bull Only expansion workflow is supported to create a mixed cluster Initial cluster creation with mixedM4M5 servers is not supported

bull All M5 servers must match the form factor (220240) type (HybridAF) security capability (Non-SEDonly) amp disk configuration (QTY capacity and non-SED) of the existing M4 servers

bull HX220-M5 will use a maximum of 6 capacity disks (2 disk slots to remain empty) when mixedwith HX220-M4

bull HX Edge SED LFF Hyper-V and Stretched Clusters do not support mixed M4 and M5 clusters

Security FixesThe following security issues are resolved

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-13132 CVE-2019-9924CSCvq6313840(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-11719 CVE-2019-11727CVE-2019-11729

CSCvq7124040(2a)

Release Notes for Cisco HX Data Platform Release 4022

Mixed Cluster Expansion Guidelines

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-1125CSCvr0633940(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2015-9383 CVE-2018-14498CVE-2018-20406 CVE-2018-20852CVE-2019-10160 CVE-2019-13117CVE-2019-13118 CVE-2019-14287CVE-2019-14973 CVE-2019-15903CVE-2019-17546 CVE-2019-18197CVE-2019-18218 CVE-2019-5010CVE-2019-5094 CVE-2019-5481CVE-2019-5482 CVE-2019-9636CVE-2019-9740 CVE-2019-9947CVE-2019-9948

CSCvs0609440(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2017-13168 CVE-2017-18174CVE-2017-18216 CVE-2018-10876CVE-2018-10877 CVE-2018-10878CVE-2018-10879 CVE-2018-10880CVE-2018-10881 CVE-2018-10882CVE-2018-10902 CVE-2018-10938CVE-2018-12233 CVE-2018-12896CVE-2018-13053 CVE-2018-13094CVE-2018-13096 CVE-2018-13405CVE-2018-13406 CVE-2018-14609CVE-2018-14617 CVE-2018-14633CVE-2018-14734 CVE-2018-15572CVE-2018-15594 CVE-2018-16276CVE-2018-16658 CVE-2018-17182CVE-2018-17972 CVE-2018-18021CVE-2018-18690 CVE-2018-18710CVE-2018-6554 CVE-2018-6555CVE-2018-9363

CSCvp6501940(2a)

This vulnerability is due to insufficientHTML iframe protection An attackercould exploit this vulnerability bydirecting a user to an attacker-controlledweb page that contains amaliciousHTMLiframe A successful exploit could allowthe attacker to conduct clickjacking orother clientside browser attacks

NACSCvo9851640(2a)

Release Notes for Cisco HX Data Platform Release 4023

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerability is due to insufficientauthentication for the statistics collectionservice An attacker could exploit thisvulnerability by sending properlyformatted data values to the statisticscollection service of an affected deviceA successful exploit could allow theattacker to cause the web interfacestatistics view to present invalid data tousers

NACSCvj9558440(2a)

The vulnerability is due to insufficientCSRF protections for the web UI on anaffected device An attacker could exploitthis vulnerability by persuading a user ofthe interface to follow a malicious linkA successful exploit could allow theattacker to perform arbitrary actions withthe privilege level of the affected user

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvp2434340(2a)

Vulnerabilities with a version of cJSONidentified by CVE-2019-11834 andCVE-2019-11835

For more information see the relatedCisco Security Advisory

CVE-2019-11834 CVE-2019-11835CSCvq1994940(2a)

HX ESXi image patches for MCEPSCand TAA vulnerabilities with VMwareESXi identified by CVE-2018-12207 andCVE-20190-11135

For more information see the relatedCisco Security Advisory

CVE-2018-12207 CVE-2019-11135CSCvr5439840(2a)

CP networking vulnerabilities affectingthe Linux kernel identified byCVE-2019-11477 CVE-2019-11478CVE-2019-11479

For more information see the relatedCisco Security Advisory

CVE-2019-11477 CVE-2019-11478CVE-2019-11479

CSCvq1954640(2a)

Qualification of Microsoft securitypatches for vulnerabilities withMicrosoftHyper-V hypervisor identified withCVE-2018-12207 CVE-2019-11135

For more information see the relatedCisco Security Advisory

CVE-2018-12207 CVE-2019-11135CSCvr5439940(2a)

Release Notes for Cisco HX Data Platform Release 4024

Security Fixes

DescriptionCVEDefect IDRelease

OpenSSH vulnerabilities identified byCVE-2019-6111

For more information see the relatedCisco Security Advisory

CVE-2016-10708 CVE-2018-15473CVE-2018-20685 CVE-2019-6109CVE-2019-6111

CSCvp7646340(2a)

Multiple vulnerabilities associated withOpenSSL and LibNSS

For more information see the relatedCisco Security Advisory

CVE-2016-2105 CVE-2016-2106CVE-2016-2107 CVE-2016-2109CVE-2016-2176 CVE-2016-2177CVE-2016-2178 CVE-2016-2179CVE-2016-2180 CVE-2016-2181CVE-2016-2182 CVE-2016-2183CVE-2016-6302 CVE-2016-6303CVE-2016-6304 CVE-2016-6305CVE-2016-6306 CVE-2016-6307CVE-2016-6308 CVE-2016-7055CVE-2016-8610 CVE-2017-3731CVE-2017-3732 CVE-2017-3735CVE-2017-3736 CVE-2017-3737CVE-2017-3738 CVE-2018-0495CVE-2018-0732 CVE-2018-0734CVE-2018-0735 CVE-2018-0737CVE-2018-0739 CVE-2018-12384CVE-2018-12404 CVE-2018-5407CVE-2019-1559

CSCvp6667940(2a)

libpng vulnerability identified byCVE-2016-10087

For more information see the relatedCisco Security Advisory

CVE-2016-10087CSCvp6655540(2a)

Vulnerabilities associated with libjpg andlibpng

For more information see the relatedCisco Security Advisory

CVE-2014-9092 CVE-2016-3616CVE-2017-15232 CVE-2018-11212CVE-2018-11213 CVE-2018-11214CVE-2018-1152 CVE-2018-13785

CSCvp3458640(2a)

Vulnerabilities associated with Glibidentified by CVE-2018-16428 andCVE-2018-16429

For more information see the relatedCisco Security Advisory

CVE-2018-16428 CVE-2018-16429CSCvp3120740(2a)

Multiple vulnerabilities associated withcurl expat python 27 python 35 36and 37 freetype and giflib

For more information see the relatedCisco Security Advisory

CVE-2019-15133 CVE-2019-15903CVE-2019-5010 CVE-2019-5481CVE-2019-5482 CVE-2019-9636CVE-2019-9740 CVE-2019-9947CVE-2019-9948

CSCvr3690340(2a)

Release Notes for Cisco HX Data Platform Release 4025

Security Fixes

DescriptionCVEDefect IDRelease

Multiple third party softwarevulnerabilities

For more information see the relatedCisco Security Advisory

CVE-2019-14379 CVE-2019-12384CVE-2019-14439

CSCvq9203240(2a)

Vulnerabilities associated with elfutils

For more information see the relatedCisco Security Advisory

CVE-2018-16062 CVE-2018-16402CVE-2018-16403 CVE-2018-18310CVE-2018-18520 CVE-2018-18521CVE-2019-7149 CVE-2019-7150CVE-2019-7665

CSCvq4325040(2a)

Vulnerabilities associated with vim

For more information see the relatedCisco Security Advisory

CVE-2017-5953 CVE-2019-12735CSCvq4323040(2a)

Vulnerabilities associated with sqlite3

For more information see the relatedCisco Security Advisory

CVE-2016-6153 CVE-2017-10989CVE-2017-13685 CVE-2017-2518CVE-2017-2519 CVE-2017-2520CVE-2018-20346 CVE-2018-20505CVE-2018-20506 CVE-2019-8457CVE-2019-9936 CVE-2019-9937

CSCvq4321340(2a)

Vulnerabilities associated with glib20

For more information see the relatedCisco Security Advisory

CVE-2018-20843CSCvq4320940(2a)

Vulnerabilities associated with expat

For more information see the relatedCisco Security Advisory

CVE-2018-20843CSCvq4320540(2a)

Vulnerabilities associated with bzip2

For more information see the relatedCisco Security Advisory

CVE-2016-3189 CVE-2019-12900CSCvq4319440(2a)

Mulitple Vulnerabilities associated wtihNodeJS

For more information see the relatedCisco Security Advisory

CVE-2018-12115 CVE-2018-0734CVE-2018-5407 CVE-2018-12120CVE-2018-12121 CVE-2018-12122CVE-2018-12123 CVE-2018-12116CVE-2019-5737 CVE-2019-5739

CSCvq1069440(2a)

Vulnerabilities associated with PalletsJinja strformat_map and identified byCVE-2019-10906 and CVE-2016-10745

For more information see the relatedCisco Security Advisory

CVE-2019-10906 CVE-2016-10745CSCvq1038840(2a)

Release Notes for Cisco HX Data Platform Release 4026

Security Fixes

DescriptionCVEDefect IDRelease

Multiple Security Vulnerabilities

For more information see the relatedCisco Security Advisory

CVE-2014-8501 CVE-2014-9939CVE-2015-9262 CVE-2016-10087CVE-2016-2226 CVE-2016-4487CVE-2016-4488 CVE-2016-4489CVE-2016-4490 CVE-2016-4491CVE-2016-4492 CVE-2016-4493CVE-2016-6131 CVE-2018-10963CVE-2018-13785 CVE-2018-17100CVE-2018-17101 CVE-2018-18557CVE-2018-18661 CVE-2018-7456CVE-2018-8905

CSCvq0917840(2a)

Vulnerabilities associated withlibseccomp

For more information see the relatedCisco Security Advisory

CVE-2019-9893CSCvq0756840(2a)

Vulnerabilities associated withgdk-pixbuf

For more information see the relatedCisco Security Advisory

CVE-2017-12447CSCvq0675540(2a)

Vulnerabilities associated withpython-crypto

For more information see the relatedCisco Security Advisory

CVE-2018-6594CSCvp9381740(2a)

Vulnerabilities associated with wget

For more information see the relatedCisco Security Advisory

CVE-2018-20483 CVE-2019-5953CSCvp8672140(2a)

Multiple vulnerabilities associated withPython Crypto

For more information see the relatedCisco Security Advisory

CVE-2018-6594CSCvp6674840(2a)

Vulnerabilities associated withsensible-utils

For more information see the relatedCisco Security Advisory

CVE-2017-17512CSCvp6673440(2a)

Multiple vulnerabilities associated withPython

For more information see the relatedCisco Security Advisory

CVE-2018-1000030CSCvp6668940(2a)

Release Notes for Cisco HX Data Platform Release 4027

Security Fixes

DescriptionCVEDefect IDRelease

Multiple patch vulnerabilities identifiedby CVE-2016-10713

For more information see the relatedCisco Security Advisory

CVE-2016-10713CVE-2018-1000156 CVE-2018-6951

CSCvp6667240(2a)

Vulnerabilities associatedwith Little CMSidentified by CVE-2016-10165

For more information see the relatedCisco Security Advisory

CVE-2016-10165 CVE-2018-16435CSCvp6666440(2a)

Vulnerabilities associated with TenableScan identified byCVE-2011-5325CVE-2019-11068CVE-2016-7076CVE-2017-1000368

For more information see the relatedCisco Security Advisory

CVE-2011-5325 CVE-2016-7076CVE-2017-1000368CVE-2019-11068

CSCvp6474640(2a)

Vulnerabilities associated with libxml2294 and earlier as used in XMLSec1223 and earlier and other products andidentified by CVE-2016-9318

For more information see the relatedCisco Security Advisory

CVE-2016-9318 CVE-2017-16932CVE-2017-18258 CVE-2018-14404CVE-2018-14567

CSCvp3479240(2a)

Vulnerabilities associated with LFTPRemote File Names Unauthorized AccessVulnerability and identified byCVE-2018-10916

For more information see the relatedCisco Security Advisory

CVE-2018-10916CSCvp2926640(2a)

Vulnerabilities associated withCryptographyDeprecationWarning signerand verifier have been deprecated

For more information see the relatedCisco Security Advisory

CVE-2018-7750CSCvo3409740(2a)

Multiple Vulnerabilities in python python27 35

For more information see the relatedCisco Security Advisory

CVE-2019-16056CSCvr3979340(2a)

Release Notes for Cisco HX Data Platform Release 4028

Security Fixes

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toexecute commands as the root user

The vulnerability is due to insufficientinput validation An attacker could exploitthis vulnerability by connecting to thecluster service manager and injectingcommands into the bound process Asuccessful exploit could allow the attackerto run commands on the affected host asthe root user

Cisco has released software updates thataddress this vulnerability There areworkarounds that address thisvulnerability

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvq2417640(1b)35(2g)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

The vulnerability is due to an unprotectedlistening interface An attacker couldexploit this vulnerability by connectingto the listening interface and injectingcommands to the bound process Anexploit could allow the attacker to runcommands on the affected host as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvj9560640(1b)35(2g)

Release Notes for Cisco HX Data Platform Release 4029

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerabilities associated with theJVM18U121memory leak found duringVC alarms (concurrent 40 calls usingREST API)

CVE-2017-10053 CVE-2017-10067CVE-2017-10074 CVE-2017-10078CVE-2017-10081 CVE-2017-10087CVE-2017-10089 CVE-2017-10090CVE-2017-10096 CVE-2017-10101CVE-2017-10102 CVE-2017-10107CVE-2017-10108 CVE-2017-10109CVE-2017-10110 CVE-2017-10111CVE-2017-10115 CVE-2017-10116CVE-2017-10118 CVE-2017-10135CVE-2017-10176 CVE-2017-10193CVE-2017-10198 CVE-2017-10243CVE-2017-10274 CVE-2017-10281CVE-2017-10285 CVE-2017-10295CVE-2017-10345 CVE-2017-10346CVE-2017-10347 CVE-2017-10348CVE-2017-10349 CVE-2017-10350CVE-2017-10355 CVE-2017-10356CVE-2017-10357 CVE-2017-10388CVE-2017-3509 CVE-2017-3511CVE-2017-3526 CVE-2017-3533CVE-2017-3539 CVE-2017-3544CVE-2018-2579 CVE-2018-2582CVE-2018-2588 CVE-2018-2599CVE-2018-2602 CVE-2018-2603CVE-2018-2618 CVE-2018-2629CVE-2018-2633 CVE-2018-2634CVE-2018-2637 CVE-2018-2641CVE-2018-2663 CVE-2018-2677CVE-2018-2678 CVE-2018-2783CVE-2018-2790 CVE-2018-2794CVE-2018-2795 CVE-2018-2796CVE-2018-2797 CVE-2018-2798CVE-2018-2799 CVE-2018-2800CVE-2018-2814 CVE-2018-2815CVE-2018-2952 CVE-2018-3136CVE-2018-3139 CVE-2018-3149CVE-2018-3150 CVE-2018-3169CVE-2018-3180 CVE-2018-3183CVE-2018-3214 CVE-2019-2422

CSCvo8899740(1b)

Tomcat and Nginx logs are not beingcollected in the support bundle generatedthrough HX Connect in Release 35

NACSCvm5803140(1b)

The vulnerabilities associated with thelibmspack software package versionincluded in Cisco HX Data Platform

CVE-2018-18584

CVE-2018-18585

CSCvn3511940(1a)

Release Notes for Cisco HX Data Platform Release 4030

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerabilities associated withFasterXML Jackson-Databind TimeValueField Denial of Service

CVE-2018-14719

CVE-2018-14720

CVE-2018-1000873

CVE-2018-14721

CVE-2018-19360

CVE-2018-19362

CVE-2018-19361

CVE-2018-14718

CSCvn8228240(1a)

The vulnerabilities associated with theOpenSSL Protocol software packageversion included in Cisco HX DataPlatform

CVE-2013-3587CSCvo0505440(1a)

The vulnerabilities associated withThird-Party Software Denial of Service

CVE-2018-16487

CVE-2018-19361

CSCvo2781840(1a)

The vulnerabilities associated withLinuxkernel for Ubuntu 1710

CVE-2018-1092

CVE-2018-7492

CVE-2018-8087

CVE-2018-1068

CVE-2018-8781

CSCvm5314935(2a)

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)25(1d)stcli services dns remove should remove the DNS serverinfo from the interface files

CSCvs70967

40(2b)40(1b)HX 40(1b) new deployment from Intersight productioncloud

User receives following error while enabling CSI(Kubernetes integration)

Failure occurred during volume_access_enable Errorwas Nonzero exit code 1

CSCvt10522

40(2b)40(2a)Receive the following alertevent in HX Connect HXController VM HOSTNAME one or more configuredDNS servers not responding

CSCvt13947

Release Notes for Cisco HX Data Platform Release 4031

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)40(2a)API UI CLI show few Drive slots Empty after upgradeto 402a

The drives are seen in the sysmtool lsscsi and stclicommand outputs

The cluster is healthy with no errors

CSCvt14914

40(2b)35(2d)After successfully changing the DNS server on the HXcontroller we still can see the original DNS entry thatwas added during the deployment

CSCvs69154

40(2b)35(2e)HyperFlex Connect and VCenter show APD alarms forHX and non HX datastores This implies there is an issuewith HyperFlex when there may not be

CSCvs30080

40(2b)40(2a)An issue where all the disks in the node get locked afterrebooting the node

We successfully unlocked using sed-clientsh -Ucommand but wanted to test with another reboot anddrives locked again

CSCvs74286

40(2b)40(2a)When running stcli license commands on HyperFlexthe following error is seen

rootSpringpathControllertmp stcli license show all

Show smart licensing failed

Smart Agent is not ready please wait a minute and tryagain

CSCvt13929

40(2b)40(2a)After upgrading to HX 40(2a) the first RoboEdge nodethat gets rebooted then ignores persistent drivesPIDHX-HD24TB10K4KN

Cluster will be in degraded state and have reducedcapacity

CSCvt20144

40(2b)35(2b)Zookeeper fails to start while Exhibitor is runninghowever echo srvr returns nothing

CSCvs21562

40(2b)40(2a)When performing an HyperFlex upgrade a validationwarning may occur due to the host not having EnterprisePlus or Enterprise hypervisor licensing

Upgrade Validation Warning

ESXi host esx1labtest should be configured withVMware Enterprise license for upgrade to continue

CSCvs91787

Release Notes for Cisco HX Data Platform Release 4032

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)35(2g)Cluster expansion fails at Config Installer stage whenthe root and admin password for the storage controller(SCVM) are different

CSCvs69317

40(2b)35(2g)Panic while upgrading ESXiCSCvt06983

40(2b)40(2a)A cluster node running may hang in the Linux kernelThis is classified as an oops and a deviation from theexpected behavior

CSCvs54285

40(2b)35(2g)Rebalance failing on 10+10 node HX 35(2g) stretchcluster

CSCvs69007

40(2b)35(2d)The cluster becomes inaccessible to the IOVisorCSCvr54687

40(2b)40(2a)Panic on the storage controllerCSCvt61297

40(2b)40(2a)The size of support bundle is very large when collectedwith storfs-support command

CSCvt63306

40(2b)40(2a)storfs crash associated withFileSystemUsageWarningEvent orFileSystemUsageAlertEvent due to high usage ofvarstv above 80

CSCvt72807

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

40(2a)35(2d)In normal scenarios HX upgrade gets pendingACK afterupdating service profile In this case UCS only upgradewas stalled because after updating service profile HXupgrade was not getting pending ACK and it was alwayswaiting state

CSCvo39912

40(2a)35(1a)When using motherBoardReplace-12 to clean ZK fromold stNodepNodes resulted in unmounted datastore andmaking the size of the Hyperflex Datastores 0 resultingin all VMs in the cluster going offline

CSCvq39471

40(2a)26(1e)Upgrading cluster and getting error

Failed upgrade validations Checking vCenterconfiguration Reason Upgrade validations failed DRSFault Insufficient resources to satisfy configured failover

CSCvm77294

Release Notes for Cisco HX Data Platform Release 4033

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)The cluster expand fails on a node with DR replicationconfigured

When a HX cluster which has DR replication configuredis expanded we see the installer UI pulling in thereplicationVLAN information instead of themanagementVLAN information Even if we change that informationto the correct mgmt VLAN id and name it it does notseem to work as the node is configured with the VLANof the replicationVLAN in ESXi This leads to the failureof the node add with the host unreachable error

CSCvo70650

40(2a)35(1a)Customer reported that server didnt complete thefirmware upgrade automatically one by one

The user manually needs to put the host into maintenancemode and they manually acknowledged the pendingrequests to finish the UCS firmware upgrade

Maintenance policy is set to default (User-ack) as perthe design

CSCvo91624

40(2a)35(1i)When the cluster is in Failed state and stcli noderemove is attempted the output shows successful thoughthe node failed to get removed from the cluster

CSCvo93017

40(2a)35(1a)HyperFlex cluster upgrade may fail during validationwith a DRS Validation failed error

CSCvp31021

40(2a)35(2b)HX cluster expansion will fail with the following errormessage

MAC address pool configuration failure150[ErrorDescription] bad address block range definitioncollision

CSCvp58318

40(2a)35(2b)Memory usage by carbon cacheCSCvq34873

40(2a)35(2g)Upgrading ESXi cluster fails with error Nodemaintenance mode failed

CSCvr03240

40(2a)35(2e)storfs process does not automatically start on Exit HXMaintenance Mode or other tasks which restart StorageController

CSCvr88978

40(2a)35(2a)Post install script fails with message

Failed to execute ipmitool on HX node

CSCvp10707

40(2a)40(1a)HyperFlex expansionworkflow doesnt pull VLANnamecorrectly

CSCvp46539

Release Notes for Cisco HX Data Platform Release 4034

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)During HX cluster deployment - Validate ClusterCreation phase - HX Installer might fail with the error

fromvarlogspringpathstDeploylog

CSCvq45087

40(2a)35(2b)HX Installer fails to login to the SOL for configuringESX hosts

CSCvq91380

40(2a)35(2d)If DRS is enabled with configuration parameters otherthen fully automated ie manual or partially automatedthan this will change to fully automated during clusterexpansion

CSCvr44222

40(2a)35(1a)One of the validation task name has a typo The taskname is Validating node for mixed of SED and non-SEDdisks

It should be Validating node for mix of SED andnon-SED disks

CSCvp82175

40(2a)30(1i)This combination of using smaller NVMe drives is notsupported in expand operation For example if the drivesare only 375GB and they cannot be added to an existingcluster with larger caching SSD

CSCvo12359

40(2a)35(1a)HyperFlex includes a version of OpenSSL that is affectedby the vulnerabilities identified by the followingCommon Vulnerability and Exposures

CVE-2018-0495

CSCvp66679

40(2a)30(1e)SDK invoke exception when taking a native snapshot ona VM with GPU pci passthrough attached

CSCvo00511

40(2a)30(1i)Node replacement script fails due to EAM errorCSCvo62867

40(2a)35(1a)While pairing a cluster (HX release gt= 35) and cluster(HX release lt 35) remote replication network test failson the cluster (HX release lt 35)

CSCvo79760

40(2a)30(1b)Fixed the issue in the latest support-workflow bundleThis should not be hit anymore

CSCvo87061

40(2a)35(2a)On a 3 node cluster the MBreplace script fails as thescript looks for HEALTHY cluster status

CSCvo87080

40(2a)35(2a)MbReplace script (tar) hangs when running on 352b or40

CSCvp12359

Release Notes for Cisco HX Data Platform Release 4035

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)HX replication cleanup failing with error NA 35(2a) -INFODR state is not clean

CSCvp63958

40(2a)40(1a)The check_stig_parameters API incorrectly shows thecompliance state of the cluster as not STIG compliant

CSCvp66277

40(2a)35(2c)The snapshot creation on CBT enabled VM fails witherror Failed in vmreparent vmkfstools clone1

CSCvq06952

40(2a)35(2a)HyperFlex GUI when setting up remote DR-replicationpartnerpeer should show remote replication partner TCPport reachability test results

This will provide quick upfront connectivity results thatfrequently takes customer a while to manually check andconfirm

If reachability tests fail an info window and HX alertshould trigger that list the destination peer IP and thespecific port(s) that could not be reache

CSCvo60587

40(2a)35(2b)If you have more than one IP pool configured for areplication network and local replication network testsare failing you might be running into this issue

CSCvr67130

40(2a)35(2a)Skip task option in MBreplace and Software redeployscripts The scripts should have the option to skip tasksas user inputs and not by modifying the json file

CSCvp05204

40(2a)40(1a)Error noticed on SCVM Console print_req_error IOerror dev fd0 sector 0 on HyperV

It doesnt come up on SSH session

CSCvq34357

40(2a)30(1d)The VMNetwork switch is already created as part of theinstall process

Ideally the ip address assignment should be done duringthe initial install process as well No post install stepsshould be required altrnatively it could be done as partof a post install script

CSCvk36222

40(2a)25(1c)storfs service impacted during HX upgrade from HX21(1) to 25(1c)

HX node which is not being upgraded storfs servicestopped causing outage

CSCvg53223

40(2a)35(1b)HX Stretch Cluster Witness VM reverts to DHCP onreboot

CSCvp37536

40(2a)35(2b)HyperFlex Stretch Cluster Witness does not getprogrammed with an NTP server

CSCvp46578

Release Notes for Cisco HX Data Platform Release 4036

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2b)HyperFlex cluster does not heal after maintenance onnodeserver

CSCvr18528

40(2a)40(2a)High percentage of packet loss on the eth1 data interface9K MTU

CSCvr97089

40(2a)35(2f)The HyperFlex cluster healing state is stuck at 87 andis unable to progress

CSCvr16760

40(2a)40(2a)In some circumstances two replications can occur forthe same VM

From the GUI the replication status for this VM willshow stuck in an In Progress state but there are nojobs stuck in replication layer Subsequent Replicationsshould succeed

CSCvq22898

40(2a)35(2a)After an unexpected power outage the hyperflex clusteris unable to come up correctly

All processes are running time is synced and all nodescan vmkping each other as required

CSCvp13990

40(2a)40(1a)When specifying IP address instead of node ID stclinode maintenanceMode commands fail

CSCvp33657

40(2a)40(1a)2 node edge cluster if experiencing Intersightarbitratorconnectivity issues should NOT allowHXMaintenancemode to initiate as this will bring the cluster down

CSCvr31573

40(2a)35(2c)Old files in varsupportZKTxnlog are not purged withthe daily zklog-cleanup cron job

CSCvr89066

40(2a)35(2a)When a node is in MaintenanceMode any disk removalor replacement will be reflected in UI only after the nodeis brought back frommaintenancemode This is becausestorfs is not running on the node in maintenance andwill not be able to detect disk activities until it is broughtout of Maintenance Mode

CSCvo86431

40(2a)35(2c)Cluster might go offline and stop serving IO requestsrootcvm~ stcli cluster storage-summary--detailGet cluster storage summary failedjavanetConnectException Connection refused19216814210010207

CSCvr01645

40(2a)35(2d)A stretch cluster deployment fails with a messageFormatting nodes Node in Use on the Hyperflex installerand crmZoneType value is seen as 1 in storfscfg oroptspringpathconfigstretchtunes

CSCvq80340

Release Notes for Cisco HX Data Platform Release 4037

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)30(1c)During upgrade we upload the upgrade packagedownloaded from Ciscocom Upgrade packages are offile type tgz hxconnect acceptsstorfs-packages-401a-33028tar file a different type ofcompression and allows upgrade to start and fail whenit cannot decompress the file for bootstrap and scvmclientupgrade on ESXi

CSCvq63888

40(2a)40(2a)While attempting to install hyperflex via intersight thefollowing error is seen

Failed in Task Add hosts to vCenter Cluster with ErrorTry adding hosts manually to vCenter and retry Failedto get addressFailed to host by namelookupHostnamecompanycom on 000053 read udp12700145430-gt12700153 read connection refused

CSCvr52098

40(2a)35(2a)CoreAPI call may time out for cluster management APIwhile doing cluster create validation

CSCvo92952

40(2a)35(2d)Validation fails with Mixed mode expansion checkdue to Enhanced vMotion Compatibility Incompatibilitywhen the ESXi version is different

CSCvq95460

40(2a)30(1i)This failure is seen in installer when custom workflowis used hypervisor configuration + deploy (clean diskpartition=no) at Installing software packages on storagecontroller VM

CSCvr66309

40(2a)35(2d)The HX Installer apparently replaced a comma bysemicolon on the VLAN ID range while exporting theconfiguration to a JSON file

CSCvq93831

40(2a)40(1a)When using unsupported browsers HX Connect userswill not be able to login These users see an errorindicating unauthorized user

CSCvp22693

Release Notes for Cisco HX Data Platform Release 4038

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)40(1a)A vulnerability in the web-based management interfaceof Cisco HyperFlex Software could allow anunauthenticated remote attacker to conduct a cross-siterequest forgery (CSRF) attack on an affected system

The vulnerability is due to insufficient CSRF protectionsfor the web UI on an affected device An attacker couldexploit this vulnerability by persuading a user of theinterface to follow a malicious link A successful exploitcould allow the attacker to perform arbitrary actions withthe privilege level of the affected user

Cisco has released software updates that address thisvulnerability There are no workarounds that address thisvulnerability

This advisory is available at the following link

httpstoolsciscocomsecuritycentercontentCiscoSecurityAdvisorycisco-sa-20190807-hypflex-csrf

CSCvp24343

40(2a)35(2d)Add message to HX Connect Progress flow to notAcknowledge Pending Activities and reboot the serverson UCSM

HX Connect is doing a controlled rolling server upgradein the background

CSCvq22844

40(2a)40(1b)Compute Node expand operation fails due to incorrectcluster name

CSCvr43786

40(2a)26(1e)RF-2 cluster shutdown during data resynchronizationafter a node failure (non storfs) followed by multipledisk read failures and a HardBlacklist of a disk

CSCvp41241

40(2a)40(1a)HyperFlex Edge nodes do not properly set the ESXihostname during deployment

CSCvo13143

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4039

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4040

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1b) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401b-33133p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4041

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4042

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1b)40(1a)While running the HyperFlex installer with WindowsServer Hyper-V cluster creation process fails with theerror Failure occurred during ClusterCreation process Unable to post thecontent to the downstream This symptomis encountered while deploying HyperFlex cluster withHyperFlex nodes configuredordered with Cisco VIC1457 MLOM (PID HX-MLOM-C25Q-04) andWindows Server Datacenter or Core with Hyper-Vversion 2016 or 2019

CSCvp64140

Release Notes for Cisco HX Data Platform Release 4043

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)35(2b)Adding Micron 5200 drive to a cluster fails to increasecluster capacity

CSCvo69067

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4044

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4045

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1a) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401a-33028p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4046

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4047

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1a)30(1d)Cluster instability when disks of different sector sizeplaced in HX node

CSCvk17250

40(1a)

35(2c)

35(1a)When logged in using a local HX user account insteadof a Virtual Center account an error message appearsintermittently indicating Virtual Centerunreachable or Resource informationcannot be updated when VC is reachable

CSCvo36198

40(1a)

35(2b)

30(1d)HXDP does not work with EMC RecoverPoint - needsto support VMware API (FSS-Readdir)

CSCvk38003

Release Notes for Cisco HX Data Platform Release 4048

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)

35(2a)

30(1i)

25(1c)

25(1d)

Zookeeper (Exhibitor) process on HyperFlex StorageController VMs may fail to respond to requests whenvarzookeeper has low or no free space

CSCvm90352

40(1a)26(1c)Use Asynchronous consolidation for HX SnapshotsCSCvn02151

40(1a)35(2b)HX Quiesced Snapshot for Backup VendorsCSCvo90713

40(1a)35(2a)The cluster creationcluster expansion workflow stopswith the following error message at the validation step

FIRMWARE-Check UCSC-SAS-M5HD

FIRMWARE-Check UCSC-SAS-M5HD Required00000029000000320000003500000050 Found00000058

Action Needed Update the Controller Firmware toRequired Version

CSCvn17787

35(2b)30(1d)Kernel migration fails when a local datastore is searchedfor in the ESXi host

CSCvn73127

35(2a)26(1b)A node shuts down when two disks are replaced (acaching disk and another capacity disk) where thecapacity disk is inserted first and then the caching diskis inserted

CSCvk46364

Hyper-V

40(1a)35(2a)Cluster expansion may fail with an error code 500 -operation timed out

CSCvn28721

35(2b)35(2a)During upgrade remove the VLAN on team created foruser vSwitch During fresh installation only one VLANtag is set to the vSwitch and team although multipleVLANs were entered

CSCvn54300

35(2b)35(2a)While upgrading a Hyper-V cluster on account of a rarerace condition between the stUpgradeService andZookeeper servers the upgrade orchestration throws anupgrade validation error and the upgrade process isaborted

CSCvn60486

Release Notes for Cisco HX Data Platform Release 4049

Resolved Caveats in Release 40(1a)

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Startrestart EAM service invCenter

When EAM service is not runningvCenter registration is failing

CSCvt55712

35(2a)Contact Cisco TAC if you believeyou are seeing similar conditions

HyperFlex Stretch Cluster saw poorapplication performance due to writelatency cluster remained unhealthy andrebalance was stuck

This defect is currently being used forinvestigative purposes

CSCvt45344

35(2g)Remove log files fromvarlogspringpath folder when thelog files are not getting rotatedStop the relevant service removelarge log files fromvarlogspringpath folder andrestart the service

varstv folder in Controller VM maybecome full

CSCvt36374

40(2b)Restarting the stMgr will initializeit again

In a Cluster configured with DR networkStMgr may not get initialized and stuckin a deadlock while enabling the IPTablerules (can be verified from the stMgrlog)

CSCvt89709

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Click recover for the selectedVM again under the standalonemode This will move the VM torecovered state

When the user recovers a VM protectedin a group it along with other VMs in thegroup move to recovering state instandalonemode The selectedVM shouldmove to recovered state

CSCvs75553

35(2e)Clean up the fail over cluster andthen recreate the fail over clusterNo need to touch the HX storagecluster

Hyper-V When replicated DC was usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2d)40(1a)

Keep cluster utilization as low aspossible To avoid read-only statekeep below 92

HyperFlex clusters storage utilization isabove 92 usable capacity

CSCvr20922

Release Notes for Cisco HX Data Platform Release 4050

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1b)NAAll the disks in the node got locked afterrebooting the node

We successfully unlocked usingsed-clientsh -U command but wantedto test with another reboot and driveslocked again

CSCvs74286

35(2g)To recover the VM copy over thedata disks and attach them to thenew VM

After enable encryption HX connectencryption status showing Caution butall of the disks enable encryptionsuccessful

-Self Encrypting Drives Service isrunning on all of the nodes

-There is no error message of encryption

- USB0 interface is up

-All SED disks showing supported 1enabled 1 locked 0

CSCvs93245

40(1b)Enable kernelpanic_on_oops as apersistent configuration This willcause the node to panic and rebootimmediately

A cluster node running HX release40(1b) may hang in the Linux kernelThis is classified as an oops and adeviation from the expected behavior

CSCvs54285

Release Notes for Cisco HX Data Platform Release 4051

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Enabling K8iscsi stack in Hx clusterCSCvs41324

Release Notes for Cisco HX Data Platform Release 4052

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Steps to enable K8 on HX cluster

Perform following steps on ALLthe controller VMs

1 Updateetcinitscvmclientconf toenable tune iscsiEnable totrue

sed -iesiscsiEnable=falseiscsiEnable=trueetcinitscvmclientconf

2 Run following initctlcommand to reloadconfiguration

initctl reload-configuration

3 Restart scvmclient process

stop scvmclient startscvmclient

4 Run following initctlcommand to mount SYSTEMDS

initctl emit --no-waitsystem-datastore-created

5 Verify that iscsiEnable tuneis set to true using thefollowing command

ps -eaf | grep scvmclient |grep -v grep

root 6241 1 1 Dec06 014007optspringpathstorfs-corescvmclient-T logEnabled=false -TlogSyslogEnabled=true -TlogEchoToScreen=false -TstatLoggingToFile=false -TstatLoggingToSyslog=true-TlogDir=varlogspringpath-TnfsBackendServerList=1010748100-T iscsiEnable=true -TiscsiUseAsync=true -T

Release Notes for Cisco HX Data Platform Release 4053

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

iscsiConfigFilePath=nfsSYSTEMistgtconfps -eaf | grep scvmclient |grep -v grep

35(2e)Following procedure can be usedto check NFS Queue Depth andincrease if needed

rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth256 lt- Low value[rootHXESXI1] vsish -e setvmkModulesnfsclientmnt[DATASTORE]propertiesmaxQDepth 1024[rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth1024 lt- Optimalvalue

This requires an ESXi host rebootto take effect

Please place the node in HyperflexMaintenanceMode and gracefullyreboot the node for the changes tobe applied

HyperFlex Datastore NFS Queue Depthshows as 256 which can lead toperformance (including latency) issues

CSCvr83056

40(1b)This is cosmetic The error will goaway once the cluster returns tohealthy status

When a single node is offline in clusterstcli cluster storage-summary shows twonodes unavailable rootSCVM~ stclicluster storage-summary

CSCvs47735

35(2g)Modify both the root and adminpassword for the SCVMs to be thesame

stcli security password set -uadmin

Cluster expansion fails at Config Installerstage when the root and admin passwordfor the storage controller (SCVM) aredifferent

CSCvs69317

35(2d)NAAfter successfully changing the DNSserver on the HX controller we still cansee the original DNS entry that was addedduring the deployment

CSCvs69154

Release Notes for Cisco HX Data Platform Release 4054

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Customers are strongly encouragedto work with TAC in order toidentify and remediate this issue

1 Change getcluster from 2m to8m inoptspringpathstorfs-mgmtstMgr-10confapplicationconfon all nodes

2 Restart stMgr on all nodes

You may see this error message after afailed upgrade or other task such asattempting to enter a node into HXmaintenance mode

CSCvs53555

40(2a)Run the following command oneach storage controller VM as root

grep -i monitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp sed -iesmonitor_dns_serverstruemonitor_dns_serversfalseoptspringpathhx-diag-toolswatchdog_configjsonampamp grep -imonitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp restart watchdog

Receive the following alertevent in HXConnect HX Controller VMHOSTNAME one or more configuredDNS servers not responding

CSCvt13947

35(2b)Delete any empty (Size 0) log filesunder varlogzookeeperversion-2ORvarlogzookeeperstandaloneversion-2

Zookeeper fails to start while Exhibitor isrunning however echo srvr returnsnothing

CSCvs21562

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

40(4e)1 De-commission the server

2 Power drain the server -REMOVE BOTH POWERCORDS ON THE BACK OFTHE SERVER FOR 10SECONDS THENREINSERTPOWERCORDS

3 Re-commission the server

M2 boot disk is missing from serverinventory after upgrade to server firmware40(4e) As a result server fails to boot toOS installed in the M2 disk The issuepersists after re-acknowledgement as wellas de-commission andre-acknowledgement of the server

CSCvs02466

Release Notes for Cisco HX Data Platform Release 4055

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Re-start the flusher manuallyMany disks taken offline by a fault ormanually resulting in ENOSPC oncluster

CSCvq32721

35(2e)Clean up fail over cluster and thenrecreate the fail over cluster Thereis no need to touch the HX storagecluster

[Hyper-V]When replicated DCwas usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2b)Manually install upgradescvmclient VIB andmake sure thatit matches with HXDP version

Upgrade from release HX 35(2a) to35(2x) 4X does not upgrade scvmclientRecovery point may not work

CSCvq54992

Management

30(1b)To recover the VM copy over thedata disks and attach them to thenew VM

[HyperV]VMshows up onmultiple nodesCSCvj22992

35(1a)1 Do not change MTU at ESXlevel on running cluster

2 Reset back to the original value

ChangingMTU from 9000 to 1500 on oneESX node causes storfs process to restarton all nodes

CSCvm99150

Release Notes for Cisco HX Data Platform Release 4056

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)After the installation (for newdeployments) or after an upgradeto 401a (for existingdeployments) perform thefollowing steps on all ControllerVMs

1 Edit the following tune files (onall Controller VMs)

optspringpathconfiglfftunes

optspringpathconfigvsi_16tbtunes

2 SetcleanerEnableSegSummaryCleaning

to false

3 After editing the above tunefiles

a SSH in to all the ControllerVMs

b Run storfstool -- -Z

c Run the below command andcheck the values Tune valueshould be true

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=true

d Type below commands andapplied the tunes changesdynamically

echo false gttmpstprocfssystemtunecleanerEnableSegSummaryCleaning

e Verify that tune values arechanged Run below commandValue should be false

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=false

f umount tmpstprocfs

A cluster with 8TB or 12TB disk drivesmay experience IO stalls for severalminutes after another node fails in thecluster

CSCvp23718

Release Notes for Cisco HX Data Platform Release 4057

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)1 Update the conf file manuallyto change to the link local addressto

2 Restart scvmclient on thecontroller and ESX

HX 35(2b) FlexVol to HX 40 CSIupgrade does not work FlexVol continuesto work

CSCvp26319

35(2a)Disable rebalance until node healsthen reenable rebalance

7-node cluster healing very slowlybecause rebalance change AMS reqs areblocking healing process

CSCvp62512

35(2a)NAWith moderate cluster storage capacityusage under site failover cluster clustermay experience an all paths (APD) downcondition

CSCvq49412

40(1a)NAThis product includes Third-partySoftware that is affected by thevulnerabilities identified by the followingCommon Vulnerability and Exposures(CVE) IDs

CVE-2016-0762 CVE-2016-6797CVE-2016-6816 CVE-2016-8735CVE-2017-5647 CVE-2017-12615CVE-2017-12616 CVE-2017-12617CVE-2017-7674 CVE-2018-1304CVE-2018-8014 CVE-2018-1336CVE-2018-8034 CVE-2018-11784CVE-2019-0232

CSCvp36364

35(2d)NAstcli cluster info command needs toprovide UCSM VIP address Currently itshows ucsm-hostcom

CSCvq11456

35(2a)No workarounds currently Thestale entries do not affect the dayto day operations of HyperFlex

HyperFlex upgrade validation failedbecause cluster has extraneous stNodes ininternal database

CSCvq32530

35(2a)NAWhen not in maintenancemode all drivesappear in HX Connect and get updated asexpected

When in maintenance mode extra drivesare shown in HX Connect with no slotnumbers

CSCvq42378

35(2b)Contact TAC to cleanup duplicatenode

Nodes duplicated in HX Connect aftercluster recreate

CSCvq61825

Release Notes for Cisco HX Data Platform Release 4058

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)NACurrently HyperFlex installs on Hyper-Vdo not contain the stcli security whitelistcommandset

CSCvq66245

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

Release Notes for Cisco HX Data Platform Release 4059

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)[Hyper-V] Converged node expansionfails for incoming node with VIC 14xx

CSCvq39523

Release Notes for Cisco HX Data Platform Release 4060

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

1 Set Jumbo frame inWindowsHypervisor for incomingnode

Get-NetAdapter storage-data-a|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

Get-NetAdapter storage-data-b|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

2 Verify all interfaces are setcorrectly to support Jumbo frames

Get-NetAdapter storage |Get-NetAdapterAdvancedProperty| RegistryKeyword -matchjumbo | ft -auto

You should receive the followingmessage

Name DisplayName DisplayValue

RegistryKeyword RegistryValue

---- ----------- ------------

--------------- -------------

vswitch-hx-storage-data Jumbo

Packet 9014 Bytes

JumboPacket 9014

storage-data-b Jumbo Packet

Bytes 9014 JumboPacket

9014

storage-data-a Jumbo Packet

Bytes 9014 JumboPacket

9014

3 Reboot only the incoming nodeor the node being expanded

4 Retry Cluster Expansion in

Release Notes for Cisco HX Data Platform Release 4061

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

NVMe Caching SSDs slot information is unavailable from HX-Connect for all AF server PIDs except forthe All-NVMe server PIDs Please refer to UCSM management console for NVMe SSD slot information

Note

Applicable PlatformsDrive PIDDrive Function

All existing HX M5 serversHX-M2-960GBAlternate boot drive

All existing HX M5 servers except AllNVMe

HX-SD480G6I1X-EVAlternate system (or housekeeping) drive

All existing HX M5 servers except AllNVMe

HX-SD480GM1X-EVAlternate system (or housekeeping) drive

The following HX M5 serversHXAF220C-M5SX HXAF240C-M5SXHXAF-E-220M5SX

HX-SD800G123X-EP800G 12G SAS Cache drive option forAll Flash

All NVMe HXAF220C-M5SNHX-NVME2H-I1000All NVMe 1TB Capacity drive

All NVMe HXAF220C-M5SNHX-NVME2H-I4000All NVMe 4TB Capacity drive

All NVMe HXAF220C-M5SNHX-NVMEHW-I8000New high density All NVMe 8TBCapacity drive

All Flash Configuration ndash namelyHXAF220C-M5SX HXAF240C-M5SXHXAF-E-220M5SX

ESX support only

HX-SD76T61X-EVNew high density All Flash 76TBCapacity drive

Maximum number of drives per node is12 drives on HX AF C240

HX240C-M5LHX-HD8T7K4KANAlternate drive for 8TB LFF capacity

HXAF220C-M5SX HXAF240C-M5SXHX-SD38T2HTNK9New 38TB FIPS compliant SED SSDdata drives

HXAF220C-M5SX HXAF240C-M5SXHX-SD960G2HTNK9New 960GFIPS compliant SEDSSD datadrives

New Features in Release 40(1b)

bull Support for Second Generation Intelreg Xeonreg Scalable Processor RefreshmdashThis release includessupport for the Second Generation Intelreg Xeonreg Scalable processor refresh (formerly Cascade Lake)

New Features in Release 40(1a)

The following new features are in Release 40(1a)

bull Ultra-Light HyperFlex Edge ClustersmdashThis release introduces support for two-node HyperFlex Edgeclusters enabling HyperFlex to run in environments requiring a small footprint Cisco Intersight providescomprehensive lifecycle management and includes remote cloud-based installation centralized upgradesand invisible witnessing Both 1GE and 10GE networking topology options are available

Release Notes for Cisco HX Data Platform Release 407

New Features

bull Scaled-Up HyperFlex Edge ClustersmdashThis release adds support for four node HyperFlex Edge clustersenabling a full range of size options for remote and branch offices Size the branch office environmentsto suit current needs with a two three or four node HyperFlex Edge cluster Cisco Intersight providesfull-lifecycle management and 1GE and 10GE networking options are available

bull Cisco Intersight Invisible Cloud WitnessmdashFor two node clusters this feature eliminates the need forwitness VMs the infrastructure to run those VMs and the management overhead to deploy scale andpatch witnessing software The Invisible Cloud Witness is responsible for maintaining cluster HA in theevent of failure scenarios This feature is included at no additional cost and is automatically deployedand managed by Cisco Intersight

bull Cloud-delivered HyperFlex Edge UpgradesmdashPowered by Cisco Intersight this feature adds supportfor multi-site orchestrated remote upgrades of the HyperFlex Data Platform This feature will be enabledwith the next 40 patch release and will allowHyperFlex Edge clusters deployed via Intersight to performorchestrated upgrades across one or many sites in parallel

bull All-NVMe HyperFlexmdashStarting with this release a new high-end performance node powered by allNVMe drives is available for HyperFlex clusters Co-engineered with Intel to support Intel VMD forhot-plug and surprise removal this offering represents an industry first an enterprise-ready and fullyvalidated all-NVMe HCI appliance The all-NVMe offering is available in the 220 form factor (1RU)and is powered by Intel Optane cache drives for maximum performance and highest endurance

bull VMware Site Recovery Manager (SRM) IntegrationmdashThis release brings support for a Cisco developedStorage Replication Adapter (SRA) for SRM The SRA provides the ability to leverage HyperFlex nativeasync replication with the powerful orchestration and runbook capabilities of SRM The SRA includesthe ability to perform test recoveries planned migrations and full disaster recovery

HX SRA is certified by VMware and is available for download from VMWareSRM site

Note

bull HyperFlex DR Powershell RunbooksmdashNew Powershell cmdlets are included for automated runbookgeneration when using HyperFlex native disaster recovery The New-HXRunbook cmdlet supports thefollowing workflows Test Recovery Planned Migration and Disaster Recovery These runbooks canbe used to orchestrate DR workflows without the requirement for any third-party software

bull Windows Server 2019 with Hyper-VmdashSupport has been added in this release for the Windows Server2019 operating system for Hyper-V based HyperFlex deployments

bull Kubernetes CSI PluginmdashThis release adds support for the HyperFlex CSI (HX-CSI) plugin based onthe Kubernetes Container Storage Interface (CSI) specification Customers can now use the HX-CSIplugin to provision andmanage persistent volumes in Kubernetes version 113 and later Note Kubernetes113 support for Cisco Container Platform amp Openshift Container Platform is forthcoming in theirrespective future releases

bull C480 ML Compute only nodemdashThis release introduces support for C480 ML as a new compute-onlynode for Deep LearningMachine Learning Workloads Data scientists can now use the power of up toeight NVidia SXM2 V100 GPUs to accelerate deep learning workloads VMs running deep-learningworkloads will need to use PCIe pass-through for access to GPUs

bull Higher Capacity DrivesmdashAnew 24TB 10k rpm SAS HDD option for SFF Hybrid HyperFlex clustersand a 12TB 72K rpmSASHDDoption for LFFHybrid are now available Both HyperFlex andHyperFlexEdge support the 24TB capacity point for maximum density in this form factor Note that HyperFlex

Release Notes for Cisco HX Data Platform Release 408

New Features

Edge does not support LFF-drives HyperFlex HyperV version does not yet support the new 12TB driveoption See the HyperFlex spec sheets for a full list of configurable options

bull New Cache and increased scale for Hyper-V mdashNVMe amp Optane SSDs are now supported as cachedrives for Hyper-V deployments Furthermore scale limits have been increased to 16+16(Converged+Compute-only) for both SFF (AF and Hybrid) amp LFF (Hybrid) clusters

bull Centralized Audit Log ExportmdashThis release adds support for audit logging via a remote syslog serverThis capability enables customers to retain audit logs from all HyperFlex nodes in a centralized remotesyslog server to meet retention and compliance requirements

bull DISA STIG CompliancemdashThis release adds new HX REST APIs for setting removing and checkingstatus of DISA STIGs for Controller VMs ESXi hosts and vCenter These APIs enable customers tomeet DISA security requirements by centrally and securely applying STIGs detecting and correctingfor drifts in any STIG settings

Supported Versions and System RequirementsCisco HX Data Platform requires specific software and hardware versions and networking settings forsuccessful installation

For a complete list of requirements see

bull Cisco HyperFlex Systems Installation Guide for VMware ESXi or

bull Cisco HyperFlex Systems Installation Guide for Microsoft Hyper-V

Hardware and Software Interoperability

For a complete list of hardware and software inter-dependencies refer to respective Cisco UCS Managerrelease version of Hardware and Software Interoperability for Cisco HyperFlex HX-Series

HyperFlex Software Versions

The HX componentsmdashCisco HX Data Platform Installer Cisco HX Data Platform and Cisco UCSfirmwaremdashare installed on different servers Verify that each component on each server used with and withinan HX Storage Cluster are compatible

bull HyperFlex does not support UCS Manager and UCS Server Firmware versions 40(4a) 40(4b)and 40(4c)

Do not upgrade to these versions of firmware

Do not upgrade to these versions of UCS Manager

Important

bull Verify that the preconfigured HX servers have the same version of Cisco UCS server firmware installedIf the Cisco UCS Fabric Interconnects (FI) firmware versions are different see the Cisco HyperFlexSystems Upgrade Guide for VMware ESXi Release 40 for steps to align the firmware versions

bull M4 For NEW hybrid or All Flash (Cisco HyperFlex HX240c M4 or HX220c M4) deploymentsverify that Cisco UCS Manager 31(3k) 32(3i) or 40(2b) or higher is installed For moreinformation see Recommended Cisco HyperFlex HX Data Platform Software Releases

Release Notes for Cisco HX Data Platform Release 409

Supported Versions and System Requirements

bull M5 For NEW hybrid or All Flash (Cisco HyperFlex HX240c M5 or HX220c M5) deploymentsverify that the recommended UCS firmware version is installed

For SED-based HyperFlex systems ensure that the A (Infrastructure) B (Bladeserver) and C (Rack server) bundles are at Cisco UCS Manager version 40(2b)or later for all SED M4M5 systems For more details see CSCvh04307 ForSED-based HyperFlex systems also ensure that all clusters are at HyperFlexRelease 35(2b) or later For more information see Field Notice (70234) andCSCvk17250

Important

bull To reinstall an HX server download supported and compatible versions of the software See theCisco HyperFlex Systems Installation Guide for VMware ESXi for the requirements and steps

bull Important For Intersight edge servers running older than 40(1a) CIMC version HUU is the suggestedmechanism to update the firmware

Table 1 HyperFlex Software Versions for M4M5 Servers

M5 Recommended FIServer Firmware

(be sure to review important notes above)

M4 Recommended FIServer Firmware

(be sure to review important notes above)

HyperFlexRelease

40(4h)40(4h)40(2b)

40(4h)40(4h)40(2a)

40(4h)40(4h)40(1b)

40(4e)40(4e)40(1a)

HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments

Cisco HX Data Platform Release 4x based Deployments

Confirm the component firmware on the server meets the minimum versions listed in the following tables

HyperFlex Edge does not support Cisco IMC versions 40(4a) 40(4b) 40(4c) 40(4d) and 40(4e)Important

Table 2 HX220c M4 HXAF220c M4 Cluster

Recommended Firmware Version - HXDP 4x

(be sure to review important note(s) above)

Component

402d0BIOS

40(2h)Cisco Integrated Management Controller(CIMC)

Release Notes for Cisco HX Data Platform Release 4010

HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments

Recommended Firmware Version - HXDP 4x

(be sure to review important note(s) above)

Component

13000012SAS HBA

43(2b)VIC 1227

0103 (Toshiba)

CS07 (Intel 480)

SSD

CN03 (Seagate)

CK03 (Seagate 18)

CK03 (Seagate 24)

sF3Q (Samsung)

3F3Q (Samsung)

HDD

40(2h)

Download Software

Host Upgrade Utility (HUU) Download Link

Table 3 HX220c M5 HXAF220c M5 Cluster

Recommended Firmware Version - HXDP 4x

(be sure to review important notes above)

Component

C220M5404p00224200755BIOS

40(4i)Cisco Integrated Management Controller(CIMC)

09000008SAS HBA

43(3c)VIC 1387

50(3e)VIC 1457

G201CS01 (Intel)

0104 (Toshiba)

0107 (Toshiba)

SSD

2F3Q (Samsung)

3F3Q (Samsung)

NOA4 (Seagate)

CK03 (Seagate 18tb)

CK03 (Seagate 24)

HDD

Release Notes for Cisco HX Data Platform Release 4011

HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments

Recommended Firmware Version - HXDP 4x

(be sure to review important notes above)

Component

40(4k)

Download Software

Host Upgrade Utility (HUU) Download Link

HyperFlex Licensing

As of version 26(1a) HyperFlex supports VMware PAC licensing Existing VMware embedded licenses willcontinue to be supported

As of version 25(1a) HyperFlex uses a smart licensing mechanism to apply your licenses See the CiscoHyperFlex Systems Installation Guide for VMware ESXi for details and steps

VMware vSphere Licensing Requirements

How you purchase your vSphere license determines how your license applies to your HyperFlex system

bull If you purchased your vSphere license with HyperFlex

Each HyperFlex server either has the Enterprise or Enterprise Plus edition preinstalled at the factory

bull HX Nodes have OEM licenses preinstalled If you delete or overwrite thecontent of the boot drives after receiving the HX servers you also delete thefactory-installed licenses

bull OEM license keys is a newVMware vCenter 60 U1b feature Earlier versionsdo not support OEM licenses

bull All factory-installed HX nodes share the same OEM license key WithvSphere OEM keys the Usage count can exceed the Capacity value

bull When you add an HX host to vCenter through the Add Host wizard in theAssign license section select the OEM license

We obfuscate the actual vSphere OEM license key for example0N085-XXXXX-XXXXX-XXXXX-10LHH

bull Standard Essentials Plus and ROBO editions are not available preinstalledon HX servers

Note

bull If you did NOT purchase your vSphere license with HyperFlex

The HX nodes have a vSphere Foundation license preinstalled After initial setup you can apply thelicense to a supported version of vSphere

bull If you purchased a vSphere PAC license

Follow the instructions in your PAC license letter from VMware to add the license to your MY VMwareaccount then follow the instructions to add your HX host to vCenter and assign the PAC license

Release Notes for Cisco HX Data Platform Release 4012

HyperFlex Licensing

HX Data Platform Software Versions for HyperFlex Witness Node

Witness Node VersionHyperFlex Release

10840(2b)

10840(2a)

10440(1b)

10440(1a)

Software Requirements for VMware ESXi

The software requirements include verification that you are using compatible versions of Cisco HyperFlexSystems (HX) components and VMware vSphere VMware vCenter and VMware ESXi

bull Verify that all HX servers have a compatible version of vSphere preinstalled

bull Verify that the vCenter version is the same or later than the ESXi version

bull Verify that the vCenter and ESXi versions are compatible by consulting the VMware ProductInteroperability Matrix Newer vCenter versions may be used with older ESXi versions so long as bothESXi and vCenter are supported in the table below

bull Verify that you have a vCenter administrator account with root-level privileges and the associatedpassword

For VIC1457 there is no support for ESXi 60Note

The below table applies for all of the following VMware vSphere Editions Enterprise Enterprise PlusStandard Essentials Plus ROBO

VMware vCenter VersionsVMware ESXi VersionsHyperFlex Version

60 U3 65 U3 67 U360 U3 65 U3 67 U340(2b)1

60 U3 65 U3 67 U360 U3 65 U3 67 U340(2a)2

60 U3 65 U3 67 U260 U3 65 U3 67 U2340(1b)

60 U3 65 U2 67 U260 U3 65 U2 67 U2440(1a)

1 Cisco HyperFlex Release 40(2) is the last major HyperFlex release that will support vSphere 60 (ESXiand vCenter) due to those versions reaching end of VMware general support on March 12 2020

2 Cisco HyperFlex Release 40(2) is the last major HyperFlex release that will support vSphere 60 (ESXiand vCenter) due to those versions reaching end of VMware general support on March 12 2020

3 Use of 67 U2 is not recommended see software advisory for CSCvq06952 and CSCvp88515 for furtherdetails

4 Use of 67 U2 is not recommended see software advisory for CSCvq06952 and CSCvp88515 for furtherdetails

Release Notes for Cisco HX Data Platform Release 4013

HX Data Platform Software Versions for HyperFlex Witness Node

Software Requirements for Microsoft Hyper-V

The software requirements include verification that you are using compatible versions of Cisco HyperFlexSystems (HX) components and Microsoft Hyper-V (Hyper-V) components

HyperFlex Software versions

The HX componentsmdashCisco HX Data Platform Installer Cisco HX Data Platform and Cisco UCSfirmwaremdashare installed on different servers Verify that each component on each server used with and withinthe HX Storage Cluster are compatible

bull Cisco HyperFlex M5 Converged nodesmdash For Hybrid (Cisco HyperFlex HX240c M5 HX220c M5)and All Flash (Cisco HyperFlex HXAF240cM5 HXAF220cM5) verify that Cisco UCSManager 40(2b)is installed HX 40 (1a) does not support Hyper-V on the All NVMe (HXAF220C-M5SN) nodes Fordetailed information on installation requirements and steps see theCisco HyperFlex Systems InstallationGuide on Microsoft Hyper-V

Table 4 Supported HyperFlex Software versions for M5 Servers on Hyper-V

M5 Recommended Server FirmwareHyperFlex Release

40(4h)40(2b)

40(4g)40(2a)

40(4e)40(1b)

40(4e)40(1a)

Release Notes for Cisco HX Data Platform Release 4014

Software Requirements for Microsoft Hyper-V

Table 5 Supported Microsoft Software versions

VersionMicrosoft Component

Windows Server 2016 Datacenter Core amp DesktopExperience

ForWindows Server 2016Datacenter Coreand Desktop Experience the Windows2016 ISO image should be Update BuildRevision (UBR) 1884 at a minimum

Note

Windows Server 2019Datacenter-DesktopExperienceis supported starting from HXDP 401(a) onwards

For Windows Server 2019 DesktopExperience theWindows 2019 ISO imageshould be Update Build Revision (UBR)107 at a minimum

Note

Windows Server 2019 DatacenterndashCore is notsupported currently

Also note that the following are currently notsupported

OEM activated ISOs and Retail ISOs are notsupported

Earlier versions ofWindows Server such asWindows2012r2 are not supported

Non-English versions of the ISO are not supported

Windows Operating System (Windows OS)

A Windows 2012 or later domain and forestfunctionality level

Active Directory

Supported Microsoft License Editions

The Microsoft Windows Server version that is installed on one or more HyperFlex hosts must be licensed asper Microsoft licensing requirements listed on Microsoft Licensing

Browser Recommendations

Use one of the following browsers to run the listed HyperFlex components These browsers have been testedand approved Other browsers might work but full functionality has not been tested and confirmed

Table 6 Supported Browsers

HX ConnectHX Data PlatformInstaller

Cisco UCS ManagerCisco IntersightBrowser

11 or higher11 or higher11 or higherNAMicrosoft InternetExplorer

70 or higher70 or higher57 or higher62 or higherGoogle Chrome

Release Notes for Cisco HX Data Platform Release 4015

Browser Recommendations

HX ConnectHX Data PlatformInstaller

Cisco UCS ManagerCisco IntersightBrowser

60 or higher60 or higher45 or higher57 or higherMozilla Firefox

NANA9 or higher10 or higherApple Safari

NANA35 or higherNAOpera

Notes

bull Cisco HyperFlex Connect

The minimum recommended resolution is 1024 X 768

bull Cisco HX Data Platform Plug-in

The Cisco HX Data Platform Plug-in runs in vSphere For VMware Host Client System browserrequirements see the VMware documentation at httpswwwvmwarecomsupportpubs

The HX Data Platform Plug-in is not displayed in the vCenter HTML client You must use the vCenterflash client

bull Cisco UCS Manager

The browser must support the following

bull Java Runtime Environment 16 or later

bull Adobe Flash Player 10 or higher is required for some features

For the latest browser information about Cisco UCSManager refer to themost recent Cisco UCSManagerGetting Started Guide

Cisco HX Data Platform Storage Cluster SpecificationsCluster Limits

bull Cisco HX Data Platform supports up to 100 clusters managed per vCenter as per VMware configurationmaximums

bull Cisco HX Data Platform supports any number of clusters on a single FI domain Each HX convergednode must be directly connected to a dedicated FI port on fabric A and fabric B without the use of a FEXC-series compute only nodes must also directly connect to both FIs B-series compute only nodes willconnect through a chassis IO module to both fabrics In the end the number of physical ports on the FIwill dictate the maximum cluster size and maximum number of individual clusters supported in a UCSdomain

The following table provides Cisco HX Data Platform Storage Cluster Specifications

Release Notes for Cisco HX Data Platform Release 4016

Cisco HX Data Platform Storage Cluster Specifications

Stretched Cluster (Availableon ESX Only)

Microsoft Hyper-VVMware ESXiNode

AllNVMe -HXAF220c-M5SN

HX240c-M5LHX220cM5

HX220cAF M5

HX240cM5

HX240cAF M5

HX240c-M5LHX220cM5

HX220cAF M5

HX240cM5

HX240cAF M5

HXAF220c-M5SN

AllNVMe -HXAF220c-M5SN

ESXi 65or 67only NotsupportedwithESXi 60orHyper-V

HX220cM5 Edge

HXAF220cM5 Edge

HX220cM4 Edge

HXAF220cM4 Edge

HX240c-M5LHX220cM5

HX220cAF M5

HX240cM5

HX240cAF M5

HX220cM4

HX220cAF M4

HX240cM4

HX240cAF M4

HXServers

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

C220M5C240M5B200M4B200M5

C240M5C220M5B200M4B200M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

mdashB200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

Compute-OnlyUCSB-SeriesC-SeriesServers

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

Convergednodesonly

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

SupportedNodes

Release Notes for Cisco HX Data Platform Release 4017

Cisco HX Data Platform Storage Cluster Specifications

Stretched Cluster (Availableon ESX Only)

Microsoft Hyper-VVMware ESXiNode

NA(requiresEnterpriseHXDP-PLicense)

NA(requiresEnterpriseHXDP-PLicense)

NA(requiresEnterpriseHXDP-PLicense)

Convergednodes3-16(12TBHDDoption isnotsupportedforHyperV)

Compute-onlynodes0-16

Convergednodes3-16

Compute-onlynodes0-16

NA(requiresEnterpriseHXDP-PLicense)

M4Convergednodes 3

M5Convergednodes23or 4

RequiresHXDP-ELicense

Convergednodes3-16(12TBdrive islimited toamaximumof 8nodes)

Compute-onlynodes0-16(12TBdrive islimited toamaximumof 8nodes)

Convergednodes3-32(76TBdriveconfigsarelimited tomaximumof 12drivesnodeon AF240)

Compute-onlynodes0-32

HXDP-SLicensedNodeLimits

11 ratioofHXDP-StoComputeonlynodes

(MinmdashMax)

Convergednodes2-16 perSite

Compute-onlynodes0-21 perSite

(up tomaxclustersize)

Convergednodes2-8 perSite

Compute-onlynodes0-16 perSite

(up tomaxclustersize)

Convergednodes2-16 perSite

Compute-onlynodes0-21 perSite

(up tomaxclustersize)

Convergednodes3-16(12TBHDDoption isnotsupportedforHyperV)

Compute-onlynodes0-16

Convergednodes3-16

Compute-onlynodes0-16

Convergednodes3-32

ComputeOnlynodes0-32

(up tomaxclustersize)

Convergednodes 3

(requiresHXDP-ELicense)

Convergednodes3-16(12TBdrive islimited toamaximumof 8nodes)

Compute-onlynodes0-32(12TBdrive islimited toamaximumof 16nodes)

Convergednodes3-32(76TBdriveconfigsarelimited tomaximumof 12drivesnodeon AF240)

Compute-onlynodes0-32

(up tomaxclustersize)

HXDP-PLicensedNodeLimits

12 ratioofHXDP-PtoComputeonlynodes

(MinmdashMax)

24 perSite 64percluster

24 perSite 48percluster

32 perSite 64percluster

32326434864MaxClusterSize

Release Notes for Cisco HX Data Platform Release 4018

Cisco HX Data Platform Storage Cluster Specifications

Stretched Cluster (Availableon ESX Only)

Microsoft Hyper-VVMware ESXiNode

212121111121mdash2121MaxComputetoConvergedratio

NoExpansion

Requires Enterprise license

Requires uniform expansion across both sites

Guidelines and Limitationsbull Starting with release 40(2a) SCVM is no longer needed on a Compute node

bull HX REST API Access Token Management ndash Applications leveraging HX REST APIs should re-useaccess tokens when making API calls Once obtained using the AAA Obtain Access Token API accesstokens are valid for 18 days (1555200 seconds) In addition AAA enforces rate limiting on ObtainAccess Token API requests in a 15 minute window auth can be invoked (successfully) a maximum of5 times A user is allowed to create a maximum of 8 unrevoked tokens Subsequent call to auth willautomatically revoke the oldest issued token tomake room for the new token Amaximum of 16 unrevokedtokens can be present in system In order to prevent brute-force attacks after 10 consecutive failedauthentication attempts a user account is locked for a period of 120 secondsFor more information seeCisco HyperFlex Systems REST API Reference guide

HxConnect makes use of AAA Authentication REST API for login and the above rate limit applies toHxConnect also

Upgrade Guidelines

The following list is a highlight of critical criteria for performing an upgrade of your HyperFlex system

bull Unsupported HX Data Platform 17x 18x 20 21x 25x and 26x clustersmdashUsers from anyversion prior to 26(1a) must step through an intermediate version before upgrading to 40 or later releasesIf you need to upgrade your environment from a Cisco HyperFlex HX Data Platform software releasethat is past the last date of support to the latest suggested release on the Cisco Software Download sitesee Cisco HyperFlex Systems Upgrade Guide for Unsupported Cisco HXReleases For more informationsee the Software Advisory for CSCvq66867 WARNING Only Use HXDP 26(1e) Upgrade PackageWhen Upgrading From HXDP 18(1a)-18(1e)

bull Hypercheck Health Check UtilitymdashCisco recommends running this proactive health check utility onyour HyperFlex cluster prior to upgrade These checks provide early visibility into any areas that mayneed attention andwill help ensure a seamless upgrade experience For more information see the HyperFlexHealth amp Pre-Upgrade Check Tool TechNote for full instructions on how to install and run Hypercheck

bull vSphere 67 Software AdvisorymdashDo not upgrade to Cisco HX Data Platform Release 40(1a) whenrunning ESXi 67U1 EP06 (build 11675023) Do not upgrade to 67U1 EP06 (build 11675023) ifrunning Cisco HX Data Platform Release 40(1a) See the Software Advisory CSCvo56350 for furtherdetails

Release Notes for Cisco HX Data Platform Release 4019

Guidelines and Limitations

The software build version posted at release will override any other local versions

bull Required vCenter upgrademdashFor enhanced security Cisco HX Data Platform Release 35(1a) or laterrequires the use of TLS 12 Therefore vCenter must be upgraded to 60 U3f or later before upgradingto Cisco HX Data Platform Release 35 or later In addition ESXi should be upgraded as required tomeet HX Data Platform compatibility requirements

bull Minimum HXDP version for upgrademdashHXData Platform clusters running 26(1a) or later may upgradedirectly to 40 using the HX Connect UI

bull Cluster ReadinessmdashEnsure that the cluster is properly bootstrapped and the updated plug-in is loadedbefore proceeding Manual cluster bootstrap is required for upgrade from a pre-35 release

bull Cluster ReadinessmdashEnsure that the cluster is properly bootstrapped and the updated plug-in is loadedbefore proceeding Manual cluster bootstrap is required for HX releases earlier than 35(1a) For moreinformation see theManual Bootstrap Upgrade Process in the Cisco HyperFlex Systems Upgrade Guidefor VMware ESXi Release 40 Do not skip this cluster bootstrap step it is required for all upgradesuntil HX Release 35(1a) Auto bootstrap is supported beginning with HX release 35(1a) For moreinformation see the Auto Bootstrap Upgrade Process from HX Connect UI in the Cisco HyperFlexSystems Upgrade Guide for VMware ESXi Release 40

Manual bootstrap is not supported on Intersight clusters

bull Initiating Upgrade―Use the HX Connect UI or CLI stcli commands when upgrading from 25(1a)or later releases Use either the CLI stcli commands or the HX Data Platform Plug-in to the vSphereWeb Client when upgrading from a pre-25(1a) release The vCenter plug-in should not be used forupgrades starting with the 25(1a) release The vCenter plug-in should not be used for upgrades startingwith the 25(1a) release

If the current cluster version is at 35(1a) or above you do not need to use the stcli command Directupgrade to 40 is possible

bull Complete your Upgrade―The self-healing (or rebalance) capability is disabled temporarily during theupgrade window If the upgrade fails you should complete the upgrade as soon as possible

bull ESXi and HXDP Compatibility―Ensure your cluster is running a compatible version of ESXi basedon the running the HXData Platform version (see the section Software Requirements for VMware ESXi)ESXi compatibility is determined by the major version and update release of ESXi It is generally bestto upgrade HXDP and ESXi together if combining the upgrade operations into a single optimized rebootWhen running a split upgrade first upgrade the HX Data Platform then proceed to upgrade ESXi

bull If Upgrading to vSphere 65

bull Certain cluster functions such as native and scheduled snapshotsReadyClones and Enter or Exit HX Maintenance Mode will not operatefrom the time the upgrade is started until the HX Data Platform upgrade iscomplete

bull After upgrading ESXi using the offline zip bundle use the ESX ExitMaintenance Mode option The HX Exit Maintenance Mode option doesnot operate in the vSphere Web Client until the HX Data Platform upgradeis complete

Note

Release Notes for Cisco HX Data Platform Release 4020

Guidelines and Limitations

bull vSphere 60 UpgradesmdashUsers on vSphere 60 migrating to 65 upgrade components in the followingorder

1 Upgrade HX Data Platform and UCS firmware

2 Upgrade HX Data Platform and ESXi

3 Upgrade HX Data Platform only first then upgrade ESXi or UCS firmware or both

bull M4 Server Firmware UpgradesmdashUpgrade server firmware to ensure smooth operation and to correctknown issues Specifically newer SAS HBA firmware is available in this release and is recommendedfor long-term stability

bull Users are encouraged to upgrade to 31(3c) C-bundle or later whenever possible

bull Users running C-bundle versions before 31(2f) must upgrade server firmware by performing acombined upgrade of UCS server firmware (C-bundle) to 31(3c) or later and HX Data Platform to25 Do not split the upgrade into two separate operations

bull If the cluster is already on 31(2f) C-bundle or later you may perform an HX Data Platform onlyor combined upgrade as required

bull M5 Server Firmware UpgradesmdashM5 generation servers must run firmware version 32(2d) or later

bull Firmware Downgrades mdashDowngrading UCSM from the HX-installer is not supported

bull M4M5 Mixed DomainsmdashA mixed domain occurs when a new separate M5 cluster is installed underthe same UCS domain that contains existing M4 clusters Under these conditions orchestrated UCSserver firmware upgrade will not operate until Cisco HX Data Platform Release 26 or later is installedon the M4 clusters Therefore it is best practice to first upgrade UCS server firmware to the latest 31(3)or 32(2) patch release before adding a new M5 cluster to the existing UCS domain Additionally any17 HX Data Platform clusters must first be upgraded before adding any new M5 clusters to the samedomain

bull Maintenance WindowmdashIf upgrading both HX Data Platform and UCS firmware you can select eithera combined or split upgrade through the vSphere HX Data Platform Plug-in depending on the length ofthe maintenance window Cisco UCSManager infrastructure upgrade is only supported using AutoInstalland the direct server firmware upgrade should be performed only through the upgrade orchestrationframework provided by the HX Data Platform Plug-in

bull Unsupported Self-Encrypting Drives (SEDs)mdashIf adding or replacing self-encrypting drives (SEDs)that have been recently qualified in newer versions of HX Data Platform insert the new drives only afterupgrading HX Data Platform to a compatible version All drives must be SED drives mixing SED andnon-SED is not supported

bullbull Enabling External Host AccessmdashWith Cisco HX Data Platform Release 40(1a) port 445 on themanagement network is blocked for enhanced security Note that prior to 40 port 445 port was openenabling external host access If you are upgrading to 40(1a) from a prior release and would like tocontinue external host access you can use a utility to open select hosts For more information aboutenabling external host access see the ConfiguringHyperFlex Share to SCVMM section in the InstallationGuide for Microsoft Hyper-V

Release Notes for Cisco HX Data Platform Release 4021

Guidelines and Limitations

Mixed Cluster Expansion Guidelinesbull Expanding existing M4 cluster with M5 converged nodes is supported

bull Expanding existing M5 cluster with M4 converged nodes is not supported

bull Expanding existing mixed M4M5 cluster with M4 or M5 converged nodes is supported

bull Adding any supported compute-only nodes is permitted with all M4 M5 and mixed M4M5 clustersusing the HX Data Platform 26 or later Installer Some example combinations are listed here manyother combinations are possibleExample combinationsExpand mixed M4M5 cluster with compute-only B200 C220 C240 M4M5Expand M4 cluster with compute-only B200 M5 C220 M5 C240M5

bull Only expansion workflow is supported to create a mixed cluster Initial cluster creation with mixedM4M5 servers is not supported

bull All M5 servers must match the form factor (220240) type (HybridAF) security capability (Non-SEDonly) amp disk configuration (QTY capacity and non-SED) of the existing M4 servers

bull HX220-M5 will use a maximum of 6 capacity disks (2 disk slots to remain empty) when mixedwith HX220-M4

bull HX Edge SED LFF Hyper-V and Stretched Clusters do not support mixed M4 and M5 clusters

Security FixesThe following security issues are resolved

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-13132 CVE-2019-9924CSCvq6313840(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-11719 CVE-2019-11727CVE-2019-11729

CSCvq7124040(2a)

Release Notes for Cisco HX Data Platform Release 4022

Mixed Cluster Expansion Guidelines

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-1125CSCvr0633940(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2015-9383 CVE-2018-14498CVE-2018-20406 CVE-2018-20852CVE-2019-10160 CVE-2019-13117CVE-2019-13118 CVE-2019-14287CVE-2019-14973 CVE-2019-15903CVE-2019-17546 CVE-2019-18197CVE-2019-18218 CVE-2019-5010CVE-2019-5094 CVE-2019-5481CVE-2019-5482 CVE-2019-9636CVE-2019-9740 CVE-2019-9947CVE-2019-9948

CSCvs0609440(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2017-13168 CVE-2017-18174CVE-2017-18216 CVE-2018-10876CVE-2018-10877 CVE-2018-10878CVE-2018-10879 CVE-2018-10880CVE-2018-10881 CVE-2018-10882CVE-2018-10902 CVE-2018-10938CVE-2018-12233 CVE-2018-12896CVE-2018-13053 CVE-2018-13094CVE-2018-13096 CVE-2018-13405CVE-2018-13406 CVE-2018-14609CVE-2018-14617 CVE-2018-14633CVE-2018-14734 CVE-2018-15572CVE-2018-15594 CVE-2018-16276CVE-2018-16658 CVE-2018-17182CVE-2018-17972 CVE-2018-18021CVE-2018-18690 CVE-2018-18710CVE-2018-6554 CVE-2018-6555CVE-2018-9363

CSCvp6501940(2a)

This vulnerability is due to insufficientHTML iframe protection An attackercould exploit this vulnerability bydirecting a user to an attacker-controlledweb page that contains amaliciousHTMLiframe A successful exploit could allowthe attacker to conduct clickjacking orother clientside browser attacks

NACSCvo9851640(2a)

Release Notes for Cisco HX Data Platform Release 4023

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerability is due to insufficientauthentication for the statistics collectionservice An attacker could exploit thisvulnerability by sending properlyformatted data values to the statisticscollection service of an affected deviceA successful exploit could allow theattacker to cause the web interfacestatistics view to present invalid data tousers

NACSCvj9558440(2a)

The vulnerability is due to insufficientCSRF protections for the web UI on anaffected device An attacker could exploitthis vulnerability by persuading a user ofthe interface to follow a malicious linkA successful exploit could allow theattacker to perform arbitrary actions withthe privilege level of the affected user

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvp2434340(2a)

Vulnerabilities with a version of cJSONidentified by CVE-2019-11834 andCVE-2019-11835

For more information see the relatedCisco Security Advisory

CVE-2019-11834 CVE-2019-11835CSCvq1994940(2a)

HX ESXi image patches for MCEPSCand TAA vulnerabilities with VMwareESXi identified by CVE-2018-12207 andCVE-20190-11135

For more information see the relatedCisco Security Advisory

CVE-2018-12207 CVE-2019-11135CSCvr5439840(2a)

CP networking vulnerabilities affectingthe Linux kernel identified byCVE-2019-11477 CVE-2019-11478CVE-2019-11479

For more information see the relatedCisco Security Advisory

CVE-2019-11477 CVE-2019-11478CVE-2019-11479

CSCvq1954640(2a)

Qualification of Microsoft securitypatches for vulnerabilities withMicrosoftHyper-V hypervisor identified withCVE-2018-12207 CVE-2019-11135

For more information see the relatedCisco Security Advisory

CVE-2018-12207 CVE-2019-11135CSCvr5439940(2a)

Release Notes for Cisco HX Data Platform Release 4024

Security Fixes

DescriptionCVEDefect IDRelease

OpenSSH vulnerabilities identified byCVE-2019-6111

For more information see the relatedCisco Security Advisory

CVE-2016-10708 CVE-2018-15473CVE-2018-20685 CVE-2019-6109CVE-2019-6111

CSCvp7646340(2a)

Multiple vulnerabilities associated withOpenSSL and LibNSS

For more information see the relatedCisco Security Advisory

CVE-2016-2105 CVE-2016-2106CVE-2016-2107 CVE-2016-2109CVE-2016-2176 CVE-2016-2177CVE-2016-2178 CVE-2016-2179CVE-2016-2180 CVE-2016-2181CVE-2016-2182 CVE-2016-2183CVE-2016-6302 CVE-2016-6303CVE-2016-6304 CVE-2016-6305CVE-2016-6306 CVE-2016-6307CVE-2016-6308 CVE-2016-7055CVE-2016-8610 CVE-2017-3731CVE-2017-3732 CVE-2017-3735CVE-2017-3736 CVE-2017-3737CVE-2017-3738 CVE-2018-0495CVE-2018-0732 CVE-2018-0734CVE-2018-0735 CVE-2018-0737CVE-2018-0739 CVE-2018-12384CVE-2018-12404 CVE-2018-5407CVE-2019-1559

CSCvp6667940(2a)

libpng vulnerability identified byCVE-2016-10087

For more information see the relatedCisco Security Advisory

CVE-2016-10087CSCvp6655540(2a)

Vulnerabilities associated with libjpg andlibpng

For more information see the relatedCisco Security Advisory

CVE-2014-9092 CVE-2016-3616CVE-2017-15232 CVE-2018-11212CVE-2018-11213 CVE-2018-11214CVE-2018-1152 CVE-2018-13785

CSCvp3458640(2a)

Vulnerabilities associated with Glibidentified by CVE-2018-16428 andCVE-2018-16429

For more information see the relatedCisco Security Advisory

CVE-2018-16428 CVE-2018-16429CSCvp3120740(2a)

Multiple vulnerabilities associated withcurl expat python 27 python 35 36and 37 freetype and giflib

For more information see the relatedCisco Security Advisory

CVE-2019-15133 CVE-2019-15903CVE-2019-5010 CVE-2019-5481CVE-2019-5482 CVE-2019-9636CVE-2019-9740 CVE-2019-9947CVE-2019-9948

CSCvr3690340(2a)

Release Notes for Cisco HX Data Platform Release 4025

Security Fixes

DescriptionCVEDefect IDRelease

Multiple third party softwarevulnerabilities

For more information see the relatedCisco Security Advisory

CVE-2019-14379 CVE-2019-12384CVE-2019-14439

CSCvq9203240(2a)

Vulnerabilities associated with elfutils

For more information see the relatedCisco Security Advisory

CVE-2018-16062 CVE-2018-16402CVE-2018-16403 CVE-2018-18310CVE-2018-18520 CVE-2018-18521CVE-2019-7149 CVE-2019-7150CVE-2019-7665

CSCvq4325040(2a)

Vulnerabilities associated with vim

For more information see the relatedCisco Security Advisory

CVE-2017-5953 CVE-2019-12735CSCvq4323040(2a)

Vulnerabilities associated with sqlite3

For more information see the relatedCisco Security Advisory

CVE-2016-6153 CVE-2017-10989CVE-2017-13685 CVE-2017-2518CVE-2017-2519 CVE-2017-2520CVE-2018-20346 CVE-2018-20505CVE-2018-20506 CVE-2019-8457CVE-2019-9936 CVE-2019-9937

CSCvq4321340(2a)

Vulnerabilities associated with glib20

For more information see the relatedCisco Security Advisory

CVE-2018-20843CSCvq4320940(2a)

Vulnerabilities associated with expat

For more information see the relatedCisco Security Advisory

CVE-2018-20843CSCvq4320540(2a)

Vulnerabilities associated with bzip2

For more information see the relatedCisco Security Advisory

CVE-2016-3189 CVE-2019-12900CSCvq4319440(2a)

Mulitple Vulnerabilities associated wtihNodeJS

For more information see the relatedCisco Security Advisory

CVE-2018-12115 CVE-2018-0734CVE-2018-5407 CVE-2018-12120CVE-2018-12121 CVE-2018-12122CVE-2018-12123 CVE-2018-12116CVE-2019-5737 CVE-2019-5739

CSCvq1069440(2a)

Vulnerabilities associated with PalletsJinja strformat_map and identified byCVE-2019-10906 and CVE-2016-10745

For more information see the relatedCisco Security Advisory

CVE-2019-10906 CVE-2016-10745CSCvq1038840(2a)

Release Notes for Cisco HX Data Platform Release 4026

Security Fixes

DescriptionCVEDefect IDRelease

Multiple Security Vulnerabilities

For more information see the relatedCisco Security Advisory

CVE-2014-8501 CVE-2014-9939CVE-2015-9262 CVE-2016-10087CVE-2016-2226 CVE-2016-4487CVE-2016-4488 CVE-2016-4489CVE-2016-4490 CVE-2016-4491CVE-2016-4492 CVE-2016-4493CVE-2016-6131 CVE-2018-10963CVE-2018-13785 CVE-2018-17100CVE-2018-17101 CVE-2018-18557CVE-2018-18661 CVE-2018-7456CVE-2018-8905

CSCvq0917840(2a)

Vulnerabilities associated withlibseccomp

For more information see the relatedCisco Security Advisory

CVE-2019-9893CSCvq0756840(2a)

Vulnerabilities associated withgdk-pixbuf

For more information see the relatedCisco Security Advisory

CVE-2017-12447CSCvq0675540(2a)

Vulnerabilities associated withpython-crypto

For more information see the relatedCisco Security Advisory

CVE-2018-6594CSCvp9381740(2a)

Vulnerabilities associated with wget

For more information see the relatedCisco Security Advisory

CVE-2018-20483 CVE-2019-5953CSCvp8672140(2a)

Multiple vulnerabilities associated withPython Crypto

For more information see the relatedCisco Security Advisory

CVE-2018-6594CSCvp6674840(2a)

Vulnerabilities associated withsensible-utils

For more information see the relatedCisco Security Advisory

CVE-2017-17512CSCvp6673440(2a)

Multiple vulnerabilities associated withPython

For more information see the relatedCisco Security Advisory

CVE-2018-1000030CSCvp6668940(2a)

Release Notes for Cisco HX Data Platform Release 4027

Security Fixes

DescriptionCVEDefect IDRelease

Multiple patch vulnerabilities identifiedby CVE-2016-10713

For more information see the relatedCisco Security Advisory

CVE-2016-10713CVE-2018-1000156 CVE-2018-6951

CSCvp6667240(2a)

Vulnerabilities associatedwith Little CMSidentified by CVE-2016-10165

For more information see the relatedCisco Security Advisory

CVE-2016-10165 CVE-2018-16435CSCvp6666440(2a)

Vulnerabilities associated with TenableScan identified byCVE-2011-5325CVE-2019-11068CVE-2016-7076CVE-2017-1000368

For more information see the relatedCisco Security Advisory

CVE-2011-5325 CVE-2016-7076CVE-2017-1000368CVE-2019-11068

CSCvp6474640(2a)

Vulnerabilities associated with libxml2294 and earlier as used in XMLSec1223 and earlier and other products andidentified by CVE-2016-9318

For more information see the relatedCisco Security Advisory

CVE-2016-9318 CVE-2017-16932CVE-2017-18258 CVE-2018-14404CVE-2018-14567

CSCvp3479240(2a)

Vulnerabilities associated with LFTPRemote File Names Unauthorized AccessVulnerability and identified byCVE-2018-10916

For more information see the relatedCisco Security Advisory

CVE-2018-10916CSCvp2926640(2a)

Vulnerabilities associated withCryptographyDeprecationWarning signerand verifier have been deprecated

For more information see the relatedCisco Security Advisory

CVE-2018-7750CSCvo3409740(2a)

Multiple Vulnerabilities in python python27 35

For more information see the relatedCisco Security Advisory

CVE-2019-16056CSCvr3979340(2a)

Release Notes for Cisco HX Data Platform Release 4028

Security Fixes

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toexecute commands as the root user

The vulnerability is due to insufficientinput validation An attacker could exploitthis vulnerability by connecting to thecluster service manager and injectingcommands into the bound process Asuccessful exploit could allow the attackerto run commands on the affected host asthe root user

Cisco has released software updates thataddress this vulnerability There areworkarounds that address thisvulnerability

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvq2417640(1b)35(2g)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

The vulnerability is due to an unprotectedlistening interface An attacker couldexploit this vulnerability by connectingto the listening interface and injectingcommands to the bound process Anexploit could allow the attacker to runcommands on the affected host as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvj9560640(1b)35(2g)

Release Notes for Cisco HX Data Platform Release 4029

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerabilities associated with theJVM18U121memory leak found duringVC alarms (concurrent 40 calls usingREST API)

CVE-2017-10053 CVE-2017-10067CVE-2017-10074 CVE-2017-10078CVE-2017-10081 CVE-2017-10087CVE-2017-10089 CVE-2017-10090CVE-2017-10096 CVE-2017-10101CVE-2017-10102 CVE-2017-10107CVE-2017-10108 CVE-2017-10109CVE-2017-10110 CVE-2017-10111CVE-2017-10115 CVE-2017-10116CVE-2017-10118 CVE-2017-10135CVE-2017-10176 CVE-2017-10193CVE-2017-10198 CVE-2017-10243CVE-2017-10274 CVE-2017-10281CVE-2017-10285 CVE-2017-10295CVE-2017-10345 CVE-2017-10346CVE-2017-10347 CVE-2017-10348CVE-2017-10349 CVE-2017-10350CVE-2017-10355 CVE-2017-10356CVE-2017-10357 CVE-2017-10388CVE-2017-3509 CVE-2017-3511CVE-2017-3526 CVE-2017-3533CVE-2017-3539 CVE-2017-3544CVE-2018-2579 CVE-2018-2582CVE-2018-2588 CVE-2018-2599CVE-2018-2602 CVE-2018-2603CVE-2018-2618 CVE-2018-2629CVE-2018-2633 CVE-2018-2634CVE-2018-2637 CVE-2018-2641CVE-2018-2663 CVE-2018-2677CVE-2018-2678 CVE-2018-2783CVE-2018-2790 CVE-2018-2794CVE-2018-2795 CVE-2018-2796CVE-2018-2797 CVE-2018-2798CVE-2018-2799 CVE-2018-2800CVE-2018-2814 CVE-2018-2815CVE-2018-2952 CVE-2018-3136CVE-2018-3139 CVE-2018-3149CVE-2018-3150 CVE-2018-3169CVE-2018-3180 CVE-2018-3183CVE-2018-3214 CVE-2019-2422

CSCvo8899740(1b)

Tomcat and Nginx logs are not beingcollected in the support bundle generatedthrough HX Connect in Release 35

NACSCvm5803140(1b)

The vulnerabilities associated with thelibmspack software package versionincluded in Cisco HX Data Platform

CVE-2018-18584

CVE-2018-18585

CSCvn3511940(1a)

Release Notes for Cisco HX Data Platform Release 4030

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerabilities associated withFasterXML Jackson-Databind TimeValueField Denial of Service

CVE-2018-14719

CVE-2018-14720

CVE-2018-1000873

CVE-2018-14721

CVE-2018-19360

CVE-2018-19362

CVE-2018-19361

CVE-2018-14718

CSCvn8228240(1a)

The vulnerabilities associated with theOpenSSL Protocol software packageversion included in Cisco HX DataPlatform

CVE-2013-3587CSCvo0505440(1a)

The vulnerabilities associated withThird-Party Software Denial of Service

CVE-2018-16487

CVE-2018-19361

CSCvo2781840(1a)

The vulnerabilities associated withLinuxkernel for Ubuntu 1710

CVE-2018-1092

CVE-2018-7492

CVE-2018-8087

CVE-2018-1068

CVE-2018-8781

CSCvm5314935(2a)

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)25(1d)stcli services dns remove should remove the DNS serverinfo from the interface files

CSCvs70967

40(2b)40(1b)HX 40(1b) new deployment from Intersight productioncloud

User receives following error while enabling CSI(Kubernetes integration)

Failure occurred during volume_access_enable Errorwas Nonzero exit code 1

CSCvt10522

40(2b)40(2a)Receive the following alertevent in HX Connect HXController VM HOSTNAME one or more configuredDNS servers not responding

CSCvt13947

Release Notes for Cisco HX Data Platform Release 4031

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)40(2a)API UI CLI show few Drive slots Empty after upgradeto 402a

The drives are seen in the sysmtool lsscsi and stclicommand outputs

The cluster is healthy with no errors

CSCvt14914

40(2b)35(2d)After successfully changing the DNS server on the HXcontroller we still can see the original DNS entry thatwas added during the deployment

CSCvs69154

40(2b)35(2e)HyperFlex Connect and VCenter show APD alarms forHX and non HX datastores This implies there is an issuewith HyperFlex when there may not be

CSCvs30080

40(2b)40(2a)An issue where all the disks in the node get locked afterrebooting the node

We successfully unlocked using sed-clientsh -Ucommand but wanted to test with another reboot anddrives locked again

CSCvs74286

40(2b)40(2a)When running stcli license commands on HyperFlexthe following error is seen

rootSpringpathControllertmp stcli license show all

Show smart licensing failed

Smart Agent is not ready please wait a minute and tryagain

CSCvt13929

40(2b)40(2a)After upgrading to HX 40(2a) the first RoboEdge nodethat gets rebooted then ignores persistent drivesPIDHX-HD24TB10K4KN

Cluster will be in degraded state and have reducedcapacity

CSCvt20144

40(2b)35(2b)Zookeeper fails to start while Exhibitor is runninghowever echo srvr returns nothing

CSCvs21562

40(2b)40(2a)When performing an HyperFlex upgrade a validationwarning may occur due to the host not having EnterprisePlus or Enterprise hypervisor licensing

Upgrade Validation Warning

ESXi host esx1labtest should be configured withVMware Enterprise license for upgrade to continue

CSCvs91787

Release Notes for Cisco HX Data Platform Release 4032

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)35(2g)Cluster expansion fails at Config Installer stage whenthe root and admin password for the storage controller(SCVM) are different

CSCvs69317

40(2b)35(2g)Panic while upgrading ESXiCSCvt06983

40(2b)40(2a)A cluster node running may hang in the Linux kernelThis is classified as an oops and a deviation from theexpected behavior

CSCvs54285

40(2b)35(2g)Rebalance failing on 10+10 node HX 35(2g) stretchcluster

CSCvs69007

40(2b)35(2d)The cluster becomes inaccessible to the IOVisorCSCvr54687

40(2b)40(2a)Panic on the storage controllerCSCvt61297

40(2b)40(2a)The size of support bundle is very large when collectedwith storfs-support command

CSCvt63306

40(2b)40(2a)storfs crash associated withFileSystemUsageWarningEvent orFileSystemUsageAlertEvent due to high usage ofvarstv above 80

CSCvt72807

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

40(2a)35(2d)In normal scenarios HX upgrade gets pendingACK afterupdating service profile In this case UCS only upgradewas stalled because after updating service profile HXupgrade was not getting pending ACK and it was alwayswaiting state

CSCvo39912

40(2a)35(1a)When using motherBoardReplace-12 to clean ZK fromold stNodepNodes resulted in unmounted datastore andmaking the size of the Hyperflex Datastores 0 resultingin all VMs in the cluster going offline

CSCvq39471

40(2a)26(1e)Upgrading cluster and getting error

Failed upgrade validations Checking vCenterconfiguration Reason Upgrade validations failed DRSFault Insufficient resources to satisfy configured failover

CSCvm77294

Release Notes for Cisco HX Data Platform Release 4033

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)The cluster expand fails on a node with DR replicationconfigured

When a HX cluster which has DR replication configuredis expanded we see the installer UI pulling in thereplicationVLAN information instead of themanagementVLAN information Even if we change that informationto the correct mgmt VLAN id and name it it does notseem to work as the node is configured with the VLANof the replicationVLAN in ESXi This leads to the failureof the node add with the host unreachable error

CSCvo70650

40(2a)35(1a)Customer reported that server didnt complete thefirmware upgrade automatically one by one

The user manually needs to put the host into maintenancemode and they manually acknowledged the pendingrequests to finish the UCS firmware upgrade

Maintenance policy is set to default (User-ack) as perthe design

CSCvo91624

40(2a)35(1i)When the cluster is in Failed state and stcli noderemove is attempted the output shows successful thoughthe node failed to get removed from the cluster

CSCvo93017

40(2a)35(1a)HyperFlex cluster upgrade may fail during validationwith a DRS Validation failed error

CSCvp31021

40(2a)35(2b)HX cluster expansion will fail with the following errormessage

MAC address pool configuration failure150[ErrorDescription] bad address block range definitioncollision

CSCvp58318

40(2a)35(2b)Memory usage by carbon cacheCSCvq34873

40(2a)35(2g)Upgrading ESXi cluster fails with error Nodemaintenance mode failed

CSCvr03240

40(2a)35(2e)storfs process does not automatically start on Exit HXMaintenance Mode or other tasks which restart StorageController

CSCvr88978

40(2a)35(2a)Post install script fails with message

Failed to execute ipmitool on HX node

CSCvp10707

40(2a)40(1a)HyperFlex expansionworkflow doesnt pull VLANnamecorrectly

CSCvp46539

Release Notes for Cisco HX Data Platform Release 4034

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)During HX cluster deployment - Validate ClusterCreation phase - HX Installer might fail with the error

fromvarlogspringpathstDeploylog

CSCvq45087

40(2a)35(2b)HX Installer fails to login to the SOL for configuringESX hosts

CSCvq91380

40(2a)35(2d)If DRS is enabled with configuration parameters otherthen fully automated ie manual or partially automatedthan this will change to fully automated during clusterexpansion

CSCvr44222

40(2a)35(1a)One of the validation task name has a typo The taskname is Validating node for mixed of SED and non-SEDdisks

It should be Validating node for mix of SED andnon-SED disks

CSCvp82175

40(2a)30(1i)This combination of using smaller NVMe drives is notsupported in expand operation For example if the drivesare only 375GB and they cannot be added to an existingcluster with larger caching SSD

CSCvo12359

40(2a)35(1a)HyperFlex includes a version of OpenSSL that is affectedby the vulnerabilities identified by the followingCommon Vulnerability and Exposures

CVE-2018-0495

CSCvp66679

40(2a)30(1e)SDK invoke exception when taking a native snapshot ona VM with GPU pci passthrough attached

CSCvo00511

40(2a)30(1i)Node replacement script fails due to EAM errorCSCvo62867

40(2a)35(1a)While pairing a cluster (HX release gt= 35) and cluster(HX release lt 35) remote replication network test failson the cluster (HX release lt 35)

CSCvo79760

40(2a)30(1b)Fixed the issue in the latest support-workflow bundleThis should not be hit anymore

CSCvo87061

40(2a)35(2a)On a 3 node cluster the MBreplace script fails as thescript looks for HEALTHY cluster status

CSCvo87080

40(2a)35(2a)MbReplace script (tar) hangs when running on 352b or40

CSCvp12359

Release Notes for Cisco HX Data Platform Release 4035

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)HX replication cleanup failing with error NA 35(2a) -INFODR state is not clean

CSCvp63958

40(2a)40(1a)The check_stig_parameters API incorrectly shows thecompliance state of the cluster as not STIG compliant

CSCvp66277

40(2a)35(2c)The snapshot creation on CBT enabled VM fails witherror Failed in vmreparent vmkfstools clone1

CSCvq06952

40(2a)35(2a)HyperFlex GUI when setting up remote DR-replicationpartnerpeer should show remote replication partner TCPport reachability test results

This will provide quick upfront connectivity results thatfrequently takes customer a while to manually check andconfirm

If reachability tests fail an info window and HX alertshould trigger that list the destination peer IP and thespecific port(s) that could not be reache

CSCvo60587

40(2a)35(2b)If you have more than one IP pool configured for areplication network and local replication network testsare failing you might be running into this issue

CSCvr67130

40(2a)35(2a)Skip task option in MBreplace and Software redeployscripts The scripts should have the option to skip tasksas user inputs and not by modifying the json file

CSCvp05204

40(2a)40(1a)Error noticed on SCVM Console print_req_error IOerror dev fd0 sector 0 on HyperV

It doesnt come up on SSH session

CSCvq34357

40(2a)30(1d)The VMNetwork switch is already created as part of theinstall process

Ideally the ip address assignment should be done duringthe initial install process as well No post install stepsshould be required altrnatively it could be done as partof a post install script

CSCvk36222

40(2a)25(1c)storfs service impacted during HX upgrade from HX21(1) to 25(1c)

HX node which is not being upgraded storfs servicestopped causing outage

CSCvg53223

40(2a)35(1b)HX Stretch Cluster Witness VM reverts to DHCP onreboot

CSCvp37536

40(2a)35(2b)HyperFlex Stretch Cluster Witness does not getprogrammed with an NTP server

CSCvp46578

Release Notes for Cisco HX Data Platform Release 4036

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2b)HyperFlex cluster does not heal after maintenance onnodeserver

CSCvr18528

40(2a)40(2a)High percentage of packet loss on the eth1 data interface9K MTU

CSCvr97089

40(2a)35(2f)The HyperFlex cluster healing state is stuck at 87 andis unable to progress

CSCvr16760

40(2a)40(2a)In some circumstances two replications can occur forthe same VM

From the GUI the replication status for this VM willshow stuck in an In Progress state but there are nojobs stuck in replication layer Subsequent Replicationsshould succeed

CSCvq22898

40(2a)35(2a)After an unexpected power outage the hyperflex clusteris unable to come up correctly

All processes are running time is synced and all nodescan vmkping each other as required

CSCvp13990

40(2a)40(1a)When specifying IP address instead of node ID stclinode maintenanceMode commands fail

CSCvp33657

40(2a)40(1a)2 node edge cluster if experiencing Intersightarbitratorconnectivity issues should NOT allowHXMaintenancemode to initiate as this will bring the cluster down

CSCvr31573

40(2a)35(2c)Old files in varsupportZKTxnlog are not purged withthe daily zklog-cleanup cron job

CSCvr89066

40(2a)35(2a)When a node is in MaintenanceMode any disk removalor replacement will be reflected in UI only after the nodeis brought back frommaintenancemode This is becausestorfs is not running on the node in maintenance andwill not be able to detect disk activities until it is broughtout of Maintenance Mode

CSCvo86431

40(2a)35(2c)Cluster might go offline and stop serving IO requestsrootcvm~ stcli cluster storage-summary--detailGet cluster storage summary failedjavanetConnectException Connection refused19216814210010207

CSCvr01645

40(2a)35(2d)A stretch cluster deployment fails with a messageFormatting nodes Node in Use on the Hyperflex installerand crmZoneType value is seen as 1 in storfscfg oroptspringpathconfigstretchtunes

CSCvq80340

Release Notes for Cisco HX Data Platform Release 4037

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)30(1c)During upgrade we upload the upgrade packagedownloaded from Ciscocom Upgrade packages are offile type tgz hxconnect acceptsstorfs-packages-401a-33028tar file a different type ofcompression and allows upgrade to start and fail whenit cannot decompress the file for bootstrap and scvmclientupgrade on ESXi

CSCvq63888

40(2a)40(2a)While attempting to install hyperflex via intersight thefollowing error is seen

Failed in Task Add hosts to vCenter Cluster with ErrorTry adding hosts manually to vCenter and retry Failedto get addressFailed to host by namelookupHostnamecompanycom on 000053 read udp12700145430-gt12700153 read connection refused

CSCvr52098

40(2a)35(2a)CoreAPI call may time out for cluster management APIwhile doing cluster create validation

CSCvo92952

40(2a)35(2d)Validation fails with Mixed mode expansion checkdue to Enhanced vMotion Compatibility Incompatibilitywhen the ESXi version is different

CSCvq95460

40(2a)30(1i)This failure is seen in installer when custom workflowis used hypervisor configuration + deploy (clean diskpartition=no) at Installing software packages on storagecontroller VM

CSCvr66309

40(2a)35(2d)The HX Installer apparently replaced a comma bysemicolon on the VLAN ID range while exporting theconfiguration to a JSON file

CSCvq93831

40(2a)40(1a)When using unsupported browsers HX Connect userswill not be able to login These users see an errorindicating unauthorized user

CSCvp22693

Release Notes for Cisco HX Data Platform Release 4038

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)40(1a)A vulnerability in the web-based management interfaceof Cisco HyperFlex Software could allow anunauthenticated remote attacker to conduct a cross-siterequest forgery (CSRF) attack on an affected system

The vulnerability is due to insufficient CSRF protectionsfor the web UI on an affected device An attacker couldexploit this vulnerability by persuading a user of theinterface to follow a malicious link A successful exploitcould allow the attacker to perform arbitrary actions withthe privilege level of the affected user

Cisco has released software updates that address thisvulnerability There are no workarounds that address thisvulnerability

This advisory is available at the following link

httpstoolsciscocomsecuritycentercontentCiscoSecurityAdvisorycisco-sa-20190807-hypflex-csrf

CSCvp24343

40(2a)35(2d)Add message to HX Connect Progress flow to notAcknowledge Pending Activities and reboot the serverson UCSM

HX Connect is doing a controlled rolling server upgradein the background

CSCvq22844

40(2a)40(1b)Compute Node expand operation fails due to incorrectcluster name

CSCvr43786

40(2a)26(1e)RF-2 cluster shutdown during data resynchronizationafter a node failure (non storfs) followed by multipledisk read failures and a HardBlacklist of a disk

CSCvp41241

40(2a)40(1a)HyperFlex Edge nodes do not properly set the ESXihostname during deployment

CSCvo13143

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4039

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4040

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1b) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401b-33133p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4041

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4042

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1b)40(1a)While running the HyperFlex installer with WindowsServer Hyper-V cluster creation process fails with theerror Failure occurred during ClusterCreation process Unable to post thecontent to the downstream This symptomis encountered while deploying HyperFlex cluster withHyperFlex nodes configuredordered with Cisco VIC1457 MLOM (PID HX-MLOM-C25Q-04) andWindows Server Datacenter or Core with Hyper-Vversion 2016 or 2019

CSCvp64140

Release Notes for Cisco HX Data Platform Release 4043

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)35(2b)Adding Micron 5200 drive to a cluster fails to increasecluster capacity

CSCvo69067

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4044

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4045

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1a) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401a-33028p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4046

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4047

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1a)30(1d)Cluster instability when disks of different sector sizeplaced in HX node

CSCvk17250

40(1a)

35(2c)

35(1a)When logged in using a local HX user account insteadof a Virtual Center account an error message appearsintermittently indicating Virtual Centerunreachable or Resource informationcannot be updated when VC is reachable

CSCvo36198

40(1a)

35(2b)

30(1d)HXDP does not work with EMC RecoverPoint - needsto support VMware API (FSS-Readdir)

CSCvk38003

Release Notes for Cisco HX Data Platform Release 4048

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)

35(2a)

30(1i)

25(1c)

25(1d)

Zookeeper (Exhibitor) process on HyperFlex StorageController VMs may fail to respond to requests whenvarzookeeper has low or no free space

CSCvm90352

40(1a)26(1c)Use Asynchronous consolidation for HX SnapshotsCSCvn02151

40(1a)35(2b)HX Quiesced Snapshot for Backup VendorsCSCvo90713

40(1a)35(2a)The cluster creationcluster expansion workflow stopswith the following error message at the validation step

FIRMWARE-Check UCSC-SAS-M5HD

FIRMWARE-Check UCSC-SAS-M5HD Required00000029000000320000003500000050 Found00000058

Action Needed Update the Controller Firmware toRequired Version

CSCvn17787

35(2b)30(1d)Kernel migration fails when a local datastore is searchedfor in the ESXi host

CSCvn73127

35(2a)26(1b)A node shuts down when two disks are replaced (acaching disk and another capacity disk) where thecapacity disk is inserted first and then the caching diskis inserted

CSCvk46364

Hyper-V

40(1a)35(2a)Cluster expansion may fail with an error code 500 -operation timed out

CSCvn28721

35(2b)35(2a)During upgrade remove the VLAN on team created foruser vSwitch During fresh installation only one VLANtag is set to the vSwitch and team although multipleVLANs were entered

CSCvn54300

35(2b)35(2a)While upgrading a Hyper-V cluster on account of a rarerace condition between the stUpgradeService andZookeeper servers the upgrade orchestration throws anupgrade validation error and the upgrade process isaborted

CSCvn60486

Release Notes for Cisco HX Data Platform Release 4049

Resolved Caveats in Release 40(1a)

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Startrestart EAM service invCenter

When EAM service is not runningvCenter registration is failing

CSCvt55712

35(2a)Contact Cisco TAC if you believeyou are seeing similar conditions

HyperFlex Stretch Cluster saw poorapplication performance due to writelatency cluster remained unhealthy andrebalance was stuck

This defect is currently being used forinvestigative purposes

CSCvt45344

35(2g)Remove log files fromvarlogspringpath folder when thelog files are not getting rotatedStop the relevant service removelarge log files fromvarlogspringpath folder andrestart the service

varstv folder in Controller VM maybecome full

CSCvt36374

40(2b)Restarting the stMgr will initializeit again

In a Cluster configured with DR networkStMgr may not get initialized and stuckin a deadlock while enabling the IPTablerules (can be verified from the stMgrlog)

CSCvt89709

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Click recover for the selectedVM again under the standalonemode This will move the VM torecovered state

When the user recovers a VM protectedin a group it along with other VMs in thegroup move to recovering state instandalonemode The selectedVM shouldmove to recovered state

CSCvs75553

35(2e)Clean up the fail over cluster andthen recreate the fail over clusterNo need to touch the HX storagecluster

Hyper-V When replicated DC was usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2d)40(1a)

Keep cluster utilization as low aspossible To avoid read-only statekeep below 92

HyperFlex clusters storage utilization isabove 92 usable capacity

CSCvr20922

Release Notes for Cisco HX Data Platform Release 4050

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1b)NAAll the disks in the node got locked afterrebooting the node

We successfully unlocked usingsed-clientsh -U command but wantedto test with another reboot and driveslocked again

CSCvs74286

35(2g)To recover the VM copy over thedata disks and attach them to thenew VM

After enable encryption HX connectencryption status showing Caution butall of the disks enable encryptionsuccessful

-Self Encrypting Drives Service isrunning on all of the nodes

-There is no error message of encryption

- USB0 interface is up

-All SED disks showing supported 1enabled 1 locked 0

CSCvs93245

40(1b)Enable kernelpanic_on_oops as apersistent configuration This willcause the node to panic and rebootimmediately

A cluster node running HX release40(1b) may hang in the Linux kernelThis is classified as an oops and adeviation from the expected behavior

CSCvs54285

Release Notes for Cisco HX Data Platform Release 4051

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Enabling K8iscsi stack in Hx clusterCSCvs41324

Release Notes for Cisco HX Data Platform Release 4052

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Steps to enable K8 on HX cluster

Perform following steps on ALLthe controller VMs

1 Updateetcinitscvmclientconf toenable tune iscsiEnable totrue

sed -iesiscsiEnable=falseiscsiEnable=trueetcinitscvmclientconf

2 Run following initctlcommand to reloadconfiguration

initctl reload-configuration

3 Restart scvmclient process

stop scvmclient startscvmclient

4 Run following initctlcommand to mount SYSTEMDS

initctl emit --no-waitsystem-datastore-created

5 Verify that iscsiEnable tuneis set to true using thefollowing command

ps -eaf | grep scvmclient |grep -v grep

root 6241 1 1 Dec06 014007optspringpathstorfs-corescvmclient-T logEnabled=false -TlogSyslogEnabled=true -TlogEchoToScreen=false -TstatLoggingToFile=false -TstatLoggingToSyslog=true-TlogDir=varlogspringpath-TnfsBackendServerList=1010748100-T iscsiEnable=true -TiscsiUseAsync=true -T

Release Notes for Cisco HX Data Platform Release 4053

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

iscsiConfigFilePath=nfsSYSTEMistgtconfps -eaf | grep scvmclient |grep -v grep

35(2e)Following procedure can be usedto check NFS Queue Depth andincrease if needed

rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth256 lt- Low value[rootHXESXI1] vsish -e setvmkModulesnfsclientmnt[DATASTORE]propertiesmaxQDepth 1024[rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth1024 lt- Optimalvalue

This requires an ESXi host rebootto take effect

Please place the node in HyperflexMaintenanceMode and gracefullyreboot the node for the changes tobe applied

HyperFlex Datastore NFS Queue Depthshows as 256 which can lead toperformance (including latency) issues

CSCvr83056

40(1b)This is cosmetic The error will goaway once the cluster returns tohealthy status

When a single node is offline in clusterstcli cluster storage-summary shows twonodes unavailable rootSCVM~ stclicluster storage-summary

CSCvs47735

35(2g)Modify both the root and adminpassword for the SCVMs to be thesame

stcli security password set -uadmin

Cluster expansion fails at Config Installerstage when the root and admin passwordfor the storage controller (SCVM) aredifferent

CSCvs69317

35(2d)NAAfter successfully changing the DNSserver on the HX controller we still cansee the original DNS entry that was addedduring the deployment

CSCvs69154

Release Notes for Cisco HX Data Platform Release 4054

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Customers are strongly encouragedto work with TAC in order toidentify and remediate this issue

1 Change getcluster from 2m to8m inoptspringpathstorfs-mgmtstMgr-10confapplicationconfon all nodes

2 Restart stMgr on all nodes

You may see this error message after afailed upgrade or other task such asattempting to enter a node into HXmaintenance mode

CSCvs53555

40(2a)Run the following command oneach storage controller VM as root

grep -i monitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp sed -iesmonitor_dns_serverstruemonitor_dns_serversfalseoptspringpathhx-diag-toolswatchdog_configjsonampamp grep -imonitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp restart watchdog

Receive the following alertevent in HXConnect HX Controller VMHOSTNAME one or more configuredDNS servers not responding

CSCvt13947

35(2b)Delete any empty (Size 0) log filesunder varlogzookeeperversion-2ORvarlogzookeeperstandaloneversion-2

Zookeeper fails to start while Exhibitor isrunning however echo srvr returnsnothing

CSCvs21562

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

40(4e)1 De-commission the server

2 Power drain the server -REMOVE BOTH POWERCORDS ON THE BACK OFTHE SERVER FOR 10SECONDS THENREINSERTPOWERCORDS

3 Re-commission the server

M2 boot disk is missing from serverinventory after upgrade to server firmware40(4e) As a result server fails to boot toOS installed in the M2 disk The issuepersists after re-acknowledgement as wellas de-commission andre-acknowledgement of the server

CSCvs02466

Release Notes for Cisco HX Data Platform Release 4055

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Re-start the flusher manuallyMany disks taken offline by a fault ormanually resulting in ENOSPC oncluster

CSCvq32721

35(2e)Clean up fail over cluster and thenrecreate the fail over cluster Thereis no need to touch the HX storagecluster

[Hyper-V]When replicated DCwas usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2b)Manually install upgradescvmclient VIB andmake sure thatit matches with HXDP version

Upgrade from release HX 35(2a) to35(2x) 4X does not upgrade scvmclientRecovery point may not work

CSCvq54992

Management

30(1b)To recover the VM copy over thedata disks and attach them to thenew VM

[HyperV]VMshows up onmultiple nodesCSCvj22992

35(1a)1 Do not change MTU at ESXlevel on running cluster

2 Reset back to the original value

ChangingMTU from 9000 to 1500 on oneESX node causes storfs process to restarton all nodes

CSCvm99150

Release Notes for Cisco HX Data Platform Release 4056

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)After the installation (for newdeployments) or after an upgradeto 401a (for existingdeployments) perform thefollowing steps on all ControllerVMs

1 Edit the following tune files (onall Controller VMs)

optspringpathconfiglfftunes

optspringpathconfigvsi_16tbtunes

2 SetcleanerEnableSegSummaryCleaning

to false

3 After editing the above tunefiles

a SSH in to all the ControllerVMs

b Run storfstool -- -Z

c Run the below command andcheck the values Tune valueshould be true

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=true

d Type below commands andapplied the tunes changesdynamically

echo false gttmpstprocfssystemtunecleanerEnableSegSummaryCleaning

e Verify that tune values arechanged Run below commandValue should be false

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=false

f umount tmpstprocfs

A cluster with 8TB or 12TB disk drivesmay experience IO stalls for severalminutes after another node fails in thecluster

CSCvp23718

Release Notes for Cisco HX Data Platform Release 4057

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)1 Update the conf file manuallyto change to the link local addressto

2 Restart scvmclient on thecontroller and ESX

HX 35(2b) FlexVol to HX 40 CSIupgrade does not work FlexVol continuesto work

CSCvp26319

35(2a)Disable rebalance until node healsthen reenable rebalance

7-node cluster healing very slowlybecause rebalance change AMS reqs areblocking healing process

CSCvp62512

35(2a)NAWith moderate cluster storage capacityusage under site failover cluster clustermay experience an all paths (APD) downcondition

CSCvq49412

40(1a)NAThis product includes Third-partySoftware that is affected by thevulnerabilities identified by the followingCommon Vulnerability and Exposures(CVE) IDs

CVE-2016-0762 CVE-2016-6797CVE-2016-6816 CVE-2016-8735CVE-2017-5647 CVE-2017-12615CVE-2017-12616 CVE-2017-12617CVE-2017-7674 CVE-2018-1304CVE-2018-8014 CVE-2018-1336CVE-2018-8034 CVE-2018-11784CVE-2019-0232

CSCvp36364

35(2d)NAstcli cluster info command needs toprovide UCSM VIP address Currently itshows ucsm-hostcom

CSCvq11456

35(2a)No workarounds currently Thestale entries do not affect the dayto day operations of HyperFlex

HyperFlex upgrade validation failedbecause cluster has extraneous stNodes ininternal database

CSCvq32530

35(2a)NAWhen not in maintenancemode all drivesappear in HX Connect and get updated asexpected

When in maintenance mode extra drivesare shown in HX Connect with no slotnumbers

CSCvq42378

35(2b)Contact TAC to cleanup duplicatenode

Nodes duplicated in HX Connect aftercluster recreate

CSCvq61825

Release Notes for Cisco HX Data Platform Release 4058

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)NACurrently HyperFlex installs on Hyper-Vdo not contain the stcli security whitelistcommandset

CSCvq66245

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

Release Notes for Cisco HX Data Platform Release 4059

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)[Hyper-V] Converged node expansionfails for incoming node with VIC 14xx

CSCvq39523

Release Notes for Cisco HX Data Platform Release 4060

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

1 Set Jumbo frame inWindowsHypervisor for incomingnode

Get-NetAdapter storage-data-a|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

Get-NetAdapter storage-data-b|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

2 Verify all interfaces are setcorrectly to support Jumbo frames

Get-NetAdapter storage |Get-NetAdapterAdvancedProperty| RegistryKeyword -matchjumbo | ft -auto

You should receive the followingmessage

Name DisplayName DisplayValue

RegistryKeyword RegistryValue

---- ----------- ------------

--------------- -------------

vswitch-hx-storage-data Jumbo

Packet 9014 Bytes

JumboPacket 9014

storage-data-b Jumbo Packet

Bytes 9014 JumboPacket

9014

storage-data-a Jumbo Packet

Bytes 9014 JumboPacket

9014

3 Reboot only the incoming nodeor the node being expanded

4 Retry Cluster Expansion in

Release Notes for Cisco HX Data Platform Release 4061

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

bull Scaled-Up HyperFlex Edge ClustersmdashThis release adds support for four node HyperFlex Edge clustersenabling a full range of size options for remote and branch offices Size the branch office environmentsto suit current needs with a two three or four node HyperFlex Edge cluster Cisco Intersight providesfull-lifecycle management and 1GE and 10GE networking options are available

bull Cisco Intersight Invisible Cloud WitnessmdashFor two node clusters this feature eliminates the need forwitness VMs the infrastructure to run those VMs and the management overhead to deploy scale andpatch witnessing software The Invisible Cloud Witness is responsible for maintaining cluster HA in theevent of failure scenarios This feature is included at no additional cost and is automatically deployedand managed by Cisco Intersight

bull Cloud-delivered HyperFlex Edge UpgradesmdashPowered by Cisco Intersight this feature adds supportfor multi-site orchestrated remote upgrades of the HyperFlex Data Platform This feature will be enabledwith the next 40 patch release and will allowHyperFlex Edge clusters deployed via Intersight to performorchestrated upgrades across one or many sites in parallel

bull All-NVMe HyperFlexmdashStarting with this release a new high-end performance node powered by allNVMe drives is available for HyperFlex clusters Co-engineered with Intel to support Intel VMD forhot-plug and surprise removal this offering represents an industry first an enterprise-ready and fullyvalidated all-NVMe HCI appliance The all-NVMe offering is available in the 220 form factor (1RU)and is powered by Intel Optane cache drives for maximum performance and highest endurance

bull VMware Site Recovery Manager (SRM) IntegrationmdashThis release brings support for a Cisco developedStorage Replication Adapter (SRA) for SRM The SRA provides the ability to leverage HyperFlex nativeasync replication with the powerful orchestration and runbook capabilities of SRM The SRA includesthe ability to perform test recoveries planned migrations and full disaster recovery

HX SRA is certified by VMware and is available for download from VMWareSRM site

Note

bull HyperFlex DR Powershell RunbooksmdashNew Powershell cmdlets are included for automated runbookgeneration when using HyperFlex native disaster recovery The New-HXRunbook cmdlet supports thefollowing workflows Test Recovery Planned Migration and Disaster Recovery These runbooks canbe used to orchestrate DR workflows without the requirement for any third-party software

bull Windows Server 2019 with Hyper-VmdashSupport has been added in this release for the Windows Server2019 operating system for Hyper-V based HyperFlex deployments

bull Kubernetes CSI PluginmdashThis release adds support for the HyperFlex CSI (HX-CSI) plugin based onthe Kubernetes Container Storage Interface (CSI) specification Customers can now use the HX-CSIplugin to provision andmanage persistent volumes in Kubernetes version 113 and later Note Kubernetes113 support for Cisco Container Platform amp Openshift Container Platform is forthcoming in theirrespective future releases

bull C480 ML Compute only nodemdashThis release introduces support for C480 ML as a new compute-onlynode for Deep LearningMachine Learning Workloads Data scientists can now use the power of up toeight NVidia SXM2 V100 GPUs to accelerate deep learning workloads VMs running deep-learningworkloads will need to use PCIe pass-through for access to GPUs

bull Higher Capacity DrivesmdashAnew 24TB 10k rpm SAS HDD option for SFF Hybrid HyperFlex clustersand a 12TB 72K rpmSASHDDoption for LFFHybrid are now available Both HyperFlex andHyperFlexEdge support the 24TB capacity point for maximum density in this form factor Note that HyperFlex

Release Notes for Cisco HX Data Platform Release 408

New Features

Edge does not support LFF-drives HyperFlex HyperV version does not yet support the new 12TB driveoption See the HyperFlex spec sheets for a full list of configurable options

bull New Cache and increased scale for Hyper-V mdashNVMe amp Optane SSDs are now supported as cachedrives for Hyper-V deployments Furthermore scale limits have been increased to 16+16(Converged+Compute-only) for both SFF (AF and Hybrid) amp LFF (Hybrid) clusters

bull Centralized Audit Log ExportmdashThis release adds support for audit logging via a remote syslog serverThis capability enables customers to retain audit logs from all HyperFlex nodes in a centralized remotesyslog server to meet retention and compliance requirements

bull DISA STIG CompliancemdashThis release adds new HX REST APIs for setting removing and checkingstatus of DISA STIGs for Controller VMs ESXi hosts and vCenter These APIs enable customers tomeet DISA security requirements by centrally and securely applying STIGs detecting and correctingfor drifts in any STIG settings

Supported Versions and System RequirementsCisco HX Data Platform requires specific software and hardware versions and networking settings forsuccessful installation

For a complete list of requirements see

bull Cisco HyperFlex Systems Installation Guide for VMware ESXi or

bull Cisco HyperFlex Systems Installation Guide for Microsoft Hyper-V

Hardware and Software Interoperability

For a complete list of hardware and software inter-dependencies refer to respective Cisco UCS Managerrelease version of Hardware and Software Interoperability for Cisco HyperFlex HX-Series

HyperFlex Software Versions

The HX componentsmdashCisco HX Data Platform Installer Cisco HX Data Platform and Cisco UCSfirmwaremdashare installed on different servers Verify that each component on each server used with and withinan HX Storage Cluster are compatible

bull HyperFlex does not support UCS Manager and UCS Server Firmware versions 40(4a) 40(4b)and 40(4c)

Do not upgrade to these versions of firmware

Do not upgrade to these versions of UCS Manager

Important

bull Verify that the preconfigured HX servers have the same version of Cisco UCS server firmware installedIf the Cisco UCS Fabric Interconnects (FI) firmware versions are different see the Cisco HyperFlexSystems Upgrade Guide for VMware ESXi Release 40 for steps to align the firmware versions

bull M4 For NEW hybrid or All Flash (Cisco HyperFlex HX240c M4 or HX220c M4) deploymentsverify that Cisco UCS Manager 31(3k) 32(3i) or 40(2b) or higher is installed For moreinformation see Recommended Cisco HyperFlex HX Data Platform Software Releases

Release Notes for Cisco HX Data Platform Release 409

Supported Versions and System Requirements

bull M5 For NEW hybrid or All Flash (Cisco HyperFlex HX240c M5 or HX220c M5) deploymentsverify that the recommended UCS firmware version is installed

For SED-based HyperFlex systems ensure that the A (Infrastructure) B (Bladeserver) and C (Rack server) bundles are at Cisco UCS Manager version 40(2b)or later for all SED M4M5 systems For more details see CSCvh04307 ForSED-based HyperFlex systems also ensure that all clusters are at HyperFlexRelease 35(2b) or later For more information see Field Notice (70234) andCSCvk17250

Important

bull To reinstall an HX server download supported and compatible versions of the software See theCisco HyperFlex Systems Installation Guide for VMware ESXi for the requirements and steps

bull Important For Intersight edge servers running older than 40(1a) CIMC version HUU is the suggestedmechanism to update the firmware

Table 1 HyperFlex Software Versions for M4M5 Servers

M5 Recommended FIServer Firmware

(be sure to review important notes above)

M4 Recommended FIServer Firmware

(be sure to review important notes above)

HyperFlexRelease

40(4h)40(4h)40(2b)

40(4h)40(4h)40(2a)

40(4h)40(4h)40(1b)

40(4e)40(4e)40(1a)

HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments

Cisco HX Data Platform Release 4x based Deployments

Confirm the component firmware on the server meets the minimum versions listed in the following tables

HyperFlex Edge does not support Cisco IMC versions 40(4a) 40(4b) 40(4c) 40(4d) and 40(4e)Important

Table 2 HX220c M4 HXAF220c M4 Cluster

Recommended Firmware Version - HXDP 4x

(be sure to review important note(s) above)

Component

402d0BIOS

40(2h)Cisco Integrated Management Controller(CIMC)

Release Notes for Cisco HX Data Platform Release 4010

HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments

Recommended Firmware Version - HXDP 4x

(be sure to review important note(s) above)

Component

13000012SAS HBA

43(2b)VIC 1227

0103 (Toshiba)

CS07 (Intel 480)

SSD

CN03 (Seagate)

CK03 (Seagate 18)

CK03 (Seagate 24)

sF3Q (Samsung)

3F3Q (Samsung)

HDD

40(2h)

Download Software

Host Upgrade Utility (HUU) Download Link

Table 3 HX220c M5 HXAF220c M5 Cluster

Recommended Firmware Version - HXDP 4x

(be sure to review important notes above)

Component

C220M5404p00224200755BIOS

40(4i)Cisco Integrated Management Controller(CIMC)

09000008SAS HBA

43(3c)VIC 1387

50(3e)VIC 1457

G201CS01 (Intel)

0104 (Toshiba)

0107 (Toshiba)

SSD

2F3Q (Samsung)

3F3Q (Samsung)

NOA4 (Seagate)

CK03 (Seagate 18tb)

CK03 (Seagate 24)

HDD

Release Notes for Cisco HX Data Platform Release 4011

HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments

Recommended Firmware Version - HXDP 4x

(be sure to review important notes above)

Component

40(4k)

Download Software

Host Upgrade Utility (HUU) Download Link

HyperFlex Licensing

As of version 26(1a) HyperFlex supports VMware PAC licensing Existing VMware embedded licenses willcontinue to be supported

As of version 25(1a) HyperFlex uses a smart licensing mechanism to apply your licenses See the CiscoHyperFlex Systems Installation Guide for VMware ESXi for details and steps

VMware vSphere Licensing Requirements

How you purchase your vSphere license determines how your license applies to your HyperFlex system

bull If you purchased your vSphere license with HyperFlex

Each HyperFlex server either has the Enterprise or Enterprise Plus edition preinstalled at the factory

bull HX Nodes have OEM licenses preinstalled If you delete or overwrite thecontent of the boot drives after receiving the HX servers you also delete thefactory-installed licenses

bull OEM license keys is a newVMware vCenter 60 U1b feature Earlier versionsdo not support OEM licenses

bull All factory-installed HX nodes share the same OEM license key WithvSphere OEM keys the Usage count can exceed the Capacity value

bull When you add an HX host to vCenter through the Add Host wizard in theAssign license section select the OEM license

We obfuscate the actual vSphere OEM license key for example0N085-XXXXX-XXXXX-XXXXX-10LHH

bull Standard Essentials Plus and ROBO editions are not available preinstalledon HX servers

Note

bull If you did NOT purchase your vSphere license with HyperFlex

The HX nodes have a vSphere Foundation license preinstalled After initial setup you can apply thelicense to a supported version of vSphere

bull If you purchased a vSphere PAC license

Follow the instructions in your PAC license letter from VMware to add the license to your MY VMwareaccount then follow the instructions to add your HX host to vCenter and assign the PAC license

Release Notes for Cisco HX Data Platform Release 4012

HyperFlex Licensing

HX Data Platform Software Versions for HyperFlex Witness Node

Witness Node VersionHyperFlex Release

10840(2b)

10840(2a)

10440(1b)

10440(1a)

Software Requirements for VMware ESXi

The software requirements include verification that you are using compatible versions of Cisco HyperFlexSystems (HX) components and VMware vSphere VMware vCenter and VMware ESXi

bull Verify that all HX servers have a compatible version of vSphere preinstalled

bull Verify that the vCenter version is the same or later than the ESXi version

bull Verify that the vCenter and ESXi versions are compatible by consulting the VMware ProductInteroperability Matrix Newer vCenter versions may be used with older ESXi versions so long as bothESXi and vCenter are supported in the table below

bull Verify that you have a vCenter administrator account with root-level privileges and the associatedpassword

For VIC1457 there is no support for ESXi 60Note

The below table applies for all of the following VMware vSphere Editions Enterprise Enterprise PlusStandard Essentials Plus ROBO

VMware vCenter VersionsVMware ESXi VersionsHyperFlex Version

60 U3 65 U3 67 U360 U3 65 U3 67 U340(2b)1

60 U3 65 U3 67 U360 U3 65 U3 67 U340(2a)2

60 U3 65 U3 67 U260 U3 65 U3 67 U2340(1b)

60 U3 65 U2 67 U260 U3 65 U2 67 U2440(1a)

1 Cisco HyperFlex Release 40(2) is the last major HyperFlex release that will support vSphere 60 (ESXiand vCenter) due to those versions reaching end of VMware general support on March 12 2020

2 Cisco HyperFlex Release 40(2) is the last major HyperFlex release that will support vSphere 60 (ESXiand vCenter) due to those versions reaching end of VMware general support on March 12 2020

3 Use of 67 U2 is not recommended see software advisory for CSCvq06952 and CSCvp88515 for furtherdetails

4 Use of 67 U2 is not recommended see software advisory for CSCvq06952 and CSCvp88515 for furtherdetails

Release Notes for Cisco HX Data Platform Release 4013

HX Data Platform Software Versions for HyperFlex Witness Node

Software Requirements for Microsoft Hyper-V

The software requirements include verification that you are using compatible versions of Cisco HyperFlexSystems (HX) components and Microsoft Hyper-V (Hyper-V) components

HyperFlex Software versions

The HX componentsmdashCisco HX Data Platform Installer Cisco HX Data Platform and Cisco UCSfirmwaremdashare installed on different servers Verify that each component on each server used with and withinthe HX Storage Cluster are compatible

bull Cisco HyperFlex M5 Converged nodesmdash For Hybrid (Cisco HyperFlex HX240c M5 HX220c M5)and All Flash (Cisco HyperFlex HXAF240cM5 HXAF220cM5) verify that Cisco UCSManager 40(2b)is installed HX 40 (1a) does not support Hyper-V on the All NVMe (HXAF220C-M5SN) nodes Fordetailed information on installation requirements and steps see theCisco HyperFlex Systems InstallationGuide on Microsoft Hyper-V

Table 4 Supported HyperFlex Software versions for M5 Servers on Hyper-V

M5 Recommended Server FirmwareHyperFlex Release

40(4h)40(2b)

40(4g)40(2a)

40(4e)40(1b)

40(4e)40(1a)

Release Notes for Cisco HX Data Platform Release 4014

Software Requirements for Microsoft Hyper-V

Table 5 Supported Microsoft Software versions

VersionMicrosoft Component

Windows Server 2016 Datacenter Core amp DesktopExperience

ForWindows Server 2016Datacenter Coreand Desktop Experience the Windows2016 ISO image should be Update BuildRevision (UBR) 1884 at a minimum

Note

Windows Server 2019Datacenter-DesktopExperienceis supported starting from HXDP 401(a) onwards

For Windows Server 2019 DesktopExperience theWindows 2019 ISO imageshould be Update Build Revision (UBR)107 at a minimum

Note

Windows Server 2019 DatacenterndashCore is notsupported currently

Also note that the following are currently notsupported

OEM activated ISOs and Retail ISOs are notsupported

Earlier versions ofWindows Server such asWindows2012r2 are not supported

Non-English versions of the ISO are not supported

Windows Operating System (Windows OS)

A Windows 2012 or later domain and forestfunctionality level

Active Directory

Supported Microsoft License Editions

The Microsoft Windows Server version that is installed on one or more HyperFlex hosts must be licensed asper Microsoft licensing requirements listed on Microsoft Licensing

Browser Recommendations

Use one of the following browsers to run the listed HyperFlex components These browsers have been testedand approved Other browsers might work but full functionality has not been tested and confirmed

Table 6 Supported Browsers

HX ConnectHX Data PlatformInstaller

Cisco UCS ManagerCisco IntersightBrowser

11 or higher11 or higher11 or higherNAMicrosoft InternetExplorer

70 or higher70 or higher57 or higher62 or higherGoogle Chrome

Release Notes for Cisco HX Data Platform Release 4015

Browser Recommendations

HX ConnectHX Data PlatformInstaller

Cisco UCS ManagerCisco IntersightBrowser

60 or higher60 or higher45 or higher57 or higherMozilla Firefox

NANA9 or higher10 or higherApple Safari

NANA35 or higherNAOpera

Notes

bull Cisco HyperFlex Connect

The minimum recommended resolution is 1024 X 768

bull Cisco HX Data Platform Plug-in

The Cisco HX Data Platform Plug-in runs in vSphere For VMware Host Client System browserrequirements see the VMware documentation at httpswwwvmwarecomsupportpubs

The HX Data Platform Plug-in is not displayed in the vCenter HTML client You must use the vCenterflash client

bull Cisco UCS Manager

The browser must support the following

bull Java Runtime Environment 16 or later

bull Adobe Flash Player 10 or higher is required for some features

For the latest browser information about Cisco UCSManager refer to themost recent Cisco UCSManagerGetting Started Guide

Cisco HX Data Platform Storage Cluster SpecificationsCluster Limits

bull Cisco HX Data Platform supports up to 100 clusters managed per vCenter as per VMware configurationmaximums

bull Cisco HX Data Platform supports any number of clusters on a single FI domain Each HX convergednode must be directly connected to a dedicated FI port on fabric A and fabric B without the use of a FEXC-series compute only nodes must also directly connect to both FIs B-series compute only nodes willconnect through a chassis IO module to both fabrics In the end the number of physical ports on the FIwill dictate the maximum cluster size and maximum number of individual clusters supported in a UCSdomain

The following table provides Cisco HX Data Platform Storage Cluster Specifications

Release Notes for Cisco HX Data Platform Release 4016

Cisco HX Data Platform Storage Cluster Specifications

Stretched Cluster (Availableon ESX Only)

Microsoft Hyper-VVMware ESXiNode

AllNVMe -HXAF220c-M5SN

HX240c-M5LHX220cM5

HX220cAF M5

HX240cM5

HX240cAF M5

HX240c-M5LHX220cM5

HX220cAF M5

HX240cM5

HX240cAF M5

HXAF220c-M5SN

AllNVMe -HXAF220c-M5SN

ESXi 65or 67only NotsupportedwithESXi 60orHyper-V

HX220cM5 Edge

HXAF220cM5 Edge

HX220cM4 Edge

HXAF220cM4 Edge

HX240c-M5LHX220cM5

HX220cAF M5

HX240cM5

HX240cAF M5

HX220cM4

HX220cAF M4

HX240cM4

HX240cAF M4

HXServers

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

C220M5C240M5B200M4B200M5

C240M5C220M5B200M4B200M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

mdashB200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

Compute-OnlyUCSB-SeriesC-SeriesServers

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

Convergednodesonly

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

SupportedNodes

Release Notes for Cisco HX Data Platform Release 4017

Cisco HX Data Platform Storage Cluster Specifications

Stretched Cluster (Availableon ESX Only)

Microsoft Hyper-VVMware ESXiNode

NA(requiresEnterpriseHXDP-PLicense)

NA(requiresEnterpriseHXDP-PLicense)

NA(requiresEnterpriseHXDP-PLicense)

Convergednodes3-16(12TBHDDoption isnotsupportedforHyperV)

Compute-onlynodes0-16

Convergednodes3-16

Compute-onlynodes0-16

NA(requiresEnterpriseHXDP-PLicense)

M4Convergednodes 3

M5Convergednodes23or 4

RequiresHXDP-ELicense

Convergednodes3-16(12TBdrive islimited toamaximumof 8nodes)

Compute-onlynodes0-16(12TBdrive islimited toamaximumof 8nodes)

Convergednodes3-32(76TBdriveconfigsarelimited tomaximumof 12drivesnodeon AF240)

Compute-onlynodes0-32

HXDP-SLicensedNodeLimits

11 ratioofHXDP-StoComputeonlynodes

(MinmdashMax)

Convergednodes2-16 perSite

Compute-onlynodes0-21 perSite

(up tomaxclustersize)

Convergednodes2-8 perSite

Compute-onlynodes0-16 perSite

(up tomaxclustersize)

Convergednodes2-16 perSite

Compute-onlynodes0-21 perSite

(up tomaxclustersize)

Convergednodes3-16(12TBHDDoption isnotsupportedforHyperV)

Compute-onlynodes0-16

Convergednodes3-16

Compute-onlynodes0-16

Convergednodes3-32

ComputeOnlynodes0-32

(up tomaxclustersize)

Convergednodes 3

(requiresHXDP-ELicense)

Convergednodes3-16(12TBdrive islimited toamaximumof 8nodes)

Compute-onlynodes0-32(12TBdrive islimited toamaximumof 16nodes)

Convergednodes3-32(76TBdriveconfigsarelimited tomaximumof 12drivesnodeon AF240)

Compute-onlynodes0-32

(up tomaxclustersize)

HXDP-PLicensedNodeLimits

12 ratioofHXDP-PtoComputeonlynodes

(MinmdashMax)

24 perSite 64percluster

24 perSite 48percluster

32 perSite 64percluster

32326434864MaxClusterSize

Release Notes for Cisco HX Data Platform Release 4018

Cisco HX Data Platform Storage Cluster Specifications

Stretched Cluster (Availableon ESX Only)

Microsoft Hyper-VVMware ESXiNode

212121111121mdash2121MaxComputetoConvergedratio

NoExpansion

Requires Enterprise license

Requires uniform expansion across both sites

Guidelines and Limitationsbull Starting with release 40(2a) SCVM is no longer needed on a Compute node

bull HX REST API Access Token Management ndash Applications leveraging HX REST APIs should re-useaccess tokens when making API calls Once obtained using the AAA Obtain Access Token API accesstokens are valid for 18 days (1555200 seconds) In addition AAA enforces rate limiting on ObtainAccess Token API requests in a 15 minute window auth can be invoked (successfully) a maximum of5 times A user is allowed to create a maximum of 8 unrevoked tokens Subsequent call to auth willautomatically revoke the oldest issued token tomake room for the new token Amaximum of 16 unrevokedtokens can be present in system In order to prevent brute-force attacks after 10 consecutive failedauthentication attempts a user account is locked for a period of 120 secondsFor more information seeCisco HyperFlex Systems REST API Reference guide

HxConnect makes use of AAA Authentication REST API for login and the above rate limit applies toHxConnect also

Upgrade Guidelines

The following list is a highlight of critical criteria for performing an upgrade of your HyperFlex system

bull Unsupported HX Data Platform 17x 18x 20 21x 25x and 26x clustersmdashUsers from anyversion prior to 26(1a) must step through an intermediate version before upgrading to 40 or later releasesIf you need to upgrade your environment from a Cisco HyperFlex HX Data Platform software releasethat is past the last date of support to the latest suggested release on the Cisco Software Download sitesee Cisco HyperFlex Systems Upgrade Guide for Unsupported Cisco HXReleases For more informationsee the Software Advisory for CSCvq66867 WARNING Only Use HXDP 26(1e) Upgrade PackageWhen Upgrading From HXDP 18(1a)-18(1e)

bull Hypercheck Health Check UtilitymdashCisco recommends running this proactive health check utility onyour HyperFlex cluster prior to upgrade These checks provide early visibility into any areas that mayneed attention andwill help ensure a seamless upgrade experience For more information see the HyperFlexHealth amp Pre-Upgrade Check Tool TechNote for full instructions on how to install and run Hypercheck

bull vSphere 67 Software AdvisorymdashDo not upgrade to Cisco HX Data Platform Release 40(1a) whenrunning ESXi 67U1 EP06 (build 11675023) Do not upgrade to 67U1 EP06 (build 11675023) ifrunning Cisco HX Data Platform Release 40(1a) See the Software Advisory CSCvo56350 for furtherdetails

Release Notes for Cisco HX Data Platform Release 4019

Guidelines and Limitations

The software build version posted at release will override any other local versions

bull Required vCenter upgrademdashFor enhanced security Cisco HX Data Platform Release 35(1a) or laterrequires the use of TLS 12 Therefore vCenter must be upgraded to 60 U3f or later before upgradingto Cisco HX Data Platform Release 35 or later In addition ESXi should be upgraded as required tomeet HX Data Platform compatibility requirements

bull Minimum HXDP version for upgrademdashHXData Platform clusters running 26(1a) or later may upgradedirectly to 40 using the HX Connect UI

bull Cluster ReadinessmdashEnsure that the cluster is properly bootstrapped and the updated plug-in is loadedbefore proceeding Manual cluster bootstrap is required for upgrade from a pre-35 release

bull Cluster ReadinessmdashEnsure that the cluster is properly bootstrapped and the updated plug-in is loadedbefore proceeding Manual cluster bootstrap is required for HX releases earlier than 35(1a) For moreinformation see theManual Bootstrap Upgrade Process in the Cisco HyperFlex Systems Upgrade Guidefor VMware ESXi Release 40 Do not skip this cluster bootstrap step it is required for all upgradesuntil HX Release 35(1a) Auto bootstrap is supported beginning with HX release 35(1a) For moreinformation see the Auto Bootstrap Upgrade Process from HX Connect UI in the Cisco HyperFlexSystems Upgrade Guide for VMware ESXi Release 40

Manual bootstrap is not supported on Intersight clusters

bull Initiating Upgrade―Use the HX Connect UI or CLI stcli commands when upgrading from 25(1a)or later releases Use either the CLI stcli commands or the HX Data Platform Plug-in to the vSphereWeb Client when upgrading from a pre-25(1a) release The vCenter plug-in should not be used forupgrades starting with the 25(1a) release The vCenter plug-in should not be used for upgrades startingwith the 25(1a) release

If the current cluster version is at 35(1a) or above you do not need to use the stcli command Directupgrade to 40 is possible

bull Complete your Upgrade―The self-healing (or rebalance) capability is disabled temporarily during theupgrade window If the upgrade fails you should complete the upgrade as soon as possible

bull ESXi and HXDP Compatibility―Ensure your cluster is running a compatible version of ESXi basedon the running the HXData Platform version (see the section Software Requirements for VMware ESXi)ESXi compatibility is determined by the major version and update release of ESXi It is generally bestto upgrade HXDP and ESXi together if combining the upgrade operations into a single optimized rebootWhen running a split upgrade first upgrade the HX Data Platform then proceed to upgrade ESXi

bull If Upgrading to vSphere 65

bull Certain cluster functions such as native and scheduled snapshotsReadyClones and Enter or Exit HX Maintenance Mode will not operatefrom the time the upgrade is started until the HX Data Platform upgrade iscomplete

bull After upgrading ESXi using the offline zip bundle use the ESX ExitMaintenance Mode option The HX Exit Maintenance Mode option doesnot operate in the vSphere Web Client until the HX Data Platform upgradeis complete

Note

Release Notes for Cisco HX Data Platform Release 4020

Guidelines and Limitations

bull vSphere 60 UpgradesmdashUsers on vSphere 60 migrating to 65 upgrade components in the followingorder

1 Upgrade HX Data Platform and UCS firmware

2 Upgrade HX Data Platform and ESXi

3 Upgrade HX Data Platform only first then upgrade ESXi or UCS firmware or both

bull M4 Server Firmware UpgradesmdashUpgrade server firmware to ensure smooth operation and to correctknown issues Specifically newer SAS HBA firmware is available in this release and is recommendedfor long-term stability

bull Users are encouraged to upgrade to 31(3c) C-bundle or later whenever possible

bull Users running C-bundle versions before 31(2f) must upgrade server firmware by performing acombined upgrade of UCS server firmware (C-bundle) to 31(3c) or later and HX Data Platform to25 Do not split the upgrade into two separate operations

bull If the cluster is already on 31(2f) C-bundle or later you may perform an HX Data Platform onlyor combined upgrade as required

bull M5 Server Firmware UpgradesmdashM5 generation servers must run firmware version 32(2d) or later

bull Firmware Downgrades mdashDowngrading UCSM from the HX-installer is not supported

bull M4M5 Mixed DomainsmdashA mixed domain occurs when a new separate M5 cluster is installed underthe same UCS domain that contains existing M4 clusters Under these conditions orchestrated UCSserver firmware upgrade will not operate until Cisco HX Data Platform Release 26 or later is installedon the M4 clusters Therefore it is best practice to first upgrade UCS server firmware to the latest 31(3)or 32(2) patch release before adding a new M5 cluster to the existing UCS domain Additionally any17 HX Data Platform clusters must first be upgraded before adding any new M5 clusters to the samedomain

bull Maintenance WindowmdashIf upgrading both HX Data Platform and UCS firmware you can select eithera combined or split upgrade through the vSphere HX Data Platform Plug-in depending on the length ofthe maintenance window Cisco UCSManager infrastructure upgrade is only supported using AutoInstalland the direct server firmware upgrade should be performed only through the upgrade orchestrationframework provided by the HX Data Platform Plug-in

bull Unsupported Self-Encrypting Drives (SEDs)mdashIf adding or replacing self-encrypting drives (SEDs)that have been recently qualified in newer versions of HX Data Platform insert the new drives only afterupgrading HX Data Platform to a compatible version All drives must be SED drives mixing SED andnon-SED is not supported

bullbull Enabling External Host AccessmdashWith Cisco HX Data Platform Release 40(1a) port 445 on themanagement network is blocked for enhanced security Note that prior to 40 port 445 port was openenabling external host access If you are upgrading to 40(1a) from a prior release and would like tocontinue external host access you can use a utility to open select hosts For more information aboutenabling external host access see the ConfiguringHyperFlex Share to SCVMM section in the InstallationGuide for Microsoft Hyper-V

Release Notes for Cisco HX Data Platform Release 4021

Guidelines and Limitations

Mixed Cluster Expansion Guidelinesbull Expanding existing M4 cluster with M5 converged nodes is supported

bull Expanding existing M5 cluster with M4 converged nodes is not supported

bull Expanding existing mixed M4M5 cluster with M4 or M5 converged nodes is supported

bull Adding any supported compute-only nodes is permitted with all M4 M5 and mixed M4M5 clustersusing the HX Data Platform 26 or later Installer Some example combinations are listed here manyother combinations are possibleExample combinationsExpand mixed M4M5 cluster with compute-only B200 C220 C240 M4M5Expand M4 cluster with compute-only B200 M5 C220 M5 C240M5

bull Only expansion workflow is supported to create a mixed cluster Initial cluster creation with mixedM4M5 servers is not supported

bull All M5 servers must match the form factor (220240) type (HybridAF) security capability (Non-SEDonly) amp disk configuration (QTY capacity and non-SED) of the existing M4 servers

bull HX220-M5 will use a maximum of 6 capacity disks (2 disk slots to remain empty) when mixedwith HX220-M4

bull HX Edge SED LFF Hyper-V and Stretched Clusters do not support mixed M4 and M5 clusters

Security FixesThe following security issues are resolved

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-13132 CVE-2019-9924CSCvq6313840(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-11719 CVE-2019-11727CVE-2019-11729

CSCvq7124040(2a)

Release Notes for Cisco HX Data Platform Release 4022

Mixed Cluster Expansion Guidelines

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-1125CSCvr0633940(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2015-9383 CVE-2018-14498CVE-2018-20406 CVE-2018-20852CVE-2019-10160 CVE-2019-13117CVE-2019-13118 CVE-2019-14287CVE-2019-14973 CVE-2019-15903CVE-2019-17546 CVE-2019-18197CVE-2019-18218 CVE-2019-5010CVE-2019-5094 CVE-2019-5481CVE-2019-5482 CVE-2019-9636CVE-2019-9740 CVE-2019-9947CVE-2019-9948

CSCvs0609440(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2017-13168 CVE-2017-18174CVE-2017-18216 CVE-2018-10876CVE-2018-10877 CVE-2018-10878CVE-2018-10879 CVE-2018-10880CVE-2018-10881 CVE-2018-10882CVE-2018-10902 CVE-2018-10938CVE-2018-12233 CVE-2018-12896CVE-2018-13053 CVE-2018-13094CVE-2018-13096 CVE-2018-13405CVE-2018-13406 CVE-2018-14609CVE-2018-14617 CVE-2018-14633CVE-2018-14734 CVE-2018-15572CVE-2018-15594 CVE-2018-16276CVE-2018-16658 CVE-2018-17182CVE-2018-17972 CVE-2018-18021CVE-2018-18690 CVE-2018-18710CVE-2018-6554 CVE-2018-6555CVE-2018-9363

CSCvp6501940(2a)

This vulnerability is due to insufficientHTML iframe protection An attackercould exploit this vulnerability bydirecting a user to an attacker-controlledweb page that contains amaliciousHTMLiframe A successful exploit could allowthe attacker to conduct clickjacking orother clientside browser attacks

NACSCvo9851640(2a)

Release Notes for Cisco HX Data Platform Release 4023

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerability is due to insufficientauthentication for the statistics collectionservice An attacker could exploit thisvulnerability by sending properlyformatted data values to the statisticscollection service of an affected deviceA successful exploit could allow theattacker to cause the web interfacestatistics view to present invalid data tousers

NACSCvj9558440(2a)

The vulnerability is due to insufficientCSRF protections for the web UI on anaffected device An attacker could exploitthis vulnerability by persuading a user ofthe interface to follow a malicious linkA successful exploit could allow theattacker to perform arbitrary actions withthe privilege level of the affected user

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvp2434340(2a)

Vulnerabilities with a version of cJSONidentified by CVE-2019-11834 andCVE-2019-11835

For more information see the relatedCisco Security Advisory

CVE-2019-11834 CVE-2019-11835CSCvq1994940(2a)

HX ESXi image patches for MCEPSCand TAA vulnerabilities with VMwareESXi identified by CVE-2018-12207 andCVE-20190-11135

For more information see the relatedCisco Security Advisory

CVE-2018-12207 CVE-2019-11135CSCvr5439840(2a)

CP networking vulnerabilities affectingthe Linux kernel identified byCVE-2019-11477 CVE-2019-11478CVE-2019-11479

For more information see the relatedCisco Security Advisory

CVE-2019-11477 CVE-2019-11478CVE-2019-11479

CSCvq1954640(2a)

Qualification of Microsoft securitypatches for vulnerabilities withMicrosoftHyper-V hypervisor identified withCVE-2018-12207 CVE-2019-11135

For more information see the relatedCisco Security Advisory

CVE-2018-12207 CVE-2019-11135CSCvr5439940(2a)

Release Notes for Cisco HX Data Platform Release 4024

Security Fixes

DescriptionCVEDefect IDRelease

OpenSSH vulnerabilities identified byCVE-2019-6111

For more information see the relatedCisco Security Advisory

CVE-2016-10708 CVE-2018-15473CVE-2018-20685 CVE-2019-6109CVE-2019-6111

CSCvp7646340(2a)

Multiple vulnerabilities associated withOpenSSL and LibNSS

For more information see the relatedCisco Security Advisory

CVE-2016-2105 CVE-2016-2106CVE-2016-2107 CVE-2016-2109CVE-2016-2176 CVE-2016-2177CVE-2016-2178 CVE-2016-2179CVE-2016-2180 CVE-2016-2181CVE-2016-2182 CVE-2016-2183CVE-2016-6302 CVE-2016-6303CVE-2016-6304 CVE-2016-6305CVE-2016-6306 CVE-2016-6307CVE-2016-6308 CVE-2016-7055CVE-2016-8610 CVE-2017-3731CVE-2017-3732 CVE-2017-3735CVE-2017-3736 CVE-2017-3737CVE-2017-3738 CVE-2018-0495CVE-2018-0732 CVE-2018-0734CVE-2018-0735 CVE-2018-0737CVE-2018-0739 CVE-2018-12384CVE-2018-12404 CVE-2018-5407CVE-2019-1559

CSCvp6667940(2a)

libpng vulnerability identified byCVE-2016-10087

For more information see the relatedCisco Security Advisory

CVE-2016-10087CSCvp6655540(2a)

Vulnerabilities associated with libjpg andlibpng

For more information see the relatedCisco Security Advisory

CVE-2014-9092 CVE-2016-3616CVE-2017-15232 CVE-2018-11212CVE-2018-11213 CVE-2018-11214CVE-2018-1152 CVE-2018-13785

CSCvp3458640(2a)

Vulnerabilities associated with Glibidentified by CVE-2018-16428 andCVE-2018-16429

For more information see the relatedCisco Security Advisory

CVE-2018-16428 CVE-2018-16429CSCvp3120740(2a)

Multiple vulnerabilities associated withcurl expat python 27 python 35 36and 37 freetype and giflib

For more information see the relatedCisco Security Advisory

CVE-2019-15133 CVE-2019-15903CVE-2019-5010 CVE-2019-5481CVE-2019-5482 CVE-2019-9636CVE-2019-9740 CVE-2019-9947CVE-2019-9948

CSCvr3690340(2a)

Release Notes for Cisco HX Data Platform Release 4025

Security Fixes

DescriptionCVEDefect IDRelease

Multiple third party softwarevulnerabilities

For more information see the relatedCisco Security Advisory

CVE-2019-14379 CVE-2019-12384CVE-2019-14439

CSCvq9203240(2a)

Vulnerabilities associated with elfutils

For more information see the relatedCisco Security Advisory

CVE-2018-16062 CVE-2018-16402CVE-2018-16403 CVE-2018-18310CVE-2018-18520 CVE-2018-18521CVE-2019-7149 CVE-2019-7150CVE-2019-7665

CSCvq4325040(2a)

Vulnerabilities associated with vim

For more information see the relatedCisco Security Advisory

CVE-2017-5953 CVE-2019-12735CSCvq4323040(2a)

Vulnerabilities associated with sqlite3

For more information see the relatedCisco Security Advisory

CVE-2016-6153 CVE-2017-10989CVE-2017-13685 CVE-2017-2518CVE-2017-2519 CVE-2017-2520CVE-2018-20346 CVE-2018-20505CVE-2018-20506 CVE-2019-8457CVE-2019-9936 CVE-2019-9937

CSCvq4321340(2a)

Vulnerabilities associated with glib20

For more information see the relatedCisco Security Advisory

CVE-2018-20843CSCvq4320940(2a)

Vulnerabilities associated with expat

For more information see the relatedCisco Security Advisory

CVE-2018-20843CSCvq4320540(2a)

Vulnerabilities associated with bzip2

For more information see the relatedCisco Security Advisory

CVE-2016-3189 CVE-2019-12900CSCvq4319440(2a)

Mulitple Vulnerabilities associated wtihNodeJS

For more information see the relatedCisco Security Advisory

CVE-2018-12115 CVE-2018-0734CVE-2018-5407 CVE-2018-12120CVE-2018-12121 CVE-2018-12122CVE-2018-12123 CVE-2018-12116CVE-2019-5737 CVE-2019-5739

CSCvq1069440(2a)

Vulnerabilities associated with PalletsJinja strformat_map and identified byCVE-2019-10906 and CVE-2016-10745

For more information see the relatedCisco Security Advisory

CVE-2019-10906 CVE-2016-10745CSCvq1038840(2a)

Release Notes for Cisco HX Data Platform Release 4026

Security Fixes

DescriptionCVEDefect IDRelease

Multiple Security Vulnerabilities

For more information see the relatedCisco Security Advisory

CVE-2014-8501 CVE-2014-9939CVE-2015-9262 CVE-2016-10087CVE-2016-2226 CVE-2016-4487CVE-2016-4488 CVE-2016-4489CVE-2016-4490 CVE-2016-4491CVE-2016-4492 CVE-2016-4493CVE-2016-6131 CVE-2018-10963CVE-2018-13785 CVE-2018-17100CVE-2018-17101 CVE-2018-18557CVE-2018-18661 CVE-2018-7456CVE-2018-8905

CSCvq0917840(2a)

Vulnerabilities associated withlibseccomp

For more information see the relatedCisco Security Advisory

CVE-2019-9893CSCvq0756840(2a)

Vulnerabilities associated withgdk-pixbuf

For more information see the relatedCisco Security Advisory

CVE-2017-12447CSCvq0675540(2a)

Vulnerabilities associated withpython-crypto

For more information see the relatedCisco Security Advisory

CVE-2018-6594CSCvp9381740(2a)

Vulnerabilities associated with wget

For more information see the relatedCisco Security Advisory

CVE-2018-20483 CVE-2019-5953CSCvp8672140(2a)

Multiple vulnerabilities associated withPython Crypto

For more information see the relatedCisco Security Advisory

CVE-2018-6594CSCvp6674840(2a)

Vulnerabilities associated withsensible-utils

For more information see the relatedCisco Security Advisory

CVE-2017-17512CSCvp6673440(2a)

Multiple vulnerabilities associated withPython

For more information see the relatedCisco Security Advisory

CVE-2018-1000030CSCvp6668940(2a)

Release Notes for Cisco HX Data Platform Release 4027

Security Fixes

DescriptionCVEDefect IDRelease

Multiple patch vulnerabilities identifiedby CVE-2016-10713

For more information see the relatedCisco Security Advisory

CVE-2016-10713CVE-2018-1000156 CVE-2018-6951

CSCvp6667240(2a)

Vulnerabilities associatedwith Little CMSidentified by CVE-2016-10165

For more information see the relatedCisco Security Advisory

CVE-2016-10165 CVE-2018-16435CSCvp6666440(2a)

Vulnerabilities associated with TenableScan identified byCVE-2011-5325CVE-2019-11068CVE-2016-7076CVE-2017-1000368

For more information see the relatedCisco Security Advisory

CVE-2011-5325 CVE-2016-7076CVE-2017-1000368CVE-2019-11068

CSCvp6474640(2a)

Vulnerabilities associated with libxml2294 and earlier as used in XMLSec1223 and earlier and other products andidentified by CVE-2016-9318

For more information see the relatedCisco Security Advisory

CVE-2016-9318 CVE-2017-16932CVE-2017-18258 CVE-2018-14404CVE-2018-14567

CSCvp3479240(2a)

Vulnerabilities associated with LFTPRemote File Names Unauthorized AccessVulnerability and identified byCVE-2018-10916

For more information see the relatedCisco Security Advisory

CVE-2018-10916CSCvp2926640(2a)

Vulnerabilities associated withCryptographyDeprecationWarning signerand verifier have been deprecated

For more information see the relatedCisco Security Advisory

CVE-2018-7750CSCvo3409740(2a)

Multiple Vulnerabilities in python python27 35

For more information see the relatedCisco Security Advisory

CVE-2019-16056CSCvr3979340(2a)

Release Notes for Cisco HX Data Platform Release 4028

Security Fixes

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toexecute commands as the root user

The vulnerability is due to insufficientinput validation An attacker could exploitthis vulnerability by connecting to thecluster service manager and injectingcommands into the bound process Asuccessful exploit could allow the attackerto run commands on the affected host asthe root user

Cisco has released software updates thataddress this vulnerability There areworkarounds that address thisvulnerability

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvq2417640(1b)35(2g)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

The vulnerability is due to an unprotectedlistening interface An attacker couldexploit this vulnerability by connectingto the listening interface and injectingcommands to the bound process Anexploit could allow the attacker to runcommands on the affected host as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvj9560640(1b)35(2g)

Release Notes for Cisco HX Data Platform Release 4029

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerabilities associated with theJVM18U121memory leak found duringVC alarms (concurrent 40 calls usingREST API)

CVE-2017-10053 CVE-2017-10067CVE-2017-10074 CVE-2017-10078CVE-2017-10081 CVE-2017-10087CVE-2017-10089 CVE-2017-10090CVE-2017-10096 CVE-2017-10101CVE-2017-10102 CVE-2017-10107CVE-2017-10108 CVE-2017-10109CVE-2017-10110 CVE-2017-10111CVE-2017-10115 CVE-2017-10116CVE-2017-10118 CVE-2017-10135CVE-2017-10176 CVE-2017-10193CVE-2017-10198 CVE-2017-10243CVE-2017-10274 CVE-2017-10281CVE-2017-10285 CVE-2017-10295CVE-2017-10345 CVE-2017-10346CVE-2017-10347 CVE-2017-10348CVE-2017-10349 CVE-2017-10350CVE-2017-10355 CVE-2017-10356CVE-2017-10357 CVE-2017-10388CVE-2017-3509 CVE-2017-3511CVE-2017-3526 CVE-2017-3533CVE-2017-3539 CVE-2017-3544CVE-2018-2579 CVE-2018-2582CVE-2018-2588 CVE-2018-2599CVE-2018-2602 CVE-2018-2603CVE-2018-2618 CVE-2018-2629CVE-2018-2633 CVE-2018-2634CVE-2018-2637 CVE-2018-2641CVE-2018-2663 CVE-2018-2677CVE-2018-2678 CVE-2018-2783CVE-2018-2790 CVE-2018-2794CVE-2018-2795 CVE-2018-2796CVE-2018-2797 CVE-2018-2798CVE-2018-2799 CVE-2018-2800CVE-2018-2814 CVE-2018-2815CVE-2018-2952 CVE-2018-3136CVE-2018-3139 CVE-2018-3149CVE-2018-3150 CVE-2018-3169CVE-2018-3180 CVE-2018-3183CVE-2018-3214 CVE-2019-2422

CSCvo8899740(1b)

Tomcat and Nginx logs are not beingcollected in the support bundle generatedthrough HX Connect in Release 35

NACSCvm5803140(1b)

The vulnerabilities associated with thelibmspack software package versionincluded in Cisco HX Data Platform

CVE-2018-18584

CVE-2018-18585

CSCvn3511940(1a)

Release Notes for Cisco HX Data Platform Release 4030

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerabilities associated withFasterXML Jackson-Databind TimeValueField Denial of Service

CVE-2018-14719

CVE-2018-14720

CVE-2018-1000873

CVE-2018-14721

CVE-2018-19360

CVE-2018-19362

CVE-2018-19361

CVE-2018-14718

CSCvn8228240(1a)

The vulnerabilities associated with theOpenSSL Protocol software packageversion included in Cisco HX DataPlatform

CVE-2013-3587CSCvo0505440(1a)

The vulnerabilities associated withThird-Party Software Denial of Service

CVE-2018-16487

CVE-2018-19361

CSCvo2781840(1a)

The vulnerabilities associated withLinuxkernel for Ubuntu 1710

CVE-2018-1092

CVE-2018-7492

CVE-2018-8087

CVE-2018-1068

CVE-2018-8781

CSCvm5314935(2a)

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)25(1d)stcli services dns remove should remove the DNS serverinfo from the interface files

CSCvs70967

40(2b)40(1b)HX 40(1b) new deployment from Intersight productioncloud

User receives following error while enabling CSI(Kubernetes integration)

Failure occurred during volume_access_enable Errorwas Nonzero exit code 1

CSCvt10522

40(2b)40(2a)Receive the following alertevent in HX Connect HXController VM HOSTNAME one or more configuredDNS servers not responding

CSCvt13947

Release Notes for Cisco HX Data Platform Release 4031

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)40(2a)API UI CLI show few Drive slots Empty after upgradeto 402a

The drives are seen in the sysmtool lsscsi and stclicommand outputs

The cluster is healthy with no errors

CSCvt14914

40(2b)35(2d)After successfully changing the DNS server on the HXcontroller we still can see the original DNS entry thatwas added during the deployment

CSCvs69154

40(2b)35(2e)HyperFlex Connect and VCenter show APD alarms forHX and non HX datastores This implies there is an issuewith HyperFlex when there may not be

CSCvs30080

40(2b)40(2a)An issue where all the disks in the node get locked afterrebooting the node

We successfully unlocked using sed-clientsh -Ucommand but wanted to test with another reboot anddrives locked again

CSCvs74286

40(2b)40(2a)When running stcli license commands on HyperFlexthe following error is seen

rootSpringpathControllertmp stcli license show all

Show smart licensing failed

Smart Agent is not ready please wait a minute and tryagain

CSCvt13929

40(2b)40(2a)After upgrading to HX 40(2a) the first RoboEdge nodethat gets rebooted then ignores persistent drivesPIDHX-HD24TB10K4KN

Cluster will be in degraded state and have reducedcapacity

CSCvt20144

40(2b)35(2b)Zookeeper fails to start while Exhibitor is runninghowever echo srvr returns nothing

CSCvs21562

40(2b)40(2a)When performing an HyperFlex upgrade a validationwarning may occur due to the host not having EnterprisePlus or Enterprise hypervisor licensing

Upgrade Validation Warning

ESXi host esx1labtest should be configured withVMware Enterprise license for upgrade to continue

CSCvs91787

Release Notes for Cisco HX Data Platform Release 4032

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)35(2g)Cluster expansion fails at Config Installer stage whenthe root and admin password for the storage controller(SCVM) are different

CSCvs69317

40(2b)35(2g)Panic while upgrading ESXiCSCvt06983

40(2b)40(2a)A cluster node running may hang in the Linux kernelThis is classified as an oops and a deviation from theexpected behavior

CSCvs54285

40(2b)35(2g)Rebalance failing on 10+10 node HX 35(2g) stretchcluster

CSCvs69007

40(2b)35(2d)The cluster becomes inaccessible to the IOVisorCSCvr54687

40(2b)40(2a)Panic on the storage controllerCSCvt61297

40(2b)40(2a)The size of support bundle is very large when collectedwith storfs-support command

CSCvt63306

40(2b)40(2a)storfs crash associated withFileSystemUsageWarningEvent orFileSystemUsageAlertEvent due to high usage ofvarstv above 80

CSCvt72807

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

40(2a)35(2d)In normal scenarios HX upgrade gets pendingACK afterupdating service profile In this case UCS only upgradewas stalled because after updating service profile HXupgrade was not getting pending ACK and it was alwayswaiting state

CSCvo39912

40(2a)35(1a)When using motherBoardReplace-12 to clean ZK fromold stNodepNodes resulted in unmounted datastore andmaking the size of the Hyperflex Datastores 0 resultingin all VMs in the cluster going offline

CSCvq39471

40(2a)26(1e)Upgrading cluster and getting error

Failed upgrade validations Checking vCenterconfiguration Reason Upgrade validations failed DRSFault Insufficient resources to satisfy configured failover

CSCvm77294

Release Notes for Cisco HX Data Platform Release 4033

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)The cluster expand fails on a node with DR replicationconfigured

When a HX cluster which has DR replication configuredis expanded we see the installer UI pulling in thereplicationVLAN information instead of themanagementVLAN information Even if we change that informationto the correct mgmt VLAN id and name it it does notseem to work as the node is configured with the VLANof the replicationVLAN in ESXi This leads to the failureof the node add with the host unreachable error

CSCvo70650

40(2a)35(1a)Customer reported that server didnt complete thefirmware upgrade automatically one by one

The user manually needs to put the host into maintenancemode and they manually acknowledged the pendingrequests to finish the UCS firmware upgrade

Maintenance policy is set to default (User-ack) as perthe design

CSCvo91624

40(2a)35(1i)When the cluster is in Failed state and stcli noderemove is attempted the output shows successful thoughthe node failed to get removed from the cluster

CSCvo93017

40(2a)35(1a)HyperFlex cluster upgrade may fail during validationwith a DRS Validation failed error

CSCvp31021

40(2a)35(2b)HX cluster expansion will fail with the following errormessage

MAC address pool configuration failure150[ErrorDescription] bad address block range definitioncollision

CSCvp58318

40(2a)35(2b)Memory usage by carbon cacheCSCvq34873

40(2a)35(2g)Upgrading ESXi cluster fails with error Nodemaintenance mode failed

CSCvr03240

40(2a)35(2e)storfs process does not automatically start on Exit HXMaintenance Mode or other tasks which restart StorageController

CSCvr88978

40(2a)35(2a)Post install script fails with message

Failed to execute ipmitool on HX node

CSCvp10707

40(2a)40(1a)HyperFlex expansionworkflow doesnt pull VLANnamecorrectly

CSCvp46539

Release Notes for Cisco HX Data Platform Release 4034

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)During HX cluster deployment - Validate ClusterCreation phase - HX Installer might fail with the error

fromvarlogspringpathstDeploylog

CSCvq45087

40(2a)35(2b)HX Installer fails to login to the SOL for configuringESX hosts

CSCvq91380

40(2a)35(2d)If DRS is enabled with configuration parameters otherthen fully automated ie manual or partially automatedthan this will change to fully automated during clusterexpansion

CSCvr44222

40(2a)35(1a)One of the validation task name has a typo The taskname is Validating node for mixed of SED and non-SEDdisks

It should be Validating node for mix of SED andnon-SED disks

CSCvp82175

40(2a)30(1i)This combination of using smaller NVMe drives is notsupported in expand operation For example if the drivesare only 375GB and they cannot be added to an existingcluster with larger caching SSD

CSCvo12359

40(2a)35(1a)HyperFlex includes a version of OpenSSL that is affectedby the vulnerabilities identified by the followingCommon Vulnerability and Exposures

CVE-2018-0495

CSCvp66679

40(2a)30(1e)SDK invoke exception when taking a native snapshot ona VM with GPU pci passthrough attached

CSCvo00511

40(2a)30(1i)Node replacement script fails due to EAM errorCSCvo62867

40(2a)35(1a)While pairing a cluster (HX release gt= 35) and cluster(HX release lt 35) remote replication network test failson the cluster (HX release lt 35)

CSCvo79760

40(2a)30(1b)Fixed the issue in the latest support-workflow bundleThis should not be hit anymore

CSCvo87061

40(2a)35(2a)On a 3 node cluster the MBreplace script fails as thescript looks for HEALTHY cluster status

CSCvo87080

40(2a)35(2a)MbReplace script (tar) hangs when running on 352b or40

CSCvp12359

Release Notes for Cisco HX Data Platform Release 4035

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)HX replication cleanup failing with error NA 35(2a) -INFODR state is not clean

CSCvp63958

40(2a)40(1a)The check_stig_parameters API incorrectly shows thecompliance state of the cluster as not STIG compliant

CSCvp66277

40(2a)35(2c)The snapshot creation on CBT enabled VM fails witherror Failed in vmreparent vmkfstools clone1

CSCvq06952

40(2a)35(2a)HyperFlex GUI when setting up remote DR-replicationpartnerpeer should show remote replication partner TCPport reachability test results

This will provide quick upfront connectivity results thatfrequently takes customer a while to manually check andconfirm

If reachability tests fail an info window and HX alertshould trigger that list the destination peer IP and thespecific port(s) that could not be reache

CSCvo60587

40(2a)35(2b)If you have more than one IP pool configured for areplication network and local replication network testsare failing you might be running into this issue

CSCvr67130

40(2a)35(2a)Skip task option in MBreplace and Software redeployscripts The scripts should have the option to skip tasksas user inputs and not by modifying the json file

CSCvp05204

40(2a)40(1a)Error noticed on SCVM Console print_req_error IOerror dev fd0 sector 0 on HyperV

It doesnt come up on SSH session

CSCvq34357

40(2a)30(1d)The VMNetwork switch is already created as part of theinstall process

Ideally the ip address assignment should be done duringthe initial install process as well No post install stepsshould be required altrnatively it could be done as partof a post install script

CSCvk36222

40(2a)25(1c)storfs service impacted during HX upgrade from HX21(1) to 25(1c)

HX node which is not being upgraded storfs servicestopped causing outage

CSCvg53223

40(2a)35(1b)HX Stretch Cluster Witness VM reverts to DHCP onreboot

CSCvp37536

40(2a)35(2b)HyperFlex Stretch Cluster Witness does not getprogrammed with an NTP server

CSCvp46578

Release Notes for Cisco HX Data Platform Release 4036

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2b)HyperFlex cluster does not heal after maintenance onnodeserver

CSCvr18528

40(2a)40(2a)High percentage of packet loss on the eth1 data interface9K MTU

CSCvr97089

40(2a)35(2f)The HyperFlex cluster healing state is stuck at 87 andis unable to progress

CSCvr16760

40(2a)40(2a)In some circumstances two replications can occur forthe same VM

From the GUI the replication status for this VM willshow stuck in an In Progress state but there are nojobs stuck in replication layer Subsequent Replicationsshould succeed

CSCvq22898

40(2a)35(2a)After an unexpected power outage the hyperflex clusteris unable to come up correctly

All processes are running time is synced and all nodescan vmkping each other as required

CSCvp13990

40(2a)40(1a)When specifying IP address instead of node ID stclinode maintenanceMode commands fail

CSCvp33657

40(2a)40(1a)2 node edge cluster if experiencing Intersightarbitratorconnectivity issues should NOT allowHXMaintenancemode to initiate as this will bring the cluster down

CSCvr31573

40(2a)35(2c)Old files in varsupportZKTxnlog are not purged withthe daily zklog-cleanup cron job

CSCvr89066

40(2a)35(2a)When a node is in MaintenanceMode any disk removalor replacement will be reflected in UI only after the nodeis brought back frommaintenancemode This is becausestorfs is not running on the node in maintenance andwill not be able to detect disk activities until it is broughtout of Maintenance Mode

CSCvo86431

40(2a)35(2c)Cluster might go offline and stop serving IO requestsrootcvm~ stcli cluster storage-summary--detailGet cluster storage summary failedjavanetConnectException Connection refused19216814210010207

CSCvr01645

40(2a)35(2d)A stretch cluster deployment fails with a messageFormatting nodes Node in Use on the Hyperflex installerand crmZoneType value is seen as 1 in storfscfg oroptspringpathconfigstretchtunes

CSCvq80340

Release Notes for Cisco HX Data Platform Release 4037

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)30(1c)During upgrade we upload the upgrade packagedownloaded from Ciscocom Upgrade packages are offile type tgz hxconnect acceptsstorfs-packages-401a-33028tar file a different type ofcompression and allows upgrade to start and fail whenit cannot decompress the file for bootstrap and scvmclientupgrade on ESXi

CSCvq63888

40(2a)40(2a)While attempting to install hyperflex via intersight thefollowing error is seen

Failed in Task Add hosts to vCenter Cluster with ErrorTry adding hosts manually to vCenter and retry Failedto get addressFailed to host by namelookupHostnamecompanycom on 000053 read udp12700145430-gt12700153 read connection refused

CSCvr52098

40(2a)35(2a)CoreAPI call may time out for cluster management APIwhile doing cluster create validation

CSCvo92952

40(2a)35(2d)Validation fails with Mixed mode expansion checkdue to Enhanced vMotion Compatibility Incompatibilitywhen the ESXi version is different

CSCvq95460

40(2a)30(1i)This failure is seen in installer when custom workflowis used hypervisor configuration + deploy (clean diskpartition=no) at Installing software packages on storagecontroller VM

CSCvr66309

40(2a)35(2d)The HX Installer apparently replaced a comma bysemicolon on the VLAN ID range while exporting theconfiguration to a JSON file

CSCvq93831

40(2a)40(1a)When using unsupported browsers HX Connect userswill not be able to login These users see an errorindicating unauthorized user

CSCvp22693

Release Notes for Cisco HX Data Platform Release 4038

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)40(1a)A vulnerability in the web-based management interfaceof Cisco HyperFlex Software could allow anunauthenticated remote attacker to conduct a cross-siterequest forgery (CSRF) attack on an affected system

The vulnerability is due to insufficient CSRF protectionsfor the web UI on an affected device An attacker couldexploit this vulnerability by persuading a user of theinterface to follow a malicious link A successful exploitcould allow the attacker to perform arbitrary actions withthe privilege level of the affected user

Cisco has released software updates that address thisvulnerability There are no workarounds that address thisvulnerability

This advisory is available at the following link

httpstoolsciscocomsecuritycentercontentCiscoSecurityAdvisorycisco-sa-20190807-hypflex-csrf

CSCvp24343

40(2a)35(2d)Add message to HX Connect Progress flow to notAcknowledge Pending Activities and reboot the serverson UCSM

HX Connect is doing a controlled rolling server upgradein the background

CSCvq22844

40(2a)40(1b)Compute Node expand operation fails due to incorrectcluster name

CSCvr43786

40(2a)26(1e)RF-2 cluster shutdown during data resynchronizationafter a node failure (non storfs) followed by multipledisk read failures and a HardBlacklist of a disk

CSCvp41241

40(2a)40(1a)HyperFlex Edge nodes do not properly set the ESXihostname during deployment

CSCvo13143

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4039

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4040

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1b) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401b-33133p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4041

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4042

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1b)40(1a)While running the HyperFlex installer with WindowsServer Hyper-V cluster creation process fails with theerror Failure occurred during ClusterCreation process Unable to post thecontent to the downstream This symptomis encountered while deploying HyperFlex cluster withHyperFlex nodes configuredordered with Cisco VIC1457 MLOM (PID HX-MLOM-C25Q-04) andWindows Server Datacenter or Core with Hyper-Vversion 2016 or 2019

CSCvp64140

Release Notes for Cisco HX Data Platform Release 4043

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)35(2b)Adding Micron 5200 drive to a cluster fails to increasecluster capacity

CSCvo69067

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4044

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4045

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1a) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401a-33028p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4046

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4047

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1a)30(1d)Cluster instability when disks of different sector sizeplaced in HX node

CSCvk17250

40(1a)

35(2c)

35(1a)When logged in using a local HX user account insteadof a Virtual Center account an error message appearsintermittently indicating Virtual Centerunreachable or Resource informationcannot be updated when VC is reachable

CSCvo36198

40(1a)

35(2b)

30(1d)HXDP does not work with EMC RecoverPoint - needsto support VMware API (FSS-Readdir)

CSCvk38003

Release Notes for Cisco HX Data Platform Release 4048

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)

35(2a)

30(1i)

25(1c)

25(1d)

Zookeeper (Exhibitor) process on HyperFlex StorageController VMs may fail to respond to requests whenvarzookeeper has low or no free space

CSCvm90352

40(1a)26(1c)Use Asynchronous consolidation for HX SnapshotsCSCvn02151

40(1a)35(2b)HX Quiesced Snapshot for Backup VendorsCSCvo90713

40(1a)35(2a)The cluster creationcluster expansion workflow stopswith the following error message at the validation step

FIRMWARE-Check UCSC-SAS-M5HD

FIRMWARE-Check UCSC-SAS-M5HD Required00000029000000320000003500000050 Found00000058

Action Needed Update the Controller Firmware toRequired Version

CSCvn17787

35(2b)30(1d)Kernel migration fails when a local datastore is searchedfor in the ESXi host

CSCvn73127

35(2a)26(1b)A node shuts down when two disks are replaced (acaching disk and another capacity disk) where thecapacity disk is inserted first and then the caching diskis inserted

CSCvk46364

Hyper-V

40(1a)35(2a)Cluster expansion may fail with an error code 500 -operation timed out

CSCvn28721

35(2b)35(2a)During upgrade remove the VLAN on team created foruser vSwitch During fresh installation only one VLANtag is set to the vSwitch and team although multipleVLANs were entered

CSCvn54300

35(2b)35(2a)While upgrading a Hyper-V cluster on account of a rarerace condition between the stUpgradeService andZookeeper servers the upgrade orchestration throws anupgrade validation error and the upgrade process isaborted

CSCvn60486

Release Notes for Cisco HX Data Platform Release 4049

Resolved Caveats in Release 40(1a)

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Startrestart EAM service invCenter

When EAM service is not runningvCenter registration is failing

CSCvt55712

35(2a)Contact Cisco TAC if you believeyou are seeing similar conditions

HyperFlex Stretch Cluster saw poorapplication performance due to writelatency cluster remained unhealthy andrebalance was stuck

This defect is currently being used forinvestigative purposes

CSCvt45344

35(2g)Remove log files fromvarlogspringpath folder when thelog files are not getting rotatedStop the relevant service removelarge log files fromvarlogspringpath folder andrestart the service

varstv folder in Controller VM maybecome full

CSCvt36374

40(2b)Restarting the stMgr will initializeit again

In a Cluster configured with DR networkStMgr may not get initialized and stuckin a deadlock while enabling the IPTablerules (can be verified from the stMgrlog)

CSCvt89709

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Click recover for the selectedVM again under the standalonemode This will move the VM torecovered state

When the user recovers a VM protectedin a group it along with other VMs in thegroup move to recovering state instandalonemode The selectedVM shouldmove to recovered state

CSCvs75553

35(2e)Clean up the fail over cluster andthen recreate the fail over clusterNo need to touch the HX storagecluster

Hyper-V When replicated DC was usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2d)40(1a)

Keep cluster utilization as low aspossible To avoid read-only statekeep below 92

HyperFlex clusters storage utilization isabove 92 usable capacity

CSCvr20922

Release Notes for Cisco HX Data Platform Release 4050

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1b)NAAll the disks in the node got locked afterrebooting the node

We successfully unlocked usingsed-clientsh -U command but wantedto test with another reboot and driveslocked again

CSCvs74286

35(2g)To recover the VM copy over thedata disks and attach them to thenew VM

After enable encryption HX connectencryption status showing Caution butall of the disks enable encryptionsuccessful

-Self Encrypting Drives Service isrunning on all of the nodes

-There is no error message of encryption

- USB0 interface is up

-All SED disks showing supported 1enabled 1 locked 0

CSCvs93245

40(1b)Enable kernelpanic_on_oops as apersistent configuration This willcause the node to panic and rebootimmediately

A cluster node running HX release40(1b) may hang in the Linux kernelThis is classified as an oops and adeviation from the expected behavior

CSCvs54285

Release Notes for Cisco HX Data Platform Release 4051

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Enabling K8iscsi stack in Hx clusterCSCvs41324

Release Notes for Cisco HX Data Platform Release 4052

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Steps to enable K8 on HX cluster

Perform following steps on ALLthe controller VMs

1 Updateetcinitscvmclientconf toenable tune iscsiEnable totrue

sed -iesiscsiEnable=falseiscsiEnable=trueetcinitscvmclientconf

2 Run following initctlcommand to reloadconfiguration

initctl reload-configuration

3 Restart scvmclient process

stop scvmclient startscvmclient

4 Run following initctlcommand to mount SYSTEMDS

initctl emit --no-waitsystem-datastore-created

5 Verify that iscsiEnable tuneis set to true using thefollowing command

ps -eaf | grep scvmclient |grep -v grep

root 6241 1 1 Dec06 014007optspringpathstorfs-corescvmclient-T logEnabled=false -TlogSyslogEnabled=true -TlogEchoToScreen=false -TstatLoggingToFile=false -TstatLoggingToSyslog=true-TlogDir=varlogspringpath-TnfsBackendServerList=1010748100-T iscsiEnable=true -TiscsiUseAsync=true -T

Release Notes for Cisco HX Data Platform Release 4053

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

iscsiConfigFilePath=nfsSYSTEMistgtconfps -eaf | grep scvmclient |grep -v grep

35(2e)Following procedure can be usedto check NFS Queue Depth andincrease if needed

rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth256 lt- Low value[rootHXESXI1] vsish -e setvmkModulesnfsclientmnt[DATASTORE]propertiesmaxQDepth 1024[rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth1024 lt- Optimalvalue

This requires an ESXi host rebootto take effect

Please place the node in HyperflexMaintenanceMode and gracefullyreboot the node for the changes tobe applied

HyperFlex Datastore NFS Queue Depthshows as 256 which can lead toperformance (including latency) issues

CSCvr83056

40(1b)This is cosmetic The error will goaway once the cluster returns tohealthy status

When a single node is offline in clusterstcli cluster storage-summary shows twonodes unavailable rootSCVM~ stclicluster storage-summary

CSCvs47735

35(2g)Modify both the root and adminpassword for the SCVMs to be thesame

stcli security password set -uadmin

Cluster expansion fails at Config Installerstage when the root and admin passwordfor the storage controller (SCVM) aredifferent

CSCvs69317

35(2d)NAAfter successfully changing the DNSserver on the HX controller we still cansee the original DNS entry that was addedduring the deployment

CSCvs69154

Release Notes for Cisco HX Data Platform Release 4054

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Customers are strongly encouragedto work with TAC in order toidentify and remediate this issue

1 Change getcluster from 2m to8m inoptspringpathstorfs-mgmtstMgr-10confapplicationconfon all nodes

2 Restart stMgr on all nodes

You may see this error message after afailed upgrade or other task such asattempting to enter a node into HXmaintenance mode

CSCvs53555

40(2a)Run the following command oneach storage controller VM as root

grep -i monitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp sed -iesmonitor_dns_serverstruemonitor_dns_serversfalseoptspringpathhx-diag-toolswatchdog_configjsonampamp grep -imonitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp restart watchdog

Receive the following alertevent in HXConnect HX Controller VMHOSTNAME one or more configuredDNS servers not responding

CSCvt13947

35(2b)Delete any empty (Size 0) log filesunder varlogzookeeperversion-2ORvarlogzookeeperstandaloneversion-2

Zookeeper fails to start while Exhibitor isrunning however echo srvr returnsnothing

CSCvs21562

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

40(4e)1 De-commission the server

2 Power drain the server -REMOVE BOTH POWERCORDS ON THE BACK OFTHE SERVER FOR 10SECONDS THENREINSERTPOWERCORDS

3 Re-commission the server

M2 boot disk is missing from serverinventory after upgrade to server firmware40(4e) As a result server fails to boot toOS installed in the M2 disk The issuepersists after re-acknowledgement as wellas de-commission andre-acknowledgement of the server

CSCvs02466

Release Notes for Cisco HX Data Platform Release 4055

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Re-start the flusher manuallyMany disks taken offline by a fault ormanually resulting in ENOSPC oncluster

CSCvq32721

35(2e)Clean up fail over cluster and thenrecreate the fail over cluster Thereis no need to touch the HX storagecluster

[Hyper-V]When replicated DCwas usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2b)Manually install upgradescvmclient VIB andmake sure thatit matches with HXDP version

Upgrade from release HX 35(2a) to35(2x) 4X does not upgrade scvmclientRecovery point may not work

CSCvq54992

Management

30(1b)To recover the VM copy over thedata disks and attach them to thenew VM

[HyperV]VMshows up onmultiple nodesCSCvj22992

35(1a)1 Do not change MTU at ESXlevel on running cluster

2 Reset back to the original value

ChangingMTU from 9000 to 1500 on oneESX node causes storfs process to restarton all nodes

CSCvm99150

Release Notes for Cisco HX Data Platform Release 4056

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)After the installation (for newdeployments) or after an upgradeto 401a (for existingdeployments) perform thefollowing steps on all ControllerVMs

1 Edit the following tune files (onall Controller VMs)

optspringpathconfiglfftunes

optspringpathconfigvsi_16tbtunes

2 SetcleanerEnableSegSummaryCleaning

to false

3 After editing the above tunefiles

a SSH in to all the ControllerVMs

b Run storfstool -- -Z

c Run the below command andcheck the values Tune valueshould be true

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=true

d Type below commands andapplied the tunes changesdynamically

echo false gttmpstprocfssystemtunecleanerEnableSegSummaryCleaning

e Verify that tune values arechanged Run below commandValue should be false

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=false

f umount tmpstprocfs

A cluster with 8TB or 12TB disk drivesmay experience IO stalls for severalminutes after another node fails in thecluster

CSCvp23718

Release Notes for Cisco HX Data Platform Release 4057

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)1 Update the conf file manuallyto change to the link local addressto

2 Restart scvmclient on thecontroller and ESX

HX 35(2b) FlexVol to HX 40 CSIupgrade does not work FlexVol continuesto work

CSCvp26319

35(2a)Disable rebalance until node healsthen reenable rebalance

7-node cluster healing very slowlybecause rebalance change AMS reqs areblocking healing process

CSCvp62512

35(2a)NAWith moderate cluster storage capacityusage under site failover cluster clustermay experience an all paths (APD) downcondition

CSCvq49412

40(1a)NAThis product includes Third-partySoftware that is affected by thevulnerabilities identified by the followingCommon Vulnerability and Exposures(CVE) IDs

CVE-2016-0762 CVE-2016-6797CVE-2016-6816 CVE-2016-8735CVE-2017-5647 CVE-2017-12615CVE-2017-12616 CVE-2017-12617CVE-2017-7674 CVE-2018-1304CVE-2018-8014 CVE-2018-1336CVE-2018-8034 CVE-2018-11784CVE-2019-0232

CSCvp36364

35(2d)NAstcli cluster info command needs toprovide UCSM VIP address Currently itshows ucsm-hostcom

CSCvq11456

35(2a)No workarounds currently Thestale entries do not affect the dayto day operations of HyperFlex

HyperFlex upgrade validation failedbecause cluster has extraneous stNodes ininternal database

CSCvq32530

35(2a)NAWhen not in maintenancemode all drivesappear in HX Connect and get updated asexpected

When in maintenance mode extra drivesare shown in HX Connect with no slotnumbers

CSCvq42378

35(2b)Contact TAC to cleanup duplicatenode

Nodes duplicated in HX Connect aftercluster recreate

CSCvq61825

Release Notes for Cisco HX Data Platform Release 4058

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)NACurrently HyperFlex installs on Hyper-Vdo not contain the stcli security whitelistcommandset

CSCvq66245

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

Release Notes for Cisco HX Data Platform Release 4059

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)[Hyper-V] Converged node expansionfails for incoming node with VIC 14xx

CSCvq39523

Release Notes for Cisco HX Data Platform Release 4060

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

1 Set Jumbo frame inWindowsHypervisor for incomingnode

Get-NetAdapter storage-data-a|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

Get-NetAdapter storage-data-b|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

2 Verify all interfaces are setcorrectly to support Jumbo frames

Get-NetAdapter storage |Get-NetAdapterAdvancedProperty| RegistryKeyword -matchjumbo | ft -auto

You should receive the followingmessage

Name DisplayName DisplayValue

RegistryKeyword RegistryValue

---- ----------- ------------

--------------- -------------

vswitch-hx-storage-data Jumbo

Packet 9014 Bytes

JumboPacket 9014

storage-data-b Jumbo Packet

Bytes 9014 JumboPacket

9014

storage-data-a Jumbo Packet

Bytes 9014 JumboPacket

9014

3 Reboot only the incoming nodeor the node being expanded

4 Retry Cluster Expansion in

Release Notes for Cisco HX Data Platform Release 4061

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

Edge does not support LFF-drives HyperFlex HyperV version does not yet support the new 12TB driveoption See the HyperFlex spec sheets for a full list of configurable options

bull New Cache and increased scale for Hyper-V mdashNVMe amp Optane SSDs are now supported as cachedrives for Hyper-V deployments Furthermore scale limits have been increased to 16+16(Converged+Compute-only) for both SFF (AF and Hybrid) amp LFF (Hybrid) clusters

bull Centralized Audit Log ExportmdashThis release adds support for audit logging via a remote syslog serverThis capability enables customers to retain audit logs from all HyperFlex nodes in a centralized remotesyslog server to meet retention and compliance requirements

bull DISA STIG CompliancemdashThis release adds new HX REST APIs for setting removing and checkingstatus of DISA STIGs for Controller VMs ESXi hosts and vCenter These APIs enable customers tomeet DISA security requirements by centrally and securely applying STIGs detecting and correctingfor drifts in any STIG settings

Supported Versions and System RequirementsCisco HX Data Platform requires specific software and hardware versions and networking settings forsuccessful installation

For a complete list of requirements see

bull Cisco HyperFlex Systems Installation Guide for VMware ESXi or

bull Cisco HyperFlex Systems Installation Guide for Microsoft Hyper-V

Hardware and Software Interoperability

For a complete list of hardware and software inter-dependencies refer to respective Cisco UCS Managerrelease version of Hardware and Software Interoperability for Cisco HyperFlex HX-Series

HyperFlex Software Versions

The HX componentsmdashCisco HX Data Platform Installer Cisco HX Data Platform and Cisco UCSfirmwaremdashare installed on different servers Verify that each component on each server used with and withinan HX Storage Cluster are compatible

bull HyperFlex does not support UCS Manager and UCS Server Firmware versions 40(4a) 40(4b)and 40(4c)

Do not upgrade to these versions of firmware

Do not upgrade to these versions of UCS Manager

Important

bull Verify that the preconfigured HX servers have the same version of Cisco UCS server firmware installedIf the Cisco UCS Fabric Interconnects (FI) firmware versions are different see the Cisco HyperFlexSystems Upgrade Guide for VMware ESXi Release 40 for steps to align the firmware versions

bull M4 For NEW hybrid or All Flash (Cisco HyperFlex HX240c M4 or HX220c M4) deploymentsverify that Cisco UCS Manager 31(3k) 32(3i) or 40(2b) or higher is installed For moreinformation see Recommended Cisco HyperFlex HX Data Platform Software Releases

Release Notes for Cisco HX Data Platform Release 409

Supported Versions and System Requirements

bull M5 For NEW hybrid or All Flash (Cisco HyperFlex HX240c M5 or HX220c M5) deploymentsverify that the recommended UCS firmware version is installed

For SED-based HyperFlex systems ensure that the A (Infrastructure) B (Bladeserver) and C (Rack server) bundles are at Cisco UCS Manager version 40(2b)or later for all SED M4M5 systems For more details see CSCvh04307 ForSED-based HyperFlex systems also ensure that all clusters are at HyperFlexRelease 35(2b) or later For more information see Field Notice (70234) andCSCvk17250

Important

bull To reinstall an HX server download supported and compatible versions of the software See theCisco HyperFlex Systems Installation Guide for VMware ESXi for the requirements and steps

bull Important For Intersight edge servers running older than 40(1a) CIMC version HUU is the suggestedmechanism to update the firmware

Table 1 HyperFlex Software Versions for M4M5 Servers

M5 Recommended FIServer Firmware

(be sure to review important notes above)

M4 Recommended FIServer Firmware

(be sure to review important notes above)

HyperFlexRelease

40(4h)40(4h)40(2b)

40(4h)40(4h)40(2a)

40(4h)40(4h)40(1b)

40(4e)40(4e)40(1a)

HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments

Cisco HX Data Platform Release 4x based Deployments

Confirm the component firmware on the server meets the minimum versions listed in the following tables

HyperFlex Edge does not support Cisco IMC versions 40(4a) 40(4b) 40(4c) 40(4d) and 40(4e)Important

Table 2 HX220c M4 HXAF220c M4 Cluster

Recommended Firmware Version - HXDP 4x

(be sure to review important note(s) above)

Component

402d0BIOS

40(2h)Cisco Integrated Management Controller(CIMC)

Release Notes for Cisco HX Data Platform Release 4010

HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments

Recommended Firmware Version - HXDP 4x

(be sure to review important note(s) above)

Component

13000012SAS HBA

43(2b)VIC 1227

0103 (Toshiba)

CS07 (Intel 480)

SSD

CN03 (Seagate)

CK03 (Seagate 18)

CK03 (Seagate 24)

sF3Q (Samsung)

3F3Q (Samsung)

HDD

40(2h)

Download Software

Host Upgrade Utility (HUU) Download Link

Table 3 HX220c M5 HXAF220c M5 Cluster

Recommended Firmware Version - HXDP 4x

(be sure to review important notes above)

Component

C220M5404p00224200755BIOS

40(4i)Cisco Integrated Management Controller(CIMC)

09000008SAS HBA

43(3c)VIC 1387

50(3e)VIC 1457

G201CS01 (Intel)

0104 (Toshiba)

0107 (Toshiba)

SSD

2F3Q (Samsung)

3F3Q (Samsung)

NOA4 (Seagate)

CK03 (Seagate 18tb)

CK03 (Seagate 24)

HDD

Release Notes for Cisco HX Data Platform Release 4011

HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments

Recommended Firmware Version - HXDP 4x

(be sure to review important notes above)

Component

40(4k)

Download Software

Host Upgrade Utility (HUU) Download Link

HyperFlex Licensing

As of version 26(1a) HyperFlex supports VMware PAC licensing Existing VMware embedded licenses willcontinue to be supported

As of version 25(1a) HyperFlex uses a smart licensing mechanism to apply your licenses See the CiscoHyperFlex Systems Installation Guide for VMware ESXi for details and steps

VMware vSphere Licensing Requirements

How you purchase your vSphere license determines how your license applies to your HyperFlex system

bull If you purchased your vSphere license with HyperFlex

Each HyperFlex server either has the Enterprise or Enterprise Plus edition preinstalled at the factory

bull HX Nodes have OEM licenses preinstalled If you delete or overwrite thecontent of the boot drives after receiving the HX servers you also delete thefactory-installed licenses

bull OEM license keys is a newVMware vCenter 60 U1b feature Earlier versionsdo not support OEM licenses

bull All factory-installed HX nodes share the same OEM license key WithvSphere OEM keys the Usage count can exceed the Capacity value

bull When you add an HX host to vCenter through the Add Host wizard in theAssign license section select the OEM license

We obfuscate the actual vSphere OEM license key for example0N085-XXXXX-XXXXX-XXXXX-10LHH

bull Standard Essentials Plus and ROBO editions are not available preinstalledon HX servers

Note

bull If you did NOT purchase your vSphere license with HyperFlex

The HX nodes have a vSphere Foundation license preinstalled After initial setup you can apply thelicense to a supported version of vSphere

bull If you purchased a vSphere PAC license

Follow the instructions in your PAC license letter from VMware to add the license to your MY VMwareaccount then follow the instructions to add your HX host to vCenter and assign the PAC license

Release Notes for Cisco HX Data Platform Release 4012

HyperFlex Licensing

HX Data Platform Software Versions for HyperFlex Witness Node

Witness Node VersionHyperFlex Release

10840(2b)

10840(2a)

10440(1b)

10440(1a)

Software Requirements for VMware ESXi

The software requirements include verification that you are using compatible versions of Cisco HyperFlexSystems (HX) components and VMware vSphere VMware vCenter and VMware ESXi

bull Verify that all HX servers have a compatible version of vSphere preinstalled

bull Verify that the vCenter version is the same or later than the ESXi version

bull Verify that the vCenter and ESXi versions are compatible by consulting the VMware ProductInteroperability Matrix Newer vCenter versions may be used with older ESXi versions so long as bothESXi and vCenter are supported in the table below

bull Verify that you have a vCenter administrator account with root-level privileges and the associatedpassword

For VIC1457 there is no support for ESXi 60Note

The below table applies for all of the following VMware vSphere Editions Enterprise Enterprise PlusStandard Essentials Plus ROBO

VMware vCenter VersionsVMware ESXi VersionsHyperFlex Version

60 U3 65 U3 67 U360 U3 65 U3 67 U340(2b)1

60 U3 65 U3 67 U360 U3 65 U3 67 U340(2a)2

60 U3 65 U3 67 U260 U3 65 U3 67 U2340(1b)

60 U3 65 U2 67 U260 U3 65 U2 67 U2440(1a)

1 Cisco HyperFlex Release 40(2) is the last major HyperFlex release that will support vSphere 60 (ESXiand vCenter) due to those versions reaching end of VMware general support on March 12 2020

2 Cisco HyperFlex Release 40(2) is the last major HyperFlex release that will support vSphere 60 (ESXiand vCenter) due to those versions reaching end of VMware general support on March 12 2020

3 Use of 67 U2 is not recommended see software advisory for CSCvq06952 and CSCvp88515 for furtherdetails

4 Use of 67 U2 is not recommended see software advisory for CSCvq06952 and CSCvp88515 for furtherdetails

Release Notes for Cisco HX Data Platform Release 4013

HX Data Platform Software Versions for HyperFlex Witness Node

Software Requirements for Microsoft Hyper-V

The software requirements include verification that you are using compatible versions of Cisco HyperFlexSystems (HX) components and Microsoft Hyper-V (Hyper-V) components

HyperFlex Software versions

The HX componentsmdashCisco HX Data Platform Installer Cisco HX Data Platform and Cisco UCSfirmwaremdashare installed on different servers Verify that each component on each server used with and withinthe HX Storage Cluster are compatible

bull Cisco HyperFlex M5 Converged nodesmdash For Hybrid (Cisco HyperFlex HX240c M5 HX220c M5)and All Flash (Cisco HyperFlex HXAF240cM5 HXAF220cM5) verify that Cisco UCSManager 40(2b)is installed HX 40 (1a) does not support Hyper-V on the All NVMe (HXAF220C-M5SN) nodes Fordetailed information on installation requirements and steps see theCisco HyperFlex Systems InstallationGuide on Microsoft Hyper-V

Table 4 Supported HyperFlex Software versions for M5 Servers on Hyper-V

M5 Recommended Server FirmwareHyperFlex Release

40(4h)40(2b)

40(4g)40(2a)

40(4e)40(1b)

40(4e)40(1a)

Release Notes for Cisco HX Data Platform Release 4014

Software Requirements for Microsoft Hyper-V

Table 5 Supported Microsoft Software versions

VersionMicrosoft Component

Windows Server 2016 Datacenter Core amp DesktopExperience

ForWindows Server 2016Datacenter Coreand Desktop Experience the Windows2016 ISO image should be Update BuildRevision (UBR) 1884 at a minimum

Note

Windows Server 2019Datacenter-DesktopExperienceis supported starting from HXDP 401(a) onwards

For Windows Server 2019 DesktopExperience theWindows 2019 ISO imageshould be Update Build Revision (UBR)107 at a minimum

Note

Windows Server 2019 DatacenterndashCore is notsupported currently

Also note that the following are currently notsupported

OEM activated ISOs and Retail ISOs are notsupported

Earlier versions ofWindows Server such asWindows2012r2 are not supported

Non-English versions of the ISO are not supported

Windows Operating System (Windows OS)

A Windows 2012 or later domain and forestfunctionality level

Active Directory

Supported Microsoft License Editions

The Microsoft Windows Server version that is installed on one or more HyperFlex hosts must be licensed asper Microsoft licensing requirements listed on Microsoft Licensing

Browser Recommendations

Use one of the following browsers to run the listed HyperFlex components These browsers have been testedand approved Other browsers might work but full functionality has not been tested and confirmed

Table 6 Supported Browsers

HX ConnectHX Data PlatformInstaller

Cisco UCS ManagerCisco IntersightBrowser

11 or higher11 or higher11 or higherNAMicrosoft InternetExplorer

70 or higher70 or higher57 or higher62 or higherGoogle Chrome

Release Notes for Cisco HX Data Platform Release 4015

Browser Recommendations

HX ConnectHX Data PlatformInstaller

Cisco UCS ManagerCisco IntersightBrowser

60 or higher60 or higher45 or higher57 or higherMozilla Firefox

NANA9 or higher10 or higherApple Safari

NANA35 or higherNAOpera

Notes

bull Cisco HyperFlex Connect

The minimum recommended resolution is 1024 X 768

bull Cisco HX Data Platform Plug-in

The Cisco HX Data Platform Plug-in runs in vSphere For VMware Host Client System browserrequirements see the VMware documentation at httpswwwvmwarecomsupportpubs

The HX Data Platform Plug-in is not displayed in the vCenter HTML client You must use the vCenterflash client

bull Cisco UCS Manager

The browser must support the following

bull Java Runtime Environment 16 or later

bull Adobe Flash Player 10 or higher is required for some features

For the latest browser information about Cisco UCSManager refer to themost recent Cisco UCSManagerGetting Started Guide

Cisco HX Data Platform Storage Cluster SpecificationsCluster Limits

bull Cisco HX Data Platform supports up to 100 clusters managed per vCenter as per VMware configurationmaximums

bull Cisco HX Data Platform supports any number of clusters on a single FI domain Each HX convergednode must be directly connected to a dedicated FI port on fabric A and fabric B without the use of a FEXC-series compute only nodes must also directly connect to both FIs B-series compute only nodes willconnect through a chassis IO module to both fabrics In the end the number of physical ports on the FIwill dictate the maximum cluster size and maximum number of individual clusters supported in a UCSdomain

The following table provides Cisco HX Data Platform Storage Cluster Specifications

Release Notes for Cisco HX Data Platform Release 4016

Cisco HX Data Platform Storage Cluster Specifications

Stretched Cluster (Availableon ESX Only)

Microsoft Hyper-VVMware ESXiNode

AllNVMe -HXAF220c-M5SN

HX240c-M5LHX220cM5

HX220cAF M5

HX240cM5

HX240cAF M5

HX240c-M5LHX220cM5

HX220cAF M5

HX240cM5

HX240cAF M5

HXAF220c-M5SN

AllNVMe -HXAF220c-M5SN

ESXi 65or 67only NotsupportedwithESXi 60orHyper-V

HX220cM5 Edge

HXAF220cM5 Edge

HX220cM4 Edge

HXAF220cM4 Edge

HX240c-M5LHX220cM5

HX220cAF M5

HX240cM5

HX240cAF M5

HX220cM4

HX220cAF M4

HX240cM4

HX240cAF M4

HXServers

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

C220M5C240M5B200M4B200M5

C240M5C220M5B200M4B200M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

mdashB200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

Compute-OnlyUCSB-SeriesC-SeriesServers

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

Convergednodesonly

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

SupportedNodes

Release Notes for Cisco HX Data Platform Release 4017

Cisco HX Data Platform Storage Cluster Specifications

Stretched Cluster (Availableon ESX Only)

Microsoft Hyper-VVMware ESXiNode

NA(requiresEnterpriseHXDP-PLicense)

NA(requiresEnterpriseHXDP-PLicense)

NA(requiresEnterpriseHXDP-PLicense)

Convergednodes3-16(12TBHDDoption isnotsupportedforHyperV)

Compute-onlynodes0-16

Convergednodes3-16

Compute-onlynodes0-16

NA(requiresEnterpriseHXDP-PLicense)

M4Convergednodes 3

M5Convergednodes23or 4

RequiresHXDP-ELicense

Convergednodes3-16(12TBdrive islimited toamaximumof 8nodes)

Compute-onlynodes0-16(12TBdrive islimited toamaximumof 8nodes)

Convergednodes3-32(76TBdriveconfigsarelimited tomaximumof 12drivesnodeon AF240)

Compute-onlynodes0-32

HXDP-SLicensedNodeLimits

11 ratioofHXDP-StoComputeonlynodes

(MinmdashMax)

Convergednodes2-16 perSite

Compute-onlynodes0-21 perSite

(up tomaxclustersize)

Convergednodes2-8 perSite

Compute-onlynodes0-16 perSite

(up tomaxclustersize)

Convergednodes2-16 perSite

Compute-onlynodes0-21 perSite

(up tomaxclustersize)

Convergednodes3-16(12TBHDDoption isnotsupportedforHyperV)

Compute-onlynodes0-16

Convergednodes3-16

Compute-onlynodes0-16

Convergednodes3-32

ComputeOnlynodes0-32

(up tomaxclustersize)

Convergednodes 3

(requiresHXDP-ELicense)

Convergednodes3-16(12TBdrive islimited toamaximumof 8nodes)

Compute-onlynodes0-32(12TBdrive islimited toamaximumof 16nodes)

Convergednodes3-32(76TBdriveconfigsarelimited tomaximumof 12drivesnodeon AF240)

Compute-onlynodes0-32

(up tomaxclustersize)

HXDP-PLicensedNodeLimits

12 ratioofHXDP-PtoComputeonlynodes

(MinmdashMax)

24 perSite 64percluster

24 perSite 48percluster

32 perSite 64percluster

32326434864MaxClusterSize

Release Notes for Cisco HX Data Platform Release 4018

Cisco HX Data Platform Storage Cluster Specifications

Stretched Cluster (Availableon ESX Only)

Microsoft Hyper-VVMware ESXiNode

212121111121mdash2121MaxComputetoConvergedratio

NoExpansion

Requires Enterprise license

Requires uniform expansion across both sites

Guidelines and Limitationsbull Starting with release 40(2a) SCVM is no longer needed on a Compute node

bull HX REST API Access Token Management ndash Applications leveraging HX REST APIs should re-useaccess tokens when making API calls Once obtained using the AAA Obtain Access Token API accesstokens are valid for 18 days (1555200 seconds) In addition AAA enforces rate limiting on ObtainAccess Token API requests in a 15 minute window auth can be invoked (successfully) a maximum of5 times A user is allowed to create a maximum of 8 unrevoked tokens Subsequent call to auth willautomatically revoke the oldest issued token tomake room for the new token Amaximum of 16 unrevokedtokens can be present in system In order to prevent brute-force attacks after 10 consecutive failedauthentication attempts a user account is locked for a period of 120 secondsFor more information seeCisco HyperFlex Systems REST API Reference guide

HxConnect makes use of AAA Authentication REST API for login and the above rate limit applies toHxConnect also

Upgrade Guidelines

The following list is a highlight of critical criteria for performing an upgrade of your HyperFlex system

bull Unsupported HX Data Platform 17x 18x 20 21x 25x and 26x clustersmdashUsers from anyversion prior to 26(1a) must step through an intermediate version before upgrading to 40 or later releasesIf you need to upgrade your environment from a Cisco HyperFlex HX Data Platform software releasethat is past the last date of support to the latest suggested release on the Cisco Software Download sitesee Cisco HyperFlex Systems Upgrade Guide for Unsupported Cisco HXReleases For more informationsee the Software Advisory for CSCvq66867 WARNING Only Use HXDP 26(1e) Upgrade PackageWhen Upgrading From HXDP 18(1a)-18(1e)

bull Hypercheck Health Check UtilitymdashCisco recommends running this proactive health check utility onyour HyperFlex cluster prior to upgrade These checks provide early visibility into any areas that mayneed attention andwill help ensure a seamless upgrade experience For more information see the HyperFlexHealth amp Pre-Upgrade Check Tool TechNote for full instructions on how to install and run Hypercheck

bull vSphere 67 Software AdvisorymdashDo not upgrade to Cisco HX Data Platform Release 40(1a) whenrunning ESXi 67U1 EP06 (build 11675023) Do not upgrade to 67U1 EP06 (build 11675023) ifrunning Cisco HX Data Platform Release 40(1a) See the Software Advisory CSCvo56350 for furtherdetails

Release Notes for Cisco HX Data Platform Release 4019

Guidelines and Limitations

The software build version posted at release will override any other local versions

bull Required vCenter upgrademdashFor enhanced security Cisco HX Data Platform Release 35(1a) or laterrequires the use of TLS 12 Therefore vCenter must be upgraded to 60 U3f or later before upgradingto Cisco HX Data Platform Release 35 or later In addition ESXi should be upgraded as required tomeet HX Data Platform compatibility requirements

bull Minimum HXDP version for upgrademdashHXData Platform clusters running 26(1a) or later may upgradedirectly to 40 using the HX Connect UI

bull Cluster ReadinessmdashEnsure that the cluster is properly bootstrapped and the updated plug-in is loadedbefore proceeding Manual cluster bootstrap is required for upgrade from a pre-35 release

bull Cluster ReadinessmdashEnsure that the cluster is properly bootstrapped and the updated plug-in is loadedbefore proceeding Manual cluster bootstrap is required for HX releases earlier than 35(1a) For moreinformation see theManual Bootstrap Upgrade Process in the Cisco HyperFlex Systems Upgrade Guidefor VMware ESXi Release 40 Do not skip this cluster bootstrap step it is required for all upgradesuntil HX Release 35(1a) Auto bootstrap is supported beginning with HX release 35(1a) For moreinformation see the Auto Bootstrap Upgrade Process from HX Connect UI in the Cisco HyperFlexSystems Upgrade Guide for VMware ESXi Release 40

Manual bootstrap is not supported on Intersight clusters

bull Initiating Upgrade―Use the HX Connect UI or CLI stcli commands when upgrading from 25(1a)or later releases Use either the CLI stcli commands or the HX Data Platform Plug-in to the vSphereWeb Client when upgrading from a pre-25(1a) release The vCenter plug-in should not be used forupgrades starting with the 25(1a) release The vCenter plug-in should not be used for upgrades startingwith the 25(1a) release

If the current cluster version is at 35(1a) or above you do not need to use the stcli command Directupgrade to 40 is possible

bull Complete your Upgrade―The self-healing (or rebalance) capability is disabled temporarily during theupgrade window If the upgrade fails you should complete the upgrade as soon as possible

bull ESXi and HXDP Compatibility―Ensure your cluster is running a compatible version of ESXi basedon the running the HXData Platform version (see the section Software Requirements for VMware ESXi)ESXi compatibility is determined by the major version and update release of ESXi It is generally bestto upgrade HXDP and ESXi together if combining the upgrade operations into a single optimized rebootWhen running a split upgrade first upgrade the HX Data Platform then proceed to upgrade ESXi

bull If Upgrading to vSphere 65

bull Certain cluster functions such as native and scheduled snapshotsReadyClones and Enter or Exit HX Maintenance Mode will not operatefrom the time the upgrade is started until the HX Data Platform upgrade iscomplete

bull After upgrading ESXi using the offline zip bundle use the ESX ExitMaintenance Mode option The HX Exit Maintenance Mode option doesnot operate in the vSphere Web Client until the HX Data Platform upgradeis complete

Note

Release Notes for Cisco HX Data Platform Release 4020

Guidelines and Limitations

bull vSphere 60 UpgradesmdashUsers on vSphere 60 migrating to 65 upgrade components in the followingorder

1 Upgrade HX Data Platform and UCS firmware

2 Upgrade HX Data Platform and ESXi

3 Upgrade HX Data Platform only first then upgrade ESXi or UCS firmware or both

bull M4 Server Firmware UpgradesmdashUpgrade server firmware to ensure smooth operation and to correctknown issues Specifically newer SAS HBA firmware is available in this release and is recommendedfor long-term stability

bull Users are encouraged to upgrade to 31(3c) C-bundle or later whenever possible

bull Users running C-bundle versions before 31(2f) must upgrade server firmware by performing acombined upgrade of UCS server firmware (C-bundle) to 31(3c) or later and HX Data Platform to25 Do not split the upgrade into two separate operations

bull If the cluster is already on 31(2f) C-bundle or later you may perform an HX Data Platform onlyor combined upgrade as required

bull M5 Server Firmware UpgradesmdashM5 generation servers must run firmware version 32(2d) or later

bull Firmware Downgrades mdashDowngrading UCSM from the HX-installer is not supported

bull M4M5 Mixed DomainsmdashA mixed domain occurs when a new separate M5 cluster is installed underthe same UCS domain that contains existing M4 clusters Under these conditions orchestrated UCSserver firmware upgrade will not operate until Cisco HX Data Platform Release 26 or later is installedon the M4 clusters Therefore it is best practice to first upgrade UCS server firmware to the latest 31(3)or 32(2) patch release before adding a new M5 cluster to the existing UCS domain Additionally any17 HX Data Platform clusters must first be upgraded before adding any new M5 clusters to the samedomain

bull Maintenance WindowmdashIf upgrading both HX Data Platform and UCS firmware you can select eithera combined or split upgrade through the vSphere HX Data Platform Plug-in depending on the length ofthe maintenance window Cisco UCSManager infrastructure upgrade is only supported using AutoInstalland the direct server firmware upgrade should be performed only through the upgrade orchestrationframework provided by the HX Data Platform Plug-in

bull Unsupported Self-Encrypting Drives (SEDs)mdashIf adding or replacing self-encrypting drives (SEDs)that have been recently qualified in newer versions of HX Data Platform insert the new drives only afterupgrading HX Data Platform to a compatible version All drives must be SED drives mixing SED andnon-SED is not supported

bullbull Enabling External Host AccessmdashWith Cisco HX Data Platform Release 40(1a) port 445 on themanagement network is blocked for enhanced security Note that prior to 40 port 445 port was openenabling external host access If you are upgrading to 40(1a) from a prior release and would like tocontinue external host access you can use a utility to open select hosts For more information aboutenabling external host access see the ConfiguringHyperFlex Share to SCVMM section in the InstallationGuide for Microsoft Hyper-V

Release Notes for Cisco HX Data Platform Release 4021

Guidelines and Limitations

Mixed Cluster Expansion Guidelinesbull Expanding existing M4 cluster with M5 converged nodes is supported

bull Expanding existing M5 cluster with M4 converged nodes is not supported

bull Expanding existing mixed M4M5 cluster with M4 or M5 converged nodes is supported

bull Adding any supported compute-only nodes is permitted with all M4 M5 and mixed M4M5 clustersusing the HX Data Platform 26 or later Installer Some example combinations are listed here manyother combinations are possibleExample combinationsExpand mixed M4M5 cluster with compute-only B200 C220 C240 M4M5Expand M4 cluster with compute-only B200 M5 C220 M5 C240M5

bull Only expansion workflow is supported to create a mixed cluster Initial cluster creation with mixedM4M5 servers is not supported

bull All M5 servers must match the form factor (220240) type (HybridAF) security capability (Non-SEDonly) amp disk configuration (QTY capacity and non-SED) of the existing M4 servers

bull HX220-M5 will use a maximum of 6 capacity disks (2 disk slots to remain empty) when mixedwith HX220-M4

bull HX Edge SED LFF Hyper-V and Stretched Clusters do not support mixed M4 and M5 clusters

Security FixesThe following security issues are resolved

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-13132 CVE-2019-9924CSCvq6313840(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-11719 CVE-2019-11727CVE-2019-11729

CSCvq7124040(2a)

Release Notes for Cisco HX Data Platform Release 4022

Mixed Cluster Expansion Guidelines

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-1125CSCvr0633940(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2015-9383 CVE-2018-14498CVE-2018-20406 CVE-2018-20852CVE-2019-10160 CVE-2019-13117CVE-2019-13118 CVE-2019-14287CVE-2019-14973 CVE-2019-15903CVE-2019-17546 CVE-2019-18197CVE-2019-18218 CVE-2019-5010CVE-2019-5094 CVE-2019-5481CVE-2019-5482 CVE-2019-9636CVE-2019-9740 CVE-2019-9947CVE-2019-9948

CSCvs0609440(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2017-13168 CVE-2017-18174CVE-2017-18216 CVE-2018-10876CVE-2018-10877 CVE-2018-10878CVE-2018-10879 CVE-2018-10880CVE-2018-10881 CVE-2018-10882CVE-2018-10902 CVE-2018-10938CVE-2018-12233 CVE-2018-12896CVE-2018-13053 CVE-2018-13094CVE-2018-13096 CVE-2018-13405CVE-2018-13406 CVE-2018-14609CVE-2018-14617 CVE-2018-14633CVE-2018-14734 CVE-2018-15572CVE-2018-15594 CVE-2018-16276CVE-2018-16658 CVE-2018-17182CVE-2018-17972 CVE-2018-18021CVE-2018-18690 CVE-2018-18710CVE-2018-6554 CVE-2018-6555CVE-2018-9363

CSCvp6501940(2a)

This vulnerability is due to insufficientHTML iframe protection An attackercould exploit this vulnerability bydirecting a user to an attacker-controlledweb page that contains amaliciousHTMLiframe A successful exploit could allowthe attacker to conduct clickjacking orother clientside browser attacks

NACSCvo9851640(2a)

Release Notes for Cisco HX Data Platform Release 4023

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerability is due to insufficientauthentication for the statistics collectionservice An attacker could exploit thisvulnerability by sending properlyformatted data values to the statisticscollection service of an affected deviceA successful exploit could allow theattacker to cause the web interfacestatistics view to present invalid data tousers

NACSCvj9558440(2a)

The vulnerability is due to insufficientCSRF protections for the web UI on anaffected device An attacker could exploitthis vulnerability by persuading a user ofthe interface to follow a malicious linkA successful exploit could allow theattacker to perform arbitrary actions withthe privilege level of the affected user

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvp2434340(2a)

Vulnerabilities with a version of cJSONidentified by CVE-2019-11834 andCVE-2019-11835

For more information see the relatedCisco Security Advisory

CVE-2019-11834 CVE-2019-11835CSCvq1994940(2a)

HX ESXi image patches for MCEPSCand TAA vulnerabilities with VMwareESXi identified by CVE-2018-12207 andCVE-20190-11135

For more information see the relatedCisco Security Advisory

CVE-2018-12207 CVE-2019-11135CSCvr5439840(2a)

CP networking vulnerabilities affectingthe Linux kernel identified byCVE-2019-11477 CVE-2019-11478CVE-2019-11479

For more information see the relatedCisco Security Advisory

CVE-2019-11477 CVE-2019-11478CVE-2019-11479

CSCvq1954640(2a)

Qualification of Microsoft securitypatches for vulnerabilities withMicrosoftHyper-V hypervisor identified withCVE-2018-12207 CVE-2019-11135

For more information see the relatedCisco Security Advisory

CVE-2018-12207 CVE-2019-11135CSCvr5439940(2a)

Release Notes for Cisco HX Data Platform Release 4024

Security Fixes

DescriptionCVEDefect IDRelease

OpenSSH vulnerabilities identified byCVE-2019-6111

For more information see the relatedCisco Security Advisory

CVE-2016-10708 CVE-2018-15473CVE-2018-20685 CVE-2019-6109CVE-2019-6111

CSCvp7646340(2a)

Multiple vulnerabilities associated withOpenSSL and LibNSS

For more information see the relatedCisco Security Advisory

CVE-2016-2105 CVE-2016-2106CVE-2016-2107 CVE-2016-2109CVE-2016-2176 CVE-2016-2177CVE-2016-2178 CVE-2016-2179CVE-2016-2180 CVE-2016-2181CVE-2016-2182 CVE-2016-2183CVE-2016-6302 CVE-2016-6303CVE-2016-6304 CVE-2016-6305CVE-2016-6306 CVE-2016-6307CVE-2016-6308 CVE-2016-7055CVE-2016-8610 CVE-2017-3731CVE-2017-3732 CVE-2017-3735CVE-2017-3736 CVE-2017-3737CVE-2017-3738 CVE-2018-0495CVE-2018-0732 CVE-2018-0734CVE-2018-0735 CVE-2018-0737CVE-2018-0739 CVE-2018-12384CVE-2018-12404 CVE-2018-5407CVE-2019-1559

CSCvp6667940(2a)

libpng vulnerability identified byCVE-2016-10087

For more information see the relatedCisco Security Advisory

CVE-2016-10087CSCvp6655540(2a)

Vulnerabilities associated with libjpg andlibpng

For more information see the relatedCisco Security Advisory

CVE-2014-9092 CVE-2016-3616CVE-2017-15232 CVE-2018-11212CVE-2018-11213 CVE-2018-11214CVE-2018-1152 CVE-2018-13785

CSCvp3458640(2a)

Vulnerabilities associated with Glibidentified by CVE-2018-16428 andCVE-2018-16429

For more information see the relatedCisco Security Advisory

CVE-2018-16428 CVE-2018-16429CSCvp3120740(2a)

Multiple vulnerabilities associated withcurl expat python 27 python 35 36and 37 freetype and giflib

For more information see the relatedCisco Security Advisory

CVE-2019-15133 CVE-2019-15903CVE-2019-5010 CVE-2019-5481CVE-2019-5482 CVE-2019-9636CVE-2019-9740 CVE-2019-9947CVE-2019-9948

CSCvr3690340(2a)

Release Notes for Cisco HX Data Platform Release 4025

Security Fixes

DescriptionCVEDefect IDRelease

Multiple third party softwarevulnerabilities

For more information see the relatedCisco Security Advisory

CVE-2019-14379 CVE-2019-12384CVE-2019-14439

CSCvq9203240(2a)

Vulnerabilities associated with elfutils

For more information see the relatedCisco Security Advisory

CVE-2018-16062 CVE-2018-16402CVE-2018-16403 CVE-2018-18310CVE-2018-18520 CVE-2018-18521CVE-2019-7149 CVE-2019-7150CVE-2019-7665

CSCvq4325040(2a)

Vulnerabilities associated with vim

For more information see the relatedCisco Security Advisory

CVE-2017-5953 CVE-2019-12735CSCvq4323040(2a)

Vulnerabilities associated with sqlite3

For more information see the relatedCisco Security Advisory

CVE-2016-6153 CVE-2017-10989CVE-2017-13685 CVE-2017-2518CVE-2017-2519 CVE-2017-2520CVE-2018-20346 CVE-2018-20505CVE-2018-20506 CVE-2019-8457CVE-2019-9936 CVE-2019-9937

CSCvq4321340(2a)

Vulnerabilities associated with glib20

For more information see the relatedCisco Security Advisory

CVE-2018-20843CSCvq4320940(2a)

Vulnerabilities associated with expat

For more information see the relatedCisco Security Advisory

CVE-2018-20843CSCvq4320540(2a)

Vulnerabilities associated with bzip2

For more information see the relatedCisco Security Advisory

CVE-2016-3189 CVE-2019-12900CSCvq4319440(2a)

Mulitple Vulnerabilities associated wtihNodeJS

For more information see the relatedCisco Security Advisory

CVE-2018-12115 CVE-2018-0734CVE-2018-5407 CVE-2018-12120CVE-2018-12121 CVE-2018-12122CVE-2018-12123 CVE-2018-12116CVE-2019-5737 CVE-2019-5739

CSCvq1069440(2a)

Vulnerabilities associated with PalletsJinja strformat_map and identified byCVE-2019-10906 and CVE-2016-10745

For more information see the relatedCisco Security Advisory

CVE-2019-10906 CVE-2016-10745CSCvq1038840(2a)

Release Notes for Cisco HX Data Platform Release 4026

Security Fixes

DescriptionCVEDefect IDRelease

Multiple Security Vulnerabilities

For more information see the relatedCisco Security Advisory

CVE-2014-8501 CVE-2014-9939CVE-2015-9262 CVE-2016-10087CVE-2016-2226 CVE-2016-4487CVE-2016-4488 CVE-2016-4489CVE-2016-4490 CVE-2016-4491CVE-2016-4492 CVE-2016-4493CVE-2016-6131 CVE-2018-10963CVE-2018-13785 CVE-2018-17100CVE-2018-17101 CVE-2018-18557CVE-2018-18661 CVE-2018-7456CVE-2018-8905

CSCvq0917840(2a)

Vulnerabilities associated withlibseccomp

For more information see the relatedCisco Security Advisory

CVE-2019-9893CSCvq0756840(2a)

Vulnerabilities associated withgdk-pixbuf

For more information see the relatedCisco Security Advisory

CVE-2017-12447CSCvq0675540(2a)

Vulnerabilities associated withpython-crypto

For more information see the relatedCisco Security Advisory

CVE-2018-6594CSCvp9381740(2a)

Vulnerabilities associated with wget

For more information see the relatedCisco Security Advisory

CVE-2018-20483 CVE-2019-5953CSCvp8672140(2a)

Multiple vulnerabilities associated withPython Crypto

For more information see the relatedCisco Security Advisory

CVE-2018-6594CSCvp6674840(2a)

Vulnerabilities associated withsensible-utils

For more information see the relatedCisco Security Advisory

CVE-2017-17512CSCvp6673440(2a)

Multiple vulnerabilities associated withPython

For more information see the relatedCisco Security Advisory

CVE-2018-1000030CSCvp6668940(2a)

Release Notes for Cisco HX Data Platform Release 4027

Security Fixes

DescriptionCVEDefect IDRelease

Multiple patch vulnerabilities identifiedby CVE-2016-10713

For more information see the relatedCisco Security Advisory

CVE-2016-10713CVE-2018-1000156 CVE-2018-6951

CSCvp6667240(2a)

Vulnerabilities associatedwith Little CMSidentified by CVE-2016-10165

For more information see the relatedCisco Security Advisory

CVE-2016-10165 CVE-2018-16435CSCvp6666440(2a)

Vulnerabilities associated with TenableScan identified byCVE-2011-5325CVE-2019-11068CVE-2016-7076CVE-2017-1000368

For more information see the relatedCisco Security Advisory

CVE-2011-5325 CVE-2016-7076CVE-2017-1000368CVE-2019-11068

CSCvp6474640(2a)

Vulnerabilities associated with libxml2294 and earlier as used in XMLSec1223 and earlier and other products andidentified by CVE-2016-9318

For more information see the relatedCisco Security Advisory

CVE-2016-9318 CVE-2017-16932CVE-2017-18258 CVE-2018-14404CVE-2018-14567

CSCvp3479240(2a)

Vulnerabilities associated with LFTPRemote File Names Unauthorized AccessVulnerability and identified byCVE-2018-10916

For more information see the relatedCisco Security Advisory

CVE-2018-10916CSCvp2926640(2a)

Vulnerabilities associated withCryptographyDeprecationWarning signerand verifier have been deprecated

For more information see the relatedCisco Security Advisory

CVE-2018-7750CSCvo3409740(2a)

Multiple Vulnerabilities in python python27 35

For more information see the relatedCisco Security Advisory

CVE-2019-16056CSCvr3979340(2a)

Release Notes for Cisco HX Data Platform Release 4028

Security Fixes

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toexecute commands as the root user

The vulnerability is due to insufficientinput validation An attacker could exploitthis vulnerability by connecting to thecluster service manager and injectingcommands into the bound process Asuccessful exploit could allow the attackerto run commands on the affected host asthe root user

Cisco has released software updates thataddress this vulnerability There areworkarounds that address thisvulnerability

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvq2417640(1b)35(2g)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

The vulnerability is due to an unprotectedlistening interface An attacker couldexploit this vulnerability by connectingto the listening interface and injectingcommands to the bound process Anexploit could allow the attacker to runcommands on the affected host as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvj9560640(1b)35(2g)

Release Notes for Cisco HX Data Platform Release 4029

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerabilities associated with theJVM18U121memory leak found duringVC alarms (concurrent 40 calls usingREST API)

CVE-2017-10053 CVE-2017-10067CVE-2017-10074 CVE-2017-10078CVE-2017-10081 CVE-2017-10087CVE-2017-10089 CVE-2017-10090CVE-2017-10096 CVE-2017-10101CVE-2017-10102 CVE-2017-10107CVE-2017-10108 CVE-2017-10109CVE-2017-10110 CVE-2017-10111CVE-2017-10115 CVE-2017-10116CVE-2017-10118 CVE-2017-10135CVE-2017-10176 CVE-2017-10193CVE-2017-10198 CVE-2017-10243CVE-2017-10274 CVE-2017-10281CVE-2017-10285 CVE-2017-10295CVE-2017-10345 CVE-2017-10346CVE-2017-10347 CVE-2017-10348CVE-2017-10349 CVE-2017-10350CVE-2017-10355 CVE-2017-10356CVE-2017-10357 CVE-2017-10388CVE-2017-3509 CVE-2017-3511CVE-2017-3526 CVE-2017-3533CVE-2017-3539 CVE-2017-3544CVE-2018-2579 CVE-2018-2582CVE-2018-2588 CVE-2018-2599CVE-2018-2602 CVE-2018-2603CVE-2018-2618 CVE-2018-2629CVE-2018-2633 CVE-2018-2634CVE-2018-2637 CVE-2018-2641CVE-2018-2663 CVE-2018-2677CVE-2018-2678 CVE-2018-2783CVE-2018-2790 CVE-2018-2794CVE-2018-2795 CVE-2018-2796CVE-2018-2797 CVE-2018-2798CVE-2018-2799 CVE-2018-2800CVE-2018-2814 CVE-2018-2815CVE-2018-2952 CVE-2018-3136CVE-2018-3139 CVE-2018-3149CVE-2018-3150 CVE-2018-3169CVE-2018-3180 CVE-2018-3183CVE-2018-3214 CVE-2019-2422

CSCvo8899740(1b)

Tomcat and Nginx logs are not beingcollected in the support bundle generatedthrough HX Connect in Release 35

NACSCvm5803140(1b)

The vulnerabilities associated with thelibmspack software package versionincluded in Cisco HX Data Platform

CVE-2018-18584

CVE-2018-18585

CSCvn3511940(1a)

Release Notes for Cisco HX Data Platform Release 4030

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerabilities associated withFasterXML Jackson-Databind TimeValueField Denial of Service

CVE-2018-14719

CVE-2018-14720

CVE-2018-1000873

CVE-2018-14721

CVE-2018-19360

CVE-2018-19362

CVE-2018-19361

CVE-2018-14718

CSCvn8228240(1a)

The vulnerabilities associated with theOpenSSL Protocol software packageversion included in Cisco HX DataPlatform

CVE-2013-3587CSCvo0505440(1a)

The vulnerabilities associated withThird-Party Software Denial of Service

CVE-2018-16487

CVE-2018-19361

CSCvo2781840(1a)

The vulnerabilities associated withLinuxkernel for Ubuntu 1710

CVE-2018-1092

CVE-2018-7492

CVE-2018-8087

CVE-2018-1068

CVE-2018-8781

CSCvm5314935(2a)

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)25(1d)stcli services dns remove should remove the DNS serverinfo from the interface files

CSCvs70967

40(2b)40(1b)HX 40(1b) new deployment from Intersight productioncloud

User receives following error while enabling CSI(Kubernetes integration)

Failure occurred during volume_access_enable Errorwas Nonzero exit code 1

CSCvt10522

40(2b)40(2a)Receive the following alertevent in HX Connect HXController VM HOSTNAME one or more configuredDNS servers not responding

CSCvt13947

Release Notes for Cisco HX Data Platform Release 4031

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)40(2a)API UI CLI show few Drive slots Empty after upgradeto 402a

The drives are seen in the sysmtool lsscsi and stclicommand outputs

The cluster is healthy with no errors

CSCvt14914

40(2b)35(2d)After successfully changing the DNS server on the HXcontroller we still can see the original DNS entry thatwas added during the deployment

CSCvs69154

40(2b)35(2e)HyperFlex Connect and VCenter show APD alarms forHX and non HX datastores This implies there is an issuewith HyperFlex when there may not be

CSCvs30080

40(2b)40(2a)An issue where all the disks in the node get locked afterrebooting the node

We successfully unlocked using sed-clientsh -Ucommand but wanted to test with another reboot anddrives locked again

CSCvs74286

40(2b)40(2a)When running stcli license commands on HyperFlexthe following error is seen

rootSpringpathControllertmp stcli license show all

Show smart licensing failed

Smart Agent is not ready please wait a minute and tryagain

CSCvt13929

40(2b)40(2a)After upgrading to HX 40(2a) the first RoboEdge nodethat gets rebooted then ignores persistent drivesPIDHX-HD24TB10K4KN

Cluster will be in degraded state and have reducedcapacity

CSCvt20144

40(2b)35(2b)Zookeeper fails to start while Exhibitor is runninghowever echo srvr returns nothing

CSCvs21562

40(2b)40(2a)When performing an HyperFlex upgrade a validationwarning may occur due to the host not having EnterprisePlus or Enterprise hypervisor licensing

Upgrade Validation Warning

ESXi host esx1labtest should be configured withVMware Enterprise license for upgrade to continue

CSCvs91787

Release Notes for Cisco HX Data Platform Release 4032

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)35(2g)Cluster expansion fails at Config Installer stage whenthe root and admin password for the storage controller(SCVM) are different

CSCvs69317

40(2b)35(2g)Panic while upgrading ESXiCSCvt06983

40(2b)40(2a)A cluster node running may hang in the Linux kernelThis is classified as an oops and a deviation from theexpected behavior

CSCvs54285

40(2b)35(2g)Rebalance failing on 10+10 node HX 35(2g) stretchcluster

CSCvs69007

40(2b)35(2d)The cluster becomes inaccessible to the IOVisorCSCvr54687

40(2b)40(2a)Panic on the storage controllerCSCvt61297

40(2b)40(2a)The size of support bundle is very large when collectedwith storfs-support command

CSCvt63306

40(2b)40(2a)storfs crash associated withFileSystemUsageWarningEvent orFileSystemUsageAlertEvent due to high usage ofvarstv above 80

CSCvt72807

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

40(2a)35(2d)In normal scenarios HX upgrade gets pendingACK afterupdating service profile In this case UCS only upgradewas stalled because after updating service profile HXupgrade was not getting pending ACK and it was alwayswaiting state

CSCvo39912

40(2a)35(1a)When using motherBoardReplace-12 to clean ZK fromold stNodepNodes resulted in unmounted datastore andmaking the size of the Hyperflex Datastores 0 resultingin all VMs in the cluster going offline

CSCvq39471

40(2a)26(1e)Upgrading cluster and getting error

Failed upgrade validations Checking vCenterconfiguration Reason Upgrade validations failed DRSFault Insufficient resources to satisfy configured failover

CSCvm77294

Release Notes for Cisco HX Data Platform Release 4033

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)The cluster expand fails on a node with DR replicationconfigured

When a HX cluster which has DR replication configuredis expanded we see the installer UI pulling in thereplicationVLAN information instead of themanagementVLAN information Even if we change that informationto the correct mgmt VLAN id and name it it does notseem to work as the node is configured with the VLANof the replicationVLAN in ESXi This leads to the failureof the node add with the host unreachable error

CSCvo70650

40(2a)35(1a)Customer reported that server didnt complete thefirmware upgrade automatically one by one

The user manually needs to put the host into maintenancemode and they manually acknowledged the pendingrequests to finish the UCS firmware upgrade

Maintenance policy is set to default (User-ack) as perthe design

CSCvo91624

40(2a)35(1i)When the cluster is in Failed state and stcli noderemove is attempted the output shows successful thoughthe node failed to get removed from the cluster

CSCvo93017

40(2a)35(1a)HyperFlex cluster upgrade may fail during validationwith a DRS Validation failed error

CSCvp31021

40(2a)35(2b)HX cluster expansion will fail with the following errormessage

MAC address pool configuration failure150[ErrorDescription] bad address block range definitioncollision

CSCvp58318

40(2a)35(2b)Memory usage by carbon cacheCSCvq34873

40(2a)35(2g)Upgrading ESXi cluster fails with error Nodemaintenance mode failed

CSCvr03240

40(2a)35(2e)storfs process does not automatically start on Exit HXMaintenance Mode or other tasks which restart StorageController

CSCvr88978

40(2a)35(2a)Post install script fails with message

Failed to execute ipmitool on HX node

CSCvp10707

40(2a)40(1a)HyperFlex expansionworkflow doesnt pull VLANnamecorrectly

CSCvp46539

Release Notes for Cisco HX Data Platform Release 4034

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)During HX cluster deployment - Validate ClusterCreation phase - HX Installer might fail with the error

fromvarlogspringpathstDeploylog

CSCvq45087

40(2a)35(2b)HX Installer fails to login to the SOL for configuringESX hosts

CSCvq91380

40(2a)35(2d)If DRS is enabled with configuration parameters otherthen fully automated ie manual or partially automatedthan this will change to fully automated during clusterexpansion

CSCvr44222

40(2a)35(1a)One of the validation task name has a typo The taskname is Validating node for mixed of SED and non-SEDdisks

It should be Validating node for mix of SED andnon-SED disks

CSCvp82175

40(2a)30(1i)This combination of using smaller NVMe drives is notsupported in expand operation For example if the drivesare only 375GB and they cannot be added to an existingcluster with larger caching SSD

CSCvo12359

40(2a)35(1a)HyperFlex includes a version of OpenSSL that is affectedby the vulnerabilities identified by the followingCommon Vulnerability and Exposures

CVE-2018-0495

CSCvp66679

40(2a)30(1e)SDK invoke exception when taking a native snapshot ona VM with GPU pci passthrough attached

CSCvo00511

40(2a)30(1i)Node replacement script fails due to EAM errorCSCvo62867

40(2a)35(1a)While pairing a cluster (HX release gt= 35) and cluster(HX release lt 35) remote replication network test failson the cluster (HX release lt 35)

CSCvo79760

40(2a)30(1b)Fixed the issue in the latest support-workflow bundleThis should not be hit anymore

CSCvo87061

40(2a)35(2a)On a 3 node cluster the MBreplace script fails as thescript looks for HEALTHY cluster status

CSCvo87080

40(2a)35(2a)MbReplace script (tar) hangs when running on 352b or40

CSCvp12359

Release Notes for Cisco HX Data Platform Release 4035

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)HX replication cleanup failing with error NA 35(2a) -INFODR state is not clean

CSCvp63958

40(2a)40(1a)The check_stig_parameters API incorrectly shows thecompliance state of the cluster as not STIG compliant

CSCvp66277

40(2a)35(2c)The snapshot creation on CBT enabled VM fails witherror Failed in vmreparent vmkfstools clone1

CSCvq06952

40(2a)35(2a)HyperFlex GUI when setting up remote DR-replicationpartnerpeer should show remote replication partner TCPport reachability test results

This will provide quick upfront connectivity results thatfrequently takes customer a while to manually check andconfirm

If reachability tests fail an info window and HX alertshould trigger that list the destination peer IP and thespecific port(s) that could not be reache

CSCvo60587

40(2a)35(2b)If you have more than one IP pool configured for areplication network and local replication network testsare failing you might be running into this issue

CSCvr67130

40(2a)35(2a)Skip task option in MBreplace and Software redeployscripts The scripts should have the option to skip tasksas user inputs and not by modifying the json file

CSCvp05204

40(2a)40(1a)Error noticed on SCVM Console print_req_error IOerror dev fd0 sector 0 on HyperV

It doesnt come up on SSH session

CSCvq34357

40(2a)30(1d)The VMNetwork switch is already created as part of theinstall process

Ideally the ip address assignment should be done duringthe initial install process as well No post install stepsshould be required altrnatively it could be done as partof a post install script

CSCvk36222

40(2a)25(1c)storfs service impacted during HX upgrade from HX21(1) to 25(1c)

HX node which is not being upgraded storfs servicestopped causing outage

CSCvg53223

40(2a)35(1b)HX Stretch Cluster Witness VM reverts to DHCP onreboot

CSCvp37536

40(2a)35(2b)HyperFlex Stretch Cluster Witness does not getprogrammed with an NTP server

CSCvp46578

Release Notes for Cisco HX Data Platform Release 4036

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2b)HyperFlex cluster does not heal after maintenance onnodeserver

CSCvr18528

40(2a)40(2a)High percentage of packet loss on the eth1 data interface9K MTU

CSCvr97089

40(2a)35(2f)The HyperFlex cluster healing state is stuck at 87 andis unable to progress

CSCvr16760

40(2a)40(2a)In some circumstances two replications can occur forthe same VM

From the GUI the replication status for this VM willshow stuck in an In Progress state but there are nojobs stuck in replication layer Subsequent Replicationsshould succeed

CSCvq22898

40(2a)35(2a)After an unexpected power outage the hyperflex clusteris unable to come up correctly

All processes are running time is synced and all nodescan vmkping each other as required

CSCvp13990

40(2a)40(1a)When specifying IP address instead of node ID stclinode maintenanceMode commands fail

CSCvp33657

40(2a)40(1a)2 node edge cluster if experiencing Intersightarbitratorconnectivity issues should NOT allowHXMaintenancemode to initiate as this will bring the cluster down

CSCvr31573

40(2a)35(2c)Old files in varsupportZKTxnlog are not purged withthe daily zklog-cleanup cron job

CSCvr89066

40(2a)35(2a)When a node is in MaintenanceMode any disk removalor replacement will be reflected in UI only after the nodeis brought back frommaintenancemode This is becausestorfs is not running on the node in maintenance andwill not be able to detect disk activities until it is broughtout of Maintenance Mode

CSCvo86431

40(2a)35(2c)Cluster might go offline and stop serving IO requestsrootcvm~ stcli cluster storage-summary--detailGet cluster storage summary failedjavanetConnectException Connection refused19216814210010207

CSCvr01645

40(2a)35(2d)A stretch cluster deployment fails with a messageFormatting nodes Node in Use on the Hyperflex installerand crmZoneType value is seen as 1 in storfscfg oroptspringpathconfigstretchtunes

CSCvq80340

Release Notes for Cisco HX Data Platform Release 4037

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)30(1c)During upgrade we upload the upgrade packagedownloaded from Ciscocom Upgrade packages are offile type tgz hxconnect acceptsstorfs-packages-401a-33028tar file a different type ofcompression and allows upgrade to start and fail whenit cannot decompress the file for bootstrap and scvmclientupgrade on ESXi

CSCvq63888

40(2a)40(2a)While attempting to install hyperflex via intersight thefollowing error is seen

Failed in Task Add hosts to vCenter Cluster with ErrorTry adding hosts manually to vCenter and retry Failedto get addressFailed to host by namelookupHostnamecompanycom on 000053 read udp12700145430-gt12700153 read connection refused

CSCvr52098

40(2a)35(2a)CoreAPI call may time out for cluster management APIwhile doing cluster create validation

CSCvo92952

40(2a)35(2d)Validation fails with Mixed mode expansion checkdue to Enhanced vMotion Compatibility Incompatibilitywhen the ESXi version is different

CSCvq95460

40(2a)30(1i)This failure is seen in installer when custom workflowis used hypervisor configuration + deploy (clean diskpartition=no) at Installing software packages on storagecontroller VM

CSCvr66309

40(2a)35(2d)The HX Installer apparently replaced a comma bysemicolon on the VLAN ID range while exporting theconfiguration to a JSON file

CSCvq93831

40(2a)40(1a)When using unsupported browsers HX Connect userswill not be able to login These users see an errorindicating unauthorized user

CSCvp22693

Release Notes for Cisco HX Data Platform Release 4038

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)40(1a)A vulnerability in the web-based management interfaceof Cisco HyperFlex Software could allow anunauthenticated remote attacker to conduct a cross-siterequest forgery (CSRF) attack on an affected system

The vulnerability is due to insufficient CSRF protectionsfor the web UI on an affected device An attacker couldexploit this vulnerability by persuading a user of theinterface to follow a malicious link A successful exploitcould allow the attacker to perform arbitrary actions withthe privilege level of the affected user

Cisco has released software updates that address thisvulnerability There are no workarounds that address thisvulnerability

This advisory is available at the following link

httpstoolsciscocomsecuritycentercontentCiscoSecurityAdvisorycisco-sa-20190807-hypflex-csrf

CSCvp24343

40(2a)35(2d)Add message to HX Connect Progress flow to notAcknowledge Pending Activities and reboot the serverson UCSM

HX Connect is doing a controlled rolling server upgradein the background

CSCvq22844

40(2a)40(1b)Compute Node expand operation fails due to incorrectcluster name

CSCvr43786

40(2a)26(1e)RF-2 cluster shutdown during data resynchronizationafter a node failure (non storfs) followed by multipledisk read failures and a HardBlacklist of a disk

CSCvp41241

40(2a)40(1a)HyperFlex Edge nodes do not properly set the ESXihostname during deployment

CSCvo13143

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4039

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4040

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1b) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401b-33133p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4041

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4042

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1b)40(1a)While running the HyperFlex installer with WindowsServer Hyper-V cluster creation process fails with theerror Failure occurred during ClusterCreation process Unable to post thecontent to the downstream This symptomis encountered while deploying HyperFlex cluster withHyperFlex nodes configuredordered with Cisco VIC1457 MLOM (PID HX-MLOM-C25Q-04) andWindows Server Datacenter or Core with Hyper-Vversion 2016 or 2019

CSCvp64140

Release Notes for Cisco HX Data Platform Release 4043

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)35(2b)Adding Micron 5200 drive to a cluster fails to increasecluster capacity

CSCvo69067

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4044

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4045

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1a) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401a-33028p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4046

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4047

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1a)30(1d)Cluster instability when disks of different sector sizeplaced in HX node

CSCvk17250

40(1a)

35(2c)

35(1a)When logged in using a local HX user account insteadof a Virtual Center account an error message appearsintermittently indicating Virtual Centerunreachable or Resource informationcannot be updated when VC is reachable

CSCvo36198

40(1a)

35(2b)

30(1d)HXDP does not work with EMC RecoverPoint - needsto support VMware API (FSS-Readdir)

CSCvk38003

Release Notes for Cisco HX Data Platform Release 4048

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)

35(2a)

30(1i)

25(1c)

25(1d)

Zookeeper (Exhibitor) process on HyperFlex StorageController VMs may fail to respond to requests whenvarzookeeper has low or no free space

CSCvm90352

40(1a)26(1c)Use Asynchronous consolidation for HX SnapshotsCSCvn02151

40(1a)35(2b)HX Quiesced Snapshot for Backup VendorsCSCvo90713

40(1a)35(2a)The cluster creationcluster expansion workflow stopswith the following error message at the validation step

FIRMWARE-Check UCSC-SAS-M5HD

FIRMWARE-Check UCSC-SAS-M5HD Required00000029000000320000003500000050 Found00000058

Action Needed Update the Controller Firmware toRequired Version

CSCvn17787

35(2b)30(1d)Kernel migration fails when a local datastore is searchedfor in the ESXi host

CSCvn73127

35(2a)26(1b)A node shuts down when two disks are replaced (acaching disk and another capacity disk) where thecapacity disk is inserted first and then the caching diskis inserted

CSCvk46364

Hyper-V

40(1a)35(2a)Cluster expansion may fail with an error code 500 -operation timed out

CSCvn28721

35(2b)35(2a)During upgrade remove the VLAN on team created foruser vSwitch During fresh installation only one VLANtag is set to the vSwitch and team although multipleVLANs were entered

CSCvn54300

35(2b)35(2a)While upgrading a Hyper-V cluster on account of a rarerace condition between the stUpgradeService andZookeeper servers the upgrade orchestration throws anupgrade validation error and the upgrade process isaborted

CSCvn60486

Release Notes for Cisco HX Data Platform Release 4049

Resolved Caveats in Release 40(1a)

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Startrestart EAM service invCenter

When EAM service is not runningvCenter registration is failing

CSCvt55712

35(2a)Contact Cisco TAC if you believeyou are seeing similar conditions

HyperFlex Stretch Cluster saw poorapplication performance due to writelatency cluster remained unhealthy andrebalance was stuck

This defect is currently being used forinvestigative purposes

CSCvt45344

35(2g)Remove log files fromvarlogspringpath folder when thelog files are not getting rotatedStop the relevant service removelarge log files fromvarlogspringpath folder andrestart the service

varstv folder in Controller VM maybecome full

CSCvt36374

40(2b)Restarting the stMgr will initializeit again

In a Cluster configured with DR networkStMgr may not get initialized and stuckin a deadlock while enabling the IPTablerules (can be verified from the stMgrlog)

CSCvt89709

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Click recover for the selectedVM again under the standalonemode This will move the VM torecovered state

When the user recovers a VM protectedin a group it along with other VMs in thegroup move to recovering state instandalonemode The selectedVM shouldmove to recovered state

CSCvs75553

35(2e)Clean up the fail over cluster andthen recreate the fail over clusterNo need to touch the HX storagecluster

Hyper-V When replicated DC was usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2d)40(1a)

Keep cluster utilization as low aspossible To avoid read-only statekeep below 92

HyperFlex clusters storage utilization isabove 92 usable capacity

CSCvr20922

Release Notes for Cisco HX Data Platform Release 4050

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1b)NAAll the disks in the node got locked afterrebooting the node

We successfully unlocked usingsed-clientsh -U command but wantedto test with another reboot and driveslocked again

CSCvs74286

35(2g)To recover the VM copy over thedata disks and attach them to thenew VM

After enable encryption HX connectencryption status showing Caution butall of the disks enable encryptionsuccessful

-Self Encrypting Drives Service isrunning on all of the nodes

-There is no error message of encryption

- USB0 interface is up

-All SED disks showing supported 1enabled 1 locked 0

CSCvs93245

40(1b)Enable kernelpanic_on_oops as apersistent configuration This willcause the node to panic and rebootimmediately

A cluster node running HX release40(1b) may hang in the Linux kernelThis is classified as an oops and adeviation from the expected behavior

CSCvs54285

Release Notes for Cisco HX Data Platform Release 4051

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Enabling K8iscsi stack in Hx clusterCSCvs41324

Release Notes for Cisco HX Data Platform Release 4052

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Steps to enable K8 on HX cluster

Perform following steps on ALLthe controller VMs

1 Updateetcinitscvmclientconf toenable tune iscsiEnable totrue

sed -iesiscsiEnable=falseiscsiEnable=trueetcinitscvmclientconf

2 Run following initctlcommand to reloadconfiguration

initctl reload-configuration

3 Restart scvmclient process

stop scvmclient startscvmclient

4 Run following initctlcommand to mount SYSTEMDS

initctl emit --no-waitsystem-datastore-created

5 Verify that iscsiEnable tuneis set to true using thefollowing command

ps -eaf | grep scvmclient |grep -v grep

root 6241 1 1 Dec06 014007optspringpathstorfs-corescvmclient-T logEnabled=false -TlogSyslogEnabled=true -TlogEchoToScreen=false -TstatLoggingToFile=false -TstatLoggingToSyslog=true-TlogDir=varlogspringpath-TnfsBackendServerList=1010748100-T iscsiEnable=true -TiscsiUseAsync=true -T

Release Notes for Cisco HX Data Platform Release 4053

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

iscsiConfigFilePath=nfsSYSTEMistgtconfps -eaf | grep scvmclient |grep -v grep

35(2e)Following procedure can be usedto check NFS Queue Depth andincrease if needed

rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth256 lt- Low value[rootHXESXI1] vsish -e setvmkModulesnfsclientmnt[DATASTORE]propertiesmaxQDepth 1024[rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth1024 lt- Optimalvalue

This requires an ESXi host rebootto take effect

Please place the node in HyperflexMaintenanceMode and gracefullyreboot the node for the changes tobe applied

HyperFlex Datastore NFS Queue Depthshows as 256 which can lead toperformance (including latency) issues

CSCvr83056

40(1b)This is cosmetic The error will goaway once the cluster returns tohealthy status

When a single node is offline in clusterstcli cluster storage-summary shows twonodes unavailable rootSCVM~ stclicluster storage-summary

CSCvs47735

35(2g)Modify both the root and adminpassword for the SCVMs to be thesame

stcli security password set -uadmin

Cluster expansion fails at Config Installerstage when the root and admin passwordfor the storage controller (SCVM) aredifferent

CSCvs69317

35(2d)NAAfter successfully changing the DNSserver on the HX controller we still cansee the original DNS entry that was addedduring the deployment

CSCvs69154

Release Notes for Cisco HX Data Platform Release 4054

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Customers are strongly encouragedto work with TAC in order toidentify and remediate this issue

1 Change getcluster from 2m to8m inoptspringpathstorfs-mgmtstMgr-10confapplicationconfon all nodes

2 Restart stMgr on all nodes

You may see this error message after afailed upgrade or other task such asattempting to enter a node into HXmaintenance mode

CSCvs53555

40(2a)Run the following command oneach storage controller VM as root

grep -i monitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp sed -iesmonitor_dns_serverstruemonitor_dns_serversfalseoptspringpathhx-diag-toolswatchdog_configjsonampamp grep -imonitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp restart watchdog

Receive the following alertevent in HXConnect HX Controller VMHOSTNAME one or more configuredDNS servers not responding

CSCvt13947

35(2b)Delete any empty (Size 0) log filesunder varlogzookeeperversion-2ORvarlogzookeeperstandaloneversion-2

Zookeeper fails to start while Exhibitor isrunning however echo srvr returnsnothing

CSCvs21562

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

40(4e)1 De-commission the server

2 Power drain the server -REMOVE BOTH POWERCORDS ON THE BACK OFTHE SERVER FOR 10SECONDS THENREINSERTPOWERCORDS

3 Re-commission the server

M2 boot disk is missing from serverinventory after upgrade to server firmware40(4e) As a result server fails to boot toOS installed in the M2 disk The issuepersists after re-acknowledgement as wellas de-commission andre-acknowledgement of the server

CSCvs02466

Release Notes for Cisco HX Data Platform Release 4055

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Re-start the flusher manuallyMany disks taken offline by a fault ormanually resulting in ENOSPC oncluster

CSCvq32721

35(2e)Clean up fail over cluster and thenrecreate the fail over cluster Thereis no need to touch the HX storagecluster

[Hyper-V]When replicated DCwas usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2b)Manually install upgradescvmclient VIB andmake sure thatit matches with HXDP version

Upgrade from release HX 35(2a) to35(2x) 4X does not upgrade scvmclientRecovery point may not work

CSCvq54992

Management

30(1b)To recover the VM copy over thedata disks and attach them to thenew VM

[HyperV]VMshows up onmultiple nodesCSCvj22992

35(1a)1 Do not change MTU at ESXlevel on running cluster

2 Reset back to the original value

ChangingMTU from 9000 to 1500 on oneESX node causes storfs process to restarton all nodes

CSCvm99150

Release Notes for Cisco HX Data Platform Release 4056

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)After the installation (for newdeployments) or after an upgradeto 401a (for existingdeployments) perform thefollowing steps on all ControllerVMs

1 Edit the following tune files (onall Controller VMs)

optspringpathconfiglfftunes

optspringpathconfigvsi_16tbtunes

2 SetcleanerEnableSegSummaryCleaning

to false

3 After editing the above tunefiles

a SSH in to all the ControllerVMs

b Run storfstool -- -Z

c Run the below command andcheck the values Tune valueshould be true

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=true

d Type below commands andapplied the tunes changesdynamically

echo false gttmpstprocfssystemtunecleanerEnableSegSummaryCleaning

e Verify that tune values arechanged Run below commandValue should be false

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=false

f umount tmpstprocfs

A cluster with 8TB or 12TB disk drivesmay experience IO stalls for severalminutes after another node fails in thecluster

CSCvp23718

Release Notes for Cisco HX Data Platform Release 4057

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)1 Update the conf file manuallyto change to the link local addressto

2 Restart scvmclient on thecontroller and ESX

HX 35(2b) FlexVol to HX 40 CSIupgrade does not work FlexVol continuesto work

CSCvp26319

35(2a)Disable rebalance until node healsthen reenable rebalance

7-node cluster healing very slowlybecause rebalance change AMS reqs areblocking healing process

CSCvp62512

35(2a)NAWith moderate cluster storage capacityusage under site failover cluster clustermay experience an all paths (APD) downcondition

CSCvq49412

40(1a)NAThis product includes Third-partySoftware that is affected by thevulnerabilities identified by the followingCommon Vulnerability and Exposures(CVE) IDs

CVE-2016-0762 CVE-2016-6797CVE-2016-6816 CVE-2016-8735CVE-2017-5647 CVE-2017-12615CVE-2017-12616 CVE-2017-12617CVE-2017-7674 CVE-2018-1304CVE-2018-8014 CVE-2018-1336CVE-2018-8034 CVE-2018-11784CVE-2019-0232

CSCvp36364

35(2d)NAstcli cluster info command needs toprovide UCSM VIP address Currently itshows ucsm-hostcom

CSCvq11456

35(2a)No workarounds currently Thestale entries do not affect the dayto day operations of HyperFlex

HyperFlex upgrade validation failedbecause cluster has extraneous stNodes ininternal database

CSCvq32530

35(2a)NAWhen not in maintenancemode all drivesappear in HX Connect and get updated asexpected

When in maintenance mode extra drivesare shown in HX Connect with no slotnumbers

CSCvq42378

35(2b)Contact TAC to cleanup duplicatenode

Nodes duplicated in HX Connect aftercluster recreate

CSCvq61825

Release Notes for Cisco HX Data Platform Release 4058

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)NACurrently HyperFlex installs on Hyper-Vdo not contain the stcli security whitelistcommandset

CSCvq66245

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

Release Notes for Cisco HX Data Platform Release 4059

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)[Hyper-V] Converged node expansionfails for incoming node with VIC 14xx

CSCvq39523

Release Notes for Cisco HX Data Platform Release 4060

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

1 Set Jumbo frame inWindowsHypervisor for incomingnode

Get-NetAdapter storage-data-a|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

Get-NetAdapter storage-data-b|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

2 Verify all interfaces are setcorrectly to support Jumbo frames

Get-NetAdapter storage |Get-NetAdapterAdvancedProperty| RegistryKeyword -matchjumbo | ft -auto

You should receive the followingmessage

Name DisplayName DisplayValue

RegistryKeyword RegistryValue

---- ----------- ------------

--------------- -------------

vswitch-hx-storage-data Jumbo

Packet 9014 Bytes

JumboPacket 9014

storage-data-b Jumbo Packet

Bytes 9014 JumboPacket

9014

storage-data-a Jumbo Packet

Bytes 9014 JumboPacket

9014

3 Reboot only the incoming nodeor the node being expanded

4 Retry Cluster Expansion in

Release Notes for Cisco HX Data Platform Release 4061

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

bull M5 For NEW hybrid or All Flash (Cisco HyperFlex HX240c M5 or HX220c M5) deploymentsverify that the recommended UCS firmware version is installed

For SED-based HyperFlex systems ensure that the A (Infrastructure) B (Bladeserver) and C (Rack server) bundles are at Cisco UCS Manager version 40(2b)or later for all SED M4M5 systems For more details see CSCvh04307 ForSED-based HyperFlex systems also ensure that all clusters are at HyperFlexRelease 35(2b) or later For more information see Field Notice (70234) andCSCvk17250

Important

bull To reinstall an HX server download supported and compatible versions of the software See theCisco HyperFlex Systems Installation Guide for VMware ESXi for the requirements and steps

bull Important For Intersight edge servers running older than 40(1a) CIMC version HUU is the suggestedmechanism to update the firmware

Table 1 HyperFlex Software Versions for M4M5 Servers

M5 Recommended FIServer Firmware

(be sure to review important notes above)

M4 Recommended FIServer Firmware

(be sure to review important notes above)

HyperFlexRelease

40(4h)40(4h)40(2b)

40(4h)40(4h)40(2a)

40(4h)40(4h)40(1b)

40(4e)40(4e)40(1a)

HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments

Cisco HX Data Platform Release 4x based Deployments

Confirm the component firmware on the server meets the minimum versions listed in the following tables

HyperFlex Edge does not support Cisco IMC versions 40(4a) 40(4b) 40(4c) 40(4d) and 40(4e)Important

Table 2 HX220c M4 HXAF220c M4 Cluster

Recommended Firmware Version - HXDP 4x

(be sure to review important note(s) above)

Component

402d0BIOS

40(2h)Cisco Integrated Management Controller(CIMC)

Release Notes for Cisco HX Data Platform Release 4010

HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments

Recommended Firmware Version - HXDP 4x

(be sure to review important note(s) above)

Component

13000012SAS HBA

43(2b)VIC 1227

0103 (Toshiba)

CS07 (Intel 480)

SSD

CN03 (Seagate)

CK03 (Seagate 18)

CK03 (Seagate 24)

sF3Q (Samsung)

3F3Q (Samsung)

HDD

40(2h)

Download Software

Host Upgrade Utility (HUU) Download Link

Table 3 HX220c M5 HXAF220c M5 Cluster

Recommended Firmware Version - HXDP 4x

(be sure to review important notes above)

Component

C220M5404p00224200755BIOS

40(4i)Cisco Integrated Management Controller(CIMC)

09000008SAS HBA

43(3c)VIC 1387

50(3e)VIC 1457

G201CS01 (Intel)

0104 (Toshiba)

0107 (Toshiba)

SSD

2F3Q (Samsung)

3F3Q (Samsung)

NOA4 (Seagate)

CK03 (Seagate 18tb)

CK03 (Seagate 24)

HDD

Release Notes for Cisco HX Data Platform Release 4011

HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments

Recommended Firmware Version - HXDP 4x

(be sure to review important notes above)

Component

40(4k)

Download Software

Host Upgrade Utility (HUU) Download Link

HyperFlex Licensing

As of version 26(1a) HyperFlex supports VMware PAC licensing Existing VMware embedded licenses willcontinue to be supported

As of version 25(1a) HyperFlex uses a smart licensing mechanism to apply your licenses See the CiscoHyperFlex Systems Installation Guide for VMware ESXi for details and steps

VMware vSphere Licensing Requirements

How you purchase your vSphere license determines how your license applies to your HyperFlex system

bull If you purchased your vSphere license with HyperFlex

Each HyperFlex server either has the Enterprise or Enterprise Plus edition preinstalled at the factory

bull HX Nodes have OEM licenses preinstalled If you delete or overwrite thecontent of the boot drives after receiving the HX servers you also delete thefactory-installed licenses

bull OEM license keys is a newVMware vCenter 60 U1b feature Earlier versionsdo not support OEM licenses

bull All factory-installed HX nodes share the same OEM license key WithvSphere OEM keys the Usage count can exceed the Capacity value

bull When you add an HX host to vCenter through the Add Host wizard in theAssign license section select the OEM license

We obfuscate the actual vSphere OEM license key for example0N085-XXXXX-XXXXX-XXXXX-10LHH

bull Standard Essentials Plus and ROBO editions are not available preinstalledon HX servers

Note

bull If you did NOT purchase your vSphere license with HyperFlex

The HX nodes have a vSphere Foundation license preinstalled After initial setup you can apply thelicense to a supported version of vSphere

bull If you purchased a vSphere PAC license

Follow the instructions in your PAC license letter from VMware to add the license to your MY VMwareaccount then follow the instructions to add your HX host to vCenter and assign the PAC license

Release Notes for Cisco HX Data Platform Release 4012

HyperFlex Licensing

HX Data Platform Software Versions for HyperFlex Witness Node

Witness Node VersionHyperFlex Release

10840(2b)

10840(2a)

10440(1b)

10440(1a)

Software Requirements for VMware ESXi

The software requirements include verification that you are using compatible versions of Cisco HyperFlexSystems (HX) components and VMware vSphere VMware vCenter and VMware ESXi

bull Verify that all HX servers have a compatible version of vSphere preinstalled

bull Verify that the vCenter version is the same or later than the ESXi version

bull Verify that the vCenter and ESXi versions are compatible by consulting the VMware ProductInteroperability Matrix Newer vCenter versions may be used with older ESXi versions so long as bothESXi and vCenter are supported in the table below

bull Verify that you have a vCenter administrator account with root-level privileges and the associatedpassword

For VIC1457 there is no support for ESXi 60Note

The below table applies for all of the following VMware vSphere Editions Enterprise Enterprise PlusStandard Essentials Plus ROBO

VMware vCenter VersionsVMware ESXi VersionsHyperFlex Version

60 U3 65 U3 67 U360 U3 65 U3 67 U340(2b)1

60 U3 65 U3 67 U360 U3 65 U3 67 U340(2a)2

60 U3 65 U3 67 U260 U3 65 U3 67 U2340(1b)

60 U3 65 U2 67 U260 U3 65 U2 67 U2440(1a)

1 Cisco HyperFlex Release 40(2) is the last major HyperFlex release that will support vSphere 60 (ESXiand vCenter) due to those versions reaching end of VMware general support on March 12 2020

2 Cisco HyperFlex Release 40(2) is the last major HyperFlex release that will support vSphere 60 (ESXiand vCenter) due to those versions reaching end of VMware general support on March 12 2020

3 Use of 67 U2 is not recommended see software advisory for CSCvq06952 and CSCvp88515 for furtherdetails

4 Use of 67 U2 is not recommended see software advisory for CSCvq06952 and CSCvp88515 for furtherdetails

Release Notes for Cisco HX Data Platform Release 4013

HX Data Platform Software Versions for HyperFlex Witness Node

Software Requirements for Microsoft Hyper-V

The software requirements include verification that you are using compatible versions of Cisco HyperFlexSystems (HX) components and Microsoft Hyper-V (Hyper-V) components

HyperFlex Software versions

The HX componentsmdashCisco HX Data Platform Installer Cisco HX Data Platform and Cisco UCSfirmwaremdashare installed on different servers Verify that each component on each server used with and withinthe HX Storage Cluster are compatible

bull Cisco HyperFlex M5 Converged nodesmdash For Hybrid (Cisco HyperFlex HX240c M5 HX220c M5)and All Flash (Cisco HyperFlex HXAF240cM5 HXAF220cM5) verify that Cisco UCSManager 40(2b)is installed HX 40 (1a) does not support Hyper-V on the All NVMe (HXAF220C-M5SN) nodes Fordetailed information on installation requirements and steps see theCisco HyperFlex Systems InstallationGuide on Microsoft Hyper-V

Table 4 Supported HyperFlex Software versions for M5 Servers on Hyper-V

M5 Recommended Server FirmwareHyperFlex Release

40(4h)40(2b)

40(4g)40(2a)

40(4e)40(1b)

40(4e)40(1a)

Release Notes for Cisco HX Data Platform Release 4014

Software Requirements for Microsoft Hyper-V

Table 5 Supported Microsoft Software versions

VersionMicrosoft Component

Windows Server 2016 Datacenter Core amp DesktopExperience

ForWindows Server 2016Datacenter Coreand Desktop Experience the Windows2016 ISO image should be Update BuildRevision (UBR) 1884 at a minimum

Note

Windows Server 2019Datacenter-DesktopExperienceis supported starting from HXDP 401(a) onwards

For Windows Server 2019 DesktopExperience theWindows 2019 ISO imageshould be Update Build Revision (UBR)107 at a minimum

Note

Windows Server 2019 DatacenterndashCore is notsupported currently

Also note that the following are currently notsupported

OEM activated ISOs and Retail ISOs are notsupported

Earlier versions ofWindows Server such asWindows2012r2 are not supported

Non-English versions of the ISO are not supported

Windows Operating System (Windows OS)

A Windows 2012 or later domain and forestfunctionality level

Active Directory

Supported Microsoft License Editions

The Microsoft Windows Server version that is installed on one or more HyperFlex hosts must be licensed asper Microsoft licensing requirements listed on Microsoft Licensing

Browser Recommendations

Use one of the following browsers to run the listed HyperFlex components These browsers have been testedand approved Other browsers might work but full functionality has not been tested and confirmed

Table 6 Supported Browsers

HX ConnectHX Data PlatformInstaller

Cisco UCS ManagerCisco IntersightBrowser

11 or higher11 or higher11 or higherNAMicrosoft InternetExplorer

70 or higher70 or higher57 or higher62 or higherGoogle Chrome

Release Notes for Cisco HX Data Platform Release 4015

Browser Recommendations

HX ConnectHX Data PlatformInstaller

Cisco UCS ManagerCisco IntersightBrowser

60 or higher60 or higher45 or higher57 or higherMozilla Firefox

NANA9 or higher10 or higherApple Safari

NANA35 or higherNAOpera

Notes

bull Cisco HyperFlex Connect

The minimum recommended resolution is 1024 X 768

bull Cisco HX Data Platform Plug-in

The Cisco HX Data Platform Plug-in runs in vSphere For VMware Host Client System browserrequirements see the VMware documentation at httpswwwvmwarecomsupportpubs

The HX Data Platform Plug-in is not displayed in the vCenter HTML client You must use the vCenterflash client

bull Cisco UCS Manager

The browser must support the following

bull Java Runtime Environment 16 or later

bull Adobe Flash Player 10 or higher is required for some features

For the latest browser information about Cisco UCSManager refer to themost recent Cisco UCSManagerGetting Started Guide

Cisco HX Data Platform Storage Cluster SpecificationsCluster Limits

bull Cisco HX Data Platform supports up to 100 clusters managed per vCenter as per VMware configurationmaximums

bull Cisco HX Data Platform supports any number of clusters on a single FI domain Each HX convergednode must be directly connected to a dedicated FI port on fabric A and fabric B without the use of a FEXC-series compute only nodes must also directly connect to both FIs B-series compute only nodes willconnect through a chassis IO module to both fabrics In the end the number of physical ports on the FIwill dictate the maximum cluster size and maximum number of individual clusters supported in a UCSdomain

The following table provides Cisco HX Data Platform Storage Cluster Specifications

Release Notes for Cisco HX Data Platform Release 4016

Cisco HX Data Platform Storage Cluster Specifications

Stretched Cluster (Availableon ESX Only)

Microsoft Hyper-VVMware ESXiNode

AllNVMe -HXAF220c-M5SN

HX240c-M5LHX220cM5

HX220cAF M5

HX240cM5

HX240cAF M5

HX240c-M5LHX220cM5

HX220cAF M5

HX240cM5

HX240cAF M5

HXAF220c-M5SN

AllNVMe -HXAF220c-M5SN

ESXi 65or 67only NotsupportedwithESXi 60orHyper-V

HX220cM5 Edge

HXAF220cM5 Edge

HX220cM4 Edge

HXAF220cM4 Edge

HX240c-M5LHX220cM5

HX220cAF M5

HX240cM5

HX240cAF M5

HX220cM4

HX220cAF M4

HX240cM4

HX240cAF M4

HXServers

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

C220M5C240M5B200M4B200M5

C240M5C220M5B200M4B200M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

mdashB200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

Compute-OnlyUCSB-SeriesC-SeriesServers

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

Convergednodesonly

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

SupportedNodes

Release Notes for Cisco HX Data Platform Release 4017

Cisco HX Data Platform Storage Cluster Specifications

Stretched Cluster (Availableon ESX Only)

Microsoft Hyper-VVMware ESXiNode

NA(requiresEnterpriseHXDP-PLicense)

NA(requiresEnterpriseHXDP-PLicense)

NA(requiresEnterpriseHXDP-PLicense)

Convergednodes3-16(12TBHDDoption isnotsupportedforHyperV)

Compute-onlynodes0-16

Convergednodes3-16

Compute-onlynodes0-16

NA(requiresEnterpriseHXDP-PLicense)

M4Convergednodes 3

M5Convergednodes23or 4

RequiresHXDP-ELicense

Convergednodes3-16(12TBdrive islimited toamaximumof 8nodes)

Compute-onlynodes0-16(12TBdrive islimited toamaximumof 8nodes)

Convergednodes3-32(76TBdriveconfigsarelimited tomaximumof 12drivesnodeon AF240)

Compute-onlynodes0-32

HXDP-SLicensedNodeLimits

11 ratioofHXDP-StoComputeonlynodes

(MinmdashMax)

Convergednodes2-16 perSite

Compute-onlynodes0-21 perSite

(up tomaxclustersize)

Convergednodes2-8 perSite

Compute-onlynodes0-16 perSite

(up tomaxclustersize)

Convergednodes2-16 perSite

Compute-onlynodes0-21 perSite

(up tomaxclustersize)

Convergednodes3-16(12TBHDDoption isnotsupportedforHyperV)

Compute-onlynodes0-16

Convergednodes3-16

Compute-onlynodes0-16

Convergednodes3-32

ComputeOnlynodes0-32

(up tomaxclustersize)

Convergednodes 3

(requiresHXDP-ELicense)

Convergednodes3-16(12TBdrive islimited toamaximumof 8nodes)

Compute-onlynodes0-32(12TBdrive islimited toamaximumof 16nodes)

Convergednodes3-32(76TBdriveconfigsarelimited tomaximumof 12drivesnodeon AF240)

Compute-onlynodes0-32

(up tomaxclustersize)

HXDP-PLicensedNodeLimits

12 ratioofHXDP-PtoComputeonlynodes

(MinmdashMax)

24 perSite 64percluster

24 perSite 48percluster

32 perSite 64percluster

32326434864MaxClusterSize

Release Notes for Cisco HX Data Platform Release 4018

Cisco HX Data Platform Storage Cluster Specifications

Stretched Cluster (Availableon ESX Only)

Microsoft Hyper-VVMware ESXiNode

212121111121mdash2121MaxComputetoConvergedratio

NoExpansion

Requires Enterprise license

Requires uniform expansion across both sites

Guidelines and Limitationsbull Starting with release 40(2a) SCVM is no longer needed on a Compute node

bull HX REST API Access Token Management ndash Applications leveraging HX REST APIs should re-useaccess tokens when making API calls Once obtained using the AAA Obtain Access Token API accesstokens are valid for 18 days (1555200 seconds) In addition AAA enforces rate limiting on ObtainAccess Token API requests in a 15 minute window auth can be invoked (successfully) a maximum of5 times A user is allowed to create a maximum of 8 unrevoked tokens Subsequent call to auth willautomatically revoke the oldest issued token tomake room for the new token Amaximum of 16 unrevokedtokens can be present in system In order to prevent brute-force attacks after 10 consecutive failedauthentication attempts a user account is locked for a period of 120 secondsFor more information seeCisco HyperFlex Systems REST API Reference guide

HxConnect makes use of AAA Authentication REST API for login and the above rate limit applies toHxConnect also

Upgrade Guidelines

The following list is a highlight of critical criteria for performing an upgrade of your HyperFlex system

bull Unsupported HX Data Platform 17x 18x 20 21x 25x and 26x clustersmdashUsers from anyversion prior to 26(1a) must step through an intermediate version before upgrading to 40 or later releasesIf you need to upgrade your environment from a Cisco HyperFlex HX Data Platform software releasethat is past the last date of support to the latest suggested release on the Cisco Software Download sitesee Cisco HyperFlex Systems Upgrade Guide for Unsupported Cisco HXReleases For more informationsee the Software Advisory for CSCvq66867 WARNING Only Use HXDP 26(1e) Upgrade PackageWhen Upgrading From HXDP 18(1a)-18(1e)

bull Hypercheck Health Check UtilitymdashCisco recommends running this proactive health check utility onyour HyperFlex cluster prior to upgrade These checks provide early visibility into any areas that mayneed attention andwill help ensure a seamless upgrade experience For more information see the HyperFlexHealth amp Pre-Upgrade Check Tool TechNote for full instructions on how to install and run Hypercheck

bull vSphere 67 Software AdvisorymdashDo not upgrade to Cisco HX Data Platform Release 40(1a) whenrunning ESXi 67U1 EP06 (build 11675023) Do not upgrade to 67U1 EP06 (build 11675023) ifrunning Cisco HX Data Platform Release 40(1a) See the Software Advisory CSCvo56350 for furtherdetails

Release Notes for Cisco HX Data Platform Release 4019

Guidelines and Limitations

The software build version posted at release will override any other local versions

bull Required vCenter upgrademdashFor enhanced security Cisco HX Data Platform Release 35(1a) or laterrequires the use of TLS 12 Therefore vCenter must be upgraded to 60 U3f or later before upgradingto Cisco HX Data Platform Release 35 or later In addition ESXi should be upgraded as required tomeet HX Data Platform compatibility requirements

bull Minimum HXDP version for upgrademdashHXData Platform clusters running 26(1a) or later may upgradedirectly to 40 using the HX Connect UI

bull Cluster ReadinessmdashEnsure that the cluster is properly bootstrapped and the updated plug-in is loadedbefore proceeding Manual cluster bootstrap is required for upgrade from a pre-35 release

bull Cluster ReadinessmdashEnsure that the cluster is properly bootstrapped and the updated plug-in is loadedbefore proceeding Manual cluster bootstrap is required for HX releases earlier than 35(1a) For moreinformation see theManual Bootstrap Upgrade Process in the Cisco HyperFlex Systems Upgrade Guidefor VMware ESXi Release 40 Do not skip this cluster bootstrap step it is required for all upgradesuntil HX Release 35(1a) Auto bootstrap is supported beginning with HX release 35(1a) For moreinformation see the Auto Bootstrap Upgrade Process from HX Connect UI in the Cisco HyperFlexSystems Upgrade Guide for VMware ESXi Release 40

Manual bootstrap is not supported on Intersight clusters

bull Initiating Upgrade―Use the HX Connect UI or CLI stcli commands when upgrading from 25(1a)or later releases Use either the CLI stcli commands or the HX Data Platform Plug-in to the vSphereWeb Client when upgrading from a pre-25(1a) release The vCenter plug-in should not be used forupgrades starting with the 25(1a) release The vCenter plug-in should not be used for upgrades startingwith the 25(1a) release

If the current cluster version is at 35(1a) or above you do not need to use the stcli command Directupgrade to 40 is possible

bull Complete your Upgrade―The self-healing (or rebalance) capability is disabled temporarily during theupgrade window If the upgrade fails you should complete the upgrade as soon as possible

bull ESXi and HXDP Compatibility―Ensure your cluster is running a compatible version of ESXi basedon the running the HXData Platform version (see the section Software Requirements for VMware ESXi)ESXi compatibility is determined by the major version and update release of ESXi It is generally bestto upgrade HXDP and ESXi together if combining the upgrade operations into a single optimized rebootWhen running a split upgrade first upgrade the HX Data Platform then proceed to upgrade ESXi

bull If Upgrading to vSphere 65

bull Certain cluster functions such as native and scheduled snapshotsReadyClones and Enter or Exit HX Maintenance Mode will not operatefrom the time the upgrade is started until the HX Data Platform upgrade iscomplete

bull After upgrading ESXi using the offline zip bundle use the ESX ExitMaintenance Mode option The HX Exit Maintenance Mode option doesnot operate in the vSphere Web Client until the HX Data Platform upgradeis complete

Note

Release Notes for Cisco HX Data Platform Release 4020

Guidelines and Limitations

bull vSphere 60 UpgradesmdashUsers on vSphere 60 migrating to 65 upgrade components in the followingorder

1 Upgrade HX Data Platform and UCS firmware

2 Upgrade HX Data Platform and ESXi

3 Upgrade HX Data Platform only first then upgrade ESXi or UCS firmware or both

bull M4 Server Firmware UpgradesmdashUpgrade server firmware to ensure smooth operation and to correctknown issues Specifically newer SAS HBA firmware is available in this release and is recommendedfor long-term stability

bull Users are encouraged to upgrade to 31(3c) C-bundle or later whenever possible

bull Users running C-bundle versions before 31(2f) must upgrade server firmware by performing acombined upgrade of UCS server firmware (C-bundle) to 31(3c) or later and HX Data Platform to25 Do not split the upgrade into two separate operations

bull If the cluster is already on 31(2f) C-bundle or later you may perform an HX Data Platform onlyor combined upgrade as required

bull M5 Server Firmware UpgradesmdashM5 generation servers must run firmware version 32(2d) or later

bull Firmware Downgrades mdashDowngrading UCSM from the HX-installer is not supported

bull M4M5 Mixed DomainsmdashA mixed domain occurs when a new separate M5 cluster is installed underthe same UCS domain that contains existing M4 clusters Under these conditions orchestrated UCSserver firmware upgrade will not operate until Cisco HX Data Platform Release 26 or later is installedon the M4 clusters Therefore it is best practice to first upgrade UCS server firmware to the latest 31(3)or 32(2) patch release before adding a new M5 cluster to the existing UCS domain Additionally any17 HX Data Platform clusters must first be upgraded before adding any new M5 clusters to the samedomain

bull Maintenance WindowmdashIf upgrading both HX Data Platform and UCS firmware you can select eithera combined or split upgrade through the vSphere HX Data Platform Plug-in depending on the length ofthe maintenance window Cisco UCSManager infrastructure upgrade is only supported using AutoInstalland the direct server firmware upgrade should be performed only through the upgrade orchestrationframework provided by the HX Data Platform Plug-in

bull Unsupported Self-Encrypting Drives (SEDs)mdashIf adding or replacing self-encrypting drives (SEDs)that have been recently qualified in newer versions of HX Data Platform insert the new drives only afterupgrading HX Data Platform to a compatible version All drives must be SED drives mixing SED andnon-SED is not supported

bullbull Enabling External Host AccessmdashWith Cisco HX Data Platform Release 40(1a) port 445 on themanagement network is blocked for enhanced security Note that prior to 40 port 445 port was openenabling external host access If you are upgrading to 40(1a) from a prior release and would like tocontinue external host access you can use a utility to open select hosts For more information aboutenabling external host access see the ConfiguringHyperFlex Share to SCVMM section in the InstallationGuide for Microsoft Hyper-V

Release Notes for Cisco HX Data Platform Release 4021

Guidelines and Limitations

Mixed Cluster Expansion Guidelinesbull Expanding existing M4 cluster with M5 converged nodes is supported

bull Expanding existing M5 cluster with M4 converged nodes is not supported

bull Expanding existing mixed M4M5 cluster with M4 or M5 converged nodes is supported

bull Adding any supported compute-only nodes is permitted with all M4 M5 and mixed M4M5 clustersusing the HX Data Platform 26 or later Installer Some example combinations are listed here manyother combinations are possibleExample combinationsExpand mixed M4M5 cluster with compute-only B200 C220 C240 M4M5Expand M4 cluster with compute-only B200 M5 C220 M5 C240M5

bull Only expansion workflow is supported to create a mixed cluster Initial cluster creation with mixedM4M5 servers is not supported

bull All M5 servers must match the form factor (220240) type (HybridAF) security capability (Non-SEDonly) amp disk configuration (QTY capacity and non-SED) of the existing M4 servers

bull HX220-M5 will use a maximum of 6 capacity disks (2 disk slots to remain empty) when mixedwith HX220-M4

bull HX Edge SED LFF Hyper-V and Stretched Clusters do not support mixed M4 and M5 clusters

Security FixesThe following security issues are resolved

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-13132 CVE-2019-9924CSCvq6313840(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-11719 CVE-2019-11727CVE-2019-11729

CSCvq7124040(2a)

Release Notes for Cisco HX Data Platform Release 4022

Mixed Cluster Expansion Guidelines

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-1125CSCvr0633940(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2015-9383 CVE-2018-14498CVE-2018-20406 CVE-2018-20852CVE-2019-10160 CVE-2019-13117CVE-2019-13118 CVE-2019-14287CVE-2019-14973 CVE-2019-15903CVE-2019-17546 CVE-2019-18197CVE-2019-18218 CVE-2019-5010CVE-2019-5094 CVE-2019-5481CVE-2019-5482 CVE-2019-9636CVE-2019-9740 CVE-2019-9947CVE-2019-9948

CSCvs0609440(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2017-13168 CVE-2017-18174CVE-2017-18216 CVE-2018-10876CVE-2018-10877 CVE-2018-10878CVE-2018-10879 CVE-2018-10880CVE-2018-10881 CVE-2018-10882CVE-2018-10902 CVE-2018-10938CVE-2018-12233 CVE-2018-12896CVE-2018-13053 CVE-2018-13094CVE-2018-13096 CVE-2018-13405CVE-2018-13406 CVE-2018-14609CVE-2018-14617 CVE-2018-14633CVE-2018-14734 CVE-2018-15572CVE-2018-15594 CVE-2018-16276CVE-2018-16658 CVE-2018-17182CVE-2018-17972 CVE-2018-18021CVE-2018-18690 CVE-2018-18710CVE-2018-6554 CVE-2018-6555CVE-2018-9363

CSCvp6501940(2a)

This vulnerability is due to insufficientHTML iframe protection An attackercould exploit this vulnerability bydirecting a user to an attacker-controlledweb page that contains amaliciousHTMLiframe A successful exploit could allowthe attacker to conduct clickjacking orother clientside browser attacks

NACSCvo9851640(2a)

Release Notes for Cisco HX Data Platform Release 4023

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerability is due to insufficientauthentication for the statistics collectionservice An attacker could exploit thisvulnerability by sending properlyformatted data values to the statisticscollection service of an affected deviceA successful exploit could allow theattacker to cause the web interfacestatistics view to present invalid data tousers

NACSCvj9558440(2a)

The vulnerability is due to insufficientCSRF protections for the web UI on anaffected device An attacker could exploitthis vulnerability by persuading a user ofthe interface to follow a malicious linkA successful exploit could allow theattacker to perform arbitrary actions withthe privilege level of the affected user

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvp2434340(2a)

Vulnerabilities with a version of cJSONidentified by CVE-2019-11834 andCVE-2019-11835

For more information see the relatedCisco Security Advisory

CVE-2019-11834 CVE-2019-11835CSCvq1994940(2a)

HX ESXi image patches for MCEPSCand TAA vulnerabilities with VMwareESXi identified by CVE-2018-12207 andCVE-20190-11135

For more information see the relatedCisco Security Advisory

CVE-2018-12207 CVE-2019-11135CSCvr5439840(2a)

CP networking vulnerabilities affectingthe Linux kernel identified byCVE-2019-11477 CVE-2019-11478CVE-2019-11479

For more information see the relatedCisco Security Advisory

CVE-2019-11477 CVE-2019-11478CVE-2019-11479

CSCvq1954640(2a)

Qualification of Microsoft securitypatches for vulnerabilities withMicrosoftHyper-V hypervisor identified withCVE-2018-12207 CVE-2019-11135

For more information see the relatedCisco Security Advisory

CVE-2018-12207 CVE-2019-11135CSCvr5439940(2a)

Release Notes for Cisco HX Data Platform Release 4024

Security Fixes

DescriptionCVEDefect IDRelease

OpenSSH vulnerabilities identified byCVE-2019-6111

For more information see the relatedCisco Security Advisory

CVE-2016-10708 CVE-2018-15473CVE-2018-20685 CVE-2019-6109CVE-2019-6111

CSCvp7646340(2a)

Multiple vulnerabilities associated withOpenSSL and LibNSS

For more information see the relatedCisco Security Advisory

CVE-2016-2105 CVE-2016-2106CVE-2016-2107 CVE-2016-2109CVE-2016-2176 CVE-2016-2177CVE-2016-2178 CVE-2016-2179CVE-2016-2180 CVE-2016-2181CVE-2016-2182 CVE-2016-2183CVE-2016-6302 CVE-2016-6303CVE-2016-6304 CVE-2016-6305CVE-2016-6306 CVE-2016-6307CVE-2016-6308 CVE-2016-7055CVE-2016-8610 CVE-2017-3731CVE-2017-3732 CVE-2017-3735CVE-2017-3736 CVE-2017-3737CVE-2017-3738 CVE-2018-0495CVE-2018-0732 CVE-2018-0734CVE-2018-0735 CVE-2018-0737CVE-2018-0739 CVE-2018-12384CVE-2018-12404 CVE-2018-5407CVE-2019-1559

CSCvp6667940(2a)

libpng vulnerability identified byCVE-2016-10087

For more information see the relatedCisco Security Advisory

CVE-2016-10087CSCvp6655540(2a)

Vulnerabilities associated with libjpg andlibpng

For more information see the relatedCisco Security Advisory

CVE-2014-9092 CVE-2016-3616CVE-2017-15232 CVE-2018-11212CVE-2018-11213 CVE-2018-11214CVE-2018-1152 CVE-2018-13785

CSCvp3458640(2a)

Vulnerabilities associated with Glibidentified by CVE-2018-16428 andCVE-2018-16429

For more information see the relatedCisco Security Advisory

CVE-2018-16428 CVE-2018-16429CSCvp3120740(2a)

Multiple vulnerabilities associated withcurl expat python 27 python 35 36and 37 freetype and giflib

For more information see the relatedCisco Security Advisory

CVE-2019-15133 CVE-2019-15903CVE-2019-5010 CVE-2019-5481CVE-2019-5482 CVE-2019-9636CVE-2019-9740 CVE-2019-9947CVE-2019-9948

CSCvr3690340(2a)

Release Notes for Cisco HX Data Platform Release 4025

Security Fixes

DescriptionCVEDefect IDRelease

Multiple third party softwarevulnerabilities

For more information see the relatedCisco Security Advisory

CVE-2019-14379 CVE-2019-12384CVE-2019-14439

CSCvq9203240(2a)

Vulnerabilities associated with elfutils

For more information see the relatedCisco Security Advisory

CVE-2018-16062 CVE-2018-16402CVE-2018-16403 CVE-2018-18310CVE-2018-18520 CVE-2018-18521CVE-2019-7149 CVE-2019-7150CVE-2019-7665

CSCvq4325040(2a)

Vulnerabilities associated with vim

For more information see the relatedCisco Security Advisory

CVE-2017-5953 CVE-2019-12735CSCvq4323040(2a)

Vulnerabilities associated with sqlite3

For more information see the relatedCisco Security Advisory

CVE-2016-6153 CVE-2017-10989CVE-2017-13685 CVE-2017-2518CVE-2017-2519 CVE-2017-2520CVE-2018-20346 CVE-2018-20505CVE-2018-20506 CVE-2019-8457CVE-2019-9936 CVE-2019-9937

CSCvq4321340(2a)

Vulnerabilities associated with glib20

For more information see the relatedCisco Security Advisory

CVE-2018-20843CSCvq4320940(2a)

Vulnerabilities associated with expat

For more information see the relatedCisco Security Advisory

CVE-2018-20843CSCvq4320540(2a)

Vulnerabilities associated with bzip2

For more information see the relatedCisco Security Advisory

CVE-2016-3189 CVE-2019-12900CSCvq4319440(2a)

Mulitple Vulnerabilities associated wtihNodeJS

For more information see the relatedCisco Security Advisory

CVE-2018-12115 CVE-2018-0734CVE-2018-5407 CVE-2018-12120CVE-2018-12121 CVE-2018-12122CVE-2018-12123 CVE-2018-12116CVE-2019-5737 CVE-2019-5739

CSCvq1069440(2a)

Vulnerabilities associated with PalletsJinja strformat_map and identified byCVE-2019-10906 and CVE-2016-10745

For more information see the relatedCisco Security Advisory

CVE-2019-10906 CVE-2016-10745CSCvq1038840(2a)

Release Notes for Cisco HX Data Platform Release 4026

Security Fixes

DescriptionCVEDefect IDRelease

Multiple Security Vulnerabilities

For more information see the relatedCisco Security Advisory

CVE-2014-8501 CVE-2014-9939CVE-2015-9262 CVE-2016-10087CVE-2016-2226 CVE-2016-4487CVE-2016-4488 CVE-2016-4489CVE-2016-4490 CVE-2016-4491CVE-2016-4492 CVE-2016-4493CVE-2016-6131 CVE-2018-10963CVE-2018-13785 CVE-2018-17100CVE-2018-17101 CVE-2018-18557CVE-2018-18661 CVE-2018-7456CVE-2018-8905

CSCvq0917840(2a)

Vulnerabilities associated withlibseccomp

For more information see the relatedCisco Security Advisory

CVE-2019-9893CSCvq0756840(2a)

Vulnerabilities associated withgdk-pixbuf

For more information see the relatedCisco Security Advisory

CVE-2017-12447CSCvq0675540(2a)

Vulnerabilities associated withpython-crypto

For more information see the relatedCisco Security Advisory

CVE-2018-6594CSCvp9381740(2a)

Vulnerabilities associated with wget

For more information see the relatedCisco Security Advisory

CVE-2018-20483 CVE-2019-5953CSCvp8672140(2a)

Multiple vulnerabilities associated withPython Crypto

For more information see the relatedCisco Security Advisory

CVE-2018-6594CSCvp6674840(2a)

Vulnerabilities associated withsensible-utils

For more information see the relatedCisco Security Advisory

CVE-2017-17512CSCvp6673440(2a)

Multiple vulnerabilities associated withPython

For more information see the relatedCisco Security Advisory

CVE-2018-1000030CSCvp6668940(2a)

Release Notes for Cisco HX Data Platform Release 4027

Security Fixes

DescriptionCVEDefect IDRelease

Multiple patch vulnerabilities identifiedby CVE-2016-10713

For more information see the relatedCisco Security Advisory

CVE-2016-10713CVE-2018-1000156 CVE-2018-6951

CSCvp6667240(2a)

Vulnerabilities associatedwith Little CMSidentified by CVE-2016-10165

For more information see the relatedCisco Security Advisory

CVE-2016-10165 CVE-2018-16435CSCvp6666440(2a)

Vulnerabilities associated with TenableScan identified byCVE-2011-5325CVE-2019-11068CVE-2016-7076CVE-2017-1000368

For more information see the relatedCisco Security Advisory

CVE-2011-5325 CVE-2016-7076CVE-2017-1000368CVE-2019-11068

CSCvp6474640(2a)

Vulnerabilities associated with libxml2294 and earlier as used in XMLSec1223 and earlier and other products andidentified by CVE-2016-9318

For more information see the relatedCisco Security Advisory

CVE-2016-9318 CVE-2017-16932CVE-2017-18258 CVE-2018-14404CVE-2018-14567

CSCvp3479240(2a)

Vulnerabilities associated with LFTPRemote File Names Unauthorized AccessVulnerability and identified byCVE-2018-10916

For more information see the relatedCisco Security Advisory

CVE-2018-10916CSCvp2926640(2a)

Vulnerabilities associated withCryptographyDeprecationWarning signerand verifier have been deprecated

For more information see the relatedCisco Security Advisory

CVE-2018-7750CSCvo3409740(2a)

Multiple Vulnerabilities in python python27 35

For more information see the relatedCisco Security Advisory

CVE-2019-16056CSCvr3979340(2a)

Release Notes for Cisco HX Data Platform Release 4028

Security Fixes

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toexecute commands as the root user

The vulnerability is due to insufficientinput validation An attacker could exploitthis vulnerability by connecting to thecluster service manager and injectingcommands into the bound process Asuccessful exploit could allow the attackerto run commands on the affected host asthe root user

Cisco has released software updates thataddress this vulnerability There areworkarounds that address thisvulnerability

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvq2417640(1b)35(2g)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

The vulnerability is due to an unprotectedlistening interface An attacker couldexploit this vulnerability by connectingto the listening interface and injectingcommands to the bound process Anexploit could allow the attacker to runcommands on the affected host as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvj9560640(1b)35(2g)

Release Notes for Cisco HX Data Platform Release 4029

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerabilities associated with theJVM18U121memory leak found duringVC alarms (concurrent 40 calls usingREST API)

CVE-2017-10053 CVE-2017-10067CVE-2017-10074 CVE-2017-10078CVE-2017-10081 CVE-2017-10087CVE-2017-10089 CVE-2017-10090CVE-2017-10096 CVE-2017-10101CVE-2017-10102 CVE-2017-10107CVE-2017-10108 CVE-2017-10109CVE-2017-10110 CVE-2017-10111CVE-2017-10115 CVE-2017-10116CVE-2017-10118 CVE-2017-10135CVE-2017-10176 CVE-2017-10193CVE-2017-10198 CVE-2017-10243CVE-2017-10274 CVE-2017-10281CVE-2017-10285 CVE-2017-10295CVE-2017-10345 CVE-2017-10346CVE-2017-10347 CVE-2017-10348CVE-2017-10349 CVE-2017-10350CVE-2017-10355 CVE-2017-10356CVE-2017-10357 CVE-2017-10388CVE-2017-3509 CVE-2017-3511CVE-2017-3526 CVE-2017-3533CVE-2017-3539 CVE-2017-3544CVE-2018-2579 CVE-2018-2582CVE-2018-2588 CVE-2018-2599CVE-2018-2602 CVE-2018-2603CVE-2018-2618 CVE-2018-2629CVE-2018-2633 CVE-2018-2634CVE-2018-2637 CVE-2018-2641CVE-2018-2663 CVE-2018-2677CVE-2018-2678 CVE-2018-2783CVE-2018-2790 CVE-2018-2794CVE-2018-2795 CVE-2018-2796CVE-2018-2797 CVE-2018-2798CVE-2018-2799 CVE-2018-2800CVE-2018-2814 CVE-2018-2815CVE-2018-2952 CVE-2018-3136CVE-2018-3139 CVE-2018-3149CVE-2018-3150 CVE-2018-3169CVE-2018-3180 CVE-2018-3183CVE-2018-3214 CVE-2019-2422

CSCvo8899740(1b)

Tomcat and Nginx logs are not beingcollected in the support bundle generatedthrough HX Connect in Release 35

NACSCvm5803140(1b)

The vulnerabilities associated with thelibmspack software package versionincluded in Cisco HX Data Platform

CVE-2018-18584

CVE-2018-18585

CSCvn3511940(1a)

Release Notes for Cisco HX Data Platform Release 4030

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerabilities associated withFasterXML Jackson-Databind TimeValueField Denial of Service

CVE-2018-14719

CVE-2018-14720

CVE-2018-1000873

CVE-2018-14721

CVE-2018-19360

CVE-2018-19362

CVE-2018-19361

CVE-2018-14718

CSCvn8228240(1a)

The vulnerabilities associated with theOpenSSL Protocol software packageversion included in Cisco HX DataPlatform

CVE-2013-3587CSCvo0505440(1a)

The vulnerabilities associated withThird-Party Software Denial of Service

CVE-2018-16487

CVE-2018-19361

CSCvo2781840(1a)

The vulnerabilities associated withLinuxkernel for Ubuntu 1710

CVE-2018-1092

CVE-2018-7492

CVE-2018-8087

CVE-2018-1068

CVE-2018-8781

CSCvm5314935(2a)

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)25(1d)stcli services dns remove should remove the DNS serverinfo from the interface files

CSCvs70967

40(2b)40(1b)HX 40(1b) new deployment from Intersight productioncloud

User receives following error while enabling CSI(Kubernetes integration)

Failure occurred during volume_access_enable Errorwas Nonzero exit code 1

CSCvt10522

40(2b)40(2a)Receive the following alertevent in HX Connect HXController VM HOSTNAME one or more configuredDNS servers not responding

CSCvt13947

Release Notes for Cisco HX Data Platform Release 4031

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)40(2a)API UI CLI show few Drive slots Empty after upgradeto 402a

The drives are seen in the sysmtool lsscsi and stclicommand outputs

The cluster is healthy with no errors

CSCvt14914

40(2b)35(2d)After successfully changing the DNS server on the HXcontroller we still can see the original DNS entry thatwas added during the deployment

CSCvs69154

40(2b)35(2e)HyperFlex Connect and VCenter show APD alarms forHX and non HX datastores This implies there is an issuewith HyperFlex when there may not be

CSCvs30080

40(2b)40(2a)An issue where all the disks in the node get locked afterrebooting the node

We successfully unlocked using sed-clientsh -Ucommand but wanted to test with another reboot anddrives locked again

CSCvs74286

40(2b)40(2a)When running stcli license commands on HyperFlexthe following error is seen

rootSpringpathControllertmp stcli license show all

Show smart licensing failed

Smart Agent is not ready please wait a minute and tryagain

CSCvt13929

40(2b)40(2a)After upgrading to HX 40(2a) the first RoboEdge nodethat gets rebooted then ignores persistent drivesPIDHX-HD24TB10K4KN

Cluster will be in degraded state and have reducedcapacity

CSCvt20144

40(2b)35(2b)Zookeeper fails to start while Exhibitor is runninghowever echo srvr returns nothing

CSCvs21562

40(2b)40(2a)When performing an HyperFlex upgrade a validationwarning may occur due to the host not having EnterprisePlus or Enterprise hypervisor licensing

Upgrade Validation Warning

ESXi host esx1labtest should be configured withVMware Enterprise license for upgrade to continue

CSCvs91787

Release Notes for Cisco HX Data Platform Release 4032

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)35(2g)Cluster expansion fails at Config Installer stage whenthe root and admin password for the storage controller(SCVM) are different

CSCvs69317

40(2b)35(2g)Panic while upgrading ESXiCSCvt06983

40(2b)40(2a)A cluster node running may hang in the Linux kernelThis is classified as an oops and a deviation from theexpected behavior

CSCvs54285

40(2b)35(2g)Rebalance failing on 10+10 node HX 35(2g) stretchcluster

CSCvs69007

40(2b)35(2d)The cluster becomes inaccessible to the IOVisorCSCvr54687

40(2b)40(2a)Panic on the storage controllerCSCvt61297

40(2b)40(2a)The size of support bundle is very large when collectedwith storfs-support command

CSCvt63306

40(2b)40(2a)storfs crash associated withFileSystemUsageWarningEvent orFileSystemUsageAlertEvent due to high usage ofvarstv above 80

CSCvt72807

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

40(2a)35(2d)In normal scenarios HX upgrade gets pendingACK afterupdating service profile In this case UCS only upgradewas stalled because after updating service profile HXupgrade was not getting pending ACK and it was alwayswaiting state

CSCvo39912

40(2a)35(1a)When using motherBoardReplace-12 to clean ZK fromold stNodepNodes resulted in unmounted datastore andmaking the size of the Hyperflex Datastores 0 resultingin all VMs in the cluster going offline

CSCvq39471

40(2a)26(1e)Upgrading cluster and getting error

Failed upgrade validations Checking vCenterconfiguration Reason Upgrade validations failed DRSFault Insufficient resources to satisfy configured failover

CSCvm77294

Release Notes for Cisco HX Data Platform Release 4033

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)The cluster expand fails on a node with DR replicationconfigured

When a HX cluster which has DR replication configuredis expanded we see the installer UI pulling in thereplicationVLAN information instead of themanagementVLAN information Even if we change that informationto the correct mgmt VLAN id and name it it does notseem to work as the node is configured with the VLANof the replicationVLAN in ESXi This leads to the failureof the node add with the host unreachable error

CSCvo70650

40(2a)35(1a)Customer reported that server didnt complete thefirmware upgrade automatically one by one

The user manually needs to put the host into maintenancemode and they manually acknowledged the pendingrequests to finish the UCS firmware upgrade

Maintenance policy is set to default (User-ack) as perthe design

CSCvo91624

40(2a)35(1i)When the cluster is in Failed state and stcli noderemove is attempted the output shows successful thoughthe node failed to get removed from the cluster

CSCvo93017

40(2a)35(1a)HyperFlex cluster upgrade may fail during validationwith a DRS Validation failed error

CSCvp31021

40(2a)35(2b)HX cluster expansion will fail with the following errormessage

MAC address pool configuration failure150[ErrorDescription] bad address block range definitioncollision

CSCvp58318

40(2a)35(2b)Memory usage by carbon cacheCSCvq34873

40(2a)35(2g)Upgrading ESXi cluster fails with error Nodemaintenance mode failed

CSCvr03240

40(2a)35(2e)storfs process does not automatically start on Exit HXMaintenance Mode or other tasks which restart StorageController

CSCvr88978

40(2a)35(2a)Post install script fails with message

Failed to execute ipmitool on HX node

CSCvp10707

40(2a)40(1a)HyperFlex expansionworkflow doesnt pull VLANnamecorrectly

CSCvp46539

Release Notes for Cisco HX Data Platform Release 4034

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)During HX cluster deployment - Validate ClusterCreation phase - HX Installer might fail with the error

fromvarlogspringpathstDeploylog

CSCvq45087

40(2a)35(2b)HX Installer fails to login to the SOL for configuringESX hosts

CSCvq91380

40(2a)35(2d)If DRS is enabled with configuration parameters otherthen fully automated ie manual or partially automatedthan this will change to fully automated during clusterexpansion

CSCvr44222

40(2a)35(1a)One of the validation task name has a typo The taskname is Validating node for mixed of SED and non-SEDdisks

It should be Validating node for mix of SED andnon-SED disks

CSCvp82175

40(2a)30(1i)This combination of using smaller NVMe drives is notsupported in expand operation For example if the drivesare only 375GB and they cannot be added to an existingcluster with larger caching SSD

CSCvo12359

40(2a)35(1a)HyperFlex includes a version of OpenSSL that is affectedby the vulnerabilities identified by the followingCommon Vulnerability and Exposures

CVE-2018-0495

CSCvp66679

40(2a)30(1e)SDK invoke exception when taking a native snapshot ona VM with GPU pci passthrough attached

CSCvo00511

40(2a)30(1i)Node replacement script fails due to EAM errorCSCvo62867

40(2a)35(1a)While pairing a cluster (HX release gt= 35) and cluster(HX release lt 35) remote replication network test failson the cluster (HX release lt 35)

CSCvo79760

40(2a)30(1b)Fixed the issue in the latest support-workflow bundleThis should not be hit anymore

CSCvo87061

40(2a)35(2a)On a 3 node cluster the MBreplace script fails as thescript looks for HEALTHY cluster status

CSCvo87080

40(2a)35(2a)MbReplace script (tar) hangs when running on 352b or40

CSCvp12359

Release Notes for Cisco HX Data Platform Release 4035

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)HX replication cleanup failing with error NA 35(2a) -INFODR state is not clean

CSCvp63958

40(2a)40(1a)The check_stig_parameters API incorrectly shows thecompliance state of the cluster as not STIG compliant

CSCvp66277

40(2a)35(2c)The snapshot creation on CBT enabled VM fails witherror Failed in vmreparent vmkfstools clone1

CSCvq06952

40(2a)35(2a)HyperFlex GUI when setting up remote DR-replicationpartnerpeer should show remote replication partner TCPport reachability test results

This will provide quick upfront connectivity results thatfrequently takes customer a while to manually check andconfirm

If reachability tests fail an info window and HX alertshould trigger that list the destination peer IP and thespecific port(s) that could not be reache

CSCvo60587

40(2a)35(2b)If you have more than one IP pool configured for areplication network and local replication network testsare failing you might be running into this issue

CSCvr67130

40(2a)35(2a)Skip task option in MBreplace and Software redeployscripts The scripts should have the option to skip tasksas user inputs and not by modifying the json file

CSCvp05204

40(2a)40(1a)Error noticed on SCVM Console print_req_error IOerror dev fd0 sector 0 on HyperV

It doesnt come up on SSH session

CSCvq34357

40(2a)30(1d)The VMNetwork switch is already created as part of theinstall process

Ideally the ip address assignment should be done duringthe initial install process as well No post install stepsshould be required altrnatively it could be done as partof a post install script

CSCvk36222

40(2a)25(1c)storfs service impacted during HX upgrade from HX21(1) to 25(1c)

HX node which is not being upgraded storfs servicestopped causing outage

CSCvg53223

40(2a)35(1b)HX Stretch Cluster Witness VM reverts to DHCP onreboot

CSCvp37536

40(2a)35(2b)HyperFlex Stretch Cluster Witness does not getprogrammed with an NTP server

CSCvp46578

Release Notes for Cisco HX Data Platform Release 4036

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2b)HyperFlex cluster does not heal after maintenance onnodeserver

CSCvr18528

40(2a)40(2a)High percentage of packet loss on the eth1 data interface9K MTU

CSCvr97089

40(2a)35(2f)The HyperFlex cluster healing state is stuck at 87 andis unable to progress

CSCvr16760

40(2a)40(2a)In some circumstances two replications can occur forthe same VM

From the GUI the replication status for this VM willshow stuck in an In Progress state but there are nojobs stuck in replication layer Subsequent Replicationsshould succeed

CSCvq22898

40(2a)35(2a)After an unexpected power outage the hyperflex clusteris unable to come up correctly

All processes are running time is synced and all nodescan vmkping each other as required

CSCvp13990

40(2a)40(1a)When specifying IP address instead of node ID stclinode maintenanceMode commands fail

CSCvp33657

40(2a)40(1a)2 node edge cluster if experiencing Intersightarbitratorconnectivity issues should NOT allowHXMaintenancemode to initiate as this will bring the cluster down

CSCvr31573

40(2a)35(2c)Old files in varsupportZKTxnlog are not purged withthe daily zklog-cleanup cron job

CSCvr89066

40(2a)35(2a)When a node is in MaintenanceMode any disk removalor replacement will be reflected in UI only after the nodeis brought back frommaintenancemode This is becausestorfs is not running on the node in maintenance andwill not be able to detect disk activities until it is broughtout of Maintenance Mode

CSCvo86431

40(2a)35(2c)Cluster might go offline and stop serving IO requestsrootcvm~ stcli cluster storage-summary--detailGet cluster storage summary failedjavanetConnectException Connection refused19216814210010207

CSCvr01645

40(2a)35(2d)A stretch cluster deployment fails with a messageFormatting nodes Node in Use on the Hyperflex installerand crmZoneType value is seen as 1 in storfscfg oroptspringpathconfigstretchtunes

CSCvq80340

Release Notes for Cisco HX Data Platform Release 4037

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)30(1c)During upgrade we upload the upgrade packagedownloaded from Ciscocom Upgrade packages are offile type tgz hxconnect acceptsstorfs-packages-401a-33028tar file a different type ofcompression and allows upgrade to start and fail whenit cannot decompress the file for bootstrap and scvmclientupgrade on ESXi

CSCvq63888

40(2a)40(2a)While attempting to install hyperflex via intersight thefollowing error is seen

Failed in Task Add hosts to vCenter Cluster with ErrorTry adding hosts manually to vCenter and retry Failedto get addressFailed to host by namelookupHostnamecompanycom on 000053 read udp12700145430-gt12700153 read connection refused

CSCvr52098

40(2a)35(2a)CoreAPI call may time out for cluster management APIwhile doing cluster create validation

CSCvo92952

40(2a)35(2d)Validation fails with Mixed mode expansion checkdue to Enhanced vMotion Compatibility Incompatibilitywhen the ESXi version is different

CSCvq95460

40(2a)30(1i)This failure is seen in installer when custom workflowis used hypervisor configuration + deploy (clean diskpartition=no) at Installing software packages on storagecontroller VM

CSCvr66309

40(2a)35(2d)The HX Installer apparently replaced a comma bysemicolon on the VLAN ID range while exporting theconfiguration to a JSON file

CSCvq93831

40(2a)40(1a)When using unsupported browsers HX Connect userswill not be able to login These users see an errorindicating unauthorized user

CSCvp22693

Release Notes for Cisco HX Data Platform Release 4038

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)40(1a)A vulnerability in the web-based management interfaceof Cisco HyperFlex Software could allow anunauthenticated remote attacker to conduct a cross-siterequest forgery (CSRF) attack on an affected system

The vulnerability is due to insufficient CSRF protectionsfor the web UI on an affected device An attacker couldexploit this vulnerability by persuading a user of theinterface to follow a malicious link A successful exploitcould allow the attacker to perform arbitrary actions withthe privilege level of the affected user

Cisco has released software updates that address thisvulnerability There are no workarounds that address thisvulnerability

This advisory is available at the following link

httpstoolsciscocomsecuritycentercontentCiscoSecurityAdvisorycisco-sa-20190807-hypflex-csrf

CSCvp24343

40(2a)35(2d)Add message to HX Connect Progress flow to notAcknowledge Pending Activities and reboot the serverson UCSM

HX Connect is doing a controlled rolling server upgradein the background

CSCvq22844

40(2a)40(1b)Compute Node expand operation fails due to incorrectcluster name

CSCvr43786

40(2a)26(1e)RF-2 cluster shutdown during data resynchronizationafter a node failure (non storfs) followed by multipledisk read failures and a HardBlacklist of a disk

CSCvp41241

40(2a)40(1a)HyperFlex Edge nodes do not properly set the ESXihostname during deployment

CSCvo13143

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4039

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4040

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1b) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401b-33133p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4041

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4042

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1b)40(1a)While running the HyperFlex installer with WindowsServer Hyper-V cluster creation process fails with theerror Failure occurred during ClusterCreation process Unable to post thecontent to the downstream This symptomis encountered while deploying HyperFlex cluster withHyperFlex nodes configuredordered with Cisco VIC1457 MLOM (PID HX-MLOM-C25Q-04) andWindows Server Datacenter or Core with Hyper-Vversion 2016 or 2019

CSCvp64140

Release Notes for Cisco HX Data Platform Release 4043

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)35(2b)Adding Micron 5200 drive to a cluster fails to increasecluster capacity

CSCvo69067

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4044

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4045

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1a) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401a-33028p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4046

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4047

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1a)30(1d)Cluster instability when disks of different sector sizeplaced in HX node

CSCvk17250

40(1a)

35(2c)

35(1a)When logged in using a local HX user account insteadof a Virtual Center account an error message appearsintermittently indicating Virtual Centerunreachable or Resource informationcannot be updated when VC is reachable

CSCvo36198

40(1a)

35(2b)

30(1d)HXDP does not work with EMC RecoverPoint - needsto support VMware API (FSS-Readdir)

CSCvk38003

Release Notes for Cisco HX Data Platform Release 4048

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)

35(2a)

30(1i)

25(1c)

25(1d)

Zookeeper (Exhibitor) process on HyperFlex StorageController VMs may fail to respond to requests whenvarzookeeper has low or no free space

CSCvm90352

40(1a)26(1c)Use Asynchronous consolidation for HX SnapshotsCSCvn02151

40(1a)35(2b)HX Quiesced Snapshot for Backup VendorsCSCvo90713

40(1a)35(2a)The cluster creationcluster expansion workflow stopswith the following error message at the validation step

FIRMWARE-Check UCSC-SAS-M5HD

FIRMWARE-Check UCSC-SAS-M5HD Required00000029000000320000003500000050 Found00000058

Action Needed Update the Controller Firmware toRequired Version

CSCvn17787

35(2b)30(1d)Kernel migration fails when a local datastore is searchedfor in the ESXi host

CSCvn73127

35(2a)26(1b)A node shuts down when two disks are replaced (acaching disk and another capacity disk) where thecapacity disk is inserted first and then the caching diskis inserted

CSCvk46364

Hyper-V

40(1a)35(2a)Cluster expansion may fail with an error code 500 -operation timed out

CSCvn28721

35(2b)35(2a)During upgrade remove the VLAN on team created foruser vSwitch During fresh installation only one VLANtag is set to the vSwitch and team although multipleVLANs were entered

CSCvn54300

35(2b)35(2a)While upgrading a Hyper-V cluster on account of a rarerace condition between the stUpgradeService andZookeeper servers the upgrade orchestration throws anupgrade validation error and the upgrade process isaborted

CSCvn60486

Release Notes for Cisco HX Data Platform Release 4049

Resolved Caveats in Release 40(1a)

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Startrestart EAM service invCenter

When EAM service is not runningvCenter registration is failing

CSCvt55712

35(2a)Contact Cisco TAC if you believeyou are seeing similar conditions

HyperFlex Stretch Cluster saw poorapplication performance due to writelatency cluster remained unhealthy andrebalance was stuck

This defect is currently being used forinvestigative purposes

CSCvt45344

35(2g)Remove log files fromvarlogspringpath folder when thelog files are not getting rotatedStop the relevant service removelarge log files fromvarlogspringpath folder andrestart the service

varstv folder in Controller VM maybecome full

CSCvt36374

40(2b)Restarting the stMgr will initializeit again

In a Cluster configured with DR networkStMgr may not get initialized and stuckin a deadlock while enabling the IPTablerules (can be verified from the stMgrlog)

CSCvt89709

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Click recover for the selectedVM again under the standalonemode This will move the VM torecovered state

When the user recovers a VM protectedin a group it along with other VMs in thegroup move to recovering state instandalonemode The selectedVM shouldmove to recovered state

CSCvs75553

35(2e)Clean up the fail over cluster andthen recreate the fail over clusterNo need to touch the HX storagecluster

Hyper-V When replicated DC was usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2d)40(1a)

Keep cluster utilization as low aspossible To avoid read-only statekeep below 92

HyperFlex clusters storage utilization isabove 92 usable capacity

CSCvr20922

Release Notes for Cisco HX Data Platform Release 4050

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1b)NAAll the disks in the node got locked afterrebooting the node

We successfully unlocked usingsed-clientsh -U command but wantedto test with another reboot and driveslocked again

CSCvs74286

35(2g)To recover the VM copy over thedata disks and attach them to thenew VM

After enable encryption HX connectencryption status showing Caution butall of the disks enable encryptionsuccessful

-Self Encrypting Drives Service isrunning on all of the nodes

-There is no error message of encryption

- USB0 interface is up

-All SED disks showing supported 1enabled 1 locked 0

CSCvs93245

40(1b)Enable kernelpanic_on_oops as apersistent configuration This willcause the node to panic and rebootimmediately

A cluster node running HX release40(1b) may hang in the Linux kernelThis is classified as an oops and adeviation from the expected behavior

CSCvs54285

Release Notes for Cisco HX Data Platform Release 4051

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Enabling K8iscsi stack in Hx clusterCSCvs41324

Release Notes for Cisco HX Data Platform Release 4052

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Steps to enable K8 on HX cluster

Perform following steps on ALLthe controller VMs

1 Updateetcinitscvmclientconf toenable tune iscsiEnable totrue

sed -iesiscsiEnable=falseiscsiEnable=trueetcinitscvmclientconf

2 Run following initctlcommand to reloadconfiguration

initctl reload-configuration

3 Restart scvmclient process

stop scvmclient startscvmclient

4 Run following initctlcommand to mount SYSTEMDS

initctl emit --no-waitsystem-datastore-created

5 Verify that iscsiEnable tuneis set to true using thefollowing command

ps -eaf | grep scvmclient |grep -v grep

root 6241 1 1 Dec06 014007optspringpathstorfs-corescvmclient-T logEnabled=false -TlogSyslogEnabled=true -TlogEchoToScreen=false -TstatLoggingToFile=false -TstatLoggingToSyslog=true-TlogDir=varlogspringpath-TnfsBackendServerList=1010748100-T iscsiEnable=true -TiscsiUseAsync=true -T

Release Notes for Cisco HX Data Platform Release 4053

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

iscsiConfigFilePath=nfsSYSTEMistgtconfps -eaf | grep scvmclient |grep -v grep

35(2e)Following procedure can be usedto check NFS Queue Depth andincrease if needed

rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth256 lt- Low value[rootHXESXI1] vsish -e setvmkModulesnfsclientmnt[DATASTORE]propertiesmaxQDepth 1024[rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth1024 lt- Optimalvalue

This requires an ESXi host rebootto take effect

Please place the node in HyperflexMaintenanceMode and gracefullyreboot the node for the changes tobe applied

HyperFlex Datastore NFS Queue Depthshows as 256 which can lead toperformance (including latency) issues

CSCvr83056

40(1b)This is cosmetic The error will goaway once the cluster returns tohealthy status

When a single node is offline in clusterstcli cluster storage-summary shows twonodes unavailable rootSCVM~ stclicluster storage-summary

CSCvs47735

35(2g)Modify both the root and adminpassword for the SCVMs to be thesame

stcli security password set -uadmin

Cluster expansion fails at Config Installerstage when the root and admin passwordfor the storage controller (SCVM) aredifferent

CSCvs69317

35(2d)NAAfter successfully changing the DNSserver on the HX controller we still cansee the original DNS entry that was addedduring the deployment

CSCvs69154

Release Notes for Cisco HX Data Platform Release 4054

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Customers are strongly encouragedto work with TAC in order toidentify and remediate this issue

1 Change getcluster from 2m to8m inoptspringpathstorfs-mgmtstMgr-10confapplicationconfon all nodes

2 Restart stMgr on all nodes

You may see this error message after afailed upgrade or other task such asattempting to enter a node into HXmaintenance mode

CSCvs53555

40(2a)Run the following command oneach storage controller VM as root

grep -i monitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp sed -iesmonitor_dns_serverstruemonitor_dns_serversfalseoptspringpathhx-diag-toolswatchdog_configjsonampamp grep -imonitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp restart watchdog

Receive the following alertevent in HXConnect HX Controller VMHOSTNAME one or more configuredDNS servers not responding

CSCvt13947

35(2b)Delete any empty (Size 0) log filesunder varlogzookeeperversion-2ORvarlogzookeeperstandaloneversion-2

Zookeeper fails to start while Exhibitor isrunning however echo srvr returnsnothing

CSCvs21562

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

40(4e)1 De-commission the server

2 Power drain the server -REMOVE BOTH POWERCORDS ON THE BACK OFTHE SERVER FOR 10SECONDS THENREINSERTPOWERCORDS

3 Re-commission the server

M2 boot disk is missing from serverinventory after upgrade to server firmware40(4e) As a result server fails to boot toOS installed in the M2 disk The issuepersists after re-acknowledgement as wellas de-commission andre-acknowledgement of the server

CSCvs02466

Release Notes for Cisco HX Data Platform Release 4055

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Re-start the flusher manuallyMany disks taken offline by a fault ormanually resulting in ENOSPC oncluster

CSCvq32721

35(2e)Clean up fail over cluster and thenrecreate the fail over cluster Thereis no need to touch the HX storagecluster

[Hyper-V]When replicated DCwas usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2b)Manually install upgradescvmclient VIB andmake sure thatit matches with HXDP version

Upgrade from release HX 35(2a) to35(2x) 4X does not upgrade scvmclientRecovery point may not work

CSCvq54992

Management

30(1b)To recover the VM copy over thedata disks and attach them to thenew VM

[HyperV]VMshows up onmultiple nodesCSCvj22992

35(1a)1 Do not change MTU at ESXlevel on running cluster

2 Reset back to the original value

ChangingMTU from 9000 to 1500 on oneESX node causes storfs process to restarton all nodes

CSCvm99150

Release Notes for Cisco HX Data Platform Release 4056

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)After the installation (for newdeployments) or after an upgradeto 401a (for existingdeployments) perform thefollowing steps on all ControllerVMs

1 Edit the following tune files (onall Controller VMs)

optspringpathconfiglfftunes

optspringpathconfigvsi_16tbtunes

2 SetcleanerEnableSegSummaryCleaning

to false

3 After editing the above tunefiles

a SSH in to all the ControllerVMs

b Run storfstool -- -Z

c Run the below command andcheck the values Tune valueshould be true

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=true

d Type below commands andapplied the tunes changesdynamically

echo false gttmpstprocfssystemtunecleanerEnableSegSummaryCleaning

e Verify that tune values arechanged Run below commandValue should be false

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=false

f umount tmpstprocfs

A cluster with 8TB or 12TB disk drivesmay experience IO stalls for severalminutes after another node fails in thecluster

CSCvp23718

Release Notes for Cisco HX Data Platform Release 4057

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)1 Update the conf file manuallyto change to the link local addressto

2 Restart scvmclient on thecontroller and ESX

HX 35(2b) FlexVol to HX 40 CSIupgrade does not work FlexVol continuesto work

CSCvp26319

35(2a)Disable rebalance until node healsthen reenable rebalance

7-node cluster healing very slowlybecause rebalance change AMS reqs areblocking healing process

CSCvp62512

35(2a)NAWith moderate cluster storage capacityusage under site failover cluster clustermay experience an all paths (APD) downcondition

CSCvq49412

40(1a)NAThis product includes Third-partySoftware that is affected by thevulnerabilities identified by the followingCommon Vulnerability and Exposures(CVE) IDs

CVE-2016-0762 CVE-2016-6797CVE-2016-6816 CVE-2016-8735CVE-2017-5647 CVE-2017-12615CVE-2017-12616 CVE-2017-12617CVE-2017-7674 CVE-2018-1304CVE-2018-8014 CVE-2018-1336CVE-2018-8034 CVE-2018-11784CVE-2019-0232

CSCvp36364

35(2d)NAstcli cluster info command needs toprovide UCSM VIP address Currently itshows ucsm-hostcom

CSCvq11456

35(2a)No workarounds currently Thestale entries do not affect the dayto day operations of HyperFlex

HyperFlex upgrade validation failedbecause cluster has extraneous stNodes ininternal database

CSCvq32530

35(2a)NAWhen not in maintenancemode all drivesappear in HX Connect and get updated asexpected

When in maintenance mode extra drivesare shown in HX Connect with no slotnumbers

CSCvq42378

35(2b)Contact TAC to cleanup duplicatenode

Nodes duplicated in HX Connect aftercluster recreate

CSCvq61825

Release Notes for Cisco HX Data Platform Release 4058

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)NACurrently HyperFlex installs on Hyper-Vdo not contain the stcli security whitelistcommandset

CSCvq66245

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

Release Notes for Cisco HX Data Platform Release 4059

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)[Hyper-V] Converged node expansionfails for incoming node with VIC 14xx

CSCvq39523

Release Notes for Cisco HX Data Platform Release 4060

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

1 Set Jumbo frame inWindowsHypervisor for incomingnode

Get-NetAdapter storage-data-a|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

Get-NetAdapter storage-data-b|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

2 Verify all interfaces are setcorrectly to support Jumbo frames

Get-NetAdapter storage |Get-NetAdapterAdvancedProperty| RegistryKeyword -matchjumbo | ft -auto

You should receive the followingmessage

Name DisplayName DisplayValue

RegistryKeyword RegistryValue

---- ----------- ------------

--------------- -------------

vswitch-hx-storage-data Jumbo

Packet 9014 Bytes

JumboPacket 9014

storage-data-b Jumbo Packet

Bytes 9014 JumboPacket

9014

storage-data-a Jumbo Packet

Bytes 9014 JumboPacket

9014

3 Reboot only the incoming nodeor the node being expanded

4 Retry Cluster Expansion in

Release Notes for Cisco HX Data Platform Release 4061

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

Recommended Firmware Version - HXDP 4x

(be sure to review important note(s) above)

Component

13000012SAS HBA

43(2b)VIC 1227

0103 (Toshiba)

CS07 (Intel 480)

SSD

CN03 (Seagate)

CK03 (Seagate 18)

CK03 (Seagate 24)

sF3Q (Samsung)

3F3Q (Samsung)

HDD

40(2h)

Download Software

Host Upgrade Utility (HUU) Download Link

Table 3 HX220c M5 HXAF220c M5 Cluster

Recommended Firmware Version - HXDP 4x

(be sure to review important notes above)

Component

C220M5404p00224200755BIOS

40(4i)Cisco Integrated Management Controller(CIMC)

09000008SAS HBA

43(3c)VIC 1387

50(3e)VIC 1457

G201CS01 (Intel)

0104 (Toshiba)

0107 (Toshiba)

SSD

2F3Q (Samsung)

3F3Q (Samsung)

NOA4 (Seagate)

CK03 (Seagate 18tb)

CK03 (Seagate 24)

HDD

Release Notes for Cisco HX Data Platform Release 4011

HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments

Recommended Firmware Version - HXDP 4x

(be sure to review important notes above)

Component

40(4k)

Download Software

Host Upgrade Utility (HUU) Download Link

HyperFlex Licensing

As of version 26(1a) HyperFlex supports VMware PAC licensing Existing VMware embedded licenses willcontinue to be supported

As of version 25(1a) HyperFlex uses a smart licensing mechanism to apply your licenses See the CiscoHyperFlex Systems Installation Guide for VMware ESXi for details and steps

VMware vSphere Licensing Requirements

How you purchase your vSphere license determines how your license applies to your HyperFlex system

bull If you purchased your vSphere license with HyperFlex

Each HyperFlex server either has the Enterprise or Enterprise Plus edition preinstalled at the factory

bull HX Nodes have OEM licenses preinstalled If you delete or overwrite thecontent of the boot drives after receiving the HX servers you also delete thefactory-installed licenses

bull OEM license keys is a newVMware vCenter 60 U1b feature Earlier versionsdo not support OEM licenses

bull All factory-installed HX nodes share the same OEM license key WithvSphere OEM keys the Usage count can exceed the Capacity value

bull When you add an HX host to vCenter through the Add Host wizard in theAssign license section select the OEM license

We obfuscate the actual vSphere OEM license key for example0N085-XXXXX-XXXXX-XXXXX-10LHH

bull Standard Essentials Plus and ROBO editions are not available preinstalledon HX servers

Note

bull If you did NOT purchase your vSphere license with HyperFlex

The HX nodes have a vSphere Foundation license preinstalled After initial setup you can apply thelicense to a supported version of vSphere

bull If you purchased a vSphere PAC license

Follow the instructions in your PAC license letter from VMware to add the license to your MY VMwareaccount then follow the instructions to add your HX host to vCenter and assign the PAC license

Release Notes for Cisco HX Data Platform Release 4012

HyperFlex Licensing

HX Data Platform Software Versions for HyperFlex Witness Node

Witness Node VersionHyperFlex Release

10840(2b)

10840(2a)

10440(1b)

10440(1a)

Software Requirements for VMware ESXi

The software requirements include verification that you are using compatible versions of Cisco HyperFlexSystems (HX) components and VMware vSphere VMware vCenter and VMware ESXi

bull Verify that all HX servers have a compatible version of vSphere preinstalled

bull Verify that the vCenter version is the same or later than the ESXi version

bull Verify that the vCenter and ESXi versions are compatible by consulting the VMware ProductInteroperability Matrix Newer vCenter versions may be used with older ESXi versions so long as bothESXi and vCenter are supported in the table below

bull Verify that you have a vCenter administrator account with root-level privileges and the associatedpassword

For VIC1457 there is no support for ESXi 60Note

The below table applies for all of the following VMware vSphere Editions Enterprise Enterprise PlusStandard Essentials Plus ROBO

VMware vCenter VersionsVMware ESXi VersionsHyperFlex Version

60 U3 65 U3 67 U360 U3 65 U3 67 U340(2b)1

60 U3 65 U3 67 U360 U3 65 U3 67 U340(2a)2

60 U3 65 U3 67 U260 U3 65 U3 67 U2340(1b)

60 U3 65 U2 67 U260 U3 65 U2 67 U2440(1a)

1 Cisco HyperFlex Release 40(2) is the last major HyperFlex release that will support vSphere 60 (ESXiand vCenter) due to those versions reaching end of VMware general support on March 12 2020

2 Cisco HyperFlex Release 40(2) is the last major HyperFlex release that will support vSphere 60 (ESXiand vCenter) due to those versions reaching end of VMware general support on March 12 2020

3 Use of 67 U2 is not recommended see software advisory for CSCvq06952 and CSCvp88515 for furtherdetails

4 Use of 67 U2 is not recommended see software advisory for CSCvq06952 and CSCvp88515 for furtherdetails

Release Notes for Cisco HX Data Platform Release 4013

HX Data Platform Software Versions for HyperFlex Witness Node

Software Requirements for Microsoft Hyper-V

The software requirements include verification that you are using compatible versions of Cisco HyperFlexSystems (HX) components and Microsoft Hyper-V (Hyper-V) components

HyperFlex Software versions

The HX componentsmdashCisco HX Data Platform Installer Cisco HX Data Platform and Cisco UCSfirmwaremdashare installed on different servers Verify that each component on each server used with and withinthe HX Storage Cluster are compatible

bull Cisco HyperFlex M5 Converged nodesmdash For Hybrid (Cisco HyperFlex HX240c M5 HX220c M5)and All Flash (Cisco HyperFlex HXAF240cM5 HXAF220cM5) verify that Cisco UCSManager 40(2b)is installed HX 40 (1a) does not support Hyper-V on the All NVMe (HXAF220C-M5SN) nodes Fordetailed information on installation requirements and steps see theCisco HyperFlex Systems InstallationGuide on Microsoft Hyper-V

Table 4 Supported HyperFlex Software versions for M5 Servers on Hyper-V

M5 Recommended Server FirmwareHyperFlex Release

40(4h)40(2b)

40(4g)40(2a)

40(4e)40(1b)

40(4e)40(1a)

Release Notes for Cisco HX Data Platform Release 4014

Software Requirements for Microsoft Hyper-V

Table 5 Supported Microsoft Software versions

VersionMicrosoft Component

Windows Server 2016 Datacenter Core amp DesktopExperience

ForWindows Server 2016Datacenter Coreand Desktop Experience the Windows2016 ISO image should be Update BuildRevision (UBR) 1884 at a minimum

Note

Windows Server 2019Datacenter-DesktopExperienceis supported starting from HXDP 401(a) onwards

For Windows Server 2019 DesktopExperience theWindows 2019 ISO imageshould be Update Build Revision (UBR)107 at a minimum

Note

Windows Server 2019 DatacenterndashCore is notsupported currently

Also note that the following are currently notsupported

OEM activated ISOs and Retail ISOs are notsupported

Earlier versions ofWindows Server such asWindows2012r2 are not supported

Non-English versions of the ISO are not supported

Windows Operating System (Windows OS)

A Windows 2012 or later domain and forestfunctionality level

Active Directory

Supported Microsoft License Editions

The Microsoft Windows Server version that is installed on one or more HyperFlex hosts must be licensed asper Microsoft licensing requirements listed on Microsoft Licensing

Browser Recommendations

Use one of the following browsers to run the listed HyperFlex components These browsers have been testedand approved Other browsers might work but full functionality has not been tested and confirmed

Table 6 Supported Browsers

HX ConnectHX Data PlatformInstaller

Cisco UCS ManagerCisco IntersightBrowser

11 or higher11 or higher11 or higherNAMicrosoft InternetExplorer

70 or higher70 or higher57 or higher62 or higherGoogle Chrome

Release Notes for Cisco HX Data Platform Release 4015

Browser Recommendations

HX ConnectHX Data PlatformInstaller

Cisco UCS ManagerCisco IntersightBrowser

60 or higher60 or higher45 or higher57 or higherMozilla Firefox

NANA9 or higher10 or higherApple Safari

NANA35 or higherNAOpera

Notes

bull Cisco HyperFlex Connect

The minimum recommended resolution is 1024 X 768

bull Cisco HX Data Platform Plug-in

The Cisco HX Data Platform Plug-in runs in vSphere For VMware Host Client System browserrequirements see the VMware documentation at httpswwwvmwarecomsupportpubs

The HX Data Platform Plug-in is not displayed in the vCenter HTML client You must use the vCenterflash client

bull Cisco UCS Manager

The browser must support the following

bull Java Runtime Environment 16 or later

bull Adobe Flash Player 10 or higher is required for some features

For the latest browser information about Cisco UCSManager refer to themost recent Cisco UCSManagerGetting Started Guide

Cisco HX Data Platform Storage Cluster SpecificationsCluster Limits

bull Cisco HX Data Platform supports up to 100 clusters managed per vCenter as per VMware configurationmaximums

bull Cisco HX Data Platform supports any number of clusters on a single FI domain Each HX convergednode must be directly connected to a dedicated FI port on fabric A and fabric B without the use of a FEXC-series compute only nodes must also directly connect to both FIs B-series compute only nodes willconnect through a chassis IO module to both fabrics In the end the number of physical ports on the FIwill dictate the maximum cluster size and maximum number of individual clusters supported in a UCSdomain

The following table provides Cisco HX Data Platform Storage Cluster Specifications

Release Notes for Cisco HX Data Platform Release 4016

Cisco HX Data Platform Storage Cluster Specifications

Stretched Cluster (Availableon ESX Only)

Microsoft Hyper-VVMware ESXiNode

AllNVMe -HXAF220c-M5SN

HX240c-M5LHX220cM5

HX220cAF M5

HX240cM5

HX240cAF M5

HX240c-M5LHX220cM5

HX220cAF M5

HX240cM5

HX240cAF M5

HXAF220c-M5SN

AllNVMe -HXAF220c-M5SN

ESXi 65or 67only NotsupportedwithESXi 60orHyper-V

HX220cM5 Edge

HXAF220cM5 Edge

HX220cM4 Edge

HXAF220cM4 Edge

HX240c-M5LHX220cM5

HX220cAF M5

HX240cM5

HX240cAF M5

HX220cM4

HX220cAF M4

HX240cM4

HX240cAF M4

HXServers

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

C220M5C240M5B200M4B200M5

C240M5C220M5B200M4B200M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

mdashB200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

Compute-OnlyUCSB-SeriesC-SeriesServers

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

Convergednodesonly

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

SupportedNodes

Release Notes for Cisco HX Data Platform Release 4017

Cisco HX Data Platform Storage Cluster Specifications

Stretched Cluster (Availableon ESX Only)

Microsoft Hyper-VVMware ESXiNode

NA(requiresEnterpriseHXDP-PLicense)

NA(requiresEnterpriseHXDP-PLicense)

NA(requiresEnterpriseHXDP-PLicense)

Convergednodes3-16(12TBHDDoption isnotsupportedforHyperV)

Compute-onlynodes0-16

Convergednodes3-16

Compute-onlynodes0-16

NA(requiresEnterpriseHXDP-PLicense)

M4Convergednodes 3

M5Convergednodes23or 4

RequiresHXDP-ELicense

Convergednodes3-16(12TBdrive islimited toamaximumof 8nodes)

Compute-onlynodes0-16(12TBdrive islimited toamaximumof 8nodes)

Convergednodes3-32(76TBdriveconfigsarelimited tomaximumof 12drivesnodeon AF240)

Compute-onlynodes0-32

HXDP-SLicensedNodeLimits

11 ratioofHXDP-StoComputeonlynodes

(MinmdashMax)

Convergednodes2-16 perSite

Compute-onlynodes0-21 perSite

(up tomaxclustersize)

Convergednodes2-8 perSite

Compute-onlynodes0-16 perSite

(up tomaxclustersize)

Convergednodes2-16 perSite

Compute-onlynodes0-21 perSite

(up tomaxclustersize)

Convergednodes3-16(12TBHDDoption isnotsupportedforHyperV)

Compute-onlynodes0-16

Convergednodes3-16

Compute-onlynodes0-16

Convergednodes3-32

ComputeOnlynodes0-32

(up tomaxclustersize)

Convergednodes 3

(requiresHXDP-ELicense)

Convergednodes3-16(12TBdrive islimited toamaximumof 8nodes)

Compute-onlynodes0-32(12TBdrive islimited toamaximumof 16nodes)

Convergednodes3-32(76TBdriveconfigsarelimited tomaximumof 12drivesnodeon AF240)

Compute-onlynodes0-32

(up tomaxclustersize)

HXDP-PLicensedNodeLimits

12 ratioofHXDP-PtoComputeonlynodes

(MinmdashMax)

24 perSite 64percluster

24 perSite 48percluster

32 perSite 64percluster

32326434864MaxClusterSize

Release Notes for Cisco HX Data Platform Release 4018

Cisco HX Data Platform Storage Cluster Specifications

Stretched Cluster (Availableon ESX Only)

Microsoft Hyper-VVMware ESXiNode

212121111121mdash2121MaxComputetoConvergedratio

NoExpansion

Requires Enterprise license

Requires uniform expansion across both sites

Guidelines and Limitationsbull Starting with release 40(2a) SCVM is no longer needed on a Compute node

bull HX REST API Access Token Management ndash Applications leveraging HX REST APIs should re-useaccess tokens when making API calls Once obtained using the AAA Obtain Access Token API accesstokens are valid for 18 days (1555200 seconds) In addition AAA enforces rate limiting on ObtainAccess Token API requests in a 15 minute window auth can be invoked (successfully) a maximum of5 times A user is allowed to create a maximum of 8 unrevoked tokens Subsequent call to auth willautomatically revoke the oldest issued token tomake room for the new token Amaximum of 16 unrevokedtokens can be present in system In order to prevent brute-force attacks after 10 consecutive failedauthentication attempts a user account is locked for a period of 120 secondsFor more information seeCisco HyperFlex Systems REST API Reference guide

HxConnect makes use of AAA Authentication REST API for login and the above rate limit applies toHxConnect also

Upgrade Guidelines

The following list is a highlight of critical criteria for performing an upgrade of your HyperFlex system

bull Unsupported HX Data Platform 17x 18x 20 21x 25x and 26x clustersmdashUsers from anyversion prior to 26(1a) must step through an intermediate version before upgrading to 40 or later releasesIf you need to upgrade your environment from a Cisco HyperFlex HX Data Platform software releasethat is past the last date of support to the latest suggested release on the Cisco Software Download sitesee Cisco HyperFlex Systems Upgrade Guide for Unsupported Cisco HXReleases For more informationsee the Software Advisory for CSCvq66867 WARNING Only Use HXDP 26(1e) Upgrade PackageWhen Upgrading From HXDP 18(1a)-18(1e)

bull Hypercheck Health Check UtilitymdashCisco recommends running this proactive health check utility onyour HyperFlex cluster prior to upgrade These checks provide early visibility into any areas that mayneed attention andwill help ensure a seamless upgrade experience For more information see the HyperFlexHealth amp Pre-Upgrade Check Tool TechNote for full instructions on how to install and run Hypercheck

bull vSphere 67 Software AdvisorymdashDo not upgrade to Cisco HX Data Platform Release 40(1a) whenrunning ESXi 67U1 EP06 (build 11675023) Do not upgrade to 67U1 EP06 (build 11675023) ifrunning Cisco HX Data Platform Release 40(1a) See the Software Advisory CSCvo56350 for furtherdetails

Release Notes for Cisco HX Data Platform Release 4019

Guidelines and Limitations

The software build version posted at release will override any other local versions

bull Required vCenter upgrademdashFor enhanced security Cisco HX Data Platform Release 35(1a) or laterrequires the use of TLS 12 Therefore vCenter must be upgraded to 60 U3f or later before upgradingto Cisco HX Data Platform Release 35 or later In addition ESXi should be upgraded as required tomeet HX Data Platform compatibility requirements

bull Minimum HXDP version for upgrademdashHXData Platform clusters running 26(1a) or later may upgradedirectly to 40 using the HX Connect UI

bull Cluster ReadinessmdashEnsure that the cluster is properly bootstrapped and the updated plug-in is loadedbefore proceeding Manual cluster bootstrap is required for upgrade from a pre-35 release

bull Cluster ReadinessmdashEnsure that the cluster is properly bootstrapped and the updated plug-in is loadedbefore proceeding Manual cluster bootstrap is required for HX releases earlier than 35(1a) For moreinformation see theManual Bootstrap Upgrade Process in the Cisco HyperFlex Systems Upgrade Guidefor VMware ESXi Release 40 Do not skip this cluster bootstrap step it is required for all upgradesuntil HX Release 35(1a) Auto bootstrap is supported beginning with HX release 35(1a) For moreinformation see the Auto Bootstrap Upgrade Process from HX Connect UI in the Cisco HyperFlexSystems Upgrade Guide for VMware ESXi Release 40

Manual bootstrap is not supported on Intersight clusters

bull Initiating Upgrade―Use the HX Connect UI or CLI stcli commands when upgrading from 25(1a)or later releases Use either the CLI stcli commands or the HX Data Platform Plug-in to the vSphereWeb Client when upgrading from a pre-25(1a) release The vCenter plug-in should not be used forupgrades starting with the 25(1a) release The vCenter plug-in should not be used for upgrades startingwith the 25(1a) release

If the current cluster version is at 35(1a) or above you do not need to use the stcli command Directupgrade to 40 is possible

bull Complete your Upgrade―The self-healing (or rebalance) capability is disabled temporarily during theupgrade window If the upgrade fails you should complete the upgrade as soon as possible

bull ESXi and HXDP Compatibility―Ensure your cluster is running a compatible version of ESXi basedon the running the HXData Platform version (see the section Software Requirements for VMware ESXi)ESXi compatibility is determined by the major version and update release of ESXi It is generally bestto upgrade HXDP and ESXi together if combining the upgrade operations into a single optimized rebootWhen running a split upgrade first upgrade the HX Data Platform then proceed to upgrade ESXi

bull If Upgrading to vSphere 65

bull Certain cluster functions such as native and scheduled snapshotsReadyClones and Enter or Exit HX Maintenance Mode will not operatefrom the time the upgrade is started until the HX Data Platform upgrade iscomplete

bull After upgrading ESXi using the offline zip bundle use the ESX ExitMaintenance Mode option The HX Exit Maintenance Mode option doesnot operate in the vSphere Web Client until the HX Data Platform upgradeis complete

Note

Release Notes for Cisco HX Data Platform Release 4020

Guidelines and Limitations

bull vSphere 60 UpgradesmdashUsers on vSphere 60 migrating to 65 upgrade components in the followingorder

1 Upgrade HX Data Platform and UCS firmware

2 Upgrade HX Data Platform and ESXi

3 Upgrade HX Data Platform only first then upgrade ESXi or UCS firmware or both

bull M4 Server Firmware UpgradesmdashUpgrade server firmware to ensure smooth operation and to correctknown issues Specifically newer SAS HBA firmware is available in this release and is recommendedfor long-term stability

bull Users are encouraged to upgrade to 31(3c) C-bundle or later whenever possible

bull Users running C-bundle versions before 31(2f) must upgrade server firmware by performing acombined upgrade of UCS server firmware (C-bundle) to 31(3c) or later and HX Data Platform to25 Do not split the upgrade into two separate operations

bull If the cluster is already on 31(2f) C-bundle or later you may perform an HX Data Platform onlyor combined upgrade as required

bull M5 Server Firmware UpgradesmdashM5 generation servers must run firmware version 32(2d) or later

bull Firmware Downgrades mdashDowngrading UCSM from the HX-installer is not supported

bull M4M5 Mixed DomainsmdashA mixed domain occurs when a new separate M5 cluster is installed underthe same UCS domain that contains existing M4 clusters Under these conditions orchestrated UCSserver firmware upgrade will not operate until Cisco HX Data Platform Release 26 or later is installedon the M4 clusters Therefore it is best practice to first upgrade UCS server firmware to the latest 31(3)or 32(2) patch release before adding a new M5 cluster to the existing UCS domain Additionally any17 HX Data Platform clusters must first be upgraded before adding any new M5 clusters to the samedomain

bull Maintenance WindowmdashIf upgrading both HX Data Platform and UCS firmware you can select eithera combined or split upgrade through the vSphere HX Data Platform Plug-in depending on the length ofthe maintenance window Cisco UCSManager infrastructure upgrade is only supported using AutoInstalland the direct server firmware upgrade should be performed only through the upgrade orchestrationframework provided by the HX Data Platform Plug-in

bull Unsupported Self-Encrypting Drives (SEDs)mdashIf adding or replacing self-encrypting drives (SEDs)that have been recently qualified in newer versions of HX Data Platform insert the new drives only afterupgrading HX Data Platform to a compatible version All drives must be SED drives mixing SED andnon-SED is not supported

bullbull Enabling External Host AccessmdashWith Cisco HX Data Platform Release 40(1a) port 445 on themanagement network is blocked for enhanced security Note that prior to 40 port 445 port was openenabling external host access If you are upgrading to 40(1a) from a prior release and would like tocontinue external host access you can use a utility to open select hosts For more information aboutenabling external host access see the ConfiguringHyperFlex Share to SCVMM section in the InstallationGuide for Microsoft Hyper-V

Release Notes for Cisco HX Data Platform Release 4021

Guidelines and Limitations

Mixed Cluster Expansion Guidelinesbull Expanding existing M4 cluster with M5 converged nodes is supported

bull Expanding existing M5 cluster with M4 converged nodes is not supported

bull Expanding existing mixed M4M5 cluster with M4 or M5 converged nodes is supported

bull Adding any supported compute-only nodes is permitted with all M4 M5 and mixed M4M5 clustersusing the HX Data Platform 26 or later Installer Some example combinations are listed here manyother combinations are possibleExample combinationsExpand mixed M4M5 cluster with compute-only B200 C220 C240 M4M5Expand M4 cluster with compute-only B200 M5 C220 M5 C240M5

bull Only expansion workflow is supported to create a mixed cluster Initial cluster creation with mixedM4M5 servers is not supported

bull All M5 servers must match the form factor (220240) type (HybridAF) security capability (Non-SEDonly) amp disk configuration (QTY capacity and non-SED) of the existing M4 servers

bull HX220-M5 will use a maximum of 6 capacity disks (2 disk slots to remain empty) when mixedwith HX220-M4

bull HX Edge SED LFF Hyper-V and Stretched Clusters do not support mixed M4 and M5 clusters

Security FixesThe following security issues are resolved

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-13132 CVE-2019-9924CSCvq6313840(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-11719 CVE-2019-11727CVE-2019-11729

CSCvq7124040(2a)

Release Notes for Cisco HX Data Platform Release 4022

Mixed Cluster Expansion Guidelines

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-1125CSCvr0633940(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2015-9383 CVE-2018-14498CVE-2018-20406 CVE-2018-20852CVE-2019-10160 CVE-2019-13117CVE-2019-13118 CVE-2019-14287CVE-2019-14973 CVE-2019-15903CVE-2019-17546 CVE-2019-18197CVE-2019-18218 CVE-2019-5010CVE-2019-5094 CVE-2019-5481CVE-2019-5482 CVE-2019-9636CVE-2019-9740 CVE-2019-9947CVE-2019-9948

CSCvs0609440(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2017-13168 CVE-2017-18174CVE-2017-18216 CVE-2018-10876CVE-2018-10877 CVE-2018-10878CVE-2018-10879 CVE-2018-10880CVE-2018-10881 CVE-2018-10882CVE-2018-10902 CVE-2018-10938CVE-2018-12233 CVE-2018-12896CVE-2018-13053 CVE-2018-13094CVE-2018-13096 CVE-2018-13405CVE-2018-13406 CVE-2018-14609CVE-2018-14617 CVE-2018-14633CVE-2018-14734 CVE-2018-15572CVE-2018-15594 CVE-2018-16276CVE-2018-16658 CVE-2018-17182CVE-2018-17972 CVE-2018-18021CVE-2018-18690 CVE-2018-18710CVE-2018-6554 CVE-2018-6555CVE-2018-9363

CSCvp6501940(2a)

This vulnerability is due to insufficientHTML iframe protection An attackercould exploit this vulnerability bydirecting a user to an attacker-controlledweb page that contains amaliciousHTMLiframe A successful exploit could allowthe attacker to conduct clickjacking orother clientside browser attacks

NACSCvo9851640(2a)

Release Notes for Cisco HX Data Platform Release 4023

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerability is due to insufficientauthentication for the statistics collectionservice An attacker could exploit thisvulnerability by sending properlyformatted data values to the statisticscollection service of an affected deviceA successful exploit could allow theattacker to cause the web interfacestatistics view to present invalid data tousers

NACSCvj9558440(2a)

The vulnerability is due to insufficientCSRF protections for the web UI on anaffected device An attacker could exploitthis vulnerability by persuading a user ofthe interface to follow a malicious linkA successful exploit could allow theattacker to perform arbitrary actions withthe privilege level of the affected user

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvp2434340(2a)

Vulnerabilities with a version of cJSONidentified by CVE-2019-11834 andCVE-2019-11835

For more information see the relatedCisco Security Advisory

CVE-2019-11834 CVE-2019-11835CSCvq1994940(2a)

HX ESXi image patches for MCEPSCand TAA vulnerabilities with VMwareESXi identified by CVE-2018-12207 andCVE-20190-11135

For more information see the relatedCisco Security Advisory

CVE-2018-12207 CVE-2019-11135CSCvr5439840(2a)

CP networking vulnerabilities affectingthe Linux kernel identified byCVE-2019-11477 CVE-2019-11478CVE-2019-11479

For more information see the relatedCisco Security Advisory

CVE-2019-11477 CVE-2019-11478CVE-2019-11479

CSCvq1954640(2a)

Qualification of Microsoft securitypatches for vulnerabilities withMicrosoftHyper-V hypervisor identified withCVE-2018-12207 CVE-2019-11135

For more information see the relatedCisco Security Advisory

CVE-2018-12207 CVE-2019-11135CSCvr5439940(2a)

Release Notes for Cisco HX Data Platform Release 4024

Security Fixes

DescriptionCVEDefect IDRelease

OpenSSH vulnerabilities identified byCVE-2019-6111

For more information see the relatedCisco Security Advisory

CVE-2016-10708 CVE-2018-15473CVE-2018-20685 CVE-2019-6109CVE-2019-6111

CSCvp7646340(2a)

Multiple vulnerabilities associated withOpenSSL and LibNSS

For more information see the relatedCisco Security Advisory

CVE-2016-2105 CVE-2016-2106CVE-2016-2107 CVE-2016-2109CVE-2016-2176 CVE-2016-2177CVE-2016-2178 CVE-2016-2179CVE-2016-2180 CVE-2016-2181CVE-2016-2182 CVE-2016-2183CVE-2016-6302 CVE-2016-6303CVE-2016-6304 CVE-2016-6305CVE-2016-6306 CVE-2016-6307CVE-2016-6308 CVE-2016-7055CVE-2016-8610 CVE-2017-3731CVE-2017-3732 CVE-2017-3735CVE-2017-3736 CVE-2017-3737CVE-2017-3738 CVE-2018-0495CVE-2018-0732 CVE-2018-0734CVE-2018-0735 CVE-2018-0737CVE-2018-0739 CVE-2018-12384CVE-2018-12404 CVE-2018-5407CVE-2019-1559

CSCvp6667940(2a)

libpng vulnerability identified byCVE-2016-10087

For more information see the relatedCisco Security Advisory

CVE-2016-10087CSCvp6655540(2a)

Vulnerabilities associated with libjpg andlibpng

For more information see the relatedCisco Security Advisory

CVE-2014-9092 CVE-2016-3616CVE-2017-15232 CVE-2018-11212CVE-2018-11213 CVE-2018-11214CVE-2018-1152 CVE-2018-13785

CSCvp3458640(2a)

Vulnerabilities associated with Glibidentified by CVE-2018-16428 andCVE-2018-16429

For more information see the relatedCisco Security Advisory

CVE-2018-16428 CVE-2018-16429CSCvp3120740(2a)

Multiple vulnerabilities associated withcurl expat python 27 python 35 36and 37 freetype and giflib

For more information see the relatedCisco Security Advisory

CVE-2019-15133 CVE-2019-15903CVE-2019-5010 CVE-2019-5481CVE-2019-5482 CVE-2019-9636CVE-2019-9740 CVE-2019-9947CVE-2019-9948

CSCvr3690340(2a)

Release Notes for Cisco HX Data Platform Release 4025

Security Fixes

DescriptionCVEDefect IDRelease

Multiple third party softwarevulnerabilities

For more information see the relatedCisco Security Advisory

CVE-2019-14379 CVE-2019-12384CVE-2019-14439

CSCvq9203240(2a)

Vulnerabilities associated with elfutils

For more information see the relatedCisco Security Advisory

CVE-2018-16062 CVE-2018-16402CVE-2018-16403 CVE-2018-18310CVE-2018-18520 CVE-2018-18521CVE-2019-7149 CVE-2019-7150CVE-2019-7665

CSCvq4325040(2a)

Vulnerabilities associated with vim

For more information see the relatedCisco Security Advisory

CVE-2017-5953 CVE-2019-12735CSCvq4323040(2a)

Vulnerabilities associated with sqlite3

For more information see the relatedCisco Security Advisory

CVE-2016-6153 CVE-2017-10989CVE-2017-13685 CVE-2017-2518CVE-2017-2519 CVE-2017-2520CVE-2018-20346 CVE-2018-20505CVE-2018-20506 CVE-2019-8457CVE-2019-9936 CVE-2019-9937

CSCvq4321340(2a)

Vulnerabilities associated with glib20

For more information see the relatedCisco Security Advisory

CVE-2018-20843CSCvq4320940(2a)

Vulnerabilities associated with expat

For more information see the relatedCisco Security Advisory

CVE-2018-20843CSCvq4320540(2a)

Vulnerabilities associated with bzip2

For more information see the relatedCisco Security Advisory

CVE-2016-3189 CVE-2019-12900CSCvq4319440(2a)

Mulitple Vulnerabilities associated wtihNodeJS

For more information see the relatedCisco Security Advisory

CVE-2018-12115 CVE-2018-0734CVE-2018-5407 CVE-2018-12120CVE-2018-12121 CVE-2018-12122CVE-2018-12123 CVE-2018-12116CVE-2019-5737 CVE-2019-5739

CSCvq1069440(2a)

Vulnerabilities associated with PalletsJinja strformat_map and identified byCVE-2019-10906 and CVE-2016-10745

For more information see the relatedCisco Security Advisory

CVE-2019-10906 CVE-2016-10745CSCvq1038840(2a)

Release Notes for Cisco HX Data Platform Release 4026

Security Fixes

DescriptionCVEDefect IDRelease

Multiple Security Vulnerabilities

For more information see the relatedCisco Security Advisory

CVE-2014-8501 CVE-2014-9939CVE-2015-9262 CVE-2016-10087CVE-2016-2226 CVE-2016-4487CVE-2016-4488 CVE-2016-4489CVE-2016-4490 CVE-2016-4491CVE-2016-4492 CVE-2016-4493CVE-2016-6131 CVE-2018-10963CVE-2018-13785 CVE-2018-17100CVE-2018-17101 CVE-2018-18557CVE-2018-18661 CVE-2018-7456CVE-2018-8905

CSCvq0917840(2a)

Vulnerabilities associated withlibseccomp

For more information see the relatedCisco Security Advisory

CVE-2019-9893CSCvq0756840(2a)

Vulnerabilities associated withgdk-pixbuf

For more information see the relatedCisco Security Advisory

CVE-2017-12447CSCvq0675540(2a)

Vulnerabilities associated withpython-crypto

For more information see the relatedCisco Security Advisory

CVE-2018-6594CSCvp9381740(2a)

Vulnerabilities associated with wget

For more information see the relatedCisco Security Advisory

CVE-2018-20483 CVE-2019-5953CSCvp8672140(2a)

Multiple vulnerabilities associated withPython Crypto

For more information see the relatedCisco Security Advisory

CVE-2018-6594CSCvp6674840(2a)

Vulnerabilities associated withsensible-utils

For more information see the relatedCisco Security Advisory

CVE-2017-17512CSCvp6673440(2a)

Multiple vulnerabilities associated withPython

For more information see the relatedCisco Security Advisory

CVE-2018-1000030CSCvp6668940(2a)

Release Notes for Cisco HX Data Platform Release 4027

Security Fixes

DescriptionCVEDefect IDRelease

Multiple patch vulnerabilities identifiedby CVE-2016-10713

For more information see the relatedCisco Security Advisory

CVE-2016-10713CVE-2018-1000156 CVE-2018-6951

CSCvp6667240(2a)

Vulnerabilities associatedwith Little CMSidentified by CVE-2016-10165

For more information see the relatedCisco Security Advisory

CVE-2016-10165 CVE-2018-16435CSCvp6666440(2a)

Vulnerabilities associated with TenableScan identified byCVE-2011-5325CVE-2019-11068CVE-2016-7076CVE-2017-1000368

For more information see the relatedCisco Security Advisory

CVE-2011-5325 CVE-2016-7076CVE-2017-1000368CVE-2019-11068

CSCvp6474640(2a)

Vulnerabilities associated with libxml2294 and earlier as used in XMLSec1223 and earlier and other products andidentified by CVE-2016-9318

For more information see the relatedCisco Security Advisory

CVE-2016-9318 CVE-2017-16932CVE-2017-18258 CVE-2018-14404CVE-2018-14567

CSCvp3479240(2a)

Vulnerabilities associated with LFTPRemote File Names Unauthorized AccessVulnerability and identified byCVE-2018-10916

For more information see the relatedCisco Security Advisory

CVE-2018-10916CSCvp2926640(2a)

Vulnerabilities associated withCryptographyDeprecationWarning signerand verifier have been deprecated

For more information see the relatedCisco Security Advisory

CVE-2018-7750CSCvo3409740(2a)

Multiple Vulnerabilities in python python27 35

For more information see the relatedCisco Security Advisory

CVE-2019-16056CSCvr3979340(2a)

Release Notes for Cisco HX Data Platform Release 4028

Security Fixes

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toexecute commands as the root user

The vulnerability is due to insufficientinput validation An attacker could exploitthis vulnerability by connecting to thecluster service manager and injectingcommands into the bound process Asuccessful exploit could allow the attackerto run commands on the affected host asthe root user

Cisco has released software updates thataddress this vulnerability There areworkarounds that address thisvulnerability

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvq2417640(1b)35(2g)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

The vulnerability is due to an unprotectedlistening interface An attacker couldexploit this vulnerability by connectingto the listening interface and injectingcommands to the bound process Anexploit could allow the attacker to runcommands on the affected host as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvj9560640(1b)35(2g)

Release Notes for Cisco HX Data Platform Release 4029

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerabilities associated with theJVM18U121memory leak found duringVC alarms (concurrent 40 calls usingREST API)

CVE-2017-10053 CVE-2017-10067CVE-2017-10074 CVE-2017-10078CVE-2017-10081 CVE-2017-10087CVE-2017-10089 CVE-2017-10090CVE-2017-10096 CVE-2017-10101CVE-2017-10102 CVE-2017-10107CVE-2017-10108 CVE-2017-10109CVE-2017-10110 CVE-2017-10111CVE-2017-10115 CVE-2017-10116CVE-2017-10118 CVE-2017-10135CVE-2017-10176 CVE-2017-10193CVE-2017-10198 CVE-2017-10243CVE-2017-10274 CVE-2017-10281CVE-2017-10285 CVE-2017-10295CVE-2017-10345 CVE-2017-10346CVE-2017-10347 CVE-2017-10348CVE-2017-10349 CVE-2017-10350CVE-2017-10355 CVE-2017-10356CVE-2017-10357 CVE-2017-10388CVE-2017-3509 CVE-2017-3511CVE-2017-3526 CVE-2017-3533CVE-2017-3539 CVE-2017-3544CVE-2018-2579 CVE-2018-2582CVE-2018-2588 CVE-2018-2599CVE-2018-2602 CVE-2018-2603CVE-2018-2618 CVE-2018-2629CVE-2018-2633 CVE-2018-2634CVE-2018-2637 CVE-2018-2641CVE-2018-2663 CVE-2018-2677CVE-2018-2678 CVE-2018-2783CVE-2018-2790 CVE-2018-2794CVE-2018-2795 CVE-2018-2796CVE-2018-2797 CVE-2018-2798CVE-2018-2799 CVE-2018-2800CVE-2018-2814 CVE-2018-2815CVE-2018-2952 CVE-2018-3136CVE-2018-3139 CVE-2018-3149CVE-2018-3150 CVE-2018-3169CVE-2018-3180 CVE-2018-3183CVE-2018-3214 CVE-2019-2422

CSCvo8899740(1b)

Tomcat and Nginx logs are not beingcollected in the support bundle generatedthrough HX Connect in Release 35

NACSCvm5803140(1b)

The vulnerabilities associated with thelibmspack software package versionincluded in Cisco HX Data Platform

CVE-2018-18584

CVE-2018-18585

CSCvn3511940(1a)

Release Notes for Cisco HX Data Platform Release 4030

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerabilities associated withFasterXML Jackson-Databind TimeValueField Denial of Service

CVE-2018-14719

CVE-2018-14720

CVE-2018-1000873

CVE-2018-14721

CVE-2018-19360

CVE-2018-19362

CVE-2018-19361

CVE-2018-14718

CSCvn8228240(1a)

The vulnerabilities associated with theOpenSSL Protocol software packageversion included in Cisco HX DataPlatform

CVE-2013-3587CSCvo0505440(1a)

The vulnerabilities associated withThird-Party Software Denial of Service

CVE-2018-16487

CVE-2018-19361

CSCvo2781840(1a)

The vulnerabilities associated withLinuxkernel for Ubuntu 1710

CVE-2018-1092

CVE-2018-7492

CVE-2018-8087

CVE-2018-1068

CVE-2018-8781

CSCvm5314935(2a)

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)25(1d)stcli services dns remove should remove the DNS serverinfo from the interface files

CSCvs70967

40(2b)40(1b)HX 40(1b) new deployment from Intersight productioncloud

User receives following error while enabling CSI(Kubernetes integration)

Failure occurred during volume_access_enable Errorwas Nonzero exit code 1

CSCvt10522

40(2b)40(2a)Receive the following alertevent in HX Connect HXController VM HOSTNAME one or more configuredDNS servers not responding

CSCvt13947

Release Notes for Cisco HX Data Platform Release 4031

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)40(2a)API UI CLI show few Drive slots Empty after upgradeto 402a

The drives are seen in the sysmtool lsscsi and stclicommand outputs

The cluster is healthy with no errors

CSCvt14914

40(2b)35(2d)After successfully changing the DNS server on the HXcontroller we still can see the original DNS entry thatwas added during the deployment

CSCvs69154

40(2b)35(2e)HyperFlex Connect and VCenter show APD alarms forHX and non HX datastores This implies there is an issuewith HyperFlex when there may not be

CSCvs30080

40(2b)40(2a)An issue where all the disks in the node get locked afterrebooting the node

We successfully unlocked using sed-clientsh -Ucommand but wanted to test with another reboot anddrives locked again

CSCvs74286

40(2b)40(2a)When running stcli license commands on HyperFlexthe following error is seen

rootSpringpathControllertmp stcli license show all

Show smart licensing failed

Smart Agent is not ready please wait a minute and tryagain

CSCvt13929

40(2b)40(2a)After upgrading to HX 40(2a) the first RoboEdge nodethat gets rebooted then ignores persistent drivesPIDHX-HD24TB10K4KN

Cluster will be in degraded state and have reducedcapacity

CSCvt20144

40(2b)35(2b)Zookeeper fails to start while Exhibitor is runninghowever echo srvr returns nothing

CSCvs21562

40(2b)40(2a)When performing an HyperFlex upgrade a validationwarning may occur due to the host not having EnterprisePlus or Enterprise hypervisor licensing

Upgrade Validation Warning

ESXi host esx1labtest should be configured withVMware Enterprise license for upgrade to continue

CSCvs91787

Release Notes for Cisco HX Data Platform Release 4032

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)35(2g)Cluster expansion fails at Config Installer stage whenthe root and admin password for the storage controller(SCVM) are different

CSCvs69317

40(2b)35(2g)Panic while upgrading ESXiCSCvt06983

40(2b)40(2a)A cluster node running may hang in the Linux kernelThis is classified as an oops and a deviation from theexpected behavior

CSCvs54285

40(2b)35(2g)Rebalance failing on 10+10 node HX 35(2g) stretchcluster

CSCvs69007

40(2b)35(2d)The cluster becomes inaccessible to the IOVisorCSCvr54687

40(2b)40(2a)Panic on the storage controllerCSCvt61297

40(2b)40(2a)The size of support bundle is very large when collectedwith storfs-support command

CSCvt63306

40(2b)40(2a)storfs crash associated withFileSystemUsageWarningEvent orFileSystemUsageAlertEvent due to high usage ofvarstv above 80

CSCvt72807

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

40(2a)35(2d)In normal scenarios HX upgrade gets pendingACK afterupdating service profile In this case UCS only upgradewas stalled because after updating service profile HXupgrade was not getting pending ACK and it was alwayswaiting state

CSCvo39912

40(2a)35(1a)When using motherBoardReplace-12 to clean ZK fromold stNodepNodes resulted in unmounted datastore andmaking the size of the Hyperflex Datastores 0 resultingin all VMs in the cluster going offline

CSCvq39471

40(2a)26(1e)Upgrading cluster and getting error

Failed upgrade validations Checking vCenterconfiguration Reason Upgrade validations failed DRSFault Insufficient resources to satisfy configured failover

CSCvm77294

Release Notes for Cisco HX Data Platform Release 4033

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)The cluster expand fails on a node with DR replicationconfigured

When a HX cluster which has DR replication configuredis expanded we see the installer UI pulling in thereplicationVLAN information instead of themanagementVLAN information Even if we change that informationto the correct mgmt VLAN id and name it it does notseem to work as the node is configured with the VLANof the replicationVLAN in ESXi This leads to the failureof the node add with the host unreachable error

CSCvo70650

40(2a)35(1a)Customer reported that server didnt complete thefirmware upgrade automatically one by one

The user manually needs to put the host into maintenancemode and they manually acknowledged the pendingrequests to finish the UCS firmware upgrade

Maintenance policy is set to default (User-ack) as perthe design

CSCvo91624

40(2a)35(1i)When the cluster is in Failed state and stcli noderemove is attempted the output shows successful thoughthe node failed to get removed from the cluster

CSCvo93017

40(2a)35(1a)HyperFlex cluster upgrade may fail during validationwith a DRS Validation failed error

CSCvp31021

40(2a)35(2b)HX cluster expansion will fail with the following errormessage

MAC address pool configuration failure150[ErrorDescription] bad address block range definitioncollision

CSCvp58318

40(2a)35(2b)Memory usage by carbon cacheCSCvq34873

40(2a)35(2g)Upgrading ESXi cluster fails with error Nodemaintenance mode failed

CSCvr03240

40(2a)35(2e)storfs process does not automatically start on Exit HXMaintenance Mode or other tasks which restart StorageController

CSCvr88978

40(2a)35(2a)Post install script fails with message

Failed to execute ipmitool on HX node

CSCvp10707

40(2a)40(1a)HyperFlex expansionworkflow doesnt pull VLANnamecorrectly

CSCvp46539

Release Notes for Cisco HX Data Platform Release 4034

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)During HX cluster deployment - Validate ClusterCreation phase - HX Installer might fail with the error

fromvarlogspringpathstDeploylog

CSCvq45087

40(2a)35(2b)HX Installer fails to login to the SOL for configuringESX hosts

CSCvq91380

40(2a)35(2d)If DRS is enabled with configuration parameters otherthen fully automated ie manual or partially automatedthan this will change to fully automated during clusterexpansion

CSCvr44222

40(2a)35(1a)One of the validation task name has a typo The taskname is Validating node for mixed of SED and non-SEDdisks

It should be Validating node for mix of SED andnon-SED disks

CSCvp82175

40(2a)30(1i)This combination of using smaller NVMe drives is notsupported in expand operation For example if the drivesare only 375GB and they cannot be added to an existingcluster with larger caching SSD

CSCvo12359

40(2a)35(1a)HyperFlex includes a version of OpenSSL that is affectedby the vulnerabilities identified by the followingCommon Vulnerability and Exposures

CVE-2018-0495

CSCvp66679

40(2a)30(1e)SDK invoke exception when taking a native snapshot ona VM with GPU pci passthrough attached

CSCvo00511

40(2a)30(1i)Node replacement script fails due to EAM errorCSCvo62867

40(2a)35(1a)While pairing a cluster (HX release gt= 35) and cluster(HX release lt 35) remote replication network test failson the cluster (HX release lt 35)

CSCvo79760

40(2a)30(1b)Fixed the issue in the latest support-workflow bundleThis should not be hit anymore

CSCvo87061

40(2a)35(2a)On a 3 node cluster the MBreplace script fails as thescript looks for HEALTHY cluster status

CSCvo87080

40(2a)35(2a)MbReplace script (tar) hangs when running on 352b or40

CSCvp12359

Release Notes for Cisco HX Data Platform Release 4035

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)HX replication cleanup failing with error NA 35(2a) -INFODR state is not clean

CSCvp63958

40(2a)40(1a)The check_stig_parameters API incorrectly shows thecompliance state of the cluster as not STIG compliant

CSCvp66277

40(2a)35(2c)The snapshot creation on CBT enabled VM fails witherror Failed in vmreparent vmkfstools clone1

CSCvq06952

40(2a)35(2a)HyperFlex GUI when setting up remote DR-replicationpartnerpeer should show remote replication partner TCPport reachability test results

This will provide quick upfront connectivity results thatfrequently takes customer a while to manually check andconfirm

If reachability tests fail an info window and HX alertshould trigger that list the destination peer IP and thespecific port(s) that could not be reache

CSCvo60587

40(2a)35(2b)If you have more than one IP pool configured for areplication network and local replication network testsare failing you might be running into this issue

CSCvr67130

40(2a)35(2a)Skip task option in MBreplace and Software redeployscripts The scripts should have the option to skip tasksas user inputs and not by modifying the json file

CSCvp05204

40(2a)40(1a)Error noticed on SCVM Console print_req_error IOerror dev fd0 sector 0 on HyperV

It doesnt come up on SSH session

CSCvq34357

40(2a)30(1d)The VMNetwork switch is already created as part of theinstall process

Ideally the ip address assignment should be done duringthe initial install process as well No post install stepsshould be required altrnatively it could be done as partof a post install script

CSCvk36222

40(2a)25(1c)storfs service impacted during HX upgrade from HX21(1) to 25(1c)

HX node which is not being upgraded storfs servicestopped causing outage

CSCvg53223

40(2a)35(1b)HX Stretch Cluster Witness VM reverts to DHCP onreboot

CSCvp37536

40(2a)35(2b)HyperFlex Stretch Cluster Witness does not getprogrammed with an NTP server

CSCvp46578

Release Notes for Cisco HX Data Platform Release 4036

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2b)HyperFlex cluster does not heal after maintenance onnodeserver

CSCvr18528

40(2a)40(2a)High percentage of packet loss on the eth1 data interface9K MTU

CSCvr97089

40(2a)35(2f)The HyperFlex cluster healing state is stuck at 87 andis unable to progress

CSCvr16760

40(2a)40(2a)In some circumstances two replications can occur forthe same VM

From the GUI the replication status for this VM willshow stuck in an In Progress state but there are nojobs stuck in replication layer Subsequent Replicationsshould succeed

CSCvq22898

40(2a)35(2a)After an unexpected power outage the hyperflex clusteris unable to come up correctly

All processes are running time is synced and all nodescan vmkping each other as required

CSCvp13990

40(2a)40(1a)When specifying IP address instead of node ID stclinode maintenanceMode commands fail

CSCvp33657

40(2a)40(1a)2 node edge cluster if experiencing Intersightarbitratorconnectivity issues should NOT allowHXMaintenancemode to initiate as this will bring the cluster down

CSCvr31573

40(2a)35(2c)Old files in varsupportZKTxnlog are not purged withthe daily zklog-cleanup cron job

CSCvr89066

40(2a)35(2a)When a node is in MaintenanceMode any disk removalor replacement will be reflected in UI only after the nodeis brought back frommaintenancemode This is becausestorfs is not running on the node in maintenance andwill not be able to detect disk activities until it is broughtout of Maintenance Mode

CSCvo86431

40(2a)35(2c)Cluster might go offline and stop serving IO requestsrootcvm~ stcli cluster storage-summary--detailGet cluster storage summary failedjavanetConnectException Connection refused19216814210010207

CSCvr01645

40(2a)35(2d)A stretch cluster deployment fails with a messageFormatting nodes Node in Use on the Hyperflex installerand crmZoneType value is seen as 1 in storfscfg oroptspringpathconfigstretchtunes

CSCvq80340

Release Notes for Cisco HX Data Platform Release 4037

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)30(1c)During upgrade we upload the upgrade packagedownloaded from Ciscocom Upgrade packages are offile type tgz hxconnect acceptsstorfs-packages-401a-33028tar file a different type ofcompression and allows upgrade to start and fail whenit cannot decompress the file for bootstrap and scvmclientupgrade on ESXi

CSCvq63888

40(2a)40(2a)While attempting to install hyperflex via intersight thefollowing error is seen

Failed in Task Add hosts to vCenter Cluster with ErrorTry adding hosts manually to vCenter and retry Failedto get addressFailed to host by namelookupHostnamecompanycom on 000053 read udp12700145430-gt12700153 read connection refused

CSCvr52098

40(2a)35(2a)CoreAPI call may time out for cluster management APIwhile doing cluster create validation

CSCvo92952

40(2a)35(2d)Validation fails with Mixed mode expansion checkdue to Enhanced vMotion Compatibility Incompatibilitywhen the ESXi version is different

CSCvq95460

40(2a)30(1i)This failure is seen in installer when custom workflowis used hypervisor configuration + deploy (clean diskpartition=no) at Installing software packages on storagecontroller VM

CSCvr66309

40(2a)35(2d)The HX Installer apparently replaced a comma bysemicolon on the VLAN ID range while exporting theconfiguration to a JSON file

CSCvq93831

40(2a)40(1a)When using unsupported browsers HX Connect userswill not be able to login These users see an errorindicating unauthorized user

CSCvp22693

Release Notes for Cisco HX Data Platform Release 4038

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)40(1a)A vulnerability in the web-based management interfaceof Cisco HyperFlex Software could allow anunauthenticated remote attacker to conduct a cross-siterequest forgery (CSRF) attack on an affected system

The vulnerability is due to insufficient CSRF protectionsfor the web UI on an affected device An attacker couldexploit this vulnerability by persuading a user of theinterface to follow a malicious link A successful exploitcould allow the attacker to perform arbitrary actions withthe privilege level of the affected user

Cisco has released software updates that address thisvulnerability There are no workarounds that address thisvulnerability

This advisory is available at the following link

httpstoolsciscocomsecuritycentercontentCiscoSecurityAdvisorycisco-sa-20190807-hypflex-csrf

CSCvp24343

40(2a)35(2d)Add message to HX Connect Progress flow to notAcknowledge Pending Activities and reboot the serverson UCSM

HX Connect is doing a controlled rolling server upgradein the background

CSCvq22844

40(2a)40(1b)Compute Node expand operation fails due to incorrectcluster name

CSCvr43786

40(2a)26(1e)RF-2 cluster shutdown during data resynchronizationafter a node failure (non storfs) followed by multipledisk read failures and a HardBlacklist of a disk

CSCvp41241

40(2a)40(1a)HyperFlex Edge nodes do not properly set the ESXihostname during deployment

CSCvo13143

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4039

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4040

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1b) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401b-33133p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4041

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4042

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1b)40(1a)While running the HyperFlex installer with WindowsServer Hyper-V cluster creation process fails with theerror Failure occurred during ClusterCreation process Unable to post thecontent to the downstream This symptomis encountered while deploying HyperFlex cluster withHyperFlex nodes configuredordered with Cisco VIC1457 MLOM (PID HX-MLOM-C25Q-04) andWindows Server Datacenter or Core with Hyper-Vversion 2016 or 2019

CSCvp64140

Release Notes for Cisco HX Data Platform Release 4043

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)35(2b)Adding Micron 5200 drive to a cluster fails to increasecluster capacity

CSCvo69067

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4044

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4045

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1a) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401a-33028p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4046

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4047

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1a)30(1d)Cluster instability when disks of different sector sizeplaced in HX node

CSCvk17250

40(1a)

35(2c)

35(1a)When logged in using a local HX user account insteadof a Virtual Center account an error message appearsintermittently indicating Virtual Centerunreachable or Resource informationcannot be updated when VC is reachable

CSCvo36198

40(1a)

35(2b)

30(1d)HXDP does not work with EMC RecoverPoint - needsto support VMware API (FSS-Readdir)

CSCvk38003

Release Notes for Cisco HX Data Platform Release 4048

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)

35(2a)

30(1i)

25(1c)

25(1d)

Zookeeper (Exhibitor) process on HyperFlex StorageController VMs may fail to respond to requests whenvarzookeeper has low or no free space

CSCvm90352

40(1a)26(1c)Use Asynchronous consolidation for HX SnapshotsCSCvn02151

40(1a)35(2b)HX Quiesced Snapshot for Backup VendorsCSCvo90713

40(1a)35(2a)The cluster creationcluster expansion workflow stopswith the following error message at the validation step

FIRMWARE-Check UCSC-SAS-M5HD

FIRMWARE-Check UCSC-SAS-M5HD Required00000029000000320000003500000050 Found00000058

Action Needed Update the Controller Firmware toRequired Version

CSCvn17787

35(2b)30(1d)Kernel migration fails when a local datastore is searchedfor in the ESXi host

CSCvn73127

35(2a)26(1b)A node shuts down when two disks are replaced (acaching disk and another capacity disk) where thecapacity disk is inserted first and then the caching diskis inserted

CSCvk46364

Hyper-V

40(1a)35(2a)Cluster expansion may fail with an error code 500 -operation timed out

CSCvn28721

35(2b)35(2a)During upgrade remove the VLAN on team created foruser vSwitch During fresh installation only one VLANtag is set to the vSwitch and team although multipleVLANs were entered

CSCvn54300

35(2b)35(2a)While upgrading a Hyper-V cluster on account of a rarerace condition between the stUpgradeService andZookeeper servers the upgrade orchestration throws anupgrade validation error and the upgrade process isaborted

CSCvn60486

Release Notes for Cisco HX Data Platform Release 4049

Resolved Caveats in Release 40(1a)

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Startrestart EAM service invCenter

When EAM service is not runningvCenter registration is failing

CSCvt55712

35(2a)Contact Cisco TAC if you believeyou are seeing similar conditions

HyperFlex Stretch Cluster saw poorapplication performance due to writelatency cluster remained unhealthy andrebalance was stuck

This defect is currently being used forinvestigative purposes

CSCvt45344

35(2g)Remove log files fromvarlogspringpath folder when thelog files are not getting rotatedStop the relevant service removelarge log files fromvarlogspringpath folder andrestart the service

varstv folder in Controller VM maybecome full

CSCvt36374

40(2b)Restarting the stMgr will initializeit again

In a Cluster configured with DR networkStMgr may not get initialized and stuckin a deadlock while enabling the IPTablerules (can be verified from the stMgrlog)

CSCvt89709

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Click recover for the selectedVM again under the standalonemode This will move the VM torecovered state

When the user recovers a VM protectedin a group it along with other VMs in thegroup move to recovering state instandalonemode The selectedVM shouldmove to recovered state

CSCvs75553

35(2e)Clean up the fail over cluster andthen recreate the fail over clusterNo need to touch the HX storagecluster

Hyper-V When replicated DC was usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2d)40(1a)

Keep cluster utilization as low aspossible To avoid read-only statekeep below 92

HyperFlex clusters storage utilization isabove 92 usable capacity

CSCvr20922

Release Notes for Cisco HX Data Platform Release 4050

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1b)NAAll the disks in the node got locked afterrebooting the node

We successfully unlocked usingsed-clientsh -U command but wantedto test with another reboot and driveslocked again

CSCvs74286

35(2g)To recover the VM copy over thedata disks and attach them to thenew VM

After enable encryption HX connectencryption status showing Caution butall of the disks enable encryptionsuccessful

-Self Encrypting Drives Service isrunning on all of the nodes

-There is no error message of encryption

- USB0 interface is up

-All SED disks showing supported 1enabled 1 locked 0

CSCvs93245

40(1b)Enable kernelpanic_on_oops as apersistent configuration This willcause the node to panic and rebootimmediately

A cluster node running HX release40(1b) may hang in the Linux kernelThis is classified as an oops and adeviation from the expected behavior

CSCvs54285

Release Notes for Cisco HX Data Platform Release 4051

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Enabling K8iscsi stack in Hx clusterCSCvs41324

Release Notes for Cisco HX Data Platform Release 4052

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Steps to enable K8 on HX cluster

Perform following steps on ALLthe controller VMs

1 Updateetcinitscvmclientconf toenable tune iscsiEnable totrue

sed -iesiscsiEnable=falseiscsiEnable=trueetcinitscvmclientconf

2 Run following initctlcommand to reloadconfiguration

initctl reload-configuration

3 Restart scvmclient process

stop scvmclient startscvmclient

4 Run following initctlcommand to mount SYSTEMDS

initctl emit --no-waitsystem-datastore-created

5 Verify that iscsiEnable tuneis set to true using thefollowing command

ps -eaf | grep scvmclient |grep -v grep

root 6241 1 1 Dec06 014007optspringpathstorfs-corescvmclient-T logEnabled=false -TlogSyslogEnabled=true -TlogEchoToScreen=false -TstatLoggingToFile=false -TstatLoggingToSyslog=true-TlogDir=varlogspringpath-TnfsBackendServerList=1010748100-T iscsiEnable=true -TiscsiUseAsync=true -T

Release Notes for Cisco HX Data Platform Release 4053

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

iscsiConfigFilePath=nfsSYSTEMistgtconfps -eaf | grep scvmclient |grep -v grep

35(2e)Following procedure can be usedto check NFS Queue Depth andincrease if needed

rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth256 lt- Low value[rootHXESXI1] vsish -e setvmkModulesnfsclientmnt[DATASTORE]propertiesmaxQDepth 1024[rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth1024 lt- Optimalvalue

This requires an ESXi host rebootto take effect

Please place the node in HyperflexMaintenanceMode and gracefullyreboot the node for the changes tobe applied

HyperFlex Datastore NFS Queue Depthshows as 256 which can lead toperformance (including latency) issues

CSCvr83056

40(1b)This is cosmetic The error will goaway once the cluster returns tohealthy status

When a single node is offline in clusterstcli cluster storage-summary shows twonodes unavailable rootSCVM~ stclicluster storage-summary

CSCvs47735

35(2g)Modify both the root and adminpassword for the SCVMs to be thesame

stcli security password set -uadmin

Cluster expansion fails at Config Installerstage when the root and admin passwordfor the storage controller (SCVM) aredifferent

CSCvs69317

35(2d)NAAfter successfully changing the DNSserver on the HX controller we still cansee the original DNS entry that was addedduring the deployment

CSCvs69154

Release Notes for Cisco HX Data Platform Release 4054

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Customers are strongly encouragedto work with TAC in order toidentify and remediate this issue

1 Change getcluster from 2m to8m inoptspringpathstorfs-mgmtstMgr-10confapplicationconfon all nodes

2 Restart stMgr on all nodes

You may see this error message after afailed upgrade or other task such asattempting to enter a node into HXmaintenance mode

CSCvs53555

40(2a)Run the following command oneach storage controller VM as root

grep -i monitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp sed -iesmonitor_dns_serverstruemonitor_dns_serversfalseoptspringpathhx-diag-toolswatchdog_configjsonampamp grep -imonitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp restart watchdog

Receive the following alertevent in HXConnect HX Controller VMHOSTNAME one or more configuredDNS servers not responding

CSCvt13947

35(2b)Delete any empty (Size 0) log filesunder varlogzookeeperversion-2ORvarlogzookeeperstandaloneversion-2

Zookeeper fails to start while Exhibitor isrunning however echo srvr returnsnothing

CSCvs21562

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

40(4e)1 De-commission the server

2 Power drain the server -REMOVE BOTH POWERCORDS ON THE BACK OFTHE SERVER FOR 10SECONDS THENREINSERTPOWERCORDS

3 Re-commission the server

M2 boot disk is missing from serverinventory after upgrade to server firmware40(4e) As a result server fails to boot toOS installed in the M2 disk The issuepersists after re-acknowledgement as wellas de-commission andre-acknowledgement of the server

CSCvs02466

Release Notes for Cisco HX Data Platform Release 4055

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Re-start the flusher manuallyMany disks taken offline by a fault ormanually resulting in ENOSPC oncluster

CSCvq32721

35(2e)Clean up fail over cluster and thenrecreate the fail over cluster Thereis no need to touch the HX storagecluster

[Hyper-V]When replicated DCwas usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2b)Manually install upgradescvmclient VIB andmake sure thatit matches with HXDP version

Upgrade from release HX 35(2a) to35(2x) 4X does not upgrade scvmclientRecovery point may not work

CSCvq54992

Management

30(1b)To recover the VM copy over thedata disks and attach them to thenew VM

[HyperV]VMshows up onmultiple nodesCSCvj22992

35(1a)1 Do not change MTU at ESXlevel on running cluster

2 Reset back to the original value

ChangingMTU from 9000 to 1500 on oneESX node causes storfs process to restarton all nodes

CSCvm99150

Release Notes for Cisco HX Data Platform Release 4056

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)After the installation (for newdeployments) or after an upgradeto 401a (for existingdeployments) perform thefollowing steps on all ControllerVMs

1 Edit the following tune files (onall Controller VMs)

optspringpathconfiglfftunes

optspringpathconfigvsi_16tbtunes

2 SetcleanerEnableSegSummaryCleaning

to false

3 After editing the above tunefiles

a SSH in to all the ControllerVMs

b Run storfstool -- -Z

c Run the below command andcheck the values Tune valueshould be true

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=true

d Type below commands andapplied the tunes changesdynamically

echo false gttmpstprocfssystemtunecleanerEnableSegSummaryCleaning

e Verify that tune values arechanged Run below commandValue should be false

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=false

f umount tmpstprocfs

A cluster with 8TB or 12TB disk drivesmay experience IO stalls for severalminutes after another node fails in thecluster

CSCvp23718

Release Notes for Cisco HX Data Platform Release 4057

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)1 Update the conf file manuallyto change to the link local addressto

2 Restart scvmclient on thecontroller and ESX

HX 35(2b) FlexVol to HX 40 CSIupgrade does not work FlexVol continuesto work

CSCvp26319

35(2a)Disable rebalance until node healsthen reenable rebalance

7-node cluster healing very slowlybecause rebalance change AMS reqs areblocking healing process

CSCvp62512

35(2a)NAWith moderate cluster storage capacityusage under site failover cluster clustermay experience an all paths (APD) downcondition

CSCvq49412

40(1a)NAThis product includes Third-partySoftware that is affected by thevulnerabilities identified by the followingCommon Vulnerability and Exposures(CVE) IDs

CVE-2016-0762 CVE-2016-6797CVE-2016-6816 CVE-2016-8735CVE-2017-5647 CVE-2017-12615CVE-2017-12616 CVE-2017-12617CVE-2017-7674 CVE-2018-1304CVE-2018-8014 CVE-2018-1336CVE-2018-8034 CVE-2018-11784CVE-2019-0232

CSCvp36364

35(2d)NAstcli cluster info command needs toprovide UCSM VIP address Currently itshows ucsm-hostcom

CSCvq11456

35(2a)No workarounds currently Thestale entries do not affect the dayto day operations of HyperFlex

HyperFlex upgrade validation failedbecause cluster has extraneous stNodes ininternal database

CSCvq32530

35(2a)NAWhen not in maintenancemode all drivesappear in HX Connect and get updated asexpected

When in maintenance mode extra drivesare shown in HX Connect with no slotnumbers

CSCvq42378

35(2b)Contact TAC to cleanup duplicatenode

Nodes duplicated in HX Connect aftercluster recreate

CSCvq61825

Release Notes for Cisco HX Data Platform Release 4058

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)NACurrently HyperFlex installs on Hyper-Vdo not contain the stcli security whitelistcommandset

CSCvq66245

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

Release Notes for Cisco HX Data Platform Release 4059

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)[Hyper-V] Converged node expansionfails for incoming node with VIC 14xx

CSCvq39523

Release Notes for Cisco HX Data Platform Release 4060

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

1 Set Jumbo frame inWindowsHypervisor for incomingnode

Get-NetAdapter storage-data-a|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

Get-NetAdapter storage-data-b|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

2 Verify all interfaces are setcorrectly to support Jumbo frames

Get-NetAdapter storage |Get-NetAdapterAdvancedProperty| RegistryKeyword -matchjumbo | ft -auto

You should receive the followingmessage

Name DisplayName DisplayValue

RegistryKeyword RegistryValue

---- ----------- ------------

--------------- -------------

vswitch-hx-storage-data Jumbo

Packet 9014 Bytes

JumboPacket 9014

storage-data-b Jumbo Packet

Bytes 9014 JumboPacket

9014

storage-data-a Jumbo Packet

Bytes 9014 JumboPacket

9014

3 Reboot only the incoming nodeor the node being expanded

4 Retry Cluster Expansion in

Release Notes for Cisco HX Data Platform Release 4061

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

Recommended Firmware Version - HXDP 4x

(be sure to review important notes above)

Component

40(4k)

Download Software

Host Upgrade Utility (HUU) Download Link

HyperFlex Licensing

As of version 26(1a) HyperFlex supports VMware PAC licensing Existing VMware embedded licenses willcontinue to be supported

As of version 25(1a) HyperFlex uses a smart licensing mechanism to apply your licenses See the CiscoHyperFlex Systems Installation Guide for VMware ESXi for details and steps

VMware vSphere Licensing Requirements

How you purchase your vSphere license determines how your license applies to your HyperFlex system

bull If you purchased your vSphere license with HyperFlex

Each HyperFlex server either has the Enterprise or Enterprise Plus edition preinstalled at the factory

bull HX Nodes have OEM licenses preinstalled If you delete or overwrite thecontent of the boot drives after receiving the HX servers you also delete thefactory-installed licenses

bull OEM license keys is a newVMware vCenter 60 U1b feature Earlier versionsdo not support OEM licenses

bull All factory-installed HX nodes share the same OEM license key WithvSphere OEM keys the Usage count can exceed the Capacity value

bull When you add an HX host to vCenter through the Add Host wizard in theAssign license section select the OEM license

We obfuscate the actual vSphere OEM license key for example0N085-XXXXX-XXXXX-XXXXX-10LHH

bull Standard Essentials Plus and ROBO editions are not available preinstalledon HX servers

Note

bull If you did NOT purchase your vSphere license with HyperFlex

The HX nodes have a vSphere Foundation license preinstalled After initial setup you can apply thelicense to a supported version of vSphere

bull If you purchased a vSphere PAC license

Follow the instructions in your PAC license letter from VMware to add the license to your MY VMwareaccount then follow the instructions to add your HX host to vCenter and assign the PAC license

Release Notes for Cisco HX Data Platform Release 4012

HyperFlex Licensing

HX Data Platform Software Versions for HyperFlex Witness Node

Witness Node VersionHyperFlex Release

10840(2b)

10840(2a)

10440(1b)

10440(1a)

Software Requirements for VMware ESXi

The software requirements include verification that you are using compatible versions of Cisco HyperFlexSystems (HX) components and VMware vSphere VMware vCenter and VMware ESXi

bull Verify that all HX servers have a compatible version of vSphere preinstalled

bull Verify that the vCenter version is the same or later than the ESXi version

bull Verify that the vCenter and ESXi versions are compatible by consulting the VMware ProductInteroperability Matrix Newer vCenter versions may be used with older ESXi versions so long as bothESXi and vCenter are supported in the table below

bull Verify that you have a vCenter administrator account with root-level privileges and the associatedpassword

For VIC1457 there is no support for ESXi 60Note

The below table applies for all of the following VMware vSphere Editions Enterprise Enterprise PlusStandard Essentials Plus ROBO

VMware vCenter VersionsVMware ESXi VersionsHyperFlex Version

60 U3 65 U3 67 U360 U3 65 U3 67 U340(2b)1

60 U3 65 U3 67 U360 U3 65 U3 67 U340(2a)2

60 U3 65 U3 67 U260 U3 65 U3 67 U2340(1b)

60 U3 65 U2 67 U260 U3 65 U2 67 U2440(1a)

1 Cisco HyperFlex Release 40(2) is the last major HyperFlex release that will support vSphere 60 (ESXiand vCenter) due to those versions reaching end of VMware general support on March 12 2020

2 Cisco HyperFlex Release 40(2) is the last major HyperFlex release that will support vSphere 60 (ESXiand vCenter) due to those versions reaching end of VMware general support on March 12 2020

3 Use of 67 U2 is not recommended see software advisory for CSCvq06952 and CSCvp88515 for furtherdetails

4 Use of 67 U2 is not recommended see software advisory for CSCvq06952 and CSCvp88515 for furtherdetails

Release Notes for Cisco HX Data Platform Release 4013

HX Data Platform Software Versions for HyperFlex Witness Node

Software Requirements for Microsoft Hyper-V

The software requirements include verification that you are using compatible versions of Cisco HyperFlexSystems (HX) components and Microsoft Hyper-V (Hyper-V) components

HyperFlex Software versions

The HX componentsmdashCisco HX Data Platform Installer Cisco HX Data Platform and Cisco UCSfirmwaremdashare installed on different servers Verify that each component on each server used with and withinthe HX Storage Cluster are compatible

bull Cisco HyperFlex M5 Converged nodesmdash For Hybrid (Cisco HyperFlex HX240c M5 HX220c M5)and All Flash (Cisco HyperFlex HXAF240cM5 HXAF220cM5) verify that Cisco UCSManager 40(2b)is installed HX 40 (1a) does not support Hyper-V on the All NVMe (HXAF220C-M5SN) nodes Fordetailed information on installation requirements and steps see theCisco HyperFlex Systems InstallationGuide on Microsoft Hyper-V

Table 4 Supported HyperFlex Software versions for M5 Servers on Hyper-V

M5 Recommended Server FirmwareHyperFlex Release

40(4h)40(2b)

40(4g)40(2a)

40(4e)40(1b)

40(4e)40(1a)

Release Notes for Cisco HX Data Platform Release 4014

Software Requirements for Microsoft Hyper-V

Table 5 Supported Microsoft Software versions

VersionMicrosoft Component

Windows Server 2016 Datacenter Core amp DesktopExperience

ForWindows Server 2016Datacenter Coreand Desktop Experience the Windows2016 ISO image should be Update BuildRevision (UBR) 1884 at a minimum

Note

Windows Server 2019Datacenter-DesktopExperienceis supported starting from HXDP 401(a) onwards

For Windows Server 2019 DesktopExperience theWindows 2019 ISO imageshould be Update Build Revision (UBR)107 at a minimum

Note

Windows Server 2019 DatacenterndashCore is notsupported currently

Also note that the following are currently notsupported

OEM activated ISOs and Retail ISOs are notsupported

Earlier versions ofWindows Server such asWindows2012r2 are not supported

Non-English versions of the ISO are not supported

Windows Operating System (Windows OS)

A Windows 2012 or later domain and forestfunctionality level

Active Directory

Supported Microsoft License Editions

The Microsoft Windows Server version that is installed on one or more HyperFlex hosts must be licensed asper Microsoft licensing requirements listed on Microsoft Licensing

Browser Recommendations

Use one of the following browsers to run the listed HyperFlex components These browsers have been testedand approved Other browsers might work but full functionality has not been tested and confirmed

Table 6 Supported Browsers

HX ConnectHX Data PlatformInstaller

Cisco UCS ManagerCisco IntersightBrowser

11 or higher11 or higher11 or higherNAMicrosoft InternetExplorer

70 or higher70 or higher57 or higher62 or higherGoogle Chrome

Release Notes for Cisco HX Data Platform Release 4015

Browser Recommendations

HX ConnectHX Data PlatformInstaller

Cisco UCS ManagerCisco IntersightBrowser

60 or higher60 or higher45 or higher57 or higherMozilla Firefox

NANA9 or higher10 or higherApple Safari

NANA35 or higherNAOpera

Notes

bull Cisco HyperFlex Connect

The minimum recommended resolution is 1024 X 768

bull Cisco HX Data Platform Plug-in

The Cisco HX Data Platform Plug-in runs in vSphere For VMware Host Client System browserrequirements see the VMware documentation at httpswwwvmwarecomsupportpubs

The HX Data Platform Plug-in is not displayed in the vCenter HTML client You must use the vCenterflash client

bull Cisco UCS Manager

The browser must support the following

bull Java Runtime Environment 16 or later

bull Adobe Flash Player 10 or higher is required for some features

For the latest browser information about Cisco UCSManager refer to themost recent Cisco UCSManagerGetting Started Guide

Cisco HX Data Platform Storage Cluster SpecificationsCluster Limits

bull Cisco HX Data Platform supports up to 100 clusters managed per vCenter as per VMware configurationmaximums

bull Cisco HX Data Platform supports any number of clusters on a single FI domain Each HX convergednode must be directly connected to a dedicated FI port on fabric A and fabric B without the use of a FEXC-series compute only nodes must also directly connect to both FIs B-series compute only nodes willconnect through a chassis IO module to both fabrics In the end the number of physical ports on the FIwill dictate the maximum cluster size and maximum number of individual clusters supported in a UCSdomain

The following table provides Cisco HX Data Platform Storage Cluster Specifications

Release Notes for Cisco HX Data Platform Release 4016

Cisco HX Data Platform Storage Cluster Specifications

Stretched Cluster (Availableon ESX Only)

Microsoft Hyper-VVMware ESXiNode

AllNVMe -HXAF220c-M5SN

HX240c-M5LHX220cM5

HX220cAF M5

HX240cM5

HX240cAF M5

HX240c-M5LHX220cM5

HX220cAF M5

HX240cM5

HX240cAF M5

HXAF220c-M5SN

AllNVMe -HXAF220c-M5SN

ESXi 65or 67only NotsupportedwithESXi 60orHyper-V

HX220cM5 Edge

HXAF220cM5 Edge

HX220cM4 Edge

HXAF220cM4 Edge

HX240c-M5LHX220cM5

HX220cAF M5

HX240cM5

HX240cAF M5

HX220cM4

HX220cAF M4

HX240cM4

HX240cAF M4

HXServers

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

C220M5C240M5B200M4B200M5

C240M5C220M5B200M4B200M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

mdashB200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

Compute-OnlyUCSB-SeriesC-SeriesServers

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

Convergednodesonly

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

SupportedNodes

Release Notes for Cisco HX Data Platform Release 4017

Cisco HX Data Platform Storage Cluster Specifications

Stretched Cluster (Availableon ESX Only)

Microsoft Hyper-VVMware ESXiNode

NA(requiresEnterpriseHXDP-PLicense)

NA(requiresEnterpriseHXDP-PLicense)

NA(requiresEnterpriseHXDP-PLicense)

Convergednodes3-16(12TBHDDoption isnotsupportedforHyperV)

Compute-onlynodes0-16

Convergednodes3-16

Compute-onlynodes0-16

NA(requiresEnterpriseHXDP-PLicense)

M4Convergednodes 3

M5Convergednodes23or 4

RequiresHXDP-ELicense

Convergednodes3-16(12TBdrive islimited toamaximumof 8nodes)

Compute-onlynodes0-16(12TBdrive islimited toamaximumof 8nodes)

Convergednodes3-32(76TBdriveconfigsarelimited tomaximumof 12drivesnodeon AF240)

Compute-onlynodes0-32

HXDP-SLicensedNodeLimits

11 ratioofHXDP-StoComputeonlynodes

(MinmdashMax)

Convergednodes2-16 perSite

Compute-onlynodes0-21 perSite

(up tomaxclustersize)

Convergednodes2-8 perSite

Compute-onlynodes0-16 perSite

(up tomaxclustersize)

Convergednodes2-16 perSite

Compute-onlynodes0-21 perSite

(up tomaxclustersize)

Convergednodes3-16(12TBHDDoption isnotsupportedforHyperV)

Compute-onlynodes0-16

Convergednodes3-16

Compute-onlynodes0-16

Convergednodes3-32

ComputeOnlynodes0-32

(up tomaxclustersize)

Convergednodes 3

(requiresHXDP-ELicense)

Convergednodes3-16(12TBdrive islimited toamaximumof 8nodes)

Compute-onlynodes0-32(12TBdrive islimited toamaximumof 16nodes)

Convergednodes3-32(76TBdriveconfigsarelimited tomaximumof 12drivesnodeon AF240)

Compute-onlynodes0-32

(up tomaxclustersize)

HXDP-PLicensedNodeLimits

12 ratioofHXDP-PtoComputeonlynodes

(MinmdashMax)

24 perSite 64percluster

24 perSite 48percluster

32 perSite 64percluster

32326434864MaxClusterSize

Release Notes for Cisco HX Data Platform Release 4018

Cisco HX Data Platform Storage Cluster Specifications

Stretched Cluster (Availableon ESX Only)

Microsoft Hyper-VVMware ESXiNode

212121111121mdash2121MaxComputetoConvergedratio

NoExpansion

Requires Enterprise license

Requires uniform expansion across both sites

Guidelines and Limitationsbull Starting with release 40(2a) SCVM is no longer needed on a Compute node

bull HX REST API Access Token Management ndash Applications leveraging HX REST APIs should re-useaccess tokens when making API calls Once obtained using the AAA Obtain Access Token API accesstokens are valid for 18 days (1555200 seconds) In addition AAA enforces rate limiting on ObtainAccess Token API requests in a 15 minute window auth can be invoked (successfully) a maximum of5 times A user is allowed to create a maximum of 8 unrevoked tokens Subsequent call to auth willautomatically revoke the oldest issued token tomake room for the new token Amaximum of 16 unrevokedtokens can be present in system In order to prevent brute-force attacks after 10 consecutive failedauthentication attempts a user account is locked for a period of 120 secondsFor more information seeCisco HyperFlex Systems REST API Reference guide

HxConnect makes use of AAA Authentication REST API for login and the above rate limit applies toHxConnect also

Upgrade Guidelines

The following list is a highlight of critical criteria for performing an upgrade of your HyperFlex system

bull Unsupported HX Data Platform 17x 18x 20 21x 25x and 26x clustersmdashUsers from anyversion prior to 26(1a) must step through an intermediate version before upgrading to 40 or later releasesIf you need to upgrade your environment from a Cisco HyperFlex HX Data Platform software releasethat is past the last date of support to the latest suggested release on the Cisco Software Download sitesee Cisco HyperFlex Systems Upgrade Guide for Unsupported Cisco HXReleases For more informationsee the Software Advisory for CSCvq66867 WARNING Only Use HXDP 26(1e) Upgrade PackageWhen Upgrading From HXDP 18(1a)-18(1e)

bull Hypercheck Health Check UtilitymdashCisco recommends running this proactive health check utility onyour HyperFlex cluster prior to upgrade These checks provide early visibility into any areas that mayneed attention andwill help ensure a seamless upgrade experience For more information see the HyperFlexHealth amp Pre-Upgrade Check Tool TechNote for full instructions on how to install and run Hypercheck

bull vSphere 67 Software AdvisorymdashDo not upgrade to Cisco HX Data Platform Release 40(1a) whenrunning ESXi 67U1 EP06 (build 11675023) Do not upgrade to 67U1 EP06 (build 11675023) ifrunning Cisco HX Data Platform Release 40(1a) See the Software Advisory CSCvo56350 for furtherdetails

Release Notes for Cisco HX Data Platform Release 4019

Guidelines and Limitations

The software build version posted at release will override any other local versions

bull Required vCenter upgrademdashFor enhanced security Cisco HX Data Platform Release 35(1a) or laterrequires the use of TLS 12 Therefore vCenter must be upgraded to 60 U3f or later before upgradingto Cisco HX Data Platform Release 35 or later In addition ESXi should be upgraded as required tomeet HX Data Platform compatibility requirements

bull Minimum HXDP version for upgrademdashHXData Platform clusters running 26(1a) or later may upgradedirectly to 40 using the HX Connect UI

bull Cluster ReadinessmdashEnsure that the cluster is properly bootstrapped and the updated plug-in is loadedbefore proceeding Manual cluster bootstrap is required for upgrade from a pre-35 release

bull Cluster ReadinessmdashEnsure that the cluster is properly bootstrapped and the updated plug-in is loadedbefore proceeding Manual cluster bootstrap is required for HX releases earlier than 35(1a) For moreinformation see theManual Bootstrap Upgrade Process in the Cisco HyperFlex Systems Upgrade Guidefor VMware ESXi Release 40 Do not skip this cluster bootstrap step it is required for all upgradesuntil HX Release 35(1a) Auto bootstrap is supported beginning with HX release 35(1a) For moreinformation see the Auto Bootstrap Upgrade Process from HX Connect UI in the Cisco HyperFlexSystems Upgrade Guide for VMware ESXi Release 40

Manual bootstrap is not supported on Intersight clusters

bull Initiating Upgrade―Use the HX Connect UI or CLI stcli commands when upgrading from 25(1a)or later releases Use either the CLI stcli commands or the HX Data Platform Plug-in to the vSphereWeb Client when upgrading from a pre-25(1a) release The vCenter plug-in should not be used forupgrades starting with the 25(1a) release The vCenter plug-in should not be used for upgrades startingwith the 25(1a) release

If the current cluster version is at 35(1a) or above you do not need to use the stcli command Directupgrade to 40 is possible

bull Complete your Upgrade―The self-healing (or rebalance) capability is disabled temporarily during theupgrade window If the upgrade fails you should complete the upgrade as soon as possible

bull ESXi and HXDP Compatibility―Ensure your cluster is running a compatible version of ESXi basedon the running the HXData Platform version (see the section Software Requirements for VMware ESXi)ESXi compatibility is determined by the major version and update release of ESXi It is generally bestto upgrade HXDP and ESXi together if combining the upgrade operations into a single optimized rebootWhen running a split upgrade first upgrade the HX Data Platform then proceed to upgrade ESXi

bull If Upgrading to vSphere 65

bull Certain cluster functions such as native and scheduled snapshotsReadyClones and Enter or Exit HX Maintenance Mode will not operatefrom the time the upgrade is started until the HX Data Platform upgrade iscomplete

bull After upgrading ESXi using the offline zip bundle use the ESX ExitMaintenance Mode option The HX Exit Maintenance Mode option doesnot operate in the vSphere Web Client until the HX Data Platform upgradeis complete

Note

Release Notes for Cisco HX Data Platform Release 4020

Guidelines and Limitations

bull vSphere 60 UpgradesmdashUsers on vSphere 60 migrating to 65 upgrade components in the followingorder

1 Upgrade HX Data Platform and UCS firmware

2 Upgrade HX Data Platform and ESXi

3 Upgrade HX Data Platform only first then upgrade ESXi or UCS firmware or both

bull M4 Server Firmware UpgradesmdashUpgrade server firmware to ensure smooth operation and to correctknown issues Specifically newer SAS HBA firmware is available in this release and is recommendedfor long-term stability

bull Users are encouraged to upgrade to 31(3c) C-bundle or later whenever possible

bull Users running C-bundle versions before 31(2f) must upgrade server firmware by performing acombined upgrade of UCS server firmware (C-bundle) to 31(3c) or later and HX Data Platform to25 Do not split the upgrade into two separate operations

bull If the cluster is already on 31(2f) C-bundle or later you may perform an HX Data Platform onlyor combined upgrade as required

bull M5 Server Firmware UpgradesmdashM5 generation servers must run firmware version 32(2d) or later

bull Firmware Downgrades mdashDowngrading UCSM from the HX-installer is not supported

bull M4M5 Mixed DomainsmdashA mixed domain occurs when a new separate M5 cluster is installed underthe same UCS domain that contains existing M4 clusters Under these conditions orchestrated UCSserver firmware upgrade will not operate until Cisco HX Data Platform Release 26 or later is installedon the M4 clusters Therefore it is best practice to first upgrade UCS server firmware to the latest 31(3)or 32(2) patch release before adding a new M5 cluster to the existing UCS domain Additionally any17 HX Data Platform clusters must first be upgraded before adding any new M5 clusters to the samedomain

bull Maintenance WindowmdashIf upgrading both HX Data Platform and UCS firmware you can select eithera combined or split upgrade through the vSphere HX Data Platform Plug-in depending on the length ofthe maintenance window Cisco UCSManager infrastructure upgrade is only supported using AutoInstalland the direct server firmware upgrade should be performed only through the upgrade orchestrationframework provided by the HX Data Platform Plug-in

bull Unsupported Self-Encrypting Drives (SEDs)mdashIf adding or replacing self-encrypting drives (SEDs)that have been recently qualified in newer versions of HX Data Platform insert the new drives only afterupgrading HX Data Platform to a compatible version All drives must be SED drives mixing SED andnon-SED is not supported

bullbull Enabling External Host AccessmdashWith Cisco HX Data Platform Release 40(1a) port 445 on themanagement network is blocked for enhanced security Note that prior to 40 port 445 port was openenabling external host access If you are upgrading to 40(1a) from a prior release and would like tocontinue external host access you can use a utility to open select hosts For more information aboutenabling external host access see the ConfiguringHyperFlex Share to SCVMM section in the InstallationGuide for Microsoft Hyper-V

Release Notes for Cisco HX Data Platform Release 4021

Guidelines and Limitations

Mixed Cluster Expansion Guidelinesbull Expanding existing M4 cluster with M5 converged nodes is supported

bull Expanding existing M5 cluster with M4 converged nodes is not supported

bull Expanding existing mixed M4M5 cluster with M4 or M5 converged nodes is supported

bull Adding any supported compute-only nodes is permitted with all M4 M5 and mixed M4M5 clustersusing the HX Data Platform 26 or later Installer Some example combinations are listed here manyother combinations are possibleExample combinationsExpand mixed M4M5 cluster with compute-only B200 C220 C240 M4M5Expand M4 cluster with compute-only B200 M5 C220 M5 C240M5

bull Only expansion workflow is supported to create a mixed cluster Initial cluster creation with mixedM4M5 servers is not supported

bull All M5 servers must match the form factor (220240) type (HybridAF) security capability (Non-SEDonly) amp disk configuration (QTY capacity and non-SED) of the existing M4 servers

bull HX220-M5 will use a maximum of 6 capacity disks (2 disk slots to remain empty) when mixedwith HX220-M4

bull HX Edge SED LFF Hyper-V and Stretched Clusters do not support mixed M4 and M5 clusters

Security FixesThe following security issues are resolved

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-13132 CVE-2019-9924CSCvq6313840(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-11719 CVE-2019-11727CVE-2019-11729

CSCvq7124040(2a)

Release Notes for Cisco HX Data Platform Release 4022

Mixed Cluster Expansion Guidelines

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-1125CSCvr0633940(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2015-9383 CVE-2018-14498CVE-2018-20406 CVE-2018-20852CVE-2019-10160 CVE-2019-13117CVE-2019-13118 CVE-2019-14287CVE-2019-14973 CVE-2019-15903CVE-2019-17546 CVE-2019-18197CVE-2019-18218 CVE-2019-5010CVE-2019-5094 CVE-2019-5481CVE-2019-5482 CVE-2019-9636CVE-2019-9740 CVE-2019-9947CVE-2019-9948

CSCvs0609440(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2017-13168 CVE-2017-18174CVE-2017-18216 CVE-2018-10876CVE-2018-10877 CVE-2018-10878CVE-2018-10879 CVE-2018-10880CVE-2018-10881 CVE-2018-10882CVE-2018-10902 CVE-2018-10938CVE-2018-12233 CVE-2018-12896CVE-2018-13053 CVE-2018-13094CVE-2018-13096 CVE-2018-13405CVE-2018-13406 CVE-2018-14609CVE-2018-14617 CVE-2018-14633CVE-2018-14734 CVE-2018-15572CVE-2018-15594 CVE-2018-16276CVE-2018-16658 CVE-2018-17182CVE-2018-17972 CVE-2018-18021CVE-2018-18690 CVE-2018-18710CVE-2018-6554 CVE-2018-6555CVE-2018-9363

CSCvp6501940(2a)

This vulnerability is due to insufficientHTML iframe protection An attackercould exploit this vulnerability bydirecting a user to an attacker-controlledweb page that contains amaliciousHTMLiframe A successful exploit could allowthe attacker to conduct clickjacking orother clientside browser attacks

NACSCvo9851640(2a)

Release Notes for Cisco HX Data Platform Release 4023

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerability is due to insufficientauthentication for the statistics collectionservice An attacker could exploit thisvulnerability by sending properlyformatted data values to the statisticscollection service of an affected deviceA successful exploit could allow theattacker to cause the web interfacestatistics view to present invalid data tousers

NACSCvj9558440(2a)

The vulnerability is due to insufficientCSRF protections for the web UI on anaffected device An attacker could exploitthis vulnerability by persuading a user ofthe interface to follow a malicious linkA successful exploit could allow theattacker to perform arbitrary actions withthe privilege level of the affected user

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvp2434340(2a)

Vulnerabilities with a version of cJSONidentified by CVE-2019-11834 andCVE-2019-11835

For more information see the relatedCisco Security Advisory

CVE-2019-11834 CVE-2019-11835CSCvq1994940(2a)

HX ESXi image patches for MCEPSCand TAA vulnerabilities with VMwareESXi identified by CVE-2018-12207 andCVE-20190-11135

For more information see the relatedCisco Security Advisory

CVE-2018-12207 CVE-2019-11135CSCvr5439840(2a)

CP networking vulnerabilities affectingthe Linux kernel identified byCVE-2019-11477 CVE-2019-11478CVE-2019-11479

For more information see the relatedCisco Security Advisory

CVE-2019-11477 CVE-2019-11478CVE-2019-11479

CSCvq1954640(2a)

Qualification of Microsoft securitypatches for vulnerabilities withMicrosoftHyper-V hypervisor identified withCVE-2018-12207 CVE-2019-11135

For more information see the relatedCisco Security Advisory

CVE-2018-12207 CVE-2019-11135CSCvr5439940(2a)

Release Notes for Cisco HX Data Platform Release 4024

Security Fixes

DescriptionCVEDefect IDRelease

OpenSSH vulnerabilities identified byCVE-2019-6111

For more information see the relatedCisco Security Advisory

CVE-2016-10708 CVE-2018-15473CVE-2018-20685 CVE-2019-6109CVE-2019-6111

CSCvp7646340(2a)

Multiple vulnerabilities associated withOpenSSL and LibNSS

For more information see the relatedCisco Security Advisory

CVE-2016-2105 CVE-2016-2106CVE-2016-2107 CVE-2016-2109CVE-2016-2176 CVE-2016-2177CVE-2016-2178 CVE-2016-2179CVE-2016-2180 CVE-2016-2181CVE-2016-2182 CVE-2016-2183CVE-2016-6302 CVE-2016-6303CVE-2016-6304 CVE-2016-6305CVE-2016-6306 CVE-2016-6307CVE-2016-6308 CVE-2016-7055CVE-2016-8610 CVE-2017-3731CVE-2017-3732 CVE-2017-3735CVE-2017-3736 CVE-2017-3737CVE-2017-3738 CVE-2018-0495CVE-2018-0732 CVE-2018-0734CVE-2018-0735 CVE-2018-0737CVE-2018-0739 CVE-2018-12384CVE-2018-12404 CVE-2018-5407CVE-2019-1559

CSCvp6667940(2a)

libpng vulnerability identified byCVE-2016-10087

For more information see the relatedCisco Security Advisory

CVE-2016-10087CSCvp6655540(2a)

Vulnerabilities associated with libjpg andlibpng

For more information see the relatedCisco Security Advisory

CVE-2014-9092 CVE-2016-3616CVE-2017-15232 CVE-2018-11212CVE-2018-11213 CVE-2018-11214CVE-2018-1152 CVE-2018-13785

CSCvp3458640(2a)

Vulnerabilities associated with Glibidentified by CVE-2018-16428 andCVE-2018-16429

For more information see the relatedCisco Security Advisory

CVE-2018-16428 CVE-2018-16429CSCvp3120740(2a)

Multiple vulnerabilities associated withcurl expat python 27 python 35 36and 37 freetype and giflib

For more information see the relatedCisco Security Advisory

CVE-2019-15133 CVE-2019-15903CVE-2019-5010 CVE-2019-5481CVE-2019-5482 CVE-2019-9636CVE-2019-9740 CVE-2019-9947CVE-2019-9948

CSCvr3690340(2a)

Release Notes for Cisco HX Data Platform Release 4025

Security Fixes

DescriptionCVEDefect IDRelease

Multiple third party softwarevulnerabilities

For more information see the relatedCisco Security Advisory

CVE-2019-14379 CVE-2019-12384CVE-2019-14439

CSCvq9203240(2a)

Vulnerabilities associated with elfutils

For more information see the relatedCisco Security Advisory

CVE-2018-16062 CVE-2018-16402CVE-2018-16403 CVE-2018-18310CVE-2018-18520 CVE-2018-18521CVE-2019-7149 CVE-2019-7150CVE-2019-7665

CSCvq4325040(2a)

Vulnerabilities associated with vim

For more information see the relatedCisco Security Advisory

CVE-2017-5953 CVE-2019-12735CSCvq4323040(2a)

Vulnerabilities associated with sqlite3

For more information see the relatedCisco Security Advisory

CVE-2016-6153 CVE-2017-10989CVE-2017-13685 CVE-2017-2518CVE-2017-2519 CVE-2017-2520CVE-2018-20346 CVE-2018-20505CVE-2018-20506 CVE-2019-8457CVE-2019-9936 CVE-2019-9937

CSCvq4321340(2a)

Vulnerabilities associated with glib20

For more information see the relatedCisco Security Advisory

CVE-2018-20843CSCvq4320940(2a)

Vulnerabilities associated with expat

For more information see the relatedCisco Security Advisory

CVE-2018-20843CSCvq4320540(2a)

Vulnerabilities associated with bzip2

For more information see the relatedCisco Security Advisory

CVE-2016-3189 CVE-2019-12900CSCvq4319440(2a)

Mulitple Vulnerabilities associated wtihNodeJS

For more information see the relatedCisco Security Advisory

CVE-2018-12115 CVE-2018-0734CVE-2018-5407 CVE-2018-12120CVE-2018-12121 CVE-2018-12122CVE-2018-12123 CVE-2018-12116CVE-2019-5737 CVE-2019-5739

CSCvq1069440(2a)

Vulnerabilities associated with PalletsJinja strformat_map and identified byCVE-2019-10906 and CVE-2016-10745

For more information see the relatedCisco Security Advisory

CVE-2019-10906 CVE-2016-10745CSCvq1038840(2a)

Release Notes for Cisco HX Data Platform Release 4026

Security Fixes

DescriptionCVEDefect IDRelease

Multiple Security Vulnerabilities

For more information see the relatedCisco Security Advisory

CVE-2014-8501 CVE-2014-9939CVE-2015-9262 CVE-2016-10087CVE-2016-2226 CVE-2016-4487CVE-2016-4488 CVE-2016-4489CVE-2016-4490 CVE-2016-4491CVE-2016-4492 CVE-2016-4493CVE-2016-6131 CVE-2018-10963CVE-2018-13785 CVE-2018-17100CVE-2018-17101 CVE-2018-18557CVE-2018-18661 CVE-2018-7456CVE-2018-8905

CSCvq0917840(2a)

Vulnerabilities associated withlibseccomp

For more information see the relatedCisco Security Advisory

CVE-2019-9893CSCvq0756840(2a)

Vulnerabilities associated withgdk-pixbuf

For more information see the relatedCisco Security Advisory

CVE-2017-12447CSCvq0675540(2a)

Vulnerabilities associated withpython-crypto

For more information see the relatedCisco Security Advisory

CVE-2018-6594CSCvp9381740(2a)

Vulnerabilities associated with wget

For more information see the relatedCisco Security Advisory

CVE-2018-20483 CVE-2019-5953CSCvp8672140(2a)

Multiple vulnerabilities associated withPython Crypto

For more information see the relatedCisco Security Advisory

CVE-2018-6594CSCvp6674840(2a)

Vulnerabilities associated withsensible-utils

For more information see the relatedCisco Security Advisory

CVE-2017-17512CSCvp6673440(2a)

Multiple vulnerabilities associated withPython

For more information see the relatedCisco Security Advisory

CVE-2018-1000030CSCvp6668940(2a)

Release Notes for Cisco HX Data Platform Release 4027

Security Fixes

DescriptionCVEDefect IDRelease

Multiple patch vulnerabilities identifiedby CVE-2016-10713

For more information see the relatedCisco Security Advisory

CVE-2016-10713CVE-2018-1000156 CVE-2018-6951

CSCvp6667240(2a)

Vulnerabilities associatedwith Little CMSidentified by CVE-2016-10165

For more information see the relatedCisco Security Advisory

CVE-2016-10165 CVE-2018-16435CSCvp6666440(2a)

Vulnerabilities associated with TenableScan identified byCVE-2011-5325CVE-2019-11068CVE-2016-7076CVE-2017-1000368

For more information see the relatedCisco Security Advisory

CVE-2011-5325 CVE-2016-7076CVE-2017-1000368CVE-2019-11068

CSCvp6474640(2a)

Vulnerabilities associated with libxml2294 and earlier as used in XMLSec1223 and earlier and other products andidentified by CVE-2016-9318

For more information see the relatedCisco Security Advisory

CVE-2016-9318 CVE-2017-16932CVE-2017-18258 CVE-2018-14404CVE-2018-14567

CSCvp3479240(2a)

Vulnerabilities associated with LFTPRemote File Names Unauthorized AccessVulnerability and identified byCVE-2018-10916

For more information see the relatedCisco Security Advisory

CVE-2018-10916CSCvp2926640(2a)

Vulnerabilities associated withCryptographyDeprecationWarning signerand verifier have been deprecated

For more information see the relatedCisco Security Advisory

CVE-2018-7750CSCvo3409740(2a)

Multiple Vulnerabilities in python python27 35

For more information see the relatedCisco Security Advisory

CVE-2019-16056CSCvr3979340(2a)

Release Notes for Cisco HX Data Platform Release 4028

Security Fixes

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toexecute commands as the root user

The vulnerability is due to insufficientinput validation An attacker could exploitthis vulnerability by connecting to thecluster service manager and injectingcommands into the bound process Asuccessful exploit could allow the attackerto run commands on the affected host asthe root user

Cisco has released software updates thataddress this vulnerability There areworkarounds that address thisvulnerability

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvq2417640(1b)35(2g)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

The vulnerability is due to an unprotectedlistening interface An attacker couldexploit this vulnerability by connectingto the listening interface and injectingcommands to the bound process Anexploit could allow the attacker to runcommands on the affected host as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvj9560640(1b)35(2g)

Release Notes for Cisco HX Data Platform Release 4029

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerabilities associated with theJVM18U121memory leak found duringVC alarms (concurrent 40 calls usingREST API)

CVE-2017-10053 CVE-2017-10067CVE-2017-10074 CVE-2017-10078CVE-2017-10081 CVE-2017-10087CVE-2017-10089 CVE-2017-10090CVE-2017-10096 CVE-2017-10101CVE-2017-10102 CVE-2017-10107CVE-2017-10108 CVE-2017-10109CVE-2017-10110 CVE-2017-10111CVE-2017-10115 CVE-2017-10116CVE-2017-10118 CVE-2017-10135CVE-2017-10176 CVE-2017-10193CVE-2017-10198 CVE-2017-10243CVE-2017-10274 CVE-2017-10281CVE-2017-10285 CVE-2017-10295CVE-2017-10345 CVE-2017-10346CVE-2017-10347 CVE-2017-10348CVE-2017-10349 CVE-2017-10350CVE-2017-10355 CVE-2017-10356CVE-2017-10357 CVE-2017-10388CVE-2017-3509 CVE-2017-3511CVE-2017-3526 CVE-2017-3533CVE-2017-3539 CVE-2017-3544CVE-2018-2579 CVE-2018-2582CVE-2018-2588 CVE-2018-2599CVE-2018-2602 CVE-2018-2603CVE-2018-2618 CVE-2018-2629CVE-2018-2633 CVE-2018-2634CVE-2018-2637 CVE-2018-2641CVE-2018-2663 CVE-2018-2677CVE-2018-2678 CVE-2018-2783CVE-2018-2790 CVE-2018-2794CVE-2018-2795 CVE-2018-2796CVE-2018-2797 CVE-2018-2798CVE-2018-2799 CVE-2018-2800CVE-2018-2814 CVE-2018-2815CVE-2018-2952 CVE-2018-3136CVE-2018-3139 CVE-2018-3149CVE-2018-3150 CVE-2018-3169CVE-2018-3180 CVE-2018-3183CVE-2018-3214 CVE-2019-2422

CSCvo8899740(1b)

Tomcat and Nginx logs are not beingcollected in the support bundle generatedthrough HX Connect in Release 35

NACSCvm5803140(1b)

The vulnerabilities associated with thelibmspack software package versionincluded in Cisco HX Data Platform

CVE-2018-18584

CVE-2018-18585

CSCvn3511940(1a)

Release Notes for Cisco HX Data Platform Release 4030

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerabilities associated withFasterXML Jackson-Databind TimeValueField Denial of Service

CVE-2018-14719

CVE-2018-14720

CVE-2018-1000873

CVE-2018-14721

CVE-2018-19360

CVE-2018-19362

CVE-2018-19361

CVE-2018-14718

CSCvn8228240(1a)

The vulnerabilities associated with theOpenSSL Protocol software packageversion included in Cisco HX DataPlatform

CVE-2013-3587CSCvo0505440(1a)

The vulnerabilities associated withThird-Party Software Denial of Service

CVE-2018-16487

CVE-2018-19361

CSCvo2781840(1a)

The vulnerabilities associated withLinuxkernel for Ubuntu 1710

CVE-2018-1092

CVE-2018-7492

CVE-2018-8087

CVE-2018-1068

CVE-2018-8781

CSCvm5314935(2a)

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)25(1d)stcli services dns remove should remove the DNS serverinfo from the interface files

CSCvs70967

40(2b)40(1b)HX 40(1b) new deployment from Intersight productioncloud

User receives following error while enabling CSI(Kubernetes integration)

Failure occurred during volume_access_enable Errorwas Nonzero exit code 1

CSCvt10522

40(2b)40(2a)Receive the following alertevent in HX Connect HXController VM HOSTNAME one or more configuredDNS servers not responding

CSCvt13947

Release Notes for Cisco HX Data Platform Release 4031

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)40(2a)API UI CLI show few Drive slots Empty after upgradeto 402a

The drives are seen in the sysmtool lsscsi and stclicommand outputs

The cluster is healthy with no errors

CSCvt14914

40(2b)35(2d)After successfully changing the DNS server on the HXcontroller we still can see the original DNS entry thatwas added during the deployment

CSCvs69154

40(2b)35(2e)HyperFlex Connect and VCenter show APD alarms forHX and non HX datastores This implies there is an issuewith HyperFlex when there may not be

CSCvs30080

40(2b)40(2a)An issue where all the disks in the node get locked afterrebooting the node

We successfully unlocked using sed-clientsh -Ucommand but wanted to test with another reboot anddrives locked again

CSCvs74286

40(2b)40(2a)When running stcli license commands on HyperFlexthe following error is seen

rootSpringpathControllertmp stcli license show all

Show smart licensing failed

Smart Agent is not ready please wait a minute and tryagain

CSCvt13929

40(2b)40(2a)After upgrading to HX 40(2a) the first RoboEdge nodethat gets rebooted then ignores persistent drivesPIDHX-HD24TB10K4KN

Cluster will be in degraded state and have reducedcapacity

CSCvt20144

40(2b)35(2b)Zookeeper fails to start while Exhibitor is runninghowever echo srvr returns nothing

CSCvs21562

40(2b)40(2a)When performing an HyperFlex upgrade a validationwarning may occur due to the host not having EnterprisePlus or Enterprise hypervisor licensing

Upgrade Validation Warning

ESXi host esx1labtest should be configured withVMware Enterprise license for upgrade to continue

CSCvs91787

Release Notes for Cisco HX Data Platform Release 4032

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)35(2g)Cluster expansion fails at Config Installer stage whenthe root and admin password for the storage controller(SCVM) are different

CSCvs69317

40(2b)35(2g)Panic while upgrading ESXiCSCvt06983

40(2b)40(2a)A cluster node running may hang in the Linux kernelThis is classified as an oops and a deviation from theexpected behavior

CSCvs54285

40(2b)35(2g)Rebalance failing on 10+10 node HX 35(2g) stretchcluster

CSCvs69007

40(2b)35(2d)The cluster becomes inaccessible to the IOVisorCSCvr54687

40(2b)40(2a)Panic on the storage controllerCSCvt61297

40(2b)40(2a)The size of support bundle is very large when collectedwith storfs-support command

CSCvt63306

40(2b)40(2a)storfs crash associated withFileSystemUsageWarningEvent orFileSystemUsageAlertEvent due to high usage ofvarstv above 80

CSCvt72807

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

40(2a)35(2d)In normal scenarios HX upgrade gets pendingACK afterupdating service profile In this case UCS only upgradewas stalled because after updating service profile HXupgrade was not getting pending ACK and it was alwayswaiting state

CSCvo39912

40(2a)35(1a)When using motherBoardReplace-12 to clean ZK fromold stNodepNodes resulted in unmounted datastore andmaking the size of the Hyperflex Datastores 0 resultingin all VMs in the cluster going offline

CSCvq39471

40(2a)26(1e)Upgrading cluster and getting error

Failed upgrade validations Checking vCenterconfiguration Reason Upgrade validations failed DRSFault Insufficient resources to satisfy configured failover

CSCvm77294

Release Notes for Cisco HX Data Platform Release 4033

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)The cluster expand fails on a node with DR replicationconfigured

When a HX cluster which has DR replication configuredis expanded we see the installer UI pulling in thereplicationVLAN information instead of themanagementVLAN information Even if we change that informationto the correct mgmt VLAN id and name it it does notseem to work as the node is configured with the VLANof the replicationVLAN in ESXi This leads to the failureof the node add with the host unreachable error

CSCvo70650

40(2a)35(1a)Customer reported that server didnt complete thefirmware upgrade automatically one by one

The user manually needs to put the host into maintenancemode and they manually acknowledged the pendingrequests to finish the UCS firmware upgrade

Maintenance policy is set to default (User-ack) as perthe design

CSCvo91624

40(2a)35(1i)When the cluster is in Failed state and stcli noderemove is attempted the output shows successful thoughthe node failed to get removed from the cluster

CSCvo93017

40(2a)35(1a)HyperFlex cluster upgrade may fail during validationwith a DRS Validation failed error

CSCvp31021

40(2a)35(2b)HX cluster expansion will fail with the following errormessage

MAC address pool configuration failure150[ErrorDescription] bad address block range definitioncollision

CSCvp58318

40(2a)35(2b)Memory usage by carbon cacheCSCvq34873

40(2a)35(2g)Upgrading ESXi cluster fails with error Nodemaintenance mode failed

CSCvr03240

40(2a)35(2e)storfs process does not automatically start on Exit HXMaintenance Mode or other tasks which restart StorageController

CSCvr88978

40(2a)35(2a)Post install script fails with message

Failed to execute ipmitool on HX node

CSCvp10707

40(2a)40(1a)HyperFlex expansionworkflow doesnt pull VLANnamecorrectly

CSCvp46539

Release Notes for Cisco HX Data Platform Release 4034

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)During HX cluster deployment - Validate ClusterCreation phase - HX Installer might fail with the error

fromvarlogspringpathstDeploylog

CSCvq45087

40(2a)35(2b)HX Installer fails to login to the SOL for configuringESX hosts

CSCvq91380

40(2a)35(2d)If DRS is enabled with configuration parameters otherthen fully automated ie manual or partially automatedthan this will change to fully automated during clusterexpansion

CSCvr44222

40(2a)35(1a)One of the validation task name has a typo The taskname is Validating node for mixed of SED and non-SEDdisks

It should be Validating node for mix of SED andnon-SED disks

CSCvp82175

40(2a)30(1i)This combination of using smaller NVMe drives is notsupported in expand operation For example if the drivesare only 375GB and they cannot be added to an existingcluster with larger caching SSD

CSCvo12359

40(2a)35(1a)HyperFlex includes a version of OpenSSL that is affectedby the vulnerabilities identified by the followingCommon Vulnerability and Exposures

CVE-2018-0495

CSCvp66679

40(2a)30(1e)SDK invoke exception when taking a native snapshot ona VM with GPU pci passthrough attached

CSCvo00511

40(2a)30(1i)Node replacement script fails due to EAM errorCSCvo62867

40(2a)35(1a)While pairing a cluster (HX release gt= 35) and cluster(HX release lt 35) remote replication network test failson the cluster (HX release lt 35)

CSCvo79760

40(2a)30(1b)Fixed the issue in the latest support-workflow bundleThis should not be hit anymore

CSCvo87061

40(2a)35(2a)On a 3 node cluster the MBreplace script fails as thescript looks for HEALTHY cluster status

CSCvo87080

40(2a)35(2a)MbReplace script (tar) hangs when running on 352b or40

CSCvp12359

Release Notes for Cisco HX Data Platform Release 4035

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)HX replication cleanup failing with error NA 35(2a) -INFODR state is not clean

CSCvp63958

40(2a)40(1a)The check_stig_parameters API incorrectly shows thecompliance state of the cluster as not STIG compliant

CSCvp66277

40(2a)35(2c)The snapshot creation on CBT enabled VM fails witherror Failed in vmreparent vmkfstools clone1

CSCvq06952

40(2a)35(2a)HyperFlex GUI when setting up remote DR-replicationpartnerpeer should show remote replication partner TCPport reachability test results

This will provide quick upfront connectivity results thatfrequently takes customer a while to manually check andconfirm

If reachability tests fail an info window and HX alertshould trigger that list the destination peer IP and thespecific port(s) that could not be reache

CSCvo60587

40(2a)35(2b)If you have more than one IP pool configured for areplication network and local replication network testsare failing you might be running into this issue

CSCvr67130

40(2a)35(2a)Skip task option in MBreplace and Software redeployscripts The scripts should have the option to skip tasksas user inputs and not by modifying the json file

CSCvp05204

40(2a)40(1a)Error noticed on SCVM Console print_req_error IOerror dev fd0 sector 0 on HyperV

It doesnt come up on SSH session

CSCvq34357

40(2a)30(1d)The VMNetwork switch is already created as part of theinstall process

Ideally the ip address assignment should be done duringthe initial install process as well No post install stepsshould be required altrnatively it could be done as partof a post install script

CSCvk36222

40(2a)25(1c)storfs service impacted during HX upgrade from HX21(1) to 25(1c)

HX node which is not being upgraded storfs servicestopped causing outage

CSCvg53223

40(2a)35(1b)HX Stretch Cluster Witness VM reverts to DHCP onreboot

CSCvp37536

40(2a)35(2b)HyperFlex Stretch Cluster Witness does not getprogrammed with an NTP server

CSCvp46578

Release Notes for Cisco HX Data Platform Release 4036

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2b)HyperFlex cluster does not heal after maintenance onnodeserver

CSCvr18528

40(2a)40(2a)High percentage of packet loss on the eth1 data interface9K MTU

CSCvr97089

40(2a)35(2f)The HyperFlex cluster healing state is stuck at 87 andis unable to progress

CSCvr16760

40(2a)40(2a)In some circumstances two replications can occur forthe same VM

From the GUI the replication status for this VM willshow stuck in an In Progress state but there are nojobs stuck in replication layer Subsequent Replicationsshould succeed

CSCvq22898

40(2a)35(2a)After an unexpected power outage the hyperflex clusteris unable to come up correctly

All processes are running time is synced and all nodescan vmkping each other as required

CSCvp13990

40(2a)40(1a)When specifying IP address instead of node ID stclinode maintenanceMode commands fail

CSCvp33657

40(2a)40(1a)2 node edge cluster if experiencing Intersightarbitratorconnectivity issues should NOT allowHXMaintenancemode to initiate as this will bring the cluster down

CSCvr31573

40(2a)35(2c)Old files in varsupportZKTxnlog are not purged withthe daily zklog-cleanup cron job

CSCvr89066

40(2a)35(2a)When a node is in MaintenanceMode any disk removalor replacement will be reflected in UI only after the nodeis brought back frommaintenancemode This is becausestorfs is not running on the node in maintenance andwill not be able to detect disk activities until it is broughtout of Maintenance Mode

CSCvo86431

40(2a)35(2c)Cluster might go offline and stop serving IO requestsrootcvm~ stcli cluster storage-summary--detailGet cluster storage summary failedjavanetConnectException Connection refused19216814210010207

CSCvr01645

40(2a)35(2d)A stretch cluster deployment fails with a messageFormatting nodes Node in Use on the Hyperflex installerand crmZoneType value is seen as 1 in storfscfg oroptspringpathconfigstretchtunes

CSCvq80340

Release Notes for Cisco HX Data Platform Release 4037

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)30(1c)During upgrade we upload the upgrade packagedownloaded from Ciscocom Upgrade packages are offile type tgz hxconnect acceptsstorfs-packages-401a-33028tar file a different type ofcompression and allows upgrade to start and fail whenit cannot decompress the file for bootstrap and scvmclientupgrade on ESXi

CSCvq63888

40(2a)40(2a)While attempting to install hyperflex via intersight thefollowing error is seen

Failed in Task Add hosts to vCenter Cluster with ErrorTry adding hosts manually to vCenter and retry Failedto get addressFailed to host by namelookupHostnamecompanycom on 000053 read udp12700145430-gt12700153 read connection refused

CSCvr52098

40(2a)35(2a)CoreAPI call may time out for cluster management APIwhile doing cluster create validation

CSCvo92952

40(2a)35(2d)Validation fails with Mixed mode expansion checkdue to Enhanced vMotion Compatibility Incompatibilitywhen the ESXi version is different

CSCvq95460

40(2a)30(1i)This failure is seen in installer when custom workflowis used hypervisor configuration + deploy (clean diskpartition=no) at Installing software packages on storagecontroller VM

CSCvr66309

40(2a)35(2d)The HX Installer apparently replaced a comma bysemicolon on the VLAN ID range while exporting theconfiguration to a JSON file

CSCvq93831

40(2a)40(1a)When using unsupported browsers HX Connect userswill not be able to login These users see an errorindicating unauthorized user

CSCvp22693

Release Notes for Cisco HX Data Platform Release 4038

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)40(1a)A vulnerability in the web-based management interfaceof Cisco HyperFlex Software could allow anunauthenticated remote attacker to conduct a cross-siterequest forgery (CSRF) attack on an affected system

The vulnerability is due to insufficient CSRF protectionsfor the web UI on an affected device An attacker couldexploit this vulnerability by persuading a user of theinterface to follow a malicious link A successful exploitcould allow the attacker to perform arbitrary actions withthe privilege level of the affected user

Cisco has released software updates that address thisvulnerability There are no workarounds that address thisvulnerability

This advisory is available at the following link

httpstoolsciscocomsecuritycentercontentCiscoSecurityAdvisorycisco-sa-20190807-hypflex-csrf

CSCvp24343

40(2a)35(2d)Add message to HX Connect Progress flow to notAcknowledge Pending Activities and reboot the serverson UCSM

HX Connect is doing a controlled rolling server upgradein the background

CSCvq22844

40(2a)40(1b)Compute Node expand operation fails due to incorrectcluster name

CSCvr43786

40(2a)26(1e)RF-2 cluster shutdown during data resynchronizationafter a node failure (non storfs) followed by multipledisk read failures and a HardBlacklist of a disk

CSCvp41241

40(2a)40(1a)HyperFlex Edge nodes do not properly set the ESXihostname during deployment

CSCvo13143

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4039

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4040

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1b) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401b-33133p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4041

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4042

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1b)40(1a)While running the HyperFlex installer with WindowsServer Hyper-V cluster creation process fails with theerror Failure occurred during ClusterCreation process Unable to post thecontent to the downstream This symptomis encountered while deploying HyperFlex cluster withHyperFlex nodes configuredordered with Cisco VIC1457 MLOM (PID HX-MLOM-C25Q-04) andWindows Server Datacenter or Core with Hyper-Vversion 2016 or 2019

CSCvp64140

Release Notes for Cisco HX Data Platform Release 4043

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)35(2b)Adding Micron 5200 drive to a cluster fails to increasecluster capacity

CSCvo69067

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4044

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4045

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1a) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401a-33028p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4046

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4047

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1a)30(1d)Cluster instability when disks of different sector sizeplaced in HX node

CSCvk17250

40(1a)

35(2c)

35(1a)When logged in using a local HX user account insteadof a Virtual Center account an error message appearsintermittently indicating Virtual Centerunreachable or Resource informationcannot be updated when VC is reachable

CSCvo36198

40(1a)

35(2b)

30(1d)HXDP does not work with EMC RecoverPoint - needsto support VMware API (FSS-Readdir)

CSCvk38003

Release Notes for Cisco HX Data Platform Release 4048

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)

35(2a)

30(1i)

25(1c)

25(1d)

Zookeeper (Exhibitor) process on HyperFlex StorageController VMs may fail to respond to requests whenvarzookeeper has low or no free space

CSCvm90352

40(1a)26(1c)Use Asynchronous consolidation for HX SnapshotsCSCvn02151

40(1a)35(2b)HX Quiesced Snapshot for Backup VendorsCSCvo90713

40(1a)35(2a)The cluster creationcluster expansion workflow stopswith the following error message at the validation step

FIRMWARE-Check UCSC-SAS-M5HD

FIRMWARE-Check UCSC-SAS-M5HD Required00000029000000320000003500000050 Found00000058

Action Needed Update the Controller Firmware toRequired Version

CSCvn17787

35(2b)30(1d)Kernel migration fails when a local datastore is searchedfor in the ESXi host

CSCvn73127

35(2a)26(1b)A node shuts down when two disks are replaced (acaching disk and another capacity disk) where thecapacity disk is inserted first and then the caching diskis inserted

CSCvk46364

Hyper-V

40(1a)35(2a)Cluster expansion may fail with an error code 500 -operation timed out

CSCvn28721

35(2b)35(2a)During upgrade remove the VLAN on team created foruser vSwitch During fresh installation only one VLANtag is set to the vSwitch and team although multipleVLANs were entered

CSCvn54300

35(2b)35(2a)While upgrading a Hyper-V cluster on account of a rarerace condition between the stUpgradeService andZookeeper servers the upgrade orchestration throws anupgrade validation error and the upgrade process isaborted

CSCvn60486

Release Notes for Cisco HX Data Platform Release 4049

Resolved Caveats in Release 40(1a)

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Startrestart EAM service invCenter

When EAM service is not runningvCenter registration is failing

CSCvt55712

35(2a)Contact Cisco TAC if you believeyou are seeing similar conditions

HyperFlex Stretch Cluster saw poorapplication performance due to writelatency cluster remained unhealthy andrebalance was stuck

This defect is currently being used forinvestigative purposes

CSCvt45344

35(2g)Remove log files fromvarlogspringpath folder when thelog files are not getting rotatedStop the relevant service removelarge log files fromvarlogspringpath folder andrestart the service

varstv folder in Controller VM maybecome full

CSCvt36374

40(2b)Restarting the stMgr will initializeit again

In a Cluster configured with DR networkStMgr may not get initialized and stuckin a deadlock while enabling the IPTablerules (can be verified from the stMgrlog)

CSCvt89709

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Click recover for the selectedVM again under the standalonemode This will move the VM torecovered state

When the user recovers a VM protectedin a group it along with other VMs in thegroup move to recovering state instandalonemode The selectedVM shouldmove to recovered state

CSCvs75553

35(2e)Clean up the fail over cluster andthen recreate the fail over clusterNo need to touch the HX storagecluster

Hyper-V When replicated DC was usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2d)40(1a)

Keep cluster utilization as low aspossible To avoid read-only statekeep below 92

HyperFlex clusters storage utilization isabove 92 usable capacity

CSCvr20922

Release Notes for Cisco HX Data Platform Release 4050

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1b)NAAll the disks in the node got locked afterrebooting the node

We successfully unlocked usingsed-clientsh -U command but wantedto test with another reboot and driveslocked again

CSCvs74286

35(2g)To recover the VM copy over thedata disks and attach them to thenew VM

After enable encryption HX connectencryption status showing Caution butall of the disks enable encryptionsuccessful

-Self Encrypting Drives Service isrunning on all of the nodes

-There is no error message of encryption

- USB0 interface is up

-All SED disks showing supported 1enabled 1 locked 0

CSCvs93245

40(1b)Enable kernelpanic_on_oops as apersistent configuration This willcause the node to panic and rebootimmediately

A cluster node running HX release40(1b) may hang in the Linux kernelThis is classified as an oops and adeviation from the expected behavior

CSCvs54285

Release Notes for Cisco HX Data Platform Release 4051

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Enabling K8iscsi stack in Hx clusterCSCvs41324

Release Notes for Cisco HX Data Platform Release 4052

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Steps to enable K8 on HX cluster

Perform following steps on ALLthe controller VMs

1 Updateetcinitscvmclientconf toenable tune iscsiEnable totrue

sed -iesiscsiEnable=falseiscsiEnable=trueetcinitscvmclientconf

2 Run following initctlcommand to reloadconfiguration

initctl reload-configuration

3 Restart scvmclient process

stop scvmclient startscvmclient

4 Run following initctlcommand to mount SYSTEMDS

initctl emit --no-waitsystem-datastore-created

5 Verify that iscsiEnable tuneis set to true using thefollowing command

ps -eaf | grep scvmclient |grep -v grep

root 6241 1 1 Dec06 014007optspringpathstorfs-corescvmclient-T logEnabled=false -TlogSyslogEnabled=true -TlogEchoToScreen=false -TstatLoggingToFile=false -TstatLoggingToSyslog=true-TlogDir=varlogspringpath-TnfsBackendServerList=1010748100-T iscsiEnable=true -TiscsiUseAsync=true -T

Release Notes for Cisco HX Data Platform Release 4053

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

iscsiConfigFilePath=nfsSYSTEMistgtconfps -eaf | grep scvmclient |grep -v grep

35(2e)Following procedure can be usedto check NFS Queue Depth andincrease if needed

rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth256 lt- Low value[rootHXESXI1] vsish -e setvmkModulesnfsclientmnt[DATASTORE]propertiesmaxQDepth 1024[rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth1024 lt- Optimalvalue

This requires an ESXi host rebootto take effect

Please place the node in HyperflexMaintenanceMode and gracefullyreboot the node for the changes tobe applied

HyperFlex Datastore NFS Queue Depthshows as 256 which can lead toperformance (including latency) issues

CSCvr83056

40(1b)This is cosmetic The error will goaway once the cluster returns tohealthy status

When a single node is offline in clusterstcli cluster storage-summary shows twonodes unavailable rootSCVM~ stclicluster storage-summary

CSCvs47735

35(2g)Modify both the root and adminpassword for the SCVMs to be thesame

stcli security password set -uadmin

Cluster expansion fails at Config Installerstage when the root and admin passwordfor the storage controller (SCVM) aredifferent

CSCvs69317

35(2d)NAAfter successfully changing the DNSserver on the HX controller we still cansee the original DNS entry that was addedduring the deployment

CSCvs69154

Release Notes for Cisco HX Data Platform Release 4054

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Customers are strongly encouragedto work with TAC in order toidentify and remediate this issue

1 Change getcluster from 2m to8m inoptspringpathstorfs-mgmtstMgr-10confapplicationconfon all nodes

2 Restart stMgr on all nodes

You may see this error message after afailed upgrade or other task such asattempting to enter a node into HXmaintenance mode

CSCvs53555

40(2a)Run the following command oneach storage controller VM as root

grep -i monitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp sed -iesmonitor_dns_serverstruemonitor_dns_serversfalseoptspringpathhx-diag-toolswatchdog_configjsonampamp grep -imonitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp restart watchdog

Receive the following alertevent in HXConnect HX Controller VMHOSTNAME one or more configuredDNS servers not responding

CSCvt13947

35(2b)Delete any empty (Size 0) log filesunder varlogzookeeperversion-2ORvarlogzookeeperstandaloneversion-2

Zookeeper fails to start while Exhibitor isrunning however echo srvr returnsnothing

CSCvs21562

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

40(4e)1 De-commission the server

2 Power drain the server -REMOVE BOTH POWERCORDS ON THE BACK OFTHE SERVER FOR 10SECONDS THENREINSERTPOWERCORDS

3 Re-commission the server

M2 boot disk is missing from serverinventory after upgrade to server firmware40(4e) As a result server fails to boot toOS installed in the M2 disk The issuepersists after re-acknowledgement as wellas de-commission andre-acknowledgement of the server

CSCvs02466

Release Notes for Cisco HX Data Platform Release 4055

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Re-start the flusher manuallyMany disks taken offline by a fault ormanually resulting in ENOSPC oncluster

CSCvq32721

35(2e)Clean up fail over cluster and thenrecreate the fail over cluster Thereis no need to touch the HX storagecluster

[Hyper-V]When replicated DCwas usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2b)Manually install upgradescvmclient VIB andmake sure thatit matches with HXDP version

Upgrade from release HX 35(2a) to35(2x) 4X does not upgrade scvmclientRecovery point may not work

CSCvq54992

Management

30(1b)To recover the VM copy over thedata disks and attach them to thenew VM

[HyperV]VMshows up onmultiple nodesCSCvj22992

35(1a)1 Do not change MTU at ESXlevel on running cluster

2 Reset back to the original value

ChangingMTU from 9000 to 1500 on oneESX node causes storfs process to restarton all nodes

CSCvm99150

Release Notes for Cisco HX Data Platform Release 4056

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)After the installation (for newdeployments) or after an upgradeto 401a (for existingdeployments) perform thefollowing steps on all ControllerVMs

1 Edit the following tune files (onall Controller VMs)

optspringpathconfiglfftunes

optspringpathconfigvsi_16tbtunes

2 SetcleanerEnableSegSummaryCleaning

to false

3 After editing the above tunefiles

a SSH in to all the ControllerVMs

b Run storfstool -- -Z

c Run the below command andcheck the values Tune valueshould be true

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=true

d Type below commands andapplied the tunes changesdynamically

echo false gttmpstprocfssystemtunecleanerEnableSegSummaryCleaning

e Verify that tune values arechanged Run below commandValue should be false

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=false

f umount tmpstprocfs

A cluster with 8TB or 12TB disk drivesmay experience IO stalls for severalminutes after another node fails in thecluster

CSCvp23718

Release Notes for Cisco HX Data Platform Release 4057

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)1 Update the conf file manuallyto change to the link local addressto

2 Restart scvmclient on thecontroller and ESX

HX 35(2b) FlexVol to HX 40 CSIupgrade does not work FlexVol continuesto work

CSCvp26319

35(2a)Disable rebalance until node healsthen reenable rebalance

7-node cluster healing very slowlybecause rebalance change AMS reqs areblocking healing process

CSCvp62512

35(2a)NAWith moderate cluster storage capacityusage under site failover cluster clustermay experience an all paths (APD) downcondition

CSCvq49412

40(1a)NAThis product includes Third-partySoftware that is affected by thevulnerabilities identified by the followingCommon Vulnerability and Exposures(CVE) IDs

CVE-2016-0762 CVE-2016-6797CVE-2016-6816 CVE-2016-8735CVE-2017-5647 CVE-2017-12615CVE-2017-12616 CVE-2017-12617CVE-2017-7674 CVE-2018-1304CVE-2018-8014 CVE-2018-1336CVE-2018-8034 CVE-2018-11784CVE-2019-0232

CSCvp36364

35(2d)NAstcli cluster info command needs toprovide UCSM VIP address Currently itshows ucsm-hostcom

CSCvq11456

35(2a)No workarounds currently Thestale entries do not affect the dayto day operations of HyperFlex

HyperFlex upgrade validation failedbecause cluster has extraneous stNodes ininternal database

CSCvq32530

35(2a)NAWhen not in maintenancemode all drivesappear in HX Connect and get updated asexpected

When in maintenance mode extra drivesare shown in HX Connect with no slotnumbers

CSCvq42378

35(2b)Contact TAC to cleanup duplicatenode

Nodes duplicated in HX Connect aftercluster recreate

CSCvq61825

Release Notes for Cisco HX Data Platform Release 4058

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)NACurrently HyperFlex installs on Hyper-Vdo not contain the stcli security whitelistcommandset

CSCvq66245

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

Release Notes for Cisco HX Data Platform Release 4059

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)[Hyper-V] Converged node expansionfails for incoming node with VIC 14xx

CSCvq39523

Release Notes for Cisco HX Data Platform Release 4060

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

1 Set Jumbo frame inWindowsHypervisor for incomingnode

Get-NetAdapter storage-data-a|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

Get-NetAdapter storage-data-b|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

2 Verify all interfaces are setcorrectly to support Jumbo frames

Get-NetAdapter storage |Get-NetAdapterAdvancedProperty| RegistryKeyword -matchjumbo | ft -auto

You should receive the followingmessage

Name DisplayName DisplayValue

RegistryKeyword RegistryValue

---- ----------- ------------

--------------- -------------

vswitch-hx-storage-data Jumbo

Packet 9014 Bytes

JumboPacket 9014

storage-data-b Jumbo Packet

Bytes 9014 JumboPacket

9014

storage-data-a Jumbo Packet

Bytes 9014 JumboPacket

9014

3 Reboot only the incoming nodeor the node being expanded

4 Retry Cluster Expansion in

Release Notes for Cisco HX Data Platform Release 4061

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

HX Data Platform Software Versions for HyperFlex Witness Node

Witness Node VersionHyperFlex Release

10840(2b)

10840(2a)

10440(1b)

10440(1a)

Software Requirements for VMware ESXi

The software requirements include verification that you are using compatible versions of Cisco HyperFlexSystems (HX) components and VMware vSphere VMware vCenter and VMware ESXi

bull Verify that all HX servers have a compatible version of vSphere preinstalled

bull Verify that the vCenter version is the same or later than the ESXi version

bull Verify that the vCenter and ESXi versions are compatible by consulting the VMware ProductInteroperability Matrix Newer vCenter versions may be used with older ESXi versions so long as bothESXi and vCenter are supported in the table below

bull Verify that you have a vCenter administrator account with root-level privileges and the associatedpassword

For VIC1457 there is no support for ESXi 60Note

The below table applies for all of the following VMware vSphere Editions Enterprise Enterprise PlusStandard Essentials Plus ROBO

VMware vCenter VersionsVMware ESXi VersionsHyperFlex Version

60 U3 65 U3 67 U360 U3 65 U3 67 U340(2b)1

60 U3 65 U3 67 U360 U3 65 U3 67 U340(2a)2

60 U3 65 U3 67 U260 U3 65 U3 67 U2340(1b)

60 U3 65 U2 67 U260 U3 65 U2 67 U2440(1a)

1 Cisco HyperFlex Release 40(2) is the last major HyperFlex release that will support vSphere 60 (ESXiand vCenter) due to those versions reaching end of VMware general support on March 12 2020

2 Cisco HyperFlex Release 40(2) is the last major HyperFlex release that will support vSphere 60 (ESXiand vCenter) due to those versions reaching end of VMware general support on March 12 2020

3 Use of 67 U2 is not recommended see software advisory for CSCvq06952 and CSCvp88515 for furtherdetails

4 Use of 67 U2 is not recommended see software advisory for CSCvq06952 and CSCvp88515 for furtherdetails

Release Notes for Cisco HX Data Platform Release 4013

HX Data Platform Software Versions for HyperFlex Witness Node

Software Requirements for Microsoft Hyper-V

The software requirements include verification that you are using compatible versions of Cisco HyperFlexSystems (HX) components and Microsoft Hyper-V (Hyper-V) components

HyperFlex Software versions

The HX componentsmdashCisco HX Data Platform Installer Cisco HX Data Platform and Cisco UCSfirmwaremdashare installed on different servers Verify that each component on each server used with and withinthe HX Storage Cluster are compatible

bull Cisco HyperFlex M5 Converged nodesmdash For Hybrid (Cisco HyperFlex HX240c M5 HX220c M5)and All Flash (Cisco HyperFlex HXAF240cM5 HXAF220cM5) verify that Cisco UCSManager 40(2b)is installed HX 40 (1a) does not support Hyper-V on the All NVMe (HXAF220C-M5SN) nodes Fordetailed information on installation requirements and steps see theCisco HyperFlex Systems InstallationGuide on Microsoft Hyper-V

Table 4 Supported HyperFlex Software versions for M5 Servers on Hyper-V

M5 Recommended Server FirmwareHyperFlex Release

40(4h)40(2b)

40(4g)40(2a)

40(4e)40(1b)

40(4e)40(1a)

Release Notes for Cisco HX Data Platform Release 4014

Software Requirements for Microsoft Hyper-V

Table 5 Supported Microsoft Software versions

VersionMicrosoft Component

Windows Server 2016 Datacenter Core amp DesktopExperience

ForWindows Server 2016Datacenter Coreand Desktop Experience the Windows2016 ISO image should be Update BuildRevision (UBR) 1884 at a minimum

Note

Windows Server 2019Datacenter-DesktopExperienceis supported starting from HXDP 401(a) onwards

For Windows Server 2019 DesktopExperience theWindows 2019 ISO imageshould be Update Build Revision (UBR)107 at a minimum

Note

Windows Server 2019 DatacenterndashCore is notsupported currently

Also note that the following are currently notsupported

OEM activated ISOs and Retail ISOs are notsupported

Earlier versions ofWindows Server such asWindows2012r2 are not supported

Non-English versions of the ISO are not supported

Windows Operating System (Windows OS)

A Windows 2012 or later domain and forestfunctionality level

Active Directory

Supported Microsoft License Editions

The Microsoft Windows Server version that is installed on one or more HyperFlex hosts must be licensed asper Microsoft licensing requirements listed on Microsoft Licensing

Browser Recommendations

Use one of the following browsers to run the listed HyperFlex components These browsers have been testedand approved Other browsers might work but full functionality has not been tested and confirmed

Table 6 Supported Browsers

HX ConnectHX Data PlatformInstaller

Cisco UCS ManagerCisco IntersightBrowser

11 or higher11 or higher11 or higherNAMicrosoft InternetExplorer

70 or higher70 or higher57 or higher62 or higherGoogle Chrome

Release Notes for Cisco HX Data Platform Release 4015

Browser Recommendations

HX ConnectHX Data PlatformInstaller

Cisco UCS ManagerCisco IntersightBrowser

60 or higher60 or higher45 or higher57 or higherMozilla Firefox

NANA9 or higher10 or higherApple Safari

NANA35 or higherNAOpera

Notes

bull Cisco HyperFlex Connect

The minimum recommended resolution is 1024 X 768

bull Cisco HX Data Platform Plug-in

The Cisco HX Data Platform Plug-in runs in vSphere For VMware Host Client System browserrequirements see the VMware documentation at httpswwwvmwarecomsupportpubs

The HX Data Platform Plug-in is not displayed in the vCenter HTML client You must use the vCenterflash client

bull Cisco UCS Manager

The browser must support the following

bull Java Runtime Environment 16 or later

bull Adobe Flash Player 10 or higher is required for some features

For the latest browser information about Cisco UCSManager refer to themost recent Cisco UCSManagerGetting Started Guide

Cisco HX Data Platform Storage Cluster SpecificationsCluster Limits

bull Cisco HX Data Platform supports up to 100 clusters managed per vCenter as per VMware configurationmaximums

bull Cisco HX Data Platform supports any number of clusters on a single FI domain Each HX convergednode must be directly connected to a dedicated FI port on fabric A and fabric B without the use of a FEXC-series compute only nodes must also directly connect to both FIs B-series compute only nodes willconnect through a chassis IO module to both fabrics In the end the number of physical ports on the FIwill dictate the maximum cluster size and maximum number of individual clusters supported in a UCSdomain

The following table provides Cisco HX Data Platform Storage Cluster Specifications

Release Notes for Cisco HX Data Platform Release 4016

Cisco HX Data Platform Storage Cluster Specifications

Stretched Cluster (Availableon ESX Only)

Microsoft Hyper-VVMware ESXiNode

AllNVMe -HXAF220c-M5SN

HX240c-M5LHX220cM5

HX220cAF M5

HX240cM5

HX240cAF M5

HX240c-M5LHX220cM5

HX220cAF M5

HX240cM5

HX240cAF M5

HXAF220c-M5SN

AllNVMe -HXAF220c-M5SN

ESXi 65or 67only NotsupportedwithESXi 60orHyper-V

HX220cM5 Edge

HXAF220cM5 Edge

HX220cM4 Edge

HXAF220cM4 Edge

HX240c-M5LHX220cM5

HX220cAF M5

HX240cM5

HX240cAF M5

HX220cM4

HX220cAF M4

HX240cM4

HX240cAF M4

HXServers

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

C220M5C240M5B200M4B200M5

C240M5C220M5B200M4B200M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

mdashB200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

Compute-OnlyUCSB-SeriesC-SeriesServers

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

Convergednodesonly

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

SupportedNodes

Release Notes for Cisco HX Data Platform Release 4017

Cisco HX Data Platform Storage Cluster Specifications

Stretched Cluster (Availableon ESX Only)

Microsoft Hyper-VVMware ESXiNode

NA(requiresEnterpriseHXDP-PLicense)

NA(requiresEnterpriseHXDP-PLicense)

NA(requiresEnterpriseHXDP-PLicense)

Convergednodes3-16(12TBHDDoption isnotsupportedforHyperV)

Compute-onlynodes0-16

Convergednodes3-16

Compute-onlynodes0-16

NA(requiresEnterpriseHXDP-PLicense)

M4Convergednodes 3

M5Convergednodes23or 4

RequiresHXDP-ELicense

Convergednodes3-16(12TBdrive islimited toamaximumof 8nodes)

Compute-onlynodes0-16(12TBdrive islimited toamaximumof 8nodes)

Convergednodes3-32(76TBdriveconfigsarelimited tomaximumof 12drivesnodeon AF240)

Compute-onlynodes0-32

HXDP-SLicensedNodeLimits

11 ratioofHXDP-StoComputeonlynodes

(MinmdashMax)

Convergednodes2-16 perSite

Compute-onlynodes0-21 perSite

(up tomaxclustersize)

Convergednodes2-8 perSite

Compute-onlynodes0-16 perSite

(up tomaxclustersize)

Convergednodes2-16 perSite

Compute-onlynodes0-21 perSite

(up tomaxclustersize)

Convergednodes3-16(12TBHDDoption isnotsupportedforHyperV)

Compute-onlynodes0-16

Convergednodes3-16

Compute-onlynodes0-16

Convergednodes3-32

ComputeOnlynodes0-32

(up tomaxclustersize)

Convergednodes 3

(requiresHXDP-ELicense)

Convergednodes3-16(12TBdrive islimited toamaximumof 8nodes)

Compute-onlynodes0-32(12TBdrive islimited toamaximumof 16nodes)

Convergednodes3-32(76TBdriveconfigsarelimited tomaximumof 12drivesnodeon AF240)

Compute-onlynodes0-32

(up tomaxclustersize)

HXDP-PLicensedNodeLimits

12 ratioofHXDP-PtoComputeonlynodes

(MinmdashMax)

24 perSite 64percluster

24 perSite 48percluster

32 perSite 64percluster

32326434864MaxClusterSize

Release Notes for Cisco HX Data Platform Release 4018

Cisco HX Data Platform Storage Cluster Specifications

Stretched Cluster (Availableon ESX Only)

Microsoft Hyper-VVMware ESXiNode

212121111121mdash2121MaxComputetoConvergedratio

NoExpansion

Requires Enterprise license

Requires uniform expansion across both sites

Guidelines and Limitationsbull Starting with release 40(2a) SCVM is no longer needed on a Compute node

bull HX REST API Access Token Management ndash Applications leveraging HX REST APIs should re-useaccess tokens when making API calls Once obtained using the AAA Obtain Access Token API accesstokens are valid for 18 days (1555200 seconds) In addition AAA enforces rate limiting on ObtainAccess Token API requests in a 15 minute window auth can be invoked (successfully) a maximum of5 times A user is allowed to create a maximum of 8 unrevoked tokens Subsequent call to auth willautomatically revoke the oldest issued token tomake room for the new token Amaximum of 16 unrevokedtokens can be present in system In order to prevent brute-force attacks after 10 consecutive failedauthentication attempts a user account is locked for a period of 120 secondsFor more information seeCisco HyperFlex Systems REST API Reference guide

HxConnect makes use of AAA Authentication REST API for login and the above rate limit applies toHxConnect also

Upgrade Guidelines

The following list is a highlight of critical criteria for performing an upgrade of your HyperFlex system

bull Unsupported HX Data Platform 17x 18x 20 21x 25x and 26x clustersmdashUsers from anyversion prior to 26(1a) must step through an intermediate version before upgrading to 40 or later releasesIf you need to upgrade your environment from a Cisco HyperFlex HX Data Platform software releasethat is past the last date of support to the latest suggested release on the Cisco Software Download sitesee Cisco HyperFlex Systems Upgrade Guide for Unsupported Cisco HXReleases For more informationsee the Software Advisory for CSCvq66867 WARNING Only Use HXDP 26(1e) Upgrade PackageWhen Upgrading From HXDP 18(1a)-18(1e)

bull Hypercheck Health Check UtilitymdashCisco recommends running this proactive health check utility onyour HyperFlex cluster prior to upgrade These checks provide early visibility into any areas that mayneed attention andwill help ensure a seamless upgrade experience For more information see the HyperFlexHealth amp Pre-Upgrade Check Tool TechNote for full instructions on how to install and run Hypercheck

bull vSphere 67 Software AdvisorymdashDo not upgrade to Cisco HX Data Platform Release 40(1a) whenrunning ESXi 67U1 EP06 (build 11675023) Do not upgrade to 67U1 EP06 (build 11675023) ifrunning Cisco HX Data Platform Release 40(1a) See the Software Advisory CSCvo56350 for furtherdetails

Release Notes for Cisco HX Data Platform Release 4019

Guidelines and Limitations

The software build version posted at release will override any other local versions

bull Required vCenter upgrademdashFor enhanced security Cisco HX Data Platform Release 35(1a) or laterrequires the use of TLS 12 Therefore vCenter must be upgraded to 60 U3f or later before upgradingto Cisco HX Data Platform Release 35 or later In addition ESXi should be upgraded as required tomeet HX Data Platform compatibility requirements

bull Minimum HXDP version for upgrademdashHXData Platform clusters running 26(1a) or later may upgradedirectly to 40 using the HX Connect UI

bull Cluster ReadinessmdashEnsure that the cluster is properly bootstrapped and the updated plug-in is loadedbefore proceeding Manual cluster bootstrap is required for upgrade from a pre-35 release

bull Cluster ReadinessmdashEnsure that the cluster is properly bootstrapped and the updated plug-in is loadedbefore proceeding Manual cluster bootstrap is required for HX releases earlier than 35(1a) For moreinformation see theManual Bootstrap Upgrade Process in the Cisco HyperFlex Systems Upgrade Guidefor VMware ESXi Release 40 Do not skip this cluster bootstrap step it is required for all upgradesuntil HX Release 35(1a) Auto bootstrap is supported beginning with HX release 35(1a) For moreinformation see the Auto Bootstrap Upgrade Process from HX Connect UI in the Cisco HyperFlexSystems Upgrade Guide for VMware ESXi Release 40

Manual bootstrap is not supported on Intersight clusters

bull Initiating Upgrade―Use the HX Connect UI or CLI stcli commands when upgrading from 25(1a)or later releases Use either the CLI stcli commands or the HX Data Platform Plug-in to the vSphereWeb Client when upgrading from a pre-25(1a) release The vCenter plug-in should not be used forupgrades starting with the 25(1a) release The vCenter plug-in should not be used for upgrades startingwith the 25(1a) release

If the current cluster version is at 35(1a) or above you do not need to use the stcli command Directupgrade to 40 is possible

bull Complete your Upgrade―The self-healing (or rebalance) capability is disabled temporarily during theupgrade window If the upgrade fails you should complete the upgrade as soon as possible

bull ESXi and HXDP Compatibility―Ensure your cluster is running a compatible version of ESXi basedon the running the HXData Platform version (see the section Software Requirements for VMware ESXi)ESXi compatibility is determined by the major version and update release of ESXi It is generally bestto upgrade HXDP and ESXi together if combining the upgrade operations into a single optimized rebootWhen running a split upgrade first upgrade the HX Data Platform then proceed to upgrade ESXi

bull If Upgrading to vSphere 65

bull Certain cluster functions such as native and scheduled snapshotsReadyClones and Enter or Exit HX Maintenance Mode will not operatefrom the time the upgrade is started until the HX Data Platform upgrade iscomplete

bull After upgrading ESXi using the offline zip bundle use the ESX ExitMaintenance Mode option The HX Exit Maintenance Mode option doesnot operate in the vSphere Web Client until the HX Data Platform upgradeis complete

Note

Release Notes for Cisco HX Data Platform Release 4020

Guidelines and Limitations

bull vSphere 60 UpgradesmdashUsers on vSphere 60 migrating to 65 upgrade components in the followingorder

1 Upgrade HX Data Platform and UCS firmware

2 Upgrade HX Data Platform and ESXi

3 Upgrade HX Data Platform only first then upgrade ESXi or UCS firmware or both

bull M4 Server Firmware UpgradesmdashUpgrade server firmware to ensure smooth operation and to correctknown issues Specifically newer SAS HBA firmware is available in this release and is recommendedfor long-term stability

bull Users are encouraged to upgrade to 31(3c) C-bundle or later whenever possible

bull Users running C-bundle versions before 31(2f) must upgrade server firmware by performing acombined upgrade of UCS server firmware (C-bundle) to 31(3c) or later and HX Data Platform to25 Do not split the upgrade into two separate operations

bull If the cluster is already on 31(2f) C-bundle or later you may perform an HX Data Platform onlyor combined upgrade as required

bull M5 Server Firmware UpgradesmdashM5 generation servers must run firmware version 32(2d) or later

bull Firmware Downgrades mdashDowngrading UCSM from the HX-installer is not supported

bull M4M5 Mixed DomainsmdashA mixed domain occurs when a new separate M5 cluster is installed underthe same UCS domain that contains existing M4 clusters Under these conditions orchestrated UCSserver firmware upgrade will not operate until Cisco HX Data Platform Release 26 or later is installedon the M4 clusters Therefore it is best practice to first upgrade UCS server firmware to the latest 31(3)or 32(2) patch release before adding a new M5 cluster to the existing UCS domain Additionally any17 HX Data Platform clusters must first be upgraded before adding any new M5 clusters to the samedomain

bull Maintenance WindowmdashIf upgrading both HX Data Platform and UCS firmware you can select eithera combined or split upgrade through the vSphere HX Data Platform Plug-in depending on the length ofthe maintenance window Cisco UCSManager infrastructure upgrade is only supported using AutoInstalland the direct server firmware upgrade should be performed only through the upgrade orchestrationframework provided by the HX Data Platform Plug-in

bull Unsupported Self-Encrypting Drives (SEDs)mdashIf adding or replacing self-encrypting drives (SEDs)that have been recently qualified in newer versions of HX Data Platform insert the new drives only afterupgrading HX Data Platform to a compatible version All drives must be SED drives mixing SED andnon-SED is not supported

bullbull Enabling External Host AccessmdashWith Cisco HX Data Platform Release 40(1a) port 445 on themanagement network is blocked for enhanced security Note that prior to 40 port 445 port was openenabling external host access If you are upgrading to 40(1a) from a prior release and would like tocontinue external host access you can use a utility to open select hosts For more information aboutenabling external host access see the ConfiguringHyperFlex Share to SCVMM section in the InstallationGuide for Microsoft Hyper-V

Release Notes for Cisco HX Data Platform Release 4021

Guidelines and Limitations

Mixed Cluster Expansion Guidelinesbull Expanding existing M4 cluster with M5 converged nodes is supported

bull Expanding existing M5 cluster with M4 converged nodes is not supported

bull Expanding existing mixed M4M5 cluster with M4 or M5 converged nodes is supported

bull Adding any supported compute-only nodes is permitted with all M4 M5 and mixed M4M5 clustersusing the HX Data Platform 26 or later Installer Some example combinations are listed here manyother combinations are possibleExample combinationsExpand mixed M4M5 cluster with compute-only B200 C220 C240 M4M5Expand M4 cluster with compute-only B200 M5 C220 M5 C240M5

bull Only expansion workflow is supported to create a mixed cluster Initial cluster creation with mixedM4M5 servers is not supported

bull All M5 servers must match the form factor (220240) type (HybridAF) security capability (Non-SEDonly) amp disk configuration (QTY capacity and non-SED) of the existing M4 servers

bull HX220-M5 will use a maximum of 6 capacity disks (2 disk slots to remain empty) when mixedwith HX220-M4

bull HX Edge SED LFF Hyper-V and Stretched Clusters do not support mixed M4 and M5 clusters

Security FixesThe following security issues are resolved

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-13132 CVE-2019-9924CSCvq6313840(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-11719 CVE-2019-11727CVE-2019-11729

CSCvq7124040(2a)

Release Notes for Cisco HX Data Platform Release 4022

Mixed Cluster Expansion Guidelines

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-1125CSCvr0633940(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2015-9383 CVE-2018-14498CVE-2018-20406 CVE-2018-20852CVE-2019-10160 CVE-2019-13117CVE-2019-13118 CVE-2019-14287CVE-2019-14973 CVE-2019-15903CVE-2019-17546 CVE-2019-18197CVE-2019-18218 CVE-2019-5010CVE-2019-5094 CVE-2019-5481CVE-2019-5482 CVE-2019-9636CVE-2019-9740 CVE-2019-9947CVE-2019-9948

CSCvs0609440(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2017-13168 CVE-2017-18174CVE-2017-18216 CVE-2018-10876CVE-2018-10877 CVE-2018-10878CVE-2018-10879 CVE-2018-10880CVE-2018-10881 CVE-2018-10882CVE-2018-10902 CVE-2018-10938CVE-2018-12233 CVE-2018-12896CVE-2018-13053 CVE-2018-13094CVE-2018-13096 CVE-2018-13405CVE-2018-13406 CVE-2018-14609CVE-2018-14617 CVE-2018-14633CVE-2018-14734 CVE-2018-15572CVE-2018-15594 CVE-2018-16276CVE-2018-16658 CVE-2018-17182CVE-2018-17972 CVE-2018-18021CVE-2018-18690 CVE-2018-18710CVE-2018-6554 CVE-2018-6555CVE-2018-9363

CSCvp6501940(2a)

This vulnerability is due to insufficientHTML iframe protection An attackercould exploit this vulnerability bydirecting a user to an attacker-controlledweb page that contains amaliciousHTMLiframe A successful exploit could allowthe attacker to conduct clickjacking orother clientside browser attacks

NACSCvo9851640(2a)

Release Notes for Cisco HX Data Platform Release 4023

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerability is due to insufficientauthentication for the statistics collectionservice An attacker could exploit thisvulnerability by sending properlyformatted data values to the statisticscollection service of an affected deviceA successful exploit could allow theattacker to cause the web interfacestatistics view to present invalid data tousers

NACSCvj9558440(2a)

The vulnerability is due to insufficientCSRF protections for the web UI on anaffected device An attacker could exploitthis vulnerability by persuading a user ofthe interface to follow a malicious linkA successful exploit could allow theattacker to perform arbitrary actions withthe privilege level of the affected user

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvp2434340(2a)

Vulnerabilities with a version of cJSONidentified by CVE-2019-11834 andCVE-2019-11835

For more information see the relatedCisco Security Advisory

CVE-2019-11834 CVE-2019-11835CSCvq1994940(2a)

HX ESXi image patches for MCEPSCand TAA vulnerabilities with VMwareESXi identified by CVE-2018-12207 andCVE-20190-11135

For more information see the relatedCisco Security Advisory

CVE-2018-12207 CVE-2019-11135CSCvr5439840(2a)

CP networking vulnerabilities affectingthe Linux kernel identified byCVE-2019-11477 CVE-2019-11478CVE-2019-11479

For more information see the relatedCisco Security Advisory

CVE-2019-11477 CVE-2019-11478CVE-2019-11479

CSCvq1954640(2a)

Qualification of Microsoft securitypatches for vulnerabilities withMicrosoftHyper-V hypervisor identified withCVE-2018-12207 CVE-2019-11135

For more information see the relatedCisco Security Advisory

CVE-2018-12207 CVE-2019-11135CSCvr5439940(2a)

Release Notes for Cisco HX Data Platform Release 4024

Security Fixes

DescriptionCVEDefect IDRelease

OpenSSH vulnerabilities identified byCVE-2019-6111

For more information see the relatedCisco Security Advisory

CVE-2016-10708 CVE-2018-15473CVE-2018-20685 CVE-2019-6109CVE-2019-6111

CSCvp7646340(2a)

Multiple vulnerabilities associated withOpenSSL and LibNSS

For more information see the relatedCisco Security Advisory

CVE-2016-2105 CVE-2016-2106CVE-2016-2107 CVE-2016-2109CVE-2016-2176 CVE-2016-2177CVE-2016-2178 CVE-2016-2179CVE-2016-2180 CVE-2016-2181CVE-2016-2182 CVE-2016-2183CVE-2016-6302 CVE-2016-6303CVE-2016-6304 CVE-2016-6305CVE-2016-6306 CVE-2016-6307CVE-2016-6308 CVE-2016-7055CVE-2016-8610 CVE-2017-3731CVE-2017-3732 CVE-2017-3735CVE-2017-3736 CVE-2017-3737CVE-2017-3738 CVE-2018-0495CVE-2018-0732 CVE-2018-0734CVE-2018-0735 CVE-2018-0737CVE-2018-0739 CVE-2018-12384CVE-2018-12404 CVE-2018-5407CVE-2019-1559

CSCvp6667940(2a)

libpng vulnerability identified byCVE-2016-10087

For more information see the relatedCisco Security Advisory

CVE-2016-10087CSCvp6655540(2a)

Vulnerabilities associated with libjpg andlibpng

For more information see the relatedCisco Security Advisory

CVE-2014-9092 CVE-2016-3616CVE-2017-15232 CVE-2018-11212CVE-2018-11213 CVE-2018-11214CVE-2018-1152 CVE-2018-13785

CSCvp3458640(2a)

Vulnerabilities associated with Glibidentified by CVE-2018-16428 andCVE-2018-16429

For more information see the relatedCisco Security Advisory

CVE-2018-16428 CVE-2018-16429CSCvp3120740(2a)

Multiple vulnerabilities associated withcurl expat python 27 python 35 36and 37 freetype and giflib

For more information see the relatedCisco Security Advisory

CVE-2019-15133 CVE-2019-15903CVE-2019-5010 CVE-2019-5481CVE-2019-5482 CVE-2019-9636CVE-2019-9740 CVE-2019-9947CVE-2019-9948

CSCvr3690340(2a)

Release Notes for Cisco HX Data Platform Release 4025

Security Fixes

DescriptionCVEDefect IDRelease

Multiple third party softwarevulnerabilities

For more information see the relatedCisco Security Advisory

CVE-2019-14379 CVE-2019-12384CVE-2019-14439

CSCvq9203240(2a)

Vulnerabilities associated with elfutils

For more information see the relatedCisco Security Advisory

CVE-2018-16062 CVE-2018-16402CVE-2018-16403 CVE-2018-18310CVE-2018-18520 CVE-2018-18521CVE-2019-7149 CVE-2019-7150CVE-2019-7665

CSCvq4325040(2a)

Vulnerabilities associated with vim

For more information see the relatedCisco Security Advisory

CVE-2017-5953 CVE-2019-12735CSCvq4323040(2a)

Vulnerabilities associated with sqlite3

For more information see the relatedCisco Security Advisory

CVE-2016-6153 CVE-2017-10989CVE-2017-13685 CVE-2017-2518CVE-2017-2519 CVE-2017-2520CVE-2018-20346 CVE-2018-20505CVE-2018-20506 CVE-2019-8457CVE-2019-9936 CVE-2019-9937

CSCvq4321340(2a)

Vulnerabilities associated with glib20

For more information see the relatedCisco Security Advisory

CVE-2018-20843CSCvq4320940(2a)

Vulnerabilities associated with expat

For more information see the relatedCisco Security Advisory

CVE-2018-20843CSCvq4320540(2a)

Vulnerabilities associated with bzip2

For more information see the relatedCisco Security Advisory

CVE-2016-3189 CVE-2019-12900CSCvq4319440(2a)

Mulitple Vulnerabilities associated wtihNodeJS

For more information see the relatedCisco Security Advisory

CVE-2018-12115 CVE-2018-0734CVE-2018-5407 CVE-2018-12120CVE-2018-12121 CVE-2018-12122CVE-2018-12123 CVE-2018-12116CVE-2019-5737 CVE-2019-5739

CSCvq1069440(2a)

Vulnerabilities associated with PalletsJinja strformat_map and identified byCVE-2019-10906 and CVE-2016-10745

For more information see the relatedCisco Security Advisory

CVE-2019-10906 CVE-2016-10745CSCvq1038840(2a)

Release Notes for Cisco HX Data Platform Release 4026

Security Fixes

DescriptionCVEDefect IDRelease

Multiple Security Vulnerabilities

For more information see the relatedCisco Security Advisory

CVE-2014-8501 CVE-2014-9939CVE-2015-9262 CVE-2016-10087CVE-2016-2226 CVE-2016-4487CVE-2016-4488 CVE-2016-4489CVE-2016-4490 CVE-2016-4491CVE-2016-4492 CVE-2016-4493CVE-2016-6131 CVE-2018-10963CVE-2018-13785 CVE-2018-17100CVE-2018-17101 CVE-2018-18557CVE-2018-18661 CVE-2018-7456CVE-2018-8905

CSCvq0917840(2a)

Vulnerabilities associated withlibseccomp

For more information see the relatedCisco Security Advisory

CVE-2019-9893CSCvq0756840(2a)

Vulnerabilities associated withgdk-pixbuf

For more information see the relatedCisco Security Advisory

CVE-2017-12447CSCvq0675540(2a)

Vulnerabilities associated withpython-crypto

For more information see the relatedCisco Security Advisory

CVE-2018-6594CSCvp9381740(2a)

Vulnerabilities associated with wget

For more information see the relatedCisco Security Advisory

CVE-2018-20483 CVE-2019-5953CSCvp8672140(2a)

Multiple vulnerabilities associated withPython Crypto

For more information see the relatedCisco Security Advisory

CVE-2018-6594CSCvp6674840(2a)

Vulnerabilities associated withsensible-utils

For more information see the relatedCisco Security Advisory

CVE-2017-17512CSCvp6673440(2a)

Multiple vulnerabilities associated withPython

For more information see the relatedCisco Security Advisory

CVE-2018-1000030CSCvp6668940(2a)

Release Notes for Cisco HX Data Platform Release 4027

Security Fixes

DescriptionCVEDefect IDRelease

Multiple patch vulnerabilities identifiedby CVE-2016-10713

For more information see the relatedCisco Security Advisory

CVE-2016-10713CVE-2018-1000156 CVE-2018-6951

CSCvp6667240(2a)

Vulnerabilities associatedwith Little CMSidentified by CVE-2016-10165

For more information see the relatedCisco Security Advisory

CVE-2016-10165 CVE-2018-16435CSCvp6666440(2a)

Vulnerabilities associated with TenableScan identified byCVE-2011-5325CVE-2019-11068CVE-2016-7076CVE-2017-1000368

For more information see the relatedCisco Security Advisory

CVE-2011-5325 CVE-2016-7076CVE-2017-1000368CVE-2019-11068

CSCvp6474640(2a)

Vulnerabilities associated with libxml2294 and earlier as used in XMLSec1223 and earlier and other products andidentified by CVE-2016-9318

For more information see the relatedCisco Security Advisory

CVE-2016-9318 CVE-2017-16932CVE-2017-18258 CVE-2018-14404CVE-2018-14567

CSCvp3479240(2a)

Vulnerabilities associated with LFTPRemote File Names Unauthorized AccessVulnerability and identified byCVE-2018-10916

For more information see the relatedCisco Security Advisory

CVE-2018-10916CSCvp2926640(2a)

Vulnerabilities associated withCryptographyDeprecationWarning signerand verifier have been deprecated

For more information see the relatedCisco Security Advisory

CVE-2018-7750CSCvo3409740(2a)

Multiple Vulnerabilities in python python27 35

For more information see the relatedCisco Security Advisory

CVE-2019-16056CSCvr3979340(2a)

Release Notes for Cisco HX Data Platform Release 4028

Security Fixes

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toexecute commands as the root user

The vulnerability is due to insufficientinput validation An attacker could exploitthis vulnerability by connecting to thecluster service manager and injectingcommands into the bound process Asuccessful exploit could allow the attackerto run commands on the affected host asthe root user

Cisco has released software updates thataddress this vulnerability There areworkarounds that address thisvulnerability

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvq2417640(1b)35(2g)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

The vulnerability is due to an unprotectedlistening interface An attacker couldexploit this vulnerability by connectingto the listening interface and injectingcommands to the bound process Anexploit could allow the attacker to runcommands on the affected host as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvj9560640(1b)35(2g)

Release Notes for Cisco HX Data Platform Release 4029

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerabilities associated with theJVM18U121memory leak found duringVC alarms (concurrent 40 calls usingREST API)

CVE-2017-10053 CVE-2017-10067CVE-2017-10074 CVE-2017-10078CVE-2017-10081 CVE-2017-10087CVE-2017-10089 CVE-2017-10090CVE-2017-10096 CVE-2017-10101CVE-2017-10102 CVE-2017-10107CVE-2017-10108 CVE-2017-10109CVE-2017-10110 CVE-2017-10111CVE-2017-10115 CVE-2017-10116CVE-2017-10118 CVE-2017-10135CVE-2017-10176 CVE-2017-10193CVE-2017-10198 CVE-2017-10243CVE-2017-10274 CVE-2017-10281CVE-2017-10285 CVE-2017-10295CVE-2017-10345 CVE-2017-10346CVE-2017-10347 CVE-2017-10348CVE-2017-10349 CVE-2017-10350CVE-2017-10355 CVE-2017-10356CVE-2017-10357 CVE-2017-10388CVE-2017-3509 CVE-2017-3511CVE-2017-3526 CVE-2017-3533CVE-2017-3539 CVE-2017-3544CVE-2018-2579 CVE-2018-2582CVE-2018-2588 CVE-2018-2599CVE-2018-2602 CVE-2018-2603CVE-2018-2618 CVE-2018-2629CVE-2018-2633 CVE-2018-2634CVE-2018-2637 CVE-2018-2641CVE-2018-2663 CVE-2018-2677CVE-2018-2678 CVE-2018-2783CVE-2018-2790 CVE-2018-2794CVE-2018-2795 CVE-2018-2796CVE-2018-2797 CVE-2018-2798CVE-2018-2799 CVE-2018-2800CVE-2018-2814 CVE-2018-2815CVE-2018-2952 CVE-2018-3136CVE-2018-3139 CVE-2018-3149CVE-2018-3150 CVE-2018-3169CVE-2018-3180 CVE-2018-3183CVE-2018-3214 CVE-2019-2422

CSCvo8899740(1b)

Tomcat and Nginx logs are not beingcollected in the support bundle generatedthrough HX Connect in Release 35

NACSCvm5803140(1b)

The vulnerabilities associated with thelibmspack software package versionincluded in Cisco HX Data Platform

CVE-2018-18584

CVE-2018-18585

CSCvn3511940(1a)

Release Notes for Cisco HX Data Platform Release 4030

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerabilities associated withFasterXML Jackson-Databind TimeValueField Denial of Service

CVE-2018-14719

CVE-2018-14720

CVE-2018-1000873

CVE-2018-14721

CVE-2018-19360

CVE-2018-19362

CVE-2018-19361

CVE-2018-14718

CSCvn8228240(1a)

The vulnerabilities associated with theOpenSSL Protocol software packageversion included in Cisco HX DataPlatform

CVE-2013-3587CSCvo0505440(1a)

The vulnerabilities associated withThird-Party Software Denial of Service

CVE-2018-16487

CVE-2018-19361

CSCvo2781840(1a)

The vulnerabilities associated withLinuxkernel for Ubuntu 1710

CVE-2018-1092

CVE-2018-7492

CVE-2018-8087

CVE-2018-1068

CVE-2018-8781

CSCvm5314935(2a)

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)25(1d)stcli services dns remove should remove the DNS serverinfo from the interface files

CSCvs70967

40(2b)40(1b)HX 40(1b) new deployment from Intersight productioncloud

User receives following error while enabling CSI(Kubernetes integration)

Failure occurred during volume_access_enable Errorwas Nonzero exit code 1

CSCvt10522

40(2b)40(2a)Receive the following alertevent in HX Connect HXController VM HOSTNAME one or more configuredDNS servers not responding

CSCvt13947

Release Notes for Cisco HX Data Platform Release 4031

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)40(2a)API UI CLI show few Drive slots Empty after upgradeto 402a

The drives are seen in the sysmtool lsscsi and stclicommand outputs

The cluster is healthy with no errors

CSCvt14914

40(2b)35(2d)After successfully changing the DNS server on the HXcontroller we still can see the original DNS entry thatwas added during the deployment

CSCvs69154

40(2b)35(2e)HyperFlex Connect and VCenter show APD alarms forHX and non HX datastores This implies there is an issuewith HyperFlex when there may not be

CSCvs30080

40(2b)40(2a)An issue where all the disks in the node get locked afterrebooting the node

We successfully unlocked using sed-clientsh -Ucommand but wanted to test with another reboot anddrives locked again

CSCvs74286

40(2b)40(2a)When running stcli license commands on HyperFlexthe following error is seen

rootSpringpathControllertmp stcli license show all

Show smart licensing failed

Smart Agent is not ready please wait a minute and tryagain

CSCvt13929

40(2b)40(2a)After upgrading to HX 40(2a) the first RoboEdge nodethat gets rebooted then ignores persistent drivesPIDHX-HD24TB10K4KN

Cluster will be in degraded state and have reducedcapacity

CSCvt20144

40(2b)35(2b)Zookeeper fails to start while Exhibitor is runninghowever echo srvr returns nothing

CSCvs21562

40(2b)40(2a)When performing an HyperFlex upgrade a validationwarning may occur due to the host not having EnterprisePlus or Enterprise hypervisor licensing

Upgrade Validation Warning

ESXi host esx1labtest should be configured withVMware Enterprise license for upgrade to continue

CSCvs91787

Release Notes for Cisco HX Data Platform Release 4032

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)35(2g)Cluster expansion fails at Config Installer stage whenthe root and admin password for the storage controller(SCVM) are different

CSCvs69317

40(2b)35(2g)Panic while upgrading ESXiCSCvt06983

40(2b)40(2a)A cluster node running may hang in the Linux kernelThis is classified as an oops and a deviation from theexpected behavior

CSCvs54285

40(2b)35(2g)Rebalance failing on 10+10 node HX 35(2g) stretchcluster

CSCvs69007

40(2b)35(2d)The cluster becomes inaccessible to the IOVisorCSCvr54687

40(2b)40(2a)Panic on the storage controllerCSCvt61297

40(2b)40(2a)The size of support bundle is very large when collectedwith storfs-support command

CSCvt63306

40(2b)40(2a)storfs crash associated withFileSystemUsageWarningEvent orFileSystemUsageAlertEvent due to high usage ofvarstv above 80

CSCvt72807

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

40(2a)35(2d)In normal scenarios HX upgrade gets pendingACK afterupdating service profile In this case UCS only upgradewas stalled because after updating service profile HXupgrade was not getting pending ACK and it was alwayswaiting state

CSCvo39912

40(2a)35(1a)When using motherBoardReplace-12 to clean ZK fromold stNodepNodes resulted in unmounted datastore andmaking the size of the Hyperflex Datastores 0 resultingin all VMs in the cluster going offline

CSCvq39471

40(2a)26(1e)Upgrading cluster and getting error

Failed upgrade validations Checking vCenterconfiguration Reason Upgrade validations failed DRSFault Insufficient resources to satisfy configured failover

CSCvm77294

Release Notes for Cisco HX Data Platform Release 4033

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)The cluster expand fails on a node with DR replicationconfigured

When a HX cluster which has DR replication configuredis expanded we see the installer UI pulling in thereplicationVLAN information instead of themanagementVLAN information Even if we change that informationto the correct mgmt VLAN id and name it it does notseem to work as the node is configured with the VLANof the replicationVLAN in ESXi This leads to the failureof the node add with the host unreachable error

CSCvo70650

40(2a)35(1a)Customer reported that server didnt complete thefirmware upgrade automatically one by one

The user manually needs to put the host into maintenancemode and they manually acknowledged the pendingrequests to finish the UCS firmware upgrade

Maintenance policy is set to default (User-ack) as perthe design

CSCvo91624

40(2a)35(1i)When the cluster is in Failed state and stcli noderemove is attempted the output shows successful thoughthe node failed to get removed from the cluster

CSCvo93017

40(2a)35(1a)HyperFlex cluster upgrade may fail during validationwith a DRS Validation failed error

CSCvp31021

40(2a)35(2b)HX cluster expansion will fail with the following errormessage

MAC address pool configuration failure150[ErrorDescription] bad address block range definitioncollision

CSCvp58318

40(2a)35(2b)Memory usage by carbon cacheCSCvq34873

40(2a)35(2g)Upgrading ESXi cluster fails with error Nodemaintenance mode failed

CSCvr03240

40(2a)35(2e)storfs process does not automatically start on Exit HXMaintenance Mode or other tasks which restart StorageController

CSCvr88978

40(2a)35(2a)Post install script fails with message

Failed to execute ipmitool on HX node

CSCvp10707

40(2a)40(1a)HyperFlex expansionworkflow doesnt pull VLANnamecorrectly

CSCvp46539

Release Notes for Cisco HX Data Platform Release 4034

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)During HX cluster deployment - Validate ClusterCreation phase - HX Installer might fail with the error

fromvarlogspringpathstDeploylog

CSCvq45087

40(2a)35(2b)HX Installer fails to login to the SOL for configuringESX hosts

CSCvq91380

40(2a)35(2d)If DRS is enabled with configuration parameters otherthen fully automated ie manual or partially automatedthan this will change to fully automated during clusterexpansion

CSCvr44222

40(2a)35(1a)One of the validation task name has a typo The taskname is Validating node for mixed of SED and non-SEDdisks

It should be Validating node for mix of SED andnon-SED disks

CSCvp82175

40(2a)30(1i)This combination of using smaller NVMe drives is notsupported in expand operation For example if the drivesare only 375GB and they cannot be added to an existingcluster with larger caching SSD

CSCvo12359

40(2a)35(1a)HyperFlex includes a version of OpenSSL that is affectedby the vulnerabilities identified by the followingCommon Vulnerability and Exposures

CVE-2018-0495

CSCvp66679

40(2a)30(1e)SDK invoke exception when taking a native snapshot ona VM with GPU pci passthrough attached

CSCvo00511

40(2a)30(1i)Node replacement script fails due to EAM errorCSCvo62867

40(2a)35(1a)While pairing a cluster (HX release gt= 35) and cluster(HX release lt 35) remote replication network test failson the cluster (HX release lt 35)

CSCvo79760

40(2a)30(1b)Fixed the issue in the latest support-workflow bundleThis should not be hit anymore

CSCvo87061

40(2a)35(2a)On a 3 node cluster the MBreplace script fails as thescript looks for HEALTHY cluster status

CSCvo87080

40(2a)35(2a)MbReplace script (tar) hangs when running on 352b or40

CSCvp12359

Release Notes for Cisco HX Data Platform Release 4035

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)HX replication cleanup failing with error NA 35(2a) -INFODR state is not clean

CSCvp63958

40(2a)40(1a)The check_stig_parameters API incorrectly shows thecompliance state of the cluster as not STIG compliant

CSCvp66277

40(2a)35(2c)The snapshot creation on CBT enabled VM fails witherror Failed in vmreparent vmkfstools clone1

CSCvq06952

40(2a)35(2a)HyperFlex GUI when setting up remote DR-replicationpartnerpeer should show remote replication partner TCPport reachability test results

This will provide quick upfront connectivity results thatfrequently takes customer a while to manually check andconfirm

If reachability tests fail an info window and HX alertshould trigger that list the destination peer IP and thespecific port(s) that could not be reache

CSCvo60587

40(2a)35(2b)If you have more than one IP pool configured for areplication network and local replication network testsare failing you might be running into this issue

CSCvr67130

40(2a)35(2a)Skip task option in MBreplace and Software redeployscripts The scripts should have the option to skip tasksas user inputs and not by modifying the json file

CSCvp05204

40(2a)40(1a)Error noticed on SCVM Console print_req_error IOerror dev fd0 sector 0 on HyperV

It doesnt come up on SSH session

CSCvq34357

40(2a)30(1d)The VMNetwork switch is already created as part of theinstall process

Ideally the ip address assignment should be done duringthe initial install process as well No post install stepsshould be required altrnatively it could be done as partof a post install script

CSCvk36222

40(2a)25(1c)storfs service impacted during HX upgrade from HX21(1) to 25(1c)

HX node which is not being upgraded storfs servicestopped causing outage

CSCvg53223

40(2a)35(1b)HX Stretch Cluster Witness VM reverts to DHCP onreboot

CSCvp37536

40(2a)35(2b)HyperFlex Stretch Cluster Witness does not getprogrammed with an NTP server

CSCvp46578

Release Notes for Cisco HX Data Platform Release 4036

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2b)HyperFlex cluster does not heal after maintenance onnodeserver

CSCvr18528

40(2a)40(2a)High percentage of packet loss on the eth1 data interface9K MTU

CSCvr97089

40(2a)35(2f)The HyperFlex cluster healing state is stuck at 87 andis unable to progress

CSCvr16760

40(2a)40(2a)In some circumstances two replications can occur forthe same VM

From the GUI the replication status for this VM willshow stuck in an In Progress state but there are nojobs stuck in replication layer Subsequent Replicationsshould succeed

CSCvq22898

40(2a)35(2a)After an unexpected power outage the hyperflex clusteris unable to come up correctly

All processes are running time is synced and all nodescan vmkping each other as required

CSCvp13990

40(2a)40(1a)When specifying IP address instead of node ID stclinode maintenanceMode commands fail

CSCvp33657

40(2a)40(1a)2 node edge cluster if experiencing Intersightarbitratorconnectivity issues should NOT allowHXMaintenancemode to initiate as this will bring the cluster down

CSCvr31573

40(2a)35(2c)Old files in varsupportZKTxnlog are not purged withthe daily zklog-cleanup cron job

CSCvr89066

40(2a)35(2a)When a node is in MaintenanceMode any disk removalor replacement will be reflected in UI only after the nodeis brought back frommaintenancemode This is becausestorfs is not running on the node in maintenance andwill not be able to detect disk activities until it is broughtout of Maintenance Mode

CSCvo86431

40(2a)35(2c)Cluster might go offline and stop serving IO requestsrootcvm~ stcli cluster storage-summary--detailGet cluster storage summary failedjavanetConnectException Connection refused19216814210010207

CSCvr01645

40(2a)35(2d)A stretch cluster deployment fails with a messageFormatting nodes Node in Use on the Hyperflex installerand crmZoneType value is seen as 1 in storfscfg oroptspringpathconfigstretchtunes

CSCvq80340

Release Notes for Cisco HX Data Platform Release 4037

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)30(1c)During upgrade we upload the upgrade packagedownloaded from Ciscocom Upgrade packages are offile type tgz hxconnect acceptsstorfs-packages-401a-33028tar file a different type ofcompression and allows upgrade to start and fail whenit cannot decompress the file for bootstrap and scvmclientupgrade on ESXi

CSCvq63888

40(2a)40(2a)While attempting to install hyperflex via intersight thefollowing error is seen

Failed in Task Add hosts to vCenter Cluster with ErrorTry adding hosts manually to vCenter and retry Failedto get addressFailed to host by namelookupHostnamecompanycom on 000053 read udp12700145430-gt12700153 read connection refused

CSCvr52098

40(2a)35(2a)CoreAPI call may time out for cluster management APIwhile doing cluster create validation

CSCvo92952

40(2a)35(2d)Validation fails with Mixed mode expansion checkdue to Enhanced vMotion Compatibility Incompatibilitywhen the ESXi version is different

CSCvq95460

40(2a)30(1i)This failure is seen in installer when custom workflowis used hypervisor configuration + deploy (clean diskpartition=no) at Installing software packages on storagecontroller VM

CSCvr66309

40(2a)35(2d)The HX Installer apparently replaced a comma bysemicolon on the VLAN ID range while exporting theconfiguration to a JSON file

CSCvq93831

40(2a)40(1a)When using unsupported browsers HX Connect userswill not be able to login These users see an errorindicating unauthorized user

CSCvp22693

Release Notes for Cisco HX Data Platform Release 4038

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)40(1a)A vulnerability in the web-based management interfaceof Cisco HyperFlex Software could allow anunauthenticated remote attacker to conduct a cross-siterequest forgery (CSRF) attack on an affected system

The vulnerability is due to insufficient CSRF protectionsfor the web UI on an affected device An attacker couldexploit this vulnerability by persuading a user of theinterface to follow a malicious link A successful exploitcould allow the attacker to perform arbitrary actions withthe privilege level of the affected user

Cisco has released software updates that address thisvulnerability There are no workarounds that address thisvulnerability

This advisory is available at the following link

httpstoolsciscocomsecuritycentercontentCiscoSecurityAdvisorycisco-sa-20190807-hypflex-csrf

CSCvp24343

40(2a)35(2d)Add message to HX Connect Progress flow to notAcknowledge Pending Activities and reboot the serverson UCSM

HX Connect is doing a controlled rolling server upgradein the background

CSCvq22844

40(2a)40(1b)Compute Node expand operation fails due to incorrectcluster name

CSCvr43786

40(2a)26(1e)RF-2 cluster shutdown during data resynchronizationafter a node failure (non storfs) followed by multipledisk read failures and a HardBlacklist of a disk

CSCvp41241

40(2a)40(1a)HyperFlex Edge nodes do not properly set the ESXihostname during deployment

CSCvo13143

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4039

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4040

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1b) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401b-33133p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4041

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4042

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1b)40(1a)While running the HyperFlex installer with WindowsServer Hyper-V cluster creation process fails with theerror Failure occurred during ClusterCreation process Unable to post thecontent to the downstream This symptomis encountered while deploying HyperFlex cluster withHyperFlex nodes configuredordered with Cisco VIC1457 MLOM (PID HX-MLOM-C25Q-04) andWindows Server Datacenter or Core with Hyper-Vversion 2016 or 2019

CSCvp64140

Release Notes for Cisco HX Data Platform Release 4043

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)35(2b)Adding Micron 5200 drive to a cluster fails to increasecluster capacity

CSCvo69067

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4044

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4045

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1a) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401a-33028p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4046

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4047

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1a)30(1d)Cluster instability when disks of different sector sizeplaced in HX node

CSCvk17250

40(1a)

35(2c)

35(1a)When logged in using a local HX user account insteadof a Virtual Center account an error message appearsintermittently indicating Virtual Centerunreachable or Resource informationcannot be updated when VC is reachable

CSCvo36198

40(1a)

35(2b)

30(1d)HXDP does not work with EMC RecoverPoint - needsto support VMware API (FSS-Readdir)

CSCvk38003

Release Notes for Cisco HX Data Platform Release 4048

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)

35(2a)

30(1i)

25(1c)

25(1d)

Zookeeper (Exhibitor) process on HyperFlex StorageController VMs may fail to respond to requests whenvarzookeeper has low or no free space

CSCvm90352

40(1a)26(1c)Use Asynchronous consolidation for HX SnapshotsCSCvn02151

40(1a)35(2b)HX Quiesced Snapshot for Backup VendorsCSCvo90713

40(1a)35(2a)The cluster creationcluster expansion workflow stopswith the following error message at the validation step

FIRMWARE-Check UCSC-SAS-M5HD

FIRMWARE-Check UCSC-SAS-M5HD Required00000029000000320000003500000050 Found00000058

Action Needed Update the Controller Firmware toRequired Version

CSCvn17787

35(2b)30(1d)Kernel migration fails when a local datastore is searchedfor in the ESXi host

CSCvn73127

35(2a)26(1b)A node shuts down when two disks are replaced (acaching disk and another capacity disk) where thecapacity disk is inserted first and then the caching diskis inserted

CSCvk46364

Hyper-V

40(1a)35(2a)Cluster expansion may fail with an error code 500 -operation timed out

CSCvn28721

35(2b)35(2a)During upgrade remove the VLAN on team created foruser vSwitch During fresh installation only one VLANtag is set to the vSwitch and team although multipleVLANs were entered

CSCvn54300

35(2b)35(2a)While upgrading a Hyper-V cluster on account of a rarerace condition between the stUpgradeService andZookeeper servers the upgrade orchestration throws anupgrade validation error and the upgrade process isaborted

CSCvn60486

Release Notes for Cisco HX Data Platform Release 4049

Resolved Caveats in Release 40(1a)

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Startrestart EAM service invCenter

When EAM service is not runningvCenter registration is failing

CSCvt55712

35(2a)Contact Cisco TAC if you believeyou are seeing similar conditions

HyperFlex Stretch Cluster saw poorapplication performance due to writelatency cluster remained unhealthy andrebalance was stuck

This defect is currently being used forinvestigative purposes

CSCvt45344

35(2g)Remove log files fromvarlogspringpath folder when thelog files are not getting rotatedStop the relevant service removelarge log files fromvarlogspringpath folder andrestart the service

varstv folder in Controller VM maybecome full

CSCvt36374

40(2b)Restarting the stMgr will initializeit again

In a Cluster configured with DR networkStMgr may not get initialized and stuckin a deadlock while enabling the IPTablerules (can be verified from the stMgrlog)

CSCvt89709

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Click recover for the selectedVM again under the standalonemode This will move the VM torecovered state

When the user recovers a VM protectedin a group it along with other VMs in thegroup move to recovering state instandalonemode The selectedVM shouldmove to recovered state

CSCvs75553

35(2e)Clean up the fail over cluster andthen recreate the fail over clusterNo need to touch the HX storagecluster

Hyper-V When replicated DC was usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2d)40(1a)

Keep cluster utilization as low aspossible To avoid read-only statekeep below 92

HyperFlex clusters storage utilization isabove 92 usable capacity

CSCvr20922

Release Notes for Cisco HX Data Platform Release 4050

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1b)NAAll the disks in the node got locked afterrebooting the node

We successfully unlocked usingsed-clientsh -U command but wantedto test with another reboot and driveslocked again

CSCvs74286

35(2g)To recover the VM copy over thedata disks and attach them to thenew VM

After enable encryption HX connectencryption status showing Caution butall of the disks enable encryptionsuccessful

-Self Encrypting Drives Service isrunning on all of the nodes

-There is no error message of encryption

- USB0 interface is up

-All SED disks showing supported 1enabled 1 locked 0

CSCvs93245

40(1b)Enable kernelpanic_on_oops as apersistent configuration This willcause the node to panic and rebootimmediately

A cluster node running HX release40(1b) may hang in the Linux kernelThis is classified as an oops and adeviation from the expected behavior

CSCvs54285

Release Notes for Cisco HX Data Platform Release 4051

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Enabling K8iscsi stack in Hx clusterCSCvs41324

Release Notes for Cisco HX Data Platform Release 4052

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Steps to enable K8 on HX cluster

Perform following steps on ALLthe controller VMs

1 Updateetcinitscvmclientconf toenable tune iscsiEnable totrue

sed -iesiscsiEnable=falseiscsiEnable=trueetcinitscvmclientconf

2 Run following initctlcommand to reloadconfiguration

initctl reload-configuration

3 Restart scvmclient process

stop scvmclient startscvmclient

4 Run following initctlcommand to mount SYSTEMDS

initctl emit --no-waitsystem-datastore-created

5 Verify that iscsiEnable tuneis set to true using thefollowing command

ps -eaf | grep scvmclient |grep -v grep

root 6241 1 1 Dec06 014007optspringpathstorfs-corescvmclient-T logEnabled=false -TlogSyslogEnabled=true -TlogEchoToScreen=false -TstatLoggingToFile=false -TstatLoggingToSyslog=true-TlogDir=varlogspringpath-TnfsBackendServerList=1010748100-T iscsiEnable=true -TiscsiUseAsync=true -T

Release Notes for Cisco HX Data Platform Release 4053

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

iscsiConfigFilePath=nfsSYSTEMistgtconfps -eaf | grep scvmclient |grep -v grep

35(2e)Following procedure can be usedto check NFS Queue Depth andincrease if needed

rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth256 lt- Low value[rootHXESXI1] vsish -e setvmkModulesnfsclientmnt[DATASTORE]propertiesmaxQDepth 1024[rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth1024 lt- Optimalvalue

This requires an ESXi host rebootto take effect

Please place the node in HyperflexMaintenanceMode and gracefullyreboot the node for the changes tobe applied

HyperFlex Datastore NFS Queue Depthshows as 256 which can lead toperformance (including latency) issues

CSCvr83056

40(1b)This is cosmetic The error will goaway once the cluster returns tohealthy status

When a single node is offline in clusterstcli cluster storage-summary shows twonodes unavailable rootSCVM~ stclicluster storage-summary

CSCvs47735

35(2g)Modify both the root and adminpassword for the SCVMs to be thesame

stcli security password set -uadmin

Cluster expansion fails at Config Installerstage when the root and admin passwordfor the storage controller (SCVM) aredifferent

CSCvs69317

35(2d)NAAfter successfully changing the DNSserver on the HX controller we still cansee the original DNS entry that was addedduring the deployment

CSCvs69154

Release Notes for Cisco HX Data Platform Release 4054

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Customers are strongly encouragedto work with TAC in order toidentify and remediate this issue

1 Change getcluster from 2m to8m inoptspringpathstorfs-mgmtstMgr-10confapplicationconfon all nodes

2 Restart stMgr on all nodes

You may see this error message after afailed upgrade or other task such asattempting to enter a node into HXmaintenance mode

CSCvs53555

40(2a)Run the following command oneach storage controller VM as root

grep -i monitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp sed -iesmonitor_dns_serverstruemonitor_dns_serversfalseoptspringpathhx-diag-toolswatchdog_configjsonampamp grep -imonitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp restart watchdog

Receive the following alertevent in HXConnect HX Controller VMHOSTNAME one or more configuredDNS servers not responding

CSCvt13947

35(2b)Delete any empty (Size 0) log filesunder varlogzookeeperversion-2ORvarlogzookeeperstandaloneversion-2

Zookeeper fails to start while Exhibitor isrunning however echo srvr returnsnothing

CSCvs21562

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

40(4e)1 De-commission the server

2 Power drain the server -REMOVE BOTH POWERCORDS ON THE BACK OFTHE SERVER FOR 10SECONDS THENREINSERTPOWERCORDS

3 Re-commission the server

M2 boot disk is missing from serverinventory after upgrade to server firmware40(4e) As a result server fails to boot toOS installed in the M2 disk The issuepersists after re-acknowledgement as wellas de-commission andre-acknowledgement of the server

CSCvs02466

Release Notes for Cisco HX Data Platform Release 4055

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Re-start the flusher manuallyMany disks taken offline by a fault ormanually resulting in ENOSPC oncluster

CSCvq32721

35(2e)Clean up fail over cluster and thenrecreate the fail over cluster Thereis no need to touch the HX storagecluster

[Hyper-V]When replicated DCwas usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2b)Manually install upgradescvmclient VIB andmake sure thatit matches with HXDP version

Upgrade from release HX 35(2a) to35(2x) 4X does not upgrade scvmclientRecovery point may not work

CSCvq54992

Management

30(1b)To recover the VM copy over thedata disks and attach them to thenew VM

[HyperV]VMshows up onmultiple nodesCSCvj22992

35(1a)1 Do not change MTU at ESXlevel on running cluster

2 Reset back to the original value

ChangingMTU from 9000 to 1500 on oneESX node causes storfs process to restarton all nodes

CSCvm99150

Release Notes for Cisco HX Data Platform Release 4056

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)After the installation (for newdeployments) or after an upgradeto 401a (for existingdeployments) perform thefollowing steps on all ControllerVMs

1 Edit the following tune files (onall Controller VMs)

optspringpathconfiglfftunes

optspringpathconfigvsi_16tbtunes

2 SetcleanerEnableSegSummaryCleaning

to false

3 After editing the above tunefiles

a SSH in to all the ControllerVMs

b Run storfstool -- -Z

c Run the below command andcheck the values Tune valueshould be true

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=true

d Type below commands andapplied the tunes changesdynamically

echo false gttmpstprocfssystemtunecleanerEnableSegSummaryCleaning

e Verify that tune values arechanged Run below commandValue should be false

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=false

f umount tmpstprocfs

A cluster with 8TB or 12TB disk drivesmay experience IO stalls for severalminutes after another node fails in thecluster

CSCvp23718

Release Notes for Cisco HX Data Platform Release 4057

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)1 Update the conf file manuallyto change to the link local addressto

2 Restart scvmclient on thecontroller and ESX

HX 35(2b) FlexVol to HX 40 CSIupgrade does not work FlexVol continuesto work

CSCvp26319

35(2a)Disable rebalance until node healsthen reenable rebalance

7-node cluster healing very slowlybecause rebalance change AMS reqs areblocking healing process

CSCvp62512

35(2a)NAWith moderate cluster storage capacityusage under site failover cluster clustermay experience an all paths (APD) downcondition

CSCvq49412

40(1a)NAThis product includes Third-partySoftware that is affected by thevulnerabilities identified by the followingCommon Vulnerability and Exposures(CVE) IDs

CVE-2016-0762 CVE-2016-6797CVE-2016-6816 CVE-2016-8735CVE-2017-5647 CVE-2017-12615CVE-2017-12616 CVE-2017-12617CVE-2017-7674 CVE-2018-1304CVE-2018-8014 CVE-2018-1336CVE-2018-8034 CVE-2018-11784CVE-2019-0232

CSCvp36364

35(2d)NAstcli cluster info command needs toprovide UCSM VIP address Currently itshows ucsm-hostcom

CSCvq11456

35(2a)No workarounds currently Thestale entries do not affect the dayto day operations of HyperFlex

HyperFlex upgrade validation failedbecause cluster has extraneous stNodes ininternal database

CSCvq32530

35(2a)NAWhen not in maintenancemode all drivesappear in HX Connect and get updated asexpected

When in maintenance mode extra drivesare shown in HX Connect with no slotnumbers

CSCvq42378

35(2b)Contact TAC to cleanup duplicatenode

Nodes duplicated in HX Connect aftercluster recreate

CSCvq61825

Release Notes for Cisco HX Data Platform Release 4058

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)NACurrently HyperFlex installs on Hyper-Vdo not contain the stcli security whitelistcommandset

CSCvq66245

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

Release Notes for Cisco HX Data Platform Release 4059

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)[Hyper-V] Converged node expansionfails for incoming node with VIC 14xx

CSCvq39523

Release Notes for Cisco HX Data Platform Release 4060

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

1 Set Jumbo frame inWindowsHypervisor for incomingnode

Get-NetAdapter storage-data-a|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

Get-NetAdapter storage-data-b|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

2 Verify all interfaces are setcorrectly to support Jumbo frames

Get-NetAdapter storage |Get-NetAdapterAdvancedProperty| RegistryKeyword -matchjumbo | ft -auto

You should receive the followingmessage

Name DisplayName DisplayValue

RegistryKeyword RegistryValue

---- ----------- ------------

--------------- -------------

vswitch-hx-storage-data Jumbo

Packet 9014 Bytes

JumboPacket 9014

storage-data-b Jumbo Packet

Bytes 9014 JumboPacket

9014

storage-data-a Jumbo Packet

Bytes 9014 JumboPacket

9014

3 Reboot only the incoming nodeor the node being expanded

4 Retry Cluster Expansion in

Release Notes for Cisco HX Data Platform Release 4061

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

Software Requirements for Microsoft Hyper-V

The software requirements include verification that you are using compatible versions of Cisco HyperFlexSystems (HX) components and Microsoft Hyper-V (Hyper-V) components

HyperFlex Software versions

The HX componentsmdashCisco HX Data Platform Installer Cisco HX Data Platform and Cisco UCSfirmwaremdashare installed on different servers Verify that each component on each server used with and withinthe HX Storage Cluster are compatible

bull Cisco HyperFlex M5 Converged nodesmdash For Hybrid (Cisco HyperFlex HX240c M5 HX220c M5)and All Flash (Cisco HyperFlex HXAF240cM5 HXAF220cM5) verify that Cisco UCSManager 40(2b)is installed HX 40 (1a) does not support Hyper-V on the All NVMe (HXAF220C-M5SN) nodes Fordetailed information on installation requirements and steps see theCisco HyperFlex Systems InstallationGuide on Microsoft Hyper-V

Table 4 Supported HyperFlex Software versions for M5 Servers on Hyper-V

M5 Recommended Server FirmwareHyperFlex Release

40(4h)40(2b)

40(4g)40(2a)

40(4e)40(1b)

40(4e)40(1a)

Release Notes for Cisco HX Data Platform Release 4014

Software Requirements for Microsoft Hyper-V

Table 5 Supported Microsoft Software versions

VersionMicrosoft Component

Windows Server 2016 Datacenter Core amp DesktopExperience

ForWindows Server 2016Datacenter Coreand Desktop Experience the Windows2016 ISO image should be Update BuildRevision (UBR) 1884 at a minimum

Note

Windows Server 2019Datacenter-DesktopExperienceis supported starting from HXDP 401(a) onwards

For Windows Server 2019 DesktopExperience theWindows 2019 ISO imageshould be Update Build Revision (UBR)107 at a minimum

Note

Windows Server 2019 DatacenterndashCore is notsupported currently

Also note that the following are currently notsupported

OEM activated ISOs and Retail ISOs are notsupported

Earlier versions ofWindows Server such asWindows2012r2 are not supported

Non-English versions of the ISO are not supported

Windows Operating System (Windows OS)

A Windows 2012 or later domain and forestfunctionality level

Active Directory

Supported Microsoft License Editions

The Microsoft Windows Server version that is installed on one or more HyperFlex hosts must be licensed asper Microsoft licensing requirements listed on Microsoft Licensing

Browser Recommendations

Use one of the following browsers to run the listed HyperFlex components These browsers have been testedand approved Other browsers might work but full functionality has not been tested and confirmed

Table 6 Supported Browsers

HX ConnectHX Data PlatformInstaller

Cisco UCS ManagerCisco IntersightBrowser

11 or higher11 or higher11 or higherNAMicrosoft InternetExplorer

70 or higher70 or higher57 or higher62 or higherGoogle Chrome

Release Notes for Cisco HX Data Platform Release 4015

Browser Recommendations

HX ConnectHX Data PlatformInstaller

Cisco UCS ManagerCisco IntersightBrowser

60 or higher60 or higher45 or higher57 or higherMozilla Firefox

NANA9 or higher10 or higherApple Safari

NANA35 or higherNAOpera

Notes

bull Cisco HyperFlex Connect

The minimum recommended resolution is 1024 X 768

bull Cisco HX Data Platform Plug-in

The Cisco HX Data Platform Plug-in runs in vSphere For VMware Host Client System browserrequirements see the VMware documentation at httpswwwvmwarecomsupportpubs

The HX Data Platform Plug-in is not displayed in the vCenter HTML client You must use the vCenterflash client

bull Cisco UCS Manager

The browser must support the following

bull Java Runtime Environment 16 or later

bull Adobe Flash Player 10 or higher is required for some features

For the latest browser information about Cisco UCSManager refer to themost recent Cisco UCSManagerGetting Started Guide

Cisco HX Data Platform Storage Cluster SpecificationsCluster Limits

bull Cisco HX Data Platform supports up to 100 clusters managed per vCenter as per VMware configurationmaximums

bull Cisco HX Data Platform supports any number of clusters on a single FI domain Each HX convergednode must be directly connected to a dedicated FI port on fabric A and fabric B without the use of a FEXC-series compute only nodes must also directly connect to both FIs B-series compute only nodes willconnect through a chassis IO module to both fabrics In the end the number of physical ports on the FIwill dictate the maximum cluster size and maximum number of individual clusters supported in a UCSdomain

The following table provides Cisco HX Data Platform Storage Cluster Specifications

Release Notes for Cisco HX Data Platform Release 4016

Cisco HX Data Platform Storage Cluster Specifications

Stretched Cluster (Availableon ESX Only)

Microsoft Hyper-VVMware ESXiNode

AllNVMe -HXAF220c-M5SN

HX240c-M5LHX220cM5

HX220cAF M5

HX240cM5

HX240cAF M5

HX240c-M5LHX220cM5

HX220cAF M5

HX240cM5

HX240cAF M5

HXAF220c-M5SN

AllNVMe -HXAF220c-M5SN

ESXi 65or 67only NotsupportedwithESXi 60orHyper-V

HX220cM5 Edge

HXAF220cM5 Edge

HX220cM4 Edge

HXAF220cM4 Edge

HX240c-M5LHX220cM5

HX220cAF M5

HX240cM5

HX240cAF M5

HX220cM4

HX220cAF M4

HX240cM4

HX240cAF M4

HXServers

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

C220M5C240M5B200M4B200M5

C240M5C220M5B200M4B200M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

mdashB200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

Compute-OnlyUCSB-SeriesC-SeriesServers

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

Convergednodesonly

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

SupportedNodes

Release Notes for Cisco HX Data Platform Release 4017

Cisco HX Data Platform Storage Cluster Specifications

Stretched Cluster (Availableon ESX Only)

Microsoft Hyper-VVMware ESXiNode

NA(requiresEnterpriseHXDP-PLicense)

NA(requiresEnterpriseHXDP-PLicense)

NA(requiresEnterpriseHXDP-PLicense)

Convergednodes3-16(12TBHDDoption isnotsupportedforHyperV)

Compute-onlynodes0-16

Convergednodes3-16

Compute-onlynodes0-16

NA(requiresEnterpriseHXDP-PLicense)

M4Convergednodes 3

M5Convergednodes23or 4

RequiresHXDP-ELicense

Convergednodes3-16(12TBdrive islimited toamaximumof 8nodes)

Compute-onlynodes0-16(12TBdrive islimited toamaximumof 8nodes)

Convergednodes3-32(76TBdriveconfigsarelimited tomaximumof 12drivesnodeon AF240)

Compute-onlynodes0-32

HXDP-SLicensedNodeLimits

11 ratioofHXDP-StoComputeonlynodes

(MinmdashMax)

Convergednodes2-16 perSite

Compute-onlynodes0-21 perSite

(up tomaxclustersize)

Convergednodes2-8 perSite

Compute-onlynodes0-16 perSite

(up tomaxclustersize)

Convergednodes2-16 perSite

Compute-onlynodes0-21 perSite

(up tomaxclustersize)

Convergednodes3-16(12TBHDDoption isnotsupportedforHyperV)

Compute-onlynodes0-16

Convergednodes3-16

Compute-onlynodes0-16

Convergednodes3-32

ComputeOnlynodes0-32

(up tomaxclustersize)

Convergednodes 3

(requiresHXDP-ELicense)

Convergednodes3-16(12TBdrive islimited toamaximumof 8nodes)

Compute-onlynodes0-32(12TBdrive islimited toamaximumof 16nodes)

Convergednodes3-32(76TBdriveconfigsarelimited tomaximumof 12drivesnodeon AF240)

Compute-onlynodes0-32

(up tomaxclustersize)

HXDP-PLicensedNodeLimits

12 ratioofHXDP-PtoComputeonlynodes

(MinmdashMax)

24 perSite 64percluster

24 perSite 48percluster

32 perSite 64percluster

32326434864MaxClusterSize

Release Notes for Cisco HX Data Platform Release 4018

Cisco HX Data Platform Storage Cluster Specifications

Stretched Cluster (Availableon ESX Only)

Microsoft Hyper-VVMware ESXiNode

212121111121mdash2121MaxComputetoConvergedratio

NoExpansion

Requires Enterprise license

Requires uniform expansion across both sites

Guidelines and Limitationsbull Starting with release 40(2a) SCVM is no longer needed on a Compute node

bull HX REST API Access Token Management ndash Applications leveraging HX REST APIs should re-useaccess tokens when making API calls Once obtained using the AAA Obtain Access Token API accesstokens are valid for 18 days (1555200 seconds) In addition AAA enforces rate limiting on ObtainAccess Token API requests in a 15 minute window auth can be invoked (successfully) a maximum of5 times A user is allowed to create a maximum of 8 unrevoked tokens Subsequent call to auth willautomatically revoke the oldest issued token tomake room for the new token Amaximum of 16 unrevokedtokens can be present in system In order to prevent brute-force attacks after 10 consecutive failedauthentication attempts a user account is locked for a period of 120 secondsFor more information seeCisco HyperFlex Systems REST API Reference guide

HxConnect makes use of AAA Authentication REST API for login and the above rate limit applies toHxConnect also

Upgrade Guidelines

The following list is a highlight of critical criteria for performing an upgrade of your HyperFlex system

bull Unsupported HX Data Platform 17x 18x 20 21x 25x and 26x clustersmdashUsers from anyversion prior to 26(1a) must step through an intermediate version before upgrading to 40 or later releasesIf you need to upgrade your environment from a Cisco HyperFlex HX Data Platform software releasethat is past the last date of support to the latest suggested release on the Cisco Software Download sitesee Cisco HyperFlex Systems Upgrade Guide for Unsupported Cisco HXReleases For more informationsee the Software Advisory for CSCvq66867 WARNING Only Use HXDP 26(1e) Upgrade PackageWhen Upgrading From HXDP 18(1a)-18(1e)

bull Hypercheck Health Check UtilitymdashCisco recommends running this proactive health check utility onyour HyperFlex cluster prior to upgrade These checks provide early visibility into any areas that mayneed attention andwill help ensure a seamless upgrade experience For more information see the HyperFlexHealth amp Pre-Upgrade Check Tool TechNote for full instructions on how to install and run Hypercheck

bull vSphere 67 Software AdvisorymdashDo not upgrade to Cisco HX Data Platform Release 40(1a) whenrunning ESXi 67U1 EP06 (build 11675023) Do not upgrade to 67U1 EP06 (build 11675023) ifrunning Cisco HX Data Platform Release 40(1a) See the Software Advisory CSCvo56350 for furtherdetails

Release Notes for Cisco HX Data Platform Release 4019

Guidelines and Limitations

The software build version posted at release will override any other local versions

bull Required vCenter upgrademdashFor enhanced security Cisco HX Data Platform Release 35(1a) or laterrequires the use of TLS 12 Therefore vCenter must be upgraded to 60 U3f or later before upgradingto Cisco HX Data Platform Release 35 or later In addition ESXi should be upgraded as required tomeet HX Data Platform compatibility requirements

bull Minimum HXDP version for upgrademdashHXData Platform clusters running 26(1a) or later may upgradedirectly to 40 using the HX Connect UI

bull Cluster ReadinessmdashEnsure that the cluster is properly bootstrapped and the updated plug-in is loadedbefore proceeding Manual cluster bootstrap is required for upgrade from a pre-35 release

bull Cluster ReadinessmdashEnsure that the cluster is properly bootstrapped and the updated plug-in is loadedbefore proceeding Manual cluster bootstrap is required for HX releases earlier than 35(1a) For moreinformation see theManual Bootstrap Upgrade Process in the Cisco HyperFlex Systems Upgrade Guidefor VMware ESXi Release 40 Do not skip this cluster bootstrap step it is required for all upgradesuntil HX Release 35(1a) Auto bootstrap is supported beginning with HX release 35(1a) For moreinformation see the Auto Bootstrap Upgrade Process from HX Connect UI in the Cisco HyperFlexSystems Upgrade Guide for VMware ESXi Release 40

Manual bootstrap is not supported on Intersight clusters

bull Initiating Upgrade―Use the HX Connect UI or CLI stcli commands when upgrading from 25(1a)or later releases Use either the CLI stcli commands or the HX Data Platform Plug-in to the vSphereWeb Client when upgrading from a pre-25(1a) release The vCenter plug-in should not be used forupgrades starting with the 25(1a) release The vCenter plug-in should not be used for upgrades startingwith the 25(1a) release

If the current cluster version is at 35(1a) or above you do not need to use the stcli command Directupgrade to 40 is possible

bull Complete your Upgrade―The self-healing (or rebalance) capability is disabled temporarily during theupgrade window If the upgrade fails you should complete the upgrade as soon as possible

bull ESXi and HXDP Compatibility―Ensure your cluster is running a compatible version of ESXi basedon the running the HXData Platform version (see the section Software Requirements for VMware ESXi)ESXi compatibility is determined by the major version and update release of ESXi It is generally bestto upgrade HXDP and ESXi together if combining the upgrade operations into a single optimized rebootWhen running a split upgrade first upgrade the HX Data Platform then proceed to upgrade ESXi

bull If Upgrading to vSphere 65

bull Certain cluster functions such as native and scheduled snapshotsReadyClones and Enter or Exit HX Maintenance Mode will not operatefrom the time the upgrade is started until the HX Data Platform upgrade iscomplete

bull After upgrading ESXi using the offline zip bundle use the ESX ExitMaintenance Mode option The HX Exit Maintenance Mode option doesnot operate in the vSphere Web Client until the HX Data Platform upgradeis complete

Note

Release Notes for Cisco HX Data Platform Release 4020

Guidelines and Limitations

bull vSphere 60 UpgradesmdashUsers on vSphere 60 migrating to 65 upgrade components in the followingorder

1 Upgrade HX Data Platform and UCS firmware

2 Upgrade HX Data Platform and ESXi

3 Upgrade HX Data Platform only first then upgrade ESXi or UCS firmware or both

bull M4 Server Firmware UpgradesmdashUpgrade server firmware to ensure smooth operation and to correctknown issues Specifically newer SAS HBA firmware is available in this release and is recommendedfor long-term stability

bull Users are encouraged to upgrade to 31(3c) C-bundle or later whenever possible

bull Users running C-bundle versions before 31(2f) must upgrade server firmware by performing acombined upgrade of UCS server firmware (C-bundle) to 31(3c) or later and HX Data Platform to25 Do not split the upgrade into two separate operations

bull If the cluster is already on 31(2f) C-bundle or later you may perform an HX Data Platform onlyor combined upgrade as required

bull M5 Server Firmware UpgradesmdashM5 generation servers must run firmware version 32(2d) or later

bull Firmware Downgrades mdashDowngrading UCSM from the HX-installer is not supported

bull M4M5 Mixed DomainsmdashA mixed domain occurs when a new separate M5 cluster is installed underthe same UCS domain that contains existing M4 clusters Under these conditions orchestrated UCSserver firmware upgrade will not operate until Cisco HX Data Platform Release 26 or later is installedon the M4 clusters Therefore it is best practice to first upgrade UCS server firmware to the latest 31(3)or 32(2) patch release before adding a new M5 cluster to the existing UCS domain Additionally any17 HX Data Platform clusters must first be upgraded before adding any new M5 clusters to the samedomain

bull Maintenance WindowmdashIf upgrading both HX Data Platform and UCS firmware you can select eithera combined or split upgrade through the vSphere HX Data Platform Plug-in depending on the length ofthe maintenance window Cisco UCSManager infrastructure upgrade is only supported using AutoInstalland the direct server firmware upgrade should be performed only through the upgrade orchestrationframework provided by the HX Data Platform Plug-in

bull Unsupported Self-Encrypting Drives (SEDs)mdashIf adding or replacing self-encrypting drives (SEDs)that have been recently qualified in newer versions of HX Data Platform insert the new drives only afterupgrading HX Data Platform to a compatible version All drives must be SED drives mixing SED andnon-SED is not supported

bullbull Enabling External Host AccessmdashWith Cisco HX Data Platform Release 40(1a) port 445 on themanagement network is blocked for enhanced security Note that prior to 40 port 445 port was openenabling external host access If you are upgrading to 40(1a) from a prior release and would like tocontinue external host access you can use a utility to open select hosts For more information aboutenabling external host access see the ConfiguringHyperFlex Share to SCVMM section in the InstallationGuide for Microsoft Hyper-V

Release Notes for Cisco HX Data Platform Release 4021

Guidelines and Limitations

Mixed Cluster Expansion Guidelinesbull Expanding existing M4 cluster with M5 converged nodes is supported

bull Expanding existing M5 cluster with M4 converged nodes is not supported

bull Expanding existing mixed M4M5 cluster with M4 or M5 converged nodes is supported

bull Adding any supported compute-only nodes is permitted with all M4 M5 and mixed M4M5 clustersusing the HX Data Platform 26 or later Installer Some example combinations are listed here manyother combinations are possibleExample combinationsExpand mixed M4M5 cluster with compute-only B200 C220 C240 M4M5Expand M4 cluster with compute-only B200 M5 C220 M5 C240M5

bull Only expansion workflow is supported to create a mixed cluster Initial cluster creation with mixedM4M5 servers is not supported

bull All M5 servers must match the form factor (220240) type (HybridAF) security capability (Non-SEDonly) amp disk configuration (QTY capacity and non-SED) of the existing M4 servers

bull HX220-M5 will use a maximum of 6 capacity disks (2 disk slots to remain empty) when mixedwith HX220-M4

bull HX Edge SED LFF Hyper-V and Stretched Clusters do not support mixed M4 and M5 clusters

Security FixesThe following security issues are resolved

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-13132 CVE-2019-9924CSCvq6313840(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-11719 CVE-2019-11727CVE-2019-11729

CSCvq7124040(2a)

Release Notes for Cisco HX Data Platform Release 4022

Mixed Cluster Expansion Guidelines

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-1125CSCvr0633940(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2015-9383 CVE-2018-14498CVE-2018-20406 CVE-2018-20852CVE-2019-10160 CVE-2019-13117CVE-2019-13118 CVE-2019-14287CVE-2019-14973 CVE-2019-15903CVE-2019-17546 CVE-2019-18197CVE-2019-18218 CVE-2019-5010CVE-2019-5094 CVE-2019-5481CVE-2019-5482 CVE-2019-9636CVE-2019-9740 CVE-2019-9947CVE-2019-9948

CSCvs0609440(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2017-13168 CVE-2017-18174CVE-2017-18216 CVE-2018-10876CVE-2018-10877 CVE-2018-10878CVE-2018-10879 CVE-2018-10880CVE-2018-10881 CVE-2018-10882CVE-2018-10902 CVE-2018-10938CVE-2018-12233 CVE-2018-12896CVE-2018-13053 CVE-2018-13094CVE-2018-13096 CVE-2018-13405CVE-2018-13406 CVE-2018-14609CVE-2018-14617 CVE-2018-14633CVE-2018-14734 CVE-2018-15572CVE-2018-15594 CVE-2018-16276CVE-2018-16658 CVE-2018-17182CVE-2018-17972 CVE-2018-18021CVE-2018-18690 CVE-2018-18710CVE-2018-6554 CVE-2018-6555CVE-2018-9363

CSCvp6501940(2a)

This vulnerability is due to insufficientHTML iframe protection An attackercould exploit this vulnerability bydirecting a user to an attacker-controlledweb page that contains amaliciousHTMLiframe A successful exploit could allowthe attacker to conduct clickjacking orother clientside browser attacks

NACSCvo9851640(2a)

Release Notes for Cisco HX Data Platform Release 4023

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerability is due to insufficientauthentication for the statistics collectionservice An attacker could exploit thisvulnerability by sending properlyformatted data values to the statisticscollection service of an affected deviceA successful exploit could allow theattacker to cause the web interfacestatistics view to present invalid data tousers

NACSCvj9558440(2a)

The vulnerability is due to insufficientCSRF protections for the web UI on anaffected device An attacker could exploitthis vulnerability by persuading a user ofthe interface to follow a malicious linkA successful exploit could allow theattacker to perform arbitrary actions withthe privilege level of the affected user

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvp2434340(2a)

Vulnerabilities with a version of cJSONidentified by CVE-2019-11834 andCVE-2019-11835

For more information see the relatedCisco Security Advisory

CVE-2019-11834 CVE-2019-11835CSCvq1994940(2a)

HX ESXi image patches for MCEPSCand TAA vulnerabilities with VMwareESXi identified by CVE-2018-12207 andCVE-20190-11135

For more information see the relatedCisco Security Advisory

CVE-2018-12207 CVE-2019-11135CSCvr5439840(2a)

CP networking vulnerabilities affectingthe Linux kernel identified byCVE-2019-11477 CVE-2019-11478CVE-2019-11479

For more information see the relatedCisco Security Advisory

CVE-2019-11477 CVE-2019-11478CVE-2019-11479

CSCvq1954640(2a)

Qualification of Microsoft securitypatches for vulnerabilities withMicrosoftHyper-V hypervisor identified withCVE-2018-12207 CVE-2019-11135

For more information see the relatedCisco Security Advisory

CVE-2018-12207 CVE-2019-11135CSCvr5439940(2a)

Release Notes for Cisco HX Data Platform Release 4024

Security Fixes

DescriptionCVEDefect IDRelease

OpenSSH vulnerabilities identified byCVE-2019-6111

For more information see the relatedCisco Security Advisory

CVE-2016-10708 CVE-2018-15473CVE-2018-20685 CVE-2019-6109CVE-2019-6111

CSCvp7646340(2a)

Multiple vulnerabilities associated withOpenSSL and LibNSS

For more information see the relatedCisco Security Advisory

CVE-2016-2105 CVE-2016-2106CVE-2016-2107 CVE-2016-2109CVE-2016-2176 CVE-2016-2177CVE-2016-2178 CVE-2016-2179CVE-2016-2180 CVE-2016-2181CVE-2016-2182 CVE-2016-2183CVE-2016-6302 CVE-2016-6303CVE-2016-6304 CVE-2016-6305CVE-2016-6306 CVE-2016-6307CVE-2016-6308 CVE-2016-7055CVE-2016-8610 CVE-2017-3731CVE-2017-3732 CVE-2017-3735CVE-2017-3736 CVE-2017-3737CVE-2017-3738 CVE-2018-0495CVE-2018-0732 CVE-2018-0734CVE-2018-0735 CVE-2018-0737CVE-2018-0739 CVE-2018-12384CVE-2018-12404 CVE-2018-5407CVE-2019-1559

CSCvp6667940(2a)

libpng vulnerability identified byCVE-2016-10087

For more information see the relatedCisco Security Advisory

CVE-2016-10087CSCvp6655540(2a)

Vulnerabilities associated with libjpg andlibpng

For more information see the relatedCisco Security Advisory

CVE-2014-9092 CVE-2016-3616CVE-2017-15232 CVE-2018-11212CVE-2018-11213 CVE-2018-11214CVE-2018-1152 CVE-2018-13785

CSCvp3458640(2a)

Vulnerabilities associated with Glibidentified by CVE-2018-16428 andCVE-2018-16429

For more information see the relatedCisco Security Advisory

CVE-2018-16428 CVE-2018-16429CSCvp3120740(2a)

Multiple vulnerabilities associated withcurl expat python 27 python 35 36and 37 freetype and giflib

For more information see the relatedCisco Security Advisory

CVE-2019-15133 CVE-2019-15903CVE-2019-5010 CVE-2019-5481CVE-2019-5482 CVE-2019-9636CVE-2019-9740 CVE-2019-9947CVE-2019-9948

CSCvr3690340(2a)

Release Notes for Cisco HX Data Platform Release 4025

Security Fixes

DescriptionCVEDefect IDRelease

Multiple third party softwarevulnerabilities

For more information see the relatedCisco Security Advisory

CVE-2019-14379 CVE-2019-12384CVE-2019-14439

CSCvq9203240(2a)

Vulnerabilities associated with elfutils

For more information see the relatedCisco Security Advisory

CVE-2018-16062 CVE-2018-16402CVE-2018-16403 CVE-2018-18310CVE-2018-18520 CVE-2018-18521CVE-2019-7149 CVE-2019-7150CVE-2019-7665

CSCvq4325040(2a)

Vulnerabilities associated with vim

For more information see the relatedCisco Security Advisory

CVE-2017-5953 CVE-2019-12735CSCvq4323040(2a)

Vulnerabilities associated with sqlite3

For more information see the relatedCisco Security Advisory

CVE-2016-6153 CVE-2017-10989CVE-2017-13685 CVE-2017-2518CVE-2017-2519 CVE-2017-2520CVE-2018-20346 CVE-2018-20505CVE-2018-20506 CVE-2019-8457CVE-2019-9936 CVE-2019-9937

CSCvq4321340(2a)

Vulnerabilities associated with glib20

For more information see the relatedCisco Security Advisory

CVE-2018-20843CSCvq4320940(2a)

Vulnerabilities associated with expat

For more information see the relatedCisco Security Advisory

CVE-2018-20843CSCvq4320540(2a)

Vulnerabilities associated with bzip2

For more information see the relatedCisco Security Advisory

CVE-2016-3189 CVE-2019-12900CSCvq4319440(2a)

Mulitple Vulnerabilities associated wtihNodeJS

For more information see the relatedCisco Security Advisory

CVE-2018-12115 CVE-2018-0734CVE-2018-5407 CVE-2018-12120CVE-2018-12121 CVE-2018-12122CVE-2018-12123 CVE-2018-12116CVE-2019-5737 CVE-2019-5739

CSCvq1069440(2a)

Vulnerabilities associated with PalletsJinja strformat_map and identified byCVE-2019-10906 and CVE-2016-10745

For more information see the relatedCisco Security Advisory

CVE-2019-10906 CVE-2016-10745CSCvq1038840(2a)

Release Notes for Cisco HX Data Platform Release 4026

Security Fixes

DescriptionCVEDefect IDRelease

Multiple Security Vulnerabilities

For more information see the relatedCisco Security Advisory

CVE-2014-8501 CVE-2014-9939CVE-2015-9262 CVE-2016-10087CVE-2016-2226 CVE-2016-4487CVE-2016-4488 CVE-2016-4489CVE-2016-4490 CVE-2016-4491CVE-2016-4492 CVE-2016-4493CVE-2016-6131 CVE-2018-10963CVE-2018-13785 CVE-2018-17100CVE-2018-17101 CVE-2018-18557CVE-2018-18661 CVE-2018-7456CVE-2018-8905

CSCvq0917840(2a)

Vulnerabilities associated withlibseccomp

For more information see the relatedCisco Security Advisory

CVE-2019-9893CSCvq0756840(2a)

Vulnerabilities associated withgdk-pixbuf

For more information see the relatedCisco Security Advisory

CVE-2017-12447CSCvq0675540(2a)

Vulnerabilities associated withpython-crypto

For more information see the relatedCisco Security Advisory

CVE-2018-6594CSCvp9381740(2a)

Vulnerabilities associated with wget

For more information see the relatedCisco Security Advisory

CVE-2018-20483 CVE-2019-5953CSCvp8672140(2a)

Multiple vulnerabilities associated withPython Crypto

For more information see the relatedCisco Security Advisory

CVE-2018-6594CSCvp6674840(2a)

Vulnerabilities associated withsensible-utils

For more information see the relatedCisco Security Advisory

CVE-2017-17512CSCvp6673440(2a)

Multiple vulnerabilities associated withPython

For more information see the relatedCisco Security Advisory

CVE-2018-1000030CSCvp6668940(2a)

Release Notes for Cisco HX Data Platform Release 4027

Security Fixes

DescriptionCVEDefect IDRelease

Multiple patch vulnerabilities identifiedby CVE-2016-10713

For more information see the relatedCisco Security Advisory

CVE-2016-10713CVE-2018-1000156 CVE-2018-6951

CSCvp6667240(2a)

Vulnerabilities associatedwith Little CMSidentified by CVE-2016-10165

For more information see the relatedCisco Security Advisory

CVE-2016-10165 CVE-2018-16435CSCvp6666440(2a)

Vulnerabilities associated with TenableScan identified byCVE-2011-5325CVE-2019-11068CVE-2016-7076CVE-2017-1000368

For more information see the relatedCisco Security Advisory

CVE-2011-5325 CVE-2016-7076CVE-2017-1000368CVE-2019-11068

CSCvp6474640(2a)

Vulnerabilities associated with libxml2294 and earlier as used in XMLSec1223 and earlier and other products andidentified by CVE-2016-9318

For more information see the relatedCisco Security Advisory

CVE-2016-9318 CVE-2017-16932CVE-2017-18258 CVE-2018-14404CVE-2018-14567

CSCvp3479240(2a)

Vulnerabilities associated with LFTPRemote File Names Unauthorized AccessVulnerability and identified byCVE-2018-10916

For more information see the relatedCisco Security Advisory

CVE-2018-10916CSCvp2926640(2a)

Vulnerabilities associated withCryptographyDeprecationWarning signerand verifier have been deprecated

For more information see the relatedCisco Security Advisory

CVE-2018-7750CSCvo3409740(2a)

Multiple Vulnerabilities in python python27 35

For more information see the relatedCisco Security Advisory

CVE-2019-16056CSCvr3979340(2a)

Release Notes for Cisco HX Data Platform Release 4028

Security Fixes

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toexecute commands as the root user

The vulnerability is due to insufficientinput validation An attacker could exploitthis vulnerability by connecting to thecluster service manager and injectingcommands into the bound process Asuccessful exploit could allow the attackerto run commands on the affected host asthe root user

Cisco has released software updates thataddress this vulnerability There areworkarounds that address thisvulnerability

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvq2417640(1b)35(2g)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

The vulnerability is due to an unprotectedlistening interface An attacker couldexploit this vulnerability by connectingto the listening interface and injectingcommands to the bound process Anexploit could allow the attacker to runcommands on the affected host as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvj9560640(1b)35(2g)

Release Notes for Cisco HX Data Platform Release 4029

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerabilities associated with theJVM18U121memory leak found duringVC alarms (concurrent 40 calls usingREST API)

CVE-2017-10053 CVE-2017-10067CVE-2017-10074 CVE-2017-10078CVE-2017-10081 CVE-2017-10087CVE-2017-10089 CVE-2017-10090CVE-2017-10096 CVE-2017-10101CVE-2017-10102 CVE-2017-10107CVE-2017-10108 CVE-2017-10109CVE-2017-10110 CVE-2017-10111CVE-2017-10115 CVE-2017-10116CVE-2017-10118 CVE-2017-10135CVE-2017-10176 CVE-2017-10193CVE-2017-10198 CVE-2017-10243CVE-2017-10274 CVE-2017-10281CVE-2017-10285 CVE-2017-10295CVE-2017-10345 CVE-2017-10346CVE-2017-10347 CVE-2017-10348CVE-2017-10349 CVE-2017-10350CVE-2017-10355 CVE-2017-10356CVE-2017-10357 CVE-2017-10388CVE-2017-3509 CVE-2017-3511CVE-2017-3526 CVE-2017-3533CVE-2017-3539 CVE-2017-3544CVE-2018-2579 CVE-2018-2582CVE-2018-2588 CVE-2018-2599CVE-2018-2602 CVE-2018-2603CVE-2018-2618 CVE-2018-2629CVE-2018-2633 CVE-2018-2634CVE-2018-2637 CVE-2018-2641CVE-2018-2663 CVE-2018-2677CVE-2018-2678 CVE-2018-2783CVE-2018-2790 CVE-2018-2794CVE-2018-2795 CVE-2018-2796CVE-2018-2797 CVE-2018-2798CVE-2018-2799 CVE-2018-2800CVE-2018-2814 CVE-2018-2815CVE-2018-2952 CVE-2018-3136CVE-2018-3139 CVE-2018-3149CVE-2018-3150 CVE-2018-3169CVE-2018-3180 CVE-2018-3183CVE-2018-3214 CVE-2019-2422

CSCvo8899740(1b)

Tomcat and Nginx logs are not beingcollected in the support bundle generatedthrough HX Connect in Release 35

NACSCvm5803140(1b)

The vulnerabilities associated with thelibmspack software package versionincluded in Cisco HX Data Platform

CVE-2018-18584

CVE-2018-18585

CSCvn3511940(1a)

Release Notes for Cisco HX Data Platform Release 4030

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerabilities associated withFasterXML Jackson-Databind TimeValueField Denial of Service

CVE-2018-14719

CVE-2018-14720

CVE-2018-1000873

CVE-2018-14721

CVE-2018-19360

CVE-2018-19362

CVE-2018-19361

CVE-2018-14718

CSCvn8228240(1a)

The vulnerabilities associated with theOpenSSL Protocol software packageversion included in Cisco HX DataPlatform

CVE-2013-3587CSCvo0505440(1a)

The vulnerabilities associated withThird-Party Software Denial of Service

CVE-2018-16487

CVE-2018-19361

CSCvo2781840(1a)

The vulnerabilities associated withLinuxkernel for Ubuntu 1710

CVE-2018-1092

CVE-2018-7492

CVE-2018-8087

CVE-2018-1068

CVE-2018-8781

CSCvm5314935(2a)

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)25(1d)stcli services dns remove should remove the DNS serverinfo from the interface files

CSCvs70967

40(2b)40(1b)HX 40(1b) new deployment from Intersight productioncloud

User receives following error while enabling CSI(Kubernetes integration)

Failure occurred during volume_access_enable Errorwas Nonzero exit code 1

CSCvt10522

40(2b)40(2a)Receive the following alertevent in HX Connect HXController VM HOSTNAME one or more configuredDNS servers not responding

CSCvt13947

Release Notes for Cisco HX Data Platform Release 4031

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)40(2a)API UI CLI show few Drive slots Empty after upgradeto 402a

The drives are seen in the sysmtool lsscsi and stclicommand outputs

The cluster is healthy with no errors

CSCvt14914

40(2b)35(2d)After successfully changing the DNS server on the HXcontroller we still can see the original DNS entry thatwas added during the deployment

CSCvs69154

40(2b)35(2e)HyperFlex Connect and VCenter show APD alarms forHX and non HX datastores This implies there is an issuewith HyperFlex when there may not be

CSCvs30080

40(2b)40(2a)An issue where all the disks in the node get locked afterrebooting the node

We successfully unlocked using sed-clientsh -Ucommand but wanted to test with another reboot anddrives locked again

CSCvs74286

40(2b)40(2a)When running stcli license commands on HyperFlexthe following error is seen

rootSpringpathControllertmp stcli license show all

Show smart licensing failed

Smart Agent is not ready please wait a minute and tryagain

CSCvt13929

40(2b)40(2a)After upgrading to HX 40(2a) the first RoboEdge nodethat gets rebooted then ignores persistent drivesPIDHX-HD24TB10K4KN

Cluster will be in degraded state and have reducedcapacity

CSCvt20144

40(2b)35(2b)Zookeeper fails to start while Exhibitor is runninghowever echo srvr returns nothing

CSCvs21562

40(2b)40(2a)When performing an HyperFlex upgrade a validationwarning may occur due to the host not having EnterprisePlus or Enterprise hypervisor licensing

Upgrade Validation Warning

ESXi host esx1labtest should be configured withVMware Enterprise license for upgrade to continue

CSCvs91787

Release Notes for Cisco HX Data Platform Release 4032

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)35(2g)Cluster expansion fails at Config Installer stage whenthe root and admin password for the storage controller(SCVM) are different

CSCvs69317

40(2b)35(2g)Panic while upgrading ESXiCSCvt06983

40(2b)40(2a)A cluster node running may hang in the Linux kernelThis is classified as an oops and a deviation from theexpected behavior

CSCvs54285

40(2b)35(2g)Rebalance failing on 10+10 node HX 35(2g) stretchcluster

CSCvs69007

40(2b)35(2d)The cluster becomes inaccessible to the IOVisorCSCvr54687

40(2b)40(2a)Panic on the storage controllerCSCvt61297

40(2b)40(2a)The size of support bundle is very large when collectedwith storfs-support command

CSCvt63306

40(2b)40(2a)storfs crash associated withFileSystemUsageWarningEvent orFileSystemUsageAlertEvent due to high usage ofvarstv above 80

CSCvt72807

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

40(2a)35(2d)In normal scenarios HX upgrade gets pendingACK afterupdating service profile In this case UCS only upgradewas stalled because after updating service profile HXupgrade was not getting pending ACK and it was alwayswaiting state

CSCvo39912

40(2a)35(1a)When using motherBoardReplace-12 to clean ZK fromold stNodepNodes resulted in unmounted datastore andmaking the size of the Hyperflex Datastores 0 resultingin all VMs in the cluster going offline

CSCvq39471

40(2a)26(1e)Upgrading cluster and getting error

Failed upgrade validations Checking vCenterconfiguration Reason Upgrade validations failed DRSFault Insufficient resources to satisfy configured failover

CSCvm77294

Release Notes for Cisco HX Data Platform Release 4033

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)The cluster expand fails on a node with DR replicationconfigured

When a HX cluster which has DR replication configuredis expanded we see the installer UI pulling in thereplicationVLAN information instead of themanagementVLAN information Even if we change that informationto the correct mgmt VLAN id and name it it does notseem to work as the node is configured with the VLANof the replicationVLAN in ESXi This leads to the failureof the node add with the host unreachable error

CSCvo70650

40(2a)35(1a)Customer reported that server didnt complete thefirmware upgrade automatically one by one

The user manually needs to put the host into maintenancemode and they manually acknowledged the pendingrequests to finish the UCS firmware upgrade

Maintenance policy is set to default (User-ack) as perthe design

CSCvo91624

40(2a)35(1i)When the cluster is in Failed state and stcli noderemove is attempted the output shows successful thoughthe node failed to get removed from the cluster

CSCvo93017

40(2a)35(1a)HyperFlex cluster upgrade may fail during validationwith a DRS Validation failed error

CSCvp31021

40(2a)35(2b)HX cluster expansion will fail with the following errormessage

MAC address pool configuration failure150[ErrorDescription] bad address block range definitioncollision

CSCvp58318

40(2a)35(2b)Memory usage by carbon cacheCSCvq34873

40(2a)35(2g)Upgrading ESXi cluster fails with error Nodemaintenance mode failed

CSCvr03240

40(2a)35(2e)storfs process does not automatically start on Exit HXMaintenance Mode or other tasks which restart StorageController

CSCvr88978

40(2a)35(2a)Post install script fails with message

Failed to execute ipmitool on HX node

CSCvp10707

40(2a)40(1a)HyperFlex expansionworkflow doesnt pull VLANnamecorrectly

CSCvp46539

Release Notes for Cisco HX Data Platform Release 4034

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)During HX cluster deployment - Validate ClusterCreation phase - HX Installer might fail with the error

fromvarlogspringpathstDeploylog

CSCvq45087

40(2a)35(2b)HX Installer fails to login to the SOL for configuringESX hosts

CSCvq91380

40(2a)35(2d)If DRS is enabled with configuration parameters otherthen fully automated ie manual or partially automatedthan this will change to fully automated during clusterexpansion

CSCvr44222

40(2a)35(1a)One of the validation task name has a typo The taskname is Validating node for mixed of SED and non-SEDdisks

It should be Validating node for mix of SED andnon-SED disks

CSCvp82175

40(2a)30(1i)This combination of using smaller NVMe drives is notsupported in expand operation For example if the drivesare only 375GB and they cannot be added to an existingcluster with larger caching SSD

CSCvo12359

40(2a)35(1a)HyperFlex includes a version of OpenSSL that is affectedby the vulnerabilities identified by the followingCommon Vulnerability and Exposures

CVE-2018-0495

CSCvp66679

40(2a)30(1e)SDK invoke exception when taking a native snapshot ona VM with GPU pci passthrough attached

CSCvo00511

40(2a)30(1i)Node replacement script fails due to EAM errorCSCvo62867

40(2a)35(1a)While pairing a cluster (HX release gt= 35) and cluster(HX release lt 35) remote replication network test failson the cluster (HX release lt 35)

CSCvo79760

40(2a)30(1b)Fixed the issue in the latest support-workflow bundleThis should not be hit anymore

CSCvo87061

40(2a)35(2a)On a 3 node cluster the MBreplace script fails as thescript looks for HEALTHY cluster status

CSCvo87080

40(2a)35(2a)MbReplace script (tar) hangs when running on 352b or40

CSCvp12359

Release Notes for Cisco HX Data Platform Release 4035

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)HX replication cleanup failing with error NA 35(2a) -INFODR state is not clean

CSCvp63958

40(2a)40(1a)The check_stig_parameters API incorrectly shows thecompliance state of the cluster as not STIG compliant

CSCvp66277

40(2a)35(2c)The snapshot creation on CBT enabled VM fails witherror Failed in vmreparent vmkfstools clone1

CSCvq06952

40(2a)35(2a)HyperFlex GUI when setting up remote DR-replicationpartnerpeer should show remote replication partner TCPport reachability test results

This will provide quick upfront connectivity results thatfrequently takes customer a while to manually check andconfirm

If reachability tests fail an info window and HX alertshould trigger that list the destination peer IP and thespecific port(s) that could not be reache

CSCvo60587

40(2a)35(2b)If you have more than one IP pool configured for areplication network and local replication network testsare failing you might be running into this issue

CSCvr67130

40(2a)35(2a)Skip task option in MBreplace and Software redeployscripts The scripts should have the option to skip tasksas user inputs and not by modifying the json file

CSCvp05204

40(2a)40(1a)Error noticed on SCVM Console print_req_error IOerror dev fd0 sector 0 on HyperV

It doesnt come up on SSH session

CSCvq34357

40(2a)30(1d)The VMNetwork switch is already created as part of theinstall process

Ideally the ip address assignment should be done duringthe initial install process as well No post install stepsshould be required altrnatively it could be done as partof a post install script

CSCvk36222

40(2a)25(1c)storfs service impacted during HX upgrade from HX21(1) to 25(1c)

HX node which is not being upgraded storfs servicestopped causing outage

CSCvg53223

40(2a)35(1b)HX Stretch Cluster Witness VM reverts to DHCP onreboot

CSCvp37536

40(2a)35(2b)HyperFlex Stretch Cluster Witness does not getprogrammed with an NTP server

CSCvp46578

Release Notes for Cisco HX Data Platform Release 4036

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2b)HyperFlex cluster does not heal after maintenance onnodeserver

CSCvr18528

40(2a)40(2a)High percentage of packet loss on the eth1 data interface9K MTU

CSCvr97089

40(2a)35(2f)The HyperFlex cluster healing state is stuck at 87 andis unable to progress

CSCvr16760

40(2a)40(2a)In some circumstances two replications can occur forthe same VM

From the GUI the replication status for this VM willshow stuck in an In Progress state but there are nojobs stuck in replication layer Subsequent Replicationsshould succeed

CSCvq22898

40(2a)35(2a)After an unexpected power outage the hyperflex clusteris unable to come up correctly

All processes are running time is synced and all nodescan vmkping each other as required

CSCvp13990

40(2a)40(1a)When specifying IP address instead of node ID stclinode maintenanceMode commands fail

CSCvp33657

40(2a)40(1a)2 node edge cluster if experiencing Intersightarbitratorconnectivity issues should NOT allowHXMaintenancemode to initiate as this will bring the cluster down

CSCvr31573

40(2a)35(2c)Old files in varsupportZKTxnlog are not purged withthe daily zklog-cleanup cron job

CSCvr89066

40(2a)35(2a)When a node is in MaintenanceMode any disk removalor replacement will be reflected in UI only after the nodeis brought back frommaintenancemode This is becausestorfs is not running on the node in maintenance andwill not be able to detect disk activities until it is broughtout of Maintenance Mode

CSCvo86431

40(2a)35(2c)Cluster might go offline and stop serving IO requestsrootcvm~ stcli cluster storage-summary--detailGet cluster storage summary failedjavanetConnectException Connection refused19216814210010207

CSCvr01645

40(2a)35(2d)A stretch cluster deployment fails with a messageFormatting nodes Node in Use on the Hyperflex installerand crmZoneType value is seen as 1 in storfscfg oroptspringpathconfigstretchtunes

CSCvq80340

Release Notes for Cisco HX Data Platform Release 4037

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)30(1c)During upgrade we upload the upgrade packagedownloaded from Ciscocom Upgrade packages are offile type tgz hxconnect acceptsstorfs-packages-401a-33028tar file a different type ofcompression and allows upgrade to start and fail whenit cannot decompress the file for bootstrap and scvmclientupgrade on ESXi

CSCvq63888

40(2a)40(2a)While attempting to install hyperflex via intersight thefollowing error is seen

Failed in Task Add hosts to vCenter Cluster with ErrorTry adding hosts manually to vCenter and retry Failedto get addressFailed to host by namelookupHostnamecompanycom on 000053 read udp12700145430-gt12700153 read connection refused

CSCvr52098

40(2a)35(2a)CoreAPI call may time out for cluster management APIwhile doing cluster create validation

CSCvo92952

40(2a)35(2d)Validation fails with Mixed mode expansion checkdue to Enhanced vMotion Compatibility Incompatibilitywhen the ESXi version is different

CSCvq95460

40(2a)30(1i)This failure is seen in installer when custom workflowis used hypervisor configuration + deploy (clean diskpartition=no) at Installing software packages on storagecontroller VM

CSCvr66309

40(2a)35(2d)The HX Installer apparently replaced a comma bysemicolon on the VLAN ID range while exporting theconfiguration to a JSON file

CSCvq93831

40(2a)40(1a)When using unsupported browsers HX Connect userswill not be able to login These users see an errorindicating unauthorized user

CSCvp22693

Release Notes for Cisco HX Data Platform Release 4038

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)40(1a)A vulnerability in the web-based management interfaceof Cisco HyperFlex Software could allow anunauthenticated remote attacker to conduct a cross-siterequest forgery (CSRF) attack on an affected system

The vulnerability is due to insufficient CSRF protectionsfor the web UI on an affected device An attacker couldexploit this vulnerability by persuading a user of theinterface to follow a malicious link A successful exploitcould allow the attacker to perform arbitrary actions withthe privilege level of the affected user

Cisco has released software updates that address thisvulnerability There are no workarounds that address thisvulnerability

This advisory is available at the following link

httpstoolsciscocomsecuritycentercontentCiscoSecurityAdvisorycisco-sa-20190807-hypflex-csrf

CSCvp24343

40(2a)35(2d)Add message to HX Connect Progress flow to notAcknowledge Pending Activities and reboot the serverson UCSM

HX Connect is doing a controlled rolling server upgradein the background

CSCvq22844

40(2a)40(1b)Compute Node expand operation fails due to incorrectcluster name

CSCvr43786

40(2a)26(1e)RF-2 cluster shutdown during data resynchronizationafter a node failure (non storfs) followed by multipledisk read failures and a HardBlacklist of a disk

CSCvp41241

40(2a)40(1a)HyperFlex Edge nodes do not properly set the ESXihostname during deployment

CSCvo13143

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4039

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4040

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1b) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401b-33133p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4041

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4042

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1b)40(1a)While running the HyperFlex installer with WindowsServer Hyper-V cluster creation process fails with theerror Failure occurred during ClusterCreation process Unable to post thecontent to the downstream This symptomis encountered while deploying HyperFlex cluster withHyperFlex nodes configuredordered with Cisco VIC1457 MLOM (PID HX-MLOM-C25Q-04) andWindows Server Datacenter or Core with Hyper-Vversion 2016 or 2019

CSCvp64140

Release Notes for Cisco HX Data Platform Release 4043

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)35(2b)Adding Micron 5200 drive to a cluster fails to increasecluster capacity

CSCvo69067

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4044

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4045

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1a) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401a-33028p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4046

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4047

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1a)30(1d)Cluster instability when disks of different sector sizeplaced in HX node

CSCvk17250

40(1a)

35(2c)

35(1a)When logged in using a local HX user account insteadof a Virtual Center account an error message appearsintermittently indicating Virtual Centerunreachable or Resource informationcannot be updated when VC is reachable

CSCvo36198

40(1a)

35(2b)

30(1d)HXDP does not work with EMC RecoverPoint - needsto support VMware API (FSS-Readdir)

CSCvk38003

Release Notes for Cisco HX Data Platform Release 4048

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)

35(2a)

30(1i)

25(1c)

25(1d)

Zookeeper (Exhibitor) process on HyperFlex StorageController VMs may fail to respond to requests whenvarzookeeper has low or no free space

CSCvm90352

40(1a)26(1c)Use Asynchronous consolidation for HX SnapshotsCSCvn02151

40(1a)35(2b)HX Quiesced Snapshot for Backup VendorsCSCvo90713

40(1a)35(2a)The cluster creationcluster expansion workflow stopswith the following error message at the validation step

FIRMWARE-Check UCSC-SAS-M5HD

FIRMWARE-Check UCSC-SAS-M5HD Required00000029000000320000003500000050 Found00000058

Action Needed Update the Controller Firmware toRequired Version

CSCvn17787

35(2b)30(1d)Kernel migration fails when a local datastore is searchedfor in the ESXi host

CSCvn73127

35(2a)26(1b)A node shuts down when two disks are replaced (acaching disk and another capacity disk) where thecapacity disk is inserted first and then the caching diskis inserted

CSCvk46364

Hyper-V

40(1a)35(2a)Cluster expansion may fail with an error code 500 -operation timed out

CSCvn28721

35(2b)35(2a)During upgrade remove the VLAN on team created foruser vSwitch During fresh installation only one VLANtag is set to the vSwitch and team although multipleVLANs were entered

CSCvn54300

35(2b)35(2a)While upgrading a Hyper-V cluster on account of a rarerace condition between the stUpgradeService andZookeeper servers the upgrade orchestration throws anupgrade validation error and the upgrade process isaborted

CSCvn60486

Release Notes for Cisco HX Data Platform Release 4049

Resolved Caveats in Release 40(1a)

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Startrestart EAM service invCenter

When EAM service is not runningvCenter registration is failing

CSCvt55712

35(2a)Contact Cisco TAC if you believeyou are seeing similar conditions

HyperFlex Stretch Cluster saw poorapplication performance due to writelatency cluster remained unhealthy andrebalance was stuck

This defect is currently being used forinvestigative purposes

CSCvt45344

35(2g)Remove log files fromvarlogspringpath folder when thelog files are not getting rotatedStop the relevant service removelarge log files fromvarlogspringpath folder andrestart the service

varstv folder in Controller VM maybecome full

CSCvt36374

40(2b)Restarting the stMgr will initializeit again

In a Cluster configured with DR networkStMgr may not get initialized and stuckin a deadlock while enabling the IPTablerules (can be verified from the stMgrlog)

CSCvt89709

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Click recover for the selectedVM again under the standalonemode This will move the VM torecovered state

When the user recovers a VM protectedin a group it along with other VMs in thegroup move to recovering state instandalonemode The selectedVM shouldmove to recovered state

CSCvs75553

35(2e)Clean up the fail over cluster andthen recreate the fail over clusterNo need to touch the HX storagecluster

Hyper-V When replicated DC was usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2d)40(1a)

Keep cluster utilization as low aspossible To avoid read-only statekeep below 92

HyperFlex clusters storage utilization isabove 92 usable capacity

CSCvr20922

Release Notes for Cisco HX Data Platform Release 4050

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1b)NAAll the disks in the node got locked afterrebooting the node

We successfully unlocked usingsed-clientsh -U command but wantedto test with another reboot and driveslocked again

CSCvs74286

35(2g)To recover the VM copy over thedata disks and attach them to thenew VM

After enable encryption HX connectencryption status showing Caution butall of the disks enable encryptionsuccessful

-Self Encrypting Drives Service isrunning on all of the nodes

-There is no error message of encryption

- USB0 interface is up

-All SED disks showing supported 1enabled 1 locked 0

CSCvs93245

40(1b)Enable kernelpanic_on_oops as apersistent configuration This willcause the node to panic and rebootimmediately

A cluster node running HX release40(1b) may hang in the Linux kernelThis is classified as an oops and adeviation from the expected behavior

CSCvs54285

Release Notes for Cisco HX Data Platform Release 4051

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Enabling K8iscsi stack in Hx clusterCSCvs41324

Release Notes for Cisco HX Data Platform Release 4052

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Steps to enable K8 on HX cluster

Perform following steps on ALLthe controller VMs

1 Updateetcinitscvmclientconf toenable tune iscsiEnable totrue

sed -iesiscsiEnable=falseiscsiEnable=trueetcinitscvmclientconf

2 Run following initctlcommand to reloadconfiguration

initctl reload-configuration

3 Restart scvmclient process

stop scvmclient startscvmclient

4 Run following initctlcommand to mount SYSTEMDS

initctl emit --no-waitsystem-datastore-created

5 Verify that iscsiEnable tuneis set to true using thefollowing command

ps -eaf | grep scvmclient |grep -v grep

root 6241 1 1 Dec06 014007optspringpathstorfs-corescvmclient-T logEnabled=false -TlogSyslogEnabled=true -TlogEchoToScreen=false -TstatLoggingToFile=false -TstatLoggingToSyslog=true-TlogDir=varlogspringpath-TnfsBackendServerList=1010748100-T iscsiEnable=true -TiscsiUseAsync=true -T

Release Notes for Cisco HX Data Platform Release 4053

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

iscsiConfigFilePath=nfsSYSTEMistgtconfps -eaf | grep scvmclient |grep -v grep

35(2e)Following procedure can be usedto check NFS Queue Depth andincrease if needed

rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth256 lt- Low value[rootHXESXI1] vsish -e setvmkModulesnfsclientmnt[DATASTORE]propertiesmaxQDepth 1024[rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth1024 lt- Optimalvalue

This requires an ESXi host rebootto take effect

Please place the node in HyperflexMaintenanceMode and gracefullyreboot the node for the changes tobe applied

HyperFlex Datastore NFS Queue Depthshows as 256 which can lead toperformance (including latency) issues

CSCvr83056

40(1b)This is cosmetic The error will goaway once the cluster returns tohealthy status

When a single node is offline in clusterstcli cluster storage-summary shows twonodes unavailable rootSCVM~ stclicluster storage-summary

CSCvs47735

35(2g)Modify both the root and adminpassword for the SCVMs to be thesame

stcli security password set -uadmin

Cluster expansion fails at Config Installerstage when the root and admin passwordfor the storage controller (SCVM) aredifferent

CSCvs69317

35(2d)NAAfter successfully changing the DNSserver on the HX controller we still cansee the original DNS entry that was addedduring the deployment

CSCvs69154

Release Notes for Cisco HX Data Platform Release 4054

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Customers are strongly encouragedto work with TAC in order toidentify and remediate this issue

1 Change getcluster from 2m to8m inoptspringpathstorfs-mgmtstMgr-10confapplicationconfon all nodes

2 Restart stMgr on all nodes

You may see this error message after afailed upgrade or other task such asattempting to enter a node into HXmaintenance mode

CSCvs53555

40(2a)Run the following command oneach storage controller VM as root

grep -i monitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp sed -iesmonitor_dns_serverstruemonitor_dns_serversfalseoptspringpathhx-diag-toolswatchdog_configjsonampamp grep -imonitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp restart watchdog

Receive the following alertevent in HXConnect HX Controller VMHOSTNAME one or more configuredDNS servers not responding

CSCvt13947

35(2b)Delete any empty (Size 0) log filesunder varlogzookeeperversion-2ORvarlogzookeeperstandaloneversion-2

Zookeeper fails to start while Exhibitor isrunning however echo srvr returnsnothing

CSCvs21562

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

40(4e)1 De-commission the server

2 Power drain the server -REMOVE BOTH POWERCORDS ON THE BACK OFTHE SERVER FOR 10SECONDS THENREINSERTPOWERCORDS

3 Re-commission the server

M2 boot disk is missing from serverinventory after upgrade to server firmware40(4e) As a result server fails to boot toOS installed in the M2 disk The issuepersists after re-acknowledgement as wellas de-commission andre-acknowledgement of the server

CSCvs02466

Release Notes for Cisco HX Data Platform Release 4055

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Re-start the flusher manuallyMany disks taken offline by a fault ormanually resulting in ENOSPC oncluster

CSCvq32721

35(2e)Clean up fail over cluster and thenrecreate the fail over cluster Thereis no need to touch the HX storagecluster

[Hyper-V]When replicated DCwas usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2b)Manually install upgradescvmclient VIB andmake sure thatit matches with HXDP version

Upgrade from release HX 35(2a) to35(2x) 4X does not upgrade scvmclientRecovery point may not work

CSCvq54992

Management

30(1b)To recover the VM copy over thedata disks and attach them to thenew VM

[HyperV]VMshows up onmultiple nodesCSCvj22992

35(1a)1 Do not change MTU at ESXlevel on running cluster

2 Reset back to the original value

ChangingMTU from 9000 to 1500 on oneESX node causes storfs process to restarton all nodes

CSCvm99150

Release Notes for Cisco HX Data Platform Release 4056

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)After the installation (for newdeployments) or after an upgradeto 401a (for existingdeployments) perform thefollowing steps on all ControllerVMs

1 Edit the following tune files (onall Controller VMs)

optspringpathconfiglfftunes

optspringpathconfigvsi_16tbtunes

2 SetcleanerEnableSegSummaryCleaning

to false

3 After editing the above tunefiles

a SSH in to all the ControllerVMs

b Run storfstool -- -Z

c Run the below command andcheck the values Tune valueshould be true

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=true

d Type below commands andapplied the tunes changesdynamically

echo false gttmpstprocfssystemtunecleanerEnableSegSummaryCleaning

e Verify that tune values arechanged Run below commandValue should be false

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=false

f umount tmpstprocfs

A cluster with 8TB or 12TB disk drivesmay experience IO stalls for severalminutes after another node fails in thecluster

CSCvp23718

Release Notes for Cisco HX Data Platform Release 4057

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)1 Update the conf file manuallyto change to the link local addressto

2 Restart scvmclient on thecontroller and ESX

HX 35(2b) FlexVol to HX 40 CSIupgrade does not work FlexVol continuesto work

CSCvp26319

35(2a)Disable rebalance until node healsthen reenable rebalance

7-node cluster healing very slowlybecause rebalance change AMS reqs areblocking healing process

CSCvp62512

35(2a)NAWith moderate cluster storage capacityusage under site failover cluster clustermay experience an all paths (APD) downcondition

CSCvq49412

40(1a)NAThis product includes Third-partySoftware that is affected by thevulnerabilities identified by the followingCommon Vulnerability and Exposures(CVE) IDs

CVE-2016-0762 CVE-2016-6797CVE-2016-6816 CVE-2016-8735CVE-2017-5647 CVE-2017-12615CVE-2017-12616 CVE-2017-12617CVE-2017-7674 CVE-2018-1304CVE-2018-8014 CVE-2018-1336CVE-2018-8034 CVE-2018-11784CVE-2019-0232

CSCvp36364

35(2d)NAstcli cluster info command needs toprovide UCSM VIP address Currently itshows ucsm-hostcom

CSCvq11456

35(2a)No workarounds currently Thestale entries do not affect the dayto day operations of HyperFlex

HyperFlex upgrade validation failedbecause cluster has extraneous stNodes ininternal database

CSCvq32530

35(2a)NAWhen not in maintenancemode all drivesappear in HX Connect and get updated asexpected

When in maintenance mode extra drivesare shown in HX Connect with no slotnumbers

CSCvq42378

35(2b)Contact TAC to cleanup duplicatenode

Nodes duplicated in HX Connect aftercluster recreate

CSCvq61825

Release Notes for Cisco HX Data Platform Release 4058

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)NACurrently HyperFlex installs on Hyper-Vdo not contain the stcli security whitelistcommandset

CSCvq66245

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

Release Notes for Cisco HX Data Platform Release 4059

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)[Hyper-V] Converged node expansionfails for incoming node with VIC 14xx

CSCvq39523

Release Notes for Cisco HX Data Platform Release 4060

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

1 Set Jumbo frame inWindowsHypervisor for incomingnode

Get-NetAdapter storage-data-a|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

Get-NetAdapter storage-data-b|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

2 Verify all interfaces are setcorrectly to support Jumbo frames

Get-NetAdapter storage |Get-NetAdapterAdvancedProperty| RegistryKeyword -matchjumbo | ft -auto

You should receive the followingmessage

Name DisplayName DisplayValue

RegistryKeyword RegistryValue

---- ----------- ------------

--------------- -------------

vswitch-hx-storage-data Jumbo

Packet 9014 Bytes

JumboPacket 9014

storage-data-b Jumbo Packet

Bytes 9014 JumboPacket

9014

storage-data-a Jumbo Packet

Bytes 9014 JumboPacket

9014

3 Reboot only the incoming nodeor the node being expanded

4 Retry Cluster Expansion in

Release Notes for Cisco HX Data Platform Release 4061

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

Table 5 Supported Microsoft Software versions

VersionMicrosoft Component

Windows Server 2016 Datacenter Core amp DesktopExperience

ForWindows Server 2016Datacenter Coreand Desktop Experience the Windows2016 ISO image should be Update BuildRevision (UBR) 1884 at a minimum

Note

Windows Server 2019Datacenter-DesktopExperienceis supported starting from HXDP 401(a) onwards

For Windows Server 2019 DesktopExperience theWindows 2019 ISO imageshould be Update Build Revision (UBR)107 at a minimum

Note

Windows Server 2019 DatacenterndashCore is notsupported currently

Also note that the following are currently notsupported

OEM activated ISOs and Retail ISOs are notsupported

Earlier versions ofWindows Server such asWindows2012r2 are not supported

Non-English versions of the ISO are not supported

Windows Operating System (Windows OS)

A Windows 2012 or later domain and forestfunctionality level

Active Directory

Supported Microsoft License Editions

The Microsoft Windows Server version that is installed on one or more HyperFlex hosts must be licensed asper Microsoft licensing requirements listed on Microsoft Licensing

Browser Recommendations

Use one of the following browsers to run the listed HyperFlex components These browsers have been testedand approved Other browsers might work but full functionality has not been tested and confirmed

Table 6 Supported Browsers

HX ConnectHX Data PlatformInstaller

Cisco UCS ManagerCisco IntersightBrowser

11 or higher11 or higher11 or higherNAMicrosoft InternetExplorer

70 or higher70 or higher57 or higher62 or higherGoogle Chrome

Release Notes for Cisco HX Data Platform Release 4015

Browser Recommendations

HX ConnectHX Data PlatformInstaller

Cisco UCS ManagerCisco IntersightBrowser

60 or higher60 or higher45 or higher57 or higherMozilla Firefox

NANA9 or higher10 or higherApple Safari

NANA35 or higherNAOpera

Notes

bull Cisco HyperFlex Connect

The minimum recommended resolution is 1024 X 768

bull Cisco HX Data Platform Plug-in

The Cisco HX Data Platform Plug-in runs in vSphere For VMware Host Client System browserrequirements see the VMware documentation at httpswwwvmwarecomsupportpubs

The HX Data Platform Plug-in is not displayed in the vCenter HTML client You must use the vCenterflash client

bull Cisco UCS Manager

The browser must support the following

bull Java Runtime Environment 16 or later

bull Adobe Flash Player 10 or higher is required for some features

For the latest browser information about Cisco UCSManager refer to themost recent Cisco UCSManagerGetting Started Guide

Cisco HX Data Platform Storage Cluster SpecificationsCluster Limits

bull Cisco HX Data Platform supports up to 100 clusters managed per vCenter as per VMware configurationmaximums

bull Cisco HX Data Platform supports any number of clusters on a single FI domain Each HX convergednode must be directly connected to a dedicated FI port on fabric A and fabric B without the use of a FEXC-series compute only nodes must also directly connect to both FIs B-series compute only nodes willconnect through a chassis IO module to both fabrics In the end the number of physical ports on the FIwill dictate the maximum cluster size and maximum number of individual clusters supported in a UCSdomain

The following table provides Cisco HX Data Platform Storage Cluster Specifications

Release Notes for Cisco HX Data Platform Release 4016

Cisco HX Data Platform Storage Cluster Specifications

Stretched Cluster (Availableon ESX Only)

Microsoft Hyper-VVMware ESXiNode

AllNVMe -HXAF220c-M5SN

HX240c-M5LHX220cM5

HX220cAF M5

HX240cM5

HX240cAF M5

HX240c-M5LHX220cM5

HX220cAF M5

HX240cM5

HX240cAF M5

HXAF220c-M5SN

AllNVMe -HXAF220c-M5SN

ESXi 65or 67only NotsupportedwithESXi 60orHyper-V

HX220cM5 Edge

HXAF220cM5 Edge

HX220cM4 Edge

HXAF220cM4 Edge

HX240c-M5LHX220cM5

HX220cAF M5

HX240cM5

HX240cAF M5

HX220cM4

HX220cAF M4

HX240cM4

HX240cAF M4

HXServers

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

C220M5C240M5B200M4B200M5

C240M5C220M5B200M4B200M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

mdashB200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

Compute-OnlyUCSB-SeriesC-SeriesServers

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

Convergednodesonly

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

SupportedNodes

Release Notes for Cisco HX Data Platform Release 4017

Cisco HX Data Platform Storage Cluster Specifications

Stretched Cluster (Availableon ESX Only)

Microsoft Hyper-VVMware ESXiNode

NA(requiresEnterpriseHXDP-PLicense)

NA(requiresEnterpriseHXDP-PLicense)

NA(requiresEnterpriseHXDP-PLicense)

Convergednodes3-16(12TBHDDoption isnotsupportedforHyperV)

Compute-onlynodes0-16

Convergednodes3-16

Compute-onlynodes0-16

NA(requiresEnterpriseHXDP-PLicense)

M4Convergednodes 3

M5Convergednodes23or 4

RequiresHXDP-ELicense

Convergednodes3-16(12TBdrive islimited toamaximumof 8nodes)

Compute-onlynodes0-16(12TBdrive islimited toamaximumof 8nodes)

Convergednodes3-32(76TBdriveconfigsarelimited tomaximumof 12drivesnodeon AF240)

Compute-onlynodes0-32

HXDP-SLicensedNodeLimits

11 ratioofHXDP-StoComputeonlynodes

(MinmdashMax)

Convergednodes2-16 perSite

Compute-onlynodes0-21 perSite

(up tomaxclustersize)

Convergednodes2-8 perSite

Compute-onlynodes0-16 perSite

(up tomaxclustersize)

Convergednodes2-16 perSite

Compute-onlynodes0-21 perSite

(up tomaxclustersize)

Convergednodes3-16(12TBHDDoption isnotsupportedforHyperV)

Compute-onlynodes0-16

Convergednodes3-16

Compute-onlynodes0-16

Convergednodes3-32

ComputeOnlynodes0-32

(up tomaxclustersize)

Convergednodes 3

(requiresHXDP-ELicense)

Convergednodes3-16(12TBdrive islimited toamaximumof 8nodes)

Compute-onlynodes0-32(12TBdrive islimited toamaximumof 16nodes)

Convergednodes3-32(76TBdriveconfigsarelimited tomaximumof 12drivesnodeon AF240)

Compute-onlynodes0-32

(up tomaxclustersize)

HXDP-PLicensedNodeLimits

12 ratioofHXDP-PtoComputeonlynodes

(MinmdashMax)

24 perSite 64percluster

24 perSite 48percluster

32 perSite 64percluster

32326434864MaxClusterSize

Release Notes for Cisco HX Data Platform Release 4018

Cisco HX Data Platform Storage Cluster Specifications

Stretched Cluster (Availableon ESX Only)

Microsoft Hyper-VVMware ESXiNode

212121111121mdash2121MaxComputetoConvergedratio

NoExpansion

Requires Enterprise license

Requires uniform expansion across both sites

Guidelines and Limitationsbull Starting with release 40(2a) SCVM is no longer needed on a Compute node

bull HX REST API Access Token Management ndash Applications leveraging HX REST APIs should re-useaccess tokens when making API calls Once obtained using the AAA Obtain Access Token API accesstokens are valid for 18 days (1555200 seconds) In addition AAA enforces rate limiting on ObtainAccess Token API requests in a 15 minute window auth can be invoked (successfully) a maximum of5 times A user is allowed to create a maximum of 8 unrevoked tokens Subsequent call to auth willautomatically revoke the oldest issued token tomake room for the new token Amaximum of 16 unrevokedtokens can be present in system In order to prevent brute-force attacks after 10 consecutive failedauthentication attempts a user account is locked for a period of 120 secondsFor more information seeCisco HyperFlex Systems REST API Reference guide

HxConnect makes use of AAA Authentication REST API for login and the above rate limit applies toHxConnect also

Upgrade Guidelines

The following list is a highlight of critical criteria for performing an upgrade of your HyperFlex system

bull Unsupported HX Data Platform 17x 18x 20 21x 25x and 26x clustersmdashUsers from anyversion prior to 26(1a) must step through an intermediate version before upgrading to 40 or later releasesIf you need to upgrade your environment from a Cisco HyperFlex HX Data Platform software releasethat is past the last date of support to the latest suggested release on the Cisco Software Download sitesee Cisco HyperFlex Systems Upgrade Guide for Unsupported Cisco HXReleases For more informationsee the Software Advisory for CSCvq66867 WARNING Only Use HXDP 26(1e) Upgrade PackageWhen Upgrading From HXDP 18(1a)-18(1e)

bull Hypercheck Health Check UtilitymdashCisco recommends running this proactive health check utility onyour HyperFlex cluster prior to upgrade These checks provide early visibility into any areas that mayneed attention andwill help ensure a seamless upgrade experience For more information see the HyperFlexHealth amp Pre-Upgrade Check Tool TechNote for full instructions on how to install and run Hypercheck

bull vSphere 67 Software AdvisorymdashDo not upgrade to Cisco HX Data Platform Release 40(1a) whenrunning ESXi 67U1 EP06 (build 11675023) Do not upgrade to 67U1 EP06 (build 11675023) ifrunning Cisco HX Data Platform Release 40(1a) See the Software Advisory CSCvo56350 for furtherdetails

Release Notes for Cisco HX Data Platform Release 4019

Guidelines and Limitations

The software build version posted at release will override any other local versions

bull Required vCenter upgrademdashFor enhanced security Cisco HX Data Platform Release 35(1a) or laterrequires the use of TLS 12 Therefore vCenter must be upgraded to 60 U3f or later before upgradingto Cisco HX Data Platform Release 35 or later In addition ESXi should be upgraded as required tomeet HX Data Platform compatibility requirements

bull Minimum HXDP version for upgrademdashHXData Platform clusters running 26(1a) or later may upgradedirectly to 40 using the HX Connect UI

bull Cluster ReadinessmdashEnsure that the cluster is properly bootstrapped and the updated plug-in is loadedbefore proceeding Manual cluster bootstrap is required for upgrade from a pre-35 release

bull Cluster ReadinessmdashEnsure that the cluster is properly bootstrapped and the updated plug-in is loadedbefore proceeding Manual cluster bootstrap is required for HX releases earlier than 35(1a) For moreinformation see theManual Bootstrap Upgrade Process in the Cisco HyperFlex Systems Upgrade Guidefor VMware ESXi Release 40 Do not skip this cluster bootstrap step it is required for all upgradesuntil HX Release 35(1a) Auto bootstrap is supported beginning with HX release 35(1a) For moreinformation see the Auto Bootstrap Upgrade Process from HX Connect UI in the Cisco HyperFlexSystems Upgrade Guide for VMware ESXi Release 40

Manual bootstrap is not supported on Intersight clusters

bull Initiating Upgrade―Use the HX Connect UI or CLI stcli commands when upgrading from 25(1a)or later releases Use either the CLI stcli commands or the HX Data Platform Plug-in to the vSphereWeb Client when upgrading from a pre-25(1a) release The vCenter plug-in should not be used forupgrades starting with the 25(1a) release The vCenter plug-in should not be used for upgrades startingwith the 25(1a) release

If the current cluster version is at 35(1a) or above you do not need to use the stcli command Directupgrade to 40 is possible

bull Complete your Upgrade―The self-healing (or rebalance) capability is disabled temporarily during theupgrade window If the upgrade fails you should complete the upgrade as soon as possible

bull ESXi and HXDP Compatibility―Ensure your cluster is running a compatible version of ESXi basedon the running the HXData Platform version (see the section Software Requirements for VMware ESXi)ESXi compatibility is determined by the major version and update release of ESXi It is generally bestto upgrade HXDP and ESXi together if combining the upgrade operations into a single optimized rebootWhen running a split upgrade first upgrade the HX Data Platform then proceed to upgrade ESXi

bull If Upgrading to vSphere 65

bull Certain cluster functions such as native and scheduled snapshotsReadyClones and Enter or Exit HX Maintenance Mode will not operatefrom the time the upgrade is started until the HX Data Platform upgrade iscomplete

bull After upgrading ESXi using the offline zip bundle use the ESX ExitMaintenance Mode option The HX Exit Maintenance Mode option doesnot operate in the vSphere Web Client until the HX Data Platform upgradeis complete

Note

Release Notes for Cisco HX Data Platform Release 4020

Guidelines and Limitations

bull vSphere 60 UpgradesmdashUsers on vSphere 60 migrating to 65 upgrade components in the followingorder

1 Upgrade HX Data Platform and UCS firmware

2 Upgrade HX Data Platform and ESXi

3 Upgrade HX Data Platform only first then upgrade ESXi or UCS firmware or both

bull M4 Server Firmware UpgradesmdashUpgrade server firmware to ensure smooth operation and to correctknown issues Specifically newer SAS HBA firmware is available in this release and is recommendedfor long-term stability

bull Users are encouraged to upgrade to 31(3c) C-bundle or later whenever possible

bull Users running C-bundle versions before 31(2f) must upgrade server firmware by performing acombined upgrade of UCS server firmware (C-bundle) to 31(3c) or later and HX Data Platform to25 Do not split the upgrade into two separate operations

bull If the cluster is already on 31(2f) C-bundle or later you may perform an HX Data Platform onlyor combined upgrade as required

bull M5 Server Firmware UpgradesmdashM5 generation servers must run firmware version 32(2d) or later

bull Firmware Downgrades mdashDowngrading UCSM from the HX-installer is not supported

bull M4M5 Mixed DomainsmdashA mixed domain occurs when a new separate M5 cluster is installed underthe same UCS domain that contains existing M4 clusters Under these conditions orchestrated UCSserver firmware upgrade will not operate until Cisco HX Data Platform Release 26 or later is installedon the M4 clusters Therefore it is best practice to first upgrade UCS server firmware to the latest 31(3)or 32(2) patch release before adding a new M5 cluster to the existing UCS domain Additionally any17 HX Data Platform clusters must first be upgraded before adding any new M5 clusters to the samedomain

bull Maintenance WindowmdashIf upgrading both HX Data Platform and UCS firmware you can select eithera combined or split upgrade through the vSphere HX Data Platform Plug-in depending on the length ofthe maintenance window Cisco UCSManager infrastructure upgrade is only supported using AutoInstalland the direct server firmware upgrade should be performed only through the upgrade orchestrationframework provided by the HX Data Platform Plug-in

bull Unsupported Self-Encrypting Drives (SEDs)mdashIf adding or replacing self-encrypting drives (SEDs)that have been recently qualified in newer versions of HX Data Platform insert the new drives only afterupgrading HX Data Platform to a compatible version All drives must be SED drives mixing SED andnon-SED is not supported

bullbull Enabling External Host AccessmdashWith Cisco HX Data Platform Release 40(1a) port 445 on themanagement network is blocked for enhanced security Note that prior to 40 port 445 port was openenabling external host access If you are upgrading to 40(1a) from a prior release and would like tocontinue external host access you can use a utility to open select hosts For more information aboutenabling external host access see the ConfiguringHyperFlex Share to SCVMM section in the InstallationGuide for Microsoft Hyper-V

Release Notes for Cisco HX Data Platform Release 4021

Guidelines and Limitations

Mixed Cluster Expansion Guidelinesbull Expanding existing M4 cluster with M5 converged nodes is supported

bull Expanding existing M5 cluster with M4 converged nodes is not supported

bull Expanding existing mixed M4M5 cluster with M4 or M5 converged nodes is supported

bull Adding any supported compute-only nodes is permitted with all M4 M5 and mixed M4M5 clustersusing the HX Data Platform 26 or later Installer Some example combinations are listed here manyother combinations are possibleExample combinationsExpand mixed M4M5 cluster with compute-only B200 C220 C240 M4M5Expand M4 cluster with compute-only B200 M5 C220 M5 C240M5

bull Only expansion workflow is supported to create a mixed cluster Initial cluster creation with mixedM4M5 servers is not supported

bull All M5 servers must match the form factor (220240) type (HybridAF) security capability (Non-SEDonly) amp disk configuration (QTY capacity and non-SED) of the existing M4 servers

bull HX220-M5 will use a maximum of 6 capacity disks (2 disk slots to remain empty) when mixedwith HX220-M4

bull HX Edge SED LFF Hyper-V and Stretched Clusters do not support mixed M4 and M5 clusters

Security FixesThe following security issues are resolved

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-13132 CVE-2019-9924CSCvq6313840(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-11719 CVE-2019-11727CVE-2019-11729

CSCvq7124040(2a)

Release Notes for Cisco HX Data Platform Release 4022

Mixed Cluster Expansion Guidelines

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-1125CSCvr0633940(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2015-9383 CVE-2018-14498CVE-2018-20406 CVE-2018-20852CVE-2019-10160 CVE-2019-13117CVE-2019-13118 CVE-2019-14287CVE-2019-14973 CVE-2019-15903CVE-2019-17546 CVE-2019-18197CVE-2019-18218 CVE-2019-5010CVE-2019-5094 CVE-2019-5481CVE-2019-5482 CVE-2019-9636CVE-2019-9740 CVE-2019-9947CVE-2019-9948

CSCvs0609440(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2017-13168 CVE-2017-18174CVE-2017-18216 CVE-2018-10876CVE-2018-10877 CVE-2018-10878CVE-2018-10879 CVE-2018-10880CVE-2018-10881 CVE-2018-10882CVE-2018-10902 CVE-2018-10938CVE-2018-12233 CVE-2018-12896CVE-2018-13053 CVE-2018-13094CVE-2018-13096 CVE-2018-13405CVE-2018-13406 CVE-2018-14609CVE-2018-14617 CVE-2018-14633CVE-2018-14734 CVE-2018-15572CVE-2018-15594 CVE-2018-16276CVE-2018-16658 CVE-2018-17182CVE-2018-17972 CVE-2018-18021CVE-2018-18690 CVE-2018-18710CVE-2018-6554 CVE-2018-6555CVE-2018-9363

CSCvp6501940(2a)

This vulnerability is due to insufficientHTML iframe protection An attackercould exploit this vulnerability bydirecting a user to an attacker-controlledweb page that contains amaliciousHTMLiframe A successful exploit could allowthe attacker to conduct clickjacking orother clientside browser attacks

NACSCvo9851640(2a)

Release Notes for Cisco HX Data Platform Release 4023

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerability is due to insufficientauthentication for the statistics collectionservice An attacker could exploit thisvulnerability by sending properlyformatted data values to the statisticscollection service of an affected deviceA successful exploit could allow theattacker to cause the web interfacestatistics view to present invalid data tousers

NACSCvj9558440(2a)

The vulnerability is due to insufficientCSRF protections for the web UI on anaffected device An attacker could exploitthis vulnerability by persuading a user ofthe interface to follow a malicious linkA successful exploit could allow theattacker to perform arbitrary actions withthe privilege level of the affected user

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvp2434340(2a)

Vulnerabilities with a version of cJSONidentified by CVE-2019-11834 andCVE-2019-11835

For more information see the relatedCisco Security Advisory

CVE-2019-11834 CVE-2019-11835CSCvq1994940(2a)

HX ESXi image patches for MCEPSCand TAA vulnerabilities with VMwareESXi identified by CVE-2018-12207 andCVE-20190-11135

For more information see the relatedCisco Security Advisory

CVE-2018-12207 CVE-2019-11135CSCvr5439840(2a)

CP networking vulnerabilities affectingthe Linux kernel identified byCVE-2019-11477 CVE-2019-11478CVE-2019-11479

For more information see the relatedCisco Security Advisory

CVE-2019-11477 CVE-2019-11478CVE-2019-11479

CSCvq1954640(2a)

Qualification of Microsoft securitypatches for vulnerabilities withMicrosoftHyper-V hypervisor identified withCVE-2018-12207 CVE-2019-11135

For more information see the relatedCisco Security Advisory

CVE-2018-12207 CVE-2019-11135CSCvr5439940(2a)

Release Notes for Cisco HX Data Platform Release 4024

Security Fixes

DescriptionCVEDefect IDRelease

OpenSSH vulnerabilities identified byCVE-2019-6111

For more information see the relatedCisco Security Advisory

CVE-2016-10708 CVE-2018-15473CVE-2018-20685 CVE-2019-6109CVE-2019-6111

CSCvp7646340(2a)

Multiple vulnerabilities associated withOpenSSL and LibNSS

For more information see the relatedCisco Security Advisory

CVE-2016-2105 CVE-2016-2106CVE-2016-2107 CVE-2016-2109CVE-2016-2176 CVE-2016-2177CVE-2016-2178 CVE-2016-2179CVE-2016-2180 CVE-2016-2181CVE-2016-2182 CVE-2016-2183CVE-2016-6302 CVE-2016-6303CVE-2016-6304 CVE-2016-6305CVE-2016-6306 CVE-2016-6307CVE-2016-6308 CVE-2016-7055CVE-2016-8610 CVE-2017-3731CVE-2017-3732 CVE-2017-3735CVE-2017-3736 CVE-2017-3737CVE-2017-3738 CVE-2018-0495CVE-2018-0732 CVE-2018-0734CVE-2018-0735 CVE-2018-0737CVE-2018-0739 CVE-2018-12384CVE-2018-12404 CVE-2018-5407CVE-2019-1559

CSCvp6667940(2a)

libpng vulnerability identified byCVE-2016-10087

For more information see the relatedCisco Security Advisory

CVE-2016-10087CSCvp6655540(2a)

Vulnerabilities associated with libjpg andlibpng

For more information see the relatedCisco Security Advisory

CVE-2014-9092 CVE-2016-3616CVE-2017-15232 CVE-2018-11212CVE-2018-11213 CVE-2018-11214CVE-2018-1152 CVE-2018-13785

CSCvp3458640(2a)

Vulnerabilities associated with Glibidentified by CVE-2018-16428 andCVE-2018-16429

For more information see the relatedCisco Security Advisory

CVE-2018-16428 CVE-2018-16429CSCvp3120740(2a)

Multiple vulnerabilities associated withcurl expat python 27 python 35 36and 37 freetype and giflib

For more information see the relatedCisco Security Advisory

CVE-2019-15133 CVE-2019-15903CVE-2019-5010 CVE-2019-5481CVE-2019-5482 CVE-2019-9636CVE-2019-9740 CVE-2019-9947CVE-2019-9948

CSCvr3690340(2a)

Release Notes for Cisco HX Data Platform Release 4025

Security Fixes

DescriptionCVEDefect IDRelease

Multiple third party softwarevulnerabilities

For more information see the relatedCisco Security Advisory

CVE-2019-14379 CVE-2019-12384CVE-2019-14439

CSCvq9203240(2a)

Vulnerabilities associated with elfutils

For more information see the relatedCisco Security Advisory

CVE-2018-16062 CVE-2018-16402CVE-2018-16403 CVE-2018-18310CVE-2018-18520 CVE-2018-18521CVE-2019-7149 CVE-2019-7150CVE-2019-7665

CSCvq4325040(2a)

Vulnerabilities associated with vim

For more information see the relatedCisco Security Advisory

CVE-2017-5953 CVE-2019-12735CSCvq4323040(2a)

Vulnerabilities associated with sqlite3

For more information see the relatedCisco Security Advisory

CVE-2016-6153 CVE-2017-10989CVE-2017-13685 CVE-2017-2518CVE-2017-2519 CVE-2017-2520CVE-2018-20346 CVE-2018-20505CVE-2018-20506 CVE-2019-8457CVE-2019-9936 CVE-2019-9937

CSCvq4321340(2a)

Vulnerabilities associated with glib20

For more information see the relatedCisco Security Advisory

CVE-2018-20843CSCvq4320940(2a)

Vulnerabilities associated with expat

For more information see the relatedCisco Security Advisory

CVE-2018-20843CSCvq4320540(2a)

Vulnerabilities associated with bzip2

For more information see the relatedCisco Security Advisory

CVE-2016-3189 CVE-2019-12900CSCvq4319440(2a)

Mulitple Vulnerabilities associated wtihNodeJS

For more information see the relatedCisco Security Advisory

CVE-2018-12115 CVE-2018-0734CVE-2018-5407 CVE-2018-12120CVE-2018-12121 CVE-2018-12122CVE-2018-12123 CVE-2018-12116CVE-2019-5737 CVE-2019-5739

CSCvq1069440(2a)

Vulnerabilities associated with PalletsJinja strformat_map and identified byCVE-2019-10906 and CVE-2016-10745

For more information see the relatedCisco Security Advisory

CVE-2019-10906 CVE-2016-10745CSCvq1038840(2a)

Release Notes for Cisco HX Data Platform Release 4026

Security Fixes

DescriptionCVEDefect IDRelease

Multiple Security Vulnerabilities

For more information see the relatedCisco Security Advisory

CVE-2014-8501 CVE-2014-9939CVE-2015-9262 CVE-2016-10087CVE-2016-2226 CVE-2016-4487CVE-2016-4488 CVE-2016-4489CVE-2016-4490 CVE-2016-4491CVE-2016-4492 CVE-2016-4493CVE-2016-6131 CVE-2018-10963CVE-2018-13785 CVE-2018-17100CVE-2018-17101 CVE-2018-18557CVE-2018-18661 CVE-2018-7456CVE-2018-8905

CSCvq0917840(2a)

Vulnerabilities associated withlibseccomp

For more information see the relatedCisco Security Advisory

CVE-2019-9893CSCvq0756840(2a)

Vulnerabilities associated withgdk-pixbuf

For more information see the relatedCisco Security Advisory

CVE-2017-12447CSCvq0675540(2a)

Vulnerabilities associated withpython-crypto

For more information see the relatedCisco Security Advisory

CVE-2018-6594CSCvp9381740(2a)

Vulnerabilities associated with wget

For more information see the relatedCisco Security Advisory

CVE-2018-20483 CVE-2019-5953CSCvp8672140(2a)

Multiple vulnerabilities associated withPython Crypto

For more information see the relatedCisco Security Advisory

CVE-2018-6594CSCvp6674840(2a)

Vulnerabilities associated withsensible-utils

For more information see the relatedCisco Security Advisory

CVE-2017-17512CSCvp6673440(2a)

Multiple vulnerabilities associated withPython

For more information see the relatedCisco Security Advisory

CVE-2018-1000030CSCvp6668940(2a)

Release Notes for Cisco HX Data Platform Release 4027

Security Fixes

DescriptionCVEDefect IDRelease

Multiple patch vulnerabilities identifiedby CVE-2016-10713

For more information see the relatedCisco Security Advisory

CVE-2016-10713CVE-2018-1000156 CVE-2018-6951

CSCvp6667240(2a)

Vulnerabilities associatedwith Little CMSidentified by CVE-2016-10165

For more information see the relatedCisco Security Advisory

CVE-2016-10165 CVE-2018-16435CSCvp6666440(2a)

Vulnerabilities associated with TenableScan identified byCVE-2011-5325CVE-2019-11068CVE-2016-7076CVE-2017-1000368

For more information see the relatedCisco Security Advisory

CVE-2011-5325 CVE-2016-7076CVE-2017-1000368CVE-2019-11068

CSCvp6474640(2a)

Vulnerabilities associated with libxml2294 and earlier as used in XMLSec1223 and earlier and other products andidentified by CVE-2016-9318

For more information see the relatedCisco Security Advisory

CVE-2016-9318 CVE-2017-16932CVE-2017-18258 CVE-2018-14404CVE-2018-14567

CSCvp3479240(2a)

Vulnerabilities associated with LFTPRemote File Names Unauthorized AccessVulnerability and identified byCVE-2018-10916

For more information see the relatedCisco Security Advisory

CVE-2018-10916CSCvp2926640(2a)

Vulnerabilities associated withCryptographyDeprecationWarning signerand verifier have been deprecated

For more information see the relatedCisco Security Advisory

CVE-2018-7750CSCvo3409740(2a)

Multiple Vulnerabilities in python python27 35

For more information see the relatedCisco Security Advisory

CVE-2019-16056CSCvr3979340(2a)

Release Notes for Cisco HX Data Platform Release 4028

Security Fixes

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toexecute commands as the root user

The vulnerability is due to insufficientinput validation An attacker could exploitthis vulnerability by connecting to thecluster service manager and injectingcommands into the bound process Asuccessful exploit could allow the attackerto run commands on the affected host asthe root user

Cisco has released software updates thataddress this vulnerability There areworkarounds that address thisvulnerability

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvq2417640(1b)35(2g)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

The vulnerability is due to an unprotectedlistening interface An attacker couldexploit this vulnerability by connectingto the listening interface and injectingcommands to the bound process Anexploit could allow the attacker to runcommands on the affected host as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvj9560640(1b)35(2g)

Release Notes for Cisco HX Data Platform Release 4029

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerabilities associated with theJVM18U121memory leak found duringVC alarms (concurrent 40 calls usingREST API)

CVE-2017-10053 CVE-2017-10067CVE-2017-10074 CVE-2017-10078CVE-2017-10081 CVE-2017-10087CVE-2017-10089 CVE-2017-10090CVE-2017-10096 CVE-2017-10101CVE-2017-10102 CVE-2017-10107CVE-2017-10108 CVE-2017-10109CVE-2017-10110 CVE-2017-10111CVE-2017-10115 CVE-2017-10116CVE-2017-10118 CVE-2017-10135CVE-2017-10176 CVE-2017-10193CVE-2017-10198 CVE-2017-10243CVE-2017-10274 CVE-2017-10281CVE-2017-10285 CVE-2017-10295CVE-2017-10345 CVE-2017-10346CVE-2017-10347 CVE-2017-10348CVE-2017-10349 CVE-2017-10350CVE-2017-10355 CVE-2017-10356CVE-2017-10357 CVE-2017-10388CVE-2017-3509 CVE-2017-3511CVE-2017-3526 CVE-2017-3533CVE-2017-3539 CVE-2017-3544CVE-2018-2579 CVE-2018-2582CVE-2018-2588 CVE-2018-2599CVE-2018-2602 CVE-2018-2603CVE-2018-2618 CVE-2018-2629CVE-2018-2633 CVE-2018-2634CVE-2018-2637 CVE-2018-2641CVE-2018-2663 CVE-2018-2677CVE-2018-2678 CVE-2018-2783CVE-2018-2790 CVE-2018-2794CVE-2018-2795 CVE-2018-2796CVE-2018-2797 CVE-2018-2798CVE-2018-2799 CVE-2018-2800CVE-2018-2814 CVE-2018-2815CVE-2018-2952 CVE-2018-3136CVE-2018-3139 CVE-2018-3149CVE-2018-3150 CVE-2018-3169CVE-2018-3180 CVE-2018-3183CVE-2018-3214 CVE-2019-2422

CSCvo8899740(1b)

Tomcat and Nginx logs are not beingcollected in the support bundle generatedthrough HX Connect in Release 35

NACSCvm5803140(1b)

The vulnerabilities associated with thelibmspack software package versionincluded in Cisco HX Data Platform

CVE-2018-18584

CVE-2018-18585

CSCvn3511940(1a)

Release Notes for Cisco HX Data Platform Release 4030

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerabilities associated withFasterXML Jackson-Databind TimeValueField Denial of Service

CVE-2018-14719

CVE-2018-14720

CVE-2018-1000873

CVE-2018-14721

CVE-2018-19360

CVE-2018-19362

CVE-2018-19361

CVE-2018-14718

CSCvn8228240(1a)

The vulnerabilities associated with theOpenSSL Protocol software packageversion included in Cisco HX DataPlatform

CVE-2013-3587CSCvo0505440(1a)

The vulnerabilities associated withThird-Party Software Denial of Service

CVE-2018-16487

CVE-2018-19361

CSCvo2781840(1a)

The vulnerabilities associated withLinuxkernel for Ubuntu 1710

CVE-2018-1092

CVE-2018-7492

CVE-2018-8087

CVE-2018-1068

CVE-2018-8781

CSCvm5314935(2a)

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)25(1d)stcli services dns remove should remove the DNS serverinfo from the interface files

CSCvs70967

40(2b)40(1b)HX 40(1b) new deployment from Intersight productioncloud

User receives following error while enabling CSI(Kubernetes integration)

Failure occurred during volume_access_enable Errorwas Nonzero exit code 1

CSCvt10522

40(2b)40(2a)Receive the following alertevent in HX Connect HXController VM HOSTNAME one or more configuredDNS servers not responding

CSCvt13947

Release Notes for Cisco HX Data Platform Release 4031

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)40(2a)API UI CLI show few Drive slots Empty after upgradeto 402a

The drives are seen in the sysmtool lsscsi and stclicommand outputs

The cluster is healthy with no errors

CSCvt14914

40(2b)35(2d)After successfully changing the DNS server on the HXcontroller we still can see the original DNS entry thatwas added during the deployment

CSCvs69154

40(2b)35(2e)HyperFlex Connect and VCenter show APD alarms forHX and non HX datastores This implies there is an issuewith HyperFlex when there may not be

CSCvs30080

40(2b)40(2a)An issue where all the disks in the node get locked afterrebooting the node

We successfully unlocked using sed-clientsh -Ucommand but wanted to test with another reboot anddrives locked again

CSCvs74286

40(2b)40(2a)When running stcli license commands on HyperFlexthe following error is seen

rootSpringpathControllertmp stcli license show all

Show smart licensing failed

Smart Agent is not ready please wait a minute and tryagain

CSCvt13929

40(2b)40(2a)After upgrading to HX 40(2a) the first RoboEdge nodethat gets rebooted then ignores persistent drivesPIDHX-HD24TB10K4KN

Cluster will be in degraded state and have reducedcapacity

CSCvt20144

40(2b)35(2b)Zookeeper fails to start while Exhibitor is runninghowever echo srvr returns nothing

CSCvs21562

40(2b)40(2a)When performing an HyperFlex upgrade a validationwarning may occur due to the host not having EnterprisePlus or Enterprise hypervisor licensing

Upgrade Validation Warning

ESXi host esx1labtest should be configured withVMware Enterprise license for upgrade to continue

CSCvs91787

Release Notes for Cisco HX Data Platform Release 4032

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)35(2g)Cluster expansion fails at Config Installer stage whenthe root and admin password for the storage controller(SCVM) are different

CSCvs69317

40(2b)35(2g)Panic while upgrading ESXiCSCvt06983

40(2b)40(2a)A cluster node running may hang in the Linux kernelThis is classified as an oops and a deviation from theexpected behavior

CSCvs54285

40(2b)35(2g)Rebalance failing on 10+10 node HX 35(2g) stretchcluster

CSCvs69007

40(2b)35(2d)The cluster becomes inaccessible to the IOVisorCSCvr54687

40(2b)40(2a)Panic on the storage controllerCSCvt61297

40(2b)40(2a)The size of support bundle is very large when collectedwith storfs-support command

CSCvt63306

40(2b)40(2a)storfs crash associated withFileSystemUsageWarningEvent orFileSystemUsageAlertEvent due to high usage ofvarstv above 80

CSCvt72807

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

40(2a)35(2d)In normal scenarios HX upgrade gets pendingACK afterupdating service profile In this case UCS only upgradewas stalled because after updating service profile HXupgrade was not getting pending ACK and it was alwayswaiting state

CSCvo39912

40(2a)35(1a)When using motherBoardReplace-12 to clean ZK fromold stNodepNodes resulted in unmounted datastore andmaking the size of the Hyperflex Datastores 0 resultingin all VMs in the cluster going offline

CSCvq39471

40(2a)26(1e)Upgrading cluster and getting error

Failed upgrade validations Checking vCenterconfiguration Reason Upgrade validations failed DRSFault Insufficient resources to satisfy configured failover

CSCvm77294

Release Notes for Cisco HX Data Platform Release 4033

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)The cluster expand fails on a node with DR replicationconfigured

When a HX cluster which has DR replication configuredis expanded we see the installer UI pulling in thereplicationVLAN information instead of themanagementVLAN information Even if we change that informationto the correct mgmt VLAN id and name it it does notseem to work as the node is configured with the VLANof the replicationVLAN in ESXi This leads to the failureof the node add with the host unreachable error

CSCvo70650

40(2a)35(1a)Customer reported that server didnt complete thefirmware upgrade automatically one by one

The user manually needs to put the host into maintenancemode and they manually acknowledged the pendingrequests to finish the UCS firmware upgrade

Maintenance policy is set to default (User-ack) as perthe design

CSCvo91624

40(2a)35(1i)When the cluster is in Failed state and stcli noderemove is attempted the output shows successful thoughthe node failed to get removed from the cluster

CSCvo93017

40(2a)35(1a)HyperFlex cluster upgrade may fail during validationwith a DRS Validation failed error

CSCvp31021

40(2a)35(2b)HX cluster expansion will fail with the following errormessage

MAC address pool configuration failure150[ErrorDescription] bad address block range definitioncollision

CSCvp58318

40(2a)35(2b)Memory usage by carbon cacheCSCvq34873

40(2a)35(2g)Upgrading ESXi cluster fails with error Nodemaintenance mode failed

CSCvr03240

40(2a)35(2e)storfs process does not automatically start on Exit HXMaintenance Mode or other tasks which restart StorageController

CSCvr88978

40(2a)35(2a)Post install script fails with message

Failed to execute ipmitool on HX node

CSCvp10707

40(2a)40(1a)HyperFlex expansionworkflow doesnt pull VLANnamecorrectly

CSCvp46539

Release Notes for Cisco HX Data Platform Release 4034

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)During HX cluster deployment - Validate ClusterCreation phase - HX Installer might fail with the error

fromvarlogspringpathstDeploylog

CSCvq45087

40(2a)35(2b)HX Installer fails to login to the SOL for configuringESX hosts

CSCvq91380

40(2a)35(2d)If DRS is enabled with configuration parameters otherthen fully automated ie manual or partially automatedthan this will change to fully automated during clusterexpansion

CSCvr44222

40(2a)35(1a)One of the validation task name has a typo The taskname is Validating node for mixed of SED and non-SEDdisks

It should be Validating node for mix of SED andnon-SED disks

CSCvp82175

40(2a)30(1i)This combination of using smaller NVMe drives is notsupported in expand operation For example if the drivesare only 375GB and they cannot be added to an existingcluster with larger caching SSD

CSCvo12359

40(2a)35(1a)HyperFlex includes a version of OpenSSL that is affectedby the vulnerabilities identified by the followingCommon Vulnerability and Exposures

CVE-2018-0495

CSCvp66679

40(2a)30(1e)SDK invoke exception when taking a native snapshot ona VM with GPU pci passthrough attached

CSCvo00511

40(2a)30(1i)Node replacement script fails due to EAM errorCSCvo62867

40(2a)35(1a)While pairing a cluster (HX release gt= 35) and cluster(HX release lt 35) remote replication network test failson the cluster (HX release lt 35)

CSCvo79760

40(2a)30(1b)Fixed the issue in the latest support-workflow bundleThis should not be hit anymore

CSCvo87061

40(2a)35(2a)On a 3 node cluster the MBreplace script fails as thescript looks for HEALTHY cluster status

CSCvo87080

40(2a)35(2a)MbReplace script (tar) hangs when running on 352b or40

CSCvp12359

Release Notes for Cisco HX Data Platform Release 4035

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)HX replication cleanup failing with error NA 35(2a) -INFODR state is not clean

CSCvp63958

40(2a)40(1a)The check_stig_parameters API incorrectly shows thecompliance state of the cluster as not STIG compliant

CSCvp66277

40(2a)35(2c)The snapshot creation on CBT enabled VM fails witherror Failed in vmreparent vmkfstools clone1

CSCvq06952

40(2a)35(2a)HyperFlex GUI when setting up remote DR-replicationpartnerpeer should show remote replication partner TCPport reachability test results

This will provide quick upfront connectivity results thatfrequently takes customer a while to manually check andconfirm

If reachability tests fail an info window and HX alertshould trigger that list the destination peer IP and thespecific port(s) that could not be reache

CSCvo60587

40(2a)35(2b)If you have more than one IP pool configured for areplication network and local replication network testsare failing you might be running into this issue

CSCvr67130

40(2a)35(2a)Skip task option in MBreplace and Software redeployscripts The scripts should have the option to skip tasksas user inputs and not by modifying the json file

CSCvp05204

40(2a)40(1a)Error noticed on SCVM Console print_req_error IOerror dev fd0 sector 0 on HyperV

It doesnt come up on SSH session

CSCvq34357

40(2a)30(1d)The VMNetwork switch is already created as part of theinstall process

Ideally the ip address assignment should be done duringthe initial install process as well No post install stepsshould be required altrnatively it could be done as partof a post install script

CSCvk36222

40(2a)25(1c)storfs service impacted during HX upgrade from HX21(1) to 25(1c)

HX node which is not being upgraded storfs servicestopped causing outage

CSCvg53223

40(2a)35(1b)HX Stretch Cluster Witness VM reverts to DHCP onreboot

CSCvp37536

40(2a)35(2b)HyperFlex Stretch Cluster Witness does not getprogrammed with an NTP server

CSCvp46578

Release Notes for Cisco HX Data Platform Release 4036

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2b)HyperFlex cluster does not heal after maintenance onnodeserver

CSCvr18528

40(2a)40(2a)High percentage of packet loss on the eth1 data interface9K MTU

CSCvr97089

40(2a)35(2f)The HyperFlex cluster healing state is stuck at 87 andis unable to progress

CSCvr16760

40(2a)40(2a)In some circumstances two replications can occur forthe same VM

From the GUI the replication status for this VM willshow stuck in an In Progress state but there are nojobs stuck in replication layer Subsequent Replicationsshould succeed

CSCvq22898

40(2a)35(2a)After an unexpected power outage the hyperflex clusteris unable to come up correctly

All processes are running time is synced and all nodescan vmkping each other as required

CSCvp13990

40(2a)40(1a)When specifying IP address instead of node ID stclinode maintenanceMode commands fail

CSCvp33657

40(2a)40(1a)2 node edge cluster if experiencing Intersightarbitratorconnectivity issues should NOT allowHXMaintenancemode to initiate as this will bring the cluster down

CSCvr31573

40(2a)35(2c)Old files in varsupportZKTxnlog are not purged withthe daily zklog-cleanup cron job

CSCvr89066

40(2a)35(2a)When a node is in MaintenanceMode any disk removalor replacement will be reflected in UI only after the nodeis brought back frommaintenancemode This is becausestorfs is not running on the node in maintenance andwill not be able to detect disk activities until it is broughtout of Maintenance Mode

CSCvo86431

40(2a)35(2c)Cluster might go offline and stop serving IO requestsrootcvm~ stcli cluster storage-summary--detailGet cluster storage summary failedjavanetConnectException Connection refused19216814210010207

CSCvr01645

40(2a)35(2d)A stretch cluster deployment fails with a messageFormatting nodes Node in Use on the Hyperflex installerand crmZoneType value is seen as 1 in storfscfg oroptspringpathconfigstretchtunes

CSCvq80340

Release Notes for Cisco HX Data Platform Release 4037

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)30(1c)During upgrade we upload the upgrade packagedownloaded from Ciscocom Upgrade packages are offile type tgz hxconnect acceptsstorfs-packages-401a-33028tar file a different type ofcompression and allows upgrade to start and fail whenit cannot decompress the file for bootstrap and scvmclientupgrade on ESXi

CSCvq63888

40(2a)40(2a)While attempting to install hyperflex via intersight thefollowing error is seen

Failed in Task Add hosts to vCenter Cluster with ErrorTry adding hosts manually to vCenter and retry Failedto get addressFailed to host by namelookupHostnamecompanycom on 000053 read udp12700145430-gt12700153 read connection refused

CSCvr52098

40(2a)35(2a)CoreAPI call may time out for cluster management APIwhile doing cluster create validation

CSCvo92952

40(2a)35(2d)Validation fails with Mixed mode expansion checkdue to Enhanced vMotion Compatibility Incompatibilitywhen the ESXi version is different

CSCvq95460

40(2a)30(1i)This failure is seen in installer when custom workflowis used hypervisor configuration + deploy (clean diskpartition=no) at Installing software packages on storagecontroller VM

CSCvr66309

40(2a)35(2d)The HX Installer apparently replaced a comma bysemicolon on the VLAN ID range while exporting theconfiguration to a JSON file

CSCvq93831

40(2a)40(1a)When using unsupported browsers HX Connect userswill not be able to login These users see an errorindicating unauthorized user

CSCvp22693

Release Notes for Cisco HX Data Platform Release 4038

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)40(1a)A vulnerability in the web-based management interfaceof Cisco HyperFlex Software could allow anunauthenticated remote attacker to conduct a cross-siterequest forgery (CSRF) attack on an affected system

The vulnerability is due to insufficient CSRF protectionsfor the web UI on an affected device An attacker couldexploit this vulnerability by persuading a user of theinterface to follow a malicious link A successful exploitcould allow the attacker to perform arbitrary actions withthe privilege level of the affected user

Cisco has released software updates that address thisvulnerability There are no workarounds that address thisvulnerability

This advisory is available at the following link

httpstoolsciscocomsecuritycentercontentCiscoSecurityAdvisorycisco-sa-20190807-hypflex-csrf

CSCvp24343

40(2a)35(2d)Add message to HX Connect Progress flow to notAcknowledge Pending Activities and reboot the serverson UCSM

HX Connect is doing a controlled rolling server upgradein the background

CSCvq22844

40(2a)40(1b)Compute Node expand operation fails due to incorrectcluster name

CSCvr43786

40(2a)26(1e)RF-2 cluster shutdown during data resynchronizationafter a node failure (non storfs) followed by multipledisk read failures and a HardBlacklist of a disk

CSCvp41241

40(2a)40(1a)HyperFlex Edge nodes do not properly set the ESXihostname during deployment

CSCvo13143

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4039

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4040

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1b) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401b-33133p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4041

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4042

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1b)40(1a)While running the HyperFlex installer with WindowsServer Hyper-V cluster creation process fails with theerror Failure occurred during ClusterCreation process Unable to post thecontent to the downstream This symptomis encountered while deploying HyperFlex cluster withHyperFlex nodes configuredordered with Cisco VIC1457 MLOM (PID HX-MLOM-C25Q-04) andWindows Server Datacenter or Core with Hyper-Vversion 2016 or 2019

CSCvp64140

Release Notes for Cisco HX Data Platform Release 4043

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)35(2b)Adding Micron 5200 drive to a cluster fails to increasecluster capacity

CSCvo69067

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4044

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4045

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1a) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401a-33028p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4046

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4047

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1a)30(1d)Cluster instability when disks of different sector sizeplaced in HX node

CSCvk17250

40(1a)

35(2c)

35(1a)When logged in using a local HX user account insteadof a Virtual Center account an error message appearsintermittently indicating Virtual Centerunreachable or Resource informationcannot be updated when VC is reachable

CSCvo36198

40(1a)

35(2b)

30(1d)HXDP does not work with EMC RecoverPoint - needsto support VMware API (FSS-Readdir)

CSCvk38003

Release Notes for Cisco HX Data Platform Release 4048

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)

35(2a)

30(1i)

25(1c)

25(1d)

Zookeeper (Exhibitor) process on HyperFlex StorageController VMs may fail to respond to requests whenvarzookeeper has low or no free space

CSCvm90352

40(1a)26(1c)Use Asynchronous consolidation for HX SnapshotsCSCvn02151

40(1a)35(2b)HX Quiesced Snapshot for Backup VendorsCSCvo90713

40(1a)35(2a)The cluster creationcluster expansion workflow stopswith the following error message at the validation step

FIRMWARE-Check UCSC-SAS-M5HD

FIRMWARE-Check UCSC-SAS-M5HD Required00000029000000320000003500000050 Found00000058

Action Needed Update the Controller Firmware toRequired Version

CSCvn17787

35(2b)30(1d)Kernel migration fails when a local datastore is searchedfor in the ESXi host

CSCvn73127

35(2a)26(1b)A node shuts down when two disks are replaced (acaching disk and another capacity disk) where thecapacity disk is inserted first and then the caching diskis inserted

CSCvk46364

Hyper-V

40(1a)35(2a)Cluster expansion may fail with an error code 500 -operation timed out

CSCvn28721

35(2b)35(2a)During upgrade remove the VLAN on team created foruser vSwitch During fresh installation only one VLANtag is set to the vSwitch and team although multipleVLANs were entered

CSCvn54300

35(2b)35(2a)While upgrading a Hyper-V cluster on account of a rarerace condition between the stUpgradeService andZookeeper servers the upgrade orchestration throws anupgrade validation error and the upgrade process isaborted

CSCvn60486

Release Notes for Cisco HX Data Platform Release 4049

Resolved Caveats in Release 40(1a)

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Startrestart EAM service invCenter

When EAM service is not runningvCenter registration is failing

CSCvt55712

35(2a)Contact Cisco TAC if you believeyou are seeing similar conditions

HyperFlex Stretch Cluster saw poorapplication performance due to writelatency cluster remained unhealthy andrebalance was stuck

This defect is currently being used forinvestigative purposes

CSCvt45344

35(2g)Remove log files fromvarlogspringpath folder when thelog files are not getting rotatedStop the relevant service removelarge log files fromvarlogspringpath folder andrestart the service

varstv folder in Controller VM maybecome full

CSCvt36374

40(2b)Restarting the stMgr will initializeit again

In a Cluster configured with DR networkStMgr may not get initialized and stuckin a deadlock while enabling the IPTablerules (can be verified from the stMgrlog)

CSCvt89709

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Click recover for the selectedVM again under the standalonemode This will move the VM torecovered state

When the user recovers a VM protectedin a group it along with other VMs in thegroup move to recovering state instandalonemode The selectedVM shouldmove to recovered state

CSCvs75553

35(2e)Clean up the fail over cluster andthen recreate the fail over clusterNo need to touch the HX storagecluster

Hyper-V When replicated DC was usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2d)40(1a)

Keep cluster utilization as low aspossible To avoid read-only statekeep below 92

HyperFlex clusters storage utilization isabove 92 usable capacity

CSCvr20922

Release Notes for Cisco HX Data Platform Release 4050

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1b)NAAll the disks in the node got locked afterrebooting the node

We successfully unlocked usingsed-clientsh -U command but wantedto test with another reboot and driveslocked again

CSCvs74286

35(2g)To recover the VM copy over thedata disks and attach them to thenew VM

After enable encryption HX connectencryption status showing Caution butall of the disks enable encryptionsuccessful

-Self Encrypting Drives Service isrunning on all of the nodes

-There is no error message of encryption

- USB0 interface is up

-All SED disks showing supported 1enabled 1 locked 0

CSCvs93245

40(1b)Enable kernelpanic_on_oops as apersistent configuration This willcause the node to panic and rebootimmediately

A cluster node running HX release40(1b) may hang in the Linux kernelThis is classified as an oops and adeviation from the expected behavior

CSCvs54285

Release Notes for Cisco HX Data Platform Release 4051

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Enabling K8iscsi stack in Hx clusterCSCvs41324

Release Notes for Cisco HX Data Platform Release 4052

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Steps to enable K8 on HX cluster

Perform following steps on ALLthe controller VMs

1 Updateetcinitscvmclientconf toenable tune iscsiEnable totrue

sed -iesiscsiEnable=falseiscsiEnable=trueetcinitscvmclientconf

2 Run following initctlcommand to reloadconfiguration

initctl reload-configuration

3 Restart scvmclient process

stop scvmclient startscvmclient

4 Run following initctlcommand to mount SYSTEMDS

initctl emit --no-waitsystem-datastore-created

5 Verify that iscsiEnable tuneis set to true using thefollowing command

ps -eaf | grep scvmclient |grep -v grep

root 6241 1 1 Dec06 014007optspringpathstorfs-corescvmclient-T logEnabled=false -TlogSyslogEnabled=true -TlogEchoToScreen=false -TstatLoggingToFile=false -TstatLoggingToSyslog=true-TlogDir=varlogspringpath-TnfsBackendServerList=1010748100-T iscsiEnable=true -TiscsiUseAsync=true -T

Release Notes for Cisco HX Data Platform Release 4053

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

iscsiConfigFilePath=nfsSYSTEMistgtconfps -eaf | grep scvmclient |grep -v grep

35(2e)Following procedure can be usedto check NFS Queue Depth andincrease if needed

rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth256 lt- Low value[rootHXESXI1] vsish -e setvmkModulesnfsclientmnt[DATASTORE]propertiesmaxQDepth 1024[rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth1024 lt- Optimalvalue

This requires an ESXi host rebootto take effect

Please place the node in HyperflexMaintenanceMode and gracefullyreboot the node for the changes tobe applied

HyperFlex Datastore NFS Queue Depthshows as 256 which can lead toperformance (including latency) issues

CSCvr83056

40(1b)This is cosmetic The error will goaway once the cluster returns tohealthy status

When a single node is offline in clusterstcli cluster storage-summary shows twonodes unavailable rootSCVM~ stclicluster storage-summary

CSCvs47735

35(2g)Modify both the root and adminpassword for the SCVMs to be thesame

stcli security password set -uadmin

Cluster expansion fails at Config Installerstage when the root and admin passwordfor the storage controller (SCVM) aredifferent

CSCvs69317

35(2d)NAAfter successfully changing the DNSserver on the HX controller we still cansee the original DNS entry that was addedduring the deployment

CSCvs69154

Release Notes for Cisco HX Data Platform Release 4054

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Customers are strongly encouragedto work with TAC in order toidentify and remediate this issue

1 Change getcluster from 2m to8m inoptspringpathstorfs-mgmtstMgr-10confapplicationconfon all nodes

2 Restart stMgr on all nodes

You may see this error message after afailed upgrade or other task such asattempting to enter a node into HXmaintenance mode

CSCvs53555

40(2a)Run the following command oneach storage controller VM as root

grep -i monitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp sed -iesmonitor_dns_serverstruemonitor_dns_serversfalseoptspringpathhx-diag-toolswatchdog_configjsonampamp grep -imonitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp restart watchdog

Receive the following alertevent in HXConnect HX Controller VMHOSTNAME one or more configuredDNS servers not responding

CSCvt13947

35(2b)Delete any empty (Size 0) log filesunder varlogzookeeperversion-2ORvarlogzookeeperstandaloneversion-2

Zookeeper fails to start while Exhibitor isrunning however echo srvr returnsnothing

CSCvs21562

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

40(4e)1 De-commission the server

2 Power drain the server -REMOVE BOTH POWERCORDS ON THE BACK OFTHE SERVER FOR 10SECONDS THENREINSERTPOWERCORDS

3 Re-commission the server

M2 boot disk is missing from serverinventory after upgrade to server firmware40(4e) As a result server fails to boot toOS installed in the M2 disk The issuepersists after re-acknowledgement as wellas de-commission andre-acknowledgement of the server

CSCvs02466

Release Notes for Cisco HX Data Platform Release 4055

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Re-start the flusher manuallyMany disks taken offline by a fault ormanually resulting in ENOSPC oncluster

CSCvq32721

35(2e)Clean up fail over cluster and thenrecreate the fail over cluster Thereis no need to touch the HX storagecluster

[Hyper-V]When replicated DCwas usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2b)Manually install upgradescvmclient VIB andmake sure thatit matches with HXDP version

Upgrade from release HX 35(2a) to35(2x) 4X does not upgrade scvmclientRecovery point may not work

CSCvq54992

Management

30(1b)To recover the VM copy over thedata disks and attach them to thenew VM

[HyperV]VMshows up onmultiple nodesCSCvj22992

35(1a)1 Do not change MTU at ESXlevel on running cluster

2 Reset back to the original value

ChangingMTU from 9000 to 1500 on oneESX node causes storfs process to restarton all nodes

CSCvm99150

Release Notes for Cisco HX Data Platform Release 4056

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)After the installation (for newdeployments) or after an upgradeto 401a (for existingdeployments) perform thefollowing steps on all ControllerVMs

1 Edit the following tune files (onall Controller VMs)

optspringpathconfiglfftunes

optspringpathconfigvsi_16tbtunes

2 SetcleanerEnableSegSummaryCleaning

to false

3 After editing the above tunefiles

a SSH in to all the ControllerVMs

b Run storfstool -- -Z

c Run the below command andcheck the values Tune valueshould be true

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=true

d Type below commands andapplied the tunes changesdynamically

echo false gttmpstprocfssystemtunecleanerEnableSegSummaryCleaning

e Verify that tune values arechanged Run below commandValue should be false

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=false

f umount tmpstprocfs

A cluster with 8TB or 12TB disk drivesmay experience IO stalls for severalminutes after another node fails in thecluster

CSCvp23718

Release Notes for Cisco HX Data Platform Release 4057

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)1 Update the conf file manuallyto change to the link local addressto

2 Restart scvmclient on thecontroller and ESX

HX 35(2b) FlexVol to HX 40 CSIupgrade does not work FlexVol continuesto work

CSCvp26319

35(2a)Disable rebalance until node healsthen reenable rebalance

7-node cluster healing very slowlybecause rebalance change AMS reqs areblocking healing process

CSCvp62512

35(2a)NAWith moderate cluster storage capacityusage under site failover cluster clustermay experience an all paths (APD) downcondition

CSCvq49412

40(1a)NAThis product includes Third-partySoftware that is affected by thevulnerabilities identified by the followingCommon Vulnerability and Exposures(CVE) IDs

CVE-2016-0762 CVE-2016-6797CVE-2016-6816 CVE-2016-8735CVE-2017-5647 CVE-2017-12615CVE-2017-12616 CVE-2017-12617CVE-2017-7674 CVE-2018-1304CVE-2018-8014 CVE-2018-1336CVE-2018-8034 CVE-2018-11784CVE-2019-0232

CSCvp36364

35(2d)NAstcli cluster info command needs toprovide UCSM VIP address Currently itshows ucsm-hostcom

CSCvq11456

35(2a)No workarounds currently Thestale entries do not affect the dayto day operations of HyperFlex

HyperFlex upgrade validation failedbecause cluster has extraneous stNodes ininternal database

CSCvq32530

35(2a)NAWhen not in maintenancemode all drivesappear in HX Connect and get updated asexpected

When in maintenance mode extra drivesare shown in HX Connect with no slotnumbers

CSCvq42378

35(2b)Contact TAC to cleanup duplicatenode

Nodes duplicated in HX Connect aftercluster recreate

CSCvq61825

Release Notes for Cisco HX Data Platform Release 4058

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)NACurrently HyperFlex installs on Hyper-Vdo not contain the stcli security whitelistcommandset

CSCvq66245

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

Release Notes for Cisco HX Data Platform Release 4059

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)[Hyper-V] Converged node expansionfails for incoming node with VIC 14xx

CSCvq39523

Release Notes for Cisco HX Data Platform Release 4060

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

1 Set Jumbo frame inWindowsHypervisor for incomingnode

Get-NetAdapter storage-data-a|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

Get-NetAdapter storage-data-b|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

2 Verify all interfaces are setcorrectly to support Jumbo frames

Get-NetAdapter storage |Get-NetAdapterAdvancedProperty| RegistryKeyword -matchjumbo | ft -auto

You should receive the followingmessage

Name DisplayName DisplayValue

RegistryKeyword RegistryValue

---- ----------- ------------

--------------- -------------

vswitch-hx-storage-data Jumbo

Packet 9014 Bytes

JumboPacket 9014

storage-data-b Jumbo Packet

Bytes 9014 JumboPacket

9014

storage-data-a Jumbo Packet

Bytes 9014 JumboPacket

9014

3 Reboot only the incoming nodeor the node being expanded

4 Retry Cluster Expansion in

Release Notes for Cisco HX Data Platform Release 4061

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

HX ConnectHX Data PlatformInstaller

Cisco UCS ManagerCisco IntersightBrowser

60 or higher60 or higher45 or higher57 or higherMozilla Firefox

NANA9 or higher10 or higherApple Safari

NANA35 or higherNAOpera

Notes

bull Cisco HyperFlex Connect

The minimum recommended resolution is 1024 X 768

bull Cisco HX Data Platform Plug-in

The Cisco HX Data Platform Plug-in runs in vSphere For VMware Host Client System browserrequirements see the VMware documentation at httpswwwvmwarecomsupportpubs

The HX Data Platform Plug-in is not displayed in the vCenter HTML client You must use the vCenterflash client

bull Cisco UCS Manager

The browser must support the following

bull Java Runtime Environment 16 or later

bull Adobe Flash Player 10 or higher is required for some features

For the latest browser information about Cisco UCSManager refer to themost recent Cisco UCSManagerGetting Started Guide

Cisco HX Data Platform Storage Cluster SpecificationsCluster Limits

bull Cisco HX Data Platform supports up to 100 clusters managed per vCenter as per VMware configurationmaximums

bull Cisco HX Data Platform supports any number of clusters on a single FI domain Each HX convergednode must be directly connected to a dedicated FI port on fabric A and fabric B without the use of a FEXC-series compute only nodes must also directly connect to both FIs B-series compute only nodes willconnect through a chassis IO module to both fabrics In the end the number of physical ports on the FIwill dictate the maximum cluster size and maximum number of individual clusters supported in a UCSdomain

The following table provides Cisco HX Data Platform Storage Cluster Specifications

Release Notes for Cisco HX Data Platform Release 4016

Cisco HX Data Platform Storage Cluster Specifications

Stretched Cluster (Availableon ESX Only)

Microsoft Hyper-VVMware ESXiNode

AllNVMe -HXAF220c-M5SN

HX240c-M5LHX220cM5

HX220cAF M5

HX240cM5

HX240cAF M5

HX240c-M5LHX220cM5

HX220cAF M5

HX240cM5

HX240cAF M5

HXAF220c-M5SN

AllNVMe -HXAF220c-M5SN

ESXi 65or 67only NotsupportedwithESXi 60orHyper-V

HX220cM5 Edge

HXAF220cM5 Edge

HX220cM4 Edge

HXAF220cM4 Edge

HX240c-M5LHX220cM5

HX220cAF M5

HX240cM5

HX240cAF M5

HX220cM4

HX220cAF M4

HX240cM4

HX240cAF M4

HXServers

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

C220M5C240M5B200M4B200M5

C240M5C220M5B200M4B200M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

mdashB200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

Compute-OnlyUCSB-SeriesC-SeriesServers

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

Convergednodesonly

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

SupportedNodes

Release Notes for Cisco HX Data Platform Release 4017

Cisco HX Data Platform Storage Cluster Specifications

Stretched Cluster (Availableon ESX Only)

Microsoft Hyper-VVMware ESXiNode

NA(requiresEnterpriseHXDP-PLicense)

NA(requiresEnterpriseHXDP-PLicense)

NA(requiresEnterpriseHXDP-PLicense)

Convergednodes3-16(12TBHDDoption isnotsupportedforHyperV)

Compute-onlynodes0-16

Convergednodes3-16

Compute-onlynodes0-16

NA(requiresEnterpriseHXDP-PLicense)

M4Convergednodes 3

M5Convergednodes23or 4

RequiresHXDP-ELicense

Convergednodes3-16(12TBdrive islimited toamaximumof 8nodes)

Compute-onlynodes0-16(12TBdrive islimited toamaximumof 8nodes)

Convergednodes3-32(76TBdriveconfigsarelimited tomaximumof 12drivesnodeon AF240)

Compute-onlynodes0-32

HXDP-SLicensedNodeLimits

11 ratioofHXDP-StoComputeonlynodes

(MinmdashMax)

Convergednodes2-16 perSite

Compute-onlynodes0-21 perSite

(up tomaxclustersize)

Convergednodes2-8 perSite

Compute-onlynodes0-16 perSite

(up tomaxclustersize)

Convergednodes2-16 perSite

Compute-onlynodes0-21 perSite

(up tomaxclustersize)

Convergednodes3-16(12TBHDDoption isnotsupportedforHyperV)

Compute-onlynodes0-16

Convergednodes3-16

Compute-onlynodes0-16

Convergednodes3-32

ComputeOnlynodes0-32

(up tomaxclustersize)

Convergednodes 3

(requiresHXDP-ELicense)

Convergednodes3-16(12TBdrive islimited toamaximumof 8nodes)

Compute-onlynodes0-32(12TBdrive islimited toamaximumof 16nodes)

Convergednodes3-32(76TBdriveconfigsarelimited tomaximumof 12drivesnodeon AF240)

Compute-onlynodes0-32

(up tomaxclustersize)

HXDP-PLicensedNodeLimits

12 ratioofHXDP-PtoComputeonlynodes

(MinmdashMax)

24 perSite 64percluster

24 perSite 48percluster

32 perSite 64percluster

32326434864MaxClusterSize

Release Notes for Cisco HX Data Platform Release 4018

Cisco HX Data Platform Storage Cluster Specifications

Stretched Cluster (Availableon ESX Only)

Microsoft Hyper-VVMware ESXiNode

212121111121mdash2121MaxComputetoConvergedratio

NoExpansion

Requires Enterprise license

Requires uniform expansion across both sites

Guidelines and Limitationsbull Starting with release 40(2a) SCVM is no longer needed on a Compute node

bull HX REST API Access Token Management ndash Applications leveraging HX REST APIs should re-useaccess tokens when making API calls Once obtained using the AAA Obtain Access Token API accesstokens are valid for 18 days (1555200 seconds) In addition AAA enforces rate limiting on ObtainAccess Token API requests in a 15 minute window auth can be invoked (successfully) a maximum of5 times A user is allowed to create a maximum of 8 unrevoked tokens Subsequent call to auth willautomatically revoke the oldest issued token tomake room for the new token Amaximum of 16 unrevokedtokens can be present in system In order to prevent brute-force attacks after 10 consecutive failedauthentication attempts a user account is locked for a period of 120 secondsFor more information seeCisco HyperFlex Systems REST API Reference guide

HxConnect makes use of AAA Authentication REST API for login and the above rate limit applies toHxConnect also

Upgrade Guidelines

The following list is a highlight of critical criteria for performing an upgrade of your HyperFlex system

bull Unsupported HX Data Platform 17x 18x 20 21x 25x and 26x clustersmdashUsers from anyversion prior to 26(1a) must step through an intermediate version before upgrading to 40 or later releasesIf you need to upgrade your environment from a Cisco HyperFlex HX Data Platform software releasethat is past the last date of support to the latest suggested release on the Cisco Software Download sitesee Cisco HyperFlex Systems Upgrade Guide for Unsupported Cisco HXReleases For more informationsee the Software Advisory for CSCvq66867 WARNING Only Use HXDP 26(1e) Upgrade PackageWhen Upgrading From HXDP 18(1a)-18(1e)

bull Hypercheck Health Check UtilitymdashCisco recommends running this proactive health check utility onyour HyperFlex cluster prior to upgrade These checks provide early visibility into any areas that mayneed attention andwill help ensure a seamless upgrade experience For more information see the HyperFlexHealth amp Pre-Upgrade Check Tool TechNote for full instructions on how to install and run Hypercheck

bull vSphere 67 Software AdvisorymdashDo not upgrade to Cisco HX Data Platform Release 40(1a) whenrunning ESXi 67U1 EP06 (build 11675023) Do not upgrade to 67U1 EP06 (build 11675023) ifrunning Cisco HX Data Platform Release 40(1a) See the Software Advisory CSCvo56350 for furtherdetails

Release Notes for Cisco HX Data Platform Release 4019

Guidelines and Limitations

The software build version posted at release will override any other local versions

bull Required vCenter upgrademdashFor enhanced security Cisco HX Data Platform Release 35(1a) or laterrequires the use of TLS 12 Therefore vCenter must be upgraded to 60 U3f or later before upgradingto Cisco HX Data Platform Release 35 or later In addition ESXi should be upgraded as required tomeet HX Data Platform compatibility requirements

bull Minimum HXDP version for upgrademdashHXData Platform clusters running 26(1a) or later may upgradedirectly to 40 using the HX Connect UI

bull Cluster ReadinessmdashEnsure that the cluster is properly bootstrapped and the updated plug-in is loadedbefore proceeding Manual cluster bootstrap is required for upgrade from a pre-35 release

bull Cluster ReadinessmdashEnsure that the cluster is properly bootstrapped and the updated plug-in is loadedbefore proceeding Manual cluster bootstrap is required for HX releases earlier than 35(1a) For moreinformation see theManual Bootstrap Upgrade Process in the Cisco HyperFlex Systems Upgrade Guidefor VMware ESXi Release 40 Do not skip this cluster bootstrap step it is required for all upgradesuntil HX Release 35(1a) Auto bootstrap is supported beginning with HX release 35(1a) For moreinformation see the Auto Bootstrap Upgrade Process from HX Connect UI in the Cisco HyperFlexSystems Upgrade Guide for VMware ESXi Release 40

Manual bootstrap is not supported on Intersight clusters

bull Initiating Upgrade―Use the HX Connect UI or CLI stcli commands when upgrading from 25(1a)or later releases Use either the CLI stcli commands or the HX Data Platform Plug-in to the vSphereWeb Client when upgrading from a pre-25(1a) release The vCenter plug-in should not be used forupgrades starting with the 25(1a) release The vCenter plug-in should not be used for upgrades startingwith the 25(1a) release

If the current cluster version is at 35(1a) or above you do not need to use the stcli command Directupgrade to 40 is possible

bull Complete your Upgrade―The self-healing (or rebalance) capability is disabled temporarily during theupgrade window If the upgrade fails you should complete the upgrade as soon as possible

bull ESXi and HXDP Compatibility―Ensure your cluster is running a compatible version of ESXi basedon the running the HXData Platform version (see the section Software Requirements for VMware ESXi)ESXi compatibility is determined by the major version and update release of ESXi It is generally bestto upgrade HXDP and ESXi together if combining the upgrade operations into a single optimized rebootWhen running a split upgrade first upgrade the HX Data Platform then proceed to upgrade ESXi

bull If Upgrading to vSphere 65

bull Certain cluster functions such as native and scheduled snapshotsReadyClones and Enter or Exit HX Maintenance Mode will not operatefrom the time the upgrade is started until the HX Data Platform upgrade iscomplete

bull After upgrading ESXi using the offline zip bundle use the ESX ExitMaintenance Mode option The HX Exit Maintenance Mode option doesnot operate in the vSphere Web Client until the HX Data Platform upgradeis complete

Note

Release Notes for Cisco HX Data Platform Release 4020

Guidelines and Limitations

bull vSphere 60 UpgradesmdashUsers on vSphere 60 migrating to 65 upgrade components in the followingorder

1 Upgrade HX Data Platform and UCS firmware

2 Upgrade HX Data Platform and ESXi

3 Upgrade HX Data Platform only first then upgrade ESXi or UCS firmware or both

bull M4 Server Firmware UpgradesmdashUpgrade server firmware to ensure smooth operation and to correctknown issues Specifically newer SAS HBA firmware is available in this release and is recommendedfor long-term stability

bull Users are encouraged to upgrade to 31(3c) C-bundle or later whenever possible

bull Users running C-bundle versions before 31(2f) must upgrade server firmware by performing acombined upgrade of UCS server firmware (C-bundle) to 31(3c) or later and HX Data Platform to25 Do not split the upgrade into two separate operations

bull If the cluster is already on 31(2f) C-bundle or later you may perform an HX Data Platform onlyor combined upgrade as required

bull M5 Server Firmware UpgradesmdashM5 generation servers must run firmware version 32(2d) or later

bull Firmware Downgrades mdashDowngrading UCSM from the HX-installer is not supported

bull M4M5 Mixed DomainsmdashA mixed domain occurs when a new separate M5 cluster is installed underthe same UCS domain that contains existing M4 clusters Under these conditions orchestrated UCSserver firmware upgrade will not operate until Cisco HX Data Platform Release 26 or later is installedon the M4 clusters Therefore it is best practice to first upgrade UCS server firmware to the latest 31(3)or 32(2) patch release before adding a new M5 cluster to the existing UCS domain Additionally any17 HX Data Platform clusters must first be upgraded before adding any new M5 clusters to the samedomain

bull Maintenance WindowmdashIf upgrading both HX Data Platform and UCS firmware you can select eithera combined or split upgrade through the vSphere HX Data Platform Plug-in depending on the length ofthe maintenance window Cisco UCSManager infrastructure upgrade is only supported using AutoInstalland the direct server firmware upgrade should be performed only through the upgrade orchestrationframework provided by the HX Data Platform Plug-in

bull Unsupported Self-Encrypting Drives (SEDs)mdashIf adding or replacing self-encrypting drives (SEDs)that have been recently qualified in newer versions of HX Data Platform insert the new drives only afterupgrading HX Data Platform to a compatible version All drives must be SED drives mixing SED andnon-SED is not supported

bullbull Enabling External Host AccessmdashWith Cisco HX Data Platform Release 40(1a) port 445 on themanagement network is blocked for enhanced security Note that prior to 40 port 445 port was openenabling external host access If you are upgrading to 40(1a) from a prior release and would like tocontinue external host access you can use a utility to open select hosts For more information aboutenabling external host access see the ConfiguringHyperFlex Share to SCVMM section in the InstallationGuide for Microsoft Hyper-V

Release Notes for Cisco HX Data Platform Release 4021

Guidelines and Limitations

Mixed Cluster Expansion Guidelinesbull Expanding existing M4 cluster with M5 converged nodes is supported

bull Expanding existing M5 cluster with M4 converged nodes is not supported

bull Expanding existing mixed M4M5 cluster with M4 or M5 converged nodes is supported

bull Adding any supported compute-only nodes is permitted with all M4 M5 and mixed M4M5 clustersusing the HX Data Platform 26 or later Installer Some example combinations are listed here manyother combinations are possibleExample combinationsExpand mixed M4M5 cluster with compute-only B200 C220 C240 M4M5Expand M4 cluster with compute-only B200 M5 C220 M5 C240M5

bull Only expansion workflow is supported to create a mixed cluster Initial cluster creation with mixedM4M5 servers is not supported

bull All M5 servers must match the form factor (220240) type (HybridAF) security capability (Non-SEDonly) amp disk configuration (QTY capacity and non-SED) of the existing M4 servers

bull HX220-M5 will use a maximum of 6 capacity disks (2 disk slots to remain empty) when mixedwith HX220-M4

bull HX Edge SED LFF Hyper-V and Stretched Clusters do not support mixed M4 and M5 clusters

Security FixesThe following security issues are resolved

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-13132 CVE-2019-9924CSCvq6313840(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-11719 CVE-2019-11727CVE-2019-11729

CSCvq7124040(2a)

Release Notes for Cisco HX Data Platform Release 4022

Mixed Cluster Expansion Guidelines

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-1125CSCvr0633940(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2015-9383 CVE-2018-14498CVE-2018-20406 CVE-2018-20852CVE-2019-10160 CVE-2019-13117CVE-2019-13118 CVE-2019-14287CVE-2019-14973 CVE-2019-15903CVE-2019-17546 CVE-2019-18197CVE-2019-18218 CVE-2019-5010CVE-2019-5094 CVE-2019-5481CVE-2019-5482 CVE-2019-9636CVE-2019-9740 CVE-2019-9947CVE-2019-9948

CSCvs0609440(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2017-13168 CVE-2017-18174CVE-2017-18216 CVE-2018-10876CVE-2018-10877 CVE-2018-10878CVE-2018-10879 CVE-2018-10880CVE-2018-10881 CVE-2018-10882CVE-2018-10902 CVE-2018-10938CVE-2018-12233 CVE-2018-12896CVE-2018-13053 CVE-2018-13094CVE-2018-13096 CVE-2018-13405CVE-2018-13406 CVE-2018-14609CVE-2018-14617 CVE-2018-14633CVE-2018-14734 CVE-2018-15572CVE-2018-15594 CVE-2018-16276CVE-2018-16658 CVE-2018-17182CVE-2018-17972 CVE-2018-18021CVE-2018-18690 CVE-2018-18710CVE-2018-6554 CVE-2018-6555CVE-2018-9363

CSCvp6501940(2a)

This vulnerability is due to insufficientHTML iframe protection An attackercould exploit this vulnerability bydirecting a user to an attacker-controlledweb page that contains amaliciousHTMLiframe A successful exploit could allowthe attacker to conduct clickjacking orother clientside browser attacks

NACSCvo9851640(2a)

Release Notes for Cisco HX Data Platform Release 4023

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerability is due to insufficientauthentication for the statistics collectionservice An attacker could exploit thisvulnerability by sending properlyformatted data values to the statisticscollection service of an affected deviceA successful exploit could allow theattacker to cause the web interfacestatistics view to present invalid data tousers

NACSCvj9558440(2a)

The vulnerability is due to insufficientCSRF protections for the web UI on anaffected device An attacker could exploitthis vulnerability by persuading a user ofthe interface to follow a malicious linkA successful exploit could allow theattacker to perform arbitrary actions withthe privilege level of the affected user

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvp2434340(2a)

Vulnerabilities with a version of cJSONidentified by CVE-2019-11834 andCVE-2019-11835

For more information see the relatedCisco Security Advisory

CVE-2019-11834 CVE-2019-11835CSCvq1994940(2a)

HX ESXi image patches for MCEPSCand TAA vulnerabilities with VMwareESXi identified by CVE-2018-12207 andCVE-20190-11135

For more information see the relatedCisco Security Advisory

CVE-2018-12207 CVE-2019-11135CSCvr5439840(2a)

CP networking vulnerabilities affectingthe Linux kernel identified byCVE-2019-11477 CVE-2019-11478CVE-2019-11479

For more information see the relatedCisco Security Advisory

CVE-2019-11477 CVE-2019-11478CVE-2019-11479

CSCvq1954640(2a)

Qualification of Microsoft securitypatches for vulnerabilities withMicrosoftHyper-V hypervisor identified withCVE-2018-12207 CVE-2019-11135

For more information see the relatedCisco Security Advisory

CVE-2018-12207 CVE-2019-11135CSCvr5439940(2a)

Release Notes for Cisco HX Data Platform Release 4024

Security Fixes

DescriptionCVEDefect IDRelease

OpenSSH vulnerabilities identified byCVE-2019-6111

For more information see the relatedCisco Security Advisory

CVE-2016-10708 CVE-2018-15473CVE-2018-20685 CVE-2019-6109CVE-2019-6111

CSCvp7646340(2a)

Multiple vulnerabilities associated withOpenSSL and LibNSS

For more information see the relatedCisco Security Advisory

CVE-2016-2105 CVE-2016-2106CVE-2016-2107 CVE-2016-2109CVE-2016-2176 CVE-2016-2177CVE-2016-2178 CVE-2016-2179CVE-2016-2180 CVE-2016-2181CVE-2016-2182 CVE-2016-2183CVE-2016-6302 CVE-2016-6303CVE-2016-6304 CVE-2016-6305CVE-2016-6306 CVE-2016-6307CVE-2016-6308 CVE-2016-7055CVE-2016-8610 CVE-2017-3731CVE-2017-3732 CVE-2017-3735CVE-2017-3736 CVE-2017-3737CVE-2017-3738 CVE-2018-0495CVE-2018-0732 CVE-2018-0734CVE-2018-0735 CVE-2018-0737CVE-2018-0739 CVE-2018-12384CVE-2018-12404 CVE-2018-5407CVE-2019-1559

CSCvp6667940(2a)

libpng vulnerability identified byCVE-2016-10087

For more information see the relatedCisco Security Advisory

CVE-2016-10087CSCvp6655540(2a)

Vulnerabilities associated with libjpg andlibpng

For more information see the relatedCisco Security Advisory

CVE-2014-9092 CVE-2016-3616CVE-2017-15232 CVE-2018-11212CVE-2018-11213 CVE-2018-11214CVE-2018-1152 CVE-2018-13785

CSCvp3458640(2a)

Vulnerabilities associated with Glibidentified by CVE-2018-16428 andCVE-2018-16429

For more information see the relatedCisco Security Advisory

CVE-2018-16428 CVE-2018-16429CSCvp3120740(2a)

Multiple vulnerabilities associated withcurl expat python 27 python 35 36and 37 freetype and giflib

For more information see the relatedCisco Security Advisory

CVE-2019-15133 CVE-2019-15903CVE-2019-5010 CVE-2019-5481CVE-2019-5482 CVE-2019-9636CVE-2019-9740 CVE-2019-9947CVE-2019-9948

CSCvr3690340(2a)

Release Notes for Cisco HX Data Platform Release 4025

Security Fixes

DescriptionCVEDefect IDRelease

Multiple third party softwarevulnerabilities

For more information see the relatedCisco Security Advisory

CVE-2019-14379 CVE-2019-12384CVE-2019-14439

CSCvq9203240(2a)

Vulnerabilities associated with elfutils

For more information see the relatedCisco Security Advisory

CVE-2018-16062 CVE-2018-16402CVE-2018-16403 CVE-2018-18310CVE-2018-18520 CVE-2018-18521CVE-2019-7149 CVE-2019-7150CVE-2019-7665

CSCvq4325040(2a)

Vulnerabilities associated with vim

For more information see the relatedCisco Security Advisory

CVE-2017-5953 CVE-2019-12735CSCvq4323040(2a)

Vulnerabilities associated with sqlite3

For more information see the relatedCisco Security Advisory

CVE-2016-6153 CVE-2017-10989CVE-2017-13685 CVE-2017-2518CVE-2017-2519 CVE-2017-2520CVE-2018-20346 CVE-2018-20505CVE-2018-20506 CVE-2019-8457CVE-2019-9936 CVE-2019-9937

CSCvq4321340(2a)

Vulnerabilities associated with glib20

For more information see the relatedCisco Security Advisory

CVE-2018-20843CSCvq4320940(2a)

Vulnerabilities associated with expat

For more information see the relatedCisco Security Advisory

CVE-2018-20843CSCvq4320540(2a)

Vulnerabilities associated with bzip2

For more information see the relatedCisco Security Advisory

CVE-2016-3189 CVE-2019-12900CSCvq4319440(2a)

Mulitple Vulnerabilities associated wtihNodeJS

For more information see the relatedCisco Security Advisory

CVE-2018-12115 CVE-2018-0734CVE-2018-5407 CVE-2018-12120CVE-2018-12121 CVE-2018-12122CVE-2018-12123 CVE-2018-12116CVE-2019-5737 CVE-2019-5739

CSCvq1069440(2a)

Vulnerabilities associated with PalletsJinja strformat_map and identified byCVE-2019-10906 and CVE-2016-10745

For more information see the relatedCisco Security Advisory

CVE-2019-10906 CVE-2016-10745CSCvq1038840(2a)

Release Notes for Cisco HX Data Platform Release 4026

Security Fixes

DescriptionCVEDefect IDRelease

Multiple Security Vulnerabilities

For more information see the relatedCisco Security Advisory

CVE-2014-8501 CVE-2014-9939CVE-2015-9262 CVE-2016-10087CVE-2016-2226 CVE-2016-4487CVE-2016-4488 CVE-2016-4489CVE-2016-4490 CVE-2016-4491CVE-2016-4492 CVE-2016-4493CVE-2016-6131 CVE-2018-10963CVE-2018-13785 CVE-2018-17100CVE-2018-17101 CVE-2018-18557CVE-2018-18661 CVE-2018-7456CVE-2018-8905

CSCvq0917840(2a)

Vulnerabilities associated withlibseccomp

For more information see the relatedCisco Security Advisory

CVE-2019-9893CSCvq0756840(2a)

Vulnerabilities associated withgdk-pixbuf

For more information see the relatedCisco Security Advisory

CVE-2017-12447CSCvq0675540(2a)

Vulnerabilities associated withpython-crypto

For more information see the relatedCisco Security Advisory

CVE-2018-6594CSCvp9381740(2a)

Vulnerabilities associated with wget

For more information see the relatedCisco Security Advisory

CVE-2018-20483 CVE-2019-5953CSCvp8672140(2a)

Multiple vulnerabilities associated withPython Crypto

For more information see the relatedCisco Security Advisory

CVE-2018-6594CSCvp6674840(2a)

Vulnerabilities associated withsensible-utils

For more information see the relatedCisco Security Advisory

CVE-2017-17512CSCvp6673440(2a)

Multiple vulnerabilities associated withPython

For more information see the relatedCisco Security Advisory

CVE-2018-1000030CSCvp6668940(2a)

Release Notes for Cisco HX Data Platform Release 4027

Security Fixes

DescriptionCVEDefect IDRelease

Multiple patch vulnerabilities identifiedby CVE-2016-10713

For more information see the relatedCisco Security Advisory

CVE-2016-10713CVE-2018-1000156 CVE-2018-6951

CSCvp6667240(2a)

Vulnerabilities associatedwith Little CMSidentified by CVE-2016-10165

For more information see the relatedCisco Security Advisory

CVE-2016-10165 CVE-2018-16435CSCvp6666440(2a)

Vulnerabilities associated with TenableScan identified byCVE-2011-5325CVE-2019-11068CVE-2016-7076CVE-2017-1000368

For more information see the relatedCisco Security Advisory

CVE-2011-5325 CVE-2016-7076CVE-2017-1000368CVE-2019-11068

CSCvp6474640(2a)

Vulnerabilities associated with libxml2294 and earlier as used in XMLSec1223 and earlier and other products andidentified by CVE-2016-9318

For more information see the relatedCisco Security Advisory

CVE-2016-9318 CVE-2017-16932CVE-2017-18258 CVE-2018-14404CVE-2018-14567

CSCvp3479240(2a)

Vulnerabilities associated with LFTPRemote File Names Unauthorized AccessVulnerability and identified byCVE-2018-10916

For more information see the relatedCisco Security Advisory

CVE-2018-10916CSCvp2926640(2a)

Vulnerabilities associated withCryptographyDeprecationWarning signerand verifier have been deprecated

For more information see the relatedCisco Security Advisory

CVE-2018-7750CSCvo3409740(2a)

Multiple Vulnerabilities in python python27 35

For more information see the relatedCisco Security Advisory

CVE-2019-16056CSCvr3979340(2a)

Release Notes for Cisco HX Data Platform Release 4028

Security Fixes

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toexecute commands as the root user

The vulnerability is due to insufficientinput validation An attacker could exploitthis vulnerability by connecting to thecluster service manager and injectingcommands into the bound process Asuccessful exploit could allow the attackerto run commands on the affected host asthe root user

Cisco has released software updates thataddress this vulnerability There areworkarounds that address thisvulnerability

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvq2417640(1b)35(2g)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

The vulnerability is due to an unprotectedlistening interface An attacker couldexploit this vulnerability by connectingto the listening interface and injectingcommands to the bound process Anexploit could allow the attacker to runcommands on the affected host as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvj9560640(1b)35(2g)

Release Notes for Cisco HX Data Platform Release 4029

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerabilities associated with theJVM18U121memory leak found duringVC alarms (concurrent 40 calls usingREST API)

CVE-2017-10053 CVE-2017-10067CVE-2017-10074 CVE-2017-10078CVE-2017-10081 CVE-2017-10087CVE-2017-10089 CVE-2017-10090CVE-2017-10096 CVE-2017-10101CVE-2017-10102 CVE-2017-10107CVE-2017-10108 CVE-2017-10109CVE-2017-10110 CVE-2017-10111CVE-2017-10115 CVE-2017-10116CVE-2017-10118 CVE-2017-10135CVE-2017-10176 CVE-2017-10193CVE-2017-10198 CVE-2017-10243CVE-2017-10274 CVE-2017-10281CVE-2017-10285 CVE-2017-10295CVE-2017-10345 CVE-2017-10346CVE-2017-10347 CVE-2017-10348CVE-2017-10349 CVE-2017-10350CVE-2017-10355 CVE-2017-10356CVE-2017-10357 CVE-2017-10388CVE-2017-3509 CVE-2017-3511CVE-2017-3526 CVE-2017-3533CVE-2017-3539 CVE-2017-3544CVE-2018-2579 CVE-2018-2582CVE-2018-2588 CVE-2018-2599CVE-2018-2602 CVE-2018-2603CVE-2018-2618 CVE-2018-2629CVE-2018-2633 CVE-2018-2634CVE-2018-2637 CVE-2018-2641CVE-2018-2663 CVE-2018-2677CVE-2018-2678 CVE-2018-2783CVE-2018-2790 CVE-2018-2794CVE-2018-2795 CVE-2018-2796CVE-2018-2797 CVE-2018-2798CVE-2018-2799 CVE-2018-2800CVE-2018-2814 CVE-2018-2815CVE-2018-2952 CVE-2018-3136CVE-2018-3139 CVE-2018-3149CVE-2018-3150 CVE-2018-3169CVE-2018-3180 CVE-2018-3183CVE-2018-3214 CVE-2019-2422

CSCvo8899740(1b)

Tomcat and Nginx logs are not beingcollected in the support bundle generatedthrough HX Connect in Release 35

NACSCvm5803140(1b)

The vulnerabilities associated with thelibmspack software package versionincluded in Cisco HX Data Platform

CVE-2018-18584

CVE-2018-18585

CSCvn3511940(1a)

Release Notes for Cisco HX Data Platform Release 4030

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerabilities associated withFasterXML Jackson-Databind TimeValueField Denial of Service

CVE-2018-14719

CVE-2018-14720

CVE-2018-1000873

CVE-2018-14721

CVE-2018-19360

CVE-2018-19362

CVE-2018-19361

CVE-2018-14718

CSCvn8228240(1a)

The vulnerabilities associated with theOpenSSL Protocol software packageversion included in Cisco HX DataPlatform

CVE-2013-3587CSCvo0505440(1a)

The vulnerabilities associated withThird-Party Software Denial of Service

CVE-2018-16487

CVE-2018-19361

CSCvo2781840(1a)

The vulnerabilities associated withLinuxkernel for Ubuntu 1710

CVE-2018-1092

CVE-2018-7492

CVE-2018-8087

CVE-2018-1068

CVE-2018-8781

CSCvm5314935(2a)

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)25(1d)stcli services dns remove should remove the DNS serverinfo from the interface files

CSCvs70967

40(2b)40(1b)HX 40(1b) new deployment from Intersight productioncloud

User receives following error while enabling CSI(Kubernetes integration)

Failure occurred during volume_access_enable Errorwas Nonzero exit code 1

CSCvt10522

40(2b)40(2a)Receive the following alertevent in HX Connect HXController VM HOSTNAME one or more configuredDNS servers not responding

CSCvt13947

Release Notes for Cisco HX Data Platform Release 4031

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)40(2a)API UI CLI show few Drive slots Empty after upgradeto 402a

The drives are seen in the sysmtool lsscsi and stclicommand outputs

The cluster is healthy with no errors

CSCvt14914

40(2b)35(2d)After successfully changing the DNS server on the HXcontroller we still can see the original DNS entry thatwas added during the deployment

CSCvs69154

40(2b)35(2e)HyperFlex Connect and VCenter show APD alarms forHX and non HX datastores This implies there is an issuewith HyperFlex when there may not be

CSCvs30080

40(2b)40(2a)An issue where all the disks in the node get locked afterrebooting the node

We successfully unlocked using sed-clientsh -Ucommand but wanted to test with another reboot anddrives locked again

CSCvs74286

40(2b)40(2a)When running stcli license commands on HyperFlexthe following error is seen

rootSpringpathControllertmp stcli license show all

Show smart licensing failed

Smart Agent is not ready please wait a minute and tryagain

CSCvt13929

40(2b)40(2a)After upgrading to HX 40(2a) the first RoboEdge nodethat gets rebooted then ignores persistent drivesPIDHX-HD24TB10K4KN

Cluster will be in degraded state and have reducedcapacity

CSCvt20144

40(2b)35(2b)Zookeeper fails to start while Exhibitor is runninghowever echo srvr returns nothing

CSCvs21562

40(2b)40(2a)When performing an HyperFlex upgrade a validationwarning may occur due to the host not having EnterprisePlus or Enterprise hypervisor licensing

Upgrade Validation Warning

ESXi host esx1labtest should be configured withVMware Enterprise license for upgrade to continue

CSCvs91787

Release Notes for Cisco HX Data Platform Release 4032

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)35(2g)Cluster expansion fails at Config Installer stage whenthe root and admin password for the storage controller(SCVM) are different

CSCvs69317

40(2b)35(2g)Panic while upgrading ESXiCSCvt06983

40(2b)40(2a)A cluster node running may hang in the Linux kernelThis is classified as an oops and a deviation from theexpected behavior

CSCvs54285

40(2b)35(2g)Rebalance failing on 10+10 node HX 35(2g) stretchcluster

CSCvs69007

40(2b)35(2d)The cluster becomes inaccessible to the IOVisorCSCvr54687

40(2b)40(2a)Panic on the storage controllerCSCvt61297

40(2b)40(2a)The size of support bundle is very large when collectedwith storfs-support command

CSCvt63306

40(2b)40(2a)storfs crash associated withFileSystemUsageWarningEvent orFileSystemUsageAlertEvent due to high usage ofvarstv above 80

CSCvt72807

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

40(2a)35(2d)In normal scenarios HX upgrade gets pendingACK afterupdating service profile In this case UCS only upgradewas stalled because after updating service profile HXupgrade was not getting pending ACK and it was alwayswaiting state

CSCvo39912

40(2a)35(1a)When using motherBoardReplace-12 to clean ZK fromold stNodepNodes resulted in unmounted datastore andmaking the size of the Hyperflex Datastores 0 resultingin all VMs in the cluster going offline

CSCvq39471

40(2a)26(1e)Upgrading cluster and getting error

Failed upgrade validations Checking vCenterconfiguration Reason Upgrade validations failed DRSFault Insufficient resources to satisfy configured failover

CSCvm77294

Release Notes for Cisco HX Data Platform Release 4033

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)The cluster expand fails on a node with DR replicationconfigured

When a HX cluster which has DR replication configuredis expanded we see the installer UI pulling in thereplicationVLAN information instead of themanagementVLAN information Even if we change that informationto the correct mgmt VLAN id and name it it does notseem to work as the node is configured with the VLANof the replicationVLAN in ESXi This leads to the failureof the node add with the host unreachable error

CSCvo70650

40(2a)35(1a)Customer reported that server didnt complete thefirmware upgrade automatically one by one

The user manually needs to put the host into maintenancemode and they manually acknowledged the pendingrequests to finish the UCS firmware upgrade

Maintenance policy is set to default (User-ack) as perthe design

CSCvo91624

40(2a)35(1i)When the cluster is in Failed state and stcli noderemove is attempted the output shows successful thoughthe node failed to get removed from the cluster

CSCvo93017

40(2a)35(1a)HyperFlex cluster upgrade may fail during validationwith a DRS Validation failed error

CSCvp31021

40(2a)35(2b)HX cluster expansion will fail with the following errormessage

MAC address pool configuration failure150[ErrorDescription] bad address block range definitioncollision

CSCvp58318

40(2a)35(2b)Memory usage by carbon cacheCSCvq34873

40(2a)35(2g)Upgrading ESXi cluster fails with error Nodemaintenance mode failed

CSCvr03240

40(2a)35(2e)storfs process does not automatically start on Exit HXMaintenance Mode or other tasks which restart StorageController

CSCvr88978

40(2a)35(2a)Post install script fails with message

Failed to execute ipmitool on HX node

CSCvp10707

40(2a)40(1a)HyperFlex expansionworkflow doesnt pull VLANnamecorrectly

CSCvp46539

Release Notes for Cisco HX Data Platform Release 4034

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)During HX cluster deployment - Validate ClusterCreation phase - HX Installer might fail with the error

fromvarlogspringpathstDeploylog

CSCvq45087

40(2a)35(2b)HX Installer fails to login to the SOL for configuringESX hosts

CSCvq91380

40(2a)35(2d)If DRS is enabled with configuration parameters otherthen fully automated ie manual or partially automatedthan this will change to fully automated during clusterexpansion

CSCvr44222

40(2a)35(1a)One of the validation task name has a typo The taskname is Validating node for mixed of SED and non-SEDdisks

It should be Validating node for mix of SED andnon-SED disks

CSCvp82175

40(2a)30(1i)This combination of using smaller NVMe drives is notsupported in expand operation For example if the drivesare only 375GB and they cannot be added to an existingcluster with larger caching SSD

CSCvo12359

40(2a)35(1a)HyperFlex includes a version of OpenSSL that is affectedby the vulnerabilities identified by the followingCommon Vulnerability and Exposures

CVE-2018-0495

CSCvp66679

40(2a)30(1e)SDK invoke exception when taking a native snapshot ona VM with GPU pci passthrough attached

CSCvo00511

40(2a)30(1i)Node replacement script fails due to EAM errorCSCvo62867

40(2a)35(1a)While pairing a cluster (HX release gt= 35) and cluster(HX release lt 35) remote replication network test failson the cluster (HX release lt 35)

CSCvo79760

40(2a)30(1b)Fixed the issue in the latest support-workflow bundleThis should not be hit anymore

CSCvo87061

40(2a)35(2a)On a 3 node cluster the MBreplace script fails as thescript looks for HEALTHY cluster status

CSCvo87080

40(2a)35(2a)MbReplace script (tar) hangs when running on 352b or40

CSCvp12359

Release Notes for Cisco HX Data Platform Release 4035

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)HX replication cleanup failing with error NA 35(2a) -INFODR state is not clean

CSCvp63958

40(2a)40(1a)The check_stig_parameters API incorrectly shows thecompliance state of the cluster as not STIG compliant

CSCvp66277

40(2a)35(2c)The snapshot creation on CBT enabled VM fails witherror Failed in vmreparent vmkfstools clone1

CSCvq06952

40(2a)35(2a)HyperFlex GUI when setting up remote DR-replicationpartnerpeer should show remote replication partner TCPport reachability test results

This will provide quick upfront connectivity results thatfrequently takes customer a while to manually check andconfirm

If reachability tests fail an info window and HX alertshould trigger that list the destination peer IP and thespecific port(s) that could not be reache

CSCvo60587

40(2a)35(2b)If you have more than one IP pool configured for areplication network and local replication network testsare failing you might be running into this issue

CSCvr67130

40(2a)35(2a)Skip task option in MBreplace and Software redeployscripts The scripts should have the option to skip tasksas user inputs and not by modifying the json file

CSCvp05204

40(2a)40(1a)Error noticed on SCVM Console print_req_error IOerror dev fd0 sector 0 on HyperV

It doesnt come up on SSH session

CSCvq34357

40(2a)30(1d)The VMNetwork switch is already created as part of theinstall process

Ideally the ip address assignment should be done duringthe initial install process as well No post install stepsshould be required altrnatively it could be done as partof a post install script

CSCvk36222

40(2a)25(1c)storfs service impacted during HX upgrade from HX21(1) to 25(1c)

HX node which is not being upgraded storfs servicestopped causing outage

CSCvg53223

40(2a)35(1b)HX Stretch Cluster Witness VM reverts to DHCP onreboot

CSCvp37536

40(2a)35(2b)HyperFlex Stretch Cluster Witness does not getprogrammed with an NTP server

CSCvp46578

Release Notes for Cisco HX Data Platform Release 4036

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2b)HyperFlex cluster does not heal after maintenance onnodeserver

CSCvr18528

40(2a)40(2a)High percentage of packet loss on the eth1 data interface9K MTU

CSCvr97089

40(2a)35(2f)The HyperFlex cluster healing state is stuck at 87 andis unable to progress

CSCvr16760

40(2a)40(2a)In some circumstances two replications can occur forthe same VM

From the GUI the replication status for this VM willshow stuck in an In Progress state but there are nojobs stuck in replication layer Subsequent Replicationsshould succeed

CSCvq22898

40(2a)35(2a)After an unexpected power outage the hyperflex clusteris unable to come up correctly

All processes are running time is synced and all nodescan vmkping each other as required

CSCvp13990

40(2a)40(1a)When specifying IP address instead of node ID stclinode maintenanceMode commands fail

CSCvp33657

40(2a)40(1a)2 node edge cluster if experiencing Intersightarbitratorconnectivity issues should NOT allowHXMaintenancemode to initiate as this will bring the cluster down

CSCvr31573

40(2a)35(2c)Old files in varsupportZKTxnlog are not purged withthe daily zklog-cleanup cron job

CSCvr89066

40(2a)35(2a)When a node is in MaintenanceMode any disk removalor replacement will be reflected in UI only after the nodeis brought back frommaintenancemode This is becausestorfs is not running on the node in maintenance andwill not be able to detect disk activities until it is broughtout of Maintenance Mode

CSCvo86431

40(2a)35(2c)Cluster might go offline and stop serving IO requestsrootcvm~ stcli cluster storage-summary--detailGet cluster storage summary failedjavanetConnectException Connection refused19216814210010207

CSCvr01645

40(2a)35(2d)A stretch cluster deployment fails with a messageFormatting nodes Node in Use on the Hyperflex installerand crmZoneType value is seen as 1 in storfscfg oroptspringpathconfigstretchtunes

CSCvq80340

Release Notes for Cisco HX Data Platform Release 4037

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)30(1c)During upgrade we upload the upgrade packagedownloaded from Ciscocom Upgrade packages are offile type tgz hxconnect acceptsstorfs-packages-401a-33028tar file a different type ofcompression and allows upgrade to start and fail whenit cannot decompress the file for bootstrap and scvmclientupgrade on ESXi

CSCvq63888

40(2a)40(2a)While attempting to install hyperflex via intersight thefollowing error is seen

Failed in Task Add hosts to vCenter Cluster with ErrorTry adding hosts manually to vCenter and retry Failedto get addressFailed to host by namelookupHostnamecompanycom on 000053 read udp12700145430-gt12700153 read connection refused

CSCvr52098

40(2a)35(2a)CoreAPI call may time out for cluster management APIwhile doing cluster create validation

CSCvo92952

40(2a)35(2d)Validation fails with Mixed mode expansion checkdue to Enhanced vMotion Compatibility Incompatibilitywhen the ESXi version is different

CSCvq95460

40(2a)30(1i)This failure is seen in installer when custom workflowis used hypervisor configuration + deploy (clean diskpartition=no) at Installing software packages on storagecontroller VM

CSCvr66309

40(2a)35(2d)The HX Installer apparently replaced a comma bysemicolon on the VLAN ID range while exporting theconfiguration to a JSON file

CSCvq93831

40(2a)40(1a)When using unsupported browsers HX Connect userswill not be able to login These users see an errorindicating unauthorized user

CSCvp22693

Release Notes for Cisco HX Data Platform Release 4038

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)40(1a)A vulnerability in the web-based management interfaceof Cisco HyperFlex Software could allow anunauthenticated remote attacker to conduct a cross-siterequest forgery (CSRF) attack on an affected system

The vulnerability is due to insufficient CSRF protectionsfor the web UI on an affected device An attacker couldexploit this vulnerability by persuading a user of theinterface to follow a malicious link A successful exploitcould allow the attacker to perform arbitrary actions withthe privilege level of the affected user

Cisco has released software updates that address thisvulnerability There are no workarounds that address thisvulnerability

This advisory is available at the following link

httpstoolsciscocomsecuritycentercontentCiscoSecurityAdvisorycisco-sa-20190807-hypflex-csrf

CSCvp24343

40(2a)35(2d)Add message to HX Connect Progress flow to notAcknowledge Pending Activities and reboot the serverson UCSM

HX Connect is doing a controlled rolling server upgradein the background

CSCvq22844

40(2a)40(1b)Compute Node expand operation fails due to incorrectcluster name

CSCvr43786

40(2a)26(1e)RF-2 cluster shutdown during data resynchronizationafter a node failure (non storfs) followed by multipledisk read failures and a HardBlacklist of a disk

CSCvp41241

40(2a)40(1a)HyperFlex Edge nodes do not properly set the ESXihostname during deployment

CSCvo13143

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4039

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4040

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1b) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401b-33133p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4041

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4042

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1b)40(1a)While running the HyperFlex installer with WindowsServer Hyper-V cluster creation process fails with theerror Failure occurred during ClusterCreation process Unable to post thecontent to the downstream This symptomis encountered while deploying HyperFlex cluster withHyperFlex nodes configuredordered with Cisco VIC1457 MLOM (PID HX-MLOM-C25Q-04) andWindows Server Datacenter or Core with Hyper-Vversion 2016 or 2019

CSCvp64140

Release Notes for Cisco HX Data Platform Release 4043

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)35(2b)Adding Micron 5200 drive to a cluster fails to increasecluster capacity

CSCvo69067

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4044

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4045

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1a) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401a-33028p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4046

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4047

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1a)30(1d)Cluster instability when disks of different sector sizeplaced in HX node

CSCvk17250

40(1a)

35(2c)

35(1a)When logged in using a local HX user account insteadof a Virtual Center account an error message appearsintermittently indicating Virtual Centerunreachable or Resource informationcannot be updated when VC is reachable

CSCvo36198

40(1a)

35(2b)

30(1d)HXDP does not work with EMC RecoverPoint - needsto support VMware API (FSS-Readdir)

CSCvk38003

Release Notes for Cisco HX Data Platform Release 4048

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)

35(2a)

30(1i)

25(1c)

25(1d)

Zookeeper (Exhibitor) process on HyperFlex StorageController VMs may fail to respond to requests whenvarzookeeper has low or no free space

CSCvm90352

40(1a)26(1c)Use Asynchronous consolidation for HX SnapshotsCSCvn02151

40(1a)35(2b)HX Quiesced Snapshot for Backup VendorsCSCvo90713

40(1a)35(2a)The cluster creationcluster expansion workflow stopswith the following error message at the validation step

FIRMWARE-Check UCSC-SAS-M5HD

FIRMWARE-Check UCSC-SAS-M5HD Required00000029000000320000003500000050 Found00000058

Action Needed Update the Controller Firmware toRequired Version

CSCvn17787

35(2b)30(1d)Kernel migration fails when a local datastore is searchedfor in the ESXi host

CSCvn73127

35(2a)26(1b)A node shuts down when two disks are replaced (acaching disk and another capacity disk) where thecapacity disk is inserted first and then the caching diskis inserted

CSCvk46364

Hyper-V

40(1a)35(2a)Cluster expansion may fail with an error code 500 -operation timed out

CSCvn28721

35(2b)35(2a)During upgrade remove the VLAN on team created foruser vSwitch During fresh installation only one VLANtag is set to the vSwitch and team although multipleVLANs were entered

CSCvn54300

35(2b)35(2a)While upgrading a Hyper-V cluster on account of a rarerace condition between the stUpgradeService andZookeeper servers the upgrade orchestration throws anupgrade validation error and the upgrade process isaborted

CSCvn60486

Release Notes for Cisco HX Data Platform Release 4049

Resolved Caveats in Release 40(1a)

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Startrestart EAM service invCenter

When EAM service is not runningvCenter registration is failing

CSCvt55712

35(2a)Contact Cisco TAC if you believeyou are seeing similar conditions

HyperFlex Stretch Cluster saw poorapplication performance due to writelatency cluster remained unhealthy andrebalance was stuck

This defect is currently being used forinvestigative purposes

CSCvt45344

35(2g)Remove log files fromvarlogspringpath folder when thelog files are not getting rotatedStop the relevant service removelarge log files fromvarlogspringpath folder andrestart the service

varstv folder in Controller VM maybecome full

CSCvt36374

40(2b)Restarting the stMgr will initializeit again

In a Cluster configured with DR networkStMgr may not get initialized and stuckin a deadlock while enabling the IPTablerules (can be verified from the stMgrlog)

CSCvt89709

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Click recover for the selectedVM again under the standalonemode This will move the VM torecovered state

When the user recovers a VM protectedin a group it along with other VMs in thegroup move to recovering state instandalonemode The selectedVM shouldmove to recovered state

CSCvs75553

35(2e)Clean up the fail over cluster andthen recreate the fail over clusterNo need to touch the HX storagecluster

Hyper-V When replicated DC was usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2d)40(1a)

Keep cluster utilization as low aspossible To avoid read-only statekeep below 92

HyperFlex clusters storage utilization isabove 92 usable capacity

CSCvr20922

Release Notes for Cisco HX Data Platform Release 4050

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1b)NAAll the disks in the node got locked afterrebooting the node

We successfully unlocked usingsed-clientsh -U command but wantedto test with another reboot and driveslocked again

CSCvs74286

35(2g)To recover the VM copy over thedata disks and attach them to thenew VM

After enable encryption HX connectencryption status showing Caution butall of the disks enable encryptionsuccessful

-Self Encrypting Drives Service isrunning on all of the nodes

-There is no error message of encryption

- USB0 interface is up

-All SED disks showing supported 1enabled 1 locked 0

CSCvs93245

40(1b)Enable kernelpanic_on_oops as apersistent configuration This willcause the node to panic and rebootimmediately

A cluster node running HX release40(1b) may hang in the Linux kernelThis is classified as an oops and adeviation from the expected behavior

CSCvs54285

Release Notes for Cisco HX Data Platform Release 4051

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Enabling K8iscsi stack in Hx clusterCSCvs41324

Release Notes for Cisco HX Data Platform Release 4052

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Steps to enable K8 on HX cluster

Perform following steps on ALLthe controller VMs

1 Updateetcinitscvmclientconf toenable tune iscsiEnable totrue

sed -iesiscsiEnable=falseiscsiEnable=trueetcinitscvmclientconf

2 Run following initctlcommand to reloadconfiguration

initctl reload-configuration

3 Restart scvmclient process

stop scvmclient startscvmclient

4 Run following initctlcommand to mount SYSTEMDS

initctl emit --no-waitsystem-datastore-created

5 Verify that iscsiEnable tuneis set to true using thefollowing command

ps -eaf | grep scvmclient |grep -v grep

root 6241 1 1 Dec06 014007optspringpathstorfs-corescvmclient-T logEnabled=false -TlogSyslogEnabled=true -TlogEchoToScreen=false -TstatLoggingToFile=false -TstatLoggingToSyslog=true-TlogDir=varlogspringpath-TnfsBackendServerList=1010748100-T iscsiEnable=true -TiscsiUseAsync=true -T

Release Notes for Cisco HX Data Platform Release 4053

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

iscsiConfigFilePath=nfsSYSTEMistgtconfps -eaf | grep scvmclient |grep -v grep

35(2e)Following procedure can be usedto check NFS Queue Depth andincrease if needed

rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth256 lt- Low value[rootHXESXI1] vsish -e setvmkModulesnfsclientmnt[DATASTORE]propertiesmaxQDepth 1024[rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth1024 lt- Optimalvalue

This requires an ESXi host rebootto take effect

Please place the node in HyperflexMaintenanceMode and gracefullyreboot the node for the changes tobe applied

HyperFlex Datastore NFS Queue Depthshows as 256 which can lead toperformance (including latency) issues

CSCvr83056

40(1b)This is cosmetic The error will goaway once the cluster returns tohealthy status

When a single node is offline in clusterstcli cluster storage-summary shows twonodes unavailable rootSCVM~ stclicluster storage-summary

CSCvs47735

35(2g)Modify both the root and adminpassword for the SCVMs to be thesame

stcli security password set -uadmin

Cluster expansion fails at Config Installerstage when the root and admin passwordfor the storage controller (SCVM) aredifferent

CSCvs69317

35(2d)NAAfter successfully changing the DNSserver on the HX controller we still cansee the original DNS entry that was addedduring the deployment

CSCvs69154

Release Notes for Cisco HX Data Platform Release 4054

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Customers are strongly encouragedto work with TAC in order toidentify and remediate this issue

1 Change getcluster from 2m to8m inoptspringpathstorfs-mgmtstMgr-10confapplicationconfon all nodes

2 Restart stMgr on all nodes

You may see this error message after afailed upgrade or other task such asattempting to enter a node into HXmaintenance mode

CSCvs53555

40(2a)Run the following command oneach storage controller VM as root

grep -i monitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp sed -iesmonitor_dns_serverstruemonitor_dns_serversfalseoptspringpathhx-diag-toolswatchdog_configjsonampamp grep -imonitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp restart watchdog

Receive the following alertevent in HXConnect HX Controller VMHOSTNAME one or more configuredDNS servers not responding

CSCvt13947

35(2b)Delete any empty (Size 0) log filesunder varlogzookeeperversion-2ORvarlogzookeeperstandaloneversion-2

Zookeeper fails to start while Exhibitor isrunning however echo srvr returnsnothing

CSCvs21562

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

40(4e)1 De-commission the server

2 Power drain the server -REMOVE BOTH POWERCORDS ON THE BACK OFTHE SERVER FOR 10SECONDS THENREINSERTPOWERCORDS

3 Re-commission the server

M2 boot disk is missing from serverinventory after upgrade to server firmware40(4e) As a result server fails to boot toOS installed in the M2 disk The issuepersists after re-acknowledgement as wellas de-commission andre-acknowledgement of the server

CSCvs02466

Release Notes for Cisco HX Data Platform Release 4055

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Re-start the flusher manuallyMany disks taken offline by a fault ormanually resulting in ENOSPC oncluster

CSCvq32721

35(2e)Clean up fail over cluster and thenrecreate the fail over cluster Thereis no need to touch the HX storagecluster

[Hyper-V]When replicated DCwas usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2b)Manually install upgradescvmclient VIB andmake sure thatit matches with HXDP version

Upgrade from release HX 35(2a) to35(2x) 4X does not upgrade scvmclientRecovery point may not work

CSCvq54992

Management

30(1b)To recover the VM copy over thedata disks and attach them to thenew VM

[HyperV]VMshows up onmultiple nodesCSCvj22992

35(1a)1 Do not change MTU at ESXlevel on running cluster

2 Reset back to the original value

ChangingMTU from 9000 to 1500 on oneESX node causes storfs process to restarton all nodes

CSCvm99150

Release Notes for Cisco HX Data Platform Release 4056

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)After the installation (for newdeployments) or after an upgradeto 401a (for existingdeployments) perform thefollowing steps on all ControllerVMs

1 Edit the following tune files (onall Controller VMs)

optspringpathconfiglfftunes

optspringpathconfigvsi_16tbtunes

2 SetcleanerEnableSegSummaryCleaning

to false

3 After editing the above tunefiles

a SSH in to all the ControllerVMs

b Run storfstool -- -Z

c Run the below command andcheck the values Tune valueshould be true

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=true

d Type below commands andapplied the tunes changesdynamically

echo false gttmpstprocfssystemtunecleanerEnableSegSummaryCleaning

e Verify that tune values arechanged Run below commandValue should be false

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=false

f umount tmpstprocfs

A cluster with 8TB or 12TB disk drivesmay experience IO stalls for severalminutes after another node fails in thecluster

CSCvp23718

Release Notes for Cisco HX Data Platform Release 4057

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)1 Update the conf file manuallyto change to the link local addressto

2 Restart scvmclient on thecontroller and ESX

HX 35(2b) FlexVol to HX 40 CSIupgrade does not work FlexVol continuesto work

CSCvp26319

35(2a)Disable rebalance until node healsthen reenable rebalance

7-node cluster healing very slowlybecause rebalance change AMS reqs areblocking healing process

CSCvp62512

35(2a)NAWith moderate cluster storage capacityusage under site failover cluster clustermay experience an all paths (APD) downcondition

CSCvq49412

40(1a)NAThis product includes Third-partySoftware that is affected by thevulnerabilities identified by the followingCommon Vulnerability and Exposures(CVE) IDs

CVE-2016-0762 CVE-2016-6797CVE-2016-6816 CVE-2016-8735CVE-2017-5647 CVE-2017-12615CVE-2017-12616 CVE-2017-12617CVE-2017-7674 CVE-2018-1304CVE-2018-8014 CVE-2018-1336CVE-2018-8034 CVE-2018-11784CVE-2019-0232

CSCvp36364

35(2d)NAstcli cluster info command needs toprovide UCSM VIP address Currently itshows ucsm-hostcom

CSCvq11456

35(2a)No workarounds currently Thestale entries do not affect the dayto day operations of HyperFlex

HyperFlex upgrade validation failedbecause cluster has extraneous stNodes ininternal database

CSCvq32530

35(2a)NAWhen not in maintenancemode all drivesappear in HX Connect and get updated asexpected

When in maintenance mode extra drivesare shown in HX Connect with no slotnumbers

CSCvq42378

35(2b)Contact TAC to cleanup duplicatenode

Nodes duplicated in HX Connect aftercluster recreate

CSCvq61825

Release Notes for Cisco HX Data Platform Release 4058

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)NACurrently HyperFlex installs on Hyper-Vdo not contain the stcli security whitelistcommandset

CSCvq66245

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

Release Notes for Cisco HX Data Platform Release 4059

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)[Hyper-V] Converged node expansionfails for incoming node with VIC 14xx

CSCvq39523

Release Notes for Cisco HX Data Platform Release 4060

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

1 Set Jumbo frame inWindowsHypervisor for incomingnode

Get-NetAdapter storage-data-a|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

Get-NetAdapter storage-data-b|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

2 Verify all interfaces are setcorrectly to support Jumbo frames

Get-NetAdapter storage |Get-NetAdapterAdvancedProperty| RegistryKeyword -matchjumbo | ft -auto

You should receive the followingmessage

Name DisplayName DisplayValue

RegistryKeyword RegistryValue

---- ----------- ------------

--------------- -------------

vswitch-hx-storage-data Jumbo

Packet 9014 Bytes

JumboPacket 9014

storage-data-b Jumbo Packet

Bytes 9014 JumboPacket

9014

storage-data-a Jumbo Packet

Bytes 9014 JumboPacket

9014

3 Reboot only the incoming nodeor the node being expanded

4 Retry Cluster Expansion in

Release Notes for Cisco HX Data Platform Release 4061

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

Stretched Cluster (Availableon ESX Only)

Microsoft Hyper-VVMware ESXiNode

AllNVMe -HXAF220c-M5SN

HX240c-M5LHX220cM5

HX220cAF M5

HX240cM5

HX240cAF M5

HX240c-M5LHX220cM5

HX220cAF M5

HX240cM5

HX240cAF M5

HXAF220c-M5SN

AllNVMe -HXAF220c-M5SN

ESXi 65or 67only NotsupportedwithESXi 60orHyper-V

HX220cM5 Edge

HXAF220cM5 Edge

HX220cM4 Edge

HXAF220cM4 Edge

HX240c-M5LHX220cM5

HX220cAF M5

HX240cM5

HX240cAF M5

HX220cM4

HX220cAF M4

HX240cM4

HX240cAF M4

HXServers

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

C220M5C240M5B200M4B200M5

C240M5C220M5B200M4B200M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

mdashB200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

B200M5M4M3B260M4B420M4B460M4B480M5C220M5M4M3C240M5M4M3C460M4C480M5

Compute-OnlyUCSB-SeriesC-SeriesServers

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

Convergednodesonly

ConvergedandCompute-onlynodes

ConvergedandCompute-onlynodes

SupportedNodes

Release Notes for Cisco HX Data Platform Release 4017

Cisco HX Data Platform Storage Cluster Specifications

Stretched Cluster (Availableon ESX Only)

Microsoft Hyper-VVMware ESXiNode

NA(requiresEnterpriseHXDP-PLicense)

NA(requiresEnterpriseHXDP-PLicense)

NA(requiresEnterpriseHXDP-PLicense)

Convergednodes3-16(12TBHDDoption isnotsupportedforHyperV)

Compute-onlynodes0-16

Convergednodes3-16

Compute-onlynodes0-16

NA(requiresEnterpriseHXDP-PLicense)

M4Convergednodes 3

M5Convergednodes23or 4

RequiresHXDP-ELicense

Convergednodes3-16(12TBdrive islimited toamaximumof 8nodes)

Compute-onlynodes0-16(12TBdrive islimited toamaximumof 8nodes)

Convergednodes3-32(76TBdriveconfigsarelimited tomaximumof 12drivesnodeon AF240)

Compute-onlynodes0-32

HXDP-SLicensedNodeLimits

11 ratioofHXDP-StoComputeonlynodes

(MinmdashMax)

Convergednodes2-16 perSite

Compute-onlynodes0-21 perSite

(up tomaxclustersize)

Convergednodes2-8 perSite

Compute-onlynodes0-16 perSite

(up tomaxclustersize)

Convergednodes2-16 perSite

Compute-onlynodes0-21 perSite

(up tomaxclustersize)

Convergednodes3-16(12TBHDDoption isnotsupportedforHyperV)

Compute-onlynodes0-16

Convergednodes3-16

Compute-onlynodes0-16

Convergednodes3-32

ComputeOnlynodes0-32

(up tomaxclustersize)

Convergednodes 3

(requiresHXDP-ELicense)

Convergednodes3-16(12TBdrive islimited toamaximumof 8nodes)

Compute-onlynodes0-32(12TBdrive islimited toamaximumof 16nodes)

Convergednodes3-32(76TBdriveconfigsarelimited tomaximumof 12drivesnodeon AF240)

Compute-onlynodes0-32

(up tomaxclustersize)

HXDP-PLicensedNodeLimits

12 ratioofHXDP-PtoComputeonlynodes

(MinmdashMax)

24 perSite 64percluster

24 perSite 48percluster

32 perSite 64percluster

32326434864MaxClusterSize

Release Notes for Cisco HX Data Platform Release 4018

Cisco HX Data Platform Storage Cluster Specifications

Stretched Cluster (Availableon ESX Only)

Microsoft Hyper-VVMware ESXiNode

212121111121mdash2121MaxComputetoConvergedratio

NoExpansion

Requires Enterprise license

Requires uniform expansion across both sites

Guidelines and Limitationsbull Starting with release 40(2a) SCVM is no longer needed on a Compute node

bull HX REST API Access Token Management ndash Applications leveraging HX REST APIs should re-useaccess tokens when making API calls Once obtained using the AAA Obtain Access Token API accesstokens are valid for 18 days (1555200 seconds) In addition AAA enforces rate limiting on ObtainAccess Token API requests in a 15 minute window auth can be invoked (successfully) a maximum of5 times A user is allowed to create a maximum of 8 unrevoked tokens Subsequent call to auth willautomatically revoke the oldest issued token tomake room for the new token Amaximum of 16 unrevokedtokens can be present in system In order to prevent brute-force attacks after 10 consecutive failedauthentication attempts a user account is locked for a period of 120 secondsFor more information seeCisco HyperFlex Systems REST API Reference guide

HxConnect makes use of AAA Authentication REST API for login and the above rate limit applies toHxConnect also

Upgrade Guidelines

The following list is a highlight of critical criteria for performing an upgrade of your HyperFlex system

bull Unsupported HX Data Platform 17x 18x 20 21x 25x and 26x clustersmdashUsers from anyversion prior to 26(1a) must step through an intermediate version before upgrading to 40 or later releasesIf you need to upgrade your environment from a Cisco HyperFlex HX Data Platform software releasethat is past the last date of support to the latest suggested release on the Cisco Software Download sitesee Cisco HyperFlex Systems Upgrade Guide for Unsupported Cisco HXReleases For more informationsee the Software Advisory for CSCvq66867 WARNING Only Use HXDP 26(1e) Upgrade PackageWhen Upgrading From HXDP 18(1a)-18(1e)

bull Hypercheck Health Check UtilitymdashCisco recommends running this proactive health check utility onyour HyperFlex cluster prior to upgrade These checks provide early visibility into any areas that mayneed attention andwill help ensure a seamless upgrade experience For more information see the HyperFlexHealth amp Pre-Upgrade Check Tool TechNote for full instructions on how to install and run Hypercheck

bull vSphere 67 Software AdvisorymdashDo not upgrade to Cisco HX Data Platform Release 40(1a) whenrunning ESXi 67U1 EP06 (build 11675023) Do not upgrade to 67U1 EP06 (build 11675023) ifrunning Cisco HX Data Platform Release 40(1a) See the Software Advisory CSCvo56350 for furtherdetails

Release Notes for Cisco HX Data Platform Release 4019

Guidelines and Limitations

The software build version posted at release will override any other local versions

bull Required vCenter upgrademdashFor enhanced security Cisco HX Data Platform Release 35(1a) or laterrequires the use of TLS 12 Therefore vCenter must be upgraded to 60 U3f or later before upgradingto Cisco HX Data Platform Release 35 or later In addition ESXi should be upgraded as required tomeet HX Data Platform compatibility requirements

bull Minimum HXDP version for upgrademdashHXData Platform clusters running 26(1a) or later may upgradedirectly to 40 using the HX Connect UI

bull Cluster ReadinessmdashEnsure that the cluster is properly bootstrapped and the updated plug-in is loadedbefore proceeding Manual cluster bootstrap is required for upgrade from a pre-35 release

bull Cluster ReadinessmdashEnsure that the cluster is properly bootstrapped and the updated plug-in is loadedbefore proceeding Manual cluster bootstrap is required for HX releases earlier than 35(1a) For moreinformation see theManual Bootstrap Upgrade Process in the Cisco HyperFlex Systems Upgrade Guidefor VMware ESXi Release 40 Do not skip this cluster bootstrap step it is required for all upgradesuntil HX Release 35(1a) Auto bootstrap is supported beginning with HX release 35(1a) For moreinformation see the Auto Bootstrap Upgrade Process from HX Connect UI in the Cisco HyperFlexSystems Upgrade Guide for VMware ESXi Release 40

Manual bootstrap is not supported on Intersight clusters

bull Initiating Upgrade―Use the HX Connect UI or CLI stcli commands when upgrading from 25(1a)or later releases Use either the CLI stcli commands or the HX Data Platform Plug-in to the vSphereWeb Client when upgrading from a pre-25(1a) release The vCenter plug-in should not be used forupgrades starting with the 25(1a) release The vCenter plug-in should not be used for upgrades startingwith the 25(1a) release

If the current cluster version is at 35(1a) or above you do not need to use the stcli command Directupgrade to 40 is possible

bull Complete your Upgrade―The self-healing (or rebalance) capability is disabled temporarily during theupgrade window If the upgrade fails you should complete the upgrade as soon as possible

bull ESXi and HXDP Compatibility―Ensure your cluster is running a compatible version of ESXi basedon the running the HXData Platform version (see the section Software Requirements for VMware ESXi)ESXi compatibility is determined by the major version and update release of ESXi It is generally bestto upgrade HXDP and ESXi together if combining the upgrade operations into a single optimized rebootWhen running a split upgrade first upgrade the HX Data Platform then proceed to upgrade ESXi

bull If Upgrading to vSphere 65

bull Certain cluster functions such as native and scheduled snapshotsReadyClones and Enter or Exit HX Maintenance Mode will not operatefrom the time the upgrade is started until the HX Data Platform upgrade iscomplete

bull After upgrading ESXi using the offline zip bundle use the ESX ExitMaintenance Mode option The HX Exit Maintenance Mode option doesnot operate in the vSphere Web Client until the HX Data Platform upgradeis complete

Note

Release Notes for Cisco HX Data Platform Release 4020

Guidelines and Limitations

bull vSphere 60 UpgradesmdashUsers on vSphere 60 migrating to 65 upgrade components in the followingorder

1 Upgrade HX Data Platform and UCS firmware

2 Upgrade HX Data Platform and ESXi

3 Upgrade HX Data Platform only first then upgrade ESXi or UCS firmware or both

bull M4 Server Firmware UpgradesmdashUpgrade server firmware to ensure smooth operation and to correctknown issues Specifically newer SAS HBA firmware is available in this release and is recommendedfor long-term stability

bull Users are encouraged to upgrade to 31(3c) C-bundle or later whenever possible

bull Users running C-bundle versions before 31(2f) must upgrade server firmware by performing acombined upgrade of UCS server firmware (C-bundle) to 31(3c) or later and HX Data Platform to25 Do not split the upgrade into two separate operations

bull If the cluster is already on 31(2f) C-bundle or later you may perform an HX Data Platform onlyor combined upgrade as required

bull M5 Server Firmware UpgradesmdashM5 generation servers must run firmware version 32(2d) or later

bull Firmware Downgrades mdashDowngrading UCSM from the HX-installer is not supported

bull M4M5 Mixed DomainsmdashA mixed domain occurs when a new separate M5 cluster is installed underthe same UCS domain that contains existing M4 clusters Under these conditions orchestrated UCSserver firmware upgrade will not operate until Cisco HX Data Platform Release 26 or later is installedon the M4 clusters Therefore it is best practice to first upgrade UCS server firmware to the latest 31(3)or 32(2) patch release before adding a new M5 cluster to the existing UCS domain Additionally any17 HX Data Platform clusters must first be upgraded before adding any new M5 clusters to the samedomain

bull Maintenance WindowmdashIf upgrading both HX Data Platform and UCS firmware you can select eithera combined or split upgrade through the vSphere HX Data Platform Plug-in depending on the length ofthe maintenance window Cisco UCSManager infrastructure upgrade is only supported using AutoInstalland the direct server firmware upgrade should be performed only through the upgrade orchestrationframework provided by the HX Data Platform Plug-in

bull Unsupported Self-Encrypting Drives (SEDs)mdashIf adding or replacing self-encrypting drives (SEDs)that have been recently qualified in newer versions of HX Data Platform insert the new drives only afterupgrading HX Data Platform to a compatible version All drives must be SED drives mixing SED andnon-SED is not supported

bullbull Enabling External Host AccessmdashWith Cisco HX Data Platform Release 40(1a) port 445 on themanagement network is blocked for enhanced security Note that prior to 40 port 445 port was openenabling external host access If you are upgrading to 40(1a) from a prior release and would like tocontinue external host access you can use a utility to open select hosts For more information aboutenabling external host access see the ConfiguringHyperFlex Share to SCVMM section in the InstallationGuide for Microsoft Hyper-V

Release Notes for Cisco HX Data Platform Release 4021

Guidelines and Limitations

Mixed Cluster Expansion Guidelinesbull Expanding existing M4 cluster with M5 converged nodes is supported

bull Expanding existing M5 cluster with M4 converged nodes is not supported

bull Expanding existing mixed M4M5 cluster with M4 or M5 converged nodes is supported

bull Adding any supported compute-only nodes is permitted with all M4 M5 and mixed M4M5 clustersusing the HX Data Platform 26 or later Installer Some example combinations are listed here manyother combinations are possibleExample combinationsExpand mixed M4M5 cluster with compute-only B200 C220 C240 M4M5Expand M4 cluster with compute-only B200 M5 C220 M5 C240M5

bull Only expansion workflow is supported to create a mixed cluster Initial cluster creation with mixedM4M5 servers is not supported

bull All M5 servers must match the form factor (220240) type (HybridAF) security capability (Non-SEDonly) amp disk configuration (QTY capacity and non-SED) of the existing M4 servers

bull HX220-M5 will use a maximum of 6 capacity disks (2 disk slots to remain empty) when mixedwith HX220-M4

bull HX Edge SED LFF Hyper-V and Stretched Clusters do not support mixed M4 and M5 clusters

Security FixesThe following security issues are resolved

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-13132 CVE-2019-9924CSCvq6313840(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-11719 CVE-2019-11727CVE-2019-11729

CSCvq7124040(2a)

Release Notes for Cisco HX Data Platform Release 4022

Mixed Cluster Expansion Guidelines

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-1125CSCvr0633940(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2015-9383 CVE-2018-14498CVE-2018-20406 CVE-2018-20852CVE-2019-10160 CVE-2019-13117CVE-2019-13118 CVE-2019-14287CVE-2019-14973 CVE-2019-15903CVE-2019-17546 CVE-2019-18197CVE-2019-18218 CVE-2019-5010CVE-2019-5094 CVE-2019-5481CVE-2019-5482 CVE-2019-9636CVE-2019-9740 CVE-2019-9947CVE-2019-9948

CSCvs0609440(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2017-13168 CVE-2017-18174CVE-2017-18216 CVE-2018-10876CVE-2018-10877 CVE-2018-10878CVE-2018-10879 CVE-2018-10880CVE-2018-10881 CVE-2018-10882CVE-2018-10902 CVE-2018-10938CVE-2018-12233 CVE-2018-12896CVE-2018-13053 CVE-2018-13094CVE-2018-13096 CVE-2018-13405CVE-2018-13406 CVE-2018-14609CVE-2018-14617 CVE-2018-14633CVE-2018-14734 CVE-2018-15572CVE-2018-15594 CVE-2018-16276CVE-2018-16658 CVE-2018-17182CVE-2018-17972 CVE-2018-18021CVE-2018-18690 CVE-2018-18710CVE-2018-6554 CVE-2018-6555CVE-2018-9363

CSCvp6501940(2a)

This vulnerability is due to insufficientHTML iframe protection An attackercould exploit this vulnerability bydirecting a user to an attacker-controlledweb page that contains amaliciousHTMLiframe A successful exploit could allowthe attacker to conduct clickjacking orother clientside browser attacks

NACSCvo9851640(2a)

Release Notes for Cisco HX Data Platform Release 4023

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerability is due to insufficientauthentication for the statistics collectionservice An attacker could exploit thisvulnerability by sending properlyformatted data values to the statisticscollection service of an affected deviceA successful exploit could allow theattacker to cause the web interfacestatistics view to present invalid data tousers

NACSCvj9558440(2a)

The vulnerability is due to insufficientCSRF protections for the web UI on anaffected device An attacker could exploitthis vulnerability by persuading a user ofthe interface to follow a malicious linkA successful exploit could allow theattacker to perform arbitrary actions withthe privilege level of the affected user

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvp2434340(2a)

Vulnerabilities with a version of cJSONidentified by CVE-2019-11834 andCVE-2019-11835

For more information see the relatedCisco Security Advisory

CVE-2019-11834 CVE-2019-11835CSCvq1994940(2a)

HX ESXi image patches for MCEPSCand TAA vulnerabilities with VMwareESXi identified by CVE-2018-12207 andCVE-20190-11135

For more information see the relatedCisco Security Advisory

CVE-2018-12207 CVE-2019-11135CSCvr5439840(2a)

CP networking vulnerabilities affectingthe Linux kernel identified byCVE-2019-11477 CVE-2019-11478CVE-2019-11479

For more information see the relatedCisco Security Advisory

CVE-2019-11477 CVE-2019-11478CVE-2019-11479

CSCvq1954640(2a)

Qualification of Microsoft securitypatches for vulnerabilities withMicrosoftHyper-V hypervisor identified withCVE-2018-12207 CVE-2019-11135

For more information see the relatedCisco Security Advisory

CVE-2018-12207 CVE-2019-11135CSCvr5439940(2a)

Release Notes for Cisco HX Data Platform Release 4024

Security Fixes

DescriptionCVEDefect IDRelease

OpenSSH vulnerabilities identified byCVE-2019-6111

For more information see the relatedCisco Security Advisory

CVE-2016-10708 CVE-2018-15473CVE-2018-20685 CVE-2019-6109CVE-2019-6111

CSCvp7646340(2a)

Multiple vulnerabilities associated withOpenSSL and LibNSS

For more information see the relatedCisco Security Advisory

CVE-2016-2105 CVE-2016-2106CVE-2016-2107 CVE-2016-2109CVE-2016-2176 CVE-2016-2177CVE-2016-2178 CVE-2016-2179CVE-2016-2180 CVE-2016-2181CVE-2016-2182 CVE-2016-2183CVE-2016-6302 CVE-2016-6303CVE-2016-6304 CVE-2016-6305CVE-2016-6306 CVE-2016-6307CVE-2016-6308 CVE-2016-7055CVE-2016-8610 CVE-2017-3731CVE-2017-3732 CVE-2017-3735CVE-2017-3736 CVE-2017-3737CVE-2017-3738 CVE-2018-0495CVE-2018-0732 CVE-2018-0734CVE-2018-0735 CVE-2018-0737CVE-2018-0739 CVE-2018-12384CVE-2018-12404 CVE-2018-5407CVE-2019-1559

CSCvp6667940(2a)

libpng vulnerability identified byCVE-2016-10087

For more information see the relatedCisco Security Advisory

CVE-2016-10087CSCvp6655540(2a)

Vulnerabilities associated with libjpg andlibpng

For more information see the relatedCisco Security Advisory

CVE-2014-9092 CVE-2016-3616CVE-2017-15232 CVE-2018-11212CVE-2018-11213 CVE-2018-11214CVE-2018-1152 CVE-2018-13785

CSCvp3458640(2a)

Vulnerabilities associated with Glibidentified by CVE-2018-16428 andCVE-2018-16429

For more information see the relatedCisco Security Advisory

CVE-2018-16428 CVE-2018-16429CSCvp3120740(2a)

Multiple vulnerabilities associated withcurl expat python 27 python 35 36and 37 freetype and giflib

For more information see the relatedCisco Security Advisory

CVE-2019-15133 CVE-2019-15903CVE-2019-5010 CVE-2019-5481CVE-2019-5482 CVE-2019-9636CVE-2019-9740 CVE-2019-9947CVE-2019-9948

CSCvr3690340(2a)

Release Notes for Cisco HX Data Platform Release 4025

Security Fixes

DescriptionCVEDefect IDRelease

Multiple third party softwarevulnerabilities

For more information see the relatedCisco Security Advisory

CVE-2019-14379 CVE-2019-12384CVE-2019-14439

CSCvq9203240(2a)

Vulnerabilities associated with elfutils

For more information see the relatedCisco Security Advisory

CVE-2018-16062 CVE-2018-16402CVE-2018-16403 CVE-2018-18310CVE-2018-18520 CVE-2018-18521CVE-2019-7149 CVE-2019-7150CVE-2019-7665

CSCvq4325040(2a)

Vulnerabilities associated with vim

For more information see the relatedCisco Security Advisory

CVE-2017-5953 CVE-2019-12735CSCvq4323040(2a)

Vulnerabilities associated with sqlite3

For more information see the relatedCisco Security Advisory

CVE-2016-6153 CVE-2017-10989CVE-2017-13685 CVE-2017-2518CVE-2017-2519 CVE-2017-2520CVE-2018-20346 CVE-2018-20505CVE-2018-20506 CVE-2019-8457CVE-2019-9936 CVE-2019-9937

CSCvq4321340(2a)

Vulnerabilities associated with glib20

For more information see the relatedCisco Security Advisory

CVE-2018-20843CSCvq4320940(2a)

Vulnerabilities associated with expat

For more information see the relatedCisco Security Advisory

CVE-2018-20843CSCvq4320540(2a)

Vulnerabilities associated with bzip2

For more information see the relatedCisco Security Advisory

CVE-2016-3189 CVE-2019-12900CSCvq4319440(2a)

Mulitple Vulnerabilities associated wtihNodeJS

For more information see the relatedCisco Security Advisory

CVE-2018-12115 CVE-2018-0734CVE-2018-5407 CVE-2018-12120CVE-2018-12121 CVE-2018-12122CVE-2018-12123 CVE-2018-12116CVE-2019-5737 CVE-2019-5739

CSCvq1069440(2a)

Vulnerabilities associated with PalletsJinja strformat_map and identified byCVE-2019-10906 and CVE-2016-10745

For more information see the relatedCisco Security Advisory

CVE-2019-10906 CVE-2016-10745CSCvq1038840(2a)

Release Notes for Cisco HX Data Platform Release 4026

Security Fixes

DescriptionCVEDefect IDRelease

Multiple Security Vulnerabilities

For more information see the relatedCisco Security Advisory

CVE-2014-8501 CVE-2014-9939CVE-2015-9262 CVE-2016-10087CVE-2016-2226 CVE-2016-4487CVE-2016-4488 CVE-2016-4489CVE-2016-4490 CVE-2016-4491CVE-2016-4492 CVE-2016-4493CVE-2016-6131 CVE-2018-10963CVE-2018-13785 CVE-2018-17100CVE-2018-17101 CVE-2018-18557CVE-2018-18661 CVE-2018-7456CVE-2018-8905

CSCvq0917840(2a)

Vulnerabilities associated withlibseccomp

For more information see the relatedCisco Security Advisory

CVE-2019-9893CSCvq0756840(2a)

Vulnerabilities associated withgdk-pixbuf

For more information see the relatedCisco Security Advisory

CVE-2017-12447CSCvq0675540(2a)

Vulnerabilities associated withpython-crypto

For more information see the relatedCisco Security Advisory

CVE-2018-6594CSCvp9381740(2a)

Vulnerabilities associated with wget

For more information see the relatedCisco Security Advisory

CVE-2018-20483 CVE-2019-5953CSCvp8672140(2a)

Multiple vulnerabilities associated withPython Crypto

For more information see the relatedCisco Security Advisory

CVE-2018-6594CSCvp6674840(2a)

Vulnerabilities associated withsensible-utils

For more information see the relatedCisco Security Advisory

CVE-2017-17512CSCvp6673440(2a)

Multiple vulnerabilities associated withPython

For more information see the relatedCisco Security Advisory

CVE-2018-1000030CSCvp6668940(2a)

Release Notes for Cisco HX Data Platform Release 4027

Security Fixes

DescriptionCVEDefect IDRelease

Multiple patch vulnerabilities identifiedby CVE-2016-10713

For more information see the relatedCisco Security Advisory

CVE-2016-10713CVE-2018-1000156 CVE-2018-6951

CSCvp6667240(2a)

Vulnerabilities associatedwith Little CMSidentified by CVE-2016-10165

For more information see the relatedCisco Security Advisory

CVE-2016-10165 CVE-2018-16435CSCvp6666440(2a)

Vulnerabilities associated with TenableScan identified byCVE-2011-5325CVE-2019-11068CVE-2016-7076CVE-2017-1000368

For more information see the relatedCisco Security Advisory

CVE-2011-5325 CVE-2016-7076CVE-2017-1000368CVE-2019-11068

CSCvp6474640(2a)

Vulnerabilities associated with libxml2294 and earlier as used in XMLSec1223 and earlier and other products andidentified by CVE-2016-9318

For more information see the relatedCisco Security Advisory

CVE-2016-9318 CVE-2017-16932CVE-2017-18258 CVE-2018-14404CVE-2018-14567

CSCvp3479240(2a)

Vulnerabilities associated with LFTPRemote File Names Unauthorized AccessVulnerability and identified byCVE-2018-10916

For more information see the relatedCisco Security Advisory

CVE-2018-10916CSCvp2926640(2a)

Vulnerabilities associated withCryptographyDeprecationWarning signerand verifier have been deprecated

For more information see the relatedCisco Security Advisory

CVE-2018-7750CSCvo3409740(2a)

Multiple Vulnerabilities in python python27 35

For more information see the relatedCisco Security Advisory

CVE-2019-16056CSCvr3979340(2a)

Release Notes for Cisco HX Data Platform Release 4028

Security Fixes

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toexecute commands as the root user

The vulnerability is due to insufficientinput validation An attacker could exploitthis vulnerability by connecting to thecluster service manager and injectingcommands into the bound process Asuccessful exploit could allow the attackerto run commands on the affected host asthe root user

Cisco has released software updates thataddress this vulnerability There areworkarounds that address thisvulnerability

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvq2417640(1b)35(2g)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

The vulnerability is due to an unprotectedlistening interface An attacker couldexploit this vulnerability by connectingto the listening interface and injectingcommands to the bound process Anexploit could allow the attacker to runcommands on the affected host as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvj9560640(1b)35(2g)

Release Notes for Cisco HX Data Platform Release 4029

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerabilities associated with theJVM18U121memory leak found duringVC alarms (concurrent 40 calls usingREST API)

CVE-2017-10053 CVE-2017-10067CVE-2017-10074 CVE-2017-10078CVE-2017-10081 CVE-2017-10087CVE-2017-10089 CVE-2017-10090CVE-2017-10096 CVE-2017-10101CVE-2017-10102 CVE-2017-10107CVE-2017-10108 CVE-2017-10109CVE-2017-10110 CVE-2017-10111CVE-2017-10115 CVE-2017-10116CVE-2017-10118 CVE-2017-10135CVE-2017-10176 CVE-2017-10193CVE-2017-10198 CVE-2017-10243CVE-2017-10274 CVE-2017-10281CVE-2017-10285 CVE-2017-10295CVE-2017-10345 CVE-2017-10346CVE-2017-10347 CVE-2017-10348CVE-2017-10349 CVE-2017-10350CVE-2017-10355 CVE-2017-10356CVE-2017-10357 CVE-2017-10388CVE-2017-3509 CVE-2017-3511CVE-2017-3526 CVE-2017-3533CVE-2017-3539 CVE-2017-3544CVE-2018-2579 CVE-2018-2582CVE-2018-2588 CVE-2018-2599CVE-2018-2602 CVE-2018-2603CVE-2018-2618 CVE-2018-2629CVE-2018-2633 CVE-2018-2634CVE-2018-2637 CVE-2018-2641CVE-2018-2663 CVE-2018-2677CVE-2018-2678 CVE-2018-2783CVE-2018-2790 CVE-2018-2794CVE-2018-2795 CVE-2018-2796CVE-2018-2797 CVE-2018-2798CVE-2018-2799 CVE-2018-2800CVE-2018-2814 CVE-2018-2815CVE-2018-2952 CVE-2018-3136CVE-2018-3139 CVE-2018-3149CVE-2018-3150 CVE-2018-3169CVE-2018-3180 CVE-2018-3183CVE-2018-3214 CVE-2019-2422

CSCvo8899740(1b)

Tomcat and Nginx logs are not beingcollected in the support bundle generatedthrough HX Connect in Release 35

NACSCvm5803140(1b)

The vulnerabilities associated with thelibmspack software package versionincluded in Cisco HX Data Platform

CVE-2018-18584

CVE-2018-18585

CSCvn3511940(1a)

Release Notes for Cisco HX Data Platform Release 4030

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerabilities associated withFasterXML Jackson-Databind TimeValueField Denial of Service

CVE-2018-14719

CVE-2018-14720

CVE-2018-1000873

CVE-2018-14721

CVE-2018-19360

CVE-2018-19362

CVE-2018-19361

CVE-2018-14718

CSCvn8228240(1a)

The vulnerabilities associated with theOpenSSL Protocol software packageversion included in Cisco HX DataPlatform

CVE-2013-3587CSCvo0505440(1a)

The vulnerabilities associated withThird-Party Software Denial of Service

CVE-2018-16487

CVE-2018-19361

CSCvo2781840(1a)

The vulnerabilities associated withLinuxkernel for Ubuntu 1710

CVE-2018-1092

CVE-2018-7492

CVE-2018-8087

CVE-2018-1068

CVE-2018-8781

CSCvm5314935(2a)

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)25(1d)stcli services dns remove should remove the DNS serverinfo from the interface files

CSCvs70967

40(2b)40(1b)HX 40(1b) new deployment from Intersight productioncloud

User receives following error while enabling CSI(Kubernetes integration)

Failure occurred during volume_access_enable Errorwas Nonzero exit code 1

CSCvt10522

40(2b)40(2a)Receive the following alertevent in HX Connect HXController VM HOSTNAME one or more configuredDNS servers not responding

CSCvt13947

Release Notes for Cisco HX Data Platform Release 4031

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)40(2a)API UI CLI show few Drive slots Empty after upgradeto 402a

The drives are seen in the sysmtool lsscsi and stclicommand outputs

The cluster is healthy with no errors

CSCvt14914

40(2b)35(2d)After successfully changing the DNS server on the HXcontroller we still can see the original DNS entry thatwas added during the deployment

CSCvs69154

40(2b)35(2e)HyperFlex Connect and VCenter show APD alarms forHX and non HX datastores This implies there is an issuewith HyperFlex when there may not be

CSCvs30080

40(2b)40(2a)An issue where all the disks in the node get locked afterrebooting the node

We successfully unlocked using sed-clientsh -Ucommand but wanted to test with another reboot anddrives locked again

CSCvs74286

40(2b)40(2a)When running stcli license commands on HyperFlexthe following error is seen

rootSpringpathControllertmp stcli license show all

Show smart licensing failed

Smart Agent is not ready please wait a minute and tryagain

CSCvt13929

40(2b)40(2a)After upgrading to HX 40(2a) the first RoboEdge nodethat gets rebooted then ignores persistent drivesPIDHX-HD24TB10K4KN

Cluster will be in degraded state and have reducedcapacity

CSCvt20144

40(2b)35(2b)Zookeeper fails to start while Exhibitor is runninghowever echo srvr returns nothing

CSCvs21562

40(2b)40(2a)When performing an HyperFlex upgrade a validationwarning may occur due to the host not having EnterprisePlus or Enterprise hypervisor licensing

Upgrade Validation Warning

ESXi host esx1labtest should be configured withVMware Enterprise license for upgrade to continue

CSCvs91787

Release Notes for Cisco HX Data Platform Release 4032

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)35(2g)Cluster expansion fails at Config Installer stage whenthe root and admin password for the storage controller(SCVM) are different

CSCvs69317

40(2b)35(2g)Panic while upgrading ESXiCSCvt06983

40(2b)40(2a)A cluster node running may hang in the Linux kernelThis is classified as an oops and a deviation from theexpected behavior

CSCvs54285

40(2b)35(2g)Rebalance failing on 10+10 node HX 35(2g) stretchcluster

CSCvs69007

40(2b)35(2d)The cluster becomes inaccessible to the IOVisorCSCvr54687

40(2b)40(2a)Panic on the storage controllerCSCvt61297

40(2b)40(2a)The size of support bundle is very large when collectedwith storfs-support command

CSCvt63306

40(2b)40(2a)storfs crash associated withFileSystemUsageWarningEvent orFileSystemUsageAlertEvent due to high usage ofvarstv above 80

CSCvt72807

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

40(2a)35(2d)In normal scenarios HX upgrade gets pendingACK afterupdating service profile In this case UCS only upgradewas stalled because after updating service profile HXupgrade was not getting pending ACK and it was alwayswaiting state

CSCvo39912

40(2a)35(1a)When using motherBoardReplace-12 to clean ZK fromold stNodepNodes resulted in unmounted datastore andmaking the size of the Hyperflex Datastores 0 resultingin all VMs in the cluster going offline

CSCvq39471

40(2a)26(1e)Upgrading cluster and getting error

Failed upgrade validations Checking vCenterconfiguration Reason Upgrade validations failed DRSFault Insufficient resources to satisfy configured failover

CSCvm77294

Release Notes for Cisco HX Data Platform Release 4033

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)The cluster expand fails on a node with DR replicationconfigured

When a HX cluster which has DR replication configuredis expanded we see the installer UI pulling in thereplicationVLAN information instead of themanagementVLAN information Even if we change that informationto the correct mgmt VLAN id and name it it does notseem to work as the node is configured with the VLANof the replicationVLAN in ESXi This leads to the failureof the node add with the host unreachable error

CSCvo70650

40(2a)35(1a)Customer reported that server didnt complete thefirmware upgrade automatically one by one

The user manually needs to put the host into maintenancemode and they manually acknowledged the pendingrequests to finish the UCS firmware upgrade

Maintenance policy is set to default (User-ack) as perthe design

CSCvo91624

40(2a)35(1i)When the cluster is in Failed state and stcli noderemove is attempted the output shows successful thoughthe node failed to get removed from the cluster

CSCvo93017

40(2a)35(1a)HyperFlex cluster upgrade may fail during validationwith a DRS Validation failed error

CSCvp31021

40(2a)35(2b)HX cluster expansion will fail with the following errormessage

MAC address pool configuration failure150[ErrorDescription] bad address block range definitioncollision

CSCvp58318

40(2a)35(2b)Memory usage by carbon cacheCSCvq34873

40(2a)35(2g)Upgrading ESXi cluster fails with error Nodemaintenance mode failed

CSCvr03240

40(2a)35(2e)storfs process does not automatically start on Exit HXMaintenance Mode or other tasks which restart StorageController

CSCvr88978

40(2a)35(2a)Post install script fails with message

Failed to execute ipmitool on HX node

CSCvp10707

40(2a)40(1a)HyperFlex expansionworkflow doesnt pull VLANnamecorrectly

CSCvp46539

Release Notes for Cisco HX Data Platform Release 4034

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)During HX cluster deployment - Validate ClusterCreation phase - HX Installer might fail with the error

fromvarlogspringpathstDeploylog

CSCvq45087

40(2a)35(2b)HX Installer fails to login to the SOL for configuringESX hosts

CSCvq91380

40(2a)35(2d)If DRS is enabled with configuration parameters otherthen fully automated ie manual or partially automatedthan this will change to fully automated during clusterexpansion

CSCvr44222

40(2a)35(1a)One of the validation task name has a typo The taskname is Validating node for mixed of SED and non-SEDdisks

It should be Validating node for mix of SED andnon-SED disks

CSCvp82175

40(2a)30(1i)This combination of using smaller NVMe drives is notsupported in expand operation For example if the drivesare only 375GB and they cannot be added to an existingcluster with larger caching SSD

CSCvo12359

40(2a)35(1a)HyperFlex includes a version of OpenSSL that is affectedby the vulnerabilities identified by the followingCommon Vulnerability and Exposures

CVE-2018-0495

CSCvp66679

40(2a)30(1e)SDK invoke exception when taking a native snapshot ona VM with GPU pci passthrough attached

CSCvo00511

40(2a)30(1i)Node replacement script fails due to EAM errorCSCvo62867

40(2a)35(1a)While pairing a cluster (HX release gt= 35) and cluster(HX release lt 35) remote replication network test failson the cluster (HX release lt 35)

CSCvo79760

40(2a)30(1b)Fixed the issue in the latest support-workflow bundleThis should not be hit anymore

CSCvo87061

40(2a)35(2a)On a 3 node cluster the MBreplace script fails as thescript looks for HEALTHY cluster status

CSCvo87080

40(2a)35(2a)MbReplace script (tar) hangs when running on 352b or40

CSCvp12359

Release Notes for Cisco HX Data Platform Release 4035

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)HX replication cleanup failing with error NA 35(2a) -INFODR state is not clean

CSCvp63958

40(2a)40(1a)The check_stig_parameters API incorrectly shows thecompliance state of the cluster as not STIG compliant

CSCvp66277

40(2a)35(2c)The snapshot creation on CBT enabled VM fails witherror Failed in vmreparent vmkfstools clone1

CSCvq06952

40(2a)35(2a)HyperFlex GUI when setting up remote DR-replicationpartnerpeer should show remote replication partner TCPport reachability test results

This will provide quick upfront connectivity results thatfrequently takes customer a while to manually check andconfirm

If reachability tests fail an info window and HX alertshould trigger that list the destination peer IP and thespecific port(s) that could not be reache

CSCvo60587

40(2a)35(2b)If you have more than one IP pool configured for areplication network and local replication network testsare failing you might be running into this issue

CSCvr67130

40(2a)35(2a)Skip task option in MBreplace and Software redeployscripts The scripts should have the option to skip tasksas user inputs and not by modifying the json file

CSCvp05204

40(2a)40(1a)Error noticed on SCVM Console print_req_error IOerror dev fd0 sector 0 on HyperV

It doesnt come up on SSH session

CSCvq34357

40(2a)30(1d)The VMNetwork switch is already created as part of theinstall process

Ideally the ip address assignment should be done duringthe initial install process as well No post install stepsshould be required altrnatively it could be done as partof a post install script

CSCvk36222

40(2a)25(1c)storfs service impacted during HX upgrade from HX21(1) to 25(1c)

HX node which is not being upgraded storfs servicestopped causing outage

CSCvg53223

40(2a)35(1b)HX Stretch Cluster Witness VM reverts to DHCP onreboot

CSCvp37536

40(2a)35(2b)HyperFlex Stretch Cluster Witness does not getprogrammed with an NTP server

CSCvp46578

Release Notes for Cisco HX Data Platform Release 4036

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2b)HyperFlex cluster does not heal after maintenance onnodeserver

CSCvr18528

40(2a)40(2a)High percentage of packet loss on the eth1 data interface9K MTU

CSCvr97089

40(2a)35(2f)The HyperFlex cluster healing state is stuck at 87 andis unable to progress

CSCvr16760

40(2a)40(2a)In some circumstances two replications can occur forthe same VM

From the GUI the replication status for this VM willshow stuck in an In Progress state but there are nojobs stuck in replication layer Subsequent Replicationsshould succeed

CSCvq22898

40(2a)35(2a)After an unexpected power outage the hyperflex clusteris unable to come up correctly

All processes are running time is synced and all nodescan vmkping each other as required

CSCvp13990

40(2a)40(1a)When specifying IP address instead of node ID stclinode maintenanceMode commands fail

CSCvp33657

40(2a)40(1a)2 node edge cluster if experiencing Intersightarbitratorconnectivity issues should NOT allowHXMaintenancemode to initiate as this will bring the cluster down

CSCvr31573

40(2a)35(2c)Old files in varsupportZKTxnlog are not purged withthe daily zklog-cleanup cron job

CSCvr89066

40(2a)35(2a)When a node is in MaintenanceMode any disk removalor replacement will be reflected in UI only after the nodeis brought back frommaintenancemode This is becausestorfs is not running on the node in maintenance andwill not be able to detect disk activities until it is broughtout of Maintenance Mode

CSCvo86431

40(2a)35(2c)Cluster might go offline and stop serving IO requestsrootcvm~ stcli cluster storage-summary--detailGet cluster storage summary failedjavanetConnectException Connection refused19216814210010207

CSCvr01645

40(2a)35(2d)A stretch cluster deployment fails with a messageFormatting nodes Node in Use on the Hyperflex installerand crmZoneType value is seen as 1 in storfscfg oroptspringpathconfigstretchtunes

CSCvq80340

Release Notes for Cisco HX Data Platform Release 4037

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)30(1c)During upgrade we upload the upgrade packagedownloaded from Ciscocom Upgrade packages are offile type tgz hxconnect acceptsstorfs-packages-401a-33028tar file a different type ofcompression and allows upgrade to start and fail whenit cannot decompress the file for bootstrap and scvmclientupgrade on ESXi

CSCvq63888

40(2a)40(2a)While attempting to install hyperflex via intersight thefollowing error is seen

Failed in Task Add hosts to vCenter Cluster with ErrorTry adding hosts manually to vCenter and retry Failedto get addressFailed to host by namelookupHostnamecompanycom on 000053 read udp12700145430-gt12700153 read connection refused

CSCvr52098

40(2a)35(2a)CoreAPI call may time out for cluster management APIwhile doing cluster create validation

CSCvo92952

40(2a)35(2d)Validation fails with Mixed mode expansion checkdue to Enhanced vMotion Compatibility Incompatibilitywhen the ESXi version is different

CSCvq95460

40(2a)30(1i)This failure is seen in installer when custom workflowis used hypervisor configuration + deploy (clean diskpartition=no) at Installing software packages on storagecontroller VM

CSCvr66309

40(2a)35(2d)The HX Installer apparently replaced a comma bysemicolon on the VLAN ID range while exporting theconfiguration to a JSON file

CSCvq93831

40(2a)40(1a)When using unsupported browsers HX Connect userswill not be able to login These users see an errorindicating unauthorized user

CSCvp22693

Release Notes for Cisco HX Data Platform Release 4038

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)40(1a)A vulnerability in the web-based management interfaceof Cisco HyperFlex Software could allow anunauthenticated remote attacker to conduct a cross-siterequest forgery (CSRF) attack on an affected system

The vulnerability is due to insufficient CSRF protectionsfor the web UI on an affected device An attacker couldexploit this vulnerability by persuading a user of theinterface to follow a malicious link A successful exploitcould allow the attacker to perform arbitrary actions withthe privilege level of the affected user

Cisco has released software updates that address thisvulnerability There are no workarounds that address thisvulnerability

This advisory is available at the following link

httpstoolsciscocomsecuritycentercontentCiscoSecurityAdvisorycisco-sa-20190807-hypflex-csrf

CSCvp24343

40(2a)35(2d)Add message to HX Connect Progress flow to notAcknowledge Pending Activities and reboot the serverson UCSM

HX Connect is doing a controlled rolling server upgradein the background

CSCvq22844

40(2a)40(1b)Compute Node expand operation fails due to incorrectcluster name

CSCvr43786

40(2a)26(1e)RF-2 cluster shutdown during data resynchronizationafter a node failure (non storfs) followed by multipledisk read failures and a HardBlacklist of a disk

CSCvp41241

40(2a)40(1a)HyperFlex Edge nodes do not properly set the ESXihostname during deployment

CSCvo13143

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4039

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4040

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1b) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401b-33133p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4041

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4042

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1b)40(1a)While running the HyperFlex installer with WindowsServer Hyper-V cluster creation process fails with theerror Failure occurred during ClusterCreation process Unable to post thecontent to the downstream This symptomis encountered while deploying HyperFlex cluster withHyperFlex nodes configuredordered with Cisco VIC1457 MLOM (PID HX-MLOM-C25Q-04) andWindows Server Datacenter or Core with Hyper-Vversion 2016 or 2019

CSCvp64140

Release Notes for Cisco HX Data Platform Release 4043

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)35(2b)Adding Micron 5200 drive to a cluster fails to increasecluster capacity

CSCvo69067

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4044

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4045

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1a) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401a-33028p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4046

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4047

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1a)30(1d)Cluster instability when disks of different sector sizeplaced in HX node

CSCvk17250

40(1a)

35(2c)

35(1a)When logged in using a local HX user account insteadof a Virtual Center account an error message appearsintermittently indicating Virtual Centerunreachable or Resource informationcannot be updated when VC is reachable

CSCvo36198

40(1a)

35(2b)

30(1d)HXDP does not work with EMC RecoverPoint - needsto support VMware API (FSS-Readdir)

CSCvk38003

Release Notes for Cisco HX Data Platform Release 4048

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)

35(2a)

30(1i)

25(1c)

25(1d)

Zookeeper (Exhibitor) process on HyperFlex StorageController VMs may fail to respond to requests whenvarzookeeper has low or no free space

CSCvm90352

40(1a)26(1c)Use Asynchronous consolidation for HX SnapshotsCSCvn02151

40(1a)35(2b)HX Quiesced Snapshot for Backup VendorsCSCvo90713

40(1a)35(2a)The cluster creationcluster expansion workflow stopswith the following error message at the validation step

FIRMWARE-Check UCSC-SAS-M5HD

FIRMWARE-Check UCSC-SAS-M5HD Required00000029000000320000003500000050 Found00000058

Action Needed Update the Controller Firmware toRequired Version

CSCvn17787

35(2b)30(1d)Kernel migration fails when a local datastore is searchedfor in the ESXi host

CSCvn73127

35(2a)26(1b)A node shuts down when two disks are replaced (acaching disk and another capacity disk) where thecapacity disk is inserted first and then the caching diskis inserted

CSCvk46364

Hyper-V

40(1a)35(2a)Cluster expansion may fail with an error code 500 -operation timed out

CSCvn28721

35(2b)35(2a)During upgrade remove the VLAN on team created foruser vSwitch During fresh installation only one VLANtag is set to the vSwitch and team although multipleVLANs were entered

CSCvn54300

35(2b)35(2a)While upgrading a Hyper-V cluster on account of a rarerace condition between the stUpgradeService andZookeeper servers the upgrade orchestration throws anupgrade validation error and the upgrade process isaborted

CSCvn60486

Release Notes for Cisco HX Data Platform Release 4049

Resolved Caveats in Release 40(1a)

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Startrestart EAM service invCenter

When EAM service is not runningvCenter registration is failing

CSCvt55712

35(2a)Contact Cisco TAC if you believeyou are seeing similar conditions

HyperFlex Stretch Cluster saw poorapplication performance due to writelatency cluster remained unhealthy andrebalance was stuck

This defect is currently being used forinvestigative purposes

CSCvt45344

35(2g)Remove log files fromvarlogspringpath folder when thelog files are not getting rotatedStop the relevant service removelarge log files fromvarlogspringpath folder andrestart the service

varstv folder in Controller VM maybecome full

CSCvt36374

40(2b)Restarting the stMgr will initializeit again

In a Cluster configured with DR networkStMgr may not get initialized and stuckin a deadlock while enabling the IPTablerules (can be verified from the stMgrlog)

CSCvt89709

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Click recover for the selectedVM again under the standalonemode This will move the VM torecovered state

When the user recovers a VM protectedin a group it along with other VMs in thegroup move to recovering state instandalonemode The selectedVM shouldmove to recovered state

CSCvs75553

35(2e)Clean up the fail over cluster andthen recreate the fail over clusterNo need to touch the HX storagecluster

Hyper-V When replicated DC was usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2d)40(1a)

Keep cluster utilization as low aspossible To avoid read-only statekeep below 92

HyperFlex clusters storage utilization isabove 92 usable capacity

CSCvr20922

Release Notes for Cisco HX Data Platform Release 4050

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1b)NAAll the disks in the node got locked afterrebooting the node

We successfully unlocked usingsed-clientsh -U command but wantedto test with another reboot and driveslocked again

CSCvs74286

35(2g)To recover the VM copy over thedata disks and attach them to thenew VM

After enable encryption HX connectencryption status showing Caution butall of the disks enable encryptionsuccessful

-Self Encrypting Drives Service isrunning on all of the nodes

-There is no error message of encryption

- USB0 interface is up

-All SED disks showing supported 1enabled 1 locked 0

CSCvs93245

40(1b)Enable kernelpanic_on_oops as apersistent configuration This willcause the node to panic and rebootimmediately

A cluster node running HX release40(1b) may hang in the Linux kernelThis is classified as an oops and adeviation from the expected behavior

CSCvs54285

Release Notes for Cisco HX Data Platform Release 4051

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Enabling K8iscsi stack in Hx clusterCSCvs41324

Release Notes for Cisco HX Data Platform Release 4052

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Steps to enable K8 on HX cluster

Perform following steps on ALLthe controller VMs

1 Updateetcinitscvmclientconf toenable tune iscsiEnable totrue

sed -iesiscsiEnable=falseiscsiEnable=trueetcinitscvmclientconf

2 Run following initctlcommand to reloadconfiguration

initctl reload-configuration

3 Restart scvmclient process

stop scvmclient startscvmclient

4 Run following initctlcommand to mount SYSTEMDS

initctl emit --no-waitsystem-datastore-created

5 Verify that iscsiEnable tuneis set to true using thefollowing command

ps -eaf | grep scvmclient |grep -v grep

root 6241 1 1 Dec06 014007optspringpathstorfs-corescvmclient-T logEnabled=false -TlogSyslogEnabled=true -TlogEchoToScreen=false -TstatLoggingToFile=false -TstatLoggingToSyslog=true-TlogDir=varlogspringpath-TnfsBackendServerList=1010748100-T iscsiEnable=true -TiscsiUseAsync=true -T

Release Notes for Cisco HX Data Platform Release 4053

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

iscsiConfigFilePath=nfsSYSTEMistgtconfps -eaf | grep scvmclient |grep -v grep

35(2e)Following procedure can be usedto check NFS Queue Depth andincrease if needed

rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth256 lt- Low value[rootHXESXI1] vsish -e setvmkModulesnfsclientmnt[DATASTORE]propertiesmaxQDepth 1024[rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth1024 lt- Optimalvalue

This requires an ESXi host rebootto take effect

Please place the node in HyperflexMaintenanceMode and gracefullyreboot the node for the changes tobe applied

HyperFlex Datastore NFS Queue Depthshows as 256 which can lead toperformance (including latency) issues

CSCvr83056

40(1b)This is cosmetic The error will goaway once the cluster returns tohealthy status

When a single node is offline in clusterstcli cluster storage-summary shows twonodes unavailable rootSCVM~ stclicluster storage-summary

CSCvs47735

35(2g)Modify both the root and adminpassword for the SCVMs to be thesame

stcli security password set -uadmin

Cluster expansion fails at Config Installerstage when the root and admin passwordfor the storage controller (SCVM) aredifferent

CSCvs69317

35(2d)NAAfter successfully changing the DNSserver on the HX controller we still cansee the original DNS entry that was addedduring the deployment

CSCvs69154

Release Notes for Cisco HX Data Platform Release 4054

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Customers are strongly encouragedto work with TAC in order toidentify and remediate this issue

1 Change getcluster from 2m to8m inoptspringpathstorfs-mgmtstMgr-10confapplicationconfon all nodes

2 Restart stMgr on all nodes

You may see this error message after afailed upgrade or other task such asattempting to enter a node into HXmaintenance mode

CSCvs53555

40(2a)Run the following command oneach storage controller VM as root

grep -i monitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp sed -iesmonitor_dns_serverstruemonitor_dns_serversfalseoptspringpathhx-diag-toolswatchdog_configjsonampamp grep -imonitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp restart watchdog

Receive the following alertevent in HXConnect HX Controller VMHOSTNAME one or more configuredDNS servers not responding

CSCvt13947

35(2b)Delete any empty (Size 0) log filesunder varlogzookeeperversion-2ORvarlogzookeeperstandaloneversion-2

Zookeeper fails to start while Exhibitor isrunning however echo srvr returnsnothing

CSCvs21562

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

40(4e)1 De-commission the server

2 Power drain the server -REMOVE BOTH POWERCORDS ON THE BACK OFTHE SERVER FOR 10SECONDS THENREINSERTPOWERCORDS

3 Re-commission the server

M2 boot disk is missing from serverinventory after upgrade to server firmware40(4e) As a result server fails to boot toOS installed in the M2 disk The issuepersists after re-acknowledgement as wellas de-commission andre-acknowledgement of the server

CSCvs02466

Release Notes for Cisco HX Data Platform Release 4055

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Re-start the flusher manuallyMany disks taken offline by a fault ormanually resulting in ENOSPC oncluster

CSCvq32721

35(2e)Clean up fail over cluster and thenrecreate the fail over cluster Thereis no need to touch the HX storagecluster

[Hyper-V]When replicated DCwas usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2b)Manually install upgradescvmclient VIB andmake sure thatit matches with HXDP version

Upgrade from release HX 35(2a) to35(2x) 4X does not upgrade scvmclientRecovery point may not work

CSCvq54992

Management

30(1b)To recover the VM copy over thedata disks and attach them to thenew VM

[HyperV]VMshows up onmultiple nodesCSCvj22992

35(1a)1 Do not change MTU at ESXlevel on running cluster

2 Reset back to the original value

ChangingMTU from 9000 to 1500 on oneESX node causes storfs process to restarton all nodes

CSCvm99150

Release Notes for Cisco HX Data Platform Release 4056

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)After the installation (for newdeployments) or after an upgradeto 401a (for existingdeployments) perform thefollowing steps on all ControllerVMs

1 Edit the following tune files (onall Controller VMs)

optspringpathconfiglfftunes

optspringpathconfigvsi_16tbtunes

2 SetcleanerEnableSegSummaryCleaning

to false

3 After editing the above tunefiles

a SSH in to all the ControllerVMs

b Run storfstool -- -Z

c Run the below command andcheck the values Tune valueshould be true

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=true

d Type below commands andapplied the tunes changesdynamically

echo false gttmpstprocfssystemtunecleanerEnableSegSummaryCleaning

e Verify that tune values arechanged Run below commandValue should be false

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=false

f umount tmpstprocfs

A cluster with 8TB or 12TB disk drivesmay experience IO stalls for severalminutes after another node fails in thecluster

CSCvp23718

Release Notes for Cisco HX Data Platform Release 4057

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)1 Update the conf file manuallyto change to the link local addressto

2 Restart scvmclient on thecontroller and ESX

HX 35(2b) FlexVol to HX 40 CSIupgrade does not work FlexVol continuesto work

CSCvp26319

35(2a)Disable rebalance until node healsthen reenable rebalance

7-node cluster healing very slowlybecause rebalance change AMS reqs areblocking healing process

CSCvp62512

35(2a)NAWith moderate cluster storage capacityusage under site failover cluster clustermay experience an all paths (APD) downcondition

CSCvq49412

40(1a)NAThis product includes Third-partySoftware that is affected by thevulnerabilities identified by the followingCommon Vulnerability and Exposures(CVE) IDs

CVE-2016-0762 CVE-2016-6797CVE-2016-6816 CVE-2016-8735CVE-2017-5647 CVE-2017-12615CVE-2017-12616 CVE-2017-12617CVE-2017-7674 CVE-2018-1304CVE-2018-8014 CVE-2018-1336CVE-2018-8034 CVE-2018-11784CVE-2019-0232

CSCvp36364

35(2d)NAstcli cluster info command needs toprovide UCSM VIP address Currently itshows ucsm-hostcom

CSCvq11456

35(2a)No workarounds currently Thestale entries do not affect the dayto day operations of HyperFlex

HyperFlex upgrade validation failedbecause cluster has extraneous stNodes ininternal database

CSCvq32530

35(2a)NAWhen not in maintenancemode all drivesappear in HX Connect and get updated asexpected

When in maintenance mode extra drivesare shown in HX Connect with no slotnumbers

CSCvq42378

35(2b)Contact TAC to cleanup duplicatenode

Nodes duplicated in HX Connect aftercluster recreate

CSCvq61825

Release Notes for Cisco HX Data Platform Release 4058

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)NACurrently HyperFlex installs on Hyper-Vdo not contain the stcli security whitelistcommandset

CSCvq66245

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

Release Notes for Cisco HX Data Platform Release 4059

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)[Hyper-V] Converged node expansionfails for incoming node with VIC 14xx

CSCvq39523

Release Notes for Cisco HX Data Platform Release 4060

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

1 Set Jumbo frame inWindowsHypervisor for incomingnode

Get-NetAdapter storage-data-a|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

Get-NetAdapter storage-data-b|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

2 Verify all interfaces are setcorrectly to support Jumbo frames

Get-NetAdapter storage |Get-NetAdapterAdvancedProperty| RegistryKeyword -matchjumbo | ft -auto

You should receive the followingmessage

Name DisplayName DisplayValue

RegistryKeyword RegistryValue

---- ----------- ------------

--------------- -------------

vswitch-hx-storage-data Jumbo

Packet 9014 Bytes

JumboPacket 9014

storage-data-b Jumbo Packet

Bytes 9014 JumboPacket

9014

storage-data-a Jumbo Packet

Bytes 9014 JumboPacket

9014

3 Reboot only the incoming nodeor the node being expanded

4 Retry Cluster Expansion in

Release Notes for Cisco HX Data Platform Release 4061

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

Stretched Cluster (Availableon ESX Only)

Microsoft Hyper-VVMware ESXiNode

NA(requiresEnterpriseHXDP-PLicense)

NA(requiresEnterpriseHXDP-PLicense)

NA(requiresEnterpriseHXDP-PLicense)

Convergednodes3-16(12TBHDDoption isnotsupportedforHyperV)

Compute-onlynodes0-16

Convergednodes3-16

Compute-onlynodes0-16

NA(requiresEnterpriseHXDP-PLicense)

M4Convergednodes 3

M5Convergednodes23or 4

RequiresHXDP-ELicense

Convergednodes3-16(12TBdrive islimited toamaximumof 8nodes)

Compute-onlynodes0-16(12TBdrive islimited toamaximumof 8nodes)

Convergednodes3-32(76TBdriveconfigsarelimited tomaximumof 12drivesnodeon AF240)

Compute-onlynodes0-32

HXDP-SLicensedNodeLimits

11 ratioofHXDP-StoComputeonlynodes

(MinmdashMax)

Convergednodes2-16 perSite

Compute-onlynodes0-21 perSite

(up tomaxclustersize)

Convergednodes2-8 perSite

Compute-onlynodes0-16 perSite

(up tomaxclustersize)

Convergednodes2-16 perSite

Compute-onlynodes0-21 perSite

(up tomaxclustersize)

Convergednodes3-16(12TBHDDoption isnotsupportedforHyperV)

Compute-onlynodes0-16

Convergednodes3-16

Compute-onlynodes0-16

Convergednodes3-32

ComputeOnlynodes0-32

(up tomaxclustersize)

Convergednodes 3

(requiresHXDP-ELicense)

Convergednodes3-16(12TBdrive islimited toamaximumof 8nodes)

Compute-onlynodes0-32(12TBdrive islimited toamaximumof 16nodes)

Convergednodes3-32(76TBdriveconfigsarelimited tomaximumof 12drivesnodeon AF240)

Compute-onlynodes0-32

(up tomaxclustersize)

HXDP-PLicensedNodeLimits

12 ratioofHXDP-PtoComputeonlynodes

(MinmdashMax)

24 perSite 64percluster

24 perSite 48percluster

32 perSite 64percluster

32326434864MaxClusterSize

Release Notes for Cisco HX Data Platform Release 4018

Cisco HX Data Platform Storage Cluster Specifications

Stretched Cluster (Availableon ESX Only)

Microsoft Hyper-VVMware ESXiNode

212121111121mdash2121MaxComputetoConvergedratio

NoExpansion

Requires Enterprise license

Requires uniform expansion across both sites

Guidelines and Limitationsbull Starting with release 40(2a) SCVM is no longer needed on a Compute node

bull HX REST API Access Token Management ndash Applications leveraging HX REST APIs should re-useaccess tokens when making API calls Once obtained using the AAA Obtain Access Token API accesstokens are valid for 18 days (1555200 seconds) In addition AAA enforces rate limiting on ObtainAccess Token API requests in a 15 minute window auth can be invoked (successfully) a maximum of5 times A user is allowed to create a maximum of 8 unrevoked tokens Subsequent call to auth willautomatically revoke the oldest issued token tomake room for the new token Amaximum of 16 unrevokedtokens can be present in system In order to prevent brute-force attacks after 10 consecutive failedauthentication attempts a user account is locked for a period of 120 secondsFor more information seeCisco HyperFlex Systems REST API Reference guide

HxConnect makes use of AAA Authentication REST API for login and the above rate limit applies toHxConnect also

Upgrade Guidelines

The following list is a highlight of critical criteria for performing an upgrade of your HyperFlex system

bull Unsupported HX Data Platform 17x 18x 20 21x 25x and 26x clustersmdashUsers from anyversion prior to 26(1a) must step through an intermediate version before upgrading to 40 or later releasesIf you need to upgrade your environment from a Cisco HyperFlex HX Data Platform software releasethat is past the last date of support to the latest suggested release on the Cisco Software Download sitesee Cisco HyperFlex Systems Upgrade Guide for Unsupported Cisco HXReleases For more informationsee the Software Advisory for CSCvq66867 WARNING Only Use HXDP 26(1e) Upgrade PackageWhen Upgrading From HXDP 18(1a)-18(1e)

bull Hypercheck Health Check UtilitymdashCisco recommends running this proactive health check utility onyour HyperFlex cluster prior to upgrade These checks provide early visibility into any areas that mayneed attention andwill help ensure a seamless upgrade experience For more information see the HyperFlexHealth amp Pre-Upgrade Check Tool TechNote for full instructions on how to install and run Hypercheck

bull vSphere 67 Software AdvisorymdashDo not upgrade to Cisco HX Data Platform Release 40(1a) whenrunning ESXi 67U1 EP06 (build 11675023) Do not upgrade to 67U1 EP06 (build 11675023) ifrunning Cisco HX Data Platform Release 40(1a) See the Software Advisory CSCvo56350 for furtherdetails

Release Notes for Cisco HX Data Platform Release 4019

Guidelines and Limitations

The software build version posted at release will override any other local versions

bull Required vCenter upgrademdashFor enhanced security Cisco HX Data Platform Release 35(1a) or laterrequires the use of TLS 12 Therefore vCenter must be upgraded to 60 U3f or later before upgradingto Cisco HX Data Platform Release 35 or later In addition ESXi should be upgraded as required tomeet HX Data Platform compatibility requirements

bull Minimum HXDP version for upgrademdashHXData Platform clusters running 26(1a) or later may upgradedirectly to 40 using the HX Connect UI

bull Cluster ReadinessmdashEnsure that the cluster is properly bootstrapped and the updated plug-in is loadedbefore proceeding Manual cluster bootstrap is required for upgrade from a pre-35 release

bull Cluster ReadinessmdashEnsure that the cluster is properly bootstrapped and the updated plug-in is loadedbefore proceeding Manual cluster bootstrap is required for HX releases earlier than 35(1a) For moreinformation see theManual Bootstrap Upgrade Process in the Cisco HyperFlex Systems Upgrade Guidefor VMware ESXi Release 40 Do not skip this cluster bootstrap step it is required for all upgradesuntil HX Release 35(1a) Auto bootstrap is supported beginning with HX release 35(1a) For moreinformation see the Auto Bootstrap Upgrade Process from HX Connect UI in the Cisco HyperFlexSystems Upgrade Guide for VMware ESXi Release 40

Manual bootstrap is not supported on Intersight clusters

bull Initiating Upgrade―Use the HX Connect UI or CLI stcli commands when upgrading from 25(1a)or later releases Use either the CLI stcli commands or the HX Data Platform Plug-in to the vSphereWeb Client when upgrading from a pre-25(1a) release The vCenter plug-in should not be used forupgrades starting with the 25(1a) release The vCenter plug-in should not be used for upgrades startingwith the 25(1a) release

If the current cluster version is at 35(1a) or above you do not need to use the stcli command Directupgrade to 40 is possible

bull Complete your Upgrade―The self-healing (or rebalance) capability is disabled temporarily during theupgrade window If the upgrade fails you should complete the upgrade as soon as possible

bull ESXi and HXDP Compatibility―Ensure your cluster is running a compatible version of ESXi basedon the running the HXData Platform version (see the section Software Requirements for VMware ESXi)ESXi compatibility is determined by the major version and update release of ESXi It is generally bestto upgrade HXDP and ESXi together if combining the upgrade operations into a single optimized rebootWhen running a split upgrade first upgrade the HX Data Platform then proceed to upgrade ESXi

bull If Upgrading to vSphere 65

bull Certain cluster functions such as native and scheduled snapshotsReadyClones and Enter or Exit HX Maintenance Mode will not operatefrom the time the upgrade is started until the HX Data Platform upgrade iscomplete

bull After upgrading ESXi using the offline zip bundle use the ESX ExitMaintenance Mode option The HX Exit Maintenance Mode option doesnot operate in the vSphere Web Client until the HX Data Platform upgradeis complete

Note

Release Notes for Cisco HX Data Platform Release 4020

Guidelines and Limitations

bull vSphere 60 UpgradesmdashUsers on vSphere 60 migrating to 65 upgrade components in the followingorder

1 Upgrade HX Data Platform and UCS firmware

2 Upgrade HX Data Platform and ESXi

3 Upgrade HX Data Platform only first then upgrade ESXi or UCS firmware or both

bull M4 Server Firmware UpgradesmdashUpgrade server firmware to ensure smooth operation and to correctknown issues Specifically newer SAS HBA firmware is available in this release and is recommendedfor long-term stability

bull Users are encouraged to upgrade to 31(3c) C-bundle or later whenever possible

bull Users running C-bundle versions before 31(2f) must upgrade server firmware by performing acombined upgrade of UCS server firmware (C-bundle) to 31(3c) or later and HX Data Platform to25 Do not split the upgrade into two separate operations

bull If the cluster is already on 31(2f) C-bundle or later you may perform an HX Data Platform onlyor combined upgrade as required

bull M5 Server Firmware UpgradesmdashM5 generation servers must run firmware version 32(2d) or later

bull Firmware Downgrades mdashDowngrading UCSM from the HX-installer is not supported

bull M4M5 Mixed DomainsmdashA mixed domain occurs when a new separate M5 cluster is installed underthe same UCS domain that contains existing M4 clusters Under these conditions orchestrated UCSserver firmware upgrade will not operate until Cisco HX Data Platform Release 26 or later is installedon the M4 clusters Therefore it is best practice to first upgrade UCS server firmware to the latest 31(3)or 32(2) patch release before adding a new M5 cluster to the existing UCS domain Additionally any17 HX Data Platform clusters must first be upgraded before adding any new M5 clusters to the samedomain

bull Maintenance WindowmdashIf upgrading both HX Data Platform and UCS firmware you can select eithera combined or split upgrade through the vSphere HX Data Platform Plug-in depending on the length ofthe maintenance window Cisco UCSManager infrastructure upgrade is only supported using AutoInstalland the direct server firmware upgrade should be performed only through the upgrade orchestrationframework provided by the HX Data Platform Plug-in

bull Unsupported Self-Encrypting Drives (SEDs)mdashIf adding or replacing self-encrypting drives (SEDs)that have been recently qualified in newer versions of HX Data Platform insert the new drives only afterupgrading HX Data Platform to a compatible version All drives must be SED drives mixing SED andnon-SED is not supported

bullbull Enabling External Host AccessmdashWith Cisco HX Data Platform Release 40(1a) port 445 on themanagement network is blocked for enhanced security Note that prior to 40 port 445 port was openenabling external host access If you are upgrading to 40(1a) from a prior release and would like tocontinue external host access you can use a utility to open select hosts For more information aboutenabling external host access see the ConfiguringHyperFlex Share to SCVMM section in the InstallationGuide for Microsoft Hyper-V

Release Notes for Cisco HX Data Platform Release 4021

Guidelines and Limitations

Mixed Cluster Expansion Guidelinesbull Expanding existing M4 cluster with M5 converged nodes is supported

bull Expanding existing M5 cluster with M4 converged nodes is not supported

bull Expanding existing mixed M4M5 cluster with M4 or M5 converged nodes is supported

bull Adding any supported compute-only nodes is permitted with all M4 M5 and mixed M4M5 clustersusing the HX Data Platform 26 or later Installer Some example combinations are listed here manyother combinations are possibleExample combinationsExpand mixed M4M5 cluster with compute-only B200 C220 C240 M4M5Expand M4 cluster with compute-only B200 M5 C220 M5 C240M5

bull Only expansion workflow is supported to create a mixed cluster Initial cluster creation with mixedM4M5 servers is not supported

bull All M5 servers must match the form factor (220240) type (HybridAF) security capability (Non-SEDonly) amp disk configuration (QTY capacity and non-SED) of the existing M4 servers

bull HX220-M5 will use a maximum of 6 capacity disks (2 disk slots to remain empty) when mixedwith HX220-M4

bull HX Edge SED LFF Hyper-V and Stretched Clusters do not support mixed M4 and M5 clusters

Security FixesThe following security issues are resolved

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-13132 CVE-2019-9924CSCvq6313840(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-11719 CVE-2019-11727CVE-2019-11729

CSCvq7124040(2a)

Release Notes for Cisco HX Data Platform Release 4022

Mixed Cluster Expansion Guidelines

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-1125CSCvr0633940(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2015-9383 CVE-2018-14498CVE-2018-20406 CVE-2018-20852CVE-2019-10160 CVE-2019-13117CVE-2019-13118 CVE-2019-14287CVE-2019-14973 CVE-2019-15903CVE-2019-17546 CVE-2019-18197CVE-2019-18218 CVE-2019-5010CVE-2019-5094 CVE-2019-5481CVE-2019-5482 CVE-2019-9636CVE-2019-9740 CVE-2019-9947CVE-2019-9948

CSCvs0609440(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2017-13168 CVE-2017-18174CVE-2017-18216 CVE-2018-10876CVE-2018-10877 CVE-2018-10878CVE-2018-10879 CVE-2018-10880CVE-2018-10881 CVE-2018-10882CVE-2018-10902 CVE-2018-10938CVE-2018-12233 CVE-2018-12896CVE-2018-13053 CVE-2018-13094CVE-2018-13096 CVE-2018-13405CVE-2018-13406 CVE-2018-14609CVE-2018-14617 CVE-2018-14633CVE-2018-14734 CVE-2018-15572CVE-2018-15594 CVE-2018-16276CVE-2018-16658 CVE-2018-17182CVE-2018-17972 CVE-2018-18021CVE-2018-18690 CVE-2018-18710CVE-2018-6554 CVE-2018-6555CVE-2018-9363

CSCvp6501940(2a)

This vulnerability is due to insufficientHTML iframe protection An attackercould exploit this vulnerability bydirecting a user to an attacker-controlledweb page that contains amaliciousHTMLiframe A successful exploit could allowthe attacker to conduct clickjacking orother clientside browser attacks

NACSCvo9851640(2a)

Release Notes for Cisco HX Data Platform Release 4023

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerability is due to insufficientauthentication for the statistics collectionservice An attacker could exploit thisvulnerability by sending properlyformatted data values to the statisticscollection service of an affected deviceA successful exploit could allow theattacker to cause the web interfacestatistics view to present invalid data tousers

NACSCvj9558440(2a)

The vulnerability is due to insufficientCSRF protections for the web UI on anaffected device An attacker could exploitthis vulnerability by persuading a user ofthe interface to follow a malicious linkA successful exploit could allow theattacker to perform arbitrary actions withthe privilege level of the affected user

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvp2434340(2a)

Vulnerabilities with a version of cJSONidentified by CVE-2019-11834 andCVE-2019-11835

For more information see the relatedCisco Security Advisory

CVE-2019-11834 CVE-2019-11835CSCvq1994940(2a)

HX ESXi image patches for MCEPSCand TAA vulnerabilities with VMwareESXi identified by CVE-2018-12207 andCVE-20190-11135

For more information see the relatedCisco Security Advisory

CVE-2018-12207 CVE-2019-11135CSCvr5439840(2a)

CP networking vulnerabilities affectingthe Linux kernel identified byCVE-2019-11477 CVE-2019-11478CVE-2019-11479

For more information see the relatedCisco Security Advisory

CVE-2019-11477 CVE-2019-11478CVE-2019-11479

CSCvq1954640(2a)

Qualification of Microsoft securitypatches for vulnerabilities withMicrosoftHyper-V hypervisor identified withCVE-2018-12207 CVE-2019-11135

For more information see the relatedCisco Security Advisory

CVE-2018-12207 CVE-2019-11135CSCvr5439940(2a)

Release Notes for Cisco HX Data Platform Release 4024

Security Fixes

DescriptionCVEDefect IDRelease

OpenSSH vulnerabilities identified byCVE-2019-6111

For more information see the relatedCisco Security Advisory

CVE-2016-10708 CVE-2018-15473CVE-2018-20685 CVE-2019-6109CVE-2019-6111

CSCvp7646340(2a)

Multiple vulnerabilities associated withOpenSSL and LibNSS

For more information see the relatedCisco Security Advisory

CVE-2016-2105 CVE-2016-2106CVE-2016-2107 CVE-2016-2109CVE-2016-2176 CVE-2016-2177CVE-2016-2178 CVE-2016-2179CVE-2016-2180 CVE-2016-2181CVE-2016-2182 CVE-2016-2183CVE-2016-6302 CVE-2016-6303CVE-2016-6304 CVE-2016-6305CVE-2016-6306 CVE-2016-6307CVE-2016-6308 CVE-2016-7055CVE-2016-8610 CVE-2017-3731CVE-2017-3732 CVE-2017-3735CVE-2017-3736 CVE-2017-3737CVE-2017-3738 CVE-2018-0495CVE-2018-0732 CVE-2018-0734CVE-2018-0735 CVE-2018-0737CVE-2018-0739 CVE-2018-12384CVE-2018-12404 CVE-2018-5407CVE-2019-1559

CSCvp6667940(2a)

libpng vulnerability identified byCVE-2016-10087

For more information see the relatedCisco Security Advisory

CVE-2016-10087CSCvp6655540(2a)

Vulnerabilities associated with libjpg andlibpng

For more information see the relatedCisco Security Advisory

CVE-2014-9092 CVE-2016-3616CVE-2017-15232 CVE-2018-11212CVE-2018-11213 CVE-2018-11214CVE-2018-1152 CVE-2018-13785

CSCvp3458640(2a)

Vulnerabilities associated with Glibidentified by CVE-2018-16428 andCVE-2018-16429

For more information see the relatedCisco Security Advisory

CVE-2018-16428 CVE-2018-16429CSCvp3120740(2a)

Multiple vulnerabilities associated withcurl expat python 27 python 35 36and 37 freetype and giflib

For more information see the relatedCisco Security Advisory

CVE-2019-15133 CVE-2019-15903CVE-2019-5010 CVE-2019-5481CVE-2019-5482 CVE-2019-9636CVE-2019-9740 CVE-2019-9947CVE-2019-9948

CSCvr3690340(2a)

Release Notes for Cisco HX Data Platform Release 4025

Security Fixes

DescriptionCVEDefect IDRelease

Multiple third party softwarevulnerabilities

For more information see the relatedCisco Security Advisory

CVE-2019-14379 CVE-2019-12384CVE-2019-14439

CSCvq9203240(2a)

Vulnerabilities associated with elfutils

For more information see the relatedCisco Security Advisory

CVE-2018-16062 CVE-2018-16402CVE-2018-16403 CVE-2018-18310CVE-2018-18520 CVE-2018-18521CVE-2019-7149 CVE-2019-7150CVE-2019-7665

CSCvq4325040(2a)

Vulnerabilities associated with vim

For more information see the relatedCisco Security Advisory

CVE-2017-5953 CVE-2019-12735CSCvq4323040(2a)

Vulnerabilities associated with sqlite3

For more information see the relatedCisco Security Advisory

CVE-2016-6153 CVE-2017-10989CVE-2017-13685 CVE-2017-2518CVE-2017-2519 CVE-2017-2520CVE-2018-20346 CVE-2018-20505CVE-2018-20506 CVE-2019-8457CVE-2019-9936 CVE-2019-9937

CSCvq4321340(2a)

Vulnerabilities associated with glib20

For more information see the relatedCisco Security Advisory

CVE-2018-20843CSCvq4320940(2a)

Vulnerabilities associated with expat

For more information see the relatedCisco Security Advisory

CVE-2018-20843CSCvq4320540(2a)

Vulnerabilities associated with bzip2

For more information see the relatedCisco Security Advisory

CVE-2016-3189 CVE-2019-12900CSCvq4319440(2a)

Mulitple Vulnerabilities associated wtihNodeJS

For more information see the relatedCisco Security Advisory

CVE-2018-12115 CVE-2018-0734CVE-2018-5407 CVE-2018-12120CVE-2018-12121 CVE-2018-12122CVE-2018-12123 CVE-2018-12116CVE-2019-5737 CVE-2019-5739

CSCvq1069440(2a)

Vulnerabilities associated with PalletsJinja strformat_map and identified byCVE-2019-10906 and CVE-2016-10745

For more information see the relatedCisco Security Advisory

CVE-2019-10906 CVE-2016-10745CSCvq1038840(2a)

Release Notes for Cisco HX Data Platform Release 4026

Security Fixes

DescriptionCVEDefect IDRelease

Multiple Security Vulnerabilities

For more information see the relatedCisco Security Advisory

CVE-2014-8501 CVE-2014-9939CVE-2015-9262 CVE-2016-10087CVE-2016-2226 CVE-2016-4487CVE-2016-4488 CVE-2016-4489CVE-2016-4490 CVE-2016-4491CVE-2016-4492 CVE-2016-4493CVE-2016-6131 CVE-2018-10963CVE-2018-13785 CVE-2018-17100CVE-2018-17101 CVE-2018-18557CVE-2018-18661 CVE-2018-7456CVE-2018-8905

CSCvq0917840(2a)

Vulnerabilities associated withlibseccomp

For more information see the relatedCisco Security Advisory

CVE-2019-9893CSCvq0756840(2a)

Vulnerabilities associated withgdk-pixbuf

For more information see the relatedCisco Security Advisory

CVE-2017-12447CSCvq0675540(2a)

Vulnerabilities associated withpython-crypto

For more information see the relatedCisco Security Advisory

CVE-2018-6594CSCvp9381740(2a)

Vulnerabilities associated with wget

For more information see the relatedCisco Security Advisory

CVE-2018-20483 CVE-2019-5953CSCvp8672140(2a)

Multiple vulnerabilities associated withPython Crypto

For more information see the relatedCisco Security Advisory

CVE-2018-6594CSCvp6674840(2a)

Vulnerabilities associated withsensible-utils

For more information see the relatedCisco Security Advisory

CVE-2017-17512CSCvp6673440(2a)

Multiple vulnerabilities associated withPython

For more information see the relatedCisco Security Advisory

CVE-2018-1000030CSCvp6668940(2a)

Release Notes for Cisco HX Data Platform Release 4027

Security Fixes

DescriptionCVEDefect IDRelease

Multiple patch vulnerabilities identifiedby CVE-2016-10713

For more information see the relatedCisco Security Advisory

CVE-2016-10713CVE-2018-1000156 CVE-2018-6951

CSCvp6667240(2a)

Vulnerabilities associatedwith Little CMSidentified by CVE-2016-10165

For more information see the relatedCisco Security Advisory

CVE-2016-10165 CVE-2018-16435CSCvp6666440(2a)

Vulnerabilities associated with TenableScan identified byCVE-2011-5325CVE-2019-11068CVE-2016-7076CVE-2017-1000368

For more information see the relatedCisco Security Advisory

CVE-2011-5325 CVE-2016-7076CVE-2017-1000368CVE-2019-11068

CSCvp6474640(2a)

Vulnerabilities associated with libxml2294 and earlier as used in XMLSec1223 and earlier and other products andidentified by CVE-2016-9318

For more information see the relatedCisco Security Advisory

CVE-2016-9318 CVE-2017-16932CVE-2017-18258 CVE-2018-14404CVE-2018-14567

CSCvp3479240(2a)

Vulnerabilities associated with LFTPRemote File Names Unauthorized AccessVulnerability and identified byCVE-2018-10916

For more information see the relatedCisco Security Advisory

CVE-2018-10916CSCvp2926640(2a)

Vulnerabilities associated withCryptographyDeprecationWarning signerand verifier have been deprecated

For more information see the relatedCisco Security Advisory

CVE-2018-7750CSCvo3409740(2a)

Multiple Vulnerabilities in python python27 35

For more information see the relatedCisco Security Advisory

CVE-2019-16056CSCvr3979340(2a)

Release Notes for Cisco HX Data Platform Release 4028

Security Fixes

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toexecute commands as the root user

The vulnerability is due to insufficientinput validation An attacker could exploitthis vulnerability by connecting to thecluster service manager and injectingcommands into the bound process Asuccessful exploit could allow the attackerto run commands on the affected host asthe root user

Cisco has released software updates thataddress this vulnerability There areworkarounds that address thisvulnerability

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvq2417640(1b)35(2g)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

The vulnerability is due to an unprotectedlistening interface An attacker couldexploit this vulnerability by connectingto the listening interface and injectingcommands to the bound process Anexploit could allow the attacker to runcommands on the affected host as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvj9560640(1b)35(2g)

Release Notes for Cisco HX Data Platform Release 4029

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerabilities associated with theJVM18U121memory leak found duringVC alarms (concurrent 40 calls usingREST API)

CVE-2017-10053 CVE-2017-10067CVE-2017-10074 CVE-2017-10078CVE-2017-10081 CVE-2017-10087CVE-2017-10089 CVE-2017-10090CVE-2017-10096 CVE-2017-10101CVE-2017-10102 CVE-2017-10107CVE-2017-10108 CVE-2017-10109CVE-2017-10110 CVE-2017-10111CVE-2017-10115 CVE-2017-10116CVE-2017-10118 CVE-2017-10135CVE-2017-10176 CVE-2017-10193CVE-2017-10198 CVE-2017-10243CVE-2017-10274 CVE-2017-10281CVE-2017-10285 CVE-2017-10295CVE-2017-10345 CVE-2017-10346CVE-2017-10347 CVE-2017-10348CVE-2017-10349 CVE-2017-10350CVE-2017-10355 CVE-2017-10356CVE-2017-10357 CVE-2017-10388CVE-2017-3509 CVE-2017-3511CVE-2017-3526 CVE-2017-3533CVE-2017-3539 CVE-2017-3544CVE-2018-2579 CVE-2018-2582CVE-2018-2588 CVE-2018-2599CVE-2018-2602 CVE-2018-2603CVE-2018-2618 CVE-2018-2629CVE-2018-2633 CVE-2018-2634CVE-2018-2637 CVE-2018-2641CVE-2018-2663 CVE-2018-2677CVE-2018-2678 CVE-2018-2783CVE-2018-2790 CVE-2018-2794CVE-2018-2795 CVE-2018-2796CVE-2018-2797 CVE-2018-2798CVE-2018-2799 CVE-2018-2800CVE-2018-2814 CVE-2018-2815CVE-2018-2952 CVE-2018-3136CVE-2018-3139 CVE-2018-3149CVE-2018-3150 CVE-2018-3169CVE-2018-3180 CVE-2018-3183CVE-2018-3214 CVE-2019-2422

CSCvo8899740(1b)

Tomcat and Nginx logs are not beingcollected in the support bundle generatedthrough HX Connect in Release 35

NACSCvm5803140(1b)

The vulnerabilities associated with thelibmspack software package versionincluded in Cisco HX Data Platform

CVE-2018-18584

CVE-2018-18585

CSCvn3511940(1a)

Release Notes for Cisco HX Data Platform Release 4030

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerabilities associated withFasterXML Jackson-Databind TimeValueField Denial of Service

CVE-2018-14719

CVE-2018-14720

CVE-2018-1000873

CVE-2018-14721

CVE-2018-19360

CVE-2018-19362

CVE-2018-19361

CVE-2018-14718

CSCvn8228240(1a)

The vulnerabilities associated with theOpenSSL Protocol software packageversion included in Cisco HX DataPlatform

CVE-2013-3587CSCvo0505440(1a)

The vulnerabilities associated withThird-Party Software Denial of Service

CVE-2018-16487

CVE-2018-19361

CSCvo2781840(1a)

The vulnerabilities associated withLinuxkernel for Ubuntu 1710

CVE-2018-1092

CVE-2018-7492

CVE-2018-8087

CVE-2018-1068

CVE-2018-8781

CSCvm5314935(2a)

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)25(1d)stcli services dns remove should remove the DNS serverinfo from the interface files

CSCvs70967

40(2b)40(1b)HX 40(1b) new deployment from Intersight productioncloud

User receives following error while enabling CSI(Kubernetes integration)

Failure occurred during volume_access_enable Errorwas Nonzero exit code 1

CSCvt10522

40(2b)40(2a)Receive the following alertevent in HX Connect HXController VM HOSTNAME one or more configuredDNS servers not responding

CSCvt13947

Release Notes for Cisco HX Data Platform Release 4031

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)40(2a)API UI CLI show few Drive slots Empty after upgradeto 402a

The drives are seen in the sysmtool lsscsi and stclicommand outputs

The cluster is healthy with no errors

CSCvt14914

40(2b)35(2d)After successfully changing the DNS server on the HXcontroller we still can see the original DNS entry thatwas added during the deployment

CSCvs69154

40(2b)35(2e)HyperFlex Connect and VCenter show APD alarms forHX and non HX datastores This implies there is an issuewith HyperFlex when there may not be

CSCvs30080

40(2b)40(2a)An issue where all the disks in the node get locked afterrebooting the node

We successfully unlocked using sed-clientsh -Ucommand but wanted to test with another reboot anddrives locked again

CSCvs74286

40(2b)40(2a)When running stcli license commands on HyperFlexthe following error is seen

rootSpringpathControllertmp stcli license show all

Show smart licensing failed

Smart Agent is not ready please wait a minute and tryagain

CSCvt13929

40(2b)40(2a)After upgrading to HX 40(2a) the first RoboEdge nodethat gets rebooted then ignores persistent drivesPIDHX-HD24TB10K4KN

Cluster will be in degraded state and have reducedcapacity

CSCvt20144

40(2b)35(2b)Zookeeper fails to start while Exhibitor is runninghowever echo srvr returns nothing

CSCvs21562

40(2b)40(2a)When performing an HyperFlex upgrade a validationwarning may occur due to the host not having EnterprisePlus or Enterprise hypervisor licensing

Upgrade Validation Warning

ESXi host esx1labtest should be configured withVMware Enterprise license for upgrade to continue

CSCvs91787

Release Notes for Cisco HX Data Platform Release 4032

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)35(2g)Cluster expansion fails at Config Installer stage whenthe root and admin password for the storage controller(SCVM) are different

CSCvs69317

40(2b)35(2g)Panic while upgrading ESXiCSCvt06983

40(2b)40(2a)A cluster node running may hang in the Linux kernelThis is classified as an oops and a deviation from theexpected behavior

CSCvs54285

40(2b)35(2g)Rebalance failing on 10+10 node HX 35(2g) stretchcluster

CSCvs69007

40(2b)35(2d)The cluster becomes inaccessible to the IOVisorCSCvr54687

40(2b)40(2a)Panic on the storage controllerCSCvt61297

40(2b)40(2a)The size of support bundle is very large when collectedwith storfs-support command

CSCvt63306

40(2b)40(2a)storfs crash associated withFileSystemUsageWarningEvent orFileSystemUsageAlertEvent due to high usage ofvarstv above 80

CSCvt72807

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

40(2a)35(2d)In normal scenarios HX upgrade gets pendingACK afterupdating service profile In this case UCS only upgradewas stalled because after updating service profile HXupgrade was not getting pending ACK and it was alwayswaiting state

CSCvo39912

40(2a)35(1a)When using motherBoardReplace-12 to clean ZK fromold stNodepNodes resulted in unmounted datastore andmaking the size of the Hyperflex Datastores 0 resultingin all VMs in the cluster going offline

CSCvq39471

40(2a)26(1e)Upgrading cluster and getting error

Failed upgrade validations Checking vCenterconfiguration Reason Upgrade validations failed DRSFault Insufficient resources to satisfy configured failover

CSCvm77294

Release Notes for Cisco HX Data Platform Release 4033

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)The cluster expand fails on a node with DR replicationconfigured

When a HX cluster which has DR replication configuredis expanded we see the installer UI pulling in thereplicationVLAN information instead of themanagementVLAN information Even if we change that informationto the correct mgmt VLAN id and name it it does notseem to work as the node is configured with the VLANof the replicationVLAN in ESXi This leads to the failureof the node add with the host unreachable error

CSCvo70650

40(2a)35(1a)Customer reported that server didnt complete thefirmware upgrade automatically one by one

The user manually needs to put the host into maintenancemode and they manually acknowledged the pendingrequests to finish the UCS firmware upgrade

Maintenance policy is set to default (User-ack) as perthe design

CSCvo91624

40(2a)35(1i)When the cluster is in Failed state and stcli noderemove is attempted the output shows successful thoughthe node failed to get removed from the cluster

CSCvo93017

40(2a)35(1a)HyperFlex cluster upgrade may fail during validationwith a DRS Validation failed error

CSCvp31021

40(2a)35(2b)HX cluster expansion will fail with the following errormessage

MAC address pool configuration failure150[ErrorDescription] bad address block range definitioncollision

CSCvp58318

40(2a)35(2b)Memory usage by carbon cacheCSCvq34873

40(2a)35(2g)Upgrading ESXi cluster fails with error Nodemaintenance mode failed

CSCvr03240

40(2a)35(2e)storfs process does not automatically start on Exit HXMaintenance Mode or other tasks which restart StorageController

CSCvr88978

40(2a)35(2a)Post install script fails with message

Failed to execute ipmitool on HX node

CSCvp10707

40(2a)40(1a)HyperFlex expansionworkflow doesnt pull VLANnamecorrectly

CSCvp46539

Release Notes for Cisco HX Data Platform Release 4034

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)During HX cluster deployment - Validate ClusterCreation phase - HX Installer might fail with the error

fromvarlogspringpathstDeploylog

CSCvq45087

40(2a)35(2b)HX Installer fails to login to the SOL for configuringESX hosts

CSCvq91380

40(2a)35(2d)If DRS is enabled with configuration parameters otherthen fully automated ie manual or partially automatedthan this will change to fully automated during clusterexpansion

CSCvr44222

40(2a)35(1a)One of the validation task name has a typo The taskname is Validating node for mixed of SED and non-SEDdisks

It should be Validating node for mix of SED andnon-SED disks

CSCvp82175

40(2a)30(1i)This combination of using smaller NVMe drives is notsupported in expand operation For example if the drivesare only 375GB and they cannot be added to an existingcluster with larger caching SSD

CSCvo12359

40(2a)35(1a)HyperFlex includes a version of OpenSSL that is affectedby the vulnerabilities identified by the followingCommon Vulnerability and Exposures

CVE-2018-0495

CSCvp66679

40(2a)30(1e)SDK invoke exception when taking a native snapshot ona VM with GPU pci passthrough attached

CSCvo00511

40(2a)30(1i)Node replacement script fails due to EAM errorCSCvo62867

40(2a)35(1a)While pairing a cluster (HX release gt= 35) and cluster(HX release lt 35) remote replication network test failson the cluster (HX release lt 35)

CSCvo79760

40(2a)30(1b)Fixed the issue in the latest support-workflow bundleThis should not be hit anymore

CSCvo87061

40(2a)35(2a)On a 3 node cluster the MBreplace script fails as thescript looks for HEALTHY cluster status

CSCvo87080

40(2a)35(2a)MbReplace script (tar) hangs when running on 352b or40

CSCvp12359

Release Notes for Cisco HX Data Platform Release 4035

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)HX replication cleanup failing with error NA 35(2a) -INFODR state is not clean

CSCvp63958

40(2a)40(1a)The check_stig_parameters API incorrectly shows thecompliance state of the cluster as not STIG compliant

CSCvp66277

40(2a)35(2c)The snapshot creation on CBT enabled VM fails witherror Failed in vmreparent vmkfstools clone1

CSCvq06952

40(2a)35(2a)HyperFlex GUI when setting up remote DR-replicationpartnerpeer should show remote replication partner TCPport reachability test results

This will provide quick upfront connectivity results thatfrequently takes customer a while to manually check andconfirm

If reachability tests fail an info window and HX alertshould trigger that list the destination peer IP and thespecific port(s) that could not be reache

CSCvo60587

40(2a)35(2b)If you have more than one IP pool configured for areplication network and local replication network testsare failing you might be running into this issue

CSCvr67130

40(2a)35(2a)Skip task option in MBreplace and Software redeployscripts The scripts should have the option to skip tasksas user inputs and not by modifying the json file

CSCvp05204

40(2a)40(1a)Error noticed on SCVM Console print_req_error IOerror dev fd0 sector 0 on HyperV

It doesnt come up on SSH session

CSCvq34357

40(2a)30(1d)The VMNetwork switch is already created as part of theinstall process

Ideally the ip address assignment should be done duringthe initial install process as well No post install stepsshould be required altrnatively it could be done as partof a post install script

CSCvk36222

40(2a)25(1c)storfs service impacted during HX upgrade from HX21(1) to 25(1c)

HX node which is not being upgraded storfs servicestopped causing outage

CSCvg53223

40(2a)35(1b)HX Stretch Cluster Witness VM reverts to DHCP onreboot

CSCvp37536

40(2a)35(2b)HyperFlex Stretch Cluster Witness does not getprogrammed with an NTP server

CSCvp46578

Release Notes for Cisco HX Data Platform Release 4036

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2b)HyperFlex cluster does not heal after maintenance onnodeserver

CSCvr18528

40(2a)40(2a)High percentage of packet loss on the eth1 data interface9K MTU

CSCvr97089

40(2a)35(2f)The HyperFlex cluster healing state is stuck at 87 andis unable to progress

CSCvr16760

40(2a)40(2a)In some circumstances two replications can occur forthe same VM

From the GUI the replication status for this VM willshow stuck in an In Progress state but there are nojobs stuck in replication layer Subsequent Replicationsshould succeed

CSCvq22898

40(2a)35(2a)After an unexpected power outage the hyperflex clusteris unable to come up correctly

All processes are running time is synced and all nodescan vmkping each other as required

CSCvp13990

40(2a)40(1a)When specifying IP address instead of node ID stclinode maintenanceMode commands fail

CSCvp33657

40(2a)40(1a)2 node edge cluster if experiencing Intersightarbitratorconnectivity issues should NOT allowHXMaintenancemode to initiate as this will bring the cluster down

CSCvr31573

40(2a)35(2c)Old files in varsupportZKTxnlog are not purged withthe daily zklog-cleanup cron job

CSCvr89066

40(2a)35(2a)When a node is in MaintenanceMode any disk removalor replacement will be reflected in UI only after the nodeis brought back frommaintenancemode This is becausestorfs is not running on the node in maintenance andwill not be able to detect disk activities until it is broughtout of Maintenance Mode

CSCvo86431

40(2a)35(2c)Cluster might go offline and stop serving IO requestsrootcvm~ stcli cluster storage-summary--detailGet cluster storage summary failedjavanetConnectException Connection refused19216814210010207

CSCvr01645

40(2a)35(2d)A stretch cluster deployment fails with a messageFormatting nodes Node in Use on the Hyperflex installerand crmZoneType value is seen as 1 in storfscfg oroptspringpathconfigstretchtunes

CSCvq80340

Release Notes for Cisco HX Data Platform Release 4037

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)30(1c)During upgrade we upload the upgrade packagedownloaded from Ciscocom Upgrade packages are offile type tgz hxconnect acceptsstorfs-packages-401a-33028tar file a different type ofcompression and allows upgrade to start and fail whenit cannot decompress the file for bootstrap and scvmclientupgrade on ESXi

CSCvq63888

40(2a)40(2a)While attempting to install hyperflex via intersight thefollowing error is seen

Failed in Task Add hosts to vCenter Cluster with ErrorTry adding hosts manually to vCenter and retry Failedto get addressFailed to host by namelookupHostnamecompanycom on 000053 read udp12700145430-gt12700153 read connection refused

CSCvr52098

40(2a)35(2a)CoreAPI call may time out for cluster management APIwhile doing cluster create validation

CSCvo92952

40(2a)35(2d)Validation fails with Mixed mode expansion checkdue to Enhanced vMotion Compatibility Incompatibilitywhen the ESXi version is different

CSCvq95460

40(2a)30(1i)This failure is seen in installer when custom workflowis used hypervisor configuration + deploy (clean diskpartition=no) at Installing software packages on storagecontroller VM

CSCvr66309

40(2a)35(2d)The HX Installer apparently replaced a comma bysemicolon on the VLAN ID range while exporting theconfiguration to a JSON file

CSCvq93831

40(2a)40(1a)When using unsupported browsers HX Connect userswill not be able to login These users see an errorindicating unauthorized user

CSCvp22693

Release Notes for Cisco HX Data Platform Release 4038

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)40(1a)A vulnerability in the web-based management interfaceof Cisco HyperFlex Software could allow anunauthenticated remote attacker to conduct a cross-siterequest forgery (CSRF) attack on an affected system

The vulnerability is due to insufficient CSRF protectionsfor the web UI on an affected device An attacker couldexploit this vulnerability by persuading a user of theinterface to follow a malicious link A successful exploitcould allow the attacker to perform arbitrary actions withthe privilege level of the affected user

Cisco has released software updates that address thisvulnerability There are no workarounds that address thisvulnerability

This advisory is available at the following link

httpstoolsciscocomsecuritycentercontentCiscoSecurityAdvisorycisco-sa-20190807-hypflex-csrf

CSCvp24343

40(2a)35(2d)Add message to HX Connect Progress flow to notAcknowledge Pending Activities and reboot the serverson UCSM

HX Connect is doing a controlled rolling server upgradein the background

CSCvq22844

40(2a)40(1b)Compute Node expand operation fails due to incorrectcluster name

CSCvr43786

40(2a)26(1e)RF-2 cluster shutdown during data resynchronizationafter a node failure (non storfs) followed by multipledisk read failures and a HardBlacklist of a disk

CSCvp41241

40(2a)40(1a)HyperFlex Edge nodes do not properly set the ESXihostname during deployment

CSCvo13143

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4039

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4040

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1b) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401b-33133p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4041

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4042

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1b)40(1a)While running the HyperFlex installer with WindowsServer Hyper-V cluster creation process fails with theerror Failure occurred during ClusterCreation process Unable to post thecontent to the downstream This symptomis encountered while deploying HyperFlex cluster withHyperFlex nodes configuredordered with Cisco VIC1457 MLOM (PID HX-MLOM-C25Q-04) andWindows Server Datacenter or Core with Hyper-Vversion 2016 or 2019

CSCvp64140

Release Notes for Cisco HX Data Platform Release 4043

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)35(2b)Adding Micron 5200 drive to a cluster fails to increasecluster capacity

CSCvo69067

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4044

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4045

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1a) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401a-33028p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4046

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4047

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1a)30(1d)Cluster instability when disks of different sector sizeplaced in HX node

CSCvk17250

40(1a)

35(2c)

35(1a)When logged in using a local HX user account insteadof a Virtual Center account an error message appearsintermittently indicating Virtual Centerunreachable or Resource informationcannot be updated when VC is reachable

CSCvo36198

40(1a)

35(2b)

30(1d)HXDP does not work with EMC RecoverPoint - needsto support VMware API (FSS-Readdir)

CSCvk38003

Release Notes for Cisco HX Data Platform Release 4048

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)

35(2a)

30(1i)

25(1c)

25(1d)

Zookeeper (Exhibitor) process on HyperFlex StorageController VMs may fail to respond to requests whenvarzookeeper has low or no free space

CSCvm90352

40(1a)26(1c)Use Asynchronous consolidation for HX SnapshotsCSCvn02151

40(1a)35(2b)HX Quiesced Snapshot for Backup VendorsCSCvo90713

40(1a)35(2a)The cluster creationcluster expansion workflow stopswith the following error message at the validation step

FIRMWARE-Check UCSC-SAS-M5HD

FIRMWARE-Check UCSC-SAS-M5HD Required00000029000000320000003500000050 Found00000058

Action Needed Update the Controller Firmware toRequired Version

CSCvn17787

35(2b)30(1d)Kernel migration fails when a local datastore is searchedfor in the ESXi host

CSCvn73127

35(2a)26(1b)A node shuts down when two disks are replaced (acaching disk and another capacity disk) where thecapacity disk is inserted first and then the caching diskis inserted

CSCvk46364

Hyper-V

40(1a)35(2a)Cluster expansion may fail with an error code 500 -operation timed out

CSCvn28721

35(2b)35(2a)During upgrade remove the VLAN on team created foruser vSwitch During fresh installation only one VLANtag is set to the vSwitch and team although multipleVLANs were entered

CSCvn54300

35(2b)35(2a)While upgrading a Hyper-V cluster on account of a rarerace condition between the stUpgradeService andZookeeper servers the upgrade orchestration throws anupgrade validation error and the upgrade process isaborted

CSCvn60486

Release Notes for Cisco HX Data Platform Release 4049

Resolved Caveats in Release 40(1a)

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Startrestart EAM service invCenter

When EAM service is not runningvCenter registration is failing

CSCvt55712

35(2a)Contact Cisco TAC if you believeyou are seeing similar conditions

HyperFlex Stretch Cluster saw poorapplication performance due to writelatency cluster remained unhealthy andrebalance was stuck

This defect is currently being used forinvestigative purposes

CSCvt45344

35(2g)Remove log files fromvarlogspringpath folder when thelog files are not getting rotatedStop the relevant service removelarge log files fromvarlogspringpath folder andrestart the service

varstv folder in Controller VM maybecome full

CSCvt36374

40(2b)Restarting the stMgr will initializeit again

In a Cluster configured with DR networkStMgr may not get initialized and stuckin a deadlock while enabling the IPTablerules (can be verified from the stMgrlog)

CSCvt89709

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Click recover for the selectedVM again under the standalonemode This will move the VM torecovered state

When the user recovers a VM protectedin a group it along with other VMs in thegroup move to recovering state instandalonemode The selectedVM shouldmove to recovered state

CSCvs75553

35(2e)Clean up the fail over cluster andthen recreate the fail over clusterNo need to touch the HX storagecluster

Hyper-V When replicated DC was usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2d)40(1a)

Keep cluster utilization as low aspossible To avoid read-only statekeep below 92

HyperFlex clusters storage utilization isabove 92 usable capacity

CSCvr20922

Release Notes for Cisco HX Data Platform Release 4050

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1b)NAAll the disks in the node got locked afterrebooting the node

We successfully unlocked usingsed-clientsh -U command but wantedto test with another reboot and driveslocked again

CSCvs74286

35(2g)To recover the VM copy over thedata disks and attach them to thenew VM

After enable encryption HX connectencryption status showing Caution butall of the disks enable encryptionsuccessful

-Self Encrypting Drives Service isrunning on all of the nodes

-There is no error message of encryption

- USB0 interface is up

-All SED disks showing supported 1enabled 1 locked 0

CSCvs93245

40(1b)Enable kernelpanic_on_oops as apersistent configuration This willcause the node to panic and rebootimmediately

A cluster node running HX release40(1b) may hang in the Linux kernelThis is classified as an oops and adeviation from the expected behavior

CSCvs54285

Release Notes for Cisco HX Data Platform Release 4051

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Enabling K8iscsi stack in Hx clusterCSCvs41324

Release Notes for Cisco HX Data Platform Release 4052

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Steps to enable K8 on HX cluster

Perform following steps on ALLthe controller VMs

1 Updateetcinitscvmclientconf toenable tune iscsiEnable totrue

sed -iesiscsiEnable=falseiscsiEnable=trueetcinitscvmclientconf

2 Run following initctlcommand to reloadconfiguration

initctl reload-configuration

3 Restart scvmclient process

stop scvmclient startscvmclient

4 Run following initctlcommand to mount SYSTEMDS

initctl emit --no-waitsystem-datastore-created

5 Verify that iscsiEnable tuneis set to true using thefollowing command

ps -eaf | grep scvmclient |grep -v grep

root 6241 1 1 Dec06 014007optspringpathstorfs-corescvmclient-T logEnabled=false -TlogSyslogEnabled=true -TlogEchoToScreen=false -TstatLoggingToFile=false -TstatLoggingToSyslog=true-TlogDir=varlogspringpath-TnfsBackendServerList=1010748100-T iscsiEnable=true -TiscsiUseAsync=true -T

Release Notes for Cisco HX Data Platform Release 4053

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

iscsiConfigFilePath=nfsSYSTEMistgtconfps -eaf | grep scvmclient |grep -v grep

35(2e)Following procedure can be usedto check NFS Queue Depth andincrease if needed

rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth256 lt- Low value[rootHXESXI1] vsish -e setvmkModulesnfsclientmnt[DATASTORE]propertiesmaxQDepth 1024[rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth1024 lt- Optimalvalue

This requires an ESXi host rebootto take effect

Please place the node in HyperflexMaintenanceMode and gracefullyreboot the node for the changes tobe applied

HyperFlex Datastore NFS Queue Depthshows as 256 which can lead toperformance (including latency) issues

CSCvr83056

40(1b)This is cosmetic The error will goaway once the cluster returns tohealthy status

When a single node is offline in clusterstcli cluster storage-summary shows twonodes unavailable rootSCVM~ stclicluster storage-summary

CSCvs47735

35(2g)Modify both the root and adminpassword for the SCVMs to be thesame

stcli security password set -uadmin

Cluster expansion fails at Config Installerstage when the root and admin passwordfor the storage controller (SCVM) aredifferent

CSCvs69317

35(2d)NAAfter successfully changing the DNSserver on the HX controller we still cansee the original DNS entry that was addedduring the deployment

CSCvs69154

Release Notes for Cisco HX Data Platform Release 4054

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Customers are strongly encouragedto work with TAC in order toidentify and remediate this issue

1 Change getcluster from 2m to8m inoptspringpathstorfs-mgmtstMgr-10confapplicationconfon all nodes

2 Restart stMgr on all nodes

You may see this error message after afailed upgrade or other task such asattempting to enter a node into HXmaintenance mode

CSCvs53555

40(2a)Run the following command oneach storage controller VM as root

grep -i monitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp sed -iesmonitor_dns_serverstruemonitor_dns_serversfalseoptspringpathhx-diag-toolswatchdog_configjsonampamp grep -imonitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp restart watchdog

Receive the following alertevent in HXConnect HX Controller VMHOSTNAME one or more configuredDNS servers not responding

CSCvt13947

35(2b)Delete any empty (Size 0) log filesunder varlogzookeeperversion-2ORvarlogzookeeperstandaloneversion-2

Zookeeper fails to start while Exhibitor isrunning however echo srvr returnsnothing

CSCvs21562

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

40(4e)1 De-commission the server

2 Power drain the server -REMOVE BOTH POWERCORDS ON THE BACK OFTHE SERVER FOR 10SECONDS THENREINSERTPOWERCORDS

3 Re-commission the server

M2 boot disk is missing from serverinventory after upgrade to server firmware40(4e) As a result server fails to boot toOS installed in the M2 disk The issuepersists after re-acknowledgement as wellas de-commission andre-acknowledgement of the server

CSCvs02466

Release Notes for Cisco HX Data Platform Release 4055

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Re-start the flusher manuallyMany disks taken offline by a fault ormanually resulting in ENOSPC oncluster

CSCvq32721

35(2e)Clean up fail over cluster and thenrecreate the fail over cluster Thereis no need to touch the HX storagecluster

[Hyper-V]When replicated DCwas usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2b)Manually install upgradescvmclient VIB andmake sure thatit matches with HXDP version

Upgrade from release HX 35(2a) to35(2x) 4X does not upgrade scvmclientRecovery point may not work

CSCvq54992

Management

30(1b)To recover the VM copy over thedata disks and attach them to thenew VM

[HyperV]VMshows up onmultiple nodesCSCvj22992

35(1a)1 Do not change MTU at ESXlevel on running cluster

2 Reset back to the original value

ChangingMTU from 9000 to 1500 on oneESX node causes storfs process to restarton all nodes

CSCvm99150

Release Notes for Cisco HX Data Platform Release 4056

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)After the installation (for newdeployments) or after an upgradeto 401a (for existingdeployments) perform thefollowing steps on all ControllerVMs

1 Edit the following tune files (onall Controller VMs)

optspringpathconfiglfftunes

optspringpathconfigvsi_16tbtunes

2 SetcleanerEnableSegSummaryCleaning

to false

3 After editing the above tunefiles

a SSH in to all the ControllerVMs

b Run storfstool -- -Z

c Run the below command andcheck the values Tune valueshould be true

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=true

d Type below commands andapplied the tunes changesdynamically

echo false gttmpstprocfssystemtunecleanerEnableSegSummaryCleaning

e Verify that tune values arechanged Run below commandValue should be false

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=false

f umount tmpstprocfs

A cluster with 8TB or 12TB disk drivesmay experience IO stalls for severalminutes after another node fails in thecluster

CSCvp23718

Release Notes for Cisco HX Data Platform Release 4057

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)1 Update the conf file manuallyto change to the link local addressto

2 Restart scvmclient on thecontroller and ESX

HX 35(2b) FlexVol to HX 40 CSIupgrade does not work FlexVol continuesto work

CSCvp26319

35(2a)Disable rebalance until node healsthen reenable rebalance

7-node cluster healing very slowlybecause rebalance change AMS reqs areblocking healing process

CSCvp62512

35(2a)NAWith moderate cluster storage capacityusage under site failover cluster clustermay experience an all paths (APD) downcondition

CSCvq49412

40(1a)NAThis product includes Third-partySoftware that is affected by thevulnerabilities identified by the followingCommon Vulnerability and Exposures(CVE) IDs

CVE-2016-0762 CVE-2016-6797CVE-2016-6816 CVE-2016-8735CVE-2017-5647 CVE-2017-12615CVE-2017-12616 CVE-2017-12617CVE-2017-7674 CVE-2018-1304CVE-2018-8014 CVE-2018-1336CVE-2018-8034 CVE-2018-11784CVE-2019-0232

CSCvp36364

35(2d)NAstcli cluster info command needs toprovide UCSM VIP address Currently itshows ucsm-hostcom

CSCvq11456

35(2a)No workarounds currently Thestale entries do not affect the dayto day operations of HyperFlex

HyperFlex upgrade validation failedbecause cluster has extraneous stNodes ininternal database

CSCvq32530

35(2a)NAWhen not in maintenancemode all drivesappear in HX Connect and get updated asexpected

When in maintenance mode extra drivesare shown in HX Connect with no slotnumbers

CSCvq42378

35(2b)Contact TAC to cleanup duplicatenode

Nodes duplicated in HX Connect aftercluster recreate

CSCvq61825

Release Notes for Cisco HX Data Platform Release 4058

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)NACurrently HyperFlex installs on Hyper-Vdo not contain the stcli security whitelistcommandset

CSCvq66245

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

Release Notes for Cisco HX Data Platform Release 4059

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)[Hyper-V] Converged node expansionfails for incoming node with VIC 14xx

CSCvq39523

Release Notes for Cisco HX Data Platform Release 4060

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

1 Set Jumbo frame inWindowsHypervisor for incomingnode

Get-NetAdapter storage-data-a|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

Get-NetAdapter storage-data-b|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

2 Verify all interfaces are setcorrectly to support Jumbo frames

Get-NetAdapter storage |Get-NetAdapterAdvancedProperty| RegistryKeyword -matchjumbo | ft -auto

You should receive the followingmessage

Name DisplayName DisplayValue

RegistryKeyword RegistryValue

---- ----------- ------------

--------------- -------------

vswitch-hx-storage-data Jumbo

Packet 9014 Bytes

JumboPacket 9014

storage-data-b Jumbo Packet

Bytes 9014 JumboPacket

9014

storage-data-a Jumbo Packet

Bytes 9014 JumboPacket

9014

3 Reboot only the incoming nodeor the node being expanded

4 Retry Cluster Expansion in

Release Notes for Cisco HX Data Platform Release 4061

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

Stretched Cluster (Availableon ESX Only)

Microsoft Hyper-VVMware ESXiNode

212121111121mdash2121MaxComputetoConvergedratio

NoExpansion

Requires Enterprise license

Requires uniform expansion across both sites

Guidelines and Limitationsbull Starting with release 40(2a) SCVM is no longer needed on a Compute node

bull HX REST API Access Token Management ndash Applications leveraging HX REST APIs should re-useaccess tokens when making API calls Once obtained using the AAA Obtain Access Token API accesstokens are valid for 18 days (1555200 seconds) In addition AAA enforces rate limiting on ObtainAccess Token API requests in a 15 minute window auth can be invoked (successfully) a maximum of5 times A user is allowed to create a maximum of 8 unrevoked tokens Subsequent call to auth willautomatically revoke the oldest issued token tomake room for the new token Amaximum of 16 unrevokedtokens can be present in system In order to prevent brute-force attacks after 10 consecutive failedauthentication attempts a user account is locked for a period of 120 secondsFor more information seeCisco HyperFlex Systems REST API Reference guide

HxConnect makes use of AAA Authentication REST API for login and the above rate limit applies toHxConnect also

Upgrade Guidelines

The following list is a highlight of critical criteria for performing an upgrade of your HyperFlex system

bull Unsupported HX Data Platform 17x 18x 20 21x 25x and 26x clustersmdashUsers from anyversion prior to 26(1a) must step through an intermediate version before upgrading to 40 or later releasesIf you need to upgrade your environment from a Cisco HyperFlex HX Data Platform software releasethat is past the last date of support to the latest suggested release on the Cisco Software Download sitesee Cisco HyperFlex Systems Upgrade Guide for Unsupported Cisco HXReleases For more informationsee the Software Advisory for CSCvq66867 WARNING Only Use HXDP 26(1e) Upgrade PackageWhen Upgrading From HXDP 18(1a)-18(1e)

bull Hypercheck Health Check UtilitymdashCisco recommends running this proactive health check utility onyour HyperFlex cluster prior to upgrade These checks provide early visibility into any areas that mayneed attention andwill help ensure a seamless upgrade experience For more information see the HyperFlexHealth amp Pre-Upgrade Check Tool TechNote for full instructions on how to install and run Hypercheck

bull vSphere 67 Software AdvisorymdashDo not upgrade to Cisco HX Data Platform Release 40(1a) whenrunning ESXi 67U1 EP06 (build 11675023) Do not upgrade to 67U1 EP06 (build 11675023) ifrunning Cisco HX Data Platform Release 40(1a) See the Software Advisory CSCvo56350 for furtherdetails

Release Notes for Cisco HX Data Platform Release 4019

Guidelines and Limitations

The software build version posted at release will override any other local versions

bull Required vCenter upgrademdashFor enhanced security Cisco HX Data Platform Release 35(1a) or laterrequires the use of TLS 12 Therefore vCenter must be upgraded to 60 U3f or later before upgradingto Cisco HX Data Platform Release 35 or later In addition ESXi should be upgraded as required tomeet HX Data Platform compatibility requirements

bull Minimum HXDP version for upgrademdashHXData Platform clusters running 26(1a) or later may upgradedirectly to 40 using the HX Connect UI

bull Cluster ReadinessmdashEnsure that the cluster is properly bootstrapped and the updated plug-in is loadedbefore proceeding Manual cluster bootstrap is required for upgrade from a pre-35 release

bull Cluster ReadinessmdashEnsure that the cluster is properly bootstrapped and the updated plug-in is loadedbefore proceeding Manual cluster bootstrap is required for HX releases earlier than 35(1a) For moreinformation see theManual Bootstrap Upgrade Process in the Cisco HyperFlex Systems Upgrade Guidefor VMware ESXi Release 40 Do not skip this cluster bootstrap step it is required for all upgradesuntil HX Release 35(1a) Auto bootstrap is supported beginning with HX release 35(1a) For moreinformation see the Auto Bootstrap Upgrade Process from HX Connect UI in the Cisco HyperFlexSystems Upgrade Guide for VMware ESXi Release 40

Manual bootstrap is not supported on Intersight clusters

bull Initiating Upgrade―Use the HX Connect UI or CLI stcli commands when upgrading from 25(1a)or later releases Use either the CLI stcli commands or the HX Data Platform Plug-in to the vSphereWeb Client when upgrading from a pre-25(1a) release The vCenter plug-in should not be used forupgrades starting with the 25(1a) release The vCenter plug-in should not be used for upgrades startingwith the 25(1a) release

If the current cluster version is at 35(1a) or above you do not need to use the stcli command Directupgrade to 40 is possible

bull Complete your Upgrade―The self-healing (or rebalance) capability is disabled temporarily during theupgrade window If the upgrade fails you should complete the upgrade as soon as possible

bull ESXi and HXDP Compatibility―Ensure your cluster is running a compatible version of ESXi basedon the running the HXData Platform version (see the section Software Requirements for VMware ESXi)ESXi compatibility is determined by the major version and update release of ESXi It is generally bestto upgrade HXDP and ESXi together if combining the upgrade operations into a single optimized rebootWhen running a split upgrade first upgrade the HX Data Platform then proceed to upgrade ESXi

bull If Upgrading to vSphere 65

bull Certain cluster functions such as native and scheduled snapshotsReadyClones and Enter or Exit HX Maintenance Mode will not operatefrom the time the upgrade is started until the HX Data Platform upgrade iscomplete

bull After upgrading ESXi using the offline zip bundle use the ESX ExitMaintenance Mode option The HX Exit Maintenance Mode option doesnot operate in the vSphere Web Client until the HX Data Platform upgradeis complete

Note

Release Notes for Cisco HX Data Platform Release 4020

Guidelines and Limitations

bull vSphere 60 UpgradesmdashUsers on vSphere 60 migrating to 65 upgrade components in the followingorder

1 Upgrade HX Data Platform and UCS firmware

2 Upgrade HX Data Platform and ESXi

3 Upgrade HX Data Platform only first then upgrade ESXi or UCS firmware or both

bull M4 Server Firmware UpgradesmdashUpgrade server firmware to ensure smooth operation and to correctknown issues Specifically newer SAS HBA firmware is available in this release and is recommendedfor long-term stability

bull Users are encouraged to upgrade to 31(3c) C-bundle or later whenever possible

bull Users running C-bundle versions before 31(2f) must upgrade server firmware by performing acombined upgrade of UCS server firmware (C-bundle) to 31(3c) or later and HX Data Platform to25 Do not split the upgrade into two separate operations

bull If the cluster is already on 31(2f) C-bundle or later you may perform an HX Data Platform onlyor combined upgrade as required

bull M5 Server Firmware UpgradesmdashM5 generation servers must run firmware version 32(2d) or later

bull Firmware Downgrades mdashDowngrading UCSM from the HX-installer is not supported

bull M4M5 Mixed DomainsmdashA mixed domain occurs when a new separate M5 cluster is installed underthe same UCS domain that contains existing M4 clusters Under these conditions orchestrated UCSserver firmware upgrade will not operate until Cisco HX Data Platform Release 26 or later is installedon the M4 clusters Therefore it is best practice to first upgrade UCS server firmware to the latest 31(3)or 32(2) patch release before adding a new M5 cluster to the existing UCS domain Additionally any17 HX Data Platform clusters must first be upgraded before adding any new M5 clusters to the samedomain

bull Maintenance WindowmdashIf upgrading both HX Data Platform and UCS firmware you can select eithera combined or split upgrade through the vSphere HX Data Platform Plug-in depending on the length ofthe maintenance window Cisco UCSManager infrastructure upgrade is only supported using AutoInstalland the direct server firmware upgrade should be performed only through the upgrade orchestrationframework provided by the HX Data Platform Plug-in

bull Unsupported Self-Encrypting Drives (SEDs)mdashIf adding or replacing self-encrypting drives (SEDs)that have been recently qualified in newer versions of HX Data Platform insert the new drives only afterupgrading HX Data Platform to a compatible version All drives must be SED drives mixing SED andnon-SED is not supported

bullbull Enabling External Host AccessmdashWith Cisco HX Data Platform Release 40(1a) port 445 on themanagement network is blocked for enhanced security Note that prior to 40 port 445 port was openenabling external host access If you are upgrading to 40(1a) from a prior release and would like tocontinue external host access you can use a utility to open select hosts For more information aboutenabling external host access see the ConfiguringHyperFlex Share to SCVMM section in the InstallationGuide for Microsoft Hyper-V

Release Notes for Cisco HX Data Platform Release 4021

Guidelines and Limitations

Mixed Cluster Expansion Guidelinesbull Expanding existing M4 cluster with M5 converged nodes is supported

bull Expanding existing M5 cluster with M4 converged nodes is not supported

bull Expanding existing mixed M4M5 cluster with M4 or M5 converged nodes is supported

bull Adding any supported compute-only nodes is permitted with all M4 M5 and mixed M4M5 clustersusing the HX Data Platform 26 or later Installer Some example combinations are listed here manyother combinations are possibleExample combinationsExpand mixed M4M5 cluster with compute-only B200 C220 C240 M4M5Expand M4 cluster with compute-only B200 M5 C220 M5 C240M5

bull Only expansion workflow is supported to create a mixed cluster Initial cluster creation with mixedM4M5 servers is not supported

bull All M5 servers must match the form factor (220240) type (HybridAF) security capability (Non-SEDonly) amp disk configuration (QTY capacity and non-SED) of the existing M4 servers

bull HX220-M5 will use a maximum of 6 capacity disks (2 disk slots to remain empty) when mixedwith HX220-M4

bull HX Edge SED LFF Hyper-V and Stretched Clusters do not support mixed M4 and M5 clusters

Security FixesThe following security issues are resolved

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-13132 CVE-2019-9924CSCvq6313840(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-11719 CVE-2019-11727CVE-2019-11729

CSCvq7124040(2a)

Release Notes for Cisco HX Data Platform Release 4022

Mixed Cluster Expansion Guidelines

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-1125CSCvr0633940(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2015-9383 CVE-2018-14498CVE-2018-20406 CVE-2018-20852CVE-2019-10160 CVE-2019-13117CVE-2019-13118 CVE-2019-14287CVE-2019-14973 CVE-2019-15903CVE-2019-17546 CVE-2019-18197CVE-2019-18218 CVE-2019-5010CVE-2019-5094 CVE-2019-5481CVE-2019-5482 CVE-2019-9636CVE-2019-9740 CVE-2019-9947CVE-2019-9948

CSCvs0609440(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2017-13168 CVE-2017-18174CVE-2017-18216 CVE-2018-10876CVE-2018-10877 CVE-2018-10878CVE-2018-10879 CVE-2018-10880CVE-2018-10881 CVE-2018-10882CVE-2018-10902 CVE-2018-10938CVE-2018-12233 CVE-2018-12896CVE-2018-13053 CVE-2018-13094CVE-2018-13096 CVE-2018-13405CVE-2018-13406 CVE-2018-14609CVE-2018-14617 CVE-2018-14633CVE-2018-14734 CVE-2018-15572CVE-2018-15594 CVE-2018-16276CVE-2018-16658 CVE-2018-17182CVE-2018-17972 CVE-2018-18021CVE-2018-18690 CVE-2018-18710CVE-2018-6554 CVE-2018-6555CVE-2018-9363

CSCvp6501940(2a)

This vulnerability is due to insufficientHTML iframe protection An attackercould exploit this vulnerability bydirecting a user to an attacker-controlledweb page that contains amaliciousHTMLiframe A successful exploit could allowthe attacker to conduct clickjacking orother clientside browser attacks

NACSCvo9851640(2a)

Release Notes for Cisco HX Data Platform Release 4023

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerability is due to insufficientauthentication for the statistics collectionservice An attacker could exploit thisvulnerability by sending properlyformatted data values to the statisticscollection service of an affected deviceA successful exploit could allow theattacker to cause the web interfacestatistics view to present invalid data tousers

NACSCvj9558440(2a)

The vulnerability is due to insufficientCSRF protections for the web UI on anaffected device An attacker could exploitthis vulnerability by persuading a user ofthe interface to follow a malicious linkA successful exploit could allow theattacker to perform arbitrary actions withthe privilege level of the affected user

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvp2434340(2a)

Vulnerabilities with a version of cJSONidentified by CVE-2019-11834 andCVE-2019-11835

For more information see the relatedCisco Security Advisory

CVE-2019-11834 CVE-2019-11835CSCvq1994940(2a)

HX ESXi image patches for MCEPSCand TAA vulnerabilities with VMwareESXi identified by CVE-2018-12207 andCVE-20190-11135

For more information see the relatedCisco Security Advisory

CVE-2018-12207 CVE-2019-11135CSCvr5439840(2a)

CP networking vulnerabilities affectingthe Linux kernel identified byCVE-2019-11477 CVE-2019-11478CVE-2019-11479

For more information see the relatedCisco Security Advisory

CVE-2019-11477 CVE-2019-11478CVE-2019-11479

CSCvq1954640(2a)

Qualification of Microsoft securitypatches for vulnerabilities withMicrosoftHyper-V hypervisor identified withCVE-2018-12207 CVE-2019-11135

For more information see the relatedCisco Security Advisory

CVE-2018-12207 CVE-2019-11135CSCvr5439940(2a)

Release Notes for Cisco HX Data Platform Release 4024

Security Fixes

DescriptionCVEDefect IDRelease

OpenSSH vulnerabilities identified byCVE-2019-6111

For more information see the relatedCisco Security Advisory

CVE-2016-10708 CVE-2018-15473CVE-2018-20685 CVE-2019-6109CVE-2019-6111

CSCvp7646340(2a)

Multiple vulnerabilities associated withOpenSSL and LibNSS

For more information see the relatedCisco Security Advisory

CVE-2016-2105 CVE-2016-2106CVE-2016-2107 CVE-2016-2109CVE-2016-2176 CVE-2016-2177CVE-2016-2178 CVE-2016-2179CVE-2016-2180 CVE-2016-2181CVE-2016-2182 CVE-2016-2183CVE-2016-6302 CVE-2016-6303CVE-2016-6304 CVE-2016-6305CVE-2016-6306 CVE-2016-6307CVE-2016-6308 CVE-2016-7055CVE-2016-8610 CVE-2017-3731CVE-2017-3732 CVE-2017-3735CVE-2017-3736 CVE-2017-3737CVE-2017-3738 CVE-2018-0495CVE-2018-0732 CVE-2018-0734CVE-2018-0735 CVE-2018-0737CVE-2018-0739 CVE-2018-12384CVE-2018-12404 CVE-2018-5407CVE-2019-1559

CSCvp6667940(2a)

libpng vulnerability identified byCVE-2016-10087

For more information see the relatedCisco Security Advisory

CVE-2016-10087CSCvp6655540(2a)

Vulnerabilities associated with libjpg andlibpng

For more information see the relatedCisco Security Advisory

CVE-2014-9092 CVE-2016-3616CVE-2017-15232 CVE-2018-11212CVE-2018-11213 CVE-2018-11214CVE-2018-1152 CVE-2018-13785

CSCvp3458640(2a)

Vulnerabilities associated with Glibidentified by CVE-2018-16428 andCVE-2018-16429

For more information see the relatedCisco Security Advisory

CVE-2018-16428 CVE-2018-16429CSCvp3120740(2a)

Multiple vulnerabilities associated withcurl expat python 27 python 35 36and 37 freetype and giflib

For more information see the relatedCisco Security Advisory

CVE-2019-15133 CVE-2019-15903CVE-2019-5010 CVE-2019-5481CVE-2019-5482 CVE-2019-9636CVE-2019-9740 CVE-2019-9947CVE-2019-9948

CSCvr3690340(2a)

Release Notes for Cisco HX Data Platform Release 4025

Security Fixes

DescriptionCVEDefect IDRelease

Multiple third party softwarevulnerabilities

For more information see the relatedCisco Security Advisory

CVE-2019-14379 CVE-2019-12384CVE-2019-14439

CSCvq9203240(2a)

Vulnerabilities associated with elfutils

For more information see the relatedCisco Security Advisory

CVE-2018-16062 CVE-2018-16402CVE-2018-16403 CVE-2018-18310CVE-2018-18520 CVE-2018-18521CVE-2019-7149 CVE-2019-7150CVE-2019-7665

CSCvq4325040(2a)

Vulnerabilities associated with vim

For more information see the relatedCisco Security Advisory

CVE-2017-5953 CVE-2019-12735CSCvq4323040(2a)

Vulnerabilities associated with sqlite3

For more information see the relatedCisco Security Advisory

CVE-2016-6153 CVE-2017-10989CVE-2017-13685 CVE-2017-2518CVE-2017-2519 CVE-2017-2520CVE-2018-20346 CVE-2018-20505CVE-2018-20506 CVE-2019-8457CVE-2019-9936 CVE-2019-9937

CSCvq4321340(2a)

Vulnerabilities associated with glib20

For more information see the relatedCisco Security Advisory

CVE-2018-20843CSCvq4320940(2a)

Vulnerabilities associated with expat

For more information see the relatedCisco Security Advisory

CVE-2018-20843CSCvq4320540(2a)

Vulnerabilities associated with bzip2

For more information see the relatedCisco Security Advisory

CVE-2016-3189 CVE-2019-12900CSCvq4319440(2a)

Mulitple Vulnerabilities associated wtihNodeJS

For more information see the relatedCisco Security Advisory

CVE-2018-12115 CVE-2018-0734CVE-2018-5407 CVE-2018-12120CVE-2018-12121 CVE-2018-12122CVE-2018-12123 CVE-2018-12116CVE-2019-5737 CVE-2019-5739

CSCvq1069440(2a)

Vulnerabilities associated with PalletsJinja strformat_map and identified byCVE-2019-10906 and CVE-2016-10745

For more information see the relatedCisco Security Advisory

CVE-2019-10906 CVE-2016-10745CSCvq1038840(2a)

Release Notes for Cisco HX Data Platform Release 4026

Security Fixes

DescriptionCVEDefect IDRelease

Multiple Security Vulnerabilities

For more information see the relatedCisco Security Advisory

CVE-2014-8501 CVE-2014-9939CVE-2015-9262 CVE-2016-10087CVE-2016-2226 CVE-2016-4487CVE-2016-4488 CVE-2016-4489CVE-2016-4490 CVE-2016-4491CVE-2016-4492 CVE-2016-4493CVE-2016-6131 CVE-2018-10963CVE-2018-13785 CVE-2018-17100CVE-2018-17101 CVE-2018-18557CVE-2018-18661 CVE-2018-7456CVE-2018-8905

CSCvq0917840(2a)

Vulnerabilities associated withlibseccomp

For more information see the relatedCisco Security Advisory

CVE-2019-9893CSCvq0756840(2a)

Vulnerabilities associated withgdk-pixbuf

For more information see the relatedCisco Security Advisory

CVE-2017-12447CSCvq0675540(2a)

Vulnerabilities associated withpython-crypto

For more information see the relatedCisco Security Advisory

CVE-2018-6594CSCvp9381740(2a)

Vulnerabilities associated with wget

For more information see the relatedCisco Security Advisory

CVE-2018-20483 CVE-2019-5953CSCvp8672140(2a)

Multiple vulnerabilities associated withPython Crypto

For more information see the relatedCisco Security Advisory

CVE-2018-6594CSCvp6674840(2a)

Vulnerabilities associated withsensible-utils

For more information see the relatedCisco Security Advisory

CVE-2017-17512CSCvp6673440(2a)

Multiple vulnerabilities associated withPython

For more information see the relatedCisco Security Advisory

CVE-2018-1000030CSCvp6668940(2a)

Release Notes for Cisco HX Data Platform Release 4027

Security Fixes

DescriptionCVEDefect IDRelease

Multiple patch vulnerabilities identifiedby CVE-2016-10713

For more information see the relatedCisco Security Advisory

CVE-2016-10713CVE-2018-1000156 CVE-2018-6951

CSCvp6667240(2a)

Vulnerabilities associatedwith Little CMSidentified by CVE-2016-10165

For more information see the relatedCisco Security Advisory

CVE-2016-10165 CVE-2018-16435CSCvp6666440(2a)

Vulnerabilities associated with TenableScan identified byCVE-2011-5325CVE-2019-11068CVE-2016-7076CVE-2017-1000368

For more information see the relatedCisco Security Advisory

CVE-2011-5325 CVE-2016-7076CVE-2017-1000368CVE-2019-11068

CSCvp6474640(2a)

Vulnerabilities associated with libxml2294 and earlier as used in XMLSec1223 and earlier and other products andidentified by CVE-2016-9318

For more information see the relatedCisco Security Advisory

CVE-2016-9318 CVE-2017-16932CVE-2017-18258 CVE-2018-14404CVE-2018-14567

CSCvp3479240(2a)

Vulnerabilities associated with LFTPRemote File Names Unauthorized AccessVulnerability and identified byCVE-2018-10916

For more information see the relatedCisco Security Advisory

CVE-2018-10916CSCvp2926640(2a)

Vulnerabilities associated withCryptographyDeprecationWarning signerand verifier have been deprecated

For more information see the relatedCisco Security Advisory

CVE-2018-7750CSCvo3409740(2a)

Multiple Vulnerabilities in python python27 35

For more information see the relatedCisco Security Advisory

CVE-2019-16056CSCvr3979340(2a)

Release Notes for Cisco HX Data Platform Release 4028

Security Fixes

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toexecute commands as the root user

The vulnerability is due to insufficientinput validation An attacker could exploitthis vulnerability by connecting to thecluster service manager and injectingcommands into the bound process Asuccessful exploit could allow the attackerto run commands on the affected host asthe root user

Cisco has released software updates thataddress this vulnerability There areworkarounds that address thisvulnerability

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvq2417640(1b)35(2g)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

The vulnerability is due to an unprotectedlistening interface An attacker couldexploit this vulnerability by connectingto the listening interface and injectingcommands to the bound process Anexploit could allow the attacker to runcommands on the affected host as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvj9560640(1b)35(2g)

Release Notes for Cisco HX Data Platform Release 4029

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerabilities associated with theJVM18U121memory leak found duringVC alarms (concurrent 40 calls usingREST API)

CVE-2017-10053 CVE-2017-10067CVE-2017-10074 CVE-2017-10078CVE-2017-10081 CVE-2017-10087CVE-2017-10089 CVE-2017-10090CVE-2017-10096 CVE-2017-10101CVE-2017-10102 CVE-2017-10107CVE-2017-10108 CVE-2017-10109CVE-2017-10110 CVE-2017-10111CVE-2017-10115 CVE-2017-10116CVE-2017-10118 CVE-2017-10135CVE-2017-10176 CVE-2017-10193CVE-2017-10198 CVE-2017-10243CVE-2017-10274 CVE-2017-10281CVE-2017-10285 CVE-2017-10295CVE-2017-10345 CVE-2017-10346CVE-2017-10347 CVE-2017-10348CVE-2017-10349 CVE-2017-10350CVE-2017-10355 CVE-2017-10356CVE-2017-10357 CVE-2017-10388CVE-2017-3509 CVE-2017-3511CVE-2017-3526 CVE-2017-3533CVE-2017-3539 CVE-2017-3544CVE-2018-2579 CVE-2018-2582CVE-2018-2588 CVE-2018-2599CVE-2018-2602 CVE-2018-2603CVE-2018-2618 CVE-2018-2629CVE-2018-2633 CVE-2018-2634CVE-2018-2637 CVE-2018-2641CVE-2018-2663 CVE-2018-2677CVE-2018-2678 CVE-2018-2783CVE-2018-2790 CVE-2018-2794CVE-2018-2795 CVE-2018-2796CVE-2018-2797 CVE-2018-2798CVE-2018-2799 CVE-2018-2800CVE-2018-2814 CVE-2018-2815CVE-2018-2952 CVE-2018-3136CVE-2018-3139 CVE-2018-3149CVE-2018-3150 CVE-2018-3169CVE-2018-3180 CVE-2018-3183CVE-2018-3214 CVE-2019-2422

CSCvo8899740(1b)

Tomcat and Nginx logs are not beingcollected in the support bundle generatedthrough HX Connect in Release 35

NACSCvm5803140(1b)

The vulnerabilities associated with thelibmspack software package versionincluded in Cisco HX Data Platform

CVE-2018-18584

CVE-2018-18585

CSCvn3511940(1a)

Release Notes for Cisco HX Data Platform Release 4030

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerabilities associated withFasterXML Jackson-Databind TimeValueField Denial of Service

CVE-2018-14719

CVE-2018-14720

CVE-2018-1000873

CVE-2018-14721

CVE-2018-19360

CVE-2018-19362

CVE-2018-19361

CVE-2018-14718

CSCvn8228240(1a)

The vulnerabilities associated with theOpenSSL Protocol software packageversion included in Cisco HX DataPlatform

CVE-2013-3587CSCvo0505440(1a)

The vulnerabilities associated withThird-Party Software Denial of Service

CVE-2018-16487

CVE-2018-19361

CSCvo2781840(1a)

The vulnerabilities associated withLinuxkernel for Ubuntu 1710

CVE-2018-1092

CVE-2018-7492

CVE-2018-8087

CVE-2018-1068

CVE-2018-8781

CSCvm5314935(2a)

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)25(1d)stcli services dns remove should remove the DNS serverinfo from the interface files

CSCvs70967

40(2b)40(1b)HX 40(1b) new deployment from Intersight productioncloud

User receives following error while enabling CSI(Kubernetes integration)

Failure occurred during volume_access_enable Errorwas Nonzero exit code 1

CSCvt10522

40(2b)40(2a)Receive the following alertevent in HX Connect HXController VM HOSTNAME one or more configuredDNS servers not responding

CSCvt13947

Release Notes for Cisco HX Data Platform Release 4031

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)40(2a)API UI CLI show few Drive slots Empty after upgradeto 402a

The drives are seen in the sysmtool lsscsi and stclicommand outputs

The cluster is healthy with no errors

CSCvt14914

40(2b)35(2d)After successfully changing the DNS server on the HXcontroller we still can see the original DNS entry thatwas added during the deployment

CSCvs69154

40(2b)35(2e)HyperFlex Connect and VCenter show APD alarms forHX and non HX datastores This implies there is an issuewith HyperFlex when there may not be

CSCvs30080

40(2b)40(2a)An issue where all the disks in the node get locked afterrebooting the node

We successfully unlocked using sed-clientsh -Ucommand but wanted to test with another reboot anddrives locked again

CSCvs74286

40(2b)40(2a)When running stcli license commands on HyperFlexthe following error is seen

rootSpringpathControllertmp stcli license show all

Show smart licensing failed

Smart Agent is not ready please wait a minute and tryagain

CSCvt13929

40(2b)40(2a)After upgrading to HX 40(2a) the first RoboEdge nodethat gets rebooted then ignores persistent drivesPIDHX-HD24TB10K4KN

Cluster will be in degraded state and have reducedcapacity

CSCvt20144

40(2b)35(2b)Zookeeper fails to start while Exhibitor is runninghowever echo srvr returns nothing

CSCvs21562

40(2b)40(2a)When performing an HyperFlex upgrade a validationwarning may occur due to the host not having EnterprisePlus or Enterprise hypervisor licensing

Upgrade Validation Warning

ESXi host esx1labtest should be configured withVMware Enterprise license for upgrade to continue

CSCvs91787

Release Notes for Cisco HX Data Platform Release 4032

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)35(2g)Cluster expansion fails at Config Installer stage whenthe root and admin password for the storage controller(SCVM) are different

CSCvs69317

40(2b)35(2g)Panic while upgrading ESXiCSCvt06983

40(2b)40(2a)A cluster node running may hang in the Linux kernelThis is classified as an oops and a deviation from theexpected behavior

CSCvs54285

40(2b)35(2g)Rebalance failing on 10+10 node HX 35(2g) stretchcluster

CSCvs69007

40(2b)35(2d)The cluster becomes inaccessible to the IOVisorCSCvr54687

40(2b)40(2a)Panic on the storage controllerCSCvt61297

40(2b)40(2a)The size of support bundle is very large when collectedwith storfs-support command

CSCvt63306

40(2b)40(2a)storfs crash associated withFileSystemUsageWarningEvent orFileSystemUsageAlertEvent due to high usage ofvarstv above 80

CSCvt72807

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

40(2a)35(2d)In normal scenarios HX upgrade gets pendingACK afterupdating service profile In this case UCS only upgradewas stalled because after updating service profile HXupgrade was not getting pending ACK and it was alwayswaiting state

CSCvo39912

40(2a)35(1a)When using motherBoardReplace-12 to clean ZK fromold stNodepNodes resulted in unmounted datastore andmaking the size of the Hyperflex Datastores 0 resultingin all VMs in the cluster going offline

CSCvq39471

40(2a)26(1e)Upgrading cluster and getting error

Failed upgrade validations Checking vCenterconfiguration Reason Upgrade validations failed DRSFault Insufficient resources to satisfy configured failover

CSCvm77294

Release Notes for Cisco HX Data Platform Release 4033

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)The cluster expand fails on a node with DR replicationconfigured

When a HX cluster which has DR replication configuredis expanded we see the installer UI pulling in thereplicationVLAN information instead of themanagementVLAN information Even if we change that informationto the correct mgmt VLAN id and name it it does notseem to work as the node is configured with the VLANof the replicationVLAN in ESXi This leads to the failureof the node add with the host unreachable error

CSCvo70650

40(2a)35(1a)Customer reported that server didnt complete thefirmware upgrade automatically one by one

The user manually needs to put the host into maintenancemode and they manually acknowledged the pendingrequests to finish the UCS firmware upgrade

Maintenance policy is set to default (User-ack) as perthe design

CSCvo91624

40(2a)35(1i)When the cluster is in Failed state and stcli noderemove is attempted the output shows successful thoughthe node failed to get removed from the cluster

CSCvo93017

40(2a)35(1a)HyperFlex cluster upgrade may fail during validationwith a DRS Validation failed error

CSCvp31021

40(2a)35(2b)HX cluster expansion will fail with the following errormessage

MAC address pool configuration failure150[ErrorDescription] bad address block range definitioncollision

CSCvp58318

40(2a)35(2b)Memory usage by carbon cacheCSCvq34873

40(2a)35(2g)Upgrading ESXi cluster fails with error Nodemaintenance mode failed

CSCvr03240

40(2a)35(2e)storfs process does not automatically start on Exit HXMaintenance Mode or other tasks which restart StorageController

CSCvr88978

40(2a)35(2a)Post install script fails with message

Failed to execute ipmitool on HX node

CSCvp10707

40(2a)40(1a)HyperFlex expansionworkflow doesnt pull VLANnamecorrectly

CSCvp46539

Release Notes for Cisco HX Data Platform Release 4034

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)During HX cluster deployment - Validate ClusterCreation phase - HX Installer might fail with the error

fromvarlogspringpathstDeploylog

CSCvq45087

40(2a)35(2b)HX Installer fails to login to the SOL for configuringESX hosts

CSCvq91380

40(2a)35(2d)If DRS is enabled with configuration parameters otherthen fully automated ie manual or partially automatedthan this will change to fully automated during clusterexpansion

CSCvr44222

40(2a)35(1a)One of the validation task name has a typo The taskname is Validating node for mixed of SED and non-SEDdisks

It should be Validating node for mix of SED andnon-SED disks

CSCvp82175

40(2a)30(1i)This combination of using smaller NVMe drives is notsupported in expand operation For example if the drivesare only 375GB and they cannot be added to an existingcluster with larger caching SSD

CSCvo12359

40(2a)35(1a)HyperFlex includes a version of OpenSSL that is affectedby the vulnerabilities identified by the followingCommon Vulnerability and Exposures

CVE-2018-0495

CSCvp66679

40(2a)30(1e)SDK invoke exception when taking a native snapshot ona VM with GPU pci passthrough attached

CSCvo00511

40(2a)30(1i)Node replacement script fails due to EAM errorCSCvo62867

40(2a)35(1a)While pairing a cluster (HX release gt= 35) and cluster(HX release lt 35) remote replication network test failson the cluster (HX release lt 35)

CSCvo79760

40(2a)30(1b)Fixed the issue in the latest support-workflow bundleThis should not be hit anymore

CSCvo87061

40(2a)35(2a)On a 3 node cluster the MBreplace script fails as thescript looks for HEALTHY cluster status

CSCvo87080

40(2a)35(2a)MbReplace script (tar) hangs when running on 352b or40

CSCvp12359

Release Notes for Cisco HX Data Platform Release 4035

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)HX replication cleanup failing with error NA 35(2a) -INFODR state is not clean

CSCvp63958

40(2a)40(1a)The check_stig_parameters API incorrectly shows thecompliance state of the cluster as not STIG compliant

CSCvp66277

40(2a)35(2c)The snapshot creation on CBT enabled VM fails witherror Failed in vmreparent vmkfstools clone1

CSCvq06952

40(2a)35(2a)HyperFlex GUI when setting up remote DR-replicationpartnerpeer should show remote replication partner TCPport reachability test results

This will provide quick upfront connectivity results thatfrequently takes customer a while to manually check andconfirm

If reachability tests fail an info window and HX alertshould trigger that list the destination peer IP and thespecific port(s) that could not be reache

CSCvo60587

40(2a)35(2b)If you have more than one IP pool configured for areplication network and local replication network testsare failing you might be running into this issue

CSCvr67130

40(2a)35(2a)Skip task option in MBreplace and Software redeployscripts The scripts should have the option to skip tasksas user inputs and not by modifying the json file

CSCvp05204

40(2a)40(1a)Error noticed on SCVM Console print_req_error IOerror dev fd0 sector 0 on HyperV

It doesnt come up on SSH session

CSCvq34357

40(2a)30(1d)The VMNetwork switch is already created as part of theinstall process

Ideally the ip address assignment should be done duringthe initial install process as well No post install stepsshould be required altrnatively it could be done as partof a post install script

CSCvk36222

40(2a)25(1c)storfs service impacted during HX upgrade from HX21(1) to 25(1c)

HX node which is not being upgraded storfs servicestopped causing outage

CSCvg53223

40(2a)35(1b)HX Stretch Cluster Witness VM reverts to DHCP onreboot

CSCvp37536

40(2a)35(2b)HyperFlex Stretch Cluster Witness does not getprogrammed with an NTP server

CSCvp46578

Release Notes for Cisco HX Data Platform Release 4036

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2b)HyperFlex cluster does not heal after maintenance onnodeserver

CSCvr18528

40(2a)40(2a)High percentage of packet loss on the eth1 data interface9K MTU

CSCvr97089

40(2a)35(2f)The HyperFlex cluster healing state is stuck at 87 andis unable to progress

CSCvr16760

40(2a)40(2a)In some circumstances two replications can occur forthe same VM

From the GUI the replication status for this VM willshow stuck in an In Progress state but there are nojobs stuck in replication layer Subsequent Replicationsshould succeed

CSCvq22898

40(2a)35(2a)After an unexpected power outage the hyperflex clusteris unable to come up correctly

All processes are running time is synced and all nodescan vmkping each other as required

CSCvp13990

40(2a)40(1a)When specifying IP address instead of node ID stclinode maintenanceMode commands fail

CSCvp33657

40(2a)40(1a)2 node edge cluster if experiencing Intersightarbitratorconnectivity issues should NOT allowHXMaintenancemode to initiate as this will bring the cluster down

CSCvr31573

40(2a)35(2c)Old files in varsupportZKTxnlog are not purged withthe daily zklog-cleanup cron job

CSCvr89066

40(2a)35(2a)When a node is in MaintenanceMode any disk removalor replacement will be reflected in UI only after the nodeis brought back frommaintenancemode This is becausestorfs is not running on the node in maintenance andwill not be able to detect disk activities until it is broughtout of Maintenance Mode

CSCvo86431

40(2a)35(2c)Cluster might go offline and stop serving IO requestsrootcvm~ stcli cluster storage-summary--detailGet cluster storage summary failedjavanetConnectException Connection refused19216814210010207

CSCvr01645

40(2a)35(2d)A stretch cluster deployment fails with a messageFormatting nodes Node in Use on the Hyperflex installerand crmZoneType value is seen as 1 in storfscfg oroptspringpathconfigstretchtunes

CSCvq80340

Release Notes for Cisco HX Data Platform Release 4037

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)30(1c)During upgrade we upload the upgrade packagedownloaded from Ciscocom Upgrade packages are offile type tgz hxconnect acceptsstorfs-packages-401a-33028tar file a different type ofcompression and allows upgrade to start and fail whenit cannot decompress the file for bootstrap and scvmclientupgrade on ESXi

CSCvq63888

40(2a)40(2a)While attempting to install hyperflex via intersight thefollowing error is seen

Failed in Task Add hosts to vCenter Cluster with ErrorTry adding hosts manually to vCenter and retry Failedto get addressFailed to host by namelookupHostnamecompanycom on 000053 read udp12700145430-gt12700153 read connection refused

CSCvr52098

40(2a)35(2a)CoreAPI call may time out for cluster management APIwhile doing cluster create validation

CSCvo92952

40(2a)35(2d)Validation fails with Mixed mode expansion checkdue to Enhanced vMotion Compatibility Incompatibilitywhen the ESXi version is different

CSCvq95460

40(2a)30(1i)This failure is seen in installer when custom workflowis used hypervisor configuration + deploy (clean diskpartition=no) at Installing software packages on storagecontroller VM

CSCvr66309

40(2a)35(2d)The HX Installer apparently replaced a comma bysemicolon on the VLAN ID range while exporting theconfiguration to a JSON file

CSCvq93831

40(2a)40(1a)When using unsupported browsers HX Connect userswill not be able to login These users see an errorindicating unauthorized user

CSCvp22693

Release Notes for Cisco HX Data Platform Release 4038

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)40(1a)A vulnerability in the web-based management interfaceof Cisco HyperFlex Software could allow anunauthenticated remote attacker to conduct a cross-siterequest forgery (CSRF) attack on an affected system

The vulnerability is due to insufficient CSRF protectionsfor the web UI on an affected device An attacker couldexploit this vulnerability by persuading a user of theinterface to follow a malicious link A successful exploitcould allow the attacker to perform arbitrary actions withthe privilege level of the affected user

Cisco has released software updates that address thisvulnerability There are no workarounds that address thisvulnerability

This advisory is available at the following link

httpstoolsciscocomsecuritycentercontentCiscoSecurityAdvisorycisco-sa-20190807-hypflex-csrf

CSCvp24343

40(2a)35(2d)Add message to HX Connect Progress flow to notAcknowledge Pending Activities and reboot the serverson UCSM

HX Connect is doing a controlled rolling server upgradein the background

CSCvq22844

40(2a)40(1b)Compute Node expand operation fails due to incorrectcluster name

CSCvr43786

40(2a)26(1e)RF-2 cluster shutdown during data resynchronizationafter a node failure (non storfs) followed by multipledisk read failures and a HardBlacklist of a disk

CSCvp41241

40(2a)40(1a)HyperFlex Edge nodes do not properly set the ESXihostname during deployment

CSCvo13143

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4039

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4040

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1b) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401b-33133p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4041

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4042

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1b)40(1a)While running the HyperFlex installer with WindowsServer Hyper-V cluster creation process fails with theerror Failure occurred during ClusterCreation process Unable to post thecontent to the downstream This symptomis encountered while deploying HyperFlex cluster withHyperFlex nodes configuredordered with Cisco VIC1457 MLOM (PID HX-MLOM-C25Q-04) andWindows Server Datacenter or Core with Hyper-Vversion 2016 or 2019

CSCvp64140

Release Notes for Cisco HX Data Platform Release 4043

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)35(2b)Adding Micron 5200 drive to a cluster fails to increasecluster capacity

CSCvo69067

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4044

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4045

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1a) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401a-33028p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4046

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4047

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1a)30(1d)Cluster instability when disks of different sector sizeplaced in HX node

CSCvk17250

40(1a)

35(2c)

35(1a)When logged in using a local HX user account insteadof a Virtual Center account an error message appearsintermittently indicating Virtual Centerunreachable or Resource informationcannot be updated when VC is reachable

CSCvo36198

40(1a)

35(2b)

30(1d)HXDP does not work with EMC RecoverPoint - needsto support VMware API (FSS-Readdir)

CSCvk38003

Release Notes for Cisco HX Data Platform Release 4048

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)

35(2a)

30(1i)

25(1c)

25(1d)

Zookeeper (Exhibitor) process on HyperFlex StorageController VMs may fail to respond to requests whenvarzookeeper has low or no free space

CSCvm90352

40(1a)26(1c)Use Asynchronous consolidation for HX SnapshotsCSCvn02151

40(1a)35(2b)HX Quiesced Snapshot for Backup VendorsCSCvo90713

40(1a)35(2a)The cluster creationcluster expansion workflow stopswith the following error message at the validation step

FIRMWARE-Check UCSC-SAS-M5HD

FIRMWARE-Check UCSC-SAS-M5HD Required00000029000000320000003500000050 Found00000058

Action Needed Update the Controller Firmware toRequired Version

CSCvn17787

35(2b)30(1d)Kernel migration fails when a local datastore is searchedfor in the ESXi host

CSCvn73127

35(2a)26(1b)A node shuts down when two disks are replaced (acaching disk and another capacity disk) where thecapacity disk is inserted first and then the caching diskis inserted

CSCvk46364

Hyper-V

40(1a)35(2a)Cluster expansion may fail with an error code 500 -operation timed out

CSCvn28721

35(2b)35(2a)During upgrade remove the VLAN on team created foruser vSwitch During fresh installation only one VLANtag is set to the vSwitch and team although multipleVLANs were entered

CSCvn54300

35(2b)35(2a)While upgrading a Hyper-V cluster on account of a rarerace condition between the stUpgradeService andZookeeper servers the upgrade orchestration throws anupgrade validation error and the upgrade process isaborted

CSCvn60486

Release Notes for Cisco HX Data Platform Release 4049

Resolved Caveats in Release 40(1a)

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Startrestart EAM service invCenter

When EAM service is not runningvCenter registration is failing

CSCvt55712

35(2a)Contact Cisco TAC if you believeyou are seeing similar conditions

HyperFlex Stretch Cluster saw poorapplication performance due to writelatency cluster remained unhealthy andrebalance was stuck

This defect is currently being used forinvestigative purposes

CSCvt45344

35(2g)Remove log files fromvarlogspringpath folder when thelog files are not getting rotatedStop the relevant service removelarge log files fromvarlogspringpath folder andrestart the service

varstv folder in Controller VM maybecome full

CSCvt36374

40(2b)Restarting the stMgr will initializeit again

In a Cluster configured with DR networkStMgr may not get initialized and stuckin a deadlock while enabling the IPTablerules (can be verified from the stMgrlog)

CSCvt89709

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Click recover for the selectedVM again under the standalonemode This will move the VM torecovered state

When the user recovers a VM protectedin a group it along with other VMs in thegroup move to recovering state instandalonemode The selectedVM shouldmove to recovered state

CSCvs75553

35(2e)Clean up the fail over cluster andthen recreate the fail over clusterNo need to touch the HX storagecluster

Hyper-V When replicated DC was usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2d)40(1a)

Keep cluster utilization as low aspossible To avoid read-only statekeep below 92

HyperFlex clusters storage utilization isabove 92 usable capacity

CSCvr20922

Release Notes for Cisco HX Data Platform Release 4050

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1b)NAAll the disks in the node got locked afterrebooting the node

We successfully unlocked usingsed-clientsh -U command but wantedto test with another reboot and driveslocked again

CSCvs74286

35(2g)To recover the VM copy over thedata disks and attach them to thenew VM

After enable encryption HX connectencryption status showing Caution butall of the disks enable encryptionsuccessful

-Self Encrypting Drives Service isrunning on all of the nodes

-There is no error message of encryption

- USB0 interface is up

-All SED disks showing supported 1enabled 1 locked 0

CSCvs93245

40(1b)Enable kernelpanic_on_oops as apersistent configuration This willcause the node to panic and rebootimmediately

A cluster node running HX release40(1b) may hang in the Linux kernelThis is classified as an oops and adeviation from the expected behavior

CSCvs54285

Release Notes for Cisco HX Data Platform Release 4051

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Enabling K8iscsi stack in Hx clusterCSCvs41324

Release Notes for Cisco HX Data Platform Release 4052

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Steps to enable K8 on HX cluster

Perform following steps on ALLthe controller VMs

1 Updateetcinitscvmclientconf toenable tune iscsiEnable totrue

sed -iesiscsiEnable=falseiscsiEnable=trueetcinitscvmclientconf

2 Run following initctlcommand to reloadconfiguration

initctl reload-configuration

3 Restart scvmclient process

stop scvmclient startscvmclient

4 Run following initctlcommand to mount SYSTEMDS

initctl emit --no-waitsystem-datastore-created

5 Verify that iscsiEnable tuneis set to true using thefollowing command

ps -eaf | grep scvmclient |grep -v grep

root 6241 1 1 Dec06 014007optspringpathstorfs-corescvmclient-T logEnabled=false -TlogSyslogEnabled=true -TlogEchoToScreen=false -TstatLoggingToFile=false -TstatLoggingToSyslog=true-TlogDir=varlogspringpath-TnfsBackendServerList=1010748100-T iscsiEnable=true -TiscsiUseAsync=true -T

Release Notes for Cisco HX Data Platform Release 4053

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

iscsiConfigFilePath=nfsSYSTEMistgtconfps -eaf | grep scvmclient |grep -v grep

35(2e)Following procedure can be usedto check NFS Queue Depth andincrease if needed

rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth256 lt- Low value[rootHXESXI1] vsish -e setvmkModulesnfsclientmnt[DATASTORE]propertiesmaxQDepth 1024[rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth1024 lt- Optimalvalue

This requires an ESXi host rebootto take effect

Please place the node in HyperflexMaintenanceMode and gracefullyreboot the node for the changes tobe applied

HyperFlex Datastore NFS Queue Depthshows as 256 which can lead toperformance (including latency) issues

CSCvr83056

40(1b)This is cosmetic The error will goaway once the cluster returns tohealthy status

When a single node is offline in clusterstcli cluster storage-summary shows twonodes unavailable rootSCVM~ stclicluster storage-summary

CSCvs47735

35(2g)Modify both the root and adminpassword for the SCVMs to be thesame

stcli security password set -uadmin

Cluster expansion fails at Config Installerstage when the root and admin passwordfor the storage controller (SCVM) aredifferent

CSCvs69317

35(2d)NAAfter successfully changing the DNSserver on the HX controller we still cansee the original DNS entry that was addedduring the deployment

CSCvs69154

Release Notes for Cisco HX Data Platform Release 4054

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Customers are strongly encouragedto work with TAC in order toidentify and remediate this issue

1 Change getcluster from 2m to8m inoptspringpathstorfs-mgmtstMgr-10confapplicationconfon all nodes

2 Restart stMgr on all nodes

You may see this error message after afailed upgrade or other task such asattempting to enter a node into HXmaintenance mode

CSCvs53555

40(2a)Run the following command oneach storage controller VM as root

grep -i monitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp sed -iesmonitor_dns_serverstruemonitor_dns_serversfalseoptspringpathhx-diag-toolswatchdog_configjsonampamp grep -imonitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp restart watchdog

Receive the following alertevent in HXConnect HX Controller VMHOSTNAME one or more configuredDNS servers not responding

CSCvt13947

35(2b)Delete any empty (Size 0) log filesunder varlogzookeeperversion-2ORvarlogzookeeperstandaloneversion-2

Zookeeper fails to start while Exhibitor isrunning however echo srvr returnsnothing

CSCvs21562

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

40(4e)1 De-commission the server

2 Power drain the server -REMOVE BOTH POWERCORDS ON THE BACK OFTHE SERVER FOR 10SECONDS THENREINSERTPOWERCORDS

3 Re-commission the server

M2 boot disk is missing from serverinventory after upgrade to server firmware40(4e) As a result server fails to boot toOS installed in the M2 disk The issuepersists after re-acknowledgement as wellas de-commission andre-acknowledgement of the server

CSCvs02466

Release Notes for Cisco HX Data Platform Release 4055

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Re-start the flusher manuallyMany disks taken offline by a fault ormanually resulting in ENOSPC oncluster

CSCvq32721

35(2e)Clean up fail over cluster and thenrecreate the fail over cluster Thereis no need to touch the HX storagecluster

[Hyper-V]When replicated DCwas usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2b)Manually install upgradescvmclient VIB andmake sure thatit matches with HXDP version

Upgrade from release HX 35(2a) to35(2x) 4X does not upgrade scvmclientRecovery point may not work

CSCvq54992

Management

30(1b)To recover the VM copy over thedata disks and attach them to thenew VM

[HyperV]VMshows up onmultiple nodesCSCvj22992

35(1a)1 Do not change MTU at ESXlevel on running cluster

2 Reset back to the original value

ChangingMTU from 9000 to 1500 on oneESX node causes storfs process to restarton all nodes

CSCvm99150

Release Notes for Cisco HX Data Platform Release 4056

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)After the installation (for newdeployments) or after an upgradeto 401a (for existingdeployments) perform thefollowing steps on all ControllerVMs

1 Edit the following tune files (onall Controller VMs)

optspringpathconfiglfftunes

optspringpathconfigvsi_16tbtunes

2 SetcleanerEnableSegSummaryCleaning

to false

3 After editing the above tunefiles

a SSH in to all the ControllerVMs

b Run storfstool -- -Z

c Run the below command andcheck the values Tune valueshould be true

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=true

d Type below commands andapplied the tunes changesdynamically

echo false gttmpstprocfssystemtunecleanerEnableSegSummaryCleaning

e Verify that tune values arechanged Run below commandValue should be false

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=false

f umount tmpstprocfs

A cluster with 8TB or 12TB disk drivesmay experience IO stalls for severalminutes after another node fails in thecluster

CSCvp23718

Release Notes for Cisco HX Data Platform Release 4057

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)1 Update the conf file manuallyto change to the link local addressto

2 Restart scvmclient on thecontroller and ESX

HX 35(2b) FlexVol to HX 40 CSIupgrade does not work FlexVol continuesto work

CSCvp26319

35(2a)Disable rebalance until node healsthen reenable rebalance

7-node cluster healing very slowlybecause rebalance change AMS reqs areblocking healing process

CSCvp62512

35(2a)NAWith moderate cluster storage capacityusage under site failover cluster clustermay experience an all paths (APD) downcondition

CSCvq49412

40(1a)NAThis product includes Third-partySoftware that is affected by thevulnerabilities identified by the followingCommon Vulnerability and Exposures(CVE) IDs

CVE-2016-0762 CVE-2016-6797CVE-2016-6816 CVE-2016-8735CVE-2017-5647 CVE-2017-12615CVE-2017-12616 CVE-2017-12617CVE-2017-7674 CVE-2018-1304CVE-2018-8014 CVE-2018-1336CVE-2018-8034 CVE-2018-11784CVE-2019-0232

CSCvp36364

35(2d)NAstcli cluster info command needs toprovide UCSM VIP address Currently itshows ucsm-hostcom

CSCvq11456

35(2a)No workarounds currently Thestale entries do not affect the dayto day operations of HyperFlex

HyperFlex upgrade validation failedbecause cluster has extraneous stNodes ininternal database

CSCvq32530

35(2a)NAWhen not in maintenancemode all drivesappear in HX Connect and get updated asexpected

When in maintenance mode extra drivesare shown in HX Connect with no slotnumbers

CSCvq42378

35(2b)Contact TAC to cleanup duplicatenode

Nodes duplicated in HX Connect aftercluster recreate

CSCvq61825

Release Notes for Cisco HX Data Platform Release 4058

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)NACurrently HyperFlex installs on Hyper-Vdo not contain the stcli security whitelistcommandset

CSCvq66245

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

Release Notes for Cisco HX Data Platform Release 4059

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)[Hyper-V] Converged node expansionfails for incoming node with VIC 14xx

CSCvq39523

Release Notes for Cisco HX Data Platform Release 4060

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

1 Set Jumbo frame inWindowsHypervisor for incomingnode

Get-NetAdapter storage-data-a|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

Get-NetAdapter storage-data-b|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

2 Verify all interfaces are setcorrectly to support Jumbo frames

Get-NetAdapter storage |Get-NetAdapterAdvancedProperty| RegistryKeyword -matchjumbo | ft -auto

You should receive the followingmessage

Name DisplayName DisplayValue

RegistryKeyword RegistryValue

---- ----------- ------------

--------------- -------------

vswitch-hx-storage-data Jumbo

Packet 9014 Bytes

JumboPacket 9014

storage-data-b Jumbo Packet

Bytes 9014 JumboPacket

9014

storage-data-a Jumbo Packet

Bytes 9014 JumboPacket

9014

3 Reboot only the incoming nodeor the node being expanded

4 Retry Cluster Expansion in

Release Notes for Cisco HX Data Platform Release 4061

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

The software build version posted at release will override any other local versions

bull Required vCenter upgrademdashFor enhanced security Cisco HX Data Platform Release 35(1a) or laterrequires the use of TLS 12 Therefore vCenter must be upgraded to 60 U3f or later before upgradingto Cisco HX Data Platform Release 35 or later In addition ESXi should be upgraded as required tomeet HX Data Platform compatibility requirements

bull Minimum HXDP version for upgrademdashHXData Platform clusters running 26(1a) or later may upgradedirectly to 40 using the HX Connect UI

bull Cluster ReadinessmdashEnsure that the cluster is properly bootstrapped and the updated plug-in is loadedbefore proceeding Manual cluster bootstrap is required for upgrade from a pre-35 release

bull Cluster ReadinessmdashEnsure that the cluster is properly bootstrapped and the updated plug-in is loadedbefore proceeding Manual cluster bootstrap is required for HX releases earlier than 35(1a) For moreinformation see theManual Bootstrap Upgrade Process in the Cisco HyperFlex Systems Upgrade Guidefor VMware ESXi Release 40 Do not skip this cluster bootstrap step it is required for all upgradesuntil HX Release 35(1a) Auto bootstrap is supported beginning with HX release 35(1a) For moreinformation see the Auto Bootstrap Upgrade Process from HX Connect UI in the Cisco HyperFlexSystems Upgrade Guide for VMware ESXi Release 40

Manual bootstrap is not supported on Intersight clusters

bull Initiating Upgrade―Use the HX Connect UI or CLI stcli commands when upgrading from 25(1a)or later releases Use either the CLI stcli commands or the HX Data Platform Plug-in to the vSphereWeb Client when upgrading from a pre-25(1a) release The vCenter plug-in should not be used forupgrades starting with the 25(1a) release The vCenter plug-in should not be used for upgrades startingwith the 25(1a) release

If the current cluster version is at 35(1a) or above you do not need to use the stcli command Directupgrade to 40 is possible

bull Complete your Upgrade―The self-healing (or rebalance) capability is disabled temporarily during theupgrade window If the upgrade fails you should complete the upgrade as soon as possible

bull ESXi and HXDP Compatibility―Ensure your cluster is running a compatible version of ESXi basedon the running the HXData Platform version (see the section Software Requirements for VMware ESXi)ESXi compatibility is determined by the major version and update release of ESXi It is generally bestto upgrade HXDP and ESXi together if combining the upgrade operations into a single optimized rebootWhen running a split upgrade first upgrade the HX Data Platform then proceed to upgrade ESXi

bull If Upgrading to vSphere 65

bull Certain cluster functions such as native and scheduled snapshotsReadyClones and Enter or Exit HX Maintenance Mode will not operatefrom the time the upgrade is started until the HX Data Platform upgrade iscomplete

bull After upgrading ESXi using the offline zip bundle use the ESX ExitMaintenance Mode option The HX Exit Maintenance Mode option doesnot operate in the vSphere Web Client until the HX Data Platform upgradeis complete

Note

Release Notes for Cisco HX Data Platform Release 4020

Guidelines and Limitations

bull vSphere 60 UpgradesmdashUsers on vSphere 60 migrating to 65 upgrade components in the followingorder

1 Upgrade HX Data Platform and UCS firmware

2 Upgrade HX Data Platform and ESXi

3 Upgrade HX Data Platform only first then upgrade ESXi or UCS firmware or both

bull M4 Server Firmware UpgradesmdashUpgrade server firmware to ensure smooth operation and to correctknown issues Specifically newer SAS HBA firmware is available in this release and is recommendedfor long-term stability

bull Users are encouraged to upgrade to 31(3c) C-bundle or later whenever possible

bull Users running C-bundle versions before 31(2f) must upgrade server firmware by performing acombined upgrade of UCS server firmware (C-bundle) to 31(3c) or later and HX Data Platform to25 Do not split the upgrade into two separate operations

bull If the cluster is already on 31(2f) C-bundle or later you may perform an HX Data Platform onlyor combined upgrade as required

bull M5 Server Firmware UpgradesmdashM5 generation servers must run firmware version 32(2d) or later

bull Firmware Downgrades mdashDowngrading UCSM from the HX-installer is not supported

bull M4M5 Mixed DomainsmdashA mixed domain occurs when a new separate M5 cluster is installed underthe same UCS domain that contains existing M4 clusters Under these conditions orchestrated UCSserver firmware upgrade will not operate until Cisco HX Data Platform Release 26 or later is installedon the M4 clusters Therefore it is best practice to first upgrade UCS server firmware to the latest 31(3)or 32(2) patch release before adding a new M5 cluster to the existing UCS domain Additionally any17 HX Data Platform clusters must first be upgraded before adding any new M5 clusters to the samedomain

bull Maintenance WindowmdashIf upgrading both HX Data Platform and UCS firmware you can select eithera combined or split upgrade through the vSphere HX Data Platform Plug-in depending on the length ofthe maintenance window Cisco UCSManager infrastructure upgrade is only supported using AutoInstalland the direct server firmware upgrade should be performed only through the upgrade orchestrationframework provided by the HX Data Platform Plug-in

bull Unsupported Self-Encrypting Drives (SEDs)mdashIf adding or replacing self-encrypting drives (SEDs)that have been recently qualified in newer versions of HX Data Platform insert the new drives only afterupgrading HX Data Platform to a compatible version All drives must be SED drives mixing SED andnon-SED is not supported

bullbull Enabling External Host AccessmdashWith Cisco HX Data Platform Release 40(1a) port 445 on themanagement network is blocked for enhanced security Note that prior to 40 port 445 port was openenabling external host access If you are upgrading to 40(1a) from a prior release and would like tocontinue external host access you can use a utility to open select hosts For more information aboutenabling external host access see the ConfiguringHyperFlex Share to SCVMM section in the InstallationGuide for Microsoft Hyper-V

Release Notes for Cisco HX Data Platform Release 4021

Guidelines and Limitations

Mixed Cluster Expansion Guidelinesbull Expanding existing M4 cluster with M5 converged nodes is supported

bull Expanding existing M5 cluster with M4 converged nodes is not supported

bull Expanding existing mixed M4M5 cluster with M4 or M5 converged nodes is supported

bull Adding any supported compute-only nodes is permitted with all M4 M5 and mixed M4M5 clustersusing the HX Data Platform 26 or later Installer Some example combinations are listed here manyother combinations are possibleExample combinationsExpand mixed M4M5 cluster with compute-only B200 C220 C240 M4M5Expand M4 cluster with compute-only B200 M5 C220 M5 C240M5

bull Only expansion workflow is supported to create a mixed cluster Initial cluster creation with mixedM4M5 servers is not supported

bull All M5 servers must match the form factor (220240) type (HybridAF) security capability (Non-SEDonly) amp disk configuration (QTY capacity and non-SED) of the existing M4 servers

bull HX220-M5 will use a maximum of 6 capacity disks (2 disk slots to remain empty) when mixedwith HX220-M4

bull HX Edge SED LFF Hyper-V and Stretched Clusters do not support mixed M4 and M5 clusters

Security FixesThe following security issues are resolved

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-13132 CVE-2019-9924CSCvq6313840(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-11719 CVE-2019-11727CVE-2019-11729

CSCvq7124040(2a)

Release Notes for Cisco HX Data Platform Release 4022

Mixed Cluster Expansion Guidelines

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-1125CSCvr0633940(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2015-9383 CVE-2018-14498CVE-2018-20406 CVE-2018-20852CVE-2019-10160 CVE-2019-13117CVE-2019-13118 CVE-2019-14287CVE-2019-14973 CVE-2019-15903CVE-2019-17546 CVE-2019-18197CVE-2019-18218 CVE-2019-5010CVE-2019-5094 CVE-2019-5481CVE-2019-5482 CVE-2019-9636CVE-2019-9740 CVE-2019-9947CVE-2019-9948

CSCvs0609440(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2017-13168 CVE-2017-18174CVE-2017-18216 CVE-2018-10876CVE-2018-10877 CVE-2018-10878CVE-2018-10879 CVE-2018-10880CVE-2018-10881 CVE-2018-10882CVE-2018-10902 CVE-2018-10938CVE-2018-12233 CVE-2018-12896CVE-2018-13053 CVE-2018-13094CVE-2018-13096 CVE-2018-13405CVE-2018-13406 CVE-2018-14609CVE-2018-14617 CVE-2018-14633CVE-2018-14734 CVE-2018-15572CVE-2018-15594 CVE-2018-16276CVE-2018-16658 CVE-2018-17182CVE-2018-17972 CVE-2018-18021CVE-2018-18690 CVE-2018-18710CVE-2018-6554 CVE-2018-6555CVE-2018-9363

CSCvp6501940(2a)

This vulnerability is due to insufficientHTML iframe protection An attackercould exploit this vulnerability bydirecting a user to an attacker-controlledweb page that contains amaliciousHTMLiframe A successful exploit could allowthe attacker to conduct clickjacking orother clientside browser attacks

NACSCvo9851640(2a)

Release Notes for Cisco HX Data Platform Release 4023

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerability is due to insufficientauthentication for the statistics collectionservice An attacker could exploit thisvulnerability by sending properlyformatted data values to the statisticscollection service of an affected deviceA successful exploit could allow theattacker to cause the web interfacestatistics view to present invalid data tousers

NACSCvj9558440(2a)

The vulnerability is due to insufficientCSRF protections for the web UI on anaffected device An attacker could exploitthis vulnerability by persuading a user ofthe interface to follow a malicious linkA successful exploit could allow theattacker to perform arbitrary actions withthe privilege level of the affected user

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvp2434340(2a)

Vulnerabilities with a version of cJSONidentified by CVE-2019-11834 andCVE-2019-11835

For more information see the relatedCisco Security Advisory

CVE-2019-11834 CVE-2019-11835CSCvq1994940(2a)

HX ESXi image patches for MCEPSCand TAA vulnerabilities with VMwareESXi identified by CVE-2018-12207 andCVE-20190-11135

For more information see the relatedCisco Security Advisory

CVE-2018-12207 CVE-2019-11135CSCvr5439840(2a)

CP networking vulnerabilities affectingthe Linux kernel identified byCVE-2019-11477 CVE-2019-11478CVE-2019-11479

For more information see the relatedCisco Security Advisory

CVE-2019-11477 CVE-2019-11478CVE-2019-11479

CSCvq1954640(2a)

Qualification of Microsoft securitypatches for vulnerabilities withMicrosoftHyper-V hypervisor identified withCVE-2018-12207 CVE-2019-11135

For more information see the relatedCisco Security Advisory

CVE-2018-12207 CVE-2019-11135CSCvr5439940(2a)

Release Notes for Cisco HX Data Platform Release 4024

Security Fixes

DescriptionCVEDefect IDRelease

OpenSSH vulnerabilities identified byCVE-2019-6111

For more information see the relatedCisco Security Advisory

CVE-2016-10708 CVE-2018-15473CVE-2018-20685 CVE-2019-6109CVE-2019-6111

CSCvp7646340(2a)

Multiple vulnerabilities associated withOpenSSL and LibNSS

For more information see the relatedCisco Security Advisory

CVE-2016-2105 CVE-2016-2106CVE-2016-2107 CVE-2016-2109CVE-2016-2176 CVE-2016-2177CVE-2016-2178 CVE-2016-2179CVE-2016-2180 CVE-2016-2181CVE-2016-2182 CVE-2016-2183CVE-2016-6302 CVE-2016-6303CVE-2016-6304 CVE-2016-6305CVE-2016-6306 CVE-2016-6307CVE-2016-6308 CVE-2016-7055CVE-2016-8610 CVE-2017-3731CVE-2017-3732 CVE-2017-3735CVE-2017-3736 CVE-2017-3737CVE-2017-3738 CVE-2018-0495CVE-2018-0732 CVE-2018-0734CVE-2018-0735 CVE-2018-0737CVE-2018-0739 CVE-2018-12384CVE-2018-12404 CVE-2018-5407CVE-2019-1559

CSCvp6667940(2a)

libpng vulnerability identified byCVE-2016-10087

For more information see the relatedCisco Security Advisory

CVE-2016-10087CSCvp6655540(2a)

Vulnerabilities associated with libjpg andlibpng

For more information see the relatedCisco Security Advisory

CVE-2014-9092 CVE-2016-3616CVE-2017-15232 CVE-2018-11212CVE-2018-11213 CVE-2018-11214CVE-2018-1152 CVE-2018-13785

CSCvp3458640(2a)

Vulnerabilities associated with Glibidentified by CVE-2018-16428 andCVE-2018-16429

For more information see the relatedCisco Security Advisory

CVE-2018-16428 CVE-2018-16429CSCvp3120740(2a)

Multiple vulnerabilities associated withcurl expat python 27 python 35 36and 37 freetype and giflib

For more information see the relatedCisco Security Advisory

CVE-2019-15133 CVE-2019-15903CVE-2019-5010 CVE-2019-5481CVE-2019-5482 CVE-2019-9636CVE-2019-9740 CVE-2019-9947CVE-2019-9948

CSCvr3690340(2a)

Release Notes for Cisco HX Data Platform Release 4025

Security Fixes

DescriptionCVEDefect IDRelease

Multiple third party softwarevulnerabilities

For more information see the relatedCisco Security Advisory

CVE-2019-14379 CVE-2019-12384CVE-2019-14439

CSCvq9203240(2a)

Vulnerabilities associated with elfutils

For more information see the relatedCisco Security Advisory

CVE-2018-16062 CVE-2018-16402CVE-2018-16403 CVE-2018-18310CVE-2018-18520 CVE-2018-18521CVE-2019-7149 CVE-2019-7150CVE-2019-7665

CSCvq4325040(2a)

Vulnerabilities associated with vim

For more information see the relatedCisco Security Advisory

CVE-2017-5953 CVE-2019-12735CSCvq4323040(2a)

Vulnerabilities associated with sqlite3

For more information see the relatedCisco Security Advisory

CVE-2016-6153 CVE-2017-10989CVE-2017-13685 CVE-2017-2518CVE-2017-2519 CVE-2017-2520CVE-2018-20346 CVE-2018-20505CVE-2018-20506 CVE-2019-8457CVE-2019-9936 CVE-2019-9937

CSCvq4321340(2a)

Vulnerabilities associated with glib20

For more information see the relatedCisco Security Advisory

CVE-2018-20843CSCvq4320940(2a)

Vulnerabilities associated with expat

For more information see the relatedCisco Security Advisory

CVE-2018-20843CSCvq4320540(2a)

Vulnerabilities associated with bzip2

For more information see the relatedCisco Security Advisory

CVE-2016-3189 CVE-2019-12900CSCvq4319440(2a)

Mulitple Vulnerabilities associated wtihNodeJS

For more information see the relatedCisco Security Advisory

CVE-2018-12115 CVE-2018-0734CVE-2018-5407 CVE-2018-12120CVE-2018-12121 CVE-2018-12122CVE-2018-12123 CVE-2018-12116CVE-2019-5737 CVE-2019-5739

CSCvq1069440(2a)

Vulnerabilities associated with PalletsJinja strformat_map and identified byCVE-2019-10906 and CVE-2016-10745

For more information see the relatedCisco Security Advisory

CVE-2019-10906 CVE-2016-10745CSCvq1038840(2a)

Release Notes for Cisco HX Data Platform Release 4026

Security Fixes

DescriptionCVEDefect IDRelease

Multiple Security Vulnerabilities

For more information see the relatedCisco Security Advisory

CVE-2014-8501 CVE-2014-9939CVE-2015-9262 CVE-2016-10087CVE-2016-2226 CVE-2016-4487CVE-2016-4488 CVE-2016-4489CVE-2016-4490 CVE-2016-4491CVE-2016-4492 CVE-2016-4493CVE-2016-6131 CVE-2018-10963CVE-2018-13785 CVE-2018-17100CVE-2018-17101 CVE-2018-18557CVE-2018-18661 CVE-2018-7456CVE-2018-8905

CSCvq0917840(2a)

Vulnerabilities associated withlibseccomp

For more information see the relatedCisco Security Advisory

CVE-2019-9893CSCvq0756840(2a)

Vulnerabilities associated withgdk-pixbuf

For more information see the relatedCisco Security Advisory

CVE-2017-12447CSCvq0675540(2a)

Vulnerabilities associated withpython-crypto

For more information see the relatedCisco Security Advisory

CVE-2018-6594CSCvp9381740(2a)

Vulnerabilities associated with wget

For more information see the relatedCisco Security Advisory

CVE-2018-20483 CVE-2019-5953CSCvp8672140(2a)

Multiple vulnerabilities associated withPython Crypto

For more information see the relatedCisco Security Advisory

CVE-2018-6594CSCvp6674840(2a)

Vulnerabilities associated withsensible-utils

For more information see the relatedCisco Security Advisory

CVE-2017-17512CSCvp6673440(2a)

Multiple vulnerabilities associated withPython

For more information see the relatedCisco Security Advisory

CVE-2018-1000030CSCvp6668940(2a)

Release Notes for Cisco HX Data Platform Release 4027

Security Fixes

DescriptionCVEDefect IDRelease

Multiple patch vulnerabilities identifiedby CVE-2016-10713

For more information see the relatedCisco Security Advisory

CVE-2016-10713CVE-2018-1000156 CVE-2018-6951

CSCvp6667240(2a)

Vulnerabilities associatedwith Little CMSidentified by CVE-2016-10165

For more information see the relatedCisco Security Advisory

CVE-2016-10165 CVE-2018-16435CSCvp6666440(2a)

Vulnerabilities associated with TenableScan identified byCVE-2011-5325CVE-2019-11068CVE-2016-7076CVE-2017-1000368

For more information see the relatedCisco Security Advisory

CVE-2011-5325 CVE-2016-7076CVE-2017-1000368CVE-2019-11068

CSCvp6474640(2a)

Vulnerabilities associated with libxml2294 and earlier as used in XMLSec1223 and earlier and other products andidentified by CVE-2016-9318

For more information see the relatedCisco Security Advisory

CVE-2016-9318 CVE-2017-16932CVE-2017-18258 CVE-2018-14404CVE-2018-14567

CSCvp3479240(2a)

Vulnerabilities associated with LFTPRemote File Names Unauthorized AccessVulnerability and identified byCVE-2018-10916

For more information see the relatedCisco Security Advisory

CVE-2018-10916CSCvp2926640(2a)

Vulnerabilities associated withCryptographyDeprecationWarning signerand verifier have been deprecated

For more information see the relatedCisco Security Advisory

CVE-2018-7750CSCvo3409740(2a)

Multiple Vulnerabilities in python python27 35

For more information see the relatedCisco Security Advisory

CVE-2019-16056CSCvr3979340(2a)

Release Notes for Cisco HX Data Platform Release 4028

Security Fixes

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toexecute commands as the root user

The vulnerability is due to insufficientinput validation An attacker could exploitthis vulnerability by connecting to thecluster service manager and injectingcommands into the bound process Asuccessful exploit could allow the attackerto run commands on the affected host asthe root user

Cisco has released software updates thataddress this vulnerability There areworkarounds that address thisvulnerability

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvq2417640(1b)35(2g)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

The vulnerability is due to an unprotectedlistening interface An attacker couldexploit this vulnerability by connectingto the listening interface and injectingcommands to the bound process Anexploit could allow the attacker to runcommands on the affected host as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvj9560640(1b)35(2g)

Release Notes for Cisco HX Data Platform Release 4029

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerabilities associated with theJVM18U121memory leak found duringVC alarms (concurrent 40 calls usingREST API)

CVE-2017-10053 CVE-2017-10067CVE-2017-10074 CVE-2017-10078CVE-2017-10081 CVE-2017-10087CVE-2017-10089 CVE-2017-10090CVE-2017-10096 CVE-2017-10101CVE-2017-10102 CVE-2017-10107CVE-2017-10108 CVE-2017-10109CVE-2017-10110 CVE-2017-10111CVE-2017-10115 CVE-2017-10116CVE-2017-10118 CVE-2017-10135CVE-2017-10176 CVE-2017-10193CVE-2017-10198 CVE-2017-10243CVE-2017-10274 CVE-2017-10281CVE-2017-10285 CVE-2017-10295CVE-2017-10345 CVE-2017-10346CVE-2017-10347 CVE-2017-10348CVE-2017-10349 CVE-2017-10350CVE-2017-10355 CVE-2017-10356CVE-2017-10357 CVE-2017-10388CVE-2017-3509 CVE-2017-3511CVE-2017-3526 CVE-2017-3533CVE-2017-3539 CVE-2017-3544CVE-2018-2579 CVE-2018-2582CVE-2018-2588 CVE-2018-2599CVE-2018-2602 CVE-2018-2603CVE-2018-2618 CVE-2018-2629CVE-2018-2633 CVE-2018-2634CVE-2018-2637 CVE-2018-2641CVE-2018-2663 CVE-2018-2677CVE-2018-2678 CVE-2018-2783CVE-2018-2790 CVE-2018-2794CVE-2018-2795 CVE-2018-2796CVE-2018-2797 CVE-2018-2798CVE-2018-2799 CVE-2018-2800CVE-2018-2814 CVE-2018-2815CVE-2018-2952 CVE-2018-3136CVE-2018-3139 CVE-2018-3149CVE-2018-3150 CVE-2018-3169CVE-2018-3180 CVE-2018-3183CVE-2018-3214 CVE-2019-2422

CSCvo8899740(1b)

Tomcat and Nginx logs are not beingcollected in the support bundle generatedthrough HX Connect in Release 35

NACSCvm5803140(1b)

The vulnerabilities associated with thelibmspack software package versionincluded in Cisco HX Data Platform

CVE-2018-18584

CVE-2018-18585

CSCvn3511940(1a)

Release Notes for Cisco HX Data Platform Release 4030

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerabilities associated withFasterXML Jackson-Databind TimeValueField Denial of Service

CVE-2018-14719

CVE-2018-14720

CVE-2018-1000873

CVE-2018-14721

CVE-2018-19360

CVE-2018-19362

CVE-2018-19361

CVE-2018-14718

CSCvn8228240(1a)

The vulnerabilities associated with theOpenSSL Protocol software packageversion included in Cisco HX DataPlatform

CVE-2013-3587CSCvo0505440(1a)

The vulnerabilities associated withThird-Party Software Denial of Service

CVE-2018-16487

CVE-2018-19361

CSCvo2781840(1a)

The vulnerabilities associated withLinuxkernel for Ubuntu 1710

CVE-2018-1092

CVE-2018-7492

CVE-2018-8087

CVE-2018-1068

CVE-2018-8781

CSCvm5314935(2a)

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)25(1d)stcli services dns remove should remove the DNS serverinfo from the interface files

CSCvs70967

40(2b)40(1b)HX 40(1b) new deployment from Intersight productioncloud

User receives following error while enabling CSI(Kubernetes integration)

Failure occurred during volume_access_enable Errorwas Nonzero exit code 1

CSCvt10522

40(2b)40(2a)Receive the following alertevent in HX Connect HXController VM HOSTNAME one or more configuredDNS servers not responding

CSCvt13947

Release Notes for Cisco HX Data Platform Release 4031

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)40(2a)API UI CLI show few Drive slots Empty after upgradeto 402a

The drives are seen in the sysmtool lsscsi and stclicommand outputs

The cluster is healthy with no errors

CSCvt14914

40(2b)35(2d)After successfully changing the DNS server on the HXcontroller we still can see the original DNS entry thatwas added during the deployment

CSCvs69154

40(2b)35(2e)HyperFlex Connect and VCenter show APD alarms forHX and non HX datastores This implies there is an issuewith HyperFlex when there may not be

CSCvs30080

40(2b)40(2a)An issue where all the disks in the node get locked afterrebooting the node

We successfully unlocked using sed-clientsh -Ucommand but wanted to test with another reboot anddrives locked again

CSCvs74286

40(2b)40(2a)When running stcli license commands on HyperFlexthe following error is seen

rootSpringpathControllertmp stcli license show all

Show smart licensing failed

Smart Agent is not ready please wait a minute and tryagain

CSCvt13929

40(2b)40(2a)After upgrading to HX 40(2a) the first RoboEdge nodethat gets rebooted then ignores persistent drivesPIDHX-HD24TB10K4KN

Cluster will be in degraded state and have reducedcapacity

CSCvt20144

40(2b)35(2b)Zookeeper fails to start while Exhibitor is runninghowever echo srvr returns nothing

CSCvs21562

40(2b)40(2a)When performing an HyperFlex upgrade a validationwarning may occur due to the host not having EnterprisePlus or Enterprise hypervisor licensing

Upgrade Validation Warning

ESXi host esx1labtest should be configured withVMware Enterprise license for upgrade to continue

CSCvs91787

Release Notes for Cisco HX Data Platform Release 4032

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)35(2g)Cluster expansion fails at Config Installer stage whenthe root and admin password for the storage controller(SCVM) are different

CSCvs69317

40(2b)35(2g)Panic while upgrading ESXiCSCvt06983

40(2b)40(2a)A cluster node running may hang in the Linux kernelThis is classified as an oops and a deviation from theexpected behavior

CSCvs54285

40(2b)35(2g)Rebalance failing on 10+10 node HX 35(2g) stretchcluster

CSCvs69007

40(2b)35(2d)The cluster becomes inaccessible to the IOVisorCSCvr54687

40(2b)40(2a)Panic on the storage controllerCSCvt61297

40(2b)40(2a)The size of support bundle is very large when collectedwith storfs-support command

CSCvt63306

40(2b)40(2a)storfs crash associated withFileSystemUsageWarningEvent orFileSystemUsageAlertEvent due to high usage ofvarstv above 80

CSCvt72807

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

40(2a)35(2d)In normal scenarios HX upgrade gets pendingACK afterupdating service profile In this case UCS only upgradewas stalled because after updating service profile HXupgrade was not getting pending ACK and it was alwayswaiting state

CSCvo39912

40(2a)35(1a)When using motherBoardReplace-12 to clean ZK fromold stNodepNodes resulted in unmounted datastore andmaking the size of the Hyperflex Datastores 0 resultingin all VMs in the cluster going offline

CSCvq39471

40(2a)26(1e)Upgrading cluster and getting error

Failed upgrade validations Checking vCenterconfiguration Reason Upgrade validations failed DRSFault Insufficient resources to satisfy configured failover

CSCvm77294

Release Notes for Cisco HX Data Platform Release 4033

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)The cluster expand fails on a node with DR replicationconfigured

When a HX cluster which has DR replication configuredis expanded we see the installer UI pulling in thereplicationVLAN information instead of themanagementVLAN information Even if we change that informationto the correct mgmt VLAN id and name it it does notseem to work as the node is configured with the VLANof the replicationVLAN in ESXi This leads to the failureof the node add with the host unreachable error

CSCvo70650

40(2a)35(1a)Customer reported that server didnt complete thefirmware upgrade automatically one by one

The user manually needs to put the host into maintenancemode and they manually acknowledged the pendingrequests to finish the UCS firmware upgrade

Maintenance policy is set to default (User-ack) as perthe design

CSCvo91624

40(2a)35(1i)When the cluster is in Failed state and stcli noderemove is attempted the output shows successful thoughthe node failed to get removed from the cluster

CSCvo93017

40(2a)35(1a)HyperFlex cluster upgrade may fail during validationwith a DRS Validation failed error

CSCvp31021

40(2a)35(2b)HX cluster expansion will fail with the following errormessage

MAC address pool configuration failure150[ErrorDescription] bad address block range definitioncollision

CSCvp58318

40(2a)35(2b)Memory usage by carbon cacheCSCvq34873

40(2a)35(2g)Upgrading ESXi cluster fails with error Nodemaintenance mode failed

CSCvr03240

40(2a)35(2e)storfs process does not automatically start on Exit HXMaintenance Mode or other tasks which restart StorageController

CSCvr88978

40(2a)35(2a)Post install script fails with message

Failed to execute ipmitool on HX node

CSCvp10707

40(2a)40(1a)HyperFlex expansionworkflow doesnt pull VLANnamecorrectly

CSCvp46539

Release Notes for Cisco HX Data Platform Release 4034

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)During HX cluster deployment - Validate ClusterCreation phase - HX Installer might fail with the error

fromvarlogspringpathstDeploylog

CSCvq45087

40(2a)35(2b)HX Installer fails to login to the SOL for configuringESX hosts

CSCvq91380

40(2a)35(2d)If DRS is enabled with configuration parameters otherthen fully automated ie manual or partially automatedthan this will change to fully automated during clusterexpansion

CSCvr44222

40(2a)35(1a)One of the validation task name has a typo The taskname is Validating node for mixed of SED and non-SEDdisks

It should be Validating node for mix of SED andnon-SED disks

CSCvp82175

40(2a)30(1i)This combination of using smaller NVMe drives is notsupported in expand operation For example if the drivesare only 375GB and they cannot be added to an existingcluster with larger caching SSD

CSCvo12359

40(2a)35(1a)HyperFlex includes a version of OpenSSL that is affectedby the vulnerabilities identified by the followingCommon Vulnerability and Exposures

CVE-2018-0495

CSCvp66679

40(2a)30(1e)SDK invoke exception when taking a native snapshot ona VM with GPU pci passthrough attached

CSCvo00511

40(2a)30(1i)Node replacement script fails due to EAM errorCSCvo62867

40(2a)35(1a)While pairing a cluster (HX release gt= 35) and cluster(HX release lt 35) remote replication network test failson the cluster (HX release lt 35)

CSCvo79760

40(2a)30(1b)Fixed the issue in the latest support-workflow bundleThis should not be hit anymore

CSCvo87061

40(2a)35(2a)On a 3 node cluster the MBreplace script fails as thescript looks for HEALTHY cluster status

CSCvo87080

40(2a)35(2a)MbReplace script (tar) hangs when running on 352b or40

CSCvp12359

Release Notes for Cisco HX Data Platform Release 4035

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)HX replication cleanup failing with error NA 35(2a) -INFODR state is not clean

CSCvp63958

40(2a)40(1a)The check_stig_parameters API incorrectly shows thecompliance state of the cluster as not STIG compliant

CSCvp66277

40(2a)35(2c)The snapshot creation on CBT enabled VM fails witherror Failed in vmreparent vmkfstools clone1

CSCvq06952

40(2a)35(2a)HyperFlex GUI when setting up remote DR-replicationpartnerpeer should show remote replication partner TCPport reachability test results

This will provide quick upfront connectivity results thatfrequently takes customer a while to manually check andconfirm

If reachability tests fail an info window and HX alertshould trigger that list the destination peer IP and thespecific port(s) that could not be reache

CSCvo60587

40(2a)35(2b)If you have more than one IP pool configured for areplication network and local replication network testsare failing you might be running into this issue

CSCvr67130

40(2a)35(2a)Skip task option in MBreplace and Software redeployscripts The scripts should have the option to skip tasksas user inputs and not by modifying the json file

CSCvp05204

40(2a)40(1a)Error noticed on SCVM Console print_req_error IOerror dev fd0 sector 0 on HyperV

It doesnt come up on SSH session

CSCvq34357

40(2a)30(1d)The VMNetwork switch is already created as part of theinstall process

Ideally the ip address assignment should be done duringthe initial install process as well No post install stepsshould be required altrnatively it could be done as partof a post install script

CSCvk36222

40(2a)25(1c)storfs service impacted during HX upgrade from HX21(1) to 25(1c)

HX node which is not being upgraded storfs servicestopped causing outage

CSCvg53223

40(2a)35(1b)HX Stretch Cluster Witness VM reverts to DHCP onreboot

CSCvp37536

40(2a)35(2b)HyperFlex Stretch Cluster Witness does not getprogrammed with an NTP server

CSCvp46578

Release Notes for Cisco HX Data Platform Release 4036

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2b)HyperFlex cluster does not heal after maintenance onnodeserver

CSCvr18528

40(2a)40(2a)High percentage of packet loss on the eth1 data interface9K MTU

CSCvr97089

40(2a)35(2f)The HyperFlex cluster healing state is stuck at 87 andis unable to progress

CSCvr16760

40(2a)40(2a)In some circumstances two replications can occur forthe same VM

From the GUI the replication status for this VM willshow stuck in an In Progress state but there are nojobs stuck in replication layer Subsequent Replicationsshould succeed

CSCvq22898

40(2a)35(2a)After an unexpected power outage the hyperflex clusteris unable to come up correctly

All processes are running time is synced and all nodescan vmkping each other as required

CSCvp13990

40(2a)40(1a)When specifying IP address instead of node ID stclinode maintenanceMode commands fail

CSCvp33657

40(2a)40(1a)2 node edge cluster if experiencing Intersightarbitratorconnectivity issues should NOT allowHXMaintenancemode to initiate as this will bring the cluster down

CSCvr31573

40(2a)35(2c)Old files in varsupportZKTxnlog are not purged withthe daily zklog-cleanup cron job

CSCvr89066

40(2a)35(2a)When a node is in MaintenanceMode any disk removalor replacement will be reflected in UI only after the nodeis brought back frommaintenancemode This is becausestorfs is not running on the node in maintenance andwill not be able to detect disk activities until it is broughtout of Maintenance Mode

CSCvo86431

40(2a)35(2c)Cluster might go offline and stop serving IO requestsrootcvm~ stcli cluster storage-summary--detailGet cluster storage summary failedjavanetConnectException Connection refused19216814210010207

CSCvr01645

40(2a)35(2d)A stretch cluster deployment fails with a messageFormatting nodes Node in Use on the Hyperflex installerand crmZoneType value is seen as 1 in storfscfg oroptspringpathconfigstretchtunes

CSCvq80340

Release Notes for Cisco HX Data Platform Release 4037

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)30(1c)During upgrade we upload the upgrade packagedownloaded from Ciscocom Upgrade packages are offile type tgz hxconnect acceptsstorfs-packages-401a-33028tar file a different type ofcompression and allows upgrade to start and fail whenit cannot decompress the file for bootstrap and scvmclientupgrade on ESXi

CSCvq63888

40(2a)40(2a)While attempting to install hyperflex via intersight thefollowing error is seen

Failed in Task Add hosts to vCenter Cluster with ErrorTry adding hosts manually to vCenter and retry Failedto get addressFailed to host by namelookupHostnamecompanycom on 000053 read udp12700145430-gt12700153 read connection refused

CSCvr52098

40(2a)35(2a)CoreAPI call may time out for cluster management APIwhile doing cluster create validation

CSCvo92952

40(2a)35(2d)Validation fails with Mixed mode expansion checkdue to Enhanced vMotion Compatibility Incompatibilitywhen the ESXi version is different

CSCvq95460

40(2a)30(1i)This failure is seen in installer when custom workflowis used hypervisor configuration + deploy (clean diskpartition=no) at Installing software packages on storagecontroller VM

CSCvr66309

40(2a)35(2d)The HX Installer apparently replaced a comma bysemicolon on the VLAN ID range while exporting theconfiguration to a JSON file

CSCvq93831

40(2a)40(1a)When using unsupported browsers HX Connect userswill not be able to login These users see an errorindicating unauthorized user

CSCvp22693

Release Notes for Cisco HX Data Platform Release 4038

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)40(1a)A vulnerability in the web-based management interfaceof Cisco HyperFlex Software could allow anunauthenticated remote attacker to conduct a cross-siterequest forgery (CSRF) attack on an affected system

The vulnerability is due to insufficient CSRF protectionsfor the web UI on an affected device An attacker couldexploit this vulnerability by persuading a user of theinterface to follow a malicious link A successful exploitcould allow the attacker to perform arbitrary actions withthe privilege level of the affected user

Cisco has released software updates that address thisvulnerability There are no workarounds that address thisvulnerability

This advisory is available at the following link

httpstoolsciscocomsecuritycentercontentCiscoSecurityAdvisorycisco-sa-20190807-hypflex-csrf

CSCvp24343

40(2a)35(2d)Add message to HX Connect Progress flow to notAcknowledge Pending Activities and reboot the serverson UCSM

HX Connect is doing a controlled rolling server upgradein the background

CSCvq22844

40(2a)40(1b)Compute Node expand operation fails due to incorrectcluster name

CSCvr43786

40(2a)26(1e)RF-2 cluster shutdown during data resynchronizationafter a node failure (non storfs) followed by multipledisk read failures and a HardBlacklist of a disk

CSCvp41241

40(2a)40(1a)HyperFlex Edge nodes do not properly set the ESXihostname during deployment

CSCvo13143

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4039

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4040

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1b) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401b-33133p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4041

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4042

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1b)40(1a)While running the HyperFlex installer with WindowsServer Hyper-V cluster creation process fails with theerror Failure occurred during ClusterCreation process Unable to post thecontent to the downstream This symptomis encountered while deploying HyperFlex cluster withHyperFlex nodes configuredordered with Cisco VIC1457 MLOM (PID HX-MLOM-C25Q-04) andWindows Server Datacenter or Core with Hyper-Vversion 2016 or 2019

CSCvp64140

Release Notes for Cisco HX Data Platform Release 4043

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)35(2b)Adding Micron 5200 drive to a cluster fails to increasecluster capacity

CSCvo69067

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4044

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4045

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1a) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401a-33028p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4046

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4047

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1a)30(1d)Cluster instability when disks of different sector sizeplaced in HX node

CSCvk17250

40(1a)

35(2c)

35(1a)When logged in using a local HX user account insteadof a Virtual Center account an error message appearsintermittently indicating Virtual Centerunreachable or Resource informationcannot be updated when VC is reachable

CSCvo36198

40(1a)

35(2b)

30(1d)HXDP does not work with EMC RecoverPoint - needsto support VMware API (FSS-Readdir)

CSCvk38003

Release Notes for Cisco HX Data Platform Release 4048

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)

35(2a)

30(1i)

25(1c)

25(1d)

Zookeeper (Exhibitor) process on HyperFlex StorageController VMs may fail to respond to requests whenvarzookeeper has low or no free space

CSCvm90352

40(1a)26(1c)Use Asynchronous consolidation for HX SnapshotsCSCvn02151

40(1a)35(2b)HX Quiesced Snapshot for Backup VendorsCSCvo90713

40(1a)35(2a)The cluster creationcluster expansion workflow stopswith the following error message at the validation step

FIRMWARE-Check UCSC-SAS-M5HD

FIRMWARE-Check UCSC-SAS-M5HD Required00000029000000320000003500000050 Found00000058

Action Needed Update the Controller Firmware toRequired Version

CSCvn17787

35(2b)30(1d)Kernel migration fails when a local datastore is searchedfor in the ESXi host

CSCvn73127

35(2a)26(1b)A node shuts down when two disks are replaced (acaching disk and another capacity disk) where thecapacity disk is inserted first and then the caching diskis inserted

CSCvk46364

Hyper-V

40(1a)35(2a)Cluster expansion may fail with an error code 500 -operation timed out

CSCvn28721

35(2b)35(2a)During upgrade remove the VLAN on team created foruser vSwitch During fresh installation only one VLANtag is set to the vSwitch and team although multipleVLANs were entered

CSCvn54300

35(2b)35(2a)While upgrading a Hyper-V cluster on account of a rarerace condition between the stUpgradeService andZookeeper servers the upgrade orchestration throws anupgrade validation error and the upgrade process isaborted

CSCvn60486

Release Notes for Cisco HX Data Platform Release 4049

Resolved Caveats in Release 40(1a)

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Startrestart EAM service invCenter

When EAM service is not runningvCenter registration is failing

CSCvt55712

35(2a)Contact Cisco TAC if you believeyou are seeing similar conditions

HyperFlex Stretch Cluster saw poorapplication performance due to writelatency cluster remained unhealthy andrebalance was stuck

This defect is currently being used forinvestigative purposes

CSCvt45344

35(2g)Remove log files fromvarlogspringpath folder when thelog files are not getting rotatedStop the relevant service removelarge log files fromvarlogspringpath folder andrestart the service

varstv folder in Controller VM maybecome full

CSCvt36374

40(2b)Restarting the stMgr will initializeit again

In a Cluster configured with DR networkStMgr may not get initialized and stuckin a deadlock while enabling the IPTablerules (can be verified from the stMgrlog)

CSCvt89709

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Click recover for the selectedVM again under the standalonemode This will move the VM torecovered state

When the user recovers a VM protectedin a group it along with other VMs in thegroup move to recovering state instandalonemode The selectedVM shouldmove to recovered state

CSCvs75553

35(2e)Clean up the fail over cluster andthen recreate the fail over clusterNo need to touch the HX storagecluster

Hyper-V When replicated DC was usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2d)40(1a)

Keep cluster utilization as low aspossible To avoid read-only statekeep below 92

HyperFlex clusters storage utilization isabove 92 usable capacity

CSCvr20922

Release Notes for Cisco HX Data Platform Release 4050

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1b)NAAll the disks in the node got locked afterrebooting the node

We successfully unlocked usingsed-clientsh -U command but wantedto test with another reboot and driveslocked again

CSCvs74286

35(2g)To recover the VM copy over thedata disks and attach them to thenew VM

After enable encryption HX connectencryption status showing Caution butall of the disks enable encryptionsuccessful

-Self Encrypting Drives Service isrunning on all of the nodes

-There is no error message of encryption

- USB0 interface is up

-All SED disks showing supported 1enabled 1 locked 0

CSCvs93245

40(1b)Enable kernelpanic_on_oops as apersistent configuration This willcause the node to panic and rebootimmediately

A cluster node running HX release40(1b) may hang in the Linux kernelThis is classified as an oops and adeviation from the expected behavior

CSCvs54285

Release Notes for Cisco HX Data Platform Release 4051

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Enabling K8iscsi stack in Hx clusterCSCvs41324

Release Notes for Cisco HX Data Platform Release 4052

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Steps to enable K8 on HX cluster

Perform following steps on ALLthe controller VMs

1 Updateetcinitscvmclientconf toenable tune iscsiEnable totrue

sed -iesiscsiEnable=falseiscsiEnable=trueetcinitscvmclientconf

2 Run following initctlcommand to reloadconfiguration

initctl reload-configuration

3 Restart scvmclient process

stop scvmclient startscvmclient

4 Run following initctlcommand to mount SYSTEMDS

initctl emit --no-waitsystem-datastore-created

5 Verify that iscsiEnable tuneis set to true using thefollowing command

ps -eaf | grep scvmclient |grep -v grep

root 6241 1 1 Dec06 014007optspringpathstorfs-corescvmclient-T logEnabled=false -TlogSyslogEnabled=true -TlogEchoToScreen=false -TstatLoggingToFile=false -TstatLoggingToSyslog=true-TlogDir=varlogspringpath-TnfsBackendServerList=1010748100-T iscsiEnable=true -TiscsiUseAsync=true -T

Release Notes for Cisco HX Data Platform Release 4053

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

iscsiConfigFilePath=nfsSYSTEMistgtconfps -eaf | grep scvmclient |grep -v grep

35(2e)Following procedure can be usedto check NFS Queue Depth andincrease if needed

rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth256 lt- Low value[rootHXESXI1] vsish -e setvmkModulesnfsclientmnt[DATASTORE]propertiesmaxQDepth 1024[rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth1024 lt- Optimalvalue

This requires an ESXi host rebootto take effect

Please place the node in HyperflexMaintenanceMode and gracefullyreboot the node for the changes tobe applied

HyperFlex Datastore NFS Queue Depthshows as 256 which can lead toperformance (including latency) issues

CSCvr83056

40(1b)This is cosmetic The error will goaway once the cluster returns tohealthy status

When a single node is offline in clusterstcli cluster storage-summary shows twonodes unavailable rootSCVM~ stclicluster storage-summary

CSCvs47735

35(2g)Modify both the root and adminpassword for the SCVMs to be thesame

stcli security password set -uadmin

Cluster expansion fails at Config Installerstage when the root and admin passwordfor the storage controller (SCVM) aredifferent

CSCvs69317

35(2d)NAAfter successfully changing the DNSserver on the HX controller we still cansee the original DNS entry that was addedduring the deployment

CSCvs69154

Release Notes for Cisco HX Data Platform Release 4054

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Customers are strongly encouragedto work with TAC in order toidentify and remediate this issue

1 Change getcluster from 2m to8m inoptspringpathstorfs-mgmtstMgr-10confapplicationconfon all nodes

2 Restart stMgr on all nodes

You may see this error message after afailed upgrade or other task such asattempting to enter a node into HXmaintenance mode

CSCvs53555

40(2a)Run the following command oneach storage controller VM as root

grep -i monitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp sed -iesmonitor_dns_serverstruemonitor_dns_serversfalseoptspringpathhx-diag-toolswatchdog_configjsonampamp grep -imonitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp restart watchdog

Receive the following alertevent in HXConnect HX Controller VMHOSTNAME one or more configuredDNS servers not responding

CSCvt13947

35(2b)Delete any empty (Size 0) log filesunder varlogzookeeperversion-2ORvarlogzookeeperstandaloneversion-2

Zookeeper fails to start while Exhibitor isrunning however echo srvr returnsnothing

CSCvs21562

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

40(4e)1 De-commission the server

2 Power drain the server -REMOVE BOTH POWERCORDS ON THE BACK OFTHE SERVER FOR 10SECONDS THENREINSERTPOWERCORDS

3 Re-commission the server

M2 boot disk is missing from serverinventory after upgrade to server firmware40(4e) As a result server fails to boot toOS installed in the M2 disk The issuepersists after re-acknowledgement as wellas de-commission andre-acknowledgement of the server

CSCvs02466

Release Notes for Cisco HX Data Platform Release 4055

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Re-start the flusher manuallyMany disks taken offline by a fault ormanually resulting in ENOSPC oncluster

CSCvq32721

35(2e)Clean up fail over cluster and thenrecreate the fail over cluster Thereis no need to touch the HX storagecluster

[Hyper-V]When replicated DCwas usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2b)Manually install upgradescvmclient VIB andmake sure thatit matches with HXDP version

Upgrade from release HX 35(2a) to35(2x) 4X does not upgrade scvmclientRecovery point may not work

CSCvq54992

Management

30(1b)To recover the VM copy over thedata disks and attach them to thenew VM

[HyperV]VMshows up onmultiple nodesCSCvj22992

35(1a)1 Do not change MTU at ESXlevel on running cluster

2 Reset back to the original value

ChangingMTU from 9000 to 1500 on oneESX node causes storfs process to restarton all nodes

CSCvm99150

Release Notes for Cisco HX Data Platform Release 4056

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)After the installation (for newdeployments) or after an upgradeto 401a (for existingdeployments) perform thefollowing steps on all ControllerVMs

1 Edit the following tune files (onall Controller VMs)

optspringpathconfiglfftunes

optspringpathconfigvsi_16tbtunes

2 SetcleanerEnableSegSummaryCleaning

to false

3 After editing the above tunefiles

a SSH in to all the ControllerVMs

b Run storfstool -- -Z

c Run the below command andcheck the values Tune valueshould be true

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=true

d Type below commands andapplied the tunes changesdynamically

echo false gttmpstprocfssystemtunecleanerEnableSegSummaryCleaning

e Verify that tune values arechanged Run below commandValue should be false

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=false

f umount tmpstprocfs

A cluster with 8TB or 12TB disk drivesmay experience IO stalls for severalminutes after another node fails in thecluster

CSCvp23718

Release Notes for Cisco HX Data Platform Release 4057

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)1 Update the conf file manuallyto change to the link local addressto

2 Restart scvmclient on thecontroller and ESX

HX 35(2b) FlexVol to HX 40 CSIupgrade does not work FlexVol continuesto work

CSCvp26319

35(2a)Disable rebalance until node healsthen reenable rebalance

7-node cluster healing very slowlybecause rebalance change AMS reqs areblocking healing process

CSCvp62512

35(2a)NAWith moderate cluster storage capacityusage under site failover cluster clustermay experience an all paths (APD) downcondition

CSCvq49412

40(1a)NAThis product includes Third-partySoftware that is affected by thevulnerabilities identified by the followingCommon Vulnerability and Exposures(CVE) IDs

CVE-2016-0762 CVE-2016-6797CVE-2016-6816 CVE-2016-8735CVE-2017-5647 CVE-2017-12615CVE-2017-12616 CVE-2017-12617CVE-2017-7674 CVE-2018-1304CVE-2018-8014 CVE-2018-1336CVE-2018-8034 CVE-2018-11784CVE-2019-0232

CSCvp36364

35(2d)NAstcli cluster info command needs toprovide UCSM VIP address Currently itshows ucsm-hostcom

CSCvq11456

35(2a)No workarounds currently Thestale entries do not affect the dayto day operations of HyperFlex

HyperFlex upgrade validation failedbecause cluster has extraneous stNodes ininternal database

CSCvq32530

35(2a)NAWhen not in maintenancemode all drivesappear in HX Connect and get updated asexpected

When in maintenance mode extra drivesare shown in HX Connect with no slotnumbers

CSCvq42378

35(2b)Contact TAC to cleanup duplicatenode

Nodes duplicated in HX Connect aftercluster recreate

CSCvq61825

Release Notes for Cisco HX Data Platform Release 4058

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)NACurrently HyperFlex installs on Hyper-Vdo not contain the stcli security whitelistcommandset

CSCvq66245

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

Release Notes for Cisco HX Data Platform Release 4059

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)[Hyper-V] Converged node expansionfails for incoming node with VIC 14xx

CSCvq39523

Release Notes for Cisco HX Data Platform Release 4060

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

1 Set Jumbo frame inWindowsHypervisor for incomingnode

Get-NetAdapter storage-data-a|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

Get-NetAdapter storage-data-b|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

2 Verify all interfaces are setcorrectly to support Jumbo frames

Get-NetAdapter storage |Get-NetAdapterAdvancedProperty| RegistryKeyword -matchjumbo | ft -auto

You should receive the followingmessage

Name DisplayName DisplayValue

RegistryKeyword RegistryValue

---- ----------- ------------

--------------- -------------

vswitch-hx-storage-data Jumbo

Packet 9014 Bytes

JumboPacket 9014

storage-data-b Jumbo Packet

Bytes 9014 JumboPacket

9014

storage-data-a Jumbo Packet

Bytes 9014 JumboPacket

9014

3 Reboot only the incoming nodeor the node being expanded

4 Retry Cluster Expansion in

Release Notes for Cisco HX Data Platform Release 4061

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

bull vSphere 60 UpgradesmdashUsers on vSphere 60 migrating to 65 upgrade components in the followingorder

1 Upgrade HX Data Platform and UCS firmware

2 Upgrade HX Data Platform and ESXi

3 Upgrade HX Data Platform only first then upgrade ESXi or UCS firmware or both

bull M4 Server Firmware UpgradesmdashUpgrade server firmware to ensure smooth operation and to correctknown issues Specifically newer SAS HBA firmware is available in this release and is recommendedfor long-term stability

bull Users are encouraged to upgrade to 31(3c) C-bundle or later whenever possible

bull Users running C-bundle versions before 31(2f) must upgrade server firmware by performing acombined upgrade of UCS server firmware (C-bundle) to 31(3c) or later and HX Data Platform to25 Do not split the upgrade into two separate operations

bull If the cluster is already on 31(2f) C-bundle or later you may perform an HX Data Platform onlyor combined upgrade as required

bull M5 Server Firmware UpgradesmdashM5 generation servers must run firmware version 32(2d) or later

bull Firmware Downgrades mdashDowngrading UCSM from the HX-installer is not supported

bull M4M5 Mixed DomainsmdashA mixed domain occurs when a new separate M5 cluster is installed underthe same UCS domain that contains existing M4 clusters Under these conditions orchestrated UCSserver firmware upgrade will not operate until Cisco HX Data Platform Release 26 or later is installedon the M4 clusters Therefore it is best practice to first upgrade UCS server firmware to the latest 31(3)or 32(2) patch release before adding a new M5 cluster to the existing UCS domain Additionally any17 HX Data Platform clusters must first be upgraded before adding any new M5 clusters to the samedomain

bull Maintenance WindowmdashIf upgrading both HX Data Platform and UCS firmware you can select eithera combined or split upgrade through the vSphere HX Data Platform Plug-in depending on the length ofthe maintenance window Cisco UCSManager infrastructure upgrade is only supported using AutoInstalland the direct server firmware upgrade should be performed only through the upgrade orchestrationframework provided by the HX Data Platform Plug-in

bull Unsupported Self-Encrypting Drives (SEDs)mdashIf adding or replacing self-encrypting drives (SEDs)that have been recently qualified in newer versions of HX Data Platform insert the new drives only afterupgrading HX Data Platform to a compatible version All drives must be SED drives mixing SED andnon-SED is not supported

bullbull Enabling External Host AccessmdashWith Cisco HX Data Platform Release 40(1a) port 445 on themanagement network is blocked for enhanced security Note that prior to 40 port 445 port was openenabling external host access If you are upgrading to 40(1a) from a prior release and would like tocontinue external host access you can use a utility to open select hosts For more information aboutenabling external host access see the ConfiguringHyperFlex Share to SCVMM section in the InstallationGuide for Microsoft Hyper-V

Release Notes for Cisco HX Data Platform Release 4021

Guidelines and Limitations

Mixed Cluster Expansion Guidelinesbull Expanding existing M4 cluster with M5 converged nodes is supported

bull Expanding existing M5 cluster with M4 converged nodes is not supported

bull Expanding existing mixed M4M5 cluster with M4 or M5 converged nodes is supported

bull Adding any supported compute-only nodes is permitted with all M4 M5 and mixed M4M5 clustersusing the HX Data Platform 26 or later Installer Some example combinations are listed here manyother combinations are possibleExample combinationsExpand mixed M4M5 cluster with compute-only B200 C220 C240 M4M5Expand M4 cluster with compute-only B200 M5 C220 M5 C240M5

bull Only expansion workflow is supported to create a mixed cluster Initial cluster creation with mixedM4M5 servers is not supported

bull All M5 servers must match the form factor (220240) type (HybridAF) security capability (Non-SEDonly) amp disk configuration (QTY capacity and non-SED) of the existing M4 servers

bull HX220-M5 will use a maximum of 6 capacity disks (2 disk slots to remain empty) when mixedwith HX220-M4

bull HX Edge SED LFF Hyper-V and Stretched Clusters do not support mixed M4 and M5 clusters

Security FixesThe following security issues are resolved

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-13132 CVE-2019-9924CSCvq6313840(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-11719 CVE-2019-11727CVE-2019-11729

CSCvq7124040(2a)

Release Notes for Cisco HX Data Platform Release 4022

Mixed Cluster Expansion Guidelines

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-1125CSCvr0633940(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2015-9383 CVE-2018-14498CVE-2018-20406 CVE-2018-20852CVE-2019-10160 CVE-2019-13117CVE-2019-13118 CVE-2019-14287CVE-2019-14973 CVE-2019-15903CVE-2019-17546 CVE-2019-18197CVE-2019-18218 CVE-2019-5010CVE-2019-5094 CVE-2019-5481CVE-2019-5482 CVE-2019-9636CVE-2019-9740 CVE-2019-9947CVE-2019-9948

CSCvs0609440(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2017-13168 CVE-2017-18174CVE-2017-18216 CVE-2018-10876CVE-2018-10877 CVE-2018-10878CVE-2018-10879 CVE-2018-10880CVE-2018-10881 CVE-2018-10882CVE-2018-10902 CVE-2018-10938CVE-2018-12233 CVE-2018-12896CVE-2018-13053 CVE-2018-13094CVE-2018-13096 CVE-2018-13405CVE-2018-13406 CVE-2018-14609CVE-2018-14617 CVE-2018-14633CVE-2018-14734 CVE-2018-15572CVE-2018-15594 CVE-2018-16276CVE-2018-16658 CVE-2018-17182CVE-2018-17972 CVE-2018-18021CVE-2018-18690 CVE-2018-18710CVE-2018-6554 CVE-2018-6555CVE-2018-9363

CSCvp6501940(2a)

This vulnerability is due to insufficientHTML iframe protection An attackercould exploit this vulnerability bydirecting a user to an attacker-controlledweb page that contains amaliciousHTMLiframe A successful exploit could allowthe attacker to conduct clickjacking orother clientside browser attacks

NACSCvo9851640(2a)

Release Notes for Cisco HX Data Platform Release 4023

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerability is due to insufficientauthentication for the statistics collectionservice An attacker could exploit thisvulnerability by sending properlyformatted data values to the statisticscollection service of an affected deviceA successful exploit could allow theattacker to cause the web interfacestatistics view to present invalid data tousers

NACSCvj9558440(2a)

The vulnerability is due to insufficientCSRF protections for the web UI on anaffected device An attacker could exploitthis vulnerability by persuading a user ofthe interface to follow a malicious linkA successful exploit could allow theattacker to perform arbitrary actions withthe privilege level of the affected user

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvp2434340(2a)

Vulnerabilities with a version of cJSONidentified by CVE-2019-11834 andCVE-2019-11835

For more information see the relatedCisco Security Advisory

CVE-2019-11834 CVE-2019-11835CSCvq1994940(2a)

HX ESXi image patches for MCEPSCand TAA vulnerabilities with VMwareESXi identified by CVE-2018-12207 andCVE-20190-11135

For more information see the relatedCisco Security Advisory

CVE-2018-12207 CVE-2019-11135CSCvr5439840(2a)

CP networking vulnerabilities affectingthe Linux kernel identified byCVE-2019-11477 CVE-2019-11478CVE-2019-11479

For more information see the relatedCisco Security Advisory

CVE-2019-11477 CVE-2019-11478CVE-2019-11479

CSCvq1954640(2a)

Qualification of Microsoft securitypatches for vulnerabilities withMicrosoftHyper-V hypervisor identified withCVE-2018-12207 CVE-2019-11135

For more information see the relatedCisco Security Advisory

CVE-2018-12207 CVE-2019-11135CSCvr5439940(2a)

Release Notes for Cisco HX Data Platform Release 4024

Security Fixes

DescriptionCVEDefect IDRelease

OpenSSH vulnerabilities identified byCVE-2019-6111

For more information see the relatedCisco Security Advisory

CVE-2016-10708 CVE-2018-15473CVE-2018-20685 CVE-2019-6109CVE-2019-6111

CSCvp7646340(2a)

Multiple vulnerabilities associated withOpenSSL and LibNSS

For more information see the relatedCisco Security Advisory

CVE-2016-2105 CVE-2016-2106CVE-2016-2107 CVE-2016-2109CVE-2016-2176 CVE-2016-2177CVE-2016-2178 CVE-2016-2179CVE-2016-2180 CVE-2016-2181CVE-2016-2182 CVE-2016-2183CVE-2016-6302 CVE-2016-6303CVE-2016-6304 CVE-2016-6305CVE-2016-6306 CVE-2016-6307CVE-2016-6308 CVE-2016-7055CVE-2016-8610 CVE-2017-3731CVE-2017-3732 CVE-2017-3735CVE-2017-3736 CVE-2017-3737CVE-2017-3738 CVE-2018-0495CVE-2018-0732 CVE-2018-0734CVE-2018-0735 CVE-2018-0737CVE-2018-0739 CVE-2018-12384CVE-2018-12404 CVE-2018-5407CVE-2019-1559

CSCvp6667940(2a)

libpng vulnerability identified byCVE-2016-10087

For more information see the relatedCisco Security Advisory

CVE-2016-10087CSCvp6655540(2a)

Vulnerabilities associated with libjpg andlibpng

For more information see the relatedCisco Security Advisory

CVE-2014-9092 CVE-2016-3616CVE-2017-15232 CVE-2018-11212CVE-2018-11213 CVE-2018-11214CVE-2018-1152 CVE-2018-13785

CSCvp3458640(2a)

Vulnerabilities associated with Glibidentified by CVE-2018-16428 andCVE-2018-16429

For more information see the relatedCisco Security Advisory

CVE-2018-16428 CVE-2018-16429CSCvp3120740(2a)

Multiple vulnerabilities associated withcurl expat python 27 python 35 36and 37 freetype and giflib

For more information see the relatedCisco Security Advisory

CVE-2019-15133 CVE-2019-15903CVE-2019-5010 CVE-2019-5481CVE-2019-5482 CVE-2019-9636CVE-2019-9740 CVE-2019-9947CVE-2019-9948

CSCvr3690340(2a)

Release Notes for Cisco HX Data Platform Release 4025

Security Fixes

DescriptionCVEDefect IDRelease

Multiple third party softwarevulnerabilities

For more information see the relatedCisco Security Advisory

CVE-2019-14379 CVE-2019-12384CVE-2019-14439

CSCvq9203240(2a)

Vulnerabilities associated with elfutils

For more information see the relatedCisco Security Advisory

CVE-2018-16062 CVE-2018-16402CVE-2018-16403 CVE-2018-18310CVE-2018-18520 CVE-2018-18521CVE-2019-7149 CVE-2019-7150CVE-2019-7665

CSCvq4325040(2a)

Vulnerabilities associated with vim

For more information see the relatedCisco Security Advisory

CVE-2017-5953 CVE-2019-12735CSCvq4323040(2a)

Vulnerabilities associated with sqlite3

For more information see the relatedCisco Security Advisory

CVE-2016-6153 CVE-2017-10989CVE-2017-13685 CVE-2017-2518CVE-2017-2519 CVE-2017-2520CVE-2018-20346 CVE-2018-20505CVE-2018-20506 CVE-2019-8457CVE-2019-9936 CVE-2019-9937

CSCvq4321340(2a)

Vulnerabilities associated with glib20

For more information see the relatedCisco Security Advisory

CVE-2018-20843CSCvq4320940(2a)

Vulnerabilities associated with expat

For more information see the relatedCisco Security Advisory

CVE-2018-20843CSCvq4320540(2a)

Vulnerabilities associated with bzip2

For more information see the relatedCisco Security Advisory

CVE-2016-3189 CVE-2019-12900CSCvq4319440(2a)

Mulitple Vulnerabilities associated wtihNodeJS

For more information see the relatedCisco Security Advisory

CVE-2018-12115 CVE-2018-0734CVE-2018-5407 CVE-2018-12120CVE-2018-12121 CVE-2018-12122CVE-2018-12123 CVE-2018-12116CVE-2019-5737 CVE-2019-5739

CSCvq1069440(2a)

Vulnerabilities associated with PalletsJinja strformat_map and identified byCVE-2019-10906 and CVE-2016-10745

For more information see the relatedCisco Security Advisory

CVE-2019-10906 CVE-2016-10745CSCvq1038840(2a)

Release Notes for Cisco HX Data Platform Release 4026

Security Fixes

DescriptionCVEDefect IDRelease

Multiple Security Vulnerabilities

For more information see the relatedCisco Security Advisory

CVE-2014-8501 CVE-2014-9939CVE-2015-9262 CVE-2016-10087CVE-2016-2226 CVE-2016-4487CVE-2016-4488 CVE-2016-4489CVE-2016-4490 CVE-2016-4491CVE-2016-4492 CVE-2016-4493CVE-2016-6131 CVE-2018-10963CVE-2018-13785 CVE-2018-17100CVE-2018-17101 CVE-2018-18557CVE-2018-18661 CVE-2018-7456CVE-2018-8905

CSCvq0917840(2a)

Vulnerabilities associated withlibseccomp

For more information see the relatedCisco Security Advisory

CVE-2019-9893CSCvq0756840(2a)

Vulnerabilities associated withgdk-pixbuf

For more information see the relatedCisco Security Advisory

CVE-2017-12447CSCvq0675540(2a)

Vulnerabilities associated withpython-crypto

For more information see the relatedCisco Security Advisory

CVE-2018-6594CSCvp9381740(2a)

Vulnerabilities associated with wget

For more information see the relatedCisco Security Advisory

CVE-2018-20483 CVE-2019-5953CSCvp8672140(2a)

Multiple vulnerabilities associated withPython Crypto

For more information see the relatedCisco Security Advisory

CVE-2018-6594CSCvp6674840(2a)

Vulnerabilities associated withsensible-utils

For more information see the relatedCisco Security Advisory

CVE-2017-17512CSCvp6673440(2a)

Multiple vulnerabilities associated withPython

For more information see the relatedCisco Security Advisory

CVE-2018-1000030CSCvp6668940(2a)

Release Notes for Cisco HX Data Platform Release 4027

Security Fixes

DescriptionCVEDefect IDRelease

Multiple patch vulnerabilities identifiedby CVE-2016-10713

For more information see the relatedCisco Security Advisory

CVE-2016-10713CVE-2018-1000156 CVE-2018-6951

CSCvp6667240(2a)

Vulnerabilities associatedwith Little CMSidentified by CVE-2016-10165

For more information see the relatedCisco Security Advisory

CVE-2016-10165 CVE-2018-16435CSCvp6666440(2a)

Vulnerabilities associated with TenableScan identified byCVE-2011-5325CVE-2019-11068CVE-2016-7076CVE-2017-1000368

For more information see the relatedCisco Security Advisory

CVE-2011-5325 CVE-2016-7076CVE-2017-1000368CVE-2019-11068

CSCvp6474640(2a)

Vulnerabilities associated with libxml2294 and earlier as used in XMLSec1223 and earlier and other products andidentified by CVE-2016-9318

For more information see the relatedCisco Security Advisory

CVE-2016-9318 CVE-2017-16932CVE-2017-18258 CVE-2018-14404CVE-2018-14567

CSCvp3479240(2a)

Vulnerabilities associated with LFTPRemote File Names Unauthorized AccessVulnerability and identified byCVE-2018-10916

For more information see the relatedCisco Security Advisory

CVE-2018-10916CSCvp2926640(2a)

Vulnerabilities associated withCryptographyDeprecationWarning signerand verifier have been deprecated

For more information see the relatedCisco Security Advisory

CVE-2018-7750CSCvo3409740(2a)

Multiple Vulnerabilities in python python27 35

For more information see the relatedCisco Security Advisory

CVE-2019-16056CSCvr3979340(2a)

Release Notes for Cisco HX Data Platform Release 4028

Security Fixes

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toexecute commands as the root user

The vulnerability is due to insufficientinput validation An attacker could exploitthis vulnerability by connecting to thecluster service manager and injectingcommands into the bound process Asuccessful exploit could allow the attackerto run commands on the affected host asthe root user

Cisco has released software updates thataddress this vulnerability There areworkarounds that address thisvulnerability

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvq2417640(1b)35(2g)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

The vulnerability is due to an unprotectedlistening interface An attacker couldexploit this vulnerability by connectingto the listening interface and injectingcommands to the bound process Anexploit could allow the attacker to runcommands on the affected host as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvj9560640(1b)35(2g)

Release Notes for Cisco HX Data Platform Release 4029

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerabilities associated with theJVM18U121memory leak found duringVC alarms (concurrent 40 calls usingREST API)

CVE-2017-10053 CVE-2017-10067CVE-2017-10074 CVE-2017-10078CVE-2017-10081 CVE-2017-10087CVE-2017-10089 CVE-2017-10090CVE-2017-10096 CVE-2017-10101CVE-2017-10102 CVE-2017-10107CVE-2017-10108 CVE-2017-10109CVE-2017-10110 CVE-2017-10111CVE-2017-10115 CVE-2017-10116CVE-2017-10118 CVE-2017-10135CVE-2017-10176 CVE-2017-10193CVE-2017-10198 CVE-2017-10243CVE-2017-10274 CVE-2017-10281CVE-2017-10285 CVE-2017-10295CVE-2017-10345 CVE-2017-10346CVE-2017-10347 CVE-2017-10348CVE-2017-10349 CVE-2017-10350CVE-2017-10355 CVE-2017-10356CVE-2017-10357 CVE-2017-10388CVE-2017-3509 CVE-2017-3511CVE-2017-3526 CVE-2017-3533CVE-2017-3539 CVE-2017-3544CVE-2018-2579 CVE-2018-2582CVE-2018-2588 CVE-2018-2599CVE-2018-2602 CVE-2018-2603CVE-2018-2618 CVE-2018-2629CVE-2018-2633 CVE-2018-2634CVE-2018-2637 CVE-2018-2641CVE-2018-2663 CVE-2018-2677CVE-2018-2678 CVE-2018-2783CVE-2018-2790 CVE-2018-2794CVE-2018-2795 CVE-2018-2796CVE-2018-2797 CVE-2018-2798CVE-2018-2799 CVE-2018-2800CVE-2018-2814 CVE-2018-2815CVE-2018-2952 CVE-2018-3136CVE-2018-3139 CVE-2018-3149CVE-2018-3150 CVE-2018-3169CVE-2018-3180 CVE-2018-3183CVE-2018-3214 CVE-2019-2422

CSCvo8899740(1b)

Tomcat and Nginx logs are not beingcollected in the support bundle generatedthrough HX Connect in Release 35

NACSCvm5803140(1b)

The vulnerabilities associated with thelibmspack software package versionincluded in Cisco HX Data Platform

CVE-2018-18584

CVE-2018-18585

CSCvn3511940(1a)

Release Notes for Cisco HX Data Platform Release 4030

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerabilities associated withFasterXML Jackson-Databind TimeValueField Denial of Service

CVE-2018-14719

CVE-2018-14720

CVE-2018-1000873

CVE-2018-14721

CVE-2018-19360

CVE-2018-19362

CVE-2018-19361

CVE-2018-14718

CSCvn8228240(1a)

The vulnerabilities associated with theOpenSSL Protocol software packageversion included in Cisco HX DataPlatform

CVE-2013-3587CSCvo0505440(1a)

The vulnerabilities associated withThird-Party Software Denial of Service

CVE-2018-16487

CVE-2018-19361

CSCvo2781840(1a)

The vulnerabilities associated withLinuxkernel for Ubuntu 1710

CVE-2018-1092

CVE-2018-7492

CVE-2018-8087

CVE-2018-1068

CVE-2018-8781

CSCvm5314935(2a)

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)25(1d)stcli services dns remove should remove the DNS serverinfo from the interface files

CSCvs70967

40(2b)40(1b)HX 40(1b) new deployment from Intersight productioncloud

User receives following error while enabling CSI(Kubernetes integration)

Failure occurred during volume_access_enable Errorwas Nonzero exit code 1

CSCvt10522

40(2b)40(2a)Receive the following alertevent in HX Connect HXController VM HOSTNAME one or more configuredDNS servers not responding

CSCvt13947

Release Notes for Cisco HX Data Platform Release 4031

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)40(2a)API UI CLI show few Drive slots Empty after upgradeto 402a

The drives are seen in the sysmtool lsscsi and stclicommand outputs

The cluster is healthy with no errors

CSCvt14914

40(2b)35(2d)After successfully changing the DNS server on the HXcontroller we still can see the original DNS entry thatwas added during the deployment

CSCvs69154

40(2b)35(2e)HyperFlex Connect and VCenter show APD alarms forHX and non HX datastores This implies there is an issuewith HyperFlex when there may not be

CSCvs30080

40(2b)40(2a)An issue where all the disks in the node get locked afterrebooting the node

We successfully unlocked using sed-clientsh -Ucommand but wanted to test with another reboot anddrives locked again

CSCvs74286

40(2b)40(2a)When running stcli license commands on HyperFlexthe following error is seen

rootSpringpathControllertmp stcli license show all

Show smart licensing failed

Smart Agent is not ready please wait a minute and tryagain

CSCvt13929

40(2b)40(2a)After upgrading to HX 40(2a) the first RoboEdge nodethat gets rebooted then ignores persistent drivesPIDHX-HD24TB10K4KN

Cluster will be in degraded state and have reducedcapacity

CSCvt20144

40(2b)35(2b)Zookeeper fails to start while Exhibitor is runninghowever echo srvr returns nothing

CSCvs21562

40(2b)40(2a)When performing an HyperFlex upgrade a validationwarning may occur due to the host not having EnterprisePlus or Enterprise hypervisor licensing

Upgrade Validation Warning

ESXi host esx1labtest should be configured withVMware Enterprise license for upgrade to continue

CSCvs91787

Release Notes for Cisco HX Data Platform Release 4032

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)35(2g)Cluster expansion fails at Config Installer stage whenthe root and admin password for the storage controller(SCVM) are different

CSCvs69317

40(2b)35(2g)Panic while upgrading ESXiCSCvt06983

40(2b)40(2a)A cluster node running may hang in the Linux kernelThis is classified as an oops and a deviation from theexpected behavior

CSCvs54285

40(2b)35(2g)Rebalance failing on 10+10 node HX 35(2g) stretchcluster

CSCvs69007

40(2b)35(2d)The cluster becomes inaccessible to the IOVisorCSCvr54687

40(2b)40(2a)Panic on the storage controllerCSCvt61297

40(2b)40(2a)The size of support bundle is very large when collectedwith storfs-support command

CSCvt63306

40(2b)40(2a)storfs crash associated withFileSystemUsageWarningEvent orFileSystemUsageAlertEvent due to high usage ofvarstv above 80

CSCvt72807

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

40(2a)35(2d)In normal scenarios HX upgrade gets pendingACK afterupdating service profile In this case UCS only upgradewas stalled because after updating service profile HXupgrade was not getting pending ACK and it was alwayswaiting state

CSCvo39912

40(2a)35(1a)When using motherBoardReplace-12 to clean ZK fromold stNodepNodes resulted in unmounted datastore andmaking the size of the Hyperflex Datastores 0 resultingin all VMs in the cluster going offline

CSCvq39471

40(2a)26(1e)Upgrading cluster and getting error

Failed upgrade validations Checking vCenterconfiguration Reason Upgrade validations failed DRSFault Insufficient resources to satisfy configured failover

CSCvm77294

Release Notes for Cisco HX Data Platform Release 4033

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)The cluster expand fails on a node with DR replicationconfigured

When a HX cluster which has DR replication configuredis expanded we see the installer UI pulling in thereplicationVLAN information instead of themanagementVLAN information Even if we change that informationto the correct mgmt VLAN id and name it it does notseem to work as the node is configured with the VLANof the replicationVLAN in ESXi This leads to the failureof the node add with the host unreachable error

CSCvo70650

40(2a)35(1a)Customer reported that server didnt complete thefirmware upgrade automatically one by one

The user manually needs to put the host into maintenancemode and they manually acknowledged the pendingrequests to finish the UCS firmware upgrade

Maintenance policy is set to default (User-ack) as perthe design

CSCvo91624

40(2a)35(1i)When the cluster is in Failed state and stcli noderemove is attempted the output shows successful thoughthe node failed to get removed from the cluster

CSCvo93017

40(2a)35(1a)HyperFlex cluster upgrade may fail during validationwith a DRS Validation failed error

CSCvp31021

40(2a)35(2b)HX cluster expansion will fail with the following errormessage

MAC address pool configuration failure150[ErrorDescription] bad address block range definitioncollision

CSCvp58318

40(2a)35(2b)Memory usage by carbon cacheCSCvq34873

40(2a)35(2g)Upgrading ESXi cluster fails with error Nodemaintenance mode failed

CSCvr03240

40(2a)35(2e)storfs process does not automatically start on Exit HXMaintenance Mode or other tasks which restart StorageController

CSCvr88978

40(2a)35(2a)Post install script fails with message

Failed to execute ipmitool on HX node

CSCvp10707

40(2a)40(1a)HyperFlex expansionworkflow doesnt pull VLANnamecorrectly

CSCvp46539

Release Notes for Cisco HX Data Platform Release 4034

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)During HX cluster deployment - Validate ClusterCreation phase - HX Installer might fail with the error

fromvarlogspringpathstDeploylog

CSCvq45087

40(2a)35(2b)HX Installer fails to login to the SOL for configuringESX hosts

CSCvq91380

40(2a)35(2d)If DRS is enabled with configuration parameters otherthen fully automated ie manual or partially automatedthan this will change to fully automated during clusterexpansion

CSCvr44222

40(2a)35(1a)One of the validation task name has a typo The taskname is Validating node for mixed of SED and non-SEDdisks

It should be Validating node for mix of SED andnon-SED disks

CSCvp82175

40(2a)30(1i)This combination of using smaller NVMe drives is notsupported in expand operation For example if the drivesare only 375GB and they cannot be added to an existingcluster with larger caching SSD

CSCvo12359

40(2a)35(1a)HyperFlex includes a version of OpenSSL that is affectedby the vulnerabilities identified by the followingCommon Vulnerability and Exposures

CVE-2018-0495

CSCvp66679

40(2a)30(1e)SDK invoke exception when taking a native snapshot ona VM with GPU pci passthrough attached

CSCvo00511

40(2a)30(1i)Node replacement script fails due to EAM errorCSCvo62867

40(2a)35(1a)While pairing a cluster (HX release gt= 35) and cluster(HX release lt 35) remote replication network test failson the cluster (HX release lt 35)

CSCvo79760

40(2a)30(1b)Fixed the issue in the latest support-workflow bundleThis should not be hit anymore

CSCvo87061

40(2a)35(2a)On a 3 node cluster the MBreplace script fails as thescript looks for HEALTHY cluster status

CSCvo87080

40(2a)35(2a)MbReplace script (tar) hangs when running on 352b or40

CSCvp12359

Release Notes for Cisco HX Data Platform Release 4035

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)HX replication cleanup failing with error NA 35(2a) -INFODR state is not clean

CSCvp63958

40(2a)40(1a)The check_stig_parameters API incorrectly shows thecompliance state of the cluster as not STIG compliant

CSCvp66277

40(2a)35(2c)The snapshot creation on CBT enabled VM fails witherror Failed in vmreparent vmkfstools clone1

CSCvq06952

40(2a)35(2a)HyperFlex GUI when setting up remote DR-replicationpartnerpeer should show remote replication partner TCPport reachability test results

This will provide quick upfront connectivity results thatfrequently takes customer a while to manually check andconfirm

If reachability tests fail an info window and HX alertshould trigger that list the destination peer IP and thespecific port(s) that could not be reache

CSCvo60587

40(2a)35(2b)If you have more than one IP pool configured for areplication network and local replication network testsare failing you might be running into this issue

CSCvr67130

40(2a)35(2a)Skip task option in MBreplace and Software redeployscripts The scripts should have the option to skip tasksas user inputs and not by modifying the json file

CSCvp05204

40(2a)40(1a)Error noticed on SCVM Console print_req_error IOerror dev fd0 sector 0 on HyperV

It doesnt come up on SSH session

CSCvq34357

40(2a)30(1d)The VMNetwork switch is already created as part of theinstall process

Ideally the ip address assignment should be done duringthe initial install process as well No post install stepsshould be required altrnatively it could be done as partof a post install script

CSCvk36222

40(2a)25(1c)storfs service impacted during HX upgrade from HX21(1) to 25(1c)

HX node which is not being upgraded storfs servicestopped causing outage

CSCvg53223

40(2a)35(1b)HX Stretch Cluster Witness VM reverts to DHCP onreboot

CSCvp37536

40(2a)35(2b)HyperFlex Stretch Cluster Witness does not getprogrammed with an NTP server

CSCvp46578

Release Notes for Cisco HX Data Platform Release 4036

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2b)HyperFlex cluster does not heal after maintenance onnodeserver

CSCvr18528

40(2a)40(2a)High percentage of packet loss on the eth1 data interface9K MTU

CSCvr97089

40(2a)35(2f)The HyperFlex cluster healing state is stuck at 87 andis unable to progress

CSCvr16760

40(2a)40(2a)In some circumstances two replications can occur forthe same VM

From the GUI the replication status for this VM willshow stuck in an In Progress state but there are nojobs stuck in replication layer Subsequent Replicationsshould succeed

CSCvq22898

40(2a)35(2a)After an unexpected power outage the hyperflex clusteris unable to come up correctly

All processes are running time is synced and all nodescan vmkping each other as required

CSCvp13990

40(2a)40(1a)When specifying IP address instead of node ID stclinode maintenanceMode commands fail

CSCvp33657

40(2a)40(1a)2 node edge cluster if experiencing Intersightarbitratorconnectivity issues should NOT allowHXMaintenancemode to initiate as this will bring the cluster down

CSCvr31573

40(2a)35(2c)Old files in varsupportZKTxnlog are not purged withthe daily zklog-cleanup cron job

CSCvr89066

40(2a)35(2a)When a node is in MaintenanceMode any disk removalor replacement will be reflected in UI only after the nodeis brought back frommaintenancemode This is becausestorfs is not running on the node in maintenance andwill not be able to detect disk activities until it is broughtout of Maintenance Mode

CSCvo86431

40(2a)35(2c)Cluster might go offline and stop serving IO requestsrootcvm~ stcli cluster storage-summary--detailGet cluster storage summary failedjavanetConnectException Connection refused19216814210010207

CSCvr01645

40(2a)35(2d)A stretch cluster deployment fails with a messageFormatting nodes Node in Use on the Hyperflex installerand crmZoneType value is seen as 1 in storfscfg oroptspringpathconfigstretchtunes

CSCvq80340

Release Notes for Cisco HX Data Platform Release 4037

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)30(1c)During upgrade we upload the upgrade packagedownloaded from Ciscocom Upgrade packages are offile type tgz hxconnect acceptsstorfs-packages-401a-33028tar file a different type ofcompression and allows upgrade to start and fail whenit cannot decompress the file for bootstrap and scvmclientupgrade on ESXi

CSCvq63888

40(2a)40(2a)While attempting to install hyperflex via intersight thefollowing error is seen

Failed in Task Add hosts to vCenter Cluster with ErrorTry adding hosts manually to vCenter and retry Failedto get addressFailed to host by namelookupHostnamecompanycom on 000053 read udp12700145430-gt12700153 read connection refused

CSCvr52098

40(2a)35(2a)CoreAPI call may time out for cluster management APIwhile doing cluster create validation

CSCvo92952

40(2a)35(2d)Validation fails with Mixed mode expansion checkdue to Enhanced vMotion Compatibility Incompatibilitywhen the ESXi version is different

CSCvq95460

40(2a)30(1i)This failure is seen in installer when custom workflowis used hypervisor configuration + deploy (clean diskpartition=no) at Installing software packages on storagecontroller VM

CSCvr66309

40(2a)35(2d)The HX Installer apparently replaced a comma bysemicolon on the VLAN ID range while exporting theconfiguration to a JSON file

CSCvq93831

40(2a)40(1a)When using unsupported browsers HX Connect userswill not be able to login These users see an errorindicating unauthorized user

CSCvp22693

Release Notes for Cisco HX Data Platform Release 4038

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)40(1a)A vulnerability in the web-based management interfaceof Cisco HyperFlex Software could allow anunauthenticated remote attacker to conduct a cross-siterequest forgery (CSRF) attack on an affected system

The vulnerability is due to insufficient CSRF protectionsfor the web UI on an affected device An attacker couldexploit this vulnerability by persuading a user of theinterface to follow a malicious link A successful exploitcould allow the attacker to perform arbitrary actions withthe privilege level of the affected user

Cisco has released software updates that address thisvulnerability There are no workarounds that address thisvulnerability

This advisory is available at the following link

httpstoolsciscocomsecuritycentercontentCiscoSecurityAdvisorycisco-sa-20190807-hypflex-csrf

CSCvp24343

40(2a)35(2d)Add message to HX Connect Progress flow to notAcknowledge Pending Activities and reboot the serverson UCSM

HX Connect is doing a controlled rolling server upgradein the background

CSCvq22844

40(2a)40(1b)Compute Node expand operation fails due to incorrectcluster name

CSCvr43786

40(2a)26(1e)RF-2 cluster shutdown during data resynchronizationafter a node failure (non storfs) followed by multipledisk read failures and a HardBlacklist of a disk

CSCvp41241

40(2a)40(1a)HyperFlex Edge nodes do not properly set the ESXihostname during deployment

CSCvo13143

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4039

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4040

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1b) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401b-33133p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4041

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4042

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1b)40(1a)While running the HyperFlex installer with WindowsServer Hyper-V cluster creation process fails with theerror Failure occurred during ClusterCreation process Unable to post thecontent to the downstream This symptomis encountered while deploying HyperFlex cluster withHyperFlex nodes configuredordered with Cisco VIC1457 MLOM (PID HX-MLOM-C25Q-04) andWindows Server Datacenter or Core with Hyper-Vversion 2016 or 2019

CSCvp64140

Release Notes for Cisco HX Data Platform Release 4043

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)35(2b)Adding Micron 5200 drive to a cluster fails to increasecluster capacity

CSCvo69067

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4044

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4045

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1a) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401a-33028p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4046

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4047

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1a)30(1d)Cluster instability when disks of different sector sizeplaced in HX node

CSCvk17250

40(1a)

35(2c)

35(1a)When logged in using a local HX user account insteadof a Virtual Center account an error message appearsintermittently indicating Virtual Centerunreachable or Resource informationcannot be updated when VC is reachable

CSCvo36198

40(1a)

35(2b)

30(1d)HXDP does not work with EMC RecoverPoint - needsto support VMware API (FSS-Readdir)

CSCvk38003

Release Notes for Cisco HX Data Platform Release 4048

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)

35(2a)

30(1i)

25(1c)

25(1d)

Zookeeper (Exhibitor) process on HyperFlex StorageController VMs may fail to respond to requests whenvarzookeeper has low or no free space

CSCvm90352

40(1a)26(1c)Use Asynchronous consolidation for HX SnapshotsCSCvn02151

40(1a)35(2b)HX Quiesced Snapshot for Backup VendorsCSCvo90713

40(1a)35(2a)The cluster creationcluster expansion workflow stopswith the following error message at the validation step

FIRMWARE-Check UCSC-SAS-M5HD

FIRMWARE-Check UCSC-SAS-M5HD Required00000029000000320000003500000050 Found00000058

Action Needed Update the Controller Firmware toRequired Version

CSCvn17787

35(2b)30(1d)Kernel migration fails when a local datastore is searchedfor in the ESXi host

CSCvn73127

35(2a)26(1b)A node shuts down when two disks are replaced (acaching disk and another capacity disk) where thecapacity disk is inserted first and then the caching diskis inserted

CSCvk46364

Hyper-V

40(1a)35(2a)Cluster expansion may fail with an error code 500 -operation timed out

CSCvn28721

35(2b)35(2a)During upgrade remove the VLAN on team created foruser vSwitch During fresh installation only one VLANtag is set to the vSwitch and team although multipleVLANs were entered

CSCvn54300

35(2b)35(2a)While upgrading a Hyper-V cluster on account of a rarerace condition between the stUpgradeService andZookeeper servers the upgrade orchestration throws anupgrade validation error and the upgrade process isaborted

CSCvn60486

Release Notes for Cisco HX Data Platform Release 4049

Resolved Caveats in Release 40(1a)

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Startrestart EAM service invCenter

When EAM service is not runningvCenter registration is failing

CSCvt55712

35(2a)Contact Cisco TAC if you believeyou are seeing similar conditions

HyperFlex Stretch Cluster saw poorapplication performance due to writelatency cluster remained unhealthy andrebalance was stuck

This defect is currently being used forinvestigative purposes

CSCvt45344

35(2g)Remove log files fromvarlogspringpath folder when thelog files are not getting rotatedStop the relevant service removelarge log files fromvarlogspringpath folder andrestart the service

varstv folder in Controller VM maybecome full

CSCvt36374

40(2b)Restarting the stMgr will initializeit again

In a Cluster configured with DR networkStMgr may not get initialized and stuckin a deadlock while enabling the IPTablerules (can be verified from the stMgrlog)

CSCvt89709

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Click recover for the selectedVM again under the standalonemode This will move the VM torecovered state

When the user recovers a VM protectedin a group it along with other VMs in thegroup move to recovering state instandalonemode The selectedVM shouldmove to recovered state

CSCvs75553

35(2e)Clean up the fail over cluster andthen recreate the fail over clusterNo need to touch the HX storagecluster

Hyper-V When replicated DC was usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2d)40(1a)

Keep cluster utilization as low aspossible To avoid read-only statekeep below 92

HyperFlex clusters storage utilization isabove 92 usable capacity

CSCvr20922

Release Notes for Cisco HX Data Platform Release 4050

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1b)NAAll the disks in the node got locked afterrebooting the node

We successfully unlocked usingsed-clientsh -U command but wantedto test with another reboot and driveslocked again

CSCvs74286

35(2g)To recover the VM copy over thedata disks and attach them to thenew VM

After enable encryption HX connectencryption status showing Caution butall of the disks enable encryptionsuccessful

-Self Encrypting Drives Service isrunning on all of the nodes

-There is no error message of encryption

- USB0 interface is up

-All SED disks showing supported 1enabled 1 locked 0

CSCvs93245

40(1b)Enable kernelpanic_on_oops as apersistent configuration This willcause the node to panic and rebootimmediately

A cluster node running HX release40(1b) may hang in the Linux kernelThis is classified as an oops and adeviation from the expected behavior

CSCvs54285

Release Notes for Cisco HX Data Platform Release 4051

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Enabling K8iscsi stack in Hx clusterCSCvs41324

Release Notes for Cisco HX Data Platform Release 4052

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Steps to enable K8 on HX cluster

Perform following steps on ALLthe controller VMs

1 Updateetcinitscvmclientconf toenable tune iscsiEnable totrue

sed -iesiscsiEnable=falseiscsiEnable=trueetcinitscvmclientconf

2 Run following initctlcommand to reloadconfiguration

initctl reload-configuration

3 Restart scvmclient process

stop scvmclient startscvmclient

4 Run following initctlcommand to mount SYSTEMDS

initctl emit --no-waitsystem-datastore-created

5 Verify that iscsiEnable tuneis set to true using thefollowing command

ps -eaf | grep scvmclient |grep -v grep

root 6241 1 1 Dec06 014007optspringpathstorfs-corescvmclient-T logEnabled=false -TlogSyslogEnabled=true -TlogEchoToScreen=false -TstatLoggingToFile=false -TstatLoggingToSyslog=true-TlogDir=varlogspringpath-TnfsBackendServerList=1010748100-T iscsiEnable=true -TiscsiUseAsync=true -T

Release Notes for Cisco HX Data Platform Release 4053

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

iscsiConfigFilePath=nfsSYSTEMistgtconfps -eaf | grep scvmclient |grep -v grep

35(2e)Following procedure can be usedto check NFS Queue Depth andincrease if needed

rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth256 lt- Low value[rootHXESXI1] vsish -e setvmkModulesnfsclientmnt[DATASTORE]propertiesmaxQDepth 1024[rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth1024 lt- Optimalvalue

This requires an ESXi host rebootto take effect

Please place the node in HyperflexMaintenanceMode and gracefullyreboot the node for the changes tobe applied

HyperFlex Datastore NFS Queue Depthshows as 256 which can lead toperformance (including latency) issues

CSCvr83056

40(1b)This is cosmetic The error will goaway once the cluster returns tohealthy status

When a single node is offline in clusterstcli cluster storage-summary shows twonodes unavailable rootSCVM~ stclicluster storage-summary

CSCvs47735

35(2g)Modify both the root and adminpassword for the SCVMs to be thesame

stcli security password set -uadmin

Cluster expansion fails at Config Installerstage when the root and admin passwordfor the storage controller (SCVM) aredifferent

CSCvs69317

35(2d)NAAfter successfully changing the DNSserver on the HX controller we still cansee the original DNS entry that was addedduring the deployment

CSCvs69154

Release Notes for Cisco HX Data Platform Release 4054

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Customers are strongly encouragedto work with TAC in order toidentify and remediate this issue

1 Change getcluster from 2m to8m inoptspringpathstorfs-mgmtstMgr-10confapplicationconfon all nodes

2 Restart stMgr on all nodes

You may see this error message after afailed upgrade or other task such asattempting to enter a node into HXmaintenance mode

CSCvs53555

40(2a)Run the following command oneach storage controller VM as root

grep -i monitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp sed -iesmonitor_dns_serverstruemonitor_dns_serversfalseoptspringpathhx-diag-toolswatchdog_configjsonampamp grep -imonitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp restart watchdog

Receive the following alertevent in HXConnect HX Controller VMHOSTNAME one or more configuredDNS servers not responding

CSCvt13947

35(2b)Delete any empty (Size 0) log filesunder varlogzookeeperversion-2ORvarlogzookeeperstandaloneversion-2

Zookeeper fails to start while Exhibitor isrunning however echo srvr returnsnothing

CSCvs21562

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

40(4e)1 De-commission the server

2 Power drain the server -REMOVE BOTH POWERCORDS ON THE BACK OFTHE SERVER FOR 10SECONDS THENREINSERTPOWERCORDS

3 Re-commission the server

M2 boot disk is missing from serverinventory after upgrade to server firmware40(4e) As a result server fails to boot toOS installed in the M2 disk The issuepersists after re-acknowledgement as wellas de-commission andre-acknowledgement of the server

CSCvs02466

Release Notes for Cisco HX Data Platform Release 4055

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Re-start the flusher manuallyMany disks taken offline by a fault ormanually resulting in ENOSPC oncluster

CSCvq32721

35(2e)Clean up fail over cluster and thenrecreate the fail over cluster Thereis no need to touch the HX storagecluster

[Hyper-V]When replicated DCwas usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2b)Manually install upgradescvmclient VIB andmake sure thatit matches with HXDP version

Upgrade from release HX 35(2a) to35(2x) 4X does not upgrade scvmclientRecovery point may not work

CSCvq54992

Management

30(1b)To recover the VM copy over thedata disks and attach them to thenew VM

[HyperV]VMshows up onmultiple nodesCSCvj22992

35(1a)1 Do not change MTU at ESXlevel on running cluster

2 Reset back to the original value

ChangingMTU from 9000 to 1500 on oneESX node causes storfs process to restarton all nodes

CSCvm99150

Release Notes for Cisco HX Data Platform Release 4056

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)After the installation (for newdeployments) or after an upgradeto 401a (for existingdeployments) perform thefollowing steps on all ControllerVMs

1 Edit the following tune files (onall Controller VMs)

optspringpathconfiglfftunes

optspringpathconfigvsi_16tbtunes

2 SetcleanerEnableSegSummaryCleaning

to false

3 After editing the above tunefiles

a SSH in to all the ControllerVMs

b Run storfstool -- -Z

c Run the below command andcheck the values Tune valueshould be true

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=true

d Type below commands andapplied the tunes changesdynamically

echo false gttmpstprocfssystemtunecleanerEnableSegSummaryCleaning

e Verify that tune values arechanged Run below commandValue should be false

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=false

f umount tmpstprocfs

A cluster with 8TB or 12TB disk drivesmay experience IO stalls for severalminutes after another node fails in thecluster

CSCvp23718

Release Notes for Cisco HX Data Platform Release 4057

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)1 Update the conf file manuallyto change to the link local addressto

2 Restart scvmclient on thecontroller and ESX

HX 35(2b) FlexVol to HX 40 CSIupgrade does not work FlexVol continuesto work

CSCvp26319

35(2a)Disable rebalance until node healsthen reenable rebalance

7-node cluster healing very slowlybecause rebalance change AMS reqs areblocking healing process

CSCvp62512

35(2a)NAWith moderate cluster storage capacityusage under site failover cluster clustermay experience an all paths (APD) downcondition

CSCvq49412

40(1a)NAThis product includes Third-partySoftware that is affected by thevulnerabilities identified by the followingCommon Vulnerability and Exposures(CVE) IDs

CVE-2016-0762 CVE-2016-6797CVE-2016-6816 CVE-2016-8735CVE-2017-5647 CVE-2017-12615CVE-2017-12616 CVE-2017-12617CVE-2017-7674 CVE-2018-1304CVE-2018-8014 CVE-2018-1336CVE-2018-8034 CVE-2018-11784CVE-2019-0232

CSCvp36364

35(2d)NAstcli cluster info command needs toprovide UCSM VIP address Currently itshows ucsm-hostcom

CSCvq11456

35(2a)No workarounds currently Thestale entries do not affect the dayto day operations of HyperFlex

HyperFlex upgrade validation failedbecause cluster has extraneous stNodes ininternal database

CSCvq32530

35(2a)NAWhen not in maintenancemode all drivesappear in HX Connect and get updated asexpected

When in maintenance mode extra drivesare shown in HX Connect with no slotnumbers

CSCvq42378

35(2b)Contact TAC to cleanup duplicatenode

Nodes duplicated in HX Connect aftercluster recreate

CSCvq61825

Release Notes for Cisco HX Data Platform Release 4058

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)NACurrently HyperFlex installs on Hyper-Vdo not contain the stcli security whitelistcommandset

CSCvq66245

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

Release Notes for Cisco HX Data Platform Release 4059

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)[Hyper-V] Converged node expansionfails for incoming node with VIC 14xx

CSCvq39523

Release Notes for Cisco HX Data Platform Release 4060

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

1 Set Jumbo frame inWindowsHypervisor for incomingnode

Get-NetAdapter storage-data-a|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

Get-NetAdapter storage-data-b|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

2 Verify all interfaces are setcorrectly to support Jumbo frames

Get-NetAdapter storage |Get-NetAdapterAdvancedProperty| RegistryKeyword -matchjumbo | ft -auto

You should receive the followingmessage

Name DisplayName DisplayValue

RegistryKeyword RegistryValue

---- ----------- ------------

--------------- -------------

vswitch-hx-storage-data Jumbo

Packet 9014 Bytes

JumboPacket 9014

storage-data-b Jumbo Packet

Bytes 9014 JumboPacket

9014

storage-data-a Jumbo Packet

Bytes 9014 JumboPacket

9014

3 Reboot only the incoming nodeor the node being expanded

4 Retry Cluster Expansion in

Release Notes for Cisco HX Data Platform Release 4061

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

Mixed Cluster Expansion Guidelinesbull Expanding existing M4 cluster with M5 converged nodes is supported

bull Expanding existing M5 cluster with M4 converged nodes is not supported

bull Expanding existing mixed M4M5 cluster with M4 or M5 converged nodes is supported

bull Adding any supported compute-only nodes is permitted with all M4 M5 and mixed M4M5 clustersusing the HX Data Platform 26 or later Installer Some example combinations are listed here manyother combinations are possibleExample combinationsExpand mixed M4M5 cluster with compute-only B200 C220 C240 M4M5Expand M4 cluster with compute-only B200 M5 C220 M5 C240M5

bull Only expansion workflow is supported to create a mixed cluster Initial cluster creation with mixedM4M5 servers is not supported

bull All M5 servers must match the form factor (220240) type (HybridAF) security capability (Non-SEDonly) amp disk configuration (QTY capacity and non-SED) of the existing M4 servers

bull HX220-M5 will use a maximum of 6 capacity disks (2 disk slots to remain empty) when mixedwith HX220-M4

bull HX Edge SED LFF Hyper-V and Stretched Clusters do not support mixed M4 and M5 clusters

Security FixesThe following security issues are resolved

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-13132 CVE-2019-9924CSCvq6313840(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-11719 CVE-2019-11727CVE-2019-11729

CSCvq7124040(2a)

Release Notes for Cisco HX Data Platform Release 4022

Mixed Cluster Expansion Guidelines

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-1125CSCvr0633940(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2015-9383 CVE-2018-14498CVE-2018-20406 CVE-2018-20852CVE-2019-10160 CVE-2019-13117CVE-2019-13118 CVE-2019-14287CVE-2019-14973 CVE-2019-15903CVE-2019-17546 CVE-2019-18197CVE-2019-18218 CVE-2019-5010CVE-2019-5094 CVE-2019-5481CVE-2019-5482 CVE-2019-9636CVE-2019-9740 CVE-2019-9947CVE-2019-9948

CSCvs0609440(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2017-13168 CVE-2017-18174CVE-2017-18216 CVE-2018-10876CVE-2018-10877 CVE-2018-10878CVE-2018-10879 CVE-2018-10880CVE-2018-10881 CVE-2018-10882CVE-2018-10902 CVE-2018-10938CVE-2018-12233 CVE-2018-12896CVE-2018-13053 CVE-2018-13094CVE-2018-13096 CVE-2018-13405CVE-2018-13406 CVE-2018-14609CVE-2018-14617 CVE-2018-14633CVE-2018-14734 CVE-2018-15572CVE-2018-15594 CVE-2018-16276CVE-2018-16658 CVE-2018-17182CVE-2018-17972 CVE-2018-18021CVE-2018-18690 CVE-2018-18710CVE-2018-6554 CVE-2018-6555CVE-2018-9363

CSCvp6501940(2a)

This vulnerability is due to insufficientHTML iframe protection An attackercould exploit this vulnerability bydirecting a user to an attacker-controlledweb page that contains amaliciousHTMLiframe A successful exploit could allowthe attacker to conduct clickjacking orother clientside browser attacks

NACSCvo9851640(2a)

Release Notes for Cisco HX Data Platform Release 4023

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerability is due to insufficientauthentication for the statistics collectionservice An attacker could exploit thisvulnerability by sending properlyformatted data values to the statisticscollection service of an affected deviceA successful exploit could allow theattacker to cause the web interfacestatistics view to present invalid data tousers

NACSCvj9558440(2a)

The vulnerability is due to insufficientCSRF protections for the web UI on anaffected device An attacker could exploitthis vulnerability by persuading a user ofthe interface to follow a malicious linkA successful exploit could allow theattacker to perform arbitrary actions withthe privilege level of the affected user

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvp2434340(2a)

Vulnerabilities with a version of cJSONidentified by CVE-2019-11834 andCVE-2019-11835

For more information see the relatedCisco Security Advisory

CVE-2019-11834 CVE-2019-11835CSCvq1994940(2a)

HX ESXi image patches for MCEPSCand TAA vulnerabilities with VMwareESXi identified by CVE-2018-12207 andCVE-20190-11135

For more information see the relatedCisco Security Advisory

CVE-2018-12207 CVE-2019-11135CSCvr5439840(2a)

CP networking vulnerabilities affectingthe Linux kernel identified byCVE-2019-11477 CVE-2019-11478CVE-2019-11479

For more information see the relatedCisco Security Advisory

CVE-2019-11477 CVE-2019-11478CVE-2019-11479

CSCvq1954640(2a)

Qualification of Microsoft securitypatches for vulnerabilities withMicrosoftHyper-V hypervisor identified withCVE-2018-12207 CVE-2019-11135

For more information see the relatedCisco Security Advisory

CVE-2018-12207 CVE-2019-11135CSCvr5439940(2a)

Release Notes for Cisco HX Data Platform Release 4024

Security Fixes

DescriptionCVEDefect IDRelease

OpenSSH vulnerabilities identified byCVE-2019-6111

For more information see the relatedCisco Security Advisory

CVE-2016-10708 CVE-2018-15473CVE-2018-20685 CVE-2019-6109CVE-2019-6111

CSCvp7646340(2a)

Multiple vulnerabilities associated withOpenSSL and LibNSS

For more information see the relatedCisco Security Advisory

CVE-2016-2105 CVE-2016-2106CVE-2016-2107 CVE-2016-2109CVE-2016-2176 CVE-2016-2177CVE-2016-2178 CVE-2016-2179CVE-2016-2180 CVE-2016-2181CVE-2016-2182 CVE-2016-2183CVE-2016-6302 CVE-2016-6303CVE-2016-6304 CVE-2016-6305CVE-2016-6306 CVE-2016-6307CVE-2016-6308 CVE-2016-7055CVE-2016-8610 CVE-2017-3731CVE-2017-3732 CVE-2017-3735CVE-2017-3736 CVE-2017-3737CVE-2017-3738 CVE-2018-0495CVE-2018-0732 CVE-2018-0734CVE-2018-0735 CVE-2018-0737CVE-2018-0739 CVE-2018-12384CVE-2018-12404 CVE-2018-5407CVE-2019-1559

CSCvp6667940(2a)

libpng vulnerability identified byCVE-2016-10087

For more information see the relatedCisco Security Advisory

CVE-2016-10087CSCvp6655540(2a)

Vulnerabilities associated with libjpg andlibpng

For more information see the relatedCisco Security Advisory

CVE-2014-9092 CVE-2016-3616CVE-2017-15232 CVE-2018-11212CVE-2018-11213 CVE-2018-11214CVE-2018-1152 CVE-2018-13785

CSCvp3458640(2a)

Vulnerabilities associated with Glibidentified by CVE-2018-16428 andCVE-2018-16429

For more information see the relatedCisco Security Advisory

CVE-2018-16428 CVE-2018-16429CSCvp3120740(2a)

Multiple vulnerabilities associated withcurl expat python 27 python 35 36and 37 freetype and giflib

For more information see the relatedCisco Security Advisory

CVE-2019-15133 CVE-2019-15903CVE-2019-5010 CVE-2019-5481CVE-2019-5482 CVE-2019-9636CVE-2019-9740 CVE-2019-9947CVE-2019-9948

CSCvr3690340(2a)

Release Notes for Cisco HX Data Platform Release 4025

Security Fixes

DescriptionCVEDefect IDRelease

Multiple third party softwarevulnerabilities

For more information see the relatedCisco Security Advisory

CVE-2019-14379 CVE-2019-12384CVE-2019-14439

CSCvq9203240(2a)

Vulnerabilities associated with elfutils

For more information see the relatedCisco Security Advisory

CVE-2018-16062 CVE-2018-16402CVE-2018-16403 CVE-2018-18310CVE-2018-18520 CVE-2018-18521CVE-2019-7149 CVE-2019-7150CVE-2019-7665

CSCvq4325040(2a)

Vulnerabilities associated with vim

For more information see the relatedCisco Security Advisory

CVE-2017-5953 CVE-2019-12735CSCvq4323040(2a)

Vulnerabilities associated with sqlite3

For more information see the relatedCisco Security Advisory

CVE-2016-6153 CVE-2017-10989CVE-2017-13685 CVE-2017-2518CVE-2017-2519 CVE-2017-2520CVE-2018-20346 CVE-2018-20505CVE-2018-20506 CVE-2019-8457CVE-2019-9936 CVE-2019-9937

CSCvq4321340(2a)

Vulnerabilities associated with glib20

For more information see the relatedCisco Security Advisory

CVE-2018-20843CSCvq4320940(2a)

Vulnerabilities associated with expat

For more information see the relatedCisco Security Advisory

CVE-2018-20843CSCvq4320540(2a)

Vulnerabilities associated with bzip2

For more information see the relatedCisco Security Advisory

CVE-2016-3189 CVE-2019-12900CSCvq4319440(2a)

Mulitple Vulnerabilities associated wtihNodeJS

For more information see the relatedCisco Security Advisory

CVE-2018-12115 CVE-2018-0734CVE-2018-5407 CVE-2018-12120CVE-2018-12121 CVE-2018-12122CVE-2018-12123 CVE-2018-12116CVE-2019-5737 CVE-2019-5739

CSCvq1069440(2a)

Vulnerabilities associated with PalletsJinja strformat_map and identified byCVE-2019-10906 and CVE-2016-10745

For more information see the relatedCisco Security Advisory

CVE-2019-10906 CVE-2016-10745CSCvq1038840(2a)

Release Notes for Cisco HX Data Platform Release 4026

Security Fixes

DescriptionCVEDefect IDRelease

Multiple Security Vulnerabilities

For more information see the relatedCisco Security Advisory

CVE-2014-8501 CVE-2014-9939CVE-2015-9262 CVE-2016-10087CVE-2016-2226 CVE-2016-4487CVE-2016-4488 CVE-2016-4489CVE-2016-4490 CVE-2016-4491CVE-2016-4492 CVE-2016-4493CVE-2016-6131 CVE-2018-10963CVE-2018-13785 CVE-2018-17100CVE-2018-17101 CVE-2018-18557CVE-2018-18661 CVE-2018-7456CVE-2018-8905

CSCvq0917840(2a)

Vulnerabilities associated withlibseccomp

For more information see the relatedCisco Security Advisory

CVE-2019-9893CSCvq0756840(2a)

Vulnerabilities associated withgdk-pixbuf

For more information see the relatedCisco Security Advisory

CVE-2017-12447CSCvq0675540(2a)

Vulnerabilities associated withpython-crypto

For more information see the relatedCisco Security Advisory

CVE-2018-6594CSCvp9381740(2a)

Vulnerabilities associated with wget

For more information see the relatedCisco Security Advisory

CVE-2018-20483 CVE-2019-5953CSCvp8672140(2a)

Multiple vulnerabilities associated withPython Crypto

For more information see the relatedCisco Security Advisory

CVE-2018-6594CSCvp6674840(2a)

Vulnerabilities associated withsensible-utils

For more information see the relatedCisco Security Advisory

CVE-2017-17512CSCvp6673440(2a)

Multiple vulnerabilities associated withPython

For more information see the relatedCisco Security Advisory

CVE-2018-1000030CSCvp6668940(2a)

Release Notes for Cisco HX Data Platform Release 4027

Security Fixes

DescriptionCVEDefect IDRelease

Multiple patch vulnerabilities identifiedby CVE-2016-10713

For more information see the relatedCisco Security Advisory

CVE-2016-10713CVE-2018-1000156 CVE-2018-6951

CSCvp6667240(2a)

Vulnerabilities associatedwith Little CMSidentified by CVE-2016-10165

For more information see the relatedCisco Security Advisory

CVE-2016-10165 CVE-2018-16435CSCvp6666440(2a)

Vulnerabilities associated with TenableScan identified byCVE-2011-5325CVE-2019-11068CVE-2016-7076CVE-2017-1000368

For more information see the relatedCisco Security Advisory

CVE-2011-5325 CVE-2016-7076CVE-2017-1000368CVE-2019-11068

CSCvp6474640(2a)

Vulnerabilities associated with libxml2294 and earlier as used in XMLSec1223 and earlier and other products andidentified by CVE-2016-9318

For more information see the relatedCisco Security Advisory

CVE-2016-9318 CVE-2017-16932CVE-2017-18258 CVE-2018-14404CVE-2018-14567

CSCvp3479240(2a)

Vulnerabilities associated with LFTPRemote File Names Unauthorized AccessVulnerability and identified byCVE-2018-10916

For more information see the relatedCisco Security Advisory

CVE-2018-10916CSCvp2926640(2a)

Vulnerabilities associated withCryptographyDeprecationWarning signerand verifier have been deprecated

For more information see the relatedCisco Security Advisory

CVE-2018-7750CSCvo3409740(2a)

Multiple Vulnerabilities in python python27 35

For more information see the relatedCisco Security Advisory

CVE-2019-16056CSCvr3979340(2a)

Release Notes for Cisco HX Data Platform Release 4028

Security Fixes

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toexecute commands as the root user

The vulnerability is due to insufficientinput validation An attacker could exploitthis vulnerability by connecting to thecluster service manager and injectingcommands into the bound process Asuccessful exploit could allow the attackerto run commands on the affected host asthe root user

Cisco has released software updates thataddress this vulnerability There areworkarounds that address thisvulnerability

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvq2417640(1b)35(2g)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

The vulnerability is due to an unprotectedlistening interface An attacker couldexploit this vulnerability by connectingto the listening interface and injectingcommands to the bound process Anexploit could allow the attacker to runcommands on the affected host as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvj9560640(1b)35(2g)

Release Notes for Cisco HX Data Platform Release 4029

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerabilities associated with theJVM18U121memory leak found duringVC alarms (concurrent 40 calls usingREST API)

CVE-2017-10053 CVE-2017-10067CVE-2017-10074 CVE-2017-10078CVE-2017-10081 CVE-2017-10087CVE-2017-10089 CVE-2017-10090CVE-2017-10096 CVE-2017-10101CVE-2017-10102 CVE-2017-10107CVE-2017-10108 CVE-2017-10109CVE-2017-10110 CVE-2017-10111CVE-2017-10115 CVE-2017-10116CVE-2017-10118 CVE-2017-10135CVE-2017-10176 CVE-2017-10193CVE-2017-10198 CVE-2017-10243CVE-2017-10274 CVE-2017-10281CVE-2017-10285 CVE-2017-10295CVE-2017-10345 CVE-2017-10346CVE-2017-10347 CVE-2017-10348CVE-2017-10349 CVE-2017-10350CVE-2017-10355 CVE-2017-10356CVE-2017-10357 CVE-2017-10388CVE-2017-3509 CVE-2017-3511CVE-2017-3526 CVE-2017-3533CVE-2017-3539 CVE-2017-3544CVE-2018-2579 CVE-2018-2582CVE-2018-2588 CVE-2018-2599CVE-2018-2602 CVE-2018-2603CVE-2018-2618 CVE-2018-2629CVE-2018-2633 CVE-2018-2634CVE-2018-2637 CVE-2018-2641CVE-2018-2663 CVE-2018-2677CVE-2018-2678 CVE-2018-2783CVE-2018-2790 CVE-2018-2794CVE-2018-2795 CVE-2018-2796CVE-2018-2797 CVE-2018-2798CVE-2018-2799 CVE-2018-2800CVE-2018-2814 CVE-2018-2815CVE-2018-2952 CVE-2018-3136CVE-2018-3139 CVE-2018-3149CVE-2018-3150 CVE-2018-3169CVE-2018-3180 CVE-2018-3183CVE-2018-3214 CVE-2019-2422

CSCvo8899740(1b)

Tomcat and Nginx logs are not beingcollected in the support bundle generatedthrough HX Connect in Release 35

NACSCvm5803140(1b)

The vulnerabilities associated with thelibmspack software package versionincluded in Cisco HX Data Platform

CVE-2018-18584

CVE-2018-18585

CSCvn3511940(1a)

Release Notes for Cisco HX Data Platform Release 4030

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerabilities associated withFasterXML Jackson-Databind TimeValueField Denial of Service

CVE-2018-14719

CVE-2018-14720

CVE-2018-1000873

CVE-2018-14721

CVE-2018-19360

CVE-2018-19362

CVE-2018-19361

CVE-2018-14718

CSCvn8228240(1a)

The vulnerabilities associated with theOpenSSL Protocol software packageversion included in Cisco HX DataPlatform

CVE-2013-3587CSCvo0505440(1a)

The vulnerabilities associated withThird-Party Software Denial of Service

CVE-2018-16487

CVE-2018-19361

CSCvo2781840(1a)

The vulnerabilities associated withLinuxkernel for Ubuntu 1710

CVE-2018-1092

CVE-2018-7492

CVE-2018-8087

CVE-2018-1068

CVE-2018-8781

CSCvm5314935(2a)

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)25(1d)stcli services dns remove should remove the DNS serverinfo from the interface files

CSCvs70967

40(2b)40(1b)HX 40(1b) new deployment from Intersight productioncloud

User receives following error while enabling CSI(Kubernetes integration)

Failure occurred during volume_access_enable Errorwas Nonzero exit code 1

CSCvt10522

40(2b)40(2a)Receive the following alertevent in HX Connect HXController VM HOSTNAME one or more configuredDNS servers not responding

CSCvt13947

Release Notes for Cisco HX Data Platform Release 4031

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)40(2a)API UI CLI show few Drive slots Empty after upgradeto 402a

The drives are seen in the sysmtool lsscsi and stclicommand outputs

The cluster is healthy with no errors

CSCvt14914

40(2b)35(2d)After successfully changing the DNS server on the HXcontroller we still can see the original DNS entry thatwas added during the deployment

CSCvs69154

40(2b)35(2e)HyperFlex Connect and VCenter show APD alarms forHX and non HX datastores This implies there is an issuewith HyperFlex when there may not be

CSCvs30080

40(2b)40(2a)An issue where all the disks in the node get locked afterrebooting the node

We successfully unlocked using sed-clientsh -Ucommand but wanted to test with another reboot anddrives locked again

CSCvs74286

40(2b)40(2a)When running stcli license commands on HyperFlexthe following error is seen

rootSpringpathControllertmp stcli license show all

Show smart licensing failed

Smart Agent is not ready please wait a minute and tryagain

CSCvt13929

40(2b)40(2a)After upgrading to HX 40(2a) the first RoboEdge nodethat gets rebooted then ignores persistent drivesPIDHX-HD24TB10K4KN

Cluster will be in degraded state and have reducedcapacity

CSCvt20144

40(2b)35(2b)Zookeeper fails to start while Exhibitor is runninghowever echo srvr returns nothing

CSCvs21562

40(2b)40(2a)When performing an HyperFlex upgrade a validationwarning may occur due to the host not having EnterprisePlus or Enterprise hypervisor licensing

Upgrade Validation Warning

ESXi host esx1labtest should be configured withVMware Enterprise license for upgrade to continue

CSCvs91787

Release Notes for Cisco HX Data Platform Release 4032

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)35(2g)Cluster expansion fails at Config Installer stage whenthe root and admin password for the storage controller(SCVM) are different

CSCvs69317

40(2b)35(2g)Panic while upgrading ESXiCSCvt06983

40(2b)40(2a)A cluster node running may hang in the Linux kernelThis is classified as an oops and a deviation from theexpected behavior

CSCvs54285

40(2b)35(2g)Rebalance failing on 10+10 node HX 35(2g) stretchcluster

CSCvs69007

40(2b)35(2d)The cluster becomes inaccessible to the IOVisorCSCvr54687

40(2b)40(2a)Panic on the storage controllerCSCvt61297

40(2b)40(2a)The size of support bundle is very large when collectedwith storfs-support command

CSCvt63306

40(2b)40(2a)storfs crash associated withFileSystemUsageWarningEvent orFileSystemUsageAlertEvent due to high usage ofvarstv above 80

CSCvt72807

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

40(2a)35(2d)In normal scenarios HX upgrade gets pendingACK afterupdating service profile In this case UCS only upgradewas stalled because after updating service profile HXupgrade was not getting pending ACK and it was alwayswaiting state

CSCvo39912

40(2a)35(1a)When using motherBoardReplace-12 to clean ZK fromold stNodepNodes resulted in unmounted datastore andmaking the size of the Hyperflex Datastores 0 resultingin all VMs in the cluster going offline

CSCvq39471

40(2a)26(1e)Upgrading cluster and getting error

Failed upgrade validations Checking vCenterconfiguration Reason Upgrade validations failed DRSFault Insufficient resources to satisfy configured failover

CSCvm77294

Release Notes for Cisco HX Data Platform Release 4033

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)The cluster expand fails on a node with DR replicationconfigured

When a HX cluster which has DR replication configuredis expanded we see the installer UI pulling in thereplicationVLAN information instead of themanagementVLAN information Even if we change that informationto the correct mgmt VLAN id and name it it does notseem to work as the node is configured with the VLANof the replicationVLAN in ESXi This leads to the failureof the node add with the host unreachable error

CSCvo70650

40(2a)35(1a)Customer reported that server didnt complete thefirmware upgrade automatically one by one

The user manually needs to put the host into maintenancemode and they manually acknowledged the pendingrequests to finish the UCS firmware upgrade

Maintenance policy is set to default (User-ack) as perthe design

CSCvo91624

40(2a)35(1i)When the cluster is in Failed state and stcli noderemove is attempted the output shows successful thoughthe node failed to get removed from the cluster

CSCvo93017

40(2a)35(1a)HyperFlex cluster upgrade may fail during validationwith a DRS Validation failed error

CSCvp31021

40(2a)35(2b)HX cluster expansion will fail with the following errormessage

MAC address pool configuration failure150[ErrorDescription] bad address block range definitioncollision

CSCvp58318

40(2a)35(2b)Memory usage by carbon cacheCSCvq34873

40(2a)35(2g)Upgrading ESXi cluster fails with error Nodemaintenance mode failed

CSCvr03240

40(2a)35(2e)storfs process does not automatically start on Exit HXMaintenance Mode or other tasks which restart StorageController

CSCvr88978

40(2a)35(2a)Post install script fails with message

Failed to execute ipmitool on HX node

CSCvp10707

40(2a)40(1a)HyperFlex expansionworkflow doesnt pull VLANnamecorrectly

CSCvp46539

Release Notes for Cisco HX Data Platform Release 4034

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)During HX cluster deployment - Validate ClusterCreation phase - HX Installer might fail with the error

fromvarlogspringpathstDeploylog

CSCvq45087

40(2a)35(2b)HX Installer fails to login to the SOL for configuringESX hosts

CSCvq91380

40(2a)35(2d)If DRS is enabled with configuration parameters otherthen fully automated ie manual or partially automatedthan this will change to fully automated during clusterexpansion

CSCvr44222

40(2a)35(1a)One of the validation task name has a typo The taskname is Validating node for mixed of SED and non-SEDdisks

It should be Validating node for mix of SED andnon-SED disks

CSCvp82175

40(2a)30(1i)This combination of using smaller NVMe drives is notsupported in expand operation For example if the drivesare only 375GB and they cannot be added to an existingcluster with larger caching SSD

CSCvo12359

40(2a)35(1a)HyperFlex includes a version of OpenSSL that is affectedby the vulnerabilities identified by the followingCommon Vulnerability and Exposures

CVE-2018-0495

CSCvp66679

40(2a)30(1e)SDK invoke exception when taking a native snapshot ona VM with GPU pci passthrough attached

CSCvo00511

40(2a)30(1i)Node replacement script fails due to EAM errorCSCvo62867

40(2a)35(1a)While pairing a cluster (HX release gt= 35) and cluster(HX release lt 35) remote replication network test failson the cluster (HX release lt 35)

CSCvo79760

40(2a)30(1b)Fixed the issue in the latest support-workflow bundleThis should not be hit anymore

CSCvo87061

40(2a)35(2a)On a 3 node cluster the MBreplace script fails as thescript looks for HEALTHY cluster status

CSCvo87080

40(2a)35(2a)MbReplace script (tar) hangs when running on 352b or40

CSCvp12359

Release Notes for Cisco HX Data Platform Release 4035

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)HX replication cleanup failing with error NA 35(2a) -INFODR state is not clean

CSCvp63958

40(2a)40(1a)The check_stig_parameters API incorrectly shows thecompliance state of the cluster as not STIG compliant

CSCvp66277

40(2a)35(2c)The snapshot creation on CBT enabled VM fails witherror Failed in vmreparent vmkfstools clone1

CSCvq06952

40(2a)35(2a)HyperFlex GUI when setting up remote DR-replicationpartnerpeer should show remote replication partner TCPport reachability test results

This will provide quick upfront connectivity results thatfrequently takes customer a while to manually check andconfirm

If reachability tests fail an info window and HX alertshould trigger that list the destination peer IP and thespecific port(s) that could not be reache

CSCvo60587

40(2a)35(2b)If you have more than one IP pool configured for areplication network and local replication network testsare failing you might be running into this issue

CSCvr67130

40(2a)35(2a)Skip task option in MBreplace and Software redeployscripts The scripts should have the option to skip tasksas user inputs and not by modifying the json file

CSCvp05204

40(2a)40(1a)Error noticed on SCVM Console print_req_error IOerror dev fd0 sector 0 on HyperV

It doesnt come up on SSH session

CSCvq34357

40(2a)30(1d)The VMNetwork switch is already created as part of theinstall process

Ideally the ip address assignment should be done duringthe initial install process as well No post install stepsshould be required altrnatively it could be done as partof a post install script

CSCvk36222

40(2a)25(1c)storfs service impacted during HX upgrade from HX21(1) to 25(1c)

HX node which is not being upgraded storfs servicestopped causing outage

CSCvg53223

40(2a)35(1b)HX Stretch Cluster Witness VM reverts to DHCP onreboot

CSCvp37536

40(2a)35(2b)HyperFlex Stretch Cluster Witness does not getprogrammed with an NTP server

CSCvp46578

Release Notes for Cisco HX Data Platform Release 4036

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2b)HyperFlex cluster does not heal after maintenance onnodeserver

CSCvr18528

40(2a)40(2a)High percentage of packet loss on the eth1 data interface9K MTU

CSCvr97089

40(2a)35(2f)The HyperFlex cluster healing state is stuck at 87 andis unable to progress

CSCvr16760

40(2a)40(2a)In some circumstances two replications can occur forthe same VM

From the GUI the replication status for this VM willshow stuck in an In Progress state but there are nojobs stuck in replication layer Subsequent Replicationsshould succeed

CSCvq22898

40(2a)35(2a)After an unexpected power outage the hyperflex clusteris unable to come up correctly

All processes are running time is synced and all nodescan vmkping each other as required

CSCvp13990

40(2a)40(1a)When specifying IP address instead of node ID stclinode maintenanceMode commands fail

CSCvp33657

40(2a)40(1a)2 node edge cluster if experiencing Intersightarbitratorconnectivity issues should NOT allowHXMaintenancemode to initiate as this will bring the cluster down

CSCvr31573

40(2a)35(2c)Old files in varsupportZKTxnlog are not purged withthe daily zklog-cleanup cron job

CSCvr89066

40(2a)35(2a)When a node is in MaintenanceMode any disk removalor replacement will be reflected in UI only after the nodeis brought back frommaintenancemode This is becausestorfs is not running on the node in maintenance andwill not be able to detect disk activities until it is broughtout of Maintenance Mode

CSCvo86431

40(2a)35(2c)Cluster might go offline and stop serving IO requestsrootcvm~ stcli cluster storage-summary--detailGet cluster storage summary failedjavanetConnectException Connection refused19216814210010207

CSCvr01645

40(2a)35(2d)A stretch cluster deployment fails with a messageFormatting nodes Node in Use on the Hyperflex installerand crmZoneType value is seen as 1 in storfscfg oroptspringpathconfigstretchtunes

CSCvq80340

Release Notes for Cisco HX Data Platform Release 4037

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)30(1c)During upgrade we upload the upgrade packagedownloaded from Ciscocom Upgrade packages are offile type tgz hxconnect acceptsstorfs-packages-401a-33028tar file a different type ofcompression and allows upgrade to start and fail whenit cannot decompress the file for bootstrap and scvmclientupgrade on ESXi

CSCvq63888

40(2a)40(2a)While attempting to install hyperflex via intersight thefollowing error is seen

Failed in Task Add hosts to vCenter Cluster with ErrorTry adding hosts manually to vCenter and retry Failedto get addressFailed to host by namelookupHostnamecompanycom on 000053 read udp12700145430-gt12700153 read connection refused

CSCvr52098

40(2a)35(2a)CoreAPI call may time out for cluster management APIwhile doing cluster create validation

CSCvo92952

40(2a)35(2d)Validation fails with Mixed mode expansion checkdue to Enhanced vMotion Compatibility Incompatibilitywhen the ESXi version is different

CSCvq95460

40(2a)30(1i)This failure is seen in installer when custom workflowis used hypervisor configuration + deploy (clean diskpartition=no) at Installing software packages on storagecontroller VM

CSCvr66309

40(2a)35(2d)The HX Installer apparently replaced a comma bysemicolon on the VLAN ID range while exporting theconfiguration to a JSON file

CSCvq93831

40(2a)40(1a)When using unsupported browsers HX Connect userswill not be able to login These users see an errorindicating unauthorized user

CSCvp22693

Release Notes for Cisco HX Data Platform Release 4038

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)40(1a)A vulnerability in the web-based management interfaceof Cisco HyperFlex Software could allow anunauthenticated remote attacker to conduct a cross-siterequest forgery (CSRF) attack on an affected system

The vulnerability is due to insufficient CSRF protectionsfor the web UI on an affected device An attacker couldexploit this vulnerability by persuading a user of theinterface to follow a malicious link A successful exploitcould allow the attacker to perform arbitrary actions withthe privilege level of the affected user

Cisco has released software updates that address thisvulnerability There are no workarounds that address thisvulnerability

This advisory is available at the following link

httpstoolsciscocomsecuritycentercontentCiscoSecurityAdvisorycisco-sa-20190807-hypflex-csrf

CSCvp24343

40(2a)35(2d)Add message to HX Connect Progress flow to notAcknowledge Pending Activities and reboot the serverson UCSM

HX Connect is doing a controlled rolling server upgradein the background

CSCvq22844

40(2a)40(1b)Compute Node expand operation fails due to incorrectcluster name

CSCvr43786

40(2a)26(1e)RF-2 cluster shutdown during data resynchronizationafter a node failure (non storfs) followed by multipledisk read failures and a HardBlacklist of a disk

CSCvp41241

40(2a)40(1a)HyperFlex Edge nodes do not properly set the ESXihostname during deployment

CSCvo13143

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4039

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4040

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1b) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401b-33133p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4041

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4042

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1b)40(1a)While running the HyperFlex installer with WindowsServer Hyper-V cluster creation process fails with theerror Failure occurred during ClusterCreation process Unable to post thecontent to the downstream This symptomis encountered while deploying HyperFlex cluster withHyperFlex nodes configuredordered with Cisco VIC1457 MLOM (PID HX-MLOM-C25Q-04) andWindows Server Datacenter or Core with Hyper-Vversion 2016 or 2019

CSCvp64140

Release Notes for Cisco HX Data Platform Release 4043

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)35(2b)Adding Micron 5200 drive to a cluster fails to increasecluster capacity

CSCvo69067

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4044

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4045

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1a) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401a-33028p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4046

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4047

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1a)30(1d)Cluster instability when disks of different sector sizeplaced in HX node

CSCvk17250

40(1a)

35(2c)

35(1a)When logged in using a local HX user account insteadof a Virtual Center account an error message appearsintermittently indicating Virtual Centerunreachable or Resource informationcannot be updated when VC is reachable

CSCvo36198

40(1a)

35(2b)

30(1d)HXDP does not work with EMC RecoverPoint - needsto support VMware API (FSS-Readdir)

CSCvk38003

Release Notes for Cisco HX Data Platform Release 4048

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)

35(2a)

30(1i)

25(1c)

25(1d)

Zookeeper (Exhibitor) process on HyperFlex StorageController VMs may fail to respond to requests whenvarzookeeper has low or no free space

CSCvm90352

40(1a)26(1c)Use Asynchronous consolidation for HX SnapshotsCSCvn02151

40(1a)35(2b)HX Quiesced Snapshot for Backup VendorsCSCvo90713

40(1a)35(2a)The cluster creationcluster expansion workflow stopswith the following error message at the validation step

FIRMWARE-Check UCSC-SAS-M5HD

FIRMWARE-Check UCSC-SAS-M5HD Required00000029000000320000003500000050 Found00000058

Action Needed Update the Controller Firmware toRequired Version

CSCvn17787

35(2b)30(1d)Kernel migration fails when a local datastore is searchedfor in the ESXi host

CSCvn73127

35(2a)26(1b)A node shuts down when two disks are replaced (acaching disk and another capacity disk) where thecapacity disk is inserted first and then the caching diskis inserted

CSCvk46364

Hyper-V

40(1a)35(2a)Cluster expansion may fail with an error code 500 -operation timed out

CSCvn28721

35(2b)35(2a)During upgrade remove the VLAN on team created foruser vSwitch During fresh installation only one VLANtag is set to the vSwitch and team although multipleVLANs were entered

CSCvn54300

35(2b)35(2a)While upgrading a Hyper-V cluster on account of a rarerace condition between the stUpgradeService andZookeeper servers the upgrade orchestration throws anupgrade validation error and the upgrade process isaborted

CSCvn60486

Release Notes for Cisco HX Data Platform Release 4049

Resolved Caveats in Release 40(1a)

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Startrestart EAM service invCenter

When EAM service is not runningvCenter registration is failing

CSCvt55712

35(2a)Contact Cisco TAC if you believeyou are seeing similar conditions

HyperFlex Stretch Cluster saw poorapplication performance due to writelatency cluster remained unhealthy andrebalance was stuck

This defect is currently being used forinvestigative purposes

CSCvt45344

35(2g)Remove log files fromvarlogspringpath folder when thelog files are not getting rotatedStop the relevant service removelarge log files fromvarlogspringpath folder andrestart the service

varstv folder in Controller VM maybecome full

CSCvt36374

40(2b)Restarting the stMgr will initializeit again

In a Cluster configured with DR networkStMgr may not get initialized and stuckin a deadlock while enabling the IPTablerules (can be verified from the stMgrlog)

CSCvt89709

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Click recover for the selectedVM again under the standalonemode This will move the VM torecovered state

When the user recovers a VM protectedin a group it along with other VMs in thegroup move to recovering state instandalonemode The selectedVM shouldmove to recovered state

CSCvs75553

35(2e)Clean up the fail over cluster andthen recreate the fail over clusterNo need to touch the HX storagecluster

Hyper-V When replicated DC was usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2d)40(1a)

Keep cluster utilization as low aspossible To avoid read-only statekeep below 92

HyperFlex clusters storage utilization isabove 92 usable capacity

CSCvr20922

Release Notes for Cisco HX Data Platform Release 4050

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1b)NAAll the disks in the node got locked afterrebooting the node

We successfully unlocked usingsed-clientsh -U command but wantedto test with another reboot and driveslocked again

CSCvs74286

35(2g)To recover the VM copy over thedata disks and attach them to thenew VM

After enable encryption HX connectencryption status showing Caution butall of the disks enable encryptionsuccessful

-Self Encrypting Drives Service isrunning on all of the nodes

-There is no error message of encryption

- USB0 interface is up

-All SED disks showing supported 1enabled 1 locked 0

CSCvs93245

40(1b)Enable kernelpanic_on_oops as apersistent configuration This willcause the node to panic and rebootimmediately

A cluster node running HX release40(1b) may hang in the Linux kernelThis is classified as an oops and adeviation from the expected behavior

CSCvs54285

Release Notes for Cisco HX Data Platform Release 4051

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Enabling K8iscsi stack in Hx clusterCSCvs41324

Release Notes for Cisco HX Data Platform Release 4052

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Steps to enable K8 on HX cluster

Perform following steps on ALLthe controller VMs

1 Updateetcinitscvmclientconf toenable tune iscsiEnable totrue

sed -iesiscsiEnable=falseiscsiEnable=trueetcinitscvmclientconf

2 Run following initctlcommand to reloadconfiguration

initctl reload-configuration

3 Restart scvmclient process

stop scvmclient startscvmclient

4 Run following initctlcommand to mount SYSTEMDS

initctl emit --no-waitsystem-datastore-created

5 Verify that iscsiEnable tuneis set to true using thefollowing command

ps -eaf | grep scvmclient |grep -v grep

root 6241 1 1 Dec06 014007optspringpathstorfs-corescvmclient-T logEnabled=false -TlogSyslogEnabled=true -TlogEchoToScreen=false -TstatLoggingToFile=false -TstatLoggingToSyslog=true-TlogDir=varlogspringpath-TnfsBackendServerList=1010748100-T iscsiEnable=true -TiscsiUseAsync=true -T

Release Notes for Cisco HX Data Platform Release 4053

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

iscsiConfigFilePath=nfsSYSTEMistgtconfps -eaf | grep scvmclient |grep -v grep

35(2e)Following procedure can be usedto check NFS Queue Depth andincrease if needed

rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth256 lt- Low value[rootHXESXI1] vsish -e setvmkModulesnfsclientmnt[DATASTORE]propertiesmaxQDepth 1024[rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth1024 lt- Optimalvalue

This requires an ESXi host rebootto take effect

Please place the node in HyperflexMaintenanceMode and gracefullyreboot the node for the changes tobe applied

HyperFlex Datastore NFS Queue Depthshows as 256 which can lead toperformance (including latency) issues

CSCvr83056

40(1b)This is cosmetic The error will goaway once the cluster returns tohealthy status

When a single node is offline in clusterstcli cluster storage-summary shows twonodes unavailable rootSCVM~ stclicluster storage-summary

CSCvs47735

35(2g)Modify both the root and adminpassword for the SCVMs to be thesame

stcli security password set -uadmin

Cluster expansion fails at Config Installerstage when the root and admin passwordfor the storage controller (SCVM) aredifferent

CSCvs69317

35(2d)NAAfter successfully changing the DNSserver on the HX controller we still cansee the original DNS entry that was addedduring the deployment

CSCvs69154

Release Notes for Cisco HX Data Platform Release 4054

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Customers are strongly encouragedto work with TAC in order toidentify and remediate this issue

1 Change getcluster from 2m to8m inoptspringpathstorfs-mgmtstMgr-10confapplicationconfon all nodes

2 Restart stMgr on all nodes

You may see this error message after afailed upgrade or other task such asattempting to enter a node into HXmaintenance mode

CSCvs53555

40(2a)Run the following command oneach storage controller VM as root

grep -i monitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp sed -iesmonitor_dns_serverstruemonitor_dns_serversfalseoptspringpathhx-diag-toolswatchdog_configjsonampamp grep -imonitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp restart watchdog

Receive the following alertevent in HXConnect HX Controller VMHOSTNAME one or more configuredDNS servers not responding

CSCvt13947

35(2b)Delete any empty (Size 0) log filesunder varlogzookeeperversion-2ORvarlogzookeeperstandaloneversion-2

Zookeeper fails to start while Exhibitor isrunning however echo srvr returnsnothing

CSCvs21562

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

40(4e)1 De-commission the server

2 Power drain the server -REMOVE BOTH POWERCORDS ON THE BACK OFTHE SERVER FOR 10SECONDS THENREINSERTPOWERCORDS

3 Re-commission the server

M2 boot disk is missing from serverinventory after upgrade to server firmware40(4e) As a result server fails to boot toOS installed in the M2 disk The issuepersists after re-acknowledgement as wellas de-commission andre-acknowledgement of the server

CSCvs02466

Release Notes for Cisco HX Data Platform Release 4055

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Re-start the flusher manuallyMany disks taken offline by a fault ormanually resulting in ENOSPC oncluster

CSCvq32721

35(2e)Clean up fail over cluster and thenrecreate the fail over cluster Thereis no need to touch the HX storagecluster

[Hyper-V]When replicated DCwas usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2b)Manually install upgradescvmclient VIB andmake sure thatit matches with HXDP version

Upgrade from release HX 35(2a) to35(2x) 4X does not upgrade scvmclientRecovery point may not work

CSCvq54992

Management

30(1b)To recover the VM copy over thedata disks and attach them to thenew VM

[HyperV]VMshows up onmultiple nodesCSCvj22992

35(1a)1 Do not change MTU at ESXlevel on running cluster

2 Reset back to the original value

ChangingMTU from 9000 to 1500 on oneESX node causes storfs process to restarton all nodes

CSCvm99150

Release Notes for Cisco HX Data Platform Release 4056

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)After the installation (for newdeployments) or after an upgradeto 401a (for existingdeployments) perform thefollowing steps on all ControllerVMs

1 Edit the following tune files (onall Controller VMs)

optspringpathconfiglfftunes

optspringpathconfigvsi_16tbtunes

2 SetcleanerEnableSegSummaryCleaning

to false

3 After editing the above tunefiles

a SSH in to all the ControllerVMs

b Run storfstool -- -Z

c Run the below command andcheck the values Tune valueshould be true

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=true

d Type below commands andapplied the tunes changesdynamically

echo false gttmpstprocfssystemtunecleanerEnableSegSummaryCleaning

e Verify that tune values arechanged Run below commandValue should be false

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=false

f umount tmpstprocfs

A cluster with 8TB or 12TB disk drivesmay experience IO stalls for severalminutes after another node fails in thecluster

CSCvp23718

Release Notes for Cisco HX Data Platform Release 4057

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)1 Update the conf file manuallyto change to the link local addressto

2 Restart scvmclient on thecontroller and ESX

HX 35(2b) FlexVol to HX 40 CSIupgrade does not work FlexVol continuesto work

CSCvp26319

35(2a)Disable rebalance until node healsthen reenable rebalance

7-node cluster healing very slowlybecause rebalance change AMS reqs areblocking healing process

CSCvp62512

35(2a)NAWith moderate cluster storage capacityusage under site failover cluster clustermay experience an all paths (APD) downcondition

CSCvq49412

40(1a)NAThis product includes Third-partySoftware that is affected by thevulnerabilities identified by the followingCommon Vulnerability and Exposures(CVE) IDs

CVE-2016-0762 CVE-2016-6797CVE-2016-6816 CVE-2016-8735CVE-2017-5647 CVE-2017-12615CVE-2017-12616 CVE-2017-12617CVE-2017-7674 CVE-2018-1304CVE-2018-8014 CVE-2018-1336CVE-2018-8034 CVE-2018-11784CVE-2019-0232

CSCvp36364

35(2d)NAstcli cluster info command needs toprovide UCSM VIP address Currently itshows ucsm-hostcom

CSCvq11456

35(2a)No workarounds currently Thestale entries do not affect the dayto day operations of HyperFlex

HyperFlex upgrade validation failedbecause cluster has extraneous stNodes ininternal database

CSCvq32530

35(2a)NAWhen not in maintenancemode all drivesappear in HX Connect and get updated asexpected

When in maintenance mode extra drivesare shown in HX Connect with no slotnumbers

CSCvq42378

35(2b)Contact TAC to cleanup duplicatenode

Nodes duplicated in HX Connect aftercluster recreate

CSCvq61825

Release Notes for Cisco HX Data Platform Release 4058

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)NACurrently HyperFlex installs on Hyper-Vdo not contain the stcli security whitelistcommandset

CSCvq66245

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

Release Notes for Cisco HX Data Platform Release 4059

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)[Hyper-V] Converged node expansionfails for incoming node with VIC 14xx

CSCvq39523

Release Notes for Cisco HX Data Platform Release 4060

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

1 Set Jumbo frame inWindowsHypervisor for incomingnode

Get-NetAdapter storage-data-a|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

Get-NetAdapter storage-data-b|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

2 Verify all interfaces are setcorrectly to support Jumbo frames

Get-NetAdapter storage |Get-NetAdapterAdvancedProperty| RegistryKeyword -matchjumbo | ft -auto

You should receive the followingmessage

Name DisplayName DisplayValue

RegistryKeyword RegistryValue

---- ----------- ------------

--------------- -------------

vswitch-hx-storage-data Jumbo

Packet 9014 Bytes

JumboPacket 9014

storage-data-b Jumbo Packet

Bytes 9014 JumboPacket

9014

storage-data-a Jumbo Packet

Bytes 9014 JumboPacket

9014

3 Reboot only the incoming nodeor the node being expanded

4 Retry Cluster Expansion in

Release Notes for Cisco HX Data Platform Release 4061

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2019-1125CSCvr0633940(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2015-9383 CVE-2018-14498CVE-2018-20406 CVE-2018-20852CVE-2019-10160 CVE-2019-13117CVE-2019-13118 CVE-2019-14287CVE-2019-14973 CVE-2019-15903CVE-2019-17546 CVE-2019-18197CVE-2019-18218 CVE-2019-5010CVE-2019-5094 CVE-2019-5481CVE-2019-5482 CVE-2019-9636CVE-2019-9740 CVE-2019-9947CVE-2019-9948

CSCvs0609440(2a)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2017-13168 CVE-2017-18174CVE-2017-18216 CVE-2018-10876CVE-2018-10877 CVE-2018-10878CVE-2018-10879 CVE-2018-10880CVE-2018-10881 CVE-2018-10882CVE-2018-10902 CVE-2018-10938CVE-2018-12233 CVE-2018-12896CVE-2018-13053 CVE-2018-13094CVE-2018-13096 CVE-2018-13405CVE-2018-13406 CVE-2018-14609CVE-2018-14617 CVE-2018-14633CVE-2018-14734 CVE-2018-15572CVE-2018-15594 CVE-2018-16276CVE-2018-16658 CVE-2018-17182CVE-2018-17972 CVE-2018-18021CVE-2018-18690 CVE-2018-18710CVE-2018-6554 CVE-2018-6555CVE-2018-9363

CSCvp6501940(2a)

This vulnerability is due to insufficientHTML iframe protection An attackercould exploit this vulnerability bydirecting a user to an attacker-controlledweb page that contains amaliciousHTMLiframe A successful exploit could allowthe attacker to conduct clickjacking orother clientside browser attacks

NACSCvo9851640(2a)

Release Notes for Cisco HX Data Platform Release 4023

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerability is due to insufficientauthentication for the statistics collectionservice An attacker could exploit thisvulnerability by sending properlyformatted data values to the statisticscollection service of an affected deviceA successful exploit could allow theattacker to cause the web interfacestatistics view to present invalid data tousers

NACSCvj9558440(2a)

The vulnerability is due to insufficientCSRF protections for the web UI on anaffected device An attacker could exploitthis vulnerability by persuading a user ofthe interface to follow a malicious linkA successful exploit could allow theattacker to perform arbitrary actions withthe privilege level of the affected user

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvp2434340(2a)

Vulnerabilities with a version of cJSONidentified by CVE-2019-11834 andCVE-2019-11835

For more information see the relatedCisco Security Advisory

CVE-2019-11834 CVE-2019-11835CSCvq1994940(2a)

HX ESXi image patches for MCEPSCand TAA vulnerabilities with VMwareESXi identified by CVE-2018-12207 andCVE-20190-11135

For more information see the relatedCisco Security Advisory

CVE-2018-12207 CVE-2019-11135CSCvr5439840(2a)

CP networking vulnerabilities affectingthe Linux kernel identified byCVE-2019-11477 CVE-2019-11478CVE-2019-11479

For more information see the relatedCisco Security Advisory

CVE-2019-11477 CVE-2019-11478CVE-2019-11479

CSCvq1954640(2a)

Qualification of Microsoft securitypatches for vulnerabilities withMicrosoftHyper-V hypervisor identified withCVE-2018-12207 CVE-2019-11135

For more information see the relatedCisco Security Advisory

CVE-2018-12207 CVE-2019-11135CSCvr5439940(2a)

Release Notes for Cisco HX Data Platform Release 4024

Security Fixes

DescriptionCVEDefect IDRelease

OpenSSH vulnerabilities identified byCVE-2019-6111

For more information see the relatedCisco Security Advisory

CVE-2016-10708 CVE-2018-15473CVE-2018-20685 CVE-2019-6109CVE-2019-6111

CSCvp7646340(2a)

Multiple vulnerabilities associated withOpenSSL and LibNSS

For more information see the relatedCisco Security Advisory

CVE-2016-2105 CVE-2016-2106CVE-2016-2107 CVE-2016-2109CVE-2016-2176 CVE-2016-2177CVE-2016-2178 CVE-2016-2179CVE-2016-2180 CVE-2016-2181CVE-2016-2182 CVE-2016-2183CVE-2016-6302 CVE-2016-6303CVE-2016-6304 CVE-2016-6305CVE-2016-6306 CVE-2016-6307CVE-2016-6308 CVE-2016-7055CVE-2016-8610 CVE-2017-3731CVE-2017-3732 CVE-2017-3735CVE-2017-3736 CVE-2017-3737CVE-2017-3738 CVE-2018-0495CVE-2018-0732 CVE-2018-0734CVE-2018-0735 CVE-2018-0737CVE-2018-0739 CVE-2018-12384CVE-2018-12404 CVE-2018-5407CVE-2019-1559

CSCvp6667940(2a)

libpng vulnerability identified byCVE-2016-10087

For more information see the relatedCisco Security Advisory

CVE-2016-10087CSCvp6655540(2a)

Vulnerabilities associated with libjpg andlibpng

For more information see the relatedCisco Security Advisory

CVE-2014-9092 CVE-2016-3616CVE-2017-15232 CVE-2018-11212CVE-2018-11213 CVE-2018-11214CVE-2018-1152 CVE-2018-13785

CSCvp3458640(2a)

Vulnerabilities associated with Glibidentified by CVE-2018-16428 andCVE-2018-16429

For more information see the relatedCisco Security Advisory

CVE-2018-16428 CVE-2018-16429CSCvp3120740(2a)

Multiple vulnerabilities associated withcurl expat python 27 python 35 36and 37 freetype and giflib

For more information see the relatedCisco Security Advisory

CVE-2019-15133 CVE-2019-15903CVE-2019-5010 CVE-2019-5481CVE-2019-5482 CVE-2019-9636CVE-2019-9740 CVE-2019-9947CVE-2019-9948

CSCvr3690340(2a)

Release Notes for Cisco HX Data Platform Release 4025

Security Fixes

DescriptionCVEDefect IDRelease

Multiple third party softwarevulnerabilities

For more information see the relatedCisco Security Advisory

CVE-2019-14379 CVE-2019-12384CVE-2019-14439

CSCvq9203240(2a)

Vulnerabilities associated with elfutils

For more information see the relatedCisco Security Advisory

CVE-2018-16062 CVE-2018-16402CVE-2018-16403 CVE-2018-18310CVE-2018-18520 CVE-2018-18521CVE-2019-7149 CVE-2019-7150CVE-2019-7665

CSCvq4325040(2a)

Vulnerabilities associated with vim

For more information see the relatedCisco Security Advisory

CVE-2017-5953 CVE-2019-12735CSCvq4323040(2a)

Vulnerabilities associated with sqlite3

For more information see the relatedCisco Security Advisory

CVE-2016-6153 CVE-2017-10989CVE-2017-13685 CVE-2017-2518CVE-2017-2519 CVE-2017-2520CVE-2018-20346 CVE-2018-20505CVE-2018-20506 CVE-2019-8457CVE-2019-9936 CVE-2019-9937

CSCvq4321340(2a)

Vulnerabilities associated with glib20

For more information see the relatedCisco Security Advisory

CVE-2018-20843CSCvq4320940(2a)

Vulnerabilities associated with expat

For more information see the relatedCisco Security Advisory

CVE-2018-20843CSCvq4320540(2a)

Vulnerabilities associated with bzip2

For more information see the relatedCisco Security Advisory

CVE-2016-3189 CVE-2019-12900CSCvq4319440(2a)

Mulitple Vulnerabilities associated wtihNodeJS

For more information see the relatedCisco Security Advisory

CVE-2018-12115 CVE-2018-0734CVE-2018-5407 CVE-2018-12120CVE-2018-12121 CVE-2018-12122CVE-2018-12123 CVE-2018-12116CVE-2019-5737 CVE-2019-5739

CSCvq1069440(2a)

Vulnerabilities associated with PalletsJinja strformat_map and identified byCVE-2019-10906 and CVE-2016-10745

For more information see the relatedCisco Security Advisory

CVE-2019-10906 CVE-2016-10745CSCvq1038840(2a)

Release Notes for Cisco HX Data Platform Release 4026

Security Fixes

DescriptionCVEDefect IDRelease

Multiple Security Vulnerabilities

For more information see the relatedCisco Security Advisory

CVE-2014-8501 CVE-2014-9939CVE-2015-9262 CVE-2016-10087CVE-2016-2226 CVE-2016-4487CVE-2016-4488 CVE-2016-4489CVE-2016-4490 CVE-2016-4491CVE-2016-4492 CVE-2016-4493CVE-2016-6131 CVE-2018-10963CVE-2018-13785 CVE-2018-17100CVE-2018-17101 CVE-2018-18557CVE-2018-18661 CVE-2018-7456CVE-2018-8905

CSCvq0917840(2a)

Vulnerabilities associated withlibseccomp

For more information see the relatedCisco Security Advisory

CVE-2019-9893CSCvq0756840(2a)

Vulnerabilities associated withgdk-pixbuf

For more information see the relatedCisco Security Advisory

CVE-2017-12447CSCvq0675540(2a)

Vulnerabilities associated withpython-crypto

For more information see the relatedCisco Security Advisory

CVE-2018-6594CSCvp9381740(2a)

Vulnerabilities associated with wget

For more information see the relatedCisco Security Advisory

CVE-2018-20483 CVE-2019-5953CSCvp8672140(2a)

Multiple vulnerabilities associated withPython Crypto

For more information see the relatedCisco Security Advisory

CVE-2018-6594CSCvp6674840(2a)

Vulnerabilities associated withsensible-utils

For more information see the relatedCisco Security Advisory

CVE-2017-17512CSCvp6673440(2a)

Multiple vulnerabilities associated withPython

For more information see the relatedCisco Security Advisory

CVE-2018-1000030CSCvp6668940(2a)

Release Notes for Cisco HX Data Platform Release 4027

Security Fixes

DescriptionCVEDefect IDRelease

Multiple patch vulnerabilities identifiedby CVE-2016-10713

For more information see the relatedCisco Security Advisory

CVE-2016-10713CVE-2018-1000156 CVE-2018-6951

CSCvp6667240(2a)

Vulnerabilities associatedwith Little CMSidentified by CVE-2016-10165

For more information see the relatedCisco Security Advisory

CVE-2016-10165 CVE-2018-16435CSCvp6666440(2a)

Vulnerabilities associated with TenableScan identified byCVE-2011-5325CVE-2019-11068CVE-2016-7076CVE-2017-1000368

For more information see the relatedCisco Security Advisory

CVE-2011-5325 CVE-2016-7076CVE-2017-1000368CVE-2019-11068

CSCvp6474640(2a)

Vulnerabilities associated with libxml2294 and earlier as used in XMLSec1223 and earlier and other products andidentified by CVE-2016-9318

For more information see the relatedCisco Security Advisory

CVE-2016-9318 CVE-2017-16932CVE-2017-18258 CVE-2018-14404CVE-2018-14567

CSCvp3479240(2a)

Vulnerabilities associated with LFTPRemote File Names Unauthorized AccessVulnerability and identified byCVE-2018-10916

For more information see the relatedCisco Security Advisory

CVE-2018-10916CSCvp2926640(2a)

Vulnerabilities associated withCryptographyDeprecationWarning signerand verifier have been deprecated

For more information see the relatedCisco Security Advisory

CVE-2018-7750CSCvo3409740(2a)

Multiple Vulnerabilities in python python27 35

For more information see the relatedCisco Security Advisory

CVE-2019-16056CSCvr3979340(2a)

Release Notes for Cisco HX Data Platform Release 4028

Security Fixes

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toexecute commands as the root user

The vulnerability is due to insufficientinput validation An attacker could exploitthis vulnerability by connecting to thecluster service manager and injectingcommands into the bound process Asuccessful exploit could allow the attackerto run commands on the affected host asthe root user

Cisco has released software updates thataddress this vulnerability There areworkarounds that address thisvulnerability

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvq2417640(1b)35(2g)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

The vulnerability is due to an unprotectedlistening interface An attacker couldexploit this vulnerability by connectingto the listening interface and injectingcommands to the bound process Anexploit could allow the attacker to runcommands on the affected host as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvj9560640(1b)35(2g)

Release Notes for Cisco HX Data Platform Release 4029

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerabilities associated with theJVM18U121memory leak found duringVC alarms (concurrent 40 calls usingREST API)

CVE-2017-10053 CVE-2017-10067CVE-2017-10074 CVE-2017-10078CVE-2017-10081 CVE-2017-10087CVE-2017-10089 CVE-2017-10090CVE-2017-10096 CVE-2017-10101CVE-2017-10102 CVE-2017-10107CVE-2017-10108 CVE-2017-10109CVE-2017-10110 CVE-2017-10111CVE-2017-10115 CVE-2017-10116CVE-2017-10118 CVE-2017-10135CVE-2017-10176 CVE-2017-10193CVE-2017-10198 CVE-2017-10243CVE-2017-10274 CVE-2017-10281CVE-2017-10285 CVE-2017-10295CVE-2017-10345 CVE-2017-10346CVE-2017-10347 CVE-2017-10348CVE-2017-10349 CVE-2017-10350CVE-2017-10355 CVE-2017-10356CVE-2017-10357 CVE-2017-10388CVE-2017-3509 CVE-2017-3511CVE-2017-3526 CVE-2017-3533CVE-2017-3539 CVE-2017-3544CVE-2018-2579 CVE-2018-2582CVE-2018-2588 CVE-2018-2599CVE-2018-2602 CVE-2018-2603CVE-2018-2618 CVE-2018-2629CVE-2018-2633 CVE-2018-2634CVE-2018-2637 CVE-2018-2641CVE-2018-2663 CVE-2018-2677CVE-2018-2678 CVE-2018-2783CVE-2018-2790 CVE-2018-2794CVE-2018-2795 CVE-2018-2796CVE-2018-2797 CVE-2018-2798CVE-2018-2799 CVE-2018-2800CVE-2018-2814 CVE-2018-2815CVE-2018-2952 CVE-2018-3136CVE-2018-3139 CVE-2018-3149CVE-2018-3150 CVE-2018-3169CVE-2018-3180 CVE-2018-3183CVE-2018-3214 CVE-2019-2422

CSCvo8899740(1b)

Tomcat and Nginx logs are not beingcollected in the support bundle generatedthrough HX Connect in Release 35

NACSCvm5803140(1b)

The vulnerabilities associated with thelibmspack software package versionincluded in Cisco HX Data Platform

CVE-2018-18584

CVE-2018-18585

CSCvn3511940(1a)

Release Notes for Cisco HX Data Platform Release 4030

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerabilities associated withFasterXML Jackson-Databind TimeValueField Denial of Service

CVE-2018-14719

CVE-2018-14720

CVE-2018-1000873

CVE-2018-14721

CVE-2018-19360

CVE-2018-19362

CVE-2018-19361

CVE-2018-14718

CSCvn8228240(1a)

The vulnerabilities associated with theOpenSSL Protocol software packageversion included in Cisco HX DataPlatform

CVE-2013-3587CSCvo0505440(1a)

The vulnerabilities associated withThird-Party Software Denial of Service

CVE-2018-16487

CVE-2018-19361

CSCvo2781840(1a)

The vulnerabilities associated withLinuxkernel for Ubuntu 1710

CVE-2018-1092

CVE-2018-7492

CVE-2018-8087

CVE-2018-1068

CVE-2018-8781

CSCvm5314935(2a)

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)25(1d)stcli services dns remove should remove the DNS serverinfo from the interface files

CSCvs70967

40(2b)40(1b)HX 40(1b) new deployment from Intersight productioncloud

User receives following error while enabling CSI(Kubernetes integration)

Failure occurred during volume_access_enable Errorwas Nonzero exit code 1

CSCvt10522

40(2b)40(2a)Receive the following alertevent in HX Connect HXController VM HOSTNAME one or more configuredDNS servers not responding

CSCvt13947

Release Notes for Cisco HX Data Platform Release 4031

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)40(2a)API UI CLI show few Drive slots Empty after upgradeto 402a

The drives are seen in the sysmtool lsscsi and stclicommand outputs

The cluster is healthy with no errors

CSCvt14914

40(2b)35(2d)After successfully changing the DNS server on the HXcontroller we still can see the original DNS entry thatwas added during the deployment

CSCvs69154

40(2b)35(2e)HyperFlex Connect and VCenter show APD alarms forHX and non HX datastores This implies there is an issuewith HyperFlex when there may not be

CSCvs30080

40(2b)40(2a)An issue where all the disks in the node get locked afterrebooting the node

We successfully unlocked using sed-clientsh -Ucommand but wanted to test with another reboot anddrives locked again

CSCvs74286

40(2b)40(2a)When running stcli license commands on HyperFlexthe following error is seen

rootSpringpathControllertmp stcli license show all

Show smart licensing failed

Smart Agent is not ready please wait a minute and tryagain

CSCvt13929

40(2b)40(2a)After upgrading to HX 40(2a) the first RoboEdge nodethat gets rebooted then ignores persistent drivesPIDHX-HD24TB10K4KN

Cluster will be in degraded state and have reducedcapacity

CSCvt20144

40(2b)35(2b)Zookeeper fails to start while Exhibitor is runninghowever echo srvr returns nothing

CSCvs21562

40(2b)40(2a)When performing an HyperFlex upgrade a validationwarning may occur due to the host not having EnterprisePlus or Enterprise hypervisor licensing

Upgrade Validation Warning

ESXi host esx1labtest should be configured withVMware Enterprise license for upgrade to continue

CSCvs91787

Release Notes for Cisco HX Data Platform Release 4032

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)35(2g)Cluster expansion fails at Config Installer stage whenthe root and admin password for the storage controller(SCVM) are different

CSCvs69317

40(2b)35(2g)Panic while upgrading ESXiCSCvt06983

40(2b)40(2a)A cluster node running may hang in the Linux kernelThis is classified as an oops and a deviation from theexpected behavior

CSCvs54285

40(2b)35(2g)Rebalance failing on 10+10 node HX 35(2g) stretchcluster

CSCvs69007

40(2b)35(2d)The cluster becomes inaccessible to the IOVisorCSCvr54687

40(2b)40(2a)Panic on the storage controllerCSCvt61297

40(2b)40(2a)The size of support bundle is very large when collectedwith storfs-support command

CSCvt63306

40(2b)40(2a)storfs crash associated withFileSystemUsageWarningEvent orFileSystemUsageAlertEvent due to high usage ofvarstv above 80

CSCvt72807

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

40(2a)35(2d)In normal scenarios HX upgrade gets pendingACK afterupdating service profile In this case UCS only upgradewas stalled because after updating service profile HXupgrade was not getting pending ACK and it was alwayswaiting state

CSCvo39912

40(2a)35(1a)When using motherBoardReplace-12 to clean ZK fromold stNodepNodes resulted in unmounted datastore andmaking the size of the Hyperflex Datastores 0 resultingin all VMs in the cluster going offline

CSCvq39471

40(2a)26(1e)Upgrading cluster and getting error

Failed upgrade validations Checking vCenterconfiguration Reason Upgrade validations failed DRSFault Insufficient resources to satisfy configured failover

CSCvm77294

Release Notes for Cisco HX Data Platform Release 4033

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)The cluster expand fails on a node with DR replicationconfigured

When a HX cluster which has DR replication configuredis expanded we see the installer UI pulling in thereplicationVLAN information instead of themanagementVLAN information Even if we change that informationto the correct mgmt VLAN id and name it it does notseem to work as the node is configured with the VLANof the replicationVLAN in ESXi This leads to the failureof the node add with the host unreachable error

CSCvo70650

40(2a)35(1a)Customer reported that server didnt complete thefirmware upgrade automatically one by one

The user manually needs to put the host into maintenancemode and they manually acknowledged the pendingrequests to finish the UCS firmware upgrade

Maintenance policy is set to default (User-ack) as perthe design

CSCvo91624

40(2a)35(1i)When the cluster is in Failed state and stcli noderemove is attempted the output shows successful thoughthe node failed to get removed from the cluster

CSCvo93017

40(2a)35(1a)HyperFlex cluster upgrade may fail during validationwith a DRS Validation failed error

CSCvp31021

40(2a)35(2b)HX cluster expansion will fail with the following errormessage

MAC address pool configuration failure150[ErrorDescription] bad address block range definitioncollision

CSCvp58318

40(2a)35(2b)Memory usage by carbon cacheCSCvq34873

40(2a)35(2g)Upgrading ESXi cluster fails with error Nodemaintenance mode failed

CSCvr03240

40(2a)35(2e)storfs process does not automatically start on Exit HXMaintenance Mode or other tasks which restart StorageController

CSCvr88978

40(2a)35(2a)Post install script fails with message

Failed to execute ipmitool on HX node

CSCvp10707

40(2a)40(1a)HyperFlex expansionworkflow doesnt pull VLANnamecorrectly

CSCvp46539

Release Notes for Cisco HX Data Platform Release 4034

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)During HX cluster deployment - Validate ClusterCreation phase - HX Installer might fail with the error

fromvarlogspringpathstDeploylog

CSCvq45087

40(2a)35(2b)HX Installer fails to login to the SOL for configuringESX hosts

CSCvq91380

40(2a)35(2d)If DRS is enabled with configuration parameters otherthen fully automated ie manual or partially automatedthan this will change to fully automated during clusterexpansion

CSCvr44222

40(2a)35(1a)One of the validation task name has a typo The taskname is Validating node for mixed of SED and non-SEDdisks

It should be Validating node for mix of SED andnon-SED disks

CSCvp82175

40(2a)30(1i)This combination of using smaller NVMe drives is notsupported in expand operation For example if the drivesare only 375GB and they cannot be added to an existingcluster with larger caching SSD

CSCvo12359

40(2a)35(1a)HyperFlex includes a version of OpenSSL that is affectedby the vulnerabilities identified by the followingCommon Vulnerability and Exposures

CVE-2018-0495

CSCvp66679

40(2a)30(1e)SDK invoke exception when taking a native snapshot ona VM with GPU pci passthrough attached

CSCvo00511

40(2a)30(1i)Node replacement script fails due to EAM errorCSCvo62867

40(2a)35(1a)While pairing a cluster (HX release gt= 35) and cluster(HX release lt 35) remote replication network test failson the cluster (HX release lt 35)

CSCvo79760

40(2a)30(1b)Fixed the issue in the latest support-workflow bundleThis should not be hit anymore

CSCvo87061

40(2a)35(2a)On a 3 node cluster the MBreplace script fails as thescript looks for HEALTHY cluster status

CSCvo87080

40(2a)35(2a)MbReplace script (tar) hangs when running on 352b or40

CSCvp12359

Release Notes for Cisco HX Data Platform Release 4035

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)HX replication cleanup failing with error NA 35(2a) -INFODR state is not clean

CSCvp63958

40(2a)40(1a)The check_stig_parameters API incorrectly shows thecompliance state of the cluster as not STIG compliant

CSCvp66277

40(2a)35(2c)The snapshot creation on CBT enabled VM fails witherror Failed in vmreparent vmkfstools clone1

CSCvq06952

40(2a)35(2a)HyperFlex GUI when setting up remote DR-replicationpartnerpeer should show remote replication partner TCPport reachability test results

This will provide quick upfront connectivity results thatfrequently takes customer a while to manually check andconfirm

If reachability tests fail an info window and HX alertshould trigger that list the destination peer IP and thespecific port(s) that could not be reache

CSCvo60587

40(2a)35(2b)If you have more than one IP pool configured for areplication network and local replication network testsare failing you might be running into this issue

CSCvr67130

40(2a)35(2a)Skip task option in MBreplace and Software redeployscripts The scripts should have the option to skip tasksas user inputs and not by modifying the json file

CSCvp05204

40(2a)40(1a)Error noticed on SCVM Console print_req_error IOerror dev fd0 sector 0 on HyperV

It doesnt come up on SSH session

CSCvq34357

40(2a)30(1d)The VMNetwork switch is already created as part of theinstall process

Ideally the ip address assignment should be done duringthe initial install process as well No post install stepsshould be required altrnatively it could be done as partof a post install script

CSCvk36222

40(2a)25(1c)storfs service impacted during HX upgrade from HX21(1) to 25(1c)

HX node which is not being upgraded storfs servicestopped causing outage

CSCvg53223

40(2a)35(1b)HX Stretch Cluster Witness VM reverts to DHCP onreboot

CSCvp37536

40(2a)35(2b)HyperFlex Stretch Cluster Witness does not getprogrammed with an NTP server

CSCvp46578

Release Notes for Cisco HX Data Platform Release 4036

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2b)HyperFlex cluster does not heal after maintenance onnodeserver

CSCvr18528

40(2a)40(2a)High percentage of packet loss on the eth1 data interface9K MTU

CSCvr97089

40(2a)35(2f)The HyperFlex cluster healing state is stuck at 87 andis unable to progress

CSCvr16760

40(2a)40(2a)In some circumstances two replications can occur forthe same VM

From the GUI the replication status for this VM willshow stuck in an In Progress state but there are nojobs stuck in replication layer Subsequent Replicationsshould succeed

CSCvq22898

40(2a)35(2a)After an unexpected power outage the hyperflex clusteris unable to come up correctly

All processes are running time is synced and all nodescan vmkping each other as required

CSCvp13990

40(2a)40(1a)When specifying IP address instead of node ID stclinode maintenanceMode commands fail

CSCvp33657

40(2a)40(1a)2 node edge cluster if experiencing Intersightarbitratorconnectivity issues should NOT allowHXMaintenancemode to initiate as this will bring the cluster down

CSCvr31573

40(2a)35(2c)Old files in varsupportZKTxnlog are not purged withthe daily zklog-cleanup cron job

CSCvr89066

40(2a)35(2a)When a node is in MaintenanceMode any disk removalor replacement will be reflected in UI only after the nodeis brought back frommaintenancemode This is becausestorfs is not running on the node in maintenance andwill not be able to detect disk activities until it is broughtout of Maintenance Mode

CSCvo86431

40(2a)35(2c)Cluster might go offline and stop serving IO requestsrootcvm~ stcli cluster storage-summary--detailGet cluster storage summary failedjavanetConnectException Connection refused19216814210010207

CSCvr01645

40(2a)35(2d)A stretch cluster deployment fails with a messageFormatting nodes Node in Use on the Hyperflex installerand crmZoneType value is seen as 1 in storfscfg oroptspringpathconfigstretchtunes

CSCvq80340

Release Notes for Cisco HX Data Platform Release 4037

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)30(1c)During upgrade we upload the upgrade packagedownloaded from Ciscocom Upgrade packages are offile type tgz hxconnect acceptsstorfs-packages-401a-33028tar file a different type ofcompression and allows upgrade to start and fail whenit cannot decompress the file for bootstrap and scvmclientupgrade on ESXi

CSCvq63888

40(2a)40(2a)While attempting to install hyperflex via intersight thefollowing error is seen

Failed in Task Add hosts to vCenter Cluster with ErrorTry adding hosts manually to vCenter and retry Failedto get addressFailed to host by namelookupHostnamecompanycom on 000053 read udp12700145430-gt12700153 read connection refused

CSCvr52098

40(2a)35(2a)CoreAPI call may time out for cluster management APIwhile doing cluster create validation

CSCvo92952

40(2a)35(2d)Validation fails with Mixed mode expansion checkdue to Enhanced vMotion Compatibility Incompatibilitywhen the ESXi version is different

CSCvq95460

40(2a)30(1i)This failure is seen in installer when custom workflowis used hypervisor configuration + deploy (clean diskpartition=no) at Installing software packages on storagecontroller VM

CSCvr66309

40(2a)35(2d)The HX Installer apparently replaced a comma bysemicolon on the VLAN ID range while exporting theconfiguration to a JSON file

CSCvq93831

40(2a)40(1a)When using unsupported browsers HX Connect userswill not be able to login These users see an errorindicating unauthorized user

CSCvp22693

Release Notes for Cisco HX Data Platform Release 4038

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)40(1a)A vulnerability in the web-based management interfaceof Cisco HyperFlex Software could allow anunauthenticated remote attacker to conduct a cross-siterequest forgery (CSRF) attack on an affected system

The vulnerability is due to insufficient CSRF protectionsfor the web UI on an affected device An attacker couldexploit this vulnerability by persuading a user of theinterface to follow a malicious link A successful exploitcould allow the attacker to perform arbitrary actions withthe privilege level of the affected user

Cisco has released software updates that address thisvulnerability There are no workarounds that address thisvulnerability

This advisory is available at the following link

httpstoolsciscocomsecuritycentercontentCiscoSecurityAdvisorycisco-sa-20190807-hypflex-csrf

CSCvp24343

40(2a)35(2d)Add message to HX Connect Progress flow to notAcknowledge Pending Activities and reboot the serverson UCSM

HX Connect is doing a controlled rolling server upgradein the background

CSCvq22844

40(2a)40(1b)Compute Node expand operation fails due to incorrectcluster name

CSCvr43786

40(2a)26(1e)RF-2 cluster shutdown during data resynchronizationafter a node failure (non storfs) followed by multipledisk read failures and a HardBlacklist of a disk

CSCvp41241

40(2a)40(1a)HyperFlex Edge nodes do not properly set the ESXihostname during deployment

CSCvo13143

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4039

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4040

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1b) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401b-33133p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4041

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4042

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1b)40(1a)While running the HyperFlex installer with WindowsServer Hyper-V cluster creation process fails with theerror Failure occurred during ClusterCreation process Unable to post thecontent to the downstream This symptomis encountered while deploying HyperFlex cluster withHyperFlex nodes configuredordered with Cisco VIC1457 MLOM (PID HX-MLOM-C25Q-04) andWindows Server Datacenter or Core with Hyper-Vversion 2016 or 2019

CSCvp64140

Release Notes for Cisco HX Data Platform Release 4043

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)35(2b)Adding Micron 5200 drive to a cluster fails to increasecluster capacity

CSCvo69067

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4044

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4045

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1a) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401a-33028p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4046

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4047

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1a)30(1d)Cluster instability when disks of different sector sizeplaced in HX node

CSCvk17250

40(1a)

35(2c)

35(1a)When logged in using a local HX user account insteadof a Virtual Center account an error message appearsintermittently indicating Virtual Centerunreachable or Resource informationcannot be updated when VC is reachable

CSCvo36198

40(1a)

35(2b)

30(1d)HXDP does not work with EMC RecoverPoint - needsto support VMware API (FSS-Readdir)

CSCvk38003

Release Notes for Cisco HX Data Platform Release 4048

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)

35(2a)

30(1i)

25(1c)

25(1d)

Zookeeper (Exhibitor) process on HyperFlex StorageController VMs may fail to respond to requests whenvarzookeeper has low or no free space

CSCvm90352

40(1a)26(1c)Use Asynchronous consolidation for HX SnapshotsCSCvn02151

40(1a)35(2b)HX Quiesced Snapshot for Backup VendorsCSCvo90713

40(1a)35(2a)The cluster creationcluster expansion workflow stopswith the following error message at the validation step

FIRMWARE-Check UCSC-SAS-M5HD

FIRMWARE-Check UCSC-SAS-M5HD Required00000029000000320000003500000050 Found00000058

Action Needed Update the Controller Firmware toRequired Version

CSCvn17787

35(2b)30(1d)Kernel migration fails when a local datastore is searchedfor in the ESXi host

CSCvn73127

35(2a)26(1b)A node shuts down when two disks are replaced (acaching disk and another capacity disk) where thecapacity disk is inserted first and then the caching diskis inserted

CSCvk46364

Hyper-V

40(1a)35(2a)Cluster expansion may fail with an error code 500 -operation timed out

CSCvn28721

35(2b)35(2a)During upgrade remove the VLAN on team created foruser vSwitch During fresh installation only one VLANtag is set to the vSwitch and team although multipleVLANs were entered

CSCvn54300

35(2b)35(2a)While upgrading a Hyper-V cluster on account of a rarerace condition between the stUpgradeService andZookeeper servers the upgrade orchestration throws anupgrade validation error and the upgrade process isaborted

CSCvn60486

Release Notes for Cisco HX Data Platform Release 4049

Resolved Caveats in Release 40(1a)

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Startrestart EAM service invCenter

When EAM service is not runningvCenter registration is failing

CSCvt55712

35(2a)Contact Cisco TAC if you believeyou are seeing similar conditions

HyperFlex Stretch Cluster saw poorapplication performance due to writelatency cluster remained unhealthy andrebalance was stuck

This defect is currently being used forinvestigative purposes

CSCvt45344

35(2g)Remove log files fromvarlogspringpath folder when thelog files are not getting rotatedStop the relevant service removelarge log files fromvarlogspringpath folder andrestart the service

varstv folder in Controller VM maybecome full

CSCvt36374

40(2b)Restarting the stMgr will initializeit again

In a Cluster configured with DR networkStMgr may not get initialized and stuckin a deadlock while enabling the IPTablerules (can be verified from the stMgrlog)

CSCvt89709

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Click recover for the selectedVM again under the standalonemode This will move the VM torecovered state

When the user recovers a VM protectedin a group it along with other VMs in thegroup move to recovering state instandalonemode The selectedVM shouldmove to recovered state

CSCvs75553

35(2e)Clean up the fail over cluster andthen recreate the fail over clusterNo need to touch the HX storagecluster

Hyper-V When replicated DC was usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2d)40(1a)

Keep cluster utilization as low aspossible To avoid read-only statekeep below 92

HyperFlex clusters storage utilization isabove 92 usable capacity

CSCvr20922

Release Notes for Cisco HX Data Platform Release 4050

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1b)NAAll the disks in the node got locked afterrebooting the node

We successfully unlocked usingsed-clientsh -U command but wantedto test with another reboot and driveslocked again

CSCvs74286

35(2g)To recover the VM copy over thedata disks and attach them to thenew VM

After enable encryption HX connectencryption status showing Caution butall of the disks enable encryptionsuccessful

-Self Encrypting Drives Service isrunning on all of the nodes

-There is no error message of encryption

- USB0 interface is up

-All SED disks showing supported 1enabled 1 locked 0

CSCvs93245

40(1b)Enable kernelpanic_on_oops as apersistent configuration This willcause the node to panic and rebootimmediately

A cluster node running HX release40(1b) may hang in the Linux kernelThis is classified as an oops and adeviation from the expected behavior

CSCvs54285

Release Notes for Cisco HX Data Platform Release 4051

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Enabling K8iscsi stack in Hx clusterCSCvs41324

Release Notes for Cisco HX Data Platform Release 4052

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Steps to enable K8 on HX cluster

Perform following steps on ALLthe controller VMs

1 Updateetcinitscvmclientconf toenable tune iscsiEnable totrue

sed -iesiscsiEnable=falseiscsiEnable=trueetcinitscvmclientconf

2 Run following initctlcommand to reloadconfiguration

initctl reload-configuration

3 Restart scvmclient process

stop scvmclient startscvmclient

4 Run following initctlcommand to mount SYSTEMDS

initctl emit --no-waitsystem-datastore-created

5 Verify that iscsiEnable tuneis set to true using thefollowing command

ps -eaf | grep scvmclient |grep -v grep

root 6241 1 1 Dec06 014007optspringpathstorfs-corescvmclient-T logEnabled=false -TlogSyslogEnabled=true -TlogEchoToScreen=false -TstatLoggingToFile=false -TstatLoggingToSyslog=true-TlogDir=varlogspringpath-TnfsBackendServerList=1010748100-T iscsiEnable=true -TiscsiUseAsync=true -T

Release Notes for Cisco HX Data Platform Release 4053

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

iscsiConfigFilePath=nfsSYSTEMistgtconfps -eaf | grep scvmclient |grep -v grep

35(2e)Following procedure can be usedto check NFS Queue Depth andincrease if needed

rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth256 lt- Low value[rootHXESXI1] vsish -e setvmkModulesnfsclientmnt[DATASTORE]propertiesmaxQDepth 1024[rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth1024 lt- Optimalvalue

This requires an ESXi host rebootto take effect

Please place the node in HyperflexMaintenanceMode and gracefullyreboot the node for the changes tobe applied

HyperFlex Datastore NFS Queue Depthshows as 256 which can lead toperformance (including latency) issues

CSCvr83056

40(1b)This is cosmetic The error will goaway once the cluster returns tohealthy status

When a single node is offline in clusterstcli cluster storage-summary shows twonodes unavailable rootSCVM~ stclicluster storage-summary

CSCvs47735

35(2g)Modify both the root and adminpassword for the SCVMs to be thesame

stcli security password set -uadmin

Cluster expansion fails at Config Installerstage when the root and admin passwordfor the storage controller (SCVM) aredifferent

CSCvs69317

35(2d)NAAfter successfully changing the DNSserver on the HX controller we still cansee the original DNS entry that was addedduring the deployment

CSCvs69154

Release Notes for Cisco HX Data Platform Release 4054

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Customers are strongly encouragedto work with TAC in order toidentify and remediate this issue

1 Change getcluster from 2m to8m inoptspringpathstorfs-mgmtstMgr-10confapplicationconfon all nodes

2 Restart stMgr on all nodes

You may see this error message after afailed upgrade or other task such asattempting to enter a node into HXmaintenance mode

CSCvs53555

40(2a)Run the following command oneach storage controller VM as root

grep -i monitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp sed -iesmonitor_dns_serverstruemonitor_dns_serversfalseoptspringpathhx-diag-toolswatchdog_configjsonampamp grep -imonitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp restart watchdog

Receive the following alertevent in HXConnect HX Controller VMHOSTNAME one or more configuredDNS servers not responding

CSCvt13947

35(2b)Delete any empty (Size 0) log filesunder varlogzookeeperversion-2ORvarlogzookeeperstandaloneversion-2

Zookeeper fails to start while Exhibitor isrunning however echo srvr returnsnothing

CSCvs21562

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

40(4e)1 De-commission the server

2 Power drain the server -REMOVE BOTH POWERCORDS ON THE BACK OFTHE SERVER FOR 10SECONDS THENREINSERTPOWERCORDS

3 Re-commission the server

M2 boot disk is missing from serverinventory after upgrade to server firmware40(4e) As a result server fails to boot toOS installed in the M2 disk The issuepersists after re-acknowledgement as wellas de-commission andre-acknowledgement of the server

CSCvs02466

Release Notes for Cisco HX Data Platform Release 4055

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Re-start the flusher manuallyMany disks taken offline by a fault ormanually resulting in ENOSPC oncluster

CSCvq32721

35(2e)Clean up fail over cluster and thenrecreate the fail over cluster Thereis no need to touch the HX storagecluster

[Hyper-V]When replicated DCwas usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2b)Manually install upgradescvmclient VIB andmake sure thatit matches with HXDP version

Upgrade from release HX 35(2a) to35(2x) 4X does not upgrade scvmclientRecovery point may not work

CSCvq54992

Management

30(1b)To recover the VM copy over thedata disks and attach them to thenew VM

[HyperV]VMshows up onmultiple nodesCSCvj22992

35(1a)1 Do not change MTU at ESXlevel on running cluster

2 Reset back to the original value

ChangingMTU from 9000 to 1500 on oneESX node causes storfs process to restarton all nodes

CSCvm99150

Release Notes for Cisco HX Data Platform Release 4056

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)After the installation (for newdeployments) or after an upgradeto 401a (for existingdeployments) perform thefollowing steps on all ControllerVMs

1 Edit the following tune files (onall Controller VMs)

optspringpathconfiglfftunes

optspringpathconfigvsi_16tbtunes

2 SetcleanerEnableSegSummaryCleaning

to false

3 After editing the above tunefiles

a SSH in to all the ControllerVMs

b Run storfstool -- -Z

c Run the below command andcheck the values Tune valueshould be true

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=true

d Type below commands andapplied the tunes changesdynamically

echo false gttmpstprocfssystemtunecleanerEnableSegSummaryCleaning

e Verify that tune values arechanged Run below commandValue should be false

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=false

f umount tmpstprocfs

A cluster with 8TB or 12TB disk drivesmay experience IO stalls for severalminutes after another node fails in thecluster

CSCvp23718

Release Notes for Cisco HX Data Platform Release 4057

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)1 Update the conf file manuallyto change to the link local addressto

2 Restart scvmclient on thecontroller and ESX

HX 35(2b) FlexVol to HX 40 CSIupgrade does not work FlexVol continuesto work

CSCvp26319

35(2a)Disable rebalance until node healsthen reenable rebalance

7-node cluster healing very slowlybecause rebalance change AMS reqs areblocking healing process

CSCvp62512

35(2a)NAWith moderate cluster storage capacityusage under site failover cluster clustermay experience an all paths (APD) downcondition

CSCvq49412

40(1a)NAThis product includes Third-partySoftware that is affected by thevulnerabilities identified by the followingCommon Vulnerability and Exposures(CVE) IDs

CVE-2016-0762 CVE-2016-6797CVE-2016-6816 CVE-2016-8735CVE-2017-5647 CVE-2017-12615CVE-2017-12616 CVE-2017-12617CVE-2017-7674 CVE-2018-1304CVE-2018-8014 CVE-2018-1336CVE-2018-8034 CVE-2018-11784CVE-2019-0232

CSCvp36364

35(2d)NAstcli cluster info command needs toprovide UCSM VIP address Currently itshows ucsm-hostcom

CSCvq11456

35(2a)No workarounds currently Thestale entries do not affect the dayto day operations of HyperFlex

HyperFlex upgrade validation failedbecause cluster has extraneous stNodes ininternal database

CSCvq32530

35(2a)NAWhen not in maintenancemode all drivesappear in HX Connect and get updated asexpected

When in maintenance mode extra drivesare shown in HX Connect with no slotnumbers

CSCvq42378

35(2b)Contact TAC to cleanup duplicatenode

Nodes duplicated in HX Connect aftercluster recreate

CSCvq61825

Release Notes for Cisco HX Data Platform Release 4058

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)NACurrently HyperFlex installs on Hyper-Vdo not contain the stcli security whitelistcommandset

CSCvq66245

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

Release Notes for Cisco HX Data Platform Release 4059

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)[Hyper-V] Converged node expansionfails for incoming node with VIC 14xx

CSCvq39523

Release Notes for Cisco HX Data Platform Release 4060

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

1 Set Jumbo frame inWindowsHypervisor for incomingnode

Get-NetAdapter storage-data-a|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

Get-NetAdapter storage-data-b|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

2 Verify all interfaces are setcorrectly to support Jumbo frames

Get-NetAdapter storage |Get-NetAdapterAdvancedProperty| RegistryKeyword -matchjumbo | ft -auto

You should receive the followingmessage

Name DisplayName DisplayValue

RegistryKeyword RegistryValue

---- ----------- ------------

--------------- -------------

vswitch-hx-storage-data Jumbo

Packet 9014 Bytes

JumboPacket 9014

storage-data-b Jumbo Packet

Bytes 9014 JumboPacket

9014

storage-data-a Jumbo Packet

Bytes 9014 JumboPacket

9014

3 Reboot only the incoming nodeor the node being expanded

4 Retry Cluster Expansion in

Release Notes for Cisco HX Data Platform Release 4061

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

DescriptionCVEDefect IDRelease

The vulnerability is due to insufficientauthentication for the statistics collectionservice An attacker could exploit thisvulnerability by sending properlyformatted data values to the statisticscollection service of an affected deviceA successful exploit could allow theattacker to cause the web interfacestatistics view to present invalid data tousers

NACSCvj9558440(2a)

The vulnerability is due to insufficientCSRF protections for the web UI on anaffected device An attacker could exploitthis vulnerability by persuading a user ofthe interface to follow a malicious linkA successful exploit could allow theattacker to perform arbitrary actions withthe privilege level of the affected user

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvp2434340(2a)

Vulnerabilities with a version of cJSONidentified by CVE-2019-11834 andCVE-2019-11835

For more information see the relatedCisco Security Advisory

CVE-2019-11834 CVE-2019-11835CSCvq1994940(2a)

HX ESXi image patches for MCEPSCand TAA vulnerabilities with VMwareESXi identified by CVE-2018-12207 andCVE-20190-11135

For more information see the relatedCisco Security Advisory

CVE-2018-12207 CVE-2019-11135CSCvr5439840(2a)

CP networking vulnerabilities affectingthe Linux kernel identified byCVE-2019-11477 CVE-2019-11478CVE-2019-11479

For more information see the relatedCisco Security Advisory

CVE-2019-11477 CVE-2019-11478CVE-2019-11479

CSCvq1954640(2a)

Qualification of Microsoft securitypatches for vulnerabilities withMicrosoftHyper-V hypervisor identified withCVE-2018-12207 CVE-2019-11135

For more information see the relatedCisco Security Advisory

CVE-2018-12207 CVE-2019-11135CSCvr5439940(2a)

Release Notes for Cisco HX Data Platform Release 4024

Security Fixes

DescriptionCVEDefect IDRelease

OpenSSH vulnerabilities identified byCVE-2019-6111

For more information see the relatedCisco Security Advisory

CVE-2016-10708 CVE-2018-15473CVE-2018-20685 CVE-2019-6109CVE-2019-6111

CSCvp7646340(2a)

Multiple vulnerabilities associated withOpenSSL and LibNSS

For more information see the relatedCisco Security Advisory

CVE-2016-2105 CVE-2016-2106CVE-2016-2107 CVE-2016-2109CVE-2016-2176 CVE-2016-2177CVE-2016-2178 CVE-2016-2179CVE-2016-2180 CVE-2016-2181CVE-2016-2182 CVE-2016-2183CVE-2016-6302 CVE-2016-6303CVE-2016-6304 CVE-2016-6305CVE-2016-6306 CVE-2016-6307CVE-2016-6308 CVE-2016-7055CVE-2016-8610 CVE-2017-3731CVE-2017-3732 CVE-2017-3735CVE-2017-3736 CVE-2017-3737CVE-2017-3738 CVE-2018-0495CVE-2018-0732 CVE-2018-0734CVE-2018-0735 CVE-2018-0737CVE-2018-0739 CVE-2018-12384CVE-2018-12404 CVE-2018-5407CVE-2019-1559

CSCvp6667940(2a)

libpng vulnerability identified byCVE-2016-10087

For more information see the relatedCisco Security Advisory

CVE-2016-10087CSCvp6655540(2a)

Vulnerabilities associated with libjpg andlibpng

For more information see the relatedCisco Security Advisory

CVE-2014-9092 CVE-2016-3616CVE-2017-15232 CVE-2018-11212CVE-2018-11213 CVE-2018-11214CVE-2018-1152 CVE-2018-13785

CSCvp3458640(2a)

Vulnerabilities associated with Glibidentified by CVE-2018-16428 andCVE-2018-16429

For more information see the relatedCisco Security Advisory

CVE-2018-16428 CVE-2018-16429CSCvp3120740(2a)

Multiple vulnerabilities associated withcurl expat python 27 python 35 36and 37 freetype and giflib

For more information see the relatedCisco Security Advisory

CVE-2019-15133 CVE-2019-15903CVE-2019-5010 CVE-2019-5481CVE-2019-5482 CVE-2019-9636CVE-2019-9740 CVE-2019-9947CVE-2019-9948

CSCvr3690340(2a)

Release Notes for Cisco HX Data Platform Release 4025

Security Fixes

DescriptionCVEDefect IDRelease

Multiple third party softwarevulnerabilities

For more information see the relatedCisco Security Advisory

CVE-2019-14379 CVE-2019-12384CVE-2019-14439

CSCvq9203240(2a)

Vulnerabilities associated with elfutils

For more information see the relatedCisco Security Advisory

CVE-2018-16062 CVE-2018-16402CVE-2018-16403 CVE-2018-18310CVE-2018-18520 CVE-2018-18521CVE-2019-7149 CVE-2019-7150CVE-2019-7665

CSCvq4325040(2a)

Vulnerabilities associated with vim

For more information see the relatedCisco Security Advisory

CVE-2017-5953 CVE-2019-12735CSCvq4323040(2a)

Vulnerabilities associated with sqlite3

For more information see the relatedCisco Security Advisory

CVE-2016-6153 CVE-2017-10989CVE-2017-13685 CVE-2017-2518CVE-2017-2519 CVE-2017-2520CVE-2018-20346 CVE-2018-20505CVE-2018-20506 CVE-2019-8457CVE-2019-9936 CVE-2019-9937

CSCvq4321340(2a)

Vulnerabilities associated with glib20

For more information see the relatedCisco Security Advisory

CVE-2018-20843CSCvq4320940(2a)

Vulnerabilities associated with expat

For more information see the relatedCisco Security Advisory

CVE-2018-20843CSCvq4320540(2a)

Vulnerabilities associated with bzip2

For more information see the relatedCisco Security Advisory

CVE-2016-3189 CVE-2019-12900CSCvq4319440(2a)

Mulitple Vulnerabilities associated wtihNodeJS

For more information see the relatedCisco Security Advisory

CVE-2018-12115 CVE-2018-0734CVE-2018-5407 CVE-2018-12120CVE-2018-12121 CVE-2018-12122CVE-2018-12123 CVE-2018-12116CVE-2019-5737 CVE-2019-5739

CSCvq1069440(2a)

Vulnerabilities associated with PalletsJinja strformat_map and identified byCVE-2019-10906 and CVE-2016-10745

For more information see the relatedCisco Security Advisory

CVE-2019-10906 CVE-2016-10745CSCvq1038840(2a)

Release Notes for Cisco HX Data Platform Release 4026

Security Fixes

DescriptionCVEDefect IDRelease

Multiple Security Vulnerabilities

For more information see the relatedCisco Security Advisory

CVE-2014-8501 CVE-2014-9939CVE-2015-9262 CVE-2016-10087CVE-2016-2226 CVE-2016-4487CVE-2016-4488 CVE-2016-4489CVE-2016-4490 CVE-2016-4491CVE-2016-4492 CVE-2016-4493CVE-2016-6131 CVE-2018-10963CVE-2018-13785 CVE-2018-17100CVE-2018-17101 CVE-2018-18557CVE-2018-18661 CVE-2018-7456CVE-2018-8905

CSCvq0917840(2a)

Vulnerabilities associated withlibseccomp

For more information see the relatedCisco Security Advisory

CVE-2019-9893CSCvq0756840(2a)

Vulnerabilities associated withgdk-pixbuf

For more information see the relatedCisco Security Advisory

CVE-2017-12447CSCvq0675540(2a)

Vulnerabilities associated withpython-crypto

For more information see the relatedCisco Security Advisory

CVE-2018-6594CSCvp9381740(2a)

Vulnerabilities associated with wget

For more information see the relatedCisco Security Advisory

CVE-2018-20483 CVE-2019-5953CSCvp8672140(2a)

Multiple vulnerabilities associated withPython Crypto

For more information see the relatedCisco Security Advisory

CVE-2018-6594CSCvp6674840(2a)

Vulnerabilities associated withsensible-utils

For more information see the relatedCisco Security Advisory

CVE-2017-17512CSCvp6673440(2a)

Multiple vulnerabilities associated withPython

For more information see the relatedCisco Security Advisory

CVE-2018-1000030CSCvp6668940(2a)

Release Notes for Cisco HX Data Platform Release 4027

Security Fixes

DescriptionCVEDefect IDRelease

Multiple patch vulnerabilities identifiedby CVE-2016-10713

For more information see the relatedCisco Security Advisory

CVE-2016-10713CVE-2018-1000156 CVE-2018-6951

CSCvp6667240(2a)

Vulnerabilities associatedwith Little CMSidentified by CVE-2016-10165

For more information see the relatedCisco Security Advisory

CVE-2016-10165 CVE-2018-16435CSCvp6666440(2a)

Vulnerabilities associated with TenableScan identified byCVE-2011-5325CVE-2019-11068CVE-2016-7076CVE-2017-1000368

For more information see the relatedCisco Security Advisory

CVE-2011-5325 CVE-2016-7076CVE-2017-1000368CVE-2019-11068

CSCvp6474640(2a)

Vulnerabilities associated with libxml2294 and earlier as used in XMLSec1223 and earlier and other products andidentified by CVE-2016-9318

For more information see the relatedCisco Security Advisory

CVE-2016-9318 CVE-2017-16932CVE-2017-18258 CVE-2018-14404CVE-2018-14567

CSCvp3479240(2a)

Vulnerabilities associated with LFTPRemote File Names Unauthorized AccessVulnerability and identified byCVE-2018-10916

For more information see the relatedCisco Security Advisory

CVE-2018-10916CSCvp2926640(2a)

Vulnerabilities associated withCryptographyDeprecationWarning signerand verifier have been deprecated

For more information see the relatedCisco Security Advisory

CVE-2018-7750CSCvo3409740(2a)

Multiple Vulnerabilities in python python27 35

For more information see the relatedCisco Security Advisory

CVE-2019-16056CSCvr3979340(2a)

Release Notes for Cisco HX Data Platform Release 4028

Security Fixes

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toexecute commands as the root user

The vulnerability is due to insufficientinput validation An attacker could exploitthis vulnerability by connecting to thecluster service manager and injectingcommands into the bound process Asuccessful exploit could allow the attackerto run commands on the affected host asthe root user

Cisco has released software updates thataddress this vulnerability There areworkarounds that address thisvulnerability

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvq2417640(1b)35(2g)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

The vulnerability is due to an unprotectedlistening interface An attacker couldexploit this vulnerability by connectingto the listening interface and injectingcommands to the bound process Anexploit could allow the attacker to runcommands on the affected host as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvj9560640(1b)35(2g)

Release Notes for Cisco HX Data Platform Release 4029

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerabilities associated with theJVM18U121memory leak found duringVC alarms (concurrent 40 calls usingREST API)

CVE-2017-10053 CVE-2017-10067CVE-2017-10074 CVE-2017-10078CVE-2017-10081 CVE-2017-10087CVE-2017-10089 CVE-2017-10090CVE-2017-10096 CVE-2017-10101CVE-2017-10102 CVE-2017-10107CVE-2017-10108 CVE-2017-10109CVE-2017-10110 CVE-2017-10111CVE-2017-10115 CVE-2017-10116CVE-2017-10118 CVE-2017-10135CVE-2017-10176 CVE-2017-10193CVE-2017-10198 CVE-2017-10243CVE-2017-10274 CVE-2017-10281CVE-2017-10285 CVE-2017-10295CVE-2017-10345 CVE-2017-10346CVE-2017-10347 CVE-2017-10348CVE-2017-10349 CVE-2017-10350CVE-2017-10355 CVE-2017-10356CVE-2017-10357 CVE-2017-10388CVE-2017-3509 CVE-2017-3511CVE-2017-3526 CVE-2017-3533CVE-2017-3539 CVE-2017-3544CVE-2018-2579 CVE-2018-2582CVE-2018-2588 CVE-2018-2599CVE-2018-2602 CVE-2018-2603CVE-2018-2618 CVE-2018-2629CVE-2018-2633 CVE-2018-2634CVE-2018-2637 CVE-2018-2641CVE-2018-2663 CVE-2018-2677CVE-2018-2678 CVE-2018-2783CVE-2018-2790 CVE-2018-2794CVE-2018-2795 CVE-2018-2796CVE-2018-2797 CVE-2018-2798CVE-2018-2799 CVE-2018-2800CVE-2018-2814 CVE-2018-2815CVE-2018-2952 CVE-2018-3136CVE-2018-3139 CVE-2018-3149CVE-2018-3150 CVE-2018-3169CVE-2018-3180 CVE-2018-3183CVE-2018-3214 CVE-2019-2422

CSCvo8899740(1b)

Tomcat and Nginx logs are not beingcollected in the support bundle generatedthrough HX Connect in Release 35

NACSCvm5803140(1b)

The vulnerabilities associated with thelibmspack software package versionincluded in Cisco HX Data Platform

CVE-2018-18584

CVE-2018-18585

CSCvn3511940(1a)

Release Notes for Cisco HX Data Platform Release 4030

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerabilities associated withFasterXML Jackson-Databind TimeValueField Denial of Service

CVE-2018-14719

CVE-2018-14720

CVE-2018-1000873

CVE-2018-14721

CVE-2018-19360

CVE-2018-19362

CVE-2018-19361

CVE-2018-14718

CSCvn8228240(1a)

The vulnerabilities associated with theOpenSSL Protocol software packageversion included in Cisco HX DataPlatform

CVE-2013-3587CSCvo0505440(1a)

The vulnerabilities associated withThird-Party Software Denial of Service

CVE-2018-16487

CVE-2018-19361

CSCvo2781840(1a)

The vulnerabilities associated withLinuxkernel for Ubuntu 1710

CVE-2018-1092

CVE-2018-7492

CVE-2018-8087

CVE-2018-1068

CVE-2018-8781

CSCvm5314935(2a)

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)25(1d)stcli services dns remove should remove the DNS serverinfo from the interface files

CSCvs70967

40(2b)40(1b)HX 40(1b) new deployment from Intersight productioncloud

User receives following error while enabling CSI(Kubernetes integration)

Failure occurred during volume_access_enable Errorwas Nonzero exit code 1

CSCvt10522

40(2b)40(2a)Receive the following alertevent in HX Connect HXController VM HOSTNAME one or more configuredDNS servers not responding

CSCvt13947

Release Notes for Cisco HX Data Platform Release 4031

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)40(2a)API UI CLI show few Drive slots Empty after upgradeto 402a

The drives are seen in the sysmtool lsscsi and stclicommand outputs

The cluster is healthy with no errors

CSCvt14914

40(2b)35(2d)After successfully changing the DNS server on the HXcontroller we still can see the original DNS entry thatwas added during the deployment

CSCvs69154

40(2b)35(2e)HyperFlex Connect and VCenter show APD alarms forHX and non HX datastores This implies there is an issuewith HyperFlex when there may not be

CSCvs30080

40(2b)40(2a)An issue where all the disks in the node get locked afterrebooting the node

We successfully unlocked using sed-clientsh -Ucommand but wanted to test with another reboot anddrives locked again

CSCvs74286

40(2b)40(2a)When running stcli license commands on HyperFlexthe following error is seen

rootSpringpathControllertmp stcli license show all

Show smart licensing failed

Smart Agent is not ready please wait a minute and tryagain

CSCvt13929

40(2b)40(2a)After upgrading to HX 40(2a) the first RoboEdge nodethat gets rebooted then ignores persistent drivesPIDHX-HD24TB10K4KN

Cluster will be in degraded state and have reducedcapacity

CSCvt20144

40(2b)35(2b)Zookeeper fails to start while Exhibitor is runninghowever echo srvr returns nothing

CSCvs21562

40(2b)40(2a)When performing an HyperFlex upgrade a validationwarning may occur due to the host not having EnterprisePlus or Enterprise hypervisor licensing

Upgrade Validation Warning

ESXi host esx1labtest should be configured withVMware Enterprise license for upgrade to continue

CSCvs91787

Release Notes for Cisco HX Data Platform Release 4032

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)35(2g)Cluster expansion fails at Config Installer stage whenthe root and admin password for the storage controller(SCVM) are different

CSCvs69317

40(2b)35(2g)Panic while upgrading ESXiCSCvt06983

40(2b)40(2a)A cluster node running may hang in the Linux kernelThis is classified as an oops and a deviation from theexpected behavior

CSCvs54285

40(2b)35(2g)Rebalance failing on 10+10 node HX 35(2g) stretchcluster

CSCvs69007

40(2b)35(2d)The cluster becomes inaccessible to the IOVisorCSCvr54687

40(2b)40(2a)Panic on the storage controllerCSCvt61297

40(2b)40(2a)The size of support bundle is very large when collectedwith storfs-support command

CSCvt63306

40(2b)40(2a)storfs crash associated withFileSystemUsageWarningEvent orFileSystemUsageAlertEvent due to high usage ofvarstv above 80

CSCvt72807

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

40(2a)35(2d)In normal scenarios HX upgrade gets pendingACK afterupdating service profile In this case UCS only upgradewas stalled because after updating service profile HXupgrade was not getting pending ACK and it was alwayswaiting state

CSCvo39912

40(2a)35(1a)When using motherBoardReplace-12 to clean ZK fromold stNodepNodes resulted in unmounted datastore andmaking the size of the Hyperflex Datastores 0 resultingin all VMs in the cluster going offline

CSCvq39471

40(2a)26(1e)Upgrading cluster and getting error

Failed upgrade validations Checking vCenterconfiguration Reason Upgrade validations failed DRSFault Insufficient resources to satisfy configured failover

CSCvm77294

Release Notes for Cisco HX Data Platform Release 4033

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)The cluster expand fails on a node with DR replicationconfigured

When a HX cluster which has DR replication configuredis expanded we see the installer UI pulling in thereplicationVLAN information instead of themanagementVLAN information Even if we change that informationto the correct mgmt VLAN id and name it it does notseem to work as the node is configured with the VLANof the replicationVLAN in ESXi This leads to the failureof the node add with the host unreachable error

CSCvo70650

40(2a)35(1a)Customer reported that server didnt complete thefirmware upgrade automatically one by one

The user manually needs to put the host into maintenancemode and they manually acknowledged the pendingrequests to finish the UCS firmware upgrade

Maintenance policy is set to default (User-ack) as perthe design

CSCvo91624

40(2a)35(1i)When the cluster is in Failed state and stcli noderemove is attempted the output shows successful thoughthe node failed to get removed from the cluster

CSCvo93017

40(2a)35(1a)HyperFlex cluster upgrade may fail during validationwith a DRS Validation failed error

CSCvp31021

40(2a)35(2b)HX cluster expansion will fail with the following errormessage

MAC address pool configuration failure150[ErrorDescription] bad address block range definitioncollision

CSCvp58318

40(2a)35(2b)Memory usage by carbon cacheCSCvq34873

40(2a)35(2g)Upgrading ESXi cluster fails with error Nodemaintenance mode failed

CSCvr03240

40(2a)35(2e)storfs process does not automatically start on Exit HXMaintenance Mode or other tasks which restart StorageController

CSCvr88978

40(2a)35(2a)Post install script fails with message

Failed to execute ipmitool on HX node

CSCvp10707

40(2a)40(1a)HyperFlex expansionworkflow doesnt pull VLANnamecorrectly

CSCvp46539

Release Notes for Cisco HX Data Platform Release 4034

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)During HX cluster deployment - Validate ClusterCreation phase - HX Installer might fail with the error

fromvarlogspringpathstDeploylog

CSCvq45087

40(2a)35(2b)HX Installer fails to login to the SOL for configuringESX hosts

CSCvq91380

40(2a)35(2d)If DRS is enabled with configuration parameters otherthen fully automated ie manual or partially automatedthan this will change to fully automated during clusterexpansion

CSCvr44222

40(2a)35(1a)One of the validation task name has a typo The taskname is Validating node for mixed of SED and non-SEDdisks

It should be Validating node for mix of SED andnon-SED disks

CSCvp82175

40(2a)30(1i)This combination of using smaller NVMe drives is notsupported in expand operation For example if the drivesare only 375GB and they cannot be added to an existingcluster with larger caching SSD

CSCvo12359

40(2a)35(1a)HyperFlex includes a version of OpenSSL that is affectedby the vulnerabilities identified by the followingCommon Vulnerability and Exposures

CVE-2018-0495

CSCvp66679

40(2a)30(1e)SDK invoke exception when taking a native snapshot ona VM with GPU pci passthrough attached

CSCvo00511

40(2a)30(1i)Node replacement script fails due to EAM errorCSCvo62867

40(2a)35(1a)While pairing a cluster (HX release gt= 35) and cluster(HX release lt 35) remote replication network test failson the cluster (HX release lt 35)

CSCvo79760

40(2a)30(1b)Fixed the issue in the latest support-workflow bundleThis should not be hit anymore

CSCvo87061

40(2a)35(2a)On a 3 node cluster the MBreplace script fails as thescript looks for HEALTHY cluster status

CSCvo87080

40(2a)35(2a)MbReplace script (tar) hangs when running on 352b or40

CSCvp12359

Release Notes for Cisco HX Data Platform Release 4035

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)HX replication cleanup failing with error NA 35(2a) -INFODR state is not clean

CSCvp63958

40(2a)40(1a)The check_stig_parameters API incorrectly shows thecompliance state of the cluster as not STIG compliant

CSCvp66277

40(2a)35(2c)The snapshot creation on CBT enabled VM fails witherror Failed in vmreparent vmkfstools clone1

CSCvq06952

40(2a)35(2a)HyperFlex GUI when setting up remote DR-replicationpartnerpeer should show remote replication partner TCPport reachability test results

This will provide quick upfront connectivity results thatfrequently takes customer a while to manually check andconfirm

If reachability tests fail an info window and HX alertshould trigger that list the destination peer IP and thespecific port(s) that could not be reache

CSCvo60587

40(2a)35(2b)If you have more than one IP pool configured for areplication network and local replication network testsare failing you might be running into this issue

CSCvr67130

40(2a)35(2a)Skip task option in MBreplace and Software redeployscripts The scripts should have the option to skip tasksas user inputs and not by modifying the json file

CSCvp05204

40(2a)40(1a)Error noticed on SCVM Console print_req_error IOerror dev fd0 sector 0 on HyperV

It doesnt come up on SSH session

CSCvq34357

40(2a)30(1d)The VMNetwork switch is already created as part of theinstall process

Ideally the ip address assignment should be done duringthe initial install process as well No post install stepsshould be required altrnatively it could be done as partof a post install script

CSCvk36222

40(2a)25(1c)storfs service impacted during HX upgrade from HX21(1) to 25(1c)

HX node which is not being upgraded storfs servicestopped causing outage

CSCvg53223

40(2a)35(1b)HX Stretch Cluster Witness VM reverts to DHCP onreboot

CSCvp37536

40(2a)35(2b)HyperFlex Stretch Cluster Witness does not getprogrammed with an NTP server

CSCvp46578

Release Notes for Cisco HX Data Platform Release 4036

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2b)HyperFlex cluster does not heal after maintenance onnodeserver

CSCvr18528

40(2a)40(2a)High percentage of packet loss on the eth1 data interface9K MTU

CSCvr97089

40(2a)35(2f)The HyperFlex cluster healing state is stuck at 87 andis unable to progress

CSCvr16760

40(2a)40(2a)In some circumstances two replications can occur forthe same VM

From the GUI the replication status for this VM willshow stuck in an In Progress state but there are nojobs stuck in replication layer Subsequent Replicationsshould succeed

CSCvq22898

40(2a)35(2a)After an unexpected power outage the hyperflex clusteris unable to come up correctly

All processes are running time is synced and all nodescan vmkping each other as required

CSCvp13990

40(2a)40(1a)When specifying IP address instead of node ID stclinode maintenanceMode commands fail

CSCvp33657

40(2a)40(1a)2 node edge cluster if experiencing Intersightarbitratorconnectivity issues should NOT allowHXMaintenancemode to initiate as this will bring the cluster down

CSCvr31573

40(2a)35(2c)Old files in varsupportZKTxnlog are not purged withthe daily zklog-cleanup cron job

CSCvr89066

40(2a)35(2a)When a node is in MaintenanceMode any disk removalor replacement will be reflected in UI only after the nodeis brought back frommaintenancemode This is becausestorfs is not running on the node in maintenance andwill not be able to detect disk activities until it is broughtout of Maintenance Mode

CSCvo86431

40(2a)35(2c)Cluster might go offline and stop serving IO requestsrootcvm~ stcli cluster storage-summary--detailGet cluster storage summary failedjavanetConnectException Connection refused19216814210010207

CSCvr01645

40(2a)35(2d)A stretch cluster deployment fails with a messageFormatting nodes Node in Use on the Hyperflex installerand crmZoneType value is seen as 1 in storfscfg oroptspringpathconfigstretchtunes

CSCvq80340

Release Notes for Cisco HX Data Platform Release 4037

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)30(1c)During upgrade we upload the upgrade packagedownloaded from Ciscocom Upgrade packages are offile type tgz hxconnect acceptsstorfs-packages-401a-33028tar file a different type ofcompression and allows upgrade to start and fail whenit cannot decompress the file for bootstrap and scvmclientupgrade on ESXi

CSCvq63888

40(2a)40(2a)While attempting to install hyperflex via intersight thefollowing error is seen

Failed in Task Add hosts to vCenter Cluster with ErrorTry adding hosts manually to vCenter and retry Failedto get addressFailed to host by namelookupHostnamecompanycom on 000053 read udp12700145430-gt12700153 read connection refused

CSCvr52098

40(2a)35(2a)CoreAPI call may time out for cluster management APIwhile doing cluster create validation

CSCvo92952

40(2a)35(2d)Validation fails with Mixed mode expansion checkdue to Enhanced vMotion Compatibility Incompatibilitywhen the ESXi version is different

CSCvq95460

40(2a)30(1i)This failure is seen in installer when custom workflowis used hypervisor configuration + deploy (clean diskpartition=no) at Installing software packages on storagecontroller VM

CSCvr66309

40(2a)35(2d)The HX Installer apparently replaced a comma bysemicolon on the VLAN ID range while exporting theconfiguration to a JSON file

CSCvq93831

40(2a)40(1a)When using unsupported browsers HX Connect userswill not be able to login These users see an errorindicating unauthorized user

CSCvp22693

Release Notes for Cisco HX Data Platform Release 4038

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)40(1a)A vulnerability in the web-based management interfaceof Cisco HyperFlex Software could allow anunauthenticated remote attacker to conduct a cross-siterequest forgery (CSRF) attack on an affected system

The vulnerability is due to insufficient CSRF protectionsfor the web UI on an affected device An attacker couldexploit this vulnerability by persuading a user of theinterface to follow a malicious link A successful exploitcould allow the attacker to perform arbitrary actions withthe privilege level of the affected user

Cisco has released software updates that address thisvulnerability There are no workarounds that address thisvulnerability

This advisory is available at the following link

httpstoolsciscocomsecuritycentercontentCiscoSecurityAdvisorycisco-sa-20190807-hypflex-csrf

CSCvp24343

40(2a)35(2d)Add message to HX Connect Progress flow to notAcknowledge Pending Activities and reboot the serverson UCSM

HX Connect is doing a controlled rolling server upgradein the background

CSCvq22844

40(2a)40(1b)Compute Node expand operation fails due to incorrectcluster name

CSCvr43786

40(2a)26(1e)RF-2 cluster shutdown during data resynchronizationafter a node failure (non storfs) followed by multipledisk read failures and a HardBlacklist of a disk

CSCvp41241

40(2a)40(1a)HyperFlex Edge nodes do not properly set the ESXihostname during deployment

CSCvo13143

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4039

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4040

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1b) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401b-33133p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4041

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4042

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1b)40(1a)While running the HyperFlex installer with WindowsServer Hyper-V cluster creation process fails with theerror Failure occurred during ClusterCreation process Unable to post thecontent to the downstream This symptomis encountered while deploying HyperFlex cluster withHyperFlex nodes configuredordered with Cisco VIC1457 MLOM (PID HX-MLOM-C25Q-04) andWindows Server Datacenter or Core with Hyper-Vversion 2016 or 2019

CSCvp64140

Release Notes for Cisco HX Data Platform Release 4043

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)35(2b)Adding Micron 5200 drive to a cluster fails to increasecluster capacity

CSCvo69067

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4044

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4045

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1a) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401a-33028p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4046

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4047

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1a)30(1d)Cluster instability when disks of different sector sizeplaced in HX node

CSCvk17250

40(1a)

35(2c)

35(1a)When logged in using a local HX user account insteadof a Virtual Center account an error message appearsintermittently indicating Virtual Centerunreachable or Resource informationcannot be updated when VC is reachable

CSCvo36198

40(1a)

35(2b)

30(1d)HXDP does not work with EMC RecoverPoint - needsto support VMware API (FSS-Readdir)

CSCvk38003

Release Notes for Cisco HX Data Platform Release 4048

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)

35(2a)

30(1i)

25(1c)

25(1d)

Zookeeper (Exhibitor) process on HyperFlex StorageController VMs may fail to respond to requests whenvarzookeeper has low or no free space

CSCvm90352

40(1a)26(1c)Use Asynchronous consolidation for HX SnapshotsCSCvn02151

40(1a)35(2b)HX Quiesced Snapshot for Backup VendorsCSCvo90713

40(1a)35(2a)The cluster creationcluster expansion workflow stopswith the following error message at the validation step

FIRMWARE-Check UCSC-SAS-M5HD

FIRMWARE-Check UCSC-SAS-M5HD Required00000029000000320000003500000050 Found00000058

Action Needed Update the Controller Firmware toRequired Version

CSCvn17787

35(2b)30(1d)Kernel migration fails when a local datastore is searchedfor in the ESXi host

CSCvn73127

35(2a)26(1b)A node shuts down when two disks are replaced (acaching disk and another capacity disk) where thecapacity disk is inserted first and then the caching diskis inserted

CSCvk46364

Hyper-V

40(1a)35(2a)Cluster expansion may fail with an error code 500 -operation timed out

CSCvn28721

35(2b)35(2a)During upgrade remove the VLAN on team created foruser vSwitch During fresh installation only one VLANtag is set to the vSwitch and team although multipleVLANs were entered

CSCvn54300

35(2b)35(2a)While upgrading a Hyper-V cluster on account of a rarerace condition between the stUpgradeService andZookeeper servers the upgrade orchestration throws anupgrade validation error and the upgrade process isaborted

CSCvn60486

Release Notes for Cisco HX Data Platform Release 4049

Resolved Caveats in Release 40(1a)

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Startrestart EAM service invCenter

When EAM service is not runningvCenter registration is failing

CSCvt55712

35(2a)Contact Cisco TAC if you believeyou are seeing similar conditions

HyperFlex Stretch Cluster saw poorapplication performance due to writelatency cluster remained unhealthy andrebalance was stuck

This defect is currently being used forinvestigative purposes

CSCvt45344

35(2g)Remove log files fromvarlogspringpath folder when thelog files are not getting rotatedStop the relevant service removelarge log files fromvarlogspringpath folder andrestart the service

varstv folder in Controller VM maybecome full

CSCvt36374

40(2b)Restarting the stMgr will initializeit again

In a Cluster configured with DR networkStMgr may not get initialized and stuckin a deadlock while enabling the IPTablerules (can be verified from the stMgrlog)

CSCvt89709

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Click recover for the selectedVM again under the standalonemode This will move the VM torecovered state

When the user recovers a VM protectedin a group it along with other VMs in thegroup move to recovering state instandalonemode The selectedVM shouldmove to recovered state

CSCvs75553

35(2e)Clean up the fail over cluster andthen recreate the fail over clusterNo need to touch the HX storagecluster

Hyper-V When replicated DC was usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2d)40(1a)

Keep cluster utilization as low aspossible To avoid read-only statekeep below 92

HyperFlex clusters storage utilization isabove 92 usable capacity

CSCvr20922

Release Notes for Cisco HX Data Platform Release 4050

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1b)NAAll the disks in the node got locked afterrebooting the node

We successfully unlocked usingsed-clientsh -U command but wantedto test with another reboot and driveslocked again

CSCvs74286

35(2g)To recover the VM copy over thedata disks and attach them to thenew VM

After enable encryption HX connectencryption status showing Caution butall of the disks enable encryptionsuccessful

-Self Encrypting Drives Service isrunning on all of the nodes

-There is no error message of encryption

- USB0 interface is up

-All SED disks showing supported 1enabled 1 locked 0

CSCvs93245

40(1b)Enable kernelpanic_on_oops as apersistent configuration This willcause the node to panic and rebootimmediately

A cluster node running HX release40(1b) may hang in the Linux kernelThis is classified as an oops and adeviation from the expected behavior

CSCvs54285

Release Notes for Cisco HX Data Platform Release 4051

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Enabling K8iscsi stack in Hx clusterCSCvs41324

Release Notes for Cisco HX Data Platform Release 4052

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Steps to enable K8 on HX cluster

Perform following steps on ALLthe controller VMs

1 Updateetcinitscvmclientconf toenable tune iscsiEnable totrue

sed -iesiscsiEnable=falseiscsiEnable=trueetcinitscvmclientconf

2 Run following initctlcommand to reloadconfiguration

initctl reload-configuration

3 Restart scvmclient process

stop scvmclient startscvmclient

4 Run following initctlcommand to mount SYSTEMDS

initctl emit --no-waitsystem-datastore-created

5 Verify that iscsiEnable tuneis set to true using thefollowing command

ps -eaf | grep scvmclient |grep -v grep

root 6241 1 1 Dec06 014007optspringpathstorfs-corescvmclient-T logEnabled=false -TlogSyslogEnabled=true -TlogEchoToScreen=false -TstatLoggingToFile=false -TstatLoggingToSyslog=true-TlogDir=varlogspringpath-TnfsBackendServerList=1010748100-T iscsiEnable=true -TiscsiUseAsync=true -T

Release Notes for Cisco HX Data Platform Release 4053

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

iscsiConfigFilePath=nfsSYSTEMistgtconfps -eaf | grep scvmclient |grep -v grep

35(2e)Following procedure can be usedto check NFS Queue Depth andincrease if needed

rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth256 lt- Low value[rootHXESXI1] vsish -e setvmkModulesnfsclientmnt[DATASTORE]propertiesmaxQDepth 1024[rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth1024 lt- Optimalvalue

This requires an ESXi host rebootto take effect

Please place the node in HyperflexMaintenanceMode and gracefullyreboot the node for the changes tobe applied

HyperFlex Datastore NFS Queue Depthshows as 256 which can lead toperformance (including latency) issues

CSCvr83056

40(1b)This is cosmetic The error will goaway once the cluster returns tohealthy status

When a single node is offline in clusterstcli cluster storage-summary shows twonodes unavailable rootSCVM~ stclicluster storage-summary

CSCvs47735

35(2g)Modify both the root and adminpassword for the SCVMs to be thesame

stcli security password set -uadmin

Cluster expansion fails at Config Installerstage when the root and admin passwordfor the storage controller (SCVM) aredifferent

CSCvs69317

35(2d)NAAfter successfully changing the DNSserver on the HX controller we still cansee the original DNS entry that was addedduring the deployment

CSCvs69154

Release Notes for Cisco HX Data Platform Release 4054

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Customers are strongly encouragedto work with TAC in order toidentify and remediate this issue

1 Change getcluster from 2m to8m inoptspringpathstorfs-mgmtstMgr-10confapplicationconfon all nodes

2 Restart stMgr on all nodes

You may see this error message after afailed upgrade or other task such asattempting to enter a node into HXmaintenance mode

CSCvs53555

40(2a)Run the following command oneach storage controller VM as root

grep -i monitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp sed -iesmonitor_dns_serverstruemonitor_dns_serversfalseoptspringpathhx-diag-toolswatchdog_configjsonampamp grep -imonitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp restart watchdog

Receive the following alertevent in HXConnect HX Controller VMHOSTNAME one or more configuredDNS servers not responding

CSCvt13947

35(2b)Delete any empty (Size 0) log filesunder varlogzookeeperversion-2ORvarlogzookeeperstandaloneversion-2

Zookeeper fails to start while Exhibitor isrunning however echo srvr returnsnothing

CSCvs21562

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

40(4e)1 De-commission the server

2 Power drain the server -REMOVE BOTH POWERCORDS ON THE BACK OFTHE SERVER FOR 10SECONDS THENREINSERTPOWERCORDS

3 Re-commission the server

M2 boot disk is missing from serverinventory after upgrade to server firmware40(4e) As a result server fails to boot toOS installed in the M2 disk The issuepersists after re-acknowledgement as wellas de-commission andre-acknowledgement of the server

CSCvs02466

Release Notes for Cisco HX Data Platform Release 4055

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Re-start the flusher manuallyMany disks taken offline by a fault ormanually resulting in ENOSPC oncluster

CSCvq32721

35(2e)Clean up fail over cluster and thenrecreate the fail over cluster Thereis no need to touch the HX storagecluster

[Hyper-V]When replicated DCwas usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2b)Manually install upgradescvmclient VIB andmake sure thatit matches with HXDP version

Upgrade from release HX 35(2a) to35(2x) 4X does not upgrade scvmclientRecovery point may not work

CSCvq54992

Management

30(1b)To recover the VM copy over thedata disks and attach them to thenew VM

[HyperV]VMshows up onmultiple nodesCSCvj22992

35(1a)1 Do not change MTU at ESXlevel on running cluster

2 Reset back to the original value

ChangingMTU from 9000 to 1500 on oneESX node causes storfs process to restarton all nodes

CSCvm99150

Release Notes for Cisco HX Data Platform Release 4056

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)After the installation (for newdeployments) or after an upgradeto 401a (for existingdeployments) perform thefollowing steps on all ControllerVMs

1 Edit the following tune files (onall Controller VMs)

optspringpathconfiglfftunes

optspringpathconfigvsi_16tbtunes

2 SetcleanerEnableSegSummaryCleaning

to false

3 After editing the above tunefiles

a SSH in to all the ControllerVMs

b Run storfstool -- -Z

c Run the below command andcheck the values Tune valueshould be true

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=true

d Type below commands andapplied the tunes changesdynamically

echo false gttmpstprocfssystemtunecleanerEnableSegSummaryCleaning

e Verify that tune values arechanged Run below commandValue should be false

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=false

f umount tmpstprocfs

A cluster with 8TB or 12TB disk drivesmay experience IO stalls for severalminutes after another node fails in thecluster

CSCvp23718

Release Notes for Cisco HX Data Platform Release 4057

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)1 Update the conf file manuallyto change to the link local addressto

2 Restart scvmclient on thecontroller and ESX

HX 35(2b) FlexVol to HX 40 CSIupgrade does not work FlexVol continuesto work

CSCvp26319

35(2a)Disable rebalance until node healsthen reenable rebalance

7-node cluster healing very slowlybecause rebalance change AMS reqs areblocking healing process

CSCvp62512

35(2a)NAWith moderate cluster storage capacityusage under site failover cluster clustermay experience an all paths (APD) downcondition

CSCvq49412

40(1a)NAThis product includes Third-partySoftware that is affected by thevulnerabilities identified by the followingCommon Vulnerability and Exposures(CVE) IDs

CVE-2016-0762 CVE-2016-6797CVE-2016-6816 CVE-2016-8735CVE-2017-5647 CVE-2017-12615CVE-2017-12616 CVE-2017-12617CVE-2017-7674 CVE-2018-1304CVE-2018-8014 CVE-2018-1336CVE-2018-8034 CVE-2018-11784CVE-2019-0232

CSCvp36364

35(2d)NAstcli cluster info command needs toprovide UCSM VIP address Currently itshows ucsm-hostcom

CSCvq11456

35(2a)No workarounds currently Thestale entries do not affect the dayto day operations of HyperFlex

HyperFlex upgrade validation failedbecause cluster has extraneous stNodes ininternal database

CSCvq32530

35(2a)NAWhen not in maintenancemode all drivesappear in HX Connect and get updated asexpected

When in maintenance mode extra drivesare shown in HX Connect with no slotnumbers

CSCvq42378

35(2b)Contact TAC to cleanup duplicatenode

Nodes duplicated in HX Connect aftercluster recreate

CSCvq61825

Release Notes for Cisco HX Data Platform Release 4058

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)NACurrently HyperFlex installs on Hyper-Vdo not contain the stcli security whitelistcommandset

CSCvq66245

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

Release Notes for Cisco HX Data Platform Release 4059

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)[Hyper-V] Converged node expansionfails for incoming node with VIC 14xx

CSCvq39523

Release Notes for Cisco HX Data Platform Release 4060

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

1 Set Jumbo frame inWindowsHypervisor for incomingnode

Get-NetAdapter storage-data-a|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

Get-NetAdapter storage-data-b|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

2 Verify all interfaces are setcorrectly to support Jumbo frames

Get-NetAdapter storage |Get-NetAdapterAdvancedProperty| RegistryKeyword -matchjumbo | ft -auto

You should receive the followingmessage

Name DisplayName DisplayValue

RegistryKeyword RegistryValue

---- ----------- ------------

--------------- -------------

vswitch-hx-storage-data Jumbo

Packet 9014 Bytes

JumboPacket 9014

storage-data-b Jumbo Packet

Bytes 9014 JumboPacket

9014

storage-data-a Jumbo Packet

Bytes 9014 JumboPacket

9014

3 Reboot only the incoming nodeor the node being expanded

4 Retry Cluster Expansion in

Release Notes for Cisco HX Data Platform Release 4061

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

DescriptionCVEDefect IDRelease

OpenSSH vulnerabilities identified byCVE-2019-6111

For more information see the relatedCisco Security Advisory

CVE-2016-10708 CVE-2018-15473CVE-2018-20685 CVE-2019-6109CVE-2019-6111

CSCvp7646340(2a)

Multiple vulnerabilities associated withOpenSSL and LibNSS

For more information see the relatedCisco Security Advisory

CVE-2016-2105 CVE-2016-2106CVE-2016-2107 CVE-2016-2109CVE-2016-2176 CVE-2016-2177CVE-2016-2178 CVE-2016-2179CVE-2016-2180 CVE-2016-2181CVE-2016-2182 CVE-2016-2183CVE-2016-6302 CVE-2016-6303CVE-2016-6304 CVE-2016-6305CVE-2016-6306 CVE-2016-6307CVE-2016-6308 CVE-2016-7055CVE-2016-8610 CVE-2017-3731CVE-2017-3732 CVE-2017-3735CVE-2017-3736 CVE-2017-3737CVE-2017-3738 CVE-2018-0495CVE-2018-0732 CVE-2018-0734CVE-2018-0735 CVE-2018-0737CVE-2018-0739 CVE-2018-12384CVE-2018-12404 CVE-2018-5407CVE-2019-1559

CSCvp6667940(2a)

libpng vulnerability identified byCVE-2016-10087

For more information see the relatedCisco Security Advisory

CVE-2016-10087CSCvp6655540(2a)

Vulnerabilities associated with libjpg andlibpng

For more information see the relatedCisco Security Advisory

CVE-2014-9092 CVE-2016-3616CVE-2017-15232 CVE-2018-11212CVE-2018-11213 CVE-2018-11214CVE-2018-1152 CVE-2018-13785

CSCvp3458640(2a)

Vulnerabilities associated with Glibidentified by CVE-2018-16428 andCVE-2018-16429

For more information see the relatedCisco Security Advisory

CVE-2018-16428 CVE-2018-16429CSCvp3120740(2a)

Multiple vulnerabilities associated withcurl expat python 27 python 35 36and 37 freetype and giflib

For more information see the relatedCisco Security Advisory

CVE-2019-15133 CVE-2019-15903CVE-2019-5010 CVE-2019-5481CVE-2019-5482 CVE-2019-9636CVE-2019-9740 CVE-2019-9947CVE-2019-9948

CSCvr3690340(2a)

Release Notes for Cisco HX Data Platform Release 4025

Security Fixes

DescriptionCVEDefect IDRelease

Multiple third party softwarevulnerabilities

For more information see the relatedCisco Security Advisory

CVE-2019-14379 CVE-2019-12384CVE-2019-14439

CSCvq9203240(2a)

Vulnerabilities associated with elfutils

For more information see the relatedCisco Security Advisory

CVE-2018-16062 CVE-2018-16402CVE-2018-16403 CVE-2018-18310CVE-2018-18520 CVE-2018-18521CVE-2019-7149 CVE-2019-7150CVE-2019-7665

CSCvq4325040(2a)

Vulnerabilities associated with vim

For more information see the relatedCisco Security Advisory

CVE-2017-5953 CVE-2019-12735CSCvq4323040(2a)

Vulnerabilities associated with sqlite3

For more information see the relatedCisco Security Advisory

CVE-2016-6153 CVE-2017-10989CVE-2017-13685 CVE-2017-2518CVE-2017-2519 CVE-2017-2520CVE-2018-20346 CVE-2018-20505CVE-2018-20506 CVE-2019-8457CVE-2019-9936 CVE-2019-9937

CSCvq4321340(2a)

Vulnerabilities associated with glib20

For more information see the relatedCisco Security Advisory

CVE-2018-20843CSCvq4320940(2a)

Vulnerabilities associated with expat

For more information see the relatedCisco Security Advisory

CVE-2018-20843CSCvq4320540(2a)

Vulnerabilities associated with bzip2

For more information see the relatedCisco Security Advisory

CVE-2016-3189 CVE-2019-12900CSCvq4319440(2a)

Mulitple Vulnerabilities associated wtihNodeJS

For more information see the relatedCisco Security Advisory

CVE-2018-12115 CVE-2018-0734CVE-2018-5407 CVE-2018-12120CVE-2018-12121 CVE-2018-12122CVE-2018-12123 CVE-2018-12116CVE-2019-5737 CVE-2019-5739

CSCvq1069440(2a)

Vulnerabilities associated with PalletsJinja strformat_map and identified byCVE-2019-10906 and CVE-2016-10745

For more information see the relatedCisco Security Advisory

CVE-2019-10906 CVE-2016-10745CSCvq1038840(2a)

Release Notes for Cisco HX Data Platform Release 4026

Security Fixes

DescriptionCVEDefect IDRelease

Multiple Security Vulnerabilities

For more information see the relatedCisco Security Advisory

CVE-2014-8501 CVE-2014-9939CVE-2015-9262 CVE-2016-10087CVE-2016-2226 CVE-2016-4487CVE-2016-4488 CVE-2016-4489CVE-2016-4490 CVE-2016-4491CVE-2016-4492 CVE-2016-4493CVE-2016-6131 CVE-2018-10963CVE-2018-13785 CVE-2018-17100CVE-2018-17101 CVE-2018-18557CVE-2018-18661 CVE-2018-7456CVE-2018-8905

CSCvq0917840(2a)

Vulnerabilities associated withlibseccomp

For more information see the relatedCisco Security Advisory

CVE-2019-9893CSCvq0756840(2a)

Vulnerabilities associated withgdk-pixbuf

For more information see the relatedCisco Security Advisory

CVE-2017-12447CSCvq0675540(2a)

Vulnerabilities associated withpython-crypto

For more information see the relatedCisco Security Advisory

CVE-2018-6594CSCvp9381740(2a)

Vulnerabilities associated with wget

For more information see the relatedCisco Security Advisory

CVE-2018-20483 CVE-2019-5953CSCvp8672140(2a)

Multiple vulnerabilities associated withPython Crypto

For more information see the relatedCisco Security Advisory

CVE-2018-6594CSCvp6674840(2a)

Vulnerabilities associated withsensible-utils

For more information see the relatedCisco Security Advisory

CVE-2017-17512CSCvp6673440(2a)

Multiple vulnerabilities associated withPython

For more information see the relatedCisco Security Advisory

CVE-2018-1000030CSCvp6668940(2a)

Release Notes for Cisco HX Data Platform Release 4027

Security Fixes

DescriptionCVEDefect IDRelease

Multiple patch vulnerabilities identifiedby CVE-2016-10713

For more information see the relatedCisco Security Advisory

CVE-2016-10713CVE-2018-1000156 CVE-2018-6951

CSCvp6667240(2a)

Vulnerabilities associatedwith Little CMSidentified by CVE-2016-10165

For more information see the relatedCisco Security Advisory

CVE-2016-10165 CVE-2018-16435CSCvp6666440(2a)

Vulnerabilities associated with TenableScan identified byCVE-2011-5325CVE-2019-11068CVE-2016-7076CVE-2017-1000368

For more information see the relatedCisco Security Advisory

CVE-2011-5325 CVE-2016-7076CVE-2017-1000368CVE-2019-11068

CSCvp6474640(2a)

Vulnerabilities associated with libxml2294 and earlier as used in XMLSec1223 and earlier and other products andidentified by CVE-2016-9318

For more information see the relatedCisco Security Advisory

CVE-2016-9318 CVE-2017-16932CVE-2017-18258 CVE-2018-14404CVE-2018-14567

CSCvp3479240(2a)

Vulnerabilities associated with LFTPRemote File Names Unauthorized AccessVulnerability and identified byCVE-2018-10916

For more information see the relatedCisco Security Advisory

CVE-2018-10916CSCvp2926640(2a)

Vulnerabilities associated withCryptographyDeprecationWarning signerand verifier have been deprecated

For more information see the relatedCisco Security Advisory

CVE-2018-7750CSCvo3409740(2a)

Multiple Vulnerabilities in python python27 35

For more information see the relatedCisco Security Advisory

CVE-2019-16056CSCvr3979340(2a)

Release Notes for Cisco HX Data Platform Release 4028

Security Fixes

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toexecute commands as the root user

The vulnerability is due to insufficientinput validation An attacker could exploitthis vulnerability by connecting to thecluster service manager and injectingcommands into the bound process Asuccessful exploit could allow the attackerto run commands on the affected host asthe root user

Cisco has released software updates thataddress this vulnerability There areworkarounds that address thisvulnerability

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvq2417640(1b)35(2g)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

The vulnerability is due to an unprotectedlistening interface An attacker couldexploit this vulnerability by connectingto the listening interface and injectingcommands to the bound process Anexploit could allow the attacker to runcommands on the affected host as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvj9560640(1b)35(2g)

Release Notes for Cisco HX Data Platform Release 4029

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerabilities associated with theJVM18U121memory leak found duringVC alarms (concurrent 40 calls usingREST API)

CVE-2017-10053 CVE-2017-10067CVE-2017-10074 CVE-2017-10078CVE-2017-10081 CVE-2017-10087CVE-2017-10089 CVE-2017-10090CVE-2017-10096 CVE-2017-10101CVE-2017-10102 CVE-2017-10107CVE-2017-10108 CVE-2017-10109CVE-2017-10110 CVE-2017-10111CVE-2017-10115 CVE-2017-10116CVE-2017-10118 CVE-2017-10135CVE-2017-10176 CVE-2017-10193CVE-2017-10198 CVE-2017-10243CVE-2017-10274 CVE-2017-10281CVE-2017-10285 CVE-2017-10295CVE-2017-10345 CVE-2017-10346CVE-2017-10347 CVE-2017-10348CVE-2017-10349 CVE-2017-10350CVE-2017-10355 CVE-2017-10356CVE-2017-10357 CVE-2017-10388CVE-2017-3509 CVE-2017-3511CVE-2017-3526 CVE-2017-3533CVE-2017-3539 CVE-2017-3544CVE-2018-2579 CVE-2018-2582CVE-2018-2588 CVE-2018-2599CVE-2018-2602 CVE-2018-2603CVE-2018-2618 CVE-2018-2629CVE-2018-2633 CVE-2018-2634CVE-2018-2637 CVE-2018-2641CVE-2018-2663 CVE-2018-2677CVE-2018-2678 CVE-2018-2783CVE-2018-2790 CVE-2018-2794CVE-2018-2795 CVE-2018-2796CVE-2018-2797 CVE-2018-2798CVE-2018-2799 CVE-2018-2800CVE-2018-2814 CVE-2018-2815CVE-2018-2952 CVE-2018-3136CVE-2018-3139 CVE-2018-3149CVE-2018-3150 CVE-2018-3169CVE-2018-3180 CVE-2018-3183CVE-2018-3214 CVE-2019-2422

CSCvo8899740(1b)

Tomcat and Nginx logs are not beingcollected in the support bundle generatedthrough HX Connect in Release 35

NACSCvm5803140(1b)

The vulnerabilities associated with thelibmspack software package versionincluded in Cisco HX Data Platform

CVE-2018-18584

CVE-2018-18585

CSCvn3511940(1a)

Release Notes for Cisco HX Data Platform Release 4030

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerabilities associated withFasterXML Jackson-Databind TimeValueField Denial of Service

CVE-2018-14719

CVE-2018-14720

CVE-2018-1000873

CVE-2018-14721

CVE-2018-19360

CVE-2018-19362

CVE-2018-19361

CVE-2018-14718

CSCvn8228240(1a)

The vulnerabilities associated with theOpenSSL Protocol software packageversion included in Cisco HX DataPlatform

CVE-2013-3587CSCvo0505440(1a)

The vulnerabilities associated withThird-Party Software Denial of Service

CVE-2018-16487

CVE-2018-19361

CSCvo2781840(1a)

The vulnerabilities associated withLinuxkernel for Ubuntu 1710

CVE-2018-1092

CVE-2018-7492

CVE-2018-8087

CVE-2018-1068

CVE-2018-8781

CSCvm5314935(2a)

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)25(1d)stcli services dns remove should remove the DNS serverinfo from the interface files

CSCvs70967

40(2b)40(1b)HX 40(1b) new deployment from Intersight productioncloud

User receives following error while enabling CSI(Kubernetes integration)

Failure occurred during volume_access_enable Errorwas Nonzero exit code 1

CSCvt10522

40(2b)40(2a)Receive the following alertevent in HX Connect HXController VM HOSTNAME one or more configuredDNS servers not responding

CSCvt13947

Release Notes for Cisco HX Data Platform Release 4031

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)40(2a)API UI CLI show few Drive slots Empty after upgradeto 402a

The drives are seen in the sysmtool lsscsi and stclicommand outputs

The cluster is healthy with no errors

CSCvt14914

40(2b)35(2d)After successfully changing the DNS server on the HXcontroller we still can see the original DNS entry thatwas added during the deployment

CSCvs69154

40(2b)35(2e)HyperFlex Connect and VCenter show APD alarms forHX and non HX datastores This implies there is an issuewith HyperFlex when there may not be

CSCvs30080

40(2b)40(2a)An issue where all the disks in the node get locked afterrebooting the node

We successfully unlocked using sed-clientsh -Ucommand but wanted to test with another reboot anddrives locked again

CSCvs74286

40(2b)40(2a)When running stcli license commands on HyperFlexthe following error is seen

rootSpringpathControllertmp stcli license show all

Show smart licensing failed

Smart Agent is not ready please wait a minute and tryagain

CSCvt13929

40(2b)40(2a)After upgrading to HX 40(2a) the first RoboEdge nodethat gets rebooted then ignores persistent drivesPIDHX-HD24TB10K4KN

Cluster will be in degraded state and have reducedcapacity

CSCvt20144

40(2b)35(2b)Zookeeper fails to start while Exhibitor is runninghowever echo srvr returns nothing

CSCvs21562

40(2b)40(2a)When performing an HyperFlex upgrade a validationwarning may occur due to the host not having EnterprisePlus or Enterprise hypervisor licensing

Upgrade Validation Warning

ESXi host esx1labtest should be configured withVMware Enterprise license for upgrade to continue

CSCvs91787

Release Notes for Cisco HX Data Platform Release 4032

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)35(2g)Cluster expansion fails at Config Installer stage whenthe root and admin password for the storage controller(SCVM) are different

CSCvs69317

40(2b)35(2g)Panic while upgrading ESXiCSCvt06983

40(2b)40(2a)A cluster node running may hang in the Linux kernelThis is classified as an oops and a deviation from theexpected behavior

CSCvs54285

40(2b)35(2g)Rebalance failing on 10+10 node HX 35(2g) stretchcluster

CSCvs69007

40(2b)35(2d)The cluster becomes inaccessible to the IOVisorCSCvr54687

40(2b)40(2a)Panic on the storage controllerCSCvt61297

40(2b)40(2a)The size of support bundle is very large when collectedwith storfs-support command

CSCvt63306

40(2b)40(2a)storfs crash associated withFileSystemUsageWarningEvent orFileSystemUsageAlertEvent due to high usage ofvarstv above 80

CSCvt72807

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

40(2a)35(2d)In normal scenarios HX upgrade gets pendingACK afterupdating service profile In this case UCS only upgradewas stalled because after updating service profile HXupgrade was not getting pending ACK and it was alwayswaiting state

CSCvo39912

40(2a)35(1a)When using motherBoardReplace-12 to clean ZK fromold stNodepNodes resulted in unmounted datastore andmaking the size of the Hyperflex Datastores 0 resultingin all VMs in the cluster going offline

CSCvq39471

40(2a)26(1e)Upgrading cluster and getting error

Failed upgrade validations Checking vCenterconfiguration Reason Upgrade validations failed DRSFault Insufficient resources to satisfy configured failover

CSCvm77294

Release Notes for Cisco HX Data Platform Release 4033

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)The cluster expand fails on a node with DR replicationconfigured

When a HX cluster which has DR replication configuredis expanded we see the installer UI pulling in thereplicationVLAN information instead of themanagementVLAN information Even if we change that informationto the correct mgmt VLAN id and name it it does notseem to work as the node is configured with the VLANof the replicationVLAN in ESXi This leads to the failureof the node add with the host unreachable error

CSCvo70650

40(2a)35(1a)Customer reported that server didnt complete thefirmware upgrade automatically one by one

The user manually needs to put the host into maintenancemode and they manually acknowledged the pendingrequests to finish the UCS firmware upgrade

Maintenance policy is set to default (User-ack) as perthe design

CSCvo91624

40(2a)35(1i)When the cluster is in Failed state and stcli noderemove is attempted the output shows successful thoughthe node failed to get removed from the cluster

CSCvo93017

40(2a)35(1a)HyperFlex cluster upgrade may fail during validationwith a DRS Validation failed error

CSCvp31021

40(2a)35(2b)HX cluster expansion will fail with the following errormessage

MAC address pool configuration failure150[ErrorDescription] bad address block range definitioncollision

CSCvp58318

40(2a)35(2b)Memory usage by carbon cacheCSCvq34873

40(2a)35(2g)Upgrading ESXi cluster fails with error Nodemaintenance mode failed

CSCvr03240

40(2a)35(2e)storfs process does not automatically start on Exit HXMaintenance Mode or other tasks which restart StorageController

CSCvr88978

40(2a)35(2a)Post install script fails with message

Failed to execute ipmitool on HX node

CSCvp10707

40(2a)40(1a)HyperFlex expansionworkflow doesnt pull VLANnamecorrectly

CSCvp46539

Release Notes for Cisco HX Data Platform Release 4034

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)During HX cluster deployment - Validate ClusterCreation phase - HX Installer might fail with the error

fromvarlogspringpathstDeploylog

CSCvq45087

40(2a)35(2b)HX Installer fails to login to the SOL for configuringESX hosts

CSCvq91380

40(2a)35(2d)If DRS is enabled with configuration parameters otherthen fully automated ie manual or partially automatedthan this will change to fully automated during clusterexpansion

CSCvr44222

40(2a)35(1a)One of the validation task name has a typo The taskname is Validating node for mixed of SED and non-SEDdisks

It should be Validating node for mix of SED andnon-SED disks

CSCvp82175

40(2a)30(1i)This combination of using smaller NVMe drives is notsupported in expand operation For example if the drivesare only 375GB and they cannot be added to an existingcluster with larger caching SSD

CSCvo12359

40(2a)35(1a)HyperFlex includes a version of OpenSSL that is affectedby the vulnerabilities identified by the followingCommon Vulnerability and Exposures

CVE-2018-0495

CSCvp66679

40(2a)30(1e)SDK invoke exception when taking a native snapshot ona VM with GPU pci passthrough attached

CSCvo00511

40(2a)30(1i)Node replacement script fails due to EAM errorCSCvo62867

40(2a)35(1a)While pairing a cluster (HX release gt= 35) and cluster(HX release lt 35) remote replication network test failson the cluster (HX release lt 35)

CSCvo79760

40(2a)30(1b)Fixed the issue in the latest support-workflow bundleThis should not be hit anymore

CSCvo87061

40(2a)35(2a)On a 3 node cluster the MBreplace script fails as thescript looks for HEALTHY cluster status

CSCvo87080

40(2a)35(2a)MbReplace script (tar) hangs when running on 352b or40

CSCvp12359

Release Notes for Cisco HX Data Platform Release 4035

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)HX replication cleanup failing with error NA 35(2a) -INFODR state is not clean

CSCvp63958

40(2a)40(1a)The check_stig_parameters API incorrectly shows thecompliance state of the cluster as not STIG compliant

CSCvp66277

40(2a)35(2c)The snapshot creation on CBT enabled VM fails witherror Failed in vmreparent vmkfstools clone1

CSCvq06952

40(2a)35(2a)HyperFlex GUI when setting up remote DR-replicationpartnerpeer should show remote replication partner TCPport reachability test results

This will provide quick upfront connectivity results thatfrequently takes customer a while to manually check andconfirm

If reachability tests fail an info window and HX alertshould trigger that list the destination peer IP and thespecific port(s) that could not be reache

CSCvo60587

40(2a)35(2b)If you have more than one IP pool configured for areplication network and local replication network testsare failing you might be running into this issue

CSCvr67130

40(2a)35(2a)Skip task option in MBreplace and Software redeployscripts The scripts should have the option to skip tasksas user inputs and not by modifying the json file

CSCvp05204

40(2a)40(1a)Error noticed on SCVM Console print_req_error IOerror dev fd0 sector 0 on HyperV

It doesnt come up on SSH session

CSCvq34357

40(2a)30(1d)The VMNetwork switch is already created as part of theinstall process

Ideally the ip address assignment should be done duringthe initial install process as well No post install stepsshould be required altrnatively it could be done as partof a post install script

CSCvk36222

40(2a)25(1c)storfs service impacted during HX upgrade from HX21(1) to 25(1c)

HX node which is not being upgraded storfs servicestopped causing outage

CSCvg53223

40(2a)35(1b)HX Stretch Cluster Witness VM reverts to DHCP onreboot

CSCvp37536

40(2a)35(2b)HyperFlex Stretch Cluster Witness does not getprogrammed with an NTP server

CSCvp46578

Release Notes for Cisco HX Data Platform Release 4036

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2b)HyperFlex cluster does not heal after maintenance onnodeserver

CSCvr18528

40(2a)40(2a)High percentage of packet loss on the eth1 data interface9K MTU

CSCvr97089

40(2a)35(2f)The HyperFlex cluster healing state is stuck at 87 andis unable to progress

CSCvr16760

40(2a)40(2a)In some circumstances two replications can occur forthe same VM

From the GUI the replication status for this VM willshow stuck in an In Progress state but there are nojobs stuck in replication layer Subsequent Replicationsshould succeed

CSCvq22898

40(2a)35(2a)After an unexpected power outage the hyperflex clusteris unable to come up correctly

All processes are running time is synced and all nodescan vmkping each other as required

CSCvp13990

40(2a)40(1a)When specifying IP address instead of node ID stclinode maintenanceMode commands fail

CSCvp33657

40(2a)40(1a)2 node edge cluster if experiencing Intersightarbitratorconnectivity issues should NOT allowHXMaintenancemode to initiate as this will bring the cluster down

CSCvr31573

40(2a)35(2c)Old files in varsupportZKTxnlog are not purged withthe daily zklog-cleanup cron job

CSCvr89066

40(2a)35(2a)When a node is in MaintenanceMode any disk removalor replacement will be reflected in UI only after the nodeis brought back frommaintenancemode This is becausestorfs is not running on the node in maintenance andwill not be able to detect disk activities until it is broughtout of Maintenance Mode

CSCvo86431

40(2a)35(2c)Cluster might go offline and stop serving IO requestsrootcvm~ stcli cluster storage-summary--detailGet cluster storage summary failedjavanetConnectException Connection refused19216814210010207

CSCvr01645

40(2a)35(2d)A stretch cluster deployment fails with a messageFormatting nodes Node in Use on the Hyperflex installerand crmZoneType value is seen as 1 in storfscfg oroptspringpathconfigstretchtunes

CSCvq80340

Release Notes for Cisco HX Data Platform Release 4037

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)30(1c)During upgrade we upload the upgrade packagedownloaded from Ciscocom Upgrade packages are offile type tgz hxconnect acceptsstorfs-packages-401a-33028tar file a different type ofcompression and allows upgrade to start and fail whenit cannot decompress the file for bootstrap and scvmclientupgrade on ESXi

CSCvq63888

40(2a)40(2a)While attempting to install hyperflex via intersight thefollowing error is seen

Failed in Task Add hosts to vCenter Cluster with ErrorTry adding hosts manually to vCenter and retry Failedto get addressFailed to host by namelookupHostnamecompanycom on 000053 read udp12700145430-gt12700153 read connection refused

CSCvr52098

40(2a)35(2a)CoreAPI call may time out for cluster management APIwhile doing cluster create validation

CSCvo92952

40(2a)35(2d)Validation fails with Mixed mode expansion checkdue to Enhanced vMotion Compatibility Incompatibilitywhen the ESXi version is different

CSCvq95460

40(2a)30(1i)This failure is seen in installer when custom workflowis used hypervisor configuration + deploy (clean diskpartition=no) at Installing software packages on storagecontroller VM

CSCvr66309

40(2a)35(2d)The HX Installer apparently replaced a comma bysemicolon on the VLAN ID range while exporting theconfiguration to a JSON file

CSCvq93831

40(2a)40(1a)When using unsupported browsers HX Connect userswill not be able to login These users see an errorindicating unauthorized user

CSCvp22693

Release Notes for Cisco HX Data Platform Release 4038

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)40(1a)A vulnerability in the web-based management interfaceof Cisco HyperFlex Software could allow anunauthenticated remote attacker to conduct a cross-siterequest forgery (CSRF) attack on an affected system

The vulnerability is due to insufficient CSRF protectionsfor the web UI on an affected device An attacker couldexploit this vulnerability by persuading a user of theinterface to follow a malicious link A successful exploitcould allow the attacker to perform arbitrary actions withthe privilege level of the affected user

Cisco has released software updates that address thisvulnerability There are no workarounds that address thisvulnerability

This advisory is available at the following link

httpstoolsciscocomsecuritycentercontentCiscoSecurityAdvisorycisco-sa-20190807-hypflex-csrf

CSCvp24343

40(2a)35(2d)Add message to HX Connect Progress flow to notAcknowledge Pending Activities and reboot the serverson UCSM

HX Connect is doing a controlled rolling server upgradein the background

CSCvq22844

40(2a)40(1b)Compute Node expand operation fails due to incorrectcluster name

CSCvr43786

40(2a)26(1e)RF-2 cluster shutdown during data resynchronizationafter a node failure (non storfs) followed by multipledisk read failures and a HardBlacklist of a disk

CSCvp41241

40(2a)40(1a)HyperFlex Edge nodes do not properly set the ESXihostname during deployment

CSCvo13143

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4039

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4040

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1b) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401b-33133p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4041

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4042

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1b)40(1a)While running the HyperFlex installer with WindowsServer Hyper-V cluster creation process fails with theerror Failure occurred during ClusterCreation process Unable to post thecontent to the downstream This symptomis encountered while deploying HyperFlex cluster withHyperFlex nodes configuredordered with Cisco VIC1457 MLOM (PID HX-MLOM-C25Q-04) andWindows Server Datacenter or Core with Hyper-Vversion 2016 or 2019

CSCvp64140

Release Notes for Cisco HX Data Platform Release 4043

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)35(2b)Adding Micron 5200 drive to a cluster fails to increasecluster capacity

CSCvo69067

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4044

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4045

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1a) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401a-33028p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4046

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4047

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1a)30(1d)Cluster instability when disks of different sector sizeplaced in HX node

CSCvk17250

40(1a)

35(2c)

35(1a)When logged in using a local HX user account insteadof a Virtual Center account an error message appearsintermittently indicating Virtual Centerunreachable or Resource informationcannot be updated when VC is reachable

CSCvo36198

40(1a)

35(2b)

30(1d)HXDP does not work with EMC RecoverPoint - needsto support VMware API (FSS-Readdir)

CSCvk38003

Release Notes for Cisco HX Data Platform Release 4048

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)

35(2a)

30(1i)

25(1c)

25(1d)

Zookeeper (Exhibitor) process on HyperFlex StorageController VMs may fail to respond to requests whenvarzookeeper has low or no free space

CSCvm90352

40(1a)26(1c)Use Asynchronous consolidation for HX SnapshotsCSCvn02151

40(1a)35(2b)HX Quiesced Snapshot for Backup VendorsCSCvo90713

40(1a)35(2a)The cluster creationcluster expansion workflow stopswith the following error message at the validation step

FIRMWARE-Check UCSC-SAS-M5HD

FIRMWARE-Check UCSC-SAS-M5HD Required00000029000000320000003500000050 Found00000058

Action Needed Update the Controller Firmware toRequired Version

CSCvn17787

35(2b)30(1d)Kernel migration fails when a local datastore is searchedfor in the ESXi host

CSCvn73127

35(2a)26(1b)A node shuts down when two disks are replaced (acaching disk and another capacity disk) where thecapacity disk is inserted first and then the caching diskis inserted

CSCvk46364

Hyper-V

40(1a)35(2a)Cluster expansion may fail with an error code 500 -operation timed out

CSCvn28721

35(2b)35(2a)During upgrade remove the VLAN on team created foruser vSwitch During fresh installation only one VLANtag is set to the vSwitch and team although multipleVLANs were entered

CSCvn54300

35(2b)35(2a)While upgrading a Hyper-V cluster on account of a rarerace condition between the stUpgradeService andZookeeper servers the upgrade orchestration throws anupgrade validation error and the upgrade process isaborted

CSCvn60486

Release Notes for Cisco HX Data Platform Release 4049

Resolved Caveats in Release 40(1a)

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Startrestart EAM service invCenter

When EAM service is not runningvCenter registration is failing

CSCvt55712

35(2a)Contact Cisco TAC if you believeyou are seeing similar conditions

HyperFlex Stretch Cluster saw poorapplication performance due to writelatency cluster remained unhealthy andrebalance was stuck

This defect is currently being used forinvestigative purposes

CSCvt45344

35(2g)Remove log files fromvarlogspringpath folder when thelog files are not getting rotatedStop the relevant service removelarge log files fromvarlogspringpath folder andrestart the service

varstv folder in Controller VM maybecome full

CSCvt36374

40(2b)Restarting the stMgr will initializeit again

In a Cluster configured with DR networkStMgr may not get initialized and stuckin a deadlock while enabling the IPTablerules (can be verified from the stMgrlog)

CSCvt89709

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Click recover for the selectedVM again under the standalonemode This will move the VM torecovered state

When the user recovers a VM protectedin a group it along with other VMs in thegroup move to recovering state instandalonemode The selectedVM shouldmove to recovered state

CSCvs75553

35(2e)Clean up the fail over cluster andthen recreate the fail over clusterNo need to touch the HX storagecluster

Hyper-V When replicated DC was usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2d)40(1a)

Keep cluster utilization as low aspossible To avoid read-only statekeep below 92

HyperFlex clusters storage utilization isabove 92 usable capacity

CSCvr20922

Release Notes for Cisco HX Data Platform Release 4050

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1b)NAAll the disks in the node got locked afterrebooting the node

We successfully unlocked usingsed-clientsh -U command but wantedto test with another reboot and driveslocked again

CSCvs74286

35(2g)To recover the VM copy over thedata disks and attach them to thenew VM

After enable encryption HX connectencryption status showing Caution butall of the disks enable encryptionsuccessful

-Self Encrypting Drives Service isrunning on all of the nodes

-There is no error message of encryption

- USB0 interface is up

-All SED disks showing supported 1enabled 1 locked 0

CSCvs93245

40(1b)Enable kernelpanic_on_oops as apersistent configuration This willcause the node to panic and rebootimmediately

A cluster node running HX release40(1b) may hang in the Linux kernelThis is classified as an oops and adeviation from the expected behavior

CSCvs54285

Release Notes for Cisco HX Data Platform Release 4051

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Enabling K8iscsi stack in Hx clusterCSCvs41324

Release Notes for Cisco HX Data Platform Release 4052

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Steps to enable K8 on HX cluster

Perform following steps on ALLthe controller VMs

1 Updateetcinitscvmclientconf toenable tune iscsiEnable totrue

sed -iesiscsiEnable=falseiscsiEnable=trueetcinitscvmclientconf

2 Run following initctlcommand to reloadconfiguration

initctl reload-configuration

3 Restart scvmclient process

stop scvmclient startscvmclient

4 Run following initctlcommand to mount SYSTEMDS

initctl emit --no-waitsystem-datastore-created

5 Verify that iscsiEnable tuneis set to true using thefollowing command

ps -eaf | grep scvmclient |grep -v grep

root 6241 1 1 Dec06 014007optspringpathstorfs-corescvmclient-T logEnabled=false -TlogSyslogEnabled=true -TlogEchoToScreen=false -TstatLoggingToFile=false -TstatLoggingToSyslog=true-TlogDir=varlogspringpath-TnfsBackendServerList=1010748100-T iscsiEnable=true -TiscsiUseAsync=true -T

Release Notes for Cisco HX Data Platform Release 4053

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

iscsiConfigFilePath=nfsSYSTEMistgtconfps -eaf | grep scvmclient |grep -v grep

35(2e)Following procedure can be usedto check NFS Queue Depth andincrease if needed

rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth256 lt- Low value[rootHXESXI1] vsish -e setvmkModulesnfsclientmnt[DATASTORE]propertiesmaxQDepth 1024[rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth1024 lt- Optimalvalue

This requires an ESXi host rebootto take effect

Please place the node in HyperflexMaintenanceMode and gracefullyreboot the node for the changes tobe applied

HyperFlex Datastore NFS Queue Depthshows as 256 which can lead toperformance (including latency) issues

CSCvr83056

40(1b)This is cosmetic The error will goaway once the cluster returns tohealthy status

When a single node is offline in clusterstcli cluster storage-summary shows twonodes unavailable rootSCVM~ stclicluster storage-summary

CSCvs47735

35(2g)Modify both the root and adminpassword for the SCVMs to be thesame

stcli security password set -uadmin

Cluster expansion fails at Config Installerstage when the root and admin passwordfor the storage controller (SCVM) aredifferent

CSCvs69317

35(2d)NAAfter successfully changing the DNSserver on the HX controller we still cansee the original DNS entry that was addedduring the deployment

CSCvs69154

Release Notes for Cisco HX Data Platform Release 4054

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Customers are strongly encouragedto work with TAC in order toidentify and remediate this issue

1 Change getcluster from 2m to8m inoptspringpathstorfs-mgmtstMgr-10confapplicationconfon all nodes

2 Restart stMgr on all nodes

You may see this error message after afailed upgrade or other task such asattempting to enter a node into HXmaintenance mode

CSCvs53555

40(2a)Run the following command oneach storage controller VM as root

grep -i monitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp sed -iesmonitor_dns_serverstruemonitor_dns_serversfalseoptspringpathhx-diag-toolswatchdog_configjsonampamp grep -imonitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp restart watchdog

Receive the following alertevent in HXConnect HX Controller VMHOSTNAME one or more configuredDNS servers not responding

CSCvt13947

35(2b)Delete any empty (Size 0) log filesunder varlogzookeeperversion-2ORvarlogzookeeperstandaloneversion-2

Zookeeper fails to start while Exhibitor isrunning however echo srvr returnsnothing

CSCvs21562

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

40(4e)1 De-commission the server

2 Power drain the server -REMOVE BOTH POWERCORDS ON THE BACK OFTHE SERVER FOR 10SECONDS THENREINSERTPOWERCORDS

3 Re-commission the server

M2 boot disk is missing from serverinventory after upgrade to server firmware40(4e) As a result server fails to boot toOS installed in the M2 disk The issuepersists after re-acknowledgement as wellas de-commission andre-acknowledgement of the server

CSCvs02466

Release Notes for Cisco HX Data Platform Release 4055

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Re-start the flusher manuallyMany disks taken offline by a fault ormanually resulting in ENOSPC oncluster

CSCvq32721

35(2e)Clean up fail over cluster and thenrecreate the fail over cluster Thereis no need to touch the HX storagecluster

[Hyper-V]When replicated DCwas usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2b)Manually install upgradescvmclient VIB andmake sure thatit matches with HXDP version

Upgrade from release HX 35(2a) to35(2x) 4X does not upgrade scvmclientRecovery point may not work

CSCvq54992

Management

30(1b)To recover the VM copy over thedata disks and attach them to thenew VM

[HyperV]VMshows up onmultiple nodesCSCvj22992

35(1a)1 Do not change MTU at ESXlevel on running cluster

2 Reset back to the original value

ChangingMTU from 9000 to 1500 on oneESX node causes storfs process to restarton all nodes

CSCvm99150

Release Notes for Cisco HX Data Platform Release 4056

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)After the installation (for newdeployments) or after an upgradeto 401a (for existingdeployments) perform thefollowing steps on all ControllerVMs

1 Edit the following tune files (onall Controller VMs)

optspringpathconfiglfftunes

optspringpathconfigvsi_16tbtunes

2 SetcleanerEnableSegSummaryCleaning

to false

3 After editing the above tunefiles

a SSH in to all the ControllerVMs

b Run storfstool -- -Z

c Run the below command andcheck the values Tune valueshould be true

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=true

d Type below commands andapplied the tunes changesdynamically

echo false gttmpstprocfssystemtunecleanerEnableSegSummaryCleaning

e Verify that tune values arechanged Run below commandValue should be false

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=false

f umount tmpstprocfs

A cluster with 8TB or 12TB disk drivesmay experience IO stalls for severalminutes after another node fails in thecluster

CSCvp23718

Release Notes for Cisco HX Data Platform Release 4057

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)1 Update the conf file manuallyto change to the link local addressto

2 Restart scvmclient on thecontroller and ESX

HX 35(2b) FlexVol to HX 40 CSIupgrade does not work FlexVol continuesto work

CSCvp26319

35(2a)Disable rebalance until node healsthen reenable rebalance

7-node cluster healing very slowlybecause rebalance change AMS reqs areblocking healing process

CSCvp62512

35(2a)NAWith moderate cluster storage capacityusage under site failover cluster clustermay experience an all paths (APD) downcondition

CSCvq49412

40(1a)NAThis product includes Third-partySoftware that is affected by thevulnerabilities identified by the followingCommon Vulnerability and Exposures(CVE) IDs

CVE-2016-0762 CVE-2016-6797CVE-2016-6816 CVE-2016-8735CVE-2017-5647 CVE-2017-12615CVE-2017-12616 CVE-2017-12617CVE-2017-7674 CVE-2018-1304CVE-2018-8014 CVE-2018-1336CVE-2018-8034 CVE-2018-11784CVE-2019-0232

CSCvp36364

35(2d)NAstcli cluster info command needs toprovide UCSM VIP address Currently itshows ucsm-hostcom

CSCvq11456

35(2a)No workarounds currently Thestale entries do not affect the dayto day operations of HyperFlex

HyperFlex upgrade validation failedbecause cluster has extraneous stNodes ininternal database

CSCvq32530

35(2a)NAWhen not in maintenancemode all drivesappear in HX Connect and get updated asexpected

When in maintenance mode extra drivesare shown in HX Connect with no slotnumbers

CSCvq42378

35(2b)Contact TAC to cleanup duplicatenode

Nodes duplicated in HX Connect aftercluster recreate

CSCvq61825

Release Notes for Cisco HX Data Platform Release 4058

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)NACurrently HyperFlex installs on Hyper-Vdo not contain the stcli security whitelistcommandset

CSCvq66245

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

Release Notes for Cisco HX Data Platform Release 4059

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)[Hyper-V] Converged node expansionfails for incoming node with VIC 14xx

CSCvq39523

Release Notes for Cisco HX Data Platform Release 4060

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

1 Set Jumbo frame inWindowsHypervisor for incomingnode

Get-NetAdapter storage-data-a|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

Get-NetAdapter storage-data-b|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

2 Verify all interfaces are setcorrectly to support Jumbo frames

Get-NetAdapter storage |Get-NetAdapterAdvancedProperty| RegistryKeyword -matchjumbo | ft -auto

You should receive the followingmessage

Name DisplayName DisplayValue

RegistryKeyword RegistryValue

---- ----------- ------------

--------------- -------------

vswitch-hx-storage-data Jumbo

Packet 9014 Bytes

JumboPacket 9014

storage-data-b Jumbo Packet

Bytes 9014 JumboPacket

9014

storage-data-a Jumbo Packet

Bytes 9014 JumboPacket

9014

3 Reboot only the incoming nodeor the node being expanded

4 Retry Cluster Expansion in

Release Notes for Cisco HX Data Platform Release 4061

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

DescriptionCVEDefect IDRelease

Multiple third party softwarevulnerabilities

For more information see the relatedCisco Security Advisory

CVE-2019-14379 CVE-2019-12384CVE-2019-14439

CSCvq9203240(2a)

Vulnerabilities associated with elfutils

For more information see the relatedCisco Security Advisory

CVE-2018-16062 CVE-2018-16402CVE-2018-16403 CVE-2018-18310CVE-2018-18520 CVE-2018-18521CVE-2019-7149 CVE-2019-7150CVE-2019-7665

CSCvq4325040(2a)

Vulnerabilities associated with vim

For more information see the relatedCisco Security Advisory

CVE-2017-5953 CVE-2019-12735CSCvq4323040(2a)

Vulnerabilities associated with sqlite3

For more information see the relatedCisco Security Advisory

CVE-2016-6153 CVE-2017-10989CVE-2017-13685 CVE-2017-2518CVE-2017-2519 CVE-2017-2520CVE-2018-20346 CVE-2018-20505CVE-2018-20506 CVE-2019-8457CVE-2019-9936 CVE-2019-9937

CSCvq4321340(2a)

Vulnerabilities associated with glib20

For more information see the relatedCisco Security Advisory

CVE-2018-20843CSCvq4320940(2a)

Vulnerabilities associated with expat

For more information see the relatedCisco Security Advisory

CVE-2018-20843CSCvq4320540(2a)

Vulnerabilities associated with bzip2

For more information see the relatedCisco Security Advisory

CVE-2016-3189 CVE-2019-12900CSCvq4319440(2a)

Mulitple Vulnerabilities associated wtihNodeJS

For more information see the relatedCisco Security Advisory

CVE-2018-12115 CVE-2018-0734CVE-2018-5407 CVE-2018-12120CVE-2018-12121 CVE-2018-12122CVE-2018-12123 CVE-2018-12116CVE-2019-5737 CVE-2019-5739

CSCvq1069440(2a)

Vulnerabilities associated with PalletsJinja strformat_map and identified byCVE-2019-10906 and CVE-2016-10745

For more information see the relatedCisco Security Advisory

CVE-2019-10906 CVE-2016-10745CSCvq1038840(2a)

Release Notes for Cisco HX Data Platform Release 4026

Security Fixes

DescriptionCVEDefect IDRelease

Multiple Security Vulnerabilities

For more information see the relatedCisco Security Advisory

CVE-2014-8501 CVE-2014-9939CVE-2015-9262 CVE-2016-10087CVE-2016-2226 CVE-2016-4487CVE-2016-4488 CVE-2016-4489CVE-2016-4490 CVE-2016-4491CVE-2016-4492 CVE-2016-4493CVE-2016-6131 CVE-2018-10963CVE-2018-13785 CVE-2018-17100CVE-2018-17101 CVE-2018-18557CVE-2018-18661 CVE-2018-7456CVE-2018-8905

CSCvq0917840(2a)

Vulnerabilities associated withlibseccomp

For more information see the relatedCisco Security Advisory

CVE-2019-9893CSCvq0756840(2a)

Vulnerabilities associated withgdk-pixbuf

For more information see the relatedCisco Security Advisory

CVE-2017-12447CSCvq0675540(2a)

Vulnerabilities associated withpython-crypto

For more information see the relatedCisco Security Advisory

CVE-2018-6594CSCvp9381740(2a)

Vulnerabilities associated with wget

For more information see the relatedCisco Security Advisory

CVE-2018-20483 CVE-2019-5953CSCvp8672140(2a)

Multiple vulnerabilities associated withPython Crypto

For more information see the relatedCisco Security Advisory

CVE-2018-6594CSCvp6674840(2a)

Vulnerabilities associated withsensible-utils

For more information see the relatedCisco Security Advisory

CVE-2017-17512CSCvp6673440(2a)

Multiple vulnerabilities associated withPython

For more information see the relatedCisco Security Advisory

CVE-2018-1000030CSCvp6668940(2a)

Release Notes for Cisco HX Data Platform Release 4027

Security Fixes

DescriptionCVEDefect IDRelease

Multiple patch vulnerabilities identifiedby CVE-2016-10713

For more information see the relatedCisco Security Advisory

CVE-2016-10713CVE-2018-1000156 CVE-2018-6951

CSCvp6667240(2a)

Vulnerabilities associatedwith Little CMSidentified by CVE-2016-10165

For more information see the relatedCisco Security Advisory

CVE-2016-10165 CVE-2018-16435CSCvp6666440(2a)

Vulnerabilities associated with TenableScan identified byCVE-2011-5325CVE-2019-11068CVE-2016-7076CVE-2017-1000368

For more information see the relatedCisco Security Advisory

CVE-2011-5325 CVE-2016-7076CVE-2017-1000368CVE-2019-11068

CSCvp6474640(2a)

Vulnerabilities associated with libxml2294 and earlier as used in XMLSec1223 and earlier and other products andidentified by CVE-2016-9318

For more information see the relatedCisco Security Advisory

CVE-2016-9318 CVE-2017-16932CVE-2017-18258 CVE-2018-14404CVE-2018-14567

CSCvp3479240(2a)

Vulnerabilities associated with LFTPRemote File Names Unauthorized AccessVulnerability and identified byCVE-2018-10916

For more information see the relatedCisco Security Advisory

CVE-2018-10916CSCvp2926640(2a)

Vulnerabilities associated withCryptographyDeprecationWarning signerand verifier have been deprecated

For more information see the relatedCisco Security Advisory

CVE-2018-7750CSCvo3409740(2a)

Multiple Vulnerabilities in python python27 35

For more information see the relatedCisco Security Advisory

CVE-2019-16056CSCvr3979340(2a)

Release Notes for Cisco HX Data Platform Release 4028

Security Fixes

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toexecute commands as the root user

The vulnerability is due to insufficientinput validation An attacker could exploitthis vulnerability by connecting to thecluster service manager and injectingcommands into the bound process Asuccessful exploit could allow the attackerto run commands on the affected host asthe root user

Cisco has released software updates thataddress this vulnerability There areworkarounds that address thisvulnerability

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvq2417640(1b)35(2g)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

The vulnerability is due to an unprotectedlistening interface An attacker couldexploit this vulnerability by connectingto the listening interface and injectingcommands to the bound process Anexploit could allow the attacker to runcommands on the affected host as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvj9560640(1b)35(2g)

Release Notes for Cisco HX Data Platform Release 4029

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerabilities associated with theJVM18U121memory leak found duringVC alarms (concurrent 40 calls usingREST API)

CVE-2017-10053 CVE-2017-10067CVE-2017-10074 CVE-2017-10078CVE-2017-10081 CVE-2017-10087CVE-2017-10089 CVE-2017-10090CVE-2017-10096 CVE-2017-10101CVE-2017-10102 CVE-2017-10107CVE-2017-10108 CVE-2017-10109CVE-2017-10110 CVE-2017-10111CVE-2017-10115 CVE-2017-10116CVE-2017-10118 CVE-2017-10135CVE-2017-10176 CVE-2017-10193CVE-2017-10198 CVE-2017-10243CVE-2017-10274 CVE-2017-10281CVE-2017-10285 CVE-2017-10295CVE-2017-10345 CVE-2017-10346CVE-2017-10347 CVE-2017-10348CVE-2017-10349 CVE-2017-10350CVE-2017-10355 CVE-2017-10356CVE-2017-10357 CVE-2017-10388CVE-2017-3509 CVE-2017-3511CVE-2017-3526 CVE-2017-3533CVE-2017-3539 CVE-2017-3544CVE-2018-2579 CVE-2018-2582CVE-2018-2588 CVE-2018-2599CVE-2018-2602 CVE-2018-2603CVE-2018-2618 CVE-2018-2629CVE-2018-2633 CVE-2018-2634CVE-2018-2637 CVE-2018-2641CVE-2018-2663 CVE-2018-2677CVE-2018-2678 CVE-2018-2783CVE-2018-2790 CVE-2018-2794CVE-2018-2795 CVE-2018-2796CVE-2018-2797 CVE-2018-2798CVE-2018-2799 CVE-2018-2800CVE-2018-2814 CVE-2018-2815CVE-2018-2952 CVE-2018-3136CVE-2018-3139 CVE-2018-3149CVE-2018-3150 CVE-2018-3169CVE-2018-3180 CVE-2018-3183CVE-2018-3214 CVE-2019-2422

CSCvo8899740(1b)

Tomcat and Nginx logs are not beingcollected in the support bundle generatedthrough HX Connect in Release 35

NACSCvm5803140(1b)

The vulnerabilities associated with thelibmspack software package versionincluded in Cisco HX Data Platform

CVE-2018-18584

CVE-2018-18585

CSCvn3511940(1a)

Release Notes for Cisco HX Data Platform Release 4030

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerabilities associated withFasterXML Jackson-Databind TimeValueField Denial of Service

CVE-2018-14719

CVE-2018-14720

CVE-2018-1000873

CVE-2018-14721

CVE-2018-19360

CVE-2018-19362

CVE-2018-19361

CVE-2018-14718

CSCvn8228240(1a)

The vulnerabilities associated with theOpenSSL Protocol software packageversion included in Cisco HX DataPlatform

CVE-2013-3587CSCvo0505440(1a)

The vulnerabilities associated withThird-Party Software Denial of Service

CVE-2018-16487

CVE-2018-19361

CSCvo2781840(1a)

The vulnerabilities associated withLinuxkernel for Ubuntu 1710

CVE-2018-1092

CVE-2018-7492

CVE-2018-8087

CVE-2018-1068

CVE-2018-8781

CSCvm5314935(2a)

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)25(1d)stcli services dns remove should remove the DNS serverinfo from the interface files

CSCvs70967

40(2b)40(1b)HX 40(1b) new deployment from Intersight productioncloud

User receives following error while enabling CSI(Kubernetes integration)

Failure occurred during volume_access_enable Errorwas Nonzero exit code 1

CSCvt10522

40(2b)40(2a)Receive the following alertevent in HX Connect HXController VM HOSTNAME one or more configuredDNS servers not responding

CSCvt13947

Release Notes for Cisco HX Data Platform Release 4031

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)40(2a)API UI CLI show few Drive slots Empty after upgradeto 402a

The drives are seen in the sysmtool lsscsi and stclicommand outputs

The cluster is healthy with no errors

CSCvt14914

40(2b)35(2d)After successfully changing the DNS server on the HXcontroller we still can see the original DNS entry thatwas added during the deployment

CSCvs69154

40(2b)35(2e)HyperFlex Connect and VCenter show APD alarms forHX and non HX datastores This implies there is an issuewith HyperFlex when there may not be

CSCvs30080

40(2b)40(2a)An issue where all the disks in the node get locked afterrebooting the node

We successfully unlocked using sed-clientsh -Ucommand but wanted to test with another reboot anddrives locked again

CSCvs74286

40(2b)40(2a)When running stcli license commands on HyperFlexthe following error is seen

rootSpringpathControllertmp stcli license show all

Show smart licensing failed

Smart Agent is not ready please wait a minute and tryagain

CSCvt13929

40(2b)40(2a)After upgrading to HX 40(2a) the first RoboEdge nodethat gets rebooted then ignores persistent drivesPIDHX-HD24TB10K4KN

Cluster will be in degraded state and have reducedcapacity

CSCvt20144

40(2b)35(2b)Zookeeper fails to start while Exhibitor is runninghowever echo srvr returns nothing

CSCvs21562

40(2b)40(2a)When performing an HyperFlex upgrade a validationwarning may occur due to the host not having EnterprisePlus or Enterprise hypervisor licensing

Upgrade Validation Warning

ESXi host esx1labtest should be configured withVMware Enterprise license for upgrade to continue

CSCvs91787

Release Notes for Cisco HX Data Platform Release 4032

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)35(2g)Cluster expansion fails at Config Installer stage whenthe root and admin password for the storage controller(SCVM) are different

CSCvs69317

40(2b)35(2g)Panic while upgrading ESXiCSCvt06983

40(2b)40(2a)A cluster node running may hang in the Linux kernelThis is classified as an oops and a deviation from theexpected behavior

CSCvs54285

40(2b)35(2g)Rebalance failing on 10+10 node HX 35(2g) stretchcluster

CSCvs69007

40(2b)35(2d)The cluster becomes inaccessible to the IOVisorCSCvr54687

40(2b)40(2a)Panic on the storage controllerCSCvt61297

40(2b)40(2a)The size of support bundle is very large when collectedwith storfs-support command

CSCvt63306

40(2b)40(2a)storfs crash associated withFileSystemUsageWarningEvent orFileSystemUsageAlertEvent due to high usage ofvarstv above 80

CSCvt72807

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

40(2a)35(2d)In normal scenarios HX upgrade gets pendingACK afterupdating service profile In this case UCS only upgradewas stalled because after updating service profile HXupgrade was not getting pending ACK and it was alwayswaiting state

CSCvo39912

40(2a)35(1a)When using motherBoardReplace-12 to clean ZK fromold stNodepNodes resulted in unmounted datastore andmaking the size of the Hyperflex Datastores 0 resultingin all VMs in the cluster going offline

CSCvq39471

40(2a)26(1e)Upgrading cluster and getting error

Failed upgrade validations Checking vCenterconfiguration Reason Upgrade validations failed DRSFault Insufficient resources to satisfy configured failover

CSCvm77294

Release Notes for Cisco HX Data Platform Release 4033

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)The cluster expand fails on a node with DR replicationconfigured

When a HX cluster which has DR replication configuredis expanded we see the installer UI pulling in thereplicationVLAN information instead of themanagementVLAN information Even if we change that informationto the correct mgmt VLAN id and name it it does notseem to work as the node is configured with the VLANof the replicationVLAN in ESXi This leads to the failureof the node add with the host unreachable error

CSCvo70650

40(2a)35(1a)Customer reported that server didnt complete thefirmware upgrade automatically one by one

The user manually needs to put the host into maintenancemode and they manually acknowledged the pendingrequests to finish the UCS firmware upgrade

Maintenance policy is set to default (User-ack) as perthe design

CSCvo91624

40(2a)35(1i)When the cluster is in Failed state and stcli noderemove is attempted the output shows successful thoughthe node failed to get removed from the cluster

CSCvo93017

40(2a)35(1a)HyperFlex cluster upgrade may fail during validationwith a DRS Validation failed error

CSCvp31021

40(2a)35(2b)HX cluster expansion will fail with the following errormessage

MAC address pool configuration failure150[ErrorDescription] bad address block range definitioncollision

CSCvp58318

40(2a)35(2b)Memory usage by carbon cacheCSCvq34873

40(2a)35(2g)Upgrading ESXi cluster fails with error Nodemaintenance mode failed

CSCvr03240

40(2a)35(2e)storfs process does not automatically start on Exit HXMaintenance Mode or other tasks which restart StorageController

CSCvr88978

40(2a)35(2a)Post install script fails with message

Failed to execute ipmitool on HX node

CSCvp10707

40(2a)40(1a)HyperFlex expansionworkflow doesnt pull VLANnamecorrectly

CSCvp46539

Release Notes for Cisco HX Data Platform Release 4034

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)During HX cluster deployment - Validate ClusterCreation phase - HX Installer might fail with the error

fromvarlogspringpathstDeploylog

CSCvq45087

40(2a)35(2b)HX Installer fails to login to the SOL for configuringESX hosts

CSCvq91380

40(2a)35(2d)If DRS is enabled with configuration parameters otherthen fully automated ie manual or partially automatedthan this will change to fully automated during clusterexpansion

CSCvr44222

40(2a)35(1a)One of the validation task name has a typo The taskname is Validating node for mixed of SED and non-SEDdisks

It should be Validating node for mix of SED andnon-SED disks

CSCvp82175

40(2a)30(1i)This combination of using smaller NVMe drives is notsupported in expand operation For example if the drivesare only 375GB and they cannot be added to an existingcluster with larger caching SSD

CSCvo12359

40(2a)35(1a)HyperFlex includes a version of OpenSSL that is affectedby the vulnerabilities identified by the followingCommon Vulnerability and Exposures

CVE-2018-0495

CSCvp66679

40(2a)30(1e)SDK invoke exception when taking a native snapshot ona VM with GPU pci passthrough attached

CSCvo00511

40(2a)30(1i)Node replacement script fails due to EAM errorCSCvo62867

40(2a)35(1a)While pairing a cluster (HX release gt= 35) and cluster(HX release lt 35) remote replication network test failson the cluster (HX release lt 35)

CSCvo79760

40(2a)30(1b)Fixed the issue in the latest support-workflow bundleThis should not be hit anymore

CSCvo87061

40(2a)35(2a)On a 3 node cluster the MBreplace script fails as thescript looks for HEALTHY cluster status

CSCvo87080

40(2a)35(2a)MbReplace script (tar) hangs when running on 352b or40

CSCvp12359

Release Notes for Cisco HX Data Platform Release 4035

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)HX replication cleanup failing with error NA 35(2a) -INFODR state is not clean

CSCvp63958

40(2a)40(1a)The check_stig_parameters API incorrectly shows thecompliance state of the cluster as not STIG compliant

CSCvp66277

40(2a)35(2c)The snapshot creation on CBT enabled VM fails witherror Failed in vmreparent vmkfstools clone1

CSCvq06952

40(2a)35(2a)HyperFlex GUI when setting up remote DR-replicationpartnerpeer should show remote replication partner TCPport reachability test results

This will provide quick upfront connectivity results thatfrequently takes customer a while to manually check andconfirm

If reachability tests fail an info window and HX alertshould trigger that list the destination peer IP and thespecific port(s) that could not be reache

CSCvo60587

40(2a)35(2b)If you have more than one IP pool configured for areplication network and local replication network testsare failing you might be running into this issue

CSCvr67130

40(2a)35(2a)Skip task option in MBreplace and Software redeployscripts The scripts should have the option to skip tasksas user inputs and not by modifying the json file

CSCvp05204

40(2a)40(1a)Error noticed on SCVM Console print_req_error IOerror dev fd0 sector 0 on HyperV

It doesnt come up on SSH session

CSCvq34357

40(2a)30(1d)The VMNetwork switch is already created as part of theinstall process

Ideally the ip address assignment should be done duringthe initial install process as well No post install stepsshould be required altrnatively it could be done as partof a post install script

CSCvk36222

40(2a)25(1c)storfs service impacted during HX upgrade from HX21(1) to 25(1c)

HX node which is not being upgraded storfs servicestopped causing outage

CSCvg53223

40(2a)35(1b)HX Stretch Cluster Witness VM reverts to DHCP onreboot

CSCvp37536

40(2a)35(2b)HyperFlex Stretch Cluster Witness does not getprogrammed with an NTP server

CSCvp46578

Release Notes for Cisco HX Data Platform Release 4036

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2b)HyperFlex cluster does not heal after maintenance onnodeserver

CSCvr18528

40(2a)40(2a)High percentage of packet loss on the eth1 data interface9K MTU

CSCvr97089

40(2a)35(2f)The HyperFlex cluster healing state is stuck at 87 andis unable to progress

CSCvr16760

40(2a)40(2a)In some circumstances two replications can occur forthe same VM

From the GUI the replication status for this VM willshow stuck in an In Progress state but there are nojobs stuck in replication layer Subsequent Replicationsshould succeed

CSCvq22898

40(2a)35(2a)After an unexpected power outage the hyperflex clusteris unable to come up correctly

All processes are running time is synced and all nodescan vmkping each other as required

CSCvp13990

40(2a)40(1a)When specifying IP address instead of node ID stclinode maintenanceMode commands fail

CSCvp33657

40(2a)40(1a)2 node edge cluster if experiencing Intersightarbitratorconnectivity issues should NOT allowHXMaintenancemode to initiate as this will bring the cluster down

CSCvr31573

40(2a)35(2c)Old files in varsupportZKTxnlog are not purged withthe daily zklog-cleanup cron job

CSCvr89066

40(2a)35(2a)When a node is in MaintenanceMode any disk removalor replacement will be reflected in UI only after the nodeis brought back frommaintenancemode This is becausestorfs is not running on the node in maintenance andwill not be able to detect disk activities until it is broughtout of Maintenance Mode

CSCvo86431

40(2a)35(2c)Cluster might go offline and stop serving IO requestsrootcvm~ stcli cluster storage-summary--detailGet cluster storage summary failedjavanetConnectException Connection refused19216814210010207

CSCvr01645

40(2a)35(2d)A stretch cluster deployment fails with a messageFormatting nodes Node in Use on the Hyperflex installerand crmZoneType value is seen as 1 in storfscfg oroptspringpathconfigstretchtunes

CSCvq80340

Release Notes for Cisco HX Data Platform Release 4037

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)30(1c)During upgrade we upload the upgrade packagedownloaded from Ciscocom Upgrade packages are offile type tgz hxconnect acceptsstorfs-packages-401a-33028tar file a different type ofcompression and allows upgrade to start and fail whenit cannot decompress the file for bootstrap and scvmclientupgrade on ESXi

CSCvq63888

40(2a)40(2a)While attempting to install hyperflex via intersight thefollowing error is seen

Failed in Task Add hosts to vCenter Cluster with ErrorTry adding hosts manually to vCenter and retry Failedto get addressFailed to host by namelookupHostnamecompanycom on 000053 read udp12700145430-gt12700153 read connection refused

CSCvr52098

40(2a)35(2a)CoreAPI call may time out for cluster management APIwhile doing cluster create validation

CSCvo92952

40(2a)35(2d)Validation fails with Mixed mode expansion checkdue to Enhanced vMotion Compatibility Incompatibilitywhen the ESXi version is different

CSCvq95460

40(2a)30(1i)This failure is seen in installer when custom workflowis used hypervisor configuration + deploy (clean diskpartition=no) at Installing software packages on storagecontroller VM

CSCvr66309

40(2a)35(2d)The HX Installer apparently replaced a comma bysemicolon on the VLAN ID range while exporting theconfiguration to a JSON file

CSCvq93831

40(2a)40(1a)When using unsupported browsers HX Connect userswill not be able to login These users see an errorindicating unauthorized user

CSCvp22693

Release Notes for Cisco HX Data Platform Release 4038

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)40(1a)A vulnerability in the web-based management interfaceof Cisco HyperFlex Software could allow anunauthenticated remote attacker to conduct a cross-siterequest forgery (CSRF) attack on an affected system

The vulnerability is due to insufficient CSRF protectionsfor the web UI on an affected device An attacker couldexploit this vulnerability by persuading a user of theinterface to follow a malicious link A successful exploitcould allow the attacker to perform arbitrary actions withthe privilege level of the affected user

Cisco has released software updates that address thisvulnerability There are no workarounds that address thisvulnerability

This advisory is available at the following link

httpstoolsciscocomsecuritycentercontentCiscoSecurityAdvisorycisco-sa-20190807-hypflex-csrf

CSCvp24343

40(2a)35(2d)Add message to HX Connect Progress flow to notAcknowledge Pending Activities and reboot the serverson UCSM

HX Connect is doing a controlled rolling server upgradein the background

CSCvq22844

40(2a)40(1b)Compute Node expand operation fails due to incorrectcluster name

CSCvr43786

40(2a)26(1e)RF-2 cluster shutdown during data resynchronizationafter a node failure (non storfs) followed by multipledisk read failures and a HardBlacklist of a disk

CSCvp41241

40(2a)40(1a)HyperFlex Edge nodes do not properly set the ESXihostname during deployment

CSCvo13143

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4039

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4040

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1b) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401b-33133p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4041

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4042

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1b)40(1a)While running the HyperFlex installer with WindowsServer Hyper-V cluster creation process fails with theerror Failure occurred during ClusterCreation process Unable to post thecontent to the downstream This symptomis encountered while deploying HyperFlex cluster withHyperFlex nodes configuredordered with Cisco VIC1457 MLOM (PID HX-MLOM-C25Q-04) andWindows Server Datacenter or Core with Hyper-Vversion 2016 or 2019

CSCvp64140

Release Notes for Cisco HX Data Platform Release 4043

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)35(2b)Adding Micron 5200 drive to a cluster fails to increasecluster capacity

CSCvo69067

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4044

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4045

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1a) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401a-33028p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4046

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4047

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1a)30(1d)Cluster instability when disks of different sector sizeplaced in HX node

CSCvk17250

40(1a)

35(2c)

35(1a)When logged in using a local HX user account insteadof a Virtual Center account an error message appearsintermittently indicating Virtual Centerunreachable or Resource informationcannot be updated when VC is reachable

CSCvo36198

40(1a)

35(2b)

30(1d)HXDP does not work with EMC RecoverPoint - needsto support VMware API (FSS-Readdir)

CSCvk38003

Release Notes for Cisco HX Data Platform Release 4048

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)

35(2a)

30(1i)

25(1c)

25(1d)

Zookeeper (Exhibitor) process on HyperFlex StorageController VMs may fail to respond to requests whenvarzookeeper has low or no free space

CSCvm90352

40(1a)26(1c)Use Asynchronous consolidation for HX SnapshotsCSCvn02151

40(1a)35(2b)HX Quiesced Snapshot for Backup VendorsCSCvo90713

40(1a)35(2a)The cluster creationcluster expansion workflow stopswith the following error message at the validation step

FIRMWARE-Check UCSC-SAS-M5HD

FIRMWARE-Check UCSC-SAS-M5HD Required00000029000000320000003500000050 Found00000058

Action Needed Update the Controller Firmware toRequired Version

CSCvn17787

35(2b)30(1d)Kernel migration fails when a local datastore is searchedfor in the ESXi host

CSCvn73127

35(2a)26(1b)A node shuts down when two disks are replaced (acaching disk and another capacity disk) where thecapacity disk is inserted first and then the caching diskis inserted

CSCvk46364

Hyper-V

40(1a)35(2a)Cluster expansion may fail with an error code 500 -operation timed out

CSCvn28721

35(2b)35(2a)During upgrade remove the VLAN on team created foruser vSwitch During fresh installation only one VLANtag is set to the vSwitch and team although multipleVLANs were entered

CSCvn54300

35(2b)35(2a)While upgrading a Hyper-V cluster on account of a rarerace condition between the stUpgradeService andZookeeper servers the upgrade orchestration throws anupgrade validation error and the upgrade process isaborted

CSCvn60486

Release Notes for Cisco HX Data Platform Release 4049

Resolved Caveats in Release 40(1a)

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Startrestart EAM service invCenter

When EAM service is not runningvCenter registration is failing

CSCvt55712

35(2a)Contact Cisco TAC if you believeyou are seeing similar conditions

HyperFlex Stretch Cluster saw poorapplication performance due to writelatency cluster remained unhealthy andrebalance was stuck

This defect is currently being used forinvestigative purposes

CSCvt45344

35(2g)Remove log files fromvarlogspringpath folder when thelog files are not getting rotatedStop the relevant service removelarge log files fromvarlogspringpath folder andrestart the service

varstv folder in Controller VM maybecome full

CSCvt36374

40(2b)Restarting the stMgr will initializeit again

In a Cluster configured with DR networkStMgr may not get initialized and stuckin a deadlock while enabling the IPTablerules (can be verified from the stMgrlog)

CSCvt89709

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Click recover for the selectedVM again under the standalonemode This will move the VM torecovered state

When the user recovers a VM protectedin a group it along with other VMs in thegroup move to recovering state instandalonemode The selectedVM shouldmove to recovered state

CSCvs75553

35(2e)Clean up the fail over cluster andthen recreate the fail over clusterNo need to touch the HX storagecluster

Hyper-V When replicated DC was usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2d)40(1a)

Keep cluster utilization as low aspossible To avoid read-only statekeep below 92

HyperFlex clusters storage utilization isabove 92 usable capacity

CSCvr20922

Release Notes for Cisco HX Data Platform Release 4050

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1b)NAAll the disks in the node got locked afterrebooting the node

We successfully unlocked usingsed-clientsh -U command but wantedto test with another reboot and driveslocked again

CSCvs74286

35(2g)To recover the VM copy over thedata disks and attach them to thenew VM

After enable encryption HX connectencryption status showing Caution butall of the disks enable encryptionsuccessful

-Self Encrypting Drives Service isrunning on all of the nodes

-There is no error message of encryption

- USB0 interface is up

-All SED disks showing supported 1enabled 1 locked 0

CSCvs93245

40(1b)Enable kernelpanic_on_oops as apersistent configuration This willcause the node to panic and rebootimmediately

A cluster node running HX release40(1b) may hang in the Linux kernelThis is classified as an oops and adeviation from the expected behavior

CSCvs54285

Release Notes for Cisco HX Data Platform Release 4051

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Enabling K8iscsi stack in Hx clusterCSCvs41324

Release Notes for Cisco HX Data Platform Release 4052

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Steps to enable K8 on HX cluster

Perform following steps on ALLthe controller VMs

1 Updateetcinitscvmclientconf toenable tune iscsiEnable totrue

sed -iesiscsiEnable=falseiscsiEnable=trueetcinitscvmclientconf

2 Run following initctlcommand to reloadconfiguration

initctl reload-configuration

3 Restart scvmclient process

stop scvmclient startscvmclient

4 Run following initctlcommand to mount SYSTEMDS

initctl emit --no-waitsystem-datastore-created

5 Verify that iscsiEnable tuneis set to true using thefollowing command

ps -eaf | grep scvmclient |grep -v grep

root 6241 1 1 Dec06 014007optspringpathstorfs-corescvmclient-T logEnabled=false -TlogSyslogEnabled=true -TlogEchoToScreen=false -TstatLoggingToFile=false -TstatLoggingToSyslog=true-TlogDir=varlogspringpath-TnfsBackendServerList=1010748100-T iscsiEnable=true -TiscsiUseAsync=true -T

Release Notes for Cisco HX Data Platform Release 4053

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

iscsiConfigFilePath=nfsSYSTEMistgtconfps -eaf | grep scvmclient |grep -v grep

35(2e)Following procedure can be usedto check NFS Queue Depth andincrease if needed

rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth256 lt- Low value[rootHXESXI1] vsish -e setvmkModulesnfsclientmnt[DATASTORE]propertiesmaxQDepth 1024[rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth1024 lt- Optimalvalue

This requires an ESXi host rebootto take effect

Please place the node in HyperflexMaintenanceMode and gracefullyreboot the node for the changes tobe applied

HyperFlex Datastore NFS Queue Depthshows as 256 which can lead toperformance (including latency) issues

CSCvr83056

40(1b)This is cosmetic The error will goaway once the cluster returns tohealthy status

When a single node is offline in clusterstcli cluster storage-summary shows twonodes unavailable rootSCVM~ stclicluster storage-summary

CSCvs47735

35(2g)Modify both the root and adminpassword for the SCVMs to be thesame

stcli security password set -uadmin

Cluster expansion fails at Config Installerstage when the root and admin passwordfor the storage controller (SCVM) aredifferent

CSCvs69317

35(2d)NAAfter successfully changing the DNSserver on the HX controller we still cansee the original DNS entry that was addedduring the deployment

CSCvs69154

Release Notes for Cisco HX Data Platform Release 4054

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Customers are strongly encouragedto work with TAC in order toidentify and remediate this issue

1 Change getcluster from 2m to8m inoptspringpathstorfs-mgmtstMgr-10confapplicationconfon all nodes

2 Restart stMgr on all nodes

You may see this error message after afailed upgrade or other task such asattempting to enter a node into HXmaintenance mode

CSCvs53555

40(2a)Run the following command oneach storage controller VM as root

grep -i monitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp sed -iesmonitor_dns_serverstruemonitor_dns_serversfalseoptspringpathhx-diag-toolswatchdog_configjsonampamp grep -imonitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp restart watchdog

Receive the following alertevent in HXConnect HX Controller VMHOSTNAME one or more configuredDNS servers not responding

CSCvt13947

35(2b)Delete any empty (Size 0) log filesunder varlogzookeeperversion-2ORvarlogzookeeperstandaloneversion-2

Zookeeper fails to start while Exhibitor isrunning however echo srvr returnsnothing

CSCvs21562

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

40(4e)1 De-commission the server

2 Power drain the server -REMOVE BOTH POWERCORDS ON THE BACK OFTHE SERVER FOR 10SECONDS THENREINSERTPOWERCORDS

3 Re-commission the server

M2 boot disk is missing from serverinventory after upgrade to server firmware40(4e) As a result server fails to boot toOS installed in the M2 disk The issuepersists after re-acknowledgement as wellas de-commission andre-acknowledgement of the server

CSCvs02466

Release Notes for Cisco HX Data Platform Release 4055

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Re-start the flusher manuallyMany disks taken offline by a fault ormanually resulting in ENOSPC oncluster

CSCvq32721

35(2e)Clean up fail over cluster and thenrecreate the fail over cluster Thereis no need to touch the HX storagecluster

[Hyper-V]When replicated DCwas usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2b)Manually install upgradescvmclient VIB andmake sure thatit matches with HXDP version

Upgrade from release HX 35(2a) to35(2x) 4X does not upgrade scvmclientRecovery point may not work

CSCvq54992

Management

30(1b)To recover the VM copy over thedata disks and attach them to thenew VM

[HyperV]VMshows up onmultiple nodesCSCvj22992

35(1a)1 Do not change MTU at ESXlevel on running cluster

2 Reset back to the original value

ChangingMTU from 9000 to 1500 on oneESX node causes storfs process to restarton all nodes

CSCvm99150

Release Notes for Cisco HX Data Platform Release 4056

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)After the installation (for newdeployments) or after an upgradeto 401a (for existingdeployments) perform thefollowing steps on all ControllerVMs

1 Edit the following tune files (onall Controller VMs)

optspringpathconfiglfftunes

optspringpathconfigvsi_16tbtunes

2 SetcleanerEnableSegSummaryCleaning

to false

3 After editing the above tunefiles

a SSH in to all the ControllerVMs

b Run storfstool -- -Z

c Run the below command andcheck the values Tune valueshould be true

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=true

d Type below commands andapplied the tunes changesdynamically

echo false gttmpstprocfssystemtunecleanerEnableSegSummaryCleaning

e Verify that tune values arechanged Run below commandValue should be false

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=false

f umount tmpstprocfs

A cluster with 8TB or 12TB disk drivesmay experience IO stalls for severalminutes after another node fails in thecluster

CSCvp23718

Release Notes for Cisco HX Data Platform Release 4057

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)1 Update the conf file manuallyto change to the link local addressto

2 Restart scvmclient on thecontroller and ESX

HX 35(2b) FlexVol to HX 40 CSIupgrade does not work FlexVol continuesto work

CSCvp26319

35(2a)Disable rebalance until node healsthen reenable rebalance

7-node cluster healing very slowlybecause rebalance change AMS reqs areblocking healing process

CSCvp62512

35(2a)NAWith moderate cluster storage capacityusage under site failover cluster clustermay experience an all paths (APD) downcondition

CSCvq49412

40(1a)NAThis product includes Third-partySoftware that is affected by thevulnerabilities identified by the followingCommon Vulnerability and Exposures(CVE) IDs

CVE-2016-0762 CVE-2016-6797CVE-2016-6816 CVE-2016-8735CVE-2017-5647 CVE-2017-12615CVE-2017-12616 CVE-2017-12617CVE-2017-7674 CVE-2018-1304CVE-2018-8014 CVE-2018-1336CVE-2018-8034 CVE-2018-11784CVE-2019-0232

CSCvp36364

35(2d)NAstcli cluster info command needs toprovide UCSM VIP address Currently itshows ucsm-hostcom

CSCvq11456

35(2a)No workarounds currently Thestale entries do not affect the dayto day operations of HyperFlex

HyperFlex upgrade validation failedbecause cluster has extraneous stNodes ininternal database

CSCvq32530

35(2a)NAWhen not in maintenancemode all drivesappear in HX Connect and get updated asexpected

When in maintenance mode extra drivesare shown in HX Connect with no slotnumbers

CSCvq42378

35(2b)Contact TAC to cleanup duplicatenode

Nodes duplicated in HX Connect aftercluster recreate

CSCvq61825

Release Notes for Cisco HX Data Platform Release 4058

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)NACurrently HyperFlex installs on Hyper-Vdo not contain the stcli security whitelistcommandset

CSCvq66245

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

Release Notes for Cisco HX Data Platform Release 4059

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)[Hyper-V] Converged node expansionfails for incoming node with VIC 14xx

CSCvq39523

Release Notes for Cisco HX Data Platform Release 4060

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

1 Set Jumbo frame inWindowsHypervisor for incomingnode

Get-NetAdapter storage-data-a|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

Get-NetAdapter storage-data-b|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

2 Verify all interfaces are setcorrectly to support Jumbo frames

Get-NetAdapter storage |Get-NetAdapterAdvancedProperty| RegistryKeyword -matchjumbo | ft -auto

You should receive the followingmessage

Name DisplayName DisplayValue

RegistryKeyword RegistryValue

---- ----------- ------------

--------------- -------------

vswitch-hx-storage-data Jumbo

Packet 9014 Bytes

JumboPacket 9014

storage-data-b Jumbo Packet

Bytes 9014 JumboPacket

9014

storage-data-a Jumbo Packet

Bytes 9014 JumboPacket

9014

3 Reboot only the incoming nodeor the node being expanded

4 Retry Cluster Expansion in

Release Notes for Cisco HX Data Platform Release 4061

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

DescriptionCVEDefect IDRelease

Multiple Security Vulnerabilities

For more information see the relatedCisco Security Advisory

CVE-2014-8501 CVE-2014-9939CVE-2015-9262 CVE-2016-10087CVE-2016-2226 CVE-2016-4487CVE-2016-4488 CVE-2016-4489CVE-2016-4490 CVE-2016-4491CVE-2016-4492 CVE-2016-4493CVE-2016-6131 CVE-2018-10963CVE-2018-13785 CVE-2018-17100CVE-2018-17101 CVE-2018-18557CVE-2018-18661 CVE-2018-7456CVE-2018-8905

CSCvq0917840(2a)

Vulnerabilities associated withlibseccomp

For more information see the relatedCisco Security Advisory

CVE-2019-9893CSCvq0756840(2a)

Vulnerabilities associated withgdk-pixbuf

For more information see the relatedCisco Security Advisory

CVE-2017-12447CSCvq0675540(2a)

Vulnerabilities associated withpython-crypto

For more information see the relatedCisco Security Advisory

CVE-2018-6594CSCvp9381740(2a)

Vulnerabilities associated with wget

For more information see the relatedCisco Security Advisory

CVE-2018-20483 CVE-2019-5953CSCvp8672140(2a)

Multiple vulnerabilities associated withPython Crypto

For more information see the relatedCisco Security Advisory

CVE-2018-6594CSCvp6674840(2a)

Vulnerabilities associated withsensible-utils

For more information see the relatedCisco Security Advisory

CVE-2017-17512CSCvp6673440(2a)

Multiple vulnerabilities associated withPython

For more information see the relatedCisco Security Advisory

CVE-2018-1000030CSCvp6668940(2a)

Release Notes for Cisco HX Data Platform Release 4027

Security Fixes

DescriptionCVEDefect IDRelease

Multiple patch vulnerabilities identifiedby CVE-2016-10713

For more information see the relatedCisco Security Advisory

CVE-2016-10713CVE-2018-1000156 CVE-2018-6951

CSCvp6667240(2a)

Vulnerabilities associatedwith Little CMSidentified by CVE-2016-10165

For more information see the relatedCisco Security Advisory

CVE-2016-10165 CVE-2018-16435CSCvp6666440(2a)

Vulnerabilities associated with TenableScan identified byCVE-2011-5325CVE-2019-11068CVE-2016-7076CVE-2017-1000368

For more information see the relatedCisco Security Advisory

CVE-2011-5325 CVE-2016-7076CVE-2017-1000368CVE-2019-11068

CSCvp6474640(2a)

Vulnerabilities associated with libxml2294 and earlier as used in XMLSec1223 and earlier and other products andidentified by CVE-2016-9318

For more information see the relatedCisco Security Advisory

CVE-2016-9318 CVE-2017-16932CVE-2017-18258 CVE-2018-14404CVE-2018-14567

CSCvp3479240(2a)

Vulnerabilities associated with LFTPRemote File Names Unauthorized AccessVulnerability and identified byCVE-2018-10916

For more information see the relatedCisco Security Advisory

CVE-2018-10916CSCvp2926640(2a)

Vulnerabilities associated withCryptographyDeprecationWarning signerand verifier have been deprecated

For more information see the relatedCisco Security Advisory

CVE-2018-7750CSCvo3409740(2a)

Multiple Vulnerabilities in python python27 35

For more information see the relatedCisco Security Advisory

CVE-2019-16056CSCvr3979340(2a)

Release Notes for Cisco HX Data Platform Release 4028

Security Fixes

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toexecute commands as the root user

The vulnerability is due to insufficientinput validation An attacker could exploitthis vulnerability by connecting to thecluster service manager and injectingcommands into the bound process Asuccessful exploit could allow the attackerto run commands on the affected host asthe root user

Cisco has released software updates thataddress this vulnerability There areworkarounds that address thisvulnerability

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvq2417640(1b)35(2g)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

The vulnerability is due to an unprotectedlistening interface An attacker couldexploit this vulnerability by connectingto the listening interface and injectingcommands to the bound process Anexploit could allow the attacker to runcommands on the affected host as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvj9560640(1b)35(2g)

Release Notes for Cisco HX Data Platform Release 4029

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerabilities associated with theJVM18U121memory leak found duringVC alarms (concurrent 40 calls usingREST API)

CVE-2017-10053 CVE-2017-10067CVE-2017-10074 CVE-2017-10078CVE-2017-10081 CVE-2017-10087CVE-2017-10089 CVE-2017-10090CVE-2017-10096 CVE-2017-10101CVE-2017-10102 CVE-2017-10107CVE-2017-10108 CVE-2017-10109CVE-2017-10110 CVE-2017-10111CVE-2017-10115 CVE-2017-10116CVE-2017-10118 CVE-2017-10135CVE-2017-10176 CVE-2017-10193CVE-2017-10198 CVE-2017-10243CVE-2017-10274 CVE-2017-10281CVE-2017-10285 CVE-2017-10295CVE-2017-10345 CVE-2017-10346CVE-2017-10347 CVE-2017-10348CVE-2017-10349 CVE-2017-10350CVE-2017-10355 CVE-2017-10356CVE-2017-10357 CVE-2017-10388CVE-2017-3509 CVE-2017-3511CVE-2017-3526 CVE-2017-3533CVE-2017-3539 CVE-2017-3544CVE-2018-2579 CVE-2018-2582CVE-2018-2588 CVE-2018-2599CVE-2018-2602 CVE-2018-2603CVE-2018-2618 CVE-2018-2629CVE-2018-2633 CVE-2018-2634CVE-2018-2637 CVE-2018-2641CVE-2018-2663 CVE-2018-2677CVE-2018-2678 CVE-2018-2783CVE-2018-2790 CVE-2018-2794CVE-2018-2795 CVE-2018-2796CVE-2018-2797 CVE-2018-2798CVE-2018-2799 CVE-2018-2800CVE-2018-2814 CVE-2018-2815CVE-2018-2952 CVE-2018-3136CVE-2018-3139 CVE-2018-3149CVE-2018-3150 CVE-2018-3169CVE-2018-3180 CVE-2018-3183CVE-2018-3214 CVE-2019-2422

CSCvo8899740(1b)

Tomcat and Nginx logs are not beingcollected in the support bundle generatedthrough HX Connect in Release 35

NACSCvm5803140(1b)

The vulnerabilities associated with thelibmspack software package versionincluded in Cisco HX Data Platform

CVE-2018-18584

CVE-2018-18585

CSCvn3511940(1a)

Release Notes for Cisco HX Data Platform Release 4030

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerabilities associated withFasterXML Jackson-Databind TimeValueField Denial of Service

CVE-2018-14719

CVE-2018-14720

CVE-2018-1000873

CVE-2018-14721

CVE-2018-19360

CVE-2018-19362

CVE-2018-19361

CVE-2018-14718

CSCvn8228240(1a)

The vulnerabilities associated with theOpenSSL Protocol software packageversion included in Cisco HX DataPlatform

CVE-2013-3587CSCvo0505440(1a)

The vulnerabilities associated withThird-Party Software Denial of Service

CVE-2018-16487

CVE-2018-19361

CSCvo2781840(1a)

The vulnerabilities associated withLinuxkernel for Ubuntu 1710

CVE-2018-1092

CVE-2018-7492

CVE-2018-8087

CVE-2018-1068

CVE-2018-8781

CSCvm5314935(2a)

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)25(1d)stcli services dns remove should remove the DNS serverinfo from the interface files

CSCvs70967

40(2b)40(1b)HX 40(1b) new deployment from Intersight productioncloud

User receives following error while enabling CSI(Kubernetes integration)

Failure occurred during volume_access_enable Errorwas Nonzero exit code 1

CSCvt10522

40(2b)40(2a)Receive the following alertevent in HX Connect HXController VM HOSTNAME one or more configuredDNS servers not responding

CSCvt13947

Release Notes for Cisco HX Data Platform Release 4031

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)40(2a)API UI CLI show few Drive slots Empty after upgradeto 402a

The drives are seen in the sysmtool lsscsi and stclicommand outputs

The cluster is healthy with no errors

CSCvt14914

40(2b)35(2d)After successfully changing the DNS server on the HXcontroller we still can see the original DNS entry thatwas added during the deployment

CSCvs69154

40(2b)35(2e)HyperFlex Connect and VCenter show APD alarms forHX and non HX datastores This implies there is an issuewith HyperFlex when there may not be

CSCvs30080

40(2b)40(2a)An issue where all the disks in the node get locked afterrebooting the node

We successfully unlocked using sed-clientsh -Ucommand but wanted to test with another reboot anddrives locked again

CSCvs74286

40(2b)40(2a)When running stcli license commands on HyperFlexthe following error is seen

rootSpringpathControllertmp stcli license show all

Show smart licensing failed

Smart Agent is not ready please wait a minute and tryagain

CSCvt13929

40(2b)40(2a)After upgrading to HX 40(2a) the first RoboEdge nodethat gets rebooted then ignores persistent drivesPIDHX-HD24TB10K4KN

Cluster will be in degraded state and have reducedcapacity

CSCvt20144

40(2b)35(2b)Zookeeper fails to start while Exhibitor is runninghowever echo srvr returns nothing

CSCvs21562

40(2b)40(2a)When performing an HyperFlex upgrade a validationwarning may occur due to the host not having EnterprisePlus or Enterprise hypervisor licensing

Upgrade Validation Warning

ESXi host esx1labtest should be configured withVMware Enterprise license for upgrade to continue

CSCvs91787

Release Notes for Cisco HX Data Platform Release 4032

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)35(2g)Cluster expansion fails at Config Installer stage whenthe root and admin password for the storage controller(SCVM) are different

CSCvs69317

40(2b)35(2g)Panic while upgrading ESXiCSCvt06983

40(2b)40(2a)A cluster node running may hang in the Linux kernelThis is classified as an oops and a deviation from theexpected behavior

CSCvs54285

40(2b)35(2g)Rebalance failing on 10+10 node HX 35(2g) stretchcluster

CSCvs69007

40(2b)35(2d)The cluster becomes inaccessible to the IOVisorCSCvr54687

40(2b)40(2a)Panic on the storage controllerCSCvt61297

40(2b)40(2a)The size of support bundle is very large when collectedwith storfs-support command

CSCvt63306

40(2b)40(2a)storfs crash associated withFileSystemUsageWarningEvent orFileSystemUsageAlertEvent due to high usage ofvarstv above 80

CSCvt72807

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

40(2a)35(2d)In normal scenarios HX upgrade gets pendingACK afterupdating service profile In this case UCS only upgradewas stalled because after updating service profile HXupgrade was not getting pending ACK and it was alwayswaiting state

CSCvo39912

40(2a)35(1a)When using motherBoardReplace-12 to clean ZK fromold stNodepNodes resulted in unmounted datastore andmaking the size of the Hyperflex Datastores 0 resultingin all VMs in the cluster going offline

CSCvq39471

40(2a)26(1e)Upgrading cluster and getting error

Failed upgrade validations Checking vCenterconfiguration Reason Upgrade validations failed DRSFault Insufficient resources to satisfy configured failover

CSCvm77294

Release Notes for Cisco HX Data Platform Release 4033

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)The cluster expand fails on a node with DR replicationconfigured

When a HX cluster which has DR replication configuredis expanded we see the installer UI pulling in thereplicationVLAN information instead of themanagementVLAN information Even if we change that informationto the correct mgmt VLAN id and name it it does notseem to work as the node is configured with the VLANof the replicationVLAN in ESXi This leads to the failureof the node add with the host unreachable error

CSCvo70650

40(2a)35(1a)Customer reported that server didnt complete thefirmware upgrade automatically one by one

The user manually needs to put the host into maintenancemode and they manually acknowledged the pendingrequests to finish the UCS firmware upgrade

Maintenance policy is set to default (User-ack) as perthe design

CSCvo91624

40(2a)35(1i)When the cluster is in Failed state and stcli noderemove is attempted the output shows successful thoughthe node failed to get removed from the cluster

CSCvo93017

40(2a)35(1a)HyperFlex cluster upgrade may fail during validationwith a DRS Validation failed error

CSCvp31021

40(2a)35(2b)HX cluster expansion will fail with the following errormessage

MAC address pool configuration failure150[ErrorDescription] bad address block range definitioncollision

CSCvp58318

40(2a)35(2b)Memory usage by carbon cacheCSCvq34873

40(2a)35(2g)Upgrading ESXi cluster fails with error Nodemaintenance mode failed

CSCvr03240

40(2a)35(2e)storfs process does not automatically start on Exit HXMaintenance Mode or other tasks which restart StorageController

CSCvr88978

40(2a)35(2a)Post install script fails with message

Failed to execute ipmitool on HX node

CSCvp10707

40(2a)40(1a)HyperFlex expansionworkflow doesnt pull VLANnamecorrectly

CSCvp46539

Release Notes for Cisco HX Data Platform Release 4034

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)During HX cluster deployment - Validate ClusterCreation phase - HX Installer might fail with the error

fromvarlogspringpathstDeploylog

CSCvq45087

40(2a)35(2b)HX Installer fails to login to the SOL for configuringESX hosts

CSCvq91380

40(2a)35(2d)If DRS is enabled with configuration parameters otherthen fully automated ie manual or partially automatedthan this will change to fully automated during clusterexpansion

CSCvr44222

40(2a)35(1a)One of the validation task name has a typo The taskname is Validating node for mixed of SED and non-SEDdisks

It should be Validating node for mix of SED andnon-SED disks

CSCvp82175

40(2a)30(1i)This combination of using smaller NVMe drives is notsupported in expand operation For example if the drivesare only 375GB and they cannot be added to an existingcluster with larger caching SSD

CSCvo12359

40(2a)35(1a)HyperFlex includes a version of OpenSSL that is affectedby the vulnerabilities identified by the followingCommon Vulnerability and Exposures

CVE-2018-0495

CSCvp66679

40(2a)30(1e)SDK invoke exception when taking a native snapshot ona VM with GPU pci passthrough attached

CSCvo00511

40(2a)30(1i)Node replacement script fails due to EAM errorCSCvo62867

40(2a)35(1a)While pairing a cluster (HX release gt= 35) and cluster(HX release lt 35) remote replication network test failson the cluster (HX release lt 35)

CSCvo79760

40(2a)30(1b)Fixed the issue in the latest support-workflow bundleThis should not be hit anymore

CSCvo87061

40(2a)35(2a)On a 3 node cluster the MBreplace script fails as thescript looks for HEALTHY cluster status

CSCvo87080

40(2a)35(2a)MbReplace script (tar) hangs when running on 352b or40

CSCvp12359

Release Notes for Cisco HX Data Platform Release 4035

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)HX replication cleanup failing with error NA 35(2a) -INFODR state is not clean

CSCvp63958

40(2a)40(1a)The check_stig_parameters API incorrectly shows thecompliance state of the cluster as not STIG compliant

CSCvp66277

40(2a)35(2c)The snapshot creation on CBT enabled VM fails witherror Failed in vmreparent vmkfstools clone1

CSCvq06952

40(2a)35(2a)HyperFlex GUI when setting up remote DR-replicationpartnerpeer should show remote replication partner TCPport reachability test results

This will provide quick upfront connectivity results thatfrequently takes customer a while to manually check andconfirm

If reachability tests fail an info window and HX alertshould trigger that list the destination peer IP and thespecific port(s) that could not be reache

CSCvo60587

40(2a)35(2b)If you have more than one IP pool configured for areplication network and local replication network testsare failing you might be running into this issue

CSCvr67130

40(2a)35(2a)Skip task option in MBreplace and Software redeployscripts The scripts should have the option to skip tasksas user inputs and not by modifying the json file

CSCvp05204

40(2a)40(1a)Error noticed on SCVM Console print_req_error IOerror dev fd0 sector 0 on HyperV

It doesnt come up on SSH session

CSCvq34357

40(2a)30(1d)The VMNetwork switch is already created as part of theinstall process

Ideally the ip address assignment should be done duringthe initial install process as well No post install stepsshould be required altrnatively it could be done as partof a post install script

CSCvk36222

40(2a)25(1c)storfs service impacted during HX upgrade from HX21(1) to 25(1c)

HX node which is not being upgraded storfs servicestopped causing outage

CSCvg53223

40(2a)35(1b)HX Stretch Cluster Witness VM reverts to DHCP onreboot

CSCvp37536

40(2a)35(2b)HyperFlex Stretch Cluster Witness does not getprogrammed with an NTP server

CSCvp46578

Release Notes for Cisco HX Data Platform Release 4036

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2b)HyperFlex cluster does not heal after maintenance onnodeserver

CSCvr18528

40(2a)40(2a)High percentage of packet loss on the eth1 data interface9K MTU

CSCvr97089

40(2a)35(2f)The HyperFlex cluster healing state is stuck at 87 andis unable to progress

CSCvr16760

40(2a)40(2a)In some circumstances two replications can occur forthe same VM

From the GUI the replication status for this VM willshow stuck in an In Progress state but there are nojobs stuck in replication layer Subsequent Replicationsshould succeed

CSCvq22898

40(2a)35(2a)After an unexpected power outage the hyperflex clusteris unable to come up correctly

All processes are running time is synced and all nodescan vmkping each other as required

CSCvp13990

40(2a)40(1a)When specifying IP address instead of node ID stclinode maintenanceMode commands fail

CSCvp33657

40(2a)40(1a)2 node edge cluster if experiencing Intersightarbitratorconnectivity issues should NOT allowHXMaintenancemode to initiate as this will bring the cluster down

CSCvr31573

40(2a)35(2c)Old files in varsupportZKTxnlog are not purged withthe daily zklog-cleanup cron job

CSCvr89066

40(2a)35(2a)When a node is in MaintenanceMode any disk removalor replacement will be reflected in UI only after the nodeis brought back frommaintenancemode This is becausestorfs is not running on the node in maintenance andwill not be able to detect disk activities until it is broughtout of Maintenance Mode

CSCvo86431

40(2a)35(2c)Cluster might go offline and stop serving IO requestsrootcvm~ stcli cluster storage-summary--detailGet cluster storage summary failedjavanetConnectException Connection refused19216814210010207

CSCvr01645

40(2a)35(2d)A stretch cluster deployment fails with a messageFormatting nodes Node in Use on the Hyperflex installerand crmZoneType value is seen as 1 in storfscfg oroptspringpathconfigstretchtunes

CSCvq80340

Release Notes for Cisco HX Data Platform Release 4037

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)30(1c)During upgrade we upload the upgrade packagedownloaded from Ciscocom Upgrade packages are offile type tgz hxconnect acceptsstorfs-packages-401a-33028tar file a different type ofcompression and allows upgrade to start and fail whenit cannot decompress the file for bootstrap and scvmclientupgrade on ESXi

CSCvq63888

40(2a)40(2a)While attempting to install hyperflex via intersight thefollowing error is seen

Failed in Task Add hosts to vCenter Cluster with ErrorTry adding hosts manually to vCenter and retry Failedto get addressFailed to host by namelookupHostnamecompanycom on 000053 read udp12700145430-gt12700153 read connection refused

CSCvr52098

40(2a)35(2a)CoreAPI call may time out for cluster management APIwhile doing cluster create validation

CSCvo92952

40(2a)35(2d)Validation fails with Mixed mode expansion checkdue to Enhanced vMotion Compatibility Incompatibilitywhen the ESXi version is different

CSCvq95460

40(2a)30(1i)This failure is seen in installer when custom workflowis used hypervisor configuration + deploy (clean diskpartition=no) at Installing software packages on storagecontroller VM

CSCvr66309

40(2a)35(2d)The HX Installer apparently replaced a comma bysemicolon on the VLAN ID range while exporting theconfiguration to a JSON file

CSCvq93831

40(2a)40(1a)When using unsupported browsers HX Connect userswill not be able to login These users see an errorindicating unauthorized user

CSCvp22693

Release Notes for Cisco HX Data Platform Release 4038

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)40(1a)A vulnerability in the web-based management interfaceof Cisco HyperFlex Software could allow anunauthenticated remote attacker to conduct a cross-siterequest forgery (CSRF) attack on an affected system

The vulnerability is due to insufficient CSRF protectionsfor the web UI on an affected device An attacker couldexploit this vulnerability by persuading a user of theinterface to follow a malicious link A successful exploitcould allow the attacker to perform arbitrary actions withthe privilege level of the affected user

Cisco has released software updates that address thisvulnerability There are no workarounds that address thisvulnerability

This advisory is available at the following link

httpstoolsciscocomsecuritycentercontentCiscoSecurityAdvisorycisco-sa-20190807-hypflex-csrf

CSCvp24343

40(2a)35(2d)Add message to HX Connect Progress flow to notAcknowledge Pending Activities and reboot the serverson UCSM

HX Connect is doing a controlled rolling server upgradein the background

CSCvq22844

40(2a)40(1b)Compute Node expand operation fails due to incorrectcluster name

CSCvr43786

40(2a)26(1e)RF-2 cluster shutdown during data resynchronizationafter a node failure (non storfs) followed by multipledisk read failures and a HardBlacklist of a disk

CSCvp41241

40(2a)40(1a)HyperFlex Edge nodes do not properly set the ESXihostname during deployment

CSCvo13143

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4039

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4040

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1b) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401b-33133p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4041

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4042

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1b)40(1a)While running the HyperFlex installer with WindowsServer Hyper-V cluster creation process fails with theerror Failure occurred during ClusterCreation process Unable to post thecontent to the downstream This symptomis encountered while deploying HyperFlex cluster withHyperFlex nodes configuredordered with Cisco VIC1457 MLOM (PID HX-MLOM-C25Q-04) andWindows Server Datacenter or Core with Hyper-Vversion 2016 or 2019

CSCvp64140

Release Notes for Cisco HX Data Platform Release 4043

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)35(2b)Adding Micron 5200 drive to a cluster fails to increasecluster capacity

CSCvo69067

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4044

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4045

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1a) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401a-33028p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4046

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4047

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1a)30(1d)Cluster instability when disks of different sector sizeplaced in HX node

CSCvk17250

40(1a)

35(2c)

35(1a)When logged in using a local HX user account insteadof a Virtual Center account an error message appearsintermittently indicating Virtual Centerunreachable or Resource informationcannot be updated when VC is reachable

CSCvo36198

40(1a)

35(2b)

30(1d)HXDP does not work with EMC RecoverPoint - needsto support VMware API (FSS-Readdir)

CSCvk38003

Release Notes for Cisco HX Data Platform Release 4048

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)

35(2a)

30(1i)

25(1c)

25(1d)

Zookeeper (Exhibitor) process on HyperFlex StorageController VMs may fail to respond to requests whenvarzookeeper has low or no free space

CSCvm90352

40(1a)26(1c)Use Asynchronous consolidation for HX SnapshotsCSCvn02151

40(1a)35(2b)HX Quiesced Snapshot for Backup VendorsCSCvo90713

40(1a)35(2a)The cluster creationcluster expansion workflow stopswith the following error message at the validation step

FIRMWARE-Check UCSC-SAS-M5HD

FIRMWARE-Check UCSC-SAS-M5HD Required00000029000000320000003500000050 Found00000058

Action Needed Update the Controller Firmware toRequired Version

CSCvn17787

35(2b)30(1d)Kernel migration fails when a local datastore is searchedfor in the ESXi host

CSCvn73127

35(2a)26(1b)A node shuts down when two disks are replaced (acaching disk and another capacity disk) where thecapacity disk is inserted first and then the caching diskis inserted

CSCvk46364

Hyper-V

40(1a)35(2a)Cluster expansion may fail with an error code 500 -operation timed out

CSCvn28721

35(2b)35(2a)During upgrade remove the VLAN on team created foruser vSwitch During fresh installation only one VLANtag is set to the vSwitch and team although multipleVLANs were entered

CSCvn54300

35(2b)35(2a)While upgrading a Hyper-V cluster on account of a rarerace condition between the stUpgradeService andZookeeper servers the upgrade orchestration throws anupgrade validation error and the upgrade process isaborted

CSCvn60486

Release Notes for Cisco HX Data Platform Release 4049

Resolved Caveats in Release 40(1a)

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Startrestart EAM service invCenter

When EAM service is not runningvCenter registration is failing

CSCvt55712

35(2a)Contact Cisco TAC if you believeyou are seeing similar conditions

HyperFlex Stretch Cluster saw poorapplication performance due to writelatency cluster remained unhealthy andrebalance was stuck

This defect is currently being used forinvestigative purposes

CSCvt45344

35(2g)Remove log files fromvarlogspringpath folder when thelog files are not getting rotatedStop the relevant service removelarge log files fromvarlogspringpath folder andrestart the service

varstv folder in Controller VM maybecome full

CSCvt36374

40(2b)Restarting the stMgr will initializeit again

In a Cluster configured with DR networkStMgr may not get initialized and stuckin a deadlock while enabling the IPTablerules (can be verified from the stMgrlog)

CSCvt89709

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Click recover for the selectedVM again under the standalonemode This will move the VM torecovered state

When the user recovers a VM protectedin a group it along with other VMs in thegroup move to recovering state instandalonemode The selectedVM shouldmove to recovered state

CSCvs75553

35(2e)Clean up the fail over cluster andthen recreate the fail over clusterNo need to touch the HX storagecluster

Hyper-V When replicated DC was usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2d)40(1a)

Keep cluster utilization as low aspossible To avoid read-only statekeep below 92

HyperFlex clusters storage utilization isabove 92 usable capacity

CSCvr20922

Release Notes for Cisco HX Data Platform Release 4050

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1b)NAAll the disks in the node got locked afterrebooting the node

We successfully unlocked usingsed-clientsh -U command but wantedto test with another reboot and driveslocked again

CSCvs74286

35(2g)To recover the VM copy over thedata disks and attach them to thenew VM

After enable encryption HX connectencryption status showing Caution butall of the disks enable encryptionsuccessful

-Self Encrypting Drives Service isrunning on all of the nodes

-There is no error message of encryption

- USB0 interface is up

-All SED disks showing supported 1enabled 1 locked 0

CSCvs93245

40(1b)Enable kernelpanic_on_oops as apersistent configuration This willcause the node to panic and rebootimmediately

A cluster node running HX release40(1b) may hang in the Linux kernelThis is classified as an oops and adeviation from the expected behavior

CSCvs54285

Release Notes for Cisco HX Data Platform Release 4051

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Enabling K8iscsi stack in Hx clusterCSCvs41324

Release Notes for Cisco HX Data Platform Release 4052

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Steps to enable K8 on HX cluster

Perform following steps on ALLthe controller VMs

1 Updateetcinitscvmclientconf toenable tune iscsiEnable totrue

sed -iesiscsiEnable=falseiscsiEnable=trueetcinitscvmclientconf

2 Run following initctlcommand to reloadconfiguration

initctl reload-configuration

3 Restart scvmclient process

stop scvmclient startscvmclient

4 Run following initctlcommand to mount SYSTEMDS

initctl emit --no-waitsystem-datastore-created

5 Verify that iscsiEnable tuneis set to true using thefollowing command

ps -eaf | grep scvmclient |grep -v grep

root 6241 1 1 Dec06 014007optspringpathstorfs-corescvmclient-T logEnabled=false -TlogSyslogEnabled=true -TlogEchoToScreen=false -TstatLoggingToFile=false -TstatLoggingToSyslog=true-TlogDir=varlogspringpath-TnfsBackendServerList=1010748100-T iscsiEnable=true -TiscsiUseAsync=true -T

Release Notes for Cisco HX Data Platform Release 4053

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

iscsiConfigFilePath=nfsSYSTEMistgtconfps -eaf | grep scvmclient |grep -v grep

35(2e)Following procedure can be usedto check NFS Queue Depth andincrease if needed

rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth256 lt- Low value[rootHXESXI1] vsish -e setvmkModulesnfsclientmnt[DATASTORE]propertiesmaxQDepth 1024[rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth1024 lt- Optimalvalue

This requires an ESXi host rebootto take effect

Please place the node in HyperflexMaintenanceMode and gracefullyreboot the node for the changes tobe applied

HyperFlex Datastore NFS Queue Depthshows as 256 which can lead toperformance (including latency) issues

CSCvr83056

40(1b)This is cosmetic The error will goaway once the cluster returns tohealthy status

When a single node is offline in clusterstcli cluster storage-summary shows twonodes unavailable rootSCVM~ stclicluster storage-summary

CSCvs47735

35(2g)Modify both the root and adminpassword for the SCVMs to be thesame

stcli security password set -uadmin

Cluster expansion fails at Config Installerstage when the root and admin passwordfor the storage controller (SCVM) aredifferent

CSCvs69317

35(2d)NAAfter successfully changing the DNSserver on the HX controller we still cansee the original DNS entry that was addedduring the deployment

CSCvs69154

Release Notes for Cisco HX Data Platform Release 4054

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Customers are strongly encouragedto work with TAC in order toidentify and remediate this issue

1 Change getcluster from 2m to8m inoptspringpathstorfs-mgmtstMgr-10confapplicationconfon all nodes

2 Restart stMgr on all nodes

You may see this error message after afailed upgrade or other task such asattempting to enter a node into HXmaintenance mode

CSCvs53555

40(2a)Run the following command oneach storage controller VM as root

grep -i monitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp sed -iesmonitor_dns_serverstruemonitor_dns_serversfalseoptspringpathhx-diag-toolswatchdog_configjsonampamp grep -imonitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp restart watchdog

Receive the following alertevent in HXConnect HX Controller VMHOSTNAME one or more configuredDNS servers not responding

CSCvt13947

35(2b)Delete any empty (Size 0) log filesunder varlogzookeeperversion-2ORvarlogzookeeperstandaloneversion-2

Zookeeper fails to start while Exhibitor isrunning however echo srvr returnsnothing

CSCvs21562

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

40(4e)1 De-commission the server

2 Power drain the server -REMOVE BOTH POWERCORDS ON THE BACK OFTHE SERVER FOR 10SECONDS THENREINSERTPOWERCORDS

3 Re-commission the server

M2 boot disk is missing from serverinventory after upgrade to server firmware40(4e) As a result server fails to boot toOS installed in the M2 disk The issuepersists after re-acknowledgement as wellas de-commission andre-acknowledgement of the server

CSCvs02466

Release Notes for Cisco HX Data Platform Release 4055

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Re-start the flusher manuallyMany disks taken offline by a fault ormanually resulting in ENOSPC oncluster

CSCvq32721

35(2e)Clean up fail over cluster and thenrecreate the fail over cluster Thereis no need to touch the HX storagecluster

[Hyper-V]When replicated DCwas usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2b)Manually install upgradescvmclient VIB andmake sure thatit matches with HXDP version

Upgrade from release HX 35(2a) to35(2x) 4X does not upgrade scvmclientRecovery point may not work

CSCvq54992

Management

30(1b)To recover the VM copy over thedata disks and attach them to thenew VM

[HyperV]VMshows up onmultiple nodesCSCvj22992

35(1a)1 Do not change MTU at ESXlevel on running cluster

2 Reset back to the original value

ChangingMTU from 9000 to 1500 on oneESX node causes storfs process to restarton all nodes

CSCvm99150

Release Notes for Cisco HX Data Platform Release 4056

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)After the installation (for newdeployments) or after an upgradeto 401a (for existingdeployments) perform thefollowing steps on all ControllerVMs

1 Edit the following tune files (onall Controller VMs)

optspringpathconfiglfftunes

optspringpathconfigvsi_16tbtunes

2 SetcleanerEnableSegSummaryCleaning

to false

3 After editing the above tunefiles

a SSH in to all the ControllerVMs

b Run storfstool -- -Z

c Run the below command andcheck the values Tune valueshould be true

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=true

d Type below commands andapplied the tunes changesdynamically

echo false gttmpstprocfssystemtunecleanerEnableSegSummaryCleaning

e Verify that tune values arechanged Run below commandValue should be false

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=false

f umount tmpstprocfs

A cluster with 8TB or 12TB disk drivesmay experience IO stalls for severalminutes after another node fails in thecluster

CSCvp23718

Release Notes for Cisco HX Data Platform Release 4057

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)1 Update the conf file manuallyto change to the link local addressto

2 Restart scvmclient on thecontroller and ESX

HX 35(2b) FlexVol to HX 40 CSIupgrade does not work FlexVol continuesto work

CSCvp26319

35(2a)Disable rebalance until node healsthen reenable rebalance

7-node cluster healing very slowlybecause rebalance change AMS reqs areblocking healing process

CSCvp62512

35(2a)NAWith moderate cluster storage capacityusage under site failover cluster clustermay experience an all paths (APD) downcondition

CSCvq49412

40(1a)NAThis product includes Third-partySoftware that is affected by thevulnerabilities identified by the followingCommon Vulnerability and Exposures(CVE) IDs

CVE-2016-0762 CVE-2016-6797CVE-2016-6816 CVE-2016-8735CVE-2017-5647 CVE-2017-12615CVE-2017-12616 CVE-2017-12617CVE-2017-7674 CVE-2018-1304CVE-2018-8014 CVE-2018-1336CVE-2018-8034 CVE-2018-11784CVE-2019-0232

CSCvp36364

35(2d)NAstcli cluster info command needs toprovide UCSM VIP address Currently itshows ucsm-hostcom

CSCvq11456

35(2a)No workarounds currently Thestale entries do not affect the dayto day operations of HyperFlex

HyperFlex upgrade validation failedbecause cluster has extraneous stNodes ininternal database

CSCvq32530

35(2a)NAWhen not in maintenancemode all drivesappear in HX Connect and get updated asexpected

When in maintenance mode extra drivesare shown in HX Connect with no slotnumbers

CSCvq42378

35(2b)Contact TAC to cleanup duplicatenode

Nodes duplicated in HX Connect aftercluster recreate

CSCvq61825

Release Notes for Cisco HX Data Platform Release 4058

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)NACurrently HyperFlex installs on Hyper-Vdo not contain the stcli security whitelistcommandset

CSCvq66245

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

Release Notes for Cisco HX Data Platform Release 4059

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)[Hyper-V] Converged node expansionfails for incoming node with VIC 14xx

CSCvq39523

Release Notes for Cisco HX Data Platform Release 4060

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

1 Set Jumbo frame inWindowsHypervisor for incomingnode

Get-NetAdapter storage-data-a|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

Get-NetAdapter storage-data-b|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

2 Verify all interfaces are setcorrectly to support Jumbo frames

Get-NetAdapter storage |Get-NetAdapterAdvancedProperty| RegistryKeyword -matchjumbo | ft -auto

You should receive the followingmessage

Name DisplayName DisplayValue

RegistryKeyword RegistryValue

---- ----------- ------------

--------------- -------------

vswitch-hx-storage-data Jumbo

Packet 9014 Bytes

JumboPacket 9014

storage-data-b Jumbo Packet

Bytes 9014 JumboPacket

9014

storage-data-a Jumbo Packet

Bytes 9014 JumboPacket

9014

3 Reboot only the incoming nodeor the node being expanded

4 Retry Cluster Expansion in

Release Notes for Cisco HX Data Platform Release 4061

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

DescriptionCVEDefect IDRelease

Multiple patch vulnerabilities identifiedby CVE-2016-10713

For more information see the relatedCisco Security Advisory

CVE-2016-10713CVE-2018-1000156 CVE-2018-6951

CSCvp6667240(2a)

Vulnerabilities associatedwith Little CMSidentified by CVE-2016-10165

For more information see the relatedCisco Security Advisory

CVE-2016-10165 CVE-2018-16435CSCvp6666440(2a)

Vulnerabilities associated with TenableScan identified byCVE-2011-5325CVE-2019-11068CVE-2016-7076CVE-2017-1000368

For more information see the relatedCisco Security Advisory

CVE-2011-5325 CVE-2016-7076CVE-2017-1000368CVE-2019-11068

CSCvp6474640(2a)

Vulnerabilities associated with libxml2294 and earlier as used in XMLSec1223 and earlier and other products andidentified by CVE-2016-9318

For more information see the relatedCisco Security Advisory

CVE-2016-9318 CVE-2017-16932CVE-2017-18258 CVE-2018-14404CVE-2018-14567

CSCvp3479240(2a)

Vulnerabilities associated with LFTPRemote File Names Unauthorized AccessVulnerability and identified byCVE-2018-10916

For more information see the relatedCisco Security Advisory

CVE-2018-10916CSCvp2926640(2a)

Vulnerabilities associated withCryptographyDeprecationWarning signerand verifier have been deprecated

For more information see the relatedCisco Security Advisory

CVE-2018-7750CSCvo3409740(2a)

Multiple Vulnerabilities in python python27 35

For more information see the relatedCisco Security Advisory

CVE-2019-16056CSCvr3979340(2a)

Release Notes for Cisco HX Data Platform Release 4028

Security Fixes

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toexecute commands as the root user

The vulnerability is due to insufficientinput validation An attacker could exploitthis vulnerability by connecting to thecluster service manager and injectingcommands into the bound process Asuccessful exploit could allow the attackerto run commands on the affected host asthe root user

Cisco has released software updates thataddress this vulnerability There areworkarounds that address thisvulnerability

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvq2417640(1b)35(2g)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

The vulnerability is due to an unprotectedlistening interface An attacker couldexploit this vulnerability by connectingto the listening interface and injectingcommands to the bound process Anexploit could allow the attacker to runcommands on the affected host as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvj9560640(1b)35(2g)

Release Notes for Cisco HX Data Platform Release 4029

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerabilities associated with theJVM18U121memory leak found duringVC alarms (concurrent 40 calls usingREST API)

CVE-2017-10053 CVE-2017-10067CVE-2017-10074 CVE-2017-10078CVE-2017-10081 CVE-2017-10087CVE-2017-10089 CVE-2017-10090CVE-2017-10096 CVE-2017-10101CVE-2017-10102 CVE-2017-10107CVE-2017-10108 CVE-2017-10109CVE-2017-10110 CVE-2017-10111CVE-2017-10115 CVE-2017-10116CVE-2017-10118 CVE-2017-10135CVE-2017-10176 CVE-2017-10193CVE-2017-10198 CVE-2017-10243CVE-2017-10274 CVE-2017-10281CVE-2017-10285 CVE-2017-10295CVE-2017-10345 CVE-2017-10346CVE-2017-10347 CVE-2017-10348CVE-2017-10349 CVE-2017-10350CVE-2017-10355 CVE-2017-10356CVE-2017-10357 CVE-2017-10388CVE-2017-3509 CVE-2017-3511CVE-2017-3526 CVE-2017-3533CVE-2017-3539 CVE-2017-3544CVE-2018-2579 CVE-2018-2582CVE-2018-2588 CVE-2018-2599CVE-2018-2602 CVE-2018-2603CVE-2018-2618 CVE-2018-2629CVE-2018-2633 CVE-2018-2634CVE-2018-2637 CVE-2018-2641CVE-2018-2663 CVE-2018-2677CVE-2018-2678 CVE-2018-2783CVE-2018-2790 CVE-2018-2794CVE-2018-2795 CVE-2018-2796CVE-2018-2797 CVE-2018-2798CVE-2018-2799 CVE-2018-2800CVE-2018-2814 CVE-2018-2815CVE-2018-2952 CVE-2018-3136CVE-2018-3139 CVE-2018-3149CVE-2018-3150 CVE-2018-3169CVE-2018-3180 CVE-2018-3183CVE-2018-3214 CVE-2019-2422

CSCvo8899740(1b)

Tomcat and Nginx logs are not beingcollected in the support bundle generatedthrough HX Connect in Release 35

NACSCvm5803140(1b)

The vulnerabilities associated with thelibmspack software package versionincluded in Cisco HX Data Platform

CVE-2018-18584

CVE-2018-18585

CSCvn3511940(1a)

Release Notes for Cisco HX Data Platform Release 4030

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerabilities associated withFasterXML Jackson-Databind TimeValueField Denial of Service

CVE-2018-14719

CVE-2018-14720

CVE-2018-1000873

CVE-2018-14721

CVE-2018-19360

CVE-2018-19362

CVE-2018-19361

CVE-2018-14718

CSCvn8228240(1a)

The vulnerabilities associated with theOpenSSL Protocol software packageversion included in Cisco HX DataPlatform

CVE-2013-3587CSCvo0505440(1a)

The vulnerabilities associated withThird-Party Software Denial of Service

CVE-2018-16487

CVE-2018-19361

CSCvo2781840(1a)

The vulnerabilities associated withLinuxkernel for Ubuntu 1710

CVE-2018-1092

CVE-2018-7492

CVE-2018-8087

CVE-2018-1068

CVE-2018-8781

CSCvm5314935(2a)

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)25(1d)stcli services dns remove should remove the DNS serverinfo from the interface files

CSCvs70967

40(2b)40(1b)HX 40(1b) new deployment from Intersight productioncloud

User receives following error while enabling CSI(Kubernetes integration)

Failure occurred during volume_access_enable Errorwas Nonzero exit code 1

CSCvt10522

40(2b)40(2a)Receive the following alertevent in HX Connect HXController VM HOSTNAME one or more configuredDNS servers not responding

CSCvt13947

Release Notes for Cisco HX Data Platform Release 4031

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)40(2a)API UI CLI show few Drive slots Empty after upgradeto 402a

The drives are seen in the sysmtool lsscsi and stclicommand outputs

The cluster is healthy with no errors

CSCvt14914

40(2b)35(2d)After successfully changing the DNS server on the HXcontroller we still can see the original DNS entry thatwas added during the deployment

CSCvs69154

40(2b)35(2e)HyperFlex Connect and VCenter show APD alarms forHX and non HX datastores This implies there is an issuewith HyperFlex when there may not be

CSCvs30080

40(2b)40(2a)An issue where all the disks in the node get locked afterrebooting the node

We successfully unlocked using sed-clientsh -Ucommand but wanted to test with another reboot anddrives locked again

CSCvs74286

40(2b)40(2a)When running stcli license commands on HyperFlexthe following error is seen

rootSpringpathControllertmp stcli license show all

Show smart licensing failed

Smart Agent is not ready please wait a minute and tryagain

CSCvt13929

40(2b)40(2a)After upgrading to HX 40(2a) the first RoboEdge nodethat gets rebooted then ignores persistent drivesPIDHX-HD24TB10K4KN

Cluster will be in degraded state and have reducedcapacity

CSCvt20144

40(2b)35(2b)Zookeeper fails to start while Exhibitor is runninghowever echo srvr returns nothing

CSCvs21562

40(2b)40(2a)When performing an HyperFlex upgrade a validationwarning may occur due to the host not having EnterprisePlus or Enterprise hypervisor licensing

Upgrade Validation Warning

ESXi host esx1labtest should be configured withVMware Enterprise license for upgrade to continue

CSCvs91787

Release Notes for Cisco HX Data Platform Release 4032

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)35(2g)Cluster expansion fails at Config Installer stage whenthe root and admin password for the storage controller(SCVM) are different

CSCvs69317

40(2b)35(2g)Panic while upgrading ESXiCSCvt06983

40(2b)40(2a)A cluster node running may hang in the Linux kernelThis is classified as an oops and a deviation from theexpected behavior

CSCvs54285

40(2b)35(2g)Rebalance failing on 10+10 node HX 35(2g) stretchcluster

CSCvs69007

40(2b)35(2d)The cluster becomes inaccessible to the IOVisorCSCvr54687

40(2b)40(2a)Panic on the storage controllerCSCvt61297

40(2b)40(2a)The size of support bundle is very large when collectedwith storfs-support command

CSCvt63306

40(2b)40(2a)storfs crash associated withFileSystemUsageWarningEvent orFileSystemUsageAlertEvent due to high usage ofvarstv above 80

CSCvt72807

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

40(2a)35(2d)In normal scenarios HX upgrade gets pendingACK afterupdating service profile In this case UCS only upgradewas stalled because after updating service profile HXupgrade was not getting pending ACK and it was alwayswaiting state

CSCvo39912

40(2a)35(1a)When using motherBoardReplace-12 to clean ZK fromold stNodepNodes resulted in unmounted datastore andmaking the size of the Hyperflex Datastores 0 resultingin all VMs in the cluster going offline

CSCvq39471

40(2a)26(1e)Upgrading cluster and getting error

Failed upgrade validations Checking vCenterconfiguration Reason Upgrade validations failed DRSFault Insufficient resources to satisfy configured failover

CSCvm77294

Release Notes for Cisco HX Data Platform Release 4033

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)The cluster expand fails on a node with DR replicationconfigured

When a HX cluster which has DR replication configuredis expanded we see the installer UI pulling in thereplicationVLAN information instead of themanagementVLAN information Even if we change that informationto the correct mgmt VLAN id and name it it does notseem to work as the node is configured with the VLANof the replicationVLAN in ESXi This leads to the failureof the node add with the host unreachable error

CSCvo70650

40(2a)35(1a)Customer reported that server didnt complete thefirmware upgrade automatically one by one

The user manually needs to put the host into maintenancemode and they manually acknowledged the pendingrequests to finish the UCS firmware upgrade

Maintenance policy is set to default (User-ack) as perthe design

CSCvo91624

40(2a)35(1i)When the cluster is in Failed state and stcli noderemove is attempted the output shows successful thoughthe node failed to get removed from the cluster

CSCvo93017

40(2a)35(1a)HyperFlex cluster upgrade may fail during validationwith a DRS Validation failed error

CSCvp31021

40(2a)35(2b)HX cluster expansion will fail with the following errormessage

MAC address pool configuration failure150[ErrorDescription] bad address block range definitioncollision

CSCvp58318

40(2a)35(2b)Memory usage by carbon cacheCSCvq34873

40(2a)35(2g)Upgrading ESXi cluster fails with error Nodemaintenance mode failed

CSCvr03240

40(2a)35(2e)storfs process does not automatically start on Exit HXMaintenance Mode or other tasks which restart StorageController

CSCvr88978

40(2a)35(2a)Post install script fails with message

Failed to execute ipmitool on HX node

CSCvp10707

40(2a)40(1a)HyperFlex expansionworkflow doesnt pull VLANnamecorrectly

CSCvp46539

Release Notes for Cisco HX Data Platform Release 4034

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)During HX cluster deployment - Validate ClusterCreation phase - HX Installer might fail with the error

fromvarlogspringpathstDeploylog

CSCvq45087

40(2a)35(2b)HX Installer fails to login to the SOL for configuringESX hosts

CSCvq91380

40(2a)35(2d)If DRS is enabled with configuration parameters otherthen fully automated ie manual or partially automatedthan this will change to fully automated during clusterexpansion

CSCvr44222

40(2a)35(1a)One of the validation task name has a typo The taskname is Validating node for mixed of SED and non-SEDdisks

It should be Validating node for mix of SED andnon-SED disks

CSCvp82175

40(2a)30(1i)This combination of using smaller NVMe drives is notsupported in expand operation For example if the drivesare only 375GB and they cannot be added to an existingcluster with larger caching SSD

CSCvo12359

40(2a)35(1a)HyperFlex includes a version of OpenSSL that is affectedby the vulnerabilities identified by the followingCommon Vulnerability and Exposures

CVE-2018-0495

CSCvp66679

40(2a)30(1e)SDK invoke exception when taking a native snapshot ona VM with GPU pci passthrough attached

CSCvo00511

40(2a)30(1i)Node replacement script fails due to EAM errorCSCvo62867

40(2a)35(1a)While pairing a cluster (HX release gt= 35) and cluster(HX release lt 35) remote replication network test failson the cluster (HX release lt 35)

CSCvo79760

40(2a)30(1b)Fixed the issue in the latest support-workflow bundleThis should not be hit anymore

CSCvo87061

40(2a)35(2a)On a 3 node cluster the MBreplace script fails as thescript looks for HEALTHY cluster status

CSCvo87080

40(2a)35(2a)MbReplace script (tar) hangs when running on 352b or40

CSCvp12359

Release Notes for Cisco HX Data Platform Release 4035

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)HX replication cleanup failing with error NA 35(2a) -INFODR state is not clean

CSCvp63958

40(2a)40(1a)The check_stig_parameters API incorrectly shows thecompliance state of the cluster as not STIG compliant

CSCvp66277

40(2a)35(2c)The snapshot creation on CBT enabled VM fails witherror Failed in vmreparent vmkfstools clone1

CSCvq06952

40(2a)35(2a)HyperFlex GUI when setting up remote DR-replicationpartnerpeer should show remote replication partner TCPport reachability test results

This will provide quick upfront connectivity results thatfrequently takes customer a while to manually check andconfirm

If reachability tests fail an info window and HX alertshould trigger that list the destination peer IP and thespecific port(s) that could not be reache

CSCvo60587

40(2a)35(2b)If you have more than one IP pool configured for areplication network and local replication network testsare failing you might be running into this issue

CSCvr67130

40(2a)35(2a)Skip task option in MBreplace and Software redeployscripts The scripts should have the option to skip tasksas user inputs and not by modifying the json file

CSCvp05204

40(2a)40(1a)Error noticed on SCVM Console print_req_error IOerror dev fd0 sector 0 on HyperV

It doesnt come up on SSH session

CSCvq34357

40(2a)30(1d)The VMNetwork switch is already created as part of theinstall process

Ideally the ip address assignment should be done duringthe initial install process as well No post install stepsshould be required altrnatively it could be done as partof a post install script

CSCvk36222

40(2a)25(1c)storfs service impacted during HX upgrade from HX21(1) to 25(1c)

HX node which is not being upgraded storfs servicestopped causing outage

CSCvg53223

40(2a)35(1b)HX Stretch Cluster Witness VM reverts to DHCP onreboot

CSCvp37536

40(2a)35(2b)HyperFlex Stretch Cluster Witness does not getprogrammed with an NTP server

CSCvp46578

Release Notes for Cisco HX Data Platform Release 4036

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2b)HyperFlex cluster does not heal after maintenance onnodeserver

CSCvr18528

40(2a)40(2a)High percentage of packet loss on the eth1 data interface9K MTU

CSCvr97089

40(2a)35(2f)The HyperFlex cluster healing state is stuck at 87 andis unable to progress

CSCvr16760

40(2a)40(2a)In some circumstances two replications can occur forthe same VM

From the GUI the replication status for this VM willshow stuck in an In Progress state but there are nojobs stuck in replication layer Subsequent Replicationsshould succeed

CSCvq22898

40(2a)35(2a)After an unexpected power outage the hyperflex clusteris unable to come up correctly

All processes are running time is synced and all nodescan vmkping each other as required

CSCvp13990

40(2a)40(1a)When specifying IP address instead of node ID stclinode maintenanceMode commands fail

CSCvp33657

40(2a)40(1a)2 node edge cluster if experiencing Intersightarbitratorconnectivity issues should NOT allowHXMaintenancemode to initiate as this will bring the cluster down

CSCvr31573

40(2a)35(2c)Old files in varsupportZKTxnlog are not purged withthe daily zklog-cleanup cron job

CSCvr89066

40(2a)35(2a)When a node is in MaintenanceMode any disk removalor replacement will be reflected in UI only after the nodeis brought back frommaintenancemode This is becausestorfs is not running on the node in maintenance andwill not be able to detect disk activities until it is broughtout of Maintenance Mode

CSCvo86431

40(2a)35(2c)Cluster might go offline and stop serving IO requestsrootcvm~ stcli cluster storage-summary--detailGet cluster storage summary failedjavanetConnectException Connection refused19216814210010207

CSCvr01645

40(2a)35(2d)A stretch cluster deployment fails with a messageFormatting nodes Node in Use on the Hyperflex installerand crmZoneType value is seen as 1 in storfscfg oroptspringpathconfigstretchtunes

CSCvq80340

Release Notes for Cisco HX Data Platform Release 4037

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)30(1c)During upgrade we upload the upgrade packagedownloaded from Ciscocom Upgrade packages are offile type tgz hxconnect acceptsstorfs-packages-401a-33028tar file a different type ofcompression and allows upgrade to start and fail whenit cannot decompress the file for bootstrap and scvmclientupgrade on ESXi

CSCvq63888

40(2a)40(2a)While attempting to install hyperflex via intersight thefollowing error is seen

Failed in Task Add hosts to vCenter Cluster with ErrorTry adding hosts manually to vCenter and retry Failedto get addressFailed to host by namelookupHostnamecompanycom on 000053 read udp12700145430-gt12700153 read connection refused

CSCvr52098

40(2a)35(2a)CoreAPI call may time out for cluster management APIwhile doing cluster create validation

CSCvo92952

40(2a)35(2d)Validation fails with Mixed mode expansion checkdue to Enhanced vMotion Compatibility Incompatibilitywhen the ESXi version is different

CSCvq95460

40(2a)30(1i)This failure is seen in installer when custom workflowis used hypervisor configuration + deploy (clean diskpartition=no) at Installing software packages on storagecontroller VM

CSCvr66309

40(2a)35(2d)The HX Installer apparently replaced a comma bysemicolon on the VLAN ID range while exporting theconfiguration to a JSON file

CSCvq93831

40(2a)40(1a)When using unsupported browsers HX Connect userswill not be able to login These users see an errorindicating unauthorized user

CSCvp22693

Release Notes for Cisco HX Data Platform Release 4038

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)40(1a)A vulnerability in the web-based management interfaceof Cisco HyperFlex Software could allow anunauthenticated remote attacker to conduct a cross-siterequest forgery (CSRF) attack on an affected system

The vulnerability is due to insufficient CSRF protectionsfor the web UI on an affected device An attacker couldexploit this vulnerability by persuading a user of theinterface to follow a malicious link A successful exploitcould allow the attacker to perform arbitrary actions withthe privilege level of the affected user

Cisco has released software updates that address thisvulnerability There are no workarounds that address thisvulnerability

This advisory is available at the following link

httpstoolsciscocomsecuritycentercontentCiscoSecurityAdvisorycisco-sa-20190807-hypflex-csrf

CSCvp24343

40(2a)35(2d)Add message to HX Connect Progress flow to notAcknowledge Pending Activities and reboot the serverson UCSM

HX Connect is doing a controlled rolling server upgradein the background

CSCvq22844

40(2a)40(1b)Compute Node expand operation fails due to incorrectcluster name

CSCvr43786

40(2a)26(1e)RF-2 cluster shutdown during data resynchronizationafter a node failure (non storfs) followed by multipledisk read failures and a HardBlacklist of a disk

CSCvp41241

40(2a)40(1a)HyperFlex Edge nodes do not properly set the ESXihostname during deployment

CSCvo13143

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4039

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4040

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1b) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401b-33133p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4041

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4042

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1b)40(1a)While running the HyperFlex installer with WindowsServer Hyper-V cluster creation process fails with theerror Failure occurred during ClusterCreation process Unable to post thecontent to the downstream This symptomis encountered while deploying HyperFlex cluster withHyperFlex nodes configuredordered with Cisco VIC1457 MLOM (PID HX-MLOM-C25Q-04) andWindows Server Datacenter or Core with Hyper-Vversion 2016 or 2019

CSCvp64140

Release Notes for Cisco HX Data Platform Release 4043

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)35(2b)Adding Micron 5200 drive to a cluster fails to increasecluster capacity

CSCvo69067

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4044

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4045

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1a) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401a-33028p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4046

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4047

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1a)30(1d)Cluster instability when disks of different sector sizeplaced in HX node

CSCvk17250

40(1a)

35(2c)

35(1a)When logged in using a local HX user account insteadof a Virtual Center account an error message appearsintermittently indicating Virtual Centerunreachable or Resource informationcannot be updated when VC is reachable

CSCvo36198

40(1a)

35(2b)

30(1d)HXDP does not work with EMC RecoverPoint - needsto support VMware API (FSS-Readdir)

CSCvk38003

Release Notes for Cisco HX Data Platform Release 4048

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)

35(2a)

30(1i)

25(1c)

25(1d)

Zookeeper (Exhibitor) process on HyperFlex StorageController VMs may fail to respond to requests whenvarzookeeper has low or no free space

CSCvm90352

40(1a)26(1c)Use Asynchronous consolidation for HX SnapshotsCSCvn02151

40(1a)35(2b)HX Quiesced Snapshot for Backup VendorsCSCvo90713

40(1a)35(2a)The cluster creationcluster expansion workflow stopswith the following error message at the validation step

FIRMWARE-Check UCSC-SAS-M5HD

FIRMWARE-Check UCSC-SAS-M5HD Required00000029000000320000003500000050 Found00000058

Action Needed Update the Controller Firmware toRequired Version

CSCvn17787

35(2b)30(1d)Kernel migration fails when a local datastore is searchedfor in the ESXi host

CSCvn73127

35(2a)26(1b)A node shuts down when two disks are replaced (acaching disk and another capacity disk) where thecapacity disk is inserted first and then the caching diskis inserted

CSCvk46364

Hyper-V

40(1a)35(2a)Cluster expansion may fail with an error code 500 -operation timed out

CSCvn28721

35(2b)35(2a)During upgrade remove the VLAN on team created foruser vSwitch During fresh installation only one VLANtag is set to the vSwitch and team although multipleVLANs were entered

CSCvn54300

35(2b)35(2a)While upgrading a Hyper-V cluster on account of a rarerace condition between the stUpgradeService andZookeeper servers the upgrade orchestration throws anupgrade validation error and the upgrade process isaborted

CSCvn60486

Release Notes for Cisco HX Data Platform Release 4049

Resolved Caveats in Release 40(1a)

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Startrestart EAM service invCenter

When EAM service is not runningvCenter registration is failing

CSCvt55712

35(2a)Contact Cisco TAC if you believeyou are seeing similar conditions

HyperFlex Stretch Cluster saw poorapplication performance due to writelatency cluster remained unhealthy andrebalance was stuck

This defect is currently being used forinvestigative purposes

CSCvt45344

35(2g)Remove log files fromvarlogspringpath folder when thelog files are not getting rotatedStop the relevant service removelarge log files fromvarlogspringpath folder andrestart the service

varstv folder in Controller VM maybecome full

CSCvt36374

40(2b)Restarting the stMgr will initializeit again

In a Cluster configured with DR networkStMgr may not get initialized and stuckin a deadlock while enabling the IPTablerules (can be verified from the stMgrlog)

CSCvt89709

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Click recover for the selectedVM again under the standalonemode This will move the VM torecovered state

When the user recovers a VM protectedin a group it along with other VMs in thegroup move to recovering state instandalonemode The selectedVM shouldmove to recovered state

CSCvs75553

35(2e)Clean up the fail over cluster andthen recreate the fail over clusterNo need to touch the HX storagecluster

Hyper-V When replicated DC was usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2d)40(1a)

Keep cluster utilization as low aspossible To avoid read-only statekeep below 92

HyperFlex clusters storage utilization isabove 92 usable capacity

CSCvr20922

Release Notes for Cisco HX Data Platform Release 4050

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1b)NAAll the disks in the node got locked afterrebooting the node

We successfully unlocked usingsed-clientsh -U command but wantedto test with another reboot and driveslocked again

CSCvs74286

35(2g)To recover the VM copy over thedata disks and attach them to thenew VM

After enable encryption HX connectencryption status showing Caution butall of the disks enable encryptionsuccessful

-Self Encrypting Drives Service isrunning on all of the nodes

-There is no error message of encryption

- USB0 interface is up

-All SED disks showing supported 1enabled 1 locked 0

CSCvs93245

40(1b)Enable kernelpanic_on_oops as apersistent configuration This willcause the node to panic and rebootimmediately

A cluster node running HX release40(1b) may hang in the Linux kernelThis is classified as an oops and adeviation from the expected behavior

CSCvs54285

Release Notes for Cisco HX Data Platform Release 4051

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Enabling K8iscsi stack in Hx clusterCSCvs41324

Release Notes for Cisco HX Data Platform Release 4052

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Steps to enable K8 on HX cluster

Perform following steps on ALLthe controller VMs

1 Updateetcinitscvmclientconf toenable tune iscsiEnable totrue

sed -iesiscsiEnable=falseiscsiEnable=trueetcinitscvmclientconf

2 Run following initctlcommand to reloadconfiguration

initctl reload-configuration

3 Restart scvmclient process

stop scvmclient startscvmclient

4 Run following initctlcommand to mount SYSTEMDS

initctl emit --no-waitsystem-datastore-created

5 Verify that iscsiEnable tuneis set to true using thefollowing command

ps -eaf | grep scvmclient |grep -v grep

root 6241 1 1 Dec06 014007optspringpathstorfs-corescvmclient-T logEnabled=false -TlogSyslogEnabled=true -TlogEchoToScreen=false -TstatLoggingToFile=false -TstatLoggingToSyslog=true-TlogDir=varlogspringpath-TnfsBackendServerList=1010748100-T iscsiEnable=true -TiscsiUseAsync=true -T

Release Notes for Cisco HX Data Platform Release 4053

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

iscsiConfigFilePath=nfsSYSTEMistgtconfps -eaf | grep scvmclient |grep -v grep

35(2e)Following procedure can be usedto check NFS Queue Depth andincrease if needed

rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth256 lt- Low value[rootHXESXI1] vsish -e setvmkModulesnfsclientmnt[DATASTORE]propertiesmaxQDepth 1024[rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth1024 lt- Optimalvalue

This requires an ESXi host rebootto take effect

Please place the node in HyperflexMaintenanceMode and gracefullyreboot the node for the changes tobe applied

HyperFlex Datastore NFS Queue Depthshows as 256 which can lead toperformance (including latency) issues

CSCvr83056

40(1b)This is cosmetic The error will goaway once the cluster returns tohealthy status

When a single node is offline in clusterstcli cluster storage-summary shows twonodes unavailable rootSCVM~ stclicluster storage-summary

CSCvs47735

35(2g)Modify both the root and adminpassword for the SCVMs to be thesame

stcli security password set -uadmin

Cluster expansion fails at Config Installerstage when the root and admin passwordfor the storage controller (SCVM) aredifferent

CSCvs69317

35(2d)NAAfter successfully changing the DNSserver on the HX controller we still cansee the original DNS entry that was addedduring the deployment

CSCvs69154

Release Notes for Cisco HX Data Platform Release 4054

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Customers are strongly encouragedto work with TAC in order toidentify and remediate this issue

1 Change getcluster from 2m to8m inoptspringpathstorfs-mgmtstMgr-10confapplicationconfon all nodes

2 Restart stMgr on all nodes

You may see this error message after afailed upgrade or other task such asattempting to enter a node into HXmaintenance mode

CSCvs53555

40(2a)Run the following command oneach storage controller VM as root

grep -i monitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp sed -iesmonitor_dns_serverstruemonitor_dns_serversfalseoptspringpathhx-diag-toolswatchdog_configjsonampamp grep -imonitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp restart watchdog

Receive the following alertevent in HXConnect HX Controller VMHOSTNAME one or more configuredDNS servers not responding

CSCvt13947

35(2b)Delete any empty (Size 0) log filesunder varlogzookeeperversion-2ORvarlogzookeeperstandaloneversion-2

Zookeeper fails to start while Exhibitor isrunning however echo srvr returnsnothing

CSCvs21562

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

40(4e)1 De-commission the server

2 Power drain the server -REMOVE BOTH POWERCORDS ON THE BACK OFTHE SERVER FOR 10SECONDS THENREINSERTPOWERCORDS

3 Re-commission the server

M2 boot disk is missing from serverinventory after upgrade to server firmware40(4e) As a result server fails to boot toOS installed in the M2 disk The issuepersists after re-acknowledgement as wellas de-commission andre-acknowledgement of the server

CSCvs02466

Release Notes for Cisco HX Data Platform Release 4055

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Re-start the flusher manuallyMany disks taken offline by a fault ormanually resulting in ENOSPC oncluster

CSCvq32721

35(2e)Clean up fail over cluster and thenrecreate the fail over cluster Thereis no need to touch the HX storagecluster

[Hyper-V]When replicated DCwas usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2b)Manually install upgradescvmclient VIB andmake sure thatit matches with HXDP version

Upgrade from release HX 35(2a) to35(2x) 4X does not upgrade scvmclientRecovery point may not work

CSCvq54992

Management

30(1b)To recover the VM copy over thedata disks and attach them to thenew VM

[HyperV]VMshows up onmultiple nodesCSCvj22992

35(1a)1 Do not change MTU at ESXlevel on running cluster

2 Reset back to the original value

ChangingMTU from 9000 to 1500 on oneESX node causes storfs process to restarton all nodes

CSCvm99150

Release Notes for Cisco HX Data Platform Release 4056

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)After the installation (for newdeployments) or after an upgradeto 401a (for existingdeployments) perform thefollowing steps on all ControllerVMs

1 Edit the following tune files (onall Controller VMs)

optspringpathconfiglfftunes

optspringpathconfigvsi_16tbtunes

2 SetcleanerEnableSegSummaryCleaning

to false

3 After editing the above tunefiles

a SSH in to all the ControllerVMs

b Run storfstool -- -Z

c Run the below command andcheck the values Tune valueshould be true

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=true

d Type below commands andapplied the tunes changesdynamically

echo false gttmpstprocfssystemtunecleanerEnableSegSummaryCleaning

e Verify that tune values arechanged Run below commandValue should be false

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=false

f umount tmpstprocfs

A cluster with 8TB or 12TB disk drivesmay experience IO stalls for severalminutes after another node fails in thecluster

CSCvp23718

Release Notes for Cisco HX Data Platform Release 4057

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)1 Update the conf file manuallyto change to the link local addressto

2 Restart scvmclient on thecontroller and ESX

HX 35(2b) FlexVol to HX 40 CSIupgrade does not work FlexVol continuesto work

CSCvp26319

35(2a)Disable rebalance until node healsthen reenable rebalance

7-node cluster healing very slowlybecause rebalance change AMS reqs areblocking healing process

CSCvp62512

35(2a)NAWith moderate cluster storage capacityusage under site failover cluster clustermay experience an all paths (APD) downcondition

CSCvq49412

40(1a)NAThis product includes Third-partySoftware that is affected by thevulnerabilities identified by the followingCommon Vulnerability and Exposures(CVE) IDs

CVE-2016-0762 CVE-2016-6797CVE-2016-6816 CVE-2016-8735CVE-2017-5647 CVE-2017-12615CVE-2017-12616 CVE-2017-12617CVE-2017-7674 CVE-2018-1304CVE-2018-8014 CVE-2018-1336CVE-2018-8034 CVE-2018-11784CVE-2019-0232

CSCvp36364

35(2d)NAstcli cluster info command needs toprovide UCSM VIP address Currently itshows ucsm-hostcom

CSCvq11456

35(2a)No workarounds currently Thestale entries do not affect the dayto day operations of HyperFlex

HyperFlex upgrade validation failedbecause cluster has extraneous stNodes ininternal database

CSCvq32530

35(2a)NAWhen not in maintenancemode all drivesappear in HX Connect and get updated asexpected

When in maintenance mode extra drivesare shown in HX Connect with no slotnumbers

CSCvq42378

35(2b)Contact TAC to cleanup duplicatenode

Nodes duplicated in HX Connect aftercluster recreate

CSCvq61825

Release Notes for Cisco HX Data Platform Release 4058

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)NACurrently HyperFlex installs on Hyper-Vdo not contain the stcli security whitelistcommandset

CSCvq66245

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

Release Notes for Cisco HX Data Platform Release 4059

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)[Hyper-V] Converged node expansionfails for incoming node with VIC 14xx

CSCvq39523

Release Notes for Cisco HX Data Platform Release 4060

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

1 Set Jumbo frame inWindowsHypervisor for incomingnode

Get-NetAdapter storage-data-a|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

Get-NetAdapter storage-data-b|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

2 Verify all interfaces are setcorrectly to support Jumbo frames

Get-NetAdapter storage |Get-NetAdapterAdvancedProperty| RegistryKeyword -matchjumbo | ft -auto

You should receive the followingmessage

Name DisplayName DisplayValue

RegistryKeyword RegistryValue

---- ----------- ------------

--------------- -------------

vswitch-hx-storage-data Jumbo

Packet 9014 Bytes

JumboPacket 9014

storage-data-b Jumbo Packet

Bytes 9014 JumboPacket

9014

storage-data-a Jumbo Packet

Bytes 9014 JumboPacket

9014

3 Reboot only the incoming nodeor the node being expanded

4 Retry Cluster Expansion in

Release Notes for Cisco HX Data Platform Release 4061

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

DescriptionCVEDefect IDRelease

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toexecute commands as the root user

The vulnerability is due to insufficientinput validation An attacker could exploitthis vulnerability by connecting to thecluster service manager and injectingcommands into the bound process Asuccessful exploit could allow the attackerto run commands on the affected host asthe root user

Cisco has released software updates thataddress this vulnerability There areworkarounds that address thisvulnerability

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvq2417640(1b)35(2g)

A vulnerability in the cluster servicemanager of Cisco HyperFlex could allowan unauthenticated adjacent attacker toperform a command injection as the rootuser

The vulnerability is due to an unprotectedlistening interface An attacker couldexploit this vulnerability by connectingto the listening interface and injectingcommands to the bound process Anexploit could allow the attacker to runcommands on the affected host as the rootuser

For more information see the relatedCisco Security Advisory

CVE-2018-15380CSCvj9560640(1b)35(2g)

Release Notes for Cisco HX Data Platform Release 4029

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerabilities associated with theJVM18U121memory leak found duringVC alarms (concurrent 40 calls usingREST API)

CVE-2017-10053 CVE-2017-10067CVE-2017-10074 CVE-2017-10078CVE-2017-10081 CVE-2017-10087CVE-2017-10089 CVE-2017-10090CVE-2017-10096 CVE-2017-10101CVE-2017-10102 CVE-2017-10107CVE-2017-10108 CVE-2017-10109CVE-2017-10110 CVE-2017-10111CVE-2017-10115 CVE-2017-10116CVE-2017-10118 CVE-2017-10135CVE-2017-10176 CVE-2017-10193CVE-2017-10198 CVE-2017-10243CVE-2017-10274 CVE-2017-10281CVE-2017-10285 CVE-2017-10295CVE-2017-10345 CVE-2017-10346CVE-2017-10347 CVE-2017-10348CVE-2017-10349 CVE-2017-10350CVE-2017-10355 CVE-2017-10356CVE-2017-10357 CVE-2017-10388CVE-2017-3509 CVE-2017-3511CVE-2017-3526 CVE-2017-3533CVE-2017-3539 CVE-2017-3544CVE-2018-2579 CVE-2018-2582CVE-2018-2588 CVE-2018-2599CVE-2018-2602 CVE-2018-2603CVE-2018-2618 CVE-2018-2629CVE-2018-2633 CVE-2018-2634CVE-2018-2637 CVE-2018-2641CVE-2018-2663 CVE-2018-2677CVE-2018-2678 CVE-2018-2783CVE-2018-2790 CVE-2018-2794CVE-2018-2795 CVE-2018-2796CVE-2018-2797 CVE-2018-2798CVE-2018-2799 CVE-2018-2800CVE-2018-2814 CVE-2018-2815CVE-2018-2952 CVE-2018-3136CVE-2018-3139 CVE-2018-3149CVE-2018-3150 CVE-2018-3169CVE-2018-3180 CVE-2018-3183CVE-2018-3214 CVE-2019-2422

CSCvo8899740(1b)

Tomcat and Nginx logs are not beingcollected in the support bundle generatedthrough HX Connect in Release 35

NACSCvm5803140(1b)

The vulnerabilities associated with thelibmspack software package versionincluded in Cisco HX Data Platform

CVE-2018-18584

CVE-2018-18585

CSCvn3511940(1a)

Release Notes for Cisco HX Data Platform Release 4030

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerabilities associated withFasterXML Jackson-Databind TimeValueField Denial of Service

CVE-2018-14719

CVE-2018-14720

CVE-2018-1000873

CVE-2018-14721

CVE-2018-19360

CVE-2018-19362

CVE-2018-19361

CVE-2018-14718

CSCvn8228240(1a)

The vulnerabilities associated with theOpenSSL Protocol software packageversion included in Cisco HX DataPlatform

CVE-2013-3587CSCvo0505440(1a)

The vulnerabilities associated withThird-Party Software Denial of Service

CVE-2018-16487

CVE-2018-19361

CSCvo2781840(1a)

The vulnerabilities associated withLinuxkernel for Ubuntu 1710

CVE-2018-1092

CVE-2018-7492

CVE-2018-8087

CVE-2018-1068

CVE-2018-8781

CSCvm5314935(2a)

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)25(1d)stcli services dns remove should remove the DNS serverinfo from the interface files

CSCvs70967

40(2b)40(1b)HX 40(1b) new deployment from Intersight productioncloud

User receives following error while enabling CSI(Kubernetes integration)

Failure occurred during volume_access_enable Errorwas Nonzero exit code 1

CSCvt10522

40(2b)40(2a)Receive the following alertevent in HX Connect HXController VM HOSTNAME one or more configuredDNS servers not responding

CSCvt13947

Release Notes for Cisco HX Data Platform Release 4031

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)40(2a)API UI CLI show few Drive slots Empty after upgradeto 402a

The drives are seen in the sysmtool lsscsi and stclicommand outputs

The cluster is healthy with no errors

CSCvt14914

40(2b)35(2d)After successfully changing the DNS server on the HXcontroller we still can see the original DNS entry thatwas added during the deployment

CSCvs69154

40(2b)35(2e)HyperFlex Connect and VCenter show APD alarms forHX and non HX datastores This implies there is an issuewith HyperFlex when there may not be

CSCvs30080

40(2b)40(2a)An issue where all the disks in the node get locked afterrebooting the node

We successfully unlocked using sed-clientsh -Ucommand but wanted to test with another reboot anddrives locked again

CSCvs74286

40(2b)40(2a)When running stcli license commands on HyperFlexthe following error is seen

rootSpringpathControllertmp stcli license show all

Show smart licensing failed

Smart Agent is not ready please wait a minute and tryagain

CSCvt13929

40(2b)40(2a)After upgrading to HX 40(2a) the first RoboEdge nodethat gets rebooted then ignores persistent drivesPIDHX-HD24TB10K4KN

Cluster will be in degraded state and have reducedcapacity

CSCvt20144

40(2b)35(2b)Zookeeper fails to start while Exhibitor is runninghowever echo srvr returns nothing

CSCvs21562

40(2b)40(2a)When performing an HyperFlex upgrade a validationwarning may occur due to the host not having EnterprisePlus or Enterprise hypervisor licensing

Upgrade Validation Warning

ESXi host esx1labtest should be configured withVMware Enterprise license for upgrade to continue

CSCvs91787

Release Notes for Cisco HX Data Platform Release 4032

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)35(2g)Cluster expansion fails at Config Installer stage whenthe root and admin password for the storage controller(SCVM) are different

CSCvs69317

40(2b)35(2g)Panic while upgrading ESXiCSCvt06983

40(2b)40(2a)A cluster node running may hang in the Linux kernelThis is classified as an oops and a deviation from theexpected behavior

CSCvs54285

40(2b)35(2g)Rebalance failing on 10+10 node HX 35(2g) stretchcluster

CSCvs69007

40(2b)35(2d)The cluster becomes inaccessible to the IOVisorCSCvr54687

40(2b)40(2a)Panic on the storage controllerCSCvt61297

40(2b)40(2a)The size of support bundle is very large when collectedwith storfs-support command

CSCvt63306

40(2b)40(2a)storfs crash associated withFileSystemUsageWarningEvent orFileSystemUsageAlertEvent due to high usage ofvarstv above 80

CSCvt72807

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

40(2a)35(2d)In normal scenarios HX upgrade gets pendingACK afterupdating service profile In this case UCS only upgradewas stalled because after updating service profile HXupgrade was not getting pending ACK and it was alwayswaiting state

CSCvo39912

40(2a)35(1a)When using motherBoardReplace-12 to clean ZK fromold stNodepNodes resulted in unmounted datastore andmaking the size of the Hyperflex Datastores 0 resultingin all VMs in the cluster going offline

CSCvq39471

40(2a)26(1e)Upgrading cluster and getting error

Failed upgrade validations Checking vCenterconfiguration Reason Upgrade validations failed DRSFault Insufficient resources to satisfy configured failover

CSCvm77294

Release Notes for Cisco HX Data Platform Release 4033

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)The cluster expand fails on a node with DR replicationconfigured

When a HX cluster which has DR replication configuredis expanded we see the installer UI pulling in thereplicationVLAN information instead of themanagementVLAN information Even if we change that informationto the correct mgmt VLAN id and name it it does notseem to work as the node is configured with the VLANof the replicationVLAN in ESXi This leads to the failureof the node add with the host unreachable error

CSCvo70650

40(2a)35(1a)Customer reported that server didnt complete thefirmware upgrade automatically one by one

The user manually needs to put the host into maintenancemode and they manually acknowledged the pendingrequests to finish the UCS firmware upgrade

Maintenance policy is set to default (User-ack) as perthe design

CSCvo91624

40(2a)35(1i)When the cluster is in Failed state and stcli noderemove is attempted the output shows successful thoughthe node failed to get removed from the cluster

CSCvo93017

40(2a)35(1a)HyperFlex cluster upgrade may fail during validationwith a DRS Validation failed error

CSCvp31021

40(2a)35(2b)HX cluster expansion will fail with the following errormessage

MAC address pool configuration failure150[ErrorDescription] bad address block range definitioncollision

CSCvp58318

40(2a)35(2b)Memory usage by carbon cacheCSCvq34873

40(2a)35(2g)Upgrading ESXi cluster fails with error Nodemaintenance mode failed

CSCvr03240

40(2a)35(2e)storfs process does not automatically start on Exit HXMaintenance Mode or other tasks which restart StorageController

CSCvr88978

40(2a)35(2a)Post install script fails with message

Failed to execute ipmitool on HX node

CSCvp10707

40(2a)40(1a)HyperFlex expansionworkflow doesnt pull VLANnamecorrectly

CSCvp46539

Release Notes for Cisco HX Data Platform Release 4034

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)During HX cluster deployment - Validate ClusterCreation phase - HX Installer might fail with the error

fromvarlogspringpathstDeploylog

CSCvq45087

40(2a)35(2b)HX Installer fails to login to the SOL for configuringESX hosts

CSCvq91380

40(2a)35(2d)If DRS is enabled with configuration parameters otherthen fully automated ie manual or partially automatedthan this will change to fully automated during clusterexpansion

CSCvr44222

40(2a)35(1a)One of the validation task name has a typo The taskname is Validating node for mixed of SED and non-SEDdisks

It should be Validating node for mix of SED andnon-SED disks

CSCvp82175

40(2a)30(1i)This combination of using smaller NVMe drives is notsupported in expand operation For example if the drivesare only 375GB and they cannot be added to an existingcluster with larger caching SSD

CSCvo12359

40(2a)35(1a)HyperFlex includes a version of OpenSSL that is affectedby the vulnerabilities identified by the followingCommon Vulnerability and Exposures

CVE-2018-0495

CSCvp66679

40(2a)30(1e)SDK invoke exception when taking a native snapshot ona VM with GPU pci passthrough attached

CSCvo00511

40(2a)30(1i)Node replacement script fails due to EAM errorCSCvo62867

40(2a)35(1a)While pairing a cluster (HX release gt= 35) and cluster(HX release lt 35) remote replication network test failson the cluster (HX release lt 35)

CSCvo79760

40(2a)30(1b)Fixed the issue in the latest support-workflow bundleThis should not be hit anymore

CSCvo87061

40(2a)35(2a)On a 3 node cluster the MBreplace script fails as thescript looks for HEALTHY cluster status

CSCvo87080

40(2a)35(2a)MbReplace script (tar) hangs when running on 352b or40

CSCvp12359

Release Notes for Cisco HX Data Platform Release 4035

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)HX replication cleanup failing with error NA 35(2a) -INFODR state is not clean

CSCvp63958

40(2a)40(1a)The check_stig_parameters API incorrectly shows thecompliance state of the cluster as not STIG compliant

CSCvp66277

40(2a)35(2c)The snapshot creation on CBT enabled VM fails witherror Failed in vmreparent vmkfstools clone1

CSCvq06952

40(2a)35(2a)HyperFlex GUI when setting up remote DR-replicationpartnerpeer should show remote replication partner TCPport reachability test results

This will provide quick upfront connectivity results thatfrequently takes customer a while to manually check andconfirm

If reachability tests fail an info window and HX alertshould trigger that list the destination peer IP and thespecific port(s) that could not be reache

CSCvo60587

40(2a)35(2b)If you have more than one IP pool configured for areplication network and local replication network testsare failing you might be running into this issue

CSCvr67130

40(2a)35(2a)Skip task option in MBreplace and Software redeployscripts The scripts should have the option to skip tasksas user inputs and not by modifying the json file

CSCvp05204

40(2a)40(1a)Error noticed on SCVM Console print_req_error IOerror dev fd0 sector 0 on HyperV

It doesnt come up on SSH session

CSCvq34357

40(2a)30(1d)The VMNetwork switch is already created as part of theinstall process

Ideally the ip address assignment should be done duringthe initial install process as well No post install stepsshould be required altrnatively it could be done as partof a post install script

CSCvk36222

40(2a)25(1c)storfs service impacted during HX upgrade from HX21(1) to 25(1c)

HX node which is not being upgraded storfs servicestopped causing outage

CSCvg53223

40(2a)35(1b)HX Stretch Cluster Witness VM reverts to DHCP onreboot

CSCvp37536

40(2a)35(2b)HyperFlex Stretch Cluster Witness does not getprogrammed with an NTP server

CSCvp46578

Release Notes for Cisco HX Data Platform Release 4036

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2b)HyperFlex cluster does not heal after maintenance onnodeserver

CSCvr18528

40(2a)40(2a)High percentage of packet loss on the eth1 data interface9K MTU

CSCvr97089

40(2a)35(2f)The HyperFlex cluster healing state is stuck at 87 andis unable to progress

CSCvr16760

40(2a)40(2a)In some circumstances two replications can occur forthe same VM

From the GUI the replication status for this VM willshow stuck in an In Progress state but there are nojobs stuck in replication layer Subsequent Replicationsshould succeed

CSCvq22898

40(2a)35(2a)After an unexpected power outage the hyperflex clusteris unable to come up correctly

All processes are running time is synced and all nodescan vmkping each other as required

CSCvp13990

40(2a)40(1a)When specifying IP address instead of node ID stclinode maintenanceMode commands fail

CSCvp33657

40(2a)40(1a)2 node edge cluster if experiencing Intersightarbitratorconnectivity issues should NOT allowHXMaintenancemode to initiate as this will bring the cluster down

CSCvr31573

40(2a)35(2c)Old files in varsupportZKTxnlog are not purged withthe daily zklog-cleanup cron job

CSCvr89066

40(2a)35(2a)When a node is in MaintenanceMode any disk removalor replacement will be reflected in UI only after the nodeis brought back frommaintenancemode This is becausestorfs is not running on the node in maintenance andwill not be able to detect disk activities until it is broughtout of Maintenance Mode

CSCvo86431

40(2a)35(2c)Cluster might go offline and stop serving IO requestsrootcvm~ stcli cluster storage-summary--detailGet cluster storage summary failedjavanetConnectException Connection refused19216814210010207

CSCvr01645

40(2a)35(2d)A stretch cluster deployment fails with a messageFormatting nodes Node in Use on the Hyperflex installerand crmZoneType value is seen as 1 in storfscfg oroptspringpathconfigstretchtunes

CSCvq80340

Release Notes for Cisco HX Data Platform Release 4037

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)30(1c)During upgrade we upload the upgrade packagedownloaded from Ciscocom Upgrade packages are offile type tgz hxconnect acceptsstorfs-packages-401a-33028tar file a different type ofcompression and allows upgrade to start and fail whenit cannot decompress the file for bootstrap and scvmclientupgrade on ESXi

CSCvq63888

40(2a)40(2a)While attempting to install hyperflex via intersight thefollowing error is seen

Failed in Task Add hosts to vCenter Cluster with ErrorTry adding hosts manually to vCenter and retry Failedto get addressFailed to host by namelookupHostnamecompanycom on 000053 read udp12700145430-gt12700153 read connection refused

CSCvr52098

40(2a)35(2a)CoreAPI call may time out for cluster management APIwhile doing cluster create validation

CSCvo92952

40(2a)35(2d)Validation fails with Mixed mode expansion checkdue to Enhanced vMotion Compatibility Incompatibilitywhen the ESXi version is different

CSCvq95460

40(2a)30(1i)This failure is seen in installer when custom workflowis used hypervisor configuration + deploy (clean diskpartition=no) at Installing software packages on storagecontroller VM

CSCvr66309

40(2a)35(2d)The HX Installer apparently replaced a comma bysemicolon on the VLAN ID range while exporting theconfiguration to a JSON file

CSCvq93831

40(2a)40(1a)When using unsupported browsers HX Connect userswill not be able to login These users see an errorindicating unauthorized user

CSCvp22693

Release Notes for Cisco HX Data Platform Release 4038

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)40(1a)A vulnerability in the web-based management interfaceof Cisco HyperFlex Software could allow anunauthenticated remote attacker to conduct a cross-siterequest forgery (CSRF) attack on an affected system

The vulnerability is due to insufficient CSRF protectionsfor the web UI on an affected device An attacker couldexploit this vulnerability by persuading a user of theinterface to follow a malicious link A successful exploitcould allow the attacker to perform arbitrary actions withthe privilege level of the affected user

Cisco has released software updates that address thisvulnerability There are no workarounds that address thisvulnerability

This advisory is available at the following link

httpstoolsciscocomsecuritycentercontentCiscoSecurityAdvisorycisco-sa-20190807-hypflex-csrf

CSCvp24343

40(2a)35(2d)Add message to HX Connect Progress flow to notAcknowledge Pending Activities and reboot the serverson UCSM

HX Connect is doing a controlled rolling server upgradein the background

CSCvq22844

40(2a)40(1b)Compute Node expand operation fails due to incorrectcluster name

CSCvr43786

40(2a)26(1e)RF-2 cluster shutdown during data resynchronizationafter a node failure (non storfs) followed by multipledisk read failures and a HardBlacklist of a disk

CSCvp41241

40(2a)40(1a)HyperFlex Edge nodes do not properly set the ESXihostname during deployment

CSCvo13143

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4039

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4040

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1b) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401b-33133p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4041

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4042

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1b)40(1a)While running the HyperFlex installer with WindowsServer Hyper-V cluster creation process fails with theerror Failure occurred during ClusterCreation process Unable to post thecontent to the downstream This symptomis encountered while deploying HyperFlex cluster withHyperFlex nodes configuredordered with Cisco VIC1457 MLOM (PID HX-MLOM-C25Q-04) andWindows Server Datacenter or Core with Hyper-Vversion 2016 or 2019

CSCvp64140

Release Notes for Cisco HX Data Platform Release 4043

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)35(2b)Adding Micron 5200 drive to a cluster fails to increasecluster capacity

CSCvo69067

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4044

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4045

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1a) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401a-33028p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4046

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4047

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1a)30(1d)Cluster instability when disks of different sector sizeplaced in HX node

CSCvk17250

40(1a)

35(2c)

35(1a)When logged in using a local HX user account insteadof a Virtual Center account an error message appearsintermittently indicating Virtual Centerunreachable or Resource informationcannot be updated when VC is reachable

CSCvo36198

40(1a)

35(2b)

30(1d)HXDP does not work with EMC RecoverPoint - needsto support VMware API (FSS-Readdir)

CSCvk38003

Release Notes for Cisco HX Data Platform Release 4048

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)

35(2a)

30(1i)

25(1c)

25(1d)

Zookeeper (Exhibitor) process on HyperFlex StorageController VMs may fail to respond to requests whenvarzookeeper has low or no free space

CSCvm90352

40(1a)26(1c)Use Asynchronous consolidation for HX SnapshotsCSCvn02151

40(1a)35(2b)HX Quiesced Snapshot for Backup VendorsCSCvo90713

40(1a)35(2a)The cluster creationcluster expansion workflow stopswith the following error message at the validation step

FIRMWARE-Check UCSC-SAS-M5HD

FIRMWARE-Check UCSC-SAS-M5HD Required00000029000000320000003500000050 Found00000058

Action Needed Update the Controller Firmware toRequired Version

CSCvn17787

35(2b)30(1d)Kernel migration fails when a local datastore is searchedfor in the ESXi host

CSCvn73127

35(2a)26(1b)A node shuts down when two disks are replaced (acaching disk and another capacity disk) where thecapacity disk is inserted first and then the caching diskis inserted

CSCvk46364

Hyper-V

40(1a)35(2a)Cluster expansion may fail with an error code 500 -operation timed out

CSCvn28721

35(2b)35(2a)During upgrade remove the VLAN on team created foruser vSwitch During fresh installation only one VLANtag is set to the vSwitch and team although multipleVLANs were entered

CSCvn54300

35(2b)35(2a)While upgrading a Hyper-V cluster on account of a rarerace condition between the stUpgradeService andZookeeper servers the upgrade orchestration throws anupgrade validation error and the upgrade process isaborted

CSCvn60486

Release Notes for Cisco HX Data Platform Release 4049

Resolved Caveats in Release 40(1a)

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Startrestart EAM service invCenter

When EAM service is not runningvCenter registration is failing

CSCvt55712

35(2a)Contact Cisco TAC if you believeyou are seeing similar conditions

HyperFlex Stretch Cluster saw poorapplication performance due to writelatency cluster remained unhealthy andrebalance was stuck

This defect is currently being used forinvestigative purposes

CSCvt45344

35(2g)Remove log files fromvarlogspringpath folder when thelog files are not getting rotatedStop the relevant service removelarge log files fromvarlogspringpath folder andrestart the service

varstv folder in Controller VM maybecome full

CSCvt36374

40(2b)Restarting the stMgr will initializeit again

In a Cluster configured with DR networkStMgr may not get initialized and stuckin a deadlock while enabling the IPTablerules (can be verified from the stMgrlog)

CSCvt89709

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Click recover for the selectedVM again under the standalonemode This will move the VM torecovered state

When the user recovers a VM protectedin a group it along with other VMs in thegroup move to recovering state instandalonemode The selectedVM shouldmove to recovered state

CSCvs75553

35(2e)Clean up the fail over cluster andthen recreate the fail over clusterNo need to touch the HX storagecluster

Hyper-V When replicated DC was usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2d)40(1a)

Keep cluster utilization as low aspossible To avoid read-only statekeep below 92

HyperFlex clusters storage utilization isabove 92 usable capacity

CSCvr20922

Release Notes for Cisco HX Data Platform Release 4050

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1b)NAAll the disks in the node got locked afterrebooting the node

We successfully unlocked usingsed-clientsh -U command but wantedto test with another reboot and driveslocked again

CSCvs74286

35(2g)To recover the VM copy over thedata disks and attach them to thenew VM

After enable encryption HX connectencryption status showing Caution butall of the disks enable encryptionsuccessful

-Self Encrypting Drives Service isrunning on all of the nodes

-There is no error message of encryption

- USB0 interface is up

-All SED disks showing supported 1enabled 1 locked 0

CSCvs93245

40(1b)Enable kernelpanic_on_oops as apersistent configuration This willcause the node to panic and rebootimmediately

A cluster node running HX release40(1b) may hang in the Linux kernelThis is classified as an oops and adeviation from the expected behavior

CSCvs54285

Release Notes for Cisco HX Data Platform Release 4051

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Enabling K8iscsi stack in Hx clusterCSCvs41324

Release Notes for Cisco HX Data Platform Release 4052

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Steps to enable K8 on HX cluster

Perform following steps on ALLthe controller VMs

1 Updateetcinitscvmclientconf toenable tune iscsiEnable totrue

sed -iesiscsiEnable=falseiscsiEnable=trueetcinitscvmclientconf

2 Run following initctlcommand to reloadconfiguration

initctl reload-configuration

3 Restart scvmclient process

stop scvmclient startscvmclient

4 Run following initctlcommand to mount SYSTEMDS

initctl emit --no-waitsystem-datastore-created

5 Verify that iscsiEnable tuneis set to true using thefollowing command

ps -eaf | grep scvmclient |grep -v grep

root 6241 1 1 Dec06 014007optspringpathstorfs-corescvmclient-T logEnabled=false -TlogSyslogEnabled=true -TlogEchoToScreen=false -TstatLoggingToFile=false -TstatLoggingToSyslog=true-TlogDir=varlogspringpath-TnfsBackendServerList=1010748100-T iscsiEnable=true -TiscsiUseAsync=true -T

Release Notes for Cisco HX Data Platform Release 4053

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

iscsiConfigFilePath=nfsSYSTEMistgtconfps -eaf | grep scvmclient |grep -v grep

35(2e)Following procedure can be usedto check NFS Queue Depth andincrease if needed

rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth256 lt- Low value[rootHXESXI1] vsish -e setvmkModulesnfsclientmnt[DATASTORE]propertiesmaxQDepth 1024[rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth1024 lt- Optimalvalue

This requires an ESXi host rebootto take effect

Please place the node in HyperflexMaintenanceMode and gracefullyreboot the node for the changes tobe applied

HyperFlex Datastore NFS Queue Depthshows as 256 which can lead toperformance (including latency) issues

CSCvr83056

40(1b)This is cosmetic The error will goaway once the cluster returns tohealthy status

When a single node is offline in clusterstcli cluster storage-summary shows twonodes unavailable rootSCVM~ stclicluster storage-summary

CSCvs47735

35(2g)Modify both the root and adminpassword for the SCVMs to be thesame

stcli security password set -uadmin

Cluster expansion fails at Config Installerstage when the root and admin passwordfor the storage controller (SCVM) aredifferent

CSCvs69317

35(2d)NAAfter successfully changing the DNSserver on the HX controller we still cansee the original DNS entry that was addedduring the deployment

CSCvs69154

Release Notes for Cisco HX Data Platform Release 4054

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Customers are strongly encouragedto work with TAC in order toidentify and remediate this issue

1 Change getcluster from 2m to8m inoptspringpathstorfs-mgmtstMgr-10confapplicationconfon all nodes

2 Restart stMgr on all nodes

You may see this error message after afailed upgrade or other task such asattempting to enter a node into HXmaintenance mode

CSCvs53555

40(2a)Run the following command oneach storage controller VM as root

grep -i monitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp sed -iesmonitor_dns_serverstruemonitor_dns_serversfalseoptspringpathhx-diag-toolswatchdog_configjsonampamp grep -imonitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp restart watchdog

Receive the following alertevent in HXConnect HX Controller VMHOSTNAME one or more configuredDNS servers not responding

CSCvt13947

35(2b)Delete any empty (Size 0) log filesunder varlogzookeeperversion-2ORvarlogzookeeperstandaloneversion-2

Zookeeper fails to start while Exhibitor isrunning however echo srvr returnsnothing

CSCvs21562

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

40(4e)1 De-commission the server

2 Power drain the server -REMOVE BOTH POWERCORDS ON THE BACK OFTHE SERVER FOR 10SECONDS THENREINSERTPOWERCORDS

3 Re-commission the server

M2 boot disk is missing from serverinventory after upgrade to server firmware40(4e) As a result server fails to boot toOS installed in the M2 disk The issuepersists after re-acknowledgement as wellas de-commission andre-acknowledgement of the server

CSCvs02466

Release Notes for Cisco HX Data Platform Release 4055

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Re-start the flusher manuallyMany disks taken offline by a fault ormanually resulting in ENOSPC oncluster

CSCvq32721

35(2e)Clean up fail over cluster and thenrecreate the fail over cluster Thereis no need to touch the HX storagecluster

[Hyper-V]When replicated DCwas usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2b)Manually install upgradescvmclient VIB andmake sure thatit matches with HXDP version

Upgrade from release HX 35(2a) to35(2x) 4X does not upgrade scvmclientRecovery point may not work

CSCvq54992

Management

30(1b)To recover the VM copy over thedata disks and attach them to thenew VM

[HyperV]VMshows up onmultiple nodesCSCvj22992

35(1a)1 Do not change MTU at ESXlevel on running cluster

2 Reset back to the original value

ChangingMTU from 9000 to 1500 on oneESX node causes storfs process to restarton all nodes

CSCvm99150

Release Notes for Cisco HX Data Platform Release 4056

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)After the installation (for newdeployments) or after an upgradeto 401a (for existingdeployments) perform thefollowing steps on all ControllerVMs

1 Edit the following tune files (onall Controller VMs)

optspringpathconfiglfftunes

optspringpathconfigvsi_16tbtunes

2 SetcleanerEnableSegSummaryCleaning

to false

3 After editing the above tunefiles

a SSH in to all the ControllerVMs

b Run storfstool -- -Z

c Run the below command andcheck the values Tune valueshould be true

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=true

d Type below commands andapplied the tunes changesdynamically

echo false gttmpstprocfssystemtunecleanerEnableSegSummaryCleaning

e Verify that tune values arechanged Run below commandValue should be false

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=false

f umount tmpstprocfs

A cluster with 8TB or 12TB disk drivesmay experience IO stalls for severalminutes after another node fails in thecluster

CSCvp23718

Release Notes for Cisco HX Data Platform Release 4057

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)1 Update the conf file manuallyto change to the link local addressto

2 Restart scvmclient on thecontroller and ESX

HX 35(2b) FlexVol to HX 40 CSIupgrade does not work FlexVol continuesto work

CSCvp26319

35(2a)Disable rebalance until node healsthen reenable rebalance

7-node cluster healing very slowlybecause rebalance change AMS reqs areblocking healing process

CSCvp62512

35(2a)NAWith moderate cluster storage capacityusage under site failover cluster clustermay experience an all paths (APD) downcondition

CSCvq49412

40(1a)NAThis product includes Third-partySoftware that is affected by thevulnerabilities identified by the followingCommon Vulnerability and Exposures(CVE) IDs

CVE-2016-0762 CVE-2016-6797CVE-2016-6816 CVE-2016-8735CVE-2017-5647 CVE-2017-12615CVE-2017-12616 CVE-2017-12617CVE-2017-7674 CVE-2018-1304CVE-2018-8014 CVE-2018-1336CVE-2018-8034 CVE-2018-11784CVE-2019-0232

CSCvp36364

35(2d)NAstcli cluster info command needs toprovide UCSM VIP address Currently itshows ucsm-hostcom

CSCvq11456

35(2a)No workarounds currently Thestale entries do not affect the dayto day operations of HyperFlex

HyperFlex upgrade validation failedbecause cluster has extraneous stNodes ininternal database

CSCvq32530

35(2a)NAWhen not in maintenancemode all drivesappear in HX Connect and get updated asexpected

When in maintenance mode extra drivesare shown in HX Connect with no slotnumbers

CSCvq42378

35(2b)Contact TAC to cleanup duplicatenode

Nodes duplicated in HX Connect aftercluster recreate

CSCvq61825

Release Notes for Cisco HX Data Platform Release 4058

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)NACurrently HyperFlex installs on Hyper-Vdo not contain the stcli security whitelistcommandset

CSCvq66245

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

Release Notes for Cisco HX Data Platform Release 4059

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)[Hyper-V] Converged node expansionfails for incoming node with VIC 14xx

CSCvq39523

Release Notes for Cisco HX Data Platform Release 4060

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

1 Set Jumbo frame inWindowsHypervisor for incomingnode

Get-NetAdapter storage-data-a|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

Get-NetAdapter storage-data-b|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

2 Verify all interfaces are setcorrectly to support Jumbo frames

Get-NetAdapter storage |Get-NetAdapterAdvancedProperty| RegistryKeyword -matchjumbo | ft -auto

You should receive the followingmessage

Name DisplayName DisplayValue

RegistryKeyword RegistryValue

---- ----------- ------------

--------------- -------------

vswitch-hx-storage-data Jumbo

Packet 9014 Bytes

JumboPacket 9014

storage-data-b Jumbo Packet

Bytes 9014 JumboPacket

9014

storage-data-a Jumbo Packet

Bytes 9014 JumboPacket

9014

3 Reboot only the incoming nodeor the node being expanded

4 Retry Cluster Expansion in

Release Notes for Cisco HX Data Platform Release 4061

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

DescriptionCVEDefect IDRelease

The vulnerabilities associated with theJVM18U121memory leak found duringVC alarms (concurrent 40 calls usingREST API)

CVE-2017-10053 CVE-2017-10067CVE-2017-10074 CVE-2017-10078CVE-2017-10081 CVE-2017-10087CVE-2017-10089 CVE-2017-10090CVE-2017-10096 CVE-2017-10101CVE-2017-10102 CVE-2017-10107CVE-2017-10108 CVE-2017-10109CVE-2017-10110 CVE-2017-10111CVE-2017-10115 CVE-2017-10116CVE-2017-10118 CVE-2017-10135CVE-2017-10176 CVE-2017-10193CVE-2017-10198 CVE-2017-10243CVE-2017-10274 CVE-2017-10281CVE-2017-10285 CVE-2017-10295CVE-2017-10345 CVE-2017-10346CVE-2017-10347 CVE-2017-10348CVE-2017-10349 CVE-2017-10350CVE-2017-10355 CVE-2017-10356CVE-2017-10357 CVE-2017-10388CVE-2017-3509 CVE-2017-3511CVE-2017-3526 CVE-2017-3533CVE-2017-3539 CVE-2017-3544CVE-2018-2579 CVE-2018-2582CVE-2018-2588 CVE-2018-2599CVE-2018-2602 CVE-2018-2603CVE-2018-2618 CVE-2018-2629CVE-2018-2633 CVE-2018-2634CVE-2018-2637 CVE-2018-2641CVE-2018-2663 CVE-2018-2677CVE-2018-2678 CVE-2018-2783CVE-2018-2790 CVE-2018-2794CVE-2018-2795 CVE-2018-2796CVE-2018-2797 CVE-2018-2798CVE-2018-2799 CVE-2018-2800CVE-2018-2814 CVE-2018-2815CVE-2018-2952 CVE-2018-3136CVE-2018-3139 CVE-2018-3149CVE-2018-3150 CVE-2018-3169CVE-2018-3180 CVE-2018-3183CVE-2018-3214 CVE-2019-2422

CSCvo8899740(1b)

Tomcat and Nginx logs are not beingcollected in the support bundle generatedthrough HX Connect in Release 35

NACSCvm5803140(1b)

The vulnerabilities associated with thelibmspack software package versionincluded in Cisco HX Data Platform

CVE-2018-18584

CVE-2018-18585

CSCvn3511940(1a)

Release Notes for Cisco HX Data Platform Release 4030

Security Fixes

DescriptionCVEDefect IDRelease

The vulnerabilities associated withFasterXML Jackson-Databind TimeValueField Denial of Service

CVE-2018-14719

CVE-2018-14720

CVE-2018-1000873

CVE-2018-14721

CVE-2018-19360

CVE-2018-19362

CVE-2018-19361

CVE-2018-14718

CSCvn8228240(1a)

The vulnerabilities associated with theOpenSSL Protocol software packageversion included in Cisco HX DataPlatform

CVE-2013-3587CSCvo0505440(1a)

The vulnerabilities associated withThird-Party Software Denial of Service

CVE-2018-16487

CVE-2018-19361

CSCvo2781840(1a)

The vulnerabilities associated withLinuxkernel for Ubuntu 1710

CVE-2018-1092

CVE-2018-7492

CVE-2018-8087

CVE-2018-1068

CVE-2018-8781

CSCvm5314935(2a)

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)25(1d)stcli services dns remove should remove the DNS serverinfo from the interface files

CSCvs70967

40(2b)40(1b)HX 40(1b) new deployment from Intersight productioncloud

User receives following error while enabling CSI(Kubernetes integration)

Failure occurred during volume_access_enable Errorwas Nonzero exit code 1

CSCvt10522

40(2b)40(2a)Receive the following alertevent in HX Connect HXController VM HOSTNAME one or more configuredDNS servers not responding

CSCvt13947

Release Notes for Cisco HX Data Platform Release 4031

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)40(2a)API UI CLI show few Drive slots Empty after upgradeto 402a

The drives are seen in the sysmtool lsscsi and stclicommand outputs

The cluster is healthy with no errors

CSCvt14914

40(2b)35(2d)After successfully changing the DNS server on the HXcontroller we still can see the original DNS entry thatwas added during the deployment

CSCvs69154

40(2b)35(2e)HyperFlex Connect and VCenter show APD alarms forHX and non HX datastores This implies there is an issuewith HyperFlex when there may not be

CSCvs30080

40(2b)40(2a)An issue where all the disks in the node get locked afterrebooting the node

We successfully unlocked using sed-clientsh -Ucommand but wanted to test with another reboot anddrives locked again

CSCvs74286

40(2b)40(2a)When running stcli license commands on HyperFlexthe following error is seen

rootSpringpathControllertmp stcli license show all

Show smart licensing failed

Smart Agent is not ready please wait a minute and tryagain

CSCvt13929

40(2b)40(2a)After upgrading to HX 40(2a) the first RoboEdge nodethat gets rebooted then ignores persistent drivesPIDHX-HD24TB10K4KN

Cluster will be in degraded state and have reducedcapacity

CSCvt20144

40(2b)35(2b)Zookeeper fails to start while Exhibitor is runninghowever echo srvr returns nothing

CSCvs21562

40(2b)40(2a)When performing an HyperFlex upgrade a validationwarning may occur due to the host not having EnterprisePlus or Enterprise hypervisor licensing

Upgrade Validation Warning

ESXi host esx1labtest should be configured withVMware Enterprise license for upgrade to continue

CSCvs91787

Release Notes for Cisco HX Data Platform Release 4032

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)35(2g)Cluster expansion fails at Config Installer stage whenthe root and admin password for the storage controller(SCVM) are different

CSCvs69317

40(2b)35(2g)Panic while upgrading ESXiCSCvt06983

40(2b)40(2a)A cluster node running may hang in the Linux kernelThis is classified as an oops and a deviation from theexpected behavior

CSCvs54285

40(2b)35(2g)Rebalance failing on 10+10 node HX 35(2g) stretchcluster

CSCvs69007

40(2b)35(2d)The cluster becomes inaccessible to the IOVisorCSCvr54687

40(2b)40(2a)Panic on the storage controllerCSCvt61297

40(2b)40(2a)The size of support bundle is very large when collectedwith storfs-support command

CSCvt63306

40(2b)40(2a)storfs crash associated withFileSystemUsageWarningEvent orFileSystemUsageAlertEvent due to high usage ofvarstv above 80

CSCvt72807

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

40(2a)35(2d)In normal scenarios HX upgrade gets pendingACK afterupdating service profile In this case UCS only upgradewas stalled because after updating service profile HXupgrade was not getting pending ACK and it was alwayswaiting state

CSCvo39912

40(2a)35(1a)When using motherBoardReplace-12 to clean ZK fromold stNodepNodes resulted in unmounted datastore andmaking the size of the Hyperflex Datastores 0 resultingin all VMs in the cluster going offline

CSCvq39471

40(2a)26(1e)Upgrading cluster and getting error

Failed upgrade validations Checking vCenterconfiguration Reason Upgrade validations failed DRSFault Insufficient resources to satisfy configured failover

CSCvm77294

Release Notes for Cisco HX Data Platform Release 4033

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)The cluster expand fails on a node with DR replicationconfigured

When a HX cluster which has DR replication configuredis expanded we see the installer UI pulling in thereplicationVLAN information instead of themanagementVLAN information Even if we change that informationto the correct mgmt VLAN id and name it it does notseem to work as the node is configured with the VLANof the replicationVLAN in ESXi This leads to the failureof the node add with the host unreachable error

CSCvo70650

40(2a)35(1a)Customer reported that server didnt complete thefirmware upgrade automatically one by one

The user manually needs to put the host into maintenancemode and they manually acknowledged the pendingrequests to finish the UCS firmware upgrade

Maintenance policy is set to default (User-ack) as perthe design

CSCvo91624

40(2a)35(1i)When the cluster is in Failed state and stcli noderemove is attempted the output shows successful thoughthe node failed to get removed from the cluster

CSCvo93017

40(2a)35(1a)HyperFlex cluster upgrade may fail during validationwith a DRS Validation failed error

CSCvp31021

40(2a)35(2b)HX cluster expansion will fail with the following errormessage

MAC address pool configuration failure150[ErrorDescription] bad address block range definitioncollision

CSCvp58318

40(2a)35(2b)Memory usage by carbon cacheCSCvq34873

40(2a)35(2g)Upgrading ESXi cluster fails with error Nodemaintenance mode failed

CSCvr03240

40(2a)35(2e)storfs process does not automatically start on Exit HXMaintenance Mode or other tasks which restart StorageController

CSCvr88978

40(2a)35(2a)Post install script fails with message

Failed to execute ipmitool on HX node

CSCvp10707

40(2a)40(1a)HyperFlex expansionworkflow doesnt pull VLANnamecorrectly

CSCvp46539

Release Notes for Cisco HX Data Platform Release 4034

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)During HX cluster deployment - Validate ClusterCreation phase - HX Installer might fail with the error

fromvarlogspringpathstDeploylog

CSCvq45087

40(2a)35(2b)HX Installer fails to login to the SOL for configuringESX hosts

CSCvq91380

40(2a)35(2d)If DRS is enabled with configuration parameters otherthen fully automated ie manual or partially automatedthan this will change to fully automated during clusterexpansion

CSCvr44222

40(2a)35(1a)One of the validation task name has a typo The taskname is Validating node for mixed of SED and non-SEDdisks

It should be Validating node for mix of SED andnon-SED disks

CSCvp82175

40(2a)30(1i)This combination of using smaller NVMe drives is notsupported in expand operation For example if the drivesare only 375GB and they cannot be added to an existingcluster with larger caching SSD

CSCvo12359

40(2a)35(1a)HyperFlex includes a version of OpenSSL that is affectedby the vulnerabilities identified by the followingCommon Vulnerability and Exposures

CVE-2018-0495

CSCvp66679

40(2a)30(1e)SDK invoke exception when taking a native snapshot ona VM with GPU pci passthrough attached

CSCvo00511

40(2a)30(1i)Node replacement script fails due to EAM errorCSCvo62867

40(2a)35(1a)While pairing a cluster (HX release gt= 35) and cluster(HX release lt 35) remote replication network test failson the cluster (HX release lt 35)

CSCvo79760

40(2a)30(1b)Fixed the issue in the latest support-workflow bundleThis should not be hit anymore

CSCvo87061

40(2a)35(2a)On a 3 node cluster the MBreplace script fails as thescript looks for HEALTHY cluster status

CSCvo87080

40(2a)35(2a)MbReplace script (tar) hangs when running on 352b or40

CSCvp12359

Release Notes for Cisco HX Data Platform Release 4035

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)HX replication cleanup failing with error NA 35(2a) -INFODR state is not clean

CSCvp63958

40(2a)40(1a)The check_stig_parameters API incorrectly shows thecompliance state of the cluster as not STIG compliant

CSCvp66277

40(2a)35(2c)The snapshot creation on CBT enabled VM fails witherror Failed in vmreparent vmkfstools clone1

CSCvq06952

40(2a)35(2a)HyperFlex GUI when setting up remote DR-replicationpartnerpeer should show remote replication partner TCPport reachability test results

This will provide quick upfront connectivity results thatfrequently takes customer a while to manually check andconfirm

If reachability tests fail an info window and HX alertshould trigger that list the destination peer IP and thespecific port(s) that could not be reache

CSCvo60587

40(2a)35(2b)If you have more than one IP pool configured for areplication network and local replication network testsare failing you might be running into this issue

CSCvr67130

40(2a)35(2a)Skip task option in MBreplace and Software redeployscripts The scripts should have the option to skip tasksas user inputs and not by modifying the json file

CSCvp05204

40(2a)40(1a)Error noticed on SCVM Console print_req_error IOerror dev fd0 sector 0 on HyperV

It doesnt come up on SSH session

CSCvq34357

40(2a)30(1d)The VMNetwork switch is already created as part of theinstall process

Ideally the ip address assignment should be done duringthe initial install process as well No post install stepsshould be required altrnatively it could be done as partof a post install script

CSCvk36222

40(2a)25(1c)storfs service impacted during HX upgrade from HX21(1) to 25(1c)

HX node which is not being upgraded storfs servicestopped causing outage

CSCvg53223

40(2a)35(1b)HX Stretch Cluster Witness VM reverts to DHCP onreboot

CSCvp37536

40(2a)35(2b)HyperFlex Stretch Cluster Witness does not getprogrammed with an NTP server

CSCvp46578

Release Notes for Cisco HX Data Platform Release 4036

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2b)HyperFlex cluster does not heal after maintenance onnodeserver

CSCvr18528

40(2a)40(2a)High percentage of packet loss on the eth1 data interface9K MTU

CSCvr97089

40(2a)35(2f)The HyperFlex cluster healing state is stuck at 87 andis unable to progress

CSCvr16760

40(2a)40(2a)In some circumstances two replications can occur forthe same VM

From the GUI the replication status for this VM willshow stuck in an In Progress state but there are nojobs stuck in replication layer Subsequent Replicationsshould succeed

CSCvq22898

40(2a)35(2a)After an unexpected power outage the hyperflex clusteris unable to come up correctly

All processes are running time is synced and all nodescan vmkping each other as required

CSCvp13990

40(2a)40(1a)When specifying IP address instead of node ID stclinode maintenanceMode commands fail

CSCvp33657

40(2a)40(1a)2 node edge cluster if experiencing Intersightarbitratorconnectivity issues should NOT allowHXMaintenancemode to initiate as this will bring the cluster down

CSCvr31573

40(2a)35(2c)Old files in varsupportZKTxnlog are not purged withthe daily zklog-cleanup cron job

CSCvr89066

40(2a)35(2a)When a node is in MaintenanceMode any disk removalor replacement will be reflected in UI only after the nodeis brought back frommaintenancemode This is becausestorfs is not running on the node in maintenance andwill not be able to detect disk activities until it is broughtout of Maintenance Mode

CSCvo86431

40(2a)35(2c)Cluster might go offline and stop serving IO requestsrootcvm~ stcli cluster storage-summary--detailGet cluster storage summary failedjavanetConnectException Connection refused19216814210010207

CSCvr01645

40(2a)35(2d)A stretch cluster deployment fails with a messageFormatting nodes Node in Use on the Hyperflex installerand crmZoneType value is seen as 1 in storfscfg oroptspringpathconfigstretchtunes

CSCvq80340

Release Notes for Cisco HX Data Platform Release 4037

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)30(1c)During upgrade we upload the upgrade packagedownloaded from Ciscocom Upgrade packages are offile type tgz hxconnect acceptsstorfs-packages-401a-33028tar file a different type ofcompression and allows upgrade to start and fail whenit cannot decompress the file for bootstrap and scvmclientupgrade on ESXi

CSCvq63888

40(2a)40(2a)While attempting to install hyperflex via intersight thefollowing error is seen

Failed in Task Add hosts to vCenter Cluster with ErrorTry adding hosts manually to vCenter and retry Failedto get addressFailed to host by namelookupHostnamecompanycom on 000053 read udp12700145430-gt12700153 read connection refused

CSCvr52098

40(2a)35(2a)CoreAPI call may time out for cluster management APIwhile doing cluster create validation

CSCvo92952

40(2a)35(2d)Validation fails with Mixed mode expansion checkdue to Enhanced vMotion Compatibility Incompatibilitywhen the ESXi version is different

CSCvq95460

40(2a)30(1i)This failure is seen in installer when custom workflowis used hypervisor configuration + deploy (clean diskpartition=no) at Installing software packages on storagecontroller VM

CSCvr66309

40(2a)35(2d)The HX Installer apparently replaced a comma bysemicolon on the VLAN ID range while exporting theconfiguration to a JSON file

CSCvq93831

40(2a)40(1a)When using unsupported browsers HX Connect userswill not be able to login These users see an errorindicating unauthorized user

CSCvp22693

Release Notes for Cisco HX Data Platform Release 4038

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)40(1a)A vulnerability in the web-based management interfaceof Cisco HyperFlex Software could allow anunauthenticated remote attacker to conduct a cross-siterequest forgery (CSRF) attack on an affected system

The vulnerability is due to insufficient CSRF protectionsfor the web UI on an affected device An attacker couldexploit this vulnerability by persuading a user of theinterface to follow a malicious link A successful exploitcould allow the attacker to perform arbitrary actions withthe privilege level of the affected user

Cisco has released software updates that address thisvulnerability There are no workarounds that address thisvulnerability

This advisory is available at the following link

httpstoolsciscocomsecuritycentercontentCiscoSecurityAdvisorycisco-sa-20190807-hypflex-csrf

CSCvp24343

40(2a)35(2d)Add message to HX Connect Progress flow to notAcknowledge Pending Activities and reboot the serverson UCSM

HX Connect is doing a controlled rolling server upgradein the background

CSCvq22844

40(2a)40(1b)Compute Node expand operation fails due to incorrectcluster name

CSCvr43786

40(2a)26(1e)RF-2 cluster shutdown during data resynchronizationafter a node failure (non storfs) followed by multipledisk read failures and a HardBlacklist of a disk

CSCvp41241

40(2a)40(1a)HyperFlex Edge nodes do not properly set the ESXihostname during deployment

CSCvo13143

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4039

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4040

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1b) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401b-33133p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4041

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4042

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1b)40(1a)While running the HyperFlex installer with WindowsServer Hyper-V cluster creation process fails with theerror Failure occurred during ClusterCreation process Unable to post thecontent to the downstream This symptomis encountered while deploying HyperFlex cluster withHyperFlex nodes configuredordered with Cisco VIC1457 MLOM (PID HX-MLOM-C25Q-04) andWindows Server Datacenter or Core with Hyper-Vversion 2016 or 2019

CSCvp64140

Release Notes for Cisco HX Data Platform Release 4043

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)35(2b)Adding Micron 5200 drive to a cluster fails to increasecluster capacity

CSCvo69067

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4044

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4045

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1a) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401a-33028p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4046

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4047

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1a)30(1d)Cluster instability when disks of different sector sizeplaced in HX node

CSCvk17250

40(1a)

35(2c)

35(1a)When logged in using a local HX user account insteadof a Virtual Center account an error message appearsintermittently indicating Virtual Centerunreachable or Resource informationcannot be updated when VC is reachable

CSCvo36198

40(1a)

35(2b)

30(1d)HXDP does not work with EMC RecoverPoint - needsto support VMware API (FSS-Readdir)

CSCvk38003

Release Notes for Cisco HX Data Platform Release 4048

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)

35(2a)

30(1i)

25(1c)

25(1d)

Zookeeper (Exhibitor) process on HyperFlex StorageController VMs may fail to respond to requests whenvarzookeeper has low or no free space

CSCvm90352

40(1a)26(1c)Use Asynchronous consolidation for HX SnapshotsCSCvn02151

40(1a)35(2b)HX Quiesced Snapshot for Backup VendorsCSCvo90713

40(1a)35(2a)The cluster creationcluster expansion workflow stopswith the following error message at the validation step

FIRMWARE-Check UCSC-SAS-M5HD

FIRMWARE-Check UCSC-SAS-M5HD Required00000029000000320000003500000050 Found00000058

Action Needed Update the Controller Firmware toRequired Version

CSCvn17787

35(2b)30(1d)Kernel migration fails when a local datastore is searchedfor in the ESXi host

CSCvn73127

35(2a)26(1b)A node shuts down when two disks are replaced (acaching disk and another capacity disk) where thecapacity disk is inserted first and then the caching diskis inserted

CSCvk46364

Hyper-V

40(1a)35(2a)Cluster expansion may fail with an error code 500 -operation timed out

CSCvn28721

35(2b)35(2a)During upgrade remove the VLAN on team created foruser vSwitch During fresh installation only one VLANtag is set to the vSwitch and team although multipleVLANs were entered

CSCvn54300

35(2b)35(2a)While upgrading a Hyper-V cluster on account of a rarerace condition between the stUpgradeService andZookeeper servers the upgrade orchestration throws anupgrade validation error and the upgrade process isaborted

CSCvn60486

Release Notes for Cisco HX Data Platform Release 4049

Resolved Caveats in Release 40(1a)

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Startrestart EAM service invCenter

When EAM service is not runningvCenter registration is failing

CSCvt55712

35(2a)Contact Cisco TAC if you believeyou are seeing similar conditions

HyperFlex Stretch Cluster saw poorapplication performance due to writelatency cluster remained unhealthy andrebalance was stuck

This defect is currently being used forinvestigative purposes

CSCvt45344

35(2g)Remove log files fromvarlogspringpath folder when thelog files are not getting rotatedStop the relevant service removelarge log files fromvarlogspringpath folder andrestart the service

varstv folder in Controller VM maybecome full

CSCvt36374

40(2b)Restarting the stMgr will initializeit again

In a Cluster configured with DR networkStMgr may not get initialized and stuckin a deadlock while enabling the IPTablerules (can be verified from the stMgrlog)

CSCvt89709

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Click recover for the selectedVM again under the standalonemode This will move the VM torecovered state

When the user recovers a VM protectedin a group it along with other VMs in thegroup move to recovering state instandalonemode The selectedVM shouldmove to recovered state

CSCvs75553

35(2e)Clean up the fail over cluster andthen recreate the fail over clusterNo need to touch the HX storagecluster

Hyper-V When replicated DC was usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2d)40(1a)

Keep cluster utilization as low aspossible To avoid read-only statekeep below 92

HyperFlex clusters storage utilization isabove 92 usable capacity

CSCvr20922

Release Notes for Cisco HX Data Platform Release 4050

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1b)NAAll the disks in the node got locked afterrebooting the node

We successfully unlocked usingsed-clientsh -U command but wantedto test with another reboot and driveslocked again

CSCvs74286

35(2g)To recover the VM copy over thedata disks and attach them to thenew VM

After enable encryption HX connectencryption status showing Caution butall of the disks enable encryptionsuccessful

-Self Encrypting Drives Service isrunning on all of the nodes

-There is no error message of encryption

- USB0 interface is up

-All SED disks showing supported 1enabled 1 locked 0

CSCvs93245

40(1b)Enable kernelpanic_on_oops as apersistent configuration This willcause the node to panic and rebootimmediately

A cluster node running HX release40(1b) may hang in the Linux kernelThis is classified as an oops and adeviation from the expected behavior

CSCvs54285

Release Notes for Cisco HX Data Platform Release 4051

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Enabling K8iscsi stack in Hx clusterCSCvs41324

Release Notes for Cisco HX Data Platform Release 4052

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Steps to enable K8 on HX cluster

Perform following steps on ALLthe controller VMs

1 Updateetcinitscvmclientconf toenable tune iscsiEnable totrue

sed -iesiscsiEnable=falseiscsiEnable=trueetcinitscvmclientconf

2 Run following initctlcommand to reloadconfiguration

initctl reload-configuration

3 Restart scvmclient process

stop scvmclient startscvmclient

4 Run following initctlcommand to mount SYSTEMDS

initctl emit --no-waitsystem-datastore-created

5 Verify that iscsiEnable tuneis set to true using thefollowing command

ps -eaf | grep scvmclient |grep -v grep

root 6241 1 1 Dec06 014007optspringpathstorfs-corescvmclient-T logEnabled=false -TlogSyslogEnabled=true -TlogEchoToScreen=false -TstatLoggingToFile=false -TstatLoggingToSyslog=true-TlogDir=varlogspringpath-TnfsBackendServerList=1010748100-T iscsiEnable=true -TiscsiUseAsync=true -T

Release Notes for Cisco HX Data Platform Release 4053

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

iscsiConfigFilePath=nfsSYSTEMistgtconfps -eaf | grep scvmclient |grep -v grep

35(2e)Following procedure can be usedto check NFS Queue Depth andincrease if needed

rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth256 lt- Low value[rootHXESXI1] vsish -e setvmkModulesnfsclientmnt[DATASTORE]propertiesmaxQDepth 1024[rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth1024 lt- Optimalvalue

This requires an ESXi host rebootto take effect

Please place the node in HyperflexMaintenanceMode and gracefullyreboot the node for the changes tobe applied

HyperFlex Datastore NFS Queue Depthshows as 256 which can lead toperformance (including latency) issues

CSCvr83056

40(1b)This is cosmetic The error will goaway once the cluster returns tohealthy status

When a single node is offline in clusterstcli cluster storage-summary shows twonodes unavailable rootSCVM~ stclicluster storage-summary

CSCvs47735

35(2g)Modify both the root and adminpassword for the SCVMs to be thesame

stcli security password set -uadmin

Cluster expansion fails at Config Installerstage when the root and admin passwordfor the storage controller (SCVM) aredifferent

CSCvs69317

35(2d)NAAfter successfully changing the DNSserver on the HX controller we still cansee the original DNS entry that was addedduring the deployment

CSCvs69154

Release Notes for Cisco HX Data Platform Release 4054

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Customers are strongly encouragedto work with TAC in order toidentify and remediate this issue

1 Change getcluster from 2m to8m inoptspringpathstorfs-mgmtstMgr-10confapplicationconfon all nodes

2 Restart stMgr on all nodes

You may see this error message after afailed upgrade or other task such asattempting to enter a node into HXmaintenance mode

CSCvs53555

40(2a)Run the following command oneach storage controller VM as root

grep -i monitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp sed -iesmonitor_dns_serverstruemonitor_dns_serversfalseoptspringpathhx-diag-toolswatchdog_configjsonampamp grep -imonitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp restart watchdog

Receive the following alertevent in HXConnect HX Controller VMHOSTNAME one or more configuredDNS servers not responding

CSCvt13947

35(2b)Delete any empty (Size 0) log filesunder varlogzookeeperversion-2ORvarlogzookeeperstandaloneversion-2

Zookeeper fails to start while Exhibitor isrunning however echo srvr returnsnothing

CSCvs21562

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

40(4e)1 De-commission the server

2 Power drain the server -REMOVE BOTH POWERCORDS ON THE BACK OFTHE SERVER FOR 10SECONDS THENREINSERTPOWERCORDS

3 Re-commission the server

M2 boot disk is missing from serverinventory after upgrade to server firmware40(4e) As a result server fails to boot toOS installed in the M2 disk The issuepersists after re-acknowledgement as wellas de-commission andre-acknowledgement of the server

CSCvs02466

Release Notes for Cisco HX Data Platform Release 4055

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Re-start the flusher manuallyMany disks taken offline by a fault ormanually resulting in ENOSPC oncluster

CSCvq32721

35(2e)Clean up fail over cluster and thenrecreate the fail over cluster Thereis no need to touch the HX storagecluster

[Hyper-V]When replicated DCwas usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2b)Manually install upgradescvmclient VIB andmake sure thatit matches with HXDP version

Upgrade from release HX 35(2a) to35(2x) 4X does not upgrade scvmclientRecovery point may not work

CSCvq54992

Management

30(1b)To recover the VM copy over thedata disks and attach them to thenew VM

[HyperV]VMshows up onmultiple nodesCSCvj22992

35(1a)1 Do not change MTU at ESXlevel on running cluster

2 Reset back to the original value

ChangingMTU from 9000 to 1500 on oneESX node causes storfs process to restarton all nodes

CSCvm99150

Release Notes for Cisco HX Data Platform Release 4056

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)After the installation (for newdeployments) or after an upgradeto 401a (for existingdeployments) perform thefollowing steps on all ControllerVMs

1 Edit the following tune files (onall Controller VMs)

optspringpathconfiglfftunes

optspringpathconfigvsi_16tbtunes

2 SetcleanerEnableSegSummaryCleaning

to false

3 After editing the above tunefiles

a SSH in to all the ControllerVMs

b Run storfstool -- -Z

c Run the below command andcheck the values Tune valueshould be true

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=true

d Type below commands andapplied the tunes changesdynamically

echo false gttmpstprocfssystemtunecleanerEnableSegSummaryCleaning

e Verify that tune values arechanged Run below commandValue should be false

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=false

f umount tmpstprocfs

A cluster with 8TB or 12TB disk drivesmay experience IO stalls for severalminutes after another node fails in thecluster

CSCvp23718

Release Notes for Cisco HX Data Platform Release 4057

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)1 Update the conf file manuallyto change to the link local addressto

2 Restart scvmclient on thecontroller and ESX

HX 35(2b) FlexVol to HX 40 CSIupgrade does not work FlexVol continuesto work

CSCvp26319

35(2a)Disable rebalance until node healsthen reenable rebalance

7-node cluster healing very slowlybecause rebalance change AMS reqs areblocking healing process

CSCvp62512

35(2a)NAWith moderate cluster storage capacityusage under site failover cluster clustermay experience an all paths (APD) downcondition

CSCvq49412

40(1a)NAThis product includes Third-partySoftware that is affected by thevulnerabilities identified by the followingCommon Vulnerability and Exposures(CVE) IDs

CVE-2016-0762 CVE-2016-6797CVE-2016-6816 CVE-2016-8735CVE-2017-5647 CVE-2017-12615CVE-2017-12616 CVE-2017-12617CVE-2017-7674 CVE-2018-1304CVE-2018-8014 CVE-2018-1336CVE-2018-8034 CVE-2018-11784CVE-2019-0232

CSCvp36364

35(2d)NAstcli cluster info command needs toprovide UCSM VIP address Currently itshows ucsm-hostcom

CSCvq11456

35(2a)No workarounds currently Thestale entries do not affect the dayto day operations of HyperFlex

HyperFlex upgrade validation failedbecause cluster has extraneous stNodes ininternal database

CSCvq32530

35(2a)NAWhen not in maintenancemode all drivesappear in HX Connect and get updated asexpected

When in maintenance mode extra drivesare shown in HX Connect with no slotnumbers

CSCvq42378

35(2b)Contact TAC to cleanup duplicatenode

Nodes duplicated in HX Connect aftercluster recreate

CSCvq61825

Release Notes for Cisco HX Data Platform Release 4058

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)NACurrently HyperFlex installs on Hyper-Vdo not contain the stcli security whitelistcommandset

CSCvq66245

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

Release Notes for Cisco HX Data Platform Release 4059

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)[Hyper-V] Converged node expansionfails for incoming node with VIC 14xx

CSCvq39523

Release Notes for Cisco HX Data Platform Release 4060

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

1 Set Jumbo frame inWindowsHypervisor for incomingnode

Get-NetAdapter storage-data-a|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

Get-NetAdapter storage-data-b|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

2 Verify all interfaces are setcorrectly to support Jumbo frames

Get-NetAdapter storage |Get-NetAdapterAdvancedProperty| RegistryKeyword -matchjumbo | ft -auto

You should receive the followingmessage

Name DisplayName DisplayValue

RegistryKeyword RegistryValue

---- ----------- ------------

--------------- -------------

vswitch-hx-storage-data Jumbo

Packet 9014 Bytes

JumboPacket 9014

storage-data-b Jumbo Packet

Bytes 9014 JumboPacket

9014

storage-data-a Jumbo Packet

Bytes 9014 JumboPacket

9014

3 Reboot only the incoming nodeor the node being expanded

4 Retry Cluster Expansion in

Release Notes for Cisco HX Data Platform Release 4061

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

DescriptionCVEDefect IDRelease

The vulnerabilities associated withFasterXML Jackson-Databind TimeValueField Denial of Service

CVE-2018-14719

CVE-2018-14720

CVE-2018-1000873

CVE-2018-14721

CVE-2018-19360

CVE-2018-19362

CVE-2018-19361

CVE-2018-14718

CSCvn8228240(1a)

The vulnerabilities associated with theOpenSSL Protocol software packageversion included in Cisco HX DataPlatform

CVE-2013-3587CSCvo0505440(1a)

The vulnerabilities associated withThird-Party Software Denial of Service

CVE-2018-16487

CVE-2018-19361

CSCvo2781840(1a)

The vulnerabilities associated withLinuxkernel for Ubuntu 1710

CVE-2018-1092

CVE-2018-7492

CVE-2018-8087

CVE-2018-1068

CVE-2018-8781

CSCvm5314935(2a)

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)25(1d)stcli services dns remove should remove the DNS serverinfo from the interface files

CSCvs70967

40(2b)40(1b)HX 40(1b) new deployment from Intersight productioncloud

User receives following error while enabling CSI(Kubernetes integration)

Failure occurred during volume_access_enable Errorwas Nonzero exit code 1

CSCvt10522

40(2b)40(2a)Receive the following alertevent in HX Connect HXController VM HOSTNAME one or more configuredDNS servers not responding

CSCvt13947

Release Notes for Cisco HX Data Platform Release 4031

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)40(2a)API UI CLI show few Drive slots Empty after upgradeto 402a

The drives are seen in the sysmtool lsscsi and stclicommand outputs

The cluster is healthy with no errors

CSCvt14914

40(2b)35(2d)After successfully changing the DNS server on the HXcontroller we still can see the original DNS entry thatwas added during the deployment

CSCvs69154

40(2b)35(2e)HyperFlex Connect and VCenter show APD alarms forHX and non HX datastores This implies there is an issuewith HyperFlex when there may not be

CSCvs30080

40(2b)40(2a)An issue where all the disks in the node get locked afterrebooting the node

We successfully unlocked using sed-clientsh -Ucommand but wanted to test with another reboot anddrives locked again

CSCvs74286

40(2b)40(2a)When running stcli license commands on HyperFlexthe following error is seen

rootSpringpathControllertmp stcli license show all

Show smart licensing failed

Smart Agent is not ready please wait a minute and tryagain

CSCvt13929

40(2b)40(2a)After upgrading to HX 40(2a) the first RoboEdge nodethat gets rebooted then ignores persistent drivesPIDHX-HD24TB10K4KN

Cluster will be in degraded state and have reducedcapacity

CSCvt20144

40(2b)35(2b)Zookeeper fails to start while Exhibitor is runninghowever echo srvr returns nothing

CSCvs21562

40(2b)40(2a)When performing an HyperFlex upgrade a validationwarning may occur due to the host not having EnterprisePlus or Enterprise hypervisor licensing

Upgrade Validation Warning

ESXi host esx1labtest should be configured withVMware Enterprise license for upgrade to continue

CSCvs91787

Release Notes for Cisco HX Data Platform Release 4032

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)35(2g)Cluster expansion fails at Config Installer stage whenthe root and admin password for the storage controller(SCVM) are different

CSCvs69317

40(2b)35(2g)Panic while upgrading ESXiCSCvt06983

40(2b)40(2a)A cluster node running may hang in the Linux kernelThis is classified as an oops and a deviation from theexpected behavior

CSCvs54285

40(2b)35(2g)Rebalance failing on 10+10 node HX 35(2g) stretchcluster

CSCvs69007

40(2b)35(2d)The cluster becomes inaccessible to the IOVisorCSCvr54687

40(2b)40(2a)Panic on the storage controllerCSCvt61297

40(2b)40(2a)The size of support bundle is very large when collectedwith storfs-support command

CSCvt63306

40(2b)40(2a)storfs crash associated withFileSystemUsageWarningEvent orFileSystemUsageAlertEvent due to high usage ofvarstv above 80

CSCvt72807

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

40(2a)35(2d)In normal scenarios HX upgrade gets pendingACK afterupdating service profile In this case UCS only upgradewas stalled because after updating service profile HXupgrade was not getting pending ACK and it was alwayswaiting state

CSCvo39912

40(2a)35(1a)When using motherBoardReplace-12 to clean ZK fromold stNodepNodes resulted in unmounted datastore andmaking the size of the Hyperflex Datastores 0 resultingin all VMs in the cluster going offline

CSCvq39471

40(2a)26(1e)Upgrading cluster and getting error

Failed upgrade validations Checking vCenterconfiguration Reason Upgrade validations failed DRSFault Insufficient resources to satisfy configured failover

CSCvm77294

Release Notes for Cisco HX Data Platform Release 4033

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)The cluster expand fails on a node with DR replicationconfigured

When a HX cluster which has DR replication configuredis expanded we see the installer UI pulling in thereplicationVLAN information instead of themanagementVLAN information Even if we change that informationto the correct mgmt VLAN id and name it it does notseem to work as the node is configured with the VLANof the replicationVLAN in ESXi This leads to the failureof the node add with the host unreachable error

CSCvo70650

40(2a)35(1a)Customer reported that server didnt complete thefirmware upgrade automatically one by one

The user manually needs to put the host into maintenancemode and they manually acknowledged the pendingrequests to finish the UCS firmware upgrade

Maintenance policy is set to default (User-ack) as perthe design

CSCvo91624

40(2a)35(1i)When the cluster is in Failed state and stcli noderemove is attempted the output shows successful thoughthe node failed to get removed from the cluster

CSCvo93017

40(2a)35(1a)HyperFlex cluster upgrade may fail during validationwith a DRS Validation failed error

CSCvp31021

40(2a)35(2b)HX cluster expansion will fail with the following errormessage

MAC address pool configuration failure150[ErrorDescription] bad address block range definitioncollision

CSCvp58318

40(2a)35(2b)Memory usage by carbon cacheCSCvq34873

40(2a)35(2g)Upgrading ESXi cluster fails with error Nodemaintenance mode failed

CSCvr03240

40(2a)35(2e)storfs process does not automatically start on Exit HXMaintenance Mode or other tasks which restart StorageController

CSCvr88978

40(2a)35(2a)Post install script fails with message

Failed to execute ipmitool on HX node

CSCvp10707

40(2a)40(1a)HyperFlex expansionworkflow doesnt pull VLANnamecorrectly

CSCvp46539

Release Notes for Cisco HX Data Platform Release 4034

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)During HX cluster deployment - Validate ClusterCreation phase - HX Installer might fail with the error

fromvarlogspringpathstDeploylog

CSCvq45087

40(2a)35(2b)HX Installer fails to login to the SOL for configuringESX hosts

CSCvq91380

40(2a)35(2d)If DRS is enabled with configuration parameters otherthen fully automated ie manual or partially automatedthan this will change to fully automated during clusterexpansion

CSCvr44222

40(2a)35(1a)One of the validation task name has a typo The taskname is Validating node for mixed of SED and non-SEDdisks

It should be Validating node for mix of SED andnon-SED disks

CSCvp82175

40(2a)30(1i)This combination of using smaller NVMe drives is notsupported in expand operation For example if the drivesare only 375GB and they cannot be added to an existingcluster with larger caching SSD

CSCvo12359

40(2a)35(1a)HyperFlex includes a version of OpenSSL that is affectedby the vulnerabilities identified by the followingCommon Vulnerability and Exposures

CVE-2018-0495

CSCvp66679

40(2a)30(1e)SDK invoke exception when taking a native snapshot ona VM with GPU pci passthrough attached

CSCvo00511

40(2a)30(1i)Node replacement script fails due to EAM errorCSCvo62867

40(2a)35(1a)While pairing a cluster (HX release gt= 35) and cluster(HX release lt 35) remote replication network test failson the cluster (HX release lt 35)

CSCvo79760

40(2a)30(1b)Fixed the issue in the latest support-workflow bundleThis should not be hit anymore

CSCvo87061

40(2a)35(2a)On a 3 node cluster the MBreplace script fails as thescript looks for HEALTHY cluster status

CSCvo87080

40(2a)35(2a)MbReplace script (tar) hangs when running on 352b or40

CSCvp12359

Release Notes for Cisco HX Data Platform Release 4035

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)HX replication cleanup failing with error NA 35(2a) -INFODR state is not clean

CSCvp63958

40(2a)40(1a)The check_stig_parameters API incorrectly shows thecompliance state of the cluster as not STIG compliant

CSCvp66277

40(2a)35(2c)The snapshot creation on CBT enabled VM fails witherror Failed in vmreparent vmkfstools clone1

CSCvq06952

40(2a)35(2a)HyperFlex GUI when setting up remote DR-replicationpartnerpeer should show remote replication partner TCPport reachability test results

This will provide quick upfront connectivity results thatfrequently takes customer a while to manually check andconfirm

If reachability tests fail an info window and HX alertshould trigger that list the destination peer IP and thespecific port(s) that could not be reache

CSCvo60587

40(2a)35(2b)If you have more than one IP pool configured for areplication network and local replication network testsare failing you might be running into this issue

CSCvr67130

40(2a)35(2a)Skip task option in MBreplace and Software redeployscripts The scripts should have the option to skip tasksas user inputs and not by modifying the json file

CSCvp05204

40(2a)40(1a)Error noticed on SCVM Console print_req_error IOerror dev fd0 sector 0 on HyperV

It doesnt come up on SSH session

CSCvq34357

40(2a)30(1d)The VMNetwork switch is already created as part of theinstall process

Ideally the ip address assignment should be done duringthe initial install process as well No post install stepsshould be required altrnatively it could be done as partof a post install script

CSCvk36222

40(2a)25(1c)storfs service impacted during HX upgrade from HX21(1) to 25(1c)

HX node which is not being upgraded storfs servicestopped causing outage

CSCvg53223

40(2a)35(1b)HX Stretch Cluster Witness VM reverts to DHCP onreboot

CSCvp37536

40(2a)35(2b)HyperFlex Stretch Cluster Witness does not getprogrammed with an NTP server

CSCvp46578

Release Notes for Cisco HX Data Platform Release 4036

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2b)HyperFlex cluster does not heal after maintenance onnodeserver

CSCvr18528

40(2a)40(2a)High percentage of packet loss on the eth1 data interface9K MTU

CSCvr97089

40(2a)35(2f)The HyperFlex cluster healing state is stuck at 87 andis unable to progress

CSCvr16760

40(2a)40(2a)In some circumstances two replications can occur forthe same VM

From the GUI the replication status for this VM willshow stuck in an In Progress state but there are nojobs stuck in replication layer Subsequent Replicationsshould succeed

CSCvq22898

40(2a)35(2a)After an unexpected power outage the hyperflex clusteris unable to come up correctly

All processes are running time is synced and all nodescan vmkping each other as required

CSCvp13990

40(2a)40(1a)When specifying IP address instead of node ID stclinode maintenanceMode commands fail

CSCvp33657

40(2a)40(1a)2 node edge cluster if experiencing Intersightarbitratorconnectivity issues should NOT allowHXMaintenancemode to initiate as this will bring the cluster down

CSCvr31573

40(2a)35(2c)Old files in varsupportZKTxnlog are not purged withthe daily zklog-cleanup cron job

CSCvr89066

40(2a)35(2a)When a node is in MaintenanceMode any disk removalor replacement will be reflected in UI only after the nodeis brought back frommaintenancemode This is becausestorfs is not running on the node in maintenance andwill not be able to detect disk activities until it is broughtout of Maintenance Mode

CSCvo86431

40(2a)35(2c)Cluster might go offline and stop serving IO requestsrootcvm~ stcli cluster storage-summary--detailGet cluster storage summary failedjavanetConnectException Connection refused19216814210010207

CSCvr01645

40(2a)35(2d)A stretch cluster deployment fails with a messageFormatting nodes Node in Use on the Hyperflex installerand crmZoneType value is seen as 1 in storfscfg oroptspringpathconfigstretchtunes

CSCvq80340

Release Notes for Cisco HX Data Platform Release 4037

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)30(1c)During upgrade we upload the upgrade packagedownloaded from Ciscocom Upgrade packages are offile type tgz hxconnect acceptsstorfs-packages-401a-33028tar file a different type ofcompression and allows upgrade to start and fail whenit cannot decompress the file for bootstrap and scvmclientupgrade on ESXi

CSCvq63888

40(2a)40(2a)While attempting to install hyperflex via intersight thefollowing error is seen

Failed in Task Add hosts to vCenter Cluster with ErrorTry adding hosts manually to vCenter and retry Failedto get addressFailed to host by namelookupHostnamecompanycom on 000053 read udp12700145430-gt12700153 read connection refused

CSCvr52098

40(2a)35(2a)CoreAPI call may time out for cluster management APIwhile doing cluster create validation

CSCvo92952

40(2a)35(2d)Validation fails with Mixed mode expansion checkdue to Enhanced vMotion Compatibility Incompatibilitywhen the ESXi version is different

CSCvq95460

40(2a)30(1i)This failure is seen in installer when custom workflowis used hypervisor configuration + deploy (clean diskpartition=no) at Installing software packages on storagecontroller VM

CSCvr66309

40(2a)35(2d)The HX Installer apparently replaced a comma bysemicolon on the VLAN ID range while exporting theconfiguration to a JSON file

CSCvq93831

40(2a)40(1a)When using unsupported browsers HX Connect userswill not be able to login These users see an errorindicating unauthorized user

CSCvp22693

Release Notes for Cisco HX Data Platform Release 4038

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)40(1a)A vulnerability in the web-based management interfaceof Cisco HyperFlex Software could allow anunauthenticated remote attacker to conduct a cross-siterequest forgery (CSRF) attack on an affected system

The vulnerability is due to insufficient CSRF protectionsfor the web UI on an affected device An attacker couldexploit this vulnerability by persuading a user of theinterface to follow a malicious link A successful exploitcould allow the attacker to perform arbitrary actions withthe privilege level of the affected user

Cisco has released software updates that address thisvulnerability There are no workarounds that address thisvulnerability

This advisory is available at the following link

httpstoolsciscocomsecuritycentercontentCiscoSecurityAdvisorycisco-sa-20190807-hypflex-csrf

CSCvp24343

40(2a)35(2d)Add message to HX Connect Progress flow to notAcknowledge Pending Activities and reboot the serverson UCSM

HX Connect is doing a controlled rolling server upgradein the background

CSCvq22844

40(2a)40(1b)Compute Node expand operation fails due to incorrectcluster name

CSCvr43786

40(2a)26(1e)RF-2 cluster shutdown during data resynchronizationafter a node failure (non storfs) followed by multipledisk read failures and a HardBlacklist of a disk

CSCvp41241

40(2a)40(1a)HyperFlex Edge nodes do not properly set the ESXihostname during deployment

CSCvo13143

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4039

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4040

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1b) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401b-33133p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4041

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4042

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1b)40(1a)While running the HyperFlex installer with WindowsServer Hyper-V cluster creation process fails with theerror Failure occurred during ClusterCreation process Unable to post thecontent to the downstream This symptomis encountered while deploying HyperFlex cluster withHyperFlex nodes configuredordered with Cisco VIC1457 MLOM (PID HX-MLOM-C25Q-04) andWindows Server Datacenter or Core with Hyper-Vversion 2016 or 2019

CSCvp64140

Release Notes for Cisco HX Data Platform Release 4043

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)35(2b)Adding Micron 5200 drive to a cluster fails to increasecluster capacity

CSCvo69067

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4044

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4045

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1a) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401a-33028p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4046

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4047

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1a)30(1d)Cluster instability when disks of different sector sizeplaced in HX node

CSCvk17250

40(1a)

35(2c)

35(1a)When logged in using a local HX user account insteadof a Virtual Center account an error message appearsintermittently indicating Virtual Centerunreachable or Resource informationcannot be updated when VC is reachable

CSCvo36198

40(1a)

35(2b)

30(1d)HXDP does not work with EMC RecoverPoint - needsto support VMware API (FSS-Readdir)

CSCvk38003

Release Notes for Cisco HX Data Platform Release 4048

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)

35(2a)

30(1i)

25(1c)

25(1d)

Zookeeper (Exhibitor) process on HyperFlex StorageController VMs may fail to respond to requests whenvarzookeeper has low or no free space

CSCvm90352

40(1a)26(1c)Use Asynchronous consolidation for HX SnapshotsCSCvn02151

40(1a)35(2b)HX Quiesced Snapshot for Backup VendorsCSCvo90713

40(1a)35(2a)The cluster creationcluster expansion workflow stopswith the following error message at the validation step

FIRMWARE-Check UCSC-SAS-M5HD

FIRMWARE-Check UCSC-SAS-M5HD Required00000029000000320000003500000050 Found00000058

Action Needed Update the Controller Firmware toRequired Version

CSCvn17787

35(2b)30(1d)Kernel migration fails when a local datastore is searchedfor in the ESXi host

CSCvn73127

35(2a)26(1b)A node shuts down when two disks are replaced (acaching disk and another capacity disk) where thecapacity disk is inserted first and then the caching diskis inserted

CSCvk46364

Hyper-V

40(1a)35(2a)Cluster expansion may fail with an error code 500 -operation timed out

CSCvn28721

35(2b)35(2a)During upgrade remove the VLAN on team created foruser vSwitch During fresh installation only one VLANtag is set to the vSwitch and team although multipleVLANs were entered

CSCvn54300

35(2b)35(2a)While upgrading a Hyper-V cluster on account of a rarerace condition between the stUpgradeService andZookeeper servers the upgrade orchestration throws anupgrade validation error and the upgrade process isaborted

CSCvn60486

Release Notes for Cisco HX Data Platform Release 4049

Resolved Caveats in Release 40(1a)

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Startrestart EAM service invCenter

When EAM service is not runningvCenter registration is failing

CSCvt55712

35(2a)Contact Cisco TAC if you believeyou are seeing similar conditions

HyperFlex Stretch Cluster saw poorapplication performance due to writelatency cluster remained unhealthy andrebalance was stuck

This defect is currently being used forinvestigative purposes

CSCvt45344

35(2g)Remove log files fromvarlogspringpath folder when thelog files are not getting rotatedStop the relevant service removelarge log files fromvarlogspringpath folder andrestart the service

varstv folder in Controller VM maybecome full

CSCvt36374

40(2b)Restarting the stMgr will initializeit again

In a Cluster configured with DR networkStMgr may not get initialized and stuckin a deadlock while enabling the IPTablerules (can be verified from the stMgrlog)

CSCvt89709

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Click recover for the selectedVM again under the standalonemode This will move the VM torecovered state

When the user recovers a VM protectedin a group it along with other VMs in thegroup move to recovering state instandalonemode The selectedVM shouldmove to recovered state

CSCvs75553

35(2e)Clean up the fail over cluster andthen recreate the fail over clusterNo need to touch the HX storagecluster

Hyper-V When replicated DC was usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2d)40(1a)

Keep cluster utilization as low aspossible To avoid read-only statekeep below 92

HyperFlex clusters storage utilization isabove 92 usable capacity

CSCvr20922

Release Notes for Cisco HX Data Platform Release 4050

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1b)NAAll the disks in the node got locked afterrebooting the node

We successfully unlocked usingsed-clientsh -U command but wantedto test with another reboot and driveslocked again

CSCvs74286

35(2g)To recover the VM copy over thedata disks and attach them to thenew VM

After enable encryption HX connectencryption status showing Caution butall of the disks enable encryptionsuccessful

-Self Encrypting Drives Service isrunning on all of the nodes

-There is no error message of encryption

- USB0 interface is up

-All SED disks showing supported 1enabled 1 locked 0

CSCvs93245

40(1b)Enable kernelpanic_on_oops as apersistent configuration This willcause the node to panic and rebootimmediately

A cluster node running HX release40(1b) may hang in the Linux kernelThis is classified as an oops and adeviation from the expected behavior

CSCvs54285

Release Notes for Cisco HX Data Platform Release 4051

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Enabling K8iscsi stack in Hx clusterCSCvs41324

Release Notes for Cisco HX Data Platform Release 4052

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Steps to enable K8 on HX cluster

Perform following steps on ALLthe controller VMs

1 Updateetcinitscvmclientconf toenable tune iscsiEnable totrue

sed -iesiscsiEnable=falseiscsiEnable=trueetcinitscvmclientconf

2 Run following initctlcommand to reloadconfiguration

initctl reload-configuration

3 Restart scvmclient process

stop scvmclient startscvmclient

4 Run following initctlcommand to mount SYSTEMDS

initctl emit --no-waitsystem-datastore-created

5 Verify that iscsiEnable tuneis set to true using thefollowing command

ps -eaf | grep scvmclient |grep -v grep

root 6241 1 1 Dec06 014007optspringpathstorfs-corescvmclient-T logEnabled=false -TlogSyslogEnabled=true -TlogEchoToScreen=false -TstatLoggingToFile=false -TstatLoggingToSyslog=true-TlogDir=varlogspringpath-TnfsBackendServerList=1010748100-T iscsiEnable=true -TiscsiUseAsync=true -T

Release Notes for Cisco HX Data Platform Release 4053

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

iscsiConfigFilePath=nfsSYSTEMistgtconfps -eaf | grep scvmclient |grep -v grep

35(2e)Following procedure can be usedto check NFS Queue Depth andincrease if needed

rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth256 lt- Low value[rootHXESXI1] vsish -e setvmkModulesnfsclientmnt[DATASTORE]propertiesmaxQDepth 1024[rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth1024 lt- Optimalvalue

This requires an ESXi host rebootto take effect

Please place the node in HyperflexMaintenanceMode and gracefullyreboot the node for the changes tobe applied

HyperFlex Datastore NFS Queue Depthshows as 256 which can lead toperformance (including latency) issues

CSCvr83056

40(1b)This is cosmetic The error will goaway once the cluster returns tohealthy status

When a single node is offline in clusterstcli cluster storage-summary shows twonodes unavailable rootSCVM~ stclicluster storage-summary

CSCvs47735

35(2g)Modify both the root and adminpassword for the SCVMs to be thesame

stcli security password set -uadmin

Cluster expansion fails at Config Installerstage when the root and admin passwordfor the storage controller (SCVM) aredifferent

CSCvs69317

35(2d)NAAfter successfully changing the DNSserver on the HX controller we still cansee the original DNS entry that was addedduring the deployment

CSCvs69154

Release Notes for Cisco HX Data Platform Release 4054

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Customers are strongly encouragedto work with TAC in order toidentify and remediate this issue

1 Change getcluster from 2m to8m inoptspringpathstorfs-mgmtstMgr-10confapplicationconfon all nodes

2 Restart stMgr on all nodes

You may see this error message after afailed upgrade or other task such asattempting to enter a node into HXmaintenance mode

CSCvs53555

40(2a)Run the following command oneach storage controller VM as root

grep -i monitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp sed -iesmonitor_dns_serverstruemonitor_dns_serversfalseoptspringpathhx-diag-toolswatchdog_configjsonampamp grep -imonitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp restart watchdog

Receive the following alertevent in HXConnect HX Controller VMHOSTNAME one or more configuredDNS servers not responding

CSCvt13947

35(2b)Delete any empty (Size 0) log filesunder varlogzookeeperversion-2ORvarlogzookeeperstandaloneversion-2

Zookeeper fails to start while Exhibitor isrunning however echo srvr returnsnothing

CSCvs21562

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

40(4e)1 De-commission the server

2 Power drain the server -REMOVE BOTH POWERCORDS ON THE BACK OFTHE SERVER FOR 10SECONDS THENREINSERTPOWERCORDS

3 Re-commission the server

M2 boot disk is missing from serverinventory after upgrade to server firmware40(4e) As a result server fails to boot toOS installed in the M2 disk The issuepersists after re-acknowledgement as wellas de-commission andre-acknowledgement of the server

CSCvs02466

Release Notes for Cisco HX Data Platform Release 4055

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Re-start the flusher manuallyMany disks taken offline by a fault ormanually resulting in ENOSPC oncluster

CSCvq32721

35(2e)Clean up fail over cluster and thenrecreate the fail over cluster Thereis no need to touch the HX storagecluster

[Hyper-V]When replicated DCwas usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2b)Manually install upgradescvmclient VIB andmake sure thatit matches with HXDP version

Upgrade from release HX 35(2a) to35(2x) 4X does not upgrade scvmclientRecovery point may not work

CSCvq54992

Management

30(1b)To recover the VM copy over thedata disks and attach them to thenew VM

[HyperV]VMshows up onmultiple nodesCSCvj22992

35(1a)1 Do not change MTU at ESXlevel on running cluster

2 Reset back to the original value

ChangingMTU from 9000 to 1500 on oneESX node causes storfs process to restarton all nodes

CSCvm99150

Release Notes for Cisco HX Data Platform Release 4056

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)After the installation (for newdeployments) or after an upgradeto 401a (for existingdeployments) perform thefollowing steps on all ControllerVMs

1 Edit the following tune files (onall Controller VMs)

optspringpathconfiglfftunes

optspringpathconfigvsi_16tbtunes

2 SetcleanerEnableSegSummaryCleaning

to false

3 After editing the above tunefiles

a SSH in to all the ControllerVMs

b Run storfstool -- -Z

c Run the below command andcheck the values Tune valueshould be true

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=true

d Type below commands andapplied the tunes changesdynamically

echo false gttmpstprocfssystemtunecleanerEnableSegSummaryCleaning

e Verify that tune values arechanged Run below commandValue should be false

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=false

f umount tmpstprocfs

A cluster with 8TB or 12TB disk drivesmay experience IO stalls for severalminutes after another node fails in thecluster

CSCvp23718

Release Notes for Cisco HX Data Platform Release 4057

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)1 Update the conf file manuallyto change to the link local addressto

2 Restart scvmclient on thecontroller and ESX

HX 35(2b) FlexVol to HX 40 CSIupgrade does not work FlexVol continuesto work

CSCvp26319

35(2a)Disable rebalance until node healsthen reenable rebalance

7-node cluster healing very slowlybecause rebalance change AMS reqs areblocking healing process

CSCvp62512

35(2a)NAWith moderate cluster storage capacityusage under site failover cluster clustermay experience an all paths (APD) downcondition

CSCvq49412

40(1a)NAThis product includes Third-partySoftware that is affected by thevulnerabilities identified by the followingCommon Vulnerability and Exposures(CVE) IDs

CVE-2016-0762 CVE-2016-6797CVE-2016-6816 CVE-2016-8735CVE-2017-5647 CVE-2017-12615CVE-2017-12616 CVE-2017-12617CVE-2017-7674 CVE-2018-1304CVE-2018-8014 CVE-2018-1336CVE-2018-8034 CVE-2018-11784CVE-2019-0232

CSCvp36364

35(2d)NAstcli cluster info command needs toprovide UCSM VIP address Currently itshows ucsm-hostcom

CSCvq11456

35(2a)No workarounds currently Thestale entries do not affect the dayto day operations of HyperFlex

HyperFlex upgrade validation failedbecause cluster has extraneous stNodes ininternal database

CSCvq32530

35(2a)NAWhen not in maintenancemode all drivesappear in HX Connect and get updated asexpected

When in maintenance mode extra drivesare shown in HX Connect with no slotnumbers

CSCvq42378

35(2b)Contact TAC to cleanup duplicatenode

Nodes duplicated in HX Connect aftercluster recreate

CSCvq61825

Release Notes for Cisco HX Data Platform Release 4058

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)NACurrently HyperFlex installs on Hyper-Vdo not contain the stcli security whitelistcommandset

CSCvq66245

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

Release Notes for Cisco HX Data Platform Release 4059

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)[Hyper-V] Converged node expansionfails for incoming node with VIC 14xx

CSCvq39523

Release Notes for Cisco HX Data Platform Release 4060

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

1 Set Jumbo frame inWindowsHypervisor for incomingnode

Get-NetAdapter storage-data-a|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

Get-NetAdapter storage-data-b|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

2 Verify all interfaces are setcorrectly to support Jumbo frames

Get-NetAdapter storage |Get-NetAdapterAdvancedProperty| RegistryKeyword -matchjumbo | ft -auto

You should receive the followingmessage

Name DisplayName DisplayValue

RegistryKeyword RegistryValue

---- ----------- ------------

--------------- -------------

vswitch-hx-storage-data Jumbo

Packet 9014 Bytes

JumboPacket 9014

storage-data-b Jumbo Packet

Bytes 9014 JumboPacket

9014

storage-data-a Jumbo Packet

Bytes 9014 JumboPacket

9014

3 Reboot only the incoming nodeor the node being expanded

4 Retry Cluster Expansion in

Release Notes for Cisco HX Data Platform Release 4061

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)40(2a)API UI CLI show few Drive slots Empty after upgradeto 402a

The drives are seen in the sysmtool lsscsi and stclicommand outputs

The cluster is healthy with no errors

CSCvt14914

40(2b)35(2d)After successfully changing the DNS server on the HXcontroller we still can see the original DNS entry thatwas added during the deployment

CSCvs69154

40(2b)35(2e)HyperFlex Connect and VCenter show APD alarms forHX and non HX datastores This implies there is an issuewith HyperFlex when there may not be

CSCvs30080

40(2b)40(2a)An issue where all the disks in the node get locked afterrebooting the node

We successfully unlocked using sed-clientsh -Ucommand but wanted to test with another reboot anddrives locked again

CSCvs74286

40(2b)40(2a)When running stcli license commands on HyperFlexthe following error is seen

rootSpringpathControllertmp stcli license show all

Show smart licensing failed

Smart Agent is not ready please wait a minute and tryagain

CSCvt13929

40(2b)40(2a)After upgrading to HX 40(2a) the first RoboEdge nodethat gets rebooted then ignores persistent drivesPIDHX-HD24TB10K4KN

Cluster will be in degraded state and have reducedcapacity

CSCvt20144

40(2b)35(2b)Zookeeper fails to start while Exhibitor is runninghowever echo srvr returns nothing

CSCvs21562

40(2b)40(2a)When performing an HyperFlex upgrade a validationwarning may occur due to the host not having EnterprisePlus or Enterprise hypervisor licensing

Upgrade Validation Warning

ESXi host esx1labtest should be configured withVMware Enterprise license for upgrade to continue

CSCvs91787

Release Notes for Cisco HX Data Platform Release 4032

Resolved Caveats in Release 40(2b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)35(2g)Cluster expansion fails at Config Installer stage whenthe root and admin password for the storage controller(SCVM) are different

CSCvs69317

40(2b)35(2g)Panic while upgrading ESXiCSCvt06983

40(2b)40(2a)A cluster node running may hang in the Linux kernelThis is classified as an oops and a deviation from theexpected behavior

CSCvs54285

40(2b)35(2g)Rebalance failing on 10+10 node HX 35(2g) stretchcluster

CSCvs69007

40(2b)35(2d)The cluster becomes inaccessible to the IOVisorCSCvr54687

40(2b)40(2a)Panic on the storage controllerCSCvt61297

40(2b)40(2a)The size of support bundle is very large when collectedwith storfs-support command

CSCvt63306

40(2b)40(2a)storfs crash associated withFileSystemUsageWarningEvent orFileSystemUsageAlertEvent due to high usage ofvarstv above 80

CSCvt72807

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

40(2a)35(2d)In normal scenarios HX upgrade gets pendingACK afterupdating service profile In this case UCS only upgradewas stalled because after updating service profile HXupgrade was not getting pending ACK and it was alwayswaiting state

CSCvo39912

40(2a)35(1a)When using motherBoardReplace-12 to clean ZK fromold stNodepNodes resulted in unmounted datastore andmaking the size of the Hyperflex Datastores 0 resultingin all VMs in the cluster going offline

CSCvq39471

40(2a)26(1e)Upgrading cluster and getting error

Failed upgrade validations Checking vCenterconfiguration Reason Upgrade validations failed DRSFault Insufficient resources to satisfy configured failover

CSCvm77294

Release Notes for Cisco HX Data Platform Release 4033

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)The cluster expand fails on a node with DR replicationconfigured

When a HX cluster which has DR replication configuredis expanded we see the installer UI pulling in thereplicationVLAN information instead of themanagementVLAN information Even if we change that informationto the correct mgmt VLAN id and name it it does notseem to work as the node is configured with the VLANof the replicationVLAN in ESXi This leads to the failureof the node add with the host unreachable error

CSCvo70650

40(2a)35(1a)Customer reported that server didnt complete thefirmware upgrade automatically one by one

The user manually needs to put the host into maintenancemode and they manually acknowledged the pendingrequests to finish the UCS firmware upgrade

Maintenance policy is set to default (User-ack) as perthe design

CSCvo91624

40(2a)35(1i)When the cluster is in Failed state and stcli noderemove is attempted the output shows successful thoughthe node failed to get removed from the cluster

CSCvo93017

40(2a)35(1a)HyperFlex cluster upgrade may fail during validationwith a DRS Validation failed error

CSCvp31021

40(2a)35(2b)HX cluster expansion will fail with the following errormessage

MAC address pool configuration failure150[ErrorDescription] bad address block range definitioncollision

CSCvp58318

40(2a)35(2b)Memory usage by carbon cacheCSCvq34873

40(2a)35(2g)Upgrading ESXi cluster fails with error Nodemaintenance mode failed

CSCvr03240

40(2a)35(2e)storfs process does not automatically start on Exit HXMaintenance Mode or other tasks which restart StorageController

CSCvr88978

40(2a)35(2a)Post install script fails with message

Failed to execute ipmitool on HX node

CSCvp10707

40(2a)40(1a)HyperFlex expansionworkflow doesnt pull VLANnamecorrectly

CSCvp46539

Release Notes for Cisco HX Data Platform Release 4034

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)During HX cluster deployment - Validate ClusterCreation phase - HX Installer might fail with the error

fromvarlogspringpathstDeploylog

CSCvq45087

40(2a)35(2b)HX Installer fails to login to the SOL for configuringESX hosts

CSCvq91380

40(2a)35(2d)If DRS is enabled with configuration parameters otherthen fully automated ie manual or partially automatedthan this will change to fully automated during clusterexpansion

CSCvr44222

40(2a)35(1a)One of the validation task name has a typo The taskname is Validating node for mixed of SED and non-SEDdisks

It should be Validating node for mix of SED andnon-SED disks

CSCvp82175

40(2a)30(1i)This combination of using smaller NVMe drives is notsupported in expand operation For example if the drivesare only 375GB and they cannot be added to an existingcluster with larger caching SSD

CSCvo12359

40(2a)35(1a)HyperFlex includes a version of OpenSSL that is affectedby the vulnerabilities identified by the followingCommon Vulnerability and Exposures

CVE-2018-0495

CSCvp66679

40(2a)30(1e)SDK invoke exception when taking a native snapshot ona VM with GPU pci passthrough attached

CSCvo00511

40(2a)30(1i)Node replacement script fails due to EAM errorCSCvo62867

40(2a)35(1a)While pairing a cluster (HX release gt= 35) and cluster(HX release lt 35) remote replication network test failson the cluster (HX release lt 35)

CSCvo79760

40(2a)30(1b)Fixed the issue in the latest support-workflow bundleThis should not be hit anymore

CSCvo87061

40(2a)35(2a)On a 3 node cluster the MBreplace script fails as thescript looks for HEALTHY cluster status

CSCvo87080

40(2a)35(2a)MbReplace script (tar) hangs when running on 352b or40

CSCvp12359

Release Notes for Cisco HX Data Platform Release 4035

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)HX replication cleanup failing with error NA 35(2a) -INFODR state is not clean

CSCvp63958

40(2a)40(1a)The check_stig_parameters API incorrectly shows thecompliance state of the cluster as not STIG compliant

CSCvp66277

40(2a)35(2c)The snapshot creation on CBT enabled VM fails witherror Failed in vmreparent vmkfstools clone1

CSCvq06952

40(2a)35(2a)HyperFlex GUI when setting up remote DR-replicationpartnerpeer should show remote replication partner TCPport reachability test results

This will provide quick upfront connectivity results thatfrequently takes customer a while to manually check andconfirm

If reachability tests fail an info window and HX alertshould trigger that list the destination peer IP and thespecific port(s) that could not be reache

CSCvo60587

40(2a)35(2b)If you have more than one IP pool configured for areplication network and local replication network testsare failing you might be running into this issue

CSCvr67130

40(2a)35(2a)Skip task option in MBreplace and Software redeployscripts The scripts should have the option to skip tasksas user inputs and not by modifying the json file

CSCvp05204

40(2a)40(1a)Error noticed on SCVM Console print_req_error IOerror dev fd0 sector 0 on HyperV

It doesnt come up on SSH session

CSCvq34357

40(2a)30(1d)The VMNetwork switch is already created as part of theinstall process

Ideally the ip address assignment should be done duringthe initial install process as well No post install stepsshould be required altrnatively it could be done as partof a post install script

CSCvk36222

40(2a)25(1c)storfs service impacted during HX upgrade from HX21(1) to 25(1c)

HX node which is not being upgraded storfs servicestopped causing outage

CSCvg53223

40(2a)35(1b)HX Stretch Cluster Witness VM reverts to DHCP onreboot

CSCvp37536

40(2a)35(2b)HyperFlex Stretch Cluster Witness does not getprogrammed with an NTP server

CSCvp46578

Release Notes for Cisco HX Data Platform Release 4036

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2b)HyperFlex cluster does not heal after maintenance onnodeserver

CSCvr18528

40(2a)40(2a)High percentage of packet loss on the eth1 data interface9K MTU

CSCvr97089

40(2a)35(2f)The HyperFlex cluster healing state is stuck at 87 andis unable to progress

CSCvr16760

40(2a)40(2a)In some circumstances two replications can occur forthe same VM

From the GUI the replication status for this VM willshow stuck in an In Progress state but there are nojobs stuck in replication layer Subsequent Replicationsshould succeed

CSCvq22898

40(2a)35(2a)After an unexpected power outage the hyperflex clusteris unable to come up correctly

All processes are running time is synced and all nodescan vmkping each other as required

CSCvp13990

40(2a)40(1a)When specifying IP address instead of node ID stclinode maintenanceMode commands fail

CSCvp33657

40(2a)40(1a)2 node edge cluster if experiencing Intersightarbitratorconnectivity issues should NOT allowHXMaintenancemode to initiate as this will bring the cluster down

CSCvr31573

40(2a)35(2c)Old files in varsupportZKTxnlog are not purged withthe daily zklog-cleanup cron job

CSCvr89066

40(2a)35(2a)When a node is in MaintenanceMode any disk removalor replacement will be reflected in UI only after the nodeis brought back frommaintenancemode This is becausestorfs is not running on the node in maintenance andwill not be able to detect disk activities until it is broughtout of Maintenance Mode

CSCvo86431

40(2a)35(2c)Cluster might go offline and stop serving IO requestsrootcvm~ stcli cluster storage-summary--detailGet cluster storage summary failedjavanetConnectException Connection refused19216814210010207

CSCvr01645

40(2a)35(2d)A stretch cluster deployment fails with a messageFormatting nodes Node in Use on the Hyperflex installerand crmZoneType value is seen as 1 in storfscfg oroptspringpathconfigstretchtunes

CSCvq80340

Release Notes for Cisco HX Data Platform Release 4037

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)30(1c)During upgrade we upload the upgrade packagedownloaded from Ciscocom Upgrade packages are offile type tgz hxconnect acceptsstorfs-packages-401a-33028tar file a different type ofcompression and allows upgrade to start and fail whenit cannot decompress the file for bootstrap and scvmclientupgrade on ESXi

CSCvq63888

40(2a)40(2a)While attempting to install hyperflex via intersight thefollowing error is seen

Failed in Task Add hosts to vCenter Cluster with ErrorTry adding hosts manually to vCenter and retry Failedto get addressFailed to host by namelookupHostnamecompanycom on 000053 read udp12700145430-gt12700153 read connection refused

CSCvr52098

40(2a)35(2a)CoreAPI call may time out for cluster management APIwhile doing cluster create validation

CSCvo92952

40(2a)35(2d)Validation fails with Mixed mode expansion checkdue to Enhanced vMotion Compatibility Incompatibilitywhen the ESXi version is different

CSCvq95460

40(2a)30(1i)This failure is seen in installer when custom workflowis used hypervisor configuration + deploy (clean diskpartition=no) at Installing software packages on storagecontroller VM

CSCvr66309

40(2a)35(2d)The HX Installer apparently replaced a comma bysemicolon on the VLAN ID range while exporting theconfiguration to a JSON file

CSCvq93831

40(2a)40(1a)When using unsupported browsers HX Connect userswill not be able to login These users see an errorindicating unauthorized user

CSCvp22693

Release Notes for Cisco HX Data Platform Release 4038

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)40(1a)A vulnerability in the web-based management interfaceof Cisco HyperFlex Software could allow anunauthenticated remote attacker to conduct a cross-siterequest forgery (CSRF) attack on an affected system

The vulnerability is due to insufficient CSRF protectionsfor the web UI on an affected device An attacker couldexploit this vulnerability by persuading a user of theinterface to follow a malicious link A successful exploitcould allow the attacker to perform arbitrary actions withthe privilege level of the affected user

Cisco has released software updates that address thisvulnerability There are no workarounds that address thisvulnerability

This advisory is available at the following link

httpstoolsciscocomsecuritycentercontentCiscoSecurityAdvisorycisco-sa-20190807-hypflex-csrf

CSCvp24343

40(2a)35(2d)Add message to HX Connect Progress flow to notAcknowledge Pending Activities and reboot the serverson UCSM

HX Connect is doing a controlled rolling server upgradein the background

CSCvq22844

40(2a)40(1b)Compute Node expand operation fails due to incorrectcluster name

CSCvr43786

40(2a)26(1e)RF-2 cluster shutdown during data resynchronizationafter a node failure (non storfs) followed by multipledisk read failures and a HardBlacklist of a disk

CSCvp41241

40(2a)40(1a)HyperFlex Edge nodes do not properly set the ESXihostname during deployment

CSCvo13143

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4039

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4040

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1b) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401b-33133p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4041

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4042

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1b)40(1a)While running the HyperFlex installer with WindowsServer Hyper-V cluster creation process fails with theerror Failure occurred during ClusterCreation process Unable to post thecontent to the downstream This symptomis encountered while deploying HyperFlex cluster withHyperFlex nodes configuredordered with Cisco VIC1457 MLOM (PID HX-MLOM-C25Q-04) andWindows Server Datacenter or Core with Hyper-Vversion 2016 or 2019

CSCvp64140

Release Notes for Cisco HX Data Platform Release 4043

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)35(2b)Adding Micron 5200 drive to a cluster fails to increasecluster capacity

CSCvo69067

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4044

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4045

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1a) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401a-33028p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4046

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4047

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1a)30(1d)Cluster instability when disks of different sector sizeplaced in HX node

CSCvk17250

40(1a)

35(2c)

35(1a)When logged in using a local HX user account insteadof a Virtual Center account an error message appearsintermittently indicating Virtual Centerunreachable or Resource informationcannot be updated when VC is reachable

CSCvo36198

40(1a)

35(2b)

30(1d)HXDP does not work with EMC RecoverPoint - needsto support VMware API (FSS-Readdir)

CSCvk38003

Release Notes for Cisco HX Data Platform Release 4048

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)

35(2a)

30(1i)

25(1c)

25(1d)

Zookeeper (Exhibitor) process on HyperFlex StorageController VMs may fail to respond to requests whenvarzookeeper has low or no free space

CSCvm90352

40(1a)26(1c)Use Asynchronous consolidation for HX SnapshotsCSCvn02151

40(1a)35(2b)HX Quiesced Snapshot for Backup VendorsCSCvo90713

40(1a)35(2a)The cluster creationcluster expansion workflow stopswith the following error message at the validation step

FIRMWARE-Check UCSC-SAS-M5HD

FIRMWARE-Check UCSC-SAS-M5HD Required00000029000000320000003500000050 Found00000058

Action Needed Update the Controller Firmware toRequired Version

CSCvn17787

35(2b)30(1d)Kernel migration fails when a local datastore is searchedfor in the ESXi host

CSCvn73127

35(2a)26(1b)A node shuts down when two disks are replaced (acaching disk and another capacity disk) where thecapacity disk is inserted first and then the caching diskis inserted

CSCvk46364

Hyper-V

40(1a)35(2a)Cluster expansion may fail with an error code 500 -operation timed out

CSCvn28721

35(2b)35(2a)During upgrade remove the VLAN on team created foruser vSwitch During fresh installation only one VLANtag is set to the vSwitch and team although multipleVLANs were entered

CSCvn54300

35(2b)35(2a)While upgrading a Hyper-V cluster on account of a rarerace condition between the stUpgradeService andZookeeper servers the upgrade orchestration throws anupgrade validation error and the upgrade process isaborted

CSCvn60486

Release Notes for Cisco HX Data Platform Release 4049

Resolved Caveats in Release 40(1a)

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Startrestart EAM service invCenter

When EAM service is not runningvCenter registration is failing

CSCvt55712

35(2a)Contact Cisco TAC if you believeyou are seeing similar conditions

HyperFlex Stretch Cluster saw poorapplication performance due to writelatency cluster remained unhealthy andrebalance was stuck

This defect is currently being used forinvestigative purposes

CSCvt45344

35(2g)Remove log files fromvarlogspringpath folder when thelog files are not getting rotatedStop the relevant service removelarge log files fromvarlogspringpath folder andrestart the service

varstv folder in Controller VM maybecome full

CSCvt36374

40(2b)Restarting the stMgr will initializeit again

In a Cluster configured with DR networkStMgr may not get initialized and stuckin a deadlock while enabling the IPTablerules (can be verified from the stMgrlog)

CSCvt89709

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Click recover for the selectedVM again under the standalonemode This will move the VM torecovered state

When the user recovers a VM protectedin a group it along with other VMs in thegroup move to recovering state instandalonemode The selectedVM shouldmove to recovered state

CSCvs75553

35(2e)Clean up the fail over cluster andthen recreate the fail over clusterNo need to touch the HX storagecluster

Hyper-V When replicated DC was usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2d)40(1a)

Keep cluster utilization as low aspossible To avoid read-only statekeep below 92

HyperFlex clusters storage utilization isabove 92 usable capacity

CSCvr20922

Release Notes for Cisco HX Data Platform Release 4050

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1b)NAAll the disks in the node got locked afterrebooting the node

We successfully unlocked usingsed-clientsh -U command but wantedto test with another reboot and driveslocked again

CSCvs74286

35(2g)To recover the VM copy over thedata disks and attach them to thenew VM

After enable encryption HX connectencryption status showing Caution butall of the disks enable encryptionsuccessful

-Self Encrypting Drives Service isrunning on all of the nodes

-There is no error message of encryption

- USB0 interface is up

-All SED disks showing supported 1enabled 1 locked 0

CSCvs93245

40(1b)Enable kernelpanic_on_oops as apersistent configuration This willcause the node to panic and rebootimmediately

A cluster node running HX release40(1b) may hang in the Linux kernelThis is classified as an oops and adeviation from the expected behavior

CSCvs54285

Release Notes for Cisco HX Data Platform Release 4051

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Enabling K8iscsi stack in Hx clusterCSCvs41324

Release Notes for Cisco HX Data Platform Release 4052

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Steps to enable K8 on HX cluster

Perform following steps on ALLthe controller VMs

1 Updateetcinitscvmclientconf toenable tune iscsiEnable totrue

sed -iesiscsiEnable=falseiscsiEnable=trueetcinitscvmclientconf

2 Run following initctlcommand to reloadconfiguration

initctl reload-configuration

3 Restart scvmclient process

stop scvmclient startscvmclient

4 Run following initctlcommand to mount SYSTEMDS

initctl emit --no-waitsystem-datastore-created

5 Verify that iscsiEnable tuneis set to true using thefollowing command

ps -eaf | grep scvmclient |grep -v grep

root 6241 1 1 Dec06 014007optspringpathstorfs-corescvmclient-T logEnabled=false -TlogSyslogEnabled=true -TlogEchoToScreen=false -TstatLoggingToFile=false -TstatLoggingToSyslog=true-TlogDir=varlogspringpath-TnfsBackendServerList=1010748100-T iscsiEnable=true -TiscsiUseAsync=true -T

Release Notes for Cisco HX Data Platform Release 4053

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

iscsiConfigFilePath=nfsSYSTEMistgtconfps -eaf | grep scvmclient |grep -v grep

35(2e)Following procedure can be usedto check NFS Queue Depth andincrease if needed

rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth256 lt- Low value[rootHXESXI1] vsish -e setvmkModulesnfsclientmnt[DATASTORE]propertiesmaxQDepth 1024[rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth1024 lt- Optimalvalue

This requires an ESXi host rebootto take effect

Please place the node in HyperflexMaintenanceMode and gracefullyreboot the node for the changes tobe applied

HyperFlex Datastore NFS Queue Depthshows as 256 which can lead toperformance (including latency) issues

CSCvr83056

40(1b)This is cosmetic The error will goaway once the cluster returns tohealthy status

When a single node is offline in clusterstcli cluster storage-summary shows twonodes unavailable rootSCVM~ stclicluster storage-summary

CSCvs47735

35(2g)Modify both the root and adminpassword for the SCVMs to be thesame

stcli security password set -uadmin

Cluster expansion fails at Config Installerstage when the root and admin passwordfor the storage controller (SCVM) aredifferent

CSCvs69317

35(2d)NAAfter successfully changing the DNSserver on the HX controller we still cansee the original DNS entry that was addedduring the deployment

CSCvs69154

Release Notes for Cisco HX Data Platform Release 4054

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Customers are strongly encouragedto work with TAC in order toidentify and remediate this issue

1 Change getcluster from 2m to8m inoptspringpathstorfs-mgmtstMgr-10confapplicationconfon all nodes

2 Restart stMgr on all nodes

You may see this error message after afailed upgrade or other task such asattempting to enter a node into HXmaintenance mode

CSCvs53555

40(2a)Run the following command oneach storage controller VM as root

grep -i monitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp sed -iesmonitor_dns_serverstruemonitor_dns_serversfalseoptspringpathhx-diag-toolswatchdog_configjsonampamp grep -imonitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp restart watchdog

Receive the following alertevent in HXConnect HX Controller VMHOSTNAME one or more configuredDNS servers not responding

CSCvt13947

35(2b)Delete any empty (Size 0) log filesunder varlogzookeeperversion-2ORvarlogzookeeperstandaloneversion-2

Zookeeper fails to start while Exhibitor isrunning however echo srvr returnsnothing

CSCvs21562

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

40(4e)1 De-commission the server

2 Power drain the server -REMOVE BOTH POWERCORDS ON THE BACK OFTHE SERVER FOR 10SECONDS THENREINSERTPOWERCORDS

3 Re-commission the server

M2 boot disk is missing from serverinventory after upgrade to server firmware40(4e) As a result server fails to boot toOS installed in the M2 disk The issuepersists after re-acknowledgement as wellas de-commission andre-acknowledgement of the server

CSCvs02466

Release Notes for Cisco HX Data Platform Release 4055

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Re-start the flusher manuallyMany disks taken offline by a fault ormanually resulting in ENOSPC oncluster

CSCvq32721

35(2e)Clean up fail over cluster and thenrecreate the fail over cluster Thereis no need to touch the HX storagecluster

[Hyper-V]When replicated DCwas usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2b)Manually install upgradescvmclient VIB andmake sure thatit matches with HXDP version

Upgrade from release HX 35(2a) to35(2x) 4X does not upgrade scvmclientRecovery point may not work

CSCvq54992

Management

30(1b)To recover the VM copy over thedata disks and attach them to thenew VM

[HyperV]VMshows up onmultiple nodesCSCvj22992

35(1a)1 Do not change MTU at ESXlevel on running cluster

2 Reset back to the original value

ChangingMTU from 9000 to 1500 on oneESX node causes storfs process to restarton all nodes

CSCvm99150

Release Notes for Cisco HX Data Platform Release 4056

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)After the installation (for newdeployments) or after an upgradeto 401a (for existingdeployments) perform thefollowing steps on all ControllerVMs

1 Edit the following tune files (onall Controller VMs)

optspringpathconfiglfftunes

optspringpathconfigvsi_16tbtunes

2 SetcleanerEnableSegSummaryCleaning

to false

3 After editing the above tunefiles

a SSH in to all the ControllerVMs

b Run storfstool -- -Z

c Run the below command andcheck the values Tune valueshould be true

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=true

d Type below commands andapplied the tunes changesdynamically

echo false gttmpstprocfssystemtunecleanerEnableSegSummaryCleaning

e Verify that tune values arechanged Run below commandValue should be false

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=false

f umount tmpstprocfs

A cluster with 8TB or 12TB disk drivesmay experience IO stalls for severalminutes after another node fails in thecluster

CSCvp23718

Release Notes for Cisco HX Data Platform Release 4057

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)1 Update the conf file manuallyto change to the link local addressto

2 Restart scvmclient on thecontroller and ESX

HX 35(2b) FlexVol to HX 40 CSIupgrade does not work FlexVol continuesto work

CSCvp26319

35(2a)Disable rebalance until node healsthen reenable rebalance

7-node cluster healing very slowlybecause rebalance change AMS reqs areblocking healing process

CSCvp62512

35(2a)NAWith moderate cluster storage capacityusage under site failover cluster clustermay experience an all paths (APD) downcondition

CSCvq49412

40(1a)NAThis product includes Third-partySoftware that is affected by thevulnerabilities identified by the followingCommon Vulnerability and Exposures(CVE) IDs

CVE-2016-0762 CVE-2016-6797CVE-2016-6816 CVE-2016-8735CVE-2017-5647 CVE-2017-12615CVE-2017-12616 CVE-2017-12617CVE-2017-7674 CVE-2018-1304CVE-2018-8014 CVE-2018-1336CVE-2018-8034 CVE-2018-11784CVE-2019-0232

CSCvp36364

35(2d)NAstcli cluster info command needs toprovide UCSM VIP address Currently itshows ucsm-hostcom

CSCvq11456

35(2a)No workarounds currently Thestale entries do not affect the dayto day operations of HyperFlex

HyperFlex upgrade validation failedbecause cluster has extraneous stNodes ininternal database

CSCvq32530

35(2a)NAWhen not in maintenancemode all drivesappear in HX Connect and get updated asexpected

When in maintenance mode extra drivesare shown in HX Connect with no slotnumbers

CSCvq42378

35(2b)Contact TAC to cleanup duplicatenode

Nodes duplicated in HX Connect aftercluster recreate

CSCvq61825

Release Notes for Cisco HX Data Platform Release 4058

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)NACurrently HyperFlex installs on Hyper-Vdo not contain the stcli security whitelistcommandset

CSCvq66245

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

Release Notes for Cisco HX Data Platform Release 4059

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)[Hyper-V] Converged node expansionfails for incoming node with VIC 14xx

CSCvq39523

Release Notes for Cisco HX Data Platform Release 4060

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

1 Set Jumbo frame inWindowsHypervisor for incomingnode

Get-NetAdapter storage-data-a|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

Get-NetAdapter storage-data-b|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

2 Verify all interfaces are setcorrectly to support Jumbo frames

Get-NetAdapter storage |Get-NetAdapterAdvancedProperty| RegistryKeyword -matchjumbo | ft -auto

You should receive the followingmessage

Name DisplayName DisplayValue

RegistryKeyword RegistryValue

---- ----------- ------------

--------------- -------------

vswitch-hx-storage-data Jumbo

Packet 9014 Bytes

JumboPacket 9014

storage-data-b Jumbo Packet

Bytes 9014 JumboPacket

9014

storage-data-a Jumbo Packet

Bytes 9014 JumboPacket

9014

3 Reboot only the incoming nodeor the node being expanded

4 Retry Cluster Expansion in

Release Notes for Cisco HX Data Platform Release 4061

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2b)35(2g)Cluster expansion fails at Config Installer stage whenthe root and admin password for the storage controller(SCVM) are different

CSCvs69317

40(2b)35(2g)Panic while upgrading ESXiCSCvt06983

40(2b)40(2a)A cluster node running may hang in the Linux kernelThis is classified as an oops and a deviation from theexpected behavior

CSCvs54285

40(2b)35(2g)Rebalance failing on 10+10 node HX 35(2g) stretchcluster

CSCvs69007

40(2b)35(2d)The cluster becomes inaccessible to the IOVisorCSCvr54687

40(2b)40(2a)Panic on the storage controllerCSCvt61297

40(2b)40(2a)The size of support bundle is very large when collectedwith storfs-support command

CSCvt63306

40(2b)40(2a)storfs crash associated withFileSystemUsageWarningEvent orFileSystemUsageAlertEvent due to high usage ofvarstv above 80

CSCvt72807

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

40(2a)35(2d)In normal scenarios HX upgrade gets pendingACK afterupdating service profile In this case UCS only upgradewas stalled because after updating service profile HXupgrade was not getting pending ACK and it was alwayswaiting state

CSCvo39912

40(2a)35(1a)When using motherBoardReplace-12 to clean ZK fromold stNodepNodes resulted in unmounted datastore andmaking the size of the Hyperflex Datastores 0 resultingin all VMs in the cluster going offline

CSCvq39471

40(2a)26(1e)Upgrading cluster and getting error

Failed upgrade validations Checking vCenterconfiguration Reason Upgrade validations failed DRSFault Insufficient resources to satisfy configured failover

CSCvm77294

Release Notes for Cisco HX Data Platform Release 4033

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)The cluster expand fails on a node with DR replicationconfigured

When a HX cluster which has DR replication configuredis expanded we see the installer UI pulling in thereplicationVLAN information instead of themanagementVLAN information Even if we change that informationto the correct mgmt VLAN id and name it it does notseem to work as the node is configured with the VLANof the replicationVLAN in ESXi This leads to the failureof the node add with the host unreachable error

CSCvo70650

40(2a)35(1a)Customer reported that server didnt complete thefirmware upgrade automatically one by one

The user manually needs to put the host into maintenancemode and they manually acknowledged the pendingrequests to finish the UCS firmware upgrade

Maintenance policy is set to default (User-ack) as perthe design

CSCvo91624

40(2a)35(1i)When the cluster is in Failed state and stcli noderemove is attempted the output shows successful thoughthe node failed to get removed from the cluster

CSCvo93017

40(2a)35(1a)HyperFlex cluster upgrade may fail during validationwith a DRS Validation failed error

CSCvp31021

40(2a)35(2b)HX cluster expansion will fail with the following errormessage

MAC address pool configuration failure150[ErrorDescription] bad address block range definitioncollision

CSCvp58318

40(2a)35(2b)Memory usage by carbon cacheCSCvq34873

40(2a)35(2g)Upgrading ESXi cluster fails with error Nodemaintenance mode failed

CSCvr03240

40(2a)35(2e)storfs process does not automatically start on Exit HXMaintenance Mode or other tasks which restart StorageController

CSCvr88978

40(2a)35(2a)Post install script fails with message

Failed to execute ipmitool on HX node

CSCvp10707

40(2a)40(1a)HyperFlex expansionworkflow doesnt pull VLANnamecorrectly

CSCvp46539

Release Notes for Cisco HX Data Platform Release 4034

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)During HX cluster deployment - Validate ClusterCreation phase - HX Installer might fail with the error

fromvarlogspringpathstDeploylog

CSCvq45087

40(2a)35(2b)HX Installer fails to login to the SOL for configuringESX hosts

CSCvq91380

40(2a)35(2d)If DRS is enabled with configuration parameters otherthen fully automated ie manual or partially automatedthan this will change to fully automated during clusterexpansion

CSCvr44222

40(2a)35(1a)One of the validation task name has a typo The taskname is Validating node for mixed of SED and non-SEDdisks

It should be Validating node for mix of SED andnon-SED disks

CSCvp82175

40(2a)30(1i)This combination of using smaller NVMe drives is notsupported in expand operation For example if the drivesare only 375GB and they cannot be added to an existingcluster with larger caching SSD

CSCvo12359

40(2a)35(1a)HyperFlex includes a version of OpenSSL that is affectedby the vulnerabilities identified by the followingCommon Vulnerability and Exposures

CVE-2018-0495

CSCvp66679

40(2a)30(1e)SDK invoke exception when taking a native snapshot ona VM with GPU pci passthrough attached

CSCvo00511

40(2a)30(1i)Node replacement script fails due to EAM errorCSCvo62867

40(2a)35(1a)While pairing a cluster (HX release gt= 35) and cluster(HX release lt 35) remote replication network test failson the cluster (HX release lt 35)

CSCvo79760

40(2a)30(1b)Fixed the issue in the latest support-workflow bundleThis should not be hit anymore

CSCvo87061

40(2a)35(2a)On a 3 node cluster the MBreplace script fails as thescript looks for HEALTHY cluster status

CSCvo87080

40(2a)35(2a)MbReplace script (tar) hangs when running on 352b or40

CSCvp12359

Release Notes for Cisco HX Data Platform Release 4035

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)HX replication cleanup failing with error NA 35(2a) -INFODR state is not clean

CSCvp63958

40(2a)40(1a)The check_stig_parameters API incorrectly shows thecompliance state of the cluster as not STIG compliant

CSCvp66277

40(2a)35(2c)The snapshot creation on CBT enabled VM fails witherror Failed in vmreparent vmkfstools clone1

CSCvq06952

40(2a)35(2a)HyperFlex GUI when setting up remote DR-replicationpartnerpeer should show remote replication partner TCPport reachability test results

This will provide quick upfront connectivity results thatfrequently takes customer a while to manually check andconfirm

If reachability tests fail an info window and HX alertshould trigger that list the destination peer IP and thespecific port(s) that could not be reache

CSCvo60587

40(2a)35(2b)If you have more than one IP pool configured for areplication network and local replication network testsare failing you might be running into this issue

CSCvr67130

40(2a)35(2a)Skip task option in MBreplace and Software redeployscripts The scripts should have the option to skip tasksas user inputs and not by modifying the json file

CSCvp05204

40(2a)40(1a)Error noticed on SCVM Console print_req_error IOerror dev fd0 sector 0 on HyperV

It doesnt come up on SSH session

CSCvq34357

40(2a)30(1d)The VMNetwork switch is already created as part of theinstall process

Ideally the ip address assignment should be done duringthe initial install process as well No post install stepsshould be required altrnatively it could be done as partof a post install script

CSCvk36222

40(2a)25(1c)storfs service impacted during HX upgrade from HX21(1) to 25(1c)

HX node which is not being upgraded storfs servicestopped causing outage

CSCvg53223

40(2a)35(1b)HX Stretch Cluster Witness VM reverts to DHCP onreboot

CSCvp37536

40(2a)35(2b)HyperFlex Stretch Cluster Witness does not getprogrammed with an NTP server

CSCvp46578

Release Notes for Cisco HX Data Platform Release 4036

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2b)HyperFlex cluster does not heal after maintenance onnodeserver

CSCvr18528

40(2a)40(2a)High percentage of packet loss on the eth1 data interface9K MTU

CSCvr97089

40(2a)35(2f)The HyperFlex cluster healing state is stuck at 87 andis unable to progress

CSCvr16760

40(2a)40(2a)In some circumstances two replications can occur forthe same VM

From the GUI the replication status for this VM willshow stuck in an In Progress state but there are nojobs stuck in replication layer Subsequent Replicationsshould succeed

CSCvq22898

40(2a)35(2a)After an unexpected power outage the hyperflex clusteris unable to come up correctly

All processes are running time is synced and all nodescan vmkping each other as required

CSCvp13990

40(2a)40(1a)When specifying IP address instead of node ID stclinode maintenanceMode commands fail

CSCvp33657

40(2a)40(1a)2 node edge cluster if experiencing Intersightarbitratorconnectivity issues should NOT allowHXMaintenancemode to initiate as this will bring the cluster down

CSCvr31573

40(2a)35(2c)Old files in varsupportZKTxnlog are not purged withthe daily zklog-cleanup cron job

CSCvr89066

40(2a)35(2a)When a node is in MaintenanceMode any disk removalor replacement will be reflected in UI only after the nodeis brought back frommaintenancemode This is becausestorfs is not running on the node in maintenance andwill not be able to detect disk activities until it is broughtout of Maintenance Mode

CSCvo86431

40(2a)35(2c)Cluster might go offline and stop serving IO requestsrootcvm~ stcli cluster storage-summary--detailGet cluster storage summary failedjavanetConnectException Connection refused19216814210010207

CSCvr01645

40(2a)35(2d)A stretch cluster deployment fails with a messageFormatting nodes Node in Use on the Hyperflex installerand crmZoneType value is seen as 1 in storfscfg oroptspringpathconfigstretchtunes

CSCvq80340

Release Notes for Cisco HX Data Platform Release 4037

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)30(1c)During upgrade we upload the upgrade packagedownloaded from Ciscocom Upgrade packages are offile type tgz hxconnect acceptsstorfs-packages-401a-33028tar file a different type ofcompression and allows upgrade to start and fail whenit cannot decompress the file for bootstrap and scvmclientupgrade on ESXi

CSCvq63888

40(2a)40(2a)While attempting to install hyperflex via intersight thefollowing error is seen

Failed in Task Add hosts to vCenter Cluster with ErrorTry adding hosts manually to vCenter and retry Failedto get addressFailed to host by namelookupHostnamecompanycom on 000053 read udp12700145430-gt12700153 read connection refused

CSCvr52098

40(2a)35(2a)CoreAPI call may time out for cluster management APIwhile doing cluster create validation

CSCvo92952

40(2a)35(2d)Validation fails with Mixed mode expansion checkdue to Enhanced vMotion Compatibility Incompatibilitywhen the ESXi version is different

CSCvq95460

40(2a)30(1i)This failure is seen in installer when custom workflowis used hypervisor configuration + deploy (clean diskpartition=no) at Installing software packages on storagecontroller VM

CSCvr66309

40(2a)35(2d)The HX Installer apparently replaced a comma bysemicolon on the VLAN ID range while exporting theconfiguration to a JSON file

CSCvq93831

40(2a)40(1a)When using unsupported browsers HX Connect userswill not be able to login These users see an errorindicating unauthorized user

CSCvp22693

Release Notes for Cisco HX Data Platform Release 4038

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)40(1a)A vulnerability in the web-based management interfaceof Cisco HyperFlex Software could allow anunauthenticated remote attacker to conduct a cross-siterequest forgery (CSRF) attack on an affected system

The vulnerability is due to insufficient CSRF protectionsfor the web UI on an affected device An attacker couldexploit this vulnerability by persuading a user of theinterface to follow a malicious link A successful exploitcould allow the attacker to perform arbitrary actions withthe privilege level of the affected user

Cisco has released software updates that address thisvulnerability There are no workarounds that address thisvulnerability

This advisory is available at the following link

httpstoolsciscocomsecuritycentercontentCiscoSecurityAdvisorycisco-sa-20190807-hypflex-csrf

CSCvp24343

40(2a)35(2d)Add message to HX Connect Progress flow to notAcknowledge Pending Activities and reboot the serverson UCSM

HX Connect is doing a controlled rolling server upgradein the background

CSCvq22844

40(2a)40(1b)Compute Node expand operation fails due to incorrectcluster name

CSCvr43786

40(2a)26(1e)RF-2 cluster shutdown during data resynchronizationafter a node failure (non storfs) followed by multipledisk read failures and a HardBlacklist of a disk

CSCvp41241

40(2a)40(1a)HyperFlex Edge nodes do not properly set the ESXihostname during deployment

CSCvo13143

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4039

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4040

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1b) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401b-33133p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4041

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4042

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1b)40(1a)While running the HyperFlex installer with WindowsServer Hyper-V cluster creation process fails with theerror Failure occurred during ClusterCreation process Unable to post thecontent to the downstream This symptomis encountered while deploying HyperFlex cluster withHyperFlex nodes configuredordered with Cisco VIC1457 MLOM (PID HX-MLOM-C25Q-04) andWindows Server Datacenter or Core with Hyper-Vversion 2016 or 2019

CSCvp64140

Release Notes for Cisco HX Data Platform Release 4043

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)35(2b)Adding Micron 5200 drive to a cluster fails to increasecluster capacity

CSCvo69067

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4044

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4045

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1a) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401a-33028p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4046

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4047

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1a)30(1d)Cluster instability when disks of different sector sizeplaced in HX node

CSCvk17250

40(1a)

35(2c)

35(1a)When logged in using a local HX user account insteadof a Virtual Center account an error message appearsintermittently indicating Virtual Centerunreachable or Resource informationcannot be updated when VC is reachable

CSCvo36198

40(1a)

35(2b)

30(1d)HXDP does not work with EMC RecoverPoint - needsto support VMware API (FSS-Readdir)

CSCvk38003

Release Notes for Cisco HX Data Platform Release 4048

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)

35(2a)

30(1i)

25(1c)

25(1d)

Zookeeper (Exhibitor) process on HyperFlex StorageController VMs may fail to respond to requests whenvarzookeeper has low or no free space

CSCvm90352

40(1a)26(1c)Use Asynchronous consolidation for HX SnapshotsCSCvn02151

40(1a)35(2b)HX Quiesced Snapshot for Backup VendorsCSCvo90713

40(1a)35(2a)The cluster creationcluster expansion workflow stopswith the following error message at the validation step

FIRMWARE-Check UCSC-SAS-M5HD

FIRMWARE-Check UCSC-SAS-M5HD Required00000029000000320000003500000050 Found00000058

Action Needed Update the Controller Firmware toRequired Version

CSCvn17787

35(2b)30(1d)Kernel migration fails when a local datastore is searchedfor in the ESXi host

CSCvn73127

35(2a)26(1b)A node shuts down when two disks are replaced (acaching disk and another capacity disk) where thecapacity disk is inserted first and then the caching diskis inserted

CSCvk46364

Hyper-V

40(1a)35(2a)Cluster expansion may fail with an error code 500 -operation timed out

CSCvn28721

35(2b)35(2a)During upgrade remove the VLAN on team created foruser vSwitch During fresh installation only one VLANtag is set to the vSwitch and team although multipleVLANs were entered

CSCvn54300

35(2b)35(2a)While upgrading a Hyper-V cluster on account of a rarerace condition between the stUpgradeService andZookeeper servers the upgrade orchestration throws anupgrade validation error and the upgrade process isaborted

CSCvn60486

Release Notes for Cisco HX Data Platform Release 4049

Resolved Caveats in Release 40(1a)

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Startrestart EAM service invCenter

When EAM service is not runningvCenter registration is failing

CSCvt55712

35(2a)Contact Cisco TAC if you believeyou are seeing similar conditions

HyperFlex Stretch Cluster saw poorapplication performance due to writelatency cluster remained unhealthy andrebalance was stuck

This defect is currently being used forinvestigative purposes

CSCvt45344

35(2g)Remove log files fromvarlogspringpath folder when thelog files are not getting rotatedStop the relevant service removelarge log files fromvarlogspringpath folder andrestart the service

varstv folder in Controller VM maybecome full

CSCvt36374

40(2b)Restarting the stMgr will initializeit again

In a Cluster configured with DR networkStMgr may not get initialized and stuckin a deadlock while enabling the IPTablerules (can be verified from the stMgrlog)

CSCvt89709

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Click recover for the selectedVM again under the standalonemode This will move the VM torecovered state

When the user recovers a VM protectedin a group it along with other VMs in thegroup move to recovering state instandalonemode The selectedVM shouldmove to recovered state

CSCvs75553

35(2e)Clean up the fail over cluster andthen recreate the fail over clusterNo need to touch the HX storagecluster

Hyper-V When replicated DC was usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2d)40(1a)

Keep cluster utilization as low aspossible To avoid read-only statekeep below 92

HyperFlex clusters storage utilization isabove 92 usable capacity

CSCvr20922

Release Notes for Cisco HX Data Platform Release 4050

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1b)NAAll the disks in the node got locked afterrebooting the node

We successfully unlocked usingsed-clientsh -U command but wantedto test with another reboot and driveslocked again

CSCvs74286

35(2g)To recover the VM copy over thedata disks and attach them to thenew VM

After enable encryption HX connectencryption status showing Caution butall of the disks enable encryptionsuccessful

-Self Encrypting Drives Service isrunning on all of the nodes

-There is no error message of encryption

- USB0 interface is up

-All SED disks showing supported 1enabled 1 locked 0

CSCvs93245

40(1b)Enable kernelpanic_on_oops as apersistent configuration This willcause the node to panic and rebootimmediately

A cluster node running HX release40(1b) may hang in the Linux kernelThis is classified as an oops and adeviation from the expected behavior

CSCvs54285

Release Notes for Cisco HX Data Platform Release 4051

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Enabling K8iscsi stack in Hx clusterCSCvs41324

Release Notes for Cisco HX Data Platform Release 4052

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Steps to enable K8 on HX cluster

Perform following steps on ALLthe controller VMs

1 Updateetcinitscvmclientconf toenable tune iscsiEnable totrue

sed -iesiscsiEnable=falseiscsiEnable=trueetcinitscvmclientconf

2 Run following initctlcommand to reloadconfiguration

initctl reload-configuration

3 Restart scvmclient process

stop scvmclient startscvmclient

4 Run following initctlcommand to mount SYSTEMDS

initctl emit --no-waitsystem-datastore-created

5 Verify that iscsiEnable tuneis set to true using thefollowing command

ps -eaf | grep scvmclient |grep -v grep

root 6241 1 1 Dec06 014007optspringpathstorfs-corescvmclient-T logEnabled=false -TlogSyslogEnabled=true -TlogEchoToScreen=false -TstatLoggingToFile=false -TstatLoggingToSyslog=true-TlogDir=varlogspringpath-TnfsBackendServerList=1010748100-T iscsiEnable=true -TiscsiUseAsync=true -T

Release Notes for Cisco HX Data Platform Release 4053

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

iscsiConfigFilePath=nfsSYSTEMistgtconfps -eaf | grep scvmclient |grep -v grep

35(2e)Following procedure can be usedto check NFS Queue Depth andincrease if needed

rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth256 lt- Low value[rootHXESXI1] vsish -e setvmkModulesnfsclientmnt[DATASTORE]propertiesmaxQDepth 1024[rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth1024 lt- Optimalvalue

This requires an ESXi host rebootto take effect

Please place the node in HyperflexMaintenanceMode and gracefullyreboot the node for the changes tobe applied

HyperFlex Datastore NFS Queue Depthshows as 256 which can lead toperformance (including latency) issues

CSCvr83056

40(1b)This is cosmetic The error will goaway once the cluster returns tohealthy status

When a single node is offline in clusterstcli cluster storage-summary shows twonodes unavailable rootSCVM~ stclicluster storage-summary

CSCvs47735

35(2g)Modify both the root and adminpassword for the SCVMs to be thesame

stcli security password set -uadmin

Cluster expansion fails at Config Installerstage when the root and admin passwordfor the storage controller (SCVM) aredifferent

CSCvs69317

35(2d)NAAfter successfully changing the DNSserver on the HX controller we still cansee the original DNS entry that was addedduring the deployment

CSCvs69154

Release Notes for Cisco HX Data Platform Release 4054

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Customers are strongly encouragedto work with TAC in order toidentify and remediate this issue

1 Change getcluster from 2m to8m inoptspringpathstorfs-mgmtstMgr-10confapplicationconfon all nodes

2 Restart stMgr on all nodes

You may see this error message after afailed upgrade or other task such asattempting to enter a node into HXmaintenance mode

CSCvs53555

40(2a)Run the following command oneach storage controller VM as root

grep -i monitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp sed -iesmonitor_dns_serverstruemonitor_dns_serversfalseoptspringpathhx-diag-toolswatchdog_configjsonampamp grep -imonitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp restart watchdog

Receive the following alertevent in HXConnect HX Controller VMHOSTNAME one or more configuredDNS servers not responding

CSCvt13947

35(2b)Delete any empty (Size 0) log filesunder varlogzookeeperversion-2ORvarlogzookeeperstandaloneversion-2

Zookeeper fails to start while Exhibitor isrunning however echo srvr returnsnothing

CSCvs21562

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

40(4e)1 De-commission the server

2 Power drain the server -REMOVE BOTH POWERCORDS ON THE BACK OFTHE SERVER FOR 10SECONDS THENREINSERTPOWERCORDS

3 Re-commission the server

M2 boot disk is missing from serverinventory after upgrade to server firmware40(4e) As a result server fails to boot toOS installed in the M2 disk The issuepersists after re-acknowledgement as wellas de-commission andre-acknowledgement of the server

CSCvs02466

Release Notes for Cisco HX Data Platform Release 4055

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Re-start the flusher manuallyMany disks taken offline by a fault ormanually resulting in ENOSPC oncluster

CSCvq32721

35(2e)Clean up fail over cluster and thenrecreate the fail over cluster Thereis no need to touch the HX storagecluster

[Hyper-V]When replicated DCwas usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2b)Manually install upgradescvmclient VIB andmake sure thatit matches with HXDP version

Upgrade from release HX 35(2a) to35(2x) 4X does not upgrade scvmclientRecovery point may not work

CSCvq54992

Management

30(1b)To recover the VM copy over thedata disks and attach them to thenew VM

[HyperV]VMshows up onmultiple nodesCSCvj22992

35(1a)1 Do not change MTU at ESXlevel on running cluster

2 Reset back to the original value

ChangingMTU from 9000 to 1500 on oneESX node causes storfs process to restarton all nodes

CSCvm99150

Release Notes for Cisco HX Data Platform Release 4056

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)After the installation (for newdeployments) or after an upgradeto 401a (for existingdeployments) perform thefollowing steps on all ControllerVMs

1 Edit the following tune files (onall Controller VMs)

optspringpathconfiglfftunes

optspringpathconfigvsi_16tbtunes

2 SetcleanerEnableSegSummaryCleaning

to false

3 After editing the above tunefiles

a SSH in to all the ControllerVMs

b Run storfstool -- -Z

c Run the below command andcheck the values Tune valueshould be true

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=true

d Type below commands andapplied the tunes changesdynamically

echo false gttmpstprocfssystemtunecleanerEnableSegSummaryCleaning

e Verify that tune values arechanged Run below commandValue should be false

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=false

f umount tmpstprocfs

A cluster with 8TB or 12TB disk drivesmay experience IO stalls for severalminutes after another node fails in thecluster

CSCvp23718

Release Notes for Cisco HX Data Platform Release 4057

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)1 Update the conf file manuallyto change to the link local addressto

2 Restart scvmclient on thecontroller and ESX

HX 35(2b) FlexVol to HX 40 CSIupgrade does not work FlexVol continuesto work

CSCvp26319

35(2a)Disable rebalance until node healsthen reenable rebalance

7-node cluster healing very slowlybecause rebalance change AMS reqs areblocking healing process

CSCvp62512

35(2a)NAWith moderate cluster storage capacityusage under site failover cluster clustermay experience an all paths (APD) downcondition

CSCvq49412

40(1a)NAThis product includes Third-partySoftware that is affected by thevulnerabilities identified by the followingCommon Vulnerability and Exposures(CVE) IDs

CVE-2016-0762 CVE-2016-6797CVE-2016-6816 CVE-2016-8735CVE-2017-5647 CVE-2017-12615CVE-2017-12616 CVE-2017-12617CVE-2017-7674 CVE-2018-1304CVE-2018-8014 CVE-2018-1336CVE-2018-8034 CVE-2018-11784CVE-2019-0232

CSCvp36364

35(2d)NAstcli cluster info command needs toprovide UCSM VIP address Currently itshows ucsm-hostcom

CSCvq11456

35(2a)No workarounds currently Thestale entries do not affect the dayto day operations of HyperFlex

HyperFlex upgrade validation failedbecause cluster has extraneous stNodes ininternal database

CSCvq32530

35(2a)NAWhen not in maintenancemode all drivesappear in HX Connect and get updated asexpected

When in maintenance mode extra drivesare shown in HX Connect with no slotnumbers

CSCvq42378

35(2b)Contact TAC to cleanup duplicatenode

Nodes duplicated in HX Connect aftercluster recreate

CSCvq61825

Release Notes for Cisco HX Data Platform Release 4058

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)NACurrently HyperFlex installs on Hyper-Vdo not contain the stcli security whitelistcommandset

CSCvq66245

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

Release Notes for Cisco HX Data Platform Release 4059

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)[Hyper-V] Converged node expansionfails for incoming node with VIC 14xx

CSCvq39523

Release Notes for Cisco HX Data Platform Release 4060

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

1 Set Jumbo frame inWindowsHypervisor for incomingnode

Get-NetAdapter storage-data-a|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

Get-NetAdapter storage-data-b|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

2 Verify all interfaces are setcorrectly to support Jumbo frames

Get-NetAdapter storage |Get-NetAdapterAdvancedProperty| RegistryKeyword -matchjumbo | ft -auto

You should receive the followingmessage

Name DisplayName DisplayValue

RegistryKeyword RegistryValue

---- ----------- ------------

--------------- -------------

vswitch-hx-storage-data Jumbo

Packet 9014 Bytes

JumboPacket 9014

storage-data-b Jumbo Packet

Bytes 9014 JumboPacket

9014

storage-data-a Jumbo Packet

Bytes 9014 JumboPacket

9014

3 Reboot only the incoming nodeor the node being expanded

4 Retry Cluster Expansion in

Release Notes for Cisco HX Data Platform Release 4061

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)The cluster expand fails on a node with DR replicationconfigured

When a HX cluster which has DR replication configuredis expanded we see the installer UI pulling in thereplicationVLAN information instead of themanagementVLAN information Even if we change that informationto the correct mgmt VLAN id and name it it does notseem to work as the node is configured with the VLANof the replicationVLAN in ESXi This leads to the failureof the node add with the host unreachable error

CSCvo70650

40(2a)35(1a)Customer reported that server didnt complete thefirmware upgrade automatically one by one

The user manually needs to put the host into maintenancemode and they manually acknowledged the pendingrequests to finish the UCS firmware upgrade

Maintenance policy is set to default (User-ack) as perthe design

CSCvo91624

40(2a)35(1i)When the cluster is in Failed state and stcli noderemove is attempted the output shows successful thoughthe node failed to get removed from the cluster

CSCvo93017

40(2a)35(1a)HyperFlex cluster upgrade may fail during validationwith a DRS Validation failed error

CSCvp31021

40(2a)35(2b)HX cluster expansion will fail with the following errormessage

MAC address pool configuration failure150[ErrorDescription] bad address block range definitioncollision

CSCvp58318

40(2a)35(2b)Memory usage by carbon cacheCSCvq34873

40(2a)35(2g)Upgrading ESXi cluster fails with error Nodemaintenance mode failed

CSCvr03240

40(2a)35(2e)storfs process does not automatically start on Exit HXMaintenance Mode or other tasks which restart StorageController

CSCvr88978

40(2a)35(2a)Post install script fails with message

Failed to execute ipmitool on HX node

CSCvp10707

40(2a)40(1a)HyperFlex expansionworkflow doesnt pull VLANnamecorrectly

CSCvp46539

Release Notes for Cisco HX Data Platform Release 4034

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)During HX cluster deployment - Validate ClusterCreation phase - HX Installer might fail with the error

fromvarlogspringpathstDeploylog

CSCvq45087

40(2a)35(2b)HX Installer fails to login to the SOL for configuringESX hosts

CSCvq91380

40(2a)35(2d)If DRS is enabled with configuration parameters otherthen fully automated ie manual or partially automatedthan this will change to fully automated during clusterexpansion

CSCvr44222

40(2a)35(1a)One of the validation task name has a typo The taskname is Validating node for mixed of SED and non-SEDdisks

It should be Validating node for mix of SED andnon-SED disks

CSCvp82175

40(2a)30(1i)This combination of using smaller NVMe drives is notsupported in expand operation For example if the drivesare only 375GB and they cannot be added to an existingcluster with larger caching SSD

CSCvo12359

40(2a)35(1a)HyperFlex includes a version of OpenSSL that is affectedby the vulnerabilities identified by the followingCommon Vulnerability and Exposures

CVE-2018-0495

CSCvp66679

40(2a)30(1e)SDK invoke exception when taking a native snapshot ona VM with GPU pci passthrough attached

CSCvo00511

40(2a)30(1i)Node replacement script fails due to EAM errorCSCvo62867

40(2a)35(1a)While pairing a cluster (HX release gt= 35) and cluster(HX release lt 35) remote replication network test failson the cluster (HX release lt 35)

CSCvo79760

40(2a)30(1b)Fixed the issue in the latest support-workflow bundleThis should not be hit anymore

CSCvo87061

40(2a)35(2a)On a 3 node cluster the MBreplace script fails as thescript looks for HEALTHY cluster status

CSCvo87080

40(2a)35(2a)MbReplace script (tar) hangs when running on 352b or40

CSCvp12359

Release Notes for Cisco HX Data Platform Release 4035

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)HX replication cleanup failing with error NA 35(2a) -INFODR state is not clean

CSCvp63958

40(2a)40(1a)The check_stig_parameters API incorrectly shows thecompliance state of the cluster as not STIG compliant

CSCvp66277

40(2a)35(2c)The snapshot creation on CBT enabled VM fails witherror Failed in vmreparent vmkfstools clone1

CSCvq06952

40(2a)35(2a)HyperFlex GUI when setting up remote DR-replicationpartnerpeer should show remote replication partner TCPport reachability test results

This will provide quick upfront connectivity results thatfrequently takes customer a while to manually check andconfirm

If reachability tests fail an info window and HX alertshould trigger that list the destination peer IP and thespecific port(s) that could not be reache

CSCvo60587

40(2a)35(2b)If you have more than one IP pool configured for areplication network and local replication network testsare failing you might be running into this issue

CSCvr67130

40(2a)35(2a)Skip task option in MBreplace and Software redeployscripts The scripts should have the option to skip tasksas user inputs and not by modifying the json file

CSCvp05204

40(2a)40(1a)Error noticed on SCVM Console print_req_error IOerror dev fd0 sector 0 on HyperV

It doesnt come up on SSH session

CSCvq34357

40(2a)30(1d)The VMNetwork switch is already created as part of theinstall process

Ideally the ip address assignment should be done duringthe initial install process as well No post install stepsshould be required altrnatively it could be done as partof a post install script

CSCvk36222

40(2a)25(1c)storfs service impacted during HX upgrade from HX21(1) to 25(1c)

HX node which is not being upgraded storfs servicestopped causing outage

CSCvg53223

40(2a)35(1b)HX Stretch Cluster Witness VM reverts to DHCP onreboot

CSCvp37536

40(2a)35(2b)HyperFlex Stretch Cluster Witness does not getprogrammed with an NTP server

CSCvp46578

Release Notes for Cisco HX Data Platform Release 4036

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2b)HyperFlex cluster does not heal after maintenance onnodeserver

CSCvr18528

40(2a)40(2a)High percentage of packet loss on the eth1 data interface9K MTU

CSCvr97089

40(2a)35(2f)The HyperFlex cluster healing state is stuck at 87 andis unable to progress

CSCvr16760

40(2a)40(2a)In some circumstances two replications can occur forthe same VM

From the GUI the replication status for this VM willshow stuck in an In Progress state but there are nojobs stuck in replication layer Subsequent Replicationsshould succeed

CSCvq22898

40(2a)35(2a)After an unexpected power outage the hyperflex clusteris unable to come up correctly

All processes are running time is synced and all nodescan vmkping each other as required

CSCvp13990

40(2a)40(1a)When specifying IP address instead of node ID stclinode maintenanceMode commands fail

CSCvp33657

40(2a)40(1a)2 node edge cluster if experiencing Intersightarbitratorconnectivity issues should NOT allowHXMaintenancemode to initiate as this will bring the cluster down

CSCvr31573

40(2a)35(2c)Old files in varsupportZKTxnlog are not purged withthe daily zklog-cleanup cron job

CSCvr89066

40(2a)35(2a)When a node is in MaintenanceMode any disk removalor replacement will be reflected in UI only after the nodeis brought back frommaintenancemode This is becausestorfs is not running on the node in maintenance andwill not be able to detect disk activities until it is broughtout of Maintenance Mode

CSCvo86431

40(2a)35(2c)Cluster might go offline and stop serving IO requestsrootcvm~ stcli cluster storage-summary--detailGet cluster storage summary failedjavanetConnectException Connection refused19216814210010207

CSCvr01645

40(2a)35(2d)A stretch cluster deployment fails with a messageFormatting nodes Node in Use on the Hyperflex installerand crmZoneType value is seen as 1 in storfscfg oroptspringpathconfigstretchtunes

CSCvq80340

Release Notes for Cisco HX Data Platform Release 4037

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)30(1c)During upgrade we upload the upgrade packagedownloaded from Ciscocom Upgrade packages are offile type tgz hxconnect acceptsstorfs-packages-401a-33028tar file a different type ofcompression and allows upgrade to start and fail whenit cannot decompress the file for bootstrap and scvmclientupgrade on ESXi

CSCvq63888

40(2a)40(2a)While attempting to install hyperflex via intersight thefollowing error is seen

Failed in Task Add hosts to vCenter Cluster with ErrorTry adding hosts manually to vCenter and retry Failedto get addressFailed to host by namelookupHostnamecompanycom on 000053 read udp12700145430-gt12700153 read connection refused

CSCvr52098

40(2a)35(2a)CoreAPI call may time out for cluster management APIwhile doing cluster create validation

CSCvo92952

40(2a)35(2d)Validation fails with Mixed mode expansion checkdue to Enhanced vMotion Compatibility Incompatibilitywhen the ESXi version is different

CSCvq95460

40(2a)30(1i)This failure is seen in installer when custom workflowis used hypervisor configuration + deploy (clean diskpartition=no) at Installing software packages on storagecontroller VM

CSCvr66309

40(2a)35(2d)The HX Installer apparently replaced a comma bysemicolon on the VLAN ID range while exporting theconfiguration to a JSON file

CSCvq93831

40(2a)40(1a)When using unsupported browsers HX Connect userswill not be able to login These users see an errorindicating unauthorized user

CSCvp22693

Release Notes for Cisco HX Data Platform Release 4038

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)40(1a)A vulnerability in the web-based management interfaceof Cisco HyperFlex Software could allow anunauthenticated remote attacker to conduct a cross-siterequest forgery (CSRF) attack on an affected system

The vulnerability is due to insufficient CSRF protectionsfor the web UI on an affected device An attacker couldexploit this vulnerability by persuading a user of theinterface to follow a malicious link A successful exploitcould allow the attacker to perform arbitrary actions withthe privilege level of the affected user

Cisco has released software updates that address thisvulnerability There are no workarounds that address thisvulnerability

This advisory is available at the following link

httpstoolsciscocomsecuritycentercontentCiscoSecurityAdvisorycisco-sa-20190807-hypflex-csrf

CSCvp24343

40(2a)35(2d)Add message to HX Connect Progress flow to notAcknowledge Pending Activities and reboot the serverson UCSM

HX Connect is doing a controlled rolling server upgradein the background

CSCvq22844

40(2a)40(1b)Compute Node expand operation fails due to incorrectcluster name

CSCvr43786

40(2a)26(1e)RF-2 cluster shutdown during data resynchronizationafter a node failure (non storfs) followed by multipledisk read failures and a HardBlacklist of a disk

CSCvp41241

40(2a)40(1a)HyperFlex Edge nodes do not properly set the ESXihostname during deployment

CSCvo13143

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4039

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4040

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1b) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401b-33133p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4041

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4042

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1b)40(1a)While running the HyperFlex installer with WindowsServer Hyper-V cluster creation process fails with theerror Failure occurred during ClusterCreation process Unable to post thecontent to the downstream This symptomis encountered while deploying HyperFlex cluster withHyperFlex nodes configuredordered with Cisco VIC1457 MLOM (PID HX-MLOM-C25Q-04) andWindows Server Datacenter or Core with Hyper-Vversion 2016 or 2019

CSCvp64140

Release Notes for Cisco HX Data Platform Release 4043

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)35(2b)Adding Micron 5200 drive to a cluster fails to increasecluster capacity

CSCvo69067

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4044

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4045

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1a) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401a-33028p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4046

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4047

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1a)30(1d)Cluster instability when disks of different sector sizeplaced in HX node

CSCvk17250

40(1a)

35(2c)

35(1a)When logged in using a local HX user account insteadof a Virtual Center account an error message appearsintermittently indicating Virtual Centerunreachable or Resource informationcannot be updated when VC is reachable

CSCvo36198

40(1a)

35(2b)

30(1d)HXDP does not work with EMC RecoverPoint - needsto support VMware API (FSS-Readdir)

CSCvk38003

Release Notes for Cisco HX Data Platform Release 4048

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)

35(2a)

30(1i)

25(1c)

25(1d)

Zookeeper (Exhibitor) process on HyperFlex StorageController VMs may fail to respond to requests whenvarzookeeper has low or no free space

CSCvm90352

40(1a)26(1c)Use Asynchronous consolidation for HX SnapshotsCSCvn02151

40(1a)35(2b)HX Quiesced Snapshot for Backup VendorsCSCvo90713

40(1a)35(2a)The cluster creationcluster expansion workflow stopswith the following error message at the validation step

FIRMWARE-Check UCSC-SAS-M5HD

FIRMWARE-Check UCSC-SAS-M5HD Required00000029000000320000003500000050 Found00000058

Action Needed Update the Controller Firmware toRequired Version

CSCvn17787

35(2b)30(1d)Kernel migration fails when a local datastore is searchedfor in the ESXi host

CSCvn73127

35(2a)26(1b)A node shuts down when two disks are replaced (acaching disk and another capacity disk) where thecapacity disk is inserted first and then the caching diskis inserted

CSCvk46364

Hyper-V

40(1a)35(2a)Cluster expansion may fail with an error code 500 -operation timed out

CSCvn28721

35(2b)35(2a)During upgrade remove the VLAN on team created foruser vSwitch During fresh installation only one VLANtag is set to the vSwitch and team although multipleVLANs were entered

CSCvn54300

35(2b)35(2a)While upgrading a Hyper-V cluster on account of a rarerace condition between the stUpgradeService andZookeeper servers the upgrade orchestration throws anupgrade validation error and the upgrade process isaborted

CSCvn60486

Release Notes for Cisco HX Data Platform Release 4049

Resolved Caveats in Release 40(1a)

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Startrestart EAM service invCenter

When EAM service is not runningvCenter registration is failing

CSCvt55712

35(2a)Contact Cisco TAC if you believeyou are seeing similar conditions

HyperFlex Stretch Cluster saw poorapplication performance due to writelatency cluster remained unhealthy andrebalance was stuck

This defect is currently being used forinvestigative purposes

CSCvt45344

35(2g)Remove log files fromvarlogspringpath folder when thelog files are not getting rotatedStop the relevant service removelarge log files fromvarlogspringpath folder andrestart the service

varstv folder in Controller VM maybecome full

CSCvt36374

40(2b)Restarting the stMgr will initializeit again

In a Cluster configured with DR networkStMgr may not get initialized and stuckin a deadlock while enabling the IPTablerules (can be verified from the stMgrlog)

CSCvt89709

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Click recover for the selectedVM again under the standalonemode This will move the VM torecovered state

When the user recovers a VM protectedin a group it along with other VMs in thegroup move to recovering state instandalonemode The selectedVM shouldmove to recovered state

CSCvs75553

35(2e)Clean up the fail over cluster andthen recreate the fail over clusterNo need to touch the HX storagecluster

Hyper-V When replicated DC was usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2d)40(1a)

Keep cluster utilization as low aspossible To avoid read-only statekeep below 92

HyperFlex clusters storage utilization isabove 92 usable capacity

CSCvr20922

Release Notes for Cisco HX Data Platform Release 4050

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1b)NAAll the disks in the node got locked afterrebooting the node

We successfully unlocked usingsed-clientsh -U command but wantedto test with another reboot and driveslocked again

CSCvs74286

35(2g)To recover the VM copy over thedata disks and attach them to thenew VM

After enable encryption HX connectencryption status showing Caution butall of the disks enable encryptionsuccessful

-Self Encrypting Drives Service isrunning on all of the nodes

-There is no error message of encryption

- USB0 interface is up

-All SED disks showing supported 1enabled 1 locked 0

CSCvs93245

40(1b)Enable kernelpanic_on_oops as apersistent configuration This willcause the node to panic and rebootimmediately

A cluster node running HX release40(1b) may hang in the Linux kernelThis is classified as an oops and adeviation from the expected behavior

CSCvs54285

Release Notes for Cisco HX Data Platform Release 4051

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Enabling K8iscsi stack in Hx clusterCSCvs41324

Release Notes for Cisco HX Data Platform Release 4052

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Steps to enable K8 on HX cluster

Perform following steps on ALLthe controller VMs

1 Updateetcinitscvmclientconf toenable tune iscsiEnable totrue

sed -iesiscsiEnable=falseiscsiEnable=trueetcinitscvmclientconf

2 Run following initctlcommand to reloadconfiguration

initctl reload-configuration

3 Restart scvmclient process

stop scvmclient startscvmclient

4 Run following initctlcommand to mount SYSTEMDS

initctl emit --no-waitsystem-datastore-created

5 Verify that iscsiEnable tuneis set to true using thefollowing command

ps -eaf | grep scvmclient |grep -v grep

root 6241 1 1 Dec06 014007optspringpathstorfs-corescvmclient-T logEnabled=false -TlogSyslogEnabled=true -TlogEchoToScreen=false -TstatLoggingToFile=false -TstatLoggingToSyslog=true-TlogDir=varlogspringpath-TnfsBackendServerList=1010748100-T iscsiEnable=true -TiscsiUseAsync=true -T

Release Notes for Cisco HX Data Platform Release 4053

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

iscsiConfigFilePath=nfsSYSTEMistgtconfps -eaf | grep scvmclient |grep -v grep

35(2e)Following procedure can be usedto check NFS Queue Depth andincrease if needed

rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth256 lt- Low value[rootHXESXI1] vsish -e setvmkModulesnfsclientmnt[DATASTORE]propertiesmaxQDepth 1024[rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth1024 lt- Optimalvalue

This requires an ESXi host rebootto take effect

Please place the node in HyperflexMaintenanceMode and gracefullyreboot the node for the changes tobe applied

HyperFlex Datastore NFS Queue Depthshows as 256 which can lead toperformance (including latency) issues

CSCvr83056

40(1b)This is cosmetic The error will goaway once the cluster returns tohealthy status

When a single node is offline in clusterstcli cluster storage-summary shows twonodes unavailable rootSCVM~ stclicluster storage-summary

CSCvs47735

35(2g)Modify both the root and adminpassword for the SCVMs to be thesame

stcli security password set -uadmin

Cluster expansion fails at Config Installerstage when the root and admin passwordfor the storage controller (SCVM) aredifferent

CSCvs69317

35(2d)NAAfter successfully changing the DNSserver on the HX controller we still cansee the original DNS entry that was addedduring the deployment

CSCvs69154

Release Notes for Cisco HX Data Platform Release 4054

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Customers are strongly encouragedto work with TAC in order toidentify and remediate this issue

1 Change getcluster from 2m to8m inoptspringpathstorfs-mgmtstMgr-10confapplicationconfon all nodes

2 Restart stMgr on all nodes

You may see this error message after afailed upgrade or other task such asattempting to enter a node into HXmaintenance mode

CSCvs53555

40(2a)Run the following command oneach storage controller VM as root

grep -i monitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp sed -iesmonitor_dns_serverstruemonitor_dns_serversfalseoptspringpathhx-diag-toolswatchdog_configjsonampamp grep -imonitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp restart watchdog

Receive the following alertevent in HXConnect HX Controller VMHOSTNAME one or more configuredDNS servers not responding

CSCvt13947

35(2b)Delete any empty (Size 0) log filesunder varlogzookeeperversion-2ORvarlogzookeeperstandaloneversion-2

Zookeeper fails to start while Exhibitor isrunning however echo srvr returnsnothing

CSCvs21562

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

40(4e)1 De-commission the server

2 Power drain the server -REMOVE BOTH POWERCORDS ON THE BACK OFTHE SERVER FOR 10SECONDS THENREINSERTPOWERCORDS

3 Re-commission the server

M2 boot disk is missing from serverinventory after upgrade to server firmware40(4e) As a result server fails to boot toOS installed in the M2 disk The issuepersists after re-acknowledgement as wellas de-commission andre-acknowledgement of the server

CSCvs02466

Release Notes for Cisco HX Data Platform Release 4055

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Re-start the flusher manuallyMany disks taken offline by a fault ormanually resulting in ENOSPC oncluster

CSCvq32721

35(2e)Clean up fail over cluster and thenrecreate the fail over cluster Thereis no need to touch the HX storagecluster

[Hyper-V]When replicated DCwas usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2b)Manually install upgradescvmclient VIB andmake sure thatit matches with HXDP version

Upgrade from release HX 35(2a) to35(2x) 4X does not upgrade scvmclientRecovery point may not work

CSCvq54992

Management

30(1b)To recover the VM copy over thedata disks and attach them to thenew VM

[HyperV]VMshows up onmultiple nodesCSCvj22992

35(1a)1 Do not change MTU at ESXlevel on running cluster

2 Reset back to the original value

ChangingMTU from 9000 to 1500 on oneESX node causes storfs process to restarton all nodes

CSCvm99150

Release Notes for Cisco HX Data Platform Release 4056

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)After the installation (for newdeployments) or after an upgradeto 401a (for existingdeployments) perform thefollowing steps on all ControllerVMs

1 Edit the following tune files (onall Controller VMs)

optspringpathconfiglfftunes

optspringpathconfigvsi_16tbtunes

2 SetcleanerEnableSegSummaryCleaning

to false

3 After editing the above tunefiles

a SSH in to all the ControllerVMs

b Run storfstool -- -Z

c Run the below command andcheck the values Tune valueshould be true

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=true

d Type below commands andapplied the tunes changesdynamically

echo false gttmpstprocfssystemtunecleanerEnableSegSummaryCleaning

e Verify that tune values arechanged Run below commandValue should be false

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=false

f umount tmpstprocfs

A cluster with 8TB or 12TB disk drivesmay experience IO stalls for severalminutes after another node fails in thecluster

CSCvp23718

Release Notes for Cisco HX Data Platform Release 4057

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)1 Update the conf file manuallyto change to the link local addressto

2 Restart scvmclient on thecontroller and ESX

HX 35(2b) FlexVol to HX 40 CSIupgrade does not work FlexVol continuesto work

CSCvp26319

35(2a)Disable rebalance until node healsthen reenable rebalance

7-node cluster healing very slowlybecause rebalance change AMS reqs areblocking healing process

CSCvp62512

35(2a)NAWith moderate cluster storage capacityusage under site failover cluster clustermay experience an all paths (APD) downcondition

CSCvq49412

40(1a)NAThis product includes Third-partySoftware that is affected by thevulnerabilities identified by the followingCommon Vulnerability and Exposures(CVE) IDs

CVE-2016-0762 CVE-2016-6797CVE-2016-6816 CVE-2016-8735CVE-2017-5647 CVE-2017-12615CVE-2017-12616 CVE-2017-12617CVE-2017-7674 CVE-2018-1304CVE-2018-8014 CVE-2018-1336CVE-2018-8034 CVE-2018-11784CVE-2019-0232

CSCvp36364

35(2d)NAstcli cluster info command needs toprovide UCSM VIP address Currently itshows ucsm-hostcom

CSCvq11456

35(2a)No workarounds currently Thestale entries do not affect the dayto day operations of HyperFlex

HyperFlex upgrade validation failedbecause cluster has extraneous stNodes ininternal database

CSCvq32530

35(2a)NAWhen not in maintenancemode all drivesappear in HX Connect and get updated asexpected

When in maintenance mode extra drivesare shown in HX Connect with no slotnumbers

CSCvq42378

35(2b)Contact TAC to cleanup duplicatenode

Nodes duplicated in HX Connect aftercluster recreate

CSCvq61825

Release Notes for Cisco HX Data Platform Release 4058

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)NACurrently HyperFlex installs on Hyper-Vdo not contain the stcli security whitelistcommandset

CSCvq66245

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

Release Notes for Cisco HX Data Platform Release 4059

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)[Hyper-V] Converged node expansionfails for incoming node with VIC 14xx

CSCvq39523

Release Notes for Cisco HX Data Platform Release 4060

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

1 Set Jumbo frame inWindowsHypervisor for incomingnode

Get-NetAdapter storage-data-a|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

Get-NetAdapter storage-data-b|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

2 Verify all interfaces are setcorrectly to support Jumbo frames

Get-NetAdapter storage |Get-NetAdapterAdvancedProperty| RegistryKeyword -matchjumbo | ft -auto

You should receive the followingmessage

Name DisplayName DisplayValue

RegistryKeyword RegistryValue

---- ----------- ------------

--------------- -------------

vswitch-hx-storage-data Jumbo

Packet 9014 Bytes

JumboPacket 9014

storage-data-b Jumbo Packet

Bytes 9014 JumboPacket

9014

storage-data-a Jumbo Packet

Bytes 9014 JumboPacket

9014

3 Reboot only the incoming nodeor the node being expanded

4 Retry Cluster Expansion in

Release Notes for Cisco HX Data Platform Release 4061

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)During HX cluster deployment - Validate ClusterCreation phase - HX Installer might fail with the error

fromvarlogspringpathstDeploylog

CSCvq45087

40(2a)35(2b)HX Installer fails to login to the SOL for configuringESX hosts

CSCvq91380

40(2a)35(2d)If DRS is enabled with configuration parameters otherthen fully automated ie manual or partially automatedthan this will change to fully automated during clusterexpansion

CSCvr44222

40(2a)35(1a)One of the validation task name has a typo The taskname is Validating node for mixed of SED and non-SEDdisks

It should be Validating node for mix of SED andnon-SED disks

CSCvp82175

40(2a)30(1i)This combination of using smaller NVMe drives is notsupported in expand operation For example if the drivesare only 375GB and they cannot be added to an existingcluster with larger caching SSD

CSCvo12359

40(2a)35(1a)HyperFlex includes a version of OpenSSL that is affectedby the vulnerabilities identified by the followingCommon Vulnerability and Exposures

CVE-2018-0495

CSCvp66679

40(2a)30(1e)SDK invoke exception when taking a native snapshot ona VM with GPU pci passthrough attached

CSCvo00511

40(2a)30(1i)Node replacement script fails due to EAM errorCSCvo62867

40(2a)35(1a)While pairing a cluster (HX release gt= 35) and cluster(HX release lt 35) remote replication network test failson the cluster (HX release lt 35)

CSCvo79760

40(2a)30(1b)Fixed the issue in the latest support-workflow bundleThis should not be hit anymore

CSCvo87061

40(2a)35(2a)On a 3 node cluster the MBreplace script fails as thescript looks for HEALTHY cluster status

CSCvo87080

40(2a)35(2a)MbReplace script (tar) hangs when running on 352b or40

CSCvp12359

Release Notes for Cisco HX Data Platform Release 4035

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)HX replication cleanup failing with error NA 35(2a) -INFODR state is not clean

CSCvp63958

40(2a)40(1a)The check_stig_parameters API incorrectly shows thecompliance state of the cluster as not STIG compliant

CSCvp66277

40(2a)35(2c)The snapshot creation on CBT enabled VM fails witherror Failed in vmreparent vmkfstools clone1

CSCvq06952

40(2a)35(2a)HyperFlex GUI when setting up remote DR-replicationpartnerpeer should show remote replication partner TCPport reachability test results

This will provide quick upfront connectivity results thatfrequently takes customer a while to manually check andconfirm

If reachability tests fail an info window and HX alertshould trigger that list the destination peer IP and thespecific port(s) that could not be reache

CSCvo60587

40(2a)35(2b)If you have more than one IP pool configured for areplication network and local replication network testsare failing you might be running into this issue

CSCvr67130

40(2a)35(2a)Skip task option in MBreplace and Software redeployscripts The scripts should have the option to skip tasksas user inputs and not by modifying the json file

CSCvp05204

40(2a)40(1a)Error noticed on SCVM Console print_req_error IOerror dev fd0 sector 0 on HyperV

It doesnt come up on SSH session

CSCvq34357

40(2a)30(1d)The VMNetwork switch is already created as part of theinstall process

Ideally the ip address assignment should be done duringthe initial install process as well No post install stepsshould be required altrnatively it could be done as partof a post install script

CSCvk36222

40(2a)25(1c)storfs service impacted during HX upgrade from HX21(1) to 25(1c)

HX node which is not being upgraded storfs servicestopped causing outage

CSCvg53223

40(2a)35(1b)HX Stretch Cluster Witness VM reverts to DHCP onreboot

CSCvp37536

40(2a)35(2b)HyperFlex Stretch Cluster Witness does not getprogrammed with an NTP server

CSCvp46578

Release Notes for Cisco HX Data Platform Release 4036

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2b)HyperFlex cluster does not heal after maintenance onnodeserver

CSCvr18528

40(2a)40(2a)High percentage of packet loss on the eth1 data interface9K MTU

CSCvr97089

40(2a)35(2f)The HyperFlex cluster healing state is stuck at 87 andis unable to progress

CSCvr16760

40(2a)40(2a)In some circumstances two replications can occur forthe same VM

From the GUI the replication status for this VM willshow stuck in an In Progress state but there are nojobs stuck in replication layer Subsequent Replicationsshould succeed

CSCvq22898

40(2a)35(2a)After an unexpected power outage the hyperflex clusteris unable to come up correctly

All processes are running time is synced and all nodescan vmkping each other as required

CSCvp13990

40(2a)40(1a)When specifying IP address instead of node ID stclinode maintenanceMode commands fail

CSCvp33657

40(2a)40(1a)2 node edge cluster if experiencing Intersightarbitratorconnectivity issues should NOT allowHXMaintenancemode to initiate as this will bring the cluster down

CSCvr31573

40(2a)35(2c)Old files in varsupportZKTxnlog are not purged withthe daily zklog-cleanup cron job

CSCvr89066

40(2a)35(2a)When a node is in MaintenanceMode any disk removalor replacement will be reflected in UI only after the nodeis brought back frommaintenancemode This is becausestorfs is not running on the node in maintenance andwill not be able to detect disk activities until it is broughtout of Maintenance Mode

CSCvo86431

40(2a)35(2c)Cluster might go offline and stop serving IO requestsrootcvm~ stcli cluster storage-summary--detailGet cluster storage summary failedjavanetConnectException Connection refused19216814210010207

CSCvr01645

40(2a)35(2d)A stretch cluster deployment fails with a messageFormatting nodes Node in Use on the Hyperflex installerand crmZoneType value is seen as 1 in storfscfg oroptspringpathconfigstretchtunes

CSCvq80340

Release Notes for Cisco HX Data Platform Release 4037

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)30(1c)During upgrade we upload the upgrade packagedownloaded from Ciscocom Upgrade packages are offile type tgz hxconnect acceptsstorfs-packages-401a-33028tar file a different type ofcompression and allows upgrade to start and fail whenit cannot decompress the file for bootstrap and scvmclientupgrade on ESXi

CSCvq63888

40(2a)40(2a)While attempting to install hyperflex via intersight thefollowing error is seen

Failed in Task Add hosts to vCenter Cluster with ErrorTry adding hosts manually to vCenter and retry Failedto get addressFailed to host by namelookupHostnamecompanycom on 000053 read udp12700145430-gt12700153 read connection refused

CSCvr52098

40(2a)35(2a)CoreAPI call may time out for cluster management APIwhile doing cluster create validation

CSCvo92952

40(2a)35(2d)Validation fails with Mixed mode expansion checkdue to Enhanced vMotion Compatibility Incompatibilitywhen the ESXi version is different

CSCvq95460

40(2a)30(1i)This failure is seen in installer when custom workflowis used hypervisor configuration + deploy (clean diskpartition=no) at Installing software packages on storagecontroller VM

CSCvr66309

40(2a)35(2d)The HX Installer apparently replaced a comma bysemicolon on the VLAN ID range while exporting theconfiguration to a JSON file

CSCvq93831

40(2a)40(1a)When using unsupported browsers HX Connect userswill not be able to login These users see an errorindicating unauthorized user

CSCvp22693

Release Notes for Cisco HX Data Platform Release 4038

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)40(1a)A vulnerability in the web-based management interfaceof Cisco HyperFlex Software could allow anunauthenticated remote attacker to conduct a cross-siterequest forgery (CSRF) attack on an affected system

The vulnerability is due to insufficient CSRF protectionsfor the web UI on an affected device An attacker couldexploit this vulnerability by persuading a user of theinterface to follow a malicious link A successful exploitcould allow the attacker to perform arbitrary actions withthe privilege level of the affected user

Cisco has released software updates that address thisvulnerability There are no workarounds that address thisvulnerability

This advisory is available at the following link

httpstoolsciscocomsecuritycentercontentCiscoSecurityAdvisorycisco-sa-20190807-hypflex-csrf

CSCvp24343

40(2a)35(2d)Add message to HX Connect Progress flow to notAcknowledge Pending Activities and reboot the serverson UCSM

HX Connect is doing a controlled rolling server upgradein the background

CSCvq22844

40(2a)40(1b)Compute Node expand operation fails due to incorrectcluster name

CSCvr43786

40(2a)26(1e)RF-2 cluster shutdown during data resynchronizationafter a node failure (non storfs) followed by multipledisk read failures and a HardBlacklist of a disk

CSCvp41241

40(2a)40(1a)HyperFlex Edge nodes do not properly set the ESXihostname during deployment

CSCvo13143

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4039

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4040

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1b) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401b-33133p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4041

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4042

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1b)40(1a)While running the HyperFlex installer with WindowsServer Hyper-V cluster creation process fails with theerror Failure occurred during ClusterCreation process Unable to post thecontent to the downstream This symptomis encountered while deploying HyperFlex cluster withHyperFlex nodes configuredordered with Cisco VIC1457 MLOM (PID HX-MLOM-C25Q-04) andWindows Server Datacenter or Core with Hyper-Vversion 2016 or 2019

CSCvp64140

Release Notes for Cisco HX Data Platform Release 4043

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)35(2b)Adding Micron 5200 drive to a cluster fails to increasecluster capacity

CSCvo69067

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4044

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4045

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1a) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401a-33028p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4046

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4047

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1a)30(1d)Cluster instability when disks of different sector sizeplaced in HX node

CSCvk17250

40(1a)

35(2c)

35(1a)When logged in using a local HX user account insteadof a Virtual Center account an error message appearsintermittently indicating Virtual Centerunreachable or Resource informationcannot be updated when VC is reachable

CSCvo36198

40(1a)

35(2b)

30(1d)HXDP does not work with EMC RecoverPoint - needsto support VMware API (FSS-Readdir)

CSCvk38003

Release Notes for Cisco HX Data Platform Release 4048

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)

35(2a)

30(1i)

25(1c)

25(1d)

Zookeeper (Exhibitor) process on HyperFlex StorageController VMs may fail to respond to requests whenvarzookeeper has low or no free space

CSCvm90352

40(1a)26(1c)Use Asynchronous consolidation for HX SnapshotsCSCvn02151

40(1a)35(2b)HX Quiesced Snapshot for Backup VendorsCSCvo90713

40(1a)35(2a)The cluster creationcluster expansion workflow stopswith the following error message at the validation step

FIRMWARE-Check UCSC-SAS-M5HD

FIRMWARE-Check UCSC-SAS-M5HD Required00000029000000320000003500000050 Found00000058

Action Needed Update the Controller Firmware toRequired Version

CSCvn17787

35(2b)30(1d)Kernel migration fails when a local datastore is searchedfor in the ESXi host

CSCvn73127

35(2a)26(1b)A node shuts down when two disks are replaced (acaching disk and another capacity disk) where thecapacity disk is inserted first and then the caching diskis inserted

CSCvk46364

Hyper-V

40(1a)35(2a)Cluster expansion may fail with an error code 500 -operation timed out

CSCvn28721

35(2b)35(2a)During upgrade remove the VLAN on team created foruser vSwitch During fresh installation only one VLANtag is set to the vSwitch and team although multipleVLANs were entered

CSCvn54300

35(2b)35(2a)While upgrading a Hyper-V cluster on account of a rarerace condition between the stUpgradeService andZookeeper servers the upgrade orchestration throws anupgrade validation error and the upgrade process isaborted

CSCvn60486

Release Notes for Cisco HX Data Platform Release 4049

Resolved Caveats in Release 40(1a)

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Startrestart EAM service invCenter

When EAM service is not runningvCenter registration is failing

CSCvt55712

35(2a)Contact Cisco TAC if you believeyou are seeing similar conditions

HyperFlex Stretch Cluster saw poorapplication performance due to writelatency cluster remained unhealthy andrebalance was stuck

This defect is currently being used forinvestigative purposes

CSCvt45344

35(2g)Remove log files fromvarlogspringpath folder when thelog files are not getting rotatedStop the relevant service removelarge log files fromvarlogspringpath folder andrestart the service

varstv folder in Controller VM maybecome full

CSCvt36374

40(2b)Restarting the stMgr will initializeit again

In a Cluster configured with DR networkStMgr may not get initialized and stuckin a deadlock while enabling the IPTablerules (can be verified from the stMgrlog)

CSCvt89709

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Click recover for the selectedVM again under the standalonemode This will move the VM torecovered state

When the user recovers a VM protectedin a group it along with other VMs in thegroup move to recovering state instandalonemode The selectedVM shouldmove to recovered state

CSCvs75553

35(2e)Clean up the fail over cluster andthen recreate the fail over clusterNo need to touch the HX storagecluster

Hyper-V When replicated DC was usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2d)40(1a)

Keep cluster utilization as low aspossible To avoid read-only statekeep below 92

HyperFlex clusters storage utilization isabove 92 usable capacity

CSCvr20922

Release Notes for Cisco HX Data Platform Release 4050

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1b)NAAll the disks in the node got locked afterrebooting the node

We successfully unlocked usingsed-clientsh -U command but wantedto test with another reboot and driveslocked again

CSCvs74286

35(2g)To recover the VM copy over thedata disks and attach them to thenew VM

After enable encryption HX connectencryption status showing Caution butall of the disks enable encryptionsuccessful

-Self Encrypting Drives Service isrunning on all of the nodes

-There is no error message of encryption

- USB0 interface is up

-All SED disks showing supported 1enabled 1 locked 0

CSCvs93245

40(1b)Enable kernelpanic_on_oops as apersistent configuration This willcause the node to panic and rebootimmediately

A cluster node running HX release40(1b) may hang in the Linux kernelThis is classified as an oops and adeviation from the expected behavior

CSCvs54285

Release Notes for Cisco HX Data Platform Release 4051

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Enabling K8iscsi stack in Hx clusterCSCvs41324

Release Notes for Cisco HX Data Platform Release 4052

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Steps to enable K8 on HX cluster

Perform following steps on ALLthe controller VMs

1 Updateetcinitscvmclientconf toenable tune iscsiEnable totrue

sed -iesiscsiEnable=falseiscsiEnable=trueetcinitscvmclientconf

2 Run following initctlcommand to reloadconfiguration

initctl reload-configuration

3 Restart scvmclient process

stop scvmclient startscvmclient

4 Run following initctlcommand to mount SYSTEMDS

initctl emit --no-waitsystem-datastore-created

5 Verify that iscsiEnable tuneis set to true using thefollowing command

ps -eaf | grep scvmclient |grep -v grep

root 6241 1 1 Dec06 014007optspringpathstorfs-corescvmclient-T logEnabled=false -TlogSyslogEnabled=true -TlogEchoToScreen=false -TstatLoggingToFile=false -TstatLoggingToSyslog=true-TlogDir=varlogspringpath-TnfsBackendServerList=1010748100-T iscsiEnable=true -TiscsiUseAsync=true -T

Release Notes for Cisco HX Data Platform Release 4053

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

iscsiConfigFilePath=nfsSYSTEMistgtconfps -eaf | grep scvmclient |grep -v grep

35(2e)Following procedure can be usedto check NFS Queue Depth andincrease if needed

rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth256 lt- Low value[rootHXESXI1] vsish -e setvmkModulesnfsclientmnt[DATASTORE]propertiesmaxQDepth 1024[rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth1024 lt- Optimalvalue

This requires an ESXi host rebootto take effect

Please place the node in HyperflexMaintenanceMode and gracefullyreboot the node for the changes tobe applied

HyperFlex Datastore NFS Queue Depthshows as 256 which can lead toperformance (including latency) issues

CSCvr83056

40(1b)This is cosmetic The error will goaway once the cluster returns tohealthy status

When a single node is offline in clusterstcli cluster storage-summary shows twonodes unavailable rootSCVM~ stclicluster storage-summary

CSCvs47735

35(2g)Modify both the root and adminpassword for the SCVMs to be thesame

stcli security password set -uadmin

Cluster expansion fails at Config Installerstage when the root and admin passwordfor the storage controller (SCVM) aredifferent

CSCvs69317

35(2d)NAAfter successfully changing the DNSserver on the HX controller we still cansee the original DNS entry that was addedduring the deployment

CSCvs69154

Release Notes for Cisco HX Data Platform Release 4054

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Customers are strongly encouragedto work with TAC in order toidentify and remediate this issue

1 Change getcluster from 2m to8m inoptspringpathstorfs-mgmtstMgr-10confapplicationconfon all nodes

2 Restart stMgr on all nodes

You may see this error message after afailed upgrade or other task such asattempting to enter a node into HXmaintenance mode

CSCvs53555

40(2a)Run the following command oneach storage controller VM as root

grep -i monitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp sed -iesmonitor_dns_serverstruemonitor_dns_serversfalseoptspringpathhx-diag-toolswatchdog_configjsonampamp grep -imonitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp restart watchdog

Receive the following alertevent in HXConnect HX Controller VMHOSTNAME one or more configuredDNS servers not responding

CSCvt13947

35(2b)Delete any empty (Size 0) log filesunder varlogzookeeperversion-2ORvarlogzookeeperstandaloneversion-2

Zookeeper fails to start while Exhibitor isrunning however echo srvr returnsnothing

CSCvs21562

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

40(4e)1 De-commission the server

2 Power drain the server -REMOVE BOTH POWERCORDS ON THE BACK OFTHE SERVER FOR 10SECONDS THENREINSERTPOWERCORDS

3 Re-commission the server

M2 boot disk is missing from serverinventory after upgrade to server firmware40(4e) As a result server fails to boot toOS installed in the M2 disk The issuepersists after re-acknowledgement as wellas de-commission andre-acknowledgement of the server

CSCvs02466

Release Notes for Cisco HX Data Platform Release 4055

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Re-start the flusher manuallyMany disks taken offline by a fault ormanually resulting in ENOSPC oncluster

CSCvq32721

35(2e)Clean up fail over cluster and thenrecreate the fail over cluster Thereis no need to touch the HX storagecluster

[Hyper-V]When replicated DCwas usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2b)Manually install upgradescvmclient VIB andmake sure thatit matches with HXDP version

Upgrade from release HX 35(2a) to35(2x) 4X does not upgrade scvmclientRecovery point may not work

CSCvq54992

Management

30(1b)To recover the VM copy over thedata disks and attach them to thenew VM

[HyperV]VMshows up onmultiple nodesCSCvj22992

35(1a)1 Do not change MTU at ESXlevel on running cluster

2 Reset back to the original value

ChangingMTU from 9000 to 1500 on oneESX node causes storfs process to restarton all nodes

CSCvm99150

Release Notes for Cisco HX Data Platform Release 4056

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)After the installation (for newdeployments) or after an upgradeto 401a (for existingdeployments) perform thefollowing steps on all ControllerVMs

1 Edit the following tune files (onall Controller VMs)

optspringpathconfiglfftunes

optspringpathconfigvsi_16tbtunes

2 SetcleanerEnableSegSummaryCleaning

to false

3 After editing the above tunefiles

a SSH in to all the ControllerVMs

b Run storfstool -- -Z

c Run the below command andcheck the values Tune valueshould be true

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=true

d Type below commands andapplied the tunes changesdynamically

echo false gttmpstprocfssystemtunecleanerEnableSegSummaryCleaning

e Verify that tune values arechanged Run below commandValue should be false

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=false

f umount tmpstprocfs

A cluster with 8TB or 12TB disk drivesmay experience IO stalls for severalminutes after another node fails in thecluster

CSCvp23718

Release Notes for Cisco HX Data Platform Release 4057

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)1 Update the conf file manuallyto change to the link local addressto

2 Restart scvmclient on thecontroller and ESX

HX 35(2b) FlexVol to HX 40 CSIupgrade does not work FlexVol continuesto work

CSCvp26319

35(2a)Disable rebalance until node healsthen reenable rebalance

7-node cluster healing very slowlybecause rebalance change AMS reqs areblocking healing process

CSCvp62512

35(2a)NAWith moderate cluster storage capacityusage under site failover cluster clustermay experience an all paths (APD) downcondition

CSCvq49412

40(1a)NAThis product includes Third-partySoftware that is affected by thevulnerabilities identified by the followingCommon Vulnerability and Exposures(CVE) IDs

CVE-2016-0762 CVE-2016-6797CVE-2016-6816 CVE-2016-8735CVE-2017-5647 CVE-2017-12615CVE-2017-12616 CVE-2017-12617CVE-2017-7674 CVE-2018-1304CVE-2018-8014 CVE-2018-1336CVE-2018-8034 CVE-2018-11784CVE-2019-0232

CSCvp36364

35(2d)NAstcli cluster info command needs toprovide UCSM VIP address Currently itshows ucsm-hostcom

CSCvq11456

35(2a)No workarounds currently Thestale entries do not affect the dayto day operations of HyperFlex

HyperFlex upgrade validation failedbecause cluster has extraneous stNodes ininternal database

CSCvq32530

35(2a)NAWhen not in maintenancemode all drivesappear in HX Connect and get updated asexpected

When in maintenance mode extra drivesare shown in HX Connect with no slotnumbers

CSCvq42378

35(2b)Contact TAC to cleanup duplicatenode

Nodes duplicated in HX Connect aftercluster recreate

CSCvq61825

Release Notes for Cisco HX Data Platform Release 4058

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)NACurrently HyperFlex installs on Hyper-Vdo not contain the stcli security whitelistcommandset

CSCvq66245

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

Release Notes for Cisco HX Data Platform Release 4059

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)[Hyper-V] Converged node expansionfails for incoming node with VIC 14xx

CSCvq39523

Release Notes for Cisco HX Data Platform Release 4060

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

1 Set Jumbo frame inWindowsHypervisor for incomingnode

Get-NetAdapter storage-data-a|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

Get-NetAdapter storage-data-b|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

2 Verify all interfaces are setcorrectly to support Jumbo frames

Get-NetAdapter storage |Get-NetAdapterAdvancedProperty| RegistryKeyword -matchjumbo | ft -auto

You should receive the followingmessage

Name DisplayName DisplayValue

RegistryKeyword RegistryValue

---- ----------- ------------

--------------- -------------

vswitch-hx-storage-data Jumbo

Packet 9014 Bytes

JumboPacket 9014

storage-data-b Jumbo Packet

Bytes 9014 JumboPacket

9014

storage-data-a Jumbo Packet

Bytes 9014 JumboPacket

9014

3 Reboot only the incoming nodeor the node being expanded

4 Retry Cluster Expansion in

Release Notes for Cisco HX Data Platform Release 4061

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2a)HX replication cleanup failing with error NA 35(2a) -INFODR state is not clean

CSCvp63958

40(2a)40(1a)The check_stig_parameters API incorrectly shows thecompliance state of the cluster as not STIG compliant

CSCvp66277

40(2a)35(2c)The snapshot creation on CBT enabled VM fails witherror Failed in vmreparent vmkfstools clone1

CSCvq06952

40(2a)35(2a)HyperFlex GUI when setting up remote DR-replicationpartnerpeer should show remote replication partner TCPport reachability test results

This will provide quick upfront connectivity results thatfrequently takes customer a while to manually check andconfirm

If reachability tests fail an info window and HX alertshould trigger that list the destination peer IP and thespecific port(s) that could not be reache

CSCvo60587

40(2a)35(2b)If you have more than one IP pool configured for areplication network and local replication network testsare failing you might be running into this issue

CSCvr67130

40(2a)35(2a)Skip task option in MBreplace and Software redeployscripts The scripts should have the option to skip tasksas user inputs and not by modifying the json file

CSCvp05204

40(2a)40(1a)Error noticed on SCVM Console print_req_error IOerror dev fd0 sector 0 on HyperV

It doesnt come up on SSH session

CSCvq34357

40(2a)30(1d)The VMNetwork switch is already created as part of theinstall process

Ideally the ip address assignment should be done duringthe initial install process as well No post install stepsshould be required altrnatively it could be done as partof a post install script

CSCvk36222

40(2a)25(1c)storfs service impacted during HX upgrade from HX21(1) to 25(1c)

HX node which is not being upgraded storfs servicestopped causing outage

CSCvg53223

40(2a)35(1b)HX Stretch Cluster Witness VM reverts to DHCP onreboot

CSCvp37536

40(2a)35(2b)HyperFlex Stretch Cluster Witness does not getprogrammed with an NTP server

CSCvp46578

Release Notes for Cisco HX Data Platform Release 4036

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2b)HyperFlex cluster does not heal after maintenance onnodeserver

CSCvr18528

40(2a)40(2a)High percentage of packet loss on the eth1 data interface9K MTU

CSCvr97089

40(2a)35(2f)The HyperFlex cluster healing state is stuck at 87 andis unable to progress

CSCvr16760

40(2a)40(2a)In some circumstances two replications can occur forthe same VM

From the GUI the replication status for this VM willshow stuck in an In Progress state but there are nojobs stuck in replication layer Subsequent Replicationsshould succeed

CSCvq22898

40(2a)35(2a)After an unexpected power outage the hyperflex clusteris unable to come up correctly

All processes are running time is synced and all nodescan vmkping each other as required

CSCvp13990

40(2a)40(1a)When specifying IP address instead of node ID stclinode maintenanceMode commands fail

CSCvp33657

40(2a)40(1a)2 node edge cluster if experiencing Intersightarbitratorconnectivity issues should NOT allowHXMaintenancemode to initiate as this will bring the cluster down

CSCvr31573

40(2a)35(2c)Old files in varsupportZKTxnlog are not purged withthe daily zklog-cleanup cron job

CSCvr89066

40(2a)35(2a)When a node is in MaintenanceMode any disk removalor replacement will be reflected in UI only after the nodeis brought back frommaintenancemode This is becausestorfs is not running on the node in maintenance andwill not be able to detect disk activities until it is broughtout of Maintenance Mode

CSCvo86431

40(2a)35(2c)Cluster might go offline and stop serving IO requestsrootcvm~ stcli cluster storage-summary--detailGet cluster storage summary failedjavanetConnectException Connection refused19216814210010207

CSCvr01645

40(2a)35(2d)A stretch cluster deployment fails with a messageFormatting nodes Node in Use on the Hyperflex installerand crmZoneType value is seen as 1 in storfscfg oroptspringpathconfigstretchtunes

CSCvq80340

Release Notes for Cisco HX Data Platform Release 4037

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)30(1c)During upgrade we upload the upgrade packagedownloaded from Ciscocom Upgrade packages are offile type tgz hxconnect acceptsstorfs-packages-401a-33028tar file a different type ofcompression and allows upgrade to start and fail whenit cannot decompress the file for bootstrap and scvmclientupgrade on ESXi

CSCvq63888

40(2a)40(2a)While attempting to install hyperflex via intersight thefollowing error is seen

Failed in Task Add hosts to vCenter Cluster with ErrorTry adding hosts manually to vCenter and retry Failedto get addressFailed to host by namelookupHostnamecompanycom on 000053 read udp12700145430-gt12700153 read connection refused

CSCvr52098

40(2a)35(2a)CoreAPI call may time out for cluster management APIwhile doing cluster create validation

CSCvo92952

40(2a)35(2d)Validation fails with Mixed mode expansion checkdue to Enhanced vMotion Compatibility Incompatibilitywhen the ESXi version is different

CSCvq95460

40(2a)30(1i)This failure is seen in installer when custom workflowis used hypervisor configuration + deploy (clean diskpartition=no) at Installing software packages on storagecontroller VM

CSCvr66309

40(2a)35(2d)The HX Installer apparently replaced a comma bysemicolon on the VLAN ID range while exporting theconfiguration to a JSON file

CSCvq93831

40(2a)40(1a)When using unsupported browsers HX Connect userswill not be able to login These users see an errorindicating unauthorized user

CSCvp22693

Release Notes for Cisco HX Data Platform Release 4038

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)40(1a)A vulnerability in the web-based management interfaceof Cisco HyperFlex Software could allow anunauthenticated remote attacker to conduct a cross-siterequest forgery (CSRF) attack on an affected system

The vulnerability is due to insufficient CSRF protectionsfor the web UI on an affected device An attacker couldexploit this vulnerability by persuading a user of theinterface to follow a malicious link A successful exploitcould allow the attacker to perform arbitrary actions withthe privilege level of the affected user

Cisco has released software updates that address thisvulnerability There are no workarounds that address thisvulnerability

This advisory is available at the following link

httpstoolsciscocomsecuritycentercontentCiscoSecurityAdvisorycisco-sa-20190807-hypflex-csrf

CSCvp24343

40(2a)35(2d)Add message to HX Connect Progress flow to notAcknowledge Pending Activities and reboot the serverson UCSM

HX Connect is doing a controlled rolling server upgradein the background

CSCvq22844

40(2a)40(1b)Compute Node expand operation fails due to incorrectcluster name

CSCvr43786

40(2a)26(1e)RF-2 cluster shutdown during data resynchronizationafter a node failure (non storfs) followed by multipledisk read failures and a HardBlacklist of a disk

CSCvp41241

40(2a)40(1a)HyperFlex Edge nodes do not properly set the ESXihostname during deployment

CSCvo13143

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4039

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4040

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1b) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401b-33133p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4041

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4042

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1b)40(1a)While running the HyperFlex installer with WindowsServer Hyper-V cluster creation process fails with theerror Failure occurred during ClusterCreation process Unable to post thecontent to the downstream This symptomis encountered while deploying HyperFlex cluster withHyperFlex nodes configuredordered with Cisco VIC1457 MLOM (PID HX-MLOM-C25Q-04) andWindows Server Datacenter or Core with Hyper-Vversion 2016 or 2019

CSCvp64140

Release Notes for Cisco HX Data Platform Release 4043

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)35(2b)Adding Micron 5200 drive to a cluster fails to increasecluster capacity

CSCvo69067

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4044

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4045

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1a) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401a-33028p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4046

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4047

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1a)30(1d)Cluster instability when disks of different sector sizeplaced in HX node

CSCvk17250

40(1a)

35(2c)

35(1a)When logged in using a local HX user account insteadof a Virtual Center account an error message appearsintermittently indicating Virtual Centerunreachable or Resource informationcannot be updated when VC is reachable

CSCvo36198

40(1a)

35(2b)

30(1d)HXDP does not work with EMC RecoverPoint - needsto support VMware API (FSS-Readdir)

CSCvk38003

Release Notes for Cisco HX Data Platform Release 4048

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)

35(2a)

30(1i)

25(1c)

25(1d)

Zookeeper (Exhibitor) process on HyperFlex StorageController VMs may fail to respond to requests whenvarzookeeper has low or no free space

CSCvm90352

40(1a)26(1c)Use Asynchronous consolidation for HX SnapshotsCSCvn02151

40(1a)35(2b)HX Quiesced Snapshot for Backup VendorsCSCvo90713

40(1a)35(2a)The cluster creationcluster expansion workflow stopswith the following error message at the validation step

FIRMWARE-Check UCSC-SAS-M5HD

FIRMWARE-Check UCSC-SAS-M5HD Required00000029000000320000003500000050 Found00000058

Action Needed Update the Controller Firmware toRequired Version

CSCvn17787

35(2b)30(1d)Kernel migration fails when a local datastore is searchedfor in the ESXi host

CSCvn73127

35(2a)26(1b)A node shuts down when two disks are replaced (acaching disk and another capacity disk) where thecapacity disk is inserted first and then the caching diskis inserted

CSCvk46364

Hyper-V

40(1a)35(2a)Cluster expansion may fail with an error code 500 -operation timed out

CSCvn28721

35(2b)35(2a)During upgrade remove the VLAN on team created foruser vSwitch During fresh installation only one VLANtag is set to the vSwitch and team although multipleVLANs were entered

CSCvn54300

35(2b)35(2a)While upgrading a Hyper-V cluster on account of a rarerace condition between the stUpgradeService andZookeeper servers the upgrade orchestration throws anupgrade validation error and the upgrade process isaborted

CSCvn60486

Release Notes for Cisco HX Data Platform Release 4049

Resolved Caveats in Release 40(1a)

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Startrestart EAM service invCenter

When EAM service is not runningvCenter registration is failing

CSCvt55712

35(2a)Contact Cisco TAC if you believeyou are seeing similar conditions

HyperFlex Stretch Cluster saw poorapplication performance due to writelatency cluster remained unhealthy andrebalance was stuck

This defect is currently being used forinvestigative purposes

CSCvt45344

35(2g)Remove log files fromvarlogspringpath folder when thelog files are not getting rotatedStop the relevant service removelarge log files fromvarlogspringpath folder andrestart the service

varstv folder in Controller VM maybecome full

CSCvt36374

40(2b)Restarting the stMgr will initializeit again

In a Cluster configured with DR networkStMgr may not get initialized and stuckin a deadlock while enabling the IPTablerules (can be verified from the stMgrlog)

CSCvt89709

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Click recover for the selectedVM again under the standalonemode This will move the VM torecovered state

When the user recovers a VM protectedin a group it along with other VMs in thegroup move to recovering state instandalonemode The selectedVM shouldmove to recovered state

CSCvs75553

35(2e)Clean up the fail over cluster andthen recreate the fail over clusterNo need to touch the HX storagecluster

Hyper-V When replicated DC was usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2d)40(1a)

Keep cluster utilization as low aspossible To avoid read-only statekeep below 92

HyperFlex clusters storage utilization isabove 92 usable capacity

CSCvr20922

Release Notes for Cisco HX Data Platform Release 4050

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1b)NAAll the disks in the node got locked afterrebooting the node

We successfully unlocked usingsed-clientsh -U command but wantedto test with another reboot and driveslocked again

CSCvs74286

35(2g)To recover the VM copy over thedata disks and attach them to thenew VM

After enable encryption HX connectencryption status showing Caution butall of the disks enable encryptionsuccessful

-Self Encrypting Drives Service isrunning on all of the nodes

-There is no error message of encryption

- USB0 interface is up

-All SED disks showing supported 1enabled 1 locked 0

CSCvs93245

40(1b)Enable kernelpanic_on_oops as apersistent configuration This willcause the node to panic and rebootimmediately

A cluster node running HX release40(1b) may hang in the Linux kernelThis is classified as an oops and adeviation from the expected behavior

CSCvs54285

Release Notes for Cisco HX Data Platform Release 4051

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Enabling K8iscsi stack in Hx clusterCSCvs41324

Release Notes for Cisco HX Data Platform Release 4052

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Steps to enable K8 on HX cluster

Perform following steps on ALLthe controller VMs

1 Updateetcinitscvmclientconf toenable tune iscsiEnable totrue

sed -iesiscsiEnable=falseiscsiEnable=trueetcinitscvmclientconf

2 Run following initctlcommand to reloadconfiguration

initctl reload-configuration

3 Restart scvmclient process

stop scvmclient startscvmclient

4 Run following initctlcommand to mount SYSTEMDS

initctl emit --no-waitsystem-datastore-created

5 Verify that iscsiEnable tuneis set to true using thefollowing command

ps -eaf | grep scvmclient |grep -v grep

root 6241 1 1 Dec06 014007optspringpathstorfs-corescvmclient-T logEnabled=false -TlogSyslogEnabled=true -TlogEchoToScreen=false -TstatLoggingToFile=false -TstatLoggingToSyslog=true-TlogDir=varlogspringpath-TnfsBackendServerList=1010748100-T iscsiEnable=true -TiscsiUseAsync=true -T

Release Notes for Cisco HX Data Platform Release 4053

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

iscsiConfigFilePath=nfsSYSTEMistgtconfps -eaf | grep scvmclient |grep -v grep

35(2e)Following procedure can be usedto check NFS Queue Depth andincrease if needed

rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth256 lt- Low value[rootHXESXI1] vsish -e setvmkModulesnfsclientmnt[DATASTORE]propertiesmaxQDepth 1024[rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth1024 lt- Optimalvalue

This requires an ESXi host rebootto take effect

Please place the node in HyperflexMaintenanceMode and gracefullyreboot the node for the changes tobe applied

HyperFlex Datastore NFS Queue Depthshows as 256 which can lead toperformance (including latency) issues

CSCvr83056

40(1b)This is cosmetic The error will goaway once the cluster returns tohealthy status

When a single node is offline in clusterstcli cluster storage-summary shows twonodes unavailable rootSCVM~ stclicluster storage-summary

CSCvs47735

35(2g)Modify both the root and adminpassword for the SCVMs to be thesame

stcli security password set -uadmin

Cluster expansion fails at Config Installerstage when the root and admin passwordfor the storage controller (SCVM) aredifferent

CSCvs69317

35(2d)NAAfter successfully changing the DNSserver on the HX controller we still cansee the original DNS entry that was addedduring the deployment

CSCvs69154

Release Notes for Cisco HX Data Platform Release 4054

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Customers are strongly encouragedto work with TAC in order toidentify and remediate this issue

1 Change getcluster from 2m to8m inoptspringpathstorfs-mgmtstMgr-10confapplicationconfon all nodes

2 Restart stMgr on all nodes

You may see this error message after afailed upgrade or other task such asattempting to enter a node into HXmaintenance mode

CSCvs53555

40(2a)Run the following command oneach storage controller VM as root

grep -i monitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp sed -iesmonitor_dns_serverstruemonitor_dns_serversfalseoptspringpathhx-diag-toolswatchdog_configjsonampamp grep -imonitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp restart watchdog

Receive the following alertevent in HXConnect HX Controller VMHOSTNAME one or more configuredDNS servers not responding

CSCvt13947

35(2b)Delete any empty (Size 0) log filesunder varlogzookeeperversion-2ORvarlogzookeeperstandaloneversion-2

Zookeeper fails to start while Exhibitor isrunning however echo srvr returnsnothing

CSCvs21562

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

40(4e)1 De-commission the server

2 Power drain the server -REMOVE BOTH POWERCORDS ON THE BACK OFTHE SERVER FOR 10SECONDS THENREINSERTPOWERCORDS

3 Re-commission the server

M2 boot disk is missing from serverinventory after upgrade to server firmware40(4e) As a result server fails to boot toOS installed in the M2 disk The issuepersists after re-acknowledgement as wellas de-commission andre-acknowledgement of the server

CSCvs02466

Release Notes for Cisco HX Data Platform Release 4055

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Re-start the flusher manuallyMany disks taken offline by a fault ormanually resulting in ENOSPC oncluster

CSCvq32721

35(2e)Clean up fail over cluster and thenrecreate the fail over cluster Thereis no need to touch the HX storagecluster

[Hyper-V]When replicated DCwas usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2b)Manually install upgradescvmclient VIB andmake sure thatit matches with HXDP version

Upgrade from release HX 35(2a) to35(2x) 4X does not upgrade scvmclientRecovery point may not work

CSCvq54992

Management

30(1b)To recover the VM copy over thedata disks and attach them to thenew VM

[HyperV]VMshows up onmultiple nodesCSCvj22992

35(1a)1 Do not change MTU at ESXlevel on running cluster

2 Reset back to the original value

ChangingMTU from 9000 to 1500 on oneESX node causes storfs process to restarton all nodes

CSCvm99150

Release Notes for Cisco HX Data Platform Release 4056

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)After the installation (for newdeployments) or after an upgradeto 401a (for existingdeployments) perform thefollowing steps on all ControllerVMs

1 Edit the following tune files (onall Controller VMs)

optspringpathconfiglfftunes

optspringpathconfigvsi_16tbtunes

2 SetcleanerEnableSegSummaryCleaning

to false

3 After editing the above tunefiles

a SSH in to all the ControllerVMs

b Run storfstool -- -Z

c Run the below command andcheck the values Tune valueshould be true

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=true

d Type below commands andapplied the tunes changesdynamically

echo false gttmpstprocfssystemtunecleanerEnableSegSummaryCleaning

e Verify that tune values arechanged Run below commandValue should be false

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=false

f umount tmpstprocfs

A cluster with 8TB or 12TB disk drivesmay experience IO stalls for severalminutes after another node fails in thecluster

CSCvp23718

Release Notes for Cisco HX Data Platform Release 4057

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)1 Update the conf file manuallyto change to the link local addressto

2 Restart scvmclient on thecontroller and ESX

HX 35(2b) FlexVol to HX 40 CSIupgrade does not work FlexVol continuesto work

CSCvp26319

35(2a)Disable rebalance until node healsthen reenable rebalance

7-node cluster healing very slowlybecause rebalance change AMS reqs areblocking healing process

CSCvp62512

35(2a)NAWith moderate cluster storage capacityusage under site failover cluster clustermay experience an all paths (APD) downcondition

CSCvq49412

40(1a)NAThis product includes Third-partySoftware that is affected by thevulnerabilities identified by the followingCommon Vulnerability and Exposures(CVE) IDs

CVE-2016-0762 CVE-2016-6797CVE-2016-6816 CVE-2016-8735CVE-2017-5647 CVE-2017-12615CVE-2017-12616 CVE-2017-12617CVE-2017-7674 CVE-2018-1304CVE-2018-8014 CVE-2018-1336CVE-2018-8034 CVE-2018-11784CVE-2019-0232

CSCvp36364

35(2d)NAstcli cluster info command needs toprovide UCSM VIP address Currently itshows ucsm-hostcom

CSCvq11456

35(2a)No workarounds currently Thestale entries do not affect the dayto day operations of HyperFlex

HyperFlex upgrade validation failedbecause cluster has extraneous stNodes ininternal database

CSCvq32530

35(2a)NAWhen not in maintenancemode all drivesappear in HX Connect and get updated asexpected

When in maintenance mode extra drivesare shown in HX Connect with no slotnumbers

CSCvq42378

35(2b)Contact TAC to cleanup duplicatenode

Nodes duplicated in HX Connect aftercluster recreate

CSCvq61825

Release Notes for Cisco HX Data Platform Release 4058

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)NACurrently HyperFlex installs on Hyper-Vdo not contain the stcli security whitelistcommandset

CSCvq66245

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

Release Notes for Cisco HX Data Platform Release 4059

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)[Hyper-V] Converged node expansionfails for incoming node with VIC 14xx

CSCvq39523

Release Notes for Cisco HX Data Platform Release 4060

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

1 Set Jumbo frame inWindowsHypervisor for incomingnode

Get-NetAdapter storage-data-a|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

Get-NetAdapter storage-data-b|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

2 Verify all interfaces are setcorrectly to support Jumbo frames

Get-NetAdapter storage |Get-NetAdapterAdvancedProperty| RegistryKeyword -matchjumbo | ft -auto

You should receive the followingmessage

Name DisplayName DisplayValue

RegistryKeyword RegistryValue

---- ----------- ------------

--------------- -------------

vswitch-hx-storage-data Jumbo

Packet 9014 Bytes

JumboPacket 9014

storage-data-b Jumbo Packet

Bytes 9014 JumboPacket

9014

storage-data-a Jumbo Packet

Bytes 9014 JumboPacket

9014

3 Reboot only the incoming nodeor the node being expanded

4 Retry Cluster Expansion in

Release Notes for Cisco HX Data Platform Release 4061

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)35(2b)HyperFlex cluster does not heal after maintenance onnodeserver

CSCvr18528

40(2a)40(2a)High percentage of packet loss on the eth1 data interface9K MTU

CSCvr97089

40(2a)35(2f)The HyperFlex cluster healing state is stuck at 87 andis unable to progress

CSCvr16760

40(2a)40(2a)In some circumstances two replications can occur forthe same VM

From the GUI the replication status for this VM willshow stuck in an In Progress state but there are nojobs stuck in replication layer Subsequent Replicationsshould succeed

CSCvq22898

40(2a)35(2a)After an unexpected power outage the hyperflex clusteris unable to come up correctly

All processes are running time is synced and all nodescan vmkping each other as required

CSCvp13990

40(2a)40(1a)When specifying IP address instead of node ID stclinode maintenanceMode commands fail

CSCvp33657

40(2a)40(1a)2 node edge cluster if experiencing Intersightarbitratorconnectivity issues should NOT allowHXMaintenancemode to initiate as this will bring the cluster down

CSCvr31573

40(2a)35(2c)Old files in varsupportZKTxnlog are not purged withthe daily zklog-cleanup cron job

CSCvr89066

40(2a)35(2a)When a node is in MaintenanceMode any disk removalor replacement will be reflected in UI only after the nodeis brought back frommaintenancemode This is becausestorfs is not running on the node in maintenance andwill not be able to detect disk activities until it is broughtout of Maintenance Mode

CSCvo86431

40(2a)35(2c)Cluster might go offline and stop serving IO requestsrootcvm~ stcli cluster storage-summary--detailGet cluster storage summary failedjavanetConnectException Connection refused19216814210010207

CSCvr01645

40(2a)35(2d)A stretch cluster deployment fails with a messageFormatting nodes Node in Use on the Hyperflex installerand crmZoneType value is seen as 1 in storfscfg oroptspringpathconfigstretchtunes

CSCvq80340

Release Notes for Cisco HX Data Platform Release 4037

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)30(1c)During upgrade we upload the upgrade packagedownloaded from Ciscocom Upgrade packages are offile type tgz hxconnect acceptsstorfs-packages-401a-33028tar file a different type ofcompression and allows upgrade to start and fail whenit cannot decompress the file for bootstrap and scvmclientupgrade on ESXi

CSCvq63888

40(2a)40(2a)While attempting to install hyperflex via intersight thefollowing error is seen

Failed in Task Add hosts to vCenter Cluster with ErrorTry adding hosts manually to vCenter and retry Failedto get addressFailed to host by namelookupHostnamecompanycom on 000053 read udp12700145430-gt12700153 read connection refused

CSCvr52098

40(2a)35(2a)CoreAPI call may time out for cluster management APIwhile doing cluster create validation

CSCvo92952

40(2a)35(2d)Validation fails with Mixed mode expansion checkdue to Enhanced vMotion Compatibility Incompatibilitywhen the ESXi version is different

CSCvq95460

40(2a)30(1i)This failure is seen in installer when custom workflowis used hypervisor configuration + deploy (clean diskpartition=no) at Installing software packages on storagecontroller VM

CSCvr66309

40(2a)35(2d)The HX Installer apparently replaced a comma bysemicolon on the VLAN ID range while exporting theconfiguration to a JSON file

CSCvq93831

40(2a)40(1a)When using unsupported browsers HX Connect userswill not be able to login These users see an errorindicating unauthorized user

CSCvp22693

Release Notes for Cisco HX Data Platform Release 4038

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)40(1a)A vulnerability in the web-based management interfaceof Cisco HyperFlex Software could allow anunauthenticated remote attacker to conduct a cross-siterequest forgery (CSRF) attack on an affected system

The vulnerability is due to insufficient CSRF protectionsfor the web UI on an affected device An attacker couldexploit this vulnerability by persuading a user of theinterface to follow a malicious link A successful exploitcould allow the attacker to perform arbitrary actions withthe privilege level of the affected user

Cisco has released software updates that address thisvulnerability There are no workarounds that address thisvulnerability

This advisory is available at the following link

httpstoolsciscocomsecuritycentercontentCiscoSecurityAdvisorycisco-sa-20190807-hypflex-csrf

CSCvp24343

40(2a)35(2d)Add message to HX Connect Progress flow to notAcknowledge Pending Activities and reboot the serverson UCSM

HX Connect is doing a controlled rolling server upgradein the background

CSCvq22844

40(2a)40(1b)Compute Node expand operation fails due to incorrectcluster name

CSCvr43786

40(2a)26(1e)RF-2 cluster shutdown during data resynchronizationafter a node failure (non storfs) followed by multipledisk read failures and a HardBlacklist of a disk

CSCvp41241

40(2a)40(1a)HyperFlex Edge nodes do not properly set the ESXihostname during deployment

CSCvo13143

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4039

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4040

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1b) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401b-33133p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4041

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4042

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1b)40(1a)While running the HyperFlex installer with WindowsServer Hyper-V cluster creation process fails with theerror Failure occurred during ClusterCreation process Unable to post thecontent to the downstream This symptomis encountered while deploying HyperFlex cluster withHyperFlex nodes configuredordered with Cisco VIC1457 MLOM (PID HX-MLOM-C25Q-04) andWindows Server Datacenter or Core with Hyper-Vversion 2016 or 2019

CSCvp64140

Release Notes for Cisco HX Data Platform Release 4043

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)35(2b)Adding Micron 5200 drive to a cluster fails to increasecluster capacity

CSCvo69067

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4044

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4045

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1a) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401a-33028p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4046

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4047

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1a)30(1d)Cluster instability when disks of different sector sizeplaced in HX node

CSCvk17250

40(1a)

35(2c)

35(1a)When logged in using a local HX user account insteadof a Virtual Center account an error message appearsintermittently indicating Virtual Centerunreachable or Resource informationcannot be updated when VC is reachable

CSCvo36198

40(1a)

35(2b)

30(1d)HXDP does not work with EMC RecoverPoint - needsto support VMware API (FSS-Readdir)

CSCvk38003

Release Notes for Cisco HX Data Platform Release 4048

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)

35(2a)

30(1i)

25(1c)

25(1d)

Zookeeper (Exhibitor) process on HyperFlex StorageController VMs may fail to respond to requests whenvarzookeeper has low or no free space

CSCvm90352

40(1a)26(1c)Use Asynchronous consolidation for HX SnapshotsCSCvn02151

40(1a)35(2b)HX Quiesced Snapshot for Backup VendorsCSCvo90713

40(1a)35(2a)The cluster creationcluster expansion workflow stopswith the following error message at the validation step

FIRMWARE-Check UCSC-SAS-M5HD

FIRMWARE-Check UCSC-SAS-M5HD Required00000029000000320000003500000050 Found00000058

Action Needed Update the Controller Firmware toRequired Version

CSCvn17787

35(2b)30(1d)Kernel migration fails when a local datastore is searchedfor in the ESXi host

CSCvn73127

35(2a)26(1b)A node shuts down when two disks are replaced (acaching disk and another capacity disk) where thecapacity disk is inserted first and then the caching diskis inserted

CSCvk46364

Hyper-V

40(1a)35(2a)Cluster expansion may fail with an error code 500 -operation timed out

CSCvn28721

35(2b)35(2a)During upgrade remove the VLAN on team created foruser vSwitch During fresh installation only one VLANtag is set to the vSwitch and team although multipleVLANs were entered

CSCvn54300

35(2b)35(2a)While upgrading a Hyper-V cluster on account of a rarerace condition between the stUpgradeService andZookeeper servers the upgrade orchestration throws anupgrade validation error and the upgrade process isaborted

CSCvn60486

Release Notes for Cisco HX Data Platform Release 4049

Resolved Caveats in Release 40(1a)

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Startrestart EAM service invCenter

When EAM service is not runningvCenter registration is failing

CSCvt55712

35(2a)Contact Cisco TAC if you believeyou are seeing similar conditions

HyperFlex Stretch Cluster saw poorapplication performance due to writelatency cluster remained unhealthy andrebalance was stuck

This defect is currently being used forinvestigative purposes

CSCvt45344

35(2g)Remove log files fromvarlogspringpath folder when thelog files are not getting rotatedStop the relevant service removelarge log files fromvarlogspringpath folder andrestart the service

varstv folder in Controller VM maybecome full

CSCvt36374

40(2b)Restarting the stMgr will initializeit again

In a Cluster configured with DR networkStMgr may not get initialized and stuckin a deadlock while enabling the IPTablerules (can be verified from the stMgrlog)

CSCvt89709

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Click recover for the selectedVM again under the standalonemode This will move the VM torecovered state

When the user recovers a VM protectedin a group it along with other VMs in thegroup move to recovering state instandalonemode The selectedVM shouldmove to recovered state

CSCvs75553

35(2e)Clean up the fail over cluster andthen recreate the fail over clusterNo need to touch the HX storagecluster

Hyper-V When replicated DC was usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2d)40(1a)

Keep cluster utilization as low aspossible To avoid read-only statekeep below 92

HyperFlex clusters storage utilization isabove 92 usable capacity

CSCvr20922

Release Notes for Cisco HX Data Platform Release 4050

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1b)NAAll the disks in the node got locked afterrebooting the node

We successfully unlocked usingsed-clientsh -U command but wantedto test with another reboot and driveslocked again

CSCvs74286

35(2g)To recover the VM copy over thedata disks and attach them to thenew VM

After enable encryption HX connectencryption status showing Caution butall of the disks enable encryptionsuccessful

-Self Encrypting Drives Service isrunning on all of the nodes

-There is no error message of encryption

- USB0 interface is up

-All SED disks showing supported 1enabled 1 locked 0

CSCvs93245

40(1b)Enable kernelpanic_on_oops as apersistent configuration This willcause the node to panic and rebootimmediately

A cluster node running HX release40(1b) may hang in the Linux kernelThis is classified as an oops and adeviation from the expected behavior

CSCvs54285

Release Notes for Cisco HX Data Platform Release 4051

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Enabling K8iscsi stack in Hx clusterCSCvs41324

Release Notes for Cisco HX Data Platform Release 4052

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Steps to enable K8 on HX cluster

Perform following steps on ALLthe controller VMs

1 Updateetcinitscvmclientconf toenable tune iscsiEnable totrue

sed -iesiscsiEnable=falseiscsiEnable=trueetcinitscvmclientconf

2 Run following initctlcommand to reloadconfiguration

initctl reload-configuration

3 Restart scvmclient process

stop scvmclient startscvmclient

4 Run following initctlcommand to mount SYSTEMDS

initctl emit --no-waitsystem-datastore-created

5 Verify that iscsiEnable tuneis set to true using thefollowing command

ps -eaf | grep scvmclient |grep -v grep

root 6241 1 1 Dec06 014007optspringpathstorfs-corescvmclient-T logEnabled=false -TlogSyslogEnabled=true -TlogEchoToScreen=false -TstatLoggingToFile=false -TstatLoggingToSyslog=true-TlogDir=varlogspringpath-TnfsBackendServerList=1010748100-T iscsiEnable=true -TiscsiUseAsync=true -T

Release Notes for Cisco HX Data Platform Release 4053

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

iscsiConfigFilePath=nfsSYSTEMistgtconfps -eaf | grep scvmclient |grep -v grep

35(2e)Following procedure can be usedto check NFS Queue Depth andincrease if needed

rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth256 lt- Low value[rootHXESXI1] vsish -e setvmkModulesnfsclientmnt[DATASTORE]propertiesmaxQDepth 1024[rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth1024 lt- Optimalvalue

This requires an ESXi host rebootto take effect

Please place the node in HyperflexMaintenanceMode and gracefullyreboot the node for the changes tobe applied

HyperFlex Datastore NFS Queue Depthshows as 256 which can lead toperformance (including latency) issues

CSCvr83056

40(1b)This is cosmetic The error will goaway once the cluster returns tohealthy status

When a single node is offline in clusterstcli cluster storage-summary shows twonodes unavailable rootSCVM~ stclicluster storage-summary

CSCvs47735

35(2g)Modify both the root and adminpassword for the SCVMs to be thesame

stcli security password set -uadmin

Cluster expansion fails at Config Installerstage when the root and admin passwordfor the storage controller (SCVM) aredifferent

CSCvs69317

35(2d)NAAfter successfully changing the DNSserver on the HX controller we still cansee the original DNS entry that was addedduring the deployment

CSCvs69154

Release Notes for Cisco HX Data Platform Release 4054

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Customers are strongly encouragedto work with TAC in order toidentify and remediate this issue

1 Change getcluster from 2m to8m inoptspringpathstorfs-mgmtstMgr-10confapplicationconfon all nodes

2 Restart stMgr on all nodes

You may see this error message after afailed upgrade or other task such asattempting to enter a node into HXmaintenance mode

CSCvs53555

40(2a)Run the following command oneach storage controller VM as root

grep -i monitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp sed -iesmonitor_dns_serverstruemonitor_dns_serversfalseoptspringpathhx-diag-toolswatchdog_configjsonampamp grep -imonitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp restart watchdog

Receive the following alertevent in HXConnect HX Controller VMHOSTNAME one or more configuredDNS servers not responding

CSCvt13947

35(2b)Delete any empty (Size 0) log filesunder varlogzookeeperversion-2ORvarlogzookeeperstandaloneversion-2

Zookeeper fails to start while Exhibitor isrunning however echo srvr returnsnothing

CSCvs21562

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

40(4e)1 De-commission the server

2 Power drain the server -REMOVE BOTH POWERCORDS ON THE BACK OFTHE SERVER FOR 10SECONDS THENREINSERTPOWERCORDS

3 Re-commission the server

M2 boot disk is missing from serverinventory after upgrade to server firmware40(4e) As a result server fails to boot toOS installed in the M2 disk The issuepersists after re-acknowledgement as wellas de-commission andre-acknowledgement of the server

CSCvs02466

Release Notes for Cisco HX Data Platform Release 4055

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Re-start the flusher manuallyMany disks taken offline by a fault ormanually resulting in ENOSPC oncluster

CSCvq32721

35(2e)Clean up fail over cluster and thenrecreate the fail over cluster Thereis no need to touch the HX storagecluster

[Hyper-V]When replicated DCwas usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2b)Manually install upgradescvmclient VIB andmake sure thatit matches with HXDP version

Upgrade from release HX 35(2a) to35(2x) 4X does not upgrade scvmclientRecovery point may not work

CSCvq54992

Management

30(1b)To recover the VM copy over thedata disks and attach them to thenew VM

[HyperV]VMshows up onmultiple nodesCSCvj22992

35(1a)1 Do not change MTU at ESXlevel on running cluster

2 Reset back to the original value

ChangingMTU from 9000 to 1500 on oneESX node causes storfs process to restarton all nodes

CSCvm99150

Release Notes for Cisco HX Data Platform Release 4056

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)After the installation (for newdeployments) or after an upgradeto 401a (for existingdeployments) perform thefollowing steps on all ControllerVMs

1 Edit the following tune files (onall Controller VMs)

optspringpathconfiglfftunes

optspringpathconfigvsi_16tbtunes

2 SetcleanerEnableSegSummaryCleaning

to false

3 After editing the above tunefiles

a SSH in to all the ControllerVMs

b Run storfstool -- -Z

c Run the below command andcheck the values Tune valueshould be true

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=true

d Type below commands andapplied the tunes changesdynamically

echo false gttmpstprocfssystemtunecleanerEnableSegSummaryCleaning

e Verify that tune values arechanged Run below commandValue should be false

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=false

f umount tmpstprocfs

A cluster with 8TB or 12TB disk drivesmay experience IO stalls for severalminutes after another node fails in thecluster

CSCvp23718

Release Notes for Cisco HX Data Platform Release 4057

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)1 Update the conf file manuallyto change to the link local addressto

2 Restart scvmclient on thecontroller and ESX

HX 35(2b) FlexVol to HX 40 CSIupgrade does not work FlexVol continuesto work

CSCvp26319

35(2a)Disable rebalance until node healsthen reenable rebalance

7-node cluster healing very slowlybecause rebalance change AMS reqs areblocking healing process

CSCvp62512

35(2a)NAWith moderate cluster storage capacityusage under site failover cluster clustermay experience an all paths (APD) downcondition

CSCvq49412

40(1a)NAThis product includes Third-partySoftware that is affected by thevulnerabilities identified by the followingCommon Vulnerability and Exposures(CVE) IDs

CVE-2016-0762 CVE-2016-6797CVE-2016-6816 CVE-2016-8735CVE-2017-5647 CVE-2017-12615CVE-2017-12616 CVE-2017-12617CVE-2017-7674 CVE-2018-1304CVE-2018-8014 CVE-2018-1336CVE-2018-8034 CVE-2018-11784CVE-2019-0232

CSCvp36364

35(2d)NAstcli cluster info command needs toprovide UCSM VIP address Currently itshows ucsm-hostcom

CSCvq11456

35(2a)No workarounds currently Thestale entries do not affect the dayto day operations of HyperFlex

HyperFlex upgrade validation failedbecause cluster has extraneous stNodes ininternal database

CSCvq32530

35(2a)NAWhen not in maintenancemode all drivesappear in HX Connect and get updated asexpected

When in maintenance mode extra drivesare shown in HX Connect with no slotnumbers

CSCvq42378

35(2b)Contact TAC to cleanup duplicatenode

Nodes duplicated in HX Connect aftercluster recreate

CSCvq61825

Release Notes for Cisco HX Data Platform Release 4058

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)NACurrently HyperFlex installs on Hyper-Vdo not contain the stcli security whitelistcommandset

CSCvq66245

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

Release Notes for Cisco HX Data Platform Release 4059

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)[Hyper-V] Converged node expansionfails for incoming node with VIC 14xx

CSCvq39523

Release Notes for Cisco HX Data Platform Release 4060

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

1 Set Jumbo frame inWindowsHypervisor for incomingnode

Get-NetAdapter storage-data-a|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

Get-NetAdapter storage-data-b|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

2 Verify all interfaces are setcorrectly to support Jumbo frames

Get-NetAdapter storage |Get-NetAdapterAdvancedProperty| RegistryKeyword -matchjumbo | ft -auto

You should receive the followingmessage

Name DisplayName DisplayValue

RegistryKeyword RegistryValue

---- ----------- ------------

--------------- -------------

vswitch-hx-storage-data Jumbo

Packet 9014 Bytes

JumboPacket 9014

storage-data-b Jumbo Packet

Bytes 9014 JumboPacket

9014

storage-data-a Jumbo Packet

Bytes 9014 JumboPacket

9014

3 Reboot only the incoming nodeor the node being expanded

4 Retry Cluster Expansion in

Release Notes for Cisco HX Data Platform Release 4061

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)30(1c)During upgrade we upload the upgrade packagedownloaded from Ciscocom Upgrade packages are offile type tgz hxconnect acceptsstorfs-packages-401a-33028tar file a different type ofcompression and allows upgrade to start and fail whenit cannot decompress the file for bootstrap and scvmclientupgrade on ESXi

CSCvq63888

40(2a)40(2a)While attempting to install hyperflex via intersight thefollowing error is seen

Failed in Task Add hosts to vCenter Cluster with ErrorTry adding hosts manually to vCenter and retry Failedto get addressFailed to host by namelookupHostnamecompanycom on 000053 read udp12700145430-gt12700153 read connection refused

CSCvr52098

40(2a)35(2a)CoreAPI call may time out for cluster management APIwhile doing cluster create validation

CSCvo92952

40(2a)35(2d)Validation fails with Mixed mode expansion checkdue to Enhanced vMotion Compatibility Incompatibilitywhen the ESXi version is different

CSCvq95460

40(2a)30(1i)This failure is seen in installer when custom workflowis used hypervisor configuration + deploy (clean diskpartition=no) at Installing software packages on storagecontroller VM

CSCvr66309

40(2a)35(2d)The HX Installer apparently replaced a comma bysemicolon on the VLAN ID range while exporting theconfiguration to a JSON file

CSCvq93831

40(2a)40(1a)When using unsupported browsers HX Connect userswill not be able to login These users see an errorindicating unauthorized user

CSCvp22693

Release Notes for Cisco HX Data Platform Release 4038

Resolved Caveats in Release 40(2a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)40(1a)A vulnerability in the web-based management interfaceof Cisco HyperFlex Software could allow anunauthenticated remote attacker to conduct a cross-siterequest forgery (CSRF) attack on an affected system

The vulnerability is due to insufficient CSRF protectionsfor the web UI on an affected device An attacker couldexploit this vulnerability by persuading a user of theinterface to follow a malicious link A successful exploitcould allow the attacker to perform arbitrary actions withthe privilege level of the affected user

Cisco has released software updates that address thisvulnerability There are no workarounds that address thisvulnerability

This advisory is available at the following link

httpstoolsciscocomsecuritycentercontentCiscoSecurityAdvisorycisco-sa-20190807-hypflex-csrf

CSCvp24343

40(2a)35(2d)Add message to HX Connect Progress flow to notAcknowledge Pending Activities and reboot the serverson UCSM

HX Connect is doing a controlled rolling server upgradein the background

CSCvq22844

40(2a)40(1b)Compute Node expand operation fails due to incorrectcluster name

CSCvr43786

40(2a)26(1e)RF-2 cluster shutdown during data resynchronizationafter a node failure (non storfs) followed by multipledisk read failures and a HardBlacklist of a disk

CSCvp41241

40(2a)40(1a)HyperFlex Edge nodes do not properly set the ESXihostname during deployment

CSCvo13143

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4039

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4040

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1b) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401b-33133p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4041

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4042

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1b)40(1a)While running the HyperFlex installer with WindowsServer Hyper-V cluster creation process fails with theerror Failure occurred during ClusterCreation process Unable to post thecontent to the downstream This symptomis encountered while deploying HyperFlex cluster withHyperFlex nodes configuredordered with Cisco VIC1457 MLOM (PID HX-MLOM-C25Q-04) andWindows Server Datacenter or Core with Hyper-Vversion 2016 or 2019

CSCvp64140

Release Notes for Cisco HX Data Platform Release 4043

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)35(2b)Adding Micron 5200 drive to a cluster fails to increasecluster capacity

CSCvo69067

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4044

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4045

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1a) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401a-33028p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4046

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4047

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1a)30(1d)Cluster instability when disks of different sector sizeplaced in HX node

CSCvk17250

40(1a)

35(2c)

35(1a)When logged in using a local HX user account insteadof a Virtual Center account an error message appearsintermittently indicating Virtual Centerunreachable or Resource informationcannot be updated when VC is reachable

CSCvo36198

40(1a)

35(2b)

30(1d)HXDP does not work with EMC RecoverPoint - needsto support VMware API (FSS-Readdir)

CSCvk38003

Release Notes for Cisco HX Data Platform Release 4048

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)

35(2a)

30(1i)

25(1c)

25(1d)

Zookeeper (Exhibitor) process on HyperFlex StorageController VMs may fail to respond to requests whenvarzookeeper has low or no free space

CSCvm90352

40(1a)26(1c)Use Asynchronous consolidation for HX SnapshotsCSCvn02151

40(1a)35(2b)HX Quiesced Snapshot for Backup VendorsCSCvo90713

40(1a)35(2a)The cluster creationcluster expansion workflow stopswith the following error message at the validation step

FIRMWARE-Check UCSC-SAS-M5HD

FIRMWARE-Check UCSC-SAS-M5HD Required00000029000000320000003500000050 Found00000058

Action Needed Update the Controller Firmware toRequired Version

CSCvn17787

35(2b)30(1d)Kernel migration fails when a local datastore is searchedfor in the ESXi host

CSCvn73127

35(2a)26(1b)A node shuts down when two disks are replaced (acaching disk and another capacity disk) where thecapacity disk is inserted first and then the caching diskis inserted

CSCvk46364

Hyper-V

40(1a)35(2a)Cluster expansion may fail with an error code 500 -operation timed out

CSCvn28721

35(2b)35(2a)During upgrade remove the VLAN on team created foruser vSwitch During fresh installation only one VLANtag is set to the vSwitch and team although multipleVLANs were entered

CSCvn54300

35(2b)35(2a)While upgrading a Hyper-V cluster on account of a rarerace condition between the stUpgradeService andZookeeper servers the upgrade orchestration throws anupgrade validation error and the upgrade process isaborted

CSCvn60486

Release Notes for Cisco HX Data Platform Release 4049

Resolved Caveats in Release 40(1a)

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Startrestart EAM service invCenter

When EAM service is not runningvCenter registration is failing

CSCvt55712

35(2a)Contact Cisco TAC if you believeyou are seeing similar conditions

HyperFlex Stretch Cluster saw poorapplication performance due to writelatency cluster remained unhealthy andrebalance was stuck

This defect is currently being used forinvestigative purposes

CSCvt45344

35(2g)Remove log files fromvarlogspringpath folder when thelog files are not getting rotatedStop the relevant service removelarge log files fromvarlogspringpath folder andrestart the service

varstv folder in Controller VM maybecome full

CSCvt36374

40(2b)Restarting the stMgr will initializeit again

In a Cluster configured with DR networkStMgr may not get initialized and stuckin a deadlock while enabling the IPTablerules (can be verified from the stMgrlog)

CSCvt89709

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Click recover for the selectedVM again under the standalonemode This will move the VM torecovered state

When the user recovers a VM protectedin a group it along with other VMs in thegroup move to recovering state instandalonemode The selectedVM shouldmove to recovered state

CSCvs75553

35(2e)Clean up the fail over cluster andthen recreate the fail over clusterNo need to touch the HX storagecluster

Hyper-V When replicated DC was usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2d)40(1a)

Keep cluster utilization as low aspossible To avoid read-only statekeep below 92

HyperFlex clusters storage utilization isabove 92 usable capacity

CSCvr20922

Release Notes for Cisco HX Data Platform Release 4050

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1b)NAAll the disks in the node got locked afterrebooting the node

We successfully unlocked usingsed-clientsh -U command but wantedto test with another reboot and driveslocked again

CSCvs74286

35(2g)To recover the VM copy over thedata disks and attach them to thenew VM

After enable encryption HX connectencryption status showing Caution butall of the disks enable encryptionsuccessful

-Self Encrypting Drives Service isrunning on all of the nodes

-There is no error message of encryption

- USB0 interface is up

-All SED disks showing supported 1enabled 1 locked 0

CSCvs93245

40(1b)Enable kernelpanic_on_oops as apersistent configuration This willcause the node to panic and rebootimmediately

A cluster node running HX release40(1b) may hang in the Linux kernelThis is classified as an oops and adeviation from the expected behavior

CSCvs54285

Release Notes for Cisco HX Data Platform Release 4051

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Enabling K8iscsi stack in Hx clusterCSCvs41324

Release Notes for Cisco HX Data Platform Release 4052

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Steps to enable K8 on HX cluster

Perform following steps on ALLthe controller VMs

1 Updateetcinitscvmclientconf toenable tune iscsiEnable totrue

sed -iesiscsiEnable=falseiscsiEnable=trueetcinitscvmclientconf

2 Run following initctlcommand to reloadconfiguration

initctl reload-configuration

3 Restart scvmclient process

stop scvmclient startscvmclient

4 Run following initctlcommand to mount SYSTEMDS

initctl emit --no-waitsystem-datastore-created

5 Verify that iscsiEnable tuneis set to true using thefollowing command

ps -eaf | grep scvmclient |grep -v grep

root 6241 1 1 Dec06 014007optspringpathstorfs-corescvmclient-T logEnabled=false -TlogSyslogEnabled=true -TlogEchoToScreen=false -TstatLoggingToFile=false -TstatLoggingToSyslog=true-TlogDir=varlogspringpath-TnfsBackendServerList=1010748100-T iscsiEnable=true -TiscsiUseAsync=true -T

Release Notes for Cisco HX Data Platform Release 4053

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

iscsiConfigFilePath=nfsSYSTEMistgtconfps -eaf | grep scvmclient |grep -v grep

35(2e)Following procedure can be usedto check NFS Queue Depth andincrease if needed

rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth256 lt- Low value[rootHXESXI1] vsish -e setvmkModulesnfsclientmnt[DATASTORE]propertiesmaxQDepth 1024[rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth1024 lt- Optimalvalue

This requires an ESXi host rebootto take effect

Please place the node in HyperflexMaintenanceMode and gracefullyreboot the node for the changes tobe applied

HyperFlex Datastore NFS Queue Depthshows as 256 which can lead toperformance (including latency) issues

CSCvr83056

40(1b)This is cosmetic The error will goaway once the cluster returns tohealthy status

When a single node is offline in clusterstcli cluster storage-summary shows twonodes unavailable rootSCVM~ stclicluster storage-summary

CSCvs47735

35(2g)Modify both the root and adminpassword for the SCVMs to be thesame

stcli security password set -uadmin

Cluster expansion fails at Config Installerstage when the root and admin passwordfor the storage controller (SCVM) aredifferent

CSCvs69317

35(2d)NAAfter successfully changing the DNSserver on the HX controller we still cansee the original DNS entry that was addedduring the deployment

CSCvs69154

Release Notes for Cisco HX Data Platform Release 4054

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Customers are strongly encouragedto work with TAC in order toidentify and remediate this issue

1 Change getcluster from 2m to8m inoptspringpathstorfs-mgmtstMgr-10confapplicationconfon all nodes

2 Restart stMgr on all nodes

You may see this error message after afailed upgrade or other task such asattempting to enter a node into HXmaintenance mode

CSCvs53555

40(2a)Run the following command oneach storage controller VM as root

grep -i monitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp sed -iesmonitor_dns_serverstruemonitor_dns_serversfalseoptspringpathhx-diag-toolswatchdog_configjsonampamp grep -imonitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp restart watchdog

Receive the following alertevent in HXConnect HX Controller VMHOSTNAME one or more configuredDNS servers not responding

CSCvt13947

35(2b)Delete any empty (Size 0) log filesunder varlogzookeeperversion-2ORvarlogzookeeperstandaloneversion-2

Zookeeper fails to start while Exhibitor isrunning however echo srvr returnsnothing

CSCvs21562

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

40(4e)1 De-commission the server

2 Power drain the server -REMOVE BOTH POWERCORDS ON THE BACK OFTHE SERVER FOR 10SECONDS THENREINSERTPOWERCORDS

3 Re-commission the server

M2 boot disk is missing from serverinventory after upgrade to server firmware40(4e) As a result server fails to boot toOS installed in the M2 disk The issuepersists after re-acknowledgement as wellas de-commission andre-acknowledgement of the server

CSCvs02466

Release Notes for Cisco HX Data Platform Release 4055

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Re-start the flusher manuallyMany disks taken offline by a fault ormanually resulting in ENOSPC oncluster

CSCvq32721

35(2e)Clean up fail over cluster and thenrecreate the fail over cluster Thereis no need to touch the HX storagecluster

[Hyper-V]When replicated DCwas usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2b)Manually install upgradescvmclient VIB andmake sure thatit matches with HXDP version

Upgrade from release HX 35(2a) to35(2x) 4X does not upgrade scvmclientRecovery point may not work

CSCvq54992

Management

30(1b)To recover the VM copy over thedata disks and attach them to thenew VM

[HyperV]VMshows up onmultiple nodesCSCvj22992

35(1a)1 Do not change MTU at ESXlevel on running cluster

2 Reset back to the original value

ChangingMTU from 9000 to 1500 on oneESX node causes storfs process to restarton all nodes

CSCvm99150

Release Notes for Cisco HX Data Platform Release 4056

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)After the installation (for newdeployments) or after an upgradeto 401a (for existingdeployments) perform thefollowing steps on all ControllerVMs

1 Edit the following tune files (onall Controller VMs)

optspringpathconfiglfftunes

optspringpathconfigvsi_16tbtunes

2 SetcleanerEnableSegSummaryCleaning

to false

3 After editing the above tunefiles

a SSH in to all the ControllerVMs

b Run storfstool -- -Z

c Run the below command andcheck the values Tune valueshould be true

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=true

d Type below commands andapplied the tunes changesdynamically

echo false gttmpstprocfssystemtunecleanerEnableSegSummaryCleaning

e Verify that tune values arechanged Run below commandValue should be false

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=false

f umount tmpstprocfs

A cluster with 8TB or 12TB disk drivesmay experience IO stalls for severalminutes after another node fails in thecluster

CSCvp23718

Release Notes for Cisco HX Data Platform Release 4057

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)1 Update the conf file manuallyto change to the link local addressto

2 Restart scvmclient on thecontroller and ESX

HX 35(2b) FlexVol to HX 40 CSIupgrade does not work FlexVol continuesto work

CSCvp26319

35(2a)Disable rebalance until node healsthen reenable rebalance

7-node cluster healing very slowlybecause rebalance change AMS reqs areblocking healing process

CSCvp62512

35(2a)NAWith moderate cluster storage capacityusage under site failover cluster clustermay experience an all paths (APD) downcondition

CSCvq49412

40(1a)NAThis product includes Third-partySoftware that is affected by thevulnerabilities identified by the followingCommon Vulnerability and Exposures(CVE) IDs

CVE-2016-0762 CVE-2016-6797CVE-2016-6816 CVE-2016-8735CVE-2017-5647 CVE-2017-12615CVE-2017-12616 CVE-2017-12617CVE-2017-7674 CVE-2018-1304CVE-2018-8014 CVE-2018-1336CVE-2018-8034 CVE-2018-11784CVE-2019-0232

CSCvp36364

35(2d)NAstcli cluster info command needs toprovide UCSM VIP address Currently itshows ucsm-hostcom

CSCvq11456

35(2a)No workarounds currently Thestale entries do not affect the dayto day operations of HyperFlex

HyperFlex upgrade validation failedbecause cluster has extraneous stNodes ininternal database

CSCvq32530

35(2a)NAWhen not in maintenancemode all drivesappear in HX Connect and get updated asexpected

When in maintenance mode extra drivesare shown in HX Connect with no slotnumbers

CSCvq42378

35(2b)Contact TAC to cleanup duplicatenode

Nodes duplicated in HX Connect aftercluster recreate

CSCvq61825

Release Notes for Cisco HX Data Platform Release 4058

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)NACurrently HyperFlex installs on Hyper-Vdo not contain the stcli security whitelistcommandset

CSCvq66245

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

Release Notes for Cisco HX Data Platform Release 4059

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)[Hyper-V] Converged node expansionfails for incoming node with VIC 14xx

CSCvq39523

Release Notes for Cisco HX Data Platform Release 4060

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

1 Set Jumbo frame inWindowsHypervisor for incomingnode

Get-NetAdapter storage-data-a|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

Get-NetAdapter storage-data-b|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

2 Verify all interfaces are setcorrectly to support Jumbo frames

Get-NetAdapter storage |Get-NetAdapterAdvancedProperty| RegistryKeyword -matchjumbo | ft -auto

You should receive the followingmessage

Name DisplayName DisplayValue

RegistryKeyword RegistryValue

---- ----------- ------------

--------------- -------------

vswitch-hx-storage-data Jumbo

Packet 9014 Bytes

JumboPacket 9014

storage-data-b Jumbo Packet

Bytes 9014 JumboPacket

9014

storage-data-a Jumbo Packet

Bytes 9014 JumboPacket

9014

3 Reboot only the incoming nodeor the node being expanded

4 Retry Cluster Expansion in

Release Notes for Cisco HX Data Platform Release 4061

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(2a)40(1a)A vulnerability in the web-based management interfaceof Cisco HyperFlex Software could allow anunauthenticated remote attacker to conduct a cross-siterequest forgery (CSRF) attack on an affected system

The vulnerability is due to insufficient CSRF protectionsfor the web UI on an affected device An attacker couldexploit this vulnerability by persuading a user of theinterface to follow a malicious link A successful exploitcould allow the attacker to perform arbitrary actions withthe privilege level of the affected user

Cisco has released software updates that address thisvulnerability There are no workarounds that address thisvulnerability

This advisory is available at the following link

httpstoolsciscocomsecuritycentercontentCiscoSecurityAdvisorycisco-sa-20190807-hypflex-csrf

CSCvp24343

40(2a)35(2d)Add message to HX Connect Progress flow to notAcknowledge Pending Activities and reboot the serverson UCSM

HX Connect is doing a controlled rolling server upgradein the background

CSCvq22844

40(2a)40(1b)Compute Node expand operation fails due to incorrectcluster name

CSCvr43786

40(2a)26(1e)RF-2 cluster shutdown during data resynchronizationafter a node failure (non storfs) followed by multipledisk read failures and a HardBlacklist of a disk

CSCvp41241

40(2a)40(1a)HyperFlex Edge nodes do not properly set the ESXihostname during deployment

CSCvo13143

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4039

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4040

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1b) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401b-33133p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4041

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4042

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1b)40(1a)While running the HyperFlex installer with WindowsServer Hyper-V cluster creation process fails with theerror Failure occurred during ClusterCreation process Unable to post thecontent to the downstream This symptomis encountered while deploying HyperFlex cluster withHyperFlex nodes configuredordered with Cisco VIC1457 MLOM (PID HX-MLOM-C25Q-04) andWindows Server Datacenter or Core with Hyper-Vversion 2016 or 2019

CSCvp64140

Release Notes for Cisco HX Data Platform Release 4043

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)35(2b)Adding Micron 5200 drive to a cluster fails to increasecluster capacity

CSCvo69067

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4044

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4045

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1a) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401a-33028p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4046

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4047

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1a)30(1d)Cluster instability when disks of different sector sizeplaced in HX node

CSCvk17250

40(1a)

35(2c)

35(1a)When logged in using a local HX user account insteadof a Virtual Center account an error message appearsintermittently indicating Virtual Centerunreachable or Resource informationcannot be updated when VC is reachable

CSCvo36198

40(1a)

35(2b)

30(1d)HXDP does not work with EMC RecoverPoint - needsto support VMware API (FSS-Readdir)

CSCvk38003

Release Notes for Cisco HX Data Platform Release 4048

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)

35(2a)

30(1i)

25(1c)

25(1d)

Zookeeper (Exhibitor) process on HyperFlex StorageController VMs may fail to respond to requests whenvarzookeeper has low or no free space

CSCvm90352

40(1a)26(1c)Use Asynchronous consolidation for HX SnapshotsCSCvn02151

40(1a)35(2b)HX Quiesced Snapshot for Backup VendorsCSCvo90713

40(1a)35(2a)The cluster creationcluster expansion workflow stopswith the following error message at the validation step

FIRMWARE-Check UCSC-SAS-M5HD

FIRMWARE-Check UCSC-SAS-M5HD Required00000029000000320000003500000050 Found00000058

Action Needed Update the Controller Firmware toRequired Version

CSCvn17787

35(2b)30(1d)Kernel migration fails when a local datastore is searchedfor in the ESXi host

CSCvn73127

35(2a)26(1b)A node shuts down when two disks are replaced (acaching disk and another capacity disk) where thecapacity disk is inserted first and then the caching diskis inserted

CSCvk46364

Hyper-V

40(1a)35(2a)Cluster expansion may fail with an error code 500 -operation timed out

CSCvn28721

35(2b)35(2a)During upgrade remove the VLAN on team created foruser vSwitch During fresh installation only one VLANtag is set to the vSwitch and team although multipleVLANs were entered

CSCvn54300

35(2b)35(2a)While upgrading a Hyper-V cluster on account of a rarerace condition between the stUpgradeService andZookeeper servers the upgrade orchestration throws anupgrade validation error and the upgrade process isaborted

CSCvn60486

Release Notes for Cisco HX Data Platform Release 4049

Resolved Caveats in Release 40(1a)

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Startrestart EAM service invCenter

When EAM service is not runningvCenter registration is failing

CSCvt55712

35(2a)Contact Cisco TAC if you believeyou are seeing similar conditions

HyperFlex Stretch Cluster saw poorapplication performance due to writelatency cluster remained unhealthy andrebalance was stuck

This defect is currently being used forinvestigative purposes

CSCvt45344

35(2g)Remove log files fromvarlogspringpath folder when thelog files are not getting rotatedStop the relevant service removelarge log files fromvarlogspringpath folder andrestart the service

varstv folder in Controller VM maybecome full

CSCvt36374

40(2b)Restarting the stMgr will initializeit again

In a Cluster configured with DR networkStMgr may not get initialized and stuckin a deadlock while enabling the IPTablerules (can be verified from the stMgrlog)

CSCvt89709

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Click recover for the selectedVM again under the standalonemode This will move the VM torecovered state

When the user recovers a VM protectedin a group it along with other VMs in thegroup move to recovering state instandalonemode The selectedVM shouldmove to recovered state

CSCvs75553

35(2e)Clean up the fail over cluster andthen recreate the fail over clusterNo need to touch the HX storagecluster

Hyper-V When replicated DC was usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2d)40(1a)

Keep cluster utilization as low aspossible To avoid read-only statekeep below 92

HyperFlex clusters storage utilization isabove 92 usable capacity

CSCvr20922

Release Notes for Cisco HX Data Platform Release 4050

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1b)NAAll the disks in the node got locked afterrebooting the node

We successfully unlocked usingsed-clientsh -U command but wantedto test with another reboot and driveslocked again

CSCvs74286

35(2g)To recover the VM copy over thedata disks and attach them to thenew VM

After enable encryption HX connectencryption status showing Caution butall of the disks enable encryptionsuccessful

-Self Encrypting Drives Service isrunning on all of the nodes

-There is no error message of encryption

- USB0 interface is up

-All SED disks showing supported 1enabled 1 locked 0

CSCvs93245

40(1b)Enable kernelpanic_on_oops as apersistent configuration This willcause the node to panic and rebootimmediately

A cluster node running HX release40(1b) may hang in the Linux kernelThis is classified as an oops and adeviation from the expected behavior

CSCvs54285

Release Notes for Cisco HX Data Platform Release 4051

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Enabling K8iscsi stack in Hx clusterCSCvs41324

Release Notes for Cisco HX Data Platform Release 4052

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Steps to enable K8 on HX cluster

Perform following steps on ALLthe controller VMs

1 Updateetcinitscvmclientconf toenable tune iscsiEnable totrue

sed -iesiscsiEnable=falseiscsiEnable=trueetcinitscvmclientconf

2 Run following initctlcommand to reloadconfiguration

initctl reload-configuration

3 Restart scvmclient process

stop scvmclient startscvmclient

4 Run following initctlcommand to mount SYSTEMDS

initctl emit --no-waitsystem-datastore-created

5 Verify that iscsiEnable tuneis set to true using thefollowing command

ps -eaf | grep scvmclient |grep -v grep

root 6241 1 1 Dec06 014007optspringpathstorfs-corescvmclient-T logEnabled=false -TlogSyslogEnabled=true -TlogEchoToScreen=false -TstatLoggingToFile=false -TstatLoggingToSyslog=true-TlogDir=varlogspringpath-TnfsBackendServerList=1010748100-T iscsiEnable=true -TiscsiUseAsync=true -T

Release Notes for Cisco HX Data Platform Release 4053

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

iscsiConfigFilePath=nfsSYSTEMistgtconfps -eaf | grep scvmclient |grep -v grep

35(2e)Following procedure can be usedto check NFS Queue Depth andincrease if needed

rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth256 lt- Low value[rootHXESXI1] vsish -e setvmkModulesnfsclientmnt[DATASTORE]propertiesmaxQDepth 1024[rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth1024 lt- Optimalvalue

This requires an ESXi host rebootto take effect

Please place the node in HyperflexMaintenanceMode and gracefullyreboot the node for the changes tobe applied

HyperFlex Datastore NFS Queue Depthshows as 256 which can lead toperformance (including latency) issues

CSCvr83056

40(1b)This is cosmetic The error will goaway once the cluster returns tohealthy status

When a single node is offline in clusterstcli cluster storage-summary shows twonodes unavailable rootSCVM~ stclicluster storage-summary

CSCvs47735

35(2g)Modify both the root and adminpassword for the SCVMs to be thesame

stcli security password set -uadmin

Cluster expansion fails at Config Installerstage when the root and admin passwordfor the storage controller (SCVM) aredifferent

CSCvs69317

35(2d)NAAfter successfully changing the DNSserver on the HX controller we still cansee the original DNS entry that was addedduring the deployment

CSCvs69154

Release Notes for Cisco HX Data Platform Release 4054

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Customers are strongly encouragedto work with TAC in order toidentify and remediate this issue

1 Change getcluster from 2m to8m inoptspringpathstorfs-mgmtstMgr-10confapplicationconfon all nodes

2 Restart stMgr on all nodes

You may see this error message after afailed upgrade or other task such asattempting to enter a node into HXmaintenance mode

CSCvs53555

40(2a)Run the following command oneach storage controller VM as root

grep -i monitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp sed -iesmonitor_dns_serverstruemonitor_dns_serversfalseoptspringpathhx-diag-toolswatchdog_configjsonampamp grep -imonitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp restart watchdog

Receive the following alertevent in HXConnect HX Controller VMHOSTNAME one or more configuredDNS servers not responding

CSCvt13947

35(2b)Delete any empty (Size 0) log filesunder varlogzookeeperversion-2ORvarlogzookeeperstandaloneversion-2

Zookeeper fails to start while Exhibitor isrunning however echo srvr returnsnothing

CSCvs21562

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

40(4e)1 De-commission the server

2 Power drain the server -REMOVE BOTH POWERCORDS ON THE BACK OFTHE SERVER FOR 10SECONDS THENREINSERTPOWERCORDS

3 Re-commission the server

M2 boot disk is missing from serverinventory after upgrade to server firmware40(4e) As a result server fails to boot toOS installed in the M2 disk The issuepersists after re-acknowledgement as wellas de-commission andre-acknowledgement of the server

CSCvs02466

Release Notes for Cisco HX Data Platform Release 4055

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Re-start the flusher manuallyMany disks taken offline by a fault ormanually resulting in ENOSPC oncluster

CSCvq32721

35(2e)Clean up fail over cluster and thenrecreate the fail over cluster Thereis no need to touch the HX storagecluster

[Hyper-V]When replicated DCwas usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2b)Manually install upgradescvmclient VIB andmake sure thatit matches with HXDP version

Upgrade from release HX 35(2a) to35(2x) 4X does not upgrade scvmclientRecovery point may not work

CSCvq54992

Management

30(1b)To recover the VM copy over thedata disks and attach them to thenew VM

[HyperV]VMshows up onmultiple nodesCSCvj22992

35(1a)1 Do not change MTU at ESXlevel on running cluster

2 Reset back to the original value

ChangingMTU from 9000 to 1500 on oneESX node causes storfs process to restarton all nodes

CSCvm99150

Release Notes for Cisco HX Data Platform Release 4056

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)After the installation (for newdeployments) or after an upgradeto 401a (for existingdeployments) perform thefollowing steps on all ControllerVMs

1 Edit the following tune files (onall Controller VMs)

optspringpathconfiglfftunes

optspringpathconfigvsi_16tbtunes

2 SetcleanerEnableSegSummaryCleaning

to false

3 After editing the above tunefiles

a SSH in to all the ControllerVMs

b Run storfstool -- -Z

c Run the below command andcheck the values Tune valueshould be true

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=true

d Type below commands andapplied the tunes changesdynamically

echo false gttmpstprocfssystemtunecleanerEnableSegSummaryCleaning

e Verify that tune values arechanged Run below commandValue should be false

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=false

f umount tmpstprocfs

A cluster with 8TB or 12TB disk drivesmay experience IO stalls for severalminutes after another node fails in thecluster

CSCvp23718

Release Notes for Cisco HX Data Platform Release 4057

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)1 Update the conf file manuallyto change to the link local addressto

2 Restart scvmclient on thecontroller and ESX

HX 35(2b) FlexVol to HX 40 CSIupgrade does not work FlexVol continuesto work

CSCvp26319

35(2a)Disable rebalance until node healsthen reenable rebalance

7-node cluster healing very slowlybecause rebalance change AMS reqs areblocking healing process

CSCvp62512

35(2a)NAWith moderate cluster storage capacityusage under site failover cluster clustermay experience an all paths (APD) downcondition

CSCvq49412

40(1a)NAThis product includes Third-partySoftware that is affected by thevulnerabilities identified by the followingCommon Vulnerability and Exposures(CVE) IDs

CVE-2016-0762 CVE-2016-6797CVE-2016-6816 CVE-2016-8735CVE-2017-5647 CVE-2017-12615CVE-2017-12616 CVE-2017-12617CVE-2017-7674 CVE-2018-1304CVE-2018-8014 CVE-2018-1336CVE-2018-8034 CVE-2018-11784CVE-2019-0232

CSCvp36364

35(2d)NAstcli cluster info command needs toprovide UCSM VIP address Currently itshows ucsm-hostcom

CSCvq11456

35(2a)No workarounds currently Thestale entries do not affect the dayto day operations of HyperFlex

HyperFlex upgrade validation failedbecause cluster has extraneous stNodes ininternal database

CSCvq32530

35(2a)NAWhen not in maintenancemode all drivesappear in HX Connect and get updated asexpected

When in maintenance mode extra drivesare shown in HX Connect with no slotnumbers

CSCvq42378

35(2b)Contact TAC to cleanup duplicatenode

Nodes duplicated in HX Connect aftercluster recreate

CSCvq61825

Release Notes for Cisco HX Data Platform Release 4058

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)NACurrently HyperFlex installs on Hyper-Vdo not contain the stcli security whitelistcommandset

CSCvq66245

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

Release Notes for Cisco HX Data Platform Release 4059

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)[Hyper-V] Converged node expansionfails for incoming node with VIC 14xx

CSCvq39523

Release Notes for Cisco HX Data Platform Release 4060

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

1 Set Jumbo frame inWindowsHypervisor for incomingnode

Get-NetAdapter storage-data-a|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

Get-NetAdapter storage-data-b|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

2 Verify all interfaces are setcorrectly to support Jumbo frames

Get-NetAdapter storage |Get-NetAdapterAdvancedProperty| RegistryKeyword -matchjumbo | ft -auto

You should receive the followingmessage

Name DisplayName DisplayValue

RegistryKeyword RegistryValue

---- ----------- ------------

--------------- -------------

vswitch-hx-storage-data Jumbo

Packet 9014 Bytes

JumboPacket 9014

storage-data-b Jumbo Packet

Bytes 9014 JumboPacket

9014

storage-data-a Jumbo Packet

Bytes 9014 JumboPacket

9014

3 Reboot only the incoming nodeor the node being expanded

4 Retry Cluster Expansion in

Release Notes for Cisco HX Data Platform Release 4061

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4040

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1b) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401b-33133p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4041

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4042

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1b)40(1a)While running the HyperFlex installer with WindowsServer Hyper-V cluster creation process fails with theerror Failure occurred during ClusterCreation process Unable to post thecontent to the downstream This symptomis encountered while deploying HyperFlex cluster withHyperFlex nodes configuredordered with Cisco VIC1457 MLOM (PID HX-MLOM-C25Q-04) andWindows Server Datacenter or Core with Hyper-Vversion 2016 or 2019

CSCvp64140

Release Notes for Cisco HX Data Platform Release 4043

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)35(2b)Adding Micron 5200 drive to a cluster fails to increasecluster capacity

CSCvo69067

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4044

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4045

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1a) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401a-33028p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4046

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4047

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1a)30(1d)Cluster instability when disks of different sector sizeplaced in HX node

CSCvk17250

40(1a)

35(2c)

35(1a)When logged in using a local HX user account insteadof a Virtual Center account an error message appearsintermittently indicating Virtual Centerunreachable or Resource informationcannot be updated when VC is reachable

CSCvo36198

40(1a)

35(2b)

30(1d)HXDP does not work with EMC RecoverPoint - needsto support VMware API (FSS-Readdir)

CSCvk38003

Release Notes for Cisco HX Data Platform Release 4048

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)

35(2a)

30(1i)

25(1c)

25(1d)

Zookeeper (Exhibitor) process on HyperFlex StorageController VMs may fail to respond to requests whenvarzookeeper has low or no free space

CSCvm90352

40(1a)26(1c)Use Asynchronous consolidation for HX SnapshotsCSCvn02151

40(1a)35(2b)HX Quiesced Snapshot for Backup VendorsCSCvo90713

40(1a)35(2a)The cluster creationcluster expansion workflow stopswith the following error message at the validation step

FIRMWARE-Check UCSC-SAS-M5HD

FIRMWARE-Check UCSC-SAS-M5HD Required00000029000000320000003500000050 Found00000058

Action Needed Update the Controller Firmware toRequired Version

CSCvn17787

35(2b)30(1d)Kernel migration fails when a local datastore is searchedfor in the ESXi host

CSCvn73127

35(2a)26(1b)A node shuts down when two disks are replaced (acaching disk and another capacity disk) where thecapacity disk is inserted first and then the caching diskis inserted

CSCvk46364

Hyper-V

40(1a)35(2a)Cluster expansion may fail with an error code 500 -operation timed out

CSCvn28721

35(2b)35(2a)During upgrade remove the VLAN on team created foruser vSwitch During fresh installation only one VLANtag is set to the vSwitch and team although multipleVLANs were entered

CSCvn54300

35(2b)35(2a)While upgrading a Hyper-V cluster on account of a rarerace condition between the stUpgradeService andZookeeper servers the upgrade orchestration throws anupgrade validation error and the upgrade process isaborted

CSCvn60486

Release Notes for Cisco HX Data Platform Release 4049

Resolved Caveats in Release 40(1a)

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Startrestart EAM service invCenter

When EAM service is not runningvCenter registration is failing

CSCvt55712

35(2a)Contact Cisco TAC if you believeyou are seeing similar conditions

HyperFlex Stretch Cluster saw poorapplication performance due to writelatency cluster remained unhealthy andrebalance was stuck

This defect is currently being used forinvestigative purposes

CSCvt45344

35(2g)Remove log files fromvarlogspringpath folder when thelog files are not getting rotatedStop the relevant service removelarge log files fromvarlogspringpath folder andrestart the service

varstv folder in Controller VM maybecome full

CSCvt36374

40(2b)Restarting the stMgr will initializeit again

In a Cluster configured with DR networkStMgr may not get initialized and stuckin a deadlock while enabling the IPTablerules (can be verified from the stMgrlog)

CSCvt89709

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Click recover for the selectedVM again under the standalonemode This will move the VM torecovered state

When the user recovers a VM protectedin a group it along with other VMs in thegroup move to recovering state instandalonemode The selectedVM shouldmove to recovered state

CSCvs75553

35(2e)Clean up the fail over cluster andthen recreate the fail over clusterNo need to touch the HX storagecluster

Hyper-V When replicated DC was usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2d)40(1a)

Keep cluster utilization as low aspossible To avoid read-only statekeep below 92

HyperFlex clusters storage utilization isabove 92 usable capacity

CSCvr20922

Release Notes for Cisco HX Data Platform Release 4050

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1b)NAAll the disks in the node got locked afterrebooting the node

We successfully unlocked usingsed-clientsh -U command but wantedto test with another reboot and driveslocked again

CSCvs74286

35(2g)To recover the VM copy over thedata disks and attach them to thenew VM

After enable encryption HX connectencryption status showing Caution butall of the disks enable encryptionsuccessful

-Self Encrypting Drives Service isrunning on all of the nodes

-There is no error message of encryption

- USB0 interface is up

-All SED disks showing supported 1enabled 1 locked 0

CSCvs93245

40(1b)Enable kernelpanic_on_oops as apersistent configuration This willcause the node to panic and rebootimmediately

A cluster node running HX release40(1b) may hang in the Linux kernelThis is classified as an oops and adeviation from the expected behavior

CSCvs54285

Release Notes for Cisco HX Data Platform Release 4051

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Enabling K8iscsi stack in Hx clusterCSCvs41324

Release Notes for Cisco HX Data Platform Release 4052

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Steps to enable K8 on HX cluster

Perform following steps on ALLthe controller VMs

1 Updateetcinitscvmclientconf toenable tune iscsiEnable totrue

sed -iesiscsiEnable=falseiscsiEnable=trueetcinitscvmclientconf

2 Run following initctlcommand to reloadconfiguration

initctl reload-configuration

3 Restart scvmclient process

stop scvmclient startscvmclient

4 Run following initctlcommand to mount SYSTEMDS

initctl emit --no-waitsystem-datastore-created

5 Verify that iscsiEnable tuneis set to true using thefollowing command

ps -eaf | grep scvmclient |grep -v grep

root 6241 1 1 Dec06 014007optspringpathstorfs-corescvmclient-T logEnabled=false -TlogSyslogEnabled=true -TlogEchoToScreen=false -TstatLoggingToFile=false -TstatLoggingToSyslog=true-TlogDir=varlogspringpath-TnfsBackendServerList=1010748100-T iscsiEnable=true -TiscsiUseAsync=true -T

Release Notes for Cisco HX Data Platform Release 4053

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

iscsiConfigFilePath=nfsSYSTEMistgtconfps -eaf | grep scvmclient |grep -v grep

35(2e)Following procedure can be usedto check NFS Queue Depth andincrease if needed

rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth256 lt- Low value[rootHXESXI1] vsish -e setvmkModulesnfsclientmnt[DATASTORE]propertiesmaxQDepth 1024[rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth1024 lt- Optimalvalue

This requires an ESXi host rebootto take effect

Please place the node in HyperflexMaintenanceMode and gracefullyreboot the node for the changes tobe applied

HyperFlex Datastore NFS Queue Depthshows as 256 which can lead toperformance (including latency) issues

CSCvr83056

40(1b)This is cosmetic The error will goaway once the cluster returns tohealthy status

When a single node is offline in clusterstcli cluster storage-summary shows twonodes unavailable rootSCVM~ stclicluster storage-summary

CSCvs47735

35(2g)Modify both the root and adminpassword for the SCVMs to be thesame

stcli security password set -uadmin

Cluster expansion fails at Config Installerstage when the root and admin passwordfor the storage controller (SCVM) aredifferent

CSCvs69317

35(2d)NAAfter successfully changing the DNSserver on the HX controller we still cansee the original DNS entry that was addedduring the deployment

CSCvs69154

Release Notes for Cisco HX Data Platform Release 4054

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Customers are strongly encouragedto work with TAC in order toidentify and remediate this issue

1 Change getcluster from 2m to8m inoptspringpathstorfs-mgmtstMgr-10confapplicationconfon all nodes

2 Restart stMgr on all nodes

You may see this error message after afailed upgrade or other task such asattempting to enter a node into HXmaintenance mode

CSCvs53555

40(2a)Run the following command oneach storage controller VM as root

grep -i monitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp sed -iesmonitor_dns_serverstruemonitor_dns_serversfalseoptspringpathhx-diag-toolswatchdog_configjsonampamp grep -imonitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp restart watchdog

Receive the following alertevent in HXConnect HX Controller VMHOSTNAME one or more configuredDNS servers not responding

CSCvt13947

35(2b)Delete any empty (Size 0) log filesunder varlogzookeeperversion-2ORvarlogzookeeperstandaloneversion-2

Zookeeper fails to start while Exhibitor isrunning however echo srvr returnsnothing

CSCvs21562

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

40(4e)1 De-commission the server

2 Power drain the server -REMOVE BOTH POWERCORDS ON THE BACK OFTHE SERVER FOR 10SECONDS THENREINSERTPOWERCORDS

3 Re-commission the server

M2 boot disk is missing from serverinventory after upgrade to server firmware40(4e) As a result server fails to boot toOS installed in the M2 disk The issuepersists after re-acknowledgement as wellas de-commission andre-acknowledgement of the server

CSCvs02466

Release Notes for Cisco HX Data Platform Release 4055

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Re-start the flusher manuallyMany disks taken offline by a fault ormanually resulting in ENOSPC oncluster

CSCvq32721

35(2e)Clean up fail over cluster and thenrecreate the fail over cluster Thereis no need to touch the HX storagecluster

[Hyper-V]When replicated DCwas usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2b)Manually install upgradescvmclient VIB andmake sure thatit matches with HXDP version

Upgrade from release HX 35(2a) to35(2x) 4X does not upgrade scvmclientRecovery point may not work

CSCvq54992

Management

30(1b)To recover the VM copy over thedata disks and attach them to thenew VM

[HyperV]VMshows up onmultiple nodesCSCvj22992

35(1a)1 Do not change MTU at ESXlevel on running cluster

2 Reset back to the original value

ChangingMTU from 9000 to 1500 on oneESX node causes storfs process to restarton all nodes

CSCvm99150

Release Notes for Cisco HX Data Platform Release 4056

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)After the installation (for newdeployments) or after an upgradeto 401a (for existingdeployments) perform thefollowing steps on all ControllerVMs

1 Edit the following tune files (onall Controller VMs)

optspringpathconfiglfftunes

optspringpathconfigvsi_16tbtunes

2 SetcleanerEnableSegSummaryCleaning

to false

3 After editing the above tunefiles

a SSH in to all the ControllerVMs

b Run storfstool -- -Z

c Run the below command andcheck the values Tune valueshould be true

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=true

d Type below commands andapplied the tunes changesdynamically

echo false gttmpstprocfssystemtunecleanerEnableSegSummaryCleaning

e Verify that tune values arechanged Run below commandValue should be false

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=false

f umount tmpstprocfs

A cluster with 8TB or 12TB disk drivesmay experience IO stalls for severalminutes after another node fails in thecluster

CSCvp23718

Release Notes for Cisco HX Data Platform Release 4057

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)1 Update the conf file manuallyto change to the link local addressto

2 Restart scvmclient on thecontroller and ESX

HX 35(2b) FlexVol to HX 40 CSIupgrade does not work FlexVol continuesto work

CSCvp26319

35(2a)Disable rebalance until node healsthen reenable rebalance

7-node cluster healing very slowlybecause rebalance change AMS reqs areblocking healing process

CSCvp62512

35(2a)NAWith moderate cluster storage capacityusage under site failover cluster clustermay experience an all paths (APD) downcondition

CSCvq49412

40(1a)NAThis product includes Third-partySoftware that is affected by thevulnerabilities identified by the followingCommon Vulnerability and Exposures(CVE) IDs

CVE-2016-0762 CVE-2016-6797CVE-2016-6816 CVE-2016-8735CVE-2017-5647 CVE-2017-12615CVE-2017-12616 CVE-2017-12617CVE-2017-7674 CVE-2018-1304CVE-2018-8014 CVE-2018-1336CVE-2018-8034 CVE-2018-11784CVE-2019-0232

CSCvp36364

35(2d)NAstcli cluster info command needs toprovide UCSM VIP address Currently itshows ucsm-hostcom

CSCvq11456

35(2a)No workarounds currently Thestale entries do not affect the dayto day operations of HyperFlex

HyperFlex upgrade validation failedbecause cluster has extraneous stNodes ininternal database

CSCvq32530

35(2a)NAWhen not in maintenancemode all drivesappear in HX Connect and get updated asexpected

When in maintenance mode extra drivesare shown in HX Connect with no slotnumbers

CSCvq42378

35(2b)Contact TAC to cleanup duplicatenode

Nodes duplicated in HX Connect aftercluster recreate

CSCvq61825

Release Notes for Cisco HX Data Platform Release 4058

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)NACurrently HyperFlex installs on Hyper-Vdo not contain the stcli security whitelistcommandset

CSCvq66245

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

Release Notes for Cisco HX Data Platform Release 4059

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)[Hyper-V] Converged node expansionfails for incoming node with VIC 14xx

CSCvq39523

Release Notes for Cisco HX Data Platform Release 4060

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

1 Set Jumbo frame inWindowsHypervisor for incomingnode

Get-NetAdapter storage-data-a|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

Get-NetAdapter storage-data-b|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

2 Verify all interfaces are setcorrectly to support Jumbo frames

Get-NetAdapter storage |Get-NetAdapterAdvancedProperty| RegistryKeyword -matchjumbo | ft -auto

You should receive the followingmessage

Name DisplayName DisplayValue

RegistryKeyword RegistryValue

---- ----------- ------------

--------------- -------------

vswitch-hx-storage-data Jumbo

Packet 9014 Bytes

JumboPacket 9014

storage-data-b Jumbo Packet

Bytes 9014 JumboPacket

9014

storage-data-a Jumbo Packet

Bytes 9014 JumboPacket

9014

3 Reboot only the incoming nodeor the node being expanded

4 Retry Cluster Expansion in

Release Notes for Cisco HX Data Platform Release 4061

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1b) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401b-33133p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4041

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4042

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1b)40(1a)While running the HyperFlex installer with WindowsServer Hyper-V cluster creation process fails with theerror Failure occurred during ClusterCreation process Unable to post thecontent to the downstream This symptomis encountered while deploying HyperFlex cluster withHyperFlex nodes configuredordered with Cisco VIC1457 MLOM (PID HX-MLOM-C25Q-04) andWindows Server Datacenter or Core with Hyper-Vversion 2016 or 2019

CSCvp64140

Release Notes for Cisco HX Data Platform Release 4043

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)35(2b)Adding Micron 5200 drive to a cluster fails to increasecluster capacity

CSCvo69067

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4044

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4045

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1a) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401a-33028p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4046

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4047

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1a)30(1d)Cluster instability when disks of different sector sizeplaced in HX node

CSCvk17250

40(1a)

35(2c)

35(1a)When logged in using a local HX user account insteadof a Virtual Center account an error message appearsintermittently indicating Virtual Centerunreachable or Resource informationcannot be updated when VC is reachable

CSCvo36198

40(1a)

35(2b)

30(1d)HXDP does not work with EMC RecoverPoint - needsto support VMware API (FSS-Readdir)

CSCvk38003

Release Notes for Cisco HX Data Platform Release 4048

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)

35(2a)

30(1i)

25(1c)

25(1d)

Zookeeper (Exhibitor) process on HyperFlex StorageController VMs may fail to respond to requests whenvarzookeeper has low or no free space

CSCvm90352

40(1a)26(1c)Use Asynchronous consolidation for HX SnapshotsCSCvn02151

40(1a)35(2b)HX Quiesced Snapshot for Backup VendorsCSCvo90713

40(1a)35(2a)The cluster creationcluster expansion workflow stopswith the following error message at the validation step

FIRMWARE-Check UCSC-SAS-M5HD

FIRMWARE-Check UCSC-SAS-M5HD Required00000029000000320000003500000050 Found00000058

Action Needed Update the Controller Firmware toRequired Version

CSCvn17787

35(2b)30(1d)Kernel migration fails when a local datastore is searchedfor in the ESXi host

CSCvn73127

35(2a)26(1b)A node shuts down when two disks are replaced (acaching disk and another capacity disk) where thecapacity disk is inserted first and then the caching diskis inserted

CSCvk46364

Hyper-V

40(1a)35(2a)Cluster expansion may fail with an error code 500 -operation timed out

CSCvn28721

35(2b)35(2a)During upgrade remove the VLAN on team created foruser vSwitch During fresh installation only one VLANtag is set to the vSwitch and team although multipleVLANs were entered

CSCvn54300

35(2b)35(2a)While upgrading a Hyper-V cluster on account of a rarerace condition between the stUpgradeService andZookeeper servers the upgrade orchestration throws anupgrade validation error and the upgrade process isaborted

CSCvn60486

Release Notes for Cisco HX Data Platform Release 4049

Resolved Caveats in Release 40(1a)

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Startrestart EAM service invCenter

When EAM service is not runningvCenter registration is failing

CSCvt55712

35(2a)Contact Cisco TAC if you believeyou are seeing similar conditions

HyperFlex Stretch Cluster saw poorapplication performance due to writelatency cluster remained unhealthy andrebalance was stuck

This defect is currently being used forinvestigative purposes

CSCvt45344

35(2g)Remove log files fromvarlogspringpath folder when thelog files are not getting rotatedStop the relevant service removelarge log files fromvarlogspringpath folder andrestart the service

varstv folder in Controller VM maybecome full

CSCvt36374

40(2b)Restarting the stMgr will initializeit again

In a Cluster configured with DR networkStMgr may not get initialized and stuckin a deadlock while enabling the IPTablerules (can be verified from the stMgrlog)

CSCvt89709

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Click recover for the selectedVM again under the standalonemode This will move the VM torecovered state

When the user recovers a VM protectedin a group it along with other VMs in thegroup move to recovering state instandalonemode The selectedVM shouldmove to recovered state

CSCvs75553

35(2e)Clean up the fail over cluster andthen recreate the fail over clusterNo need to touch the HX storagecluster

Hyper-V When replicated DC was usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2d)40(1a)

Keep cluster utilization as low aspossible To avoid read-only statekeep below 92

HyperFlex clusters storage utilization isabove 92 usable capacity

CSCvr20922

Release Notes for Cisco HX Data Platform Release 4050

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1b)NAAll the disks in the node got locked afterrebooting the node

We successfully unlocked usingsed-clientsh -U command but wantedto test with another reboot and driveslocked again

CSCvs74286

35(2g)To recover the VM copy over thedata disks and attach them to thenew VM

After enable encryption HX connectencryption status showing Caution butall of the disks enable encryptionsuccessful

-Self Encrypting Drives Service isrunning on all of the nodes

-There is no error message of encryption

- USB0 interface is up

-All SED disks showing supported 1enabled 1 locked 0

CSCvs93245

40(1b)Enable kernelpanic_on_oops as apersistent configuration This willcause the node to panic and rebootimmediately

A cluster node running HX release40(1b) may hang in the Linux kernelThis is classified as an oops and adeviation from the expected behavior

CSCvs54285

Release Notes for Cisco HX Data Platform Release 4051

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Enabling K8iscsi stack in Hx clusterCSCvs41324

Release Notes for Cisco HX Data Platform Release 4052

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Steps to enable K8 on HX cluster

Perform following steps on ALLthe controller VMs

1 Updateetcinitscvmclientconf toenable tune iscsiEnable totrue

sed -iesiscsiEnable=falseiscsiEnable=trueetcinitscvmclientconf

2 Run following initctlcommand to reloadconfiguration

initctl reload-configuration

3 Restart scvmclient process

stop scvmclient startscvmclient

4 Run following initctlcommand to mount SYSTEMDS

initctl emit --no-waitsystem-datastore-created

5 Verify that iscsiEnable tuneis set to true using thefollowing command

ps -eaf | grep scvmclient |grep -v grep

root 6241 1 1 Dec06 014007optspringpathstorfs-corescvmclient-T logEnabled=false -TlogSyslogEnabled=true -TlogEchoToScreen=false -TstatLoggingToFile=false -TstatLoggingToSyslog=true-TlogDir=varlogspringpath-TnfsBackendServerList=1010748100-T iscsiEnable=true -TiscsiUseAsync=true -T

Release Notes for Cisco HX Data Platform Release 4053

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

iscsiConfigFilePath=nfsSYSTEMistgtconfps -eaf | grep scvmclient |grep -v grep

35(2e)Following procedure can be usedto check NFS Queue Depth andincrease if needed

rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth256 lt- Low value[rootHXESXI1] vsish -e setvmkModulesnfsclientmnt[DATASTORE]propertiesmaxQDepth 1024[rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth1024 lt- Optimalvalue

This requires an ESXi host rebootto take effect

Please place the node in HyperflexMaintenanceMode and gracefullyreboot the node for the changes tobe applied

HyperFlex Datastore NFS Queue Depthshows as 256 which can lead toperformance (including latency) issues

CSCvr83056

40(1b)This is cosmetic The error will goaway once the cluster returns tohealthy status

When a single node is offline in clusterstcli cluster storage-summary shows twonodes unavailable rootSCVM~ stclicluster storage-summary

CSCvs47735

35(2g)Modify both the root and adminpassword for the SCVMs to be thesame

stcli security password set -uadmin

Cluster expansion fails at Config Installerstage when the root and admin passwordfor the storage controller (SCVM) aredifferent

CSCvs69317

35(2d)NAAfter successfully changing the DNSserver on the HX controller we still cansee the original DNS entry that was addedduring the deployment

CSCvs69154

Release Notes for Cisco HX Data Platform Release 4054

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Customers are strongly encouragedto work with TAC in order toidentify and remediate this issue

1 Change getcluster from 2m to8m inoptspringpathstorfs-mgmtstMgr-10confapplicationconfon all nodes

2 Restart stMgr on all nodes

You may see this error message after afailed upgrade or other task such asattempting to enter a node into HXmaintenance mode

CSCvs53555

40(2a)Run the following command oneach storage controller VM as root

grep -i monitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp sed -iesmonitor_dns_serverstruemonitor_dns_serversfalseoptspringpathhx-diag-toolswatchdog_configjsonampamp grep -imonitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp restart watchdog

Receive the following alertevent in HXConnect HX Controller VMHOSTNAME one or more configuredDNS servers not responding

CSCvt13947

35(2b)Delete any empty (Size 0) log filesunder varlogzookeeperversion-2ORvarlogzookeeperstandaloneversion-2

Zookeeper fails to start while Exhibitor isrunning however echo srvr returnsnothing

CSCvs21562

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

40(4e)1 De-commission the server

2 Power drain the server -REMOVE BOTH POWERCORDS ON THE BACK OFTHE SERVER FOR 10SECONDS THENREINSERTPOWERCORDS

3 Re-commission the server

M2 boot disk is missing from serverinventory after upgrade to server firmware40(4e) As a result server fails to boot toOS installed in the M2 disk The issuepersists after re-acknowledgement as wellas de-commission andre-acknowledgement of the server

CSCvs02466

Release Notes for Cisco HX Data Platform Release 4055

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Re-start the flusher manuallyMany disks taken offline by a fault ormanually resulting in ENOSPC oncluster

CSCvq32721

35(2e)Clean up fail over cluster and thenrecreate the fail over cluster Thereis no need to touch the HX storagecluster

[Hyper-V]When replicated DCwas usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2b)Manually install upgradescvmclient VIB andmake sure thatit matches with HXDP version

Upgrade from release HX 35(2a) to35(2x) 4X does not upgrade scvmclientRecovery point may not work

CSCvq54992

Management

30(1b)To recover the VM copy over thedata disks and attach them to thenew VM

[HyperV]VMshows up onmultiple nodesCSCvj22992

35(1a)1 Do not change MTU at ESXlevel on running cluster

2 Reset back to the original value

ChangingMTU from 9000 to 1500 on oneESX node causes storfs process to restarton all nodes

CSCvm99150

Release Notes for Cisco HX Data Platform Release 4056

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)After the installation (for newdeployments) or after an upgradeto 401a (for existingdeployments) perform thefollowing steps on all ControllerVMs

1 Edit the following tune files (onall Controller VMs)

optspringpathconfiglfftunes

optspringpathconfigvsi_16tbtunes

2 SetcleanerEnableSegSummaryCleaning

to false

3 After editing the above tunefiles

a SSH in to all the ControllerVMs

b Run storfstool -- -Z

c Run the below command andcheck the values Tune valueshould be true

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=true

d Type below commands andapplied the tunes changesdynamically

echo false gttmpstprocfssystemtunecleanerEnableSegSummaryCleaning

e Verify that tune values arechanged Run below commandValue should be false

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=false

f umount tmpstprocfs

A cluster with 8TB or 12TB disk drivesmay experience IO stalls for severalminutes after another node fails in thecluster

CSCvp23718

Release Notes for Cisco HX Data Platform Release 4057

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)1 Update the conf file manuallyto change to the link local addressto

2 Restart scvmclient on thecontroller and ESX

HX 35(2b) FlexVol to HX 40 CSIupgrade does not work FlexVol continuesto work

CSCvp26319

35(2a)Disable rebalance until node healsthen reenable rebalance

7-node cluster healing very slowlybecause rebalance change AMS reqs areblocking healing process

CSCvp62512

35(2a)NAWith moderate cluster storage capacityusage under site failover cluster clustermay experience an all paths (APD) downcondition

CSCvq49412

40(1a)NAThis product includes Third-partySoftware that is affected by thevulnerabilities identified by the followingCommon Vulnerability and Exposures(CVE) IDs

CVE-2016-0762 CVE-2016-6797CVE-2016-6816 CVE-2016-8735CVE-2017-5647 CVE-2017-12615CVE-2017-12616 CVE-2017-12617CVE-2017-7674 CVE-2018-1304CVE-2018-8014 CVE-2018-1336CVE-2018-8034 CVE-2018-11784CVE-2019-0232

CSCvp36364

35(2d)NAstcli cluster info command needs toprovide UCSM VIP address Currently itshows ucsm-hostcom

CSCvq11456

35(2a)No workarounds currently Thestale entries do not affect the dayto day operations of HyperFlex

HyperFlex upgrade validation failedbecause cluster has extraneous stNodes ininternal database

CSCvq32530

35(2a)NAWhen not in maintenancemode all drivesappear in HX Connect and get updated asexpected

When in maintenance mode extra drivesare shown in HX Connect with no slotnumbers

CSCvq42378

35(2b)Contact TAC to cleanup duplicatenode

Nodes duplicated in HX Connect aftercluster recreate

CSCvq61825

Release Notes for Cisco HX Data Platform Release 4058

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)NACurrently HyperFlex installs on Hyper-Vdo not contain the stcli security whitelistcommandset

CSCvq66245

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

Release Notes for Cisco HX Data Platform Release 4059

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)[Hyper-V] Converged node expansionfails for incoming node with VIC 14xx

CSCvq39523

Release Notes for Cisco HX Data Platform Release 4060

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

1 Set Jumbo frame inWindowsHypervisor for incomingnode

Get-NetAdapter storage-data-a|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

Get-NetAdapter storage-data-b|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

2 Verify all interfaces are setcorrectly to support Jumbo frames

Get-NetAdapter storage |Get-NetAdapterAdvancedProperty| RegistryKeyword -matchjumbo | ft -auto

You should receive the followingmessage

Name DisplayName DisplayValue

RegistryKeyword RegistryValue

---- ----------- ------------

--------------- -------------

vswitch-hx-storage-data Jumbo

Packet 9014 Bytes

JumboPacket 9014

storage-data-b Jumbo Packet

Bytes 9014 JumboPacket

9014

storage-data-a Jumbo Packet

Bytes 9014 JumboPacket

9014

3 Reboot only the incoming nodeor the node being expanded

4 Retry Cluster Expansion in

Release Notes for Cisco HX Data Platform Release 4061

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4042

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1b)40(1a)While running the HyperFlex installer with WindowsServer Hyper-V cluster creation process fails with theerror Failure occurred during ClusterCreation process Unable to post thecontent to the downstream This symptomis encountered while deploying HyperFlex cluster withHyperFlex nodes configuredordered with Cisco VIC1457 MLOM (PID HX-MLOM-C25Q-04) andWindows Server Datacenter or Core with Hyper-Vversion 2016 or 2019

CSCvp64140

Release Notes for Cisco HX Data Platform Release 4043

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)35(2b)Adding Micron 5200 drive to a cluster fails to increasecluster capacity

CSCvo69067

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4044

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4045

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1a) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401a-33028p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4046

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4047

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1a)30(1d)Cluster instability when disks of different sector sizeplaced in HX node

CSCvk17250

40(1a)

35(2c)

35(1a)When logged in using a local HX user account insteadof a Virtual Center account an error message appearsintermittently indicating Virtual Centerunreachable or Resource informationcannot be updated when VC is reachable

CSCvo36198

40(1a)

35(2b)

30(1d)HXDP does not work with EMC RecoverPoint - needsto support VMware API (FSS-Readdir)

CSCvk38003

Release Notes for Cisco HX Data Platform Release 4048

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)

35(2a)

30(1i)

25(1c)

25(1d)

Zookeeper (Exhibitor) process on HyperFlex StorageController VMs may fail to respond to requests whenvarzookeeper has low or no free space

CSCvm90352

40(1a)26(1c)Use Asynchronous consolidation for HX SnapshotsCSCvn02151

40(1a)35(2b)HX Quiesced Snapshot for Backup VendorsCSCvo90713

40(1a)35(2a)The cluster creationcluster expansion workflow stopswith the following error message at the validation step

FIRMWARE-Check UCSC-SAS-M5HD

FIRMWARE-Check UCSC-SAS-M5HD Required00000029000000320000003500000050 Found00000058

Action Needed Update the Controller Firmware toRequired Version

CSCvn17787

35(2b)30(1d)Kernel migration fails when a local datastore is searchedfor in the ESXi host

CSCvn73127

35(2a)26(1b)A node shuts down when two disks are replaced (acaching disk and another capacity disk) where thecapacity disk is inserted first and then the caching diskis inserted

CSCvk46364

Hyper-V

40(1a)35(2a)Cluster expansion may fail with an error code 500 -operation timed out

CSCvn28721

35(2b)35(2a)During upgrade remove the VLAN on team created foruser vSwitch During fresh installation only one VLANtag is set to the vSwitch and team although multipleVLANs were entered

CSCvn54300

35(2b)35(2a)While upgrading a Hyper-V cluster on account of a rarerace condition between the stUpgradeService andZookeeper servers the upgrade orchestration throws anupgrade validation error and the upgrade process isaborted

CSCvn60486

Release Notes for Cisco HX Data Platform Release 4049

Resolved Caveats in Release 40(1a)

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Startrestart EAM service invCenter

When EAM service is not runningvCenter registration is failing

CSCvt55712

35(2a)Contact Cisco TAC if you believeyou are seeing similar conditions

HyperFlex Stretch Cluster saw poorapplication performance due to writelatency cluster remained unhealthy andrebalance was stuck

This defect is currently being used forinvestigative purposes

CSCvt45344

35(2g)Remove log files fromvarlogspringpath folder when thelog files are not getting rotatedStop the relevant service removelarge log files fromvarlogspringpath folder andrestart the service

varstv folder in Controller VM maybecome full

CSCvt36374

40(2b)Restarting the stMgr will initializeit again

In a Cluster configured with DR networkStMgr may not get initialized and stuckin a deadlock while enabling the IPTablerules (can be verified from the stMgrlog)

CSCvt89709

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Click recover for the selectedVM again under the standalonemode This will move the VM torecovered state

When the user recovers a VM protectedin a group it along with other VMs in thegroup move to recovering state instandalonemode The selectedVM shouldmove to recovered state

CSCvs75553

35(2e)Clean up the fail over cluster andthen recreate the fail over clusterNo need to touch the HX storagecluster

Hyper-V When replicated DC was usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2d)40(1a)

Keep cluster utilization as low aspossible To avoid read-only statekeep below 92

HyperFlex clusters storage utilization isabove 92 usable capacity

CSCvr20922

Release Notes for Cisco HX Data Platform Release 4050

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1b)NAAll the disks in the node got locked afterrebooting the node

We successfully unlocked usingsed-clientsh -U command but wantedto test with another reboot and driveslocked again

CSCvs74286

35(2g)To recover the VM copy over thedata disks and attach them to thenew VM

After enable encryption HX connectencryption status showing Caution butall of the disks enable encryptionsuccessful

-Self Encrypting Drives Service isrunning on all of the nodes

-There is no error message of encryption

- USB0 interface is up

-All SED disks showing supported 1enabled 1 locked 0

CSCvs93245

40(1b)Enable kernelpanic_on_oops as apersistent configuration This willcause the node to panic and rebootimmediately

A cluster node running HX release40(1b) may hang in the Linux kernelThis is classified as an oops and adeviation from the expected behavior

CSCvs54285

Release Notes for Cisco HX Data Platform Release 4051

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Enabling K8iscsi stack in Hx clusterCSCvs41324

Release Notes for Cisco HX Data Platform Release 4052

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Steps to enable K8 on HX cluster

Perform following steps on ALLthe controller VMs

1 Updateetcinitscvmclientconf toenable tune iscsiEnable totrue

sed -iesiscsiEnable=falseiscsiEnable=trueetcinitscvmclientconf

2 Run following initctlcommand to reloadconfiguration

initctl reload-configuration

3 Restart scvmclient process

stop scvmclient startscvmclient

4 Run following initctlcommand to mount SYSTEMDS

initctl emit --no-waitsystem-datastore-created

5 Verify that iscsiEnable tuneis set to true using thefollowing command

ps -eaf | grep scvmclient |grep -v grep

root 6241 1 1 Dec06 014007optspringpathstorfs-corescvmclient-T logEnabled=false -TlogSyslogEnabled=true -TlogEchoToScreen=false -TstatLoggingToFile=false -TstatLoggingToSyslog=true-TlogDir=varlogspringpath-TnfsBackendServerList=1010748100-T iscsiEnable=true -TiscsiUseAsync=true -T

Release Notes for Cisco HX Data Platform Release 4053

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

iscsiConfigFilePath=nfsSYSTEMistgtconfps -eaf | grep scvmclient |grep -v grep

35(2e)Following procedure can be usedto check NFS Queue Depth andincrease if needed

rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth256 lt- Low value[rootHXESXI1] vsish -e setvmkModulesnfsclientmnt[DATASTORE]propertiesmaxQDepth 1024[rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth1024 lt- Optimalvalue

This requires an ESXi host rebootto take effect

Please place the node in HyperflexMaintenanceMode and gracefullyreboot the node for the changes tobe applied

HyperFlex Datastore NFS Queue Depthshows as 256 which can lead toperformance (including latency) issues

CSCvr83056

40(1b)This is cosmetic The error will goaway once the cluster returns tohealthy status

When a single node is offline in clusterstcli cluster storage-summary shows twonodes unavailable rootSCVM~ stclicluster storage-summary

CSCvs47735

35(2g)Modify both the root and adminpassword for the SCVMs to be thesame

stcli security password set -uadmin

Cluster expansion fails at Config Installerstage when the root and admin passwordfor the storage controller (SCVM) aredifferent

CSCvs69317

35(2d)NAAfter successfully changing the DNSserver on the HX controller we still cansee the original DNS entry that was addedduring the deployment

CSCvs69154

Release Notes for Cisco HX Data Platform Release 4054

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Customers are strongly encouragedto work with TAC in order toidentify and remediate this issue

1 Change getcluster from 2m to8m inoptspringpathstorfs-mgmtstMgr-10confapplicationconfon all nodes

2 Restart stMgr on all nodes

You may see this error message after afailed upgrade or other task such asattempting to enter a node into HXmaintenance mode

CSCvs53555

40(2a)Run the following command oneach storage controller VM as root

grep -i monitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp sed -iesmonitor_dns_serverstruemonitor_dns_serversfalseoptspringpathhx-diag-toolswatchdog_configjsonampamp grep -imonitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp restart watchdog

Receive the following alertevent in HXConnect HX Controller VMHOSTNAME one or more configuredDNS servers not responding

CSCvt13947

35(2b)Delete any empty (Size 0) log filesunder varlogzookeeperversion-2ORvarlogzookeeperstandaloneversion-2

Zookeeper fails to start while Exhibitor isrunning however echo srvr returnsnothing

CSCvs21562

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

40(4e)1 De-commission the server

2 Power drain the server -REMOVE BOTH POWERCORDS ON THE BACK OFTHE SERVER FOR 10SECONDS THENREINSERTPOWERCORDS

3 Re-commission the server

M2 boot disk is missing from serverinventory after upgrade to server firmware40(4e) As a result server fails to boot toOS installed in the M2 disk The issuepersists after re-acknowledgement as wellas de-commission andre-acknowledgement of the server

CSCvs02466

Release Notes for Cisco HX Data Platform Release 4055

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Re-start the flusher manuallyMany disks taken offline by a fault ormanually resulting in ENOSPC oncluster

CSCvq32721

35(2e)Clean up fail over cluster and thenrecreate the fail over cluster Thereis no need to touch the HX storagecluster

[Hyper-V]When replicated DCwas usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2b)Manually install upgradescvmclient VIB andmake sure thatit matches with HXDP version

Upgrade from release HX 35(2a) to35(2x) 4X does not upgrade scvmclientRecovery point may not work

CSCvq54992

Management

30(1b)To recover the VM copy over thedata disks and attach them to thenew VM

[HyperV]VMshows up onmultiple nodesCSCvj22992

35(1a)1 Do not change MTU at ESXlevel on running cluster

2 Reset back to the original value

ChangingMTU from 9000 to 1500 on oneESX node causes storfs process to restarton all nodes

CSCvm99150

Release Notes for Cisco HX Data Platform Release 4056

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)After the installation (for newdeployments) or after an upgradeto 401a (for existingdeployments) perform thefollowing steps on all ControllerVMs

1 Edit the following tune files (onall Controller VMs)

optspringpathconfiglfftunes

optspringpathconfigvsi_16tbtunes

2 SetcleanerEnableSegSummaryCleaning

to false

3 After editing the above tunefiles

a SSH in to all the ControllerVMs

b Run storfstool -- -Z

c Run the below command andcheck the values Tune valueshould be true

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=true

d Type below commands andapplied the tunes changesdynamically

echo false gttmpstprocfssystemtunecleanerEnableSegSummaryCleaning

e Verify that tune values arechanged Run below commandValue should be false

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=false

f umount tmpstprocfs

A cluster with 8TB or 12TB disk drivesmay experience IO stalls for severalminutes after another node fails in thecluster

CSCvp23718

Release Notes for Cisco HX Data Platform Release 4057

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)1 Update the conf file manuallyto change to the link local addressto

2 Restart scvmclient on thecontroller and ESX

HX 35(2b) FlexVol to HX 40 CSIupgrade does not work FlexVol continuesto work

CSCvp26319

35(2a)Disable rebalance until node healsthen reenable rebalance

7-node cluster healing very slowlybecause rebalance change AMS reqs areblocking healing process

CSCvp62512

35(2a)NAWith moderate cluster storage capacityusage under site failover cluster clustermay experience an all paths (APD) downcondition

CSCvq49412

40(1a)NAThis product includes Third-partySoftware that is affected by thevulnerabilities identified by the followingCommon Vulnerability and Exposures(CVE) IDs

CVE-2016-0762 CVE-2016-6797CVE-2016-6816 CVE-2016-8735CVE-2017-5647 CVE-2017-12615CVE-2017-12616 CVE-2017-12617CVE-2017-7674 CVE-2018-1304CVE-2018-8014 CVE-2018-1336CVE-2018-8034 CVE-2018-11784CVE-2019-0232

CSCvp36364

35(2d)NAstcli cluster info command needs toprovide UCSM VIP address Currently itshows ucsm-hostcom

CSCvq11456

35(2a)No workarounds currently Thestale entries do not affect the dayto day operations of HyperFlex

HyperFlex upgrade validation failedbecause cluster has extraneous stNodes ininternal database

CSCvq32530

35(2a)NAWhen not in maintenancemode all drivesappear in HX Connect and get updated asexpected

When in maintenance mode extra drivesare shown in HX Connect with no slotnumbers

CSCvq42378

35(2b)Contact TAC to cleanup duplicatenode

Nodes duplicated in HX Connect aftercluster recreate

CSCvq61825

Release Notes for Cisco HX Data Platform Release 4058

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)NACurrently HyperFlex installs on Hyper-Vdo not contain the stcli security whitelistcommandset

CSCvq66245

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

Release Notes for Cisco HX Data Platform Release 4059

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)[Hyper-V] Converged node expansionfails for incoming node with VIC 14xx

CSCvq39523

Release Notes for Cisco HX Data Platform Release 4060

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

1 Set Jumbo frame inWindowsHypervisor for incomingnode

Get-NetAdapter storage-data-a|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

Get-NetAdapter storage-data-b|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

2 Verify all interfaces are setcorrectly to support Jumbo frames

Get-NetAdapter storage |Get-NetAdapterAdvancedProperty| RegistryKeyword -matchjumbo | ft -auto

You should receive the followingmessage

Name DisplayName DisplayValue

RegistryKeyword RegistryValue

---- ----------- ------------

--------------- -------------

vswitch-hx-storage-data Jumbo

Packet 9014 Bytes

JumboPacket 9014

storage-data-b Jumbo Packet

Bytes 9014 JumboPacket

9014

storage-data-a Jumbo Packet

Bytes 9014 JumboPacket

9014

3 Reboot only the incoming nodeor the node being expanded

4 Retry Cluster Expansion in

Release Notes for Cisco HX Data Platform Release 4061

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1b)40(1a)While running the HyperFlex installer with WindowsServer Hyper-V cluster creation process fails with theerror Failure occurred during ClusterCreation process Unable to post thecontent to the downstream This symptomis encountered while deploying HyperFlex cluster withHyperFlex nodes configuredordered with Cisco VIC1457 MLOM (PID HX-MLOM-C25Q-04) andWindows Server Datacenter or Core with Hyper-Vversion 2016 or 2019

CSCvp64140

Release Notes for Cisco HX Data Platform Release 4043

Resolved Caveats in Release 40(1b)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)35(2b)Adding Micron 5200 drive to a cluster fails to increasecluster capacity

CSCvo69067

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4044

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4045

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1a) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401a-33028p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4046

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4047

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1a)30(1d)Cluster instability when disks of different sector sizeplaced in HX node

CSCvk17250

40(1a)

35(2c)

35(1a)When logged in using a local HX user account insteadof a Virtual Center account an error message appearsintermittently indicating Virtual Centerunreachable or Resource informationcannot be updated when VC is reachable

CSCvo36198

40(1a)

35(2b)

30(1d)HXDP does not work with EMC RecoverPoint - needsto support VMware API (FSS-Readdir)

CSCvk38003

Release Notes for Cisco HX Data Platform Release 4048

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)

35(2a)

30(1i)

25(1c)

25(1d)

Zookeeper (Exhibitor) process on HyperFlex StorageController VMs may fail to respond to requests whenvarzookeeper has low or no free space

CSCvm90352

40(1a)26(1c)Use Asynchronous consolidation for HX SnapshotsCSCvn02151

40(1a)35(2b)HX Quiesced Snapshot for Backup VendorsCSCvo90713

40(1a)35(2a)The cluster creationcluster expansion workflow stopswith the following error message at the validation step

FIRMWARE-Check UCSC-SAS-M5HD

FIRMWARE-Check UCSC-SAS-M5HD Required00000029000000320000003500000050 Found00000058

Action Needed Update the Controller Firmware toRequired Version

CSCvn17787

35(2b)30(1d)Kernel migration fails when a local datastore is searchedfor in the ESXi host

CSCvn73127

35(2a)26(1b)A node shuts down when two disks are replaced (acaching disk and another capacity disk) where thecapacity disk is inserted first and then the caching diskis inserted

CSCvk46364

Hyper-V

40(1a)35(2a)Cluster expansion may fail with an error code 500 -operation timed out

CSCvn28721

35(2b)35(2a)During upgrade remove the VLAN on team created foruser vSwitch During fresh installation only one VLANtag is set to the vSwitch and team although multipleVLANs were entered

CSCvn54300

35(2b)35(2a)While upgrading a Hyper-V cluster on account of a rarerace condition between the stUpgradeService andZookeeper servers the upgrade orchestration throws anupgrade validation error and the upgrade process isaborted

CSCvn60486

Release Notes for Cisco HX Data Platform Release 4049

Resolved Caveats in Release 40(1a)

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Startrestart EAM service invCenter

When EAM service is not runningvCenter registration is failing

CSCvt55712

35(2a)Contact Cisco TAC if you believeyou are seeing similar conditions

HyperFlex Stretch Cluster saw poorapplication performance due to writelatency cluster remained unhealthy andrebalance was stuck

This defect is currently being used forinvestigative purposes

CSCvt45344

35(2g)Remove log files fromvarlogspringpath folder when thelog files are not getting rotatedStop the relevant service removelarge log files fromvarlogspringpath folder andrestart the service

varstv folder in Controller VM maybecome full

CSCvt36374

40(2b)Restarting the stMgr will initializeit again

In a Cluster configured with DR networkStMgr may not get initialized and stuckin a deadlock while enabling the IPTablerules (can be verified from the stMgrlog)

CSCvt89709

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Click recover for the selectedVM again under the standalonemode This will move the VM torecovered state

When the user recovers a VM protectedin a group it along with other VMs in thegroup move to recovering state instandalonemode The selectedVM shouldmove to recovered state

CSCvs75553

35(2e)Clean up the fail over cluster andthen recreate the fail over clusterNo need to touch the HX storagecluster

Hyper-V When replicated DC was usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2d)40(1a)

Keep cluster utilization as low aspossible To avoid read-only statekeep below 92

HyperFlex clusters storage utilization isabove 92 usable capacity

CSCvr20922

Release Notes for Cisco HX Data Platform Release 4050

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1b)NAAll the disks in the node got locked afterrebooting the node

We successfully unlocked usingsed-clientsh -U command but wantedto test with another reboot and driveslocked again

CSCvs74286

35(2g)To recover the VM copy over thedata disks and attach them to thenew VM

After enable encryption HX connectencryption status showing Caution butall of the disks enable encryptionsuccessful

-Self Encrypting Drives Service isrunning on all of the nodes

-There is no error message of encryption

- USB0 interface is up

-All SED disks showing supported 1enabled 1 locked 0

CSCvs93245

40(1b)Enable kernelpanic_on_oops as apersistent configuration This willcause the node to panic and rebootimmediately

A cluster node running HX release40(1b) may hang in the Linux kernelThis is classified as an oops and adeviation from the expected behavior

CSCvs54285

Release Notes for Cisco HX Data Platform Release 4051

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Enabling K8iscsi stack in Hx clusterCSCvs41324

Release Notes for Cisco HX Data Platform Release 4052

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Steps to enable K8 on HX cluster

Perform following steps on ALLthe controller VMs

1 Updateetcinitscvmclientconf toenable tune iscsiEnable totrue

sed -iesiscsiEnable=falseiscsiEnable=trueetcinitscvmclientconf

2 Run following initctlcommand to reloadconfiguration

initctl reload-configuration

3 Restart scvmclient process

stop scvmclient startscvmclient

4 Run following initctlcommand to mount SYSTEMDS

initctl emit --no-waitsystem-datastore-created

5 Verify that iscsiEnable tuneis set to true using thefollowing command

ps -eaf | grep scvmclient |grep -v grep

root 6241 1 1 Dec06 014007optspringpathstorfs-corescvmclient-T logEnabled=false -TlogSyslogEnabled=true -TlogEchoToScreen=false -TstatLoggingToFile=false -TstatLoggingToSyslog=true-TlogDir=varlogspringpath-TnfsBackendServerList=1010748100-T iscsiEnable=true -TiscsiUseAsync=true -T

Release Notes for Cisco HX Data Platform Release 4053

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

iscsiConfigFilePath=nfsSYSTEMistgtconfps -eaf | grep scvmclient |grep -v grep

35(2e)Following procedure can be usedto check NFS Queue Depth andincrease if needed

rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth256 lt- Low value[rootHXESXI1] vsish -e setvmkModulesnfsclientmnt[DATASTORE]propertiesmaxQDepth 1024[rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth1024 lt- Optimalvalue

This requires an ESXi host rebootto take effect

Please place the node in HyperflexMaintenanceMode and gracefullyreboot the node for the changes tobe applied

HyperFlex Datastore NFS Queue Depthshows as 256 which can lead toperformance (including latency) issues

CSCvr83056

40(1b)This is cosmetic The error will goaway once the cluster returns tohealthy status

When a single node is offline in clusterstcli cluster storage-summary shows twonodes unavailable rootSCVM~ stclicluster storage-summary

CSCvs47735

35(2g)Modify both the root and adminpassword for the SCVMs to be thesame

stcli security password set -uadmin

Cluster expansion fails at Config Installerstage when the root and admin passwordfor the storage controller (SCVM) aredifferent

CSCvs69317

35(2d)NAAfter successfully changing the DNSserver on the HX controller we still cansee the original DNS entry that was addedduring the deployment

CSCvs69154

Release Notes for Cisco HX Data Platform Release 4054

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Customers are strongly encouragedto work with TAC in order toidentify and remediate this issue

1 Change getcluster from 2m to8m inoptspringpathstorfs-mgmtstMgr-10confapplicationconfon all nodes

2 Restart stMgr on all nodes

You may see this error message after afailed upgrade or other task such asattempting to enter a node into HXmaintenance mode

CSCvs53555

40(2a)Run the following command oneach storage controller VM as root

grep -i monitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp sed -iesmonitor_dns_serverstruemonitor_dns_serversfalseoptspringpathhx-diag-toolswatchdog_configjsonampamp grep -imonitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp restart watchdog

Receive the following alertevent in HXConnect HX Controller VMHOSTNAME one or more configuredDNS servers not responding

CSCvt13947

35(2b)Delete any empty (Size 0) log filesunder varlogzookeeperversion-2ORvarlogzookeeperstandaloneversion-2

Zookeeper fails to start while Exhibitor isrunning however echo srvr returnsnothing

CSCvs21562

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

40(4e)1 De-commission the server

2 Power drain the server -REMOVE BOTH POWERCORDS ON THE BACK OFTHE SERVER FOR 10SECONDS THENREINSERTPOWERCORDS

3 Re-commission the server

M2 boot disk is missing from serverinventory after upgrade to server firmware40(4e) As a result server fails to boot toOS installed in the M2 disk The issuepersists after re-acknowledgement as wellas de-commission andre-acknowledgement of the server

CSCvs02466

Release Notes for Cisco HX Data Platform Release 4055

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Re-start the flusher manuallyMany disks taken offline by a fault ormanually resulting in ENOSPC oncluster

CSCvq32721

35(2e)Clean up fail over cluster and thenrecreate the fail over cluster Thereis no need to touch the HX storagecluster

[Hyper-V]When replicated DCwas usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2b)Manually install upgradescvmclient VIB andmake sure thatit matches with HXDP version

Upgrade from release HX 35(2a) to35(2x) 4X does not upgrade scvmclientRecovery point may not work

CSCvq54992

Management

30(1b)To recover the VM copy over thedata disks and attach them to thenew VM

[HyperV]VMshows up onmultiple nodesCSCvj22992

35(1a)1 Do not change MTU at ESXlevel on running cluster

2 Reset back to the original value

ChangingMTU from 9000 to 1500 on oneESX node causes storfs process to restarton all nodes

CSCvm99150

Release Notes for Cisco HX Data Platform Release 4056

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)After the installation (for newdeployments) or after an upgradeto 401a (for existingdeployments) perform thefollowing steps on all ControllerVMs

1 Edit the following tune files (onall Controller VMs)

optspringpathconfiglfftunes

optspringpathconfigvsi_16tbtunes

2 SetcleanerEnableSegSummaryCleaning

to false

3 After editing the above tunefiles

a SSH in to all the ControllerVMs

b Run storfstool -- -Z

c Run the below command andcheck the values Tune valueshould be true

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=true

d Type below commands andapplied the tunes changesdynamically

echo false gttmpstprocfssystemtunecleanerEnableSegSummaryCleaning

e Verify that tune values arechanged Run below commandValue should be false

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=false

f umount tmpstprocfs

A cluster with 8TB or 12TB disk drivesmay experience IO stalls for severalminutes after another node fails in thecluster

CSCvp23718

Release Notes for Cisco HX Data Platform Release 4057

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)1 Update the conf file manuallyto change to the link local addressto

2 Restart scvmclient on thecontroller and ESX

HX 35(2b) FlexVol to HX 40 CSIupgrade does not work FlexVol continuesto work

CSCvp26319

35(2a)Disable rebalance until node healsthen reenable rebalance

7-node cluster healing very slowlybecause rebalance change AMS reqs areblocking healing process

CSCvp62512

35(2a)NAWith moderate cluster storage capacityusage under site failover cluster clustermay experience an all paths (APD) downcondition

CSCvq49412

40(1a)NAThis product includes Third-partySoftware that is affected by thevulnerabilities identified by the followingCommon Vulnerability and Exposures(CVE) IDs

CVE-2016-0762 CVE-2016-6797CVE-2016-6816 CVE-2016-8735CVE-2017-5647 CVE-2017-12615CVE-2017-12616 CVE-2017-12617CVE-2017-7674 CVE-2018-1304CVE-2018-8014 CVE-2018-1336CVE-2018-8034 CVE-2018-11784CVE-2019-0232

CSCvp36364

35(2d)NAstcli cluster info command needs toprovide UCSM VIP address Currently itshows ucsm-hostcom

CSCvq11456

35(2a)No workarounds currently Thestale entries do not affect the dayto day operations of HyperFlex

HyperFlex upgrade validation failedbecause cluster has extraneous stNodes ininternal database

CSCvq32530

35(2a)NAWhen not in maintenancemode all drivesappear in HX Connect and get updated asexpected

When in maintenance mode extra drivesare shown in HX Connect with no slotnumbers

CSCvq42378

35(2b)Contact TAC to cleanup duplicatenode

Nodes duplicated in HX Connect aftercluster recreate

CSCvq61825

Release Notes for Cisco HX Data Platform Release 4058

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)NACurrently HyperFlex installs on Hyper-Vdo not contain the stcli security whitelistcommandset

CSCvq66245

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

Release Notes for Cisco HX Data Platform Release 4059

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)[Hyper-V] Converged node expansionfails for incoming node with VIC 14xx

CSCvq39523

Release Notes for Cisco HX Data Platform Release 4060

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

1 Set Jumbo frame inWindowsHypervisor for incomingnode

Get-NetAdapter storage-data-a|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

Get-NetAdapter storage-data-b|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

2 Verify all interfaces are setcorrectly to support Jumbo frames

Get-NetAdapter storage |Get-NetAdapterAdvancedProperty| RegistryKeyword -matchjumbo | ft -auto

You should receive the followingmessage

Name DisplayName DisplayValue

RegistryKeyword RegistryValue

---- ----------- ------------

--------------- -------------

vswitch-hx-storage-data Jumbo

Packet 9014 Bytes

JumboPacket 9014

storage-data-b Jumbo Packet

Bytes 9014 JumboPacket

9014

storage-data-a Jumbo Packet

Bytes 9014 JumboPacket

9014

3 Reboot only the incoming nodeor the node being expanded

4 Retry Cluster Expansion in

Release Notes for Cisco HX Data Platform Release 4061

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1b)35(2b)Adding Micron 5200 drive to a cluster fails to increasecluster capacity

CSCvo69067

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

ESXi Installation Upgrade Expansion Management

Release Notes for Cisco HX Data Platform Release 4044

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4045

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1a) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401a-33028p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4046

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4047

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1a)30(1d)Cluster instability when disks of different sector sizeplaced in HX node

CSCvk17250

40(1a)

35(2c)

35(1a)When logged in using a local HX user account insteadof a Virtual Center account an error message appearsintermittently indicating Virtual Centerunreachable or Resource informationcannot be updated when VC is reachable

CSCvo36198

40(1a)

35(2b)

30(1d)HXDP does not work with EMC RecoverPoint - needsto support VMware API (FSS-Readdir)

CSCvk38003

Release Notes for Cisco HX Data Platform Release 4048

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)

35(2a)

30(1i)

25(1c)

25(1d)

Zookeeper (Exhibitor) process on HyperFlex StorageController VMs may fail to respond to requests whenvarzookeeper has low or no free space

CSCvm90352

40(1a)26(1c)Use Asynchronous consolidation for HX SnapshotsCSCvn02151

40(1a)35(2b)HX Quiesced Snapshot for Backup VendorsCSCvo90713

40(1a)35(2a)The cluster creationcluster expansion workflow stopswith the following error message at the validation step

FIRMWARE-Check UCSC-SAS-M5HD

FIRMWARE-Check UCSC-SAS-M5HD Required00000029000000320000003500000050 Found00000058

Action Needed Update the Controller Firmware toRequired Version

CSCvn17787

35(2b)30(1d)Kernel migration fails when a local datastore is searchedfor in the ESXi host

CSCvn73127

35(2a)26(1b)A node shuts down when two disks are replaced (acaching disk and another capacity disk) where thecapacity disk is inserted first and then the caching diskis inserted

CSCvk46364

Hyper-V

40(1a)35(2a)Cluster expansion may fail with an error code 500 -operation timed out

CSCvn28721

35(2b)35(2a)During upgrade remove the VLAN on team created foruser vSwitch During fresh installation only one VLANtag is set to the vSwitch and team although multipleVLANs were entered

CSCvn54300

35(2b)35(2a)While upgrading a Hyper-V cluster on account of a rarerace condition between the stUpgradeService andZookeeper servers the upgrade orchestration throws anupgrade validation error and the upgrade process isaborted

CSCvn60486

Release Notes for Cisco HX Data Platform Release 4049

Resolved Caveats in Release 40(1a)

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Startrestart EAM service invCenter

When EAM service is not runningvCenter registration is failing

CSCvt55712

35(2a)Contact Cisco TAC if you believeyou are seeing similar conditions

HyperFlex Stretch Cluster saw poorapplication performance due to writelatency cluster remained unhealthy andrebalance was stuck

This defect is currently being used forinvestigative purposes

CSCvt45344

35(2g)Remove log files fromvarlogspringpath folder when thelog files are not getting rotatedStop the relevant service removelarge log files fromvarlogspringpath folder andrestart the service

varstv folder in Controller VM maybecome full

CSCvt36374

40(2b)Restarting the stMgr will initializeit again

In a Cluster configured with DR networkStMgr may not get initialized and stuckin a deadlock while enabling the IPTablerules (can be verified from the stMgrlog)

CSCvt89709

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Click recover for the selectedVM again under the standalonemode This will move the VM torecovered state

When the user recovers a VM protectedin a group it along with other VMs in thegroup move to recovering state instandalonemode The selectedVM shouldmove to recovered state

CSCvs75553

35(2e)Clean up the fail over cluster andthen recreate the fail over clusterNo need to touch the HX storagecluster

Hyper-V When replicated DC was usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2d)40(1a)

Keep cluster utilization as low aspossible To avoid read-only statekeep below 92

HyperFlex clusters storage utilization isabove 92 usable capacity

CSCvr20922

Release Notes for Cisco HX Data Platform Release 4050

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1b)NAAll the disks in the node got locked afterrebooting the node

We successfully unlocked usingsed-clientsh -U command but wantedto test with another reboot and driveslocked again

CSCvs74286

35(2g)To recover the VM copy over thedata disks and attach them to thenew VM

After enable encryption HX connectencryption status showing Caution butall of the disks enable encryptionsuccessful

-Self Encrypting Drives Service isrunning on all of the nodes

-There is no error message of encryption

- USB0 interface is up

-All SED disks showing supported 1enabled 1 locked 0

CSCvs93245

40(1b)Enable kernelpanic_on_oops as apersistent configuration This willcause the node to panic and rebootimmediately

A cluster node running HX release40(1b) may hang in the Linux kernelThis is classified as an oops and adeviation from the expected behavior

CSCvs54285

Release Notes for Cisco HX Data Platform Release 4051

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Enabling K8iscsi stack in Hx clusterCSCvs41324

Release Notes for Cisco HX Data Platform Release 4052

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Steps to enable K8 on HX cluster

Perform following steps on ALLthe controller VMs

1 Updateetcinitscvmclientconf toenable tune iscsiEnable totrue

sed -iesiscsiEnable=falseiscsiEnable=trueetcinitscvmclientconf

2 Run following initctlcommand to reloadconfiguration

initctl reload-configuration

3 Restart scvmclient process

stop scvmclient startscvmclient

4 Run following initctlcommand to mount SYSTEMDS

initctl emit --no-waitsystem-datastore-created

5 Verify that iscsiEnable tuneis set to true using thefollowing command

ps -eaf | grep scvmclient |grep -v grep

root 6241 1 1 Dec06 014007optspringpathstorfs-corescvmclient-T logEnabled=false -TlogSyslogEnabled=true -TlogEchoToScreen=false -TstatLoggingToFile=false -TstatLoggingToSyslog=true-TlogDir=varlogspringpath-TnfsBackendServerList=1010748100-T iscsiEnable=true -TiscsiUseAsync=true -T

Release Notes for Cisco HX Data Platform Release 4053

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

iscsiConfigFilePath=nfsSYSTEMistgtconfps -eaf | grep scvmclient |grep -v grep

35(2e)Following procedure can be usedto check NFS Queue Depth andincrease if needed

rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth256 lt- Low value[rootHXESXI1] vsish -e setvmkModulesnfsclientmnt[DATASTORE]propertiesmaxQDepth 1024[rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth1024 lt- Optimalvalue

This requires an ESXi host rebootto take effect

Please place the node in HyperflexMaintenanceMode and gracefullyreboot the node for the changes tobe applied

HyperFlex Datastore NFS Queue Depthshows as 256 which can lead toperformance (including latency) issues

CSCvr83056

40(1b)This is cosmetic The error will goaway once the cluster returns tohealthy status

When a single node is offline in clusterstcli cluster storage-summary shows twonodes unavailable rootSCVM~ stclicluster storage-summary

CSCvs47735

35(2g)Modify both the root and adminpassword for the SCVMs to be thesame

stcli security password set -uadmin

Cluster expansion fails at Config Installerstage when the root and admin passwordfor the storage controller (SCVM) aredifferent

CSCvs69317

35(2d)NAAfter successfully changing the DNSserver on the HX controller we still cansee the original DNS entry that was addedduring the deployment

CSCvs69154

Release Notes for Cisco HX Data Platform Release 4054

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Customers are strongly encouragedto work with TAC in order toidentify and remediate this issue

1 Change getcluster from 2m to8m inoptspringpathstorfs-mgmtstMgr-10confapplicationconfon all nodes

2 Restart stMgr on all nodes

You may see this error message after afailed upgrade or other task such asattempting to enter a node into HXmaintenance mode

CSCvs53555

40(2a)Run the following command oneach storage controller VM as root

grep -i monitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp sed -iesmonitor_dns_serverstruemonitor_dns_serversfalseoptspringpathhx-diag-toolswatchdog_configjsonampamp grep -imonitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp restart watchdog

Receive the following alertevent in HXConnect HX Controller VMHOSTNAME one or more configuredDNS servers not responding

CSCvt13947

35(2b)Delete any empty (Size 0) log filesunder varlogzookeeperversion-2ORvarlogzookeeperstandaloneversion-2

Zookeeper fails to start while Exhibitor isrunning however echo srvr returnsnothing

CSCvs21562

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

40(4e)1 De-commission the server

2 Power drain the server -REMOVE BOTH POWERCORDS ON THE BACK OFTHE SERVER FOR 10SECONDS THENREINSERTPOWERCORDS

3 Re-commission the server

M2 boot disk is missing from serverinventory after upgrade to server firmware40(4e) As a result server fails to boot toOS installed in the M2 disk The issuepersists after re-acknowledgement as wellas de-commission andre-acknowledgement of the server

CSCvs02466

Release Notes for Cisco HX Data Platform Release 4055

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Re-start the flusher manuallyMany disks taken offline by a fault ormanually resulting in ENOSPC oncluster

CSCvq32721

35(2e)Clean up fail over cluster and thenrecreate the fail over cluster Thereis no need to touch the HX storagecluster

[Hyper-V]When replicated DCwas usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2b)Manually install upgradescvmclient VIB andmake sure thatit matches with HXDP version

Upgrade from release HX 35(2a) to35(2x) 4X does not upgrade scvmclientRecovery point may not work

CSCvq54992

Management

30(1b)To recover the VM copy over thedata disks and attach them to thenew VM

[HyperV]VMshows up onmultiple nodesCSCvj22992

35(1a)1 Do not change MTU at ESXlevel on running cluster

2 Reset back to the original value

ChangingMTU from 9000 to 1500 on oneESX node causes storfs process to restarton all nodes

CSCvm99150

Release Notes for Cisco HX Data Platform Release 4056

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)After the installation (for newdeployments) or after an upgradeto 401a (for existingdeployments) perform thefollowing steps on all ControllerVMs

1 Edit the following tune files (onall Controller VMs)

optspringpathconfiglfftunes

optspringpathconfigvsi_16tbtunes

2 SetcleanerEnableSegSummaryCleaning

to false

3 After editing the above tunefiles

a SSH in to all the ControllerVMs

b Run storfstool -- -Z

c Run the below command andcheck the values Tune valueshould be true

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=true

d Type below commands andapplied the tunes changesdynamically

echo false gttmpstprocfssystemtunecleanerEnableSegSummaryCleaning

e Verify that tune values arechanged Run below commandValue should be false

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=false

f umount tmpstprocfs

A cluster with 8TB or 12TB disk drivesmay experience IO stalls for severalminutes after another node fails in thecluster

CSCvp23718

Release Notes for Cisco HX Data Platform Release 4057

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)1 Update the conf file manuallyto change to the link local addressto

2 Restart scvmclient on thecontroller and ESX

HX 35(2b) FlexVol to HX 40 CSIupgrade does not work FlexVol continuesto work

CSCvp26319

35(2a)Disable rebalance until node healsthen reenable rebalance

7-node cluster healing very slowlybecause rebalance change AMS reqs areblocking healing process

CSCvp62512

35(2a)NAWith moderate cluster storage capacityusage under site failover cluster clustermay experience an all paths (APD) downcondition

CSCvq49412

40(1a)NAThis product includes Third-partySoftware that is affected by thevulnerabilities identified by the followingCommon Vulnerability and Exposures(CVE) IDs

CVE-2016-0762 CVE-2016-6797CVE-2016-6816 CVE-2016-8735CVE-2017-5647 CVE-2017-12615CVE-2017-12616 CVE-2017-12617CVE-2017-7674 CVE-2018-1304CVE-2018-8014 CVE-2018-1336CVE-2018-8034 CVE-2018-11784CVE-2019-0232

CSCvp36364

35(2d)NAstcli cluster info command needs toprovide UCSM VIP address Currently itshows ucsm-hostcom

CSCvq11456

35(2a)No workarounds currently Thestale entries do not affect the dayto day operations of HyperFlex

HyperFlex upgrade validation failedbecause cluster has extraneous stNodes ininternal database

CSCvq32530

35(2a)NAWhen not in maintenancemode all drivesappear in HX Connect and get updated asexpected

When in maintenance mode extra drivesare shown in HX Connect with no slotnumbers

CSCvq42378

35(2b)Contact TAC to cleanup duplicatenode

Nodes duplicated in HX Connect aftercluster recreate

CSCvq61825

Release Notes for Cisco HX Data Platform Release 4058

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)NACurrently HyperFlex installs on Hyper-Vdo not contain the stcli security whitelistcommandset

CSCvq66245

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

Release Notes for Cisco HX Data Platform Release 4059

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)[Hyper-V] Converged node expansionfails for incoming node with VIC 14xx

CSCvq39523

Release Notes for Cisco HX Data Platform Release 4060

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

1 Set Jumbo frame inWindowsHypervisor for incomingnode

Get-NetAdapter storage-data-a|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

Get-NetAdapter storage-data-b|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

2 Verify all interfaces are setcorrectly to support Jumbo frames

Get-NetAdapter storage |Get-NetAdapterAdvancedProperty| RegistryKeyword -matchjumbo | ft -auto

You should receive the followingmessage

Name DisplayName DisplayValue

RegistryKeyword RegistryValue

---- ----------- ------------

--------------- -------------

vswitch-hx-storage-data Jumbo

Packet 9014 Bytes

JumboPacket 9014

storage-data-b Jumbo Packet

Bytes 9014 JumboPacket

9014

storage-data-a Jumbo Packet

Bytes 9014 JumboPacket

9014

3 Reboot only the incoming nodeor the node being expanded

4 Retry Cluster Expansion in

Release Notes for Cisco HX Data Platform Release 4061

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)26(1e)CSCvs28167

Release Notes for Cisco HX Data Platform Release 4045

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1a) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401a-33028p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4046

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4047

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1a)30(1d)Cluster instability when disks of different sector sizeplaced in HX node

CSCvk17250

40(1a)

35(2c)

35(1a)When logged in using a local HX user account insteadof a Virtual Center account an error message appearsintermittently indicating Virtual Centerunreachable or Resource informationcannot be updated when VC is reachable

CSCvo36198

40(1a)

35(2b)

30(1d)HXDP does not work with EMC RecoverPoint - needsto support VMware API (FSS-Readdir)

CSCvk38003

Release Notes for Cisco HX Data Platform Release 4048

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)

35(2a)

30(1i)

25(1c)

25(1d)

Zookeeper (Exhibitor) process on HyperFlex StorageController VMs may fail to respond to requests whenvarzookeeper has low or no free space

CSCvm90352

40(1a)26(1c)Use Asynchronous consolidation for HX SnapshotsCSCvn02151

40(1a)35(2b)HX Quiesced Snapshot for Backup VendorsCSCvo90713

40(1a)35(2a)The cluster creationcluster expansion workflow stopswith the following error message at the validation step

FIRMWARE-Check UCSC-SAS-M5HD

FIRMWARE-Check UCSC-SAS-M5HD Required00000029000000320000003500000050 Found00000058

Action Needed Update the Controller Firmware toRequired Version

CSCvn17787

35(2b)30(1d)Kernel migration fails when a local datastore is searchedfor in the ESXi host

CSCvn73127

35(2a)26(1b)A node shuts down when two disks are replaced (acaching disk and another capacity disk) where thecapacity disk is inserted first and then the caching diskis inserted

CSCvk46364

Hyper-V

40(1a)35(2a)Cluster expansion may fail with an error code 500 -operation timed out

CSCvn28721

35(2b)35(2a)During upgrade remove the VLAN on team created foruser vSwitch During fresh installation only one VLANtag is set to the vSwitch and team although multipleVLANs were entered

CSCvn54300

35(2b)35(2a)While upgrading a Hyper-V cluster on account of a rarerace condition between the stUpgradeService andZookeeper servers the upgrade orchestration throws anupgrade validation error and the upgrade process isaborted

CSCvn60486

Release Notes for Cisco HX Data Platform Release 4049

Resolved Caveats in Release 40(1a)

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Startrestart EAM service invCenter

When EAM service is not runningvCenter registration is failing

CSCvt55712

35(2a)Contact Cisco TAC if you believeyou are seeing similar conditions

HyperFlex Stretch Cluster saw poorapplication performance due to writelatency cluster remained unhealthy andrebalance was stuck

This defect is currently being used forinvestigative purposes

CSCvt45344

35(2g)Remove log files fromvarlogspringpath folder when thelog files are not getting rotatedStop the relevant service removelarge log files fromvarlogspringpath folder andrestart the service

varstv folder in Controller VM maybecome full

CSCvt36374

40(2b)Restarting the stMgr will initializeit again

In a Cluster configured with DR networkStMgr may not get initialized and stuckin a deadlock while enabling the IPTablerules (can be verified from the stMgrlog)

CSCvt89709

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Click recover for the selectedVM again under the standalonemode This will move the VM torecovered state

When the user recovers a VM protectedin a group it along with other VMs in thegroup move to recovering state instandalonemode The selectedVM shouldmove to recovered state

CSCvs75553

35(2e)Clean up the fail over cluster andthen recreate the fail over clusterNo need to touch the HX storagecluster

Hyper-V When replicated DC was usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2d)40(1a)

Keep cluster utilization as low aspossible To avoid read-only statekeep below 92

HyperFlex clusters storage utilization isabove 92 usable capacity

CSCvr20922

Release Notes for Cisco HX Data Platform Release 4050

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1b)NAAll the disks in the node got locked afterrebooting the node

We successfully unlocked usingsed-clientsh -U command but wantedto test with another reboot and driveslocked again

CSCvs74286

35(2g)To recover the VM copy over thedata disks and attach them to thenew VM

After enable encryption HX connectencryption status showing Caution butall of the disks enable encryptionsuccessful

-Self Encrypting Drives Service isrunning on all of the nodes

-There is no error message of encryption

- USB0 interface is up

-All SED disks showing supported 1enabled 1 locked 0

CSCvs93245

40(1b)Enable kernelpanic_on_oops as apersistent configuration This willcause the node to panic and rebootimmediately

A cluster node running HX release40(1b) may hang in the Linux kernelThis is classified as an oops and adeviation from the expected behavior

CSCvs54285

Release Notes for Cisco HX Data Platform Release 4051

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Enabling K8iscsi stack in Hx clusterCSCvs41324

Release Notes for Cisco HX Data Platform Release 4052

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Steps to enable K8 on HX cluster

Perform following steps on ALLthe controller VMs

1 Updateetcinitscvmclientconf toenable tune iscsiEnable totrue

sed -iesiscsiEnable=falseiscsiEnable=trueetcinitscvmclientconf

2 Run following initctlcommand to reloadconfiguration

initctl reload-configuration

3 Restart scvmclient process

stop scvmclient startscvmclient

4 Run following initctlcommand to mount SYSTEMDS

initctl emit --no-waitsystem-datastore-created

5 Verify that iscsiEnable tuneis set to true using thefollowing command

ps -eaf | grep scvmclient |grep -v grep

root 6241 1 1 Dec06 014007optspringpathstorfs-corescvmclient-T logEnabled=false -TlogSyslogEnabled=true -TlogEchoToScreen=false -TstatLoggingToFile=false -TstatLoggingToSyslog=true-TlogDir=varlogspringpath-TnfsBackendServerList=1010748100-T iscsiEnable=true -TiscsiUseAsync=true -T

Release Notes for Cisco HX Data Platform Release 4053

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

iscsiConfigFilePath=nfsSYSTEMistgtconfps -eaf | grep scvmclient |grep -v grep

35(2e)Following procedure can be usedto check NFS Queue Depth andincrease if needed

rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth256 lt- Low value[rootHXESXI1] vsish -e setvmkModulesnfsclientmnt[DATASTORE]propertiesmaxQDepth 1024[rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth1024 lt- Optimalvalue

This requires an ESXi host rebootto take effect

Please place the node in HyperflexMaintenanceMode and gracefullyreboot the node for the changes tobe applied

HyperFlex Datastore NFS Queue Depthshows as 256 which can lead toperformance (including latency) issues

CSCvr83056

40(1b)This is cosmetic The error will goaway once the cluster returns tohealthy status

When a single node is offline in clusterstcli cluster storage-summary shows twonodes unavailable rootSCVM~ stclicluster storage-summary

CSCvs47735

35(2g)Modify both the root and adminpassword for the SCVMs to be thesame

stcli security password set -uadmin

Cluster expansion fails at Config Installerstage when the root and admin passwordfor the storage controller (SCVM) aredifferent

CSCvs69317

35(2d)NAAfter successfully changing the DNSserver on the HX controller we still cansee the original DNS entry that was addedduring the deployment

CSCvs69154

Release Notes for Cisco HX Data Platform Release 4054

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Customers are strongly encouragedto work with TAC in order toidentify and remediate this issue

1 Change getcluster from 2m to8m inoptspringpathstorfs-mgmtstMgr-10confapplicationconfon all nodes

2 Restart stMgr on all nodes

You may see this error message after afailed upgrade or other task such asattempting to enter a node into HXmaintenance mode

CSCvs53555

40(2a)Run the following command oneach storage controller VM as root

grep -i monitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp sed -iesmonitor_dns_serverstruemonitor_dns_serversfalseoptspringpathhx-diag-toolswatchdog_configjsonampamp grep -imonitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp restart watchdog

Receive the following alertevent in HXConnect HX Controller VMHOSTNAME one or more configuredDNS servers not responding

CSCvt13947

35(2b)Delete any empty (Size 0) log filesunder varlogzookeeperversion-2ORvarlogzookeeperstandaloneversion-2

Zookeeper fails to start while Exhibitor isrunning however echo srvr returnsnothing

CSCvs21562

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

40(4e)1 De-commission the server

2 Power drain the server -REMOVE BOTH POWERCORDS ON THE BACK OFTHE SERVER FOR 10SECONDS THENREINSERTPOWERCORDS

3 Re-commission the server

M2 boot disk is missing from serverinventory after upgrade to server firmware40(4e) As a result server fails to boot toOS installed in the M2 disk The issuepersists after re-acknowledgement as wellas de-commission andre-acknowledgement of the server

CSCvs02466

Release Notes for Cisco HX Data Platform Release 4055

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Re-start the flusher manuallyMany disks taken offline by a fault ormanually resulting in ENOSPC oncluster

CSCvq32721

35(2e)Clean up fail over cluster and thenrecreate the fail over cluster Thereis no need to touch the HX storagecluster

[Hyper-V]When replicated DCwas usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2b)Manually install upgradescvmclient VIB andmake sure thatit matches with HXDP version

Upgrade from release HX 35(2a) to35(2x) 4X does not upgrade scvmclientRecovery point may not work

CSCvq54992

Management

30(1b)To recover the VM copy over thedata disks and attach them to thenew VM

[HyperV]VMshows up onmultiple nodesCSCvj22992

35(1a)1 Do not change MTU at ESXlevel on running cluster

2 Reset back to the original value

ChangingMTU from 9000 to 1500 on oneESX node causes storfs process to restarton all nodes

CSCvm99150

Release Notes for Cisco HX Data Platform Release 4056

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)After the installation (for newdeployments) or after an upgradeto 401a (for existingdeployments) perform thefollowing steps on all ControllerVMs

1 Edit the following tune files (onall Controller VMs)

optspringpathconfiglfftunes

optspringpathconfigvsi_16tbtunes

2 SetcleanerEnableSegSummaryCleaning

to false

3 After editing the above tunefiles

a SSH in to all the ControllerVMs

b Run storfstool -- -Z

c Run the below command andcheck the values Tune valueshould be true

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=true

d Type below commands andapplied the tunes changesdynamically

echo false gttmpstprocfssystemtunecleanerEnableSegSummaryCleaning

e Verify that tune values arechanged Run below commandValue should be false

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=false

f umount tmpstprocfs

A cluster with 8TB or 12TB disk drivesmay experience IO stalls for severalminutes after another node fails in thecluster

CSCvp23718

Release Notes for Cisco HX Data Platform Release 4057

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)1 Update the conf file manuallyto change to the link local addressto

2 Restart scvmclient on thecontroller and ESX

HX 35(2b) FlexVol to HX 40 CSIupgrade does not work FlexVol continuesto work

CSCvp26319

35(2a)Disable rebalance until node healsthen reenable rebalance

7-node cluster healing very slowlybecause rebalance change AMS reqs areblocking healing process

CSCvp62512

35(2a)NAWith moderate cluster storage capacityusage under site failover cluster clustermay experience an all paths (APD) downcondition

CSCvq49412

40(1a)NAThis product includes Third-partySoftware that is affected by thevulnerabilities identified by the followingCommon Vulnerability and Exposures(CVE) IDs

CVE-2016-0762 CVE-2016-6797CVE-2016-6816 CVE-2016-8735CVE-2017-5647 CVE-2017-12615CVE-2017-12616 CVE-2017-12617CVE-2017-7674 CVE-2018-1304CVE-2018-8014 CVE-2018-1336CVE-2018-8034 CVE-2018-11784CVE-2019-0232

CSCvp36364

35(2d)NAstcli cluster info command needs toprovide UCSM VIP address Currently itshows ucsm-hostcom

CSCvq11456

35(2a)No workarounds currently Thestale entries do not affect the dayto day operations of HyperFlex

HyperFlex upgrade validation failedbecause cluster has extraneous stNodes ininternal database

CSCvq32530

35(2a)NAWhen not in maintenancemode all drivesappear in HX Connect and get updated asexpected

When in maintenance mode extra drivesare shown in HX Connect with no slotnumbers

CSCvq42378

35(2b)Contact TAC to cleanup duplicatenode

Nodes duplicated in HX Connect aftercluster recreate

CSCvq61825

Release Notes for Cisco HX Data Platform Release 4058

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)NACurrently HyperFlex installs on Hyper-Vdo not contain the stcli security whitelistcommandset

CSCvq66245

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

Release Notes for Cisco HX Data Platform Release 4059

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)[Hyper-V] Converged node expansionfails for incoming node with VIC 14xx

CSCvq39523

Release Notes for Cisco HX Data Platform Release 4060

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

1 Set Jumbo frame inWindowsHypervisor for incomingnode

Get-NetAdapter storage-data-a|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

Get-NetAdapter storage-data-b|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

2 Verify all interfaces are setcorrectly to support Jumbo frames

Get-NetAdapter storage |Get-NetAdapterAdvancedProperty| RegistryKeyword -matchjumbo | ft -auto

You should receive the followingmessage

Name DisplayName DisplayValue

RegistryKeyword RegistryValue

---- ----------- ------------

--------------- -------------

vswitch-hx-storage-data Jumbo

Packet 9014 Bytes

JumboPacket 9014

storage-data-b Jumbo Packet

Bytes 9014 JumboPacket

9014

storage-data-a Jumbo Packet

Bytes 9014 JumboPacket

9014

3 Reboot only the incoming nodeor the node being expanded

4 Retry Cluster Expansion in

Release Notes for Cisco HX Data Platform Release 4061

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

In order to install or complete a node replacement onCisco HyperFlex customers need to download an HXInstaller OVA (Open Virtual Appliance) file in order todeploy a stretched cluster customers additionally needto download a Witness OVA All of the code posted onCCO prior to the posting of release HX 35(2g) wasdiscovered to have expired certificates as of 112619Cisco has re-signed and re-posted OVA files associatedwith HX releases 35(2e) 352(f) 352(g) 40(1a) and40(1b) with updated certificates For other releasesattempts to deploy an OVF template with an expiredOVA will fail with the following error message ldquoTheOVF package is signed with an invalid certificaterdquo

Conditions

If customers are deploying HX 35(2e) 352(f) 352(g)40(1a) or 40(1b) Cisco has re-signed and re-postedOVA files and customers will not experience the problemif they use the patched OVA files Look for a ldquop1rdquo suffixin the OVA filenames which indicates that OVA filehas been fixed

File Name Examples

HX 40(1a) patched OVA file for Cisco HyperFlex DataPlatform Installer for VMware ESXi

Cisco-HX-Data-Platform-Installer-v401a-33028p1-esxova

Cisco HyperFlex Data Platform Stretched ClusterWitness

HyperFlex-Witness-104p1ova

Customers using the OVA files for other HX releasesrefer to the following workaround

Workaround

There are two options to move forward after failing todeploy with an OVA file that is affected (applies to theinstaller and witness OVA files)

Option A - Remove the local manifest file

The manifest file can be deleted so vCenter does notcheck the validity of the certificate

1 Download and extract the OVA file to a localdirectory

2 Remove the mf file

3 Add the remaining files to a new archive and changethe file extension from tar to ova

Release Notes for Cisco HX Data Platform Release 4046

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4047

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1a)30(1d)Cluster instability when disks of different sector sizeplaced in HX node

CSCvk17250

40(1a)

35(2c)

35(1a)When logged in using a local HX user account insteadof a Virtual Center account an error message appearsintermittently indicating Virtual Centerunreachable or Resource informationcannot be updated when VC is reachable

CSCvo36198

40(1a)

35(2b)

30(1d)HXDP does not work with EMC RecoverPoint - needsto support VMware API (FSS-Readdir)

CSCvk38003

Release Notes for Cisco HX Data Platform Release 4048

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)

35(2a)

30(1i)

25(1c)

25(1d)

Zookeeper (Exhibitor) process on HyperFlex StorageController VMs may fail to respond to requests whenvarzookeeper has low or no free space

CSCvm90352

40(1a)26(1c)Use Asynchronous consolidation for HX SnapshotsCSCvn02151

40(1a)35(2b)HX Quiesced Snapshot for Backup VendorsCSCvo90713

40(1a)35(2a)The cluster creationcluster expansion workflow stopswith the following error message at the validation step

FIRMWARE-Check UCSC-SAS-M5HD

FIRMWARE-Check UCSC-SAS-M5HD Required00000029000000320000003500000050 Found00000058

Action Needed Update the Controller Firmware toRequired Version

CSCvn17787

35(2b)30(1d)Kernel migration fails when a local datastore is searchedfor in the ESXi host

CSCvn73127

35(2a)26(1b)A node shuts down when two disks are replaced (acaching disk and another capacity disk) where thecapacity disk is inserted first and then the caching diskis inserted

CSCvk46364

Hyper-V

40(1a)35(2a)Cluster expansion may fail with an error code 500 -operation timed out

CSCvn28721

35(2b)35(2a)During upgrade remove the VLAN on team created foruser vSwitch During fresh installation only one VLANtag is set to the vSwitch and team although multipleVLANs were entered

CSCvn54300

35(2b)35(2a)While upgrading a Hyper-V cluster on account of a rarerace condition between the stUpgradeService andZookeeper servers the upgrade orchestration throws anupgrade validation error and the upgrade process isaborted

CSCvn60486

Release Notes for Cisco HX Data Platform Release 4049

Resolved Caveats in Release 40(1a)

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Startrestart EAM service invCenter

When EAM service is not runningvCenter registration is failing

CSCvt55712

35(2a)Contact Cisco TAC if you believeyou are seeing similar conditions

HyperFlex Stretch Cluster saw poorapplication performance due to writelatency cluster remained unhealthy andrebalance was stuck

This defect is currently being used forinvestigative purposes

CSCvt45344

35(2g)Remove log files fromvarlogspringpath folder when thelog files are not getting rotatedStop the relevant service removelarge log files fromvarlogspringpath folder andrestart the service

varstv folder in Controller VM maybecome full

CSCvt36374

40(2b)Restarting the stMgr will initializeit again

In a Cluster configured with DR networkStMgr may not get initialized and stuckin a deadlock while enabling the IPTablerules (can be verified from the stMgrlog)

CSCvt89709

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Click recover for the selectedVM again under the standalonemode This will move the VM torecovered state

When the user recovers a VM protectedin a group it along with other VMs in thegroup move to recovering state instandalonemode The selectedVM shouldmove to recovered state

CSCvs75553

35(2e)Clean up the fail over cluster andthen recreate the fail over clusterNo need to touch the HX storagecluster

Hyper-V When replicated DC was usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2d)40(1a)

Keep cluster utilization as low aspossible To avoid read-only statekeep below 92

HyperFlex clusters storage utilization isabove 92 usable capacity

CSCvr20922

Release Notes for Cisco HX Data Platform Release 4050

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1b)NAAll the disks in the node got locked afterrebooting the node

We successfully unlocked usingsed-clientsh -U command but wantedto test with another reboot and driveslocked again

CSCvs74286

35(2g)To recover the VM copy over thedata disks and attach them to thenew VM

After enable encryption HX connectencryption status showing Caution butall of the disks enable encryptionsuccessful

-Self Encrypting Drives Service isrunning on all of the nodes

-There is no error message of encryption

- USB0 interface is up

-All SED disks showing supported 1enabled 1 locked 0

CSCvs93245

40(1b)Enable kernelpanic_on_oops as apersistent configuration This willcause the node to panic and rebootimmediately

A cluster node running HX release40(1b) may hang in the Linux kernelThis is classified as an oops and adeviation from the expected behavior

CSCvs54285

Release Notes for Cisco HX Data Platform Release 4051

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Enabling K8iscsi stack in Hx clusterCSCvs41324

Release Notes for Cisco HX Data Platform Release 4052

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Steps to enable K8 on HX cluster

Perform following steps on ALLthe controller VMs

1 Updateetcinitscvmclientconf toenable tune iscsiEnable totrue

sed -iesiscsiEnable=falseiscsiEnable=trueetcinitscvmclientconf

2 Run following initctlcommand to reloadconfiguration

initctl reload-configuration

3 Restart scvmclient process

stop scvmclient startscvmclient

4 Run following initctlcommand to mount SYSTEMDS

initctl emit --no-waitsystem-datastore-created

5 Verify that iscsiEnable tuneis set to true using thefollowing command

ps -eaf | grep scvmclient |grep -v grep

root 6241 1 1 Dec06 014007optspringpathstorfs-corescvmclient-T logEnabled=false -TlogSyslogEnabled=true -TlogEchoToScreen=false -TstatLoggingToFile=false -TstatLoggingToSyslog=true-TlogDir=varlogspringpath-TnfsBackendServerList=1010748100-T iscsiEnable=true -TiscsiUseAsync=true -T

Release Notes for Cisco HX Data Platform Release 4053

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

iscsiConfigFilePath=nfsSYSTEMistgtconfps -eaf | grep scvmclient |grep -v grep

35(2e)Following procedure can be usedto check NFS Queue Depth andincrease if needed

rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth256 lt- Low value[rootHXESXI1] vsish -e setvmkModulesnfsclientmnt[DATASTORE]propertiesmaxQDepth 1024[rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth1024 lt- Optimalvalue

This requires an ESXi host rebootto take effect

Please place the node in HyperflexMaintenanceMode and gracefullyreboot the node for the changes tobe applied

HyperFlex Datastore NFS Queue Depthshows as 256 which can lead toperformance (including latency) issues

CSCvr83056

40(1b)This is cosmetic The error will goaway once the cluster returns tohealthy status

When a single node is offline in clusterstcli cluster storage-summary shows twonodes unavailable rootSCVM~ stclicluster storage-summary

CSCvs47735

35(2g)Modify both the root and adminpassword for the SCVMs to be thesame

stcli security password set -uadmin

Cluster expansion fails at Config Installerstage when the root and admin passwordfor the storage controller (SCVM) aredifferent

CSCvs69317

35(2d)NAAfter successfully changing the DNSserver on the HX controller we still cansee the original DNS entry that was addedduring the deployment

CSCvs69154

Release Notes for Cisco HX Data Platform Release 4054

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Customers are strongly encouragedto work with TAC in order toidentify and remediate this issue

1 Change getcluster from 2m to8m inoptspringpathstorfs-mgmtstMgr-10confapplicationconfon all nodes

2 Restart stMgr on all nodes

You may see this error message after afailed upgrade or other task such asattempting to enter a node into HXmaintenance mode

CSCvs53555

40(2a)Run the following command oneach storage controller VM as root

grep -i monitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp sed -iesmonitor_dns_serverstruemonitor_dns_serversfalseoptspringpathhx-diag-toolswatchdog_configjsonampamp grep -imonitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp restart watchdog

Receive the following alertevent in HXConnect HX Controller VMHOSTNAME one or more configuredDNS servers not responding

CSCvt13947

35(2b)Delete any empty (Size 0) log filesunder varlogzookeeperversion-2ORvarlogzookeeperstandaloneversion-2

Zookeeper fails to start while Exhibitor isrunning however echo srvr returnsnothing

CSCvs21562

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

40(4e)1 De-commission the server

2 Power drain the server -REMOVE BOTH POWERCORDS ON THE BACK OFTHE SERVER FOR 10SECONDS THENREINSERTPOWERCORDS

3 Re-commission the server

M2 boot disk is missing from serverinventory after upgrade to server firmware40(4e) As a result server fails to boot toOS installed in the M2 disk The issuepersists after re-acknowledgement as wellas de-commission andre-acknowledgement of the server

CSCvs02466

Release Notes for Cisco HX Data Platform Release 4055

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Re-start the flusher manuallyMany disks taken offline by a fault ormanually resulting in ENOSPC oncluster

CSCvq32721

35(2e)Clean up fail over cluster and thenrecreate the fail over cluster Thereis no need to touch the HX storagecluster

[Hyper-V]When replicated DCwas usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2b)Manually install upgradescvmclient VIB andmake sure thatit matches with HXDP version

Upgrade from release HX 35(2a) to35(2x) 4X does not upgrade scvmclientRecovery point may not work

CSCvq54992

Management

30(1b)To recover the VM copy over thedata disks and attach them to thenew VM

[HyperV]VMshows up onmultiple nodesCSCvj22992

35(1a)1 Do not change MTU at ESXlevel on running cluster

2 Reset back to the original value

ChangingMTU from 9000 to 1500 on oneESX node causes storfs process to restarton all nodes

CSCvm99150

Release Notes for Cisco HX Data Platform Release 4056

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)After the installation (for newdeployments) or after an upgradeto 401a (for existingdeployments) perform thefollowing steps on all ControllerVMs

1 Edit the following tune files (onall Controller VMs)

optspringpathconfiglfftunes

optspringpathconfigvsi_16tbtunes

2 SetcleanerEnableSegSummaryCleaning

to false

3 After editing the above tunefiles

a SSH in to all the ControllerVMs

b Run storfstool -- -Z

c Run the below command andcheck the values Tune valueshould be true

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=true

d Type below commands andapplied the tunes changesdynamically

echo false gttmpstprocfssystemtunecleanerEnableSegSummaryCleaning

e Verify that tune values arechanged Run below commandValue should be false

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=false

f umount tmpstprocfs

A cluster with 8TB or 12TB disk drivesmay experience IO stalls for severalminutes after another node fails in thecluster

CSCvp23718

Release Notes for Cisco HX Data Platform Release 4057

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)1 Update the conf file manuallyto change to the link local addressto

2 Restart scvmclient on thecontroller and ESX

HX 35(2b) FlexVol to HX 40 CSIupgrade does not work FlexVol continuesto work

CSCvp26319

35(2a)Disable rebalance until node healsthen reenable rebalance

7-node cluster healing very slowlybecause rebalance change AMS reqs areblocking healing process

CSCvp62512

35(2a)NAWith moderate cluster storage capacityusage under site failover cluster clustermay experience an all paths (APD) downcondition

CSCvq49412

40(1a)NAThis product includes Third-partySoftware that is affected by thevulnerabilities identified by the followingCommon Vulnerability and Exposures(CVE) IDs

CVE-2016-0762 CVE-2016-6797CVE-2016-6816 CVE-2016-8735CVE-2017-5647 CVE-2017-12615CVE-2017-12616 CVE-2017-12617CVE-2017-7674 CVE-2018-1304CVE-2018-8014 CVE-2018-1336CVE-2018-8034 CVE-2018-11784CVE-2019-0232

CSCvp36364

35(2d)NAstcli cluster info command needs toprovide UCSM VIP address Currently itshows ucsm-hostcom

CSCvq11456

35(2a)No workarounds currently Thestale entries do not affect the dayto day operations of HyperFlex

HyperFlex upgrade validation failedbecause cluster has extraneous stNodes ininternal database

CSCvq32530

35(2a)NAWhen not in maintenancemode all drivesappear in HX Connect and get updated asexpected

When in maintenance mode extra drivesare shown in HX Connect with no slotnumbers

CSCvq42378

35(2b)Contact TAC to cleanup duplicatenode

Nodes duplicated in HX Connect aftercluster recreate

CSCvq61825

Release Notes for Cisco HX Data Platform Release 4058

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)NACurrently HyperFlex installs on Hyper-Vdo not contain the stcli security whitelistcommandset

CSCvq66245

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

Release Notes for Cisco HX Data Platform Release 4059

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)[Hyper-V] Converged node expansionfails for incoming node with VIC 14xx

CSCvq39523

Release Notes for Cisco HX Data Platform Release 4060

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

1 Set Jumbo frame inWindowsHypervisor for incomingnode

Get-NetAdapter storage-data-a|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

Get-NetAdapter storage-data-b|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

2 Verify all interfaces are setcorrectly to support Jumbo frames

Get-NetAdapter storage |Get-NetAdapterAdvancedProperty| RegistryKeyword -matchjumbo | ft -auto

You should receive the followingmessage

Name DisplayName DisplayValue

RegistryKeyword RegistryValue

---- ----------- ------------

--------------- -------------

vswitch-hx-storage-data Jumbo

Packet 9014 Bytes

JumboPacket 9014

storage-data-b Jumbo Packet

Bytes 9014 JumboPacket

9014

storage-data-a Jumbo Packet

Bytes 9014 JumboPacket

9014

3 Reboot only the incoming nodeor the node being expanded

4 Retry Cluster Expansion in

Release Notes for Cisco HX Data Platform Release 4061

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

Proceed to deploy that newly created OVA file usingldquoDeploy by OVF Templaterdquo in vCenter vCenter will

4

show the file as not having a certificate This isexpected and the deployment should continuewithoutissue

Option B - Remove the local manifest file

Manually deploy with ovftool ndash Use VMwares ovftoolto deploy the OVAwhile bypassing the certificate checkThe ovftool can be downloaded and run on customerscomputer The ovftool also comes pre-installed on HXController VMs This is helpful for node replacementsand cluster expansions

Release Notes for Cisco HX Data Platform Release 4047

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1a)30(1d)Cluster instability when disks of different sector sizeplaced in HX node

CSCvk17250

40(1a)

35(2c)

35(1a)When logged in using a local HX user account insteadof a Virtual Center account an error message appearsintermittently indicating Virtual Centerunreachable or Resource informationcannot be updated when VC is reachable

CSCvo36198

40(1a)

35(2b)

30(1d)HXDP does not work with EMC RecoverPoint - needsto support VMware API (FSS-Readdir)

CSCvk38003

Release Notes for Cisco HX Data Platform Release 4048

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)

35(2a)

30(1i)

25(1c)

25(1d)

Zookeeper (Exhibitor) process on HyperFlex StorageController VMs may fail to respond to requests whenvarzookeeper has low or no free space

CSCvm90352

40(1a)26(1c)Use Asynchronous consolidation for HX SnapshotsCSCvn02151

40(1a)35(2b)HX Quiesced Snapshot for Backup VendorsCSCvo90713

40(1a)35(2a)The cluster creationcluster expansion workflow stopswith the following error message at the validation step

FIRMWARE-Check UCSC-SAS-M5HD

FIRMWARE-Check UCSC-SAS-M5HD Required00000029000000320000003500000050 Found00000058

Action Needed Update the Controller Firmware toRequired Version

CSCvn17787

35(2b)30(1d)Kernel migration fails when a local datastore is searchedfor in the ESXi host

CSCvn73127

35(2a)26(1b)A node shuts down when two disks are replaced (acaching disk and another capacity disk) where thecapacity disk is inserted first and then the caching diskis inserted

CSCvk46364

Hyper-V

40(1a)35(2a)Cluster expansion may fail with an error code 500 -operation timed out

CSCvn28721

35(2b)35(2a)During upgrade remove the VLAN on team created foruser vSwitch During fresh installation only one VLANtag is set to the vSwitch and team although multipleVLANs were entered

CSCvn54300

35(2b)35(2a)While upgrading a Hyper-V cluster on account of a rarerace condition between the stUpgradeService andZookeeper servers the upgrade orchestration throws anupgrade validation error and the upgrade process isaborted

CSCvn60486

Release Notes for Cisco HX Data Platform Release 4049

Resolved Caveats in Release 40(1a)

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Startrestart EAM service invCenter

When EAM service is not runningvCenter registration is failing

CSCvt55712

35(2a)Contact Cisco TAC if you believeyou are seeing similar conditions

HyperFlex Stretch Cluster saw poorapplication performance due to writelatency cluster remained unhealthy andrebalance was stuck

This defect is currently being used forinvestigative purposes

CSCvt45344

35(2g)Remove log files fromvarlogspringpath folder when thelog files are not getting rotatedStop the relevant service removelarge log files fromvarlogspringpath folder andrestart the service

varstv folder in Controller VM maybecome full

CSCvt36374

40(2b)Restarting the stMgr will initializeit again

In a Cluster configured with DR networkStMgr may not get initialized and stuckin a deadlock while enabling the IPTablerules (can be verified from the stMgrlog)

CSCvt89709

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Click recover for the selectedVM again under the standalonemode This will move the VM torecovered state

When the user recovers a VM protectedin a group it along with other VMs in thegroup move to recovering state instandalonemode The selectedVM shouldmove to recovered state

CSCvs75553

35(2e)Clean up the fail over cluster andthen recreate the fail over clusterNo need to touch the HX storagecluster

Hyper-V When replicated DC was usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2d)40(1a)

Keep cluster utilization as low aspossible To avoid read-only statekeep below 92

HyperFlex clusters storage utilization isabove 92 usable capacity

CSCvr20922

Release Notes for Cisco HX Data Platform Release 4050

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1b)NAAll the disks in the node got locked afterrebooting the node

We successfully unlocked usingsed-clientsh -U command but wantedto test with another reboot and driveslocked again

CSCvs74286

35(2g)To recover the VM copy over thedata disks and attach them to thenew VM

After enable encryption HX connectencryption status showing Caution butall of the disks enable encryptionsuccessful

-Self Encrypting Drives Service isrunning on all of the nodes

-There is no error message of encryption

- USB0 interface is up

-All SED disks showing supported 1enabled 1 locked 0

CSCvs93245

40(1b)Enable kernelpanic_on_oops as apersistent configuration This willcause the node to panic and rebootimmediately

A cluster node running HX release40(1b) may hang in the Linux kernelThis is classified as an oops and adeviation from the expected behavior

CSCvs54285

Release Notes for Cisco HX Data Platform Release 4051

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Enabling K8iscsi stack in Hx clusterCSCvs41324

Release Notes for Cisco HX Data Platform Release 4052

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Steps to enable K8 on HX cluster

Perform following steps on ALLthe controller VMs

1 Updateetcinitscvmclientconf toenable tune iscsiEnable totrue

sed -iesiscsiEnable=falseiscsiEnable=trueetcinitscvmclientconf

2 Run following initctlcommand to reloadconfiguration

initctl reload-configuration

3 Restart scvmclient process

stop scvmclient startscvmclient

4 Run following initctlcommand to mount SYSTEMDS

initctl emit --no-waitsystem-datastore-created

5 Verify that iscsiEnable tuneis set to true using thefollowing command

ps -eaf | grep scvmclient |grep -v grep

root 6241 1 1 Dec06 014007optspringpathstorfs-corescvmclient-T logEnabled=false -TlogSyslogEnabled=true -TlogEchoToScreen=false -TstatLoggingToFile=false -TstatLoggingToSyslog=true-TlogDir=varlogspringpath-TnfsBackendServerList=1010748100-T iscsiEnable=true -TiscsiUseAsync=true -T

Release Notes for Cisco HX Data Platform Release 4053

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

iscsiConfigFilePath=nfsSYSTEMistgtconfps -eaf | grep scvmclient |grep -v grep

35(2e)Following procedure can be usedto check NFS Queue Depth andincrease if needed

rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth256 lt- Low value[rootHXESXI1] vsish -e setvmkModulesnfsclientmnt[DATASTORE]propertiesmaxQDepth 1024[rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth1024 lt- Optimalvalue

This requires an ESXi host rebootto take effect

Please place the node in HyperflexMaintenanceMode and gracefullyreboot the node for the changes tobe applied

HyperFlex Datastore NFS Queue Depthshows as 256 which can lead toperformance (including latency) issues

CSCvr83056

40(1b)This is cosmetic The error will goaway once the cluster returns tohealthy status

When a single node is offline in clusterstcli cluster storage-summary shows twonodes unavailable rootSCVM~ stclicluster storage-summary

CSCvs47735

35(2g)Modify both the root and adminpassword for the SCVMs to be thesame

stcli security password set -uadmin

Cluster expansion fails at Config Installerstage when the root and admin passwordfor the storage controller (SCVM) aredifferent

CSCvs69317

35(2d)NAAfter successfully changing the DNSserver on the HX controller we still cansee the original DNS entry that was addedduring the deployment

CSCvs69154

Release Notes for Cisco HX Data Platform Release 4054

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Customers are strongly encouragedto work with TAC in order toidentify and remediate this issue

1 Change getcluster from 2m to8m inoptspringpathstorfs-mgmtstMgr-10confapplicationconfon all nodes

2 Restart stMgr on all nodes

You may see this error message after afailed upgrade or other task such asattempting to enter a node into HXmaintenance mode

CSCvs53555

40(2a)Run the following command oneach storage controller VM as root

grep -i monitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp sed -iesmonitor_dns_serverstruemonitor_dns_serversfalseoptspringpathhx-diag-toolswatchdog_configjsonampamp grep -imonitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp restart watchdog

Receive the following alertevent in HXConnect HX Controller VMHOSTNAME one or more configuredDNS servers not responding

CSCvt13947

35(2b)Delete any empty (Size 0) log filesunder varlogzookeeperversion-2ORvarlogzookeeperstandaloneversion-2

Zookeeper fails to start while Exhibitor isrunning however echo srvr returnsnothing

CSCvs21562

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

40(4e)1 De-commission the server

2 Power drain the server -REMOVE BOTH POWERCORDS ON THE BACK OFTHE SERVER FOR 10SECONDS THENREINSERTPOWERCORDS

3 Re-commission the server

M2 boot disk is missing from serverinventory after upgrade to server firmware40(4e) As a result server fails to boot toOS installed in the M2 disk The issuepersists after re-acknowledgement as wellas de-commission andre-acknowledgement of the server

CSCvs02466

Release Notes for Cisco HX Data Platform Release 4055

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Re-start the flusher manuallyMany disks taken offline by a fault ormanually resulting in ENOSPC oncluster

CSCvq32721

35(2e)Clean up fail over cluster and thenrecreate the fail over cluster Thereis no need to touch the HX storagecluster

[Hyper-V]When replicated DCwas usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2b)Manually install upgradescvmclient VIB andmake sure thatit matches with HXDP version

Upgrade from release HX 35(2a) to35(2x) 4X does not upgrade scvmclientRecovery point may not work

CSCvq54992

Management

30(1b)To recover the VM copy over thedata disks and attach them to thenew VM

[HyperV]VMshows up onmultiple nodesCSCvj22992

35(1a)1 Do not change MTU at ESXlevel on running cluster

2 Reset back to the original value

ChangingMTU from 9000 to 1500 on oneESX node causes storfs process to restarton all nodes

CSCvm99150

Release Notes for Cisco HX Data Platform Release 4056

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)After the installation (for newdeployments) or after an upgradeto 401a (for existingdeployments) perform thefollowing steps on all ControllerVMs

1 Edit the following tune files (onall Controller VMs)

optspringpathconfiglfftunes

optspringpathconfigvsi_16tbtunes

2 SetcleanerEnableSegSummaryCleaning

to false

3 After editing the above tunefiles

a SSH in to all the ControllerVMs

b Run storfstool -- -Z

c Run the below command andcheck the values Tune valueshould be true

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=true

d Type below commands andapplied the tunes changesdynamically

echo false gttmpstprocfssystemtunecleanerEnableSegSummaryCleaning

e Verify that tune values arechanged Run below commandValue should be false

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=false

f umount tmpstprocfs

A cluster with 8TB or 12TB disk drivesmay experience IO stalls for severalminutes after another node fails in thecluster

CSCvp23718

Release Notes for Cisco HX Data Platform Release 4057

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)1 Update the conf file manuallyto change to the link local addressto

2 Restart scvmclient on thecontroller and ESX

HX 35(2b) FlexVol to HX 40 CSIupgrade does not work FlexVol continuesto work

CSCvp26319

35(2a)Disable rebalance until node healsthen reenable rebalance

7-node cluster healing very slowlybecause rebalance change AMS reqs areblocking healing process

CSCvp62512

35(2a)NAWith moderate cluster storage capacityusage under site failover cluster clustermay experience an all paths (APD) downcondition

CSCvq49412

40(1a)NAThis product includes Third-partySoftware that is affected by thevulnerabilities identified by the followingCommon Vulnerability and Exposures(CVE) IDs

CVE-2016-0762 CVE-2016-6797CVE-2016-6816 CVE-2016-8735CVE-2017-5647 CVE-2017-12615CVE-2017-12616 CVE-2017-12617CVE-2017-7674 CVE-2018-1304CVE-2018-8014 CVE-2018-1336CVE-2018-8034 CVE-2018-11784CVE-2019-0232

CSCvp36364

35(2d)NAstcli cluster info command needs toprovide UCSM VIP address Currently itshows ucsm-hostcom

CSCvq11456

35(2a)No workarounds currently Thestale entries do not affect the dayto day operations of HyperFlex

HyperFlex upgrade validation failedbecause cluster has extraneous stNodes ininternal database

CSCvq32530

35(2a)NAWhen not in maintenancemode all drivesappear in HX Connect and get updated asexpected

When in maintenance mode extra drivesare shown in HX Connect with no slotnumbers

CSCvq42378

35(2b)Contact TAC to cleanup duplicatenode

Nodes duplicated in HX Connect aftercluster recreate

CSCvq61825

Release Notes for Cisco HX Data Platform Release 4058

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)NACurrently HyperFlex installs on Hyper-Vdo not contain the stcli security whitelistcommandset

CSCvq66245

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

Release Notes for Cisco HX Data Platform Release 4059

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)[Hyper-V] Converged node expansionfails for incoming node with VIC 14xx

CSCvq39523

Release Notes for Cisco HX Data Platform Release 4060

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

1 Set Jumbo frame inWindowsHypervisor for incomingnode

Get-NetAdapter storage-data-a|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

Get-NetAdapter storage-data-b|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

2 Verify all interfaces are setcorrectly to support Jumbo frames

Get-NetAdapter storage |Get-NetAdapterAdvancedProperty| RegistryKeyword -matchjumbo | ft -auto

You should receive the followingmessage

Name DisplayName DisplayValue

RegistryKeyword RegistryValue

---- ----------- ------------

--------------- -------------

vswitch-hx-storage-data Jumbo

Packet 9014 Bytes

JumboPacket 9014

storage-data-b Jumbo Packet

Bytes 9014 JumboPacket

9014

storage-data-a Jumbo Packet

Bytes 9014 JumboPacket

9014

3 Reboot only the incoming nodeor the node being expanded

4 Retry Cluster Expansion in

Release Notes for Cisco HX Data Platform Release 4061

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

1 Use ovftool to deploy the OVA file to a datastorewhile raising the --skipManifestcheck switch ForexamplerootSpringpathControllerABCDEFGH~ ovftool--skipManifestCheck -ds=datastorehttpltpath to

ovagtCisco-HX-Data-Platform-Installer-v352c-31725-esxovavirootltIP of management ESX

hostgt

2 TheOVA should be deployed and present in vCenteron the ESXi host previously specified

3 Power on the VM and console into it

4 Login to the VMwith the default usernamepasswordcombination of root Cisco123

5 Set the IP of the VM statically by issuing vietcnetworketh0interface

6 Change iface eth0 inet dhcp to iface eth0 inet staticEach of the following needs to be on their own lineand tab indentedaddress ltdesired ip address of installergt

netmask XXXX

gateway XXXX

ltescgt wq

7 After the file is reviewed and saved restart the VMThe VM should now boot with the desired IP address

8 The first login via the WebGUI (still using defaultusernamepassword combination) will have the userchange the password

9 After the password change the user can begin thedesired installexpandnode replacement activity

40(1a)30(1d)Cluster instability when disks of different sector sizeplaced in HX node

CSCvk17250

40(1a)

35(2c)

35(1a)When logged in using a local HX user account insteadof a Virtual Center account an error message appearsintermittently indicating Virtual Centerunreachable or Resource informationcannot be updated when VC is reachable

CSCvo36198

40(1a)

35(2b)

30(1d)HXDP does not work with EMC RecoverPoint - needsto support VMware API (FSS-Readdir)

CSCvk38003

Release Notes for Cisco HX Data Platform Release 4048

Resolved Caveats in Release 40(1a)

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)

35(2a)

30(1i)

25(1c)

25(1d)

Zookeeper (Exhibitor) process on HyperFlex StorageController VMs may fail to respond to requests whenvarzookeeper has low or no free space

CSCvm90352

40(1a)26(1c)Use Asynchronous consolidation for HX SnapshotsCSCvn02151

40(1a)35(2b)HX Quiesced Snapshot for Backup VendorsCSCvo90713

40(1a)35(2a)The cluster creationcluster expansion workflow stopswith the following error message at the validation step

FIRMWARE-Check UCSC-SAS-M5HD

FIRMWARE-Check UCSC-SAS-M5HD Required00000029000000320000003500000050 Found00000058

Action Needed Update the Controller Firmware toRequired Version

CSCvn17787

35(2b)30(1d)Kernel migration fails when a local datastore is searchedfor in the ESXi host

CSCvn73127

35(2a)26(1b)A node shuts down when two disks are replaced (acaching disk and another capacity disk) where thecapacity disk is inserted first and then the caching diskis inserted

CSCvk46364

Hyper-V

40(1a)35(2a)Cluster expansion may fail with an error code 500 -operation timed out

CSCvn28721

35(2b)35(2a)During upgrade remove the VLAN on team created foruser vSwitch During fresh installation only one VLANtag is set to the vSwitch and team although multipleVLANs were entered

CSCvn54300

35(2b)35(2a)While upgrading a Hyper-V cluster on account of a rarerace condition between the stUpgradeService andZookeeper servers the upgrade orchestration throws anupgrade validation error and the upgrade process isaborted

CSCvn60486

Release Notes for Cisco HX Data Platform Release 4049

Resolved Caveats in Release 40(1a)

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Startrestart EAM service invCenter

When EAM service is not runningvCenter registration is failing

CSCvt55712

35(2a)Contact Cisco TAC if you believeyou are seeing similar conditions

HyperFlex Stretch Cluster saw poorapplication performance due to writelatency cluster remained unhealthy andrebalance was stuck

This defect is currently being used forinvestigative purposes

CSCvt45344

35(2g)Remove log files fromvarlogspringpath folder when thelog files are not getting rotatedStop the relevant service removelarge log files fromvarlogspringpath folder andrestart the service

varstv folder in Controller VM maybecome full

CSCvt36374

40(2b)Restarting the stMgr will initializeit again

In a Cluster configured with DR networkStMgr may not get initialized and stuckin a deadlock while enabling the IPTablerules (can be verified from the stMgrlog)

CSCvt89709

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Click recover for the selectedVM again under the standalonemode This will move the VM torecovered state

When the user recovers a VM protectedin a group it along with other VMs in thegroup move to recovering state instandalonemode The selectedVM shouldmove to recovered state

CSCvs75553

35(2e)Clean up the fail over cluster andthen recreate the fail over clusterNo need to touch the HX storagecluster

Hyper-V When replicated DC was usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2d)40(1a)

Keep cluster utilization as low aspossible To avoid read-only statekeep below 92

HyperFlex clusters storage utilization isabove 92 usable capacity

CSCvr20922

Release Notes for Cisco HX Data Platform Release 4050

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1b)NAAll the disks in the node got locked afterrebooting the node

We successfully unlocked usingsed-clientsh -U command but wantedto test with another reboot and driveslocked again

CSCvs74286

35(2g)To recover the VM copy over thedata disks and attach them to thenew VM

After enable encryption HX connectencryption status showing Caution butall of the disks enable encryptionsuccessful

-Self Encrypting Drives Service isrunning on all of the nodes

-There is no error message of encryption

- USB0 interface is up

-All SED disks showing supported 1enabled 1 locked 0

CSCvs93245

40(1b)Enable kernelpanic_on_oops as apersistent configuration This willcause the node to panic and rebootimmediately

A cluster node running HX release40(1b) may hang in the Linux kernelThis is classified as an oops and adeviation from the expected behavior

CSCvs54285

Release Notes for Cisco HX Data Platform Release 4051

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Enabling K8iscsi stack in Hx clusterCSCvs41324

Release Notes for Cisco HX Data Platform Release 4052

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Steps to enable K8 on HX cluster

Perform following steps on ALLthe controller VMs

1 Updateetcinitscvmclientconf toenable tune iscsiEnable totrue

sed -iesiscsiEnable=falseiscsiEnable=trueetcinitscvmclientconf

2 Run following initctlcommand to reloadconfiguration

initctl reload-configuration

3 Restart scvmclient process

stop scvmclient startscvmclient

4 Run following initctlcommand to mount SYSTEMDS

initctl emit --no-waitsystem-datastore-created

5 Verify that iscsiEnable tuneis set to true using thefollowing command

ps -eaf | grep scvmclient |grep -v grep

root 6241 1 1 Dec06 014007optspringpathstorfs-corescvmclient-T logEnabled=false -TlogSyslogEnabled=true -TlogEchoToScreen=false -TstatLoggingToFile=false -TstatLoggingToSyslog=true-TlogDir=varlogspringpath-TnfsBackendServerList=1010748100-T iscsiEnable=true -TiscsiUseAsync=true -T

Release Notes for Cisco HX Data Platform Release 4053

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

iscsiConfigFilePath=nfsSYSTEMistgtconfps -eaf | grep scvmclient |grep -v grep

35(2e)Following procedure can be usedto check NFS Queue Depth andincrease if needed

rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth256 lt- Low value[rootHXESXI1] vsish -e setvmkModulesnfsclientmnt[DATASTORE]propertiesmaxQDepth 1024[rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth1024 lt- Optimalvalue

This requires an ESXi host rebootto take effect

Please place the node in HyperflexMaintenanceMode and gracefullyreboot the node for the changes tobe applied

HyperFlex Datastore NFS Queue Depthshows as 256 which can lead toperformance (including latency) issues

CSCvr83056

40(1b)This is cosmetic The error will goaway once the cluster returns tohealthy status

When a single node is offline in clusterstcli cluster storage-summary shows twonodes unavailable rootSCVM~ stclicluster storage-summary

CSCvs47735

35(2g)Modify both the root and adminpassword for the SCVMs to be thesame

stcli security password set -uadmin

Cluster expansion fails at Config Installerstage when the root and admin passwordfor the storage controller (SCVM) aredifferent

CSCvs69317

35(2d)NAAfter successfully changing the DNSserver on the HX controller we still cansee the original DNS entry that was addedduring the deployment

CSCvs69154

Release Notes for Cisco HX Data Platform Release 4054

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Customers are strongly encouragedto work with TAC in order toidentify and remediate this issue

1 Change getcluster from 2m to8m inoptspringpathstorfs-mgmtstMgr-10confapplicationconfon all nodes

2 Restart stMgr on all nodes

You may see this error message after afailed upgrade or other task such asattempting to enter a node into HXmaintenance mode

CSCvs53555

40(2a)Run the following command oneach storage controller VM as root

grep -i monitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp sed -iesmonitor_dns_serverstruemonitor_dns_serversfalseoptspringpathhx-diag-toolswatchdog_configjsonampamp grep -imonitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp restart watchdog

Receive the following alertevent in HXConnect HX Controller VMHOSTNAME one or more configuredDNS servers not responding

CSCvt13947

35(2b)Delete any empty (Size 0) log filesunder varlogzookeeperversion-2ORvarlogzookeeperstandaloneversion-2

Zookeeper fails to start while Exhibitor isrunning however echo srvr returnsnothing

CSCvs21562

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

40(4e)1 De-commission the server

2 Power drain the server -REMOVE BOTH POWERCORDS ON THE BACK OFTHE SERVER FOR 10SECONDS THENREINSERTPOWERCORDS

3 Re-commission the server

M2 boot disk is missing from serverinventory after upgrade to server firmware40(4e) As a result server fails to boot toOS installed in the M2 disk The issuepersists after re-acknowledgement as wellas de-commission andre-acknowledgement of the server

CSCvs02466

Release Notes for Cisco HX Data Platform Release 4055

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Re-start the flusher manuallyMany disks taken offline by a fault ormanually resulting in ENOSPC oncluster

CSCvq32721

35(2e)Clean up fail over cluster and thenrecreate the fail over cluster Thereis no need to touch the HX storagecluster

[Hyper-V]When replicated DCwas usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2b)Manually install upgradescvmclient VIB andmake sure thatit matches with HXDP version

Upgrade from release HX 35(2a) to35(2x) 4X does not upgrade scvmclientRecovery point may not work

CSCvq54992

Management

30(1b)To recover the VM copy over thedata disks and attach them to thenew VM

[HyperV]VMshows up onmultiple nodesCSCvj22992

35(1a)1 Do not change MTU at ESXlevel on running cluster

2 Reset back to the original value

ChangingMTU from 9000 to 1500 on oneESX node causes storfs process to restarton all nodes

CSCvm99150

Release Notes for Cisco HX Data Platform Release 4056

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)After the installation (for newdeployments) or after an upgradeto 401a (for existingdeployments) perform thefollowing steps on all ControllerVMs

1 Edit the following tune files (onall Controller VMs)

optspringpathconfiglfftunes

optspringpathconfigvsi_16tbtunes

2 SetcleanerEnableSegSummaryCleaning

to false

3 After editing the above tunefiles

a SSH in to all the ControllerVMs

b Run storfstool -- -Z

c Run the below command andcheck the values Tune valueshould be true

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=true

d Type below commands andapplied the tunes changesdynamically

echo false gttmpstprocfssystemtunecleanerEnableSegSummaryCleaning

e Verify that tune values arechanged Run below commandValue should be false

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=false

f umount tmpstprocfs

A cluster with 8TB or 12TB disk drivesmay experience IO stalls for severalminutes after another node fails in thecluster

CSCvp23718

Release Notes for Cisco HX Data Platform Release 4057

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)1 Update the conf file manuallyto change to the link local addressto

2 Restart scvmclient on thecontroller and ESX

HX 35(2b) FlexVol to HX 40 CSIupgrade does not work FlexVol continuesto work

CSCvp26319

35(2a)Disable rebalance until node healsthen reenable rebalance

7-node cluster healing very slowlybecause rebalance change AMS reqs areblocking healing process

CSCvp62512

35(2a)NAWith moderate cluster storage capacityusage under site failover cluster clustermay experience an all paths (APD) downcondition

CSCvq49412

40(1a)NAThis product includes Third-partySoftware that is affected by thevulnerabilities identified by the followingCommon Vulnerability and Exposures(CVE) IDs

CVE-2016-0762 CVE-2016-6797CVE-2016-6816 CVE-2016-8735CVE-2017-5647 CVE-2017-12615CVE-2017-12616 CVE-2017-12617CVE-2017-7674 CVE-2018-1304CVE-2018-8014 CVE-2018-1336CVE-2018-8034 CVE-2018-11784CVE-2019-0232

CSCvp36364

35(2d)NAstcli cluster info command needs toprovide UCSM VIP address Currently itshows ucsm-hostcom

CSCvq11456

35(2a)No workarounds currently Thestale entries do not affect the dayto day operations of HyperFlex

HyperFlex upgrade validation failedbecause cluster has extraneous stNodes ininternal database

CSCvq32530

35(2a)NAWhen not in maintenancemode all drivesappear in HX Connect and get updated asexpected

When in maintenance mode extra drivesare shown in HX Connect with no slotnumbers

CSCvq42378

35(2b)Contact TAC to cleanup duplicatenode

Nodes duplicated in HX Connect aftercluster recreate

CSCvq61825

Release Notes for Cisco HX Data Platform Release 4058

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)NACurrently HyperFlex installs on Hyper-Vdo not contain the stcli security whitelistcommandset

CSCvq66245

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

Release Notes for Cisco HX Data Platform Release 4059

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)[Hyper-V] Converged node expansionfails for incoming node with VIC 14xx

CSCvq39523

Release Notes for Cisco HX Data Platform Release 4060

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

1 Set Jumbo frame inWindowsHypervisor for incomingnode

Get-NetAdapter storage-data-a|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

Get-NetAdapter storage-data-b|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

2 Verify all interfaces are setcorrectly to support Jumbo frames

Get-NetAdapter storage |Get-NetAdapterAdvancedProperty| RegistryKeyword -matchjumbo | ft -auto

You should receive the followingmessage

Name DisplayName DisplayValue

RegistryKeyword RegistryValue

---- ----------- ------------

--------------- -------------

vswitch-hx-storage-data Jumbo

Packet 9014 Bytes

JumboPacket 9014

storage-data-b Jumbo Packet

Bytes 9014 JumboPacket

9014

storage-data-a Jumbo Packet

Bytes 9014 JumboPacket

9014

3 Reboot only the incoming nodeor the node being expanded

4 Retry Cluster Expansion in

Release Notes for Cisco HX Data Platform Release 4061

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

Resolved inRelease

First ReleaseAffected

SymptomDefect ID

40(1a)

35(2a)

30(1i)

25(1c)

25(1d)

Zookeeper (Exhibitor) process on HyperFlex StorageController VMs may fail to respond to requests whenvarzookeeper has low or no free space

CSCvm90352

40(1a)26(1c)Use Asynchronous consolidation for HX SnapshotsCSCvn02151

40(1a)35(2b)HX Quiesced Snapshot for Backup VendorsCSCvo90713

40(1a)35(2a)The cluster creationcluster expansion workflow stopswith the following error message at the validation step

FIRMWARE-Check UCSC-SAS-M5HD

FIRMWARE-Check UCSC-SAS-M5HD Required00000029000000320000003500000050 Found00000058

Action Needed Update the Controller Firmware toRequired Version

CSCvn17787

35(2b)30(1d)Kernel migration fails when a local datastore is searchedfor in the ESXi host

CSCvn73127

35(2a)26(1b)A node shuts down when two disks are replaced (acaching disk and another capacity disk) where thecapacity disk is inserted first and then the caching diskis inserted

CSCvk46364

Hyper-V

40(1a)35(2a)Cluster expansion may fail with an error code 500 -operation timed out

CSCvn28721

35(2b)35(2a)During upgrade remove the VLAN on team created foruser vSwitch During fresh installation only one VLANtag is set to the vSwitch and team although multipleVLANs were entered

CSCvn54300

35(2b)35(2a)While upgrading a Hyper-V cluster on account of a rarerace condition between the stUpgradeService andZookeeper servers the upgrade orchestration throws anupgrade validation error and the upgrade process isaborted

CSCvn60486

Release Notes for Cisco HX Data Platform Release 4049

Resolved Caveats in Release 40(1a)

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Startrestart EAM service invCenter

When EAM service is not runningvCenter registration is failing

CSCvt55712

35(2a)Contact Cisco TAC if you believeyou are seeing similar conditions

HyperFlex Stretch Cluster saw poorapplication performance due to writelatency cluster remained unhealthy andrebalance was stuck

This defect is currently being used forinvestigative purposes

CSCvt45344

35(2g)Remove log files fromvarlogspringpath folder when thelog files are not getting rotatedStop the relevant service removelarge log files fromvarlogspringpath folder andrestart the service

varstv folder in Controller VM maybecome full

CSCvt36374

40(2b)Restarting the stMgr will initializeit again

In a Cluster configured with DR networkStMgr may not get initialized and stuckin a deadlock while enabling the IPTablerules (can be verified from the stMgrlog)

CSCvt89709

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Click recover for the selectedVM again under the standalonemode This will move the VM torecovered state

When the user recovers a VM protectedin a group it along with other VMs in thegroup move to recovering state instandalonemode The selectedVM shouldmove to recovered state

CSCvs75553

35(2e)Clean up the fail over cluster andthen recreate the fail over clusterNo need to touch the HX storagecluster

Hyper-V When replicated DC was usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2d)40(1a)

Keep cluster utilization as low aspossible To avoid read-only statekeep below 92

HyperFlex clusters storage utilization isabove 92 usable capacity

CSCvr20922

Release Notes for Cisco HX Data Platform Release 4050

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1b)NAAll the disks in the node got locked afterrebooting the node

We successfully unlocked usingsed-clientsh -U command but wantedto test with another reboot and driveslocked again

CSCvs74286

35(2g)To recover the VM copy over thedata disks and attach them to thenew VM

After enable encryption HX connectencryption status showing Caution butall of the disks enable encryptionsuccessful

-Self Encrypting Drives Service isrunning on all of the nodes

-There is no error message of encryption

- USB0 interface is up

-All SED disks showing supported 1enabled 1 locked 0

CSCvs93245

40(1b)Enable kernelpanic_on_oops as apersistent configuration This willcause the node to panic and rebootimmediately

A cluster node running HX release40(1b) may hang in the Linux kernelThis is classified as an oops and adeviation from the expected behavior

CSCvs54285

Release Notes for Cisco HX Data Platform Release 4051

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Enabling K8iscsi stack in Hx clusterCSCvs41324

Release Notes for Cisco HX Data Platform Release 4052

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Steps to enable K8 on HX cluster

Perform following steps on ALLthe controller VMs

1 Updateetcinitscvmclientconf toenable tune iscsiEnable totrue

sed -iesiscsiEnable=falseiscsiEnable=trueetcinitscvmclientconf

2 Run following initctlcommand to reloadconfiguration

initctl reload-configuration

3 Restart scvmclient process

stop scvmclient startscvmclient

4 Run following initctlcommand to mount SYSTEMDS

initctl emit --no-waitsystem-datastore-created

5 Verify that iscsiEnable tuneis set to true using thefollowing command

ps -eaf | grep scvmclient |grep -v grep

root 6241 1 1 Dec06 014007optspringpathstorfs-corescvmclient-T logEnabled=false -TlogSyslogEnabled=true -TlogEchoToScreen=false -TstatLoggingToFile=false -TstatLoggingToSyslog=true-TlogDir=varlogspringpath-TnfsBackendServerList=1010748100-T iscsiEnable=true -TiscsiUseAsync=true -T

Release Notes for Cisco HX Data Platform Release 4053

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

iscsiConfigFilePath=nfsSYSTEMistgtconfps -eaf | grep scvmclient |grep -v grep

35(2e)Following procedure can be usedto check NFS Queue Depth andincrease if needed

rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth256 lt- Low value[rootHXESXI1] vsish -e setvmkModulesnfsclientmnt[DATASTORE]propertiesmaxQDepth 1024[rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth1024 lt- Optimalvalue

This requires an ESXi host rebootto take effect

Please place the node in HyperflexMaintenanceMode and gracefullyreboot the node for the changes tobe applied

HyperFlex Datastore NFS Queue Depthshows as 256 which can lead toperformance (including latency) issues

CSCvr83056

40(1b)This is cosmetic The error will goaway once the cluster returns tohealthy status

When a single node is offline in clusterstcli cluster storage-summary shows twonodes unavailable rootSCVM~ stclicluster storage-summary

CSCvs47735

35(2g)Modify both the root and adminpassword for the SCVMs to be thesame

stcli security password set -uadmin

Cluster expansion fails at Config Installerstage when the root and admin passwordfor the storage controller (SCVM) aredifferent

CSCvs69317

35(2d)NAAfter successfully changing the DNSserver on the HX controller we still cansee the original DNS entry that was addedduring the deployment

CSCvs69154

Release Notes for Cisco HX Data Platform Release 4054

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Customers are strongly encouragedto work with TAC in order toidentify and remediate this issue

1 Change getcluster from 2m to8m inoptspringpathstorfs-mgmtstMgr-10confapplicationconfon all nodes

2 Restart stMgr on all nodes

You may see this error message after afailed upgrade or other task such asattempting to enter a node into HXmaintenance mode

CSCvs53555

40(2a)Run the following command oneach storage controller VM as root

grep -i monitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp sed -iesmonitor_dns_serverstruemonitor_dns_serversfalseoptspringpathhx-diag-toolswatchdog_configjsonampamp grep -imonitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp restart watchdog

Receive the following alertevent in HXConnect HX Controller VMHOSTNAME one or more configuredDNS servers not responding

CSCvt13947

35(2b)Delete any empty (Size 0) log filesunder varlogzookeeperversion-2ORvarlogzookeeperstandaloneversion-2

Zookeeper fails to start while Exhibitor isrunning however echo srvr returnsnothing

CSCvs21562

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

40(4e)1 De-commission the server

2 Power drain the server -REMOVE BOTH POWERCORDS ON THE BACK OFTHE SERVER FOR 10SECONDS THENREINSERTPOWERCORDS

3 Re-commission the server

M2 boot disk is missing from serverinventory after upgrade to server firmware40(4e) As a result server fails to boot toOS installed in the M2 disk The issuepersists after re-acknowledgement as wellas de-commission andre-acknowledgement of the server

CSCvs02466

Release Notes for Cisco HX Data Platform Release 4055

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Re-start the flusher manuallyMany disks taken offline by a fault ormanually resulting in ENOSPC oncluster

CSCvq32721

35(2e)Clean up fail over cluster and thenrecreate the fail over cluster Thereis no need to touch the HX storagecluster

[Hyper-V]When replicated DCwas usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2b)Manually install upgradescvmclient VIB andmake sure thatit matches with HXDP version

Upgrade from release HX 35(2a) to35(2x) 4X does not upgrade scvmclientRecovery point may not work

CSCvq54992

Management

30(1b)To recover the VM copy over thedata disks and attach them to thenew VM

[HyperV]VMshows up onmultiple nodesCSCvj22992

35(1a)1 Do not change MTU at ESXlevel on running cluster

2 Reset back to the original value

ChangingMTU from 9000 to 1500 on oneESX node causes storfs process to restarton all nodes

CSCvm99150

Release Notes for Cisco HX Data Platform Release 4056

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)After the installation (for newdeployments) or after an upgradeto 401a (for existingdeployments) perform thefollowing steps on all ControllerVMs

1 Edit the following tune files (onall Controller VMs)

optspringpathconfiglfftunes

optspringpathconfigvsi_16tbtunes

2 SetcleanerEnableSegSummaryCleaning

to false

3 After editing the above tunefiles

a SSH in to all the ControllerVMs

b Run storfstool -- -Z

c Run the below command andcheck the values Tune valueshould be true

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=true

d Type below commands andapplied the tunes changesdynamically

echo false gttmpstprocfssystemtunecleanerEnableSegSummaryCleaning

e Verify that tune values arechanged Run below commandValue should be false

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=false

f umount tmpstprocfs

A cluster with 8TB or 12TB disk drivesmay experience IO stalls for severalminutes after another node fails in thecluster

CSCvp23718

Release Notes for Cisco HX Data Platform Release 4057

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)1 Update the conf file manuallyto change to the link local addressto

2 Restart scvmclient on thecontroller and ESX

HX 35(2b) FlexVol to HX 40 CSIupgrade does not work FlexVol continuesto work

CSCvp26319

35(2a)Disable rebalance until node healsthen reenable rebalance

7-node cluster healing very slowlybecause rebalance change AMS reqs areblocking healing process

CSCvp62512

35(2a)NAWith moderate cluster storage capacityusage under site failover cluster clustermay experience an all paths (APD) downcondition

CSCvq49412

40(1a)NAThis product includes Third-partySoftware that is affected by thevulnerabilities identified by the followingCommon Vulnerability and Exposures(CVE) IDs

CVE-2016-0762 CVE-2016-6797CVE-2016-6816 CVE-2016-8735CVE-2017-5647 CVE-2017-12615CVE-2017-12616 CVE-2017-12617CVE-2017-7674 CVE-2018-1304CVE-2018-8014 CVE-2018-1336CVE-2018-8034 CVE-2018-11784CVE-2019-0232

CSCvp36364

35(2d)NAstcli cluster info command needs toprovide UCSM VIP address Currently itshows ucsm-hostcom

CSCvq11456

35(2a)No workarounds currently Thestale entries do not affect the dayto day operations of HyperFlex

HyperFlex upgrade validation failedbecause cluster has extraneous stNodes ininternal database

CSCvq32530

35(2a)NAWhen not in maintenancemode all drivesappear in HX Connect and get updated asexpected

When in maintenance mode extra drivesare shown in HX Connect with no slotnumbers

CSCvq42378

35(2b)Contact TAC to cleanup duplicatenode

Nodes duplicated in HX Connect aftercluster recreate

CSCvq61825

Release Notes for Cisco HX Data Platform Release 4058

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)NACurrently HyperFlex installs on Hyper-Vdo not contain the stcli security whitelistcommandset

CSCvq66245

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

Release Notes for Cisco HX Data Platform Release 4059

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)[Hyper-V] Converged node expansionfails for incoming node with VIC 14xx

CSCvq39523

Release Notes for Cisco HX Data Platform Release 4060

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

1 Set Jumbo frame inWindowsHypervisor for incomingnode

Get-NetAdapter storage-data-a|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

Get-NetAdapter storage-data-b|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

2 Verify all interfaces are setcorrectly to support Jumbo frames

Get-NetAdapter storage |Get-NetAdapterAdvancedProperty| RegistryKeyword -matchjumbo | ft -auto

You should receive the followingmessage

Name DisplayName DisplayValue

RegistryKeyword RegistryValue

---- ----------- ------------

--------------- -------------

vswitch-hx-storage-data Jumbo

Packet 9014 Bytes

JumboPacket 9014

storage-data-b Jumbo Packet

Bytes 9014 JumboPacket

9014

storage-data-a Jumbo Packet

Bytes 9014 JumboPacket

9014

3 Reboot only the incoming nodeor the node being expanded

4 Retry Cluster Expansion in

Release Notes for Cisco HX Data Platform Release 4061

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Startrestart EAM service invCenter

When EAM service is not runningvCenter registration is failing

CSCvt55712

35(2a)Contact Cisco TAC if you believeyou are seeing similar conditions

HyperFlex Stretch Cluster saw poorapplication performance due to writelatency cluster remained unhealthy andrebalance was stuck

This defect is currently being used forinvestigative purposes

CSCvt45344

35(2g)Remove log files fromvarlogspringpath folder when thelog files are not getting rotatedStop the relevant service removelarge log files fromvarlogspringpath folder andrestart the service

varstv folder in Controller VM maybecome full

CSCvt36374

40(2b)Restarting the stMgr will initializeit again

In a Cluster configured with DR networkStMgr may not get initialized and stuckin a deadlock while enabling the IPTablerules (can be verified from the stMgrlog)

CSCvt89709

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Click recover for the selectedVM again under the standalonemode This will move the VM torecovered state

When the user recovers a VM protectedin a group it along with other VMs in thegroup move to recovering state instandalonemode The selectedVM shouldmove to recovered state

CSCvs75553

35(2e)Clean up the fail over cluster andthen recreate the fail over clusterNo need to touch the HX storagecluster

Hyper-V When replicated DC was usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2d)40(1a)

Keep cluster utilization as low aspossible To avoid read-only statekeep below 92

HyperFlex clusters storage utilization isabove 92 usable capacity

CSCvr20922

Release Notes for Cisco HX Data Platform Release 4050

Open Caveats in Release 40(2b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1b)NAAll the disks in the node got locked afterrebooting the node

We successfully unlocked usingsed-clientsh -U command but wantedto test with another reboot and driveslocked again

CSCvs74286

35(2g)To recover the VM copy over thedata disks and attach them to thenew VM

After enable encryption HX connectencryption status showing Caution butall of the disks enable encryptionsuccessful

-Self Encrypting Drives Service isrunning on all of the nodes

-There is no error message of encryption

- USB0 interface is up

-All SED disks showing supported 1enabled 1 locked 0

CSCvs93245

40(1b)Enable kernelpanic_on_oops as apersistent configuration This willcause the node to panic and rebootimmediately

A cluster node running HX release40(1b) may hang in the Linux kernelThis is classified as an oops and adeviation from the expected behavior

CSCvs54285

Release Notes for Cisco HX Data Platform Release 4051

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Enabling K8iscsi stack in Hx clusterCSCvs41324

Release Notes for Cisco HX Data Platform Release 4052

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Steps to enable K8 on HX cluster

Perform following steps on ALLthe controller VMs

1 Updateetcinitscvmclientconf toenable tune iscsiEnable totrue

sed -iesiscsiEnable=falseiscsiEnable=trueetcinitscvmclientconf

2 Run following initctlcommand to reloadconfiguration

initctl reload-configuration

3 Restart scvmclient process

stop scvmclient startscvmclient

4 Run following initctlcommand to mount SYSTEMDS

initctl emit --no-waitsystem-datastore-created

5 Verify that iscsiEnable tuneis set to true using thefollowing command

ps -eaf | grep scvmclient |grep -v grep

root 6241 1 1 Dec06 014007optspringpathstorfs-corescvmclient-T logEnabled=false -TlogSyslogEnabled=true -TlogEchoToScreen=false -TstatLoggingToFile=false -TstatLoggingToSyslog=true-TlogDir=varlogspringpath-TnfsBackendServerList=1010748100-T iscsiEnable=true -TiscsiUseAsync=true -T

Release Notes for Cisco HX Data Platform Release 4053

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

iscsiConfigFilePath=nfsSYSTEMistgtconfps -eaf | grep scvmclient |grep -v grep

35(2e)Following procedure can be usedto check NFS Queue Depth andincrease if needed

rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth256 lt- Low value[rootHXESXI1] vsish -e setvmkModulesnfsclientmnt[DATASTORE]propertiesmaxQDepth 1024[rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth1024 lt- Optimalvalue

This requires an ESXi host rebootto take effect

Please place the node in HyperflexMaintenanceMode and gracefullyreboot the node for the changes tobe applied

HyperFlex Datastore NFS Queue Depthshows as 256 which can lead toperformance (including latency) issues

CSCvr83056

40(1b)This is cosmetic The error will goaway once the cluster returns tohealthy status

When a single node is offline in clusterstcli cluster storage-summary shows twonodes unavailable rootSCVM~ stclicluster storage-summary

CSCvs47735

35(2g)Modify both the root and adminpassword for the SCVMs to be thesame

stcli security password set -uadmin

Cluster expansion fails at Config Installerstage when the root and admin passwordfor the storage controller (SCVM) aredifferent

CSCvs69317

35(2d)NAAfter successfully changing the DNSserver on the HX controller we still cansee the original DNS entry that was addedduring the deployment

CSCvs69154

Release Notes for Cisco HX Data Platform Release 4054

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Customers are strongly encouragedto work with TAC in order toidentify and remediate this issue

1 Change getcluster from 2m to8m inoptspringpathstorfs-mgmtstMgr-10confapplicationconfon all nodes

2 Restart stMgr on all nodes

You may see this error message after afailed upgrade or other task such asattempting to enter a node into HXmaintenance mode

CSCvs53555

40(2a)Run the following command oneach storage controller VM as root

grep -i monitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp sed -iesmonitor_dns_serverstruemonitor_dns_serversfalseoptspringpathhx-diag-toolswatchdog_configjsonampamp grep -imonitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp restart watchdog

Receive the following alertevent in HXConnect HX Controller VMHOSTNAME one or more configuredDNS servers not responding

CSCvt13947

35(2b)Delete any empty (Size 0) log filesunder varlogzookeeperversion-2ORvarlogzookeeperstandaloneversion-2

Zookeeper fails to start while Exhibitor isrunning however echo srvr returnsnothing

CSCvs21562

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

40(4e)1 De-commission the server

2 Power drain the server -REMOVE BOTH POWERCORDS ON THE BACK OFTHE SERVER FOR 10SECONDS THENREINSERTPOWERCORDS

3 Re-commission the server

M2 boot disk is missing from serverinventory after upgrade to server firmware40(4e) As a result server fails to boot toOS installed in the M2 disk The issuepersists after re-acknowledgement as wellas de-commission andre-acknowledgement of the server

CSCvs02466

Release Notes for Cisco HX Data Platform Release 4055

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Re-start the flusher manuallyMany disks taken offline by a fault ormanually resulting in ENOSPC oncluster

CSCvq32721

35(2e)Clean up fail over cluster and thenrecreate the fail over cluster Thereis no need to touch the HX storagecluster

[Hyper-V]When replicated DCwas usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2b)Manually install upgradescvmclient VIB andmake sure thatit matches with HXDP version

Upgrade from release HX 35(2a) to35(2x) 4X does not upgrade scvmclientRecovery point may not work

CSCvq54992

Management

30(1b)To recover the VM copy over thedata disks and attach them to thenew VM

[HyperV]VMshows up onmultiple nodesCSCvj22992

35(1a)1 Do not change MTU at ESXlevel on running cluster

2 Reset back to the original value

ChangingMTU from 9000 to 1500 on oneESX node causes storfs process to restarton all nodes

CSCvm99150

Release Notes for Cisco HX Data Platform Release 4056

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)After the installation (for newdeployments) or after an upgradeto 401a (for existingdeployments) perform thefollowing steps on all ControllerVMs

1 Edit the following tune files (onall Controller VMs)

optspringpathconfiglfftunes

optspringpathconfigvsi_16tbtunes

2 SetcleanerEnableSegSummaryCleaning

to false

3 After editing the above tunefiles

a SSH in to all the ControllerVMs

b Run storfstool -- -Z

c Run the below command andcheck the values Tune valueshould be true

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=true

d Type below commands andapplied the tunes changesdynamically

echo false gttmpstprocfssystemtunecleanerEnableSegSummaryCleaning

e Verify that tune values arechanged Run below commandValue should be false

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=false

f umount tmpstprocfs

A cluster with 8TB or 12TB disk drivesmay experience IO stalls for severalminutes after another node fails in thecluster

CSCvp23718

Release Notes for Cisco HX Data Platform Release 4057

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)1 Update the conf file manuallyto change to the link local addressto

2 Restart scvmclient on thecontroller and ESX

HX 35(2b) FlexVol to HX 40 CSIupgrade does not work FlexVol continuesto work

CSCvp26319

35(2a)Disable rebalance until node healsthen reenable rebalance

7-node cluster healing very slowlybecause rebalance change AMS reqs areblocking healing process

CSCvp62512

35(2a)NAWith moderate cluster storage capacityusage under site failover cluster clustermay experience an all paths (APD) downcondition

CSCvq49412

40(1a)NAThis product includes Third-partySoftware that is affected by thevulnerabilities identified by the followingCommon Vulnerability and Exposures(CVE) IDs

CVE-2016-0762 CVE-2016-6797CVE-2016-6816 CVE-2016-8735CVE-2017-5647 CVE-2017-12615CVE-2017-12616 CVE-2017-12617CVE-2017-7674 CVE-2018-1304CVE-2018-8014 CVE-2018-1336CVE-2018-8034 CVE-2018-11784CVE-2019-0232

CSCvp36364

35(2d)NAstcli cluster info command needs toprovide UCSM VIP address Currently itshows ucsm-hostcom

CSCvq11456

35(2a)No workarounds currently Thestale entries do not affect the dayto day operations of HyperFlex

HyperFlex upgrade validation failedbecause cluster has extraneous stNodes ininternal database

CSCvq32530

35(2a)NAWhen not in maintenancemode all drivesappear in HX Connect and get updated asexpected

When in maintenance mode extra drivesare shown in HX Connect with no slotnumbers

CSCvq42378

35(2b)Contact TAC to cleanup duplicatenode

Nodes duplicated in HX Connect aftercluster recreate

CSCvq61825

Release Notes for Cisco HX Data Platform Release 4058

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)NACurrently HyperFlex installs on Hyper-Vdo not contain the stcli security whitelistcommandset

CSCvq66245

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

Release Notes for Cisco HX Data Platform Release 4059

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)[Hyper-V] Converged node expansionfails for incoming node with VIC 14xx

CSCvq39523

Release Notes for Cisco HX Data Platform Release 4060

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

1 Set Jumbo frame inWindowsHypervisor for incomingnode

Get-NetAdapter storage-data-a|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

Get-NetAdapter storage-data-b|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

2 Verify all interfaces are setcorrectly to support Jumbo frames

Get-NetAdapter storage |Get-NetAdapterAdvancedProperty| RegistryKeyword -matchjumbo | ft -auto

You should receive the followingmessage

Name DisplayName DisplayValue

RegistryKeyword RegistryValue

---- ----------- ------------

--------------- -------------

vswitch-hx-storage-data Jumbo

Packet 9014 Bytes

JumboPacket 9014

storage-data-b Jumbo Packet

Bytes 9014 JumboPacket

9014

storage-data-a Jumbo Packet

Bytes 9014 JumboPacket

9014

3 Reboot only the incoming nodeor the node being expanded

4 Retry Cluster Expansion in

Release Notes for Cisco HX Data Platform Release 4061

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1b)NAAll the disks in the node got locked afterrebooting the node

We successfully unlocked usingsed-clientsh -U command but wantedto test with another reboot and driveslocked again

CSCvs74286

35(2g)To recover the VM copy over thedata disks and attach them to thenew VM

After enable encryption HX connectencryption status showing Caution butall of the disks enable encryptionsuccessful

-Self Encrypting Drives Service isrunning on all of the nodes

-There is no error message of encryption

- USB0 interface is up

-All SED disks showing supported 1enabled 1 locked 0

CSCvs93245

40(1b)Enable kernelpanic_on_oops as apersistent configuration This willcause the node to panic and rebootimmediately

A cluster node running HX release40(1b) may hang in the Linux kernelThis is classified as an oops and adeviation from the expected behavior

CSCvs54285

Release Notes for Cisco HX Data Platform Release 4051

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Enabling K8iscsi stack in Hx clusterCSCvs41324

Release Notes for Cisco HX Data Platform Release 4052

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Steps to enable K8 on HX cluster

Perform following steps on ALLthe controller VMs

1 Updateetcinitscvmclientconf toenable tune iscsiEnable totrue

sed -iesiscsiEnable=falseiscsiEnable=trueetcinitscvmclientconf

2 Run following initctlcommand to reloadconfiguration

initctl reload-configuration

3 Restart scvmclient process

stop scvmclient startscvmclient

4 Run following initctlcommand to mount SYSTEMDS

initctl emit --no-waitsystem-datastore-created

5 Verify that iscsiEnable tuneis set to true using thefollowing command

ps -eaf | grep scvmclient |grep -v grep

root 6241 1 1 Dec06 014007optspringpathstorfs-corescvmclient-T logEnabled=false -TlogSyslogEnabled=true -TlogEchoToScreen=false -TstatLoggingToFile=false -TstatLoggingToSyslog=true-TlogDir=varlogspringpath-TnfsBackendServerList=1010748100-T iscsiEnable=true -TiscsiUseAsync=true -T

Release Notes for Cisco HX Data Platform Release 4053

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

iscsiConfigFilePath=nfsSYSTEMistgtconfps -eaf | grep scvmclient |grep -v grep

35(2e)Following procedure can be usedto check NFS Queue Depth andincrease if needed

rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth256 lt- Low value[rootHXESXI1] vsish -e setvmkModulesnfsclientmnt[DATASTORE]propertiesmaxQDepth 1024[rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth1024 lt- Optimalvalue

This requires an ESXi host rebootto take effect

Please place the node in HyperflexMaintenanceMode and gracefullyreboot the node for the changes tobe applied

HyperFlex Datastore NFS Queue Depthshows as 256 which can lead toperformance (including latency) issues

CSCvr83056

40(1b)This is cosmetic The error will goaway once the cluster returns tohealthy status

When a single node is offline in clusterstcli cluster storage-summary shows twonodes unavailable rootSCVM~ stclicluster storage-summary

CSCvs47735

35(2g)Modify both the root and adminpassword for the SCVMs to be thesame

stcli security password set -uadmin

Cluster expansion fails at Config Installerstage when the root and admin passwordfor the storage controller (SCVM) aredifferent

CSCvs69317

35(2d)NAAfter successfully changing the DNSserver on the HX controller we still cansee the original DNS entry that was addedduring the deployment

CSCvs69154

Release Notes for Cisco HX Data Platform Release 4054

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Customers are strongly encouragedto work with TAC in order toidentify and remediate this issue

1 Change getcluster from 2m to8m inoptspringpathstorfs-mgmtstMgr-10confapplicationconfon all nodes

2 Restart stMgr on all nodes

You may see this error message after afailed upgrade or other task such asattempting to enter a node into HXmaintenance mode

CSCvs53555

40(2a)Run the following command oneach storage controller VM as root

grep -i monitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp sed -iesmonitor_dns_serverstruemonitor_dns_serversfalseoptspringpathhx-diag-toolswatchdog_configjsonampamp grep -imonitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp restart watchdog

Receive the following alertevent in HXConnect HX Controller VMHOSTNAME one or more configuredDNS servers not responding

CSCvt13947

35(2b)Delete any empty (Size 0) log filesunder varlogzookeeperversion-2ORvarlogzookeeperstandaloneversion-2

Zookeeper fails to start while Exhibitor isrunning however echo srvr returnsnothing

CSCvs21562

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

40(4e)1 De-commission the server

2 Power drain the server -REMOVE BOTH POWERCORDS ON THE BACK OFTHE SERVER FOR 10SECONDS THENREINSERTPOWERCORDS

3 Re-commission the server

M2 boot disk is missing from serverinventory after upgrade to server firmware40(4e) As a result server fails to boot toOS installed in the M2 disk The issuepersists after re-acknowledgement as wellas de-commission andre-acknowledgement of the server

CSCvs02466

Release Notes for Cisco HX Data Platform Release 4055

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Re-start the flusher manuallyMany disks taken offline by a fault ormanually resulting in ENOSPC oncluster

CSCvq32721

35(2e)Clean up fail over cluster and thenrecreate the fail over cluster Thereis no need to touch the HX storagecluster

[Hyper-V]When replicated DCwas usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2b)Manually install upgradescvmclient VIB andmake sure thatit matches with HXDP version

Upgrade from release HX 35(2a) to35(2x) 4X does not upgrade scvmclientRecovery point may not work

CSCvq54992

Management

30(1b)To recover the VM copy over thedata disks and attach them to thenew VM

[HyperV]VMshows up onmultiple nodesCSCvj22992

35(1a)1 Do not change MTU at ESXlevel on running cluster

2 Reset back to the original value

ChangingMTU from 9000 to 1500 on oneESX node causes storfs process to restarton all nodes

CSCvm99150

Release Notes for Cisco HX Data Platform Release 4056

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)After the installation (for newdeployments) or after an upgradeto 401a (for existingdeployments) perform thefollowing steps on all ControllerVMs

1 Edit the following tune files (onall Controller VMs)

optspringpathconfiglfftunes

optspringpathconfigvsi_16tbtunes

2 SetcleanerEnableSegSummaryCleaning

to false

3 After editing the above tunefiles

a SSH in to all the ControllerVMs

b Run storfstool -- -Z

c Run the below command andcheck the values Tune valueshould be true

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=true

d Type below commands andapplied the tunes changesdynamically

echo false gttmpstprocfssystemtunecleanerEnableSegSummaryCleaning

e Verify that tune values arechanged Run below commandValue should be false

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=false

f umount tmpstprocfs

A cluster with 8TB or 12TB disk drivesmay experience IO stalls for severalminutes after another node fails in thecluster

CSCvp23718

Release Notes for Cisco HX Data Platform Release 4057

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)1 Update the conf file manuallyto change to the link local addressto

2 Restart scvmclient on thecontroller and ESX

HX 35(2b) FlexVol to HX 40 CSIupgrade does not work FlexVol continuesto work

CSCvp26319

35(2a)Disable rebalance until node healsthen reenable rebalance

7-node cluster healing very slowlybecause rebalance change AMS reqs areblocking healing process

CSCvp62512

35(2a)NAWith moderate cluster storage capacityusage under site failover cluster clustermay experience an all paths (APD) downcondition

CSCvq49412

40(1a)NAThis product includes Third-partySoftware that is affected by thevulnerabilities identified by the followingCommon Vulnerability and Exposures(CVE) IDs

CVE-2016-0762 CVE-2016-6797CVE-2016-6816 CVE-2016-8735CVE-2017-5647 CVE-2017-12615CVE-2017-12616 CVE-2017-12617CVE-2017-7674 CVE-2018-1304CVE-2018-8014 CVE-2018-1336CVE-2018-8034 CVE-2018-11784CVE-2019-0232

CSCvp36364

35(2d)NAstcli cluster info command needs toprovide UCSM VIP address Currently itshows ucsm-hostcom

CSCvq11456

35(2a)No workarounds currently Thestale entries do not affect the dayto day operations of HyperFlex

HyperFlex upgrade validation failedbecause cluster has extraneous stNodes ininternal database

CSCvq32530

35(2a)NAWhen not in maintenancemode all drivesappear in HX Connect and get updated asexpected

When in maintenance mode extra drivesare shown in HX Connect with no slotnumbers

CSCvq42378

35(2b)Contact TAC to cleanup duplicatenode

Nodes duplicated in HX Connect aftercluster recreate

CSCvq61825

Release Notes for Cisco HX Data Platform Release 4058

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)NACurrently HyperFlex installs on Hyper-Vdo not contain the stcli security whitelistcommandset

CSCvq66245

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

Release Notes for Cisco HX Data Platform Release 4059

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)[Hyper-V] Converged node expansionfails for incoming node with VIC 14xx

CSCvq39523

Release Notes for Cisco HX Data Platform Release 4060

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

1 Set Jumbo frame inWindowsHypervisor for incomingnode

Get-NetAdapter storage-data-a|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

Get-NetAdapter storage-data-b|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

2 Verify all interfaces are setcorrectly to support Jumbo frames

Get-NetAdapter storage |Get-NetAdapterAdvancedProperty| RegistryKeyword -matchjumbo | ft -auto

You should receive the followingmessage

Name DisplayName DisplayValue

RegistryKeyword RegistryValue

---- ----------- ------------

--------------- -------------

vswitch-hx-storage-data Jumbo

Packet 9014 Bytes

JumboPacket 9014

storage-data-b Jumbo Packet

Bytes 9014 JumboPacket

9014

storage-data-a Jumbo Packet

Bytes 9014 JumboPacket

9014

3 Reboot only the incoming nodeor the node being expanded

4 Retry Cluster Expansion in

Release Notes for Cisco HX Data Platform Release 4061

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

DefectFound inRelease

WorkaroundSymptomDefect ID

40(2a)Enabling K8iscsi stack in Hx clusterCSCvs41324

Release Notes for Cisco HX Data Platform Release 4052

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Steps to enable K8 on HX cluster

Perform following steps on ALLthe controller VMs

1 Updateetcinitscvmclientconf toenable tune iscsiEnable totrue

sed -iesiscsiEnable=falseiscsiEnable=trueetcinitscvmclientconf

2 Run following initctlcommand to reloadconfiguration

initctl reload-configuration

3 Restart scvmclient process

stop scvmclient startscvmclient

4 Run following initctlcommand to mount SYSTEMDS

initctl emit --no-waitsystem-datastore-created

5 Verify that iscsiEnable tuneis set to true using thefollowing command

ps -eaf | grep scvmclient |grep -v grep

root 6241 1 1 Dec06 014007optspringpathstorfs-corescvmclient-T logEnabled=false -TlogSyslogEnabled=true -TlogEchoToScreen=false -TstatLoggingToFile=false -TstatLoggingToSyslog=true-TlogDir=varlogspringpath-TnfsBackendServerList=1010748100-T iscsiEnable=true -TiscsiUseAsync=true -T

Release Notes for Cisco HX Data Platform Release 4053

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

iscsiConfigFilePath=nfsSYSTEMistgtconfps -eaf | grep scvmclient |grep -v grep

35(2e)Following procedure can be usedto check NFS Queue Depth andincrease if needed

rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth256 lt- Low value[rootHXESXI1] vsish -e setvmkModulesnfsclientmnt[DATASTORE]propertiesmaxQDepth 1024[rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth1024 lt- Optimalvalue

This requires an ESXi host rebootto take effect

Please place the node in HyperflexMaintenanceMode and gracefullyreboot the node for the changes tobe applied

HyperFlex Datastore NFS Queue Depthshows as 256 which can lead toperformance (including latency) issues

CSCvr83056

40(1b)This is cosmetic The error will goaway once the cluster returns tohealthy status

When a single node is offline in clusterstcli cluster storage-summary shows twonodes unavailable rootSCVM~ stclicluster storage-summary

CSCvs47735

35(2g)Modify both the root and adminpassword for the SCVMs to be thesame

stcli security password set -uadmin

Cluster expansion fails at Config Installerstage when the root and admin passwordfor the storage controller (SCVM) aredifferent

CSCvs69317

35(2d)NAAfter successfully changing the DNSserver on the HX controller we still cansee the original DNS entry that was addedduring the deployment

CSCvs69154

Release Notes for Cisco HX Data Platform Release 4054

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Customers are strongly encouragedto work with TAC in order toidentify and remediate this issue

1 Change getcluster from 2m to8m inoptspringpathstorfs-mgmtstMgr-10confapplicationconfon all nodes

2 Restart stMgr on all nodes

You may see this error message after afailed upgrade or other task such asattempting to enter a node into HXmaintenance mode

CSCvs53555

40(2a)Run the following command oneach storage controller VM as root

grep -i monitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp sed -iesmonitor_dns_serverstruemonitor_dns_serversfalseoptspringpathhx-diag-toolswatchdog_configjsonampamp grep -imonitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp restart watchdog

Receive the following alertevent in HXConnect HX Controller VMHOSTNAME one or more configuredDNS servers not responding

CSCvt13947

35(2b)Delete any empty (Size 0) log filesunder varlogzookeeperversion-2ORvarlogzookeeperstandaloneversion-2

Zookeeper fails to start while Exhibitor isrunning however echo srvr returnsnothing

CSCvs21562

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

40(4e)1 De-commission the server

2 Power drain the server -REMOVE BOTH POWERCORDS ON THE BACK OFTHE SERVER FOR 10SECONDS THENREINSERTPOWERCORDS

3 Re-commission the server

M2 boot disk is missing from serverinventory after upgrade to server firmware40(4e) As a result server fails to boot toOS installed in the M2 disk The issuepersists after re-acknowledgement as wellas de-commission andre-acknowledgement of the server

CSCvs02466

Release Notes for Cisco HX Data Platform Release 4055

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Re-start the flusher manuallyMany disks taken offline by a fault ormanually resulting in ENOSPC oncluster

CSCvq32721

35(2e)Clean up fail over cluster and thenrecreate the fail over cluster Thereis no need to touch the HX storagecluster

[Hyper-V]When replicated DCwas usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2b)Manually install upgradescvmclient VIB andmake sure thatit matches with HXDP version

Upgrade from release HX 35(2a) to35(2x) 4X does not upgrade scvmclientRecovery point may not work

CSCvq54992

Management

30(1b)To recover the VM copy over thedata disks and attach them to thenew VM

[HyperV]VMshows up onmultiple nodesCSCvj22992

35(1a)1 Do not change MTU at ESXlevel on running cluster

2 Reset back to the original value

ChangingMTU from 9000 to 1500 on oneESX node causes storfs process to restarton all nodes

CSCvm99150

Release Notes for Cisco HX Data Platform Release 4056

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)After the installation (for newdeployments) or after an upgradeto 401a (for existingdeployments) perform thefollowing steps on all ControllerVMs

1 Edit the following tune files (onall Controller VMs)

optspringpathconfiglfftunes

optspringpathconfigvsi_16tbtunes

2 SetcleanerEnableSegSummaryCleaning

to false

3 After editing the above tunefiles

a SSH in to all the ControllerVMs

b Run storfstool -- -Z

c Run the below command andcheck the values Tune valueshould be true

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=true

d Type below commands andapplied the tunes changesdynamically

echo false gttmpstprocfssystemtunecleanerEnableSegSummaryCleaning

e Verify that tune values arechanged Run below commandValue should be false

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=false

f umount tmpstprocfs

A cluster with 8TB or 12TB disk drivesmay experience IO stalls for severalminutes after another node fails in thecluster

CSCvp23718

Release Notes for Cisco HX Data Platform Release 4057

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)1 Update the conf file manuallyto change to the link local addressto

2 Restart scvmclient on thecontroller and ESX

HX 35(2b) FlexVol to HX 40 CSIupgrade does not work FlexVol continuesto work

CSCvp26319

35(2a)Disable rebalance until node healsthen reenable rebalance

7-node cluster healing very slowlybecause rebalance change AMS reqs areblocking healing process

CSCvp62512

35(2a)NAWith moderate cluster storage capacityusage under site failover cluster clustermay experience an all paths (APD) downcondition

CSCvq49412

40(1a)NAThis product includes Third-partySoftware that is affected by thevulnerabilities identified by the followingCommon Vulnerability and Exposures(CVE) IDs

CVE-2016-0762 CVE-2016-6797CVE-2016-6816 CVE-2016-8735CVE-2017-5647 CVE-2017-12615CVE-2017-12616 CVE-2017-12617CVE-2017-7674 CVE-2018-1304CVE-2018-8014 CVE-2018-1336CVE-2018-8034 CVE-2018-11784CVE-2019-0232

CSCvp36364

35(2d)NAstcli cluster info command needs toprovide UCSM VIP address Currently itshows ucsm-hostcom

CSCvq11456

35(2a)No workarounds currently Thestale entries do not affect the dayto day operations of HyperFlex

HyperFlex upgrade validation failedbecause cluster has extraneous stNodes ininternal database

CSCvq32530

35(2a)NAWhen not in maintenancemode all drivesappear in HX Connect and get updated asexpected

When in maintenance mode extra drivesare shown in HX Connect with no slotnumbers

CSCvq42378

35(2b)Contact TAC to cleanup duplicatenode

Nodes duplicated in HX Connect aftercluster recreate

CSCvq61825

Release Notes for Cisco HX Data Platform Release 4058

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)NACurrently HyperFlex installs on Hyper-Vdo not contain the stcli security whitelistcommandset

CSCvq66245

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

Release Notes for Cisco HX Data Platform Release 4059

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)[Hyper-V] Converged node expansionfails for incoming node with VIC 14xx

CSCvq39523

Release Notes for Cisco HX Data Platform Release 4060

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

1 Set Jumbo frame inWindowsHypervisor for incomingnode

Get-NetAdapter storage-data-a|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

Get-NetAdapter storage-data-b|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

2 Verify all interfaces are setcorrectly to support Jumbo frames

Get-NetAdapter storage |Get-NetAdapterAdvancedProperty| RegistryKeyword -matchjumbo | ft -auto

You should receive the followingmessage

Name DisplayName DisplayValue

RegistryKeyword RegistryValue

---- ----------- ------------

--------------- -------------

vswitch-hx-storage-data Jumbo

Packet 9014 Bytes

JumboPacket 9014

storage-data-b Jumbo Packet

Bytes 9014 JumboPacket

9014

storage-data-a Jumbo Packet

Bytes 9014 JumboPacket

9014

3 Reboot only the incoming nodeor the node being expanded

4 Retry Cluster Expansion in

Release Notes for Cisco HX Data Platform Release 4061

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

DefectFound inRelease

WorkaroundSymptomDefect ID

Steps to enable K8 on HX cluster

Perform following steps on ALLthe controller VMs

1 Updateetcinitscvmclientconf toenable tune iscsiEnable totrue

sed -iesiscsiEnable=falseiscsiEnable=trueetcinitscvmclientconf

2 Run following initctlcommand to reloadconfiguration

initctl reload-configuration

3 Restart scvmclient process

stop scvmclient startscvmclient

4 Run following initctlcommand to mount SYSTEMDS

initctl emit --no-waitsystem-datastore-created

5 Verify that iscsiEnable tuneis set to true using thefollowing command

ps -eaf | grep scvmclient |grep -v grep

root 6241 1 1 Dec06 014007optspringpathstorfs-corescvmclient-T logEnabled=false -TlogSyslogEnabled=true -TlogEchoToScreen=false -TstatLoggingToFile=false -TstatLoggingToSyslog=true-TlogDir=varlogspringpath-TnfsBackendServerList=1010748100-T iscsiEnable=true -TiscsiUseAsync=true -T

Release Notes for Cisco HX Data Platform Release 4053

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

iscsiConfigFilePath=nfsSYSTEMistgtconfps -eaf | grep scvmclient |grep -v grep

35(2e)Following procedure can be usedto check NFS Queue Depth andincrease if needed

rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth256 lt- Low value[rootHXESXI1] vsish -e setvmkModulesnfsclientmnt[DATASTORE]propertiesmaxQDepth 1024[rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth1024 lt- Optimalvalue

This requires an ESXi host rebootto take effect

Please place the node in HyperflexMaintenanceMode and gracefullyreboot the node for the changes tobe applied

HyperFlex Datastore NFS Queue Depthshows as 256 which can lead toperformance (including latency) issues

CSCvr83056

40(1b)This is cosmetic The error will goaway once the cluster returns tohealthy status

When a single node is offline in clusterstcli cluster storage-summary shows twonodes unavailable rootSCVM~ stclicluster storage-summary

CSCvs47735

35(2g)Modify both the root and adminpassword for the SCVMs to be thesame

stcli security password set -uadmin

Cluster expansion fails at Config Installerstage when the root and admin passwordfor the storage controller (SCVM) aredifferent

CSCvs69317

35(2d)NAAfter successfully changing the DNSserver on the HX controller we still cansee the original DNS entry that was addedduring the deployment

CSCvs69154

Release Notes for Cisco HX Data Platform Release 4054

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Customers are strongly encouragedto work with TAC in order toidentify and remediate this issue

1 Change getcluster from 2m to8m inoptspringpathstorfs-mgmtstMgr-10confapplicationconfon all nodes

2 Restart stMgr on all nodes

You may see this error message after afailed upgrade or other task such asattempting to enter a node into HXmaintenance mode

CSCvs53555

40(2a)Run the following command oneach storage controller VM as root

grep -i monitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp sed -iesmonitor_dns_serverstruemonitor_dns_serversfalseoptspringpathhx-diag-toolswatchdog_configjsonampamp grep -imonitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp restart watchdog

Receive the following alertevent in HXConnect HX Controller VMHOSTNAME one or more configuredDNS servers not responding

CSCvt13947

35(2b)Delete any empty (Size 0) log filesunder varlogzookeeperversion-2ORvarlogzookeeperstandaloneversion-2

Zookeeper fails to start while Exhibitor isrunning however echo srvr returnsnothing

CSCvs21562

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

40(4e)1 De-commission the server

2 Power drain the server -REMOVE BOTH POWERCORDS ON THE BACK OFTHE SERVER FOR 10SECONDS THENREINSERTPOWERCORDS

3 Re-commission the server

M2 boot disk is missing from serverinventory after upgrade to server firmware40(4e) As a result server fails to boot toOS installed in the M2 disk The issuepersists after re-acknowledgement as wellas de-commission andre-acknowledgement of the server

CSCvs02466

Release Notes for Cisco HX Data Platform Release 4055

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Re-start the flusher manuallyMany disks taken offline by a fault ormanually resulting in ENOSPC oncluster

CSCvq32721

35(2e)Clean up fail over cluster and thenrecreate the fail over cluster Thereis no need to touch the HX storagecluster

[Hyper-V]When replicated DCwas usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2b)Manually install upgradescvmclient VIB andmake sure thatit matches with HXDP version

Upgrade from release HX 35(2a) to35(2x) 4X does not upgrade scvmclientRecovery point may not work

CSCvq54992

Management

30(1b)To recover the VM copy over thedata disks and attach them to thenew VM

[HyperV]VMshows up onmultiple nodesCSCvj22992

35(1a)1 Do not change MTU at ESXlevel on running cluster

2 Reset back to the original value

ChangingMTU from 9000 to 1500 on oneESX node causes storfs process to restarton all nodes

CSCvm99150

Release Notes for Cisco HX Data Platform Release 4056

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)After the installation (for newdeployments) or after an upgradeto 401a (for existingdeployments) perform thefollowing steps on all ControllerVMs

1 Edit the following tune files (onall Controller VMs)

optspringpathconfiglfftunes

optspringpathconfigvsi_16tbtunes

2 SetcleanerEnableSegSummaryCleaning

to false

3 After editing the above tunefiles

a SSH in to all the ControllerVMs

b Run storfstool -- -Z

c Run the below command andcheck the values Tune valueshould be true

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=true

d Type below commands andapplied the tunes changesdynamically

echo false gttmpstprocfssystemtunecleanerEnableSegSummaryCleaning

e Verify that tune values arechanged Run below commandValue should be false

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=false

f umount tmpstprocfs

A cluster with 8TB or 12TB disk drivesmay experience IO stalls for severalminutes after another node fails in thecluster

CSCvp23718

Release Notes for Cisco HX Data Platform Release 4057

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)1 Update the conf file manuallyto change to the link local addressto

2 Restart scvmclient on thecontroller and ESX

HX 35(2b) FlexVol to HX 40 CSIupgrade does not work FlexVol continuesto work

CSCvp26319

35(2a)Disable rebalance until node healsthen reenable rebalance

7-node cluster healing very slowlybecause rebalance change AMS reqs areblocking healing process

CSCvp62512

35(2a)NAWith moderate cluster storage capacityusage under site failover cluster clustermay experience an all paths (APD) downcondition

CSCvq49412

40(1a)NAThis product includes Third-partySoftware that is affected by thevulnerabilities identified by the followingCommon Vulnerability and Exposures(CVE) IDs

CVE-2016-0762 CVE-2016-6797CVE-2016-6816 CVE-2016-8735CVE-2017-5647 CVE-2017-12615CVE-2017-12616 CVE-2017-12617CVE-2017-7674 CVE-2018-1304CVE-2018-8014 CVE-2018-1336CVE-2018-8034 CVE-2018-11784CVE-2019-0232

CSCvp36364

35(2d)NAstcli cluster info command needs toprovide UCSM VIP address Currently itshows ucsm-hostcom

CSCvq11456

35(2a)No workarounds currently Thestale entries do not affect the dayto day operations of HyperFlex

HyperFlex upgrade validation failedbecause cluster has extraneous stNodes ininternal database

CSCvq32530

35(2a)NAWhen not in maintenancemode all drivesappear in HX Connect and get updated asexpected

When in maintenance mode extra drivesare shown in HX Connect with no slotnumbers

CSCvq42378

35(2b)Contact TAC to cleanup duplicatenode

Nodes duplicated in HX Connect aftercluster recreate

CSCvq61825

Release Notes for Cisco HX Data Platform Release 4058

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)NACurrently HyperFlex installs on Hyper-Vdo not contain the stcli security whitelistcommandset

CSCvq66245

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

Release Notes for Cisco HX Data Platform Release 4059

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)[Hyper-V] Converged node expansionfails for incoming node with VIC 14xx

CSCvq39523

Release Notes for Cisco HX Data Platform Release 4060

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

1 Set Jumbo frame inWindowsHypervisor for incomingnode

Get-NetAdapter storage-data-a|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

Get-NetAdapter storage-data-b|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

2 Verify all interfaces are setcorrectly to support Jumbo frames

Get-NetAdapter storage |Get-NetAdapterAdvancedProperty| RegistryKeyword -matchjumbo | ft -auto

You should receive the followingmessage

Name DisplayName DisplayValue

RegistryKeyword RegistryValue

---- ----------- ------------

--------------- -------------

vswitch-hx-storage-data Jumbo

Packet 9014 Bytes

JumboPacket 9014

storage-data-b Jumbo Packet

Bytes 9014 JumboPacket

9014

storage-data-a Jumbo Packet

Bytes 9014 JumboPacket

9014

3 Reboot only the incoming nodeor the node being expanded

4 Retry Cluster Expansion in

Release Notes for Cisco HX Data Platform Release 4061

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

DefectFound inRelease

WorkaroundSymptomDefect ID

iscsiConfigFilePath=nfsSYSTEMistgtconfps -eaf | grep scvmclient |grep -v grep

35(2e)Following procedure can be usedto check NFS Queue Depth andincrease if needed

rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth256 lt- Low value[rootHXESXI1] vsish -e setvmkModulesnfsclientmnt[DATASTORE]propertiesmaxQDepth 1024[rootHXESXI1] vsish -e getvmkModulesnfsclientmnt[DATASTORE]properties| grep -i maxqdepthmaxQDepth1024 lt- Optimalvalue

This requires an ESXi host rebootto take effect

Please place the node in HyperflexMaintenanceMode and gracefullyreboot the node for the changes tobe applied

HyperFlex Datastore NFS Queue Depthshows as 256 which can lead toperformance (including latency) issues

CSCvr83056

40(1b)This is cosmetic The error will goaway once the cluster returns tohealthy status

When a single node is offline in clusterstcli cluster storage-summary shows twonodes unavailable rootSCVM~ stclicluster storage-summary

CSCvs47735

35(2g)Modify both the root and adminpassword for the SCVMs to be thesame

stcli security password set -uadmin

Cluster expansion fails at Config Installerstage when the root and admin passwordfor the storage controller (SCVM) aredifferent

CSCvs69317

35(2d)NAAfter successfully changing the DNSserver on the HX controller we still cansee the original DNS entry that was addedduring the deployment

CSCvs69154

Release Notes for Cisco HX Data Platform Release 4054

Open Caveats in Release 40(2a)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Customers are strongly encouragedto work with TAC in order toidentify and remediate this issue

1 Change getcluster from 2m to8m inoptspringpathstorfs-mgmtstMgr-10confapplicationconfon all nodes

2 Restart stMgr on all nodes

You may see this error message after afailed upgrade or other task such asattempting to enter a node into HXmaintenance mode

CSCvs53555

40(2a)Run the following command oneach storage controller VM as root

grep -i monitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp sed -iesmonitor_dns_serverstruemonitor_dns_serversfalseoptspringpathhx-diag-toolswatchdog_configjsonampamp grep -imonitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp restart watchdog

Receive the following alertevent in HXConnect HX Controller VMHOSTNAME one or more configuredDNS servers not responding

CSCvt13947

35(2b)Delete any empty (Size 0) log filesunder varlogzookeeperversion-2ORvarlogzookeeperstandaloneversion-2

Zookeeper fails to start while Exhibitor isrunning however echo srvr returnsnothing

CSCvs21562

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

40(4e)1 De-commission the server

2 Power drain the server -REMOVE BOTH POWERCORDS ON THE BACK OFTHE SERVER FOR 10SECONDS THENREINSERTPOWERCORDS

3 Re-commission the server

M2 boot disk is missing from serverinventory after upgrade to server firmware40(4e) As a result server fails to boot toOS installed in the M2 disk The issuepersists after re-acknowledgement as wellas de-commission andre-acknowledgement of the server

CSCvs02466

Release Notes for Cisco HX Data Platform Release 4055

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Re-start the flusher manuallyMany disks taken offline by a fault ormanually resulting in ENOSPC oncluster

CSCvq32721

35(2e)Clean up fail over cluster and thenrecreate the fail over cluster Thereis no need to touch the HX storagecluster

[Hyper-V]When replicated DCwas usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2b)Manually install upgradescvmclient VIB andmake sure thatit matches with HXDP version

Upgrade from release HX 35(2a) to35(2x) 4X does not upgrade scvmclientRecovery point may not work

CSCvq54992

Management

30(1b)To recover the VM copy over thedata disks and attach them to thenew VM

[HyperV]VMshows up onmultiple nodesCSCvj22992

35(1a)1 Do not change MTU at ESXlevel on running cluster

2 Reset back to the original value

ChangingMTU from 9000 to 1500 on oneESX node causes storfs process to restarton all nodes

CSCvm99150

Release Notes for Cisco HX Data Platform Release 4056

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)After the installation (for newdeployments) or after an upgradeto 401a (for existingdeployments) perform thefollowing steps on all ControllerVMs

1 Edit the following tune files (onall Controller VMs)

optspringpathconfiglfftunes

optspringpathconfigvsi_16tbtunes

2 SetcleanerEnableSegSummaryCleaning

to false

3 After editing the above tunefiles

a SSH in to all the ControllerVMs

b Run storfstool -- -Z

c Run the below command andcheck the values Tune valueshould be true

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=true

d Type below commands andapplied the tunes changesdynamically

echo false gttmpstprocfssystemtunecleanerEnableSegSummaryCleaning

e Verify that tune values arechanged Run below commandValue should be false

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=false

f umount tmpstprocfs

A cluster with 8TB or 12TB disk drivesmay experience IO stalls for severalminutes after another node fails in thecluster

CSCvp23718

Release Notes for Cisco HX Data Platform Release 4057

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)1 Update the conf file manuallyto change to the link local addressto

2 Restart scvmclient on thecontroller and ESX

HX 35(2b) FlexVol to HX 40 CSIupgrade does not work FlexVol continuesto work

CSCvp26319

35(2a)Disable rebalance until node healsthen reenable rebalance

7-node cluster healing very slowlybecause rebalance change AMS reqs areblocking healing process

CSCvp62512

35(2a)NAWith moderate cluster storage capacityusage under site failover cluster clustermay experience an all paths (APD) downcondition

CSCvq49412

40(1a)NAThis product includes Third-partySoftware that is affected by thevulnerabilities identified by the followingCommon Vulnerability and Exposures(CVE) IDs

CVE-2016-0762 CVE-2016-6797CVE-2016-6816 CVE-2016-8735CVE-2017-5647 CVE-2017-12615CVE-2017-12616 CVE-2017-12617CVE-2017-7674 CVE-2018-1304CVE-2018-8014 CVE-2018-1336CVE-2018-8034 CVE-2018-11784CVE-2019-0232

CSCvp36364

35(2d)NAstcli cluster info command needs toprovide UCSM VIP address Currently itshows ucsm-hostcom

CSCvq11456

35(2a)No workarounds currently Thestale entries do not affect the dayto day operations of HyperFlex

HyperFlex upgrade validation failedbecause cluster has extraneous stNodes ininternal database

CSCvq32530

35(2a)NAWhen not in maintenancemode all drivesappear in HX Connect and get updated asexpected

When in maintenance mode extra drivesare shown in HX Connect with no slotnumbers

CSCvq42378

35(2b)Contact TAC to cleanup duplicatenode

Nodes duplicated in HX Connect aftercluster recreate

CSCvq61825

Release Notes for Cisco HX Data Platform Release 4058

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)NACurrently HyperFlex installs on Hyper-Vdo not contain the stcli security whitelistcommandset

CSCvq66245

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

Release Notes for Cisco HX Data Platform Release 4059

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)[Hyper-V] Converged node expansionfails for incoming node with VIC 14xx

CSCvq39523

Release Notes for Cisco HX Data Platform Release 4060

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

1 Set Jumbo frame inWindowsHypervisor for incomingnode

Get-NetAdapter storage-data-a|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

Get-NetAdapter storage-data-b|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

2 Verify all interfaces are setcorrectly to support Jumbo frames

Get-NetAdapter storage |Get-NetAdapterAdvancedProperty| RegistryKeyword -matchjumbo | ft -auto

You should receive the followingmessage

Name DisplayName DisplayValue

RegistryKeyword RegistryValue

---- ----------- ------------

--------------- -------------

vswitch-hx-storage-data Jumbo

Packet 9014 Bytes

JumboPacket 9014

storage-data-b Jumbo Packet

Bytes 9014 JumboPacket

9014

storage-data-a Jumbo Packet

Bytes 9014 JumboPacket

9014

3 Reboot only the incoming nodeor the node being expanded

4 Retry Cluster Expansion in

Release Notes for Cisco HX Data Platform Release 4061

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Customers are strongly encouragedto work with TAC in order toidentify and remediate this issue

1 Change getcluster from 2m to8m inoptspringpathstorfs-mgmtstMgr-10confapplicationconfon all nodes

2 Restart stMgr on all nodes

You may see this error message after afailed upgrade or other task such asattempting to enter a node into HXmaintenance mode

CSCvs53555

40(2a)Run the following command oneach storage controller VM as root

grep -i monitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp sed -iesmonitor_dns_serverstruemonitor_dns_serversfalseoptspringpathhx-diag-toolswatchdog_configjsonampamp grep -imonitor_dns_serversoptspringpathhx-diag-toolswatchdog_configjsonampamp restart watchdog

Receive the following alertevent in HXConnect HX Controller VMHOSTNAME one or more configuredDNS servers not responding

CSCvt13947

35(2b)Delete any empty (Size 0) log filesunder varlogzookeeperversion-2ORvarlogzookeeperstandaloneversion-2

Zookeeper fails to start while Exhibitor isrunning however echo srvr returnsnothing

CSCvs21562

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

40(4e)1 De-commission the server

2 Power drain the server -REMOVE BOTH POWERCORDS ON THE BACK OFTHE SERVER FOR 10SECONDS THENREINSERTPOWERCORDS

3 Re-commission the server

M2 boot disk is missing from serverinventory after upgrade to server firmware40(4e) As a result server fails to boot toOS installed in the M2 disk The issuepersists after re-acknowledgement as wellas de-commission andre-acknowledgement of the server

CSCvs02466

Release Notes for Cisco HX Data Platform Release 4055

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Re-start the flusher manuallyMany disks taken offline by a fault ormanually resulting in ENOSPC oncluster

CSCvq32721

35(2e)Clean up fail over cluster and thenrecreate the fail over cluster Thereis no need to touch the HX storagecluster

[Hyper-V]When replicated DCwas usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2b)Manually install upgradescvmclient VIB andmake sure thatit matches with HXDP version

Upgrade from release HX 35(2a) to35(2x) 4X does not upgrade scvmclientRecovery point may not work

CSCvq54992

Management

30(1b)To recover the VM copy over thedata disks and attach them to thenew VM

[HyperV]VMshows up onmultiple nodesCSCvj22992

35(1a)1 Do not change MTU at ESXlevel on running cluster

2 Reset back to the original value

ChangingMTU from 9000 to 1500 on oneESX node causes storfs process to restarton all nodes

CSCvm99150

Release Notes for Cisco HX Data Platform Release 4056

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)After the installation (for newdeployments) or after an upgradeto 401a (for existingdeployments) perform thefollowing steps on all ControllerVMs

1 Edit the following tune files (onall Controller VMs)

optspringpathconfiglfftunes

optspringpathconfigvsi_16tbtunes

2 SetcleanerEnableSegSummaryCleaning

to false

3 After editing the above tunefiles

a SSH in to all the ControllerVMs

b Run storfstool -- -Z

c Run the below command andcheck the values Tune valueshould be true

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=true

d Type below commands andapplied the tunes changesdynamically

echo false gttmpstprocfssystemtunecleanerEnableSegSummaryCleaning

e Verify that tune values arechanged Run below commandValue should be false

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=false

f umount tmpstprocfs

A cluster with 8TB or 12TB disk drivesmay experience IO stalls for severalminutes after another node fails in thecluster

CSCvp23718

Release Notes for Cisco HX Data Platform Release 4057

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)1 Update the conf file manuallyto change to the link local addressto

2 Restart scvmclient on thecontroller and ESX

HX 35(2b) FlexVol to HX 40 CSIupgrade does not work FlexVol continuesto work

CSCvp26319

35(2a)Disable rebalance until node healsthen reenable rebalance

7-node cluster healing very slowlybecause rebalance change AMS reqs areblocking healing process

CSCvp62512

35(2a)NAWith moderate cluster storage capacityusage under site failover cluster clustermay experience an all paths (APD) downcondition

CSCvq49412

40(1a)NAThis product includes Third-partySoftware that is affected by thevulnerabilities identified by the followingCommon Vulnerability and Exposures(CVE) IDs

CVE-2016-0762 CVE-2016-6797CVE-2016-6816 CVE-2016-8735CVE-2017-5647 CVE-2017-12615CVE-2017-12616 CVE-2017-12617CVE-2017-7674 CVE-2018-1304CVE-2018-8014 CVE-2018-1336CVE-2018-8034 CVE-2018-11784CVE-2019-0232

CSCvp36364

35(2d)NAstcli cluster info command needs toprovide UCSM VIP address Currently itshows ucsm-hostcom

CSCvq11456

35(2a)No workarounds currently Thestale entries do not affect the dayto day operations of HyperFlex

HyperFlex upgrade validation failedbecause cluster has extraneous stNodes ininternal database

CSCvq32530

35(2a)NAWhen not in maintenancemode all drivesappear in HX Connect and get updated asexpected

When in maintenance mode extra drivesare shown in HX Connect with no slotnumbers

CSCvq42378

35(2b)Contact TAC to cleanup duplicatenode

Nodes duplicated in HX Connect aftercluster recreate

CSCvq61825

Release Notes for Cisco HX Data Platform Release 4058

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)NACurrently HyperFlex installs on Hyper-Vdo not contain the stcli security whitelistcommandset

CSCvq66245

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

Release Notes for Cisco HX Data Platform Release 4059

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)[Hyper-V] Converged node expansionfails for incoming node with VIC 14xx

CSCvq39523

Release Notes for Cisco HX Data Platform Release 4060

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

1 Set Jumbo frame inWindowsHypervisor for incomingnode

Get-NetAdapter storage-data-a|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

Get-NetAdapter storage-data-b|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

2 Verify all interfaces are setcorrectly to support Jumbo frames

Get-NetAdapter storage |Get-NetAdapterAdvancedProperty| RegistryKeyword -matchjumbo | ft -auto

You should receive the followingmessage

Name DisplayName DisplayValue

RegistryKeyword RegistryValue

---- ----------- ------------

--------------- -------------

vswitch-hx-storage-data Jumbo

Packet 9014 Bytes

JumboPacket 9014

storage-data-b Jumbo Packet

Bytes 9014 JumboPacket

9014

storage-data-a Jumbo Packet

Bytes 9014 JumboPacket

9014

3 Reboot only the incoming nodeor the node being expanded

4 Retry Cluster Expansion in

Release Notes for Cisco HX Data Platform Release 4061

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

DefectFound inRelease

WorkaroundSymptomDefect ID

35(2e)Re-start the flusher manuallyMany disks taken offline by a fault ormanually resulting in ENOSPC oncluster

CSCvq32721

35(2e)Clean up fail over cluster and thenrecreate the fail over cluster Thereis no need to touch the HX storagecluster

[Hyper-V]When replicated DCwas usedthen during install timeWindows failovercluster was not created successfully

CSCvq38279

35(2b)Manually install upgradescvmclient VIB andmake sure thatit matches with HXDP version

Upgrade from release HX 35(2a) to35(2x) 4X does not upgrade scvmclientRecovery point may not work

CSCvq54992

Management

30(1b)To recover the VM copy over thedata disks and attach them to thenew VM

[HyperV]VMshows up onmultiple nodesCSCvj22992

35(1a)1 Do not change MTU at ESXlevel on running cluster

2 Reset back to the original value

ChangingMTU from 9000 to 1500 on oneESX node causes storfs process to restarton all nodes

CSCvm99150

Release Notes for Cisco HX Data Platform Release 4056

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)After the installation (for newdeployments) or after an upgradeto 401a (for existingdeployments) perform thefollowing steps on all ControllerVMs

1 Edit the following tune files (onall Controller VMs)

optspringpathconfiglfftunes

optspringpathconfigvsi_16tbtunes

2 SetcleanerEnableSegSummaryCleaning

to false

3 After editing the above tunefiles

a SSH in to all the ControllerVMs

b Run storfstool -- -Z

c Run the below command andcheck the values Tune valueshould be true

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=true

d Type below commands andapplied the tunes changesdynamically

echo false gttmpstprocfssystemtunecleanerEnableSegSummaryCleaning

e Verify that tune values arechanged Run below commandValue should be false

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=false

f umount tmpstprocfs

A cluster with 8TB or 12TB disk drivesmay experience IO stalls for severalminutes after another node fails in thecluster

CSCvp23718

Release Notes for Cisco HX Data Platform Release 4057

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)1 Update the conf file manuallyto change to the link local addressto

2 Restart scvmclient on thecontroller and ESX

HX 35(2b) FlexVol to HX 40 CSIupgrade does not work FlexVol continuesto work

CSCvp26319

35(2a)Disable rebalance until node healsthen reenable rebalance

7-node cluster healing very slowlybecause rebalance change AMS reqs areblocking healing process

CSCvp62512

35(2a)NAWith moderate cluster storage capacityusage under site failover cluster clustermay experience an all paths (APD) downcondition

CSCvq49412

40(1a)NAThis product includes Third-partySoftware that is affected by thevulnerabilities identified by the followingCommon Vulnerability and Exposures(CVE) IDs

CVE-2016-0762 CVE-2016-6797CVE-2016-6816 CVE-2016-8735CVE-2017-5647 CVE-2017-12615CVE-2017-12616 CVE-2017-12617CVE-2017-7674 CVE-2018-1304CVE-2018-8014 CVE-2018-1336CVE-2018-8034 CVE-2018-11784CVE-2019-0232

CSCvp36364

35(2d)NAstcli cluster info command needs toprovide UCSM VIP address Currently itshows ucsm-hostcom

CSCvq11456

35(2a)No workarounds currently Thestale entries do not affect the dayto day operations of HyperFlex

HyperFlex upgrade validation failedbecause cluster has extraneous stNodes ininternal database

CSCvq32530

35(2a)NAWhen not in maintenancemode all drivesappear in HX Connect and get updated asexpected

When in maintenance mode extra drivesare shown in HX Connect with no slotnumbers

CSCvq42378

35(2b)Contact TAC to cleanup duplicatenode

Nodes duplicated in HX Connect aftercluster recreate

CSCvq61825

Release Notes for Cisco HX Data Platform Release 4058

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)NACurrently HyperFlex installs on Hyper-Vdo not contain the stcli security whitelistcommandset

CSCvq66245

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

Release Notes for Cisco HX Data Platform Release 4059

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)[Hyper-V] Converged node expansionfails for incoming node with VIC 14xx

CSCvq39523

Release Notes for Cisco HX Data Platform Release 4060

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

1 Set Jumbo frame inWindowsHypervisor for incomingnode

Get-NetAdapter storage-data-a|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

Get-NetAdapter storage-data-b|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

2 Verify all interfaces are setcorrectly to support Jumbo frames

Get-NetAdapter storage |Get-NetAdapterAdvancedProperty| RegistryKeyword -matchjumbo | ft -auto

You should receive the followingmessage

Name DisplayName DisplayValue

RegistryKeyword RegistryValue

---- ----------- ------------

--------------- -------------

vswitch-hx-storage-data Jumbo

Packet 9014 Bytes

JumboPacket 9014

storage-data-b Jumbo Packet

Bytes 9014 JumboPacket

9014

storage-data-a Jumbo Packet

Bytes 9014 JumboPacket

9014

3 Reboot only the incoming nodeor the node being expanded

4 Retry Cluster Expansion in

Release Notes for Cisco HX Data Platform Release 4061

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)After the installation (for newdeployments) or after an upgradeto 401a (for existingdeployments) perform thefollowing steps on all ControllerVMs

1 Edit the following tune files (onall Controller VMs)

optspringpathconfiglfftunes

optspringpathconfigvsi_16tbtunes

2 SetcleanerEnableSegSummaryCleaning

to false

3 After editing the above tunefiles

a SSH in to all the ControllerVMs

b Run storfstool -- -Z

c Run the below command andcheck the values Tune valueshould be true

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=true

d Type below commands andapplied the tunes changesdynamically

echo false gttmpstprocfssystemtunecleanerEnableSegSummaryCleaning

e Verify that tune values arechanged Run below commandValue should be false

cattmpstprocfssystemtunecleanerEnableSegSummaryCleaning

cleanerEnableSegSummaryCleaning=false

f umount tmpstprocfs

A cluster with 8TB or 12TB disk drivesmay experience IO stalls for severalminutes after another node fails in thecluster

CSCvp23718

Release Notes for Cisco HX Data Platform Release 4057

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)1 Update the conf file manuallyto change to the link local addressto

2 Restart scvmclient on thecontroller and ESX

HX 35(2b) FlexVol to HX 40 CSIupgrade does not work FlexVol continuesto work

CSCvp26319

35(2a)Disable rebalance until node healsthen reenable rebalance

7-node cluster healing very slowlybecause rebalance change AMS reqs areblocking healing process

CSCvp62512

35(2a)NAWith moderate cluster storage capacityusage under site failover cluster clustermay experience an all paths (APD) downcondition

CSCvq49412

40(1a)NAThis product includes Third-partySoftware that is affected by thevulnerabilities identified by the followingCommon Vulnerability and Exposures(CVE) IDs

CVE-2016-0762 CVE-2016-6797CVE-2016-6816 CVE-2016-8735CVE-2017-5647 CVE-2017-12615CVE-2017-12616 CVE-2017-12617CVE-2017-7674 CVE-2018-1304CVE-2018-8014 CVE-2018-1336CVE-2018-8034 CVE-2018-11784CVE-2019-0232

CSCvp36364

35(2d)NAstcli cluster info command needs toprovide UCSM VIP address Currently itshows ucsm-hostcom

CSCvq11456

35(2a)No workarounds currently Thestale entries do not affect the dayto day operations of HyperFlex

HyperFlex upgrade validation failedbecause cluster has extraneous stNodes ininternal database

CSCvq32530

35(2a)NAWhen not in maintenancemode all drivesappear in HX Connect and get updated asexpected

When in maintenance mode extra drivesare shown in HX Connect with no slotnumbers

CSCvq42378

35(2b)Contact TAC to cleanup duplicatenode

Nodes duplicated in HX Connect aftercluster recreate

CSCvq61825

Release Notes for Cisco HX Data Platform Release 4058

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)NACurrently HyperFlex installs on Hyper-Vdo not contain the stcli security whitelistcommandset

CSCvq66245

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

Release Notes for Cisco HX Data Platform Release 4059

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)[Hyper-V] Converged node expansionfails for incoming node with VIC 14xx

CSCvq39523

Release Notes for Cisco HX Data Platform Release 4060

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

1 Set Jumbo frame inWindowsHypervisor for incomingnode

Get-NetAdapter storage-data-a|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

Get-NetAdapter storage-data-b|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

2 Verify all interfaces are setcorrectly to support Jumbo frames

Get-NetAdapter storage |Get-NetAdapterAdvancedProperty| RegistryKeyword -matchjumbo | ft -auto

You should receive the followingmessage

Name DisplayName DisplayValue

RegistryKeyword RegistryValue

---- ----------- ------------

--------------- -------------

vswitch-hx-storage-data Jumbo

Packet 9014 Bytes

JumboPacket 9014

storage-data-b Jumbo Packet

Bytes 9014 JumboPacket

9014

storage-data-a Jumbo Packet

Bytes 9014 JumboPacket

9014

3 Reboot only the incoming nodeor the node being expanded

4 Retry Cluster Expansion in

Release Notes for Cisco HX Data Platform Release 4061

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)1 Update the conf file manuallyto change to the link local addressto

2 Restart scvmclient on thecontroller and ESX

HX 35(2b) FlexVol to HX 40 CSIupgrade does not work FlexVol continuesto work

CSCvp26319

35(2a)Disable rebalance until node healsthen reenable rebalance

7-node cluster healing very slowlybecause rebalance change AMS reqs areblocking healing process

CSCvp62512

35(2a)NAWith moderate cluster storage capacityusage under site failover cluster clustermay experience an all paths (APD) downcondition

CSCvq49412

40(1a)NAThis product includes Third-partySoftware that is affected by thevulnerabilities identified by the followingCommon Vulnerability and Exposures(CVE) IDs

CVE-2016-0762 CVE-2016-6797CVE-2016-6816 CVE-2016-8735CVE-2017-5647 CVE-2017-12615CVE-2017-12616 CVE-2017-12617CVE-2017-7674 CVE-2018-1304CVE-2018-8014 CVE-2018-1336CVE-2018-8034 CVE-2018-11784CVE-2019-0232

CSCvp36364

35(2d)NAstcli cluster info command needs toprovide UCSM VIP address Currently itshows ucsm-hostcom

CSCvq11456

35(2a)No workarounds currently Thestale entries do not affect the dayto day operations of HyperFlex

HyperFlex upgrade validation failedbecause cluster has extraneous stNodes ininternal database

CSCvq32530

35(2a)NAWhen not in maintenancemode all drivesappear in HX Connect and get updated asexpected

When in maintenance mode extra drivesare shown in HX Connect with no slotnumbers

CSCvq42378

35(2b)Contact TAC to cleanup duplicatenode

Nodes duplicated in HX Connect aftercluster recreate

CSCvq61825

Release Notes for Cisco HX Data Platform Release 4058

Open Caveats in Release 40(1b)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)NACurrently HyperFlex installs on Hyper-Vdo not contain the stcli security whitelistcommandset

CSCvq66245

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

Release Notes for Cisco HX Data Platform Release 4059

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)[Hyper-V] Converged node expansionfails for incoming node with VIC 14xx

CSCvq39523

Release Notes for Cisco HX Data Platform Release 4060

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

1 Set Jumbo frame inWindowsHypervisor for incomingnode

Get-NetAdapter storage-data-a|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

Get-NetAdapter storage-data-b|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

2 Verify all interfaces are setcorrectly to support Jumbo frames

Get-NetAdapter storage |Get-NetAdapterAdvancedProperty| RegistryKeyword -matchjumbo | ft -auto

You should receive the followingmessage

Name DisplayName DisplayValue

RegistryKeyword RegistryValue

---- ----------- ------------

--------------- -------------

vswitch-hx-storage-data Jumbo

Packet 9014 Bytes

JumboPacket 9014

storage-data-b Jumbo Packet

Bytes 9014 JumboPacket

9014

storage-data-a Jumbo Packet

Bytes 9014 JumboPacket

9014

3 Reboot only the incoming nodeor the node being expanded

4 Retry Cluster Expansion in

Release Notes for Cisco HX Data Platform Release 4061

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)NACurrently HyperFlex installs on Hyper-Vdo not contain the stcli security whitelistcommandset

CSCvq66245

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Install Upgrade Expand

Release Notes for Cisco HX Data Platform Release 4059

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)[Hyper-V] Converged node expansionfails for incoming node with VIC 14xx

CSCvq39523

Release Notes for Cisco HX Data Platform Release 4060

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

1 Set Jumbo frame inWindowsHypervisor for incomingnode

Get-NetAdapter storage-data-a|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

Get-NetAdapter storage-data-b|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

2 Verify all interfaces are setcorrectly to support Jumbo frames

Get-NetAdapter storage |Get-NetAdapterAdvancedProperty| RegistryKeyword -matchjumbo | ft -auto

You should receive the followingmessage

Name DisplayName DisplayValue

RegistryKeyword RegistryValue

---- ----------- ------------

--------------- -------------

vswitch-hx-storage-data Jumbo

Packet 9014 Bytes

JumboPacket 9014

storage-data-b Jumbo Packet

Bytes 9014 JumboPacket

9014

storage-data-a Jumbo Packet

Bytes 9014 JumboPacket

9014

3 Reboot only the incoming nodeor the node being expanded

4 Retry Cluster Expansion in

Release Notes for Cisco HX Data Platform Release 4061

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)[Hyper-V] Converged node expansionfails for incoming node with VIC 14xx

CSCvq39523

Release Notes for Cisco HX Data Platform Release 4060

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

1 Set Jumbo frame inWindowsHypervisor for incomingnode

Get-NetAdapter storage-data-a|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

Get-NetAdapter storage-data-b|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

2 Verify all interfaces are setcorrectly to support Jumbo frames

Get-NetAdapter storage |Get-NetAdapterAdvancedProperty| RegistryKeyword -matchjumbo | ft -auto

You should receive the followingmessage

Name DisplayName DisplayValue

RegistryKeyword RegistryValue

---- ----------- ------------

--------------- -------------

vswitch-hx-storage-data Jumbo

Packet 9014 Bytes

JumboPacket 9014

storage-data-b Jumbo Packet

Bytes 9014 JumboPacket

9014

storage-data-a Jumbo Packet

Bytes 9014 JumboPacket

9014

3 Reboot only the incoming nodeor the node being expanded

4 Retry Cluster Expansion in

Release Notes for Cisco HX Data Platform Release 4061

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

DefectFound inRelease

WorkaroundSymptomDefect ID

1 Set Jumbo frame inWindowsHypervisor for incomingnode

Get-NetAdapter storage-data-a|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

Get-NetAdapter storage-data-b|Get-NetAdapterAdvancedProperty-RegistryKeywordJumboPacket |Set-NetAdapterAdvancedProperty-RegistryValue 9014

2 Verify all interfaces are setcorrectly to support Jumbo frames

Get-NetAdapter storage |Get-NetAdapterAdvancedProperty| RegistryKeyword -matchjumbo | ft -auto

You should receive the followingmessage

Name DisplayName DisplayValue

RegistryKeyword RegistryValue

---- ----------- ------------

--------------- -------------

vswitch-hx-storage-data Jumbo

Packet 9014 Bytes

JumboPacket 9014

storage-data-b Jumbo Packet

Bytes 9014 JumboPacket

9014

storage-data-a Jumbo Packet

Bytes 9014 JumboPacket

9014

3 Reboot only the incoming nodeor the node being expanded

4 Retry Cluster Expansion in

Release Notes for Cisco HX Data Platform Release 4061

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

DefectFound inRelease

WorkaroundSymptomDefect ID

installer (from the point where iterrored out previously)

40(1a)Confirm that both nodes are up andrunning and you have given itsome time to heal (a few hours)before attempting the workaroundIf not healed and failed-back runthe following command on bothcontroller VMs (preferablysimultaneously on both nodes) torestart restarthxRoboController

Two node HyperFlex Edge cluster maynot failback successfully and return tohealthy This can occur if connection toIntersight is highly impaired (egtransaction latencies exceeding 100ms)

CSCvp12241

40(1a)Ensure VC is available beforeperforming DS operations

Datastore createdelete fails when VC isnot available for ESXi HX cluster (regularor stretch or 2N robo)

CSCvp20102

30(1b)

35(1a)

Avoid removal and re-insertion ofdrives that are regarded as goodIf you have performed testing thathas removed and reinserted gooddrives contact Cisco TAC forfurther instructions

Emergency cluster shuts down in HX30(1b) after exiting host from HXMaintenance Mode and storfs panic onother node

CSCvk23212

35(1a)Wait at least 30 minutes for theAD policy to take effect If theissue is not resolved reboot 1 hostat a time usingmaintenancemode

During Hyper-V installation if youchoose to perform constrained delegationlater sometimes it takes an excessiveamount of time to reflect on HX ConnectUI

CSCvm55176

35(2a)Add the ISCSI interface manuallyafter install

Installer process fails when user selectsoption to setup ISCSI interface

CSCvo88857

35(2b)

35(1a)

1 Run the following commandstcli cluster upgrade--components hxdp --clean

2 Re-run the HX upgrade

3 Repeat steps 1 and 2 asnecessary

Stretch cluster upgrade fails randomly dueto Cluster invalid state

CSCvp20230

35(1a)

30(1e)

This error indicates that Windowsfailed to find a domain controllerPlease add a specific IP of DomainController in the Advanced inputof HXInstaller

HX Hyper-V installation fails andHXBootstraplog contains thefollowing message

Unable to find a defaultserver with Active DirectoryWeb Services running

CSCvm53679

Release Notes for Cisco HX Data Platform Release 4062

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

DefectFound inRelease

WorkaroundSymptomDefect ID

Management

40(1a)There are two options

A) Wait the 15-20 minutes for theARP entry to time out then thedatastore will become mountedagain

B) Manually reset the ARP entryby following these steps

1 Check the current eth10 macaddress from both HyperFlexController VMs and node theIPMac

Run ifconfig eth10 from theshell of the controller VM

2 On the node where the datastoreis unavailable check the macaddress entries for the above IPaddresses in the ESXi ARP cacheusing the following command

esxcli network ip neighbor list

3 If the IP address is assigned tothe incorrect mac address purgethe entry from the ARP table asshown below

esxcli network ip neighbor

remove -a ltIP_addressgt -v 4

After 2 node network partition healsdatastore on the isolated node is notavailable for 15 minutes

CSCvp98910

35(2b)If upgrading to 35(2b) or 40I(1a)enabling the RecoverPoint featuremay require you to perform arolling node by node HXMaintenance Mode in the clusterEnsure that the cluster is healthyand can tolerate one node failure(for 3 or 4 node clusters) and 2node failures (for 5 or greater than5 node clusters)

Deploy of EMC RecoverPoint fails witherror Failed findingrepository device in thevRPA view

CSCvp21417

Release Notes for Cisco HX Data Platform Release 4063

Open Caveats in Release 40(1a)

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

DefectFound inRelease

WorkaroundSymptomDefect ID

40(1a)

35(2b)

Remove the Micron 5200 drivesfrom the system Upgrade toRelease 35(2b) then followdisk-add expansion workflow

In the event of adding unsupportedMicron5200 drives to an HX cluster and thenupgrading HX to a release that supportsthem the drives could get locked if thecluster has remote security enabled(during certain cases like continuousreboot of controller VMs)

CSCvo89507

35(1a)NAHX Datastore usage is higher than thecombined usage of the VMs on thedatastore

CSCvn76916

35(1a)Retake the snapshotVM powers off during backup VMsnapshot

CSCvo83276

35(1a)

30(1e)

1 Verify if the interface is up andif you can ping the loopbackinterface

ifconfig -aping 127001

2 Bring up the loopback interfaceip link set lo up

3 Check that the service isrunningstatus scvmclientstatusstorfs

4 Start the followingservicesstart scvmclientstartstorfs

HX node keeps crashing after node isrestarted

CSCvn11045

35(2b)Use the command stcli servicessch show instead

Cluster info shows Smart call home isenabled even though it is disabled

CSCvp09978

Related Caveats

Resolved inRelease

Defect Found inRelease

SymptomDefect ID

OpenCisco IMC40(1a)

When attempting to install ESXi 65 or 67 from a CIMCmounted ISO with an embedded kickstart file theinstallationmay fail when reading the embeddedKSCFGfile In the ESXi installer a popup error will state

Could not open file ltpathgtKSCFG

CSCvq41985

Release Notes for Cisco HX Data Platform Release 4064

Related Caveats

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

Related Documentation

DescriptionDocument

Provides an editable file for gathering required configurationinformation prior to starting an installation This checklist must be filledout and returned to a Cisco account team

Preinstallation Checklist

Provides detailed information about Day 0 configuration of HyperFlexSystems and related post cluster configuration tasks It also describeshow to set up multiple HX clusters expand an HX cluster set up amixed HX cluster and attach external storage

Installation Guide for VMwareESXi

Provides installation and configuration procedures for HyperFlexStretched cluster enabling you to deploy an Active-Active disasteravoidance solution for mission critical workloads

Stretched Cluster Guide

Provides installation and configuration procedures on how to installand configure Cisco HyperFlex Systems on Microsoft Hyper-V

Installation Guide on MicrosoftHyper-V

Provides deployment procedures for HyperFlex Edge designed to bringhyperconvergence to remote and branch office (ROBO) and edgeenvironments

Edge Deployment Guide

Provides information about how to manage and monitor the clusterencryption data protection (replication and recovery) ReadyClonesNative snapshots and user management Interfaces includeHXConnectHX Data Platform Plug-in and the stcli commands

Administration Guide

Provides information about how to manage and monitor the Hyper-Vcluster encryption data protection (replication and recovery)ReadyClones Hyper-V Checkpoints and user management Interfacesinclude Cisco HyperFlex Systems and the hxcli commands

Administration Guide for Hyper-V

Provides information about HyperFlex storage integration forKubernetes information on Kubernetes support in HyperFlex Connectand instructions on how to configure CiscoHyperFlex Container StorageInterface (CSI) storage integration for both the Cisco container platformand the RedHat OpenShift container platform

Administration Guide forKubernetes

Provides installation configuration and deployment procedures for aHyperFlex system to connect to CitrixWorkspaces and associated CitrixCloud subscription services such as Citrix Virtual Apps and DesktopsServices The Citrix Ready HCIWorkspace Appliance program enablesa Cisco HyperFlex System deployed on Microsoft Hyper-V to connectto Citrix Cloud

Administration Guide for CitrixWorkspace Appliance

Provides installation configuration and deployment procedures forHyperFlex Intersight designed to deliver secure infrastructuremanagement anywhere from the cloud

HyperFlex Intersight InstallationGuide

Release Notes for Cisco HX Data Platform Release 4065

Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation

DescriptionDocument

Provides information on how to upgrade an existing installation of CiscoHX Data Platform upgrade guidelines and information about variousupgrade tasks

Upgrade Guide

Provides information about HyperFlex Systems specific network andexternal storage management tasks

Network and External StorageManagement Guide

Provides CLI reference information for HX Data Platform stcli

commandsCommand Line Interface (CLI)Guide

Provides information on how to use the Cisco PowerShell CiscoHXPowerCLI cmdlets for Data Protection

Cisco HyperFlex PowerShellCmdlets for Disaster Recovery

Provides information related to REST APIs that enable externalapplications to interface directly with the Cisco HyperFlex managementplane

REST API Getting Started Guide

REST API Reference

Provides troubleshooting for installation configuration to configurationand to configuration In addition this guide provides information aboutunderstanding system events errors Smart Call Home and Ciscosupport

Troubleshooting Guide

Provides independent knowledge base articlesTechNotes

Provides information on recommended FIServer firmwareRelease Notes for UCS ManagerRelease 40

Release Notes for Cisco HX Data Platform Release 4066

Related Documentation

  • Release Notes for Cisco HX Data Platform Release 40
  • Introduction
    • Revision History
    • New Features
    • Supported Versions and System Requirements
      • HyperFlex Software Versions
      • HyperFlex Edge and Firmware Compatibility Matrix for 4x Deployments
      • HyperFlex Licensing
      • HX Data Platform Software Versions for HyperFlex Witness Node
      • Software Requirements for VMware ESXi
      • Software Requirements for Microsoft Hyper-V
      • Browser Recommendations
        • Cisco HX Data Platform Storage Cluster Specifications
        • Guidelines and Limitations
        • Mixed Cluster Expansion Guidelines
        • Security Fixes
        • Resolved Caveats in Release 40(2b)
        • Resolved Caveats in Release 40(2a)
        • Resolved Caveats in Release 40(1b)
        • Resolved Caveats in Release 40(1a)
        • Open Caveats in Release 40(2b)
        • Open Caveats in Release 40(2a)
        • Open Caveats in Release 40(1b)
        • Open Caveats in Release 40(1a)
        • Related Caveats
        • Related Documentation