1712 dell emc cloud for microsoft azure stack · 10 1712 dell emc cloud for microsoft azure stack...

1712 Dell EMC Cloud for Microsoft Azure Stack Patch and Update Guide

Version A01 Dell Engineering January 2018

2 1712 Dell EMC Cloud for Microsoft Azure Stack Patch and Update Guide | version A00

Revisions

Date Version Description

Jan 2018 A00 Initial release

Jan 2018 A01 HLH updates, Meltdown, and Spectre

THIS GUIDE IS FOR INFORMATIONAL PURPOSES ONLY, AND MAY CONTAIN TYPOGRAPHICAL ERRORS AND TECHNICAL INACCURACIES.

THE CONTENT IS PROVIDED AS IS, WITHOUT EXPRESS OR IMPLIED WARRANTIES OF ANY KIND BY DELL EMC or MICROSOFT

Copyright © 2018 Dell Inc. All rights reserved. Dell and the Dell EMC logo are trademarks of Dell Inc. in the United States and/or other jurisdictions. All

other marks and names mentioned herein may be trademarks of their respective companies.


Table of contents Revisions............................................................................................................................................................................. 2

Patch and Update Overview ............................................................................................................................................... 5

Goal . ................................................................................................................................................................ 5

About Speculative Execution Side-Channel Vulnerabilities ......................................................................................... 5

Installing Patch and Updates ........................................................................................................................................ 6

Phase 1: Installing Microsoft updates ................................................................................................................................. 7

1a. Updating the Hardware Lifecycle Management Server [HLH] ............................................................................... 7

Installing the Windows Server 2016 Cumulative Update on the HLH Host ................................................................. 7

Enable the Mitigations by Setting Windows Server 2016 Registry Values .................................................................. 9

Updating the OME and OMNM VMs ............................................................................................................................ 9

1b. Running the Microsoft Patch and Updates on Azure Stack Scale Nodes. .......................................................... 13

Build reference ........................................................................................................................................................... 13

Prerequisites ............................................................................................................................................................... 13

Procedure ................................................................................................................................................................... 14

Phase 2: Running Dell EMC firmware Patch and Update framework .............................................................................. 22

Downloading contents ................................................................................................................................................ 22

Preparing the stamp for updates ................................................................................................................................ 22

2a. Applying the Dell EMC firmware Patch and Update on the Hardware Lifecycle Management Host [HLH] ......... 25

2b. Applying the Dell EMC firmware Patch and Update on Azure Stack Scale Nodes ............................................. 31

Draining the node – Maintenance mode .................................................................................................................... 31

Invoking the Dell EMC Patch and Update Script ........................................................................................................ 33

How to review updates ...................................................................................................................................................... 41

Monitor updates in Azure Stack using the privileged endpoint .................................................................................. 47

Verify the cmdlets are available ................................................................................................................................. 47

Use the update management cmdlets ....................................................................................................................... 48

Connect to the privileged endpoint and assign session variable ............................................................................... 49

Get high-level status of the current update run .......................................................................................................... 49

Get the full update run status with details .................................................................................................................. 49

Get the verbose progress log ..................................................................................................................................... 50

Actively view the verbose logging .............................................................................................................................. 50

Resume a failed update operation ............................................................................................................................. 51

Troubleshooting .......................................................................................................................................................... 51


New features and fixes ............................................................................................................................................... 51

Windows Server 2016 new features and fixes ........................................................................................................... 51

Known issues with the update process ............................................................................................................................. 52

Microsoft Known Issues ............................................................................................................................................. 52

Known issues (post-installation) ................................................................................................................................. 52

DELL EMC Known Issues .......................................................................................................................................... 55

Appendix A: Updating Security Policies on the OME VM ................................................................................................. 56


Patch and Update Overview

Goal .

Azure Stack operators are faced with the enormous challenge of keeping their solution both secure and

functional. They must ensure the solution is not vulnerable to threats–external or internal–while maintaining

negotiated service-level agreements.

About Speculative Execution Side-Channel Vulnerabilities

Dell EMC is aware of the side-channel analysis vulnerabilities (also known as Meltdown and Spectre)

affecting many modern microprocessors that were publicly described a team of security researchers on

January 3, 2018. This document addresses the specific steps for securing the servers within the Dell EMC

Cloud for Microsoft Azure Stack from these specific vulnerabilities.

In general, there are three steps that must be taken to implement full mitigations against these

attacks. These can be summarized as follows:

1. Patch the Operating System on the server (install Windows Server 2016 KB4056890)

2. Enable specific mitigations within the Operating System (apply registry modifications)

3. Update the Intel processor microcode on the server (flash the server BIOS).

In the context of the Azure stack solution:

The Installing the Windows Server 2016 Cumulative Update on the HLH Host, Enable the Mitigations

by Setting Windows Server 2016 Registry Values, and Updating the OME and OMNM VMs

subsections within this document explain how to apply these same updates to the physical HLH host,

as well as the OpenManage Essentials / Support Assist Enterprise and OpenManage Network

Manager Virtual Machines that reside on that host.

The Microsoft Azure Stack 1712 Update (Build 20180106.1) addresses steps #1 and #2 for the scale

unit hosts and the infrastructure VMs that comprise the Azure Stack solution. Following the

procedures in the 1b. Running the Microsoft Patch and Updates on Azure Stack Scale Nodes section

of this document will apply the necessary OS updates and registry configuration settings for the

physical scale unit hosts and the infrastructure VMs.

As of January 22, 2018, Intel has communicated new guidance regarding "reboot issues and

unpredictable system behavior" with the microcode included in the BIOS updates released to address

Spectre (Variant 2), CVE-2017-5715. Dell is advising that all customers should not deploy the BIOS

update for the Spectre (Variant 2) vulnerability at this time. We have removed the impacted BIOS

updates from our support pages and are working with Intel on a new BIOS update that will include

new microcode from Intel. For the latest information and recommended BIOS versions, please refer

to: http://www.dell.com/support/article/us/en/04/sln308588/microprocessor-side-channel-

vulnerabilities-cve-2017-5715-cve-2017-5753-cve-2017-5754-impact-on-dell-emc-products-dell-

enterprise-servers-storage-and-networking-?lang=en

https://newsroom.intel.com/news/root-cause-of-reboot-issue-identified-updated-guidance-for-customers-and-partners/

http://www.dell.com/support/article/us/en/04/sln308588/microprocessor-side-channel-vulnerabilities-cve-2017-5715-cve-2017-5753-cve-2017-5754-impact-on-dell-emc-products-dell-enterprise-servers-storage-and-networking-?lang=en




When these BIOS updates are released, Dell EMC will issue an updated Dell EMC Tools package for

Microsoft Azure Stack. This update will contain the BIOS update packages that can be applied using

the steps in 2a. Applying the Dell EMC firmware Patch and Update on the Hardware Lifecycle

Management Host [HLH] (which explains the procedure for the physical HLH host, including

suspending BitLocker prior to performing the update) and 2b. Applying the Dell EMC firmware Patch

and Update on Azure Stack Scale Nodes (which explains how to apply these updates to the physical

scale unit nodes).

Installing Patch and Updates

Installing Patch and Updates includes firmware updates for the hardware, and software updates for the

operating system and drivers. The Patch and Update process is a two-phase process:

1. Running Microsoft software Patch and Update framework

a. Hardware Lifecycle Management Server [HLH]

b. Azure Stack Scale Nodes

2. Running Dell EMC firmware Patch and Update framework

a. Hardware Lifecycle Management Server [HLH]

b. Azure Stack Scale Nodes

IMPORTANT: Normally, firmware patches and updates need to be installed first before running software

patches and updates. For 1712, due to side-channel analysis vulnerabilities (also known as Meltdown

and Spectre), the Microsoft Patch and Update process will need to be run first, and Dell EMC Patch and

Update process second.


Phase 1: Installing Microsoft updates Section 1a below provides the procedure for updating the hardware lifecycle manaement server. Section 1b

provides the details for running the Microsoft P&U framework against the scale nodes. These operations can

be performed in either order.

1a. Updating the Hardware Lifecycle Management Server [HLH]

Installing the Windows Server 2016 Cumulative Update on the HLH Host Complete the following steps to update Windows Server 2016 on the HLH host:

Step Activity

1 Log in to your HLH server.

2 Browse to C:\DELLEMCTools\ folder on your OME-VM where you extracted the DellEMC toolkit.

Go to the HLH_Cumulative_Windows_Update folder for the Windows Server 2016 x86_64

Cumulative Update for January 2018 (KB 4056890) and copy it onto the HLH host.

3 Run the update package.

Click “Yes” to allow the update to execute.


Step Activity

4 The update will progress through multiple stages and take several minutes to complete.

Note: This may take 25 minutes or more to complete.

5 When the update package has finished running, click “Restart Now” to reboot the computer and finish applying the updates.


Step Activity

6 Log in to Windows and open an elevated (Administrator) PowerShell.

Issue the following command and verify that KB4056890 has been installed:

Get-HotFix

Enable the Mitigations by Setting Windows Server 2016 Registry Values Complete the following steps to set registry values that enable the mitigations:

Step Activity

1 Open an elevated (Administrator) PowerShell. Issue the following three commands to set the registry values (each command should be entered as one continuous line, despite the wrapping in this document):

reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management" /v FeatureSettingsOverride /t REG_DWORD /d 0 /f

reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management" /v FeatureSettingsOverrideMask /t REG_DWORD /d 3 /f

reg add "HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Virtualization" /v MinVmVersionForCpuBasedMitigations /t REG_SZ /d "1.0" /f

For more information, refer to https://support.microsoft.com/en-us/help/4072698/windows-server-guidance-to-protect-against-the-speculative-execution

2 Execute the Restart-Computer cmdlet to reboot the HLH.

Updating the OME and OMNM VMs To further ensure that guest-to-host and guest-to-guest memory access is protected against potential exploits,

the OS update and registry settings should also be applied to the virtual machines that run on the HLH host.

https://support.microsoft.com/en-us/help/4072698/windows-server-guidance-to-protect-against-the-speculative-execution



Install the Cumulative Update on the OME and OMNM Virtual Machines Complete the following steps to update Windows Server 2016 on the OME and OMNM virtual machines:

Step Activity

1 From the HLH console, connect to the OME VM via RDP (or from Hyper-V Manager). Log on as the local Administrator.

2 Either place a copy of the Windows Server 2016 x86_64 Cumulative Update for January 2018 (KB 4056890) locally on the VM, or connect to a file share location that contains the update package.

3 Run the update package.

Click “Yes” to allow the update to execute.


Step Activity

4 The update will progress through multiple stages and take several minutes to complete.

Note: This may take 25 minutes or more to complete.

5 When the update package has finished running, click “Restart Now” to reboot the computer and finish applying the updates.

Note: The reboot may take several minutes due to portions of the update that run before Windows Server 2016 shuts down.


Step Activity

6 Log in to the VM (as in step 1) and open an elevated (Administrator) PowerShell. Issue the following command and verify that KB4056890 has been installed:

Get-HotFix

7 Repeat these same steps to apply the update on the OMNM virtual machine.


Enable Mitigations on the OME and OMNM Virtual Machines Complete the following steps to set registry values that enable the mitigations:

Step Activity

1 From the HLH console, connect to the OME VM via RDP (or from Hyper-V Manager). Log on as the local Administrator.

2 Open an elevated (Administrator) PowerShell. Issue the following three commands to set the registry values (each command should be entered as one continuous line, despite the wrapping in this document):

reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management" /v FeatureSettingsOverride /t REG_DWORD /d 0 /f

reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management" /v FeatureSettingsOverrideMask /t REG_DWORD /d 3 /f

reg add "HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Virtualization" /v MinVmVersionForCpuBasedMitigations /t REG_SZ /d "1.0" /f

For more information, refer to https://support.microsoft.com/en-us/help/4072698/windows-server-guidance-to-protect-against-the-speculative-execution

3 Execute the Restart-Computer cmdlet to reboot the virtual machine.

4 Repeat these same steps to apply the update on the OMNM virtual machine.

1b. Running the Microsoft Patch and Updates on Azure Stack Scale Nodes.

IMPORTANT: This update package is only applicable for Azure Stack integrated systems. Do not apply

this update package to the Azure Stack Development Kit.

Build reference The Azure Stack 1712 update build number is 180106.1. If a customer has deployed 180103.2 previously, you

do not need to apply 180106.1.

Prerequisites You must first install the Azure Stack 1710 Update and 1711 Update before applying this update.



https://docs.microsoft.com/azure/azure-stack/azure-stack-update-1710

https://docs.microsoft.com/azure/azure-stack/azure-stack-update-1711


Procedure The following procedure shows how to import and install updates as an Azure Stack operator.

Step Activity

1 Download the update package from Microsoft from the The Azure Stack 1712 update build number 180106.1 website. Scroll down the page to the section “Download the update” and download the package. An update package will typically consist of a single self-extracting executable (.exe), corresponding bin files (.bin) and a single metadata (.xml) file.

The <package>.exe file contains the payload for the update, for example the latest cumulative update for

Windows Server.

The corresponding <package>.bin file(s) provide compression for the payload as associated with the

executable.

The metadata.xml file contains essential information about the update, for example the publisher, name,

prerequisite, size and support path URL.

https://docs.microsoft.com/en-us/azure/azure-stack/azure-stack-update-1712


Step Activity

2 To import the update package to Azure Stack, in the administrator portal, under Data + Storage, click Storage Accounts.

3 In the filter box, type update, and select the updateadminaccount.


Step Activity

4 In the updateadminaccount storage account details, under Services, select Blobs.

5 On the Blob service tile, click + Container to create a new container, give it a name (for example, update-1709), and then click OK.


Step Activity

6 After the container is created, click Upload to upload the <package>.exe, any associated .bin files, and the metadata.xml files into the container.


Step Activity

7 Browse to the <package>.exe file, and then click Open in the file explorer window.

8 Next, click Upload in the administrator portal.


Step Activity

9 Do the same for the <package>.bin and metadata.xml files.

10 When done, you can review the Notifications. A notification should indicate that upload has completed.


Step Activity

11 Navigate back to the Update tile to review the newly-added update package.

12 To install an update, select the package marked as Ready and either right-click and select Update now, or click Update now in the command bar at the top of the window.


Step Activity


Phase 2: Running Dell EMC firmware Patch and Update framework

Note: When you run Dell EMC firmware Patch and Update and it informs that no updates available, that

means you are at the latest level of firmware level.

Downloading contents

Download the DellEMC Tools compressed file (Cloud for Microsoft Azure Stack 13G Toolkit

1.0.1712.2.zip) from the Dell EMC Support Downloads Webpage onto your OME-VM. Right click the zip file to

extract its contents into a folder, for example C:\DELLEMCTools. This toolkit has, among various DELLEMC

tools, the framework and firmware installation files required for running Dell EMC post-deployment / FRU

scenario firmware Patch and Update process.

Preparing the stamp for updates

Step Activity

1 Browse to the folder C:\DELLEMCTools where the extracted contents are located. In this folder, the contents required for the post-deployment Dell EMC Firmware Patch and Update are:

The Post Deploy Firmware Patch and Update Azure Stack folder

The Firmware folder.

https://support.emc.com/downloads/42238_Cloud-for-Microsoft-Azure-Stack


Step Activity

2 Right-click the Firmware folder (this folder holds all the firmware executables that iDRAC will use to update the firmware). Now select the Sharing tab. Then click Advanced Sharing… and share this folder. In our example we called the share “Firmware”, which is the default. You can call it a different name.

IMPORTANT: Please take note of this share name as we will use it later when we invoke the Firmware P&U script. Click OK. After share creation, we move to draining the node into maintenance mode.

3 Check if the SMB v1 is enabled on the share.

On the “firmware” share that was created in the prior step, where the DUPs executables are located, perform the following:

a. Check to see if SMBv1 is enabled in the server configuration by running the following command: Get-SmbServerConfiguration

Note: The commands indicated must be run in a PowerShell window as Administrator.


Step Activity

b. If SMBv1 is enabled (displays True; highlighted yellow in the following screenshot), proceed to the next step (Draining the Node – Maintenance Mode).

i. Post completion of P&U framework disable the SMBv1 configuration (See steps below to disable

SMBv1 configuration) c. If SMBv1 support is disabled, then run the following command to enable SMBv1 configuration on the

node and acknowledge the operation with a "Y" when prompted. Sample output is listed below: PS C:\Windows\system32> Set-SmbServerConfiguration -EnableSMB1Protocol $true

i. Confirm ii. Are you sure you want to perform this action? iii. Performing operation 'Modify' on Target 'SMB Server Configuration'. iv. [Y] Yes [A] Yes to All [N] No [L] No to All [S] Suspend [?] Help (default is "Y"): Y v. Ensure that the configuration now indicates that SMBv1 is enabled (show in the image above).

d. Run the P&U process as per the documentation. e. Post completion of P&U process disable the SMBv1 configuration (see the steps below).

Steps to disable SMBv1 configuration

Note: The commands indicated must be run in a PowerShell window as Administrator.

1. Disable SMBv1 support by running the following command:

Set-SmbServerConfiguration -EnableSMB1Protocol $false

a. Acknowledge by typing in a "Y" at the prompt.

b. Ensure that SMBv1 support is now off by using the Get-SmbServerConfiguration cmdlet. Now

the EnableSMB1Protocol will display False.


2a. Applying the Dell EMC firmware Patch and Update on the Hardware

Lifecycle Management Host [HLH]

1. Suspend BitLocker by performing the following steps:

Step Activity

a Log into your HLH server.

b Verify that a BitLocker Recovery Password for the C: drive is available: a) Open an elevated (Administrator) PowerShell. b) Run the following command:

(Get-BitLockerVolume -MountPoint “C:”).KeyProtector

c) Make sure that a Key Protector of Type “RecoveryPassword” is listed, and that this numerical password has been saved somewhere outside of the HLH host. This will make it possible to boot the OS in the event that something goes wrong with the update process.

d) If a RecoveryPassword is NOT present, run the following command:

Add-BitLockerKeyProtector –MountPoint “C:” –

RecoveryPasswordProtector

The new password will be displayed on screen when it is created, or you can repeat the command from “b” above to display it again. It is also advisable to check whether a Recovery Password is available for the D: drive, and to create this protector if it is not present. Simply substitute “D:” in place of “C:” in the commands within this step to check for the presence of the password or to create it if it is absent.


Step Activity

c Suspend BitLocker on the OS volume: a) Open an elevated (Administrator) PowerShell. b) Run the following command:

Suspend-BitLocker –MountPoint “C:”

c) Verify that the “Protection Status” shows “Off”.

2. Now switch over and log into the OME VM to invoke the Firmware P&U script.

3. Open a new PowerShell window with Administrator privileges.

4. Browse to the C:\DELLEMCTools folder where you extracted the DELLEMC Tools zip file and

change directory to the “Post Deploy Firmware Patch and Update Azure Stack” folder.

5. Type the following command at the prompt and press Enter to load all the modules required for

Firmware Patch and Update framework.

Import-Module .\DELLEMCFirmwareUpdate.ps1


6. Type the following command at the prompt.

Note: For the -IPAddress parameter, enter the IP address of the [HLH] server’s iDRAC. The -

Remediate paramater installs the firmware. If you do not use the -Remediate parameter, only the

inventory of the firmware will be printed out. Press Enter.

Invoke-CheckFirmwareBaseline -IPAddress 10.10.10.10 -Remediate

Note: After invoking the script if you receive and error that looks something like “WARNING: Caught

exception -> WinRM cannot complete the operation”, please consult the Dell EMC Known issues

section

Note: If you encounter errors such as “Cannot invoke method. Method invocation is

supported only on core types in this language mode.” when attempting to execute

the Dell EMC firmware updates from the OME VM, see Appendix A: Updating Security Policies on the

OME VM.

7. After you press Enter, you will be prompted for the iDRAC username (idracUser). Enter the correct

iDRAC username. In the example below, we used root.


8. Next, you will be prompted to provide the iDRAC password (idracPass). Enter the correct iDRAC

password.

9. Next, you will be prompted to provide the name of the share where the firmware executables are

located (FirmwareShare). This is the same share we created in Step 2 of Preparing the stamp for

updates. In our example, we called it Firmware.

10. Next, you will be prompted for the Host IP address where the Firmware Share is located

(FirmwareShareHost). This is the IP address of the OMEVM where we created the share and

executing this script. Since we are invoking the scripts from OME-VM, it will be the local IPv4 address

of the OME-VM.


11. Next, you will be prompted to enter the credentials for the account on the OMEVM that has access to

the Firmware Share (FirmwareShareCredential). In our example below, we used Administrator.

At this step, the framework will parse the catalog file and perform health checks, among other

functions.


The framework will now compare the firmware inventory against the catalog file (Support Matrix

compatible) and print out the results of firmware that is not compliant and needs remediation.

If -Remediate option was used, the framework will start the remediation process now. This includes

creating update jobs and then polling the firmware update jobs. This can be seen from the few

screenshots below.

IMPORTANT: The OME-VM runs on the HLH server. Since the OME-VM is trying to update the HLH

Firmware, the host will restart and the PowerShell session will be lost. Once HLH reboots, please log

into the OME-VM and run the same Firmware Update process for HLH until all the Firmware have

been remediated


2b. Applying the Dell EMC firmware Patch and Update on Azure Stack

Scale Nodes

Draining the node – Maintenance mode

1. Log in to the Azure Stack Administration portal, and from the Dashboard, click the Region

management tile.


2. In the left pane, click Scale Units and then in the right pane, click S-Cluster.

Clicking the S-Cluster displays all the nodes in the cluster.

3. Click the first node, (in the example below; SAC21-Node01). A new tile appears on the right, showing

the option to drain the node into maintenance mode. Click the Drain button for the first node.


The notification center will indicate Draining the node – Running.

Once the node has successfully drained, the Notification Center will show the message “Successfully

drained the node …”. Now the node is in maintenance mode and ready for firmware updates. In the

next section, we will invoke the Dell EMC firmware updates on the drained node.

Invoking the Dell EMC Patch and Update Script

To invoke the Firmware P&U script:

1. Log into the OME VM.

2. Open a new PowerShell window with Administrator privileges.

3. Browse to the folder where you extracted the DELLEMC Tools zip file and change directory to the

“Post Deploy Firmware Patch and Update Azure Stack” folder.


4. Type the following command at the prompt and press Enter to load all the modules required for

Firmware Patch and Update framework.

Import-Module .\DELLEMCFirmwareUpdate.ps1

5. Type the following command at the prompt.

Note: For the -IPAddress parameter, enter the IP address of the host’s iDRAC that is in

Maintenance Mode. The -Remediate paramater installs the firmware. If you do not use the -

Remediate parameter, only the inventory of the firmware will be printed out. Press Enter.

Invoke-CheckFirmwareBaseline -IPAddress 10.10.10.10 -Remediate

Note: After invoking the script if you receive and error that looks something like “WARNING: Caught

exception -> WinRM cannot complete the operation”, please consult the Dell EMC Known issues

section.

Note: If you encounter errors such as “Cannot invoke method. Method invocation is

supported only on core types in this language mode.” when attempting to execute

the Dell EMC firmware updates from the OME VM, see Appendix A: Updating Security Policies on the

OME VM.


6. After you press Enter, you will be prompted for the iDRAC username (idracUser). Enter the correct

iDRAC username. In the example below, we used root.

7. Next, you will be prompted to provide the iDRAC password (idracPass). Enter the correct iDRAC

password.

8. Next, you will be prompted to provide the name of the share where the firmware executables are

located (FirmwareShare). This is the same share we created in Step 2 of Preparing the stamp for

updates. In our example, we called it Firmware.

9. Next, you will be prompted for the Host IP address where the Firmware Share is located

(FirmwareShareHost). This is the IP address of the OMEVM where we created the share and

executing this script.


10. Next, you will be prompted to enter the credentials for the account on the OMEVM that has access to

the Firmware Share (FirmwareShareCredential). In our example below, we used Administrator.

At this step, the framework will parse the catalog file and perform health checks, among other

functions.


The framework will now compare the firmware inventory against the catalog file (Support Matrix

compatible) and print out the results of firmware that is not compliant and needs remediation.

If -Remediate option was used, the framework will start the remediation process now. This includes

creating update jobs and then polling the firmware update jobs. This can be seen from the few

screenshots below.


You can also open an iDRAC virtual console and the Job Queue during the updates process to get

real-time updates. If the iDRAC itself needs updating, you will lose access to it and will have to

reconnect.


Once the firmware updates are complete, go back to the Azure Stack Administration portal and

Resume the node.

IMPORTANT: Now repeat the same process for the remaining scale unit nodes.


How to review updates You can drill into the Update tile to view information about updates that may have already been imported, or

updates you plan to install on a certain date.

After an update package is uploaded to Azure Stack, the top-level Update tile will indicate that an update is

available, and show the current version of the stamp. See the following screenshots to review the in-line

logging and “download full logs” features. Sometimes the update availability information is not updated right

away, so please refresh the portal and it should reflect “Update available”.


You could also click on the “Download full logs” to get the upgrade summary information log in JSON format.


Monitor updates in Azure Stack using the privileged endpoint

The following information is also available at Microsoft’s Azure Stack website, Monitor updates in Azure Stack

using the privileged endpoint. It is recommended to visit the website in order to get the latest updates and

changes made by the Microsoft Azure Stack team.

Applies to: Azure Stack integrated systems

You can use the privileged endpoint to monitor the progress of an Azure Stack update run, and to resume a

failed update run from the last successful step.

The following new PowerShell cmdlets for update management are included in the 1710 update for Azure

Stack integrated systems.

Cmdlet Description

Get-AzureStackUpdateStatus Returns the status of the currently running, completed, or failed

update. Provides the high-level status of the update operation,

and an XML document that describes both the current step and

the corresponding state.

Get-AzureStackUpdateVerboseLog Returns the verbose logs that are generated by the update.

Resume-AzureStackUpdate Resumes a failed update at the point where it failed. In certain

scenarios, you may have to complete mitigation steps before you

resume the update.

Verify the cmdlets are available Because the cmdlets are new in the 1710 update package for Azure Stack, the 1710 update process needs to

get to a certain point before the monitoring capability is available. Typically, the cmdlets are available if the

status in the administrator portal indicates that the 1710 update is at the Restart Storage Hosts step.

Specifically, the cmdlet update occurs during Step: Running step 2.6 - Update PrivilegedEndpoint whitelist.

You can also determine whether the cmdlets are available programmatically by querying the command list

from the privileged endpoint. To do this, run the following commands from the hardware lifecycle host or from

a Privileged Access Workstation. Also, make sure the privileged endpoint is a trusted host. For more

information, see step 1 of Access the privileged endpoint.

https://docs.microsoft.com/en-us/azure/azure-stack/azure-stack-monitor-update

https://docs.microsoft.com/en-us/azure/azure-stack/azure-stack-monitor-update

https://github.com/MicrosoftDocs/azure-docs/blob/master/articles/azure-stack/azure-stack-privileged-endpoint.md#access-the-privileged-endpoint


1. Create a PowerShell session on any of the ERCS virtual machines in your Azure Stack environment

(Prefix-ERCS01, Prefix-ERCS02, or Prefix-ERCS03). Replace Prefix with the virtual machine prefix

string that’s specific to your environment.

$cred = Get-Credential $pepSession = New-PSSession -ComputerName <Prefix>-ercs01 -Credential $cred -ConfigurationName PrivilegedEndpoint

When prompted for credentials, use the <Azure Stack domain>\cloudadmin account, or an account

that's a member of the CloudAdmins group. For the CloudAdmin account, enter the same password

that was provided during installation for the AzureStackAdmin domain administrator account.

2. Get the full list of commands that are available in the privileged endpoint.

$commands = Invoke-Command -Session $pepSession -ScriptBlock { Get-Command }

3. Determine if the privileged endpoint was updated.

$updateManagementModuleName = "Microsoft.Azurestack.UpdateManagement" if (($commands | ? Source -eq $updateManagementModuleName)) { Write-Host "Privileged endpoint was updated to support update monitoring tools." } else { Write-Host "Privileged endpoint has not been updated yet. Please try again later."

}

4. List the commands specific to the Microsoft.AzureStack.UpdateManagement module.

$commands | ? Source -eq $updateManagementModuleName

For example:

$commands | ? Source -eq $updateManagementModuleName CommandType Name Version Source PSComputerName ----------- ---- ------- ------ -------------- Function Get-AzureStackUpdateStatus 0.0 Microsoft.Azurestack.UpdateManagement Contoso-ercs01 Function Get-AzureStackUpdateVerboseLog 0.0 Microsoft.Azurestack.UpdateManagement Contoso-ercs01 Function Resume-AzureStackUpdate 0.0 Microsoft.Azurestack.UpdateManagement Contoso-ercs01

Use the update management cmdlets

Note: Run the following commands from the hardware lifecycle host or from a Privileged Access

Workstation. Also, make sure the privileged endpoint is a trusted host. For more information, see step 1

of Access the privileged endpoint.

https://github.com/MicrosoftDocs/azure-docs/blob/master/articles/azure-stack/azure-stack-privileged-endpoint.md#access-the-privileged-endpoint


Connect to the privileged endpoint and assign session variable Run the following commands to create a PowerShell session on any of the ERCS virtual machines in your

Azure Stack environment (Prefix-ERCS01, Prefix-ERCS02, or Prefix-ERCS03), and to assign a session

variable.

$cred = Get-Credential $pepSession = New-PSSession -ComputerName <Prefix>-ercs01 -Credential $cred -ConfigurationName PrivilegedEndpoint

When prompted for credentials, use the <Azure Stack domain>\cloudadmin account, or an account that's a

member of the CloudAdmins group. For the CloudAdmin account, enter the same password that was

provided during installation for the AzureStackAdmin domain administrator account.

Get high-level status of the current update run To get a high-level status of the current update run, run the following commands:

$statusString = Invoke-Command -Session $pepSession -ScriptBlock { Get-AzureStackUpdateStatus -StatusOnly } $statusString.Value

Possible values include:

Running

Completed

Failed

Canceled

You can run these commands repeatedly to see the most up-to-date status. You don't have to re-establish a

connection to check again.

Get the full update run status with details You can get the full update run summary as an XML string. You can write the string to a file for examination,

or convert it to an XML document and use PowerShell to parse it. The following command parses the XML to

get a hierarchical list of the currently running steps.

[xml]$updateStatus = Invoke-Command -Session $pepSession -ScriptBlock { Get-AzureStackUpdateStatus } $updateStatus.SelectNodes("//Step[@Status='InProgress']")

In the following example, the top-level step (Cloud Update) has a child plan to update and restart the storage

hosts. It shows that the Restart Storage Hosts plan is updating the Blob Storage service on one of the hosts.

[xml]$updateStatus = Invoke-Command -Session $pepSession -ScriptBlock { Get-AzureStackUpdateStatus }


$updateStatus.SelectNodes("//Step[@Status='InProgress']") FullStepIndex : 2 Index : 2 Name : Cloud Update Description : Perform cloud update. StartTimeUtc : 2017-10-13T12:50:39.9020351Z Status : InProgress Task : Task FullStepIndex : 2.9 Index : 9 Name : Restart Storage Hosts Description : Restart Storage Hosts. EceErrorAction : Stop StartTimeUtc : 2017-10-13T15:44:06.7431447Z Status : InProgress Task : Task FullStepIndex : 2.9.2 Index : 2 Name : PreUpdate ACS Blob Service Description : Check function level, update deployment artifacts, configure Blob service settings StartTimeUtc : 2017-10-13T15:44:26.0708525Z Status : InProgress Task : Task

Get the verbose progress log You can write the log to a file for examination. This can help you diagnose an update failure.

$log = Invoke-Command -Session $pepSession -ScriptBlock { Get-AzureStackUpdateVerboseLog } $log > ".\UpdateVerboseLog.txt"

Actively view the verbose logging To actively view the verbose log during an update run, and jump to the most recent entries, run the following

commands to enter the session in interactive mode, and to show the log:

Enter-PSSession -Session $pepSession Get-AzureStackUpdateVerboseLog -Wait

The log updates every 60 seconds, and new content (if available) is written to the console.

During long-running background processes, the console output may not be written to the console for some

time. To cancel the interactive output, press Ctrl+C.


Resume a failed update operation If the update fails, you can resume the update run where it left off.

Invoke-Command -Session $pepSession -ScriptBlock { Resume-AzureStackUpdate }

Troubleshooting

The privileged endpoint is available on all ERCS virtual machines in the Azure Stack environment. Because

the connection is not made to a highly available endpoint, you may experience occasional interruptions,

warning, or error messages. These messages may indicate that the session was disconnected or that there

was an error communicating with the ECE Service. This behavior is expected. You can retry the operation in

a few minutes or create a new privileged endpoint session on one of the other ERCS virtual machines.

New features and fixes This update includes the following improvements and fixes for Azure Stack.

New features Test-AzureStack cmdlet to validate Azure Stack Cloud available via privileged endpoint

Ability to register a disconnected deployment of Azure Stack

Monitoring alerts for certificate and user account expiration

Added Set-BmcPassword cmdlet in PEP for BMC password rotation

Network logging updates to support on-demand logging

Support reimage operation for Virtual Machine Scales Sets (VMSS)

Enable kiosk mode on ERCS VM for CloudAdmin login

Tenants can activate Windows VMs automatically

Fixes Fix to show Node Operational Status in maintenance while running repair

Fix to correct Public IP usage records time/date stamp

Various other performance, stability and security fixes

TimeSource and Defender privileged endpoint module bug fixes

Windows Server 2016 new features and fixes

January, 3rd - 2018—KB4056890 (OS Build 14393.2007)

o This update includes the software fixes for the industry-wide security issue described by MSRC

Security Advisory ADV 180002.

https://support.microsoft.com/help/4056890/windows-10-update-kb4056890

https://portal.msrc.microsoft.com/en-US/security-guidance/advisory/ADV180002

https://portal.msrc.microsoft.com/en-US/security-guidance/advisory/ADV180002


Known issues with the update process

Microsoft Known Issues

This section contains known issues that you may encounter during the 1712 update installation.

1. Symptom: Azure Stack operators may see the following error during the update process: "Type 'CheckHealth' of Role 'VirtualMachines' raised an exception:\n\nVirtual

Machine health check for -ACS01 produced the following errors.\nThere was

an error getting VM information from hosts. Exception details:\nGet-VM :

The operation on computer 'Node03' failed: The WS-Management service

cannot process the request. The WMI \nservice or the WMI provider returned

an unknown error: HRESULT 0x8004106c."

a) Cause: This issue is caused by a Windows Server issue that is intended to be addressed in subsequent Window server updates.

b) Resolution: Contact Microsoft Customer Service and Support (CSS) for assistance.

2. Symptom: Azure Stack operators may see the following error during the update process:"Enabling the seed ring VM failed on node Host-Node03 with an error: [Host-Node03]

Connecting to remote server Host-Node03 failed with the following error

message : The WinRM client received an HTTP server error status (500), but

the remote service did not include any other information about the cause

of the failure."

a) Cause: This issue is caused by a Windows Server issue that is intended to be addressed in subsequent Window server updates.

b) Resolution: Contact Microsoft Customer Service and Support (CSS) for assistance.

Known issues (post-installation) This section contains post-installation known issues with build 20171201.3.

Portal It may not be possible to view compute or storage resources in the administrator portal. This indicates

that an error occurred during the installation of the update and that the update was incorrectly

reported as successful. If this issue occurs, please contact Microsoft CSS for assistance.

You may see a blank dashboard in the portal. To recover the dashboard, select the gear icon in the

upper right corner of the portal, and then select Restore default settings.

When you view the properties of a resource group, the Move button is disabled. This behavior is

expected. Moving resource groups between subscriptions is not currently supported.

For any workflow where you select a subscription, resource group, or location in a drop-down list, you

may experience one or more of the following issues:

You may see a blank row at the top of the list. You should still be able to select an item as expected.

If the list of items in the drop-down list is short, you may not be able to view any of the item names.

If you have multiple user subscriptions, the resource group drop-down list may be empty.


Note: To work around the last two issues, you can type the name of the subscription or resource

group (if you know it), or you can use PowerShell instead.

Deleting user subscriptions results in orphaned resources. As a workaround, first delete user

resources or the entire resource group, and then delete user subscriptions.

You are not able to view permissions to your subscription using the Azure Stack portals. As a

workaround, you can verify permissions by using PowerShell.

Health and monitoring If you reboot an infrastructure role instance, you may receive a message indicating that the reboot

failed. However, the reboot actually succeeded.

Marketplace Some marketplace items are being removed in this release due to compatibility concerns. These will

be re-enabled after further validation.

Users can browse the full marketplace without a subscription, and can see administrative items like

plans and offers. These items are non-functional to users.

Compute Users are given the option to create a virtual machine with geo-redundant storage. This configuration

causes virtual machine creation to fail.

You can configure a virtual machine availability set only with a fault domain of one, and an update

domain of one.

There is no marketplace experience to create virtual machine scale sets. You can create a scale set

by using a template.

Scaling settings for virtual machine scale sets are not available in the portal. As a workaround, you

can use Azure PowerShell. Because of PowerShell version differences, you must use the -

Name parameter instead of -VMScaleSetName .

Networking You can't create a load balancer with a public IP address by using the portal. As a workaround, you

can use PowerShell to create the load balancer.

You must create a network address translation (NAT) rule when you create a network load balancer.

If you don't, you'll receive an error when you try to add a NAT rule after the load balancer is created.

You can't disassociate a public IP address from a virtual machine (VM) after the VM has been created

and associated with that IP address. Disassociation will appear to work, but the previously assigned

public IP address remains associated with the original VM. This behavior occurs even if you reassign

the IP address to a new VM (commonly referred to as a VIP swap). All future attempts to connect

through this IP address result in a connection to the originally associated VM, and not to the new one.

Currently, you must only use new public IP addresses for new VM creation.

Azure Stack operators may be unable to deploy, delete, modify VNETs or Network Security Groups.

This issue is primarily seen on subsequent update attempts of the same package. This is caused by a

packaging issue with an update which is currently under investigation.

https://docs.microsoft.com/azure/virtual-machine-scale-sets/virtual-machine-scale-sets-manage-powershell#change-the-capacity-of-a-scale-set


Internal Load Balancing (ILB) improperly handles MAC addresses for back-end VMs which breaks

Linux instances.


SQL/MySQL It can take up to an hour before tenants can create databases in a new SQL or MySQL SKU.

Creation of items directly on SQL and MySQL hosting servers that are not performed by the resource

provider is not supported and may result in a mismatched state.

Note: You should not have impact to your existing SQL or MySQL resource provider users when

updating your Azure Stack Integrated Systems to the 1712 version. You can continue to use your

current SQL or MySQL resource provider builds until a new Azure Stack update is available.

App Service A user must register the storage resource provider before they create their first Azure Function in the

subscription.

Identity In Azure Active Directory Federation Services (ADFS) deployed environments,

the azurestack\azurestackadmin account is no longer the owner of the Default Provider Subscription.

Instead of logging into the Admin portal / adminmanagement endpoint with

the azurestack\azurestackadmin, you can use the azurestack\cloudadmin account, so that you can

manage and use the Default Provider Subscription.

IMPORTANT: Even though the azurestack\cloudadmin account is the owner of the Default Provider

Subscription in ADFS deployed environments, it does not have permissions to RDP into the host.

Continue to use the azurestack\azurestackadmin account or the local administrator account to login,

access and manage the host as needed.

DELL EMC Known Issues

Symptom: Sometimes when running the DELLEMC Firmware Patch and Update framework, right at the start

of validating the firmware, the PowerShell outputs the following exception.

WARNING: Caught exception -> WinRM cannot complete the operation. Verify that

the specified computer name is valid, that the computer is accessible over the

network, and that a firewall exception for the WinRM service is enabled and

allows access from this computer. By default, the WinRM firewall exception for

public profiles limits access to remote computers within the same local subnet.

Cause: The root cause of this particular issue is unknown at the moment.

Resolution: The workaround is simply to re-try the command again and when you try the second time, the

framework connects to the iDRAC just fine and validates the firmware baseline.


Appendix A: Updating Security Policies on the OME VM If you encounter errors, such as “Cannot invoke method. Method invocation is supported

only on core types in this language mode.” when attempting to execute the Dell EMC firmware

updates from the OME VM, it will be necessary to update the security policies to white-list the scripts that

perform the updates.

Step Activity

1 Download the Latest DellEMC Tools from the following link:


2 a.) Extract the update files from the DellEMC Tools package to C:\PU on the OME VM.

b.) Copy “UpdateWDAC\” to C:\Security on your OME VM.

3 Open a PowerShell session with Administrator privileges on the OME VM.

4 Navigate to C:\Security\UpdateWDAC.

5 Run the following script:

.\CreateOMEAuditPolicy.ps1

6 Reboot the VM.

7 Log back in to the OME VM and Open PowerShell with Administrator privileges.



Step Activity

8 Run the following command, which will clear the existing code integrity events.

Wevtutil.exe cl Microsoft-Windows-CodeIntegrity/Operational

9 Navigate to C:\PU to locate the SupportAssist Enterprise installer.

10 Execute SupportAssistEnterprise_1.2.0.36.exe and select “Upgrade”.

11 Navigate back to C:\Security\OMEWDACUpdate.


Step Activity

12 Create the final enforced whitelist by executing the following command:

.\CreateOMEEnforcedPolicy.ps1

14 Reboot the OME VM so that the policy takes effect.

15 The DellEMC update scripts should now be white-listed, and should execute without errors.

1712 dell emc cloud for microsoft azure stack · 10 1712 dell emc cloud for microsoft azure stack...

Documents