dell emc cloud for microsoft azure stack vxrack as

A Dell EMC Technical White Paper

Dell EMC Cloud for Microsoft Azure Stack VxRack AS Concepts Guide

VxRack AS Version A00 Dell Engineering November 2017

2 Dell EMC Cloud for Microsoft Azure Stack VxRack AS | Concept Guide | version A00

Revisions

Date Version Description

Nov 2017 A00 Initial release

THIS GUIDE IS FOR INFORMATIONAL PURPOSES ONLY, AND MAY CONTAIN TYPOGRAPHICAL ERRORS AND TECHNICAL INACCURACIES.

THE CONTENT IS PROVIDED AS IS, WITHOUT EXPRESS OR IMPLIED WARRANTIES OF ANY KIND.

Copyright © 2017 Dell Inc. All rights reserved. Dell and the Dell EMC logo are trademarks of Dell Inc. in the United States and/or other jurisdictions. All

other marks and names mentioned herein may be trademarks of their respective companies.


Table of contents Revisions............................................................................................................................................................................. 2

Table of contents ................................................................................................................................................................ 3

Overview ............................................................................................................................................................................. 6

Dell EMC Cloud for Microsoft Azure Stack Scale Unit ................................................................................................. 7

Dell EMC Cloud for Microsoft Azure Stack Hardware Lifecycle Host .......................................................................... 7

Configuration options ................................................................................................................................................... 8

Order and deployment process .................................................................................................................................... 9

Applications and Azure Services .................................................................................................................................. 9

Prerequisites ..................................................................................................................................................................... 10

Environmental requirements ...................................................................................................................................... 10

Stack PDU Power Drop requirements ........................................................................................................................ 10

Azure Connection, Identity Store, Billing Model Decisions ........................................................................................ 11

Choose identity store .................................................................................................................................................. 12

Azure Active Directory identity store .......................................................................................................................... 12

Active Directory Federated Services identity store .................................................................................................... 12

Choosing Disconnected From Azure ......................................................................................................................... 12

Features that are impaired or unavailable in Disconnected Mode ............................................................................. 13

Required customer-provided security certificates ...................................................................................................... 15

Azure Stack certificates required ................................................................................................................................ 15

PaaS certificates (optional) ........................................................................................................................................ 17

Requesting certificates using an INF file .................................................................................................................... 18

License requirements ................................................................................................................................................. 19

Azure Stack endpoints and customer port requirements ........................................................................................... 20

Hardware infrastructure .................................................................................................................................................... 23

Hardware components ............................................................................................................................................... 23

Scale Unit configuration ............................................................................................................................................ 24

Supported PDU options .............................................................................................................................................. 27

Networking .................................................................................................................................................................. 30

Server and Switch Port Description References ........................................................................................................ 30

Cable placement and port mapping ........................................................................................................................... 31

Hardware Lifecycle Host management network connectivity ..................................................................................... 32

Scale Unit – R740XD connectivity.............................................................................................................................. 33


Border connectivity ..................................................................................................................................................... 34

BGP routing ................................................................................................................................................................ 35

Static routing ............................................................................................................................................................... 36

Transparent proxy ...................................................................................................................................................... 37

Firewall Integration ..................................................................................................................................................... 38

Deployment ............................................................................................................................................................... 38

Register your Azure Stack system (activate the system) ........................................................................................... 39

Operations and management software ............................................................................................................................. 40

Microsoft Azure Stack ................................................................................................................................................ 40

Hardware Lifecycle Host software .............................................................................................................................. 42

Security ............................................................................................................................................................................. 43

Least privilege – Minimum authority required for each operation .............................................................................. 43

Secrets rotation (change password on a regular cadence) ....................................................................................... 43

Maintaining the Dell EMC Hybrid Cloud for Microsoft Azure Stack .................................................................................. 45

Monitoring and alerting in Azure Stack ...................................................................................................................... 45

Patch and Update ....................................................................................................................................................... 45

Backup and recovery .................................................................................................................................................. 46

Azure Stack Infrastructure Backup – the Backup Controller ...................................................................................... 47

Backup ........................................................................................................................................................................ 47

What data is actually captured by the Infrastructure Backup Controller? .................................................................. 48

Data In-Scope............................................................................................................................................................. 48

What about PAAS data and Resource Provider VMs? .............................................................................................. 49

Modern web application BCDR approach .................................................................................................................. 49

Third party solutions ................................................................................................................................................... 50

What about my custom images and blob collateral for Marketplace? ........................................................................ 50

Hardware Lifecycle Host and switch configuration ..................................................................................................... 51

Microsoft recommended SMB target folder structure example .................................................................................. 51

Recovery from a catastrophic failure high-level workflow .......................................................................................... 52

Dell EMC support and consulting offerings ................................................................................................................ 53

Field Replacement of Parts ........................................................................................................................................ 53

ProSupport Plus for Enterprise ................................................................................................................................... 53

Consulting service offerings ....................................................................................................................................... 53

Cautions ............................................................................................................................................................................ 55


Additional resources ......................................................................................................................................................... 56

Tools for using Azure and Azure Stack https://github.com/Azure/AzureStack-Tools ................................................ 56


Overview Adopting a hybrid cloud strategy as a means to achieving

digital transformation can be complicated. Often, IT and the

organization they support have processes, procedures,

personnel, and tools that are not aligned for optimal cloud

brokerage and consumption.

The most common hurdles to overcome are complexity of

disaggregate applications and tools, legacy IT vs cloud

competency, confidence that anytime, anywhere, always-on

availability is achievable and that the cost of acquisition is affordable.

Enterprise IT organizations are expected to deliver a consistent end-user experience but most public and

private cloud implementations are not reflective of one another making all phases of the life-cycle; acquisition,

deployment, operation, maintenance more of a kludge, than a repeatable, predictable, positive experience.

Dell EMC Cloud for Microsoft Azure Stack is engineered with best in

class hyper-converged VxRack AS infrastructure, networking, backup

and encryption from Dell EMC, along with application development

tools from Microsoft. Furthermore, Dell EMC manages the component

lifecycle of the entire Azure Stack platform to ensure all phases

(acquisition, deployment, operation and maintenance) have a

repeatable, predictable, turnkey experience.

This powerful combination brings together Microsoft Azure Stack with the expertise of Dell EMC in the

development of hybrid cloud platforms. Dell EMC offers a robust end-to-end solution from the integration of

hardware, software and services, to lifecycle management and seamless upgrades. Our approach delivers

our customers better results for IT and digital transformation – we make the complex simple with a fully

engineered, trusted hybrid cloud platform for Microsoft Azure Stack.

https://cloudchatsblog.emc.com/wp-content/uploads/2017/05/cloud-2.png


VxRack AS Hyper-converged Infrastructure The Dell EMC Cloud for Microsoft Azure Stack is a fully-engineered hybrid cloud platform built on the VxRack

AS hyper-converged architecture, consisting of common modular building blocks that scale linearly from 4 to

12 (16 future) nodes in a scale unit. It provides a simple, cost-effective solution that delivers multiple

performance and capacity options to match any use case and cover a wide variety of cloud native

applications and workloads. Based Microsoft’s Windows 2016 software defined architecture and built with

new 5th generation Intel™ Xeon™ processors, the Dell EMC VxRack AS allows customers to start small and

grow, scaling capacity and performance easily with minimal disruption. Scaling in predictable units ensures a

“pay-as-you-grow” approach for future growth.

Dell EMC Cloud for Microsoft Azure Stack Scale Unit

Dell EMC Cloud for Microsoft Azure Stack is built around a Scale Unit (SU). At the Scale Unit, Dell EMC

Cloud for Microsoft Azure Stack is a hyper converged Azure Stack engineered system with the option to start

with 4, 8, or 12 nodes.

Within the Scale Unit, Dell EMC Cloud for Microsoft Azure Stack provides flexibility at a component level to

optimize processor, memory, storage capacity, and caching ratios.

Dell EMC Cloud for Microsoft Azure Stack Hardware Lifecycle Host

The Hardware Lifecyce Host (HLH) is Dell EMC designed to enable monitoring and updates for your Azure

stack. The Host is a PowerEdge R640 Management server with Dell EMC management software and tools to

enable Server and Network monitoring, call home capability if desired and Patch and Update capability of the

Dell EMC provided components.


Configuration options Scale Units:

Small (4 PowerEdge R740XD nodes)

Medium (8 PowerEdge R740XD nodes)

Large (12 PowerEdge R740XD nodes)

Each of the Scale Units supports three capacity and performance options:

Note: The three capacity and performance options must be homogenous. There is no mixing and matching within

a Scale Unit.

Standard components In addition each Scale Unit also includes the required Hardware Life Cycle Host server and network switches

1 x Dell EMC PowerEdge R640 Management server (Hardware Lifecycle Host)

2 x Dell EMC Networking S4048-ON Top of Rack switches

1 x Dell EMC Networking S3048-ON Management switch

Dell EMC Cloud for Microsoft Azure Stack includes the following services offerings:

Dell EMC Support Services

Dell EMMC Deployment Services

Optional Dell EMC or Partner Professional Consulting Services

Configuration Processor Memory Cache Data Storage

Low Gold 5118 - 12 core 2.3Ghz 384GB 6 x 960/800GB SSD = ~5.7TB SAS

10x4TB (40TB) SAS

Mid Gold 6130- 16 core 2.1Ghz 512GB 6x1.92TB(11.5TB) SAS 10X8TB (80TB) SAS

High Platinum 8160 - 24 core 2.1Ghz 788GB 6x1.92TB(11.5TB) SAS 10x10TB (100TB) SAS


Order and deployment process

Applications and Azure Services

Dell EMC Cloud for Microsoft Azure Stack is designed to run Infrastructure and Platform services consistent

with what is available in Azure public. With Azure Services available on-premises, customers can:

Use the cloud computing model for Azure IaaS services that go much beyond traditional virtualization.

For instance, Virtual Machine Scale Sets enable rapid deployments with scaling options for modern

workloads (for example, containerized applications).

Incorporate consistent Azure PaaS services that simplify development and enable hybrid deployment

choice and portability for cloud applications. Run high-productivity PaaS (Azure App Service) and

Serverless computing (Azure functions) in on-premises environments.

Adopt common operational practices across Azure and Azure Stack: Deploy and operate Azure IaaS/

PaaS services using the same administrative experiences and tools as Azure.

Use an Azure Active Directory (AAD) subscription to administer Azure Stack identities, including

secure multitenant access (i.e., enabling users across multiple AAD tenants to access Azure Stack

resources).

Build for the future as Microsoft delivers continuous Azure innovation to Azure Stack, including new

Azure services, updates to existing services, and additional Azure Marketplace applications.


Prerequisites

Environmental requirements

14G Configuration Totals for 200V AC Input Voltage and 35C Max Ambient

4 node 8 node 12 node 16 node*

Watts BTU/hr Watts BTU/hr Watts BTU/hr Watts BTU/hr

Input Power

Min 3395 11577 5979 20388 8563 29200 11147 38011

Mid 3691 12586 6571 22407 9451 32228 12331 42049

Max 3927 13391 7043 24017 10159 34642 13275 45268

Input Current (Amps)

Min 17.2 30.3 43.4 56.5

Mid 18.7 33.3 47.8 62.4

Max 19.9 35.6 51.4 67.1

Weight (pounds) 790 1082 1374 1666

Stack PDU Power Drop requirements

Number of Scale Power Drops Reqiured

Units (R740XDs) Single Phase 3 Phase Delta 3 Phase Y

4 2 2 2

8 4 2 2

12 6 2 2

16* 8 2 2

*Note: 16 node expected to be supported by Microsoft in 2018

Data Source - Legal Notice: Results shown in the previous table are from Dell EMC Lab measurements and the EMC Power Calculator. The EMC Power Calculator is subject to change without notice and is provided “AS IS” without warrant of any kind, express or implied. EMC does not make any representations regarding the use, validity, accuracy or reliability of the tool or the results of the use of the tool. The entire risk arising out of the use of this tool remains solely with the customer. In no event shall EMC be liable for any direct, consequential, incidental, special, punitive or other damages, even if EMC is negligent or has been advised of the possibility of such damages, arising from use of the tool or the information provided herein Output values obtained from this tool are intended solely for customer facilities planning purposes and are approximate and conservative. Actual results may vary.


Azure Connection, Identity Store, Billing Model Decisions

You can deploy Azure Stack to an environment that is connected to Azure (the default) or disconnected from Azure. This

choice defines which options are available for your identity store (Azure Active Directory or Active Directory Federation

Services) and billing model (pay-as-you-use billing or capacity-based billing). See the following diagram and chart:

This is a key decision point! Choosing ADFS or AAD is a one-time decision that you must make at

deployment time. You cannot change this later without re-deploying the entire system.

CHOOSING CONNECTED TO AZURE

If you choose the Connect to Azure option, your Azure Stack deployment will have connectivity to Azure. This means that

you can have either Azure Active Directory or Active Directory Federation Services (ADFS) for your identity store. You can

also choose from either billing model: consumption-based or capacity-based. A connected deployment is the default

option because it allows customers to get the most value out of Azure Stack, particularly for hybrid scenarios that involve

both Azure and Azure Stack.


Choose identity store

With a connected deployment, you can choose between Azure Active Directory or ADFS for your identity store. A

disconnected deployment can only use ADFS.

Your identity store choice has no bearing on tenant VMs, the identity store and accounts that they use, whether or not

they can join an Active Directory Domain, and so on. This is separate.

For example: If you deploy IaaS tenant VMs on top of Azure Stack, and want them to join a Corporate Active Directory

Domain and use accounts from there, you can still do this. You are not required to use the AAD identity store you select

here for those accounts.

Azure Active Directory identity store

When you use Azure Active Directory for your identity store, you need two Azure Active Directory accounts. These

accounts can be the same account, or different accounts. While using the same account might be simpler and useful if

you have a limited number of Azure accounts, your business needs might suggest using two accounts.

1. Global admin account (only required for connected deployments). This is an Azure account that is used to create

applications and service principals for Azure Stack infrastructure services in Azure Active Directory. This account

must have directory admin privileges to the directory that your Azure Stack system will be deployed under. It will

become the Global Admin for the Azure Active Directory tenant. It will be used:

a. To provision and delegate applications and service principals for all Azure Stack services that need to

interact with Azure Active Directory and Graph API.

b. As the Service Administrator account. This is the owner of the default provider subscription (which you

can later change). You can log into the Azure Stack admin portal with this account, and can use it to

create offers and plans, set quotas, and perform other administrative functions in Azure Stack.

2. Billing account (required for both connected and disconnected deployments). This Azure account that is used to

establish the billing relationship between your Azure Stack system with the Azure commerce backend. This is the

account that will be billed for Azure Stack fees. This account will also be used for marketplace syndication and

other hybrid scenarios.

Active Directory Federated Services identity store

Choose this option if you want to use your own identity store, such as Active Directory, for your Service Administrator

accounts. If you want to use your Corporate Active Directory to manage your Service Administrator accounts, then this is

the option for you.

Choosing Disconnected From Azure

With this option, you can deploy and use Azure Stack without a connection to the Internet. Choose this option if you:

Have security or other restrictions that require you to deploy Azure Stack in an environment that is not connected

to the Internet.

Want to block data (including usage data) from being sent to Azure.

Want to use Azure Stack purely as a private cloud solution that is deployed to your corporate Intranet, and are not

interested in hybrid scenarios.


Sometimes, this type of environment is also referred to as a “submarine scenario”.

With a disconnected deployment, you are limited to an ADFS identity store and a capacity-based billing model.

A disconnected deployment does not strictly mean that you cannot later connect your Azure Stack instance to Azure for

hybrid scenarios for tenant VMs. It means that you do not have connectivity to Azure during deployment, or you do not

want to use Azure Active Directory as your identity store. However, if you want to have connectivity to Azure after

deployment, regardless of what you want to use as your identity store, you should choose the Connect to Azure

deployment option.

Physically disconnected Physically connected

Billing Must be capacity EA only

Capacity or consumption EA or CSP

Identity Must be ADFS AAD or ADFS

Marketplace syndication

Not available Supported BYOL licensing of syndicated images

Registration Not available Automated

P&U Required, requires removable media and a separate connected device

Automated

Features that are impaired or unavailable in Disconnected Mode

Azure Stack was designed to work best when connected to Azure, so it is important to note that there are some features

and functionality that are either impaired or completely unavailable in the Disconnected mode.

Feature Impact in Disconnected mode

VM deployment with DSC extension to configure VM post deployment

Impaired – DSC extension looks to the Internet for the latest WMF.

VM deployment with Docker Extension to run Docker commands

Impaired – Docker will check the Internet for the latest version and this check will fail.

Documentation links in the Azure Stack Portal Unavailable – Links such as Give Feedback, Help, Quickstart, etc. that use an Internet URL will not work.

Alert remediation/mitigation that references an online remediation guide

Unavailable – Any alert remediation links that use an Internet URL will not work.

Marketplace syndication – The ability to select and add Gallery packages directly from the Azure Marketplace

Unavailable – This feature requires connectivity to Azure and an Azure Active Directory account.


Feature Impact in Disconnected mode

Using Azure Active Directory federation accounts to manage an Azure Stack deployment

Unavailable – This feature requires connectivity to Azure. ADFS with a local Active Directory instance must be used instead.

Resource Providers such as WebApps and SQL Unavailable - Resource Providers such as WebApps and SQL require Internet access for content.

Command Line Interface (CLI) Impaired – CLI has reduced functionality in terms of authentication and provisioning of Service Principles.

Visual Studio – Cloud discovery Impaired – Cloud Discovery will either discover different clouds or will not work at all.

Visual Studio – ADFS Impaired – Only Visual Studio Enterprise supports ADFS.

Telemetry Unavailable – Telemetry data for Azure Stack as well as any third-party gallery packages that depend on telemetry data.

Certificates Unavailable – Internet connectivity is required for Certificate Revocation List (CRL) and Online Certificate Status Protocol (OSCP) services in the context of HTTPS.

Key-Vault Impaired – A common use case for Key Vault is to have an application read secrets at runtime. For this the application needs a service principal in the directory. In Azure Active Directory, regular users (non-admins) are by default allowed to add service principals. In AD (using ADFS) they are not. This places a hurdle in the end-to-end experience because one must always go through a directory admin to add any application.


Required customer-provided security certificates

Azure Stack has a public infrastructure network that contains the external-accessible or public IP addresses

that are assigned to a small set of Azure Stack services. The remainder are used by the tenant VMs. You

must provide certificates with the appropriate DNS names for these Azure Stack public infrastructure

endpoints.

Note that there are some certificate restrictions in the current Azure Stack version. Below is a list of the

certificate requirements that are needed to deploy Azure Stack:

Certificate must be from either an internal Certificate Authority, or a Public Certificate Authority who is

included in the base OS image as part of the Microsoft Trusted Root Authority Program. You can find

the full list here: https://gallery.technet.microsoft.com/Trusted-Root-Certificate-123665ca

The certificate can be a single wild card certificate covering all name spaces in the Subject Alternative

Name (SAN) field or can be a set of individual certificates only using wild cards for endpoints such as

storage and Key Vault where they are required.

The certificate signature algorithm cannot be SHA1, as it must be stronger.

The certificate format must be PFX, as both the public and private keys are required for Azure Stack

installation.

The certificate pfx files must have a value "Digital Signature", "KeyEncipherment", and

"DataEncipherment" in its “Key Usage" field.

The passwords to all certificate pfx files must be the same at the time of deployment

Ensure that the Subject Names and Subject Alternative Names of all certificates provided by the

Azure Stack Administrator match the specifications outlines in “Certificates Required”. Failure to do

so we result in failed deployments attempts.

Azure Stack certificates required As described above, you must provide certificates with the appropriate DNS names for the different Azure

Stack public infrastructure endpoints. Each endpoint’s DNS name is expressed in the format:

<PREFIX>.<REGION>.<EXTERNALFQDN>

For your deployment, the REGION and EXTERNALFQDN values must match the region and external domain

names that you chose for your Azure Stack system. As an example, if my region name was “Redmond” and

my external domain name was “Contoso.com”, my DNS names would have the format

<PREFIX>.redmond.contoso.com. PREFIX values are predesignated by Microsoft to describe the endpoint

secured by the certificate.

The PREFIX values of the external infrastructure endpoints depend on the Azure Stack service that uses the

specific endpoint. Table C1 below describes the different Azure Stack public endpoints required for Azure

Stack deployments in both AAD and ADFS modes, grouped by area, as well as the namespaces used and

the certificates that are required for each namespace. Please note that the table below also describes the

folder to which you must copy the different certificates per public endpoint:

https://gallery.technet.microsoft.com/Trusted-Root-Certificate-123665ca


Note: You MUST copy the certificates to each folder in the folder structure that matches the identity provider

you are deploying against, AAD or ADFS. If you are using a single certificate for all endpoints, you must copy

that certificate file into each deployment folder outlined in the tables below. The folder structure is pre-built in

the DVM and can be found here: C:\CloudDeployment\Setup\Certificates.

The following table lists the required certificates for all Azure Stack deployments (AAD and ADFS):

Table C1

If you deploy Azure Stack using the AAD deployment mode, you only need to request the certificates listed in

the previous table (C1). However, if you deploy Azure Stack using the ADFS deployment mode, you must

request the certificates listed in the previous table (C1) AND the additional certificates listed in the following

table (C2).

The following table lists the additional required certificates for deployments using ADFS as the identity

management system:

Scope (per region)

Namespace Certificate Deployment Folder

ADFS <REGION>.<EXTERNALFQDN> adfs.<REGION>.<EXTERNALFQDN> SSL Certificate

ADFS

Graph <REGION>.<EXTERNALFQDN> graph.<REGION>.<EXTERNALFQDN> SSL Certificate

Graph

Table C2

Note: All of the certificates listed on both tables above (C1 and C2) must have the same password.

Scope (per region)

Namespace Certificate Deployment Folder

Portals

ARM

<REGION>.<EXTERNALFQDN> portal. <REGION>.<EXTERNALFQDN> adminportal. <REGION>.<EXTERNALFQDN> management. <REGION>.<EXTERNALFQDN> adminmanagement. <REGION>.<EXTERNALFQDN>

SSL Certificate with SANs

Public Portal Admin Portal ARM Public

ARM Admin

Storage blob.<REGION>.<EXTERNALFQDN> *.blob.<REGION>.<EXTERNALFQDN> Wildcard SSL Certificate

ACS

table.<REGION>.<EXTERNALFQDN> *.queue.<REGION>.<EXTERNALFQDN> Wildcard SSL Certificate

queue.<REGION>.<EXTERNALFQDN> *.table.<REGION>.<EXTERNALFQDN> Wildcard SSL Certificate

Key Vault vault.<REGION>.<EXTERNALFQDN> *.vault.<REGION>.<EXTERNALFQDN> Wildcard SSL Certificate

KeyVault

adminvault.<REGION>.<EXTERNALFQDN> *.adminvault.<REGION>.<EXTERNALFQDN> Wildcard SSL Certificate

KeyVaultInternal


PaaS certificates (optional) If you are planning to deploy the additional Azure Stack PaaS services (SQL, MySQL, and App Service) after

Azure Stack has been deployed and configured, you will need to request additional certificates to cover the

endpoints of the PaaS services.

IMPORTANT: The certificates that you use for App Service and SQL/MySQL resource providers need to

have the same root authority as those used for the public Azure Stack endpoints.

Table C3 below describes the endpoints and certificates required for the SQL/MySQL adapters and for App

Service. Please note that you do not need to copy these certificates to the Azure Stack deployment folder.

Instead, you will be asked to provide these certificates when you install the additional resource providers.

The following table lists the certificates required for additional Azure Stack PaaS services:

Scope (per region)

Namespace Certificate Used for

SQL MySQL

dbadapter.<REGION>.<EXTERNALFQDN> *.dbadapter.<REGION>.<EXTERNALFQDN> Wildcard SSL Certificate

SQL and MySQL

App Service

appservice.<REGION>.<EXTERNALFQDN> *.appservice.<REGION>.<EXTERNALFQDN> *.scm.appservice.<REGION>.<EXTERNALFQDN> Multi Domain Wildcard SSL Certificate1

Web Traffic Default SSL Cert

api.appservice.<REGION>.<EXTERNALFQDN> SSL Certificate

API

sso.appservice.<REGION>.<EXTERNALFQDN> SSL Certificate

SSO

Table C3 1 May not be supported by all Public Certificate Authorities

Dell EMC required certificates

Table C4 below describes the endpoints and certificates required for the Open Manage Essentials and

Support Assist Enterprise. Please note that you do not need to copy these certificates to the Azure Stack

deployment folder. Instead, you will need provide these certificates during install of OME and SAE.

The following table lists the certificates required:

Scope Namespace Certificate Used for

OME <OMESRVNAME>.<customerFQDN> <OMESRVNAME>.<REGION>.<customerFQDN> SSL Certificate with SANs

OME

OMNM <OMNMSRVNAME>.<customerFQDN>

<OMNMSRVNAME>.<REGION>.<customerFQDN> SSL Certificate with SANs

OMNM

Table C4


Requesting certificates using an INF file One way to request certificates from either a Public CA or an Internal CA is by using an INF file to specify

details of the certificate, and then use the Windows built-in certreq.exe utility to generate a request file using

that INF. This process is described in the sections below.

Sample INF file

Below is a sample certrequest INF file that can be used to create an offline certreq file for submission to a CA

(either internal or public) that covers all of the required endpoints (including the PaaS services) in a single

wildcard certificate.

The sample INF file below assumes that:

Region = SEA

External FQDN = contoso.com

[Version] Signature="$Windows NT$"

[NewRequest] Subject = "C=US, O=Microsoft, L=Redmond, ST=Washington, CN=portal.sea.contoso.com"

Exportable = TRUE ; Private key is not exportable KeyLength = 2048 ; Common key sizes: 512, 1024, 2048, 4096, 8192, 16384 KeySpec = 1 ; AT_KEYEXCHANGE KeyUsage = 0xA0 ; Digital Signature, Key Encipherment MachineKeySet = True ; The key belongs to the local computer account ProviderName = "Microsoft RSA SChannel Cryptographic Provider" ProviderType = 12 SMIME = FALSE RequestType = PKCS10 HashAlgorithm = SHA256

; At least certreq.exe shipping with Windows Vista/Server 2008 is required to interpret the [Strings] and [Extensions] sections below

[Strings] szOID_SUBJECT_ALT_NAME2 = "2.5.29.17" szOID_ENHANCED_KEY_USAGE = "2.5.29.37" szOID_PKIX_KP_SERVER_AUTH = "1.3.6.1.5.5.7.3.1" szOID_PKIX_KP_CLIENT_AUTH = "1.3.6.1.5.5.7.3.2"

[Extensions] %szOID_SUBJECT_ALT_NAME2% = "{text}dns=*.sea.contoso.com&dns=*.blob.sea.contoso.com&dns=*.queue.sea.contoso.com&dns=*.table.sea.contoso.com&dns=*.vault.sea.contoso.com&dns=*.adminvault.sea.contoso.com&dns=*.dbadapter.sea.contoso.com&dns=*.appservice.sea.contoso.com&dns=*.scm.appservice.sea.contoso.com&dns=api.appservice.sea.contoso.com&dns=sso.appservice.sea.contoso.com&dns=adminportal.sea.contoso.com&dns=management.sea.contoso.com&dns=adminmanagement.sea.contoso.com" %szOID_ENHANCED_KEY_USAGE% = "{text}%szOID_PKIX_KP_SERVER_AUTH%,%szOID_PKIX_KP_CLIENT_AUTH%"

[RequestAttributes]


CertificateCheck script

This script is intended to be given to the customer in order to validate that the certificates are suitable before

Azure Stack deployment.

The script checks the following:

PFX can be read.

Signature algorithm is not SHA1.

Private Key is present and exported from the local machine certificate store.

Key Usage contains Digital Signature, Key Encipherment.

DNS names match the required DNS names by Azure Stack.

License requirements

An Azure subscription including Active Directory must be available before deploying Azure Stack. This

subscription can be purchased from Dell EMC, Microsoft, or other providers.

Dell EMC Hybrid Cloud for Microsoft Azure Stack comes with the required Dell EMC and Microsoft licenses,

including:

Azure Stack

o Windows Server 2016 Datacenter edition (provided as part of the Azure Stack license)

OpenManage Essentials (OME) Configuration Manager license — OME is designed for server

lifecycle management. The OME license itself is embedded in all of your Azure Stack servers from

the factory.

OpenManage Network Manager (OMNM) license — OMNM is designed for switch and networking

lifecycle management. The OMNM Licence will be provided to you before deployment. This licence

needs to be provided to the Dell EMC deployment team to be added during deployment.

Azure Stack Licensing

Dell EMC Cloud for Microsoft Azure Stack is licensed through “pay-as-you-use” metering and

consumption billing. Azure Stack consumption includes both public and private cloud workloads, and

the metering information for this usage is aggregated by Microsoft at regular intervals. The only

licensing options that can be utilized for Azure Stack consumption billing are Enterprise Agreements

(EA) and the Cloud Solution Provider (CSP) program. Note that the customer or partner is

responsible for the licensing of any 3rd party software utilized in an Azure Stack tenant.

Enterprise Agreements are ideal for organizations that already use an EA for other Microsoft software

programs. An EA agreement offers complete control of the Azure subscriptions running on the Stack

solution. Azure Stack usage is applied to the monetary commitment in the EA and support for the

Azure services is provided directly from Microsoft. An EA agreement is also the only method to

license Azure Stack if it is intended to be run in a disconnected mode. This “Capacity Model” requires

an annual subscription.

As a Azure CSP Direct and Indirect provider, Dell EMC will offer consumption-based licensing on

Azure Stack to enterprise organizations and our channel partners. Through CSP, Dell EMC provides

https://azure.microsoft.com/mediahandler/files/resourcefiles/5bc3f30c-cd57-4513-989e-056325eb95e1/Azure-Stack-packaging-and-pricing-datasheet.pdf


sales, provisioning, billing, and support. Dell EMC will bill our enterprise customers on a monthly

basis, but the CSP agreement is non-contractual. Our partners using the CSP Indirect program will

bill their end customers for their Azure usage in the format they choose, whether bundled with other

services or simply pass-through. Find out more about Azure CSP here.

Azure Stack endpoints and customer port requirements

Overview Azure Stack sets up various endpoints (VIPs - virtual IP addresses) for its infrastructure roles. These VIPs are

allocated from the public IP address pool. Each VIP is secured with an access control list (ACL) in the

software-defined network layer. ACLs are also used across the physical switches (ToRs and BMC) to further

harden the solution. A DNS entry is created for each endpoint in the external DNS zone that was specified at

deployment time.

The following architectural diagram shows the different network layers and ACLs:

Ports and Protocols (inbound) The infrastructure VIPs that are required for publishing Azure Stack endpoints to external networks are listed

in the table below. The list shows each endpoint, the required port, and protocol. Endpoints required for

additional resource providers like the SQL resource provider and others are covered in the specific resource

provider deployment documentation.

Internal infrastructure VIPs are not listed because they are not required for publishing Azure Stack.

Note: Tenant VIPs are dynamic, defined by the tenants themselves with no control by the infrastructure

operator.

https://azure.microsoft.com/en-us/offers/ms-azr-0145p/


Endpoint (VIP) DNS Host A Entry Protocol Ports

AD FS Adfs.[Region].[External FQDN] HTTPS 443

Portal (administrator) Adminportal. [Region].[External FQDN] HTTPS 443 12495 12499 12646 12647 12648 12649 12650 13001 13003 13010 13011 13020 13021 13026 30015

Azure Resource Manager (administrator)

Adminmanagement. [Region].[External FQDN]

HTTPS 443 30024

Portal (user) Portal. [Region].[External FQDN] HTTPS 443 12495 12649 13001 13010 13011 13020 13021 30015 13003

Azure Resource Manager (user)

Management. [Region].[External FQDN] HTTPS 443 30024

Graph Graph. [Region].[External FQDN] HTTPS 443

Certificate revocation list Crl. [Region].[External FQDN] HTTP 80

DNS *.[Region].[External FQDN] TCP&UDP 53

Key Vault (user) *.vault. [Region].[External FQDN] TCP TCP

443 12490

Key Vault (administrator) *.adminvault. [Region].[External FQDN] TCP TCP

443 12492

Storage Queue *.queue. [Region].[External FQDN] HTTP HTTPS

80 443

Storage Table *.table. [Region].[External FQDN] HTTP HTTPS

80 443

Storage Blob *.blob. [Region].[External FQDN] HTTP HTTPS

80 443


Ports and URLs (outbound) Azure Stack supports only transparent proxy servers. In a deployment where a transparent proxy uplinks to a

traditional proxy server, you must allow the following ports and URLs for outbound communication.

Firewall publishing The ports listed in the previous section apply to inbound communication when publishing Azure Stack

Services through an existing firewall.

We recommend that you use a firewall device to help secure Azure Stack. However, it is not a strict

requirement. Although firewalls can help for things like distributed denial-of-service (DDOS) attacks and

content inspection, they can also become a throughput bottleneck for Azure storage services like blobs,

tables, and queues.

Purpose URL Port Protocol

Identity login.windows.net login.microsoftonline.com graph.windows.net

80 & 443 http, https

Marketplace syndication

https://management.azure.com https://*.blob.core.windows.net https://*.azureedge.net https://*.microsoftazurestack.com

443 https

Patch & Update https://*.azureedge.net 443 https

Registration https://management.azure.com 443 https

Usage https://*.microsoftazurestack.com https://*.trafficmanager.com

443 https

https://*.azureedge.net/

https://*.trafficmanager.com/


Hardware infrastructure

Hardware components Minimum of 4 x PowerEdge R740xd to a maximum of 12 x R740xd.

1 x Dell EMC PowerEdge R640 Management server (Hardware Lifecycle Host)

2 x Dell EMC Networking S4048-ON Top of Rack (ToR) switches

1 x Dell EMC Networking S3048-ON Management switch

PowerEdge R740XD 2-socket, 2U rack system for demanding environments, provides ideal balance between storage, I/O and

application acceleration with superior configuration flexibility In the Dell EMC Cloud for Microsoft Azure Stack,

R740XD is configured with a total of 18 drives. Allowing for 2 SSD boot drives, 6 SSD cache drives, and 10

HDDs for storage capacity.

PowerEdge R640 Hardware Lifecycle Host

Scalable computing and storage in a 1U, 2-socket platform with an ideal mix of performance, cost and density

for most data centers.


Dell EMC Networking S4048-ON S4048 top-of-rack switches

The Dell EMC Networking S-Series

S4048-ON is an ultra-low-latency

10/40GbE top-of-rack (ToR) switch

built for applications in high

performance datacenter and computing environments. Leveraging a non-blocking switching architecture, the

S4048-ON delivers line-rate L2 and L3 forwarding capacity with ultra-low-latency to maximize network

performance. The compact S4048-ON design provides industry-leading density of 48 dual-speed 1/10GbE

(SFP+) ports as well as six 40GbE QSFP+ uplinks to conserve valuable rack space and simplify the migration

to 40Gbps in the datacenter core (each 40GbE QSFP+ uplink can also support four 10GbE ports with a

breakout cable). In addition, the S4048-ON incorporates multiple architectural features that optimize

datacenter network flexibility, efficiency and availability, PSU to I/O panel airflow for hot/cold aisle

environments, and redundant, hot-swappable power supplies and fans.

Dell EMC Networking S3048-ON Management switch

The Dell EMC Networking S-Series

S3048-ON is a low-latency switch that

features 48 x 1GbE and 4 x 10GbE

ports, a dense 1U design and up to 260Gbps performance.

Scale Unit configuration

The following images show switches and servers placement for the 12 node configuration. Dell EMC Cloud

for Microsoft Azure Stack comes pre racked stacked and cabled ready for a Dell EMC Engineer to configure

into your datacenter and complete the deployment as an IaaS platform. Additional Dell EMC consulting

services are available to help you tailor for your use.


Minimum configuration elevation: four node Scale Unit

17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 321 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 51 5249 50

Stack-ID

LNK1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

ACT50 52 5433 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

49 51 53

Stack-ID

LNK1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

ACT50 52 5433 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

49 51 53

Node01 (R740xd)

Node02 (R740xd)

Node03 (R740xd)

Node04 (R740xd)

ToR-1 (S4048-ON)

ToR-2 (S4048-ON)

Mgmt (S3048-ON)

Mgmt-Node (R640)

Rear

Server Shipping Bracket-Rear Mount


Front


Maximum configuration elevation: twelve node Scale Unit

Node01 (R740xd)

Node02 (R740xd)

Node03 (R740xd)

Node04 (R740xd)

Node06 (R740xd)

Node07 (R740xd)

Node08 (R740xd)

Node05 (R740xd)

Node09 (R740xd)

Node10 (R740xd)

Node11 (R740xd)

Node12 (R740xd)

ToR-1 (S4048-ON)

ToR-2 (S4048-ON)

Mgmt (S3048-ON)

Mgmt-Node (R640)

Rear

17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 321 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 51 5249 50

Stack-ID

LNK1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

ACT50 52 5433 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

49 51 53

Stack-ID

LNK1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

ACT50 52 5433 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

49 51 53



Front


Supported PDU options

Single Phase


Three Phase Delta


Three Phase Y


Networking

Server and Switch Port Description References

The following descriptions are used to define the server and switch port connections shown in the port mapping table below.

GMTP – Management Ports on ToR switch (S4048)

BMC – Management Switch (S3048)

HLH – Hardware Lifecyle Host (R640)

OoB – Connects to iDRAC management ports

rNDC1 – Describes the left port on Mellanox Connectx-4

rNDC2 – Describes the right port on Mellanox Connectx-4

HLH-rNDC1 – Describes left port on Intel NDC card


Cable placement and port mapping

TOR-1 (S4048) Switch TOR-2 (S4048) Switch MGMTS (S3048) Switch

Origination Port Destination Origination Port Destination Origination Port Destination

NODE-01 rNDC1-1 To

TOR1Port.01 NODE-01 rNDC2-2 To TOR2Port.01 NODE-01 OoB To BMCPort.01

NODE-02 rNDC1-1 To


NODE-03 rNDC1-1 To


NODE-04 rNDC1-1 To


NODE-05 rNDC1-1 To


NODE-06 rNDC1-1 To


NODE-07 rNDC1-1 To


NODE-08 rNDC1-1 To


NODE-09 rNDC1-1 To


NODE-10 rNDC1-1 To


NODE-11 rNDC1-1 To


NODE-12 rNDC1-1 To


NODE-13 rNDC1-1 To


NODE-14 rNDC1-1 To


NODE-15 rNDC1-1 To


NODE-16 rNDC1-1 To


TOR2Port.44 10Gb To TOR1Port.44 TOR1Port.44 10Gb To TOR2Port.44 HLH-iDRAC OoB To BMCPort.46

TOR2Port 45 10Gb To TOR1Port 45 TOR1Port 45 10Gb To TOR2Port 45 TOR1-MGMTP

1Gb To BMCPort.47

BMCPort.51 10Gb To TOR1Port.46 BMCPort.52 10Gb To TOR2Port.46 TOR2-MGMTP

1Gb To BMCPort.48

Customer Border-1

10Gb To TOR1Port.47 Customer Border-1

10Gb To TOR2Port.47 HLH- rNDC1-1 10Gb To BMCPort.49

Customer Border-2

10Gb To TOR1Port.48 Customer Border-2

10Gb To TOR2Port.48 TOR1Port.46 10Gb To BMCPort.51

TOR2Port.49 40Gb To TOR1Port.49 TOR1Port.49 40Gb To TOR2Port.49 TOR2Port.46 10Gb To BMCPort.52

TOR2Port.50 40Gb To TOR1Port.50 TOR1Port.50 40Gb To TOR2Port.50


Hardware Lifecycle Host management network connectivity R640 Rear View Server


Scale Unit – R740XD connectivity R740XD Server rear view.


Border connectivity Network integration planning is an important prerequisite for proper operation and management of the Azure

Stack solution. Planning begins during the IP distribution when you choose whether or not to use dynamic routing

with BGP. This requires assigning a 16-bit BGP autonomous system number (public or private) or using static

routing, where we assign a static default route to the border devices.

10GbE DAC

40GbE DAC

1GbE copper

Legend

10GbE Fibre

ToR-11/44 <-> ToR-1 1/441/45 <-> ToR-2 1/451/46 <-> Mgmt 1/51

1/47 <-> Customer Border1/48 <-> Customer Border

1/49 <-> ToR-2 1/491/50 <-> ToR-2 1/50

ToR-21/44 <-> ToR-1 1/441/45 <-> ToR-1 1/451/46 <-> Mgmt 1/52

1/47 <-> Customer Border1/48 <-> Customer Border

1/49 <-> ToR-1 1/491/50 <-> ToR-1 1/50

Stack-ID

LNK1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

ACT50 52 5433 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

49 51 53

Stack-ID

LNK1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

ACT50 52 5433 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

49 51 53

ToR-1 (S4048-ON)

17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 321 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 51 5249 50

ToR-2 (S4048-ON)

Mgmt (S3048-ON)

Customer Network


BGP routing

Fault Domain

Azure Stack Cloud Network BGP Routing

SDN – Software Load BalancerBGP Advertisement to TORs

peer with Router IP

Edge BGP ASN

TOR BGP ASN

BGP Prefix-ListDeny Private Network routing

Dynamic BGP Peering LinksInfrastructure network

Software BGP

ASN

Private Network - Storage and Internal VIPs

External Network – Public VIPs


BMC

TOR 1 TOR 2MLAG Peer Link

iBGP Backup Link

Using a dynamic routing protocol like BGP guarantees that your system is always aware of network changes

and facilitates administration.

As shown on this diagram, we restrict advertising of the private IP space on the ToR using a prefix-list that

denies the private IP subnets and applying it as a route-map on the connection between the ToR and the

border.

The Software Load Balancer (SLB) running inside the Azure Stack solution peers to the ToR devices so it can

dynamically advertise the VIP addresses.

To ensure that user traffic immediately and transparently recovers from failure, the VPC or MLAG configured

between the ToR devices allows the use of multi-chassis link aggregation to the hosts and HSRP or VRRP

that provides network redundancy for the IP networks.


Static routing

Fault Domain

Azure Stack Cloud Network Static Routing

SDN – Software Load BalancerBGP Advertisement to TORs

peer with Router IP

Static Routes

TOR BGP ASN

Dynamic BGP Peering LinksInfrastructure network

Software BGP

ASN

External Network – Public VIPs


BMC


iBGP Backup Link

Customer border assign static route to TOR P2P Infrastructure Network BMC Network *(Optional) Switch Infrastructure Network External NetworkTOR Switches Static Rroute 0.0.0.0/0 to Border P2P

address. Inside Azure Stack Network will use a

default BGP configuration.

Using static routes adds more fixed configuration to the border and ToR devices. It requires thorough analysis

before any change. Issues caused by a configuration error may take more time to rollback depending on the

changes made. It is not the best method, but it is supported.

To integrate using this method, the border device must be configured with static routes pointing to the ToR

devices for traffic destined to any of the networks listed on the graphic inside the yellow box.

The ToR devices must be configured with a static default route sending all traffic to the border devices. The

one traffic exception to this rule is for the private space which will be blocked using an Access Control List

applied on the ToR to border connection.

Everything else should be the same as the first method. The BGP dynamic routing will still be used inside the

rack because it is an essential tool for the SLB and other components and cannot be disabled or removed.


Transparent proxy A transparent proxy (also known as an intercepting, inline, or forced proxy) intercepts normal communication

at the network layer without requiring any special client configuration. Clients need not to be aware of the

existence of the proxy.

The Azure Stack solution does not support normal proxies. If the datacenter requires all traffic to use a proxy,

you must configure a transparent proxy to process all traffic from the rack to handle it according to policy,

separating between the zones on your network.

DMZ/WEB Server/Other Services

Azure Stack

Datacenter

BMC


iBGP Backup Link

Internet

Border 1 Border 2

Firewall, Router or

Proxy


Firewall Integration

We recommend that you use a firewall device to help secure Azure Stack. Although firewalls can help with things like

distributed denial-of-service (DDOS) attacks, intrusion detection and content inspection, they can also become a throughput

bottleneck for Azure storage services like blobs, tables, and queues.

Please read the Publish Endpoints (https://docs.microsoft.com/en-us/azure/azure-stack/azure-stack-integrate-endpoints)

article from the Datacenter Integration documentation to plan for the Firewall Integration, the article will list the

inbound/outbound ports and protocols required for Azure Stack. Additional information is also available in the Dell EMC

Cloud for Microsoft Azure Stack Planning Guide available from your Dell EMC planning consultant.

Deployment

One of Dell EMC’s primary design goals was to get our customers operational in days. This requires substantial engineering

rigor before the system gets to the customer. This results in the least amount of time spent on-site (keeping deployments

predictable and costs low), but also ensures a smooth transition for customers to get started building plans and onboarding

tenants.

To achieve this goal, software from Microsoft, and hardware, software and firmware from Dell EMC are put through a suite

of functional, performance and reliability tests in the Dell EMC engineering labs with a focus on standardizing and

automating as much as possible. Next, additional pre-deployment tests are run at the Dell EMC factory to ensure that every

system is not only fully integrated, but all possible issues are eliminated prior to shipping to the customer.

Once the rack is in place, Dell EMC technical engineers will quickly configure and integrate the hybrid cloud environment,

resulting in a fully operational platform that’s ready – within days – to deliver services with Microsoft Azure Stack.

https://docs.microsoft.com/en-us/azure/azure-stack/azure-stack-integrate-endpoints


Register your Azure Stack system (activate the system) After deployment, you must register the new Azure Stack system by following these steps:

Registration is mandatory if the customer has chosen the pay-as-you-go billing model. They will be in violation

of the licensing terms if the Azure Stack deployment is not registered and they do not report usage.

• Decide the Azure subscription for Azure Stack billing association

• Obtain agreement number for capacity-based billing model

• Obtain Azure Stack Deployment GUID

Obtain Registration Prerequisites

• Register Azure Stack from the DVM in a connected deployment, or

• Register Azure Stack from an Internet connected computer in a disconnected deployment

• Obtain the activation key

Register Azure Stack • Take the registration string to

the Azure Stack system

• Activate the system with the registration string.

Activate Azure Stack

• Renew capacity-based yearly subscription

• Change billing model (consumption v.s. capacity)

• Scale changes (add/remove nodes) for capacity-based billing

Renew / Change Registration


Operations and management software

Microsoft Azure Stack Azure Stack is an extension of Azure, bringing the agility and fast-paced innovation of cloud computing to on-

premises environments. Only Azure Stack lets you deliver Azure services from your organization’s

datacenter, while balancing the right amount of flexibility and control—for truly consistent hybrid cloud

deployments.

Read the whitepaper for more details about Azure Stack

Accessing the Azure Stack There are two portals in Azure Stack; the administrator portal and the user portal (also referred to as the

tenantportal). The following table shows how to connect to the portals.

Portal Portal URL

Administrator https://adminportal.Rack9.DellEmcAzureStack.onmicrosoft.com

User https://portal.rack9.DellEmcAzureStack.onmicrosoft.com

https://go.microsoft.com/fwlink/?LinkId=842846&clcid=0x409


The administrator portal The administrator portal enables a cloud operator to perform administrative and operational tasks. A cloud

operator can do things such as:

Monitor health and alerts

Manage capacity

Populate the marketplace

Create plans and offers

Create subscriptions for tenants

A cloud operator can also create resources such as virtual machines, virtual networks, and storage accounts.

The user portal The user portal does not provide access to any of the administrative or operational capabilities of the

administrator portal. In the user portal, a user can subscribe to public offers, and use the services that are

made available through those offers.

Privileged Endpoint (PEP) The Privileged Endpoint is a PowerShell Just Enough Access (JEA) endpoint. The endpoint is accessed via

the ERCS infrastructure VMs ERCS = Emergency Recovery Console Server.

JEA restricts the PowerShell commands that a user/admin account may run to a specified list, with control

parameters such as level of privilege and time/duration of that privilege.

Since Azure Stack is by default a locked-down system, JEA provides the necessary elevated privilege to

enable Microsoft or Dell EMC support access for deeper diagnostic and troubleshooting actions.


There is no access to MMC snap-ins, Service Fabric Explorer etc. Unlocking the Privileged Endpoint is known as “Breaking the Glass” – only Microsoft or Dell EMC support can break the glass.

Hardware Lifecycle Host software

Windows Server 2016 Datacenter edition Windows Server 2016 is the cloud-ready operating system that supports your current workloads while

introducing new technologies that make it easy to transition to cloud computing when you are ready. The Dell

EMC HLH utilizes Windows Sever 2016 Datacenter edition with Hyper-V role to host the Dell EMC

management VMs and Patch & Update tools.

OpenManage Essentials (OME) Designed for easy installation and use, OpenManage Essentials also monitors the health status of both Dell

and multi-vendor hardware environments – including anytime, anywhere access to status and alerts through

OpenManage Mobile-equipped handheld devices.

http://www.dell.com/en-us/work/learn/openmanage-essentials

OpenManage Network Manager (OMNM) Featuring an intuitive web-based interface, support for the Dell family of network switches and multi-vendor

support, OpenManage™ Network Manager makes it easier than ever to manage your converged network

infrastructure with:

One-to-many functionality to automate configuration management.

Easy monitoring and diagnoses of networking health and performance.

Ability to deploy firmware, backup and restore configurations across many switches and routers.

Affordable subscription-based model.

http://www.dell.com/en-us/work/shop/cty/pdp/spd/dell-openmanage-network-manager/force10_omnm_1438

Dell EMC SupportAssist SupportAssist is installed and enabled during HLH deployment if allowed by the customer and integrates with

OME to proactively contact Dell EMC support.

The best time to solve a problem is before it happens. Using proactive and predictive technology,

SupportAssist helps reduce your steps and time to resolution; often detecting issues before they become

critical. Benefits include:

Value — SupportAssist is available to all Azure Stack customers at no additional charge.

Improve productivity — replaces manual, high-effort routines with automated support.

Accelerate time to resolution — receive issue alerts, automatic case creation, and proactive contact

from Dell experts.

Gain insight and control — optimize enterprise devices with monthly ProSupport Plus reporting and

get predictive issue detection before the problem starts.

http://www.dell.com/en-us/work/learn/supportassist

http://www.dell.com/en-us/work/learn/openmanage-essentials

http://www.dell.com/en-us/work/shop/cty/pdp/spd/dell-openmanage-network-manager/force10_omnm_1438

http://www.dell.com/en-us/work/learn/supportassist


Security Security incorporated into the design is a key tenant of Azure Stack. The key security features enabled are:

Firmware

o TPM 2.0 and SecureBoot are enabled.

o All firmware and driver update packages are signed.

o Firmware update is secured.

Leverages Windows Cryptograms implementations.

Software

o BitLocker enabled on all physical drives.

o Defense Informations System Agency (DISA) Security Technical Implementation Guides

(STIGs) class of security policies applied enabled.

o Device guard and credential guard enabled.

o Whitelisting enabled to ensure unknown software cannot be run on host systems.

o Defender enabled on HLH host for anti-malware.

o Federal Information Processing Standards (FIPS) 140-2 compliant crypto algorithms used for

internal stack communication.

Network traffic

o Encrypted

Least privilege – Minimum authority required for each operation

Dell EMC hardware and software have the ability to enable multiple roles and users. To ensure security and meet

least privilege authority best practices and requirements for Azure Stack, we define Operator and Administrator

Roles at deployment:

Operator

o Minimum privilege to read but not modify

Server Admin

o Full access to update, modify reboot, etc.

Switch Admin

o Full access can reboot and update

As desired your Dell EMC Deployment Engineers can help you enable additional users and roles for the

Hardware Lifecyle Host.

Azure Stack roles are defined and controlled by Microsoft so may not be changed.

Secrets rotation (change password on a regular cadence) Secrets (for example, passwords, certificates, string keys) contained in the Hardware Lifecycle Host and

iDRACs should be rotated from on a regular cadence. At the end of deployment time, we will assist the

operator if desired to set up desired accounts and remove any well-known usernames and passwords.


Important: Well-known user names such as ADMIN, admin, root, Administrator, USERID, etc., are not

recommended for use. Also, passwords such as Password, Password1!, P@ssW0rd, Welcome, 1234567,

Winter10, calvin, etc., are not recommended.


Maintaining the Dell EMC Hybrid Cloud for Microsoft Azure Stack

Monitoring and alerting in Azure Stack

Patch and Update

One of the key challenges System Operators face is to safely and reliably update their Azure Stack

infrastructure while providing highly-available, mission critical services to their customers. Updates can range

in scope from software to hardware—across core components of the system. Microsoft and Dell EMC provide

customers with the ability to update their infrastructure while ensuring that business applications, services and

workloads are highly available.

Dell EMC provides tools located on the Hardware Lifecycle Host to update Dell EMC software and Azure

Stack Firmware. Microsoft Provides an Update Resource Provider and Updates tile in the Administrator portal

native to a multi-node Azure Stack deployment to simplify the update process. The Updates tile allows

operators to:

View important information such as the current stamp version.

Install updates.

Review update history for previously installed updates.

As updates are installed, an operator can view high-level status as the update process iterates through

various subsystems in Azure Stack. Example subsystems include physical hosts, service fabric, infrastructure

virtual machines, and services that provide both the administrator and user portals.

Starting at general availability, Microsoft and Dell EMC will release update packages that contain both

security and non-security related payload. It is important that customers keep their stamps current to maintain

both security and functional environments.


It is also important to note that maintenance operations may affect tenant workloads. We strongly recommend

that you notify users of the maintenance operation and that you schedule normal maintenance windows

during non-business hours as much as possible during the entire update process.

You can view Dell EMC updates at: https://support.emc.com/products/42238 and the most current Microsoft

Azure Stack information by visiting http://aka.ms/azurestackupdate.

The Patch and Update process is a two-phase process:

1. Running Dell EMC firmware Patch and Update framework

2. Running Microsoft software Patch and Update framework

Firmware patches and updates need to be installed first before running software patches and updates.

A key tenant of Azure Stack is to maintain consistency with Azure cloud. To ensure this consistency Microsoft

and Dell EMC recommend that operators keep their Azure Stack up to date with the latest updates and the

stack should not be allowed to be more than three months behind on updates to ensure timely support.

For more information on the Microsoft Servicing policy see https://docs.microsoft.com/en-us/azure/azure-stack/azure-stack-servicing-policy

Backup and recovery

The purpose of this section is to identify BCDR recommendations to help a Cloud Operator to effect a full

recovery of their Azure Stack Infrastructure deployment from a Catastrophic Event, requiring a re-deployment

of Azure Stack on hardware.

This document does not cover the steps required to recover In-Guest or Tenant data.

This guide is intended to complement the Microsoft-provided recovery steps for Azure Stack for Dell EMC

Customer Deployments.

https://support.emc.com/products/42238

http://aka.ms/azurestackupdate

https://docs.microsoft.com/en-us/azure/azure-stack/azure-stack-servicing-policy

https://docs.microsoft.com/en-us/azure/azure-stack/azure-stack-servicing-policy


Azure Stack Infrastructure Backup – the Backup Controller

Today, Microsoft Azure Stack’s Infrastructure Business Continuity and Disaster Recovery (BCDR) options are

somewhat limited. At GA, Microsoft is delivering an integrated Infrastructure Backup framework that is

available within the Azure Administrator portal:

Infrastructure data from multiple internal services is backed up by Azure Stack using the Infrastructure

Backup Controller service.

The expectation, at GA, is that the customer will provide an SMB target (NAS or otherwise) to store the

Infrastructure backup. From a space planning perspective, Microsoft is estimating 1TB will be required to

cover the Infrastructure Backup storage requirements. Given the ephemeral value of the data that actually IS

protected, Microsoft is indicating that this should address 1 week’s worth of backups.

Please note that at GA, this operation is a MANUAL one. There is no scheduler, and therefore it will be crucial

for customers to identify how frequently they will want to capture this data.

Backup Service Provider provides external share to store Azure Stack “tarball.”

Full backup periodically

Service Provider can use an existing backup solution to protect the share


Dell EMC recommends planning for sufficient storage in order to insure that there is sufficient room for

aspects that are NOT covered by the Backup Controller as well.

This includes:

Switch configuration data

Unique HLH data

Bringing into focus other aspects of recovery such as Resource Providers

What data is actually captured by the Infrastructure Backup Controller? Azure Stack separates infrastructure data from tenant data. This document will only speak to the

Infrastructure aspects of Recovery.

Tenants of Azure Stack are responsible for protecting their workloads and backing up data.

Scenarios 1. Recover Azure Stack stamp impacted by catastrophic data loss

2. Recover individual services impacted by data loss

Data In-Scope Azure Stack service data and tenant/app meta-data only. Tenant and app data must be protected separately.

Infrastructure services include all the services and micro-services like ARM, KeyVault, CRP, NRP, SRP, etc.

Azure Stack will support backup of all the data contained in each service that needs to be protected. For

example: subscriptions, plans, offers, keys, etc. This data is unique to Azure Stack and does not exist on a

system external to that cloud.

The plan is to backup at the service/micros-service level to optimize the backup time and payload size. This

also gives is the ability to control the granularity of restore.

From an admin perspective, an external file share is required so the backup engine (also referred to as the

Backup Controller) can export a compressed, encrypted file that contains all the data that gets backed up. At

GA, Backup will a manual process.

The backup engine will eventually automatically purge backups older a fixed number of days (# of days of

retention has not been established and will not be admin configurable at GA).

Customers and ISV backup partners will never be exposed to the internal implementation of how

backup and restore work in Azure Stack. This is by design.


What about PAAS data and Resource Provider VMs?

Microsoft PAAS Resource Providers At GA, Microsoft is offering the following resource providers (RPs):

SQL RP

MySQL RP

App Service (Preview)

The RPs themselves are comprised of the following:

The (My)SQL resource provider adapter VM, which is a Windows virtual machine running the provider

services.

The resource provider itself, which processes provisioning requests and exposes database

resources.

Servers that host (My)SQL Server, which provide capacity for databases, called Hosting Servers.

For SQL and MySQL, the deployment does not create SQL servers for you – it is a customer responsibility to

create “external” SQL instances themselves. These can be Azure Stack IAAS VMs, or even be outside of the

Azure Stack stamp.

The SQL instance must be allocated exclusively to the RP. It is advised that the in-guest workload backup

solution be leveraged to protect the SQL Databases as you would any other tenant workload. For example,

Avamar or Dell EMC Networker agents.

Modern web application BCDR approach Protecting modern/cloud born apps requires a richer discussion and a clear understanding of the apps BC/DR

strategy from the top/down. A bottoms/up approach where the underlying physical/hypervisor is the source is

a non-starter, especially for PaaS-based apps.

We need to start our journey from the cloud and understand how tenants are protecting their cloud

born/modern apps in Azure (or AWS, GCS, etc.). In all cases, the services do not expose an infrastructure

backup that targets the underlying machines running complex multi-tenant services. Backup is delegated up

the stack all the way to the app/tenant.

For example, most services expose primitives/CRUD operations that admin, dev, devops can use to protect a

specific resource. For example, backup of an App Service, database, replication of a blob, etc. There is no

single operation that will back-up all data repositories across all apps and subscriptions. We already know this

approach has its limits if you want an app consistent backup across multiple independent data repositories

(db, blob, table, file share, etc.). There is no such thing as “VSS” for PaaS. Long term, the most sophisticated

application will provide native backup and restore capabilities that account for consistency, item level restore,

failover, etc.

For Azure Stack, as Microsoft ships new PaaS offerings, the plan is to offer a consistent set of capabilities like

you would see in Azure. We know each service will not offer 100% of capabilities day 1 but Microsoft will

close any gaps overtime. Over time, Microsoft will document the backup/restore workflows that will work for

each service. An example of what is not in the GA release - RA-GRS/GRS support for Blob Storage. This will

https://docs.microsoft.com/en-us/azure/azure-stack/azure-stack-sql-resource-provider-deploy

https://docs.microsoft.com/en-us/azure/azure-stack/azure-stack-mysql-resource-provider-deploy

https://docs.microsoft.com/en-us/azure/azure-stack/azure-stack-app-service-deploy


impact how you design BC/DR for an app. You will have the ability to snapshot a blob, copy it to another

storage account, but native replication of blobs between two regions is not available at GA.

Third party solutions Given that modern web applications leverage standard CRUD operations (Create, Read, Update, Delete),

there are viable third party solutions that can address continuity.

A solution such as ZeroNine’s ZeroDown can fill the niche for synchronizing data inbound to a web app by

journaling the CRUD operation and playing back across multiple cloud targets. This de-coupling of the

inbound URI/CRUD command is a more modern approach to address BCRD for web apps.

http://www.zerodownsoftware.com/

What about my custom images and blob collateral for Marketplace?

Given the finite focus of Backup Controller, what about such items that fall between the cracks? The

Infrastructure aspects that are represented by Custom VM images, stored as blobs within the Azure Stack

Cloud.

Custom VM Images are, once ingested, stored in the VM Image Repository. A description of the workflow is

located here:

https://docs.microsoft.com/en-us/azure/azure-stack/azure-stack-add-vm-image

You can generate a list of VM image names by executing:

Get-AzureVMImage

AzCopy One such utility that can be leveraged to protect bespoke, customer-generated data within Azure Stack is

AzCopy.

AzCopy is a command-line utility designed to copy data to and from Microsoft Azure Blob and Table storage

using simple commands with optimal performance. You can copy data from one object to another within your

storage account, or between storage accounts. There are two version of the AzCopy: AzCopy on Windows

and AzCopy on Linux. Azure Stack only supports the Windows version.

https://docs.microsoft.com/en-us/azure/storage/common/storage-use-azcopy

Note: While AzCopy has a Windows and Linux distribution, only the Windows version is supported at GA.

AzCopy is a free, relatively performant utility that can copy Azure (and Azure Stack) BLOBS to a local target

or Azure Consistent Storage Cloud target.

Example of local copy operation: AzCopy.exe /source:https://myaccount.blob.local.azurestack.external/mycontainer

/dest:C:\myfolder /sourcekey:<key> /S

http://www.zerodownsoftware.com/

https://docs.microsoft.com/en-us/azure/azure-stack/azure-stack-add-vm-image

https://docs.microsoft.com/en-us/azure/storage/common/storage-use-azcopy


Hardware Lifecycle Host and switch configuration Dell EMC recommends that this SMB share or NAS targets ALSO be used to house the collateral used to

deploy your Azure Stack, as well as the Switch Configuration Backup information.

During the HLH and Azure Stack deployment, the Deployment Engineer will copy important configuration files

such as switch configurations, BitLocker Recovery key and deployment files to a folder on the HLH. At the

end of the deployment he will provide this files to be backed up along with your other backups.

Microsoft recommended SMB target folder structure example


Recovery from a catastrophic failure high-level workflow


Dell EMC support and consulting offerings

Field Replacement of Parts If there’s a failure of a part while in the customer’s datacenter, the customer will not be responsible for

resolving the problem. Dell EMC will own the replacement process. An onsite resource will not only come

and fix the broken part, they will also bring the system back to its functioning state. Additionally, SLAs

prevent troubleshooting beyond a handful of times. After a reasonable number of tries, we will replace the

entire node. Due to the pre-deployment testing process, Dell EMC Cloud for Microsoft Azure Stack has built-

in automation that provides alerts of any failures to enable rapid replacement for minimal disruption. In fact,

we will know there’s an issue before the customer does.

ProSupport Plus for Enterprise Better system performance and health: Dell EMC experts and

tools can help you avoid problems associated with incompatible

hardware, software, BIOS and firmware versions.

Collaborative: Dell EMC and your Technology Service Manager

work with you during the entire process, from data collection through delivery and will perform the

analysis for you.

Automation: SupportAssist and Secure Remote Services (ESRS) provide proactive, automated issue

detection, notification, case creation and reporting that reduces

systems maintenance data collection effort.

Dell EMC Hybrid Cloud for Azure Stack comes standard with Next

Business Day parts replacement but can be updated to 4 hour

replacement in many service areas.

Consulting service offerings

Our services for Dell EMC Cloud for Microsoft Azure Stack will help customers implement and integrate Azure

Stack into their existing environments.

This service helps you prepare for the solution deployment by

understanding the best use of cloud for your business and how to

optimize your integration.

Speed your path to productivity with deployment and integration services:

Leveraging our experience and expertise with hybrid cloud platforms, we have engineered an optimal

Rack Integration that ensures a consistent technology build, quality assurance, and comprehensive

oversight from configuration to delivery.


Once the rack is in place, Dell EMC technical engineers will rapidly configure and integrate your

hybrid cloud environment, resulting in a fully operational platform that is ready to deliver services with

Microsoft Azure Stack.

Many customers want to expand their Hybrid Cloud solution to deliver more value to their business. Dell EMC

Services offer optional custom services to optimize the Hybrid Cloud.

Extend your on-premises Active Directory with Azure Active Directory Federation providing a cloud-

ready directory services platform, single sign-on.

Consume and integrate with Azure Stack Public cloud.

Develop simple IaaS blueprints integrated into a Service Catalog to create complex XaaS such as

Database as a Service using Microsoft SQL Server.

For ongoing day two operations, you can take advantage of services to extend existing monitoring

and metering systems using Microsoft System Center.

When you purchase the Hybrid Cloud for Azure Stack, you also receive single contact support with ProSupport Plus – the highest level of support available, giving IT teams the confidence that each component will be fully supported by Dell EMC experts, a dedicated Technical Account Manager, 24x7 access to elite hardware and software engineers and collaborative third-party assistance. All of this with the end goal of accelerating your time to value of your hybrid cloud platform.


Cautions

Important: The recommendations and guidelines in this document are based on industry best practices, Azure

Stack architecture requirements, and Dell EMC lab testing. If not followed, the functionality and or management of

the solution may not work as designed or expected, and problem resolution may be limited, delayed, or not viable.


Additional resources

Tools for using Azure and Azure Stack https://github.com/Azure/AzureStack-Tools To use these tools, obtain Azure Stack compatible Azure PowerShell module. Unless you have installed from other

sources, one way to do it is to obtain from public package repositories as follows. Note that both of these could still be

used to operate against Azure as well as Azure Stack, but may lack some of the latest Azure features.

For PowerShell, install the following:

Install-Module -Name 'AzureRm.Bootstrapper' Install-AzureRmProfile -profile '2017-03-09-profile' -Force Install-Module -Name AzureStack -RequiredVersion 1.2.10

Obtain the tools by cloning the git repository.

git clone https://github.com/Azure/AzureStack-Tools.git --recursive cd AzureStack-Tools

Otherwise download the tools as follows:

invoke-webrequest https://github.com/Azure/AzureStack-Tools/archive/master.zip -OutFile master.zip expand-archive master.zip -DestinationPath . -Force cd AzureStack-Tools-master

Azure Resource Manager policy for Azure Stack

Constrains Azure subscription to the capabilities available in the Azure Stack.

Apply Azure Stack policy to Azure subscriptions and resource groups

Deployment of Azure Stack

Helps prepare for Azure Stack deployment.

Prepare to Deploy (boot from VHD)

Prepare to Redeploy (boot back to original/base OS)

Connecting to Azure Stack

Connect to an Azure Stack instance from your personal computer/laptop.

Connect via VPN to an Azure Stack installation

Setting up Identity for Azure Stack

Create and manage identity related objects and configurations for Azure Stack

Create Service Principals in a disconnected topology

https://github.com/Azure/AzureStack-Tools

https://github.com/Azure/AzureStack-Tools/blob/master/Policy

https://github.com/Azure/AzureStack-Tools/blob/master/Deployment

https://github.com/Azure/AzureStack-Tools/blob/master/Connect

https://github.com/Azure/AzureStack-Tools/blob/master/Identity


Azure Stack Service Administration

Manage plans and subscriptions in Azure Stack.

Add default (unlimited) plans and quotas so that tenants can create new subscriptions

Azure Stack Compute Administration

Manage compute (VM) service in Azure Stack.

Add VM Image to the Azure Stack Marketplace

Azure Stack Infrastructure Administration

Manage Azure Stack Infrastructure

Get Infrastructure Roles

Get Infrastructure Role Instances

Start Infrastructure Role Instance

Stop Infrastructure Role Instance

Restart Infrastructure Role Instance

Get Storage Capacity

Get Storage Shares

Get Scale Unit

Get Scale Unit Node

Get Gateway Pool

Get Gateway

Get SLB MUX

Get IP Pool

Add IP Pool

Get MAC Address Pool

Get Logical network

Get Alert

Close Alert

Get Update Region Summary

Get Update

Apply Update

Get Update run

AzureRM Template Validator

Validate Azure ARM Template Capabilities

Resources – Types, Location, Apiversion

Compute Capabilities – extensions, images, sizes

Storage Capabilities – SKUs

https://github.com/Azure/AzureStack-Tools/blob/master/ServiceAdmin

https://github.com/Azure/AzureStack-Tools/blob/master/ComputeAdmin

https://github.com/Azure/AzureStack-Tools/blob/master/Infrastructure

https://github.com/Azure/AzureStack-Tools/blob/master/TemplateValidator

dell emc cloud for microsoft azure stack vxrack as

Documents