dell emc cloud for microsoft azure stack · 5 dell emc cloud for microsoft azure stack | deployment...

A Dell EMC Deployment and Configuration Guide

Dell EMC Cloud for Microsoft Azure Stack Deployment Planning Guide

Version A03

Dell EMC and Microsoft Engineering November 2017

2 Dell EMC Cloud for Microsoft Azure Stack | Deployment Planning Guide | version A03

Revisions

Date Version Description

Oct 2017 A00 Initial release

Nov 2017 A01 Added updates from Microsoft version 1.2 Deployment Planning Guide Added Certificate and License requirements

Nov 2017 A02 Updates from Microsoft Deployment Companion Guide Version 1.21 to 1.25

Removed requirement for dataencipherment on external certificates

Added explicit statement for support of intermediary CAs in the chain-of-trusts of public infrastructure certificates

Added Certificate Validation Steps

Nov 2017 A03 Updates to Azure Stack Licensing section

THIS GUIDE IS FOR INFORMATIONAL PURPOSES ONLY, AND MAY CONTAIN TYPOGRAPHICAL ERRORS AND TECHNICAL INACCURACIES.

THE CONTENT IS PROVIDED AS IS, WITHOUT EXPRESS OR IMPLIED WARRANTIES OF ANY KIND BY DELL EMC or MICROSOFT

Copyright © 2017 Dell Inc. All rights reserved. Dell and the Dell EMC logo are trademarks of Dell Inc. in the United States and/or other jurisdictions. All

other marks and names mentioned herein may be trademarks of their respective companies.


Contents

CONTENTS OVERVIEW .......................................................................................................................................................................... 5

DEPLOYMENT WORKSHEET ............................................................................................................................................... 5

CUSTOMER AND ENVIRONMENT INFORMATION ..................................................................................................... 5

AZURE CONNECTION, IDENTITY STORE, BILLING MODEL DECISIONS .................................................................... 6

CUSTOMER INFORMATION ....................................................................................................................................... 10

ENVIRONMENT INFORMATION ................................................................................................................................ 13

NETWORK SETTINGS ........................................................................................................................................................ 15

NETWORK DESIGN AND INFRASTRUCTURE ............................................................................................................. 16

BMC NETWORK............................................................................................................................................................. 17

PRIVATE NETWORK ................................................................................................................................................... 18

AZURE STACK INFRASTRUCTURE NETWORK .................................................................................................................... 18

PUBLIC INFRASTRUCTURE NETWORK .............................................................................................................................. 18

PUBLIC VIP NETWORK ................................................................................................................................................... 18

SWITCH INFRASTRUCTURE NETWORK............................................................................................................................. 18

SWITCH MANAGEMENT NETWORK.......................................................................................................................... 18

IP ASSIGNMENTS ON THE DEPLOYMENT WORKSHEET .......................................................................................... 19

PHYSICAL SWITCH ACCESS CONTROL LIST .............................................................................................................. 23

DATACENTER INTEGRATION ..................................................................................................................................... 25

BORDER CONNECTIVITY ............................................................................................................................................ 25

BGP ROUTING ............................................................................................................................................................... 26

STATIC ROUTING ........................................................................................................................................................... 27

TRANSPARENT PROXY ................................................................................................................................................... 28

FIREWALL INTEGRATION .......................................................................................................................................... 29

EDGE FIREWALL SCENARIO ............................................................................................................................................ 30

ENTERPRISE/INTRANET/PERIMETER NETWORK FIREWALL SCENARIO ................................................................................ 30

NAT 31

SSL DECRYPTION ........................................................................................................................................................... 32

REQUIRED CUSTOMER-PROVIDED SECURITY CERTIFICATES ................................................................................. 32

AZURE STACK CERTIFICATES REQUIRED .................................................................................................................. 33

PAAS CERTIFICATES (OPTIONAL) ............................................................................................................................. 34


DELL EMC REQUIRED CERTIFICATES ........................................................................................................................ 35

REQUESTING CERTIFICATES USING AN INF FILE .................................................................................................... 35

VALIDATING CERTIFICATES BEFORE DEPLOYMENT ................................................................................................ 37

PERFORMING CERTIFICATE VALIDATION ................................................................................................................ 37

LICENSE REQUIREMENTS .......................................................................................................................................... 39

REGISTER YOUR AZURE STACK SYSTEM (ACTIVATE THE SYSTEM) ........................................................................ 40

REGISTER WHEN CONNECTED .................................................................................................................................. 40

REGISTER AZURE STACK WITH AZURE – PAY-AS-YOU-USE BILLING MODEL ........................................................ 41

REGISTER AZURE STACK WITH AZURE – CAPACITY BILLING MODEL .................................................................... 41

RENEW / CHANGE REGISTRATION ........................................................................................................................... 42

APPENDIX ......................................................................................................................................................................... 43

PORTS AND PROTOCOLS ........................................................................................................................................... 43

POST-DEPLOYMENT - CUSTOMER DEFINED NETWORK CONFIGURATION ........................................................................... 43

ENABLING HTTPS USING A CA-ISSUED SSL CERTIFICATE .................................................................................................... 43


OVERVIEW

This guide helps customers and Dell EMC Engineers gather pre-deployment information and make important infrastructure decisions

for Dell EMC Cloud for Microsoft Azure Stack. This information is required to correctly deploy Azure Stack to the customer

datacenter.

DEPLOYMENT WORKSHEET

The Azure Stack Deployment Worksheet is an Excel spreadsheet that collects all the necessary information and decisions in one

place. You should complete the Deployment Worksheet during your planning process and have it ready for the hardware vendor

before deployment starts, in order to help the process go quickly and smoothly.

The Azure Stack Deployment Worksheet has three tabs:

Customer and Environment Info

Network Settings

Dell EMC Consulting

The information required by the worksheet ranges across networking, security, and identity information, with many important

decisions that may require knowledge from many different areas. Therefore, you might have to pull in people from multiple teams in

your organization to complete the worksheet. It can help to talk to your hardware vendor while filling out the worksheet, as they

may have advice helpful to making your decisions.

While filling out the worksheet, you may need to make some pre-deployment configuration changes to your network environment.

This can include reserving IP address spaces for the Azure Stack solution and/or configuring your routers, switches, and firewalls to

prepare for connectivity to the new Azure Stack solution switches.

CUSTOMER AND ENVIRONMENT INFORMATION

The Customer and Environment Info tab has three sections to fill out:

Azure Connection, Identity Store, Billing Model Decisions

Customer Information

Environment Information


AZURE CONNECTION, IDENTITY STORE, BILLING MODEL DECISIONS

You can deploy Azure Stack to an environment that is connected to Azure (the default) or disconnected from Azure. This choice

defines which options are available for your identity store (Azure Active Directory or Active Directory Federation Services) and billing

model (pay-as-you-use billing or capacity-based billing). See the following diagram and chart:

This is a key decision point! Choosing ADFS or AAD is a one-time decision that you must make at

deployment time. You cannot change this later without re-deploying the entire system.

CHOOSING CONNECTED TO AZURE

If you choose the Connect to Azure option, your Azure Stack deployment will have connectivity to Azure. This means that you can

have either Azure Active Directory or Active Directory Federation Services (ADFS) for your identity store. You can also choose from

either billing model: consumption-based or capacity-based. A connected deployment is the default option because it allows

customers to get the most value out of Azure Stack, particularly for hybrid scenarios that involve both Azure and Azure Stack.

CHOOSE IDENTITY STORE

With a connected deployment, you can choose between Azure Active Directory or ADFS for your identity store. A disconnected

deployment can only use ADFS.


Your identity store choice has no bearing on tenant VMs, the identity store and accounts that they use, whether or not they can join

an Active Directory Domain, and so on. This is separate.

For example: If you deploy IaaS tenant VMs on top of Azure Stack, and want them to join a Corporate Active Directory Domain and

use accounts from there, you can still do this. You are not required to use the AAD identity store you select here for those accounts.

AZURE ACTIVE DIRECTORY IDENTITY STORE

When you use Azure Active Directory for your identity store, you need two Azure Active Directory accounts. These accounts can be

the same account, or different accounts. While using the same account might be simpler and useful if you have a limited number of

Azure accounts, your business needs might suggest using two accounts.

1. Global admin account (only required for connected deployments). This is an Azure account that is used to create

applications and service principals for Azure Stack infrastructure services in Azure Active Directory. This account must have

directory admin privileges to the directory that your Azure Stack system will be deployed under. It will become the Global

Admin for the Azure Active Directory tenant. It will be used:

a. To provision and delegate applications and service principals for all Azure Stack services that need to interact with

Azure Active Directory and Graph API.

b. As the Service Administrator account. This is the owner of the default provider subscription (which you can later

change). You can log into the Azure Stack admin portal with this account, and can use it to create offers and plans,

set quotas, and perform other administrative functions in Azure Stack.

2. Billing account (required for both connected and disconnected deployments). This Azure account that is used to establish

the billing relationship between your Azure Stack system with the Azure commerce backend. This is the account that will be

billed for Azure Stack fees. This account will also be used for marketplace syndication and other hybrid scenarios.

ACTIVE DIRECTORY FEDERATED SERVICES IDENTITY STORE

Choose this option if you want to use your own identity store, such as Active Directory, for your Service Administrator accounts. If

you want to use your Corporate Active Directory to manage your Service Administrator accounts, then this is the option for you.

CHOOSE BILLING MODEL

You can choose either Pay As You Use billing model or Capacity billing model. Pay-as-you-use deployments must be able to report

usage through a connection to Azure at least once every 30 days; therefore, if connectivity will not available, the Capacity model is

the only option.

With the pay-as-you-use billing model, usage is charged to an Azure subscription, and you only pay when you actually use the

system. If this is the model you choose, you will need an Azure subscription and the account ID associated with (for example,

[email protected]). EA, CSP, and CSL subscriptions are supported. Usage reporting is configured during Azure

Stack Registration.

In most cases, Enterprise customers will use EA subscriptions, and Service Providers will use CSP or CSL subscriptions.


If you are going to use a CSP subscription, please review the table below to identify what subscription to use, as the correct

approach depends on the exact CSP scenario:

Scenario Domain and subscription to use

If you are a Direct CSP Partner, and you will operate the Azure Stack…

Use a CSL (Common Service Layer) subscription, or in Partner Center, create an AAD tenant with a descriptive name, for example <your organization>CSPAdmin, and an Azure CSP subscription associated with it. If you are an Indirect CSP Provider, and you will operate the

Azure Stack…

If you are an Indirect CSP Reseller, and you will operate the Azure Stack…

Ask your Indirect CSP Provider to create, using Partner Center, an AAD tenant for your organization, and an Azure CSP subscription associated with it.

If you decide to use a capacity-based billing model, you must purchase an Azure Stack Capacity Plan SKU based on the capacity of your system. You will need to know the number of physical cores in your Azure Stack to purchase the correct quantity. The Agreement Number is required during the registration step for a capacity-based billing model. Please note that Capacity customers will need an EA Azure subscription for registration. The reason is that registration sets up syndication, which requires an Azure subscription. The subscription is not used for Azure Stack usage. For details on the differences between the two models, see the https://azure.microsoft.com/mediahandler/files/resourcefiles/5bc3f30c-cd57-4513-989e-056325eb95e1/Azure-Stack-packaging-and-pricing-datasheet.pdf.

CHOOSING DISCONNECTED FROM AZURE

With this option, you can deploy and use Azure Stack without a connection to the Internet. Choose this option if you:

Have security or other restrictions that require you to deploy Azure Stack in an environment that is not connected to the

Internet.

Want to block data (including usage data) from being sent to Azure.

Want to use Azure Stack purely as a private cloud solution that is deployed to your corporate Intranet, and are not

interested in hybrid scenarios.

Sometimes, this type of environment is also referred to as a “submarine scenario”.

With a disconnected deployment, you are limited to an ADFS identity store and a capacity-based billing model.

A disconnected deployment does not strictly mean that you cannot later connect your Azure Stack instance to Azure for hybrid

scenarios for tenant VMs. It means that you do not have connectivity to Azure during deployment, or you do not want to use Azure

Active Directory as your identity store. However, if you want to have connectivity to Azure after deployment, regardless of what you

want to use as your identity store, you should choose the Connect to Azure deployment option.

https://azure.microsoft.com/mediahandler/files/resourcefiles/5bc3f30c-cd57-4513-989e-056325eb95e1/Azure-Stack-packaging-and-pricing-datasheet.pdf



Physically disconnected Physically connected

Billing Must be capacity EA only

Capacity or consumption EA or CSP

Identity Must be ADFS AAD or ADFS

Marketplace syndication

Not available Supported BYOL licensing of syndicated images

Registration Not available Automated

P&U Required, requires removable media and a separate connected device

Automated

FEATURES THAT ARE IMPAIRED OR UNAVAILABLE IN DISCONNECTED MODE

Azure Stack was designed to work best when connected to Azure, so it is important to note that there are some features and

functionality that are either impaired or completely unavailable in the Disconnected mode.

Feature Impact in Disconnected mode

VM deployment with DSC extension to configure VM post deployment

Impaired – DSC extension looks to the Internet for the latest WMF.

VM deployment with Docker Extension to run Docker commands

Impaired – Docker will check the Internet for the latest version and this check will fail.

Documentation links in the Azure Stack Portal Unavailable – Links such as Give Feedback, Help, Quickstart, etc. that use an Internet URL will not work.

Alert remediation/mitigation that references an online remediation guide

Unavailable – Any alert remediation links that use an Internet URL will not work.

Marketplace syndication – The ability to select and add Gallery packages directly from the Azure Marketplace

Unavailable – This feature requires connectivity to Azure and an Azure Active Directory account.

Using Azure Active Directory federation accounts to manage an Azure Stack deployment

Unavailable – This feature requires connectivity to Azure. ADFS with a local Active Directory instance must be used instead.

Resource Providers such as WebApps and SQL Unavailable – Resource Providers such as WebApps and SQL require Internet access for content.

Command Line Interface (CLI) Impaired – CLI has reduced functionality in terms of authentication and provisioning of Service Principles.


Feature Impact in Disconnected mode

Visual Studio – Cloud discovery Impaired – Cloud Discovery will either discover different clouds or will not work at all.

Visual Studio – ADFS Impaired – Only Visual Studio Enterprise supports ADFS.

Telemetry Unavailable – Telemetry data for Azure Stack as well as any third-party gallery packages that depend on telemetry data.

Certificates Unavailable – Internet connectivity is required for Certificate Revocation List (CRL) and Online Certificate Status Protocol (OSCP) services in the context of HTTPS.

Key-Vault Impaired – A common use case for Key Vault is to have an application read secrets at runtime. For this the application needs a service principal in the directory. In Azure Active Directory, regular users (non-admins) are by default allowed to add service principals. In AD (using ADFS) they are not. This places a hurdle in the end-to-end experience because one must always go through a directory admin to add any application.

CUSTOMER INFORMATION

The Customer information section is where you provide information to integrate Azure Stack with your organizations IT

infrastructure.

COMPANY NAME

The name of your organization.

REGION NAME

This value is pre-pended to your External Domain Name suffix (see below) and used to create the FQDN of your external endpoints

(for example, regionname.cloudapp.externaldomainname.com). Even if there is only one region, you will still need to provide a

region name. It must consist of only letters and numbers between 0-9.

This is a key decision point! Choosing your region name and external domain name should be

done with careful consideration and planning, as this will form the basis of your DNS namespace. These

values cannot be changed without re-deploying Azure Stack.


If you plan to have more than one region in the future, carefully consider the following tips:

Use a region name that gives some indication of the physical location of the Azure Stack scale-units. In Azure, the region

names correspond to the geographic location of the datacenters where the compute, storage, and network resources are

located (USWest, EastAsia, NorthEurope, etc.), to give users a clear idea of where their resources will be physical located.

Use a naming convention that will be intuitive for your users. Datacenter locations are a popular choice for region names.

Make sure that your tenants can make a good choice as to where to deploy their resources based on the region name.

Keep the region name short. The region will be pre-pended to your external domain name to create the FQDN for that

region.

For example, when your tenants create a public IP address, they can create a DNS name label to associate with that public IP

address. This is useful if you want to associate the public IP address with a load balancer that will balance traffic for a web

application, for example. Tenants can pick the prefix or “Host Name”, but the suffix will be based on the region name, and the

external domain name that you choose during deployment.

For example, in Azure if you are creating a public IP address in a resource group in the WestUS Region, the DNS name label field

looks like this:

In Azure Stack, it will look similar to this, except instead of .westus, you will see the region name you chose for this field, and instead

of cloudapp.azure.com, you will see cloudapp.[ExternalDomainName] where [ExternalDomainName] will be the value you choose

for external domain name. As you can see, the value that you choose here will be used to build the URLs that will be used for your

tenant services and can get long quickly, so choose carefully.

These considerations are important even if you only have a single region, as these values cannot be changed without re-deploying

Azure Stack.

EXTERNAL DOMAIN NAME

This is the external DNS zone for your Azure Stack instance. This value, along with the region name, will be used to construct the

FQDN for all external endpoints for this Azure Stack region (for example, regionname.cloudapp.externaldomainname.com).

This is a key decision point! Choosing your region name and external domain Name should be

done with careful consideration and planning, as this will form the basis of your DNS namespace. These

values cannot be changed without re-deploying Azure Stack.

As with the region name, you should choose the external domain name very carefully, as this will be used to form all the URLs for

external endpoints that your tenants will access. It cannot be changed after you have deployed Azure Stack.


EXAMPLE: CONTOSO.COM

Let’s look at a sample deployment of a fictitious company to help illustrate how these values would be used.

Contoso wants to deploy Azure Stack and it already owns the DNS Domain, Contoso.com. They would like to leverage this existing

DNS name, since their customers are already familiar with it and their brand, and so they want to use an external domain name for

Azure Stack that is a subdomain of Contoso.com. They’re going to start out with a single region in their Chicago datacenter with

plans to add more regions in the future. They have chosen to call this Azure cloud MAST because it is simple and they like the way it

sounds.

Taking this into consideration, Contoso chooses the following values for their deployment.

Company name: Contoso

Region name: CHI

External domain name: mast.contoso.com

With this combination, here are a few examples of what the resulting URLS would be for this deployment.

The Azure Stack Tenant Portal URL for this deployment would be:

https://publicportal.chi.mast.contoso.com

What if a tenant wants to create a load balancer with a public IP address for his web application and give it a DNS name label? The

application it will be used for is a teamwork application, so the tenant uses the DNS name label “Teams”. The resulting URL for the

web application would look like this:

http://teams.chi.cloudapp.mast.contoso.com

In this example, Contoso chose an external domain name that was a subdomain of a DNS domain name that already existed. In this

case, Contoso can set up a DNS delegation for that zone down to the Azure Stack DNS so that tenants can resolve these names from

outside of the Azure Stack instance. Contoso could also, for example, set up a CNAME or alias for Azure Stack to point to

portal.mast.contoso.com that in turn points to portal.chi.mast.contoso.com. Later on, when they want to expand and add another

region in Seattle, they can set up load balancing rules to route the name portal.mast.contoso.com to either

portal.chi.mast.contoso.com or portal.sea.mast.contoso.com depending on proximity, availability, or other business rules.

Organizations can, of course, set this up differently according to their business needs. This is merely an example of the

considerations you must take into account during your namespace planning.

PRIVATE DOMAIN INFORMATION

This is used to create the internal, Active Directory integrated DNS domain that will be used for Azure Stack infrastructure services.

This domain is used for internal endpoints, service-to-service communications, infrastructure role machine accounts, group-

managed service accounts, etc. This domain and the endpoints in it are accessible only from the infrastructure subnet (see the

section on Network Settings), and are NOT exposed externally to tenants.

There are three fields that you need to complete in this section. These values are used to create the internal infrastructure domain

and domain controllers for Azure Stack.


Fully Qualified Domain Name: This is the DNS-friendly FQDN of the private infrastructure domain. You can choose your own or use the sample value (azurestack.local) provided in the Deployment Worksheet. The most important thing to keep in mind here is that this domain is completely self-contained and used only for Azure Stack infrastructure services. It is not meant to be integrated with your organizations on-premise Active Directory instance. It will, for the most part, be a black box that you will not access or configure directly. It is configured completely via automation.

Domain admin user name: The Domain Admin User name is not actually captured in the Deployment Worksheet, and is there only to let you know that you will need to provide this at deployment time.

Domain admin password: Similar to the domain user name, the domain admin password is not actually entered into the Deployment Worksheet. The field is simply meant to let you know that you will need to provide this at deployment time.

NAMING PREFIXES

During the deployment, a set of computer names and corresponding IP assignments will be automatically generated for both physical devices as well as deployment-related items (for example, management virtual machines, AD object names, etc). Provide two alpha-numeric prefix strings up to eight characters long, and it will be prepended to those environment resources for easy identification. These prefixes are used with well-known suffixes to make names consistent across all Azure Stack installations and to facilitate troubleshooting and diagnostics. For example, in the event that trace logs need to be collected, it is easier to diagnose issues if we recognize the naming pattern we see in the logs.

The Physical Machine Prefix will be used for physical switch and physical compute nodes.

The Deployment Prefix will be used by Azure Stack deployment for the infrastructure role machine names.

These two options are provided as often different teams with different naming conventions manage network devices, physical computer devices, and service specific VMs. These can be the same string if desired.

ENVIRONMENT INFORMATION

This section collects time and DNS information.

TIME ZONE

Only one time-zone setting is permitted per Azure Stack Region. This value will default to whatever time zone is configured on the DVM. However, if you want to specify this explicitly, you can do so here. For a list of valid time zone values, refer to this article.

DNS FORWARDER

Azure Stack deploys its own recursive DNS servers that are part of the solutions infrastructure. If they do not have the proper authority, these recursive DNS servers will forward DNS name queries to an upstream DNS server. This is to make sure that the authoritative resolver for that DNS name can be found, the name resolved, and the result returned to original requester.

Azure Stack DNS servers are only authoritative for the external domain name zone., For queries for DNS names outside of the Azure Stack solution, you must provide the IP Address of a DNS server in your environment that can either resolve these names or forward them as appropriate.

It is recommended that you provide at least two entries (separated by commas) in the Upstream DNS Servers field. These entries must be IP addresses of valid DNS servers accessible from your Azure Stack Public Infrastructure network (see Networking Design

https://technet.microsoft.com/en-us/library/cc749073(v=ws.10).aspx


and Infrastructure). If you do not provide these entries, or if these entries are unavailable, queries for DNS names for endpoints outside of the Azure Stack (for example, Internet endpoints like www.bing.com ) will fail.

TIME SYNCHRONIZATION

You must choose a specific time server used to synchronize Azure Stack. Time symbolization is critical to Azure Stack and its Infrastructure Roles, as it is used to generate Kerberos tickets, which are used to authenticate internal services with each other.

You must specify an IP for the time synchronization server, although most of the components in the infrastructure can resolve a URL, some can only support IP addresses. If you are using the Disconnected deployment option, you must specify a time server on your corporate network that you are sure can be reached from the infrastructure network in Azure Stack.

http://www.bing.com/


NETWORK SETTINGS

The following information helps you understand the network infrastructure for your Azure Stack deployment and integration on the

datacenter, it will also guide you on how to fill in the Deployment Worksheet with details about important decisions that will require

network knowledge of the environment. Although the configuration may vary from vendor to vendor, the requirements and concepts

are the same for all of them.


NETWORK DESIGN AND INFRASTRUCTURE

Physical Network Design

The Azure Stack solution requires a resilient and highly available physical infrastructure to support its operation and services. Below

is a diagram of our recommended design:

BMC

45

iBGP Backup Link

Border – Customer Network

Azure Stack HostNode 1

BMC

NIC2

NIC1

HLH

BMC

NIC2

NIC1


BMC

NIC2

NIC1

Azure Stack HostNode ...

BMC

NIC2

NIC1


BMC

NIC2

NIC1

46 <- -> 51

46 <- -> 52

1

1

2

2

...

...

12

1

46

50 *(OPTIONAL)

49

...

12

12

TOR 1 TOR 2

49MLAG Peer Link


Logical Networks

Logical networks represent an abstraction of the underlying physical network infrastructure. They are used to organize and simplify

network assignments for hosts, virtual machines, and services. As part of logical network creation, network sites are created to define

the virtual local area networks (VLANs), IP subnets, and IP subnet/VLAN pairs that are associated with the logical network in each

physical location.

The network infrastructure for Azure Stack uses the following logical networks that are configured on the switches:

Hyper-VVirtualSwitch

BMC

BMC

SDNSoftware BGP ASN

BMC Network

Switch Infrastructure Network

Infrastructure Network

External Network – Public VIPs

Private Network - Internal VIPs

Private Network - Storage

Hyper-VVirtual Switch

TOR SwitchTOR BGP ASN

BMC Switch

Border Switch*Edge BGP ASN

(Optional)

Deployment VM

Hardware Lifecycle Host

Azure Stack Hosts

BMC NETWORK

This network is dedicated to connecting all the baseboard management controllers (also known as service processors, for example,

iDRAC, iLO, iBMC, etc.). The (HLH) hardware lifecycle host will be located on this network and provides Dell EMC specific software for

hardware maintenance and/or monitoring. During deployment, the Azure Stack DVM runs on the BMC network and the deployment

automation tools require access to the BMC of all hosts and to the Internet where it validates some of the required resources like NTP,

DNS and Azure authentication. The minimum size of this network is /27 (30 host IP’s), this network should be routable to allow internet

access to the DVM, the ACLs (discussed in detail on the section below) block access to other components on the BMC network but

these can be allowed based on customer security preferences.


PRIVATE NETWORK

This /24 (254 host IPs) network is private to the Azure Stack region (does not expand beyond the border switch devices of the Azure

Stack region) and is divided into two subnets:

Storage network: A /25 (126 host IPs) network used to support the use of Spaces Direct and SMB (Server Message Block)

storage traffic and virtual machine live migration.

Internal virtual IP network: A /25 network dedicated to internal-only VIPs for the software load balancer.

AZURE STACK INFRASTRUCTURE NETWORK

This /24 network is dedicated to internal Azure Stack components so that they can communicate and exchange data among

themselves. This subnet requires routable IP addresses but is kept private to the solution by using (ACLs) access control lists, It isn’t

expected to be routed beyond the border switches except for a very small range equivalent in size to a /27 network utilized by some

of these services when they require access to external resources and/or the internet.

PUBLIC INFRASTRUCTURE NETWORK

This /27 network is the very small range from the Infrastructure subnet mentioned above, it does not require public IP addresses, but

it does require Internet access through a NAT or Transparent Proxy. This network will be allocated for the ERCS (Emergency Recovery

Console System), the ERCS VM requires Internet access during registration to Azure and should be routable to your management

network for troubleshooting purposes.

PUBLIC VIP NETWORK

The Public VIP Network is assigned to the network controller in Azure Stack. It’s not a logical network on the switch. The SLB uses the

pool of addresses and assigns /32 networks for tenant workloads. On the switch routing table, these /32 IPs are advertised as an

available route via BGP. This network contains the external-accessible or public IP addresses, the Azure Stack infrastructure uses at

least 8 addresses from this Public VIP Network while the remainder is used by the tenant VMs. The network size on this subnet can

range from a minimum of /26 (64 hosts) to a maximum of /22 (1022 hosts), we recommend that you plan for a /24 network.

SWITCH INFRASTRUCTURE NETWORK

This /26 network is the subnet that contains the routable point-to-point IP /30 (2 host IP’s) subnets and the loopbacks which are

dedicated /32 subnets for in-band switch management and BGP router ID. This range of IP addresses must be routable externally of

the Azure Stack solution to your datacenter, they may be private or public IPs.

SWITCH MANAGEMENT NETWORK

This /29 (6 host IPs) network is dedicated to connecting the management ports of the switches. It allows out-of-band access for

deployment, management, and troubleshooting. It is calculated from the switch infrastructure network mentioned above.


IP ASSIGNMENTS ON THE DEPLOYMENT WORKSHEET

In the Deployment Worksheet, customers are asked to provide the following network addresses to support the Azure Stack

deployment process. The deployment team uses the Deployment Worksheet tool to break out the IP networks into all the smaller

networks required by the system. For detailed descriptions of each network, see the “NETWORK DESIGN AND INFRASTRUCTURE”

section above.

In this example, we will fill the Network Settings tab of the Deployment Worksheet with the following values:

BMC Network: 10.193.132.0 /27

Private Network Storage Network & Internal VIPs: 11.11.128.0 /24

Infrastructure Network: 12.193.130.0 /24

Public Virtual IP (VIP) Network: 13.200.132.0 /24

Switch Infrastructure Network: 10.193.132.128 /26

When you run the Generate function of the Deployment Worksheet tool, it creates two new tabs on the spreadsheet. The first tab is

the Subnet Summary and it shows how the supernets were split to create all the networks required by the system. In our example

below there is only a subset of the columns found on this tab. The actual result has more details of each network listed:

Rack Subnet Type

Name IPv4 Subnet IPv4 Addresses

Border P2P Link P2P_Border/Border1_To_Rack1/TOR1 10.193.132.128/30 4




Border P2P Link P2P_Rack1/TOR1_To_Rack1/BMC 10.193.132.144/30 4

Border P2P Link P2P_Rack1/TOR2_To_Rack1/BMC 10.193.132.148/30 4

Rack1 Loopback Loopback0_Rack1_TOR1 10.193.132.152/32 1

Rack1 Loopback Loopback0_Rack1_TOR2 10.193.132.153/32 1

Rack1 Loopback Loopback0_Rack1_BMC 10.193.132.154/32 1

Rack1 P2P Link P2P_Rack1/TOR1-ibgp-1_To_Rack1/TOR2-ibgp-1 10.193.132.156/30 4

Rack1 P2P Link P2P_Rack1/TOR1-ibgp-2_To_Rack1/TOR2-ibgp-2 10.193.132.160/30 4

Rack1 VLAN BMCMgmt 10.193.132.0/27 32

Rack1 VLAN SwitchMgmt 10.193.132.168/29 8

Rack1 VLAN CL01-RG01-SU01-Storage 11.11.128.0/25 128


Rack Subnet Type

Name IPv4 Subnet IPv4 Addresses

Rack1 VLAN CL01-RG01-SU01-Infra 12.193.130.0/24 256

Rack1 Other CL01-RG01-SU01-VIPS 13.200.132.0/24 256

Rack1 Other CL01-RG01-SU01-InternalVIPS 11.11.128.128/25 128

The second tab is IP Address Usage and it shows how the IPs are consumed:

BMC Network

The supernet for the BMC network requires a /26 network at a minimum. The gateway uses the first IP in the network followed by the

BMC devices in the rack. The hardware lifecycle host has multiple addresses assigned on this network and can be used to deploy,

monitor, and support the rack. These IPs are distributed into 3 groups: DVM, InternalAccessible and ExternalAccessible.

Rack: Rack1

Name: BMCMgmt

Assigned To IPv4 Address

Network 10.193.132.0

Gateway 10.193.132.1

HLH-BMC 10.193.132.2

AzS-Node01 10.193.132.3

AzS-Node02 10.193.132.4

AzS-Node03 10.193.132.5

AzS-Node04 10.193.132.6

ExternalAccessible-1 10.193.132.19





InternalAccessible-1 10.193.132.24





CL01-RG01-SU01-DVM00 10.193.132.29

HLH-OS 10.193.132.30

Broadcast 10.193.132.31


Storage Network

The Storage network is a private network and is not intended to be routed beyond the rack. It is the first half of the Private Network

supernet and it is used by the switch distributed as shown on the table below. The gateway is the first IP in the subnet. The second

half used for the Internal VIPs is a private pool of addresses that is managed by Azure Stack SLB, and not shown on the IP Address

Usage tab. These networks support Azure Stack, and the ACLs on the ToR switches that prevent these networks from been

advertised and/or accessed outside the solution.

Rack: Rack1

Name: CL01-RG01-SU01-Storage


Network 11.11.128.0

Gateway 11.11.128.1

TOR1 11.11.128.2

TOR2 11.11.128.3

Broadcast 11.11.128.127

Azure Stack infrastructure network

The infrastructure network supernet requires a /24 network and this continues to be a /24 after the Deployment Worksheet tool

runs. The gateway will be the first IP in the subnet.

Rack: Rack1

Name: CL01-RG01-SU01-Infra


Network 12.193.130.0

Gateway 12.193.130.1

TOR1 12.193.130.2

TOR2 12.193.130.3

Broadcast 12.193.130.255


Switch Infrastructure Network

The infra network is broken into multiple networks used by the physical switch infrastructure. This is different from the Azure Stack

Infrastructure which only supports the Azure Stack software. The Switch Infra Network supports only the physical switch infrastructure.

The networks that are supported by infra are:

Name IPv4 Subnet

P2P_Border/Border1_To_Rack1/TOR1 10.193.132.128/30




P2P_Rack1/TOR1_To_Rack1/BMC 10.193.132.144/30

P2P_Rack1/TOR2_To_Rack1/BMC 10.193.132.148/30

Loopback0_Rack1_TOR1 10.193.132.152/32

Loopback0_Rack1_TOR2 10.193.132.153/32

Loopback0_Rack1_BMC 10.193.132.154/32

P2P_Rack1/TOR1-ibgp-1_To_Rack1/TOR2-ibgp-1 10.193.132.156/30

P2P_Rack1/TOR1-ibgp-2_To_Rack1/TOR2-ibgp-2 10.193.132.160/30

SwitchMgmt 10.193.132.168/29

Point-to-point (P2P): These networks allow connectivity between all switches. The subnet size is a /30 network for each P2P.

The lowest IP is always assigned to the upstream (North) device on the stack.

Loopback: These addresses are /32 networks that are assigned to each switch used in the rack. The border devices are not

assigned a loopback since they are not expected to be part of the Azure Stack solution.

Switch Mgmt or Switch Management: This /29 network supports the dedicated management interfaces of the switches in the

rack. The IPs are assigned as follows, this table can also be found on the IP Address Usage tab of the Deployment Worksheet:

Rack: Rack1

Name: SwitchMgmt


Network 10.193.132.168

Gateway 10.193.132.169

TOR1 10.193.132.170


TOR2 10.193.132.171

Broadcast 10.193.132.175

PHYSICAL SWITCH ACCESS CONTROL LIST

To protect the Azure Stack solution, we have implemented Access Control Lists (ACL) on the ToR switches. This section describes

how this security is implemented. The following table shows the sources and destinations of every network inside the Azure Stack

solution:

The table below correlates the ACL references with the Azure Stack networks:

Network ACL Reference

Description

BMC Network BMC Mgmt Deployment VM, BMC Interface, HLH server, HLH VMs.

HLH External Accessible

A set of addresses that are hosted on a HLH node, the ACL denies IP access beyond the border.

HLH Internal Accessible

A set of addresses that are hosted on the HLH node, these have access to IP resources beyond the border.

HLH DVM Azure Stack Deployment Virtual Machine, access to resources on the internet.

SwitchInfraNetwork Switch Mgmt Dedicated switch management interfaces.

Tor1 / Tor2 RouterIP

Loopback interface of the switch used for BGP peering between the SLB and switch/router.

BMC

Mgmt

HLH

Internal

Accessible

HLH

External

AccessibleHLH

DVM

Switch

Mgmt

Azure Stack

Infrastructure

Azure Stack

Infrastructure

Public

Tor1,Tor2

RouterIP Storage Internal VIPs Public-VIPs Public-Admin-VIPs

Customer/Internet

0.0.0.0

BMC Mgmt Permit Permit Permit Permit Permit Permit Permit Deny Deny Deny Deny Deny Deny

HLH Internal

Accessible Permit Permit Permit Permit Permit Permit Permit Deny Deny Permit Deny Permit Deny

HLH External

Accessible Permit Permit Permit Permit Permit Permit Permit Deny Deny Permit Permit Permit Permit

HLH DVM Permit Permit Permit Permit Permit Permit Permit Deny Deny Permit Permit Permit Permit

Switch Mgmt Permit Permit Permit Permit Permit Permit Permit Deny Deny Deny Deny Permit Deny

Azure Stack

Infrastructure Permit Permit Permit Permit Permit Permit Permit

Permit

TCP BGP Deny Permit Deny Permit Deny

Azure Stack

Infrastructure

Public Permit Permit Permit Permit Permit Permit Permit

Permit

TCP BGP Deny Permit Permit Permit Permit*

Tor1,Tor2

RouterIP Deny Deny Deny Deny Deny

Permit

TCP BGP

Permit

TCP BGP

Permit

TCP BGP Deny Deny Deny Deny Deny

Storage Deny Deny Deny Deny Deny Deny Deny Deny Permit Deny Deny Deny Deny

Internal VIPs Deny Permit Permit Permit Deny Permit Permit Deny Deny Permit Deny Permit Deny

Public-VIPs Deny Deny Permit Permit Deny Deny Permit Deny Deny Deny Permit Permit Permit

Public-Admin-VIPs Deny Permit Permit Permit Permit Permit Permit Deny Deny Permit Permit Permit Permit

Customer/Internet

0.0.0.0 Deny Deny Permit Permit Deny Deny Permit* Deny Deny Deny Permit Permit Permit

Destination

TOR

BMC

SLB

BMC TOR SLB


Network ACL Reference

Description

AzureStackInfraNetwork Azure Stack Infrastructure

Azure Stack Infrastructure services and VMs, restricted network.

Azure Stack Infrastructure

Public

Azure Stack Infrastructure services that need to talk to the internet and tenant's (NTP, DNS, AD).

StorageNetwork Storage Private IPs not routed outside of the stamp.

Internal VIPs Private IPs not routed outside of the stamp.

Public-VIPS Public VIPs Tenant network address space managed by the network controller.

Public Admin VIPs

Small subset of addresses in the Tenant pool that are required to talk to Internal-VIPs and Azure Stack Infrastructure.

Customer Network (Not on Spreadsheet)

Customer / Internet 0.0.0.0

Customer defined network. From the perspective of Azure Stack, 0.0.0.0 is the border device.

Deny* Customer can update this field to allow additional management capabilities.

Permit* Customer datacenter network; this will be defined by the customer.


DATACENTER INTEGRATION

BORDER CONNECTIVITY

Network integration planning is an important prerequisite for proper deployment, operation and management of the Azure Stack

solution. The TORs requires Layer 3 uplinks with Point-to-Point IPs (/30 networks) configured on the physical interfaces, our solution

does not support Layer 2 uplinks. Planning begins during the IP distribution when you choose whether or not to use dynamic routing

with BGP. This requires assigning a 16-bit BGP autonomous system number (public or private) or using static routing, where we

assign a static default route to the border devices.


BGP ROUTING

Fault Domain

Azure Stack Cloud Network BGP Routing

SDN – Software Load BalancerBGP Advertisement to TORs

peer with Router IP

Edge BGP ASN

TOR BGP ASN

BGP Prefix-ListDeny Private Network routing

Dynamic BGP Peering LinksInfrastructure network

Software BGP

ASN

Private Network - Storage and Internal VIPs



BMC

TOR 1 TOR 2MLAG Peer Link

iBGP Backup Link

Using a dynamic routing protocol like BGP guarantees that your system is always aware of network changes and facilitates

administration.

As shown on this drawing, we restrict advertising of the private IP space on the ToR using a prefix-list that denies the private IP

subnets and applying it as a route-map on the connection between the ToR and the border.

The Software Load Balancer (SLB) running inside the Azure Stack solution peers to the ToR devices so it can dynamically advertise

the VIP addresses.


To ensure that user traffic immediately and transparently recovers from failure, the VPC or MLAG configured between the ToR

devices allows the use of multi-chassis link aggregation to the hosts and HSRP or VRRP that provides network redundancy for the IP

networks.

STATIC ROUTING

Fault Domain

Azure Stack Cloud Network Static Routing

SDN – Software Load BalancerBGP Advertisement to TORs

peer with Router IP

Static Routes

TOR BGP ASN

Dynamic BGP Peering LinksInfrastructure network

Software BGP

ASN



BMC


iBGP Backup Link

Customer border assign static route to TOR P2P Infrastructure Network BMC Network *(Optional) Switch Infrastructure Network External NetworkTOR Switches Static Rroute 0.0.0.0/0 to Border P2P

address. Inside Azure Stack Network will use a

default BGP configuration.

Using static routes adds more fixed configuration to the border and ToR devices. It requires thorough analysis before any change.

Issues caused by a configuration error may take more time to roll back depending on the changes made. It is not the best method, but

it is supported.

To integrate using this method, the border device must be configured with static routes pointing to the ToR devices for traffic destined

to any of the networks listed on the graphic inside the yellow box.


The ToR devices must be configured with a static default route sending all traffic to the border devices. The one traffic exception to

this rule is for the private space, which will be blocked using an Access Control List applied on the ToR to border connection.

Everything else should be the same as the first method. The BGP dynamic routing will still be used inside the rack because it is an

essential tool for the SLB and other components and cannot be disabled or removed.

TRANSPARENT PROXY

A transparent proxy (also known as an intercepting, inline, or forced proxy) intercepts normal communication at the network layer

without requiring any special client configuration. Clients do not need to be aware of the existence of the proxy.

DMZ/WEB Server/Other Services

Azure Stack

Datacenter

BMC


iBGP Backup Link

Internet

Border 1 Border 2

Firewall, Router or

Proxy

The Azure Stack solution does not support normal proxies. If the datacenter requires all traffic to use a proxy, you must configure a

transparent proxy to process all traffic from the rack to handle it according to policy, separating traffic between the zones on your

network.


FIREWALL INTEGRATION

We recommend that you use a firewall device to help secure Azure Stack. Although firewalls can help with things like distributed

denial-of-service (DDOS) attacks, intrusion detection and content inspection, they can also become a throughput bottleneck for Azure

storage services like blobs, tables, and queues.

Please read the Publish Endpoints (https://docs.microsoft.com/en-us/azure/azure-stack/azure-stack-integrate-endpoints) article

from the Datacenter Integration documentation to plan for the Firewall Integration, the article will list the inbound/outbound ports

and protocols required for Azure Stack.

On this document you will find detailed scenarios for Firewall Integration and recommendations on best practices.

https://docs.microsoft.com/en-us/azure/azure-stack/azure-stack-integrate-endpoints


EDGE FIREWALL SCENARIO

In an edge deployment, Azure Stack is deployed directly behind the edge router or the firewall. On these scenarios, we support the

firewall above the border or acting as the border device if they support ECMP (Equal Cost Multi Path) on either BGP or Static Routing.

Azure Stack

Datacenter

BMC


iBGP Backup Link

Internet

Border 1 Border 2

Firewall

Azure Stack

Datacenter

BMC


iBGP Backup Link

Internet

Firewall 1 Firewall 2

Edge Router

The Public routable IP addresses are specified for the Public VIP on the External Network, we don’t recommend using Public routable

IPs on any other network for security purposes.

ENTERPRISE/INTRANET/PERIMETER NETWORK FIREWALL SCENARIO

In an Enterprise/Intranet/Perimeter deployment, Azure Stack is deployed on a multi zoned firewall or in between the edge firewall

and the internal/corporate network firewall, and its traffic will be distributed between the Secure, DMZ and Unsecure zones.

- Secure: This is the internal network, it uses internal or corporate routable IP addresses, it can be divided or not, it can have

Internet outbound access through NAT on the Firewall, it’s usually accessible from anywhere inside your datacenter and/or


the customer’s network. All Azure Stack networks should reside on the Secure zone except for the External Network/Public

VIP pool.

- DMZ: The perimeter network is where we usually deploy external or internet facing applications like Web servers. It is usually

monitored by a firewall to avoid attacks like DDoS and/or intrusion (hacking) while still allowing specified inbound traffic from

the Internet. Only the External Network/Public VIP pool of Azure Stack should reside on the DMZ zone.

- Unsecure: This is the external network, the Internet. We DO NOT recommend deploying Azure Stack on an Unsecure zone.

-

Azure Stack

Datacenter

BMC


iBGP Backup Link

Internet

Border 1 Border 2

Edge Firewall

Internal Firewall

Router/Network Core

DMZ/Transit Switch/Router

NAT

Network Address Translation (NAT) is the recommended method to allow the DVM to access the external resources/Internet during

deployment as well as the ERCS VMs during registration and troubleshooting.

NAT can also be an alternative to Public IP addresses on the External Network/Public VIPs, it is not recommended because it limits the

tenant user experience and raises complexity. The two option would be a 1:1 NAT which still requires one Public IP per user IP on the

pool or Many:1 NAT which requires a NAT rule per user VIP that contains associations to all ports a user might use.


Some of the downsides of using NAT for Public VIP are:

NAT adds overhead when managing firewall rules because users control their own endpoints and their own publishing rules

in the software-defined networking (SDN) stack. Users must contact the Azure Stack operator to get their VIPs published, and

to update the port list.

While NAT usage limits the user experience, it gives full control to the operator over publishing requests.

For hybrid cloud scenarios with Azure, consider that Azure does not support setting up a VPN tunnel to an endpoint using

NAT.

SSL DECRYPTION

Currently our recommendation on SSL Decryption is to disable it on all Azure Stack traffic, in the future we will provide guidance on

how to enable SSL Decryption for Azure Stack.

REQUIRED CUSTOMER-PROVIDED SECURITY CERTIFICATES

Azure Stack has a public infrastructure network that contains the external-accessible or public IP addresses

that are assigned to a small set of Azure Stack services. The remainder are used by the tenant VMs. You

must provide certificates with the appropriate DNS names for these Azure Stack public infrastructure

endpoints.

Note that there are some certificate restrictions in the current Azure Stack version. Below is a list of the

certificate requirements that are needed to deploy Azure Stack:

Certificate must be from a Public Certificate Authority who is included in the base OS image as part of

the Microsoft Trusted Root Authority Program. You can find the full list here:

https://gallery.technet.microsoft.com/Trusted-Root-Certificate-123665ca

The certificate can be a single wild card certificate covering all name spaces in the Subject Alternative

Name (SAN) field or can be a set of individual certificates only using wild cards for endpoints such as

storage and Key Vault where they are required.

The certificate signature algorithm cannot be SHA1, as it must be stronger.

The certificate format must be PFX, as both the public and private keys are required for Azure Stack

installation.

The certificate pfx files must have a value "Digital Signature", "KeyEncipherment"

The passwords to all certificate pfx files must be the same at the time of deployment

Ensure that the Subject Names and Subject Alternative Names of all certificates provided by the

Azure Stack Administrator match the specifications outlines in “Certificates Required”. Failure to do

so we result in failed deployments attempts.

Note: The presence of Intermediary Certificate Authorities in certificates’ chain-of-trusts IS now supported.

https://gallery.technet.microsoft.com/Trusted-Root-Certificate-123665ca


AZURE STACK CERTIFICATES REQUIRED

As described above, you must provide certificates with the appropriate DNS names for the different Azure

Stack public infrastructure endpoints. Each endpoint’s DNS name is expressed in the format:

<PREFIX>.<REGION>.<EXTERNALFQDN>

For your deployment, the REGION and EXTERNALFQDN values must match the region and external domain

names that you chose for your Azure Stack system. As an example, if my region name was “Redmond” and

my external domain name was “Contoso.com”, my DNS names would have the format

<PREFIX>.redmond.contoso.com. PREFIX values are predesignated by Microsoft to describe the endpoint

secured by the certificate.

The PREFIX values of the external infrastructure endpoints depend on the Azure Stack service that uses the

specific endpoint. Table C1 below describes the different Azure Stack public endpoints required for Azure

Stack deployments in both AAD and ADFS modes, grouped by area, as well as the namespaces used and

the certificates that are required for each namespace. Please note that the table below also describes the

folder to which you must copy the different certificates per public endpoint:

Note: You MUST copy the certificates to each folder in the folder structure that matches the identity provider

you are deploying against, AAD or ADFS. If you are using a single certificate for all endpoints, you must copy

that certificate file into each deployment folder outlined in the tables below. The folder structure is pre-built in

the DVM and can be found here: C:\CloudDeployment\Setup\Certificates.

The following table lists the required certificates for all Azure Stack deployments (AAD and ADFS):

Scope (per region)

Namespace Certificate Deployment Folder

Portals

ARM

<REGION>.<EXTERNALFQDN> portal. <REGION>.<EXTERNALFQDN> adminportal. <REGION>.<EXTERNALFQDN> management. <REGION>.<EXTERNALFQDN> adminmanagement. <REGION>.<EXTERNALFQDN>

SSL Certificate with SANs

Public Portal Admin Portal ARM Public

ARM Admin

Storage blob.<REGION>.<EXTERNALFQDN> *.blob.<REGION>.<EXTERNALFQDN> Wildcard SSL Certificate

ACS

table.<REGION>.<EXTERNALFQDN> *.queue.<REGION>.<EXTERNALFQDN> Wildcard SSL Certificate

queue.<REGION>.<EXTERNALFQDN> *.table.<REGION>.<EXTERNALFQDN> Wildcard SSL Certificate

Key Vault vault.<REGION>.<EXTERNALFQDN> *.vault.<REGION>.<EXTERNALFQDN> Wildcard SSL Certificate

KeyVault

adminvault.<REGION>.<EXTERNALFQDN> *.adminvault.<REGION>.<EXTERNALFQDN> Wildcard SSL Certificate

KeyVaultInternal

Table C1

If you deploy Azure Stack using the AAD deployment mode, you only need to request the certificates listed in

the previous table (C1). However, if you deploy Azure Stack using the ADFS deployment mode, you must


request the certificates listed in the previous table (C1) AND the additional certificates listed in the following

table (C2).

The following table lists the additional required certificates for deployments using ADFS as the identity

management system:

Scope (per region)

Namespace Certificate Deployment Folder

ADFS <REGION>.<EXTERNALFQDN> adfs.<REGION>.<EXTERNALFQDN> SSL Certificate

ADFS

Graph <REGION>.<EXTERNALFQDN> graph.<REGION>.<EXTERNALFQDN> SSL Certificate

Graph

Table C2

Note: All of the certificates listed on both tables above (C1 and C2) must have the same password.

PAAS CERTIFICATES (OPTIONAL)

If you are planning to deploy the additional Azure Stack PaaS services (SQL, MySQL, and App Service) after

Azure Stack has been deployed and configured, you will need to request additional certificates to cover the

endpoints of the PaaS services.

IMPORTANT: The certificates that you use for App Service and SQL/MySQL resource providers need

to have the same root authority as those used for the public Azure Stack endpoints.

Table C3 below describes the endpoints and certificates required for the SQL/MySQL adapters and for App

Service. Please note that you do not need to copy these certificates to the Azure Stack deployment folder.

Instead, you will be asked to provide these certificates when you install the additional resource providers.

The following table lists the certificates required for additional Azure Stack PaaS services:

Scope (per region)

Namespace Certificate Used for

SQL MySQL

dbadapter.<REGION>.<EXTERNALFQDN> *.dbadapter.<REGION>.<EXTERNALFQDN> Wildcard SSL Certificate

SQL and MySQL

App Service

appservice.<REGION>.<EXTERNALFQDN> *.appservice.<REGION>.<EXTERNALFQDN> *.scm.appservice.<REGION>.<EXTERNALFQDN> Multi Domain Wildcard SSL Certificate1

Web Traffic Default SSL Cert

api.appservice.<REGION>.<EXTERNALFQDN> SSL Certificate

API

sso.appservice.<REGION>.<EXTERNALFQDN> SSL Certificate

SSO

Table C3

1 May not be supported by all Public Certificate Authorities


DELL EMC REQUIRED CERTIFICATES

Table C4 below describes the endpoints and certificates required for the Open Manage Essentials and

Support Assist Enterprise. Please note that you do not need to copy these certificates to the Azure Stack

deployment folder. Instead, you will need provide these certificates during install of OME and SAE.

The following table lists the certificates required:

Scope Namespace Certificate Used for

OME <OMESRVNAME>.<customerFQDN> <OMESRVNAME>.<REGION>.<customerFQDN> SSL Certificate with SANs

OME

OMNM <OMNMSRVNAME>.<customerFQDN>

<OMNMSRVNAME>.<REGION>.<customerFQDN> SSL Certificate with SANs

OMNM

Table C4

REQUESTING CERTIFICATES USING AN INF FILE

One way to request certificates from either a Public CA or an Internal CA is by using an INF file to specify

details of the certificate, and then use the Windows built-in certreq.exe utility to generate a request file using

that INF. This process is described in the sections below.

Sample INF File

Below is a sample certrequest INF file that can be used to create an offline certreq file for submission to a CA

(either internal or public) that covers all of the required endpoints (including the PaaS services) in a single

wildcard certificate.

The sample INF file below assumes that:

Region = SEA

External FQDN = contoso.com

[Version] Signature="$Windows NT$"

[NewRequest] Subject = "C=US, O=Microsoft, L=Redmond, ST=Washington, CN=portal.sea.contoso.com"

Exportable = TRUE ; Private key is not exportable KeyLength = 2048 ; Common key sizes: 512, 1024, 2048, 4096, 8192, 16384 KeySpec = 1 ; AT_KEYEXCHANGE KeyUsage = 0xA0 ; Digital Signature, Key Encipherment MachineKeySet = True ; The key belongs to the local computer account ProviderName = "Microsoft RSA SChannel Cryptographic Provider" ProviderType = 12 SMIME = FALSE RequestType = PKCS10 HashAlgorithm = SHA256


; At least certreq.exe shipping with Windows Vista/Server 2008 is required to interpret the [Strings] and [Extensions] sections below

[Strings] szOID_SUBJECT_ALT_NAME2 = "2.5.29.17" szOID_ENHANCED_KEY_USAGE = "2.5.29.37" szOID_PKIX_KP_SERVER_AUTH = "1.3.6.1.5.5.7.3.1" szOID_PKIX_KP_CLIENT_AUTH = "1.3.6.1.5.5.7.3.2"

[Extensions] %szOID_SUBJECT_ALT_NAME2% = "{text}dns=*.sea.contoso.com&dns=*.blob.sea.contoso.com&dns=*.queue.sea.contoso.com&dns=*.table.sea.contoso.com&dns=*.vault.sea.contoso.com&dns=*.adminvault.sea.contoso.com&dns=*.dbadapter.sea.contoso.com&dns=*.appservice.sea.contoso.com&dns=*.scm.appservice.sea.contoso.com&dns=api.appservice.sea.contoso.com&dns=sso.appservice.sea.contoso.com&dns=adminportal.sea.contoso.com&dns=management.sea.contoso.com&dns=adminmanagement.sea.contoso.com" %szOID_ENHANCED_KEY_USAGE% = "{text}%szOID_PKIX_KP_SERVER_AUTH%,%szOID_PKIX_KP_CLIENT_AUTH%"

[RequestAttributes]


VALIDATING CERTIFICATES BEFORE DEPLOYMENT

The Certificate Checker tool described in this section is intended to be given to the cusomer along with their deploymentdata.json in

order to validate the certificates are suitable pre-deployment. This should be provided with enough time to test and get certificates

re-issued if needed. The PFX file and password should be treated as sensitive information and should remain solely with the

customer.

The script checks the following:

PFX can be read

Signature Algorithm is not SHA1

Private Key is present and exported from the local machine certificate store. Key Usage contains Digital Signature, Key Encipherment

DNS Name match the required DNS names by Azure Stack.

PERFORMING CERTIFICATE VALIDATION 1) Extract contents to of utils\certchecker.

2) Copy the customer certificates to a new directory, for example, certchecker\certs

3) Replace the customer certificate in either ADFS or AAD replace ssl.pfx in each child directory with the certificate you want

to check. The resultant paths should be:

c:\certchecker\certs\AAD\ACS\CustomerCertificate.pfx,

c:\certchecker\certs\AAD\Admin Portal\CustomerCertificate.pfx

c:\certchecker\certs\AAD\ARM Admin\CustomerCertificate.pfx

and so on…

4) Copy deploymentdata.json for the install into c:\certchecker

5) Open an elevated PowerShell window and run the following:

Cd certchecker

$password = Read-Host -Prompt "Enter PFX Password" -AsSecureString

.\CertChecker.ps1 -CertificatePath .\Certificates\ -pfxPassword $password -deploymentDataJSONPath

.\DeploymentData.json

6) The output should contain OK for all certificates and all attributes checked:

Test PFX Certificate .\certs\ADFS\ACS

Read PFX SSL.pfx: OK

Signature Algorithm: OK

Private Key: OK

Key Usage: OK

DNS Names: OK

No Profile: OK

Test PFX Certificate .\certs\ADFS\Public Portal

Read PFX SSL.pfx: OK

Signature Algorithm: OK

Private Key: OK

Key Usage: OK


DNS Names: OK

No Profile: OK


LICENSE REQUIREMENTS

An Azure subscription including Active Directory must be available before deploying Azure Stack. This

subscription can be purchased from Dell EMC, Microsoft, or other providers.

Dell EMC Hybrid Cloud for Microsoft Azure Stack comes with the required Dell EMC and Microsoft licenses,

including:

Azure Stack

o Windows Server 2016 Datacenter edition (provided as part of the Azure Stack license)

OpenManage Essentials (OME) Configuration Manager license — OME is designed for server

lifecycle management. The OME license itself is embedded in all of your Azure Stack servers from

the factory.

OpenManage Network Manager (OMNM) license — OMNM is designed for switch and networking

lifecycle management. The OMNM Licence will be provided to you before deployment. This licence

needs to be provided to the Dell EMC deployment team to be added during deployment.

Azure Stack Licensing

Dell EMC Cloud for Microsoft Azure Stack is licensed through “pay-as-you-use” metering and

consumption billing. Azure Stack consumption includes both public and private cloud workloads, and

the metering information for this usage is aggregated by Microsoft at regular intervals. The only

licensing options that can be utilized for Azure Stack consumption billing are Enterprise Agreements

(EA) and the Cloud Solution Provider (CSP) program. Note that the customer or partner is

responsible for the licensing of any 3rd party software utilized in an Azure Stack tenant.

Enterprise Agreements are ideal for organizations that already use an EA for other Microsoft software

programs. An EA agreement offers complete control of the Azure subscriptions running on the Stack

solution. Azure Stack usage is applied to the monetary commitment in the EA and support for the

Azure services is provided directly from Microsoft. An EA agreement is also the only method to

license Azure Stack if it is intended to be run in a disconnected mode. This “Capacity Model” requires

an annual subscription.

As a Azure CSP Direct and Indirect provider, Dell EMC will offer consumption-based licensing on

Azure Stack to enterprise organizations and our channel partners. Through CSP, Dell EMC provides

sales, provisioning, billing, and support. Dell EMC will bill our enterprise customers on a monthly

basis, but the CSP agreement is non-contractual. Our partners using the CSP Indirect program will

bill their end customers for their Azure usage in the format they choose, whether bundled with other

services or simply pass-through. Find out more about Azure CSP here.


https://azure.microsoft.com/en-us/offers/ms-azr-0145p/


REGISTER YOUR AZURE STACK SYSTEM (ACTIVATE THE SYSTEM)

Registration connects Azure Stack to Azure to enable pay-as-you-use billing and marketplace syndication. After deployment, you must register the new Azure Stack system by following these steps:

Registration is mandatory if the customer has chosen the pay-as-you-use billing model. They will be in violation of the licensing terms if the Azure Stack deployment is not registered and they do not report usage.

REGISTER WHEN CONNECTED

GET AZURE SUBSCRIPTION

Before registering Azure Stack with Azure, you must have:

The subscription ID for an Azure subscription. To get the ID, sign in to Azure, click More services > Subscriptions, click the

subscription you want to use, and under Essentials you can find the Subscription ID. China, Germany, and US government

cloud subscriptions are not currently supported.

The username and password for an account that is an owner for the subscription (MSA/2FA accounts are supported).

The Azure Active Directory for the Azure subscription. You can find this directory in Azure by hovering over your avatar at

the top-right corner of the Azure portal.

Registered the Azure Stack resource provider (see the Register Azure Stack Resource Provider section below for details).

Registering Azure Stack incurs no cost on your Azure subscription.

• Decide the Azure subscription for Azure Stack billing association

• Obtain agreement number for capacity-based billing model

• Obtain Azure Stack Deployment GUID

Obtain Registration Prerequisites

• Register Azure Stack from the DVM in a connected deployment, or

• Register Azure Stack from an Internet connected computer in a disconnected deployment

• Obtain the activation key

Register Azure Stack • Take the registration string to

the Azure Stack system

• Activate the system with the registration string.

Activate Azure Stack

• Renew capacity-based yearly subscription

• Change billing model (consumption v.s. capacity)

• Scale changes (add/remove nodes) for capacity-based billing

Renew / Change Registration


REGISTER AZURE STACK RESOURCE PROVIDER IN AZURE

1. Start PowerShell ISE as an administrator.

2. Log in to the Azure account that is an owner of the Azure subscription with -EnvironmentName parameter set to

"AzureCloud".

3. Register the Azure resource provider "Microsoft.AzureStack".

Example:

Login-AzureRmAccount -EnvironmentName "AzureCloud" Register-AzureRmResourceProvider -ProviderNamespace Microsoft.AzureStack -Force

REGISTER AZURE STACK WITH AZURE – PAY-AS-YOU-USE BILLING MODEL 1. Install PowerShell for Azure Stack.

2. Copy the RegisterWithAzure.ps1 script to a folder (such as C:\Temp).

3. Start PowerShell ISE as an administrator.

4. Run the Add-AzsRegistration module from the RegisterWithAzure.ps1 script, replacing the following placeholders:

CLOUDADMINCREDENTIAL is a PowerShell object that contains credential information (user name and password)

for the owner of the azure subscription.

AZUREDIRECTORYTENANTNAME is the Azure tenant directory where you would like your registration resource in

azure to be created.

AZURESUBSCRIPTIONID is the Azure subscription ID that you want to use to register Azure Stack.

.PrivilegedEndpoint is the Just-Enough-Access Computer Name, also known as Emergency Console VM.

Add-AzsRegistration -CloudAdminCredential $cloudAdminCredential -AzureDirectoryTenantName $azureDirectoryTenantName -AzureSubscriptionId $azureSubscriptionId -PrivilegedEndpoint $privilegedEndpoint -BillingModel PayAsYouUse

REGISTER AZURE STACK WITH AZURE – CAPACITY BILLING MODEL

Follow the same instruction as for pay-as-you-use, but invoke the Add-AzsRegistration module in this form.

AgreementNumber is the agreement number of the EA under which the Capacity license was purchased.

Also note that the BillingModel parameter is set to Capacity.

The other parameters are unchanged.

Add-AzsRegistration -CloudAdminCredential $cloudAdminCredential -AzureDirectoryTenantName

$azureDirectoryTenantName -AzureSubscriptionId $azureSubscriptionId -PrivilegedEndpoint $privilegedEndpoint -BillingModel

Capacity – AgreementNumber $agreementNumber

https://docs.microsoft.com/en-us/azure/azure-stack/azure-stack-powershell-install

https://github.com/Azure/AzureStack-Tools/tree/vnext/Registration


RENEW / CHANGE REGISTRATION

You will need to update or renew your registration in the following circumstances:

After you renew your capacity-based yearly subscription.

When you change your billing model.

When you scale changes (add/remove nodes) for capacity-based billing.

When any of these events happens, you must register your Azure Stack system again as explained above.


APPENDIX

PORTS AND PROTOCOLS

This information can be found in detail in the Datacenter Integration Guide.

POST-DEPLOYMENT - CUSTOMER DEFINED NETWORK CONFIGURATION

Currently the Azure Stack solution will not be shipped with a TACACS or RADIUS solution for access control of devices like the switches

and routers, nor a Syslog solution to capture switch logs, but all these devices support those services. To help integrate with an existing

TACACS, RADIUS and/or Syslog server on your environment, we will provide an extra file with the Network Switch Configuration which

will allow the engineer onsite to customize the switch to the customer’s needs.

ENABLING HTTPS USING A CA-ISSUED SSL CERTIFICATE

1. Navigate to the following directory:

<INSTALL_DIR>\oware\synergy\tomcat-7.0.70\bin\certs

2. Edit the 'makecert.sh' script:

a. Edit the group of lines starting with 'KT_' and set appropriate values for: 'KT_HOSTNAME' and 'KT_IPADDRESS'.

b. Note that changing values of other 'KT_' variables is optional.


<INSTALL_DIR>\oware\synergy\tomcat-7.0.70\bin

4. Edit the 'setenv.bat' script (Windows) or the 'sentenv.sh' script (Linux):

a. Change line FROM: "ENABLE_SSL=false" TO: "ENABLE_SSL=true"

b. Change line with 'SSL_PASSWORD=' to match 'KT_PASSWORD' value, if changed in step 2.

c. Change line with 'SSL_CERTKEYFILE=' to match 'KT_FILENAME' value, if changed in step 2.



6. Execute the 'makecert.sh' script.

a. Open a Windows command prompt, and execute: oware

Execute: cd "$OWARE_USER_ROOT/oware/synergy/tomcat-7.0.70/bin/certs"

Execute: ./makecert.sh

Script will generate three files: 1) 'selfsigned.cer' 2) 'selfsigned.jks' 3) 'selfsigned-csr.txt'


<INSTALL_DIR>\oware\synergy\tomcat-7.0.70\conf

8. Edit file: 'web.xml’ file:

a. Copy and paste contents of file: 'web-ssl-addition.xml' into file: 'web.xml' just BEFORE the last line that contains:

"</web-app>"

9. Locate the Certificate Signing Request (CSR) file you generated ('selfsigned-csr.txt').

10. Provide this generated CSR file to your CA provider.

11. Obtain your SSL certificate and ALL required Root and Intermediate certificates from your CA provider.

12. Copy all certificate files into the following directory:



13. Edit the 'importcerts-EXAMPLE.sh' script:

a. Edit values for: 'KT_PASSWORD' and 'KT_FILENAME', if changed in step 1.

b. Modify the script to match the specific alias names, files names, and number of certificates issued by your CA

provider.

Note: The alias 'server' MUST be used when importing the Domain certificate.

14. Execute the 'importcerts-EXAMPLE.sh' script.

15. Stop, update, and restart the Web Server service:

a. Navigate to the following directory:

<INSTALL_DIR>\oware\synergy\tomcat-7.0.70\bin

i. From an administrative command prompt, execute: net stop synergy

ii. From an administrative command prompt, execute: service.bat update

iii. From an administrative command prompt, execute: net start synergy

16. Close ALL open browser windows BEFORE opening the application.

All HTTP requests should now redirect to HTTPS on port 8443.

dell emc cloud for microsoft azure stack · 5 dell emc cloud for microsoft azure stack | deployment...

Documents