for red hat openstack platform 13 dell emc network edge … · dell emc network edge reference...

Dell EMC Network Edge Reference Architecturefor Red Hat OpenStack Platform 13

Dell EMC Service Provider Solutions

ii | Contents

ContentsList of Figures.................................................................................................................... iv

List of Tables...................................................................................................................... v

Trademarks........................................................................................................................ viNotes, cautions, and warnings......................................................................................... vii

Chapter 1: Overview........................................................................................................... 8Dell EMC Network Edge Reference Architecture for Red Hat OpenStack Platform 13...................... 9

Red Hat OpenStack Platform 13..............................................................................................9Distributed Compute Nodes (DCN).......................................................................................... 9DCN Topology Example..........................................................................................................10Key benefits............................................................................................................................ 11Red Hat OpenStack Platform and DCN Further Reading...................................................... 12

Hardware options...............................................................................................................................12Networking and network services......................................................................................................13Storage considerations...................................................................................................................... 13Taxonomy........................................................................................................................................... 13

Chapter 2: Edge compute server options........................................................................ 15Dell EMC PowerEdge R640 server...................................................................................................16Dell EMC PowerEdge R740 servers................................................................................................. 16Dell EMC PowerEdge R740xd servers............................................................................................. 16Configuration notes............................................................................................................................16

Chapter 3: Network architecture.......................................................................................18Network architecture overview...........................................................................................................19Infrastructure layouts..........................................................................................................................19Network components......................................................................................................................... 19

Server nodes...........................................................................................................................19Access switch or ToR............................................................................................................. 20Layer-2 and Layer-3 switching............................................................................................... 20Out of Band Management network.........................................................................................20

Chapter 4: Operational notes........................................................................................... 21High availability.................................................................................................................................. 22Service layout.....................................................................................................................................22Deployment overview.........................................................................................................................22

Chapter 5: Solution architecture.......................................................................................24Solution bundle with 25GbE/100GbE networking overview.............................................................. 25

DCN site networks overview...................................................................................................25Node type 802.1q tagging information................................................................................... 25Dell EMC Edge Reference Architecture for RHOSP layout................................................... 25Solution bundle 25GbE/100GbE network configuration......................................................... 28

Solution expansion.............................................................................................................................28

Dell EMC Network Edge Reference Architecture for Red Hat OpenStack Platform 13

Contents | iii

Chapter 6: Professional services......................................................................................30Consulting services............................................................................................................................31

Appendix A: Update history..............................................................................................33

Appendix B: Bill of materials............................................................................................ 34Bill of Materials for Dell EMC Network Edge Reference Architecture for Red Hat OpenStack

Platform DCN................................................................................................................................35Edge compute node configuration with 25GbE networking....................................................35

Subscriptions and network switches in the solution..........................................................................36Dell EMC Networking S3048-ON switch................................................................................ 36Dell EMC Networking S4048-ON optional switch...................................................................36Dell EMC Networking S5232F-ON switch.............................................................................. 37

Appendix C: References...................................................................................................38To learn more.....................................................................................................................................39

Glossary............................................................................................................................40


iv | List of Figures

List of FiguresFigure 1: DCN Topology.................................................................................................. 11

Figure 2: Dell EMC Network Edge Reference Architecture for Red Hat OpenStackPlatform DCN Taxonomy............................................................................................. 14

Figure 3: Edge solution rack layout.................................................................................27

Figure 4: 25GbE/100GbE cluster network logical architecture for DCN.......................... 28

Figure 5: Dell EMC Services for RHOSP and NFVi Ready Architecture......................... 31


List of Tables | v

List of TablesTable 1: Deployed core services......................................................................................13

Table 2: Dell EMC PowerEdge R640/Dell EMC PowerEdge R740/Dell EMCPowerEdge R740xd Tested BIOS and Firmware Versions..........................................16

Table 3: Dell EMC tested firmware versions................................................................... 17

Table 4: Channel bonding modes supported...................................................................19

Table 5: Edge compute node services............................................................................ 22

Table 6: OpenStack Edge compute network 802.1q tagging...........................................25

Table 7: Edge Compute Architecture Guide update history.............................................33

Table 8: Compute node Dell EMC PowerEdge R640......................................................35

Table 9: Compute node Dell EMC PowerEdge R740......................................................35

Table 10: Compute node Dell EMC PowerEdge R740xd................................................ 35

Table 11: S3048-ON switch............................................................................................. 36

Table 12: S4048-ON switch............................................................................................. 36

Table 13: S5232F-ON...................................................................................................... 37


vi | Trademarks

Trademarks

Copyright © 2014-2019 Dell EMC or its subsidiaries. All rights reserved.

Intel® and Xeon® are registered trademarks of Intel Corporation.

DISCLAIMER: The OpenStack® Word Mark and OpenStack Logo are either registered trademarks/service marks or trademarks/service marks of the OpenStack Foundation, in the United States and othercountries, and are used with the OpenStack Foundation's permission. We are not affiliated with, endorsedor sponsored by the OpenStack Foundation or the OpenStack community.


Notes, cautions, and warnings | vii

Notes, cautions, and warnings

A Note indicates important information that helps you make better use of your system.

A Caution indicates potential damage to hardware or loss of data if instructions are notfollowed.

A Warning indicates a potential for property damage, personal injury, or death.

This document is for informational purposes only and may contain typographical errors and technicalinaccuracies. The content is provided as is, without express or implied warranties of any kind.


8 | Overview

Chapter

1Overview

Topics:

• Dell EMC Network EdgeReference Architecture for RedHat OpenStack Platform 13

• Hardware options• Networking and network

services• Storage considerations• Taxonomy

An OpenStack® based cloud is now a common need by manyorganizations and Dell EMC with Red Hat have worked togetherto build a jointly engineered and validated edge architecture, usingRed Hat OpenStack Platform's Distributed Compute Nodes (DCN)capability, that details software, hardware, and integration points of allsolution components. The architecture provides prescriptive guidanceand recommendations for:

• Edge compute nodes• Network design• Software layout• Offers suggestion for other system configurations


Overview | 9

Dell EMC Network Edge Reference Architecture for Red Hat OpenStackPlatform 13

This reference architecture guide is built on Red Hat OpenStack Platform's (DCN), which is based on the17th OpenStack release codename Queens. Red Hat OpenStack Platform 13 is a containerized versionoffering greater scalability, resiliency and user experience.

Red Hat OpenStack Platform 13The Red Hat OpenStack Platform (RHOSP) provides the foundation to build a private or publicInfrastructure-as-a-Service (IaaS) cloud on Red Hat Enterprise Linux Server 7.7. It offers a massivelyscalable, fault-tolerant platform for the development of cloud-enabled workloads or Telco Cloud for ServiceProviders.

It is packaged so that available physical hardware can be turned into a private, public, or hybrid cloudplatform including:

• High-availability control plane• Fully distributed object storage• Persistent block-level storage• Virtual-machine provisioning engine and image storage• Authentication and authorization mechanism• Integrated networking

Distributed Compute Nodes (DCN)DCN, as part of Red Hat OpenStack Platform 13, leverages OpenStack features like Availability Zonesand provisioning over routed L3 networks with Ironic, to enable deployment of compute nodes to remotelocations. For example, a service provider may deploy several DCN sites to scale out a virtual RadioAccess Network (vRAN) implementation.

DCN has several caveats that must be considered when planning remote compute site deployment(s):

• Only Compute can be run at an edge site, other services such as persistent block storage are notsupported at this time

• Image considerations - Overcloud images for bare-metal provisioning of the remote compute nodesare pulled from the undercloud. Also, instance images for VMs running on edge nodes will initially befetched from the control plane the first time they are used, subsequent instances will use the locallycached image. Images are large files, implying a fast, reliable connection to the director node andcontrol plane is required.

• Networking:

• Latency - a round-trip between the control plane and remote site must be under 100ms or stability ofthe system could become compromised

• Drop-outs - If an edge site temporarily loses its connection to the control plane, then no OpenStackcontrol plane API or CLI operations can be executed until connectivity is restored to the site. Forexample, existing workloads will continue to run, but no new instances can be started until theconnection is restored. Any control functions like snapshotting, live migration etc. cannot occur untilthe link between the central cloud and edge site is restored, as all control features are dependent onthe control plane being able to communicate with the edge site.

Note: Connectivity issues are DCN site specific, losing connection to one DCN site does noteffect other DCN sites.

• This guide recommends using Provider networks for DCN workloads at this time. Depending on thetype of workloads you are running on the edge nodes, and existing networking policies, there areseveral ways for configuring instance IP addressing:


10 | Overview

• Static IPs using config-drive in conjunction with cloud-init - Utilizing config-drive as the metadataAPI server leverages the virtual media capabilities of Nova, which means there are no NeutronMetadata agents or DCHP relay required to assign a IP address to edge instances.

• DHCP relay - Forwards DHCP requests to Neutron at central site.

Note: A separate DHCP relay instance is required for each provider network.

• External DCHP server at edge site - in this case instance IP addresses are not managed byneutron.

• Inter-compute node awareness - A limitation of Neutron is that it is not able to identify individualcompute nodes as remote or local. Therefore each compute node, across all DCN sites, includingthe central cloud, will have a list of every other compute node. Depending on your networkingconfiguration this can happen by:

• When using VXLAN - First, the same Neutron networks must be configured at every site, thenevery Compute node will build a VXLAN tunnel (through the control plane) to all Controllers andcompute nodes regardless if they are remote or local.

• VLAN only - This method requires that identical network bridges and VLANs are used across allsites.

DCN Topology ExampleIt is possible with DCN to deploy any number of remote compute sites, each with it's own Availability Zone(AZ). Once the site is provisioned, specifying the site's AZ when launching an instance is all that is requiredto launch an instance, for example:

openstack server create --flavor tiny --image cirros \ --network edge-management-vxlan \ --security-group edge-management \ --availability-zone AZ_DCN_2 \ edge_vm_1


Overview | 11

Figure 1: DCN Topology

Key benefitsThe Dell EMC Edge Reference Architecture for RHOSP offers several benefits to help service providersand high-end enterprises rapidly implement Dell EMC hardware and Red Hat OpenStack Platform softwareat the network edge:

• Ready-to-use solution: The reference architecture has been fully engineered, validated, tested by DellEMC laboratories and documented by Dell EMC. This decreases your investment and deployment risk,and it enables faster deployment time.


12 | Overview

• Long lifecycle deployment: PowerEdge R-Series servers, recommended in the architecture, includelong-life Intel® Xeon® processors which reduces your investment risk and protects your investment forthe long-term.

• World-class professional services: The reference architecture includes Dell EMC professional servicesthat spans consulting, deployment, and design support to guide your deployment needs.

• Customizable solution: The architecture is prescriptive, but it can be customized to address eachcustomer’s unique virtual network function (vNF), like virtual Radio Access Network (vRAN) or otheredge workload requirements.

• Co-engineered and Integrated: OpenStack depends upon Linux for performance, security, hardwareenablement, networking, storage, and other primary services. The Red Hat OpenStack Platformdelivers an OpenStack distribution with the proven performance, stability, and scalability of RHEL7.7 enabling you to focus on delivering the services your customers want, instead of focusing on theunderlying operating platform.

• Deploy with confidence, as the Red Hat OpenStack Platform DCN provides hardened and stable branchreleases of OpenStack and Linux. The Red Hat OpenStack Platform is a long life release productsupported by Red Hat for a three (3) year “production phase” life cycle, well beyond the six-monthrelease cycle of unsupported, community OpenStack. Red Hat OpenStack Platform life cycle supportpolicies can be found at https://access.redhat.com/support/policy/updates/openstack/platform

• Take advantage of broad application support. Red Hat Enterprise Linux, running as guest virtualmachines, provides a stable application development platform with a broad set of ISV certifications. Youcan therefore rapidly build and deploy your cloud applications.

• Avoid vendor lock-in by moving to open technologies, while maintaining your existing infrastructureinvestments.

• Benefit from the world’s largest partner ecosystem: Red Hat has assembled the world’s largestecosystem of certified partners for OpenStack compute, storage, networking, ISV software, andservices for Red Hat OpenStack Platform deployments. This ensures the same level of broad supportand compatibility that customers enjoy today in the Red Hat Enterprise Linux ecosystem.

• Dell EMC is the largest infrastructure provider in the world with all workloads and platforms beingvalidated on it.

• Upgrade of Red Hat OpenStack Platform director-based installations.• Bring security to the cloud. Rely upon the SELinux military-grade security and container technologies of

Red Hat Enterprise Linux to prevent intrusions and protect your data, when running in public or privateclouds.

Red Hat OpenStack Platform and DCN Further ReadingFor more on Red Hat OpenStack Platform 13 and DCN please see:

• Red Hat OpenStack Platform 13• Deploying Distributed Compute Nodes to Edge Sites• Routed L3 Networks (Spine Leaf Networking)• Scaling the Overcloud

Hardware optionsTo reduce time spent on specifying hardware for an initial edge deployment, this Architecture Guide offersa full solution using validated Dell EMC PowerEdge server hardware designed to allow a wide range ofconfiguration options, including optimized configurations for edge compute nodes.

Dell EMC recommends starting an edge site deployment using components from this Architecture Guide -Version 1.0 because the hardware and operations processes comprise a flexible foundation upon which toexpand as your edge site deployment(s) grow, so your investment is protected.


https://access.redhat.com/support/policy/updates/openstack/platform

https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/

https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html/deploying_distributed_compute_nodes_to_edge_sites/deploying_distributed_compute_nodes_to_edge_sites

https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html-single/spine_leaf_networking

https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html/deploying_distributed_compute_nodes_to_edge_sites/deploying_distributed_compute_nodes_to_edge_sites#scaling_the_overcloud

Overview | 13

As noted throughout this Edge Compute Architecture Guide - Version 1.0, Dell EMC constantly addscapabilities to expand this offering, and other hardware may be available at the time of this reading. Pleasecontact your Dell EMC sales representative for more information on new hardware releases.

Networking and network servicesNetwork configuration is based upon using the Neutron-based options supported by the RHOSP codebase, and does not rely upon third-party drivers. This reference configuration is based upon the Neutronnetworking services using the ML2 drivers for OpenVswitch.

Physical networking includes:

• Core and layered networking capabilities• 10GbE or/and 25GbE networking• NIC bonding• Redundant trunking top-of-rack (ToR) switches into core routers

This enables the Dell EMC Network Edge Reference Architecture for Red Hat OpenStack Platform tooperate in a full production environment.

See Network architecture on page 18 for guidelines. Detailed designs are available through Dell EMCConsulting Services.

Storage considerationsCurrently, DCN only supports local storage (ephemeral), where each Compute node will host theephemeral disks associated with each virtual machine. A destroyed VM also removes the ephemeral disk.With the hardware configuration for the compute nodes (see Edge compute node configuration with 25GbEnetworking on page 35) using up to 24 x 600GB disks in a RAID 10, there will be approximately 6.5 TBof storage available.

Note: Future releases of DCN will support persistent volumes.

TaxonomyThe Dell EMC Network Edge Reference Architecture for Red Hat OpenStack Platform is built using thefollowing core OpenStack components, as delivered in the Red Hat OpenStack Platform. See Table 1:Deployed core services on page 13.

Note: Dell EMC Network Edge Reference Architecture for Red Hat OpenStack Platform assumesyou already have a full central Red Hat OpenStack Platform 13 deployment running the controlplane and all other core services, such as Cinder (block storage), Swift (object storage), Heat(orchestration), Keystone (identity) etc., see: Dell EMC Ready Architecture for RHOSP

Table 1: Deployed core services

Component Code name

Compute Nova

Networking Neutron

Deployment/management Tripleo (director)

Orchestration Heat

Bare-metal provisioning Ironic


https://www.dell.com/support/article/us/en/04/sln310368/dell-emc-ready-architecture-for-red-hat-openstack-platform

14 | Overview

The taxonomy presented in Figure 2: Dell EMC Network Edge Reference Architecture for Red HatOpenStack Platform DCN Taxonomy on page 14 reflects infrastructure components, and OpenStack-specific components, that are under active development by the community, Dell EMC, and Red Hat. Thetaxonomy reflects that there are two sides for cloud users:

• Dell EMC site-specific infrastructure• Standards-based API (shown in pink) interactions

The standards-based APIs are the same between all OpenStack deployments, and let customers andvendor ecosystems operate across multiple clouds. The edge site-specific infrastructure combines openand proprietary software, Dell EMC hardware, and operational processes to deliver edge computingresources.

The implementation choices are highly specific to the requirements of each edge site. Many of thesechoices can be standardized and automated using the tools in this Edge Compute Architecture Guide.Conforming to best practices helps reduce operational risk by leveraging the accumulated experience ofDell EMC, Red Hat and the broader OpenStack community.

Red Hat OpenStack Platform director (Undercloud) is used to deploy the main overcloud and edge sites(DCN). The Undercloud is a single server that runs the Red Hat OpenStack Platform director, which is asubset of OpenStack services used to deploy, manage (hardware and software) and update the Overcloudand DCN sites. In the Dell EMC Network Edge Reference Architecture for Red Hat OpenStack Platform theOpenStack edge compute servers comprise each edge site's DCN servers.

Figure 2: Dell EMC Network Edge Reference Architecture for Red Hat OpenStack Platform DCNTaxonomy


Edge compute server options | 15

Chapter

2Edge compute server options

Topics:

• Dell EMC PowerEdge R640server

• Dell EMC PowerEdge R740servers

• Dell EMC PowerEdge R740xdservers

• Configuration notes

The base validated solution supports the Dell EMC PowerEdge R640,Dell EMC PowerEdge R740xd and Dell EMC PowerEdge R740 serverlines.

Note: Please contact your Dell EMC sales representative fordetailed parts lists.


16 | Edge compute server options

Dell EMC PowerEdge R640 serverThe Dell EMC PowerEdge R640 is the ideal dual-socket, 1U platform for dense scale-out cloud computing.The scalable business architecture of the Dell EMC PowerEdge R640 is designed to maximize applicationperformance and provide the flexibility to optimize configurations based on the application and use case.

With the Dell EMC PowerEdge R640 you can create an NVMe cache pool and use either 2.5” or 3.5”drives for data storage. Combined with up to 24 DIMM’s, 12 of which can be NVDIMM’s, you have theresources to create the optimum configuration to maximize application performance in only a 1U chassis.This can simplify and speed-up deployments of the Red Hat OpenStack Platform.

Dell EMC PowerEdge R740 serversMaximize your application performance with the Dell EMC PowerEdge R740 server, which brings youthe perfect balance of accelerator cards, storage and compute resources in a 2U, 2-socket platform.With a wide range of GPU and FPGA options, the Dell EMC PowerEdge R740 has the versatility toadapt to virtually any application and provides the optimum platform for VDI deployments. The Dell EMCPowerEdge R740 offers up to 16 x 2.5” or 8 x 3.5” drives and iDRAC9, so you can scale to meet demandsand simplify the entire IT lifecycle.

Dell EMC PowerEdge R740xd serversThe Dell EMC PowerEdge R740xd delivers a perfect balance between storage scalability andperformance. The 2U two-socket platform is ideal for software defined storage. The R740xd versatility ishighlighted with the ability to mix any drive type to create the optimum configuration of SSD and HDD foreither performance, capacity or both.

The Dell EMC PowerEdge R740xd is the platform of choice for software defined storage and is thefoundation of Red Hat Ceph Storage for this Architecture Guide - Version 1.0

Configuration notesThis guide contains the full bill of materials (BOM) listing for the Dell EMC PowerEdge R640, Dell EMCPowerEdge R740 and Dell EMC PowerEdge R740xd server configurations as used to support thisArchitecture Guide.

The Dell EMC PowerEdge R640, Dell EMC PowerEdge R740 and Dell EMC PowerEdge R740xdconfigurations are used with 10GbE or/and 25GbE networking. To ensure that the network is HighAvailability (HA) ready, an additional network card is required in each node. This guide outlines thesupported cards and includes them as part of the solution.

CAUTION: You must ensure that the firmware on all servers is up to date. Otherwise, unexpectedresults may occur.

Table 2: Dell EMC PowerEdge R640/Dell EMC PowerEdge R740/Dell EMC PowerEdge R740xdTested BIOS and Firmware Versions

Product Version

BIOS 2.2.11

iDRAC with Lifecycle controller 3.34.34.34

Intel® XXV710 NIC 18.8.9


Edge compute server options | 17

Product Version

PERC H730P Mini controller (Dell EMCPowerEdge R640)

25.5.5.0005

PERC H740P Mini controller (Dell EMCPowerEdge R640)

50.5.1-2633

HBA330 Mini (Dell EMC PowerEdge R740xd) 16.17.00.3

BOSS-S1 (Dell EMC PowerEdge R740xd) 2.5.13.3020

Table 3: Dell EMC tested firmware versions

Product Version

S3048-ON firmware Dell EMC Networking OS10 Enterprise OSversion: 10.4.3.4 Build version: 10.4.3.4.213

S4048-ON firmware (optional) Dell EMC Networking OS10 Enterprise OSversion: 10.4.3.4 Build version: 10.4.3.4.213

S5232F-ON firmware Dell EMC Networking OS10 Enterprise OSversion: 10.4.3.4 Build version: 10.4.3.4.213


18 | Network architecture

Chapter

3Network architecture

Topics:

• Network architecture overview• Infrastructure layouts• Network components

This networking guide supports consistency in rapid deploymentsthrough minimal network configuration in an edge computingenvironment.


Network architecture | 19

Network architecture overviewThe Dell EMC Edge Reference Architecture for RHOSP uses the following switches:

• 2 - S5232F-ON as the Top-of-Rack (ToR) switches. ToR switches need to be redundant as they area critical component for DCN to communicate with the central Overcloud control plane and workloadprovider network.

• 1 - S3048-ON as the management switch for remote out-of-band provisioning of the physical serversand communicating with the iDRAC.

Infrastructure layoutsThe network consists of the following major network infrastructure layouts:

• Data network infrastructure - The server NICs and ToR switches.• Management network infrastructure - The BMC management network, consisting of iDRAC ports and

the out-of-band management ports the 1-rack unit (RU) S3048-ON switch.

Network componentsThe data network is primarily composed of the ToR and the aggregation switches. The followingcomponent blocks make up this network:

• Server nodes on page 19• Access switch or ToR on page 20• Layer-2 and Layer-3 switching on page 20• DCN site networks overview on page 25• Out of Band Management network on page 20

Server nodesIn order to create a highly-available solution, the network must be resilient to loss of a single networkswitch, network interface card (NIC) or bad cable. To achieve this, the network configuration uses channelbonding across the servers and switches.

There are several types (or modes) of channel bonding, but only one is recommended for the Solution. TheOpenStack edge compute nodes should use:

• 802.3ad or LACP (mode = 4)

Note: Other modes, such as balance-rr (mode=0), balance-xor (mode=2), broadcast(mode=3), balance-tlb (mode=5), and balance-alb (mode=6), are not supported. Pleasecheck with your technician for current support status of active-backup (mode = 1).

All nodes' endpoints are terminated to switch ports that have been configured for LACP bonding modeacross two S5232F-ON ToR switches for 25GbE/100GbE configured with a Virtual Link Trunkinginterconnect (VLTi) across them.

Please contact your Dell EMC sales representative for other viable options.

Table 4: Channel bonding modes supported

Channel bonding type

Node type 802.3ad (LACP mode 4)

OpenStack compute nodes Yes (solution default)


20 | Network architecture

A single port is an option when bonding is not required. However, it is neither used nor validated in theDell EMC Network Edge Reference Architecture for Red Hat OpenStack Platform. The need to eliminatesingle points of failure is taken into consideration as part of the design, and this option has been eliminatedwherever possible.

Please contact your Dell EMC sales representative for other viable options.

Access switch or ToRDell EMC’s recommended architecture uses VLT for HA between the two ToR switches, which enables theservers to terminate their Link Aggregation Group (LAG) interfaces (or bonds) into two different switchesinstead of one. This configuration enables active-active bandwidth utilization and provides redundancywithin the rack if one ToR switch fails or requires maintenance. Dell EMC recommended ToR switch is:

• 10/25/100GbE connectivity – S5232F-ON

The ToR switches are responsible for providing the different network connections to the OpenStack controlplane such as tenant networks and communication between the compute nodes of the DCN deployment.

ToR switches are also configured to enable different network function virtualization (NFV) features such asOVS-DPDK, SR-IOV, DVR, VLAN-Aware VMs etc.

Layer-2 and Layer-3 switchingDCN cluster network access requirements:

• Provisioning servers can happen solely over L3 with the exception that a DHCP relay is required formarshaling DHCP requests to the OpenStack control plane.

• The two network links - Provisioning, and Management - can have uplinks to a gateway device. TheProvisioning network can use the Red Hat OpenStack Platform director as a proxy for pulling packagesfrom a subscription server, or a gateway can be added.

• There are many tools available for Out of Band management, (OOB) for the iDRAC, which you can useafter first adding the gateway to the network, and then updating the iDRAC.

• The ToR switches are connected to a gateway device, usually a router or firewall. This device willhandle routing for all networks external to the DCN site.

Out of Band Management networkThe management network for all the servers and switches is aggregated into a Dell EMC NetworkingS3048-ON switch and it is uplinked to the S5232F-ON switches.

The Out of Band (OOB) management network is used for several functions:

• The highly available software uses it to introspect, provision, reboot and partition servers. For example,OpenStack Ironic leverages the iDRAC for some of the provisioning steps.

• When an uplink to a router is added and the iDRACs configured to use it as a gateway, there are toolsfor monitoring the servers and gathering metrics on them.

Note: Discussion of this topic is beyond the scope of this document.


Operational notes | 21

Chapter

4Operational notes

Topics:

• High availability• Service layout• Deployment overview

This section provides a basic overview of several important systemaspects.


22 | Operational notes

High availabilityIn order for the solution to be ready for production, different systems need to be fault-tolerant. This EdgeCompute Architecture Guide utilizes both hardware-based and software-based redundancy. This includes,but is not limited to:

• Operating systems are hosted on a RAID 10 hard drive set.• Critical network connections from server to switch utilize network bonding.• ToR switches are redundant.

Note:

• For external provider networks for VMs running at a DCN site, which are not managed byNeutron, network redundancy will need to be handled by the customer.

• It is assumed that the central cloud control plane has HA enabled according to the Dell EMCReady Architecture for RHOSP Guide, see: https://www.dell.com/support/article/us/en/19/sln310368/dell-emc-ready-architecture-for-red-hat-openstack-platform.

Service layoutDuring a DCN site deployment, a subset of OpenStack services will be installed each Compute node.

Table 5: Edge compute node services

Hypervisor (KVM)

Nova compute services

Neutron (ovs/ml2 agent)

Heat agent

Deployment overviewThis is an overview of the DCN deployment process that can be utilized for planning purposes:

1. Hardware setup:

• Rack and stack• Cabling• iDRAC setup• PXE NIC configuration• Server BIOS and RAID configuration• Switch configuration

2. Software setup at each DCN site:

• Install a DCHP relay(s) used to forward DCHP traffic to Neutron running on the central cloud controlplane

• At least one DCHP relay is required for provisioning compute nodes at the DCN site• Where the DHCP relay(s) are installed is up to the customer

• Discover edge site nodes• Import discovered nodes into Red Hat OpenStack Platform director• Configure Overcloud files for each DCN site• Provision DCN site• Validate DCN nodes' networking




Operational notes | 23

3. Environment tests

• Once Red Hat OpenStack Platform director has completed a DCN site deployment, a typical set ofvalidation tasks will include:

• Create a host aggregate, availability zone and flavor, with the supporting metadata to target theDCN site when launching an instance, for example:

$ openstack aggregate create dcn-site-1-agg$ openstack aggregate add host dcn-site-1-agg dcn-site-1-nova-1$ openstack aggregate set --zone dcn-site-1-az dcn-site-1-agg$ openstack aggregate set --property dcn-site-1=true dcn-site-1-agg+-------------------+-----------------------------------+| Field | Value |+-------------------+-----------------------------------+| availability_zone | dcn-site-az-1 || created_at | 2019-10-02T17:22:27.000000 || deleted | False || deleted_at | None || hosts | ['dcn-site-1-nova-1'] || id | 1 || name | dcn-site-agg-1 || properties | dcn-site-1='true' || updated_at | None |+-------------------+-----------------------------------+

$ openstack flavor create --property dcn-site-1=true dcn-site-1-flavor+----------------------------+--------------------------------------+| Field | Value |+----------------------------+--------------------------------------+| id | 4bc6c3e7-42be-4620-ae84-819459baf496 || name | dcn-site-1-flavor || os-flavor-access:is_public | True || properties | dcn-site-1='true' |+----------------------------+--------------------------------------+

• Launch an instance using the new flavor and DCN site provider network, for example:

$ openstack server create --flavor dcn-site-1-flavor \--network dcn-site-1-provider-net dcn-site-1-vm-1 ...+-----------+---------------------------------------+| Field | Value |+-----------+---------------------------------------+| flavor | dcn-site-1-flavor (4bc6c3e7-42be-...) || hostId | 2ef6c3e7-32ae-2120-ae84-819459baf488 || name | dcn-site-1-vm-1 || addresses | dcn-site-1-provider-net=192.168.1.123 || ... | ... |+--------+------------------------------------------+

• Validate instance is running and that networking and addressing is correct


24 | Solution architecture

Chapter

5Solution architecture

Topics:

• Solution bundle with25GbE/100GbE networkingoverview

• Solution expansion

This core architecture provides prescriptive guidance andrecommendations, jointly engineered by Dell EMC and Red Hat, fordeploying Dell EMC Network Edge Reference Architecture for Red HatOpenStack Platform version 13 DCN with Dell EMC infrastructure.

The goals are to:

• Provide practical system design guidance and recommendedconfigurations

• Develop sample networking configurations capable of supportingyour production system

The development of this architecture builds upon the experienceand engineering skills of Dell EMC and Red Hat, and encapsulatesbest practices developed in numerous real-world deployments. Thedesigns and configurations in this architecture have been tested in DellEMC and Red Hat labs to verify system functionality and operationalrobustness.

The solution consists of the components shown in Figure 3: Edgesolution rack layout on page 27, and represents a sample of asingle DCN site.

Note: Currently, Red Hat OpenStack Platform 13 supports upto a total of 300 compute nodes across all sites, including thecore OpenStack installation, see Scaling the Overcloud

This guide assumes you have already deployed Red Hat OpenStackPlatform 13 following our Dell EMC Ready Architecture for RHOSP 13,and are using this guide to implement DCN sites.



Solution architecture | 25

Solution bundle with 25GbE/100GbE networking overviewSince the solution bundle is designed for a production environment, the networking is based upon 25GbEbonds for data networks, and the network switches are configured for HA. The Out of Band Management(iDRAC's) network is not HA, and is 1GbE. The 100GbE networking is used in the solution for the user/tenant traffic.

For basic hardware configuration refer to Bill of materials on page 34.

DCN site networks overviewEach edge site is configured with the servers internal bridged networks for VM workloads, servermanagement and control plane access. This Edge Compute Architecture Guide implements at a minimumfive (5) separate networks:

• Management/Out of Band (OOB) - VLAN network for external access to iDRACs. The OpenStackcontrol plane needs direct access to this network for IPMI/WSMAN and other out-of-band operations.

• Provisioning (Bare Metal) network - Routed Neutron L3 network that requires a DCHP relay toforward DCHP requests to the central cloud control plane.

• Private API - VLAN network used by DCN compute nodes to access the OpenStack control planemessaging system.

• Tenant network - VLAN or VXLAN tunnel for accessing VMs from the central cloud and communicationbetween VMs at the DCN site.

• External network - Provider network using config-drive and cloud-init for instance metadata (IPaddress, hostname etc.). This network enables external access to VMs running at the DCN site.

Note: Provider networks can be implemented in a wide variety of ways which is beyond thescope of this document. Please contact Dell EMC for more information.

Node type 802.1q tagging informationThe solution is designed with the idea that different network traffic should be segregated from other traffic.This is accomplished by utilizing 802.1q VLAN Tagging for the different segments. The table Table 6:OpenStack Edge compute network 802.1q tagging on page 25 summarizes this. This segregation isindependent of network speed.

Table 6: OpenStack Edge compute network 802.1q tagging

Network Tagging

iDRAC physical connection to the management/OOBVLAN

Untagged

Provisioning VLAN Untagged

Tenant network (VLAN/VXLAN) Tagged

Private API network VLAN Tagged

Provider network (external access for VMs) Dependent on type of provider networkimplementation

Dell EMC Edge Reference Architecture for RHOSP layout• Network switches:

• Two (2) Dell EMC Networking S5232F-ON• One (1) Dell EMC Networking S3048-ON

• Compute nodes - Up to 20 servers per DCN site, in any combination of:



• Dell EMC PowerEdge R640• Dell EMC PowerEdge R740• Dell EMC PowerEdge R740xd

See Edge compute node configuration with 25GbE networking on page 35 for edge compute nodehardware configurations.



Figure 3: Edge solution rack layout



Solution bundle 25GbE/100GbE network configurationThe network for the Dell EMC Network Edge Reference Architecture for Red Hat OpenStack Platform hasbeen designed to support production-ready servers with a highly available network configuration.

The basic logical network for a single DCN site will look like :

Figure 4: 25GbE/100GbE cluster network logical architecture for DCN

Solution expansionThe Dell EMC Network Edge Reference Architecture for Red Hat OpenStack Platform can be expandedby:



• Adding compute nodes to an existing DCN site deployment(s). No more than 20 compute nodes per siteare supported at this time.

• Deploy additional DCN site(s)

Note: Currently, Red Hat OpenStack Platform 13 supports a up to a total of 300 compute nodesacross all sites, including the core OpenStack installation. For other expansion details, pleasespeak with your Dell EMC sales representative.


30 | Professional services

Chapter

6Professional services

Topics:

• Consulting services

Dell EMC Professional Services understands the complexity andchallenges in an accelerated deployment, integration and operationsof Red Hat OpenStack Platform. At the core, the challenges are notsolely dealing with new technology – but it also includes people andprocess changes needed to be successful. Over the years Dell EMCConsulting has led hundreds of successful cloud implementations.

Dell EMC's Consulting is well equipped in tackling the challengesin implementing and integrating Red Hat OpenStack Platform andnetwork functions virtualization infrastructure, (NFVi) technologies.Furthermore, Dell EMC has a robust, comprehensive portfolio focusedon Red Hat OpenStack Platform based solutions that have evolvedfrom our heritage in enterprise cloud solutions – and is specificallytailored to the unique and demanding Telco/Cloud service providerrequirements for NFVi implementations. The figure below details thecore service from the Dell EMC Services portfolio that our customerscan leverage to support Red Hat OpenStack Platform and NFViinitiatives.


Professional services | 31

Consulting services

Figure 5: Dell EMC Services for RHOSP and NFVi Ready Architecture

• Dell EMC Red Hat OpenStack Platform, NFVi Strategy Workshop – Dell EMC NFV/SDN/Openstackworkshop is intended to help customers who are starting their cloud transformation journey to develop acomprehensive NFV, RHOSP adoption strategy.

It helps customers prioritize the use cases based on business drivers, understand the technology andplatform choices such as VMware VIO, Red Hat, etc., identify infrastructure virtualization opportunities,develop strategic roadmap for implementation phases.

This includes understanding current state, understand requirements, GAP analysis and collaborativeservices to define technology, process implementation paths to ensure successful transformation.

• Dell EMC Red Hat OpenStack Platform, NFVi Business Advisory – This service is intended to helpcustomers develop a business case to adopt NFV based business drivers. It helps to demonstrate tosenior management the benefits of moving towards a shared NFV, RHOSP cloud infrastructure.

This services includes an independent and collaborative review of the business objectives to developa comprehensive business case for the transformation including the Total cost of ownership/ Return onInvestment financial model.

• Dell EMC Red Hat OpenStack Platform, NFVi Design Services – Dell EMC will help the customersdevelop a Low Level Design (LLD) and High Level Design (HLD) based on customer requirements anduse cases. Dell EMC consultants will work with the customer to gather data, use cases and plans forthe OpenStack, NFVI deployment.

Based on that, the team will help develop the end-to-end design that can be used for implementation ofthe RHOSP ready architecture leveraging JetPack automation.

• Dell EMC Red Hat OpenStack Platform, NFVi Implementation Services – Dell EMC Edge ReferenceArchitecture for RHOSP and solutions offer a risk free option for our customers.

To ensure faster time to value Dell EMC Implementation services provide on or off-site infrastructureplanning and implementation of the RHOSP ready architecture leveraging JetPack automation.


32 | Professional services

• Dell EMC Red Hat OpenStack Platform, NFVi Residency/Operate Services – Dell EMC will helpcustomers operate the new cloud infrastructure by providing resources as on-site/off-site residents.These residents will manage the lifecycle of the cloud and also assist with automation and onboardingof workload to the OpenStack cloud.

To engage Dell EMC consulting services, please contact your Dell EMC sales representative or send anemail.


http://[email protected]

http://[email protected]

Update history | 33

Appendix

AUpdate history

Table 7: Edge Compute Architecture Guide update history

Version Comments

1.0 Initial release


34 | Bill of materials

Appendix

BBill of materials

Topics:

• Bill of Materials for Dell EMCNetwork Edge ReferenceArchitecture for Red HatOpenStack Platform DCN

• Subscriptions and networkswitches in the solution

This guide provides Bill of material information necessary to purchasethe proper hardware to deploy the Dell EMC Network Edge ReferenceArchitecture for Red Hat OpenStack Platform.

Note: For cable, racks, power please contact your Dell EMCsupport representative.


Bill of materials | 35

Bill of Materials for Dell EMC Network Edge Reference Architecture for RedHat OpenStack Platform DCN

Up to 20 Dell EMC PowerEdge R-Series compute servers are supported per DCN site in any combinationof the following:

• Dell EMC PowerEdge R640• Dell EMC PowerEdge R740• Dell EMC PowerEdge R740xd

Edge compute node configuration with 25GbE networking

Table 8: Compute node Dell EMC PowerEdge R640

Platform Dell EMC PowerEdge R640

CPU Intel® Xeon® Gold 6230 CPU @ 2.10GHz

RAM (minimum) 192GB DDR-4 2933 MHz

LOM 1 x Intel® X710 10GbE SFP+

Add-in network 2 x Intel® XXV710 DP 25GbE DA/SFP+ Adapter

Disk 8 x 600GB 10k SAS 12Gbps

Storage controller PERC H740 Mini controller

RAID RAID 10

Table 9: Compute node Dell EMC PowerEdge R740

Platform Dell EMC PowerEdge R740





Disk 8 x 600GB 10k SAS 12Gbps


RAID RAID 10

Table 10: Compute node Dell EMC PowerEdge R740xd

Platform Dell EMC PowerEdge R740xd





Disk* 8-24 x 600GB 10k SAS 12Gbps


36 | Bill of materials

Platform Dell EMC PowerEdge R740xd


RAID RAID 10

Note: * When choosing Dell EMC PowerEdge R740xd as compute nodes, all nodes must have thesame number of disks.

Note: Be sure to consult your Dell EMC account representative before changing the recommendedhardware configurations.

Subscriptions and network switches in the solutionA Dell EMC sales representative will determine the correct software subscriptions needed for the Dell EMCNetwork Edge Reference Architecture for Red Hat OpenStack Platform and Dell EMC Networking OS10subscriptions.

Required subscriptions:

• Red Hat OpenStack Platform• Dell EMC Networking OS10• Red Hat Satellite - optional

Note: Please contact your Dell EMC sales representative.

Dell EMC Networking S3048-ON switch

Table 11: S3048-ON switch

Product Description

S3048-ON 48 line-rate 1000BASE-T ports, 4 line-rate 10GbESFP+ ports (1 qty)

Redundant power supplies AC Power Supply

or

DC Power Supply

Fans Fan Module I/O Panel to PSU Airflow

or

Fan Module PSU to I/O Panel Airflow

Validated operating systems Dell EMC Networking OS10

Dell EMC Networking S4048-ON optional switch

Table 12: S4048-ON switch

Product Description

S4048-ON 48x 10GbE SFP+, 6x QSFP+ (1 qty) - optional


Bill of materials | 37

Product Description


or

DC Power Supply


or



Dell EMC Networking S5232F-ON switch

Table 13: S5232F-ON

Product Description

S5232F-ON 100GbE, 40GbE, and 25GbE (2 qty)


or

DC Power Supply


or




38 | References

Appendix

CReferences

Topics:

• To learn more

Additional information found at https://www.dell.com/support/article/us/en/19/sln310368/dell-emc-ready-architecture-for-red-hat-openstack-platform?lang=en or by e-mailing [email protected].

For more on Red Hat OpenStack Platform 13 and DCN, please see:

• Red Hat OpenStack Platform 13• Deploying Distributed Compute Nodes to Edge Sites• Routed L3 Networks (Spine Leaf Networking)• Scaling the Overcloud

Note: If you need additional services or implementation help,please contact your Dell EMC sales representative.


https://www.dell.com/support/article/us/en/19/sln310368/dell-emc-ready-architecture-for-red-hat-openstack-platform?lang=en



mailto:[email protected]

https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/

https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html/deploying_distributed_compute_nodes_to_edge_sites/deploying_distributed_compute_nodes_to_edge_sites

https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html-single/spine_leaf_networking


References | 39

To learn moreFor more information on the Dell EMC Network Edge Reference Architecture for Red Hat OpenStackPlatform, visit https://www.dell.com/support/article/us/en/19/sln310368/dell-emc-ready-architecture-for-red-hat-openstack-platform?lang=en.

For more information on Dell EMC Service Provider Solutions, visit https://www.dellemc.com/en-us/service-providers/index.htm




https://www.dellemc.com/en-us/service-providers/index.htm

https://www.dellemc.com/en-us/service-providers/index.htm

40 | Glossary

Glossary

APIApplication Programing Interface is a specification that defines how software components can interact.

BMC/iDRAC EnterpriseBaseboard management controller. An on-board microcontroller that monitors the system for critical eventsby communicating with various sensors on the system board, and sends alerts and log events when certainparameters exceed their preset thresholds.

BOSSThe Boot Optimized Storage Solution (BOSS) enables customers to segregate operating system anddata on server-internal storage. This is helpful in the Hyper-Converged Infrastructure (HCI) and SoftwareDefined Storage (SDS) arenas, to separate operating system drives from data drives, and implementhardware RAID mirroring (RAID1) for OS drives.

CDHCloudera Distribution for Apache Hadoop

Cloud computingSee http://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-145.pdf

Cloud computing is a model for enabling ubiquitous, convenient, on-demand network access to a sharedpool of configurable computing resources (e.g., networks, servers, storage, applications, and services) thatcan be rapidly provisioned and released with minimal management effort or service provider interaction.

ClusterA set of servers dedicated to OpenStack that can be attached to multiple distribution switches.

Control PlaneThe cloud control plane provides the central management system for OpenStack deployments. Cloudcontrollers are mission critical and should be clustered for high-availability. The cloud controller managesauthentication and the message bus for all service intercommunication.

The control plane manages the following services:

• Databases providing authentication and persistance for OpenStack services• Message queue services• Conductor services• Proxy requests to a database• Authentication and authorization• Image-management services


http://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-145.pdf

Glossary | 41

• Scheduling services• The OpenStack dashboard (Horizon)• RESTful API endpoints that OpenStack services expose for consumption

Compute nodeThe hardware configuration that best supports the hypervisor server or Nova compute roles.

DCNRed Hat's edge computing offering based on OpenStack.

DevOpsDevelopment Operations (DevOps) is an operational model for managing data centers using improvedautomated deployments, shortened lead times between fixes, and faster mean time to recovery. Seehttps://en.wikipedia.org/wiki/DevOps.

DIMMDual In-line Memory Module

DNSThe domain name system (DNS) defines how Internet domain names are located, and translated intoInternet Protocol (IP) addresses.

Edge ComputingA distributed computing model that brings computing power closer to the location where it is needed, toimprove network latency and reduce data plane backhaul.

FQDDA fully qualified device descriptor (FQDD) is a method used to describe a particular component within asystem or subsystem, and is used for system management and other purposes.

FQDNA fully qualified domain name (FQDN) is the portion of an Internet Uniform Resource Locator (URL) thatfully identifies the server to which an Internet request is addressed. The FQDN includes the second-leveldomain name, such as "dell.com", and any other levels as required.

GUIGraphical User Interface - A visual interface for human interaction with the software, taking inputs andgenerating easy to understand visual outputs.


https://en.wikipedia.org/wiki/DevOps

https://en.wikipedia.org/wiki/DevOps

42 | Glossary

HypervisorSoftware that runs virtual machines (VMs).

IaaSInfrastructure as a Service.

Infrastructure nodeSystems that handle the control plane and deployment functions.

ISVIndependent Software Vendor.

JBODJust a Bunch of Disks

LAGLink Aggregation Group.

LOMLAN on motherboard.

LVMLogical Volume Management

ML2The Modular Layer 2 plug-in is a framework that allows OpenStack to utilize different layer 2 networkingtechnologies.

NFSThe Network File System (NFS) is a distributed filesystem that allows a computer user to access,manipulate, and store files on a remote computer, as though they resided on a local file directory.

NICNetwork Interface Card


Glossary | 43

NodeOne of the servers in the cluster.

NUMANon-Uniform Memory Access

OvercloudThe functional cloud that is available to run guest VMs and workloads.

PodAn installation comprised of three racks, and consisting of servers, storage, and networking.

RANRAN - Radio Access Network. Part of a mobile telecommunication system that implements radio accesstechnology.

RESTREST - Representational State Transfer (also ReST). Relies upon stateless, client-server, cacheablecommunications protocol to access the API.

RHOSPRed Hat OpenStack Platform

RPCRemote Procedure Call

SAHThe Solution Admin Host (SAH) is a physical server that supports VMs for the Undercloud machinesneeded for the cluster to be deployed and operated.

SDSSoftware-defined storage (SDS) is an approach to computer data storage in which software is used tomanage policy-based provisioning and management of data storage, independent of the underlyinghardware.


44 | Glossary

SDNSoftware-defined Network (SDN) is where the software will define, create, use and destroy differentnetworks as needed.

StampA stamp is the collection of all servers and network switches in the solution.

Storage NodeThe hardware configuration that best supports SDS functions such as Red Hat Ceph Storage.

ToRTop-of-rack switch/router.

UU used in the definition of the size of server, example 1U or 2U. A "U" is a unit of measure equal to 1.75inches in height.

UndercloudThe Undercloud is the system used to control, deploy, and monitor the Overcloud - it is a single nodeOpenStack deployment completely under the administrators control. The Undercloud is not HA configured.

VLTA Virtual Link Trunk (VLT) is the combined port channel between an attached device (ToR switch) and theVLT peer switches.

VLTiA Virtual Link Trunk Interconnect (VLTi) is an interconnect used to synchronize states between theVLT peer switches. Both endpoints must be the same speed, i.e. 40Gb → 40Gb; 1G interfaces are notsupported.

VMVirtual Machine - a simulation of a computer system.

vRANVirtual Radio Access Network - A Radio Access Network (RAN) implemented as software vs a solelyhardware-based solution.


for red hat openstack platform 13 dell emc network edge … · dell emc network edge reference...

Documents