scaling neutron networks - deutsche openstack tage · scaling neutron beyond the limits of vlan...

Scaling NeutronIn Heterogeneous Environments

Martin Klein, SAP

June 21, 2016

© 2016 SAP SE or an SAP affiliate company. All rights reserved. 2

Agenda

Where do we come from

Overview of the SAP Monsoon Converged Cloud

Where do we want to go

Transitioning a private IAAS Platform to OpenStack

How do we want to get there

Architecting OpenStack Neutron to enable growth


SAP Monsoon Converged Cloud

SAP Internal IAAS/PAAS Platform

Currently running In-House SW Stack with some OpenStack Components. Offering

Custom API and Amazon EC2 compatible API’s.

IAAS

Global footprint with currently 6 Regions on 4 Continents

Provide a unified global platform for SAP’s Cloud offerings

Offers Compute, Block Storage and limited Networking

PAAS

Focus on automation and Continuous Delivery using OpsCode Chef

Optional for customers which don’t bring their own PAAS service


SAP Monsoon Converged Cloud Current Scale

Platform is offered in 6 Regions extending to 15 In 2016

Absolute Size

CPU 17.000 Cores

Memory 600 TB

Storage 6.2 PB

Instances 27.000

Volumes 58.000

Operations

Instance Operations 3.000/Day

Instance Growth 250/Day


SAP Monsoon Converged Cloud OpenStack Transition

Replace all infrastructure controllers with OpenStack implementation

Existing Implementations

Replace with standard OpenStack implementations

Transition custom services to an OpenStack like schema

Introduce a thin layer on top of Keystone to to reflect special requirements

New Features

Swift - Object Storage

Neutron – Networking including L2/L3, LBaaS, VPNaaS and FWaaS

Designate - DNSaaS

Manila - NFSaaS

Monasca - Monitoring as a Service


SAP Monsoon Converged Cloud Key Challenges

Running OpenStack on an Enterprise Hardware Stack

OpenStack Challenges

Running Nova in a Multi Hypervisor Environment

Scaling Neutron beyond the limits of VLAN

Infrastructure Challenges

Scaling a network Fabric beyond 4k L2 Networks

Attaching arbitrary devices to the Fabric without additional requirements on connected

devices

Finding a universally available overlay protocol

Scaling in OpenStack


OpenStack Scaling I

Neutron

Nova

Cinder

Manila

No

Ne Ci

Ma

No

NoNe

Ma


OpenStack Scaling III

Neutron

Nova

Cinder

Manila

No

Ne Ci

Ma

No

NoNe

Ma

No No

NoNo

Ne

NeCi

Ci

Ma

No

Ma

No

Neutron – L2

• OpenStack Services are designed to scale out

• Exception: Neutron L2 needs to scale up

Neutron ML2 Network Model


Neutron ML2 Network Model

Network Type

Defines Network Protocol

Is type ID of Neutron networks

Mechanism Driver

Is responsible for implementing

the network types on a specific

Device

Is responsible for binding the

ports

Core Plugin

Type Manager Mechanism Manager

Linux

Bridge

Type Driver Mechanism Driver

VendorLinux

OVSGRE VxLANVLAN


Neutron ML2 VLAN Networks

Neutron VLAN Networks

Network Level

All VLAN ID’s available

everywhere

All Host ports in trunk mode

Host Level

Trunk port as uplink for OVS

OVS Handles tag for vnic

VM Level

Interface in access mode

VLAN ID 1-4096

VLAN ID 1-4096 VLAN ID 1-4096

VM

vnic0

OVS

eth0

Trunk Trunk

Trunk

Trunk

Access

VM

vnic0

OVS

eth0

Trunk

Trunk

Access

Network Level

Host Level

VM Level


Neutron ML2 VxLAN Networks

Neutron VxLAN Networks

Network Level

Provides L3 Underlay Network

All Host have access

Host Level

Has access to undelay

OVS Handles Encap-Decap

VM Level


VLAN ID 10

VLAN ID 10 VLAN ID 10

VM

vnic0

OVS

eth0

Trunk Trunk

Access

Encap

Access

VM

vnic0

OVS

eth0

Access

Encap

Access

Network Level

Host Level

VM Level


Neutron VxLAN vs VLAN Type Networks

VLAN Type Networks

PRO: Available Everywhere

PRO: Low overhead on the Host

CON: Very limited scale (4K theoretically, 2k realistically)

CON: Entire Fabric needs to be aware of all available VLAN’s

VxLAN Type Networks

PRO: Large ID Space (16M theoretically, 20K realistically with Flood and Learn )

PRO: Fabric needs only be L3 and not aware of VxLAN

CON: Not available on every OS / Device (Only Flood and Learn commonly available)

CON: High CPU Overhead for Encap-Decap

CON: Large number of VTEP if termination on every device.


OpenStack Scaling IIISo what’s the problem again

No Common Overlay protocol available in the Enterprise environment

No

Ne Ci

Ma

No

NoNe

Ma

No No

NoNo

Ne

NeCi

Ci

Ma

No

Ma

No

Neutron – L2

Neutron Hierarchical Port Binding


Neutron Flat Port Binding

Neutron Network

Network Details (Global)

network_type: vxlan

segmentation_id: 14410

Network requirements

All devices use one protocol

All network layers are protocol

aware

Core Network

Edge Switch Edge Switch

HVStorage

Device

Network

DeviceHV

VM VM

Port Port

VNI:14410 VNI:14410

VNI:14411 VNI:14411Port Port


Neutron Hierarchical Port Binding

Neutron Network

Network Details (Global)

network_type: vxlan

segmentation_id: 14410/11

Network Local (local)

network_type: vlan

segmentation_id: local

Network requirements

Core/Edge devices share

protocol

Connected devices are not

overlay aware

HVStorage

Device

Network

DeviceHV

VM VM

Port Port

VLAN: 40 VLAN:10

VLAN:10 VLAN:15

Core Network

Edge SwitchVNI:14410-14411

VLAN:15 VLAN:10Edge Switch

VNI:14410-14411

VLAN: 40 VLAN:10

Port Port


Neutron Hierarchical Port Binding II

Neutron HPB Networks

Network Core Level

Provides Underlay for VXLAN

Can be routed networks

Network Edge Level


Translation local VLAN VxLAN

Host Level



VM Level


VxLAN ID 1-16M

Local VLAN ID 1-4096 Local VLAN ID 1-4096

VM

vnic0

OVS

eth0

L3 L3

Trunk

Trunk

Access

VM

vnic0

OVS

eth0

Trunk

Trunk

Access

Network Level

Host Level

VM Level


Neutron Hierarchical Port Binding IVOpenStack Configuration

Without Hierarchical Port Binding

With Hierarchical Port Binding


Neutron Hierarchical Port Binding IVOpenStack Binding Levels

> net-show 4f904b00-9705-4098-85ff-023eb57ec29e….....

| segments | {“network_type": "vxlan", “physical_network": null, “segmentation_id": 10059} |

| | {“network_type": "vlan", “physical_network": "bb98-fi", “segmentation_id": 3927}|

Each Network has multiple segments

neutron=# select * from ml2_port_binding_levels

port_id | host | level | driver | segment_id

-------------+---------------------------+-------+-------------+-------------------------

55f29793-... | l3-asr1.labx.mo.sap.corp | 0 | aci | d27f202d-...

55f29793-... | l3-asr1.labx.mo.sap.corp | 1 | openvswitch | de7b0947-...

Each Port can be bound against mutilple segments,

dependant on the binding host

Neutron HPB Ports

Network Edge Level

Performs Partial Binding

Creates Lower Level segment

Assigns lower segment ID

Host Level

OVS Driver completes Binding



Scaling NeutronSummary

VLAN Type Networks

• Very efficient on the Host Level

• Very limited Scale

VxLAN Type Networks

Very large overhead on the Host Level

Not commonly available

Provide large ID space

HPB Networks

Provide overlay type scale with VLAN type host complexity

Encapsulation done on network edge in wire speed

scaling neutron networks - deutsche openstack tage · scaling neutron beyond the limits of vlan...

Documents