openstack networking internals - first part
DESCRIPTION
Openstack Networking Internals - first part Description of the Virtual Network Infrastructure inside an OpenStack cluster The pictures of the VNI were taken with the "Show my network state" tool https://sites.google.com/site/showmynetworkstate/TRANSCRIPT
![Page 1: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/1.jpg)
Giuliano Santandrea – CIRI ICT University of Bologna
![Page 2: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/2.jpg)
● OpenStack description● Openstack components and allocation● Neutron abstractions● The virtual network infrastructure
![Page 3: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/3.jpg)
vif: virtual network interface VR: virtual router VNI: virtual network infrastructure OVS: Open vSwitch© virtual bridge LB: Linux bridge
![Page 4: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/4.jpg)
![Page 5: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/5.jpg)
● OpenStack is a cloud platform management software● A cloud platform is a cluster of machines that host
some servers (instances): the servers are offered to the users as as a “service”. The user is able to create a “virtual infrastructure” composed of servers and network appliances (firewalls, routers, ...)
● The servers can be implemented as:– VM (KVM,VMWare,..)– light container (LXC-Docker,..)– bare metal (PXE boot,…)
![Page 6: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/6.jpg)
OS is composed of the following components:
● Web dashboard (Horizon)
● Compute (Nova): it manages instances lifecycle
● Keystone: credentials, service catalog of all the OS services (list of REST service endpoints)
● Glance: image management. An image is a blob file containing a file system with a “pre-cooked” VM: it can be used by hypervisors to boot a new instance!
● Networking(Neutron): network management
● Block storage (Cinder): persistent storage (volumes)
● Other services: – Object storage (Swift): distributed storage for non-structured data
![Page 7: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/7.jpg)
External net
Management net
Data net (flat)
CPU node 1 CPU node 2 CPU node 3Controller Network node
Internet
![Page 8: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/8.jpg)
● Data net● Mgmt net● External/API net
These networks are implemented as physical separated networks.
Cesena cluster: I configured a switch with “port-based, access mode” VLANs
![Page 9: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/9.jpg)
External net
Management net
Data net (flat)
CPU node 1 CPU node 2 CPU node 3Controller Network node
Internet
Allows the admin to access the cluster nodes and it is used for inter-service communicationEVERY NODE IS ON THIS NET
![Page 10: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/10.jpg)
External net
Management net
Data net (flat)
CPU node 1 CPU node 2 CPU node 3Controller Network node
Internet
Used for inter-VM communication.Depending on the chosen network virtualization mechanism, the packets will beVLAN tagged packets, or encapsulated packets (VXLAN, GRE)
![Page 11: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/11.jpg)
External net
Management net
Data net (flat)
CPU node 1 CPU node 2 CPU node 3Controller Network node
Internet
It allows the VMs to access the Internet, the user to access the VMs
![Page 12: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/12.jpg)
![Page 13: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/13.jpg)
External net
Management net
Data net (flat)
CPU node 1 CPU node 2 CPU node 3Controller Network node
Internet
●Keystone●Nova
●API: REST endpoint, receives user requests●Scheduler: chooses a compute node
●Glance:●API●Registry
●Neutron●Server: REST API endpoint ●plugin: implements the VNI
●Cinder:●API,…
●message queue: middleware for inter-service communication
![Page 14: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/14.jpg)
External net
Management net
Data net (flat)
CPU node 1 CPU node 2 CPU node 3Controller Network node
Internet
Neutron• plugin: implements the VNI• L3: virtual routers creation• dhcp• metadata
![Page 15: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/15.jpg)
External net
Management net
Data net (flat)
CPU node 1 CPU node 2 CPU node 3Controller Network node
Internet
In each Compute node:●Neutron
●plugin: implements VNI●Nova
●compute: manages the hypervisor
![Page 16: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/16.jpg)
● User send REST API calls to the service enpoints, using:– the web dashboard– CLI clients
● OS components communicate between each other using:– Message passing (an AMQP server resides in the
controller node)– rest API calls
● Some components (neutron-server, keystone, etc.) access directly to a DB to save/modify their state
![Page 17: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/17.jpg)
Sometimes the external network is notdirectly connected the internet, but there is a datacenter gateway allowing the access to the internet
Public net
gatewayREST APIInternet
Private cloud
External net
![Page 18: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/18.jpg)
Public net
gateway
External net
Mgmt net
data net
CPU node 1 CPU node 2 CPU node 3Controller
nova api
Network node
REST API
User
port forwarding
Internet
REST protocol
![Page 19: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/19.jpg)
Public net
gateway
External net
CPU node 1 CPU node 2 CPU node 3Controller Network node
REST API
User
port forwarding
Internet
Horizon(Web server
apache2)
HTTP request
Mgmt net
data net
![Page 20: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/20.jpg)
![Page 21: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/21.jpg)
Public net
gateway
External net
rete dati (flat)
CPU node 1 CPU node 2 CPU node 3Controller Network node
REST API
User
port forwarding
Internet
xvncproxy
VNC protocol
Port forwarding on 6080 port!!
The hypervisor pipes the VM video output through the network
hypervisor
sudo iptables -t nat -I PREROUTING -i eth1 -p tcp --dport 80 -j DNAT --to-destination 10.250.0.1:80
VM
Mgmt net
![Page 22: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/22.jpg)
![Page 23: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/23.jpg)
Public net
gateway
External net
CPU node 1 CPU node 2 CPU node 3Controller Network node
REST APIInternet
novacompute
nova api
messagequeue server
Mgmt net
data net
![Page 24: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/24.jpg)
Public net
gateway
External net
CPU node 1 CPU node 2 CPU node 3Controller Network node
REST APIInternet
novacompute
Glance API REST endpoint
REST API call
Mgmt net
data net
![Page 25: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/25.jpg)
Public net
gateway
External net
CPU node 1 CPU node 2 CPU node 3Controller Network node
REST APIInternet
Mgmt net
data net
VM VM
![Page 26: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/26.jpg)
Public net
gateway
External net
CPU node 1 CPU node 2 CPU node 3Controller Network node
REST APIInternet
Mgmt net
data net
VM VMVirtual router
![Page 27: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/27.jpg)
Public net
gateway
External net
CPU node 1 CPU node 2 CPU node 3Controller Network node
REST APIInternet
Mgmt net
data net
VMVirtual router
NAT/port forwarding
User
![Page 28: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/28.jpg)
Public net
gateway
External net
CPU node 1 CPU node 2 CPU node 3Controller Network node
REST APIInternet
Mgmt net
data net
VMVirtual router
(NAT/Floating IP)
NAT/port forwarding
User
The VM has:• a fixed private IP on a
private tenant net• an optional floating IP on
the external network
The VR does:• The NAT for
private IPs• DNAT of the
floating IPs
![Page 29: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/29.jpg)
Public net
gateway
External net
CPU node 1 CPU node 2 CPU node 3Controller Network node
REST APIInternet
Mgmt net
data net
hypervisorVM
storage
The VM sees the disk (block device) as a local device!
Network file system protocol (NFS, iSCSI, …)
Storage node
![Page 30: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/30.jpg)
![Page 31: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/31.jpg)
Neutron defines these network abstractions: Network – an isolated L2 network segment Subnet – an IP address block on a certain network Router – a gateway between subnets Fixed IP – an IP on a tenant network Floating IP – a mapping between IP of external
networks and a private fixed IP Port – an attachment point to a network.
Users only see these abstractions!!!
Neutron implements these abstraction inside the VNI in the cluster nodes (i.e. at the host level) so that VMs(guest level) can see the virtual networks
![Page 32: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/32.jpg)
Tenant network: a network created by a clouduser. The VM takes a fixed IP from this net (notmodifiable after the VM creation)◦ implementation detail: the VM receives the IP from a
DHCP server configured to give always the same fixed IP to that VM!
Provider network: network external to the cluster, allows outside connectivity, passing through the network node. A VM can allocate a floating IP to gain external visibility (OpenStack maps eachfloating IP to the related fixed IP). Floating Ips are deallocable.
![Page 33: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/33.jpg)
VIRTUAL NETWORKManaged by Neutron
Physicalinterface of the network node
![Page 34: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/34.jpg)
They are “leaky abstractions”! For example:◦ Net creation is limited to the effective VLAN ID
availability for the physical network!◦ During the network creation you could specify low
level implementation details ( such as the VLAN ID) or leave OpenStack decide them for you
![Page 35: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/35.jpg)
server (REST API endpoint): receives API requests, saves allthe network info in a database, instructs the agents
plugin agent: implements the VNI inside the cluster node,using the technology specified (GRE tunnel, VLAN, VXLAN, …).
dhcp: implements the dhcp servers L3: implements the virtual routers Metadata: the VMs contact the metadata service at the
creation
![Page 36: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/36.jpg)
![Page 37: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/37.jpg)
● Create a L2 network– neutron net-create net1
● Associate a L3 subnet to the network– neutron subnet-create net1 10.0.0.0/24 --name subnet1
● Boot a VM on that subnet– nova boot --image img --flavor flv --nic net-id=uuid vm_name
![Page 38: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/38.jpg)
A user that wants to create a VM:
1. Sends a REST API (via CLI client or web dashboartd) to the Keystone REST endpoint (request+auth) and Nova endpoint
2. “nova-scheduler”, internally, chooses the best suitable compute (CPU) node that will host the VM
3. In that CPU node, the “nova-compute” component does the following things:
1. prepares the hypervisor
2. asks Glance the VM image
3. asks Neutron components to allocate the VNI
4. asks Cinder to allocate the persistent block storage (volumes) for the VM
![Page 39: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/39.jpg)
source: http://goo.gl/n3Bb5s
![Page 40: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/40.jpg)
Network namespaces is a technology that allows to separate/isolate multiple network domains inside a single host by replicating the network software stack
A process executed in a namespace sees only specific:◦ physical/virtual network interfaces◦ routing/arp tables◦ firewall/NAT rules
You can: ◦ Create a netns◦ Create a process/virtual network component inside that netns
![Page 41: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/41.jpg)
Networknamespace 1
Linux kernelglobal namespace(PID 1)
Hardware
Network namespace N
Network namespace 2
...
A global namespace createdat boot time. Processes
usually reside here
You can:• Create a netns• create a vif inside the netns• start a linux process
VIF
processprocess
process
process
![Page 42: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/42.jpg)
Networknamespace 1
Linux kernelglobal namespace(PID 1)
Hardware
Network namespace N
Network namespace 2
...
Namespaces guarantee L3 isolation, so the interfaces can have overlapping IP addresses!
VIFVIF
![Page 43: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/43.jpg)
The virtual bridges can be connected to (physical or virtual) interfaces thatreside in different namespaces: the virtual bridges act as bridges betweenthe namespaces
Networknamespace 1
Network namespace 2
VIF packet
PIF
Virtual bridge
process
Physical host
![Page 44: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/44.jpg)
Physical host
process process
process process
They are completelyisolated!
![Page 45: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/45.jpg)
![Page 46: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/46.jpg)
In each node there is:◦ An integration bridge◦ A bridge for each physical network, connected to
The integration bridge The physical network interface (pif)
![Page 47: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/47.jpg)
public net
gatewayExternal net
Mgmt net
Data net
CPU node 1Controller Network node
br-data
br-intlinuxbridge
VM
br-data
br-int
br-ex
br-data
br-int
br-data: connected to the data net
Internet
![Page 48: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/48.jpg)
gateway
CPU node 1Controller Network node
br-data
br-intlinuxbridge
VM
br-data
br-int
br-ex
br-data
br-int
br-int: intermediate bridge, act as a hub of a
star network
External net
Mgmt net
Data net
public net
Internet
![Page 49: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/49.jpg)
gateway
CPU node 1Controller Network node
br-data
br-intlinuxbridge
VM
br-data
br-int
br-ex
br-data
br-int
br-ex: connected to the external network (only
present only in the network node!!)
External net
Mgmt net
Data net
public net
Internet
![Page 50: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/50.jpg)
Advantages: namespaces allow to manage multiple L3 functions at the host level in the same node.
This is a key enabler is implementing VNIs with advancedfunctionalities: the cloud users to create overlapping virtualL3 networks!!!◦ Two tenants can create isolated L3 networks with the same IP addresses
The namespaces are used only in the network node (because L3 functionalities at the host level are present only inside the network node)
A new network namespace is created by OpenStack when you create…◦ … a new virtual L2 network◦ … a new virtual router
![Page 51: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/51.jpg)
In the compute node:◦ nova-compute makes a REST API call to neutron-
server asking for a port allocation (and a fixed IP for the VM)
◦ neutron-agent configures the virtual bridges (br-data, br-int) via OpenFlow protocol
◦ nova-compute boots the VM
![Page 52: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/52.jpg)
In the network node:◦ neutron-dhcp
creates a netns ("qdhcp-…") creates a vif inside that netns spawns a dnsmasq process (DHCP server) using that vif
◦ neutron-l3 creates a netns for the virtual router ("qrouter-…") creates a vif inside that netns configure the routing tables inside that netns
![Page 53: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/53.jpg)
In the configuration files Neutron associateseach “physical network” to a virtual bridge. For example:◦ Physnet1: br-data◦ Extphysnet: br-ex
The admin, during the creation of a provider network, must specify the associated physicalnetwork
![Page 54: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/54.jpg)
Use case: a user◦ creates a private user network( 10.0.0.0/24)◦ boots a VM on this network
source ~/devstack/openrc demo demo pass && nova boot --key-name hc01 --image cirros-0.3.1-x86_64-disk --flavor 2 --nic net-id=61821a27-69b8-43c2-afa8-633304d8be50,v4-fixed-ip=10.0.0.66 myserver
![Page 55: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/55.jpg)
The admin can even specify the VLAN Id used on the physical data network (even outside the VLAN ID pool specified in the configurationfile)
The user cloud is not allowed to do this (OpenStack autonomously picksa VLAN from the VLAN pool available in the data network)
source ~/devstack/openrc admin admin pass && neutron net-create net2 --provider:network_typevlan --provider:physical_network physnet1 --provider:segmentation_id 1000
+---------------------------+--------------------------------------+| Field | Value |+---------------------------+--------------------------------------+| admin_state_up | True || id | 61821a27-69b8-43c2-afa8-633304d8be50 || name | private || provider:network_type | vlan || provider:physical_network | physnet1 || provider:segmentation_id | 1000 || router:external | False || shared | False || status | ACTIVE || subnets | bbc97757-f297-4c7b-b032-e70768fe8485 || tenant_id | a370af83e43a432abb3adfbf976d1cf8 |+---------------------------+--------------------------------------+
NB this VLAN is that used on the physical network!
![Page 56: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/56.jpg)
PIF on the data network
Veth pair: a pair of vif that act as a pipe (everythingentering from oneexits from the other)
![Page 57: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/57.jpg)
VMeth0
LB managment interface
Tap interface (host-level view of the VM interface)
Veth pair che collega br-int al linux bridge
Veth pair interfacesconnecting the bridges
![Page 58: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/58.jpg)
![Page 59: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/59.jpg)
specificroutingtables
dhcp server
dhcp server
Network namespaces
No traffic here
![Page 60: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/60.jpg)
VMeth0 eth0
![Page 61: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/61.jpg)
User network (internal)The VMs will receive a fixed IP on thisnet.
Provider network (external)NB it’s the physical external network of the cluster
![Page 62: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/62.jpg)
![Page 63: Openstack Networking Internals - first part](https://reader035.vdocument.in/reader035/viewer/2022062313/559c583b1a28ab07588b4638/html5/thumbnails/63.jpg)