network node is not needed anymore - completed distributed virtual router / fujitsu part
TRANSCRIPT
Copyright 2015 FUJITSU LIMITED1
Network Node is Not Needed Anymore- Completed Distributed Virtual Router
OpenStack Summit October 2015 TokyoTuesday, October 27 - 2:50pm – 3:30pm
Takanori MiyagishiFujitsu LimitedE-mail address: [email protected] name: miygishi_t
Copyright 2015 FUJITSU LIMITED4
Vision Deployment of enterprise systems on OpenStack
To achieve this from the network perspective, we need more scalability and availability. Current Neutron has the following issues. Performance bottle neck Single point of failure (SPOF)
Copyright 2015 FUJITSU LIMITED6
Legacy Network Architecture (Legacy Router)
Neutron centralizes networking services to Network Node Most of traffics need to go through Network Node
Compute Node Network Node
ML2 pluginOVS-agent
DHCP-agent
Controller Node
Neutron Server
L3-agent
ML2 pluginOVS-agentOVS-agent
metadata-agent
: Management network: External network: Data network
Copyright 2015 FUJITSU LIMITED7
Legacy Network Architecture (Legacy Router)
Compute Node Network Node
ML2 pluginOVS-agent
DHCP-agent
Controller Node
Neutron Server
L3-agent
ML2 pluginOVS-agentOVS-agent
metadata-agent
: Management network: External network: Data network
Neutron centralizes networking services to Network Node Most of traffics need to go through Network Node
What happens due to this Architecture?
Copyright 2015 FUJITSU LIMITED8
Issue #1: Network Node is Bottle neck There are 6 Network Traffics
East-West Traffic• Between VMs
- Same Subnet -- (1)
- Different Subnets -- (2)• VM accessing to DHCP server -- (3)• VM getting metadata from vRouter -- (4)
North-South Traffic• Between VM and out of OpenStack
- Using Floating IP -- (5)
- Using SNAT -- (6)
Compute node #1 Network NodeCompute node #2
br-tun
br-int
br-tun
br-int
br-ex
VM VM VM
: External Network: Data Network
DHCP
br-tun
br-int
vRouter
Copyright 2015 FUJITSU LIMITED9
Issue #1: Network Node is Bottle neck There are 6 Network Traffics
East-West Traffic• Between VMs
- Same Subnet -- (1)
- Different Subnets -- (2)• VM accessing to DHCP server -- (3)• VM getting metadata from vRouter -- (4)
North-South Traffic• Between VM and out of OpenStack
- Using Floating IP -- (5)
- Using SNAT -- (6) Not need to go through Network Node in theboth cases:- VMs are on the same host- VMs are on different hosts
Compute node #1 Network NodeCompute node #2
br-tun br-tun
br-int
br-ex
VM VM VM
: External Network: Data Network
DHCP
br-tun
br-int
vRouter
br-int
Copyright 2015 FUJITSU LIMITED10
Issue #1: Network Node is Bottle neck
Compute node #1 Network NodeCompute node #2
br-ex
VM VM VM
: External Network: Data Network
DHCP
br-tun
br-int
vRouter
br-tun
br-int
br-tun
br-int
There are 6 Network Traffics East-West Traffic• Between VMs
- Same Subnet -- (1)
- Different Subnets -- (2)• VM accessing to DHCP server -- (3)• VM getting metadata from vRouter -- (4)
North-South Traffic• Between VM and out of OpenStack
- Using Floating IP -- (5)
- Using SNAT -- (6) Not need to go through Network Node in theboth cases:- VMs are on the same host- VMs are on different hosts
Copyright 2015 FUJITSU LIMITED11
Issue #1: Network Node is Bottle neck
Compute node #1 Network NodeCompute node #2
br-ex
VM VM VM
: External Network: Data Network
DHCP
vRouter
br-tun
br-int
br-tun
br-int
br-tun
br-int
There are 6 Network Traffics East-West Traffic• Between VMs
- Same Subnet -- (1)
- Different Subnets -- (2)• VM accessing to DHCP server -- (3)• VM getting metadata from vRouter -- (4)
North-South Traffic• Between VM and out of OpenStack
- Using Floating IP -- (5)
- Using SNAT -- (6) Need to go through Network Node in theboth cases:- VMs are on the same host- VMs are on different hosts
Copyright 2015 FUJITSU LIMITED12
Issue #1: Network Node is Bottle neck
Need to go through Network Node in theboth cases:- VMs are on the same host- VMs are on different hosts
Compute node #1 Network NodeCompute node #2
br-ex
VM VM VM
: External Network: Data Network
DHCP
vRouter
br-tun
br-int
br-tun
br-int
br-tun
br-int
There are 6 Network Traffics East-West Traffic• Between VMs
- Same Subnet -- (1)
- Different Subnets -- (2)• VM accessing to DHCP server -- (3)• VM getting metadata from vRouter -- (4)
North-South Traffic• Between VM and out of OpenStack
- Using Floating IP -- (5)
- Using SNAT -- (6)
Copyright 2015 FUJITSU LIMITED13
Issue #1: Network Node is Bottle neck
Compute node #1 Network NodeCompute node #2
br-ex
VM VM VM
: External Network: Data Network
DHCP
vRouter
br-tun
br-int
br-tun
br-int
br-tun
br-int
There are 6 Network Traffics East-West Traffic• Between VMs
- Same Subnet -- (1)
- Different Subnets -- (2)• VM accessing to DHCP server -- (3)• VM getting metadata from vRouter -- (4)
North-South Traffic• Between VM and out of OpenStack
- Using Floating IP -- (5)
- Using SNAT -- (6)
Copyright 2015 FUJITSU LIMITED14
Issue #1: Network Node is Bottle neck
Compute node #1 Network NodeCompute node #2
br-tun
br-int
br-tun
br-int
br-ex
VM VM VM
: External Network: Data Network
DHCP
br-tun
br-int
vRouter
There are 6 Network Traffics East-West Traffic• Between VMs
- Same Subnet -- (1)
- Different Subnets -- (2)• VM accessing to DHCP server -- (3)• VM getting metadata from vRouter -- (4)
North-South Traffic• Between VM and out of OpenStack
- Using Floating IP -- (5)
- Using SNAT -- (6)
Copyright 2015 FUJITSU LIMITED15
Issue #1: Network Node is Bottle neck
Compute node #1 Network NodeCompute node #2
br-tun
br-int
br-tun
br-int
br-ex
VM VM VM
: External Network: Data Network
DHCP
br-tun
br-int
vRouter
There are 6 Network Traffics East-West Traffic• Between VMs
- Same Subnet -- (1)
- Different Subnets -- (2)• VM accessing to DHCP server -- (3)• VM getting metadata from vRouter -- (4)
North-South Traffic• Between VM and out of OpenStack
- Using Floating IP -- (5)
- Using SNAT -- (6)
Copyright 2015 FUJITSU LIMITED16
Issue #1: Network Node is Bottle neck Five out of six traffic patterns need to go through Network Node
Compute node #1 Network NodeCompute node #2
br-tun
br-int
br-tun
br-int
br-ex
VM VM VM DHCP
br-tun
br-int
vRouter
Need through Network Node?
East-West
(1) Between VMs (same subnet) No(2) Between VMs (different subnet) Yes(3) VM accessing to DHCP Yes(4) VM getting metadata Yes
North-South
(5) Using Floating IP Yes(6) Using SNAT Yes
(1)
(1)
(2),(3),(4),(5),(6)
Bottle neck
Copyright 2015 FUJITSU LIMITED17
Issue #2: Network Node is SPOF If Network Node fails, that affects many traffics
Compute node #1 Network NodeCompute node #2
br-ex
VM VM VM DHCP
br-tun
br-int
vRouter
Affected Trafficsby Network Node Failure
East-West
(1) Between VMs (same subnet) No(2) Between VMs (different subnet) Yes(3) VM accessing to DHCP Yes(4) VM getting metadata Yes
North-South
(5) Using Floating IP Yes(6) Using SNAT Yes
br-tun
br-int
br-tun
br-int
Copyright 2015 FUJITSU LIMITED19
DVR Architecture L3-agent and Metadata-agent can be distributed on each Compute
Node But a part of L3-agent (SNAT) and DHCP-agent still need to
run on Network Node
Compute Node Network Node
ML2 pluginOVS-agent
DHCP-agent
Controller Node
Neutron Server
L3-agent(SNAT)
ML2 pluginOVS-agentOVS-agentL3-agent
metadata-agent Metadata-agent
Copyright 2015 FUJITSU LIMITED20
DVR Architecture
Compute Node Network Node
ML2 pluginOVS-agent
DHCP-agent
Controller Node
Neutron Server
L3-agent(SNAT)
ML2 pluginOVS-agentOVS-agentL3-agent
metadata-agent Metadata-agent
L3-agent and Metadata-agent can be distributed on each Compute Node But a part of L3-agent (SNAT) and DHCP-agent still need to
run on Network NodeWhat are benefits of the distribution of agents?- Will explain, comparing with Legacy Router
21
Traffics with Legacy Router again five of six types of traffics needs through Network Node
Copyright 2015 FUJITSU LIMITED
Need through Network Node?
East-West
(1) Between VMs (same subnet) No(2) Between VMs (different subnet) Yes(3) VM accessing to DHCP Yes(4) Gets metadata Yes
North-South
(5) Using Floating IP Yes(6) Using SNAT Yes
Compute node #1 Network NodeCompute node #2
br-tun
br-int
br-tun
br-int
br-ex
VM VM VM DHCP
br-tun
br-int
vRouter
(1)
(1)
(2),(3),(4),(5),(6)
22
Changes in the Architecture by DVR These components are distributed to each compute node
vRouter, Floating IP, and bridge (interface) to external network
Copyright 2015 FUJITSU LIMITED
Compute node #1 Network NodeCompute node #2
br-tun
br-int
br-tun
br-int
br-ex
VM VMDHCP
br-tun
br-int
br-exbr-ex
fip fip
Need through Network Node?
East-West
(1) Between VMs (same subnet) No(2) Between VMs (different subnet) Yes(3) VM accessing to DHCP Yes(4) VM getting metadata Yes
North-South
(5) Using Floating IP Yes(6) Using SNAT Yes
23
Network flow of DVR
Copyright 2015 FUJITSU LIMITED
Compute node #1 Network NodeCompute node #2
br-ex
VM VMDHCP
br-tun
br-int
br-exbr-ex
fip fip
br-tun
br-int
br-tun
br-int
East-West traffic - Between VMs(same subnet) Not need to go through Network Node in the both cases:
VMs are on the same host VMs are on different hosts
Need through Network Node?
East-West
(1) Between VMs (same subnet) No(2) Between VMs (different subnet) Yes(3) VM accessing to DHCP Yes(4) VM getting metadata Yes
North-South
(5) Using Floating IP Yes(6) Using SNAT Yes
24
Network flow of DVR
Copyright 2015 FUJITSU LIMITED
Compute node #1 Network NodeCompute node #2
br-ex
VM VMDHCP
br-tun
br-int
br-exbr-ex
fip fip
br-tun br-tun
br-intbr-int
East-West traffic - Between VMs(different subnet) No longer need to go through Network Node VM access to vRouter that exists on own Compute Node
Need through Network Node?
East-West
(1) Between VMs (same subnet) No(2) Between VMs (different subnet) Yes -> NO
(3) VM accessing to DHCP Yes(4) VM getting metadata Yes
North-South
(5) Using Floating IP Yes(6) Using SNAT Yes
25
Network flow of DVR
Copyright 2015 FUJITSU LIMITED
Compute node #1 Network NodeCompute node #2
br-ex
VM VMDHCP
br-tun
br-int
br-tun
br-int
br-tun
br-int
br-exbr-ex
fip fip
East-West traffic – Access to DHCP Still need to go through Network Node
Need through Network Node?
East-West
(1) Between VMs (same subnet) No(2) Between VMs (different subnet) Yes -> NO
(3) VM accessing to DHCP Yes(4) VM getting metadata Yes
North-South
(5) Using Floating IP Yes(6) Using SNAT Yes
26
Network flow of DVR
Copyright 2015 FUJITSU LIMITED
Compute node #1 Network NodeCompute node #2
br-ex
VM VMDHCP
br-tun br-tun
br-int
br-tun
br-int
br-exbr-ex
fip fipbr-int
East-West traffic – Gets metadata No longer need to go through Network Node VM access to vRouter that exists on own Compute Node
Need through Network Node?
East-West
(1) Between VMs (same subnet) No(2) Between VMs (different subnet) Yes -> NO
(3) VM accessing to DHCP Yes(4) VM getting metadata Yes -> NO
North-South
(5) Using Floating IP Yes(6) Using SNAT Yes
27
Network flow of DVR
Copyright 2015 FUJITSU LIMITED
Compute node #1 Network NodeCompute node #2
br-ex
VM VMDHCP
br-tun br-tun
br-int
br-tun
br-int
br-ex
fipbr-int
br-ex
fip
North-South traffic – Using Floating IP No longer need to go through Netowork Node VM access to vRouter and translate to floating IP that exists on own
Compute node
Need through Network Node?
East-West
(1) Between VMs (same subnet) No(2) Between VMs (different subnet) Yes -> NO
(3) VM accessing to DHCP Yes(4) VM getting metadata Yes -> NO
North-South
(5) Using Floating IP Yes -> NO(6) Using SNAT Yes
28
Network flow of DVR
Copyright 2015 FUJITSU LIMITED
Compute node #1 Network NodeCompute node #2
VM VMDHCP
br-tun
br-int
br-exbr-ex
fip fip
br-exbr-tun
br-int
br-tun
br-int
North-South traffic – Using SNAT Still need to go through Network Node
Need through Network Node?
East-West
(1) Between VMs (same subnet) No(2) Between VMs (different subnet) Yes -> NO
(3) VM accessing to DHCP Yes(4) VM getting metadata Yes -> NO
North-South
(5) Using Floating IP Yes -> NO(6) Using SNAT Yes
29
Network flow of DVR 3 types of traffic are no longer need to go through
Network Node due to DVR
Copyright 2015 FUJITSU LIMITED
Compute node #1 Network NodeCompute node #2
br-ex
VM VMDHCP
br-ex
fip
br-tun
br-int
(4)
(5)
(2)
br-tun
br-int
br-ex
fip
br-tun
br-int
Need through Network Node?
East-West
(1) Between VMs (same subnet) No(2) Between VMs (different subnet) Yes -> NO
(3) VM accessing to DHCP Yes(4) VM getting metadata Yes -> NO
North-South
(5) Using Floating IP Yes -> NO(6) Using SNAT Yes
30
Network flow of DVR 2 types of traffics still need to go through Network Node
Meaning there are still two issues (Bottleneck and SPOF)
Copyright 2015 FUJITSU LIMITED
Compute node #1 Network NodeCompute node #2
VM VMDHCP
br-tun
br-int
br-exbr-ex
fip fip
(6)
(3)
br-exbr-tun
br-int
br-tun
br-int
Need through Network Node?
East-West
(1) Between VMs (same subnet) No(2) Between VMs (different subnet) Yes -> NO
(3) VM accessing to DHCP Yes(4) VM getting metadata Yes -> NO
North-South
(5) Using Floating IP Yes -> NO(6) Using SNAT Yes
Copyright 2015 FUJITSU LIMITED31
Completely Distributed Virtual Router
To eliminate bottle neck and SPOF completely Distributed SNAT and DHCP are required
Compute Node Network Node
ML2 pluginOVS-agent
DHCP-agent
Controller Node
Neutron Server
L3-agent(SNAT)
ML2 pluginOVS-agentOVS-agentL3-agent
metadata-agentDHCP-agent
Copyright 2015 FUJITSU LIMITED33
Distributed SNAT Distribute remaining l3-agent(SNAT) on Network Node to each Compute
Node
Compute Node Network Node
ML2 pluginOVS-agent
DHCP-agent
Controller Node
Neutron Server
L3-agent(SNAT)
ML2 pluginOVS-agentOVS-agentL3-agent
metadata-agent
Copyright 2015 FUJITSU LIMITED34
Conventional behavior of SNAT Create SNAT on Network Node for each vRouter
If a tenant has multiple vRouters, then Neutron creates multiple SNATs The amount of public IP consumption
# of vRouter that sets a gateway - 3 public IPs are needed in the following configuration
Compute Node #1 Network NodeCompute Node #2VM VM VM
SNATvRouterA1 vRouterA1'vRouterA2 vRouterA2'vRouterB1 vRouterB1'SNAT
SNATVM VM VM VM
Copyright 2015 FUJITSU LIMITED35
Possible Idea for Distributed SNAT Five ideas that I devised w/ some hints in the past discussions
#1 One SNAT for one Router #2 One SNAT for one Compute Node #3 One SNAT for one Compute Node w/o Security Concern #4 Double NAT #5 BGP(Border Gateway Protocol) Understood why Distributed SNAT has not been implemented yet
It’s not easy at all to devise the best design…
Will explain these ideas from the next page For each idea, will show an actual amount of public IP consumption in an example system
configuration Compute Node#1 and #2 have two VMs of tenant#1 and a VM of tenant#2 respectively
Copyright 2015 FUJITSU LIMITED36
#1 One SNAT for one Router Distribute SNAT simply … Follow the conventional behavior
The number of SNAT increases because each vRouter requires one SNAT
Compute Node #1 Compute Node #2VM VM
SNATSNAT SNAT
VMVM VM
SNATSNAT SNAT
VM
.1 .3 .5 .2 .4 .6
Copyright 2015 FUJITSU LIMITED37
#1 One SNAT for one Router The amount of public IP consumption
(# of Compute Nodes) * (# of vRouters per Compute Node) … Too many!! 6 public IPs are needed in the example system configuration
Compute Node #1 Compute Node #2VM VM
SNATSNAT SNAT
VMVM VM
SNATSNAT SNAT
VM
.1 .3 .5 .2 .4 .6
Copyright 2015 FUJITSU LIMITED38
#2 One SNAT for one Compute Node One SNAT shared by vRouters in Compute Node
In other words, sharing SNAT between different tenantsNeeds an exclusion of address scope between different tenants
Compute Node #1 Compute Node #2VM VM
SNAT
VMVM VM
SNAT
VM
.1 .2
Copyright 2015 FUJITSU LIMITED39
#2 One SNAT for one Compute Node The amount of public IP consumption
# of Compute nodes … Sufficiently small 2 Public IPs are needed in the example system configuration
Security concern exists If an admin makes a wrong setting on SNAT, VMs in the different tenants can
communicate.
Compute Node #1 Compute Node #2VM VM
SNAT
VMVM VM
SNAT
VM
.1 .2
Copyright 2015 FUJITSU LIMITED40
#3 One SNAT for one Compute Node w/o Security Concern SNAT shared by vRouters of the same tenant in Compute Node
Compute Node #1 Compute Node #2VMVM VM
SNAT
VM
.1 SNAT .3 SNAT .4
VM VM
SNAT .2
Copyright 2015 FUJITSU LIMITED41
#3 One SNAT for one Compute Node w/o Security Concern The amount of public IP consumption
(# of Compute nodes) * (# of Tenants) If each tenant only has one vRouter, the consumption will be the same
as idea #1 because (# of Tenants) becomes (# of vRouter) 4 public IPs are needed in the example system configuration
Compute Node #1 Compute Node #2VMVM VM
SNAT
VM
.1 SNAT .3 SNAT .4
VM VM
SNAT .2
Copyright 2015 FUJITSU LIMITED42
#4 Double NAT Double NAT on SNAT and upstream physical router
SNAT: translates private IP to a dedicated local IP (like ISP shared IP) Upstream physical router: translates the local IP to a public IP
Need some changes to Floating IP mechanism
- Floating IP needs to be able to allocate the local IPs to VMs
Compute Node #1 Compute Node #2VM VM
SNATSNAT SNAT
VMVM VM
SNATSNAT SNAT
VM
.1 .3 .5 .2 .4 .6
Upstream Router
NAT table
Copyright 2015 FUJITSU LIMITED43
#4 Double NAT Dynamic setting to upstream physical router is required
OpenStack doesn’t have such a feature yet The amount of public IP consumption
(# of vRouter) 3 public IPs are needed
Compute Node #1 Compute Node #2VM VM
SNATSNAT SNAT
VMVM VM
SNATSNAT SNAT
VM
.1 .3 .5 .2 .4 .6
Upstream Router
NAT table(Need dynamic setting)
Copyright 2015 FUJITSU LIMITED44
#5 BGP(Border Gateway Protocol) vRouters in the same group share one public IP
Upstream physical router as a gateway BGP speaker uses BGP to advertise the router which SNAT
to connect in the same group Neutron doesn’t have a feature like BGP Speaker yet
Compute Node #1 BGP SpeakerCompute Node #2
Upstream Router
Peering
VM
SNAT.6
VM VM
SNAT .2 SNAT .4
VM
SNAT.5
VM VM
SNAT .1 SNAT .3
Copyright 2015 FUJITSU LIMITED45
#5 BGP(Border Gateway Protocol) High cost
Development cost: BGP support CAPEX: Linux server for BGP speaker Special Requirement: upstream physical router that supports BGP
Compute Node #1 BGP SpeakerCompute Node #2
Upstream Router
Peering
VM
SNAT.6
VM VM
SNAT .2 SNAT .4
VM
SNAT.5
VM VM
SNAT .1 SNAT .3
Copyright 2015 FUJITSU LIMITED46
#5 BGP(Border Gateway Protocol) The amount of public IP consumption
# of vRouter … same as the current DVR where SNAT is not distributed “BGP support” is under development now
SPEC(merged): https://review.openstack.org/#/c/90833/
Compute Node #1 BGP SpeakerCompute Node #2
Upstream Router
Peering
VM
SNAT.6
VM VM
SNAT .2 SNAT .4
VM
SNAT.5
VM VM
SNAT .1 SNAT .3
Copyright 2015 FUJITSU LIMITED47
#5 BGP(Border Gateway Protocol) Need more enhancements for “BGP support” to realize this idea because
BGP Speaker will be another SPOF Possible solution is to make vRouter behave as BGP Speaker
Compute Node #1 BGP SpeakerCompute Node #2
Upstream Router
Peering
VM
SNAT.6
VM VM
SNAT .2 SNAT .4
VM
SNAT.5
VM VM
SNAT .1 SNAT .3
Copyright 2015 FUJITSU LIMITED48
Summary of the Ideas Ideas in the order of the amount of public IP consumption
Name Consumption Cost for Infrastructure
#1 One SNAT for one Router (# of Compute nodes) * (# of Routers)
#3 One SNAT for one Compute Node w/o Security Concern
(# of Compute nodes) * (# of Tenants)
#2 One SNAT for one Compute Node # of Compute Nodes
#4 Double NAT # of vRouters Physical router required
#5 BGP # of vRouters A server for BGP Speaker and physical router required
Copyright 2015 FUJITSU LIMITED49
Summarized in table Trade-off between IP consumption and cost for infrastructure
Name Consumption Cost for Infrastructure
#1 One SNAT for one Router (# of Compute nodes) * (# of Routers)
#3 One SNAT for one Compute Node w/o Security Concern
(# of Compute nodes) * (# of Tenants)
#2 One SNAT for one Compute Node # of Compute Nodes
#4 Double NAT # of vRouters Physical router required
#5 BGP # of vRouters A server for BGP Speaker and physical router required
Many
A fewHeavy
Light
Copyright 2015 FUJITSU LIMITED51
Future Plan
I’ll propose idea #3 again at the Design Summit, “One SNAT for one Compute Node w/o Security Concern” It doesn’t require any specific hardware Still many public IPs can be consumed in a particular environment If you cannot get much advantages in your environment, you can use the
centralized SNAT (Network Node)
Let’s discuss at the Design SummitWill discuss at Neutron’s Contributors meetup on Friday!