operationalizing evpn in the data center: part 2
TRANSCRIPT
1
Nov 1, 2017
Dinesh G Dutt, Vivek Venkataraman | Cumulus Networks
Part 2: Routing, Deployment Use Cases & Best Practices
Operationalizing EVPN in the DC
2Cumulus Networks
EVPN Summary
Routing Models
Configuring Routing
Troubleshooting EVPN
Deployment Models and Recommendations
Agenda
3Cumulus Networks
Key Takeaways
• EVPN supports routing as well as bridging• Since L2 is no longer behind a single rack, multiple routing
models are possible▪ VRF is supported in all models
• Pick right routing model based on use case• FRR/Cumulus continues the simple configuration model
even with EVPN routing
4Cumulus Networks
The Story So Far
• Designed to address the twin issues of:▪ Multi-tenancy over an L3 network▪ Allow disjointed L2 segments over an L3 network
• Dataplane:▪ Supports multiple encapsulations: MPLS, VxLAN, NVGRE…▪ VxLAN is the common choice within the data center
• Control plane is BGP• Standards-based
▪ IETF original draft for MPLS: RFC 7432▪ IETF draft for support with VxLAN: draft-ietf-bess-evpn-overlay
5Cumulus Networks
Why Now ?
• Adoption of leaf-spine based IP fabrics to build data centers• Rise of switching silicon that supports VxLAN routing• Multi-vendor support for EVPN
▪ Lack of widespread adoption of controller-based overlays
6Cumulus Networks
The Next Chapter
• EVPN is more than just multi-tenancy L2:▪ supports routing, multicast handling, MAC/VM mobility etc.
• This part will cover these other aspects• Plus, deployment models
7Cumulus Networks
VXLAN Summary
• UDP/IP based encapsulation carrying L2 payloads▪ RFC 7438
• Source port hashing allows fine-grained traffic spreading of overlay traffic without requiring deep packet parsing
• 24-bit Virtual Network Identifier (VNI) identifies the VPN• Tunnel ingress and egress are called VTEP (VXLAN Tunnel
Endpoint)
8Cumulus Networks
• Protocol aspects based on BGP-based MPLS VPNs:▪ Routes of a tenant kept separate with Route Distinguisher (RD)▪ Routes contain Route Targets (RTs) to identify the VPN (L2 and/or L3 )▪ Uses MP-BGP AFI L2VPN (25) SAFI EVPN (70)▪ Various new BGP attributes (extended communities) - MAC Mobility,
Default Gateway, Encapsulation, Router MAC etc.• Multiple pieces of information exchanged in EVPN:
▪ Another level of encoding, called route types, to identify the information carried
EVPN Summary: Protocol
9Cumulus Networks
EVPN Summary - key route types
Route Type
Name Usage
RT-2 MAC/IP Advertisement Route Advertise MACs and/or MACIPs
RT-3 Inclusive Multicast Ethernet Tag Route
Advertise VNI membership (primarily to prune recipients of BUM traffic)
RT-5 IP Prefix Route Advertise routes to subnet prefixes
RT-1 Ethernet AutoDiscovery (A-D) Route For multi-homing, used to let remote VTEPs know about connectivity to an Ethernet Segment and VLANs reachable on it.
RT-4 Ethernet Segment Route For designated forwarder (DF) election for BUM traffic handling in multi-homing scenarios.
RT-6 Selective Multicast Ethernet Tag Route
To carry IGMP multicast group membership information for a tenant using EVPN.
Route/VNIinfo
Dual attachsupport
MulticastInfo
10Cumulus Networks
H11 -> H41: VXLAN Bridging (Packet Forwarding Level Set)
50.1.1.11 (VL 100) 50.1.1.41 (VL 100)
L1
L2 L3
L4
S1 S2
H11 H41
Unencapsulatedpacket: DMAC is H41
Encapsulated packet:
Routed from L1 -> S1 Encapsu
lated packet:
Routed from S1 ->
L4
Unencapsulated
packet: DMAC is H41
11Cumulus Networks
H11 -> H41: VXLAN Bridging (Packet Forwarding Level Set)
50.1.1.11 50.1.1.41
L1
L2 L3
L4
S1 S2
H11 H41
Unencapsulatedpacket: DMAC is H41
Encapsulated packet:
Routed from L1 -> S1 Encapsu
lated packet:
Routed from S1 ->
L4
● Spines use only the VXLAN Header to route the packet● Inner packet is carried practically unmodified● L1 maps brown VLAN to brown VNI, L4 does the opposite
Unencapsulated
packet: DMAC is H42
DMAC: H41SMAC: H11DstP: H41SrcIP: H11
DMAC: H41SMAC: H11DstP: H41SrcIP: H11
DMAC: H41SMAC: H11DstP: H41SrcIP: H11
DMAC: H41SMAC: H11DstP: H41SrcIP: H11
Data
DataData
Data
DMAC: S1SMAC: L1DstIP: L4SrcIP: L1VNI: Brown
DMAC: S1SMAC: L1DstIP: L4SrcIP: L1VNI: Brown
VXLANHeader
12
Routing Models
13Cumulus Networks
Regular Routing (H11 -> H12), No VxLAN: Case 1
50.1.1.11 (VLAN 100) 50.1.2.22 (VLAN 110)
L1L2 L3
L4
S1 S2
H11 H12
1. H11 bridges to L1, default gateway
2. L1:a. routes to Blue subnetb. L1 identifies Blue subnet
as being localc. L1 does neighbor lookup
on H123. L1 bridges to H12
12
14Cumulus Networks
Regular Routing (H11 -> H42), No VxLAN: Case 2
50.1.1.11 (VLAN 100) 50.1.2.42 (VLAN 110)
L1L2 L3
L4
S1 S2
H11 H42
L1 and L4 have exchanged subnet routes
1. H11 bridges to L1, default gateway
2. L1 routes to next hop S1 (or S2)
3. S1 (or S2) routes to L44. On L4, destination is on a
local subnet. L4 does neighbor lookup and bridges to H42
1
2 3
4
15Cumulus Networks
Routing (H11 -> H42) with VxLAN
• Where is H11’s (and H42’s) default router ?
• If L1 is the default router, what happens after initial routing?
▪ Bridge to H42 (case 1) ?▪ Routing at next hop L4 (case
2)?• L1 and L4 always encapsulate
and decapsulate VXLAN packet
• Spines only route encapsulated packets
50.1.1.11 (VL 100) 50.1.2.42 (VL 110)
L1L2 L3
L4
S1 S2
H11 H42
16Cumulus Networks
The Rise of the Routing Models
• Where is H11’s (and H42’s) default router ?▪ Specific per-VNI (or all VNI) gateways (Centralized routing)▪ All ingress VTEPs are gateways (Distributed routing)
• So, what happens after the initial routing ?▪ Bridge (case 1): Asymmetric Routing▪ Route (case 2): Symmetric Routing
17Cumulus Networks
Asymmetric vs Symmetric: Observations
• Asymmetric Model assumes all subnets are locally attached• Symmetric model assumes all subnets are NOT locally
attached• This choice plays a role in what’s suitable for what
deployment
18Cumulus Networks
Asymmetric Routing H11 -> H42: Step by Step
50.1.1.11 (VLAN 100) 50.1.2.42 (VLAN 110)
L1L2 L3
L4
S1 S2
H11 H42
1. H11 sends unencapsulated to L1a. DMAC = L1’s MAC, DIP = H42’s IP
1
19Cumulus Networks
Asymmetric Routing H11 -> H42: Step by Step
50.1.1.11 (VLAN 100) 50.1.2.42 (VLAN 110)
L1L2 L3
L4
S1 S2
H11 H42
1. H11 sends unencapsulated to L1a. DMAC = L1’s MAC, DIP = H42’s IP
2. L1:a. routes the packet (in tenant’s VRF) to
blue subnetb. identifies it is a local subnet and does a
neighbor lookup to get H42’s MAC*
c. Determines H42’s MAC is behind L4d. L1 encapsulates the packet with VxLAN
header:i. Payload: DMAC = H42’s MAC.
SMAC = L1’s MACii. DIP = L4’s VTEP, SIP = L1’s
VTEP, VNI = Blueiii. DMAC = S1’s MAC, SMAC = L1’s
MAC
1
2
20Cumulus Networks
Asymmetric Routing H11 -> H42: Step by Step
50.1.1.11 (VLAN 100) 50.1.2.42 (VLAN 110)
L1L2 L3
L4
S1 S2
H11 H42
1. H11 sends unencapsulated to L1a. DMAC = L1’s MAC, DIP = H42’s IP
2. L1:a. routes the packet (in tenant’s VRF) to
blue subnetb. identifies it is a local subnet and does a
neighbor lookup to get H42’s MAC*
c. Determines H42’s MAC is behind L4d. L1 encapsulates the packet with VxLAN
header:i. Payload: DMAC = H42’s MAC.
SMAC = L1’s MACii. DIP = L4’s VTEP, SIP = L1’s
VTEP, VNI = Blueiii. DMAC = S1’s MAC, SMAC = L1’s
MAC3. S1 routes to L44. L4:
a. decapsulates the packet; VNI = Blueb. Looks up DMAC of H42 on
corresponding VLAN, bridges out port
1
2 3
4
21Cumulus Networks
Asymmetric Routing: Putting It All Together
1. Host sends packet to gateway router2. Ingress VTEP (GW):
a. Routesb. Bridgesc. Encapsulates
3. Spine switches (underlay) route4. Egress VTEP:
a. Decapsulatesb. Bridges to end host
Packets are transported through the fabric in the final destination VNI
50.1.1.11 (VLAN 100) 50.1.2.42 (VLAN 110)
L1L2 L3
L4
S1 S2
H11 H42
1
2 3
4
22Cumulus Networks
Symmetric Routing H11 -> H42: Step by Step
50.1.1.11 (VLAN 100) 50.1.2.42 (VLAN 110)
L1L2 L3
L4
S1 S2
H11 H42
1. H11 sends unencapsulated to L1a. DMAC = L1’s MAC, DstIP = H42
1
23Cumulus Networks
Symmetric Routing H11 -> H42: Step by Step
50.1.1.11 (VLAN 100) 50.1.2.42 (VLAN 110)
L1L2 L3
L4
S1 S2
H11 H42
1. H11 sends unencapsulated to L1a. DMAC = L1’s MAC, DIP = H42’s IP
2. L1:a. routes the packet (/32 route) to next hop
L4 - DMAC is L4’s Router MACb. L1 encapsulates the packet with VxLAN
header:i. Payload: DMAC = L4’s Router
MAC. SMAC = L1’s MACii. DIP = L4’s VTEP, SIP = L1’s
VTEP, VNI = ??iii. DMAC = S1’s MAC, SMAC = L1’s
MAC1
2
24Cumulus Networks
Symmetric Routing H11 -> H42: Step by Step
50.1.1.11 (VLAN 100) 50.1.2.42 (VLAN 110)
L1L2 L3
L4
S1 S2
H11 H42
1. H11 sends unencapsulated to L1a. DMAC = L1’s MAC, DIP = H42’s IP
2. L1:a. routes the packet (/32 route) to next hop
L4 - DMAC is L4’s Router MACb. L1 encapsulates the packet with VxLAN
header:i. Payload: DMAC = L4’s Router
MAC. SMAC = L1’s MACii. DIP = L4’s VTEP, SIP = L1’s
VTEP, VNI = ??iii. DMAC = S1’s MAC, SMAC = L1’s
MAC
Question: What VNI to use to transport the frame to L4 ?
1. Brown (ingress VNI)2. Blue (egress VNI, but how do I know ?)3. Some other VNI
1
2
25Cumulus Networks
Symmetric Routing H11 -> H42: Step by Step
50.1.1.11 (VLAN 100) 50.1.2.42 (VLAN 110)
L1L2 L3
L4
S1 S2
H11 H42
1. H11 sends unencapsulated to L1a. DMAC = L1’s MAC, DIP = H42’s IP
2. L1:a. routes the packet (/32 route) to next hop
L4 - DMAC is L4’s Router MACb. L1 encapsulates the packet with VxLAN
header:i. Payload: DMAC = L4’s Router
MAC. SMAC = L1’s MACii. DIP = L4’s VTEP, SIP = L1’s
VTEP, VNI = per-tenant L3 transport VNI
iii. DMAC = S1’s MAC, SMAC = L1’s MAC
3. S1 routes to L44. L4:
a. decapsulates the packet. VNI is the L3 VNI - identifies the VRF.
b. Looks up the DIP in VRF and routes to local subnet
c. Looks up neighbor table for H42d. Bridges to H42
1
2 3
4
26Cumulus Networks
Symmetric Routing: Putting It All Together
50.1.1.11 (VLAN 100) 50.1.2.42 (VLAN 110)
L1L2 L3
L4
S1 S2
H11 H42
1. Host sends packet to gateway router2. Ingress VTEP (GW):
a. Routes to egress VTEPb. Encapsulates
3. Spine switches (underlay) route4. Egress VTEP:
a. Decapsulatesb. Routes to local subnetc. Bridges to end host
Packets are transported through the fabric in a per-tenant L3 VNI.
1
2 3
4
27Cumulus Networks
• L3 VNI - configured and exchanged in control plane and carried in routed packets.
▪ Additional configuration▪ Corresponds to VRF associated with the L2 VNI(s)▪ Different number space from L2 VNI
• Router MAC - Automatically derived (in Cumulus Linux/FRR) and exchanged in the control plane. Used in routed packets to indicate packet should be routed by egress VTEP (next hop)
Symmetric routing - L3 Transport VNI and Router MAC
28Cumulus Networks
Asymmetric vs Symmetric: Packet Header View
50.1.1.11 50.1.2.42
L1
L2 L3
L4
S1 S2
H11 H42
DMAC: L1SMAC: H11DstP: H42SrcIP: H11
Data
DMAC: H42SMAC: L1DstP: H42SrcIP: H11
Data
DMAC: S1SMAC: L1DstIP: L4SrcIP: L1VNI: L3 VNI
DMAC: L4SMAC: S1DstIP: L4SrcIP: L1VNI: L3 VNI
DMAC: L4SMAC: L1DstP: H42SrcIP: H11
Data
DMAC: L4SMAC: L1DstP: H42SrcIP: H11
Data
DMAC: H42SMAC: L1DstP: H42SrcIP: H11
Data
DMAC: S1SMAC: L1DstIP: L4SrcIP: L1VNI: Blue
DMAC: L4SMAC: S1DstIP: L4SrcIP: L1VNI: Blue
ASYMM SYMM ASYMMSYMM
DMAC: H42SMAC: L1DstP: H42SrcIP: H11
29Cumulus Networks
Asymmetric vs Symmetric: Forwarding Tables View
Asymmetric Symmetric
MAC Table All end stations End stations in all locally known subnets plus remote VTEPs
Neighbor Table All end stations End stations in all locally known subnets* plus remote VTEPs
Route Table Locally attached prefixes All end stations plus local subnets
VNIs All VNIs in fabric Locally attached VNIs plus L3 transport VNIs
* - Needed for ARP Suppression
30Cumulus Networks
Asymmetric vs Symmetric: Configuration View
Asymmetric Symmetric
Uniform configuration Yes No, since not all VNIs are present everywhere
Need Orchestrator No Most likely, since VNIs and their VLAN mappings will need to be configured or torn down as hosts/VMs move
Scaling Yes, breaking mobility up into pods
Yes
Miscellaneous Need configuring and mapping additional L3 transport VNIs
31Cumulus Networks
Asymmetric vs Symmetric: Vendor Interop View
Aymmetric Symmetric
Arista X
Cisco X
Juniper X
Cumulus/FRR X X*
* - Supported in upcoming 3.5 release of Cumulus Linux
32Cumulus Networks
Distributed Routing Model
• Since end station IP/MAC is spread throughout the network, no specific first hop router can be first hop router
• Distributed model assumes every ToR switch is the first hop router for all locally attached subnets
▪ Anycast IP and anycast MAC model▪ Similar to VRR used today (VARP in Arista lingo)
• Most common deployed: when used to replace existing VLAN-based access-agg-core networks with VXLAN-based Clos networks
33Cumulus Networks
Centralized Routing Model
• Encapsulated packets bridged to a designated first hop router
• Packets are routed by this router• Encapsulated packets bridged to final destination by this
router• Primary switching silicon requirement:
▪ To decapsulate, route, bridge, encapsulate, route on underlay header
• Most commonly deployed: when EVPN is used for multi-tenancy in cloud-like environments
34Cumulus Networks
Centralized Routing H11 -> H42: Sample Packet Flow
50.1.1.11 (VLAN 100) 50.1.2.42 (VLAN 110)
L1L2 L3
L4
S1 S2
H11 H42
1. Host sends packet to gateway router (L2)2. Ingress VTEP (GW):
a. Bridges to egress VTEP/router L2b. Encapsulates packet & sends out
3. Spine switches (underlay) route4. Gateway VTEP:
a. Decapsulatesb. Routes to local subnetc. Bridges to end hostd. Encapsulates packet & sends out
5. Spine switches (underlay) route6. Egress VTEP:
a. Decapsulatesb. Bridges to end host
Packets are transported through the fabric in the bridge VNI.
1
2 3 4 5
6
35Cumulus Networks
How do I talk to the outside world?
• Routing/Packet Forwarding was all based on /32 routes or neighbor entries.
• To route to external networks, we need to route to prefixes. ▪ Enter EVPN type-5 routes (RT-5).
• RT-5 allows an IP prefix to be advertised, not just MAC+IP. ▪ For the common scenario of connecting to another subnet or external
network, the advertising VTEP is itself the next hop. RT-5 contains the Router MAC of this VTEP.
▪ Specified in draft-ietf-bess-evpn-prefix-advertisement
36Cumulus Networks
Control Plane Illustration for External Routing
L1L2 L3
L4
S1 S2
● Per-tenant VRF peering between Border Leaf BL1 and WAN edge router R1
● R1 advertises prefixes relevant to a tenant (e.g., default route) on corresponding peering.
● BLs are typically deployed in pairs for redundancy.
● For internal destinations to be reachable, BLs will advertise corresponding subnets to R1.
BL1R1
WAN
37Cumulus Networks
Control Plane Illustration for External Routing
L1L2 L3
L4
S1 S2
● Per-tenant VRF peering between Border Leaf BL1 and WAN edge router R1
● R1 advertises prefixes relevant to a tenant (e.g., default route) on corresponding peering.
● BLs are typically deployed in pairs for redundancy.
● For internal destinations to be reachable, BLs will advertise corresponding subnets to R1.
BL1
● BL1 installs routes in VRF routing table
● BL1 exports these routes into EVPN as RT-5.
● RT-5 advertised to other VTEPs with L3 VNI of associated VRF. Next hop is BL1..
R1WAN
38Cumulus Networks
Control Plane Illustration for External Routing
L1L2 L3
L4
S1 S2
Receiving VTEPs (L1, …) install routes into VRF routing table - next hop is BL1, MAC is BL1’s RMAC
● Per-tenant VRF peering between Border Leaf BL1 and WAN edge router R1
● R1 advertises prefixes relevant to a tenant (e.g., default route) on corresponding peering.
● BLs are typically deployed in pairs for redundancy.
● For internal destinations to be reachable, BLs will advertise corresponding subnets to R1.
BL1
● BL1 installs routes in VRF routing table
● BL1 exports these routes into EVPN as RT-5.
● RT-5 advertised to other VTEPs with L3 VNI of associated VRF. Next hop is BL1..
R1WAN
Note: This is for illustration purposes, a real deployment is likely to have NAT, FW etc.
39Cumulus Networks
External Routing: Packet Flow
L1L2 L3
L4
S1 S2
BL1R1
WAN
50.1.1.11 (VL 100)
H11201.11.1.45
H100
H11 sends the packet for H100 to L1 - its default GW
40Cumulus Networks
External Routing: Packet Flow
L1L2 L3
L4
S1 S2
BL1R1
WAN
50.1.1.11 (VL 100)
H11201.11.1.45
H100
H11 sends the packet for H100 to L1 - its default GW
● L1 matches packet against external route (default or prefix advertised by BL1) and routes to next hop VTEP BL1.
● Packet routed over core with DMAC = BL1’s Router MAC. VNI is the L3 VNI for this VRF.
41Cumulus Networks
External Routing: Packet Flow
L1L2 L3
L4
S1 S2
BL1 terminates the VxLAN tunnel and routes the packet in the tenant VRF - on to R1..
BL1R1
WAN
50.1.1.11 (VL 100)
H11201.11.1.45
H100
H11 sends the packet for H100 to L1 - its default GW
● L1 matches packet against external route (default or prefix advertised by BL1) and routes to next hop VTEP BL1.
● Packet routed over core with DMAC = BL1’s Router MAC. VNI is the L3 VNI for this VRF.
42Cumulus Networks
Wait...Is RT-5 used only for external connectivity?
• No! RT-5 can also be used for inter-POD and inter-DC communication.
• It really depends on how the subnets have been provisioned i.e., contained within a POD or DC.
• Cumulus Linux (and FRR) supports RT-5 for external and inter-POD/inter-DC communication - available in upcoming release.
43
Configuration Example
44Cumulus Networks
Configuration Steps: Asymmetric Routing
• Provision VLANs and VNIs on all leaves• Provision subnets for all relevant VLANs (SVIs)• Map SVIs to appropriate VRF• Configure eBGP between leaf and spine• Activate and advertise information about all locally active
VNIs
45Cumulus Networks
Configuration Steps: Symmetric Routing
• Provision relevant locally attached VLANs and VNIs on the leaves (dynamic, non-uniform compared to asymmetric)
• Provision subnets for all locally attached VLANs (SVIs)• Map SVIs to appropriate VRF• For each VRF, provision an L3 VNI (additional step
compared to asymmetric)• Configure eBGP between leaf and spine• Activate and advertise information about all locally active
VNIs
46Cumulus Networks
Asymmetric vs Symmetric Routing: FRR Configuration
# BGP/EVPN configurationrouter bgp 65456
bgp router-id 110.0.0.1neighbor fabric peer-groupneighbor fabric remote-as externalneighbor uplink-1 interface peer-group fabricneighbor uplink-2 interface peer-group fabricaddress-family ipv4 unicast neighbor fabric activate redistribute connectedaddress-family l2vpn evpn neighbor fabric activate advertise-all-vni
# L3 VNI configuration for tenant VRFvrf vrf-tenant1
vni 104001
# BGP/EVPN configurationrouter bgp 65456
bgp router-id 110.0.0.1neighbor fabric peer-groupneighbor fabric remote-as externalneighbor uplink-1 interface peer-group fabricneighbor uplink-2 interface peer-group fabricaddress-family ipv4 unicast neighbor fabric activate redistribute connectedaddress-family l2vpn evpn neighbor fabric activate advertise-all-vni
47Cumulus Networks
Centralized routing
• Fundamental configuration on Gateway VTEP(s) is same as in the distributed case.
• Gateway VTEP(s) need to be configured to advertise their own MACIP.
# BGP/EVPN configurationrouter bgp 65456
bgp router-id 110.0.0.5neighbor fabric peer-groupneighbor fabric remote-as externalneighbor uplink-1 interface peer-group fabricneighbor uplink-2 interface peer-group fabricaddress-family ipv4 unicast neighbor fabric activate redistribute connectedaddress-family l2vpn evpn neighbor fabric activate advertise-all-vni advertise-default-gw
48Cumulus Networks
Switching Silicon Support
• Considering only native, single-pass support for VxLAN routing
• Cavium and Barefoot chipsets are supposed to have support for all modes
T2 T2+ T3 Tomahawk family
Spectrum/A0
Spectrum/A1
Spectrum2
Asymmetric - X X - X X X
Symmetric - X X - X X X
Centralized - X X - - X X
49
What about multicast?
50Cumulus Networks
The jury is still out
• Multicast routing in EVPN is still evolving.
• There are at least two key aspects:▪ Optimized intra-subnet multicast (only to VTEPs behind which
interested receivers are present)▪ Optimized inter-subnet multicast - local/distributed routing wherever
possible
• There are multiple proposals being discussed - including leveraging MVPN and VPLS Multicast.
• Stay tuned for a future update on this topic!
51Cumulus Networks
Summary
• EVPN supports routing besides bridging• Due to the distributed nature of L2 in EVPN, several routing
models are possible• Choose the right model based on deployment use case
▪ Choose wisely• Cumulus/FRR supports (or will shortly support) all of the
routing models, including interop with other vendors▪ Most other vendors support only a subset of these
• Cumulus/FRR provides a radically simplified config for EVPN routing
52
Thank you!Visit us at cumulusnetworks.com or follow us @cumulusnetworks or
slack.cumulusnetworks.com
© 2017 Cumulus Networks. Cumulus Networks, the Cumulus Networks Logo, and Cumulus Linux are trademarks or registered trademarks of Cumulus Networks, Inc. or its affiliates in the U.S. and other countries. Other names may be trademarks of their respective owners. The registered trademark
Linux® is used pursuant to a sublicense from LMI, the exclusive licensee of Linus Torvalds, owner of the mark on a world-wide basis.
53Cumulus Networks
Flood multicast only where there are receivers
• Basic BUM handling will flood to all remote VTEPs.
• What if there is real multicast traffic (i.e., non link-local) for a tenant - e.g., system monitoring, discovery, data dissemination using Pub/Sub etc? Receivers may be dispersed in the DC.
▪ Enter Selective Multicast and EVPN Type-6 (RT-6) routes▪ IGMP/MLD state on attachment circuits (ACs) conveyed using EVPN
RT-6 to remote VTEPs▪ Receiving VTEPs generate proxy reports on their ACs▪ Receiving VTEPs also build state indicating which VTEPs need traffic
for a particular (C-*, C-G) or (C-S, C-G)
54Cumulus Networks
Distributed multicast routing
• When multicast sources and receivers are on different subnets, the (inter-subnet) multicast routing can get hairy:
▪ Only one VTEP can be the Designated Router (DR) on a subnet, so even for local receivers on a different subnet from source, packet may have to be routed by a remote VTEP.
▪ A VTEP could get multiple copies of the packet, one for each subnet
• Distributed multicast routing is the solution. In one proposal:▪ Each VTEP routes to local receivers on all subnets.▪ Only one copy sent to remote VTEPs - on source subnet▪ Receivers will receive on a special broadcast domain if they don’t have
the source subnet.
55Cumulus Networks
Symmetric routing - sample topology
50.1.1.11 (VL 100)
50.1.4.44 (VL 130)
L1L2 L3
L4
S1 S2
H11
50.1.2.12 (VL 110)H12
50.1.3.43 (VL 120)H43
VL 130
H44
● Tenant has 4 VLANs:○ VL 100 - 50.1.1.x/24○ VL 110 - 50.1.2.x/24○ VL 120 - 50.1.3.x/24○ VL 130 - 50.1.4.x/24
● VLANs 100 and 110 (and corresponding SVIs) are provisioned on {L1, L2} and VLANs 120 and 130 on {L3, L4}
● Anycast GW IP is 50.1.x.250 - provisioned on all Leafs.
● VLAN - VNI mappings:○ VL 100 - VNI 10100○ VL 110 - VNI 10110○ VL 120 - VNI 10120○ VL 130 - VNI 10130
● L3 VLAN and VNI for tenant are 4001 and 104001 respectively
56Cumulus Networks
Symmetric routing - sample interface configuration (L1)
# VxLAN interfaces and VLAN-VNI mappings (local ones)auto vxlan100iface vxlan100 vxlan-id 10100 vxlan-local-tunnelip 110.0.0.1 bridge-learning off bridge-access 100 bridge-arp-nd-suppress on
# VxLAN interface and VLAN-VNI mapping for the L3VNIauto vxlan4001iface vxlan4001 vxlan-id 104001 vxlan-local-tunnelip 110.0.0.1 bridge-learning off bridge-access 4001
# Bridge with member ports (VLAN-aware)auto br0iface br0 bridge-vlan-aware yes bridge-ports swp3 swp4 swp5 swp6 vxlan100 vxlan110 vxlan4001 bridge-stp on bridge-vids 100 110 4001
# Tenant VRF configuration - if multiple tenants existauto vrf-tenant1iface vrf-tenant1 vrf-table auto
# SVI with anycast GW IP (for local tenant subnets)auto vlan100iface vlan100 address 50.1.1.1/24 vlan-id 100 vlan-raw-device br0 address-virtual 00:00:5e:00:01:01 50.1.1.250/24 vrf vrf-tenant1
# L3 VLAN interface per tenant (for L3 VNI)auto vlan4001iface vlan4001 vlan-id 4001 vlan-raw-device br0 vrf vrf-tenant1
57Cumulus Networks
Symmetric routing - sample FRR configuration (L1)
# L3 VNI configuration for tenant VRFvrf vrf-tenant1
vni 104001
# BGP/EVPN configurationrouter bgp 65456
bgp router-id 110.0.0.1neighbor fabric peer-groupneighbor fabric remote-as externalneighbor uplink-1 interface peer-group fabricneighbor uplink-2 interface peer-group fabricaddress-family ipv4 unicast neighbor fabric activate redistribute connectedaddress-family l2vpn evpn neighbor fabric activate advertise-all-vni