![Page 1: Distributed load balancing...Global Traffic Distribution pavlos.parissis@booking.com. users websiteA ... Load balancing at wire-speed Adding/removing a path reshuffles flows. Equal-Cost](https://reader036.vdocument.in/reader036/viewer/2022070809/5f0789247e708231d41d777e/html5/thumbnails/1.jpg)
Distributed load balancingReal case example using open source on commodity hardware
Pavlos Parissis | LinuxConf Berlin 2016
![Page 3: Distributed load balancing...Global Traffic Distribution pavlos.parissis@booking.com. users websiteA ... Load balancing at wire-speed Adding/removing a path reshuffles flows. Equal-Cost](https://reader036.vdocument.in/reader036/viewer/2022070809/5f0789247e708231d41d777e/html5/thumbnails/3.jpg)
users
websiteA● Scales only vertically
● Single point of failure
● Choke point for (D)DOS
● Very expensive
Active Node Standby Node
The traditional way
![Page 4: Distributed load balancing...Global Traffic Distribution pavlos.parissis@booking.com. users websiteA ... Load balancing at wire-speed Adding/removing a path reshuffles flows. Equal-Cost](https://reader036.vdocument.in/reader036/viewer/2022070809/5f0789247e708231d41d777e/html5/thumbnails/4.jpg)
users
websiteA
A better way
![Page 5: Distributed load balancing...Global Traffic Distribution pavlos.parissis@booking.com. users websiteA ... Load balancing at wire-speed Adding/removing a path reshuffles flows. Equal-Cost](https://reader036.vdocument.in/reader036/viewer/2022070809/5f0789247e708231d41d777e/html5/thumbnails/5.jpg)
How to get there
● Equal-Cost Multi-Pathing routing
● Anycast network address
scheme
● Bird Internet Routing Daemon
● A healthchecker for Anycasted
services
● HAProxy Layer4-7 load balancer
![Page 6: Distributed load balancing...Global Traffic Distribution pavlos.parissis@booking.com. users websiteA ... Load balancing at wire-speed Adding/removing a path reshuffles flows. Equal-Cost](https://reader036.vdocument.in/reader036/viewer/2022070809/5f0789247e708231d41d777e/html5/thumbnails/6.jpg)
Equal-Cost Multi-Pathing routing
ECMP
1 2 3 4
Destination IP Next hop
5.56.17.220/32 node1
5.56.17.220/32 node2
5.56.17.220/32 node3
5.56.17.220/32 node4
● Nodes are distributed across multiple networks● Preserves source and destination addresses● Cheapest form of balancing● Load balancing at wire-speed● Adding/removing a path reshuffles flows
![Page 7: Distributed load balancing...Global Traffic Distribution pavlos.parissis@booking.com. users websiteA ... Load balancing at wire-speed Adding/removing a path reshuffles flows. Equal-Cost](https://reader036.vdocument.in/reader036/viewer/2022070809/5f0789247e708231d41d777e/html5/thumbnails/7.jpg)
Equal-Cost Multi-Pathing
Tier 1 Load balancer
Tier 2 Load balancer
users
Layer 3
Layer 7 Layer 7 Layer 7
![Page 8: Distributed load balancing...Global Traffic Distribution pavlos.parissis@booking.com. users websiteA ... Load balancing at wire-speed Adding/removing a path reshuffles flows. Equal-Cost](https://reader036.vdocument.in/reader036/viewer/2022070809/5f0789247e708231d41d777e/html5/thumbnails/8.jpg)
2-Tier setup in production
users
Layer 3
Tier 1 Load balancer
Layer 3
Layer 7 Layer 7 Layer 7 Layer 7 Layer 7Tier 2 Load balancer
Fabric Layer Layer 3
Layer 3ToR Layer Layer 3 Layer 3 Layer 3 Layer 3
Layer 7 Layer 7Layer 7Layer 7 Layer 7Layer 7 Layer 7Layer 7 Layer 7Layer 7
![Page 9: Distributed load balancing...Global Traffic Distribution pavlos.parissis@booking.com. users websiteA ... Load balancing at wire-speed Adding/removing a path reshuffles flows. Equal-Cost](https://reader036.vdocument.in/reader036/viewer/2022070809/5f0789247e708231d41d777e/html5/thumbnails/9.jpg)
Benefits of 2-Tier setup
● Horizontally scalable
● Scaling and managing each tier independently
● Single device becomes less critical
![Page 10: Distributed load balancing...Global Traffic Distribution pavlos.parissis@booking.com. users websiteA ... Load balancing at wire-speed Adding/removing a path reshuffles flows. Equal-Cost](https://reader036.vdocument.in/reader036/viewer/2022070809/5f0789247e708231d41d777e/html5/thumbnails/10.jpg)
Anycast network address scheme
sender
receiver A
receiver B
receiver C
distance in number of hops
![Page 11: Distributed load balancing...Global Traffic Distribution pavlos.parissis@booking.com. users websiteA ... Load balancing at wire-speed Adding/removing a path reshuffles flows. Equal-Cost](https://reader036.vdocument.in/reader036/viewer/2022070809/5f0789247e708231d41d777e/html5/thumbnails/11.jpg)
Anycast in production
Data-center A
LB platform
local users
Data-center B
LB platform
local users
users
transition time ~20ms
![Page 12: Distributed load balancing...Global Traffic Distribution pavlos.parissis@booking.com. users websiteA ... Load balancing at wire-speed Adding/removing a path reshuffles flows. Equal-Cost](https://reader036.vdocument.in/reader036/viewer/2022070809/5f0789247e708231d41d777e/html5/thumbnails/12.jpg)
Benefits of Anycast in production
● Network detect failures within 1.2secs ( BFD protocol helps a lot)
● Switches traffic to other location within 1sec
● Reduces network distance which lowers response time
● Provides a very fast and without manual intervention fail-over
which improves service reliability
● Works for TCP protocol
![Page 13: Distributed load balancing...Global Traffic Distribution pavlos.parissis@booking.com. users websiteA ... Load balancing at wire-speed Adding/removing a path reshuffles flows. Equal-Cost](https://reader036.vdocument.in/reader036/viewer/2022070809/5f0789247e708231d41d777e/html5/thumbnails/13.jpg)
Dive into details
● Bird Internet Routing daemon
● A healthchecker for anycasted
services
● HAProxy Layer4-7 load balancer
![Page 14: Distributed load balancing...Global Traffic Distribution pavlos.parissis@booking.com. users websiteA ... Load balancing at wire-speed Adding/removing a path reshuffles flows. Equal-Cost](https://reader036.vdocument.in/reader036/viewer/2022070809/5f0789247e708231d41d777e/html5/thumbnails/14.jpg)
appsapps
How it works
Load balancer node
HAProxy
Bird
anycasthealthchecker
check
ToRswitch
Fabricswitch
Users
apps
![Page 15: Distributed load balancing...Global Traffic Distribution pavlos.parissis@booking.com. users websiteA ... Load balancing at wire-speed Adding/removing a path reshuffles flows. Equal-Cost](https://reader036.vdocument.in/reader036/viewer/2022070809/5f0789247e708231d41d777e/html5/thumbnails/15.jpg)
How Bird advertise routes
Bird daemonLoad balancer node: 10.1.1.1
directprotocol
1.2.3.1/32 dev lo [direct1 2016-09-19] * (240)1.2.3.2./32 dev lo [direct1 2016-09-19] * (240)
BGP protocol
BGPpeer
loopback interface1.2.3.1/321.2.3.2/32
import routes
export routes
![Page 16: Distributed load balancing...Global Traffic Distribution pavlos.parissis@booking.com. users websiteA ... Load balancing at wire-speed Adding/removing a path reshuffles flows. Equal-Cost](https://reader036.vdocument.in/reader036/viewer/2022070809/5f0789247e708231d41d777e/html5/thumbnails/16.jpg)
Filtering routes for unhealthy services
directprotocol
1.12.3.1/32 dev lo [direct1 2016-09-19] * (240)1.12.3.2./32 dev lo [direct1 2016-09-19] * (240)
BGPprotocol
BGPpeer
loopback interface1.2.3.1/321.2.3.2/32
import routes
exported routes:1.2.3.1/32
filter
routein LIST
LIST= [ 1.2.3.1/32]
anycast-healthchecker
service
![Page 17: Distributed load balancing...Global Traffic Distribution pavlos.parissis@booking.com. users websiteA ... Load balancing at wire-speed Adding/removing a path reshuffles flows. Equal-Cost](https://reader036.vdocument.in/reader036/viewer/2022070809/5f0789247e708231d41d777e/html5/thumbnails/17.jpg)
HAProxy load balancer
● Highly configurable
● Rock solid
● Excellent support
● Supports Lua
● Faster than Nginx in our setup, benchmark yours
![Page 18: Distributed load balancing...Global Traffic Distribution pavlos.parissis@booking.com. users websiteA ... Load balancing at wire-speed Adding/removing a path reshuffles flows. Equal-Cost](https://reader036.vdocument.in/reader036/viewer/2022070809/5f0789247e708231d41d777e/html5/thumbnails/18.jpg)
HAProxy load balancer performance
![Page 19: Distributed load balancing...Global Traffic Distribution pavlos.parissis@booking.com. users websiteA ... Load balancing at wire-speed Adding/removing a path reshuffles flows. Equal-Cost](https://reader036.vdocument.in/reader036/viewer/2022070809/5f0789247e708231d41d777e/html5/thumbnails/19.jpg)
Software and Hardware we use
● Arista switches
● 2 x 10GbE interfaces on servers and 160GbE (4 x 40GbE) on switches
● Bird Internet Routing Daemon http://bird.network.cz
● HAProxy load balancer http://www.haproxy.org
● https://github.com/unixsurfer/anycast_healthchecker
● https://github.com/unixsurfer/haproxystats
● https://github.com/unixsurfer/haproxyadmin
● HP discrete/blade servers
![Page 20: Distributed load balancing...Global Traffic Distribution pavlos.parissis@booking.com. users websiteA ... Load balancing at wire-speed Adding/removing a path reshuffles flows. Equal-Cost](https://reader036.vdocument.in/reader036/viewer/2022070809/5f0789247e708231d41d777e/html5/thumbnails/20.jpg)
We are hiring
Site Reliability Engineers
https://workingatbooking.com