routeros multi-wan loadbal

7/22/2019 RouterOS Multi-WAN LoadBal

1/128

Load Balancing Using

PCC & RouterOS


2/128

About Me

Steve Discher, from College Station, Texas, USA

Class of 87 Texas A&M University

Using MikroTik since early 2004 when I started my first WISP Author of the book RouterOS by Example

MikroTik Certified Trainer and teach RouterOS classes,MyWISPTraining.com

Operate a wireless distribution company, ISPSupplies.com


3/128

1. What is load balancing

andwhy would I want it?


4/128

2. Which method should Ipick and how does it work?


5/128

3. Ok, I want it but how do Iset it up?


6/128

Typical Scenario Requiring Load BalancingProblem: No high capacity circuits available, DSL only

Distribution: Fiber, Copper, Wireless, etc.

Hotel, Apartments, etc.


7/128


Distribution: Fiber, Copper, Wireless, etc.

Typical Scenario Requiring Load BalancingSolution: Multiple low capacity circuits, RouterOS load balancing


8/128

1. What is load balancing and why would I

want it?


9/128


want it?

Process to utilize multiple internet connections in such amanner as to proportionately distribute internet traffic across

all the connections.


10/128


want it?


all the connections. Distribution may be symmetrical or asymmetrical dependingon circuit availability.


11/128


want it?


all the connections. Distribution may be symmetrical or asymmetrical dependingon circuit availability.

Useful when the downstream bandwidth requirement to asingle routing device exceeds the capabilities of a single

internet circuit.


12/128

Load Balancing in General

Options Available


13/128


First, the type of load balancing we are discussing today shouldnot be confused with any type of bonding protocol or sub-packet

based load balancing.

Options Available


14/128



based load balancing. Bonding, MLPP, etc. require that the protocol be recognized onboth the subscriber and provider ends. Not available withcommodity internet connections.

Options Available


15/128




Cant simply bridge two DSL or Cable modem connections,

doesnt work.

Options Available


16/128




Cant simply bridge two DSL or Cable modem connections,

doesnt work.There are several methods to provide load balancing inRouterOS.

Options Available


17/128

Example


18/128

Options Available

Load Balancing Options With RouterOS:


19/128

ECMP - Equal Cost Multi-Path Routing

Options Available



20/128

ECMP - Equal Cost Multi-Path Routing Per-address pair load balancing

Options Available



21/128

ECMP - Equal Cost Multi-Path Routing Per-address pair load balancing Doesnt work well for certain protocols, connections break when routingtable flushes every ten minutes to prevent DOS attacks

Options Available



22/128


Nth Load Balancing - Per connection load balancing, with the additionof persistent connections

Options Available



23/128



Hybrid / Custom Setups - Solutions based on one or more methodsabove with the addition of scripts or policy routing to make the solutionmore intelligent.

Options Available



24/128




Bandwidth based load balancing - MPLS, Traffic Engineering, etc.

Options Available



25/128




Bandwidth based load balancing - MPLS, Traffic Engineering, etc.PCC - Per Connection Classifier

Options Available



26/128




Bandwidth based load balancing - MPLS, Traffic Engineering, etc.PCC - Per Connection Classifier Simple, effective, scalable, no nasty side effects

Options Available



27/128




Bandwidth based load balancing - MPLS, Traffic Engineering, etc.PCC - Per Connection Classifier Simple, effective, scalable, no nasty side effects Per-address pair load balancing method

Options Available



28/128

2. How does it work?


29/128


PCC divides the incoming data into streams and then usesrouting rules to sort the traffic evenly (or not) across multipleWAN connections.


30/128



This is done by:


31/128



This is done by:1. Using a hashing algorithm to first sort the traffic based onsource address, source port, destination address, destinationport or various combinations thereof.


32/128




2. Using packet marking and routing marks and severalrouting tables to ensure traffic follows a specified route outthe specified WAN interface.


33/128




2. Using packet marking and routing marks and severalrouting tables to ensure traffic follows a specified route outthe specified WAN interface.


34/128

Understand the Solution


35/128


MikroTik RouterOS is extremely powerful and configurable,so this can be a double edged sword, several possible

solutions to the same problem


36/128



solutions to the same problem Each has multiple moving pieces


37/128



solutions to the same problem Each has multiple moving pieces Greatest success with any solution by understanding thepieces and what they do.


38/128

Understanding the PCC Load Balancing Solution

Terms...


39/128


1. Packet - The container for our data, header and payload.

Terms...


40/128



2. Connections - Conduit through which host to hostcommunication occurs, based on Src/Dst addresses and ports

Terms...


41/128



2. Connections - Conduit through which host to hostcommunication occurs, based on Src/Dst addresses and ports3.Mangle Facility- Firewall function within RouterOS that

allows you to create a mark which is then associated with packetsthat can be identified later by other functions like firewall rules or

routing tables.

Terms...


42/128



43/128


4. PCC - Per Connection Classifier, function contained with theMangle Facility to sort traffic into streams


44/128


4. PCC - Per Connection Classifier, function contained with theMangle Facility to sort traffic into streams

5. Routing Table - Route rules, the rules the router uses todetermine what to do with a packet. By comparing the destinationaddress in the packet to the list of routes, the router decideswhich interface to send the packet out. By adding a routing markwith mangle, we can have multiple routing tables!


45/128


A packet is like a letter & envelope.The front is the header and the letter inside the envelope is thepayload.

Source Address& Port

Destination Address& Port

1. What is a packet?


46/128


IPv4 HeaderSource Address

(sender)

Destination Address

(receiver)

Port Port

Protocol


47/128

2. What are connections?



48/128

3. What is the mangle facility?



49/128

4. What is PCC?

Per Connection Classifier is a mangle option that sortsdata into streams that can be marked for identification later.


(s)

Unsorted In Sorting Sorted Streams


50/128

Where is it found?


1

2

3


51/128

How does PCC work?


"PCC takes selected fields from IP header, and with the help of a hashingalgorithm converts selected fields into 32-bit value.

This value then is divided by a Denominator and the Remainder then iscompared to a specified Remainder, if equal then packet will be captured.

You can choose from src-address, dst-address, src-port, dst-port (or variouscombinations) from the header to use in this operation."


52/128

PCC uses a hashing algorithm.


A hashing algorithm is a mathematical function that takesan input and returns an output.

The output will always be the same for a specified input.

Example of a simple hash:Input x 100 = hash value


53/128

PCC uses modular arithmetic (clock arithmetic).


Numerators, Denominators and Remainders are parts of modular arithmetic.

It is represented by a% sign and it is spoken as mod.To work modular math, think of it as "how many are left over (Remainder) afteryou've subtracted the second value (Denominator) from the first (Numerator) as

many times as possible without going negative?"

Here are some examples of modular math:

Numerator =3 Denominator = 3

3% 3 = 0 because 3 - 3 = 0 left overor4% 3 = 1 because 4 - 3 = 1 left over5% 3 = 2 because 5 - 3 = 2 left over6% 3 = 0, because 6 - 3 = 3, subtract 3 again = 0 left over


54/128

Example: 2 WAN Connections


The first line means "produce the output of the hash functiongiven the packet's source IP address, divide it by 2 and if theremainder is 0, perform the action of marking the connection asWAN1".

The second line means "produce the output of the hash functiongiven the packet's source IP address, divide it by 2 and if theremainder is 1, perform the action of marking the connection asWAN2".

Denominator Remainder

Modular math helps us understand how to create the PCC rules!

2 PCC Rules Required


55/128

How to set PCC, Remember:


2 WAN connections:

2 / 0 First WAN

2 / 1 Second WAN3 WAN connections:

3 / 0 First WAN

3 / 1 Second WAN

3 / 2 Third WAN and so on...


56/128

5. What is a routing table?



57/128

Details of a route, key pieces are destination and gateway.



58/128

Multiple routing tables with route marks



59/128




60/128




61/128




62/128


Review:


63/128


1. Packet - Container for IP data

Review:


64/128


1. Packet - Container for IP data2. Connections - Bi-directional conduit for communication

between two hosts

Review:


65/128



between two hosts3. Mangle Facility - Manipulates packets by adding marks

Review:


66/128




4. PCC - Divides data into streams (based on marks)

Review:


67/128




4. PCC - Divides data into streams (based on marks)5. Routing Table - List of route rules to direct packets and wecan have multiple tables based on routing marks

Review:


68/128

3. Ok, I want it but how do I set it up?



69/128

3. Ok, I want it but how do I set it up?

Scenario: One router, many clients, three DSL connections



70/128

Step by Step Configuration

Test Setup


71/128


1. Set up the basic portion of the network (MTCNA, Wiki, etc): Private IP address on LAN interface DHCP Server on LAN interface DNS server Static IP for WAN or DHCP client on WAN

Firewall if required


72/128




2.Create load balancing part of the configuration:

Mangle rules

Routing tables


73/128





Mangle rules

Routing tables


74/128


Physical interface connections

ether1 ether2 ether3 ether4 ether5


75/128


IP Addresses

DHCP Client - WAN

Masquerade Rules

DNS Client & Caching

1.Set up the basic portion of the network (MTCNA, Wiki, etc):


76/128



Create various mangle rules to mark connections Create mangle rules to associate routing marks with packetsbased on their connection mark.

Create routes to send traffic out the WAN connections in apredetermined manner.


77/128


Step 1: Create some accept rules.

We have to manually force local traffic to connected

networks to stay in the main routing table.

Background - Any subnet for which the router has an IPaddress configured is called a connected network, meaningpackets to that network are sent out an interface and can reach

their destination without using another router to get there.


78/128


Connected network example

The problem using mangles here is it willforce traffic to follow alternate routing tables(not main)Traffic to these connected networks would

go out the WAN interfaces and not reachtheir intended destinations.

Step 1 continued...


79/128


Step 1 continued...

Solution:

The accept action causes the packet to leave the mangle chain, therebynot marking it and allowing that traffic to use the main routing table.


80/128


One rule for each connected network, in this example these areour WAN networks

Step 1 continued...


81/128


/ip firewall mangle

add action=accept chain=prerouting disabled=no dst-address=172.17.0.0/24



Step 1 Completed


82/128


Create the PCC mangles:

We will use optimal mangle method of marking connections firstand then packets because it is the most efficient way to marktraffic, uses least resources.

First identify traffic and mark the connection.

Second, look for that connection mark and mark the routes.

Step 2: Create Mangle rules that will sort the traffic intostreams.


83/128


Step 2 Continued...


84/128


Step 2 Continued...


85/128


Step 2 Continued...


86/128


Step 2 Continued...


87/128


Step 2 Continued...


88/128


Step 2 Continued...


89/128


Step 2 Continued...


90/128


Step 2 Continued...

S S C fi i


91/128


ip firewall mangle

add action=mark-connection chain=prerouting connection-mark=no-mark disabled=no dst-address-type=!local \in-interface=ether5 new-connection-mark=WAN1 passthrough=yes per-connection-classifier=both-addresses:3/0

add action=mark-connection chain=prerouting connection-mark=no-mark disabled=no dst-address-type=!local \

in-interface=ether5 new-connection-mark=WAN2 passthrough=yes per-connection-classifier=both-addresses:3/1

add action=mark-connection chain=prerouting connection-mark=no-mark disabled=no dst-address-type=!local \

in-interface=ether5 new-connection-mark=WAN3 passthrough=yes per-connection-classifier=both-addresses:3/2

Step 2 Completed

S b S C fi i


92/128


Step 3: Create the mangles to add the routing marks to the packets based onthe connection mark in the PREROUTING CHAIN:

St b St C fi ti


93/128



St b St C fi ti


94/128



St b St C fi ti


95/128



St b St C fi ti


96/128


Step 3 Continued for OUTPUT CHAIN...

St b St C fi ti


97/128





98/128





99/128





100/128


Step 3 Completed

/ip firewall mangle

add action=mark-routing chain=prerouting connection-mark=WAN1 disabled=no in-interface=ether5 \

new-routing-mark=ether1-mark passthrough=yes





add action=mark-routing chain=output connection-mark=WAN1 disabled=no new-routing-mark=ether1-mark passthrough=yes





101/128


Step 4: Identify which WAN interface the traffic came in andmark the connections appropriately.



102/128





103/128





104/128


Step 4 Completed

/ip firewall mangle

add action=mark-connection chain=prerouting connection-mark=no-mark disabled=no \

in-interface=ether1 new-connection-mark=WAN1 passthrough=yes







105/128


Final result: Connections should be marked, route marks added topackets based on connection mark.



106/128


Mangles are done, we now create the routes:



107/128


We will need one default route for each routing mark,

corresponding to each of the WAN connections.




108/128


We will need one default route for each routing mark,

corresponding to each of the WAN connections.We will also need one unmarked default route corresponding toeach of the WAN connections.




109/128


Step 5: Create the unmarked default routes.



110/128





111/128





112/128





113/128





114/128





115/128


Step 6: Create the marked default routes.



116/128

p y p g




117/128

p y p g




118/128

p y p g




119/128

p y p g




120/128

p y p g

Final result - Routing table



121/128

p y p g

Final result!Actual screen shots from a load balance configuration inproduction with 2 WAN connections.

Common Problems


122/128

I use DHCP for my WAN addressing, how can I get the marked

routes created properly?/system script

add name=ConfigureDHCPRoutes policy=\

ftp,reboot,read,write,policy,test,winbox,password,sniff,sensitive,api source=":local cli\

entcounter\r\

\n:local routecounter\r\

\n:local duplicatecounter\r\

\n:local routeupdated \"no\"\r\

\n:foreach clientcounter in=[/ip dhcp-client find] do={\r\

\n:local routingmarkname ([/ip dhcp-client get \$clientcounter interface] . \"-mark\")\r\\n:local newroutinggateway [ip dhcp-client get \$clientcounter gateway]\r\

\n:foreach routecounter in=[/ip route find where routing-mark=\$routingmarkname] do={\r\

\n :local routinggateway [/ip route get [find routing-mark=\$routingmarkname] gateway]\

\r\

\n\t:if ([:len \$newroutinggateway] > 0) do={\r\

\n\t :if (\$\"routinggateway\" != \$\"newroutinggateway\") do={ \r\

\n\t /ip route set \$routecounter gateway=\$newroutinggateway \r\

\n\t\t:set routeupdated \"yes\"\r\

\n\t }\r\

\n\t}\r\

\n } \r\\n :if ([:len \$newroutinggateway] > 0) do={\r\

\n :if (\$routeupdated = \"no\") do={\r\

\n /ip route add routing-mark=\$routingmarkname gateway=\$newroutinggateway dst-add\

ress=0.0.0.0/0\r\

\n }\r\

\n }\r\

\n}\r\

\n"

Script by Andrew Cox

Common Problems


123/128

Before Running Script

After Running Script

Common Problems


124/128

PCC doesnt seem to work properly with HotSpot or IP

Webproxy

It is possible to make it work but the rules get very detailed andcomplicated.

Solution: Use two routers, one for load balancing, one for HotSpotor IP Webproxy.

Solution: Use metarouter with host router doing main routingfunctions, virtual router doing the load balancing.

Common Problems


125/128

DNS resolves from some clients, not others

If you are using two different ISPs and their respective nameservers, possibly some clients are accessing ISP1s DNS server

through ISP2s connection and ISP1 is blocking DNS requests fromoutside their IP space.

Solution: Consider OpenDNS, destination NAT with redirect toDNS cache, etc.

Common Problems


126/128

Strange http issues, some images load, other dont, problems withsome secure sites

Solution: Try using both addresses or source address for PCCclassifier. While both addresses and ports gives the greatestchance for randomization and better possibility for evendistribution, it can create these types of issues.

Common Problems


127/128

I can only get asymmetrical connections, one DSL and one cablemodem.

Solution: You can weight one interface higher and force more trafficthrough it by repeating the connection marking PCC rule more than oncefor that connection.

Example for added weight to WAN3.

/ip firewall mangle

add action=mark-connection chain=prerouting connection-mark=no-mark disabled=\

no dst-address-type=!local in-interface=ether5 new-connection-mark=WAN1 \

passthrough=yes per-connection-classifier=both-addresses:3/0










Thank You!


128/128

MyWISPTraining.com

LearnMikroTik.com ISPSupplies.com

RouterOS by Example available formany distributors or Amazon.com, iTunes

routeros multi-wan loadbal

Documents