Mikrotik load balancing with PCC port forwarding, ospf and more…

Mikrotik RouterOS is a very popular platform of choice for many enthusiast and SMB networks. Though it is nowhere close to perfect and has a lot bugs you can always try VyOS for x86 deployments. You can read about this here. PCC or per connection classifier is a method of load balancing in Router OS based on connections. However, I have multiple VPN tunnels with my friend’s routers. Some are L2TP/IPSec while others are GRE over IPSec. I run OSPF to learn their routes over VPN tunnels.

Now coming to the problem with PCC load balancing since PCC needs to maintain multiple routing tables, OSPF routes are unusable as they are present in the Main routing table. Port forwarding doesn’t work properly because, as the outgoing connection is randomized between the multiple WAN links it may not be sent out through WAN link from where the incoming request was received. Since the client on the internet had sent request to WAN 1 IP address, it expects the reply to come from the WAN1 IP address but it may be sent from WAN2.

To overcome this problem we will be using some mangle rules. to ensure that WAN 1 incoming requests leave through WAN1 only and WAN2 incoming requests leave through WAN2 only. For the OSPF and connected routes to be bypassed from PCC load balancing we would be using an OSPF script and a mangle rule. Here I am using two ISPs Alliance Broadband Service Private Limited and Srishti Sanchar Webnet Limited. The former is a 100/50 mbps link while the later is a symmetric 100mbps link. We want PCC port forwarding to work along with dynamic routing protocol routes.

Lets set up interface WAN as WAN1 and ether2 as WAN2. We have three local network interfaces LAN, WLAN, ether5. The interface names with -PoP are GRE tunnel interfaces.

/ip address
add address=192.168.254.1 interface=loopback network=192.168.254.1

add address=172.22.146.1/26 interface=LAN network=172.22.146.0

add address=172.22.146.65/26 interface=WLAN network=172.22.146.64
add address=192.168.168.21/30 interface=Dumdum-PoP network=192.168.168.20

add address=192.168.168.5/30 interface=Madhyamgram-PoP network=192.168.168.4

add address=192.168.168.9/30 interface=Bally-PoP network=192.168.168.8

add address=192.168.72.1/23 interface=ether5 network=192.168.72.0

add address=10.28.115.18/30 interface=WAN network=10.28.115.16

add address=172.28.55.3/24 interface=ether2 network=172.28.55.0

We create an Interface list ‘lans’ and add interfaces LAN and WLAN as we only want to loadbalance traffic on these interfaces. Ether 5 is connected to my Proxmox server and we want all its traffic to use only WAN1.

/interface list member
add interface=WLAN list=lans

add interface=LAN list=lans

Create NAT rules for both WAN links.

/ip firewall nat
add action=masquerade chain=srcnat out-interface=WAN

add action=masquerade chain=srcnat out-interface=ether2

Next we create a script to automatically create an address list ‘ospf’ with routes learnt over OSPF. It checks for any updates every 60 seconds.

:do {
:foreach i in=[/ip route find ospf] do={
:local network [/ip route get $i dst-address];
/ip firewall address-list remove [/ip firewall address-list find where timeout=00:00:00];
/ip fire addr add list=OSPF address=$network timeout=00:01:00;
}
:delay 60s;
} while=(true)

We now create an address list for connected routes under /ip firewall address-list with the name connected with our lan interface networks. This will be used for bypassing inter-lan connections from PCC load balancing process.

Now we add rules to mark connections to ospf routes and connected routes and route them only via main routing table.

/ip firewall mangle
chain=prerouting action=mark-connection new-connection-mark=vpn_conn passthrough=yes connection-state=established,related,new dst-address-list=OSPF in-interface-list=lans
chain=prerouting action=mark-connection new-connection-mark=vpn_conn passthrough=yes connection-state=established,related,new dst-address-list=connected in-interface-list=lans
chain=prerouting action=mark-routing new-routing-mark=main passthrough=yes connection-mark=vpn_conn

Now we add mangle rules for making incoming connections to leave only via the same WAN interface.

chain=input action=mark-connection new-connection-mark=abspl_out_conn passthrough=yes in-interface=WAN
chain=output action=mark-routing new-routing-mark=abspl_out_traffic passthrough=no connection-mark=abspl_out_conn
chain=input action=mark-connection new-connection-mark=sswl_out_conn passthrough=yes in-interface=ether2
chain=output action=mark-routing new-routing-mark=sswl_out_traffic passthrough=no connection-mark=sswl_out_conn
chain=forward action=mark-connection new-connection-mark=abspl_out_pfw passthrough=no connection-state=new in-interface=WAN
chain=prerouting action=mark-routing new-routing-mark=abspl_out_traffic passthrough=no connection-mark=abspl_out_pfw in-interface-list=lans
chain=forward action=mark-connection new-connection-mark=sswl_out_pfw passthrough=no connection-state=new in-interface=ether2
chain=prerouting action=mark-routing new-routing-mark=sswl_out_traffic passthrough=no connection-mark=sswl_out_pfw in-interface-list=lans

Now we add mangle rules for the PCC which is very straight forward:

chain=prerouting action=accept in-interface=WAN 


chain=prerouting action=accept in-interface=ether2

chain=prerouting action=mark-connection new-connection-mark=abspl_conn passthrough=yes dst-address-type=!local connection-mark=!vpn_conn per-connection-classifier=src-address-and-port:2/0 

chain=prerouting action=mark-connection new-connection-mark=sswl_conn passthrough=yes dst-address-type=!local connection-mark=!vpn_conn per-connection-classifier=src-address-and-port:2/1


chain=prerouting action=mark-routing new-routing-mark=to_abspl passthrough=yes connection-mark=abspl_conn in-interface-list=lans 
chain=prerouting action=mark-routing new-routing-mark=to_sswl passthrough=yes connection-mark=sswl_conn in-interface-list=lans 

Now we add the routes into the routing table:

/ip route
add distance=1 gateway=10.28.115.17 routing-mark=abspl_out_traffic
add distance=1 gateway=172.28.55.1 routing-mark=sswl_out_traffic
add check-gateway=ping distance=1 gateway=10.28.115.17 routing-mark=to_abspl

add check-gateway=ping distance=2 gateway=172.28.55.1 routing-mark=to_sswl

add check-gateway=ping distance=1 gateway=10.28.115.17
add check-gateway=ping distance=2 gateway=172.28.55.1

Now we just normally configure OSPF and GRE tunnels. After that we can easily enjoy load balancing as well as access VPN routes over OSPF. I would like to thank Sayed Jahanzaib for the port forwarding mangle rules.