De-configuring Dual WAN Load Balancing advice

[3pointD]

Hello,

I've been running dual wan connections using ECMP Weighted Load Balance on my FWF80CM (v4 MR3 Patch 15) for a couple of years now. The solution has worked out pretty well but now I want to change this design to a more manual setup to improve control over how traffic flows through the device.

My goals:

WAN2 - Primary Gateway

WAN1 - Failover Gateway

Needs:

I will still need to Policy route some specific traffic out WAN1

I will still need to access VIPs setup on WAN1 externally

I will still need to access HTTPS admin on WAN1 externally

To start off I assumed that I could just change ECMP back to Source IP (default) and then give WAN1 administrative distance higher than WAN2. Both WAN1 and WAN2 have static routes with the same distance currently. When I did this working remotely over HTTPS on WAN1 as soon as I applied the Admin Distance on WAN1 I lost my remote connection and found that I was also unable to access HTTPS admin on WAN2. In addition when I arrived at the office HTTP/HTTPS traffic was not working. I could ping external addresses (I think) but DNS wouldn't resolve. After changing the Distance back to 0 on both interfaces and a reboot of the fortigate HTTP/HTTPS started to flow again.

I have an internal DNS server and do not use the fortigate. Can anyone tell me what might have happened. Do I need to clear the routing table or cache after changing these settings? Should I be using Priority instead of Distance? How did this affect my inbound HTTPS admin session? How does this affect inbound external traffic?

I don't know a whole lot about routing aside from what I've read in the fortigate manual and here in the forums so be gentle.

Thanks 

[obfuscated · Accepted solution]

The distance variable dictates what routes go into the routing table and only the best (or the equal best in the case of ECMP) will go into the routing table.  By changing this you have removed one of the routes.

The Priority is the next consideration so if there are multiple routes in the routing table (ie same distance) it will use the priority to select which one it chooses to send traffic to.

Then we are down to the load balancing of the ECMP ie based on source address or a weighting.

I am not sure what version you are using but version 5.2 has a 'WAN link interface' feature which allows you to more easily bundle the interfaces and may be worth exploring:-

http://www.google.co.uk/u...;bvm=bv.96041959,d.ZGU

[3pointD]

Hi, Anyone have any insight on this? Any hints greatly appreciated!!

Thank you

[obfuscated · Accepted solution]

The distance variable dictates what routes go into the routing table and only the best (or the equal best in the case of ECMP) will go into the routing table.  By changing this you have removed one of the routes.

The Priority is the next consideration so if there are multiple routes in the routing table (ie same distance) it will use the priority to select which one it chooses to send traffic to.

Then we are down to the load balancing of the ECMP ie based on source address or a weighting.

I am not sure what version you are using but version 5.2 has a 'WAN link interface' feature which allows you to more easily bundle the interfaces and may be worth exploring:-

http://www.google.co.uk/u...;bvm=bv.96041959,d.ZGU

[3pointD]

Hey, thanks for the reply.

Both WAN interfaces currently have static entries. If I remove one of the static routes will that effectively turn ECMP off?

I understand that 5.2 does have some better features to deal with multiple WANs but I've got an older FWF80CM. From what I've heard the newer versions can be rough on these older, lower end units.

[obfuscated · Accepted solution]

Removing the route will turn the ECMP off as the box will only send traffic out one of the WAN links.

For the outbound traffic you could retain the routes and distance and add 'priority'  This would use the WAN2 for all traffic unless the interface went down at which point WAN1 would be used.  Adding policy based routing can then  direct certain (assuming important traffic) out WAN1.

The consideration here is that by changing this you have traffic coming in via WAN1 for your externally presented systems and then routed out via WAN2 as its the default route.  This could then result in another walk/drive into the office if you try doing the changes remotely (We have all done it at least once).  Although if you have been using ECMP for sometime then the stateful firewall has clearly handled this successfully.

As for 5.2 on your box I would agree :)

[3pointD]

Yes, the configuration you mention was my second attempt. I did that on site.

1. I changed ECMP Load Balancing Method to "Source IP based"

2. Set the Priority of WAN1 static route to 5 (WAN2 is still 0)

3. Change the Weight values on both WAN interfaces back to default 0

After those changes I lose internet access. I tried rebooting the Fortigate to no avail. Is there a cache I need to clear?

Thanks very much for the input! 

[obfuscated]

Some better visibility may be an idea.  Is it possible to run the following to see whats going on now?

show router policy

get router info routing-table database   (All Potential Routes)

get router info routing-table all  (Best Routes which are injected into the Routing Table)

[3pointD]

Everything those commands return is exactly what I see under my Routing Policies and Routing Monitor in gui. Routing Table database and all return the same values. The only thing that looks strange to me is that WAN1 doesn't have S next to it identifying it as Static but I've never looked at the routing table in console before so it might be normal...

S *> 0.0.0.0/0 [10/0] via XX.XX.XXX.XXX, wan2, [0/255]    *> [10/0] via XXX.XX.XX.XX, wan1, [0/50] C *> 10.0.0.0/24 is directly connected, internal2 C *> 10.0.1.0/24 is directly connected, Wifi3 C *> 10.1.17.0/24 is directly connected, wifi C *> 10.1.18.0/24 is directly connected, wifi2 C *> 10.1.19.0/24 is directly connected, Wifi4 C *> XX.XX.XXX.XXX/28 is directly connected, wan2 C *> 192.168.15.0/24 is directly connected, internal4 C *> 192.168.16.0/24 is directly connected, dmz C *> 192.168.17.0/24 is directly connected, internal1 C *> XXX.XX.XX.XX/28 is directly connected, wan1

Also in the routing-table all I see an * for WAN2 which indicates "candidate default". Would this be because it is currently weighted higher.

I'm positive I read that with ECMP new connections go to the same existing destination will always be routed through the same interface regardless of Weight, Spillover .etc. Where would I see or clear this?

[obfuscated]

This looks like the 'database' command based on the *'s and >'s

S *> 0.0.0.0/0 [10/0] via XX.XX.XXX.XXX, wan2, [0/255]    *> [10/0] via XXX.XX.XX.XX, wan1, [0/50]

So you have two routes out two interfaces with the same Distance and Metric [10/0] ).   The Priorities are the same and the weighting approx 5:1  [0/50] and [0/255]  This means that they are both in the routing table and ECMP is being used to weight the traffic accordingly.

If you change the distance as per your first post then the WAN1 route will be removed from the routing table hence why 'stuff broke'

This clearly works as long as you are happy with the 5:1.  I guess the next step based on your 'needs' is to policy route traffic out WAN1, have you got any Policy routes in place now?  We have to remember that these override the routing table as they are checked before the routing table.

Ob

[3pointD]

Hi Sorry I was out on vacation for a bit.

OK yes I understand everything you mentioned in your last post. Moving forward I do not want to keep the ECMP Weighting. I want to have WAN2 as primary and WAN1 as failover. My understanding is that I need to set ECMP back to Source IP Based, change the Weight to 0 on each interface and then set Distance of WAN1 to a higher value than WAN2. Dead Gateway Detection is set up to ping both WAN gateways.

My problem is that as soon as do all of this internet access seems to stop on both WAN interfaces. At this point I'm not touching Policy routes. What tests can I run to figure out what is happening?

Thanks a bunch for the help!