Dual WAN Load Balancing or Policy route with redundant interface

[farhad_plasma]

hi,

I have a situation in my network which there is 2 WAN links and I have to use both of them for internet as described bellow:

servers must use WAN1 primarily which has public ip addresses and serves remote access vpn and other public services,

clients must use WAN2 primarily which does not have public ip address.

both links must failover to the other for internet usage. Also both links receive their ip and gateway from pppoe connection.

So for this implementation I first tried WAN LLB. this implementation works really fine but the problem is that in this situation I lose incoming connections like VPN. I don't know why. I even defined a specific LLB Rule to prefer WAN1 for vpn address range, but again no luck.

The other way crossed my mined is using policy routes. I defined WAN2 default route distance with lower value and defined a policy route saying that all client traffic default route is WAN2. in this situation I have vpn and services working fine but when WAN2 goes down, clients lose internet access because policy route does not track any link state or something else to detect it. If I could write such a track like a router the problem is solved.

or I could find problem related to situation one again problem is solved.

can anyone help me in this situation please?

[tanr]

Your options are a little different if you're running FortiOS 5.2.x or 5.4.x.  What is your version?

To do load balancing with policy routes you need to set the default static routes with the *same* distance, but with different priorities.  That way they both stay in the routing table and the policy route can force you to one or the other interface.

Some documentation:

http://kb.fortinet.com/kb/viewContent.do?externalId=FD32103  http://kb.fortinet.com/kb/documentLink.do?externalID=FD36462  http://kb.fortinet.com/kb/documentLink.do?externalID=100116

You'll also need to set up something to check if the link is down to allow you to failover by removing the route to that interface from the table when it is down.  A discussion of this is https://forum.fortinet.com/tm.aspx?m=139366#139478.  There are somewhat different options for this between 5.2.x and 5.4.x.

Most of this has been discussed in the forums, so you should be able to find more detail with some searches.

[farhad_plasma]

hi tanr

thank you for replying, first I should say that the os is 5.4.

You provided very useful links in your reply. as I understood I should do a policy route without specifying GW and also write same distance default routes with higher priority to WAN1 so that servers and services prefer this route and client choose WAN2 due to the policy route. And when WAN2 goes down clients failover to WAN1. And when WAN1 goes down servers outgoing traffic failover to WAN2. (using this link http://kb.fortinet.com/kb/documentLink.do?externalID=100116)

With this everything seems correct except that fail link detection must be done using link-monitor (as mentioned in this post https://forum.fortinet.com/tm.aspx?m=139366#139478), am I right?

[tanr]

I believe that's it.  I'm off site right now so can't verify my actual config that implements this in 5.4.

There is one difference I ran into that might be 5.4 specific.

In 5.4 I found that I needed to have the policy route specify not just the gateway interface but also the gateway IP. That is, I couldn't leave 0.0.0.0 as the GW IP or it wouldn't properly policy route.

[farhad_plasma]

I think I have felt the same thing though I can't remember exactly but if it's true then how will WAN2 traffic failover to WAN1 when WAN2 is not reachable?

I will test it as soon as possible and share the results.

[tanr]

I believe the static route to WAN2 gets removed from the routing table when the link monitor sees that it is down, making the only viable route the one using WAN1.

Do let us know how the test goes.

[farhad_plasma]

Alright I'm back and I'm really disappointed (and angry). Today I spent 5 precious hours on this useless device and I got no results. I did as I wrote previously. First I wrote the static route which I then I found that routing engine does not insert them into the routing table and I think that the reason is that these connections get their IP addresses dynamically with mask /32, so when I tell the interface not to get GW dynamically and I insert the default routes statically, the route goes to inactive state which I can approve in routing database. So I had to find a way to set the pppoe ip address manually and I couldn't which let me to use dynamic GW. In this method I realized that when pppoe connection goes down the related GW is omitted from routing table too. In this situation I have equal cost routes with same priority.

So I just had to write the policy routes which seems easy as a pie. I wrote 2 policy routes like bellow:

clients ===> WAN2

servers ===> WAN1

with this configuration the policy route works fine but when I disconnect the telephone cable and make WAN1 or WAN2 go down, the traffic does not failover to other WAN link.

So I thought this is one of those many strange behaviors of fortigate that hides many configuration options to be accessible from CLI. I tried to check the policy route from cli and I found a configuration that seemed relevant "set dst-negate enable". I enabled it and tested again. The failover happened but I realized something worse. The policy route now was not working and all traffic flow went through WAN2. I can't understand this strange behavior. Why did it even choose WAN2. Why not WAN1? they are both equal cost.

This useless box is driving me crazy. Even the lowest mikrotik os can perform this simple scenario. Actually I have done this before both with mikrotik and cisco routers, so easy and working well. I just needed UTM features so I chose fortigate but it really disappointed me.

Is it a bug or something? Do you have any idea tanr? 

os detailed version: v5.4.build5335 (GA)

[tanr]

Sorry - that sounds like a frustrating morning.

I'm still off-site, so can't easily check my own config.  My config has gateways with static IPs, and the one pppoe connection is handled by a modem, so my FortiGate isn't directly dealing with any pppoe interfaces.

If you've got the static and policy routes in place I don't see why it wouldn't fail over to the other wan interface.  Some questions you have probably already checked:

- Did you verify before you made one fail that the other route was in the routing table?  

- If so, how did you confirm it wasn't routing?

- Do you have security policies in place to allow both routing solutions?

Beyond that, I'm afraid I don't have any other suggestions.  

Anybody else in the forum have thoughts on this?

BTW, this is the sort of thing that Fortinet can be helpful with.  You might just want to open a ticket with them.

[tanr]

Just one other thought -- is it possible your issue is from PPPoE interfaces having a default distance of 5 instead of 10?    See https://forum.fortinet.com/tm.aspx?m=107557 for a mention of this -- I think it is also mentioned in the KB articles.

[adnan_sabir]

i have exactly the same scenario.. i have one static ip and one normal internet connection. so i want to use static ip for onle remote desktop and vpn services only and want all other traffic through normal internet connection. Actually i want to give access to remote users through static ip so i want to save bandwidth as well om my static ip. please help me out in this regard..