Dual Physical Wan Interfaces (2 ISPs) with Multiple IPs per Interface – SNAT and Failover

[tanr]

Hi all,

I’ve been searching through the forums and docs without finding a full answer to this, so thought I’d post.

I have a 300D (v5.4.1) using two physical interfaces to connect to two different ISPs. 

For each physical interface I’ve got a block of IPs.

I can’t use more physical interfaces for these wan connections, and I don’t have an L2 switch between the FGT and the ISP.

The internal network has a number of segregated lans and vlans.

For simplicity, I’ll call the physical wan ports wan1 and wan2, and the IPs on them wan1.1, wan1.2, wan2.1, wan2.2, etc.

In normal circumstances I need to route specific source subnets out using a specific ISP and a specific IP within my block of IPs for that ISP.  For example, the source vlan hw-test is routed out wan1, SNAT’d to wan1.3, the source vlan seclan is routed out wan2, SNAT’d to wan2.1.

I also map some of the specific wan facing IPs to internal servers.  For example, say IP wan2.2 maps to one of my web servers.

This is all doable with multiple IP Pools for SNAT (turning off ARP reply for security) used by multiple security policies for routing out, and VIPs to DNAT the connections from the wan to my servers.  No server load balancing being done.

MY PROBLEM / QUESTION:

Given the scenario, what is the best method to handle failover? 

I’m not interested in load balancing, just failover.  I’ve found ways to do it – just none that I’m happy with.

My requirements for failover make this more complicated.  I only want to support failover for certain of my subnets. 

For example, source seclan should failover from using wan2 (SNAT wan2.1) to wan1 (SNAT wan1.1), but hw-test should never failover.

Solutions I’ve looked at so far: 

[ol]

WAN Link Load Balancing (WAN LLB) isn’t an option because it only works with physical interfaces. I suppose this could be an option if I added a FortiSwitch between the FGT and the ISP to give me more physical interfaces – Don’t really want to do that.

I don’t think ECMP (Equal Cost Multiple Path) gives me the source based control I want, its “source-based” method is just to use a hash of the source to choose an interface.  Also, the docs say it’s not recommended with 5.4.1 (not sure why) and GUI access to it has been removed.

Multiple VDOMs containing the various vlans/subnets, allow me to specify routes with the same distance but different priorities.  This does give me simple static routes since each VDOMs would only have the source subnets/vlans I want.  But I’d rather not move to multiple VDOMs if I don’t really need to!

Policy routes can route to wan1 or wan2 based on source.  If a policy route won’t apply when an interface is down (can anyone confirm?) then I could still use IP pools in the security policies to do the SNAT – I’d just need two versions of the security policy for each source subnet/vlan that supported failover, so that the security policy could apply the correct SNAT depending on if it was going out wan1 or wan2.  This adds many more security policies.[/ol]

Am I missing something?  I’d love a cleaner solution than the four above.

[ede_pfau]

I'd opt for option 3. Your requirements and the effort you're willing to make are special, as is this solution.

Policy based routing is explicitly no option here, as PBR are not switched off if the egress interface goes down. I'd love to see that they'd be treated like static routes (route is dropped if interface is down) but they aren't.

[tanr]

Ouch, didn't realise that policy routes didn't go down in that case.

One other negative with VDOMs is a small note in the new 5.4.1 docs: Cooperative Security Fabric (CSF) firewalls do not support multiple VDOMs (365260).

How about this possibility:

My understanding is that if a policy route specifies an output-device, but doesn't specify a gateway, that routing then goes to any static route that has that destination interface (per case 5 in http://kb.fortinet.com/kb/viewContent.do?externalId=FD32103).  Maybe that gives me a somewhat different option?

I can adjust the setup so that all sources that would normally go out wan2 are the same that I don't want to failover from wan2 to wan1, and only some of the sources that would normally go out of wan1 should not failover to wan2.

Then I set up two default static routes for wan1 and wan2, with the priority for wan2 being a higher number

config router static   edit 1     set device wan1

    set distance 10     set priority 8     set gateway 50.50.50.2   next   edit 2

    set device wan2     set dist 10     set priority 20     set gateway 30.30.30.3   end

Then I set up policy routes, JUST for the sources I don't want to fail over.

The policy routes will specify an output-device of wan1 or wan2, appropriately, but with no gateway IP set.  There would be policy routes for all sources that normally go out wan2.  This way, the policy routes force those sources that shouldn't fail over to be forced to the interfaces static route, even if that interface is down.

Or am I just being overly optimistic here?

[tanr]

Some more information after a few tests.

Scenario 5 from the documentation I quoted above, which says that a policy route that specifies an output interface but no gateway IP seems to be incorrect for 5.4.1 at least.  Without a gateway IP specified, the policy routes don't override the static routes that have a higher priority (with the same distance).  This is supported by a different KB article, http://kb.fortinet.com/kb/documentLink.do?externalID=FD36462, which contradicts the first one I quoted with the statement: 

  "Policy routes can route traffic out independently of the routing table only if the gateway is defined."

Interestingly, even with a gateway IP specified, if the policy routes destination interface is disconnected (just unplugged - I didn't try disconnecting farther up the chain), the policy route doesn't route.  So it appears (though more testing is needed to verify) that 5.4.1 policy routes don't route to disconnected interfaces. 

That means I can't use the method I was hoping to use from my second post, though I probably could put in multiple policy routes for the same sources (since they're evaluated in order) to redirect as needed.  That would be too many policy routes!

Checking the rather scant documentation on 5.4.1 WAN Link Load Balancing (LLB) from http://help.fortinet.com/fos50hlp/54/Content/FortiOS/fortigate-advanced-routing-54/whatsnew54.htm the new WAN LLB Priority Rules (not load balancing algorithm) allow specifying an outgoing interface based on source address.  I thought that combined with health checks (or NOT doing health checks) this might work for my case.  However, the effect of a disconnected dst interface was the same as with the policy routes.  Even though I had only given a single wan interface that my source was supposed to get routed to, when the interface was disconnected the WAN LLB stopped having any effect and packets from the source were routed out the other wan interface.

So, unless anybody has other suggestions, I’ve got:

[ol]

Neither WAN LLB with Priority Rules based on Source nor Policy Based Routing solve the problem of failover routing of only some sources by themselves.  I suppose I could make them work by having additional security policies that would drop packets from sources that got incorrectly re-routed.  Kind of complicated, messy, and not nice to support.

Multiple VDOMs can solve this for me.  I’d rather not do VDOMs if I don’t need to, especially with the odd documentation warning Cooperative Security Fabric firewalls don’t support multiple VDOMs.  Probably the cleanest solution.

I could add an additional small FortiGate (60D) for the vlans that need to be kept off the other wan.  They don’t need VPN access or shared resources anyway.  Cost, rack space, more cables, and moving over some config by hand.[/ol]