Policy route outbound to 0.0.0.0 and response works fine, but inbound from 0.0.0.0 doesn't

[bhall]

I had a situation arise in which I need to host a service using https on a different interface than WAN1.

Typical setup with the default routing table using WAN1 and that ISP next hop router as the default gateway for 0.0.0.0/0.0.0.0, and all of that has been working fine for years.

Now I have a need to host a new https service, but through an alternate ISP that is connected to a subnet that is routed through the Internal LAN connected to the Internal switch interface on the firewall.

I created a new DMZ using a spare port and created a policy route for the server in that DMZ to send traffic through the Internal interface instead of WAN1. That server gets out fine, and can get Internet with no issues through the alternate ISP. It also shows a specific NAT address correctly that is set up for that ISP connection on whatismyip.com.

So outbound traffic for this server on the new DMZ with the policy route works perfectly. Inbound traffic is an issue.

For remote internal networks that I have defined on the firewall that come in through the Internal interface, I can connect to HTTPS on the server in the new DMZ perfectly. However, anything from 0.0.0.0/0.0.0.0 from the alternate ISP does not work. I do have the firewall rule policy set to allow from source 0.0.0.0 to the server in the new DMZ to allow all HTTPS traffic. But again, anything not defined in the firewall routing table can't connect.

I suspect that the policy route is working with traffic originating from the server in the new DMZ, but when traffic originating from 0.0.0.0, but from Internal rather than WAN1, is being received correctly, but then routed out WAN1 instead of being returned through the Internal interface.

Is this even possible? Or will traffic from undefined networks always be routed out WAN1, even if it comes from Internal and I have a policy route for that DMZ?

[ede_pfau · Accepted solution]

There is a catch with Policy Routing.

PBR does indeed direct traffic _before_ routes in the routing table are looked up. But, and that is the catch, it then relies on the routing table to prevent RPF (reverse path failure, 'anti-spoof') checking to kill that traffic.

In short, with two source networks/ISPs with unknown addresses, you need two default routes. And with Fortigates, there is only one default route per system (that is, per VDOM).

But you can get a second default route into the live routing table, with these parameters:

- create a new static route,

destination: '0.0.0.0/0',

gateway: <ISP2 next hop router address>,

distance: SAME as for your first default route, by default 10,

now 'Advanced', 'Priority': set to 10 (anything > 0).

Net effect: the second default route will be placed into the routing table because it's got the same distance as the first one. Check that in 'Monitor'>'Routing'.

As it's priority (translate: 'cost') is higher, outbound traffic will still use the first default route. But, if you've got a PBR forcing traffic out to the second ISP, RPF check will now be satisfied and reply traffic from unknown sources into the second WAN port will be allowed.

You only need an additionial matching firewall policy to allow outbound traffic.

Please try it out and post back.

[bhall]

Thanks for the detailed info. I'll give this a shot on Monday and see how it goes.

[bhall]

This seems to have done the trick. It's working as intended now, and all original services are still working correctly through the original ISP.

Thanks for the help!

[ede_pfau]

Yay!

[bhall]

I have a follow-up question for this.

Everything, as previously mentioned, is working smoothly, but I've noticed something. I'm sure it's just a result of the 2nd default route and it's priority, but wanted to ask if anything can be done.

When onsite and accessing the new https service from a network connected to the firewall (or defined in the routing table) the https page loads instantly as would be expected.

However, when connecting from an unknown network not defined in the routing table, it takes about 10 seconds for the page to load. I realize that this is probably from the firewall checking the routing table, looking at the default route with more priority, then looking at the route that matches the policy route, then allowing the traffic.

Is there anything that can be done to speed this up, or is that just a side effect of having to use the 2nd default route with a policy route?

[ede_pfau]

Side effect? None that I know of.

10 seconds warrant that you have a look at how and where the packet/request is going, using

diag debug flow

In short:

di de flow show cons en (might be superfluous in v6)

di de flow sh fu en

di de flow sh ip en

di de flow filter saddr <IP of host where request comes from>

di de flow filter proto 6

di de flow filter dport 80            ...to just look at tcp/80 = HTTP

start tracing:

di de flow tra start 10

and initiate the HTTP request

Tracing will stop after 10 packets.

My guess is that traffic bounces between wan1 and wan2 for a while...let's see.

[bhall]

Thanks again. I'll give this a shot and see what comes back.

[bhall]

I was going to do some debugging, but when I got home to do some testing off-site, everything was moving as fast as it should be, but I haven't made any changes or done any research on the firewall at all.

So it was apparently unrelated to the 2nd default route I added. I believe the secondary ISP I'm using to host that service was having some issues that just coincidentally happened after I completed the migration of this host to that connection.

[ede_pfau]

Good to hear your problem is solved. With green glasses on, you see the world in green...been there, done that.