Remote management was working

[rc42]

The firewall has a Primary and Secondary internet connection.

The Primary(wan1) is PPOE and IP/DNS/GW are automatically assigned (the IP is static, but is assigned via PPOE). The default route assigned is given a distance of 5.

The Secondary(wan2) is a basic static address.

I had 2 static 0.0.0.0 routes configured, one on wan1 pointed to that GW with a distance of 10, the other on wan2 pointed to it's GW with a distance of 15.

The active route was the one created by the PPOE (obviously), with the distance of 5. The route monitor shows this as having an interface of ppp1.

I have two local-in-policy that allows https in to the wan1 and wan2 ONLY from my other site, to allow easy remote management.

I'm using link-monitor to fail-over from wan1 to wan2 (no I don't want to use SD-WAN). It is configured with the wan1 interface and GW.

The remote access was working as expected. But the fail-over was not. Apparently, if traffic on wan1 stops but the PPOE stays up, the dynamically assigned PPOE default route with GW x.x.x.x on ppp1 with distance 5 stays active.

This makes sense, sort of, since the link monitor is tied to the static default route on wan1 with it's distance of 10.

So I changed the static default route for wan1 to a distance of 1, putting it ahead of the PPOE assigned default route. I also changed the secondary default route for wan2 to a distance of 3, also putting it ahead of the PPOE route.

So now the route monitor shows the static wan1 default route as active under normal conditions. And if the link-monitor goes down the static wan2 route becomes active. (I suppose if wan2 also goes down the PPOE router would again become active, but at that point who cares, there is no internet anyway at that point).

But now that this is all working, the remote management won't connect.

I can't think of anything I changed that would affect this, but I change the distance from 10 to 1 and my remote management session disconnected. The site is up and everything else is working but no remote management.

Any ideas?

[Toshi_Esumi]

First, I would recommend "set defaultgw disable" on wan1 to stop injecting pppoe default route so that you don't fight against it by tweaking distance on those two static default routes you care.

Then probably you need to post your local-in-policies (CLI) on both wan1 and wan2, of course, after masking the public IPs so that anyone esle can guess and comment what might be going on.

[rc42]

I'm little nervous about changing the defaultgw setting remotely, this may have to wait until I'm back on-site. The reason is the the PPOE is creating 2 "connected" entries in the routing table. One is xxx.xxx.xxx.1/32 on ppp1 which is the address of the gateway, and the other is xxx.xxx.xxx.111/32 on ppp1 which is the IP of the wan1 interface.

So my concern is that turning off defaultgw will cause the xxxx.xxx.xxx.1/32 to drop from the routing table and my static route to that same address won't know how to reach it (since the interface IP is /32).

While I think about this and maybe go on-site, here is the local-in-policy;

config firewall local-in-policy edit 1 set intf "wan1" set srcaddr "allowed-public-IPs" set dstaddr "all" set action accept set service "HTTPS" set schedule "always" next edit 2 set intf "wan1" set srcaddr "all" set dstaddr "all" set service "HTTPS" set schedule "always" next

[Toshi_Esumi]

That might be the origin of the problem. Any static routes to wan1 including the default route shouldn't have GW IP configured because it's assigned over PPPoE although it doesn't chang. You need to enable "dynamic-gateway" like below:

edit 0

  set device wan1

  set dynamic-gateway enable

next

I would suggest you change this in a maintenance window and test the fail-over and the remote access.

[Toshi_Esumi]

And below is my home FGT config for CenturyLink pppoe over a vlan interface and its result in the routing-table.

config router static

edit 4

  set dst x.x.x.x 255.255.255.255

  set device "ClinkVlan"

  set dynamic-gateway enable

next

edit 8

  set device "ClinkVlan"

  set dynamic-gateway enable

next

end

get router info routing-t all

S*    0.0.0.0/0 [10/0] via GW_IP_VIA_PPPOE, ppp1

S      x.x.x.x/32 [10/0] via GW_IP_VIA_PPPOE, ppp1

[rc42]

Just to add to the confusion, the primary internet is down this morning, no traffic is being passed. But the PPOE is still getting an IP, GW, and route to the gateway. 

This stresses why I used the link-monitor and not just relying on PPOE dropping and losing that GW. If I was relying on the loss of the PPOE issued GW, the firewall would still be trying to use the PPOE route and the site would be down.

[brycemd]

You don't have to use the GW as the target to determine if the link is up.

[rc42]

It is my understanding that link-monitor uses the combination of interface and gateway to determine which route to remove from the routing table.

Or are you referring to the address I ping to determine the interface can pass traffic, in which case I would never use the gateway simply because the gateway is often the providers CPE and is very likely to be up when their service is actually down.

[rc42]

OK, it looks like there were some old policy routes that were causing problems, I also found that not all the fail-over routs had the same distance. Removed the policy routes and fixed the distance, cleaned up the priority as well. And it all looks to be working.

I still have it getting a route from the PPOE, along with the static route. This could cause it to not fail-over if the PPOE is up but not passing traffic. I'm going to fix this as well but I want to be on-site and test the fail-over so it will have to wait till next week.