Unusual routing error

[atsak]

I have 3 FG60E's with route based IPSEC tunnels back to our datacenter.

Once a month or so, they'll suddenly start routing traffic over WAN1 for the private subnet instead of the VPN tunnel interface (viewable in the taffic logs).   All three of them do it.   None of my Juniper's do (we are slowly converting from SSG-5's)

Routing table has x.x.x.x/x TunnelName distance 1

x.x.x.x/x BackupTunnelonWWAN distance 2

0.0.0.0/0 blackhole distance 200

And the WAN1 interface is DHCP from the ISP, and is set at distance 5.

I can fix it by resetting all the sessions or rebooting the firewall.

How is this possible that it's ignoring the distance settings and an explicit route and instead using the implicit 0.0.0.0 received on the WAN interface?   More importantly, how do I stop it?

[tanr]

It looks like you've got duplicate destination subnets in your static routes with different distances? 

So the route to your "TunnelName" interface will be active, but the route to BackupTunnelonWWAN will be inactive since its a duplicate destination with a greater distance.

Maybe what is happening is that TunnelName goes down temporarily and the traffic then gets routed out whatever default route you have to WAN1 (you didn't list it above) before the BackupTunnelonWWAN becomes active.  Since it is then an existing session it doesn't get changed back to your BackupTunnelonWWAN once that becomes active.

You should be able to work around that by setting the routes to both TunnelName and BackupTunnelonWWAN to be the same distance but have different priorities.  That way both routes would be kept active.

http://kb.fortinet.com/kb/viewContent.do?externalId=FD32103 discusses this.

Note that you should still have blackhole routes for bogon addresses so that if both of your VPN tunnels go down you don't end up routing vpn traffic out the wan.  Discussion at https://forum.fortinet.com/tm.aspx?m=100686. 

[atsak]

The WAN1 connection is a DHCP provided next hop (Cableco connections), so it's implicit and has never shown in the static route table (because it's dynamic - it does show though in the overall routing table of course or it wouldn't work).

I have a blackhole for 0.0.0.0 - should that not automatically also include the 10.x.x.x subnets I'm using or do they need to be explicit?

[tanr]

The blackhole for 0.0.0.0 probably won't ever get used because its distance is greater than the distance for your wan 0.0.0.0 default route.   

If you create blackhole routes specifically for the bogons (10.x.x.x, etc.) but with a large distance those routes will get hit (if your vpn tunnels are down) instead of the default wan route because the bogon routes are more specific.  At least, that's my understanding.

[atsak]

K thanks let me try the bogon blackhole for the 10 subnet (which is what we use) and see if that corrects the issue; its so intermittent it's hard to know for sure :)

[emnoc]

The blackhole for 0.0.0.0 probably won't ever get used because its distance is greater than the distance for your wan 0.0.0.0 default route.   

The bigger distance would not enter the database if the smaller distance is available.

You should use blackhole for BCP with ipsec

Op, when your tunnel gooes down do you see a "destination in the route table" and label null summary for the destinations that are covered in  the IPsec-tunnel?

if your answer is yes, than your blackholes are working and traffic will NOT leak out the wan1.

Ken

[tanr]

@emnoc, would you expect the secondary BackupTunnelonWWAN route to get skipped briefly when the shorter distance tunnel route goes down since the backup route isn't active?  Or could that be a routing bug?

[emnoc]

Don't know, if you have access  tot he ipsec tunnels shutdown the tunnel and confirm the  blackhole routes populate the  table. If they do , than you know they work.

Execute cli cmd  get router  info routing-table details <insert the exat route-prefix here>

E.g

FGT90DSOC # get router  info routing-table details 192.168.3.0/23 Routing entry for 192.168.3.0/23   Known via "static", distance 10, metric 0, best   * directly connected, IPSECTUN0

 if you clear  the vpn-tunnel the route should come up in "null" b4 the ipsec-ike gateway resets like in  the below examples

e.g

FGT90DSOC # get router  info routing-table  static  | grep 172.16.11 S       172.16.11.0/24 [10/0] is directly connected, AWSvpn1 FGT90DSOC # diagnose  vpn ike  gateway  clear name AWSvpn1 FGT90DSOC # get router  info routing-table  static  | grep 172.16.11 S       172.16.11.0/24 [254/0] is a summary, Null FGT90DSOC # get router  info routing-table  static  | grep 172.16.11 S       172.16.11.0/24 [10/0] is directly connected, AWSvpn1

That would be the last test, you have todo it quick b4 phase1/2 is re-established