Default route across VPN tunnel

[jmlux]

We are using IPsec over a Metro Ethernet connection. This connection provides no Internet service, only service to HQ. The goal is to send all traffic across the tunnel. Now I've found a solution but I don't know why it works... One default route is created by DHCP for WAN1 and given distance 5. Still I want to use the tunnel for everything, except IPsec traffic itself obviously! Apparently I have to use the same distance for a second default route (the one across the tunnel) or everything breaks (it sends everything across the tunnel or to wan).

Additionally in order to distinguish both default routes I have: 1) dropped priority (not distance) of default route set by DHCP of wan1 to 3 (instead of 0) 2) specified a second default route 0.0.0.0/0 -> VPN_HQ (the tunnel interface) with priority 0 (hence making it the boss) The kernel routing table, and therefore also route cache now look like this: tab=254 vf=1 scope=0 type=1 proto=11 prio=0 0.0.0.0/0.0.0.0/0->0.0.0.0/0 pref=0.0.0.0 gwy=0.0.0.0 dev=34(VPN_HQ) tab=254 vf=1 scope=0 type=1 proto=11 prio=3 0.0.0.0/0.0.0.0/0->0.0.0.0/0 pref=0.0.0.0 gwy=192.168.178.1 dev=5(wan1) It does what I want it to do. However I don't really understand how it knows that the outer-tunnel traffic should use wan1, while the inner-tunnel traffic uses VPN_HQ.

I would like to exclude that the fact that it is working is a bug. Or some side-effect which will make my life miserable in the future, like a ticking timebomb. In any case I haven't found any documents that would describe how to (correctly) create the setup that we desire... Any enlightenment on this matter is very welcome. Thanks, Marki

[Toshi_Esumi]

The tunnel comes up/stays up initiated by HQ side, not from local side.

If HQ's IP is static, I would set the /32 route toward wan if no internet should go out locally.

[jmlux]

I don't think it's possible for HQ to bring up the tunnel in this case since there is another firewall at branch side which doesn't forward (4)500 to the Fortigate. Ports (4)500 are only forwarded on HQ side to HQ Fortigate. And in fact HQ address is supposed to be dynamic...

[Toshi_Esumi]

Port forwarding forward all traffic destined to the port(s) you specified to the local device. So all IPsec related packets HQ send to the NAT outside IP reach the local FGT.

[jmlux]

You do not understand what I write.

Or I do not understand what I write.

There is (at least) another firewall between both Fortigate.

The one in front of the local Fortigate does NOT forward IPsec ports.

As such, HQ FGT cannot start IPsec communication.

[Toshi_Esumi]

You just didn't describe the detail for anybody to understand your set upon both ends related to the VPN. If you draw a diagram, it would tell a thousand words.

[Toshi_Esumi]

Also I recommend you run sniffing on both ends with "any" for the interface then add option "4" so that you can see what interface it's coming in and going out. Then run IKE debugging to see how the IKE negotiation settles initiated by which side.

[jmlux]

I'd like to summarize my findings for anyone trying to accomplish this in the future. Apparently the requirement of having a default route across the tunnel is not a common thing... Requirements: * WAN1 using DHCP * Tunnel setup using (D)DNS * all traffic must go through the tunnel The first two requirements make your setup portable. You can move the FGT around as long as DHCP/IP service is there and you get a DNS resolution for the hardcoded name. To emphasize the last point, the requirement is to have all traffic, both locally originating (from the FGT itself) as well as the users behind the FGT, go through the tunnel. This requires a default route pointing to the tunnel interface. In a first step this will result in a setup where the kernel routing table will contain two default routes like this: # get router info kernel tab=254 vf=1 scope=0 type=1 proto=11 prio=0 0.0.0.0/0.0.0.0/0->0.0.0.0/0 pref=0.0.0.0 gwy=0.0.0.0 dev=34(VPN_HQ) tab=254 vf=1 scope=0 type=1 proto=11 prio=0 0.0.0.0/0.0.0.0/0->0.0.0.0/0 pref=0.0.0.0 gwy=192.168.178.1 dev=5(wan1) The router's routing table will contain this: # get route info routing-table detail Routing table for VRF=0 S*      0.0.0.0/0 [10/0] is directly connected, VPN_HQ                   [5/0] via 192.168.178.1, wan1 This won't work because by default the route learned by DHCP has a distance of 5, while the default settings of the static route you point to the tunnel will be 10. You have to manually use the same distance as the route learned from DHCP. The router's routing table will now look like this: # get route info routing-table detail Routing table for VRF=0 S*      0.0.0.0/0 [5/0] is directly connected, VPN_HQ                   [5/0] via 192.168.178.1, wan1                    Now at least the IPsec traffic itself flows correctly because it probably is hardcoded somehow that the actual IPsec traffic is supposed to keep using the only present physical interface (wan1), even though there are two equal routes (in theory, the "directly connected" interface might even "win") As far as the rest of the traffic goes, the FGT will perform ECMP and try to distribute the flows across the two routes/links. This will end up in some connections across the tunnel working flawlessly while others just don't establish (AFAIK a hash algorithm is used to distribute the flows, like odd IP uses link1, even IP uses link2, or the like). In our case this made for example FGT's syslog daemon go to 100% CPU because that local traffic was probably hashed to be egressed through wan1 but there was no policy allowing that.

To fix that (no more ECMP) you have to change the priority (a second setting beside the distance) of the route learned from WAN1 DHCP. In that case you have to configure this in interface settings and the resulting routing table then contains: # get route info routing-table detail Routing table for VRF=0 S*      0.0.0.0/0 [5/0] is directly connected, VPN_HQ                   [5/0] via 192.168.178.1, wan1, [3/0] You can see the priority (here: 3) appended to the route to wan1 which makes this route LESS preferred. IPsec traffic will continue to flow undisturbed (hope this is a feature, and not a bug :) whereas all other traffic will prefer VPN_HQ. This is also reflected in the kernel routing table: # get router info kernel tab=254 vf=1 scope=0 type=1 proto=11 prio=0 0.0.0.0/0.0.0.0/0->0.0.0.0/0 pref=0.0.0.0 gwy=0.0.0.0 dev=34(VPN_HQ) tab=254 vf=1 scope=0 type=1 proto=11 prio=3 0.0.0.0/0.0.0.0/0->0.0.0.0/0 pref=0.0.0.0 gwy=192.168.178.1 dev=5(wan1) Happy tunnelling.

[40james_FTNT]

I do topologies where I need to default all traffic to the overlay (ipsec) a lot. Other options that what you did with the priority. - disable default gw being received from dhcp. Add a static route to the tunnel ip you are connecting so your tunnel can come up and then add the default to the tunnel. - If you have ECMP routes to the underlay and overlay you could just enable sdwan. Then set a rule that all traffic uses the overlay. Glad you figured it out!

[jmlux]

james_h wrote:
- disable default gw being received from dhcp. Add a static route to the tunnel ip you are connecting so your tunnel can come up and then add the default to the tunnel.

Concerning alternative 1: That wouldn't work when the remote IP is not static but (D)DNS is used, so we do not know it in advance.