mpls connection with ipsec vpn as backup

[Fullmoon]

hi folks have similar issue same as this thread https://forum.fortinet.com/tm.aspx?m=135847 The only difference is that we are using FOS 5.4 and 5.6. we're playing with static routes (distance and metric) seems failover is not working as expected. If you ask regarding the design and static routes similar with the link stated above. Any help is much appreciated.

[Toshi_Esumi]

So your main connection goes directly into MPLS over L2 circuit then your backup is via IPsec VPN into the same MPLS cloud over another ISP's internet circuit (we call it a "L3 circuit")? If so, you can handle outgoing routing with some static route arrangement at this location with no problem. But the issue comes up on the opposite direction: coming from other MPLS connected locations into this location. Depending on your MPLS provider's setup or their product offers, they might not fail over to the backup IPSec path when the main circuit is down. You need to talk to the provider how to set up the failover. Likely they need to change something and use a routing protocol with this location to be able to move the incoming routes around. We are actually one of MPLS network providers and we offer BGP setup to our customers who needs a failover.

[Fullmoon]

Hi Toshi. thank you for your time. Please see revised network design if you shed some insights for this setup. Existing and working setup, remote site having no fortigate firewall all computers are routed to router device (192.168.2.1). All workstations from Remote site able to access local resources to Head Quarters side via IPVPN.   End user wants to achieve failover/redundancy in the event IPVPN goes down. So based on the attached diagram, WANx interface is connected to Internet and Fortigate LAN interface is connected to L2 switch with an ip address of 192.168.2.254. Now all workstations in remote site reconfigured to route via local interface of Fortigate. Outcome of the revised design, workstations from remote site still able to ping vlans from HQ via IPVPN, Second, workstations from remote site still able to ping vlans from HQ via IPSEC VPN, the catch here we need to changed the Distance manually in the Static Route in order the traffic to sway to IPSEC Tunnel the moment we altered the route/gateway to IPVPN link (e.g from 192.168.2.2 to 192.168.2.100) just to assume that link is down or not reachable)   Does it matter because during our testing the workstation is connected to L2 switch not on fortigate switch interfaces, so even we altered the gateway, fortigate able to detect that the ipvpn is still connected physically?   Again, a million thanks!

[Toshi_Esumi]

Let me ask this. Is the remote router and the core switch (L2/L3 switch/router?) are GW to each other over the IPVPN? My guess is no and both sides have different GW IPs lives inside of the IPVPN cloud. Then this is a flaw No.1: you're relying on L1/L2 down to detect the connection down on both sides. There are situations the link is up but connectivity is down. Or one side of link down but the other side of link is up. And with your design with a set of FGs, FGs can't detect connection down. This means you need to rely on the routing change capability on the router and the core switch. Adding FG-FG vpn is just adding the secondary connection between the router and the core switch. You can't utilize FG's capability. A simple solution is to set up a routing protocol like ospf, bgp, etc. between the router and the core switch if they're capable. Then the routing protocol detect neighboring down and can failover at the same time. If you have to use static routing, you need to have a way to detect the connection down on both side and remove the static routes, like Cisco's IP SLA&track. Another option would be letting those FGs to terminate both IPVPN and IPSec on both side and let FGs to detect IPVPN connection down, with like link-monitor. It works like just Cisco's IP SLA&track, it detests the connection down (with pinging) and remove static routes toward the link. Again, the key design concept here is what device can detect the down and make the routing decision, then the routing change need to happen on both end at the same time.

[Fullmoon]

toshiesumi wrote:
Another option would be letting those FGs to terminate both IPVPN and IPSec on both side and let FGs to detect IPVPN connection down, with like link-monitor. It works like just Cisco's IP SLA&track, it detests the connection down (with pinging) and remove static routes toward the link.  

Yes, IPSEC VPN was terminated on both FG's. T'was configured that remote site will Dialup UP to HQ to established ipsec tunnel and to accessed HQ local resources.   Please correct me if I'm wrong based on your statement. Meaning to say given that WAN(1) is facing Internet ISP, I need to assign 192.168.2.x on WAN(2) interface and introduce new network address on LAN? With this, I have the flexibility to monitor WAN link failure. Your thoughts please. thank you.

[Toshi_Esumi]

If you want to go that design, the FG#1 should terminate the ISP with wan1(or2) and terminates IPVPN with wan2(or1), then LAN(internal) has 192.168.2.1/24, which will eliminate the router. Then you want to set up link-monitor toward the other end to detect connection down. I think there were multiple discussions in the forum as well as FTNT KB, documentations about link-monitor. Then again, you need to do the same on HQ side. You should test your design well in a maintenance window(s) before deploying so that you and your customer can be sure the design and config work as you/they expect.

[Toshi_Esumi]

Or keep .2.254 on FG#1 if you don't want to change the default route for all devices.