Fortigate IPsec Site-to-Site Tunnel traffic is not passing through the other MPLS connection

[olivern4]

Hi All,

We are having issues in our MPLS - IPsec VPN Tunnel, please see attached network diagram for reference.

Whenever we up the tunnel in ISP1, we have no problem.

1. We can access the ATM and Blackbox Switch from the server and vice versa. We also have a test laptop in the client side which I forgot to in include in the diagram.

2. We can traceroute the traffic and see that it is passing through the expected path, which shows the IP address from both sides in ISP1.

3. Even on Fortigate logs, we can see that traffic is using the right policy and static route.

The problem start when we swing to ISP2, for HA testing.

1. We can still access the test laptop from the server and vice versa. We use the test laptop first before we proceed with adding the ATM and Blackbox, which is what we did in the ISP1.

2. When we traceroute the traffic, we can still see that it is passing through the router in the ISP1 of the client side.

3. The client also tried to traceroute from their end, and it can reach the server, but we cannot see the IP address of our Head Office router and Fortigate firewall when they send the screenshot of the traceroute result.

4. We tried to disable the tunnel of ISP1, disable the static route from both firewall and even the policy, and still when we do a traceroute, we can still see that it is passing through the ISP1 of the client router. The client tried again to traceroute, but still we cannot see our IP addresses.

5. When we check the Forward Traffic in the Fortigate, it shows that it is passing through the right policy, which is using the ISP2 tunnel.
6. When I checked the "diag debug application ike -1" command and enabled it, it is still passing through the tunnel in ISP1 with error "error 101:Network is unreachable" and "could not send IKE Packet(ident_i1send)". Maybe because the tunnel in ISP1 is down.

Does anyone knows why is this happening?

Thank you so much who can help us.

[sjoshi]

Hi,

Take the sniffer on both FGT simultanesouly to see which interface it egress out

diag sniff packet any 'host x.x.x.x and icmp' 4 0 l >> where x.x.x.x is the dst IP

[olivern4]

Hi @sjoshi 

Noted on this. We'll try and update this thread.

Thank you.

[sjoshi]

Hi 

sure waiting for your update

[olivern4]

Hi All,

Just to update this thread, the client is still busy at this moment, so we cannot proceed with the activity. We will try to do it maybe next week.

Thank you.

[sjoshi]

yes u can try to share the packet capture to identify the flow

[dingjerry_FTNT]

Hi @olivern4 ,

Please run the debug flow commands, the output will tell us more info:

https://community.fortinet.com/t5/FortiGate/Troubleshooting-Tip-First-steps-to-troubleshoot-connecti...

Please use the debug flow commands in Step 5.

[Toshi_Esumi]

Do you see the tunnel over ISP2 up at both side of the FGTs? You should run the IKE debug by specifying the ISP2 side of the public IP on the client side when you run it on the head office FGT.
"diag vpn ike log-filter dst-addr4 [peer_public_ip]"
before the ike app debug command.

You mentioned "for HA testing" and "use the test laptop first before we proceed with adding the ATM and Blackbox". How exactly are you doing this test? If they're in the same sbunet, you generally can test all at once only, like by taking the tunnel over ISP1 down, so that both ends can failover to the ISP2 tunnel at that same time, which is necessary to avoid asymmetric routing per direction.

Toshi

[DPadula]

Adding to @Toshi_Esumi's reply. I see a router in front of the FGT on ISP2 on the client side but not on the ISP1. Is this the real topology or you just forgot to add it?
Assuming you have the same settings on both sides for ISP1 and ISP2 but a extra router is between FGT1 and FGT2 via ISP2 maybe it is the reason of the issue. This is just a hypothesis. Follow the instruction suggested by @Toshi_Esumi first so we can get a better understanding of what is happening.

[callmeahero]

You mentioned that even after disabling the tunnel for ISP1 and the static route, traffic is still passing through ISP1.

Double-check the static routes on both firewalls.

There could be a persistent route pointing to ISP1 that is overriding the routing for ISP2.

Make sure that the routes for ISP2 are correctly prioritized.

Since you are testing HA failover, confirm that when you switch to ISP2, the HA failover settings on the FortiGate are correctly configured to switch both the IPsec tunnel and routing policies to ISP2.

Sometimes, when HA fails over, certain routing or VPN settings may not switch over as expected.

The issue might be related to how the MPLS paths and IPsec tunnels are being routed.

You could try adding policy-based routing (PBR) to ensure that traffic from your network is specifically routed through the correct tunnel (ISP2 in this case).

The error "Network is unreachable" in your IKE debug logs could suggest that the tunnel is still trying to use ISP1 even after the failover.

The "could not send IKE Packet" error might indicate that the FortiGate is unable to establish a connection over ISP2, possibly because it’s still trying to route the traffic via ISP1.

This could be due to incorrect routing or firewall policy settings not being applied properly after the failover.

Make sure split tunneling is not misconfigured.

If your FortiGate is incorrectly using ISP1 even after switching, it might be due to the way the traffic is being routed for IPsec.

Double-check the routing table and make sure that when ISP2 is active, the routes for the IPsec tunnel are correctly pointing to it.

Review your HA settings and make sure both the IPsec tunnel and the routing policies are failing over properly to ISP2.

Confirm that policy-based routing (PBR) is in place if necessary.

Look at the debug logs for both ISPs and check if there’s any misconfiguration preventing the tunnel over ISP2 from establishing properly.

By following these steps, you should be able to identify where the routing is going wrong when switching to ISP2.