Explorer

Question

FortiOS 7.4.2 Bug Causes IPsec VPN Tunnel Phase 2 Instability

Forum|Forum|2 years ago
January 22, 2024
28 replies
70064 views

I have had many site-to-site IPsec tunnels working fine for several years until I upgraded to FortiOS 7.4.2. Shortly afterward, my tunnels began dropping connections on random Phase 2 connections. I have had to bring down the phases or entire tunnel to get traffic flowing again many times. I opened a ticket with Fortinet and had three technicians working with me at various times but none found a solution.

I finally downgraded to 7.4.1 and all my problems went away. There is obviously a bug in 7.4.2 and I hope Fortinet finds and acknowledges it and fixes it for the next release.

FortiGate

Show previous replies

OLiH

Explorer

Fortinet support is still refusing to acknowledge the issue btw. This is unacceptable.

Hello team

we cannot investigate further if we don't have logs from the issue. And even if the customer mentioned on the link provide those

we have to confirm if your case matches with that issue

Additionally as you have rolled back I will adjust the case priority to

Best regards
Orestis

rtwright68

New Member

We are experiencing the same issue on 7.4.3 on our 100F. Fortinet needs to fix this, it is not isolated to one customer or type of router. We are using 100F's (HA) at our corporate location and 90E's (HA) at our other locations. The 90E's are running 7.0.14 (SSL vulnerability fix).

Updating to 7.4.3 has introduced major instability with our VPN connections, which prior to this firmware upgrade were ROCK SOLID.

RepareIT

Explorer II

I updated Kangming by email with my configs. My ipsec vpn configs are very basic, the only thing I could think of is that we have traffic shapping, I wonder if anyone else has traffic shapping enabled ? Strangely the tunnel connection drop is between 10 and 30 min most of the time, 2 days ago it was for 3 hours. I also get these every 2 seconds - Error Number Invalid ESP packet detected (replayed packet). when the tunnel is down in the logs.

OLiH

Explorer

We do not have traffic shaping. Drops lasted for minutes most of the time, but we had a couple of hours too. I don't remember about the ESP packet detected lines in the log.

Kangming

Staff

Deer Customer,

We can reproduce it in the lab. The problem is related to np6xlite, so np6xlite related devices will be affected in V7.4.2GA and V7.4.3GA. Dev is investigating it further and we will officially fix it on V7.4.4GA

If you are an internal employee, we can track the progress of this issue through #1003830.

This is a workaround we found, which can solve the problem of blocked IPsec VPN traffic in this way.

Execute "set replay disable" on phase2-interface on both sides of the IPsec VPN

BOS-101 (Interim)# config vpn ipsec phase2-interface
BOS-101 (phase2-interface) (Interim)# edit "XXXXXX"
BOS-101 (xxxxxx) (Interim)# set replay disable
BOS-101 (xxxxxx) (Interim)# end

BOS-101 (Interim)# diagnose vpn ike gateway flush

IPsec VPN Tunnel Phase 2 Instability after upgrade to 7.4.2GA on NP6xlite platform.

Workaround: Execute "set replay disable" on phase2-interface on both sides of the IPsec VPN

This part will be updated to the release note soon.

OLiH

Explorer

AFAIK, I am still facing a support techie who is requesting logs... Zero help. The issue is still not acknowledged...

Hello team

I have tried to recreate the issue in my lab between two FGTs in 7.4.1 and updated to 7.4.2 and it worked as expected

additionally the examples you mention in the cases are different since the first update says

"and Dev has confirmed that the problem is in FGT_100F-v7-build2597-FORTINET.out, and I am verifying on this build whether it is successfully fixed."

which doesn't apply to your case as you have FG200F

and the second one concerns np6xlite related devices which could related to your case but since the suggested workaround ( "set replay disable" ) has no effect in your case it is not the same issue

so the two options we have is either provide logs so we can investigate the issue or wait until the 7.4.4 version released ( expected between Mar 26, 2024 and Apr 26, 2024 )

Best regards
Orestis

Kangming

Staff

Hi OliH,

If we can see the constant changes of np6xlite DROP_IPSEC0_ENGINB through the following command "diagnose npu np6xlite dce" when the IPsec VPN status is UP, routing and policies are normal, but ESP traffic is blocked, especially when inbound packets cannot be seen, it should match this bug.

FGT100F/FGT200F/FGT60F/FGT80F... They are both np6xlite platforms and therefore have the same issues.

BOS-101 # diagnose npu np6xlite dce DROP_IPSEC0_ENGINB0:0000000000000008[80]  BOS-101 # diagnose npu np6xlite dce DROP_IPSEC0_ENGINB0:0000000000000003[80]  BOS-101 # diagnose npu np6xlite dce DROP_IPSEC0_ENGINB0:0000000000000010[80]  BOS-101 # diagnose npu np6xlite dce DROP_IPSEC0_ENGINB0:0000000000000005[80]  BOS-101 # diagnose npu np6xlite dce DROP_IPSEC0_ENGINB0:0000000000000006[80]  BOS-101 # BOS-101 # diagnose npu np6xlite dce DROP_IPSEC0_ENGINB0:0000000000007676[80] DROP_IPSEC0_ENGINB1:0000000000000010[81] DROP_IPSEC0_ENGINB3:0000000000000001[83] STAT_IVS_COPY_CNT:0000000000001558[9a]  BOS-101 # diagnose npu np6xlite dce DROP_IPSEC0_ENGINB0:0000000000000006[80]

sfmf305

Visitor III

Experienced the same issue on a couple of 100F's on v7.4.3

Implemneted the work around.
Monitoring now.
Thanks!!!

computos

Visitor III

Hi, in my company we upgraded to fortios 7.4.3 and have the same situation. Do you have any news of this incident ? Thks

People_First

New Member

Currently our enterprise is using 7.4.3 across all of our gates they range from 60fs to 200fs we use advpn tunnels to communicate to all branch site. We were definitely experiencing tunnel instability across the board. After working with support over many sessions the root of the issue seems to be BGP Flapping/becoming unreachable that seems to poison the routing table. One of the Fixes that seems to be working was to set phase one on any locations having the issue to

set npu-offload disable

Our tech also suggested disabling the short cut routes. (we have not done this yet) if the issue continues.

as always I suggest you discuss any advice with support before trying changes and if possible not on a production environment or during production hours.

BillH_FTNT

Staff

Hi @People_First

1. set npu-offload disable : it will not offload on NPU however all packets will be processed by CPU. If your traffic is not much, then it is quite a good option.

2. You can check feedback from Kangmin in this thread about disabling the replay in phase 2 of IPsec by commanding "set replay disable." Remember "diagnose VPN like gateway flush."

3. You can check something more in my post https://community.fortinet.com/t5/Support-Forum/FortiOS-7-4-2-Bug-Causes-IPsec-VPN-Tunnel-Phase-2-Instability/td-p/295462

HTH

Bill

aguerriero

Explorer

This doesn't just affect NP6 lite. I have an 1100E that is constantly dropping phase 2 connections on dozens of tunnels. If I only ever create 1 phase 2 SA it will work just fine. Any more than 2 and the problem gets worse and worse.

BillH_FTNT

Staff

Hi @aguerriero

If the post above doesn't help you so, I think you should contact TAC for support.
Bill

aguerriero

Explorer

I did. I am tracking a bunch of threads that have this major problem. Alot of chatter on the web keeps saying TAC keeps trying to push this back as some kind of configuration issue... but there are way to many people complaining about it.