FortiOS 7.4.2 Bug Causes IPsec VPN Tunnel Phase 2 Instability

[andybarker]

I have had many site-to-site IPsec tunnels working fine for several years until I upgraded to FortiOS 7.4.2. Shortly afterward, my tunnels began dropping connections on random Phase 2 connections. I have had to bring down the phases or entire tunnel to get traffic flowing again many times. I opened a ticket with Fortinet and had three technicians working with me at various times but none found a solution.

I finally downgraded to 7.4.1 and all my problems went away. There is obviously a bug in 7.4.2 and I hope Fortinet finds and acknowledges it and fixes it for the next release.

[OLiH]

Thanks for being helpful Kangming. If you need more config files, please let me know.

[OLiH]

Fortinet support is still refusing to acknowledge the issue btw. This is unacceptable.

Hello team

we cannot investigate further if we don't have logs from the issue. And even if the customer mentioned on the link provide those

we have to confirm if your case matches with that issue

Additionally as you have rolled back I will adjust the case priority to

Best regards
Orestis

[rtwright68]

We are experiencing the same issue on 7.4.3 on our 100F.  Fortinet needs to fix this, it is not isolated to one customer or type of router.  We are using 100F's (HA) at our corporate location and 90E's (HA) at our other locations.  The 90E's are running 7.0.14 (SSL vulnerability fix).  

Updating to 7.4.3 has introduced major instability with our VPN connections, which prior to this firmware upgrade were ROCK SOLID.

[RepareIT]

I updated Kangming by email with my configs. My ipsec vpn configs are very basic, the only thing I could think of is that we have traffic shapping, I wonder if anyone else has traffic shapping enabled ? Strangely the tunnel connection drop is between 10 and 30 min most of the time, 2 days ago it was for 3 hours. I also get these every 2 seconds - Error Number    Invalid ESP packet detected (replayed packet). when the tunnel is down in the logs.

[OLiH]

We do not have traffic shaping. Drops lasted for minutes most of the time, but we had a couple of hours too. I don't remember about the ESP packet detected lines in the log.

[Kangming]

Thank you for your help.

I have reproduced it in the lab, and Dev has confirmed that the problem is in FGT_100F-v7-build2597-FORTINET.out, and I am verifying on this build whether it is successfully fixed.

However, the GA of V7.4.4 has not been released yet, so you cannot get the official GA build. We are trying to see if there is a suitable workaround to avoid this problem.

We will continue to follow up and provide feedback on our Dev processing results.

Thanks for your feedback.

[Kangming]

Deer  Customer,

We can reproduce it in the lab. The problem is related to np6xlite, so np6xlite related devices will be affected in V7.4.2GA and V7.4.3GA.  Dev is investigating it further and we will officially fix it on V7.4.4GA

If you are an internal employee, we can track the progress of this issue through #1003830.

This is a workaround we found, which can solve the problem of blocked IPsec VPN traffic in this way.

Execute "set replay disable" on phase2-interface on both sides of the IPsec VPN

BOS-101 (Interim)# config vpn ipsec phase2-interface
BOS-101 (phase2-interface) (Interim)# edit "XXXXXX"
BOS-101 (xxxxxx) (Interim)# set replay disable
BOS-101 (xxxxxx) (Interim)# end

BOS-101 (Interim)# diagnose vpn ike gateway flush

IPsec VPN Tunnel Phase 2 Instability after upgrade to 7.4.2GA on NP6xlite platform.

Workaround: Execute "set replay disable" on phase2-interface on both sides of the IPsec VPN

This part will be updated to the release note soon.

[OLiH]

AFAIK, I am still facing a support techie who is requesting logs... Zero help. The issue is still not acknowledged...

Hello team

I have tried to recreate the issue in my lab between two FGTs in 7.4.1 and updated to 7.4.2 and it worked as expected

additionally the examples you mention in the cases are different since the first update says

"and Dev has confirmed that the problem is in FGT_100F-v7-build2597-FORTINET.out, and I am verifying on this build whether it is successfully fixed."

which doesn't apply to your case as you have FG200F

and the second one concerns np6xlite related devices which could related to your case but since the suggested workaround ( "set replay disable" ) has no effect in your case it is not the same issue

so the two options we have is either provide logs so we can investigate the issue or wait until the 7.4.4 version released ( expected between Mar 26, 2024 and Apr 26, 2024 )

Best regards
Orestis

[Kangming]

Hi OliH,

If we can see the constant changes of np6xlite DROP_IPSEC0_ENGINB through the following command "diagnose npu np6xlite dce" when the IPsec VPN status is UP, routing and policies are normal, but ESP traffic is blocked, especially when inbound packets cannot be seen, it should match this bug.

FGT100F/FGT200F/FGT60F/FGT80F... They are both np6xlite platforms and therefore have the same issues.

BOS-101 # diagnose npu np6xlite dce
DROP_IPSEC0_ENGINB0:0000000000000008[80]

BOS-101 # diagnose npu np6xlite dce
DROP_IPSEC0_ENGINB0:0000000000000003[80]

BOS-101 # diagnose npu np6xlite dce
DROP_IPSEC0_ENGINB0:0000000000000010[80]

BOS-101 # diagnose npu np6xlite dce
DROP_IPSEC0_ENGINB0:0000000000000005[80]

BOS-101 # diagnose npu np6xlite dce
DROP_IPSEC0_ENGINB0:0000000000000006[80]

BOS-101 #
BOS-101 # diagnose npu np6xlite dce
DROP_IPSEC0_ENGINB0:0000000000007676[80] DROP_IPSEC0_ENGINB1:0000000000000010[81] DROP_IPSEC0_ENGINB3:0000000000000001[83] STAT_IVS_COPY_CNT:0000000000001558[9a]

BOS-101 # diagnose npu np6xlite dce
DROP_IPSEC0_ENGINB0:0000000000000006[80]

[sfmf305]

Experienced the same issue on a couple of 100F's on v7.4.3

Implemneted the work around.
Monitoring now. 
Thanks!!!