Explorer

Question

FortiOS 7.4.2 Bug Causes IPsec VPN Tunnel Phase 2 Instability

Forum|Forum|2 years ago
January 22, 2024
28 replies
69851 views

I have had many site-to-site IPsec tunnels working fine for several years until I upgraded to FortiOS 7.4.2. Shortly afterward, my tunnels began dropping connections on random Phase 2 connections. I have had to bring down the phases or entire tunnel to get traffic flowing again many times. I opened a ticket with Fortinet and had three technicians working with me at various times but none found a solution.

I finally downgraded to 7.4.1 and all my problems went away. There is obviously a bug in 7.4.2 and I hope Fortinet finds and acknowledges it and fixes it for the next release.

FortiGate

BillH_FTNT

Staff

Hi andybarker,

Please help to share more details about your network and device. What is your device version? What kind of traffic is in your VPN tunnel? if it is okay for you, pls share the ticket number we can access to take a look at your configuration. Thanks

Regards

Bill

andybarkerAuthor

Explorer

Thanks for your reply, Bill. We have two 200Fs in HA Active-Passive mode on both ends of the tunnel. This tunnel had been in place and working flawlessly for over two years. I worked with three different support techs on ticket 9118289. They tried many things, including disabling DPD and replay detection on the Phase 2s. They also disabled NPU offload. None of that helped at all. Only downgrading to 7.4.1 fixed everything.

BillH_FTNT

Staff

Hi Andybarker

1. To check any replay error counters in NP6 (depending on your NP, you can type 2 commands for sure)

diag npu np6 dce 0

dia npu np6xlite dce 0

2. Check RX error (rxe) reported at your IPsec interface. Check "rxe" counters at "stat"

diag netlink interface list xxx

3. You can test disable Anti-Replay :

config vpn ipsec phase2(-interface)

edit …

set replay { enable* | disable }

4. To check anti-replay , you can use the command below, with replaywin=0, which means it is disabled, and different 0, which means it is still enabled.

diag vpn tunnel list name “xxx”

5. You can try disabling ipsec-inbound-cache.

config system npu

set ipsec-inbound-cache disable

End

6.Try to reduce MTU

FGT # config system interface

FGT (interface) # edit vpninterface

FGT (vpninterface) # set mtu-override enable

FGT (vpninterface) # set mtu 1300

FGT (vpninterface) # end

Let's try to apply these on non-business time. Then, monitor the result and roll back after testing.

HTH

Bill

wendelin

Explorer

Hi andybarker, hi Bill,

i experienced the exact same behaviour on an FG101F running OS 7.4.2 talking to a Sophos appliance. To restart data flow if have to bring down the stalled Phase2 tunnel manually (!); tunnel restarts automatically.

The FG didnt recognized the stopped data flow by itself; there is no automatic detection/restart and the subtunnel is shown as green until manual stop.

We have 5 Phase2 tunnels active at that connection, only the most used one stops working from time to time. The duration between the stops vary from a minute to serveral hours.

Same behaviour have been seen (but only a few times) on an other ipsec tunnel talking to an FG60E (OS 7.0.13)

regards

Michael

maulishshah

Staff

@wendelin, can you please confirm if are you still on version 7.4.2?

If yes, could you please disable the npu-offloading on the affected phase2 and monitor whether the issue persist

config vpn ipsec phase2-interface

edit <name of phase2>

set npu-offload disable

end

By following changes, the issue has been resolved it means the issue with the NPU and we need to collect the logs for the same.

maulishshah

Staff

@andybarker, could you please share your Fortinet Case number to get more information?

Thank you.

andybarkerAuthor

Explorer

9118289

anthonyfoerster

Explorer

Good Morning,
any news to this case?

Regards,

BillH_FTNT

Staff

Hi Anthony,
I am building a lab for your case. Will share the result soon.

Bill

bogyon

New Member

We have the same problem.
I upgraded all Fortigate (61F) to 7.4.2 firmware, except one. The latter (101F) remained on version 6.4.14.
All are communicating with one starpoint (VM1).
At this time, only 6.4.14 could establish a stable IPSec site-to-site connection with the star point. All the others dropped the connection at various times or failed to establish it.
A downgrade to 7.4.1 solved the problem, but the star point remained 7.4.2, and so now everything works fine.
But because of the SSLVPN vulnerability, it would be urgent to upgrade to 7.4.3, which would probably cause problems again with IPSec site-to-site connections.
What could be the solution?

BillH_FTNT

Staff

Hi bogyon

The vulnerability solution is essential; therefore, you should upgrade to the solution suggestion version. If you get an issue, just call Fortinet to get support.

RG/Bill

bogyon

New Member

Hi Bill

I did the upgrade on the starpoint (VM1) to 7.4.3. It worked fine so far with the 7.4.2 firmware. But after the upgrade, no route-based IPSec connections were established. However, the policy-based IPSec connection was successfully established.
The next interesting thing is that the LDAP connection was also broken after the upgrade.
I then gave up and I downgraded to 7.2.7 and all IPSec was successfully established and the LDAP connection was also restored.

After that all devices were upgraded to 7.2.7 firmware and now everything works fine.

Yurisk

SuperUser

Had the same issue - cluster of 200F 7.4.2 A/P to FGT 100F 7.4.1, - IPSec tunnels stop transferring data, tunnel by all indicators stayed up but no data entered the tunnel, flushing IPSec SAs solved issue each time. Usual debug showed no problem. Temporarily "solved" by creating Automation stitch to flush/refresh problematic tunnels daily, until client rolled back to 7.4.1 on FGT200F. Now is the same dilemma - upgrade to 7.4.3 to fix SSL VPN vulnerability and suffer IPSec downs or roll back all the way to 7.0.14.

OLiH

Explorer

Same here. Disabling Anti-Replay did not fix the issue. Downgrade to 7.4.1 did. Defo an issue with 7.4.2, most probably not fixed in 7.4.3. Fortinet support seem to count on one of us to test.

BillH_FTNT

Staff

Hi @OLiH

What is your HW version? Can you share the ticket number? We can try to find your issue ASAP.

Bill

OLiH

Explorer

The ticket # is 9166522 downgraded to 7.4.1.

OLiH

Explorer

Orestis suggest that we upgrade and trigger the issue to be able to get logs. I am not keen, we need the tunnels UP. Any volunteer?

The next suggestion is to wait for next version.

I do not feel like we are getting support, especially since it is not still acknowledged it is a bug......

Here is the answer:

Hello team

I followed up with the suggested link but without logs I cannot open an engineering report or confirm that this is a bug

I believe that you facing a problem but if we cannot investigate further we cannot specify what the issue is

in this case either create a maintenance window and update and provide us the logs to investigate or wait for the next version

Best regards
Orestis

Kangming

Staff

Thanks for your feedback. Could you share your configuration file with me? I will try reproducing it and file a bug internally to let Dev investigate the cause.

My email is: kmliu@fortinet.com, you could upload the configuration to the ticket or send it to me directly, thank you so much.

andybarkerAuthor

Explorer

I cannot spend any more time helping Fortinet Support troubleshoot this issue. I gave them five days of my time and many logs, remote sessions, etc. The first two technicians changed how my IPsec VPNs operated, and the third tech said the changes they made were wrong and needed to be reversed to the original settings. None of the changes they made helped at all.

I will be waiting to see if they admit it is a bug and say it is fixed.

Kangming

Staff

Hi andybarker,

Can you share the output of this command?

#get sys status

License Status: Low-Encryption(LENC) ----------------------->

---> FortiOS: 7.4.2

If you match this result, may have matched a known bug.

RepareIT

Explorer II

I just checked on both of my firewall (200F and 100F), I don't have that License Status line at all. I have the same bug and stuck on 7.4.3 for the vulnerability fixes.

Kangming

Staff

Thanks for your feedback. This should be another problem, but there is no internal bug record yet. Can you share your configuration file with me? I will try reproducing it and file a bug internally to let Dev investigate the cause.

My email is: kmliu@fortinet.com, you could upload the configuration to the ticket or send it to me directly, thank you so much.

Show more replies

Sign up

Login with SSO

Login to the community

Login with SSO

Scanning file for viruses.

This file cannot be downloaded