60F frequently crashing

[MeoDub]

I recently replaced our FortiGate 60E devices with brand new 60F units (clean config build). Since going live, the HQ 60F has been highly unstable.   We've experienced multiple unexpected reboots, almost on a daily basis.  The 60E's were rock solid for years, but this new device has been nothing short of a disaster. 

• kernel entering extreme low memory and conserve mode
• sslvpn process timeouts
• watchdog triggered lockups
• spontaneous reboots

I'm a bit shocked at how slow official support has been to respond/assist, so I'm hoping the community can provide better guidance or insight.  

We are running the latest mature release, v7.2.11 build1740 (GA.M).  Has anyone experienced anything similar with 7.2.11 or this hardware? Should I consider downgrading to 7.2.9 or earlier?  

Outside of looking at different vendors, which is not a quick fix, I'm at a loss. 

Thank you in advance, and I will gladly provide more config/logs.

Crashlog examples:

2796: 2025-07-22 04:09:43 msg="Kernel enters extreme low memory mode"
2797: 2025-07-22 04:22:32 SSL VPN enter conserve mode.
2798: 2025-07-22 04:22:32 sslvpn watchdog timeout, pid 245
2799: 2025-07-22 04:22:33 <00245> firmware FortiGate-60F v7.2.11,build1740b1740,250210 (GA.M) (Release)
2800: 2025-07-22 04:22:33 <00245> application sslvpnd

[22394.945844] INFO: rcu_sched self-detected stall on CPU
[22394.946440] 1-...: (1 GPs behind) idle=28e/1/0x4000000000000000 softirq=111690/111691 fqs=5245
...
[22394.946616] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [system_task:1]
...
[22394.947316] CPU1: stopping other CPUs
...
[22394.952649] Rebooting in 1 seconds..

[FortiDor]

Hi @MeoDub 

Have you done a factory reset in CLI of the 60F ?
Have you setup some IPS security profile or others which use more ressources on your FGT ?

I never sounds this kind of problème perhaps a hardware default 

You can check the hardware status if there is already a failure 

[AEK]

Check these two article and try optimize memory usage.

https://community.fortinet.com/t5/FortiGate/Technical-Tip-Low-end-FortiGate-models-with-RAM-2GB-ente...

https://community.fortinet.com/t5/FortiGate/Troubleshooting-Tip-How-to-optimize-memory-usage-specifi...

I'd also upgrade to 7.4.8 which currently is the recommended version, ans also because Fortinet introduced special memory optimizations for FGTs with 2GB RAM in 7.4.x.

Hope it helps

[Toshi_Esumi]

First, you can call in any time to get immediate support instead of waiting for somebody to respond to your ticket opened at the support page. You might need to be in the queue for 5 - 10 min though if no engineer is available after the initial rep put your call on-hold. 

The first thing you need to find out is what is/are taking up much of memmory by like "diag sys top 5 30", then hit Shift-m before it goes into conserve mode. If healthy, you wouldn't see number beyond 10% in the 2nd column from the last like below.

Run Time: 135 days, 15 hours and 46 minutes
0U, 24N, 0S, 76I, 0WA, 0HI, 0SI, 0ST; 1918T, 296F
              ipsengine  439    S <    0.0  6.8  0
              ipsengine  441    S <    0.0  6.7  2
              ipsengine  440    S <    0.0  6.7  3
                      node  179    S      0.0  4.8  2
                       wad  322    S       0.0  3.8  2
                       wad  320    S       0.0  3.1  2
              scanunitd 23273 S <    0.0  2.9  0
               ipshelper  438    S <    0.0  2.9  3
                 forticron  169    S       1.3  1.8  1
                  cw_acd  220    S       0.0  1.7  3
                 cmdbsvr  121    S       0.0  1.7  0
                  miglogd  178    S       0.0  1.6  3
                  sslvpnd  180    S        0.0  1.5  1
                  forticldd  170    S        0.0  1.4  3
                        wad  183    S       0.0   1.3  3
                        csfd  235    S       0.0   1.3  2
                     httpsd  163    S       0.0   1.3  0
                      fgfmd  219    S       0.0   1.2  2
                   sslvpnd  240    S       0.0   1.2  1
                   sslvpnd  238    S       0.0   1.2  3
                   sslvpnd  239    S       0.0   1.2  1
initXXXXXXXXXXX  1        S       0.0   1.1  3
                     newcli 23298 S       0.0   1.1  3
                           cid  207    S      0.0   1.0  3
                 dnsproxy  229    S      0.0   1.0  3
                     fgtlogd  189    S      0.0   0.9  1
                   miglogd  315    S      0.0   0.9  2
                extenderd  234    S      0.0   0.9  3
                     fcnacd  176    S       0.0   0.8  1
                       autod  236    S       0.0   0.8  2

Then if you see a particular process(es) is(are) taking up majority of memory, you might want to kill it/them by "diag sys kill 11 [pid]" or "fnsysctl killall [process_name]". The first number next to the process name in above list is the pid/process ID.
https://community.fortinet.com/t5/FortiGate/Technical-Tip-Find-and-restart-kill-a-process-on-a-Forti...

That would temporarily restore the memory. But eventually need to get it figured out why by TAC and you might have a way to avoid it's from happening or change the version. TAC would tell you those options.

Toshi

 

[Toshi_Esumi]

And, this is the KB for "diag sys top".
https://community.fortinet.com/t5/FortiGate/Technical-Tip-Using-the-diagnose-sys-top-CLI-command/ta-...

[BillH_FTNT]

Hi MeoDub,

I think there are two issues with your device.

1. Memory-related issue – You can refer to the links that AEK shared for more details.
2. Reboot-related issue – It appears the CPU experienced a stall.

Could you please share the complete comlogs related to the issue? It would be helpful if you could provide the following information:

• Comlog at the time the issue occurred
• System, event, and traffic logs from before, during, and after the issue (preferably from FortiAnalyzer or Syslog)
• Output of diag debug crashlog read
• Output of execute tac report
• Full configuration

Thank you!

My name is Bill from Fortinet, my email is bhoang@fortinet.com

Thank you

Bill

[MeoDub]

Thank you all for the great advice and help!  I didn't know I could call in directly, but I was able to finally connect with a tech.  Apparently they are working on this issue as my case is not unique.  Here is what they did for me to temporarily fix while they continue to investigate and work on a solution.

Here is a recap of the session:

- fortigate was showing memory usage at 71%

- ipsengine consuming more than others

- checked system events and found fortigate did manual update this morning and as soon as it         finished fortigate went to conserve mode

-  changed update to weekly on Sunday

- also made the following changes to mitigate memory issues

config system global

    set sslvpn-max-worker-count 2

    set wad-worker-count 2

    set scanunit-count 2

config ips global

   set engine-count 2

Edit:  Almost forgot...we also set up an auto-script to restart the IPS Engine every 12 hours.

[BillH_FTNT]

Hi MeoDub,

Could you share me the Ticket number I will get the information through ticket. Thank you

Regards

Bill

[MeoDub]

Certainly, it's 10914977

[BillH_FTNT]

Hi MeoDub

Do you have Microsoft Teams ?  Could I ask you some information through Teams ? Thanks 

Bill