60F frequently crashing

Hi @MeoDub 

Have you done a factory reset in CLI of the 60F ?
Have you setup some IPS security profile or others which use more ressources on your FGT ?

I never sounds this kind of problème perhaps a hardware default 

You can check the hardware status if there is already a failure 

Check these two article and try optimize memory usage.

https://community.fortinet.com/t5/FortiGate/Technical-Tip-Low-end-FortiGate-models-with-RAM-2GB-entering/ta-p/295489

https://community.fortinet.com/t5/FortiGate/Troubleshooting-Tip-How-to-optimize-memory-usage-specifically/ta-p/304981

I'd also upgrade to 7.4.8 which currently is the recommended version, ans also because Fortinet introduced special memory optimizations for FGTs with 2GB RAM in 7.4.x.

Hope it helps

First, you can call in any time to get immediate support instead of waiting for somebody to respond to your ticket opened at the support page. You might need to be in the queue for 5 - 10 min though if no engineer is available after the initial rep put your call on-hold. 

The first thing you need to find out is what is/are taking up much of memmory by like "diag sys top 5 30", then hit Shift-m before it goes into conserve mode. If healthy, you wouldn't see number beyond 10% in the 2nd column from the last like below.

Run Time: 135 days, 15 hours and 46 minutes
0U, 24N, 0S, 76I, 0WA, 0HI, 0SI, 0ST; 1918T, 296F
              ipsengine  439    S <    0.0  6.8  0
              ipsengine  441    S <    0.0  6.7  2
              ipsengine  440    S <    0.0  6.7  3
                      node  179    S      0.0  4.8  2
                       wad  322    S       0.0  3.8  2
                       wad  320    S       0.0  3.1  2
              scanunitd 23273 S <    0.0  2.9  0
               ipshelper  438    S <    0.0  2.9  3
                 forticron  169    S       1.3  1.8  1
                  cw_acd  220    S       0.0  1.7  3
                 cmdbsvr  121    S       0.0  1.7  0
                  miglogd  178    S       0.0  1.6  3
                  sslvpnd  180    S        0.0  1.5  1
                  forticldd  170    S        0.0  1.4  3
                        wad  183    S       0.0   1.3  3
                        csfd  235    S       0.0   1.3  2
                     httpsd  163    S       0.0   1.3  0
                      fgfmd  219    S       0.0   1.2  2
                   sslvpnd  240    S       0.0   1.2  1
                   sslvpnd  238    S       0.0   1.2  3
                   sslvpnd  239    S       0.0   1.2  1
initXXXXXXXXXXX  1        S       0.0   1.1  3
                     newcli 23298 S       0.0   1.1  3
                           cid  207    S      0.0   1.0  3
                 dnsproxy  229    S      0.0   1.0  3
                     fgtlogd  189    S      0.0   0.9  1
                   miglogd  315    S      0.0   0.9  2
                extenderd  234    S      0.0   0.9  3
                     fcnacd  176    S       0.0   0.8  1
                       autod  236    S       0.0   0.8  2

Then if you see a particular process(es) is(are) taking up majority of memory, you might want to kill it/them by "diag sys kill 11 [pid]" or "fnsysctl killall [process_name]". The first number next to the process name in above list is the pid/process ID.
https://community.fortinet.com/t5/FortiGate/Technical-Tip-Find-and-restart-kill-a-process-on-a-FortiGate-by/ta-p/208496

That would temporarily restore the memory. But eventually need to get it figured out why by TAC and you might have a way to avoid it's from happening or change the version. TAC would tell you those options.

Toshi

 

Hi MeoDub,

I think there are two issues with your device.

1. Memory-related issue – You can refer to the links that AEK shared for more details.
2. Reboot-related issue – It appears the CPU experienced a stall.

Could you please share the complete comlogs related to the issue? It would be helpful if you could provide the following information:

• Comlog at the time the issue occurred
• System, event, and traffic logs from before, during, and after the issue (preferably from FortiAnalyzer or Syslog)
• Output of diag debug crashlog read
• Output of execute tac report
• Full configuration

Thank you!

My name is Bill from Fortinet, my email is bhoang@fortinet.com

Thank you

Bill

Thank you all for the great advice and help!  I didn't know I could call in directly, but I was able to finally connect with a tech.  Apparently they are working on this issue as my case is not unique.  Here is what they did for me to temporarily fix while they continue to investigate and work on a solution.

Here is a recap of the session:

- fortigate was showing memory usage at 71%

- ipsengine consuming more than others

- checked system events and found fortigate did manual update this morning and as soon as it         finished fortigate went to conserve mode

-  changed update to weekly on Sunday

- also made the following changes to mitigate memory issues

config system global

    set sslvpn-max-worker-count 2

    set wad-worker-count 2

    set scanunit-count 2

config ips global

   set engine-count 2

Edit:  Almost forgot...we also set up an auto-script to restart the IPS Engine every 12 hours.

Hi MeoDub and all,

The is a log "[22394.945844] INFO: rcu_sched self-detected stall on CPU
[22394.946440] 1-...: (1 GPs behind) idle=28e/1/0x4000000000000000 softirq=111690/111691 fqs=5245
...
[22394.946616] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [system_task:1]
...
[22394.947316] CPU1: stopping other CPUs
..." 

I think there was a large amount of traffic reaching the box at that moment.
@MeoDub, could you please share the system logs, event logs, and traffic logs here or through the FortiCare ticket? I will cross-check and try to reproduce the issue in the lab. If needed, I may create an internal ticket for the Engineering team to investigate further. I’ll share any updates once I receive support from the team.

Regards

Bill