Conserve Mode, FGT-60F & FortiOS 7.4

[NotMine]

Hi,

Anyone out there using FortiOS v7.4.4,build2662 on the FortiGate-60F? How is your RAM usage?

I've installed v7.4.4,build2662 a couple of weeks ago, and the device was entering conserve mode every few days or so. Usual RAM utilization was around 75%, right after boot, so no wonder it was pushing it into conserve mode.

I've since downgraded to 7.2 (now usual RAM usage i 60-65%) but with this version we're having other issues which I would love to resolve (long connection times, need to refresh a web page a few times to open it etc...).

Here is the info I got during the last conserve mode:

firewall01  get system status

Version: FortiGate-60F v7.4.4,build2662,240514 (GA.F)

First GA patch build date: 230509

Security Level: 2

Firmware Signature: certified

Virus-DB: 92.05717(2024-07-10 07:26)

Extended DB: 92.05717(2024-07-10 07:25)

AV AI/ML Model: 2.17065(2024-07-10 07:45)

IPS-DB: 28.00824(2024-07-10 00:15)

IPS-ETDB: 0.00000(2001-01-01 00:00)

APP-DB: 28.00823(2024-07-08 23:57)

FMWP-DB: 24.00070(2024-07-05 17:45)

IPS Malicious URL Database: 5.00107(2024-07-10 08:52)

IoT-Detect: 28.00824(2024-07-09 17:07)

OT-Detect-DB: 28.00824(2024-07-09 17:07)

OT-Patch-DB: 28.00824(2024-07-09 17:11)

OT-Threat-DB: 28.00823(2024-07-08 23:57)

IPS-Engine: 7.00539(2024-05-09 00:27)

Serial-Number: FGT60F*********

BIOS version: 05000030

System Part-Number: P24286-07

Log hard disk: Not available

Hostname: firewall01

Private Encryption: Disable

Operation Mode: NAT

Current virtual domain: root

Max number of virtual domains: 10

Virtual domains status: 1 in NAT mode, 0 in TP mode

Virtual domain configuration: disable

FIPS-CC mode: disable

Current HA mode: standalone

Branch point: 2662

Release Version Information: GA

System time: Wed Jul 10 18:32:42 2024

Last reboot reason: warm reboot

firewall01  diag sys top

[H[JRun Time:  0 days, 22 hours and 34 minutes

12U, 0N, 0S, 85I, 3WA, 0HI, 0SI, 0ST; 1917T, 301F

       ipshelper      186      R <    99.9     9.0    6

           quard      208      S       2.9     0.8    4

           snmpd      197      S       0.4     0.6    0

            node      169      S       0.0     4.1    6

       ipsengine      346      S <     0.0     3.3    5

       ipsengine      347      D <     0.0     3.3    7

       ipsengine      348      S <     0.0     3.1    6

             wad      298      S       0.0     2.6    2

       forticron      174      S       0.0     2.3    2

             wad      300      S       0.0     2.1    6

         cmdbsvr      132      S       0.0     2.1    0

         miglogd      183      S       0.0     2.0    0

          cw_acd      221      S       0.0     1.8    1

       forticron     3677      S       0.0     1.6    2

             wad      190      S       0.0     1.5    5

       forticron     3678      R       0.0     1.5    3

       forticron     3676      S       0.0     1.5    4

         sslvpnd      187      S       0.0     1.4    3

            csfd      228      S       0.0     1.3    5

       scanunitd     3645      S <     0.0     1.2    2

[H[JRun Time:  0 days, 22 hours and 34 minutes

2U, 0N, 1S, 73I, 24WA, 0HI, 0SI, 0ST; 1917T, 304F

       ipshelper      186      D <    11.7     7.0    1

            iked      192      S       2.9     0.9    4

       ipsengine      348      S <     1.9     3.7    6

       ipsengine      346      S <     1.3     3.8    5

       ipsengine      347      S <     1.3     3.8    7

         miglogd      306      S       0.3     1.3    0

       urlfilter      290      S <     0.3     0.8    1

           radvd      213      S       0.3     0.6    2

       forticron     3678      R       0.1     1.5    3

         sslvpnd      235      S       0.1     1.1    3

         sslvpnd      236      S       0.1     1.1    1

           authd      176      S       0.1     0.7    1

         syslogd      194      S       0.1     0.7    1

        dnsproxy      215      S       0.1     0.5    1

             acd      200      S       0.1     0.4    7

  merged_daemons      172      S       0.1     0.4    2

            node      169      S       0.0     4.1    6

             wad      298      S       0.0     2.6    2

       forticron      174      S       0.0     2.3    2

             wad      300      S       0.0     2.1    2

[H[JRun Time:  0 days, 22 hours and 34 minutes

10U, 0N, 0S, 87I, 3WA, 0HI, 0SI, 0ST; 1917T, 316F

       ipshelper      186      R <    83.1     7.4    1

       forticron      174      S       0.7     2.3    3

       ipsengine      346      S <     0.5     3.9    5

       ipsengine      347      S <     0.5     3.8    7

       ipsengine      348      S <     0.1     3.8    6

          cw_acd      221      S       0.1     1.8    0

         sslvpnd      238      S       0.1     1.1    7

            node      169      S       0.0     4.1    6

             wad      298      S       0.0     2.6    2

             wad      300      S       0.0     2.1    0

         cmdbsvr      132      S       0.0     2.1    0

         miglogd      183      S       0.0     2.1    5

       forticron     3677      S       0.0     1.6    2

             wad      190      S       0.0     1.5    6

       forticron     3678      R       0.0     1.5    3

       forticron     3676      S       0.0     1.5    4

         sslvpnd      187      S       0.0     1.4    5

         miglogd      306      S       0.0     1.3    2

            csfd      228      S       0.0     1.3    5

       scanunitd     3645      S <     0.0     1.2    2

[H[JRun Time:  0 days, 22 hours and 34 minutes

11U, 0N, 0S, 86I, 3WA, 0HI, 0SI, 0ST; 1917T, 330F

       ipshelper      186      R <    94.8     7.4    2

       ipsengine      348      D <     1.1     3.9    6

          cw_acd      221      S       0.1     1.8    3

       forticron     3678      R       0.1     1.5    3

         sslvpnd      235      S       0.1     1.1    4

           snmpd      197      S       0.1     0.6    3

            node      169      S       0.0     4.1    7

       ipsengine      346      S <     0.0     3.9    5

       ipsengine      347      S <     0.0     3.8    7

             wad      298      S       0.0     2.6    5

       forticron      174      S       0.0     2.3    3

             wad      300      S       0.0     2.1    5

         miglogd      183      S       0.0     2.1    0

         cmdbsvr      132      S       0.0     2.1    0

       forticron     3677      S       0.0     1.6    2

             wad      190      S       0.0     1.5    6

       forticron     3676      S       0.0     1.5    4

         sslvpnd      187      S       0.0     1.4    5

         miglogd      306      S       0.0     1.3    3

            csfd      228      S       0.0     1.3    6

[NotMine]

Thank you @dbhavsar and @amuda.

Yesterday I've implemented some of the changes recommended in KB @dbhavsar recommended and the device did NOT enter the conserve mode last night. Prior to the config changes, it would enter the conserve mode at least once a day, usually late afternoon. So, hopefully, these config changes helped.

I'll update the topic in the next few days with the result and exact config changes I've implemented.

Cheers!

[NotMine]

Unfortunately, after working fine for several days with the "optimize memory" configuration changes suggested here, the device went full retard. It simply stopped working and it would not boot after the restart. I had to format the boot device, reinstall the firmware from scratch and restore the configuration.

Now it goes to conserve mode every few days or so and I have to drive to the office and cold restart it. I did upgrade to 7.4.5 as soon as it was available.

This morning I've turned off as much of UTP scanning (AV especially) and logging in polices as I could, and I've also implemented some of the "memory optimization" tweaks. Hopefully, it will help.

Here are the config changes I've implemented (just for future reference):

config ips global
  set engine-count 2

  set socket-size 48
end

config log memory setting
  set status disable
end

config system session-ttl
  set default 600
end

config system dns
  set dns-cache-limit 300
end

[NotMine]

After implementing the above changes, device is still entering conserve mode. Furthermore, two of the ports, connected to two separate network devices, are behaving strangely (flapping). HQIP passed, no hardware fault found. So i have a ticket opened, hoping for resolution.

[pochtakazah]

@NotMine Dude, just schedule killing of high-memory-consuming processes, idk for example every 3 hours... Here mine CLI script (FGT-60F):


fnsysctl killall wad
fnsysctl killall miglogd
fnsysctl killall ipsengine


without that FGT-60F (7.4.3) enters conserve mode every single day at same time (period 24 hours).

now it floats from 55% to 65% and again to 55% every 3 hours (with tweaks may be even lower). 

also you can create automation to reboot FGT when it enters conserve mode (to not driving to reboot it manualy).


NSE 5

[NotMine]

Thanks, If it comes to that, I'll try your method. But I really think that we should not be forced to 'hack' the device this way.

Anyway, tech support initiated RMA of the device. Hopefully, it will solve at least some problems. But most likely, we'll upgrade to 70F, as I understand it has 4GB of RAM.

[EME]

Hello, I have a Fortigate 60F @Home. I upgraded it to 7.4.5 last Sunday (a maintenance window is rare at my house). My principle is to wait for patch .4 before upgrading to a new level. I was hoping I would be safe with 7.4.5.
But now my Fortigate enters “Kernel enters memory conserve mode” every day.
Once I had to reboot and twice it came out on its own.

I agree with @NotMine, that this Fortinet should fix this as it is clearly a bug. Changing the configuration, especially by disabling or limiting IPS scanning, is not an option for a Firewall.
By the way, this would certainly not be the first time either. I have also experienced it with FTG 1200Ds. Turned out to be a bug in the IPS engine after a very long case.

[jblyon]

We're seeing the same thing happen on our 60Fs running 7.4.5. It enters conserve mode and then extreme low memory mode a few seconds later. This is immediately after a Fortiguard update occurs and the unit needs to reload the AV database. The unit will drop all connections until it is either rebooted or about 20 minutes pass. If you let the 20 minutes pass memory use drops right back down to the 66% that it's normally always at. There's no sign of any other memory leaks, it's sudden after an AV database reload following an update.

We have a ticket open, but Fortinet has not responded to it yet. In the meantime we've scheduled updates to occur outside business hours where it doesn't matter if a branch office drops offline for 20 minutes afterward. This has kept the units from dropping off during business hours.

Edit: Support replied trying to blame the fact that the unit was already at 66% memory use before the update process. I asked them to explain how an increase of at least 29% on 7.4.5 could be considered normal when on 7.2 and 6.4 memory use increases by only 1-2% during Fortiguard updates...

[EME]

Yes I can confirm. After a “FortiSandbox AV database updated” I get “Low Memory” and “conserve mode” messages. I have listed 3 of them:

 

[NotMine]

Kudos to both of you, @jblyon and @EME, for narrowing down the potential cause!

We've replaced the unit in RMA, but the device is still entering the conserve mode.

I can substantiate your finding that FortiGuard update is causing this, because our device stopped responding yesterday around 1:30 PM, with FortiGuard updates scheduled daily at 1 PM. It was completely unresponsive, even thru the console connection.

Strangely enough, nothing in the crashlog about the conserve mode.

[EME]

I just registered a case with Fortinet :)