Conserve Mode, FGT-60F & FortiOS 7.4

[NotMine]

Hi,

Anyone out there using FortiOS v7.4.4,build2662 on the FortiGate-60F? How is your RAM usage?

I've installed v7.4.4,build2662 a couple of weeks ago, and the device was entering conserve mode every few days or so. Usual RAM utilization was around 75%, right after boot, so no wonder it was pushing it into conserve mode.

I've since downgraded to 7.2 (now usual RAM usage i 60-65%) but with this version we're having other issues which I would love to resolve (long connection times, need to refresh a web page a few times to open it etc...).

Here is the info I got during the last conserve mode:

firewall01  get system status

Version: FortiGate-60F v7.4.4,build2662,240514 (GA.F)

First GA patch build date: 230509

Security Level: 2

Firmware Signature: certified

Virus-DB: 92.05717(2024-07-10 07:26)

Extended DB: 92.05717(2024-07-10 07:25)

AV AI/ML Model: 2.17065(2024-07-10 07:45)

IPS-DB: 28.00824(2024-07-10 00:15)

IPS-ETDB: 0.00000(2001-01-01 00:00)

APP-DB: 28.00823(2024-07-08 23:57)

FMWP-DB: 24.00070(2024-07-05 17:45)

IPS Malicious URL Database: 5.00107(2024-07-10 08:52)

IoT-Detect: 28.00824(2024-07-09 17:07)

OT-Detect-DB: 28.00824(2024-07-09 17:07)

OT-Patch-DB: 28.00824(2024-07-09 17:11)

OT-Threat-DB: 28.00823(2024-07-08 23:57)

IPS-Engine: 7.00539(2024-05-09 00:27)

Serial-Number: FGT60F*********

BIOS version: 05000030

System Part-Number: P24286-07

Log hard disk: Not available

Hostname: firewall01

Private Encryption: Disable

Operation Mode: NAT

Current virtual domain: root

Max number of virtual domains: 10

Virtual domains status: 1 in NAT mode, 0 in TP mode

Virtual domain configuration: disable

FIPS-CC mode: disable

Current HA mode: standalone

Branch point: 2662

Release Version Information: GA

System time: Wed Jul 10 18:32:42 2024

Last reboot reason: warm reboot

firewall01  diag sys top

[H[JRun Time:  0 days, 22 hours and 34 minutes

12U, 0N, 0S, 85I, 3WA, 0HI, 0SI, 0ST; 1917T, 301F

       ipshelper      186      R <    99.9     9.0    6

           quard      208      S       2.9     0.8    4

           snmpd      197      S       0.4     0.6    0

            node      169      S       0.0     4.1    6

       ipsengine      346      S <     0.0     3.3    5

       ipsengine      347      D <     0.0     3.3    7

       ipsengine      348      S <     0.0     3.1    6

             wad      298      S       0.0     2.6    2

       forticron      174      S       0.0     2.3    2

             wad      300      S       0.0     2.1    6

         cmdbsvr      132      S       0.0     2.1    0

         miglogd      183      S       0.0     2.0    0

          cw_acd      221      S       0.0     1.8    1

       forticron     3677      S       0.0     1.6    2

             wad      190      S       0.0     1.5    5

       forticron     3678      R       0.0     1.5    3

       forticron     3676      S       0.0     1.5    4

         sslvpnd      187      S       0.0     1.4    3

            csfd      228      S       0.0     1.3    5

       scanunitd     3645      S <     0.0     1.2    2

[H[JRun Time:  0 days, 22 hours and 34 minutes

2U, 0N, 1S, 73I, 24WA, 0HI, 0SI, 0ST; 1917T, 304F

       ipshelper      186      D <    11.7     7.0    1

            iked      192      S       2.9     0.9    4

       ipsengine      348      S <     1.9     3.7    6

       ipsengine      346      S <     1.3     3.8    5

       ipsengine      347      S <     1.3     3.8    7

         miglogd      306      S       0.3     1.3    0

       urlfilter      290      S <     0.3     0.8    1

           radvd      213      S       0.3     0.6    2

       forticron     3678      R       0.1     1.5    3

         sslvpnd      235      S       0.1     1.1    3

         sslvpnd      236      S       0.1     1.1    1

           authd      176      S       0.1     0.7    1

         syslogd      194      S       0.1     0.7    1

        dnsproxy      215      S       0.1     0.5    1

             acd      200      S       0.1     0.4    7

  merged_daemons      172      S       0.1     0.4    2

            node      169      S       0.0     4.1    6

             wad      298      S       0.0     2.6    2

       forticron      174      S       0.0     2.3    2

             wad      300      S       0.0     2.1    2

[H[JRun Time:  0 days, 22 hours and 34 minutes

10U, 0N, 0S, 87I, 3WA, 0HI, 0SI, 0ST; 1917T, 316F

       ipshelper      186      R <    83.1     7.4    1

       forticron      174      S       0.7     2.3    3

       ipsengine      346      S <     0.5     3.9    5

       ipsengine      347      S <     0.5     3.8    7

       ipsengine      348      S <     0.1     3.8    6

          cw_acd      221      S       0.1     1.8    0

         sslvpnd      238      S       0.1     1.1    7

            node      169      S       0.0     4.1    6

             wad      298      S       0.0     2.6    2

             wad      300      S       0.0     2.1    0

         cmdbsvr      132      S       0.0     2.1    0

         miglogd      183      S       0.0     2.1    5

       forticron     3677      S       0.0     1.6    2

             wad      190      S       0.0     1.5    6

       forticron     3678      R       0.0     1.5    3

       forticron     3676      S       0.0     1.5    4

         sslvpnd      187      S       0.0     1.4    5

         miglogd      306      S       0.0     1.3    2

            csfd      228      S       0.0     1.3    5

       scanunitd     3645      S <     0.0     1.2    2

[H[JRun Time:  0 days, 22 hours and 34 minutes

11U, 0N, 0S, 86I, 3WA, 0HI, 0SI, 0ST; 1917T, 330F

       ipshelper      186      R <    94.8     7.4    2

       ipsengine      348      D <     1.1     3.9    6

          cw_acd      221      S       0.1     1.8    3

       forticron     3678      R       0.1     1.5    3

         sslvpnd      235      S       0.1     1.1    4

           snmpd      197      S       0.1     0.6    3

            node      169      S       0.0     4.1    7

       ipsengine      346      S <     0.0     3.9    5

       ipsengine      347      S <     0.0     3.8    7

             wad      298      S       0.0     2.6    5

       forticron      174      S       0.0     2.3    3

             wad      300      S       0.0     2.1    5

         miglogd      183      S       0.0     2.1    0

         cmdbsvr      132      S       0.0     2.1    0

       forticron     3677      S       0.0     1.6    2

             wad      190      S       0.0     1.5    6

       forticron     3676      S       0.0     1.5    4

         sslvpnd      187      S       0.0     1.4    5

         miglogd      306      S       0.0     1.3    3

            csfd      228      S       0.0     1.3    6

[jblyon]

I got nowhere on a support call this morning. They want logs to try to correlate the issue with other identical reports that they've received (at least support finally admitted there were other reports.) The problem is the logs they want quit being logged when the unit hits the extreme memory threshold. I've run a script to collect the logs when the issue happens and they simply don't get recorded. It's a Catch 22...they want the logs to correlate the issue before sending it to engineering, but the logs don't exist because of the issue.

The sad thing is others with 100/200F units also seem to be experiencing high memory utilization, but those units have enough to basically handle the leaks and keep running. The 2GB units just don't have that luxury. The bug(s) isn't just limited to the 2GB models, they're just the only ones crashing due to it.

We're already planning to downgrade to 7.2.10 this weekend. We can't even schedule Fortiguard updates outside business hours now without the update crashing the Fortigate. We're just going to wing it this week without current Fortiguard definitions, which is NOT a position any business should need to be in.

This is being handled terribly. We're probably going to end up skipping 7.4 entirely. Boss just scheduled a meeting to discuss options to jump ship when our current licenses expire in a little over a year...

[NotMine]

I will open another ticket (previous one being closed with RMA), not because I expect any real resolution, but to "pump up the numbers" and hopefully increase awareness and pressure inside the organization.

[EME]

I just told Support that I do not find the “solution” acceptable and that he should escalate it internally. I also pointed out that as a company we have 50 40F and 61F in the field and that it is unacceptable that these will soon have to be replaced, because at some point we will be forced to 7.4.

[S2I]

The blue line is the monitoring of RAM usage of my forti 61F.
I recently upgrade from 6.4.last to 7.4.last.... guess when ?
(about 15% more !!!)

[NotMine]

The scariest part about this picture is the obvious trend of increasing RAM usage... :D

[dbhavsar]

Hello Everyone,

Have you tried this workaround:
https://community.fortinet.com/t5/FortiGate/Technical-Tip-FortiGate-is-entering-into-Conserve-Mode-d... 

[jblyon]

It still hits conserve mode even on the mini database and with acceleration disabled.

We're also seeing other issues with 7.4.5 crop up now. Fortiguard at times will fail to update because it detects a self signed certificate in the chain of the factory hardware cert (which is properly registered to the serial number), showing spinning circles for the status of various Fortiguard licensed services, then suddenly it'll stop yelling about the cert, the status of the features will show normally, and updates will run.

We're also seeing spoke devices just randomly lose their BGP routes from the hubs. BGP will still be established, there's no errors logged, but the routes are just gone. Routes to other sites through the hub are still present and working, but the hub's local network routes just drop until we either reboot the spoke or forcefully rebuild BGP.

This is on top of the finally acknowledged IPSEC memory leak.

I just finished downgrade testing for 7.2.10. I doubt we'll entertain the idea of any 7.4 release anytime soon. I've never seen a product this badly broken this far into the release cycle, and I work for a Microsoft Partner...

[EME]

@jblyon,

Fortunately, I don't have BGP on my home firewall, but we do use it in the company, so another thing to consider.

My ticket @FotiSupport was escalated to senior Support on Monday, but I haven't heard back from them yet. Not even the statement, “The finally acknowledged IPSEC memory leak”.
I will wait some time for them and decide what to do.

[EME]

@dbhavsar 

Of these measures, I scheduled the FortiGuard update at 2 a.m. every day. At least this way we don't have the Conserve mode during the day. I have to check every morning if it has been recovered. I hope it will be fixed soon, otherwise I will go back to 7.2.

[EME]

I have given up :(
Last Saturday I went back to 7.2.10 on my home firewall. Conserve Mode was usually set to 2-3 AM, due to scheduled FortiGuard updates, but I had also experienced that the WiFi (FortiAP) was not accepting new clients. After rebooting the firewall, it worked again. This was the final push to a rollback