Conserve Mode, FGT-60F & FortiOS 7.4

[NotMine]

Hi,

Anyone out there using FortiOS v7.4.4,build2662 on the FortiGate-60F? How is your RAM usage?

I've installed v7.4.4,build2662 a couple of weeks ago, and the device was entering conserve mode every few days or so. Usual RAM utilization was around 75%, right after boot, so no wonder it was pushing it into conserve mode.

I've since downgraded to 7.2 (now usual RAM usage i 60-65%) but with this version we're having other issues which I would love to resolve (long connection times, need to refresh a web page a few times to open it etc...).

Here is the info I got during the last conserve mode:

firewall01  get system status

Version: FortiGate-60F v7.4.4,build2662,240514 (GA.F)

First GA patch build date: 230509

Security Level: 2

Firmware Signature: certified

Virus-DB: 92.05717(2024-07-10 07:26)

Extended DB: 92.05717(2024-07-10 07:25)

AV AI/ML Model: 2.17065(2024-07-10 07:45)

IPS-DB: 28.00824(2024-07-10 00:15)

IPS-ETDB: 0.00000(2001-01-01 00:00)

APP-DB: 28.00823(2024-07-08 23:57)

FMWP-DB: 24.00070(2024-07-05 17:45)

IPS Malicious URL Database: 5.00107(2024-07-10 08:52)

IoT-Detect: 28.00824(2024-07-09 17:07)

OT-Detect-DB: 28.00824(2024-07-09 17:07)

OT-Patch-DB: 28.00824(2024-07-09 17:11)

OT-Threat-DB: 28.00823(2024-07-08 23:57)

IPS-Engine: 7.00539(2024-05-09 00:27)

Serial-Number: FGT60F*********

BIOS version: 05000030

System Part-Number: P24286-07

Log hard disk: Not available

Hostname: firewall01

Private Encryption: Disable

Operation Mode: NAT

Current virtual domain: root

Max number of virtual domains: 10

Virtual domains status: 1 in NAT mode, 0 in TP mode

Virtual domain configuration: disable

FIPS-CC mode: disable

Current HA mode: standalone

Branch point: 2662

Release Version Information: GA

System time: Wed Jul 10 18:32:42 2024

Last reboot reason: warm reboot

firewall01  diag sys top

[H[JRun Time:  0 days, 22 hours and 34 minutes

12U, 0N, 0S, 85I, 3WA, 0HI, 0SI, 0ST; 1917T, 301F

       ipshelper      186      R <    99.9     9.0    6

           quard      208      S       2.9     0.8    4

           snmpd      197      S       0.4     0.6    0

            node      169      S       0.0     4.1    6

       ipsengine      346      S <     0.0     3.3    5

       ipsengine      347      D <     0.0     3.3    7

       ipsengine      348      S <     0.0     3.1    6

             wad      298      S       0.0     2.6    2

       forticron      174      S       0.0     2.3    2

             wad      300      S       0.0     2.1    6

         cmdbsvr      132      S       0.0     2.1    0

         miglogd      183      S       0.0     2.0    0

          cw_acd      221      S       0.0     1.8    1

       forticron     3677      S       0.0     1.6    2

             wad      190      S       0.0     1.5    5

       forticron     3678      R       0.0     1.5    3

       forticron     3676      S       0.0     1.5    4

         sslvpnd      187      S       0.0     1.4    3

            csfd      228      S       0.0     1.3    5

       scanunitd     3645      S <     0.0     1.2    2

[H[JRun Time:  0 days, 22 hours and 34 minutes

2U, 0N, 1S, 73I, 24WA, 0HI, 0SI, 0ST; 1917T, 304F

       ipshelper      186      D <    11.7     7.0    1

            iked      192      S       2.9     0.9    4

       ipsengine      348      S <     1.9     3.7    6

       ipsengine      346      S <     1.3     3.8    5

       ipsengine      347      S <     1.3     3.8    7

         miglogd      306      S       0.3     1.3    0

       urlfilter      290      S <     0.3     0.8    1

           radvd      213      S       0.3     0.6    2

       forticron     3678      R       0.1     1.5    3

         sslvpnd      235      S       0.1     1.1    3

         sslvpnd      236      S       0.1     1.1    1

           authd      176      S       0.1     0.7    1

         syslogd      194      S       0.1     0.7    1

        dnsproxy      215      S       0.1     0.5    1

             acd      200      S       0.1     0.4    7

  merged_daemons      172      S       0.1     0.4    2

            node      169      S       0.0     4.1    6

             wad      298      S       0.0     2.6    2

       forticron      174      S       0.0     2.3    2

             wad      300      S       0.0     2.1    2

[H[JRun Time:  0 days, 22 hours and 34 minutes

10U, 0N, 0S, 87I, 3WA, 0HI, 0SI, 0ST; 1917T, 316F

       ipshelper      186      R <    83.1     7.4    1

       forticron      174      S       0.7     2.3    3

       ipsengine      346      S <     0.5     3.9    5

       ipsengine      347      S <     0.5     3.8    7

       ipsengine      348      S <     0.1     3.8    6

          cw_acd      221      S       0.1     1.8    0

         sslvpnd      238      S       0.1     1.1    7

            node      169      S       0.0     4.1    6

             wad      298      S       0.0     2.6    2

             wad      300      S       0.0     2.1    0

         cmdbsvr      132      S       0.0     2.1    0

         miglogd      183      S       0.0     2.1    5

       forticron     3677      S       0.0     1.6    2

             wad      190      S       0.0     1.5    6

       forticron     3678      R       0.0     1.5    3

       forticron     3676      S       0.0     1.5    4

         sslvpnd      187      S       0.0     1.4    5

         miglogd      306      S       0.0     1.3    2

            csfd      228      S       0.0     1.3    5

       scanunitd     3645      S <     0.0     1.2    2

[H[JRun Time:  0 days, 22 hours and 34 minutes

11U, 0N, 0S, 86I, 3WA, 0HI, 0SI, 0ST; 1917T, 330F

       ipshelper      186      R <    94.8     7.4    2

       ipsengine      348      D <     1.1     3.9    6

          cw_acd      221      S       0.1     1.8    3

       forticron     3678      R       0.1     1.5    3

         sslvpnd      235      S       0.1     1.1    4

           snmpd      197      S       0.1     0.6    3

            node      169      S       0.0     4.1    7

       ipsengine      346      S <     0.0     3.9    5

       ipsengine      347      S <     0.0     3.8    7

             wad      298      S       0.0     2.6    5

       forticron      174      S       0.0     2.3    3

             wad      300      S       0.0     2.1    5

         miglogd      183      S       0.0     2.1    0

         cmdbsvr      132      S       0.0     2.1    0

       forticron     3677      S       0.0     1.6    2

             wad      190      S       0.0     1.5    6

       forticron     3676      S       0.0     1.5    4

         sslvpnd      187      S       0.0     1.4    5

         miglogd      306      S       0.0     1.3    3

            csfd      228      S       0.0     1.3    6

[swissroot]

I have exactly the same. Also done all tweaks mentioned by fortinet except the "killing" tasks and still get the conserve mode exactly at the time of the fortiguard update.

I upgraded this morning after the next down (this time even serial was not accessible) to 7.6 even this is feature and .0 release... read on some redit from a guy which has the same issues after that it was "more stable" :) also opened a ticket with forti. 

Let's see if the community get the solution before the vendor... ;)

[NotMine]

Great to have someone brave enough to try 7.6, @swissroot! :) Looking forward to see if it will resolve the issue. Although for us it is not a viable solution because we need SSL VPN (for IOS devices), which is discontinued for low-end devices in 7.6+.

[swissroot]

The "brave" journey stopped hard over the weekend... First it was looking good until the update of the FortiGuard and then one CPU spiked to 100% and stayed there. After that randomly it gave no connection or extremely slow responses for surfing. The WebGui was still accessible without any issues and no special things are logged. The other day it was that bad with the random responses that I had to reboot it. This solved the CPU spike after 15min delay of the reboot... updating a IDS/AV???? But then it started again after about 3-4h uptime to get random delay's in surfing or streaming. Sometimes no delay sometimes even to the point of not reachable.

I was watching a movie on Sunday and this was doing it right in the middle again. I then decided to go the short way even with the knowledge of loosing something but being up hopefully in short time again and downgraded it to 7.2.10. This release is actually working fine on my 61F so I gave it a try. The downgrade took some time but since then (klapp on wood) it's stable on the 60F. I think I would doo a factory reset and fresh config on the 7.2.x train. 

Hard to see that Fortinet is rolling out such bad releases more and more and in the same time removing functions which where working on the 7.0 and 7.2 without any bigger issues on the "low-end" models if you stay in the spec's of small business amount of users.

Also as mentioned below why not put 2 or 4 gb extra of ram in it this will not make the whole unit much more expensive in terms of manufacturing, except the are buying the memory from apple...;-)

will post an update when I made the fresh config to get rid of the downgrade mess if it's stable now. 

[EME]

On request of Fortinet support, I added a stitch to run debugging when in Conserve Mode. First result was "auto-script cannot run because of high memory usage (96%)" :p

Second one did deliver a complete debug report. Uploaded to the case.

Did not configure the memory tweaking Support suggested, because of the "low end Fortigate". I find this hs. This same FortiGate with same config run perfect on 7.0 and 7.2 without any memory problem. I still think it will be solved after a bug is fixed, probably in the IPS engine. Like I told, would not be the first time.

I also still wondering, why memory is still a problem in modern day equipment. What is the production cost of 2 Gb of memory? Maybe a dollar or 2. So why not put in 4 or 8 Gb, will make the FortiGate max $10,- more expensive.

[NotMine]

All great points @EME. Regarding the RAM - totally agree with you! 4GB should be bare minimum! If it is any consolation, I did implement the memory tweaks - they did not help. :)

However, it looks like I've found an acceptable 'workaround' for our environment: since we can all agree that FortiGuard updates trigger the Conserve Mode, I've scheduled daily update for 6AM. I've also created an automation stitch to restart the FortiGate each morning at 5:40AM, just to lower RAM usage 5-10% prior to the update.

[swissroot]

to be honest we are speaking about "enterprise solutions" even the entry level fortigates have an enterprise price tag. So in this segment I'm not discussing about rebooting it every day to prevent a conserve mode. This can be done in consumer hw with a consumer price tag but not in a business environment. Forti should check their firmwares and fix those flaws. I had a 61F for x years on the older trains of firmwares working with all features enabled without any conserve mode during the whole life of it. So it's possible and we are speaking still about the same feature set of AV/IDS/WEB and so on nothing really new and fancy. And it was even possible to do ssl-vpn on top without any issues :-).

[NotMine]

I agree 100%! There is definitely something wrong with these new releases. Either it is a bug which they are not willing to acknowledge and deal with, or it is "planned obsolescence". Considering the "low end devices" narrative, and removal of the features, I'm inclined to think it is the latter.

[topogigione]

Me too. They lowered prices of old 40F/60F, then made this model not working with new OSs.

Instead of simply declare that 40F/60F are "cheap and old" devices.

[EME]

Hi @NotMine

I did the same thing to update FortiGuard updates at 2 a.m. as a workaround. For my home network, this is temporarily fine. For a corporate network, it's a choice between continuity and security. You could miss an important update. And yes, the window of opertunity is small, but not 0.

[EME]

And so, it begins :(

Feedback from support:

Please note that based on the output provided, i can see that the firewall entered the conserve mode due to low memory issue caused by the IPS engine (AV failed to open).

Please refer to the following document that explains the cause behind this behavior and the remedy that you can implement to prevent this issue:
https://community.fortinet.com/t5/FortiGate/Technical-Tip-IPS-socket-size-and-fail-open-mode/ta-p/19...

Also, i would recommend to follow the document below since you're using a small series if the firewall that has a 2GB of RAM.
https://community.fortinet.com/t5/FortiGate/Technical-Tip-Steps-to-optimize-the-Memory-consumption/t...

They keep throwing it at “only 2Gb” of memory.

Again, they push me to tweak the memory and now they also want me to configure my FortiGate in “fail-open”.

These “solutions” are driving me crazy.

Like @swissroot said, this is no way to handle customers that use Enterprise equipment. I should not be forced to degrade my security, to be able to maintain continuity,

Even this small unit at my home does maybe seem to them as a small customer, but they make the mistake that I work at a company that owns and manages more than 60 FortiGate’s in all sizes, with also FortiSwitches, FortiAP’s, FortiWeb, FortiManager and FortiAnalyser.

When is this sent from support to engineering, so they can say, O wait, we have a bug, here is the update and it works fine again?