Explorer III

Question

Conserve Mode, FGT-60F & FortiOS 7.4

Forum|Forum|1 year ago
July 18, 2024
16 replies
35612 views

Hi,

Anyone out there using FortiOS v7.4.4,build2662 on the FortiGate-60F? How is your RAM usage?

I've installed v7.4.4,build2662 a couple of weeks ago, and the device was entering conserve mode every few days or so. Usual RAM utilization was around 75%, right after boot, so no wonder it was pushing it into conserve mode.

I've since downgraded to 7.2 (now usual RAM usage i 60-65%) but with this version we're having other issues which I would love to resolve (long connection times, need to refresh a web page a few times to open it etc...).

Here is the info I got during the last conserve mode:

firewall01 get system status

Version: FortiGate-60F v7.4.4,build2662,240514 (GA.F)

First GA patch build date: 230509

Security Level: 2

Firmware Signature: certified

Virus-DB: 92.05717(2024-07-10 07:26)

Extended DB: 92.05717(2024-07-10 07:25)

AV AI/ML Model: 2.17065(2024-07-10 07:45)

IPS-DB: 28.00824(2024-07-10 00:15)

IPS-ETDB: 0.00000(2001-01-01 00:00)

APP-DB: 28.00823(2024-07-08 23:57)

FMWP-DB: 24.00070(2024-07-05 17:45)

IPS Malicious URL Database: 5.00107(2024-07-10 08:52)

IoT-Detect: 28.00824(2024-07-09 17:07)

OT-Detect-DB: 28.00824(2024-07-09 17:07)

OT-Patch-DB: 28.00824(2024-07-09 17:11)

OT-Threat-DB: 28.00823(2024-07-08 23:57)

IPS-Engine: 7.00539(2024-05-09 00:27)

Serial-Number: FGT60F*********

BIOS version: 05000030

System Part-Number: P24286-07

Log hard disk: Not available

Hostname: firewall01

Private Encryption: Disable

Operation Mode: NAT

Current virtual domain: root

Max number of virtual domains: 10

Virtual domains status: 1 in NAT mode, 0 in TP mode

Virtual domain configuration: disable

FIPS-CC mode: disable

Current HA mode: standalone

Branch point: 2662

Release Version Information: GA

System time: Wed Jul 10 18:32:42 2024

Last reboot reason: warm reboot

firewall01 diag sys top

[H[JRun Time: 0 days, 22 hours and 34 minutes

12U, 0N, 0S, 85I, 3WA, 0HI, 0SI, 0ST; 1917T, 301F

ipshelper 186 R < 99.9 9.0 6

quard 208 S 2.9 0.8 4

snmpd 197 S 0.4 0.6 0

node 169 S 0.0 4.1 6

ipsengine 346 S < 0.0 3.3 5

ipsengine 347 D < 0.0 3.3 7

ipsengine 348 S < 0.0 3.1 6

wad 298 S 0.0 2.6 2

forticron 174 S 0.0 2.3 2

wad 300 S 0.0 2.1 6

cmdbsvr 132 S 0.0 2.1 0

miglogd 183 S 0.0 2.0 0

cw_acd 221 S 0.0 1.8 1

forticron 3677 S 0.0 1.6 2

wad 190 S 0.0 1.5 5

forticron 3678 R 0.0 1.5 3

forticron 3676 S 0.0 1.5 4

sslvpnd 187 S 0.0 1.4 3

csfd 228 S 0.0 1.3 5

scanunitd 3645 S < 0.0 1.2 2

[H[JRun Time: 0 days, 22 hours and 34 minutes

2U, 0N, 1S, 73I, 24WA, 0HI, 0SI, 0ST; 1917T, 304F

ipshelper 186 D < 11.7 7.0 1

iked 192 S 2.9 0.9 4

ipsengine 348 S < 1.9 3.7 6

ipsengine 346 S < 1.3 3.8 5

ipsengine 347 S < 1.3 3.8 7

miglogd 306 S 0.3 1.3 0

urlfilter 290 S < 0.3 0.8 1

radvd 213 S 0.3 0.6 2

forticron 3678 R 0.1 1.5 3

sslvpnd 235 S 0.1 1.1 3

sslvpnd 236 S 0.1 1.1 1

authd 176 S 0.1 0.7 1

syslogd 194 S 0.1 0.7 1

dnsproxy 215 S 0.1 0.5 1

acd 200 S 0.1 0.4 7

merged_daemons 172 S 0.1 0.4 2

node 169 S 0.0 4.1 6

wad 298 S 0.0 2.6 2

forticron 174 S 0.0 2.3 2

wad 300 S 0.0 2.1 2

[H[JRun Time: 0 days, 22 hours and 34 minutes

10U, 0N, 0S, 87I, 3WA, 0HI, 0SI, 0ST; 1917T, 316F

ipshelper 186 R < 83.1 7.4 1

forticron 174 S 0.7 2.3 3

ipsengine 346 S < 0.5 3.9 5

ipsengine 347 S < 0.5 3.8 7

ipsengine 348 S < 0.1 3.8 6

cw_acd 221 S 0.1 1.8 0

sslvpnd 238 S 0.1 1.1 7

node 169 S 0.0 4.1 6

wad 298 S 0.0 2.6 2

wad 300 S 0.0 2.1 0

cmdbsvr 132 S 0.0 2.1 0

miglogd 183 S 0.0 2.1 5

forticron 3677 S 0.0 1.6 2

wad 190 S 0.0 1.5 6

forticron 3678 R 0.0 1.5 3

forticron 3676 S 0.0 1.5 4

sslvpnd 187 S 0.0 1.4 5

miglogd 306 S 0.0 1.3 2

csfd 228 S 0.0 1.3 5

scanunitd 3645 S < 0.0 1.2 2

[H[JRun Time: 0 days, 22 hours and 34 minutes

11U, 0N, 0S, 86I, 3WA, 0HI, 0SI, 0ST; 1917T, 330F

ipshelper 186 R < 94.8 7.4 2

ipsengine 348 D < 1.1 3.9 6

cw_acd 221 S 0.1 1.8 3

forticron 3678 R 0.1 1.5 3

sslvpnd 235 S 0.1 1.1 4

snmpd 197 S 0.1 0.6 3

node 169 S 0.0 4.1 7

ipsengine 346 S < 0.0 3.9 5

ipsengine 347 S < 0.0 3.8 7

wad 298 S 0.0 2.6 5

forticron 174 S 0.0 2.3 3

wad 300 S 0.0 2.1 5

miglogd 183 S 0.0 2.1 0

cmdbsvr 132 S 0.0 2.1 0

forticron 3677 S 0.0 1.6 2

wad 190 S 0.0 1.5 6

forticron 3676 S 0.0 1.5 4

sslvpnd 187 S 0.0 1.4 5

miglogd 306 S 0.0 1.3 3

csfd 228 S 0.0 1.3 6

FortiGate

Show previous replies

EME

Explorer III

On request of Fortinet support, I added a stitch to run debugging when in Conserve Mode. First result was "auto-script cannot run because of high memory usage (96%)" :p

Second one did deliver a complete debug report. Uploaded to the case.

Did not configure the memory tweaking Support suggested, because of the "low end Fortigate". I find this hs. This same FortiGate with same config run perfect on 7.0 and 7.2 without any memory problem. I still think it will be solved after a bug is fixed, probably in the IPS engine. Like I told, would not be the first time.

I also still wondering, why memory is still a problem in modern day equipment. What is the production cost of 2 Gb of memory? Maybe a dollar or 2. So why not put in 4 or 8 Gb, will make the FortiGate max $10,- more expensive.

NotMineAuthor

Explorer III

All great points @EME. Regarding the RAM - totally agree with you! 4GB should be bare minimum! If it is any consolation, I did implement the memory tweaks - they did not help. :)

However, it looks like I've found an acceptable 'workaround' for our environment: since we can all agree that FortiGuard updates trigger the Conserve Mode, I've scheduled daily update for 6AM. I've also created an automation stitch to restart the FortiGate each morning at 5:40AM, just to lower RAM usage 5-10% prior to the update.

swissroot

New Member

to be honest we are speaking about "enterprise solutions" even the entry level fortigates have an enterprise price tag. So in this segment I'm not discussing about rebooting it every day to prevent a conserve mode. This can be done in consumer hw with a consumer price tag but not in a business environment. Forti should check their firmwares and fix those flaws. I had a 61F for x years on the older trains of firmwares working with all features enabled without any conserve mode during the whole life of it. So it's possible and we are speaking still about the same feature set of AV/IDS/WEB and so on nothing really new and fancy. And it was even possible to do ssl-vpn on top without any issues :-).

EME

Explorer III

And so, it begins :(

Feedback from support:

Please note that based on the output provided, i can see that the firewall entered the conserve mode due to low memory issue caused by the IPS engine (AV failed to open).

Please refer to the following document that explains the cause behind this behavior and the remedy that you can implement to prevent this issue:
https://community.fortinet.com/t5/FortiGate/Technical-Tip-IPS-socket-size-and-fail-open-mode/ta-p/191254

Also, i would recommend to follow the document below since you're using a small series if the firewall that has a 2GB of RAM.
https://community.fortinet.com/t5/FortiGate/Technical-Tip-Steps-to-optimize-the-Memory-consumption/ta-p/192323

They keep throwing it at “only 2Gb” of memory.

Again, they push me to tweak the memory and now they also want me to configure my FortiGate in “fail-open”.

These “solutions” are driving me crazy.

Like @swissroot said, this is no way to handle customers that use Enterprise equipment. I should not be forced to degrade my security, to be able to maintain continuity,

Even this small unit at my home does maybe seem to them as a small customer, but they make the mistake that I work at a company that owns and manages more than 60 FortiGate’s in all sizes, with also FortiSwitches, FortiAP’s, FortiWeb, FortiManager and FortiAnalyser.

When is this sent from support to engineering, so they can say, O wait, we have a bug, here is the update and it works fine again?

jblyon

New Member

I got nowhere on a support call this morning. They want logs to try to correlate the issue with other identical reports that they've received (at least support finally admitted there were other reports.) The problem is the logs they want quit being logged when the unit hits the extreme memory threshold. I've run a script to collect the logs when the issue happens and they simply don't get recorded. It's a Catch 22...they want the logs to correlate the issue before sending it to engineering, but the logs don't exist because of the issue.

The sad thing is others with 100/200F units also seem to be experiencing high memory utilization, but those units have enough to basically handle the leaks and keep running. The 2GB units just don't have that luxury. The bug(s) isn't just limited to the 2GB models, they're just the only ones crashing due to it.

We're already planning to downgrade to 7.2.10 this weekend. We can't even schedule Fortiguard updates outside business hours now without the update crashing the Fortigate. We're just going to wing it this week without current Fortiguard definitions, which is NOT a position any business should need to be in.

This is being handled terribly. We're probably going to end up skipping 7.4 entirely. Boss just scheduled a meeting to discuss options to jump ship when our current licenses expire in a little over a year...

S2I

New Member

The blue line is the monitoring of RAM usage of my forti 61F.
I recently upgrade from 6.4.last to 7.4.last.... guess when ?
(about 15% more !!!)
Forit 61F RAM usage.png

NotMineAuthor

Explorer III

The scariest part about this picture is the obvious trend of increasing RAM usage... :D

dbhavsar

Staff

Hello Everyone,

Have you tried this workaround:
https://community.fortinet.com/t5/FortiGate/Technical-Tip-FortiGate-is-entering-into-Conserve-Mode-during/ta-p/287053

jblyon

New Member

It still hits conserve mode even on the mini database and with acceleration disabled.

We're also seeing other issues with 7.4.5 crop up now. Fortiguard at times will fail to update because it detects a self signed certificate in the chain of the factory hardware cert (which is properly registered to the serial number), showing spinning circles for the status of various Fortiguard licensed services, then suddenly it'll stop yelling about the cert, the status of the features will show normally, and updates will run.

We're also seeing spoke devices just randomly lose their BGP routes from the hubs. BGP will still be established, there's no errors logged, but the routes are just gone. Routes to other sites through the hub are still present and working, but the hub's local network routes just drop until we either reboot the spoke or forcefully rebuild BGP.

This is on top of the finally acknowledged IPSEC memory leak.

I just finished downgrade testing for 7.2.10. I doubt we'll entertain the idea of any 7.4 release anytime soon. I've never seen a product this badly broken this far into the release cycle, and I work for a Microsoft Partner...

EME

Explorer III

@jblyon,

Fortunately, I don't have BGP on my home firewall, but we do use it in the company, so another thing to consider.

My ticket @FotiSupport was escalated to senior Support on Monday, but I haven't heard back from them yet. Not even the statement, “The finally acknowledged IPSEC memory leak”.
I will wait some time for them and decide what to do.

EME

Explorer III

I have given up :(
Last Saturday I went back to 7.2.10 on my home firewall. Conserve Mode was usually set to 2-3 AM, due to scheduled FortiGuard updates, but I had also experienced that the WiFi (FortiAP) was not accepting new clients. After rebooting the firewall, it worked again. This was the final push to a rollback

Remyt

New Member

Same issue here on a big chunk of FGT40F and FGT60F's that we are running.

I've tried a few of the memory tricks:
IPS engine-worker 2
automation stitch to restart WAD once everyday
And disabling Security Rating.

For some gates this seems to work, other not. So far the issue is happening after work hours, so users are not affected, but I would like to remove the problem all together.

I do have open ticket with Fortinet, and I hope this gets a better fix in place soon.

Sign up

Login with SSO

Login to the community

Login with SSO

Scanning file for viruses.

This file cannot be downloaded