I have fortigate 1101E version 7.2.4 after updating the IPSEngine signature database to 7.322, it started behaving strangely, momentarily an ipsengine process triggers the consumption of RAM memory causing fortigate to quickly go into conserve mode .
The event happens so quickly that it is not even possible to collect evidence. Entering conserve mode, I have a network drop and fortigate is inaccessible both via mgmt and via console. We only managed to stabilize again after the manual reboot.
At the moment we have created a auto-script as a temporary service continuity measure to automatically restart the IPSEngine process every 2 minutes.
config system auto-script edit "restart_ips" set interval 120 set repeat 0 set start auto set script "fnsysctl killall ipsmonitor"
-Crashlogs and memory logs would help further identify issues.
get system performance status (run this command 5 times in intervals of 1 minute) diag sys top 1 10(Run for 30 Sec and CTRL C to stop) diag debug crashlog read | grep 2023---check for any crash triggered
-Can you please for the time being disable logging on firewall policy and use the below article steps to optimize memory
Unfortunately I am not able to extract logs since when fortigate enters conserve mode nothing works, neither access via the mgmt interface nor via the console and after the manual reboot we lose all logs and fortigate returns to its normal state. Ipsengine processes in the normal state of fortigate consume few resources, however, in just a few minutes, a process triggers RAM consumption and fortigate crashes. In this sense, how can I act to collect the logs?
You can configure the script to collect the logs as advice by mpeddalla. As you have mentioned when the issue occurs the device is not responsive in such situation you can use com log feature to collect the log. Below is the Kb link for comlog. I hope it will be helpful.
Did you ever get this solved? I have the same issue run since upgrading to 7.2.5.
I have an email stitch setup to send sys top information and sometimes if the system can send the email out before it crashes I see it is the ipsengine eating up all available memory.
This happens really fast, as quick as a couple minutes from using its normal amount of memory to causing enough to go into conserve mode then eventually rebooting.
We ended up moving all of the SSL VPNs off of this system and setup another stitch to reboot the system so the standby can take over before it gets to conserve mode.
Right now we are at a crossroads because he have had nothing but bugs since 7.2.1 where every upgrade until 7.2.5 is making our system noticeably worse. At this point we are too apprehensive to update to 7.2.6 as it might cause even more problems.
Opening a TAC case doesn't do much because they ask for logs that are not logged when the system is crashing and you cannot force the error to happen as it is random.
The Fortinet Security Fabric brings together the concepts of convergence and consolidation to provide comprehensive cybersecurity protection for all users, devices, and applications and across all network edges.