Process IPSEngine High Memory

[DaVeiga19]

I have fortigate 1101E version 7.2.4 after updating the IPSEngine signature database to 7.322, it started behaving strangely, momentarily an ipsengine process triggers the consumption of RAM memory causing fortigate to quickly go into conserve mode .

The event happens so quickly that it is not even possible to collect evidence.
Entering conserve mode, I have a network drop and fortigate is inaccessible both via mgmt and via console.
We only managed to stabilize again after the manual reboot.

At the moment we have created a auto-script as a temporary service continuity measure to automatically restart the IPSEngine process every 2 minutes.

config system auto-script
edit "restart_ips"
set interval 120
set repeat 0
set start auto
set script "fnsysctl killall ipsmonitor"

[js2]

Hi @DaVeiga19,

You can check by adjusting the engine count and socket size.

config ips global

set engine-count 2 [integer, 0-255]
set socket-size 16 or 32 [integer, 0-512]
end

Refer link:
https://community.fortinet.com/t5/FortiGate/Technical-Tip-IPS-memory-optimization-steps/ta-p/197486

If the issue persists I would suggest to open a TAC case and share these logs for further investigation.

# Issue Timestamp
# System event logs around this time.
# Run the below commands

get system performance status (run this command 5 times in interval of 1 minutes)
diag sys top 1 40(Run for 30 Sec and CTRL C to stop)
diag debug crashlog read ---check for any crash triggered

Also collect the tac report

# execute tac report

Regards,
Joshi

[mpeddalla]

Hello Daveiga19,

Thank you for reaching on the Fortinet forum.

As suggested earlier by Joshi please collect all logs and would suggest opening a ticket with TAC for log analysis further if it matches with any known issues reported.

-As of now i couldn't see any known issues on release notes for ips memory 

https://docs.fortinet.com/document/fortigate/7.2.4/fortios-release-notes/236526/known-issues

-Crashlogs and memory logs would help further identify issues.

get system performance status (run this command 5 times in intervals of 1 minute)
diag sys top 1 10(Run for 30 Sec and CTRL C to stop)
diag debug crashlog read | grep 2023---check for any crash triggered

-Can you please for the time being disable logging on firewall policy and use the below article steps to optimize memory 

Technical Tip: IPS memory optimization steps - Fortinet Community

https://community.fortinet.com/t5/FortiGate/Technical-Tip-Reduce-memory-usage-by-reducing-the-number...

https://community.fortinet.com/t5/FortiGate/Technical-Tip-Free-up-memory-to-avoid-conserve-mode/ta-p...

https://community.fortinet.com/t5/FortiGate/Technical-Tip-Steps-on-how-to-optimize-Memory-consumptio...

Best regards,

Manasa.

[DaVeiga19]

Hi Joshi,

Hi Manasa,

Unfortunately I am not able to extract logs since when fortigate enters conserve mode nothing works, neither access via the mgmt interface nor via the console and after the manual reboot we lose all logs and fortigate returns to its normal state.
Ipsengine processes in the normal state of fortigate consume few resources, however, in just a few minutes, a process triggers RAM consumption and fortigate crashes.
In this sense, how can I act to collect the logs?

Best Regards,

Da Veiga

[mpeddalla]

Hello Daveiga19, 

Thank you for the reply.

You can configure script logs which would collect logs and send email alerts.

https://community.fortinet.com/t5/FortiGate/Technical-Tip-Execute-a-CLI-script-based-on-HIGH-memory/...

https://docs.fortinet.com/document/fortigate/6.2.15/cookbook/702937/execute-a-cli-script-based-on-cp...

Regards,

Manasa.

[pjawalekar]

Hi DaVeiga19,

You can configure the script to collect the logs as advice by mpeddalla.
As you have mentioned when the issue occurs the device is not responsive in such situation you can use com log feature to collect the log. Below is the Kb link for comlog. I hope it will be helpful.

https://community.fortinet.com/t5/FortiGate/Technical-Tip-How-to-use-the-COMLog-feature/ta-p/195390

Regards,

Pratik

[DaVeiga19]

Hello Manasa,

By configuring this automation stiches, if fortigate enters conserve mode will these event logs be available? Or will the event logs be sent by email as well?

Because my main problem is when it is in conserve mode, I lose all records after the manual reboot.
However, I will follow the suggested recommendations.

Best Regards

Da Veiga

[mpeddalla]

Hello @DaVeiga19 ,

Yes, you are correct it should send you the log as an email but as the screen doesn't stay can also parallelly log suggestions by my @pjawalekar comlog 

https://community.fortinet.com/t5/FortiGate/Technical-Tip-How-to-use-the-COMLog-feature/ta-p/195390

Regards,

Manasa

[aguerriero]

Did you ever get this solved? I have the same issue run since upgrading to 7.2.5.

I have an email stitch setup to send sys top information and sometimes if the system can send the email out before it crashes I see it is the ipsengine eating up all available memory.

This happens really fast, as quick as a couple minutes from using its normal amount of memory to causing enough to go into conserve mode then eventually rebooting.

We ended up moving all of the SSL VPNs off of this system and setup another stitch to reboot the system so the standby can take over before it gets to conserve mode.

Right now we are at a crossroads because he have had nothing but bugs since 7.2.1 where every upgrade until 7.2.5 is making our system noticeably worse. At this point we are too apprehensive to update to 7.2.6 as it might cause even more problems.

Opening a TAC case doesn't do much because they ask for logs that are not logged when the system is crashing and you cannot force the error to happen as it is random.