I have fortigate 1101E version 7.2.4 after updating the IPSEngine signature database to 7.322, it started behaving strangely, momentarily an ipsengine process triggers the consumption of RAM memory causing fortigate to quickly go into conserve mode .
The event happens so quickly that it is not even possible to collect evidence.
Entering conserve mode, I have a network drop and fortigate is inaccessible both via mgmt and via console.
We only managed to stabilize again after the manual reboot.
At the moment we have created a auto-script as a temporary service continuity measure to automatically restart the IPSEngine process every 2 minutes.
config system auto-script
edit "restart_ips"
set interval 120
set repeat 0
set start auto
set script "fnsysctl killall ipsmonitor"
Nominating a forum post submits a request to create a new Knowledge Article based on the forum post topic. Please ensure your nomination includes a solution within the reply.
Hi @DaVeiga19,
You can check by adjusting the engine count and socket size.
config ips global
set engine-count 2 [integer, 0-255]
set socket-size 16 or 32 [integer, 0-512]
end
Refer link:
https://community.fortinet.com/t5/FortiGate/Technical-Tip-IPS-memory-optimization-steps/ta-p/197486
If the issue persists I would suggest to open a TAC case and share these logs for further investigation.
# Issue Timestamp
# System event logs around this time.
# Run the below commands
get system performance status (run this command 5 times in interval of 1 minutes)
diag sys top 1 40(Run for 30 Sec and CTRL C to stop)
diag debug crashlog read ---check for any crash triggered
Also collect the tac report
# execute tac report
Regards,
Joshi
Hello Daveiga19,
Thank you for reaching on the Fortinet forum.
As suggested earlier by Joshi please collect all logs and would suggest opening a ticket with TAC for log analysis further if it matches with any known issues reported.
-As of now i couldn't see any known issues on release notes for ips memory
https://docs.fortinet.com/document/fortigate/7.2.4/fortios-release-notes/236526/known-issues
-Crashlogs and memory logs would help further identify issues.
get system performance status (run this command 5 times in intervals of 1 minute)
diag sys top 1 10(Run for 30 Sec and CTRL C to stop)
diag debug crashlog read | grep 2023---check for any crash triggered
-Can you please for the time being disable logging on firewall policy and use the below article steps to optimize memory
Technical Tip: IPS memory optimization steps - Fortinet Community
Best regards,
Manasa.
Hi Joshi,
Hi Manasa,
Unfortunately I am not able to extract logs since when fortigate enters conserve mode nothing works, neither access via the mgmt interface nor via the console and after the manual reboot we lose all logs and fortigate returns to its normal state.
Ipsengine processes in the normal state of fortigate consume few resources, however, in just a few minutes, a process triggers RAM consumption and fortigate crashes.
In this sense, how can I act to collect the logs?
Best Regards,
Da Veiga
Hello Daveiga19,
Thank you for the reply.
You can configure script logs which would collect logs and send email alerts.
Regards,
Manasa.
Hi DaVeiga19,
You can configure the script to collect the logs as advice by mpeddalla.
As you have mentioned when the issue occurs the device is not responsive in such situation you can use com log feature to collect the log. Below is the Kb link for comlog. I hope it will be helpful.
https://community.fortinet.com/t5/FortiGate/Technical-Tip-How-to-use-the-COMLog-feature/ta-p/195390
Regards,
Pratik
Hello Manasa,
By configuring this automation stiches, if fortigate enters conserve mode will these event logs be available? Or will the event logs be sent by email as well?
Because my main problem is when it is in conserve mode, I lose all records after the manual reboot.
However, I will follow the suggested recommendations.
Best Regards
Da Veiga
Hello @DaVeiga19 ,
Yes, you are correct it should send you the log as an email but as the screen doesn't stay can also parallelly log suggestions by my @pjawalekar comlog
https://community.fortinet.com/t5/FortiGate/Technical-Tip-How-to-use-the-COMLog-feature/ta-p/195390
Regards,
Manasa
Did you ever get this solved? I have the same issue run since upgrading to 7.2.5.
I have an email stitch setup to send sys top information and sometimes if the system can send the email out before it crashes I see it is the ipsengine eating up all available memory.
This happens really fast, as quick as a couple minutes from using its normal amount of memory to causing enough to go into conserve mode then eventually rebooting.
We ended up moving all of the SSL VPNs off of this system and setup another stitch to reboot the system so the standby can take over before it gets to conserve mode.
Right now we are at a crossroads because he have had nothing but bugs since 7.2.1 where every upgrade until 7.2.5 is making our system noticeably worse. At this point we are too apprehensive to update to 7.2.6 as it might cause even more problems.
Opening a TAC case doesn't do much because they ask for logs that are not logged when the system is crashing and you cannot force the error to happen as it is random.
Select Forum Responses to become Knowledge Articles!
Select the “Nominate to Knowledge Base” button to recommend a forum post to become a knowledge article.
User | Count |
---|---|
1702 | |
1092 | |
752 | |
446 | |
228 |
The Fortinet Security Fabric brings together the concepts of convergence and consolidation to provide comprehensive cybersecurity protection for all users, devices, and applications and across all network edges.
Copyright 2024 Fortinet, Inc. All Rights Reserved.