FortiGate 40F-3G4G v7.2.5 Build 1517 - FortiGate enters conservation mode and needs to be rebooted

[PSTransportation_NET]

Hello,

I'm not sure what changed recently. We have a total of 4 Fortigates; this has happened a total of 3 times so far on 2 of them, we expect this to continue to happen as the frequency of this has been increasing.

We have contacted support and they are unsure what is causing it, as everything looks correct to them. They are currently reviewing the logs and the configuration. Twice this week (One in the morning while employees were working and one of them occurred after hours) this has happened, we let it sit last night hoping it would exit conservation mode but it didn't (There were no employees working in the office).

When I look at SNMP, it looks like the memory usage hovers around 64% or so. We have IPS + AV + DNS + Webfilter + SSL inspection + APP control enabled on all our firewall policies (except the denies) and we log most events. We also have a very basic SD WAN setup. This hasn't been a issue until recently.

Can anyone offer any tips or guidance on how to fix this issue? Is there a known issue with this model/ version of firmware? 

[xshkurti]

@PSTransportation_NET 
Do you have some logs of which process is consuming most of your resources?
If the traffic you are using daily, or nothing else has happened but this changed suddenly, you have to check when your FortiOS updates happened recently (ips, av etc) 
Does the time of these updates correspond with when you started facing this issue?

[PSTransportation_NET]

Thank you for your reply. Unfortunately, we were unable to gather the processes that were consuming the most resources due to a problem with the Automation Stitch that support had a hard time figuring out why it wasn't working.

We have this set to update with Fortigate on a daily basis at 1 AM. Is there a way to check when they were installed as I can only see the date.

[xshkurti]

You can check them with
# get system auto-update versions
There will be last update time occurred: 

[PSTransportation_NET]

The output is what is listed below. These were all done in the morning, this problem with memory conservation mode happened at approximately 8:30 AM CST monday morning, 5:30 PM Wednesday Night. Anything we can get from this info?
================

FORTIGATE # get system auto-update versions

AV Engine
---------
Version: 6.00288 signed
Contract Expiry Date: Tue Apr 7 2026
Last Updated using manual update on Mon May 15 18:31:00 2023
Last Update Attempt: Wed Sep 27 01:27:58 2023
Result: No Updates

Virus Definitions
---------
Version: 91.07332 signed
Contract Expiry Date: Tue Apr 7 2026
Last Updated using scheduled update on Wed Sep 27 01:27:25 2023
Last Update Attempt: Wed Sep 27 01:27:58 2023
Result: No Updates

Extended set
---------
Version: 91.07332 signed
Contract Expiry Date: Tue Apr 7 2026
Last Updated using scheduled update on Wed Sep 27 01:27:25 2023
Last Update Attempt: Wed Sep 27 01:27:58 2023
Result: No Updates

Mobile Malware Definitions
---------
Version: 91.07332 signed
Contract Expiry Date: Tue Apr 7 2026
Last Updated using scheduled update on Wed Sep 27 01:27:25 2023
Last Update Attempt: Wed Sep 27 01:27:58 2023
Result: No Updates

IPS Attack Engine
---------
Version: 7.00322 signed
Contract Expiry Date: Tue Apr 7 2026
Last Updated using scheduled update on Wed Aug 23 01:58:33 2023
Last Update Attempt: Wed Sep 27 01:27:58 2023
Result: No Updates

IPS Config Script
---------
Version: 1.00010 signed
Contract Expiry Date: Tue Apr 7 2026
Last Updated using manual update on Mon Jan 23 16:42:00 2023
Last Update Attempt: Wed Sep 27 01:27:58 2023
Result: No Updates

Attack Definitions
---------
Version: 26.00644 signed
Contract Expiry Date: Tue Apr 7 2026
Last Updated using scheduled update on Wed Sep 27 01:27:25 2023
Last Update Attempt: Wed Sep 27 01:27:58 2023
Result: No Updates

Attack Extended Definitions
---------
Version: 0.00000
Contract Expiry Date: Tue Apr 7 2026
Last Updated using manual update on Mon Jan 1 00:00:00 2001
Last Update Attempt: n/a
Result: Updates Installed

Application Definitions
---------
Version: 26.00644 signed
Contract Expiry Date: Tue Apr 7 2026
Last Updated using scheduled update on Wed Sep 27 01:27:25 2023
Last Update Attempt: Wed Sep 27 01:27:58 2023
Result: No Updates

Industrial Attack Definitions
---------
Version: 26.00644 signed
Contract Expiry Date: Tue Apr 7 2026
Last Updated using scheduled update on Wed Sep 27 01:27:25 2023
Last Update Attempt: Wed Sep 27 01:27:58 2023
Result: No Updates

IPS Malicious URL Database
---------
Version: 4.00829 signed
Contract Expiry Date: Tue Apr 7 2026
Last Updated using scheduled update on Wed Sep 27 01:27:25 2023
Last Update Attempt: Wed Sep 27 01:27:58 2023
Result: No Updates

IoT Detect Definitions
---------
Version: 26.00644 signed
Contract Expiry Date: Tue Apr 7 2026
Last Updated using scheduled update on Wed Sep 27 01:27:25 2023
Last Update Attempt: Wed Sep 27 01:27:58 2023
Result: No Updates

Flow-based Virus Definitions
---------
Version: 91.07332 signed
Contract Expiry Date: Tue Apr 7 2026
Last Updated using scheduled update on Wed Sep 27 01:27:25 2023
Last Update Attempt: Wed Sep 27 01:27:58 2023
Result: No Updates

Botnet Domain Database
---------
Version: 3.00467 signed
Contract Expiry Date: Tue Apr 7 2026
Last Updated using scheduled update on Wed Sep 27 01:27:25 2023
Last Update Attempt: Wed Sep 27 01:27:58 2023
Result: No Updates

Internet-service Standard Database
---------
Version: 7.03391 signed
Contract Expiry Date: n/a
Last Updated using manual update on Tue Sep 26 17:36:00 2023
Last Update Attempt: n/a
Result: Updates Installed

Device and OS Identification
---------
Version: 1.00157
Contract Expiry Date: Tue Apr 7 2026
Last Updated using scheduled update on Sat Sep 23 01:57:48 2023
Last Update Attempt: Wed Sep 27 01:27:58 2023
Result: No Updates

URL Allow list
---------
Version: 3.00999
Contract Expiry Date: Tue Apr 7 2026
Last Updated using scheduled update on Wed Sep 27 01:27:44 2023
Last Update Attempt: Wed Sep 27 01:27:58 2023
Result: No Updates

IP Geography DB
---------
Version: 3.00195
Contract Expiry Date: n/a
Last Updated using manual update on Tue Sep 19 21:15:00 2023
Last Update Attempt: n/a
Result: Updates Installed

Certificate Bundle
---------
Version: 1.00046
Contract Expiry Date: n/a
Last Updated using manual update on Tue Aug 22 19:07:00 2023
Last Update Attempt: n/a
Result: Updates Installed

Malicious Certificate DB
---------
Version: 1.00448
Contract Expiry Date: Tue Apr 7 2026
Last Updated using scheduled update on Tue Sep 26 01:59:03 2023
Last Update Attempt: Wed Sep 27 01:27:58 2023
Result: No Updates

Mac Address Database
---------
Version: 1.00185
Contract Expiry Date: Tue Apr 7 2026
Last Updated using scheduled update on Wed Sep 27 01:27:25 2023
Last Update Attempt: Wed Sep 27 01:27:58 2023
Result: No Updates

AntiPhish Pattern DB
---------
Version: 1.00012
Contract Expiry Date: Tue Apr 7 2026
Last Updated using manual update on Mon Apr 17 08:52:52 2023
Last Update Attempt: Wed Sep 27 01:27:58 2023
Result: No Updates

AI/Machine Learning Malware Detection Model
---------
Version: 2.12831 signed
Contract Expiry Date: Tue Apr 7 2026
Last Updated using scheduled update on Wed Sep 27 01:27:25 2023
Last Update Attempt: Wed Sep 27 01:27:58 2023
Result: No Updates

Inline CASB Database
---------
Version: 1.00031 signed
Contract Expiry Date: Tue Apr 7 2026
Last Updated using scheduled update on Thu Jun 22 01:57:02 2023
Last Update Attempt: Wed Sep 27 01:27:58 2023
Result: No Updates

Modem List
---------
Version: 1.048

Security Rating Data Package
---------
Version: 4.00046
Contract Expiry Date: Tue Apr 7 2026
Last Updated using scheduled update on Tue Sep 12 01:58:03 2023
Last Update Attempt: Wed Sep 27 01:27:58 2023
Result: No Updates

FDS Address
---------
x.x.x.x:x

[xshkurti]

@PSTransportation_NET 

On most of them i see:
Last Updated using scheduled update on Wed Sep 27 01:27:25 2023

Since you have an open ticket, try to ask for another IPS engine and install it manually and monitor the situation.

[PSTransportation_NET]

Thank you so much for your assistance, I will check with support on this.