Technical Tip: FortiGate Enters into Memory Conserve Mode During FortiGuard Updates

vpalli · ‎11-30-2023

Description

This article describes the factors that lead to FortiGate entering Conserve Mode during scheduled or manual FortiGuard updates.

Scope

FortiGate.

Solution

'desktop-tier FortiGates' will have baseline memory usage levels at 64-72% or higher (especially when security inspection is being utilized), and so they can easily trigger memory conserve mode during FortiGuard updates.

This is especially true for models such as the FortiGate-30E, 40F, 60E, and 60F (i.e., units with less than or equal to 2GB of RAM). Refer to the Form Factor field in the product datasheet to identify the hardware type/form factor.

Form Factor.png

During the FortiGuard update process, it is possible to see system memory consumption increase by as much as 10-20%, and this can easily result in conserve mode being triggered (the default trigger entry threshold is 88% memory usage). Instances of conserve mode are especially evident during the following stages:

Downloading and installing IPS Attack Definition database updates (particularly when IPS Acceleration is enabled).
Downloading the Internet Service Database and other database objects, which require extraction and processing during updates.

Part of this increased memory consumption during updates has been due to the various FortiGuard databases increasing in size over time as they are expanded upon (i.e., added IP addresses, expanded signature lists, etc.). For example, updates implemented in October 2023 resulted in the size of the Internet Service DB increasing by about 30%, and this has inadvertently elevated the risk of FortiGates entering conserve mode during updates, particularly on FortiGates already experiencing high baseline memory consumption.

General Notes:

For FortiGate-VMs, ensure to allocate at least 4 GB of RAM to the system. This greatly reduces the risk of entering conserve mode during FortiGuard updates by simply ensuring that there is more free memory available.
Enhancements in FortiOS v7.2.11, v7.4.8, and v7.6.1 aim to reduce the likelihood of conserve mode occurring during FortiGuard updates by adding additional logic to the process (for example, further optimizing the update process, checking available memory usage before starting updates, etc.). This enhancement is tracked as part of Issue #1057131.

Recommended Changes to reduce the risk of triggering Conserve Mode during FortiGuard updates:

Configure the FortiGuard update schedule to take place during non-production hours. FortiGuard databases are increasing in size over time as they are improved, and so it can be a good idea to perform Internet Service DB updates outside of business hours for all FortiGate models (desktop, campus, and datacenter-tier) since baseline memory usage is typically lower at that time. Setting updates to occur once-daily outside of business hours can be a good starting point, but is not the only option. The following example sets the update schedule to occur once weekly between 3-4 AM on Sundays:

config system autoupdate schedule
set frequency weekly
set time 03:60 <----- If '60' is set in the minutes field, it will randomly run within 1 hour.
set day Sunday
end

Low-end platforms like 60x, 40x are known for memory problems, since these devices are installed with low memory. During FortiGuard updates, these devices may suffer from conserve mode due to a lack of memory.

In order to avoid conserving mode, the number of workers can be reduced, since every worker comes with a certain memory overhead attached to it. The following configuration should be implemented during a maintenance window and carefully monitored during production hours for any performance impact. For guidance on configuring the engine count, refer to the NOTE in the article Technical Tip: Optimizing Memory Usage by Limiting Spawned Daemons

The recommendation for low-end devices is as follows:

config system global
    set miglogd-children 1
    set sslvpn-max-worker-count 1
    set wad-worker-count 1
    set scanunit-count 2
end
config ips global
    set engine-count 1
end

Security Ratings:
The security rating result submission is enabled by default on the FortiGate. This feature enables the submission of security rating results to FortiGuard servers for data collection purposes and continuous learning.
The feature is memory-intensive and could lead to high memory usage observed on the node process, which can be disabled using the commands below.

config system global
set security-rating-result-submission disable
set security-rating-run-on-schedule disable
end

In Global config, check if the command 'set gui-proxy-inspection enable' is present. If any proxy features are not being used, it is recommended to disable this, because WAD will allocate memory resources if this command is enabled, even when the proxy is not being used.

Disable IPS Acceleration (aka set cp-accel-mode none), especially on FortiGates with <= 2GB of RAM.

During a FortiGuard update, the ipshelper process can consume as much as 20% of available system memory on desktop FortiGates for 5-10 seconds, and so it is a leading cause of conserve-mode during FortiGuard updates. This memory demand occurs when the IPS signature database is updated, as the ipshelper process needs to recompile the database before uploading it to the onboard Content Processor (CP).

To disable this feature (and thus prevent this memory consumption from occurring), use the following CLI commands:

config ips global

set cp-accel-mode none

end

In previous iterations of this article, this option was only a suggestion. It is now highly recommended to implement this for FortiGates with <= 2GB RAM, such as the FortiGate 40F/41F and 60F/61F, and in fact, it is now the default behavior as of FortiOS v7.6 (see: FortiOS v7.6 Release Notes). The following KB article discusses the cp-accel-mode option in greater depth: Troubleshooting Tip: Conserve mode due to ipshelper in lower end models.

Modify the Internet Service database (FFDB) updates to the mini or on-demand versions (v7.2.4 and above) or disable updates to the FFDB.
The recommended option is to use the on-demand Internet Service database, as this option allows admins to only download/update Internet Service objects that are actively being used in the configuration. This reduces the amount of memory required to perform updates, though it is only available as of v7.2.4 and later versions:

config system global
set internet-service-database on-demand
end

In earlier versions, the alternative option is to use the mini version of the Internet Service database, which reduces the number of available services to the most common selection (thus reducing the size of updates):

config system global
set internet-service-database mini
end

Updates for the Internet Service database can be fully disabled if required.

config system fortiguard
set update-ffdb disable <----- Enable to update the Internet Service Database only during low memory usage.
end

For more information, refer to the following KB article regarding the Internet Service database: Technical Tip: Internet-service-database: On-demand.

To check which version of the database, run the following command:

diagnose autoupdate version

Additional Optimization:

The above changes will make the largest difference for reducing/preventing conserve-mode caused by FortiGuard updates.

For additional optimization suggestions, refer to the following KB articles:

Technical Tip: Steps on how to optimize Memory consumption

Technical Tip: Free up memory to avoid conserve mode

Note:

From v7.6.3, to enhance the stability of physical FortiGate devices with 2 GB RAM, the Security Rating feature and Security Fabric topology visibility have been removed.

2 GB RAM FortiGate models no longer support Security Rating and Security Fabric topology

Related articles:

Technical Tip: FortiGate Enters into Memory Conserve Mode During FortiGuard Updates

You are leaving our website