Technical Tip: Free up memory to avoid conserve mode

xshkurti · ‎12-29-2022

Description

This article describes how to free up memory to avoid FortiGate entering conserve mode (Technical Tip: How conserve mode is triggered) when its resources are highly utilized.

Scope

FortiGate.

Solution

FortiGate goes into a conserve mode state as a self-protection mechanism when system memory is highly utilized and reaches a specific threshold.

When FortiGate enters conserve mode, it activates protection measures to recover some memory space.

When enough memory is recovered, it exits the conserve mode state and deactivates the previous state.

Three memory thresholds can be configured:

config system global

set memory-use-threshold-extreme <integer>

set memory-use-threshold-green <integer>

set memory-use-threshold-red <integer>

end

Those are in percent of total memory. And have the following values: minimum 70, maximum 97, and default as below:

memory-use-threshold-extreme - Threshold at which memory usage is considered extreme. Default value: 95

memory-use-threshold-red - Threshold at which memory usage forces the FortiGate to enter conserve mode. Default value: 88

memory-use-threshold-green - Threshold at which memory usage forces the FortiGate to exit conserve mode. Default value: 82

Reference: config system global

In some cases, it may be necessary to increase the memory conserve mode thresholds to higher values to avoid going into conserve mode too early or to work around a known issue.

This article describes how conserve mode is triggered: Technical Tip: How conserve mode is triggered - Fortinet Community.

Follow the steps below to manually free memory:

Reduce memory logging.

Memory logging and reporting, 'GUI -> Logging and Reporting'., is one of the most intensive memory uses (especially if the log is also used in firewall policies for 'all traffic'). Try to configure on each policy if possible, the Log for 'security events' or 'UTM' instead of 'ALL'.

Try to Log in to an external device, like FortiAnalyzer or Syslog, whenever possible. This is not just a way to reduce the memory footprint on FortiGate but is also a more secure method.

As such logs are difficult to tamper and compromise in case of security events. They could also contain much longer log history (depending on external device log disk capacity and settings). Or be re-used in some SIEM systems to detect, or even predict, security events.

Pay attention to:

config log memory setting
set status <enable/disable>
end

config log memory filter --> And settings in there.
config log setting

Here pay attention to global log settings like the below and consider if it is truly necessary for example that the implicit policy hits logged:

fwpolicy-implicit-log.
resolve-ip / resolve-port.
extended-log.
log-policy-comment.

Reference: CLI reference log 7.2.6

Reduce Security Rating submissions:

config system global
set security-rating-result-submission disable
set security-rating-run-on-schedule disable
end

Set the antivirus database to normal: Reduce sizes of local copies of used FortiGuard databases.

config antivirus settings
set use-extreme-db disable
end

Shift databases updates to off-peak hours.

Set updates to off-peak hours: (the example uses 3 AM, feel free to change as appropriate for the expected environment).

config system autoupdate schedule
set frequency daily
set time 03:00
end

Switch the ISDB database to be used only when needed - on-demand.

config sys global
set internet-service-database on-demand
end
execute update-ffdb-on-demand

Or:

Switch the ISDB database to keep a small-sized Internet Service database with very limited IP addresses.

config sys global
set internet-service-database mini
end

execute update-now

Optimize session timers for TCP and UDP Traffic.

In this way, FortiGate will wait a shorter time for sessions to close.

config system global

set tcp-halfclose-timer 30 <--- Default is 120 sec.

set tcp-halfopen-timer 5 <--- Default is 10 sec.

set tcp-timewait-timer 0 <--- Default is 1 sec.

set udp-idle-timer 60 <--- Default is 180 sec.

end

Optimize session TTL settings so sessions stay active for less time on FortiGate:

config system session-ttl

set default 300 <---- The default value is 3600.

end

To specify different values for different protocols, configure them as below.

In the following example for DNS requests, the FortiGate will wait for a different length of time than the default value specified above:

config system session-ttl

config port
edit 0
set protocol 17
set timeout 10
set end-port 53
set start-port 53
end
end

For steps 1 and 2, additionally, refer to this article: Technical Note: Session TTL values and Policy RST for Sessions

Reduce system service caches and free the space for other processes that require it:

config system dns
set dns-cache-limit 600 <--- The default value is 5000.

end

config system fortiguard
set webfilter-cache-ttl 600 <--- The default value is 3600.

set antispam-cache-ttl 600 <--- The default value is 1800.

end

Reduce system workers count:

config system global
set miglogd-children 1
set sslvpn-max-worker-count 1
set wad-worker-count 1
set scanunit-count 2
end

diag test application ipsmonitor 99
config ips global
set engine-count 2
set cp-accel-mode none
set exclude-signatures none

end

Reduce the maximum file size to inspect, the default is 10MB, but it can be reduced to 2-3MB.

config firewall profile-protocol-options

edit <profile name>

config [http|ftp|pop3|smtp|imap]

set over size-limit <MB>

end

Refer to the following articles for more information:

Technical Tip: Steps on how to optimize Memory consumption

Technical Note: Memory optimization techniques for FortiOS

When enabled, IPS may consume a lot of resources. The following article describes how to optimize IPS engine values:
Technical Tip: IPS memory optimization steps - Fortinet Community

Another effective way to reduce the overall memory usage of a device is to lower the amount of workers running. For more details, refer to the following article:

Technical Tip: Reduce memory usage by reducing the number of spawned daemons

FortiGates with memory usage already high might activate memory conserve mode during FortiGuard updates. As the databases grow with new objects and IP addresses on each update, the recommendation is to perform ISDB updates during quieter periods when memory usage is anticipated to be lower on the unit.

With the update implemented in October 2023, the size of the ISDB has surged by 30%. Consequently, there is an elevated risk of the system entering conserve mode, particularly on lower-end FortiGate hardware units that are already experiencing high memory consumption. Refer to the following article:

Technical Tip: FortiGate is entering into Conserve Mode during FortiGuard Updates

Note:
If the steps above do not produce satisfactory results, consider using a higher-capacity FortiGate device.

Switch the ISDB database to be used only when needed

Related articles:
Technical Tip: How to optimize the Memory consumption

Technical Tip: Script for reducing memory usage in small FortiGates experiencing conserve mode
Technical Note: Priority of session-ttl settings in FortiGate
Technical Tip: Session timeout settings

Technical Tip: Free up memory to avoid conserve mode

You are leaving our website