Description
The article describes how to check if a FortiGate is going into conserve mode due to an increased ISDB database.
In October 2023, the size of the ISDB increased significantly raising the risk of lower-end FortiGate hardware units entering conserve mode.
This is especially concerning for the lower-end unit, given that base memory usage hovers around 70-75% in these lower-end models. Even a small increase could trigger conserve mode.
Scope
The problem was observed in the 61E lower-end model. Nevertheless, the FortiGate models listed below also come with less than 2 GB of RAM.
FortiGate models ranging from 30D/30E/40C/40F/50E/60D/60E/61E/60F/61F/70D/80C/80D/80E/81E/90D/91Eand 92D.
Solution
To check memory usage, run 'diagnose hardware sysinfo memory' in both normal and conserve modes. Download the outputs and compare them side by side, focusing on:
- MemFree.
- Cached.
- Active.
- Slab.
Let's break down these memory types a bit before moving on.
Cache Memory:
Cache memory is primarily used to store copies of frequently accessed data, making it faster to retrieve that data when needed.
Active Memory:
Active memory is the portion currently in use by the system's processes. It includes the memory used by running applications and the kernel.
Slab:
The term 'slab' refers to a memory management mechanism employed by the kernel to efficiently allocate and deallocate small memory chunks.
The Slab Allocator, part of the Linux kernel's memory management subsystem, aims to enhance memory allocation performance for frequently used data structures.
A slab is a collection of pre-allocated, contiguous memory blocks of the same size, with each block used to store an instance of a specific data structure.
Let's observe a sample output of a use case scenario when the device was in its regular state, as compared to the output during conserve mode:
MemFree: 454508 kB <----- normal state == 456 MB
MemFree: 44464 kB <----- conserve state == 45 MB
Difference == 411 MB
Cached: 572924 kB <----- normal state == 573 MB
Cached: 744376 kB <----- conserve state == 744 MB
Difference == 171 MB (significant)
Active: 769540 kB <----- normal state == 770 MB
Active: 841972 kB <----- conserve state == 842 MB
Difference == 60 MB (not significant)
Slab: 106992 kB <----- normal state == 107 MB
Slab: 343040 kB <----- conserve state == 343 MB
Difference == 236 MB (significant)
Adding the differences:
171 MB (cached) + 60 MB (active) + 236 MB (slab) = 467 MB, which is quite close to the difference in memory between the normal state and conserved state.
These are the two extensively utilized memory spaces in conserve mode, and both play a role in managing cache memory. In FortiGate, cache memory includes MEMORY LOGGING, WEB FILTER CACHE, LOG queues, and also the UPDATE database cache.
Examining the crash log, it is evident that the FortiGate enters conserve mode for less than 10 seconds. Within this timeframe, the system event logs indicate FortiGuard updates for ISDB are being performed.
2023-11-03 18:20:44 green="1530 MB" msg="Kernel enters memory conserve mode
2023-11-03 18:20:46 service=kernel conserve=exit total="1866 MB" used="1522 MB" red="1642 MB"
Apply the following modifications to prevent FortiGate from entering conserve mode:
- Schedule an update outside of normal business hours:
config system autoupdate schedule
set frequency daily
set time 03:00 -> Non production hours
end
- Reduce cache timeouts:
config system fortiguard
set webfilter cache-ttl
set antispam cache-ttl
set outbreak-prevention-cache-ttl
Verify and change:
set webfilter-cache-ttl 600 <----- The default value is 3600.
set antispam-cache-ttl 600 <----- The default value is 1800.
set outbreak-prevention-cache-ttl 300 <----- The default value is 300. Verify and set to 300 if it is not.
- Reduce DNS cache size:
config system dns
show full-configuration | grep cache
set dns-cache-limit 600 <----- The default value is 5000.
set dns-cache-ttl 600 <----- The default value is 1800.
- Reduce TCP session timeouts: Reduce the value according to the type of traffic passing through the FortiGate:
config system global
show full-configuration | grep tcp
set reset-sessionless-tcp disable
set tcp-halfclose-timer 30 <----- The default value is 120 sec.
set tcp-halfopen-timer 5 <----- The default value is 10 sec.
set tcp-option enable <----- The default value is enable.
set tcp-timewait-timer 0 <----- The default value is 1 sec.
set tcp-rst-timer 5 <----- The default value is 5 sec. Verify and set to 5 if it is not already.
set udp-idle-timer 60 <----- The default value is 180 sec.
- Limit logging to essential events by adjusting log settings or disabling memory logging if using devices like FortiAnalyzer, FortiGate Cloud, and Syslog servers.
- Reduce worker count: Reduce the worker count accordingly if FortiGate is using any of the processes listed below. Be careful before changing the values. Refer to the following KB article for more information: Technical Tip: Reduce memory usage by reducing the number of spawned daemons.
config system global
set miglogd-children 1
set sslvpn-max-worker-count 1
set wad-worker-count 1
set scanunit-count 2
end
- Increase the conserve mode threshold accordingly: Technical Tip: Conserve mode changes in FortiGate 5.6 and above.
- Update FortiGuard when the memory is below the conserve mode thresholds:
config system fortiguard
set update-ffdb disable <----- Enable to update the Internet Service Database only during low memory usage.
end
- Disable CP acceleration:
config ips global
set cp-accel-mode none
end
Related document:
Default setting of cp-accel-mode is changed to none on 2GB memory models
- Check if the report function is enabled. The report function should not be enabled for 2Gb or 4Gb models as it can consume more memory. These settings can be verified from the following commands:
config report setting
set pdf-report disable
set fortiview disable
end
Delete all Report Runner results after this using the command below:
diagnose report-runner clean
- Set the ISDB to 'on-demand'. In this way, the firewall will download only the ISDB entries that will be used in policies and other configurations.
config sys global
set internet-service-database on-demand <-- The default value is standard.
end
After making this change, the following warning will appear:
Warning: Changing Internet Service database update mode will lead to the removal of all downloaded Internet Service files.
Do you want to continue? (y/n) y
Once this is confirmed, an additional message will appear:
Please run command "execute update-ffdb-on-demand" to manually initiate a download or wait for the automatic schedule update for on-demand Internet Service database. Additionally, please ensure to perform this action when enabling or changing internet service in firewall policy, route and SD-WAN settings, etc.
Note 1:
To improve performance and reduce memory-related issues on low-end devices with 2GB RAM or less than 2GB RAM, FortiOS no longer supports proxy-related features from the 7.4.4 version onwards. This only applies to FortiGate/FortiWifi 2GB RAM devices.
For more information, see this document: Proxy-related features no longer supported on FortiGate 2 GB RAM models.
Note 2:
On v7.6.3, further optimizations were made that remove or rework certain features on devices with 2GB or less of memory.
For further details, consult Optimizations for physical FortiGate devices with 2 GB RAM 7.6.3.
Note 3:
On v7.6.4, the default setting of cp-accel-mode is changed to none on 2GB memory models. This change disables CP acceleration to lower system memory usage, which can prevent some unexpected behavior due to a lack of memory. See this document: Default setting of cp-accel-mode is changed to none on 2GB memory models.
Related article:
Troubleshooting Tip: How to optimize memory usage specifically for FortiGateRugged-60F low-end model...
