Description
The article describes how to check if a FortiGate is going into conserve mode due to an increased ISDB database.
In October 2023, the size of the ISDB increased significantly raising the risk of lower-end FortiGate hardware units entering conserve mode.
This is especially concerning for the lower-end unit, given that base memory usage hovers around 70-75% in these lower-end models. Even a small increase could trigger conserve mode.
Scope
The problem was observed in the 61E lower-end model. Nevertheless, the FortiGate models listed below also come with less than 2 GB of RAM.
FortiGate models ranging from 30D/30E/40C/40F/50E/60D/60E/61E/61F/70D/70F/80C/80D/80E/81E/90D/91Eand 92D.
Solution
To check memory usage, run 'diagnose hardware sysinfo memory' in both normal and conserve modes. Download the outputs and compare them side by side, focusing on:
- MemFree
- Cached
- Active
- Slab.
Let's break down these memory types a bit before moving on.
Cache Memory:
Cache memory is primarily used to store copies of frequently accessed data, making it faster to retrieve that data when needed.
Active Memory:
Active memory is the portion currently in use by the system's processes. It includes the memory used by running applications and the kernel.
Slab:
The term 'slab' refers to a memory management mechanism employed by the kernel to efficiently allocate and deallocate small memory chunks.
The Slab Allocator, part of the Linux kernel's memory management subsystem, aims to enhance memory allocation performance for frequently used data structures.
A slab is a collection of pre-allocated, contiguous memory blocks of the same size, with each block used to store an instance of a specific data structure.
Let's observe a sample output of a use case scenario when the device was in its regular state as compared to the output during conserve mode
MemFree: 454508 kB <----- normal state == 456 MB
MemFree: 44464 kB <----- conserve state == 45 MB
Difference == 411 MB
Cached: 572924 kB <----- normal state == 573 MB
Cached: 744376 kB <----- conserve state == 744 MB
Difference == 171 MB (significant)
Active: 769540 kB <----- normal state == 770 MB
Active: 841972 kB <----- conserve state == 842 MB
Difference == 60 MB (not significant)
Slab: 106992 kB <----- normal state == 107 MB
Slab: 343040 kB <----- conserve state == 343 MB
Difference == 236 MB (significant)
Adding the differences:
171 MB (cached) + 60 MB (active) + 236 MB (slab) == 467 MB, which is quite close to the difference in memory between the normal state and conserved state.
These are the two extensively utilized memory spaces in conserve mode, and both play a role in managing cache memory. In FortiGate, cache memory includes MEMORY LOGGING, WEB FILTER CACHE, LOG queues, and also UPDATE database cache.
Examining the crash log, it's evident that the FortiGate enters conserve mode for less than 10 seconds. Within this timeframe, the system events logs indicate FortiGuard updates for ISDB are being performed.
2023-11-03 18:20:44 green="1530 MB" msg="Kernel enters memory conserve mode
2023-11-03 18:20:46 service=kernel conserve=exit total="1866 MB" used="1522 MB" red="1642 MB"
Apply the following modifications to prevent FortiGate from entering conserve mode:
- Schedule an update outside of normal business hours:
config system autoupdate schedule
set frequency daily
set time 03:00 -> Non production hours
end
- Reduce cache timeouts:
config system fortiguard
set webfilter cache-ttl
set antispam cache-ttl
Default is:
set antispam-cache-ttl 1800
set outbreak-prevention-cache-ttl 300
set webfilter-cache-ttl 3600
- Reduce DNS cache size:
config system dns
sh full-configuration | grep cache
set dns-cache-limit 5000
set dns-cache-ttl 1800
- Reduce TCP session timeouts: Reduce the value according to the type of traffic passing through the FortiGate:
config system global
show full-configuration | grep tcp
set reset-sessionless-tcp disable
set tcp-halfclose-timer 120
set tcp-halfopen-timer 10
set tcp-option enable
set tcp-rst-timer 5
set tcp-timewait-timer 1
- Limit logging to essential events by adjusting log settings or disabling memory logging if using devices like FortiAnalyzer, FortiGate Cloud, and Syslog servers.
- Reduce worker count: Reduce the worker count accordingly if your FortiGate is using any of the processes listed below heavily be careful before changing the values:
config system global
set miglogd-children 1
set sslvpn-max-worker-count 1
set wad-worker-count 1
set scanunit-count 1
end
- Increase the conserve mode threshold accordingly:
Technical Tip: Conserve mode changes in FortiGate 5.6 and above
- Update FortiGuard when the memory is below the conserve mode thresholds:
config system fortiguard
set update-ffdb disable <----- Enable to update the Internet Service Database only during low memory usage.
end
