Technical Tip: How conserve mode is triggered
Description
This article describes proxy conserve mode (sometimes referred to simply as 'conserve mode') and kernel conserve mode in the FortiGate environment.
Scope
FortiGate.
Solution
What conserve mode is:
Conserve mode is FortiGate's self-protection when memory usage reaches critical levels, dropping new sessions, disabling offloads, and reclaiming resources. In FortiOS, it uses three levels (conserve: ~82%, severe: ~88%, extreme: ~95% of total RAM) with dynamic thresholds.
The differences between proxy conserve mode and kernel conserve mode:
There are two types of conserve mode: proxy conserve mode and kernel conserve mode.
Proxy Conserve Mode:
Proxy conserve mode can be triggered when using proxy-based inspection.
The thresholds to enter and leave conserve mode depend on the amount of free memory. These threshold vary by model and are determined by the total memory available on that model.
The thresholds to enter and leave conserve mode depend on the amount of free memory. These threshold vary by model and are determined by the total memory available on that model.
| Total Memory | Enter Threshold | Exit Threshold |
| >=128 MB | 10MB | 20MB |
| >=256 MB | 40MB | 60MB |
| >=512MB | 20% | 30% |
| >= 1 GB | 12% | 18% |
Proxy conserve mode is either caused by processes consuming too much memory (rare case), or more common only by high usage of 'shared memory' (SHM).
Shared memory are buffers allocated which can be shared among different processes. It is not listed on the process memory columns as 'diagnose system top'. Shared memory is used mainly by proxies (to store the buffered data) but also by buffers (logging, quarantining...).
When entering and leaving proxy conserve mode, event logs similar to the following would be raised:
conserve=on total=<totalmemMB> free=<freememMB> entermargin=<LF> exitmargin=<HF> msg="The system has entered conserve mode"
conserve=exit total=<totalmemMB> free=<freememMB> entermargin=<LF> exitmargin=<HF> msg="The system exited conserve mode"
Kernel Conserve Mode:
The kernel conserve mode can be triggered as follows:
| Total Memory | Enter Threshold | Exit Threshold |
| 512 MB | 20% | 30% |
| >= 1GB | 200MB | 300MB |
When entering or leaving 'kernel conserve mode', event logs like the following will be raised:
'The system has entered system conserve mode'.
'The system exited system conserve mode'.
Note: A slightly different message will be shown on the GUI dashboard, such as:
'FortiGate has reached system connection limit for x seconds'.
warning: These apply to old models (<2GB RAM). In 7.6.4, use percentage-based with hysteresis; exact values vary by platform.
Unified levels: Green (<82%), Conserve (82-88%), Severe (88-95%), Extreme (>95%).
How to view the usage of the different memory sections and determine if the unit is close to conserve mode or kernel conserve mode:
By using 'diagnose hardware system memory', all of the memory counters involved in the conserve mode and kernel conserve mode calculation can be seen.
Consider the following example:
Consider the following example:
diagnose hardware sysinfo conserve <----- Legacy, still works
diagnose sys memory status <----- Modern: Shows levels explicitly
get system performance status <----- CPU/Memory overview
diagnose sys top 1 99 <----- Process memory (sort by %)
diagnose sys session stat <----- Session counts impacting memory
diagnose sys memory status <----- Modern: Shows levels explicitly
get system performance status <----- CPU/Memory overview
diagnose sys top 1 99 <----- Process memory (sort by %)
diagnose sys session stat <----- Session counts impacting memory
diagnose debug crashlog read <----- For memory-related crashes
diagnose sys kill 11 <pid> <----- Emergency process restart (caution)
FortiGate # diagnose hardware sysinfo memory
total: used: free: shared: buffers: cached: shm:
Mem: 260435968 146337792 114098176 0 221184 65974272 59985920
Swap: 0 0 0
MemTotal: 254332 kB
MemFree: 111424 kB
MemShared: 0 kB
Buffers: 216 kB
Cached: 64428 kB
SwapCached: 0 kB
Active: 26844 kB
Inactive: 37856 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 254332 kB (2)
LowFree: 111424 kB (1)
SwapTotal: 0 kB
SwapFree: 0 kB
total: used: free: shared: buffers: cached: shm:
Mem: 260435968 146337792 114098176 0 221184 65974272 59985920
Swap: 0 0 0
MemTotal: 254332 kB
MemFree: 111424 kB
MemShared: 0 kB
Buffers: 216 kB
Cached: 64428 kB
SwapCached: 0 kB
Active: 26844 kB
Inactive: 37856 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 254332 kB (2)
LowFree: 111424 kB (1)
SwapTotal: 0 kB
SwapFree: 0 kB
Use the CLI command below to check the current memory usage, threshold settings, and whether the FortiGate is currently in conserve mode.
FortiGate # diagnose hardware sysinfo conserve
memory conserve mode: off
total RAM: 387725 MB
memory used: 288203 MB 74% of total RAM
memory freeable: 561 MB 0% of total RAM
memory used + freeable threshold extreme: 368339 MB 95% of total RAM
memory used threshold red: 341198 MB 88% of total RAM
memory used threshold green: 317934 MB 82% of total RAM
Explaining the value 'Cached, Active, Inactive' that may take significant memory:
Cached = Active + Inactive
This is information cached by the FortiGate for its own system (basically, I/O buffering). The inactive part is claimed back from the system when it requires more memory.
Additionally, 'diagnose sys top' displays the memory occupied in userspace by the different processes as a percentage of total memory. Shared memory is not counted in this column. Refer to the last column on the right.
Run Time: 11 days, 19 hours and 6 minutes
0U, 0S, 14I; 248T, 108F, 56KF
newcli 414 R 1.9 5.3
ipsengine 56 S < 0.0 12.4
httpsd 72 S 0.0 7.2
cmdbsvr 17 S 0.0 7.2
httpsd 85 S 0.0 6.7
httpsd 37 S 0.0 6.1
newcli 413 S 0.0 5.3
fgfmd 78 S 0.0 5.2
miglogd 35 S 0.0 5.1
scanunitd 411 S < 0.0 4.8
updated 65 S 0.0 4.6
scanunitd 410 S < 0.0 4.5
iked 64 S 0.0 4.5
urlfilter 57 S 0.0 4.4
../..
newcli 414 R 1.9 5.3
ipsengine 56 S < 0.0 12.4
httpsd 72 S 0.0 7.2
cmdbsvr 17 S 0.0 7.2
httpsd 85 S 0.0 6.7
httpsd 37 S 0.0 6.1
newcli 413 S 0.0 5.3
fgfmd 78 S 0.0 5.2
miglogd 35 S 0.0 5.1
scanunitd 411 S < 0.0 4.8
updated 65 S 0.0 4.6
scanunitd 410 S < 0.0 4.5
iked 64 S 0.0 4.5
urlfilter 57 S 0.0 4.4
../..
Example:
The ipsengine process is using 12.4% of the unit memory in userspace.
Steps to save memory resources:
The following actions will help to save memory resources:
- Reduce the number of firewall sessions as described in the related Knowledge Base article 'Technical Note: FortiGate CPU resource optimization configuration steps'.
- Reduce the maximum file size for antivirus scanning.
- Turn off all non-mandatory features such as Logging, archiving, data leak prevention, and IPS.
- Remove 'content summary' (especially if no FortiAnalyzers are configured).
- Reduce memory caching in some features (Explicit proxy, FortiGuard Antispam/Webfiltering ...)
- This list is not exhaustive. The choice of 'non-mandatory features' is left to the administrator.
Related articles:
FortiGate CPU resource optimization configuration steps.
Technical Tip: How conserve mode is triggered.
Technical Tip: Automation stitch for the conserve mode.
Technical Tip: How to be notified via email that the FortiGate has entered conserve mode.
Troubleshooting Tip: Memory debugs needed when creating a bug ticket
Troubleshooting Tip: How to do initial troubleshooting of high memory utilization issues (conserve mode)
Technical Tip: Automation stitch for conserve mode
Checking CPU and memory resources | FortiGate / FortiOS 7.6.4 | Fortinet Document Library