Technical Tip: How conserve mode is triggered

[cgustave]

Description

This article discusses proxy conserve mode (sometimes referred to simply as "conserve mode") and kernel conserve mode in the FortiGate environment.

It addresses the following questions:

- What is conserve mode?
- What are the differences between proxy conserve mode and kernel conserve mode?
- What is the value "Cached, Active, Inactive" that may take significant memory?
- What can be done to save memory resource?

What is conserve mode?

A FortiGate goes into the "conserve mode" state as a self protection measure when a memory shortage appears on the system. When entering conserve mode the FortiGate activates protection measures in order to recover memory space. When enough memory is recovered, the system is leaving/exiting the conserve mode state and releases the protection measures.

What are the differences between proxy conserve mode and kernel conserve mode?

There are two types of conserve mode: proxy conserve mode and kernel conserve mode.

Proxy Conserve Mode

Proxy conserve mode can be triggered when using proxy-based inspection.

The thresholds to enter and leave conserve mode depend on the amount of free memory. These threshold vary by model and are determined by the total memory available on that model.

Total Memory	Enter Threshold	Exit Threshold
>=128 MB	10MB	20MB
>=256 MB	40MB	60MB
>=512MB	20%	30%
>= 1 GB	12%	18%

Proxy conserve mode is either caused by processes consuming too much memory (rare case), or more comman only by high usage of "shared memory" (SHM).

Shared memory are buffers allocated which can be shared among different processes. It is not listed on the process memory columns as diag sys top. Shared memory is used mainly by proxies (to store the buffered data) but also by buffers (logging, quarantining...).

When entering and leaving proxy conserve mode, event log as shown below would be raised:

conserve=on total=<totalmemMB> free=<freememMB> entermargin=<LF> exitmargin=<HF> msg="The system has entered conserve mode"

conserve=exit total=<totalmemMB> free=<freememMB> entermargin=<LF> exitmargin=<HF> msg="The system exited conserve mode"

Kernel Conserve Mode

The kernel conserve mode can be triggered as follows:

Total Memory	Enter Threshold	Exit Threshold
512 MB	20%	30%
>= 1GB	200MB	300MB

When entering or leaving "kernel conserve mode", event logs as shown below would be raised:

'The system has entered system conserve mode'.

'The system exited system conserve mode'.

Note: A slightly different message would be shown on the GUI dashboard such as:

'FortiGate has reached system connection limit for x seconds'.

How can the usage of the different memory sections be seen and how can it be determined if the unit is close to conserve mode or kernel conserve mode?

By using "diagnose hardware system memory", all of the memory counters involved in the conserve mode and kernel conserve mode calculation can be seen.

Consider the following example:

FWF60B # diagnose hardware sysinfo memory
total: used: free: shared: buffers: cached: shm:
Mem: 260435968 146337792 114098176 0 221184 65974272 59985920
Swap: 0 0 0
MemTotal: 254332 kB
MemFree: 111424 kB
MemShared: 0 kB
Buffers: 216 kB
Cached: 64428 kB
SwapCached: 0 kB
Active: 26844 kB
Inactive: 37856 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 254332 kB (2)
LowFree: 111424 kB (1)
SwapTotal: 0 kB
SwapFree: 0 kB

What is the value "Cached, Active, Inactive" that may take significant memory?

Cached = Active + Inactive

This is information cached by the FortiGate for its own system (basically I/O buffering). The inactive part is claimed back from the system when it requires more memory.

Additionally, '# diagnose system top' displays the memory occupied in userspace by the different processes as a percentage of total memory. Shared memory is not counted in this column. Refer to the last column on the right

Run Time: 11 days, 19 hours and 6 minutes

0U, 0S, 14I; 248T, 108F, 56KF
newcli 414 R 1.9 5.3
ipsengine 56 S < 0.0 12.4
httpsd 72 S 0.0 7.2
cmdbsvr 17 S 0.0 7.2
httpsd 85 S 0.0 6.7
httpsd 37 S 0.0 6.1
newcli 413 S 0.0 5.3
fgfmd 78 S 0.0 5.2
miglogd 35 S 0.0 5.1
scanunitd 411 S < 0.0 4.8
updated 65 S 0.0 4.6
scanunitd 410 S < 0.0 4.5
iked 64 S 0.0 4.5
urlfilter 57 S 0.0 4.4
../..

Ex: ipsengine process is using 12.4% of the unit memory in userspace.

What can be done to save memory resource?

The following actions will help to save memory resource:

- Reduce the number of firewall sessions as described in the related Knowledge Base article "Technical Note : FortiGate CPU resource optimization configuration steps".
- Reduce the maximum file size for antivirus scanning.
- Turn off all non mandatory features such as Logging, archiving, data leak prevention, IPS.
- Remove 'content summary' (especially if no FortiAnalyzers are configured).
- Reduce memory caching in some features (Explicit proxy, FortiGuard Antispam/Webfiltering ...)

- This list is not exhaustive. The choice of "non mandatory features" is left to the administrator.

Internal Notes

References:

http://wiki.fortinet.com/twiki/bin/view/Developers/ConserveMode

Top3 792 (Very good information on kernel conserve mode)

Kernel conserve mode (AV_break):

The only indication of this is through cli: diag firewall iprope state which shows by default:

 av_break=pass/off

 and when on:

 av_break=pass/pass

 av_break happens when low memory is below the 20% threshold of total memory that can be accessed by kernel, and returns when 30% is now available. In a box with 1G of physical memory, 7/8 G (870M) of memory is accessible by kernel. This is a kernel mechnism to prevent system hang.

 Question: Does this second conserve mode mechanism [kernel conserve] also apply the action defined in config>system>global av-failopen [off|one-shot|passw] or is it always bypassing ? Answer: Always by-passing afaik.


===== May 2015 =================================================

Some changes in Version 4 of this KB article:
1) updated enter/exit thresholds
2) removed reference to Low Memory as no longer relevant for 64bit models

Related Articles

FortiGate CPU resource optimization configuration steps

Technical Tip: How conserve mode is triggered