FortiGate
FortiGate Next Generation Firewall utilizes purpose-built security processors and threat intelligence security services from FortiGuard labs to deliver top-rated protection and high performance, including encrypted traffic.
bmeta
Staff
Staff
Description
This article describes general actions which could be taken and which information should be sent to Fortinet Support in case of unexpectedly entry of the unit into Conserve Mode, the unit is out of memory.

Solution
1) Run the command in CLI 'get system performance status'; the output will look similar to the sample below:
#FGT# get sys perf stat
CPU states: 1% user 0% system 0% nice 99% idle 0% iowait 0% irq 0% softirq
CPU0 states: 1% user 0% system 0% nice 99% idle 0% iowait 0% irq 0% softirq
Memory: 2004540k total, 586528k used (29%), 1418012k free (71%)
Average network usage: 1 / 0 kbps in 1 minute, 0 / 0 kbps in 10 minutes, 0 / 0 kbps in 30 minutes
Average sessions: 25 sessions in 1 minute, 25 sessions in 10 minutes, 25 sessions in 30 minutes
Average session setup rate: 0 sessions per second in last 1 minute, 0 sessions per second in last 10 minutes, 0 sessions per second in last 30 minutes
Virus caught: 0 total in 1 minute
IPS attacks blocked: 0 total in 1 minute
Uptime: 0 days,  23 hours,  41 minutes
Run the command above a few times and compare patterns of memory usage, throughput and number of sessions.

2) Check total memory usage on the output
Memory: 2004540k total, 586528k used (29%), 1418012k free (71%)
In case used memory is more than 75%, this may indicate that a further check may be required. The unit is either getting overloaded or there is a memory leak in some process/kernel or there is a lot of cached memory.

Check the amount of traffic and compare it to the data sheet (throughput section). If it is too close, the device is likely to be overloaded and there is a sizing issue. If the amount is vastly different between last 1 minute and last 30 minutes, this might indicate a traffic spike.

Average sessions: 25 sessions in 1 minute, 25 sessions in 10 minutes, 25 sessions in 30 minutes
Session table is stored in memory as well. Higher number of sessions lead to higher memory usage. FortiGate performance data sheet also defines the maximum number of sessions firewall can handle.

3) By running in CLI “diag sys top 1 45” it is possible to find memory usage per process instance.

“1” stands for refreshing period in seconds
“45” stands for a number of processes displayed. See part of it as example below:  
Run Time:  0 days, 23 hours and 54 minutes
0U, 0N, 0S, 100I, 0WA, 0HI, 0SI, 0ST; 1957T, 1271F
         newcli      308      R       0.9     0.5
           sshd      305      S       0.9     0.5
        pyfcgid      142      S       0.0     2.0
        reportd      154      S       0.0     1.8
        cmdbsvr      120      S       0.0     1.4
        pyfcgid      184      S       0.0     1.2
        pyfcgid      186      S       0.0     1.2
        pyfcgid      185      S       0.0     1.2
      forticron      149      S       0.0     1.2
        miglogd      139      S       0.0     1.1
         httpsd      141      S       0.0     1.1
      scanunitd      158      S <     0.0     1.0
In order: process name, Process ID, Process state, CPU usage %, Memory usage %.

Check % of memory usage, if any process is constantly using unreasonably high fraction memory, this might be the process causing the issue. At this moment consider opening Technical Support ticket (https://support.fortinet.com) and proceed to step 5.

Note 1: Some processes can have multiple instances like “pyfcgid” in the example above. WAD and IPSengine are also such processes. In such cases, sum up the total memory usage for all instances, and it should not exceed -20 - 25%, but it depends on the device and its total memory - for small devices with small amount of memory, it might be normal. Security Profile like Webfiltering, AV can increase the memory usage.

Note 2: In rare cases the output of the “get system performance status” command can show that memory utilization is high (e.g. more than 90%), but at the same time “diag sys top” command does not indicate any processes which is using memory. This can indicate that memory is utilized by the kernel and/or being cached. In such case please proceed with opening a Technical Support ticket step 5.

4) In order to speed up troubleshooting, run the commands below and attach the output to the ticket along with configuration and debug.log files while contacting Fortinet TAC (https://support.fortinet.com/).
# get system status
# get system performance status     <----- Use this command three times leaving a time 1 minute between each execution.

# diag sys top 2 40                        <----- Let this command run for 1 minute, then stop it via ‘q’
# diag sys top-symmary             
<----- Let this command run for 1 minute, then stop it via ‘q’ - on FortiOS 6.4 this command does not exist.
# diag hard sysinfo memory
# diag hard sysinfo slab
# diag hard sysinfo shm
# diagnose hard sys conserve
# diag debug crashlog read     
         <----- It lists all instances with timestamp for conserve modes and crashes, if any.

And this per VDOM, if configured.
# get log disk setting
# get log disk filter
# get log memory setting
# get log memory filter
Note.
The 'crashlog' might correlate with SNMP monitoring to have set up for this FortiGate.
External monitoring and recording like SNMP can greatly help to trace when such issues might have started.


Related Articles

Technical Tip: How conserve mode is triggered

Contributors