Conserve mode activated due to high memory usage

Which FortiOS version?

Some FG-100F are 4GB and some are 8GB (2nd revision).

If your FG-100F is 4GB then you are probably hitting this known issue:

try to see which process is consuming more memory

diag sys top >> ctrl+c to stop it
get sys performance status

Hello CWT-sa,

Kindly review this KB & try checking the daemon consuming high memory.

Regards,

Hi ​@CWT-sa 

There are already some useful comments. If you can share the outputs of a few commands, it would be helpful for identifying the issue.

In addition, I would like to share two common steps we often use to identify the cause of high memory usage on devices with limited hardware memory.

Step 1: In general, identify which daemons are consuming a high amount of memory or continuously increasing memory usage.

diagnose sys top-mem 250

diag hardware sysinfo slab

diagnose hardware sysinfo shm

dia debug crashlog read

Step 2: Normally, after a reboot or restarting certain processes, memory usage returns to normal. Therefore, we need a script to monitor how memory usage changes over time and to identify which daemons continue consuming a large amount of memory.

Please run the following commands multiple times, both under normal conditions and when high memory usage occurs.

get system status

fnsysctl date

get hardware status

get sys perf status

diag sys session stat

diagnose sys session6 stat

diag hardware sysinfo memory

diag hardware sysinfo slab

diagnose hardware sysinfo shm

diagnose sys top-mem 250

fnsysctl ps

diag sys vd list | grep fib

diag sys cmdb info

diag sys top-fd 30

fnsysctl date

diagnose sys top-mem 250

get sys perf firewall statistics

diag debug enable

diagnose wad stats worker show

diagnose wad memory overused

diagnose wad memory sum

diagnose wad memory workers

diagnose wad memory report

diag test application wad 10000

diag debug disable

diagnose test application ipsmonitor 24

diagnose ips session list by-flowav-mem 50

diagnose ips session list by-idle 50

diagnose ips session list by-created-queries 50

diagnose ips dissector dump

diagnose ips raw status

diagnose ips session performance

diagnose ips session list by-mem

diagnose ips memory track enable

diagnose ips memory track-size 17 480

diagnose ips memory track-print0

diagnose ips session status

diagnose ips memory status

diagnose ips packet status0

diagnose ips memory track disable

fnsysctl df -k

fnsysctl df -m

fnsysctl ls -l /tmp

fnsysctl du -i /tmp

fnsysctl du -ax /tmp

fnsysctl du -a / -d 1

fnsysctl du -i /dev/shm

fnsysctl du -ax /dev/shm

fnsysctl du -a /dev/shm

fnsysctl ls -l /dev/shm

fnsysctl du -i /node-scripts

fnsysctl du -ax /node-scripts

fnsysctl ls -l /node-scripts

exe tac report

After identifying the root cause of the high memory usage, we can then proceed to look for an appropriate workaround.

The script may generate log output, so please send it to me via email at bhoang@fortinet.com or through a private message.

Bill

As mentioned above already, it’s crucial to find out what process is taking up the memory while it’s happening to identify if it’s a known bug or could be a new one. “diag sys top 5 30” then “shift-m” would sort the output by memory usage like below. Memory usage is in the 2nd column from the right.
 

Run Time:  53 days, 22 hours and 20 minutes
0U, 0N, 0S, 100I, 0WA, 0HI, 0SI, 0ST; 1917T, 704F
              node      193      S       0.0     2.7    2
       ipshelper      190      S <     0.0     2.4    3
               wad      268      S       0.0     1.8    2
               wad      270      S       0.0     1.6    7
          cw_acd      219      S       0.0     1.6    2
         cmdbsvr      133      S       0.0     1.5    0
         forticron      181      S       0.0     1.5    6
           fgfmd    12054      S       0.0     1.3    4
                csfd      233      S       0.0     1.2    7
          newcli    31701      S <     0.0     1.2    0
             httpsd      176      S       0.0     1.2    6
          forticldd      182      S       0.0     1.1    2
initXXXXXXXXXXX        1      S       0.0     1.0    4
                 wad      195      S       0.0     0.9    5
           miglogd      191      S       0.0     0.9    5
           updated      199      S       0.0     0.8    6
           miglogd      262      S       0.0     0.8    5
        extenderd      232      S       0.0     0.7    7
         dnsproxy      239      S       0.0     0.7    6
             fcnacd      188      S       0.0     0.7    4
            httpsd    32223      S       0.0     0.7    2
            httpsd    32222      S       0.0     0.7    0
                 nsm      151      S       0.0     0.6    1
            fnbamd      180      S       0.0     0.6    4
                 iked      197      S       0.0     0.6    7
            cu_acd      223      S       0.0     0.6    7
                 wad      264      S       0.0     0.6    6
                 wad      266      S       0.0     0.6    7
                   imi      316      S       0.0     0.6    4
              bgpd      156      S <     0.0     0.6    5

Toshi