|
This article talks about high cached memory leaks where either files keep on growing in size or an endless amount of small files force the device into memory conserve mode and how to spot it.
As a general overview for any memory problem first identify where memory is allocated with ‘get system performance status’ and ‘diagnose hardware sysinfo memory’.
get system performance status
CPU states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU0 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU1 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU2 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
Memory: 16393804k total, 14652272k used (89.4%), 1468396k free (9.0%), 273136k freeable (1.7%)
Average network usage: 26 / 26 kbps in 1 minute, 25 / 26 kbps in 10 minutes, 24 / 25 kbps in 30 minutes
In this case, observe that cached memory is very high alongside with shared memory ('Shmem'). The sum of cached memory and active memory is greater than the total memory, this should not be possible. This is because in cases where cached and shared memory is high some parts of allocated memory are accounted for several categories of memory at the same time, resulting in an implausible sum.
diagnose hardware sysinfo memory
MemTotal: 16393804 kB
MemFree: 5385568 kB
Cached: 8585680 kB
Active: 9069520 kB
Shmem: 8260228 kB
Slab: 145196 kB
To identify where the files are written enter ‘fnsysctl df -k’. This is also part of the TAC report. [2] In this example, a lot of used memory is in the /tmp folder.
fnsysctl df -k
Filesystem 1K-blocks Used Available Use% Mounted on
none 14443180 12078148 2365032 84% /tmp
none 14443180 1008 14442172 0% /dev/shm
none 14443180 23716 14419464 0% /dev/cmdb
/dev/sda1 253871 108289 132475 45% /data
/dev/sda3 28327040 79904 26808208 0% /data2
FortiOS is based on Linux which lets us enter some Linux commands as long as the prefix fnysctl is entered before the command. To find the responsible file(s) one way would be to print out the entire file system via ‘fnsysctl du -alLH /’ and look for folders that consume a lot of memory.
Checking on the /tmp folder directly the appDemo_stdout, in this example, ballooned to roughly 12GB.
There are also cases where lots of small files of roughly the same size are saved to memory. In these cases, it is better to collect the outputs from these commands over some time and check the content of the affected folder as seen in 'fnsysctl df -k'.
fnsysctl ls -l /tmp
drwxr-xr-x 2 0 0 Thu Jul 4 16:23:40 2024 40 $$auto-script$$
drwxr-xr-x 2 0 0 Thu Jul 4 16:23:03 2024 40 _certs_
drwxr-xr-x 2 0 0 Thu Jul 4 16:23:03 2024 40 _keys_
lrwxrwxrwx 1 0 0 Thu Jul 4 16:22:53 2024 13 admin_server.crt -> /etc/self.crt
lrwxrwxrwx 1 0 0 Thu Jul 4 16:22:53 2024 13 KEY-FILE -> /etc/self.key
drwxr-xr-x 2 0 0 Thu Jul 11 16:00:17 2024 2100 api_cache
drwxr-xr-x 2 0 0 Thu Jul 11 15:59:45 2024 180 api_cache_stat
-rw-r--r-- 1 0 0 Thu Jul 11 16:00:34 2024 12360290727 appDemo_stdout
srwxr-xr-x 1 0 0 Thu Jul 4 16:23:42 2024 0 authd_svr_socket
drw------- 2 0 0 Thu Jul 4 16:23:40 2024 40 backtrace_log
srwxr-xr-x 1 0 0 Tue Jul 9 22:56:37 2024 0 bwl_gui_to_ha_unix_sock
-rw-r--r-- 1 0 0 Tue Jul 9 22:56:37 2024 2068 wcha_cfg_clr.conf
Unlike in the KB article FortiGate out of memory due to memory cache [1] it is not possible to interact with these files. For security and integrity reasons the FortiGate remove commands cannot be run with the fnsysctl prefix. The only way to remove these files at this point is to reboot the device or upgrade to a version where this issue is fixed.
Contact technical support if finding such files or having similar problems.
Related articles:
[1] Technical Tip: FortiGate out of memory due to memory cache on v7.0/v7.2
[2] Technical Tip: Download Debug Logs and 'execute tac report'
|
