|
This article is concerned about high cached and shared memory usage due to high flow antivirus utilization.
As good general practice for any memory problems, first identify where memory is allocated with ‘get system performance status’ and ‘diagnose hardware sysinfo memory’. This is best collected once when high memory usage is observed as well as comparisons and analysis are the easiest when no high memory usage is observed.
The following is an example of high cached and shared memory usage to illustrate the issue:
get sys performance status
CPU states: 2% user 0% system 0% nice 98% idle 0% iowait 0% irq 0% softirq
Memory: 16394796k total, 9314504k used (56.8%), 6560276k free (40.0%), 520016k freeable (3.2%)
Average network usage: 1004433 / 1020201 kbps in 1 minute, 1251377 / 1267225 kbps in 10 minutes, 933120 / 948338 kbps in 30 minutes
Average sessions: 40011 sessions in 1 minute, 40687 sessions in 10 minutes, 40080 sessions in 30 minutes
get hardware memory
MemTotal: 16394796 kB
MemFree: 4173544 kB
Cached: 4725172 kB
Active: 4369308 kB
Active(anon): 4135572 kB
Shmem: 4070296 kB
Slab: 1810260 kB
The increases in cached memory are in the /dev/shm folder.
fnsysctl df -h
Filesystem Size Used Available Use% Mounted on
none 13.4G 633.8M 12.8G 5% /tmp
none 13.4G 3.0G 10.3G 23% /dev/shm
none 13.4G 32.5M 13.4G 0% /dev/cmdb
none 13.4G 107.5M 13.3G 1% /dev/shmfile
/dev/sda1 247.9M 153.3M 81.7M 65% /data
/dev/sda3 27.0G 397.8M 25.2G 2% /data2
The next step for cached memory troubleshooting is to find a suspiciously large file or large amounts of small files in said folder. See Technical Tip: High cached memory due to increasing file-sizes.
In cases with large flow antivirus usage, no large file or large amounts of small files will be visible. When comparing the folder content when everything is quiet and when high memory usage is observed, it will look the same.
This is because the issue is not that large amounts of information is being written to a file or files: the issue is a large amount of files opened by the antivirus.
This can be investigated with 'diagnose test application ipsmonitor 24' showing a large delta between FlowAV interface file close and FlowAV interface file open for each engine.
diagnose test application ipsmonitor 24
2025-07-28 11:07:13
pid: 12290 from 20250726-08:54:23 to 20250728-11:07:13
2025-07-28 11:07:13 av_failopen: disabled
2025-07-28 11:07:13 FlowAV mmap : 0
2025-07-28 11:07:13 FlowAV file open : 0
2025-07-28 11:07:13 FlowAV timeout : 1
2025-07-28 11:07:13 FlowAV req success : 4860
2025-07-28 11:07:13 FlowAV req fail : 0
2025-07-28 11:07:13 FlowAV req retry success : 0
2025-07-28 11:07:13 FlowAV req retry fail : 0
2025-07-28 11:07:13 FlowAV bypassed scan : 0
2025-07-28 11:07:13 FlowAV buffer scan : 0
2025-07-28 11:07:13 FlowAV file scan : 0
2025-07-28 11:07:13 FlowAV interface file open : 731836
2025-07-28 11:07:13 FlowAV interface file close : 450118
2025-07-28 11:07:13 FlowAV interface file destroy : 731209
2025-07-28 11:07:13 FlowAV ignored files : 107730
2025-07-28 11:07:13 FlowAV legacy scan : 4860
2025-07-28 11:07:13 FlowAV default scan : 445258
2025-07-28 11:07:13 FlowAV buffer allocation fail : 0
2025-07-28 11:07:13 FlowAV buffer reallocation : 61420
2025-07-28 11:07:13 FlowAV buffer reallocation fail: 0
With the same command, the shared memory usage is also visible for each engine, which in turn explains why high usage is observable in the /dev/shm folder.
shm memory usage
2025-07-28 11:07:13 ==================
2025-07-28 11:07:13 total allocated memory 5283840 max 22462464 pool_size 229376 pool_max 44088852
In the flow antivirus, FortiGate buffers files and by default is allowed 50% of total memory. This can be limited with 'set av-mem-limit'. The minimum is 10%, which should be used in such a scenario.
config ips global
set av-mem-limit <value> Enter an integer value from <10> to <50> or (special = <0>). <----- <10> to <50> = 10% to 50%. 0 is the default value and is used to disable this command.
end
- Troubleshooting commands:
fnsysctl date
get sys status
diagnose sys session full-stat
diagnose hardware sysinfo memory
diagnose hardware sys slab
fnsysctl df -h
fnsysctl ls -al /dev/shm
diagnose sys top-mem 20
diagnose sys top-fd 20
diagnose sys top 1 10 10
diagnose ips session status
diagnose ips memory status
diagnose ips packet status
diagnose ips dissector status
diagnose ips session list by-mem 10
diagnose test application ipsmonitor 24
get test ipsmonitor 1
get test ipsmonitor 3
diagnose ips test cmd "ips session list http2"
diagnose ips test cmd "ips dissector statistics show http2"
fnsysctl date
|
