|
This article is concerned about high cache and shared memory usage due to high flow antivirus utilization.
As good general practice for any memory problems, first identify where memory is allocated with ‘get system performance status’ and ‘diagnose hardware sysinfo memory’. This is best collected once high memory usage is observed, as well as comparisons and analysis are the easiest when no high memory usage is observed.
The following is an example of high cache and shared memory usage to illustrate the issue:
get system performance status
CPU states: 2% user 0% system 0% nice 98% idle 0% iowait 0% irq 0% softirq
Memory: 16394796k total, 9314504k used (56.8%), 6560276k free (40.0%), 520016k freeable (3.2%)
Average network usage: 1004433 / 1020201 kbps in 1 minute, 1251377 / 1267225 kbps in 10 minutes, 933120 / 948338 kbps in 30 minutes
Average sessions: 40011 sessions in 1 minute, 40687 sessions in 10 minutes, 40080 sessions in 30 minutes
get hardware memory
MemTotal: 16394796 kB
MemFree: 4173544 kB
Cached: 4725172 kB
Active: 4369308 kB
Active(anon): 4135572 kB
Shmem: 4070296 kB
Slab: 1810260 kB
The increases in cached memory are in the /dev/shm folder.
fnsysctl df -h
Filesystem Size Used Available Use% Mounted on
none 13.4G 633.8M 12.8G 5% /tmp
none 13.4G 3.0G 10.3G 23% /dev/shm
none 13.4G 32.5M 13.4G 0% /dev/cmdb
none 13.4G 107.5M 13.3G 1% /dev/shmfile
/dev/sda1 247.9M 153.3M 81.7M 65% /data
/dev/sda3 27.0G 397.8M 25.2G 2% /data2
The next step for cached memory troubleshooting is to find a suspiciously large file or a large number of small files in said folder. See Technical Tip: High cached memory due to increasing file-sizes.
In cases with large flow antivirus usage, no large files or large amounts of small files will be visible. When comparing the folder content when everything is quiet and when high memory usage is observed, it will look the same.
This is because the issue is not that large amounts of information is being written to a file or files: the issue is a large number of files opened by the antivirus.
This can be investigated with 'diagnose test application ipsmonitor 24', showing a large difference between the FlowAV interface file close and the FlowAV interface file open for each engine.
diagnose test application ipsmonitor 24
2025-07-28 11:07:13
pid: 12290 from 20250726-08:54:23 to 20250728-11:07:13
2025-07-28 11:07:13 av_failopen: disabled
2025-07-28 11:07:13 FlowAV mmap : 0
2025-07-28 11:07:13 FlowAV file open : 0
2025-07-28 11:07:13 FlowAV timeout : 1
2025-07-28 11:07:13 FlowAV req success : 4860
2025-07-28 11:07:13 FlowAV req fail : 0
2025-07-28 11:07:13 FlowAV req retry success : 0
2025-07-28 11:07:13 FlowAV req retry fail : 0
2025-07-28 11:07:13 FlowAV bypassed scan : 0
2025-07-28 11:07:13 FlowAV buffer scan : 0
2025-07-28 11:07:13 FlowAV file scan : 0
2025-07-28 11:07:13 FlowAV interface file open : 731836
2025-07-28 11:07:13 FlowAV interface file close : 450118
2025-07-28 11:07:13 FlowAV interface file destroy : 731209
2025-07-28 11:07:13 FlowAV ignored files : 107730
2025-07-28 11:07:13 FlowAV legacy scan : 4860
2025-07-28 11:07:13 FlowAV default scan : 445258
2025-07-28 11:07:13 FlowAV buffer allocation fail : 0
2025-07-28 11:07:13 FlowAV buffer reallocation : 61420
2025-07-28 11:07:13 FlowAV buffer reallocation fail: 0
[...]
shm memory usage
2025-07-28 11:07:13 ==================
2025-07-28 11:07:13 total allocated memory 5283840 max 22462464 pool_size 229376 pool_max 44088852
The same command also shows shared memory usage for each engine in the 'shm memory usage' section, which in turn explains why high usage is observable in the /dev/shm folder.
By default, Flow-based Antivirus on the FortiGate buffers files in memory without any limit. However, starting in FortiOS v7.4.2, upcoming v7.6.5, and later, it is possible to limit this usage to a range from 10% up to 50% of total system memory using the set av-mem-limit option under config ips global:
config ips global
set av-mem-limit <10-50, default is 0**>
end
Note that the default value of 0 means that there is no limit for flow-AV memory usage.
Additionally, the above memory limit pairs with the existing av-failopen option present under config system global:
config system global
set av-failopen [ pass | off | one-shot ]
end
If the memory limits are reached and av-failopen is set to pass, then flow-AV will bypass Antivirus scanning for any currently buffered files. Otherwise, if memory limits are reached and av-failopen is not set to pass, then flow-AV will block the file and reset the session.
- Troubleshooting commands:
fnsysctl date
get system status
diagnose sys session full-stat
diagnose hardware sysinfo memory
diagnose hardware sys slab
fnsysctl df -h
fnsysctl ls -al /dev/shm
diagnose sys top-mem 20
diagnose sys top-fd 20
diagnose sys top 1 10 10
diagnose ips session status
diagnose ips memory status
diagnose ips packet status
diagnose ips dissector status
diagnose ips session list by-mem 10
diagnose test application ipsmonitor 24
get test ipsmonitor 1
get test ipsmonitor 3
diagnose ips test cmd "ips session list http2"
diagnose ips test cmd "ips dissector statistics show http2"
fnsysctl date
|
