Fortigate 60E - 6.4.4 - high gradation of memory without deallocation

[Luis_Eduardo_Oliveir]

Hey guys,

I would like to know if anyone has had difficulty with the 60E model in version 6.4.3, 6.4.4 and 6.4.5 high gradation of memory without deallocation, I could not identify high consumption of any process such as IPSMONITOR, IPSHELPER, IPSENGINE, WAD or any other another process.

I already opened a ticket with the tac but so far there has not been an effective answer about what the problem is, I decided to format some equipment but some of them again presented a problem with high memory consumption, all the equipment is clustered and the problem reserving the memory occurs on any equipment on the master or slave the increase is gradual but there is no process reserving the memory.

I appreciate the help.

Run Time: 0 days, 12 hours and 52 minutes 11U, 0N, 3S, 85I, 0WA, 0HI, 1SI, 0ST; 1866T, 663F httpsd 1545 S 8.4 0.9 httpsd 1557 S 8.3 0.9 httpsd 1571 S 7.5 0.8 httpsd 1578 S 4.5 0.4 httpsd 1581 S 4.3 0.4 httpsd 1570 S 3.3 0.6 httpsd 1524 S 2.9 0.9 httpsd 1559 S 2.1 0.6 httpsd 1562 S 2.1 0.6 httpsd 1580 S 1.9 0.6 [kworker/3:1] 75 SW 1.5 0.0 [kworker/0:1] 38 RW 1.3 0.0 httpsd 1582 S 1.1 0.4 httpsd 1583 S 1.1 0.4 httpsd 1585 S 1.1 0.4 dhcpd 159 S 0.9 0.4 hasync 161 S < 0.7 0.7 [kworker/1:1] 73 SW 0.7 0.0 [kworker/2:2] 99 SW 0.7 0.0 hatalk 160 S < 0.5 0.4

diagnose sys top-mem

ipsengine (32182): 75724kB ipsengine (32189): 75660kB ipsengine (32186): 75020kB ipsengine (32181): 74912kB cmdbsvr (101): 39096kB diag deb crash read

207: 2021-05-22 00:33:05 logdesc="Memory conserve mode entered" service=kernel conserve=on total="1866 208: 2021-05-22 00:33:05 MB" used="1642 MB" red="1642 MB" green="1530 MB" msg="Kernel enters memory 209: 2021-05-22 00:33:05 conserve mode" 210: 2021-05-22 00:33:05 MemTotal: 1911176 kB 211: 2021-05-22 00:33:05 MemFree: 139964 kB 212: 2021-05-22 00:33:05 Buffers: 42000 kB 213: 2021-05-22 00:33:05 Cached: 413104 kB 214: 2021-05-22 00:33:05 SwapCached: 0 kB 215: 2021-05-22 00:33:05 Active: 503180 kB 216: 2021-05-22 00:33:05 Inactive: 205280 kB 217: 2021-05-22 00:33:05 Active(anon): 447756 kB 218: 2021-05-22 00:33:05 Inactive(anon): 148756 kB 219: 2021-05-22 00:33:05 Active(file): 55424 kB 220: 2021-05-22 00:33:05 Inactive(file): 56524 kB 221: 2021-05-22 00:33:05 Unevictable: 0 kB 222: 2021-05-22 00:33:05 Mlocked: 0 kB 223: 2021-05-22 00:33:05 SwapTotal: 0 kB 224: 2021-05-22 00:33:05 SwapFree: 0 kB 225: 2021-05-22 00:33:05 Dirty: 0 kB 226: 2021-05-22 00:33:05 Writeback: 0 kB 227: 2021-05-22 00:33:05 AnonPages: 253256 kB 228: 2021-05-22 00:33:05 Mapped: 110640 kB 229: 2021-05-22 00:33:05 Shmem: 342972 kB 230: 2021-05-22 00:33:05 Slab: 801700 kB 231: 2021-05-22 00:33:05 SReclaimable: 5512 kB 232: 2021-05-22 00:33:05 SUnreclaim: 796188 kB 233: 2021-05-22 00:33:05 KernelStack: 1264 kB 234: 2021-05-22 00:33:05 PageTables: 17640 kB 235: 2021-05-22 00:33:05 NFS_Unstable: 0 kB 236: 2021-05-22 00:33:05 Bounce: 0 kB 237: 2021-05-22 00:33:05 WritebackTmp: 0 kB 238: 2021-05-22 00:33:05 CommitLimit: 955588 kB 239: 2021-05-22 00:33:05 Committed_AS: 8081116 kB 240: 2021-05-22 00:33:05 VmallocTotal: 663552 kB 241: 2021-05-22 00:33:05 VmallocUsed: 132552 kB 242: 2021-05-22 00:33:05 VmallocChunk: 336236 kB

get system performance status CPU states: 1% user 0% system 0% nice 98% idle 0% iowait 0% irq 1% softirq CPU0 states: 2% user 0% system 0% nice 94% idle 0% iowait 0% irq 4% softirq CPU1 states: 1% user 0% system 0% nice 99% idle 0% iowait 0% irq 0% softirq CPU2 states: 0% user 0% system 0% nice 99% idle 0% iowait 0% irq 1% softirq CPU3 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq Memory: 1911176k total, 1812220k used (94.8%), 86252k free (4.5%), 12704k freeable (0.7%) Average network usage: 1871 / 1740 kbps in 1 minute, 1885 / 1775 kbps in 10 minutes, 1931 / 1841 kbps in 30 minutes Average sessions: 999 sessions in 1 minute, 1059 sessions in 10 minutes, 1046 sessions in 30 minutes Average session setup rate: 1 sessions per second in last 1 minute, 1 sessions per second in last 10 minutes, 1 sessions per second in last 30 minutes Average NPU sessions: 486 sessions in last 1 minute, 479 sessions in last 10 minutes, 477 sessions in last 30 minutes Average nTurbo sessions: 22 sessions in last 1 minute, 17 sessions in last 10 minutes, 14 sessions in last 30 minutes Virus caught: 0 total in 1 minute IPS attacks blocked: 0 total in 1 minute Uptime: 0 days, 12 hours, 52 minutes

diagnose npu np6lite session-stats nr_cmd_raw :0 nr_ins :60030 nr_ins_fail :0 nr_del :59859 nr_del_fail :0 nr_del6 :0 nr_del6_fail :0 nr_qry :0 nr_qry_fail :0 nr_flush :0

diagnose npu np6lite session

nr_session :234 nr_tcp :92 nr_udp :41 nr_icmp :101 nr_esp :0 nr_sctp :0 nr_ipv4 :234 nr_ipv6 :0 nr_multicast :0 nr_matched_session :0

diagnose test application ipsmonitor 1 pid = 134, engine count = 2 0 - pid:178:178 cfg:1 master:0 run:1 1 - pid:259:259 cfg:0 master:1 run:1

pid: 259 index:1 master version: 06004000FLEN06800-00006.00002.00071-21020403 up time: 0 days 12 hours 52 minutes init time: 1 seconds socket size: 32(MB) database: regular bypass: disable

diagnose ips packet status

PID: 259

PACKET STATISTICS: total packets 33860 tcp packets 33824 udp packets 12 icmp packets 0 other packets 0 fast path bad packets 24 fast path other packets 0 fast path nocfg packets 24 fast path invcfg packets 0 fast path config changed packets 0 slow path invcfg packets 0 tcp PAWS packets 1 huge packets 0

PACKET ACTION STATISTICS: PASS 31759 DROP 972 RESET 0 RESET_CLIENT 0 RESET_SERVER 0 DROP_SESSION 47 PASS_SESSION 1082 CLEAR_SESSION 0 EXEMPT 0

PACKET ERROR STATISTICS: flowav file open 0 flowav file write 0 flowav external error 0 TCP asm buffer limit 0 TCP packets lost 0 dropped in conserve mode 0 SMB2 OOO buffer limit 0 flowav incomplete files 0 flowav complete files 25 flowav early passed files 0 flowav passed files 0 flowav early offloaded sessions 0 SMB2 read response status error 0 SMB2 read response pending 0 SMB2 read request timeout 0 SMB2 files with high offset 0

LMDB name "" Last LMDB position 0 LMDB error code Successful return: 0

diagnose sys flash list Partition Image TotalSize(KB) Used(KB) Use% Active 1 FGT60E-6.04-FW-build1828-210217 253920 87556 34% Yes 3 ETDB-86.00356 3021708 171920 6% No

get hardware nic internal2 Description :FortiASIC NP6LITE Adapter Driver Name :FortiASIC NP6LITE Driver Board :60E lif id :4 lif oid :68 netdev oid :68 tx group :0 Current_HWaddr 00:09:0f:09:00:03 Permanent_HWaddr 04:d5:90:80:4f:4c ========== Link Status ========== Admin :up netdev status :up autonego_setting:1 link_setting :0 speed_setting :10 duplex_setting :0 Speed :1000 Duplex :Full link_status :Up ============ Counters =========== Rx Pkts :844356 Rx Bytes :109228036 Tx Pkts :853088 Tx Bytes :138421768 Host Rx Pkts :776075 Host Rx Bytes :87899692 Host Tx Pkts :827977 Host Tx Bytes :119293571 Host Tx dropped :0 FragTxCreate :0 FragTxOk :0 FragTxDrop :0

diagnose sys session stat

misc info: session_count=550 setup_rate=1 exp_count=0 clash=25 memory_tension_drop=11971 ephemeral=0/120832 removeable=0 npu_session_count=15 nturbo_session_count=0 delete=341, flush=7, dev_down=1596/151 ses_walkers=0 TCP sessions: 57 in ESTABLISHED state 66 in SYN_SENT state 33 in SYN_RECV state 3 in FIN_WAIT state 4 in TIME_WAIT state 5 in CLOSE state 2 in CLOSE_WAIT state firewall error stat: error1=00000000 error2=00000000 error3=00000000 error4=00000000 tt=00000000 cont=000494dd ids_recv=00142486 url_recv=00000000 av_recv=018af3cc fqdn_count=00000026 fqdn6_count=00000002 global: ses_limit=0 ses6_limit=0 rt_limit=0 rt6_limit=0

get system performance firewall statistics getting traffic statistics... Browsing: 472437 packets, 212759502 bytes DNS: 19265 packets, 2113824 bytes E-Mail: 30 packets, 1720 bytes FTP: 52 packets, 3008 bytes Gaming: 0 packets, 0 bytes IM: 0 packets, 0 bytes Newsgroups: 0 packets, 0 bytes P2P: 0 packets, 0 bytes Streaming: 0 packets, 0 bytes TFTP: 194 packets, 16496 bytes VoIP: 10 packets, 5150 bytes Generic TCP: 2120216 packets, 386914506 bytes Generic UDP: 287721 packets, 55646988 bytes Generic ICMP: 494656 packets, 23662360 bytes Generic IP: 16 packets, 1523 bytes