Fortigate memory continues to grow

[DanieleS99]

Hi,

I don't understand why the memory of my Fortigate continues to grow, even if the traffic is approximately the same of 6 months ago...

Thanks for the support.

[sjoshi]

Dear DanieleS99,

Thank you for posting to the Fortinet Community Forum.

Problem Description:-
Fortigate memory continues to grow

What was your normal session count?

What was your normal memory usage?

Was there any changes in the firewall?

Can you please share me the following output:-
get sys perf stat
get sys perf firewall statistics
get sys perf firewall packet-distribution
diag sys session stat
diag sys top 2 99 --ctrl c after 30s
diag hard sysinfo memory
diag hard sysinfo conserve
diag debug crashlog read
get sys status

#conf sys global
get

Let us know if this helps.

Thanks

[DanieleS99]

Hi Joshi,

normal sessions count -> 100.000 

Sessions in this moment: 145.000

normal memory usage -> 35%

changes in the firewall -> This firewall is used only for do IPS things.. I buy the product and I have never made any changes in relation to the memory.

-------------------------

FORTIGATE # get sys perf stat
CPU states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU0 states: 0% user 1% system 0% nice 99% idle 0% iowait 0% irq 0% softirq
CPU1 states: 0% user 2% system 0% nice 98% idle 0% iowait 0% irq 0% softirq
CPU2 states: 1% user 0% system 0% nice 99% idle 0% iowait 0% irq 0% softirq
CPU3 states: 0% user 1% system 0% nice 99% idle 0% iowait 0% irq 0% softirq
CPU4 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU5 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
Memory: 8040284k total, 5937404k used (73.8%), 1669840k free (20.8%), 433040k freeable (5.4%)
Average network usage: 49048 / 50526 kbps in 1 minute, 40347 / 41626 kbps in 10 minutes, 37871 / 39144 kbps in 30 minutes
Average sessions: 151391 sessions in 1 minute, 146275 sessions in 10 minutes, 144981 sessions in 30 minutes
Average session setup rate: 197 sessions per second in last 1 minute, 185 sessions per second in last 10 minutes, 181 sessions per second in last 30 minutes
Average NPU sessions: 5487 sessions in last 1 minute, 5067 sessions in last 10 minutes, 4880 sessions in last 30 minutes
Average nTurbo sessions: 5487 sessions in last 1 minute, 5067 sessions in last 10 minutes, 4880 sessions in last 30 minutes
Virus caught: 0 total in 1 minute
IPS attacks blocked: 0 total in 1 minute
Uptime: 208 days, 3 hours, 27 minutes

----------------------------------------

FORTIGATE # get sys perf firewall statistics
getting traffic statistics...
Browsing: 63738838655 packets, 36316833226765 bytes
DNS: 3980658704 packets, 482300256471 bytes
E-Mail: 1088821775 packets, 811538202308 bytes
FTP: 4648007 packets, 3733446500 bytes
Gaming: 633 packets, 27319 bytes
IM: 3170 packets, 136412 bytes
Newsgroups: 2082 packets, 89112 bytes
P2P: 10161 packets, 445770 bytes
Streaming: 26356 packets, 1617980 bytes
TFTP: 98146 packets, 25826463 bytes
VoIP: 10229522 packets, 6818285241 bytes
Generic TCP: 1239299098 packets, 445062872848 bytes
Generic UDP: 1880994134 packets, 1370106540543 bytes
Generic ICMP: 57237297 packets, 2875309319 bytes
Generic IP: 395102 packets, 81095464 bytes

-----------------------------------------

FORTIGATE # get sys perf firewall packet-distribution
getting packet distribution statistics...
0 bytes - 63 bytes: 2444456287 packets
64 bytes - 127 bytes: 2779211593 packets
128 bytes - 255 bytes: 997826340 packets
256 bytes - 383 bytes: 243966483 packets
384 bytes - 511 bytes: 110574496 packets
512 bytes - 767 bytes: 10772404 packets
768 bytes - 1023 bytes: 3827786 packets
1024 bytes - 1279 bytes: 3822652 packets
1280 bytes - 1500 bytes: 39902344 packets
> 1500 bytes: 0 packets

-----------------------------------------

FORTIGATE # diag sys session stat
misc info: session_count=150382 setup_rate=134 exp_count=0 clash=100415
memory_tension_drop=0 ephemeral=0/588800 removeable=0
npu_session_count=5292
nturbo_session_count=5292
delete=249834, flush=190, dev_down=113/2941 ses_walkers=0
TCP sessions:
116275 in ESTABLISHED state
12 in SYN_SENT state
573 in FIN_WAIT state
12 in TIME_WAIT state
51 in CLOSE state
63 in CLOSE_WAIT state
firewall error stat:
error1=00000000
error2=00000000
error3=00000000
error4=00000000
tt=00000000
cont=000b86b0
ips_recv=04ba8b87
url_recv=00000000
av_recv=000b87cd
fqdn_count=00000009
fqdn6_count=00000000
global: ses_limit=0 ses6_limit=0 rt_limit=0 rt6_limit=0

--------------------------------------------------

Run Time: 208 days, 3 hours and 30 minutes
0U, 0N, 1S, 99I, 0WA, 0HI, 0SI, 0ST; 7851T, 1623F
snmpd 220 S 1.5 0.1 1
ipsengine 20807 S < 1.0 9.6 3
ipsengine 20808 S < 1.0 9.5 4
ipsengine 20806 S < 0.5 9.6 2
ipsengine 20805 S < 0.5 9.5 1
ipsengine 20809 S < 0.5 9.5 5
miglogd 272 S 0.5 0.8 4
hasync 10894 S < 0.5 0.5 3
syslogd 202 S 0.5 0.3 2
dnsproxy 255 S 0.5 0.2 5
ipshelper 20804 S < 0.0 3.0 3
node 198 S 0.0 1.2 0
miglogd 273 S 0.0 0.8 4
miglogd 274 S 0.0 0.8 0
cmdbsvr 173 S 0.0 0.7 5
forticron 205 S 0.0 0.5 4
miglogd 216 S 0.0 0.5 5
updated 3317 S 0.0 0.4 2
extenderd 243 S 0.0 0.3 5

----------------------------------------------------

FORTIGATE# diag hardware sysinfo memory
MemTotal: 8040284 kB
MemFree: 1669308 kB
Buffers: 22416 kB
Cached: 925400 kB
SwapCached: 0 kB
Active: 4577376 kB
Inactive: 361288 kB
Active(anon): 4291244 kB
Inactive(anon): 106112 kB
Active(file): 286132 kB
Inactive(file): 255176 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 0 kB
Writeback: 0 kB
AnonPages: 3990932 kB
Mapped: 211908 kB
Shmem: 406508 kB
Slab: 462540 kB
SReclaimable: 23200 kB
SUnreclaim: 439340 kB
KernelStack: 2624 kB
PageTables: 44644 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 4020140 kB
Committed_AS: 31223748 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 712256 kB
VmallocChunk: 34358931064 kB
DirectMap4k: 8192 kB
DirectMap2M: 8232960 kB

----------------------------------

FORTIGATE # diag hard sysinfo conserve
memory conserve mode: off
total RAM: 7851 MB
memory used: 5797 MB 73% of total RAM
memory freeable: 422 MB 5% of total RAM
memory used + freeable threshold extreme: 7458 MB 95% of total RAM
memory used threshold red: 6909 MB 88% of total RAM
memory used threshold green: 6438 MB 82% of total RAM

-----------------------------------

FORTIGATE# diag debug crashlog read
1: 2022-01-14 10:21:07 the killed daemon is /bin/eap_proxy: status=0x0
2: 2022-01-14 10:37:29 the killed daemon is /bin/sflowd: status=0x0
3: 2022-01-17 10:41:38 the killed daemon is /bin/cw_acd: status=0x0
4: 2022-01-18 12:07:34 Interface port5 is brought down. process_id=16295, process_name="newcli"
5: 2022-01-19 09:42:52 Interface port5 is brought up. process_id=173, process_name="cmdbsvr"
6: 2022-01-20 10:11:08 the killed daemon is /bin/dhcpd: status=0x0
7: 2022-01-20 10:11:08 the killed daemon is /bin/radvd: status=0x0
8: 2022-01-22 11:55:54 the killed daemon is /bin/quard: status=0xf
9: 2022-01-22 11:55:54 the killed daemon is /bin/voipd: status=0xf00
10: 2022-01-22 12:41:45 the killed daemon is /bin/getty: status=0x0
11: 2022-03-19 11:21:07 the killed daemon is /bin/updated: status=0x0
Crash log interval is 3600 seconds
Max crash log line number: 16384

---------------------------------------

AS-FORTIPS01 # get sys status
Version: FortiGate-400E-Bypass v7.0.3,build0237,211207 (GA)
Firmware Signature: certified
Virus-DB: 1.00000(2018-04-09 18:07)
Extended DB: 1.00000(2018-04-09 18:07)
Extreme DB: 1.00000(2018-04-09 18:07)
AV AI/ML Model: 0.00000(2001-01-01 00:00)
IPS-DB: 6.00741(2015-12-01 02:30)
IPS-ETDB: 21.00370(2022-08-09 01:16)
APP-DB: 21.00370(2022-08-09 01:16)
INDUSTRIAL-DB: 6.00741(2015-12-01 02:30)
IPS Malicious URL Database: 4.00434(2022-08-10 01:54)
Serial-Number: "Obscuded because i don't want to share it"
BIOS version: 05000002
System Part-Number: P26084-02
Log hard disk: Not available
Hostname: FORTIGATE
Private Encryption: Disable
Operation Mode: Transparent
Current virtual domain: root
Max number of virtual domains: 10
Virtual domains status: 0 in NAT mode, 1 in TP mode
Virtual domain configuration: disable
FIPS-CC mode: disable
Current HA mode: a-p, primary
Cluster uptime: 205 days, 3 hours, 32 minutes, 44 seconds
Cluster state change time: 2022-04-16 08:48:03
Branch point: 0237
Release Version Information: GA
FortiOS x86-64: Yes
System time: Wed Aug 10 15:10:06 2022
Last reboot reason: warm reboot

--------------------------------------------

AS-FORTIPS01 (global) # get
admin-concurrent : enable
admin-console-timeout: 0
admin-forticloud-sso-login: disable
admin-hsts-max-age : 15552000
admin-https-pki-required: disable
admin-https-redirect: enable
admin-https-ssl-banned-ciphers:
admin-https-ssl-ciphersuites: TLS-AES-128-GCM-SHA256 TLS-AES-256-GCM-SHA384 TLS-CHACHA20-POLY1305-SHA256
admin-https-ssl-versions: tlsv1-2 tlsv1-3
admin-lockout-duration: 60
admin-lockout-threshold: 3
admin-login-max : 100
admin-maintainer : enable
admin-port : 80
admin-restrict-local: disable
admin-scp : disable
admin-server-cert : self-sign
admin-sport : 443
admin-ssh-grace-time: 120
admin-ssh-password : enable
admin-ssh-port : 22
admin-ssh-v1 : disable
admin-telnet : enable
admin-telnet-port : 23
admintimeout : 60
alias : FortiGate-400E-Bypass
allow-traffic-redirect: enable
anti-replay : strict
arp-max-entry : 131072
auth-cert : Fortinet_Factory
auth-http-port : 1000
auth-https-port : 1003
auth-keepalive : disable
auth-session-limit : block-new
auto-auth-extension-device: enable
autorun-log-fsck : disable
av-affinity : 0
av-failopen : pass
av-failopen-session : disable
batch-cmdb : enable
block-session-timer : 30
br-fdb-max-entry : 8192
cert-chain-max : 8
cfg-save : automatic
check-protocol-header: loose
check-reset-range : disable
cli-audit-log : disable
cloud-communication : enable
clt-cert-req : disable
cmdbsvr-affinity : 0
cpu-use-threshold : 90
csr-ca-attribute : enable
daily-restart : disable
default-service-source-port: 1-65535
device-idle-timeout : 300
dh-params : 2048
dnsproxy-worker-count: 1
dst : enable
extender-controller-reserved-network: 10.252.0.1 255.255.0.0
fds-statistics : enable
fgd-alert-subscription:
fortiextender : enable
fortiextender-data-port: 25246
fortiextender-discovery-lockdown: disable
fortiextender-vlan-mode: disable
fortiservice-port : 8013
fortitoken-cloud : enable
gui-allow-default-hostname: disable
gui-certificates : enable
gui-custom-language : disable
gui-date-format : yyyy/MM/dd
gui-date-time-source: system
gui-device-latitude :
gui-device-longitude:
gui-display-hostname: disable
gui-firmware-upgrade-warning: enable
gui-forticare-registration-setup-warning: enable
gui-fortigate-cloud-sandbox: disable
gui-ipv6 : disable
gui-local-out : disable
gui-replacement-message-groups: disable
gui-rest-api-cache : enable
gui-theme : jade
gui-wireless-opensecurity: disable
ha-affinity : 0
honor-df : enable
hostname : FORTIGATE
igmp-state-limit : 3200
ip-src-port-range : 1024-25000
ips-affinity : 0
ipsec-asic-offload : enable
ipsec-ha-seqjump-rate: 10
ipsec-hmac-offload : enable
ipsec-soft-dec-async: disable
ipv6-accept-dad : 1
ipv6-allow-anycast-probe: disable
ipv6-allow-traffic-redirect: enable
irq-time-accounting : auto
language : english
ldapconntimeout : 500
lldp-reception : disable
lldp-transmission : disable
log-ssl-connection : disable
log-uuid-address : disable
login-timestamp : disable
management-ip :
management-port : 443
management-port-use-admin-sport: disable
management-vdom : root
max-route-cache-size: 0
memory-use-threshold-extreme: 95
memory-use-threshold-green: 82
memory-use-threshold-red: 88
miglog-affinity : 0
miglogd-children : 0
multi-factor-authentication: optional
ndp-max-entry : 0
pmtu-discovery : disable
policy-auth-concurrent: 0
post-login-banner : disable
pre-login-banner : disable
private-data-encryption: disable
proxy-auth-lifetime : disable
proxy-auth-timeout : 10
proxy-hardware-acceleration: enable
proxy-re-authentication-mode: session
proxy-resource-mode : disable
proxy-worker-count : 3
radius-port : 1812
reboot-upon-config-restore: enable
refresh : 0
remoteauthtimeout : 5
reset-sessionless-tcp: disable
revision-backup-on-logout: disable
revision-image-auto-backup: disable
scanunit-count : 6
security-rating-result-submission: enable
security-rating-run-on-schedule: enable
send-pmtu-icmp : enable
snat-route-change : disable
special-file-23-support: disable
speedtest-server : disable
ssh-enc-algo : chacha20-poly1305@openssh.com aes256-ctr aes256-gcm@openssh.com
ssh-kex-algo : diffie-hellman-group-exchange-sha256 curve25519-sha256@libssh.org ecdh-sha2-nistp256 ecdh-sha2-nistp384 ecdh-sha2-nistp521
ssh-mac-algo : hmac-sha2-256 hmac-sha2-256-etm@openssh.com hmac-sha2-512 hmac-sha2-512-etm@openssh.com
ssl-min-proto-version: TLSv1-2
ssl-static-key-ciphers: enable
sslvpn-cipher-hardware-acceleration: enable
sslvpn-ems-sn-check : disable
sslvpn-kxp-hardware-acceleration: enable
sslvpn-max-worker-count: 5
sslvpn-plugin-version-check: enable
strict-dirty-session-check: enable
strong-crypto : enable
switch-controller : enable
switch-controller-reserved-network: 10.255.0.0 255.255.0.0
sys-perf-log-interval: 5
tcp-halfclose-timer : 120
tcp-halfopen-timer : 10
tcp-option : enable
tcp-rst-timer : 5
tcp-timewait-timer : 1
timezone : (GMT+1:00) Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna
traffic-priority : tos
traffic-priority-level: medium
two-factor-email-expiry: 60
two-factor-fac-expiry: 60
two-factor-ftk-expiry: 60
two-factor-ftm-expiry: 72
two-factor-sms-expiry: 60
udp-idle-timer : 180
url-filter-affinity : 0
url-filter-count : 1
user-device-store-max-devices: 82332
user-device-store-max-unified-mem: 411662540
user-device-store-max-users: 82332
user-server-cert : Fortinet_Factory
vdom-mode : no-vdom
vip-arp-range : restricted
wad-affinity : 0
wad-csvc-cs-count : 1
wad-csvc-db-count : 3
wad-memory-change-granularity: 10
wad-source-affinity : enable
wad-worker-count : 6
wifi-ca-certificate : Fortinet_Wifi_CA
wifi-certificate : Fortinet_Wifi
wimax-4g-usb : disable
wireless-controller : enable
wireless-controller-port: 5246
fds-statistics-period: 60

-----------

Thanks for the support

[DanieleS99]

@sjoshi any update regard this topic?

Thanks

[sjoshi]

Thanks for the update.

I can see that your memory is around 73%.

I can see that IPS engine is consuming more memory and there are multiple swam of IPS engine.

Can you share me.

#Conf ips global

#get

Also please fine tune the IPS profile as much as possible

https://community.fortinet.com/t5/FortiGate/Technical-Tip-IPs-best-practices/ta-p/198360

Thanks

[DanieleS99]

Here the result:

FORTIGATE (global) # get
fail-open : disable
database : extended
traffic-submit : disable
anomaly-mode : continuous
session-limit-mode : heuristic
socket-size : 128 (MB)
engine-count : 0
sync-session-ttl : enable
np-accel-mode : basic
ips-reserve-cpu : disable
cp-accel-mode : advanced
deep-app-insp-timeout: 86400
deep-app-insp-db-limit: 500000
exclude-signatures : industrial
packet-log-queue-depth: 128
ngfw-max-scan-range : 4096
tls-active-probe:
interface-select-method: auto

Thanks for the link but I already follow the guidalines.