memory conserve threshold values are hardcoded in FortiOS

[babarmunir]

hi we are using 2 600E in ha cluster, and today lot of complaint were received from end users that services got stop working. when i login on firewall the error was displayed as per subject and memory graph was in between 82-85%. We never faced this issue before so after few googling we tried to disable IPS but this did not help out (may be take some time to get memory down), so finally firewall was rebooted and secondary firewall becomes primary, memory goes at 67%.and everything start working fine.  can somebody please suggest

1) how can we prevent this in future as our env is a very critical and cant afford any downtime due to this issue

2) how can we generate an email alert if conserve memory threshold value exceeds 75% or is there any way to prevent this error occurring again?

2) can we add more memory to device?

[HarshChavda]

Hello @babarmunir ,

You can refer this document to change threshold for conserve mode : https://docs.fortinet.com/document/fortigate/7.6.0/administration-guide/194558/conserve-mode

You can refer this document to avoid conserve mode: https://community.fortinet.com/t5/FortiGate/Technical-Tip-Free-up-memory-to-avoid-conserve-mode/ta-p...

https://community.fortinet.com/t5/FortiGate/Troubleshooting-Tip-How-to-do-initial-troubleshooting-of...

[dkochhar]

@babarmunir Can you please attach the crash logs.
Moreover, please run the following commands if again it goes into conserve mode before rebooting the device:

get system status
get system performance status <----- Use this command three times leaving a time 1 minute between each execution.
diagnose sys top 2 40 <-----Let this command run for 1 minute, then stop it via ‘q’.
diagnose sys top-summary <----- Let this command run for 1 minute, then stop it via ‘q’ - on FortiOS 6.4 this command does not exist.
diag sys top-mem <----- Run this command 4 - 5 times.
diagnose hard sysinfo memory
diagnose hard sysinfo slab
diagnose hard sysinfo shm
diagnose hard sysinfo conserve
diagnose debug crashlog read

[babarmunir]

i have disable IPS.AV and all session traffic from most of policies but still it is on 73% which is alarming for us, following process are taking memory
wad 16.3% 

node 2.4%

ipshelper 1.9%

can you please shed light what are these about and can we kill ?

also can we add more memory to device ?

firmware ver is 7.6.0

PRD (global) # diagnose debug enable

PRD (global) # diagnose test app wad 1000
Process [0]: WAD manager type=manager(0) pid=282 diagnosis=yes.
Process [1]: type=dispatcher(1) index=0 pid=466 state=running

[HarshChavda · Accepted solution]

Hello @babarmunir ,

 You can restart this process. Reference: https://community.fortinet.com/t5/FortiProxy/Technical-Tip-How-to-restart-the-WAD-process/ta-p/21278... 

However please note that stopping WAD would disrupt web-based functionalities, including web filtering and SSL inspection, while disabling IPShelper would impact any remaining IPS-related tasks. We  recommend to restart the process during quiet hours example during midnight.

[JaydenMurphy]

Thanks for the link, you made my day.

[dingjerry_FTNT]

Hi @babarmunir ,

To answer this question "can we add more memory to device?",  the answer is no.