Conserve Mode Fortigate FG80F

[RolandBaumgaertner72]

Hi,

conserve mode is something we didn't have for a long time with all the FGs we are managing right now but now it happened the 3rd time with a FG80F cluster. All 3 times the FGs were restarted right away so we didnt have time to react and check.

Normally we dont have memory issues or CPU issues in no moment when we are connected, for example right now 64% memory and 0% CPU.

We changed some configurations, disabled Features which are not necessary, deactivated IPS (something we dont like), logging in the policies, etc. but it still happened.

Any ideas or suggestions?

Thanks!

[AEK]

Hi Roland

Which firmware version?

Besides you may follow this guide for troubleshooting.

https://community.fortinet.com/t5/FortiGate/Troubleshooting-Tip-How-to-do-initial-troubleshooting-of...

[RolandBaumgaertner72]

Now 7.4.4 but we had the same problems with 7.4.2 before

[jiyong]

Hi, @RolandBaumgaertner72 

In my opinion, If the problem occurs even in the latest version, it needs necessary to analyze it by registering a Forticare ticket.

[jiyong]

Please check the command below.

# diag sys top 1 20

# fnsysctl killall [high resource process] // restart process

[rosatechnocrat]

Check the number of sessions or if there are some attacks being triggered. 

The issue occurs when suddenly there is a spike on the number of sessions. 

Visit the video below to understand the conserve mode and possible issues. 

https://youtu.be/rtrg3fI5lQw?si=6a5lYbVVbVrWYZhT

[RolandBaumgaertner72]

Hi,

again we have like 75% which doesnt seem normal since we deactivated IPS and other functionalities:

MasterXXXX # get system performance status
CPU states: 2% user 0% system 0% nice 98% idle 0% iowait 0% irq 0% softirq
CPU0 states: 1% user 0% system 0% nice 99% idle 0% iowait 0% irq 0% softirq
CPU1 states: 1% user 0% system 0% nice 99% idle 0% iowait 0% irq 0% softirq
CPU2 states: 2% user 0% system 0% nice 98% idle 0% iowait 0% irq 0% softirq
CPU3 states: 3% user 0% system 0% nice 97% idle 0% iowait 0% irq 0% softirq
CPU4 states: 1% user 0% system 0% nice 99% idle 0% iowait 0% irq 0% softirq
CPU5 states: 2% user 0% system 0% nice 98% idle 0% iowait 0% irq 0% softirq
CPU6 states: 3% user 0% system 0% nice 97% idle 0% iowait 0% irq 0% softirq
CPU7 states: 3% user 0% system 0% nice 97% idle 0% iowait 0% irq 0% softirq
Memory: 3806520k total, 2911760k used (76.5%), 543368k free (14.3%), 351392k freeable (9.2%)
Average network usage: 17270 / 17732 kbps in 1 minute, 3694 / 3899 kbps in 10 minutes, 3221 / 3348 kbps in 30 minutes
Maximal network usage: 81696 / 82154 kbps in 1 minute, 81696 / 82154 kbps in 10 minutes, 81696 / 82154 kbps in 30 minutes
Average sessions: 684 sessions in 1 minute, 489 sessions in 10 minutes, 420 sessions in 30 minutes
Maximal sessions: 742 sessions in 1 minute, 744 sessions in 10 minutes, 744 sessions in 30 minutes
Average session setup rate: 9 sessions per second in last 1 minute, 4 sessions per second in last 10 minutes, 2 sessions per second in last 30 minutes
Maximal session setup rate: 18 sessions per second in last 1 minute, 31 sessions per second in last 10 minutes, 31 sessions per second in last 30 minutes
Average NPU sessions: 260 sessions in last 1 minute, 169 sessions in last 10 minutes, 138 sessions in last 30 minutes
Maximal NPU sessions: 318 sessions in last 1 minute, 318 sessions in last 10 minutes, 318 sessions in last 30 minutes
Average nTurbo sessions: 235 sessions in last 1 minute, 148 sessions in last 10 minutes, 120 sessions in last 30 minutes
Maximal nTurbo sessions: 292 sessions in last 1 minute, 292 sessions in last 10 minutes, 292 sessions in last 30 minutes
Virus caught: 0 total in 1 minute
IPS attacks blocked: 0 total in 1 minute
Uptime: 27 days, 12 hours, 32 minutes

MasterXXXX #

Last time we checked like 2-3 days and we didnt have any internet peaks. We have one MPLS for the internal traffic and one 1GB for the WWW what was never a problem.

Any ideas?

Thanks!

[ede_pfau]

Now that you finally do see the memory peak, get the info which processes eat it up:

diag sys top-summary

sort by memory usage by pressing 'm', by CPU load by pressing 'p', terminate by 'Ctrl-C' or 'q'.

Two processes which show up often are 'httpsd' and 'ipsengine'.

You could kill all spawned processes of one kind with the 'killall' command @jiyong posted. But be careful, if you kill 'init' there might be a surprise...

I would agress to open a call with FTNT along with this data.

As a last thought, not as a solution: you can set up an automation stich to run a command (like 'fnsysctl...') each time the memory high threshold is triggered. I used that once during the Xmas holidays as a clean solution was not in sight. Finally, only the next patch update fixed it.

and some docs:

https://community.fortinet.com/t5/FortiGate/Technical-Tip-Reduce-memory-usage-by-reducing-the-number...

and for the stitch

https://community.fortinet.com/t5/FortiGate/Technical-Tip-Automation-stitch-for-conserve-mode/ta-p/2...

[RolandBaumgaertner72]

Hi,

we updated this morning to 7.4.5 and memory is at 40%, almost half of before.

This seems to be a good solution, no ;)

[mpandya]

need to check crashlogs

dia deb crashlog read