Fortigate Conserve Mode

[vex]

Hi community.

My organization has 8 locations (two datacenters) where we are using FortiGate's as NGFW. We are also using FortiClient EMS with 1200 EPP licenses and Forti Analyzer Subscription.

Enough introduction :)

In six months on our HQ location FortiGate 81F (Cluster of two in A-P HA) has entered conserve mode without any particular reason. Last time it happened was 3 weeks ago where our primary unit went into conserve mode because of memory utilization, then we did not monitor system statistics and all we had was crash-log which was not helpful. Fortinet support said that they don't know what triggered conserve mode. Then I've reset unit and upload new OS via TFTP in order to fix the device. All HW test were done, and all passed. This unit was added back into cluster as secondary unit. 

Few days ago, we experienced conserve mode again on primary unit which was secondary 3 weeks ago. This time I have added BETA sensor into PRTG to see what is going on. Of course, all debug logs and crash logs were of no use since there was only one line: Kernel triggered conserve mode or something like that. As in first case there were no signs why this happened and Fortinet support, of course doesn't know why.

Unit operates in PEAK at 15000-20000 sessions, 350 IPsec remote users (FortiClient VPN), 100 LAN users and 300Mbps traffic. And this is at peak hours. We have 90 firewall policies and on few of them have IPS, AV, WEB Filter and DNS filter active - mostly WAN facing policies.

CPU is at 1-7% and memory at 43-46% most of the time and when conserve mode activated memory jumped from 44% to 87+% in less than one minute, maybe in seconds (PRTG sensor graphs every minute so I can't be sure how fast). It took 5-20 minutes until secondary take over. Primary is inaccessible - SSH, GUI, Console - nothing works until reboot.

What changed in last 6 months is Security Fabric enable, added Forti Analyzer and all employees must use VPN but that is on all locations and only on this one we are facing conserve mode issues.

All branch FGT's are 60F, 61F, 81F units. And DCs are 101F. All operates at 7.0.7 Firmware.

I'm turning to community because Fortinet support is OMG. I can't every time deal with endless log uploads, explaining every single step over and over again to receive nothing! I feel like my technical issues are first and no one at support experienced them so far. I'm not getting straight answers and 90% issues I have to figure out myself...

Can community help me with this issue? Did someone have this situation?

If I need more powerful units, I will get them but the hell with all of this if I can't get straight answer where someone will say: Yes, our product matrix is marketing scam and units like 81 and lower (maybe even 101) can't deal with your amount of traffic, IPsec's etc. or our SMB units can't handle v7 software...

Thank you.

[gfleming]

Sounds like a memory leak. Either wait for TAC to respond to your ticket or check your log outputs yourself. The "top" command will show process memory utiliztiaon. Usually the WAD process is the culprit of mem leaks.

[vex]

Thanks for reply.

I don't think I can identify WAD process that causes mem leak before it occurs. I experienced mem leak where memory gradually rises in greater period of time, one month or so. This leak (if it is leak) occurs instant and there is no way I can be that fast to diag with top. FGT unit is inaccessible when conserve mode activates to diagnose it after.

Or maybe I'm wrong and there is way?

[gfleming]

Oh ok I see your issue now. Didn't realize it was an instant spike. Are you logging to a server? Or using FAZ or something like that? Any hints in the traffic or security logs?

Also do you happen to be using an ISDB entries in your policies?

[vex]

Hi.

I'm using FAZ as log server. I went through all traffic and security logs and nothing indicates why conserve mode activates in first place.

I'm not using ISDB on policies.

Instant was badly choosen word, it was fast. My monitoring system says that 10.15 was 48% mem utilization, at 10.16 was 62% and at 10:17 was 87% and conserve mode active.

[gfleming]

OK I suggest closely monitor the memory utilization and when you see it rising again check the mem usage for your processes on the FGT.