Several problems high memory and cpu usage blocking WAN connection after upgrade to 6.2

[gcraenen]

Hi,

After upgrading from 6.0.4 to 6.2 I have problems with WAN connectivity falling out. I'm getting this message in de Fortios GUI:

Conserve mode activated due to high memory usage

I have tried to downgrade to 6.0.4 but can't with the error message that it failed beacause it cannot download the file from fortiguard. Help.

[SMabille · Accepted solution]

Hi,

Download the image from the Fortinet support website and upload/apply it from your browser.

Best regards,

Stephane

[Dave_Hall · Accepted solution]

This is by no means a fix, but a work-around is to have the fgt perform a daily reboot. 

config system global set daily-restart enable set restart-time <time value> end

[Adrian_Lewis · Accepted solution]

Vinicius wrote:

Do you know when will the 6.2.2 be released?

No specific dates are available only targeted dates which can slip if any issues are identified. Currently target though I hear is this week.

[ISOffice · Accepted solution]

Hi all,

We upgraded our 100D appliances to 6.2.2 a week ago and noticed a slight improvement in GUI performance when viewing logs in Log & Report. However, when filters were applied the CPU once again spiked to 90+% with multiple instances of the 'log_se' process running.

I have an ongoing support call logged with Fortinet and their TAC Engineer (cheers Kevin!!) suggested that I fail over to the 'slave' appliance to see if the issue could be replicated there. To my surprise, it wasn't. The GUI was responsive and I could apply filters to viewing logs and the CPU didn't spike. The only difference between the appliances was that the 'master' appliance had used 75% of it's disk for log storage and the 'slave' appliance only 2% (we are running active-passive). When viewing logs (with filters applied) on the 'slave' appliance the 'log_se' process didn't even appear when I ran 'diag sys top'.

Long story short, on failing back to the 'master' appliance I deleted all logs on the local disk (execute log delete-all) and can now view logs, with multiple filters, quite easily and without overly stressing the CPU. We syslog traffic logs in real-time to a third party product, so it isn't critical for us to retain logs on the appliance's hard disk for extended periods of time.

Querying local disk logs, particularly if there are many of them, possibly results in high CPU usage due to the high i/o requirements to do so.

Whilst this may not be relevant to many others in this discussion, I thought I would share our experience anyway.

Best regards,

John P

[hws_kevin]

Hi,

we have the same Problem since one Year. We have 2 Fortigate 81E and Updated from 6.0.4 to 6.2.x and have these memory Problems, so we do a Downgrade to 6.0.x. There are no Problems.

Yesterday we tested the 6.4.0 and there are the same Problems again.

[NimbleIT]

We upgraded a number of our clients to 6.2.3 and noticed the "Conserve mode activated due to high memory usage" come up once web filter or AV security profiles is enabled for the LAN to WAN policy. Once this kicks in, we see random drops in internet browsing

I did upgrade one site to 6.2.4 and that site doesn't seem to have the same problem.........? Tempted to upgrade a site that has these issues to 6.2.4 to see if that makes any difference. 

[thuynh_FTNT]

hws_kevin wrote:

Hi,

we have the same Problem since one Year. We have 2 Fortigate 81E and Updated from 6.0.4 to 6.2.x and have these memory Problems, so we do a Downgrade to 6.0.x. There are no Problems.

Yesterday we tested the 6.4.0 and there are the same Problems again.

Hey Kevin, without any additional info, it's not possible for us to accurately identify your problem. Can you report your case and topology to our Customer Support? They can help collecting some debug logs in order to narrow down which process is causing high memory usage.

[NimbleIT]

We ended up downgrading our client's FGT to 6.0.4 instead and since then, have had no complaints about random internet drop outs or memory issues.

[aleilmago]

Hi.   When I had this issue in the past (a couple of times, one of these with version 6.2.0), the solution was the upgrade to the latest version of FortiOS.

On my FortiGate's no problem with version 6.4.0.   Best. Alessandro

[aroch]

aleilmago wrote:

Hi. When I had this issue in the past (a couple of times, one of these with version 6.2.0), the solution was the upgrade to the latest version of FortiOS.

On my FortiGate's no problem with version 6.4.0. Best. Alessandro

Is it still working with 6.4.x ?

Having the same problems on 6.2.2. after 227 days of uptime, only a reboot fixed the problem.

My plan was to upgrade in the same major release to 6.2.4, but this does not seem to fix the problem when i can trust the release notes.

[ODIT]

aroch wrote:

aleilmago wrote:

Hi. When I had this issue in the past (a couple of times, one of these with version 6.2.0), the solution was the upgrade to the latest version of FortiOS.

On my FortiGate's no problem with version 6.4.0. Best. Alessandro

Is it still working with 6.4.x ?

Having the same problems on 6.2.2. after 227 days of uptime, only a reboot fixed the problem.

My plan was to upgrade in the same major release to 6.2.4, but this does not seem to fix the problem when i can trust the release notes.

Well, Just had an issue with VPN and WAN dropping. Logged into our customers 60E 6.2.1 and found that the CPU was fine but the Memory was not. It was pegged at 98% usage.

After looking over the logs and past issues with the WAD using high CPU came across a few post that seem to have lowered the Memory to 55% usage.

First we did a restart / reset of the WAD process that was consuming the majority of the Memory. Below is a direct copy of the entire process with some explanations the poster used. This lowered the memory usage by 20%. I then played around with assigning different number of processor cores to the WAD service. A 60E only can have up to 4 cores. After testing each one I found that assigning 1 core was best. now the Firewall is running stable at 55% memory. Still not ideal but way better than the it was. I will separate both items I did with a line of ******************** so everyone can see the separation between the two steps I performed. 

****RESTART / RESET THE WAD PROCESS****

"Check the overall CPU and memory status: # diagnose sys top-summary CPU [|||||||||||| ] 30.7% Mem [|||||||||||||||||||||||||||| ] 71.0% 11460M/16064M Processes: 20 (running=4 sleeping=239) PID RSS CPU% ^MEM% FDS TIME+ NAME * 262 6G 60.6 38.3 41675 48:06.38 wad [x14] 247 1G 54.0 8.2 985 17:53.21 ipsmonitor [x12] 287 1G 5.0 7.1 930 21:00.71 cw_acd 256 645M 34.9 4.0 263 59:32.15 sslvpnd [x12] 272 349M 2.5 2.2 20 51:14.84 urlfilter 241 213M 4.8 1.3 61 19:52.33 miglogd [x7] 244 147M 1.7 0.9 24 00:44.52 httpsd [x5] .....   Use the "m" key on your keyboard to sort the process groups according to memory usage. Use "q" key on your keyboard to quit the diagnose command.   Show the running processes: # diagnose sys top Use the "m" key on your keyboard to sort the processes according to memory usage. Use "q" key on your keyboard to quit the diagnose command.   Locate your wad process and his process ID, let's say for now: wad 351 S 2.7 9.0 The 351 is the process ID.   Now reset and enable debuging: # diagnose debug reset # diagnose debug enable List all your wad processes and ocate your process ID (pid): # diagnose test application wad 1000 Process [0]: WAD manager type=manager(0) pid=262 diagnosis=yes. Process [1]: type=dispatcher(1) index=0 pid=345 state=running diagnosis=no debug=enable valgrind=unsupported/disabled Process [2]: type=wanopt(2) index=0 pid=346 state=running diagnosis=no debug=enable valgrind=supported/disabled Process [3]: type=worker(3) index=0 pid=347 state=running diagnosis=no debug=enable valgrind=supported/disabled Process [4]: type=worker(3) index=1 pid=348 state=running diagnosis=no debug=enable valgrind=supported/disabled Process [5]: type=worker(3) index=2 pid=349 state=running diagnosis=no debug=enable valgrind=supported/disabled Process [6]: type=worker(3) index=3 pid=350 state=running diagnosis=no debug=enable valgrind=supported/disabled Process [7]: type=worker(3) index=4 pid=351 state=running <============================= diagnosis=no debug=enable valgrind=supported/disabled Process [8]: type=worker(3) index=5 pid=352 state=running diagnosis=no debug=enable valgrind=supported/disabled Process [9]: type=worker(3) index=6 pid=353 state=running diagnosis=no debug=enable valgrind=supported/disabled Process [10]: type=worker(3) index=7 pid=354 state=running diagnosis=no debug=enable valgrind=supported/disabled Process [11]: type=worker(3) index=8 pid=355 state=running diagnosis=no debug=enable valgrind=supported/disabled Process [12]: type=worker(3) index=9 pid=356 state=running diagnosis=no debug=enable valgrind=supported/disabled Process [13]: type=informer(4) index=0 pid=344 state=running diagnosis=no debug=enable valgrind=unsupported/disabled Now access your wad process - enter into the process menu: Remember: 2xxx - the wad process always starts with 2. x3xx - the wad process type depending on whether it is a dispatcher, worker, informer etc. in brackets (). xx04 - the index number of the process (two digits). If the index is one digit, put 0 before the index.   So it looks like this in our case for wad process with pid=351 # diagnose test application wad 2304 Set diagnosis process: type=worker index=4 pid=351   Now you are working on your wad process and you can list the available options with issuing: # diagnose test application wad WAD process 351 test usage: 1: display process status 2: display total memory usage. 99: restart all WAD processes 1000: List all WAD processes. 1001: dispaly debug level name and values 1002: dispaly status of WANOpt storages ..... etc. There is a lot of them.   What you need is to restart the process so use 99 like: # diagnose test application wad 99   Reset and disable your debug: # diagnose debug disable # diagnose debug reset   Now you can check the CPU and memory again with command: # diagnose sys top-summary Compare the results with the output at the beginning of this post. More work but better than just killing processes."

****SET # OF CORES FOR THE WAD PROCESS****

config global     config system global        set wad-worker-count (1,2,3,4...)     end I found using 1 for the count on a 60E 6.2.1 Lowered this from 79% to 55% Memory usage. I am not sure if this is a fix, work around or just worked for this device. Over the entire live of Fortinet from around the mid version of the 4's or 5 firmware Fortinet seems to have had many issues with runaway processes that cause high CPU and or Memory issues on all level of devices.

@ Fortinet fix your code and stop releasing new features until all bugs are fixed. Also, stop using us and our customers for your Beta program or start paying us to Beta your firmware upgrade(s). This is getting to be a huge joke!

[tomlinson]

I have a FG1100E HA pair. I upgraded from 6.2.4 to 6.4.4 then to 6.4.5 and we are still having issues with the IPSMonitor. I have turned off IPS completely and it still seems to be leaking. The script is in place to restart IPSMonitor every two hours. I am out of options other than going back to 6.0.x. Any other suggestions. Support has been no help. 

[Tom99]

Hi

At that time we were advised to limit the maximum number of IPS engines. Maybe it will help you too.

config ips global set engine-count 4 end

When set to 0 (by default), the FortiGate unit determines the optimal number of intrusion protection engines.

Tom

[fuchs]

Subject 30E: A few days after updating from 6.2.7 to 6.2.8, the error occurred on June 1 with a 30E.

after

diagnose test application ipsmonitor 99

it works again