Several problems high memory and cpu usage blocking WAN connection after upgrade to 6.2

[gcraenen]

Hi,

After upgrading from 6.0.4 to 6.2 I have problems with WAN connectivity falling out. I'm getting this message in de Fortios GUI:

Conserve mode activated due to high memory usage

I have tried to downgrade to 6.0.4 but can't with the error message that it failed beacause it cannot download the file from fortiguard. Help.

[SMabille · Accepted solution]

Hi,

Download the image from the Fortinet support website and upload/apply it from your browser.

Best regards,

Stephane

[Dave_Hall · Accepted solution]

This is by no means a fix, but a work-around is to have the fgt perform a daily reboot. 

config system global set daily-restart enable set restart-time <time value> end

[Adrian_Lewis · Accepted solution]

Vinicius wrote:

Do you know when will the 6.2.2 be released?

No specific dates are available only targeted dates which can slip if any issues are identified. Currently target though I hear is this week.

[ISOffice · Accepted solution]

Hi all,

We upgraded our 100D appliances to 6.2.2 a week ago and noticed a slight improvement in GUI performance when viewing logs in Log & Report. However, when filters were applied the CPU once again spiked to 90+% with multiple instances of the 'log_se' process running.

I have an ongoing support call logged with Fortinet and their TAC Engineer (cheers Kevin!!) suggested that I fail over to the 'slave' appliance to see if the issue could be replicated there. To my surprise, it wasn't. The GUI was responsive and I could apply filters to viewing logs and the CPU didn't spike. The only difference between the appliances was that the 'master' appliance had used 75% of it's disk for log storage and the 'slave' appliance only 2% (we are running active-passive). When viewing logs (with filters applied) on the 'slave' appliance the 'log_se' process didn't even appear when I ran 'diag sys top'.

Long story short, on failing back to the 'master' appliance I deleted all logs on the local disk (execute log delete-all) and can now view logs, with multiple filters, quite easily and without overly stressing the CPU. We syslog traffic logs in real-time to a third party product, so it isn't critical for us to retain logs on the appliance's hard disk for extended periods of time.

Querying local disk logs, particularly if there are many of them, possibly results in high CPU usage due to the high i/o requirements to do so.

Whilst this may not be relevant to many others in this discussion, I thought I would share our experience anyway.

Best regards,

John P

[ISOffice · Accepted solution]

Hi all,

We upgraded our 100D appliances to 6.2.2 a week ago and noticed a slight improvement in GUI performance when viewing logs in Log & Report. However, when filters were applied the CPU once again spiked to 90+% with multiple instances of the 'log_se' process running.

I have an ongoing support call logged with Fortinet and their TAC Engineer (cheers Kevin!!) suggested that I fail over to the 'slave' appliance to see if the issue could be replicated there. To my surprise, it wasn't. The GUI was responsive and I could apply filters to viewing logs and the CPU didn't spike. The only difference between the appliances was that the 'master' appliance had used 75% of it's disk for log storage and the 'slave' appliance only 2% (we are running active-passive). When viewing logs (with filters applied) on the 'slave' appliance the 'log_se' process didn't even appear when I ran 'diag sys top'.

Long story short, on failing back to the 'master' appliance I deleted all logs on the local disk (execute log delete-all) and can now view logs, with multiple filters, quite easily and without overly stressing the CPU. We syslog traffic logs in real-time to a third party product, so it isn't critical for us to retain logs on the appliance's hard disk for extended periods of time.

Querying local disk logs, particularly if there are many of them, possibly results in high CPU usage due to the high i/o requirements to do so.

Whilst this may not be relevant to many others in this discussion, I thought I would share our experience anyway.

Best regards,

John P

[Aron1]

Vinicius wrote:

Hello,

Anyone having this issue on 6.2.1?

Every week i must restart those cause they run into conserve mode.

Already set-memory-threshold-green to 89 and red to 90.

I'm running 6.2.1 on a 30E and on 60E.

I have a client with 12 forti FW's. Upgraded 4 of them to 6.2.1. 2 were 5.6.7 up to 6.2.1, the other 2 6.2.0 to 6.2.1. The 2 that were running 6.2.0 and having memory issues no longer are having memory issues. On those two, the applincation that was causing the mem issue was wad on one unit and ipsmonitor on the other. Restarting those apps with the diag test app **** 99 command is only a temporary fix from what I have seen.

While they are not seeing the memory issue again, I ran into an SSL inspection issue with 6.2.1 on an online application they are running. I may be opening a Case with Fortinet on that one soon.

[FortiET]

We have Fortigate 200 E  - cluster.  Today I ran into the Conserve mode.  diagnose test application wad 99 fixed the problem at once.  But Fortinet - any updates about fix for this problem ?

[zaphod]

FortiET wrote:

We have Fortigate 200 E  - cluster.  Today I ran into the Conserve mode.  diagnose test application wad 99 fixed the problem at once.  But Fortinet - any updates about fix for this problem ?

have you updated to 6.2.1 ?

[FortiET]

Sorry, Yes, I forgot to write our version. We have run 6.2.1  Build 0932 for quite some weeks.

[scan]

Hi

Did you check why you Fortigate is in the conserve Mode?

Here some ideas to figure it out why your Fortigate is in conserve Mode:

diagnose sys top-summary #Check which process use a lot of memory oder cpudiagnose test application <Application-Name> #Restart Application 

In the most situation a kill of the process do not solve the issue. Often you get other issues sometimes a couple of days or weeks later!

[dogan_md]

I'm using 90E, I had memory problems in version 6.2, then I switched to 6.2.1. But I'm having the same problem even more often. I couldn't restart the device from the menu when this last problem occurred, I had to unplug the power cord and plug it again.

[aroch]

Hello,

using 200E Cluster with 6.2.1, uptime 36 days and today we had the same issue.

diagnose test application wad 99 fixed it for the first time.

I have seen that 6.2.2 is out.

Can anyone confirm the problem is fixed with this ?

BR

[gweeby]

Hello,

unfortunately we still have the memory conserve problem in 6.2.2

[thuynh_FTNT]

Hi gweeby, can you provide more details about your environment? You can pm me if needed.

- What is your topology and traffic load?

get system status

get system session-info full-stat

get system session-info statistics

- Can you provide the output of the following commands? Which process is using up the memory?

get sys perf stat

get sys perf top

diagnose sys top

diagnose sys top-summary "-s mem"

- Get more ips debug if it's confirmed the process is IPS

diagnose ips memory status

diagnose ips session list by-mem 10

diagnose ips session status

diagnose ips packet status 

- Any relevant crash log?

diagnose  debug crashlog read

- Does this command fix your issue?

diagnose test application wad 99 

Thanks,

Tri