Several problems high memory and cpu usage blocking WAN connection after upgrade to 6.2

[gcraenen]

Hi,

After upgrading from 6.0.4 to 6.2 I have problems with WAN connectivity falling out. I'm getting this message in de Fortios GUI:

Conserve mode activated due to high memory usage

I have tried to downgrade to 6.0.4 but can't with the error message that it failed beacause it cannot download the file from fortiguard. Help.

[SMabille · Accepted solution]

Hi,

Download the image from the Fortinet support website and upload/apply it from your browser.

Best regards,

Stephane

[Dave_Hall · Accepted solution]

This is by no means a fix, but a work-around is to have the fgt perform a daily reboot. 

config system global set daily-restart enable set restart-time <time value> end

[Adrian_Lewis · Accepted solution]

Vinicius wrote:

Do you know when will the 6.2.2 be released?

No specific dates are available only targeted dates which can slip if any issues are identified. Currently target though I hear is this week.

[ISOffice · Accepted solution]

Hi all,

We upgraded our 100D appliances to 6.2.2 a week ago and noticed a slight improvement in GUI performance when viewing logs in Log & Report. However, when filters were applied the CPU once again spiked to 90+% with multiple instances of the 'log_se' process running.

I have an ongoing support call logged with Fortinet and their TAC Engineer (cheers Kevin!!) suggested that I fail over to the 'slave' appliance to see if the issue could be replicated there. To my surprise, it wasn't. The GUI was responsive and I could apply filters to viewing logs and the CPU didn't spike. The only difference between the appliances was that the 'master' appliance had used 75% of it's disk for log storage and the 'slave' appliance only 2% (we are running active-passive). When viewing logs (with filters applied) on the 'slave' appliance the 'log_se' process didn't even appear when I ran 'diag sys top'.

Long story short, on failing back to the 'master' appliance I deleted all logs on the local disk (execute log delete-all) and can now view logs, with multiple filters, quite easily and without overly stressing the CPU. We syslog traffic logs in real-time to a third party product, so it isn't critical for us to retain logs on the appliance's hard disk for extended periods of time.

Querying local disk logs, particularly if there are many of them, possibly results in high CPU usage due to the high i/o requirements to do so.

Whilst this may not be relevant to many others in this discussion, I thought I would share our experience anyway.

Best regards,

John P

[thuynh_FTNT]

Thanks jminard for the feedbacks. >We are seeing this with several Fortigate 60E's that are running 6.0.6. I just had to have a client power-cycle theirs 10 minutes ago and another client about 4 hours ago. I didn't get a chance to see what process was consuming the memory since I couldn't even get logged in to the GUI before it was power-cycled. I'm really not sure what to do. I'd recommend the first step is to review what happened to those instances. If it keeps happening, we should be able to collect logs and debug messages for further check. Can you check if there is any relevant crashlog? >Yeah, I saw that 6.0.7 was recently released, but I haven't went through the release notes yet to see if anything like this was addressed. In this case, I'd recommend to review the release note :) Updating your firmware to latest patch version is always worth a shot. >I don't think Fortinet has ever came out and said there is a bug in any version except 6.2.1, so I'm not real hopeful that it will be listed as one of the fixes.  It's reasonable to expect bugs in early release which is why we are committed to continuously release follow-up patch version to address important issues. However, some issues are hard to catch and require complex production environment to reproduce, in which case, we rely on our community for feedbacks. So please follow up with your local SE/support to report the issue and we can escalate it accordingly.

[gcraenen]

It's been almost a year since I first posted this message. I have just upgraded to firmware 6.2.3 and this problem is still not solved. After I switch on NZBGET to download files, the ipsengine goes bezirk and puts my 61E in conserve mode due to high CPU usage.

I reverted back to firmware 6.08 and everything is fine.

Incredible that a company cannot solve this in almost a year.

[jminard]

We have a case open with support for the conserve mode issue. We were running 6.0.x and they upgraded the box to 6.2.3. Still had issues. Their latest attempt to resolve it was to switch the box from proxy mode to flow mode for UTM. We've always used proxy mode, so I'm not sure what all that is going to impact. I have to check with my tech that is working on that client to see if it has made the problem go away or not.

[tanr · Accepted solution]

@gcraenen, did you open a support ticket with TAC?  I known they've fixed a number of bugs between 6.2.0 and 6.2.3, some specific to ipsengine, but if your specific issue and repro case hasn't been reported them then it's unlikely to have gotten fixed.

Not saying that 6.2.3 is stable enough, though!  We usually wait till the .4 releases to start testing them for possible production use.  I'm reasonably hopeful

[thuynh_FTNT]

Thanks Tanr. Yes, for those who are experiencing high memory and CPU usage issue, please report it through our customer support service so we can review each case individually. We did fix a number of resource usage issues since the last release but if the issue is specific to your topology, then we'll need to take a closer look at it. You can also share your ticket number here and we can help following up.

[robertp]

Hi Guys

We're on the 100F hardware platform, just had a call with a distributor who also confirmed that 40F and 60F series hardware platforms have a known issue with the new ASIC and as such are not to be shipped.

We have engineering IPS Engine code with debugging symbols running in it to try and feed the information back to the developers however we're a commercial business, we're not here to help beta software and accept the interruptions.

Fortinet this needs to work, $10,000 for a firewall and ASE contract and you have known issues in your hardware, you know which clients have the hardware bound to ASE contracts, do the right thing, contact them and offer them replacement E series.

Rob

[thuynh_FTNT]

robertp wrote:

Hi Guys

We're on the 100F hardware platform, just had a call with a distributor who also confirmed that 40F and 60F series hardware platforms have a known issue with the new ASIC and as such are not to be shipped.

We have engineering IPS Engine code with debugging symbols running in it to try and feed the information back to the developers however we're a commercial business, we're not here to help beta software and accept the interruptions.

Fortinet this needs to work, $10,000 for a firewall and ASE contract and you have known issues in your hardware, you know which clients have the hardware bound to ASE contracts, do the right thing, contact them and offer them replacement E series.

Rob

Hi Rob, thank you for your feedback. Did you already have a Customer Support ticket for this? If so can you share it with me (can PM me if needed). We can follow up there.

[robertp]

Hi Guys

We're on the 100F hardware platform, just had a call with a distributor who also confirmed that 40F and 60F series hardware platforms have a known issue with the new ASIC and as such are not to be shipped.

We have engineering IPS Engine code with debugging symbols running in it to try and feed the information back to the developers however we're a commercial business, we're not here to help beta software and accept the interruptions.

Fortinet this needs to work, $10,000 for a firewall and ASE contract and you have known issues in your hardware, you know which clients have the hardware bound to ASE contracts, do the right thing, contact them and offer them replacement E series.

Rob

[rt72]

gcraenen wrote:

Hi,

After upgrading from 6.0.4 to 6.2 I have problems with WAN connectivity falling out. I'm getting this message in de Fortios GUI:

Conserve mode activated due to high memory usage

I have tried to downgrade to 6.0.4 but can't with the error message that it failed beacause it cannot download the file from fortiguard. Help.

What was the resolution for this issue, if any ?

We have our firewalls on 6.0.8.

[thuynh_FTNT]

rt72@queensu.ca wrote:

What was the resolution for this issue, if any ?

We have our firewalls on 6.0.8.

We have fixed a number of CPU/memory issues since then but we can only confirm the fix for the issues that were reported via Customer Support ticket, which has enough data point for the conclusion. So this really depends on your unique topology.

As a general reminder, please make sure to raise a Customer Support ticket for your own issue and provide as much debug information as possible. That's the only way we can review your issue and provide a proper fix. This forum is meant for quick report and it works well with simple issue but for complex issue, customer support ticket is the way to go.

gcraenen wrote:

Hi,

After upgrading from 6.0.4 to 6.2 I have problems with WAN connectivity falling out. I'm getting this message in de Fortios GUI:

Conserve mode activated due to high memory usage

I have tried to downgrade to 6.0.4 but can't with the error message that it failed beacause it cannot download the file from fortiguard. Help.

Hi gcraenen, not sure if you were able to downgrade the image? Where did you try to download the firmware? Was it via FortiOS GUI or FortiGuard portal?