Support Forum
The Forums are a place to find answers on a range of Fortinet products from peers and product experts.
albaker
New Contributor

Possible memory issues with 7.2.8

We recently upgraded multiple FortiGates (60F through 2600F) to 7.2.8 the day after the latest release was made available. Last week, one of these (60F) stopped passing traffic. We could ping the management interface and could do a "tnc -p 443 <IP>" where we'd see the 3-way handshake in a packet capture, but the login page would time out. We tried to console in - there was no prompt, but it'd echo back what we typed in. I did try an "exec reload", but nothing happened. But then, we couldn't get authenticated. This firewall required a hard reboot to bring back online. The only significant things in the system logs were these two events:

 

- Critical: Kernel enters memory conserve mode

- Critical: Kernel enters extreme low memory mode

 

This was just a few msec after an antivirus update, but I'm not certain if they are related.

 

We had the exact same thing happen today on another FortiGate. We have an upgrade scheduled for the main hospital this Friday, but I'm very hesitant in proceeding. I don't want any problems like this to occur. 

 

I can't find any bugs like this for versions around 7.2.5-7.2.8, but we do need to upgrade because of the recently announced vulnerabilities. Does anyone have information as to what might be going on, or maybe a better way to determine the root cause? 

 

Thank you.

1 Solution
ozkanaltas
Valued Contributor II

Hello @albaker ,

 

I think you are faced with an ASIC bug. You can find a document about this bug on the firmware download page. 

 

image.png

 

 

 

 

 

image.png

If you have found a solution, please like and accept it to make it easily accessible to others.
NSE 4-5-6-7 OT Sec - ENT FW

View solution in original post

If you have found a solution, please like and accept it to make it easily accessible to others.NSE 4-5-6-7 OT Sec - ENT FW
10 REPLIES 10
Toshi_Esumi
SuperUser
SuperUser

First, keep checking those two devices' memory usage with "get sys performance status". Then you see it's creeping up, check what daemon(s) are holding up the memory with like "diag sys top 5 40", Ctrl-m key to sort by memory usage.
Once you know the guilty daemon, look for any known issue in 7.2.8 releasenotes at the same time open a ticket at TAC to get it analyzed. They might be able to find a known issue, which is not in the releasenotes yet or might create a bug report after gathering enough information from your FGTs.

Toshi

Toshi_Esumi

There was an error in my post above. To sort the output of "diag sys top" by memory usage, use Shift-m (instead of Ctrl-m).

Toshi

ozkanaltas
Valued Contributor II

Hello @albaker ,

 

I think you are faced with an ASIC bug. You can find a document about this bug on the firmware download page. 

 

image.png

 

 

 

 

 

image.png

If you have found a solution, please like and accept it to make it easily accessible to others.
NSE 4-5-6-7 OT Sec - ENT FW
If you have found a solution, please like and accept it to make it easily accessible to others.NSE 4-5-6-7 OT Sec - ENT FW
albaker

Thanks Toshi. We've been monitoring, and things look OK so far.

 

ozkanatlas, the two firewalls we've had to hard reboot have the affected ASIC versions, although I wouldn't call the firewall being down until the power is recycled a momentary interruption. We'll contact support for this. Thanks for the info.

 

hbac

Hi @albaker,

 

For this kernel panic bug, we have a special firmware image with a fix. Please contact Fortinet TAC and the fix will be provided. 

 

Regards, 

Toshi_Esumi

@hbacDoes this happen only to the models that have an NP6xlite?

Toshi

albaker1

For us, that is the case.

hbac

Hi @Toshi_Esumi,

 

It happens to models with NP6/NP6Lite/NP6xLite. You can refer to the release notes with bug ID 1012518. https://docs.fortinet.com/document/fortigate/7.2.8/fortios-release-notes/236526/known-issues

 

Regards, 

Toshi_Esumi

Ok, thanks.

 

Toshi

Announcements

Select Forum Responses to become Knowledge Articles!

Select the “Nominate to Knowledge Base” button to recommend a forum post to become a knowledge article.

Labels
Top Kudoed Authors