Possible memory issues with 7.2.8

[albaker]

We recently upgraded multiple FortiGates (60F through 2600F) to 7.2.8 the day after the latest release was made available. Last week, one of these (60F) stopped passing traffic. We could ping the management interface and could do a "tnc -p 443 <IP>" where we'd see the 3-way handshake in a packet capture, but the login page would time out. We tried to console in - there was no prompt, but it'd echo back what we typed in. I did try an "exec reload", but nothing happened. But then, we couldn't get authenticated. This firewall required a hard reboot to bring back online. The only significant things in the system logs were these two events:

- Critical: Kernel enters memory conserve mode

- Critical: Kernel enters extreme low memory mode

This was just a few msec after an antivirus update, but I'm not certain if they are related.

We had the exact same thing happen today on another FortiGate. We have an upgrade scheduled for the main hospital this Friday, but I'm very hesitant in proceeding. I don't want any problems like this to occur. 

I can't find any bugs like this for versions around 7.2.5-7.2.8, but we do need to upgrade because of the recently announced vulnerabilities. Does anyone have information as to what might be going on, or maybe a better way to determine the root cause? 

Thank you.

[Toshi_Esumi]

First, keep checking those two devices' memory usage with "get sys performance status". Then you see it's creeping up, check what daemon(s) are holding up the memory with like "diag sys top 5 40", Ctrl-m key to sort by memory usage.
Once you know the guilty daemon, look for any known issue in 7.2.8 releasenotes at the same time open a ticket at TAC to get it analyzed. They might be able to find a known issue, which is not in the releasenotes yet or might create a bug report after gathering enough information from your FGTs.

Toshi

[Toshi_Esumi]

There was an error in my post above. To sort the output of "diag sys top" by memory usage, use Shift-m (instead of Ctrl-m).

Toshi

[ozkanaltas · Accepted solution]

Hello @albaker ,

I think you are faced with an ASIC bug. You can find a document about this bug on the firmware download page. 

[albaker]

Thanks Toshi. We've been monitoring, and things look OK so far.

ozkanatlas, the two firewalls we've had to hard reboot have the affected ASIC versions, although I wouldn't call the firewall being down until the power is recycled a momentary interruption. We'll contact support for this. Thanks for the info.

[hbac]

Hi @albaker,

For this kernel panic bug, we have a special firmware image with a fix. Please contact Fortinet TAC and the fix will be provided. 

Regards, 

[Toshi_Esumi]

@hbacDoes this happen only to the models that have an NP6xlite?

Toshi

[albaker1]

For us, that is the case.

[hbac]

Hi @Toshi_Esumi,

It happens to models with NP6/NP6Lite/NP6xLite. You can refer to the release notes with bug ID 1012518. https://docs.fortinet.com/document/fortigate/7.2.8/fortios-release-notes/236526/known-issues

Regards, 

[Toshi_Esumi]

Ok, thanks.

Toshi