- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Possible memory issues with 7.2.8
We recently upgraded multiple FortiGates (60F through 2600F) to 7.2.8 the day after the latest release was made available. Last week, one of these (60F) stopped passing traffic. We could ping the management interface and could do a "tnc -p 443 <IP>" where we'd see the 3-way handshake in a packet capture, but the login page would time out. We tried to console in - there was no prompt, but it'd echo back what we typed in. I did try an "exec reload", but nothing happened. But then, we couldn't get authenticated. This firewall required a hard reboot to bring back online. The only significant things in the system logs were these two events:
- Critical: Kernel enters memory conserve mode
- Critical: Kernel enters extreme low memory mode
This was just a few msec after an antivirus update, but I'm not certain if they are related.
We had the exact same thing happen today on another FortiGate. We have an upgrade scheduled for the main hospital this Friday, but I'm very hesitant in proceeding. I don't want any problems like this to occur.
I can't find any bugs like this for versions around 7.2.5-7.2.8, but we do need to upgrade because of the recently announced vulnerabilities. Does anyone have information as to what might be going on, or maybe a better way to determine the root cause?
Thank you.
Solved! Go to Solution.
- Labels:
-
FortiGate
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello @albaker ,
I think you are faced with an ASIC bug. You can find a document about this bug on the firmware download page.
NSE 4-5-6-7 OT Sec - ENT FW
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
First, keep checking those two devices' memory usage with "get sys performance status". Then you see it's creeping up, check what daemon(s) are holding up the memory with like "diag sys top 5 40", Ctrl-m key to sort by memory usage.
Once you know the guilty daemon, look for any known issue in 7.2.8 releasenotes at the same time open a ticket at TAC to get it analyzed. They might be able to find a known issue, which is not in the releasenotes yet or might create a bug report after gathering enough information from your FGTs.
Toshi
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
There was an error in my post above. To sort the output of "diag sys top" by memory usage, use Shift-m (instead of Ctrl-m).
Toshi
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello @albaker ,
I think you are faced with an ASIC bug. You can find a document about this bug on the firmware download page.
NSE 4-5-6-7 OT Sec - ENT FW
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thanks Toshi. We've been monitoring, and things look OK so far.
ozkanatlas, the two firewalls we've had to hard reboot have the affected ASIC versions, although I wouldn't call the firewall being down until the power is recycled a momentary interruption. We'll contact support for this. Thanks for the info.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi @albaker,
For this kernel panic bug, we have a special firmware image with a fix. Please contact Fortinet TAC and the fix will be provided.
Regards,
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@hbacDoes this happen only to the models that have an NP6xlite?
Toshi
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
For us, that is the case.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi @Toshi_Esumi,
It happens to models with NP6/NP6Lite/NP6xLite. You can refer to the release notes with bug ID 1012518. https://docs.fortinet.com/document/fortigate/7.2.8/fortios-release-notes/236526/known-issues
Regards,
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Ok, thanks.
Toshi