800c Intermittently Locks Up After 5.2.2 Update

[layer8problemsolver]

Followed supported upgrade path for 5.2.2.  Thought everything went well.  Discovered one bug/issue with the modification of our ALL service.  It was changed from 0 to 6 during the update.  After changing it back to zero we didn't have any other issues.  Everything tested fine.

A couple of weeks after the update I received a page that we were down.  I went to our core to find the unit with fans running at high rpm with power light flashing red.  It appeared to be passing traffic on the other ports but it was not.  No access to the unit from either web or console.  It was completely locked up.  Pulled power and restarted.  It came back up fine but NO logs as to what happened could be found.  Called support and they could not find anything either.  Chalked it up to a fluke but a few weeks later it happened again.  Once again, no access and no logs.  It has done this around 6 times and each time there are no logs as to why it happens.  It is random.  It can go a few weeks or a few days.  I am probably going to roll back.  Any thoughts?

[Adrian_Buckley_FTNT]

What you're describing sounds like a Kernel Panic of some sort.

Basically the OS crashes .. in which case there's no possibility for any kind of log to be created during or after the fact.

Make sure you've got some kind of active monitoring in place (SMNP) to look at CPU/Memory/Session levels (it may reveal something).

Also (most importantly) connect a device to the console port and enable logging!

If it is a Kernel panic, when it happens the FortiGate will do a memory dump through the console port.  That information can then be sent to TAC and actual investigation of the cause of the crash can happen.

[Shawn_W]

Any update?

[Flyshuffle]

Hi Everyone,

We've experienced the same situation on a 300D running firmware 5.2.2 twice recently. The firewall would simply lock up and not pass any traffic. The first time we did get logs that suggested the firewall was not tearing down sessions, and the session count just increased and increased until the firewall ran our of resources and quit. The second time, however, all of the resource logs appeared normal. Normal session counts, normal CPU and memory usage, traffic usage, etc. It just stopped working and started working again once the power was recycled. 

I see the mention about having a computer plugged into the console to log what happens if this happens again. Basically, do you just have a PC connected running Fortiexplorer and it will capture the logs should the firewall crash again?

Thanks!

[Adrian_Buckley_FTNT]

The console port is a physical port on the device.

Connecting VIA FortiExplorer (or to the CLI with SSH/Telnet) is different and won't work (if it truly is a Kernel panic).

Assuming it is a Kernel panic then the device will either reboot on it's own or lock up and be totally non-accessible.  If traffic is passing or the device is responding to management traffic (Ping, GUI, etc) then it is something totally different.