1500D's (HA pair) crashing *every night* for weeks. * Issue Identified - IPSec VPN Bug *

[cpetry]

We are running the newest 5.4 code release on a pair of HA 1500D's.  These units are crashing every night now.  It's been happening for weeks.  FortiNet support has absolutely no idea what is going on.  This is becoming detrimental to our business.

Is this happening to anyone else?  Does this FortiOS dump everything if the crash is due to a Kernel panic, or do we need to hook something up to the physical console port (a PC with console software?, if so what app would we use that could keep the console text redirection going for over 24 hours?, no time out issues?).

They crash to the point where they don't respond to anything and the power cables have to be pulled.  These are brand new units.

[Dave_Hall]

Something like this I would hook up a console/rollover cable to the console port and putty into it.  Keep the session open and log it over night or until the fgt crashes.  Never recall having to need to keep the session open, but putty does have a keepalive feature.

 

[cpetry]

Thanks, I figured that would do it.  We are going to try turning traffic shaping off for now since the issues also started happening when we enabled that.  If that doesn't correct it the next step is a PC/putty setup to log the system crash.

[ede_pfau]

hi,

from your post I assume this cluster is in production mode already. If I were in charge, I'd consider 2 things:

- first, and foremost, downgrade to v5.2.7 to reach a stable condition. Only if the whole operation would be hinged on some new v5.4 feature this would not be an option. At least you've got this option, and it's not a bad one.

- second, like with TS I'd remove one feature after the other to try to pinpoint the one which causes trouble. I would probably start with the HA setup and reduce it to standalone mode. At this moment the cluster doesn't seem to be of any help anyways.

[cpetry]

That's what I've already suggested.  We are turning off traffic shaping tonight.  If that doesn't work, A/V will go off next, etc.  Honestly, if it crashes again tonight, I'm hooking up a PC and console cable to gather more insight.  If it is a kernel panic we can't rely on SNMP or traps to see what's going on.

We did try switching up which one is primary/secondary and that didn't help.  That's what we did to rule out a hardware issue with the primary. 

I'll let you all know what we find out.  I'm sure we can't be the only ones with this issue.

[ede_pfau]

Regarding HA, I was not thinking of a hardware issue. The HA setup itself adds some complexity to the operation (internal synchronization traffic, communication, possibly dependency on or interaction with switches). That's why I'd run a single FGT first.

[cpetry]

Well it happened again with TS off.  So I'm heading into the office to setup my laptop with a console cable and putty.  Then I'll just enable console debugging and let it run overnight.  It's been happening every night (various hours), so I'm hopeful that we will catch the condition in the logging.

I'm going to setup a directory with Windows compression enabled since I'm sure debugging will generate a lot of data.

diag debug flow show console enable

[simonx]

as noted on the 800C thread, we have similar issue with 5.2.4 b668, 

there is NOTHING active, AV off, IPS off, it's pretty much firewall only 

the firewall freeze up to power cables pullout to resolve, the issue starts with one freezing, then around 30 min later the second one goes same direction.

there is nothing in logs, the console showing nothing but the HA member dying.

it seems like Kernel Panic, sorry to say that as there is not too many people complaining, 2 things came in mind.

what type of traffic are you running? is it by any chance VOIP related? if it is, then maybe it's related.

if it's not. then I think it's plain DEAD machines that needs to get RMA'd.

if forti can't fix it you should demand for RMA and new devices to be delivered.

[cpetry]

simonx wrote:

as noted on the 800C thread, we have similar issue with 5.2.4 b668, 

there is NOTHING active, AV off, IPS off, it's pretty much firewall only 

the firewall freeze up to power cables pullout to resolve, the issue starts with one freezing, then around 30 min later the second one goes same direction.

there is nothing in logs, the console showing nothing but the HA member dying.

it seems like Kernel Panic, sorry to say that as there is not too many people complaining, 2 things came in mind.

what type of traffic are you running? is it by any chance VOIP related? if it is, then maybe it's related.

if it's not. then I think it's plain DEAD machines that needs to get RMA'd.

if forti can't fix it you should demand for RMA and new devices to be delivered.

We actually did just recently start using new VOIP phones.  Before that we had a standard internal digital PBX system.  So it may be related.. I'd have to look at the timeline to see when it started happening in relation to when we started installing our cloud based VOIP phones.

[simonx]

if it is related to voip traffic. then we are a very strong case. as my client is a hard core VOIP provider.

and this might explain as an issue with voip traffic killing the machine.

important notices, while the machine is running, it's on 5% cpu, low memory consumption, nothing that might look like stress at all.