1500D's (HA pair) crashing *every night* for weeks. * Issue Identified - IPSec VPN Bug *

[cpetry]

We are running the newest 5.4 code release on a pair of HA 1500D's.  These units are crashing every night now.  It's been happening for weeks.  FortiNet support has absolutely no idea what is going on.  This is becoming detrimental to our business.

Is this happening to anyone else?  Does this FortiOS dump everything if the crash is due to a Kernel panic, or do we need to hook something up to the physical console port (a PC with console software?, if so what app would we use that could keep the console text redirection going for over 24 hours?, no time out issues?).

They crash to the point where they don't respond to anything and the power cables have to be pulled.  These are brand new units.

[cpetry]

simonx wrote:

if it is related to voip traffic. then we are a very strong case. as my client is a hard core VOIP provider.

and this might explain as an issue with voip traffic killing the machine.

important notices, while the machine is running, it's on 5% cpu, low memory consumption, nothing that might look like stress at all.

I'll let you know once I found out more about this bug they are talking about.  They don't do bug tracking for security purposes.  So I can't lookup the bug ID they have in the ticket.

[cpetry]

We figured it out - it was a bug with VPN IPSec Dialup.  So SSL VPN works but there's an issue with how the system removes the interfaces if someone connects with VPN IPSec Dialup.

So when you use IPSec VPN it creates an interface on the firewall.  When you disconnect it destroys the interface that was created.  There's a bug that causes the system to loop while trying to destroy the interface.  It can't do it, so it keeps trying, and eventually it will peg the CPU out.

So when it pegs out the CPU on the master, that firewall will fail, and HA kicks in.  Then the loop starts over on the secondary and that one goes down for the same reason.

So for now we've removed support for our users to use IPSec for VPN connectivity.  IPSec has higher performance so we made it an option - SSL or IPSec and provided instructions for both.  

Well, the bug is being fixed in 5.4.1.

Edit: Technically it happens anytime you *remove* an interface.  When a user disconnects from IPSec VPN Dialup it removes an interface and the bug surfaces.  So don't use IPSec VPN Dialup and don't remove interfaces for now (yeah I know...)