Fortigate HA cluster problem

[dmatta]

It's a second time in just 2 days that our firewall cluster has a failover.

From the logs we just see that the secondary is not able to see the primary, but we are not able to find the root cause.

We have 2 1000C and before the swap the only interesting log we have is the increasing of session from 10k to 250k...

Any idea?

[ashukla_FTNT]

It can happen due to multiple reasons:

1) Master cpu utilization very high (will cause ha heartbeat packets to be lost)

2) Ha heartbeat interface issue

3) Any monitored interface going down

10K to 250K session is significant increase but what matters more is new connections/second. This box support 190,000 new sessions per second so unless the box reach that limit it shouldn't be any issue

Check in log if there was high cpu on primary.

Also check if there was any crash by looking at diag debug crashlog read

To rule out interface issue configure second ha heartbeat interface on a different slot.

[dmatta]

1) Master cpu utilization very high (will cause ha heartbeat packets to be lost)

This seems the key, however i  the statistics i have are really low (1% maximum)

The bad thing is that we don't have a clear failover, but we start loosing part of the net cause of a SPLIT Brain.

We can see the ha virtual mac flapping between our l2 infastructure, then after 2 3 min where the situation comes to normality

the HA comes up and everything start working again.

2) Ha heartbeat interface issue

We have 2 different links and checking the port status they are fine.

3) Any monitored interface going down

Not at all

[ashukla_FTNT]

Have you configured snmp? If yes check the traffic graphs to see if there was any huge spike in traffic because as you said session increased form 10K to 250K which sound like really significant increase.

Another important point:

Is the heartbeat cable connected directly or through a switch?

Also as I mentioned earlier check the output of diag debug crashlog read and see if there was any crash at the same time of split brain.