Troubleshooting Tip: High availability intermittence issues

[Muhammad_Haiqal]

Description

This article describes share common possibilities that trigger High Availability intermittence issues.

Scope

FortiGate.

Solution

The HA design works as expected during the implementation based on existing units, VLAN, cabling, and other similar factors.
However, after some time has passed, the HA ceases to work as expected. This is a normal behavior claimed by the network administrator.

The normal changes that happened on the network:

1. A new unit or server is introduced to the network:

• Previously, the FortiGate was standalone, but is currently High-Availability.
• There is a switch from core 1 to core 2.
• An intermediate unit is introduced (such as a load balancer, proxy, or traffic management unit).
  
  

2. Changes are made to network design/topology:

• An increase to VLANs:
• Changes of routing.
• LACP configuration.
• Changes of cabling or ports on the FortiGate or switch.
• STP re-calculation.
  
  

When changes to the design occur, the traffic may not work as it did in the first implementation.
This is expected behavior.

In certain cases, there are no changes on the network.
However, failover does not work as expected.

The expected behavior is as follows:

1. Traffic on the primary unit is working as expected.
2. During failover to the secondary unit:

• Many services are down.
• A certain segment/interface is not working.
• Network intermittence / flapping occurs (for example: the connection works for 10 minutes, then goes down for 10 minutes).
  
  

From the FortiGate perspective, FortiGate only processes the traffic as it is received.

A common issue occurs due to STP (Spanning Tree Protocol) on the network level.
This frequently happens if aggregation or LACP is configured.

Basically, this issue is due to the network design itself.

Troubleshooting:

If intermittence occurs, this can be checked on the FortiGate as follows:

Version 6.0.
Go to Log & Report -> System Events.

Version 6.2 and above.
Navigate to Log & Reports -> Events -> System Events (on top right corner).

Filter for Log Description: Interface status changed.

Look for the interface that is encountering the problem. The interface status should appear as follows:

FortiGate only produces notifications - not actions.

When FortiGate detects that the port is down/up due to certain activity, FortiGate will generate a log.

This type of activity may include the following:

1. A cable is disconnected/unplugged on a port.
2. The port is shut down/disabled on the peer devices.
   
   

The most common scenario for HA issue is number 2: the port is shut down/disabled on the peer unit.

STP has a 'hold-down' timer and 're-calculation' timer to evaluate the changes on the network.

When this hold-down timer has expired, STP will refresh or recalculate the network path.

Shut down port1, then reactivate port2.

Verify the changes on the switch too.

Example scenario:

• A first FortiGate unit acting as the primary unit.
  
• A second FortiGate unit acting as the secondary unit.
  
  

The network switch should pass all of the traffic to the first FortiGate unit's side. When intermittence occurs, the network is most likely sending the traffic to the secondary FortiGate, which is not correct.

In these circumstances, the secondary unit will not process any traffic.

Refer to the 'Log & Report' mentioned previously.

For this kind of scenario, the issue is not on the FortiGate configuration itself.

However, network solutions and integrations will be required.

Consult the network administrator or Fortinet's professional services to assist further with this case.

Conclusion:

HA deployment requires proper physical and logical design on the network level.

Related article:

Technical Tip: High availability basic deployment design