FortiGate
FortiGate Next Generation Firewall utilizes purpose-built security processors and threat intelligence security services from FortiGuard labs to deliver top-rated protection and high performance, including encrypted traffic.
Not applicable
Article Id 190383
Article
Description

A network may experience packet loss when two FortiGate HA clusters have been deployed in the same broadcast domain. Deploying two HA clusters in the same broadcast domain can result in packet loss because of MAC address conflicts. The packet loss can be diagnosed by pinging from one cluster to the other or by pinging both of the clusters from a device within the broadcast domain. You can resolve the MAC address conflict by changing the HA Group ID configuration of the two clusters. The HA Group ID is sometimes also called the Cluster ID.

This article describes a topology that can result in packet loss, how to determine if packets are being lost, and how to correct the problem by changing the HA Group ID.

Note: Packet loss on a network can also be caused by IP address conflicts. Finding and fixing IP address conflicts can be difficult. However, if you are experiencing packet loss and your network contains two FortiGate HA clusters you can use the information in this article to eliminate one possible source of packet loss.

See FortiOS v3.0 HA Cluster virtual MAC addresses for more information about the HA virtual MAC address and about the HA Group ID.

Components
  • All FortiGate units running FortiOS v3.0
Example Topology

The topology below shows a FortiGate-60B cluster and a FortiGate-300A cluster. The FortiGate-60B internal interfaces and the FortiGate-300A port 1 interfaces are both connected to the same broadcast domain. In this topology the broadcast domain could be an internal network. Both clusters could also be connected to the Internet or to different networks.

Two HA clusters in the same broadcast domain

Steps or Commands

Ping testing for packet loss

If the network is experiencing packet loss, it is possible that you will not notice a problem unless you are constantly pinging both FortiGate HA clusters. During normal operation of the network you also might not notice packet loss because the loss rate may not be severe enough to timeout TCP sessions. Also many common types if TCP traffic, such as web browsing, may not be greatly affected by packet loss. However, packet loss can have a significant effect on real time protocols that deliver audio and video data.

To test for packet loss you can set up two constant ping sessions, one to each cluster. If packet loss is occurring the two ping sessions should show alternating replies and timeouts from each cluster.

FortiGate-60B   FortiGate-300A
Cluster         Cluster

reply             timeout
reply             timeout
reply             timeout
timeout           reply
timeout           reply
reply             timeout
reply             timeout
timeout           reply
timeout           reply
timeout           reply
timeout           reply

Displaying the virtual MAC address

When an HA cluster starts up, the FortiGate Clustering Protocol (FGCP) assigns a virtual MAC address to all cluster interfaces. The MAC addresses is the same for every interface on the cluster. In fact all FortiGate models assign the same virtual MAC addresses to all interfaces.

A FortiGate unit interface has two MAC addresses: the current hardware address and the permanent hardware address. The permenant hardware address cannot be changed, it is the actual MAC address of the interface hardware. The current hardware address can be changed. The current hardware address is the address seen by the network. You can change the current hardware address using the macaddr keyword of the config system interface CLI command. The current hardware address is also changed to the HA virtual MAC address by the FGCP.

You can use the command diagnose hardware deviceinfo nic to display two MAC addresses for any FortiGate interface. The command displays the currrent hadware address as Current_HWaddr and the permanant hardware address as Permanent_HWaddr.

Before HA configuration the current hardware address is the same as the permanent. The following command displays the current and permanant hardware addresses for the internal interface of a standalone FortiGate-60B unit:

FGT60B3907503171 # diagnose hardware deviceinfo nic internal
Description             MARVEL Ethernet driver
vlanid                  1024

System_Device_Name      internal
Current_HWaddr          02:09:0f:78:18:c9
Permanent_HWaddr        02:09:0f:78:18:c9
State                   up
Link                    down

During HA operation the current hardware address becomes the virtual MAC address. The following command displays the current and permanant hardware addresses for the internal interface of a FortiGate-60B unit operating in HA mode:

FGT60B3907503171 # diagnose hardware deviceinfo nic internal
Description             MARVEL Ethernet driver
vlanid                  1024

System_Device_Name      internal
Current_HWaddr          00:09:0f:09:00:02
Permanent_HWaddr        02:09:0f:78:18:c9
State                   up
Link                    down

If two HA clusters with the same virtual MAC address are connected to the same broadcast domain (L2 switch/hub), the MAC address will conflict and bounced between the two clusters. This example Cisco switch MAC address table shows the MAC address flapping between different interfaces (1/0/1 and 1/0/4).

 1    0009.0f09.0002    DYNAMIC     Gi1/0/1
 1    0009.0f09.0002    DYNAMIC     Gi1/0/4

Since the same virtual MAC address is used for all FortiGate models, MAC address conflicts can occur if the two cluster's in the same broadcast domain contain different FortiGate models.

The following comment output shows that the virtual MAC addresses for a FortiGate-60 internal interface and a FortiGate-300A port 1 interface have the same virtual MAC address. These interfaces could be used as interfaces within the same broadcast domain.

Display FortiGate-60B internal interface Information.

FGT60B3907503171 # diagnose hardware deviceinfo nic internal
Description             MARVEL Ethernet driver
vlanid                  1024

System_Device_Name      internal
Current_HWaddr          00:09:0f:09:00:02
Permanent_HWaddr        02:09:0f:78:18:c9
State                   up
Link                    down

Display FortiGate-300A port 1 internal interface Information.

FG300A2904500238 # diagnose hardware deviceinfo nic port1
Description               Intel(R) PRO/100 M Desktop Adapter
Driver_Name               e100
Driver_Version            2.1.29
PCI_Vendor                0x8086
PCI_Device_ID             0x1229
PCI_Subsystem_Vendor      0x8086
PCI_Subsystem_ID          0x0070
PCI_Revision_ID           0x0010
PCI_Bus                   3
PCI_Slot                  5
IRQ                       18
System_Device_Name        port1
Current_HWaddr            00:09:0F:09:00:02
Permanent_HWaddr          00:09:0F:85:40:FD

Avoiding MAC address conflicts

To avoid HA MAC address conflicts, set different HA group IDs for each cluster within the same broadcast domain. Changing the Group ID changes the virtual MAC address of a cluster. You can change the Group ID from the FortiGate CLI using the following command:

config system ha
    set group-id <id_integer>
end