Description
This article describes how HA VLAN tagging works between FortiGate-6000/7000 series.
Solution
When HA is configured between FortiGate Chassis, there will be different internal packets exchanged across the link to form HA Cluster. These internal packets uses different VLAN tags to communicate with each other.
When FortiGate Chassis are connected through a switch or a dedicated WAN link (Layer 2) or over VXLAN, internal VLAN tags should be transparently transferred across the link. Sometimes few third-party switches in the path may intercept and strip inner VLAN tags, that would cause HA sync to fail and Chassis will not come in sync with each other.
To address these interoperability issues, a new feature was introduced in 6.2.4 to support the different modes of VLAN tagging on HA ports under system ha settings as shown below. This is an addition to the tagging selected by the system based on the type of internal traffic.
Using the 'ha-port-outer-tpid' it is also possible to customize outer TIPD to 0x8100, 0x9100, or 0x88a8. The default TPID is 0x8100.
Changing outer TPID settings should match on both switch and FortiGate side.
By default, VLAN-tagged packets carry a tag protocol identifier (TPID) of 0x8100 in most of the switches including FortiSwitch.
When switch receives HA traffic the packet format will be as shown below based on the dtag mode selected in config system ha settings
Proprietary mode.
TPID 0x8100 VLAN hbdev-vlan-id (by default 999) + TPID 0x88a8 VLAN 10/30 + TPID 0x8100 VLAN 10/30 + ethernet packet.
Double-tagging mode.
TPID 0x8100 VLAN hbdev-vlan-id (by default 999) + TPID 0x8100 VLAN 10/30 + ethernet packet
Here is a sample config from Arista with double tagging and its default TPID 0x8100
Example diagram: QNQ tunneling for HA traffic over Layer 2 Network.
- When HA is not formed between FortiGate cluster, run a packet capture on switch side and check for VLAN tags. Capturing packets on FotiGate side is not recommended if switch already stripped the VLAN tags, so port mirroring M ports(ingress and egress) on switch is needed to capture Dot1q packets.
- VXLAN is not supported on HA interfaces of 6000/7000 series.
- Native VLAN 1 cannot be used for HA communication.
- Use dedicated VLAN for HA communication.
This article describes how HA VLAN tagging works between FortiGate-6000/7000 series.
Solution
When HA is configured between FortiGate Chassis, there will be different internal packets exchanged across the link to form HA Cluster. These internal packets uses different VLAN tags to communicate with each other.
When FortiGate Chassis are connected through a switch or a dedicated WAN link (Layer 2) or over VXLAN, internal VLAN tags should be transparently transferred across the link. Sometimes few third-party switches in the path may intercept and strip inner VLAN tags, that would cause HA sync to fail and Chassis will not come in sync with each other.
To address these interoperability issues, a new feature was introduced in 6.2.4 to support the different modes of VLAN tagging on HA ports under system ha settings as shown below. This is an addition to the tagging selected by the system based on the type of internal traffic.
# config system haThe default 'ha-port-dtag-mode' mode is proprietary as it is compatible with the current and previous tagging implementation on 6000/7000 series. However, double-tagging mode is recommended to avoid any interoperability issues.
set ha-port-dtag-mode {proprietary | double-tagging)
set ha-port-outer-tpid {0x8100 | 0x9100 | 0x88a8}
end
Using the 'ha-port-outer-tpid' it is also possible to customize outer TIPD to 0x8100, 0x9100, or 0x88a8. The default TPID is 0x8100.
Changing outer TPID settings should match on both switch and FortiGate side.
By default, VLAN-tagged packets carry a tag protocol identifier (TPID) of 0x8100 in most of the switches including FortiSwitch.
When switch receives HA traffic the packet format will be as shown below based on the dtag mode selected in config system ha settings
Proprietary mode.
TPID 0x8100 VLAN hbdev-vlan-id (by default 999) + TPID 0x88a8 VLAN 10/30 + TPID 0x8100 VLAN 10/30 + ethernet packet.
Double-tagging mode.
TPID 0x8100 VLAN hbdev-vlan-id (by default 999) + TPID 0x8100 VLAN 10/30 + ethernet packet
Here is a sample config from Arista with double tagging and its default TPID 0x8100
Example diagram: QNQ tunneling for HA traffic over Layer 2 Network.
M1 & M2 are HA ports on 7000 series FortiGates, similarly HA1 & HA2 are HA ports on 6000 series FortiGates.
M1 ports.
M1 ports.
interface Ethernet37M2 ports.
description ** FGT-7000E M1 HA HB ***
speed forced 10000full
switchport access vlan 660
switchport trunk native vlan 4091
switchport mode dot1q-tunnel
!
interface Ethernet38
description *** FGT-7000E M1 HA HB ***
speed forced 10000full
switchport access vlan 660
switchport trunk native vlan 4091
switchport mode dot1q-tunnel
!
interface Ethernet39
description *** FGT-7000E M1 HA HB ***
speed forced 10000full
switchport access vlan 660
switchport trunk native vlan 4091
switchport mode dot1q-tunnel
!
interface Ethernet40
description *** FGT-7000E M1 HA HB ****
speed forced 10000full
switchport access vlan 660
switchport trunk native vlan 4091
switchport mode dot1q-tunnel
!
interface Ethernet41Troubleshooting.
description ** FGT-7000E M2 HA HB ***
speed forced 10000full
switchport access vlan 770
switchport trunk native vlan 4092
switchport mode dot1q-tunnel
!
interface Ethernet42
description *** FGT-7000E M2 HA HB ***
speed forced 10000full
switchport access vlan 770
switchport trunk native vlan 4092
switchport mode dot1q-tunnel
interface Ethernet43
description *** FGT-7000E M2 HA HB ***
speed forced 10000full
switchport access vlan 770
switchport trunk native vlan 4092
switchport mode dot1q-tunnel
!
interface Ethernet44
description *** FGT-7000E M2 HA HB ****
speed forced 10000full
switchport access vlan 770
switchport trunk native vlan 4092
switchport mode dot1q-tunnel
!
- When HA is not formed between FortiGate cluster, run a packet capture on switch side and check for VLAN tags. Capturing packets on FotiGate side is not recommended if switch already stripped the VLAN tags, so port mirroring M ports(ingress and egress) on switch is needed to capture Dot1q packets.
- Do a ping test between 'base-mgmt' IP address of both units.
- It is possible to find the 'base-mgmt' IP address of each slot on both chassis in mgmt-vdom
For example it is possible to ping between:
For example it is possible to ping between:
10.101.10.x or 10.101.10.10x addresses <----- X: slot number of blades.
edit mgmt-vdomBelow snapshots from Wireshark shows the packet format for dtag modes Double-tagging and proprietary.
# diagnose ip address list | grep "SN\|10.101.10"
# execute enter vsys_ha
# execute-ping 10.101.10.103
Double-Tagging: Ethertype 0x8100 (2x 802.1q) - outer VLAN 999, inner VLAN 30.
Proprietary: Ethertype 0x8100/0x88a8 (q-in-q) - outer VLAN 999, inner VLAN 30/10.
Note.
- VXLAN is not supported on HA interfaces of 6000/7000 series.
- Native VLAN 1 cannot be used for HA communication.
- Use dedicated VLAN for HA communication.
