Technical Tip: FortiGate SD-WAN Packet Loss on Layer-2 Bridging underlay architecture

[stroia]

Description	This article describes the case when the packet loss is observed in a FortiGate SD-WAN massive deployment, done with FortiGate clusters and as underlay a 'Layer 2 Bridging network'. It is important to note that there is not always a packet loss issue in a network environment as described in this article.
Scope	Consider this network scheme: Packet loss scenario characteristics: Fortinet SD-WAN deployment done with the same model FortiGate clusters into different customer sites. The same WAN port and HA group (ID of FGCP Fortinet High Availability protocol) are used on different clusters. Layer 2 bridging architecture is shown in the network scheme and explained below. Layer 2 bridging architecture description: ISP (Internet Service Provider) provides customers with an MPLS connection. ISP device installed on each customer site acts like a Layer 1 Hub, forwarding frames between all active ports without any manipulation. Layer 2 routing is performed directly by ISP PE (Point of Entrance performed by a Backbone Router). The second IP address of the Point-to-Point layer 3 networks configured on the FortiGates cluster WAN port, is configured on PE. Pinging for example an IP configured on a loopback interface on Hubs, there is partial packet loss, either by sending ICMP echo packets encapsulated on an IPSec tunnel or sending it via underlay without encapsulation. There is no packet loss if the traffic is generated from a different Layer 3 device in substitution of the FortiGates cluster. ISP PE receives all customer traffic, divided per VLAN, with a different VLAN assigned to each customer site traffic. The same virtual MAC address is generated from different FortiGate clusters if conditions A, B, and C listed above are matched, so if a continuous ping is active at least on 2 FortiGate clusters into different sites, PE will receive contemporary multiple frames with the same source mac address from different VLANS. An incorrect per VLAN frame routing performed by PE causes partial packet loss. This issue is not caused by FortiGate devices, but there is some workaround available on the FortiGate side explained in the Solution section. Details about how virtual MAC address is calculated from the FortiGate are here: Cluster virtual MAC addresses.
Solution	To avoid this issue, it is possible to obtain, on different FortiGates clusters WAN ports, different Virtual Mac addresses: Use different FGCP group IDs: Via GUI:   Via CLI: config system ha     (ha) # set group-id X It is possible to enter a value from <0> to <1023>. Use different WAN ports on different sites.