Troubleshooting Tip: Duplicate ARP issue while connected with a transparent VDOM

[Stephen_G]

Description	This article describes how to avoid a duplicate ARP (Address Resolution Protocol) conflict observed in a NAT mode VDOM when connected with a transparent mode VDOM.
Scope	FortiGate, VDOMs in transparent mode.
Solution	When different VDOM modes are connected by vdom-links (especially when connected with a transparent VDOM), an issue with the ARP (Address Resolution Protocol) may be encountered: get sys arp | grep 40.100.X.1 45.100.X.1      1         76:d0:e4:e0:02:01 Int2Ext0    45.100.X.1      10        76:d0:e4:e0:02:01 L2-VLANLink0  Here, duplicate ARP entries are observed for the same IP address, 40.100.x.1, but associated with two different network interfaces Int2Ext0 and L2-VLANLink0. In an ethernet environment, switches and routers utilize MAC addresses for forwarding decisions. FortiGate maintains a routing and ARP table to determine the correct outgoing interface for each MAC address. In this case, two different VDOM link interfaces within a NAT mode VDOM share the same ARP entry. This conflict causes traffic passing through a transparent VDOM in a virtual server present in the NAT VDOM to fluctuate. When an ARP request is made, the FortiGate device learns that the IP address 40.100.X.1 is associated with the MAC address 76:d0:e4:e0:02:01. This mapping is accessible through the network interface Int2Ext0. However, while connected with a transparent VDOM, the ARP table fills up with duplicate ARP entries from the two different network interfaces, Int2Ext0 and L2-VLANLink0: One with the Int2Ext0 (40.100.X.X/24) network. Another with the L2-VLANLink0 (10.10.100.X/24) network. get sys arp | grep 40.100.X.1 40.100.X.1      1         76:d0:e4:e0:02:01 Int2Ext0    40.100.X.1      10        76:d0:e4:e0:02:01 L2-VLANLink0  diagnose ip arp  list | grep 45.115.X.1 index=551 ifname=L2-VLANLink0 40.100.X.1  76:d0:e4:e0:02:01 state=00000004 use=2085 confirm=8085 update=6 ref=0 index=516 ifname=Int2Ext0 40.100.X.1 76:d0:e4:e0:02:01 state=0000004 use=0 confirm=663253 update=7 ref=416 This duplicate ARP issue observed in two VDOM links is due to the default behavior of a transparent mode VDOM. In a transparent VDOM, ARP packets arriving on one interface are sent to all other interfaces, including VLAN sub-interfaces. As a result, all of the interfaces belong to the same broadcast domain regardless of their VLAN ID.  In this case, without forward domain configured (which is the default option), all of the interfaces become a part of the same broadcast domain. As a result, ARP packets for the VDOM link (EXT-LINK0) were propagated from all other VDOM link interfaces, including the VLAN interface. This is why duplicate entries in the ARP table were observed from two different network interfaces.  This issue causes FortiGate to select the wrong path for sending data, leading to routing confusion and resulting in data taking inefficient or incorrect routes through the network. In this case, it resulted in an inaccessible VIP server in the NAT VDOM when traffic passes through the transparent VDOM. To overcome this default behavior issue with a transparent VDOM, configure the interfaces with the same Forward Domain ID: config system interface     edit "ISP-VLAN-Link0"         set vdom "Transparent_VDOM"         set allowaccess ping         set forward-domain 10         set snmp-index 29         set interface "wan"         set vlanid 100     next end config system interface     edit "L2-VLANLink1"         set vdom "Transparent_VDOM"         set type vdom-link         set forward-domain 10         set snmp-index 28         set macaddr 7e:ec:10:6b:00:2c     next end Forward domains are like broadcast domains. Traffic arriving on one interface is broadcast only to interfaces that are in the same forward domain ID.  As a result, two interfaces intended to be in the same broadcast domain need to be configured with the same forward domain. See Technical Tip: Forwarding domain in transparent mode for more information. After the forward domain ID is configured, the interfaces will stop propagating the same ARP packet connected to other interfaces and VDOM links that are not the part of the same broadcast domain. For more information, see the FortiGate documentation's 'forward domain solution' section.