VXLAN over IPSEC - ARP Table Issue on Inter-VLAN Communication

[5q46n2te8jPWJY]

Hello,

I’ve set up two sites connected via VXLAN over IPSEC, and everything is functioning as expected.

However, I’ve noticed an issue with ARP behavior under specific conditions:

• From Site A, when I connect from another VLAN (e.g., VLAN 30) to a virtual machine in VLAN 10 or VLAN 20 on Site B, I observe a change in the ARP table on the device in Site B.

Example:

• I check the ARP table of PC B20 (a device in VLAN 20 on Site B).
  • The MAC address for 10.112.20.254 (router’s IP) initially shows 00:09:0f:09:00:00 (MAC address of the FortiGate on Site B).
• When I connect from PC A30 (a device in VLAN 30 on Site A) to PC B20, and I re-check the ARP table on PC B20, the MAC address for 10.112.20.254 changes to 00:09:0f:09:02:00 (MAC address of the FortiGate on Site A).

This unexpected behavior raises concerns about network stability and could impact communication.

Has anyone encountered a similar issue, or does anyone have insights on why the ARP entry changes in this way? Could this be related to VXLAN or routing settings?

Thanks in advance for your help!

[tachen]

Happy to see you again since from your last similar topic.

However since you have duplicated gateway IP address in same broadcast domain bridged by a single VXLAN instance, an uncontrolled ARP flooding without addition control plane helping will cause the collision and flapping.

I got what you want and I believe what you need is distributed anycast gateway and IRB (Integrated Routing and Bridging) of EVPN which FortiOS 7.4.5 not support yet.

[5q46n2te8jPWJY]

Thanks,

Can you tell me more about that ? Do you have documentation ? Wich version of FortiOS support it ?

[tachen]

No current FortiOS version support this time, so there's no official documentation. Confirm supported RFCs and MP-BGP EVPN features: 

https://docs.fortinet.com/document/fortigate/7.6.0/supported-rfcs/939093/supported-rfcs 

https://docs.fortinet.com/document/fortigate/7.4.5/administration-guide/52499/vxlan-with-mp-bgp-evpn 

As you haven't implemented MP-BGP EVPN control plane for your VXLAN network, a basic knowledge of VXLAN with MP-BGP EVPN is needed and can be acquired via the second URL above, though this is not enough.

For a further regarding anycast gateway and IRB, I advise that Google it and seeking for documents from corresponding vendors such as Cisco.

[5q46n2te8jPWJY]

Thank you, is there a way to achieve my configuration as I want it?

[tachen]

Unfortunately I have no idea since not only because of feature limitations but this topology often leads an asymmetric routing which breaks stateful firewalling (stateful packet inspection and policy evaluation performed by FortiGate). So a simple VXLAN fabric with stateful firewall integrated usually combines physical firewalls using session-synced HA like FGCP/FGSP or definitely primary/backup like VRRP, and offloads VTEP function and optional anycast gateway function to switches. Want inter-VLAN traffic control instead of directly switched by L3 switches while utilizing anycast gateway? Use VRF on switches and leak wanted E-W traffic to FortiGate.