Technical Tip: Resolving VXLAN over IPsec UDP packet drops
| Description | This article describes how to resolve UDP packet drops for VXLAN traffic traversing an IPsec tunnel, which cannot rely on TCP MSS adjustments to avoid fragmentation. |
| Scope | FortiGate, VXLAN, IPsec, UDP. |
| Solution | VXLAN encapsulates Layer 2 frames in UDP packets, introducing a 50-byte overhead for IPv4 (or 54 bytes with VLAN tagging). When this VXLAN traffic is routed through an IPsec tunnel, the additional ESP overhead often causes the total packet size to exceed the standard Internet MTU (typically 1500 bytes or lower).
In this diagram, the original Ethernet frame is encapsulated with outer headers (Ethernet, IP, and UDP) for tunneling across the network, while the inner Ethernet/IP header and payload remain unchanged from the original frame. This extra encapsulation layer used for tunneling is what differentiates VXLAN from the traditional VLAN headers defined in the 802.1Q standard. Unlike TCP, which can utilize Maximum Segment Size (MSS) adjustments to dynamically prevent IP fragmentation, UDP traffic lacks this mechanism. Therefore, oversized VXLAN packets must be fragmented to cross the IPsec tunnel.
By default, FortiOS sets the ip-fragmentation behavior on IPsec tunnels to post-encapsulation. In this mode, the FortiGate first encrypts the entire oversized VXLAN packet, then fragments the resulting ESP packet to fit within the exit interface's MTU.
The issue arises because many intermediate internet routers and firewalls drop fragmented ESP packets due to security policies or hardware processing limitations. This leads to silent drops and tunnel instability for the VXLAN traffic.
To ensure reliable delivery of UDP VXLAN traffic over IPsec, the tunnel can be configured for pre-encapsulation.
With the pre-encapsulation method, the FortiGate evaluates the packet size against the tunnel interface MTU. It fragments the inner VXLAN/UDP packet before encrypting it. Each resulting fragment is then encrypted and encapsulated with its own independent ESP header.
Consequently, intermediate routers only inspect fully formed, unfragmented ESP packets that fall within the allowed MTU limits. This hides the packet's fragmented state from the transit network and prevents intermediate drops.
To apply the change, modify the phase 1 configuration of IPsec tunnels via CLI.
Step 1: Configure pre-encapsulation on the IPsec tunnel
config vpn ipsec phase1-interface
Step 2: Verify the active fragmentation method using the following diagnostic command. Look for encap=none/40 to confirm pre-encapsulation is active. diagnose vpn tunnel list name <IPsec_tunnel_name> Important notes on Hardware Acceleration:
|
