Technical Tip: How to analyze or plan VXLAN network deployment to avoid possible MTU issue

[fwilliams]

Description	This article describes how to plan or analyze VXLAN network so it is possible to determine if there is the possibility of running into MTU issues in the future or if having one already.
Scope	FortiOS.
Solution	There are a lot of cases where VXLAN is implemented without prior evaluation of the network/underlay on which it is configured and when the users begin pushing traffic over the VXLAN, they are faced with MTU-related issues (bigger MTU than allowed) which leads to frustrating users' experience. Facts relevant or useful in this article: VXLAN is an encapsulation protocol, which means it has and adds its own overhead which increases packet size (closing up on MTU available window). VXLAN uses unicast UDP packet over port 4789 (this is the default value and can be changed on the FortiGate). VXLAN will increase the frame size by 50 bytes (8 bytes for the VXLAN header; 8 bytes for the outer UDP header; 20 bytes for the outer IPv4 header; 14 bytes for the Ethernet header). MTU is the maximum transmission unit – the maximum size of data that can be sent as a single packet without being fragmented (it is usually 1500 bytes on most Ethernet interfaces - as this is the IEEE 802.3 standard for Ethernet frames). If 1500 bytes frame is captured with Wireshark, the size in Wireshark is either 1514 bytes (Ethernet frame without VLAN tag) or 1518 bytes (with VLAN tagged). ICMP header is 8 bytes. VLAN Tag header (802.1q) is 4 bytes. IPSec header is 50 – 57 bytes (verify the exact bytes used by the IPSec deployed if implementing VXLAN over IPSec). When transmitting, the DATA is encapsulated (i.e. header(s) are added and the packet increases in size, this size needs to be maintained until it gets to the other VTEP device where it is de-encapsulated). When receiving, the DATA is de-encapsulated (i.e. header(s) are removed leaving the packet in its original form and size). Uses cases: VXLAN over layer 3 network (or internet). If the underlay has an MTU of 1500, then the maximum packet size that can be sent over the VXLAN is 1500 – 50 (1450 bytes). Note that this 1450 bytes is not raw data size, some headers contributed to the 1450: like 20 bytes IPv4 header + 8 bytes ICMP header (if using PING to test for example). Max raw data = 1422 bytes. VXLAN over IPSec network If the underlay has an MTU of 1500, and the VXLAN over IPSec is deployed then the maximum packet size that can be sent over the VXLAN in this case is 1500 –  100 to 107 (1393 - 1400 bytes). Note that this 1393 - 1400 bytes is not raw data size, some headers contributed to the 1393 - 1400: like 20 bytes IPv4 header + 8 bytes ICMP header (if using PING to test for example). Max raw data = 1365 - 1372 bytes. If using Wireshark to capture these packets, it is possible to see additional 14 bytes added, which is the Ethernet header (this does not count in MTU calculation, and it is necessary to concentrate on the ones that count only). After this analysis, it will be possible to know how to send safely over the VXLAN, if the VXLAN is facing an MTU issue or will likely face an MTU issue in the future if deployed. The trick is to ensure the packet size after encapsulation (with VXLAN) did not or will not exceed 1500 bytes or the underlay MTU. If planning to deploy VXLAN, do some checks like knowing what MTU the ISP support. If the application or intended application is strict or not with MTU size. Fix: If already deploying the VXLAN and facing an MTU issue, this is what it is possible to do. Note: Fragmentation is widely seen as a way to resolve large MTU issues, but the case is different with VXLAN as it is strict or does not work if frag. If thinking of fragmenting the VXLAN packets on the VTEPs, do not do it.  VXLAN rfc7348 warned about the use of fragmentation on VXLAN packets. See details below: Implement PMTU if possible. Lower the server MTU size.