VXLAN over IPSec MTU Problems

[jm-barreto]

Greetings

Im having some problems with my VXLAN over IPSec implementation. Im able to establish connection to the remote site. Telnet, SSH, RDP, VOIP is working fine but Outlook and some HTTP or HTTPS application don't work. I have read many article about this issue and all says that is a MTU or fragmentation issue. But I follow all the recommendation and nothing seems to work. 

First thing I notice is that VPN interface, Software-switch and vxlan mtu were set to 1370. I manage to bring the VPN and vxlan mtu to 9000 and Software-switch to 1500. My physical interface are all set to max mtu (9216). I also disable the honor-df bit but the maximum mtu that i can pass without fragmentation is 1472. And I think that is fine because 1472 + 28(header overhead) = 1500. But still cant get Outlook to work. I also adjust the mss in the policy to 1432 (1472-40). Also I lower my encryption to 3DES SHA1.

My main FW is a 100F and the remote is a 60F. Im runnig 7.2.4.

I will appreciate any information that you can provide

[Anthony_E]

Hello JBC,

Thank you for using the Community Forum. I will seek to get you an answer or help. We will reply to this thread with an update as soon as possible.

Thanks,

[ebilcari · Accepted solution]

Basically the most important MTU value is the one of the physical link between the two nodes that are doing IPSEC. In the best scenario if that link support more than 1600 bytes you can use the standard 1500 for the encapsulated data. That is not easily achievable in the standard internet. https://docs.fortinet.com/document/fortigate/7.4.0/administration-guide/38079/vxlan

If you want to make it work quickly you may decrease the MTU directly on the server or lower the MTU on the interface that is the GW for the server: https://docs.fortinet.com/document/fortigate/7.4.0/administration-guide/596096/interface-mtu-packet-...
and make sure that Path MTU Discovery (PMTUD) can work

[jm-barreto]

Thanks for the help @Anthony_E @ebilcari 

I was able to fix this adjusting the mss on the firewall policy.

[jpozo86]

Hello @jm-barreto;

I have the same problem. What mss value did you configure in the firewall policies ?

JCPV

[jm-barreto]

Hi 

For my scenario the mss value was 1303 and i apply it on receive ant transmit in the firewall policy 

[Adolfo69]

In the version 7.4.7, when you create the vxlan interfaces, these take the MTU of the interface associated, so, if you associate the vxlan to one VPN interface, be careful, because the VPN by default has a MTU of 1480 bytes. You should increase the MTU to 1500 in the VPN interfaces, delete all vxlan interfaces and create again, to take the new MTU. The formula to MTU vxlan is: MTU vxlan = MTU Interface associate - 50 bytes.

https://community.fortinet.com/t5/FortiGate/Technical-Tip-Understanding-Maximum-MTU-on-VXLAN-Interfa...