Techinal Tip: Information about MSS needs to be set for TCP communications in the policy based on the interface involved in the communication

MTU (Maximum Transmission Unit) is the largest size of a data packet or frame that can be sent in a single network transaction by the network device.

MSS (Maximum Segment Size), on the other hand, is the largest amount of data that a device can receive in a single TCP segment. MSS is a value that is exchanged during the TCP handshake between two devices and directly relates to the MTU of the communicating devices. It is the MTU of the device minus the size of the IP and TCP headers. Setting the MSS correctly in the FortiGate helps prevent fragmentation, ensuring smoother and more efficient communication.

Most of the TCP clients and servers have an Ethernet MTU of 1500 bytes. This value includes the data payload and all the headers. The standard TCP segment size is 1460 bytes. The 40-byte difference is made up of a 20-byte IP header and a 20-byte TCP header.

During the TCP handshake, they exchanged their MSS information and sent across the data segment of the respective MSS.

However, if the FortiGate interface has a different MTU due to the type of egress interface of the packet, this causes the packet to get fragmented, and in case 'Don't Fragment' is sent in the packet, FortiGate may drop the packet and send an ICMP (type 3, code 4) message to the sender.

So it is important to set the sender and receiver MSS in the policy of FortiGate for the respective TCP communication to adopt the new MSS set in the policy.

Different network interfaces can add their own headers, which reduces the effective MTU and, consequently, the MSS. The following table provides a breakdown of how various interface types affect these values.

Note:

If higher key lengths are used, the ESP header length and the IPsec tunnel MTU are reduced. That is why the MSS of the communication needs to set the possible approximate value (≈).

This will reduce the chances of A-B communication being affected by MTUs or getting fragmented.

Also consider the below in case jumbo frames are enabled, which might affect the default values: Technical Tip: MTU size and Jumbo frames support on FortiGate devices.

A packet capture can be performed during the TCP handshake to verify the MSS value being exchanged.

The document below can be followed to change the TCP MSS value in the firewall policy:

Technical Tip: Setting TCP MSS value 

The document below can be followed to change the MTU for the IPsec tunnel interface:

Technical Tip: How to override MTU for IPsec VPN interfaces on the FortiGate