Description
This article describes how to set the TCP MSS value. The Maximum Segment Size (MSS) is a parameter in the OPTIONS field of the TCP header that states the largest amount of payload (in bytes) that a communication device can handle in a single, unfragmented TCP segment.
For optimum communication, the number of bytes in the data segment and the header must add up to less than the number of bytes in the Maximum Transmission Unit (MTU).
By default it will be 40 bytes less than MTU. So if a PC's MTU has by default 1500 bytes it will send a MSS of 1460 bytes.
If the communication network has a lower MTU value, but the client PC is not aware of it, it will send its MSS value of 1460 bytes to the server. The server will therefore think that the client can receive 1500 bytes (1460 MSS layer4 +20 ip header +20 TCP header) and will send a packet with a size of 1500 bytes. Now if the MTU is lower somewhere in the path, then the packet can be fragmented. If the DF (don't fragment) bit is set then the packet can be dropped, which can cause delays or slowness in the network.
Scope
FortiGate.
Solution
To avoid such scenarios, the TCP MSS value can be changed. By doing this, the firewall will modify the TCP MSS sent by client/server in the TCP syn/syn-ack packets so the remote end receives a smaller MSS and sends smaller packets.
To change the tcp-mss on FortiGate:
config firewall policy
edit <policy id>
set tcp-mss-sender <mss value>
set tcp-mss-receiver <mss value>
tcp-mss-sender: value of the sender's TCP MSS, will modify the TCP MSS field in the TCP syn packet.
tcp-mss-receiver: value of the receiver's TCP MSS, will modify the TCP MSS field in the TCP syn packet.
Note:
If the firewall receives a packet with a segment size that is lower than the one configured in the policy, it will not modify the TCP MSS field. For example, if the firewall policy is configured to set MSS of 1440 and the packet arrives with MSS 1200, the value will not be modified and the packet will be forwarded with MSS 1200.
For some systems, the TCP MSS on the FortiGate could go as low as 1150 for them to function properly. One such example is the Cisco Phone, model: CP-XXX.
The communication affected is the one used during the phone provisioning process, where phones are supposed to contact their internet-based provisioning server to download their config, this process is via https.
When NGFW in settings under system is set to Policy-Based:
TCP-MSS can be adjusted in the 'SSL Inspection & Authentication' policy (Under policy and objects) and not in Security policies.
Create a 'SSL Inspection & Authentication' policy (From GUI under policy & Objects) or From CLI, create (Or edit) using the command 'Config firewall policy' and set the tcp-mss using the below commands :
config firewall policy
edit <policy id>
set tcp-mss-sender <mss value>
set tcp-mss-receiver <mss value>
THE PING TEST:
To find the optimum MSS/MTU value of the network, do the 'ping test' to find the MTU and then reduce 40 from that value and set that as tcp-mss in the firewall policy.
For MS Windows OS client enter:
ping -f -l 1472 <server ip>
For Apple MAC OS client enter:
ping -D -s <size> <server_ip>
For 1500 bytes MTU, 1500-28 (20 bytes=IP header and 8 bytes=ICMP header) = 1472
If the packet needs to be fragmented but the DF bit is set, the MTU is less than 1500 bytes. Suppose it is possible to ping with a size of 1400 but not with a size of 1401, this means that the MTU value is 1400+28=1428 bytes. The solution is either to fix the MTU in the network device or to set the TCP MSS in the firewall policy to 1428-40=1388 bytes.
Traffic passing over IPsec VPN tunnel:
There is an extra IPSec header. It depends on the encryption and authentication algorithm. The AES-256 and SHA1 have a maximum of 73 bytes, and another algorithm might have slightly different bytes.
For a 1500-byte MTU, the MSS for a TCP over IPSec tunnel is 1500-20(IP header)-20(TCP header)-73(IPSec header) = 1387.
For the IPSec tunnels, the MTU and TCP MSS can be configured per tunnel interface and take precedence over the settings defined by policies.
config system interface
edit "IPSecTunnelName"
set mtu-override enable
set mtu <MTU Value>
set tcp-mss <MSS Value>
Note that tcp-mss setting behavior has changed starting from FortiOS 7.0.1+ in which the least valued set for the traffic path is used for both directions of the traffic (more details available in this article)
Related articles:
MTU override of IPsec VPN interface - Fortinet Community
IP Packet fragmentation over IPSec tunnel... - Fortinet Community
Behavior of TCP-MSS setting under system ... - Fortinet Community
