Technical Tip: TCP MSS handling changes for IPsec tunnels beginning in FortiOS v7.6.1

Description

This article describes the TCP Maximum Segment Size (TCP MSS) handling changes introduced for IPsec tunnels beginning in FortiOS 7.6.1 and their impact on Path MTU Discovery (PMTUD).

Scope

 FortiOS v7.6.1 through FortiOS v8.0.0.

Solution

Beginning with FortiOS v7.6.1, FortiGate no longer automatically adjusts (clamps) the TCP MSS for traffic traversing an IPsec tunnel.

In releases earlier than FortiOS v7.6.1, when packets exceeded the effective IPsec tunnel MTU, FortiGate automatically adjusted the TCP MSS and generated ICMP 'Fragmentation Needed' messages. This behavior helped prevent oversized TCP packets from being dropped.

Beginning with FortiOS v7.6.1, FortiGate relies on Path MTU Discovery (PMTUD). When an oversized packet is detected, FortiGate generates an ICMP 'Fragmentation Needed' message, allowing the sender to discover the path MTU and reduce the TCP MSS accordingly.

As a result, successful transmission of oversized TCP packets depends on the successful operation of PMTUD and the delivery of ICMP 'Fragmentation Needed' messages.

On platforms that use NP7, NP6, NP6XLITE, and NP7XLITE network processors, IPsec traffic that is offloaded to the NPU cannot generate ICMP 'Fragmentation Needed' messages for oversized encapsulated packets.

For deployments running FortiOS v7.6.1 through FortiOS v8.0.0, the following workarounds are recommended:

1. Configure a manual TCP MSS value in the affected firewall policy to ensure packets remain below the effective IPsec tunnel MTU. This eliminates the requirement of disabling the NPU offloading on the tunnel. Technical Tip: Setting TCP MSS value.

2. Disable NPU offloading for the affected IPsec tunnel. This allows the FortiGate CPU to process the traffic and generate ICMP 'Fragmentation Needed' messages required for PMTUD: Disabling NP offloading for individual IPsec VPN phase 1s.

Disabling NPU offloading may increase CPU utilization and reduce IPsec VPN throughput.

Alternatively, when set ip-fragmentation pre-encapsulation is enabled, packets are fragmented before IPsec encapsulation. NPU offloading must be disabled on the VPN tunnel interface for this configuration. This allows fragmented packets to be encapsulated and transmitted as ESP packets, preventing packet loss in environments where the NPU cannot generate ICMP 'Fragmentation Needed' messages for oversized encapsulated packets. Fragmenting IP packets before IPsec encapsulation.

Beginning with FortiOS v8.0.1, the TCP MSS handling behavior has been reverted. A fix for FortiOS v7.6.x branch is planned for a future FortiOS v7.6.x release.