Technical Tip: Phase-1 ip-fragmentation post and pre-encapsulation settings and their relation to firewall settings

[FortiArt]

Description	This article describes the effects of the IPSec vpn phase-1 settings ip-fragmentation post-/pre-encapsulation on the MTU of a source PC when a 'don’t fragment' (DF) bit is set. Their relationship with the firewall policy settings tcp-mss-sender/tcp-mss-receiver is also described.
Scope	All supported FortiGate models.
Solution	Introduction. The following is an explanation of the default settings of the IPSec VPN phase-1 and firewall policies affecting the tunnel’s MTU, and therefore the source PC MTU, when changed from their default values. Under tunnel’s phase-1: When 'set ip-fragmentation enable' is enabled: set ip-fragmentation post-encapsulation <- Will fragment packets inside the tunnel (default). set ip-fragmentation pre-encapsulation < - Will fragment packets outside the tunnel. Under the tunnel’s firewall policy: set tcp-mss-sender 0 <- Adjust TCP’s maximum segment size (MSS) for the sender of traffic (default). set tcp-mss-receiver 0 <- Adjust TCP’s MSS for the receiver of traffic (default). Please note: MSS = MTU - 40. Please note that in this article and all of the following scenarios, it is assumed that the source PC has the DF bit set (i.e. DF = 1). In addition, the traffic passing through the tunnel uses the TCP protocol. With all settings at their default values, the tunnel’s MTU as per default post-encapsulation setting without fragmentation is equal to 1410 bytes. This mean the source PC can transmit data of 1370 bytes, which is 1410 bytes minus the 20 bytes from the TCP header and the 20 bytes from the IP header. Before Frag 1410 = 1370 (PC's Data) + 40 (TCP/IP) Headers  Need Frag 1411 = 1371 (PC's Data) + 40 (TCP/IP) Headers With all settings in their default values except for set ip-fragmentation pre-encapsulation, the tunnel’s MTU as per pre-encapsulation setting without fragmentation is equal to 1392 bytes. This mean the source PC can transmit data of up to 1352 bytes, which is equal to 1392 minus the 20 bytes from the TCP header and the 20 bytes from the IP header. Before Frag 1392 = 1352 (PC's Data) + 40 (TCP/IP) Headers Need Frag 1393 = 1353 (PC's Data) + 40 (TCP/IP) Headers In conclusion, the pre-encapsulation reduces the tunnel's MTU by 18 bytes because of the extra bytes needed to process fragmentation before the TCP traffic entering the tunnel. Scenarios to reflect the relationship of tcp-mss-sender/-receiver settings on post-/pre-encapsulation: Scenario 1: post-encapsulation: tcp-mss-sender/-receiver = 1500 -> The tunnel’s MTU without fragmentation is equal to 1410 bytes. This means the source PC can transmit data of 1370 bytes, which is equal to 1410 bytes minus the 20 bytes from the TCP header and the 20 bytes from the IP header. tcp-mss-sender/-receiver = 1000 -> The tunnel’s MTU without fragmentation is equal to 1410 bytes. This means the source PC can transmit data of 1370 bytes, which is equal to 1410 bytes minus the 20 bytes from the TCP header and the 20 bytes from the IP header. Scenario 2: pre-encapsulation: tcp-mss-sender/-receiver = 1500 -> The tunnel’s MTU without fragmentation is equal to 1392 bytes. This means the source PC can transmit data of 1352 bytes, which is equal to 1392 bytes minus the 20 bytes from the TCP header and the 20 bytes from the IP header. tcp-mss-sender/-receiver = 1000 -> The tunnel’s MTU without fragmentation is equal to 1392 bytes. This means the source PC can transmit data of 1352 bytes, which is equal to 1392 bytes minus the 20 bytes from the TCP header and the 20 bytes from the IP header. In conclusion, changing the values of tcp-mss-sender/-receiver on the relevant firewall policies of the tunnel dominated by the post-/pre-encapsulation settings means post-/pre-encapsulation settings accordingly define the maximum MTU of the source PC with a DF bit set.