Technical Tip: Fortinet-ESP Parameter remains enabled after changing transport mode, causing IPsec Traffic drops

[Dokoye]

Description	This article describes a known issue that causes FortiGate to continue encapsulating tunnel data traffic as fortinet-esp when transport mode is changed to UDP.
Scope	FortiGate v7.4.
Solution	Fortinet ESP encapsulation is a proprietary protocol intended for FortiGate-to-FortiGate IPsec tunnels, enabling ESP data traffic to be encapsulated in non-standard TCP headers. config vpn ipsec phase1-interface     edit "Tunnel1"         set fortinet-esp { enable | disable }     next end This protocol is not compatible with FortiClient or third-party VPN gateways. There is a reported CLI issue in FortiOS v7.4 firmware versions: if the phase1-interface configuration is modified to use UDP transport, fortinet-esp remains enabled but is hidden in CLI and no longer configurable. FGT-A # config vpn ipsec phase1-interface FGT-A (phase1-interface) # edit Tunnel1 FGT-A (Tunnel1) # set transport udp FGT-A (Tunnel1) # set fortinet-esp disable command parse error before 'fortinet-esp' Command fail. Return code -61 Some FortiClient versions and third-party VPN clients do support sending and receiving ESP traffic encapsulated in TCP, but this does not use the proprietary TCP encapsulation protocol. See Encapsulate ESP packets within TCP headers. In the example below, a PC connects to the dial-up IPsec VPN gateway using FortiClient, with the FortiGate acting as the dial-up server.   Example Topology:  The FortiClient is able to connect to VPN, but the endpoint does not receive any data traffic. A packet capture is taken on the FortiGate shows TCP headers are still being applied. The 'diagnose vpn tunnel list' command in FortiOS CLI shows received and transmitted packets. diagnose vpn tunnel list name Tunnel1 | grep 'pkts\|Tunnel1' <-- grep filter displays lines containing the strings 'pkts' or 'Tunnel1'. name=Tunnel1 ver=2 serial=b 172.17.97.91:4500->172.17.97.92:59018 parent=Tunnel1 index=0 proxyid=Tunnel1 proto=0 sa=1 ref=3 serial=1 add-route   dec:pkts/bytes=139/22066, enc:pkts/bytes=30/2880 Resolution: This issue is resolved in FortiOS v7.6.0 and later. In these versions fortinet-esp is correctly automatically disabled when UDP transport is configured. Workaround: On affected FortiOS v7.4 tunnels, configure the tunnel transport mode as TCP to allow disabling fortinet-esp, then change the transport mode back to udp. config vpn ipsec phase1-interface     edit "Tunnel1"         set transport tcp         set fortinet-esp disable         set transport udp     next end Related articles: Technical Tip: How to use TCP as transport for IKE/IPsec traffic Troubleshooting Tip: Encapsulating ESP packets within TCP headers Fails After Upgrade to FortiOS 7.4...