Technical Tip: IPsec VPN over TCP using FortiClient not establishing, with error message 'wrong transport, phase 1 uses non udp' in the FortiGate's IKE debug

In the IKE debugs in the FortiGate (diagnose debug application ike -1), it is possible to see the following error messages:

The message 'wrong transport, phase 1 uses non udp' indicates the FortiGate does support TCP transport, but the tunnel is configured to use UDP transport only. See the document Encapsulate ESP packets within TCP headers for a description of the TCP transport feature and configuration examples.

To resolve the issue, enable TCP transport in the FortiGate phase1-interface configuration:

1. Change the transport type to TCP:

config vpn ipsec phase1-interface
    edit "VPN_FCT"

        set ike-version 2

        set transport tcp

    next

end

2. Ensure 'fortinet-esp' is disabled.

config vpn ipsec phase1-interface
    edit "VPN_FCT"

        set fortinet-esp disable <----- Default setting. 'fortinet-esp' must remains disabled, as FortiClient does not support this protocol.

    next

end

The IKE port must match the one configured in the FortiClient, in this case, 443. If not defined, the FortiGate will use 4500 by default.

config system settings
    set ike-tcp-port 443
end

With that, the 'wrong transport, phase 1 uses non udp' error message shown in the IKE debug should no longer appear, and the IPsec VPN connection over TCP should be established.

Note:

• IPsec Dialup VPN over TCP is supported from FortiOS v7.4.6 onwards and requires FortiClient v7.4.0 or later. 
• The Fortinet ESP protocol is intended for FortiGate-to-FortiGate tunnels only, see the article How to use TCP as transport for IKE/IPsec traffic.

Related documents:

Encapsulate ESP packets within TCP headers

FortiOS 7.6.0 SSL VPN to IPsec VPN Migration

Fortinet-ESP Parameter remains enabled after changing transport mode, causing IPsec Traffic drops