Technical Tip: Dial-up VPN IPsec over TCP best practices

Description

This article describes recommended best practices for deploying an IPsec dial-up Virtual Private Network (VPN) tunnel over Transmission Control Protocol (TCP) on FortiGate devices. This configuration is typically used in environments where User Datagram Protocol (UDP) traffic is restricted or blocked by intermediate firewalls or carrier-grade NAT.

Scope

FortiGate v7.4.2 and above.

Solution

Use case:

IPsec VPN over TCP enables IPsec tunnels to function reliably in networks where standard protocols such as UDP/500 and IP protocol 50 (ESP) are not permitted. TCP encapsulation ensures that VPN traffic can traverse restrictive networks, including public Wi-Fi, mobile carrier networks, or enterprise firewalls with strict outbound rules.

Recommended best practices:

• Use TCP port 443 for transport if Secure Sockets Layer (SSL) VPN or web services are not running on the same port/interface.

• Enable TCP encapsulation only on the dial-up server FortiGate to maintain the expected traffic flow and avoid dual encapsulation.

• Configure tcp-mss-sender and tcp-mss-receiver to prevent fragmentation. The recommended value is 1360.

• Disable ESP detection on intermediate firewalls when TCP is exclusively used for transport.

• Monitor system resources, as TCP encapsulation may introduce additional CPU overhead compared to native ESP/IPsec transport.

Example CLI configuration:

On the dial-up server FortiGate (listening on TCP port 443):

config system settings 
    show full | grep ike-tcp-port
    set ike-tcp-port 443
end

config vpn ipsec phase1-interface
    edit "dialup-tcp"
        set type dynamic
        set interface "wan1"
        set ike-version2
        set peertype any
        set proposal aes256-sha256
        set dhgrp 14
        set transport tcp   
        set psksecret ENC <encrypted_psk>
    next
end

config firewall policy
    edit <id>
        set tcp-mss-sender 1360
        set tcp-mss-receiver 1360
    next
end

On the dial-up client FortiGate:

config system settings 
    show full | grep ike-tcp-port
    set ike-tcp-port 443
end

config vpn ipsec phase1-interface
    edit "client-tcp"
        set type dynamic
        set interface "wan1"
        set remote-gw x.x.x.x
        set ike-version 2
        set peertype any
        set proposal aes256-sha256
        set dhgrp 14
        set transport tcp
        set psksecret ENC <encrypted_psk>
    next
end

config firewall policy
    edit <id>
        set tcp-mss-sender 1360
        set tcp-mss-receiver 1360
    next
end

Monitoring:

Use the following command to verify the tunnel's encapsulation method and active status:

diagnose vpn tunnel list | grep encap

Additional considerations:

• TCP-encapsulated VPN traffic may be subject to inspection or throttling by deep packet inspection engines or secure web gateways.

• Do not bind both IPsec over TCP and SSL VPN services to the same interface and TCP port (for example, port 443) unless virtual IP (VIP) mapping or port forwarding is configured.

• Ensure intermediate network devices allow outbound TCP connections to the configured VPN port.

• This feature only works with IKE version 2.

• IPsec over TCP is not supported on free FortiClient VPN versions. Other limitations are described in FortiClient standalone and licensed version feature comparison.

Related documents:

• Getting started

• Technical Tip: Troubleshooting IPsec VPN tunnel connectivity