Technical Tip: TCP session over VXLAN stalls after being accepted and offloaded to NP7/NP7lite

Known issue 1179259 only affects TCP traffic ingressing or egressing over a VXLAN interface. Refer to the following article for an example VXLAN-over-IPSec configuration: Technical Tip: How to setup a VXLAN over IPsec deployment

Example topology:

VXLAN-over-IPSec is configured, and the VXLAN interface is a software switch member. A Windows endpoint (IP Address 172.16.5.25) behind FGT-A attempts to SSH to FGT-B's mgmt interface across the VXLAN over IPSec tunnel.

FGT-A has a software switch and VXLAN interface configured as follows.

config system switch-interface

    edit "software-switch"

        set vdom "root"

        set member "vxlan_int" "soft_switch_int"

        set intra-switch-policy explicit

    next

end

FGT-A has a firewall policy configured to allow SSH connections from FGT-A’s LAN to FGT-B’s management interface IP address over the VXLAN.

config firewall policy

    edit 999

        set name "SSH_To_FGT-B"

        set srcintf "soft_switch_int"

        set dstintf "vxlan_int"

        set action accept

        set srcaddr "172.16.5.25"

        set dstaddr "FGT-B_Mgmt_IP"

        set schedule "always"

        set service "SSH" "PING"

        set logtraffic all

    next

end

During an initial SSH connection attempt, the TCP handshake completes, and a debug flow shows the session is offloaded to the Network Processor. The TCP session then stalls, and no further traffic is seen. The SSH client does not receive a login prompt.

Trigger Conditions:

The issue is only observed when the following conditions are met:

• Source or Destination interface for the firewall policy is a VXLAN going over a site-to-site IPSec tunnel.
• FortiGate has an NP7 or NP7Lite chipset.
• TCP traffic is handled by a firewall policy.
• The issue is intermittent.

Additionally, the issue is more likely to trigger with the following condition: UTM features are enabled on the firewall policy.

Workaround:

Disable auto-asic-offload on firewall policies matching the trigger conditions.

config firewall policy

    edit 999

        set auto-asic-offload disable

    next

end

Resolution:

The tracking ID# for this issue is 1179259, see Known Issues. An initial fix is planned for a future FortiOS release, which will revert TCP offload for VXLAN traffic (UDP traffic remains eligible for offload). On patched versions, CPU usage may increase for devices handling large volumes of TCP traffic over VXLAN interfaces, similar to devices that have the workaround applied.