Technical Tip: TCP and ESP packets out of order

Description

This article describes that sometimes TCP packets may be sent out of order, causing sessions to drop due to heavy load on the firewall.

The same can happen for IPsec tunnel traffic in the form of ESP packets sent out of order, causing the remote router to receive those packets with errors such as 'invalid spi' or 'HMAC validation failed'.

Scope

Reducing or eliminating packets out of order for TCP and ESP traffic.

Solution

Firewall admins can try stopping this behavior with some options to alleviate the load on the firewall NPU in the following ways:

1. TCP sessions: The recommendation is to inject delay on the firewall policy level. This ensures the firewall gets more time to handle TCP sessions:

config firewall policy
 edit <policy id>
     set delay-tcp-npu-session enable
 next
end    

2. ESP packets: There are 2 commands under global and under the IPsec tunnel configuration itself that can help reduce or eliminate this issue. The first is to disable the HMAC offloading to the NPU while the second is to force fragmentation on the the IPsec tunnel:

    

config system global
    set ipsec-hmac-offload disable
end

config vpn ipsec phase1-interface
    edit <tunnel name>
        set ip-fragmentation pre-encapsulation
   next
end

3. Prevent fragmentation by configuring TCP MSS clamping either on the firewall policy or directly on the interface. This helps reduce ESP packet overhead and applies to all traffic entering and exiting the IPsec tunnel.
   

Firewall Policy MSS clamping:

config firewall policy
    edit <policy_id>
        set tcp-mss-sender 1360
        set tcp-mss-receiver 1360
    end

Interface MTU/MSS clamping:

config system interface
    edit "<vpn_interface_name>"
        set mtu-override enable
        set mtu 1400
        set tcp-mss 1360
    next
end

Note: Anti-replay can affect the traffic through the tunnel, which may lead to ESP packets being dropped.
Having the same anti-replay setting on both the local and peer IPsec is recommended.

The discarded packets will be logged with the following message in the Event Log: 'Invalid ESP packet detected (replayed packet)'.

config vpn ipsec phase2-interface
    edit <phase2-name>
        set replay <enable | disable>
    next
end