FortiGate
FortiGate Next Generation Firewall utilizes purpose-built security processors and threat intelligence security services from FortiGuard labs to deliver top-rated protection and high performance, including encrypted traffic.
fwilliams
Staff
Staff
Article Id 230343
Description

This article describes how to understand and troubleshoot 'HMAC validation failed' error. 

Scope FortiGate v6.4, v7.0 and v7.2.
Solution

The Error:

 

Invalid ESP packet detected (HMAC validation failed).

 

Reason:

The reason of this error on FortiGate is that the MAC calculated by FortiGate does not match the one inside the ESP packet received from the VPN peer.

 

The cause of this error can reside in 2 places:

  1. In the Kernel (software).
  2. In the Hardware (NPU).

 

In case it resides in the kernel, it will be visible in:

 

  • Event logs:

Similar messages will be visible in the log:

 

status=esp_error error_num=Invalid ESP packet detected (HMAC validation failed). spi=fe2b6574 seq=0000ba0d

 

  • IKE debug:

Similar messages will be visible in IKE debug:

 

ike 1:My_VPN: invalid ESP 1 (HMAC) SPI fe2c9989 seq 0000c10f 9  x.x.x.x->y.y.y.y

 

  • The RX error (rxe) counter:

Use ' diag netlink interface list My_VPN' on the IPSec interface (phase 1), repeat the command a couple of times, and the rxe counter will increase.

 

diag netlink interface list My_VPN

if=My_VPN family=00 type=768 index=15 mtu=1500 link=0 master=0

ref=15 state=off start fw_flags=0 flags=up p2p run noarp multicast

Qdisc=noqueue

stat: rxp=0 txp=0 rxb=0 txb=0 rxe=125 txe=0 rxd=0 txd=0 mc=0 collision=0

 

It can reside in hardware (NPU):

In this case, the error will not be visible in the first 3 places mentioned (Event log, IKE debug, and rxe counter). It is only possible to see the error in the NPU DCE counter (non-zero sub-engine drop count).

 

To check NPU DCE counter, use:

  1. For NP6, NP6XLite, and NP6Lite.

    diag npu np6xlite dce 0   <-----np6xlite platform .
    diag npu np6 dce 0/1       <----- np6 platform.
    diag npu np6lite dce 0    <---np6lite platform

  2. For NP7 and NP7Lite.

diag npu np7lite dce-drop-all 0   <-----np7lite platform (as NP7lite platforms have only one NP, omit the 0).

diag npu np7 dce-drop-all 0/1      <----- np7 platform (only type in NP ID 0 or 1, but use 0/1).

 

Note:

Use the below commands to display only the queue/s with drops on NP7/NP7lite, instead of printing all the queues.

 

diag npu np7lite dce-drop-all 0 brief

diag npu np7lite dce-drop-all 0 b

diag npu np7lite dce-drop-all 0 0

 

diag npu np7 dce-drop-all 0 brief

diag npu np7 dce-drop-all 0 b

diag npu np7 dce-drop-all 0 0

 

Possible causes:

  1. This error can be caused by a misconfiguration on any of the peers (local FortiGate or remote FortiGate if both ends are FortiGates, or a third-party device if peering with 3rd party).
  2. It can be because the ESP packet was altered.
    Wireshark can be used to check the HMAC of the ESP packet sent and received on the FortiGate.
  3. It can be an issue with the NPU offloading.
  4. It may be due to a calculation error during encryption on the sender or decryption on the receiver.

 

Fixes:

  1. Ensure that the proposal matches (on local and remote) 
  2. Ensure the IPsec SA Encrypt/Decrypt algorithm and HMAC are supported by the unit’s NPU 

Confirm the Algorithm that support acceleration in accord of NP model:  Encryption algorithms

 

  1. Ensure the interface bound to IPSec phase 1 is on the NPU’s port list.

 

Check with:

 

diag npu np6 port-list | grep <port_name>

diag npu np6xlite port-list | grep <port_name>

 

  1. Try to disable hardware offload under phase1:

    config vpn ipsec phase1-interface
        edit <p1_name>
            set npu-offload disable
    end

  2. CPx offload can be disabled if needed:

    config system global
       set ipsec-hmac-offload disable
       set ipsec-asic-offload enable
    end

  3. Disable anti-reply under phase 2:

    config vpn ipsec phase2-interface
        edit <p2_name>
            set replay disable
    end

  4. Disable IPsec-inbound-cache:

    config system npu
        set ipsec-inbound-cache disable
    end