Technical Tip: IPsec tunnel is not coming up due to error message AUTHENTICATION_FAILED

[anoushiravan]

Description	This article describes the possible reasons that the IPsec tunnel via ikev2 fails, usually, this issue happens when the third-party device is acting as a responder in the IPsec tunnel.
Scope	FortiGate.
Solution	In IKEv2, IKE AUTH (authentication) takes place after the SA_INIT exchange, initiator sending an AUTH message to the other side mainly for authentication purposes. Here are partial IKE negotiation logs between FortiGate and Zscaler that show the remote side is rejecting authentication messages sent by the FortiGate side: ikee 0:IPSECVPN_Zscaler:1094499: IKE SA 65d3ae182a9bf8e4/53cd353c74a2861e SK_ar 32:20171069004658BE6262ADCA4DC83BEAFDD30075C6AD5632804A0863523C26E4 ike 0:IPSECVPN_Zscaler:1094499: initiator preparing AUTH msg <-- ike 0:IPSECVPN_Zscaler:1094499: sending INITIAL-CONTACT ike 0:IPSECVPN_Zscaler:1094499: enc 2900000C010000000AA13DFE27000008000040002900002802000000A9AA939C0581928F22D90AF7AB4F462454C46EB55DB268444172CB3D5687B35521000008000040242C000028000000240103040343EC1C83030000080100000B030000080300000100000008050000002D00001801000000070000100000FFFF00000000FFFFFFFF0000001801000000070000100000FFFFB92ED400B92ED5FF03020103 ike 0:IPSECVPN_Zscaler:1094499: detected NAT ike 0:IPSECVPN_Zscaler:1094499: NAT-T float port 4500 ike 0:IPSECVPN_Zscaler:1094499: out   65D3AE182A9BF8E453CD353C74A2861E2E20230800000001000000D8230000BC856A01C42BA44075D88F5E70549ED2EEC749348587C4305BB4B7B506D58BE5D48EB8AFFA6A67ED30E393BEAA1D4D03D3A5F7C327C24024E6BA1CB380B5BDD763838383FEC5DBA5FC26316DF03A70C873BF303A02AB033026C8AB88625DF1C8518D15B8387A0EAFFA94E00128CCCEE028CA3BB7BBF80E0ACFB725F1AD0F7D6A283FECE39F4D7695A912DC32D4F5B483BD426D60F31EAAAAEF69B513B9ABBC2C30E69E39F8938D8F43E6B29FE991ABD59A162AF1C927252BD7 ike 0:IPSECVPN_Zscaler:1094499: sent IKE msg (AUTH): 100.11.48.54:4500->15.146.22.73:4500, len=216, id=65d3ae182a9bf8e4/53cd353c74a2861e:00000001 <-- FortiGate sent AUTH message to the remote side. ike 0: comes 15.146.22.73:4500->100.11.48.54:4500,ifindex=4....  <-- FortiGate received AUTH response for remote side. ike 0: IKEv2 exchange=AUTH_RESPONSE id=65d3ae182a9bf8e4/53cd353c74a2861e:00000001 len=72 ike 0: in 65D3AE182A9BF8E453CD353C74A2861E2E20232000000001000000482900002C379569B9D066C5C6A7BD1C45D3DB3A44EAA98D4E1EA232756F9A1B0EDD27213A28F35F2832BA60EC ike 0:IPSECVPN_Zscaler:1094499: dec 65D3AE182A9BF8E453CD353C74A2861E2E2023200000000100000028290000040000000801000018 ike 0:IPSECVPN_Zscaler:1094499: initiator received AUTH msg  ike 0:IPSECVPN_Zscaler:1094499: received notify type AUTHENTICATION_FAILED <-- AUTH message has been rejected by remote side Zscaler. ike 0:IPSECVPN_Zscaler:1094499: schedule delete of IKE SA 65d3ae182a9bf8e4/53cd353c74a2861e <-- ike 0:IPSECVPN_Zscaler:1094499: scheduled delete of IKE SA 65d3ae182a9bf8e4/53cd353c74a2861e ike 0:IPSECVPN_Zscaler: connection expiring due to phase1 down Note: The AUTH message is protected by the cryptographic algorithms and the keys from the SA_INIT message. Generally, 'malformed message' error describes that there is a mismatch, possible reasons that the remote side might reject the AUTH message as responder are as follows and the remote side should be checked: Phase 2 proposal mismatched/incompatibility. PFS negotiation, FortiGate does not negotiate the PFS with Auth Message so the remote side should not expect PFS when negotiating the AUTH message. Disable PFS in phase 2 on both sides to check the issue. EAP setting, which is disabled on the FortiGate side by default, EAP can be checked via the command: show full vpn ipsec phase1-interface | grep eap. EAP is used to authenticate the initiator against an EAP Server, the initiator’s AUTH payload is therefore sent in the last initiator’s IKE_AUTH request, after EAP is completed. When EAP is not used, IKE AUTH is made of a single request/response exchange, when EAP is used the IKE AUTH is made of multiple request/response exchanges, the number of exchanges depends on the EAP method being used. Mismatched preshared keys (might be a certificate issue if the certificate is used as a preshared key). Mismatched phase2 selector. By default first selector is negotiated during the IKE AUTH message, in case multiple FortiOS phase 2 are configured, they are negotiated during subsequent CREATE_CHILD_SA exchanges. Mismatched mode-cfg (IP/mask, DNS,…) in phase 1. In cloud platforms, other vendors/remote peers sometimes expect the local ID to be the FortiGate interface Public IP. It is necessary to configure the local ID and local ID type in the phase1-interface. config vpn ipsec phase1-interface     edit " tunnelname"         set localid-type keyid         set localid <(WAN-PUBLIC-IP>     end Related article: Troubleshooting Tip: Error 'received notify type AUTHENTICATION_FAILED' is obtained when the IPSEC t...  Technical Tip: IPsec VPN error 'ike Negotiate SA Error: ike ike [1470]'.