Technical Tip: IPsec site to site IKEv2 VPN between FortiGate and third-party vendor is down with error 'INVALID_SYNTAX'

Description

This article describes how to troubleshoot the issue when the IPsec IKEv2 tunnel between FortiGate and any third-party goes down and shows the error 'INVALID_SYNTAX'.

Scope

FortiGate.

Solution

Considering FortiGate to be the initiator and any third-party vendor to be the responder in the setup.

IKEv2 has two phases, IKE_SA_INIT Exchange and IKE_AUTH Exchange.

During the IKE_AUTH Exchange second message, if the notify message (Payload: Notify (41) - INVALID_SYNTAX.), it indicates that it is a Phase 2 selector mismatch.

This can be verified with the IKE debugs as well:

diagnose debug application ike -1
diagnose debug console timestamp enable
diagnose debug enable

When running the IKE debug, it shows this error as shown below.

When a packet capture is run, it shows the result as shown in the packet capture below:

To fix the issue, match the Phase 2 selectors on both units. 

Note: It is recommended that an IPsec VPN with third-party devices have individual Phase 2 selectors for different subnets instead of using an address group. This is because FortiGate builds a single Security Association (SA) that shares one SPI (Security Parameter Index) for all subnets, while third-party firewalls use a separate SPI value for each subnet. FortiGate creates different SPIs for multiple Phase-2 selectors.