Troubleshooting Tip: MacOS IKEv2 Dial-up VPN fails with 'Peer Certificate is Invalid'

Description

This article describes an issue with connecting to IPsec Dial-up VPN using a macOS FortiClient. 

Scope

FortiGate v7.4.

FortiClient macOS. 

Solution

When using FortiClient macOS to connect to an IPsec Dial-up VPN using IKEv2 with certificate authentication, the error 'Peer Certificate is Invalid' is observed. The same configuration works with FortiClient Window OS, Android, iOS phone.

The following article discusses the recommended configuration for certificate authentication for IPsec IKEv2: Technical Tip: Certificate authentication for FortiClient remote access dialup IPsec clients with SAML user authentication.

The dial-up VPN configuration is as follows:

config vpn ipsec phase1-interface
    edit "VPN"
        set type dynamic
        set interface "x1"
        set ike-version 2
        set authmethod signature
        set peertype peergrp
        set net-device disable
        set mode-cfg enable
        set proposal aes256-sha256
        set dpd on-idle
        set dhgrp 14
        set eap enable
        set eap-identity send-request
        set eap-cert-auth enable
        set certificate "test.com.ca"
        set peergrp "USER-VPN"
        set ipv4-start-ip 192.168.250.150
        set ipv4-end-ip 192.168.250.250
        set ipv4-netmask 255.255.255.0
        set dns-mode auto
        set ipv4-split-include "Internal"
        set dpd-retryinterval 60
    next
end

config vpn ipsec phase2-interface
    edit "BOD_VPN"
        set phase1name "VPN"
        set proposal aes256-sha256
        set dhgrp 14
    next
end

When collecting the debugs, the following error 'unexpected payload type 41' is observed. 

Diagnose commands:

diagnose debug application ike -1
diagnose debug enable

Output: 

ike V=root:0:BOD_VPN:472031: sent IKE msg (AUTH_RESPONSE): 103.199.6.2:4500->103.108.136.10:62140, len=206, vrf=0, id=866e7f95868e762f/4bd5ee6f963d1932:00000001, oif=25
ike V=root:0: comes 103.108.136.10:62140->103.199.6.2:4500,ifindex=25,vrf=0,len=84....
ike V=root:0: IKEv2 exchange=INFORMATIONAL id=866e7f95868e762f/4bd5ee6f963d1932:00000002 len=80
ike 0: in [hash omitted]
ike 0:BOD_VPN:472031: dec [hash omitted]
ike V=root:0:BOD_VPN:472031: responder received EAP msg
ike V=root:0:BOD_VPN:472031: unexpected payload type 41
ike V=root:0:BOD_VPN:472031: schedule delete of IKE SA 866e7f95868e762f/4bd5ee6f963d1932

The same debug outputs can be exported from FortiClient. Refer to Technical Tip: How to generate and export Debug logs from various platforms running with FortiClient and FortiClient EMS. Locate the folder named 'iked' logs. 

0x68e4 Default 0x0 2081 0 iked: [com.fortinet.forticlient:IPSec] ca_setauth: auth length 663
0x68e4 Default 0x0 2081 0 iked: [com.fortinet.forticlient:IPSec] sa_stateok: SA_INIT flags 0x0000, require 0x0009 cert,auth
0x68e4 Default 0x0 2081 0 iked: [com.fortinet.forticlient:IPSec] config_free_proposals: free 0x104e19870
0x68e1 Default 0x0 2079 0 iked: [com.fortinet.forticlient:IPSec] ca_getreq: found CA /DC=test/DC=com/DC=test/CN=test
0x68e1 Default 0x0 2079 0 iked: [com.fortinet.forticlient:IPSec] ca_cert_local: certificate key mismatch
0x68e1 Default 0x0 2079 0 iked: [com.fortinet.forticlient:IPSec] spi=0x5b76dd3aafc1ee78: ca_getreq: found cert with matching ID but without matching key.
0x68e1 Default 0x0 2079 0 iked: [com.fortinet.forticlient:IPSec] ca_getreq: found local certificate <certificate name>
0x68e4 Default 0x0 2081 0 iked: [com.fortinet.forticlient:IPSec] ikev2_pld_auth: method RSA_SIG length 256
0x68e4 Default 0x0 2081 0 iked: [com.fortinet.forticlient:IPSec] ikev2_pld_payloads: decrypted payload EAP nextpayload NONE critical 0x00 length 9
0x68e4 Default 0x0 2081 0 iked: [com.fortinet.forticlient:IPSec] spi=0x5b76dd3aafc1ee78: ikev2_pld_eap: REQUEST id 19 length 5 EAP-IDENTITY
0x68e4 Default 0x0 2081 0 iked: [com.fortinet.forticlient:IPSec] spi=0x5b76dd3aafc1ee78: sa_state: SA_INIT -> AUTH_REQUEST
0x68e4 Default 0x0 2081 0 iked: [com.fortinet.forticlient:IPSec] policy_lookup: peerid '/CN=*.test.com.ca'
0x68e4 Default 0x0 2081 0 iked: [com.fortinet.forticlient:IPSec] proposals_negotiate: score 4
0x68e4 Default 0x0 2081 0 iked: [com.fortinet.forticlient:IPSec] policy_lookup: setting policy 'vpn'
0x68e4 Default 0x0 2081 0 iked: [com.fortinet.forticlient:IPSec] ikev2_ike_auth: awaiting response from CA process
0x68e1 Default 0x0 2079 0 iked: [com.fortinet.forticlient:IPSec] ca_validate_pubkey: unsupported public key type ASN1_DN
0x68e1 Default 0x0 2079 0 iked: [com.fortinet.forticlient:IPSec] ca_validate_cert: /CN=*.test.com.ca unable to get local issuer certificate
0x68e4 Default 0x0 2081 0 iked: [com.fortinet.forticlient:IPSec] ikev2_getimsgdata: imsg 25 rspi 0x1d92ec273c838fb2 ispi 0x5b76dd3aafc1ee78 initiator 1 sa valid type 4 data length 1627
0x68e4 Default 0x0 2081 0 iked: [com.fortinet.forticlient:IPSec] spi=0x5b76dd3aafc1ee78: ikev2_dispatch_cert: peer certificate is invalid
0x68e4 Default 0x0 2081 0 iked: [com.fortinet.forticlient:IPSec] FCT send error. server addr: 103.199.6.2, error code: -304

Workaround:

1. Ensure the root CA certificate/any intermediate CA is/are imported to the local user machine.

2. Upgrade to FortiOS v7.6.2. This FortiOS version works with FortiClient macOS.