Fortigate 8.0.0 Remote VPN Using Cert + EAP

i dont think the certificate itself is failing here.

the debug shows:

auth_cert_success

and:

matched peer user 'Remote-Employee_peer'

so the cert/peer validation looks successful. the failure happens immediately after, when fnbamd tries to return the auth result:

failed send reply (2788, errno 101)

i would capture ike and fnbamd debug from the same attempt and also share the phase1 configuration, especially eap-cert-auth, peer and authusrgrp.

fortios 8.0.0 supports certificate + eap for ikev2:

https://docs.fortinet.com/document/fortigate/8.0.0/administration-guide/298520/vpn-2fa-with-eap-and-certificate-authentication

known issues:

https://docs.fortinet.com/document/fortigate/8.0.0/fortios-release-notes/236526/known-issues

resolved issues:

https://docs.fortinet.com/document/fortigate/8.0.0/fortios-release-notes/289806/resolved-issues

Hi PackerProgrammer,
 

The authentication flow stops at the point where the certificate result needs to be delivered back to the IKE daemon. The key error lines are 'fnbamd_comm_send_result - Failed send reply (2788, errno 101)'
 

Could you please run the commands below and share the output with us? This will help us check for any daemon crashes or resource-related issues on the CLI while the user is attempting to connect:
 

diagnose  debug crashlog read | grep 2026
diagnose sys top 4 40 10
 

To capture the IKE, fnbamd, and EAP debug:

diagnose debug reset
diagnose debug console timestamp enable
diagnose vpn ike log filter rem-addr4 <client_public_ip>
diagnose debug application ike -1
diagnose debug application fnbamd -1
diagnose debug application eap_proxy -1
diagnose debug enable

Once test done, run, diag debug disable to stop debugging.
 

Regards,
Aman

That Failed send reply (2788, errno 101) line is a dead giveaway. Errno 101 in Linux/FortiOS is "Network is unreachable", which basically means fnbamd (the authentication daemon) is trying to send the success result back over an internal IPC socket or to a helper process, but the payload buffer size (2776/2788 bytes) is hitting a limit or the process listening for the auth result dropped off.

Since you switched to presenting the EMS cert to handle EMS tag checks, your cert payload/SAN chain size probably bloated significantly compared to your basic internal PKI cert. A couple of things to check/try:

Check for 8.0.0 bugs: Honestly, running x.0.0 in production for FortiOS is always a wild ride. Early major releases are notorious for fnbamd memory leaks or daemon crashes under larger cert payloads. If you kill/restart fnbamd (fn fnbamd restart via CLI), does it work temporarily on the first attempt? If yes, it's almost certainly a daemon IPC bug in 8.0.0 and you'll want to open a TAC ticket or roll back to a mature patch release if possible.

Cert Chain & Payload Size: Check how many CA/Sub-CA certificates are attached or being sent in the EAP exchange. If the client is sending a full thick chain along with EMS metadata, fnbamd might be choking when trying to pass the full auth payload back to iked/eap. Try paring down the trusted chain under config user peer to only the explicit intermediate/root needed so it doesn't try to pass massive state back over the IPC.

I'd bet money this is an early 8.0.0 fnbamd IPC buffer bug with large cert/EAP payloads. Might be worth opening a ticket with TAC to see if they have a known mantis ID for it yet.

i agree with ​@kaman here. i would not assume the 2776/2788 byte value means a buffer limit yet.

in other fnbamd/eap buffer-related cases, fortios normally shows a much more explicit message such as not enough buffer for eap message.

since the certificate itself is matching successfully but the result cannot be delivered back to ike, i would first check the daemon/session state:

diagnose test application fnbamd 1

and also collect the crashlog + sys top output ​@kaman requested during the failed attempt.

there was also a known issue where fnbamd sessions could remain pending until the maximum session limit was reached with eap enabled, although fortinet lists that issue as fixed in 8.0.0.

https://community.fortinet.com/fortigate-3/troubleshooting-tip-unable-to-connect-dial-up-ipsec-vpn-ikev2-with-local-user-due-to-max-session-number-reached-228286

https://docs.fortinet.com/document/fortigate/8.0.0/fortios-release-notes/289806/resolved-issues

so i would check the fnbamd session counters first before restarting the daemon or changing the certificate chain.

so I did more testing the last 2 days and was able to successfully get IKE to succeed and move on to EAP. The catch is, I had to go back to using my PKI cert. Then i did more testing, and i was successful with the EMS issued cert if I stored in the local cert store on the Fortigate. After looking at both traces, the same error lines appear, so it seems they are not indicative of the issue. Check out these 2 traces. The first one is success, the second one is fail. This leads me to believe that the issue only arises when cert_trust_store is set to ems. set cert_trust_store local forces it to use the local store,  but, you cannot check ZTNA status. This feature may be quite new (i’m new to Fortinet) but since the option is there, i would like to use it.

ike V=root:0:Remote-Employee:594: reassembled fragmented message
ike V=root:0:Remote-Employee:594: responder received AUTH msg
ike V=root:0:Remote-Employee:594: processing notify type INITIAL_CONTACT
ike V=root:0:Remote-Employee:594: received peer identifier DER_ASN1_DN 'C = CA, ST = BC, L = Burnaby, O = Fortinet, CN = xxxxx'
ike V=root:0:Remote-Employee:594: eap-peer=yes
ike V=root:0:Remote-Employee:594: match gw peer ID by FNBAM
ike V=root:0:Remote-Employee:594: Validating X.509 certificate
ike V=root:0:Remote-Employee:594: peer cert, subject='xxxxx', issuer='xxxx'
ike V=root:0:Remote-Employee:594: peer ID verified
ike V=root:0:Remote-Employee:594: building fnbam peer candidate list
ike V=root:0:Remote-Employee:594: FNBAM_GROUP_NAME candidate 'Remote-Employee_peer'
ike V=root:0:Remote-Employee:594: certificate validation before eap pending
fnbamd_dbg_hex_pnt[50] balancer req from (8)-01 00 00 61 35 66 34 64 
[2438] handle_req-instance 1
[3253] handle_req-Rcvd auth_cert req id=84988985766092, len=2493, opt=0
[1226] __cert_auth_ctx_init-req_id=84988985766092, opt=0
[113] __cert_chg_st- 'Init'
[205] fnbamd_cert_load_certs_from_req-1 cert(s) in req.
[854] __cert_init-req_id=84988985766092
[903] __cert_build_chain-req_id=84988985766092
[323] fnbamd_chain_build-Chain discovery, opt 0x13, cur total 1
[341] fnbamd_chain_build-Following depth 0
[376] fnbamd_chain_build-Extend chain by system trust store. (good: 'CA_Cert_1')
[341] fnbamd_chain_build-Following depth 1
[355] fnbamd_chain_build-Self-sign detected.
[109] __cert_chg_st- 'Init' -> 'Validation'
[1025] __cert_verify-req_id=84988985766092
[1026] __cert_verify-Chain is complete.
[540] fnbamd_cert_verify-Chain number:2
[554] fnbamd_cert_verify-Following cert chain depth 0
[627] fnbamd_cert_verify-Issuer found: CA_Cert_1 (SSL_DPI opt 1)
[554] fnbamd_cert_verify-Following cert chain depth 1
[1057] __cert_verify-peer_info.no_ocsp_query:0 cert->status:640.
[768] fnbamd_cert_check_group_list-checking group with name 'Remote-Employee_peer'
[581] __check_add_peer-check 'Remote-Employee_peer'
[435] peer_san_subject_check-SAN: host_good 1; email_good 1; ip_good 1
[454] peer_subject_cn_check-Cert subject 'C = CA, ST = BC, L = Burnaby, O = Fortinet, CN = xxxxx'
[348] __RDN_match-Checking 'O' val 'Fortinet' -- match.
[378] __cert_subject_RDN_compare-Total matched RDNs in cert: 1
[479] peer_subject_cn_check-Subject is good.
[588] __check_add_peer-'Remote-Employee_peer' check ret:good
[703] __peer_user_clear_unmatched-Clear all user(s) other than 'Remote-Employee_peer'
[724] __peer_user_clear_unmatched-
[200] __get_default_ocsp_ctx-def_ocsp_ctx=(nil), no_ocsp_query=0, ocsp_enabled=0
[841] fnbamd_cert_check_group_list-Peer users
[844] fnbamd_cert_check_group_list-    'Remote-Employee_peer' ('N/A','N/A','N/A')
[1069] __cert_verify_do_next-req_id=84988985766092
[109] __cert_chg_st- 'Validation' -> 'Done'
[1163] __cert_done-req_id=84988985766092
[1742] fnbamd_auth_session_done-Session done, id=84988985766092
[1209] __fnbamd_cert_auth_run-Exit, req_id=84988985766092
[1785] create_auth_cert_session-fnbamd_cert_auth_init returns 0, id=84988985766092
[1698] auth_cert_success-id=84988985766092
[1321] fnbamd_cert_auth_copy_cert_status-req_id=84988985766092
[1329] fnbamd_cert_auth_copy_cert_status-Matched peer user 'Remote-Employee_peer'
[969] fnbamd_cert_check_matched_groups-checking group with name 'Remote-Employee_peer', peer_ctx->peer_user->setting.name:Remote-Employee_peer
[1033] fnbamd_cert_check_matched_groups-matched
[1362] fnbamd_cert_auth_copy_cert_status-Leaf cert status is unchecked.
[1458] fnbamd_cert_auth_copy_cert_status-Cert st 2c0, req_id=84988985766092
[356] fnbamd_comm_send_result-Sending result 0 (nid 672) for req 84988985766092, len=2798
[360] fnbamd_comm_send_result-Failed send reply (2810, errno 2)
[1567] destroy_auth_cert_session-id=84988985766092
[1293] fnbamd_cert_auth_uninit-req_id=84988985766092
[1996] fnbamd_ldaps_destroy-
[1667] fnbamd_rads_destroy-
[138] fnbamd_peer_ctx_free-Freeing peer ctx 'Remote-Employee_peer'
[2047] handle_child_rsp-Auth rsp 84988985766092, session not created, line 76
ike V=root:0:Remote-Employee:594: fnbam reply 'Remote-Employee_peer'
ike V=root:0:Remote-Employee:594: fnbam matched peer 'Remote-Employee_peer'
ike V=root:0:Remote-Employee:594: certificate validation before eap succeeded
ike V=root:0:Remote-Employee:594: responder preparing EAP identity request
 

2026-09-02 08:19:18.158865 ike V=root:0:Remote-Employee:576: reassembled fragmented message
2026-09-02 08:19:18.158877 ike V=root:0:Remote-Employee:576: responder received AUTH msg
2026-09-02 08:19:18.158885 ike V=root:0:Remote-Employee:576: processing notify type INITIAL_CONTACT
2026-09-02 08:19:18.158950 ike V=root:0:Remote-Employee:576: received peer identifier DER_ASN1_DN 'C = CA, ST = BC, L = Burnaby, O = Fortinet, CN = xxxx'
2026-09-02 08:19:18.158959 ike V=root:0:Remote-Employee:576: eap-peer=yes
2026-09-02 08:19:18.158965 ike V=root:0:Remote-Employee:576: match gw peer ID by FNBAM
2026-09-02 08:19:18.158984 ike V=root:0:Remote-Employee:576: Validating X.509 certificate
2026-09-02 08:19:18.159090 ike V=root:0:Remote-Employee:576: peer cert, subject='xxxx', issuer='xxxx'
2026-09-02 08:19:18.159104 ike V=root:0:Remote-Employee:576: peer ID verified
2026-09-02 08:19:18.159110 ike V=root:0:Remote-Employee:576: building fnbam peer candidate list
2026-09-02 08:19:18.159115 ike V=root:0:Remote-Employee:576: FNBAM_GROUP_NAME candidate 'Remote-Employee_peer'
2026-09-02 08:19:18.159144 ike V=root:0:Remote-Employee:576: certificate validation before eap pending
2026-09-02 08:19:18 fnbamd_dbg_hex_pnt[50] balancer req from (8)-01 00 00 61 35 66 34 64 
2026-09-02 08:19:18 [2438] handle_req-instance 1
2026-09-02 08:19:18 [3253] handle_req-Rcvd auth_cert req id=84988985766079, len=2493, opt=20
2026-09-02 08:19:18 [1226] __cert_auth_ctx_init-req_id=84988985766079, opt=20
2026-09-02 08:19:18 [113] __cert_chg_st- 'Init'
2026-09-02 08:19:18 [205] fnbamd_cert_load_certs_from_req-1 cert(s) in req.
2026-09-02 08:19:18 [854] __cert_init-req_id=84988985766079
2026-09-02 08:19:18 [903] __cert_build_chain-req_id=84988985766079
2026-09-02 08:19:18 [323] fnbamd_chain_build-Chain discovery, opt 0x31, cur total 1
2026-09-02 08:19:18 [341] fnbamd_chain_build-Following depth 0
2026-09-02 08:19:18 [113] __ztna_entry_lookup-vfid:-1 subject or issuer:C = CA, ST = California, L = Sunnyvale, O = Fortinet, OU = xx, CN = xxx.
2026-09-02 08:19:18 [396] fnbamd_chain_build-Extend chain by ZTNA CA cache. (good)
2026-09-02 08:19:18 [341] fnbamd_chain_build-Following depth 1
2026-09-02 08:19:18 [355] fnbamd_chain_build-Self-sign detected.
2026-09-02 08:19:18 [109] __cert_chg_st- 'Init' -> 'Validation'
2026-09-02 08:19:18 [1025] __cert_verify-req_id=84988985766079
2026-09-02 08:19:18 [1026] __cert_verify-Chain is complete.
2026-09-02 08:19:18 [540] fnbamd_cert_verify-Chain number:2
2026-09-02 08:19:18 [554] fnbamd_cert_verify-Following cert chain depth 0
2026-09-02 08:19:18 [113] __ztna_entry_lookup-vfid:-1 subject or issuer:C = CA, ST = California, L = Sunnyvale, O = Fortinet, OU = xx, CN = xxx.
2026-09-02 08:19:18 [627] fnbamd_cert_verify-Issuer found:  (SSL_DPI opt 0)
2026-09-02 08:19:18 [554] fnbamd_cert_verify-Following cert chain depth 1
2026-09-02 08:19:18 [1057] __cert_verify-peer_info.no_ocsp_query:0 cert->status:512.
2026-09-02 08:19:18 [768] fnbamd_cert_check_group_list-checking group with name 'Remote-Employee_peer'
2026-09-02 08:19:18 [581] __check_add_peer-check 'Remote-Employee_peer'
2026-09-02 08:19:18 [435] peer_san_subject_check-SAN: host_good 1; email_good 1; ip_good 1
2026-09-02 08:19:18 [454] peer_subject_cn_check-Cert subject 'C = CA, ST = BC, L = Burnaby, O = Fortinet, CN = xxxx'
2026-09-02 08:19:18 [348] __RDN_match-Checking 'O' val 'Fortinet' -- match.
2026-09-02 08:19:18 [378] __cert_subject_RDN_compare-Total matched RDNs in cert: 1
2026-09-02 08:19:18 [479] peer_subject_cn_check-Subject is good.
2026-09-02 08:19:18 [588] __check_add_peer-'Remote-Employee_peer' check ret:good
2026-09-02 08:19:18 [703] __peer_user_clear_unmatched-Clear all user(s) other than 'Remote-Employee_peer'
2026-09-02 08:19:18 [724] __peer_user_clear_unmatched-
2026-09-02 08:19:18 [200] __get_default_ocsp_ctx-def_ocsp_ctx=(nil), no_ocsp_query=0, ocsp_enabled=0
2026-09-02 08:19:18 [205] __get_default_ocsp_ctx-
2026-09-02 08:19:18 [187] fnbamd_ocsp_ctx_push_by_cert-
2026-09-02 08:19:18 [192] fnbamd_ocsp_ctx_push_by_cert-Get ocsp setting from cert
2026-09-02 08:19:18 [177] get_cert_ocsp_responder-OCSP responder not found
2026-09-02 08:19:18 [210] __get_default_ocsp_ctx-Failed to add OCSP from cert.
2026-09-02 08:19:18 [841] fnbamd_cert_check_group_list-Peer users
2026-09-02 08:19:18 [844] fnbamd_cert_check_group_list-    'Remote-Employee_peer' ('N/A','N/A','N/A')
2026-09-02 08:19:18 [1069] __cert_verify_do_next-req_id=84988985766079
2026-09-02 08:19:18 [109] __cert_chg_st- 'Validation' -> 'EC-EMS-Checking'
2026-09-02 08:19:18 [1052] fnbamd_obtain_cert_ems_info-EC UID 'xxxx', Fabric SN 'xxxxx', Fabric Tenant ID 'xxxxx'
2026-09-02 08:19:18 [1095] __cert_ec_ems_check-Check EC EMS status.
2026-09-02 08:19:18 [1097] __cert_ec_ems_check-Lookup result: 0
2026-09-02 08:19:18 [1104] __cert_ec_ems_check-Registered.
2026-09-02 08:19:18 [109] __cert_chg_st- 'EC-EMS-Checking' -> 'Done'
2026-09-02 08:19:18 [1163] __cert_done-req_id=84988985766079
2026-09-02 08:19:18 [1742] fnbamd_auth_session_done-Session done, id=84988985766079
2026-09-02 08:19:18 [1209] __fnbamd_cert_auth_run-Exit, req_id=84988985766079
2026-09-02 08:19:18 [1785] create_auth_cert_session-fnbamd_cert_auth_init returns 0, id=84988985766079
2026-09-02 08:19:18 [1698] auth_cert_success-id=84988985766079
2026-09-02 08:19:18 [1321] fnbamd_cert_auth_copy_cert_status-req_id=84988985766079
2026-09-02 08:19:18 [1329] fnbamd_cert_auth_copy_cert_status-Matched peer user 'Remote-Employee_peer'
2026-09-02 08:19:18 [1458] fnbamd_cert_auth_copy_cert_status-Cert st 210, req_id=84988985766079
2026-09-02 08:19:18 [356] fnbamd_comm_send_result-Sending result 0 (nid 672) for req 84988985766079, len=2776
2026-09-02 08:19:18 [360] fnbamd_comm_send_result-Failed send reply (2788, errno 101)
2026-09-02 08:19:18 [1567] destroy_auth_cert_session-id=84988985766079
2026-09-02 08:19:18 [2047] handle_child_rsp-Auth rsp 84988985766079, session not created, line 76
2026-09-02 08:19:18 [1293] fnbamd_cert_auth_uninit-req_id=84988985766079
2026-09-02 08:19:18 [1996] fnbamd_ldaps_destroy-
2026-09-02 08:19:18.161539 ike V=root:0:Remote-Employee:576: fnbam cert group matching failed
2026-09-02 08:19:18.161548 ike V=root:0:Remote-Employee:576: certificate validation failed
2026-09-02 08:19:18 [1667] fnbamd_rads_destroy-
2026-09-02 08:19:18.161574 ike V=root:0:Remote-Employee:576: certificate validation before eap failed
2026-09-02 08:19:18 [138] fnbamd_peer_ctx_free-Freeing peer ctx 'Remote-Employee_peer'
2026-09-02 08:19:18.161582 ike V=root:0:Remote-Employee:576: auth verify done
2026-09-02 08:19:18.161597 ike V=root:0:Remote-Employee:576: responder AUTH continuation
2026-09-02 08:19:18.161602 ike V=root:0:Remote-Employee:576: authentication failed

 

This looks somewhat similar to an 8.0.0 fnbamd/CA cache issue.

As a quick test, could you try:

diagnose test application fnbamd 5

This rebuilds the CA list used by fnbamd. After that, retry the VPN with:

set cert-trust-store ems

If it starts working after the rebuild, it would strongly point to a stale CA-cache / PKI matching issue in the same code path.