IPsec VPN - pkt-Size mismatch due to LDAP Group Overflow

[mycoolusername]

Hello everyone,
we're currently experiencing issues with some users trying to connect to our IPsec VPN. The FortiClient times out during connection, even though the configuration and password are correct.
After investigation, I found that the root cause seems to be related to users who are members of a large number of LDAP groups. For those users, the FortiGate fails to process the authentication request and throws the following error:

diagnose debug application ike -1
diagnose debug application authd 60
diagnose debug application samld -1
diagnose debug application fnbamd -1
diagnose debug application eap_proxy -1
diagnose debug console timestamp enable
diagnose debug enable

2025-09-19 11:57:45 1758275865.817240: 2025-09-19 11:57:45 EAP-TTLS/PAP: Correct user password
2025-09-19 11:57:45 1758275865.817297: 2025-09-19 11:57:45 EAP-TTLS: PHASE2_START -> SUCCESS
2025-09-19 11:57:45 1758275865.817396: 2025-09-19 11:57:45 EAP-TTLS: Derived key - hexdump(len=64):
2025-09-19 11:57:45 1758275865.817513: 2025-09-19 11:57:45 EAP: EAP entering state SELECT_ACTION
2025-09-19 11:57:45 1758275865.817569: 2025-09-19 11:57:45 EAP: getDecision: method succeeded -> SUCCESS
2025-09-19 11:57:45 1758275865.817625: 2025-09-19 11:57:45 EAP: EAP entering state SUCCESS
2025-09-19 11:57:45 1758275865.817681: 2025-09-19 11:57:45 EAP: Building EAP-Success (id=218)
2025-09-19 11:57:45 1758275865.817741: 2025-09-19 11:57:45 CTRL-EVENT-EAP-SUCCESS 00:00:00:00:00:00
2025-09-19 11:57:45 1758275865.817853: 2025-09-19 11:57:45 RADIUS SRV: Reply to 127.0.0.1:2480
2025-09-19 11:57:45 2025-09-19 11:57:45 [831] __rad_rxtx-fd 11, state 2(Challenged)
2025-09-19 11:57:45 RADIUS SRV: Removing completed session 0xf71 after timeout
2025-09-19 11:57:45 [833] __rad_rxtx-Stop rad conn timer.
2025-09-19 11:57:45 [883] __rad_rxtx-
2025-09-19 11:57:45 [432] __rad_udp_recv-Recved 8192 bytes. Buf sz 8192
2025-09-19 11:57:45 [1214] fnbamd_rad_validate_pkt-Size mismatch rx=8192 pkt=8492
2025-09-19 11:57:45 [908] __rad_rxtx-Error validating radius rsp
2025-09-19 11:57:45 [1031] __rad_error-Ret 5, st = 2.
2025-09-19 11:57:45 [301] fnbamd_radius_get_next_auth_prot-Next auth prot EAP
2025-09-19 11:57:45 [1080] __rad_error-
2025-09-19 11:57:45 [307] __rad_udp_close-closed.
2025-09-19 11:57:45 [967] __rad_conn_stop-Stop rad conn timer.
2025-09-19 11:57:45 [1301] fnbamd_rad_process-Result from radius svr 'EAP_PROXY' is 5, req 74036713144255
2025-09-19 11:57:45 [1503] fnbamd_rad_process-Challenged: 0, FTK_Challenge: 0, CHG_PWD: 0, Invaid_Digest: 0, No_Message_Authenticator_Attr: 0, State_Len: 0
2025-09-19 11:57:45 [2804] fnbamd_rad_result-Error (5) for req 74036713144255
2025-09-19 11:57:45 [133] fnbamd_comm_send_result-Not enough buffer for EAP message
2025-09-19 11:57:45 [239] fnbamd_comm_send_result-Sending result 5 (nid 0) for req 74036713144255, len=6688
2025-09-19 11:57:45 [600] destroy_auth_session-delete session 74036713144255
2025-09-19 11:57:45 [1342] fnbamd_rad_stop-
2025-09-19 11:57:45.819054 ike V=root:0:VPN_User:14365 EAP 74036713144255 result FNBAM_ERROR
2025-09-19 11:57:45.819084 ike V=root:0:VPN_User: EAP failed for user "user"

Environment:
FortiOS: 7.4.8
FortiClient: 7.4.3.1790

Configuration Overview:

config user group
    edit "LDAP_VPN_USER"
        set member "contoso-Domain-Controller"
        config match
            edit 1
                set server-name "contoso-Domain-Controller"
                set group-name "CN=GG_VPN_USER,OU=GG,OU=04_Groups,DC=contoso,DC=intern"
            next
        end
    next
end

config user ldap
    edit "contoso-Domain-Controller"
        set server "ip 1"
        set secondary-server "ip 2"
        set cnid "sAMAccountName"
        set dn "dc=contoso,dc=intern"
        set type regular
        set username "contoso\\username"
        set password ENC x
    next
end

config vpn ipsec phase1-interface
    edit "VPN_User"
        set eap enable
        set eap-identity send-request
        set authusrgrp "LDAP_VPN_USER"
    next
end

I already tried the steps in this Fortinet KB article, but unfortunately, reapplying the certificate settings didn’t resolve the issue.
https://community.fortinet.com/t5/FortiGate/Troubleshooting-Tip-How-to-resolve-the-error-Not-enough-... 

I also found a bug fix mentioned in FortiOS 7.6.1:
Bug ID 1023871
IPSec IKEv2 with SAML cannot match the Entra ID group during EAP due to a buffer size issue.

However, we are not using SAML or Entra ID in our setup.

My questions:
Is this a known bug affecting LDAP-based EAP authentication?
Is there a workaround to increase the buffer size or reduce the group data being sent?
Will Bug ID 1023871 potentially fix this issue even though it mentions SAML?

Any insights or suggestions would be greatly appreciated!

[Anthony_E]

Hello,

Thank you for using the Community Forum. I will seek to get you an answer or help. We will reply to this thread with an update as soon as possible.

Thanks,

[Anthony_E]

Hello,

We are still looking for someone to help you.

We will come back to you ASAP.

Thanks,

[kaman]

Hi mycoolusername,

Bug ID 1023871 is for IPSec IKEv2 with SAML - fail to match Entra ID group during EAP on 7.2.8 (7.4.3 works) not for the ldap. I couldn't find any bug regarding issue with the LDAP query size.

In general, LDAP can't talk to EAP Protocol which is responsible for carrying authentication data for IKEv2.

That's not an IKEv2 specific issue, it affects anything that uses EAP.

Since there is no defined RFC regarding this implementation for IKEv2 FortiOS uses a proprietary implementation to achieve LDAP auth for Dialup IKEv2. This is the reason why 3rd party clients will not be able to connect.

In this case you can use IKEv1 or check the guide below.

https://docs.fortinet.com/document/fortigate/7.6.4/administration-guide/442351/ldap-authentication-w...

https://docs.fortinet.com/document/forticlient/7.4.0/new-features/907253/eap-ttls-support-for-ipsec-...

You will need to use FortiClient 7.4.3 and newer version and you will need to modify XML file, please check the guide below:

https://community.fortinet.com/t5/FortiGate/Technical-Tip-How-to-enable-EAP-TTLS-for-IPSec-IKEv2-tun...

If still having the same issue please collect the same debug logs and share with us.

diag debug reset
diagnose debug cons time en
diag debug application fnbamd -1
diagnose debug app eap_proxy -1
diag vpn ike log filter rem-addr4 <RemoteClientpublicIp>
diag debug app ike -1
diagnose debug enable

Regards,
Aman

[mycoolusername]

Hello kaman,

thanks for the information. The VPN was already setup using the guides from the links in your post. 
The issue still exists and here is the full debug log using your debug commands.
When removing some amount of groups from the user in the Active Directory, the user is able to connect to the VPN. Re-adding some amount of groups and he fails to connect again.

diag debug reset
diagnose debug cons time en
diag debug application fnbamd -1
diagnose debug app eap_proxy -1
diag vpn ike log filter rem-addr4 91.186.6x.xxx
diag debug app ike -1
diagnose debug enable

I cant paste the full log in here, since the forum post only allows for max 150k characters.
Do you have a pastebin like services to post logs?