Issue on MacOS FortiClient IPSEC IKEv2 - packet too large

[doogz]

Now that mouthful is out of the way - I'm having an issue only on MacOS FortiClient (of course).  It was working, so I've no idea what has changed to suddenly have this behaviour.  Windows clients are working fine.  We're currently trying to migrate from SSL -> IPSec.

For the record, I've tried 7.2.12 and 7.4.3 and both exhibit the same issue.  I have a complicated auth of FortiClient -> DuoAuthProxy -> Radius -> LDAP.  That works fine with EAP-TTLS all the way through.  MacOS is 26.

So when I connect via the MacOS client, auth work, duo push works, but then SA retransmits happen and whammy Connection Timeout.  

I finally found an error I could work with from the fortigate:

2025-10-23 20:48:57.682116 ike V=root:0:ra-ops_1:343731: sent IKE msg (retransmit): xx.xx.xx.xx:4500->yy.yy.yy.yy:53479, len=9045, vrf=0, id=cbf670251e3656b1/ee13e00c20a25ee3:00000009, oif=6

Which correlates to these lines in the iked.log from the FortiClient

2025-10-24 16:22:51.625305+1300 0xac7d4a   Default     0x0                  43731  0    iked: [com.fortinet.forticlient:IPSec] fct_recvfromto: FCT ISAKMP
2025-10-24 16:22:51.625338+1300 0xac7d4a   Default     0x0                  43731  0    iked: [com.fortinet.forticlient:IPSec] fct_recvfromto: recv forwarded data length 111
2025-10-24 16:22:51.625400+1300 0xac7d4a   Default     0x0                  43731  0    iked: [com.fortinet.forticlient:IPSec] spi=0xc4c3cd1b6edcafa7: recv IKE_AUTH res 8 peer xx.xx.xx.xx:4500 local 10.26.99.237:4500, 73 bytes, policy 'vpn'
2025-10-24 16:22:51.625403+1300 0xac7d4a   Default     0x0                  43731  0    iked: [com.fortinet.forticlient:IPSec] ikev2_recv: ispi 0xc4c3cd1b6edcafa7 rspi 0xa37c722dca4fdfaf
2025-10-24 16:22:51.757349+1300 0xac7d4a   Default     0x0                  43731  0    iked: [com.fortinet.forticlient:IPSec] fct_recvfromto: FCT ISAKMP
2025-10-24 16:22:51.757380+1300 0xac7d4a   Default     0x0                  43731  0    iked: [com.fortinet.forticlient:IPSec] fct_recvfromto: recv forwarded data length 9083
2025-10-24 16:22:51.757382+1300 0xac7d4a   Default     0x0                  43731  0    iked: [com.fortinet.forticlient:IPSec] data length 9045 is greater than len 8192
2025-10-24 16:22:53.374755+1300 0xac7d4a   Default     0x0                  43731  0    iked: [com.fortinet.forticlient:IPSec] spi=0xc4c3cd1b6edcafa7: retransmit 1 IKE_AUTH req 9 peer xx.xx.xx.xx:4500 local 10.26.99.237:4500
2025-10-24 16:22:53.713642+1300 0xac7d4a   Default     0x0                  43731  0    iked: [com.fortinet.forticlient:IPSec] fct_recvfromto: FCT ISAKMP
2025-10-24 16:22:53.713678+1300 0xac7d4a   Default     0x0                  43731  0    iked: [com.fortinet.forticlient:IPSec] fct_recvfromto: recv forwarded data length 9083
2025-10-24 16:22:53.713681+1300 0xac7d4a   Default     0x0                  43731  0    iked: [com.fortinet.forticlient:IPSec] data length 9045 is greater than len 8192
2025-10-24 16:22:57.376874+1300 0xac7d4a   Default     0x0                  43731  0    iked: [com.fortinet.forticlient:IPSec] spi=0xc4c3cd1b6edcafa7: retransmit 2 IKE_AUTH req 9 peer xx.xx.xx.xx:4500 local 10.26.99.237:4500
2025-10-24 16:22:57.715004+1300 0xac7d4a   Default     0x0                  43731  0    iked: [com.fortinet.forticlient:IPSec] fct_recvfromto: FCT ISAKMP
2025-10-24 16:22:57.715047+1300 0xac7d4a   Default     0x0                  43731  0    iked: [com.fortinet.forticlient:IPSec] fct_recvfromto: recv forwarded data length 9083
2025-10-24 16:22:57.715049+1300 0xac7d4a   Default     0x0                  43731  0    iked: [com.fortinet.forticlient:IPSec] data length 9045 is greater than len 8192
2025-10-24 16:23:03.378269+1300 0xac7d4a   Default     0x0                  43731  0    iked: [com.fortinet.forticlient:IPSec] spi=0xc4c3cd1b6edcafa7: retransmit 3 IKE_AUTH req 9 peer xx.xx.xx.xx:4500 local 10.26.99.237:4500
2025-10-24 16:23:03.716198+1300 0xac7d4a   Default     0x0                  43731  0    iked: [com.fortinet.forticlient:IPSec] fct_recvfromto: FCT ISAKMP
2025-10-24 16:23:03.716208+1300 0xac7d4a   Default     0x0                  43731  0    iked: [com.fortinet.forticlient:IPSec] fct_recvfromto: recv forwarded data length 9083
2025-10-24 16:23:03.716209+1300 0xac7d4a   Default     0x0                  43731  0    iked: [com.fortinet.forticlient:IPSec] data length 9045 is greater than len 8192
2025-10-24 16:23:11.379573+1300 0xac7d4a   Default     0x0                  43731  0    iked: [com.fortinet.forticlient:IPSec] ikev2_msg_retransmit_timeout: retransmit limit reached for req 9
2025-10-24 16:23:11.379695+1300 0xac7d4a   Default     0x0                  43731  0    iked: [com.fortinet.forticlient:IPSec] FCT send error. server addr: xx.xx.xx.xx, error code: -306
2025-10-24 16:23:11.379696+1300 0xac7d4a   Default     0x0                  43731  0    iked: [com.fortinet.forticlient:IPSec] ipseccon_send: 88
2025-10-24 16:23:11.379744+1300 0xac7d4a   Default     0x0                  43731  0    iked: [com.fortinet.forticlient:IPSec] spi=0xc4c3cd1b6edcafa7: sa_free: retransmit limit reached
2025-10-24 16:23:11.379754+1300 0xac7d4a   Default     0x0                  43731  0    iked: [com.fortinet.forticlient:IPSec] config_free_proposals: free 0xc388041e0

So the client is basically saying the packet is exceeding the max MTU - makes sense.  So I read up on ike fragmentation which looks like it won't actually do anything, but I tried it anyways - no change.

This is what my phase1-interface currently looks like:

    edit "ra-ops"
        set type dynamic
        set interface "wan2"
        set ike-version 2
        set peertype one
        set net-device disable
        set mode-cfg enable
        set proposal aes128gcm-prfsha256 aes256gcm-prfsha256
        set dpd on-idle
        set dhgrp 14 19
        set eap enable
        set eap-identity send-request
        set authusrgrp "Operations"
        set peerid "operations"
        set assign-ip-from name
        set ipv4-split-include "ra-netops-split-tunnel"
        set ipv4-name "SSLVPN_TUNNEL_ADDR1"
        set save-password enable
        set client-keep-alive enable
        set psksecret ENC
        set dpd-retryinterval 60
    next

Fortigate is 7.4.3.

net (ra-ops) # get |  grep fragmentation
fragmentation       : enable 
ip-fragmentation    : post-encapsulation 
fragmentation-mtu   : 1200

Help?  TIA!

[AEK]

You said it was working.

So was there any system/library update just before the failure?

[doogz]

There's been a few updates.  Unfortunately the Mac is managed by MDM, and I get forced updates.    It was sequoia, now Tahoe, but I honestly don't remember when it broke, there has been a lot of pain and suffering getting this going due to EAP, Duo Proxy, Radius, FreeIPA etc.

My problem is now that FortiClient works on the Mac for SSL to one particular site.  But not to another, so it's almost like it can't create the firewall rules or something (socketfilterfw etc).

And IPSec doesn't work at all, no matter what I try.  I've tried, pre encapsulation, post encapsulation, MTU now seem to be fine on pre encapsultion, but I'm still getting IKE timeouts.  

The FG I'm testing with is behind CloudFlare.  So I put a secondary IP on the firewall that wasn't - same issue, IKE retransmit.

Opened a ticket with support, but they've been deathly silent since the pcaps.

[AEK]

Officially Tahoe is not supported. That's probably why.

https://docs.fortinet.com/document/forticlient/7.4.4/macos-release-notes/471180/product-integration-...