Troubleshooting Tip: How to resolve IPsec VPN issue 'failed to compute DH shared secret' with StrongSwan

Description

This article describes an issue where the IPsec connection fails to re-establish because StrongSwan sends incorrect data in the Key Exchange payload, causing DH shared secret generation to fail.

Scope

FortiGate, StrongSwan.

Solution

An intermittent issue with the dial-up IPsec tunnel between the FortiGate and StrongSwan causes the tunnel to go down. During renegotiation, StrongSwan sends incorrect data in the Key Exchange payload, which causes DH shared secret computation to fail on the FortiGate side, as shown in the debug output below.

The StrongSwan side shows the error (ECDH public value is malformed):

Nov 18 virtual-machine charon: 05[ENC] found payload of type NONCE
Nov 18 virtual-machine charon: 05[ENC] parsed IKE_SA_INIT request 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) N(REDIR_SUP) ]
Nov 18 virtual-machine charon: 05[IKE] 10.61.100.200 is initiating an IKE_SA
Nov 18 virtual-machine charon: 05[IKE] IKE_SA (unnamed)[2] state change: CREATED => CONNECTING
Nov 18 virtual-machine charon: 05[LIB] ECDH public value is malformed

FortiGate Side, failed to compute DH shared secret:

2025-11-18 13:07:38.752864 ike 3: IKEv2 exchange=SA_INIT id=0e9d4ea4007bd864/0000000000000000 len=256
2025-11-18 13:07:38.752903 ike 3:0e9d4ea4007bd864/0000000000000000:1342: responder received SA_INIT msg
2025-11-18 13:07:38.752930 ike 3:0e9d4ea4007bd864/0000000000000000:1342: received notify type NAT_DETECTION_SOURCE_IP
2025-11-18 13:07:38.752956 ike 3:0e9d4ea4007bd864/0000000000000000:1342: received notify type NAT_DETECTION_DESTINATION_IP
2025-11-18 13:07:38.752981 ike 3:0e9d4ea4007bd864/0000000000000000:1342: received notify type FRAGMENTATION_SUPPORTED
2025-11-18 13:07:38.753006 ike 3:0e9d4ea4007bd864/0000000000000000:1342: received notify type 16406
2025-11-18 13:07:38.753031 ike 3:0e9d4ea4007bd864/0000000000000000:1342: ignoring unauthenticated notify payload (16406)
2025-11-18 13:07:38.753063 ike 3:0e9d4ea4007bd864/0000000000000000:1342: incoming proposal:
2025-11-18 13:07:38.753088 ike 3:0e9d4ea4007bd864/0000000000000000:1342: proposal id = 1:
2025-11-18 13:07:38.753103 ike 3:0e9d4ea4007bd864/0000000000000000:1342: protocol = IKEv2:
2025-11-18 13:07:38.753117 ike 3:0e9d4ea4007bd864/0000000000000000:1342: encapsulation = IKEv2/none
2025-11-18 13:07:38.753133 ike 3:0e9d4ea4007bd864/0000000000000000:1342: type=ENCR, val=AES_CBC (key_len = 128)
2025-11-18 13:07:38.753149 ike 3:0e9d4ea4007bd864/0000000000000000:1342: type=INTEGR, val=AUTH_HMAC_SHA2_256_128
2025-11-18 13:07:38.753165 ike 3:0e9d4ea4007bd864/0000000000000000:1342: type=PRF, val=PRF_HMAC_SHA2_256
2025-11-18 13:07:38.753181 ike 3:0e9d4ea4007bd864/0000000000000000:1342: type=DH_GROUP, val=ECP256.
2025-11-18 13:07:38.753221 ike 3:0e9d4ea4007bd864/0000000000000000:1342: matched proposal id 1
2025-11-18 13:07:38.753246 ike 3:0e9d4ea4007bd864/0000000000000000:1342: proposal id = 1:
2025-11-18 13:07:38.753261 ike 3:0e9d4ea4007bd864/0000000000000000:1342: protocol = IKEv2:
2025-11-18 13:07:38.753275 ike 3:0e9d4ea4007bd864/0000000000000000:1342: encapsulation = IKEv2/none
2025-11-18 13:07:38.753292 ike 3:0e9d4ea4007bd864/0000000000000000:1342: type=ENCR, val=AES_CBC (key_len = 128)
2025-11-18 13:07:38.753307 ike 3:0e9d4ea4007bd864/0000000000000000:1342: type=INTEGR, val=AUTH_HMAC_SHA2_256_128
2025-11-18 13:07:38.753323 ike 3:0e9d4ea4007bd864/0000000000000000:1342: type=PRF, val=PRF_HMAC_SHA2_256
2025-11-18 13:07:38.753337 ike 3:0e9d4ea4007bd864/0000000000000000:1342: type=DH_GROUP, val=ECP256.
2025-11-18 13:07:38.753353 ike 3:0e9d4ea4007bd864/0000000000000000:1342: lifetime=86400
2025-11-18 13:07:38.753376 ike 3:0e9d4ea4007bd864/0000000000000000:1342: SA proposal chosen, matched gateway MGW-p1
2025-11-18 13:07:38.753426 ike 3:MGW-p1: created connection: 0x8a2a970 47 10.10.152.208->10.10.152.77:500.
2025-11-18 13:07:38.753466 ike 3:MGW-p1:1342: processing notify type NAT_DETECTION_SOURCE_IP
2025-11-18 13:07:38.753522 ike 3:MGW-p1:1342: processing NAT-D payload
2025-11-18 13:07:38.753545 ike 3:MGW-p1:1342: NAT detected: PEER
2025-11-18 13:07:38.753561 ike 3:MGW-p1:1342: process NAT-D
2025-11-18 13:07:38.753576 ike 3:MGW-p1:1342: processing notify type NAT_DETECTION_DESTINATION_IP
2025-11-18 13:07:38.753617 ike 3:MGW-p1:1342: processing NAT-D payload
2025-11-18 13:07:38.753636 ike 3:MGW-p1:1342: NAT detected: ME PEER
2025-11-18 13:07:38.753652 ike 3:MGW-p1:1342: process NAT-D
2025-11-18 13:07:38.753666 ike 3:MGW-p1:1342: processing notify type FRAGMENTATION_SUPPORTED
2025-11-18 13:07:38.753747 ike 3:MGW-p1:1342: responder preparing SA_INIT msg
2025-11-18 13:07:38.754251 ike 3:MGW-p1:1342: failed to compute DH shared secret
2025-11-18 13:07:38.754279 ike 3:MGW-p1:1342: schedule delete of IKE SA 0e9d4ea4007bd864/eeefb9685aab91fd
2025-11-18 13:07:38.754339 ike 3:MGW-p1:1342: scheduled delete of IKE SA 0e9d4ea4007bd864/eeefb9685aab91fd
2025-11-18 13:07:38.754422 ike 3:MGW-p1: connection expiring due to phase1 down
2025-11-18 13:07:38.754441 ike 3:MGW-p1: deleting
2025-11-18 13:07:38.754459 ike 3:MGW-p1: deleted

Workaround:

To avoid this issue, configure DH groups other than the NIST Elliptic Curve groups. Using non-NIST Elliptic Curve DH groups should resolve the issue.
Note: The NIST Elliptic Curve DH groups are 19 20 21 25 26.