| In the IPsec VPN:
In IKEv1 Main Mode, six messages are exchanged between two peers to establish a Security Association (SA) for securing the subsequent IPsec traffic.
The initiator or responder cookie is a value used to identify a specific security association. In the context of IKEv1 Main Mode, the cookie value is negotiated during the first three messages (Messages 1 & 2) as part of the Security Association (SA) payload.
Scenario 1:
Initially, in the 1st message, the initiator cookie value will be set, and the responder cookie value will be 0.
If the FortiGate sees the 1st message where the responder cookie is already set to some value, then it will be flagged as a malformed responder cookie in the IKE debug logs.
Scenario 2:
If there are multiple Tunnels established between the two peers, then If the FortiGate receives another tunnel responder cookie for the current Tunnel then it will be categorized as a malformed cookie.
To fix this issue, define the local ID for the IPsec Tunnel settings in the FortiGate and use the same value in the remote site as well to identify the connection is for that particular Tunnel. Check for pre-shared key mismatch between the peers.
Scenario 3:
If the FortiGate flags the responder cookie as malformed, then check in the previous lines of the IKE debug logs if the FortiGate has deleted the SA for the same initiator and responder cookie pair; then it is an expected behavior.
2025-05-16 00:33:18.926755 ike V=Tenant:5:InMotion-DCCor~:1554235: schedule delete of IKE SA 357d81ac28f27fed/f92dabbe96179a14
2025-05-16 00:33:18.926765 ike V=Tenant:5:InMotion-DCCor~:1554235: scheduled delete of IKE SA 357d81ac28f27fed/f92dabbe96179a14 <----- initiator cookie/responder cookie
2025-05-16 00:33:18.926780 ike V=Tenant:5:InMotion-DCCor~: connection expiring due to phase1 down
2025-05-16 00:33:18.926787 ike V=Tenant:5:InMotion-DCCor~: going to be deleted And later:
Because the FortiGate has deleted this negotiation, and the peer sends more messages for this negotiation, the FortiGate marks it as a malformed responder cookie
2025-05-16 00:33:42.920920 ike V=Tenant:5: malformed responder cookie 357d81ac28f27fed/f92dabbe96179a14 from 80.86.44.41:500->193.111.183.61 70 exchange-type Identity Protection, drop
| 
