FortiGate
FortiGate Next Generation Firewall utilizes purpose-built security processors and threat intelligence security services from FortiGuard labs to deliver top-rated protection and high performance, including encrypted traffic.
VinayHM
Staff
Staff
Article Id 271122
Description This article describes an issue where an IPsec tunnel between FortiGate and other vendors fails with a malformed message.
Scope Any supported version of FortiGate with other vendors.
Solution

Assume the following topology:

FortiGate <-> IPsec <-> Other vendors.

 

In this issue, the IPsec tunnel does not reach the 'up' status.

 

In phase 2, some vendors are unable to call a particular network or specify the network. The only option is to call 0.0.0.0/0.

This means that, on the FortiGate, the option is available to call specific networks instead of all networks.

 

However, when other vendors' phase2 has 0.0.0.0/0 and FortiGate has a specific network, the tunnel will not come up.

 

IKE Logs:

 

2023-08-29 14:23:41.124335 ike 3: comes x.x.x.x:500->x.x.x.x:500,ifindex=209....
2023-08-29 14:23:41.124377 ike 3: IKEv2 exchange=AUTH_RESPONSE id=459435ff1ac22700/2d5947a0f19e

3441:00000001 len=96
2023-08-29 14:23:41.124399 ike 3: in 459435FF1AC227002D5947A0F19E34412E2023200000000100000060290000443F6810DD6822B25C48F13FD5E0172

AD8569452CF599EEF7EE0805C6D54AF44D41
F7A665CD7413F4FD09D32F81891314C522D837D22BD7EDAB9898A6928EC94D6
2023-08-29 14:23:41.124445 ike 3:Device1:56672822: dec 459435FF1AC227002D5947A0F19E34412E2023200000000100000028290000040000000800000026
2023-08-29 14:23:41.124467 ike 3:Device1:56672822: initiator received AUTH msg
2023-08-29 14:23:41.124486 ike 3:Device1:56672822: malformed message --------> Error notification.
2023-08-29 14:23:41.124508 ike 3:Device1:56672822: schedule delete of IKE SA 459435ff1ac22700/2d5947a0f19e3441
2023-08-29 14:23:41.124531 ike 3:Device1:56672822: scheduled delete of IKE SA 459435ff1ac22700/2d5947a0f19e3441
2023-08-29 14:23:41.124605 ike 3:Device1: connection expiring due to phase1 down
2023-08-29 14:23:41.124625 ike 3:Device1: deleting
2023-08-29 14:23:41.124646 ike 3:Device1: deleted
2023-08-29 14:23:41.124665 ike 3:Device1: schedule auto-negotiate
2023-08-29 14:23:41.948269 ike shrank heap by 159744 bytes
2023-08-29 14:23:42.119686 ike 3:Device1:Device1: chosen to populate IKE_SA traffic-selectors
2023-08-29 14:23:42.119736 ike 3:Device1: no suitable IKE_SA, queuing CHILD_SA request and initiating IKE_SA negotiation
2023-08-29 14:23:42.119917 ike 3:Device1:56672829: out  

 

To fix this issue, call 0.0.0.0/0 on the FortiGate phase2 instead of specific networks.

 

In another case,  deleting the localid configured in IPsec phase 1 settings has resolved the issue. Make sure it is unset on the FortiGate under config vpn ipsec phase1-interface as shown below.

 

FortiGate (IPSEC) # config vpn ipsec phase1-interface
FortiGate (IPSEC) # edit "IPSEC"
FortiGate (IPSEC) # unset localid

FortiGate (IPSEC) # end