Troubleshooting Tip: IPsec site-to-site tunnel between FortiGate and Barracuda firewall not getting established due to 'malformed message'

[Rajan_kohli]

Description	This article explains how to fix an IPsec tunnel problem due to a malformed AUTH message between a Barracuda firewall and a FortiGate.
Scope	IPsec, FortiGate, Barracuda firewall.
Solution	In the following IPsec site-to-site tunnel setup, FortiGate is the IKE initiator while Barracuda firewall is the responder. Topology IPSec tunnel Name: CRR-T2. IKE version: IKEv2. Take debugs on the FortiGate firewall using the following commands: di vpn ike log-filter clear di vpn ike log-filter name <Phase1 name>  diag debug app ike -1 diag debug enable Note: Starting from v7.4.1, the 'di vpn ike log-filter' command has been changed to 'di vpn ike log filter'.  Useful IKE Error Logs: ike V=root:0:CRR-T2:34862: sent IKE msg (AUTH): 201.96.51.145:500->69.75.89.129:500, len=240, vrf=0, id=2f42ce8c81ac3354/5ec9d8622a374caa:00000001, oif=8 ike V=root:0: comes 69.75.89.129:500->201.96.51.145:500,ifindex=8,vrf=0,len=80.... ike V=root:0: IKEv2 exchange=INFORMATIONAL id=2f42ce8c81ac3354/5ec9d8622a374caa len=80 ike 0: in 2F42CE8C81AC33545EC9D8622A374CAA2E202500000000000000005000000034 7EB1FB50C80A427A360B04BAAB6C5C AD125DD4548D047A38FE4ABCB57B7FFCCCA2C660FF89C2373F82E9324A04655 3FF ike 0:CRR-T2:34862: dec 2F42CE8C81AC33545EC9D8622A374CAA2E202500000000000000002000000004 ike V=root:0:CRR-T2:34862: initiator received AUTH msg ike V=root:0:CRR-T2:34862: response message_id 0, expected 1 ike V=root:0:CRR-T2:34862: malformed message -------> reason for failure ike V=root:0:CRR-T2:34862: schedule delete of IKE SA 2f42ce8c81ac3354/5ec9d8622a374caa ike V=root:0:CRR-T2:34862: scheduled delete of IKE SA 2f42ce8c81ac3354/5ec9d8622a374caa ike V=root:0:CRR-T2: connection expiring due to phase1 down ike V=root:0:CRR-T2: going to be deleted Solution: Change the following settings in the IPsec configuration in Barracuda. Set Yes to 'Restart SA on Close': Disable IKE Reauthentication by unchecking the box: Refer to the following document for more IPsec tunnel configuration information on the Barracuda firewall: How to Configure a Site-to-Site IPsec IKEv2 VPN Tunnel in Barracuda