Troubleshooting Tip: IPsec site-to-site tunnel between FortiGate and Barracuda firewall not getting established due to 'malformed message'

In the following IPsec site-to-site tunnel setup, FortiGate is the IKE initiator while the Barracuda firewall is the responder.

Topology

IPSec tunnel Name: CRR-T2.

IKE version: IKEv2.

Take debugs on the FortiGate firewall using the following commands:

diagnose vpn ike log-filter clear

diagnose vpn ike log-filter name <Phase1 name> 

diagnose debug app ike -1
diagnose debug enable

Notes:

diagnose vpn ike log-filter name <Phase1 name> can be replaced by diagnose vpn ike log-filter dst-addr4 69.75.89.129.

Starting from v7.4.1, the 'diagnose vpn ike log-filter' command has been changed to 'diagnose vpn ike log filter', and the 'diagnose vpn ike log-filter dst-addr4' command has been changed to 'diagnose vpn ike log filter rem-addr4'. 

Useful IKE Error Logs:

ike V=root:0:CRR-T2:34862: sent IKE msg (AUTH): 201.96.51.145:500->69.75.89.129:500, len=240, vrf=0, id=2f42ce8c81ac3354/5ec9d8622a374caa:00000001, oif=8
ike V=root:0: comes 69.75.89.129:500->201.96.51.145:500,ifindex=8,vrf=0,len=80....
ike V=root:0: IKEv2 exchange=INFORMATIONAL id=2f42ce8c81ac3354/5ec9d8622a374caa len=80
ike 0: in 2F42CE8C81AC33545EC9D8622A374CAA2E202500000000000000005000000034

7EB1FB50C80A427A360B04BAAB6C5C

AD125DD4548D047A38FE4ABCB57B7FFCCCA2C660FF89C2373F82E9324A04655
3FF
ike 0:CRR-T2:34862: dec 2F42CE8C81AC33545EC9D8622A374CAA2E202500000000000000002000000004
ike V=root:0:CRR-T2:34862: initiator received AUTH msg
ike V=root:0:CRR-T2:34862: response message_id 0, expected 1
ike V=root:0:CRR-T2:34862: malformed message -------> Reason for failure.
ike V=root:0:CRR-T2:34862: schedule delete of IKE SA 2f42ce8c81ac3354/5ec9d8622a374caa
ike V=root:0:CRR-T2:34862: scheduled delete of IKE SA 2f42ce8c81ac3354/5ec9d8622a374caa
ike V=root:0:CRR-T2: connection expiring due to phase1 down
ike V=root:0:CRR-T2: going to be deleted

Solution:

Change the following settings in the IPsec configuration in Barracuda.

1. Set Yes to 'Restart SA on Close':

2. Disable IKE Reauthentication by unchecking the box:

Refer to the following document for more IPsec tunnel configuration information on the Barracuda firewall:
How to Configure a Site-to-Site IPsec IKEv2 VPN Tunnel in Barracuda