Troubleshooting Tip: IPsec VPN tunnel issue 'error, payload not encrypted'

This example setup is verified between a VM FortiGate and Forcepoint.

Collect the IKE debug and verify the error using below commands :

diagnose vpn ike log filter dst-addr4 <VPN remote IP address>
diagnose debug application ike -1
diagnose debug console timestamp enable
diagnose debug enable

ike V=root:0: comes 116.50.59.200:4500->10.229.224.97:4500,ifindex=4,vrf=0,len=40....
ike V=root:0: IKEv2 exchange=AUTH_RESPONSE id=39e1e703a202a2bb/2b63d2ee5311f7d6:00000001 len=36
ike 0: in 39E1E703A202A2BB2B63D2EE5311F7D6292023200000000100000024000000080000000E
ike V=root:0:Forcepoint: HA state master(2)
ike V=root:0:Forcepoint:13300: error, payload not encrypted    <- Plain text received.

It is clear from the IKE log that the two VPN peers are not able to complete phase1 negotiation (phase1 is down).

The AUTH_RESPONSE packet should be encrypted, but when taking a packet capture, the packet is not encrypted.

Change the IKE version to V1. An informational message will populate after the 1st message of ISAKMP.

Fix for the issue:

• Verify the tunnel configuration.
• When the local ID is mismatched, this error is seen.
• Local ID is expected to be configured for most of the cloud-deployed devices.

Related article:

Troubleshooting Tip: IPsec VPN tunnels