Technical Tip: IPSec VPN diagnostics – Deep analysis

[nsubramanian]

Description
This article describes how to debug IPSec VPN connectivity issues.

Solution
If the VPN fails to connect, check the following:
- Ensure that the pre-shared keys match exactly (see The pre-shared key does not match (PSK mismatch error). below).
- Ensure that both ends use the same P1 and P2 proposal settings (The SA proposals do not match (SA proposal mismatch). below).

ike 0:TRX:322: PSK auth failed: probable pre-shared key mismatch ike Negotiate SA Error:

The SA proposals do not match (SA proposal mismatch).

- Ensure that inbound and outbound traffic are allowed for all necessary network services, especially if services such as DNS or DHCP are having problems.

- Check that a static route has been configured properly to allow routing of VPN traffic.

- Ensure that the FortiGate unit is in NAT/Route mode, rather than Transparent.

- Check the NAT settings, enabling NAT traversal in the Phase 1 configuration while disabling NAT in the security policy. Pin the PAT/NAT session table, or use some of kind of NAT-T keepalive to avoid the expiration of the PAT/NAT translation.

- Ensure that both ends of the VPN tunnel are using Main mode, unless multiple dial-up tunnels are being used.   

- If multiple dial-up IPsec VPNs are configured, ensure that the peer ID is configured properly on the FortiGate and that clients have specified the correct local ID. Furthermore, in circumstances where multiple remote dialup VPN tunnels exist, each tunnel must have a peer ID set.

- If FortiClient is used, ensure that the version is compatible with the FortiGate firmware by reading the FortiOS Release Notes.

- If Perfect Forward Secrecy (PFS) is used, ensure that it is used on both peers. Use the diagnose VPN tunnel list command to troubleshoot this.

- Ensure that the Quick Mode selectors are correctly configured. If part of the setup currently uses firewall addresses or address groups, try changing it to either specify the IP addresses or use an expanded address range. This is especially useful if the remote endpoint is not a FortiGate.

- If XAUTH is enabled, ensure that the settings are the same for both ends, and that the FortiGate unit is set to Enable as Server.

- Check IPsec VPN Maximum Transmission Unit (MTU) size. A 1500 byte MTU is going to exceed the overhead of the ESP-header, including the additional ip_header,etc. Use the diagnose vpn tunnel list command to troubleshoot this.

- If the FortiGate unit is behind a NAT unit, such as a router, configure port forwarding for UDP ports 500 and 4500.

- Remove any Phase 1 or Phase 2 configurations that are not in use. If a duplicate instance of the VPN tunnel appears on the IPsec Monitor, reboot the FortiGate unit to try and clear the entry.

If the VPN tunnel goes down often, check the Phase 2 settings and either increase the Keylife value or enable Autokey Keep Alive.

If the ping or traceroute fail, it indicates a connection problem between the two ends of the tunnel.
This may or may not indicate problems with the VPN tunnel.
Confirm this by going to Monitor -> IPsec Monitor where it will be possible to see the connection.

A green arrow means the tunnel is up and currently processing traffic.
A red arrow means the tunnel is not processing traffic, and this VPN connection has a problem.

The ESP packet invalid error is due to an encryption key mismatch after a VPN tunnel has been established.
When an IPsec VPN tunnel is up, but traffic is not able to pass through the tunnel, Wireshark (or an equivalent program) can be used to determine whether there is an encryption mismatch.
A mismatch could occur for many reasons, one of the most common is the instability of an ISP link (ADSL, Cable), or it could effectively be any unit in the physical connection.