Troubleshooting Tip: IPsec VPNs tunnels

sgiannogloudis · ‎10-25-2019

Description

This article describes techniques on how to identify, debug, and troubleshoot issues with IPsec VPN tunnels.

Scope

FortiGate v7.2 and above.

Solution

Identification. As the first action, check the reachability of the destination according to the routing table with the following command:

get router info routing-table detail <destination-IP>

If the destination is reachable by multiple tunnels, isolate the problematic tunnel:

Enter the VDOM (if applicable) where the VPN is configured and type the command:

get vpn ipsec tunnel summary
'to10.174.0.182' 10.174.0.182:0 selectors(total,up): 1/1 rx(pkt,err): 1921/0 tx(pkt,err): 69/2
'to10.189.0.182' 10.189.0.182:0 selectors(total,up): 1/0 rx(pkt,err): 0/0 tx(pkt,err): 0/0

On the particular output, two VPN tunnels, to10.174.0.182 & to10.189.0.182 are visible. The second VPN tunnel on the list has its selectors in a down state so the focus will be on that tunnel.

Phase 1 checks. After the problematic tunnel has been identified, it will be possible to understand the status of Phase 1. To do so, type the below command:

diagnose vpn ike gateway list name to10.189.0.182

vd: root/0
name: to10.189.0.182
version: 1
interface: port9 10
addr: 10.189.0.31:500 -> 10.189.0.182:500
created: 15s ago
IKE SA: created 1/1
IPsec SA: created 0/0

id/spi: 19576 a83334b3c66f871b/0000000000000000
direction: responder
status: connecting, state 3, started 15s ago

The important field from this particular command is status. The status field has a discrete output that can be connected or established.

Established means Phase 1 is up and running.
Connecting means Phase 1 is down.

If Phase 1 is down, additional checks must be performed to identify the reason.

Ensure bidirectional connectivity between the VPN gateways (typically, this is the IP address on the WAN interface).

Try to traceroute (or ping) towards the VPN peer, in this example, use the commands:

execute traceroute-options source 10.189.0.31

execute traceroute 10.189.0.182

Confirm that IKE traffic for port 500 or 4500 is not blocked somewhere along the path.

To do so, perform a packet sniffer:

diag sniffer packet any "host 10.189.0.182 and (port 500 or port 4500)" 4 0 l
interfaces=[any]
filters=[host 10.189.0.182 and (port 500 or port 4500)]

Note:

If nattraversal is enabled under phase1 and FortiGate is behind the NAT, sniff traffic with 'udp port 4500'. Otherwise, sniff traffic with the filter 'udp port 500'.

IKE debugging: If both of the above checks are successful, start debugging the IKE protocol to check for possible configuration mismatches between the peers:

diagnose vpn ike log-filter dst-addr4 10.189.0.182
diagnose debug application ike -1
diagnose debug console timestamp enable
diagnose debug enable

For v7.4.0 and above, there is a slight change in command as below:

diagnose vpn ike log filter rem-addr4 10.189.0.182
diagnose debug application ike -1
diagnose debug console timestamp enable
diagnose debug enable

Note:

Using both commands will also work as intended, as shown below:

Note:

Starting from v7.4.1, the 'diagnose vpn ike log-filter dst-addr4' command has been changed to 'diagnose vpn ike log filter rem-addr4'.

To filter multiple IPv4 remote gateway addresses 'diagnose vpn ike log filter mrem-addr4' could be used. To find the list of options followed after 'diagnose vpn ike log filter ?' use a question '?' mark after the command, as shown in the example given below.

diagnose vpn ike log filter ?

list Display the current filter.
clear Erase the current filter.
vd Index of virtual domain. -1 matches all.
name Phase1 name to filter by.
ifindex Index of the interface that IKE connection is negotiated over.
loc-addr4 IPv4 local gateway address range to filter by.
mloc-addr4 Multiple IPv4 local gateway address to filter by.
rem-addr4 IPv4 remote gateway address range to filter by.
mrem-addr4 Multiple IPv4 remote gateway address to filter by.
loc-addr6 IPv6 local gateway address range to filter by.
mloc-addr6 Multiple IPv6 local gateway address to filter by.
rem-addr6 IPv6 remote gateway address range to filter by.
mrem-addr6 Multiple IPv6 remote gateway addresses to filter by.
dst-port Destination port range to filter by.
negate Negate existing setting of the specified filter parameter.

diagnose vpn ike log filter rem-addr4 10.189.0.182
diagnose debug application ike -1

diagnose debug console timestamp enable

diagnose debug enable

Phase 2 checks: If the status of Phase 1 is in an established state, then focus on Phase 2. To do so, issue the command:

diagnose vpn tunnel list name <phase1-name>

diagnose vpn tunnel list name to10.189.0.182
list all ipsec tunnel in vd 0
name=to10.189.0.182 ver=1 serial=2 10.189.0.31:0->10.189.0.182:0
bound_if=10 lgwy=static/1 tun=intf/0 mode=auto/1 encap=none/8 options[0008]=npu
proxyid_num=1 child_num=0 refcnt=10 ilast=25 olast=25 ad=/0
stat: rxp=0 txp=0 rxb=0 txb=0
dpd: mode=on-demand on=0 idle=20000ms retry=3 count=0 seqno=534
natt: mode=none draft=0 interval=0 remote_port=0
proxyid=to10.189.0.182 proto=0 sa=0 ref=1 serial=4
src: 0:172.16.170.0/255.255.255.0:0
dst: 0:192.168.50.0/255.255.255.0:0

The important field from the particular output is the ‘sa’. SA can have three values:

sa=0 indicates there is a mismatch between selectors or no traffic is being initiated.
sa=1 indicates IPsec SA is matching and there is traffic between the selectors.
sa=2 is only visible during IPsec SA rekey.

Lastly, there might be cases where the encryption and hashing algorithms in Phase 2 are mismatched as well.

The selectors are shown under the command 'get vpn ipsec tunnel details'.

To identify this kind of error, run IKE debugging as it was described above.

Note: If it is necessary to also perform packet captures, it is advised to always run the following command to match debug message time with packet sniffer time and know exactly what is happening and when:

diag debug console time en

Note:

Starting from FortiOS v7.4.1, the 'diagnose vpn ike log-filter src-addr4' command has been changed to 'diagnose vpn ike log filter loc-addr4'.

IPsec troubleshooting scenario:

A troubleshooting scenario where the following debugs were done but no relevance was seen for the tunnel seen as 'inactive':

In the GUI, the tunnel interface is 'green'.

The tunnel is inactive and the sniffer shows the traffic not passing the tunnel:

FortiGate-61F # diagnose sniffer packet any 'host 10.1.1.37 and icmp' 4 0 l

interfaces=[any]

filters=[host 10.1.1.37 and icmp]

2024-07-16 10:46:01.327162 internal in 192.168.1.251 -> 10.1.1.37: icmp: echo request

2024-07-16 10:46:02.331446 internal in 192.168.1.251 -> 10.1.1.37: icmp: echo request

2024-07-16 10:46:03.335546 internal in 192.168.1.251 -> 10.1.1.37: icmp: echo request

Debug flow does not show any drops and ends up in the last line and then repeats:

2024-07-16 10:47:54 id=20085 trace_id=4 func=iprope_dnat_check line=5191 msg="in-[internal], out-[]"

2024-07-16 10:47:54 id=20085 trace_id=4 func=iprope_dnat_check line=5204 msg="result: skb_flags-02000000, vid-0, ret-no-match, act-accept, flag-00000000"

2024-07-16 10:47:54 id=20085 trace_id=4 func=vf_ip_route_input_common line=2615 msg="find a route: flag=00000000 gw-10.1.1.37 via root"

2024-07-16 10:47:59 id=20085 trace_id=5 func=print_pkt_detail line=5746 msg="vd-root:0 received a packet(proto=1, 192.168.1.251:1->10.1.1.37:2048) from internal. type

=8, code=0, id=1, seq=288."

The Encryption is negotiated:

diagnose vpn tunnel list name Primary Tunnel

name=Primary Tunnel ver=1 serial=2 10.101.1.1:0->10.101.1.2:0 tun_id=10.101.1.2 dst_mtu=1500 dpd-link=on remote_location=0.0.0.0 weight=1

bound_if=5 lgwy=static/1 tun=intf/0 mode=auto/1 encap=none/520 options[0208]=npu frag-rfc run_state=0 accept_traffic=1 overlay_id=0

proxyid_num=1 child_num=0 refcnt=4 ilast=0 olast=0 ad=/0

stat: rxp=11746 txp=32 rxb=1939048 txb=9861

dpd: mode=on-demand on=1 idle=10000ms retry=3 count=0 seqno=0

natt: mode=none draft=0 interval=0 remote_port=0

proxyid=Net-10.1.1.0-24 proto=0 sa=1 ref=2 serial=5

src: 0:192.168.1.0/255.255.255.0:0

dst: 0:10.1.1.0/255.255.255.0:0

SA: ref=4 options=10226 type=00 soft=0 mtu=1438 expire=42832/0B replaywin=2048

seqno=1 esn=0 replaywin_lastseq=000001db itn=0 qat=0 hash_search_len=1

life: type=01 bytes=0/0 timeout=42933/43200

dec: spi=e6bb099b esp=aes key=16 4c3dd75b85f0be4c2d2b95e5dfe7e99d

ah=sha1 key=20 f9a5dbc8360e4b58aff64e74fe40137cd89dc98e

enc: spi=533e1c0c esp=aes key=16 4fddb780c9606c5c235737bd7d05c585

ah=sha1 key=20 57cc8349171a97eaf230e565bb90853cefaca7ab

dec:pkts/bytes=476/75272, enc:pkts/bytes=0

One quick way to identify two-way traffic is to check the enc and dec counters on the above output. If seeing only the enc counters increasing and dec counters not increasing, it means that the traffic is being sent out but not received from the other end.

It is also necessary to check the Link monitor settings for a health check:

In the above screenshot, the tunnel interface (Primary Tunnel) shows the status dead as the peer is not reachable.

After disabling the settings in the link monitor, the tunnel interface is active.

Note:

Ensure that NPU offloading is disabled during troubleshooting. The npu-offload option is enabled by default. If NPU offloading is active, packets may be switched via the NPU, which could prevent capturing hits for flow filters. Ensure that disabling the npu-offload option would also reset the IPsec tunnel. Verify whether the npu-offload option is enabled/disabled using the following command:

config vpn ipsec phase1-interface

edit <name of the tunnel>

show full | grep npu

Related articles:

Troubleshooting Tip: Troubleshooting IPsec Site-to-Site Tunnel Connectivity

IPsec-related diagnose command.

Technical Tip: How to configure VPN Site to Site between FortiGates (Using VPN Setup Wizard)

Technical Tip: Setting multiple DNS server for IPSec dial-up VPN

Technical Tip: NAT-traversal comparison between site-to-site and dial-up” dynamic” tunnels

Technical Tip: FortiGate Hub with multiple IPSec Dial-up phase1 using IKEv2 and PSK authentication

Technical Tip : How to configure multiple VPN tunnels from the same ISP to the same remote peer ISP.

Technical Tip: IPSec dial-up full tunnel with FortiClient