Troubleshooting Tip: IPsec VPNs tunnels

sgiannogloudis · ‎10-25-2019

Description

This article describes techniques on how to identify, debug and troubleshoot issues with IPsec VPN tunnels.

Scope

FortiGate.

Solution

Identification.

As the first action, check the reachability of the destination according to the routing table with the following command:

get router info routing-table detail <destination-IP>

If the destination is reachable by multiple tunnels, isolate the problematic tunnel:

Enter the VDOM (if applicable) where the VPN is configured and type the command:

get vpn ipsec tunnel summary
'to10.174.0.182' 10.174.0.182:0 selectors(total,up): 1/1 rx(pkt,err): 1921/0 tx(pkt,err): 69/2
'to10.189.0.182' 10.189.0.182:0 selectors(total,up): 1/0 rx(pkt,err): 0/0 tx(pkt,err): 0/0

On the particular output, two VPN tunnels, to10.174.0.182 & to10.189.0.182 are visible. The second VPN tunnel on the list has its selectors in a down state so the focus will be on that tunnel.

Phase 1 checks. After the problematic tunnel has been identified, it will be possible to understand the status of Phase 1. To do so, type the below command:

diagnose vpn ike gateway list name to10.189.0.182

vd: root/0
name: to10.189.0.182
version: 1
interface: port9 10
addr: 10.189.0.31:500 -> 10.189.0.182:500
created: 15s ago
IKE SA: created 1/1
IPsec SA: created 0/0

id/spi: 19576 a83334b3c66f871b/0000000000000000
direction: responder
status: connecting, state 3, started 15s ago

The important field from this particular command is status. The status field has a discrete output that can be connected or established.

Established means Phase 1 is up and running.
Connecting means Phase 1 is down.

If Phase 1 is down, additional checks must be performed to identify the reason.

Ensure bidirectional connectivity between the VPN gateways (typically, this is the IP address on the WAN interface).

Try to traceroute (or ping) towards the VPN peer, in this example, use the commands:

execute traceroute-options source 10.189.0.31

execute traceroute 10.189.0.182

Confirm that IKE traffic for port 500 or 4500 is not blocked somewhere along the path.

To do so, perform a packet sniffer:

diag sniffer packet any "host 10.189.0.182 and (port 500 or port 4500)" 4 0 l
interfaces=[any]
filters=[host 10.189.0.182 and (port 500 or port 4500)]

Note:

If nattraversal is enabled under phase1 and FortiGate is behind the NAT, sniff traffic with 'udp port 4500'. Otherwise, sniff traffic with the filter 'udp port 500'

IKE debugging: If both of the above checks are successful, start debugging the IKE protocol to check for possible configuration mismatches between the peers:

diagnose vpn ike log-filter dst-addr4 10.189.0.182
diagnose debug application ike -1
diagnose debug console timestamp enable
diagnose debug enable

Note:

Using both commands will also work as intended, as shown below:

Note:

Starting from v7.4.1, the 'diagnose vpn ike log-filter dst-addr4' command has been changed to 'diagnose vpn ike log filter rem-addr4'.

diagnose vpn ike log filter rem-addr4 10.189.0.182
diagnose debug application ike -1

diagnose debug console timestamp enable

diagnose debug enable

Phase 2 checks: If the status of Phase 1 is in an established state, then focus on Phase 2. To do so, issue the command:

diagnose vpn tunnel list name <phase1-name>

diagnose vpn tunnel list name to10.189.0.182
list all ipsec tunnel in vd 0
name=to10.189.0.182 ver=1 serial=2 10.189.0.31:0->10.189.0.182:0
bound_if=10 lgwy=static/1 tun=intf/0 mode=auto/1 encap=none/8 options[0008]=npu
proxyid_num=1 child_num=0 refcnt=10 ilast=25 olast=25 ad=/0
stat: rxp=0 txp=0 rxb=0 txb=0
dpd: mode=on-demand on=0 idle=20000ms retry=3 count=0 seqno=534
natt: mode=none draft=0 interval=0 remote_port=0
proxyid=to10.189.0.182 proto=0 sa=0 ref=1 serial=4
src: 0:172.16.170.0/255.255.255.0:0
dst: 0:192.168.50.0/255.255.255.0:0

The important field from the particular output is the ‘sa’. SA can have three values:

sa=0 indicates there is a mismatch between selectors or no traffic is being initiated.
sa=1 indicates IPsec SA is matching and there is traffic between the selectors.
sa=2 is only visible during IPsec SA rekey.

Lastly, there might be cases where the encryption and hashing algorithms in Phase 2 are mismatched as well.

The selectors are shown under the command 'get vpn ipsec tunnel details'.

In order to identify this kind of error, run IKE debugging as it was described above.

Note: If it is necessary to also perform packet captures, it is advised to always run the following command to match debug message time with packet sniffer time and know exactly what is happening and when:

diag debug console time en

Note:

Starting from FortiOS v7.4.1, the 'diagnose vpn ike log-filter src-addr4' command has been changed to 'diagnose vpn ike log filter loc-addr4'.

IPsec troubleshooting scenario:

A troubleshooting scenario where the following debugs were done but no relevance was seen for the tunnel seen as 'inactive':

In the GUI, the tunnel interface is 'green'.

The tunnel is inactive and the sniffer shows the traffic not passing the tunnel:

FortiGate-61F # diagnose sniffer packet any 'host 10.1.1.37 and icmp' 4 0 l

interfaces=[any]

filters=[host 10.1.1.37 and icmp]

2024-07-16 10:46:01.327162 internal in 192.168.1.251 -> 10.1.1.37: icmp: echo request

2024-07-16 10:46:02.331446 internal in 192.168.1.251 -> 10.1.1.37: icmp: echo request

2024-07-16 10:46:03.335546 internal in 192.168.1.251 -> 10.1.1.37: icmp: echo request

Debug flow does not show any drops and ends up in the last line and then repeats:

2024-07-16 10:47:54 id=20085 trace_id=4 func=iprope_dnat_check line=5191 msg="in-[internal], out-[]"

2024-07-16 10:47:54 id=20085 trace_id=4 func=iprope_dnat_check line=5204 msg="result: skb_flags-02000000, vid-0, ret-no-match, act-accept, flag-00000000"

2024-07-16 10:47:54 id=20085 trace_id=4 func=vf_ip_route_input_common line=2615 msg="find a route: flag=00000000 gw-10.1.1.37 via root"

2024-07-16 10:47:59 id=20085 trace_id=5 func=print_pkt_detail line=5746 msg="vd-root:0 received a packet(proto=1, 192.168.1.251:1->10.1.1.37:2048) from internal. type

=8, code=0, id=1, seq=288."

The Encryption is negotiated:

diagnose vpn tunnel list name Primary Tunnel

name=Primary Tunnel ver=1 serial=2 10.101.1.1:0->10.101.1.2:0 tun_id=10.101.1.2 dst_mtu=1500 dpd-link=on remote_location=0.0.0.0 weight=1

bound_if=5 lgwy=static/1 tun=intf/0 mode=auto/1 encap=none/520 options[0208]=npu frag-rfc run_state=0 accept_traffic=1 overlay_id=0

proxyid_num=1 child_num=0 refcnt=4 ilast=0 olast=0 ad=/0

stat: rxp=11746 txp=32 rxb=1939048 txb=9861

dpd: mode=on-demand on=1 idle=10000ms retry=3 count=0 seqno=0

natt: mode=none draft=0 interval=0 remote_port=0

proxyid=Net-10.1.1.0-24 proto=0 sa=1 ref=2 serial=5

src: 0:192.168.1.0/255.255.255.0:0

dst: 0:10.1.1.0/255.255.255.0:0

SA: ref=4 options=10226 type=00 soft=0 mtu=1438 expire=42832/0B replaywin=2048

seqno=1 esn=0 replaywin_lastseq=000001db itn=0 qat=0 hash_search_len=1

life: type=01 bytes=0/0 timeout=42933/43200

dec: spi=e6bb099b esp=aes key=16 4c3dd75b85f0be4c2d2b95e5dfe7e99d

ah=sha1 key=20 f9a5dbc8360e4b58aff64e74fe40137cd89dc98e

enc: spi=533e1c0c esp=aes key=16 4fddb780c9606c5c235737bd7d05c585

ah=sha1 key=20 57cc8349171a97eaf230e565bb90853cefaca7ab

dec:pkts/bytes=476/75272, enc:pkts/bytes=0

It is also necessary to check the Link monitor settings for a health check:

In the above screenshot, the tunnel interface (Primary Tunnel) shows the status dead as the peer is not reachable.

After disabling the settings in the link monitor, the tunnel interface is active.

Note:

Ensure that NPU offloading is disabled during troubleshooting. The npu-offload option is enabled by default. If NPU offloading is active, packets may be switched via the NPU, which could prevent capturing hits for flow filters. Ensure that disabling the npu-offload option would also reset the IPsec tunnel. Verify whether the npu-offload option is enabled/disabled using the following command:

config vpn ipsec phase1-interface

edit <name of the tunnel>

show full | grep npu

Related articles: