Troubleshooting Tip: IPsec VPN tunnels

Anonymous · ‎10-25-2019

Description

This article describes techniques on how to identify, debug, and troubleshoot issues with IPsec VPN tunnels.

Scope

FortiGate v7.2 and above.

Solution

Identification. As the first action, check the reachability of the destination according to the routing table with the following command:

get router info routing-table detail <destination-IP>

If the destination is reachable by multiple tunnels, isolate the problematic tunnel:

Enter the VDOM (if applicable) where the VPN is configured and type the following command:

get vpn ipsec tunnel summary
'to10.174.0.182' 10.174.0.182:0 selectors(total,up): 1/1 rx(pkt,err): 1921/0 tx(pkt,err): 69/2
'to10.189.0.182' 10.189.0.182:0 selectors(total,up): 1/0 rx(pkt,err): 0/0 tx(pkt,err): 0/0

On the particular output, two VPN tunnels, 'to10.174.0.182' and 'to10.189.0.182' are visible. The second VPN tunnel on the list has its selectors in a down state, so the focus will be on that tunnel.

Phase 1 checks.

After the problematic tunnel has been identified, it will be possible to understand the status of Phase 1. To do so, type the following command:

diagnose vpn ike gateway list name <IPsec Tunnel Name> EX: to10.189.0.182

vd: root/0
name: to10.189.0.182
version: 1
interface: port9 10
addr: 10.189.0.31:500 -> 10.189.0.182:500
created: 15s ago
IKE SA: created 1/1
IPsec SA: created 0/0

id/spi: 19576 a83334b3c66f871b/0000000000000000
direction: responder
status: connecting, state 3, started 15s ago

The key field to focus on in this command is status. This field indicates the current state of the VPN connection and typically shows either connected or established.

Established signifies that Phase 1 of the IPsec VPN tunnel is active. In this case, verify the Phase 2 configuration and its associated parameters.
Connecting indicates that Phase 1 has not been successfully established.

If Phase 1 is down, additional checks must be performed to identify the reason.

Ensure bidirectional connectivity between the VPN gateways (typically, this is the IP address on the WAN interface).

Try to traceroute (or ping) towards the VPN peer. In this example, use the commands:

execute traceroute-options source 10.189.0.31

execute traceroute 10.189.0.182

Confirm that IKE traffic for port 500 or 4500 is not blocked somewhere along the path.

To do so, perform a packet capture using the sniffer command:

diagnose sniffer packet any "host 10.189.0.182 and (port 500 or port 4500)" 4 0 l
interfaces=[any]
filters=[host 10.189.0.182 and (port 500 or port 4500)]

Note:

If nattraversal is enabled under phase1 and FortiGate is behind the NAT, sniff traffic with 'udp port 4500'. Otherwise, sniff traffic with the filter 'udp port 500'.

IKE debugging: If both of the above checks are successful, start debugging the IKE protocol to check for possible configuration mismatches between the peers:

diagnose debug reset

diagnose vpn ike log filter dst-addr4 10.189.0.182
diagnose debug application ike -1
diagnose debug console timestamp enable
diagnose debug enable

To stop the debug, use the command given below:

diagnose debug disable

diagnose debug reset

In v7.6.3, IKE debugging can be performed through the GUI using the 'CLI diagnostics' command, as shown below:

image (46).png

Here are the other options for the IKE filter:

rem-addr4 <----- IPv4 remote gateway address range to filter by.
msrc-addr4    <----- Multiple IPv4 source addresses to filter by.
mdst-addr4    <----- Multiple IPv4 destination addresses to filter by.
msrc-addr6    <----- Multiple IPv6 source address to filter by.
mdst-addr6    <----- Multiple IPv6 destination addresses to filter by.

For v7.4.0 and above, there is a slight change in the command as below:

diagnose debug reset

diagnose vpn ike log filter rem-addr4 10.189.0.182
diagnose debug application ike -1
diagnose debug console timestamp enable
diagnose debug enable

After collecting the debugging output, disable the debug processes with the following commands:

diagnose debug disable

diagnose debug reset

IKE port can be configured in case of being blocked by other devices in the path by following these articles:
Technical Tip: Configure custom IKE port between two FortiGate firewalls
Technical Tip: How to use TCP as transport for IKE/IPsec traffic

Use the following diagnostic commands to identify remote user authentication issues.

diagnose debug application fnbamd -1

Use the following diagnostic commands to identify SAML user authentication issues.

diagnose debug application samld -1

To stop the debug, use the command given below:

diagnose debug disable

diagnose debug reset

Note:

Using both commands will also work as intended, as shown below:

Note:

Starting from v7.4.1, the 'diagnose vpn ike log-filter dst-addr4' command has been changed to 'diagnose vpn ike log filter rem-addr4'.

To filter multiple IPv4 remote gateway addresses, 'diagnose vpn ike log filter mrem-addr4' could be used. To find the list of options followed after 'diagnose vpn ike log filter ?' use a question '?' mark after the command, as shown in the example given below.

diagnose vpn ike log filter ?

list Display the current filter.
clear Erase the current filter.
vd Index of virtual domain. -1 matches all.
name Phase1 name to filter by.
ifindex Index of the interface that IKE connection is negotiated over.
loc-addr4 IPv4 local gateway address range to filter by.
mloc-addr4 Multiple IPv4 local gateway address to filter by.
rem-addr4 IPv4 remote gateway address range to filter by.
mrem-addr4 Multiple IPv4 remote gateway address to filter by.
loc-addr6 IPv6 local gateway address range to filter by.
mloc-addr6 Multiple IPv6 local gateway address to filter by.
rem-addr6 IPv6 remote gateway address range to filter by.
mrem-addr6 Multiple IPv6 remote gateway addresses to filter by.
dst-port Destination port range to filter by.
negate Negate existing setting of the specified filter parameter.

diagnose vpn ike log filter rem-addr4 10.189.0.182
diagnose debug application ike -1

diagnose debug console timestamp enable

diagnose debug enable

Note:

It is best practice to clear the filter before running the debugs. To clear the filter, enter the following command:

diagnose vpn ike log filter clear

For v7.4.0 and above:

diagnose vpn ike log filter clear

Phase 2 checks: If the status of Phase 1 is in an established state, then focus on Phase 2. To do so, issue the following command:

diagnose vpn tunnel list name <phase1-name>

diagnose vpn tunnel list name to10.189.0.182
list all ipsec tunnel in vd 0
name=to10.189.0.182 ver=1 serial=2 10.189.0.31:0->10.189.0.182:0
bound_if=10 lgwy=static/1 tun=intf/0 mode=auto/1 encap=none/8 options[0008]=npu
proxyid_num=1 child_num=0 refcnt=10 ilast=25 olast=25 ad=/0
stat: rxp=0 txp=0 rxb=0 txb=0
dpd: mode=on-demand on=0 idle=20000ms retry=3 count=0 seqno=534
natt: mode=none draft=0 interval=0 remote_port=0
proxyid=to10.189.0.182 proto=0 sa=0 ref=1 serial=4
src: 0:172.16.170.0/255.255.255.0:0
dst: 0:192.168.50.0/255.255.255.0:0

The important field from the particular output is the ‘sa’. SA can have three values:

sa=0 indicates there is a mismatch between selectors or no traffic is being initiated.
sa=1 indicates IPsec SA is matching and there is traffic between the selectors.
sa=2 is only visible during IPsec SA rekey.

In some cases, the encryption and hashing algorithms in Phase 2 may be mismatched.

The selectors can be viewed using the following command: 'get vpn ipsec tunnel details'.

To identify this kind of error, run IKE debugging as it was described above.

When using a dial-up tunnel, run the command below to see all of the connection instances of the phase1:

diagnose vpn tunnel dialup-list <phase1-name>

Note:

If packet captures are required, it is recommended to first run the following command. This ensures that the debug message timestamps align with the packet sniffer timestamps, allowing precise correlation of events and timing:

diagnose debug console time enable

Note:

Starting from v7.4.1, the 'diagnose vpn ike log-filter src-addr4' command has been changed to 'diagnose vpn ike log filter loc-addr4'.

IPsec troubleshooting scenario:

A troubleshooting scenario where the following debugs were done, but no relevance was seen for the tunnel seen as 'inactive':

In the GUI, the tunnel interface is 'green'.

The tunnel is inactive, and the sniffer shows that the traffic is not passing through the tunnel:

FortiGate-61F # diagnose sniffer packet any 'host 10.1.1.37 and icmp' 4 0 l

interfaces=[any]

filters=[host 10.1.1.37 and icmp]

2024-07-16 10:46:01.327162 internal in 192.168.1.251 -> 10.1.1.37: icmp: echo request

2024-07-16 10:46:02.331446 internal in 192.168.1.251 -> 10.1.1.37: icmp: echo request

2024-07-16 10:46:03.335546 internal in 192.168.1.251 -> 10.1.1.37: icmp: echo request

Debug flow does not show any drops and ends up in the last line, and then repeats:

2024-07-16 10:47:54 id=20085 trace_id=4 func=iprope_dnat_check line=5191 msg="in-[internal], out-[]"

2024-07-16 10:47:54 id=20085 trace_id=4 func=iprope_dnat_check line=5204 msg="result: skb_flags-02000000, vid-0, ret-no-match, act-accept, flag-00000000"

2024-07-16 10:47:54 id=20085 trace_id=4 func=vf_ip_route_input_common line=2615 msg="find a route: flag=00000000 gw-10.1.1.37 via root"

2024-07-16 10:47:59 id=20085 trace_id=5 func=print_pkt_detail line=5746 msg="vd-root:0 received a packet(proto=1, 192.168.1.251:1->10.1.1.37:2048) from internal. type

=8, code=0, id=1, seq=288."

The encryption is negotiated:

diagnose vpn tunnel list name Primary Tunnel

name=Primary Tunnel ver=1 serial=2 10.101.1.1:0->10.101.1.2:0 tun_id=10.101.1.2 dst_mtu=1500 dpd-link=on remote_location=0.0.0.0 weight=1

bound_if=5 lgwy=static/1 tun=intf/0 mode=auto/1 encap=none/520 options[0208]=npu frag-rfc run_state=0 accept_traffic=1 overlay_id=0

proxyid_num=1 child_num=0 refcnt=4 ilast=0 olast=0 ad=/0

stat: rxp=11746 txp=32 rxb=1939048 txb=9861

dpd: mode=on-demand on=1 idle=10000ms retry=3 count=0 seqno=0

natt: mode=none draft=0 interval=0 remote_port=0

proxyid=Net-10.1.1.0-24 proto=0 sa=1 ref=2 serial=5

src: 0:192.168.1.0/255.255.255.0:0

dst: 0:10.1.1.0/255.255.255.0:0

SA: ref=4 options=10226 type=00 soft=0 mtu=1438 expire=42832/0B replaywin=2048

seqno=1 esn=0 replaywin_lastseq=000001db itn=0 qat=0 hash_search_len=1

life: type=01 bytes=0/0 timeout=42933/43200

dec: spi=e6bb099b esp=aes key=16 4c3dd75b85f0be4c2d2b95e5dfe7e99d

ah=sha1 key=20 f9a5dbc8360e4b58aff64e74fe40137cd89dc98e

enc: spi=533e1c0c esp=aes key=16 4fddb780c9606c5c235737bd7d05c585

ah=sha1 key=20 57cc8349171a97eaf230e565bb90853cefaca7ab

dec:pkts/bytes=476/75272, enc:pkts/bytes=0

A quick method to verify two-way traffic is to examine the enc and dec counters in the output. If the enc counters are increasing while the dec counters remain unchanged, this indicates that traffic is being sent out but not received from the remote side.

It is also necessary to check the Link monitor settings for a health check:

In the above screenshot, the tunnel interface (Primary Tunnel) shows the status dead as the peer is not reachable.

After disabling the settings in the link monitor, the tunnel interface is active.

Note:

Ensure that NPU offloading is disabled during troubleshooting. The NPU-offload option is enabled by default. When active, packets may be processed by the NPU, which can prevent capturing hits for flow filters. Disabling the npu-offload option will reset the IPsec tunnel. To check whether the npu-offload option is enabled or disabled, use the following command:

config vpn ipsec phase1-interface

edit <name of the tunnel>

show full | grep npu

Use the below-listed commands to disable NPU offloading:

config vpn ipsec phase1-interface

edit <name of the tunnel>

set npu-offload disable

Other commands that may be helpful to present the output to TAC:

diagnose vpn ike status detailed

To flush the tunnel:

diagnose vpn tunnel flush <my-phase1-name>

diagnose vpn ike gateway clear name <my-phase1-name>