Description
This article describes techniques on how to identify, debug and troubleshoot issues with IPsec VPN tunnels.
Scope
FortiGate.
Solution
- Identification.
As the first action, isolate the problematic tunnel. Enter the VDOM (if applicable) where the VPN is configured and type the command:
get vpn ipsec tunnel summary
'to10.174.0.182' 10.174.0.182:0 selectors(total,up): 1/1 rx(pkt,err): 1921/0 tx(pkt,err): 69/2
'to10.189.0.182' 10.189.0.182:0 selectors(total,up): 1/0 rx(pkt,err): 0/0 tx(pkt,err): 0/0
On the particular output, two VPN tunnels, to10.174.0.182 & to10.189.0.182 are visible. The second VPN tunnel on the list has its selectors in a down state so the focus will be on that tunnel.
- Phase 1 checks.
After the problematic tunnel has been identified, it will be possible to understand the status of Phase 1. To do so, type the below command:
diagnose vpn ike gateway list name to10.189.0.182
vd: root/0
name: to10.189.0.182
version: 1
interface: port9 10
addr: 10.189.0.31:500 -> 10.189.0.182:500
created: 15s ago
IKE SA: created 1/1
IPsec SA: created 0/0
id/spi: 19576 a83334b3c66f871b/0000000000000000
direction: responder
status: connecting, state 3, started 15s ago
The important field from this particular command is status. The status field has a discrete output that can be either connected or established.
- Established means Phase 1 is up and running.
- Connecting means Phase 1 is down.
If Phase 1 is down, perform additional checks to identify the reason.
- Ensure bidirectional connectivity exists between the VPN gateways.
Try to traceroute towards the VPN peer, in this example, use the commands:
execute traceroute-options source 10.189.0.31
execute traceroute 10.189.0.182
- Confirm that IKE traffic for port 500 or 4500 is not blocked somewhere along the path.
To do so, perform a packet sniffer:
diagnose sniffer packet any 'host 10.189.0.182 and port 500' 4 0 l
interfaces=[any]
filters=[host 10.189.0.182 and port 500]
Note:
If nattraversal is enabled under phase1 and FortiGate is behind the NAT, sniff traffic with 'udp port 4500'.
IKE debugging: If both of the above checks are successful, start debugging the IKE protocol to check for possible configuration mismatches between the peers:
diagnose vpn ike log-filter dst-addr4 10.189.0.182
diagnose debug application ike -1
diagnose debug application fnbamd -1
diagnose debug console timestamp enable
diagnose debug enable
Note: Using both commands will also work as intended, as shown below:
Note: Starting from FortiOS 7.4.1, the 'diagnose vpn ike log-filter dst-addr4' command has been changed to 'diagnose vpn ike log filter rem-addr4'.
diagnose vpn ike log filter rem-addr4 10.189.0.182
diagnose debug application ike -1
diagnose debug application fnbamd -1
diagnose debug console timestamp enable
diagnose debug enable
-
Phase 2 checks: If the status of Phase 1 is in an established state, then focus on Phase 2. To do so, issue the command:
diagnose vpn tunnel list name 10.189.0.182
list all ipsec tunnel in vd 0
name=to10.189.0.182 ver=1 serial=2 10.189.0.31:0->10.189.0.182:0
bound_if=10 lgwy=static/1 tun=intf/0 mode=auto/1 encap=none/8 options[0008]=npu
proxyid_num=1 child_num=0 refcnt=10 ilast=25 olast=25 ad=/0
stat: rxp=0 txp=0 rxb=0 txb=0
dpd: mode=on-demand on=0 idle=20000ms retry=3 count=0 seqno=534
natt: mode=none draft=0 interval=0 remote_port=0
proxyid=to10.189.0.182 proto=0 sa=0 ref=1 serial=4
src: 0:172.16.170.0/255.255.255.0:0
dst: 0:192.168.50.0/255.255.255.0:0The important field from the particular output is the ‘sa’. SA can have three values:
- sa=0 indicates there is a mismatch between selectors or no traffic is being initiated.
- sa=1 indicates IPsec SA is matching and there is traffic between the selectors.
- sa=2 is only visible during IPsec SA rekey.
Lastly, there might be cases where the encryption and hashing algorithms in Phase 2 are mismatched as well.
In order to identify this kind of error, run IKE debugging as it was described above.
Note: If it is necessary to also perform packet captures, it is advised to always run the following command to match debug message time with packet sniffer time and know exactly what is happening and when:
diag debug console time en
Note:
Starting from FortiOS v7.4.1, the 'diagnose vpn ike log-filter src-addr4' command has been changed to 'diagnose vpn ike log filter loc-addr4'.
IPsec troubleshooting scenario:
A troubleshooting scenario where the following debugs were done but no relevance was seen for the tunnel seen as 'inactive':
In the GUI, the tunnel interface is 'green'.
- The tunnel is inactive and the sniffer shows the traffic not passing the tunnel:
FortiGate-61F # diagnose sniffer packet any 'host 10.1.1.37 and icmp' 4 0 l
interfaces=[any]
filters=[host 10.1.1.37 and icmp]
2024-07-16 10:46:01.327162 internal in 192.168.1.251 -> 10.1.1.37: icmp: echo request
2024-07-16 10:46:02.331446 internal in 192.168.1.251 -> 10.1.1.37: icmp: echo request
2024-07-16 10:46:03.335546 internal in 192.168.1.251 -> 10.1.1.37: icmp: echo request
- Debug flow does not show any drops and ends up in the last line and then repeats:
2024-07-16 10:47:54 id=20085 trace_id=4 func=iprope_dnat_check line=5191 msg="in-[internal], out-[]"
2024-07-16 10:47:54 id=20085 trace_id=4 func=iprope_dnat_check line=5204 msg="result: skb_flags-02000000, vid-0, ret-no-match, act-accept, flag-00000000"
2024-07-16 10:47:54 id=20085 trace_id=4 func=vf_ip_route_input_common line=2615 msg="find a route: flag=00000000 gw-10.1.1.37 via root"
2024-07-16 10:47:59 id=20085 trace_id=5 func=print_pkt_detail line=5746 msg="vd-root:0 received a packet(proto=1, 192.168.1.251:1->10.1.1.37:2048) from internal. type
=8, code=0, id=1, seq=288."
- The Encryption is negotiated:
diagnose vpn tunnel list name Primary Tunnel
name=Primary Tunnel ver=1 serial=2 10.101.1.1:0->10.101.1.2:0 tun_id=10.101.1.2 dst_mtu=1500 dpd-link=on remote_location=0.0.0.0 weight=1
bound_if=5 lgwy=static/1 tun=intf/0 mode=auto/1 encap=none/520 options[0208]=npu frag-rfc run_state=0 accept_traffic=1 overlay_id=0
proxyid_num=1 child_num=0 refcnt=4 ilast=0 olast=0 ad=/0
stat: rxp=11746 txp=32 rxb=1939048 txb=9861
dpd: mode=on-demand on=1 idle=10000ms retry=3 count=0 seqno=0
natt: mode=none draft=0 interval=0 remote_port=0
proxyid=Net-10.1.1.0-24 proto=0 sa=1 ref=2 serial=5
src: 0:192.168.1.0/255.255.255.0:0
dst: 0:10.1.1.0/255.255.255.0:0
SA: ref=4 options=10226 type=00 soft=0 mtu=1438 expire=42832/0B replaywin=2048
seqno=1 esn=0 replaywin_lastseq=000001db itn=0 qat=0 hash_search_len=1
life: type=01 bytes=0/0 timeout=42933/43200
dec: spi=e6bb099b esp=aes key=16 4c3dd75b85f0be4c2d2b95e5dfe7e99d
ah=sha1 key=20 f9a5dbc8360e4b58aff64e74fe40137cd89dc98e
enc: spi=533e1c0c esp=aes key=16 4fddb780c9606c5c235737bd7d05c585
ah=sha1 key=20 57cc8349171a97eaf230e565bb90853cefaca7ab
dec:pkts/bytes=476/75272, enc:pkts/bytes=0
- It is also necessary to check the Link monitor settings for a health check:
In the above screenshot, the tunnel interface (Primary Tunnel) shows the status dead as the peer is not reachable.
After disabling the settings in the link monitor, the tunnel interface is active.
Note:
Ensure that NPU offloading is disabled during troubleshooting. The npu-offload option is enabled by default. If NPU offloading is active, packets may be switched via the NPU, which could prevent capturing hits for flow filters
Related documents:
Troubleshooting Tip: IPsec Site-to-Site Tunnel Connectivity.
