Troubleshooting Tip: How to use packet capture (PCAP) to troubleshoot an IPsec VPN one-way traffic issue

[fwilliams]

Description	This article explains how to use packet capture (PCAP) to troubleshoot IPsec VPN one-way traffic issues.
Scope	FortiOS
Solution	One-way traffic issues are common with IPSec VPN tunnels. Customers usually report the case as 'tunnel is up but not passing traffic'. There is a common misconception that once the tunnel is up, it should pass traffic. This is not quite accurate, as the criteria for the tunnel being up and the tunnel passing traffic are not the same. Below is an example of how an IPsec tunnel that is UP, but not passing traffic, looks during a PCAP. Note that the ISAKMP packet is going through the tunnel back and forth between the peers (an indication that the tunnel is up), but the ESP packet is only seen in one direction (from 10.5.23.25 to 197.1.1.2). This type of issue can be troubleshooted more efficiently by applying the 'follow the packet' approach. In the setup above, the VPN tunnel is terminated on the interface forvpn1 in the 'int' VDOM, which does not have direct access to the internet, but gets internet access through the 'ext' VDOM. This means that, to establish a VPN tunnel from interface 'forvpn1' in the 'int' VDOM to the Spoke FortiGate, it has to pass through the inter-VDOM link between the 'int' and 'ext' VDOMs. Assume an issues arise in this type of setup where the user in the branch cannot access applications in the HQ (hub) through the VPN tunnel; and it is known that the traffic is leaving the branch but no response is received (i.e. this is a one-way traffic issue). It is possible to use a packet capture on FortiGate to capture an ESP packet (since traffic over IPsec tunnels are wrapped in ESP, proto 50) on the following interfaces: port1 (Spoke FortiGate). port2 (ext VDOM on the hub FortiGate). forvpn0 (ext VDOM on the hub FortiGate). forvpn1 (int VDOM on the hub FortiGate). Essentially, capture packets on the source and destination interface that formed the tunnel in question, plus every interface in-between (if there are any). Go to Network -> Packet Capture -> Create New, and then select the interface to capture PCAP packets on. Filter the capture to eliminate noise (use the IP address of the tunnel source, the tunnel destination, and proto 50. These three items should be enough for the filter). Note: A small number of packets is enough because the same information is going to be repeated by subsequent packets. Set similar packet captures on the 4 interfaces or points (this case is only relevant to this specific example: the reader's environment may be different). Start the captures (all four) and then ping to the server in the HQ (hub) unit. Do not do continue pinging after approximately five pings, as this number will suffice. Note: It is possible the capture is under Network -> Diagnostics -> Packet Capture in some cases. After the ping has finished, download the four PCAP files and examine them to establish where the issue lies. In this case, it can be seen that the ESP packet made it all the way to the 'int' VDOM on the hub FortiGate. The flow from spoke to hub is as follows: spoke-port1 -> port2.ext -> forvpn0.ext -> forvpn1.int. Now, it is possible to focus on finding where the issue lies (the int vdom). Check if it was dropped by the FW policy, by a wrong 'return routing', or by NPU (if the tunnel is terminated on an NPU offload-able interface). For the FW policy drop, use a debug flow similar to the following: diag debug reset diag debug flow filter proto 50 diag debug flow show function-name enable diag debug flow show iprope enable diag debug flow trace start 20 diag debug enable Note: It is better to run the debug flow when there is no session related to this ESP in the session table. In these cases, there will be more information in the debug output because FortiGate will do a full analysis of traffic admitted into the session table. Check the session table with the following commands: diag sys session filter clear diag sys session filter src x.x.x.x <- Use the IP of the branch if checking on hub, or the IP of the hub if checking on branch. diag sys session filter proto 50 diag sys session list After, use 'diag sys session clear' to remove the session. Use this command with care: ensure the filter is set before executing, as it will otherwise clear all of the sessions in the FortiGate’s table, which will cause brief outage. In the 'debug flow' logs or output, if only 'original direction' is seen (and not 'reply direction'), it very likely that an FW policy from src=vpn_tunnel to dst=port_to_server must be made (tailored after this article example). It is also possible the ESP packet gets to the 'ext' VDOM and is not able to cross to the 'int' VDOM. Repeat the steps above on the 'ext' VDOM to eliminate the issue. If the FW policy is not the issue, the NPU might be dropping the packets in the reply direction. Disable NPU and see if that helps. If not, add 'reply disable' under phase2. If the IP address the VPN tunnel is bound to is also use in the VIP, this may cause routing issues. In this case, it will be visible in the debug logs that FortiGate is sending the response to itself (find a route: flag=xxxxx  gw-y.y.y.y via root/vdom_name) instead of the remote peer. The 'arp reply' must be disabled under the VIP object in this case: config firewall vip edit "name" set arp-reply disable end If the packet capture shows the ESP packet was on the spoke WAN interface but does not show anything on the hub’s WAN interface, it is very likely the ISP is the cause of the issue. Try disabling the VPN tunnel for around 5-10 minutes, then re-enable it - this may resolve this issue. Note: In an environment with lots of IPSec tunnels (a thousand or more), ensure the SPI on the PCAP file collected matches the SPI of the tunnel under investigation. In this article for example, the SPI on the ESP packets captured at all the four points/interfaces along the traffic path have 'SPI=0x30d4f3a5', which corresponds to the SPI for encrypting packets on the tunnel undergoing troubleshooting (from the perspective of sourcing traffic from the branch to HQ).