Troubleshooting Tip: Collecting Windows logs for troubleshooting FortiClient IPsec connection failures (Pktmon)

Overview.

It is quite common to have multiple security solutions on the same Windows workstation (AV, EDR, WF, etc.). Complications may arise when these solutions are delivered by multiple vendors which may lead to conflicts as quite commonly these solutions add up filtering drivers to the workstation networking stack. This means that traffic has to go through multiple levels of filtering which may lead to traffic loss.

The motivation of this article is to equip engineers with tools that provide a complete visibility into Windows networking stack which allows to see packet flow all the way from the application layer to NIC.

The tools that will enable that are Windows-native packet capture Pktmon.exe (Packet Monitor) and netsh wfp utility.

Why not Wireshark:

Packet Monitor captures packets at every point of their traversal through Windows networking stack (NIC, NDIS filter drivers, WFP layer, TCP/IP) as opposed to Wireshark's Npcap driver that inserts itself at a single point in the stack - typically at the NDIS protocol layer, just above the NIC miniport driver. Everything that happens above that insertion point (deeper into the OS, between the NIC and the application) is invisible to Wireshark.

In the screenshot above, the Wireshark driver is highlighted. The issue here is architectural:

• (Outbound) Any outbound packet shown in Wireshark packet capture may not actually leave the adapter as can get blocked by a security program within the WFP Native Filter.
• (Inbound) Any inbound packet shown in Wireshark packet capture may not get to the application layer as can get blocked by third-party security program's drivers (i.e. EDR).

Note: Another significant advantage of Pktmon is that it has a built-in functionality of identifying packet loss (via counters) and blocks (by filter drivers) with presenting a reason for a drop (i.e. firewall rule).

Identifying components of Windows networking stack.

It is important to understand what drivers are installed and take part in processing of networking traffic. The commands below capture information about the drivers of the NIC, NDIS, WFP layer, TCP/IP, and application layers.

When engaging with technical support, it is recommended to run the following command:

pktmon list --json > name_of_the_file.json

This command will collect output similar to the one on the screenshot above and convert it into JSON file.

The same output but without JSON conversion may be gathered as follows:

pktmon list --all

NDIS Drivers and WFP Callouts.

Before diving into collecting Pktmon captures, it is important to understand the concept of WFP (Windows Filtering Platform) Native Filter. See Introduction to Windows Filtering Platform Callout Drivers - Windows drivers | Microsoft Learn.

Modern security solutions register their filter drivers as WFP callout drivers as opposed to legacy NDIS filter drivers. As a result, FortiClient and other third-party security programs' filters may not be shown in the networking stack as they are 'hidden' inside the WFP Native Filter.

This is where most of the user-space security filtering operations occur (including packet blocks).

Above is an example of the pktmon counters --type drop --drop-reason command.

In order to view all the filters registered with WFP and identify which one(s) resulted in traffic block run the following command (elevated):

netsh wfp show filters

It will output the results in the filters.xml file where all filter drivers can be viewed.

Hint: While using tools similar to NotePad++, filter for the '<name>' or '<description>' keywords to see filters' descriptions and names. Alternatively, search for the name of the filter if known.

Follow the steps below to collect Pktmon and WFP events for IPsec connection failure troubleshooting.

Step 1 - List components of the networking stack.

Below command shows what drivers process network traffic on Windows workstation.

pktmon list --all

Or: 

pktmon list --json > pktmon_components.json

Share this with technical support.

Step 2 - Capture network traffic.

The commands below capture udp/500 and udp/4500 traffic toward X.X.X.X IP-Address.

pktmon filter add -i X.X.X.X -t UDP -p 500 
pktmon filter add -i X.X.X.X -t UDP -p 4500

Replace X.X.X.X with the VPN gateway's IP-address. If it is unknown, do not use the -i filter.

Step 3 - Start the capture and reproduce the issue.

Start the capture by running the command below.

pktmon start --capture --comp all --pkt-size 0 --file ipsec.etl

• --comp all – capture at all components (NIC, filters, WFP, TCPIP).

• --pkt-size 0 – capture full packets.

• --file – ETL file for later conversion.

Reproduce the issue after the capture has been started.

Step 4 - Stop the capture and convert to PCAPNG.

Stop the capture and convert to Wireshark-readable format.

pktmon stop

pktmon pcapng ipsec.etl -o ipsec.pcapng

Attach both ETL and PCAPNG files to the support case.

Step 5 - Export counters (to see where packets were dropped).

Run below commands to see where exactly in the networking stack packets were dropped.

pktmon counters --type all --drop-reason --include-hidden --json > pktmon_counters_ipsec.json
pktmon counters --type drop --drop-reason --include-hidden --json > pktmon_drops_ipsec.json

To see the output in the terminal, omit the --json parameter as well as file redirect operation.

Attach the output to the support case.

Step 6 - Remove Filter.

Remove filters for future troubleshooting.

pktmon filter remove

Note: If packet block happens in the WFP Native Filter Layer, proceed to the next step.

Step 7 - Collect WFP states.

Lists all WFP providers, filters, and action taken by them.

netsh wfp show state file=wfp_state.xml

Attach the output file to the case.

In total, the following files have to collected:

• Networking stack components (pktmon_components.json) from Step 1.
• ETL and PCAPNG capture files from Step 4.
• Two counter files from Step 5.
• WFP states file from step 7.

Additional resources:

1. Introduction to Windows Filtering Platform Callout Drivers - Windows drivers | Microsoft Learn
2. Pktmon command formatting | Microsoft Learn
3. Introduction to Windows Filtering Platform Callout Drivers - Windows drivers | Microsoft Learn