Troubleshooting Tip: Sample TeraTerm scripts for VPN troubleshooting and monitoring

These scripts are intended to collect diagnostic information when attempting to determine if a FortiGate is dropping IPsec tunnel traffic. This is not the cause of most IPsec tunnel issues and is not a productive first step in most cases. For general IPsec troubleshooting, see the overview article 'Troubleshooting IPsec Site-to-Site Tunnel Connectivity'.

For instructions on how to run and log a TeraTerm script, see the article 'Technical Tip: FortiGate monitoring script'.

NP6XLITE (most F-series models up to 201F) IPsec performance script:

The provided script files likely require adjustment to match the environment.

Relevant commands:

get system status
get hardware status

# take the following every ten or twenty seconds

execute time
get sys perf status
diag vpn tunnel list
diag npu np6xlite dce 0

diag npu np6 anomaly-drop 0
fnsysctl cat /proc/net/np6xlite_0/pdq
fnsysctl cat /proc/net/np6xlite_0/hif-stats
fnsysctl cat /proc/net/np6xlite_0/hifdrop
fnsysctl cat /proc/net/np6xlite_0/osw
fnsysctl cat /proc/net/np6xlite_0/fos-perf
fnsysctl cat /proc/net/np6xlite_0/ipsec-perf
diag vpn ipsec status
diag hardware device nic <ingress_intf> <----- Physical interface used to receive local traffic that would go over. IPsec tunnel
diag hardware device nic <egress_intf> <----- Physical interface used to send IPsec data traffic.
fnsysctl cat /proc/softirqs
fnsysctl cat /proc/interrupts
diag sys profile report
diag sys top-all 10

Script Files:

Single-VDOM mode:

np6xlite_ipsec_novdom.ttl

Multi-VDOM mode (take simultaneously in different TeraTerm connections):

np6xlite_ipsec_global.ttl

np6xlite_ipsec_root.ttl  <----- Requires modification for VDOMs other than 'root'. Use with care since editing a VDOM that does not exist causes FortiGate to create it.

Other relevant diagnostics:

• Simultaneous packet sniffer if possible. If the issue only occurs when auto-asic-offload is enabled on the firewall policy or npu-offload is enabled on the IPsec tunnel, not all packets will appear in a FortiGate packet sniffer.
• IPsec diagnostics from the general IPsec troubleshooting article including ike debug output.
• FortiGate configuration file.
• VPN events log.
• Diagnostics and error counters from the remote side if feasible.

Modifying IPsec scripts for use with other FortiGate models:

• Determine the NP chipset:
  
  FGT2KE-1 # get hardware status
  Model name: FortiGate-2000E
  ASIC version: CP9
  <output omitted>
  Network Card chipset: FortiASIC NP6 Adapter (rev.)
  

• Reference the platform's fast path architecture to verify which NP output should be captured.
  
  FGT2KE-1 # diagnose npu np6 port-list
  Chip XAUI Ports QSGMII Max Cross-chip
  Speed offloading
  ------ ---- ------- ------ ----- ----------
  np6_1 0 port1 NA 1G No
  0 port5 NA 1G No
  0 port9 NA 1G No
  .
  .
  .
  ------ ---- ------- ------ ----- ----------
  np6_2 2 port37 NA 10G No
  3 port38 NA 10G No
  ------ ---- ------- ------ ----- ----------
  
  
• Modify the script to refer to the correct NP unit and NP type. For example, on FortiGate-2KE if the traffic is inbound over an IPsec tunnel hosted on port37 and outbound on a VLAN hosted on port38, both ports are handled by the 'np6_2' unit and the diagnostics should be updated following this pattern:

'diag npu np6xlite dce 0' becomes 'diag npu np6 dce 2'.
'fnsysctl cat /proc/net/np6xlite_0/pdq' becomes 'fnsysctl cat /proc/net/np6_2/pdq'.
Replace other references to 'np6xlite_0' with 'np6_2'.

If the traffic spans multiple NP chips, collect diagnostics from each relevant NP unit. TAC support might request different command outputs in the context of an existing support case.

SSL VPN performance monitoring script

Relevant commands:

exec date

exec time

diag vpn ssl list

diag vpn ssl mux

diag vpn ssl mux-stat

diag sys top 1 50
diag sys mpstat 1

TeraTerm Script Files:

sslvpn_monitor_novdom.ttl <-- requires 

Other relevant diagnostics:

• Configuration file.
• Simultaneous packet sniffer filtered by SSL VPN port and client's public IP address if possible.
• Simultaneous SSL VPN debug output.

diagnose debug reset
diagnose vpn ssl debug-filter clear

diagnose debug enable

diagnose debug duration 0

diagnose debug console timestamp enable

#diagnose vpn ssl debug-filter src-addr4 <client public IP address>
diagnose debug application sslvpn -1

#collect output

diagnose debug reset
diagnose vpn ssl debug-filter clear

Note:

Super Admin privilege is required to run 'fnsysctl' command. Otherwise, FortiGate will return an error as mentioned in this article: Technical Tip: fnsysctl command returns Unknown action 0