Description
This article describes the first steps to troubleshoot connectivity problems to or through a FortiGate.
It is also helpful to provide this diagnostic information to the Fortinet Technical Assistance Center when opening a ticket to address a connectivity issue.
Assume the following diagram represents the topology:
[ PC1 ] === port1 [ FortiGate ] port2 ==== [ PC2]
Assumptions: PC1 and PC2 can be either local to port1 and port2 subnets, or on remote subnets routed via routers.
Scope
All FortiGates and FortiOS - NAT or Transparent mode.
Solution
Summary.
Step 1: Routing table check (in NAT mode).
Step 2: Verify is services are opened (if access to the FortiGate).
Step 3: Check Forward Traffic Logs
Step 4: Sniffer trace.
Step 5: Debug flow.
Step 6: Session list.
Note:
On FortiGate using NP2 interfaces, the traffic might be offloaded to the hardware processor, therefore changing the analysis with a sniffer trace or a debug flow as the traffic will not be seen with this procedure.
For using the sniffer and debug flow with NP2 ports, NP2 offloading must be disabled. Refer to the related article given further below.
Step 1: Routing table verification.
If the FortiGate is running in NAT mode, verify that all desired routes are in the routing table: local subnets, default routes, specific static routes, and dynamic routing protocol.
The FortiGate will drop packets in case of RPF check failure (see related article at the end of this page Details about RPF (Reverse Path Forwarding), also called Anti Spoofing, on FortiOS ).
To verify the routing table, use the CLI command 'get router info routing-table all' as per the example below:
get router info routing-table all
Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP
O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
* - candidate default
S* 0.0.0.0/0 [10/0] via 192.168.183.254, port1, [0/50]
C 10.0.0.0/24 is directly connected, VLAN_on_port1
C 10.160.0.0/23 is directly connected, port2
C 12.0.0.0/24 is directly connected, port1
C 172.16.78.0/24 is directly connected, VLAN_on_port3
C 192.168.182.0/23 is directly connected, port1
To check all routes including inactive routes, use the CLI command 'get router info routing-table database'.
If the connectivity issue is with a particular destination IP address or subnet, verify whether the route to the destination is correct with the CLI command 'get router info routing-table detail <destination-IP-address>'.
Step 2: Verify if services are opened on the port (if accessing the FortiGate itself).
- Verify that all appropriate services are opened on the interface that is being accessed. (telnet, http...)
Example :
show system interface port1
config system interface
edit "port1"
set vdom "root"
set ip 192.168.182.108 255.255.254.0
set allowaccess ping https ssh http telnet
set type physical
next
end
-
If the interface is accessed via another port of the FortiGate, a firewall policy must exist to allow this traffic
Example :
config firewall policy
edit 1
set srcintf "port1"
set dstintf "port2"
set srcaddr "all"
set dstaddr "all"
set action accept
set schedule "always"
set service "ANY"
next
Step 3: Check Forward Traffic Logs
Logging can be enabled on a firewall policy by accessing: Policy & Objects then a new policy can be created - or editing an existing policy - and accessing the logging section on the firewall policy:
This tool can be a very useful step to identify how the traffic is processing. There are 2 ways a firewall policy can log traffic:
- All sessionsThat option allows the policy to send all traffic it matches to the logging daemon - fgtlogd - regardless if the traffic was: Allowed, Denied, Denied by UTM, etc.
- The security eventsThis option allows logging traffic only when it is inspected by one of the Security profiles assigned to the firewall policy matching the traffic and in this case, traffic will be logged in 2 locations: 'Forward traffic' and 'Security Events'.
Note:
The third option in this setting is to disable the logging option altogether. Here how the logging option look like in the WebAdmin page:
Here is the CLI version of this option:
config firewall policy
edit <policy id>
set logtraffic <all - utm - disable>
next
end
For testing purposes and troubleshooting reasons it is recommended to keep the logging option set to 'All sessions'. Here are some examples of how the traffic log looks like:
Step 4: Sniffer trace.
Take a sniffer trace as per the following examples when running a constant ping (or TCP connection) from PC1 to PC2.
This will answer the following questions:
- Is traffic arriving at the FortiGate and does it arrive at the expected port?
- Is the ARP resolution correct for the targeted next-hop?
- Is the traffic exiting the FortiGate to the destination?
- Is the traffic sent back to the source?
diagnose sniffer packet any "host <PC1> and host <PC2>" 4
Or:
diagnose sniffer packet any "(host <PC1> and host <PC2>) and icmp" 4
Including the ARP protocol in the filter may be useful to troubleshoot a failure in the ARP resolution (for instance PC2 may be down and not responding to the FortiGate ARP requests).
diagnose sniffer packet any "host <PC1> and host <PC2> or arp" 4
To stop the sniffer, type CTRL+C.
With verbosity 4 above, the sniffer trace will display the port names where traffic ingresses/egresses.
Step 5: Debug flow.
Traffic should come in and leave the FortiGate. If not, proceed with a debug flow as follows:
diag debug enable
diag debug flow filter <- Find the options to filter below.
diag debug console timestamp enable
diag debug flow show iprope enable
diag debug flow show function-name enable
diag debug flow trace start 100 <- This will display 100 packets for this flow.
diag debug enable
To stop all other debug, type 'diag debug flow trace stop'. To clear the filter, type 'diag debug flow filter clear' .
The debug filter:
- Filter based on Protocol:
diagnose debug flow filter proto 1
Proto can be changed to be another protocol number value to focus on each protocol number as follows (proto = protocol number):
protocol number 1 = ICMP (ping).
protocol number 6 = TCP.
protocol number 17 = UDP.
etc.
- Filter only ping that relates to the IP address that wanted to focus on.
diagnose debug flow filter addr x.x.x.x
diagnose debug flow filter proto 1
Note: x.x.x.x is the IP address that we want to filter. proto 1 is ping traffic (ICMP).
These can also be adjusted for 2 IP addresses or IP address ranges:
diagnose debug flow filter addr x.x.x.x y.y.y.y and / or
diagnose debug flow filter proto 1
The added 'and' will only show packets for ICMP between x.x.x.x and y.y.y.y, 'or' will show ICMP sourced or destined from both IP Addresses.
diagnose debug flow filter addr x.x.x.x y.y.y.y
diagnose debug flow filter proto 1
Without the boolean 'and' / 'or' this will show anything in the range between x.x.x.x and y.y.y.y:
Example:
diagnose debug flow filter addr 10.0.0.1 10.0.0.254
diagnose debug flow filter proto 1
diagnose debug flow filter ?
vf: any
proto: 1-1
host addr: 10.0.0.1-10.0.0.254
Host saddr: any
Host daddr: any
port: any
sport: any
dport: any
-
Filter only port number.
diagnose debug flow filter port Y
Y = port number (such as 80 (http) ,25(smtp) ) -
Filter only IP address and port number.
diagnose debug flow filter addr x.x.x.x
diagnose debug flow filter port 80 -
Filter only source or destination port.
diagnose debug flow filter sport 80 <-- Filter with the source port 80.
Or:
diagnose debug flow filter dport 25 <-- Filter with the destination port 25.
-
Filter only the source IP address or destination IP address.
diagnose debug flow filter saddr x.x.x.x <- Filter with the source IP address x.x.x.x.
Or:
diagnose debug flow filter daddr y.y.y.y <- Filter with the destination IP address y.y.y.y.
-
Filter only the specific virtual domain.
diagnose debug flow filter vd X <- 'X' is the index of the virtual domain.
Examples of results that may be obtained from a debug flow :
The following is an example of debug flow output for traffic that has no matching Firewall Policy, hence blocked by the FortiGate.id=20085 trace_id=319 func=resolve_ip_tuple_fast line=2825 msg="vd-root received a packet(proto=6, 192.168.129.136:2854->192.168.96.153:1863) from port3."
id=20085 trace_id=319 func=resolve_ip_tuple line=2924 msg="allocate a new session-013004ac"
id=20085 trace_id=319 func=vf_ip4_route_input line=1597 msg="find a route: gw-192.168.150.129 via port1"
id=20085 trace_id=319 func=fw_forward_handler line=248 msg=" Denied by forward policy check"The following is an example of debug flow output for traffic going into an IPSec tunnel in Policy-based. This is a working scenario.
See traffic is matching and processed by Firewall Policy #2.
id=20085 trace_id=1 msg="vd-root received a packet (proto=1, 10.72.55.240:1->10.71.55.10:8) from internal."
id=20085 trace_id=1 msg="allocate a new session-00001cd3"
id=20085 trace_id=1 msg="find a route: gw-192.168.56.230 via wan1"
id=20085 trace_id=1 msg="Allowed by Policy-2: encrypt"
id=20085 trace_id=1 msg="enter IPsec tunnel-RemotePhase1"
id=20085 trace_id=1 msg="encrypted, and send to 192.168.225.22 with source 192.168.56.226"
id=20085 trace_id=1 msg="send to 192.168.56.230 via intf-wan1“
id=20085 trace_id=2 msg="vd-root received a packet (proto=1, 10.72.55.240:1-10.71.55.10:8) from internal."
id=20085 trace_id=2 msg="Find an existing session, id-00001cd3, original direction"
id=20085 trace_id=2 msg="enter IPsec ="encrypted, and send to 192.168.225.22 with source 192.168.56.226“ tunnel-RemotePhase1"
id=20085 trace_id=2 msgid=20085 trace_id=2 msg="send to 192.168.56.230 via intf-wan1"Other information messages are explained in the article 'Troubleshooting Tip: debug flow messages 'iprope_in_check() check failed, drop' - 'Denied by forward policy check' - 'reverse path check fail, drop'.
Step 6: Session list.
One further step is to look at the firewall session. For this, some filters may be used to reduce the output; see the following example:
diag sys session filter src PC1
diag sys session list
Or:
diag sys session filter dst PC1
diag sys session list
The analysis of the output of this command is further detailed in the related article below (FortiGate Firewall session list information).
To clear all sessions corresponding to a filter:
diag sys session filter dst PC1
diag sys session clear
Related articles:
Troubleshooting Tool: Using the FortiOS built-in packet sniffer
Troubleshooting Tip: FortiGate session table information
Troubleshooting Tip : How to use the FortiGate sniffer and debug flow in presence of NP2 ports
Technical Note: Details about FortiOS RPF (Reverse Path Forwarding), also called Anti-Spoofing
Technical Tip: How to filter for IP addresses and address ranges in debug flow
