Troubleshooting Tip: Common drop reasons when using diagnose debug flow commands for troubleshooting, and possible solutions

Description

This article describes possible reasons for packet drop messages when using diagnose debug flow commands for connectivity issues, and how to resolve them.

Scope

FortiGate.

Solution

When troubleshooting connectivity problems to or through a FortiGate with the 'diagnose debug flow' commands, the following messages may appear if packets are dropped:

1. 'iprope_in_check() check failed, drop'.

2. 'Denied by forward policy check'.

3. 'reverse path check fail, drop'.

4. 'iprope_in_check() check failed on policy 0, drop'.

5. 'No matching IPsec selector, drop'

6. 'Denied by quota check'

See the following article for more details about troubleshooting with the 'diagnose debug flow' command: Troubleshooting Tip: Basic FortiGate connectivity diagnostics.

Here are some of the most common reasons for such drops and the possible remediations:

1. Possible root causes for 'iprope_in_check() check failed, drop'.

1. When accessing the FortiGate for remote management (ping, telnet, SSH...), the service that is being accessed is not enabled on the interface.

For example, pinging or telnetting the DMZ interface of a FortiGate, IP address 10.50.50.2, where ping and telnet are not enabled:

diagnose debug reset      

diagnose debug flow filter daddr 10.50.50.2
diagnose debug console timestamp enable
diagnose debug flow show iprope enable

diagnose debug flow show function-name enable
diagnose debug flow trace start 100           
diagnose debug enable

id=36870 pri=emergency trace_id=1 msg="vd-root received a packet(proto=1,10.50.50.1:4608->10.50.50.2:8) from dmz."
id=36870 pri=emergency trace_id=1 msg="allocate a new session-0000d5ad"
id=36870 pri=emergency trace_id=1 msg="iprope_in_check() check failed, drop"
id=36870 pri=emergency trace_id=8 msg="vd-root received a packet(proto=6, 10.50.50.1:1160->10.50.50.2:23) from dmz."
id=36870 pri=emergency trace_id=8 msg="allocate a new session-0000d96a"
id=36870 pri=emergency trace_id=8 msg="iprope_in_check() check failed, drop"

2. When accessing the FortiGate for remote management (ping, telnet, SSH, ...), the service that is being accessed is enabled on the interface, but there are trusted hosts configured that do not match the source IP of the incoming packets.

Example: Pinging the DMZ interface of a FortiGate with IP address 10.50.50.2 from source IP 10.50.50.1 with trusted hosts configured as follows:

The 'show system admin <username>' command will show all the available administrator configurations. In the command below, an administrator named admin is used as an example.

FortiGate # show system admin admin
    config system admin
        edit "admin"
            set trusthost1 10.20.20.0 255.255.255.0
[...]

id=36870 pri=emergency trace_id=26 msg="vd-root received a packet(proto=1, 10.50.50.1:5632->10.50.50.2:8) from dmz."
id=36870 pri=emergency trace_id=26 msg="allocate a new session-0000da15"
id=36870 pri=emergency trace_id=26 msg="iprope_in_check() check failed, drop"

The ports listened to by FortiGate can be validated with the following commands:

FortiGate # diagnose sys tcpsock
FortiGate # diagnose sys udpsock

3. When accessing a FortiGate interface for remote management (ping, telnet, SSH, ...) through another interface of the same FortiGate, no firewall policy is present. Example: pinging wan2 at the IP address 10.70.70.1 through DMZ, but there is no firewall policy from DMZ to wan2.

4. Make sure that the remote management is through the customs port, the customs port is added to the service, and the firewall policy has the service added. 

5. The issue may also be due to the local-in policy, which can be verified with show full firewall local-in-policy.

Starting from v7.6.0, it is possible to view/check the local-in-policy via the GUI:

• Technical Tip: Creating a Local-In policy (IPv4 and IPv6) on GUI

• GUI support for local-in policies 

6. A VIP parameter must be set as detailed in Technical Tip: Configuration Example - Using VIP (Virtual IP) for Port Translation only.
   
   

7. An improper error can also be thrown if the default admin ports for SSH or HTTPS/HTTP are modified to custom ports and the admin is trying to access on a different port other than the configured custom port.

config system global
    set admin-sport 443   
    set admin-port 80
    set admin-ssh-port 22
end

The values shown above are the default. Cross-verify whether an attempt is being made to access the correct port.

8. If it is using the interface IP address as the gateway in a policy route (by mistake), FortiGate will try to send it to the gateway IP (which belongs to the FortiGate root VDOM). This will trigger an 'iprope_in_check() check failed, drop' message. Also, if the gateway is set to 0.0.0.0, the FortiGate will look for a route via the same interface in the routing table. In other words, it gives preference to the Interface.
   
   

9. iPrope Check Failure can be observed due to Geo-Location restrictions on SSL VPN settings. When trying to use SSL VPN from a country not included in the Geo restriction on the SSL VPN settings of FortiGate, the connection will not be successful. This is because the connection is being attempted from a different region than the one being allowed, and the debug flow will show the 'iprope_in_check() check failed on policy 0, drop' message.
   
   

10. In cases with static routes: if the gateway IP is used in a VIP/IP-Pool configuration, FortiGate will identify the gateway as a local IP address. The traffic will be denied with the error 'iprope_in_check() check failed, drop'. Remove any such wrong configuration as it is not correct logically.    

See this article for more information: Technical Tip: Iprope Check Failure observed due to Geo-Location restriction on SSL VPN settings.

2. Possible root cause for 'iprope_in_check() check failed on policy 0, drop'.

1. When trying to access the FortiGate for remote management through a dial-up IPsec, and it fails due to the 'iprope_in_check() check failed on policy 0, drop' message appearing on the debug flow, it means that it cannot match a policy for this incoming traffic. 
   If the rest of the IPSec configuration (policies, selectors) looks fine, the issue might be in the user group authentication.

When configuring authentication for the dial-up tunnel on FortiGate, there are two ways to match the users with the proper groups: 

• Match the group on the phase 1 setting with the 'set authusrgrp'.

• Match the groups on the firewall policies for the IPsec tunnel created. 
  

The 'iprope_in_check() check failed on policy 0, drop' message appears if the user group is configured both in the phase 1 settings and the corresponding policy.

Therefore, the authentication needs to take place only in one of these two ways.

The key point of the second way is that the authentication of the users can be more flexible depending on different policies, and therefore different ways of handling/inspecting them.    

2. When connecting to SSL VPN on FortiGate and the FortiClient's public IP address is not in the SSL VPN 'source-address' list, or it is in the source-address list but 'source-address-negate' is enabled. The FortiGate will drop the connection and also show the  'iprope_in_check() check failed on policy 0, drop' message on the flow debug.

config vpn ssl settings
    set source-address <address>
    set source-address-negate [enable|disable]
end

3. Another reason for this error is the captive portal settings at the interface level. 

diagnose debug reset      
diagnose debug flow filter addr <IP Address>
diagnose debug flow show iprope enable
diagnose debug flow trace start 5           
diagnose debug enable

id=65308 trace_id=11 func=iprope_policy_group_check line=4892 msg="after check: ret-matched, act-drop, flag-00000020, flag2-00000000"
id=65308 trace_id=11 func=iprope_fwd_auth_check line=902 msg="iprope_auth_portal_check() result: ret-matched, act-drop"
id=65308 trace_id=11 func=fw_forward_handler line=830 msg="Denied by forward policy check (policy 0)"

The workaround is to remove the captive portal or add exemptions in settings.

3. Possible root causes for 'Denied by forward policy check'.

1. There is no firewall policy matching the traffic that needs to be routed or forwarded by the FortiGate (traffic will hit the Implicit Deny rule).

2. The traffic matches a DENY firewall policy.

3. The traffic matches an ALLOW firewall policy, but a DISCLAIMER is enabled. Consequently, traffic will not be accepted unless the end user accepts the HTTP disclaimer proposed by FortiGate while browsing an external site. This is usually encountered on IOT devices that are unable to select the disclaimer 'Yes, I agree' option, causing all access to be denied.

For example (messages are similar for both root causes):

id=36870 pri=emergency trace_id=19 msg="vd-root received a packet(proto=1, 10.50.50.1:7680->10.60.60.1:8) from dmz."
id=36870 pri=emergency trace_id=19 msg="allocate a new session-0000007d"
id=36870 pri=emergency trace_id=19 msg="Denied by forward policy check"

4. Possible root causes for 'reverse path check fail, drop.'

This is detailed in the related article at the end of this page: 'Technical Tip: Details about FortiOS RPF (Reverse Path Forwarding), also called Anti-Spoofing'.

5. Possible root causes for 'No matching IPsec selector, drop.'

This drop message indicates that an ESP (encrypted packet) was received on the FortiGate and dropped because the SPI values in these packets for the source and destination IP addresses in the header do not match any IPSec phase 2 selectors that are currently active on the device. This usually happens if the VPN tunnel (and the corresponding phase 2 SAs) are down on one end but still active on the other side. Verify this with the CLI command 'diagnose vpn tunnel list' on both the peers and check if the tunnel is up and if phase 2 SAs for the matching source and destination subnets are active or not. There could be other phase 2 SAs for different source/destination subnets for the same tunnel that are active, but not specifically for the dropped packet's source and destination ip addresses.

The following is an example output. Verify that the highlighted 'src:' and 'dst:' subnets match the source and destination ip addresses of the packet corresponding to the drop.

FortiGate # diagnose vpn tunnel list
list all ipsec tunnel in vd 1
name=S2S-VPN ver=2 serial=1 10.10.0.1:0->10.20.0.1:0 nexthop=10.10.0.2 tun_id=10.20.0.1 tun_id6=::10.20.0.1 status=up dst_mtu=1500 weight=1
bound_if=0 real_if=7 lgwy=static/1 tun=intf mode=auto/1 encap=none/33312 options[8220]=frag-rfc run_state=0 role=primary accept_traffic=1 overlay_id=0
proxyid_num=1 child_num=0 refcnt=3 ilast=974 olast=258861 ad=/0
stat: rxp=396 txp=396 rxb=42768 txb=42768
dpd: mode=off
natt: mode=none draft=0 interval=0 remote_port=0
fec: egress=0 ingress=0
proxyid=P2_S2S-VPN proto=0 sa=1 ref=2 serial=1 auto-negotiate
src: 0:192.168.10.0/255.255.255.0:0
dst: 0:172.16.20.0/255.255.255.0:0
SA: ref=3 options=18227 type=00 soft=0 mtu=1422 expire=1416/0B replaywin=2048
seqno=1 esn=0 replaywin_lastseq=00000000 qat=0 rekey=0 hash_search_len=1
life: type=01 bytes=0/0 timeout=3299/3600
dec: spi=8bc0068e esp=aes key=32 677bed645e9bcc95055bd48c92fd96283b2ffc188795246f2b209f1cd8ae527f
ah=sha512 key=64 b558182ecd8a1f028a9d5601b834d2e5344f8863b957e32a15c50519291cf2fe0e2bad7a542578daac652a9d362ad09600a2a3871c1a1e20d90e1e680422d617
enc: spi=a37f9032 esp=aes key=32 8404c6721e2b0d06a026ba17778e9b1fe2cee7c91bb472a51849d9ae39c1abdf
ah=sha512 key=64 24a3838de3794b0448030017176bb88c44929429c1e6584b4b2adaed2508504797c0512d37b8ec1d296863667e8cd113daae1ad6686a24b571ea4a5be88bf719
dec:pkts/bytes=0/0, enc:pkts/bytes=0/0
npu_flag=00 npu_lgwy=0.0.0.0:0 npu_rgwy=0.0.0.0:0
npu_selid=0 dec_npuid=0 enc_npuid=0
FortiGate-1001F (vpn) #

6. Possible root causes for 'Denied by quota check'.

This debug message indicates that the packet was dropped by a traffic shaper, due to one of the configured (or default) thresholds being exceeded by that session. Review the traffic shaper configuration and shaper diagnostic cli outputs to verify which specific threshold has been exceeded, and tune the shaper accordingly if needed. Run the following commands multiple times to check if either shared traffic policies or per-ip traffic policies are dropping the packets and track the drop counter increments. Refer to this article to verify whether traffic shaping is being applied to the dropped packet. Technical Tip: How to verify that traffic shaping is being applied for traffic.

For shared policies:

FortiGate # diagnose firewall shaper traffic-shaper list
name guarantee-100kbps
maximum-bandwidth 131072 KB/sec
guaranteed-bandwidth 12 KB/sec
current-bandwidth 0 B/sec
priority 2
overhead 0
tos ff
packets dropped 2172
bytes dropped 831821
. . .

For per-ip policies:

diagnose firewall shaper per-ip-shaper list

For combined stats:

diagnose firewall shaper traffic-shaper stats

Related articles:

• Troubleshooting Tip: First steps to troubleshoot connectivity problems to or through a FortiGate with sniffer, debug flow, session list, routing table

• Technical Tip: Best practice when IPSec VPN is bound to loopback interface

• FortiGate log information: traffic log with firewall policy of 0 (zero) "policyid=0"

• Technical Tip: Details about FortiOS RPF (Reverse Path Forwarding), also called Anti-Spoofin

• Troubleshooting Tip: Debug flow shows error ‘iprope_auth_portal_check() result: ret-matched, act-drop’

• Technical Tip: SNMP traffic is blocked due to a local-in policy violation 

• Technical Tip: Reverse path-check failure with route present in the routing-table via inbound interface

• Troubleshooting Tip: Traffic shaping Troubleshooting