There could be different scenarios where packets enter the FortiGate but do not leave. In this case, the packets are dropped even though the firewall policy is set to 'allow' and routes are fine. One particular scenario may be responsible for this behavior.
The behavior:
Specific traffic from IP A (VLAN X) to IP Z (VLAN Y) hits the firewall. The route is available and the policy hit (action allow) is as expected, but no traffic leaves the FortiGate.
This is a real case where, after FortiGate HA failed over, the setup that previously worked has stopped working.
Troubleshooting steps:
- The packet sniffer shows traffic came in but never left this is not expected and there could be an issue with FortiGate.
diagnose sniffer packet any 'host 52.1.1.1' 4 0 l interfaces=[any] filters=[host 52.1.1.1] 2024-04-01 17:06:19.490476 port26.2222 in 10.20.90.150.59610 -> 52.1.1.1.443: syn 1418635120 2024-04-01 17:06:20.491656 port26.2222 in 10.20.90.150.59620 -> 52.1.1.1.443: syn 2217951958 2024-04-01 17:06:24.450965 port26.2222 in 10.20.90.150.42652 -> 52.1.1.1.443: syn 1063576647
- Flow debugging shows traffic hit the expected/correct firewall policy and the action is 'allow'. This proves traffic was not dropped but somehow never left the egress interface.
id=20085 trace_id=1 func=print_pkt_detail line=5919 msg="vdomX:0 received a packet(proto=6, 10.20.90.150:43522->52.1.1.1:443) tun_id=0.0.0.0 from port26.2222. flag [S], seq 3125834714, ack 0, win 29200" id=20085 trace_id=1 func=init_ip_session_common line=6099 msg="allocate a new session-b9d6da83, tun_id=0.0.0.0" id=20085 trace_id=1 func=__vf_ip_route_input_rcu line=1996 msg="find a route: flag=00000000 gw-10.10.144.1 via Port25.333" id=20085 trace_id=1 func=get_new_addr line=1227 msg="find DNAT: IP-10.10.144.18, port-43522" id=20085 trace_id=1 func=fw_forward_handler line=886 msg="Allowed by Policy-9999: SNAT" id=20085 trace_id=1 func=ip_session_confirm_final line=3180 msg="npu_state=0x4001000, hook=4" id=20085 trace_id=1 func=__ip_session_run_tuple line=3517 msg="SNAT 10.20.90.150->10.10.144.18:43522"
- The route table looks correct and it is the default route in this case. This is consistent with the debug flow result above.
get router info routing-table details 52.1.1.1
Routing table for VRF=0
Routing entry for 0.0.0.0/0 Known via "static", distance 10, metric 0, best
* 10.10.144.1, via Port25.333
A ping to the next hop was tried here and failed. Since the user was unsure if the ping is allowed, no conclusion was made.
- A check of egress interface statistics shows no traffic was ever received on this interface: traffic only left.
fnsysctl ifconfig Port25.333
Port25.333 Link encap:Ethernet HWaddr 00:09:0F:09:04:18 inet addr:10.10.144.15 Bcast:10.10.144.255 Mask:255.255.255.0 UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:765025 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0 Bytes) TX bytes:35191546 (33.6 MB)
- Upon sniffing traffic on an egress interface, the result shows all traffic leaving consists of ARP requests leaving the interface: there is no incoming traffic.
diagnose sniffer packet Port255.333 "" 4
interfaces=[Port25.333] filters=[] 2.108995 Port25.333 -- arp who-has 10.10.144.1 tell 10.10.144.15 <- This is the default gateway IP (144.1), the connecting device IP. ...
Show arp table (get system arp) also finds 10.10.144.1 not available.. this is expected as the arp request never got answered.
- After checking physical port statistics, the result (shown below) shows that there was no issue on port25. It was vlan 333 not functioning as intended. This proves there is no physical connection issue, which also explains why the port and VLAN are green in the GUI.
get hardware nic port25
Description: FortiASIC NP7 Adapter Driver Name: FortiASIC Unified NPU Driver pid :25 oid :152 vid :26 macid :24 eif_id :127 promiscous :1 mtu :1500 netdev oid :152 dev-flags :1103 dev-promis :1 Current_HWaddr 00:09:0f:09:04:18 Permanent_HWaddr d4:76:a0:b8:ce:b6 ==== Default Link Settings ===== auto-nego :Disable s_speed :10000 s_duplex :Full ==== Current Link Settings ===== auto-nego :Disable s_status :Up s_speed :10000 s_duplex :Full FEC :OFF (0x4) FEC_cap :CL91 CL74 (0x18) ==== Link Status =============== Admin :Up link_status :Up Speed :10000 Duplex :Full ==== Netdev Status ============= dev_running :Yes dev_carrier :On ==== Host Counters ============= rx_pkts :519287366 rx_bytes :24854338335 tx_pkts :768467 tx_bytes :35349878 tx_drop :0 tx_e_busy :0 tx_e_noheadroom :0 tx_e_oid :0 tx_e_adapter :0 tx_e_pad :0 tx_e_other :0 ==== Switch Poll Counters ====== tx_pkts :768467 tx_bytes :49181976 rx_pkts :519309910 rx_bytes :34244030680 ==== Netdev Counters =========== Rx Pkts :519309910 Rx Bytes :34244030680 Tx Pkts :768467 Tx Bytes :49181976
At this stage, the issue has been narrowed down to the connection between FortiGate and the connecting device (it is a switch).
As the connection used to work correctly and only failed after the HA failover, the user checked the config on the switches that connect to the previous primary unit and current primary unit and noticed a configuration difference.
In this particular case, the configuration that worked shows the switchport mode is 'trunk', while the configuration having issues shows the switchport mode is 'access'.
After the change, traffic flows as expected.
Conclusion:
When traffic does not leave the egress interface as expected, it is possible there is no ARP entry for the next hop, in such case, FortiGate cannot forward the traffic out.
|