Technical Tip: Intermittent Packet Loss or Delay of Reply Traffic on EMAC VLAN Interface Under Heavy Traffic Load
Description
This article describes an issue where the reply traffic is intermittently delayed or dropped on the EMAC VLAN interface when handling a heavy traffic load.
Scope
FortiGate v7.2.7.
Solution
When FortiGate receives a significant amount of traffic burst on the EMAC VLAN interface, packet drops or delays in forwarding the packets are seen in the reply direction leading to connectivity issues.
Sample Configuration:
config system interface
edit "VLAN2"
set vdom "root"
set ip 172.16.3.16 255.255.0.0
set allowaccess ping https ssh snmp
set type emac-vlan
set role lan
set snmp-index 61
set interface "port1"
next
edit "VLAN231"
set vdom "root"
set ip 192.168.7.1 255.255.255.0
set allowaccess ping
set type emac-vlan
set role wan
set snmp-index 57
set interface "port2"
next
From the sniffers and debug flow, reply traffic may be received by FortiGate and matches the existing session but is not forwarded out of the device. Below is a sample log trace illustrating the issue:
2024-09-11 16:07:04.705572 VLAN2 in 172.16.1.24 -> 4.2.2.2: icmp: echo request
2024-09-11 16:07:04.705622 VLAN231 out 192.168.7.1 -> 4.2.2.2: icmp: echo request
2024-09-11 16:07:04.710852 VLAN231 in 4.2.2.2 -> 192.168.7.1: icmp: echo reply
The reply packet is not forwarded out VLAN2 out.
2024-09-11 16:07:04 id=65308 trace_id=257767 func=print_pkt_detail line=5857 msg="vd-root:0 received a packet(proto=1, 172.16.1.24:185->4.2.2.2:2048) tun_id=0.0.0.0 from VLAN2. type=8, code=0, id=185, seq=29690."
2024-09-11 16:07:04 id=65308 trace_id=257767 func=init_ip_session_common line=6043 msg="allocate a new session-27c27ad2, tun_id=0.0.0.0"
2024-09-11 16:07:04 id=65308 trace_id=257767 func=__vf_ip_route_input_rcu line=2001 msg="find a route: flag=00000000 gw-205.153.115.1 via VLAN231"
2024-09-11 16:07:04 id=65308 trace_id=257767 func=__iprope_tree_check line=535 msg="gnum-100004, use addr/intf hash, len=68"
2024-09-11 16:07:04 id=65308 trace_id=257767 func=get_new_addr line=1239 msg="find SNAT: IP-192.168.7.1(from IPPOOL), port-60602"
2024-09-11 16:07:04 id=65308 trace_id=257767 func=fw_forward_handler line=1000 msg="Allowed by Policy-15: SNAT"
2024-09-11 16:07:04 id=65308 trace_id=257767 func=ip_session_confirm_final line=3090 msg="npu_state=0x4000001, hook=4"
2024-09-11 16:07:04 id=65308 trace_id=257767 func=__ip_session_run_tuple line=3432 msg="SNAT 172.16.1.24->192.168.7.1:60602"
2024-09-11 16:07:04 id=65308 trace_id=257768 func=print_pkt_detail line=5857 msg="vd-root:0 received a packet(proto=1, 4.2.2.2:60602->192.168.7.1:0) tun_id=0.0.0.0 from VLAN231. type=0, code=0, id=60602, seq=29690."
2024-09-11 16:07:04 id=65308 trace_id=257768 func=resolve_ip_tuple_fast line=5945 msg="Find an existing session, id-27c27ad2, reply direction"
2024-09-11 16:07:04 id=65308 trace_id=257768 func=__ip_session_run_tuple line=3445 msg="DNAT 192.168.7.1:0->172.16.1.24:185"
2024-09-11 16:07:04 id=65308 trace_id=257768 func=__vf_ip_route_input_rcu line=2001 msg="find a route: flag=00000000 gw-0.0.0.0 via VLAN2"
2024-09-11 16:07:04 id=65308 trace_id=257768 func=npu_handle_session44 line=1322 msg="Trying to offloading session from VLAN231 to VLAN2, skb.npu_flag=00000000 ses.state=04000204 ses.npu_state=0x04000001"
2024-09-11 16:07:04 id=65308 trace_id=257768 func=fw_forward_dirty_handler line=438 msg="state=04000204, state2=00000001, npu_state=04000001"
This issue has been resolved in v7.2.11, v7.4.8 and v7.6.1.
