Packet loss through the Fortigate act-drop

[flamer]

Hi,

we started getting packet loss through one interface of our 601E, SSH sessions on the far side of the fortigate are very slow and lagging, pinging to those ssh hosts results in around 5-20% packet loss, pinging from either side of the fortigate to the fgt IP address results in loss also. Pinging from the Fortigate outside interface to the internet is 100% clean.

I did a debug whilst running a ping, in the below example, 20 pings were sent, 3 failed with this pattern - ...!!!!!!!!!!!!!!!!!

Interestingly, I see 3 act-drops in sequence (just not the same sequence). Any explanation on what is going wrong here?

id=20085 trace_id=5289 func=print_pkt_detail line=5867 msg="vd-root:0 received a packet(proto=1, 10.10.4.50:126->10.9.4.1:2048) tun_id=0
.0.0.0 from vlan.3200. type=8, code=0, id=126, seq=3."
id=20085 trace_id=5289 func=init_ip_session_common line=6046 msg="allocate a new session-0009ca72, tun_id=0.0.0.0"
id=20085 trace_id=5289 func=iprope_dnat_check line=5336 msg="in-[vlan.3200], out-[]"
id=20085 trace_id=5289 func=iprope_dnat_tree_check line=827 msg="len=0"
id=20085 trace_id=5289 func=iprope_dnat_check line=5348 msg="result: skb_flags-02000000, vid-0, ret-no-match, act-accept, flag-00000000"
id=20085 trace_id=5289 func=vf_ip_route_input_common line=2611 msg="find a route: flag=84000000 gw-10.9.4.1 via root"
id=20085 trace_id=5289 func=iprope_access_proxy_check line=437 msg="in-[vlan.3200], out-[], skb_flags-02000000, vid-0"
id=20085 trace_id=5289 func=__iprope_check line=2272 msg="gnum-100017, check-ffffffffa002d740"
id=20085 trace_id=5289 func=iprope_policy_group_check line=4751 msg="after check: ret-no-match, act-accept, flag-00000000, flag2-00000000"
id=20085 trace_id=5289 func=iprope_in_check line=470 msg="in-[vlan.3200], out-[], skb_flags-02000000, vid-0"
id=20085 trace_id=5289 func=__iprope_check line=2272 msg="gnum-100011, check-ffffffffa002ea30"
id=20085 trace_id=5289 func=iprope_policy_group_check line=4751 msg="after check: ret-no-match, act-drop, flag-00000000, flag2-00000000"
id=20085 trace_id=5289 func=__iprope_check line=2272 msg="gnum-100001, check-ffffffffa002d740"
id=20085 trace_id=5289 func=iprope_policy_group_check line=4751 msg="after check: ret-no-match, act-accept, flag-00000000, flag2-00000000"
id=20085 trace_id=5289 func=__iprope_check line=2272 msg="gnum-10000e, check-ffffffffa002d740"
id=20085 trace_id=5289 func=__iprope_check_one_policy line=2025 msg="checked gnum-10000e policy-4294967295, ret-no-match, act-accept"
id=20085 trace_id=5289 func=__iprope_check_one_policy line=2025 msg="checked gnum-10000e policy-4294967295, ret-no-match, act-accept"
id=20085 trace_id=5289 func=__iprope_check_one_policy line=2025 msg="checked gnum-10000e policy-4294967295, ret-no-match, act-accept"
id=20085 trace_id=5289 func=__iprope_check_one_policy line=2025 msg="checked gnum-10000e policy-4294967295, ret-matched, act-accept"
id=20085 trace_id=5289 func=__iprope_check_one_policy line=2242 msg="policy-4294967295 is matched, act-drop"
id=20085 trace_id=5289 func=__iprope_check line=2289 msg="gnum-10000e check result: ret-matched, act-drop, flag-00000001, flag2-00000000"
id=20085 trace_id=5289 func=iprope_policy_group_check line=4751 msg="after check: ret-matched, act-drop, flag-00000001, flag2-00000000"
id=20085 trace_id=5289 func=__iprope_check line=2272 msg="gnum-10000f, check-ffffffffa002d740"
id=20085 trace_id=5289 func=__iprope_check_one_policy line=2025 msg="checked gnum-10000f policy-4294967295, ret-no-match, act-accept"
id=20085 trace_id=5289 func=__iprope_check_one_policy line=2025 msg="checked gnum-10000f policy-4294967295, ret-no-match, act-accept"
id=20085 trace_id=5289 func=__iprope_check_one_policy line=2025 msg="checked gnum-10000f policy-4294967295, ret-matched, act-accept"
id=20085 trace_id=5289 func=__iprope_check_one_policy line=2242 msg="policy-4294967295 is matched, act-accept"
id=20085 trace_id=5289 func=__iprope_check line=2289 msg="gnum-10000f check result: ret-matched, act-accept, flag-00000001, flag2-00000000"
id=20085 trace_id=5289 func=iprope_policy_group_check line=4751 msg="after check: ret-matched, act-accept, flag-00000001, flag2-00000000"
id=20085 trace_id=5290 func=print_pkt_detail line=5867 msg="vd-root:0 received a packet(proto=1, 10.9.4.1:126->10.10.4.50:0) tun_id=0.0.0.0 from local. type=0, code
=0, id=126, seq=3."
id=20085 trace_id=5290 func=resolve_ip_tuple_fast line=5953 msg="Find an existing session, id-0009ca72, reply direction"
id=20085 trace_id=5291 func=print_pkt_detail line=5867 msg="vd-root:0 received a packet(proto=1, 10.10.4.50:126->10.9.4.1:2048) tun_id=0.0.0.0 from vlan.3200. type=
8, code=0, id=126, seq=4."
id=20085 trace_id=5291 func=resolve_ip_tuple_fast line=5953 msg="Find an existing session, id-0009ca72, original direction"
id=20085 trace_id=5292 func=print_pkt_detail line=5867 msg="vd-root:0 received a packet(proto=1, 10.9.4.1:126->10.10.4.50:0) tun_id=0.0.0.0 from local. type=0, code
=0, id=126, seq=4."
id=20085 trace_id=5292 func=resolve_ip_tuple_fast line=5953 msg="Find an existing session, id-0009ca72, reply direction"
id=20085 trace_id=5293 func=print_pkt_detail line=5867 msg="vd-root:0 received a packet(proto=1, 10.10.4.50:126->10.9.4.1:2048) tun_id=0.0.0.0 from vlan.3200. type=
8, code=0, id=126, seq=5."
id=20085 trace_id=5293 func=resolve_ip_tuple_fast line=5953 msg="Find an existing session, id-0009ca72, original direction"
id=20085 trace_id=5294 func=print_pkt_detail line=5867 msg="vd-root:0 received a packet(proto=1, 10.9.4.1:126->10.10.4.50:0) tun_id=0.0.0.0 from local. type=0, code
=0, id=126, seq=5."
id=20085 trace_id=5294 func=resolve_ip_tuple_fast line=5953 msg="Find an existing session, id-0009ca72, reply direction"
id=20085 trace_id=5295 func=print_pkt_detail line=5867 msg="vd-root:0 received a packet(proto=1, 10.10.4.50:126->10.9.4.1:2048) tun_id=0.0.0.0 from vlan.3200. type=
8, code=0, id=126, seq=6."
id=20085 trace_id=5295 func=resolve_ip_tuple_fast line=5953 msg="Find an existing session, id-0009ca72, original direction"
id=20085 trace_id=5296 func=print_pkt_detail line=5867 msg="vd-root:0 received a packet(proto=1, 10.9.4.1:126->10.10.4.50:0) tun_id=0.0.0.0 from local. type=0, code
=0, id=126, seq=6."
id=20085 trace_id=5296 func=resolve_ip_tuple_fast line=5953 msg="Find an existing session, id-0009ca72, reply direction"
id=20085 trace_id=5297 func=print_pkt_detail line=5867 msg="vd-root:0 received a packet(proto=1, 10.10.4.50:126->10.9.4.1:2048) tun_id=0.0.0.0 from vlan.3200. type=
8, code=0, id=126, seq=7."
id=20085 trace_id=5297 func=resolve_ip_tuple_fast line=5953 msg="Find an existing session, id-0009ca72, original direction"
id=20085 trace_id=5298 func=print_pkt_detail line=5867 msg="vd-root:0 received a packet(proto=1, 10.9.4.1:126->10.10.4.50:0) tun_id=0.0.0.0 from local. type=0, code
=0, id=126, seq=7."
id=20085 trace_id=5298 func=resolve_ip_tuple_fast line=5953 msg="Find an existing session, id-0009ca72, reply direction"
id=20085 trace_id=5299 func=print_pkt_detail line=5867 msg="vd-root:0 received a packet(proto=1, 10.10.4.50:126->10.9.4.1:2048) tun_id=0.0.0.0 from vlan.3200. type=
8, code=0, id=126, seq=8."
id=20085 trace_id=5299 func=resolve_ip_tuple_fast line=5953 msg="Find an existing session, id-0009ca72, original direction"
id=20085 trace_id=5300 func=print_pkt_detail line=5867 msg="vd-root:0 received a packet(proto=1, 10.9.4.1:126->10.10.4.50:0) tun_id=0.0.0.0 from local. type=0, code
=0, id=126, seq=8."
id=20085 trace_id=5300 func=resolve_ip_tuple_fast line=5953 msg="Find an existing session, id-0009ca72, reply direction"
id=20085 trace_id=5301 func=print_pkt_detail line=5867 msg="vd-root:0 received a packet(proto=1, 10.10.4.50:126->10.9.4.1:2048) tun_id=0.0.0.0 from vlan.3200. type=
8, code=0, id=126, seq=9."
id=20085 trace_id=5301 func=resolve_ip_tuple_fast line=5953 msg="Find an existing session, id-0009ca72, original direction"
id=20085 trace_id=5302 func=print_pkt_detail line=5867 msg="vd-root:0 received a packet(proto=1, 10.9.4.1:126->10.10.4.50:0) tun_id=0.0.0.0 from local. type=0, code
=0, id=126, seq=9."
id=20085 trace_id=5302 func=resolve_ip_tuple_fast line=5953 msg="Find an existing session, id-0009ca72, reply direction"
id=20085 trace_id=5303 func=print_pkt_detail line=5867 msg="vd-root:0 received a packet(proto=1, 10.10.4.50:126->10.9.4.1:2048) tun_id=0.0.0.0 from vlan.3200. type=
8, code=0, id=126, seq=10."
id=20085 trace_id=5303 func=resolve_ip_tuple_fast line=5953 msg="Find an existing session, id-0009ca72, original direction"
id=20085 trace_id=5304 func=print_pkt_detail line=5867 msg="vd-root:0 received a packet(proto=1, 10.9.4.1:126->10.10.4.50:0) tun_id=0.0.0.0 from local. type=0, code
=0, id=126, seq=10."
id=20085 trace_id=5304 func=resolve_ip_tuple_fast line=5953 msg="Find an existing session, id-0009ca72, reply direction"
id=20085 trace_id=5305 func=print_pkt_detail line=5867 msg="vd-root:0 received a packet(proto=1, 10.10.4.50:126->10.9.4.1:2048) tun_id=0.0.0.0 from vlan.3200. type=
8, code=0, id=126, seq=11."
id=20085 trace_id=5305 func=resolve_ip_tuple_fast line=5953 msg="Find an existing session, id-0009ca72, original direction"
id=20085 trace_id=5306 func=print_pkt_detail line=5867 msg="vd-root:0 received a packet(proto=1, 10.9.4.1:126->10.10.4.50:0) tun_id=0.0.0.0 from local. type=0, code
=0, id=126, seq=11."
id=20085 trace_id=5306 func=resolve_ip_tuple_fast line=5953 msg="Find an existing session, id-0009ca72, reply direction"
id=20085 trace_id=5307 func=print_pkt_detail line=5867 msg="vd-root:0 received a packet(proto=1, 10.10.4.50:126->10.9.4.1:2048) tun_id=0.0.0.0 from vlan.3200. type=
8, code=0, id=126, seq=12."
id=20085 trace_id=5307 func=resolve_ip_tuple_fast line=5953 msg="Find an existing session, id-0009ca72, original direction"
id=20085 trace_id=5308 func=print_pkt_detail line=5867 msg="vd-root:0 received a packet(proto=1, 10.9.4.1:126->10.10.4.50:0) tun_id=0.0.0.0 from local. type=0, code
=0, id=126, seq=12."
id=20085 trace_id=5308 func=resolve_ip_tuple_fast line=5953 msg="Find an existing session, id-0009ca72, reply direction"
id=20085 trace_id=5309 func=print_pkt_detail line=5867 msg="vd-root:0 received a packet(proto=1, 10.10.4.50:126->10.9.4.1:2048) tun_id=0.0.0.0 from vlan.3200. type=
8, code=0, id=126, seq=13."
id=20085 trace_id=5309 func=resolve_ip_tuple_fast line=5953 msg="Find an existing session, id-0009ca72, original direction"
id=20085 trace_id=5310 func=print_pkt_detail line=5867 msg="vd-root:0 received a packet(proto=1, 10.9.4.1:126->10.10.4.50:0) tun_id=0.0.0.0 from local. type=0, code
=0, id=126, seq=13."
id=20085 trace_id=5310 func=resolve_ip_tuple_fast line=5953 msg="Find an existing session, id-0009ca72, reply direction"
id=20085 trace_id=5311 func=print_pkt_detail line=5867 msg="vd-root:0 received a packet(proto=1, 10.10.4.50:126->10.9.4.1:2048) tun_id=0.0.0.0 from vlan.3200. type=
8, code=0, id=126, seq=14."
id=20085 trace_id=5311 func=resolve_ip_tuple_fast line=5953 msg="Find an existing session, id-0009ca72, original direction"
id=20085 trace_id=5312 func=print_pkt_detail line=5867 msg="vd-root:0 received a packet(proto=1, 10.9.4.1:126->10.10.4.50:0) tun_id=0.0.0.0 from local. type=0, code
=0, id=126, seq=14."
id=20085 trace_id=5312 func=resolve_ip_tuple_fast line=5953 msg="Find an existing session, id-0009ca72, reply direction"
id=20085 trace_id=5313 func=print_pkt_detail line=5867 msg="vd-root:0 received a packet(proto=1, 10.10.4.50:126->10.9.4.1:2048) tun_id=0.0.0.0 from vlan.3200. type=
8, code=0, id=126, seq=15."
id=20085 trace_id=5313 func=resolve_ip_tuple_fast line=5953 msg="Find an existing session, id-0009ca72, original direction"
id=20085 trace_id=5314 func=print_pkt_detail line=5867 msg="vd-root:0 received a packet(proto=1, 10.9.4.1:126->10.10.4.50:0) tun_id=0.0.0.0 from local. type=0, code
=0, id=126, seq=15."
id=20085 trace_id=5314 func=resolve_ip_tuple_fast line=5953 msg="Find an existing session, id-0009ca72, reply direction"
id=20085 trace_id=5315 func=print_pkt_detail line=5867 msg="vd-root:0 received a packet(proto=1, 10.10.4.50:126->10.9.4.1:2048) tun_id=0.0.0.0 from vlan.3200. type=
8, code=0, id=126, seq=16."
id=20085 trace_id=5315 func=resolve_ip_tuple_fast line=5953 msg="Find an existing session, id-0009ca72, original direction"
id=20085 trace_id=5316 func=print_pkt_detail line=5867 msg="vd-root:0 received a packet(proto=1, 10.9.4.1:126->10.10.4.50:0) tun_id=0.0.0.0 from local. type=0, code
=0, id=126, seq=16."
id=20085 trace_id=5316 func=resolve_ip_tuple_fast line=5953 msg="Find an existing session, id-0009ca72, reply direction"
id=20085 trace_id=5317 func=print_pkt_detail line=5867 msg="vd-root:0 received a packet(proto=1, 10.10.4.50:126->10.9.4.1:2048) tun_id=0.0.0.0 from vlan.3200. type=
8, code=0, id=126, seq=17."
id=20085 trace_id=5317 func=resolve_ip_tuple_fast line=5953 msg="Find an existing session, id-0009ca72, original direction"
id=20085 trace_id=5318 func=print_pkt_detail line=5867 msg="vd-root:0 received a packet(proto=1, 10.9.4.1:126->10.10.4.50:0) tun_id=0.0.0.0 from local. type=0, code
=0, id=126, seq=17."
id=20085 trace_id=5318 func=resolve_ip_tuple_fast line=5953 msg="Find an existing session, id-0009ca72, reply direction"
id=20085 trace_id=5319 func=print_pkt_detail line=5867 msg="vd-root:0 received a packet(proto=1, 10.10.4.50:126->10.9.4.1:2048) tun_id=0.0.0.0 from vlan.3200. type=
8, code=0, id=126, seq=18."
id=20085 trace_id=5319 func=resolve_ip_tuple_fast line=5953 msg="Find an existing session, id-0009ca72, original direction"
id=20085 trace_id=5320 func=print_pkt_detail line=5867 msg="vd-root:0 received a packet(proto=1, 10.9.4.1:126->10.10.4.50:0) tun_id=0.0.0.0 from local. type=0, code
=0, id=126, seq=18."
id=20085 trace_id=5320 func=resolve_ip_tuple_fast line=5953 msg="Find an existing session, id-0009ca72, reply direction"
id=20085 trace_id=5321 func=print_pkt_detail line=5867 msg="vd-root:0 received a packet(proto=1, 10.10.4.50:126->10.9.4.1:2048) tun_id=0.0.0.0 from vlan.3200. type=
8, code=0, id=126, seq=19."
id=20085 trace_id=5321 func=resolve_ip_tuple_fast line=5953 msg="Find an existing session, id-0009ca72, original direction"
id=20085 trace_id=5322 func=print_pkt_detail line=5867 msg="vd-root:0 received a packet(proto=1, 10.9.4.1:126->10.10.4.50:0) tun_id=0.0.0.0 from local. type=0, code
=0, id=126, seq=19."
id=20085 trace_id=5322 func=resolve_ip_tuple_fast line=5953 msg="Find an existing session, id-0009ca72, reply direction"

[Mohamed_Gaber]

"pinging from either side of the fortigate to the fgt IP address results in loss".

Confirm that there is no loop on the network.

Do you have HA?

Test each node sparely. Try to power off one node to force the traffic through a node path.

Are the HA nodes synchronized?

[flamer]

Hi there is no loop, pinging devices in the same subnet of that interface results in loss also. It is a cluster, we have rebooted both members, both members have the same packet loss. HA is in sync.

What I mean was, traffic traversing the fgt gets packet loss, but traffic that does not traverse does not. IE fgt is .1 but .2 gets no loss to .3 but both get loss to .1 and vice versa.

The debug flow seemed to indicate packets were received and dropped