Fortigate dropping internal network and external network packets.

[Rait]

Hello!

I have a issue with fortigate 800c. It keeps dropping packets on high bandwidth load. Lets say I ping a 10.2.2.X server from 10.2.2.X server with 54 bytes, I lose around 5 packets in 30 seconds. When I increase the packet size to 65000 it loses 10-15 packets in 15 seconds.

I have debugged from console and nothing is blocked or dropped. Sniffer shows all packets are "ok", but I guess firewall drops them before it can process them. Switches are ok, cables are ok etc.

This is a new unit in our production so I tried to switch back to old Juniper and that did not drop packets at all. So it has to be the fortigate unit.

MTU on ports is default (1500 I guess).

Where could the problem be?

[emnoc]

I would start by looking at port errors. Sounds like duplex mismatch I like a few important things to look at in the below output but any errors or collision, is a good indicator of duplex mis-matches from the diag hardware dev nic < nicname >

ADFGT110C # diag hardware deviceinfo nic wan1 Description            Broadcom 570x Tigon3 Ethernet Adapter Part_Number            BCM95786T8600 PHY_Device_ID            5787 Driver_Name            tg3 Driver_Version            3.85l PCI_Vendor            0x14e4 PCI_Device_ID            0x169a PCI_Subsystem_Vendor        0x14e4 PCI_Subsystem_ID        0x969a PCI_Revision_ID            0xb002 PCI_Address            2:0.0 Memory                0xdfd00000 IRQ                10 System_Device_Name        wan1 Current_HWaddr            00:09:0f:09:01:08 Permanent_HWaddr        00:09:0f:ce:42:0f Link                up Speed                1000 Mbps Duplex                full FlowControl            Tx off, Rx off MTU_Size            1500 Rx_Packets            1143337352 Rx_Packets_Dropped        0 Tx_Packets            1021102004 Rx_Bytes            548759472 Tx_Bytes            3298808021 Rx_Errors            0 Tx_errors            0 Multicast            1601792 Collisions            0 Rx_Length_Errors        0 Rx_Over_Errors            13 Rx_Frame_Errors            0 Tx_aborted_Errors        0 Tx_carrier_errors        0 Rx_CRC_Errors            0 rx_pending            200 tx_pending            511 tg3_flags            8248ec05 tg3_flags2            380c9200 tg3_flags3            00002000 rx_rcb_ptr            00000054 rx_producer            00000054

[Rait]

emnoc wrote:

I would start by looking at port errors. Sounds like duplex mismatch I like a few important things to look at in the below output but any errors or collision, is a good indicator of duplex mis-matches from the diag hardware dev nic < nicname >

ADFGT110C # diag hardware deviceinfo nic wan1 Description            Broadcom 570x Tigon3 Ethernet Adapter Part_Number            BCM95786T8600 PHY_Device_ID            5787 Driver_Name            tg3 Driver_Version            3.85l PCI_Vendor            0x14e4 PCI_Device_ID            0x169a PCI_Subsystem_Vendor        0x14e4 PCI_Subsystem_ID        0x969a PCI_Revision_ID            0xb002 PCI_Address            2:0.0 Memory                0xdfd00000 IRQ                10 System_Device_Name        wan1 Current_HWaddr            00:09:0f:09:01:08 Permanent_HWaddr        00:09:0f:ce:42:0f Link                up Speed                1000 Mbps Duplex                full FlowControl            Tx off, Rx off MTU_Size            1500 Rx_Packets            1143337352 Rx_Packets_Dropped        0 Tx_Packets            1021102004 Rx_Bytes            548759472 Tx_Bytes            3298808021 Rx_Errors            0 Tx_errors            0 Multicast            1601792 Collisions            0 Rx_Length_Errors        0 Rx_Over_Errors            13 Rx_Frame_Errors            0 Tx_aborted_Errors        0 Tx_carrier_errors        0 Rx_CRC_Errors            0 rx_pending            200 tx_pending            511 tg3_flags            8248ec05 tg3_flags2            380c9200 tg3_flags3            00002000 rx_rcb_ptr            00000054 rx_producer            00000054

Output on port4 that has vlan interfaces.

fw1-tix # diagnose hardware deviceinfo nic port4 Description     :FortiASIC NP4 Adapter Driver Name     :FortiASIC Unified NPU Driver Version         :1.0 PCI Slot        :01:00.0 PCI_Revision_ID :1 Board           :fgt800c SN              :FG800C3914801794 Major ID        :19 Minor ID        :0 LIF ID          :5 NPU OID         :5 NPU OID_VID     :7 netdev flags    :0x00001303 Current_HWaddr   00:09:0f:09:00:14 Permanent_HWaddr 08:5b:0e:96:7a:8d Queue           :On rx_buffer_len   :2048 max_frame_size  :1522 min_frame_size  :278 MTU             :1500 Hidden          :No dyn_lif         :0 npu_cap         :0000600b ses_mask        :40027dcb cmd_in_queue    :0 half_id         :1 phy_addr        :0x07 phy_flags       :0x4002 medium_type     :Copper sw_port         :9 sw_np_port      :29 sw_stats_port   :0 phy_oid         :0x05 slot_id         :0x00 vid_phy[6]      :[0007][0000][0000][0000][0000][0000] vid_fwd[6]      :[0000][0000][0000][0000][0000][0000] oid_fwd[6]      :[0000][0000][0000][0000][0000][0000] ========== Link Status ========== Admin           :up PHY Status      :up PHY Speed       :1000 Duplex          :Full link_status     :1 rx_link_status  :1 remote_fault    :0 local_fault     :0 local_warning   :0 int_phy_link    :0 int_phy_reinit_cnt:0 link_andone     :0 serdes_mode     :SerDes ============ Counters =========== clst_tx_orig    :0 clst_tx_o_free  :0 clst_tx_redir   :0 clst_tx_rd_free :0 clst_tx_reply   :0 clst_tx_rp_free :0 dd_wo_eop       :0 tcp_udp_csum_err:49519 rx_error        :0 rx_crc_error    :0 rx_len_error    :0 rx_carrier      :0 rx_oversize     :0 rx_undersize    :0 tx_collision    :0 ip_sum_offload  :0 ipsec_dec       :0 ipsec_dec_drop  :0 ipsec_antireplay_f:0 ipsec_antireplay_p:0 aps_log         :0 Rx Pkts         :15840863264 Rx Bytes        :12734562129748 Tx Pkts         :18424647921 Tx Bytes        :14544615864915 Host Rx Pkts    :9289600739 Host Rx Bytes   :10985710481858 Host Tx Pkts    :6042606135 Host Tx Bytes   :997450169787 sw_rx_pkts      :2955977526 sw_rx_bytes     :4291837538 sw_tx_pkts      :1244797450 sw_tx_bytes     :1870594196 sw_rx_mc_pkts   :6801849 sw_rx_bc_pkts   :1825246 sw_np_rx_pkts   :18428762305 sw_np_rx_bytes  :14618787336616 sw_np_tx_pkts   :15844958527 sw_np_tx_bytes  :12798395296565 sw_np_rx_mc_pkts:15 sw_np_rx_bc_pkts:4509951

What would this mean:

tcp_udp_csum_err:49519

[ashukla_FTNT]

Rait wrote:

What would this mean:

tcp_udp_csum_err:49519

The means theh checksum was wrong and the Np4 hardware acclearator (asic) dropped these many packet. As these numbers are historical number, you have to start the traffic and monitor if this number is increasing, then it is clear that NP4 is dropping because checksum is wrong.

In pc (server) most of the network cards supports offloading the tcp/udp checksum calculation to the nic card. Disable this feature and try. At the same time do the capture on source pc and check the wireshark reports checksum as wrong.

Please note when checksum offload is enabled to NIC card, wireshark may report checksum is bad. In reality it is because of checksum offload to NIC, the actual checksum is not wrong.

[emnoc]

Op, if you want to get creative you can  get interface by interface stats using fsnsyctl

e.g

 fnsysctl cat /proc/net/dev

The output will be a column of

 Interface|       bytes    packets errs drop fifo other compressed mcast colls