Troubleshooting Tip: NP7 troubleshooting

[DiegoBernardelli]

Description	This article describes the main points of concern when troubleshooting offloading/performance issues on NP7 devices.
Scope	FortiGate, FortiOS, NP7.
Solution	When it is needed to investigate offloading the goal should be to reply to 3 main questions: Is the NP7 processor dropping packets? If yes where and why? Is the NP7 processor under heavy load? Is the NP7 processor stuck? FortiOS offers a complete set of commands to investigate and analyze the above possible issues. Is the NP7 processor dropping packets? If yes where and why? To check drops in NP the administrator can run the following command: diagnose npu np7 dce-drop-all <all | np7 id> <verbosity**> **{0|b|brief}: Show non-zero counters, {1|v|verbose}: Show all the counters, {2|c|clear}: Clear counters diagnose npu np7 dce-drop-all 0 1 <EIF drop counters> [NP7_0] Counter EIF_0 EIF_1 EIF_2 EIF_3 EIF_4 EIF_5 EIF_6 EIF_7 Total ------------------------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ------------ [0]l2_parse 0 0 0 0 0 0 0 0 0 [1]l3l4_parse 0 0 0 0 0 0 0 0 0 [2]ipv4_ver 0 0 0 0 0 0 0 0 0 ..... <HTX drop counters> [NP7_0] Counter HTX_0 HTX_1 HTX_2 HTX_3 Total ------------------------- ---------- ---------- ---------- ---------- ------------ [0]l2_parse 0 0 0 0 0 [1]l3l4_parse 0 0 0 0 0 .... <DFR drop counters> [NP7_0] Counter DFR ------------------------- ---------- [0]l2_parse 0 [1]l3l4_parse 0 .... <IPSec drop counters> [NP7_0] Counter Value ----------------- ---------- ipsec_enc_chk 0 ipsec_dec_auth 0 ipsec_dec_chk_ar 0 ----------------- ---------- .... <DSW drop counters> [NP7_0] SRC_mod -> DST_mod Drop ---------- ---------- ---------- EIF0 -> EIF0 0 EIF0 -> EIF1 0 EIF0 -> EIF2 0 EIF0 -> EIF3 0 EIF0 -> EIF4 0 .... This command can be very verbose but it contains for each NP7 module and sub-engine stats about anomalies and drop. The idea is to loop this command over time and check which counter is constantly increasing so that the focus can be moved to a specific part of the NP7 processor and eventually a ticket can be raised to the Fortinet Support. Is the NP7 processor under heavy load? NP7 is connected to the ISF through 2x CGMAC interfaces, each CGMAC is 100Gbps  for a total of 200Gbps With the command: diagnose npu np7 cgmac-stats <all | np7 id> <verbosity**> it is possible to check how many packets and throughput are received and transmitted by each NP7 processor.  diagnose npu np7 cgmac-stats 0 0 [NP7_0] CGMAC0: --------------- --------------- --------------- --------------- --------------- --------------- Counters LANE0 LANE1 LANE2 LANE3 Total --------------- --------------- --------------- --------------- --------------- --------------- rx_bcast 1568 1595 1581 1562 6306 rx_mcast 107821 107428 107685 108124 431058 rx_ucast 227803 228267 229735 229616 915421 rx_goodoctet 227165253 227183155 228702445 228728383 911779236 rx_octet 227165253 227183155 228702445 228728383 911779236 --------------- --------------- --------------- --------------- --------------- --------------- tx_bcast 350 333 341 369 1393 tx_mcast 18063 18064 18063 18063 72253 tx_ucast 533957 533972 533965 533937 2135831 tx_goodoctet 374562928 364020455 374612480 364029464 1477225327 tx_octet 374562928 364020455 374612480 364029464 1477225327 --------------- --------------- --------------- --------------- --------------- --------------- RX_RATE(pps) 0 0 0 0 0 RX_RATE(kbps) 1 0 0 0 2 TX_RATE(pps) 0 0 0 0 0 TX_RATE(kbps) 0 0 0 0 0 --------------- --------------- --------------- --------------- --------------- --------------- CG_FULL 0 CGMAC1: --------------- --------------- --------------- --------------- --------------- --------------- Counters LANE0 LANE1 LANE2 LANE3 Total --------------- --------------- --------------- --------------- --------------- --------------- rx_bcast 39572 39496 39743 39810 158621 rx_mcast 240155 241243 240570 241309 963277 rx_ucast 387952 388015 387442 387205 1550614 rx_goodoctet 405397003 406242711 405111720 405148982 1621900416 rx_octet 405397003 406242711 405111720 405148982 1621900416 --------------- --------------- --------------- --------------- --------------- --------------- tx_bcast 345 359 348 324 1376 tx_mcast 1 0 3 1 5 tx_ucast 417439 417426 417434 417459 1669758 tx_goodoctet 268194630 270121006 268254445 270368412 1076938493 tx_octet 268194630 270121006 268254445 270368412 1076938493 --------------- --------------- --------------- --------------- --------------- --------------- RX_RATE(pps) 0 0 0 0 0 RX_RATE(kbps) 1 0 0 0 2 TX_RATE(pps) 0 0 0 0 0 TX_RATE(kbps) 0 0 0 0 0 --------------- --------------- --------------- --------------- --------------- --------------- CG_FULL 0 In addition to each RX_TX rate metric to see the actual CGMAC bandwidth there is also the GC_FULL counter, this counter increases when there is congestion at the CGMAC level. With the following command, it is also possible to check the usage in percentage for every NP7 module: diagnose npu np7 pmon <all | np7 id> <verbosity**> diagnose npu np7 pmon 0 0 [NP7_0] EIF0_IGR EIF1_IGR EIF1_EGR EIF1_EGR HRX HTX DFR -------- -------- -------- -------- -------- -------- -------- -------- Usage% 0 0 0 0 0 0 0 -------- -------- -------- -------- -------- -------- -------- -------- SSE0 SSE1 SSE2 SSE3 -------- -------- -------- -------- -------- Usage% 1 1 1 1 -------- -------- -------- -------- -------- IPSEC IPTI IPTO L2TI L2TO VEP IVS -------- -------- -------- -------- -------- -------- -------- -------- Usage% 0 0 0 0 0 0 0 -------- -------- -------- -------- -------- -------- -------- -------- PLE MSE SYNK DSE NSS -------- -------- -------- -------- -------- -------- Usage% 0 0 0 12 0 -------- -------- -------- -------- -------- -------- * EIFx_IGR: EIF ingress, EIFx_EGR: EIF egress 3. Is the NP7 processor stuck? Each packet reaching the NP7 processor is stored on an allocated buffer and it is freed when the packet is processed and leaves the FortiGate (buffers used to store packets are limited). It might happen under specific conditions that buffers are not freed when packets are processed, this is called a PBA leak (PBA: packet buffer allocator). The command: diagnose npu np7 pba 0, compares a normal usage of buffers with the current usage and shows the delta. diagnose npu np7 pba all [NP7_0] normal current Delta Empty pba 00003f7c 00003f61 27 0 dba 00001ddf 00001ddc 3 hba 00000ff5 00000ff5 0 !!!Leak!!! NOTE: '!!! Leak !!!' is printed every time one of the delta is greater than 1. Having a positive value displayed is normal when traffic is being processed by the NP7, an administrator should be concerned when the delta value is high and constantly increasing over time as it means that the buffers are not being released correctly. Related documents: Troubleshooting Tip: NPU configuration commands (NP4, NP6, NP7)  NP7 acceleration  NP7 Specific Operational Topics

[vweis]

Very useful. Thank you for this.