|
When it is needed to investigate offloading the goal should be to reply to 3 main questions:
- Is the NP7 processor dropping packets? If yes where and why?
- Is the NP7 processor under heavy load?
- Is the NP7 processor stuck?
FortiOS offers a complete set of commands to investigate and analyze the above possible issues.
- Is the NP7 processor dropping packets? If yes, where and why?
To check drops in NP, the administrator can run the following command:
diagnose npu np7 dce-drop-all <all | np7 id> <verbosity**>
**{0|b|brief}: Show non-zero counters, {1|v|verbose}: Show all the counters, {2|c|clear}: Clear counters
diagnose npu np7 dce-drop-all 0 1
<EIF drop counters>
[NP7_0]
Counter EIF_0 EIF_1 EIF_2 EIF_3 EIF_4 EIF_5 EIF_6 EIF_7 Total
------------------------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ------------
[0]l2_parse 0 0 0 0 0 0 0 0 0
[1]l3l4_parse 0 0 0 0 0 0 0 0 0
[2]ipv4_ver 0 0 0 0 0 0 0 0 0
.....
<HTX drop counters>
[NP7_0]
Counter HTX_0 HTX_1 HTX_2 HTX_3 Total
------------------------- ---------- ---------- ---------- ---------- ------------
[0]l2_parse 0 0 0 0 0
[1]l3l4_parse 0 0 0 0 0
....
<DFR drop counters>
[NP7_0]
Counter DFR
------------------------- ----------
[0]l2_parse 0
[1]l3l4_parse 0
....
<IPSec drop counters>
[NP7_0]
Counter Value
----------------- ----------
ipsec_enc_chk 0
ipsec_dec_auth 0
ipsec_dec_chk_ar 0
----------------- ----------
....
<DSW drop counters>
[NP7_0]
SRC_mod -> DST_mod Drop
---------- ---------- ----------
EIF0 -> EIF0 0
EIF0 -> EIF1 0
EIF0 -> EIF2 0
EIF0 -> EIF3 0
EIF0 -> EIF4 0
....
This command can be very verbose, but it contains for each NP7 module and sub-engine stats about anomalies and drops.
The idea is to loop this command over time and check which counter is constantly increasing so that the focus can be moved to a specific part of the NP7 processor, and eventually, a ticket can be raised to Fortinet Support.
Additional commands to check drop statistics for the NP7 chip with these commands:
diagnose npu np7 dce <np7_id> <----- Shows non-zero sub-engine drop counters.
diagnose npu np7 anomaly-drop <np7_id> <----- Shows L3/L4 anomaly check drop counters.
diagnose npu np7 hrx-drop <np7_id> <----- Shows host interface drop counters.
-
Is the NP7 processor under heavy load?
NP7 is connected to the ISF through 2x CGMAC interfaces, each CGMAC is 100Gbps for a total of 200Gbps.
With the command: diagnose npu np7 cgmac-stats <all | np7 id> <verbosity**>, it is possible to check how many packets and throughput are received and transmitted by each NP7 processor.
diagnose npu np7 cgmac-stats 0 0
[NP7_0]
CGMAC0:
--------------- --------------- --------------- --------------- --------------- ---------------
Counters LANE0 LANE1 LANE2 LANE3 Total
--------------- --------------- --------------- --------------- --------------- ---------------
rx_bcast 1568 1595 1581 1562 6306
rx_mcast 107821 107428 107685 108124 431058
rx_ucast 227803 228267 229735 229616 915421
rx_goodoctet 227165253 227183155 228702445 228728383 911779236
rx_octet 227165253 227183155 228702445 228728383 911779236
--------------- --------------- --------------- --------------- --------------- ---------------
tx_bcast 350 333 341 369 1393
tx_mcast 18063 18064 18063 18063 72253
tx_ucast 533957 533972 533965 533937 2135831
tx_goodoctet 374562928 364020455 374612480 364029464 1477225327
tx_octet 374562928 364020455 374612480 364029464 1477225327
--------------- --------------- --------------- --------------- --------------- ---------------
RX_RATE(pps) 0 0 0 0 0
RX_RATE(kbps) 1 0 0 0 2
TX_RATE(pps) 0 0 0 0 0
TX_RATE(kbps) 0 0 0 0 0
--------------- --------------- --------------- --------------- --------------- ---------------
CG_FULL 0
CGMAC1:
--------------- --------------- --------------- --------------- --------------- ---------------
Counters LANE0 LANE1 LANE2 LANE3 Total
--------------- --------------- --------------- --------------- --------------- ---------------
rx_bcast 39572 39496 39743 39810 158621
rx_mcast 240155 241243 240570 241309 963277
rx_ucast 387952 388015 387442 387205 1550614
rx_goodoctet 405397003 406242711 405111720 405148982 1621900416
rx_octet 405397003 406242711 405111720 405148982 1621900416
--------------- --------------- --------------- --------------- --------------- ---------------
tx_bcast 345 359 348 324 1376
tx_mcast 1 0 3 1 5
tx_ucast 417439 417426 417434 417459 1669758
tx_goodoctet 268194630 270121006 268254445 270368412 1076938493
tx_octet 268194630 270121006 268254445 270368412 1076938493
--------------- --------------- --------------- --------------- --------------- ---------------
RX_RATE(pps) 0 0 0 0 0
RX_RATE(kbps) 1 0 0 0 2
TX_RATE(pps) 0 0 0 0 0
TX_RATE(kbps) 0 0 0 0 0
--------------- --------------- --------------- --------------- --------------- ---------------
CG_FULL 0
In addition to each RX_TX rate metric to see the actual CGMAC bandwidth, there is also the GC_FULL counter, this counter increases when there is congestion at the CGMAC level.
With the following command, it is also possible to check the usage in percentage for every NP7 module:
diagnose npu np7 pmon <all | np7 id> <verbosity**>
diagnose npu np7 pmon 0 0
[NP7_0]
EIF0_IGR EIF1_IGR EIF1_EGR EIF1_EGR HRX HTX DFR
-------- -------- -------- -------- -------- -------- -------- --------
Usage% 0 0 0 0 0 0 0
-------- -------- -------- -------- -------- -------- -------- --------
SSE0 SSE1 SSE2 SSE3
-------- -------- -------- -------- --------
Usage% 1 1 1 1
-------- -------- -------- -------- --------
IPSEC IPTI IPTO L2TI L2TO VEP IVS
-------- -------- -------- -------- -------- -------- -------- --------
Usage% 0 0 0 0 0 0 0
-------- -------- -------- -------- -------- -------- -------- --------
PLE MSE SYNK DSE NSS
-------- -------- -------- -------- -------- --------
Usage% 0 0 0 12 0
-------- -------- -------- -------- -------- --------
* EIFx_IGR: EIF ingress, EIFx_EGR: EIF egress
-
Is the NP7 processor stuck?
Each packet reaching the NP7 processor is stored on an allocated buffer, and it is freed when the packet is processed and leaves the FortiGate (buffers used to store packets are limited). It might happen under specific conditions that buffers are not freed when packets are processed, which is called a PBA leak (PBA: packet buffer allocator).
The command: diagnose npu np7 pba 0, compares a normal usage of buffers with the current usage and shows the delta.
diagnose npu np7 pba all
[NP7_0]
normal current Delta Empty
pba 00003f7c 00003f61 27 0
dba 00001ddf 00001ddc 3
hba 00000ff5 00000ff5 0
!!!Leak!!!
Note: '!!! Leak !!!' is printed every time one of the deltas is greater than 1.
Having a positive value displayed is normal when traffic is being processed by the NP7; an administrator should be concerned when the delta value is high and constantly increasing over time, as it means that the buffers are not being released correctly.
Related documents:
Troubleshooting Tip: NPU configuration commands (NP4, NP6, NP7)
Troubleshooting Tip: Understanding the FortiGate NP7 drop types
NP7 acceleration
NP7 Specific Operational Topics
|
