Technical Tip: PBA (Packet Buffer Allocator) leak

[pnayak]

Description
This article discusses about PBA (Packet Buffer Allocator) leak and explains when to consider as an actual leak.
Scope
For FortiGate 420x, 440x, 180x, 260x series.

Solution
In case of actual pba leak.

->Some of SSE/L2P threads may be held at 100%, an indication of the stuck.
•    Verify performance monitor: diag npu np7 pmon x.
•    Any module from pmon may show 100% busy.
->PBA leak is only real when numbers do not move after stopping the traffic and pba delta is is not 0.
•    Verify buffer allocation: diag npu np7 pba x
->SSE will be stuck with YES indicator.
•    Verify sse status: diag npu np7 pdq x | grep sse
•    Any pdq marked with “stuck” as 'yes' while the “rpcnt” is not moving with multiple polling.
->Contact Fortinet Support team for further investigation to find out the root cause
•    There could be n number of reasons for pba leak and it needs to be debugged separately.

Impact of pba Leak.

-> All or most of the packets will be dropped based on the number of chips are stuck depending on the product line/number of chip in the product.
-> Reboot is the solution for device to come to working state.

Not a real leak.

-> Pba counters are also still moving with traffic on.
-> Pba counters will be 0 after traffic is stopped.
-> Just ignore the Leak word when buffer numbers are small and moving.
-> PDQ is not stuck and comes to NO without any traffic or counter moves with traffic on.

Ex. Actual issue and need an action.

Following output can show that buffers are not moving and sse is stuck.

FGT-4201F-177 (global) # diag npu np7 pba 0
     normal   current  Delta    Empty
pba 00003f7c 000003ab 15313 0
dba 00001ddf 00001716 1737
hba 00000ff5 00000ff5 0

!!!Leak!!!

FGT-4201F-177 (global) # diag npu np7 pdq 0 | grep sse
sse[0].pdq 2785181839 2785181813 1762918486 1762918388 26 98 Yes
sse[1].pdq 885898977 885898951 2757591491 2757591393 26 98 Yes
sse[2].pdq 2679437802 2679437776 1348681254 1348681156 26 98 Yes
sse[3].pdq 2039845971 2039845945 3097952694 3097952599 26 95 Yes

ex. Not real and it is an example of normal scenario.

FGT-4201F-177 (global) # diag npu np7 pba 0
     normal   current  Delta    Empty
pba  00003f7c 00003f6d 15       0
dba  00001ddf 00001dda 5
hba  00000ff5 00000ff5 0

!!!Leak!!!

FGT-4201F-177 (global) # diag npu np7 pba 0
     normal   current  Delta    Empty
pba  00003f7c 00003f69 19       0
dba  00001ddf 00001ddd 2
hba  00000ff5 00000ff5 0

!!!Leak!!!

FGT-4201F-177 (global) # diag npu np7 pba 0
     normal   current  Delta    Empty
pba  00003f7c 00003f7c 0        0
dba  00001ddf 00001ddf 0
hba  00000ff5 00000ff5 0

FGT-4201F-177 (global) # diag npu np7 pdq 0 | grep sse
sse[0].pdq              1922946240 1922946240 3270996019 3270996019 0          0          No
sse[1].pdq              3430156992 3430156992 2143102263 2143102263 0          0          No
sse[2].pdq              2009731786 2009731786 3724743895 3724743895 0          0          No
sse[3].pdq              2007565015 2007565015 3675053084 3675053074 0          10         Yes

FGT-4201F-177 (global) # diag npu np7 pdq 0 | grep sse
sse[0].pdq              1925547985 1925547985 3282590608 3282590603 0          5          Yes
sse[1].pdq              3432646923 3432646923 2154134468 2154134468 0          0          No
sse[2].pdq              2012262431 2012262430 3735976705 3735976705 1          0          Yes
sse[3].pdq              2009990678 2009990678 3685754610 3685754610 0          0          No