Technical Tip: How to Verify PCIe Latency in NP7

The NP7 PCIe attachment allows the NP7 chip to communicate and exchange traffic with the FortiGate kernel (CPU) using the NP7 Host Interface (HIF). A PCIe Latency test can be run on an NP7 unit to observe latency on the PCIe channel.

Important notes:

• The available pcie_latency commands scale with the number of NP7 chips built into the FortiGate. Consequently, not all commands will return an output.
• The commands are production-impacting and should only be run during a maintenance window.
• One pcie_latency command will cause a 100% CPU spike (%sys) on a random CPU core for several seconds impacting sessions handled by that core. Running all 6 pcie_latency commands will cause 6 random CPU cores to increase to 100% CPU utilization.
• Each iteration of the pcie_latency command will cause a different CPU core to spike. It is not always the same CPU core that spikes.
• Some variation in latency is expected. Latency consistently above 2000ns (2ms) may indicate an issue.

Collecting PCIe Latency Data:

1. Confirm how many NP7 chips the device has by navigating to the FortiOS hardware acceleration guide and verifying the hardware model. FortiGate NP7 architectures.
   
   
2. Execute the commands below from the CLI several times to estimate the average latency per PCIe interface.

fnsysctl cat /proc/net/np7/np7_0/pcie_latency
fnsysctl cat /proc/net/np7/np7_1/pcie_latency
fnsysctl cat /proc/net/np7/np7_2/pcie_latency
fnsysctl cat /proc/net/np7/np7_3/pcie_latency
fnsysctl cat /proc/net/np7/np7_4/pcie_latency
fnsysctl cat /proc/net/np7/np7_5/pcie_latency

3. If latency is greater than 2000ns is observed consistently, it may indicate an issue.

FortiGate-901G # fnsysctl cat /proc/net/np7/np7_0/pcie_latency
prescale: 0
unit: 100 ns
measure[i]: max_reqid max_latency latency(ns)
measure[0]: 1 6 600
measure[1]: 80 7 700
measure[2]: 80 7 700
measure[3]: 2 c 1200
measure[4]: 2 c 1200
measure[5]: 2 c 1200
measure[6]: 2 c 1200
measure[7]: 2 c 1200
measure[8]: 2 c 1200
measure[9]: 2 c 1200