|
After enabling DPDK high CPU usage (up to 100%) can be observed. By default all CPU cores will be loaded by ipsengine. This is an expected behavior.
The reason why all IPS engines are 99% is that DPDK disables interrupts and keeps busy polling all the time.
It is by design default behavior when DPDK is enabled
config dpdk global
set status enable
end
get system performance status
CPU states: 67% user 32% system 0% nice 1% idle 0% iowait 0% irq 0% softirq
CPU0 states: 64% user 35% system 0% nice 1% idle 0% iowait 0% irq 0% softirq
CPU1 states: 71% user 28% system 0% nice 1% idle 0% iowait 0% irq 0% softirq
diagnose sys top 5 99
ipsengine 2839 R < 99.9 6.0 1
ipsengine 2838 R < 99.5 6.1 0
When DPDK is enabled, it is better to use the 'diagnose dpdk performance show' command to check DPDK engine usage. Outputs from 'get sys performance status' and 'diagnose sys top' can cause a wrong interpretation of CPU usage.
For example:
Systop shows the High CPU usage on IPSengine when DPDK is enabled.
FGTVM01 (global) # diagnose sys top 3 10 5
Run Time: 94 days, 8 hours and 49 minutes
26U, 0N, 16S, 58I, 0WA, 0HI, 0SI, 0ST; 16041T, 4252F
ipsengine 22618 R < 99.9 2.1 5 < ---
ipsengine 22619 R < 99.5 2.1 6 < ---
ipsengine 22620 R < 99.5 2.1 7 < ---
ipshelper 22372 S < 27.5 0.6 0
fnbamd 1873 S 8.0 0.2 4
node 3890 S 0.5 0.6 3
Using the command 'diagnose dpdk performance show':
FGTVM01 (global) # diagnose dpdk performance show
----------------------------------------
CPU usages
----------------------------------------
Average Engine 0 Engine 1 Engine 2
2024:05:10 10:37:35 rx: 2.2 3.8 2.6 0.1
2024:05:10 10:37:35 vnp: 1.8 1.4 1.9 2.1
2024:05:10 10:37:35 ips: 0.2 0.2 0.2 0.2
2024:05:10 10:37:35 tx: 1.1 1.6 1.6 0.0
2024:05:10 10:37:35 idle: 94.8 93.0 93.7 97.6
In case 'sleep-on-idle' is enabled only one CPU core will be overloaded by ipsengine. By default 'sleep-on-idle' is disabled. After enabling 'sleep-on-idle' latency may increase.
config dpdk global
set sleep-on-idle enable
end
get system performance status
CPU states: 34% user 15% system 0% nice 51% idle 0% iowait 0% irq 0% softirq
CPU0 states: 68% user 30% system 0% nice 2% idle 0% iowait 0% irq 0% softirq
CPU1 states: 1% user 0% system 0% nice 99% idle 0% iowait 0% irq 0% softirq
diagnose sys top 5 10
ipsengine 2830 R < 99.9 6.1 0
ipsengine 2831 S < 0.0 6.0 1
If the high CPU load on all cores or on CPU0 caused by the IPS engines is resulting in issues like traffic drops, it is recommended to disable the dpdk feature:
config dpdk global
set status disable
end
Commands to verify the dpdk settings and load:
get sys stat
get sys perf stat
diag sys top 1 20 3
diag sys mpstat 1 3
show dpdk global
show full-configuration dpdk global
show dpdk cpus
show full-configuration dpdk cpus
diagnose dpdk config show
diagnose dpdk performance show
diagnose dpdk statistics show
diagnose dpdk log show
diag debug report
|
