Fortigate VM esxi high CPU usage

[fox]

Hi, when I enable DPDK, the CPU always 100% usage, even I enable sleep-on-idle, still one core was 100%.

This was normal?　Are there any setting can reduce the CPU usage?

config:
config dpdk global
set status enable
set interface "port1" "port2" "port3"
set multiqueue enable
set sleep-on-idle enable
set elasticbuffer enable
set per-session-accounting traffic-log-only
set ipsec-offload disable
set hugepage-percentage 30
set mbufpool-percentage 25
end

config dpdk cpus
set rx-cpus "all"
set vnp-cpus "all"
set ips-cpus "all"
set tx-cpus "all"
end

log:
diagnose dpdk performance show
----------------------------------------
CPU usages
----------------------------------------
Average Engine 0 Engine 1
2023:04:10 16:58:29 rx: 0.0 0.1 0.0
2023:04:10 16:58:29 vnp: 0.3 0.0 0.6
2023:04:10 16:58:29 ips: 0.0 0.0 0.0
2023:04:10 16:58:29 tx: 0.0 0.0 0.0
2023:04:10 16:58:29 idle: 99.6 99.8 99.4

diagnose sys mpstat 2 3
Gathering data, wait 2 sec, press any key to quit.
..0..1
TIME CPU %usr %nice %sys %iowait %irq %soft %steal %idle
04:57:50 PM all 36.66 0.00 13.97 0.00 0.00 0.25 0.00 49.13
0 73.13 0.00 26.87 0.00 0.00 0.00 0.00 0.00
1 0.00 0.00 1.00 0.00 0.00 0.50 0.00 98.51

TIME CPU %usr %nice %sys %iowait %irq %soft %steal %idle
04:57:52 PM all 38.44 0.00 12.06 0.00 0.00 0.00 0.00 49.50
0 76.00 0.00 24.00 0.00 0.00 0.00 0.00 0.00
1 0.51 0.00 0.00 0.00 0.00 0.00 0.00 99.49

TIME CPU %usr %nice %sys %iowait %irq %soft %steal %idle
04:57:54 PM all 37.22 0.00 13.40 0.00 0.25 0.00 0.00 49.13
0 73.00 0.00 27.00 0.00 0.00 0.00 0.00 0.00
1 1.98 0.00 0.00 0.00 0.00 0.00 0.00 98.02

[rosatechnocrat]

Enabling DPDK in polling mode results in high CPU usage.

[kvimaladevi]

Hi Fox,

DPDK is enabled on the Firewall which does the load balancing. To understand when process is utilizing high CPU, please provide the below outputs:

diag dpdk performance show
diag sys top-summary
diag sys top 1 20

Also, please check if you are seeing any crash logs in the diag debug crashlog read output.

Regards,

Vimala

[fox]

Hi Vimala, thanks for repling, please check below log;

diag dpdk performance show
----------------------------------------
CPU usages
----------------------------------------
Average Engine 0 Engine 1
2023:04:11 20:52:56 rx: 0.0 0.1 0.0
2023:04:11 20:52:56 vnp: 0.3 0.1 0.4
2023:04:11 20:52:56 ips: 0.0 0.0 0.0
2023:04:11 20:52:56 tx: 0.0 0.0 0.0
2023:04:11 20:52:56 idle: 99.6 99.7 99.6
-----------------------------------------

diag sys top 1 20

Run Time: 29 days, 21 hours and 34 minutes
40U, 0N, 10S, 50I, 0WA, 0HI, 0SI, 0ST; 3921T, 544F
ipsengine 3845 R < 99.9 7.5 0
ipsengine 3846 S < 0.0 7.5 1
node 3619 S 0.0 3.1 1
ipshelper 3678 S < 0.0 2.7 0
cmdbsvr 3577 S 0.0 1.8 0
scanunitd 128744 S < 0.0 1.5 1
scanunitd 130583 S < 0.0 1.5 1
wad 3709 S 0.0 1.4 1
wad 3708 S 0.0 1.4 1
miglogd 3637 S 0.0 1.3 1
reportd 3639 S 0.0 1.2 1
cid 3666 S 0.0 1.2 1
miglogd 3783 S 0.0 1.1 1
cw_acd 3687 S 0.0 1.1 1
forticron 3628 S 0.0 1.0 1
scanunitd 3642 S < 0.0 1.0 1
csfd 3690 S 0.0 1.0 1
wad 3698 S 0.0 0.9 1
forticldd 3629 S 0.0 0.9 1
fgfmd 3686 S 0.0 0.9 1

[fox]

thanks for the repling, please see the log:

diag dpdk performance show
----------------------------------------
CPU usages
----------------------------------------
Average Engine 0 Engine 1
2023:04:11 22:20:03 rx: 0.9 1.6 0.1
2023:04:11 22:20:03 vnp: 2.4 0.1 4.7
2023:04:11 22:20:03 ips: 0.0 0.0 0.0
2023:04:11 22:20:03 tx: 0.7 1.0 0.4
2023:04:11 22:20:03 idle: 96.0 97.2 94.8
-----------------------------------------

diag sys top 1 20

Run Time: 29 days, 23 hours and 1 minutes
30U, 0N, 21S, 47I, 0WA, 0HI, 2SI, 0ST; 3921T, 541F
ipsengine 3845 R < 99.0 7.5 0
ipsengine 3846 R < 3.0 7.5 1
node 3619 S 1.0 3.0 1
ipshelper 3678 S < 1.0 2.7 0
cmdbsvr 3577 S 0.0 1.8 0
scanunitd 128744 S < 0.0 1.5 1
scanunitd 130583 S < 0.0 1.5 1
wad 3709 S 0.0 1.4 1
wad 3708 S 0.0 1.4 1
miglogd 3637 S 0.0 1.3 1
reportd 3639 S 0.0 1.2 1
cid 3666 S 0.0 1.2 1
miglogd 3783 S 0.0 1.1 1
cw_acd 3687 S 0.0 1.1 1
forticron 3628 S 0.0 1.0 1
scanunitd 3642 S < 0.0 1.0 1
csfd 3690 S 0.0 1.0 1
wad 3698 S 0.0 0.9 1
forticldd 3629 S 0.0 0.9 1
fgfmd 3686 S 0.0 0.9 1

[rosatechnocrat]

Seems issue is caused by ipsengine. 

ipsengine 3845 R < 99.0 7.5 0

what is the software version. Provide output of get system status

[fox]

V7.2.4

get system status
Version: FortiGate-VM64 v7.2.4,build1396,230131 (GA.F)
Virus-DB: 91.01261(2023-03-09 03:26)
Extended DB: 91.01261(2023-03-09 03:25)
Extreme DB: 1.00000(2018-04-09 18:07)
AV AI/ML Model: 0.00000(2001-01-01 00:00)
IPS-DB: 23.00508(2023-03-08 01:36)
IPS-ETDB: 23.00508(2023-03-08 01:36)
APP-DB: 6.00741(2015-12-01 02:30)
INDUSTRIAL-DB: 6.00741(2015-12-01 02:30)
IPS Malicious URL Database: 1.00001(2015-01-01 01:01)
IoT-Detect: 0.00000(2022-08-17 17:31)
Serial-Number:
License Status:
VM Resources: 2 CPU/2 allowed, 3921 MB RAM
Log hard disk: Available
Hostname:
Private Encryption: Disable
Operation Mode: NAT
Current virtual domain: root
Max number of virtual domains: 2
Virtual domains status: 1 in NAT mode, 0 in TP mode
Virtual domain configuration: disable
FIPS-CC mode: disable
Current HA mode: standalone
Branch point: 1396
Release Version Information: GA
FortiOS x86-64: Yes
System time: Wed Apr 12 16:56:30 2023
Last reboot reason: warm reboot