Description
This article describes how to troubleshoot the 'dmworker' process.
FortiManager 7.0 and upward.
Solution
The steps to understand the issue are:
get system status
The command above provides essential information about the system as the current version, the file system check, and the license status.
diagnose cdb upgrade summary
It gives the history of upgrades/downgrades which is very useful if the downgrade was made improperly.
Then it is interesting to review the system load and to know which service is using most of the system resources. For that, use the following command:
exe top -b -n 1
top - 10:23:11 up 1:00, 0 users, load average: 2.02, 1.79, 1.39
Tasks: 191 total, 2 running, 189 sleeping, 0 stopped, 0 zombie
%Cpu(s): 26.2 us, 1.5 sy, 0.0 ni, 72.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
MiB Mem : 7975.5 total, 60.2 free, 3687.0 used, 4228.2 buff/cache
MiB Swap: 2048.0 total, 2048.0 free, 0.0 used. 3395.7 avail Mem
PID USER PR NI VIRT RES %CPU %MEM TIME+ S COMMAND
2597 root 20 0 1668.4m 1.5g 100.0 19.0 56:38.92 R dmworker:proc/re+
7160 root 20 0 3.7m 2.3m 6.2 0.0 0:00.01 R /bin/top -b -n 1
The dmworker is observed keeping the CPU at 100% for a while without a change. To understand more about it the following debug commands are used:
diagnose debug application dmworker 255
diagnose debug enable
Then to stop the debugging:
diagnose debug disable
diagnose debug reset
Take the JSON part of the debug and prettify it.
{
"client": "rtmmond:1613",
"id": 734,
"method": "exec",
"params": [
{
"data": {
"dir": "/var/upload/virtual-wan/sla-log"
},
"target start": 1,
"url": "proc/rest/data"
}
],
"root": "dmworker"
}
{
"client": "rtmmond:1613",
"id": 735,
"method": "exec",
"params": [
{
"data": {
"dir": "/var/upload/status/license"
},
"target start": 1,
"url": "proc/rest/data"
}
],
"root": "dmworker"
}
{
"client": "rtmmond:1613",
"id": 737,
"method": "exec",
"params": [
{
"data": {
"dir": "/var/upload/virtual-wan/interface-log"
},
"target start": 1,
"url": "proc/rest/data"
}
],
"root": "dmworker"
}
Reviewing the repeating requests gives a few points. The license was mitigated after requesting the Entitlement file. Stop the 'sdwan monitor' under FortiManager and reboot the unit, this will help to mitigate the other requests.
config system admin setting
set sdwan-monitor-history disable
end
After the restart, the process was under observation for a while. The CPU was no longer kept at 100%.
Note:
By default, SD-WAN Monitoring History is turned off. When this feature is disabled, only real-time data from the last 10 minutes is displayed, and you can refresh the view to retrieve updated data directly from FortiGate devices. However, no historical data is stored in FortiManager while this feature remains disabled.
if SD-WAN monitoring history feature is enabled, managing devices through the central management unit, it's crucial to consider its tunnel limitations. To maintain system performance and ensure stable connections for all managed devices, we strongly recommend disabling data-intensive features such as SD-WAN historical monitoring. Applying an add-on license to the central management unit can increase its device support capacity beyond the default tunnel limit. However, even with this enhancement, seamless management of all live tunnels may still present challenges.
Troubleshooting commands:
diag dvm task repair
diagnose sys fsck harddisk
diagnose debug application dmworker 255
diagnose debug enable
To stop the debugging, type:
diagnose debug disable
diagnose debug reset
Related articles:
