Description | This article describes how to troubleshoot intermittent short CPU spikes due to configuration changes in the WAD process. |
Scope | All supported versions of FortiGate. |
Solution |
Symptoms and behavior of the WAD process:
While there may be more reasons for Wad CPU usage spikes, this article examines spikes due to configuration changes.
All WAD workers can be seen spiking up for less than 5 seconds. This load happens in the user space and potentially impacts traffic. This happens because, when a configuration change takes place, the WAD workers need to be reloaded. If the configuration file possesses a high amount of firewall policies and dynamic objects, re-loading these objects and policies will take a couple of seconds, resulting in a CPU spike. This is expected behavior.
Below is an example of such a spike.
diag sys top JRun Time: 7 days, 17 hours and 44 minutes
Reasons for changes:
Down below are possible reasons config changes occur, and WAD spikes can be triggered.
config antivirus profile edit "default" set comment "Scan files and block viruses." end
Troubleshooting wad spikes.
If it is not clear what process is causing the changes, run the command 'diagnose sys cmdb info' as often as possible, preferably every second. This will make it possible to determine when the last change was done and by what process ID (PID). Down below is an example of a change done by the process 4513. The command 'fnsysctl ps aux' gives out a list of all running processes and the associated PID on the leftmost column. In this example, it was the httpsd process. It should be noted that Fortimanager API calls are also sent thoguh the httpsd process and these API calls can perform changes in the config.
diagnose sys cmdb info version: 2 owner id: 1956 update time: 112744 conf file ver: 484373526853736 last request time: Thu Sep 19 03:43:04 2024 last request pid: 4513 last request type: CMDB_REQ_CLI_COMMIT last request done: 1
fnsysctl ps aux 4513 0 0 S /bin/httpsd
The next step is to gather a WAD debug as this will show what the WAD process is doing at the moment, and config changes will also be shown in the debug. The WAD debug produces a lot of output, so it is advised to have 'diagnose system top' run in another SSH window. Once the spike is observed in 'diag sys top', create a timestamp with 'fnsysctl date' in the SSH window running the WAD debug.
WAD debug:
diagnose debug reset
Below is an output of this kind of configuration change. This output represents the configuration change of a WAD worker with PID 413. The output will be printed for every single WAD worker since every worker will need to reload the config.
[I]2024-08-27 10:50:59.945245 [p:413] wad_conf_chg_notify :6716 process Type:worker id:3, recv notify: global_gen:6318 vd_sz=0
Related articles: |
The Fortinet Security Fabric brings together the concepts of convergence and consolidation to provide comprehensive cybersecurity protection for all users, devices, and applications and across all network edges.
Copyright 2024 Fortinet, Inc. All Rights Reserved.