|
Symptoms and behavior of the WAD process:
While there may be more reasons for Wad CPU usage spikes, this article examines spikes due to configuration changes.
All WAD workers can be seen spiking up for less than 5 seconds. This load happens in the user space and potentially impacts traffic. This happens because, when a configuration change takes place, the WAD workers need to be reloaded. If the configuration file possesses a high amount of firewall policies and dynamic objects, re-loading these objects and policies will take a couple of seconds, resulting in a CPU spike. This is expected behavior.
Below is an example of such a spike.
diag sys top
JRun Time: 7 days, 17 hours and 44 minutes
85U, 0N, 13S, 2I, 0WA, 0HI, 0SI, 0ST; 3039T, 1231F
wad 410 R 91.8 1.2 1
wad 413 R 87.7 1.2 2
wad 412 R 79.8 1.0 0
wad 411 R 73.2 1.2 3
ipshelper 192 S < 12.8 1.2 1
ipsengine 568 S < 9.1 2.0 2
Reasons for changes:
Down below are possible reasons config changes occur, and WAD spikes can be triggered.
- Manual changes: The most trivial scenario would be manual configuration changes. To check if this is the case, create a dummy firewall policy and let diag sys top run in a different CLI interface. The moment a policy is created or an existing one is changed, the WAD processes will spike up in diag sys top.
- FortiGuard updates and signature updates: When signatures update, this will also cause the WAD to restart. To rule this out, it is best to schedule these updates during out-of-business hours.
- FortiSandbox: f these spikes happen every couple of minutes, this can point towards FortiSandbox, which can trigger updates very frequently. In these cases, it is best to temporarily disable FortiSandbox in the antivirus profile settings and see if the WAD spikes still happen.
config antivirus profile
edit "default"
set comment "Scan files and block viruses."
set feature-set proxy
set fortisandbox-mode analytics-suspicious <- Disable this setting.
set analytics-ignore-filetype 3
end
Troubleshooting wad spikes.
If it is not clear what process is causing the changes, run the command 'diagnose sys cmdb info' as often as possible, preferably every second. This will make it possible to determine when the last change was done and by what process ID (PID).
Down below is an example of a change done by the process 4513. The command 'fnsysctl ps aux' gives out a list of all running processes and the associated PID on the leftmost column. In this example, it was the httpsd process. It should be noted that Fortimanager API calls are also sent thoguh the httpsd process and these API calls can perform changes in the config.
diagnose sys cmdb info
version: 2
owner id: 1956
update time: 112744
conf file ver: 484373526853736
last request time: Thu Sep 19 03:43:04 2024
last request pid: 4513
last request type: CMDB_REQ_CLI_COMMIT
last request done: 1
fnsysctl ps aux
4513 0 0 S /bin/httpsd
The next step is to gather a WAD debug as this will show what the WAD process is doing at the moment, and config changes will also be shown in the debug. The WAD debug produces a lot of output, so it is advised to have 'diagnose system top' run in another SSH window. Once the spike is observed in 'diag sys top', create a timestamp with 'fnsysctl date' in the SSH window running the WAD debug.
WAD debug:
diagnose debug reset
diagnose debug console timestamp enable
diagnose debug duration 240
diagnose wad filter clear
diagnose wad filter <FILTERS>
diagnose wad debug display pid enable
diagnose wad debug enable category all
diagnose wad debug enable level verbose
diagnose wad debug show
diagnose debug info
fnsysctl date
diagnose debug enable
Below is an output of this kind of configuration change. This output represents the configuration change of a WAD worker with PID 413. The output will be printed for every single WAD worker since every worker will need to reload the config.
[I]2024-08-27 10:50:59.945245 [p:413] wad_conf_chg_notify :6716 process Type:worker id:3, recv notify: global_gen:6318 vd_sz=0
own cmdb = 79758 own global_gen=0
[V]2024-08-27 10:50:59.945304 [p:413] wad_config_flags_dump :991 wad_worker_handle_config_change(worker conf changed): global gen=35892 flags=0x0000000000000000000000000040000000
ss[v] database '/data2/virext' valid
ss[v] database '/tmp/fsa_virdb' valid
ss[v] database '/data2/mmdb' valid
ss[v] database '/data2/avai' valid
ss[v] database '/data2/vir' valid
ss[v] try load atdb etdb fsadb mmdb avai
[V]2024-08-27 10:51:05.502664 [p:413] wad_worker_handle_config_change :1009 WadTest@WorkerConfDone
Related articles:
Technical Tip: Using the 'diagnose wad debug' command to troubleshoot Explicit Web Proxy related iss...
Scheduled updates - FortiGate administration guide
|
