Technical Tip: WAD (wad-config-notify) process consumes high Memory after upgrade to v7.4.4, v7.4.5, v7.6.0
Description
This article provides a workaround and a fix schedule for an issue in which the WAD (wad-config-notify) daemon experiences high memory usage after upgrading to v7.4.4, v7.4.5, and v7.6.0.
Scope
FortiGate v7.4.4, v7.4.5, v7.6.0.
Solution
After upgrading to v7.4.4, v7.4.5, v7.6.0, a gradual increase in WAD (wad-config-notify) memory usage is seen on FortiGates leading to memory conserve mode.
get system performance status
CPU states: 3% user 0% system 0% nice 97% idle 0% iowait 0% irq 0% softirq
CPU0 states: 1% user 0% system 0% nice 99% idle 0% iowait 0% irq 0% softirq
CPU1 states: 5% user 1% system 0% nice 94% idle 0% iowait 0% irq 0% softirq
Memory: 4057220k total, 3145100k used (77.5%), 650376k free (16.0%), 261744k freeable (6.5%)
diagnose sys top-mem
wad (275): 470138kB <- WAD process ID '275' consumes high memory.
ipshelper (211): 162549kB
node (188): 90421kB
cw_acd (221): 84123kB
ipsengine (534): 80277kB
Top-5 memory used: 887508kB
diagnose sys top 2 30
JRun Time: 17 days, 17 hours and 46 minutes
13U, 0N, 6S, 81I, 0WA, 0HI, 0SI, 0ST; 3962T, 612F
wad 278 S 0.5 12.3 1
wad 279 S 0.5 12.2 0
wad 275 S 0.0 12.2 0
diagnose wad memory report
******** Memory and CPU usage of wad processes ********
NAME PID STATE %CPU %MEM
wad-worker 278 S 0.0 12.3
wad-worker 279 S 0.0 12.3
wad-config-notify 275 S 0.0 12.2
diagnose debug enable
diagnose test app wad 1000
Process [2]: type=worker(2) index=0 pid=278 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [3]: type=worker(2) index=1 pid=279 state=running
diagnosis=no debug=enable valgrind=supported/disabled
.
Process [12]: type=config-notify(15) index=0 pid=275 state=running >> process ID "275" is of type "config-notify"
diagnosis=no debug=enable valgrind=supported/disabled
If the process type needs to be changed to a 'config-notify' WAD Process Context for Debug and Diagnostics.
diagnose test app wad 1000
Process [12]: type=config-notify(15) index=0 pid=275 state=running
diagnosis=no debug=enable valgrind=supported/disabled
diagnose test app wad 21500
Set diagnosis process: type=config-notify index=0 pid=275
diagnose test app wad 1
WAD process config status: pid=275 use_adv_mem=enabled
The issue has been resolved in FortiOS versions v7.4.6 and v7.6.1 (available for downloading in the support page). Search for BUG ID 1078385 in the Resolved Issues section of the Release Notes below:
The release date of the FortiOS firmware version can be verified through Fortinet Support,
- Select Downloads -> Firmware Download from the drop-down menu after clicking the Support icon at the top.
- In the Select Product menu, select FortiGate, then the Download tab. Drill down through the directories until the desired firmware version is reached.
The 'Date Created' column will display the public release date for that version of the firmware.
Workaround:
Configure an auto-script to restart WAD. An example script is below, this will restart WAD every 24 hours.
config system auto-script
edit restart_wad
set interval 86400
set repeat 0
set start auto
set script 'diagnose test application wad 99'
next
end
Note: Running the above command gracefully restarts all WAD worker processes. To verify that this restart has taken place, check and compare the Process ID (PID) of the WAD workers before and after the restart using the command diagnose sys process pidof wad (the PIDs will change when the process is restarted).
Alternatively, Automation Stitches may be configured to restart the WAD processes on a scheduled basis (e.g. daily at a particular time):
- Go to Security Fabric -> Automation:
- Go to the Action tab and select Create New:
- Select the CLI Script action:
- In the Script section, input the previously mentioned command diagnose test application wad 99. It is important to select 'super_admin' as the administrator profile.
- Important Note: when the FortiGate is configured with VDOMs it is crucial that the command config global be added before the above command. This ensures that the FortiGate enters the Global VDOM first before executing the WAD restart, otherwise the command will not execute successfully.
- Select OK to save the Automation Action, then go to the Trigger tab and select Create New to create a new Automation Trigger.
- Select the Schedule option:
- Specify the Frequency and the time (Hour/Minute) when the CLI Script action should be executed. Note that the time format in this configuration is based on the 24 hour clock (e.g. 10:00PM == 22:00).
- Select OK to save the Automation Trigger, then go to the Stitch tab to create a new Automation Stitch.
- Select the Trigger and Action that were created in earlier steps, then select OK to commit the result:
Related articles:
Technical Tip: How to restart WAD or IPS engine using automated script
Technical Tip: FortiGate monitoring script