Description
This article provides a workaround and a fix schedule for an issue in which the WAD (wad-config-notify) daemon experiences high memory usage after upgrading to v7.4.4, v7.4.5, and v7.6.0.
Scope
FortiGate v7.4.4, v7.4.5, v7.6.0
Solution
After upgrading to v7.4.4, v7.4.5, v7.6.0, a gradual increase in WAD (wad-config-notify) memory usage is seen on FortiGates leading to memory conserve mode.
get system performance status
CPU states: 3% user 0% system 0% nice 97% idle 0% iowait 0% irq 0% softirq
CPU0 states: 1% user 0% system 0% nice 99% idle 0% iowait 0% irq 0% softirq
CPU1 states: 5% user 1% system 0% nice 94% idle 0% iowait 0% irq 0% softirq
Memory: 4057220k total, 3145100k used (77.5%), 650376k free (16.0%), 261744k freeable (6.5%)
diagnose sys top-mem
wad (275): 470138kB <- WAD process ID '275' consumes high memory.
ipshelper (211): 162549kB
node (188): 90421kB
cw_acd (221): 84123kB
ipsengine (534): 80277kB
Top-5 memory used: 887508kB
diagnose sys top 2 30
JRun Time: 17 days, 17 hours and 46 minutes
13U, 0N, 6S, 81I, 0WA, 0HI, 0SI, 0ST; 3962T, 612F
wad 278 S 0.5 12.3 1
wad 279 S 0.5 12.2 0
wad 275 S 0.0 12.2 0
diagnose wad memory report
******** Memory and CPU usage of wad processes ********
NAME PID STATE %CPU %MEM
wad-worker 278 S 0.0 12.3
wad-worker 279 S 0.0 12.3
wad-config-notify 275 S 0.0 12.2
diagnose debug enable
diagnose test app wad 1000
Process [2]: type=worker(2) index=0 pid=278 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [3]: type=worker(2) index=1 pid=279 state=running
diagnosis=no debug=enable valgrind=supported/disabled
.
Process [12]: type=config-notify(15) index=0 pid=275 state=running >> process ID "275" is of type "config-notify"
diagnosis=no debug=enable valgrind=supported/disabled
The issue has been resolved in FortiOS versions v7.4.6 (scheduled for release in December 2024) and v7.6.1 (scheduled for release between November 2024). These timelines for firmware release are estimates and may be subject to change.
The release date of the FortiOS firmware version can be verified through Fortinet Support,
- Select Downloads -> Firmware Download from the drop-down menu after clicking the Support icon at the top.
- In the Select Product menu, select FortiGate, then the Download tab. Drill down through the directories until finding the desired firmware version.
The 'Date Created' column will display the public release date for that version of firmware.
Workaround:
Configure an auto-script to restart WAD. An example script is below, this will restart WAD every 24 hours.
config system auto-script
edit restart_wad
set interval 86400
set repeat 0
set start auto
set script 'diagnose test application wad 99'
next
end
Over the GUI is possible to configure an automation stitch to perform the restart of the 'wad' process daily, at an specific time:
- Go to Security Fabric -> Automation:
- Go to the tab 'Action' and create a new action:
- From the different options, choose 'CLI Script':
- In the script section input the previously mentioned command 'diagnose test application wad 99'. It is important to select 'super_admin' as the administrator profile.
- Save and go back to the automation page, and go to the tab 'Trigger' to create a new one.
- Select the option 'Schedule'
- Specify the frequency and the time the command will be pushed. Is important to mention that the time format is 24 hours.
- Save and go back to the automation page, and go to the tab 'Stitch' to create a new one.
- Select the Trigger and Action, that were just created a moment ago:
Related article:
Technical Tip: How to restart WAD or IPS engine using automated script
