Fortigate process " wad" consuming 62% of memory.

We had the same problem with Version 5.6.6 on our FortiGates 600D.  The good news are, since a vew days, release 5.6.7 is available and according to the release notes, "High memory usage on WAD" should be fixed.

Hope this helps,

Wolfgang

Does anyone know what triggers this wad memory usage escalation, which was fixed with 5.6.7?

When I checked it before killing them (fnsysctl killall wad) tonight, I found it came back down to normal level at about 50% two hours ago. Until then it had kept growing steadily for last two weeks based on our mem usage monitoring tool. I asked some coworkers if they did something to it but nobody seems to have done anything. It's very strange.

Hello, We upgraded our FGT-1500D A-P cluster 7 days ago to FortiOS 5.6.7, at this moment the memory is OK, but still the usage is increasing a little bit. We will see.

I would recommend you to not kill the wad process but it is better to restart it. Always restart processes in case they have command for that.

Check the overall CPU and memory status: # diagnose sys top-summary

CPU [|||||||||||| ] 30.7% Mem [|||||||||||||||||||||||||||| ] 71.0% 11460M/16064M Processes: 20 (running=4 sleeping=239) PID RSS CPU% ^MEM% FDS TIME+ NAME * 262 6G 60.6 38.3 41675 48:06.38 wad [x14] 247 1G 54.0 8.2 985 17:53.21 ipsmonitor [x12] 287 1G 5.0 7.1 930 21:00.71 cw_acd 256 645M 34.9 4.0 263 59:32.15 sslvpnd [x12] 272 349M 2.5 2.2 20 51:14.84 urlfilter 241 213M 4.8 1.3 61 19:52.33 miglogd [x7] 244 147M 1.7 0.9 24 00:44.52 httpsd [x5] .....

Use the "m" key on your keyboard to sort the process groups according to memory usage. Use "q" key on your keyboard to quit the diagnose command.

Show the running processes: # diagnose sys top Use the "m" key on your keyboard to sort the processes according to memory usage. Use "q" key on your keyboard to quit the diagnose command.

Locate your wad process and his process ID, let's say for now: wad 351 S 2.7 9.0 The 351 is the process ID.

Now reset and enable debuging: # diagnose debug reset # diagnose debug enable

List all your wad processes and ocate your process ID (pid): # diagnose test application wad 1000 Process [0]: WAD manager type=manager(0) pid=262 diagnosis=yes. Process [1]: type=dispatcher(1) index=0 pid=345 state=running diagnosis=no debug=enable valgrind=unsupported/disabled Process [2]: type=wanopt(2) index=0 pid=346 state=running diagnosis=no debug=enable valgrind=supported/disabled Process [3]: type=worker(3) index=0 pid=347 state=running diagnosis=no debug=enable valgrind=supported/disabled Process [4]: type=worker(3) index=1 pid=348 state=running diagnosis=no debug=enable valgrind=supported/disabled Process [5]: type=worker(3) index=2 pid=349 state=running diagnosis=no debug=enable valgrind=supported/disabled Process [6]: type=worker(3) index=3 pid=350 state=running diagnosis=no debug=enable valgrind=supported/disabled Process [7]: type=worker([style="background-color: #ffff00;"]3[/style]) index=[style="background-color: #ffff00;"]4[/style] pid=351 state=running <============================= diagnosis=no debug=enable valgrind=supported/disabled Process [8]: type=worker(3) index=5 pid=352 state=running diagnosis=no debug=enable valgrind=supported/disabled Process [9]: type=worker(3) index=6 pid=353 state=running diagnosis=no debug=enable valgrind=supported/disabled Process [10]: type=worker(3) index=7 pid=354 state=running diagnosis=no debug=enable valgrind=supported/disabled Process [11]: type=worker(3) index=8 pid=355 state=running diagnosis=no debug=enable valgrind=supported/disabled Process [12]: type=worker(3) index=9 pid=356 state=running diagnosis=no debug=enable valgrind=supported/disabled Process [13]: type=informer(4) index=0 pid=344 state=running diagnosis=no debug=enable valgrind=unsupported/disabled

Now access your wad process - enter into the process menu: Remember: 2xxx - the wad process always starts with 2. x3xx - the wad process type depending on whether it is a dispatcher, worker, informer etc. in brackets (). xx04 - the index number of the process (two digits). If the index is one digit, put 0 before the index.

So it looks like this in our case for wad process with pid=351 # diagnose test application wad 2304 Set diagnosis process: type=worker index=4 pid=351

Now you are working on your wad process and you can list the available options with issuing: # diagnose test application wad

WAD process 351 test usage: 1: display process status 2: display total memory usage. 99: restart all WAD processes 1000: List all WAD processes. 1001: dispaly debug level name and values 1002: dispaly status of WANOpt storages ..... etc. There is a lot of them.

What you need is to restart the process so use 99 like: # diagnose test application wad 99

Reset and disable your debug: # diagnose debug disable # diagnose debug reset

Now you can check the CPU and memory again with command: # diagnose sys top-summary

Compare the results with the output at the beginning of this post.

More work but better than just killing processes.

plindgren wrote:
i get the " CFG_CMDBAPI_ERR" when i try to make changes on my fortigate. which is other than that operational. I tired the command " diag test application ipsmonitor 99" but it did not work. So i used the command " diag sys top 1" to see what was hogging all that memory. And i found a process named " wad" that uses 62% of the memory. However this machine is in production and i dont know what the process does and i cant seem to find it anywhere. Can i kill it? What does it do? Is there a process reference for fortios out there somewhere?

pls view the current wad worker count and modify it 

    config global     config system global        set wad-worker-count (1,2,3,4...)     end