Technical Tip: Gathering WAD debugs using the 'diagnose test application' debug command

[jsevigny]

Description	This article discusses gathering WAD debugs using the 'diagnose test application' debug command to help investigate resource issues.
Scope	FortiGate.
Solution	Determine the PID of the WAD process using the most memory, to do so run one of the following commands or both: diag sys top (Hit m once command is running) diagnose sys top-mem n Time:  4 days, 23 hours and 59 minutes 2U, 0N, 2S, 95I, 0WA, 0HI, 1SI, 0ST; 7979T, 3104F              wad      318      S       0.0     5.5    3        ipshelper      311      S <     0.0     3.2    2        ipsengine      430      S <     2.3     2.1    4        ipsengine      426      S <     1.4     1.9    0        ipsengine      428      S <     2.3     1.9    2        ipsengine      429      S <     1.8     1.9    3        ipsengine      427      S <     1.8     1.9    1        ipsengine      431      S <     1.4     1.9    5        ipsengine      432      S <     1.6     1.8    6             node      245      S       0.2     1.2    0           cw_acd      292      S       2.1     1.0    2          cmdbsvr      209      S       0.0     0.8    5          src-vis      276      S       5.2     0.8    4          appDemo      175      S <     0.2     0.8    0          miglogd      264      S       0.0     0.5    4        scanunitd    30646      S <     0.0     0.4    5           cu_acd      296      S       1.6     0.4    4              wad      325      S       0.0     0.4    3              wad      327      S       0.0     0.4    0              wad      321      S       0.0     0.4    2 diagnose sys top-mem wad (318): 434687kB ipshelper (311): 248560kB node (245): 109138kB ipsengine (430): 83070kB ipsengine (428): 70281kB Top-5 memory used: 945736kB  Next, run the following: diag debug enable diag test app wad 1000 diagnose test application wad 1000 Process [0]: WAD manager type=manager(0) pid=268 diagnosis=yes. Process [1]: type=dispatcher(1) index=0 pid=320 state=running               diagnosis=no debug=enable valgrind=unsupported/disabled Process [2]: type=worker(2) index=0 pid=321 state=running               diagnosis=no debug=enable valgrind=supported/disabled Process [3]: type=worker(2) index=1 pid=322 state=running               diagnosis=no debug=enable valgrind=supported/disabled Process [4]: type=worker(2) index=2 pid=323 state=running               diagnosis=no debug=enable valgrind=supported/disabled Process [5]: type=worker(2) index=3 pid=324 state=running               diagnosis=no debug=enable valgrind=supported/disabled Process [6]: type=worker(2) index=4 pid=325 state=running               diagnosis=no debug=enable valgrind=supported/disabled Process [7]: type=worker(2) index=5 pid=326 state=running               diagnosis=no debug=enable valgrind=supported/disabled Process [8]: type=worker(2) index=6 pid=327 state=running               diagnosis=no debug=enable valgrind=supported/disabled Process [9]: type=worker(2) index=7 pid=328 state=running               diagnosis=no debug=enable valgrind=supported/disabled Process [10]: type=algo(3) index=0 pid=319 state=running               diagnosis=no debug=enable valgrind=unsupported/disabled Process [11]: type=informer(4) index=0 pid=314 state=running               diagnosis=no debug=enable valgrind=unsupported/disabled Process [12]: type=user-info(5) index=0 pid=318 state=running               diagnosis=no debug=enable valgrind=supported/disabled Process [13]: type=cert-inspection(8) index=0 pid=316 state=running               diagnosis=no debug=enable valgrind=unsupported/disabled Process [14]: type=YouTube-filter-cache-service(9) index=0 pid=317 state=running               diagnosis=no debug=enable valgrind=unsupported/disabled Process [15]: type=debug(11) index=0 pid=313 state=running               diagnosis=no debug=enable valgrind=unsupported/disabled Process [16]: type=config-notify(12) index=0 pid=315 state=running               diagnosis=no debug=enable valgrind=unsupported/disabled Since the PID in question is 318 I see the following from the above command: Process [12]: type=user-info(5) index=0 pid=318 state=running               diagnosis=no debug=enable valgrind=supported/disabled The next CLI command will tell the device that the output of the PID 318 is researched: diagnose debug enable (needed to see any debug output) diag test app wad 2yxx y= type xx=index The type is 5 and the index is 0. So the command would be 'diag test app wad 2500': diagnose test application wad 2500     set diagnosis process: type=user-info index=0 pid=318 Once this is set, it is then possible to go through the list of commands. This is different based on the type of wad process selected. Selecting the type=Manager Process(0) counts as selecting NO WAD process. I.e. default commands. For FortiOS v7.0, v7.2 and v7.4: WAD process 1972 test usage:         1: display process status         2: display total memory usage.         99: restart all WAD processes         1000: List all WAD processes.         1002: display status of WANOpt storages         1051: Enable AV scan bypass for all WAD workers.         1052: Disable AV scan bypass for all WAD workers.         1053: Enable AV unknown file type bypass for all WAD workers.         1054: Disable AV unknown file type bypass for all WAD workers.         1068: Enable debug for all WAD workers.         1069: Disable debug for all WAD workers.         1090: Toggle to write sinks for all WAD workers.         1091: Toggle to use advanced memory for new WAD daemons.         2yxx: Set No. xx process of type y (0~9) as diagnosis process.         2yyxx: Set No. xx process of type yy (10 and above) as diagnosis process.         3: display all fix-sized advanced memory stats         4: display all fix-sized advanced memory stats in details         500000..599999: cmem bucket stats (599999 for usage)         6: display memory tracking table         800..899: mem_diag commands (800 for help & usage)         800000..899999: mem_diag commands with 1 arg (800 for help & usage)         80000000..89999999: mem_diag commands with 2 args (800 for help & usage)         60: show debug stats.         61: discard all wad debug info that is currently pending         62xxx: set xxxM maximum ouput buffer size for WAD debug. 0, set back to default.         68: Enable process debug         69: Disable process debug         90: Toggle to write debug sink.         91: Crash test         98: gracefully stopping WAD process         97: Restart all WAD worker processes.         9xx: Set xx workers(0: default based on user configuration.) For FortiOS v6.4: WAD process 178 test usage:         1: display process status         2: display total memory usage.         99: restart all WAD processes         1000: List all WAD processes.         1001: display debug level name and values         1002: display status of WANOpt storages         1051: Enable AV scan bypass for all WAD workers.         1052: Disable AV scan bypass for all WAD workers.         1053: Enable AV unknown file type bypass for all WAD workers.         1054: Disable AV unknown file type bypass for all WAD workers.         1068: Enable debug for all WAD workers.         1069: Disable debug for all WAD workers.         1090: Toggle to write sinks for all WAD workers.         1091: Toggle to use advanced memory for new WAD daemons.         2yxx: Set No. xx process of type y (0~9) as diagnosis process.         2yyxx: Set No. xx process of type yy (10 and above) as diagnosis process.         3: display all fix-sized advanced memory stats         4: display all fix-sized advanced memory stats in details         500000..599999: cmem bucket stats (599999 for usage)         800..899: mem_diag commands (800 for help & usage)         800000..899999: mem_diag commands with 1 arg (800 for help & usage)         80000000..89999999: mem_diag commands with 2 args (800 for help & usage)         60: show debug stats.         61: discard all wad debug info that is currently pending         62xxx: set xxxM maximum ouput buffer size for WAD debug. 0, set back to default.         68: Enable process debug         69: Disable process debug         90: Toggle to write debug sink.         91: Crash test         98: gracefully stopping WAD process         97: Restart all WAD worker processes.         9xx: Set xx workers(0: default based on user configuration.)