| Description | This article discusses gathering WAD debugs using the 'diagnose test application' debug command to help investigate resource issues. |
| Scope | FortiGate. |
| Solution |
diag sys top (Hit m once command is running) diagnose sys top-mem
n Time: 4 days, 23 hours and 59 minutes 2U, 0N, 2S, 95I, 0WA, 0HI, 1SI, 0ST; 7979T, 3104F wad 318 S 0.0 5.5 3 ipshelper 311 S < 0.0 3.2 2 ipsengine 430 S < 2.3 2.1 4 ipsengine 426 S < 1.4 1.9 0 ipsengine 428 S < 2.3 1.9 2 ipsengine 429 S < 1.8 1.9 3 ipsengine 427 S < 1.8 1.9 1 ipsengine 431 S < 1.4 1.9 5 ipsengine 432 S < 1.6 1.8 6 node 245 S 0.2 1.2 0 cw_acd 292 S 2.1 1.0 2 cmdbsvr 209 S 0.0 0.8 5 src-vis 276 S 5.2 0.8 4 appDemo 175 S < 0.2 0.8 0 miglogd 264 S 0.0 0.5 4 scanunitd 30646 S < 0.0 0.4 5 cu_acd 296 S 1.6 0.4 4 wad 325 S 0.0 0.4 3 wad 327 S 0.0 0.4 0 wad 321 S 0.0 0.4 2
diagnose sys top-mem
wad (318): 434687kB ipshelper (311): 248560kB node (245): 109138kB ipsengine (430): 83070kB ipsengine (428): 70281kB Top-5 memory used: 945736kB
diag debug enable diag test app wad 1000
diagnose test application wad 1000 Process [0]: WAD manager type=manager(0) pid=268 diagnosis=yes. Process [1]: type=dispatcher(1) index=0 pid=320 state=running diagnosis=no debug=enable valgrind=unsupported/disabled Process [2]: type=worker(2) index=0 pid=321 state=running diagnosis=no debug=enable valgrind=supported/disabled Process [3]: type=worker(2) index=1 pid=322 state=running diagnosis=no debug=enable valgrind=supported/disabled Process [4]: type=worker(2) index=2 pid=323 state=running diagnosis=no debug=enable valgrind=supported/disabled Process [5]: type=worker(2) index=3 pid=324 state=running diagnosis=no debug=enable valgrind=supported/disabled Process [6]: type=worker(2) index=4 pid=325 state=running diagnosis=no debug=enable valgrind=supported/disabled Process [7]: type=worker(2) index=5 pid=326 state=running diagnosis=no debug=enable valgrind=supported/disabled Process [8]: type=worker(2) index=6 pid=327 state=running diagnosis=no debug=enable valgrind=supported/disabled Process [9]: type=worker(2) index=7 pid=328 state=running diagnosis=no debug=enable valgrind=supported/disabled Process [10]: type=algo(3) index=0 pid=319 state=running diagnosis=no debug=enable valgrind=unsupported/disabled Process [11]: type=informer(4) index=0 pid=314 state=running diagnosis=no debug=enable valgrind=unsupported/disabled Process [12]: type=user-info(5) index=0 pid=318 state=running diagnosis=no debug=enable valgrind=supported/disabled Process [13]: type=cert-inspection(8) index=0 pid=316 state=running diagnosis=no debug=enable valgrind=unsupported/disabled Process [14]: type=YouTube-filter-cache-service(9) index=0 pid=317 state=running diagnosis=no debug=enable valgrind=unsupported/disabled Process [15]: type=debug(11) index=0 pid=313 state=running diagnosis=no debug=enable valgrind=unsupported/disabled Process [16]: type=config-notify(12) index=0 pid=315 state=running diagnosis=no debug=enable valgrind=unsupported/disabled
Process [12]: type=user-info(5) index=0 pid=318 state=running diagnosis=no debug=enable valgrind=supported/disabled
diagnose debug enable (needed to see any debug output) diag test app wad 2yxx
y= type xx=index
diagnose test application wad 2500 set diagnosis process: type=user-info index=0 pid=318
For FortiOS v7.0, v7.2 and v7.4:
WAD process 1972 test usage: 1: display process status 2: display total memory usage. 99: restart all WAD processes 1000: List all WAD processes. 1002: display status of WANOpt storages 1051: Enable AV scan bypass for all WAD workers. 1052: Disable AV scan bypass for all WAD workers. 1053: Enable AV unknown file type bypass for all WAD workers. 1054: Disable AV unknown file type bypass for all WAD workers. 1068: Enable debug for all WAD workers. 1069: Disable debug for all WAD workers. 1090: Toggle to write sinks for all WAD workers. 1091: Toggle to use advanced memory for new WAD daemons. 2yxx: Set No. xx process of type y (0~9) as diagnosis process. 2yyxx: Set No. xx process of type yy (10 and above) as diagnosis process. 3: display all fix-sized advanced memory stats 4: display all fix-sized advanced memory stats in details 500000..599999: cmem bucket stats (599999 for usage) 6: display memory tracking table 800..899: mem_diag commands (800 for help & usage) 800000..899999: mem_diag commands with 1 arg (800 for help & usage) 80000000..89999999: mem_diag commands with 2 args (800 for help & usage) 60: show debug stats. 61: discard all wad debug info that is currently pending 62xxx: set xxxM maximum ouput buffer size for WAD debug. 0, set back to default. 68: Enable process debug 69: Disable process debug 90: Toggle to write debug sink. 91: Crash test 98: gracefully stopping WAD process 97: Restart all WAD worker processes. 9xx: Set xx workers(0: default based on user configuration.)
For FortiOS v6.4:
WAD process 178 test usage: 1: display process status 2: display total memory usage. 99: restart all WAD processes 1000: List all WAD processes. 1001: display debug level name and values 1002: display status of WANOpt storages 1051: Enable AV scan bypass for all WAD workers. 1052: Disable AV scan bypass for all WAD workers. 1053: Enable AV unknown file type bypass for all WAD workers. 1054: Disable AV unknown file type bypass for all WAD workers. 1068: Enable debug for all WAD workers. 1069: Disable debug for all WAD workers. 1090: Toggle to write sinks for all WAD workers. 1091: Toggle to use advanced memory for new WAD daemons. 2yxx: Set No. xx process of type y (0~9) as diagnosis process. 2yyxx: Set No. xx process of type yy (10 and above) as diagnosis process. 3: display all fix-sized advanced memory stats 4: display all fix-sized advanced memory stats in details 500000..599999: cmem bucket stats (599999 for usage) 800..899: mem_diag commands (800 for help & usage) 800000..899999: mem_diag commands with 1 arg (800 for help & usage) 80000000..89999999: mem_diag commands with 2 args (800 for help & usage) 60: show debug stats. 61: discard all wad debug info that is currently pending 62xxx: set xxxM maximum ouput buffer size for WAD debug. 0, set back to default. 68: Enable process debug 69: Disable process debug 90: Toggle to write debug sink. 91: Crash test 98: gracefully stopping WAD process 97: Restart all WAD worker processes. 9xx: Set xx workers(0: default based on user configuration.)
Refer to the below KB article to set up a monitoring tools to collect WAD debug: Technical Tip: TAC debug script with TeraTerm |
