Determine the PID of the WAD process using the most memory, to do so run one of the following commands or both:
diagnose sys top (Hit m once command is running)
diagnose sys top-mem
n Time:Â 4 days, 23 hours and 59 minutes
2U, 0N, 2S, 95I, 0WA, 0HI, 1SI, 0ST; 7979T, 3104F
       wad   318   S    0.0   5.5  3
    ipshelper   311   S <   0.0   3.2  2
    ipsengine   430   S <   2.3   2.1  4
    ipsengine   426   S <   1.4   1.9  0
    ipsengine   428   S <   2.3   1.9  2
    ipsengine   429   S <   1.8   1.9  3
    ipsengine   427   S <   1.8   1.9  1
    ipsengine   431   S <   1.4   1.9  5
    ipsengine   432   S <   1.6   1.8  6
     node   245   S    0.2   1.2  0
     cw_acd   292   S    2.1   1.0  2
     cmdbsvr   209   S    0.0   0.8  5
     src-vis   276   S    5.2   0.8  4
     appDemo   175   S <   0.2   0.8  0
     miglogd   264   S    0.0   0.5  4
    scanunitd  30646   S <   0.0   0.4  5
     cu_acd   296   S    1.6   0.4  4
       wad   325   S    0.0   0.4  3
       wad   327   S    0.0   0.4  0
       wad   321   S    0.0   0.4  2
Â
diagnose sys top-mem
wad (318): 434687kB
ipshelper (311): 248560kB
node (245): 109138kB
ipsengine (430): 83070kB
ipsengine (428): 70281kB
Top-5 memory used: 945736kB
  Next, run the following:
Â
diagnose debug enable
diagnose test app wad 1000
diagnose test application wad 1000
Process [0]: WAD manager type=manager(0) pid=268 diagnosis=yes.
Process [1]: type=dispatcher(1) index=0 pid=320 state=running
       diagnosis=no debug=enable valgrind=unsupported/disabled
Process [2]: type=worker(2) index=0 pid=321 state=running
       diagnosis=no debug=enable valgrind=supported/disabled
Process [3]: type=worker(2) index=1 pid=322 state=running
       diagnosis=no debug=enable valgrind=supported/disabled
Process [4]: type=worker(2) index=2 pid=323 state=running
       diagnosis=no debug=enable valgrind=supported/disabled
Process [5]: type=worker(2) index=3 pid=324 state=running
       diagnosis=no debug=enable valgrind=supported/disabled
Process [6]: type=worker(2) index=4 pid=325 state=running
       diagnosis=no debug=enable valgrind=supported/disabled
Process [7]: type=worker(2) index=5 pid=326 state=running
       diagnosis=no debug=enable valgrind=supported/disabled
Process [8]: type=worker(2) index=6 pid=327 state=running
       diagnosis=no debug=enable valgrind=supported/disabled
Process [9]: type=worker(2) index=7 pid=328 state=running
       diagnosis=no debug=enable valgrind=supported/disabled
Process [10]: type=algo(3) index=0 pid=319 state=running
       diagnosis=no debug=enable valgrind=unsupported/disabled
Process [11]: type=informer(4) index=0 pid=314 state=running
       diagnosis=no debug=enable valgrind=unsupported/disabled
Process [12]: type=user-info(5) index=0 pid=318 state=running
       diagnosis=no debug=enable valgrind=supported/disabled
Process [13]: type=cert-inspection(8) index=0 pid=316 state=running
       diagnosis=no debug=enable valgrind=unsupported/disabled
Process [14]: type=YouTube-filter-cache-service(9) index=0 pid=317 state=running
       diagnosis=no debug=enable valgrind=unsupported/disabled
Process [15]: type=debug(11) index=0 pid=313 state=running
       diagnosis=no debug=enable valgrind=unsupported/disabled
Process [16]: type=config-notify(12) index=0 pid=315 state=running
       diagnosis=no debug=enable valgrind=unsupported/disabled
 Â
Process [12]: type=user-info(5) index=0 pid=318 state=running
       diagnosis=no debug=enable valgrind=supported/disabled
 The next CLI command will tell the device that the output of the PID 318 is researched:
diagnose debug enable (needed to see any debug output)
diagnose test app wad 2yxx
Â
y= type
xx=index
 The type is 5 and the index is 0. So the command would be 'diag test app wad 2500':
diagnose test application wad 2500
  set diagnosis process: type=user-info index=0 pid=318
 Once this is set, it is then possible to go through the list of commands. This is different based on the type of wad process selected. Selecting the type=Manager Process(0) counts as selecting NO WAD process. I.e. default commands.
 For FortiOS v7.0, v7.2 and v7.4:
WAD process 1972 test usage:
       1: display process status
       2: display total memory usage.
       99: restart all WAD processes
       1000: List all WAD processes.
       1002: display status of WANOpt storages
       1051: Enable AV scan bypass for all WAD workers.
       1052: Disable AV scan bypass for all WAD workers.
       1053: Enable AV unknown file type bypass for all WAD workers.
       1054: Disable AV unknown file type bypass for all WAD workers.
       1068: Enable debug for all WAD workers.
       1069: Disable debug for all WAD workers.
       1090: Toggle to write sinks for all WAD workers.
       1091: Toggle to use advanced memory for new WAD daemons.
       2yxx: Set No. xx process of type y (0~9) as diagnosis process.
       2yyxx: Set No. xx process of type yy (10 and above) as diagnosis process.
       3: display all fix-sized advanced memory stats
       4: display all fix-sized advanced memory stats in details
       500000..599999: cmem bucket stats (599999 for usage)
       6: display memory tracking table
      800..899: mem_diag commands (800 for help & usage)
       800000..899999: mem_diag commands with 1 arg (800 for help & usage)
       80000000..89999999: mem_diag commands with 2 args (800 for help & usage)
       60: show debug stats.
       61: discard all wad debug info that is currently pending
       62xxx: set xxxM maximum ouput buffer size for WAD debug. 0, set back to default.
       68: Enable process debug
       69: Disable process debug
       90: Toggle to write debug sink.
       91: Crash test
       98: gracefully stopping WAD process
       97: Restart all WAD worker processes.
       9xx: Set xx workers(0: default based on user configuration.)
 For FortiOS v6.4:
WAD process 178 test usage:
       1: display process status
       2: display total memory usage.
       99: restart all WAD processes
       1000: List all WAD processes.
       1001: display debug level name and values
       1002: display status of WANOpt storages
       1051: Enable AV scan bypass for all WAD workers.
       1052: Disable AV scan bypass for all WAD workers.
       1053: Enable AV unknown file type bypass for all WAD workers.
       1054: Disable AV unknown file type bypass for all WAD workers.
       1068: Enable debug for all WAD workers.
       1069: Disable debug for all WAD workers.
       1090: Toggle to write sinks for all WAD workers.
       1091: Toggle to use advanced memory for new WAD daemons.
       2yxx: Set No. xx process of type y (0~9) as diagnosis process.
       2yyxx: Set No. xx process of type yy (10 and above) as diagnosis process.
       3: display all fix-sized advanced memory stats
       4: display all fix-sized advanced memory stats in details
       500000..599999: cmem bucket stats (599999 for usage)
       800..899: mem_diag commands (800 for help & usage)
       800000..899999: mem_diag commands with 1 arg (800 for help & usage)
       80000000..89999999: mem_diag commands with 2 args (800 for help & usage)
       60: show debug stats.
       61: discard all wad debug info that is currently pending
       62xxx: set xxxM maximum ouput buffer size for WAD debug. 0, set back to default.
       68: Enable process debug
       69: Disable process debug
       90: Toggle to write debug sink.
       91: Crash test
       98: gracefully stopping WAD process
       97: Restart all WAD worker processes.
       9xx: Set xx workers(0: default based on user configuration.)
Refer to the following KB article to set up monitoring tools to collect WAD debug output: Technical Tip: TAC debug script with TeraTerm.
Related articles:
|