Skip to main content
jsevigny
Staff
Staff
Oct 3, 2023 4:17 AM UTC

Gathering WAD debugs using the 'diagnose test application' debug command

  • Oct 20, 2026 5:09 PM UTC
  • 0 replies
  • 5265 views
  • Translate All

Description

This article discusses gathering WAD debugs using the 'diagnose test application' debug command to help investigate resource issues.

Scope

FortiGate.

Solution

  1. Determine the PID of the WAD process using the most memory, to do so run one of the following commands or both:


diagnose sys top (Hit m once command is running)
diagnose sys top-mem
n Time:  4 days, 23 hours and 59 minutes
2U, 0N, 2S, 95I, 0WA, 0HI, 1SI, 0ST; 7979T, 3104F
             wad      318      S       0.0     5.5    3
       ipshelper      311      S <     0.0     3.2    2
       ipsengine      430      S <     2.3     2.1    4
       ipsengine      426      S <     1.4     1.9    0
       ipsengine      428      S <     2.3     1.9    2
       ipsengine      429      S <     1.8     1.9    3
       ipsengine      427      S <     1.8     1.9    1
       ipsengine      431      S <     1.4     1.9    5
       ipsengine      432      S <     1.6     1.8    6
            node      245      S       0.2     1.2    0
          cw_acd      292      S       2.1     1.0    2
         cmdbsvr      209      S       0.0     0.8    5
         src-vis      276      S       5.2     0.8    4
         appDemo      175      S <     0.2     0.8    0
         miglogd      264      S       0.0     0.5    4
       scanunitd    30646      S <     0.0     0.4    5
          cu_acd      296      S       1.6     0.4    4
             wad      325      S       0.0     0.4    3
             wad      327      S       0.0     0.4    0
             wad      321      S       0.0     0.4    2
 
diagnose sys top-mem


wad (318): 434687kB
ipshelper (311): 248560kB
node (245): 109138kB
ipsengine (430): 83070kB
ipsengine (428): 70281kB
Top-5 memory used: 945736kB

 

  1.  Next, run the following:

 

diagnose debug enable
diagnose test app wad 1000
diagnose test application wad 1000

Process [0]: WAD manager type=manager(0) pid=268 diagnosis=yes.

Process [1]: type=dispatcher(1) index=0 pid=320 state=running
              diagnosis=no debug=enable valgrind=unsupported/disabled
Process [2]: type=worker(2) index=0 pid=321 state=running
              diagnosis=no debug=enable valgrind=supported/disabled
Process [3]: type=worker(2) index=1 pid=322 state=running
              diagnosis=no debug=enable valgrind=supported/disabled
Process [4]: type=worker(2) index=2 pid=323 state=running
              diagnosis=no debug=enable valgrind=supported/disabled
Process [5]: type=worker(2) index=3 pid=324 state=running
              diagnosis=no debug=enable valgrind=supported/disabled
Process [6]: type=worker(2) index=4 pid=325 state=running
              diagnosis=no debug=enable valgrind=supported/disabled
Process [7]: type=worker(2) index=5 pid=326 state=running
              diagnosis=no debug=enable valgrind=supported/disabled
Process [8]: type=worker(2) index=6 pid=327 state=running
              diagnosis=no debug=enable valgrind=supported/disabled
Process [9]: type=worker(2) index=7 pid=328 state=running
              diagnosis=no debug=enable valgrind=supported/disabled
Process [10]: type=algo(3) index=0 pid=319 state=running
              diagnosis=no debug=enable valgrind=unsupported/disabled
Process [11]: type=informer(4) index=0 pid=314 state=running
              diagnosis=no debug=enable valgrind=unsupported/disabled
Process [12]: type=user-info(5) index=0 pid=318 state=running
              diagnosis=no debug=enable valgrind=supported/disabled
Process [13]: type=cert-inspection(8) index=0 pid=316 state=running
              diagnosis=no debug=enable valgrind=unsupported/disabled
Process [14]: type=YouTube-filter-cache-service(9) index=0 pid=317 state=running
              diagnosis=no debug=enable valgrind=unsupported/disabled
Process [15]: type=debug(11) index=0 pid=313 state=running
              diagnosis=no debug=enable valgrind=unsupported/disabled
Process [16]: type=config-notify(12) index=0 pid=315 state=running
              diagnosis=no debug=enable valgrind=unsupported/disabled

 

 

  • Since the PID in question is 318, you can see the following in the command above:


Process [12]: type=user-info(5) index=0 pid=318 state=running

              diagnosis=no debug=enable valgrind=supported/disabled

 

  1. The next CLI command will tell the device that the output of the PID 318 is researched:


diagnose debug enable (needed to see any debug output)
diagnose test app wad 2yxx
 
y= type
xx=index

 

  1. The type is 5 and the index is 0. So the command would be 'diag test app wad 2500':


diagnose test application wad 2500

    set diagnosis process: type=user-info index=0 pid=318

 

  1. Once this is set, it is then possible to go through the list of commands. This is different based on the type of wad process selected. Selecting the type=Manager Process(0) counts as selecting NO WAD process. I.e. default commands.

 

For FortiOS v7.0, v7.2 and v7.4:

WAD process 1972 test usage:
        1: display process status
        2: display total memory usage.
        99: restart all WAD processes
        1000: List all WAD processes.
        1002: display status of WANOpt storages
        1051: Enable AV scan bypass for all WAD workers.
        1052: Disable AV scan bypass for all WAD workers.
        1053: Enable AV unknown file type bypass for all WAD workers.
        1054: Disable AV unknown file type bypass for all WAD workers.
        1068: Enable debug for all WAD workers.
        1069: Disable debug for all WAD workers.
        1090: Toggle to write sinks for all WAD workers.
        1091: Toggle to use advanced memory for new WAD daemons.
        2yxx: Set No. xx process of type y (0~9) as diagnosis process.
        2yyxx: Set No. xx process of type yy (10 and above) as diagnosis process.
        3: display all fix-sized advanced memory stats
        4: display all fix-sized advanced memory stats in details
        500000..599999: cmem bucket stats (599999 for usage)
        6: display memory tracking table
        800..899: mem_diag commands (800 for help & usage)
        800000..899999: mem_diag commands with 1 arg (800 for help & usage)
        80000000..89999999: mem_diag commands with 2 args (800 for help & usage)
        60: show debug stats.
        61: discard all wad debug info that is currently pending
        62xxx: set xxxM maximum ouput buffer size for WAD debug. 0, set back to default.
        68: Enable process debug
        69: Disable process debug
        90: Toggle to write debug sink.
        91: Crash test
        98: gracefully stopping WAD process
        97: Restart all WAD worker processes.
        9xx: Set xx workers(0: default based on user configuration.)

 

For FortiOS v6.4:

WAD process 178 test usage:
        1: display process status
        2: display total memory usage.
        99: restart all WAD processes
        1000: List all WAD processes.
        1001: display debug level name and values
        1002: display status of WANOpt storages
        1051: Enable AV scan bypass for all WAD workers.
        1052: Disable AV scan bypass for all WAD workers.
        1053: Enable AV unknown file type bypass for all WAD workers.
        1054: Disable AV unknown file type bypass for all WAD workers.
        1068: Enable debug for all WAD workers.
        1069: Disable debug for all WAD workers.
        1090: Toggle to write sinks for all WAD workers.
        1091: Toggle to use advanced memory for new WAD daemons.
        2yxx: Set No. xx process of type y (0~9) as diagnosis process.
        2yyxx: Set No. xx process of type yy (10 and above) as diagnosis process.
        3: display all fix-sized advanced memory stats
        4: display all fix-sized advanced memory stats in details
        500000..599999: cmem bucket stats (599999 for usage)
        800..899: mem_diag commands (800 for help & usage)
        800000..899999: mem_diag commands with 1 arg (800 for help & usage)
        80000000..89999999: mem_diag commands with 2 args (800 for help & usage)
        60: show debug stats.
        61: discard all wad debug info that is currently pending
        62xxx: set xxxM maximum ouput buffer size for WAD debug. 0, set back to default.
        68: Enable process debug
        69: Disable process debug
        90: Toggle to write debug sink.
        91: Crash test
        98: gracefully stopping WAD process
        97: Restart all WAD worker processes.
        9xx: Set xx workers(0: default based on user configuration.)


Refer to the following KB article to set up monitoring tools to collect WAD debug output: Technical Tip: TAC debug script with TeraTerm.


Related articles:

    Risk and Resilience in the Age of AI. An OT Thought Leadership Summit for Houston's Critical Infrastructure Leaders. Date: Thursday, December 3, 2026. Time: 8:30 am - 1:30 pm (CT). Location: The Westin Houston, Memorial City | 945 Gessner Road, Houston, TX 77024