Staff

Technical Tip: Overview of WAD process structure

Forum|Forum|7 years ago
October 1, 2018
0 replies
102157 views

Description

This article describes the components of the FortiOS webproxy process named WAD.

Scope

FortiGate.

Solution

The WAD process structure is made up of multiple processes. Depending on the firmware version, the output may differ.

Below is an example of a FortiGate-VM64-KVM v7.2.4:

diagnose test application wad 1000
Process [0]: WAD manager type=manager(0) pid=1963 diagnosis=yes.
Process [1]: type=worker(2) index=0 pid=19429 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [2]: type=algo(3) index=0 pid=19428 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled
Process [3]: type=informer(4) index=0 pid=1990 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled
Process [4]: type=user-info(5) index=0 pid=1991 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [5]: type=dev-vuln(6) index=0 pid=1992 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [6]: type=cert-inspection(9) index=0 pid=19430 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled
Process [7]: type=user-info-history(11) index=0 pid=1993 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [8]: type=debug(12) index=0 pid=1994 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled
Process [9]: type=config-notify(13) index=0 pid=1995 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled

Below is an example of a FortiGate-2000E on an older firmware release:

diagnose test application wad 1000
Process [0]: WAD manager type=manager(0) pid=236 diagnosis=yes.
Process [1]: type=dispatcher(1) index=0 pid=250 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled
Process [2]: type=wanopt(2) index=0 pid=252 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [3]: type=worker(3) index=0 pid=255 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [4]: type=worker(3) index=1 pid=257 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [5]: type=worker(3) index=2 pid=259 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [6]: type=worker(3) index=3 pid=261 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [7]: type=worker(3) index=4 pid=263 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [8]: type=worker(3) index=5 pid=264 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [9]: type=worker(3) index=6 pid=265 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [10]: type=worker(3) index=7 pid=266 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [11]: type=worker(3) index=8 pid=267 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [12]: type=worker(3) index=9 pid=268 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [13]: type=informer(4) index=0 pid=247 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled

Note the process types:

Process [0]: This is the 'WAD manager' responsible for handling/restarting the 'WAD workers'.
Process [1]: This is the 'WAD dispatcher' responsible for dispatching the requests to the 'WAD workers'.
Process [2]: This is the 'WAD wanopt' responsible for WAN optimization, but is also a 'WAD worker'.
Processes [3] to [12]: These are the 'WAD workers' responsible for handling HTTP/HTTPS requests, DNS queries, policy validation, work with UTM inspection engines like AV/URL filters, etc.
Process [13]: This is the 'WAD informer' responsible for collecting data/stats and other information from the different 'WAD workers'.

The number of WAD workers depends on the hardware. The highest unit has the highest number of WAD worker processes.

By default, one WAD worker is spawned per CPU core.

The number of WAD workers can be configured with a default value:

config system global

set wad-worker-count xx

Setting a limit on the number of workers can free up RAM since every worker spawned comes with a certain memory overhead attached to it. Conversely, reducing the number of workers will increase the CPU usage.

By default, the WAD worker count is set to 0, which will allow the system to automatically determine the number of spawned workers.

show full-configuration system global | grep wad-worker

set wad-worker-count 0

To display the list of processes or use other WAD commands, first enable debug logging with the following command:

diagnose debug enable

Enter the following command to view the list of WAD processes:

diagnose test application wad 1000

One of the processes has the 'diagnosis=yes' flag enabled. This is the 'WAD manager'.

Consider process context. By default, when the debug logging is enabled, the FortiGate CLI operates in the 'WAD manager' context.

View the context number with the following command:

diagnose test application wad 1000

Note the WAD processes in the example output:

Process [0]: WAD manager type=manager(0) pid=236 diagnosis=yes.
Process [1]: type=dispatcher(1) index=0 pid=250 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled
Process [2]: type=wanopt(2) index=0 pid=252 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [3]: type=worker(3) index=0 pid=255 state=running

Change the context of the process by specifying a process context number:

diagnose test application wad 2x00

Replace 'x' with the WAD process type number.

Examples:

diagnose test application wad 2000
Set diagnosis process to default: WAD manager process pid=236

diagnose test application wad 2100
Set diagnosis process: type=dispatcher index=0 pid=250

diagnose test application wad 2200
Set diagnosis process: type=wanopt index=0 pid=252

diagnose test application wad 2300
Set diagnosis process: type=worker index=0 pid=255

To select a different process of the worker type, replace 'y' with the index number in the following command and run it:

diagnose test application wad 230y

For example:

diagnose test application wad 2301
Set diagnosis process: type=worker index=1 pid=257

Once the FortiGate is in a 'process context', it has access to specific debug/troubleshooting commands for the process. To list all available commands, enter the following command:

diagnose test application wad

For example, enter a proxy worker context to view sessions:

diagnose test application wad 2300
Set diagnosis process: type=worker index=0 pid=255

diagnose test application wad 21
TCP stats: active=0 accepts=0 connects=307 accept_err=0
connect_err=0 bind_fails=0 make_failure=0 connected=305
early_conn_err=0, net_conn_err=0
TCP port: without_ses_ctx:0 with_ses_ctx:0

Note: If the process type consists of multiple digits (for example, '12' for 'user-info-history'), use the full value instead of a single digit.

Example:

From process list:

diagnose test application wad 1000

Process [13]: type=user-info-history(12) index=0 pid=371 state=running

diagnosis=no debug=enable valgrind=supported/disabled

To switch context:

diagnose test application wad 21200
Set diagnosis process: type=user-info-history index=0 pid=371
* '12' → user-info-history process type
* '00' → Index 0

To learn which process context FortiGate is in, use the following command:

diagnose test application wad 1000
Process [0]: WAD manager type=manager(0) pid=236 diagnosis=no.
Process [1]: type=dispatcher(1) index=0 pid=250 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled
Process [2]: type=wanopt(2) index=0 pid=252 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [3]: type=worker(3) index=0 pid=255 state=running
diagnosis=yes debug=enable valgrind=supported/disabled
Process [4]: type=worker(3) index=1 pid=257 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [5]: type=worker(3) index=2 pid=259 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [6]: type=worker(3) index=3 pid=261 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [7]: type=worker(3) index=4 pid=263 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [8]: type=worker(3) index=5 pid=264 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [9]: type=worker(3) index=6 pid=265 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [10]: type=worker(3) index=7 pid=266 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [11]: type=worker(3) index=8 pid=267 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [12]: type=worker(3) index=9 pid=268 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [13]: type=informer(4) index=0 pid=247 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled

Process [3] has the 'diagnosis=yes' flag, which means FortiGate is currently in this WAD worker context. To restart all WAD processes, follow these steps:

Enter the WAD manager context.

diagnose test application wad 2000

This sets the diagnosis process to the default: WAD manager process pid=236.

Enter the restart command:

diagnose test application wad 99

It is possible to restart a specific WAD process using the PID.

diagnose debug enable

diagnose test application wad 1000
Process [0]: WAD manager type=manager(0) pid=13762 diagnosis=yes.
Process [1]: type=dispatcher(1) index=0 pid=13774 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled
Process [2]: type=worker(2) index=0 pid=13775 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [3]: type=worker(2) index=1 pid=13776 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [4]: type=worker(2) index=2 pid=13777 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [5]: type=worker(2) index=3 pid=13778 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [6]: type=worker(2) index=4 pid=13779 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [7]: type=worker(2) index=5 pid=13780 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [8]: type=worker(2) index=6 pid=13781 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [9]: type=worker(2) index=7 pid=13782 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [10]: type=algo(3) index=0 pid=13773 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled
Process [11]: type=informer(4) index=0 pid=13765 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled
Process [12]: type=user-info(5) index=0 pid=13766 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [13]: type=dev-vuln(6) index=0 pid=13767 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [14]: type=cert-inspection(9) index=0 pid=13768 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled
Process [15]: type=YouTube-filter-cache-service(10) index=0 pid=13769 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled
Process [16]: type=user-info-history(11) index=0 pid=13770 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [17]: type=debug(12) index=0 pid=13771 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled
Process [18]: type=config-notify(13) index=0 pid=13772 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled

To determine the memory and CPU consumption of active WAD processes, run the following CLI command:

diagnose wad memory report | grep -A20 NAME
NAME PID STATE %CPU %MEM
wad-worker 13775 S 0.0 1.1
wad-worker 13777 S 0.0 1.1
wad-worker 13779 S 0.0 1.1
wad-worker 13782 S 0.0 1.1
wad-worker 13776 S 0.0 1.1
wad-worker 13778 S 0.0 1.1
wad-worker 13780 S 0.0 1.1
wad-worker 13781 S 0.0 1.1
wad-algo 13773 S 0.0 0.5
wad-manager 13762 S 0.0 0.3
wad-user-info 13766 S 0.0 0.3
wad-dev-vuln 13767 S 0.0 0.3
wad-dispatcher 13774 S 0.0 0.2
wad-config-notify 13772 S 0.0 0.2
wad-informer 13765 S 0.0 0.2
wad-debug 13771 S 0.0 0.2
wad-cert-inspection 13768 S 0.0 0.2
wad-yfcs 13769 S 0.0 0.2
wad-user-info-history 13770 S 0.0 0.2

For example, to restart the WAD user-info. PID is 13766 from the above command. The PID will change after the restart:

diagnose sys kill 11 13766

diagnose wad memory report | grep -A20 NAME
NAME PID STATE %CPU %MEM
......
wad-user-info 13856 S 2.3 0.3
......

Notes:

The WAD process is used for proxy-based inspection. In certain scenarios of high CPU/memory consumption by WAD or where WAD is crashing repeatedly, it may be necessary to restart the process as a workaround. Restarting the WAD may interrupt the inspection. It is recommended to restart the process during the maintenance period.
Find different ways to restart the WAD process and understand the impacts of restarting: Technical Tip: How to restart the WAD process.
This can be automated via the 'config system auto-script' feature: Technical Tip: Automatically restart WAD worker processes.
If further troubleshooting on the WAD process on high CPU/Memory, the TAC engineer may ask to run the TeraTerm script to collect debug continuously: Technical Tip: TAC debug script with TeraTerm or Troubleshooting Tip: Using a PID process debugging TeraTerm script.

Related article to restart a FortiGate process:

FortiGate

Sign up

Login with SSO

Login to the community

Login with SSO

Scanning file for viruses.

This file cannot be downloaded