Description
This article describes the components of the FortiOS webproxy process named WAD.
Scope
FortiGate.
Solution
The wad process structure is made of multiple processes. Depending on the firmware version, the output may differ.
Below is an example on a FortiGate-VM64-KVM v7.2.4:
diagnose test application wad 1000
Process [0]: WAD manager type=manager(0) pid=1963 diagnosis=yes.
Process [1]: type=worker(2) index=0 pid=19429 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [2]: type=algo(3) index=0 pid=19428 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled
Process [3]: type=informer(4) index=0 pid=1990 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled
Process [4]: type=user-info(5) index=0 pid=1991 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [5]: type=dev-vuln(6) index=0 pid=1992 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [6]: type=cert-inspection(9) index=0 pid=19430 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled
Process [7]: type=user-info-history(11) index=0 pid=1993 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [8]: type=debug(12) index=0 pid=1994 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled
Process [9]: type=config-notify(13) index=0 pid=1995 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled
Below is an example on an FortiGate-2000E on an older firmware release:
diagnose test application wad 1000
Process [0]: WAD manager type=manager(0) pid=236 diagnosis=yes.
Process [1]: type=dispatcher(1) index=0 pid=250 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled
Process [2]: type=wanopt(2) index=0 pid=252 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [3]: type=worker(3) index=0 pid=255 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [4]: type=worker(3) index=1 pid=257 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [5]: type=worker(3) index=2 pid=259 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [6]: type=worker(3) index=3 pid=261 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [7]: type=worker(3) index=4 pid=263 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [8]: type=worker(3) index=5 pid=264 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [9]: type=worker(3) index=6 pid=265 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [10]: type=worker(3) index=7 pid=266 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [11]: type=worker(3) index=8 pid=267 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [12]: type=worker(3) index=9 pid=268 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [13]: type=informer(4) index=0 pid=247 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled
Note the process types:
- Process [0]: This is the 'WAD manager' responsible for handling/restarting the 'WAD workers'.
- Process [1]: This is the 'WAD dispatcher' responsible for dispatching the requests to the 'WAD workers'.
- Process [2]: This is the 'WAD wanopt' responsible for WAN optimization, but is also a 'WAD worker'.
- Processes [3] to [12]: These are the 'WAD workers' responsible for handling HTTP/HTTPS requests.
- Process [13]: This is the 'WAD informer' responsible for collecting data/stats and other information from the different 'WAD workers'.
The number of WAD workers depends on the hardware. The highest unit has the highest number of WAD worker process.
By default, one WAD worker is spawned per CPU core.
The number of WAD workers can be configured with a default value:
config system global
set wad-worker-count xx
Setting a limit on the number of workers can free up RAM since every worker spawned comes with a certain memory overhead attached to it. Conversely, reducing the number of workers will increase the CPU usage.
By default, the WAD worker count is set to 0, which will allow the system to automatically determine the amount of spawned workers.
show full-configuration system global | grep wad-worker
set wad-worker-count 0
To display the list of processes or use other WAD commands, first enable debug logging with the following command:
diagnose debug enable
Enter the following command to view the list of WAD processes:
diagnose test application wad 1000
One of the processes has the 'diagnosis=yes' flag enabled. This is the 'WAD manager'.
Consider process context. By default, when the debug logging is enabled, the FortiGate CLI operates in the 'WAD manager' context.
View the context number with the following command:
diagnose test application wad 1000
Note the WAD processes in the example output:
Process [0]: WAD manager type=manager(0) pid=236 diagnosis=yes.
Process [1]: type=dispatcher(1) index=0 pid=250 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled
Process [2]: type=wanopt(2) index=0 pid=252 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [3]: type=worker(3) index=0 pid=255 state=running
Change the context of the process by specifying a process context number:
diagnose test application wad 2x00
Replace 'x' with the WAD process type number.
Examples:
diagnose test application wad 2000
Set diagnosis process to default: WAD manager process pid=236
diagnose test application wad 2100
Set diagnosis process: type=dispatcher index=0 pid=250
diagnose test application wad 2200
Set diagnosis process: type=wanopt index=0 pid=252
diagnose test application wad 2300
Set diagnosis process: type=worker index=0 pid=255
To select a different process of the worker type, replace 'y' with the index number in the following command and run it:
diagnose test application wad 230y
For example:
diagnose test application wad 2301
Set diagnosis process: type=worker index=1 pid=257
Once the FortiGate is in a 'process context', it has access to specific debug/troubleshooting commands for the process. To list all available commands enter the following command:
diagnose test application wad
For example, enter a proxy worker context to view sessions:
diagnose test application wad 2300
Set diagnosis process: type=worker index=0 pid=255
diagnose test application wad 21
TCP stats: active=0 accepts=0 connects=307 accept_err=0
connect_err=0 bind_fails=0 make_failure=0 connected=305
early_conn_err=0, net_conn_err=0
TCP port: without_ses_ctx:0 with_ses_ctx:0
To learn which process context FortiGate is in, use the following command:
diagnose test application wad 1000
Process [0]: WAD manager type=manager(0) pid=236 diagnosis=no.
Process [1]: type=dispatcher(1) index=0 pid=250 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled
Process [2]: type=wanopt(2) index=0 pid=252 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [3]: type=worker(3) index=0 pid=255 state=running
diagnosis=yes debug=enable valgrind=supported/disabled
Process [4]: type=worker(3) index=1 pid=257 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [5]: type=worker(3) index=2 pid=259 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [6]: type=worker(3) index=3 pid=261 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [7]: type=worker(3) index=4 pid=263 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [8]: type=worker(3) index=5 pid=264 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [9]: type=worker(3) index=6 pid=265 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [10]: type=worker(3) index=7 pid=266 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [11]: type=worker(3) index=8 pid=267 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [12]: type=worker(3) index=9 pid=268 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [13]: type=informer(4) index=0 pid=247 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled
Process [3] has the 'diagnosis=yes' flag, which means FortiGate is currently in this WAD worker context. To restart all WAD processes, follow these steps:
- Enter the WAD manager context.
diagnose test application wad 2000
This sets the diagnosis process to the default: WAD manager process pid=236.
- Enter the restart command:
diagnose test application wad 99
It is possible to restart a specific WAD process using the PID.
diag debug enable
diag test app wad 1000
Process [0]: WAD manager type=manager(0) pid=13762 diagnosis=yes.
Process [1]: type=dispatcher(1) index=0 pid=13774 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled
Process [2]: type=worker(2) index=0 pid=13775 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [3]: type=worker(2) index=1 pid=13776 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [4]: type=worker(2) index=2 pid=13777 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [5]: type=worker(2) index=3 pid=13778 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [6]: type=worker(2) index=4 pid=13779 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [7]: type=worker(2) index=5 pid=13780 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [8]: type=worker(2) index=6 pid=13781 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [9]: type=worker(2) index=7 pid=13782 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [10]: type=algo(3) index=0 pid=13773 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled
Process [11]: type=informer(4) index=0 pid=13765 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled
Process [12]: type=user-info(5) index=0 pid=13766 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [13]: type=dev-vuln(6) index=0 pid=13767 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [14]: type=cert-inspection(9) index=0 pid=13768 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled
Process [15]: type=YouTube-filter-cache-service(10) index=0 pid=13769 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled
Process [16]: type=user-info-history(11) index=0 pid=13770 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [17]: type=debug(12) index=0 pid=13771 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled
Process [18]: type=config-notify(13) index=0 pid=13772 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled
diag wad memory report | grep -A20 NAME
NAME PID STATE %CPU %MEM
wad-worker 13775 S 0.0 1.1
wad-worker 13777 S 0.0 1.1
wad-worker 13779 S 0.0 1.1
wad-worker 13782 S 0.0 1.1
wad-worker 13776 S 0.0 1.1
wad-worker 13778 S 0.0 1.1
wad-worker 13780 S 0.0 1.1
wad-worker 13781 S 0.0 1.1
wad-algo 13773 S 0.0 0.5
wad-manager 13762 S 0.0 0.3
wad-user-info 13766 S 0.0 0.3
wad-dev-vuln 13767 S 0.0 0.3
wad-dispatcher 13774 S 0.0 0.2
wad-config-notify 13772 S 0.0 0.2
wad-informer 13765 S 0.0 0.2
wad-debug 13771 S 0.0 0.2
wad-cert-inspection 13768 S 0.0 0.2
wad-yfcs 13769 S 0.0 0.2
wad-user-info-history 13770 S 0.0 0.2
For example, to restart the WAD user-info. PID is 13766 from the above command. The PID will change after the restart:
diag sys kill 11 13766
diag wad memory report | grep -A20 NAME
NAME PID STATE %CPU %MEM
......
wad-user-info 13856 S 2.3 0.3
......
Note:
The WAD process is used for proxy-based inspection. In certain scenarios of high CPU/memory consumption by WAD or where WAD is crashing repeatedly, it may be necessary to restart the process as a workaround. Restarting the WAD may interrupt the inspection. It is recommended to restart the process during the maintenance period.
