WAD CPU affinity

Description

An explanation of how setting WAD affinity works in FortiGate. By default, FortiGate spreads the WAD processes across all CPU cores. With the usage of the set wad-affinity these processes can be restricted to run on specific CPU cores.

Scope

FortiGate.

Solution

This setting is available under config system global and required a hex number to select all the wanted CPU cores. So for example a hex number of 0x3 would correspond to the first two CPU cores. All other cores would therefore be ineligible to be used by the WAD process.

This configuration change also requires a reboot of the device.

config system global
   set wad-affinity <HEX_BITMASK>
end

What is important to understand is that this restriction will be placed on all wad sub-processes and not only wad workers. This may cause high CPU usage since all work will be done by fewer CPU cores. For a detailed description of wad-sub processes refer to this article Overview of wad process structure.

Example:

With diagnose test application 1000, you get the process IDs (PID) and the corresponding wad-sub-processes in detail.

diagnose debug enable
diagnose test application wad 1000

Process [0]: WAD manager type=manager(0) pid=2106 diagnosis=yes.
Process [1]: type=worker(2) index=0 pid=2206 state=running
diagnosis=no debug=enable valgrind=supported/disabled

...
CUT
...

Process [13]: type=config-notify(16) index=0 pid=2203 state=running
diagnosis=no debug=enable valgrind=supported/disabled

What cores the wad is eligible to run on can be checked with the command fnsysctl cat /proc/<PID>/status. For this, you require the PID from the previous command above. Under Cpus_allowed_list, you can see all cores which are eligible for usage. In this example you have 4 cores, so all are available for the WAD.

fnsysctl cat /proc/2206/status

Name: wad
Umask: 0022
State: S (sleeping)
Tgid: 2206
Ngid: 0
Pid: 2206
PPid: 2106

...
CUT
...

Speculation_Store_Bypass: thread vulnerable
Cpus_allowed: f
Cpus_allowed_list: 0-3
Mems_allowed: 00000000,00000001
Mems_allowed_list: 0
voluntary_ctxt_switches: 1172
nonvoluntary_ctxt_switches: 36

Setting the WAD affinity will cause a restart of the device.

config system global
    set wad-affinity "0x0000000000000003"
end

Since the device rebooted, the PIDs changed so they have to be gathered again.

diagnose test app wad 1000

Process [0]: WAD manager type=manager(0) pid=2094 diagnosis=yes.
Process [1]: type=worker(2) index=0 pid=2198 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [2]: type=algo(3) index=0 pid=2196 state=running
diagnosis=no debug=enable valgrind=supported/disabled

...
CUT
...

Process [13]: type=config-notify(16) index=0 pid=2195 state=running
diagnosis=no debug=enable valgrind=supported/disabled

Checking again the Cpus_allowed_list is not only allowing the first two cores. This is true for both the WAD worker with PID 2198 and also the WAD manager process with PID 2094. In fact, all WAD processes now run on core 0 and 1.

fnsysctl cat /proc/2198/status

Name: wad
Umask: 0022
State: S (sleeping)
Tgid: 2198
Ngid: 0
Pid: 2198
PPid: 2094

...
CUT
...

Cpus_allowed: 3
Cpus_allowed_list: 0-1
Mems_allowed: 00000000,00000001
Mems_allowed_list: 0
voluntary_ctxt_switches: 178
nonvoluntary_ctxt_switches: 38

fnsysctl cat /proc/2094/status

Name: wad
Umask: 0022
State: S (sleeping)
Tgid: 2094
Ngid: 0
Pid: 2094
PPid: 1

...
CUT
...

Cpus_allowed: 3
Cpus_allowed_list: 0-1
Mems_allowed: 00000000,00000001
Mems_allowed_list: 0
voluntary_ctxt_switches: 50
nonvoluntary_ctxt_switches: 104