Description
This article describes how to reduce memory usage by reducing some processes in FortiOS such as the IPS engine, WAD and SSL VPN which spawn a child process for each CPU core.
Each of the spawned child processes will have some memory allocated to it regardless of the traffic load.
For some units with multi-core CPUs and less amount of memory, mainly 'E' or 'F' series such as 30E, 60E and 90E this can create an issue with memory usage.
Scope
FortiGate.
Solution
To solve memory usage issues, it is recommended to decrease the number of instances spawned by the aforementioned processes.
As a general guideline the count of workers should be reduced as on low end devices like the models 30/40/60/80 as follows:
config system global
set miglogd-children 1
set sslvpn-max-worker-count 1
set wad-worker-count 1
set scanunit-count 2
end
config ips global
set engine-count 1
end
For more details and different use cases refer to the steps below.
Confirm the number of instances spawned:
To list the number of running processes use the "diag sys top" command. This will only list he most busy processes:
diag sys top 1 99 1
Alternatively the command 'fnsysctl ps' can be used to list all processes running on the FortiGate. Here the count of workers has to be manually added.
fnsysctl ps
Or the command 'diag sys process pidof' can be used on current firmware releases to list all process IDs of a given process name:
diagnose sys process pidof wad
On older firmware releases the 'diag sys top-summary' provides a summarized list of the most busy processes. For firmware above v6.4 refer to the command 'diag sys top' mentioned below in this article:
diagnose sys top-summary
CPU [| ] 4.8%
Mem [||||||||||||||||| ] 44.0% 836M/1866M
Processes: 20 (running=1 sleeping=130)
PID RSS CPU% ^MEM% FDS TIME+ NAME
* 144 163M 0.0 8.7 390 00:18.49 ipsmonitor [x6]
217 66M 0.0 3.6 197 00:20.76 wad [x9]
In the example above, there are x6 instances of the IPS engine and x9 of WAD, which all consume 8.7+3.6 = 12.3% of the memory while this unit processes almost no traffic at all.
Note that the command 'diagnose sys top-summary' is not supported in v6.4 or later, however, 'diagnose sys top <delay> <number of processes>' will return the list of processes/workers. Be aware that the 'diagnose sys top' command is limited to a maximum of 99 entries. To list all processes the 'fnsysctl ps' command can be used.
More info can be found in Technical Tip: Deprecated of command '# diagnose sys top-summary'.
An example of the top command:
diagnose sys top 1 99 1
Run Time: 33 days, 4 hours and 16 minutes
0U, 0N, 0S, 100I, 0WA, 0HI, 0SI, 0ST; 1863T, 463F
cw_wtpd 196 R 0.4 0.5
newcli 6591 R 0.4 0.3
ipsengine 6826 S < 0.0 4.5
ipsengine 6828 S < 0.0 4.3
ipsengine 6829 S < 0.0 4.3
ipsengine 6827 S < 0.0 4.3
.....
It is also possible to run 'diag sys top | grep <name>' to find out the number of instances spawned:
diag sys top 1 99 1 | grep wad
wad 1988 S 0.0 1.3 0
wad 2021 S 0.0 1.3 0
wad 2016 S 0.0 1.1 0
wad 12010 S 0.0 1.1 0
In the example above, x4 instances are running for wad.
By using the commands below, it is possible to decrease the number of those instances spawned by the parent processes:
config system global
set wad-worker-count 2
end
config ips global
set engine-count 2
end
Now using the same command, to confirm:
diagnose sys top-summary
CPU [| ] 4.8%
Mem [||||||||||||||| ] 38.0% 722M/1866M
Processes: 20 (running=1 sleeping=127)
PID RSS CPU% ^MEM% FDS TIME+ NAME
* 284 102M 0.0 5.5 236 00:13.71 ipsmonitor [x4]
323 38M 0.0 2.0 132 00:11.17 wad [x7]
Verify that both have decreased their spawned child processes. There will be 2 WAD workers and 2 IPS engine instances.
Some instances cannot be disabled, like IPS monitor and IPS helper or WAD manager. That is why the x4 IPS monitor and x7 WAD are still visible.
However, notice the memory decreased to 7.5% for both processes. Almost 5% additional memory was released.
More specifically the number of workers can be listed for different processes, for example for IPS, WAD, and miglogd as follows:
IPS:
diagnose test application ipsmonitor 1
pid = 1943, engine count = 4 (+1) <- There is one master process (ipsmonitor) and 4 workers (ipsengine) running.
0 - pid:1996:1996 cfg:1 master:0 run:1
1 - pid:7038:7038 cfg:0 master:1 run:1 <- master process master:1.
2 - pid:7032:7032 cfg:0 master:0 run:1
3 - pid:7033:7033 cfg:0 master:0 run:1
4 - pid:7034:7034 cfg:0 master:0 run:1
WAD:
diagnose debug reset
diagnose debug enable
diagnose test application wad 1000
Process [0]: WAD manager type=manager(0) pid=1145 diagnosis=yes.
Process [1]: type=dispatcher(1) index=0 pid=1179 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled
Process [2]: type=worker(2) index=0 pid=1180 state=running <- worker process 1.
diagnosis=no debug=enable valgrind=supported/disabled
Process [3]: type=worker(2) index=1 pid=1185 state=running <- worker process 2.
diagnosis=no debug=enable valgrind=supported/disabled
Process [4]: type=algo(3) index=0 pid=1176 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled
Process [5]: type=informer(4) index=0 pid=1175 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled
Process [6]: type=user info(5) index=0 pid=1177 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [7]: type=debug(8) index=0 pid=1174 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled
miglogd:
diagnose test application miglogd 15
Main miglogd: ID=0, children=4, active-children=4 <- miglogd has 4 workers running.
ID=1, duration=10286129.
ID=2, duration=6.
ID=3, duration=6.
ID=4, duration=6.
The number of different spawned daemons can be reduced. The numbers used in the example below are recommended on low-end devices like 60E. These can be experimented with and adjusted as required as based on the system load.
Especially the number of sessions processed by wad, ips, sslvpnd, scan unit, etc should be checked before reducing the worker count.
If a process afterward runs at a high CPU load of i.e. 99% then the number of workers for this process should be increased again.
config system global
set miglogd-children 1
set sslvpn-max-worker-count 2
set wad-worker-count 2
set scanunit-count 2
end
config ips global
set socket-size [integer, 0-512] <----- IPS socket buffer size. The max and default values depend on available memory. Lower value reduces memory usage.
set engine-count [integer, 0-255] <- Number of IPS engines running. The default value of 0, FortiOS sets the number to optimize performance depending on the number of CPU cores. Reduce it to the number of cores the FortiGate box has.
set database [regular|extended] <- Regular protects against the latest common and in-the-wild attacks. Extended includes protection from legacy attacks.
end
After changing the engine, database, and socket size, restart the IPSEngine using the following command:
diag test app ipsmonitor 99
To reduce Spawn Processes in an HA-cluster, this process must be performed on both the Master and Slave Units because it is a hardware-specific configuration and will not sync automatically.
For example, to reduce the WAD worker count on HA-cluster, configure the following commands on both HA-cluster units:
config system global
set wad-worker-count 2
end
Note:
Do not decrease the number of daemons in cases with a high CPU load, as this may stress the CPU cores on which the daemons are running even more and create additional problems. Decrease the number of daemons only in cases of high memory usage for FortiGates with 4GB or less memory.
