Technical Tip: Optimizing memory usage by limiting spawned daemons

Description

This article describes how to optimize memory usage in FortiOS by limiting certain processes, such as the IPS engine, WAD, and SSL VPN, which create a child process for each CPU core and consume memory resources.

Each spawned child process will have some memory allocated to it, regardless of the traffic load. For some units with multi-core CPUs and less amount of memory, mainly 'E' or 'F' series such as 30E, 60E, 90E, 100/101E, 100/101F (Gen1), this can create an issue with memory usage.

Scope

FortiGate.

Solution

To address memory usage issues, it is recommended to reduce the number of instances spawned by the aforementioned processes when additional daemons are not required for the current network throughput. As a general guideline, the number of worker processes should be decreased on low-end devices, such as models 30, 40, 60, and 80, as follows:

Note: The following configuration should be implemented during a maintenance window and carefully monitored during production hours for any performance impact.

Each daemon is bound to a single CPU core. Since the number of daemons/processes is reduced, depending on the volume of traffic or number of requests handled by a single daemon, this can lead to high CPU usage on that core, potentially resulting in dropped connections. It is recommended to monitor network usage, CPU load, and memory consumption, and adjust the configuration accordingly.

For example, if the device is handling around 100-200 Mbps of traffic (as shown in the output of 'get system performance status' under average network usage), allocating more than one WAD worker or IPS engine may be necessary. This should be done while ensuring efficient memory utilization so that the device does not enter conserve mode.

config system global
    set miglogd-children 1
    set sslvpn-max-worker-count 1
    set wad-worker-count 1
    set scanunit-count 2
end

Note: 'set sslvpn-max-worker-count' , 'set wad-worker-count' , 'set scanunit-count' Commands do not work on FortiOS 7.6 and above as the features are deprecated on low-end devices. Refer to the following release notes:

• 2 GB RAM FortiGate models no longer support FortiOS proxy-related features

• SSL VPN removed from 2GB RAM models for tunnel and web mode

config ips global
    set engine-count 1
end

For more details and different use cases, refer to the steps below.

Confirm the number of instances spawned:

To list the number of running processes, use the 'diagnose sys top' command. This will only list the most busy processes:

diagnose sys top 1 99 1

Alternatively, the command 'fnsysctl ps' can be used to list all processes running on the FortiGate. Here, the count of workers has to be manually added.

fnsysctl ps

Or the command 'diagnose sys process pidof' can be used on current firmware releases to list all process IDs of a given process name:

diagnose sys process pidof wad

On older firmware releases, the 'diagnose sys top-summary' provides a summarized list of the busiest processes. For firmware above v6.4, refer to the command 'diagnose sys top':

diagnose sys top-summary
  CPU [|                                       ]   4.8% 
  Mem [|||||||||||||||||                       ]  44.0%   836M/1866M 
  Processes: 20 (running=1 sleeping=130) 
   
  PID      RSS   CPU% ^MEM%   FDS     TIME+  NAME 
* 144     163M    0.0  8.7   390  00:18.49  ipsmonitor [x6]
  217      66M    0.0  3.6   197  00:20.76  wad [x9]

In the example above, there are x6 instances of the IPS engine and x9 of WAD, which all consume 8.7+3.6 = 12.3% of the memory, while this unit processes almost no traffic at all.

Note that the command 'diagnose sys top-summary' is not supported in v6.4 or later; however, 'diagnose sys top <delay> <number of processes>' will return the list of processes/workers. Be aware that the 'diagnose sys top' command is limited to a maximum of 99 entries. To list all processes, the 'fnsysctl ps' command can be used.

More info can be found in Technical Tip: Deprecated of command 'diagnose sys top-summary'.

An example of the top command:

diagnose sys top 1 99 1

Run Time: 33 days, 4 hours and 16 minutes
0U, 0N, 0S, 100I, 0WA, 0HI, 0SI, 0ST; 1863T, 463F
cw_wtpd    196       R       0.4    0.5
newcli        6591     R       0.4    0.3
ipsengine  6826    S <    0.0    4.5
ipsengine  6828    S <    0.0    4.3
ipsengine  6829    S <    0.0    4.3
ipsengine  6827    S <    0.0    4.3

.....

It is also possible to run 'diagnose sys top | grep <name>' to find out the number of instances spawned:

diagnose sys top 1 99 1 | grep wad
 wad 1988 S 0.0 1.3 0
wad 2021 S 0.0 1.3 0
wad 2016 S 0.0 1.1 0
wad 12010 S 0.0 1.1 0

 
In the example above, x4 instances are running for WAD.

By using the commands below, it is possible to decrease the number of those instances spawned by the parent processes:

config system global
    set wad-worker-count 2
end

config ips global
    set engine-count 2
end

Now using the same command, to confirm:

diagnose sys top-summary
  CPU [|                                       ]   4.8% 
  Mem [|||||||||||||||                         ]  38.0%   722M/1866M 
  Processes: 20 (running=1 sleeping=127) 
   
  PID      RSS   CPU% ^MEM%   FDS     TIME+  NAME 
* 284     102M    0.0  5.5   236  00:13.71  ipsmonitor [x4] 
  323      38M    0.0  2.0   132  00:11.17  wad [x7]

Verify that both have decreased their spawned child processes. There will be 2 WAD workers and 2 IPS engine instances.

Some instances cannot be disabled, like the IPS monitor, and IPS helper, or the WAD manager. That is why the x4 IPS monitor and x7 WAD are still visible.

However, notice the memory decreased to 7.5% for both processes. Almost 5% additional memory was released.

More specifically, the number of workers can be listed for different processes, for example, for IPS, WAD, and miglogd as follows:

IPS:

diagnose test application ipsmonitor 1
pid = 1943, engine count = 4 (+1) <- There is one master process (ipsmonitor) and 4 workers (ipsengine) running.
0 - pid:1996:1996 cfg:1 master:0 run:1
1 - pid:7038:7038 cfg:0 master:1 run:1 <- Master process master:1.
2 - pid:7032:7032 cfg:0 master:0 run:1
3 - pid:7033:7033 cfg:0 master:0 run:1
4 - pid:7034:7034 cfg:0 master:0 run:1

WAD:

diagnose debug reset
diagnose debug enable
diagnose test application wad 1000
Process [0]: WAD manager type=manager(0) pid=1145 diagnosis=yes.
Process [1]: type=dispatcher(1) index=0 pid=1179 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled
Process [2]: type=worker(2) index=0 pid=1180 state=running <- worker process 1.
diagnosis=no debug=enable valgrind=supported/disabled
Process [3]: type=worker(2) index=1 pid=1185 state=running <- worker process 2.
diagnosis=no debug=enable valgrind=supported/disabled
Process [4]: type=algo(3) index=0 pid=1176 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled
Process [5]: type=informer(4) index=0 pid=1175 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled
Process [6]: type=user info(5) index=0 pid=1177 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [7]: type=debug(8) index=0 pid=1174 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled

miglogd:

diagnose test application miglogd 15
Main miglogd: ID=0, children=4, active-children=4 <- miglogd has 4 workers running.
ID=1, duration=10286129.
ID=2, duration=6.
ID=3, duration=6.
ID=4, duration=6.

The number of different spawned daemons can be reduced. The numbers used in the example below are recommended for low-end devices. These can be experimented with and adjusted as required as based on the system load. Especially, the number of sessions processed by wad, ips, sslvpnd, scan unit, etc, should be checked before reducing the worker count.

If a process afterward runs at a high CPU load of i.e., 99% then the number of workers for this process should be increased again.

config system global
    set miglogd-children 1
    set sslvpn-max-worker-count 2
    set wad-worker-count 2
    set scanunit-count 2
end

config ips global
    set socket-size [integer, 0-512] <----- IPS socket buffer size. The max and default values depend on available memory. Lower value reduces memory usage. 
    set engine-count [integer, 0-255] <- Number of IPS engines running. The default value of 0, FortiOS sets the number to optimize performance depending on the number of CPU cores. Reduce it to the number of cores the FortiGate box has. 
    set database [regular|extended] <- Regular protects against the latest common and in-the-wild attacks. Extended includes protection from legacy attacks.
end

After changing the engine count, database, and socket size, restart the IPSEngine using the following command:

diagnose test app ipsmonitor 99

To reduce Spawn Processes in an HA-cluster, this process must be performed on both the Master and Slave Units because it is a hardware-specific configuration and will not sync automatically.

For example, to reduce the WAD worker count on the HA-cluster, configure the following commands on both HA-cluster units:

config system global
    set wad-worker-count 2
end

Related article:

Troubleshooting Tip: initial commands to collect for memory debugging