FortiGate
FortiGate Next Generation Firewall utilizes purpose-built security processors and threat intelligence security services from FortiGuard labs to deliver top-rated protection and high performance, including encrypted traffic.
ppatel
Staff
Staff
Article Id 194322

Description


This article describes how to reduce memory usage by reducing some processes in FortiOS such as the IPS engine, WAD and SSL VPN which spawn a child process for each CPU core.

Each of the spawned child processes will have some memory allocated to it regardless of the traffic load.

For some units with multi-core CPUs and less amount of memory, mainly 'E' series such as 30E, 60E and 90E this can create an issue with memory usage.

 

Scope

 

FortiGate.

Solution


To solve memory usage issues, it is recommended to decrease the number of instances spawned by the aforementioned processes.

Confirm the number of instances spawned:

 

diagnose sys top-summary  <- For firmware above 6.4 refer to the command 'diag sys top' mentioned below in this article.
    CPU [|                                       ]   4.8%
  Mem [|||||||||||||||||                       ]  44.0%   836M/1866M
  Processes: 20 (running=1 sleeping=130)
   
  PID      RSS   CPU% ^MEM%   FDS     TIME+  NAME
* 144     163M    0.0  8.7   390  00:18.49  ipsmonitor [x6]
  217      66M    0.0  3.6   197  00:20.76  wad [x9]

 

In the example above, there are x6 instances of the IPS engine and x9 of WAD, which all consume 8.7+3.6 = 12.3% of the memory while this unit processes almost no traffic at all.

Note that the command 'diagnose sys top-summary' is not supported in FortiOS 6.4 or later, however, 'diagnose sys top <delay> <number of processes>' will return the list of processes/workers. Be aware that the 'diagnose sys top' command is limited to a maximum of 99 entries.

 

More info can be found in Technical Tip: Deprecated of command '# diagnose sys top-summary'.

 

diagnose sys top 5 99

Run Time: 33 days, 4 hours and 16 minutes
0U, 0N, 0S, 100I, 0WA, 0HI, 0SI, 0ST; 1863T, 463F
cw_wtpd    196       R       0.4    0.5
newcli        6591     R       0.4    0.3
ipsengine  6826    S <    0.0    4.5
ipsengine  6828    S <    0.0    4.3
ipsengine  6829    S <    0.0    4.3
ipsengine  6827    S <    0.0    4.3

.....


It is also possible to run 'diag sys top | grep <name>' to find out the number of instances spawned:


diag sys top 5 99 | grep wad
wad 1988 S 0.0 1.3 0
wad 2021 S 0.0 1.3 0
wad 2016 S 0.0 1.1 0
wad 12010 S 0.0 1.1 0

 

In the example above, there are x4 instances running for wad.

By using the commands below, it is possible to decrease the number of those instances spawned by the parent processes:

 

config system global
    set wad-worker-count 2
end


config ips global
    set engine-count 2
end

 

Now using the same command, to confirm:

 

diagnose sys top-summary
  CPU [|                                       ]   4.8%
  Mem [|||||||||||||||                         ]  38.0%   722M/1866M
  Processes: 20 (running=1 sleeping=127)
   
  PID      RSS   CPU% ^MEM%   FDS     TIME+  NAME
* 284     102M    0.0  5.5   236  00:13.71  ipsmonitor [x4]
  323      38M    0.0  2.0   132  00:11.17  wad [x7]

 

Verify that both have decreased their spawned child processes.
There will be 2 WAD workers and 2 IPS engine instances.

 

There are some instances that cannot be disabled, like IPS monitor and IPS helper or WAD manager.
That is why the x4 IPS monitor and x7 WAD are still visible.

However, notice the memory decreased to 7.5% for both processes.
Almost 5% additional memory was released.

 

An alternative approach to list the number of running workers would be to use the 'fnsysctl ps' command to list all processes running on the FortiGate.

 

fnsysctl ps

 

There are further methods to view the number of running processes for a given name:

For example, the number of PIDs (process IDs) can be listed with the command 'diag sys process pidof <process name>' for each process name.

 

diagnose sys process pidof miglogd
diagnose sys process pidof sslvpnd
diagnose sys process pidof scanunitd

 

More specifically the number of workers can be listed for IPSWAD and miglogd as follows:

IPS:

 

diagnose test application ipsmonitor 1
pid = 1943, engine count = 4 (+1) <- There is one master process (ipsmonitor) and 4 workers (ipsengine) running.
0 - pid:1996:1996 cfg:1 master:0 run:1
1 - pid:7038:7038 cfg:0 master:1 run:1 <- master process master:1.
2 - pid:7032:7032 cfg:0 master:0 run:1
3 - pid:7033:7033 cfg:0 master:0 run:1
4 - pid:7034:7034 cfg:0 master:0 run:1

 

WAD:

 

diagnose debug reset
diagnose debug enable
diagnose test application wad 1000
Process [0]: WAD manager type=manager(0) pid=1145 diagnosis=yes.
Process [1]: type=dispatcher(1) index=0 pid=1179 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled
Process [2]: type=worker(2) index=0 pid=1180 state=running <- worker process 1.
diagnosis=no debug=enable valgrind=supported/disabled
Process [3]: type=worker(2) index=1 pid=1185 state=running <- worker process 2.
diagnosis=no debug=enable valgrind=supported/disabled
Process [4]: type=algo(3) index=0 pid=1176 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled
Process [5]: type=informer(4) index=0 pid=1175 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled
Process [6]: type=user info(5) index=0 pid=1177 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [7]: type=debug(8) index=0 pid=1174 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled

miglogd:

 

diagnose test application miglogd 15
Main miglogd: ID=0, children=4, active-children=4
<- miglogd has 4 workers running.

ID=1, duration=10286129.
ID=2, duration=6.
ID=3, duration=6.
ID=4, duration=6.


The number of different spawned daemons can be reduced.

The numbers used in the example below are recommended on low-end devices like 60E.

These can be experimented with and adjusted as required.

 

config system global
    set miglogd-children 1
    set sslvpn-max-worker-count 2
    set wad-worker-count 2
    set scanunit-count 2
end

config ips global

set socket-size [integer, 0-512] <- IPS socket buffer size. Max and default value depend on available memory. Lower value reduces memory usage.
set engine-count [integer, 0-255]
<- Number of IPS engines running. The default value of 0, FortiOS sets the number to optimize performance depending on the number of CPU cores. Reduce it to the number of cores the FortiGate box has.
set database [regular|extended]
<- Regular protects against the latest common and in-the-wild attacks. Extended includes protection from legacy attacks.

end

 

After changing the engine, database, and socket size, restart the IPSEngine using the following command:

 

diag test app ipsmonitor 99

diag test app ipsengine 99

 

To reduce Spawn Processes in an HA-cluster, this process must be performed on both the Master and Slave Units because it is a hardware-specific configuration and will not sync automatically.

For example, to reduce the WAD worker count on HA-cluster, configure the following commands on both HA-cluster units:


config system global

set wad-worker-count 2

end