|
The WAD cert-manager process is one of many WAD processes running on the FortiGate. The following article describes the high memory usage of the wad cert-manager process. For a general introduction to the WAD process and its processes, refer to the article Technical Tip: Overview of WAD process structure.
High memory in the WAD cert-manager does not necessarily mean that it is a memory leak. The memory for the certificate manager is cached for days, so high memory usage can be observed.
As for all memory issues, first identify the category in which most memory is allocated. In this example, it is the WAD process consuming high memory, so active memory will be high.
FGT # get sys performance status
CPU states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
Memory: 7666104k total, 6413524k used (83.7%), 781220k free (10.2%), 471360k freeable (6.1%)
Average network usage: 4081 / 2359 kbps in 1 minute, 3032 / 1268 kbps in 10 minutes, 3509 / 1691 kbps in 30 minutes
Average sessions: 24699 sessions in 1 minute, 23633 sessions in 10 minutes, 24241 sessions in 30 minutes
Average session setup rate: 615 sessions per second in last 1 minute, 613 sessions per second in last 10 minutes, 616 sessions per second in last 30 minutes
Average NPU sessions: 355 sessions in last 1 minute, 553 sessions in last 10 minutes, 566 sessions in last 30 minutes
Average nTurbo sessions: 355 sessions in last 1 minute, 553 sessions in last 10 minutes, 566 sessions in last 30 minutes
get hardware memory
MemTotal: 7666104 kB
MemFree: 780072 kB
Cached: 1072156 kB
Active: 5240404 kB
Active(anon): 5094768 kB
Shmem: 492596 kB
Slab: 383408 kB
FGT # diagnose sys top-mem 99
wad (29105): 272423kB
node (1981): 103869kB
ipsengine (2396): 93961kB
ipsengine (2398): 93429kB
To identify which specific WAD process is consuming high memory, note the process ID (PID) from the diagnose sys top-mem command and look for it in the diagnose test application wad 1000 output. In this example, the process ID is 29105 and the type is cert-manager.
FGT # diagnose test application wad 1000
Process [15]: type=cert-manager(10) index=0 pid=29105 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Additional information about the cert-manager process will not be included in the diagnose wad memory report output. Instead, this process will need to be selected manually to gather information. Refer to this article: Technical Tip: Overview of WAD process structure for the manual selection process.
In diagnose wad stats output the following buffers can be seen with higher usage.
diagnose wad stats
worker.ssl.fts.str.cert_factory_bytes 830633903 <<< 830MB
worker.ssl.fts.str.cert_factory_zero_bytes 203064925 <<< 203MB
As a workaround, reduce the certificate manager cache timeout under:
config firewall ssl setting
set cert-manager-cache-timeout
The default timeout is 72 hours. In this instance, it is best to reduce it to the minimum, which is 24 hours. To get an indication of whether it is a memory leak or just normal high usage, confirm whether the WAD certificate manager process's memory usage drops after this amount of time has passed.
If it is not dropping, it indicates a memory leak. In such a scenario, open a ticket with the TAC.
For log collection, use the commands below. The process type of the WAD cert-manager might not be the same as in the example above, and the command diagnose test application wad 21000 needs to be changed accordingly.
get sys status
get sys performance status
diagnose sys vd stats
diagnose sys vd list
diagnose hardware sysinfo memory
diagnose sys top-mem 99
diagnose sys top-fd 50
diagnose sys top 1 99 5
diagnose sys mpstat 2 5
fnsysctl ps aux
fnsysctl ifconfig
diagnose ips memory status
diagnose debug reset
diagnose debug enable
diagnose test application wad 1000
diagnose wad stats summary
diagnose wad memory all
diagnose wad memory report
diagnose test application wad 21000
diagnose test application wad
diagnose test application wad 1
diagnose test application wad 2
diagnose test application wad 3
diagnose test application wad 4
diagnose test application wad 801
diagnose test application wad 803
diagnose wad memory track
diagnose debug disable
diagnose debug report
|
