|
The WAD process suffers a memory leak on v7.0.8 and v7.0.9 in WAD processes with the 'user-info' type.
To confirm if the device is suffering from this issue, run the following diagnostic commands to show the total memory usage of the device:
get sys stat
get sys perf stat
get hardware memory
diagnose sys top-mem 99
diagnose debug reset
diagnose debug enable
diagnose test app wad 1000
diagnose test app wad 2500
diagnose test app wad 803
diagnose test app wad 2
diagnose test app wad 3
diagnose debug disable
diagnose debug report
This will allow confirmation of the firmware version, as well as the current total memory usage and the kernel memory allocation.
Example outputs:
get sys stat
Version: FortiGate-200E v7.0.8,build0418,221012 (GA.F)
get sys perf stat
Memory: 4057460k total, 3063772k used (75.5%), 620072k free (15.3%), 373616k freeable (9.2%)
get hardware memory
MemTotal: 4057460 kB
MemFree: 625160 kB
Cached: 962744 kB
Active: 2525552 kB <----- 2466,36 MB.
Inactive: 270932 kB
Shmem: 502748 kB
Slab: 184460 kB
Most memory is allocated under 'Active' memory. The following command can show the user space processes using the active memory:
diagnose sys top-mem 99
wad (236:( 1165711kB <--- 1138,38 MB.
Here, a single WAD process uses approximately 1140 MB out of the total 3962 MB.
The process ID (PID) of this process is 236.
To determine which type this WAD process has, run the following:
diagnose debug reset
diagnose debug enable
diagnose test app wad 1000
To show the current focused process running:
diagnose test application wad 1 <----- This will display which process is in use (manager or any worker process).
WAD manager process status: pid=5673 n_workers=63 n_debug_workers=0
fd_conserve_mode=disabled use_adv_mem=enabled
If the process type needs to be changed to the'user-info' process:
Process [6]: type=user-info(5) index=0 pid=236 state=running
The context to the user-info process and show the memory usage type.
diagnose test application wad 2yxx <------ Replace y with process type number, and replace xx with two-digit representation of the index (e.g., 00 for 0, 01 for 1, etc.).
The command 'diagnose test app wad 2500' will switch to the user-info process context.
The values 803, 2, and 3 will dump the current memory stats for this user-info process.
diagnose test app wad 2500
diagnose test app wad 803
diagnose test app wad 2
diagnose test app wad 3
The issue can be identified if the memory usage in 'mmaped regions' from the 'diagnose test app wad 2' command is much higher than the memory in object 'wad_m_cmem_root' indicated by the 'diagnose test app wad 803' command.
For example:
diagnose test application wad 2
process malloc info:
space in mmapped regions: 1406087168
diagnose test application wad 803
cmem object stats:
id allocs frees reallocs avg_sz in_str active bytes max cmem object name
0 842340758 842310950 0 96 0 29808 2865986 3014515 wad_m_cmem_root
Workaround:
As a workaround, restart the WAD processes with the following command:
diagnose test app wad 99
This can be automated with the 'config system auto-script' feature.
Solution:
The solution is to upgrade to v7.0.10, v7.2.4, or above.
|
