Description | This article describes how to detect and resolve a WAD memory leak that occurs with 'user-info' type processes. |
Scope | FortiOS 7.0.8, 7.0.9. |
Solution |
The WAD process suffers a memory leak on FortiOS 7.0.8 and 7.0.9 in WAD processes with the 'user-info' type.
To confirm if the device is suffering from this issue, run the following diagnostic commands to show the total memory usage of the device:
# get sys stat
This will allow confirmation of the firmware version, as well as the current total memory usage and the kernel memory allocation.
Example outputs:
# get sys stat
Most memory is allocated under 'Active' memory. The following command can show the user space processes using the active memory:
# diagnose sys top-mem 99
Here, a single WAD process uses approximately 1140 MB out of the total 3962 MB. The process ID (PID) of this process is 236.
To determine which type this WAD process has, run the following:
# diagnose debug reset
If the process type is 'user-info' as shown below...
Process [6]: type=user-info(5) index=0 pid=236 state=running
... then change of the context to the user-info process and show the memory usage type.
The command 'diagnose test app wad 2500' will switch to the user-info process context.
The values 803, 2 and 3 will dump the current memory stats for this user-info process.
# diagnose test app wad 2500
The issue can be identified if the memory usage in 'mmaped regions' from the 'diagnose test app wad 2' command is much higher than the memory in object 'wad_m_cmem_root' indicated by the 'diagnose test app wad 803' command.
For example:
# diagnose test application wad 2
Workaround:
As a workaround, restart the WAD processes with the following command: # diagnose test app wad 99
This can be automated with the 'config system auto-script' feature.
Solution:
The solution is to upgrade to FortiOS versions 7.0.10, 7.2.4, or above. |