|
When memory usage is very high and increases very fast in a short period, it might be a memory leak issue, and it can be analyzed by the following steps.
Note that memory increase does not always mean a memory leak. A memory leak issue usually has these phenomena:
-
Very fast and abnormal memory increase (usually with common or low traffic levels). In a scenario where a leak happens in a matter of minutes or even seconds, manual log collection might be infeasible. An automated monitoring script such as TeraTerm script is highly advised in these cases.
-
Continuous memory increases without being de-allocated.
-
Used memory is not de-allocated even after traffic drops or stops.
The most important thing for troubleshooting a memory leak issue is to locate which module, process, or function causes the memory increase.
- Collect below log output below to identify the root cause.
get system status
diagnose hardware sysinfo memory
diagnose hardware sysinfo cpu
diagnose hardware deviceinfo disk
diagnose hardware sysinfo memory
diagnose system session stat
get hardware status
get system performance status (Run this command 5 times in intervals of 1 minute.)
diagnose system top 1 20 <--- Press CTRL+C, let it run for about 30 seconds, then press CTRL+Q to stop.
diagnose system top-mem
diagnose system top-s '-s mem' (Run for 30 seconds and press CTRL+C to stop.)
diagnose autoupdate versions
diagnose debug crashlog read | grep YYYY-MM-DD <----YYYY for year, and MM for month.
- Run 'diagnose system top 1 20' to check the top-utilized resources, and based on the top-utilizing resources, collect further logs for them. Press Shift+M to sort the output of the above command by memory.
- Run 'diagnose hardware sysinfo memory' to gain an idea of the allocated memory of current usage of firewall resources.
Example:
diagnose hardware sysinfo memory
MemTotal: 1963860 kB
Cached: 448948 kB ---> Cached mem is 440 MB.
Active: 932748 kB ---> Active mem is above 900 MB. Almost half of the memory.
- Check the 'diagnose system top-mem', calculate the memory utilization of the top utilized resource, and verify the crash logs.
The well-known processes for memory leaks are WAD, IPS, fgtlogd, and others. In case of memory leak issues, always refer to the release notes of the known issues of the firmware.
Refer to the article below to collect logs in case of WAD, IPS, and other processes consuming the memory. Based on the logs, it can be identified if there is a leak or if any network condition is triggering high memory usage.
WAD
Refer to this article if IPS memory optimization is needed: Technical Tip: IPS memory optimization steps.
-
Another common scenario is slab leaks. Slabs are kernel objects for dynamically allocating memory. In a slab leak, these slab objects are not getting deleted, and the device eventually runs out of memory.
Example:
diagnose hardware sysinfo memory
MemTotal: 2042016 kB
MemFree: 94896 kB
Cached: 421136 kB
Active: 526904 kB
Slab: 960144 kB ---> Almost half of the memory.
In these kinds of cases, it is impossible to release this memory with some kind of command. Only a reboot will release the memory.
It is possible to see which slab objects are using up the memory with the command diagnose hardware sysinfo slab. It will show the number of objects and their size, which can be used to calculate the total size of a specific slab type.
Note:
If 'diagnose system top-summary' is deprecated, instead refer to Technical Tip: Deprecated of command 'diagnose sys top-summary.
Related article:
Troubleshooting Tip: Memory debugs needed when creating a bug ticket
|
