Technical Tip: Memory leak issue

[pkumari]

Description	This article describes how to troubleshoot the memory leak issue.
Scope	FortiGate.
Solution	When memory usage is very high and increases very fast in a short period, it might be a memory leak issue, and it can be analyzed by the following steps. Note that memory increase does not always mean a memory leak. A memory leak issue usually has these phenomena: Very fast and abnormal memory increase (usually with common or low traffic levels). Continuous memory increases without de-allocated. Used memory is not de-allocated even after traffic drops or stopped. The most important thing for troubleshooting a memory leak issue is to locate which module, process, or function causes the memory increase. Collect below log output below to identify the root cause. get system status diag hardware sysinfo memory diag hardware sysinfo cpu diagnose hardware deviceinfo disk diagnose hardware sysinfo memory diagnose sys session stat get hardware status get system performance status (Run this command 5 times in intervals of 1 minute.) diag sys top 1 20 <--- Press CTRL+C, let it run for about 30 seconds, then press CTRL+Q to stop. diag sys top-mem diag sys top-s '-s mem' (Run for 30 seconds and press CTRL+C to stop.) diagnose autoupdate versions diag debug crashlog read | grep 2024 Run 'diag sys top 1 20' to check the top utilized resources and based on the top utilizing resources, collect further logs for them. Press Shift+M to sort the output of the above command by memory.   Run 'diag hardware sysinfo memory' to gain an idea of the allocated memory of current usage firewall resources. Example: diag har sysinfo memory MemTotal: 1963860 kB Cached: 448948 kB ---> Cached mem is 440 MB. Active: 932748 kB ---> Active mem is above 900 MB. Almost half of the memory. Check the 'diag sys top-mem', calculate the memory utilization of the top utilized resource, and verify the crash logs. The well-known processes for memory leaks are WAD, IPS, fgtlogd, and others. In case of memory leak issues, always refer to the release notes of the known issues of the firmware. Refer to the below article to collect logs in case of WAD, IPS, and other processes consuming the memory. Based on the logs it can be identified If there is a leak or any network condition is triggering high memory usage. WAD Refer to this article if IPS memory optimization is needed: Technical Tip: IPS memory optimization steps. Another common scenario is slab leaks. Slabs are kernel objects to dynamically allocate memory. In a slab leak, these slab objects are not getting deleted and the device eventually runs out of memory. Example:    diag har sysinfo memory    MemTotal:        2042016 kB    MemFree:           94896 kB    Cached:           421136 kB    Active:           526904 kB    Slab:             960144 kB   --->  Almost half of the memory. In these kinds of cases, it is impossible to release this memory with some kind of command. Only a reboot will release the memory. It is possible to see which slab objects are using up the memory with the command diagnose hardware sysinfo slab. It will show the number of objects and their size which can be used to calculate the total size of a specific slab type. Note: If 'diagnose sys top-summary' is deprecated, instead refer to Technical Tip: Deprecated of command 'diagnose sys top-summary.