Technical Tip: How to collect the logs needed for investigating high memory-related problem

1. Log in to Fortiweb SSH using the default 'admin' account, run the following command, and then hit 's' on the keyboard.

diagnose sys top

2. Find the daemon consuming more memory resources. For example, in the 'diagnose sys top' output taken from the lab unit, the daemon occupying the more memory resource is 'proxyd'.

diagnose sys top
slab:56340 buf:42760 cache:144004 dirty:96 write:0
Swap total:0 free:0
PID^^^VSZ^VSZRW RSS (SHR) DIRTY (SHR) STACK COMMAND
1159 994m 887m 1589m 20552 156m 1764 1100 /bin/proxyd <----- Check the RSS column.

If it is confirmed that ‘proxyd’ or ‘ml_daemon’ is consuming the most memory, collect the ‘jemalloc’ heap files of the corresponding process.

If ‘proxyd’ or ‘ml_daemon’ is not the top memory-consuming daemon, skip Step 3 and collect the output of the commands mentioned in Step 4.

3. Follow these steps to collect the ‘proxyd’ heap files. For ‘ml_daemon’, follow the same steps, replacing ‘proxyd’ with ‘ml_daemon’ in the commands.

• Clear the old jemalloc heap files. To keep a copy, back them up before clearing them.

diagnose debug jemalloc-heap clear

• Collecting Jeprof Files: Perform the following steps:
• Enable diagnostics:

Fortiweb # diagnose debug jemalloc-conf proxyd enable
prof_active:true <-----
prof_leak:true
lg_prof_sample:17
background_thread:true
dirty_decay_ms:0
muzzy_decay_ms:0​

• After a 10% increase in proxyd memory usage (proxyd RSS usage), execute:

Fortiweb# diagnose debug jemalloc proxyd dump

• Following another 10% increase in proxyd memory usage (proxyd RSS usage), rerun the command:

Fortiweb# diagnose debug jemalloc proxyd dump

• Use the show command to see the Jeprof files:

Fortiweb# diagnose debug jemalloc-heap show

• To parse the heap file:

Fortiweb# diagnose system jeprof proxy

• Disable Jemalloc Debugging:

Fortiweb# diagnose debug jemalloc-conf proxyd disable

To download jeprof.out* Files, vavigate to System -> Maintenance -> Backup & Restore -> GUI File Download/Upload to download the files.

Note:

Collecting the heap files does not significantly impact memory or other system resource usage. However, if concerned, it is possible to collect them during a maintenance window or low-traffic hours.

4. Log in to FortiWeb using the default 'admin' account and collect the output of the following commands.

Fortiweb# get system status
Fortiweb# get system performance
Fortiweb# diagnose hardware mem list

Fortiweb# diagnose debug memory
Fortiweb# fnsysctl free   <----- Works only in v7.0.x and v7.2.x.

Along with the above files, attach the configuration backup and the system debug file. To download the system debug file, go to System -> Maintenance -> Debug -> Debug Log and download the debug log file.