Technical Tip: How to collect the logs needed for investigating high memory-related problem

[ddsouza_FTNT]

Description	This article describes steps to collect the logs needed for investigating high memory-related problem.
Scope	For version 6.3 and above.
Solution	1) Login to Fortiweb SSH by using the default 'admin' account and run the following command, and then hit 's' on the keyboard. # diag sys top 2) Find the daemon consuming more memory resources. For example, in the '# diag sys top' output taken from the lab unit, the daemon occupying the more memory resource is 'proxyd'. # diag sys top slab:56340 buf:42760 cache:144004 dirty:96 write:0 Swap total:0 free:0 PID^^^VSZ^VSZRW RSS (SHR) DIRTY (SHR) STACK COMMAND 1159 994m 887m 1589m 20552 156m 1764 1100 /bin/proxyd <----- Check the RSS column. If it is confirmed 'proxyd' or 'ml_daemon' is consuming more memory resources, then it is necessary to collect 'jemalloc' heap files of the corresponding process. If 'proxyd' or 'ml_deamon' not listed as the top memory consuming daemon, then please skip Step 3) and collect the output of the commands mentioned in 4). 3) Follow the steps to collect the 'proxyd' heap files. Please follow the steps mentioned below for the 'ml_deamon' as well (replace 'proxyd' with 'ml_deamon' in the commands). - Execute the following command. Fortiweb# diagnose debug jemalloc-conf proxyd enable - Run the below command and verify prof_active is true.  Fortiweb# # diagnose debug jemalloc-conf proxyd info prof_active:true <----- prof_leak:true lg_prof_sample:17 background_thread:true dirty_decay_ms:0 muzzy_decay_ms:0​ - Monitor the memory usage. When the memory usage increases by 10%, then please run the below command to generate the heap file. Fortiweb# diagnose debug jemalloc proxyd dump - Again, monitor the memory usage. When the memory usage increases by 10% (since the last time executed the above command), execute the same command again.     Fortiweb# diagnose debug jemalloc proxyd dump - Execute the below command to decode the heap file. Save the output to a file. Fortiweb# diagnose system jeprof proxyd - Download the file by navigating to System -> Maintenance -> Backup & Restore and download the jeprof.out files 4) Login to FortiWeb by using the default 'admin' account and collect the output of the following commands. Fortiweb# get system status Fortiweb# get system performance Fortiweb# diagnose hardware mem list Fortiweb# fnsysctl cat /proc/slabinf Fortiweb# fnsysctl free Along with the above files, attach the configuration backup and the system debug file. To download the system debug file, go to System -> Maintenance -> Debug -> Debug Log and Download the debug log file.