- Log in to Fortiweb SSH using the default 'admin' account, run the following command and then hit 's' on the keyboard.
diag sys top
- Find the daemon consuming more memory resources. For example, in the 'diag sys top' output taken from the lab unit, the daemon occupying the more memory resource is 'proxyd'.
diag sys top
slab:56340 buf:42760 cache:144004 dirty:96 write:0
Swap total:0 free:0
PID^^^VSZ^VSZRW RSS (SHR) DIRTY (SHR) STACK COMMAND
1159 994m 887m 1589m 20552 156m 1764 1100 /bin/proxyd <----- Check the RSS column.
If it is confirmed that ‘proxyd’ or ‘ml_daemon’ is consuming the most memory, collect the ‘jemalloc’ heap files of the corresponding process.
If ‘proxyd’ or ‘ml_daemon’ is not the top memory-consuming daemon, skip Step 3 and collect the output of the commands mentioned in Step 4.
- Follow these steps to collect the ‘proxyd’ heap files. For ‘ml_daemon’, follow the same steps, replacing ‘proxyd’ with ‘ml_daemon’ in the commands.
- Clear the old jemalloc heap files. To keep a copy, back them up before clearing them.
diagnose debug jemalloc-heap clear
- Collecting Jeprof Files: Perform the following steps:
- Enable diagnostics:
Fortiweb # diagnose debug jemalloc-conf proxyd enable
prof_active:true <-----
prof_leak:true
lg_prof_sample:17
background_thread:true
dirty_decay_ms:0
muzzy_decay_ms:0
- After a 10% increase in proxyd memory usage (proxyd RSS usage), execute:
Fortiweb# diagnose debug jemalloc proxyd dump
- Following another 10% increase in proxyd memory usage (proxyd RSS usage), rerun the command:
Fortiweb# diagnose debug jemalloc proxyd dump
- Use the show command to see the Jeprof files:
Fortiweb# diagnose debug jemalloc-heap show
Fortiweb# diagnose system jeprof proxy
- Disable Jemalloc Debugging:
Fortiweb# diagnose debug jemalloc-conf proxyd disabl
To download jeprof.out* Files, vavigate to System -> Maintenance -> Backup & Restore -> GUI File Download/Upload to download the files.
Note:
Collecting the heap files does not significantly impact memory or other system resource usage. However, if concerned, it is possible to collect them during a maintenance window or low-traffic hours.
- Log in to FortiWeb using the default 'admin' account and collect the output of the following commands.
Fortiweb# get system status
Fortiweb# get system performance
Fortiweb# diagnose hardware mem list
Fortiweb# fnsysctl free. <----- Works only in v7.0.X and v7.2.X.
Along with the above files, attach the configuration backup and the system debug file. To download the system debug file, go to System -> Maintenance -> Debug -> Debug Log and download the debug log file.
 |
