Troubleshooting Tip: How to identify and fix memory leak issues caused by node scripts on FortiGate firewalls

[vpalli]

Description	This article outlines data collection plan and highlights a known issue reported on FortiOS firmware v7.2.7 and below.
Scope	FortiGate.
Solution	FortiGate system will enter into conserve mode when the memory usage is 88% or above. When the FortiGate is in conserve mode, node process responsible for FortiGate GUI management may not release memory properly causing entry-level devices to stay in conserve mode. This issue is fixed in FortiOS v7.2.8 and later, as well as v7.4.2 and later. Symptoms: Node or httpsd process may be consuming more than normal amount of memory. diagnose sys top 2 99 1 Run Time: 66 days, 19 hours and 26 minutes 1U, 0N, 0S, 99I, 0WA, 0HI, 0SI, 0ST; 1866T, 173F httpsd  28502   S   7.4  1.1    7 httpsd  28516   D   4.9  1.0    6 node    149     S   0.4  21.9   2  <- Freeable memory may be holding a high amount of memory and thus triggering the conserve mode event. During a normal memory consumption period, use the following: get system performance status Memory: 8171732k total, 3042064k used (37.2%), 3541636k free (43.3%), 1588032k freeable (19.4%) During Abnormal memory consumption period: get system performance status Memory: 8171732k total, 3487184k used (42.7%), 534020k free (6.5%), 4150528k freeable (50.8%) <- Errors related to Node or Node Scripts are presented in the output of the following command: diagnose debug crashlog read 1: 2022-08-08 18:47:55 <00417> ==================================================== 3: 2022-08-08 18:47:55 <00417> Error: ENOENT: no such file or directory, open '/tmp/admin_server.crt' 6: 2022-08-08 22:27:34 <01043> ==================================================== 7: 2022-08-08 22:27:34 <01043> ====== Node exiting due to uncaught exception: ====== 8: 2022-08-08 22:27:34 <01043> ==================================================== 10: 2022-08-08 22:27:34 <01043> Error: ENOMEM: not enough memory, write Important note: For collecting data during normal memory consumption period, a restart of the relevant process or a reboot of the FortiGate device may be required, which should be scheduled as part of a maintenance activity. After a daemon restart or a FortiGate reboot, another iteration of the following debug data must be captured for baseline and comparison purposes. Depending on the user process that is restarted, end users may experience traffic outage.  Data Collection Plan:  To report any new issues related to memory consumption by the node process, collect the following debug data during both normal and abnormal memory consumption periods of the daemon before submitting a support request to the Fortinet Technical Team. fnsysctl du -i /dev/cmdb fnsysctl du -a /dev/cmdb fnsysctl df -k fnsysctl ls -l /tmp fnsysctl du -i /tmp fnsysctl du -a /tmp fnsysctl du -a / -d 1 fnsysctl ls -l /dev/shm fnsysctl du -i /dev/shm fnsysctl du -a /dev/shm fnsysctl ls -l /node-scripts fnsysctl du -i /node-scripts fnsysctl du -a /node-scripts get sys perf stat diag sys top 2 99 3 diag sys top-fd diag sys top-mem 20 diag sys top-sockmem diag hardware sysinfo conserve fnsysctl du diag ips session status diag ips packet status diag ips memory status diagnose sys session stat diag sys dump-conserve-info diag sys print-conserve-info fnsysctl df fnsysctl du /node-scripts fnsysctl ls -la /node-scripts fnsysctl ls -la /node-scripts/report-runner/results fnsysctl ls -la /node-scripts/logs fnsysctl cat /proc/meminfo fnsysctl cat /proc/vmstat execute tac report Disconnect any active GUI sessions from the FortiGate and access its CLI via SSH to execute the following commands during problem state. When the httpsd daemon is consuming more memory, run the following debug commands: diagnose debug reset diagnose debug application httpsd -1 diagnose web-ui backtrace enable diagnose web-ui backtrace httpsd <Enter the process ID of the httpsd daemon> diagnose debug console timestamp enable diagnose debug duration 2 diagnose debug enable The debug commands will stop printing data after 2 minutes. When nodejs is consuming more memory, run the following debug commands: diagnose debug reset diagnose debug application nodejs -1 diagnose debug console timestamp enable diagnose debug duration 2 diagnose debug enable The debug commands will stop printing data after 2 minutes. To permanently disable/reset the debugs, execute the following commands. diagnose debug disable diagnose debug reset Capture the process dump and traces and specify the process ID. diagnose sys process pidof httpsd diagnose sys process pidof node diag sys process trace <Enter PID of node and httpsd one at a time> diag sys process dump <Enter PID of node and httpsd one at a time> diag sys process pstack <Enter PID of node and httpsd one at a time> diag sys process sock-mem <Enter PID of node and httpsd one at a time>