Troubleshooting Tip: How to troubleshoot Node.js daemon which is consuming high memory on secondary HA member

[seyuboglu]

Description	This article describes how to troubleshoot a conserve mode issue on the Secondary unit in an HA cluster caused by the node.js daemon.
Scope	FortiGate version 7.6.0.
Solution	This issue was reported after upgrading the HA cluster to v7.6.0. After upgrading to version 7.6.0, Node daemon is using high memory all the time and ending up with conserve mode on secondary device in the cluster. Memory usage increases slightly on the Secondary unit until it triggers the conserve mode.  Connect to the Secondary unit in the cluster. Detailed link about how to connect secondary node in cluster :  Technical Tip: How to access secondary unit of HA cluster via CLI Check the crashlog output to confirm the device entered conserve mode.  Example output:  diagnose debug crashlog read  122: 2024-07-11 08:00:10 service=kernel conserve=on total="3614 MB" used="3180 MB" red="3180 MB" 123: 2024-07-11 08:00:33 green="2963 MB" msg="Kernel enters memory conserve mode"  <--- Conserve mode is triggered.  Check the memory usage when the memory usage is high or when the device enters conserve mode. Example output:  get system status Version: FortiGate-100F v7.6.0,build3401,240724 (GA.F) First GA patch build date: 240724 ...(output truncated) Current HA mode: a-p, secondary         <--- The Secondary unit in the cluster. Cluster uptime: 400 days, 9 hours, 58 minutes, 3 seconds get system performance status  CPU states: 6% user 0% system 0% nice 94% idle 0% iowait 0% irq 0% softirq CPU0 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq CPU1 states: 7% user 0% system 0% nice 93% idle 0% iowait 0% irq 0% softirq Memory: 3701384k total, 3257232k used (88.0%), 247096k free (6.7%), 197056k freeable (5.3%) <---- The memory usage is high on the Secondary device.  Check the processes using high memory. Example output:  diagnose sys top 1 20 20 Run Time: 60 days, 11 hours and 12 minutes 5U, 0N, 2S, 92I, 1WA, 0HI, 0SI, 0ST; 3614T, 240F bcm.user 112 S < 1.9 0.4 7 forticron 3252 R 0.3 2.0 5 hasync 246 S 0.2 0.4 1 node 14694 S 0.0 12.7 6  <---- Node daemon consuming 12.6% of the memory. node 32403 S 0.0 3.1 5   <---- Node daemon consuming 3.1% of the memory. node 32401 S 0.0 2.9 7   <---- Node daemon consuming 2.9% of the memory. node 32404 S 0.0 2.9 0   <---- Node daemon consuming 2.9% of the memory. node 32405 S 0.0 2.5 3   <---- Node daemon consuming 2.5% of the memory. 5 of the Node daemons are consuming 24% of the memory on the Secondary unit. Restart the Node daemon on the Secondary unit. Restart the process gracefully with (available since FortiOS v7.2.1): diagnose nodejs process restart Or restart the process by killing it with: fnsysctl killall node Memory usage will get back to normal: get system performance status Memory: 3701384k total, 2118272k used (57.2%), 1219000k free (32.9%), 364112k freeable (9.9%) diagnose sys top 1 5 5 ....(output truncated)...  bcm.user 112 S < 2.9 0.5 7 node 28155 S 0.4 1.4 7   <---- Node daemon memory usage reduced to 1.4%. httpsd 28250 S 0.4 0.9 6 hasync 246 S < 0.0 1.8 5 ipshelper 399 S 0.0 1.7 2 This issue is fixed in version 7.6.3. An automation script can be configured to restart the Node daemon periodically to avoid conserve mode.  config system auto-script     edit restart_node         set interval 86400  <---- 24 hours.         set repeat 1000     <---- 1000 times.         set start auto         set script 'fnsysctl killall node'     next end Note: The script will start the interval when it got added, so to run periodically at night it needs to be added at this time.