Technical Tip: High Node.js daemon memory usage triggers Conserve Mode on Secondary HA Unit in HA cluster

A gradual increase in memory usage by the Node daemon has been observed on the secondary unit of an HA cluster after upgrading to FortiOS v7.6.0. Over time, this behavior can cause the device to enter Memory Conserve Mode.

• Log in to the Secondary unit in the cluster as outlined in the article: Technical Tip: How to access secondary unit of HA cluster via CLI

• Check the crash log to confirm that the device has entered conserve mode.

diagnose debug crashlog read 

122: 2024-07-11 08:00:10 service=kernel conserve=on total="3614 MB" used="3180 MB" red="3180 MB"
123: 2024-07-11 08:00:33 green="2963 MB" msg="Kernel enters memory conserve mode"  <--- Conserve mode is triggered. 

• Check the memory usage when the memory usage is high or when the device enters conserve mode.

get system status
Version: FortiGate-100F v7.6.0,build3401,240724 (GA.F)
First GA patch build date: 240724
...(output truncated)
Current HA mode: a-p, secondary         <--- The Secondary unit in the cluster.
Cluster uptime: 400 days, 9 hours, 58 minutes, 3 seconds

get system performance status 

CPU states: 6% user 0% system 0% nice 94% idle 0% iowait 0% irq 0% softirq
CPU0 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU1 states: 7% user 0% system 0% nice 93% idle 0% iowait 0% irq 0% softirq
Memory: 3701384k total, 3257232k used (88.0%), 247096k free (6.7%), 197056k freeable (5.3%) <---- The memory usage is high on the Secondary device. 

• Check the processes consuming high memory.

diagnose sys top 1 20 20
Run Time: 60 days, 11 hours and 12 minutes
5U, 0N, 2S, 92I, 1WA, 0HI, 0SI, 0ST; 3614T, 240F
bcm.user 112 S < 1.9 0.4 7
forticron 3252 R 0.3 2.0 5
hasync 246 S 0.2 0.4 1
node 14694 S 0.0 12.7 6  <---- Node daemon consuming 12.6% of the memory.
node 32403 S 0.0 3.1 5   <---- Node daemon consuming 3.1% of the memory.
node 32401 S 0.0 2.9 7   <---- Node daemon consuming 2.9% of the memory.
node 32404 S 0.0 2.9 0   <---- Node daemon consuming 2.9% of the memory.
node 32405 S 0.0 2.5 3   <---- Node daemon consuming 2.5% of the memory.

From the output above, five Node daemon processes collectively consume approximately 24% of total system memory on the secondary unit.

• Disabling security rating submission does not reduce memory utilization.

config system global
    set security-rating-result-submission disable
    set security-rating-run-on-schedule disable
end

This issue has been resolved in v7.6.3.

Workaround:
Until upgrading, one of the following workarounds can be applied on the secondary unit:

1. Restart the Node daemon manually on the Secondary unit using the commands below to reduce the memory utilization.

diagnose nodejs process restart

Or:

fnsysctl killall node

2. Configure an automation script to restart the Node daemon periodically.

config system auto-script
    edit restart_node
        set interval 86400 <---- 24 hours.
        set repeat 1000 <---- 1000 times.
        set start auto
        set script 'fnsysctl killall node'
    next
end

Note: The interval timer starts when the script is added. To ensure the restart occurs during off-peak hours (for example, at night), add the script at the desired start time.