Technical Tip: High memory usage of node process

This article describes the reason for high memory utilization in the node process.

FortiGate running v6.4.x, v7.0.x, v7.2.x and v7.4.

On v6.4, the node process is used for:

• Report management (which includes Security Fabric (CSF) / FortiView / Security Rating).
• WebSockets.
• Maintaining the CLI console widget when accessing the FortiGate via HTTP/HTTPS.

From v7.0 onwards, the node process is also responsible for:

Processing all incoming HTTP/HTTPS to serve static files (before v7.0, the process HTTPSD served static files).

On v7.0, the 3 main node.js scripts on a FortiGate are for:

• Report runner (Security Rating).
• CLI console.
• SSL VPN QR code generation.

The security rating result submission is enabled by default on the FortiGate.
This feature enables the submission of security rating results to FortiGuard servers for data collection purposes and continuous learning.
The feature is memory-intensive and could lead to high memory usage observed on the node process.

A high memory usage of the node process can be seen, for example, with commands:

diagnose sys top-mem
diagnose sys top 1 20 1

Example output from the 'diagnose sys top' command:

Version: FortiGate-400E v6.4.7,build1911,210825 (GA)
Run Time: 43 days, 22 hours and 40 minutes
0U, 0N, 1S, 99I, 0WA, 0HI, 0SI, 0ST; 7852T, 2818F
node 197 S 0.0 31.1 <-- 31 % memory usage of the node process.

To disable the security rating functionality, execute the following command:

config system global
    set security-rating-result-submission disable
end

In some cases, it might be required to also disable the scheduled rating and restart the Node.js process:

config system global
    set security-rating-result-submission disable
    set security-rating-run-on-schedule disable
end

In some cases, it might be required to also disable the scheduled rating and restart the Node.js process:

diagnose nodejs process restart      <-- Before v7.0.

fnsysctl killall node      <-- v7.0 Onward.

In some cases, it might be required to remove a single node process:

   diagnose sys process pidof node        <--- Lists the pids of node processes running.

   diagnose sys kill 11 <pid number>  

Running a 'killall' on a process can make the system unstable.

Another workaround suggested is to create an auto-script to periodically restart the node process, as shown below:

config system auto-script
    edit restart_node
        set interval 86400
        set repeat 100
        set start auto
        set script 'diagnose nodejs process restart'
    next
end

This script will automatically restart the node process every 24 hours (86400 seconds), which may help prevent the device from entering conserve mode.

Note:

The command 'set security-rating-result-submission' is not available anymore in v7.4.x.

High memory consumption on the Node.js process is still being observed in FortiOS v7.4.9. The manual workaround to restart the process manually or with an automation stitch still can be applied or upgrade to FortiOS v7.4.11 or v7.6.6, which include following command to auto-restart Node.js process when detected high memory usage (the following command is hidden and does not display when running the command show).

config system global
    set web-svc-auto-restart enable

end

An additional logging option can be enabled usingthe  below command for further investigation on the memory usage

config log settings
    set web-svc-perf enable
end