Troubleshooting Tip: High memory usage of node process

Description

This article describes the reason for high memory utilization in the node process.

Scope

FortiGate running FortiOS v6.4.x, v7.0.x, v7.2.x and v7.4.

Solution

On FortiOS v6.4, the node process is used for:

• Report management (which includes Security Fabric (CSF) / FortiView / Security Rating).

• WebSockets.

• Maintaining the CLI console widget when accessing the FortiGate via HTTP/HTTPS.

From FortiOS v7.0 onwards, the node process is also responsible for:

Processing all incoming HTTP/HTTPS to serve static files (before v7.0, the HTTPSD process served static files).

On FortiOS v7.0, the 3 main Node.js scripts on a FortiGate are for:

• Report runner (Security Rating).

• CLI console.

• SSL VPN QR code generation.

The security rating result submission is enabled by default on the FortiGate.
This feature enables the submission of security rating results to FortiGuard servers for data collection purposes and continuous learning.
The feature is memory-intensive and could lead to high memory usage observed on the node process.

A high memory usage of the node process can be seen, for example, with the following commands:

diagnose sys top-mem
diagnose sys top 1 20 1

Example output from the 'diagnose sys top' command:

Version: FortiGate-400E v6.4.7,build1911,210825 (GA)
Run Time: 43 days, 22 hours and 40 minutes
0U, 0N, 1S, 99I, 0WA, 0HI, 0SI, 0ST; 7852T, 2818F
node 197 S 0.0 31.1

The 'node 197 S 0.0 31.1' line indicates 31% memory usage of the node process.

To disable the security rating functionality, execute the following command:

config system global
    set security-rating-result-submission disable
end

In some cases, it might be required to also disable the scheduled rating and restart the Node.js process:

config system global
    set security-rating-result-submission disable
    set security-rating-run-on-schedule disable
end

In some cases, it might be required to also disable the scheduled rating and restart the Node.js process:

In FortiOS versions prior v7.0:

diagnose nodejs process restart

 
In FortiOS versions after FortiOS v7.0:

fnsysctl killall node

In some cases, it may be necessary to remove a single node process. List the IDs of node processes currently running:

diagnose sys process pidof node

Kill the appropriate process by ID:

diagnose sys kill 11 <Process ID>  

Running a 'killall' on a process can make the system unstable.

Another recommended workaround is to create an auto-script to periodically restart the node process, as shown below:

config system auto-script
    edit restart_node
        set interval 86400
        set repeat 100
        set start auto
        set script 'diagnose nodejs process restart'
    next
end

This script will automatically restart the node process every 24 hours (86400 seconds), which may help prevent the device from entering conserve mode.

Notes:
The command 'diagnose nodejs process restart' can be used in v7.6.5 with the auto-script option.

The command 'set security-rating-result-submission' is no longer available in v7.4.x.

High memory consumption on the Node.js process is still being observed in FortiOS v7.4.9. The manual workaround to restart the process manually or with an automation stitch can still be applied. Alternatively, upgrade to FortiOS v7.4.11 or v7.6.6, which both include the following command to auto-restart the Node.js process upon detection of high memory usage (the following command is hidden and does not display when running the command shown):

config system global
    set web-svc-auto-restart enable
end

In FortiOS v7.4.10+, v7.6.5+, and v8.0.0+, an additional logging option can be enabled by running the following command for further investigation of memory usage. After enabling it, restart the Node.js process with the following command:

diagnose nodejs process restart

config log settings
    set web-svc-perf enable
end

After enabling ‘web-svc-auto-restart’, 2 other commands will be available (but will be hidden so they won't show up in the auto-complete): 'web-svc-auto-restart-mem-threshold' and 'web-svc-auto-restart-time-threshold'.

web-svc-auto-restart-mem-threshold [100-500]

• This setting will set the memory threshold (in MB), which is the amount of memory that Node.js must be consuming when we check if Node.js should restart to conserve memory. If it passes this number, the Node process will be restarted. If set to 0, it will use the hardcoded memory thresholds in Node.js that are based on Firewall's platform:

  • 2GB models: 100MB.

  • Non-2GB models: 200MB.

web-svc-auto-restart-time-threshold [30-1140]

• This setting will set the minimum uptime (in minutes) that Node.js must be running for for it to start checking to see if the process's memory consumption is above the memory threshold to determine if it should auto-restart Node.js.

For example:

config system global
set web-svc-auto-restart enable
set web-svc-auto-restart-mem-threshold [value_here]
set web-svc-auto-restart-time-threshold [value_here]
end

Note:
The settings 'set web-svc-auto-restart enable' and 'set web-svc-perf enable' are known to cause the FortiOS Web GUI CLI console to disconnect intermittently with the message 'Connection Lost'.

This issue has been scheduled to be fixed in FortiOS v7.4.13, v7.6.8, and v8.0.1.

The setting 'set web-svc-auto-restart enable' was also identified as a cause for partial configuration loss issue after upgrading from v7.4.11 to v7.4.12 and from v7.6.6 to v7.6.7, as discussed in this article. Troubleshooting Tip: Partial Configuration Loss after Upgrade from FortiOS v7.4.11 and FortiOS v7.6.6.

Related article:

Troubleshooting Tip: CLI Console frequent 'Connection Lost' after enabling web-svc-perf under log setting.