- FortiGate system will enter into conserve mode when the memory usage is 88% or above.
- When the FortiGate is in conserve mode, node process responsible for FortiGate GUI management may not release memory properly causing entry-level devices to stay in conserve mode. This issue is fixed in FortiOS v7.2.8 and later, as well as v7.4.2 and later.
Symptoms:
- Node or httpsd process may be consuming more than normal amount of memory.
diagnose sys top 2 99 1
Run Time: 66 days, 19 hours and 26 minutes
1U, 0N, 0S, 99I, 0WA, 0HI, 0SI, 0ST; 1866T, 173F
httpsd 28502 S 7.4 1.1 7
httpsd 28516 D 4.9 1.0 6
node 149 S 0.4 21.9 2 <-
Freeable memory may be holding a high amount of memory and thus triggering the conserve mode event.
During a normal memory consumption period, use the following:
get system performance status
Memory: 8171732k total, 3042064k used (37.2%), 3541636k free (43.3%), 1588032k freeable (19.4%)
During Abnormal memory consumption period:
get system performance status
Memory: 8171732k total, 3487184k used (42.7%), 534020k free (6.5%), 4150528k freeable (50.8%) <-
Errors related to Node or Node Scripts are presented in the output of the following command:
diagnose debug crashlog read
1: 2022-08-08 18:47:55 <00417> ====================================================
3: 2022-08-08 18:47:55 <00417> Error: ENOENT: no such file or directory, open '/tmp/admin_server.crt'
6: 2022-08-08 22:27:34 <01043> ====================================================
7: 2022-08-08 22:27:34 <01043> ====== Node exiting due to uncaught exception: ======
8: 2022-08-08 22:27:34 <01043> ====================================================
10: 2022-08-08 22:27:34 <01043> Error: ENOMEM: not enough memory, write
Important note:
- For collecting data during normal memory consumption period, a restart of the relevant process or a reboot of the FortiGate device may be required, which should be scheduled as part of a maintenance activity.
- After a daemon restart or a FortiGate reboot, another iteration of the following debug data must be captured for baseline and comparison purposes.
- Depending on the user process that is restarted, end users may experience traffic outage.
Data Collection Plan:
- To report any new issues related to memory consumption by the node process, collect the following debug data during both normal and abnormal memory consumption periods of the daemon before submitting a support request to the Fortinet Technical Team.
fnsysctl du -i /dev/cmdb
fnsysctl du -a /dev/cmdb
fnsysctl df -k
fnsysctl ls -l /tmp
fnsysctl du -i /tmp
fnsysctl du -a /tmp
fnsysctl du -a / -d 1
fnsysctl ls -l /dev/shm
fnsysctl du -i /dev/shm
fnsysctl du -a /dev/shm
fnsysctl ls -l /node-scripts
fnsysctl du -i /node-scripts
fnsysctl du -a /node-scripts
get sys perf stat
diagnose sys top 2 99 3
diagnose sys top-fd
diagnose sys top-mem 20
diagnose sys top-sockmem
diagnose hardware sysinfo conserve
fnsysctl du
diagnose ips session status
diagnose ips packet status
diagnose ips memory status
diagnose sys session stat
diagnose sys dump-conserve-info
diagnose sys print-conserve-info
fnsysctl df
fnsysctl du /node-scripts
fnsysctl ls -la /node-scripts
fnsysctl ls -la /node-scripts/report-runner/results
fnsysctl ls -la /node-scripts/logs
fnsysctl cat /proc/meminfo
fnsysctl cat /proc/vmstat
execute tac report
Note: Super Admin privilege is required in order to run 'fnsysctl' command. Otherwise, FortiGate will return an error as mentioned in this article.
- Disconnect any active GUI sessions from the FortiGate and access its CLI via SSH to execute the following commands during problem state.
- When the httpsd daemon is consuming more memory, run the following debug commands:
diagnose debug reset
diagnose debug application httpsd -1
diagnose web-ui backtrace enable
diagnose web-ui backtrace httpsd <Enter the process ID of the httpsd daemon>
diagnose debug console timestamp enable
diagnose debug duration 2
diagnose debug enable
The debug commands will stop printing data after 2 minutes.
- When nodejs is consuming more memory, run the following debug commands:
diagnose debug reset
diagnose debug application nodejs -1
diagnose debug console timestamp enable
diagnose debug duration 2
diagnose debug enable
The debug commands will stop printing data after 2 minutes.
To permanently disable/reset the debugs, execute the following commands.
diagnose debug disable
diagnose debug reset
- Capture the process dump and traces and specify the process ID.
diagnose sys process pidof httpsd
diagnose sys process pidof node
diagnose sys process trace <Enter PID of node and httpsd one at a time>
diagnose sys process dump <Enter PID of node and httpsd one at a time>
diagnose sys process pstack <Enter PID of node and httpsd one at a time>
diagnose sys process sock-mem <Enter PID of node and httpsd one at a time>
Related articles:
|
