Description
This article describes how to investigate whether FortiSIEM Linux Agent causes high CPU utilization on Linux servers.
Scope
Linux Agent v5.3.0 - v7.2.6 Supervisor and Collector v6.x, v7.x, v7.1.x, v7.2.x.
Solution
- Verify if high utilization is caused by the FortiSIEM Agent by checking the CPU and Memory Utilization of the phLinuxAgent process via the Linux SSH.
top
ps -eo pid,comm,%mem,%cpu --sort=-%mem | head -n 10
-
Ensure that the system has met the pre-requisite defined in the Linux Agent Installation Guide and FortiSIEM Version Compatibility Matrix as per the installed agent version.
-
On the Linux Server, check for any errors related to FortiSIEM or phLinuxAgent in system logs:
cat /var/log/messages
cat /var/log/audit/audit.log
-
Investigate from the FortiSIEM Analytics tab to find out which event type count is maximum.
Run a historical search for 10 minutes or 1 hour :
Filters: Reporting IP = <Linux-Server-IP> AND System Event Category BETWEEN 0,6
Display conditions: Reporting IP , Event Name , Count(Matched Events)
Export Report for 10-60 minutes duration or time of cpu spike for review.
-
Temporarily exclude the high event type from the Linux Agent Template and click Apply on Template to observe if system utilization has returned to normal. Later, evaluate if the event is required in FortiSIEM or can reduce the occurrence on the Linux server.
-
Probe for any errors in Agent logs from the Linux server:
Agent Service logs are located in opt/fortinet/fortisiem/linux-agent/log/fortisiem-linux-agent.log.
Agent Application log files are located in /opt/fortinet/fortisiem/linux-agent/log/phoenix.log. -
Address the 'PH_UTIL_FILE_STAT_FAILURE' errors in Linux Agent logs.
Sample Error message in Linux Server /opt/fortinet/fortisiem/linux-agent/log/phoenix.log :
CentOS-OrgA phLinuxAgent[1652286]: [PH_UTIL_FILE_STAT_FAILURE]:[eventSeverity]=PHL_ERROR,[procName]=phLinuxAgent,[fileName]=phMiscUtils.cpp,[lineNumber]=2339,[filePath]=/proc/39/exe,[errorNoInt]=13,[phLogDetail]=Failed to stat file
CentOS-OrgB phLinuxAgent[567333]: [PH_UTIL_FILE_OPEN_FAILURE]:[eventSeverity]=PHL_ERROR,[procName]=phLinuxAgent,[fileName]=phMiscUtils.cpp,[lineNumber]=3573,[filePath]=/var/log/messages,[errorNoInt]=13,[phLogDetail]=Failed to open file
The error 'Dir could not be opened' would be generated if FortiSIEM does not have enough privileges to read that directory. The amount of these errors also causes the agent to use more CPU and memory.
Try changing the permissions for the current user or changing the access mode to global, i.e, 777 or 775.
cd
chmod -R 777 <directory-name>
Do note to consult this with the company's Linux Administrator and is it compliant to modify this file permission.
Also consider monitoring specific file instead of entire /var, /etc, /proc etc in Linux template, since this system directory has files frequently written every second for traffic logs and it would be noisy.
-
Probe for any errors in FortiSIEM collector logs, phAgentManager process :
cat /opt/phoenix/log/phoenix.log | grep -i phAgentManager
-
Inspect the Network Connectivity between the Linux server and the supervisor/collector for any latency.
-
Verify if large events are cached on the Linux server. The cache files are stored in the following directories:
/opt/fortinet/fortisiem/linux-agent/upload,
/opt/fortinet/fortisiem/linux-agent/cache
/opt/fortinet/fortisiem/linux-agent/svnUpload
ls | wc -l -> Count of event files in upload directory.
du -sh * -> Size of files in directory.
If the issue persists, then contact Fortinet Support with all the above investigated details.
