Skip to main content
mbenvenuti
Staff
mbenvenuti
Staff
June 17, 2024

Troubleshooting Tip: How to troubleshoot collector issues

  • June 17, 2024
  • 0 replies
  • 12383 views

Description

This article describes how to troubleshoot Collectors.

Scope

FortiSIEM Collector node.

Solution

In some cases, the Collector has successfully been registered in the past, but is now experiencing some issues and shows a critical health status or seems to not receive or transmit events.

Follow the next steps to identify the source of the issue:

 

  1. Check the health status seen by the super. Use the GUI to access Admin -> Health -> Collector Health.


collector_health.png

 

To get status details, hover the mouse over the Collector status and check what is in the pop-up. This will make it easy to focus on what is worth attention, such as:

  • Processes are down.

  • Cache usage.

  • RAM / CPU usage.

  • Disk issues.

 

Check the 'Last Status Updated' and the 'Last File Received' to see if the status is up-to-date or when the issue has started. This can correlate with external changes like the new firewall policy or link disconnection issues.

 

  1. Check the last collected events. Perform a basic check to make sure that the collector is sending events to the super/workers from the GUI with the Analytics menu.

  • Run a Real-time query with filter 'Collector ID = 1000x' using the proper Collector ID.

  • If no events have been displayed for 5 minutes, run a historical search with the same filter to check when the last events were received.

 

  1. Check the running processes. All ph services are down:

  • The Collector might not have been registered correctly. The Collector must be registered with the steps outlined in Register Collectors.

  • phParser process shows high CPU usage:

    • A lot of unknown events were received: This can be checked under Analytics. Run a query with the filter 'event type: Unknown_EventType'.

    • A new device may have been configured to send logs to FortiSIEM, but this device is not compliant and is not part of the External Systems Configuration Guide. To fix that, a dedicated parser can be defined or remove logging configuration on the source device.

    • Event dropping rules are configured with a complex regex. In Admin -> Settings -> Event Handling -> Dropping, rules to drop events are defined there. It is recommended to remove the logging configuration of the source device as much as possible and use the CONTAIN operator instead of REGEX as much as possible. The REGEX operator is computationally intensive, especially on high EPS.

 

  1. Check disk usage. Run the following command on the collector as root to check disk space:


df -h


If the / root disk needs to be checked:

mount -o bind / /mnt
du -h --max-depth=1 /mnt/ | sort -rh
umount /mnt #After checking directories


If /opt disk needs to be checked:

du -h --max-depth=1 /opt | sort -rh


Identify the directories that are taking up space. It could be because of system logs at:

  • /mnt/var/log and /mnt/var/log/httpd, /root/*.log, or /tmp directories where some cleanup needs to be done.

  • /opt/phoenix/log/ - FortiSIEM system logs.

  • /opt/phoenix/cache/parser/upload/ - the location of the events collected and waiting to be sent to the super/workers.

 

  1. Check for incoming events. Check for SYSLOG packets:


tcpdump udp and port 514 -vvv -w /tmp/collector.pcap


Check for SNMP packets:

tcpdump udp and port 162 -vvv -w /tmp/collector.pcap


Check for Flow packets:

tcpdump 'udp and (port 2055 or port 6343)' -vvv -w /tmp/collector.pcap


Be aware that not all Flows are supported. See the External Systems Configuration Guide for Flow Support. If the source device is sending Netflow to the FortiSIEM but cannot decode it, make sure that the Netflow Template is sent by the source device regularly.

If the tcpdump command shows 'Got 0', review the network or device configuration. Optionally, check /tmp/collector.pcap for further analysis.

Make sure phParser is listening on those ports to treat the UDP packets:

netstat -tulpn | egrep '514|162|2055|6343'
tcp6 0 0 :::514 :::* LISTEN 14161/phParser
tcp6 0 0 :::6514 :::* LISTEN 14161/phParser
udp 0 0 0.0.0.0:162 0.0.0.0:* 14161/phParser
udp6 0 0 :::514 :::* 14161/phParser
udp6 0 0 :::2055 :::* 14161/phParser
udp6 0 0 :::6343 :::* 14161/phParser

 
Check the EPS count on the collector:

tail -f /opt/phoenix/log/phoenix.log | grep PH_SYSTEM_PERF_EVENTS_PER_SEC


Example output:

2024-06-17T11:49:50.600701+02:00 fsmCol714 phParser[14161]: [PH_SYSTEM_PERF_EVENTS_PER_SEC]:[eventSeverity]=PHL_INFO,[procName]=phParser,[fileName]=LicenseEnforce.cpp,[lineNumber]=575,[eventsPerSec]=0.011111,[peakEventsPerSec]=0.050000,[phLogDetail]=

 

  1. Check for license overruns. Go to Admin -> License-> Usage -> EPS Usage, and make sure that License EPS is not below the current running EPS. If that is the case, Super will not accept new events. Make sure that the License is updated, or change device configuration to send fewer events by keeping only critical event types.

 

  1. Check for communication with the super and workers. Check for cluster configuration on the super under Admin -> Settings -> Cluster Config:

 

AdminSettingsClusterConfig.png

 

Check if the configuration is in sync in the collector:

cat /opt/phoenix/config/phoenix_config.txt | egrep 'APP_SERVER_HOST'
APP_SERVER_HOST=10.5.8.35
cat /opt/phoenix/config/phoenix_super.txt


Example output:

10.5.8.35,

 

Make sure that IPs or FQDNs listed as Event Upload Workers are reachable without filtering, proxy, or SSL inspection.

 

From the collector CLI:

curl -vk https://super_or_worker_address

* Rebuilt URL to: https://super_or_worker_address/
* Trying super_or_worker_address...
* TCP_NODELAY set
* Connected to super_or_worker_address (super_or_worker_address) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, [no content] (0):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
* subject: C=US; ST=CA; L=SunnyVale; O=Fortinet; CN=localhost
* start date: May 16 10:07:44 2024 GMT
* expire date: May 14 10:07:44 2034 GMT
* issuer: C=US; ST=CA; L=SunnyVale; O=Fortinet; CN=localhost
* SSL certificate verify result: self signed certificate (18), continuing anyway.
* TLSv1.3 (OUT), TLS app data, [no content] (0):
> GET / HTTP/1.1
> Host: super_or_worker_address
> User-Agent: curl/7.61.1
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS app data, [no content] (0):
< HTTP/1.1 200 OK
< Date: Wed, 12 Jun 2024 11:24:23 GMT
< Server: Apache
< X-XSS-Protection: 1; mode=block
< x-Frame-Options: SAMEORIGIN
< X-Content-Type-Options: nosniff
< Strict-Transport-Security: max-age=31536000; includeSubDomains
< Content-Security-Policy: default-src 'self' https://*.duosecurity.com https://*.googleapis.com https://*.gstatic.com; img-src 'self' data: https://maps.googleapis.com https://maps.gstatic.com; script-src 'self' 'unsafe-inline' 'unsafe-eval' https://*.googleapis.com; style-src 'self' 'unsafe-inline' https://*.googleapis.com;
< Referrer-Policy: no-referrer-when-downgrade
< Last-Modified: Mon, 26 Feb 2024 14:01:19 GMT
< ETag: "120-612495807c093"
< Accept-Ranges: bytes
< Content-Length: 288
< Content-Type: text/html; charset=UTF-8
<
<html>
<head>
<title></title>
<script type="text/javascript">
var hst = location.hostname
document.write("<meta HTTP-EQUIV='REFRESH' content='0; url=https://"+hst+"/phoenix' >")
</script>

</head>
<body>
<b> <h2 align="center"> </h2> </b>

<br><br><br><br><br><br>

</body>
</html>

* Connection #0 to host super_or_worker_address left intact

 

If an SSL certificate is configured on the super and workers, make sure the common name matches the Fully Qualified Domain Name of the machine. 

Check the events on the super or the worker. From the super or worker CLI, run the following:

tail -f /var/log/httpd/ssl_request_log | grep evthandler | grep Collector_IP_or_Collector_ID


Example output:

10.5.8.94 - 10001 [17/Jun/2024:12:03:53 +0200] "PUT //evthandler2?10001 HTTP/1.1" 200 - "-" "-" 


If the HTTP code is 200, this means that events have been received. If it is another code, such as 401 or 403, the collector has authorization issues. Renew registration with a phProvision script on the collector with the --update option.

 

  1. Check for the correct collector service password authentication to the super. 

  • From the GUI at Admin -> Settings -> Cluster Config, note the list of supervisor addresses.

  • From the GUI at Admin -> Health -> Collector, note the collector ID.

 

Then, for each supervisor address, run the following from the collector CLI as the root user with the previously recorded collector ID (e.g. 10001):

agent_id=`cat /opt/phoenix/config/phoenix_config.txt | grep agent_id | sed 's#agent_id=##g'`
srvpass=`phLicenseTool --showServicePassword`
curl -u "${agent_id}:${srvpass}" -vk https://Super_address:443/phoenix/rest/config/systemConfig > /tmp/systemConfig.xml


Check the result of the output command and the /tmp/systemConfig.xml. Each super address should be resolvable, the command should not result in a 4xx or 5xx code, and '/tmp/systemConfig.xml' should contain configuration data from the super.
If the page is not found, the network and firewall configuration need to be reviewed. Any proxy or SSL inspection needs to be removed.

 

If it is an authentication issue, renew registration using the phProvision script on the collector with the '--update' option.

 

  1. Check for NO_EVENT_FILE_UPLOAD or NO_SVN_FILE_UPLOAD. For safety reasons, the system may block itself from treating the events. On each FortiSIEM node, run the following command:


ls -l /opt/phoenix/cache/NO_*


If '/opt/phoenix/cache/NO_EVENT_UPLOAD_FILE' or 'NO_SVN_FILE_UPLOAD' are present, those files behave like flags.

Check for explicit errors in the logs at '/opt/phoenix/log/phoenix.log'. Check for .err files under '/opt/phoenix/cache/parser/upload/svn':

cd /opt/phoenix/cache/parser/upload/svn
ls -l
rm -rfv *.err


Remove those files with the following commands:

cd /opt/phoenix/cache/
rm -rfv NO_EVENT_UPLOAD_FILE NO_SVN_FILE_UPLOAD

 

  1. Check for cached events. Events in the collector cache can be counted with this command from the collector CLI as root:


find /opt/phoenix/cache/parser -type f | wc -l


This result must be close to 0. If this figure is growing, it either means the connection to the super/worker is in a failure state or too slow compared to the amount of incoming events. Details of events in the cache can be checked with the following command:

find /opt/phoenix/cache/parser -type f

 

  1. Check for Content upgrade access. When it is required to check if the collector can run a content upgrade, execute the next command from the collector CLI as root:

 

agent_id=`cat /opt/phoenix/config/phoenix_config.txt | grep agent_id | sed 's#agent_id=##g'`
srvpass=`phLicenseTool --showServicePassword`
export IFS=','
for super_address in `cat /opt/phoenix/config/phoenix_super.txt`; do echo "Testing access to $super_address"; curl -u "${agent_id}:${srvpass}" -kL "https://${super_address}/ContentUpgrade"; done


Example output:

Testing access to 10.5.8.44
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<html>
<head>
<title>Index of /ContentUpgrade</title>
</head>
<body>
<h1>Index of /ContentUpgrade</h1>
<table>
...
</table>
</body></html>

 

  1. Check for errors in the logs. In general, ongoing errors can be checked on the node with the next command from the collector CLI:


tail -f /opt/phoenix/log/phoenix.log | grep PHL_ERROR

 

Common known errors:

 

  1. PH_HTTP_RESPONSE_FAILURE:


2024-05-28T01:01:09.649952+02:00 collector phEventPackager[90872]: [PH_HTTP_RESPONSE_FAILURE]:[eventSeverity]=PHL_WARNING,[procName]=phEventPackager,[fileName]=phHttpClient.cpp,[lineNumber]=614,[errorNo]=500,[phLogDetail]=HTTP response code failure

 

  • Reason: The collector is not able to send the events to the worker or the super.

  • Resolution: The application Server process at Super needs to be checked.

 

  1. PH_PARSER_FILE_STAT_FAILURE:


2023-06-16T08:49:54.775570+03:00 collector phParser[4251]: [PH_PARSER_FILE_STAT_FAILURE]:[eventSeverity]=PHL_ERROR,[procName]=phParser,[fileName]=phAgentEventProcessor.cpp,[lineNumber]=401,[filePath]=/opt/phoenix/cache/parser/upload/win/TL5tpQ.gzs,[errorNoInt]=2,[phLogDetail]=Failed to stat file

 

  • Reason: The parser process does not correctly grant permissions to treat the file.

  • Resolution: Restore proper file permissions with the next commands from the collector CLI as root:

 

find /opt/phoenix/cache/parser/ -ls
chown -R admin:admin /opt/phoenix/cache/parser/
chmod -R 755 /opt/phoenix/cache/parser
chmod -R 775 /opt/phoenix/cache/parser/fwd
chmod -R 775 /opt/phoenix/cache/parser/upload/

 

  1. PH_UTIL_XML_HANDLING_ERROR:


2024-03-08T01:19:00.818830+00:00 collector phParser[100925]: [PH_UTIL_XML_HANDLING_ERROR]:[eventSeverity]=PHL_ERROR,[procName]=phParser,[fileName]=phBaseXmlParser.cpp,[lineNumber]=332,[errReason]=Exception: Expected end of tag 'data', Level: Fatal error, Line No: 1, Column No: 10668,[phLogDetail]=Failed to handle XML

 

  • Reason: The wrong parser definition has been applied.

  • Resolution: Go to FortiSIEM GUI at Admin -> Device Support -> Parsers, review the last parser changed, deactivate it, and select the 'Apply' Button to sync with collectors. Then, review its definition until the parser can be activated again and sync with 'Apply'.

 

  1. PH_GENERIC_CRITICAL (upload service thread 0 seems stuck on task):


2025-01-17T16:00:01.810575+05:30 collector1 phEventPackager[3357]: [PH_GENERIC_CRITICAL]:[eventSeverity]=PHL_CRITICAL,[procName]=phEventPackager,[fileName]=phEventPKGProcess.cpp,[lineNumber]=1280,[phLogDetail]=Upload service thread 0 seems stuck on task [ ip: 10.1.2.22, path: /evthandler2?10003, filePath: /opt/phoenix/cache/parser/events/evt_1736814170_1_9362.dat, counter: 22, upload seconds: 656

 

  • Reason: The collector cannot connect to port 443 of the Supervisor/worker IP/hostname configured in ADMIN -> Settings -> Cluster Config -> Event Upload Worker list. Use curl -kv https://hostname_or_IP/ to verify connectivity.

  • Resolution: Fix connectivity from the collector to the Supervisor/worker IP/hostname over port 443. Make sure DNS resolution also works if the hostname is used in ADMIN -> Settings -> Cluster Config -> Event Upload Worker list.

 

  1. PH_EVT_PACKAGER_FILE_UPLOAD_FAILURE:


See details in Troubleshooting Tip: How to resolve Collector Event Upload errors with Self-signed Certificates installed on Workers.

 

  1. PH_EVENT_PKG_SERVICE_LIST_EMPTY:


2025-01-17T16:00:01.810575+05:30 collector1 phEventPackager[9744]: [PH_EVENT_PKG_SERVICE_LIST_EMPTY]:[eventSeverity]=PHL_ERROR,[procName]=phEventPackager,[fileName]=phEventPKGProcess.cpp,[lineNumber]=630,[phLogDetail]=Empty upload service list

 

  • Reason: phMonitor failed to obtain the IP/FQDN to upload.

  • Resolution: Update the IP in /opt/phoenix/config/phoenix_config.txt.

 

vi /opt/phoenix/config/phoenix_config.txt
Line 23: APP_SERVER_HOST=<Super IP>
Line 935: parser_server_upload_host=<Super and Worker IP>
Line 936: svn_server_upload_host=<Super and Worker IP>
Line 960: parser_server_upload_host=<Super and Worker IP>
Line 961: svn_server_upload_host=<Super and Worker IP>

      Terms & ConditionsAccessibility statement
      Thought Leadership Security Summit. Outpace New Threats with AI - enhanced defense. Tuesday, Septmeber 15, 8:30 AM - 2:30 PM PT. The Golf Club at Newcastle, WA.
      Fortinet Flag the Hack. Wednesday, August 26, 9:00 AM - 5:00 PM ET, COSM, Atlanta, GA.
      Virtual event | September 2026. SASE summit. The age of autonomous trust. Register here!