Technical Tip: Analytics searches cause phQueryWorker to restart repeatedly

Description

This article describes how to resolve an issue where Analytics and Advanced Search operations cause the phQueryWorker process to crash repeatedly.

Scope

FortiSIEM.

Solution

After upgrading FortiSIEM, Analytics and Advanced Search queries may fail and the phQueryWorker process may repeatedly go offline and restart.

Symptoms may include:

• Analytics searches fail unexpectedly.

• Advanced Search does not return results.

• phQueryWorker repeatedly crashes and restarts.
  
  

Investigation of the logs may show that FortiSIEM is unable to create temporary files or access cache directories required by the query worker.
As an initial validation, run the following commands on the Supervisor and all Worker nodes:

df -h
mount | grep opt
ls -ld /opt/phoenix/cache/parser
ls -l /opt/phoenix/cache/parser
ls -ld /opt/phoenix/cache/querywkr
ls -l /opt/phoenix/cache/querywkr
ls -ld /opt/phoenix/cache/querywkr/result
ls -ld /opt/phoenix/cache/querywkr/report
ll /opt/phoenix/cache/query/

If the permissions on the 'querywkr' directory or its subdirectories are incorrect, run the following commands on Super :

cd /opt/phoenix/cache/
chown admin:admin querywkr/
chmod 755 querywkr/
ls -ld querywkr
ls -l querywkr
cd querywkr
ls -l
chown admin:admin completed/
chown admin:admin report/
chown admin:admin result/

Once the permissions have been corrected, restart the query processes:

su admin
phtools --stop phQueryMaster
phtools --stop phQueryWorker

Verify using the 'phstatus' command that both processes have completely stopped. Wait for few seconds and confirm whether the processes start automatically.
If they do not start automatically, start them manually:

su admin
phtools --start phQueryMaster
phtools --start phQueryWorker

After correcting the directory permissions and restarting the services, Analytics and Advanced Search queries will complete successfully and the phQueryWorker process is stable.