Troubleshooting Tip: Resolving 'Query server internal error' in FortiSIEM Analytics UI

Analytics queries in the UI return: Query server internal error.

PostgreSQL queries from the Supervisor still succeed:

psql phoenixdb phoenix -c "SELECT now();"
psql phoenixdb phoenix -c "SELECT FROM ph_dev_event_attr LIMIT 3;"

Example:

root@AH-SUPER ~] psql phoenixdb phoenix -c "SELECT now();"
now
-
2026-03-18 17:22:01.209711+00
(1 row)

This confirms the database is functioning correctly.

Check AppServer Logs.

Run the following command to identify query-related errors in the AppServer logs:

grep --color=always -iH -C 5 "Query server internal error" /opt/glassfish/domains/domain1/logs/.log | less -R

Example output:

PH_APPSERVER_QUERY_RUN_ERROR ... Query server internal error
javax.persistence.PersistenceException: SQLGrammarException
org.postgresql.util.PSQLException: syntax error at or near ")"

Check backend logs:

Review backend logs for query-related warnings:

grep --color -i phQuery /opt/phoenix/log/phoenix.log

Example:

Could not open dir /query/stopped
Could not open dir /query/completed
Could not open dir /query/result

These messages indicate missing or inaccessible query working directories.

Root Cause.

This issue is typically caused by inconsistencies on the AppServer side rather than a database failure.

Common causes include:

• Missing or corrupted cache directories.
• Missing query working directories.
• Incomplete initialization after upgrade.
• Disk space pressure on /opt.
• Stale or excessive Machine Learning cache data.

Even when query services are running and the database is healthy, the AppServer may still fail to execute queries properly if required directories or cache data are not in a valid state.

Solution.

1. Verify Query Processes.

ps -ef | grep phQuery

Ensure both processes are running:

phQueryMaster.
phQueryWorker.

2. Recreate Cache Directories.

mkdir -p /opt/phoenix/cache/query
mkdir -p /opt/phoenix/cache/querywkr
chown -R admin:admin /opt/phoenix/cache/
chmod -R 755 /opt/phoenix/cache/

3. Recreate Query Working Directories.

Create the following directories if they do not exist:

mkdir -p /opt/phoenix/cache/parser/upload/evt
mkdir -p /opt/phoenix/cache/parser/upload/win
mkdir -p /opt/phoenix/cache/parser/upload/linux

mkdir -p /querywkr/stopped
mkdir -p /querywkr/completed
mkdir -p /querywkr/result
mkdir -p /querywkr/report

mkdir -p /query/active
mkdir -p /query/completed
mkdir -p /query/report
mkdir -p /query/result
mkdir -p /query/stopped

Set permissions:

chown -R admin:admin /opt/phoenix/cache/
chown -R admin:admin /query /querywkr

chmod -R 775 /opt/phoenix/cache/parser/upload/
chmod -R 755 /query/
chmod -R 755 /querywkr/

Machine learning cache cleanup.

This issue has also been associated with the /opt/MachineLearning directory increasing in size, which can impact query performance and system behavior.

This was previously identified as a bug and has been addressed in:

7.1.4.
7.2.0.

Root cause:
The cleanup script was not properly copied or executed in some versions, leading to stale data buildup.

When to run this step:

• /opt usage is high.
• The issue appeared after an upgrade.
• Problems persist after fixing directories and permissions.

Run the cleanup script.

/opt/phoenix/phscripts/bin/phMLCacheCleaner.py

After running the script, re-test Analytics queries.

Validation Steps:

1. Log in to FortiSIEM.
2. Navigate to Analytics.
3. Run a query.

Expected results:

• Query completes successfully.
• Data is returned.
• No errors appear in the AppServer logs.