| Solution |
When running a report or any queries in the Analytics section, if the result gets stuck loading endlessly at 0% or the result is not what is expected, it can be useful to check what exact query is being run in the backend and watch for any explicit errors in the logs.
Here are the steps to follow:
- Check FortiSIEM clustering configuration
It is important to identify if workers are set up in the cluster and if the cluster is being used for queries.
- Enter the GUI and navigate to Admin -> Settings -> Cluster config / Query Worker or Clickhouse config (if using Clickhouse DB).
- List all the workers that are configured for queries.
- Check live system logs
- Connect to each query worker CLI as root, including the super, and run the following:
tail -f /opt/phoenix/log/phoenix.log /opt/glassfish/domains/domain1/logs/phoenix.log | tee -a /tmp/query.log
- From the GUI, go to Analytics and run the involved query.
- Stop the 'tail' command with the 'CTRL + C' keys and check the command output:
==> /opt/glassfish/domains/domain1/logs/phoenix.log <==
2024-06-20 15:01:40,028 [http-listener-2(10)] INFO com.ph.phoenix.framework.logging.PhAudit - [PH_AUDIT_QUERY_START]:[phCustId]=1,[eventSeverity]=PHL_INFO,[phEventCategory]=2,[procName]=AppServer,[srcIpAddr]=172.26.128.202,[osObjName]=Historical events for Group: FortiSIEM,[user]=admin,[queryId]=15969,[phLogDetail]=Starting an adhoc query
==> /opt/phoenix/log/phoenix.log <==
2024-06-20T15:01:40.447308+02:00 fsm713ClickH phQueryMaster[3293805]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phQueryMaster,[fileName]=ClickHouseResultSaver.cpp,[lineNumber]=145,[phLogDetail]=non-aggregate result data is empty
2024-06-20T15:01:40.487494+02:00 fsm713ClickH phQueryMaster[3293805]: [PH_QUERY_DURATION]:[eventSeverity]=PHL_INFO,[procName]=phQueryMaster,[fileName]=ResourceTracker.cpp,[lineNumber]=28,[reportName]=Historical events for Group: FortiSIEM,[queryId]=15969,[description]=,[durationMSec]=187,[phLogDetail]=Query statistics
==> /opt/glassfish/domains/domain1/logs/phoenix.log <==
2024-06-20 15:01:40,496 [p: thread-pool-1; w: 3] INFO com.ph.phoenix.framework.logging.PhAudit - [PH_AUDIT_QUERY_COMPLETED]:[phCustId]=1,[eventSeverity]=PHL_INFO,[phEventCategory]=2,[queryDisplay]="Event Receive Time,Reporting IP,Event Type,Event Name,Raw Event Log",[durationMSec]=579,[queryFilter]="Reporting IP = Group: FortiSIEM ",[procName]=AppServer,[osObjName]=Historical events for Group: FortiSIEM,[user]=admin,[customer]=Super,[queryId]=15969,[usageType]=GUI/API,[phLogDetail]=Query completed
This log is displayed when the query has been executed successfully. It can be used to identify the query ID, who ran it, for how long, its filter, and its display. Note down the query ID for the next step.
- Check exact query details.
- From the node CLI as the root, run the following (replace QUERY_ID with the query ID from the previous step):
find /opt/phoenix/cache/query* -type f | grep QUERY_ID
/opt/phoenix/cache/query/completed/15969/query.xml
/opt/phoenix/cache/query/completed/15969/query.sta
- The query is in the 'completed' directory when the query is finished, but it can be in 'active' or 'stopped' if the query is still going on or stopped.
- Query details can be checked with the following:
xmllint --format /opt/phoenix/cache/query/completed/15969/query.xml
<?xml version="1.0"?>
<DataRequest custId="1" dataCreationType="SYSTEM" dbId="0" end="20" entityVersion="null" id="15969" issueCustId="3" retention="0" source="Online" type="Query">
<DataSource/>
<Name>Historical events for Group: FortiSIEM</Name>
<Description/>
<CustomerScope groupByEachCustomer="false">
<Include all="true"/>
<Exclude/>
</CustomerScope>
<PatternClause>
<SubPattern id="0" name="">
<SingleEvtConstr>reptDevIpAddr=Group@PH_SYS_DEVICE_FSM</SingleEvtConstr>
<SingleEvtConstrCaseInsensitive>(reptDevIpAddr=Group@PH_SYS_DEVICE_FSM) AND phEventCategory IN (0,4,6)</SingleEvtConstrCaseInsensitive>
</SubPattern>
</PatternClause>
<SelectClause>
<AttrList>phRecvTime,reptDevIpAddr,eventType,eventName,rawEventMsg</AttrList>
</SelectClause>
<OrderByClause>
<AttrList>phRecvTime DESC</AttrList>
</OrderByClause>
<Trend>true</Trend>
<GroupSize>0</GroupSize>
<FilterInReport>false</FilterInReport>
<userRoles>
<roles custId="0">976301</roles>
</userRoles>
<SyncOrgs/>
<ReportInterval>
<Low>1718887900</Low>
<High>1718888499</High>
</ReportInterval>
<TrendInterval>auto</TrendInterval>
<TimeZone>Europe/Paris</TimeZone>
</DataRequest>
- Known issues.
Query result is loading endlessly and stays at 0%.
Reason: Storage is having issues
Solution: NFS or one Clickhouse node can be disconnected. Check for phQuery and phDataManager errors to fix storage access.
Some results are expected, but some events are not shown.
Reason:
- The role assigned to the user has 'Data Conditions', and the filter has more attributes and values in the .xml file than on the GUI.
- If groups are being used in the filter, the Redis memory limit may have been reached.
Solutions:
- Other
If required, provide the output of the commands above, /tmp/query.log files and query.xml to the support team for further analysis.
|
