|
After a worker node failure or power cycle, a FortiSIEM ClickHouse worker may fail to start clickhouse-server, preventing it from being added back to the ClickHouse cluster. This is typically caused by a missing macros.xml file required for shard and replica identification.
The following log details can be observed from clickhouse-server.log below:
2026.01.06 15:39:44.844611 [ 8182 ] {} <Error> Application: DB::Exception: No macro 'shard' in config while processing substitutions in '/clickhouse/tables/{shard}/fsiem.summary' at '20' or macro is not supported here: Cannot attach table `fsiem`.`summary` from metadata file /data-clickhouse-hot-1/clickhouse/store/3d4/3d4ed95a-74d8-4133-bc27-9874b9edbeca/summary.sql from query ATTACH TABLE fsiem.summary UUID 'd5f196ae-d298-422f-b122-ead2e5baf10b' (`phRecvTime` DateTime, `phCustId` UInt32, `customer` String, `eventType` String, `reptDevIpAddr` IPv4, `hostIpAddr` IPv4, `reptDevName` String, `phEventCategory` UInt16, `eventSeverityCat` LowCardinality(String), `reptVendor` String, `reptModel` String, `parserName` LowCardinality(String), `eventParsedOk` UInt8, `collectorId` UInt32, `SUMM_EVET_CATEGORY` UInt16, `metrics_string.name` Array(LowCardinality(String)), `metrics_string.value` Array(String), `metrics_datetime.name` Array(LowCardinality(String)), `metrics_datetime.value` Array(DateTime), `metrics_ipv4.name`
This error indicates that ClickHouse is unable to start because it cannot resolve the {shard} macro required by FortiSIEM’s distributed ClickHouse table configuration.
Most importantly, these macros are mandatory on worker nodes that participate in a ClickHouse cluster.
Resolution steps:
- Verify macros.xml is missing on the affected worker node with the command. If macros.xml is not present, proceed to step 2:
ls -lah /etc/clickhouse-server/config.d/
-
Copy the macros.xml file from a healthy worker node (Wrk_1) to the affected worker node (Wrk_2).
Modify the contents of macros.xml in (Wrk_2) to avoid any similarity conflict.
vi /etc/clickhouse-server/config.d/macros.xml
Example:
<macros>
<shard>1</shard>
<replica>5</replica>
</macros>
-
Set the correct permissions:
chmod 0755 /etc/clickhouse-server/config.d/macros.xml
chown clickhouse:clickhouse /etc/clickhouse-server/config.d/macros.xml
-
Reboot the affected worker node (Wrk_2) and verify the ClickHouse status through:
phstatus
-
Once clickhouse-server is running, execute the cleanup script using the command:
/opt/phoenix/phscripts/clickhouse/cleanup_clickhouse.sh
After that, verify the clickhouse-server status is online again through the command phstatus.
Then, proceed to re-add the worker in the FortiSIEM GUI Clickhouse configuration.
Root Cause:
The macros.xml file is missing, deleted, or corrupted after a worker node failure or power cycle.
This file is mandatory for ClickHouse cluster operation, as it defines the shard and replica identifiers used by FortiSIEM distributed tables.
Related article:
Troubleshooting Tip: How to fix 'ClickHouse test error: ClickHouse Restart Failure' error in GUI
|
