Troubleshooting Tip: Resolving ClickHouse Error 'Table is not initialized yet (NOT_INITIALIZED)' in FortiSIEM

A typical error observed in FortiSIEM logs:

grep -i "NOT_INITIALIZED" /opt/clickhouse/log/clickhouse-server.log

2026-02-02T19:40:54.309639-05:00 worker-01 phDataManager[9265]: [PH_DATAMANAGER_SUMMARYWRITER_ERROR]:[eventSeverity]=PHL_ERROR,[procName]=phDataManager,[fileName]=SummEventWriter.cpp,[lineNumber]=1001,[errorNo]=500,[errReason]=Uploading events to ClickHouse failed. respCode:500 resp:Code: 667. DB::Exception: Table is not initialized yet. (NOT_INITIALIZED) (version 23.8.16.40 (official build))

Additional symptoms may include:

• Events failing to ingest.

• Backlog growth.

• HTTP 500 errors from ClickHouse.

• Tables present in metadata but rejecting inserts.

ClickHouse tables exist in metadata but have not completed initialization.
This typically occurs when ClickHouse Keeper quorum is unstable or unavailable.

Common contributing factors:

• ClickHouse Keeper node is unstable or down.

• Keeper quorum cannot be reached.

• Replicated tables cannot fetch metadata.

• Keeper service repeatedly restarting on one node.

• Replicas stuck in NOT_INITIALIED state.

As long as a table remains uninitialized, ClickHouse rejects all inserts, causing FortiSIEM ingestion failures.

Initial Investigation:

1. Identify NOT_INITIALIZED errors.

grep -i "NOT_INITIALIZED" /opt/clickhouse/log/clickhouse-server.err.log

2. Verify Table State from ClickHouse Client.

clickhouse-client
SHOW DATABASES;

┌─name───────────────┐
│ INFORMATION_SCHEMA │
│ default │
│ fsiem │
│ information_schema │
│ system │
└────────────────────┘

3. Check replica status from the client:

SELECT * from system.replicas format Vertical;

4. Verify ClickHouse Keeper Quorum:

echo stat | nc <keeper_ip> 2181

If quorum is lost, attempt recovery:

echo rcvr | nc localhost 2181 

5. Check and Restart Keeper Service:

systemctl status ClickHouseKeeper

systemctl restart ClickHouseKeeper 

6. Force Keeper Recovery (if Service Fails to Start): 

If the Keeper service fails to start on a critical node:

• Update the following file:

/data-clickhouse-hot-1/clickhouse-keeper/.systemd_argconf 

• Ensure it contains 'ARG1=--force-recovery'.

• Restart the service:

systemctl restart ClickHouseKeeper

7. Restart Replicas:

Replica restart may fail initially, but it is still required. Run the following:

clickhouse-client --query "SYSTEM RESTART REPLICAS"

clickhouse-client --query "SYSTEM RESTORE REPLICA fsiem.events_replicated"

Advanced Recovery (Upgrade / Persistent Failure Case):

In some upgrade scenarios, the issue persists even after taking the steps above.
This is commonly caused by a faulty Keeper node flapping repeatedly.

Resolution:

Remove the problematic Keeper node from the cluster. Select Test and Deploy.

On the affected node, run the following cleanup script:

• Re-add the Keeper node to the ClickHouse cluster.

• Save and deploy the ClickHouse configuration.

Key takeaways.

• This is a ClickHouse / Keeper issue, not a FortiSIEM configuration issue.

• Restarting services alone is often not sufficient.

• A faulty Keeper node must be cleaned up and re-added.

• Once Keeper quorum is restored, FortiSIEM event ingestion recovers automatically.