Technical Tip: How to resolve if worker health showing critical due to clickhouse error table is in read only mode

In the logs, encountering errors like 'DB::Exception: Table is in read only mode since table metadata was not found in zookeeper' typically indicates issues related to table crashes in the database so this table can be restored. 

These logs can be found in: 'tail -f /opt/phoenix/log/phoenix.log'.

Step 1: Using the 'clickhouse-client' command helps to select all the tables and their status:

clickhouse-client -q "select database,table,is_readonly from system.replicas"

v7.2.2 and later versions:

Step 2: If the tables seem in read-only mode, this can be repaired using the format below.

Example:

TABLE_IS_READ_ONLY errors for 'fsiem.summary', 'fsiem.events_replicated', and 'fsiem.mv_t_events tables'.

Apply the steps below from Worker CLI for which node facing this issue.

clickhouse-client -q "SYSTEM RESTORE REPLICA fsiem.summary"

clickhouse-client -q "SYSTEM RESTORE REPLICA fsiem.events_replicated"

v7.2.2 and later versions: 

clickhouse-client -q "SYSTEM RESTORE REPLICA fsiem.summary"

clickhouse-client -q "SYSTEM RESTORE REPLICA fsiem.events_replicated"

clickhouse-client -q "SYSTEM RESTORE REPLICA fsiem.mv_t_events"

Step 3: After applying the restore, recheck using the command given in STEP 1 if the table is fixed. This is how a table in read-only mode can be restored.