Troubleshooting Tip: Upgrade stuck at ClickHouse database schema upgrade stage

Supervisor or worker node gets stuck while running the ClickHouse schema upgrade, as the following script would run and would go in loop if the/opt/phoenix/bin/clickhouse_config.db file is missing on the host that is being upgraded.

/opt/phoenix/deployment/clickhouse_upgrade/clickhouse_db_up_7.0.2_to_7.1.0.sh

Cause:

Follower supervisor or worker nodes where storage was manually formatted and mounted before GUI configuration do not complete automatic ClickHouse initialization. This results in missing /opt/phoenix/bin/clickhouse_config.db file, which stores the ClickHouse schema version needed for upgrades.

Solution:

1. Rollback using VM snapshot.
2. Copy /opt/phoenix/bin/clickhouse_config.db file from a working supervisor or worker node with the same version as the source version before the upgrade.
3. Ensure correct file permissions/ownership.

chown admin.admin /opt/phoenix/bin/clickhouse_config.db

chmod 755 /opt/phoenix/bin/clickhouse_config.db

4. Verify schema version:

 su -l admin -c '/usr/bin/sqlite3 /opt/phoenix/bin/clickhouse_config.db "SELECT * FROM configs"'

5. Retry the upgrade.