Troubleshooting Tip: How to resolve the phAnomalyWorker continuously getting restarted on the worker after installation

It has been observed that the phAnomalyWorker process continuously gets restarted on the worker after a fresh installation.

The following errors have been observed in /opt/phoenix/log/phoenix.log of the worker.

2026-03-05T14:20:39.579803+05:30 Worker02 [PH_AUDIT_ML_GENERIC_CRITICAL]:[eventSeverity]=CRITICAL,[task_id]=main,[task_name]=main,[procDetails]={Frequent restarts detected: RestartFreqExceeded('5 in 1s')}#012Traceback (most recent call last):#012 File "celery/worker/consumer/consumer.py", line 330, in start#012 self._restart_state.step()#012 File "billiard/common.py", line 152, in step#012billiard.exceptions.RestartFreqExceeded: 5 in 1s
2026-03-05T14:20:40.581329+05:30 Worker02 [PH_AUDIT_ML_GENERIC_WARNING]:[eventSeverity]=WARNING,[task_id]=main,[task_name]=main,[procDetails]={celery/worker/consumer/consumer.py:508: CPendingDeprecationWarning: The broker_connection_retry configuration setting will no longer determine#012whether broker connection retries are made during startup in Celery 6.0 and above.#012If you wish to retain the existing behavior for retrying connections on startup,#012you should set broker_connection_retry_on_startup to True.#012 warnings.warn(#012}
2026-03-05T14:20:40.584067+05:30 Worker02 [PH_AUDIT_ML_GENERIC_ERROR]:[eventSeverity]=ERROR,[task_id]=main,[task_name]=main,[procDetails]={consumer: Cannot connect to redis://default:**@dbleader.fsiem.fortinet.com:6666/1: You can't write against a read only replica..#012Trying to reconnect...#012}

It means the worker is unable to reach the supervisor using the hostname dbleader.fsiem.fortinet.com.

Check the entry for host dbleader.fsiem.fortinet.com in the/etc/hosts file on the worker.

Make sure that dbleader.fsiem.fortinet.com is resolved to the Supervisor's IP and not the worker's IP.

Check if there are interruptions for communication due to the network from the worker to the supervisor node using hostname dbleader.fsiem.fortinet.com over TCP port 6666.