|
While checking on Supervisor Jobs/Errors or health page, the following error is observed:
phIpIdentityWorker unable to keep up with incoming event rate
This error suggests that the phIpIdentityWorker process on any one node is lagging behind. This can be confirmed by browsing to Admin -> Health -> Cloud health and drilling down to the processes of each node, comparing reader and writer positions for each. The node which has this issue will have discrepancies between the reader and writer positions for the phIpIdentityWorker process.
The following errors are observed in relation to the phIpIdentityMaster process on the Supervisor:
2025-08-08T13:07:01.999034+02:00 Supervisor phIpIdentityMaster[1165020]: [PH_GENERIC_ERROR]:[eventSeverity]=PHL_ERROR,[procName]=phIpIdentityMaster,[fileName]=phNotificationServer.cpp,[lineNumber]=94,[phLogDetail]=phSSLAcceptor::accept error with errno : 336462231, duration 4 ms
2025-08-08T13:07:20.617314+02:00 Supervisor phIpIdentityMaster[1165020]: ACE_SSL (1165020|1165240) ecode: 336462231 - 140E0197:SSL routines:SSL_shutdown:shutdown while in init
The following errors are observed on the Worker:
2025-08-08T13:24:57.475075+02:00 Worker1 phIpIdentityWorker[393086]: [PH_GENERIC_ERROR]:[eventSeverity]=PHL_ERROR,[procName]=phIpIdentityWorker,[fileName]=phMiscUtils.cpp,[lineNumber]=4615,[phLogDetail]=Failed to sent data to master at: 10.10.10.10 port: 7938
2025-08-08T13:24:57.475496+02:00 Worker1 phIpIdentityWorker[393086]: [PH_IDENTITYWORKER_EVT_SEND_ERROR]:[eventSeverity]=PHL_ERROR,[procName]=phIpIdentityWorker,[fileName]=phIpIdentityWorker.cpp,[lineNumber]=971,[sentBytes64]=5601,[errReason]=Failed to send slow sync data. Retry.,[phLogDetail]=Failed to send event to identity master
2025-08-08T13:24:59.474968+02:00 Worker1 phIpIdentityWorker[393086]: ACE_SSL (393086|393086) ecode: 336462231 - 140E0197:SSL routines:SSL_shutdown:shutdown while in init
Check if port 7938 communication is healthy from the worker to the Supervisor.
On the Worker, check if any ini.err files exists in the /opt/phoenix/cache/parser/upload/svn directory.
ls -la /opt/phoenix/cache/parser/upload/svn | grep .err
Remove any ini.err files if they exist.
cd /opt/phoenix/cache/parser/upload/svn
rm -rf *.err
Restart phIpIdentityWorker on the Worker:
killall -9 phIpIdentityWorker
Restart processes on the Supervisor:
killall -9 phIpIdentityWorker
killall -9 phIpIdentityMaster
Check again in the backend logs to see if the error exists.
|
