Incidents are not generated, but events continue to arrive.

[thiago_inorpel]

I noticed a strange behavior to say the least, everything was working normally until about 4 days ago the incidents that were seen regarding events in MEA collectors stopped without explanation, that is, the initial screen that counts the incidents was reset from 4 days ago. I didn't notice anything out of the ordinary, the health of the cluster is ok, the events that the collector forwards to the supervisor continue to arrive, but for some reason the incidents don't appear, this is what is most confusing. I have already tried stopping and starting the collector, but the problem remains, and I believe that perhaps the problem is not with the collector, considering that events arrive, but the rules no longer transform them into incidents. Could anyone help me troubleshoot? I remain at your disposal to provide any additional information. Thanks

[Secusaurus · Accepted solution]

Hi @thiago_inorpel,

Do you use ClickHouse? Do you have an all-in-one deployment or Supervisor and Workers in a cluster? Can you search through past events in Analytics or do you only (or don't) see events in realtime Analytics view?

As you stated, events are coming in in general. So, the issue probably is not on the collectors, but for storing or reading the events. Depending on the setup, you can continue by checking the table states of the workers (read only?) or looking in the phoenix log for backend errors.

Best,

Christian

[Secusaurus · Accepted solution]

Hi @thiago_inorpel,

Do you use ClickHouse? Do you have an all-in-one deployment or Supervisor and Workers in a cluster? Can you search through past events in Analytics or do you only (or don't) see events in realtime Analytics view?

As you stated, events are coming in in general. So, the issue probably is not on the collectors, but for storing or reading the events. Depending on the setup, you can continue by checking the table states of the workers (read only?) or looking in the phoenix log for backend errors.

Best,

Christian

[thiago_inorpel · Accepted solution]

Hello, I would like to inform you that this problem has been resolved. We noticed that we were experiencing an inconsistency in our hypervisor that hosts the supervisor's VM. For some reason, our backup agent had frozen the VM, which we were only able to identify after a reboot, in which the VM was no longer accessible. We were then able to access the hypervisor and bring up the VM again, and the incidents started to appear again. We use Clickhouse and our architecture is Workers in a cluster.
I really appreciate your interest and promptness in this @Secusaurus case! Thank you