FortiSIEM: Find out triggering device for "PH_RULE_NO_PERFMON_VIA_AO" incidents?

[Secusaurus]

Hello everyone,

We are continuously experiencing the incident "High performance monitoring delay from Collector or Worker SIEM Supervisor" on our FortiSIEM platform. That one is triggered as soon as the Event Type "PH_DEV_MON_PERFMON_ALL_DEVICE_DELAY_HIGH" appears at least once. This event is also happening, so it seems absolutely correct.

Our health status, however is as green as it can be. No collector, worker, agent or supervisor has any issues, services down or delays.

Has anybody had a similar situation and/or an idea how to find out more context around the event?

Raw Event:

[PH_DEV_MON_PERFMON_ALL_DEVICE_DELAY_HIGH]:[phCustId]=1,[eventSeverity]=PHL_INFO,[phEventCategory]=3,[procName]=AppServer,[relayDevName]=SIEM Supervisor,[relayDevIpAddr]=(Supervisor IP),[phLogDetail]=Performance monitoring delay for all devices from a collection point crossed high water mark

Thanks already for your input!

Best,

Christian

[premchanderr]

Hi @Secusaurus ,

This event log may result from network, CPU or NTP issues causing delays in performance checks. The incident will only be resolved upon receiving the Event Type -PH_DEV_MON_PERFMON_ALL_DEVICE_DELAY_LOW.

Most of time if its minor issue within few minutes incident is cleared automatically.

If you wish to modify this, you can edit the rule, adjusting the clear condition to either a different criterion or a 10-minute interval in case the rule isn't triggered again. Additionally, you have the option to disable the ORG rule.

[Secusaurus]

Hi Prem,

Thanks for your input.

Main issue is: The Incident triggers since more than a week every five minutes. So there seems to be something wrong, but I cannot narrow it down with "Collector or Worker SIEM Supervisor", which could be just any member of the cluster.

And just disabling the rule would not be my preferred solution, if there really was an issue here.

Anyways, I talked to the TAC recently and got the hint for digging deeper in the phoenix logs (tail -f /opt/glas*/dom*/dom*/logs/phoenix.log), which I will have a look at now.

Best,

Christian

[premchanderr]

HI Christian,
You are welcome.

Yes analyzing the log would help if any process is high or errors at that time of incident. Also note even if network delays for few minutes can cause this.