FortiSOAR becomes unresponsive under heavy ingestion

[malayapanda]

In a #Fortinet #FortiSOAR multi-tenant environment, a single tenant's misconfigured Security Information and Event Management (#SIEM) detection rule can generate an excessive volume of events or alerts. This results in a substantial backlog of tasks within the #FortiSOAR task queue, leading to system unresponsiveness and impacting all tenants. While #FortiSOAR's pre-processing DROP rules mitigate the creation of new duplicate alerts, the existing task queue remains saturated, causing prolonged recovery times.

Proposed Solution and Inquiry:

To address this issue, implementing a per-tenant task queue architecture within #FortiSOAR. This would enable the selective purging of a specific tenant's task queue without affecting other tenants.

The primary questions are: @anarula

* Feasibility and Implementation: Is the implementation of per-tenant task queues technically feasible within the #FortiSOAR architecture?

* Scalability and Performance: Would this approach maintain system scalability and performance under high-load conditions, considering the potential increase in queue management overhead?

* Purging Mechanism: How can a robust tenant-specific task queue purging mechanism be implemented to ensure efficient and reliable remediation of backlog issues?"

[kaashif_m]

Hi Malaya,

I encountered the same issue in my environment and wanted to share the steps I took to resolve it.

In any SIEM ingestion process, after the alert creation step, there is a stage where related events are pulled into the incident. You can refer to the last step in the screenshot below.

These steps essentially involve database queries to the SIEM itself. Now, if 100 alerts come in and you send 100 requests to the SIEM database, it can become unresponsive, causing playbooks to remain stuck at the last step indefinitely.

Resolution Steps:

1. Remove the Link to the Last Step:

   • The final step only queries default information.
   • Instead of retrieving all related events for every alert, modify the incident response playbook to query only the necessary details at the time of incident creation.

2. Increase Compute Resources:

   • Upgrade the FortiSOAR server’s compute resources.
   • After increasing compute, raise a TAC ticket requesting fine-tuning of the node to ensure it utilizes the additional resources effectively.
   • Without this fine-tuning, the increased compute may not be fully utilized.

Hope this helps.

Best regards,
Kaashif Mohideen K

[malayapanda]

Dear @kaashif_m 

Thank you for your reply.

However, the issue is not searching co-related events from SIEM.

My above-described issue is for very initial ingestion playbook, and I am talking about 10000 alerts getting ingested in 1 hour, due to a misconfigured detection rule in SIEM / source.

Let's wait for Fortinet FortiSOAR engineering CTO response.

Regards

Malaya Manas

[kaashif_m]

Dear Malaya,

Even if 1,000 alerts are ingested:

• 1,000 playbooks will get stuck at the last step while retrieving related events.
• Since all playbooks have a default execution priority of medium, the SOAR system hangs.

At this point, you can observe the queue count either from the dashboard or via the CLI using the command:

rabbitmqctl list_queues -p fsr-cluster

The workaround I provided by removing the last step ensures that even if 1,000 alerts are ingested, your instance runs smoothly by removing the last step.

Regarding your question on implementing per-tenant task queues, I believe it is possible within a distributed tenancy model. However, licensing and cost implications need to be considered.

As you suggested, Anurla would be the best person to guide us on this.

Best regards,
Kaashif Mohideen K

[malayapanda]

Dear @kaashif_m 

I WANTED TO CLARIFY THAT THE ISSUE AT HAND IS NOT RELATED TO FORTISIEM OR SEARCHING EVENTS FOR AN ALERT.

UNFORTUNATELY, THE RESPONSES PROVIDED SO FAR HAVE NOT ADDRESSED THE CORE OF THE PROBLEM.

THANK YOU FOR YOUR UNDERSTANDING.

Thanks and Regards

Malaya Manas