Enhancing Playbooks Performance queuing through equitable schedule distribution

[akashj]

Hi @Anonymous 

Recently, we experienced performance issues with our SOAR playbook schedules. We had over 250 Schedules running, 60+ of which were ingestion playbooks. The current difficulty I was experiencing was that our entire schedule was running every 5 minutes. Example: 01:00, 01:05, 00:10, etc. Because all of the playbooks were executing at the same time, the playbook queue size ballooned for few mins and the UI became slow as a result of excessive API calls in a short period of time.

To resolve this, I attempted one solution inspired by the Splunk scheduler: https://docs.splunk.com/Documentation/SplunkCloud/9.3.2411/Alert/CronExpressions.

I divided the schedules into different minutes of the hour. For example, 10 ingestion playbooks run at 1:02, followed by another 10 at 01:03, and so on.

Schedule skewing should only be used when running a large number of Playbooks at once.

Don't apply skewing until absolutely necessary.

See the attached screenshots for further information.