Technical Note: CSTN 00032 - Redis failure on your FortiSOAR instance

Issue

You experience a Redis failure on your FortiSOAR™ instance due to high rate of playbooks execution.

Description

When the playbooks execution rate is very high on your FortiSOAR instance, for example, if 100 playbooks complete their execution in FortiSOAR in 10 seconds, and this rate of execution continues for several hours, you might face a Redis error as shown in the following image:

Root Cause

You face this issue because the Redis cache gets full due to the high rate of playbooks execution.

Resolution

1. Flush the Redis database using the # redis-cli -n 11 -a <device_uuid> flushdb command, 
   where you can retrieve the device_uuid using the csadm license --get-device-uuid command.
2. Set the value of the CELERY_TASK_RESULT_EXPIRES parameter to 1200 (this value is in seconds) in the celeryd section of the config.ini file located at /opt/cyops-workflow/sealab/sealab/ as follows:
   CELERY_TASK_RESULT_EXPIRES:1200
   Ensure that the value of the CELERY_TASK_RESULT_EXPIRES parameter is greater than the standard soft-limit time for playbook execution, which is 600 seconds.
3. Restart the uwsgi, celerybeatd, and celeryd services.

Note: In case you have a high rate of playbooks execution, it is recommended that you set the value of CELERY_TASK_RESULT_EXPIRES parameter to 1200 before the playbook execution starts, to avoid Redis failure.

Applicable Version:

5.0.0 and later