FortiSOAR Knowledge Base
FortiSOAR: Security Orchestration and Response software provides innovative case management, automation, and orchestration. It pulls together all of an organization's tools, helps unify operations, and reduce alert fatigue, context switching, and the mean time to respond to incidents.
premchanderr
Staff
Staff
Article Id 377725
Description This article describes how to analyze what causes /var/lib/rabbitmq disk full in FortiSOAR.
Scope FortiSOAR v7.x.
Solution

RabbitMQ monitors the system's performance, tracks message flow, manages queues and exchanges, and troubleshoots issues.

 

In FortiSOAR the RabbitMQ goes high mainly due to the large queue size used by different services such as Falcon, celery, crudhub, etc. 

 

SSH to FortiSOAR and run the below commands to find which queue is taking more size:

 

# su root

# df -h | grep rabbitmq

 

Example:

 

 /dev/mapper/vgos-rabbitmq 14G 14G 444M 97% /var/lib/rabbitmq 

 

# sudo rabbitmqctl list_queues -p intra-cyops

 

Example:

 

intra-cyops.png

 

# sudo rabbitmqctl list_queues -p fsr-cluster

 

 fsr-cluster.png

 

If any high value is noticed for the above queues then contact Fortinet Support and consult on purging the queue temporarily.

 

Observe how long it takes for the disk utilization to spike again and does it happens gradually or suddenly by a particular playbook.

 

To resolve this, disable any suspicious or large playbook temporarily to see if the queue size stabilizes to narrow down the issue and tune the playbook. 

 

Note:

  • Always take a backup of files or screenshot of the VM before making any changes. Revert in case of an issue.
  • Do not directly delete the files unless instructed by Fortinet Engineers.