Troubleshooting Tip: How to fix the gateway queue message consumption issues

Description

This article describes how to resolve an issue where RabbitMQ disk space continues to increase because gateway messages are not being consumed by the system.

Scope

FortiSOAR v7.5.2 or later (v7.5.x), or v7.6.1 or later (v7.6.x).

Solution

If the RabbitMQ partition '/var/lib/rabbitmq' shows high disk usage that continues to grow, follow the steps below to identify and resolve the issue:

1. Check the RabbitMQ queue:

rabbitmqctl list_queues -p intra-cyops

Review the message count for 'cyops.gateway.crudhub.data'.

If the number of messages is very high, proceed with the next steps.

2. Verify the audit logs:

/var/log/cyops-gateway/gateway.log

In the event of an error similar to the one below, it indicates that the message size exceeds the configured limit:

java.lang.IllegalStateException: Message body is too large (69320449), maximum configured size is 67108864

Perform the steps below to fix the issue:

1. Backup the configuration file:

cp -p /opt/cyops/configs/cyops-gateway/gateway.properties /opt/cyops/configs/cyops-gateway/gateway.properties_bkp 

2. Edit the configuration file to increase the size limit (default is 64).

vi /opt/cyops/configs/cyops-gateway/gateway.properties

• Add or modify the following line.

max_inbound_message_body_size=128

3. Save the file and restart the service:

systemctl restart cyops-tomcat

4. Validate the fix:

Recheck the queue. The message count should begin to decrease gradually as messages are consumed, eventually freeing up RabbitMQ disk space.

Note: This guide cannot be applied before FortiSOAR v7.5.2 or v7.6.1, since the parameter max_inbound_message_body_size was not introduced.