Technical Tip: How to troubleshoot Redis memory issues

[mbenvenuti]

Description	This article describes how to troubleshoot Redis memory issues.
Scope	FortiSIEM.
Solution	Next troubleshooting steps can be followed when one of the following behaviors is met: In Analytics, a query filtering on a group does not show the result of the item in the group. Incidents with a rule filter with the group do not trigger for elements in the group. Error while navigating through the GUI: 'IllegalArgumentException: CMDB report: invalid field src_name'. Check for Out-of-memory errors in the application server logs. As the Redis cache is fed by the application server, errors regarding the Redis memory will be rejected in its logs. From super CLI as root, run: grep OOM /opt/glassfish/domains/domain1/logs/server.log* If there is an output like: 'Caused by: redis.clients.jedis.exceptions.JedisDataException: OOM command not allowed when used memory > 'maxmemory'' It means that Redis memory needs to be increased. See section 3. It is possible that the error is not seen because logs are rotating too fast, check for Redis memory usage in section 2. Check for Redis memory usage. Check for Redis memory info with the next command from super CLI as root: redis-cli -p 6666 -a `phLicenseTool --showRedisPassword` -c info memory Check for the conditions that show a lack of memory: used_memory_peak_human > maxmemory_human used_memory_dataset_perc = 100% If one of those conditions is met, increase Redis memory by following section 3. Increase Redis memory. To identify the new memory limit to set, choose one of the next values where it might take used_memory_peak value + 1 or 2GB to make sure data will fit inside: 4294967296 6442450944 8589934592  Then from super and all the worker nodes, run the next command from CLI as root: new_mem_val=6442450944 <--Replace by the value you previously identified. cp /opt/phoenix/redis/conf/6666.conf /opt/phoenix/redis/conf/6666.conf_bak sed -i "s/maxmemory [0-9]*/maxmemory $new_mem_val/" /opt/phoenix/redis/conf/6666.conf sed -i "s/maxmemory [0-9]*/maxmemory $new_mem_val/" /opt/phoenix/redis/conf/conf.tmpl /opt/phoenix/redis/bin/redis_ops.sh stop /opt/phoenix/redis/bin/redis_ops.sh start kill -9 $(cat /opt/glassfish/domains/domain1/config/pid) <-- To run only on super node. Connect to the GUI and check if the issue is fixed. Warning: Be aware that this parameter may be reset during upgrades.