Description
This article describes how to troubleshoot an issue where the insert rate of FortiAnalyzer is not running, even after rebuilding the database.
Scope
FortiAnalyzer.
Solution
Usually, this issue happens after a FortiAnalyzer upgrade, and rebuilding the database is recommended to resolve it. But sometimes, after rebuilding the database and if the issue persists, here are the recommended steps to check and perform:
- Confirm the log rate and insert rate in CLI command: Log Rate (to confirm the raw logs are coming from devices (FortiGate, FortiWeb..):
diagnose fortilogd lograte
diagnose fortilogd lograte-adom <adom_name>
diagnose fortilogd lograte-device <device_name>
diagnose fortilogd lograte-total
Insert Rate (logs are indexed by SQL database):
diagnose test application sqlplugind 2
PID: 2017, now: 1747300163, uptime: 1230107
Thread registered: 2
DB-age: 451253147, x-rate: 0/s, xid-rate: 1/s, xid-vacuum: 0, nice-yield: 0, io-utils: 0%
Log insert speed: logs/5sec: 0.0, logs/60sec: 0.0 <----------- No logs insert to SQL database
Log stats: log-recv=0 log-insert=2089775216 bat=97858 avg-bat-sz=21355 ack=0 ack-drop=0 ack-err=0 bat-recv=0 misc-recv=0
- Check if there are any event logs related to 'siemdbd suspended', from the GUI or CLI:
diagnose test application fazcfgd 6 stat 100
Emergency/Critical elog:
175183:2025-05-01 11:28:15 tz="+0800" log_id=0028037005 type=event subtype=fazsys pri=emergency desc="FortiAnalyzer daemon suspended" user="system" userfrom="system" msg="fortilogd stopped receiving logs due to storage space at critically low level (87.60 GB free)." operation="Application suspend" performed_on="" changes="fortilogd stopped receiving logs." action="suspend"
175224:2025-05-01 11:41:17 tz="+0800" log_id=0028037005 type=event subtype=fazsys pri=emergency desc="FortiAnalyzer daemon suspended" user="system" userfrom="system" msg="siemdbd suspended due to disk full." operation="Application suspend" performed_on="" changes="siemdbd stopped inserting logs." action="suspend"
.......
.......
lograte: 1168 db insert-rate: 0.0
- Check if ADOM allocation storage is not exceeded the available quota on Gthe UI or CLI:
diagnose log device
.....
....
Total Quota Summary:
*** Warning: Total Allocated Quota is bigger than Total Quota! Please check the quota configuration of ADOMs!
Total Quota Allocated Available Allocate%
864.7GB 970.0GB 0.0KB 112.2%
System Storage Summary:
Total Used Available Use%
1078.7GB 889.3GB 189.3GB 82.4%
Reserved space: 214.0GB (19.8% of total space, system reserved: 107.0GB, Fabric Storage pool disk quota: 107.0GB).
For this example, ADOM allocation storage has been allocated to 970 GB, which causes the insert rate to run properly. It should not exceed 864 GB from the total quota.
After changing the ADOM storage to the expected quota, perform below command to restart again sqlplugind and siembd daemons:
diagnose test application sqlplugind 99
diagnose test application siemdbd 99
Related documents:
Technical Tip: Explanations about log rate/log insert speed info in some FortiAnalyzer CLI commands
Technical Tip: How to check FortiAnalyzer log rate
