Troubleshooting Tip: How to remove FortiSIEM corrupted indices

[calvinc97]

Description	This article describes how to safely remove FortiSIEM corrupted indices error found in backend.
Scope	FortiSIEM.
Solution	The possible causes of index corruption can be derived from the below: Unclean shutdown of phDataManager: If the service terminates unexpectedly, it may leave indices in an inconsistent state. Disk read/write operation failures: Issues with disk I/O during index creation or updates can lead to corruption. High disk latency – Slow disk response times may result in incomplete writes, causing index inconsistencies.   Prevention Measures: Monitor disk performance: Use command #iostat to check for high disk utilization or latency issues. Ensure sufficient disk space: Running out of disk space can lead to failures in indexing operations. Gracefully stop phDataManager process: Always use proper shutdown procedures for phDataManager to prevent abrupt interruptions. Corrupted indices can be observed in the phoenix.log of FortiSIEM nodes below: [PH_DATAMANAGER_EVTIDX_WRITE_POST_ERROR]:[eventSeverity]=PHL_ERROR,[procName]=phDataManager,[fileName]=EventIndexFile.cpp,[lineNumber]=548,[fileName]=/data/eventdb/CUSTOMER_1/internal/20160/483840-483863-168438285/tmpseg-5-8447-2762764558-1741824000-1741910399/index/110n.pst,[errReason]=Current posting is less than last posting,[phLogDetail]=Event index file write posting error The appropriate procedure is to safely remove the corrupt index without deleting the the entire date directory (e.g., /data/eventdb/CUSTOMER_1/internal/20160/) by running the following command below: # rm -rf /data/eventdb/CUSTOMER_1/internal/20160/483840-483863-168438285/tmpseg-5-8447-2762764558-1741824000-1741910399/index/ Restart the phDataManager process: # phtools --stop phDataManager # phtools --start phDataManager After restarting the phDataManager process, FortiSIEM will automatically regenerate the missing index located in: /data/eventdb/CUSTOMER_1/internal/20160/483840-483863-168438285/tmpseg-5-8447-2762764558-1741824000-1741910399/ Using this approach would ensure minimal data loss and avoid unnecessary deletion of valid event logs.