|
The possible causes of index corruption can be derived from the below:
- Unclean shutdown of phDataManager: If the service terminates unexpectedly, it may leave indices in an inconsistent state.
- Disk read/write operation failures: Issues with disk I/O during index creation or updates can lead to corruption.
- High disk latency – Slow disk response times may result in incomplete writes, causing index inconsistencies.
Prevention Measures:
- Monitor disk performance: Use command #iostat to check for high disk utilization or latency issues.
- Ensure sufficient disk space: Running out of disk space can lead to failures in indexing operations.
- Gracefully stop phDataManager process: Always use proper shutdown procedures for phDataManager to prevent abrupt interruptions.
Corrupted indices can be observed in the phoenix.log of FortiSIEM nodes below:
[PH_DATAMANAGER_EVTIDX_WRITE_POST_ERROR]:[eventSeverity]=PHL_ERROR,[procName]=phDataManager,[fileName]=EventIndexFile.cpp,[lineNumber]=548,[fileName]=/data/eventdb/CUSTOMER_1/internal/20160/483840-483863-168438285/tmpseg-5-8447-2762764558-1741824000-1741910399/index/110n.pst,[errReason]=Current posting is less than last posting,[phLogDetail]=Event index file write posting error
The appropriate procedure is to safely remove the corrupt index without deleting the the entire date directory (e.g., /data/eventdb/CUSTOMER_1/internal/20160/) by running the following command below:
# rm -rf /data/eventdb/CUSTOMER_1/internal/20160/483840-483863-168438285/tmpseg-5-8447-2762764558-1741824000-1741910399/index/
Restart the phDataManager process:
# phtools --stop phDataManager
# phtools --start phDataManager
After restarting the phDataManager process, FortiSIEM will automatically regenerate the missing index located in:
/data/eventdb/CUSTOMER_1/internal/20160/483840-483863-168438285/tmpseg-5-8447-2762764558-1741824000-1741910399/
Using this approach would ensure minimal data loss and avoid unnecessary deletion of valid event logs.
|
