Technical Tip: How to recover ignored events from ClickHouse
| Description | This article describes how to recover ignored events from Clickhouse. |
| Scope | FortiSIEM. |
| Solution | When FortiSIEM is configured with ClickHouse storage, it happens that some data is ignored in its cluster management because it cannot insert it in the local replication. As a result, some expected events might be missing from the result queries. Here are the steps to follow to recover ignored data:
From the GUI at Admin -> Settings -> ClickHouse Config, identify all the data nodes:
Connect on each data node from CLI as root and run:
du -h --max-depth=1 /data-clickhouse-*/clickhouse/data/fsiem/events_replicated/detached/ | sort -hr
Check and identify the part to recover along the size and the date(20240621) of data treated.
In the GUI from the Analytics tab, run a query for the chosen date (example 20240621) starting from 00:00:00 till 23:59:59 and count the number of events :
For example, file ignored_18250-20240621_0_6968_2577 is chosen to be recovered for the next steps. From data node CLI as root:
cd /data-clickhouse-hot-1/clickhouse/data/fsiem/events_replicated/detached/ mv ignored_18250-20240621_0_6968_2577 18250-20240621_0_6968_2577 clickhouse-client -q "alter table fsiem.events_replicated attach part '18250-20240621_0_6968_2577'"
Renew those operations on all the ignored parts and data nodes required.
In GUI from the Analytics tab, run the same query as before and compare:
|
