FortiSIEM DB Migration from EventDB to ClickHouse

[beingarif]

We are planning to upgrade our FortiSIEM deployment from version 6.4.0 to 7.3.2, and as part of this upgrade, we are also looking to migrate the event storage backend from EventDB to ClickHouse.

As per the official documentation https://help.fortinet.com/fsiem/7-3-2/Online-Help/HTML5_Help/config-storage-changing-event-database...., there is a note stating:

"In all cases of changing storage type, the old event data is not migrated to the new storage. Contact FortiSIEM Support if this is needed - some special cases may be supported."

However, in the specific section "EventDB to ClickHouse (Single Node)", this limitation is not explicitly mentioned, which raises a question in our case.

We have a customer requirement to retain access to existing event data from the current EventDB even after migrating to ClickHouse.

Could you please confirm:

1. Is it possible to migrate historical event data from EventDB to ClickHouse during or after the upgrade?

2. If supported in special cases, what would be the required steps or conditions to enable such a migration?

We want to plan the upgrade carefully and ensure that the customer’s compliance and reporting needs are not impacted due to loss of historical data.

Looking forward to your guidance and confirmation.

[beingarif]

@Anthony_E  @Secusaurus can you please help here.

[Secusaurus]

Hi @beingarif,

The documentation is a little bit confusing here...

When you change the storage type (which you usually do on the GUI), no data is migrated. This is, what the quoted sentence means.

However, when going from EventDB to ClickHouse, you can use the `phClickHouseImport` command in the backend to manually import all EventDB-events to the ClickHouse data structure. Just follow the guide you mentioned.

In case you stay on the same VM for the migration, you can skip all the sync- and mounting-stuff, since the data still is accessible on that machine. Just make sure that you use a new disk for the Click-House-DB.

I can confirm that this works (although, it's a few versions away since I did that), but it takes an awful lot of time. Take a large maintenance window (1 day or more) for that one and test the import with smaller chunks first (since there is few feedback about the progress). It will have a lot of I/O on your disks!

Another idea would be to move all the "historical" data to the archive storage in advance (NFS), which reduces the amount of events you need to convert while keeping the compliance requirements.

Best,

Christian

[Anthony_E]

Thanks Christian!