FortiSIEM: Reducing Worker Node Disk Size – Feasibility & Impact

[beingarif]

Hi Everyone,

We have a FortiSIEM deployment where each worker node currently uses a single 20TB disk, but only around 6TB of data is actively in use. Due to infrastructure limitations especially long backup and restore times the customer is considering splitting the 20TB disk into 4x6/7 TB disks per worker node.

We’re looking for insights on:

1. Can this disk restructuring be done without service interruption or downtime?
2. What performance impact (if any) should we expect during or after the process?

Has anyone implemented something similar in their FortiSIEM environment? Any best practices, lessons learned, or documentation references would be greatly appreciated.
@Secusaurus @anthony_E can you please help here..

[Secusaurus]

Hi @beingarif,

I'll try to answer that, although it's more on the Hypervisor-side than on FSM.

In a usual deployment, the backup concept of Events would be to have multiple workers on the same Shard (= Replicas) which do have the identical data and jobs. In this scenario, you could simply switch off and restructure one of the Workers without any impact on the cluster (probably some performance impact, depending on the event throughput, but no downtime).

In case you only have one node and operate on the live system, you depend on the Hypervisor. I know that V-Sphere can do disk movements and restructuring without any impacts (it copies the data to the new system while keeping the old alive, and keeps everything in sync until it switches the machine to the new datastore). The impacts then depend on the Hypervisor.

In case you like to reduce the disk space on linux side first (within the VM), you will need to look up how to configure it independently from FSM. As a start, probably something like this could help: https://linuxconfig.org/how-to-manipulate-gpt-partition-tables-with-gdisk-and-sgdisk-on-linux

Then, have a look at the ClickHouse documentation, which might include some configuration notes you might need then: https://docs.fortinet.com/document/fortisiem/7.3.5/user-guide/846327/clickhouse-usage-notes

Best,

Christian

[Rob_SIEM]

Cleanest solution would be to spin up a new worker VM, add the smaller many vdisks. 4x6TB. Then add this new worker to the same shard as the other workers. Give it time to sync a copy of the data in the shard. Then retire one of the older worker nodes in that shard. Repeat for each worker to maintain redundancy. Ideally worker nodes are added in pairs. Each pair in a shard so data is replicated to two different workers. 

Workers 1-2: Shard1 (Worker1 on physical host 1, Worker 2 on physical host 2)

Workers 3-4: Shard2 (Worker3 on physical host 1, Worker 4 on physical host 2)

Remember, every worker in a shard gets the exact same copy of data. So if you have 3 workers in shard1, each will have the exact same copy of data (3 replicas).

If you have many shards, event ingestion is spread evenly across the shards. Data is not balanced, so you have to keep a close eye on if drives in any one shard are filling up, if so you must add disks for those nodes. If the worker nodes in shard1 are at 80% capacity, add 20% more capacity to both workers in that shard.