Technical Tip: How to estimate disk space needed for Archive and Analytics logs
Description
This article describes how to estimate the disk space needed for Archive and Analytics logs based on the number of retention days required.
Scope
FortiAnalyzer v5.4.x to 7.4.x.
Solution
When setting up disk space on FortiAnalyzer-VM, it is important to size the logical disk (LVM) properly. Even if increasing the size is possible and easy to perform (see the related article linked at the bottom of hte article), it is not possible to reduce the disk size. It is therefore advised to pick a suitable size when first setting up the FortiAnalyzer.
Hard disk settings information mentioned on VMWare ESXI is as follows:
FortiAnalyzer VM ESXI Administration Guide: Configuring hardware settings
Below is a formula to estimate the minimum disk/quota size required for retaining the logs and log databases:
HDD=LR*(RA/5 + 3*RR)*1.1
Where:
HDD - Approximate required total disk/quota size [GB].
LR - Average log rate [GB/day] - Take the average of the weekly log rate statistic under System Settings -> Dashboard -> 'License Information' widget -> GB/Day -> Details.
RA - Retention period for archive/raw logs [days].
RR - Retention period for reporting/analysis/realtime [days].
'5' - When the raw logs are archived, their file size is reduced approximately 5-8 times. 5 is a bit conservative and can be replaced with up to 8 for less strict retention policies.
'3' - Multiplier - When the raw logs are inserted, the SQL DB files are approximately 3 times bigger than the original log size.
'1.1' - 10% extra, as the disk space cannot be completely utilized. There is some space reserved for cache, temporary tables, etc.
For example:
- The FortiAnalyzer receives 8 GB per day on average.
- It is necessary to retain the reporting data for 90 days at a time.
- It is also necessary to store archived raw logs for a strict 365 days.
HDD = 8 * (365/5 + 3*90) * 1.1 = 8 * (73 + 270) * 1.1 = â€3018 GB
To allow for handling spikes in the log rate, it is always better to have more than the calculated minimum space.
If the Administrative Domain [ADOM] feature is enabled, it may make more sense to calculate how much disk space is needed per ADOM, so that disk space can be allocated correctly. Per-ADOM log rates are only visible via CLI:
diagnose fortilogd logvol-adom <adom_name>
The command above will print weekly statistics about the log rate in GB/Day for all ADOMs, as well as the average log rate.
Version 6.4.3 introduced SQL table compression, which reduces disk usage:
config system sql
set compress-table-min-age <----- Minimum age of the log tables in days.
end
The estimation formula does not consider this compression factor.
Related documents: