Description
This article describes the difference between Archive and Analytic logs, and how the latter are stored in a SQL database.
Scope
FortiAnalyzer.
Solution
Every FortiGate can send logs on either port 514 (TCP or UDP). The daemons that handle logs are 'miglogd' (FortiGate) and 'oftpd' (FortiAnalyzer).
Every log has log 'fields' such as the following:
- date.
- time.
- srcip.
- dstip.
- action.
- type, etc.
An example of 1 log would be:
Logs can be viewed in two different formats.
'Raw log' (text option).
'Formatted Log' (GUI option). Most preferred by almost all users due to being easier to read.
- The size of every FortiGate log will change depending on the size of each one.
- Some logs will be smaller/bigger than others.
- The size of every log can be estimated via a PCAP file. There are different packet sizes.
The limit of logs received per day is an important metric to check. This limit will depend on the Model or VM License.
FortiAnalyzer Archive Logs.
When FortiAnalyzer receives logs, those logs are stored as Archive logs, and when the active log rolls, the resulting logfile is compressed.
'Double click' in one packet of logs. Many logs in raw format will appear.
It is possible to define the size of the packet on the option 'Roll log file when size exceeds' located in System Settings.
FortiAnalyzer Analytic Database.
The analytic database is the place where logs are indexed from the Archive to the SQL database.
Notice that even though it is possible to see 196 days on the Analytics database, it does not indicate 196 days of daily logs, it means that the oldest log is from 196 days ago.
Normally, the oldest log is frequently to see it from Type: Event logs.
Also, consider Analytic database is where reports are generated.
Log Handling Flow:
The first metric to delete logs in any database is:
- Disk Space. It is necessary to have enough disk space to archive more days.
- Data Policy. If there is enough disk space, FortiAnalyzer will automatically delete the oldest logs.
Always check the current disk space to allow the data expected to be seen.
Otherwise, consider extending disk space if there VM or enable only policies that are most important to generate logs.
Consider that every FortiAnalyzer will depend on the Analytic Sustain Rate in both VMs and physical units.
Sustained Rate: Maximum constant log message rate that the FortiAnalyzer platform can maintain for a minimum of 48 hours without SQL database and system performance degradation.
- Consider that the Analytic Logs require up to 8 times more space than the Archive logs.
- For example, if Archive logs are 100 MB, the storage needed for the Analytic Database is up to 800 MB.
- One factor explaining why Archive logs take up less space is the compression. Analytics need to be readily accessed and so the logs are not compressed.
- Consequently, it is usually recommended that more disk space be assigned to Analytics than Archive when customizing the Log Storage Policy.
- The Analytic to Archive ratio is set to 70:30 % by default, and it can be modified.
- FortiAnalyzer has 'Analyzer Mode' and 'Collector Mode'.
- Collector mode does not have an Analytics database by default. Logs are forwarded to a unit in Analyzer mode, where Analytics are found.
- The Collector Sustained Rate is higher than in Analyzer mode.
- The mode of operation depends on the network topology and individual requirements.
Related articles:
Technical Note: How to set log retention values in FortiAnalyzer
Technical Tip: How to estimate disk space needed for Archive and Analytics logs
Technical Tip: FortiAnalyzer SQL database delete and rebuild
