Description
This article describes the difference between Archive and Analytic logs, and how the latter are stored in a SQL database.
Scope
FortiAnalyzer.
Solution
Every FortiGate can send logs on either port 514 (TCP or UDP).
The daemons that handle logs are 'miglogd' (FortiGate) and 'oftpd' (FortiAnalyzer).
Every log has log 'fields' such as the following:
- date.
- time.
- srcip.
- dstip.
- action.
- type.
...
An example of 1 log, would be:
Logs can be viewed it in two different formats.
'Raw log' (text option).
'Formatted Log' (GUI option). Most preferred by almost all users due to being easier to read.
- The size of every log FortiGate will change depending on the size of each one.
- There are some logs that will be smaller/bigger than others.
- The size of every log can be estimated via pcap file. There are different packet sizes.
The limit of logs received per day is an important metric to check.
This limit will depend on the Model or VM License.
Fortianalyzer Archive Logs.
When Fortianalyzer receives logs, those logs are stored as Archive logs, and when the active log rolls, the resulting logfile is compressed.
'Double click' in one packet of logs. Many logs in raw format will appear.
It is possible to define the size of the packet on the option 'Roll log file when size exceeds' located in System Settings.
Fortianalyzer Analytic Database.
The analytic database is the place where logs are indexed from the Archive to the SQL database.
Notice that even though it is possible to see 196 days on the Analytics database, it does not indicate 196 days of daily logs, it means that the oldest log is from 196 days ago.
Normally the oldest log is frequently to see it from Type: Event logs.
Also, consider Analytic database is where reports are generated.
Log Handling Flow:
The first metric to delete logs in any database is:
- Disk Space. It is necessary to have enough disk space to archive more days.
- Data Policy. If there is enough disk space FortiAnalyzer will automatically delete the oldest logs.
Always check the current disk space is allowing the dats expected to be seen.
Otherwise, consider extending disk space if there VM or enable only policies which are most important to generate logs.
Consider that every FortiAnalyzer will depend on the Analytic Sustain Rate in both VMs and physical units.
Sustained Rate - maximum constant log message rate that the FortiAnaylzer platform can maintain for a minimum of 48 hours without SQL database and system performance degradation.
- Consider that the size of Archive Logs: Analytic Logs is 1:4 or even 1:8.#
- This means that if Archive logs are 100Mb, it is necessary to have an Analytic Database of 400Mb, or even 800Mb.
- One factor explaining why Archive logs take up less space is the compression used. Analytics need to be readily accessed and so are not compressed.
- Consequently, it is usually recommended that more disk space is assigned to Analytics than Archive when customizing the Log Storage Policy.
- The ratio of Analytic to Archive is normally 80%:20%.
- FortiAnalyzer have 'Analyzer Mode' and 'Collector Mode'.
- Collector mode does not have an Analytic database by default. Logs are forwarded to a unit in Analyzer mode where Analytics are found.
- The Collector Sustained Rate is higher than in Analyzer mode.
- The mode of operation depends on the network topology and individual requirements.
Related Article:
Technical Note: How to set log retention values in FortiAnalyzer
Technical Tip: How to estimate disk space needed for Archive and Analytics logs
