Troubleshooting Tip: Historical Logs in FortiAnalyzer are lost

[mrafat]

Description

This article describes how to resolve the loss of historical logs on Fortianalyzer due to ADOM Quota over limit.

Scope

FortiAnalyzer.

Investigation:

1. Verify the ADOM Quota utilization by going to System Settings -> Storage Info -> select the ADOM that has the device. The below window will show up the Analytics & Archive utilization in addition to the Data Policy.

2. The above can also be observed through the FortiAnalyzer CLI by issuing the command:
   
   

diagnose log device

Notice the 'used%' for both Analytics and Archive if it reaches 85% or above.

3. In this scenario, the FortiAnalyzer will start deleting old logs to free up space in the allocated ADOM storage so that it can receive the new logs and that can result in unnecessary CPU resources enforcing Quota with log deletion and database trims.

4. Verify the log rate received on the FortiAnalyzer by issuing the below command:
   
   diagnose fortilogd lograte <-- Monitoring the log rate/sec on FortiAnalyzer.
   

   last 5 seconds: 2329.6, last 30 seconds: 2300.9, last 60 seconds: 2283.4.

5. Verify the lograte per device to check which device is sending a huge amount of logs that consume high disk space by issuing the below command on FortiAnalyzer CLI:

   diag fortilogd lograte-device

   Logs per second

   Totals            Last Hour       Day      Week

        -------------------------------------------------

       Device1_Serial:      200.00      5.30      2.10

       Device2_Serial:      1136.77     72.09     10.30
   
   

6. If the allocated storage for Analytics/Archive was found not to be highly utilized, however, it still displaying that historical logs are being lost, it is necessary to verify the Device log settings and make sure in (The automatically Deleted section) that the highlighted section is not set to a value less than what having configured per ADOM.
   
   

Solutions.

To prevent the loss of Historical logs due to limited ADOM Quota, follow the below steps:

1. If there is enough storage in the Hard-disk assign an extra Quota to the ADOMs in the highlighted below section:
   
   

2. In the case of FortiAnalyzer -VM only, if there is not enough hard disk to be assigned to the ADOM, extend the hard disk of the unit.

   Check Related Articles for the KB of Hard-disk expansion

3. To know the estimated hard disk needed based on the log rate received and accordingly upgrade the hard disk.

   Check Related Articles for the KB of Formula to estimate the amount of Hard-disk needed

4. If there is not enough space and want to reduce the logging of a specific device that sends a huge amount of logs to FortiAnalyzer.

   Check Related Articles for the KB of minimizing FGT logging on FortiAnalyzer

Related Articles:

• Extending FAZ VM Hard-disk:

Technical Tip: Extending disk space in FortiAnalyzer-VM/FortiManager-VM

• Formula for Calculating the estimated Hard-disk:

Technical Tip: How to estimate disk space needed for Archive and Analytics logs

• Minimizing FortiGate logging on FortiAnalyzer:

Technical Tip: Minimizing logging from FortiGate to FortiAnalyzer