We have a FAZ VM with the 2G/Day/1TB Storage license. We noted that our daily log flows were very flat and topping out at 130Logs/Sec. Rarely if ever exceeding that amount. The current FAZ we have had been upgraded several times and we export all data it collects to Splunk for easier searching and correlation with other event sources.
So we downloaded a 6.2.3 FAZ VM OVF yesterday. We are logging from a HA pair in A/P mode of 501Es. They have a 1G connection to the Internet and 10G connection to our core. We spun up the VM and ran into the old miglogd issue. For whatever reason on our 6.0.5 FG build, if we make any changes to the log settings or disrupt the flow of logs (by rebooting the FAZ for example), they stall until we console into the Fortigate and do the following:
diag sys top-summary
View the output and locate the PID for miglogd (it is often a 5 digit number such as 11035, etc.) Then we have to kill the process:
diag sys kill 11 11035
Logs will then immediately startup again to the FAZ.
Anyway, while the new FAZ VM was running unlicensed (we configured it with the same vCPU and ram as the old one), we noticed the log flow rate peaked at 4000/second and stabilized around 1200 to 1500. So we set the management IP on the new FAZ to be the same as our old FAZ and applied our FAZ license. It rebooted and everything was happy, EXCEPT, log flow rate does NOT exceed 130 Logs/s. Why is it limiting? Is it dropping traffic or will it buffer and only injest at a max rate determined by the license level? I've been told it doesn't limit but it sure seems to be.
See the attached image that shows the peak flow and it should be obvious when we applied the license.
There is a known issue for clusterd using high CPU utilization in FortiAnalyzer 6.2.3. Check "exec top" on your FAZ to see whether that is the case (this issue can occur for standalone FAZ as well). If so, there is a special build support can provide you.
Otherwise, perhaps you could specify exactly what resources you have allocated to the VM. Or just go ahead and open a support ticket.
> I've been told it doesn't limit but it sure seems to be.
At this time, there is no intentional hard limit enforced on the logs received or forwarded by FortiAnalyzer. The license limit is a daily limit (GB/day) and you will see event logs if your sustained log rate exceeds that which would eventually cause the daily limit to be reached. In most cases, any limitation on log processing should be resource-related.
I would still advise you to open a support ticket.
We are continuing to evaluate. We added more vCPU, now 8, and left RAM at 16. According to docs this should allow 6000 lps or more. We were seeing days of exactly 30 minutes from traffic creation to display in FAZ. It is very odd. We determined this by running packet capture on Fortigate and then waiting for the same traffic to appear in FAZ logs. It was almost 30 minutes to the second.
We also went into the FortiAnalyzer filter section:
config log fortianalyzer filter
We had things cranked up a bit, so we disabled dns (we are able to obtain DNS logs directly from our ADDCs which our Gate uses anyway), multicast-traffic, local-traffic, and sniffer-traffic logging to see if that helps. We did not view excessive resource contention on the Gate or the FAZ but maybe there are some underlying limits we were hitting with such settings enabled.
The Fortinet Security Fabric brings together the concepts of convergence and consolidation to provide comprehensive cybersecurity protection for all users, devices, and applications and across all network edges.