FortiAnalyzer Throttling

[seadave]

We have a FAZ VM with the 2G/Day/1TB Storage license.  We noted that our daily log flows were very flat and topping out at 130Logs/Sec.  Rarely if ever exceeding that amount.  The current FAZ we have had been upgraded several times and we export all data it collects to Splunk for easier searching and correlation with other event sources.

So we downloaded a 6.2.3 FAZ VM OVF yesterday.  We are logging from a HA pair in A/P mode of 501Es.  They have a 1G connection to the Internet and 10G connection to our core.  We spun up the VM and ran into the old miglogd issue.  For whatever reason on our 6.0.5 FG build, if we make any changes to the log settings or disrupt the flow of logs (by rebooting the FAZ for example), they stall until we console into the Fortigate and do the following:

diag sys top-summary

View the output and locate the PID for miglogd (it is often a 5 digit number such as 11035, etc.)  Then we have to kill the process:

diag sys kill 11 11035

Logs will then immediately startup again to the FAZ. 

Anyway, while the new FAZ VM was running unlicensed (we configured it with the same vCPU and ram as the old one), we noticed the log flow rate peaked at 4000/second and stabilized around 1200 to 1500.  So we set the management IP on the new FAZ to be the same as our old FAZ and applied our FAZ license.  It rebooted and everything was happy, EXCEPT, log flow rate does NOT exceed 130 Logs/s.  Why is it limiting?  Is it dropping traffic or will it buffer and only injest at a max rate determined by the license level?  I've been told it doesn't limit but it sure seems to be.

See the attached image that shows the peak flow and it should be obvious when we applied the license.

[seadave]

Here our log data in Splunk for three days and again you can see it is very flat.  I would expect more of a bell curve with a drop around noon.

[skyhigh]

There is a known issue for clusterd using high CPU utilization in FortiAnalyzer 6.2.3.  Check "exec top" on your FAZ to see whether that is the case (this issue can occur for standalone FAZ as well).  If so, there is a special build support can provide you.

Otherwise, perhaps you could specify exactly what resources you have allocated to the VM.  Or just go ahead and open a support ticket.

[seadave]

Sorry thought I posted the specs.  FAZVM is running with 4vCPUs and 16G RAM on vSphere 6.7U3 backed by SSD based storage on Dell R630 hosts.

Our CPU util is very low.  Less than 7% across 4 vCPUs according to the GUI.  The exec top shows the following.  It seems to indicated clusterd is at 30%.

top - 10:01:01 up 17:19, 0 users, load average: 0.17, 0.15, 0.10 Tasks: 236 total, 1 running, 235 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.7 us, 0.5 sy, 0.0 ni, 98.7 id, 0.0 wa, 0.0 hi, 0.1 si, 0.0 st MiB Mem : 16063.69+total, 3856.469 free, 2907.656 used, 9299.574 buff/cache MiB Swap: 16062.99+total, 16062.99+free, 0.000 used. 9745.328 avail Mem

PID USER PR NI VIRT RES %CPU %MEM TIME+ S COMMAND 734 root 20 0 621.4m 23.7m 29.6 0.1 393:56.59 S /bin/clusterd 715 root 20 0 576.2m 27.9m 0.7 0.2 9:17.27 S /bin/logfwd 719 root 20 0 1112.2m 68.9m 0.7 0.4 2:26.07 S /bin/oftpd 7 root 20 0 0.0m 0.0m 0.3 0.0 0:34.87 S [rcu_sched] 408 root 20 0 53.0m 4.6m 0.3 0.0 0:53.90 S /bin/redis-server 127.0.0.1:6379 593 root 20 0 720.5m 379.3m 0.3 2.4 3:34.97 S dmserver 617 root 20 0 57.5m 4.8m 0.3 0.0 0:57.47 S /bin/redis-server 127.0.0.1:6380 8813 root 20 0 492.9m 67.8m 0.3 0.4 0:01.52 S /usr/local/apache2/bin/httpd -DSSL -DNO_DETACH 1 root 20 0 705.7m 387.2m 0.0 2.4 0:19.59 S /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 2 root 20 0 0.0m 0.0m 0.0 0.0 0:00.00 S [kthreadd] 3 root 20 0 0.0m 0.0m 0.0 0.0 0:00.50 S [ksoftirqd/0] 4 root 20 0 0.0m 0.0m 0.0 0.0 0:00.00 S [kworker/0:0] 5 root 0 -20 0.0m 0.0m 0.0 0.0 0:00.00 S [kworker/0:0H] 8 root 20 0 0.0m 0.0m 0.0 0.0 0:00.00 S [rcu_bh] 9 root rt 0 0.0m 0.0m 0.0 0.0 0:00.00 S [migration/0] 10 root rt 0 0.0m 0.0m 0.0 0.0 0:00.00 S [migration/1] 11 root 20 0 0.0m 0.0m 0.0 0.0 0:00.20 S [ksoftirqd/1] 13 root 0 -20 0.0m 0.0m 0.0 0.0 0:00.00 S [kworker/1:0H] 14 root rt 0 0.0m 0.0m 0.0 0.0 0:00.00 S [migration/2] 15 root 20 0 0.0m 0.0m 0.0 0.0 0:00.64 S [ksoftirqd/2] 16 root 20 0 0.0m 0.0m 0.0 0.0 0:00.00 S [kworker/2:0] 17 root 0 -20 0.0m 0.0m 0.0 0.0 0:00.00 S [kworker/2:0H] 18 root rt 0 0.0m 0.0m 0.0 0.0 0:00.00 S [migration/3] 19 root 20 0 0.0m 0.0m 0.0 0.0 0:00.17 S [ksoftirqd/3] 20 root 20 0 0.0m 0.0m 0.0 0.0 0:00.00 S [kworker/3:0] 21 root 0 -20 0.0m 0.0m 0.0 0.0 0:00.00 S [kworker/3:0H] 22 root 20 0 0.0m 0.0m 0.0 0.0 0:00.00 S [kdevtmpfs] 23 root 0 -20 0.0m 0.0m 0.0 0.0 0:00.00 S [perf] 24 root 0 -20 0.0m 0.0m 0.0 0.0 0:00.00 S [writeback] 25 root 0 -20 0.0m 0.0m 0.0 0.0 0:00.00 S [crypto] 26 root 20 0 0.0m 0.0m 0.0 0.0 0:00.63 S [kworker/0:1] 27 root 0 -20 0.0m 0.0m 0.0 0.0 0:00.00 S [bioset] 28 root 0 -20 0.0m 0.0m 0.0 0.0 0:00.00 S [kblockd] 29 root 0 -20 0.0m 0.0m 0.0 0.0 0:00.00 S [ata_sff] 30 root 20 0 0.0m 0.0m 0.0 0.0 0:00.66 S [kworker/2:1] 31 root 20 0 0.0m 0.0m 0.0 0.0 0:00.63 S [kworker/3:1] 32 root 20 0 0.0m 0.0m 0.0 0.0 0:00.84 S [kworker/1:1] 33 root 20 0 0.0m 0.0m 0.0 0.0 0:00.00 S [kswapd0] 34 root 0 -20 0.0m 0.0m 0.0 0.0 0:00.00 S [vmstat] 35 root 20 0 0.0m 0.0m 0.0 0.0 0:00.00 S [fsnotify_mark] 65 root 0 -20 0.0m 0.0m 0.0 0.0 0:00.00 S [acpi_thermal_pm] 66 root 0 -20 0.0m 0.0m 0.0 0.0 0:00.00 S [bioset] 67 root 0 -20 0.0m 0.0m 0.0 0.0 0:00.00 S [bioset] 68 root 0 -20 0.0m 0.0m 0.0 0.0 0:00.00 S [bioset]

[skyhigh]

> I've been told it doesn't limit but it sure seems to be.

At this time, there is no intentional hard limit enforced on the logs received or forwarded by FortiAnalyzer.  The license limit is a daily limit (GB/day) and you will see event logs if your sustained log rate exceeds that which would eventually cause the daily limit to be reached.  In most cases, any limitation on log processing should be resource-related.

I would still advise you to open a support ticket.

[seadave]

We are continuing to evaluate.  We added more vCPU, now 8, and left RAM at 16.  According to docs this should allow 6000 lps or more.  We were seeing days of exactly 30 minutes from traffic creation to display in FAZ.  It is very odd.  We determined this by running packet capture on Fortigate and then waiting for the same traffic to appear in FAZ logs.  It was almost 30 minutes to the second.

We also went into the FortiAnalyzer filter section:

config log fortianalyzer filter

We had things cranked up a bit, so we disabled dns (we are able to obtain DNS logs directly from our ADDCs which our Gate uses anyway), multicast-traffic, local-traffic, and sniffer-traffic logging to see if that helps.  We did not view excessive resource contention on the Gate or the FAZ but maybe there are some underlying limits we were hitting with such settings enabled.