Skip to main content
seadave
seadave
New Member
January 24, 2020
Question

FortiAnalyzer Throttling

  • January 24, 2020
  • 1 reply
  • 10281 views

We have a FAZ VM with the 2G/Day/1TB Storage license.  We noted that our daily log flows were very flat and topping out at 130Logs/Sec.  Rarely if ever exceeding that amount.  The current FAZ we have had been upgraded several times and we export all data it collects to Splunk for easier searching and correlation with other event sources.

 

So we downloaded a 6.2.3 FAZ VM OVF yesterday.  We are logging from a HA pair in A/P mode of 501Es.  They have a 1G connection to the Internet and 10G connection to our core.  We spun up the VM and ran into the old miglogd issue.  For whatever reason on our 6.0.5 FG build, if we make any changes to the log settings or disrupt the flow of logs (by rebooting the FAZ for example), they stall until we console into the Fortigate and do the following:

 

diag sys top-summary

 

View the output and locate the PID for miglogd (it is often a 5 digit number such as 11035, etc.)  Then we have to kill the process:

 

diag sys kill 11 11035

 

Logs will then immediately startup again to the FAZ. 

 

Anyway, while the new FAZ VM was running unlicensed (we configured it with the same vCPU and ram as the old one), we noticed the log flow rate peaked at 4000/second and stabilized around 1200 to 1500.  So we set the management IP on the new FAZ to be the same as our old FAZ and applied our FAZ license.  It rebooted and everything was happy, EXCEPT, log flow rate does NOT exceed 130 Logs/s.  Why is it limiting?  Is it dropping traffic or will it buffer and only injest at a max rate determined by the license level?  I've been told it doesn't limit but it sure seems to be.

 

See the attached image that shows the peak flow and it should be obvious when we applied the license.

1 reply

seadave
seadaveAuthor
New Member
January 24, 2020

Here our log data in Splunk for three days and again you can see it is very flat.  I would expect more of a bell curve with a drop around noon.

skyhigh
skyhigh
New Member
January 24, 2020

There is a known issue for clusterd using high CPU utilization in FortiAnalyzer 6.2.3.  Check "exec top" on your FAZ to see whether that is the case (this issue can occur for standalone FAZ as well).  If so, there is a special build support can provide you.

 

Otherwise, perhaps you could specify exactly what resources you have allocated to the VM.  Or just go ahead and open a support ticket.

seadave
seadaveAuthor
New Member
January 24, 2020

Sorry thought I posted the specs.  FAZVM is running with 4vCPUs and 16G RAM on vSphere 6.7U3 backed by SSD based storage on Dell R630 hosts.

 

Our CPU util is very low.  Less than 7% across 4 vCPUs according to the GUI.  The exec top shows the following.  It seems to indicated clusterd is at 30%.

 

top - 10:01:01 up 17:19, 0 users, load average: 0.17, 0.15, 0.10 Tasks: 236 total, 1 running, 235 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.7 us, 0.5 sy, 0.0 ni, 98.7 id, 0.0 wa, 0.0 hi, 0.1 si, 0.0 st MiB Mem : 16063.69+total, 3856.469 free, 2907.656 used, 9299.574 buff/cache MiB Swap: 16062.99+total, 16062.99+free, 0.000 used. 9745.328 avail Mem

PID USER PR NI VIRT RES %CPU %MEM TIME+ S COMMAND 734 root 20 0 621.4m 23.7m 29.6 0.1 393:56.59 S /bin/clusterd 715 root 20 0 576.2m 27.9m 0.7 0.2 9:17.27 S /bin/logfwd 719 root 20 0 1112.2m 68.9m 0.7 0.4 2:26.07 S /bin/oftpd 7 root 20 0 0.0m 0.0m 0.3 0.0 0:34.87 S [rcu_sched] 408 root 20 0 53.0m 4.6m 0.3 0.0 0:53.90 S /bin/redis-server 127.0.0.1:6379 593 root 20 0 720.5m 379.3m 0.3 2.4 3:34.97 S dmserver 617 root 20 0 57.5m 4.8m 0.3 0.0 0:57.47 S /bin/redis-server 127.0.0.1:6380 8813 root 20 0 492.9m 67.8m 0.3 0.4 0:01.52 S /usr/local/apache2/bin/httpd -DSSL -DNO_DETACH 1 root 20 0 705.7m 387.2m 0.0 2.4 0:19.59 S /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 2 root 20 0 0.0m 0.0m 0.0 0.0 0:00.00 S [kthreadd] 3 root 20 0 0.0m 0.0m 0.0 0.0 0:00.50 S [ksoftirqd/0] 4 root 20 0 0.0m 0.0m 0.0 0.0 0:00.00 S [kworker/0:0] 5 root 0 -20 0.0m 0.0m 0.0 0.0 0:00.00 S [kworker/0:0H] 8 root 20 0 0.0m 0.0m 0.0 0.0 0:00.00 S [rcu_bh] 9 root rt 0 0.0m 0.0m 0.0 0.0 0:00.00 S [migration/0] 10 root rt 0 0.0m 0.0m 0.0 0.0 0:00.00 S [migration/1] 11 root 20 0 0.0m 0.0m 0.0 0.0 0:00.20 S [ksoftirqd/1] 13 root 0 -20 0.0m 0.0m 0.0 0.0 0:00.00 S [kworker/1:0H] 14 root rt 0 0.0m 0.0m 0.0 0.0 0:00.00 S [migration/2] 15 root 20 0 0.0m 0.0m 0.0 0.0 0:00.64 S [ksoftirqd/2] 16 root 20 0 0.0m 0.0m 0.0 0.0 0:00.00 S [kworker/2:0] 17 root 0 -20 0.0m 0.0m 0.0 0.0 0:00.00 S [kworker/2:0H] 18 root rt 0 0.0m 0.0m 0.0 0.0 0:00.00 S [migration/3] 19 root 20 0 0.0m 0.0m 0.0 0.0 0:00.17 S [ksoftirqd/3] 20 root 20 0 0.0m 0.0m 0.0 0.0 0:00.00 S [kworker/3:0] 21 root 0 -20 0.0m 0.0m 0.0 0.0 0:00.00 S [kworker/3:0H] 22 root 20 0 0.0m 0.0m 0.0 0.0 0:00.00 S [kdevtmpfs] 23 root 0 -20 0.0m 0.0m 0.0 0.0 0:00.00 S [perf] 24 root 0 -20 0.0m 0.0m 0.0 0.0 0:00.00 S [writeback] 25 root 0 -20 0.0m 0.0m 0.0 0.0 0:00.00 S [crypto] 26 root 20 0 0.0m 0.0m 0.0 0.0 0:00.63 S [kworker/0:1] 27 root 0 -20 0.0m 0.0m 0.0 0.0 0:00.00 S [bioset] 28 root 0 -20 0.0m 0.0m 0.0 0.0 0:00.00 S [kblockd] 29 root 0 -20 0.0m 0.0m 0.0 0.0 0:00.00 S [ata_sff] 30 root 20 0 0.0m 0.0m 0.0 0.0 0:00.66 S [kworker/2:1] 31 root 20 0 0.0m 0.0m 0.0 0.0 0:00.63 S [kworker/3:1] 32 root 20 0 0.0m 0.0m 0.0 0.0 0:00.84 S [kworker/1:1] 33 root 20 0 0.0m 0.0m 0.0 0.0 0:00.00 S [kswapd0] 34 root 0 -20 0.0m 0.0m 0.0 0.0 0:00.00 S [vmstat] 35 root 20 0 0.0m 0.0m 0.0 0.0 0:00.00 S [fsnotify_mark] 65 root 0 -20 0.0m 0.0m 0.0 0.0 0:00.00 S [acpi_thermal_pm] 66 root 0 -20 0.0m 0.0m 0.0 0.0 0:00.00 S [bioset] 67 root 0 -20 0.0m 0.0m 0.0 0.0 0:00.00 S [bioset] 68 root 0 -20 0.0m 0.0m 0.0 0.0 0:00.00 S [bioset]

Terms & ConditionsAccessibility statement