FAZ VM Log Throttle

[seadave]

I have a FAZVM hosted on Dell Compellent (mostly flash) and Dell R630 servers, vSphere 6.5U3.  The VM has 16G RAM and 8vCPUs.  We have a 2GB/Day license and currently consume ~1.5G/day.  As far as I can tell storage capacity and IOPs are not rate limiting but my receive rate never exceeds 150 logs/sec.  Insertion rate is consistently less than 60s. 

We have two 501Es in A/P HA.  We are outputting logs to a Syslog destination in addition to the FAZ.  It does not appear that we are dropping logs/traffic but the receive rate appears very flat during what would be peak periods.  See attached.

Looking through docs in older FortiOS, it appears there was a min and max buffer setting for FAZ settings on the Fortigate but that is not available in FortiOS 6.0.5.  Our FAZ is the same version.  We try and log everything including denies so there should be enough volume.  Firewall is protecting ~500 devices (desktops/laptops/mobile) and servers such as Exchange.  It just seems strange that the receive rate is so flat during business hours.  If anyone has any ideas of what to test check, I'd appreciate it.

[seadave]

Partially resolved by killing the miglogd service.  Switch that connects FAZ and Gate was rebooted and for some reason that killed DNS logging.  Attempted to stop and restart logging, via GUI but DNS flows would not appear.  If you log into the Fortigate CLI   diag sys top-summary   look for the miglogd process and note the process ID (PID)   197     320M    0.0  2.0    73  09:19.30  miglogd [x5]    Press "q" to quit the monitoring of sys top.   Now that you have the PID of the miglogd process, enter the following to kill and restart it:   diag sys kill 11 197   Note in my case the PID was 197 as highlighted above.  Once we did this, DNS logs started flowing again.  Odd issue.  Will see what my log volume is over next 24 hours.

[ede_pfau]

Before reading your second post, I thought that the figures look pretty OK.

Assume 100 bytes per log entry, and 1.5 GB/86400s, would mean ~ 186 logs/s. Which wouldn't surprise me in your environment, and is far from any rate limit stated for the FAZ(-VM).

1.5 or 2 GB/day is easily taken by a FAZ-VM, even with just 2 vCPUs. If you look at the receive/insertion rate diagram, you see that everything that is received is promptly stored in the DB (no lag, no steady differences). So that looks OK as well.

Not getting any DNS logs is of course a bug situation. I just wonder why it doesn't show in the load diagrams. I'd ask Support to have a look at this, esp. for DNS logs.

You might (might) rebuild the DBs, or the data partition, with old logs being saved in advance. But, this is definitively a Support issue then.

[tanr]

Please let us know what you hear from TAC, especially about actual cause.

Makes me think a FAZ rule to notify me if daily logs change by more than 15% would be useful.