SNMP Data Missing Under Load

[40netuser]

Hi there,

We're seeing a strange issue with the SNMP data from our FortiGate 1500D firewall.

We're running version 6.4.4 and using SNMP V3.

Basically it seems that the Octet data isn't being captured on our 10Gbps WAN link when under load.

We have a 1Gbps WAN link and that graph is consistent and accurate. We also have some 100Gb switch links being monitored by our SNMP data gathering server, and those are all true as well - so I don't think it has anything to do with the link speed or throughput.

When we load up the 10G link you can see the graph information is missing from our SNMP data gathering server, but not from the dashboard in the web interface of the FortiGate.

Please see attached example photos.

Disregard the Units on the 10Gbps link, just a formatting error from the graphing system.

Here is the 1Gbps WAN link with no issues of missing or incorrect data.

Here is the 10Gbps link that while under load is missing information and incorrect data rates.

2.5KMb/s just means 2500Mb/s and you can see this doesn't tally with the actual throughput.

Here is the 10Gbps link that while under load is fine from the web interface.

[Kasey08]

I've never had this before with a previous vidmate version of FortiClient SSL VPN.

[hermann]

We have been monitoring our 1500D in the same version without issues but not having such high traffic.

Have you already checked if SNMP requests are just timed out or they are preformed in-time but do not receive any payload? 

If possible, change SNMPv3 to SNMPv1 for a test and check, if the behavior changes.

Have you checked the release notes for newer versions, if the have a fix for an issue likes yours?

Kind regards

Hermann

[40netuser]

So I haven't explicity tested if the SNMP requests are timing out, and due to the intermittent nature I'm not sure how long testing would be required to reveal such issue. Furthermore the SNMP requests about the 1Gbps port are coming in consistent and acurate - so I don't feel the SNMP server on the firewall is not responding. Also what I find odd is the information on the 10Gbps interface, when responsive, is incorrect as you can see the SNMP based graph never exceeding much over 2Gbps - whereas we know it's truly over 5Gbps of throughput.

I'll give SNMP V1 a test and see if that works as expected.

I did have a look at the release notes and they didn't suggest any issue as to what I'm seeing. There were some fixes around SNMP but not related to this issue.

Thanks

[40netuser]

So SNMP V1 is producing the same results as V3.

See below screenshots of the graphs;

[hermann · Accepted solution]

I think it is worth to ask Fortinet support for help.

Just thinking loud: AFAIK the SNMP counters for a network interface are just increased continuously with each byte flowing through the interface. At some point the counter, which may be defined as 2^32-1 = 2.147.483.648, overruns and counts from the beginning. If your SNMP probes run not too frequently, they could be affected by this behavior. Could you run those much more frequently to see if your probes get real results?

[40netuser]

Yes I just thought the same after my last post. I just tested and this is it!

I've changed the update interval to 5 seconds now, was previously 15 seconds.

See below graph now is accurate;

Glad we got there in the end!