Description
This article describes how to identify the error messages when FortiNAC is unable to process SNMPv3 MAC traps received from network switches.
Scope
FortiNAC, FortiNAC-F, Switches
Solution
When troubleshooting such scenarios, it is important to enable the following debugs in FortiNAC-F and additionally run a Packet capture to check the Trap attributes. This can be done in two separate sessions
- Enable FortiNAC debugs in the first SSH session:
execute enter-shell
logs
device -ip X.X.X.X -setAttr -name DEBUG -value "ForwardingInterface TelnetServer" <- Replace X.X.X.X with a Switch IP.
nacdebug -name TrapHandler true
nacdebug -name DeviceInterface true
nacdebug -name BridgeManager true
nacdebug -name SnmpV1 true
DumpBridgePerformance -enableMacNotifyDebug
nacdebug -logger org.snmp4j -level FINEST
nacdebug -logger org.snmp4j.MessageDispatcherImpl -level FINEST
exit
diagnose tail -F output.master | grep -i "not dispatched"
In this case, the CLI output will show only FortiNAC log events that include the string 'not dispatched', which refers to status messages related to the TRAP error. The status code in the end of the log will provide a rough idea of the error.
Example: Message from 192.168.10.2/48265 not dispatched, reason: statusInfo=1.3.6.1.6.3.15.1.1.2.0 = yyy, status=1411
2. In the second FortiNAC CLI session, enable packet capture:
execute tcpdump -i any host X.X.X.X and port 162 -w MACtrap.pcap
After these debugs are enabled, the user can recreate the issue by connecting to a switch port.
Once finished stop the Packet capture with ctrl+c.
The packet capture is automatically saved in /home/admin directory.
It can be exported by using a TFTP or SSH server.
The following article provides an example: Technical Tip: Run tcpdump in FortiNAC-F and save capture as a file.
-
Decrypting SNMPv3 packets to read V3 Mac trap attributes.
SNMPv3 traps are encrypted. To read the PCAP for the v3 trap attributes, it is necessary to decrypt the packets.
This can be done in Wireshark by going to Edit -> Preferences -> Protocols -> SNMP.
In the SNMP protocol add a new User in the 'User Table' button that contains the SNMPv3 auth attributes configured in the switch.
The attributes required are the following:
- Engine ID.
- SNMPv3 username.
- Choose the authentication model (MD5 | SHA1).
- Put the password for the authentication model.
- Choose the privacy protocol (DES | AES | AES192 | AES256).
- Put the privacy password.
Disable debugging:
execute enter-shell
logs
device -ip X.X.X.X -delAttr -name DEBUG <- Replace X.X.X.X with a Switch IP.
nacdebug -name TrapHandler false
nacdebug -name DeviceInterface false
nacdebug -name BridgeManager false
nacdebug -name SnmpV1 false
DumpBridgePerformance -disableMacNotifyDebug
nacdebug -logger org.snmp4j
nacdebug -logger org.snmp4j.MessageDispatcherImpl
Below is an example of the SNMPv3 MAC trap sent by the Switch and the corresponding error message in FNAC debugs when the Trap is received:
- MAC trap sent from a Switch:
Simple Network Management Protocol
msgVersion: snmpv3 (3)
msgGlobalData
msgAuthoritativeEngineID: YYYYYYYYYYYYYYY
1... .... = Engine ID Conformance: RFC3411 (SNMPv3)
Engine Enterprise ID: Hewlett Packard Enterprise (47196)
Engine ID Format: MAC address (3)
Engine ID Data: MAC address: ArubaHewlett_XX:XX:XX (XX:XX:XX:XX:XX:XX)
msgAuthoritativeEngineBoots: 0
msgAuthoritativeEngineTime: 0
msgUserName: fortinac
msgAuthenticationParameters: YYYYYYYYYYYYY
[Authentication: OK]
[Expert Info (Chat/Checksum): SNMP Authentication OK]
[SNMP Authentication OK]
[Severity level: Chat]
[Group: Checksum]
msgPrivacyParameters: YYYYYYYY
-
Error event logged in FortiNAC output.master.
Message from 192.168.10.2/48265 not dispatched, reason: statusInfo=1.3.6.1.6.3.15.1.1.2.0 = yyy, status=1411
The snmp4j message status is 1411, which is SNMPv3_USM_NOT_IN_TIME_WINDOW.
This appears to indicate that the time on one of the devices (switch, FNAC or both) has drifted by more than 150 seconds.
The following RFC section describes how SNMP v3 traps are processed and the different conditions for correct processing.
RFC section
Resolution:
- Verify that NTP and time/date are set the same and in sync between the Switch and FortiNAC.
- According to RFC 3414:
When an SNMP engine is first installed, it sets its local values of snmpEngineBoots and snmpEngineTime to zero. If snmpEngineTime ever reaches its maximum value (2147483647), then snmpEngineBoots is incremented as if the SNMP engine has re-booted and snmpEngineTime is reset to zero and starts incrementing again.
In this case, there is no incrementation snmpEngineTime. All SNMP traps have both boot and time counters = 0.
msgAuthoritativeEngineBoots: 0
msgAuthoritativeEngineTime: 0
The msgAuthoritativeEngineTime is the time notion of the trap sender (Network switch in this case). The time notion for the engine ID of the sender must not diverge more than 150 seconds from the receiver.
In this case, there is an issue from network switch where it is not incrementing the counters.
Aruba CX switches have a known issue reported Bug ID 312998. Aruba documentation provides more details of the symptoms.
Restarting both FortiNAC and the switch will clear snmp4js previously known cached values for these attributes and help in confirming if the switch would start sending Trap with the correct attributes.
If the counters are incrementing as expected, check the following article in order to investigate further:
Troubleshooting Tip: Identify cause of SNMPv3 MAC Trap Error: 'RFC3414 §3.2.7.a Not in time window'
Error list for V3 traps:
"SNMPv3: USM: ok", <- 1400.
"SNMPv3: USM: error", <- 1401.
"SNMPv3: USM: Configfile write error", <- 1402.
"SNMPv3: USM: Unsupported SecurityLevel", <- 1403.
"SNMPv3: USM: Unknown SecurityName", <- 1404.
"SNMPv3: USM: Encryption error", <- 1405.
"SNMPv3: USM: Decryption error", <- 1406.
"SNMPv3: USM: Authentication error", <- 1407.
"SNMPv3: USM: Authentication failure", <- 1408.
"SNMPv3: USM: Parse error", <- 1409.
"SNMPv3: USM: Unknown EngineID", <- 1410.
"SNMPv3: USM: Message not in TimeWindow", <- 1411.
"SNMPv3: USM: Unsupported AuthProtocol", <- 1412.
"SNMPv3: USM: Unsupported PrivProtocol", <- 1413.
"SNMPv3: USM: Address error", <- 1414.
"SNMPv3: USM: Could not create file", <- 1415.
"SNMPv3: USM: Could not open file", <- 1416.
"SNMPv3: USM: Could not rename file", <- 1417.
"SNMPv3: USM: Could not delete file", <- 1418.
"SNMPv3: USM: Could not write into file", <- 1419.
"SNMPv3: USM: Could not read from file", <- 1420.
Related articles:
Technical Tip: Troubleshooting SNMP communication issues
Troubleshooting Tip: Identify cause of SNMPv3 MAC Trap Error: 'RFC3414 §3.2.7.a Not in time window'
Technical Note: SNMPv3 Communication Fails for Certain Devices
