Troubleshooting Tip: Troubleshoot and debug SNMP traps

Description

This article describes how to troubleshoot Link State and MAC Notification traps when FortiNAC does not appear to be processing those received traps as expected.

Scope

FortiNAC v9.x, FortiNAC-F v7.x.

Solution

For a list of currently supported traps, see SNMP trap support in the Administration Guide.

Link State traps generate when a port’s link has come up or gone down.  

MAC Notification traps trigger under the following conditions:
Add - Device generates traffic for the first time.
Remove - MAC is removed from the address table. The time it takes for this to occur depends upon how the device is connected.

• Directly connected devices: MAC entry is removed immediately
• Devices behind an IP Phone, non-managed switch or hub: MAC entry must age out of the switch's MAC address table.  This is based on the age time configured within the switch (typically minutes).
  
  

Change - Device whose MAC is already learned on a port moves and connects to another port and generates traffic

Note: FortiAPs will not generate MAC Learned or Removed events. They are handled differently in code when it comes to syslog notifications.

For information on SNMP v3 MAC Traps, see Technical Tip: FortiNAC is not processing SNMPv3 MAC traps received from switches

Review Events in the UI:

1. Navigate to Logs -> Events & Alarms -> Management.
2. Enable the applicable events based on the trap. 'Right-click' on each event and select Log Internal.

   MAC Notification: MAC Learned and MAC Removed 

   Link State: Port Link Up and Port Link Down 

3. Once enabled, any link state or MAC Notification traps processed will generate an event.
4. Navigate to Logs -> Events & Alarms -> Events.
5. From the Add Filter drop-down menu, select Event.
6. From the Event drop-down menu, select the desired event enabled in the previous step.
7. Set any additional desired filters (such as date and time), then select Update.  

Troubleshooting steps if events are not generating:

1. Verify the sending switch is configured properly. 
   • Traps should be sent to the eth0/port1 IP address of the appliance.
   • Confirm traps are enabled on the switch ports. 
   • For details regarding MAC Notification traps, see Configuring Traps for MAC Notification in the Document Library.
2. Verify port 162 is not blocked to/from eth0/port1 on the network. If the FortiNAC-F appliance, ensure set allowaccess option includes snmp for port1.
3. Run a packet capture to confirm whether or not the traps are being received by the appliance. In addition, enable debug to gather additional information in case the traps are received but are not getting processed by FortiNAC.  

Enable debug and run packet capture (FortiNAC-OS):

1. Log in to the FortiNAC CLI as admin and enable debug. Run the following:

diagnose debug plugin enable TrapHandler
diagnose debug plugin enable DeviceInterface

2. Start a packet capture. Filter on SNMP trap traffic from the switch. Type: 

execute tcpdump -i any host <switch IP address> and port 162 -w LinkTrap.pcap

3. Recreate the behavior on the switch:

• Connect to a port.
• Disconnect from the port.
• Move to another port.

4. Press Ctrl + C to stop packet capture.
   
   
5. Export the capture (pcap) file. See Technical Tip: Run tcpdump in FortiNAC-F and save capture as a file. The trace can be viewed using Wireshark.
   
   
6. Disable the debug processes. Run the following:

diagnose debug plugin disable TrapHandler
diagnose debug plugin disable DeviceInterface 

exit

7. Collect the grab-log-snapshot log file. For instructions, see the following article: Technical Tip: How to get a debug log report from FortiNAC-CA or FortiNAC-Manager.

8. Navigate to Logs -> Events & Alarms -> Management.
   
   
9. Disable MAC Learned and MAC Removed events. 'Right-click' on each event and select Disable. 

Enable debug and run packet capture (CentOS):

1. Log in to the FortiNAC CLI as root and enable debug. Type:

nacdebug -name TrapHandler true
nacdebug -name DeviceInterface true

2. Start a packet capture. Filter on SNMP trap traffic from the switch. Type: 

logs

tcpdump -i any host <switch IP address> and port 162 -w LinkTrap.pcap

3. Recreate the behavior on the switch:

• Connect to a port.
• Disconnect from the port.

4. Ctrl +C to stop packet capture.

5. Export the capture (pcap) file. It can be downloaded from the appliance using WinSCP or a similar program. The trace can be viewed using Wireshark.

6. Disable the debug. Type:

nacdebug -name TrapHandler true
nacdebug -name DeviceInterface true

Contact the Support:

1. Collect the grab-log-snapshot log file. For instructions, see the following article: Technical Tip: How to get a debug log report from FortiNAC-CA or FortiNAC-Manager.
   
   

2. Navigate to Logs -> Events & Alarms -> Management.
   
   

3. Disable MAC Learned and MAC Removed events. 'Right-click' on each event and select Disable. 
   
   

4. Open a ticket on the Fortinet Support portal for further troubleshooting assistance. Include the following:

• Problem description.
• Switch information (screen capture of the device's Element tab under Network -> Inventory).
• Ports tested.
• The MAC address of the client was tested.
• FortiNAC version.
• Packet capture file.
• Logs.