Technical Tip: FortiNAC general troubleshooting guide

Hatibi · ‎08-26-2024

Description

This article describes the steps of how to troubleshoot FortiNAC and where to look for information to isolate a problem. Additionally, it provides a resource list of detailed troubleshooting for each FortiNAC service and function.

Scope

FortiNAC.

Solution

When troubleshooting issues with FortiNAC it is important to first have an idea of how the system is performing and if any critical Events would show system performance issues, devices being unreachable or SSH/CLI failing.

As the first step to isolate any problem, the FortiNAC administrator should first check the Event Logs and filter for the time when the issue happened. Identify if the problem is reoccurring and if it can be reproduced. Verify how many users/sites are affected and the patterns of the issue (like only in the morning between 8-9 am, when the shift starts).

Furthermore, it could be helpful to check logs from other devices such as hypervisors or Syslog Servers that might assist in further isolating the problem.

The Event logs from FortiNAC can be exported in an Excel or CSV file to search and filter for 'failure' or 'error' keywords.

The following steps need to be applied:

Set filters as needed.
Select 'Update' so the event list is updated with the filtered events.
Export the list.

Figure 1. Filtering for events and exporting the event list

In this case, it is possible to see that there is a Secondary Lost event. The next step would be to verify if the Secondary FortiNAC server is up and running and if SSH and Ping services are enabled between the two nodes.

Debug Plugins: After narrowing down the problem or gaining more insight from the event logs, it is then possible to enable specific Debug plugins in the FortiNAC CLI to further investigate the issue and find out specific error messages that might show what the problem is.

Debug plugins are either bound to the processes of the Control server (Master Loader) or the Application server (Nessus).

Control server (Master Loader).

Deals with management processes and functions such as RADIUS, LDAP, Switch communications, VLAN changes, and multiple services running on Port1 (eth0) of FortiNAC.
Log messages are written in output.master file.

Application server (Nessus).

Responsible for DHCP, DNS, and captive services provided in Port2 (eth1) of FortiNAC.
Log messages are written in output.nessus file.

Check for a list of all available Debug plugins in FortiNAC CLI:

.

As shown, there are some debug plugins bound to Nessus and some of them to MasterLoader.

This is important to take note of when troubleshooting an issue since it is necessary to know where to look for debug logs depending on the issue at hand.

Example:

There is an issue with CLI credentials failing for an Inventory device.

To investigate this problem it is necessary to enable the TelnetServer and BridgeManager debug plugins.

Find them by listing all debugs and using the 'grep' command to filter them out.

The output shows that these are debug plugins related to MasterLoader and log messages will be printed in output.master file. The output also shows that these debug plugins are currently disabled (false).

Proceed to enable the logs and confirm they are enabled:

After this, it is possible to print out output.master debug logs while the issue is recreated as follows:

naclab1 # diag tail -F output.master
yams.Yams INFO :: 2024-08-26 12:21:47:359 :: #547 :: MasterLoader Max Memory (KBytes) 4,244,608 Free Memory (KBytes) 3,876,514 Threads: 524 Up Time: 4 Days 23 Hours 50 Minutes 59 Seconds Time Zone: CEST (UTC+0200)
yams.Yams INFO :: 2024-08-26 12:21:52:281 :: #6074 :: Client connecting from IP = 127.0.0.1 name = localhost port = 47916

.......

.....

The same steps can be applied when troubleshooting another issue where the debug plugin might be bound to Nessus.

Such examples would be the ActiveFingerprint or PersistentAgent debug plugins.

In that case, the file to be checked and the log message printed out in cli is output.nessus.

Working with FortiNAC technical support: When providing logs to FortiNAC TAC it is important to always share the FortiNAC system logs and the exported Event logs covering the time of the issue.

Related articles:

Technical Tip: How to get a debug log report from FortiNAC-CA or FortiNAC-Manager.

Troubleshooting Tip: Using Events and Audit logs to identify configuration changes or new issues

FortiNAC Troubleshooting Resource List.

The below resource list provides links to other articles focused in troubleshooting specific FortiNAC issues.

They are categorized based on the following:

Issues related to FortiNAC access, resources, performance, and Licensing

TITLE	DESCRIPTION
FortiNAC data sheet	Page 14 provides VM resource sizing information. At minimum a FortiNAC VM should have 8vCPUs, 16 GB RAM and 100 GB Disk.
Technical Tip: Performance issue and some general recommendations	General recommendations for improving FortiNAC's performance.
Troubleshooting Tip: 'Licensed Without Certificates' message in UI	The message is presented to make users aware that the current Endpoint License key does not contain certificates that some new features and functions require.
Technical Tip: What consumes endpoint licenses and tracking it through Dashboards	Discusses what type of endpoints consume a license in FortiNAC. Administrators can leverage Dashboards in order to properly visualize and keep track of concurrent licenses.
Technical Tip: List devices that are currently consuming licenses	How to verify devices currently using a license.
Technical Tip: Upgrade FortiNAC-F through the CLI	How to upgrade the FortiNAC-F appliance using the CLI
Troubleshooting Tip: Admin UI not accessible after changing gateway IP and /eth0Port1 IP from FortiN...	Gain UI access after FortiNAC management IP is changed.
Technical Tip: ‘Processes are Down’ message after network or server change	The procedure needed to complete when a FortiNAC VM appliance has been migrated to another server or there have been network changes.
Technical Tip: How to add a license to FortiNAC-F 7.* from the CLI	How to add a license to FortiNAC-F 7.* from the CLI when the GUI is not reachable.

Issues related to Visibility and Registration of Hosts

TITLE	DESCRIPTION
Technical Tip: Troubleshooting SNMP communication issues	Describes basic steps to troubleshoot SNMP Communication Issues.
Troubleshooting Tip: Troubleshooting CLI credential failure	Describes how to resolve an issue where, when selecting the Validate Credentials button in Model Configuration
Technical Tip: Troubleshooting Poll failures	Steps to troubleshoot L2 or L3 Poll Failure events
Troubleshooting Tip: FortiNAC-F 'Duplicated IP address' error after deleting and adding a unit to th...	Describes how to fix the error message 'Duplicated IP address' when a unit is deleted from the topology view and tries to add it again
Troubleshooting Tip: Wireless connected hosts are not matching Location Based Policies	Describes how to investigate cases where FortiNAC policies are not matching wireless connected hosts when the Location is used as matching criteria.
Technical Tip: FortiNAC not processing SNMP v3 MAC traps from Switches	How to identify the error messages when FortiNAC is unable to process V3 MAC traps
Troubleshooting tip: Verify device support in FortiNAC-F using CLI and DB shell	How to use the CLI and DB shell to determine device support within FortiNAC-F.
Technical Tip: Device Profiling Rule with the SSH Method	How to create a Device Profiling Rule with the SSH method.
Troubleshooting Tip: Unable to complete L2 polling due to a locked device model	This article describes how to resolve device model issues that have a 'locked' state.
Technical Tip: Configure and validate Cisco SNMPv3	Describes general guidance to configure and validate Cisco SNMPv3. To configure MAC notification traps, refer to the MAC notification traps reference manual in the Fortinet Document Library.
Troubleshooting Tip: Troubleshooting a rogue not matching any device profiles	how to troubleshoot a rogue connected to the network which does not match any of the configured Device Profiling Rules.

Issues Related to RADIUS and Authentication

TITLE	DESCRIPTION
Troubleshooting Tip: Viewing FortiNAC-F local RADIUS logs from GUI	This article describes how to view local RADIUS authentication logs from GUI.
Troubleshooting Tip: Common local RADIUS failures, debug logs, and examples	Common local Radius failures in FortiNAC and provides accompanying debug logs and examples.
Technical Tip: MSCHAPv2 authentication, join FortiNAC in domain and checks	Winbind configuration and verification. In cases where authentication is required to use MSCHAPv2.
Troubleshooting Tip: Failed to add Winbind to FortiNAC due to SPNEGO bind with Kerberos failure	Describes how to fix and troubleshoot the Winbind integration with FortiNAC.
Technical Tip: FortiNAC-F FortiAuthenticator RADIUS Proxy Integration.	how to integrate FortiNAC-F acting as a RADIUS proxy with FortiAuthenticator using PEAP-MSCHAPv2.
Troubleshooting Tip: Local RADIUS log message examples	Describes about log entries that may be printed in the Local RADIUS logs and possible causes.
Technical Tip: FortiNAC RADIUS debug errors and solutions	Discusses some RADIUS errors that might appear in the RADIUS logs and how to resolve them.
Technical Tip: FortiNAC Local Radius not able to process requests due to switch MTU value	Describes cases with 802.1x EAP-TLS authentication where switches are configured with a non-default MTU value which prevents Local Radius in FortiNAC to respond to authentication requests.
Troubleshooting Tip: Local Winbind configuration fails to start	Article explains how to solve an issue that occurs when the service fails to start while configuring Local Winbind in the Administration UI
Technical Tip: Create and use a Keytab file to join FortiNAC in the domain	Join the domain using a Kerberos Keytab file for authentication, instead of requiring the admin account password during the Winbind configuration.

Issues related to enforcement: State Based Control and NAC policies

TITLE	DESCRIPTION
Technical Tip: Captive Portal is not showing for Rogue Hosts	Describes the requirements and conditions to be met in order to have state based control applied by FortiNAC.
Technical Tip: 'State based Control' concept and VLAN changes	How FortiNAC applies state-based control and the enforcement groups for each host state.
Technical Tip: Investigate Policy/Access Enforcement events for Wireless connecting endpoints	How to check Network Events and identify enforcement actions performed on Hosts connecting Wireless.
Troubleshooting Tip: Wireless connected hosts are not matching Location Based Policies	Describes how to investigate cases where FortiNAC policies are not matching wireless connected hosts when the Location is used as matching criteria.
Technical Tip: Using the CLI to verify Adapter/Host/User attributes when troubleshooting Policies	Describes the CLI commands that can be used in FortiNAC-F to check element attributes when troubleshooting policies.
Technical Note: Troubleshooting location based Network Access policies	Check if Switch/port/SSID/AP has been added to the location group in the user host profile or learned by FortiNAC.
Troubleshooting Tip: Policy not matching with access values defined in user/host profile	Describes how to solve an issue where the policy does not match with access values defined in a user/host Profile
Technical Tip: Troubleshooting policies	Explains why the expected behavior for a particular host based on a policy (e.g. Captive Portal, Endpoint Compliance, Network Access, etc) is not occurring.
Troubleshooting Tip: VLANs not changing on a wired switch	Describes steps to take when the VLAN does not change as expected on a switch port after a host connects.
Technical Tip: What causes a host to be moved to an imported LDAP Host Group	Describes what causes a host to be moved to an imported LDAP Host Group.
Troubleshooting Tip: FortiNAC fails to move rogue switches to registration VLAN	How to solve an issue where FortiSwitch fails to move rogue switches to the registration VLAN.
Technical Tip: How to find Rogue devices in a network via FortiNAC GUI console	How to List Rogue devices in a network managed by FortiNAC.

Issues related to High Availability scenarios (HA), FortiNAC Manager and Database replications

TITLE	DESCRIPTION
Technical Tip: Potential Causes of a High Availability (HA) System to Fail Over	Potential triggers for a system to failover.
Troubleshooting Tip: FortiNAC Manager synchronization errors	Describes some examples of issues that cause synchronization failures between FortiNAC managers and servers.
Technical Tip: How to restore an old Database Backup from GUI	How to restore an old Database Backup from FortiNAC-F GUI.
Technical Tip: How to Access Secondary FortiNAC-F Server Configuration Wizard on HA mode with VIP (F...	Describes how to access the Secondary FortiNAC-F Server GUI in HA mode with shared IP.
TroubleshootingTip: Clear up uncompressed database backups causing DB Sync Issues	Delete uncompressed databases on the secondary.
Technical Tip: Manually resume control to primary server via CLI	Describes how to manually resume control of the primary FortiNAC server using the CLI.
Troubleshooting Tip: Database backup Failed Event after migrating from CentOS to FortiOS	Describes the behavior when the FortiNAC server migrates from CentOS to FortiOS and has the 'System Backup Failed: Backup script exit code = 1"

VPN Integrations and FSSO/FABRIC troubleshooting

TITLE	DESCRIPTION
Troubleshooting Tip: Troubleshooting FortiGate VPN integrations managed by FortiNAC.	Steps to debug and troubleshoot IPSec and SSL VPN integrations with FortiNAC-F.
Troubleshooting Tip: FortiNAC and FortiGate VPN IPSec log example	Check messages to look for when reviewing logs for FortiGate VPN IPSec integration with FortiNAC
Technical Tip: Configure Security Fabric with FortiNAC & Fortigate	How to quickly set the Security Fabric between FortiNAC and FortiGate to use Dynamic Address Tags
Technical Tip: Configuring and troubleshooting Firewall TAGs between FortiGate and FortiNAC-F or leg...	Configure and troubleshoot Firewall TAGs between FortiGate and FortiNAC-F
Troubleshooting Tip: Using the CLI tool 'ssotool' to view FSSO and Security Fabric connections in Fo...	How to use the CLI tool "ssotool" in the FortiNAC-F.

Issues related to Persistent Agent

TITLE	DESCRIPTION
Technical Tip: Troubleshooting the Persistent agent	Describes basic steps to troubleshoot Persistent Agent.
Technical Tip: Persistent Agent Authentication/login prompt is not appearing.	Describes how to troubleshoot and configuration checks needed when the login prompt does not appear after dowloading the Persistent Agent
Troubleshooting Tip: Windows Persistent Agent logs	Collecting Persistent Agent debug logs in Windows machines for troubleshooting purposes.
Technical Note: macOS Persistent Agent logs	Collect Persistent Agent debug logging in macOS for troubleshooting purposes.
Technical Note: Linux Persistent Agent Logs	Collect Persistent Agent logs in Linux machines for troubleshooting purposes
Troubleshooting Tip: Agent is not communicating and agent logs show 'CONN_DENY'	Describes how to understand the reason for seeing 'CONN_DENY' in general.txt agent logs.
Troubleshooting Tip: Connection issues with the Fortinet Persistent Agent	Persistent Agent is not seen as connected on the FortiNAC and the Agent does not do anything on the computer.

MDM Integration Issues

TITLE	DESCRIPTION
Troubleshooting Tip: MDM registration issues	Steps needed to identify why MDM users are not registering in FortiNAC
Troubleshooting Tip: Airwatch poll fails with HTTP 429 error code	How to resolve issues where FortiNAC is no longer registering or deleting devices that have been enrolled or deleted in Airwatch

Useful CLI commands for troubleshooting and transfering files

TITLE	DESCRIPTION
Technical Tip: Useful CLI commands in FortiNAC-OS for troubleshooting	Some useful commands introduced in the new version of FortiNAC running FortiNAC-OS.
Technical Tip: File Transfer with FTP for FortiNAC-F	How to transfer a file using FTP to the FortiNAC-F. This procedure requires a FTP server.
Technical Tip: Run tcpdump in FortiNAC-F and save capture as a file	How to write packets collected through tcpdump to a .pcap file that can later be shared with Fortinet Support to investigate specific issues.
Technical Tip: FileTransfer with TFTP for FortiNAC-F	Hhow to import some data to the FortiNAC-F directory by importing a Certificate via CLI or upgrading the FortiNAC-F via CLI.
Technical Tip: How to export DataBase backup from FortiNAC-F	How to export a specific DB backup from FortiNAC-F.