Description
This article provides some general recommendations for improving FortiNAC's day-to-day operations.
This includes disabling forgotten debugs, checking for enabled logs and specific configurations that can add unnecessary load and affect FortiNAC performance. These recommendations are based on previous experience with similar cases and numerous internal discussions surrounding these topics.
It is important to note that these recommendations may not apply to every case, as each situation can have its unique considerations and factors to take into account.
Scope
FortiNAC.
Solution
- For VM deployments, the resources need to be allocated according to the Data Sheet.
| Ports in the Network | Target Environment | vCPU Qty | Memory (GB) | Disk (GB) |
| Up to 2 000 |
Small |
8* | 16* | 100 |
| Up to 15 000 | Medium | 24 | 32 | 100 |
| Up to 25 000 | Large | 32 | 96 | 100 |
Note:
A minimum of 8 vCPU and 16GB of RAM is required.
- Plugin debugs are left enabled from the previous troubleshooting session. The following command can be used to find currently enabled debugs:
> nacdebug -true
Loader | Name | Debug
__________________________|______________________________|_____________________
45 MasterLoader |SnmpV1 |true
__________________________|______________________________|_____________________
How to disable them:
> nacdebug -name SnmpV1
Setting SnmpV1 debug to false
Note:
A reboot of the system will disable all these debugs.
- Debugs are set at the network device level. These debugs can be seen in the device configuration added as attributes:
> device -all | grep DEBUG
Name = DEBUG value = TelnetServer ForwardingInterface length = 32
For each line present, it means that there is a device in the list that has debugging enabled. To disable them, it is first necessary to find the device that has them currently enabled. The output of the command 'device -all' can be extracted to an external editor and manually searched for devices that have a similar line. The content can also be filtered by the grep tool used within FortiNAC CLI as follows:
> device -all | grep DEBUG -B 60 | grep "IP\|DEBUG"
IP = 10.0.0.1
Name = DEBUG value = TelnetServer length = 12
To remove these, debug run this command:
> device -ip 10.0.0.1 -delAttr -name DEBUG
Note:
A reboot of the system will not disable debugs at the device level.
- RADIUS logs are enabled from the GUI. When there is no troubleshooting going on, the RADIUS logs have to be set to Low or better, completely disabled, None:
Leaving these logs on for a long time will cause authentication latency, high disk usage, and a very big /var/log/radius/radius.log file.
- Check the L2/L3 polling to ensure they are not set too intensively. The default scheduled L2 polling interval is 10 minutes for wireless devices, 60 minutes for wired devices, and 30 minutes for L3 polling. During normal procedures, when polling is needed, FortiNAC will trigger it automatically. The main reason for scheduled polling is to correlate the already gathered information from other sources. Lowering the polling value will give the impression of a higher responsive time from FortiNAC, but in deployments that include many network devices, this will interfere with necessarily triggered polling, and that can cause delays in host status updates.
On switches that support SNMP MAC notification traps, the best practice is to use these traps instead of the standard linkUp and linkDown traps. When MAC Notification traps are implemented, FortiNAC does not have to read the forwarding tables (L2 polling) of the switches each time a host connects or disconnects from the network: Configuring Traps for MAC Notification.
Note:
In most cases, switches do not need to have L3 Polling enabled. It is only required in specific scenarios, for example, when an L3 switch is acting as a gateway for end hosts and therefore holds IP-to-MAC mappings in its ARP table. This is the only situation where enabling L3 Polling on a switch is necessary. The same logic applies to WLCs that operate purely at Layer 2.
Several integration guides suggest the recommended configurations for 3rd party device integration with FortiNAC that can be found here: FortiNAC Reference Manuals.
- LDAP recommendation and synchronization. Search Branches should not have an entry that is the base domain. This is easy to set up for testing purposes, but it will not work well in production. Scheduled synchronization does not have to be frequent, 1 day is the default and the recommended value. Manual synchronization can be run at any time if there are changes made to the AD. This will sync the user attributes and user groups. The user's passwords are checked in real time, and there is no need for synchronization. For deployments with multiple domain controllers for a single domain, it is suggested to put only one directory with one primary and secondary server. More info can be found here: Technical Note: Best practices for LDAP configuration.
- Loggers. Same as with plugin debugs, loggers can be enabled and forgotten. Related information can be found here: Technical Tip: NACDebug logger types.
- Aging. Depending on the setup requirements and user/host dynamics, in high turnover setups, it is suggested to put some Aging values for Inactive Hosts and Users:
More info can be found here: Technical Note: Modify aging of hosts and users.
- Database and older backups. Some options help manage the information stored and archived in the DB. The default settings should be ok for most types of deployments. In case they need to be increased, and need to be planned carefully. FortiNAC will take a backup of the DB every day and will clean up the backup files that are more than 90 days old:
In cases where these files get too big and start to occupy the disk, they can be deleted manually from the CLI in this path:
> ll /bsc/campusMgr/master_loader/mysql/backup
This is another example of a manual cleanup of the DB file for a specific input type via the CLI: Technical Tip: Database tables growing large due to constant port changes
- Disk utilization and cleanup. Disk utilization can be checked from the GUI, in the Dashboard: System Performance widget. In case of high disk usage, an alarm will also be generated. Usually, the cause of high disk utilization is big log files as a result of unnecessary debugs, database backups, message files, etc.
Searching for big files:
> df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 3.9G 0 3.9G 0% /dev
tmpfs 3.9G 0 3.9G 0% /dev/shm
tmpfs 3.9G 17M 3.9G 1% /run
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/mapper/centos-root 100G 3.7G 96G 4% /
/dev/sda1 509M 287M 222M 57% /boot
tmpfs 783M 0 783M 0% /run/user/0
Change the directory to root, and recursively check within any bigger folder that is listed:
> cd /
> du -sch *
.
600K tmp
2.1G usr
90.6G var
.
> cd var
> du -sch *
.
462M lib
88G log
.
> cd var
> du -sch *
.
10.4G messages.1
1.3M mysqld.log
20.5G radius
.
> cd radius
When a big file is found (this applies to log/text files only), for example,/var/log/messages.1 or /var/log/radius/radius.log.1 it can be easily emptied with the following command:
> echo > radius.log.1
Usually, these log files can be emptied and will not cause any undesired behavior.
- Resource utilization. This can be checked from the CLI using the sar utility that is already integrated into FortiNAC. It will show the values for every 10 minutes in the last 24 hours. For example, CPU, memory, and storage IOPS:
> sar -u
> sar -r
> sar -b
Example output of the command 'sar -u':
09:30:01 AM CPU %user %nice %system %iowait %steal %idle
09:40:01 AM all 0.98 0.00 0.57 0.05 0.00 98.40
09:50:01 AM all 1.12 0.00 0.49 0.02 0.00 98.37
10:00:01 AM all 0.80 0.00 0.50 0.02 0.00 98.69
- Network devices that are not properly configured in the Network Inventory. In deployments involving multiple network devices managed by different teams, changes made to these devices often do not get reflected in FortiNAC configurations. This can lead to delays in regular operational processes for other managed devices, such as polling. It is possible to check these warnings in the output.master file after changing the directory to > logs. The commands will go through the log file and count how many times FortiNAC has tried and failed to communicate with each device:
These devices are not properly configured under the network inventory:
grep "CLI credentials not filled in for device model" output.master | cut -d ' ' -f 14-30 | sort | uniq -c
752 CLI credentials not filled in for device model, cannot telnet/SSH to device, IP = 10.6.20.43
951 CLI credentials not filled in for device model, cannot telnet/SSH to device, IP = 10.8.10.14
The user configured under credentials may have limited rights:
grep "failed to execute CLI commands" output.master | cut -d ' ' -f 10-30 | sort | uniq -c
9 failed to execute CLI commands for Router_f1 at 10.6.20.44
7 failed to execute CLI commands for SW_Floor1 at 10.8.10.15
The SSH credentials may be changed, a new SSH fingerprint is generated, or a device is replaced after an RMA:
grep "failed to create an SSH2 session" output.master | cut -d ' ' -f 10-30 | sort | uniq -c
130 failed to create an SSH2 session for Router_f2 at 10.6.20.45
789 failed to create an SSH2 session for SW_Floor2 at 10.8.10.16
.
grep "Warning: failed to connect to" output.master | cut -d ' ' -f 9-19 | sort | uniq -c
129 Warning: failed to connect to Router_f3/10.6.20.46 No more authentication methods available. code = (14)
154 Warning: failed to connect to SW_Floor3/10.8.10.17 No more authentication methods available. code = (14)
Note:
If some of these errors are happening a few times, which is an indication of delayed communications or a device being offline for a specific period, no action needs to be taken. Otherwise, the configuration needs to be changed accordingly.
The device polling is sequentially processed, and there is a queue that shows the devices that are waiting to be polled. To get an overall view of the queue size, this command can be used:
grep "Queue Size" output.master*
output.master_3491.log.0.txt: Queue Size = 13
output.master_3491.log.0.txt: Queue Size = 10
The value of the queue size is recommended to be lower than 5, having higher values is an indication that the necessary polling is going to be delayed. This is mostly caused by the reason mentioned above and may increase during peak hours when the host's status changes frequently.
- The Network Device is sending MAC traps from the Uplink ports. The network device itself can be wrongly configured to send SNMP MAC traps for hosts learned in its Uplink ports or ports that are used to communicate between network devices. This information confuses FortiNAC and affects the performance because the same MAC address for an end host is reported in different locations of the network:
grep -ia "is UPLINK and received MAC notification" output.master | cut -d ' ' -f 9-25 | sort | uniq -c | sort -r
A total count of these issues/failures is shown in the 'eventsummary.txt' file that can be found under the 'misc' folder in a log snapshot file. An example output of this file:
name count
MAC_NOTIFICATION_UPLINK 8612
SSH_FAILURE 5634
TELNET_TIMEOUT 4389
READ_CLIENTS_FAILURE 4120
Use this article to Identify Devices Sending Mac Notification Traps from Uplink Ports. Once identified, disable MAC traps sent from Uplink ports.
- Multiple Network Devices sending Syslog Events to FortiNAC: When FortiNAC receives a large number of Syslog events that it is not able to parse, it will consume a lot of CPU and memory resources. Debugs needed to verify received Syslog messages in FortiNAC:
> nacdebug -name SyslogServer true
FortiNAC will print in output.master the following messages:
yams.SecurityEventManager WARNING :: 2023-08-28 16:29:38:332 :: #734 :: Invalid Filter Tag/Column
In such cases, it is necessary to disable Syslog IDs that do not need to be sent to FortiNAC. Taking FortiGate VPN integration as an example, only specific Syslog IDs are required to be sent to FortiNAC: FortiNAC FortiGate VPN Integration
The FortiGate section 'config log syslogd filter' specifies which logs to forward to FortiNAC. The logids: 0101039947, 0101039948, 0101037129, 0101037134 are the logs related to the tunnel up/down messages.
Related documents for Log messages:
Event
Log ID numbers
- A large number of 'Database lock time exceeded' events occurred due to FortiNAC's unable to handle RADIUS proxy requests. In these cases, FortiNAC cannot keep up with the number of Radius requests. The following event logs can be seen in FortiNAC:
RADIUS (Proxy) Time Threshold <Device_name> "RADIUS process time of 6 exceeded 5 Seconds."
RADIUS (Proxy) Time Threshold <Device_name> "RADIUS process time of 7 exceeded 5 Seconds."
RADIUS (Proxy) Time Threshold <Device_name> "RADIUS process time of 9 exceeded 5 Seconds."
In FortiNAC output.master file, multiple of the following messages are seen:
yams SEVERE :: 2024-04-17 11:14:43:178 :: #535 :: java.lang.Exception: Database lock time exceeded in HostServer!
yams SEVERE :: 2024-04-17 11:14:43:686 :: #72 :: java.lang.Exception: Database lock time exceeded in DynamicClient!
yams SEVERE :: 2024-04-17 11:14:43:743 :: #400 :: java.lang.Exception: Database lock time exceeded in thread ProxyProcessThread5
yams INFO :: 2024-04-17 11:14:57:152 :: #386 :: RadiusPollThread3 verifyServerLock locked processing server X.X.X.X
yams INFO :: 2024-04-17 11:14:57:152 :: #386 :: RadiusPollThread3 verifyServerLock unlocked processing serverX.X.X.X
Note:
FortiNAC's capacity for processing RADIUS requests is approximately 60 requests per second.
Capacity is affected by the use of other features in the program, such as the Persistent Agent or MAC Notification Traps. Any requests that are not immediately processed are placed in the queue. After 5 seconds, any unprocessed requests are discarded.
If FortiNAC is deployed in a busy environment where it is expected to receive more than 60 RADIUS requests per second, additional FortiNAC appliances will be required to handle the load.
Alternatively, the FortiNAC administrator can try to:
- Increase the number of threads used by FortiNAC to process requests to quickly release the processing queue and keep up with the number of requests.
Using nano or vi editor(NACOS), modify the following file to make the following changes during a maintenance window:
vi /bsc/campusMgr/master_loader/properties_plugin/radiusManager.properties
Change the value of the 3 following attributes as noted below (or increase them further):
com.bsc.plugin.radius.RadiusServer.pollThreads=20
com.bsc.plugin.radius.RadiusServer.proxyThreads=20
com.bsc.plugin.radius.RadiusServer.maxResponseThreshold=10000
After editing, save the file and confirm the change was applied:
cat /bsc/campusMgr/master_loader/properties_plugin/radiusManager.properties
The output should show the newly configured values. After that, restart the FortiNAC processes in the CLI:
restartNAC
- Increase the RADIUS timeout value in the WLC/SWITCH acting as NAS.
These changes may help prevent RADIUS re-authentication attempts, which can impact FortiNAC during busy periods when the system is stuck in processing already-received requests. Related documentation for RADIUS can also be found in the Administration guide.
- Device Profiler features are causing unnecessary DNS queries toward FortiGuard. In FortiNAC, it is possible to enable/disable features regarding Device profiler and rogue treatment. Go to System -> Settings -> User/Host Management -> Device Profiler.
If FortiGuard IoT service and profiling are not needed, it is recommended to disable both features. output.master logs may show show following failures:
yams.FingerprintServer WARNING :: 2025-01-14 14:01:50:550 :: #2892 :: FortiGuard request failed. Connect to globaldevcollect.fortinet.net:443 [globaldevcollect.fortinet.net/X.X.X.X, globaldevcollect.fortinet.net/X.X.X.X] failed: connect timed out
This will cause FortiNAC to send DNS queries toward root DNS servers and additionally increase CPU usage due to the continuous attempts to establish connections to FortiGuard.
Related documents:
