Troubleshooting Tip: High WAD user-info CPU usage due to too many discovered devices

In this scenario, this is caused by too many discovered devices in the User & Device store. This store takes the devices from the kernel, as seen in the IPS and the output of the following command:

diagnose user device list

This usually poses an issue in an environment where there are lots of transient devices (airport, hotel, coffee shop, ... etc.). These transient devices fill up the User & Device store and will put stress upon the wad user-info process if it needs to access this list due to the sheer length of it.

This typically is then observed with high CPU usage in the system space and WAD at the top of the diagnose sys top output.

FGT01 $ diagnose sys top
Run Time: 16 days, 5 hours and 30 minutes
0U, 0N, 3S, 97I, 0WA, 0HI, 0SI, 0ST; 48377T, 21142F
wad 4516 R 99.7 0.4 4

With the process ID of the wad process and the diagnose test app wad 1000 command, the specific WAD sub-process type can be determined.

FGT01 $ diagnose debug enable

FGT01 $ diagnose test app wad 1000

Process [35]: type=user-info(5) index=0 pid=4516 state=running

diagnosis=no debug=enable valgrind=supported/disabled

With the process ID of the wad process and the diagnose test app wad 1000, the specific WAD sub-process type can be determined.

WAD user-info: 

• Handle LDAP/AD Authentication.
• Maintain user and device related data to share with other daemons.
• User & Group Cache (LDAP/AD Auth) 
• LDAP server load balance.

To show the number of discovered devices, use the diagnose user device stats command.

FGT01 $ diagnose user device stats

generation.global 216394

generation.seen 174813

generation.deletion 0

count 122821

joined 0

create_failed 0

fd 12

hash 8192

Alternatively, this information can also be gathered by selecting the wad user-info process itself. For a detailed explanation of the WAD process and how to select a specific one, refer to Technical Tip: Overview of WAD process structure.

FGT01 $ diagnose test app wad 2500
AlBustan_FGT01 $ diag test app wad 168
Interface count 175841, Max Interfaces 507278
User count 0, Max Users 507278
wad_info_trigger stats.
interface_count 175841/175889 active/total
user_count 0/0 active/total
cid_updates_count 0/19 active/total
ems_updates_count 0/0 active/total
extra_info_update_count 0/6 active/total
mac_vdoms_count 175841/175889 active/total
fw_mac_addrs_count 1/1 active/total
ip_entries_count 0/0 active/total
wad_info_inventory stats.
wad_info_node stats.
interface_count 175841/175841 active/total
users_count 0/0 active/total
user_counts_count 0/0 active/total
devices_count 175841/175841 active/total
wad_info_attr stats
attr_count 2001371/22910871 active/total
attr_byte_count 15540792/177122229 active/total
wad_m_info_attr 95595632/95595690 Cur/Peak [active 2001371, alloc 22910871, free 20909500]
wad_m_info_node 265871592/265871592 Cur/Peak [active 703364, alloc 703364, free 0]
wad_m_info_inventory 1557/1557 Cur/Peak [active 16, alloc 49, free 33]
wad_m_info_trigger 33234015/33234015 Cur/Peak [active 351683, alloc 351779, free 96]

When the active devices as seen by diagnose user device list are deleted, they may still be in the device list. This is because FortiOS stores some historical data in case of a reboot, so some information persists during such an event.

If FortiGate is configured with managing FortiSwitch, High CPU usage on WAD user-info may cause the FortiSwitch Ports and FortiSwitch Clients GUI to become inaccessible. The GUI page will just keep on spinning / loading. Restart the WAD user-info process and the GUI will be accessible. 

WiFi & Switch Controller -> FortiSwitch Ports.
WiFi & Switch Controller -> FortiSwitch Clients.

FGT01 $ diagnose debug enable

FGT01 $ diagnose test app wad 1000

Process [35]: type=user-info(5) index=0 pid=4516 state=running

diagnosis=no debug=enable valgrind=supported/disabled

.....

FGT01 $ diagnose sys kill 11 4516

Note: 4516 is the PID seen from the 'diagnose test app wad 1000'command. Refer to Technical Tip: Overview of WAD process structure.

To reduce the number of idle devices discovered, use the command set discovered-device-timeout <days>. This will reduce devices stored to those devices discovered over the last set days. The default is 28 days. The following reduces the store to all devices discovered in the last day.

config system settings
    set discovered-device-timeout 1
end

Related articles:

• Technical Tip: How to control Discovered Device Timeout for Device Monitoring
• Technical Tip: How to disable the device database