|
This error happens when the DC agent’s queue gets too full with pending logon requests, going beyond its capacity.
This is likely in environments with heavy logon traffic: like in large companies where many users log in at the same time or if there are DNS resolution issues that slow down processing.
Signs of the Problem.
- FortiGate misses some user logon events.
- The authentication process takes longer than usual.
- Security logs might have gaps related to user activities.
How DC Agent Threads Work:
The FSSO DC agent is crucial for tracking and reporting user logins from the domain controller to the FortiGate. It does this by creating multiple threads, each handling different parts of the process.
Key Points About DC Threads:
- Creating Threads: When the DC agent starts, it sets up several threads, each handling tasks like reading logon events, resolving DNS names, and communicating with FortiGate.
- Managing the Event Queue: Logon events are lined up for processing by the DC agent. Each thread picks up an event and processes it according to its role. For instance, one thread might read the event while another resolves the DNS.
- Thread Pooling: The DC agent uses a thread pool, meaning it has a set number of threads ready to handle incoming requests. If the number of logon events exceeds what these threads can manage, a backlog builds up, and the queue gets longer.
In summary, inside the DCAgent, there are three threads. The primary thread monitors system logon notifications and places the logon events into a queue. The second thread processes the events in the queue, resolves workstation names to IP addresses if possible, and then sends the login information to the collector agent. The third thread sends a keepalive signal to the collector agent every 10 seconds.
Impact of Heavy Logon Traffic:
When many logon requests come in at once, especially during peak times, the DC agent’s threads can get overwhelmed. Each logon event has to go through several steps, including capturing the event, resolving DNS, and sending it to FortiGate. Any delay in these steps can create a bottleneck.
- Event Overload: If the event queue grows too large because the threads can’t keep up, the agent will start discarding logon events, leading to the 'Too Many Requests in the Queue, Discarding Logon Event' error.
- DNS Resolution Delays: DNS resolution is often a major cause of delays. If the DC agent spends too much time resolving DNS names for each logon event, it can cause threads to lag, further increasing the queue size.
This issue is mainly caused by:
- High Volume of Logon Requests: The DC agent receives more logon requests than it can handle, causing a backlog.
- Delays in DNS Resolution: The DC agent may struggle with DNS name resolution, slowing down the processing of logon events.
How to Fix It:
Step 1: Adjust FSSO DC Agent Settings:
One effective solution is to disable DNS name resolution on the FSSO DC agent:
- On the domain controller, open the Registry Editor.
- Go to this registry key:
HKEY_LOCAL_MACHINE\SOFTWARE\Fortinet\FSAE\collectoragent
- Add or change the value of donot_resolve to 1.
- Restart the DC agent service.
This change stops the DC agent from trying to resolve DNS names, which can significantly speed up processing time for each logon event and prevent the queue from overflowing.
Step 2: Review and Optimize the FSSO Setup.
Make sure the FSSO deployment is running efficiently:
- Check the Collector Agent: Ensure it is installed on servers with enough resources and good network connectivity.
- Load Balancing: Spread the load across multiple DC agents if there are many users.
Step 3: Keep an Eye on Logon Traffic:
Regularly monitor logon traffic and the performance of the DC agents. If needed, adjust the number of agents or their processing capacity to handle busy times.
Step 4: Tuning Worker Thread Count:
Depending on the system’s capacity, it is possible to tune the Worker Thread Count on the Collector Agent to use more threads, allowing it to handle more simultaneous logon events.
Suggestion: Worker Thread to 512 (Advanced Settings -> Worker thread count).
Step 5: Balance the Load and Scale:
Distribute the processing load by deploying multiple DC agents across different domain controllers, especially in large environments. This helps ensure no single agent gets overwhelmed.
Step 6: Consider Hardware Upgrades:
If the problem continues even after optimizing settings, upgrade the hardware of the domain controllers running the DC agents to better handle the high volume of login requests.
If the issue is not resolved, reduce how often logon monitoring happens or consider upgrading the hardware of the domain controllers.
The 'Too Many Requests in the Queue, Discarding Logon Event' error happens because the DC agent cannot process logon events fast enough, often due to an overload of threads or DNS resolution delays. By understanding how the threads in the DC agent work and optimizing them, it is possible to reduce this error and ensure smoother login event processing.
Related article:
Technical Tip: Disable the DNS resolution of the FSSO DCAgent |
