Description
This article describes advanced troubleshooting and collects information to deliver to Fortinet TAC for a support ticket.
Scope
FortiGate, FSSO and FortiAuthenticator, if used as a FSSO collector.
Solution
| Flow Chart. |
| 1. Preparations. |
| 2. Getting information from one issued PC. |
| 3 Validations on Windows AD Server. |
| 4. Validations on FSSO Collector Agent. |
| 5. Communication ports needed. |
| 6. Logon Event ID poller. |
| 7. DNS issues. |
| 8. FortiGate authentication debug. |
Flow Chart.
Use this flow chart as a troubleshooting guide, HTML file with high resolution is attached at the end of this article, it can be opened in a new browser tab for a better experience.
Some charts have a specific number (1) described in this article or reference to other Fortinet Community Link (A).
- Technical Tip: FSSO choose between DC Agent mode or Polling mode
- Technical Tip: Windows event IDs used by FSSO in WinSec polling mode
- Technical Tip: How to set source IP address for FSSO and LDAP
- Technical Tip: Upgrading FSSO Agents
- Technical Tip: How and why to use the 'Ignore User List' option in FSSO Collector Agent
- Technical Tip: Update AD group of the FSSO user without logging out or logging in
- Technical Tip: FSSO Windows Directory Access Methods - Standard versus Advanced Mode
- Technical Tip: FSSO Group Filter configured on Collector Agent
- Troubleshooting Tip: How to read FSSO CA debug logon events
- Preparations. It is required to identify which Fortinet Single Sign-On Collector Agent (FSSO-CA) server is active (in case of having more than one configured) there are two ways to identify:
- In the FortiGate go to Security Fabric -> External connectors -> FSSO. Put the mouse over the connector without selecting and wait a few seconds for a descriptive box to appear, and it will indicate which is the active server highlighted in bold (img-01).
- In FSSO-CA, select the ' Show service status' Button, and the one that has the FortiGate with the identified serial number will be the active FSSO, if more than one FSSO-CA server is configured, only one will show this information others will be blank in this menu, it is expected behavior, it is possible to restart/stop Fortinet Single sign On process to force change FSSO server. (img-02), take note of the server.
On FSSO-CA, set the log level to Debug, increase the file size to 50 MB, and log on events in separate logs.
Note:
The maximum log file size that can be set is 1024 MB.
These logs are written to C:\Program Files (x86)\Fortinet\FSAE\CollectorAgent.log and Logon_event.log. CollectorAgent.log will roll over to a C:\Program Files (x86)\Fortinet\FSAE\CollectorAgent.bak file, which extends the time the log spans. If necessary, copy both logs to another location and review them.
If FSSO-CA is configured to work in DC Agent mode (see img-06), be sure the log is enabled for troubleshooting: Technical Tip: How to enable logging on DC Agent (FSSO DC Agent mode)
Important Note 1.
FSSO-CA can work in agent or polling mode to know the differences. Read this KB article: Technical Tip: Comparison between DC-Agent mode and polling mode
Keep in mind that to change from polling to agent, FSSO-CA will install the dcagent.dll in each selected AD server and will be rebooted.
When 'OK' is selected, the FSSO service will be restarted and the FSSO server may change in the FortiGate External Connector.
- Getting information from one issued PC. In an issued PC (under Windows domain and not identified by FSSO), open a cmd terminal and run the following commands:
| echo %logonserver% | It provides AD <Server Name> where PC has been authenticated. |
| nslookup <ServerName> | Replace <ServerName> for value in previous command to get IP resolution. |
| hostname | It provides the PC's hostname. |
| echo %username% | It provides a user account. |
| ipconfig /all | It provides IP and DNS information. |
| whoami /groups | It provides information about user groups. |
- Validations on Windows AD Server: Validation can be done via Event Viewer or PowerShell. Event Viewer validation:
- Access to the <ServerName>.
- Open Event Viewer -> Windows Logs -> Security.
- Make sure that the 'Event ID' and 'Task Category' columns are visible, as shown in the img-04.
- Look for the last 'logon event' registered for the user of issued PC and force a new logon event-id.
- The Logon Event ID would be 4624.
The steps to follow are:
- Refresh Logons.
- Find… write username.
- Find Next…
- Find the last login record, and take note of the time and event ID.
- Return to the issued PC, lock and unlock the computer (Windows Key+L), once unlocked, make sure again that the PC indicates that it is logged in to the same AD server (command: echo %logonserver%).
- Again on the AD server, repeat step A-D to display the new logon event, compare the time and the Event-ID of this new record, Document both records and deliver to the TAC.
PowerShell validation:
Open PowerShell CLI as administrator (elevated prompt) and execute the following command, change 192.168.95.19 to the IP of the polled domain controller:
Get-Winevent -ComputerName 192.168.95.19 -FilterHashtable @{LogName='Security';ID='4768','4624';StartTime=((Get-Date).AddMinutes(-10))}| Where-Object -Property Message -Match 'user1' | select-object -Property *
Sample output of the command:
ProviderName: Microsoft-Windows-Security-Auditing
TimeCreated Id LevelDisplayName Message
----------- -- ---------------- -------
11/28/2024 1:03:09 PM 4624 Information An account was successfully logged on....
Important Note 2.
If the AD Server does not register the new logon events, the issue must be reviewed at the Domain level, since it is a problem between the PC and the Domain Server. FSSO reads those events, if there are none, no event can be read.
- Validations on FSSO-CA: In case the new Logon-Event is registered correctly, proceed to open the FSSO-CA on the active server (identified in point 1), document the following configurations, take screenshots, and share them with the TAC (select each button indicated in red).
It is possible to share all collected information with the Fortinet TAC, here is the checklist:
- Point 2. Commands from the Issued-PC.
- Point 3. Logon event configuration, with the date and time of the affected user in the Event Viewer -> Security.
- Point 4. All the configurations of the FSSO-CA.
- Point 4. A user account with which the FSSO service is executed.
- Point 4. Logon event (log file saved in desktop).
- Point 4. Groups to which the user account with which the FSSO service is executed belongs.
If the user is an advanced FortiGate administrator, it is possible to continue with troubleshooting as well, based on the information obtained in point 2, there may be different scenarios, carry out the searches and confirm which one corresponds to the user:
- If the user searches by a user and NO results appear, then continue at point 6).
- If the user searches by a user and the hostname or IP is incorrect, then continue at point 7).
- If the user searches by hostname and the IP is incorrect, then continue at point 7).
- If the user searches by hostname and the username are incorrect, then continue at point 7).
- If the user searches by a user and the result returns the correct IP and Hostname, then continue with point 8).
- Communication ports are needed from the Domain Controller perspective (Where the DC Agent and/or the FSSO Collector Agent are installed).
Inbound:
- TCP/8003 – DC_Agent keepalive and push logon info to Collector Agent (SSL enabled/secure).
- UDP/8002 – DC_Agent keepalive and push logon info to Collector Agent.
- TCP/8001 – FortiGate to FSSO Collector Agent connection (SSL).
- TCP/8000 – FortiGate to FSSO Collector Agent connection.
- TCP/8000 – NTLM.
Outbound:
- TCP/135, TCP/139, UDP/137 – Workstation check, polling mode (fallback method).
- TCP/445 – Remote access to logon events, Workstation check (remote registry).
- TCP/389 – Group lookup using LDAP.
- TCP/636 – Group lookup using LDAPS.
- TCP/3268 – Group lookup using LDAP with global catalog.
- TCP/3269 – Group lookup using LDAPS with global catalog.
- UDP/53 – DNS for resolving hostnames of the logon events.
To check these ports and validate that they are correctly opened, Telnet tests (TCP connections) must be executed from the FortiGate to the DC.
When there are DCAgents installed on some of the Domain Controllers, it needs 8002 or 8003 port depending on if it is secure or not (8002-NoSecure or 8003-Secure) for the FortiGate connection is 8000 and 8001 depending on if it's secure or not (8000-NoSecure or 8001-Secure).
TCP/8003 – SSL enabled/secure.
UDP/8002 – No secure
TCP/8001 – SSL enabled/secure.
TCP/8000 – No secure
In case the communication for push logon info to Collector Agent is running on port 8002 (Not secure). Then this port cannot be validated from FortiGate because Telnet works over TCP and this service is over UDP.
In case the communication for push logon info to Collector Agent is running on port 8003 (SSL enabled/secure). Then this port can be validated from FortiGate because Telnet works over TCP.
The larger the user database is, the more important is fast communication. Use a packet capture tool like Wireshark, to verify the response times and leave the packet capture running for about 20 minutes to have a more representative set of packets. To review the protocols:
- DNS: use the filter dns.time > 2 to find DNS queries where the response took more than 2 seconds. It should normally be under 100ms or lower.
- LDAP: use the feature in Wireshark "GUI -> Statistics -> Service Response Time > LDAP" to get a view of the response times. These should also be normally below 100ms.
- Logon Event ID poller. Increase the level to '2' instead of '0' of visibility of LOGS in all the FSSO-CAs, On the main screen of the FSSO-CA. Select Advanced Settings -> Windows Security Event Logon -> Event IDs to poll. When 'OK' is selected, the service will be restarted and the FSSO server may change in FortiGate External Connector.
Related article:
Technical Tip: Windows event IDs used by FSSO in WinSec polling
- DNS issues. The DNS where the FSSO-CA is installed must be correctly updated concerning the hostnames of all computers, this synchronization must be resolved by the Windows Domain Administrator. It is also required that the computer equipment can dynamically update its IP every time it changes, either due to a network change or a change between Wireless and cable.
Below is an external link that may help resolve this. 'Windows DHCP Clients and DNS Dynamic Update Protocol'.
Configure DNS dynamic updates Windows server 2003
- FortiGate authentication debug. Open SSH session to the FortiGate, save all the output, and perform these diagnose commands:
diagnose debug disable
diagnose debug reset
diagnose debug application authd 8256
diagnose debug console timestamp enable
diagnose debug enable
diagnose debug authd fsso server-status <- this shows which Collector the FortiGate is connected to.
Lock and unlock the issued PC and wait 2 minutes, then stop the debug.
diagnose debug disable
diagnose debug reset
More Useful FSSO commands:
| diagnose debug authd fsso refresh-logons | Request the FSSO-CA to send the active users list to FortiGate. |
| diagnose debug authd fsso clear-logons | Clear user login info on FortiGate. Logins will be refreshed in the next polling cycle. Do not use without a filter! |
| diagnose debug authd fsso refresh-groups | Request the FSSO-CA to send the monitored groups list to FortiGate. |
| diagnose debug authd fsso server-status | Shows FortiGate connectivity status with the FSSO Collector Agent. |
| get user adgrp | List monitored groups. |
| exec fsso refresh | Manually refresh information from the DC (refresh AD and FSSO groups). |
Related articles:
Technical Tip: FSSO choose between DC Agent mode or Polling mode
Technical Tip: FSSO in DC Agent mode
Technical Tip: FSSO in polling mode
Technical Tip: FSSO Group Filter configured on Collector Agent
Technical Tip: How and why to use the 'Ignore User List' option in FSSO Collector Agent
Technical Tip: FSSO Windows Directory Access Methods - Standard versus Advanced Mode
Troubleshooting Tip: FortiGate cannot connect to FSSO Agent on Windows AD
