Description
This article expands upon the FortiAuthenticator administration guide High Availability section, in particular setting up a load-balancing (Layer 3) cluster.
Related document:
High availability
Solution
FortiAuthenticator offers two different clustering modes – active-passive (Layer 2), and load-balancing (Layer 3):
With active-passive clustering, two (or more) FortiAuthenticators will appear as a single device to the wider network, much like a FortiGate cluster, with an HA link and shared IPs they are reachable on.
With a load-balancing setup, two (or more) FortiAuthenticators fundamentally act as single devices, but synchronize a large part of their databases.
They can act as backup to each other, but may be geographically separate and utilize completely different networks.
In a load-balancing setup, a single FortiAuthenticator (or an active-passive cluster) serves as primary unit, and up to ten single FortiAuthenticators may be added as load-balancing nodes.
There is no automated failover between them, and devices utilizing the FortiAuthenticators will need to switch to a different IP address to address a different node, as if switching to a completely independent device.
Only these items are synchronized in a load-balancing setup:
- Token and seeds.
- Local user database.
- Remote user database.
- Group mappings.
- Token and user mappings.
- Certificates included in Certificate Management > End Entities > Local Services, excluding firmware (Fortinet) certificates.
- Certificate Management > Certificate Authorities > Local CAs, including firmware (Fortinet) certificates.
- Certificate binding settings for local/remote user accounts.
- SAML configurations:
- IdP settings configured in Authentication > SAML IdP > General
- SP settings configured in Authentication > SAML IdP > Service Providers
Note:
Realm tables are not synchronized, but the default realm selection (radio button) is.
Note:
Admin users (local or remote users with role admin) are NOT synced by default. Each user has a toggle to enable synchronization of the admin user to the load-balancing node starting in 6.2. This needs to be set manually.
All other settings are NOT synchronized.
This includes any remote authentication servers, RADIUS/LDAP service settings, policies, etc.
Note: Remote Authentication Servers are synced in HA LB from FAC firmware version 6.6.0 onwards.
This in turn means some settings need to be undertaken on the load-balancing nodes before clustering and synchronization can be set up correctly.
Note:
Any load-balancing nodes will need to have the same licensing as the primary node/cluster if VM, or need to be the same hardware model as the primary node/cluster. They also need to run the same firmware version.
- Preparations. Start by fully configuring the primary standalone node (or the active-passive cluster that is supposed to act as primary in a load-balancing setup).
Note.
If possible, do not add FortiTokens to the FortiAuthenticator(s) yet; they can complicate initial synchronization if already present on the primary node(s) and assigned to users.
Take a note of all configuration except the user/groups, certificate and SAML settings. Any RADIUS/LDAP configuration, SSO configuration, Portal configuration, etc.
This will need to be recreated on any load-balancing node(s) manually.
This will also need to be recreated with the exact same names!
When users/groups are synced, they will contain references to remote authentication servers; for example, if a user belonging to server LDAP1 is synced to a load-balancing node, but the load-balancing node has no entry for a server LDAP1, the synchronization will fail!
Carefully consider the planned setup and where the primary and load-balancing nodes are located.
For example, if the primary is in one location and using the authentication server(s) there, and the load-balancing node is in a different location, it might make sense to point the load-balancing node to authentication servers at its location instead of the same as the primary node.
That way, if the entire infrastructure where the primary is located fails, the secondary will still have access to working servers.
If planned to have an active-passive cluster serving as the primary unit in a load-balancing setup, please note there are additional routing considerations:
- The load-balancing node must be able to reach the active-passive cluster HA interfaces.
- The active-passive cluster must be able to reach the load-balancing node via HA interfaces.
If the primary and load-balancing nodes utilize the same infrastructure (the same LDAP/RADIUS servers, the same clients, etc.), you can work around needing to manually recreate the configuration as follows (provided console access to the load-balancing node is in place):
- Take a backup of the primary
- Restore it on the load-balancing node
- Correct the IP/network settings to what they should be (via console, same CLI syntax as FortiGate, config system interface and config router static).
- Log into the GUI of the load-balancing node.
- Correct hostname, alias, DNS settings, and anything else that should be unique on the load-balancing node.
- If HA settings are already in place, disable HA again and save this..
- Enabling High Availability.
Once the primary is fully configured, and the load-balancing nodes have the prerequisite configuration in place, High Availability can be enabled.
On the primary, set the following under System > Administration > High Availability:
- Enable HA.
- Set the Role to 'Standalone Primary'.
- Set a pre-shared key to be used by the cluster.
- If an active-passive cluster serves as primary, skip step 1-3.
- Add the IP addresses of any load-balancing nodes in the 'Load Balancers' table. Up to ten may be added.
- If an active-passive cluster serves as primary, add a route to the load-balancing node IP on the primary; set the HA interface as outgoing interface.
A routing diagram of an active-passive cluster with load-balancing node:
On the load-balancing nodes, set the following System > Administration > High Availability:
- Enable HA.
- Set the Role to 'Load Balancer'.
- Add the primary IP address under 'Load Balancing primary IP address'.
- If the primary is an active-passive cluster, set the HA interface IP of the primary unit in the cluster.
- Set the same pre-shared key as on the primary (or used by the primary cluster).
Note.
No HA interface needs to be configured on the load-balancing unit; FortiAuthenticator will use the interface as indicated by its routing table to communicate with other nodes in the load-balancing setup.
If a cluster serves as primary node, leave the cluster HA settings untouched and only add the load-balancing nodes with their IP addresses.
If not done before, FortiTokens may now be added to the primary device/cluster. If Tokens were in place before High Availability was enabled, this can cause some synchronization issues with the FortiToken/User mappings.
- Viewing High Availability status and Troubleshooting. The HA status is visible under System > Dashboard > HA status. This should show either the primary or load-balancing nodes depending on which device the HA status is viewed. It will show the synchronization status; clicking on the node will provide details on what items are or are not synchronized. If there are errors, the icons for the affected items will show as red and clicking on the icons should show some additional information on where the error is.
A few common issues that can crop up on the load-balancing node:
Error: 'Foreign Key error: Entry for name=<> not found in <> on local server' or something similar
Solution: The entry with the specified name is not present on the load-balancing node in the specified table (like a remote LDAP server missing).
Create an appropriate entry with identical name to what is on the primary, add to realms/portals/policies as required, then synchronize again
Error: '<> with reference ID <> does not exist'.
Solution.
The specified item (like user or FortiToken) was not synchronized, or was synchronized but with a different reference in the database. This can sometimes happen if a backup was taken from primary and restored on secondary with things like User/FortiToken association already in place. Deleting the appropriate section (like remote users) and synchronizing it again from primary usually resolves this.
General troubleshooting steps for synchronization issues:
These are actions that can be done in general to attempt to resolve issues with load-balancing clusters:
- Rebuilding HA tables. This can be done under System -> Dashboard -> HA Status.It will cause the load-balancing node to rebuild the database from information synched from the primary. While this is going on, the load-balancing node may be slowed down. This does not trigger a reboot and does not affect the primary.
- Checking logs and /debug section. Some details on synchronization issues may show in FortiAuthenticator GUI logs under Logging -> Log Access -> Logs. Typing 'High Availability' into the search bar in the upper right will limit the display to relevant logs. There is a debug section available under https://<FortiAuthenticator>/debug. Debug sections may be selected in the drop-down menu in the upper left corner. The “LB” and “LB HA Sync” sections are particularly relevant. These logs can also be downloaded.
- Recalculating checksums. If the configuration appears to be in sync (same number of users, same groups present etc.), but HA status still shows as out of sync, recalculating the cluster checksums can resolve this. The cluster checksums can only be recalculated on the primary node under https://<FortiAuthenticator>/debug/lb_sync
