Technical Tip: How to disconnect a member from a FortiGate HA cluster (aka 'Remove device from HA cluster')
Description
- IP addresses and administrative access settings on all interfaces are cleared to prevent conflicts with the remaining HA cluster. Existing interfaces are still retained.
- A new IP address is configured on a specified interface to allow continued network access for management and reconfiguration.
- The HA mode of the removed FortiGate is changed to standalone.
The removed FortiGate will retain the vast majority of its current configuration when removed from the cluster, including but not limited to:
- Firewall Policies.
- Interfaces/names (but as noted above, not IP address assignments).
- Admin users/passwords (including REST API admins/API keys, SSO Admins, etc.).
- Security Inspection profiles.
- HA configurations (e.g., group names, passwords, etc., but notably not HA management interfaces or the mode).
This approach is suitable for scenarios where the administrator would like to keep most of the configuration on the disconnected FortiGate. This is to make it easier to rejoin the HA cluster if required. If the disconnected FortiGate will be used for a different purpose, it is recommended to perform a factory reset instead to ensure all configurations are cleared.
Important notes:
Due to certain limitations of this feature, there may be scenarios where the IP addresses of some interfaces fail to be deleted from the FortiGate being removed. This can potentially cause a split-brain issue after removal. Therefore, it is highly recommended to pre-check the configuration dependencies mentioned below to determine whether to proceed with this approach or to perform a factory reset on the FortiGate instead.
Scope
FortiGate High Availability, All FortiOS versions since v5.2/v5.4.
Solution
Required pre-check step:
Before proceeding with HA disconnection using this method, it is necessary to verify whether any configuration references exist that could prevent interface IP addresses from being removed.
When a FortiGate is disconnected from an HA cluster, interface IP addresses are removed using the same logic as the CLI command 'unset ip'. If an interface has configuration dependencies that prevent the IP address from being unset, the interface IP will remain configured after the unit becomes standalone.
For example, if an interface is referenced by a BFD neighbor configuration as shown below, the IP address assigned to port1 will not be removed.
config system interface
edit "port1"
set vdom "root"
set ip 10.56.242.195 255.255.252.0
set allowaccess ping https ssh http telnet
set type physical
set snmp-index 1
next
end
config router bfd
config neighbor
edit 10.56.242.129
set interface "port1"
next
end
end
This interface is used in router.bfd.neighbor.
Please delete the bfd neighbor there first.
Command fail. Return code -23
In this scenario, the IP address on port1 remains after HA disconnection. If network cables remain connected, this may result in an IP conflict or a split-brain condition.
Therefore, it is necessary to check for any configuration dependencies that could prevent interface IP addresses from being removed. The configuration below could be a potential dependency that stops IP removal:
- Any interface being referenced by the BFD neighbor under 'config router bfd' (as shown above).
- Any interface referenced by OSPF, where an interface IP address is explicitly specified. Example is below:
config router ospf
config ospf-interface
edit "port1"
set interface "port1"
set ip 10.56.242.195 <--- This specified IP address is port1's IP.
next
end
end
Attempting to unset the IP address in this case results in the following error:
yoda-kvm06 (port1) # unset ip
This ip is used in router.ospf.ospf-interface.
Please unset the ip there first.
Command fail. Return code -23
If the IP address is not explicitly specified and remains at the default value of 0.0.0.0/0, no dependency issue occurs.
- Any interface IP address referenced as a source IP in system configurations such as FortiGuard, DNS, LDAP, and others. For example:
config system fortiguard
set source-ip 10.56.242.195
end
Attempting to unset the IP address in this case results in the following error:
site1 (port1) # unset ip
Error: IP address 10.56.242.195 is configured as source-ip for system.fortiguard
Command fail. Return code -23
The article below can be used to verify the use of the source-IP setting in the configuration:
Technical Tip: CLI command to check the use of 'source-ip' setting in configuration
Caution: If any of the above dependencies are present, the IP address of the affected interface will not be removed from the disconnected FortiGate. In this situation, disconnecting the FortiGate from the HA cluster using this method may result in a split-brain condition. It is recommended to take one of the following actions:
- Manually disconnect all network cables from the FortiGate being removed, except for the heartbeat and management interfaces, before proceeding. This prevents the interfaces from responding to ARP requests and helps avoid split-brain conditions.
- Alternatively, perform a factory reset on the FortiGate directly.
If no configuration dependencies are found, or if all cables except the HA and management interfaces are disconnected on the FortiGate to be removed, this procedure can be performed using either the GUI or the CLI by following the steps. However, it is strongly recommended to carry out this operation during a maintenance window and to have local console access to each device.
Disconnecting a FortiGate from the HA cluster - GUI Method:
- Log in to the Primary FortiGate via the Web GUI.
- Navigate to System -> HA (in the Global VDOM, if VDOMs are enabled). A list of FortiGate cluster members will be present.
- Select the unit to disconnect, then select the Remove device from HA cluster button:
- Select an Interface from the drop-down list after the pop-up window appears. This interface will be used as a management interface for the removed FortiGate:
- Configure an IP/Netmask for the specified interface. This IP address should be reachable over the network so that the removed FortiGate can be managed, as all other interfaces on the removed FortiGate will have the IP addressing removed to avoid conflicts with the existing HA cluster.
- Select OK to commit the change.
Disconnecting a FortiGate from the HA cluster - CLI Method (all FortiOS versions):
As mentioned previously, the existing HA cluster members will continue to provide service to the network without disruption, and the disconnected FortiGate will be accessible via the above interface and IP address, provided that appropriate routing exists to reach the device.
Post-check steps:
After disconnecting a unit from the HA cluster, it is suggested to:
-
Log in to the device via the new management IP or console.
-
Verify that all interface IP addresses have been successfully removed.
-
For any IP address that remains due to dependency:
-
Remove all related configuration dependencies and unset the IP.
-
Or, perform a factory reset on the unit directly.
-
201G-B-4627 # execute ha disconnect FGVM02TM2001363x mgmt 172.18.14.99 255.255.255.0
Starting disconnect self from HA cluster.
[__cli_action_hidden_entry:112] action error with hidden entry -23
failed to change interface port1 ip(-23)
WARNING: To avoid IP conflict, you need to manually change/unset the above interfaces' IP on the disconnected FGT.
But there are certain limitations for this feature in the v7.6 branch that:
-
The warning message is only shown on the unit disconnects itself.
-
If a unit disconnects another device from the cluster, the warning will not appear.
-
This visibility issue will be addressed in FortiOS v8.0.0.
Additionally, when the disconnect command to disconnect itself is executed through an IP-based CLI session (SSH or GUI CLI):
-
The warning does not appear
-
Because the FortiGate removes all IP addresses during the process, it causes the session drops before the warning can be displayed.
Note:
If the disconnected FortiGate is going to be reconnected to the cluster, consider the following:
- Reboot the disconnected unit before proceeding with cluster re-establishment, or run 'diagnose sys ha reset-uptime'. This ensures the cluster uptime value on the rejoining unit is lower than that of the current HA primary FortiGate, and helps to prevent the unit from attempting to assume the HA primary role.
- Assign a device priority that is lower than the existing HA primary FortiGate’s priority for the same reason. A higher priority value may cause a unit to assume the HA primary role (the default priority value is 128).
- For example, if the active Primary unit has a priority of 100, then it is recommended to set the disconnected FortiGate priority to a value lower than that.
- Reconfigure any custom HA settings that were in place before the disconnection, such as dedicated HA management interfaces.
- After the FortiGate rejoins the HA cluster, allow several minutes for the configuration to fully synchronize and for all interface configurations to be restored.
Related articles:
Troubleshooting Tip: FortiGate Cluster upgrade gets stuck when not all members have upgraded.
Technical Tip: How to confirm that Load Balancing is occurring (HA cluster).
Technical Tip: How to view the routing table on Slave/Secondary/Subordinate units in HA cluster.
Technical Tip: HA Reserved Management Interface's hidden VDOM (vsys_hamgmt VDOM).
Technical Tip: How to break a HA cluster and use one of the members as standalone
Troubleshooting Tip: Backup & Restore Cluster HA when there are many differences in tables