Technical Tip: Maximum VDOM mismatch causes HA split-brain when additional VDOMs are configured

[Matt_B]

Description	This article describes expected split-brain behavior when additional VDOMs are configured in a cluster with inconsistent VDOM licensing. While the issue is occurring, a message similar to the following is visible in hatalk: 'HA cannot be formed because the HA peer '<serial number>' has <M> vdoms. It exceeds the maximum number of vdoms allowed on this box, which only allows maximum <N> vdoms.'
Scope	FortiGates in High availability cluster.
Solution	Requirements to form a FortiGate Clustering Protocol (FGCP) HA cluster include those listed in v7.6.2 Administration Guide: Troubleshoot an HA formation. In addition, it is also required that no cluster member contains more configured VDOMs than are licensed for every cluster member. Cluster formation requirements also apply to an existing cluster. Consider an existing HA cluster with cluster members FGT-A and FGT-B. If FGT-A has an additional VDOM license applied, deploying additional VDOMs to the cluster will cause split-brain. FGT-A # config global FGT-A (global) # get system status | grep "Max number" Max number of virtual domains: 20 FGT-B # config global FGT-B (global) # get system status | grep "Max number" Max number of virtual domains: 10 FGT-A # config vdom FGT-A (vdom) # edit ? <vdom> Virtual Domain Name root vdom-02 vdom-03 vdom-04 vdom-05 vdom-06 vdom-07 vdom-08 vdom-09 vdom-10 Warning: The following configuration will cause split-brain in this scenario. FGT-A # config vdom FGT-A (vdom) # edit vdom-11 <-- Creating the eleventh VDOM causes split-brain. current vf=vdom-11:13 FGT-A (vdom-11) #  Split-brain is a serious condition where the cluster does not form and each device acts as primary. This will cause degraded network performance until the issue is resolved. See the article Technical Tip: High Availability - Split Brain. After configuring excess VDOMs on FGT-A, FGT-B will reject the other member's heartbeat packets and an informative error message shows in hatalk debug from either FortiGate. FGT-A (global) # diagnose debug application hatalk -1 Debug messages will be on for 30 minutes. FGT-A (global) # diagnose debug enable FGT-A (global) # <hatalk> parse options for 'FG6H1EBBBBBBBBBB', packet_version=8 <hatalk> HA cannot be formed because this box has 11 vdoms. It exceeds the maximum number of vdoms allowed on the HA peer 'FG6H1EBBBBBBBBBB', which only allows maximum 10 vdoms. FGT-B (global) # diagnose debug application hatalk -1 Debug messages will be on for 30 minutes. FGT-B (global) # diagnose debug enable FGT-B (global) # <hatalk> parse options for 'FG6H1EAAAAAAAAAA', packet_version=8 <hatalk> HA cannot be formed because the HA peer 'FG6H1EAAAAAAAAAA' has 11 vdoms. It exceeds the maximum number of vdoms allowed on this box, which only allows maximum 10 vdoms. Resolution: Delete the excess VDOMS. FGT-A # config vdom FGT-A (vdom) # delete vdom-11 FGT-A (vdom) #  Alternatively, purchase and apply a VDOM license to the secondary by following the steps in Technical Tip: How to activate a VDOM license from CLI. While a cluster is experiencing split-brain, physically isolating one of the members from the network by removing all network cables from it will resolve the network degradation caused by split-brain. Since, in this case, the cluster only has one member available to handle traffic, it is strongly recommended to only use this as a mitigation while continuing to troubleshoot the cause of a split-brain condition. Additional information: The typical VDOM license type is perpetual and will not expire, so it is not necessary to consider VDOM numbers for a functioning cluster to enter this state as a result of VDOM license expiry. However, if a cluster device is factory reset or formatted, the VDOM license key must be applied again before loading its intended configuration. The exception is the S-series subscription FortiGates, which can have additional VDOM support added as a subscription. This VDOM license type can expire, see Subscription-based VDOM license for FortiGate-VM S-series. If neither HA device supports additional VDOMs, attempting to configure an excess VDOM will be rejected and split-brain will not occur. FGT-A (vdom) # edit vdom-11 Could not create VD, all VD licenses have been used. Command fail. Return code -4 This issue is only relevant for models that support expanding the allowed number of VDOMs. Check a product's datasheet for its maximum and default number of supported VDOMs to verify if it supports VDOM expansion. VDOM licenses are tied to a device's serial number, see the article Troubleshooting Tip: FortiGate fails to join HA due to a VDOM License. If a cluster member is replaced and the original member has an additional VDOM license, contact Customer Support for assistance in generating a new license key for the replacement device.