Technical Tip: HA out of sync after upgraded in multi VDOMs environment
Description
This article describes how to fix HA out of sync after an upgrade in multi VDOMs environment.
When upgrading a cluster with multi VDOMs and managed by FortiManager, sometimes it will be out of sync as the SSL-SSH profile checksum is different. By removing SSL exempt from the SSL-SSH profile, it will make the cluster sync again.
Scope
FortiGate.
Solution
Troubleshooting:
The cluster is upgraded smoothly, but the slave unit is out-of-sync. Verify if the SSL-SSH profile 'custom-deep-inspection' is different. Run the following commands:
get sys ha status
FGyyy (updated 0 seconds ago): out-of-sync
FGxxx(updated 2 seconds ago): in-sync
FGyyy (updated 0 seconds ago): out-of-sync
FGxxx(updated 2 seconds ago): in-sync
diagnose sys ha checksum cluster
FGxxx
global: a0 f4 40 8d 35 29 b3 f8 7b 88 cd be ee 70 f9 a6
root: 37 98 9e 8a a6 52 c4 05 39 e5 3c 7c bd 16 62 b2
…
all: 63 20 fd 0a d9 27 07 f2 2a df ce 92 6e 83 4f 87
FGyyy
global: a0 f4 40 8d 35 29 b3 f8 7b 88 cd be ee 70 f9 a6
root: d4 3d 0b e2 f5 73 eb 7e 56 44 00 14 2d 3d 99 7a
…
all: 4e 3a ba 57 6e d6 34 25 53 ac 49 e9 bf b0 64 9c
FGxxx
global: a0 f4 40 8d 35 29 b3 f8 7b 88 cd be ee 70 f9 a6
root: 37 98 9e 8a a6 52 c4 05 39 e5 3c 7c bd 16 62 b2
…
all: 63 20 fd 0a d9 27 07 f2 2a df ce 92 6e 83 4f 87
FGyyy
global: a0 f4 40 8d 35 29 b3 f8 7b 88 cd be ee 70 f9 a6
root: d4 3d 0b e2 f5 73 eb 7e 56 44 00 14 2d 3d 99 7a
…
all: 4e 3a ba 57 6e d6 34 25 53 ac 49 e9 bf b0 64 9c
diagnose sys ha checksum show root
FGxxx
firewall.ssl-ssh-profile: b6205ab462401c94041894406b35b576
…
FGyyy
…
firewall.ssl-ssh-profile: f5277eb4bb806414c11216aa303140e5
…
FGxxx
firewall.ssl-ssh-profile: b6205ab462401c94041894406b35b576
…
FGyyy
…
firewall.ssl-ssh-profile: f5277eb4bb806414c11216aa303140e5
…
#dianose sys ha checksum show root firewall.ssl-ssh-profile
FGxxx
certificate-inspection: fa03bfe69adbe3f18e67a79a630e8e8f
custom-deep-inspection: a40395f9be9b3406e6f05c5712b3ebfb
deep-inspection: a1a2588c7b0df7715dcc674a2353f3f9
FGyyy
certificate-inspection: fa03bfe69adbe3f18e67a79a630e8e8f
custom-deep-inspection: 7f7d6b8625b969a31d21f12402a2f9fc
deep-inspection: a1a2588c7b0df7715dcc674a2353f3f9
FGxxx
certificate-inspection: fa03bfe69adbe3f18e67a79a630e8e8f
custom-deep-inspection: a40395f9be9b3406e6f05c5712b3ebfb
deep-inspection: a1a2588c7b0df7715dcc674a2353f3f9
FGyyy
certificate-inspection: fa03bfe69adbe3f18e67a79a630e8e8f
custom-deep-inspection: 7f7d6b8625b969a31d21f12402a2f9fc
deep-inspection: a1a2588c7b0df7715dcc674a2353f3f9
show firewall ssl-ssh-profile
FGxxx
config firewall ssl-ssh-profile
edit "custom-deep-inspection"
…
config ssl-exempt
…
edit 3
set type address
set address "google-play"
next
edit 4
set type address
set address "update.microsoft.com"
next
edit 5
set type address
set address "swscan.apple.com"
next
edit 6
set type address
set address "autoupdate.opera.com"
next
edit 7
set type wildcard-fqdn
…
edit 30
set type wildcard-fqdn
next
end
config firewall ssl-ssh-profile
edit "custom-deep-inspection"
…
config ssl-exempt
…
edit 3
set type address
set address "google-play"
next
edit 4
set type address
set address "update.microsoft.com"
next
edit 5
set type address
set address "swscan.apple.com"
next
edit 6
set type address
set address "autoupdate.opera.com"
next
edit 7
set type wildcard-fqdn
…
edit 30
set type wildcard-fqdn
next
end
FGyyy
config firewall ssl-ssh-profile
edit "custom-deep-inspection"
…
config ssl-exempt
…
edit 3
set type address
set address "google-play"
next
edit 4
set type address
set address "update.microsoft.com"
next
edit 5
set type address
set address "swscan.apple.com"
next
edit 6
set type address
set address "autoupdate.opera.com"
next
end
config firewall ssl-ssh-profile
edit "custom-deep-inspection"
…
config ssl-exempt
…
edit 3
set type address
set address "google-play"
next
edit 4
set type address
set address "update.microsoft.com"
next
edit 5
set type address
set address "swscan.apple.com"
next
edit 6
set type address
set address "autoupdate.opera.com"
next
end
Analysis:
- Check the user backup configuration file before upgrading. There was no custom-deep-inspection profile.
- The user uses FortiManager to manage the cluster. In FortiManager, under the same ADOM, the other cluster uses a custom-deep-inspection profile.
- The user may use a custom-deep-inspection profile in the future for this upgrade cluster.
- The primary unit contains 30 exemptions, and 4 of them do not resolve as there is no FQDN address. The secondary unit contains 4 not resolved FQDN addresses in exemptions, so it causes the cluster to become out-of-sync.
Solution:
- Upgrade to the latest.
- If the upgrade is not possible, remove 4 FQDN Addresses from 'Exempt from SSL Inspection' under 'customer-deep-inspection' from the primary unit to make the cluster sync again.
Note: In some rare cases, the checksum from the output of 'diagnose system ha checksum show <vdom>' or 'diagnose system ha checksum show global' may clearly reveal there is a mismatch in certain sections of the VDOM and the configuration (as shown below), but comparing the configuration file of both cluster members (using tools like the Notepad++ compare plugin) may show no difference. In cases like this, reboot the unit that is 'out-of-sync' or reboot both units if possible. Collect the configuration files again (from both units - primary and secondary), in the below example, that will be IPSec Phase1-Interface configuration files. The difference will be visible now. Make appropriate changes so the configuration on the Primary and Secondary unit became matched, and recalculate the checksum one more time on both (diagnose sys ha checksum recalculate) to sync the HA cluster.
root: 0d 6d 57 d6 68 3c b6 06 10 cd e9 30 56 b6 d4 17
root: 69 8e 91 6e 7f b6 44 4b be d9 58 39 62 c5 e2 70
vpn.ipsec.phase1-interface: a873595ba1f09002fca147b9214fcbc4
vpn.ipsec.phase1-interface: ca627e491119a39265cbd9c8c1d96e54
