Troubleshooting Tip: Troubleshooting a checksum mismatch in a FortiGate HA cluster

[bshingadia_FTNT]

Description

This article describes how to troubleshoot a checksum mismatch in a FortiGate High Availability (HA) cluster.

Scope

Any version of FortiGate in an HA cluster. This article focuses on versions below 5.6.

Solution

When two FortiGate units are in an HA cluster, they synchronize the configuration and act as a single unit, providing redundancy when one of the units fails.

Commands are set in global mode if VDOMs are in use.

The synchronization status of the two cluster units can be verified using the following command:

di sys ha cluster-csum

In FortiGate 5.6 and above, the command is:

diag sys ha checksum cluster

Example output:

================== FG100D3G13xxxxxx =================
 is_manage_master()=0, is_root_master()=0

debugzone
global: 89 f2 f0 0b e8 eb 0d ee f8 55 8b 47 27 7a 27 1e
root: cf 85 55 fe a7 e5 7c 6f a6 88 e5 a9 ea 26 e6 92
all: f4 62 b2 ce 81 9a c9 04 8f 67 07 ec a7 44 60 1f

checksum
global: 89 f2 f0 0b e8 eb 0d ee f8 55 8b 47 27 7a 27 1e
root: cf 85 55 fe a7 e5 7c 6f a6 88 e5 a9 ea 26 e6 92
all: f4 62 b2 ce 81 9a c9 04 8f 67 07 ec a7 44 60 1f

================== FG100D3G12xxxxxx ==================
is_manage_master()=1, is_root_master()=1

debugzone
global: 89 f2 f0 0b e8 eb 0d ee f8 55 8b 47 27 7a 27 1e
root: d8 f5 57 46 f0 b8 45 1e 00 be 45 92 a2 07 14 90
all: a7 8d cc c7 32 b5 81 a2 55 49 52 21 57 f9 3c 3b

checksum
global: 89 f2 f0 0b e8 eb 0d ee f8 55 8b 47 27 7a 27 1e
root: d8 f5 57 46 f0 b8 45 1e 00 be 45 92 a2 07 14 90
all: a7 8d cc c7 32 b5 81 a2 55 49 52 21 57 f9 3c 3b

The checksum for both of the units should match for the HA to work properly. If the checksum does not synchronize as shown in the output above, take the following steps in this article to synchronize the cluster checksums.

To identify which object(s) are not synchronized, it is typically most efficient to find the first out-of-sync object and correct it. Checksums are cumulative and so all checksums which follow the first out-of-sync object will also be incorrect.

1. Execute the following command on the slave unit to manually synchronize it with the master unit:

exec ha synchronize start

Verify the cluster-csum as mentioned above to see if the cluster is synchronized.

2. If the cluster is still not synchronized, execute the following command on both master and slave units and compare the output:

diagnose system ha showcsum 1

Example output:

system.global: d6c216d8449d75b2cd80110fa02a84e5

system.accprofile: 7df6f055a28e5d5216c4d2c2b3ee77d1

system.npu: 7df6f055a28e5d5216c4d2c2b3ee77d1

system.vdom-link: 7df6f055a28e5d5216c4d2c2b3ee77d1

wireless-controller.global: 7df6f055a28e5d5216c4d2c2b3ee77d1

wireless-controller.vap: cda65c180c25050eb83398fa23ab7fd1

system.switch-interface: cda65c180c25050eb83398fa23ab7fd1

system.interface: 56f2362fd69f51a2b6fc22a008c0c755

system.password-policy: 56f2362fd69f51a2b6fc22a008c0c755

system.sms-server: 56f2362fd69f51a2b6fc22a008c0c755

system.admin: af9c2b4f63e40551e33eabd64436fb3e

system.fsso-polling: af9c2b4f63e40551e33eabd64436fb3e

system.ha: ddfeff2ae037f615fbd83110169b70d2

FortiGate 1

FortiGate 2

diagnose system ha showcsum 1 system.global: d6c216d8449d75b2cd80110fa02a84e5 system.accprofile: 7df6f055a28e5d5216c4d2c2b3ee77d1 system.npu: 7df6f055a28e5d5216c4d2c2b3ee77d1 system.vdom-link: 7df6f055a28e5d5216c4d2c2b3ee77d1 wireless-controller.global: 7df6f055a28e5d5216c4d2c2b3ee77d1 wireless-controller.vap: cda65c180c25050eb83398fa23ab7fd1 system.switch-interface: cda65c180c25050eb83398fa23ab7fd1 system.interface: 56f2362fd69f51a2b6fc22a008c0c755 system.password-policy: 56f2362fd69f51a2b6fc22a008c0c755 system.sms-server: 56f2362fd69f51a2b6fc22a008c0c755 system.admin: af9c2b4f63e40551e33eabd64436fb3e system.fsso-polling: af9c2b4f63e40551e33eabd64436fb3e system.ha: ddfeff2ae037f615fbd83110169b70d2

diagnose system ha showcsum 1 system.global: d6c216d8449d75b2cd80110fa02a84e5 system.accprofile: 7df6f055a28e5d5216c4d2c2b3ee77d1 system.npu: 7df6f055a28e5d5216c4d2c2b3ee77d1 system.vdom-link: 7df6f055a28e5d5216c4d2c2b3ee77d1 wireless-controller.global: 7df6f055a28e5d5216c4d2c2b3ee77d1 wireless-controller.vap: cda65c180c25050eb83398fa23ab7fd1 system.switch-interface: cda65c180c25050eb83398fa23ab7fd1 system.interface: 56f2362fd69f51a2b6fc22a008c0c755 system.password-policy: 56f2362fd69f51a2b6fc22a008c0c755 system.sms-server: 56f2362fd69f51a2b6fc22a008c0c755 system.admin: 18c534445cb4e9f6fccdd0a101e31e69 system.fsso-polling: 18c534445cb4e9f6fccdd0a101e31e69 system.ha: a4de1e0fcd4add6c764ed10a61b9d022

Check the configuration parameter where the checksum mismatches. In the above example, the configuration checksum is mismatched in 'Admin Settings'.

3. Drill down to the object level.

To find out what exactly in the admin settings is causing the issue, repeat the command diagnose system ha showcsum with value 2 as shown below:

FortiGate 1

FortiGate 2

diag sys ha showcsum 2 admin.admin: vdom.root: 161f1834e4f7d7cefdc65f602f8116f5 dashboard-tabs.1: 0ce0ad276cc2a7f002ce9ac0b9ad073d dashboard-tabs.2: cb109689bf46e400e01aa1778d217523 dashboard-tabs.3: bc78e6c29c05fb268094e3ac06a6d507 dashboard-tabs.4: 7d9d58ccb29e38e5ff6b29c96e650b3c dashboard.1: 7d9d58ccb29e38e5ff6b29c96e650b3c dashboard.2: 7d9d58ccb29e38e5ff6b29c96e650b3c dashboard.3: 7d9d58ccb29e38e5ff6b29c96e650b3c dashboard.4: 7d9d58ccb29e38e5ff6b29c96e650b3c dashboard.21: 7d9d58ccb29e38e5ff6b29c96e650b3c dashboard.31: 7d9d58ccb29e38e5ff6b29c96e650b3c dashboard.41: 7d9d58ccb29e38e5ff6b29c96e650b3c dashboard.51: 5263dd1339cc964e43ea5e2799e53faa

diag sys ha showcsum 2 admin.admin: vdom.root: 161f1834e4f7d7cefdc65f602f8116f5 dashboard-tabs.1: 0ce0ad276cc2a7f002ce9ac0b9ad073d dashboard-tabs.2: cb109689bf46e400e01aa1778d217523 dashboard-tabs.3: bc78e6c29c05fb268094e3ac06a6d507 dashboard-tabs.4: 7d9d58ccb29e38e5ff6b29c96e650b3c dashboard.1: 7d9d58ccb29e38e5ff6b29c96e650b3c dashboard.2: 7d9d58ccb29e38e5ff6b29c96e650b3c dashboard.3: 7d9d58ccb29e38e5ff6b29c96e650b3c dashboard.4: 7d9d58ccb29e38e5ff6b29c96e650b3c dashboard.21: 7d9d58ccb29e38e5ff6b29c96e650b3c dashboard.31: 7d9d58ccb29e38e5ff6b29c96e650b3c dashboard.41: 7d9d58ccb29e38e5ff6b29c96e650b3c dashboard.51: 7d9d58ccb29e38e5ff6c89483effd00ca

As shown in the snippet above, an additional dashboard was configured in the 'admin' user configuration which caused the configuration to go out-of-sync. Removing that dashboard from admin user configuration will cause the HA checksum to sync again.

Alternatively, use the command 'diagnose system ha showcsum <path.object>' to find out the source of the mismatch in the admin section. This will also show the root cause of the mismatch as shown below:

(Note: As of 5.6 and above, the command is 'diag sys ha checksum show global system.admin'.)

FortiGate 1

FortiGate 2

diag sys ha showcsum system.admin admin.admin: vdom.root: 161f1834e4f7d7cefdc65f602f8116f5 dashboard-tabs.1: 0ce0ad276cc2a7f002ce9ac0b9ad073d dashboard-tabs.2: cb109689bf46e400e01aa1778d217523 dashboard-tabs.3: bc78e6c29c05fb268094e3ac06a6d507 dashboard-tabs.4: 7d9d58ccb29e38e5ff6b29c96e650b3c dashboard.1: 7d9d58ccb29e38e5ff6b29c96e650b3c dashboard.2: 7d9d58ccb29e38e5ff6b29c96e650b3c dashboard.3: 7d9d58ccb29e38e5ff6b29c96e650b3c dashboard.4: 7d9d58ccb29e38e5ff6b29c96e650b3c dashboard.21: 7d9d58ccb29e38e5ff6b29c96e650b3c dashboard.31: 7d9d58ccb29e38e5ff6b29c96e650b3c dashboard.41: 7d9d58ccb29e38e5ff6b29c96e650b3c dashboard.51: 5263dd1339cc964e43ea5e2799e53faa

diag sys ha showcsum system.admin admin.admin: vdom.root: 161f1834e4f7d7cefdc65f602f8116f5 dashboard-tabs.1: 0ce0ad276cc2a7f002ce9ac0b9ad073d dashboard-tabs.2: cb109689bf46e400e01aa1778d217523 dashboard-tabs.3: bc78e6c29c05fb268094e3ac06a6d507 dashboard-tabs.4: 7d9d58ccb29e38e5ff6b29c96e650b3c dashboard.1: 7d9d58ccb29e38e5ff6b29c96e650b3c dashboard.2: 7d9d58ccb29e38e5ff6b29c96e650b3c dashboard.3: 7d9d58ccb29e38e5ff6b29c96e650b3c dashboard.4: 7d9d58ccb29e38e5ff6b29c96e650b3c dashboard.21: 7d9d58ccb29e38e5ff6b29c96e650b3c dashboard.31: 7d9d58ccb29e38e5ff6b29c96e650b3c dashboard.41: 7d9d58ccb29e38e5ff6b29c96e650b3c dashboard.51: 7d9d58ccb29e38e5ff6c89483effd00ca

In a multi-VDOM environment, the above commands can be executed on a global level and for each VDOM. The following is the syntax for the command for individual VDOMs:

diagnose system ha showcsum <level> <vdom>      

<level> can be any value from 01 to 04 (the leading zero is important). For example:

diagnose system ha showcsum 01 root

(NOTE: From 5.6 and above, levels no longer exist. The command is 'diag sys ha checksum show <VDOM>'.)

Other tips:

Manually reconfiguring the object that is out of sync should trigger re-synchronization automatically. If necessary, manually trigger the start of synchronization with the following command:

execute ha synchronize start

This should cause the units to synchronize.

If the object in question is identical on all cluster members, it may just be the checksum that needs recalculating:

diag sys ha csum-recalculate

For newer versions, the command is:

diagnose sys ha checksum recalculate

Verify the checksum of the two units is synchronized as below:

For newer versions the command is:

diagnose sys ha checksum cluster

FortiGate 1

FortiGate 2

diagnose sys ha checksum cluster  ================== FG100D3G13xxxxxx ================== is_manage_master()=0, is_root_master()=0 debugzone global: 89 f2 f0 0b e8 eb 0d ee f8 55 8b 47 27 7a 27 1e root: cf 85 55 fe a7 e5 7c 6f a6 88 e5 a9 ea 26 e6 92 all: f4 62 b2 ce 81 9a c9 04 8f 67 07 ec a7 44 60 1f checksum global: 89 f2 f0 0b e8 eb 0d ee f8 55 8b 47 27 7a 27 1e root: cf 85 55 fe a7 e5 7c 6f a6 88 e5 a9 ea 26 e6 92 all: f4 62 b2 ce 81 9a c9 04 8f 67 07 ec a7 44 60 1f

di sys ha checksum cluster ================== FG100D3G12xxxxxx ==================   is_manage_master()=1, is_root_master()=1 debugzone global: 89 f2 f0 0b e8 eb 0d ee f8 55 8b 47 27 7a 27 1e root: cf 85 55 fe a7 e5 7c 6f a6 88 e5 a9 ea 26 e6 92 all: f4 62 b2 ce 81 9a c9 04 8f 67 07 ec a7 44 60 1f   checksum global: 89 f2 f0 0b e8 eb 0d ee f8 55 8b 47 27 7a 27 1e root: cf 85 55 fe a7 e5 7c 6f a6 88 e5 a9 ea 26 e6 92 all: f4 62 b2 ce 81 9a c9 04 8f 67 07 ec a7 44 60 1f