Fortigate HA in-sync/out-of-sync

[Baboda]

2016-10-04T14:00:20.092804+02:00 date=2016-10-04 time=14: 00:13 devname=FG800C3913801910 devid=FG800C3913801910 logid=0100037903 type=event subtype=system level=information vd="root" msg="The sync status with the master" sync_type=configurations sync_status="in-sync" 2016-10-04T14:00:25.128612+02:00 date=2016-10-04 time=14: 00:19 devname=FG800C3913801910 devid=FG800C3913801910 logid=0100037903 type=event subtype=system level=information vd="root" msg="The sync status with the master" sync_type=configurations sync_status="out-of-sync"

I have a lot of logs like the ones above. In fact it seems that master and slave are not in sync. The synchronization status of the two cluster units :

FG800C3913801256 (global) # di sys ha cluster-csum ================== FG800C3913801256 ================== is_manage_master()=1, is_root_master()=1 debugzone global: 94 ff 8c be 11 c1 c0 57 4d d7 73 dd c0 f0 46 db vdc: 3e 57 22 61 f0 58 af 18 66 9c b1 c9 ae 30 ac 75 root: 74 1b f9 bd 5c 41 a4 64 4f d3 1c b5 5c 5a 2c 84 WiFi: 12 17 17 a0 0c eb 1c 19 a8 4a aa 6d ae 5e b8 54 all: 46 ff b5 f4 54 d7 bf 22 c3 7f 4b 52 fb 03 a3 10 checksum global: 94 ff 8c be 11 c1 c0 57 4d d7 73 dd c0 f0 46 db vdc: 3e 57 22 61 f0 58 af 18 66 9c b1 c9 ae 30 ac 75 root: 74 1b f9 bd 5c 41 a4 64 4f d3 1c b5 5c 5a 2c 84 WiFi: 12 17 17 a0 0c eb 1c 19 a8 4a aa 6d ae 5e b8 54 all: 46 ff b5 f4 54 d7 bf 22 c3 7f 4b 52 fb 03 a3 10 ================== FG800C3913801910 ================== is_manage_master()=0, is_root_master()=0 debugzone global: 94 ff 8c be 11 c1 c0 57 4d d7 73 dd c0 f0 46 db vdc: 3e 57 22 61 f0 58 af 18 66 9c b1 c9 ae 30 ac 75 root: 74 1b f9 bd 5c 41 a4 64 4f d3 1c b5 5c 5a 2c 84 WiFi: 4f 3b c4 89 c9 4d e4 19 c6 24 9d 22 d6 9f 4f 8c all: 1e 56 a9 10 5b 8c c1 6d d6 12 38 fb 4a 3c 93 bc checksum global: 94 ff 8c be 11 c1 c0 57 4d d7 73 dd c0 f0 46 db vdc: 3e 57 22 61 f0 58 af 18 66 9c b1 c9 ae 30 ac 75 root: 74 1b f9 bd 5c 41 a4 64 4f d3 1c b5 5c 5a 2c 84 WiFi: 4f 3b c4 89 c9 4d e4 19 c6 24 9d 22 d6 9f 4f 8c all: 1e 56 a9 10 5b 8c c1 6d d6 12 38 fb 4a 3c 93 bc

However with diagnose sys ha showcsum 01 for each vdom and diagnose sys ha showcsum 1 on both master and slave units I've found there are no differences. So why ??? what shall I check yet ? maybe I should force recalculating checksum on slave or both units with command diag sys ha csum-recalculate ? or just forcing resync on the slave unit with the command exec ha synchronize start ?

[marneu11]

Same thing here.

Some time after updating from 6.4.8 to 7.0.6 we see the sync status flapping.

I'm aware of how to find differences in the config which cause an out-of-sync status by checksums. This only possible for me when the status keeps being out-of-sync.

This here is different. The cluster is back in sync after 1,5 to 5 minutes. not much time to debug. Happening around 30 to 40 times a day and causing lots of monitoring and email alerts.

Mostly caused by "external-files" but also sometimes by "configuration" even if there is definitly nobody logged in for changes.

Any help much appreciated.

Thanks Markus

[AlexC-FTNT]

Hi Markus
The "external-files" reason is covered by the external resources used in the firewall. They are set up to refresh periodically from your configured servers.
For the "configuration" part you need to capture what configuration changes are made at that point. Enable informational logs and check them : which account made config changes, what changes ; verify with other admins.

[marneu11]

Hi Alex,

thanks for explaination. I was familiar with that. Nevertheless this was not the issue.

There where definitly no changes by admins. (we are team of 4 and do not work by night)

Reboot of the standby member fixed it immediatelly for me.

Best

Markus