Fortigate HA in-sync/out-of-sync

[Baboda]

2016-10-04T14:00:20.092804+02:00 date=2016-10-04 time=14: 00:13 devname=FG800C3913801910 devid=FG800C3913801910 logid=0100037903 type=event subtype=system level=information vd="root" msg="The sync status with the master" sync_type=configurations sync_status="in-sync" 2016-10-04T14:00:25.128612+02:00 date=2016-10-04 time=14: 00:19 devname=FG800C3913801910 devid=FG800C3913801910 logid=0100037903 type=event subtype=system level=information vd="root" msg="The sync status with the master" sync_type=configurations sync_status="out-of-sync"

I have a lot of logs like the ones above. In fact it seems that master and slave are not in sync. The synchronization status of the two cluster units :

FG800C3913801256 (global) # di sys ha cluster-csum ================== FG800C3913801256 ================== is_manage_master()=1, is_root_master()=1 debugzone global: 94 ff 8c be 11 c1 c0 57 4d d7 73 dd c0 f0 46 db vdc: 3e 57 22 61 f0 58 af 18 66 9c b1 c9 ae 30 ac 75 root: 74 1b f9 bd 5c 41 a4 64 4f d3 1c b5 5c 5a 2c 84 WiFi: 12 17 17 a0 0c eb 1c 19 a8 4a aa 6d ae 5e b8 54 all: 46 ff b5 f4 54 d7 bf 22 c3 7f 4b 52 fb 03 a3 10 checksum global: 94 ff 8c be 11 c1 c0 57 4d d7 73 dd c0 f0 46 db vdc: 3e 57 22 61 f0 58 af 18 66 9c b1 c9 ae 30 ac 75 root: 74 1b f9 bd 5c 41 a4 64 4f d3 1c b5 5c 5a 2c 84 WiFi: 12 17 17 a0 0c eb 1c 19 a8 4a aa 6d ae 5e b8 54 all: 46 ff b5 f4 54 d7 bf 22 c3 7f 4b 52 fb 03 a3 10 ================== FG800C3913801910 ================== is_manage_master()=0, is_root_master()=0 debugzone global: 94 ff 8c be 11 c1 c0 57 4d d7 73 dd c0 f0 46 db vdc: 3e 57 22 61 f0 58 af 18 66 9c b1 c9 ae 30 ac 75 root: 74 1b f9 bd 5c 41 a4 64 4f d3 1c b5 5c 5a 2c 84 WiFi: 4f 3b c4 89 c9 4d e4 19 c6 24 9d 22 d6 9f 4f 8c all: 1e 56 a9 10 5b 8c c1 6d d6 12 38 fb 4a 3c 93 bc checksum global: 94 ff 8c be 11 c1 c0 57 4d d7 73 dd c0 f0 46 db vdc: 3e 57 22 61 f0 58 af 18 66 9c b1 c9 ae 30 ac 75 root: 74 1b f9 bd 5c 41 a4 64 4f d3 1c b5 5c 5a 2c 84 WiFi: 4f 3b c4 89 c9 4d e4 19 c6 24 9d 22 d6 9f 4f 8c all: 1e 56 a9 10 5b 8c c1 6d d6 12 38 fb 4a 3c 93 bc

However with diagnose sys ha showcsum 01 for each vdom and diagnose sys ha showcsum 1 on both master and slave units I've found there are no differences. So why ??? what shall I check yet ? maybe I should force recalculating checksum on slave or both units with command diag sys ha csum-recalculate ? or just forcing resync on the slave unit with the command exec ha synchronize start ?

[telecosistem]

Hello,

On this way, check first the hello packets knows as FGCP in Fortinet which number is 703 TCP.

Execute this command and attach the output.

diagnose sniffer packet any 'port 703' 4

Best regards,

[hklb · Accepted solution]

Hi,

Which firmware do you have ?

I had sometimes this issue with firmware 5.2.3. A diag sys ha csum-recalcutate on both unit will stop these log. 

Lucas

[TIBarigui]

Hi,   Which firmware do you have ?   I had sometimes this issue with firmware 5.2.3. A diag sys ha csum-recalcutate on both unit will stop these log.   Lucas

Passing by just to say that we were with this exactly problem.

Just did the recalculate-checksum and aparently it solved our problem.

[grunerit]

Hi

I have exactly the same problem with my 2-Node Cluster Version v5.4.4,build6003 (GA)

once per day we are getting 2 Messages/Mails mostly / always in the same time!

"out-of-sync" and "in-sync" !

Since we have Fortigate and Company!!!5 Months!

di sys ha cluster-csum -- show that  Cluster are in-sync!

On other 2-Node Cluster with the same Version v5.4.4,build6003 (GA) we didn't have this kind of messages

till last week!

What I did?:

I  create 1 FQDN Address Object and I create Firewall policy with this FQDN FQDN Address Object. FQDN Address Object: albaulicense.alsoft.net (resolved and O.K.)

Now I start getting this messages "out-of-sync" and "in-sync"  also on other HA! once per day and mostly always in the same time!

If I disable this policy, that I create above, messages doesn't stops, but if I delete this policy messages stopped coming!

Did somebody have Idea why and how can I resolve this problem and stop this messages coming?  

Thanks

gruner-it

-----

Message meets Alert condition date=2017-06-02 time=04:49:37 devname=xxxxxx devid=XXXXXXXXXXXXX logid=0108037903 type=event subtype=ha level=information vd=root logdesc="Synchronization status with master" msg="The sync status with the master" sync_type=external-files sync_status="out-of-sync"

Message meets Alert condition date=2017-06-02 time=04:50:22 devname=xxxxxx devid=XXXXXXXXXXXXX logid=0108037903 type=event subtype=ha level=information vd=root logdesc="Synchronization status with master" msg="The sync status with the master" sync_type=external-files sync_status="in-sync"

[James_G]

I get one out of sync message every night when the unit does it's ips signature update, I just live with it. My units are scheduled to pull updates at 1am and I see the same errors a few mi utes later every night.

[jim3cantos]

TIBarigui wrote:

Hi, Which firmware do you have ? I had sometimes this issue with firmware 5.2.3. A diag sys ha csum-recalcutate on both unit will stop these log. Lucas

Passing by just to say that we were with this exactly problem.

Just did the recalculate-checksum and aparently it solved our problem.

Problem solved here too recalculating checksum in master unit. The problem began when rebooting slave unit for testing. In 5.4 the command is: diagnose sys ha checksum recalculate

Reference: https://kb.fortinet.com/kb/documentLink.do?externalID=FD45183

James_G wrote:
I get one out of sync message every night when the unit does it's ips signature update, I just live with it. My units are scheduled to pull updates at 1am and I see the same errors a few minutes later every night.

This is normal behaviour.

[CWC]

Checksum recalculate worked for me as well. V6.4.4

[_kim_]

We have the same problem (Cluster in/out of sync) since upgrading from 6.4.7 to 7.0.5.

#diagnose sys ha checksum recalculate command does not help (have tried on both primary and secondary node)

Any suggestions how to fix this?

[AlexC-FTNT]

Check first what your logs are about:

 sync_type=external-files / configurations / etc
And then isolate that part of configuration that is different.

There is no universal fix for everyone!