jheadley_FTNT
Description

This article reviews a technique for troubleshooting blade synchronization issues between a config master blade and one or more slave worker blades in a FortiGate 5000 (5k) series chassis.


Scope
FortiGate 5000 series blades configured in an SLBC configuration, which requires that the chassis contains at least 1 FortiController (also known as FortiSwitch-ATCA).

Following this procedure requires access to a text comparison/diff tool, such as Notepad++ with the Compare plugin.


Solution
1. On the FortiController, run get load-balance status to identify the config master blade (confsync master), as well as which blades are not in sync.

FortiController # get load-balance status

  ELBC Master Blade: slot-3

  Confsync Master Blade: slot-3
  Blades:
     Working:  6 [  6 Active  0 Standby]
     Ready:    0 [  0 Active  0 Standby]
     Dead:     0 [  0 Active  0 Standby]
    Total:     6 [  6 Active  0 Standby]

     Slot  3: Status:Working   Function:Active
       Link:      Base: Up          Fabric: Up
       Heartbeat: Management: Good   Data: Good
       Status Message:"Running"
     Slot  5: Status:Working   Function:Active
       Link:      Base: Up          Fabric: Up
       Heartbeat: Management: Good   Data: Good
       Status Message:"Waiting for configuration sync."
     Slot  7: Status:Working   Function:Active
       Link:      Base: Up          Fabric: Up
       Heartbeat: Management: Good   Data: Good
       Status Message:"Running"
     Slot  9: Status:Working   Function:Active
       Link:      Base: Up          Fabric: Up
       Heartbeat: Management: Good   Data: Good
       Status Message:"Running"
     Slot 11: Status:Working   Function:Active
       Link:      Base: Up          Fabric: Up
       Heartbeat: Management: Good   Data: Good
       Status Message:"Running"
     Slot 14: Status:Working   Function:Active
       Link:      Base: Up          Fabric: Up
       Heartbeat: Management: Good   Data: Good
       Status Message:"Running"

Steps 2 through 4 need to be executed on both the config master blade and the out-of-sync worker FortiGate blade.

2.  From global, run diagnose sys confsync showcsum and using the text compare tool, identify what line is out of sync (not matching) between the units. The last all line can be ignored, because it is a summary of all previous lines.

FG-blade3 (global) # diagnose sys confsync showcsum

debugzone
global: 3d 1e f3 53 26 8b 7a 4f 48 1a d8 21 11 a2 d8 d4
vdom4: db cb 68 bd 40 bb 71 68 2b 45 93 51 8a f0 e6 0d
vdom3: e7 17 f6 22 1b 2e 85 fd c8 d2 ea d1 23 a1 2f a2
vdom2: 36 a0 34 47 c2 ee c8 15 3b 08 54 a6 bd c7 bb 42
vdom1: 56 9c 7e 82 dd 17 83 d7 a5 a0 50 ec b6 04 ef ad
root: ff ca 50 c0 67 54 51 a5 c8 2a 6c 3e ad 17 dc 70
elbc-mgmt: 69 39 d4 2c 6a 36 cf aa d5 00 6b 63 22 cf 28 3b
all: d5 11 62 29 4f a5 af 14 95 29 08 fc cc 25 78 62

checksum
global: 3d 1e f3 53 26 8b 7a 4f 48 1a d8 21 11 a2 d8 d4
vdom4: db cb 68 bd 40 bb 71 68 2b 45 93 51 8a f0 e6 0d
vdom3: e7 17 f6 22 1b 2e 85 fd c8 d2 ea d1 23 a1 2f a2
vdom2: 36 a0 34 47 c2 ee c8 15 3b 08 54 a6 bd c7 bb 42
vdom1: 56 9c 7e 82 dd 17 83 d7 a5 a0 50 ec b6 04 ef ad
root: ff ca 50 c0 67 54 51 a5 c8 2a 6c 3e ad 17 dc 70
elbc-mgmt: 69 39 d4 2c 6a 36 cf aa d5 00 6b 63 22 cf 28 3b
all: d5 11 62 29 4f a5 af 14 95 29 08 fc cc 25 78 62

FG-blade5 (global) # diagnose sys confsync showcsum

debugzone
global: 3d 1e f3 53 26 8b 7a 4f 48 1a d8 21 11 a2 d8 b6
vdom4: db cb 68 bd 40 bb 71 68 2b 45 93 51 8a f0 e6 0d
vdom3: e7 17 f6 22 1b 2e 85 fd c8 d2 ea d1 23 a1 2f a2
vdom2: 36 a0 34 47 c2 ee c8 15 3b 08 54 a6 bd c7 bb 42
vdom1: 56 9c 7e 82 dd 17 83 d7 a5 a0 50 ec b6 04 ef ad
root: ff ca 50 c0 67 54 51 a5 c8 2a 6c 3e ad 17 dc 70
elbc-mgmt: 69 39 d4 2c 6a 36 cf aa d5 00 6b 63 22 cf 28 3b
all: d5 11 62 29 4f a5 af 14 95 29 08 fc cc 25 78 89

checksum
global: 3d 1e f3 53 26 8b 7a 4f 48 1a d8 21 11 a2 d8 b6
vdom4: db cb 68 bd 40 bb 71 68 2b 45 93 51 8a f0 e6 0d
vdom3: e7 17 f6 22 1b 2e 85 fd c8 d2 ea d1 23 a1 2f a2
vdom2: 36 a0 34 47 c2 ee c8 15 3b 08 54 a6 bd c7 bb 42
vdom1: 56 9c 7e 82 dd 17 83 d7 a5 a0 50 ec b6 04 ef ad
root: ff ca 50 c0 67 54 51 a5 c8 2a 6c 3e ad 17 dc 70
elbc-mgmt: 69 39 d4 2c 6a 36 cf aa d5 00 6b 63 22 cf 28 3b
all: d5 11 62 29 4f a5 af 14 95 29 08 fc cc 25 78 89


If the global line is unsynchronized go to step 3a.
If the global line is synchronized, but any specific vdom is unsynchronized, go to step 4a.


3a. From global, run diagnose sys confsync showcsum 1.

FG-blade3 (global) # diagnose sys confsync showcsum 1

system.global: f8b31181ae4b93ce5a6e8fbece51d2d1

system.accprofile: 7d79452c78377be2616149264a18fd5c
system.npu: 00000000000000000000000000000000
system.vdom-link: 00000000000000000000000000000000
wireless-controller.global: 00000000000000000000000000000000
wireless-controller.vap: 00000000000000000000000000000000
system.switch-interface: 00000000000000000000000000000000
system.lte-modem: 00000000000000000000000000000000
system.interface: be3f520521f5610d30fd936d65204b19
system.password-policy: 00000000000000000000000000000000
system.password-policy-guest-admin: 00000000000000000000000000000000
...
...
...
system.ntp: 5c774215d59f7231401cc64fe23c3045
system.vdom-radius-server: 00000000000000000000000000000000
system.geoip-override: 00000000000000000000000000000000
system.fortisandbox: 00000000000000000000000000000000
FG-blade5 (global) # diagnose sys confsync showcsum 1

system.global: f8b31181ae4b93ce5a6e8fbece51d2d1

system.accprofile: 7d79452c78377be2616149264a18fd5c
system.npu: 00000000000000000000000000000000
system.vdom-link: 00000000000000000000000000000000
wireless-controller.global: 00000000000000000000000000000000
wireless-controller.vap: 00000000000000000000000000000000
system.switch-interface: 00000000000000000000000000000000
system.lte-modem: 00000000000000000000000000000000
system.interface: be3f520521f5610d30fd936d65206578
system.password-policy: 00000000000000000000000000000000
system.password-policy-guest-admin: 00000000000000000000000000000000
...
...
...
system.ntp: 5c774215d59f7231401cc64fe23c3045
system.vdom-radius-server: 00000000000000000000000000000000
system.geoip-override: 00000000000000000000000000000000
system.fortisandbox: 00000000000000000000000000000000

3b. If system.interface is unsynchronized. In global, run diagnose sys confsync showcsum system.interface.

FG-blade3 (global) # diagnose sys confsync showcsum system.interface

base-mgmt: 5873dd45edd01f09c1ef2e7819369e8e
base1: b88429a8f1a433679999849ca1f49fd7
base2: d581b02347bdd9a33674fa8bc87ecb83
elbc-base-ctrl: b8405240b754710af36156b4ca2c0f5c
elbc-ctrl/1: b86091410fd51bce8878d35aa6164128
elbc-ctrl/2: 4e6671fcb459de05a1c051a3d4fa1786
fctrl1/f1-1: c84cfa7cef653aeb33df74bf037586a6
fctrl1/f1-2: 00245e3e391b140ecbcbbca551a408f2
fctrl1/f1-3: c436c089291f85c06eddafffd486e8fa
...
...
...
mgmt1: b8405240b754710af36156b4ca2c0f5c
mgmt2: b8405240b754710af36156b4ca2c0f5c
modem: 85c640a4dce9973a6e8bd1e249857822
port1: b8405240b754710af36156b4ca2c0f5c
port2: b8405240b754710af36156b4ca2c0f5c
ssl.elbc-mgmt: 7f6415d143bc8057ca9c8880cbbda0b5
ssl.root: 6648b2f38f05c8ca365b57b1ef96c04d
FG-blade5 (global) # diagnose sys confsync showcsum system.interface

base-mgmt: 5873dd45edd01f09c1ef2e7819369e8e
base1: b88429a8f1a433679999849ca1f49ff4
base2: d581b02347bdd9a33674fa8bc87ecb83
elbc-base-ctrl: b8405240b754710af36156b4ca2c0f5c
elbc-ctrl/1: b86091410fd51bce8878d35aa6164128
elbc-ctrl/2: 4e6671fcb459de05a1c051a3d4fa1786
fctrl1/f1-1: c84cfa7cef653aeb33df74bf037586a6
fctrl1/f1-2: 00245e3e391b140ecbcbbca551a408f2
fctrl1/f1-3: c436c089291f85c06eddafffd486e8fa
...
...
...
mgmt1: b8405240b754710af36156b4ca2c0f5c
mgmt2: b8405240b754710af36156b4ca2c0f5c
modem: 85c640a4dce9973a6e8bd1e249857822
port1: b8405240b754710af36156b4ca2c0f5c
port2: b8405240b754710af36156b4ca2c0f5c
ssl.elbc-mgmt: 7f6415d143bc8057ca9c8880cbbda0b5
ssl.root: 6648b2f38f05c8ca365b57b1ef96c04d

3c. If  base1 is unsynchronized. From global, run diagnose sys confsync showcsum system.interface base1

FG-blade3 (global) # diagnose sys confsync showcsum system.interface base1

[name]='base1': 5ffbc45e893c99b462c78391d1bde20f
[vdom]='elbc-mgmt': aaad9f28801aa465e0a4d2176aa2851e
[type]='physical': 39d37257932bbbeb5593b348f9a9ce57
[snmp-index]='8': 1a87c30a608e61b92337a02dc73a5210
FG-blade5 (global) # diagnose sys confsync showcsum system.interface base1

[name]='base1': 5ffbc45e893c99b462c78391d1bde20f
[vdom]='elbc-mgmt': aaad9f28801aa465e0a4d2176aa2851e
[type]='physical': 39d37257932bbbeb5593b348f9a9ce57
[snmp-index]='12': 1a87c30a608e61b92337a02dc73a435e

3d. Go to step 5.

4a. If vdom root is unsynchronized. From global, run
diagnose sys confsync cached-csum root
.

FG-blade3 (global) # diagnose sys confsync cached-csum root

system.object-tag: 5873dd45edd01f09c1ef2e7819369e8e
system.settings: 5873dd45edd01f09c1ef2e7819369e8e
system.sit-tunnel: 5873dd45edd01f09c1ef2e7819369e8e
system.arp-table: 5873dd45edd01f09c1ef2e7819369e8e
...
...
...
wireless-controller.wids-profile: 89b021d25c69bee5d44a9d4c5fe9ac1b
wireless-controller.wtp-profile: 2fb12986b481205b07555e106ab7f63d
wireless-controller.wtp: 5873dd45edd01f09c1ef2e7819369e8e
wireless-controller.wtp-group: 5873dd45edd01f09c1ef2e7819369e8e
wireless-controller.ap-status: 5873dd45edd01f09c1ef2e7819369e8e
...
...
...
system.wccp: 5873dd45edd01f09c1ef2e7819369e8e
system.nat64: 5873dd45edd01f09c1ef2e7819369e8e
FG-blade5 (global) # diagnose sys confsync cached-csum root

system.object-tag: 5873dd45edd01f09c1ef2e7819369e8e
system.settings: 5873dd45edd01f09c1ef2e7819369e8e
system.sit-tunnel: 5873dd45edd01f09c1ef2e7819369e8e
system.arp-table: 5873dd45edd01f09c1ef2e7819369e8e
...
...
...
wireless-controller.wids-profile: 89b021d25c69bee5d44a9d4c5fe9ac1b
wireless-controller.wtp-profile: 2fb12986b481205b07555e106ab7aeef
wireless-controller.wtp: 5873dd45edd01f09c1ef2e7819369e8e
wireless-controller.wtp-group: 5873dd45edd01f09c1ef2e7819369e8e
wireless-controller.ap-status: 5873dd45edd01f09c1ef2e7819369e8e
...
...
...
system.wccp: 5873dd45edd01f09c1ef2e7819369e8e
system.nat64: 5873dd45edd01f09c1ef2e7819369e8e

4b.
If wireless-controller.wtp-profile is unsynchronized. From vdom root, run diagnose sys confsync showcsum wireless-controller.wtp-profile.

FG-blade3 (root) # diagnose sys confsync showcsum wireless-controller.wtp-profile

AP-11N-default: 4475b2a896abcf7774c506d82d46ee2c
FAP11C-default: 0471938d10a76f389737a19c2f3cb213
FAP14C-default: d1402026614d827a5faef75a7a3be6ff
FAP21D-default: 7be0b59f941a5d7f91879bb8836dfd5b
...
...
...
FAPS421E-default: a84ca5f7c3192913aac152b82af3626d
FAPS422E-default: 6112ce6bff2328a3969b05e2f1a6c833
FAPS423E-default: 739c63cd4c94adacadba8803fafe6b23
FK214B-default: e32c1e6736ee68e30b372b0a66dade95
FG-blade5 (root) # diagnose sys confsync showcsum wireless-controller.wtp-profile

AP-11N-default: 4475b2a896abcf7774c506d82d46ee2c
FAP11C-default: 0471938d10a76f389737a19c2f3cb213
FAP14C-default: d1402026614d827a5faef75a7a3be6ff
FAP21D-default: 7be0b59f941a5d7f91879bb8836dfd5b
...
...
...
FAPS421E-default: a84ca5f7c3192913aac152b82af34faa
FAPS422E-default: 6112ce6bff2328a3969b05e2f1a6c833
FAPS423E-default: 739c63cd4c94adacadba8803fafe6b23
FK214B-default: e32c1e6736ee68e30b372b0a66dade95

4
c. If FAPS421E-default is unsynchronized. From vdom root, run diagnose sys confsync showcsum wireless-controller.wtp-profile FAPS421E-default.

FG-blade3 (root) # diagnose sys confsync showcsum wireless-controller.wtp-profile FAPS421E-default

[name]='FAPS421E-default': 1822fc08ae7ea391ff2e01b0c7c5d80b
[platform]:
[type]='S421E': ec08d031ba3352cb9b2e77e87886d3c7
[ap-country]='US': 95c3cb4094c6ac7cb42f823f7d45303e
[radio-1]:
[band]='802.11n': 2fc047dafb9d65c44294c71fe8114ee6
[radio-2]:
[band]='802.11ac': fa16a841577330f4ac2a658f0189b9a6
FG-blade5 (root) # diagnose sys confsync showcsum wireless-controller.wtp-profile FAPS421E-default

[name]='FAPS421E-default': 1822fc08ae7ea391ff2e01b0c7c5d80b
[platform]:
[type]='S421E': ec08d031ba3352cb9b2e77e87886d3c7
[ap-country]='CA': 95c3cb4094c6ac7cb42f823f7d4aac45
[radio-1]:
[band]='802.11n': 2fc047dafb9d65c44294c71fe8114ee6
[radio-2]:
[band]='802.11ac': fa16a841577330f4ac2a658f0189b9a6

4d. Go to step 5.

5. The mismatched settings in step 3d or step 4d is the specific configuration section that does not match between units because it cannot sync through the config sync process.

Manually copy that configuration section from the config master blade and paste into the slave worker blade.

Alternatively, take the backup configuration file from the config master blade and restore onto the out of sync slave blade.

6. After the correction all non-matching configuration, wait 2-3 minutes for the config sync process to detect the configurations are now in sync. Verify by performing step 1 again, this time ensuring that all blades have status of Running.

Recalculation Scenario:


If step 2 shows a mismatch, but step 3 or step 4 does not show any configuration that does not match between units, a checksum recalculation is required. From global, run the command below on both the config master blade and the out of sync blade(s).

        FG-blade3 (global) # diagnose sys confsync csum-recalculate

    FG-blade5 (global) # diagnose sys confsync csum-recalculate


Related Articles

Troubleshooting Tip: FortiGate 7000 Series blade config synchronization issues (confsync)

Contributors