Description
This article reviews a technique for troubleshooting config synchronization issues between a config master blade (one of the FIMs) and one or more config slave blades (the other FIM and all FPMs) in a FortiGate 7000 (7k) series chassis.
Scope
FortiGate 7000 series.
Following this procedure requires access to a text comparison/diff tool, such as Notepad++ with the Compare plugin.
Solution
1. Identify the FIM that is the 'config master' by running get system status.
Note that in a two-chassis (active-passive) configuration, only a single FIM in the active chassis will be the config master.
FG74E43E1xxxxx63 [FIM01] # get system status
==========================================================================
Slot: 2 Module SN: FIM04E3E1xxxxx64
Version: FortiGate-7040E v5.4.3,build6284,170714 (GA)
...
Serial-Number: FG74E43E16000063
Module Serial-Number: FIM04E3E1xxxxx64
Config-Sync: Slave
==========================================================================
Slot: 3 Module SN: FPM20E3E1xxxxx04
Version: FortiGate-7040E v5.4.3,build6284,170714 (GA)
…
Serial-Number: FG74E43E1xxxxx63
Module Serial-Number: FPM20E3E1xxxxx04
Config-Sync: Slave
==========================================================================
Current slot: 1 Module SN: FIM01E3E1xxxxx72
Version: FortiGate-7040E v5.4.3,build6284,170714 (GA)
…
Serial-Number: FG74E43E1xxxxx63
Module Serial-Number: FIM01E3E1xxxxx72
Config-Sync: Master
2. On the FIM identified in Step 1, run diagnose load-balance status from global to identify which blades are not in sync.
Note that if having a second chassis, it is also needed to run this command on one of the FIMs in the passive chassis.
FG74E43E1xxxxx63 [FIM01] (global) # diagnose load-balance status
FIM01: FIM01E3E1xxxxx72
Master FPM Blade: slot-3
Slot 3: FPM20E3E1xxxxx04
Status:Working Function:Active
Link: Base: Up Fabric: Up
Heartbeat: Management: Good Data: Good
Status Message:"Running"
Slot 4: FPM20E3E1xxxxx03
Status:Working Function:Active
Link: Base: Up Fabric: Up
Heartbeat: Management: Good Data: Good
Status Message:"Waiting for configuration sync"
Steps 3 through 5 require output from both the config master FIM and the out-of-sync FIM or FPM blade.
Note: it is possible to use the 'command broadcasting' feature to only run the command once on the FIM, which will (by default) query all the rest of the blades in the chassis for the same diagnostic output. It will then be needed to cut and paste the correct sections into the text comparison tool.
3. From global, run diagnose sys confsync showcsum and using the text compare tool, identify what line is out of sync (not matching) between the units. The last line (all) can be ignored because it is a summary of all previous lines.
Note that the same lines under debugzone will also appear under checksum.
|
FG74E43E1xxxxx63 [FIM01] (global) # diagnose sys confsync showcsum
debugzone
global: 3d 1e f3 53 26 8b 7a 4f 48 1a d8 21 11 a2 d8 d4
vdom4: db cb 68 bd 40 bb 71 68 2b 45 93 51 8a f0 e6 0d
vdom3: e7 17 f6 22 1b 2e 85 fd c8 d2 ea d1 23 a1 2f a2
vdom2: 36 a0 34 47 c2 ee c8 15 3b 08 54 a6 bd c7 bb 42
vdom1: 56 9c 7e 82 dd 17 83 d7 a5 a0 50 ec b6 04 ef ad
root: ff ca 50 c0 67 54 51 a5 c8 2a 6c 3e ad 17 dc 70
dmgmt-vdom: 69 39 d4 2c 6a 36 cf aa d5 00 6b 63 22 cf 28 3b
all: d5 11 62 29 4f a5 af 14 95 29 08 fc cc 25 78 62
checksum
global: 3d 1e f3 53 26 8b 7a 4f 48 1a d8 21 11 a2 d8 d4
vdom4: db cb 68 bd 40 bb 71 68 2b 45 93 51 8a f0 e6 0d
vdom3: e7 17 f6 22 1b 2e 85 fd c8 d2 ea d1 23 a1 2f a2
vdom2: 36 a0 34 47 c2 ee c8 15 3b 08 54 a6 bd c7 bb 42
vdom1: 56 9c 7e 82 dd 17 83 d7 a5 a0 50 ec b6 04 ef ad
root: ff ca 50 c0 67 54 51 a5 c8 2a 6c 3e ad 17 dc 70
dmgmt-vdom: 69 39 d4 2c 6a 36 cf aa d5 00 6b 63 22 cf 28 3b
all: d5 11 62 29 4f a5 af 14 95 29 08 fc cc 25 78 62
|
FPM20E3E1xxxxx03 [FPM04] (global) # diagnose sys confsync showcsum
debugzone
global: 3d 1e f3 53 26 8b 7a 4f 48 1a d8 21 11 a2 d8 b6
vdom4: db cb 68 bd 40 bb 71 68 2b 45 93 51 8a f0 e6 0d
vdom3: e7 17 f6 22 1b 2e 85 fd c8 d2 ea d1 23 a1 2f a2
vdom2: 36 a0 34 47 c2 ee c8 15 3b 08 54 a6 bd c7 bb 42
vdom1: 56 9c 7e 82 dd 17 83 d7 a5 a0 50 ec b6 04 ef ad
root: ff ca 50 c0 67 54 51 a5 c8 2a 6c 3e ad 17 dc 70
dmgmt-vdom: 69 39 d4 2c 6a 36 cf aa d5 00 6b 63 22 cf 28 3b
all: d5 11 62 29 4f a5 af 14 95 29 08 fc cc 25 78 89
checksum
global: 3d 1e f3 53 26 8b 7a 4f 48 1a d8 21 11 a2 d8 b6
vdom4: db cb 68 bd 40 bb 71 68 2b 45 93 51 8a f0 e6 0d
vdom3: e7 17 f6 22 1b 2e 85 fd c8 d2 ea d1 23 a1 2f a2
vdom2: 36 a0 34 47 c2 ee c8 15 3b 08 54 a6 bd c7 bb 42
vdom1: 56 9c 7e 82 dd 17 83 d7 a5 a0 50 ec b6 04 ef ad
root: ff ca 50 c0 67 54 51 a5 c8 2a 6c 3e ad 17 dc 70
dmgmt-vdom: 69 39 d4 2c 6a 36 cf aa d5 00 6b 63 22 cf 28 3b
all: d5 11 62 29 4f a5 af 14 95 29 08 fc cc 25 78 89
|
If the global line is unsynchronized (not matching) go to step 4a.
If the global line is synchronized (matching), but any specific VDOM is unsynchronized, go to step 5a.
4a. From global, run diagnose sys confsync showcsum 1.
|
FG74E43E1xxxxx63 [FIM01] (global) # diagnose sys confsync showcsum 1
system.global: f8b31181ae4b93ce5a6e8fbece51d2d1
system.accprofile: 7d79452c78377be2616149264a18fd5c
system.npu: 00000000000000000000000000000000
system.vdom-link: 00000000000000000000000000000000
wireless-controller.global: 00000000000000000000000000000000
wireless-controller.vap: 00000000000000000000000000000000
system.switch-interface: 00000000000000000000000000000000
system.lte-modem: 00000000000000000000000000000000
system.interface: be3f520521f5610d30fd936d65204b19
system.password-policy: 00000000000000000000000000000000
system.password-policy-guest-admin: 00000000000000000000000000000000
...
...
...
system.ntp: 5c774215d59f7231401cc64fe23c3045
system.vdom-radius-server: 00000000000000000000000000000000
system.geoip-override: 00000000000000000000000000000000
system.fortisandbox: 00000000000000000000000000000000
|
FPM20E3E1xxxxx03 [FPM04] (global) # diagnose sys confsync showcsum 1
system.global: f8b31181ae4b93ce5a6e8fbece51d2d1
system.accprofile: 7d79452c78377be2616149264a18fd5c
system.npu: 00000000000000000000000000000000
system.vdom-link: 00000000000000000000000000000000
wireless-controller.global: 00000000000000000000000000000000
wireless-controller.vap: 00000000000000000000000000000000
system.switch-interface: 00000000000000000000000000000000
system.lte-modem: 00000000000000000000000000000000
system.interface: be3f520521f5610d30fd936d65206578
system.password-policy: 00000000000000000000000000000000
system.password-policy-guest-admin: 00000000000000000000000000000000
...
...
...
system.ntp: 5c774215d59f7231401cc64fe23c3045
system.vdom-radius-server: 00000000000000000000000000000000
system.geoip-override: 00000000000000000000000000000000
system.fortisandbox: 00000000000000000000000000000000
|
4b. In this example, the system.interface is unsynchronized, so in global, run diagnose sys confsync showcsum system.interface to see specifics on what is not synchronized under this configuration section.
|
FG74E43E1xxxxx63 [FIM01] (global) # diagnose sys confsync showcsum system.interface
base-mgmt: 5873dd45edd01f09c1ef2e7819369e8e
base1: b88429a8f1a433679999849ca1f49fd7
base2: d581b02347bdd9a33674fa8bc87ecb83
elbc-base-ctrl: b8405240b754710af36156b4ca2c0f5c
...
...
...
1-mgmt1: b8405240b754710af36156b4ca2c0f5c
1-mgmt2: b8405240b754710af36156b4ca2c0f5c
1-mgmt3: 85c640a4dce9973a6e8bd1e249857822
1-mgmt4: b8405240b754710af36156b4ca2c0f5c
1-M1: b8405240b754710af36156b4ca2c0f5c
|
FPM20E3E1xxxxx03 [FPM04] (global) # diagnose sys confsync showcsum system.interface
base-mgmt: 5873dd45edd01f09c1ef2e7819369e8e
base1: b88429a8f1a433679999849ca1f49ff4
base2: d581b02347bdd9a33674fa8bc87ecb83
elbc-base-ctrl: b8405240b754710af36156b4ca2c0f5c
...
...
...
1-mgmt1: b8405240b754710af36156b4ca2c0f5c
1-mgmt2: b8405240b754710af36156b4ca2c0f5c
1-mgmt3: 85c640a4dce9973a6e8bd1e249857822
1-mgmt4: b8405240b754710af36156b4ca2c0f5c
1-M1: b8405240b754710af36156b4ca2c0f5c
|
4c. If base1 is unsynchronized. From global, run diagnose sys confsync showcsum system.interface base1.
|
FG74E43E1xxxxx63 [FIM01] (global) # diagnose sys confsync showcsum system.interface base1
[name]='base1': 5ffbc45e893c99b462c78391d1bde20f
[vdom]='dmgmt-vdom': aaad9f28801aa465e0a4d2176aa2851e
[type]='physical': 39d37257932bbbeb5593b348f9a9ce57
[snmp-index]='8': 1a87c30a608e61b92337a02dc73a5210
|
FPM20E3E1xxxxx03 [FPM04] (global) # diagnose sys confsync showcsum system.interface base1
[name]='base1': 5ffbc45e893c99b462c78391d1bde20f
[vdom]='dmgmt-vdom': aaad9f28801aa465e0a4d2176aa2851e
[type]='physical': 39d37257932bbbeb5593b348f9a9ce57
[snmp-index]='12': 1a87c30a608e61b92337a02dc73a435e
|
4d. Go to step 6.
5a. If a particular VDOM, such as root, is unsynchronized, then from global, run diagnose sys confsync cached-csum root.
|
FG74E43E1xxxxx63 [FIM01] (global) # diagnose sys confsync cached-csum root
system.object-tag: 5873dd45edd01f09c1ef2e7819369e8e
system.settings: 5873dd45edd01f09c1ef2e7819369e8e
system.sit-tunnel: 5873dd45edd01f09c1ef2e7819369e8e
system.arp-table: 5873dd45edd01f09c1ef2e7819369e8e
...
...
...
wireless-controller.wids-profile: 89b021d25c69bee5d44a9d4c5fe9ac1b
wireless-controller.wtp-profile: 2fb12986b481205b07555e106ab7f63d
wireless-controller.wtp: 5873dd45edd01f09c1ef2e7819369e8e
wireless-controller.wtp-group: 5873dd45edd01f09c1ef2e7819369e8e
wireless-controller.ap-status: 5873dd45edd01f09c1ef2e7819369e8e
...
...
...
system.wccp: 5873dd45edd01f09c1ef2e7819369e8e
system.nat64: 5873dd45edd01f09c1ef2e7819369e8e
|
FPM20E3E1xxxxx03 [FPM04] (global) # diagnose sys confsync cached-csum root
system.object-tag: 5873dd45edd01f09c1ef2e7819369e8e
system.settings: 5873dd45edd01f09c1ef2e7819369e8e
system.sit-tunnel: 5873dd45edd01f09c1ef2e7819369e8e
system.arp-table: 5873dd45edd01f09c1ef2e7819369e8e
...
...
...
wireless-controller.wids-profile: 89b021d25c69bee5d44a9d4c5fe9ac1b
wireless-controller.wtp-profile: 2fb12986b481205b07555e106ab7aeef
wireless-controller.wtp: 5873dd45edd01f09c1ef2e7819369e8e
wireless-controller.wtp-group: 5873dd45edd01f09c1ef2e7819369e8e
wireless-controller.ap-status: 5873dd45edd01f09c1ef2e7819369e8e
...
...
...
system.wccp: 5873dd45edd01f09c1ef2e7819369e8e
system.nat64: 5873dd45edd01f09c1ef2e7819369e8e
|
5b. In this example, wireless-controller.wtp-profile is unsynchronized, so from VDOM root, run diagnose sys confsync showcsum wireless-controller.wtp-profile.
|
FG74E43E1xxxxx63 [FIM01] (global) # diagnose sys confsync showcsum wireless-controller.wtp-profile
AP-11N-default: 4475b2a896abcf7774c506d82d46ee2c
FAP11C-default: 0471938d10a76f389737a19c2f3cb213
FAP14C-default: d1402026614d827a5faef75a7a3be6ff
FAP21D-default: 7be0b59f941a5d7f91879bb8836dfd5b
...
...
...
FAPS421E-default: a84ca5f7c3192913aac152b82af3626d
FAPS422E-default: 6112ce6bff2328a3969b05e2f1a6c833
FAPS423E-default: 739c63cd4c94adacadba8803fafe6b23
FK214B-default: e32c1e6736ee68e30b372b0a66dade95
|
FPM20E3E1xxxxx03 [FPM04] (global) # diagnose sys confsync showcsum wireless-controller.wtp-profile
AP-11N-default: 4475b2a896abcf7774c506d82d46ee2c
FAP11C-default: 0471938d10a76f389737a19c2f3cb213
FAP14C-default: d1402026614d827a5faef75a7a3be6ff
FAP21D-default: 7be0b59f941a5d7f91879bb8836dfd5b
...
...
...
FAPS421E-default: a84ca5f7c3192913aac152b82af34faa
FAPS422E-default: 6112ce6bff2328a3969b05e2f1a6c833
FAPS423E-default: 739c63cd4c94adacadba8803fafe6b23
FK214B-default: e32c1e6736ee68e30b372b0a66dade95
|
5c. If FAPS421E-default is unsynchronized, then from VDOM root, run diagnose sys confsync showcsum wireless-controller.wtp-profile FAPS421E-default.
|
FG74E43E1xxxxx63 [FIM01] (global) # diagnose sys confsync showcsum wireless-controller.wtp-profile FAPS421E-default
[name]='FAPS421E-default': 1822fc08ae7ea391ff2e01b0c7c5d80b
[platform]:
[type]='S421E': ec08d031ba3352cb9b2e77e87886d3c7
[ap-country]='US': 95c3cb4094c6ac7cb42f823f7d45303e
[radio-1]:
[band]='802.11n': 2fc047dafb9d65c44294c71fe8114ee6
[radio-2]:
[band]='802.11ac': fa16a841577330f4ac2a658f0189b9a6
|
FPM20E3E1xxxxx03 [FPM04] (global) # diagnose sys confsync showcsum wireless-controller.wtp-profile FAPS421E-default
[name]='FAPS421E-default': 1822fc08ae7ea391ff2e01b0c7c5d80b
[platform]:
[type]='S421E': ec08d031ba3352cb9b2e77e87886d3c7
[ap-country]='CA': 95c3cb4094c6ac7cb42f823f7d4aac45
[radio-1]:
[band]='802.11n': 2fc047dafb9d65c44294c71fe8114ee6
[radio-2]:
[band]='802.11ac': fa16a841577330f4ac2a658f0189b9a6
|
5d. Go to step 6.
6. The mismatched settings in step 4d or step 5d are the specific configuration section that does not match between units because they cannot sync through the config sync process.
Manually copy that configuration section from the config master FIM and paste it into the slave FIM/FPM.
Alternatively, take the backup configuration file from the config master FIM and restore it onto the out-of-sync slave blade. Connect to a specific blade's GUI using the special management ports and restore config using the top right menu option (HA mode special management port numbers). This step is useful in case many different parts of the FPM config are out-of-sync with its master FIM.
In the example below, port 44304 connects to the FPM04 on chassis-id 1:
Bear in mind that restoring the config on a specific FPM will require a reboot of the FPM. In a HA A-P cluster, a reboot of an FPM on the primary unit will trigger a failover and it is recommended to perform an FPM config restore when the chassis unit has a secondary role.
7. After the correction of all non-matching configurations, wait 2-3 minutes for the config sync process to detect the configurations are now in sync. Verify by performing step 2 again, this time ensuring that all blades have the status of Running.
Recalculation Scenario:
If step 3 shows a mismatch, but step 4 or step 5 does not show any configuration that does not match between units, a checksum recalculation is required. From global, run the command below on both the config master blade and the out-of-sync blade(s).
FG74E43E1xxxxx63 [FIM01] (global) # diagnose sys confsync csum-recalculate
FPM20E3E1xxxxx03 [FPM04] (global) # diagnose sys confsync csum-recalculate
Related articles:
