FortiGate
FortiGate Next Generation Firewall utilizes purpose-built security processors and threat intelligence security services from FortiGuard labs to deliver top-rated protection and high performance, including encrypted traffic.
jheadley_FTNT

Description

This article reviews a technique for troubleshooting config synchronization issues between a config master blade (one of the FIMs) and one or more config slave blades (the other FIM and all FPMs) in a FortiGate 7000 (7k) series chassis.


Scope

FortiGate 7000 series.
 
Following this procedure requires access to a text comparison/diff tool, such as Notepad++ with the Compare plugin.


Solution
1. Identify the FIM that is the "config master" by running get system status

Note that in a two chassis (active-passive) configuration, only a single FIM in the active chassis will be the config master.

FG74E43E1xxxxx63 [FIM01] # get system status
==========================================================================
Slot: 2 Module SN: FIM04E3E1xxxxx64
Version: FortiGate-7040E v5.4.3,build6284,170714 (GA)
...
Serial-Number: FG74E43E16000063
Module Serial-Number: FIM04E3E1xxxxx64
Config-Sync: Slave
==========================================================================
Slot: 3 Module SN: FPM20E3E1xxxxx04
Version: FortiGate-7040E v5.4.3,build6284,170714 (GA)

Serial-Number: FG74E43E1xxxxx63
Module Serial-Number: FPM20E3E1xxxxx04
Config-Sync: Slave
==========================================================================
Current slot: 1 Module SN: FIM01E3E1xxxxx72
Version: FortiGate-7040E v5.4.3,build6284,170714 (GA)

Serial-Number: FG74E43E1xxxxx63
Module Serial-Number: FIM01E3E1xxxxx72
Config-Sync: Master


2. On the FIM identified in Step 1, run
diagnose load-balance status from global to identify which blades are not in sync.

Note that if you have a second chassis, you will also need to run this command on one of the FIMs in the passive chassis.

FG74E43E1xxxxx63 [FIM01] (global) # diagnose load-balance status

FIM01: FIM01E3E1xxxxx72
Master FPM Blade: slot-3

Slot 3: FPM20E3E1xxxxx04
Status:Working Function:Active
Link: Base: Up Fabric: Up
Heartbeat: Management: Good Data: Good
Status Message:"Running"

Slot 4: FPM20E3E1xxxxx03
Status:Working Function:Active
Link: Base: Up Fabric: Up
Heartbeat: Management: Good Data: Good
Status Message:"Waiting for configuration sync"

Steps 3 through 5 require output from both the config master FIM and the out-of-sync FIM or FPM blade.

Note you can use the “command broadcasting” feature to only run the command once on the FIM, which will (by default) query all the rest of the blades in the chassis for the same diagnostic output. You would then need to cut and paste the correct sections into your text comparison tool.


3.  From global, run diagnose sys confsync showcsumand using the text compare tool, identify what line is out of sync (not matching) between the units. The last line (all) can be ignored, because it is a summary of all previous lines.

Note that the same lines under debugzone will also appear under checksum.

FG74E43E1xxxxx63 [FIM01] (global) # diagnose sys confsync showcsum

debugzone
global: 3d 1e f3 53 26 8b 7a 4f 48 1a d8 21 11 a2 d8 d4
vdom4: db cb 68 bd 40 bb 71 68 2b 45 93 51 8a f0 e6 0d
vdom3: e7 17 f6 22 1b 2e 85 fd c8 d2 ea d1 23 a1 2f a2
vdom2: 36 a0 34 47 c2 ee c8 15 3b 08 54 a6 bd c7 bb 42
vdom1: 56 9c 7e 82 dd 17 83 d7 a5 a0 50 ec b6 04 ef ad
root: ff ca 50 c0 67 54 51 a5 c8 2a 6c 3e ad 17 dc 70
dmgmt-vdom: 69 39 d4 2c 6a 36 cf aa d5 00 6b 63 22 cf 28 3b
all: d5 11 62 29 4f a5 af 14 95 29 08 fc cc 25 78 62

checksum
global: 3d 1e f3 53 26 8b 7a 4f 48 1a d8 21 11 a2 d8 d4
vdom4: db cb 68 bd 40 bb 71 68 2b 45 93 51 8a f0 e6 0d
vdom3: e7 17 f6 22 1b 2e 85 fd c8 d2 ea d1 23 a1 2f a2
vdom2: 36 a0 34 47 c2 ee c8 15 3b 08 54 a6 bd c7 bb 42
vdom1: 56 9c 7e 82 dd 17 83 d7 a5 a0 50 ec b6 04 ef ad
root: ff ca 50 c0 67 54 51 a5 c8 2a 6c 3e ad 17 dc 70
dmgmt-vdom: 69 39 d4 2c 6a 36 cf aa d5 00 6b 63 22 cf 28 3b
all: d5 11 62 29 4f a5 af 14 95 29 08 fc cc 25 78 62

FPM20E3E1xxxxx03 [FPM04] (global) # diagnose sys confsync showcsum

debugzone
global: 3d 1e f3 53 26 8b 7a 4f 48 1a d8 21 11 a2 d8 b6
vdom4: db cb 68 bd 40 bb 71 68 2b 45 93 51 8a f0 e6 0d
vdom3: e7 17 f6 22 1b 2e 85 fd c8 d2 ea d1 23 a1 2f a2
vdom2: 36 a0 34 47 c2 ee c8 15 3b 08 54 a6 bd c7 bb 42
vdom1: 56 9c 7e 82 dd 17 83 d7 a5 a0 50 ec b6 04 ef ad
root: ff ca 50 c0 67 54 51 a5 c8 2a 6c 3e ad 17 dc 70
dmgmt-vdom: 69 39 d4 2c 6a 36 cf aa d5 00 6b 63 22 cf 28 3b
all: d5 11 62 29 4f a5 af 14 95 29 08 fc cc 25 78 89

checksum
global: 3d 1e f3 53 26 8b 7a 4f 48 1a d8 21 11 a2 d8 b6
vdom4: db cb 68 bd 40 bb 71 68 2b 45 93 51 8a f0 e6 0d
vdom3: e7 17 f6 22 1b 2e 85 fd c8 d2 ea d1 23 a1 2f a2
vdom2: 36 a0 34 47 c2 ee c8 15 3b 08 54 a6 bd c7 bb 42
vdom1: 56 9c 7e 82 dd 17 83 d7 a5 a0 50 ec b6 04 ef ad
root: ff ca 50 c0 67 54 51 a5 c8 2a 6c 3e ad 17 dc 70
dmgmt-vdom: 69 39 d4 2c 6a 36 cf aa d5 00 6b 63 22 cf 28 3b
all: d5 11 62 29 4f a5 af 14 95 29 08 fc cc 25 78 89


If the global line is unsynchronized (not matching) go to step 4a.
If the global line is synchronized (matching), but any specific vdom is unsynchronized, go to step 5a.


4a. From global, rundiagnose sys confsync showcsum 1.

 

FG74E43E1xxxxx63 [FIM01] (global) # diagnose sys confsync showcsum 1

system.global: f8b31181ae4b93ce5a6e8fbece51d2d1

system.accprofile: 7d79452c78377be2616149264a18fd5c
system.npu: 00000000000000000000000000000000
system.vdom-link: 00000000000000000000000000000000
wireless-controller.global: 00000000000000000000000000000000
wireless-controller.vap: 00000000000000000000000000000000
system.switch-interface: 00000000000000000000000000000000
system.lte-modem: 00000000000000000000000000000000
system.interface: be3f520521f5610d30fd936d65204b19
system.password-policy: 00000000000000000000000000000000
system.password-policy-guest-admin: 00000000000000000000000000000000
...
...
...
system.ntp: 5c774215d59f7231401cc64fe23c3045
system.vdom-radius-server: 00000000000000000000000000000000
system.geoip-override: 00000000000000000000000000000000
system.fortisandbox: 00000000000000000000000000000000
FPM20E3E1xxxxx03 [FPM04] (global) # diagnose sys confsync showcsum 1

system.global: f8b31181ae4b93ce5a6e8fbece51d2d1

system.accprofile: 7d79452c78377be2616149264a18fd5c
system.npu: 00000000000000000000000000000000
system.vdom-link: 00000000000000000000000000000000
wireless-controller.global: 00000000000000000000000000000000
wireless-controller.vap: 00000000000000000000000000000000
system.switch-interface: 00000000000000000000000000000000
system.lte-modem: 00000000000000000000000000000000
system.interface: be3f520521f5610d30fd936d65206578
system.password-policy: 00000000000000000000000000000000
system.password-policy-guest-admin: 00000000000000000000000000000000
...
...
...
system.ntp: 5c774215d59f7231401cc64fe23c3045
system.vdom-radius-server: 00000000000000000000000000000000
system.geoip-override: 00000000000000000000000000000000
system.fortisandbox: 00000000000000000000000000000000

4b. In this example, system.interface is unsynchronized, so in global, run diagnose sys confsync showcsum system.interfaceto see specifics on what is not synchronized under this configuration section.

 

FG74E43E1xxxxx63 [FIM01] (global) # diagnose sys confsync showcsum system.interface

base-mgmt: 5873dd45edd01f09c1ef2e7819369e8e
base1: b88429a8f1a433679999849ca1f49fd7
base2: d581b02347bdd9a33674fa8bc87ecb83
elbc-base-ctrl: b8405240b754710af36156b4ca2c0f5c
...
...
...
1-mgmt1: b8405240b754710af36156b4ca2c0f5c
1-mgmt2: b8405240b754710af36156b4ca2c0f5c
1-mgmt3: 85c640a4dce9973a6e8bd1e249857822
1-mgmt4: b8405240b754710af36156b4ca2c0f5c
1-M1: b8405240b754710af36156b4ca2c0f5c
FPM20E3E1xxxxx03 [FPM04] (global) # diagnose sys confsync showcsum system.interface

base-mgmt: 5873dd45edd01f09c1ef2e7819369e8e
base1: b88429a8f1a433679999849ca1f49ff4
base2: d581b02347bdd9a33674fa8bc87ecb83
elbc-base-ctrl: b8405240b754710af36156b4ca2c0f5c
...
...
...
1-mgmt1: b8405240b754710af36156b4ca2c0f5c
1-mgmt2: b8405240b754710af36156b4ca2c0f5c
1-mgmt3: 85c640a4dce9973a6e8bd1e249857822
1-mgmt4: b8405240b754710af36156b4ca2c0f5c
1-M1: b8405240b754710af36156b4ca2c0f5c

4c. If base1 is unsynchronized. From global, run diagnose sys confsync showcsum system.interface base1

 

FG74E43E1xxxxx63 [FIM01] (global) # diagnose sys confsync showcsum system.interface base1

[name]='base1': 5ffbc45e893c99b462c78391d1bde20f
[vdom]='dmgmt-vdom': aaad9f28801aa465e0a4d2176aa2851e
[type]='physical': 39d37257932bbbeb5593b348f9a9ce57
[snmp-index]='8': 1a87c30a608e61b92337a02dc73a5210
FPM20E3E1xxxxx03 [FPM04] (global) # diagnose sys confsync showcsum system.interface base1

[name]='base1': 5ffbc45e893c99b462c78391d1bde20f
[vdom]='dmgmt-vdom': aaad9f28801aa465e0a4d2176aa2851e
[type]='physical': 39d37257932bbbeb5593b348f9a9ce57
[snmp-index]='12': 1a87c30a608e61b92337a02dc73a435e

4d. Go to step 6.

5a. If a particular vdom, such as root,is unsynchronized, then from global, run
diagnose sys confsync cached-csum root
.

FG74E43E1xxxxx63 [FIM01] (global) # diagnose sys confsync cached-csum root

system.object-tag: 5873dd45edd01f09c1ef2e7819369e8e
system.settings: 5873dd45edd01f09c1ef2e7819369e8e
system.sit-tunnel: 5873dd45edd01f09c1ef2e7819369e8e
system.arp-table: 5873dd45edd01f09c1ef2e7819369e8e
...
...
...
wireless-controller.wids-profile: 89b021d25c69bee5d44a9d4c5fe9ac1b
wireless-controller.wtp-profile: 2fb12986b481205b07555e106ab7f63d
wireless-controller.wtp: 5873dd45edd01f09c1ef2e7819369e8e
wireless-controller.wtp-group: 5873dd45edd01f09c1ef2e7819369e8e
wireless-controller.ap-status: 5873dd45edd01f09c1ef2e7819369e8e
...
...
...
system.wccp: 5873dd45edd01f09c1ef2e7819369e8e
system.nat64: 5873dd45edd01f09c1ef2e7819369e8e
FPM20E3E1xxxxx03 [FPM04] (global) # diagnose sys confsync cached-csum root

system.object-tag: 5873dd45edd01f09c1ef2e7819369e8e
system.settings: 5873dd45edd01f09c1ef2e7819369e8e
system.sit-tunnel: 5873dd45edd01f09c1ef2e7819369e8e
system.arp-table: 5873dd45edd01f09c1ef2e7819369e8e
...
...
...
wireless-controller.wids-profile: 89b021d25c69bee5d44a9d4c5fe9ac1b
wireless-controller.wtp-profile: 2fb12986b481205b07555e106ab7aeef
wireless-controller.wtp: 5873dd45edd01f09c1ef2e7819369e8e
wireless-controller.wtp-group: 5873dd45edd01f09c1ef2e7819369e8e
wireless-controller.ap-status: 5873dd45edd01f09c1ef2e7819369e8e
...
...
...
system.wccp: 5873dd45edd01f09c1ef2e7819369e8e
system.nat64: 5873dd45edd01f09c1ef2e7819369e8e

5b.
In this example,wireless-controller.wtp-profileis unsynchronized, so from vdom root, run diagnose sys confsync showcsum wireless-controller.wtp-profile.

 

FG74E43E1xxxxx63 [FIM01] (global) # diagnose sys confsync showcsum wireless-controller.wtp-profile

AP-11N-default: 4475b2a896abcf7774c506d82d46ee2c
FAP11C-default: 0471938d10a76f389737a19c2f3cb213
FAP14C-default: d1402026614d827a5faef75a7a3be6ff
FAP21D-default: 7be0b59f941a5d7f91879bb8836dfd5b
...
...
...
FAPS421E-default: a84ca5f7c3192913aac152b82af3626d
FAPS422E-default: 6112ce6bff2328a3969b05e2f1a6c833
FAPS423E-default: 739c63cd4c94adacadba8803fafe6b23
FK214B-default: e32c1e6736ee68e30b372b0a66dade95
FPM20E3E1xxxxx03 [FPM04] (global) # diagnose sys confsync showcsum wireless-controller.wtp-profile

AP-11N-default: 4475b2a896abcf7774c506d82d46ee2c
FAP11C-default: 0471938d10a76f389737a19c2f3cb213
FAP14C-default: d1402026614d827a5faef75a7a3be6ff
FAP21D-default: 7be0b59f941a5d7f91879bb8836dfd5b
...
...
...
FAPS421E-default: a84ca5f7c3192913aac152b82af34faa
FAPS422E-default: 6112ce6bff2328a3969b05e2f1a6c833
FAPS423E-default: 739c63cd4c94adacadba8803fafe6b23
FK214B-default: e32c1e6736ee68e30b372b0a66dade95

5
c. If FAPS421E-default is unsynchronized, then from vdom root, run diagnose sys confsync showcsum wireless-controller.wtp-profile FAPS421E-default.

 

FG74E43E1xxxxx63 [FIM01] (global) # diagnose sys confsync showcsum wireless-controller.wtp-profile FAPS421E-default

[name]='FAPS421E-default': 1822fc08ae7ea391ff2e01b0c7c5d80b
[platform]:
[type]='S421E': ec08d031ba3352cb9b2e77e87886d3c7
[ap-country]='US': 95c3cb4094c6ac7cb42f823f7d45303e
[radio-1]:
[band]='802.11n': 2fc047dafb9d65c44294c71fe8114ee6
[radio-2]:
[band]='802.11ac': fa16a841577330f4ac2a658f0189b9a6
FPM20E3E1xxxxx03 [FPM04] (global) # diagnose sys confsync showcsum wireless-controller.wtp-profile FAPS421E-default

[name]='FAPS421E-default': 1822fc08ae7ea391ff2e01b0c7c5d80b
[platform]:
[type]='S421E': ec08d031ba3352cb9b2e77e87886d3c7
[ap-country]='CA': 95c3cb4094c6ac7cb42f823f7d4aac45
[radio-1]:
[band]='802.11n': 2fc047dafb9d65c44294c71fe8114ee6
[radio-2]:
[band]='802.11ac': fa16a841577330f4ac2a658f0189b9a6

5d. Go to step 6.

6. The mismatched settings in step 4d or step 5d is the specific configuration section that does not match between units because it cannot sync through the config sync process.

Manually copy that configuration section from the config master FIM and paste into the slave FIM/FPM.

Alternatively, take the backup configuration file from the config master FIM and restore onto the out of sync slave blade.

7. After the correction of all non-matching configuration, wait 2-3 minutes for the config sync process to detect the configurations are now in sync. Verify by performing step 2 again, this time ensuring that all blades have status of Running.

Recalculation Scenario:


If step 3 shows a mismatch, but step 4 or step 5 does not show any configuration that does not match between units, a checksum recalculation is required.  From global, run the command below on both the config master blade and the out of sync blade(s).

        FG74E43E1xxxxx63 [FIM01] (global) # diagnose sys confsync csum-recalculate

    FPM20E3E1xxxxx03 [FPM04] (global) # diagnose sys confsync csum-recalculate

 

Related Articles

Troubleshooting Tip: FortiGate 5000 Series blade configuration synchronization Issues (SLBC confsync...

Contributors