Technical Tip: What really happens on a FortiManager when a FortiGate is replaced that was in a HA cluster
| Description | This article describes what happens after the replacement of a FortiGate, which was the Secondary in a HA cluster, when the HA cluster was formed on the FortiManager. |
| Scope | FortiGate v7.4.5. FortiManager v7.4.5. |
| Solution | According to the documentation, when replacing a FortiGate cluster member, FortiManager will learn the new serial number through the FGFM tunnel, so nothing needs to be done. However, let's see what really happens in this process.
The FortiGate serial numbers are:
And the FortiManager serial number is FMGVMSTM22000130. The secondary was taken out of the cluster. It was the secondary at the time the HA cluster was added to FortiManager.
Start by adding the HA cluster to FortiManager:
diagnose debug application fgfm -1 On the FortiGate debug of the tunnel fgfm shows tst2 # FGFMs: Incoming ::ffff:10.5.146.8 local ::ffff:10.5.141.236. FGFMs: Create session 0x100bb160. FGFMs: checking existing sessions... FGFMs: set_fgfm_sni SNI<support.fortinet-ca2.fortinet.com> FGFMs: Load Cipher ….. FGFMs: client: reply 200 request=auth serialno=FMGVMSTM22000130 user=admin … FGFMs: immediate CA in peer cert verified chain: subject <support>, issuer <support> …. FGFMs: serial no FMGVMSTM22000130 saved to FMG detect file FGFMs: client:
On the FortiManager debug, one can see:
FGFMs(probing...): server: get ip serialno=FGVM04TM20000407 platform=FortiGate-VM64-KVM … hostname=tst2 ……
On the FortiManager debug, one can see that FortiManager forces FortiGate to accept the tunnel:
FGFMs(FGVM04TM20000407-235-10.5.141.236): Force FGT to accept this tunnel setup request FGFMs(FGVM04TM20000407-235-10.5.141.236): Update management interface to port1 into DVM FGFMs(FGVM04TM20000407-235-10.5.141.236): Update conn_mode=Active into DVM FGFMs(FGVM04TM20000407-235-10.5.141.236): Update fg_reachable=1 into DVM FGFMs(FGVM04TM20000407-235-10.5.141.236): Update tunnel_ip=169.254.0.2 into DVM FGFMs(FGVM04TM20000407-235-10.5.141.236): Update first_tunnel_up=1733752225 into DVM
The tunnel is created, and FortiManager's serial number is synced to the Secondary FortiGate. The FortiManager shows an HA cluster to be managed.
TYPE OID SN HA IP NAME ADOM IPS FIRMWARE HW_GenX fmgfaz-managed 235 FGVM04TM20000407 a-p 10.5.141.236 tst2 root 6.00741 (extended) 7.0 MR4 (2702) N/A |- STATUS: dev-db: not modified; conf: in sync; cond: OK; dm: retrieved; conn: up HA cluster member: FGVM04TM20000407 (primary); conn: up; conf: in sync HA cluster member: FGVM040000052782 (secondary 1); conn: up; conf: in sync
From the debug, one can see the secondary was added:
FGFMs(probing...): server: get auth serialno=FGVM040000052782 .. hostname=tst4 FGFMs(probing...): fgfm_get_inst_info,105: serial=, devid=0, revision=0, timestamp=0. Request [/bin/fgfmsd:4764:33]: { "client": "\/bin\/fgfmsd:4764", "id": 33, "method": "exec", "params": [{ "data": { "create_unreg": 1, "device": { "beta": -1, "branch_pt": 2702, "build": 2702, "conn_mode": 0, "dev_status": 0, "faz.perm": 15, "flags": 1, "hostname": "tst4", "ip": "10.5.141.236", "maxvdom": 10, "mgmt_mode": 1, "mgmt_uuid": "00000000-0000-0000-0000-000000000000", "mr": 4, "name": "tst4", "os_type": 0, "os_ver": -1, "patch": 5, "platform_id": -1, "platform_str": "FortiGate-VM64-KVM", "sn": "FGVM040000052782", "source": 1, "tab_status": "<unknown>", "version": 700}, "from": 1}, "url": "dvm\/cmd\/manage\/device"}], "session": -1} FGFMs: __put_script_handler,1031: devid=235. FGFMs: __put_script_handler,1035: devid=235, serialno=FGVM04TM20000407. FGFMs(FGVM04TM20000407-235-10.5.141.236): __put_script_handler,1061:open log fp 0x55b6d3e7b6c0, fd=16, OK. FGFMs(FGVM04TM20000407-235-10.5.141.236): server:send: …
The tunnel is forced from the FortiManager to the secondary FortiGate:
FGFMs(FGVM040000052782-249-10.5.141.236): __get_handler: serialno in peer cert is <FGVM040000052782> FGFMs(FGVM040000052782-249-10.5.141.236): fgfm_get_inst_info,105: serial=, devid=0, revision=0, timestamp=0. FGFMs(FGVM040000052782-249-10.5.141.236): Force FGT to accept this tunnel setup request FGFMs(FGVM040000052782-249-10.5.141.236): Update management interface to port1 into DVM FGFMs(FGVM040000052782-249-10.5.141.236): Update conn_mode=Passive into DVM FGFMs(FGVM040000052782-249-10.5.141.236): Update fg_reachable=1 into DVM FGFMs(FGVM040000052782-249-10.5.141.236): Update tunnel_ip=169.254.0.3 into DVM FGFMs(FGVM040000052782-249-10.5.141.236): Update first_tunnel_up=1733752379 into DVM FGFMs(FGVM040000052782-249-10.5.141.236): server:send:
erbium-kvm48 # diagnose dvm device list TYPE OID SN HA IP NAME ADOM IPS FIRMWARE HW_GenX fmgfaz-managed 235 FGVM04TM20000407 a-p 10.5.141.236 tst2 root 6.00741 (extended) 7.0 MR4 (2702) N/A |- STATUS: dev-db: not modified; conf: in sync; cond: OK; dm: retrieved; conn: up HA cluster member: FGVM04TM20000407 (primary); conn: up; conf: in sync |- vdom:[3]root flags:0 adom:root pkg:[imported]tst2new_root unregistered 249 FGVM040000052782 - 10.5.141.236 tst4 root N/A 7.0 MR4 (2702) N/A
FortiManager has the HA cluster as managed devices, and both FortiGates have FortiManager as the manager.
Take the Secondary FortiGate out of the cluster and add a new secondary in the cluster. After this procedure, it is clear that FortiManager has replaced the device, and the previous device is seen as a standalone and unregistered.
TYPE OID SN HA IP NAME ADOM IPS FIRMWARE HW_GenX fmgfaz-managed 235 FGVM04TM20000407 a-p 10.5.141.236 tst2 root 6.00741 (extended) 7.0 MR4 (2702) N/A |- STATUS: dev-db: not modified; conf: in sync; cond: OK; dm: retrieved; conn: up HA cluster member: FGVM04TM20000407 (primary); conn: up; conf: in sync HA cluster member: FGVM040000052818 (secondary 1); conn: up; conf: unknown |- vdom:[3]root flags:0 adom:root pkg:[imported]tst2new_root unregistered 249 FGVM040000052782 - 10.5.141.236 tst4 root N/A 7.0 MR4 (2702) N/A <---------- This is the replaced device’s serial number.
The replaced device must be deleted from FortiManager. It was done from the GUI.
fmgfaz-managed 235 FGVM04TM20000407 a-p 10.5.141.236 tst2 root 6.00741 (extended) 7.0 MR4 (2702) N/A |- STATUS: dev-db: not modified; conf: in sync; cond: OK; dm: retrieved; conn: up HA cluster member: FGVM04TM20000407 (primary); conn: up; conf: in sync HA cluster member: FGVM040000052818 (secondary 1); conn: up; conf: in sync |- vdom:[3]root flags:0 adom:root pkg:[imported]tst2new_root
The answer to the question is that, for the HA cluster on the FortiManager, it is not necessary to add the serial number, but it is necessary to delete the replaced device.
Replacing the serial number of the old FortiGate can also be achieved by executing the command first and then reclaiming the fgfm tunnel:
FMG # execute device replace sn <device_name> <FGTXXXXXXXXXXXXX> <Enter> FMG # execute fgfm reclaim-dev-tunnel <device name>
Related article: |