Need assistance with syncing firewalls after upgrade

[Thabiso]

Hi All,

We observed an issue while upgrading a FortiGate-1500D firewall HA pair from 7.0.14 to 7.2.11, with an intermediate step to 7.2.9. Following the initial upgrade to 7.2.9, FW2 came online, but FW1 remained offline for over five hours with no HATALK.

After a manual reboot, FW1 recovered to version 7.2.9, synchronized, and became primary due to its higher override setting.

However, the subsequent upgrade to 7.2.11 caused FW1 to go completely offline, with its HA interfaces failing. 

Currently we can not console to FW1 and the below logs are from FW2.

From the logs the collected the following can be seen:

13012: 2025-05-18 19:15:53 scanunit=manager pid=305: Error: failed to validate av package /data2/mmdb (no
13013: 2025-05-18 19:15:53 signature)
13014: 2025-05-18 19:15:53 scanunit=manager pid=305: Error: failed to validate av package /data2/avai (no
13015: 2025-05-18 19:15:53 signature)
13016: 2025-05-18 19:15:53 scanunit=manager pid=305 str="Some AVDBs failed to load."
13017: 2025-05-18 19:16:02 the killed daemon is /bin/sflowd: status=0x0
13018: 2025-05-18 19:16:11 the killed daemon is /bin/csfd: status=0x0
13019: 2025-05-18 19:16:48 the killed daemon is /bin/csfd: status=0x0
13020: 2025-05-18 19:16:48 the killed daemon is /bin/eap_proxy: status=0x0
13021: 2025-05-18 19:16:48 the killed daemon is /bin/eap_proxy: status=0xd
13022: 2025-05-18 19:17:13 scanunit=manager pid=305: Error: failed to validate av package /data2/mmdb (no
13023: 2025-05-18 19:17:13 signature)
13024: 2025-05-18 19:17:13 scanunit=manager pid=305: Error: failed to validate av package /data2/avai (no
13025: 2025-05-18 19:17:13 signature)
13026: 2025-05-18 19:17:13 scanunit=manager pid=305 cause='signal' str="AV database reload requested 1
13027: 2025-05-18 19:17:13 times by updated (pid 691) failed"
13028: 2025-05-18 19:20:09 the killed daemon is /bin/updated: status=0x0
13029: 2025-05-18 19:20:09 the killed daemon is /bin/csfd: status=0x0
13030: 2025-05-18 19:26:36 scanunit=manager pid=305: Error: failed to validate av package /data2/mmdb (no
13031: 2025-05-18 19:26:36 signature)
13032: 2025-05-18 19:26:36 scanunit=manager pid=305: Error: failed to validate av package /data2/avai (no
13033: 2025-05-18 19:26:36 signature)
13034: 2025-05-18 19:26:37 scanunit=manager pid=305 cause='signal' str="AV database reload requested 1
13035: 2025-05-18 19:26:37 times by quard (pid 321) failed"
13036: 2025-05-18 19:41:01 scanunit=manager pid=305: Error: failed to validate av package /data2/avai (no
13037: 2025-05-18 19:41:01 signature)
13038: 2025-05-18 19:41:01 scanunit=manager pid=305 cause='signal' str="AV database reload requested 2
13039: 2025-05-18 19:41:01 times by updated, updated (pid 1126, 1126) failed"
13040: 2025-05-18 21:03:03 Interface port31 is brought down. process_id=233, process_name="cmdbsvr"
13041: 2025-05-18 21:03:06 Interface port31 is brought up. process_id=233, process_name="cmdbsvr"
13042: 2025-05-18 21:03:09 Interface port32 is brought down. process_id=233, process_name="cmdbsvr"
13043: 2025-05-18 21:03:11 Interface port32 is brought up. process_id=233, process_name="cmdbsvr"
13044: 2025-05-18 21:07:42 Interface port34 is brought down. process_id=233, process_name="cmdbsvr"
13045: 2025-05-18 21:07:42 Interface port25 is brought down. process_id=233, process_name="cmdbsvr"
13046: 2025-05-18 21:07:42 Interface port1 is brought down. process_id=233, process_name="cmdbsvr"
13047: 2025-05-18 21:07:45 Interface port1 is brought up. process_id=233, process_name="cmdbsvr"
13048: 2025-05-18 23:01:15 Interface port1 is brought down. process_id=233, process_name="cmdbsvr"
13049: 2025-05-18 23:01:22 Interface port1 is brought up. process_id=233, process_name="cmdbsvr"
13050: 2025-05-19 00:37:32 the killed daemon is /bin/updated: status=0x0
13051: 2025-05-19 00:37:42 the killed daemon is /bin/eap_proxy: status=0x0
13052: 2025-05-19 00:37:42 the killed daemon is /bin/csfd: status=0x0
13053: 2025-05-19 00:37:44 the killed daemon is /bin/eap_proxy: status=0x0
13054: 2025-05-19 00:37:45 the killed daemon is /bin/radvd: status=0x0
13055: 2025-05-19 00:37:45 the killed daemon is /bin/eap_proxy: status=0x0
13056: 2025-05-19 00:41:48 scanunit=manager pid=305: Error: failed to validate av package /data2/mmdb (no
13057: 2025-05-19 00:41:48 signature)
13058: 2025-05-19 00:41:49 scanunit=manager pid=305 cause='signal' str="AV database reload requested 1
13059: 2025-05-19 00:41:49 times by hasync (pid 312) failed"
13060: 2025-05-19 00:42:55 scanunit=manager pid=305: Error: failed to validate av package /data2/mmdb (no
13061: 2025-05-19 00:42:55 signature)
13062: 2025-05-19 00:42:55 scanunit=manager pid=305: Error: failed to validate av package /data2/avai (no
13063: 2025-05-19 00:42:55 signature)
13064: 2025-05-19 00:42:55 scanunit=manager pid=305: Error: failed to validate av package /data2/virext
13065: 2025-05-19 00:42:55 (invalid signature)
13066: 2025-05-19 00:42:55 scanunit=manager pid=305: Error: failed to validate av package /data2/vir
13067: 2025-05-19 00:42:55 (invalid signature)
13068: 2025-05-19 00:42:55 <00305> scanunit=manager str="Not trying to load anti-virus database due to
13069: 2025-05-19 00:42:55 corrupt AVDB."
13070: 2025-05-19 00:42:55 <00305> scanunit=manager str="Not trying to load basic anti-virus database
13071: 2025-05-19 00:42:55 (ATDB) with FSA database due to corrupt AVDB."
13072: 2025-05-19 00:42:55 <00305> scanunit=manager str="Not trying to load basic anti-virus database
13073: 2025-05-19 00:42:55 (ATDB) due to corrupt AVDB."
13074: 2025-05-19 00:42:55 <00305> scanunit=manager str="Failed to move /tmp/vir.lastgood to try loading
13075: 2025-05-19 00:42:55 last-known good anti-virus database (ATDB) with FSA database."
13076: 2025-05-19 00:42:55 <00305> scanunit=manager str="Failed to move /tmp/vir.lastgood to try loading
13077: 2025-05-19 00:42:55 last-known good anti-virus database (ATDB)."
13078: 2025-05-19 00:42:56 scanunit=manager pid=305 cause='signal' str="AV database reload requested 1
13079: 2025-05-19 00:42:56 times by hasync (pid 312) failed"
13080: 2025-05-19 00:43:08 the killed daemon is /bin/csfd: status=0x0
13081: 2025-05-19 00:43:17 Interface port25 is brought up. process_id=233, process_name="cmdbsvr"
13082: 2025-05-19 00:43:17 Interface port34 is brought up. process_id=233, process_name="cmdbsvr"
13083: 2025-05-19 00:58:21 scanunit=manager pid=305: Error: failed to validate av package /data2/mmdb (no
13084: 2025-05-19 00:58:21 signature)
13085: 2025-05-19 00:58:21 scanunit=manager pid=305: Error: failed to validate av package /data2/avai (no
13086: 2025-05-19 00:58:21 signature)
13087: 2025-05-19 00:58:22 scanunit=manager pid=305 cause='signal' str="AV database reload requested 1
13088: 2025-05-19 00:58:22 times by hasync (pid 312) failed"
13089: 2025-05-19 00:58:34 the killed daemon is /bin/eap_proxy: status=0x0
13090: 2025-05-19 00:58:34 the killed daemon is /bin/csfd: status=0x0
13091: 2025-05-19 00:58:34 the killed daemon is /bin/eap_proxy: status=0x0
13092: 2025-05-19 00:58:40 the killed daemon is /bin/eap_proxy: status=0x0
13093: 2025-05-19 00:58:40 the killed daemon is /bin/eap_proxy: status=0x0
13094: 2025-05-19 00:59:25 the killed daemon is /bin/csfd: status=0x0
13095: 2025-05-19 00:59:25 the killed daemon is /bin/eap_proxy: status=0x0
13096: 2025-05-19 01:22:21 the killed daemon is /bin/sflowd: status=0x0
13097: 2025-05-19 01:22:33 the killed daemon is /bin/csfd: status=0x0
13098: 2025-05-19 01:22:46 the killed daemon is /bin/csfd: status=0x0
13099: 2025-05-19 01:22:46 the killed daemon is /bin/eap_proxy: status=0x0
13100: 2025-05-19 01:22:46 the killed daemon is /bin/eap_proxy: status=0xd
13101: 2025-05-19 01:22:49 the killed daemon is /bin/eap_proxy: status=0x0
13102: 2025-05-19 01:26:33 the killed daemon is /bin/updated: status=0x0
13103: 2025-05-19 01:26:34 the killed daemon is /bin/csfd: status=0x0
Crash log interval is 3600 seconds
Max crash log line number: 16384

Has anyone encountered something similar or know a possible workaround this issue.  

[Toshi_Esumi]

The log has no information about HA or FW1's state. You really needed to have a console connected at FW1 when you upgraded to 7.2.9 to see what happened when it didn't come up. Since it came up with 7.2.9 after a power-cycle, the upgrade process was almost done on the unit then got stuck somewhere in the final boot up, I guess.

You should open a ticket at TAC to get it examined. But TAC would ask you to connect to the console port and power-cycle. Not much you can do without a console connection.

Toshi

[adambomb1219]

What's your replacement plan for that 1500-D?