Troubleshooting Tip: How to resolve issue when CLI message 'Error: FIM1 and FIM2 boardtype mismatch detected!' is showed on a 7000F series

[fricci_FTNT]

Description

This article describes how to resolve an issue experienced on a 7000F series when the CLI on the FIM shows a mismatch message 'Error: FIM1 and FIM2 boardtype mismatch detected!' during the boot process:

----------------------------------
Reading boot image 34xxxxxx bytes.
Initializing firewall...
System is starting...
boardtype_to_str 149 boardtype:4 does not have str name
boardtype_to_str 149 boardtype:4 does not have str name
Error: FIM1 and FIM2 boardtype mismatch detected!
FIM1 is UNKNOWN BOARDTYPEFIM7921F but FIM2 is UNKNOWN BOARDTYPEFIM7921F
To continue, please make sure both FIMs are the same boardtype.
----------------------------------

As a consequence the login prompt is not displayed and it is not possible to input any login credentials from CLI.

The problem is due to the firmware image and a firmware upgrade on the FIM module, performed from the boot menu, can resolve it.
The FPMs should be still accessible from console and after login with admin credentials, the command 'get system status' from FPM should show the firmware version installed on FIM.

Scope

FortiGate-7000F series.

The file images that can be installed to resolve the issue are:

• V6.4.13 GA and above.
• V7.0.13 and above.
• V7.2.6 GA and above.

Solution

To resolve the problem, the following are needed:

• A TFTP server software installed on a laptop.
• Windows/Linux firewall rule on the laptop to temporarily allow incoming TFTP requests (default port UDP 69).
• Firmware image pre-downloaded and stored in TFTP software folder.
• Console cable (for local console access).
• Ethernet cable (to locally connect laptop to MGMT port on SMM).
• Local or remote console access to the FortiGate-7000F series.

Following this procedure, a firmware image will be downloaded from an external TFTP server (i.e. laptop) into a TFTP folder inside the FIM module.
Once the new firmware image is stored in the FIM01, it can be used to push the image to the primary FIM flash or any other 7000F series FIM/FPM module.

To connect the FortiGate to the SMM console, follow the instructions on the following KB article:

Technical Tip: How to access the console port on the management 

Once the laptop is connected to the FortiGate SMM module, Management IP and laptop IP are in the same subnet (or can reach each other), the TFTP software is running correctly and incoming TFTP requests are allowed, the procedure is the following:

1. Restart the FIM from the SMM menu and quickly go back to the FIM prompt pressing Ctrl+T. See related Kb article: Technical Tip: How to restart FPMs or FIMs through the SMM module
2. After the restart, enter the boot menu on FIM and set the TFTP parameters as shown in the related document: Installing FIM firmware from the BIOS after a reboot 
3. After the FIM is rebooted and recovered with new firmware, it should be necessary to restore the firmware on the secondary FIM using a similar procedure.
4. The secondary FIM (FIM02) can use the image downloaded in step '2' from the primary FIM. Make sure to choose the correct download port '[P]' and local IP address/gateway '[I],[G]' in FIM02 TFTP parameters.
5. Once FIMs are up and running with new correct firmware, it might be necessary to push the image to the FPMs following the instructions in the following document: Installing FPM firmware from the BIOS after a reboot  

During the procedure, if any firmware validating errors appear and a new firmware cannot be installed, it might be necessary to temporarily lower the firmware level to '0' and then restore it on the specific slot after the new firmware is in place via following KB article:

Troubleshooting Tip: Downgrade of FortiOS fails due to BIOS check 

If after installing new firmware on the FIM module, errors similar to the below are shown in the FIM console, format the log disk ('execute formatlogdisk') and rebuild the RAID ('execute disk raid enable'), the reboot will be required and errors will disappear:

-----------------------------------------------------------
EXT4-fs error (device nvme0n1p1): ext4_lookup:1576: inode #2: comm smit: deleted inode referenced: 374xxxxx
EXT4-fs error (device nvme0n1p1): ext4_lookup:1576: inode #2: comm smit: deleted inode referenced: 374xxxxx
EXT4-fs error (device nvme0n1p1): ext4_lookup:1576: inode #2: comm smit: deleted inode referenced: 374xxxxx
EXT4-fs error (device nvme0n1p1): ext4_lookup:1576: inode #2: comm cmdbsvr_iprop e: deleted inode referenced: 95xxxxx
EXT4-fs error (device nvme0n1p1): ext4_lookup:1576: inode #2: comm cmdbsvr_iprop e: deleted inode referenced: 95xxxxx
-----------------------------------------------------------

Note:
As TFTP software, Tftptd32/Tftpd64 usually works quite well.