|
When downgrading, check the release notes/special notes section to follow the factory recommendations. Follow the recommended upgrade path.
FortiManager and FortiAnalyzer have an operating system based on Linux. As with any operating system, there are a lot of processes, sub processes, and daemons running each one with a specified function and operating in a very closed behavior.
A daemon sends data in X format and another daemon receives such data in X format. This shows the entire subsystem is in good health.
Whenever a firmware upgrade process takes place, does not only upgrades the major version of the system but also updates daemons, services, and processes version changing their code, expectation, and output.
When a downgrade process takes place, logically it is possible to think that the process will be backward compatible, but it does not always work this way because the version check of the sub-processes is higher than the reference.
This left sub-processes with a higher version than the rest of the components, causing them to speak in different 'languages', with different expectations and different outputs. Returning to the first example: 'A daemon sends data in X format and another daemon receives such data in Y format.' This causes an internal error on the secondary daemon because the data received does not match the expected format/content of the data.
This causes an erratic behavior on the device, working perfectly for some specific operations but working erratic for others. The logic behind this is the relationship between the system components or services.
At the end of the day, a system with these characteristics is called 'Out of Fabric Functioning' causing the Development team to not accept Bug requests for unexpected behavior and the system to be out of Support for TAC.
Examples of erratic behavior can be observed in several cases without being able to be explained. Some examples will be listed but behavior is not limited to it:
- Cx unable to upgrade ADOM from one version to another.
- Cx reported atypical performance on FortiManager Modules.
- GUI menus not functioning as expected.
- FortiManager/FortiAnalyzer closing connections unexpectedly.
- Reporting module returning data/no matched data for the same query.
- HA process unable to synchronize between primary and secondary units.
- FortiManager PP not installing settings due to an unknown conflict.
- ZTNA Tags cannot be inserted into the database.
- Loss of Information after Firmware upgrade.
- Generated reports show empty on GUI but show the correct one on CLI.
- Default System Templates are not shown in the GUI.
Factory process when a downgrade is needed can be found on the release notes, generally only includes running 2 commands with different purposes:
execute reset {all-settings | all-except-ip}
execute format {disk | disk-ext4 | disk-ext3}
The reset command attempts to delete all incompatible content from the higher version.
Format command attempts to delete all incompatible logs from the higher version.
These actions also modify the cdb upgrade log, where it is possible to notice when a downgrade happened and did not follow the Factory instructions.
Downgrade/Unsupported upgrade path not only corrupts the system itself but also corrupts the databases. For example: a corruption of the DMF file with page/table numbers or index altered.
As sustained by current documentation, the database health check commands can only find issues but not correct them, and the only way to go is to restore from a good working backup:
See Checking FortiManager databases
diagnose pm2 check-integrity all
diagnose dvm check-integrity
diagnose cdb check adom-integrity
diagnose cdb check policy-packages
diagnose cdb upgrade check +all
'The diagnose pm2 check-integrity all command only detects errors. It cannot correct errors. If any errors are found, the only option is to restore from the last good backup before upgrading.'
How to fix this kind of scenario:
Having a good backup practice before any major change is always recommended by ITIL and ITSCM (IT Service Continuity Management) standards.
Prepare a rollback step in case the downgrade/upgrade cannot be completed successfully. Prepare the files to reestablish the device in the original version in case that rollback is needed.
If no good backup is present, redeploying from scratch is the best option to avoid importing corrupted data into the new instance (risk of data loss).
Note :
Using the corrupted backup image after redeploying will cause the same issue again. It is always recommended to keep either a snapshot from the VM or a good regular backup before performing any major changes to revert back once needed and avoid redeployment.
Some tools can be used to speed up the reconstruction process, for example, export a script or template: Export a script
Device import/export list to seed up device registration: Importing and exporting device lists
Related articles:
Technical Tip: How to check FortiManager database prior to upgrade.
Technical Tip: How to check upgrade path and upgrade details of FortiManager, FortiAnalyzer.
|
