Description
This article describes how to upgrade an ADOM on FortiManager and how to perform basic troubleshooting in case of an ADOM upgrade failure.
Scope
FortiManager versions between 5.4.x and 7.4.x.
Solution
Prerequisites and Notes:
ADOM upgrade requires system level administrator permissions and access to the respective ADOM/s (eg., Super_User admin profile).
In the firmware versions within the scope of this article (5.4.x to 6.4.x), an ADOM can only be upgraded after all the devices within this ADOM have been upgraded.
Starting in FortiManager 7.0.1, the ADOM version can be upgraded without first updating all devices and the ADOM can manage N+1 devices during migration. This is something called mixed mode or migration mode.
Starting in FortiManager 7.4.2, it is possible to have one ADOM in 7.4 that can manage FortiOS in 7.0, 7.2, and 7.4.
When a FortiManager unit is upgraded, ADOMs are not upgraded automatically.
The ADOM upgrade operations have to be done separately after the FortiManager upgrade.
When upgrading FortiManager, check if the new firmware is compatible with all existing ADOM versions. If not, make sure to upgrade the ADOMs to a supported version before proceeding with the FortiManager upgrade.
Otherwise, ADOMs in unsupported versions will become unavailable after the FortiManager upgrade.
Although it is possible to manage FortiGates with different versions within the same ADOM, there are a few limitations:
- 'Import Policy' is not supported if the FortiGate version is different than the ADOM version.
- Configuration features implemented in the newer FortiGate version may not be available in the older ADOM version.
- There might be a mismatch in the CLI syntax of some ADOM objects, causing installation or verification errors (eg., new syntax implemented in FortiOS which is not available in the database of older ADOM versions).
To upgrade an ADOM:
- Go to System Settings -> All ADOMs.
The same view in CLI:
Note:
It is not possible to upgrade an ADOM using the CLI.
This operation has to be done on the GUI.
This operation has to be done on the GUI.
-
Select an ADOM and select 'Upgrade', or select an ADOM, select 'More', and then 'Upgrade' from the toolbar.
If the ADOM has already been upgraded to the latest version, this option will not be available.
Select 'OK' in the confirmation dialog box to upgrade the device.
In the above/below picture the ADOM has been successfully upgraded.
The new ADOM version is then displayed in the 'Firmware Version' column.
Under version 6.4 and above select the ADOM that will be upgraded and go to More -> Upgrade
Synthetic diagram when ADOM upgrade is successful.
If all units within the ADOM are not already upgraded, the upgrade will be stopped and an error message will be shown.
The example below illustrates the failed ADOM upgrade: 'Please upgrade all devices to 5.6 before upgrading the ADOM'.
In such a case, upgrade the remaining devices within the ADOM, then return to step 1 to try upgrading the ADOM again.
Synthetic diagram when ADOM upgrade is failing.
Troubleshooting ADOM Upgrade:
In some cases, (if all the FortiGate into the ADOM are upgraded), the ADOM procedure may fail for many reason.
Find below the different reasons preventing a customer to upgrade an ADOM (Non exhaustive list):
In some cases, (if all the FortiGate into the ADOM are upgraded), the ADOM procedure may fail for many reason.
Find below the different reasons preventing a customer to upgrade an ADOM (Non exhaustive list):
- Name conflicts in wildcard FQDN address on SSL/ssh profile (for ADOM 5.6, 6.0, and 6.2).
- Different CLI syntax on objects/profiles.
- Miscellaneous inconsistencies on firewall objects.
- FortiGate object table size limitation changes.
All of these errors mentioned above are known fixed bugs and still exist in the customer environment because the ADOM has never been upgraded in the past even though the FortiManager is using the latest firmware version. Different firmware versions have different features, and therefore different CLI syntax. That is the reason why the ADOM upgrade might not be successful.
FortiManager Debug commands.
In such a case, Fortinet recommends using the below CLI troubleshooting commands for FortiManager versions prior to 7.4.x:
diagnose debug enable
diagnose debug service cdb 255
diagnose debug service cdb 255
For FortiManager versions 7.4 and above, use the following troubleshooting commands:
diagnose debug enable
diagnose debug service dvmdb 255
These CLI commands will help to localize and identify the root cause of the problem that prevents to upgrade of the ADOM.
In most cases, removing the concerned object/profile/interface allows to fix the issue and successfully upgrade the ADOM.
Another scenario can happen: many errors are preventing to upgrade of the ADOM.
Find the first error, then fix it and try to upgrade the ADOM: without success. In such a case, use the same method and CLI commands to identify the object/profile/interface causing the problem.
If the concerned object is used and/or important in the configuration (cannot be modified), contact Fortinet support for further assistance.
The ADOM upgrade debugging will always stop on the concerned error.
Below are some examples of FortiManager debugging after a failed ADOM upgrade:
Example 1:
--> commit copy firewall address.autoupdate.opera.com(soid=149) to dparent=1227, fail: err=-2, Name conflicts with an entry in wildcard FQDN address
name: autoupdate.opera.com ---> autoupdate.opera.com
subnet: 0.0.0.0 0.0.0.0 ---> 0.0.0.0 0.0.0.0
type: fqdn ---> fqdn
start-ip: 0.0.0.0 ---> 0.0.0.0
end-ip: 0.0.0.0 ---> 0.0.0.0
fqdn: autoupdate.opera.com ---> autoupdate.opera.com
associated-interface: any ---> any
wildcard: 0.0.0.0 0.0.0.0 ---> 0.0.0.0 0.0.0.0
cache-ttl: 0 ---> 0
color: 0 ---> 0
visibility: enable ---> enable
uuid: 2fe03af0-43b8-51ea-1233-d6844b291acd ---> 2fe03af0-43b8-51ea-1233-d6844b291acd
allow-routing: disable ---> disable
obj-id: 0 --->
Explanations of the previous error: By default, in 6.0 ADOM some firewall addresses have the same name as wildcard FQDN i.e.: 'autoupdate.opera.com', 'google-play', etc.
When upgrading to 6.2, it will hit the newly added check of not allowing firewall address to have the same name as a wildcard FQDN.
Example 2:
copy firewall address.autoupdate.opera.com(soid=149) to dparent=1227,: fail.
-> commit copy ssh.(soid=12315) to dparent=13106, fail: err=-2,Must set at least one port or enable ssh inspect-all.
======= Dump sentry and dentry======
12315 ---> 13113
status: deep-inspection ---> deep-inspection
inspect-all: disable ---> disable
unsupported-version: bypass ---> bypass
ssh-policy-check: disable --->
ssh-tun-policy-check: disable ---> disable
ssh-algorithm: compatible ---> compatible
===================================
copy ssh.(soid=12315) to dparent=13106, :fail.
copy firewall ssl-ssh-profile.custom-deep-inspection(soid=11436) to dparent=12315, :fail.
Explanations of the previous error: The ADOM upgrade failed because the configuration of SSH (set ports 22) is not set in the 'custom-deep-inspection' profile.
Global Database Specifics:
Global Database Specifics:
Only the 'Upgrade' option should be used for upgrading the Global Database to a higher version.
If the Global Database version is edited (modified) by selecting 'Edit' and then selecting either a higher or lower version, all data is deleted.To upgrade the Global Database:
- Go to System Settings -> All ADOMs.
- Select Global Database -> 'More' from the top menu bar -> Upgrade.
* If the ADOM has already been upgraded to the latest version, this option will not be available. - Select 'OK' in the Upgrade ADOM dialog box.
- After the upgrade finishes, select 'Close' to close the dialog box.
Related documents:
