Technical Guide: Understanding FortiManager core operations and device database status / policy package status with examples

Description

This article describes the FortiManager core operations that include configuration management using device database and policy package databases, and understanding the workflows that enable these core operations is crucial to ensure the deployment is running optimally to handle various scenarios. Various possible states of the system components of FortiManager are discussed in detail with examples, along with remediations for error statuses.

Scope

FortiManager. While this article primarily focuses on FortiGate management with FortiManager, the operational logic applies to other devices as well.

Solution

Table of contents:

1. Introduction

2. a. Changes pushed from FortiManager to FortiGate

3. b. Changes pulled from FortiGate to FortiManager

4. Device database status types

5. Policy package status types

6. Steps to fully restore FortiGate configs to an old copy in the revision history using FortiManager

7. Steps to fully import current configs and policies from FortiGate into FortiManager

8. Install Report logs

9. Import Report logs

10. Common errors and solutions

11. Best practices to keep device database and policy package statuses healthy

12. Related articles

Introduction:

The core operations of FortiManager at a high level include changes (configs/upgrades/scripts/etc) that are pushed from FortiManager to FortiGates, and the changes that are pulled from FortiGates into FortiManager to enable centralized configuration and revision management. Here are the main components of FortiManager to understand the core operations:

1. Device Layer.

The device layer or device manager layer represents the component that stores device-specific information for devices onboarded and managed in FortiManager. These are usually unique settings per device, such as:

• Network settings.

• System settings.

• VPN settings.

• Security fabric - part 1.

• Log and reports.

• CLI configurations.

• Provisioning templates - overrides device database settings.

• Scripts.

2. Policy Layer.

The policy layer represents the component of FortiManager where reusable objects and policies (with the logic of 'define once, use multiple times for multiple devices') are stored.

• Firewall objects like addresses, services, ISDBs, etc.

• Firewall policies like IPv4 firewall policy, shaping policy, etc.

• Security profiles like webfilter, av, etc.

• Policy Blocks.

• Security fabric - part 2.

• User and authentication.

• Scripts.

3. Global Policy layer.

This layer represents the component of FortiManager where globally reusable objects and policies (in the ADOM context) are stored. The following are some examples of the objects and policies stored here:

• Global objects (used across multiple ADOMs).

• Header policies.

• And footer policies.

Changes pushed from FortiManager to FortiGate.

Actions or operations in the direction from FortiManager -> FortiGate, wherein configuration changes are pushed from FortiManager to FortiGates.

Note: FortiManager is the central management server and the best practice is to use make config & policy changes instead of doing it directly on the FortiGates.

The following are the core operations for most day-to-day use cases on FortiManager:

1. Install operation - For device settings changes or Policies & Objects changes by an administrator (or API calls).

2. Revert & Install operation - To revert a recent change and restore the previous configs, by an administrator (or API calls).

Additionally, there are two other operations, although they are not typically used on a daily basis:

• Quick install operation (for device database only).

• Re-install policy operation (for policy package only).

Additionally, there are system template installs, and CLI script installs - all of these actions use the 'install' operation discussed above.

Changes pulled from FortiGate to FortiManager.

These are actions or operations in the direction from FortiGates -> FortiManager, wherein 'certain' (not all) changes are pulled from the FortiGates to FortiManager.

Bear in mind that making configuration changes directly on FortiGates (while being managed by a FortiManager) is not the best practice. However, there may be exceptional scenarios where changes were made on the FortiGates directly (like using scripts to push changes directly on the Remote FortiGates or config changes on the FortiGate GUI/CLI). For these scenarios, keep the following available actions in mind.

1. Retrieve - manual action on the FortiManager to pull the current configs from FortiGate and ingest them into the FortiManager.

2. Autoupdate - FortiGate triggers an automatic ingestion of config changes (or fabric status) into FortiManager when any new local changes on FortiGate are detected. A keepalive message is sent from FortiGate periodically to FortiManager, which contains the checksum of the FortiGate configuration, which determines the synchronization status, and checks if there is a need for auto-update

3. AutoRetrieve - When a new local config change is detected on the FortiGate, it is the FortiGate that is expected to initiate an 'autoupdate'. But for any reason if the autoupdate fails to initiate on the FortiGate, an autoretrieve is initiated by the FortiManager to initiate a pull request to ingest the changes into the device database.

4. Import - manual action on the FortiManager to pull the current policy and object configs from FortiGate into the FortiManager policy package database (pkg).

Important 'Autoupdate' caveat:

1. If config changes related to device settings are made directly on a FortiGate, these device setting config changes will get auto-updated in the FortiManager in the device's corresponding device-database entry, and also a new revision entry in the revision history database (with an entry named 'autoupdate' and the timestamp) will have these changes included. Additionally, the 'config status' under the Device Manager tab will show 'Auto-update'.

2. On the other hand, if config changes related to firewall policies and objects are made directly on the FortiGate, although these firewall policy/object changes are auto-updated into the revision history database (with a new 'autoupdate' revision entry containing the new firewall policy/object changes): they are not auto-updated into the device's Policy package database (pkg). The 'Policy Package Status' in the Device Manager section will still show a 'green check mark' with a sync status, but note that this does not mean the FortiGate changes are auto-imported into the policy package. Administrator should do an "import" action to ingest these policy changes into the FortiManager. If that is not done, the next time an "install" is done, FortiManager will delete those new policy changes (made directly on the FortiGate) to make it consistent with its own local policy package database.

In the example shown above, an administrator has added a new firewall policy entry on the 'FortiGate-3' directly on the FortiGate GUI (not shown in the screenshots). After this, these policy changes are 'autoupdated' into the revision history in the FortiManager (indicated by 'Auto-update' in Config Status), but the Policy Package is not updated (which is expected) but the Policy Package Status still shows a 'green check mark' with sync status but should not be misled by this status. The administrator should perform an 'import' action to inject the new policy into FortiManager's policy package. The best practice is to avoid making firewall policy changes directly on the FortiGate (and if they are made, perform an immediate import into FortiManager). Instead, make these changes on the FortiManager.

The CLI output shows much more detail about the device's status, with various fields useful for analysis and troubleshooting, as shown below:

FortiManager # diagnose dvm device list

--- There are currently 3 devices/vdoms managed ---
--- There are currently 3 devices/vdoms count for license ---

TYPE OID SN HA IP NAME ADOM IPS FIRMWARE HW_GenX

fmgfaz-managed 223 FGVM01TM2XXXXXX1 - 10.9.11.31 FortiGate-1 root 7.0 MR4 (2662) N/A
|- STATUS: dev-db: modified; conf: in sync; cond: pending; dm: retrieved; conn: up; source: FMG
|- vdom:[3]root flags:0 adom:root pkg:[imported]FortiGate-1

fmgfaz-managed 169 FGVM01TM2XXXXXX2 - 10.9.0.176 FortiGate-2 root 7.0 MR4 (2829) N/A
|- STATUS: dev-db: not modified; conf: in sync; cond: OK; dm: retrieved; conn: up; source: FMG
|- vdom:[3]root flags:0 adom:root pkg:[imported]FortiGate-2

fmgfaz-managed 188 FGVM01TM2XXXXXX3 - 10.9.11.26 FortiGate-3 root 7.0 MR4 (2829) N/A

|- STATUS: dev-db: not modified; conf: in sync; cond: OK; dm: retrieved; conn: up; source: FMG
|- vdom:[3]root flags:0 adom:root pkg:[imported]FortiGate-3

For each device (FortiGate) entry, the first line is primarily dedicated to the device database status, and the second line to the policy package status. A table summarizing what each of these fields refers to is shown below. In the next section, these fields are discussed in more detail with examples.

Device database status types.

The 'status' of the device database (dev-db) describes the current state of the device database maintained by FortiManager for that specific device. The CLI command 'diagnose dvm device list' shows many useful diagnostic details of the DVM including dev-db status as shown in the examples below.

Below is an example of a FortiManager with 3 FortiGates in the database as shown in the GUI:

The following is a table with the various fields in the 'diagnose dvm dev list' CLI output, which provides extensive information regarding device database and policy package statuses. These fields are described in detail in the subsequent sections.

The following is a reference 'diagnose dvm device list' output that will be used to discuss the various fields in the rest of the article.

FortiManager # diagnose dvm device list
--- There are currently 3 devices/vdoms managed ---
--- There are currently 3 devices/vdoms count for license ---
TYPE OID SN HA IP NAME ADOM IPS FIRMWARE HW_GenX
fmgfaz-managed 223 FGVM01TMXXXXXX1 - 10.9.11.31 FortiGate-1 root 7.0 MR4 (2662) N/A
|- STATUS: dev-db: not modified; conf: in sync; cond: OK; dm: autoupdated; conn: up; source: FMG
|- vdom:[3]root flags:0 adom:root pkg:[installed]FortiGate-1
fmgfaz-managed 169 FG80EP4MXXXXXX2 - 10.9.0.176 FortiGate-2 root 7.0 MR4 (2829) N/A
|- STATUS: dev-db: not modified; conf: in sync; cond: OK; dm: retrieved; conn: up; source: FMG
|- vdom:[3]root flags:0 adom:root pkg:[imported]FortiGate-2
fmgfaz-managed 188 FGVMULTMMXXXXX3 - 10.9.11.26 FortiGate-3 root 7.0 MR4 (2829) N/A
|- STATUS: dev-db: not modified; conf: in sync; cond: OK; dm: retrieved; conn: up; source: FMG
|- vdom:[3]root flags:0 adom:root pkg:[modified]FortiGate-3

1. Dev-db.

This field indicates the status of the device database status (device level settings), whether any configuration changes (device settings only) have been made on the FortiManager but not yet installed/pushed to the FortiGate. Note that this field does not cover policies and objects.

• Not modified:

TYPE OID SN HA IP NAME ADOM IPS FIRMWARE HW_GenX
fmgfaz-managed 223 FGVM01TMXXXXXX1 - 10.9.11.31 FortiGate-1 root 7.0 MR4 (2662) N/A
|- STATUS: dev-db: not modified; conf: in sync; cond: OK; dm: installed; conn: up; source: FMG
|- vdom:[3]root flags:0 adom:root pkg:[installed]FortiGate-1

• Modified:

When dev-db is in the modified status, the typical condition status would be 'pending', indicating an install operation is still pending. The following is an example wherein an interface config was changed on the FortiManager.

TYPE OID SN HA IP NAME ADOM IPS FIRMWARE HW_GenX
fmgfaz-managed 223 FGVM01TMXXXXXX1 - 10.9.11.31 FortiGate-1 root 7.0 MR4 (2662) N/A
|- STATUS: dev-db: modified; conf: in sync; cond: pending; dm: retrieved; conn: up; source: FMG
|- vdom:[3]root flags:0 adom:root pkg:[installed]FortiGate-1

The example configuration below would change dev-db to modified, but not the policy package change. For example, ssh was added as an additional allowed service on port2:

config system interface
    edit "port2"
        set allowaccess https fgfm ssh
    next
end

After performing an install operation, it would appear like the following:

TYPE OID SN HA IP NAME ADOM IPS FIRMWARE HW_GenX
fmgfaz-managed 223 FGVM01TMXXXXXX1 - 10.9.11.31 FortiGate-1 root 7.0 MR4 (2662) N/A
|- STATUS: dev-db: not-modified; conf: in sync; cond: OK; dm: installed; conn: up; source: FMG
|- vdom:[3]root flags:0 adom:root pkg:[installed]FortiGate-1

• Unknown.

This indicates the dev-db status is unable to be determined, which is usually when the device is unreachable, upgrade failures, not on-boarded properly, database corruption or other similar reasons.

|- STATUS: dev-db: unknown; conf: unknown; cond: unregistered; dm: none; conn: unknownn|- vdom:[3]root flags:0 adom:root pkg:[never-installed]

2. Conf.

This field indicates whether the latest and active revision entry in the revision history of FortiManager is in sync with the running configurations on the FortiGate or not. This field effectively tracks all components of configs, namely device-level settings, as well as policies and objects, because the revision entry is used to track the status, which contains the full configuration of the FortiGate. So while policies and objects are tracked by this field, this does not track the "policy package database" status, which is important to note.

• In sync:

This status indicates that the device settings, policies, and objects on the FortiManager latest revision history entry for the device are matching local configs, policies, and objects on the FortiGate.

• Out-of-sync:

This status indicates that the latest revision history configuration entry does not match the active configuration on the FortiGate, probably due to configuration changes made locally on the FortiGate or a previous partial install failure. Remediation would be to confirm the existing configs on the FortiGate are accurate and then perform a retrieve followed by import of policies into FortiManager to make the config status change to in sync.

|- STATUS: dev-db: unknown; conf: out of sync; cond: unregistered; dm: none; conn: unknown
|- vdom:[3]root flags:0 adom:root pkg:[never-installed]
|- STATUS: dev-db: modified; conf: out of sync; cond: conflict; dm: retrieved; conn: up
HA cluster member: FG120GTXXXXXXXX1 (primary); conn: up; conf: in sync
HA cluster member: FG120GTXXXXXXXX2 (secondary 1); conn: up; conf: in sync
|- vdom:[3]root flags:0 adom:Remote_offices pkg:[unknown]RO-1 wtp:[modified] fsp:[unknown]

• Unknown:

This status indicates that FortiManager is not able to determine the synchronization status because FortiGate is not reachable or due to a partial install failure. It is recommended to perform a retrieve operation on FortiManager

3. Cond.

This field indicates the condition of the 'Install' operations for a device in FortiManager for 'device-level settings' when there are device setting changes made in FortiManager that are pending an install operation. Note that this cond field only reflects the status of device-level settings and not policy package, so if there are any changes made to policy package (only), the cond status would still be 'OK'. The 'pkg' field in the next line represents the policy package status.

• OK:

This status indicates that the previous device settings install operation was successful and no pending device setting configs to be installed. This is the status a device should ideally be in.

• Pending:

This status indicates device-level settings have been made on the FortiManager, and changes are yet to be installed on the FortiGate.

The following is an example where a revert of revision was done on the FortiGate called FGT3 and how various field statuses changed.

Before reverting:

fmgfaz-managed 188 FGVM01TMXXXXXX1 - 10.9.11.26 FortiGate-3 root 7.0 MR4 (2829) N/A
|- STATUS: dev-db: not modified; conf: in sync; cond: OK; dm: installed; conn: up; source: FMG
|- vdom:[3]root flags:0 adom:root pkg:[installed]FortiGate-3

After the revert operation:

fmgfaz-managed 188 FGVM01TMXXXXXX1 - 10.9.11.26 FortiGate-3 root 7.0 MR4 (2829) N/A
|- STATUS: dev-db: modified; conf: in sync; cond: pending; dm: installed; conn: up; source: FMG
|- vdom:[3]root flags:0 adom:root pkg:[unknown]FortiGate-3

The dev-db changed to modified, with conf still showing in sync, but cond is 'pending' install. The policy package shows 'unknown'.

After an install operation:

fmgfaz-managed 188 FGVM01TMXXXXXX1 - 10.9.11.26 FortiGate-3 root 7.0 MR4 (2829) N/A
|- STATUS: dev-db: not modified; conf: in sync; cond: OK; dm: installed; conn: up; source: FMG
|- vdom:[3]root flags:0 adom:root pkg:[installed]FortiGate-3

Note: The revision revert is not pushing the previous firewall policy - check again in the lab with firewall policy + a device setting.

• Conflict:

This status indicates a condition where changes made locally on the FortiGate were not successfully retrieved, and around the same time, configuration changes are made on the FortiManager. Perform a retrieve operation or install the changes from the FortiManager to remediate this status.

|- STATUS: dev-db: modified; conf: out of sync; cond: conflict; dm: retrieved; conn: up
HA cluster member: FGVM01TMXXXXXX1 (primary); conn: up; conf: in sync
HA cluster member: FGVM01TMXXXXXX2 (secondary 1); conn: up; conf: in sync
|- vdom:[3]root flags:0 adom:Remote_offices pkg:[unknown]RO-1 wtp:[modified] fsp:[unknown]

• Unregistered:

If the FortiGate device registration is not complete, this status is shown. Complete the registration by authorizing the device on the FortiManager.

|- STATUS: dev-db: unknown; conf: out of sync; cond: unregistered; dm: none; conn: unknown
|- vdom:[3]root flags:0 adom:root pkg:[never-installed]

• Offline:

Indicates a device that is managed by FortiManager is currently offline, check the fgfm tunnel status to troubleshoot further.

|- STATUS: dev-db: not modified; conf: out of sync; cond: offline; dm: installed; conn: down
HA cluster member: FGVM01TMXXXXXX1 (primary); conn: up; conf: in sync
HA cluster member: FGVM01TMXXXXXX2 (secondary 1); conn: down; conf: unknown
|- vdom:[3]root flags:0 adom:Remote-offices pkg:[installed]RO-2​

• Out of sync:

Compares the current revision history with the FortiGate local configuration to determine if there is any difference. If there are differences, the config status is marked 'out of sync'.

|- STATUS: dev-db: not modified; conf: out of sync; cond: out of sync; dm: autoupdated; conn: up
|- vdom:[3]root flags:1 adom:root pkg:[unknown]RO-3

• Modified (recently auto-updated):

|- STATUS: dev-db: not modified; conf: in sync; cond: Modified (recent auto-updated); dm: autoupdated; conn: up; source: FMG
HA cluster member: FGVM01TMXXXXXXX1 (primary); conn: up; conf: in sync
HA cluster member: FGVM01TMXXXXXXX2 (secondary 0); conn: up; conf: in sync
|- vdom:[3]root flags:1 adom:root pkg:[conflict]RO-3:[modified]Template-1

4. Dm.

Shows the device manager status, i.e., whether a retrieve, an autoupdate, or an install operation was the last operation done concerning device settings (not policy package).

• Retrieved:

This status means the last operation on the device database for this device was a retrieve operation.

gfaz-managed 188 FGVM01TMXXXXXXX1 - 10.9.11.26 FortiGate-3 root 7.0 MR4 (2829) N/A
|- STATUS: dev-db: not modified; conf: in sync; cond: OK; dm: retrieved; conn: up; source: FMG
|- vdom:[3]root flags:0 adom:root pkg:[imported]FortiGate-3

• Autoupdated:

This status would mean the last operation on the device database for this device was an autoupdate from FortiGate to FortiManager.

TYPE OID SN HA IP NAME ADOM IPS FIRMWARE HW_GenX
fmgfaz-managed 223 FGVM01TMXXXXXXX1 - 10.9.11.31 FortiGate-1 root 7.0 MR4 (2662) N/A
|- STATUS: dev-db: not modified; conf: in sync; cond: OK; dm: autoupdated; conn: up; source: FMG
|- vdom:[3]root flags:0 adom:root pkg:[installed]FortiGate-1

• Installed:

The last operation on the device database for this device was an install operation.

TYPE OID SN HA IP NAME ADOM IPS FIRMWARE HW_GenX
fmgfaz-managed 223 FGVM01TMXXXXXXX1 - 10.9.11.31 FortiGate-1 root 7.0 MR4 (2662) N/A
|- STATUS: dev-db: not modified; conf: in sync; cond: OK; dm: installed; conn: up; source: FMG
|- vdom:[3]root flags:0 adom:root pkg:[installed]FortiGate-1

• Aborted:

This status indicates the previous operation on the device database was aborted (which could have been an install or retrieve or any such action).

|- STATUS: dev-db: modified; conf: in sync; cond: pending; dm: aborted; conn: upn|- vdom:[3]root flags:0 adom:root pkg:[installed]RO-3

5. Conn.

This field indicates the status of the FGFM tunnel between the FortiManager and the device.

Up: Indicates the fgfm tunnel is healthy and in the 'up' status.

6. VDOM.

This field indicates the VDOM to which the FortiGate instance belongs to. By default, it would be the root VDOM, but if multi-VDOM is enabled, the specific custom VDOM to which the FortiGate instance belongs is shown.

TYPE OID SN HA IP NAME ADOM IPS FIRMWARE HW_GenX
fmgfaz-managed 223 FGVM01TM2XXXXXX1 - 10.9.11.31 FortiGate-1 root 7.0 MR4 (2662) N/A
|- STATUS: dev-db: not modified; conf: in sync; cond: OK; dm: installed; conn: up; source: FMG
|- vdom:[3]root flags:0 adom:root pkg:[installed]FortiGate-1

In the example above, the VDOM is the root for the device FGVM01TM2XXXXXX1.

7. Flags.

The flags field is usually set to 0; a non-zero value might need to be analyzed further with Fortinet TAC.

8. ADOM.

If ADOM mode is enabled on FortiManager, the name of the ADOM on which the device is added is shown.

TYPE OID SN HA IP NAME ADOM IPS FIRMWARE HW_GenX
fmgfaz-managed 223 FGVM01TM2XXXXXX1 - 10.9.11.31 FortiGate-1 root 7.0 MR4 (2662) N/A
|- STATUS: dev-db: not modified; conf: in sync; cond: OK; dm: installed; conn: up; source: FMG
|- vdom:[3]root flags:0 adom:root pkg:[installed]FortiGate-1

Policy package status types:

The pkg field in the 'diagnose dvm device list' tracks the current status of the policy package status for each of the managed devices. In the GUI, Device Manager -> Devices & Groups -> Managed Devices shows both device database (device settings) status as well as policy package status, as shown in the previous section. Policy package status can also be viewed under Policy & Objects -> FortiGate -> Installation Targets, as shown in the example below:

FortiManager # diagnose dvm device list
--- There are currently 3 devices/vdoms managed ---
--- There are currently 3 devices/vdoms count for license ---
TYPE OID SN HA IP NAME ADOM IPS FIRMWARE HW_GenX
fmgfaz-managed 223 FGVM01TMXXXXXXX1 - 10.9.11.31 FortiGate-1 root 7.0 MR4 (2662) N/A
|- STATUS: dev-db: modified; conf: in sync; cond: pending; dm: retrieved; conn: up; source: FMG
|- vdom:[3]root flags:0 adom:root pkg:[imported]FortiGate-1

• Pkg:

This field represents the 'Policy Package Status' for a device in FortiManager ADOM, and here are the various status types of this field and their meanings:

• Synchronized:

The policy package status shows 'Synchronized' when all the objects and policies are in sync between the FortiManager entry and the FortiGate. While the GUI will show the policy package status as 'Synchronized', the CLI output of 'diagnose dvm device list' will show either 'installed', 'retrieved', or 'imported'.

TYPE OID SN HA IP NAME ADOM IPS FIRMWARE HW_GenX
fmgfaz-managed 223 FGVM01TMXXXXXXX1 - 10.9.11.31 FortiGate-1 root 7.0 MR4 (2662) N/A
|- STATUS: dev-db: not modified; conf: in sync; cond: OK; dm: autoupdated; conn: up; source: FMG
|- vdom:[3]root flags:0 adom:root pkg:[installed]FortiGate-1

• Never-installed:

This status indicates the policy package entry on the FortiManager (corresponding to that device) was never created and never-installed. Typically, this is the initial status during onboarding of the FortiGate, and to remediate this status, do an 'Import' operation from FortiGate into FortiManager for that device.

|- STATUS: dev-db: unknown; conf: out of sync; cond: unregistered; dm: none; conn: unknown
|- vdom:[3]root flags:0 adom:root pkg:[never-installed]

• Modified:

If the policy package is in the modified status but the config status is in sync, it means only the policy package was modified, and no changes to the device-database or device-level settings on the FortiManager.

The following is an example where a new firewall policy entry has been added to the policy package of the FortiGate called FGT3, but has not yet been installed.

fmgfaz-managed 188 FGVM01TMXXXXXXX1 - 10.9.11.26 FortiGate-3 root 7.0 MR4 (2829) N/A
|- STATUS: dev-db: not modified; conf: in sync; cond: OK; dm: installed; conn: up; source: FMG
|- vdom:[3]root flags:0 adom:root pkg:[modified]FortiGate-3

The following is the output after the 'Install' operation:

fmgfaz-managed 188 FGVM01TMXXXXXXX1 - 10.9.11.26 FortiGate-3 root 7.0 MR4 (2829) N/A
|- STATUS: dev-db: not modified; conf: in sync; cond: OK; dm: installed; conn: up; source: FMG
|- vdom:[3]root flags:0 adom:root pkg:[installed]FortiGate-3

Note that the pkg status has now changed from modified to installed.

• Unknown:

This is a scenario where, due to several conflicting changes and possible install failures, a new device being onboarded is not successful, or there are other similar scenarios due to which FortiManager is unable to determine the exact status of the policy package, and subsequently marks it as 'unknown'.

• Imported:

Policies and objects were imported from FortiGate into FortiManager using the 'Import' action, and the action was successful.

fmgfaz-managed 169 FGVM01TMXXXXXXX1 - 10.9.0.176 FortiGate-2 root 7.0 MR4 (2829) N/A
|- STATUS: dev-db: not modified; conf: in sync; cond: OK; dm: retrieved; conn: up; source: FMG
|- vdom:[3]root flags:0 adom:root pkg:[imported]FortiGate-2

• Out of Sync:

This status indicates that the latest policy package on the FortiManager does not match the policies and objects configurations on the last revision history entry, probably because configuration changes were locally made on the FortiGate, or a partial failure of a previous install. Remediation would be to confirm the existing configs on the FortiGate are accurate and then perform a retrieve followed by import of policies into FortiManager to make the config status change to in sync.

• Conflict:

The pkg status shows Conflict in a scenario when there are policy configuration changes made locally on FortiGate, but those changes were not imported into the policy package of FortiManager, and there were also changes made to the policy package for that device on the FortiManager. To remediate this situation, depending on which changes are required to be retained (i.e FortiManager changes or FortiGate changes): either perform an import of the policy package from FortiGate into FortiManager or perform an install from FortiManager to FortiGate.

• installed:

A recent 'Install' operation of the Policy package was performed, and it was successful.

fmgfaz-managed 188 FGVM01TMXXXXXXX1 - 10.9.11.26 FortiGate-3 root 7.0 MR4 (2829) N/A
|- STATUS: dev-db: not modified; conf: in sync; cond: OK; dm: installed; conn: up; source: FMG
|- vdom:[3]root flags:0 adom:root pkg:[installed]FortiGate-3

Note: A new revision entry is added to the revision history database of a device in FortiManager for 3 main operations.

1. An install action,

2. A retrieve action, and,

3. An auto-update action.

Note that an import action does not create a new revision entry; instead, it only imports the firewall policy & object changes from FortiGate into the Policy Package database (pkg) of FortiManager. This is because either an autoupdate or a retrieve event would have occurred prior (before an import is needed), which would have already created a new revision entry in the revision history database.

Steps to fully restore FortiGate configs to an old copy in the revision history using FortiManager.

There could be some scenarios wherein, after say unintended config changes to the FortiGate (either locally or from FortiManager), an administrator would want to quickly and fully restore the FortiGate configs with a previous 'Good' config copy. For such a use case, use the two steps below to fully restore the FortiGate config to a previous copy in the revision history of the FortiManager.

Step 1: Revert operation.

From the revision history tab, select the revision entry that needs to be restored to and select restore. This will create the changes ready to install the old configs to FortiGate. Next, select the Install wizard to push this to FortiGate. Here, there are two options to choose from: select the first one.

1. Choose the 'Install Device settings only' option, which will push all the configs as per the 'revision entry' to FortiGate - including device settings, policies, and objects.

2. If 'Install Policy package & Device settings' is chosen, FortiManager will combine the 'revision entry' configs with the existing policy package config (which might not be the same as what is in the revision entry), and in effect will not push the correct policies as per the intended revision entry.

Use the 'Install preview' as a best practice to review what config changes are being installed before triggering the install action. Once the install operation is complete, FortiGate will now have the old revision restored fully (device level settings, policies, and objects), but FortiManager's policy package will still be the latest configs and not the policies & objects from the revision entry that was just installed to the FortiGate. The pkg status will be in an unknown state after Step 1. The next step would be to 'import' the policy package from FortiGate to FortiManager to fix this.

fmgfaz-managed 235 FGVM01TMXXXXXXX1 - 10.9.11.26 FortiGate-3 root 7.0 MR4 (2829) N/A
|- STATUS: dev-db: not modified conf: in sync; cond: OK; dm: installed; conn: up; source: FMG
|- vdom:[3]root flags:0 adom:root pkg:[unknown]FortiGate-3

Step 2: Import operation.

Now, import the old firewall policies (corresponding to the restored revision entry) from the FortiGate into ADOM's policy package - this is needed so that FortiManager's policy package is updated with the old configs (as per the restored revision entry).

fmgfaz-managed 235 FGVM01TMXXXXXXX1 - 10.9.11.26 FortiGate-3 root 7.0 MR4 (2829) N/A
|- STATUS: dev-db: not modified; conf: in sync; cond: OK; dm: installed; conn: up; source: FMG
|- vdom:[3]root flags:0 adom:root pkg:[imported]FortiGate-3

After this step, FortiManager will also have the correct device settings and policy package settings as per the selected revision entry, corresponding to what was installed in step 1 to FortiGate.

Note: Remember that the 'Import Configuration' wizard under Device Manager in FortiManager does not import the device-level settings, it only imports the firewall policies and objects (and FortiSwitches).

Steps to fully import current configs and policies from FortiGate into FortiManager.

This can be done using the following three sequential operations on the FortiManager.

Step 1: Retrieve operation.

A Retrieve operation retrieves device-level settings from FortiGate into the FortiManager device database. Note a subtle difference with this operation: after a retrieve is triggered, a new revision entry is created with both device-level settings and firewall policies/objects, but only the device settings are updated in the device database, not the policy package database. To complete the update of the policy package database as well, do step 2 below.

FortiManager-kvm74 # diagnose dvm device list
TYPE OID SN HA IP NAME ADOM IPS FIRMWARE HW_GenX
fmgfaz-managed 235 FGVM01TMXXXXXXX1 - 10.9.11.26 FortiGate-3 root 7.0 MR4 (2878) N/A
|- STATUS: dev-db: modified; conf: in sync; cond: OK; dm: installed; conn: up; source: FMG
|- vdom:[3]root flags:0 adom:root pkg:[imported]FortiGate-3
--- End device list ---

Step 2: Import operation.

Do this to import firewall policies and objects into the policy package database of FortiManager (does not import device config settings). After Steps 1 and 2 are completed, the FortiGate configs (both device settings and policies) are now imported into FortiManager.

FortiManager-kvm74 # diagnose dvm device list
TYPE OID SN HA IP NAME ADOM IPS FIRMWARE HW_GenX
fmgfaz-managed 235 FGVM01TMXXXXXXX1 - 10.9.11.26 FortiGate-3 root 7.0 MR4 (2878) N/A
|- STATUS: dev-db: not modified; conf: in sync; cond: OK; dm: retrieved; conn: up; source: FMG
|- vdom:[3]root flags:0 adom:root pkg:[unknown]FortiGate-3

After the above steps, FortiManager would now have both device settings and policies/objects updated for that FortiGate, but an additional Install should be done to ensure that all the flags (especially pkg) are marked synchronized.

Step 3: Install Operation.

Next, perform an install operation (even though there would be no new config changes to be installed) to ensure the new revision entry created in FortiManager is synchronized with the FortiGate.

FortiManager-kvm74 # diagnose dvm device list
TYPE OID SN HA IP NAME ADOM IPS FIRMWARE HW_GenX
fmgfaz-managed 235 FGVM01TMXXXXXXX1 - 10.9.11.26 FortiGate-3 root 7.0 MR4 (2878) N/A
|- STATUS: dev-db: not modified; conf: in sync; cond: OK; dm: installed; conn: up; source: FMG
|- vdom:[3]root flags:0 adom:root pkg:[installed]FortiGate-3

Install Report logs:

Every Install operation (whether successful or failed) will have a corresponding Install report, which is very useful to monitor and troubleshoot any Install errors. This can be viewed at the following locations:

1. Install Wizard -> View Installation Log.

2. System Settings -> Task Monitor -> select an Install entry -> View Installation Log.

The following is the structure of a typical Install report:

For a successful install operation:

Starting log (Run on device)
Start installing
<config lines being installed>
---> generating verification report
<--- done generating verification report
install finished

For a failed install operation:

Starting log (Run on device)
Start installing
<config lines being installed>
<errors if any during config installs>
---> generating verification report
(vdom root: firewall proxy-policy 1:proxy)
remote original:
to be installed: explicit-web
. . .
<--- done generating verification report
If the Install fails the first time, then retries are attempted:
------- Start to retry --------
< >
---> generating verification report
< >
<--- done generating verification report
install failed

The following are some of the most common errors in Install Reports and what they mean:

1. Node check object fails.
   

This error indicates that the config line that is failing to install is dependent on another config or feature, which is enabled or configured earlier.

node_check_object fail! for < >
value parse error before <object-name>
Command fail. Return code -596

2. Entry not found in the datasource.

This error indicates the CLI syntax being used is likely not supported. Review the install preview and check what configuration lines are causing the issues.

entry not found in datasource
value parse error before <object-name>
Command fail. Return code -3

3. Object check operator error.

This error indicates that the config construct (multiple lines) was discarded due to incomplete configuration.

Must configure <object/feature>
object check operator error, -56, discard the setting
Command fail. Return code 1

Import Report logs:

Every Import operation (whether successful or failed) will also have a corresponding Import report. This can be viewed at the following locations:

1. Import Wizard -> 'Download Import Report'.

2. System Settings -> Task Monitor -> select an Install entry -> View Progress log (note that this is only the high-level progress report and will not contain the full Import Report).

The following is the structure of a typical import report with an example import operation:

Start to import config from device(FortiGate-3) vdom(root) to adom(root), package(FortiGate-3)
"authentication setting",SKIPPED,"(name=, oid=3607, DUPLICATE)"
"firewall address",SKIPPED,"(name=Internal-Network, oid=3618, DUPLICATE)"
"firewall policy",SUCCESS,"(name=2, oid=3619, new object)"
"firewall policy",SUCCESS,"(name=8, oid=3620, new object)"
"firewall profile-protocol-options",SKIPPED,"(name=default, oid=3276, DUPLICATE)"
"firewall service custom",SKIPPED,"(name=webproxy, oid=2610, DUPLICATE)"
"firewall ssh local-ca",SKIPPED"(name=Fortinet_SSH_CA, oid=3273, reason=manually)"
"firewall ssl-ssh-profile",SKIPPED,"(name=no-inspection, oid=3370, DUPLICATE)"
<End of report>

• SKIPPED: The import of the object/policy was skipped with the reason explained subsequently (for example, 'Duplicate', meaning the same object entry and value exists already in the FortiManager database, likely from the import of another FortiGate in the same ADOM). SKIPPED does not indicate any issue with the import, just that FortiManager is optimizing the database for duplicates or other reasons.

• SUCCESS: Indicates that the object/policy was successfully imported, typically because it is a 'new entry'.

• FAILED: Indicates that the object/policy could not be imported, with the reason explained subsequently in the same line.

Common errors and solutions.

The following are some of the most common device management-related error conditions in FortiManager:

1. Conflicts.

The device status will show 'Conflict' in the GUI, typically when the previous installation has failed, but also can indicate that changes made locally on the FortiGate were not retrieved properly and that, around the same time, changes were also made on the FortiManager. To remediate this Conflict status, retrieve the configuration and verify that the operation is complete, or install the changes from the FortiManager if there were expected changes made on the FortiManager.

2. Out of Sync.

The device status will show 'Out of Sync' if the latest revision history entry does not match the configuration on the FortiGate, usually due to configuration changes made locally on the FortiGate, or a previous partial install failure. To remediate this error status, perform a retrieve operation from the FortiManager and verify that this operation is complete. An 'Out of Sync' status for a Policy Package can be resolved by doing an import of policy packages from FortiGate into FortiManager.

3. Retrieve failures.

A Retrieve action, when triggered, causes FortiManager to pull the latest/current configurations (device settings) from the managed device into the device database and also creates a new revision entry. This is used typically to resynchronize the FortiGate config with FortiManager's device database. After a retrieve action, an Import should be done to ensure policy

4. Revert failures.

A Revert operation is used to revert a device database configuration (not the policies & objects, which need to be imported separately) to one of the previous recorded configuration states (in the Configuration Revision History). A revert operation should be followed by an Install operation to ensure the reverted changes are installed on the FortiGate. Common reasons for Revert failures are install failures due to different syntaxes (with previous firmware versions) or database corruptions. Remediations would be to download the config (or just the config diff) of the revision entry manually and install the changes using the Script option within FortiManager.

5. Install failures.

An Install action could fail fully or partially. Review the Install Report to carefully review if it was a partial install and remediate accordingly. The Install Report will contain details about the config install progress, the current stage, and which configs failed to install on the FortiGate. Most common reasons for failures are incorrect CLI syntax for objects (due to ADOM and FortiGate version mismatches) and database corruption, causing existing objects to be modified. For this, run the database integrity checks discussed in the next section.

6. Import failures.

An Import action could fail for multiple reasons. Download the 'Import Report' to locate the reason for the import failure. Further, the local logging level can be set to debug level, and event logs will include failed import logs as well. Some of the most common reasons for import failures are incorrect interface binding and invalid dynamic mappings. Review the import report to identify the specific configs causing the failure and remediate accordingly.

7. Reload failures.

The Reload action is used to update the device-level database with the latest (or a specific) revision history entry in the database. This operation can fail due to inconsistent or corrupt FortiGate configurations, usually caused by not following the proper upgrade path. To troubleshoot such reload action failures, use the CLI command 'diagnose test deploymanager reloadconf <devid>'. This command will show the steps at which the configuration fails to update the device-level database.

Best practices to keep the device database and policy package statuses healthy:

The following are some of the best practices to ensure a healthy status of device databases and policy package databases on FortiManager.

1. Review the install preview each time before doing an Install operation.

For every Install operation, carefully review the 'Install Preview' and verify if the config changes listed in the preview are accounted for, and if any unintended config change is seen, download the preview report, stop proceeding to the next step and then review corresponding changes in the FortiManager - why they were made and who made those changes (can be confirmed from the FortiManager event logs and task monitor under System settings). It is often the scenario that multiple admins might be accessing the FortiManager and one of the admin makes the config changes but does not complete installing those changes. This gets buffered in the config changes (device status will be in modified status), and then when the next admin makes their changes and triggers an install, all the pending config changes (including the first admin's changes) will be merged and pushed to the FortiGate. Hence, it is important to review the Install preview every time before doing the Install operation.

2. Workspace and Workflow modes to avoid config conflicts.

Related to the previous item: if the deployment has multiple administrators accessing and making changes simultaneously on the FortiManager, it is recommended to use the 'Workspace' or 'Workflow' mode to streamline the operations by enabling locking of an ADOM or even specific object/policy while making config changes to ensure conflicting sumultaneous changes by multiple admins can be addressed, more details can be found here: Workspace.

3. Do not leave dev-db and pkg in the 'modified' status for too long.

If there are any devices in the FortiManager database not in synchronized status, and especially either dev-db (device settings) or pkg (policy package) in the 'modified' status, it means there are pending config changes in the FortiManager that either needs to be removed, or installed if the changes are required. Leaving the devices in the 'modified' status for too long could mean that eventually, one of the admins might inadvertently push the previous pending config changes along with their intended configs, as both changes would be merged and pushed to the FortiGate.

4. ADOM revision periodically.

It is recommended to take ADOM revisions periodically (daily or weekly, depending on the velocity of changes) to ensure version tracking and use the option of restoring a previous ADOM version if needed.

5. Before upgrades, ensure all devices are in healthy states (device settings and policy package status).

Before initiating a FortiManager upgrade, it is best practice to ensure all the devices in the ADOMs are in healthy states with a 'Synchronized' status for both device settings and policy package status. If there are any entries in other warning or error states like Modified, out of sync, Conflict, etc, ensure it is remediated (using info from the previous section) first before proceeding with the FortiManager upgrade to avoid any database errors for those devices.

6. Ensure backups are taken before firmware upgrades.

To set a baseline before and after a FortiManager upgrade, always take a backup of the config + database from the Dashboard -> System Configuration -> Backup, which will be a .dat file (includes both the configs and the database), which can be restored if needed as a rollback plan later.

7. ADOM compatibility checks before and after upgrades.

Before planning to upgrade the FortiManager Firmware or ADOM version, always verify the compatibility for the FortiOS versions of FortiGates being managed for the current version and the new version, as well as ADOM version compatibility, using the tools below:

1. Compatibility tool

2. ADOM versions

3. Verifying ADOM upgrade readiness

8. FortiManager backups - manual or automated.
   

Backups can also be taken via the CLI with 'execute backup all-settings <>'. It is recommended to implement an automatic scheduled backup from the FortiManager. For an example of how to set it up, see Backing up the system.

9. Run ADOM DB integrity checks immediately after upgrades.

It is best practice to run the list of ADOM database integrity both before an upgrade and after the upgrade (of either the FortiManager firmware or an ADOM version upgrade). Note that the integrity check commands in FortiManager are not read-only operations: they will attempt to automatically modify the database if any errors are found, so make sure to take the config and database backup before implementing the ADOM integrity checks. The following is the list of database integrity check commands:

1. To do an integrity check on the device manager databases of all ADOMs, the list of checks done is shown in the example output below:

FortiManager-kvm74 # diagnose dvm check-integrity
[1/11] Checking object memberships ... correct
[2/11] Checking adom nodes ... correct
[3/11] Checking device nodes ... correct
[4/11] Checking device vdoms ... correct
[5/11] Checking duplicate device vdoms ... correct
[6/11] Checking device ADOM memberships ... correct
[7/11] Checking device HA Secondary ... correct
[8/11] Checking device clusters ... correct
[9/11] Checking groups ... correct
[10/11] Checking group membership ... correct
[11/11] Checking task database ... correct

2. To check any ADOM integrity errors, if found, the CLI will automatically attempt to fix them:

FortiManager # diagnose cdb check adom-integrity
General updating - adom Engineering-ADOM ... ..60%.100% No errors
General updating - adom FortiCarrier ... .100% No errors
General updating - adom FortiFirewall ... .100% No errors
General updating - adom FortiFirewallCarrier ... .100% No errors
General updating - adom QA-ADOM ... ..90%.100% No errors
General updating - adom Unmanaged_Devices ... .100% No errors
General updating - adom root ... ..90%.100% No errors
General updating - adom Global ... .100% No errors

3. To do an integrity check on the dynamic mappings of the policy package, and fix any invalid mappings:
   

FortiManager # diagnose cdb check policy-packages
Adom Engineering-ADOM
[1/7] Checking Scope ...
The scope of fext profile 235-3 is missing
1 change(s) will be made
[2/7] Checking Dynamic mappings ... correct
[3/7] Checking Policy package settings ... correct
[4/7] Checking Cross-linked objs ... correct
[5/7] Checking Object parent mismatch ... correct
[6/7] Checking Undeleted objs ... correct
[7/7] Checking Controller/Template package status ...
Package oid [6102], package name [235-3] : fortiextender invalid
1 change(s) will be made
< snippet >
The above change(s) will be made to the database, however it is recommended to perform a backup first.
Do you want to continue? (y/n)y
Adom Engineering-ADOM
[1/7] Checking Scope ... 1 change(s) made
[2/7] Checking Dynamic mappings ... correct
[3/7] Checking Policy package settings ... correct
[4/7] Checking Cross-linked objs ... correct
[5/7] Checking Object parent mismatch ... correct
[6/7] Checking Undeleted objs ... correct
[7/7] Checking Controller/Template package status ... 1 change(s) made

4. To perform an object configuration database integrity check:

FortiManager # diagnose cdb upgrade check objcfg-integrity
Checking: Object config database integrity
No error found.
FortiManager #

5. Diagnostic command to perform a check to see any issues with the database reference table, and automatically perform an upgrade/repair:

FortiManager # diagnose cdb upgrade check reference-integrity
Checking: Reference table integrity
No error found.
FortiManager #

6. Device info updates:

FortiManager # diagnose cdb check update-devinfo logdisk-size
Working on item[logdisk-size] new-value[n/a] nocheck[0] model(s)[n/a] dump_only[1]
Total checking [3] devices
100.00% checking DB Finished.ate] oid[ 235]...Done.
Dumpling summary:
csum[0][FortiGate-80E-POE][47][8][1].logdisk_size[1] log_disk_quota[0]
(Access [/var/log/update_devinfo.log] for extra log if needed.)
FortiManager #

After running the integrity check commands, if there were some issues found and the CLI outputs show they are being fixed: after the completion of the CLI command, re-issue the integrity check commands to verify no more errors are pending a fix.

10. Use FortiManager as the central management server and avoid making changes directly on FortiGate.

When a FortiGate or any Security Fabric device is managed in FortiManager, it is highly recommended to ensure all config changes are made and installed from FortiManager, and avoid doing any config changes directly on the FortiGates, as it could cause unintended conflict states in the FortiManager device databases and will require additional remediations. If there are some rare scenarios where direct config changes are needed directly on the FortiGates, use the 'Scripts' option in the FortiManager to perform the 'Remote FortiGate Directly (via CLI)' option.

11. High availability.

As with any infrastructure component deployment, implementing redundancy with High Availability for FortiManager is paramount. FortiManager supports HA cluster deployment, with the Primary automatically synchronizing configs and the database to secondary units, along with either manual or automatic (VRRP) failover modes when the Primary unit experiences an issue. More details about High Availability for FortiManager are documented here: High Availability.

Related articles:

• Technical Tip: How to perform FortiGate config backup to FortiManager

• Troubleshooting Tip: FortiGate is Out-of-sync in the Device Manager