Technical Note: FortiManager Tips and Best Practices Guide

[ccapone_FTNT]

Description

A FortiManager Best Practices Guide (originally published in August 2017) is now available in the FortiManager section of the Fortinet Document Library.  For users of FortiManager VM, sizing guidelines are now available in the FortiManager VM Installation Guide.

This document provides tips and best practice suggestions for FortiManager firmware versions 4.0 MR3 Patch 7 (also known as 4.3.7, Build 700) or later, and 5.0 GA Patch 5 (also known as 5.0.5, Build 266) or later and version 5.2 GA Patch 1 (also known as 5.2.1, Build 662) or later, and 5.4.0 GA (Build 1019) or later, and 5.6.0 GA (Build 1557) or later. The majority of the information within this document applies to older patches or MR firmware releases as well, however certain CLI command syntax might no longer be relevant. For best operation, please ensure that you are running the latest patch release for your main firmware branch (firmware train).

This document may be used as a reference for the implementation and daily usage of the FortiManager unit.

Related Fortinet Documents and Manuals

All Fortinet product documentation can be found at http://docs.fortinet.com/ . It includes Administration Guide, CLI Guide, and Installation Guide, as well as technical notes. The main categories are listed below.

FortiManager documentation:
http://docs.fortinet.com/fmgr.html

Fortinet Hardware System Test:
See related article.

Solution

FortiManager Tips and Best Practices

CLI Differences Between 4.0 MR3 and 5.0/5.2/5.4/5.6

The CLI syntax changes slightly between 4.0 MR3 and 5.0/5.2/5.4/5.6.  All version 4.0 MR3 "fmsystem" commands changed to "system" commands in 5.0/5.2/5.4/5.6.  The CLI information provided in this document is formatted for version 5.0 and later. 

Examples:

get fmsystem stat -> get system stat

diag fmsystem print df -> diag system print df

config fmsystem global -> config system global

Recommended FortiGate Firmware Versions

Various FortiGate firmware issues have been identified and corrected which directly impact the FortiGate Add and discovery process, FGFM management tunnel establishment, and Installation operations.  The currently recommended FortiGate firmware versions for most reliable FortiManager operation are:

• 4.0 MR3 Patch 15 (Build 0672) or later
• 5.0 GA Patch 10 (Build 0305) or later
• 5.2 GA Patch 11 (Build 0754) or later
• 5.4 GA Patch 5 (Build xxxx) or later

Upgrade, Downgrade and Restore Limitations

FortiManager system DOES NOT SUPPORT downgrades on a populated or factory default database.
FortiManager system DOES NOT SUPPORT the restore of a backup file on a mismatching firmware version.
FortiManager system DOES NOT SUPPORT the restore of a backup file, on matching firmware WITH an existing database (configuration).
FortiManager upgrade path MUST BE FOLLOWED as indicated in the Release Notes.

You might be able to perform some of these operations, which are not supported, without seeing any immediate problem; however, unrecoverable backend problems are to be expected during the subsequent usage.

After any firmware downgrade process on a FortiManager unit, the full factory reset procedure must be performed.

Before attempting ANY configuration restore procedure on a FortiManager unit, the full factory reset procedure must also be performed.

FortiManager Database Integrity

The FortiManager unit must NEVER be powered off without a graceful shutdown, as such action can be damaging to the internal databases.  Always use the following shutdown command prior to powering off:

execute shutdown

It is highly recommended, that FortiManager unit power cord is connected to an uninterruptible power supply (UPS), in order to prevent an unexpected power off, which can potentially damage the internal databases.

ADOM locking (or “Workspace”) feature MUST be enabled, if multiple simultaneous operators will be performing actions on the FortiManager unit, in order to prevent database corruptions. 

Not respecting the upgrade path or configuration restore and downgrade limitations as described in the "Downgrade and Restore Limitations" section, will likely result in a damaged or inconsistent database.

The following CLI commands can be used to verify and correct certain database integrity errors.  Not all integrity problems will be detected, nor could be corrected, by these commands.  They should be run when there are no active operations being performed, and if ADOM locking is enabled, ALL ADOMs must be unlocked first:

diagnose dvm check-integrity
diagnose cdb check objcfg-integrity
diagnose cdb check policy-assignment
diagnose pm2 check-integrity all
diagnose cdb check reference-integrity  [Added in 5.2.2]
diagnose cdb check policy-packages      [Added in 5.2.9/5.4.2]
diagnose cdb check adom-revision        [Added in 5.4.2]
diagnose cdb check adom-integrity       [Added in 5.2.4/5.4.0] <- [Update June 8, 2017] This command was introduced in version 5.2.4, however it did not work correctly, as it modified all Policy Package statuses after execution.  It was also possible that it did not complete successfully and aborted during execution.  These issues have been corrected as of 5.2.11/5.4.3 or later, and can now be used correctly.

If a database correction is attempted, it is recommended to run the command again a second time, in order to confirm that the changes were correctly done.

It is recommended to perform these checks and corrections prior to a firmware upgrade.  An inconsistent database which is upgraded, might end up in a worse condition.

The indication that there is a data integrity problem, might underline another issue(s) which cannot be detected and corrected by these commands.  To be absolutely safe, it is recommended that the FortiManager be wiped and that data be restored from a previously known good backup.

If the data integrity problem cannot be corrected, the FortiManager must be wiped, and data restored from a previously known good backup.

How To Factory Reset (Wipe) a FortiManager unit

The following two commands must be executed from the console port, in this particular order:

execute reset all

OR

execute reset all-except-ip  [as of 5.2.3]

This erases the "show" configuration which is stored on the flash memory, containing IP and routes, except for the new 5.2.3 command which keeps the IP and routing configuration.

execute format disk

This deletes all device information, databases, logs and re-partitions the hard disk.

If downgrading the firmware image, you MUST reformat the disk once more.  This is to ensure that the factory default database settings are correctly regenerated.  This also ensures that the disk partition layout is correctly set for that firmware version.

If upgrading to a new firmware image, it is suggested to reformat once more, but is not an absolute requirement in all cases.
Reformat is required when the new version supports a modified hard disk partition layout*, which might be beneficial for Web-Filtering/Anti-Spam services or improved Logging functionality. 

*The hard disk partition layout has been modified four times with the following firmware releases, starting with the first version shown below: 

- 3.0 MR6 and later
- 3.0 MR7 Patch 7 and later OR 4.0 and later  : (the same partition layout change was applied simultaneously to these two firmware branches)
- 4.0 MR2 Patch 8 and later OR 4.0 MR3 Patch 2 and later : (the same partition layout change was applied simultaneously to these two firmware branches)
- 5.0 and later

The FortiManager Backup File

The backup file is saved with a .dat file extension, but it is actually a .tgz file of the internal "/var" directory and its subdirectories, containing all devices and global database information, as well as the FortiManager system configuration, which is stored on the flash memory.

It does not contain any Event logs, FortiGuard Anti-Virus, IPS, Web Filtering and Anti-SPAM objects, and FortiGate firmware images.

It must be saved UNENCRYPTED (no password set) in order to be able to extract the .tgz file.

An unencrypted backup file which fails to decompress with an utility such as tar, 7-zip, WinRar, etc., is likely corrupt or incomplete, and will fail to restore as well.

It is suggested to save the file without the Encryption option, and to store it safely or to encrypt it offline if required.  An unencrypted backup file might eventually be repairable by Fortinet technical support services, should the backup file be corrupted in such a manner that it fails to restore.  

Restoring only the system level configuration

This is useful when replacing a FortiManager Slave unit for example.

It is not possible to ONLY restore the FortiManager system level configuration (such as IP address and network routing only) from a backup file. 

It is possible to extract the system level configuration from the backup file, by using a decompression utility such as tar, 7-zip or WinRar. The system configuration file is stored under /var/fwclienttemp/system.conf filename.

The CLI configuration can then be copied & pasted via a serial or terminal session.  It is best to do this in chunks of not more than 30 text lines at a time.

ADOM Usage

The simplest method of the FortiGate management is by using a single ADOM. However, multiple ADOMs will become an absolute requirement, when any of the following conditions occurs:

- Different FortiGate units (or VDOMs) must use objects with the same name, but containing different values.

- Simultaneous management operations need to be performed on different FortiGate units.

- Various FortiGate firmware versions are being managed (for example, version 5.0 together with 5.2).

- Administrative or management access to certain FortiGates or VDOMs must be restricted.

- If devices other than FortiGates need to be managed, or in order to have Logging and Reporting abilities for certain non-FortiGate devices, such as FortiCarrier, FortiMail, FortiWeb, etc.

In a single ADOM management mode, it is possible to use the device group feature, to obtain certain management flexibility. For example, it can be used to perform a single Script execution or Install operation on a grouped and restricted amount of FortiGate units.

Concurrent Operator Usage

Safe concurrent and multiple operator usage on the FortiManager unit is possible by enabling the “workspace” feature.  Enabling “workspace” feature will turn on an ADOM level or Policy Package level locking mechanism, which ensures that only one operator is performing a write operation to the FortiManager databases.

Concurrent and multiple operator usage without the “workspace” feature enabled is risky, and may very likely end up corrupting the data within the databases.

CLI Script Execution

CLI scripts can be used to provision FortiGate units or to automate configuration changes. Scripts can be executed (Run) at three different levels (Global, ADOM and Device), and therefore different databases. Scripts can also be executed directly on the FortiGate unit, which will then be followed by an automatic Retrieve operation. There are therefore four different methods of executing a CLI Script on the FortiManager unit.

It is recommended to execute CLI scripts in a top-down approach starting at the highest possible level, and to then Install the changes to the FortiGate.  The highest level is the Global database, and the lowest the Device database. 

In versions previous to 5.4, CLI script names had to be unique across all ADOMs.  A way to workaround this, was to add a short ADOM name prefix to each CLI script name.  As of version 5.4 and later, the same script name can exist in different ADOMs.

ADOM Import Operation

Firewall policies and related objects, can be created in an ADOM via the Import operation. The Import step can either be part of the device Add/Discovery process, or can be manually performed within Device Manager as an Import Policy operation.

It is important to understand, that during the Import operation, the firewall policies and objects that are imported into the ADOM database are taken from the Device-level database. Therefore, if the FortiGate policies or objects have been directly modified on the device, and the FortiGate unit is out-of-sync with the FortiManager unit, then the Import process will not update the ADOM database with those FortiGate configuration changes. An Import process is therefore also possible, if the FortiGate unit is not reachable by the FortiManager unit.

FortiManager HA Data Synchronization

FortiManager HA synchronizes all global and device level databases from primary ("master") to subordinate ("backup","slave") units.
Certain system-level configuration settings are independent on each member, and must be individually configured. For example: Logging settings, FortiGuard settings, SNMP settings.

All FortiGuard objects (Anti-Virus, IPS, Anti-Spam and Web-Filtering) are not synchronized between primary and subordinate units. Each subordinate unit operates independently from the primary unit, downloading and updating its own FortiGuard databases.

System-Level Configuration Best Practices

Certain system-level configuration settings are independent on each FortiManager HA cluster member, and must be configured individually on each unit.

Enable pre- and post-installation verifications, and increase Installation & Script logging history:

conf system dm
set dpm-logsize 10000
set force-remote-diff en
set verify-install en
set script-logsize 10000
end

Increase local Event logging level to Debug:

conf system locallog disk setting
set status en
set severity debug
end

Note: In environments where there are over 1000 managed units, and depending on the type and amount of daily activity, it is recommended to monitor disk (i/o wait states) and CPU activity after increasing this level, in order to ensure that there are no significant increases.

Enable antivirus and IPS package update and distribution event logging and Update History View:

conf fmupdate av-ips advanced-log
set log-fortigate en
set log-server en 
end

Enable antispam and web filtering package update and distribution event logging:

config fmupdate web-spam fgd-setting
set linkd-log enable/debug

set update-log enable 

set stat-log-interval 60
end

Disable all antispam and web filtering lookup logging events. The logging of these events will have a negative performance impact on the hit-rate of the AS/WF service. They will increase disk and CPU usage, and must only be enabled temporarily for debugging purposes:

config fmupdate web-spam fgd-setting
set as-log disable
set av-log disable
set wf-log disable   

set fq-log disable

end

Enable SNMP v2 (only) trap notifications concerning various events, such as redundant power supply failure, low disk usage and FortiManager HA failure:

config system snmp sysinfo
set status enable
end
config system snmp community
edit 0
set events disk_low ha_switch intf_ip_chg sys_reboot cpu_high mem_low log-alert log-rate log-data-rate lic-gbday lic-dev-quota cpu-high-exclude-nice
set name "public"
set query_v1_status disable
set trap_v1_status disable
end
config system snmp community
edit 1
config hosts
edit 0
set ip <SNMP MONITORING SERVER IP>
end
end

Configure an automated daily backup of the FortiManager database.  If possible, it is best that this is performed during an idle or quiet period of the day:

config system backup all-setting
set status enable
set protocol <ftp, scp, sftp>
set server "<FTP/SCP/SFTP SERVER IP>"
set user “<USER_NAME>"
set passwd “<PASSWORD>”
set directory “<DIRECTORY>"
set week_days monday tuesday wednesday thursday friday saturday sunday
set time "23:00:00"
end

Configure remote event logging to a FortiAnalyzer unit or Syslog server:

config system log fortianalyzer
set status enable
set ip <FortiAnalyzer IP>
end
config system locallog fortianalyzer setting
set severity debug
set status enable
end
config system locallog syslog setting
set severity debug
set status enable
set server <syslog server IP>
end

# As of v5.2.1, it is configured as follows:

config system locallog fortianalyzer setting
set status realtime
set server-ip <FortiAnalyzer IP>
set severity debug
end

config system syslog
edit mysyslogserver
set ip <syslog server IP>
end

conf system locallog syslogd setting
set status enable
set severity debug
set syslog-name mysyslogserver
end

Increase the maximum amount of Task Monitor entries that are stored prior to rolling them over.
By default, only 100 Task Monitor entries are stored.  This is usually insufficient, as it can easily be rolled within less than a day, and sometimes with a single operation (for example, an Import of a multi-VDOM unit).  It is recommended to increase this value to 2000.  This can be done via the GUI: System Settings -> Advanced -> Advanced Settings -> Task List Size.  As of 5.0.6, it is also possible to configure this via the following CLI setting:

config system global
set task-list-size 2000
end

Enable ADOM revision roll-over to avoid excessively large backup files.

conf system global
set adom-rev-auto-delete by-revisions
set adom-rev-max-backup-revisions 15
set adom-rev-max-backup-revisions 2   [Added as of 5.4.2 - reduces the amount of ADOM revisions for backup purposes only]
end

 

Event and Internal Logging Correlation

The FortiManager system continuously logs various FortiGuard activity to internal log files on the hard disk. These files can be extracted, and uploaded to a FTP/SFTP server if necessary, for investigation and troubleshooting purposes. 
In order to easily correlate timestamps between these internal log files, and any other Event log activity collected by a FortiAnalyzer unit or Syslog, it is recommended that all units (FortiManager, FortiAnalyzer, FortiGates) are configured to synchronize date and time to a common NTP server.

FortiManager NTP server configuration:

config system ntp
config ntpserver
edit 1
set server <NTP server IP>
next
end
end
config system ntp
set status enable
end
config system ntp
set sync_interval 60
end

WebUI Performance & Behavior

The WebUI performance will depend on the system specification of the FortiManager hardware platform or virtual machine, as well as the client PC and web browser used, due to the Javascript execution.
A faster client PC will improve the WebUI display performance.
Different web browsers, and their versions, may show different performance and at times different behavior as well. The currently supported web browsers are:
Firefox v32 and greater
Internet Explorer v10 and greater
Chrome v38 and greater

If encountering an odd GUI display issue, such as partial or incomplete display of a tab, an option(s), object(s), icon(s) or an entire menu, try clearing all browser cache history.

Disable any browser addons/plugins as these may have adverse performance impacts on the FMG GUI (ex: Skype Click to Call).

Also try a different supported browser to see if it behaves any differently.

Upgrade of a FortiManager unit

During the firmware upgrade, the FortiManager does not upgrade (or modify) the existing objects in the databases.  For example, all FortiGate 5.0 related objects will continue to use the same 5.0 CLI syntax, following a FortiManager 5.0 to 5.2 upgrade.

It is recommended to have console port access during the upgrade, and to log all output to a file.  There are conditions where certain upgrade error messages are only displayed on the console port, and if not captured at upgrade time, they are then no longer recoverable.  These error messages should be supplied to Fortinet technical support via a FortiCare ticket.

Verify database integrity prior to upgrading, using the commands detailed in the previous "FortiManager Database Integrity" section.  It is not recommended to upgrade if errors are detected, as these might further compromise the upgrade process.

It is recommended to verify database integrity after the upgrade as well.

It is recommended to clear the browser’s cache history following a upgrade. 

The release notes provide the details concerning the supported upgrade firmware path.

As of FortiManager version 5.0.4, an ADOM “migration” mode is supported in a 4.3 ADOM. This new feature allows for the ‘restricted’ management of 5.0 FGT devices which have been upgraded from 4.3 and continue to be managed in a 4.3 ADOM. It is a one-way only management mode – Policies and Objects from 5.0 devices can’t be Imported in a 4.3 ADOM.   Once all FortiGates have been upgraded to a 5.0 version, the 4.3 ADOM can be upgraded as well to 5.0 in order to provide full 5.0 object version support functionality. The 5.0 to 5.2 migration mode feature is available with FMG version 5.2.1 or later.  FMG 5.4.1 supports ADOM migration for FGT devices running 5.2 which are being upgraded to 5.4

FortiManager VM Memory, CPU & Disk Requirements

The base VM image is configured for only 512 MB or 2 GB of virtual memory.  The current hardware platforms support between 4GB to 128GB of memory.  The recommended amount of memory is at least 4GB. If using the FortiGuard Web Filtering & Antispam service on the FortiManager unit, then an additional 8GB of memory is required in order to cache the entire copy of the WF/AS db, as well as for the new one which gets updated regularly. 

The base VM image is configured for only 1 virtual CPU.  The current hardware platforms support between 2 and 8 CPUs.  Adding additional virtual CPUs will improve performance, especially during Install operations to multiple devices.  The current minimal recommendation is 2 CPUs.

For optimal Install performance, the recommendation is to provide 2GB of memory per CPU core. For example, a FMG-VM configured with 8 CPUs, should be allocated at least 16GB of memory (excluding additional memory required for FortiGuard services).  If FortiGuard Web Filtering services are enable, then an additional 8GB of memory needs to be allocated for that service.

The base VM image is configured with an 80GB virtual hard disk. The current hardware platforms support between 500GB and 2TB. The 80GB will be sufficient if the FortiManager RTM (Real-Time Monitoring), Log Viewing and Reporting features are NOT used. If these features are required, then the virtual disk size must be increased.
 

Rev. 3.1

Related Articles

RMA Note: HQIP - Hardware Quick Inspection Package