Staff

Technical Tip: How to configure and optimize FortiManager as Local Web filter Server

Forum|Forum|2 years ago
February 20, 2024
2 replies
12787 views

Description

The article provides instructions on setting up and optimizing FortiManager as a UDP Local Web Rating Server (web-filter) and includes troubleshooting tips. Although the article emphasizes a UDP-based web-filter setup, there is the option to choose a TCP scenario (default and more secure) by referring to the KB article linked in the notes under the 'Step 1' section.

Scope

FortiManager. This guide applies to FortiManager v7.0.x and later. Some commands or GUI options may vary in earlier versions.

Solution

Prerequisites:

Access to FortiManager with administrative privileges.
Internet access on FortiManager to fetch the latest web rating database from FortiGuard servers.
Web-filter policies are configured on FortiGate with an SSL inspection profile.
DNS server IPs configured on FortiManager (port1).

Schema of the configured environment:

Glossary and terminology:

Antivirus (AV): Software designed to detect and remove computer viruses.
Intrusion Prevention System (IPS): A Security tool that prevents unauthorized network access and attacks.
Web Filtering (WF): Restricting or allowing web content access based on predefined criteria.
Antispam (AS): Technology filtering and blocking unwanted email (spam).
Rating Database: A Repository of categorized information used for assessing the security or trustworthiness of entities, often in the context of web content, files, or outbreaks.
FDS: Fortinet Distribution Server.
FDN: Fortinet Distribution Network.

Configuration on FortiManager:

Step 1: Enable service access for web filtering on FortiManager.

After enabling service access for FortiGate Updates and Web Filtering on the FortiManager interface, there is an option to bind to an IP Address (System settings -> Network -> port1).

If the Web Filtering option is disabled, FortiManager will reset the communication, and the packages will not be deployed to FortiGates.

If 'Bind to IP Address' is 0.0.0.0/0.0.0.0 (default value), the interface IP will be used (10.5.49.190 in the screen capture above). That configuration option has been chosen for this article.

Important note:

If the IP address is set in the 'Bind to IP Address' configuration, FortiManager will utilize TCP port 443 (Not used in this article). If it is desired to configure web-filter by using a TCP scenario (more secure), follow steps 5 to 7 in Technical Tip: FortiGate configuration for using FortiManager as local FDS.

The FortiManager interfaces may have three assigned IP addresses:

The primary IP is configured for TCP/8890 with AV/IPS and UDP/8888/53 with WF/AS.
The secondary IP, designated for update services (AV/PS), operates on TCP/443.
The tertiary IP, designated for rating services (Web-Filter/AntiSpam), also operates on TCP/443.

It is essential to clarify that assigning secondary or tertiary IPs does not disrupt the primary IP's ability to listen on UDP ports. All IPs can operate simultaneously on their designated ports, enabling different FortiGates to access services on distinct ports without any interference.

Step 2: Connect FortiManager to the FortiGuard Distribution Network (FDN) for local Web-filter server use.

Via GUI: FortiGuard -> Settings -> Enable Web Filter Service.

Currently, the Web-Filter Database is empty, as FortiManager has not yet pulled the signature packages from guard.fortinet.net.

Via the CLI:

config fmupdate service
    set query-webfilter enable
end

The initial setup of the WF/AS rating databases may take several hours, possibly even exceeding a day, primarily because of the substantial downloads of delta packages for the database, which amount to more than 20GB for WF. Subsequently, the process involves merging the databases.

FortiManager has obtained the latest full web filter database when the Database version appears.

The first server contacted by FortiManager is https://guard.fortinet.net/. Upon a successful connection, it sends FortiManager a list of all available FortiGuard rating servers which FortiManager can download/update the rating databases:

Note that the CPU and I/O utilization may increase significantly during the database download and initial setup. This is expected and usually caused by the fdglinkd process downloading and writing packages to the disk, and the fgdupd process working to consolidate the updates.

FortiManager is configured to poll FDN every 10 minutes by default. A forced update can be made using the following CLI command:

diagnose fmupdate updatenow fgd

Step 3: Refine Web-Filter Configuration on FortiManager.

Execute the command 'config fmupdate web-spam fgd-setting' to Fine-Tune the Web-Filter Configuration on FortiManager.

get fmupdate web-spam fgd-setting
as-cache : 300
as-log : nospam
as-preload : disable
av-cache : 300
av-log : novirus
av-preload : disable
av2-cache : 800
av2-log : noav2
av2-preload : disable
eventlog-query : disable
fgd-pull-interval : 10
fq-cache : 300
fq-log : nofilequery
fq-preload : disable
iot-cache : 300
iot-log : nofilequery
iot-preload : disable
iotv-preload : disable
linkd-log : debug
max-client-worker : 0
max-log-quota : 6144
max-unrated-site : 500
restrict-as1-dbver : (null)
restrict-as2-dbver : (null)
restrict-as4-dbver : (null)
restrict-av-dbver : (null)
restrict-av2-dbver : (null)
restrict-fq-dbver : (null)
restrict-iots-dbver : (null)
restrict-wf-dbver : (null)
server-override:
stat-log : disable
stat-log-interval : 60
stat-sync-interval : 60
update-interval : 6
update-log : enable
wf-cache : 600
wf-dn-cache-expire-time: 30
wf-dn-cache-max-number: 10000
wf-log : nourl
wf-preload : disable

These are the recommended, optimized, and best-practice settings to ensure optimal performance and behavior for web-filtering services:

update-interval = 2
wf-cache = 0
wf-log = disable
wf-preload = enable

Memory sizing requirement:

When rating database preload is enabled, memory sizing should be calculated using the following formula:

Total required RAM = FortiManager base RAM requirement
                   + (2 x total size of all enabled rating databases configured for preload)

The rating database sizes can increase over time. Therefore, memory requirements must be recalculated based on the current database sizes, the enabled rating services, the FortiManager platform sizing, and the other services running on the appliance or VM.

The following are the most important commands to fine-tune Web-Filter Configuration on FortiManager:

fgd-pull-interval <integer>: Configure how frequently to poll the FDN servers (or override configured ones) to obtain AS/WF/FQ/GeoIP delta package updates and verify license information.

FortiGuard pull interval in minutes (1 - 1440, default = 1).

update-interval <integer>: Defines the number of hours to wait before initiating the Database merging operation if an insufficient number of delta files are received.

The default value is to wait for 6 hours for the Database to merge if insufficient delta files are received. Otherwise, merging occurs after receiving 100 delta files.
This duration can be minimized to a minimum of 2 hours, and the shorter it is, the greater the accuracy and synchronization between FortiManager's web filtering database and the FortiGuard database.

update-log {enable|disable}

enable – Enable detailed fgdupd merge process information to internal log files. This is okay for production.
disable – Disable detailed fgdupd merge process information to internal log files.

wf-log {all|disable|nourl}

all: All enable detailed WF query (and GeoIP) logging (use only for troubleshooting and disable in production). Setting it to 'all' is not recommended as it will significantly impact the performances.
disable: Disabling the web-filter logging.
nourl: Log non-URL events.

wf-dn-cache-max-number <integer>

This is the maximum number of Web filter DN (domain names) cache (0 = disable, default = 10000). Added in FortiManager v5.6.1 and used for FortiGate v5.6 and later for new Web filter categories 90 & 91.

wf-dn-cache-expire-time <integer>

This is the web filter DN cache expire time, in minutes (1 - 1440, 0 = never, default = 30). Added in FortiManager v5.6.1 and used for FortiGate v5.6 and later for new WF categories 90 & 91.

wf-cache <integer>

Web filter service maximum memory usage, in megabytes (maximum = physical memory-1024, 0 = no limit, default = 600).

Setting wf-cache to 0 allows dynamic allocation of memory up to the web-filter database size. Ensure sufficient physical memory on the hardware or VM platform to prevent web-filter process issues, such as crashes or unresponsiveness.
The web-filter database is cached in the rundb directory: 'sh-4.3# cd /var/fgd/URLs/rundb/'.
In the event of a brief disconnection from the FDS network, the local FDS server will persist in delivering local FGD service, although it may not have the latest database.

wf-preload {enable|disable}

disable: Default on all platforms, except hardware models with more than 64 GB RAM.

The web-filter database is operating on the disk. The increased disk I/O utilization in this scenario is expected to cause higher CPU utilization, as well as slower rating response if many devices are sending requests at the same time. However, this allows units with RAM less than double the total size of the rating databases to operate as web-filter rating servers.

enable: Uploads the whole web-filter database to memory.

This ensures optimal performance and reliability of the Web Filter rating server but requires a memory capacity of at least twice the combined size of all enabled rating databases (in addition to the required memory for FortiManager and other modules).

Example:

At the time of writing, the Web Filter database is approximately 11 GB and the Antispam database is approximately 1 GB, for a combined rating database size of approximately 12 GB.

When both databases are preloaded, approximately 24 GB of additional RAM is required for the rating databases:

2 x 12 GB = 24 GB

If the applicable FortiManager base memory requirement is 8 GB, the resulting illustrative minimum is approximately 32 GB:

8 GB + 24 GB = 32 GB

Note: The 24 GB and 32 GB values are examples based on the database sizes and base FortiManager requirement used at the time of writing. They are not fixed minimum requirements for every deployment. Database growth, additional rating databases such as File Query or Outbreak Prevention, managed-device scale, ADOM usage, and other FortiManager services require additional memory and must be included in the sizing calculation.

Configuration on FortiGate.

Step 1: Configure the FortiGate to use FortiManager as a local server for both AV/IPS updates and WF/AS rating:

config system central-management
    config server-list
        edit 1
            set server-type update rating
            set addr-type ipv4
            set server-address 10.5.49.190
        next
    end
    set fmg-update-port 8890
end

Step 2: Configure the Web Filter rating port on FortiGate.

In this example, FortiGate needs to set the update port to UDP 8888 FortiGuard port (default is TCP/443).

config system fortiguard
    set fortiguard-anycast disable
    set protocol udp
    set port 8888
end

Validation of Functionality: Confirming Operational Status.

Step 1: Verification of FortiGuard connectivity from FortiManager.

Run the following command on FortiManager to verify connectivity with the FortiGuard rating servers:

diagnose fmupdate view-linkd-log fgd

The status should display as 'SUCCESS', as illustrated in the example below:

Note: This command shows only the last 10 lines of the log. If the status messages are not visible, with the log running wait for the next connection (every 10 minutes by default).

Step 2: Confirm the Current Status of Device License (FortiManager/FortiGate).

Once done, check FortiGuard -> Device Licenses. Ensure the FortiGate for web filtering has a valid FortiGuard Subscription and 'up to date' Service Status. This confirms the successful communication of FortiGuard license contract information to FortiManager.

The same information can be checked with the following commands:

In FortiManager:

diagnose fmupdate dbcontract <serial_number>

FortiGate:

diagnose test update info
diagnose debug rating

Ensure that the web-filter service is enabled, the license is in 'Contract' status, and FortiManager IP is at the top of the table with flags as in the screenshot below:

If the license is valid in FortiGate but not in FortiManager, this may indicate that FortiManager is failing to reach FortiGuard servers.

Step 3: Confirm that FortiGuard connectivity is working (FortiGate).

Update debug commands can also be run on FortiGate to verify the connecting IP port number and the updated packages.

Use the below CLI commands on FortiGate:

diagnose debug reset
diagnose debug app update -1
diagnose debug enable
execute update-now

diagnose debug disable
diagnose debug reset

upd_comm_connect_fds[458]-Trying FMG 10.5.49.190:8890
upd_install_pkg[1306]-MADB002 is up-to-date
upd_install_pkg[1306]-AFDB0013 is up-to-date
upd_status_save_status[130]-try to save on status file
upd_status_save_status[196]-Wrote status file
__upd_act_update[325]-Package installed successfully
upd_comm_disconnect_fds[499]-Disconnecting FMG 10.5.49.190:8890

Troubleshooting.

Step 1: Troubleshoot FortiGuard connectivity (FortiManager).

The following is an example of a failed connection:

Use the following CLI command:

diagnose fmupdate view-linkd-log fgd
30|LastSuccessTime=10:57:57|Status=0|UpullStat=Syncing|UpullErr=Connect error|UpullServer=65.x.x.x…………
2021/04/30_08:09:22.324 error fgdlinkd[780]: DES=LOGTP_COMM:REQ_uppull_objects: Failed to connect to fds server: 65.x.x.x:443
2021/04/30_08:09:22.324 debug fgdlinkd[780]: DES=LOGTP_COMM:REQ_uppull_objects:result is: -3, respode = 0
2021/04/30_08:09:22.324 error fgdlinkd[780]: ERR=900|DES=LOGTP_MEM:pollcmd_sendto_fds_server: Failed to get info from 65.x.x.x:443

The above error indicates that the connectivity between FortiManager and FortiGuard servers is not functional. It might be due to a misconfiguration of FortiGuard on FortiManager or a firewall behind FortiManager that may block the traffic on TCP port 443.

Step 2: Troubleshoot FortiGuard connectivity (FortiGate).

Update debug commands can also be run on FortiGate to verify the connecting IP port number and the updated packages:

Use the following CLI commands on FortiGate:

diagnose debug reset
diagnose debug app update -1
diagnose debug enable
execute update-now

diagnose debug disable
diagnose debug reset

upd_daemon[1790]-Received update now request
upd_daemon[1519]-Found cached action=00000002
do_update[493]-Starting now UPDATE (final try)
upd_comm_connect_fds[458]-Trying FDS 10.5.49.190:8890
tcp_connect_fds[259]-select() timed out
upd_comm_connect_fds[472]-Failed TCP connect
do_update[505]-UPDATE failed

The error message above suggests that traffic from the downstream FortiGate, FortiGuard may be blocked by an upstream firewall.

A packet capture can also be used to verify connectivity on TCP port 8890:

diagnose sniffer packet any 'port 8890' 4 0 l

Step 3: Get the details on the downloaded web-filter package version.

To gather details about the downloaded Web Filter package version, use the following FortiManager command. Direct shell access to the log file is unavailable from FortiManager v7.6.0, v7.4.4, and v7.2.6 onward.

diagnose fmupdate view-linkd-log fgd
2023/12/14_11:24:23.041 info fgdlinkd[6164]: FCP_CONN:: received object[1/4]: id=00000000FCPR00000 ver=00000.00000-2312141024 size=200
2023/12/14_11:24:23.042 info fgdlinkd[6164]: [FDS-->FMG] Response: Protocol=4.0|Response=200|Firmware=FGD033-FW-6.8-0193|SerialNumber=FGD-FGS-
DELL0903|Server=FGDG|Persistent=false|ResponseItem=04000000FURL00000:200*00000000FCNI00000:200*00000000FDNI00000:200^M ^M
2023/12/14_11:24:23.042 info fgdlinkd[6164]: __uppull_record_event_history: svc=fgd, event=256
2023/12/14_11:24:23.042 info fgdlinkd[6164]: FCP_CONN:: receiving objhdr: objid=04000000FURL00000
2023/12/14_11:24:23.144 info fgdlinkd[6164]: __uppull_record_event_history: svc=fgd, event=512
2023/12/14_11:24:23.144 debug fgdlinkd[6164]: __obj_move_to_running_path,562:svc(fgd) package export to /var/fgd/URLs/FortiGuard_00232_52048.md5

Related articles:

Fortimanager

Sign up

Login with SSO

Login to the community

Login with SSO