Technical Tip: Using a static URL filter feature to allow/block web sites

[mzainuddinahm]

Description

This article describes how to exempt or block access to a website using the URL filter feature.

Scope

FortiGate.

Solution

As a primer, the following image shows a breakdown of the components of a URL:

When performing Static URL filtering in general, keep the following rules in mind:

• Domains/Sub-Domains and the Path can be used as part of the filter, and there is no need to specify the protocol section (for example, 'https://').
  
  
• When inspecting TLS-encrypted traffic (i.e., HTTPS), the FortiGate will have reduced scanning capabilities depending on if certificate-inspection is utilized vs. TLS/SSL deep-inspection:
  • With certificate-inspection, the FortiGate can scan the web-server's certificate and the TLS handshake process. This means that it can make web-filtering decisions based on the Server Name Indication (SNI) requested in the TLS Client Hello (TLS 1.2 and earlier), as well as the Common Name (CN) and Subject Alternative Names (SANs) present in the server certificate itself.
  • With deep-inspection, the FortiGate decrypt the HTTPS traffic and inspect the underlying HTTP payload, in addition to the options mentioned above. This means that deep-inspection grants access to inspect the HTTP Host field and the HTTP Path.
    
    
• Important: it is mentioned above, but using SSL certificate-inspection means that that it is not possible to make Static URL filtering decisions based on the HTTP Path.
  • This matters in cases where administrators want to allow/block access to specific parts of a website (for example, allow access to www.fortinet.com but disallow access to https://www.fortinet.com/solutions) and in these situations SSL deep-inspection is required (see also: FortiOS Best Practices - SSL/TLS deep inspection).
    
    
• Note: SSL certificate-inspection generally works well when inspecting HTTPS traffic with Web Filtering, IPS and Application Control (i.e. the majority of functionality/signatures work correctly). Adding deep-inspection expands functionality for the above inspection profiles, and in certain cases (such as Antivirus and DLP) it can become a necessity for HTTPS traffic inspection.

With the above in mind, there are three types of Static URL filter rules that can be defined:

1. Simple:

• A simple URL filter entry will match based on the standard URL structure, and it must be in a valid URL format.
• For example, specifying 'fortinet.com' can match 'www.fortinet.com', 'fortinet.com', and 'fortinet.com/support'
• Domains, sub-domains, and Paths can be utilized here.

To provide a better context, please refer to the scenarios below:

1. When adding a simple URL including only the domain name, such as 'fortinet.com', and blocking all other domains.

fortinet.com == simple // allow
*                 == wildcard // block

This currently would allow all the subdomains under fortinet.com, such as:

fortinet.com
www.fortinet.com --> subdomain 
support.fortinet.com --> subdomain 

And block all other websites due to the wildcard defined.

2. When adding a simple URL such as 'www.fortinet.com', and blocking all other domains.

www.fortinet.com == simple // allow
*                         == wildcard // block

This would allow only/strictly the below: 

www.fortinet.com

And block all other subdomains like support.fortinet.com, kb.fortinet.com, and, of course, other websites.

Starting next version 7.6.3 and future releases:

The same behavior above will be seen, but there will be a new command introduced via CLI to have more granular control.

The new command only applies to simple URL types.

Note: The default/current behavior explained above will not change.

config webfilter urlfilter
    edit 1
        set include-subdomains <enable/disable>
    end
end

• include-subdomains enable (default): The URL Filter matches subdomains (for example, 'fortinet.com' matches 'www.fortinet.com' and 'support.fortinet.com') 
• include-subdomains disable: The URL Filter only matches the exact domain (for example, 'fortinet.com' does not match 'support.fortinet.com').

2. Wildcard: A wildcard entry can be used to match based on a keyword/string, and it does not need to follow the standard URL structure. For example:

   • *.fortinet.com - everything before '.fortinet.com' will match this rule, like support.fortinet.com.
   • www.fortinet.com/* - everything after 'www.fortinet.com/' will match this rule, like www.fortinet.com/contact.
   • *fortinet - anything containing the word 'fortinet' will match this rule, like community.fortinet.com or fortinet.com.

3. Regular Expressions (regex): Regex can be used to match URLs based on patterns written in Perl syntax.

For example, the '*' symbol means 'match 0 or more times of the character before the symbol', so 'fortinet*.com' will match 'fortinetttttttt.com' but not 'fortinetsupport.com'. This is notably different behavior compared to the Wildcard rule type.

Note: For a more in-depth explanation of each Static URL filter type, how they work, and additional demonstration examples, refer to the following Community KB article on the subject: Technical Tip: URL Filter expressions for the FortiGate.

Configuring Static URL Filters (GUI method):

1. Go to Security Profiles -> Web Filter.
2. Select a Web Filter profile to edit.
3. Under Static URL Filter, enable the URL Filter toggle and select Create New.
4. Select the Type: Simple, Regular Expression, or Wildcard. The following example uses Wildcard.
5. The Type will dictate the accepted format of the URL section, so specifying it first can help avoid warnings/errors.
6. Enter the URL to be matched (for example, www.example*.com).
7. Select the Action to take against matching URLs: Exempt, Block, Allow, or Monitor.
8. Set the Status to Enable, then select OK to commit the new rule.
9. Select OK in the main Web Filter profile section to save the changes to the profile.

Configuring Static URL Filters (CLI method):

The syntax for configuring an entry in the CLI is as follows:

config webfilter urlfilter

    edit <ID>
        config entries
            edit 1
                set url <url>
                set referrer-host <url>
                set type {simple | regex | wildcard}
                set action {block | allow | monitor | exempt}
                set status {enable | disable}
         end
    end

end

Additional Notes:

• In the CLI, the Static URL filter table (config webfilter urlfilter) is a separate section from the main Web Filter profile itself (config webfilter profile). Static URL filter tables are linked within the Web Filter profile:

config webfilter profile

    edit "StaticFilterOnly"

        config web

            set urlfilter-table 1

            [...]

config webfilter urlfilter

    edit 1

        set name "Auto-webfilter-urlfilter_bro7efj76"
            config entries

                edit 3

                    set url "*fortinet.com"

                    set type wildcard

                next

                [...]

• By default, creating a Web Filter profile in the GUI will also create a corresponding Static URL filter table (when toggled on). However, it is possible to attach an existing Static URL filter table to a Web Filter profile by modifying the urlfilter-table config option under the config webfilter profile.

• When configuring Static URL filtering, it can be helpful to check the Web Filter logs to see what URLs are currently being blocked/identified. This can be checked under Log & Report -> Security Events -> Web Filter (aka Log & Report -> Web Filter in FortiOS v7.0 and earlier), though ensure that logging within the Firewall Policies is set to All sessions otherwise only Block/Monitor events will be logged.
  
  
• In the CLI, it is possible to configure additional options regarding the Exempt action. For example, if the website only needs to be exempted from FortiGuard Category Filtering, then set exempt fortiguard can be used instead of all:
  

config webfilter urlfilter

    edit <ID>
        config entries
            edit <ID>
                set exempt {option1}, {option2}, ...
         end
    end

end

• The following is a list of the exempt options available (use '?' in the CLI to display this list of options).

set exempt ?
av                     AntiVirus scanning.
web-content            Web filter content matching.
activex-java-cookie    ActiveX, Java, and cookie filtering.
dlp                    DLP scanning.
fortiguard             FortiGuard web filtering.
range-block            Range block feature.
pass                   Pass single connection from all.
antiphish              AntiPhish credential checking.
all                    Exempt from all security profiles.

• If changes are being made to the Static URL filter configuration during business hours, then it may be necessary to clear existing sessions on the FortiGate to enforce the change.

• Some sites will be using multiple sub-domains that are treated as different FortiGuard categories, and if those categories are blocked, then the website may not be fully accessible. In these cases, it may be necessary to exempt all of these sub-domains using Static URL filter rules to access the site successfully. To check all of the sub-domains and other resources used by a particular website, check the web browser's Developer Tools feature.
  
  
• For example, in Google Chrome, this is done by using the Ctrl+Shift+I keyboard shortcut or by opening the three-dot menu in the top-right and going to More Tools -> Developer Tools. After that, navigate to the Sources tab to see a list of sub-domains/network resources.

Note regarding Static URL rule assessment order:

Static URL filter rules are assessed in a top-down approach, which means that the first matching rule for a given URL on the list will be applied and no further rules will be matched. This means that if multiple entries in the Static URL filter list could match for a given URL address, the highest rule on the list will be utilized. The following is a scenario where this can cause a problem:

• Assume a task is to exempt a specific URL from inspection by FortiGuard Category Filtering (i.e. assume that the website's matched category is currently set to Block).
  
  
• During setup, the administrator has created two Static URL filter rules, where the top rule has the Allow action and the bottom rule has the Exempt action. The following is an example of what this would look like from the CLI:

config webfilter urlfilter
    edit 1
        set name "Auto-webfilter-urlfilter_gd0chqh2j"
            config entries
                edit 2
                    set url "example.com"
                    set action allow
                next
                edit 1
                        set url "example.com" <<< Default action is exempt, which is why it is not visible here.
                next
            end
end

• The result is that the website will match the Allow rule first and will not be exempted from FortiGuard Category Filtering. If the FortiGuard category for this URL is set with the Block action, then this configuration would lead to users not being able to access the URL. Instead, users will be redirected to the FortiGuard Deny Page and presented with a message similar to the following:

• If the rules are rearranged such that the Exempt rule is above the Allow rule (either by grabbing and moving the rules in the GUI or by using the move <id> [before|after] <id> command in the CLI), then users will instead be able to access the exempted website.

Note regarding Allow vs. Exempt Action:

• Allow: destination is allowed through the Static URL filter but is still subject to other inspections (such as FortiGuard Category Filtering within the Web Filter profile).
• Exempt: The destination is exempt from further inspection, and traffic is allowed.

Related articles:

• Technical Tip: URL Filter expressions for the FortiGate
  
• Technical Tip: Difference between action 'Allow' and 'Exempt' in static URL filter
  
  

Related Video: