Technical Tip: Using a static URL filter feature to allow/block web sites

[mzainuddinahm]

Description

This article describes how to exempt or block access to a website using the URL filter feature.

Scope

FortiGate.

Solution

Three types of URLs can be defined.

1. Simple: A simple URL filter entry could be a regular URL.

For example: www.fortinet.com.

• URL: fortinet.com.
• URL: fortinet.com/support.

2. Wildcard: A wildcard can be used to include one or more URLs to a simple URL.
   For example:

• URL: *.fortinet.com (everything before '.fortinet.com' will match this rule, like support.fortinet.com).
• URL: www.fortinet.com/* (everything after 'www.fortinet.com/' will match this rule, like www.fortinet.com/contact).

Putting 'space' after '*' on an expression will result in a wildcard match of all URLs.

Example:

 * yahoomail.com      <-- There is a space between * and yahoomail.

3. Regular Expressions (regex): Regex is used to include one or more URLs related -or not related- to a pattern using some Perl syntax.

For example:- the '*' symbol means: match 0 or more times of the character before the symbol, but no match with any character.

For example:'fortinet*.com' will match 'fortinetttttttt.com' but not 'fortinetsupport.com'.
'/i' symbols means: makes the pattern case sensitive.

For example, '/FORTINET/i' will not match with 'fortinet'.
'^' symbols means: at the beginning of the string.

For example:'^fo' will match 'fortinet.com'
'.' symbol means: match the same or different character than the one before the symbol, but is followed by the rest of the sentence.

For example: 'fortinet.com' will match 'fortinetacom', 'fortinetbcom', 'fortinetzcom'.

Configuring a URL filter:

GUI:

1. Go to Security Profiles -> Web Filter.
   
2. Select a web filter to edit.
   
3. Under Static URL Filter, enable URL Filter, and select Create New.
   
4. Enter the URL, without the 'http', for example www.example*.com
5.  Select a Type: Simple, Regular Expression, or Wildcard. In this example, select Wildcard.
   
6. Select the Action to take against matching URLs: Exempt, Block, Allow, or Monitor.
7. Select 'Enable'.
8. Select 'OK'.
   
   

CLI:

The syntax in the CLI for configuring an entry is:

config webfilter urlfilter

    edit <ID>
        config entries
            edit 1
                set url <url>
                set referrer-host <url>
                set type {simple | regex | wildcard}
                set action {block | allow | monitor | exempt}
                set status {enable | disable}
         end
    end

end

Note:

While adding UTM profiles in a Firewall policy If only web filter is configured, SSL Certificate Inspection can be used. If other UTM profiles are used (Ex: Application Control , IPS) in case Deep Inspection will be required.

To attach the URL filter table to an existing or a new web filter profile:

config webfilter profile
    edit "webfilter"               <- Name of the web filter profile.
        config web
            set urlfilter-table 1  <- Where x is the URL filter table ID, this number can be found in '3 config webfilter urlfilter' the URL filter created with an ID number.
              end
        config ftgd-wf
            unset options
        end
    next
end

If the exemption is only needed from FortiGuard filtering, then 'set exempt fortiguard' can be used, instead of all.  These options can only be configured on the CLI.

config webfilter urlfilter

    edit <ID>
        config entries
            edit <ID>
                set exempt {option1}, {option2}, ...
         end
    end

end

For all exempt actions: ? is used to show all the available options:

set exempt
av                     AntiVirus scanning.
web-content            Web filter content matching.
activex-java-cookie    ActiveX, Java, and cookie filtering.
dlp                    DLP scanning.
fortiguard             FortiGuard web filtering.
range-block            Range block feature.
pass                   Pass single connection from all.
antiphish              AntiPhish credential checking.
all                    Exempt from all security profiles.

Note:

Sometimes, it is necessary to clear the session of the source IP for the Static URL to work.
Some sites will be using multiple sub-domains that fall under different FortiGuard categories, so it will be necessary to exempt all sub-domains as well to access the site.

To check the sub-domains used by a particular site, check browser developer tools.
In Chrome -> Ctrl+Shift+I -> Sources: Here it is possible to check all the sub-domain details.
SSL/SSH deep/full inspection is mandatory for static URL filter working.

If there are multiple entries in the 'Static URL Filter' list for the same URL address, the selection for which filter that applies is a top-down approach meaning that the first rule in the list will match first and no further rules from that 'URL Filter' list will match the same URL. The following is a scenario where this can cause a problem:

• Assume a task is to exempt a specific URL from inspection by FortiGuard Categories.
• The firewall admin by error has created 2 static URL filters where the top filter is an 'allow' rule while the bottom rule is the required 'exempt' rule:

Cli Example:

config webfilter urlfilter
    edit 1
        set name "Auto-webfilter-urlfilter_gd0chqh2j"
            config entries
                edit 2
                    set url "example.com"
                    set action allow
                next
                    edit 1
                        set url "example.com"
                    next
                end
            next
        end

• The result is that the website will match the Allow rule first and will NOT be exempted from FortiGuard inspection. If The FortiGuard Category for this URL is Block this configuration would lead to users not being able to access the URL. instead, it will be redirected to the FortiGuard Deny Page similar to the following message: