Technical Tip: Using a static URL filter feature to allow/block web sites

[mzainuddinahm]

Description

This article explains how to exempt or block access to a website using the URL filter feature.

Scope

FortiGate.

Solution

There are three types of URLs that can be defined.

1. Simple: A simple URL filter entry could be a regular URL.

For example: www.fortinet.com.

• URL: fortinet.com.
• URL: fortinet.com/support.

2. Wildcard: A wildcard can be used to include one or more URLs to a simple URL
   For example:

• URL: *.fortinet.com (everything before ".fortinet.com" will match this rule, like support.fortinet.com).
• URL: www.fortinet.com/* (everything after "www.fortinet.com/" will match this rule, like www.fortinet.com/contact).

Putting 'space' after '*' on an expression will result in a wildcard match of all URLs.

Example:

 * yahoomail.com      <----- There is a space between * and yahoomail.

3. Regular Expressions (regex): Regex is used to include one or more URLs related -or not related- to a pattern using some Perl syntax.

For example:- the '*' symbol means: match 0 or more times of the character before the symbol, but no match with any character.

For example:'fortinet*.com' will match 'fortinetttttttt.com' but not 'fortinetsupport.com'.
'/i' symbols means: makes the pattern case sensitive.

For example, '/FORTINET/i' will not match with 'fortinet'.
'^' symbols means: at the beginning of the string.

For example:'^fo' will match 'fortinet.com'
'.' symbol means: match the same or different character than the one before the symbol, but is followed by the rest of the sentence.

For example: 'fortinet.com' will match 'fortinetacom', 'fortinetbcom', 'fortinetzcom'.

Configuring a URL filter:

GUI:

1. Go to Security Profiles -> Web Filter.
   
2. Select a web filter to edit.
   
3. Under Static URL Filter, enable URL Filter, and select Create New.
   
4. Enter the URL, without the “http”, for example: www.example*.com
5.  Select a Type: Simple, Regular Expression, or Wildcard. In this example, select Wildcard.
   
6. Select the Action to take against matching URLs: Exempt, Block, Allow, or Monitor.
7. Select 'Enable'.
8. Select 'OK'.
   
   

CLI:

The syntax in the CLI for configuring an entry is:

config webfilter urlfilter

    edit <ID>
        config entries
            edit 1
                set url <url>
                set referrer-host <url>
                set type {simple | regex | wildcard}
                set action {block | allow | monitor | exempt}
                set status {enable | disable}
         end
end

end

To attach the URL filter table to an existing or a new webfilter profile:

config webfilter profile
    edit "webfilter"               <----- Name of the web filter profile.
        config web
            set urlfilter-table 1  <----- Where x is the URL filter table ID, this number can be found in '3 config webfilter urlfilter' the URL filter created with an ID number.
              end
        config ftgd-wf
            unset options
        end
    next
end

If the exemption is only needed from FortiGuard filtering then 'set exempt fortiguard' can be used, instead of all.

For all exempt actions: ? is used to show all the available options:

set exempt
av --> Antivirus filtering.
web-content --> Web filter content matching.
activex-java-cookie --> ActiveX, Java, and cookie filtering.
dlp --> DLP scanning.
fortiguard --> FortiGuard web filtering.
range-block --> Exempt range block feature.
pass --> Pass single connection from all.
all --> Exempt from all.

Note:

Sometimes it is required to clear the session of the source IP for the Static URL to work.
Some sites will be using multiple sub-domains which fall under different FortiGuard categories so it will be required to exempt all sub-domains as well to access the site.

To check the sub-domains used by a particular site, check browser developer tools.
In Chrome -> Ctrl+Shift+I -> Sources: Here it is possible to check all the sub-domain details.
SSL/SSH deep/full inspection is mandatory for static URL filter working.