Description
This article describes that for the Static URL Filter to work properly when it is activated on the Web Filter profile, it must be defined with the correct type of entry. This article describes the three options that can be chosen, how they operate, and examples of their usage.
Scope
FortiGate (relevant as of FortiOS v7.2).
Solution
Simple: A simple URL Filter entry must be in the format of a standard URL, and they can include sub-domains and paths.
- Examples include: 'fortinet.com', 'fortinet.com/support', 'support.fortinet.com', 'net.com', etc.
- Note that to check/match a URL's path (e.g. 'www.fortinet.com/products/next-generation-firewall'), SSL Deep Inspection must be configured within the Firewall Policy (assuming the traffic is encrypted HTTPS).
- The HTTP Path is in the body of the HTTP request, and HTTPS encrypts that information away from the FortiGate's view (unless Deep Inspection is utilized).
- The FortiGate can still filter based upon the Domain name without needing SSL Deep Inspection, as this name is present in the TLS Certificate used by the HTTPS web server. Unencrypted HTTP traffic also does not require SSL Deep Inspection on the FortiGate.
The behavior of the Simple entry will be different depending on if flow-based inspection or proxy-based inspection is in use. See Technical Tip: Changing the inspection mode of the firewall for how to change or verify the inspection mode being used.
In flow-based inspection, a Simple entry will match a given domain if the expression is a sub-string of the actual domain.
In flow-based, 'net.com' in a Simple entry will match both 'net.com' and 'fortinet.com'.
This also applies to matching within URL paths as well (for example, 'net.com/prod' will match 'www.fortinet.com/products/next-generation-firewall').
In proxy-based inspection, Simple entries will only match a visited domain if it is a subdomain or a literal/exact match.
In proxy-based, 'net.com' in a Simple entry will match 'net.com' and 'www.net.com' but not 'fortinet.com'.
The proxy-based matching for path (the portion after the first /) is by sub-string, similar to flow-based. For example, 'net.com/prod' will match 'www.net.com/products/next-generation-firewall' but not 'www.fortinet.com/products/next-generation-
firewall'.
Note:
FortiOS version 7.6.3 added the option to control subdomain match when using a simple entry.
config webfilter urlfilter
edit <id>
set include-subdomains {enable/disable}
next
end
When include-subdomains is enabled (as is the default), the urlfilter matches subdomains (e.g., '
net.com' matches '
http://www.
net.com' and 'help.
net.com').
When disabled, it only matches the exact domain ('
net.com').
Wildcard: A Wildcard entry can be used to match multiple URLs flexibly-based on the specified keyword/string. Notably, Wildcard entries do not need to follow a specific structure/format.
A few notes regarding Wildcard behavior:
- Wildcard entries assess a given URL from left to right, and they will be considered matched as long as there is at least one match found.
Example: the entry 'for*guard' will match URLs like 'fortiguard.com'.
- A Wildcard star character (*) is needed at the start of the entry if the string is expected to be in the middle of the URL.
Example: The entry '*or*guard' will match 'fortiguard.com', but 'or*guard' (no leading wildcard star) will not since fortiguard.com starts with the letter 'f'.
- Since the string matching is performed from left to right, a wildcard star is not necessary at the end of the entry. '*fortinet' is equivalent to '*fortinet*'.
- In FortiOS v6.0.5 and later, entries beginning in '*.' will also match any subdomains as well as the base domain. For example, '*.fortinet.com' will match sub-domains of the URL 'support.fortinet.com', 'www.fortinet.com', and will also match 'fortinet.com'.
Regular Expressions (regex): Regex is used to include one or more URLs related -or not related- to a pattern using some Perl syntax, for example:
- '*' symbol means: match 0 or more times of the character before the symbol, but no match with any character.
For example: 'fortinet*.com' will match 'fortinetttttttt.com' but not 'fortinetsupport.com'.
- '/i' symbols means: makes the pattern case sensitive. By default, urlfilter matching is not case sensitive.
For example: '/FORTINET/i' will not match with 'fortinet'.
- '^' symbol means: at the beginning of the string.
For example, '^fo' will match 'fortinet.com' but not 'support.fortinet.com' or 'notfortinet.com'.
- '.' symbol means: match any single character (can be combined with '*' to match any number of any characters).
For example: 'fortinet.com' will match 'fortinetacom', 'fortinetbcom', 'fortinetzcom'.
- '\' symbol means: escape a special character (like '.' or '*') so that it acts as a literal character instead.
For example, 'fortinet\.com' will now match 'fortinet.com' but not 'fortinetacom', or 'fortinetbcom'.
Combining the above options: '^forti.*\.com' can match 'fortiguard.com' and 'fortinet.com', but not 'support.fortinet.com' (not starting with 'forti'), 'fortinet.net' (wrong top-level-domain, or 'fortizcom' ('.' is checked as a literal character and does not match 'z').
Known Issue:
Proxy-based regex urlfilter has a known issue 1093624 beginning in FortiOS v7.4.5 where '\.' and '\/' are not parsed correctly. This is fixed in v7.6.1 and above and scheduled for fix in v7.4.8.
Further reference:
See the third-party documentation regarding Perl Regular Expressions (regex).
The third-party regex 101 site is useful for testing regular expressions.
