Description
This article describes the scenario where the FortiGuard Web Filtering option "Rate URLs by domain and IP address" is enabled.
In this situation, the rating response from a FortiGuard Distribution Server (FDS) for a particular URL might differ from its IP address. This is very common in scenarios of Virtual Hosting, where one IP address of one physical server will host multiple services and URLs.
Therefore, if the IP address rating belongs to a blocked category, access to the URL will be blocked regardless of the rating of the URL.
Summary
- How to check if a URL gets two different ratings, one for the IP address and one for the URL.
- How to look for category numbers.
- How to make a live verification of rating response:
How to check if a URL gets two different ratings, one for the IP address and one for the URL.
Use the following command to dump the FortiGuard WEB cache:
diagnose test application urlfilter 3 <-- Only works if WEB filtering cache is enabled.
Example of output:
diagnose test application urlfilter 3
Saving to file [/tmp/urcCache.txt]
Cache Contents:
-=-=-=-=-=-=-=-
Cache Mode: TTL
Cache DB Ver: 93.4437
Domain |IP DB Ver T URL
29000000|34000000 93.4437 E http://www.mytestrating.fr/
34000000|34000000 13.28635 E http://www.fortinet.com/
Notes:
- In the above example, the domain www.mytestrating.fr is in category (Hex) 29, while the IP address is in category (Hex) 34.
- The numbers above are given in hexadecimal, while in the FortiGate CLI configuration, the numbers are displayed in decimal. The www.fortinet.com domain will therefore be in category Hex34 = Dec52 (see example of category below, 52 = Information Technology).
How to look for category numbers:
The numbers displayed in the columns IP and domains are the rating returned by the FortiGuard servers. The corresponding categories can be retrieved as explained in the related article: FortiGuard Web Filtering Category and Classification numbers / FortiGate configuration and troublesh....
Example:
config firewall profile
edit TEST
(TEST) get
edit TEST
(TEST) get
ftgd-wf-enable :
g01 Potentially Liable:
1 Drug Abuse
2 Occult
3 Hacking
4 Illegal or Unethical
5 Racism and Hate
6 Violence
57 Marijuana
58 Folklore
[...]
g07 Business Oriented:
49 Business
50 Information and Computer Security
51 Government and Legal Organizations
52 Information Technology
53 Armed Forces
[...]
g01 Potentially Liable:
1 Drug Abuse
2 Occult
3 Hacking
4 Illegal or Unethical
5 Racism and Hate
6 Violence
57 Marijuana
58 Folklore
[...]
g07 Business Oriented:
49 Business
50 Information and Computer Security
51 Government and Legal Organizations
52 Information Technology
53 Armed Forces
[...]
How to make a live verification of rating response:
Use the following CLI commands to check the rating requests and responses:
diagnose debug enable
diagnose debug application urlfilter -1 <-- To stop it: diagnose debug application urlfilter 0.
diagnose debug application urlfilter -1 <-- To stop it: diagnose debug application urlfilter 0.
## id=93000 msg="pid=57 urlfilter_main-723 in main.c received pkt:count=91, a=/tmp/.thttp.socket/21" id=22009 msg="received a request /tmp/.thttp.socket, addr_len=21: d= ="www.goodorg.org:80, id=12853, vfid=0, type=0, client=192.168.3.90, url=/" id=99501 user="N/A" src=192.168.3.90 sport=1321 dst=<dest_ip> dport=80 service="http" cat=43 cat_desc=“Organisation" hostname="www.goodorg.org" url="/" status=blocked msg="URL belongs to a denied category in policy"
Tested on v7.0 and later:
Web Filter Configuration: The Job Search category is set to block for demonstration.
Upon accessing https://careers.floridadental.org/, the page is blocked. The category is Job Search.
Perform an NSlookup:
>nslookup careers.floridadental.org 8.8.8.8
Server: dns.google
Address: 8.8.8.8
Non-authoritative answer:
Name: cname.boxwoodtech.com
Address: 144.202.255.70
Aliases: careers.floridadental.org
Check the category on the FortiGate:
Security Profiles -> Web Rating Overrides -> Create New -> Enter URL -> Lookup rating.
diagnose webfilter fortiguard cache dump
Caution: This command is for diagnostic purposes ONLY. The bigger the cache size is set, the more impact on performance the command has.
Do you want to continue? (y/n)y
Saving to file [/tmp/urcCache.txt]
Cache Contents:
-=-=-=-=-=-=-=-
Cache Mode: TTL
Cache DB Ver: 233.50234
Rating DB Ver DOT SLASH ORIG_FLAG T URL
22000000|22000000 233.50234 0 0 00000001 P Ahttp://144.202.255.70/
21000000|21000000 233.50234 1 0 00000001 P Ahttp://careers.floridadental.org/
........
22 Hex is 34 in Decimal.
21 Hex is 33 in Decimal.
get webfilter categories | grep 34
34 Job Search
get webfilter categories | grep 33
33 Health and Wellness
Solution:
If the rating for an IP address blocks access to a site, the solution is to disable 'Rate URLs by domain and IP address'. Alternatively, the IP address can be overridden to a different category that is allowed.
Related documents:
Rate site by URL and IP address
Verify the webfilter cache content
FortiGuard Web Filtering Override Guide ; configuration examples
