Technical Tip: Understanding the behaviour of the Block Invalid URL's option

[Jonathan_Body_FTNT]

Description

This article describes how the FortiGate behaves when setting the Block Invalid URL option from the Web Filtering UTM profile.

Scope

FortiGate.

Solution

When configuring the 'block-invalid-url' within the 'config webfilter profile', it is important to understand the behavior of the FortiGate once this option is active.

In CLI:

config webfilter profile
    edit "TEST"
        set options block-invalid-url  --------> Enabled from CLI.

Enable to blocking of websites whose SSL certificates' CN field does not contain a valid domain name.

FortiGate units always validate the CN field, regardless of whether this option is enabled. However, if this option is disabled, although validation failure does not cause the FortiGate unit to block the request, it changes the behavior of FortiGuard Web Filtering.

1. If the request is made directly to the web server, rather than a web server proxy, the FortiGate unit queries for FortiGuard Web Filtering category or class ratings using the IP address only, not the domain name.
   
   
2. If the request is to a web server proxy, the real IP address of the web server is not known, and so rating queries by either or both the IP address and the domain name is not reliable. In this case, the FortiGate unit does not perform FortiGuard Web Filtering.

When a visited URL contains a '_', the site will be blocked with 'block-invalid-url'.
As per RFC 952, 'A "name" (Net, Host, Gateway, or Domain name) is a text string up to 24 characters drawn from the alphabet (A-Z), digits (0-9), minus sign (-), and period (.).'

The invalid URL can be exempted by the URL filter in the Web Filtering profile when the firewall policy is in proxy-based inspection mode, and SSL inspection and deep inspection are enabled in the policy. The exemption will not work in the flow-based inspection mode policy.  

Webfilter log for HTTP:

date=2025-04-09 time=09:31:22 eventtime=1617985882321875966 tz="-0700" logid="0315012547" type="utm" subtype="webfilter" eventtype="urlfilter" level="notice" vd="root" proto=6 policyid=1 sessionid=17896 srcip=10.1.10.2 srcport=55210 srcintf="port2" srcintfrole="undefined" dstip=17.142.16.9 dstport=80 dstintf="port1" dstintfrole="undefined" service="HTTP" profile="webfilter" hostname="http://www.thelongestdomainnameintheworldandthensomeandthensomemoreandmore1.com" action="blocked" reqtype="direct" msg="The HTTP request contained an invalid domain name." sentbyte=136 rcvdbyte=0 url="/" crscore=30 craction=8 crlevel="high"

Webfilter log for HTTPS:

2: date=2021-04-09 time=09:33:25 eventtime=1617986005833407313 tz="-0700" logid="0315012551" type="utm" subtype="webfilter" eventtype="urlfilter" level="notice" vd="root" proto=6 policyid=1 sessionid=17953 srcip=10.1.10.2 srcport=48608 srcintf="port2" srcintfrole="undefined" dstip=17.142.16.9 dstport=443 dstintf="port1" dstintfrole="undefined" service="HTTPS" profile="webfilter" hostname="http://www.thelongestdomainnameintheworldandthensomeandthensomemoreandmore1.com" action="blocked" reqtype="direct" msg="The Server Name Indication for the HTTPS session contained an invalid domain name." sentbyte=517 rcvdbyte=0 url="/" crscore=30 craction=8 crlevel="high"