Technical Tip: FortiGate URL filtering HTTP URL query canonicalization with Proxy mode

[bbae]

Description	This article describes an issue with the Static URL Filter that prevents it from functioning as expected. In some cases, URLs that are explicitly allowed in the Static URL Filter list are still being blocked.
Scope	FortiGate v7.2, FortiGate v7.4.
Solution	This is how the FortiGate firewall policy with proxy mode canonicalizes a URL. For HTTP URL query canonicalization, the following special character sets are defined: Allowed characters: '-._~:@/?'. Reserved characters: '!$&'()*+;,='. Conditions: If the character is not percent-encoded: Keep it not encoded if: Alphabet or number or, Character of allowed characters or, Character of reserved characters. Otherwise, convert it to percent-encode. If the character is percent-encoded: Decode it if: It represents an alphabet or a number. It represents the character of allowed characters. Otherwise, keep it as encoded. For example, static URL filtering is configured as follows. internal.fortinet.it/login?next=/builder/menu/46/474/list/?tab=1820  FortiGate processes the HTTP traffic internally as follows. GET /login?next=%2Fbuilder%2Fmenu%2F46%2F474%2Flist%2F%3Ftab%3D1820 HTTP/1.1 Host: internal.fortinet.it upgrade-insecure-requests: 1 user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36 accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng, */*;q=0.8,application/signed-exchange;v=b3;q=0.7 sec-fetch-site: none sec-fetch-mode: navigate sec-fetch-user: ?1 sec-fetch-dest: document sec-ch-ua: "Chromium";v="140", "Not=A?Brand";v="24", "Google Chrome";v="140" sec-ch-ua-mobile: ?0 sec-ch-ua-platform: "Windows" accept-encoding: gzip, deflate, br, zstd accept-language: ko-KR,ko;q=0.9,en-US;q=0.8,en;q=0.7 priority: u=0, i   [I][p:279][s:2784315730][r:60641728] wad_http_parse_host :1683 host=[20]internal.fortinet.it [I][p:279][s:2784315730][r:60641728] wad_http_str_canonicalize :2198 enc=0 path=/login len=6 changes=0 [I][p:279][s:2784315730][r:60641728] wad_http_str_canonicalize :2200 end=4 path=next=/builder/menu/46/474/list/?tab%3D1820 len=42 In this case, FortiGate treats '=' as a reserved character as follows while canonicalizing the percent-encoding URL. %3D represents '=', and they are in condition 2.2; therefore, they are kept encoded. This result can cause the Static URL Filter to not function as expected. To avoid this issue, it recommends fully percent-encoding it in the URL filter entry: config webfilter urlfilter     edit 1         set name "test-1207683"             config entries                 edit 1                     set url "internal.fortinet.it/login?next=%2Fbuilder%2Fmenu%2F46%2F474%2Flist%2F%3Ftab%3D1820"                     set action allow