Technical Tip: Using pattern matching for URLs

[AlexC-FTNT]

Description

This article describes the correct usage of 'URL Path Regex' and 'Host Regex Pattern' for Explicit Proxy.

Scope

FortiGate.

Solution

Consider an example where the following sites must be matched. 

1. www.fortiexample.test.com/test/example
2. www.forti.example.test.com/directory/example
3. test.example.fortinet.com/test
4. test.example.com

Note that the word 'example' is purposely placed both in the domain name and in the directory path.

There are two possible ways to match them, depending on situational requirements:

• By hostname, using a 'Host Regex Pattern' object.
• By URL Path, using 'URL Path Regex'.

Using the 'Host Regex Pattern', FortiOS can handle the match with .*example.*, or example.* because the host part of all 4 of the URLs contains the word 'example' before the path (delimited by the first '/' ). To set this up, the destination address is enough.

However:

• To match only 2, 3, and 4, use: \.example\.* (this must include the . both before and after example).
• To match only 4, use \.example.com*.

When the address typed is 'URL Path Regex', there is no match for the words used in the path.

To match the path part of the URL (/test/example), 'URL Path Regex' needs to be used.

For example:

• To match 1,2 based on the URL Path, first define Host as 'test.com' (or 'all') then use this as URL Path Regex: \/example.*.
• To match only 3, first define Host as 'fortinet.com' (or 'all') then use this as URL Path Regex: \/test$The \ before the filter specifies that the next character is a special one and must also be matched as one.

The filters may differ according to needs, but having this object defined and used in a policy is not enough.

The unit needs to:

1. Connect first to that host: which needs to allow the CONNECT method.
2. Have visibility to the URL path: which needs deep-inspection to be enabled.

When using URL Path Regex, the configuration should look like this for an explicit proxy rule:

config firewall proxy-address
    edit "addr-connect"
        set type method
        set host "all"
        set method connect
    next

    edit "example_path"
        set host "all"
        set path ".*example.*"
    next
end

config firewall policy

edit 1

set type explicit-web

set dstintf "any"
set srcaddr "addr-connect" <- For passing CONNECT first.
set dstaddr "all"
set action accept
set schedule "always"
set service "webproxy"
set explicit-web-proxy "web-proxy"
set logtraffic all
set ssl-ssh-profile "custom-deep-inspection" <- Necessary.

next

edit 2

set type explicit-web

set dstintf "any"
set srcaddr "all"
set dstaddr "example_path" <----- Match for 'example' in the URL path.
set action accept
set schedule "always"
set service "webproxy"
set explicit-web-proxy "web-proxy"
set utm-status enable
set logtraffic all
set webcache enable
set ssl-ssh-profile "custom-deep-inspection"  <----- Necessary.
set av-profile "default"

next

For FortiGate, an additional firewall policy is needed for this:

config firewall policy

    edit 1

        set srcintf "any"

        set dstintf "any"

        set srcaddr "all"
        set dstaddr "example_path"        

        set action accept
        set schedule "always"
        set service "ALL"
        set ssl-ssh-profile "custom-deep-inspection" <----- Necessary.

    next

end

For transparent proxy traffic, one policy with set dstaddr 'example_path' is sufficient to match both 'example' in the hostname and URL path.

Make the following note when creating a URL pattern otherwise, it will not block or allow the URL:

• All URLs must be case-insensitive.
• If pattern /xxxx/yy will escape as \/xxx/yy in case YY is invalid or unsupported.
• Handling Wildcard URL with '/'  is the same as regex to match the host and path separately.
• Perl-style regex with invalid or empty flags will be handled as normal regex.