Web Filter Security Profile does not apply or log consistently

[jdoyon]

Scenario:

One "inside" server talks to another "outside" server over HTTPS. I want to monitor and log all traffic, with as much detail as possible.

I set up firewall policies to allow the traffic (from certain hosts to other certain hosts, on HTTP(S)),works great, traffic flows. I assign a web filter profile setup to "monitor" all categories, including unrated. This should have the effect of creating Web Filter Security Event Log entries for all URLs flowing through a given policy, since I monitor everything and monitoring logs, right? They all have the "certificate-inspection" profile assigned as well.

Except it doesn't.

When looking at the Forward traffic log, and the details on the right, some entries have the "Security" entry with web filtering details, while others do not. When filtering on a given firewall policy and selecting different log entries, the security tab appears and disappears, seemingly randomly, entry to entry.

Both the policy and security profile are flow-based. The only other security profile applied is the "certificate-inspection" one. So no SSL deep-inspection. I've read in a virous places however that that should be OK?

This is a FortiGate-VM 7.2.0. Lightly loaded, lots of CPU to spare, and RAM is at about 51% right now. All licensing, including FortiGuard, is current.

Any thoughts on what might be going on??

Thanks in advance.

[Anonymous · Accepted solution]

Hello @jdoyon,

                   Thank you for reaching Fortinet Community. I would recommend you to perform the following:

1) Inspection mode: Ideally web filter best works in proxy mode as most of them are HTTPS traffic and the man-in-middle does the inspection in a better way. Also flow based web-filter has limited features. More info in below document:

https://docs.fortinet.com/document/fortigate/6.2.0/cookbook/469806/inspection-mode-differences-for-w...

2) Logging: Could you verify if the logging is set to 'All sessions' and also in the respective web profile if the logging is set to 'ALL'. More info in below KB article:

https://community.fortinet.com/t5/FortiGate/Technical-Tip-Log-all-user-traffic-URLs-using-web-filter...

Hope this helps.

Thanks and regards,

[jdoyon]

Indeed, once I switched the relevant firewall policy to proxy, as well as the security profile, things started working better.

Also took me a while to realize that the URL filter only works on hostnames because with HTTPS, it only inspects SNI, instead of the whole URL.

Once I change my URL filter to be a hostname only, that also helped.

The GUI improvements in 7.2.1 also helped. Helped me realize that a session can have multiple log entries in the forward log, and that the web filter decision only appears on the last one. In flow mode this is a problem because of long-lived "keep-alive" TCP sessions. The URL would go through unfiltered in flow mode, because the session only closes much later.

Thanks for the help!