Proxy-based vs. Flow-based Inspection Mode for Web Filter profiles?

Hi, The principle is similar to the AV options. In flow based mode: the FG examine the packets passing thru, spot the URL, check it and if blocked just send a RST to both sides (or at least client side), so drop the traffic without more explanation to the user. In proxy mode: The FG act as a transparent proxy. It intercept any SYN to port 80 and act as a proxy. This mean that you have 2 sessions: one between the user and the proxy and one between the proxy and the server. This allow the FG to generate its own error page, warning etc... But is more CPU intensive Regarding the alert, the default config send an alert for " unrated" websites, and the number of sites not in Fortiguard database is relatively high compared to Bluecoat or Websense for example. I currently have very low web traffic (as only a couple of users are filtered for the moment and I get 3-5 unrated sites a day).

Thus spake SMabille:

Hi, The principle is similar to the AV options. In flow based mode: the FG examine the packets passing thru, spot the URL, check it and if blocked just send a RST to both sides (or at least client side), so drop the traffic without more explanation to the user. In proxy mode: The FG act as a transparent proxy. It intercept any SYN to port 80 and act as a proxy. This mean that you have 2 sessions: one between the user and the proxy and one between the proxy and the server. This allow the FG to generate its own error page, warning etc... But is more CPU intensive Regarding the alert, the default config send an alert for " unrated" websites, and the number of sites not in Fortiguard database is relatively high compared to Bluecoat or Websense for example. I currently have very low web traffic (as only a couple of users are filtered for the moment and I get 3-5 unrated sites a day).

Thanks SMabille. Sometimes, having lots of history and experience is NOT a convenient thing :-} I started with firewalls in 1992. We still called things " bastion hosts" back then. We installed " proxies" which later became " explicit proxies" with the advent of " transparent proxies" . My experience with (transparent) proxies was similar to what you describe as FortiNet' s " Flow-based" - that is, if you really look at what is going on inside the FortiGate appliance' s processing threads, you would see two separate TCP circuits, one between the end-client and the FortiGate, and another between the FortiGate and whatever web server the end-client really wanted to talk to, with processing going on handing packets back and forth between them. Sometimes the intermediate device would let its own IP address appear to the destination web server, and other times it would not, although in today' s common everybody-behind-NAT scenario there' s less difference than there used to be. (We had real IP addresses back in the early 1990s, rather than RFC1918 duplicated addressing + NAT). Some things have gotten a LOT easier. Back then, setting up a " firewall" meant following a long and very technical set of instructions to install and configure software on top of general purpose hosts, enable and set up the Berkeley Packet Filter, install and configure explicit proxy software and put them port-by-port in /etc/inetd.conf (we didn' t even have xinetd back then). Today, it can mean unboxing an appliance which, to a large degree, can be deployed with minimal configuration steps out of the box. But of course today it also can mean learning the product details of a hugely capable UTM appliance (like the FG100D I' m using) with its own way of speaking about things, configuration interface quirks, etc. This feels more complicated somehow than building a DEC SEAL in 1992 did! :-) Quick aside, about the alerts I' m seeing, they' re for sites like *.google.com and *.skype.com, so it shouldn' t be an unrated site producing the alert... Digging deeper into the technology of how FortiNet implements URL Filtering (Proxy, or Flow-based, and differences if any between them), I get the impression from a recent support chat, and from the brief description of the Inspection Modes in the FortiOS v5 documentation, that Flow-based inspection mode may have a higher probability of false positives. I wonder why. Focusing only on URL Filtering (be it FortiGuard categories, or explicitly entered URLs in the FortiOS v5 Security Profiles -> Web Filter -> Profiles, pick a profile, " Enable Web Site Filter" section), in a classic proxy mode where the filtering appliance (the FortiGate in this case) is intercepting TCP connections at the SYN phase, either explicit or transparent, going back a bunch of years to before when we had session keepalives and multiple HTTP commands would issue over a single TCP session, detecting the URL was fairly straightforward. Now, these intermediate devices must monitor the whole session, understanding the on-going dialogs between web browser software and web server software, to figure out each successive HTTP command which would require passing a site name/path through a URL Filter. What I wonder is whether FortiNet' s implementation of how it figures out that something is a URL which needs to be passed through a URL Filter is different in Proxy-based vs. Flow-based Inspection Mode? .. Why/how are false positives more likely in Flow-based inspection mode?

Hi, Sorry if a wasn' t clear: flow-mode, there is one TCP connection between the client and the OCS (Original Content Server). In proxy mode the TCP connection from the client finish at the FG and the FG open a second connection to the OCS. In proxy mode, as the connection is terminated on the FG, it obviously act as an HTTP server and receive all the request, parse them and filter them. In flow mode the FG scan the packet and probably won' t make the difference between a real part of an HTTP request and plain text containing HTTP-like request. For example if in this message I write something like GET /index.html HTTP/1.1 Host: www.example.com. It' s very likely that flow based will catch it and filter it.

Hi SMabille, thanks for this information.

In flow mode the FG scan the packet and probably won' t make the difference between a real part of an HTTP request and plain text containing HTTP-like request. For example if in this message I write something like GET /index.html HTTP/1.1 Host: www.example.com. It' s very likely that flow based will catch it and filter it.

Is this a supposition? .. something you' re fairly sure of based on experimenting? .. a fact confirmed by an official statement from FortiNet? As users, we need a fairly high degree of certainty in " how well" each feature works. If Flow-based inspection (or any feature) is very likely to false positive, we need to know it. FortiNet' s documentation about Flow-based inspection says that it is much more efficient but somewhat more likely to false positive. That' s not enough information for us to make an operational decision. :(

I don' t think " higher probability of false positives" is accurate, if anything the catch rate will be lower with flow-based.... From the Fortigate docs regarding flow based inspection: " Advantages of flow-based antivirus scanning include faster scanning, lower memory requirements, and no file size limitation. Clients also begin receiving download file data immediately. Disadvantages include no detection of polymorphic and self-cloaking viruses and support for fewer file archive formats."

I knew I' d seen this. FortiOS Handbook v5.0.4, page 830-840:

  Inspections Modes        Proxy            Proxy-based inspection involves buffering the traffic and examining it as a           whole before determining an action. The process of having the whole of           the data to analyze allow this process to include more points of data to           analyze than the flow-based or DNS methods.          The advantage of a proxy-based method is that the inspection can be           more thorough than the other methods, resulting in fewer false positive           or negative results in the analysis of the data.        Flow-based            The Flow-based inspection method examines the file as it passes through           the FortiGate unit without any buffering. As each packet of the traffic arrives           it is process and forwarded without waiting for the complete file or web page,          etc.          The advantage of the flow-based method is that the user sees a faster           response time for HTTP requests and there is less chance of a time-out error           due to the server at the other end responding slowly.          The disadvantages of this method are that there is a higher probability           of a false positive or negative in the analysis of the data and that a number          of points of analysis that can be used in the proxy-based method are not           available in the flow-based inspection method. There is also fewer actions           available to choose from based on the categorization of the website by           FortiGuard services.  

In my case, significantly better operational efficiency and better user experience are more valuable than somewhat better security ... but these are all highly subjective. so I repeat that it really would be appropriate for FortiNet to disclose more technical detail about how flow-based inspection can cause false positives, and some statistics on how often it happens, so that we can decide which security measure is appropriate for each customer environment.