Proxy-based vs. Flow-based Inspection Mode for Web Filter profiles?

[Jay_Libove]

Can someone explain the specific ways/differences in how the two Inspection Modes " Proxy-based" and " Flow-based" work in FortiOS v5.0.4 please? I had guessed that " Proxy-based" meant what we have clasically called " explicit proxy" , where the web browser would have to be configured to proxy to an IP address assigned to the FortiGate itself, and " Flow-based" meant what we have classically called " transparent proxy" . But I just did a test, switching the Web Filter Profile to Proxy based Inspection Mode, and without making any change in my web browser' s configuration, the FortiGuard Categories are properly applied. The FortiOS v5 handbook on page 774 gives a very brief treatment of Flow-based vs. Proxy-based, suggesting that flow-based is packet-by-packet, does no buffering, is faster; whereas proxy-based buffers up data objects which flow through the FortiGate, is slower, but could be more accurate. But I' ve been unable to find anything more detailed, and I' d prefer to know, so I can make a better decision as to what best serves my users' needs. Aside, this is brought up by the weird URL filtering effects I' ve been dicussing in another thread. So knowing more about how flow-based vs. proxy-based really works may help me understand better whether the noticeable number of apparently spurious alert messages my FortiGate is sending me about URLs which match filters, even though nothing like the " matched" URL exists in any filter on my FortiGate (in fact, I have one single filter, containing one single simple match " www.meneame.net" , so almost nothing should ever match it). thank you,

[SMabille]

Hi, The principle is similar to the AV options. In flow based mode: the FG examine the packets passing thru, spot the URL, check it and if blocked just send a RST to both sides (or at least client side), so drop the traffic without more explanation to the user. In proxy mode: The FG act as a transparent proxy. It intercept any SYN to port 80 and act as a proxy. This mean that you have 2 sessions: one between the user and the proxy and one between the proxy and the server. This allow the FG to generate its own error page, warning etc... But is more CPU intensive Regarding the alert, the default config send an alert for " unrated" websites, and the number of sites not in Fortiguard database is relatively high compared to Bluecoat or Websense for example. I currently have very low web traffic (as only a couple of users are filtered for the moment and I get 3-5 unrated sites a day).

[Jay_Libove]

Thus spake SMabille:

Hi, The principle is similar to the AV options. In flow based mode: the FG examine the packets passing thru, spot the URL, check it and if blocked just send a RST to both sides (or at least client side), so drop the traffic without more explanation to the user. In proxy mode: The FG act as a transparent proxy. It intercept any SYN to port 80 and act as a proxy. This mean that you have 2 sessions: one between the user and the proxy and one between the proxy and the server. This allow the FG to generate its own error page, warning etc... But is more CPU intensive Regarding the alert, the default config send an alert for " unrated" websites, and the number of sites not in Fortiguard database is relatively high compared to Bluecoat or Websense for example. I currently have very low web traffic (as only a couple of users are filtered for the moment and I get 3-5 unrated sites a day).

Thanks SMabille. Sometimes, having lots of history and experience is NOT a convenient thing :-} I started with firewalls in 1992. We still called things " bastion hosts" back then. We installed " proxies" which later became " explicit proxies" with the advent of " transparent proxies" . My experience with (transparent) proxies was similar to what you describe as FortiNet' s " Flow-based" - that is, if you really look at what is going on inside the FortiGate appliance' s processing threads, you would see two separate TCP circuits, one between the end-client and the FortiGate, and another between the FortiGate and whatever web server the end-client really wanted to talk to, with processing going on handing packets back and forth between them. Sometimes the intermediate device would let its own IP address appear to the destination web server, and other times it would not, although in today' s common everybody-behind-NAT scenario there' s less difference than there used to be. (We had real IP addresses back in the early 1990s, rather than RFC1918 duplicated addressing + NAT). Some things have gotten a LOT easier. Back then, setting up a " firewall" meant following a long and very technical set of instructions to install and configure software on top of general purpose hosts, enable and set up the Berkeley Packet Filter, install and configure explicit proxy software and put them port-by-port in /etc/inetd.conf (we didn' t even have xinetd back then). Today, it can mean unboxing an appliance which, to a large degree, can be deployed with minimal configuration steps out of the box. But of course today it also can mean learning the product details of a hugely capable UTM appliance (like the FG100D I' m using) with its own way of speaking about things, configuration interface quirks, etc. This feels more complicated somehow than building a DEC SEAL in 1992 did! :-) Quick aside, about the alerts I' m seeing, they' re for sites like *.google.com and *.skype.com, so it shouldn' t be an unrated site producing the alert... Digging deeper into the technology of how FortiNet implements URL Filtering (Proxy, or Flow-based, and differences if any between them), I get the impression from a recent support chat, and from the brief description of the Inspection Modes in the FortiOS v5 documentation, that Flow-based inspection mode may have a higher probability of false positives. I wonder why. Focusing only on URL Filtering (be it FortiGuard categories, or explicitly entered URLs in the FortiOS v5 Security Profiles -> Web Filter -> Profiles, pick a profile, " Enable Web Site Filter" section), in a classic proxy mode where the filtering appliance (the FortiGate in this case) is intercepting TCP connections at the SYN phase, either explicit or transparent, going back a bunch of years to before when we had session keepalives and multiple HTTP commands would issue over a single TCP session, detecting the URL was fairly straightforward. Now, these intermediate devices must monitor the whole session, understanding the on-going dialogs between web browser software and web server software, to figure out each successive HTTP command which would require passing a site name/path through a URL Filter. What I wonder is whether FortiNet' s implementation of how it figures out that something is a URL which needs to be passed through a URL Filter is different in Proxy-based vs. Flow-based Inspection Mode? .. Why/how are false positives more likely in Flow-based inspection mode?

[SMabille]

Hi, Sorry if a wasn' t clear: flow-mode, there is one TCP connection between the client and the OCS (Original Content Server). In proxy mode the TCP connection from the client finish at the FG and the FG open a second connection to the OCS. In proxy mode, as the connection is terminated on the FG, it obviously act as an HTTP server and receive all the request, parse them and filter them. In flow mode the FG scan the packet and probably won' t make the difference between a real part of an HTTP request and plain text containing HTTP-like request. For example if in this message I write something like GET /index.html HTTP/1.1 Host: www.example.com. It' s very likely that flow based will catch it and filter it.

[Jay_Libove]

Hi SMabille, thanks for this information.

In flow mode the FG scan the packet and probably won' t make the difference between a real part of an HTTP request and plain text containing HTTP-like request. For example if in this message I write something like GET /index.html HTTP/1.1 Host: www.example.com. It' s very likely that flow based will catch it and filter it.

Is this a supposition? .. something you' re fairly sure of based on experimenting? .. a fact confirmed by an official statement from FortiNet? As users, we need a fairly high degree of certainty in " how well" each feature works. If Flow-based inspection (or any feature) is very likely to false positive, we need to know it. FortiNet' s documentation about Flow-based inspection says that it is much more efficient but somewhat more likely to false positive. That' s not enough information for us to make an operational decision. :(

[Bromont_FTNT]

I don' t think " higher probability of false positives" is accurate, if anything the catch rate will be lower with flow-based.... From the Fortigate docs regarding flow based inspection: " Advantages of flow-based antivirus scanning include faster scanning, lower memory requirements, and no file size limitation. Clients also begin receiving download file data immediately. Disadvantages include no detection of polymorphic and self-cloaking viruses and support for fewer file archive formats."

[Jay_Libove]

I knew I' d seen this. FortiOS Handbook v5.0.4, page 830-840:

 Inspections Modes
 
     Proxy
 
         Proxy-based inspection involves buffering the traffic and examining it as a 
         whole before determining an action. The process of having the whole of 
         the data to analyze allow this process to include more points of data to 
         analyze than the flow-based or DNS methods.
         The advantage of a proxy-based method is that the inspection can be 
         more thorough than the other methods, resulting in fewer false positive 
         or negative results in the analysis of the data.
 
     Flow-based
 
         The Flow-based inspection method examines the file as it passes through 
         the FortiGate unit without any buffering. As each packet of the traffic arrives 
         it is process and forwarded without waiting for the complete file or web page,
         etc.
         The advantage of the flow-based method is that the user sees a faster 
         response time for HTTP requests and there is less chance of a time-out error 
         due to the server at the other end responding slowly.
         The disadvantages of this method are that there is a higher probability 
         of a false positive or negative in the analysis of the data and that a number
         of points of analysis that can be used in the proxy-based method are not 
         available in the flow-based inspection method. There is also fewer actions 
         available to choose from based on the categorization of the website by 
         FortiGuard services.
 

In my case, significantly better operational efficiency and better user experience are more valuable than somewhat better security ... but these are all highly subjective. so I repeat that it really would be appropriate for FortiNet to disclose more technical detail about how flow-based inspection can cause false positives, and some statistics on how often it happens, so that we can decide which security measure is appropriate for each customer environment.

[lonegray]

so looks like flow based is examining only the packets and proxy based is examining the HTTP-Requests (e.g., HTTP-GET or HTTP-POST).  Unclear from that documentation if it is also examining the HTTP-Response as well.