Web Filter false positives?

[Jay_Libove]

FG100D, FortiOS 5.0.4, I think I' m getting lots of false positives from the Web Filter. Under Security Profiles -> Web Filter -> Profiles, there is just the " default" profile. In the default profile, I have selected only the " Security Risk" category (including its three subcategories to be blocked. Screen shot: http://img96.imageshack.us/img96/3204/x76n.jpg This default security profile category is referenced in the one Policy rule which allows outbound web browsing traffic (basically, packets arriving on the internal Ethernet port, destined to go out the Internet Ethernet port, coming from internal IP addresses, going to anywhere; any time; any service; Accept, and apply the default Web Filter profile; and log security events). I' m getting several emails a day from the FortiGate saying things like:

Message meets Alert condition date=2013-09-30 time=11:12:48 devname=FG100D3G13807731 devid=FG100D3G13807731 logid=0315012544 type=webfilter subtype=urlfilter level=warning urlfilteridx=2 urlfilterlist=" default" policyid=25 identidx=0 sessionid=38633598 srcip=192.168.32.6 srcport=62925 srcintf=" internal2" dstip=50.18.249.41 dstport=443 dstintf=" ONO" service=" https" hostname=" 9gag.com" profile=" default" status=" blocked" reqtype=" direct" url=" /" sentbyte=0 rcvdbyte=0 msg=" URL was blocked because it is in the URL filter list" Message meets Alert condition date=2013-09-30 time=10:40:01 devname=FG100D3G13807731 devid=FG100D3G13807731 logid=0315012544 type=webfilter subtype=urlfilter level=warning urlfilteridx=2 urlfilterlist=" default" policyid=25 identidx=0 sessionid=38561211 srcip=192.168.32.6 srcport=61990 srcintf=" internal2" dstip=173.194.67.84 dstport=443 dstintf=" ONO" service=" https" hostname=" accounts.google.com" profile=" default" status=" blocked" reqtype=" direct" url=" /" sentbyte=0 rcvdbyte=0 msg=" URL was blocked because it is in the URL filter list"

Link to log screenshot: http://img542.imageshack.us/img542/5675/7lde.jpg Stupid question #1: When this message talks about the " URL was blocked because it is in the URL filter list" , does that mean that the FortiGate thinks that this URL was in one of the FortiNet-supplied categories for which I enabled blocking? .. or am I being stupid, and this is saying something about my custom URL rules (all of which permit specific URL patterns, unless I' ve done them wrong - see first screen shot)? Stupid question #2: Could these be filter service connection failures which are getting blocked by default but reported as hits, and what I need to do is tick the box to " Allow Websites When a Rating Error Occurs" ? I' ve opened a support case with FortiNet Support about this, but have been waiting more than a week for them to come up with anything useful so far, and I am quite frustrated. I don' t actually see errors appearing to the user in browsing sessions; I don' t see website failures which I think are related to specific subparts of web pages being blocked; but these messages make me doubt that this technology will reliably and predictably serve my users. Your help is appreciated. thanks,

[Bromont_FTNT]

What happens if you disable the URL filter in the webfilter profile? " Allow Websites When a Rating Error Occurs" is for if the Fortigate happens to lose connectivity to the Fortiguard servers and can' t check the rating... If this is checked then all sites will be allowed instead of all sites blocked due to Rating Error.

[Jay_Libove]

I ticked the " Allow websites when a rating error occurs" box. The false positives continue. So that wasn' t it. >What happens if you disable the URL filter in the webfilter profile? I' m not sure what this is suggesting that I test. I see: Security Profile -> Web Filter -> Profiles, profile " default" . In the " default" profile, " FortiGuard Categories" is selected; and within the inset box listing all of the FortiGuard Categories, just " Security Risk" and its three subcategories are marked with the action " Block" . Under that in the next section, I have " Enable Wed Site Filter" ticked, and I' ve manually entered six FQDNs or wildcard strings, all set to " Monitor" (so, it shouldn' t block these; and these don' t encompass the great majority of the false positives I' m seeing in the logs anyway, so this doesn' t seem to be related to the problem). So, I don' t see any place to disable the " URL filter" in the webfilter profile. What, more exactly, is it that I should do to try this theory? thanks,

[Bromont_FTNT]

The logs indicate this is caused by the URL filter (now renamed to Web Site Filter) so the test would be to disable URL filtering (Uncheck " Enable Web Site Filter" ) If that' s the cause then try deleting all the entries the re-add them.

[Jay_Libove]

Thank you Bromont. It' s very confusing how FortiNet changes the names of things but then leaves SOME of the user interface components, and parts of the documentation, listing these things under the old names... Okay, In FortiOS v5.0.4, I' ve edited Security Profiles -> Web Filter -> Profiles, " default" profile, " Web Site Filter" to be Disabled. I' ll see if I get any more of those alert messages, and update us all here. thanks, -Jay

[Jay_Libove]

Just got this: Message meets Alert condition date=2013-09-30 time=17:56:36 devname=FG100D3G13807731 devid=FG100D3G13807731 logid=0315012544 type=webfilter subtype=urlfilter level=warning urlfilteridx=0 policyid=25 identidx=0 sessionid=39489840 osname=" Mac OS X" osversion=" 10.8.5" srcip=192.168.32.6 srcport=53236 srcintf=" internal2" dstip=192.243.254.49 dstport=443 dstintf=" ONO" service=" https" hostname=" *.d2.sc.omtrdc.net" profile=" default" status=" passthrough" reqtype=" direct" url=" /" sentbyte=0 rcvdbyte=0 msg=" URL was blocked because it is in the URL filter list" " Enable Web Site Filter" is UN-checked. So it seems that it couldn' t be any URL pattern I' d entered in the Web Site Filter configuration of the Web Filter profile. Except, when it' s a category block, don' t those appear as being in a restricted category, instead of saying something about " URL filter list" ??

[Bromont_FTNT]

Interesting... the urlfiteridx is now 0 instead of 2 and the status is passthrough instead of blocked.How long since the unit was rebooted?

[Jay_Libove]

Interesting... the urlfiteridx is now 0 instead of 2 and the status is passthrough instead of blocked.How long since the unit was rebooted?

It' s been up for 39 days. Does FortiOS tend to accumulate garbage and errors in in-memory data structures, such that scheduled pro-active reboots are considered a good practice? I' ll reboot it and see if the apparent false positives continue.

[Jay_Libove]

I rebooted. I haven' t seen more URL Filter hits. (There was one brief flurry of warnings of filtering service failures, but I imagine that was a race condition between the FG100D coming back up and starting to serve connections again, and the URL Filter service taking a few seconds longer to start up). Of course, I' ll only believe that the problem of the false positives on the Web Filter categories has gone away when it has stayed away for several days. If a reboot was needed, then I have other worries, about the (lack of) quality/reliability of FortiOS 5. On the other hand, now I' m getting lots of warnings about log memory or disk being full. I reduced all logging in all of my Policies to log only on security events, and I did a delete-all of logs, and I' m still getting alerts of log memory or disk being full. I' m really disappointed in this FG100D and FortiOS 5 (5.0.4 presently).

[Jay_Libove]

Oh, h*ll. Of course I get the next email right after I made that last post:

Message meets Alert condition date=2013-10-01 time=11:12:51 devname=FG100D3G13807731 devid=FG100D3G13807731 logid=0315012544 type=webfilter subtype=urlfilter level=warning urlfilteridx=0 policyid=25 identidx=0 sessionid=164649 srcname=" roadrunner.local" osname=" Mac OS X" osversion=" 10.8.5" unauthuser=" ganguera" unauthusersource=" forticlient" srcip=192.168.32.6 srcport=49213 srcintf=" internal2" dstip=173.194.34.241 dstport=443 dstintf=" ONO" service=" https" hostname=" www.google.com" profile=" default" status=" passthrough" reqtype=" direct" url=" /" sentbyte=0 rcvdbyte=0 msg=" URL was blocked because it is in the URL filter list" Message meets Alert condition date=2013-10-01 time=11:12:50 devname=FG100D3G13807731 devid=FG100D3G13807731 logid=0315012544 type=webfilter subtype=urlfilter level=warning urlfilteridx=0 policyid=25 identidx=0 sessionid=164630 srcname=" roadrunner.local" osname=" Mac OS X" osversion=" 10.8.5" unauthuser=" ganguera" unauthusersource=" forticlient" srcip=192.168.32.6 srcport=49209 srcintf=" internal2" dstip=173.194.34.240 dstport=443 dstintf=" ONO" service=" https" hostname=" www.google.com" profile=" default" status=" passthrough" reqtype=" direct" url=" /" sentbyte=0 rcvdbyte=0 msg=" URL was blocked because it is in the URL filter list"

Security Profiles -> Web Filter -> Profiles, " default" profile, " Enable Web Site Filter" remains UN-checked. Only the " Security Risk" FortiGuard Category remains checked, as before. IN THE GUI. At the CLI, " show webfilter urlfilter ?" shows just one filter, " 2" . showing filter 2 shows all of the entries I' d manually created (in the GUI) earlier, but none of which should be active (according to the GUI). Indeed, " show webfilter profile default" does NOT include a reference to urlfilter 2 (nor to any urlfilter), so these entries, which DO seem to be active, do not seem to be configured. Weird.

 # show webfilter profile default 
 config webfilter profile
     edit " default" 
         set comment " default web filtering" 
         set replacemsg-group " web-filter-default" 
         set inspection-mode flow-based
         set ovrd-perm bannedword-override urlfilter-override fortiguard-wf-override contenttype-check-override
         set post-action comfort
             config override
                 set ovrd-user-group " " 
             end
             config ftgd-wf
                 set options error-allow
                 set category-override 140 141
                     config filters
                         edit 19
                             set category 4
                         next
                         edit 18
                             set action block
                             set category 26
                             set override-replacemsg " 26" 
                         next
                         edit 20
                             set action block
                             set category 61
                             set override-replacemsg " 26" 
                         next
                         edit 21
                             set action block
                             set category 86
                             set override-replacemsg " 26" 
                         next
                     end
             end
         set log-all-url enable
     next
 end