New Member

Question

Web Filter false positives?

Forum|Forum|12 years ago
September 30, 2013
18 replies
14806 views

FG100D, FortiOS 5.0.4, I think I' m getting lots of false positives from the Web Filter. Under Security Profiles -> Web Filter -> Profiles, there is just the " default" profile. In the default profile, I have selected only the " Security Risk" category (including its three subcategories to be blocked. Screen shot: http://img96.imageshack.us/img96/3204/x76n.jpg This default security profile category is referenced in the one Policy rule which allows outbound web browsing traffic (basically, packets arriving on the internal Ethernet port, destined to go out the Internet Ethernet port, coming from internal IP addresses, going to anywhere; any time; any service; Accept, and apply the default Web Filter profile; and log security events). I' m getting several emails a day from the FortiGate saying things like:

Message meets Alert condition date=2013-09-30 time=11:12:48 devname=FG100D3G13807731 devid=FG100D3G13807731 logid=0315012544 type=webfilter subtype=urlfilter level=warning urlfilteridx=2 urlfilterlist=" default" policyid=25 identidx=0 sessionid=38633598 srcip=192.168.32.6 srcport=62925 srcintf=" internal2" dstip=50.18.249.41 dstport=443 dstintf=" ONO" service=" https" hostname=" 9gag.com" profile=" default" status=" blocked" reqtype=" direct" url=" /" sentbyte=0 rcvdbyte=0 msg=" URL was blocked because it is in the URL filter list" Message meets Alert condition date=2013-09-30 time=10:40:01 devname=FG100D3G13807731 devid=FG100D3G13807731 logid=0315012544 type=webfilter subtype=urlfilter level=warning urlfilteridx=2 urlfilterlist=" default" policyid=25 identidx=0 sessionid=38561211 srcip=192.168.32.6 srcport=61990 srcintf=" internal2" dstip=173.194.67.84 dstport=443 dstintf=" ONO" service=" https" hostname=" accounts.google.com" profile=" default" status=" blocked" reqtype=" direct" url=" /" sentbyte=0 rcvdbyte=0 msg=" URL was blocked because it is in the URL filter list"

Link to log screenshot: http://img542.imageshack.us/img542/5675/7lde.jpg Stupid question #1: When this message talks about the " URL was blocked because it is in the URL filter list" , does that mean that the FortiGate thinks that this URL was in one of the FortiNet-supplied categories for which I enabled blocking? .. or am I being stupid, and this is saying something about my custom URL rules (all of which permit specific URL patterns, unless I' ve done them wrong - see first screen shot)? Stupid question #2: Could these be filter service connection failures which are getting blocked by default but reported as hits, and what I need to do is tick the box to " Allow Websites When a Rating Error Occurs" ? I' ve opened a support case with FortiNet Support about this, but have been waiting more than a week for them to come up with anything useful so far, and I am quite frustrated. I don' t actually see errors appearing to the user in browsing sessions; I don' t see website failures which I think are related to specific subparts of web pages being blocked; but these messages make me doubt that this technology will reliably and predictably serve my users. Your help is appreciated. thanks,

Bromont_FTNT

Staff

What happens if you disable the URL filter in the webfilter profile? " Allow Websites When a Rating Error Occurs" is for if the Fortigate happens to lose connectivity to the Fortiguard servers and can' t check the rating... If this is checked then all sites will be allowed instead of all sites blocked due to Rating Error.

Jay_LiboveAuthor

New Member

I ticked the " Allow websites when a rating error occurs" box. The false positives continue. So that wasn' t it. >What happens if you disable the URL filter in the webfilter profile? I' m not sure what this is suggesting that I test. I see: Security Profile -> Web Filter -> Profiles, profile " default" . In the " default" profile, " FortiGuard Categories" is selected; and within the inset box listing all of the FortiGuard Categories, just " Security Risk" and its three subcategories are marked with the action " Block" . Under that in the next section, I have " Enable Wed Site Filter" ticked, and I' ve manually entered six FQDNs or wildcard strings, all set to " Monitor" (so, it shouldn' t block these; and these don' t encompass the great majority of the false positives I' m seeing in the logs anyway, so this doesn' t seem to be related to the problem). So, I don' t see any place to disable the " URL filter" in the webfilter profile. What, more exactly, is it that I should do to try this theory? thanks,

Bromont_FTNT

Staff

The logs indicate this is caused by the URL filter (now renamed to Web Site Filter) so the test would be to disable URL filtering (Uncheck " Enable Web Site Filter" ) If that' s the cause then try deleting all the entries the re-add them.

Jay_LiboveAuthor

New Member

Thank you Bromont. It' s very confusing how FortiNet changes the names of things but then leaves SOME of the user interface components, and parts of the documentation, listing these things under the old names... Okay, In FortiOS v5.0.4, I' ve edited Security Profiles -> Web Filter -> Profiles, " default" profile, " Web Site Filter" to be Disabled. I' ll see if I get any more of those alert messages, and update us all here. thanks, -Jay

Jay_LiboveAuthor

New Member

Just got this: Message meets Alert condition date=2013-09-30 time=17:56:36 devname=FG100D3G13807731 devid=FG100D3G13807731 logid=0315012544 type=webfilter subtype=urlfilter level=warning urlfilteridx=0 policyid=25 identidx=0 sessionid=39489840 osname=" Mac OS X" osversion=" 10.8.5" srcip=192.168.32.6 srcport=53236 srcintf=" internal2" dstip=192.243.254.49 dstport=443 dstintf=" ONO" service=" https" hostname=" *.d2.sc.omtrdc.net" profile=" default" status=" passthrough" reqtype=" direct" url=" /" sentbyte=0 rcvdbyte=0 msg=" URL was blocked because it is in the URL filter list" " Enable Web Site Filter" is UN-checked. So it seems that it couldn' t be any URL pattern I' d entered in the Web Site Filter configuration of the Web Filter profile. Except, when it' s a category block, don' t those appear as being in a restricted category, instead of saying something about " URL filter list" ??

Bromont_FTNT

Staff

Interesting... the urlfiteridx is now 0 instead of 2 and the status is passthrough instead of blocked.How long since the unit was rebooted?

Jay_LiboveAuthor

New Member

Interesting... the urlfiteridx is now 0 instead of 2 and the status is passthrough instead of blocked.How long since the unit was rebooted?

It' s been up for 39 days. Does FortiOS tend to accumulate garbage and errors in in-memory data structures, such that scheduled pro-active reboots are considered a good practice? I' ll reboot it and see if the apparent false positives continue.

Jay_LiboveAuthor

New Member

I rebooted. I haven' t seen more URL Filter hits. (There was one brief flurry of warnings of filtering service failures, but I imagine that was a race condition between the FG100D coming back up and starting to serve connections again, and the URL Filter service taking a few seconds longer to start up). Of course, I' ll only believe that the problem of the false positives on the Web Filter categories has gone away when it has stayed away for several days. If a reboot was needed, then I have other worries, about the (lack of) quality/reliability of FortiOS 5. On the other hand, now I' m getting lots of warnings about log memory or disk being full. I reduced all logging in all of my Policies to log only on security events, and I did a delete-all of logs, and I' m still getting alerts of log memory or disk being full. I' m really disappointed in this FG100D and FortiOS 5 (5.0.4 presently).

Jay_LiboveAuthor

New Member

Oh, h*ll. Of course I get the next email right after I made that last post:

Message meets Alert condition date=2013-10-01 time=11:12:51 devname=FG100D3G13807731 devid=FG100D3G13807731 logid=0315012544 type=webfilter subtype=urlfilter level=warning urlfilteridx=0 policyid=25 identidx=0 sessionid=164649 srcname=" roadrunner.local" osname=" Mac OS X" osversion=" 10.8.5" unauthuser=" ganguera" unauthusersource=" forticlient" srcip=192.168.32.6 srcport=49213 srcintf=" internal2" dstip=173.194.34.241 dstport=443 dstintf=" ONO" service=" https" hostname=" www.google.com" profile=" default" status=" passthrough" reqtype=" direct" url=" /" sentbyte=0 rcvdbyte=0 msg=" URL was blocked because it is in the URL filter list" Message meets Alert condition date=2013-10-01 time=11:12:50 devname=FG100D3G13807731 devid=FG100D3G13807731 logid=0315012544 type=webfilter subtype=urlfilter level=warning urlfilteridx=0 policyid=25 identidx=0 sessionid=164630 srcname=" roadrunner.local" osname=" Mac OS X" osversion=" 10.8.5" unauthuser=" ganguera" unauthusersource=" forticlient" srcip=192.168.32.6 srcport=49209 srcintf=" internal2" dstip=173.194.34.240 dstport=443 dstintf=" ONO" service=" https" hostname=" www.google.com" profile=" default" status=" passthrough" reqtype=" direct" url=" /" sentbyte=0 rcvdbyte=0 msg=" URL was blocked because it is in the URL filter list"

Security Profiles -> Web Filter -> Profiles, " default" profile, " Enable Web Site Filter" remains UN-checked. Only the " Security Risk" FortiGuard Category remains checked, as before. IN THE GUI. At the CLI, " show webfilter urlfilter ?" shows just one filter, " 2" . showing filter 2 shows all of the entries I' d manually created (in the GUI) earlier, but none of which should be active (according to the GUI). Indeed, " show webfilter profile default" does NOT include a reference to urlfilter 2 (nor to any urlfilter), so these entries, which DO seem to be active, do not seem to be configured. Weird.

  # show webfilter profile default   config webfilter profile      edit " default"           set comment " default web filtering"           set replacemsg-group " web-filter-default"           set inspection-mode flow-based          set ovrd-perm bannedword-override urlfilter-override fortiguard-wf-override contenttype-check-override          set post-action comfort              config override                  set ovrd-user-group " "               end              config ftgd-wf                  set options error-allow                  set category-override 140 141                      config filters                          edit 19                              set category 4                          next                          edit 18                              set action block                              set category 26                              set override-replacemsg " 26"                           next                          edit 20                              set action block                              set category 61                              set override-replacemsg " 26"                           next                          edit 21                              set action block                              set category 86                              set override-replacemsg " 26"                           next                      end              end          set log-all-url enable      next  end

Bromont_FTNT

Staff

At this point I' d create a new webfilter profile and use the newly created profile instead of this default one.

Show more replies

Sign up

Login with SSO

Login to the community

Login with SSO

Scanning file for viruses.

This file cannot be downloaded