New Member

Question

Web Filter false positives?

18 replies
14906 views

FG100D, FortiOS 5.0.4, I think I' m getting lots of false positives from the Web Filter. Under Security Profiles -> Web Filter -> Profiles, there is just the " default" profile. In the default profile, I have selected only the " Security Risk" category (including its three subcategories to be blocked. Screen shot: http://img96.imageshack.us/img96/3204/x76n.jpg This default security profile category is referenced in the one Policy rule which allows outbound web browsing traffic (basically, packets arriving on the internal Ethernet port, destined to go out the Internet Ethernet port, coming from internal IP addresses, going to anywhere; any time; any service; Accept, and apply the default Web Filter profile; and log security events). I' m getting several emails a day from the FortiGate saying things like:

Message meets Alert condition date=2013-09-30 time=11:12:48 devname=FG100D3G13807731 devid=FG100D3G13807731 logid=0315012544 type=webfilter subtype=urlfilter level=warning urlfilteridx=2 urlfilterlist=" default" policyid=25 identidx=0 sessionid=38633598 srcip=192.168.32.6 srcport=62925 srcintf=" internal2" dstip=50.18.249.41 dstport=443 dstintf=" ONO" service=" https" hostname=" 9gag.com" profile=" default" status=" blocked" reqtype=" direct" url=" /" sentbyte=0 rcvdbyte=0 msg=" URL was blocked because it is in the URL filter list" Message meets Alert condition date=2013-09-30 time=10:40:01 devname=FG100D3G13807731 devid=FG100D3G13807731 logid=0315012544 type=webfilter subtype=urlfilter level=warning urlfilteridx=2 urlfilterlist=" default" policyid=25 identidx=0 sessionid=38561211 srcip=192.168.32.6 srcport=61990 srcintf=" internal2" dstip=173.194.67.84 dstport=443 dstintf=" ONO" service=" https" hostname=" accounts.google.com" profile=" default" status=" blocked" reqtype=" direct" url=" /" sentbyte=0 rcvdbyte=0 msg=" URL was blocked because it is in the URL filter list"

Link to log screenshot: http://img542.imageshack.us/img542/5675/7lde.jpg Stupid question #1: When this message talks about the " URL was blocked because it is in the URL filter list" , does that mean that the FortiGate thinks that this URL was in one of the FortiNet-supplied categories for which I enabled blocking? .. or am I being stupid, and this is saying something about my custom URL rules (all of which permit specific URL patterns, unless I' ve done them wrong - see first screen shot)? Stupid question #2: Could these be filter service connection failures which are getting blocked by default but reported as hits, and what I need to do is tick the box to " Allow Websites When a Rating Error Occurs" ? I' ve opened a support case with FortiNet Support about this, but have been waiting more than a week for them to come up with anything useful so far, and I am quite frustrated. I don' t actually see errors appearing to the user in browsing sessions; I don' t see website failures which I think are related to specific subparts of web pages being blocked; but these messages make me doubt that this technology will reliably and predictably serve my users. Your help is appreciated. thanks,

Show previous replies

Jay_LiboveAuthor

New Member

I blanked the webfilter entirely. There is none now. And I just got this:

Message meets Alert condition date=2013-10-01 time=18:23:25 devname=FG100D3G........ devid=FG100D3G........ logid=0315012544 type=webfilter subtype=urlfilter level=warning urlfilteridx=0 policyid=25 identidx=0 sessionid=1114237 srcname=" MacBook-MacBook-Pro-de-Borja.local" osname=" Mac OS X" osversion=" 10.8.4" unauthuser=" bj......" unauthusersource=" forticlient" srcip=192.168.32.8 srcport=59901 srcintf=" internal2" dstip=23.45.74.161 dstport=443 dstintf=" ONO" service=" https" hostname=" *.skype.com" profile=" default" status=" passthrough" reqtype=" direct" url=" /" sentbyte=0 rcvdbyte=0 msg=" URL was blocked because it is in the URL filter list"

How is that possible? The referenced firewall policy 25 is my generic " from inside to outside" policy. This policy references the default webfilter profile, which has " Enable Web Site Filter" UN-checked. And is empty anyway, see next code block:

  # show webfilter urlfilter   config webfilter urlfilter  end

So, no URL filter at all. The FortiGuard categories are here:

  # show webfilter profile default   config webfilter profile      edit " default"           set comment " default web filtering"           set replacemsg-group " web-filter-default"           set inspection-mode flow-based          set ovrd-perm bannedword-override urlfilter-override fortiguard-wf-override contenttype-check-override          set post-action comfort              config override                  set ovrd-user-group " "               end              config ftgd-wf                  set options error-allow                  set category-override 140 141                      config filters                          edit 19                              set category 4                          next                          edit 18                              set action block                              set category 26                              set override-replacemsg " 26"                           next                          edit 20                              set action block                              set category 61                              set override-replacemsg " 26"                           next                          edit 21                              set action block                              set category 86                              set override-replacemsg " 26"                           next                      end              end          set log-all-url enable      next  end

Just some FortiGuard categories (the Security Threat ones), but the error about *.skype.com being blocked isn' t from a FortiGuard category. The actual firewall policy definition, in case it helps:

      edit 25          set srcintf " internal2"           set dstintf " ONO"           set srcaddr " SP internal wired LAN2"           set dstaddr " all"           set action accept          set status enable          set schedule " always"           set service " ALL"           set utm-status enable          set logtraffic utm          set av-profile " default"           set webfilter-profile " default"           set spamfilter-profile " default"           set ips-sensor " default"           set nat enable      next

So, from where could this *.skype.com URL Filter block possibly be arising?? thanks,

Jay_LiboveAuthor

New Member

Spent an hour on the phone with FortiNet support this morning. Lots of confusion (some of it due to changes between FortiOS v4 and v5) about exactly how integrated are the internal processing of Web Category (FortiGuard) filtering and the internal processing of URL filtering. Confusion whether the messages I' m receiving which concern me, which all include the phrase " URL was blocked because it is in the URL filter list" , even though the listed URLs don' t match anything in any ' urlfilter' on my FortiGate, could possibly have anything to do with FortiGuard Category filtering. (I don' t think so). Still not clear why these " URL was blocked because it is in the URL filter list" are in fact *not* blocked (we see no evidence of web pages being blocked; we have done tests and both Category and URL filters produce the expected block webpages, so we' re pretty sure these " URL was blocked because it is in the URL filter list" are giving an inccurate message). .. And why the say " status=" passthrough" " . If the status=" passthrough" happens when a filter is checked and does not find a real hit, then I would expect to see many thousands of these messages, essentially one for every web page view. But in fact although I am getting these at the rate of several per hour, I am not getting anywhere near enough to fit this theory of " had to check, didn' t hit -> warn (unnecessarily)" . In other words, still very confused; FortiNet continues researching. I assume that I' ve found another bug, which is triggering messages which shouldn' t happen at all, or where the message itself may indicate something actually a little bit noteworthy (although I can' t yet guess what) but, like the admin timeout " configuration changed" messages is simply misleading. Will keep you informed, and would appreciate any other input you might offer, community. thanks,

Bromont_FTNT

Staff

Did you ever create a new webfilter profile and try it instead of blanking the default one?

Jay_LiboveAuthor

New Member

Thanks bromont. That will be next. Right now, as I' ve recently changed the Inspection Mode from flow-based to proxy-based, I' m waiting to see if the results change. I need to give it a day or two to be sure whether thing are the same or different. -Jay

Wayne11

Explorer

This sounds very similar to me and what we had/have with the DLP. Files were blocked without being in the DLP file pattern list. It was once in the list with 4.x but deleted it on 5.0.3. Anyway, I guess there are still a few old tables with referenced entries from 4.x, but they are not showing up in the GUI even not in the CLI. Support confirmed the strange behaviour but they are not able to reproduce in their lab with our config, so we just should do a factory reset and begin again from scratch. Waiting for 5.0.5 and hope they' ve found and solved such version 4 remnants.

vanc

New Member

Since you have both FortiGuard rating, and local URL lists, I suspect the urlfilter daemon had memory issues which lead to wrong blocking. When that happens again, please try use the following command to restart urlfilter and see if it works for you. diag test application urlfilter 99 If it works, update your tickets so that support could report a bug for dev team.

Jay_LiboveAuthor

New Member

Vanc wrote:

Since you have both FortiGuard rating, and local URL lists, I suspect the urlfilter daemon had memory issues which lead to wrong blocking. When that happens again, please try use the following command to restart urlfilter and see if it works for you.

Hi Vanc. In my particular case, memory issues would seem extremely unlikely, as these errors appeared very early in my testing, when I had only one user (me) behind the FortiGate FG100D - a lot of UTM power for one little (okay, medium sized) web browser :)

vanc

New Member

From your screen shots, you only had few entries in the list. I recommend to add the URLs one by one and see which one could trigger false match. Just like you, I' ve one 100D for myself as well. I have 11 URLs in the filter list, and with FortiGuard categories enabled. I' ve been running 5.0.4 for 2 months and haven' t found similar issues.

Sign up

Login with SSO

Login to the community

Login with SSO