Website Categories charts never seem to populate.

[neonbit]

Hi all,

I've never been able to get some certain web category charts to populate. Was wondering if anyone else has had some luck with these.

For example, this chart/dataset always show no data:

Chart: Most Commonly Visited Web Categories Dataset: apprisk-ctrl-Top-Web-Categories-Visited

The web profile is configured to monitor or block each category, and I can see them showing up fine in the logs and FortiView.

If I edit the dataset and test, it still shows no data.

Below is the web profile I'm using (condensed):

config webfilter profile     edit "strict-filtering"             config override                 set ovrd-user-group ""             end             config web                 set safe-search url                 set log-search enable             end             config ftgd-wf                 set options rate-server-ip                     config filters                         edit 1                             set category 140                         next                          ...............................

                         ...............................                         next                     end             end         set log-all-url enable     next end

Does anyone have ideas on why nothing is showing up?

[abelio]

Hi,

2 hints:

before to deal with datasets/charts, be sure you have the data (logs) indexed in your faz.

use fortiview/log browse and verify that; did you?

Then, verify your logs matches with dataset query.

hope it helps

[neonbit]

Thanks Abel, the logs are showing up correctly in the log view. Both the traffic log and security>web filter logs are populated.

The default dataset is pulling traffic from the traffic logs, and from what I can see the data that is being quereid exist in the logs.

SQL Data-set:

select catdesc, count(distinct f_user) as user_num, sum(sessions) as sessions, sum(bandwidth) as bandwidth from (###(select catdesc, coalesce(nullifna(`user`), nullifna(`unauthuser`), ipstr(`srcip`)) as f_user, count(*) as sessions, sum(coalesce(sentbyte, 0)+coalesce(rcvdbyte, 0)) as bandwidth from $log-traffic where $filter and catdesc is not null and logid_to_int(logid) not in (4, 7, 14) and utmevent in ('webfilter', 'banned-word', 'web-content', 'command-block', 'script-filter') group by f_user, catdesc order by sessions desc)### union all ###(select catdesc, coalesce(nullifna(`user`), ipstr(`srcip`)) as f_user, count(*) as sessions, sum(coalesce(sentbyte, 0)+coalesce(rcvdbyte, 0)) as bandwidth from $log-webfilter where $filter and catdesc is not null and (eventtype is null or logver>=52) group by f_user, catdesc order by sessions desc)###) t group by catdesc order by sessions desc

Traffic Log:

itime=2015-10-22 11:24:40 vd=root rcvdbyte=699 srccountry=Reserved app=HTTP.BROWSER_IE utmaction=allow transip=10.100.100.1 date=2015-10-22 devtype=Windows PC dstip=202.177.203.124 duration=7 sentbyte=948 transport=51025 logid=13 apprisk=medium group=Domain Users srcmac=00:1c:22:b2:4c:f5 service=HTTP proto=6 user=JDOE devid=FGVM010000000000 countweb=1 applist=default poluuid=74dea260-47b2-51e5-ee8d-35f450653857 dstport=80 type=traffic devname=fortigate dstname=bs.serving-sys.com dtime=2015-10-22 11:24:41 trandisp=snat osname=Windows catdesc=Advertising sessionid=12888086 itime_t=1445473480 policyid=17 srcintf=port2 srcip=10.100.100.1 sentpkt=5 osversion=7 Service Pack 1 level=notice appcat=Web.Others appid=34038 srcport=51025 appact=detected srcname=win7 subtype=forward rcvdpkt=5 dstcountry=Singapore countapp=1 time=11:24:41 action=close mastersrcmac=00:1c:22:b2:4c:f5 hostname=bs.serving-sys.com dstintf=port1

I changed the log source from Traffic to Web filter to test but it still gives me nothing.

Web filter log:

itime=2015-10-22 00:14:02 vd=root rcvdbyte=0 date=2015-10-22 dstip=96.45.33.98 sentbyte=1375 group=Domain Users service=HTTP proto=6 eventtype=ftgd_allow devid=FGVM010000000000 msg=URL belongs to an allowed category in policy dstport=80 type=utm method=domain profile=strict-filtering direction=outgoing dtime=2015-10-22 00:14:04 devname=fortigate catdesc=Information Technology sessionid=12870372 itime_t=1445473342 user=JDOE srcintf=port2 reqtype=direct srcip=10.100.100.1 level=notice url=/fdsupdate srcport=59964 cat=52 logid=13312 subtype=webfilter time=00:14:04 action=passthrough hostname=96.45.33.98 dstintf=port1

The annoying thing is that this chart is in some of the default reports, and it always comes up empty.

In any case I'll start chopping up the dataset to see if there's anything specific in it that is causing this problem.

[abelio]

Hi Neon,

I guess that dataset is ok, but that chart is not.

 Try attached chart for that dataset.

I've verified it  against my logs and it seemed go well

hope it helps

[neonbit]

Thanks Abel, I've tried that chart but no luck.

The problem is that the dataset shows no data. When you test this dataset does it get populated correctly?

I have a feeling I'm missing some configuration on the FortiGate that would allow this dataset to work, but can't seem to find it.