Fortigate 6.0.3 Traffic Log Analysis via Syslog Messages

[janderson133]

Hello,

I'm trying to summarize the amount of data used by an application, host, etc. over a given period of time.

I'm sysloggging to Splunk and seeing the following problem: long running sessions are logged every 2 minutes with the cumulative amount of data (bytes sent/received/total) up to that point.

So... for example an session that lasted an hour would have 30+ syslog messages.  Now say the total data in the last syslog show 8MB sent, 2MB received, and a total of 10MB.  The syslog before might be 7.5MB sent, 1.9MB received, and a total of 9.5MB.  A simple search utilizing SUM produces a result that is like 200MB for the 10MB session (because the last syslog message when the session terminated has the complete data for the session)

I tried to filter based on action (action != accept), but the results are like 1/3 of the actual bandwidth so that doesn't really work.

Tried the splunk fortigate app and the data model appears to have the same problem I'm describing (but maybe I'm wrong).

Anyway, hoping someone else has already solved this.

Thanks!

Jeff

[Frosty · Accepted solution]

Hi Jeff,

I've encountered a similar (in fact, I think the same) issue here:

https://forum.fortinet.com/tm.aspx?m=168332

Because I'm reporting via the Fortianalyzer directly, not using syslog data and Splunk, I came up with a workaround; see that thread for details.  It seems there is a different LogID value for the "open sessions" (everything up to but not including the final syslog entry when a session gets closed).  I was able to exclude this data from my reports.  Whether you can do anything similar with syslog data in Splunk is doubtful?!

Steve

[janderson133]

This Splunk search probably isn't efficient, but does what I need it to do (the most recent syslog for a given sessionid is used to extract the total bytes and application -> I then total bytes based on application and finally sort based on largest amount of data):

eventtype=ftnt_fgt_traffic | stats latest(bytes) as bytes, latest(app) as app by sessionid | eval GigaBytes=bytes/1024/1024/1024 | stats  sum(GigaBytes) as Total_GigaBytes by app | sort  -Total_GigaBytes

This probably isn't efficient in Splunk terms, but better than nothing for right now.  Hope someone else has advice too :)

[Frosty · Accepted solution]

Hi Jeff,

I've encountered a similar (in fact, I think the same) issue here:

https://forum.fortinet.com/tm.aspx?m=168332

Because I'm reporting via the Fortianalyzer directly, not using syslog data and Splunk, I came up with a workaround; see that thread for details.  It seems there is a different LogID value for the "open sessions" (everything up to but not including the final syslog entry when a session gets closed).  I was able to exclude this data from my reports.  Whether you can do anything similar with syslog data in Splunk is doubtful?!

Steve

[janderson133 · Accepted solution]

Thanks Frosty - that worked too:

eventtype=ftnt_fgt_traffic logid != "0000000020" | eval GigaBytes=bytes/1024/1024/1024 | stats  sum(GigaBytes) as Total_GigaBytes by app | sort  -Total_GigaBytes

I would say the only draw back is long running sessions aren't included in the report.  But... long running sessions could provide false data since a session that has run for days would be included in a 1 hour report and thus skew the results. 

[Frosty]

Hey, nice work!

You're right, not necessarily 100% accurate, but maybe close enough.

My reading of it is that a long-running session that is not yet closed would be excluded.

But a long-running session that has been closed will be included, because the LogID will have been changed on the final log entry (?).

I suppose eventually they'll fix this by only logging the difference (instead of cumulative traffic) every 2 mins.