Looking for information on "diagnose Sys session full-stat" command

[bds38]

Hello,

We have a pretty important FG-500E that is serving several thousand users and recently it has started randomly dropping sessions. It seems to be most noticeable when users are downloading files as the download suddenly halts and because the session has been dropped, the download never recovers. It happens seemingly randomly although we have been seeing a few of session clashes in the logs.

I have been trying to get to the bottom of it and when I run the "diagnose sys session full-stat" command I get an output similar to the following:

session table: table_size=4194304 max_depth=3 used=63751 misc info: session_count=29993 setup_rate=147 exp_count=1 clash=1276 memory_tension_drop=0 ephemeral=0/1113088 removeable=0 npu_session_count=15907 nturbo_session_count=2676 delete=0, flush=0, dev_down=2/229 ses_flush_filters=0 flush_work_num=0 TCP sessions: 1 in NONE state 26293 in ESTABLISHED state 43 in SYN_SENT state 6 in SYN_RECV state 9 in FIN_WAIT state 98 in TIME_WAIT state 421 in CLOSE state 715 in CLOSE_WAIT state firewall error stat: error1=00000000 error2=003be7ea error3=00000000 error4=00000000 tt=00000000 cont=01bbff8f ids_recv=c3b506e3 url_recv=00000000 av_recv=15dcfe56 fqdn_count=0000000a fqdn6_count=00000000

When I contrast the output with any other firewalls we run (we have over 20) the error2=xxxxxx column contains error2=00000000 and there are no clashes. 

Does anyone have any idea what the error counters actually represent? I have found an old article that claims error1 is to do with a mismatch of encryption state but I cannot find any information on any of the other errors.

Does anyone here have any ideas? 

[emnoc]

Clash typically means ephermal ports issues on SNAT, but what I try todo is  setup  session tracking with  snmp. Currently you do not have that many session but you need to start tracking it.

 https://kb.fortinet.com/kb/documentLink.do?externalID=FD37215

if that count is not increasing, I would look elsewhere.

Also since you mention file downloads, are you doing any filtering .If yes can you halt it? Also are upgraded to the latest FortiOS version?

Ken Felix

'

[tioeudes]

You probably facing nat por exaustion. If thats te case there should be an log entry about that. Check on system event logs for the message "NAT port is exhausted"

https://kb.fortinet.com/kb/documentLink.do?externalID=FD33056

[bds38]

Hi Tioeudes,

I have seen that error before, but not on this firewall. We are only seeing session clash errors in the logs and not NAT port exhaustion errors. 

[emnoc]

I haven't see that error b4, but is it climbing ? Also the total number for your platform is minor to say the least. I would really be interested in  UTM and flow/proxy mode inspections

Ken Felix

[bds38]

The error is climbing at a fairly steady pace. It was error2=003beaf2 when I created my post and now its incremented to error2=003bf2eb so although its not going up at an alarming rate, it is definitely a concern.

I failed to answer your other question - we are running 6.0.5 and we were planning to upgrade to 6.0.9 tomorrow until some data center shenanigans made us push it back a week.

We are running in NAT mode and the proxy inspection is using the default settings (all ports enabled and everything else disabled except for "Allow Fragmented Messages" (in emails). UTM wise we are only monitoring and aren't blocking anything on AC/IPS/AV/SSL.

[bds38]

Hi Emnoc,

Thanks for the quick reply. We have Solarwinds NPM and it constantly monitors the session counters on all of our firewalls every 15 seconds or so. Looking at the historical data for the firewall with the problem the sessions seem to peak around 30-35k, but the actual used sessions (from the command output) tends to be roughly double this number.

On our outbound policies we have application control (monitoring) SSL inspection, AV and IPS enabled, but I have tried turning these off without any success. 

I have started to move some of the end users behind a different SNAT address and I think this may be having a positive effect, but due to the random nature of the problem it is hard to get reliable reports from them when it occurs.

Any thoughts on that error2 column?