When is 5.4.1 going to drop?

[ITGuy11]

Is there an ETA as to when 5.4.1 is going to drop?  I have a brand new 300D that I am waiting to put into production as soon as 5.4.1 is ready.

[FGTuser · Accepted solution]

by end of next week (April 15)

[seadave]

This is very interesting.  I'm running 5.4 on a 500D.  Was rock solid from early March until 6/12/16 ~15:00 PDT.  Firewall entered conserve mode and required reboot.  SNMP logging showed rapid spike in CPU and MEM utilization right before this.  Further SYSLOG data shows what I think is IPS crashing?

date=2016-06-28,time=09:05:41,devname=xxx,devid=xxx,logid=0100032546,type=event,subtype=system,level=warning,vd=root,logdesc="Application crashed",action=crash,msg="Pid: 09370, application: , Firmware: FortiGate-500D v5.4.0,build1011b1011,151221 (GA) (Release), Signal 11 received, Backtrace: [0x7fc6580a3118] [0x7fc6580c1239] [0x7fc6580ccbb4] [0x7fc6580ccd51] [0x7fc6580cddaa] [0x7fc658073c3e] [0x7fc65808613e] [0x7fc658052920] [0x00cc1eb6] [0x00cc2397] [0x00cc2650] [0x00cc4845] [0x0043d3f0] [0x00cb119a] [0x00cb1e48] [0x0043d3f0] [0x00443557] [0x00441520] [0x00443188] [0x0043a917] [0x7fc65b2ae475] [0x0043a9f1]"

We have resorted to nightly scheduled reboots of the firewall to mitigate.  I have ticket 1787724 attempting to resolve the issue.

I had noticed that when we went from 5.2 to 5. 4 that my IPS log essentially stopped.  Makes me wonder if IPS has been failing this entire time.  Will not be amused if that is the case.

 

[MikePruett]

Seadave, is your environment a single 500D or is it an HA cluster. Man it would suck to do scheduled nightly reboots.

[seadave]

Fortunately (or unfortunately) no, we don't have HA.  I've always been intimidated by my perceived complexity of HA on Fortinet.  I also have the impression that there are lot of problems with it based on what I read in the forums here.  We have two 500Ds.  I keep one in production and I keep the other for backup/testing so I can swap with a config backup (with I make religiously) if necessary.  I fortunately work at a company where one hour of downtime would be annoying but not the end of the world.  People are (in my opinion) unreasonably tolerate here when it comes to downtime (which honestly is a rare event).  I'm very thankful as it makes my job easier.

I had previously only logged to our FAZVM.  I'm seeing now that watching only traffic and not kernel level SYSLOG events has obscured some issues with 5.4 and the health of the device.  I now have a separate SYSLOG stream going to our SolarWinds install so I can view only those events. (I just realized this IS redundant because I can easily use Log View/Event/System on the FAZ to view logs and search on "Level!=notice" to achieve the same result).   But if you don't have a FAZ, it is necessary to filter out all of the traffic chaff to be able to easily see such system issues.  I did that as follows:

config log syslogd setting (can be "syslogd2" if you already have a traffic syslog enabled)     set status enable     set server "123.123.123.123"     set csv enable (this can also be "disable")     set facility kernel end config log syslogd filter (needs to match the syslogd# above)     set severity warning     set forward-traffic disable     set multicast-traffic disable     set sniffer-traffic disable     set voip disable     set filter "event-level(warning)" end

I continue to see this "App Crash" every 30 minutes or so.

date=2016-06-28,time=10:31:15,devname=xxx,devid=xxx,logid=0100032546,type=event,subtype=system,level=warning,vd=root,logdesc="Application crashed",action=crash,msg="Pid: 09371, application: , Firmware: FortiGate-500D v5.4.0,build1011b1011,151221 (GA) (Release), Signal 11 received, Backtrace: [0x7fc65813a235] [0x7fc6580a30a5] [0x7fc6580a9865] [0x7fc6580c1239] [0x7fc6580cd0a3] [0x7fc6580cddaa] [0x7fc658073c3e] [0x7fc65808613e] [0x7fc658052920] [0x00cc1eb6] [0x00cc2397] [0x00cc2650] [0x00cc4845] [0x0043d3f0] [0x00cb119a] [0x00cb1e48] [0x0043d3f0] [0x00443557] [0x00441520] [0x00443188] [0x0043a917] [0x7fc65b2ae475] [0x0043a9f1]"

date=2016-06-28,time=10:04:50,devname=xxx,devid=xxx,logid=0100032546,type=event,subtype=system,level=warning,vd=root,logdesc="Application crashed",action=crash,msg="Pid: 26690, application: ࢲ, Firmware: FortiGate-500D v5.4.0,build1011b1011,151221 (GA) (Release), Signal 11 received, Backtrace: [0x7fc6580a3118] [0x7fc6580c1239] [0x7fc6580ccc82] [0x7fc6580ccd51] [0x7fc6580cddaa] [0x7fc658073c3e] [0x7fc65808613e] [0x7fc658052920] [0x00cc1eb6] [0x00cc2397] [0x00cc2650] [0x00cc4845] [0x0043d3f0] [0x00cb119a] [0x00cb1e48] [0x0043d3f0] [0x00443557] [0x00441520] [0x00443188] [0x0043a917] [0x7fc65b2ae475] [0x0043a9f1]"

Apparently I'm also seeing "session clashes".  Not sure if those are also a new problem or something that occurs naturally (I sanitized my IPs in the below example "x.x")

New Statusstate=00112386 tuple-num=3 policyid=46 identidx=0 dir=0 act=1 hook=4 x.x.11.34:51321->x.x.3.25:53(x.x.138.66:51321) dir=1 act=2 hook=0 x.x.3.25:53->x.x.138.66:51321(x.x.11.34:51321) dir=1 act=0 hook=4 x.x.3.25:53->x.x.11.34:51321(0.0.0.0:0) Old Statusstate=00004204 tuple-num=2 policyid=0 identidx=0 dir=0 act=0 hook=3 x.x.138.66:51321->x.x.3.25:53(0.0.0.0:0) dir=1 act=0 hook=1 x.x.3.25:53->x.x.138.66:51321(0.0.0.0:0)

I wonder how many other 5.4 users are having this problem without realizing it?  Our 500D historically barely breaks a sweat.  CPU is always less than 5% and MEM hovers around 30%.

I do have nearly all scanning services enabled with the exception of SPAM.  I'm doing Proxy based AV, IDS, IPS, SSL DPI, WAF, and Web Fiters.  I have about 63 rules on the device and we are using FSSO.  We are IP4 only with one ADOM.  Also no VPN, but we are working towards getting a FAC200D up and running with Fortinet FortiClient IPSec/SSL VPN with 2FA via Fortitoken Mobile, but we are still testing that.

[seadave]

Actually at 3:45AM, the schedule reboot feature in 5.4 works like a champ.  Takes less than a minute to reboot and no one seems to notice.

config system global

     set daily-restart enable

     set restart-time 03:45

end

[MikePruett]

Very interesting. I guess it is a workable solution if you don't have any tasks taking place during those times. I have a lot of external vaulting etc taking place during that time so I would be nuked if I had a similar issue!

[seadave]

Yeah, no good if you have a 24 hour online presence or doing backups over the wire.  Still dumping tapes to courier here once a week after nightly D2D.  Kind of a pain but it works.

[seadave]

Your website is great by the way.  Will give it a good look over.

[MikePruett]

Thanks! A lot of it is rehashed Fortinet documentation while I get my camera setup for video creation and what not. I try to pull everything I learn and make sure it is on there as well.

[kallbrandt]

Hi,

New forum user here... I have been following this thread for a while with great interest.

Sorry to hear you are experiencing huge problems with the 5.4 code seadave. It has to my experience always been a bit random between the models when it comes to new FortiOS versions - Some works great, other doesn't. We've had a lot of problems with 100D and new code (5.2.0-3 was a nightmare...), almost no problems except for a few minor bugs with the 600D models and up, or with 90D and down. Only one of our clients run 5.4 in production (their own choice), on 200D HA in A-A-mode. It works as it should, Rock solid for some 4 months now. Their old 211B died the same day I was going to setup the new cluster, but that's another story.... :)

Do a "diagnose sys session stat" to get the total number of clashes. If you do drown in them, you might experience NAT exhaustion - Too many sessions passing through 1 NAT address is the first example that comes to mind.

The number of session clashes should ideally be 0. If you are doing NAT with many thousands of simultaneous sessions going out on one public IP, this can be a real problem since the amount of ports that can be allocated is limited (60416 for TCP-sessions in a Fortigate running 5.2.x, same for UDP).

I use the loadbalancer feature in the Fortigate (800C HA A-A-cluster, running 5.2.7) at one of our clients. It runs in a vdom that doesn't do anything else, so it acts as an appliance so to speak. It sits in front of their Exchange 2013 CAS-servers, over 2,000 clients using Outlook. It works as a charm, but generates quite a few "session clash" entries in the log due to the intense use of NAT... No user inpact that I know of though.

[seadave]

@kallbrandt

Great info, thanks.  I've sent them quite a few logs.  Will look into this deeper.  We don't have that many users hitting it so it shouldn't be an issue.  I do see persistant "Application crashed" messages in Syslog.  Will update when I know more.