Support Forum
The Forums are a place to find answers on a range of Fortinet products from peers and product experts.
ITGuy11
New Contributor

When is 5.4.1 going to drop?

Is there an ETA as to when 5.4.1 is going to drop?  I have a brand new 300D that I am waiting to put into production as soon as 5.4.1 is ready.

2 Solutions
FGTuser
New Contributor III

by end of next week (April 15)

View solution in original post

kallbrandt

That amount of clashes is nothing to worry about I'd say. On the LB-vdom I mentioned earlier the log shows 6-digit amounts of clashes. The clash counter is reset at reboot btw, and is not related to the current amount of sessions. It is just an ongoing counter.

 

To my knowledge, all restarts of applications with restart option 11 (segmentation fault) in FortiOS is seen as a crash. It doesn't have to mean anything bad per se. The OS recycles processes all the time using option 15 (graceful restart). When that doesn't work, it moves on to try to restart with option 11 wich will generate a log entry in the syslog. The recycle process continues all the time, buffers needs to be cleared etc etc. However, a constant restarting of the same application can also mean various problems - Memory leaks, buffer overflows etc.

 

I checked your log, but I can't see anything else then the PID and some weird ASCII-signs as application name. It does look kinda odd.

 

Check your logs and keep track of if the application crash log entries correlates with odd behaviour in the firewall, we're talking sudden reboots, functions and features stopping/not working.

 

What does "diagnose debug crashlog read" say?

 

Also, do a  "diagnose sys top", a few times during the day. Do you have processes in Z or D state?

 

Richie

NSE7

View solution in original post

Richie NSE7
103 REPLIES 103
seadave

This is very interesting.  I'm running 5.4 on a 500D.  Was rock solid from early March until 6/12/16 ~15:00 PDT.  Firewall entered conserve mode and required reboot.  SNMP logging showed rapid spike in CPU and MEM utilization right before this.  Further SYSLOG data shows what I think is IPS crashing?

date=2016-06-28,time=09:05:41,devname=xxx,devid=xxx,logid=0100032546,type=event,subtype=system,level=warning,vd=root,logdesc="Application crashed",action=crash,msg="Pid: 09370, application: , Firmware: FortiGate-500D v5.4.0,build1011b1011,151221 (GA) (Release), Signal 11 received, Backtrace: [0x7fc6580a3118] [0x7fc6580c1239] [0x7fc6580ccbb4] [0x7fc6580ccd51] [0x7fc6580cddaa] [0x7fc658073c3e] [0x7fc65808613e] [0x7fc658052920] [0x00cc1eb6] [0x00cc2397] [0x00cc2650] [0x00cc4845] [0x0043d3f0] [0x00cb119a] [0x00cb1e48] [0x0043d3f0] [0x00443557] [0x00441520] [0x00443188] [0x0043a917] [0x7fc65b2ae475] [0x0043a9f1]"

 

We have resorted to nightly scheduled reboots of the firewall to mitigate.  I have ticket 1787724 attempting to resolve the issue.

 

I had noticed that when we went from 5.2 to 5. 4 that my IPS log essentially stopped.  Makes me wonder if IPS has been failing this entire time.  Will not be amused if that is the case.

 

 
MikePruett
Valued Contributor

Seadave, is your environment a single 500D or is it an HA cluster. Man it would suck to do scheduled nightly reboots.

Mike Pruett Fortinet GURU | Fortinet Training Videos
seadave

Fortunately (or unfortunately) no, we don't have HA.  I've always been intimidated by my perceived complexity of HA on Fortinet.  I also have the impression that there are lot of problems with it based on what I read in the forums here.  We have two 500Ds.  I keep one in production and I keep the other for backup/testing so I can swap with a config backup (with I make religiously) if necessary.  I fortunately work at a company where one hour of downtime would be annoying but not the end of the world.  People are (in my opinion) unreasonably tolerate here when it comes to downtime (which honestly is a rare event).  I'm very thankful as it makes my job easier.

 

I had previously only logged to our FAZVM.  I'm seeing now that watching only traffic and not kernel level SYSLOG events has obscured some issues with 5.4 and the health of the device.  I now have a separate SYSLOG stream going to our SolarWinds install so I can view only those events. (I just realized this IS redundant because I can easily use Log View/Event/System on the FAZ to view logs and search on "Level!=notice" to achieve the same result).   But if you don't have a FAZ, it is necessary to filter out all of the traffic chaff to be able to easily see such system issues.  I did that as follows:

 

config log syslogd setting (can be "syslogd2" if you already have a traffic syslog enabled)     set status enable     set server "123.123.123.123"     set csv enable (this can also be "disable")     set facility kernel end config log syslogd filter (needs to match the syslogd# above)     set severity warning     set forward-traffic disable     set multicast-traffic disable     set sniffer-traffic disable     set voip disable     set filter "event-level(warning)" end

 

I continue to see this "App Crash" every 30 minutes or so.

 

date=2016-06-28,time=10:31:15,devname=xxx,devid=xxx,logid=0100032546,type=event,subtype=system,level=warning,vd=root,logdesc="Application crashed",action=crash,msg="Pid: 09371, application: , Firmware: FortiGate-500D v5.4.0,build1011b1011,151221 (GA) (Release), Signal 11 received, Backtrace: [0x7fc65813a235] [0x7fc6580a30a5] [0x7fc6580a9865] [0x7fc6580c1239] [0x7fc6580cd0a3] [0x7fc6580cddaa] [0x7fc658073c3e] [0x7fc65808613e] [0x7fc658052920] [0x00cc1eb6] [0x00cc2397] [0x00cc2650] [0x00cc4845] [0x0043d3f0] [0x00cb119a] [0x00cb1e48] [0x0043d3f0] [0x00443557] [0x00441520] [0x00443188] [0x0043a917] [0x7fc65b2ae475] [0x0043a9f1]"

 

date=2016-06-28,time=10:04:50,devname=xxx,devid=xxx,logid=0100032546,type=event,subtype=system,level=warning,vd=root,logdesc="Application crashed",action=crash,msg="Pid: 26690, application: ࢲ, Firmware: FortiGate-500D v5.4.0,build1011b1011,151221 (GA) (Release), Signal 11 received, Backtrace: [0x7fc6580a3118] [0x7fc6580c1239] [0x7fc6580ccc82] [0x7fc6580ccd51] [0x7fc6580cddaa] [0x7fc658073c3e] [0x7fc65808613e] [0x7fc658052920] [0x00cc1eb6] [0x00cc2397] [0x00cc2650] [0x00cc4845] [0x0043d3f0] [0x00cb119a] [0x00cb1e48] [0x0043d3f0] [0x00443557] [0x00441520] [0x00443188] [0x0043a917] [0x7fc65b2ae475] [0x0043a9f1]"

 

Apparently I'm also seeing "session clashes".  Not sure if those are also a new problem or something that occurs naturally (I sanitized my IPs in the below example "x.x")

 

New Statusstate=00112386 tuple-num=3 policyid=46 identidx=0 dir=0 act=1 hook=4 x.x.11.34:51321->x.x.3.25:53(x.x.138.66:51321) dir=1 act=2 hook=0 x.x.3.25:53->x.x.138.66:51321(x.x.11.34:51321) dir=1 act=0 hook=4 x.x.3.25:53->x.x.11.34:51321(0.0.0.0:0) Old Statusstate=00004204 tuple-num=2 policyid=0 identidx=0 dir=0 act=0 hook=3 x.x.138.66:51321->x.x.3.25:53(0.0.0.0:0) dir=1 act=0 hook=1 x.x.3.25:53->x.x.138.66:51321(0.0.0.0:0)

 

I wonder how many other 5.4 users are having this problem without realizing it?  Our 500D historically barely breaks a sweat.  CPU is always less than 5% and MEM hovers around 30%.

 

I do have nearly all scanning services enabled with the exception of SPAM.  I'm doing Proxy based AV, IDS, IPS, SSL DPI, WAF, and Web Fiters.  I have about 63 rules on the device and we are using FSSO.  We are IP4 only with one ADOM.  Also no VPN, but we are working towards getting a FAC200D up and running with Fortinet FortiClient IPSec/SSL VPN with 2FA via Fortitoken Mobile, but we are still testing that.

seadave

Actually at 3:45AM, the schedule reboot feature in 5.4 works like a champ.  Takes less than a minute to reboot and no one seems to notice.

 

config system global

     set daily-restart enable

     set restart-time 03:45

end

 

MikePruett
Valued Contributor

Very interesting. I guess it is a workable solution if you don't have any tasks taking place during those times. I have a lot of external vaulting etc taking place during that time so I would be nuked if I had a similar issue!

Mike Pruett Fortinet GURU | Fortinet Training Videos
seadave

Yeah, no good if you have a 24 hour online presence or doing backups over the wire.  Still dumping tapes to courier here once a week after nightly D2D.  Kind of a pain but it works.

seadave

Your website is great by the way.  Will give it a good look over.

 

MikePruett
Valued Contributor

Thanks! A lot of it is rehashed Fortinet documentation while I get my camera setup for video creation and what not. I try to pull everything I learn and make sure it is on there as well.

Mike Pruett Fortinet GURU | Fortinet Training Videos
kallbrandt

Hi,

New forum user here... I have been following this thread for a while with great interest.

 

Sorry to hear you are experiencing huge problems with the 5.4 code seadave. It has to my experience always been a bit random between the models when it comes to new FortiOS versions - Some works great, other doesn't. We've had a lot of problems with 100D and new code (5.2.0-3 was a nightmare...), almost no problems except for a few minor bugs with the 600D models and up, or with 90D and down. Only one of our clients run 5.4 in production (their own choice), on 200D HA in A-A-mode. It works as it should, Rock solid for some 4 months now. Their old 211B died the same day I was going to setup the new cluster, but that's another story.... :)

 

Do a "diagnose sys session stat" to get the total number of clashes. If you do drown in them, you might experience NAT exhaustion - Too many sessions passing through 1 NAT address is the first example that comes to mind.

 

The number of session clashes should ideally be 0. If you are doing NAT with many thousands of simultaneous sessions going out on one public IP, this can be a real problem since the amount of ports that can be allocated is limited (60416 for TCP-sessions in a Fortigate running 5.2.x, same for UDP).

 

I use the loadbalancer feature in the Fortigate (800C HA A-A-cluster, running 5.2.7) at one of our clients. It runs in a vdom that doesn't do anything else, so it acts as an appliance so to speak. It sits in front of their Exchange 2013 CAS-servers, over 2,000 clients using Outlook. It works as a charm, but generates quite a few "session clash" entries in the log due to the intense use of NAT... No user inpact that I know of though.

Richie

NSE7

Richie NSE7
seadave

@kallbrandt

Great info, thanks.  I've sent them quite a few logs.  Will look into this deeper.  We don't have that many users hitting it so it shouldn't be an issue.  I do see persistant "Application crashed" messages in Syslog.  Will update when I know more.

Announcements

Select Forum Responses to become Knowledge Articles!

Select the “Nominate to Knowledge Base” button to recommend a forum post to become a knowledge article.

Labels
Top Kudoed Authors