CPU maxes out & all traffic stops passing

[#1]

We are using an 800 running 2.8 MR4. On Friday, we were on 2.5 MR8, were having same cpu maxed out problems, so we were told to upgrade. The problem still exists, wondering if anyone else is experiencing same issues. We are a wireless ISP with somewhere around 900+ customers coming through the core. The 800 sits at the core, and all traffic goes through it. When all traffic stops, bossman gets upset, so I need a fix soon!! I have been noticing that it' s been detecting code red viruses, not seeing it in the logs, but in the recent virus detections on system status page. Wondering if it' s really dropping those entries, or if it' s passing it to adjacent routers?? Is anyone else experiencing cpu maxing out?

[UkWizard]

take there specs with a pince of salt (as all manufacturers).

[gregs]

Thanks all for the replies. I did not get the firmware version right (brain fade) I have 2.5 build 269 which is MR9. I have been in touch with my reseller who tells me to upgrade to MR10. I am going to do that tomorrow AM. It' s a PITA now ' cos the box is in production. I am going read the instructions to prepare for the upgrade, any known caveats that I should watch for? Trombone, yes that is what I assumed. I have a po issued for 2 FG1000' s, I have put that on hold ' till this is resolved. 6Mb is nothing, we bought the 3600 and paid the big dollar because eventually we may get a 100Mb connect. Philink, as above got the ver wrong. Who woulda thunk that marketing would be exaggerated. It would be nice to know exactly what it will handle. Wiz, pinch me where? Greg

[#4]

One thing I found was that Fortigate might be able to handle the 100 Mb with a few sessions but not 10 Mb with many sessions. I noticed that when I had many sessions flowing thru Fortigate (4000+), it wolud peak my Fortigate to 98%. I guess when Fortinet spec out their units, they simply have one or two PC pushing a few session with lots of traffic thru Fortigate, instead of simulating with many PCs and many sessions just like in the real world. Just my 2 cents.

[gregs]

FYI: 2.50 299 MR10 fixes the CPU high utilization on my 3600. We did the upgrade to 2.50 299 this morning. There were absolutely no hitches. It did not lose any of the config data and it came back up in transparent mode. Mind you I was prepared, I had my laptop connected thru the console port and also thru Port 1. I also had a copy of the config ready to reload. I will also post this as a new topic. Greg

[#6]

Pasdargent, I would be very weary of using SNMP with v2.5. We have a ticket with Fortinet with our FGT500 with respect to SNMP v2c. Since I don' t have any insight into the FortiGate development this is only what it appears to me what happens when things go bad. First off it apears that the SNMP daemon uses up too much memory and start choking other process. The only way you can see this is by connecting a serial cable to the FGT500. If you do that you' ll see errors spewing out with stuff like processes being killed restarted and killed over and over. The box then starts dropping sessions... more packets get dropped and the only way to recover is to reboot the FGT500. As a work around we now restart the SNMP daemon on FGT500 after every poll. However this only works if the box is not overloaded by say an attack (by using NESSUS http://www.nessus.org/ as an example). Once you attack the box with a few concurrent attacks (like any script kitty would do ) it again keels over in the same way after about 20 minutes. I could understand it if the problem was that it dropped a few sessions when the box was overloaded. That would be expected. What is not expected is that when the load is removed the box does not recover from the crash. Fortinet refuses to fix this problem in v2.5 and keep telling us to move to v2.8. Based on my readings of the forums I don' t think v2.8 is ready for production. Fortnet also says no one else is using SNMP or they would have reported our problem as well so we must be a unique site. I can understand that as a manufacturer I wouldn' t want to spend my precious time dealing with an old release when I need to get a new release functional. My request to you is to if you' re experiencing any of these problems to please contact Fortinet with your issues and maybe if enough of us get together we could get a fix for this issue. Kind regards, SpyderGeer FGT500 v2.5 MR10 Build 299 and interim release Build 315

[UkWizard]

2.8 MR3 had worse memory leaks than all the other versions, MR4 seems better judging by the feedback form the forum users. Try the 2.5 MR10 and see what happens, then 2.8 MR4 after that if it still occurs. if it still occurs after that, i would highly recommend factory resetting the unit and restoring the config (or do it again from scratch, if poss). I have seen in the past where a box does really strange stuff, including freezing up. Caused by a firmware upgrade, seemed to screw the config somehow. Only a factory reset would fix it. Thats a last resort option though.

[#8]

UkWizard, Yeah unfortunately that is what we have done. Factory reset (again and again and again) and change nothing except turning on SNMP on the internal interface (FGT500). When we run SNMP queries against it on build v2.5 299 and 315 the FGT500 experiences these problems. We have tried v2.8 MR3 and experienced the same problems when we load the box to it' s maximum throughput and back off again the FGT500 never seems to recover. I have not tried this same test with v2.8 MR4. I also work in the critical infrastructure industry that will not accept latest builds. So while MR4 is " nice" to look at until it has been through a bank of test it will not be accepted. Thanks, SpyderGeer

[#9]

I have a Fortigate-3000 2.80,build184,040702 I' m currently running 150-180Mb/s with over 300,000 sessions. I have lost track of how many ip' s and subnets that is ;) I' ve noticed that problem too with some AV daemons. You can find the source of you problem with a ' diag sys top' : Run Time: 2 days, 4 hours and 42 minutes 29U, 4S, 66I; 1009T, 533F, 36KF ipsengine 98 R 5.9 0.3 thttp 93 S 0.0 3.4 smtp 52 S 0.0 0.7 httpsd 839 S 0.0 0.7 the process is likely to be thtttp or ftpd. Restart the process with a diag test app <name> 99 for the example above it' d be: diag sys app ipsengine 99 That' ll restart the process and the cpu will plummet. I' ll let you know how MR4 is as soon as we get it in.

[#10]

I' ve had this a few times as well. More than a few... I was at MR8 v250-build251 as of Tuesday of this week and went to v280-build219 based on a call to tech support (seems they wont even look for other issues until you are on the new OS now, at least that is what I was told). Support said it was an HTTP Proxy overflow issue. Once I updated, Wednesday and Thursday were fine. Just this morning, I had the CPU spike to 95-99% and sit there. The only solution I had was to reboot the device. I already have AV turned off on HTTP, but on for other protocols. I have file block on for all protocols. I don' t use the spam filter at all. I have IDS on and the only signature I' ve modified are the four for p2p (bit_torrent, edonkey, gnutella, and kazaa) signatures and I changed them from pass to reset based on the FortiNet support site. I work at a school and we have little control as to what students install, so this is something I would like to keep on. Not really expecting a solution here, just wanted to add my experience to the thread so there is an ongoing list of issues.

[#11]

We may have come up with a resolution. I believe the situation is a compilation of a few different items. If we have all scanning on for HTTP, FTP, POP & SMTP, all IDS anomalies selected & SNMP traps set, we continue to have the CPU spike. If we turn all scanning off, it is fine, or if we turn SNMP traps off it' s fine, but we can' t run both at the same time. Obviously scanning is more important, so we have had to turn off SNMP traps for now, and are not polling the device at all. So far, we' ve had SNMP off for 3 days, and have not had a spike since. We are going to leave it like this for a week to 10 days to see if the problem re-occurs. If it doesn' t, then Fortinet 3rd level support is going to put in a bug report to the developers for this issue. They believe it is too soon at this time to attribute explicitly that SNMP is the culprit, but I believe it is. If you look into other responses on this post, you will see that another customer also had this problem, (which is why we decided to turn it off) and he is on an older build than we are, so it sounds like no build will fix this (we are on 2.8 MR4), which is the newest. Is anyone else out there with this same problem, using SNMP traps and polling their fortinet from a monitoring station? If so, can you try turning it off & see if your problem persists? Also, could you post a response to let the rest of us know?