600C recurring crash

[billp]

I have a 600C with a recurring crash problem.  It is running 5.0.11, but the problem has persisted since 5.0.4 or so.

About every 1-3 months, the gui will show that all interfaces are connected at 1GBPS and that each port has sent/received approximately 191 billion packets if I over over a port (not shown below). 

Once the gui reaches this state, it is usually about 2-3 days before the firewall stops working altogether.

At that point, I need to do a full power down/power off reset. If I just do a warm boot, it will not recover and will stop mid-boot with an error message.

Has anyone experienced anything similar with a 600C? I am beginning to think this is a hardware issue.

Suggestions welcome. Thanks.

[FortiAdam · Accepted solution]

I would run a HQIP test on it to begin with if you haven't done that yet.  http://cookbook.fortinet....ip-test-documentation/

[ede_pfau · Accepted solution]

Bill,

I have opened a support case in August following your post. FTNT reported back that there might be a compatibility issue of the memory modules employed. As I suggested a BIOS upgrade the supporter stated that they have already included a fix into the next firmware (v5.2.4). As my customer (and me) didn't agree to update he sent me a special build based on v5.0.12 (build 8408 instead of b305).

After installing and now 35 days uptime everything looks OK. But then again, what is 35 days compared to the usual 100+ days it took to run the 600C against the wall in the past. I will have to watch and wait to see if the fix really is working. Meanwhile, maybe it's worth asking support for the special build and see if that helps in your situation.

[FortiAdam · Accepted solution]

I would run a HQIP test on it to begin with if you haven't done that yet.  http://cookbook.fortinet....ip-test-documentation/

[billp]

FWIW, tech support said this is a known bug (ID 0243461) and is expected to be fixed in FortiOS 5.2.4.

I am trying to find out if it will also be addressed in the 5.0 branch which is still supported until November, 2015.

If no other 600C users are seeing this issue, though, I might try the HQIP. Does anyone know how long it takes to run on a 4GB 600C?

Thanks.

[FortiAdam]

I have been told that the 600c has memory leak issues in both 5.0 and 5.2.  We have two 600c's running 5.0.7 and notice memory leaking but not quite as bad as what you have observed.

The HQIP test shouldn't take more than a few minutes.  You can skip the interface tests and any others that are unrelated to your issue.

[Dave_Hall]

If there is memory leak, as a temp workaround and if you can afford the brief down-time, maybe try a daily-restart.

config system global
set daily-restart {enable | disable}
set restart-time <hh:mm>
end

Edit: considering the hardware involve, a daily reboot is a bit much -- Perhaps maybe something with scripting can be done to simulate the same restart function but on weekly or semi-weekly basis.  (Still, bug should be fixed.)

[billp]

Thanks. I'll give the HQIP a shot when I can afford some downtime.

For me, the problem has definitely gotten worse in later versions of 5.0. 

It used to happen once every 3-6 months when I had 5.0.4 - 5.0.7.

With 5.0.11, it happened within 35 days of a full reboot. I can't even fix it remotely -- I have to do a cold restart to fix.

Given that tech support says this is a reported problem, I'm not too confident this is a hardware glitch. 

We are not even pushing this unit too hard.

[billp]

I thought I would post a follow-up to this problem in case anyone else has encountered it.

There is a major issue with earlier 600C's that have the original BIOS version of 04000010. The bug ID is 0229553.

Credit to this Reddit post: https://www.reddit.com/r/fortinet/comments/395l8q/all_interfaces_down_fortigate_600c/

The problem appears to be a hardware issue that can only be remedied by RMA'ing the affected box to get one with a newer BIOS version.  Another poster said that this has affected some other models as well. A fix is supposedly in the works for FortiOS 5.2.4 or later.

My 600C crashed about 1 month with this problem, although it was dependent on the amount of traffic and specific FortiOS version.

Anyone else run into this? 

[ede_pfau]

I'm afraid I have.

Running a 600C without any UTM, just firewalling. Running 5.0.9, but error occurred in  4.3.18 as well.

BIOS version: 04000009

After 100 days or so, all ports are shown "link up" in the Unit Op widget, although some are not connected at all.

Traffic usage widgets show identical graphs no matter which interface. Traffic looks "cut off" at 20 Gbps.

This has nothing to do with reality. First, we thought this was just a cosmetic problem. Then, traffic stopped.

Only a hard power cycle will bring the FGT back into operation.

Upgrading didn't help.

If FTNT really fixes this bug in v5.2 only we'll have to upgrade. Customer will only agree to that if a fix is guaranteed. As this firewall is protecting permanent data sources it's quite hard to find a window for downtime.

Bill, would you recommend a hardware exchange, or wait for a fix?

[billp]

ede_pfau wrote:

Upgrading didn't help.

If FTNT really fixes this bug in v5.2 only we'll have to upgrade. Customer will only agree to that if a fix is guaranteed. As this firewall is protecting permanent data sources it quite hard to find a window for downtime.

Bill, would you recommend a hardware exchange, or wait for a fix?

I am opting for a hardware exchange. Fortinet sent me a new-ish 600c overnight with a recent firmware. Date of manufacture was October, 2014. I am going to replace my box this weekend. 

I am guessing that this is not an easy software fix/workaround, or they would have included it in an earlier FortiOS revision. I have had this particular problem since installing my 600c about 18 months ago, but it would sometimes take 3-4 months to surface. More recently, it's been about every 30 days.

In my opinion, these units with older firmware should be recalled. That's effectively what support is doing anyway. 

Please let us know how it goes. 

Update: Tech support said it's possible to do an in-place firmware upgrade on the BIOS, but it's not simple. Not sure how complex it is, but that might be an option where you don't want to swap out the equipment. 

[billp]

Well, all is not well, even with a relatively new 600C from Fortinet with a recent firmware.

Has anyone had this issue resolved yet?

At this point, it appears that there is a serious bug in the 600C firmware with no real resolution.

The earlier-mentioned Reddit posting made it seem like there were other models that were affected, so I am curious if there are other Fortigates in the C hardware revision that also have this issue.

If someone has some wisdom to share on this, please feel free to jump in.