600C recurring crash

[billp]

I have a 600C with a recurring crash problem.  It is running 5.0.11, but the problem has persisted since 5.0.4 or so.

About every 1-3 months, the gui will show that all interfaces are connected at 1GBPS and that each port has sent/received approximately 191 billion packets if I over over a port (not shown below). 

Once the gui reaches this state, it is usually about 2-3 days before the firewall stops working altogether.

At that point, I need to do a full power down/power off reset. If I just do a warm boot, it will not recover and will stop mid-boot with an error message.

Has anyone experienced anything similar with a 600C? I am beginning to think this is a hardware issue.

Suggestions welcome. Thanks.

[FortiAdam · Accepted solution]

I would run a HQIP test on it to begin with if you haven't done that yet.  http://cookbook.fortinet....ip-test-documentation/

[ede_pfau · Accepted solution]

Bill,

I have opened a support case in August following your post. FTNT reported back that there might be a compatibility issue of the memory modules employed. As I suggested a BIOS upgrade the supporter stated that they have already included a fix into the next firmware (v5.2.4). As my customer (and me) didn't agree to update he sent me a special build based on v5.0.12 (build 8408 instead of b305).

After installing and now 35 days uptime everything looks OK. But then again, what is 35 days compared to the usual 100+ days it took to run the 600C against the wall in the past. I will have to watch and wait to see if the fix really is working. Meanwhile, maybe it's worth asking support for the special build and see if that helps in your situation.

[ede_pfau · Accepted solution]

Bill,

I have opened a support case in August following your post. FTNT reported back that there might be a compatibility issue of the memory modules employed. As I suggested a BIOS upgrade the supporter stated that they have already included a fix into the next firmware (v5.2.4). As my customer (and me) didn't agree to update he sent me a special build based on v5.0.12 (build 8408 instead of b305).

After installing and now 35 days uptime everything looks OK. But then again, what is 35 days compared to the usual 100+ days it took to run the 600C against the wall in the past. I will have to watch and wait to see if the fix really is working. Meanwhile, maybe it's worth asking support for the special build and see if that helps in your situation.

[billp]

Ede,

Thanks for the suggestion. I'll contact support to see what they say. It would be nice to have a stable 5.0.12.

5.2.4 has not received the greatest reception here on the forums, so I am not ready to jump into that hoping that it fixes the problem.

Bad memory modules would make sense given that it needs a hard power reset in order to fix the issue.

It's extremely frustrating that Fortinet has not been more forthcoming about this problem.

[ede_pfau]

So you can understand my surprise that the FTNT supporter didn't need much time to suggest the hotfix. Seems to be known now internally, in contrast to last August when I had opened a call for the same issue.

[vjoshi_FTNT · Accepted solution]

Hello, As already mentioned earlier, this is a known issue (bug :0243461) and there is a special build released in both V5.0 and V5.2. Also, there is a workaround of bringing DOWN the admin status of all the UNUSED interfaces should prevent the issue to re-occur.

Hope that helps.

billp wrote:
Well, all is not well, even with a relatively new 600C from Fortinet with a recent firmware.   Has anyone had this issue resolved yet?   At this point, it appears that there is a serious bug in the 600C firmware with no real resolution.   The earlier-mentioned Reddit posting made it seem like there were other models that were affected, so I am curious if there are other Fortigates in the C hardware revision that also have this issue.   If someone has some wisdom to share on this, please feel free to jump in.    

[billp]

vJoshi,

Thanks for the reply.  The work-around you mentioned is a life-saver. 

It would be extremely helpful if Fortinet could publish more information about the BIOS bug and bug 0243461. Tech support originally led me to believe that the BIOS bug was the sole problem I was having. Without access to the exact description, it's hard to help tech support pinpoint the exact set of symptoms and problems. A brief KB on this issue would be really appreciated.

Thanks again for your help.

vjoshi wrote:

Hello, As already mentioned earlier, this is a known issue (bug :0243461) and there is a special build released in both V5.0 and V5.2. Also, there is a workaround of bringing DOWN the admin status of all the UNUSED interfaces should prevent the issue to re-occur.

Hope that helps.

[EricG1793]

Hello,For what it's worth, in July, I had contacted Fortinet about advice for upgrading. I also mentioned my symptom of sporadic freezing during normal operation and/or hanging during reboots, citing 0243461 and 0229553. We've had no issues since we upgraded the firmware from 5.0.4 to 5.0.12, however, I want to make sure it stays that way. I requested either a fix or an RMA, but at that time, they had no official solution and could not do an RMA. Last week, the ticket was re-opened and they asked me if I wanted to upgrade the BIOS myself or RMA the unit. I replied and asked how simple the BIOS upgrade is and I'm waiting to hear back, but I have a feeling I'll elect to RMA. Just thought I'd let you all know! As frustrating as it's been having practically no information from Fortinet on the issue, I am happy that they did follow up and offer a solution.

- Eric

[mscheiber]

Hi

i dont know if i am allowed to share the BIOS update instructions i received it from FTNT Support so if you say i can share this document i will.

[billp]

I didn't do the BIOS update, but my recommendation would be to take the RMA. 

If the BIOS update fails for some reason, you could end up with a bricked firewall. 

With an RMA, you could do all the prep-work during working hours and then schedule a quick swap during off-hours. If there are any issues, you have the reassurance of having the old firewall on-hand until you ship it back.

Just my 2cents.

[ede_pfau]

@mscheiber: I think without the BIOS file you can't do any harm, and this is only available through Support. I'd be keen on knowning about the complexity (or simplicity) of the process to be able to decide which way I'd go - BIOS update or "just" firmware update.

During my latest call on this behalf, Support stated that this issue can be fixed by a FW update, apparently patching up a BIOS bug...from my guts, I'd rather patch the BIOS.

[mscheiber]

[[2. Download HQIP image (FGT_600C-HQIP.2.3.3.2339.out) and copy it under TFTP root folder[[[[[ [ FortiGate-600C (20:43-08.19.2014)Ver:04000023Serial number:FG600C3913801007RAM activationCPU(00:00020655 bfebfbff): MP initializationCPU(01:00020655 bfebfbff): MP initializationCPU(04:00020655 bfebfbff): MP initializationCPU(05:00020655 bfebfbff): MP initializationTotal RAM: 4096MBEnabling cache...Done.Scanning PCI bus...Done.Allocating PCI resources...Done.Enabling PCI resources...Done.Zeroing IRQ settings...Done.Verifying PIRQ tables...Done.Boot up, boot device capacity: 7552MB.Press any key to display configuration menu... ------- Press any key to enter BIOS menu...: Get firmware image from TFTP server.: Format boot device.: Boot with backup firmware and set as default.: Configuration and information.: Quit menu and continue to boot with default firmware.: Display this list of options. Enter Selection : Enter G,F,B,I,Q,or H:G -----Type "G" and enter Please connect TFTP server to Ethernet port "MGMT1".-----Connect cable from TFTP server to "MGMT1" Enter TFTP server address [192.168.1.168]: ---------Enter TFTP server IP hereEnter local address [192.168.1.188]: ---------Enter Local IP hereEnter firmware image file name [image.out]: FGT_600C-HQIP.2.3.3.2339.out ---Enter HQIP image nameMAC:00090FBC1A10 #############################Total 30706008 bytes data downloaded.Verifying the integrity of the firmware image. Total 262144kB unzipped.Save as Default firmware/Backup firmware/Run image without saving:[D/B/R]?R................................................................................................................................................................................................................................................................Reading boot image 2188476 bytes.Initializing firewall...System is started.  FORTITEST/FG600C3913801007 login: admin ------- loginPassword: Test program loading(HQIP, Build2339,Apr 26 2013 10:09:45) ...Engine Version: v1.0 Build 2339. Apr 26 2013 10:09:07 You are running HQIP test program. To start testing, login as "admin" without password, and type:diagnose hqip start Welcome ! FORTITEST/FG600C3913801007 # diag boup start  *******Starting BIOS online update********  Enter Fortigate serial number (FG600C3913801007): FG600C3913804474 Please plug ethernet cable into: mgmt1  : Get BIOS image from FTP server. (default): Get BIOS image from TFTP server.Enter Selection : t ------Choose "t"Enter TFTP server IP address (192.168.1.168):Enter local IP address (192.168.1.188):Getting BIOS image (FG600C/FG600C3913804474.rom) from TFTP server:T[link]FTP://192.168.1.168/FG600C/FG600C3913804474.rom[/link] Press any key to start...Connecting to server... FG600C/FG600C3913804 14% |**** | 595k 0:00:05 ETAFG600C/FG600C3913804 45% |************** | 1883k 0:00:02 ETAFG600C/FG600C3913804 73% |********************** | 3026k 0:00:01 ETAFG600C/FG600C3913804 100% |*******************************| 4096k 0:00:00 ETAFG600C/FG600C3913804 100% |*******************************| 4096k 0:00:00 ETAFG600C/FG600C3913804 100% |*******************************| 4096k 0:00:00 ETADone. 4194304 bytes received The BIOS ROM is ready to be updated. WARNING: DO NOT POEWR OFF THE UNIT DURING BIOS UPDATING!!! This process may take a few minutes.Press Enter when you are ready ... BIOS is updating...flashrom v0.9.2-runknown on Linux 2.4.37 (x86_64), built with libpci 0.0, GCC 3.4.6, little endianflashrom is free software, get the source code at [link]http://www.flashrom.org[/link] Calibrating delay loop... OK.No coreboot table found.sh: dmidecode: not founddmidecode execution unsucessfull - continuing without DMI infoFound chipset "Intel 3450", enabling flash write... OK.This chipset supports the following protocols: FWH,SPI.Found chip "Winbond W25Q32" (4096 KB, SPI) at physical address 0xffc00000.===This flash part has status UNTESTED for operations: PROBE READ ERASE WRITEThe test status of this chip may have been updated in the latest developmentversion of flashrom. If you are running the latest development version,please email a report to flashrom@flashrom.org if any of the above operationswork correctly for you with this flash part. Please include the flashromoutput with the additional -V option for all operations you tested (-V, -Vr,-Vw, -VE), and mention which mainboard or programmer you tested.Thanks for your help!===Flash image seems to be a legacy BIOS. Disabling checks.File's BIOS Fortinet Banner: FortiGate-600C (20:43-08.19.2014)  File's BIOS Product Model: FG600CFile's BIOS Serial Number: FG600C3913804474File's BIOS Version: Ver:04000023File's BIOS HW MAC address: 085b0e33e818File's BIOS OEM Serial Num: FG600C3913804474File's BIOS Licence: aa8a8d778dc82ed8File's BIOS HW Rev/Part Num: 08908-04Writing flash chip... Erasing flash before programming... Erasing flash chip... SUCCESS.done.Programming flash... ################################################################done.COMPLETE.Verifying flash..., (size:0x400000, from:0x0) Rom's BIOS Fortinet Banner: FortiGate-600C (20:43-08.19.2014)  Rom's BIOS Product Model: FG600CRom's BIOS Serial Number: FG600C3913804474Rom's BIOS Version: Ver:04000023Rom's BIOS HW MAC address: 085b0e33e818Rom's BIOS OEM Serial Num: FG600C3913804474Rom's BIOS Licence: aa8a8d778dc82ed8Rom's BIOS HW Rev/Part Num: 08908-04 File's checksum: (0X000000 ~ 0X400000) = 0X312906FBFlash's checksum: (0X000000 ~ 0X400000) = 0X312906FB Correct: File and Flash's verification is OK!  Updating BIOS ROM Done!  FORTITEST/FG600C3913801007 # exe reb --------Reboot the unit This operation will reboot the system !Do you want to continue? (y/n)y     The system is going down NOW !!  System is rebooting... FORTITEST/FG600C3913801007 # Please stand by while rebooting the system. FortiGate-600C (20:43-08.19.2014)Ver:04000023Serial number:FG600C3913804474RAM activationCPU(00:00020655 bfebfbff): MP initializationCPU(01:00020655 bfebfbff): MP initializationCPU(04:00020655 bfebfbff): MP initializationCPU(05:00020655 bfebfbff): MP initializationTotal RAM: 4096MB