billp
Contributor

600C recurring crash

I have a 600C with a recurring crash problem.  It is running 5.0.11, but the problem has persisted since 5.0.4 or so.

 

About every 1-3 months, the gui will show that all interfaces are connected at 1GBPS and that each port has sent/received approximately 191 billion packets if I over over a port (not shown below). 

 

 

 

Once the gui reaches this state, it is usually about 2-3 days before the firewall stops working altogether.

 

At that point, I need to do a full power down/power off reset. If I just do a warm boot, it will not recover and will stop mid-boot with an error message.

 

Has anyone experienced anything similar with a 600C? I am beginning to think this is a hardware issue.

 

Suggestions welcome. Thanks.

 

 

Bill ========== Fortigate 600C 5.0.12, 111C 5.0.2 Logstash 1.4.1

3 Solutions
FortiAdam
Contributor II

I would run a HQIP test on it to begin with if you haven't done that yet.  http://cookbook.fortinet....ip-test-documentation/

View solution in original post

ede_pfau
Esteemed Contributor III

Bill,

 

I have opened a support case in August following your post. FTNT reported back that there might be a compatibility issue of the memory modules employed. As I suggested a BIOS upgrade the supporter stated that they have already included a fix into the next firmware (v5.2.4). As my customer (and me) didn't agree to update he sent me a special build based on v5.0.12 (build 8408 instead of b305).

 

After installing and now 35 days uptime everything looks OK. But then again, what is 35 days compared to the usual 100+ days it took to run the 600C against the wall in the past. I will have to watch and wait to see if the fix really is working. Meanwhile, maybe it's worth asking support for the special build and see if that helps in your situation.


Ede

"Kernel panic: Aiee, killing interrupt handler!"

View solution in original post

vjoshi_FTNT
Staff
Staff

Hello, As already mentioned earlier, this is a known issue (bug :0243461) and there is a special build released in both V5.0 and V5.2. Also, there is a workaround of bringing DOWN the admin status of all the UNUSED interfaces should prevent the issue to re-occur.

 

Hope that helps.

billp wrote:
Well, all is not well, even with a relatively new 600C from Fortinet with a recent firmware.   Has anyone had this issue resolved yet?   At this point, it appears that there is a serious bug in the 600C firmware with no real resolution.   The earlier-mentioned Reddit posting made it seem like there were other models that were affected, so I am curious if there are other Fortigates in the C hardware revision that also have this issue.   If someone has some wisdom to share on this, please feel free to jump in.    

View solution in original post

21 REPLIES 21
ede_pfau
Esteemed Contributor III

Bill,

 

I have opened a support case in August following your post. FTNT reported back that there might be a compatibility issue of the memory modules employed. As I suggested a BIOS upgrade the supporter stated that they have already included a fix into the next firmware (v5.2.4). As my customer (and me) didn't agree to update he sent me a special build based on v5.0.12 (build 8408 instead of b305).

 

After installing and now 35 days uptime everything looks OK. But then again, what is 35 days compared to the usual 100+ days it took to run the 600C against the wall in the past. I will have to watch and wait to see if the fix really is working. Meanwhile, maybe it's worth asking support for the special build and see if that helps in your situation.


Ede

"Kernel panic: Aiee, killing interrupt handler!"
billp
Contributor

Ede,

 

Thanks for the suggestion. I'll contact support to see what they say. It would be nice to have a stable 5.0.12.

 

5.2.4 has not received the greatest reception here on the forums, so I am not ready to jump into that hoping that it fixes the problem.

 

Bad memory modules would make sense given that it needs a hard power reset in order to fix the issue.

 

It's extremely frustrating that Fortinet has not been more forthcoming about this problem.

 

 

Bill ========== Fortigate 600C 5.0.12, 111C 5.0.2 Logstash 1.4.1

ede_pfau
Esteemed Contributor III

So you can understand my surprise that the FTNT supporter didn't need much time to suggest the hotfix. Seems to be known now internally, in contrast to last August when I had opened a call for the same issue.


Ede

"Kernel panic: Aiee, killing interrupt handler!"
vjoshi_FTNT
Staff
Staff

Hello, As already mentioned earlier, this is a known issue (bug :0243461) and there is a special build released in both V5.0 and V5.2. Also, there is a workaround of bringing DOWN the admin status of all the UNUSED interfaces should prevent the issue to re-occur.

 

Hope that helps.

billp wrote:
Well, all is not well, even with a relatively new 600C from Fortinet with a recent firmware.   Has anyone had this issue resolved yet?   At this point, it appears that there is a serious bug in the 600C firmware with no real resolution.   The earlier-mentioned Reddit posting made it seem like there were other models that were affected, so I am curious if there are other Fortigates in the C hardware revision that also have this issue.   If someone has some wisdom to share on this, please feel free to jump in.    

billp

vJoshi,

 

Thanks for the reply.  The work-around you mentioned is a life-saver. 

 

It would be extremely helpful if Fortinet could publish more information about the BIOS bug and bug 0243461. Tech support originally led me to believe that the BIOS bug was the sole problem I was having. Without access to the exact description, it's hard to help tech support pinpoint the exact set of symptoms and problems. A brief KB on this issue would be really appreciated.

 

Thanks again for your help.

 

vjoshi wrote:

Hello, As already mentioned earlier, this is a known issue (bug :0243461) and there is a special build released in both V5.0 and V5.2. Also, there is a workaround of bringing DOWN the admin status of all the UNUSED interfaces should prevent the issue to re-occur.

 

Hope that helps.

 

Bill ========== Fortigate 600C 5.0.12, 111C 5.0.2 Logstash 1.4.1

EricG1793

Hello,For what it's worth, in July, I had contacted Fortinet about advice for upgrading. I also mentioned my symptom of sporadic freezing during normal operation and/or hanging during reboots, citing 0243461 and 0229553. We've had no issues since we upgraded the firmware from 5.0.4 to 5.0.12, however, I want to make sure it stays that way. I requested either a fix or an RMA, but at that time, they had no official solution and could not do an RMA. Last week, the ticket was re-opened and they asked me if I wanted to upgrade the BIOS myself or RMA the unit. I replied and asked how simple the BIOS upgrade is and I'm waiting to hear back, but I have a feeling I'll elect to RMA. Just thought I'd let you all know! As frustrating as it's been having practically no information from Fortinet on the issue, I am happy that they did follow up and offer a solution.

- Eric

mscheiber

Hi

 

i dont know if i am allowed to share the BIOS update instructions i received it from FTNT Support so if you say i can share this document i will.

 

 

billp

I didn't do the BIOS update, but my recommendation would be to take the RMA. 

 

If the BIOS update fails for some reason, you could end up with a bricked firewall. 

 

With an RMA, you could do all the prep-work during working hours and then schedule a quick swap during off-hours. If there are any issues, you have the reassurance of having the old firewall on-hand until you ship it back.

 

Just my 2cents.

Bill ========== Fortigate 600C 5.0.12, 111C 5.0.2 Logstash 1.4.1

ede_pfau
Esteemed Contributor III

@mscheiber: I think without the BIOS file you can't do any harm, and this is only available through Support. I'd be keen on knowning about the complexity (or simplicity) of the process to be able to decide which way I'd go - BIOS update or "just" firmware update.

 

During my latest call on this behalf, Support stated that this issue can be fixed by a FW update, apparently patching up a BIOS bug...from my guts, I'd rather patch the BIOS.


Ede

"Kernel panic: Aiee, killing interrupt handler!"
mscheiber

[[2. Download HQIP image (FGT_600C-HQIP.2.3.3.2339.out) and copy it under TFTP root folder

[[[[[ 

FortiGate-600C (20:43-08.19.2014)

Ver:04000023

Serial number:FG600C3913801007

RAM activation

CPU(00:00020655 bfebfbff): MP initialization

CPU(01:00020655 bfebfbff): MP initialization

CPU(04:00020655 bfebfbff): MP initialization

CPU(05:00020655 bfebfbff): MP initialization

Total RAM: 4096MB

Enabling cache...Done.

Scanning PCI bus...Done.

Allocating PCI resources...Done.

Enabling PCI resources...Done.

Zeroing IRQ settings...Done.

Verifying PIRQ tables...Done.

Boot up, boot device capacity: 7552MB.

Press any key to display configuration menu... ------- Press any key to enter BIOS menu

...

: Get firmware image from TFTP server.

: Format boot device.

: Boot with backup firmware and set as default.

: Configuration and information.

: Quit menu and continue to boot with default firmware.

: Display this list of options.

 

Enter Selection :

 

Enter G,F,B,I,Q,or H:G -----Type "G" and enter

 

Please connect TFTP server to Ethernet port "MGMT1".-----Connect cable from TFTP server to "MGMT1"

 

Enter TFTP server address [192.168.1.168]: ---------Enter TFTP server IP here

Enter local address [192.168.1.188]: ---------Enter Local IP here

Enter firmware image file name [image.out]: FGT_600C-HQIP.2.3.3.2339.out ---Enter HQIP image name

MAC:00090FBC1A10

 

#############################

Total 30706008 bytes data downloaded.

Verifying the integrity of the firmware image.

 

Total 262144kB unzipped.

Save as Default firmware/Backup firmware/Run image without saving:[D/B/R]?R

................................................................................................................................................................................................................................................................

Reading boot image 2188476 bytes.

Initializing firewall...

System is started.

 

 

FORTITEST/FG600C3913801007 login: admin ------- login

Password:

 

Test program loading(HQIP, Build2339,Apr 26 2013 10:09:45) ...

Engine Version: v1.0 Build 2339. Apr 26 2013 10:09:07

 

You are running HQIP test program. To start testing, login as "admin" without password, and type:

diagnose hqip start

 

Welcome !

 

FORTITEST/FG600C3913801007 # diag boup start

 

 

*******Starting BIOS online update********

 

 

Enter Fortigate serial number (FG600C3913801007): FG600C3913804474

 

Please plug ethernet cable into: mgmt1

 

 

: Get BIOS image from FTP server. (default)

: Get BIOS image from TFTP server.

Enter Selection : t ------Choose "t"

Enter TFTP server IP address (192.168.1.168):

Enter local IP address (192.168.1.188):

Getting BIOS image (FG600C/FG600C3913804474.rom) from TFTP server:

T[link]FTP://192.168.1.168/FG600C/FG600C3913804474.rom[/link]

 

Press any key to start...

Connecting to server...

 

FG600C/FG600C3913804 14% |**** | 595k 0:00:05 ETA

FG600C/FG600C3913804 45% |************** | 1883k 0:00:02 ETA

FG600C/FG600C3913804 73% |********************** | 3026k 0:00:01 ETA

FG600C/FG600C3913804 100% |*******************************| 4096k 0:00:00 ETA

FG600C/FG600C3913804 100% |*******************************| 4096k 0:00:00 ETA

FG600C/FG600C3913804 100% |*******************************| 4096k 0:00:00 ETA

Done. 4194304 bytes received

 

The BIOS ROM is ready to be updated.

 

WARNING: DO NOT POEWR OFF THE UNIT DURING BIOS UPDATING!!!

 

This process may take a few minutes.

Press Enter when you are ready ...

 

BIOS is updating...

flashrom v0.9.2-runknown on Linux 2.4.37 (x86_64), built with libpci 0.0, GCC 3.4.6, little endian

flashrom is free software, get the source code at [link]http://www.flashrom.org[/link]

 

Calibrating delay loop... OK.

No coreboot table found.

sh: dmidecode: not found

dmidecode execution unsucessfull - continuing without DMI info

Found chipset "Intel 3450", enabling flash write... OK.

This chipset supports the following protocols: FWH,SPI.

Found chip "Winbond W25Q32" (4096 KB, SPI) at physical address 0xffc00000.

===

This flash part has status UNTESTED for operations: PROBE READ ERASE WRITE

The test status of this chip may have been updated in the latest development

version of flashrom. If you are running the latest development version,

please email a report to flashrom@flashrom.org if any of the above operations

work correctly for you with this flash part. Please include the flashrom

output with the additional -V option for all operations you tested (-V, -Vr,

-Vw, -VE), and mention which mainboard or programmer you tested.

Thanks for your help!

===

Flash image seems to be a legacy BIOS. Disabling checks.

File's BIOS Fortinet Banner: FortiGate-600C (20:43-08.19.2014)

 

 

File's BIOS Product Model: FG600C

File's BIOS Serial Number: FG600C3913804474

File's BIOS Version: Ver:04000023

File's BIOS HW MAC address: 085b0e33e818

File's BIOS OEM Serial Num: FG600C3913804474

File's BIOS Licence: aa8a8d778dc82ed8

File's BIOS HW Rev/Part Num: 08908-04

Writing flash chip... Erasing flash before programming... Erasing flash chip... SUCCESS.

done.

Programming flash... ################################################################

done.

COMPLETE.

Verifying flash..., (size:0x400000, from:0x0)

 

Rom's BIOS Fortinet Banner: FortiGate-600C (20:43-08.19.2014)

 

 

Rom's BIOS Product Model: FG600C

Rom's BIOS Serial Number: FG600C3913804474

Rom's BIOS Version: Ver:04000023

Rom's BIOS HW MAC address: 085b0e33e818

Rom's BIOS OEM Serial Num: FG600C3913804474

Rom's BIOS Licence: aa8a8d778dc82ed8

Rom's BIOS HW Rev/Part Num: 08908-04

 

File's checksum: (0X000000 ~ 0X400000) = 0X312906FB

Flash's checksum: (0X000000 ~ 0X400000) = 0X312906FB

 

Correct: File and Flash's verification is OK!

 

 

Updating BIOS ROM Done!

 

 

FORTITEST/FG600C3913801007 # exe reb --------Reboot the unit

 

This operation will reboot the system !

Do you want to continue? (y/n)y

 

 

 

 

 

The system is going down NOW !!

 

 

System is rebooting...

 

FORTITEST/FG600C3913801007 #

 

Please stand by while rebooting the system.

 

FortiGate-600C (20:43-08.19.2014)

Ver:04000023

Serial number:FG600C3913804474

RAM activation

CPU(00:00020655 bfebfbff): MP initialization

CPU(01:00020655 bfebfbff): MP initialization

CPU(04:00020655 bfebfbff): MP initialization

CPU(05:00020655 bfebfbff): MP initialization

Total RAM: 4096MB