EXT3 Error - Fortigate 60D

[Mikelar]

I'm hoping someone can provide some advice on file system issues I've got on a 60D running 5.2.2. We use this firewall solely for a redundant IPSec VPN tunnel that sees very little activity, and as such it currently has no active support subscription. 

A few weeks ago I noticed the IPsec tunnel dropping with unusual VPN logs which would require a reboot to resolve the issue, except after rebooting from the GUI the unit would not come back up on it's own. A hard reboot would then successfully boot the device. I'm also unable to connect to the CLI when this occurs, but can access it after a successful reboot.

After checking the system logs I noticed an ext3 error:

EXT3-fs error (device sd(8,3)): ext3_get_inode_loc: unable to read inode block - inode=15, block=4

Since then I've not been able to access any of the units logs from the GUI at all.

I've just ran a diag disktest but it failed with the below result:

6407.3 9000(70.31%): .................................................. 10.9 4.0
6547.7 9200(71.88%): ..........................XXXXXXXXXXXXXXXXXX Read error: 4194304 bytes wanted, only 2080768 bytes read!
Test Result: Failed
Tested size: 9376MB (73.25% Coverage of whole disk)
Time used: 6791.5 sec
Read Speed: 10.9MB/s
Write Speed: 3.8MB/s
Command fail. Return code -19

I'm thinking my next step is to try the HQIP tool to check for hardware errors, which would also require a fresh firmware reload. I'm assuming if it is a hardware issue, I'll need to back-purchase an active subscription in order for it to be covered under RMA. Since I can't log a TAC ticket for this unit, I was just hoping for some advice on whether or not this is the right approach.

Cheers.

[emnoc]

Sounds like the right approach to me. You could  also reformat the flash but  you a copy of the fortinet image file.

[Ralph1973]

Hello, yes, you can always run the HQIP image (if you can obtain one). That is what TAC will suggest as well I think.

Here is how:

1.DOWNLOAD HQIP image according to product2.Reboot the Unit3.Interrupt the boot sequence4.LOAD advanced hardware test imagehttp://kb.fortinet.com/kb...stateId=0%200%204321255.Run without saving

6.During the HQIP test you will need to wire the network ports. A diagram indicating how to wire the ports will be displayed before the Network Controller Test starts.

[vjoshi_FTNT]

Hello,

Please refer to the below KB article from Fortinet :

http://kb.fortinet.com/kb/microsites/search.do?cmd=displayKC&docType=kc&externalId=FD31857

This explains the same error which you see on your device

[Mikelar]

vjoshi wrote:

Hello,

Please refer to the below KB article from Fortinet :

http://kb.fortinet.com/kb/microsites/search.do?cmd=displayKC&docType=kc&externalId=FD31857

This explains the same error which you see on your device

Thanks for the KB article. I've formatted the boot device an re-loaded the fresh image as per the fix, but after running another diag disk test unfortunately I'm getting the same result.

5911.6 9400(73.44%): XXXXXXXXXXXXXXXXXXXXXX Read error: 4194304 bytes wanted, only 647168 bytes read!

Test Result: Failed
Tested size: 9488MB (74.12% Coverage of whole disk)
Time used: 6111.4 sec
Read Speed: 16.6MB/s
Write Speed: 3.9MB/s
Command fail. Return code -29

[Ralph1973]

Thanks for the KB article. I've formatted the boot device an re-loaded the fresh image as per the fix, but after running another diag disk test unfortunately I'm getting the same result.

5911.6 9400(73.44%): XXXXXXXXXXXXXXXXXXXXXX Read error: 4194304 bytes wanted, only 647168 bytes read!

Oh that sounds annoying. I had a Fortigate 60c earlier, with defective flash. The flash is fixed to the motherboard so not replaceable, what means I couldn't use the unit anymore. Since a couple of years , Fortinet recommends not to use flash for logging purposes, to save flash use.

I think it's not done. Units work excellent, but throwing away the unit because of flash damage it's so annoying.

Regards and good luck,

Ralph

[Mikelar]

Thanks all for the input. I've run the HQIP test successfully, which passed all available tests. 

========================= Fortinet Hardware Quick Inspection Report =========================


                                            BIOS Integrity Check: PASS
                                      System Configuration Check: PASS
                                                     Memory Test: PASS
                                                        CPU Test: PASS
                                     CPU/Memory Performance Test: PASS
                                                  FortiASIC Test: N/A
                                                        USB Test: N/A (0 USB Devices)
                                                Boot Device Test: PASS
                                                  Hard Disk Test: N/A (0 Hard Disks)
                               Network Interface Controller Test: PASS
                                             NPU DDR Memory Test: N/A
                                                        LED Test: PASS
                                               Reset Button Test: PASS


========================= Fortinet Hardware Quick Inspection PASSED =======================

I then loaded a fresh copy of the 5.2.2 firmware with blank config and re-ran the disktest, which gave me the same results as before the testing.

FGT60Dxxxxx # diag disktest run

Round 1 started.
Current Test Device: /dev/sda
Total size: 12800M
Current Test Block: 4M.
Current Time Limit: No limit
Current Size Limit: No limit
Time(Sec) Size(MB) Read(MB/s) Write(MB/s)
0.0 0(0.00%): .................................................. 16.3 4.8
111.1 200(1.56%): .................................................. 16.4 4.1

5721.2 9200(71.88%): .......................................XXXXXXXXXXX 15.7 2.3
5920.8 9400(73.44%): XX Read error: 4194304 bytes wanted, only 122880 bytes read!

Test Result: Failed
Tested size: 9408MB (73.50% Coverage of whole disk)
Time used: 5939.6 sec
Read Speed: 16.6MB/s
Write Speed: 4.1MB/s
Command fail. Return code -14

I'm now kind of at a loss for what to do next. It seems that it's not likely to be a hardware issue, but reloading the firmware didn't fix the issue (which is essentially the recommended fix for all file system issues as far as I can tell). 

There is no active support subscription on this device, so I'm unable to log a support ticket.

Any further assistance would be greatly appreciated.

[Nick1]

Mikelar wrote:

Thanks all for the input. I've run the HQIP test successfully, which passed all available tests. 

========================= Fortinet Hardware Quick Inspection Report =========================


                                            BIOS Integrity Check: PASS
                                      System Configuration Check: PASS
                                                     Memory Test: PASS
                                                        CPU Test: PASS
                                     CPU/Memory Performance Test: PASS
                                                  FortiASIC Test: N/A
                                                        USB Test: N/A (0 USB Devices)
                                                Boot Device Test: PASS
                                                  Hard Disk Test: N/A (0 Hard Disks)
                               Network Interface Controller Test: PASS
                                             NPU DDR Memory Test: N/A
                                                        LED Test: PASS
                                               Reset Button Test: PASS


========================= Fortinet Hardware Quick Inspection PASSED =======================

The HOIP test shows no hard disks detected,which I think is the problem.

[Mikelar]

Nick wrote:

The HOIP test shows no hard disks detected,which I think is the problem.

Thanks for your response. The model I'm using is a 60D, which has no internal hard disks, but rather uses flash memory only.

I'm guessing the Memory Test portion of the HQIP testing only tests the RAM, does anyone know if it also tests the flash? 

I'm guessing the diag disktest runs a much more comprehensive memory test, but neither really tells me if its a hardware issue.

HQIP Memory Test

==> Memory Test

pagesize=8192, pagesizemask=FFFFE000
Detecting memory(1842MB)...[Done, 959MB will be tested]

STRESS->MEM-> 1.Random value...

STRESS->MEM-> Completed in 55 seconds (0 errors).

Release memory done.

<== Memory Test - PASS

[ede_pfau]

You're right, this absolutely looks like a hardware failure, the flash is broken.

Bad news is that without FortiCare you won't get a replacement unit.

Good news is, if you buy a contract now it will only be backdated for a maximum of 6 months even if the previous contract has expired a long time ago. For a 60D, FCare is a small amount compared to new hardware.

So, get a new contract, register it, open a ticket and send them the HQIP logs.