40C, 60C and 80C Crashed After Firmware Upgrade

[Tumi]

Hi Guys.

I upgraded one client's 80C with the latest Firmaware, build 642. Ever since then, there is no internet in the office. None of my DNS Servers can resolve any website. I tried re configuring the unit, but still, no joy. I can for example RDP in from outside, emails are working, but DNS Servers cannot resolve any sites.

I also upgraded an 60C for another client with the same new firmware. And that one is now just "dead." It doesn't boot up. Status LED Flashes perpetually and I can't access the unit in any way.

To solve the 60C client's problem, I gave them a 40C which I had also just upgraded with the new Firmware. After doing that, the unit basically blocked all incoming communications. Users can browse internet, send emails etc. But when people send to the office, those mails never reach the Exchange Server. I can't even RDP in. I ended up downgrading back to the previous Firmware, did a Factory Reset and reconfigured the unit. Still no joy. 

Now I have 3 "dead" units basically.

Please someone help. Thank you.

[Dave_Hall]

Some questions/suggestions:

1. Have you followed the recommend upgrade path?

2. Have you performed a check-sum validation on the downloaded firmware images?

3. Reviewed the patch note(s) for each firmware revision from build 642 to last previous installed firmware update+1?

4. If going from 4.3.x firmware to 5.x firmware, reviewed the 5.2 patch notes for any "gotchas"?

4. Connected a console cable to the 40C and 80C to see any error messages during boot up?

[simonorch]

And for the box that can't boot. Have you tried formating it and loading firmware via tftp?

If that doesn't get you a bootable unit, then you need to open an RMA ticket

[flynnp]

Hi

I had a similar problem after upgrading a new Fwifi 30D.

After the upgrade I couldn't resolve with DNS etc.

One thing to note is that after changing the IP address of the unit, I had to manually enter in a number of the DHCP parameters and point it to an external DNS server as it wouldn't proxy correctly.

Also it was complaining that Fortiguard couldn't access Web Filtering or Email filtering. This didn't work until I changed the port from 53 (DNS) to the alternate port 8888

I can not enable the Forticloud as the tick box is greyed out for " Send Logs to Forticloud"

Certainly appears pretty buggy to me

Regards

Paul

[Dave_Hall]

Aside from the usually stuff you should be doing when performing firmware upgrades (follow upgrade path, read the patch notes, etc.) I will load the before/after config files into WinMerge (or other text comparison tool) just to see what exactly was changed during the upgrade.  Saves a lot of headaches.  I will also use WinMerge when I'm converting/migrating some key settings from one config to another one.

[stuberman]

I have a FWF-60C which I received a couple years ago running FortiOS 4.x and promptly upgraded it to FortiOS 5.0.0 and then I would upgrade to each point release until 5.2.0 became available. That is when I started to experience similar issues with being unable to upgrade easily. My previous attempts to upgraded 5.2.1 to 5.2.2 had all failed and I restored using BIOS to 5.2.1 Last night was the worst experience for me, I was trying to upgrade from 5.2.1 to 5.2.3 and even console connections using TFTP was failing. And I was able to succeed after hours of working on it. I hope to share my experiences and observations on the issue:

1) Before the upgrade last night, I saw errors showing disk (inode) errors with a cryptic message about needing to run fs2-fsck to repair the disk problems (a linux command not part of the Fortigate commands).

2) I decided the best course was to upgrade the image to 5.2.3 using the web GUI and checking the 'format disk' option since I wanted to clear any disk problems.

3) The image upload was successful but the UTM would not boot after the format. 

4) I used a terminal session to the console port to  use this option to start from scratch: ": Format boot device."

5) I then tried the option ": Get firmware image from TFTP server." but my TFTP server showed no attempt to transfer the image.

6) I realized the UTM now had no image nor a backup image on a separate partition (this saved me in the past).

7) I also realized that the previous settings (IP addresses) and the default IP addresses were not working. [192.168.0.99 or 192.168.1.99]

8) I was able to solved this by setting the WAN1 port to DHCP ": Configuration and information" -> "

: Set firmware download port." and ": Set DHCP mode."

9) I then provided an address using "TFTP32 server" from a PC using it as a DHCP server and TFTP server.

10) I used the console (BIOS) commands to upload the image both as primary and secondary boot image.

11) Then I restored the configuration file I had recently saved. 

My thoughts on this:

A. The new FortiOS 5.2 image attempts to remove or hide the disk access as stated in Fortinet material. After the upgrade the CLI and web GUI shows "No disks available", since I had been using disk logging, the upgrades choked on the existing partitions. I counted as many as four partitions prior to the upgrade to 5.2.3.

B. The BIOS options should include a way to assign an IP address, subnet mask and default gateway to an interface or at least determine what the existing addresses are in order to TFTP a new image.

C. The FortiExplorer application was of no use in this case.

D. The new upgrade image interface in 5.2.3 looks much nicer.

E. I had tried to use the CLI under 5.2.1 to scan and repair errors on the disk but it would fail with an error code.

F. A complete format of the flash was in order but there was little help online to figure out how to recover. Most posts considered the units bricks and in need of RMA. 

[ede_pfau]

@Stuart,

sorry to hear from your odysee.

It certainly sounds like your FGT has a hardware problem with the flash disk. Hopefully, you've saved a screenshot before trying to recover the FGT, to aid in the RMA process.

If you now think that formatting the flash and reloading the image "solved" all issues you might be wrong. If the flash really has faults then formatting will have excluded these faulty cells for now. Expect to see other cells failing in the near future, resulting in a corrupted filesystem. The only way to get this corrected is to replace the hardware.

BTW, Support will ask you to run a diagnostic image and send them it's report. You can download these HQIP images after logging in to your Fortinet account. The catch is, now it will probably not show any hardware defects. At least, you're prepared for the near future.

[stuberman]

I hope you are wrong. This same upgrade problem has affected others at my company with the same model. If I see more disk error messages then I will screenshot them before requesting an RMA. I think however others that have this problem can benefit from the steps I took to recover the unit. It is far more problematic for our Fortigates around the globe, but I am hoping it only applies to the smaller models since that is where the disk logging feature was removed as part of the code upgrade.

[ede_pfau]

So you think it's the upgrade process that botches the FGT up and 'produces' filesystem errors, looking like flash cell defects? Maybe. I'll never know who was first, hen or egg. As at that point I RMA the FGT and get a (fairly) new unit back and all is bliss.

I admit HW failures have risen exponentially lately but...can you name a complicated technical device with a life span longer than 3 years today? (#joke_mode) In, say, 10 RMA cases I had to fight hard only once as the supporter didn't want to just accept my downvote on the HW. And I must say, the next time he was right (without knowing it) - it was just a screwed up filesystem from cutting of power once too often. Typical customer handling I'd say.

So I infer you'll sit it out and watch the FGT in question. I'd love to read a follow-up in case it eventually died from HW failure.