fortigate 80f rebooting randomly

[FernandoM]

Hi, I have a new FortiGate 80F located in a data center since August 2021 and it was working perfect, until October when I started getting notification that the internet was dropping out overnight. At first I thought it was the provider doing maintenance work over-night since the drop outs only happened between 1am to 3am and it was only happening once or twice a month. By December it started happening every second day or so. That's when I noticed that the router was rebooting automatically.

Logged a call with Fortinet support, they checked logs, hardware stats, etc, all was showing to be working fine. They asked me to do a hardware diagnostic check and a format to the same firmware 7.0.3.

After all those steps the unit kept on rebooting, so they replaced it with another unit.

After replacing the unit it only took 2 days and the replacement unit started rebooting again.

Escalated the issue again and now they're asking me to do other test. Downgraded to Firmware 7.0.2 and it continues to reboot randomly, They requested to disable

config system np6xlite
edit np6xlite_0
set fastpath disable  

still no joy

I'm now going around in circles with them as they want to keep on troubleshooting.

One thing I did noticed and I did mention to support is that the operating temperature seems to be too high.

Here's what the average reading shows in Celsius

TMP 1 External Temperature  53.25 C

TMP 4 Temperature                 43.00 C

CPU ON-IDLE Temperature    65.84 C

MV1680 Temperature              71.00 C

I checked the hardware specifications and it says that the operating temperature should be 40.00 C

Since the router is in the data center I asked for an air conditioning report for the past month of December, the average temperate was 24.22, checked on other network devices that are on the same rack and the average operating temperature was 36.00 C. 

I'm convinced that there's a flaw with the unit based on the temperature reading, unless I was unlucky to receive two faulty routers. 

Has anyone experienced any issues with reboots or have any suggestions?

Thanks

[FernandoM]

I've checked it today and performed the cmd you requested and saw this -

"Last reboot reason: kernel panic"

What could that mean?

[AlexC-FTNT · Accepted solution]

it means that it may be a firmware issue, but the unit is not smart enough to tell you what the problem is until you reboot while connected to the console port. That's when it should display a useful error. I suggest you upgrade the FortiOS to 7.0.4 - the problem has been fixed for the larger units where the error was visible in comlog. There is a change it was fixed for smaller units too.

[FernandoM]

Thanks @AlexC-FTNT the support technician suggested the same thing after I reported the results from the "Get System Status"

I've just upgraded to the latest firmware. I will report back in a few days.

[FernandoM]

@AlexC-FTNT The firmware upgrade has resolved the issue. Mind you that I had this issue for months and it took Fortinet months to release a critical update.

[AlexC-FTNT]

@FernandoM I have to agree with you that generally the fixes take longer than expected, but  here are two things to consider:

- to my knowledge we are the only company that provides a firmware patch every 2-4 months, as opposed to 1/2 patches per year
- in this particular case the bug was not proven on the small units because the output can be only collected at reboot via console port. And for some reason, not many customers are used to or have the possibility of using the console port

[FernandoM]

I started with both power supplies, one power supply connected to A and the other on B on the rack. It's a shared rack that has 1 x Fortigate 60E, 1 x Fortigate 100F, 4 x switches, 1 x QNAP, 3 x HP servers.  None of those devices are experiencing any outages.

While doing the initial troubleshooting I always kept both power supplies connecged even after getting the replacement router, it wasn't until a week and a half ago did I disconnect power source 2 from the Fortigate, thinking that maybe both powers connected could be overheating the router.

Thanks @ikmarwright I will ask the data center for power logs to see if there's a spike during those reboots.

[unnamed_]

Just wanted to add to this thread that I experienced the same issue in firmware version 7.0.3 but TAC support suggested upgrading to 7.0.8(as of Nov 2022) since we have found no issue on config or hardware. The logs show the FG has an unexpected shutdown and then automatically rebooted in a span of a minute to two, and the last reboot reason was kernel panic.