Error 500: Internal Server Error in web interface

[m0j0]

I've got a couple of 60C's in HA running 5.0.4.  I tried to log in to the web interface today and it logs me in and loads the top toolbar and sits there with two empty frames for a while before it finally returns a "Error 500: Internal Server Error" in the left frame.  I've tried killing all the httpsd processes but it hasn't made a difference.  A diag debug on the web-ui shows the following when the error shows in the web interface:-

[httpsd 27319] http_config.c[573] ap_invoke_handler -- handler 'fastcgi-script' completed (result==500)

[httpsd 27319] http_request.c[1443] ap_internal_redirect -- internal redirect to '/p/pubredir/httperror/'

Trying to avoid rebooting the firewall at all costs.  I've considered a forced failover to the secondary device but want to exhaust any other possible options before going down that road, and I'm not sure it will fix it anyway.  

The web interface was working fine a day or two ago and nothing has changed since then.  Firewall uptime is 223 days and looking at my Cacti graphs, memory utilisation has been sitting around 65% since November but appears to have just nudged up to 70% very recently.  Not sure if this is the cause of the problem or not.

[HA]

Hello,

Same problem for me with some 100D devices.

Normally, this issue (disk) should have been fixes since 5.2.5.

That's not really the case.

According to the support, this issue is gone with 5.4.x.

Hey Fortinet, wake up, we need clear answer about this problem !!

Regards,

HA

[Rafael_Rosseto]

Hey guys,

In all my cases, I tried to run HQIP to check disk integrity, and all of them I have to open RMA.

If you have 200D, better disable disk logging.

[dethangel]

200D stopped issuing IP addresses.

Rebooted it and got some SDCard errors in the console screen - the unit never came back.

I've got a 100D on 5.2.3 b670 (GA) that's doing fine *crosses fingers* - have downgraded 1 of my new 100D to this firmware (shipped was 5.2.5)

Why is this beginning to sound like the Ubiquiti EdgeRouter USB drive failure scenario ....

[Prometheus]

So last night we switched to the backup cluster using the 5.2.8 firmware. I'll keep you guys posted if it keeps on working or not. The faulty device also worked again after a reboot but this is still under investigation by Fortinet.

[Skippy]

Prometheus wrote:

So last night we switched to the backup cluster using the 5.2.8 firmware. I'll keep you guys posted if it keeps on working or not. The faulty device also worked again after a reboot but this is still under investigation by Fortinet.

Hi, any update if the issue came back on with the 5.2.8?

Had the same problem on a 200D with 5.2.7 and it looks like it does not like when it uses all the available disk space for disk logging. Replaced with another one to play it safe and investigating on the old one that is not in production anymore. Is it normal to have something like 28GB of disk logging in roughly 10 days of operation?

Full logging enabled for all policies with four WAN interfaces and both static and policy routes as well as BGP.

If I check in the Disk Management I can see 58GB size with 23GB for logging and archiving and 35 allocated for WAN optimization (Not in use so I could lower this down).

Allocation for disk logging is set at 0MB and the use is 28GB, does the 0MB mean no limit?

Should the new logs just overwrite the old ones or how does it work?

At the moment I disabled the disk logging on the new unit but it is something I would want to have as it looks like the realtime upload to forticloud does not work as intended (I can't see any data in forticloud).

[Prometheus · Accepted solution]

Hi Skippy,

So far so good. We did not experience the problem anymore since we installed 5.2.8. But we only installed the devices 10 days ago. It took the 5.2.7 devices like 14 days to come up with the problem, so I cannot confirm nor can I deny that 5.2.8 is the right solution.

Skippy wrote:

Prometheus wrote:

So last night we switched to the backup cluster using the 5.2.8 firmware. I'll keep you guys posted if it keeps on working or not. The faulty device also worked again after a reboot but this is still under investigation by Fortinet.

Is it normal to have something like 28GB of disk logging in roughly 10 days of operation?

Full logging enabled for all policies with four WAN interfaces and both static and policy routes as well as BGP.

I think this is quite normal. I see it with our network monitoring which collected over 2GB of logging in 12 hours. But once again i'm not sure.

[Skippy]

Thanks Prometheus,

I will probably wait for another week to see what your findings are before enabling it as we don't have an HA environment and I can't keep RMAing them and swapping them after hours, it is very time consuming.

Could you please confirm the settings you have in disk management? This is what it looks like on my 200D, I haven't touched anything from the default settings and no disk logging is enabled at the moment.

[Prometheus]

@Skippy

[Skippy]

Thanks, that looks exactly the same as mine with the only difference being the disk size as I have a 200D.

I will hold back few more days and see how you go and then re-enable disk logging if everything is good for you, just can't afford a FortiAnalyzer at the moment. Thanks

[djwilliams]

We can add two more HA pairs of 200Ds running 5.2.7 code to the growing list of devices that exhibit the exact same behavior.  I managed to force the secondary device to be active and made sure any local logging was only leveraging memory and not disk.  I will be running the HQIP image against the primary 200D, formatting and reapplying a fresh 5.2.8 image before putting it back in production.  The slightest hint that it is going wonky and I will likely be processing another RMA.