Error 500: Internal Server Error in web interface

[m0j0]

I've got a couple of 60C's in HA running 5.0.4.  I tried to log in to the web interface today and it logs me in and loads the top toolbar and sits there with two empty frames for a while before it finally returns a "Error 500: Internal Server Error" in the left frame.  I've tried killing all the httpsd processes but it hasn't made a difference.  A diag debug on the web-ui shows the following when the error shows in the web interface:-

[httpsd 27319] http_config.c[573] ap_invoke_handler -- handler 'fastcgi-script' completed (result==500)

[httpsd 27319] http_request.c[1443] ap_internal_redirect -- internal redirect to '/p/pubredir/httperror/'

Trying to avoid rebooting the firewall at all costs.  I've considered a forced failover to the secondary device but want to exhaust any other possible options before going down that road, and I'm not sure it will fix it anyway.  

The web interface was working fine a day or two ago and nothing has changed since then.  Firewall uptime is 223 days and looking at my Cacti graphs, memory utilisation has been sitting around 65% since November but appears to have just nudged up to 70% very recently.  Not sure if this is the cause of the problem or not.

[vjoshi_FTNT]

Hello,

I remember, there is a known issue for the same problem. Upgrading to the latest patch should fix this issue.

If you need a confirmation about the bug ID etc., a support ticket might help. But I am pretty sure, there was a bug on this.

Hope that helps.

[Robin_Svanberg]

vjoshi wrote:

Hello,

I remember, there is a known issue for the same problem. Upgrading to the latest patch should fix this issue.

If you need a confirmation about the bug ID etc., a support ticket might help. But I am pretty sure, there was a bug on this.

Hope that helps.

Probably Bug ID #239368. Web-based Manager : Fix Internal 500 Server Error after login.  

Solved in 5.0 Patch 8.

BR Robin

[bikash_Shaw]

Hi 

Can you connect your PC to fortigate firewall via console. I think it might be issue with disk.

Regards

Bikash

[CorneJvV]

Looks like this issue has carried over to FortiOS 5.2.3 as well.

We have a few 100D's that are now getting this issue often.

Is there another way to resolve this except for rebooting/shut down the FortiGate ?

All of these devices are in live non- HA environments.

[CorneJvV]

Issue Resolved Pending, format the device and re-install the image.

From the "diag sys top" output, there a lot of processes in Dead state.

Run Time: 13 days, 18 hours and 20 minutes 0U, 0N, 1S, 99I; 3954T, 1579F, 193KF proxyworker 112 S 0.4 1.2 sshd 3891 S 0.4 0.3 urlfilter 120 S 0.0 2.1 reportd 100 D 0.0 1.7 httpsd 72 S 0.0 1.6 proxyworker 108 S 0.0 1.2 httpsd 73 S 0.0 1.0 pyfcgid 3883 S 0.0 0.8 pyfcgid 3900 S 0.0 0.8 pyfcgid 3877 S 0.0 0.8 pyfcgid 3878 D 0.0 0.8 pyfcgid 3887 D 0.0 0.8 pyfcgid 3876 D 0.0 0.8

[Tommy_Rogers]

This is really getting to be a pain in the butt. I have a 100D with version 5.2.5 firmware. I have RMA'ed three (3) of these guys back to Fortinet. This is the fourth one they have sent me and I am still getting the error after a few days of service. The exact verbage is "Error 500: Internal Server Error". I know that it is a disk error but I don't know why. There is no way that all four 100d's can have bad flash drives unless it is a bad batch from a manufacture.

This is really embarrassing for my company to have to tell our customer that they have a defective unit. THIS IS THE FORTH ONE GUYS!!!! Come on get your engineers awake and tell me what is wrong with this thing.

The bad thing here is we recommended a Fortigate 100D over a Sonicwall, They were going to buy a Sonicwall but we talked them in to a Fortigate.

I need this fricking problem fixed ASAP!

[NotMine]

I feel you bro, we had to RMA an 200D just three months after putting in production because of a failed flash disk. Three months after the RMA, the new device started showing the same symptoms of disk failure (error 500 being one of them). Fortunately, we did not had to do a second RMA, just reinstall the firmware.

The problem, in our case, was very extensive logging. The customer had more than a hundred firewall policies, and had turned on full logging on every single one of them. So, I just turned off logging to the disk and redirected all logs to a FortiAnalyzer. These SSD disks have limited write-read cycles, and I presumed extensive logging cannot be good for them. I was right, apparently, Fortinet has issued CSB-151124-1 shortly after our problem (excerpt):

FortiGate devices with internal storage running FortiOS 5.0 or 5.2 may experience flash disk errors in cases where the flash disk has reached a finite number of program–erase cycles (typically written as P/E cycles). While Fortinet has designed all flash-based units with this limitation in mind under expected usage, experience with a very low proportion of users shows that an issue can be caused by excessive writing, updating, and modification of files on the flash disks. Features in FortiOS that may cause heavy disk usage are: 1. Disk logging 2. WanOpt & WebCache 3. Local-in policies 4. Device identification 5. DHCP and/or PPPoE 6. Excessive reboots or power cycles

[Tommy_Rogers]

Well, I have started a logdisk format. I can't even run the logdisk disable command, syas there is an error. Hopefully, if the format finishes I can get in the shell and do a disable. Trying to do this while the unit is in service from remote.

Thanks for your input though..

[Tommy_Rogers]

If I try to run a "config log delete_all" I get

"miglog.c:send_cron_msg_to_miglogd: Error sending msg. Resource temporarily unavailable"

Can't run a format during business hours because it wants to reboot when it gets done.

(lot a hair pulling going on)