Support Forum
The Forums are a place to find answers on a range of Fortinet products from peers and product experts.
m0j0
New Contributor III

Error 500: Internal Server Error in web interface

I've got a couple of 60C's in HA running 5.0.4.  I tried to log in to the web interface today and it logs me in and loads the top toolbar and sits there with two empty frames for a while before it finally returns a "Error 500: Internal Server Error" in the left frame.  I've tried killing all the httpsd processes but it hasn't made a difference.  A diag debug on the web-ui shows the following when the error shows in the web interface:-

[httpsd 27319] http_config.c[573] ap_invoke_handler -- handler 'fastcgi-script' completed (result==500)

[httpsd 27319] http_request.c[1443] ap_internal_redirect -- internal redirect to '/p/pubredir/httperror/'

 

Trying to avoid rebooting the firewall at all costs.  I've considered a forced failover to the secondary device but want to exhaust any other possible options before going down that road, and I'm not sure it will fix it anyway.  

 

The web interface was working fine a day or two ago and nothing has changed since then.  Firewall uptime is 223 days and looking at my Cacti graphs, memory utilisation has been sitting around 65% since November but appears to have just nudged up to 70% very recently.  Not sure if this is the cause of the problem or not.

 

1 Solution
Prometheus
New Contributor II

Hi Skippy,

 

So far so good. We did not experience the problem anymore since we installed 5.2.8. But we only installed the devices 10 days ago. It took the 5.2.7 devices like 14 days to come up with the problem, so I cannot confirm nor can I deny that 5.2.8 is the right solution.

Skippy wrote:

Prometheus wrote:

So last night we switched to the backup cluster using the 5.2.8 firmware. I'll keep you guys posted if it keeps on working or not. The faulty device also worked again after a reboot but this is still under investigation by Fortinet.

Is it normal to have something like 28GB of disk logging in roughly 10 days of operation?

Full logging enabled for all policies with four WAN interfaces and both static and policy routes as well as BGP.

 

I think this is quite normal. I see it with our network monitoring which collected over 2GB of logging in 12 hours. But once again i'm not sure.

 

View solution in original post

46 REPLIES 46
vjoshi_FTNT
Staff
Staff

Hello,

 

I remember, there is a known issue for the same problem. Upgrading to the latest patch should fix this issue.

 

If you need a confirmation about the bug ID etc., a support ticket might help. But I am pretty sure, there was a bug on this.

 

Hope that helps.

Robin_Svanberg

vjoshi wrote:

Hello,

 

I remember, there is a known issue for the same problem. Upgrading to the latest patch should fix this issue.

 

If you need a confirmation about the bug ID etc., a support ticket might help. But I am pretty sure, there was a bug on this.

 

Hope that helps.

Probably Bug ID #239368. Web-based Manager : Fix Internal 500 Server Error after login.  

Solved in 5.0 Patch 8.

 

BR Robin

 

Robin Svanberg Network Consultant @ Ethersec AB in Östersund, Sweden

 

robin.svanberg@ethersec.se

Robin Svanberg Network Consultant @ Ethersec AB in Östersund, Sweden robin.svanberg@ethersec.se
bikash_Shaw
New Contributor III

Hi 

 

Can you connect your PC to fortigate firewall via console. I think it might be issue with disk.

 

Regards

Bikash

CorneJvV
New Contributor

Looks like this issue has carried over to FortiOS 5.2.3 as well.

We have a few 100D's that are now getting this issue often.

 

Is there another way to resolve this except for rebooting/shut down the FortiGate ?

All of these devices are in live non- HA environments.

FCNSA FortiGate 60C, 110C, 200B, 310B FortiAnalyzer 100C FortiMail 100 FortiManager 100
FCNSA FortiGate 60C, 110C, 200B, 310B FortiAnalyzer 100C FortiMail 100 FortiManager 100
CorneJvV

Issue Resolved Pending, format the device and re-install the image.

 

From the "diag sys top" output, there a lot of processes in Dead state.

Run Time: 13 days, 18 hours and 20 minutes 0U, 0N, 1S, 99I; 3954T, 1579F, 193KF proxyworker 112 S 0.4 1.2 sshd 3891 S 0.4 0.3 urlfilter 120 S 0.0 2.1 reportd 100 D 0.0 1.7 httpsd 72 S 0.0 1.6 proxyworker 108 S 0.0 1.2 httpsd 73 S 0.0 1.0 pyfcgid 3883 S 0.0 0.8 pyfcgid 3900 S 0.0 0.8 pyfcgid 3877 S 0.0 0.8 pyfcgid 3878 D 0.0 0.8 pyfcgid 3887 D 0.0 0.8 pyfcgid 3876 D 0.0 0.8

FCNSA FortiGate 60C, 110C, 200B, 310B FortiAnalyzer 100C FortiMail 100 FortiManager 100
FCNSA FortiGate 60C, 110C, 200B, 310B FortiAnalyzer 100C FortiMail 100 FortiManager 100
Tommy_Rogers

This is really getting to be a pain in the butt. I have a 100D with version 5.2.5 firmware. I have RMA'ed three (3) of these guys back to Fortinet. This is the fourth one they have sent me and I am still getting the error after a few days of service. The exact verbage is "Error 500: Internal Server Error". I know that it is a disk error but I don't know why. There is no way that all four 100d's can have bad flash drives unless it is a bad batch from a manufacture.

 

This is really embarrassing for my company to have to tell our customer that they have a defective unit. THIS IS THE FORTH ONE GUYS!!!! Come on get your engineers awake and tell me what is wrong with this thing.

 

The bad thing here is we recommended a Fortigate 100D over a Sonicwall, They were going to buy a Sonicwall but we talked them in to a Fortigate.

I need this fricking problem fixed ASAP!

Tommy Rogers FCNSA FCNSP
Tommy Rogers FCNSA FCNSP
NotMine

I feel you bro, we had to RMA an 200D just three months after putting in production because of a failed flash disk. Three months after the RMA, the new device started showing the same symptoms of disk failure (error 500 being one of them). Fortunately, we did not had to do a second RMA, just reinstall the firmware.

 

The problem, in our case, was very extensive logging. The customer had more than a hundred firewall policies, and had turned on full logging on every single one of them. So, I just turned off logging to the disk and redirected all logs to a FortiAnalyzer. These SSD disks have limited write-read cycles, and I presumed extensive logging cannot be good for them. I was right, apparently, Fortinet has issued CSB-151124-1 shortly after our problem (excerpt):

 

FortiGate devices with internal storage running FortiOS 5.0 or 5.2 may experience flash disk errors in cases where the flash disk has reached a finite number of program–erase cycles (typically written as P/E cycles). While Fortinet has designed all flash-based units with this limitation in mind under expected usage, experience with a very low proportion of users shows that an issue can be caused by excessive writing, updating, and modification of files on the flash disks. Features in FortiOS that may cause heavy disk usage are: 1. Disk logging 2. WanOpt & WebCache 3. Local-in policies 4. Device identification 5. DHCP and/or PPPoE 6. Excessive reboots or power cycles

NSE 7

All oppinions/statements written here are my own.

NSE 7 All oppinions/statements written here are my own.
Tommy_Rogers

Well, I have started a logdisk format. I can't even run the logdisk disable command, syas there is an error. Hopefully, if the format finishes I can get in the shell and do a disable. Trying to do this while the unit is in service from remote.

Thanks for your input though..

Tommy Rogers FCNSA FCNSP
Tommy Rogers FCNSA FCNSP
Tommy_Rogers

If I try to run a "config log delete_all" I get

"miglog.c:send_cron_msg_to_miglogd: Error sending msg. Resource temporarily unavailable"

 

Can't run a format during business hours because it wants to reboot when it gets done.

(lot a hair pulling going on)

 

Tommy Rogers FCNSA FCNSP
Tommy Rogers FCNSA FCNSP
Announcements

Select Forum Responses to become Knowledge Articles!

Select the “Nominate to Knowledge Base” button to recommend a forum post to become a knowledge article.

Labels
Top Kudoed Authors