100D Locks Up Every Few Days

[billflu]

Every three days or so we have an issue where our 100D stops handing out DHCP addresses, so users are not able to get online. The only way to fix this is to hard power cycle. During the time of the issue, when trying to log into the web interface you can get through the authentication and the next page shows an error 500:

The command line interface reacts very minimally, but I was able to get the process list while this is occurring. Some of the processes are labeled "D". I wasn't able to find documentation for this, but it may refer to dead?

2U, 0N, 0S, 98I; 3954T, 1458F, 323KF
       ipsengine 120 S < 3.5 2.6
       ipsengine 118 S < 3.3 2.6
       ipsengine 117 S < 2.1 2.5
       ipsengine 119 S < 1.1 2.6
       scanunitd 8779 S < 0.7 0.3
         src-vis 92 S 0.3 1.1
       scanunitd 8731 S < 0.3 0.3
         updated 86 S 0.1 1.5
     proxyworker 88 S 0.1 1.1
     proxyworker 89 S 0.1 1.1
          newcli 8780 R 0.1 0.4
         reportd 82 D 0.0 2.0
             dsd 113 D 0.0 1.0
         pyfcgid 8629 D 0.0 0.8
          httpsd 8688 D 0.0 0.8
         pyfcgid 8716 D 0.0 0.8
         pyfcgid 8723 D 0.0 0.8
         pyfcgid 8730 D 0.0 0.8
         pyfcgid 8717 D 0.0 0.8
         pyfcgid 8737 D 0.0 0.7

Any input on this issue is appreciated. I've been restarting the firewall nightly, and that seems to keep the issue at bay. We've already done a format and re-install of the OS and also a complete hardware replacement.

Thank you

[mscheiber]

Hi

are you using vdoms? 

We have a similar problem with our 100D's. We are using vdoms and i cant go into the vdom over cli nor even can access some parts over the gui.

And the only fix is to do a hard reboot, because when i do a soft reboot the device will hang. Which FortiOS version do you use? I saw that kind of problem with 5.0.9 and 5.0.12. 

I also opened a ticket at Fortinet support and they recommend to upgrade to 5.2.3 or 5.2.4.

This is my post https://forum.fortinet.com/tm.aspx?m=130647

[billflu]

mscheiber, thanks for the response. It does sound similar, but I'm not using vdoms. The firmware is the latest v5.2.4,build688 (GA). Support is recommending I factory default and restore the configuration again, but it seems they may just be grasping at straws.

[gzarini]

billflu, i had an issue like that with a 100d unit.

It worked fine, but one night it didn't renew dhcp leases, and the next morning dhcp server didn't lease any ip.

I tried that day to enter gui and fg gave that same error, i couldn't even restarted from console so i rebbot the device from the on/off switch. Then it didn't boot.

Since day the unit is on RMA. my recommendation in to make a hardware check with fortigate technical support to check if hardware is working fine.

In my case it was the hard disk that broke up.

Regards.

[billflu]

gzarini, thank you for the input. That sounds very similar to the issue we're having, except it comes back online after rebooting. I may push for a hardware check, just to be safe.

[billflu]

To follow up on this issue, we took the following steps, and have not had any issues since.

Saved the config

Wiped and reloaded the firmware (one version back). Going back a version wasn't suggested by support, it's just an action we decided to take.

v5.2.3,build670 (GA)

Factory defaulted (just to be safe)

Restored the saved config

[HA]

Hello,

It's a well known issue.

Fortinet release a document on the partner portal.

CSB-151124-1 Fortinet 1

Technical Support Customer Support Bulletin Number: CSB-151124-1 Released: 27th November 2015 Modified: Subject: FortiGate flash disk errors Product: FortiGate low end devices Description: FortiGate devices with internal storage running FortiOS 5.0 or 5.2 may experience flash disk errors in cases where the flash disk has reached a finite number of program–erase cycles (typically written as P/E cycles). While Fortinet has designed all flash-based units with this limitation in mind under expected usage, experience with a very low proportion of users shows that an issue can be caused by excessive writing, updating, and modification of files on the flash disks. Features in FortiOS that may cause heavy disk usage are: 1. Disk logging 2. WanOpt & WebCache 3. Local-in policies 4. Device identification 5. DHCP and/or PPPoE 6. Excessive reboots or power cycles Typical symptoms experienced once this condition is met can be (but not limited to) as follows: - Problems accessing web GUI - Failure to execute CLI commands - 99% CPU usage by system - Connectivity issues - Partial or total functionality failure of device (usually DHCP stops working) - Alerts and error messages found in the event log as below: EXT3-fs: group descriptors corrupted ! EXT3-fs error (device sd(8,3)): ext3_check_descriptors: Block bitmap for group 17 not in group (block 17334272)! OR The following critical firewall event was detected: Kernel error. CSB-151124-1 Fortinet 2 Technical Support Customer Support Bulletin date=2015-10-19 time=08:49:12 devname=FortiGate devid=FGT60D3912621349 logid=0100020010 type=event subtype=system level=critical vd="root" logdesc="Kernel error" msg="EXT3-fs error (device sd(8,3)): ext3_get_inode_loc: unable to read inode block - inode=132, block=8" OR EXT2-fs error (device sd(8,3)): ext2_free_blocks: Freeing blocks not in datazone - block = 4294967295, count = 1 - Boot failures and error messages during boot up: Initializing firewall... System is starting... Starting system maintenance... Scanning /dev/mtd1... (100%) Formatting shared data partition ... done! EXT3-fs: error loading journal. EXT3-fs: error loading journal. Potentially Affected Products: Low end FortiGate/FortiWifi models with flash storage 20C, 40C, 60C, 80C, 60D, 90D, 100D Potentially Affected OS: FortiOS 4.x FortiOS 5.0 FortiOS 5.2 Remedy: The issue may be temporarily addressed by formatting boot disk and log disk. Should the issue occur, which would suggest the flash disk has reached its lifetime, you should create an RMA case and attach your current/backup configuration file and self test HQIP logs. Improvements to minimize this issue will be included in FortiOS 5.2.5 patch release and 5.4.0 minor release, the current ETA for release of both versions is December 2015. Fortinet recommend customers to upgrade to FortiOS 5.2.5 or later as soon as it is available in order to minimize flash wear. Not doing so may result in a reduced life time of the device and cause high RMA return rates. Special notes 1. Disk logging a. The 5.0.2 release notes advises against enabling this feature. Starting from 5.0.6, this CSB-151124-1 Fortinet 3 Technical Support Customer Support Bulletin feature is disabled by default on units with flash disk. b. It is possible to enable it from the CLI and when one does so a notification message is displayed: “enabling disk logging impacts overall performance and reduces the lifetime of the unit.”. c. You should avoid usage of disk logging on all units and use remote logging storage such as FortiAnalyzer or FortiCloud. Technical Support Contact Information:http://www.fortinet.com/s..rt/contact_support.html Fortinet technical support home page: [link]https://support.fortinet.com[/link] All rights reserved. Fortinet®, FortiGate®, and FortiGuard®, are registered trademarks of Fortinet, Inc., and other Fortinet names herein may also be trademarks of Fortinet. All other product or company names may be trademarks of their respective owners. Statements contained herein were attained in internal lab tests under ideal conditions, and performance may vary; network variables, different network environments and other conditions may affect performance results. Nothing herein represents any binding commitment or admission by Fortinet, and Fortinet disclaims all representations and warranties, whether express or implied, except to the extent Fortinet enters a binding written contract, signed by Fortinet’s General Counsel, with an express representation or warranty included therein. All Fortinet end-customers are bound by the terms of Fortinet’s current End User License Agreement. The information in this Customer Support Bulletin is provided for remedial purposes and is designed to assist customers in corrective action that may be helpful to the customer.

Regards,

HA

[treeborg]

This happened on our 100D. Went back and forth with support for over a week before we got the RMA approved.

Fast forward two weeks and I discovered today the replacement 100D has the same issue! Error 500: Internal Server Error and the console logs these every few seconds: EXT3-fs error (device sd(8,1)) in ext3_reserve_inode_write: IO failure

Anyway, we're getting our sales rep involved now. This is a remote office with no on-site IT, so getting these boxes swapped is a nightmare. I can't just keep RMAing the same hardware.

[ecsupport]

Had the exact same issue, until one day it just didnt come back up from a reboot, and RMA fixed for me too. I didnt check for the flash issue in that instance (which is a real problem, I've seen crazyiness like CERTAIN internet names failing to resolve DNS or be pingable as a result).

[dpaul]

msg="EXT2-fs (sdb1): previous I/O error to superblock detected" error showing on Fortigate 100D