Technical Tip: How to identify and troubleshoot a hard disk/log disk failure with FortiWeb

[Ahmed_Galal]

Description	This article describes how to identify if there is a hard disk / log disk failure and how to troubleshoot it.
Scope	FortiWeb.
Solution	Expected Issues: The following issues may occur if FortiWeb has a disk failure: FortiWeb is not logging/writing any Attack, Event, Traffic logs. FortiWeb cannot boot up And/or stuck in a boot up loop. FortiWeb is not processing the connections and is causing a complete or partial outage. FortiWeb is unable to execute various GUI actions with the error 'Request aborted. Please check your network connection.' Follow this guide to verify that the above issues are caused by disk failure and not any other security module, feature, daemon, or bug. there are two scenarios where hard disk issues may be faced: Scenario 1: FortiWeb is accessible via [GUI,SSH,CLI]: Step 1: Execute the following command: get system status International Version: FortiWeb-VM 7.4.3,build0638(GA),240405 Serial-Number: FVVM01TM22000224 license type: remote Bios version: 04000002 Log hard disk: Not Available Hostname: lister-esx04 Operation Mode: Reverse Proxy FIPS-CC mode: disabled System Uptime: [51 day(s) 23 hour(s) 1 min(s)] Current HA mode: standalone Database Status: Not Available Note 1: If the Database Status is 'Not Available' but the Log hard disk status is 'Available', it is likely to be a DB issue. Refer to this article for steps on how to fix an issue where the DB is not available. Note 2: If the status of both the Log Hard disk and also the Database is 'Not available', proceed with Step 2 below. Step 2: Execute the following command: diagnose system mount list Filesystem 1M-blocks Used Available Use% Mounted on /dev/ram0 473 311 161 65% / none 569 0 569 0% /tmp none 1897 3 1895 0% /dev/shm /dev/sdb1 362 265 78 77% /data /dev/sdb3 91 0 86 0% /home none 200 200 0 100% /var/log ==============> 100% Use 0% Available Note: Here, the /var/log has 0% availability and the size of the partition is only 200 MB. Step 3: Execute the following command: diagnose hardware check all ************************************************ CPU check    Pass core-number Pass 2 cpu-number  Pass 1 frequence     Pass 3700 cache-size    Pass 20480 model-name Pass Intel(R) Core(TM) i3-4360 CPU @ 3.70GHz ************************************************ ************************************************ Memory check Pass Total-size         Pass 8131324 frequence        Pass 1600 ************************************************ ************************************************ logdisk check Fail size                Fail 234 disk-number   Fail 1 ************************************************ ************************************************ NIC check      Pass num               Pass 8 Giga nic num Pass 8 10G nic num Pass 0 ************************************************ Note: The status of the logdisk check is 'Fail', along with the size and disk-number. Step 4: Execute the following command: diagnose hardware check logdisk logdisk check Fail size Fail 15 disk-number Fail 1 raid-level Fail no raid exists Note: The status of the logdisk check is 'Fail'. Scenario 2: FortiWeb is not accessible via the GUI, SSH, or the CLI and is stuck in the boot up process: Any of the following errors may be seen in the boot up log output: /dev/sda1: recovering journal /bin/e2fsck: No such device or address while trying to open /dev/sdb3 --------------------------------------- write error: failed to open new log file for writing: failed to open new file '/var/log/filebeat/filebeat-20240518.ndjson': open /var/log/ FWB login: find: /var/log/debug/tmp/BELGFWB01_230516_1449.nmon: Read-only file system --------------------------------------- FWB $ [1194099.855910] blk_update_request: I/O error, dev sdb, sector 3241911 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 [1194101.266938] blk_update_request: I/O error, dev sdb, sector 15828359 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 [1194101.554921] blk_update_request: I/O error, dev sdb, sector 3241911 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 [1194101.772904] blk_update_request: I/O error, dev sdb, sector 12164655 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 [1194101.933935] blk_update_request: I/O error, dev sdb, sector 4538503 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 --------------------------------------- FWB login: mln sample list not ready. [tree: error]<redisconn.c:158> Redis server is not ready, will try redis auth again later ... [tree: error]<redisconn.c:158> Redis server is not ready, will try redis auth again later ... --------------------------------------- How to Fix a hard disk/ log disk failure: Scenario 1: FortiWeb is accessible via the GUI, SSH, and/or the CLI: Rebuild the DB and format the log disk (doing so during a maintenance window is recommended as formatting the logdisk will trigger a reboot:( execute db rebuild execute formatlogdisk Note 1: Formatting the log disk will perform a reboot and will erase the current logs. The config file will not be impacted. To create a backup of the current logs, either: Use a FortiAnalyzer or similar device. Download the logs locally on the client machine: Log&Report -> Log Access -> Attack: Right-click the [Log management] icon at the top right of the page and select download. Log&Report -> Log Access -> Event: Right-click the [Log management] icon at the top right of the page and select download. Log&Report -> Log Access -> Traffic: Right-click the [Log management] icon at the top right of the page and select download. Note 2: After the reboot, check the status of 'Log hard disk' (as mentioned in step 1 of the verification section). If the status remains as 'Not available', proceed with the following 'clean installation' step, which should also be performed if the FortiWeb GUI and CLI are not accessible. Scenario 2: FortiWeb is not accessible via the GUI, SSH, or the CLI, is stuck in the boot up process, or the previous step 'DB rebuild, Format logdisk', did not fix the issue: Perform a clean install: Upload the FortiWeb image to a TFTP server on the client machine. Connect the management computer to the FortiWeb console port using a RJ-45-to-DB-9 serial cable or a null-modem cable and Initiate the console connection. Connect port1 of the FortiWeb appliance directly or to the same subnet as a TFTP server 'Step 1'. Verify that the TFTP server is currently running, and that the FortiWeb appliance can reach the TFTP server: execute ping 192.0.2.168 Where 192.0.2.168 is the IP address of the TFTP server. Execute reboot. As the FortiWeb appliance starts, a series of system startup messages appear. Press any key to display the configuration menu. Enter 'F' to Format the boot device. Enter 'G' to retrieve the image from the TFTP server. Check the troubleshooting guide for further details on how to restore firmware ('clean install'): Note: If the issue persisted after the previous steps, it is most likely a hardware related issue. Collect the following debug command outputs and open a support ticket: Scenario 1: FortiWeb is accessible via the GUI, SSH, and/or the CLI: get system status get system performance diagnose system mount list diagnose debug crashlog show diagnose debug coredumplog show diagnose hardware check all diag hardware logdisk info diag hardware harddisk list diag hardware harddisk errors fn dmesg For logdisk hardware-related issues, please follow the below diagnostic CLI commands (supported in hardware platform): execute smart self-test Then wait for the self-test completes, several minutes. You could see the info like below: Testing has begun. Please wait 2 minutes for test to complete. After it completes, get the result by using thess CLI commands: execute smart test-result execute smart info Scenario 2: FortiWeb is not accessible via the GUI, SSH, and/or the CLI: Collect the bootup outputs.