Technical Tip: RMA - HQIP test (with built-in FortiOS diagnostic commands)

Description

This article describes troubleshooting with built-in FortiOS hardware diagnostic commands. 

Scope

FortiGate with built-in diagnostic commands (E-Series and newer).

For older hardware that requires a dedicated HQIP test image, follow this article: Technical Tip: HQIP - Hardware Quick Inspection Package.

Solution

HQIP can perform tests on several hardware elements.

While it does not detect all hardware malfunctions, tests for the most common hardware problems are performed.

Note:

To prove that the hardware is faulty, visual evidence may be requested, especially when there is no other proof that the unit is indeed faulty (i.e., for checking the power adapter).

In many cases, the device may appear faulty when it is actually just running low on memory. For the lower-end FortiGate models (smaller than 200F), this does not mean there is a hardware problem. These devices may simply have limited resources and may need some configuration optimization. For more details, check the related articles at the bottom.

Traditionally, hardware diagnostics require users to download the HQIP image and then load the image on the device before the tests can be performed. Starting in FortiOS v5.4 with FortiGate E-Series models, HQIP tests are built into FortiOS itself. This allows users the convenience of performing hardware diagnostics without needing to find the HQIP image and reload the firmware image.

Note:

No HQIP images will be produced for FortiGate E-Series and above that support built-in HQIP commands. When trying to download one of these types of models, the following error will be displayed: 'An appropriate HQIP Image cannot be found for the SN that has been entered.'.

Conversely, for units that require an HQIP image, continue using their designated HQIP images. See the 'Running an HQIP Test' attachment to the related article Technical Tip: HQIP - Hardware Quick Inspection Package, for details.

Precautions and preparations:

• Some tests in HQIP will incur downtime. It is strongly recommended to run the HQIP test during a maintenance window.

• Plan to be local to the firewall to perform this test. Console access is recommended, and select tests require the use of loopback cables and re-cabling of the current device.

• Make sure to have a good backup of the configuration file.

• Have a backup of the current firmware image in case a full recovery of the system is required.

• Perform a factory reset before running the test, if possible, to get the most accurate results ('execute factoryreset').

• A reboot is required if the unit is factory reset config with some uptime.

• Make sure the device is no longer handling production traffic and the CPU and memory are low before performing the test. 

• Enable logging of the console output before the test begins

• The 'Running an HQIP Test' attachment to the related article 'Technical Tip: HQIP - Hardware Quick Inspection Package' has information about wiring the device, console connection, hardware, TFTP server, and connecting to the device. Reference the link if unfamiliar with the aforementioned operations.

What can be tested:
From the CLI, run the following command to list all the hardware test items supported on the device. This command does not execute the actual tests:

diagnose hardware test info

Typically, the output will show test cases under these categories:

• BIOS.

• System.

• Pci.

• Usb.

• Button.

• Cpu.

• Memory.

• Network.

• Npu.

• ASIC.

• Disk.

• LED.

• Wifi.

What the HQIP test does not do:

• Detect all hardware malfunctions. Tests for the most common hardware problems are performed.

• Diagnose issues that cause a device to reboot or be unstable.

• Detect software configuration errors, OS bugs, or OS Kernel Crash issues (one type of OS bug).

• Diagnose devices with multiple Hard Drives.

The built-in hardware diagnostic commands give users the flexibility to run the entire suite of test cases, a group of test cases, or a single test case. Options for performing hardware tests are found under the following command tree:

diagnose hardware test ?

Running the entire test suite (recommended):
There will be interactive prompts throughout the test, so users are advised to remain at the console for the duration of the test. To run the test suite:

diagnose hardware test suite all

Note:

If the Multi-VDOM mode is enabled on FortiGate and using SSH or a console cable to perform the HQIP Test, navigate to the GLOBAL VDOM and then execute the commands as outlined below:

config global
diagnose hardware test suite all

One of the first prompts will advise the users to connect the Ethernet cable:

diagnose hardware test suite all
Please connect ethernet cables:
[WAN - Any of PORT1...PORT4]

 
[WAN - Any of PORT1...PORT4]: This means no other cables should be connected.

Note:

ALL cables are required to be connected by default, and cabling differs significantly from one unit to another. Normally, there would not be enough transceivers or Ethernet cables to perform the test. It is possible to skip the ports that are not required to test by running the following commands before running the HQIP test :

diagnose hardware test skip <interface name>

Example:

diagnose hardware test skip port1-20
diagnose hardware test skip x5-8
diagnose hardware test skip mgmt1

To check the skipped ports:

diagnose hardware test skip show

To clear the skipped ports:

diagnose hardware test skip clear

Further information can be found in this KB article: 

Troubleshooting Tip: How to run HQIP interface test on desired ports only

Also, for certain units, self-loopback cables are required: see Technical Tip: FortiGate HQIP test self-loopback cable - Ethernet RJ45.

Example cable connection below:

Note:

Devices that support switch mode and interface mode will display different cable connection instructions based on the current mode in which the device is running.

Continue with the rest of the test cases and monitor the console for interactive prompts.

When the test is complete, a Test Report will be displayed, showing cases that Passed, Failed, or Skipped (N/A). Save this output for reference.

Running one test category or a single test case:
When running a test category, simply input the chosen category and select Enter. For example, to test the System:

diagnose hardware test system
.....
========================= Fortinet Hardware Test Report =========================
SYSTEM
  CPU Configuration Check....................................... PASS
  Memory Configuration Check.................................... PASS
  Storage Configuration Check................................... PASS
  Network Configuration Check................................... PASS
========================= Fortinet Hardware Test PASSED ========================

 To perform the self-loopback test:

diagnose hardware test network loopback
Network Interface Loopback Test
Please connect ethernet/SFP cables:
[MGMT1 - MGMT2] [HA1 - HA2] [PORT1 - PORT2] [PORT3 - PORT4]
Do you want to continue this test? (y/n) (default is n) y
Test Begin at UTC Time Thu Jul 11 05:53:44 2024

When running a single test case, input the category followed by the test case:

diagnose hardware test system cpu-config
.....
==============Fortinet Hardware Test Report=======
SYSTEM  
CPU Configuration check................................ PASS
=======Fortinet Hardware Test PASSED ===============

After the test:
Also, run this command and attach the output to the FortiCare case:

get system status

If the test shows a failure, the next step is to work with the Technical Assistance Center by opening an RMA ticket. Verify whether the device is still under warranty. Depending on the symptoms that initially led to the test and the results of the test, they may want to verify that it is not a false positive.

Attach the complete HQIP report output to the support ticket for a speedy analysis of the hardware problem.

If the factory reset of the firewall was done, then reload the configurations after the test is completed.

Common requests and issues:

FortiGate does not have the HQIP commands listed above:

The built-in HQIP commands are only supported on FortiOS v5.4 and above on E-Series or newer appliance models. Ensure that the FortiGate meets these criteria.

A non-interactive mode for running the test suite:

This does not exist. However, the test suite typically only takes a few minutes. Memory tests are known to last longer, with the duration scaling with the amount of system memory. In some cases, the test may take up to 8 hours or longer for a device with a very large amount of RAM(300GB+). If time is a concern, consider testing a group of test cases or single cases instead.

Tests overwriting disk data:

Tests do not overwrite disk data. The disk tests will only perform I/O on the disk filesystem and will not overwrite the data.

Explaining the PCBA, stress, and rack-burn-in options when running the 'diagnose hardware test suite':

These commands are for manufacturing uses and are not recommended for end users. Do not perform these tests, as they may take a very long time to run.

Note:

To proceed with an RMA process for the replacement of a device or chassis, specific supporting evidence may be required. As part of the process, it is important to provide visual evidence where the serial number of the device is clearly visible. This step ensures that the necessary verification can be completed.

For those unfamiliar with the process, it is important to note that visual evidence serves as crucial evidence to confirm the details of the device. Without this, the RMA request cannot move forward. Ensure that the serial number is legible in the media provided, as this is a standard requirement to comply with internal procedures for the RMA process and will be requested by the TAC engineer. 

RMA Note: Finding the unit serial number on a FortiGate chassis or appliance

When opening the ticket with Fortinet Support to proceed with the RMA, if the FortiGate is configured in a Cluster/HA, run the commands below on the FortiGate that is operating normally and include the outputs along with the other evidence. This will ensure that the replacement device is compatible with the functioning FortiGate.

get system status
get hardware status

Related articles:

• Technical Tip: FortiGate HQIP test self-loopback cable - Ethernet RJ45

• Technical Tip: HQIP - Hardware Quick Inspection Package

• Technical Tip: 'diagnose hardware test suite all' with non-factory reset configuration 

• Technical Tip: General Troubleshooting Library for Administrators 

• Technical Tip: How to optimize memory consumption for smaller FortiGates