Technical Guide: How to perform an RMA replacement of a FortiGate

[pjang]

Description	This article describes the process of performing an RMA replacement of a defective FortiGate, including: Initial steps for determining if an RMA replacement is required. Preparing for RMA. Opening a ticket with Fortinet TAC/RMA. License Transfers. Preparing and deploying the replacement FortiGate. Returning the defective FortiGate and completing the RMA. While this process is similar for Fortinet products in-general, this article is written specifically for handling FortiGates.
Scope	FortiGate, RMA.
Solution	Before anything else, review the FortiCompanion to RMA Services document available for download from the Fortinet Support Site's documents section. This document includes detailed information for everything related to the Fortinet RMA process, including the scope of support included in each tier of Hardware Support (Return and Replace, Advanced Replacement, Priority RMA (PRMA) and Secure RMA), explanations/definitions for RMA ticket statuses, and guidance on how Fortinet handles defective units that clients have returned as part of the RMA process. Table of Contents: Initial steps for determining if an RMA replacement is required. Preparing for RMA. Opening a ticket with Fortinet TAC/RMA. License Transfers. Preparing and deploying the replacement FortiGate. Returning the defective FortiGate and completing the RMA. Initial steps for determining if an RMA replacement is required. As a reminder, RMA replacements are used specifically to resolve hardware-related issues with the FortiGate and other Fortinet products); software-specific issues are highly unlikely to be resolved by an RMA replacement and will generally be denied unless there are clear indicators that they are caused by hardware issues. For example, kernel panics on the FortiGate may be caused by hardware issues (and with confirmation from Fortinet TAC can warrant hardware replacement), whereas FortiGates entering conserve mode largely occur due to software-related problems (and so hardware replacement will not resolve the issue). Important Note 1: all RMA requests require Fortinet TAC to verify the hardware failure based on gathered evidence (discussed further below). However, clients whose FortiGates have Priority RMA (PRMA) can choose to refuse troubleshooting and proceed straight to RMA. Important Note 2: Per the Fortinet Product License Agreement / EULA, FortiGates subjected to abnormal physical or electrical stress, misuse, negligence, or accident are not eligible for RMA. This can include units that have been subjected to lightning strikes and is true for all levels of RMA Hardware Support (see '7. Disclaimer of Other Warranties and Restrictions' in above EULA). To determine if a FortiGate is facing a hardware issue that can warrant an RMA replacement, check for the following conditions before opening a TAC ticket (this reduces the time needed to perform and RMA). Note that it is mandatory to have someone onsite with the FortiGate when validating, as hardware verification frequently requires that someone be able to make physical network/serial console connections to the FortiGate, check status LEDs, take pictures, etc. FortiGate failing to boot: The status LEDs on the FortiGate are either not lit or are showing alarm colors. Each FortiGate model includes a Quick Start Guide (QSG) document that details the available status LEDs and the different statuses they may have. See also: FortiGate Hardware Documents. Unlit status LEDs can be a good indicator for an RMA being necessary, but it is recommended to perform additional validation to ensure that an RMA can be justified. It is strongly recommended to have someone take pictures and videos of the suspect FortiGate, especially of the status LEDs. This is good evidence that can help to further justify an RMA. The FortiGate is unresponsive to serial console connections. This can be verified by connecting a laptop/workstation to the FortiGate via a serial console to USB adapter and sending some keyboard inputs to check for a response. Ensure that the serial console cable being used is guaranteed to be working correctly. The recommended method for verifying the console cable is to use it to connect to another known-working device (such as a network switch, another FortiGate, etc.) If there is no response on the serial console and the FortiGate is not accessible at all over the network then an RMA can be justified. The FortiGate reliably fails to boot up successfully after being power cycled. During the bootup process, ensure that a serial console connection to the FortiGate is active with the output logged to a text file (this evidence helps to justify an RMA). If there is no output on a known-good console cable then the FortiGate may be failing to initialize the BIOS/boot successfully, and so RMA can be justified. If there is output, then capture the output and ensure that it is included when open a TAC ticket. For FortiGates with external power supplies, try using a spare FortiGate power supply if available. For FortiGates with multiple power supplies, test all possible inputs to determine if the FortiGate can operate correctly with at least one power supply option. This is to rule out if an individual power supply module is defective. If the issue is narrowed down to a specific power supply module then it is possible to RMA just that module. It is strongly recommended to test the FortiGate with both an uninterruptible power supply (UPS) as well as direct power. In some cases, UPS power output can be incompatible with the FortiGate and prevent it from operating correctly, so validating against wall power can be useful for identifying these cases. In cases where the FortiGate is suddenly becoming unreachable without warning, have a laptop connected to the FortiGate with a logged serial console connection to capture any debug output. Larger FortiGate models come equipped with the COMLog feature, which is local flash storage that can record serial console output (useful during unexpected outage events). See also: Technical Tip: How to use the COMLog feature FortiGate operating correctly, partial hardware issue: If one of the network interfaces is non-functional, check and confirm that the physical cabling is not the problem. For Ethernet cable, try replacing the cabling between the FortiGate and its neighbor device. Alternatively, try connecting a known-good device (like a laptop with an Ethernet port/adapter) to the FortiGate directly to see if a link establishes. For optical fiber, try replacing the optical transceiver with a known-good spare, or try swapping from fiber to direct-attach copper (DAC) cables instead, if available). Use the FortiGate HQIP commands to perform additional troubleshooting: Technical Tip: HQIP - Hardware Quick Inspection Package. The HQIP commands are built-in to E-series and later FortiGates and allow administrators to run hardware diagnostic tests directly on the FortiGate. Take note that some of these tests can require temporary downtime on the FortiGate, and so a maintenance window should be scheduled to run these diagnostics. Common tests to run include disk read/write tests, network interface detection and loopback tests, and system memory read/write tests. If there are enough indicators present to confirm that a hardware issue is present and that an RMA is justified, then gather the evidence and move to the next section of this guide. Preparing for RMA. Before opening a ticket, ensure that all gathered evidence is on-hand and available to be uploaded. The previous section mentions this, but presenting the following to TAC can expedite the handling of RMA cases: A clear and concise explanation of the suspected hardware failure. Photos and videos clearly demonstrating hardware failures, Text files containing output from serial console testing or COMLog output, such as output captured when attempting to boot up the FortiGate or kernel panic messages produced during unexpected outages. Text files containing the output of any additional testing performed, such as running HQIP tests or running FortiOS debug commands on a running FortiGate. Additionally, it is recommended to explicitly state the following in the initial ticket statement, even if the information is generally included when opening the ticket: Serial Number of the suspect FortiGate. See also: RMA Note: Finding the unit serial number on a FortiGate chassis or appliance Technical Tip: How to get the serial number via CLI and GUI Whether the FortiGate is a member of an HA cluster or is standalone. Opening a ticket with Fortinet TAC/RMA. Generally speaking, it is recommended to open a Technical Support Ticket when initiating a request for a FortiGate RMA, rather than starting with a DOA/RMA Ticket. This is because Fortinet RMA requires that TAC works with the client to validate that a hardware replacement is required, and so cases that are opened as DOA/RMA will often be moved back to TAC anyway if it is unclear that there is a confirmed hardware issue. When opening a Technical Support Ticket, ensure that all of the evidence gathered in the previous step is uploaded to the ticket. With sufficient evidence available at the beginning of the ticket, TAC can be confident that a hardware issue does in-fact exist and the ticket can be moved more quickly to the RMA team for further handling. To open a ticket to request RMA, log into the Fortinet Support Site (using a user account that has permission to open FortiCare tickets), select the Support dropdown in the upper navigation bar, then select FortiCare - > Manage Tickets. Next, select the New Ticket button on the right side, then select Technical Support Ticket -> Submit Ticket. Follow the steps specified by the Fortinet Support site, and ensure that a detailed explanation is provided in the Comment section, and that the files gathered previously are successfully uploaded as Attachments. Note: in cases where there is very clear evidence of a hardware issue (such as dead-on-arrival (DOA) hardware), it can be OK to open a DOA/RMA Ticket directly, rather than going through a Technical Support Ticket for validation. Make sure to have this evidence available on the ticket, otherwise it may be redirected to TAC for validation. License Transfers. Once validation of the hardware issue is completed, TAC will move the ticket over to the RMA team. The RMA ticket will prompt clients to confirm the shipping and billing address to be used, and it will also ask if the RMA team should 'Automatically transfer all active services and support contracts to a new unit'. If this option is selected then Fortinet will automatically register the replacement unit when it is shipped out, and Fortinet will also automatically transfer all contracts and licenses associated with the defective unit to the replacement unit. Important: Do not select this option if the defective FortiGate is still active in production and handling traffic, either as a standalone unit or as an active member of an HA cluster. If the licenses are transferred away from the defective FortiGate then license-dependent features like FortiGuard Category lookups for Web Filtering will stop functioning, which could lead to further impact to the network. Only select the above option if the defective FortiGate is already fully offline and/or removed from the production environment. See also: RMA Note: Contract and Service Auto Transfer Option If the above option is not checked then the licensing will remain with the defective FortiGate for the time being. The RMA ticket will include steps so that clients can initiate the license/contract transfer after the replacement FortiGate is received (see also: Customer Service Tip: How to register Fortinet products in the Fortinet Support Portal). Preparing and deploying the replacement FortiGate. Once the replacement FortiGate is received, it will need to be prepared to take the place of the defective FortiGate. How this should be approached will depend on circumstances of the current deployment, but the first step is to migrate the existing configuration: If the defective FortiGate is a standalone and also still functional (i.e., administratively accessible over the network), then administrators should immediately take a backup of the configuration if not already done. Review the following section of the FortiGate Admin Guide for guidance on backing up/restoring the configuration on the FortiGate: Configuration backups and reset Where possible, take the backup using the GUI, as this produces a properly formatted .conf file that can be restored via the GUI to the replacement FortiGate. Additionally, take note of the specific FortiOS firmware version that is currently installed on the defective FortiGate, as the version must be matched on the replacement FortiGate in order for the config restoration to succeed. If the defective FortiGate is standalone but non-functional (i.e., failure to boot), then a backup will not be able to be created. If possible, check and utilize a backup that was taken from the FortiGate previously (e.g., taken as part of a scheduled configuration backup job, or retrieved from FortiManager). If no configuration backups exist then the FortiGate will have to be manually re-configured by the administrator. Check the following documentation regarding first-time setup of a new FortiGate: Getting started If the defective FortiGate is a member of an HA cluster (regardless of functional status), then take a configuration backup from the working HA primary FortiGate in the cluster. While it is possible to connect the replacement FortiGate to the existing HA cluster and pull all configurations via HA sync, this can occasionally fail to synchronize and apply the configuration successfully (especially with complex configurations). Instead, it is generally safer to restore an up-to-date configuration backup to the replacement FortiGate before it is integrated with the rest of the HA cluster, as this minimizes the amount of HA config sync that is required and greatly reduces the chances of HA out-of-sync errors. Once a configuration backup is available, the next step is to ensure that the replacement FortiGate is running the same firmware version as the original defective FortiGate. To do this, there are several options available: The most recommended option is to fresh-install firmware via TFTP. This is the most direct method, as it allows administrators to go directly to the target firmware version and have a clean installation. For guidance on performing a TFTP firmware install on the FortiGate, refer to the following KB article: Technical Tip: Formatting and loading FortiGate firmware image using TFTP Alternatively, the firmware on the replacement FortiGate can be upgraded to match the original FortiGate's version. It is strongly recommended to follow the supported upgrade path when performing this upgrade: Fortinet Upgrade Path Tool. This may take longer compared to a TFTP install (since multiple firmware versions may need to be installed) but is necessary to minimize the risk of unexpected issues.  Note that firmware downgrades are not supported or recommended by Fortinet TAC, as the firmware is not designed to accommodate downgrades and it can cause unexpected issues. See also: Technical Tip: FortiGate Firmware Downgrade for Minor Releases Once the replacement FortiGate is running the same version as the original FortiGate, the configuration backup can be restored. In cases where a config backup is not available, the administrator will need to manually re-configure the FortiGate before deploying the replacement (see 'Getting started' linked above). For more in-depth instructions for restoring FortiGates after RMA (standalone, HA, and chassis), refer to the following KB articles: Technical Tip: How to restore a FortiGate appliance after RMA Technical Tip: How to restore a FortiGate HA cluster after RMA Technical Tip: How to restore a chassis based FortiGate after RMA Once the replacement FortiGate is configured, it is ready to take the place of the original defective FortiGate. It is strongly recommended to schedule a maintenance window for this operation and to have someone available onsite that has physical access to the FortiGate as well as a workstation/laptop with an Internet connection (for example, using a cellular hotspot). This is especially necessary if TAC assistance is required, as Fortinet TAC can only assist if some kind of administrative access to the FortiGate is available (otherwise troubleshooting cannot occur). Note regarding FortiToken on the FortiGate: When a FortiGate is replaced via RMA, any FortiTokens assigned to users on the FortiGate must be deleted, re-registered, and re-assigned (it is not possible to restore the configuration backup and have a functioning FortiTokens on the new FortiGate). Refer to the following KB articles regarding FortiToken registration, transfer, and re-assignment following RMA: RMA Note: Self-serve FortiToken reset during RMA transfer Technical Tip: FortiToken register and provision process after RMA Technical Tip: FortiToken register and provision process after RMA in HA environment Returning the defective FortiGate and completing the RMA. After the replacement FortiGate is deployed to production, the defective FortiGate must be returned to Fortinet. Note that while Fortinet covers the cost of shipping the replacement unit to the client, the client is responsible for any import duties, taxes, or other fees associated with receiving this shipment. Additionally, the cost of return shipping for the defective FortiGate is also the responsibility of the client, unless the FortiGate has Priority RMA (in which case Fortinet also covers the cost of return shipping). Follow the instructions provided on the RMA ticket for shipping the defective FortiGate back to Fortinet. Note regarding data deletion: As discussed in the FortiCompanion to RMA Services document (under 'Handling Procedure for Customer-Returned Product'), all products returned to Fortinet have their media formatted and configuration reset to factory defaults. From there, the product has the hardware failure verified and the unit is either refurbished if viable or scrapped by a certified vendor if not (this includes physical destruction of data media). Clients in need of maximum security should strongly consider purchasing the Secure RMA service, which allows for the non-return of defective hardware and the protection of data from within the client's premises. This is a separate supplementary license that can be added per product for all RMA replacement levels.