FortiDDoS Upgrade Loop

[AtiT]

Hi,

I have a FortiDDoS-400B and provided upgrade to version 4.3.1. The unit went to upgrade loop like it is shown below. Is it a normal behaviour?

System is started.

FortiASIC-TP.0: update started. Reconfigure process takes a few minutes FortiASIC-TP.0: 0% Complete FortiASIC-TP.0: 10% Complete FortiASIC-TP.0: 20% Complete FortiASIC-TP.0: 30% Complete FortiASIC-TP.0: 40% Complete FortiASIC-TP.0: 50% Complete FortiASIC-TP.0: 60% Complete FortiASIC-TP.0: 70% Complete FortiASIC-TP.0: 80% Complete FortiASIC-TP.0: 90% Complete FortiASIC-TP.0: download complete. FortiASIC-TP.0: update success (256.521233 s). FortiASIC-TP.0: chip reconfigure, try again FortiASIC-TP.0: chip reconfigure... success. FortiASIC-TP.0: update finished rebooting FortiDDoS-400B (14:23-02.03.2014) Ver:04000007 Serial number:FI400Bxxxxxxxxxx RAM activation CPU(00:000306a9 bfebfbff): MP initialization CPU(01:000306a9 bfebfbff): MP initialization CPU(02:000306a9 bfebfbff): MP initialization CPU(03:000306a9 bfebfbff): MP initialization CPU(04:000306a9 bfebfbff): MP initialization CPU(05:000306a9 bfebfbff): MP initialization CPU(06:000306a9 bfebfbff): MP initialization CPU(07:000306a9 bfebfbff): MP initialization Total RAM: 8192MB Enabling cache...Done. Scanning PCI bus...Done. Allocating PCI resources...Done. Enabling PCI resources...Done. Zeroing IRQ settings...Done. Verifying PIRQ tables...Done. Boot up, boot device capacity: 15272MB. Press any key to display configuration menu... ......

Reading boot image 2791105 bytes. Initializing FortiDDoS...˙

System is started.

FortiASIC-TP.0: update started. Reconfigure process takes a few minutes FortiASIC-TP.0: 0% Complete

...........

[SteveDDoS_FTNT]

If it does this twice and stops, it is normal, particularly if you are upgrading from a release below 4.2.3.  FortiDDoS has 2 hardware components. The "normal" x86 CPU infrastructure (the 2nd part of the boot shown above) and the TP2 processor which is an FPGA - essentially a custom processor similar to the FortiGate ASICs but field re-configurable. Configuration of the TP2 takes a long time because the gates within the device are being re-configured to change the processor's functions.  It is not like adding firmware to RAM. If you are upgrading from pre-4.2.3, this is done twice.  The first time, baseline code is added to the TP2 - this will improve upgrades in future.  Once that is done the system goes through its normal x86 initialization but then starts a new cycle to add the 4.3.1 firmware to the TP2, after which it should again initialize the x86 CPUs and then complete.

I think the Release Note says that several reboots will occur.

If it is stuck in this boot loop then the device is likely RMA, although you have console access. If it has stopped booting and you have console access, login and at the command prompt type:

diagnose debug rrd_cmd_check

the console will scroll with a % complete notification - this can also take a long time - 5 minutes or more.  It will eventually say "rrd command check successful" or "failed".  If failed, please contact TAC.

[AtiT]

Hello Steve,

Thank you for the update.

The boot loop not finished so I tried to upload the image as backup image and boot this backup image.

The upgrade looks differnet:

FortiASIC-TP.0: update started. Reconfigure process takes a few minutes FortiASIC-TP.0: 0% Complete FortiASIC-TP.0: 10% Complete FortiASIC-TP.0: 20% Complete FortiASIC-TP.0: 30% Complete FortiASIC-TP.0: 40% Complete FortiASIC-TP.0: 50% Complete FortiASIC-TP.0: 60% Complete FortiASIC-TP.0: 70% Complete FortiASIC-TP.0: 80% Complete FortiASIC-TP.0: 90% Complete FortiASIC-TP.0: FPGA image download complete. FortiASIC-TP.0: Checking update image on FPGA..... FortiASIC-TP.0: Checking update image on FPGA.....OK, GBL_RUPD_RECONFIG_STAT = 0x3 FortiASIC-TP.0: UPDATE FPGA OK, WAIT FOR REBOOT.... FortiASIC-TP.0: update finished

There is no "chip reconfigure, try again" but messages about FPGA.

The debug:

# diagnose debug rrd_cmd_check rrdmd5 Total 21504 files scanned... Verifying the integrity of RRD commands please wait... 20 %complete Checksum mismatch for /home/rrdcmds/ICMP_45568_1_1m RRD commands check failed

I will try to format the flash and upload the image once again.

[SteveDDoS_FTNT]

Sorry for the delay on this.  That looks like a normal upgrade.  The chip reconfigure message you were getting before should happen if you are upgrading to 4.2.3 or 4.3.x. Those upgrades did extra TP2 installations to help with future upgrades.  However, the rrd_cmd info shown indicates something did not work on the upgrade - probably the logdisk.

Do you have GUI access?  Does the Dashboard show the correct Firmware Release? if so,

you can try 2 things:

diagnose debug rrd_cmd_recreate - this will fix graphing problems if it works.

Failed rrd_cmds show up as missing labels on the Monitor Graphs - they will display v0, v1, etc. instead of a text label. You can usually check by looking at the Aggregate Drop Graphs. A failed graph "under" graph this rolls up and fails this graph. 

Last resort:

execute formatlogdisk - this will delete all data but it will rebuild all the rrds and rrd_cmds, which are critical to system operation. (rrd commands are separate from the rrds themselves).

Otherwise please start a ticket or reach out to me directly at stephen@fortinet.com and we will try to recover the unit.

Steve Robinson