900D boot loop cycle

[fg66]

Hello!

I hope someone can help with a boot loop issue on a 900D.

Please excuse the NOOB description, as I am just coming into this job and have a non functioning device.

Putty is being used for CLI serial access to the device.

I have had to format the boot device, as I am starting from scratch--which appears to go fine.

Then I load from TFTP (tftpd64 program) on MGT port 1 the downloaded OS version 7.0.0 (checksum verified).

That goes well, no errors.

Next I am prompted to choose the boot program as Default/Backup/ etc.

I choose Default (I also have tried backup).

The system then goes into the "Programming firewall" - again, no errors.

After that the system reboots.

Here is the console out put that is concerning, and I am not sure what to make of it.

Sorry for the long detail, but perhaps someone can decipher it:

FortiGate-900D ( 9:56-10.13.2017)
Ver:05000004
Serial number:123456789
RAM activation
CPU(00:000306c3 bfebfbff): MP initialization
CPU(02:000306c3 bfebfbff): MP initialization
CPU(04:000306c3 bfebfbff): MP initialization
CPU(06:000306c3 bfebfbff): MP initialization
Total RAM: 16384MB
Enabling cache...Done.
Scanning PCI bus...Done.
Allocating PCI resources...Done.
Enabling PCI resources...Done.
Zeroing IRQ settings...Done.
Verifying PIRQ tables...Done.
Boot up, boot device capacity: 1918MB.
Press any key to display configuration menu...
.............................

Reading boot image 2855142 bytes.
Initializing firewall...
System is starting...
NP6: np6_hw_per_init:Failure to probe EEPROM master
NP6: E2PROM: Invalid read operation at 0
NP6: Board configuration load fail, initialize as nonfunctional Dummy board
NP6: got dummy board
BUG: unable to handle kernel NP6: register misc device np6_1 @ [10, 212]
NP6: total 2 devices probed and initialized
NP6: driver init succeeded
cp8 0000:03:00.0: PCI->APIC IRQ transform: INT A -> IRQ 18
cp8 0000:03:00.0: irq 61 for MSI/MSI-X
cp8 0000:03:00.0: irq 62 for MSI/MSI-X
cp8 0000:03:00.0: irq 63 for MSI/MSI-X
cp8 0000:03:00.0: irq 64 for MSI/MSI-X
cp8 0000:03:00.0: irq 65 for MSI/MSI-X
cp8_set_smp_affinity: irq 61 affinity to 0, idx: 0
cp8_set_smp_affinity: irq 62 affinity to 1, idx: 0
cp8_set_smp_affinity: irq 63 affinity to 2, idx: 0
cp8_set_smp_affinity: irq 64 affinity to 3, idx: 0
cp8_set_smp_affinity: irq 65 affinity to 0, idx: 0
cp8 0000:04:00.0: PCI->APIC IRQ transform: INT A -> IRQ 19
cp8 0000:04:00.0: irq 66 for MSI/MSI-X
cp8 0000:04:00.0: irq 67 for MSI/MSI-X
cp8 0000:04:00.0: irq 68 for MSI/MSI-X
cp8 0000:04:00.0: irq 69 for MSI/MSI-X
cp8 0000:04:00.0: irq 70 for MSI/MSI-X
cp8_set_smp_affinity: irq 66 affinity to 0, idx: 1
cp8_set_smp_affinity: irq 67 affinity to 1, idx: 1
cp8_set_smp_affinity: irq 68 affinity to 2, idx: 1
cp8_set_smp_affinity: irq 69 affinity to 3, idx: 1
cp8_set_smp_affinity: irq 70 affinity to 0, idx: 1
IPSA device on (PCI Bus 0000:03) mapped to cluster 0 master
IPSA device on (PCI Bus 0000:04) mapped to cluster 0 slave
found IPSA device at PCI Bus 0000:03, phys_addr ef600000, len 8000
found IPSA device at PCI Bus 0000:04, phys_addr ef700000, len 8000
IPSA register misc device ipsa_0 @ [10, 226]
===========>ha_init
===========>ip_session_init: table_size=0
paging request at fffffffffffffef0
IP: [<ffffffffa0144abe>] np6_lif_watchdog_update_link+0x4/0xff [filter4]
PGD 6e1067 PUD 6e2067 PMD 0
Oops: 0000 [#1] SMP
CPU 2

Modules linked in:
filter4(P+)

Pid: 137, comm: np6_1 Tainted: P 3.2.16 #2
RIP: 0010:[<ffffffffa0144abe>] [<ffffffffa0144abe>] np6_lif_watchdog_update_link+0x4/0xff [filter4]
RSP: 0018:ffff8803afd4be80 EFLAGS: 00010282
RAX: 0000000000000000 RBX: ffffc90024228170 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000282 RDI: fffffffffffffee0
RBP: ffffc90024228000 R08: ffff8803afd48000 R09: 0000000000000000
R10: ffff88043fd18818 R11: 0000000000000000 R12: fffffffffffffee0
R13: ffffc90024260e78 R14: ffffc90024260e28 R15: 0000000000000000
FS: 0000000000000000(0000) GS:ffff88043fd00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: fffffffffffffef0 CR3: 0000000429392000 CR4: 00000000001406a0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process np6_1 (pid: 137, threadinfo ffff8803afd48000, task ffff880428e21750)
Stack:
ffffc90024228170 ffffc90024228000 fffffffffffffee0 ffffffffa015348f
0000000000000000 ffffc900242280a8 ffff8803afd4bee0 ffff880423073c28
ffffc90024260e78 ffffffffa01533e6 0000000000000000 0000000000000000
0000000000000000 ffffffff80275140 0000000000000000 0000000000000000
ffffc90024260e78 ffff880400000000 0000000000000000 ffff8803afd4bf18
ffff8803afd4bf18 00000000e21af97d 0000000000000000 0000000000000000
0000000000000000 ffffffff8059b6b4 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000
ffffffff802750bc ffff880423073c28 ffffffffffffffff ffffffff8059b6b0
0000000000000010 0000000000000202 ffff8803afd4bf58 0000000000000018
Call Trace:
[<ffffffffa015348f>] ? np6_system_np6_thread_should_stop+0xf1/0x1f2 [filter4]
[<ffffffffa01533e6>] ? np6_system_np6_thread_should_stop+0x48/0x1f2 [filter4]
[<ffffffff80275140>] ? kthread+0x84/0x93
[<ffffffff8059b6b4>] ? kernel_thread_helper+0x4/0x10
[<ffffffff802750bc>] ? kthread_flush_work_fn+0x9/0x9
[<ffffffff8059b6b0>] ? gs_change+0xb/0xb
Code: e8 22 fe ff ff 31 c0 48 83 c4 08 c3 31 c0 c3 e8 f0 1e 01 00 31 c0 48 83 c4 08 c3 e8 87 a6 00 00 31 c0 48 83 c4 08 c3 41 54 55 53 <48> 8b 5f 10 48 8b bf e8 00 00 00 48 8b af a0 00 00 00 48 8b 47
RIP [<ffffffffa0144abe>] np6_lif_watchdog_update_link+0x4/0xff [filter4]
RSP <ffff8803afd4be80>
CR2: fffffffffffffef0
---[ end trace 1c85799ac0bbe5da ]---
Kernel panic - not syncing: Fatal exception
Pid: 137, comm: np6_1 Tainted: P D 3.2.16 #2
Call Trace:
[<ffffffff8059634c>] ? panic+0xa2/0x19e
[<ffffffff8020483e>] ? oops_end+0x8a/0x8a
[<ffffffff8021719b>] ? no_context+0xe5/0x2a8
[<ffffffff802179ff>] ? do_page_fault+0x238/0x409
[<ffffffff8025613a>] ? load_balance+0x9b/0x59f
[<ffffffff8024f5aa>] ? update_curr+0x3f/0x90
[<ffffffff80250b8c>] ? dequeue_task_fair+0xdd/0xf9
[<ffffffff80599f6f>] ? page_fault+0x1f/0x30
[<ffffffffa0144abe>] ? np6_lif_watchdog_update_link+0x4/0xff [filter4]
[<ffffffffa015348f>] ? np6_system_np6_thread_should_stop+0xf1/0x1f2 [filter4]
[<ffffffffa01533e6>] ? np6_system_np6_thread_should_stop+0x48/0x1f2 [filter4]
[<ffffffff80275140>] ? kthread+0x84/0x93
[<ffffffff8059b6b4>] ? kernel_thread_helper+0x4/0x10
[<ffffffff802750bc>] ? kthread_flush_work_fn+0x9/0x9
[<ffffffff8059b6b0>] ? gs_change+0xb/0xb
Rebooting in 5 seconds..FortiGate-900D ( 9:56-10.13.2017)

Would this be any type of fixable issue, or is it a call to support/I have a paperweight?

Thank you all very much for any insights!

[srajeswaran]

Looks like system is entering kernel panic state, if you have active support please open a ticket to investigate the core/crash .
If you don't have support, can you try using any 6.4.x version instead of 7.0.0 ?

[fg66]

Thank you for the reply. I downloaded the 6.4 code, formatted the boot device and once again everything seemed to go well, but now I get a "Open boot device failed" message.

[srajeswaran · Accepted solution]

Hi @fg66 Thanks for testing 6.4. I think we need to proceed with RMA now.

[Christian_89]

The issue you're experiencing, characterized by a boot loop and a kernel panic, seems to be related to a failure in the hardware or firmware of the device. The error messages you've shared indicate a failure to initialize the hardware properly, specifically the Network Processor (NP6).

There are a few potential causes for this issue:

1. **Corrupted firmware:** If the firmware on the device is corrupted, it could cause the device to fail during boot. You can try reinstalling the firmware to see if this resolves the issue.

2. **Hardware failure:** The error messages seem to suggest a hardware failure. In this case, it might not be possible to fix the issue without replacing the failed hardware.

3. **Incompatible firmware:** If you're using a firmware version that isn't compatible with your hardware, it could cause issues during boot. Ensure you're using a firmware version that's compatible with your FortiGate 900D.

Here are a few steps you can try:

- **Reinstall the firmware:** You can try reinstalling the firmware via TFTP again. Ensure you're using the correct firmware for your device and that the file isn't corrupted.

- **Reset to factory defaults:** If you can access the CLI, you can try resetting the device to factory defaults using the "execute factoryreset" command. Be aware that this will erase all configuration on the device.

- **Contact support:** Given the complexity of this issue, it might be best to contact Fortinet's support. They can provide more detailed troubleshooting and potentially arrange for a hardware replacement if necessary.

Please remember to backup any configuration before attempting any recovery methods. Let me know if you need help with any of these steps.

[fg66]

Hi Christian!

Thank you for the reply. I downloaded the 6.4 code, formatted the boot device and once again everything seemed to go well, but now I get an "Open boot device failed" message.

I think this may turn out to be an RMA.

[kvimaladevi]

Hi, 

If the issue is not resolved even after formatting, it should be a device issue. Please raise an RMA request.

Regards,

Vimala

[parteeksharma]

Dear fg66,

As per the logs messages shared it seems device is stuck in boot loop and kernel panic, it might be caused due to failure in the hardware or firmware of the device.  Please try flash formatting the device and try reinstalling the firmware using TFTP and then check if device boots up. Please follow below KB for flash format and reinstalling the firmware using TFTP:

https://community.fortinet.com/t5/FortiGate/Technical-Tip-FortiGate-Flash-Format-process/ta-p/198704

https://community.fortinet.com/t5/FortiGate/Technical-Tip-Formatting-and-loading-FortiGate-firmware-...

Regards,
Parteek