Help
Support Forum
The Forums are a place to find answers on a range of Fortinet products from peers and product experts.
spom
New Contributor

Created on ‎10-21-2019 11:20 AM

Working on 200D (5.4.3) that is suddenly crashing randomly

Got pulled in. Its spontaneously crashing, and just started this morning. Total hang, no console output or anything. Restart, disk check, and it might be good for 3 hours, it might be good for 10 minutes.

 

Caught a crash while I was already connected to the console (below, with a few instances of it repeatedly crashing). Any suggestions on troubleshooting? It's out of support contract so no updates available, and the cost is high enough that they aren't looking to renew something they feel is janky (no opinions on my end, I just need to resolve it one way or another).

 

 

RAM activation
CPU(00:000206a7 bfebfbff): MP initialization
CPU(02:000206a7 bfebfbff): MP initialization
Total RAM: 4096MB
Enabling cache...Done.
Scanning PCI bus...Done.
Allocating PCI resources...Done.
Enabling PCI resources...Done.
Zeroing IRQ settings...Done.
Verifying PIRQ tables...Done.
Boot up, boot device capacity: 15272MB.
Press any key to display configuration menu...
......

Reading boot image 1393095 bytes.
Initializing firewall...
System is starting...
Starting system maintenance...
Scanning /dev/sda2... (100%) 
pid=1 get sig=11 fault:0x9a5c9d50
<00000> Backtrace:
<00000> [0x0042bee4] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x2a960f4d00] => /fortidev4-x86_64/lib/libc.so.6 
<00000> [0x016e034e] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x016e03c7] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x016e03c7] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x016e0428] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x016e0aee] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x016d49a3] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x0042db18] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x2a960e1475] => /fortidev4-x86_64/lib/libc.so.6 (__libc_start_main+0x000000f5) 
pid=1 get sig=11 fault:0x1c
<00000> Backtrace:
<00000> [0x0042bee4] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x2a960f4d00] => /fortidev4-x86_64/lib/libc.so.6 
<00000> [0x0042bee4] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x2a960f4d00] => /fortidev4-x86_64/lib/libc.so.6 
<00000> [0x016e034e] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x016e03c7] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x016e03c7] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x016e0428] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x016e0aee] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x016d49a3] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x0042db18] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x2a960e1475] => /fortidev4-x86_64/lib/libc.so.6 (__libc_start_main+0x000000f5) 
pid=1 get sig=11 fault:0x1c
<00000> Backtrace:
<00000> [0x0042bee4] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x2a960f4d00] => /fortidev4-x86_64/lib/libc.so.6 
<00000> [0x0042bee4] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x2a960f4d00] => /fortidev4-x86_64/lib/libc.so.6 
<00000> [0x0042bee4] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x2a960f4d00] => /fortidev4-x86_64/lib/libc.so.6 
<00000> [0x016e034e] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x016e03c7] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x016e03c7] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x016e0428] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x016e0aee] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x016d49a3] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x0042db18] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x2a960e1475] => /fortidev4-x86_64/lib/libc.so.6 (__libc_start_main+0x000000f5) 
pid=1 get sig=11 fault:0x1c
<00000> Backtrace:
<00000> [0x0042bee4] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x2a960f4d00] => /fortidev4-x86_64/lib/libc.so.6 
<00000> [0x0042bee4] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x2a960f4d00] => /fortidev4-x86_64/lib/libc.so.6 
<00000> [0x0042bee4] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x2a960f4d00] => /fortidev4-x86_64/lib/libc.so.6 
<00000> [0x0042bee4] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x2a960f4d00] => /fortidev4-x86_64/lib/libc.so.6 
<00000> [0x016e034e] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x016e03c7] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x016e03c7] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x016e0428] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x016e0aee] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x016d49a3] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x0042db18] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x2a960e1475] => /fortidev4-x86_64/lib/libc.so.6 (__libc_start_main+0x000000f5) 
pid=1 get sig=11 fault:0x1c
<00000> Backtrace:
<00000> [0x0042bee4] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x2a960f4d00] => /fortidev4-x86_64/lib/libc.so.6 
<00000> [0x0042bee4] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x2a960f4d00] => /fortidev4-x86_64/lib/libc.so.6 
<00000> [0x0042bee4] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x2a960f4d00] => /fortidev4-x86_64/lib/libc.so.6 
<00000> [0x0042bee4] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x2a960f4d00] => /fortidev4-x86_64/lib/libc.so.6 
<00000> [0x0042bee4] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x2a960f4d00] => /fortidev4-x86_64/lib/libc.so.6 
<00000> [0x016e034e] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x016e03c7] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x016e03c7] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x016e0428] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x016e0aee] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x016d49a3] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x0042db18] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x2a960e1475] => /fortidev4-x86_64/lib/libc.so.6 (__libc_start_main+0x000000f5) 
pid=1 get sig=11 fault:0x1c
<00000> Backtrace:
<00000> [0x0042bee4] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x2a960f4d00] => /fortidev4-x86_64/lib/libc.so.6 
<00000> [0x0042bee4] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x2a960f4d00] => /fortidev4-x86_64/lib/libc.so.6 
<00000> [0x0042bee4] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x2a960f4d00] => /fortidev4-x86_64/lib/libc.so.6 
<00000> [0x0042bee4] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x2a960f4d00] => /fortidev4-x86_64/lib/libc.so.6 
<00000> [0x0042bee4] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x2a960f4d00] => /fortidev4-x86_64/lib/libc.so.6 
<00000> [0x0042bee4] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x2a960f4d00] => /fortidev4-x86_64/lib/libc.so.6 
<00000> [0x016e034e] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x016e03c7] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x016e03c7] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x016e0428] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x016e0aee] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x016d49a3] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x0042db18] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x2a960e1475] => /fortidev4-x86_64/lib/libc.so.6 (__libc_start_main+0x000000f5) 
pid=1 get sig=11 fault:0x1c
<00000> Backtrace:
<00000> [0x0042bee4] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x2a960f4d00] => /fortidev4-x86_64/lib/libc.so.6 
<00000> [0x0042bee4] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x2a960f4d00] => /fortidev4-x86_64/lib/libc.so.6 
<00000> [0x0042bee4] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x2a960f4d00] => /fortidev4-x86_64/lib/libc.so.6 
<00000> [0x0042bee4] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x2a960f4d00] => /fortidev4-x86_64/lib/libc.so.6 
<00000> [0x0042bee4] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x2a960f4d00] => /fortidev4-x86_64/lib/libc.so.6 
<00000> [0x0042bee4] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x2a960f4d00] => /fortidev4-x86_64/lib/libc.so.6 
<00000> [0x0042bee4] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x2a960f4d00] => /fortidev4-x86_64/lib/libc.so.6 
<00000> [0x016e034e] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x016e03c7] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x016e03c7] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x016e0428] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x016e0aee] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x016d49a3] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x0042db18] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x2a960e1475] => /fortidev4-x86_64/lib/libc.so.6 (__libc_start_main+0x000000f5) 
pid=1 get sig=11 fault:0x1c
<00000> Backtrace:
<00000> [0x0042bee4] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x2a960f4d00] => /fortidev4-x86_64/lib/libc.so.6 
<00000> [0x0042bee4] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x2a960f4d00] => /fortidev4-x86_64/lib/libc.so.6 
<00000> [0x0042bee4] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x2a960f4d00] => /fortidev4-x86_64/lib/libc.so.6 
<00000> [0x0042bee4] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x2a960f4d00] => /fortidev4-x86_64/lib/libc.so.6 
<00000> [0x0042bee4] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x2a960f4d00] => /fortidev4-x86_64/lib/libc.so.6 
<00000> [0x0042bee4] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x2a960f4d00] => /fortidev4-x86_64/lib/libc.so.6 
<00000> [0x0042bee4] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x2a960f4d00] => /fortidev4-x86_64/lib/libc.so.6 
<00000> [0x0042bee4] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x2a960f4d00] => /fortidev4-x86_64/lib/libc.so.6 
<00000> [0x016e034e] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x016e03c7] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x016e03c7] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x016e0428] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x016e0aee] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x016d49a3] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x0042db18] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x2a960e1475] => /fortidev4-x86_64/lib/libc.so.6 (__libc_start_main+0x000000f5) 
pid=1 get sig=11 fault:0x1c
<00000> Backtrace:
<00000> [0x0042bee4] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x2a960f4d00] => /fortidev4-x86_64/lib/libc.so.6 
<00000> [0x0042bee4] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x2a960f4d00] => /fortidev4-x86_64/lib/libc.so.6 
<00000> [0x0042bee4] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x2a960f4d00] => /fortidev4-x86_64/lib/libc.so.6 
<00000> [0x0042bee4] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x2a960f4d00] => /fortidev4-x86_64/lib/libc.so.6 
<00000> [0x0042bee4] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x2a960f4d00] => /fortidev4-x86_64/lib/libc.so.6 
<00000> [0x0042bee4] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x2a960f4d00] => /fortidev4-x86_64/lib/libc.so.6 
<00000> [0x0042bee4] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x2a960f4d00] => /fortidev4-x86_64/lib/libc.so.6 
<00000> [0x0042bee4] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x2a960f4d00] => /fortidev4-x86_64/lib/libc.so.6 
<00000> [0x0042bee4] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x2a960f4d00] => /fortidev4-x86_64/lib/libc.so.6 
<00000> [0x016e034e] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x016e03c7] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x016e03c7] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x016e0428] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x016e0aee] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x016d49a3] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x0042db18] => /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
<00000> [0x2a960e1475] => /fortidev4-x86_64/lib/libc.so.6 (__libc_start_main+0x000000f5) 
pid=1 get sig=11 fault:0x1c

 

Labels:
2659
0 Kudos
3 REPLIES 3
wkana
New Contributor

Created on ‎10-21-2019 12:01 PM

This could be helpful: 

https://kb.fortinet.com/kb/microsites/search.do?cmd=displayKC&docType=kc&externalId=FD34745

 

1665
0 Kudos
Dave_Hall
Honored Contributor
In response to wkana

Created on ‎10-22-2019 06:43 AM

Running the HQIP (Hardware Quick Inspection Package) test (as William indicated in the above link) would be a good idea, but based on the crash info provided it seems a format/reinstall firmware/restore last backup config may be in order first.  But if you can actual load/run the HQIP, it may find some hardware problem. 

NSE4/FMG-VM64/FortiAnalyzer-VM/6.0 (FWF30E/FW92D/FGT200D/FGT101E/FGT81E)/ FAP220B/221C

NSE4/FMG-VM64/FortiAnalyzer-VM/6.0 (FWF30E/FW92D/FGT200D/FGT101E/FGT81E)/ FAP220B/221C
1665
0 Kudos
ede_pfau
SuperUser
SuperUser
In response to Dave_Hall

Created on ‎10-22-2019 10:26 AM

One problem will be that you only have access to a HQIP image if you have a valid support contract.

Given the size of the model, and the ancient firmware, it might be time to think about a renewal. I would rather not invest in a new support contract.

Maybe you could gain some insight by running "di de en", "di de cr r" (crashlog read). But only maybe, a hw developer could tell you which processes died because of what, but we usually can not. Typical support case really.


Ede


"Kernel panic: Aiee, killing interrupt handler!"
Ede"Kernel panic: Aiee, killing interrupt handler!"
1665
0 Kudos
Labels
Top Kudoed Authors
View all
Broad. Integrated. Automated.

The Fortinet Security Fabric brings together the concepts of convergence and consolidation to provide comprehensive cybersecurity protection for all users, devices, and applications and across all network edges.

Social Media
Security Research
Company
News & Articles
Contact Us

Copyright 2024 Fortinet, Inc. All Rights Reserved.