5.6.11 FortiGate 200D signal 11

[dkcho]

Hello to you all,

I upgraded last week (5.6 -> 5.6.8 -> 5.6.11) The upgrade was completed normally. Then we rebooted MASTER and SLAVE once each (FailoverTest). The test was also completed normally. But I found a strange Crash Log. Signal 11 Log related to Hatalk occurred at the time of reboot of both equipment Crash log has occurred twice. I suspect BUG. If it's not BUG, I wonder why the symptoms occurred. Thanks

crash log

374: 2019-11-10 01:37:16 <00083> firmware FortiGate-200D v5.6.11,build1700b1700,190814 (GA) (Release) 375: 2019-11-10 01:37:16 <00083> application hatalk 376: 2019-11-10 01:37:16 <00083> *** signal 11 (Segmentation fault) received *** 377: 2019-11-10 01:37:16 <00083> Register dump: 378: 2019-11-10 01:37:16 <00083> RAX: 0000000000000000   RBX: 0000002a99f69000 379: 2019-11-10 01:37:16 <00083> RCX: 0000000002120fb0   RDX: 0000000002120fb0 380: 2019-11-10 01:37:16 <00083> R8:  00000000000241a0   R9:  000000000002419a 381: 2019-11-10 01:37:16 <00083> R10: 0000000000000001   R11: 0000000000000001 382: 2019-11-10 01:37:16 <00083> R12: 0000000012b8ee80   R13: 0000000012baf890 383: 2019-11-10 01:37:16 <00083> R14: 0000000012b8e460   R15: 0000000000000001 384: 2019-11-10 01:37:16 <00083> RSI: 0000000000000001   RDI: 0000000000000000 385: 2019-11-10 01:37:16 <00083> RBP: 0000007fbffff080   RSP: 0000007fbfffeff8 386: 2019-11-10 01:37:16 <00083> RIP: 00000000008d05e0   EFLAGS: 0000000000010246 387: 2019-11-10 01:37:16 <00083> CS:  0033   FS: 0000   GS: 0000 388: 2019-11-10 01:37:16 <00083> Trap: 000000000000000e   Error: 0000000000000004 389: 2019-11-10 01:37:16 <00083> OldMask: 0000000000000000 390: 2019-11-10 01:37:16 <00083> CR2: 0000000000000003 391: 2019-11-10 01:37:16 <00083> stack: 0x7fbfffeff8 - 0x7fbffffe80 392: 2019-11-10 01:37:16 <00083> Backtrace: 393: 2019-11-10 01:37:16 <00083> [0x008d05e0] => /bin/hatalk   394: 2019-11-10 01:37:16 <00083> [0x008cd257] => /bin/hatalk   395: 2019-11-10 01:37:16 <00083> [0x0131683c] => /bin/hatalk   396: 2019-11-10 01:37:16 <00083> [0x01315b82] => /bin/hatalk   397: 2019-11-10 01:37:16 <00083> [0x01315ba9] => /bin/hatalk   398: 2019-11-10 01:37:16 <00083> [0x01315c19] => /bin/hatalk   399: 2019-11-10 01:37:16 <00083> [0x008b44c4] => /bin/hatalk   400: 2019-11-10 01:37:16 <00083> [0x0042ec10] => /bin/hatalk   401: 2019-11-10 01:37:16 <00083> [0x0043550e] => /bin/hatalk   402: 2019-11-10 01:37:16 <00083> [0x00432318] => /bin/hatalk   403: 2019-11-10 01:37:16 <00083> [0x004328c1] => /bin/hatalk   404: 2019-11-10 01:37:16 <00083> [0x00434703] => /bin/hatalk   405: 2019-11-10 01:37:16 <00083> [0x004350f9] => /bin/hatalk   406: 2019-11-10 01:37:16 <00083> [0x2a96140475] => /fortidev4-x86_64/lib/libc.so.6 (__libc_start_main+0x 407: 2019-11-10 01:37:16 000000f5) liboffset 00021475 408: 2019-11-10 01:37:16 <00083> [0x0042befd] => /bin/hatalk   409: 2019-11-10 01:37:16 the killed daemon is /bin/hatalk: status=0xb 410: 2019-11-10 01:37:16 hatalk crashed 1 times. The last crash was at 2019-11-10 01:37:16 411: 2019-11-10 01:37:16 <00649> firmware FortiGate-200D v5.6.11,build1700b1700,190814 (GA) (Release) 412: 2019-11-10 01:37:16 <00649> application hatalk 413: 2019-11-10 01:37:16 <00649> *** signal 11 (Segmentation fault) received *** 414: 2019-11-10 01:37:16 <00649> Register dump: 415: 2019-11-10 01:37:16 <00649> RAX: 0000000000000000   RBX: 0000002a99f69000 416: 2019-11-10 01:37:16 <00649> RCX: 0000000002120fb0   RDX: 0000000002120fb0 417: 2019-11-10 01:37:16 <00649> R8:  0000000000000000   R9:  0000002a95582cc0 418: 2019-11-10 01:37:16 <00649> R10: 0000000000000000   R11: 0000000000000202 419: 2019-11-10 01:37:16 <00649> R12: 0000000012b8ee80   R13: 00000000127ede88 420: 2019-11-10 01:37:16 <00649> R14: 0000000012b8e460   R15: 0000000000000001 421: 2019-11-10 01:37:16 <00649> RSI: 0000000000000001   RDI: 0000000000000000 422: 2019-11-10 01:37:16 <00649> RBP: 0000007fbffff050   RSP: 0000007fbfffefc8 423: 2019-11-10 01:37:16 <00649> RIP: 00000000008d05e0   EFLAGS: 0000000000010246 424: 2019-11-10 01:37:16 <00649> CS:  0033   FS: 0000   GS: 0000 425: 2019-11-10 01:37:16 <00649> Trap: 000000000000000e   Error: 0000000000000004 426: 2019-11-10 01:37:16 <00649> OldMask: 0000000000000000 427: 2019-11-10 01:37:16 <00649> CR2: 0000000000000003 428: 2019-11-10 01:37:16 <00649> stack: 0x7fbfffefc8 - 0x7fbffffe80 429: 2019-11-10 01:37:16 <00649> Backtrace: 430: 2019-11-10 01:37:16 <00649> [0x008d05e0] => /bin/hatalk   431: 2019-11-10 01:37:16 <00649> [0x008cd257] => /bin/hatalk   432: 2019-11-10 01:37:16 <00649> [0x0131683c] => /bin/hatalk   433: 2019-11-10 01:37:16 <00649> [0x01315b82] => /bin/hatalk   434: 2019-11-10 01:37:16 <00649> [0x01315ba9] => /bin/hatalk   435: 2019-11-10 01:37:16 <00649> [0x01315ba9] => /bin/hatalk   436: 2019-11-10 01:37:16 <00649> [0x01315c19] => /bin/hatalk   437: 2019-11-10 01:37:16 <00649> [0x008b44c4] => /bin/hatalk   438: 2019-11-10 01:37:16 <00649> [0x0042ec10] => /bin/hatalk   439: 2019-11-10 01:37:16 <00649> [0x0043550e] => /bin/hatalk   440: 2019-11-10 01:37:16 <00649> [0x00432318] => /bin/hatalk   441: 2019-11-10 01:37:16 <00649> [0x004328c1] => /bin/hatalk   442: 2019-11-10 01:37:16 <00649> [0x00434703] => /bin/hatalk   443: 2019-11-10 01:37:16 <00649> [0x004350f9] => /bin/hatalk   444: 2019-11-10 01:37:16 <00649> [0x2a96140475] => /fortidev4-x86_64/lib/libc.so.6 (__libc_start_main+0x 445: 2019-11-10 01:37:16 000000f5) liboffset 00021475 446: 2019-11-10 01:37:16 <00649> [0x0042befd] => /bin/hatalk   447: 2019-11-10 01:37:19 the killed daemon is /bin/hatalk: status=0xb 448: 2019-11-10 01:37:19 hatalk crashed 2 times. The last crash was at 2019-11-10 01:37:19 449: 2019-11-10 01:37:19 <00658> firmware FortiGate-200D v5.6.11,build1700b1700,190814 (GA) (Release) 450: 2019-11-10 01:37:19 <00658> application hatalk 451: 2019-11-10 01:37:19 <00658> *** signal 11 (Segmentation fault) received *** 452: 2019-11-10 01:37:19 <00658> Register dump: 453: 2019-11-10 01:37:19 <00658> RAX: 0000000000000000   RBX: 0000002a99f69000 454: 2019-11-10 01:37:19 <00658> RCX: 0000000002120fb0   RDX: 0000000002120fb0 455: 2019-11-10 01:37:19 <00658> R8:  0000000000000000   R9:  0000002a95582cc0 456: 2019-11-10 01:37:19 <00658> R10: 0000000000000000   R11: 0000000000000202 457: 2019-11-10 01:37:19 <00658> R12: 0000000012b8ee80   R13: 00000000127ede88 458: 2019-11-10 01:37:19 <00658> R14: 0000000012b8e460   R15: 0000000000000001 459: 2019-11-10 01:37:19 <00658> RSI: 0000000000000001   RDI: 0000000000000000 460: 2019-11-10 01:37:19 <00658> RBP: 0000007fbffff050   RSP: 0000007fbfffefc8 461: 2019-11-10 01:37:19 <00658> RIP: 00000000008d05e0   EFLAGS: 0000000000010246 462: 2019-11-10 01:37:19 <00658> CS:  0033   FS: 0000   GS: 0000 463: 2019-11-10 01:37:19 <00658> Trap: 000000000000000e   Error: 0000000000000004 464: 2019-11-10 01:37:19 <00658> OldMask: 0000000000000000 465: 2019-11-10 01:37:19 <00658> CR2: 0000000000000003 466: 2019-11-10 01:37:19 <00658> stack: 0x7fbfffefc8 - 0x7fbffffe80 467: 2019-11-10 01:37:19 <00658> Backtrace: 468: 2019-11-10 01:37:19 <00658> [0x008d05e0] => /bin/hatalk   469: 2019-11-10 01:37:19 <00658> [0x008cd257] => /bin/hatalk   470: 2019-11-10 01:37:19 <00658> [0x0131683c] => /bin/hatalk   471: 2019-11-10 01:37:19 <00658> [0x01315b82] => /bin/hatalk   472: 2019-11-10 01:37:19 <00658> [0x01315ba9] => /bin/hatalk   473: 2019-11-10 01:37:19 <00658> [0x01315ba9] => /bin/hatalk   474: 2019-11-10 01:37:19 <00658> [0x01315c19] => /bin/hatalk   475: 2019-11-10 01:37:19 <00658> [0x008b44c4] => /bin/hatalk   476: 2019-11-10 01:37:19 <00658> [0x0042ec10] => /bin/hatalk   477: 2019-11-10 01:37:19 <00658> [0x0043550e] => /bin/hatalk   478: 2019-11-10 01:37:19 <00658> [0x00432318] => /bin/hatalk   479: 2019-11-10 01:37:19 <00658> [0x004328c1] => /bin/hatalk   480: 2019-11-10 01:37:19 <00658> [0x00434703] => /bin/hatalk   481: 2019-11-10 01:37:19 <00658> [0x004350f9] => /bin/hatalk   482: 2019-11-10 01:37:19 <00658> [0x2a96140475] => /fortidev4-x86_64/lib/libc.so.6 (__libc_start_main+0x 483: 2019-11-10 01:37:19 000000f5) liboffset 00021475 484: 2019-11-10 01:37:19 <00658> [0x0042befd] => /bin/hatalk   485: 2019-11-10 01:37:19 the killed daemon is /bin/hatalk: status=0xb

alret log

2019-11-10 01:37:21 System shutdown (Executed 'reboot' by "admin")

[Toshi_Esumi · Accepted solution]

Please open a ticket with TAC. Even if you power down the master, which could happen in real situations. the previous slave shouldn't clash with a segmentation fault. I hope they look into it carefully to see if it's only with 5.6.11 or a design problem with HA even with 6.0 or 6.2. I hope the former.

[live89]

Hi

From which 5.6.X you upgraded to 5.6.8 ? you followed the path ?

It might seems normal upgrade but still you should follow the path...

5.6.1 > 5.6.3 > 5.6.6 > 5.6.8 > 5.6.11

Anyway, I would recommend next time you do a reboot, then from the second member observe the following output of this debug command:

diagnose debug reset

diagnose debug application hatalk 255

diagnose debug application hasync 255 diagnose debug enable

[dkcho]

Thanks

your answer helped me a lot!