Troubleshooting Tip: FortiGate fnbamd Crash Analysis: Segmentation Fault at fnbamd_http_get_next_host

[sjoshi]

Description

This article describes the investigation and root cause analysis of a segmentation fault observed in the fnbamd process on FortiGate, specifically crashing at the fnbamd_http_get_next_host function

Scope

FortiGate v7.2, v7.4.

Solution

A lot of crashes related to fnbamd can be observed while executing the command 'diagnose debug crashlog read'. The crash was triggered by a segmentation fault (signal 11) and traced to the function fnbamd_http_get_next_host.

16354: 2025-06-30 12:36:45 fnbamd previously crashed 9 times. The last crash was at 2025-06-30 12:36:39.
16355: 2025-06-30 12:36:45 <32714> firmware FortiGate-400F v7.4.8,build2795b2795,250523 (GA.M) (Release)
16356: 2025-06-30 12:36:45 <32714> application fnbamd
16357: 2025-06-30 12:36:45 <32714> *** signal 11 (Segmentation fault) received ***
16358: 2025-06-30 12:36:45 <32714> Register dump:
16359: 2025-06-30 12:36:45 <32714> RAX: 00000000049c7af5 RBX: 0000559d59f1ae50
16360: 2025-06-30 12:36:45 <32714> RCX: 00000000049c7af4 RDX: 0000000000000000
16361: 2025-06-30 12:36:45 <32714> R08: 0000559d59efa280 R09: 0000559d59f1af20
16362: 2025-06-30 12:36:45 <32714> R10: 0000000000000007 R11: 0000000000000246
16363: 2025-06-30 12:36:45 <32714> R12: 0000000000000000 R13: 0000559d51e82034
16364: 2025-06-30 12:36:45 <32714> R14: 0000559d59f1aee8 R15: 00007ffd3cd764e0
16365: 2025-06-30 12:36:45 <32714> RSI: 0000000000000005 RDI: 0000000000000671
16366: 2025-06-30 12:36:45 <32714> RBP: 00007ffd3cd75fa0 RSP: 00007ffd3cd75f30
16367: 2025-06-30 12:36:45 <32714> RIP: 00007fded5a81701 EFLAGS: 0000000000010246
16368: 2025-06-30 12:36:45 <32714> CS: 0033 FS: 0000 GS: 0000
16369: 2025-06-30 12:36:45 <32714> Trap: 000000000000000e Error: 0000000000000004
16370: 2025-06-30 12:36:45 <32714> OldMask: 0000000000000000
16371: 2025-06-30 12:36:45 <32714> CR2: 0000000000000671
16372: 2025-06-30 12:36:45 <32714> stack: 0x7ffd3cd75f30 - 0x7ffd3cd77030
16373: 2025-06-30 12:36:45 <32714> Backtrace:
16374: 2025-06-30 12:36:45 <32714> [0x7fded5a81701] => /lib/libcrypto.so.3 {0x7fded5732000}
16375: 2025-06-30 12:36:45 <32714> [0x559d4f6c92d4] => /bin/fnbamd {0x559d4e969000}
16376: 2025-06-30 12:36:45 <32714> [0x559d4f6cae4d] => /bin/fnbamd {0x559d4e969000}
16377: 2025-06-30 12:36:45 <32714> [0x559d4f6b8c27] => /bin/fnbamd {0x559d4e969000}
16378: 2025-06-30 12:36:45 <32714> [0x559d4f6b8e44] => /bin/fnbamd {0x559d4e969000}
16379: 2025-06-30 12:36:45 <32714> [0x559d51865e09] => /bin/fnbamd {0x559d4e969000}
16380: 2025-06-30 12:36:45 <32714> [0x559d4f6d99ca] => /bin/fnbamd {0x559d4e969000}
16381: 2025-06-30 12:36:45 <32714> [0x559d4edb97f3] => /bin/fnbamd {0x559d4e969000}
16382: 2025-06-30 12:36:45 <32714> [0x7fded5d45e1b] => /lib/libc.so.6 {0x7fded5d22000}
16383: 2025-06-30 12:36:45 <32714> [0x559d4edb4f8a] => /bin/fnbamd {0x559d4e969000}
16384: 2025-06-30 12:36:45 <32714> fortidev 6.0.2.0008

Decoded Crash Log:

16354: 2025-06-30 12:36:45 fnbamd previously crashed 9 times. The last crash was at 2025-06-30 12:36:39.
16355: 2025-06-30 12:36:45 <32714> firmware FortiGate-400F v7.4.8,build2795b2795,250523 (GA.M) (Release)
16356: 2025-06-30 12:36:45 <32714> application fnbamd
16357: 2025-06-30 12:36:45 <32714> *** signal 11 (Segmentation fault) received ***
16358: 2025-06-30 12:36:45 <32714> Register dump:
16359: 2025-06-30 12:36:45 <32714> RAX: 00000000049c7af5 RBX: 0000559d59f1ae50
16360: 2025-06-30 12:36:45 <32714> RCX: 00000000049c7af4 RDX: 0000000000000000
16361: 2025-06-30 12:36:45 <32714> R08: 0000559d59efa280 R09: 0000559d59f1af20
16362: 2025-06-30 12:36:45 <32714> R10: 0000000000000007 R11: 0000000000000246
16363: 2025-06-30 12:36:45 <32714> R12: 0000000000000000 R13: 0000559d51e82034
16364: 2025-06-30 12:36:45 <32714> R14: 0000559d59f1aee8 R15: 00007ffd3cd764e0
16365: 2025-06-30 12:36:45 <32714> RSI: 0000000000000005 RDI: 0000000000000671
16366: 2025-06-30 12:36:45 <32714> RBP: 00007ffd3cd75fa0 RSP: 00007ffd3cd75f30
16367: 2025-06-30 12:36:45 <32714> RIP: 00007fded5a81701 EFLAGS: 0000000000010246
16368: 2025-06-30 12:36:45 <32714> CS: 0033 FS: 0000 GS: 0000
16369: 2025-06-30 12:36:45 <32714> Trap: 000000000000000e Error: 0000000000000004
16370: 2025-06-30 12:36:45 <32714> OldMask: 0000000000000000
16371: 2025-06-30 12:36:45 <32714> CR2: 0000000000000671
16372: 2025-06-30 12:36:45 <32714> stack: 0x7ffd3cd75f30 - 0x7ffd3cd77030
16373: 2025-06-30 12:36:45 <32714> Backtrace:
16374: 2025-06-30 12:36:45 <32714> [0x7fded5a81701] => /lib/libcrypto.so.3 {0x7fded5732000} => ?? ??:0
16375: 2025-06-30 12:36:45 <32714> [0x559d4f6c92d4] => /bin/fnbamd {0x559d4e969000} => __fnbamd_http_get_next_host at ././daemon/fnbamd/fnbamd_http.c:149
16376: 2025-06-30 12:36:45 <32714> [0x559d4f6cae4d] => /bin/fnbamd {0x559d4e969000} => __http_error at ././daemon/fnbamd/fnbamd_http.c:740
16377: 2025-06-30 12:36:45 <32714> [0x559d4f6b8c27] => /bin/fnbamd {0x559d4e969000} => fnbamd_dns_parse_resp at ././daemon/fnbamd/fnbamd_dns.c:316
16378: 2025-06-30 12:36:45 <32714> [0x559d4f6b8e44] => /bin/fnbamd {0x559d4e969000} => fnbamd_dns_poll_cb at ././daemon/fnbamd/fnbamd_dns.c:337
16379: 2025-06-30 12:36:45 <32714> [0x559d51865e09] => /bin/fnbamd {0x559d4e969000} => daemon_main at ././migbase/sysapi/daemon/daemon.c:295 (discriminator 2)
16380: 2025-06-30 12:36:45 <32714> [0x559d4f6d99ca] => /bin/fnbamd {0x559d4e969000} => fnbamd_main at ././daemon/fnbamd/fnbamd_main.c:130
16381: 2025-06-30 12:36:45 <32714> [0x559d4edb97f3] => /bin/fnbamd {0x559d4e969000} => fortiexec_call_main at ././sysinit/fortiexec.c:799
16382: 2025-06-30 12:36:45 <32714> [0x7fded5d45e1b] => /lib/libc.so.6 {0x7fded5d22000} => ?? ??:0
16383: 2025-06-30 12:36:45 <32714> [0x559d4edb4f8a] => /bin/fnbamd {0x559d4e969000} => _start at /build/glibc/glibc-2.30/csu/../sysdeps/x86_64/start.S:122
16384: 2025-06-30 12:36:45 <32714> fortidev 6.0.2.0008

A similar crash was previously addressed with a fix in FortiOS v7.4.1. However, the issue appears to have reoccurred and is now planned to be resolved in the upcoming FortiOS v7.4.9, v7.6.4 release.