Technical Tip: Brute-Force Attacks may cause the device to enter Conserve Mode with multiple http_authd daemons after upgrading to FortiOS v7.6.6
| Description | This article describes an issue where the http_authd child process is not terminated properly, resulting in multiple http_authd daemons being spawned during brute-force attacks after upgrading to FortiOS v7.6.6. Over time, the accumulation of these processes may lead to excessive memory consumption, causing the device to enter conserve mode. In FortiOS v7.6.4 and later, a new internal daemon, 'http_authd', is responsible for centralized GUI authentication and authorization. |
| Scope | FortiOS v7.6.6. |
| Solution | The symptoms observed are as follows:
The following debug output confirms that the daemon 'http_authd' is causing the high memory utilization.
FGT# diagnose debug crashlog read 16322: 2026-02-12 10:50:19 the killed daemon is /bin/http_authd: status=0x100
FGT# diagnose sys process pidof http_auth 298 740 795 860 869 883 893 906 918 929 940 950 964 975 984 993 1005 1016 1032 1042 1058 1077 1087 1097 1108 1120 1131 . . . [n]
FGT # diagnose sys top-mem 500| grep http_authd http_authd (6072): 8764kB http_authd (155): 2227kB http_authd (3893): 2215kB http_authd (3027): 2211kB http_authd (1925): 2203kB http_authd (2001): 2203kB http_authd (6074): 2189kB http_authd (1894): 2187kB http_authd (1905): 2187kB http_authd (1916): 2187kB http_authd (1991): 2187kB http_authd (2177): 2187kB http_authd (2215): 2187kB http_authd (2422): 2187kB http_authd (2436): 2187kB http_authd (2997): 2187kB http_authd (3162): 2187kB http_authd (3253): 2187kB http_authd (3288): 2187kB http_authd (3376): 2187kB http_authd (3411): 2187kB http_authd (3499): 2187kB http_authd (3533): 2187kB http_authd (3547): 2187kB http_authd (3686): 2187kB http_authd (3925): 2187kB http_authd (3938): 2187kB http_authd (3991): 2187kB http_authd (4013): 2187kB http_authd (4028): 2187kB http_authd (4053): 2187kB http_authd (4160): 2187kB http_authd (4195): 2187kB http_authd (4270): 2187kB http_authd (4431): 2187kB http_authd (4445): 2187kB http_authd (4482): 2187kB http_authd (4656): 2187kB http_authd (4674): 2187kB http_authd (4735): 2187kB http_authd (4830): 2187kB http_authd (5063): 2187kB http_authd (5074): 2187kB http_authd (5084): 2187kB http_authd (5182): 2187kB http_authd (5229): 2187kB http_authd (5296): 2187kB http_authd (5403): 2187kB http_authd (5467): 2187kB http_authd (5487): 2187kB http_authd (5651): 2187kB http_authd (5984): 2187kB http_authd (1941): 2183kB http_authd (1955): 2183kB http_authd (1966): 2183kB http_authd (1980): 2183kB http_authd (2018): 2183kB http_authd (2031): 2183kB http_authd (2142): 2183kB http_authd (2156): 2183kB http_authd (2166): 2183kB http_authd (2188): 2183kB http_authd (2205): 2183kB http_authd (2231): 2183kB http_authd (2241): 2183kB http_authd (2251): 2183kB http_authd (2386): 2183kB http_authd (2403): 2183kB http_authd (2412): 2183kB http_authd (2446): 2183kB http_authd (2582): 2183kB http_authd (2596): 2183kB http_authd (2608): 2183kB http_authd (2617): 2183kB http_authd (2762): 2183kB http_authd (2776): 2183kB http_authd (2821): 2183kB http_authd (2960): 2183kB http_authd (2969): 2183kB http_authd (2983): 2183kB http_authd (3010): 2183kB http_authd (3038): 2183kB http_authd (3055): 2183kB http_authd (3065): 2183kB http_authd (3076): 2183kB http_authd (3087): 2183kB http_authd (3102): 2183kB http_authd (3114): 2183kB http_authd (3127): 2183kB http_authd (3137): 2183kB http_authd (3151): 2183kB http_authd (3175): 2183kB http_authd (3186): 2183kB http_authd (3202): 2183kB http_authd (3212): 2183kB http_authd (3225): 2183kB http_authd (3242): 2183kB http_authd (3268): 2183kB http_authd (3307): 2183kB http_authd (3317): 2183kB http_authd (3332): 2183kB http_authd (3341): 2183kB http_authd (3355): 2183kB http_authd (3366): 2183kB http_authd (3391): 2183kB http_authd (3402): 2183kB http_authd (3427): 2183kB http_authd (3437): 2183kB http_authd (3448): 2183kB http_authd (3457): 2183kB http_authd (3471): 2183kB http_authd (3485): 2183kB http_authd (3509): 2183kB http_authd (3521): 2183kB http_authd (3559): 2183kB http_authd (3658): 2183kB http_authd (3674): 2183kB http_authd (3700): 2183kB http_authd (3714): 2183kB http_authd (3724): 2183kB
FGT# get system performance status CPU states: 0% user 4% system 0% nice 96% idle 0% iowait 0% irq 0% softirq CPU0 states: 0% user 34% system 0% nice 66% idle 0% iowait 0% irq 0% softirq CPU1 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq CPU2 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq CPU3 states: 0% user 3% system 0% nice 97% idle 0% iowait 0% irq 0% softirq CPU4 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq CPU5 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq CPU6 states: 1% user 0% system 0% nice 99% idle 0% iowait 0% irq 0% softirq CPU7 states: 1% user 0% system 0% nice 99% idle 0% iowait 0% irq 0% softirq Memory: 7769684k total, 7098508k used (91.4%), 392184k free (5.0%), 278992k freeable (3.6%)
FGT # diagnose hardware sysinfo memory MemTotal: 7769684 kB MemFree: 385572 kB Buffers: 4744 kB Cached: 910900 kB SwapCached: 0 kB Active: 5549512 kB <----- 5.5GB, ~71% of MemTotal. Inactive: 296656 kB Active(anon): 5489016 kB Inactive(anon): 279588 kB Active(file): 60496 kB Inactive(file): 17068 kB Unevictable: 298984 kB Mlocked: 42484 kB SwapTotal: 0 kB SwapFree: 0 kB Dirty: 0 kB Writeback: 0 kB AnonPages: 5229608 kB Mapped: 275136 kB Shmem: 568540 kB Slab: 433264 kB SReclaimable: 65496 kB SUnreclaim: 367768 kB KernelStack: 18144 kB PageTables: 222376 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 3884840 kB Committed_AS: 150932572 kB VmallocTotal: 260046784 kB VmallocUsed: 127948 kB VmallocChunk: 259773308 kB
date=202x-07-30 time=09:12:22 eventtime=1722345142416647315 tz="-0400" logid="0100032002" type="event" subtype="system" level="alert" vd="root" logdesc="Admin login failed" sn="0" user="testlogin" ui="https(173.25.181.118)" method="https" srcip=173.25.181.118 dstip=10.9.11.34 action="login" status="failed" reason="name_invalid" msg="Administrator testlogin login failed from https(172.25.181.118) because of invalid user name"
date=202x-07-30 time=09:10:19 eventtime=1722345019384149693 tz="-0400" logid="0100032002" type="event" subtype="system" level="alert" vd="root" logdesc="Admin login failed" sn="0" user="testlogin" ui="ssh(174.55.183.118)" method="ssh" srcip=174.55.183.118 dstip=10.9.11.34 action="login" status="failed" reason="name_invalid" msg="Administrator testlogin login failed from ssh(172.25.181.118) because of invalid user name"
This issue has been resolved in:
Workaround: Restrict administrative access to avoid brute-force attacks and cut off the root cause of the issue.
It is best practice to not to allow unrestricted administration of any edge network device via the internet, with best practice being out-of-band access. If this is not possible, it is highly recommended to apply a local-in policy to restrict the IP addresses that are able to access the administrative interface.
For a FortiGate that still has usable memory, execute the following command manually and use the automation stitches to execute a loop every 30 mins:
fnsysctl killall -9 http_authd
config system auto-script edit restart_http_authd set interval 1800 set repeat 0 set start auto set script 'fnsysctl killall -9 http_authd' next end For FortiGates that have severely depleted memory, a reboot may be the only workaround, as severely insufficient memory may be unable to execute any commands.
Note: Versions 7.6.7 and 8.0 will introduce a DoS mitigation mechanism designed to address brute-force attempts and Slowloris-style attacks.
Related articles: |
