Troubleshooting Tip: FCNACD high memory on Chassis FortiGate

[nathan_h]

Description

This article describes on how to troubleshoot high memory due to fcnacd daemon on a Chassis FortiGate. Fcnacd daemon handles communication between FortiClient and FortiGate. It is configured by FortiClient EMS connector under Fabric Connectors.

Scope

FortiGate-6000 and 7000 Series.

Solution

This issue may be triggered when EMS connection gets disconnected.

Configuration:

config endpoint-control fctems
    edit 1
        set name "EMS"
    set server "10.0.0.1"
end

get sys status
Version: FortiGate-7060E v7.2.11,build1740,250210 (GA.M)
....
Config-Sync: Primary
FPM Primary: slot-3

Enter the command below to check memory usage on FIMs and FPMs. The output shows that FIM01 has high memory usage.

config global

get system performance status

==========================================================================
Slot: 2 Module SN: FIM01EXXXXXXXXXX
CPU states: 3% user 0% system 0% nice 97% idle 0% iowait 0% irq 0% softirq
.....
Memory: 66017652k total, 58921716k used (89.3%), 6532176k free (9.9%), 563760k freeable (0.8%)
.....

==========================================================================
Slot: 3 Module SN: FPM20EXXXXXXXXXX
CPU states: 3% user 1% system 0% nice 95% idle 0% iowait 0% irq 1% softirq
.....
Memory: 132106048k total, 77958452k used (59.0%), 53585564k free (40.6%), 562032k freeable (0.4%)
.....

==========================================================================
Slot: 4 Module SN: FPM20EXXXXXXXXXX
CPU states: 6% user 2% system 0% nice 91% idle 0% iowait 0% irq 1% softirq
.....
Memory: 132106048k total, 76183504k used (57.7%), 55362240k free (41.9%), 560304k freeable (0.4%)
.....

==========================================================================
Current slot: 1 Module SN: FIM01EXXXXXXXXXX
CPU states: 2% user 1% system 0% nice 96% idle 0% iowait 0% irq 1% softirq
.....
Memory: 66017652k total, 60901552k used (92.3%), 4549860k free (6.9%), 566240k freeable (0.8%)
.....
Dataplane CPU states: 4%
Dataplane memory states: 93%
Dataplane average sessions: 950 sessions in 1 minute
Dataplane average session setup rate: 2 sessions per second in last 1 minute

Enter the command below to check the user process that is causing high memory usage.  The output below show fcnacd consume most of the memory.

config global

diagnose sys top-mem 99

fcnacd (296): 56190703kB <-----------
chlbd (300): 440325kB
locallogd (312): 169587kB
.....

diagnose debug crashlog read

14721: 2025-10-20 10:07:13 msg="Kernel enters extreme low memory mode"
14722: 2025-10-20 10:07:14 MemTotal: 66017652 kB
14723: 2025-10-20 10:07:14 MemFree: 6560156 kB
14724: 2025-10-20 10:07:14 Buffers: 17900 kB
14725: 2025-10-20 10:07:14 Cached: 2230188 kB
14726: 2025-10-20 10:07:14 SwapCached: 0 kB
14727: 2025-10-20 10:07:14 Active: 56832060 kB
14728: 2025-10-20 10:07:14 Inactive: 972772 kB
14729: 2025-10-20 10:07:14 Active(anon): 56282632 kB
14730: 2025-10-20 10:07:14 Inactive(anon): 815856 kB
.....
14739: 2025-10-20 10:07:14 AnonPages: 55576936 kB
14740: 2025-10-20 10:07:14 Mapped: 299448 kB
14741: 2025-10-20 10:07:14 Shmem: 1530388 kB
14742: 2025-10-20 10:07:14 Slab: 596376 kB
.....
14757: 2025-10-20 10:07:36 msg="Kernel exits extreme low memory mode"
14758: 2025-10-20 10:10:11 msg="Kernel enters extreme low memory mode"

Ener the command below to run fcnacd debug. 

diagnose debug reset
diagnose debug disable

diagnose debug duration 1
diagnose debug app fcnacd -1
diagnose endpoint filter show-large-data yes
diagnose debug enable

Wait for 1 minute and then disable debugging with the following commands:

diagnose debug reset
diagnose debug disable

ec_rec_set_sslvpn_conn:1042] called (FTCL UID C25AXXXXXXXXXXXXXXXXXXXXXXXXXXXX).

[fcems_recv_req_epoll:1370] called.

[fcems_recv_req:1284] called.

[ec_rec_set_sslvpn_conn:1042] called (FTCL UID C25AXXXXXXXXXXXXXXXXXXXXXXXXXXXX).

[fcems_recv_req_epoll:1370] called.

[ec_daemon_submit_sock_call:49] sent 244,244
[fcems_recv_req:1284] called.

[ec_rec_set_sslvpn_conn:1042] called (FTCL UID C25AXXXXXXXXXXXXXXXXXXXXXXXXXXXX).

[ec_daemon_submit_sock_call:49] sent 244,244
[fcems_recv_req:1284] called.

[ec_daemon_submit_sock_call:49] sent 244,244
[ec_rec_set_sslvpn_conn:1042] called (FTCL UID C25AXXXXXXXXXXXXXXXXXXXXXXXXXXXX).

[ec_rec_set_sslvpn_conn:1042] called (FTCL UID C25AXXXXXXXXXXXXXXXXXXXXXXXXXXXX).

[fcems_recv_req_epoll:1370] called.

[fcems_recv_req:1284] called.

[ec_rec_set_sslvpn_conn:1042] called (FTCL UID C25AXXXXXXXXXXXXXXXXXXXXXXXXXXXX).

[ec_daemon_submit_sock_call:49] sent 244,244
[fcems_recv_req_epoll:1370] called.

[fcems_recv_req:1284] called.

[ec_rec_set_sslvpn_conn:1042] called (FTCL UID C25AXXXXXXXXXXXXXXXXXXXXXXXXXXXX).

[ec_daemon_submit_sock_call:49] sent 244,244
[ec_rec_set_sslvpn_conn:1042] called (FTCL UID C25AXXXXXXXXXXXXXXXXXXXXXXXXXXXX).

[fcems_recv_req_epoll:1370] called.

[fcems_recv_req:1284] called.

[ec_rec_set_sslvpn_conn:1042] called (FTCL UID C25AXXXXXXXXXXXXXXXXXXXXXXXXXXXX).

[fcems_recv_req_epoll:1370] called.

[ec_daemon_submit_sock_call:49] sent 244,244
[fcems_recv_req:1284] called.

[ec_rec_set_sslvpn_conn:1042] called (FTCL UID C25AXXXXXXXXXXXXXXXXXXXXXXXXXXXX).

[fcems_recv_req_epoll:1370] called.

[fcems_recv_req:1284] called.

.....

If these symptoms are encountered, upgrade to FortiOS v7.4.x or v7.6.x - these versions are not affected.

This issue is expected to be fixed in FortiOS v7.2.12.

Related article:

Technical Tip: Basic Troubleshooting for high memory or high CPU usage