Fortigate fnsysctl command options with examples

[Yurisk]

fnsysctl is frequently helpful in troubleshooting Fortigates, and while its options are mentioned in the Forums here and there, no single article lists them, and not all options are mentioned, so I wrote a post to summarize the info.

Originally posted on https://yurisk.info/2024/10/23/fortigate-fnsysctl-command-options-with-examples/ 

Important facts about fnsysctl command:

• You have to log in with a user having super_admin profile.

• For VM Fortigate, it has to have a regular license - not free evaluation one. On free evaluation VM FGT you will get an error Unknown action 0.

• It is CLI-only command, with no GUI equivalent.

• The command runs locally on the Fortigate you are logged in, so to run the same command on a passive member of HA cluster, you will need to log in into the passive member first.

• The Tab completion does NOT work with this command (therefore this post).

• We CAN use these commands in automation stitches as set action-type cli-script.

fnsysctl ifconfig

Shows detailed info on the physical interfaces, including drops/errors/MTU. Accepts optionally name of the interface e.g. fnsysctl ifconfig port1.

FGT-Perimeter # fnsysctl ifconfig
port1   Link encap:Ethernet  HWaddr 0A:7C:2A:D2:17:6F
        inet addr:10.100.100.227  Bcast:10.100.100.255  Mask:255.255.255.0
        link-local6: fe80::87c:2aff:fed2:176f prefixlen 64
        UP BROADCAST RUNNING MULTICAST  MTU:9001  Metric:1
        RX packets:3537 errors:0 dropped:0 overruns:0 frame:0
        TX packets:5436 errors:0 dropped:0 overruns:0 carrier:0
        collisions:0 txqueuelen:1000
        RX bytes:1340257 (1.3 MB)  TX bytes:4360502 (4.2 MB)

port2   Link encap:Ethernet  HWaddr 0A:C2:8D:76:4D:8D
        inet addr:10.100.104.13  Bcast:10.100.104.255  Mask:255.255.255.0
        link-local6: fe80::8c2:8dff:fe76:4d8d prefixlen 64
        UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
        RX packets:23 errors:0 dropped:0 overruns:0 frame:0
        TX packets:129 errors:0 dropped:0 overruns:0 carrier:0
        collisions:0 txqueuelen:1000
        RX bytes:644 (644  Bytes)  TX bytes:5888 (5.8 KB)

fnsysctl ls

Lists files/folders in the filesystem. Useful for post-incident investigation of Fortigate compromises, looking for a given CVE indicators of compromise (IOCs).

It accepts only 3 flags:

• a - Show all files, including those starting with the dot in their name.

• l - Show long output, i.e. not only names but timestamps, sizes.

• A - almost all, do not show names starting with the dot (default so no need to specify).

Examples:

FGT-Perimeter #  fnsysctl ls -al  /tmp
drwxr-xr-x    2 0   0   Wed Oct 23 01:53:42 2024    40 $$auto-script$$
drwxrwxrwt   60 0   0   Wed Oct 23 02:03:46 2024  4780 .
drwxr-xr-x   18 0   0   Wed Oct 23 01:53:40 2024     0 ..
srwxr-xr-x    1 0   0   Wed Oct 23 01:53:42 2024     0 .auto_script_server
-rw-r--r--    1 0   0   Wed Oct 23 01:53:42 2024     0 .aws_addrs
srwxr-xr-x    1 0   0   Wed Oct 23 01:53:42 2024     0 .cloudapi_fconv.sock
srwxr-xr-x    1 0   0   Wed Oct 23 01:53:42 2024     0 .dhcpd.msg
srwxr-xr-x    1 0   0   Wed Oct 23 01:53:42 2024     0 .dns_local_server

FGT-Perimeter #  fnsysctl ls -a  /tmp
$$auto-script$$               .
.dns_local_server             .dns_local_server_for_proxy
.dnsproxy_unix_server  0      .fgfm_stream_clt_sock
.ipsengine001_0_0.url.socket  .ipsengine002_0_0.url.socket
.urlfilter0.sock              .wad512_0_0.url.socket
admin_server.crt              KEY-FILE
backtrace_log                 bwl_gui_to_url0_unix_sock

fnsysctl cat

Show contents of a file, not all files in the filesystem are accessible. Some examples.

• Show Linux kernel version of the Fortigate (here FortiOS 7.4.3):

FGT-Perimeter #  fnsysctl cat /proc/version
Linux version 4.19.13 (root@build) (gcc version 10.3.0 (GCC)) #1 SMP Thu Feb 1 17:10:43 UTC 2024

When trying to access a prohibited file:

FGT-Perimeter # fnsysctl cat /tmp/cw_ac_key_bak.pem
cat: /tmp/cw_ac_key_bak.pem: Not allowed

• Show open TCP connections to/from Fortigate itself:

FGT-Perimeter #  fnsysctl cat /proc/net/tcp
  sl  local_address rem_address   st tx_queue rx_queue
  tr tm->when retrnsmt   uid  timeout inode
   0: 00000000:28A0 00000000:0000 0A 00000000:00000000
   00:00000000 00000000     0        0 13871 1 ffff8880443a9200
   100 0 0 10 0 0:0/0:0/0:0 0
   1: 00000000:1E82 00000000:0000 0A 00000000:00000000
   00:00000000 00000000     0        0 17550 1 ffff88804a0ece00
   100 0 0 10 0 0:0/0:0/0:0 0
   2: 00000000:2904 00000000:0000 0A 00000000:00000000 00:00000000
   00000000     0        0 13877 1 ffff888042db2200 100 0 0 10 0
   0:0/0:0/0:0 0

The output is in hex, so it is much easier to use diagnose sys tcpsock | grep 0.0.0.0.

• Show CPU info:

FGT-Perimeter # fnsysctl cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 85
model name      : Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
stepping        : 7
microcode       : 0x5003707
cpu MHz         : 2499.998
cache size      : 36608 KB
physical id     : 0
siblings        : 2
core id         : 0
cpu cores       : 1
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic

• Get memory information:

FGT-Perimeter # fnsysctl cat /proc/meminfo
MemTotal:        1984244 kB
MemFree:          595988 kB
MemAvailable:     757016 kB
Buffers:           10140 kB
Cached:           597428 kB
SwapCached:            0 kB
Active:           591168 kB
Inactive:         141344 kB
Active(anon):     518884 kB
Inactive(anon):    47496 kB
Active(file):      72284 kB
...cut...

 

• Show nturbo acceleration statistics fnsysctl cat /proc/nturbo/<n>/drv:

fnsysctl cat /proc/nturbo/<0>/drv

Turbo interface ID: 0

============================================================================
Driver RX/TX:      760818543/759413272
TX hang:        No

Free/Used buffers:   109675/2965
Alloc fail:     0, Bad qid:     0
queue ready: 0x0000007f, 0x00000000

RXQ_0(0,20806): IN 64201109 OUT 64201142 DROP 0 NRDY 0 Fullness 0, Peak 282
TXQ_0(0,20806): IN 64083848 OUT 64083848 DROP 0 SHAPER_DROP 0 USR_DROP 117056 BUFERR 0

RXQ_1(1,20808): IN 62241175 OUT 62241191 DROP 0 NRDY 0 Fullness 0, Peak 444
TXQ_1(1,20808): IN 62092654 OUT 62092654 DROP 0 SHAPER_DROP 0 USR_DROP 148288 BUFERR 0

RXQ_2(2,20807): IN 63028145 OUT 63028179 DROP 0 NRDY 0 Fullness 0, Peak 247
TXQ_2(2,20807): IN 62856041 OUT 62856041 DROP 0 SHAPER_DROP 0 USR_DROP 171904 BUFERR 0

RXQ_3(3,20809): IN 61829939 OUT 61830044 DROP 0 NRDY 0 Fullness 0, Peak 254
TXQ_3(3,20809): IN 61684861 OUT 61684861 DROP 0 SHAPER_DROP 0 USR_DROP 144928 BUFERR 0

RXQ_4(4,20810): IN 64154116 OUT 64154184 DROP 0 NRDY 0 Fullness 0, Peak 1408
TXQ_4(4,20810): IN 64009332 OUT 64009332 DROP 0 SHAPER_DROP 0 USR_DROP 144608 BUFERR 0

RXQ_5(5,20804): IN 63186535 OUT 63186600 DROP 0 NRDY 0 Fullness 0, Peak 221
TXQ_5(5,20804): IN 63097904 OUT 63097904 DROP 0 SHAPER_DROP 0 USR_DROP 88448 BUFERR 0

RXQ_6(6,20805): IN 63168351 OUT 63168429 DROP 0 NRDY 0 Fullness 0, Peak 432
TXQ_6(6,20805): IN 62918126 OUT 62918126 DROP 0 SHAPER_DROP 0 USR_DROP 250048 BUFERR 0

To decipher the output, see https://community.fortinet.com/t5/FortiGate/Technical-Tip-Useful-diagnostics-commands-for-troublesho...

 

fnsysctl date

Show date in the Linux format, ignores any options.

FGT-Perimeter #  fnsysctl date
Wed Oct 23 02:11:03 PDT 2024

fnsysctl df

Show filesystem usage, useful when you have harddisk(s) attached to the Fortigate.

FGT-Perimeter # fnsysctl df -h
Filesystem                 Size    Used  Available Use% Mounted on
none                       1.3G   81.6M       1.2G   6% /tmp
none                       1.3G    4.7M       1.3G   0% /dev/shm
none                       1.3G   70.0M       1.2G   5% /dev/cmdb
/dev/nvme0n1p1           231.9M  129.2M      89.9M  59% /data
/dev/nvme0n1p2             1.6G  141.7M       1.4G   9% /data2
/dev/nvme1n1p1            29.4G   54.8M      27.8G   0% /var/log
/dev/nvme0n1p1           231.9M  129.2M      89.9M  59% /new_root/zebos/fortidev/etc/localtime
none                       1.3G   70.0M       1.2G   5% /new_root/eap_proxy/dev/cmdb
/dev/nvme0n1p1           231.9M  129.2M      89.9M  59% /new_root/eap_proxy/etc/cert/ca
/dev/nvme0n1p1           231.9M  129.2M      89.9M  59% /new_root/eap_proxy/fortidev/etc/localtime
/dev/nvme0n1p1           231.9M  129.2M      89.9M  59% /new_root/eap_proxy_worker/etc/cert/ca
/dev/nvme0n1p1           231.9M  129.2M      89.9M  59% /new_root/eap_proxy_worker/fortidev/etc/localtime

fnsysctl du

Show directories usage, accepts following options:

• -d n - Limit depth to n levels deep.

• -a - Show/count files as well, not only directories.

• -s - Show only the summary usage of all directories/files.

• -L - Follow all symlinks

Examples:

FGT-Perimeter #  fnsysctl du -s
715312  .

FGT-Perimeter #  fnsysctl du -L
4       ./new_root/eap_proxy_worker/fortidev/etc
4       ./new_root/eap_proxy_worker/fortidev
1256    ./new_root/eap_proxy_worker/etc/cert/ca
1256    ./new_root/eap_proxy_worker/etc/cert
1256    ./new_root/eap_proxy_worker/etc
0       ./new_root/eap_proxy_worker/dev/pts

...cut...

0       ./dev/shm/ips001
0       ./dev/shm/ips002
0       ./dev/shm/ips
3280    ./dev/shm
3280    ./dev
85811852        .

FGT-Perimeter #  fnsysctl du -d 1 -a
71960   ./new_root
20488   ./migadmin
5344    ./node-scripts
113596  ./bin
0       ./proc
0       ./fortidev
131464  ./data
142520  ./data2
0       ./boot
24      ./sbin
0       ./lib64
147440  ./tmp
11324   ./var
0       ./init
452     ./usr
0       ./etc
0       ./sys
67432   ./lib
0       ./root
3280    ./dev
715324  .

fnsysctl pwd

Show current working directory. Not very useful as we don’t have access to cd and thus cannot change directory anyway.

FGT-Perimeter # fnsysctl pwd
/

fnsysctl ps

List running processes. Useful together with the next command kill for restarting some stuck process on Fortigate. Most of the processes in Fortigate are run via Watch Dog which means killing them will shut the running process and will restart it immediately later.

FGT-Perimeter # fnsysctl ps
PID       UID     GID     STATE   CMD
1         0       0       S       /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
2         0       0       S       [kthreadd]
3         0       0       I       [rcu_gp]
4         0       0       I       [rcu_par_gp]
6         0       0       I       [kworker/0:0H-kblockd]
8         0       0       I       [mm_percpu_wq]
9         0       0       S       [ksoftirqd/0]
10        0       0       I       [rcu_sched]
11        0       0       I       [rcu_bh]
12        0       0       S       [migration/0]
13        0       0       I       [kworker/0:1-events_power_efficient]
14        0       0       S       [cpuhp/0]
15        0       0       S       [cpuhp/1]
16        0       0       S       [migration/1]
17        0       0       S       [ksoftirqd/1]
19        0       0       I       [kworker/1:0H-kblockd]
20        0       0       S       [kdevtmpfs]
32        0       0       I       [kworker/1:1-events]
37        0       0       I       [kworker/1:2-mm_percpu_wq]
217       0       0       I       [kworker/u4:2-fortilink]
345       0       0       S       [khungtaskd]
346       0       0       S       [oom_reaper]

...cut...

2019      0       0       S       /bin/autod
2020      0       0       S       /bin/cloudapid
2021      65530   65530   S       /bin/eap_proxy
2026      0       0       S       /bin/dnsproxy
2045      0       0       S       /bin/wad 4
2046      0       0       S       /bin/wad 5
2047      0       0       S       /bin/wad 6
2048      0       0       S       /bin/wad 12
2049      0       0       S       /bin/wad 13
2050      0       0       S       /bin/wad 14
2051      0       0       S       /bin/wad 9
2052      0       0       S       /bin/wad 18 0
2053      0       0       S       /bin/miglogd 1
2095      0       0       S       /bin/ipsengine
2096      0       0       S       /bin/ipsengine
2119      0       0       S       /bin/urlfilter 0
2123      65531   65531   S       /bin/imi -L 2
2124      0       0       R       /bin/sshd
2125      0       0       S       /bin/newcli
2204      0       0       I       [kworker/u4:1-events_unbound]
2319      0       0       I       [kworker/u4:0-events_unbound]
2325      0       0       S       /bin/httpsd

fnsysctl kill

Kill a process by its ID (PID). The only option accepted is -s N where N is the signal number to send as per Linux. Using the output of the fnsysctl ps above we can kill httpsd (Admin GUI process) like:

fnsysctl kill 2325

There are usually multiple processes for the same function, so it is more practical to use the next command instead - fnsysctl killall.

fnsysctl killall

Kill/restart a process by name. The only option is the name of the process. The example above for killing all httpsd processes will be:

FGT-Perimeter # fnsysctl killall httpsd

• When using killall it is not recorded in the crash log file (which you read with diagnose debug crashlog read).

• Not all processes can be killed with it, e.g. hasync.

fnsysctl mv

Move file in the filesystem. Most of the directories on the Fortigate are read-only, but some, like tmp are not. This command will ask for the username/password explicitly.

FGT-Perimeter # fnsysctl mv  /tmp/ipsshm.urldb-whitelist /tmp/ipsshm.urldb-whitelist.orig
Admin:admin
Password:

FGT-Perimeter # fnsysctl ls -al  /tmp/ipsshm.urldb-whitelist.orig
-rw-r--r--    1 0        0       Wed Oct 23 02:15:02 2024           810912 /tmp/ipsshm.urldb-whitelist.orig

Warning

Be careful with file moves as Fortigate may stop functioning if you delete a crucial file.

The obvious use for this command is for attackers who broke into Fortigate to hide their traces.

fnsysctl printenv

The only environment variable I was able to catch with this was type of Terminal used.

FGT-Perimeter # fnsysctl printenv
TERM=vt220

fnsysctl grep

Search contents of a file/files. The usual grep options are available:

        -i      Ignore case distinctions
        -l      List names of files that match
        -H      Prefix output lines with filename where match was found
        -h      Suppress the prefixing filename on output
        -n      Print line number with output lines
        -q      Quiet
        -v      Select non-matching lines
        -s      Suppress file open/read error messages
        -c      Only print count of matching lines
        -A      Print NUM lines of trailing context
        -B      Print NUM lines of leading context
        -C      Print NUM lines of output context

[kaman · Accepted solution]

Hi Yurisk,

Thank you for sharing the fnsysctl command options with examples.

You can also add troubleshooting NTrubo related issues.

fnsysctl cat /proc/nturbo/<n>/drv              <------  '<n>' is the NTurbo ID.

https://community.fortinet.com/t5/FortiGate/Technical-Tip-Useful-diagnostics-commands-for-troublesho...

[Yurisk]

Thank you, updated.

[AEK]

Thanks for sharing, Yuri

I wanted to share something similar (since it is not documented), but you did it in a more complete way.