Fortigate fnsysctl command options with examples
fnsysctl is frequently helpful in troubleshooting Fortigates, and while its options are mentioned in the Forums here and there, no single article lists them, and not all options are mentioned, so I wrote a post to summarize the info.
Originally posted on https://yurisk.info/2024/10/23/fortigate-fnsysctl-command-options-with-examples/
Important facts about fnsysctl command:
You have to log in with a user having super_admin profile.
For VM Fortigate, it has to have a regular license - not free evaluation one. On free evaluation VM FGT you will get an error Unknown action 0.
It is CLI-only command, with no GUI equivalent.
The command runs locally on the Fortigate you are logged in, so to run the same command on a passive member of HA cluster, you will need to log in into the passive member first.
The Tab completion does NOT work with this command (therefore this post).
We CAN use these commands in automation stitches as set action-type cli-script.
fnsysctl ifconfig
Shows detailed info on the physical interfaces, including drops/errors/MTU. Accepts optionally name of the interface e.g. fnsysctl ifconfig port1.
FGT-Perimeter # fnsysctl ifconfig port1 Link encap:Ethernet HWaddr 0A:7C:2A:D2:17:6F inet addr:10.100.100.227 Bcast:10.100.100.255 Mask:255.255.255.0 link-local6: fe80::87c:2aff:fed2:176f prefixlen 64 UP BROADCAST RUNNING MULTICAST MTU:9001 Metric:1 RX packets:3537 errors:0 dropped:0 overruns:0 frame:0 TX packets:5436 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:1340257 (1.3 MB) TX bytes:4360502 (4.2 MB) port2 Link encap:Ethernet HWaddr 0A:C2:8D:76:4D:8D inet addr:10.100.104.13 Bcast:10.100.104.255 Mask:255.255.255.0 link-local6: fe80::8c2:8dff:fe76:4d8d prefixlen 64 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:23 errors:0 dropped:0 overruns:0 frame:0 TX packets:129 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:644 (644 Bytes) TX bytes:5888 (5.8 KB)
fnsysctl ls
Lists files/folders in the filesystem. Useful for post-incident investigation of Fortigate compromises, looking for a given CVE indicators of compromise (IOCs).
It accepts only 3 flags:
a - Show all files, including those starting with the dot in their name.
l - Show long output, i.e. not only names but timestamps, sizes.
A - almost all, do not show names starting with the dot (default so no need to specify).
Examples:
FGT-Perimeter # fnsysctl ls -al /tmp drwxr-xr-x 2 0 0 Wed Oct 23 01:53:42 2024 40 $$auto-script$$ drwxrwxrwt 60 0 0 Wed Oct 23 02:03:46 2024 4780 . drwxr-xr-x 18 0 0 Wed Oct 23 01:53:40 2024 0 .. srwxr-xr-x 1 0 0 Wed Oct 23 01:53:42 2024 0 .auto_script_server -rw-r--r-- 1 0 0 Wed Oct 23 01:53:42 2024 0 .aws_addrs srwxr-xr-x 1 0 0 Wed Oct 23 01:53:42 2024 0 .cloudapi_fconv.sock srwxr-xr-x 1 0 0 Wed Oct 23 01:53:42 2024 0 .dhcpd.msg srwxr-xr-x 1 0 0 Wed Oct 23 01:53:42 2024 0 .dns_local_server
FGT-Perimeter # fnsysctl ls -a /tmp $$auto-script$$ . .dns_local_server .dns_local_server_for_proxy .dnsproxy_unix_server 0 .fgfm_stream_clt_sock .ipsengine001_0_0.url.socket .ipsengine002_0_0.url.socket .urlfilter0.sock .wad512_0_0.url.socket admin_server.crt KEY-FILE backtrace_log bwl_gui_to_url0_unix_sock
fnsysctl cat
Show contents of a file, not all files in the filesystem are accessible. Some examples.
Show Linux kernel version of the Fortigate (here FortiOS 7.4.3):
FGT-Perimeter # fnsysctl cat /proc/version Linux version 4.19.13 (root@build) (gcc version 10.3.0 (GCC)) #1 SMP Thu Feb 1 17:10:43 UTC 2024
When trying to access a prohibited file:
FGT-Perimeter # fnsysctl cat /tmp/cw_ac_key_bak.pem cat: /tmp/cw_ac_key_bak.pem: Not allowed
Show open TCP connections to/from Fortigate itself:
FGT-Perimeter # fnsysctl cat /proc/net/tcp sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode 0: 00000000:28A0 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 13871 1 ffff8880443a9200 100 0 0 10 0 0:0/0:0/0:0 0 1: 00000000:1E82 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 17550 1 ffff88804a0ece00 100 0 0 10 0 0:0/0:0/0:0 0 2: 00000000:2904 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 13877 1 ffff888042db2200 100 0 0 10 0 0:0/0:0/0:0 0
The output is in hex, so it is much easier to use diagnose sys tcpsock | grep 0.0.0.0.
Show CPU info:
FGT-Perimeter # fnsysctl cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 85 model name : Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz stepping : 7 microcode : 0x5003707 cpu MHz : 2499.998 cache size : 36608 KB physical id : 0 siblings : 2 core id : 0 cpu cores : 1 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic
Get memory information:
FGT-Perimeter # fnsysctl cat /proc/meminfo MemTotal: 1984244 kB MemFree: 595988 kB MemAvailable: 757016 kB Buffers: 10140 kB Cached: 597428 kB SwapCached: 0 kB Active: 591168 kB Inactive: 141344 kB Active(anon): 518884 kB Inactive(anon): 47496 kB Active(file): 72284 kB ...cut...
Show nturbo acceleration statistics fnsysctl cat /proc/nturbo/<n>/drv:
fnsysctl cat /proc/nturbo/<0>/drv Turbo interface ID: 0 ============================================================================ Driver RX/TX: 760818543/759413272 TX hang: No Free/Used buffers: 109675/2965 Alloc fail: 0, Bad qid: 0 queue ready: 0x0000007f, 0x00000000 RXQ_0(0,20806): IN 64201109 OUT 64201142 DROP 0 NRDY 0 Fullness 0, Peak 282 TXQ_0(0,20806): IN 64083848 OUT 64083848 DROP 0 SHAPER_DROP 0 USR_DROP 117056 BUFERR 0 RXQ_1(1,20808): IN 62241175 OUT 62241191 DROP 0 NRDY 0 Fullness 0, Peak 444 TXQ_1(1,20808): IN 62092654 OUT 62092654 DROP 0 SHAPER_DROP 0 USR_DROP 148288 BUFERR 0 RXQ_2(2,20807): IN 63028145 OUT 63028179 DROP 0 NRDY 0 Fullness 0, Peak 247 TXQ_2(2,20807): IN 62856041 OUT 62856041 DROP 0 SHAPER_DROP 0 USR_DROP 171904 BUFERR 0 RXQ_3(3,20809): IN 61829939 OUT 61830044 DROP 0 NRDY 0 Fullness 0, Peak 254 TXQ_3(3,20809): IN 61684861 OUT 61684861 DROP 0 SHAPER_DROP 0 USR_DROP 144928 BUFERR 0 RXQ_4(4,20810): IN 64154116 OUT 64154184 DROP 0 NRDY 0 Fullness 0, Peak 1408 TXQ_4(4,20810): IN 64009332 OUT 64009332 DROP 0 SHAPER_DROP 0 USR_DROP 144608 BUFERR 0 RXQ_5(5,20804): IN 63186535 OUT 63186600 DROP 0 NRDY 0 Fullness 0, Peak 221 TXQ_5(5,20804): IN 63097904 OUT 63097904 DROP 0 SHAPER_DROP 0 USR_DROP 88448 BUFERR 0 RXQ_6(6,20805): IN 63168351 OUT 63168429 DROP 0 NRDY 0 Fullness 0, Peak 432 TXQ_6(6,20805): IN 62918126 OUT 62918126 DROP 0 SHAPER_DROP 0 USR_DROP 250048 BUFERR 0
To decipher the output, see https://community.fortinet.com/t5/FortiGate/Technical-Tip-Useful-diagnostics-commands-for-troubleshooting/ta-p/194289
fnsysctl date
Show date in the Linux format, ignores any options.
FGT-Perimeter # fnsysctl date Wed Oct 23 02:11:03 PDT 2024
fnsysctl df
Show filesystem usage, useful when you have harddisk(s) attached to the Fortigate.
FGT-Perimeter # fnsysctl df -h Filesystem Size Used Available Use% Mounted on none 1.3G 81.6M 1.2G 6% /tmp none 1.3G 4.7M 1.3G 0% /dev/shm none 1.3G 70.0M 1.2G 5% /dev/cmdb /dev/nvme0n1p1 231.9M 129.2M 89.9M 59% /data /dev/nvme0n1p2 1.6G 141.7M 1.4G 9% /data2 /dev/nvme1n1p1 29.4G 54.8M 27.8G 0% /var/log /dev/nvme0n1p1 231.9M 129.2M 89.9M 59% /new_root/zebos/fortidev/etc/localtime none 1.3G 70.0M 1.2G 5% /new_root/eap_proxy/dev/cmdb /dev/nvme0n1p1 231.9M 129.2M 89.9M 59% /new_root/eap_proxy/etc/cert/ca /dev/nvme0n1p1 231.9M 129.2M 89.9M 59% /new_root/eap_proxy/fortidev/etc/localtime /dev/nvme0n1p1 231.9M 129.2M 89.9M 59% /new_root/eap_proxy_worker/etc/cert/ca /dev/nvme0n1p1 231.9M 129.2M 89.9M 59% /new_root/eap_proxy_worker/fortidev/etc/localtime
fnsysctl du
Show directories usage, accepts following options:
-d n - Limit depth to n levels deep.
-a - Show/count files as well, not only directories.
-s - Show only the summary usage of all directories/files.
-L - Follow all symlinks
Examples:
FGT-Perimeter # fnsysctl du -s 715312 .
FGT-Perimeter # fnsysctl du -L 4 ./new_root/eap_proxy_worker/fortidev/etc 4 ./new_root/eap_proxy_worker/fortidev 1256 ./new_root/eap_proxy_worker/etc/cert/ca 1256 ./new_root/eap_proxy_worker/etc/cert 1256 ./new_root/eap_proxy_worker/etc 0 ./new_root/eap_proxy_worker/dev/pts ...cut... 0 ./dev/shm/ips001 0 ./dev/shm/ips002 0 ./dev/shm/ips 3280 ./dev/shm 3280 ./dev 85811852 .
FGT-Perimeter # fnsysctl du -d 1 -a 71960 ./new_root 20488 ./migadmin 5344 ./node-scripts 113596 ./bin 0 ./proc 0 ./fortidev 131464 ./data 142520 ./data2 0 ./boot 24 ./sbin 0 ./lib64 147440 ./tmp 11324 ./var 0 ./init 452 ./usr 0 ./etc 0 ./sys 67432 ./lib 0 ./root 3280 ./dev 715324 .
fnsysctl pwd
Show current working directory. Not very useful as we don’t have access to cd and thus cannot change directory anyway.
FGT-Perimeter # fnsysctl pwd /
fnsysctl ps
List running processes. Useful together with the next command kill for restarting some stuck process on Fortigate. Most of the processes in Fortigate are run via Watch Dog which means killing them will shut the running process and will restart it immediately later.
FGT-Perimeter # fnsysctl ps PID UID GID STATE CMD 1 0 0 S /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 2 0 0 S [kthreadd] 3 0 0 I [rcu_gp] 4 0 0 I [rcu_par_gp] 6 0 0 I [kworker/0:0H-kblockd] 8 0 0 I [mm_percpu_wq] 9 0 0 S [ksoftirqd/0] 10 0 0 I [rcu_sched] 11 0 0 I [rcu_bh] 12 0 0 S [migration/0] 13 0 0 I [kworker/0:1-events_power_efficient] 14 0 0 S [cpuhp/0] 15 0 0 S [cpuhp/1] 16 0 0 S [migration/1] 17 0 0 S [ksoftirqd/1] 19 0 0 I [kworker/1:0H-kblockd] 20 0 0 S [kdevtmpfs] 32 0 0 I [kworker/1:1-events] 37 0 0 I [kworker/1:2-mm_percpu_wq] 217 0 0 I [kworker/u4:2-fortilink] 345 0 0 S [khungtaskd] 346 0 0 S [oom_reaper] ...cut... 2019 0 0 S /bin/autod 2020 0 0 S /bin/cloudapid 2021 65530 65530 S /bin/eap_proxy 2026 0 0 S /bin/dnsproxy 2045 0 0 S /bin/wad 4 2046 0 0 S /bin/wad 5 2047 0 0 S /bin/wad 6 2048 0 0 S /bin/wad 12 2049 0 0 S /bin/wad 13 2050 0 0 S /bin/wad 14 2051 0 0 S /bin/wad 9 2052 0 0 S /bin/wad 18 0 2053 0 0 S /bin/miglogd 1 2095 0 0 S /bin/ipsengine 2096 0 0 S /bin/ipsengine 2119 0 0 S /bin/urlfilter 0 2123 65531 65531 S /bin/imi -L 2 2124 0 0 R /bin/sshd 2125 0 0 S /bin/newcli 2204 0 0 I [kworker/u4:1-events_unbound] 2319 0 0 I [kworker/u4:0-events_unbound] 2325 0 0 S /bin/httpsd
fnsysctl kill
Kill a process by its ID (PID). The only option accepted is -s N where N is the signal number to send as per Linux. Using the output of the fnsysctl ps above we can kill httpsd (Admin GUI process) like:
fnsysctl kill 2325
There are usually multiple processes for the same function, so it is more practical to use the next command instead - fnsysctl killall.
fnsysctl killall
Kill/restart a process by name. The only option is the name of the process. The example above for killing all httpsd processes will be:
FGT-Perimeter # fnsysctl killall httpsd
When using killall it is not recorded in the crash log file (which you read with diagnose debug crashlog read).
Not all processes can be killed with it, e.g. hasync.
fnsysctl mv
Move file in the filesystem. Most of the directories on the Fortigate are read-only, but some, like tmp are not. This command will ask for the username/password explicitly.
FGT-Perimeter # fnsysctl mv /tmp/ipsshm.urldb-whitelist /tmp/ipsshm.urldb-whitelist.orig Admin:admin Password:
FGT-Perimeter # fnsysctl ls -al /tmp/ipsshm.urldb-whitelist.orig -rw-r--r-- 1 0 0 Wed Oct 23 02:15:02 2024 810912 /tmp/ipsshm.urldb-whitelist.orig
Warning | Be careful with file moves as Fortigate may stop functioning if you delete a crucial file. |
The obvious use for this command is for attackers who broke into Fortigate to hide their traces.
fnsysctl printenv
The only environment variable I was able to catch with this was type of Terminal used.
FGT-Perimeter # fnsysctl printenv TERM=vt220
fnsysctl grep
Search contents of a file/files. The usual grep options are available:
-i Ignore case distinctions -l List names of files that match -H Prefix output lines with filename where match was found -h Suppress the prefixing filename on output -n Print line number with output lines -q Quiet -v Select non-matching lines -s Suppress file open/read error messages -c Only print count of matching lines -A Print NUM lines of trailing context -B Print NUM lines of leading context -C Print NUM lines of output context
