|
Access FortiGate via the putty and log the putty session output.
The following debugs are to be captured in both working and non-working states for comparison.
Either the FortiGate debug report or 'diagnose sys top' will show this.
Run Time: 90 days, 9 hours and 30 minutes
2U, 0N, 3S, 92I, 0WA, 0HI, 3SI, 0ST; 16048T, 6133F
sslvpnd 276 S 14.5 0.3
sslvpnd 28175 S 13.0 0.4
sslvpnd 279 S 11.0 0.4
sslvpnd 25931 S 10.5 0.4
sslvpnd 8272 S 8.0 0.4
sslvpnd 26501 S 7.0 0.1
sslvpnd 3861 S 5.5 0.2
The high number of process IDs indicates possible crashes, which can also be verified by running the command 'diagnose debug crashlog read' on the FortiGate command line interface, or by searching for it in the debug report.
These additional commands are used for isolating the cause for sslvpnd high memory usage or high CPU usage:
fnsysctl date
fnsysctl ps
execute tac report
get system performance status <----- 5-6 iterations.
diagnose sys mpstat
diagnose hardware sysinfo interrupt
diagnose hardware sysinfo conserve
diagnose hardware sysinfo mem
diagnose hardware sysinfo cpu
diagnose hardware sysinfo slab
diagnose hardware sysinfo shm
fnsysctl ls -l dev/cmdb/.hidden/sslvpn/
fnsysctl ls -l dev/cmdb/.hidden/sslvpn/tmp
fnsysctl ls -l /etc/cert/local
diagnose sys process pidof sslvpnd
diagnose sys process pstack <PID> <----- Run this for each pid number from the above step.
diagnose sys process dump <PID> <----- Run this for each PID number from the above step.
diagnose sys process trace <PID> <----- Run this for each PID number from the above step.
diagnose sys process sock-mem <PID> <----- Run this for each PID number from the above step.
diagnose sys profile cpumask # <----- ID of the CPU running high, seen in get system performance status
diagnose sys profile start
<wait for 1-2 minutes>
diagnose sys profile stop
diagnose sys profile show order
diagnose sys profile show detail
diagnose sys tcpsock
diagnose sys top-fd 50
fnsysctl cat /proc/sys/fs/file-max
fnsysctl cat /proc/sys/fs/file-nr
fnsysctl cat /proc/sys/fs/inode-nr
fnsysctl cat /proc/sys/fs/nr_open
fnsysctl cat /proc/sys/fs/inode-state
diagnose vpn ssl debug-filter clear
diagnose vpn ssl debug-filter list
diagnose debug console timestamp enable
diagnose debug duration 0
diagnose debug application sslvpn -1
diagnose debug info
diagnose debug enable
diag sys top 60 <----- Collect the output of the SSL VPN debug log.
Using a script to run the below commands every 5-10 minutes:
diagnose debug enable
diagnose debug console timestamp enable
fnsysctl date
get vpn ssl monitor <----- Run this in all the VDOMS where SSL VPN is configured.
diagnose vpn ssl list <----- Run this in all the VDOMS where SSL VPN is configured.
diagnose vpn ssl stat
diagnose vpn ssl mux-stat
get system performance status
diagnose system session stat
diagnose sys top 2 30 5
diagnose sys mpstat 2 5
Once the data is captured at the time of the issue, restart the SSL VPN daemon by using the command 'diagnose sys kill 11 <PID>' (PID is the process ID of sslvpnd process), then run the 'diagnose debug crash read' command for the backtrace; there should be a new entry with for sslvpnd at the bottom.
Contact Fortinet Technical Support for assistance with capturing the debugs or automating the process with a script.
Another thing to check is the amount of TCP SYN attempts to the SSL VPN port. If there are a lot of connection attempts to the SSL VPN port but not being successful, it can also result in high CPU caused by sslvpnd.
Check successful SSLVPN logins and SSL VPN attempts:
get vpn ssl monitor
diagnose sniffer packet any 'tcp port <sslvpn.port>' 4 1000
If confirmed, try to change the SSL VPN port (temporarily) in an attempt to lower the high CPU usage caused by sslvpnd process.
Note:
To execute, the 'fnsysctl' command requires Super_admin (administrator account with super_admin permission profile) access. FortiGate will produce an error otherwise. For further information, see this KB article: Technical Tip: fnsysctl command returns Unknown action 0
Related article:
Technical Tip: SSL VPN daemon consumes high CPU due to brute force attacks
|
