Troubleshooting Tip: Collecting IPS engine related debugs

1. Show FortiGate stats and memory usage: In the VDOM environment, below below-mentioned commands work either under the 'config system global' or under:

config vdom
    edit <name of vdom>

get system status

get system performance status <----- Global VDOM.

diagnose hardware sysinfo memory <----- Global VDOM.

diagnose sys session stat

diagnose ips session list by-mem 15

diagnose ips session status

diagnose autoupdate version | grep "IPS Attack" -A 6 <----- Global VDOM.

diagnose ips memory status

diagnose test application ipsmonitor 1 <----- Global VDOM.

diagnose system top 2 30 4

diagnose system top-mem 10  <----- Global VDOM.

diagnose system top-mem 10

diagnose system top-mem 10

2. If the IPS engine's memory usage appears to be higher than normal, run 'diagnose sys process pstack $pid' (four times) to get the stack for the process.

diagnose system process pstack <PID with high usage>

fnsysctl cat /proc/$PID/smaps

3. Restart the process.

diagnose system kill 11 <PID>

4. Get the crash log as well.

diagnose debug crashlog read

Note:

Additional debugging commands for TAC.

diagnose debug reset

diagnose ips filter set "host x.x.x.x" <- x.x.x.x = testing host IP.
diagnose ips debug enable all <- Instead of 'all', use one from a list of categories displayed by pressing '?' after enable.

diagnose debug enable

Disable afterwards with the following command:

diagnose ips debug disable all

diagnose debug disable

Note: Selecting all for the IPS debug will cause high memory usage and can lead to kernel conserve mode, as this debug is copying sessions inspected by the IPS engine, which will lead to doubling the sessions.

It is always recommended to specify the type of session on the debug; also, it is strongly recommended to use the filtering command 'diagnose ips filter...' unless the goal of the debug is to collect events for all hosts.

In cases where an IPS profile is assigned to a firewall policy in proxy inspection mode, the recommended live debug commands are:

On the Filter, add several hosts, for example (x.x.x.x represents the source IP, and z.z.z.z represents the destination):

diagnose ips filter set 'host x.x.x.x and host z.z.z.z'

Filter parameters need to be used as it was used for IPS debugging.

diagnose debug reset

diagnose wad debug enable category [all, http, session ,...] <----- Use question mark '?' to view all available categories.

diagnose wad debug enable level verbose <- Verbosity level. When troubleshooting, it is recommended to use verbose.

diagnose wad filter dst 10.10.10.10  <----- Filtering the logs based on the destination address.

diagnose wad filter dport 443         <----- Filtering the logs based on the destination port.

diagnose debug console timestamp enable

diagnose debug enable

To confirm which filters are in use:

diagnose wad filter list 

drop unknown sessions: disabled 

dest ip: 10.10.10.10-10.10.10.10

dest port: 443-443

To disable the debug:

diagnose debug disable

diagnose debug reset

Related article:
Troubleshooting Tip: IPS engine new debug commands