Troubleshooting Tip: Conserve mode due to the IPS Engine or WAD

[Nivedha]

Description	This article describes how to collect logs when FortiGate is in conserve mode due to the IPS Engine or WAD.
Scope	FortiGate.
Solution	Conserve mode is triggered when memory consumption reaches the red level, and traffic starts dropping when memory consumption reaches an extreme level. Check the following references to understand how the conserve mode is triggered: Technical Tip: How conserve mode is triggered Technical Tip: Conserve mode changes in FortiGate 5.6 and above Run diagnose sys top 1 99 or diagnose sys top-mem <value> to check if IPSEngine or WAD is consuming a lot of memory. If the IPS Engine consumes a lot of memory : The second column lists the process id of the IPS Engine. Make a note of the process ID. Further, collect the following logs and open a TAC case for further troubleshooting.  fnsysctl df -h Using diagnose sys top-mem <value> to find the process ID of the IPS engine daemon, using diagnose command: diagnose sys top-mem 20 ----> list 20 processes with more Memory used. Example: diagnose sys top-mem 20 ipshelper (9078): 334500kB <---- Process ID: 9078. ipsengine (9083): 312183kB ipsengine (9085): 299577kB ipsengine (9081): 296822kB ipsengine (9080): 290039kB ipsengine (9084): 287881kB ipsengine (9079): 286787kB ipsengine (9082): 281535kB node (240): 97729kB mvl.user (159): 66131kB cid (312): 43287kB forticron (246): 28179kB miglogd (257): 26566kB cmdbsvr (194): 25987kB wad (481): 25055kB wad (485): 25011kB wad (486): 25011kB wad (488): 25011kB wad (489): 25011kB wad (490): 25011kB Top-20 memory used: 2827313kB Then run these commands: fnsysctl cat /proc/[process id]/status fnsysctl cat /proc/[process id]/maps fnsysctl cat /proc/[process id]/smaps Example: fnsysctl cat /proc/9078/status fnsysctl cat /proc/9078/maps fnsysctl cat /proc/9078/smaps Repeat the same diagnosis commands for all ID processes with high memory values. Along with this also collect the following debugs: diagnose ips memory track-glib enable diagnose ips memory track-glib dump diagnose ips memory track enable diagnose ips memory track-print diagnose ips memory track-size  get sys performance status diagnose hardware sysinfo memory diagnose sys session full-stat diagnose ips session status diagnose ips packet status diagnose ips memory status diagnose sys top-mem 50 diagnose sys top 1 99 10 <----- It will run for 10 seconds and then stop automatically. diagnose sys top-fd 50 diagnose test application ipsmonitor 3 diagnose test application ipsmonitor 14 diagnose test application ipsmonitor 15 diagnose test application ipsmonitor 24 diagnose sys top-sockmem fnsysctl df fnsysctl ls -al /tmp fnsysctl ls -al /dev/shm After finishing: diagnose ips memory track-glib disable diagnose ips memory track disable If the WAD daemon consumes high memory, collect the output of these commands in working and non-working scenarios and contact Fortinet support to further investigation: get system status diagnose hardware sysinfo memory diagnose hardware sysinfo shm diagnose hardware sysinfo slab get sys perf status diagnose sys session stat diagnose sys top-mem 50 diagnose sys vd list | grep fib diagnose sys top-fd 30 diagnose sys mpstat 1 5 <----- Wait for 5 seconds and then press q to exit diagnose sys top-all 2 30 5 <----- It will run for 5 seconds and then stop automatically  fnsysctl ps fnsysctl cat /proc/stat fnsysctl cat /proc/interrupts fnsysctl df -h fnsysctl df -k fnsysctl ls -l /tmp fnsysctl du -i /tmp fnsysctl du -a /tmp fnsysctl du -a / -d 1 fnsysctl du -i /dev/shm fnsysctl du -a /dev/shm fnsysctl du -i /node-scripts fnsysctl du -a /node-scripts diag sys process pidof wad fnsysctl cat /proc/<WAD pid>/status fnsysctl cat /proc/<WAD pid>/smaps diagnose sys process trace <WAD PID> diagnose sys process dump <WAD PID> diagnose sys process pstack <WAD PID> diagnose sys process sock-mem <WAD PID> diagnose wad memory report diagnose wad memory workers diagnose wad memory sum diagnose wad memory track diagnose wad memory overused diagnose wad memory monitor list diagnose wad stats worker diagnose wad stats worker.sysmem Note: 'fnsysctl' command requires Super Admin access to execute. FortiGate will produce an error otherwise. For further information, see Technical Tip: fnsysctl command returns Unknown action 0. Below is an example output of the initial part of the command 'diag wad memory report', where it shows the WAD process ID and name that is consuming an abnormally significant amount of memory (In the 'diag sys top' output, only the WAD process ID is known).  If this is monitored and keeps on increasing as time goes by, then it is a possible indication of a WAD memory leak. Once observed, reach out to TAC for further checking of the issue.  WAD Debugs: diagnose debug reset diagnose debug enable diagnose test application wad 1000 From the output of 'diagnose test application wad 1000', note down the type number and index value of the process (PID) consuming high memory.  For example: if WAD PID 290 is consuming high memory which is verified from the output of 'diag sys top 2 50', note down the type number and index value of the PID 290 from the output of 'diagnose test application wad 1000'. Process [8]: type=user-info(5) index=0 pid=290 state=running <----- In this case, the type value is 5 and the index is 0. Use the type and index values obtained in the previous step to run the following command: 'diagnose test application wad 2y0x'. Replace y with the type number and xx with the index value. In the above example, the command would be 'diagnose test application wad 2500'. Run the following commands: diagnose test application wad 803 diagnose test application wad 2 diagnose test application wad 3 diagnose test application wad 13 diagnose test application wad 21 diagnose test application wad 25 diagnose test application wad 27 diagnose test application wad 33 diagnose test application wad 70 diagnose test application wad 103 diagnose test application wad 104 diagnose test application wad 105 diagnose test application wad 112 diagnose test application wad 113 diagnose test application wad 114 diagnose test application wad 117 diagnose test application wad 120 diagnose test application wad 123 diagnose test application wad 130 diagnose test application wad 132 diagnose test application wad 156 diagnose test application wad 157 diagnose test application wad 158 diagnose deb disable For memory leak issues caused by WAD, repeat this for each WAD process identified with high memory consumption. Contact TAC and share all diagnostic commands. Related articles: Troubleshooting Tip: How to do initial troubleshooting of high memory utilization issues (conserve m... Technical Tip: How to restart the WAD process Technical Tip: How to restart a specific WAD worker process Technical Tip: How to troubleshoot high CPU usage caused by the IPS process Troubleshooting Tip: High CPU and MEMORY usage problem Technical Tip: How conserve mode is triggered Technical Tip: Collect report for WAD high memory related issue for Fortinet TAC assistance  Troubleshooting Tip: FortiGate enters Conserve Mode due to WAD Virtual Server Memory Leak