Technical Tip: Restart WAD or IPS when conserve mode hits (Automation Stitch)

JNDias · ‎02-08-2023

Description

This article describes how to create automation to restart a process when the FortiGate reaches conserve mode. This can be adapted to execute other commands or restart other processes depending on the issue.

This should only be applied as a temporary workaround while waiting for a bug fix.

Scope

FortiGate v7.0, v7.2, v7.4

Solution

Create an Automation Stitch to try restarting the WAD or IPS processes.

Result:

It is possible to apply these settings directly in the CLI (as shown at the bottom of this article) or in the GUI (as shown below).

Steps in the GUI:

Create Action (Automation stitches).

Script for wad process:

diagnose test application wad 99

Script option for IPS process:

diagnose test application ipsmonitor 99

Create a trigger.

Create a Stitch.

CLI Option.

config system automation-action

edit "RestartWAD"

set action-type cli-script

set minimum-interval 5

set script "diag test app wad 99"

set accprofile "super_admin"

config system automation-trigger

edit "Enters Conserve Mode"

set event-type low-memory

config system automation-stitch

edit "Restart processes"

set trigger "Enters Conserve Mode"

config actions

edit 1

set action "RestartWAD"

set required enable

Alternative time-based triggers instead of memory.

It is recommended to restart WAD or IPS daily during a time of low use to avoid impacting the network. Otherwise, the FortiGate may miss automation when in conserve mode because of non-viable memory.

config system automation-trigger
edit "Daily_at_3AM"
set trigger-type scheduled
set trigger-hour 3
next
end

config system automation-stitch
edit "Restart processes"
set trigger "Daily_at_3AM"
config actions
edit 1
set action "RestartWAD"
set required enable
next
edit 2
set action "Restart_ipsmonitor"
set required enable
next
end
next
end

To view the results of the script named 'status' (with no VDOMs):

exec auto-script result status

Script status output:
########## script name: status ##########

========== #1, 2019-10-01 14:24:04 ==========
FGT $ get system status
Version: FortiGate-100D v6.2.1,build0932,190716 (GA)
Virus-DB: 72.00005(2019-10-01 03:19)
Extended DB: 1.00000(2018-04-09 18:07)
... output continues ...

To view the results of the script named 'status' (with VDOMs - enter it in global):

config global
exec auto-script result status

Note for WAD:

There is a new alternative technique to restart WAD from FortiOS v7.2:

New FortiOS mechanism to automatically restart WAD workers.

This can be applied as a safeguarding mechanic along with the steps outlined in this article.

Related documents:

Automation stitches | docs

Technical Tip: Creating automation stitches

Technical Tip: Automation stitch for FortiGate events

Technical Tip: How to check why automation stitch is not working as expected

Technical Tip: How to restart WAD process using automated script