Skip to main content
JNDias
Staff & Editor
Staff & Editor
February 8, 2023

Technical Tip: Restart WAD or IPS when conserve mode hits (Automation Stitch)

  • February 8, 2023
  • 0 replies
  • 22313 views

Description

This article describes how to create automation to restart a process when the FortiGate reaches conserve mode. This can be adapted to execute other commands or restart other processes depending on the issue.

 

This should only be applied as a temporary workaround while waiting for a bug fix.

Scope

FortiGate v7.0, v7.2, v7.4.

Solution

Create an Automation stitch to try restarting the WAD or IPS processes.

 

Result:

 

JNDias_4-1675849122524.png

 

It is possible to apply these settings directly in the CLI (as shown at the bottom of this article) or in the GUI (as shown below).

 

Steps in the GUI:


Create Action (Automation stitches).

 

JNDias_0-1675849073012.png

 

Script for WAD process:

diagnose test application wad 99

 

Script option for IPS process:

diagnose test application ipsmonitor 99

 

A more forceful approach to restart the wad and ipsmonitor processes is to execute the fnsysctl killall wad or fnsysctl killall ipsmonitor command. This will require a super-administrator account to perform and execute, always with caution.

 

Create a trigger.

 

JNDias_1-1675849079062.png

 

JNDias_2-1675849086977.png

 

Create a stitch.

 

JNDias_3-1675849094937.png

CLI option:

config system automation-action
    edit "RestartWAD"
        set action-type cli-script
        set minimum-interval 5
        set script "diag test app wad 99"
        set accprofile "super_admin"
    next
end

config system automation-trigger
    edit "Enters Conserve Mode"
        set event-type low-memory
    next
end
 
config system automation-stitch
    edit "Restart processes"
        set trigger "Enters Conserve Mode"
            config actions
                edit 1
                    set action "RestartWAD"
                    set required enable
                next
            end
    next
end

 

Alternative time-based triggers instead of memory.

It is recommended to restart WAD or IPS daily during a time of low use to avoid impacting the network. Otherwise, the FortiGate may miss automation when in conserve mode because of non-viable memory.

2024-10-15_12-18.png

 

2024-10-15_12-25.png

 

config system automation-trigger
    edit "Daily_at_3AM"
        set trigger-type scheduled
        set trigger-hour 3
    next
end


config system automation-action
    edit "RestartWAD"
        set action-type cli-script
        set minimum-interval 5
        set script "diag test app wad 99"
        set accprofile "super_admin"
    next
    edit "Restart_ipsmonitor"
        set action-type cli-script
        set minimum-interval 5
        set script "diagnose test application ipsmonitor 99"
        set accprofile "super_admin"
    next
end


config system automation-stitch
    edit "Restart processes"
        set trigger "Daily_at_3AM"
            config actions
                edit 1
                    set action "RestartWAD"
                    set required enable
                next
                edit 2
                    set action "Restart_ipsmonitor"
                    set required enable
                next
            end
    next
end


To view the results of the script named 'status' (with no VDOMs):

execute auto-script result status

Script status output:
########## script name: status ##########

========== #1, 2019-10-01 14:24:04 ==========
FGT $ get system status
Version: FortiGate-100D v6.2.1,build0932,190716 (GA)
Virus-DB: 72.00005(2019-10-01 03:19)
Extended DB: 1.00000(2018-04-09 18:07)
... output continues ...


To view the results of the script named 'status' (with VDOMs, enter it in global):

 

config global
execute auto-script result status

 

Note for WAD:

There is a new alternative technique to restart WAD from FortiOS v7.2.

New FortiOS mechanism to automatically restart WAD workers: Technical Tip: Automatically restart WAD worker processes.

This can be applied as a safeguarding mechanism along with the steps outlined in this article.

Restart WAD or IPS at a specific memory usage before the unit goes into conserve mode: 

To configure automation stitch before the unit goes into conserve mode at a specific memory usage percentage (i.e., 80%), refer to the following article: Technical Tip: Getting automation stitch before the unit goes into Conserve Mode. Using this method, wad and IPS can be restarted at a specific memory usage percentage (even before the unit goes into conserve mode).

Note for Automation stitch result storage:

FortiGate cannot natively hold or cache the result of an automation (auto) script in memory for some minutes minutes. The automation framework processes the script’s output (%%results%%) only while the automation stitch is running and immediately passes it to the next action (for example, an email or webhook). Once the stitch completes, the data is discarded from memory.

529eec51.png


Related documents: