Technical Tip: Automatically restart WAD worker processes

FortiOS v7.2.4 has introduced a new feature to configure the auto-restart of WAD worker processes.

That option can be configured under 'config system global'.

A restart of a WAD worker process can be configured based on the following parameters:

• 'set wad-restart-mode memory': when this option is enabled, WAD will detect a potential memory leak based on a mem-leak detection algorithm which automatically restarts WAD when the algorithm detects a possible leak.

The memory leak detection algorithm works like this:

1. WAD will collect memory usage every minute and log an average value every 30 minutes.
2. WAD compares the numbers by a cycle of one day.
3. If one process uses memory this day 80% more times than yesterday's amount across the same time period, it will be considered as a potential memory leak.
4. Two consecutive days of potential leaking will cause the process to go into a stale state. At this time, a new process will be started to replace the old one. The old process will exit after the sessions running being processed.

When 'set wad-restart-mode time' is applied and time reaches the time window defined by wad-restart-start-time and wad-restart-end-time, WAD will gracefully restart;

The default option is 'wad-restart-mode none' which means that there is no automatic WAD restart.

Syntax:

set wad-restart-mode ?

none      <- Disable restart of WAD workers.

time      <- Enable daily restart of WAD workers.

memory    <- Enable restart of WAD workers based on memory usage.

Restart the WAD in a specific time frame:

config system global

    set wad-restart-end-time 04:00

    set wad-restart-mode time

    set wad-restart-start-time 01:00

end

Restart the WAD based on memory usage:

config system global

     set wad-restart-mode memory

end

Note:

This feature is specifically designed for WAD workers. If the issue persists with other WAD processes and the solution above does not resolve it, consider using Automation Stitches as an alternative method to restart the processes

Related articles:

Technical Tip: Overview of WAD process structure

Technical Tip: Restart WAD or IPS when conserve mode hits (Automation Stitch)

Technical Tip: FortiGate Resource Lists