Technical Tip: IPS memory optimization steps

[Anthony_E]

Description

This article describes how to optimize the system when high memory issue is happening with IPS process.

Scope

FortiGate.

Solution

Adjust the following settings.

config ips global

      set socket-size [integer, 0-512] <----- IPS socket buffer size. The max and default value depend on available memory. A lower value reduces memory usage. However if set too low it may cause the IPS engine to go into a fail-open state.

      set engine-count [integer, 0-255] <----- Number of IPS engines running. With the default value of 0, FortiOS sets the number to optimize performance depending on the number of CPU cores. Reduce it in small increments, and monitor the CPU usage per core, the fewer IPS engines spawned, the more load will be focused on less number of cores. Find the balance between Memory and CPU usage.

      set database [regular|extended]      <----- Regular protects against the latest common and in-the-wild attacks. Extended includes protection from legacy attacks.

  end 

After changing the engine, database, and socket size, restart the IPSEngine using the following commands:

diag test app ipsmonitor 99  
diag test app ipsengine 99

Example: In the below example engine count is set to 2:

     config ips global

         set socket-size 256
         set engine-count 2

         set database extended
     end

This will cause FortiOS to reduce number of processes for scan engines and limit amount of processes spawned by IPS. These commands need to be run on both primary and secondary FortiGates since these settings are not replicated between both HA cluster members. Performing the activity of optimizing the IPS engine will terminate IPS-related sessions so care should be taken to run them during off-peak hours.

To verify if the changes are applied or not, run the below commands: 

    diag sys top 

    CTRL + c      <----- To stop the debugs

    diag sys top-mem      <----- verify if the IPS engine process is running more than 2.

Verify overall memory usage on the FortiGate :

       get system performance status 

Note:

Multiple CLI tools can be used to investigate IPS engine memory and CPU usage, which are useful when troubleshooting such a scenario:

diagnose sys session stat
diagnose ips session status
diagnose ips session performance
diagnose ips dissector status
diagnose ips packet status
diagnose ips session list
diagnose ips ssl status
diagnose ips memory status

One way to troubleshoot memory leaks by the IPS engine or as a step to improve IPS engine memory usage is to disable hardware acceleration for the IPS engine:

config ips global
    set cp-accel-mode none
    set np-accel-mode none
end

Related articles:

Technical Tip: Changing the IPS database

Technical Tip: How to manually upgrade the IPS Engine

Technical Tip: Reduce memory usage by reducing the number of spawned daemons