| Sometimes, IPS crashes due to the IPS engine hitting a bug or exhausting resources on FortiGate. This can cause traffic disruptions where the IPS/Application control is used which are flow-based engines handled by IPS engine.
As a workaround, the following can be applied on a case-by-case basis in cases where impact is higher and causes traffic disruptions. Note: This is a temporary workaround until a permanent fix is found as the IPS scan is important for scanning traffic. Enable the fail-open on global IPS as follows:
config ips global
set fail-open enable <----- The default is disabled.
end
- Fail open can be enabled at the time of changes (upgrade or downgrade) to the IPS engine and can reset the setting after the changes. This avoids traffic disruptions.
When IPS fails open, the following crash log entry can be seen with the command 'diagnose debug crashlog read'.
IPS enter fail open mode: engines=4 socketsize=67108864
packet_action=drop In this case, it will also be useful to increase the socket size of the IPS a little and to see the current socket size 'diagnose test app ipsmonitor 1'.
config ips global
set socket-size <int>
end
- Collect the TAC report before making changes to investigate the root because of high CPU/memory:
diagnose debug reset
diagnose debug enable
execute tac report
FortiGate can be configured with the automated restart of the IPS process in case of high CPU/memory with fail-open enabled.
- Go to Security Fabric -> Automation, select 'Create New', name the automation stitch -> IPS restart, under Stitch add a Trigger, select 'Create' and select 'high CPU' or 'high Memory' then select 'Apply'.
- Add an Action, select 'Create' and 'CLI Script", name it and enter script as diagnose test app ipsmon 99, select 'Administrator Profile' as 'super_admin', select 'OK' to save changes, select the 'Add+' icon again for the action, select the CLI script created, and select 'Apply' to add the Action.
- Once all changes are done, select 'Apply-OK' at the bottom to save changes.
- Revert the changes of fail-open and automation script when a stable version/fix is found for the IPS crashing.
Refer to the following document for more information on setting automation:
Execute a CLI script based on CPU and memory thresholds
Other IPS engine debug commands: Troubleshooting Tip: IPS engine new debug commands | 
