|
- Sometimes, IPS crashes due to the IPS engine hitting a bug or exhausting resources on FortiGate.
This can cause traffic disruptions where the IPS/Application control is used which are flow-based engines handled by IPS engine.
- To overcome as a workaround, the below can be applied on a case-by-case basis in case of impact is more and causes traffic disruptions.
Note.
This is a temporary workaround until a permanent fix is found as the IPS scan is important for scanning traffic.
- Enable the fail-open on IPS global as per below:
# config ips global
set fail-open enable <----- Default is disabled.
end
- Fail open can be enabled at the time of changes (upgrade or downgrade) to the IPS engine and can reset the setting after the changes.
This avoids traffic disruptions.
In case of IPS fails open, the following crash log entry can be seen with the command 'diag debug crashlog read'.
IPS enter fail open mode: engines=4 socketsize=67108864
packet_action=drop
In this case, it will also be useful to increase the socket size of the IPS a little and to see the current socket size 'diag test app ipsmonitor 1'.
- Collect the TAC report before changes to investigate the root because of high CPU/memory:
# diag debug reset
# diag debug enable
# exec tac report
- FortiGate can be configured with the automated restart of the IPS process in case of high CPU/memory with fail-open enabled.
- Go to Security Fabric -> Automation, select 'Create New', name the automation stitch -> IPS restart, under Stitch add a Trigger, select 'Create' and select 'high CPU' or 'high Memory' then select 'Apply'.
- Add an Action, select 'Create' and 'CLI Script", name it and enter script as # diag test app ipsmon 99, select 'Administrator Profile' as 'super_admin', select 'OK' to save changes, select' Add+' icon again for action, select the CLI script created and select 'Apply' to add the Action.
- Once all changes are done, select 'Apply-OK' at the bottom to save changes.
- Revert the changes of fail-open and automation script when a stable version/fix is found for the IPS crashing.
Refer to the below article for more information on setting automation:
https://docs.fortinet.com/document/fortigate/7.0.5/administration-guide/702937/execute-a-cli-script-...
Other IPS engine debug commands:
https://community.fortinet.com/t5/FortiGate/Troubleshooting-Tip-IPS-engine-new-debug-commands/ta-p/2...
|
