IPS engine is crashing

Following TAC suggestion,  but just to be clear, you get these messages ONLY when sending traffic between subnets?

How many rules do you have IPS sensors enabled on?

Do you really need IPS rules for intra-subnet  traffic ?

How much avg/ma cpu/memory ?

Are you sure the firewall is not under sized? What the amount of traffic being sent? What model of FGT ?

Bottom line fail-open IPS is not a good thing and numerous issues can cause this issue at least your traffic is not impeded.

You will probably need to work with TAC. I would also not rule out alogrithm methods used in the  ips global cfg. I've see issues when engine-pick algorithm was used on lower end devices. You can play with that and low settings & monitor the cpu/memory and if any overall improvements.

Thanks for your quick reply here are the answer for your question? yes this messages displayed only when their is a lan traffic between two different subnets.

Q)How many rules do you have IPS sensors enabled on? ans: I don't see any column which is labled as "IPS" on policy tab. i think ips is globally enabled for all the policy. if not how to disable the ips for particular policy. provide me the steps for the same.

Q)Do you really need IPS rules for intra-subnet traffic ? ans: i do not have any idea weather ips is really required for the intra-subnet or not. but as per fortinet technical support team it is not good idea to disable ips for policy. please comment on this

Q)How much avg/ma cpu/memory ?

ans: when only internet traffic is their then avg mem is near about 50% cpu also 50% 

Q)Are you sure the firewall is not under sized? What the amount of traffic being sent? What model of FGT ?

ans:yes. firewall is under the sized. firewall model is 90 D

throughput of the firewall is 3.2 GbPS. lan data traffic is only 150 Mbps max at the time of ips engine gets crash.

below are the changes made by TAC but still issue exist.

# config ips global   # set engine-count 4   # set algorithm low   # set socket-size 1   # end   # diag test app ipsmonitor 99   Reduce the session timers to close unused sessions faster   #config system global   #set tcp-halfclose-timer 30   #set tcp-halfopen-timer 30   #set tcp-timewait-timer 0   #set udp-idle-timer 60   #end  

I m waiting for your reply 

To get ant ideal of how many sessions with active ips you could dump  the session table and look at the ips

e.g

diag sys session list | grep ips

You could also review the firewall policyId from the above and the reference sensor in the firewall config

A2; but you need to know what your inspecting. Did  support-TAC or any consultant configure these policies and for why? Was it trimmed and monitor for > & for the client-2-server ? or server-2client traffic?

A3: So support made changes, did they pull your logs and look at any events? They obviously made a ips engine count change and did my suggest "low" but  what you probably need to do which goes back to A1; you need to find what your inspecting.

>The diag ips  session list will show you active session and even helps by posting the Client and Server in the details.

>The diag ips session status will show  you  the memory used and available, some one can correct me but that's shared memory for the IPS enginer iirc.

e.g

diag ips  session status SYSTEM: memory capacity            104M memory used                23M recent pps\bps             0\0K session in-use             0 TCP:  in-use\active\total  0\0\0 UDP:  in-use\active\total  0\0\1      < ---------protocols that are enabled ICMP: in-use\active\total  0\0\0 IP:   in-use\active\total  0\0\0

Find what you have enabled, the  characteristics  of the sensor ( what 's enabled in that sensor rules ) and make sure you have the latest updates.

If you have any >> any policy with a sensor enabled and all rules  than that is probably a bad thing. i don't believe your firewall is undersize btw, probably just poor designing of the ips -sensors and/or policy-id that are enabled. I would find the latest FortiOS ips guide and study that and then make and monitor corrections  for improvements.

FWIW

The get sys performance status is a helpful status to monitor cpu/mem and ips  events but i don't know how to reset this without a reboot. So you have some work cut out for you ;)

I hope the above helps and get you started.

You still haven't determine what policies have ips protection and what rule you have enabled in the sensors. I would follow TAC and upgrade BUT also you need to trim and police the  IPS sensors. What are you trying to protect  between internal---2---internal? ( Application server, mail,web,etc....)

In your IPS details I see alot of Client to Server with service 443? Are you also deploying  SSL inspection?

And lastly, did you pull the latest  Fortigate IPS guide and review the pdf? I would read this 1st

http://docs.fortinet.com/uploaded/files/1082/fortigate-security_profiles-50.pdf

and then look at your IPS and determine if you need anomaly ( aka DoS sensor ) or signature based protection. You can't just blindly enable these and NOT understand the results and impact. Also they need a careful eye that's on going to ensure you have the best protection vrs performance.

You most likely will end up with exemptions, adjust and thresholds set and continously re-adjusted during the lifetime of the sensor deployment.

Did you check all policy from the CLI?

Another quick way to determine if you have IPS enable;

diag ips signature   status

or

diag ips anomaly  status

Did you follow TAC suggestion and upgrade?

FWIW:if you have ips_view and have your system crashing due to IPS engine, than it's mostly likely due to your IPS being enabled regardless of what features you have checked in the gui. That's just the features you have enabled per-WebGUI.

Get back in touch with TAC, and have them guide you on the problem and resolution. If you still have issues.

Hi,

If you disable the ips feature from GUI, it doesn't mean that you disable the ips engine.

You should connect in CLI and performs this command:

config fireall policy

 edit <policy ID>

   show full-config

If you don't mind post it.

Otherwise, search the ips-sensor field... it should be blank. If it's not blank, do this: unset ips-sensor.

Regards,

Radu

Hi,

If the traffic hit this policy it shouldn't be inspected by the ips, since the utm features are disabled.

   set utm-status disable  -> from the file that you attached

You could perform this command to disable the ips engine until next reboot or until you re-enable it.

diag test application ipsmonitor 2