CPU and Memory threshold best practices

CPU and memory load are handled differently.

CPU spiking, even up to 100%, is OK. Average CPU load above, say, 50%, would indicate that the HW is underpowered. That is my personal subjective opinion. The CPU is needed for session setup, DNS, NTP, NAT and more, some GUI as well, so a high CPU mark would most probably indicate that too many sessions per second are opened (for instance). Beware that handling the HA traffic needs resources too.

In my experience, permanent CPU overload occurs seldom. If it does, funny things can happen. My old 50B just rebooted after some minutes with 100% CPU load...but that's long ago.

High memory usage on the other hand has the potential to compromise functionality up to fatal breakdown. FortiOS will terminate services (processes) if memory usage exceeds 80%. This is documented. The error log message at this moment declares "conserve mode entered" which should ring all alarm bells. If memory consumption is still rising, you will lose the WebGUI at one point, and later the FGT will stall.

For instance, loading an 80C rev.1 model with 512 MB RAM with FortiOS v5.4 immediately produced 67% memory usage. I would not put that into production, or else I would have to watch it closely.

HTH.