|
When commands: 'get system performance status ' and 'diag sys mpstat' are used, high CPU utilization is seen.
There could be so many reasons why the CPU could be higher. One of the most common reasons is due to most of the traffic is being processed by the CPU and not offloaded to the hardware.
Some troubleshooting steps that can be followed to track the traffic:
- Remove IPS inspection/UTM from Policies that are not required.
- Packets that are being blocked also impact the CPU. It is important to identify the large traffic that is being sent to FortiGate and avoid it from reaching FortiGate. Some flow uses IPS to scan and cause high CPU. After the scan, it tries to match the firewall policy and gets dropped if it does not match any policy causing a high CPU. In some cases, for this large traffic, a policy can be created to match this traffic and offload it to NPU.
- Traffic can be blocked using several ways.
Sender --> Flow-Rule --> FW ACL Policy --> FW Policy
Sender: Traffic can be blocked at the sender itself.
Flow-Rule: Traffic can be blocked by creating a flow rule. This traffic will not go to the CPU.
FW ACL Policy: Traffic can be blocked by creating acl policy using CLI. In this case, very little CPU will be used. This can be configured under VDOM level: 'show firewall acl'.
FW-Policy: Traffic can be blocked using a firewall policy but this uses high CPU.
- Sniffing the traffic using 'any' filter is the most efficient way to identify interesting traffic. Use a sniffer to find the most repeated traffic (command: 'diag sniffer packet any "not port 22" 4 10000 l') For example if most UDP packets are visible, then use the command 'diag sniffer packet any 'not port 22 and udp 4 10000 l' to identify what UDP port traffic is being hit. Then based on the output, it is necessary to modify the firewall policy or configuration. A policy can be created with matching traffic and offload it to NPU by disabling UTM.
- Use 'diag sys session list' with filters to find the policy_id and VDOM.
- diag sys session filter ...... (set filters first). diag sys session list | grep "misc" (to find the interested policies and vds).
- Use flow trace to find out why packets go to slow path:
diagnose debug enable
diag debug flow filter ... <----- Set filters accordingly.
diag debug flow trace start 10 <----- Capture flow trace.
Find the reasons for packets being processed by the slow path, and then modify the config accordingly.
EX: msg =
--> Reverse path check fails, drop.
--> Denied by policy 0 (did not match any firewall policy).
--> Send to IPS.
--> Send to the application layer (UTM).
- monitor soft_irq process (diag sys mpstat).
- Fragmentation packets also can cause High CPU utilization, as they are not offloaded most of the time. The best way to handle fragmentation packets is to offload them to NPU and not send them to IPS by enabling UTM
To improve performance with fragmentation traffic, change below load-balance settings. These changes will cause all the existing sessions to drop and new sessions start forming again. Due to this, these changes should be performed during your maintenance window:
config load-balance setting
dp-fragment-session : disable<----- Change it to enable.
dp-load-distribution-method: src-dst-ip-sport-dport<----- Do not change.
sw-load-distribution-method: src-dst-ip-sport-dport<----- Change to src-dst-ip.
|
