FortiGate
FortiGate Next Generation Firewall utilizes purpose-built security processors and threat intelligence security services from FortiGuard labs to deliver top-rated protection and high performance, including encrypted traffic.
pkumari
Staff
Staff
Article Id 366726
Description This article describes how to troubleshoot high CPU issues due to 'irq' going high.
Scope FortiGate.
Solution

'IRQ' value can be seen as high sometimes in the 'get system performance status' result.

A higher 'IRQ' value indicates an issue with the console output getting printed in the background or if collecting any debug, the chances are that the command output is getting printed in multiple consoles at the same time.

 

get sys perf status
CPU states: 0% user 1% system 0% nice 87% idle 0% iowait 12% irq 0% softirq
CPU0 states: 1% user 0% system 0% nice 99% idle 0% iowait 0% irq 0% softirq
CPU1 states: 0% user 1% system 0% nice 99% idle 0% iowait 0% irq 0% softirq
CPU2 states: 0% user 3% system 0% nice 97% idle 0% iowait 0% irq 0% softirq
CPU3 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU4 states: 0% user 1% system 0% nice 96% idle 0% iowait 0% irq 3% softirq
CPU5 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU6 states: 0% user 2% system 0% nice 98% idle 0% iowait 0% irq 0% softirq
CPU7 states: 0% user 0% system 0% nice 0% idle 0% iowait 100% irq 0% softirq

 

In the 'get system performance status' command output, IRQ stands for Interrupt Requests. IRQ values in the output indicate the percentage of time the CPU is spending on handling hardware interrupts. These interrupts are signals sent by hardware devices to the CPU to request attention.

 

Here are the key points regarding IRQ in the 'get system performance status' output:

  1. IRQ allows devices to notify the kernel that they need attention.
  2. Hardware Interrupts directly affect the CPU.
  3. Software Interrupts can only be generated by the process currently running.
  4. Devices with high bandwidth may distribute interrupts to multiple cores using multiple IRQs (NP6 or NP4). To view interrupts for hardware,  use the 'diag hard sys interrupts' command. For software interrupts, use the 'fnsysctl cat /proc/softirqs' command.

 

  • Collect 'dia sys mpstat' to see the pattern of the 'irq' value.
  • Collect 'diag debug info' to confirm if there are other debugs running in the background.

 

exe time
diag deb info
exe time
get sys perf
exe time

Example:

 

graviton-kvm63 (global) # dia debug info
debug output: disable
console timestamp: disable
console no user log message: disable
zebos debug level: 306783954 (0x124926d2)
CLI debug level: 3
WAD console log: enable

===== IPS debug settings =====
IPS Engine <00235> Enabled debug actions:

===== IPS SSL debug settings =====
IPS Engine <00235> ssl debug level is "none"

===== IP router debug settings =====
BGP debugging status:
BGP debug level: ERROR
timestamp disabled

;;

 

;;

RIPng debugging status:
RIPng debugging level is CRITICAL
timestamp disabled

 

The current debug duration is 30 minutes, the timer is not started yet.

  • Use the below command to disable debug output to the console.

 

FGT-A # config global
FGT-A (global) # diag debug console no-user-log-msg enable
FGT-A (global) # end

 

Logs to be collected.

 

Putty Session 1:

 

get sys status
get sys performance status
diagnose hardware sysinfo memory
diagnose sys vd stats
diagnose sys top-mem 20
diagnose sys top-fd 50
diagnose sys top 1 99 5
diagnose sys mpstat 2 5
fnsysctl ps aux
diagnose sys process dump <PID> <-----
Replace the PID with the highest IPS Engine using a high CPU.
diagnose sys process pstack <PID>
<----- Replace the PID with the highest IPS Engine using a high CPU.
diagnose ips session status

diagnose sys profile report
diagnose debug report 

 

Putty Session 2: 

 

get sys stat
get sys perf stat
diagnose sys top 5 20 99999

 

Putty Session 3:

 

get sys stat
get sys perf stat
diagnose sys mpstat 5 9999