FortiGate
FortiGate Next Generation Firewall utilizes purpose-built security processors and threat intelligence security services from FortiGuard labs to deliver top-rated protection and high performance, including encrypted traffic.
cskuan
Staff
Staff
Article Id 194684
Description

This article describes how to troubleshoot CPU-related issues on where each daemon is running on which CPU core.

Scope

FortiOS 7.0 and later.

Solution

Use the following command to check which CPU core is used by each daemon.

The last column shows the CPU core index

 

diagnose sys top

 

 Run Time:  31 days, 18 hours and 12 minutes

0U, 0N, 0S, 100I, 0WA, 0HI, 0SI, 0ST; 2007T, 663F

         cmdbsvr      145      S       4.0     2.7    0

            node    28235      S       3.3     2.5    1

         updated      194      R       2.1     1.7    4

         reportd      190      S       6.0     1.5    3

          cw_acd      211      S       6.3     1.4    2

       forticron      180      S       2.7     1.3    3

         miglogd      189      R       1.1     1.3    1

           autod      215      S       5.8     1.2    4

 

From the above output, it is possible to conclude that the daemon updated process ID is 194 and it is running on CPU core 4.

 

The following command can also be used to check which CPU core is used by each daemon.

 

diagnose sys process dump 189 | grep Cpu   <----- 189 is the Process ID(PID) of a daemon/process
Cpus_allowed: 2                                               <----- 0010 (converted by binary).
Cpus_allowed_list: 1                                       <----- CPU no. 1 (2nd from the far right is '1').


If one core is observed (can be multiple) being used significantly more for a longer time, then, it can be helpful to run the CPU profiler.

 

get sys performance status
CPU states: 1% user 0% system 0% nice 99% idle 0% iowait 0% irq 35% softirq
CPU0 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU1 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU2 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU3 states: 5% user 0% system 0% nice 95% idle 0% iowait 0% irq 40% softirq

 

For example: CPU core 3 has the highest load in the system, softirq.

Now, proceed with CPU core profiling, which will give insight into what may be the problem (e.g. offloading).

 

diagnose sys profile cpumask <cpu core number>

diagnose sys profile start <- Run the command and leave it for 10-15 seconds.

diagnose sys profile stop <- After 10-15 seconds run it to stop the profiler.

diagnose sys profile show <- Will print the result on the console.

diagnose sys profile show order <- Shows in order.

 

Note:

If the process is in the S state (sleeping state), '# diagnose sys top' will display the last used CPU because the process is sleeping.

 

Identify the CPU core being used on older versions before FortiOS 7.0:

 

  1. Identify the Process ID (PID) of a daemon/process.

 

diagnose sys top
[JRun Time:  1 days, 23 hours and 54 minutes
0U, 0N, 0S, 100I, 0WA, 0HI, 0SI, 0ST; 7996T, 5392F
       ipshelper      330      S <     0.0     1.6
       ipsengine      333      S <     0.0     1.2
       ipsengine      331      S <     0.0     1.2
       ipsengine      332      S <     0.0     1.2
         reportd      200      S       0.0     1.1
        cmdbsvr       159      S       0.0     0.4
         pyfcgid      186      S       0.0     0.4
       scanunitd      203      S <     0.0     0.3
       scanunitd      239      S <     0.0     0.3
       scanunitd      238      S <     0.0     0.3
       scanunitd      236      S <     0.0     0.3
       scanunitd      237      S <     0.0     0.3
          httpsd      185      S       0.0     0.3
         pyfcgid      233      S       0.0     0.3
         pyfcgid      234      S       0.0     0.3
         pyfcgid      235      S       0.0     0.3
          httpsd      243      S       0.0     0.3
          httpsd      284      S       0.0     0.3
          cw_acd      218      S       0.0     0.3
       forticron      192      S       0.0     0.2

 

  1. After tracing the PID of a daemon, issue the following commands:

 

diagnose sys process dump 7091 | grep Cpus_allowed
Cpus_allowed:   f   <----- Hex Value.
Cpus_allowed_list:      0-3  <----- List of CPU cores.

Convert the HEX value to binary to find out the core using the following table:


Decimal            Binary        CPU used
1                 0001            cpu-0
2                 0010            cpu-1
3                 0011            cpu-0 and cpu-1
4                 0100            cpu-2
.......
15                 1111            cpu-0, cpu-1, cpu-2 and cpu-3
16                   10110            cpu-1, cpu-2, and cpu-4
.......

 

Use a HEX to Binary converter.