The 'diagnose sys profile report' command is a hidden command that may be highly useful to troubleshoot high CPU issues, mainly during events where the high CPU is due to 'softirq'. For more information about 'softirq', see Troubleshooting Tip: Check SoftIrq increments (recommended when experiencing high CPU usage).
As mentioned above, the 'report' parameter is hidden:
diagnose sys profile start start kernel profiling data stop copy kernel profiling data show show kernel profiling result sysmap show kernel sysmap cpumask profile which CPUs [Take 0-10 arg(s)] step set profile step module show kernel module
The command will display the following outputs if the CPU usage is low:
get system performance status CPU0 states: 1% user 0% system 0% nice 99% idle 0% iowait 0% irq 0% softirq
diagnose sys profile report CPU Kernel Percentages: 0: 1% (1 of 100). Not profiling. No busy CPUs found.
If the FortiGate has more CPUs, a line will be displayed for each one:
diag sys profile report CPU Kernel Percentages: 0: 44% (44 of 100). Not profiling. 1: 37% (37 of 100). Not profiling. 2: 45% (45 of 100). Not profiling. 3: 29% (29 of 100). Not profiling. 4: 42% (42 of 100). Not profiling. 5: 39% (39 of 99). Not profiling. 6: 43% (43 of 100). Not profiling. 7: 35% (35 of 100). Not profiling. 8: 28% (28 of 100). Not profiling.
No busy CPUs found.
The profiling will start if any CPU goes above 50%, and the kernel outputs will be displayed. More than one CPU can be profiled simultaneously.
diag sys profile report CPU Kernel Percentages: 0: 27% (28 of 101). Not profiling. 1: 25% (25 of 99). Not profiling. 2: 52% (52 of 100). Will profile. 3: 28% (29 of 101). Not profiling. 4: 36% (36 of 100). Not profiling. 5: 37% (37 of 99). Not profiling. 6: 28% (28 of 100). Not profiling. 7: 26% (27 of 102). Not profiling. 8: 20% (21 of 101). Not profiling. Kernel=0xffffffff80200000-0xffffffff80c021d1, module-gap=0x1f3fde2f Profile buffer: profile step=4, sz=15741624-7870812, last-addr=0xffffffffa140439f warning: functions cannot be profiled properly, __entry_SYSCALL_64_trampoline-pGDT32 warning: functions cannot be profiled properly, early_idt_handler_array-br_deinit diagnose vpn tunnel stat 0xffffffff809e0624: 54 cpu_idle_poll.isra.0+0x24/0x60 0xffffffffa013e78f: 13 np7_hpe_exit+0x2cf/0x340 0xffffffffa000bde7: 12 ip_session_ha_scan_walker+0x657/0x8a0 0xffffffffa0016fbb: 8 ip_session_tree_walker+0x1e8b/0x2760 0xffffffffa0140ddf: 3 np7_hw_dsw_init+0x3af/0x520 0xffffffff802f8764: 3 __local_bh_enable_ip+0x34/0x60 0xffffffffa0140fbb: 2 np7_hhtbl_free+0x6b/0x300 0xffffffffa00e9bdb: 2 ip6_session_tree_walker+0x42b/0x2280 0xffffffff80876c60: 1 neigh_resolve_output.part.0+0x0/0x100 0xffffffff809e000c: 1 schedule_hrtimeout_range_clock+0x9c/0x130 0xffffffffa003d157: 1 detect_unknown_spi6+0x397/0x7f0
These outputs will be useful for the support and development team to understand the reason of the CPU usage.
|