Technical Tip: Kernel panic in NP7 platforms after upgrading to v7.4.5

• The following errors are seen on the console after upgrading to v7.4.5 or above.
• It keeps scrolling with the following errors.
• It is related to a kernel panic.
• It affects NP7-based platforms with Traffic shaping enabled.
• Following is an example, but the kernel panic may vary. It is recommended to open a FortiCare Ticket to verify the issue with TAC.

<IRQ>
np7_sys_qtm_qqid_sch0_free+0x87/0x120 [filter4]
np7_sys_qtm_unset_per_ip_shaper+0x42/0x90 [filter4]
? log_session+0x747/0xd70 [filter4]
np7_fos_del_mcast_session+0x704/0xb10 [filter4]
ippool_ioctl+0x20b/0x1b00 [filter4]
piq_put+0x47/0x60 [filter4]
convert_k2u_ip_session_user+0xe9a/0x11f0 [filter4]
rcu_do_batch+0x9c/0x200
__rcu_process_callbacks+0xc2/0xe0
rcu_process_callbacks+0x36/0x60
__do_softirq+0xc0/0x20d
irq_exit+0x71/0x90
smp_apic_timer_interrupt+0x60/0xa0
apic_timer_interrupt+0xf/0x20
</IRQ>

System is rebooting...
rcu: INFO: rcu_sched self-detected stall on CPU
rcu: 14-....: (1 GPs behind) idle=9a2/0/0x3 softirq=1673789/1673791 fqs=98137
rcu: (t=258055 jiffies g=2485029 q=2355352)
Sending NMI from CPU 14 to CPUs 1:
NMI backtrace for cpu 1
CPU: 1 PID: 0 Comm: swapper/1 Tainted: P L 4.19.13 1
Hardware name: AMD Qogir/Qogir-VMR, BIOS 05.44.43.0005 11/03/2022
RIP: 0010:np7_tpe_init+0x3d1/0x530 [filter4]
Code: 4d 50 41 0f b7 06 48 8b 76 20 41 d3 e1 41 8d 49 ff 66 39 d0 75 0b e9 d5 00 00 00 66 45 39 c4 7...
RSP: 0018:ffff88882e843da0 EFLAGS: 00000202
RAX: 0000000000000016 RBX: 000000000000002c RCX: 00000000000007ff
RDX: ffff88880337802a RSI: ffff888803378000 RDI: 0000000000000000
RBP: ffff88882e843dd8 R08: 000000000000000d R09: 0000000000000800
R10: 000000000000229e R11: ffff8888041f5000 R12: 0000000000000016
R13: ffffffffa0c733e0 R14: ffffffffa0c73408 R15: 0000000000000016
FS: 0000000000000000(0000) GS:ffff88882e840000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fe6f4c6f000 CR3: 0000000001209000 CR4: 0000000000740ee0
PKRU: 55555554
Stack:
<IRQ>
ffff8885c95ee640 ffff88873a6e6c00 ffff8886f845a000 ffff88882e843e1e
ffffffffa0c733e0 0000000000000000 ffff88882e8679b8 ffff88882e843e00
ffffffffa01cc4a7 0000000000000000 ffff8886f845a000 000000000000000a
ffff88882e843e30 ffffffffa01ce64f ffffffffa001fc77 00360001001bfb6c
26602f4c4e895f00 ffff88874408e500 ffff88882e843e50 ffffffffa01e7514
ffff8886f845a000 ffff88874408e500 ffff88882e843e70 ffffffffa008b3fb
ffff8885c95ee728 ffff8886f845a000 ffff88882e843e88 ffffffffa008d017
ffff8885c95ee640 ffff88882e843eb0 ffffffffa000b39a ffff88882e867980
ffff88882e843ec8 ffff88880ccdaf40 ffff88882e843f18 ffffffff8034264b
ffffffff81248c00 0000000000000000 ffff88882e843ec8 ffffffffffffffe1
fffffffffffffff4 26602f4c4e895f00 ffffffff81248c00 ffff88882e867980
ffffffff81248c00 ffffffff81248e00 0000000000000009 ffff88882e843f48
ffffffff80343372 ffffffff81248c00 0000000000000009 0000000000000100
0000000000000002 ffff88882e843f60 ffffffff803433c6 ffffffff81205148
ffff88882e843fb8 ffffffff80c000c0 002000422e843fc8 ffff88880ccdaf40
0000000100224d57 000000020000000a 0000000000000000 ffff88882e854f40
0000000000000000 0000000000000000 0000000000000000 ffff88882e843fc8
ffffffff802f9201 ffff88882e843fe8 ffffffff80a01500 0000000000000000
0000000000000000 ffffc9000500be29 ffffffff80a0105f ffffc9000500be28
</IRQ>
0000000000000000 0000000000000000 0000000000000000 0000000000000001
ffffc9000500bee0 ffff88880ccdaf40 0000000000000001 0000000000000000
0000000100224d55 00000000000003df 0000000000000001 ffff88882e8594c0
000000003c6e5cf6 00000000000194c0 0000000000000000 ffffffffffffff13
ffffffff809ccb26 0000000000000010 0000000000000202 ffffc9000500bed8
0000000000000018 ffffffff809ccb0a ffff88880ccdaf40 ffffc9000500bf00
ffffffff8031d01e 0000000020000000 0000000000000087 ffffc9000500bf28
ffffffff8031d251 26602f4c4e895f00 0f6d03b424eec000 0000000000000000
ffffc9000500bf48 ffffffff8023d655 0f6d03b424eec000 024ef93ac5e31d00
0000000000000000 ffffffff802000d4 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000
Call Trace:
<IRQ>
np7_sys_qtm_qqid_sch0_free+0x87/0x120 [filter4]
np7_sys_qtm_unset_per_ip_shaper+0x4f/0xb0 [filter4]
? log_session+0x767/0xd80 [filter4]
np7_fos_del_mcast_session+0x774/0xba0 [filter4]
ippool_ioctl+0x20b/0x1c50 [filter4]
piq_put+0x47/0x60 [filter4]
convert_k2u_ip_session_user+0xe9a/0x11f0 [filter4]
rcu_do_batch+0xab/0x220
__rcu_process_callbacks+0xc2/0xe0
rcu_process_callbacks+0x36/0x60
__do_softirq+0xc0/0x20d
irq_exit+0x71/0x90
smp_apic_timer_interrupt+0x60/0xa0
apic_timer_interrupt+0xf/0x20
</IRQ>

Workaround:

Switch the QoS mode from shaping to policy (a reboot is required):

config system npu
    set default-qos-type policing
end

It will be fixed in v7.6.3 firmware or above.

Related documents: 

NP7 traffic shaping