Troubleshooting Tip: FortiGate Azure VM Crashing Intermittently and stuck in boot loop

Description

This article describes an issue where a kernel panic occurs when the device receives a large UDP packet. This condition may cause the FortiGate VM to experience intermittent errors and, in some cases, become stuck in a continuous boot loop.

Scope

 FortiGate VM on Azure, specifically fgt-vm-arm64-gcp(image type), with versions prior to FortiOS v7.6.7 and FortiOS v8.0.0.

Solution

Trigger condition: Receiving a large UDP packet on the MANA driver causes the Kernel Panic

Console output:

System is starting...
Starting system maintenance...
Scanning /dev/nvme0n1p1... (100%)
Scanning /dev/nvme0n1p2... (100%)
Serial number is FGVMSLTM25011883
fgt-eus2-vmifw01 login: skbuff: skb_over_panic: text: ffffffff b636f3ea len:1522 put:1522 head: ffff95a4d54b9000 data: ffff95a4d54b9100
kernel BUG at net/core/skbuff.c:161!
invalid opcode: 0000 [1] SMP
CPU: 2 PID: 22 Comm: ksoftirqd/2 Tainted: P 4.19.13 1
Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS Hyper-V UEFI Release v4.1 07/25/2025
RIP: 0010: skb_panic+0x4d/0x4f
Code:
Code: 00 00 57 8b b8 30 01 00 00 57 8b b8 2c 01 00 00 57 ff b0 40 01 00 00 4c 8b 88 38 01 00 00 48 c7 c7 d0 3f 68 bb e8 26 53 85 ff <0f> 0b 55 48 89 e5 48 c7 c1 00 0a 50 bb 48 8b 55 08 e8 9d ff ff ff
RSP: 0018:ffff93a31fb43e38 EFLAGS: 00010246
RAX: 000000000000008b RBX: ffff93a30e14f000 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff93a31fb55518 RDI: ffff93a31fb55518
RBP: ffff93a31fb43e58 R08: 000000000000025b R09: 0000000000000004
R10: 0000000000000000 R11: 0000000000000000 R12: ffff93a2c1c00048
R13: ffff93a2c1c08848 R14: 0000000000000001 R15: ffff93a2c1c00000
FS: 0000000000000000(0000) GS:ffff93a31fb40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000564833102000 CR3: 00000006c7009003 CR4: 0000000000360ee0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400
Stack:
 <IRQ>
 ffff93a2c21cd100 0000000000000700 00000000000006ea ffffffffbb604fda
 ffff93a31fb43e68 ffffffffbacf09d2 ffff93a31fb43ee0 ffffffffbab7295a
 ffff93a2c21cd000 000000040600eb9f ffff93a2c9f2fa80 ffffae0300000001
 ffff93a2c1c0019c ffff93a2ec41b300 000000080ce3f100 4ad12d7a8d2a1d00
 ffff93a2c1c08848 000000000000012c ffff93a31fb43ef8 ffff93a31fb68900
 0000000000000001 ffff93a31fb43f48 ffffffffbacfea8d 000000010000fc21
 ffff93a31fb43ef8 ffff93a31fb43ef8 ffff93a31fb43f08 ffff93a31fb43f08
 4ad12d7a8d2a1d00 0000000000000008 0000000000000003 0000000000000003
 ffffffffbb805218 0000000000000100 ffff93a31fb43fa0 ffffffffbb2000d8
 00200042ba420c02 ffff93a3194bc900 000000010000fc20 000000040000000a
 ffffae0385c0bdd8 0000000000000025 ffff93a309905600 0000000000000025
 0000000000000000 ffff93a31fb43fb0 ffffffffba4f9be9 ffff93a31fb43fe8
 ffffffffbb0012ef 0000000000000000 0000000000000000 0000000000000000
 0000000000000000 0000000000000000 ffffae0385c0bdd9 ffffffffbb00090f
 ffffae0385c0bdd8
 </IRQ>
 0000000000000000 0000000000000000 0000000000000000 0000000000000005
 ffffae0385c0be90 ffff93a3194bc900 0000000000000000 ffff93a319338b00
 0000000000000000 0000000000027600 0000000000000005 ffff93a31fb595a0
 00000000001022ba 00000000000195a0 0000000000000000 ffffffffffffffda
 ffffffffbae7c338 0000000000000010 0000000000000246 ffffae0385c0be88
 0000000000000018 ffff93a3194bc900 ffffffffbae7c1e0 ffffae0385c0bea0
 ffffffffba422019 ffffae0385c0beb0 ffffffffbae7c3b6 ffffae0385c0bef8
 ffffffffba520425 ffffae0385c0bee8 4ad12d7a8d2a1d00 0000000000000087
 0000000000000000 0000000000000000 0000000000000000 0000000000000000
 ffffae0385c0bf20 ffffffffba52066c 4ad12d7a8d2a1d00 aebc9dfbc0de1400
 0000000000000005 ffffae0385c0bf48 ffffffffba437832 aebc9dfbc0de1400
 0b15a6de4a473700 0000000000000000 0000000000000000 ffffffffba4000d4
 0000000000000000 0000000000000000 0000000000000000 0000000000000000
 0000000000000000 0000000000000000 0000000000000000 0000000000000000
 0000000000000000 0000000000000000 0000000000000000 0000000000000000
 0000000000000000 0000000000000000 0000000000000000 0000000000000000
 0000000000000000 0000000000000000 0000000000000000 0000000000000000
 0000000000000000
Call Trace:
 <IRQ>
 skb_put.cold+0x14/0x14
 mana_poll+0x504/0x7e7
 net_rx_action+0xe3/0x26c
 __do_softirq+0xd8/0x245
 irq_exit+0x72/0x9b
 do_IRQ+0x7f/0xcd
 common_interrupt+0xf/0xf
 </IRQ>
RIP: 0010:native_safe_halt+0x2/0x3
Code: a8 08 74 0b 65 81 25 f3 89 19 45 ff ff ff 7f c3 65 48 8b 14 25 80 4d 01 00 f0 80 4a 02 20 48 8b 12 83 e2 08 74 8b eb c1 fb f4 <c3> f4 c3 90 90 90 90 90 55 48 89 e5 53 e8 3c a4 6d ff 65 8b 05 1f
RSP: 0018:ffffae0385c0be88 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffda
RAX: 0000000000000005 RBX: ffff93a3194bc900 RCX: ffff93a31fb595a0
RDX: 00000000001022ba RSI: 00000000000195a0 RDI: 0000000000000000
RBP: ffffae0385c0be90 R08: 0000000000027600 R09: 0000000000000000
R10: ffff93a319338b00 R11: 0000000000000000 R12: 0000000000000005
R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
 ? default_idle+0x10/0x1b
 arch_cpu_idle+0x10/0x12
 default_idle_call+0x1e/0x29
 do_idle+0x1ba/0x21a
 cpu_startup_entry+0x6b/0x6f
 start_secondary+0x194/0x1c5
 secondary_startup_64+0xa4/0xb0
Modules linked in: filter4(P)
---[ end trace 155d7967a196e5ce ]---
RIP: 0010:skb_panic+0x4d/0x4f
Code: 00 00 57 8b b8 30 01 00 00 57 8b b8 2c 01 00 00 57 ff b0 40 01 00 00 4c 8b 88 38 01 00 00 48 c7 c7 d0 3f 68 bb e8 26 53 85 ff <0f> 0b 55 48 89 e5 48 c7 c1 00 0a 50 bb 48 8b 55 08 e8 9d ff ff ff
RSP: 0018:ffff93a31fb43e38 EFLAGS: 00010246
RAX: 000000000000008b RBX: ffff93a30e14f000 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff93a31fb55518 RDI: ffff93a31fb55518
RBP: ffff93a31fb43e58 R08: 000000000000025b R09: 0000000000000004
R10: 0000000000000000 R11: 0000000000000000 R12: ffff93a2c1c00048
R13: ffff93a2c1c08848 R14: 0000000000000001 R15: ffff93a2c1c00000
FS: 0000000000000000(0000) GS:ffff93a31fb40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000564833102000 CR3: 00000006c7009003 CR4: 0000000000360ee0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400
Kernel panic - not syncing: Fatal exception in interrupt
Kernel Offset: 0x3a200000 from 0xffffffff80200000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---

Workaround involves deploying a FortiGate VM using an MLX driver to avoid this issue.

This issue is tracked as ID 1274229 and resolved in FortiOS v7.6.7 and v8.0.0.