High CPU voipd

johnlloyd13 · ‎09-01-2025

hi,

i just moved a new VDOM to a fortigate and observing CPU and other stats.

i noticed the 'global' VDOM spikes to above 90% over 10 min period.

but it looks 'normal' if it's over 24 hour period.

tried to run some CLI commands as below.

can someone advise if what's the "nice" 90% plus meaning here?

i also observe "voipd" with a number "95.2" beside it. what does this also mean? is this a voip DOS attack?

what other CLI commands can i use to trace other high CPU talker?

do i need to go each VDOM and check CPU?

(global) # get system performance status

CPU states: 3% user 3% system 0% nice 92% idle 0% iowait 0% irq 2% softirq

CPU0 states: 0% user 0% system 0% nice 97% idle 0% iowait 0% irq 3% softirq

CPU1 states: 8% user 7% system 0% nice 82% idle 0% iowait 1% irq 2% softirq

CPU2 states: 40% user 39% system 0% nice 18% idle 0% iowait 1% irq 2% softirq

CPU3 states: 0% user 0% system 0% nice 97% idle 0% iowait 0% irq 3% softirq

CPU4 states: 0% user 0% system 0% nice 96% idle 0% iowait 1% irq 3% softirq

CPU5 states: 0% user 0% system 0% nice 99% idle 0% iowait 0% irq 1% softirq

CPU6 states: 0% user 0% system 0% nice 99% idle 0% iowait 0% irq 1% softirq

CPU7 states: 3% user 0% system 0% nice 93% idle 0% iowait 1% irq 3% softirq

CPU8 states: 0% user 0% system 0% nice 99% idle 0% iowait 0% irq 1% softirq

CPU9 states: 0% user 0% system 0% nice 99% idle 0% iowait 0% irq 1% softirq

CPU10 states: 2% user 0% system 0% nice 97% idle 0% iowait 0% irq 1% softirq

CPU11 states: 0% user 0% system 0% nice 98% idle 0% iowait 0% irq 2% softirq

CPU12 states: 0% user 0% system 0% nice 97% idle 0% iowait 0% irq 3% softirq

CPU13 states: 0% user 0% system 0% nice 97% idle 0% iowait 0% irq 3% softirq

CPU14 states: 0% user 0% system 0% nice 98% idle 0% iowait 0% irq 2% softirq

CPU15 states: 1% user 1% system 0% nice 96% idle 0% iowait 0% irq 2% softirq

Memory: 24720020k total, 8184684k used (33.1%), 15810920k free (64.0%), 724416k freeable (2.9%)

Average network usage: 677027 / 679541 kbps in 1 minute, 708103 / 710748 kbps in 10 minutes, 697432 / 700174 kbps in 30 minutes

Maximal network usage: 841017 / 843685 kbps in 1 minute, 880259 / 882819 kbps in 10 minutes, 890608 / 893310 kbps in 30 minutes

Average sessions: 144121 sessions in 1 minute, 142122 sessions in 10 minutes, 144300 sessions in 30 minutes

Maximal sessions: 145129 sessions in 1 minute, 147030 sessions in 10 minutes, 164704 sessions in 30 minutes

Average session setup rate: 1123 sessions per second in last 1 minute, 1147 sessions per second in last 10 minutes, 1182 sessions per second in last 30 minutes

Maximal session setup rate: 1250 sessions per second in last 1 minute, 1430 sessions per second in last 10 minutes, 2813 sessions per second in last 30 minutes

Average NPU sessions: 95363 sessions in last 1 minute, 95996 sessions in last 10 minutes, 97069 sessions in last 30 minutes

Maximal NPU sessions: 95918 sessions in last 1 minute, 98758 sessions in last 10 minutes, 108089 sessions in last 30 minutes

Average nTurbo sessions: 0 sessions in last 1 minute, 0 sessions in last 10 minutes, 0 sessions in last 30 minutes

Maximal nTurbo sessions: 0 sessions in last 1 minute, 0 sessions in last 10 minutes, 0 sessions in last 30 minutes

Virus caught: 0 total in 1 minute

IPS attacks blocked: 0 total in 1 minute

Uptime: 292 days, 5 hours, 17 minutes

diagnose sys top

Run Time: 292 days, 5 hours and 19 minutes

3U, 0N, 2S, 93I, 0WA, 0HI, 2SI, 0ST; 24140T, 15424F

voipd 30566 R 95.2 0.2 5

bcm.user 2253 S < 7.1 0.5 10

cid 3970 S 3.5 1.4 3

miglogd 4166 S 0.5 0.6 6

miglogd 4163 S 0.3 0.6 9

miglogd 4159 R 0.3 0.6 13

miglogd 4165 S 0.3 0.6 7

miglogd 4160 S 0.3 0.6 12

node 3900 S 0.3 0.5 14

httpsd 31424 S 0.3 0.1 10

sessionsync 4700 S 0.3 0.0 2

miglogd 4161 R 0.1 0.6 11

miglogd 4164 S 0.1 0.6 8

miglogd 4162 S 0.1 0.6 10

forticron 3886 S 0.1 0.5 10

hasync 3906 S < 0.1 0.3 4

dnsproxy 3929 S 0.1 0.2 8

locallogd 3903 S 0.1 0.2 0

hatalk 3905 S < 0.1 0.1 12

wad_usrinfohist 3948 S 0.1 0.0 7

BillH_FTNT · ‎09-01-2025

Hi @johnlloyd13

Does your device handle a lot of SIP (voice) traffic? What is the current firmware version of your device?
Could you please share the configuration and the results of the following commands with me via my official email: bhoang@gmail.com?

# Collect system information

show full system settings | grep default
diagnose firewall iprope list 00004e20
diagnose sys top 2 99 | grep voip
diagnose sys process pstack 30566 #voipd process, execute multiple times

# capture sip logs

On window 1 (web session 1)
diagnose sniffer packet any 'port 5060' 6

On window 2
diagnose debug console timestamp enable
diagnose debug enable
diagnose debug flow filter port 5060
diagnose debug flow trace start 100000

On window 3
diagnose debug console timestamp enable
diagnose debug enable
diagnose debug applicat sip -1

Regards
Bill

johnlloyd13 · ‎09-02-2025

hi,

we have DNAT opened but "some" might have TCP/UDP 5060 as we don't limit our downstream user applications.

the firmware is 7.2

BillH_FTNT · ‎09-02-2025

Hi @johnlloyd13

Could you please share the output of commands in my previous comments ? and also configuration ? My official email is bhoang@fortinet.com

Regards

Bill

johnlloyd13 · ‎09-03-2025

hi,

appreciate your help but i can't provide proprietary info in a public forum due to data protection laws.

can you please advise if this command is "safe" to execute?

fnsysctl killall voipd

BillH_FTNT · ‎09-03-2025

Hi John,

You can share the logs/config files through the ticket. I can access the ticket to retrieve the materials needed to reproduce or analyze the issue. Thank you !