High Memory Issue. FortiGate 1500D

[Tushara_Manoj]

Hi,

My 1500D fortiGate deceive goes  conserve mode due to high memory. It takes more that 85% of memory some times. IPS Engine take more memory. Do you have any experience on this regards?

version: v5.4.7,build1167  

Thank You!

[Fullmoon]

kindly check with FortiOS 5.6.x release notes if they addressed regarding memory issue.

[Tushara_Manoj]

Hi,

OK thanks ...

and I have another fortigate 1500D with 5.6.x. 

it is also going memory high. 80% but it didn't go to conserve mode 

Regards,

Manoj

[Fullmoon]

kindly run these command and share here the info

1. get system performance status ....(run it approximately 10 times to have representative samples) 2) diagnose sys top 2 50 ...type SHIFT+M to sort by memory (run it approximately 10 times to have representative samples) 3) diag sys top-summary (get 10 samples)

[Tushara_Manoj]

I already run those commands ....

Found some issues in IPS engine.

Followings are the current command output for 1500D 5.4.7 box.

get system performance status CPU states: 29% user 14% system 0% nice 53% idle 0% iowait 0% irq 4% softirq CPU0 states: 11% user 6% system 0% nice 77% idle 0% iowait 0% irq 6% softirq CPU1 states: 58% user 39% system 0% nice 2% idle 0% iowait 0% irq 1% softirq CPU2 states: 16% user 10% system 0% nice 65% idle 0% iowait 0% irq 9% softirq CPU3 states: 26% user 7% system 0% nice 67% idle 0% iowait 0% irq 0% softirq CPU4 states: 19% user 10% system 0% nice 63% idle 0% iowait 0% irq 8% softirq CPU5 states: 39% user 16% system 0% nice 45% idle 0% iowait 0% irq 0% softirq CPU6 states: 15% user 16% system 0% nice 60% idle 0% iowait 0% irq 9% softirq CPU7 states: 44% user 8% system 0% nice 48% idle 0% iowait 0% irq 0% softirq CPU8 states: 14% user 13% system 0% nice 64% idle 0% iowait 0% irq 9% softirq CPU9 states: 54% user 21% system 0% nice 25% idle 0% iowait 0% irq 0% softirq CPU10 states: 9% user 9% system 0% nice 73% idle 0% iowait 0% irq 9% softirq CPU11 states: 43% user 13% system 0% nice 44% idle 0% iowait 0% irq 0% softirq Memory states: 75% used Average network usage: 265544 / 293014 kbps in 1 minute, 265315 / 293892 kbps in 10 minutes, 271170 / 302609 kbps in 30 minutes Average sessions: 60675 sessions in 1 minute, 61488 sessions in 10 minutes, 6522 3 sessions in 30 minutes Average session setup rate: 2907 sessions per second in last 1 minute, 3092 sess ions per second in last 10 minutes, 3225 sessions per second in last 30 minutes Virus caught: 0 total in 1 minute IPS attacks blocked: 0 total in 1 minute Uptime: 67 days, 5 hours, 38 minutes

diagnose sys top 2 25

Run Time: 67 days, 5 hours and 40 minutes 25U, 0N, 11S, 59I, 0WA, 0HI, 5SI, 0ST; 16064T, 3939F wad 10270 S 13.3 2.0 wad 10266 S 5.9 2.0 wad 13897 S 7.9 1.9 wad 10269 S 5.4 1.7 wad 10268 S 3.9 1.6 wad 5438 S 7.9 1.6 ipsengine 15447 S < 5.9 1.2 ipsengine 15448 S < 6.4 1.2 ipsengine 15442 S < 7.4 1.2 ipsengine 15443 S < 7.9 1.2 ipsengine 15445 R < 6.9 1.2 ipsengine 15449 S < 5.9 1.2 ipsengine 15451 S < 5.9 1.2 ipsengine 15446 S < 6.4 1.2 ipsengine 15450 S < 6.9 1.2 ipsengine 15444 S < 7.9 1.2 miglogd 10189 S 31.6 0.9 miglogd 10300 S 2.4 0.6 miglogd 10298 S 2.9 0.6 miglogd 10299 S 2.4 0.6 miglogd 10301 S 2.4 0.6 miglogd 10302 S 2.4 0.6 cmdbsvr 4370 S 0.0 0.5 dnsproxy 10230 S 0.9 0.5 updated 10444 S 1.4 0.5

Top Summery:

diag sys top-summary CPU [|||||||||||||||||| ] 45.6% Mem [|||||||||||||||||||||||||||||| ] 75.0% 12153M/16064M Processes: 20 (running=1 sleeping=239)

PID RSS ^CPU% MEM% FDS TIME+ NAME * 28668 8M 0.0 0.1 8 00:00.00 getty 4370 96M 0.0 0.6 12 10:42.25 cmdbsvr 5779 136M 0.0 0.9 160 33:00.34 authd [x5] 4379 15M 0.0 0.1 87 08:09.68 zebos_launcher [x12] 15437 1G 0.0 12.7 888 43:40.88 ipsmonitor [x12] 24636 150M 0.0 0.9 87 29:33.61 sslvpnd [x12] 10444 86M 0.0 0.5 17 58:00.75 updated 10188 9M 0.0 0.1 12 00:32.12 uploadd 10189 156M 0.0 1.0 55 40:58.39 miglogd [x6] 10190 7M 0.0 0.0 11 39:12.28 ipmc_sensord 10191 7M 0.0 0.0 8 00:00.30 kmiglogd 10192 130M 0.0 0.8 20 04:46.96 httpsd [x5] 10196 8M 0.0 0.1 11 44:54.44 merged_daemons 10197 15M 0.0 0.1 14 54:30.13 fnbamd 10198 7M 0.0 0.0 12 00:52.43 fclicense 10199 43M 0.0 0.3 21 24:28.50 forticron [x2] 10200 12M 0.0 0.1 15 00:08.18 forticldd 28132 42M 0.0 0.3 15 01:18.47 pyfcgid [x7] 10203 15M 0.0 0.1 26 00:56.72 foauthd 10204 7M 0.0 0.0 10 00:00.60 httpclid CPU [||||||||||||||||| ] 43.1% Mem [|||||||||||||||||||||||||||||| ] 75.0% 12157M/16064M Processes: 20 (running=2 sleeping=239)

PID RSS ^CPU% MEM% FDS TIME+ NAME * 15437 1G 66.9 12.7 888 43:41.69 ipsmonitor [x12] 5779 136M 45.8 0.9 172 33:00.89 authd [x5] 10189 156M 39.2 1.0 55 40:58.86 miglogd [x6] 10212 1G 32.5 10.2 23896 45:17.18 wad [x10] 10225 41M 15.6 0.3 17 00:01.43 sshd [x6] 10213 81M 4.9 0.5 218 38:34.34 scanunitd [x12] 10444 86M 1.7 0.5 17 58:00.77 updated 10222 56M 1.7 0.3 19 12:19.80 urlfilter 24636 150M 0.8 0.9 87 29:33.62 sslvpnd [x12] 10192 130M 0.8 0.8 21 04:46.97 httpsd [x5] 10230 95M 0.8 0.6 30 40:14.60 dnsproxy 10232 7M 0.8 0.0 14 42:40.72 sessionsync 28668 8M 0.0 0.1 8 00:00.00 getty 4370 96M 0.0 0.6 12 10:42.25 cmdbsvr 4379 15M 0.0 0.1 87 08:09.68 zebos_launcher [x12] 10188 9M 0.0 0.1 12 00:32.12 uploadd 10190 7M 0.0 0.0 11 39:12.28 ipmc_sensord 10191 7M 0.0 0.0 8 00:00.30 kmiglogd 10196 8M 0.0 0.1 11 44:54.44 merged_daemons 10197 15M 0.0 0.1 14 54:30.13 fnbamd CPU [||||||||||||||||| ] 42.8% Mem [|||||||||||||||||||||||||||||| ] 75.0% 12161M/16064M Processes: 20 (running=3 sleeping=238)

PID RSS ^CPU% MEM% FDS TIME+ NAME * 15437 1G 68.2 12.7 888 43:42.52 ipsmonitor [x12] 10189 156M 42.1 1.0 55 40:59.38 miglogd [x6] 5779 136M 38.2 0.9 171 33:01.36 authd [x5] 10212 1G 34.9 10.2 24013 45:17.61 wad [x10] 10225 41M 16.1 0.3 17 00:01.63 sshd [x6] 10213 86M 12.9 0.5 220 38:34.50 scanunitd [x12] 10192 130M 10.6 0.8 21 04:47.10 httpsd [x5] 10222 56M 1.6 0.3 19 12:19.82 urlfilter 24636 150M 0.8 0.9 87 29:33.63 sslvpnd [x12] 10444 86M 0.8 0.5 17 58:00.78 updated 10217 11M 0.8 0.1 13 27:13.60 hatalk 10218 43M 0.8 0.3 30 52:17.33 hasync 10230 95M 0.8 0.6 30 40:14.61 dnsproxy 10232 7M 0.8 0.0 14 42:40.73 sessionsync 28668 8M 0.0 0.1 8 00:00.00 getty 4370 96M 0.0 0.6 12 10:42.25 cmdbsvr 4379 15M 0.0 0.1 87 08:09.68 zebos_launcher [x12] 10188 9M 0.0 0.1 12 00:32.12 uploadd 10190 7M 0.0 0.0 11 39:12.28 ipmc_sensord 10191 7M 0.0 0.0 8 00:00.30 kmiglogd

[Tushara_Manoj]

Hi,

still my problem could not resolved.. can you help on this regards.

still memory is 69%

thanks ...

Regards,

Manoj

[Tushara_Manoj]

Hi, can anyone help me on this regards pls.

latests output...

get system performance status CPU states: 26% user 11% system 0% nice 57% idle 0% iowait 0% irq 6% softirq CPU0 states: 13% user 4% system 0% nice 73% idle 0% iowait 0% irq 10% softirq CPU1 states: 21% user 7% system 0% nice 72% idle 0% iowait 0% irq 0% softirq CPU2 states: 13% user 12% system 0% nice 64% idle 0% iowait 0% irq 11% softirq CPU3 states: 19% user 8% system 0% nice 72% idle 0% iowait 0% irq 1% softirq CPU4 states: 15% user 10% system 0% nice 62% idle 0% iowait 0% irq 13% softirq CPU5 states: 40% user 12% system 0% nice 48% idle 0% iowait 0% irq 0% softirq CPU6 states: 14% user 10% system 0% nice 62% idle 0% iowait 0% irq 14% softirq CPU7 states: 47% user 26% system 0% nice 27% idle 0% iowait 0% irq 0% softirq CPU8 states: 15% user 6% system 0% nice 68% idle 0% iowait 0% irq 11% softirq CPU9 states: 47% user 13% system 0% nice 40% idle 0% iowait 0% irq 0% softirq CPU10 states: 12% user 7% system 0% nice 73% idle 0% iowait 0% irq 8% softirq CPU11 states: 53% user 15% system 0% nice 32% idle 0% iowait 0% irq 0% softirq Memory states: 72% used Average network usage: 332628 / 366895 kbps in 1 minute, 343516 / 377598 kbps in 10 minutes, 348577 / 380470 kbps in 30 minutes Average sessions: 74843 sessions in 1 minute, 76500 sessions in 10 minutes, 73170 sessions in 30 minutes Average session setup rate: 3413 sessions per second in last 1 minute, 3458 sessions per second in last 10 minutes, 3328 sessions per second in last 30 minutes Virus caught: 0 total in 1 minute IPS attacks blocked: 0 total in 1 minute Uptime: 1 days, 19 hours, 33 minutes

[mike_dp]

We're having the same issue on 6.0.4. Memory goes from 40% to 85% in 2 weeks and most of it is WAD process because of SD-WAN. Earlier release notes said it was fixed but it's not.