[311B] CPU@100% each day, 7:00-7:05AM

[snobs]

Since I´ve updated my 311B firewalls to 5.2.3, each day between 7:00 and 7:00am the CPU reaches 100% for some minutes

event log tells me:

146 07:15:49 Performance statistics: average CPU: 4, memory: 39, concurrent sessions: 14905, setup-rate: 96
147 07:10:49 Performance statistics: average CPU: 5, memory: 39, concurrent sessions: 9820, setup-rate: 0
148 07:05:49 Performance statistics: average CPU: 100, memory: 39, concurrent sessions: 11001, setup-rate: 67
149 07:00:48 Performance statistics: average CPU: 100, memory: 38, concurrent sessions: 9326, setup-rate: 85
150 06:55:49 Performance statistics: average CPU: 6, memory: 38, concurrent sessions: 8392, setup-rate: 82

Within that time frame, I see problems connecting to services behind that 311B firewall cluster

I´d like to know what process is the reason for that behaviour. What can I do to monitor the system?

(disk logging is disabled)

Best Regards,

Michael

[Istvan_Takacs_FTNT]

You can run

diag sys top-summary

when the CPU usage is high. That would tell you which process is hogging the CPU.

[snobs]

Well, the process "newcli" influences "ipsengine"? There´s also "miglogd". So what is happening each day?

07:02:51 Performance statistics, average CPU 100,memory 39%, concurrent sessions 13860, setup-rate 83

Run Time: 104 days, 21 hours and 16 minutes
4U, 0N, 95S, 1I; 1004T, 609F, 172KF
       ipsengine 20092 R < 56.9 8.2
          newcli 31895 S 36.5 0.8
          newcli 31695 R 10.5 0.8
            sshd 31691 S 4.8 0.3
       forticron 84 S 1.6 0.2
          sflowd 113 S 1.6 0.2
         miglogd 70 S 0.8 1.4
         telnetd 31894 S 0.8 0.2
          httpsd 157 S 0.0 3.2
          httpsd 125 S 0.0 2.3
         cmdbsvr 47 S 0.0 2.2
       ipshelper 20091 S < 0.0 1.4
         pyfcgid 15799 S 0.0 1.2
         pyfcgid 15801 S 0.0 1.1
         pyfcgid 15802 S 0.0 1.1
         pyfcgid 15803 S 0.0 1.1
         updated 8731 S 0.0 0.9
            pimd 13875 S 0.0 0.8
          newcli 31692 S 0.0 0.8
          httpsd 124 S 0.0 0.8

 

Run Time: 104 days, 21 hours and 16 minutes
1U, 0N, 98S, 1I; 1004T, 609F, 172KF
       ipsengine 20092 R < 52.1 8.2
          newcli 31695 R 15.4 0.8
         miglogd 70 R 6.3 1.4
            sshd 31691 S 4.7 0.3
          newcli 31895 S 4.2 0.8
           snmpd 93 S 3.7 0.3
          hatalk 94 S < 2.6 0.2
        dnsproxy 112 S 1.5 0.3
         telnetd 31894 S 1.5 0.2
 initXXXXXXXXXXX 1 S 0.5 0.3
          sflowd 113 S 0.5 0.2
          httpsd 157 S 0.0 3.2
          httpsd 125 S 0.0 2.3
         cmdbsvr 47 S 0.0 2.2
       ipshelper 20091 S < 0.0 1.4
         pyfcgid 15799 S 0.0 1.2
         pyfcgid 15801 S 0.0 1.1
         pyfcgid 15802 S 0.0 1.1
         pyfcgid 15803 S 0.0 1.1
         updated 8731 S 0.0 0.9

 

Run Time: 104 days, 21 hours and 15 minutes
2U, 0N, 97S, 1I; 1004T, 608F, 172KF
       ipsengine 20092 S < 75.5 8.2
          newcli 31695 R 15.2 0.8
            sshd 31691 S 5.3 0.3
         miglogd 70 S 3.0 1.4
       forticron 84 S 1.5 0.2
        dnsproxy 112 S 0.7 0.3
          hatalk 94 S < 0.7 0.2
             nsm 13868 S 0.7 0.1
          httpsd 157 S 0.0 3.2
          httpsd 125 S 0.0 2.3
         cmdbsvr 47 S 0.0 2.2
       ipshelper 20091 S < 0.0 1.4
         pyfcgid 15799 S 0.0 1.2
         pyfcgid 15801 S 0.0 1.1
         pyfcgid 15802 S 0.0 1.1
         pyfcgid 15803 S 0.0 1.1
         updated 8731 S 0.0 0.9
            pimd 13875 S 0.0 0.8
          newcli 31692 S 0.0 0.8
          newcli 31889 S 0.0 0.8

 

Run Time: 104 days, 21 hours and 15 minutes
0U, 0N, 99S, 1I; 1004T, 608F, 172KF
       ipsengine 20092 R < 38.0 8.2
          newcli 31889 S 34.2 0.8
          newcli 31695 R 16.8 0.8
            sshd 31691 S 3.8 0.3
         miglogd 70 S 2.7 1.4
         telnetd 31888 S 1.0 0.2
       ipshelper 20091 S < 0.5 1.4
 initXXXXXXXXXXX 1 S 0.5 0.3
       forticron 84 S 0.5 0.2
          httpsd 157 S 0.0 3.2
          httpsd 125 S 0.0 2.3
         cmdbsvr 47 S 0.0 2.2
         pyfcgid 15799 S 0.0 1.2
         pyfcgid 15801 S 0.0 1.1
         pyfcgid 15802 S 0.0 1.1
         pyfcgid 15803 S 0.0 1.1
         updated 8731 S 0.0 0.9
            pimd 13875 S 0.0 0.8
          newcli 31692 S 0.0 0.8
          httpsd 124 S 0.0 0.8

Run Time: 104 days, 21 hours and 14 minutes
7U, 0N, 78S, 15I; 1004T, 608F, 172KF
       ipsengine 20092 R < 57.0 8.2
          newcli 31887 S 30.8 0.8
          fcnacd 87 S 7.4 0.2
            sshd 31691 S 4.6 0.3
         miglogd 70 R 2.8 1.4
          newcli 31695 R 1.8 0.8
          sflowd 113 R 0.9 0.2
          httpsd 157 S 0.0 3.2
          httpsd 125 S 0.0 2.3
         cmdbsvr 47 S 0.0 2.2
       ipshelper 20091 S < 0.0 1.4
         pyfcgid 15799 S 0.0 1.2
         pyfcgid 15801 S 0.0 1.1
         pyfcgid 15802 S 0.0 1.1
         pyfcgid 15803 S 0.0 1.1
         updated 8731 S 0.0 0.9
            pimd 13875 S 0.0 0.8
          newcli 31692 S 0.0 0.8
          httpsd 124 S 0.0 0.8
          httpsd 73 S 0.0 0.6

 

Run Time: 104 days, 21 hours and 11 minutes
0U, 0N, 100S, 0I; 1004T, 610F, 172KF
       ipsengine 20092 R < 27.2 8.1
         miglogd 70 R 20.7 1.4
          newcli 31695 R 17.8 0.8
           snmpd 93 R 15.2 0.3
            sshd 31691 S 3.5 0.3
          hatalk 94 S < 2.5 0.2
        dnsproxy 112 S 1.2 0.3
       forticron 84 S 1.2 0.2
          sflowd 113 S 0.6 0.2
          ipldbd 12538 S 0.6 0.1
 initXXXXXXXXXXX 1 S 0.3 0.3
          httpsd 157 S 0.0 3.2
          httpsd 125 S 0.0 2.3
         cmdbsvr 47 S 0.0 2.2
       ipshelper 20091 S < 0.0 1.4
         pyfcgid 15799 S 0.0 1.2
         pyfcgid 15801 S 0.0 1.1
         pyfcgid 15802 S 0.0 1.1
         pyfcgid 15803 S 0.0 1.1
         updated 8731 S 0.0 0.9
Run Time: 104 days, 21 hours and 11 minutes
1U, 0N, 98S, 1I; 1004T, 610F, 172KF
       ipsengine 20092 R < 64.6 8.1
         miglogd 70 S 32.0 1.4
          newcli 31695 R 14.6 0.8
          hatalk 94 S < 7.0 0.2
            sshd 31691 S 5.9 0.3
           snmpd 93 S 5.9 0.3
       forticron 84 S 1.6 0.2
        dnsproxy 112 S 1.0 0.3
          hasync 99 S < 0.5 0.5
 initXXXXXXXXXXX 1 S 0.5 0.3
          sflowd 113 S 0.5 0.2
          httpsd 157 S 0.0 3.2
          httpsd 125 S 0.0 2.3
         cmdbsvr 47 S 0.0 2.2
       ipshelper 20091 S < 0.0 1.4
         pyfcgid 15799 S 0.0 1.2
         pyfcgid 15801 S 0.0 1.1
         pyfcgid 15802 S 0.0 1.1
         pyfcgid 15803 S 0.0 1.1
         updated 8731 S 0.0 0.9
Run Time: 104 days, 21 hours and 11 minutes
1U, 0N, 89S, 10I; 1004T, 610F, 172KF
       ipsengine 20092 S < 19.6 8.1
            sshd 31691 S 7.6 0.3
          newcli 31695 R 6.8 0.8
         miglogd 70 R 5.1 1.4
 initXXXXXXXXXXX 1 S 1.7 0.3
          hasync 99 S < 0.8 0.5
        dnsproxy 112 S 0.8 0.3
          httpsd 157 S 0.0 3.2
          httpsd 125 S 0.0 2.3
         cmdbsvr 47 S 0.0 2.2
       ipshelper 20091 S < 0.0 1.4
         pyfcgid 15799 S 0.0 1.2
         pyfcgid 15801 S 0.0 1.1
         pyfcgid 15802 S 0.0 1.1
         pyfcgid 15803 S 0.0 1.1
         updated 8731 S 0.0 0.9
            pimd 13875 S 0.0 0.8
          newcli 31692 S 0.0 0.8
          httpsd 124 S 0.0 0.8
          httpsd 73 S 0.0 0.6
Run Time: 104 days, 21 hours and 11 minutes
0U, 0N, 69S, 31I; 1004T, 610F, 172KF
          newcli 31695 R 6.1 0.8
       ipsengine 20092 S < 5.3 8.1
            sshd 31691 S 4.4 0.3
          httpsd 157 S 0.0 3.2
          httpsd 125 S 0.0 2.3
         cmdbsvr 47 S 0.0 2.2
         miglogd 70 S 0.0 1.4
       ipshelper 20091 S < 0.0 1.4
         pyfcgid 15799 S 0.0 1.2
         pyfcgid 15801 S 0.0 1.1
         pyfcgid 15802 S 0.0 1.1
         pyfcgid 15803 S 0.0 1.1
         updated 8731 S 0.0 0.9
            pimd 13875 S 0.0 0.8
          newcli 31692 S 0.0 0.8
          httpsd 124 S 0.0 0.8
          httpsd 73 S 0.0 0.6
          hasync 99 S < 0.0 0.5
         uploadd 69 S 0.0 0.4
          cw_acd 116 S 0.0 0.4

[emnoc]

Suggestions & Ideals

is the unit license and pulling updates? What happen if you manually pull updates doe the cpu % increase?

is your local time correct and ntp enabled?

are you logging locally or remote or both? Can you  disable this temporary at that time? Also during this period what does your logging show ?

[snobs]

[ul]

So I disabled ipsengine with "diag test application ipsmonitor 98"

Local time is correct, NTP is enabled

Today, the problem still exist. Logging (I enabled logging to Fortianalyer, only) says:[ul]

139 07:03:51 Performance%20statistics%3A%20average%20CPU%3A%20100%2C%20memory%3A%20%2032%2C%20concurrent%20sessions%3A%20%2014491%2C%20setup-rate%3A%2071
140 07:03:51 Performance%20statistics%3A%20average%20CPU%3A%20100%2C%20memory%3A%20%2032%2C%20concurrent%20sessions%3A%20%2014491%2C%20setup-rate%3A%2071

[/ul]

Let´s look at diag sys top 1 without ipsengine:[ul]

Run Time: 105 days, 21 hours and 11 minutes
0U, 0N, 99S, 1I; 1004T, 676F, 172KF
          hatalk 94 S < 25.2 0.2
         miglogd 70 R 24.9 1.6
          hasync 99 S < 22.0 0.5
          newcli 5359 R 14.5 0.5
          newcli 5213 R 12.6 0.8
          sflowd 113 S 7.5 0.2
        dnsproxy 112 S 5.0 0.8
          newcli 5361 R 5.0 0.4
            sshd 5205 S 3.4 0.3
         telnetd 109 S 3.1 0.1
         telnetd 5360 S 2.8 0.2
  merged_daemons 77 S 2.8 0.2
       fclicense 82 S 2.5 0.2
 initXXXXXXXXXXX 1 S 1.8 0.3
           snmpd 93 S 1.5 0.3
    ipmc_sensord 71 S 1.5 0.3
             nsm 13868 S 1.2 0.1
           sqldb 90 S 0.6 0.2
         harelay 100 S 0.6 0.1
             fsd 119 S 0.6 0.1

Run Time: 105 days, 21 hours and 14 minutes
0U, 0N, 100S, 0I; 1004T, 676F, 172KF
         miglogd 70 R 25.9 1.6
          hatalk 94 S < 25.9 0.2
           snmpd 93 R 23.7 0.3
          hasync 99 S < 19.9 0.5
          newcli 5213 R 13.6 0.8
        dnsproxy 112 S 4.1 0.8
          sflowd 113 S 4.1 0.2
  merged_daemons 77 S 3.7 0.2
            sshd 5205 S 2.8 0.3
     proxyworker 96 S 1.5 0.2
  zebos_launcher 13866 S 1.5 0.1
          cw_acd 116 S 1.2 0.4
 initXXXXXXXXXXX 1 S 0.9 0.3
           quard 110 S 0.6 0.3
          ipldbd 12538 S 0.6 0.1
             fsd 119 S 0.6 0.1
         usbmuxd 118 S 0.6 0.0
         harelay 100 S 0.3 0.1
          httpsd 157 S 0.0 3.2
          httpsd 125 S 0.0 2.3

[/ul][/ul]