Skip to main content
GabbaTech
New Member
February 18, 2019
Question

Analyzer upgrade from 5.6 to 6.0 becomes unresponsive after 8 hours

  • February 18, 2019
  • 5 replies
  • 5199 views

Hi,

 

I'm hoping someone can assist me. I've got an analyzer (3000D) that's been upgraded from version 5.6 to 6.0 (currently running 6.0.3)

 

The box will complete it's database rebuild as per the documentation but after about 8 hours of running, most of the processes go into a zombie state and the web gui becomes unavailable. You can ssh onto the box but an execute reboot just sits there and you have to physically power cycle the device. I've tried completely removing the db and rebuilding it but I continue to have the same issues.

 

Are there any commands that can be used to see what the box is actually doing (something like view /var/log/messages) or something similar. The diagnose test application commands sometimes work and sometimes don't and not having access to the actual unix cli is quite frustrating.

 

The box is swapping a lot and execute iotop shows sqllogd being the main culprit. It looks like the box is running out of memory

 

Total DISK READ :    1062.73 K/s | Total DISK WRITE :     221.40 K/s Actual DISK READ:    1059.04 K/s | Actual DISK WRITE:     571.95 K/s   TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO<    COMMAND  7240 be/4 root      856.09 K/s    0.00 B/s 99.74 %  0.00 % sqllogd

 

execute top again shows sqllogd using lots of memory and the box swapping

 

top - 21:08:35 up  1:06,  0 users,  load average: 10.17, 11.30, 11.31 Tasks: 369 total,   6 running, 360 sleeping,   0 stopped,   3 zombie %Cpu(s):  2.9 us, 11.9 sy,  0.0 ni, 78.8 id,  6.4 wa,  0.0 hi,  0.0 si,  0.0 st MiB Mem : 16013.96+total,   92.070 free, 15234.69+used,  687.203 buff/cache MiB Swap: 16012.99+total, 10370.72+free, 5642.270 used.   33.504 avail Mem   PID USER      PR  NI    VIRT    RES  %CPU %MEM     TIME+ S COMMAND  7240 root      20   0 16.038g 0.014t   0.7 89.8   3:57.35 D /bin/sqllogd

 

If anyone has any advice (other than completely wiping the box and starting again), I'd gladly appreciate it.

 

Cheers,

 G

    5 replies

    RCHEN_FTNT
    Staff
    Staff
    February 19, 2019

    Hi Gabba,

     

    It seems it hit the bug#502046(initd has a bug to deadlock in certain situations and stop responding to signals/respawing crashed daemons) which was fixed on B279. and it had sqllogd high memory issue, Is it possible to get some more info as below for investigation?

     

    diag deb klog

    diag log device

    diag test app  oft 2

    diag fortil lograte

    diag test app sqll 2

    diag test app sqll 205 stats

    diag test app sqll 206 stats

    diag sql st sqlp

    diag deb cr  rea

     

    Thanks!

     

    Rosie

     

     

    GabbaTech
    GabbaTechAuthor
    New Member
    February 19, 2019

    Hi Rosie,

     

    Thanks very much for the reply.

     

    I've been watching the box over the last 24 hours and it's definitely just before 8 hours during the database rebuild that the box begins to swap and inserts into the database start taking up to 30 minutes. The only way to resolve is to disable sql, reboot and re-enable sql.I've now removed the db and restarted and just let the box run without rebuilding from the archive logs.

     

    Is there an email address or something I can send the output of those commands to as diag log device for example will include firewall hostnames and serial numbers which I don't really want floating around on the internet :)

     

    Regards,

     G

    RCHEN_FTNT
    Staff
    Staff
    February 19, 2019

    Hi Gabba,

     

    It is better to  open a ticket for these mentioned  debug info.

     

    Regards,

     

    Rosie

    Thought Leadership Security Summit. Outpace New Threats with AI - enhanced defense. Tuesday, Septmeber 15, 8:30 AM - 2:30 PM PT. The Golf Club at Newcastle, WA.
    Virtual event | September 2026. SASE summit. The age of autonomous trust. Register here!