Analyzer upgrade from 5.6 to 6.0 becomes unresponsive after 8 hours

[GabbaTech]

Hi,

I'm hoping someone can assist me. I've got an analyzer (3000D) that's been upgraded from version 5.6 to 6.0 (currently running 6.0.3)

The box will complete it's database rebuild as per the documentation but after about 8 hours of running, most of the processes go into a zombie state and the web gui becomes unavailable. You can ssh onto the box but an execute reboot just sits there and you have to physically power cycle the device. I've tried completely removing the db and rebuilding it but I continue to have the same issues.

Are there any commands that can be used to see what the box is actually doing (something like view /var/log/messages) or something similar. The diagnose test application commands sometimes work and sometimes don't and not having access to the actual unix cli is quite frustrating.

The box is swapping a lot and execute iotop shows sqllogd being the main culprit. It looks like the box is running out of memory

Total DISK READ :    1062.73 K/s | Total DISK WRITE :     221.40 K/s Actual DISK READ:    1059.04 K/s | Actual DISK WRITE:     571.95 K/s   TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO<    COMMAND  7240 be/4 root      856.09 K/s    0.00 B/s 99.74 %  0.00 % sqllogd

execute top again shows sqllogd using lots of memory and the box swapping

top - 21:08:35 up  1:06,  0 users,  load average: 10.17, 11.30, 11.31 Tasks: 369 total,   6 running, 360 sleeping,   0 stopped,   3 zombie %Cpu(s):  2.9 us, 11.9 sy,  0.0 ni, 78.8 id,  6.4 wa,  0.0 hi,  0.0 si,  0.0 st MiB Mem : 16013.96+total,   92.070 free, 15234.69+used,  687.203 buff/cache MiB Swap: 16012.99+total, 10370.72+free, 5642.270 used.   33.504 avail Mem   PID USER      PR  NI    VIRT    RES  %CPU %MEM     TIME+ S COMMAND  7240 root      20   0 16.038g 0.014t   0.7 89.8   3:57.35 D /bin/sqllogd

If anyone has any advice (other than completely wiping the box and starting again), I'd gladly appreciate it.

Cheers,

 G

[RCHEN_FTNT]

Hi Gabba,

It seems it hit the bug#502046(initd has a bug to deadlock in certain situations and stop responding to signals/respawing crashed daemons) which was fixed on B279. and it had sqllogd high memory issue, Is it possible to get some more info as below for investigation?

diag deb klog

diag log device

diag test app  oft 2

diag fortil lograte

diag test app sqll 2

diag test app sqll 205 stats

diag test app sqll 206 stats

diag sql st sqlp

diag deb cr  rea

Thanks!

Rosie

[GabbaTech]

Hi Rosie,

Thanks very much for the reply.

I've been watching the box over the last 24 hours and it's definitely just before 8 hours during the database rebuild that the box begins to swap and inserts into the database start taking up to 30 minutes. The only way to resolve is to disable sql, reboot and re-enable sql.I've now removed the db and restarted and just let the box run without rebuilding from the archive logs.

Is there an email address or something I can send the output of those commands to as diag log device for example will include firewall hostnames and serial numbers which I don't really want floating around on the internet :)

Regards,

 G

[RCHEN_FTNT]

Hi Gabba,

It is better to  open a ticket for these mentioned  debug info.

Regards,

Rosie

[GabbaTech]

Thanks Rosie. I don't think the device is under maintenance though with it being an old 3000D model.

Appreciate your assistance.

Regards,

 G

[RCHEN_FTNT]

Hi Gabba,

It is better to open a ticket to upload the debug info.

Regards,

Rosie