Analyzer upgrade from 5.6 to 6.0 becomes unresponsive after 8 hours
Hi,
I'm hoping someone can assist me. I've got an analyzer (3000D) that's been upgraded from version 5.6 to 6.0 (currently running 6.0.3)
The box will complete it's database rebuild as per the documentation but after about 8 hours of running, most of the processes go into a zombie state and the web gui becomes unavailable. You can ssh onto the box but an execute reboot just sits there and you have to physically power cycle the device. I've tried completely removing the db and rebuilding it but I continue to have the same issues.
Are there any commands that can be used to see what the box is actually doing (something like view /var/log/messages) or something similar. The diagnose test application commands sometimes work and sometimes don't and not having access to the actual unix cli is quite frustrating.
The box is swapping a lot and execute iotop shows sqllogd being the main culprit. It looks like the box is running out of memory
Total DISK READ : 1062.73 K/s | Total DISK WRITE : 221.40 K/s Actual DISK READ: 1059.04 K/s | Actual DISK WRITE: 571.95 K/s TID PRIO USER DISK READ DISK WRITE SWAPIN IO< COMMAND 7240 be/4 root 856.09 K/s 0.00 B/s 99.74 % 0.00 % sqllogd
execute top again shows sqllogd using lots of memory and the box swapping
top - 21:08:35 up 1:06, 0 users, load average: 10.17, 11.30, 11.31 Tasks: 369 total, 6 running, 360 sleeping, 0 stopped, 3 zombie %Cpu(s): 2.9 us, 11.9 sy, 0.0 ni, 78.8 id, 6.4 wa, 0.0 hi, 0.0 si, 0.0 st MiB Mem : 16013.96+total, 92.070 free, 15234.69+used, 687.203 buff/cache MiB Swap: 16012.99+total, 10370.72+free, 5642.270 used. 33.504 avail Mem PID USER PR NI VIRT RES %CPU %MEM TIME+ S COMMAND 7240 root 20 0 16.038g 0.014t 0.7 89.8 3:57.35 D /bin/sqllogd
If anyone has any advice (other than completely wiping the box and starting again), I'd gladly appreciate it.
Cheers,
G
