Analyzer can't finish rebuilding Log DB

[James_Dore]

Hi all,

I was presented with a FortiAnalyzer 400E from another unit we work with, which was purchased in 2016 but never taken out of the box. Since all units in our group now have Fortigate firewalls, I've been tasked with getting it going. It's now installed, and after having a missing disk 2 sorted out (SATA connector not mating on the backplane properly) it's running. 

I have upgraded it to the latest OS for the 400E, 6.2.3, and connected our local Fortigate to the device. Logs are being sent to the Analyzer. 

However, I can't do anything with the logs, because the Analyzer is stuck with the message "Rebuilding DB - Build Log DB..." on the console. If I click on this message, it shows 1% complete, and is at Step 2: Rebuilding SQL database...., estimated time remaining 42 minutes 44 seconds. Log and report features won't be fully available till rebuilding has completed." 

It's been like this for two weeks now, and this persists between reboots, firmware upgrades, and CLI command fiddling.  In the CLI, issuing

analyzer.new # diagnose sql status rebuild-db

Rebuilding log SQL database will be starting in a moment...

Is all that is shown. 

analyzer.new # diagnose sql status rebuild-adom 
FortiAnalyzer is not in rebuild
FortiAuthenticator is not in rebuild
FortiCache is not in rebuild
FortiCarrier is not in rebuild
FortiClient is not in rebuild
FortiDDoS is not in rebuild
FortiMail is not in rebuild
FortiManager is not in rebuild
FortiNAC is not in rebuild
FortiProxy is not in rebuild
FortiSandbox is not in rebuild
FortiWeb is not in rebuild
Newc is not in rebuild
Syslog is not in rebuild
root is not in rebuild

- doesn't give anything useful either.  Actually issuing the 

analyzer.new # execute sql-local rebuild-db 
Rebuild the entire log SQL database has been requested.
This operation will remove the log SQL database and rebuild from log data.
This operation will reboot the device.
Do you want to continue? (y/n)y

Command reboots the device back in to the exact same state, achieving diddly-squat. 

Can anyone help me fix this please?

Typically, as it's nearly four years old, we have no support contract :(

Cheers,

James

[brazz_FTNT]

Hey,

can you please run 

[ul]

get system status 

diagnose cdb upgrade summary

diagnose debug crashlog read[/ul]

Thanks

[James_Dore]

Sure: 

analyzer.new # get system status 
Platform Type : FAZ400E
Platform Full Name : FortiAnalyzer-400E
Version : v6.2.3-build1235 191218 (GA)
Serial Number : FL-4HE3R16900167
BIOS version : 00020005
System Part-Number : P18712-02
Hostname : analyzer.new
Max Number of Admin Domains : 25
Admin Domain Configuration : Enabled
FIPS Mode : Disabled
Branch Point : 1235
Release Version Information : GA
Current Time : Mon Jan 27 11:30:29 GMT 2020
Daylight Time Saving : Yes
Time Zone : (GMT) London, Edinburgh.
x86-64 Applications : Yes
Disk Usage : Free 5482.83GB, Total 5501.21GB
File System : Ext4
FortiRecorder Cameras : 0 active / 12 allowed

analyzer.new # diagnose cdb upgrade summary

==== Configuration database upgraded from legacy version ====
2019-12-06 12:46:54 v6.2.2-build1183 191008 (GA)
2020-01-21 11:41:59 v6.2.3-build1235 191218 (GA)

analyzer.new # diagnose debug crashlog read

analyzer.new #

(i.e. no output for the last command).

Cheers,

James

[brazz_FTNT]

Hello, 

Thanks for the update. 

One questions 

[ul]

Does this mean there is an old FAZ ? did you use the old config from your old FAZ ?

How many devices are on your NewFAZ? (diagnose dvm device list)

Are they actively sending logs to FAZ?(Check the LogView for Historical and Real time logs)

Also if nothing works properly at the end, you might consider formatting the FAZ completely and add the devices back to it. [/ul]

[James_Dore]

Hi there - 

analyzer.new is the host name - we are New College, University of Oxford - new.ox.ac.uk - we don't have any other FAZ's. 

There are two devices sending devices to the Analyzer, each end of our Fortigate HA pair - two 501E's: 

analyzer.new # diagnose dvm device list 
--- There are currently 2 devices/vdoms managed ---

TYPE OID SN HA IP NAME ADOM IPS FIRMWARE 
faz enabled 153 FG5H1E581890643 a-p 129.67.111.9 nc-fw-a Newc N/A 6.0 MR0 (303) 
 |- STATUS: dev-db: unknown; conf: unknown; cond: unknown; dm: unknown; conn: unknown
 HA cluster member: FG5H1E581890643 (master)
 HA cluster member: FG5H1E5818906588 (slave 1)
 |- vdom:[3]root flags:0 adom:Newc pkg:[never-installed]
faz enabled 178 FG5H1E5818906431 a-p 129.67.111.9 nc-fw-b Newc N/A 6.0 MR0 (303) 
 |- STATUS: dev-db: unknown; conf: unknown; cond: unknown; dm: none; conn: unknown
 HA cluster member: FG5H1E5818906431 (master)
 |- vdom:[3]root flags:0 adom:Newc pkg:[never-installed]

--- There are currently 0 FortiAP managed ---

--- There are currently 0 FortiSwitch managed ---

--- There are currently 0 FortiExtender managed ---

--- End device list ---

They are sending logs ok - logs appear in the Log view. I;ve got Traffic, Event and VoIP logs with current timestamps (2020-01-28-14:38 is the latest). 

I'm quite happy to format and erase the FAZ, what's the process for that? I googled for things like restore to factory settings, or erase and reinstall, but obviously didn't use the correct terms!

Cheers,

James

[James_Dore]

Found it. Dang, that was easy. It even retained the network config and SSL certificate, which was nice. The rebuild message has gone, and it looks like we're in business. 

Cheers!

James

[scan]

Hi James,

I have the same issue. What exactly was your solution?

Cheers,

scan

[James_Dore]

scan wrote:

Hi James,

I have the same issue. What exactly was your solution?

Cheers,

scan

Hi there - if I recall correctly, it was 

execute sql-local rebuild-db

From the cli. 

Cheers, James