FortiAnalyzer is not Inserting Logs

[chethan]

Hi,

We have a FortiAnalyzer VM deployed on ESXi last year at our customer's place. Everything was working fine but since a week we were not able to see any logs on "Log View". When we checked the dashboard, we can see that the FortiAnalyzer is receiving logs from the FortiGate but it is not Inserting them into the database.

The logs are still present in Log Browse (Compressed). FortiAnalyzer is in Analyzer mode and not Collector mode.

How do we fix this ? 

Thank you.

FortiAnalyzer FortiGate 

[Debbie_FTNT · Accepted solution]

Hey chethan,

first thing you can try is rebuilding the database.

exe sql-local rebuild-db

That shouldn't impact anything; while rebuilding the DB there is limited access to Log View/FortiView and Reporting, but you don't have those functions working anyway.

You can also refer to this KB for the rebuild command: https://community.fortinet.com/t5/FortiAnalyzer/Technical-Tip-FortiAnalyzer-SQL-database-delete-and-...

If, after the rebuild is completed, the issue still persists, you can try deleting and rebuilding to have the database recreated from scratch. Other than that, I can only suggest digging into the sql daemons:

For example:

#dia de app sqllogd 8

#dia de en

You can also check the release notes for your Analyzer's firmware version to see if there are reported issues, check the Analyzer's own event logs and the crashlog (dia de crashlog read, I believe) to see if there are any obvious errors.

[chethan]

Hey Debbie,

I ran the diagnose debug crashlog read command and the below is the following output.

Signal 11 (Segmentation Fault) ?

Firmware FAZVM64 v6.4.7-build2412 210902 (GA)
2022-03-04 09:58:25 Application ddmd
2022-03-04 09:58:25 Signal 11 (Segmentation fault)
2022-03-04 09:58:25 Pid 6207 ppid 6201 backtrace:
2022-03-04 09:58:25 [0x7fd5eded42e8] libsegfault.so print_trace+0x23/0x138
2022-03-04 09:58:25 [0x7fd5eded4497] libsegfault.so catch_segfault+0x9a/0xfb
2022-03-04 09:58:25 [0x7fd5ec8ff90f] libc.so.6 +0x3890f
2022-03-04 09:58:25 [0x7fd5ecabc16f] libmonitor.so monitorDbSetCustomFuncs+0x6f/0x80
2022-03-04 09:58:25 [0x7fd5ecabc289] libmonitor.so +0xd289
2022-03-04 09:58:25 [0x7fd5ecabc309] libmonitor.so monitorDbStartTransaction+0x5/0x42
2022-03-04 09:58:25 [0x7fd5ecabc92e] libmonitor.so +0xd92e
2022-03-04 09:58:25 [0x7fd5ecabc3b5] libmonitor.so monitorDbBuild+0x6f/0x133
2022-03-04 09:58:25 [ 0x414b94] ddmd main+0x3b/0xd1
2022-03-04 09:58:25 [0x7fd5ec8eadea] libc.so.6 __libc_start_main+0xea/0x1be
2022-03-04 09:58:25 [ 0x4024f9] ddmd +0x24f9

I'm yet to rebuild the database.

UPDATE:

I ran the rebuild command and the process is still in progress.

But I've started to see logs in "Log View" now.

Will update again after the rebuilt process is complete.

Thank you

[Debbie_FTNT]

Hey Chetan,

the ddmd process is something with device manager if I remember correctly; and the crash was some two weeks ago, so I'm not sure if it would be relevant.
If you see that same segmentation fault repeatedly, or if the logs stop showing and you see the crash again, I would suggest reaching out to FortiAnalzyer Technical Support; it could be a bug or something else going on that needs some in-depth troubleshooting.

[chethan]

Hi Debbie,

Thank you so much for your responses.

Now the database is rebuilt.

I'll monitor the crash logs and raise a support ticket if required. 

[Debbie_FTNT]

Hey Chethan,

no problem, happy I was able to help :)