|
FortiAnalyzer v7.4 upgrade does not imply a Log Database Rebuild if the starting release is v6.4.0 or later: Firmware Upgrade Paths
Although right after the upgrade to v7.4, two common symptoms have been reported:
- High I/O disk usage:
- Low or zero insert rate, despite normal values of the receive rate:
This behavior is expected and is due to the upgrade of log tables structure, which can last from several minutes to several hours, depending on the Log Database size.
To have a confirmation that the observed behavior is caused by the log table upgrade, two checks can be done:
- The output of 'diagnose debug application sqllogd' will show 'Waiting for sqlplugind':
FAZ003 # diagnose debug application sqllogd -1
FAZ003 # diagnose debug enable
FAZ003 # diagnose debug application sqllogd 255
FAZ003 # [T1685:sqllog_main.c:455] Waiting for sqlplugind ...
diagnose debug application sqllogd 255[T1685:sqllog_main.c:455] Waiting for sqlplugind ...
[T1685:sqllog_main.c:455] Waiting for sqlplugind ...
[T1685:sqllog_main.c:455] Waiting for sqlplugind ...
[T1685:sqllog_main.c:455] Waiting for sqlplugind ...
[T1685:sqllog_main.c:455] Waiting for sqlplugind ...
[T1685:sqllog_main.c:455] Waiting for sqlplugind ...
[T1685:sqllog_main.c:455] Waiting for sqlplugind ...
^C[T1685:sqllog_main.c:455] Waiting for sqlplugind ...
^C^C^C[T1685:sqllog_main.c:455] Waiting for sqlplugind ...
- The output of 'diagnose debug application sqlplugind -1' will show the log table upgrade status in real-time
[1762258641] INFO: sqlplugind(1687):dbmaint.c:1527: Upgrading 31% done. (497/1587)
[1762258641] NOTIF: sqlplugind(1687):dbmaint.c:1680: upgrade process change to background task...
[1762258641] INFO: sqlplugind(1687):dbmaint.c:1361: Upgrading table 'FSFADOM3-FGT-ALLELSE-wlog-1762087860-0' (071678->072825)...
[1762258641] INFO: sqlplugind(1687):dbmaint.c:1527: Upgrading 31% done. (498/1587)
[1762258641] INFO: sqlplugind(1687):dbmaint.c:1527: Upgrading 31% done. (499/1587)
[1762258641] INFO: sqlplugind(1687):dbmaint.c:1361: Upgrading table 'FSFADOM3-FGT-tlog-1762087830' (071678->072825)...
[1762258641] INFO: sqlplugind(1687):dbmaint.c:1527: Upgrading 31% done. (500/1587)
....
[1762258646] INFO: sqlplugind(1687):dbmaint.c:1527: Upgrading 31% done. (501/1587)
[1762258646] INFO: sqlplugind(1687):dbmaint.c:1361: Upgrading table 'FSFADOM3-FGT-wlog-1762087860' (071678->072825)...
[1762258646] INFO: sqlplugind(1687):dbmaint.c:1527: Upgrading 31% done. (502/1587)
[1762258646] INFO: sqlplugind(1687):dbmaint.c:1361: Upgrading table 'FSFADOM3-FGT-ALLELSE-dlog-1762003650-0' (071678->072825)...
[1762258646] INFO: sqlplugind(1687):dbmaint.c:1527: Upgrading 31% done. (503/1587)
[1762258646] INFO: sqlplugind(1687):dbmaint.c:1361: Upgrading table 'FSFADOM3-FGT-dlog-1762003650' (071678->072825)...
[1762258646] INFO: sqlplugind(1687):dbmaint.c:1527: Upgrading 31% done. (504/1587)
[1762258646] INFO: sqlplugind(1687):dbmaint.c:1361: Upgrading table 'FSFADOM3-FGT-ALLELSE-rlog-1762087740-0' (071678->072825)...
[1762258646] NOTIF: sqlplugind(1687):processor.c:2513: Creating 2 writer, 2 indexer and 2 compressor threads...
[1762258647] DEBUG: sqlplugind(1687):pq_plugin.c:20: set PQconnectdb options parameter=[options='-c TimeZone=Europe/Dublin']
[1762258647] DEBUG: sqlplugind(1687):pq_plugin.c:20: set PQconnectdb options parameter=[options='-c TimeZone=Europe/Dublin']
[1762258647] DEBUG: sqlplugind(1687):pq_plugin.c:20: set PQconnectdb options parameter=[options='-c TimeZone=Europe/Dublin']
[1762258647] DEBUG: sqlplugind(1687):pq_plugin.c:20: set PQconnectdb options parameter=[options='-c TimeZone=Europe/Dublin']
[1762258647] DEBUG: sqlplugind(1687):pq_plugin.c:20: set PQconnectdb options parameter=[options='-c TimeZone=Europe/Dublin']
[1762258647] DEBUG: sqlplugind(1687):pq_plugin.c:20: set PQconnectdb options parameter=[options='-c TimeZone=Europe/Dublin']
[1762258647] ERROR: sqlplugind(1687):sql_intfstat.c:377: Failed to open dir /drive0/private/dbcommit/intfstat.
[1762258648] DEBUG: sqlplugind(1680):pgsvr_main.c:134: postgres is running...
[1762258648] DEBUG: sqlplugind(1680):pgsvr_main.c:1434: Redis(SOC Fabric) status: stopped (pid=-1)
[1762258648] DEBUG: sqlplugind(1680):pgsvr_main.c:1551: Monitor: SOC Fabric Message Broker Service(Redis) status: is_needed=0, is_changed=0, is_running=0
0:0 2000/1/1
[1762258652] INFO: sqlplugind(1687):dbmaint.c:1527: Upgrading 31% done. (505/1587)
If similar outputs are shown, it is suggested to wait for the table upgrade process to complete. After reaching 70%, the log insertion rate will start increasing, and disk I/O will decrease, but the system should come back to a normal state after full completion.
If this does not happen, it is suggested to open a Case to Fortinet support, as described here:
Technical Tip: How to open a ticket to Fortinet TAC
|
