|
FortiAnalyzer v7.4 upgrade does not imply a Log Database Rebuild if the starting release is v6.4.0 or later: Firmware Upgrade Paths.
Although right after the upgrade to v7.4, two common symptoms have been reported:
- High I/O disk usage:
- Low or zero insert rate, despite normal values of the receive rate:
This behavior is expected and is due to the upgrade of the log tables structure, which can last from several minutes to several hours, depending on the Log Database size.
To have a confirmation that the observed behavior is caused by the log table upgrade, two checks can be done:
- The output of 'diagnose debug application sqllogd' will show 'Waiting for sqlplugind':
FAZ003 # diagnose debug application sqllogd -1
FAZ003 # diagnose debug enable
FAZ003 # diagnose debug application sqllogd 255
FAZ003 # [T1685:sqllog_main.c:455] Waiting for sqlplugind ...
diagnose debug application sqllogd 255[T1685:sqllog_main.c:455] Waiting for sqlplugind ...
[T1685:sqllog_main.c:455] Waiting for sqlplugind ...
[T1685:sqllog_main.c:455] Waiting for sqlplugind ...
[T1685:sqllog_main.c:455] Waiting for sqlplugind ...
[T1685:sqllog_main.c:455] Waiting for sqlplugind ...
[T1685:sqllog_main.c:455] Waiting for sqlplugind ...
[T1685:sqllog_main.c:455] Waiting for sqlplugind ...
^C[T1685:sqllog_main.c:455] Waiting for sqlplugind ...
^C^C^C[T1685:sqllog_main.c:455] Waiting for sqlplugind ...
- The output of 'diagnose debug application sqlplugind -1' will show the log table upgrade status in real-time:
[1762258641] INFO: sqlplugind(1687):dbmaint.c:1527: Upgrading 31% done. (497/1587)
[1762258641] NOTIF: sqlplugind(1687):dbmaint.c:1680: upgrade process change to background task...
[1762258641] INFO: sqlplugind(1687):dbmaint.c:1361: Upgrading table 'FSFADOM3-FGT-ALLELSE-wlog-1762087860-0' (071678->072825)...
[1762258641] INFO: sqlplugind(1687):dbmaint.c:1527: Upgrading 31% done. (498/1587)
[1762258641] INFO: sqlplugind(1687):dbmaint.c:1527: Upgrading 31% done. (499/1587)
[1762258641] INFO: sqlplugind(1687):dbmaint.c:1361: Upgrading table 'FSFADOM3-FGT-tlog-1762087830' (071678->072825)...
[1762258641] INFO: sqlplugind(1687):dbmaint.c:1527: Upgrading 31% done. (500/1587)
....
[1762258646] INFO: sqlplugind(1687):dbmaint.c:1527: Upgrading 31% done. (501/1587)
[1762258646] INFO: sqlplugind(1687):dbmaint.c:1361: Upgrading table 'FSFADOM3-FGT-wlog-1762087860' (071678->072825)...
[1762258646] INFO: sqlplugind(1687):dbmaint.c:1527: Upgrading 31% done. (502/1587)
[1762258646] INFO: sqlplugind(1687):dbmaint.c:1361: Upgrading table 'FSFADOM3-FGT-ALLELSE-dlog-1762003650-0' (071678->072825)...
[1762258646] INFO: sqlplugind(1687):dbmaint.c:1527: Upgrading 31% done. (503/1587)
[1762258646] INFO: sqlplugind(1687):dbmaint.c:1361: Upgrading table 'FSFADOM3-FGT-dlog-1762003650' (071678->072825)...
[1762258646] INFO: sqlplugind(1687):dbmaint.c:1527: Upgrading 31% done. (504/1587)
[1762258646] INFO: sqlplugind(1687):dbmaint.c:1361: Upgrading table 'FSFADOM3-FGT-ALLELSE-rlog-1762087740-0' (071678->072825)...
[1762258646] NOTIF: sqlplugind(1687):processor.c:2513: Creating 2 writer, 2 indexer and 2 compressor threads...
[1762258647] DEBUG: sqlplugind(1687):pq_plugin.c:20: set PQconnectdb options parameter=[options='-c TimeZone=Europe/Dublin']
[1762258647] DEBUG: sqlplugind(1687):pq_plugin.c:20: set PQconnectdb options parameter=[options='-c TimeZone=Europe/Dublin']
[1762258647] DEBUG: sqlplugind(1687):pq_plugin.c:20: set PQconnectdb options parameter=[options='-c TimeZone=Europe/Dublin']
[1762258647] DEBUG: sqlplugind(1687):pq_plugin.c:20: set PQconnectdb options parameter=[options='-c TimeZone=Europe/Dublin']
[1762258647] DEBUG: sqlplugind(1687):pq_plugin.c:20: set PQconnectdb options parameter=[options='-c TimeZone=Europe/Dublin']
[1762258647] DEBUG: sqlplugind(1687):pq_plugin.c:20: set PQconnectdb options parameter=[options='-c TimeZone=Europe/Dublin']
[1762258647] ERROR: sqlplugind(1687):sql_intfstat.c:377: Failed to open dir /drive0/private/dbcommit/intfstat.
[1762258648] DEBUG: sqlplugind(1680):pgsvr_main.c:134: postgres is running...
[1762258648] DEBUG: sqlplugind(1680):pgsvr_main.c:1434: Redis(SOC Fabric) status: stopped (pid=-1)
[1762258648] DEBUG: sqlplugind(1680):pgsvr_main.c:1551: Monitor: SOC Fabric Message Broker Service(Redis) status: is_needed=0, is_changed=0, is_running=0
0:0 2000/1/1
[1762258652] INFO: sqlplugind(1687):dbmaint.c:1527: Upgrading 31% done. (505/1587)
If similar outputs are shown, it is suggested to wait for the table upgrade process to complete. After reaching 70%, the log insertion rate will start increasing, and disk I/O will decrease, but the system should come back to a normal state after full completion.
If IOwait% still spikes after the postgres+ complete upgrade, it is possible to check the disk usage in FortiAnalyzer.
CPU:
Used: 10.74%
Used(Excluded NICE): 10.74%
%used %user %nice %sys %idle %iowait %irq %softirq
CPU0 5.74 3.69 0.00 1.23 94.26 0.82 0.00 0.00
CPU1 8.38 5.11 0.00 2.04 91.62 1.02 0.00 0.20
CPU2 6.75 5.32 0.00 1.43 93.25 0.00 0.00 0.00
CPU3 11.43 6.53 0.00 3.88 88.57 1.02 0.00 0.00
CPU4 10.10 9.07 0.00 0.82 89.90 0.21 0.00 0.00
CPU5 21.95 17.07 0.00 3.46 78.05 1.22 0.00 0.20
AdomName AdomOID Type Logs Database
[Retention Quota Used( logs/quaranti/ content/ IPS) Used%] [Retention Quota Used(
SiemDB/ hcache) Used%]
root 3 FSF 600days 3194.9GB 2982.6GB(2982.4GB/ 0.0KB/ 0.0KB/ 209.7MB) 93.4% 730days 7454.7GB 6987.0GB(
2325.6GB/ 589.1GB) 93.7%
Total usage: 1 ADOMs, logs=2982.6GB(2982.4GB/0.0KB/0.0KB/209.7MB) database=6995.7GB(ADOMs usage:6987.0GB(2325.6GB, 589.1GB) + Internal U
sage:8.7GB)
Total Quota Summary:
Total Quota Allocated Available Allocate%
10700.0GB 10649.6GB 50.4GB 99.5%
System Storage Summary:
Total Used Available Use%
11000.0GB 10030.8GB 969.2GB 91.2%
When the IOwait spikes, it will cause all the FortiAnalyzer functionalities to delay, such as GUI loading.
To fix the issue, ensure the logs respect the configured ADOM storage ratio (default 70%: 30%).
Then delete the old logs when they reach 90%.
If FortiAnalyzer disk is in normal condition but IOwait% still spikes, it is recommended to run the diagnose system fsck harddisk (to repair and check the health disk). Note that the command will force the device to reboot.
|
