| Description |
This article explains how to troubleshoot the 'IPS archive is 100% full. System will stop IPS archive logging' message, which can be found in the General System Events log section with severity level ‘Emergency’.
|
| Scope |
FortiGate, Disk Logging |
| Solution |
It is important to determine if the reported error is genuine or a false alarm. The debug outputs shown in this article are taken from multiple scenarios and are provided for reference only (i.e. the commands are relevant, but the debug output may not be identical across different scenarios/environments). The commands should be used for review and analysis within the respective environment.
Additionally, in v7.2.4 and later, the locallogd process was spun-off from miglogd to handle disk-logging specifically, so utilize the locallogd-equivalent CLI commands when working on newer FortiOS versions. CLI commands not involving miglogd or locallogd remain unchanged between versions.
diagnose test application miglogd 2 <----- v7.2.3 and earlier. diagnose test application locallogd 2 <----- v7.2.4 and later.
FortiGate (global) # diagnose test application miglogd 2 [...] vdom Firewall: id=1, working, master-ip=169.254.0.1, resolve port, donot resolve IP dst-geo=1 src-geo=1 brief_format=disabled event filter: event system vpn user router wireless wad endpoint ha security-rating fortiextender connector sdwan cifs-auth-fail switch-controller rest-api disk log: enabled, paused=0, pcap_dir=/var/log/log/Firewall/pcap arch_dir=/var/log/log/Firewall/dlp_archive log-disk: disk is not full, warning=75:90:95, current-warning-level=3, alertmail is not sent log-quota=0 dlp-quota=0 quar-quota=0 ips-quota=0 report-quota=0
The pcap_dir and arch_dir filepaths will be used in later steps.
diagnose test application miglogd 16 <----- v7.2.3 and earlier. diagnose test application locallogd 10 <----- v7.2.4 and later.
diagnose sys logdisk usage diagnose hardware deviceinfo disk
FortiGate (global) # diagnose test application miglogd 16 VDOM log disk usage: root: 543273B/20000M Firewall: 11663714B/318004M
FortiGate (global) # diagnose sys logdisk usage Total HD usage: 44184MB(42138MiB)/236286MB(225340MiB) Total HD logging space: 177214MB(169005MiB) HD logging space usage for vdom "root": 290MB(277MiB)/177214MB(169005MiB)
FortiGate (global) # diagnose hardware deviceinfo disk
Disk SSD1 ref: 255 223.6GiB type: SSD [ATA Micron_5300_MTFD] dev: /dev/sda partition ref: 1 220.1GiB, 178.9GiB free mounted: Y label: LOGUSEDX9D36C17F dev: /dev/sda1 start: 2048
In this case, the overall log disk utilization is around 18%, based on the Total HD usage above.
diagnose sys logdisk quota
FortiGate (global) # diagnose sys logdisk quota type quota(MB) usage(MB) ================ ============ ============ ----- vdom Firewall ----- log disk quota 318004 MB disk log: 0 12 dlp archive: 0 0 report db data: 0 0 quarantine: 0 0 ips archive: 0 4 packet capture: 100 0 ----- vdom root ----- log disk quota 0 MB disk log: 0 0 dlp archive: 0 0 report db data: 0 0 quarantine: 0 0 ips archive: 0 0 packet capture: 100 0
fnsysctl df –k
FortiGate (global) # fnsysctl df -k Filesystem 1K-blocks Used Available Use% Mounted on none 6347016 378388 5968628 6% /tmp none 6347016 7476 6339540 0% /dev/shm none 6347016 47168 6299848 1% /dev/cmdb /dev/sda2 253871 104008 136756 43% /data /dev/sda3 28327040 294184 26593928 1% /data2 /dev/sdb1 461489512 85160760 352886356 19% /var/log
get log disk setting << check for 'status', 'ips-archive', and 'max-policy-packet-capture-size', OR
get log disk setting | grep status get log disk setting | grep ips-archive get log disk setting | grep packet-capture
FortiGate (Firewall) # get log disk setting | grep status FortiGate (Firewall) # get log disk setting | grep ips-archive ips-archive : enable FortiGate (Firewall) # get log disk setting | grep packet-capture
fnsysctl ls -al /var/log/log/<vdom_name> fnsysctl la -al /var/log/log/<vdom_name>/pcap fnsysctl ls -al /var/log/log/<vdom_name>/pcap
In particular, check for historical 'dlp_archive.#####' or 'pcap.#####' directories with dates that should have been rolled/archived/overwritten but are still present:
drwx------ 10 0 0 Mon Jul 22 09:50:06 2024 4096 dlp_archive.50667 drwx------ 10 0 0 Mon Jul 22 09:10:29 2024 4096 dlp_archive.50668 drwx------ 10 0 0 Mon Jul 22 08:33:06 2024 4096 dlp_archive.50669
drwxr-xr-x 2 0 0 Mon Jul 22 09:50:06 2024 40960 pcap.50667 drwxr-xr-x 2 0 0 Mon Jul 22 09:10:29 2024 73728 pcap.50668 drwxr-xr-x 2 0 0 Mon Jul 22 08:33:06 2024 73728 pcap.50669
fnsysctl du /var/log/log/<vdom_name>
FortiGate (global) # fnsysctl du /var/log/log/<vdom_name> [output truncated...] 42041468 /var/log << Output in # of blocksThis corresponds to 40Gig of log files
Log Disk Rolling and Identifying Orphaned Files: When the log disk reaches capacity, the system automatically deletes logs older than the configured maximum-log-age value (default: 7 days, configured under config log disk setting; see also: Technical Tip: How to set the maximum age for logs on disk).
IPS packet-capture archive files are stored in /var/log/log/<vdom_name>/pcap before being rolled. Based on the configured roll-schedule (also under config log disk setting), these files are rolled into numbered directories using a descending roll number sequence from 65535 down to 0. For example:
/var/log/log/<vdom_name>/pcap (active) /var/log/log/<vdom_name>/pcap.65534 (newer rolled archive) /var/log/log/<vdom_name>/pcap.65535 (older rolled archive)
The roll number of the archived folder should align with the roll number of the corresponding traffic and UTM logs. If the archived PCAP/DLP directories are out of sync with current logs (i.e. the roll numbers are significantly mismatched from active logs), then they are likely no longer referenced by the system and will not be removed by cleanup logic (they also cannot be manually deleted). The date of creation for these directories would be significantly older than the maximum-log-age should allow for.
This behavior did exist in older FortiOS versions and was tracked by Known Issue #954565. This issue in particular was resolved in FortiOS 7.0.14, 7.2.8, and 7.4.4, though it is possible for the issue to still potentially occur in later versions (e.g. v7.4.8) if there were unreferenced files that were created before the fix was applied.
Under normal operation, the system deletes old archives when they exceed maximum-log-age, or when disk space begins to run out. When the above issue occurs, the system does not delete certain archive folder types when the corresponding traffic logs for these archives were already removed, which leaves these archive folders as unreferenced. These unreferenced archives could accumulate excessively, leading to the event logs being generated that state 'IPS archive is 100% full, System will stop IPS archive logging.'
In resolved versions of FortiOS, the miglogd process will check for unreferenced/unneeded log archives and perform cleanup. However, if there are still unreferenced directories/files in the filesystem from before the fix was created, then the solution is to format the log disk using the command execute formatlogdisk, which will clear out all local log disk files and reformat the log disk. Ensure that any required logs are downloaded from the disk before running this command. |
