Troubleshooting Tip: How to fix the error 'IPS archive is 100% full. System will stop IPS archive logging'

Description

This article explains how to troubleshoot the 'IPS archive is 100% full. System will stop IPS archive logging' message, which can be found in the General System Events log section with severity level ‘Emergency’.

Scope

FortiGate, Disk logging.

Solution

It is important to determine if the reported error is genuine or a false alarm.

The debug outputs shown in this article are taken from multiple scenarios and are provided for reference only (i.e., the commands are relevant, but the debug output may not be identical across different scenarios/environments).

The commands should be used for review and analysis within the respective environment.

Additionally,  in v7.2.4 and later, the locallogd process was spun off from miglogd to handle disk logging specifically, so utilize the locallogd-equivalent CLI commands when working on newer FortiOS versions. CLI commands not involving miglogd or locallogd remain unchanged between versions.

1. Check if disk logging is enabled using one of the following commands:

diagnose test application miglogd 2 

This command works in v7.2.3 and earlier.

diagnose test application locallogd 2

This command works in v7.2.4 and later.

FortiGate (global) # diagnose test application miglogd 2

[...]

vdom Firewall: id=1, working, master-ip=169.254.0.1, resolve port, donot resolve IP dst-geo=1 src-geo=1 brief_format=disabled

event filter:

         event

system vpn user router wireless wad endpoint ha security-rating fortiextender connector sdwan cifs-auth-fail switch-controller rest-api

disk log: enabled, paused=0, pcap_dir=/var/log/log/Firewall/pcap

        arch_dir=/var/log/log/Firewall/dlp_archive

        log-disk: disk is not full, warning=75:90:95, current-warning-level=3, alertmail is not sent

                log-quota=0

                dlp-quota=0

                quar-quota=0

                ips-quota=0

                report-quota=0

The pcap_dir and arch_dir filepaths will be used in later steps.

2. Review log disk utilization using the following commands:

diagnose test application miglogd 16

This command works for v7.2.3 and earlier.

diagnose test application locallogd 10

This command works for v7.2.4 and above.

diagnose sys logdisk usage

diagnose hardware deviceinfo disk

 

FortiGate (global) # diagnose test application miglogd 16

VDOM log disk usage:

  root: 543273B/20000M

  Firewall: 11663714B/318004M

 

FortiGate (global) # diagnose sys logdisk usage

Total HD usage: 44184MB(42138MiB)/236286MB(225340MiB)

Total HD logging space: 177214MB(169005MiB)

HD logging space usage for vdom "root": 290MB(277MiB)/177214MB(169005MiB)

 

FortiGate (global) # diagnose hardware deviceinfo disk

 

Disk SSD1 ref: 255 223.6GiB type: SSD [ATA Micron_5300_MTFD] dev: /dev/sda

partition ref: 1 220.1GiB, 178.9GiB free mounted: Y label: LOGUSEDX9D36C17F dev: /dev/sda1 start: 2048

In this case, the overall log disk utilization is around 18%, based on the Total HD usage above.

3. Check the log disk quotas using the following command:

diagnose sys logdisk quota

 

FortiGate (global) # diagnose sys logdisk quota

type quota(MB) usage(MB)

================ ============ ============

----- vdom Firewall -----

log disk quota 318004 MB

disk log: 0 12

dlp archive: 0 0

report db data: 0 0

quarantine: 0 0

ips archive: 0 4

packet capture: 100 0

----- vdom root -----

log disk quota 0 MB

disk log: 0 0

dlp archive: 0 0

report db data: 0 0

quarantine: 0 0

ips archive: 0 0

packet capture: 100 0

4. Check disk usage of the underlying filesystem using the following command:

fnsysctl df –k

 
FortiGate (global) # fnsysctl df -k

Filesystem 1K-blocks Used Available Use% Mounted on
none 6347016 378388 5968628 6% /tmp
none 6347016 7476 6339540 0% /dev/shm
none 6347016 47168 6299848 1% /dev/cmdb
/dev/sda2 253871 104008 136756 43% /data
/dev/sda3 28327040 294184 26593928 1% /data2
/dev/sdb1 461489512 85160760 352886356 19% /var/log

5. Check if IPS packet archiving is enabled in the configuration (must be run in a non-Global VDOM):
   
   

get log disk setting 

Check for 'status', 'ips-archive', and 'max-policy-packet-capture-size' OR please check below.

get log disk setting | grep status

get log disk setting | grep ips-archive

get log disk setting | grep packet-capture

 

FortiGate (Firewall) # get log disk setting | grep status
status : enable

FortiGate (Firewall) # get log disk setting | grep ips-archive

ips-archive : enable

FortiGate (Firewall) # get log disk setting | grep packet-capture
max-policy-packet-capture-size: 100

6. Check the filesystem and inspect for archived files and packet captures using the following commands (replacing '<vdom_name>' with the appropriate VDOM):

fnsysctl ls -al /var/log/log/<vdom_name>

fnsysctl ls -al /var/log/log/<vdom_name>/pcap

fnsysctl ls -al /var/log/log/<vdom_name>/pcap

In particular, check for historical 'dlp_archive.#####' or 'pcap.#####' directories with dates that should have been rolled/archived/overwritten but are still present:

drwx------   10 0        0       Mon Jul 22 09:50:06 2024             4096 dlp_archive.50667
drwx------   10 0        0       Mon Jul 22 09:10:29 2024             4096 dlp_archive.50668
drwx------   10 0        0       Mon Jul 22 08:33:06 2024             4096 dlp_archive.50669

 

drwxr-xr-x    2 0        0       Mon Jul 22 09:50:06 2024            40960 pcap.50667
drwxr-xr-x    2 0        0       Mon Jul 22 09:10:29 2024            73728 pcap.50668
drwxr-xr-x    2 0        0       Mon Jul 22 08:33:06 2024            73728 pcap.50669

7. Check the total size of the VDOM log directory using the following command:

fnsysctl du /var/log/log/<vdom_name>


FortiGate (global) # fnsysctl du /var/log/log/<vdom_name>
[output truncated...]
42041468        /var/log << Output in # of blocksThis corresponds to 40Gig of log files

Log disk rolling and identifying orphaned files:

When the log disk reaches capacity, the system automatically deletes logs older than the configured maximum-log-age value (default: 7 days, configured under config log disk setting; see also: Technical Tip: How to set the maximum age for logs on disk).

IPS packet-capture archive files are stored in /var/log/log/<vdom_name>/pcap before being rolled. Based on the configured roll-schedule (also under the config log disk setting), these files are rolled into numbered directories using a descending roll number sequence from 65535 down to 0. For example: 

/var/log/log/<vdom_name>/pcap           (active)

/var/log/log/<vdom_name>/pcap.65534     (newer rolled archive)

/var/log/log/<vdom_name>/pcap.65535     (older rolled archive)

The roll number of the archived folder should align with the roll number of the corresponding traffic and UTM logs. 

If the archived PCAP/DLP directories are out of sync with current logs (i.e., the roll numbers are significantly mismatched from active logs), then they are likely no longer referenced by the system and will not be removed by cleanup logic (they also cannot be manually deleted).  The date of creation for these directories would be significantly older than the maximum-log-age should allow for.

This behavior did exist in older FortiOS versions and was tracked by Known Issue #954565. This issue in particular was resolved in FortiOS v7.0.14, v7.2.8, and v7.4.4, though it is possible for the issue to still potentially occur in later versions (e.g., v7.4.8) if there were unreferenced files that were created before the fix was applied. 

Under normal operation, the system deletes old archives when they exceed maximum-log-age, or when disk space begins to run out. When the above issue occurs, the system does not delete certain archive folder types when the corresponding traffic logs for these archives were already removed, which leaves these archive folders as unreferenced.

These unreferenced archives could accumulate excessively, leading to event logs being generated that state 'IPS archive is 100% full, System will stop IPS archive logging.' 

In resolved versions of FortiOS, the miglogd process will check for unreferenced/unneeded log archives and perform cleanup. However, if there are still unreferenced directories/files in the filesystem from before the fix was created, then the solution is to format the log disk using the command execute formatlogdisk, which will clear out all local log disk files and reformat the log disk. Ensure that any required logs are downloaded from the disk before running this command.

Related article:

Technical Tip: How to set the maximum age for logs on disk