Technical Tip: How Retention Policies work

[tskandamis]

Description	This article describes how Retention Policies work.
Scope	FortiSIEM storage.
Solution	When the Online database becomes full, events must be deleted to make room for new events. This can be Space-based or Policy-based. Policy-Based Retention: If the Online retention policy is created today for 90 days, the policy will move the data from online storage to archive storage, on the 91st day from the day of creating the policy. So if a new retention policy is created today, 91 days after today, the logs from today will be moved to the archive. Any older logs from today will be left untouched and will be de-moved to archive storage (if it is configured) or will be purged (if no archive storage is configured) based on the Space-Based policy, and will be subjected to FIFO. To enforce the Policy-Based Retention of old data, use the script EnforceRetentionPolicy. A date is specified as the number of days since the UNIX epoch, 1970-01-01. Below is the command to convert the date to epoch date. Command Format (Replace MM/DD/YYYY with actual value): echo $(($(date --utc --date="MM/DD/YYYY" +%s)/86400)) Example to convert 27th Feb 2022 to epoch date: echo $(($(date --utc --date="02/27/2022" +%s)/86400)) Output: 19050 Example: EnforceRetentionPolicy 16230,16233-16235. Enforce retention policies in eventDB on 6/8/2014 and from 6/11/2014 to 6/13/2014. Note 1: The script should be run from the admin user. Run 'su admin' before running the script. Note 2: Additionally, the script depends on the data size being enforced. Run the script for one day and gradually increase the range as needed. It is not recommended to use the script only for a few GBs of data. To enforce the retention policies for a large amount of data (many days), use the script many times for a few days each time. Archive Retention Policies cannot be fewer days than Online ones. In this example, data will be on Online storage for 5 days. On 6th day, data will be moved to Archive storage and they will remain there for 10 days. So, the data will be saved in total for 15 days as defined on the Archive Retention Policy: 5 days on Online storage and 10 on archive one: Space-based retention: Space-based retention is based on two thresholds defined in the /opt/phoenix/config/phoenix_config.txt file on the Supervisor node. For online data: [BEGIN phDataPurger] - Online_low_space_action_threshold_GB (default 10GB). - Online_low_space_warning_threshold_GB (default 20GB). [END] For archive data: [BEGIN phDataPurger] archive_low_space_action_threshold_GB=10. archive_low_space_warning_threshold_GB=20. [END] When Online disk space reaches the low threshold (online_low_space_action_threshold_GB) value, events are archived (if the archive directory is set) or purged. This operation continues until the Online disk space reaches the online_low_space_warning_threshold_GB value. This check is done hourly. It is possible to change these parameters to suit the environment and they will be preserved after the upgrade. To change these values, then change them on the Supervisor CLI and restart the phDataManager and phDataPurger processes: killall -9 phDataManager killall -9 phDataPurger For more information, refer to the FortiSIEM documentation: Retention Database Documentation