Technical Tip: How to Maximize data retention for Threat Hunting Repository

[dmeeker]

Description	This article describes how to maximize the data retention time for the Threat Hunting Repository. The Repository space will fill up as part of its regular function.
Scope	5.2+.
Solution	Overview The purpose of Threat Hunting is to supplement an investigation into either a triggered security event or for scheduled Threat Hunting queries for indicators of compromise (IOC). Two preliminary important question should be asked when collecting Threat Hunting data: 1) How long must the data be retained?  2) What type of Threat Hunting information needs to be collected based on the endpoints business usage? When configuring a Threat Hunting Collection profile, all corresponding data will be collected based on the profile. Examples of this are File Read, File Write, Socket Open connections and much more. The more data that is collected, the lower amount of data retention will be available. Summary of Threat Hunting Collection Profiles The below list explains the three out-of-the-box collection profiles and their estimated retention period: 1) Inventory Profile – Data retention is approximately 30 Days. This is the default out-of-the-box profile and will collect basic file level data. 2) Standard Profile – Data retention will reduce to approximately 10 to 15 days. The standard profile will collect more Threat Hunting data and is often used for more sensitive machines. 3) Comprehensive Profile – Data retention will reduce to approximately 7 or less days. This profile is designed for very sensitive machines, such as domain controllers and machines which may be more vulnerable to attackers. It is important to note that the Inventory Profile is designed to retain data for approximately 30 days if assigned to all Collectors. Any deviation of this profile will result in estimations. Recommendation: Fortinet recommends defining Threat Hunting collection exclusions for applications, processes, logs and network activity that is not going to be helpful for an incident response investigation. Examples of this may be a program such as Microsoft Teams, TeamViewer (file operations) and more. Collection exclusions are organization dependent and should be reviewed based on risk appetite. The benefits of implementing collection exclusions are: 1) Reduces noisy and redundant data from being collected and in turn makes an investigation easier. 2)  Increases data retention. 3) Can lead to system performance benefits such as returning data faster for complex queries. Collection Exclusions Configuration This is only for Threat Hunting collection and not system/security events If you export the events in Threat Hunting to csv or are looking at the Threat Hunting Events look for repetitive events. It's normal to see the same process a few times in a second; however, if you see the same process multiple times in a second (like 50 times a second), this is excessive, and you can create an exclusion for it. Please go to Security Settings > Threat Hunting > Collection Exclusions Please review the admin guide on how to Define Collection Exclusions: https://docs.fortinet.com/document/fortiedr/6.2.0/administration-guide/633468/defining-collection-ex...