FortiAnalyzer Log Retention

[xinger]

I'm somewhat new to Fortinet products so I seem to need some help that is not obvious to me in the documentation. 

We've had FortiAnalyzer for about 6 months.  Logs from various Fortigate devices are forwarded here.  But the oldest logs I can find are only 38 days old.  I'd like to have at least 90 days of logs available.  The dashboard shows 8% Hard Disk Usage and has stayed at that level for quite a while so it certainly has space to spare.  How can I increase our Hard Drive Usage, that is, retain logs longer? File Management?  None of the File Management settings are set.  Would I achieve my goal by changing the File Management's "Automatically Delete" setting for "Device log files" to "older than 3 months"? 

[scao_FTNT · Accepted solution]

Hi, xinger:

From your description, seems your per device quota not properly configured

device quota, basically controls how many raw logs and how much SQL database size the device can keep, so please do a check for "Device Manage" - right side device list, select a device and right click menu "Edit" and there is a config option for "Disk Log Quota (min. 100MB)"

Thanks

Simon

[scao_FTNT · Accepted solution]

we may have a new quota design in 5.4 so easier for user to config / manage the system quota / disk space

Thanks

Simon

[scao_FTNT · Accepted solution]

in 5.0/5.2, device quota is mainly for its SQL db size and raw log files (also for archive files if available) and for device raw log files, you can see from log view / log browse and each raw log file has from/to so you know the oldest raw log file is for when and for SQL, it is ADOM based (for all ADOM devices), so in log view, for SQL entry, the oldest log time is for all device and possible that a device, has older raw log files but can not find in SQL db which may because SQL entry has been removed by its quota check for SQL which is ADOM based, it is using a quota which is 60% for all its devices quota (for example, your ADOM has 5 devices which has default 1GB quota for each device, and then ADOM SQL table will take 3GB as its quota) there is a CLI "diagnose log device" which will give you more details for configured quota and real usage Thanks Simon

[scao_FTNT · Accepted solution]

Hi, xinger:

From your description, seems your per device quota not properly configured

device quota, basically controls how many raw logs and how much SQL database size the device can keep, so please do a check for "Device Manage" - right side device list, select a device and right click menu "Edit" and there is a config option for "Disk Log Quota (min. 100MB)"

Thanks

Simon

[xinger]

Will do.  Not what I answer wanted to hear, but I trust it is the answer I needed to hear.  Thanks Simon!  It would have been more convenient from my viewpoint to have a macro level setting to tell FortiAnalyzer to use 80% of its available disk space (or keep 90 days of logs) regardless of where logs are coming from.  But I get it.  Thanks for the quick response!

[scao_FTNT · Accepted solution]

we may have a new quota design in 5.4 so easier for user to config / manage the system quota / disk space

Thanks

Simon

[xinger]

scao_FTNT wrote:

we may have a new quota design in 5.4 so easier for user to config / manage the system quota / disk space

Thanks for that hope.  In the meantime, is there any easy way to know how many days of logs are being currently stored for a device?  I want to solve this equation:  if 25% of the quota is used in n days, then I should expect the log to hold about 4n days of logs.  But how do I know how many days n actually is?  Thanks again.

[scao_FTNT · Accepted solution]

in 5.0/5.2, device quota is mainly for its SQL db size and raw log files (also for archive files if available) and for device raw log files, you can see from log view / log browse and each raw log file has from/to so you know the oldest raw log file is for when and for SQL, it is ADOM based (for all ADOM devices), so in log view, for SQL entry, the oldest log time is for all device and possible that a device, has older raw log files but can not find in SQL db which may because SQL entry has been removed by its quota check for SQL which is ADOM based, it is using a quota which is 60% for all its devices quota (for example, your ADOM has 5 devices which has default 1GB quota for each device, and then ADOM SQL table will take 3GB as its quota) there is a CLI "diagnose log device" which will give you more details for configured quota and real usage Thanks Simon

[AndreaSoliva · Accepted solution]

Hi

I'm not 100% sure if I understand the problem here exact but give me a hint to show you what in my mind is important to calculate the FAZ storage which I do always on every installation:

1. Each FortiGate brings to the FAZ a amoutn of Logs. Example: If you configure a 60D on really full logging you have about 45 - 55 MB Logs (every log is enabled). What you have to keep in mind is that additional to this calculation of Log you have to add 25% Storage to this calculated log. This addtional storage is used for the DB running on the FAZ or also called overhead. Finally this means that in my example for a 60D you have to calculate 45 - 55 MB + 25% Storage for FAZ.

2. As second step you have to configure "rolling" of logs which means to have rolling ones a week is not a good idea on a FAZ with many devices because this is very resource intensive for CPU and RAM on FAZ. This means finally if you are "rolling" logs for a 60D on daily base you have to look at the realtime log 45 - 55 MB in the RAM. If you do not "roll" on daily base instead you roll weekly you have if you search in the log 45 - 55 MB X 7 in the RAM. From this point of view "roll" the logs on daily base.

3. As third step think about "how long I will have the logs on realtime available on the FAZ"? This means even you roll the logs on daily base there are still available on realtime under "log browse".

4. As next step think about "how long I will have to logs available on the FAZ at all" which means at which time I will delte the logs at all on the FAZ. This means also backup your logs on daily base after rolling and even you delete the logs on the FAZ at all they are available on the backup server in case of. This means if you have after 3 month a issue and you need to look at the logs which are not anymore available on the FAZ you can go to the backup server and load the log/s back to the FAZ over the gui without problems and search within this log etc.

Finally for me the answer are as following:

- Do daily based rolling (every log whatever it is will be at 00:00 rolled)

- After daily based rolling backup the file to example FTP server and zip BUT DO NOT DELETE the logs on FAZ

- After 2 Month delete the logs on FAZ at all (still available on the FTP server to be loaeded back to FAZ in case of)

- The local log of FAZ I do the same which means daily rolling and backup to FTP as after 2 Month deleting the logs

- In case of disaster I will loose at all "only the daily running logs". Restore can be done from backup server as bulk.

- Backup the config of FAZ on weekly base

Result everything is backup exept the "customized Reports". This can be done by command line if you like. At least to configure this what is mentioned under "Finally" you have to use following:

# Automatic Backup FAZ # config system backup all-settings set status enable set server [IP FTP Server] set user [User FTP Server] set directory [Dir FTP Server /example] set week_days [Day of backup example "monday"] set time [Time of backup example "06:00:00"] set protocol [Define FTP as "ftp"] set passwd [FTP Password "mypassword"] unset crptpasswd end

# Automatic Upload "Local" Log FAZ on-schedule # config system locallog disk setting set status enable set severity notification set upload enable set uploadip [FTP Server IP] set server-type [Use Protocoll "FTP "] set uploadport [FTP port 21] set uploaduser [FTP user] set uploadpass [FTP Password "mypassword"] set uploaddir [Dir FTP Server /example] set uploadtype event set uploadzip enable set uploadsched disable set upload-delete-files disable set max-log-file-size 500 set roll-schedule daily set roll-time 00:00 set diskfull overwrite set log-disk-full-percentage 80 set upload-time [Set upload Time "01:30"] end

NOTE Set the "upload-time" after 00:00 which is used for rolling logs. This takes some time!

# Automatic Upload "Device" Log FAZ on-schedule # config system log settings config rolling-regular set file-size 500 set upload enable set when daily set days mon set del-files disable set directory [Dir FTP Server /example] set gzip-format enable set hour 0 set ip [FTP Server IP] set log-format native set min 0 set password [FTP Password "mypassword"] set server-type [Use Protocoll "FTP "] set upload-hour  1 set upload-mode backup set upload-trigger on-schedule set username [FTP user] end end

# Auto Delete Files FAZ #

config system auto-delete

config dlp-files-auto-deletion

set status enable

set value 2

set when months

end

config quarantine-files-auto-deletion

set status enable

set value 2

set when months

end

config log-auto-deletion

set status enable

set value 2

set when months

end

config report-auto-deletion

set status enable

set value 6

set when months

end

end

# Manual Backup FAZ # execute backup all-settings ftp [FTP ServerIP] [Filename like "SYS_FAZ-VM0000013345_faz_[DateTime].dat [FTP user] [FTP password]

NOTE With this command you can also backup logs, reports etc.!

If you have more than you FAZ and you would like to forward the "Local" logs in realtime to another FAZ you can use NEW for 5.2.2 following:

# Forward "Local Device Log" FAZ to FortiAnalyzer

config system locallog fortianalyzer setting

set status realtime

set server-ip [IP of FAZ]

set secure-connection enable

set severity information

end

NOTE This command can also be used for FMG to forward the "Local" logs to a FAZ.

Finally this what is used here I use also for the FMG because the commands and the rolling etc. is exactly the same on a FMG as for the FAZ.

I do always the same is my system and in this way I have not to trouble about going out of space. If I'm reaching my storage capacity meaning because of too many device I have to add storage to a FAZ VM base (standard 80 GB) which is possible until 200GB which means following has to be done:

          # execute shutdown           The system will be halted.           Do you want to continue? (y/n) y

After the FAZ is down add to the instance a second disc with the needed capacity (VM base not more as 200GB at all). After adding the additional disk to the instance start the FAZ again. After the FAZ started at all do following:

          "Show all disk not in use available"

          # execute lvm extend           Disk(s) currently not in use:           disk02      32.0(GB)

         "Add the new disk not in use"

          # execute lvm extend disk02           This operation will need to reboot the system.           Do you want to continue? (y/n) y

If you like to add more as one disk use: # execute lvm extend disk02 disk03 disk04

After the FAZ is new started you can check the new disk:

          # execute lvm info           disk01 In use           80.0(GB)           disk02 In use           32.0(GB)           disk03 not present           disk04 not present           disk05 not present           disk06 not present           disk07 not present           disk08 not present           disk09 not present           disk10 not present           disk11 not present           disk12 not present

The addtional capacity will be also shown under:

# get system status

Thats it and it works for every FAZ instance if you think about as mentioned here.

hope this helps

have fun

Andrea

[tobys]

Awesome post - thanks Andrea!