Fortigate VDOM logging

[Rafa123_]

Hello.

We had a enviroment with some Fortigates of many models. The whole enviroment is in 5.2.x. We are facing a problem with VDOM logging. I need to keep in this fortigates 10 days of logs beyond the logs that are sented to fortianalyzer.

The issue is: I'm able to keep this logs while no vdom are configured but if we create a VDOM I cannot  use the full disk capacity to keep this logs.

Any thoughts about how to solve this ?

Thankyou.

[emnoc]

Qs:

Have you looked at vdom log override?What's happening in  with or without vdom ? What drives you at 10days? Can you use  upload ( compress or not  )?

example ( multivdom )

config log disk setting     set status enable     set ips-archive disable

    set upload enable

    set uploaddir  log

    set roll-schedule weekly     set roll-day sunday     set roll-time 00:00

    set uploadtype traffic event

    set uploadpass "xxxxxxxxxxxxxx"

    set uploaduser  logrollup

    set uploadip  x.x.x.x

    set uploadzip enable

end

FWIW;

Trying to compute a 10day max on disk storage is very hard to calculate, hard on the disk , and provides no retention if the unit actually fails.....imho

rollups is  the ideal method and again imho and experience.

Ken

[Rafa123_]

emnoc wrote:

Qs:

Have you looked at vdom log override?What's happening in  with or without vdom ? What drives you at 10days? Can you use  upload ( compress or not  )?

example ( multivdom )

config log disk setting    set status enable    set ips-archive disable

    set upload enable

    set uploaddir  log

    set roll-schedule weekly    set roll-day sunday    set roll-time 00:00

    set uploadtype traffic event

    set uploadpass "xxxxxxxxxxxxxx"

    set uploaduser  logrollup

    set uploadip  x.x.x.x

    set uploadzip enable

end

FWIW;

Trying to compute a 10day max on disk storage is very hard to calculate, hard on the disk , and provides no retention if the unit actually fails.....imho

rollups is  the ideal method and again imho and experience.

Ken

Hello. Thanks for your help.

I need to keep at least 10 days, for contractual reasons.

My problem is the Fortigate starts to subscribing logs before it reaches 10 days, and before the disk is full either. I do not know what is limmiting the logs.

I will check the config logdisk setting.

[emnoc]

do a cli cmd  "show fulll sys log setting" let's ensure no qutoa or other weird cfg.

e.g

show full-configuration  log disk  setting

and

show full-configuration  log memory  global-setting

and it probably will not hurt to check  misled statisics

diag test  application  miglog 6

diag test  application  miglog 16

Pay attention to the last value with  miglogs #16

e.g 

VDOM log disk usage:

  root: 235045768B/3605M

  GEFRA01: 34407844558B/3605M  <-----

  GEBER01: 0B/3605M

  SOCO:  950514964B/3605M  

I think that might shed light on your max value again and why your  not hitting what you suspect. The best command to see full max values

cli cmd  dia sys  logdisk usage 

Total HD usage: 59707MB/60093MB

Total HD logging space: 18028MB

I don't know what means can be  execute to change the size since it depends on hardware but can set quotas 

Quote are easily to be detected

e.g 

FSOCPUPCHIIL (global) $ dia sys logdisk  quota

             type    quota(MB)    usage(MB)

 ================ ============ ============

----- vdom cst1 -----

log disk quota 0 MB

        disk log:            0        32813

     dlp archive:            0            0

          report:            0           10

      quarantine:            0            0

     ips archive:            0            0

----- vdom NEXTTECH -----

log disk quota 0 MB

        disk log:            0          253

     dlp archive:            0            0

          report:            0            0

      quarantine:            0            0

     ips archive:            0            0

----- vdom VDMZ -----

log disk quota 0 MB

        disk log:            0         1039

     dlp archive:            0            0

          report:            0         1771

      quarantine:            0            0

     ips archive:            0            0

----- vdom WAN  -----

log disk quota 0 MB

        disk log:            0            0

     dlp archive:            0            0

          report:            0            0

      quarantine:            0            0

     ips archive:            0            0

----- vdom root -----

log disk quota 0 MB

        disk log:            0          2224

     dlp archive:            0            0

          report:            0            0

      quarantine:            0            0

     ips archive:            0            0

rollups are  great and still the best method imho.

Ken

[Rafa123_]

Hi Ken.

This are the info I get with the commands you provide me:

# diagnose sys logdisk quota              type    quota(MB)    usage(MB)  ================ ============ ============ ----- vdom Transparent ----- log disk quota 0 MB         disk log:            0          385      dlp archive:            0            0           report:            0         1583       quarantine:            0            0      ips archive:            0            0 ----- vdom root ----- log disk quota 0 MB         disk log:            0           24      dlp archive:            0            0           report:            0            4       quarantine:            0            0      ips archive:            0            0

# diagnose sys logdisk usage Total HD usage: 2224MB/15025MB Total HD logging space: 4507MB HD logging space usage for vdom "Transparent": 1968MB/2253MB HD logging space usage for vdom "root": 28MB/2253MB

# diagnose test application miglogd 6 mem=5098576, disk=5097191, alert=0, alarm=0, sys=0, faz=5093218, webt=0, fds=0 interface-missed=56 Queue: maxium=17100 current:0 global log dev statistics: faz 0: (logs) sent=5098558, failed=0, cached=0, relayed=0 faz 0: (packets) sent=1776301, failed=0, cached=0, relayed=0

# diagnose test application miglogd 16 VDOM log disk usage:   root: 25775950B/2253M   Transparent: 403680568B/2253M

# show full-configuration log memory global-setting config log memory global-setting     set max-size 65536     set full-first-warning-threshold 75     set full-second-warning-threshold 90     set full-final-warning-threshold 95 end

Note that diag sys logdisk usage shows that total space for HD is 15025MB. And HD logging space is only 4507MB, splited to the two VDOMs. How can I upgrade the HD Logging Space?

Thankyou.

[emnoc]

Okay so let's step back and understand what's really happen,  if you add the following value "left of the /" for the vdoms you get the total.

  root: 25775950B/2253M   Transparent: 403680568B/2253M

That gives you  4507MB max value since the logdisk overall set at   4507MB. SO no matter what you do you have 4507M.

Also as you add vdoms, they "equal" eat out of the total  HDlogging-size that available.

Let me demo on a FGT140D

This unit has the following with one vdom { root };

LAB-FGT-2 (global) # diagnose sys logdisk usage Total HD usage: 260MB/30050MB Total HD logging space: 9015MB HD logging space usage for vdom "root": 125MB/9015MB

Now here's what happens when we look at disk values with just root;

AGAIN miglod output

LAB-FGT-2 (global) # diagnose test application miglogd 16 2017-08-23 11:03:12 VDOM log disk usage: 2017-08-23 11:03:12   root: 8549274B/9015M

and if I add  root1 and root2 to  vdom;

LAB-FGT-2 (global) # diagnose test application miglogd 16 2017-08-23 11:03:59 VDOM log disk usage: 2017-08-23 11:03:59   root: 8549274B/3005M 2017-08-23 11:03:59   root1: 0B/3005M    <----my new vdom 2017-08-23 11:03:59   root2: 0B/3005M    <---- my new vdom

You see what happens, the total value is still  9015, but now it's shared equally against vd=root, root1, root2

Now to get around that you will need to find out if you can reformat the  log disk,  and see if it set the  max value to a value higher than 4507M.  I would start with a logdisk reformat BUT read the warning below

config global

execute  formatlogdisk

NOTE: it will erases all logs and could take a few minutes, and needs a reboot so be aware of the above and rolloff your logs if you need archives

Here's my  FGT140D after a execute  formatlogdisk

LAB-FGT-1 (global) # diagnose test application miglogd 16 VDOM log disk usage:   root: 79609B/9015M LAB-FGT-1 (global) #  diagnose sys logdisk usage Total HD usage: 266MB/30050MB Total HD logging space: 9015MB HD logging space usage for vdom "root": 133MB/9015MB NO CHANGE!

It made no difference, we are still stuck at  9015MB. So now we will go into cli-cmd and change the  wanopt, since with enable it will  pre-allocate space on the disk for wan-opt cache

config global

   config wanopt storage

      delete Internal

   end

config wanopt storage     edit "Internal"         set size 768         set webcache-storage-percentage 2     next end Again NO CHANGE

diagnose sys logdisk usage Total HD usage: 174MB/30050MB Total HD logging space: 9015MB HD logging space usage for vdom "root": 2MB/9015MB

Again  execute formatlog disk and see if the values , if not you need to open a ticket with FTNT but I believe the answer is going to be no and to use FAZ, FortiCLoud or Syslogd

What you can do to  stretch out the logdisk

1: reduce what you log

2: if you what traffic logs only than drop all others

3: change the severity  level ( notice vrs  info )

4: rollup logs ( again  that's what I would do )

If you execute a lot of log-writes to  the  internal disk, you  will wear out the disk  &  reduce the effectiveness of the device imo ;)

e.g

"when you  break the disk, you have no logs and now need a RMA  event , circumventing your 10day on disk archival strategy"

SYSLOG and a external SATA  drive appliance, or vmare or forticloud is cheaper ;) I've been working with fortigate conservely since v2.8 and you will never convince me to log to disk if your serious about logging.

Ken