100+ VDOM Design Best Practice

[cuneyt]

Hi Everyone,

We have a Fortigate 3950B and 100+ VDOMs on it. Dont have a fortimanager. I want to ban some malicuos IPs (100+) for all VDOMs and these IPs may change every week. Solutions:

1- Go with ACLs on router. (I dont want this solution for some reasons)

2-  Define all IPs on all VDOMs and create a firewall deny policy on all VDOMS. (If you dont die before you finish :))

3- Create a VDOM. Define all IPs and create a firewall deny policy on this VDOM. Route all other VDOMs' trafic to this VDOM.

I have not tried 3rd solution yet. Is this a good solution? Why not?

Is there another solution to handle this situation?  

Regards

[emnoc · Accepted solution]

Yes all three proposals will work  and I love your comment about  the  "don't die" if you do a massive  fw.policy  and in 100+ vdoms fw.policies ;)

Here's a few better options that you should explore and not ignore

1>

If you have a router doing  a dynamic protocol and  the fgt have a means to learn of these routes , do a  route distribute with  the /32 netmask and route to  a blackhole { null  route }. This will allow you  to drop the  /32 hosts in one single swipe. You can  do this static-route on the same  ISP-ACL router that you  mention b4. OSPF/BGP works great for this or do a static route on internet external WAN1 and enable  uRPF

{ cisco }  ip route 1.1.1.1 255.255.255.266 null0

2>

Now if you  have no access to the router or a different team controls it, you can quickly deploy  the address into the fwpolicys across the 100+ vdom with some  scripting

3>

or better yet  use the restAPI. You  could  batch job this and upload the configuration into a pre-defined fw.policy.ID ( read more )

In one of my recent rule we have 10 vdoms with public access. We maintain a fwpolicy  with  object-group. The policy was at that top of our list and used a deny and bidir, the addrgroup is only modified with the bad actor and we place the single ip.addr into the one  address.group. In this example it's called ITBLACKLISTS, which comprised of  numerous host firewall.address or firewall.addrgrps

NOTE: research the max  values in a addrgrp, I believe it 1024 or 1500 entries per group on bigger units , you should be fine in a 3950 unless your blacklist over 1500 addresses. So if you think you  might have   more than 1500   stack address.groups

e.g

config firewall addrgrp

    edit ITBLACKLISTS

      set member BHGRP1 BHGRP2 BHGRP3  OCT182018 SEPT32018

end

config firewall policy    edit 20        set uuid c5d8882e-5437-51e6-649d-8f30869eac8e        set srcintf "PORT1"        set dstintf "WAN1"        set srcaddr "ITBLACKLISTS        set dstaddr "ANY"        set action deny        set tag SOC PREMITERSECURITY        set schedule "always"        set service "ALL"        set logtraffic all        set comments " DROP IT LIKE IT'S HOT "    next   edit 21        set uuid ab0165a0-1073-51e6-ad98-74a3bcce8032        set dstintf "PORT1"        set srcintf "WAN1"        set dstaddr "ITBLACKLISTS        set srcaddr "ANY"        set action deny        set schedule "always"        set service "ALL"        set tag  SOC        set logtraffic all        set comments " DROP IT LIKE IT'S HOT "    next  fwpolicy.id 20/21 was used in ALL of our  vdoms firewall.policies and ITBLACKLISTS is  created in all  vdoms. So we only needed to modified the ITBLACKLISTS address group. We  even  crafted new groups and appended them into the parent group and set the name of the group based on DATE or  TICKET/CASE to make it easier for auditing or setting some value to know when we free the address out of the "jail" so to speak.  Another option is to null route on the  firewall lew of routing it to another vdom YMMV  pick one and test them all but your option#3 is best or  mine null route on the ISP uplinks. That latter keeps the  back actor from even touch the firewall, doesn't create a log, or consume a session or uses any cpu.  Ken 

[neonbit · Accepted solution]

If you go with option 3 it will add a little more complexity, and it will also mean that all the WAN sessions will be doubled on the firewall as the WAN VDOM will now have keep sessions for each VDOM sending traffic through it.

To be honest I would look at getting a FortiManager if you have the budget for it. You could then create a global header policy and push it to all VDOMs so it will be the first policy on each of them. This way you make a change once on the global level and it will be pushed to all VDOMs.

[emnoc · Accepted solution]

I can't   answer how to  teach you scripting , but  you will have to explore your team  access knowledge . You can do this with pyhton/perl/pwr-shell/etc...

We ( in past roles )  had a tool teams that would draft up our scripts ,  and they would do this by setting  expect or  restAPI calls to add or remove address objects in the  fortigate. So we would seed  a line  by line file of our bad-actors in a text file 

e.g ( entries pulled by our SIEM or some other device logs  like ; FEYE ,F-Secure,etc.... or you can use a BCL subscription services )

1.1.1.1

2.1.122.2

9.1.1.3

1.2.3.4

This text file would be  ran upon events or attacks and if the address didn't exist, the restAI will add the  object as an firewall.address and then append this to the firewall.addrgrp  for the ITBLACKLIST

I would suggest you test this in a lab or lab-vdom b4 running this in a  full production. Once you work out the  bugs you can  easily  update 1k plus  entries  in  a matter of a few seconds and across multiple vdoms. Granted I nevere worked in any appliance with more than 50+ vdoms ;)

e.g ( a restAPI post to add address )

curl -b cookie.txt -H "Content-Type: application/json" -H "X-CSRFTOKEN blablahblah "   -d '{ "name"; "$NAME",  "subnet" : "$NAME/32" }' -X POST https://x.x.x.x/api/v2/cmdb/firewall/address

We would  do  "for loop" for the name which == the ipv4.address, this was also  added as  subnet with  the mask 32.

Always make a config backup b4 and after and diff to ensure the changes where as planned. Also last advise , make sure you place safe guards

e.g

[ul]

  do not add a mask smaller than /32

  do not add your own subnet or address in your own subnet [/ul]

I like to mention this since I worked a case where a forcepointNGFW user  blacklisted his own subnet  causing a priority 1 outage and they didn't follow the above 2 rules :)

Ken Felix

[Alexis_G · Accepted solution]

According to feedback, (you dont like to block on router), I would definetely choose third. I m doing this all the time and works fine (normally i do this during new projects (not during production)). BUT !!!

you need to prepare the vdom links, the firewall rules and last to have ready the new routes for the intervdom routing. Because otherwise you can have a downtime in order to migrate.

Its good also if you could prepare a script to configure all routing at once.

[emnoc · Accepted solution]

Yes all three proposals will work  and I love your comment about  the  "don't die" if you do a massive  fw.policy  and in 100+ vdoms fw.policies ;)

Here's a few better options that you should explore and not ignore

1>

If you have a router doing  a dynamic protocol and  the fgt have a means to learn of these routes , do a  route distribute with  the /32 netmask and route to  a blackhole { null  route }. This will allow you  to drop the  /32 hosts in one single swipe. You can  do this static-route on the same  ISP-ACL router that you  mention b4. OSPF/BGP works great for this or do a static route on internet external WAN1 and enable  uRPF

{ cisco }  ip route 1.1.1.1 255.255.255.266 null0

2>

Now if you  have no access to the router or a different team controls it, you can quickly deploy  the address into the fwpolicys across the 100+ vdom with some  scripting

3>

or better yet  use the restAPI. You  could  batch job this and upload the configuration into a pre-defined fw.policy.ID ( read more )

In one of my recent rule we have 10 vdoms with public access. We maintain a fwpolicy  with  object-group. The policy was at that top of our list and used a deny and bidir, the addrgroup is only modified with the bad actor and we place the single ip.addr into the one  address.group. In this example it's called ITBLACKLISTS, which comprised of  numerous host firewall.address or firewall.addrgrps

NOTE: research the max  values in a addrgrp, I believe it 1024 or 1500 entries per group on bigger units , you should be fine in a 3950 unless your blacklist over 1500 addresses. So if you think you  might have   more than 1500   stack address.groups

e.g

config firewall addrgrp

    edit ITBLACKLISTS

      set member BHGRP1 BHGRP2 BHGRP3  OCT182018 SEPT32018

end

config firewall policy    edit 20        set uuid c5d8882e-5437-51e6-649d-8f30869eac8e        set srcintf "PORT1"        set dstintf "WAN1"        set srcaddr "ITBLACKLISTS        set dstaddr "ANY"        set action deny        set tag SOC PREMITERSECURITY        set schedule "always"        set service "ALL"        set logtraffic all        set comments " DROP IT LIKE IT'S HOT "    next   edit 21        set uuid ab0165a0-1073-51e6-ad98-74a3bcce8032        set dstintf "PORT1"        set srcintf "WAN1"        set dstaddr "ITBLACKLISTS        set srcaddr "ANY"        set action deny        set schedule "always"        set service "ALL"        set tag  SOC        set logtraffic all        set comments " DROP IT LIKE IT'S HOT "    next  fwpolicy.id 20/21 was used in ALL of our  vdoms firewall.policies and ITBLACKLISTS is  created in all  vdoms. So we only needed to modified the ITBLACKLISTS address group. We  even  crafted new groups and appended them into the parent group and set the name of the group based on DATE or  TICKET/CASE to make it easier for auditing or setting some value to know when we free the address out of the "jail" so to speak.  Another option is to null route on the  firewall lew of routing it to another vdom YMMV  pick one and test them all but your option#3 is best or  mine null route on the ISP uplinks. That latter keeps the  back actor from even touch the firewall, doesn't create a log, or consume a session or uses any cpu.  Ken 

[cuneyt]

Thanks a lot @emnoc for the very very comprehensive reply.

Your 3rd solution is good but before, i must create firewall policy on 100+ VDOMS. So it is hard to setup. By the way routing all VDOMs to a one VDOM is also difficult :)

2nd solution would be good but i have problems scripting on fortigate. Do you know how can i execute script on all VDOMS recursively (i mean without naming them statically)?

Your route soluiton is also quite good but i don't want to manage routers. It is like ACLs solution.

[neonbit · Accepted solution]

If you go with option 3 it will add a little more complexity, and it will also mean that all the WAN sessions will be doubled on the firewall as the WAN VDOM will now have keep sessions for each VDOM sending traffic through it.

To be honest I would look at getting a FortiManager if you have the budget for it. You could then create a global header policy and push it to all VDOMs so it will be the first policy on each of them. This way you make a change once on the global level and it will be pushed to all VDOMs.

[cuneyt]

Thanks for the reply.

"..WAN sessions will be doubled.." is a good point to consider. In 100+ VDOMs this will make sense.

[emnoc · Accepted solution]

I can't   answer how to  teach you scripting , but  you will have to explore your team  access knowledge . You can do this with pyhton/perl/pwr-shell/etc...

We ( in past roles )  had a tool teams that would draft up our scripts ,  and they would do this by setting  expect or  restAPI calls to add or remove address objects in the  fortigate. So we would seed  a line  by line file of our bad-actors in a text file 

e.g ( entries pulled by our SIEM or some other device logs  like ; FEYE ,F-Secure,etc.... or you can use a BCL subscription services )

1.1.1.1

2.1.122.2

9.1.1.3

1.2.3.4

This text file would be  ran upon events or attacks and if the address didn't exist, the restAI will add the  object as an firewall.address and then append this to the firewall.addrgrp  for the ITBLACKLIST

I would suggest you test this in a lab or lab-vdom b4 running this in a  full production. Once you work out the  bugs you can  easily  update 1k plus  entries  in  a matter of a few seconds and across multiple vdoms. Granted I nevere worked in any appliance with more than 50+ vdoms ;)

e.g ( a restAPI post to add address )

curl -b cookie.txt -H "Content-Type: application/json" -H "X-CSRFTOKEN blablahblah "   -d '{ "name"; "$NAME",  "subnet" : "$NAME/32" }' -X POST https://x.x.x.x/api/v2/cmdb/firewall/address

We would  do  "for loop" for the name which == the ipv4.address, this was also  added as  subnet with  the mask 32.

Always make a config backup b4 and after and diff to ensure the changes where as planned. Also last advise , make sure you place safe guards

e.g

[ul]

  do not add a mask smaller than /32

  do not add your own subnet or address in your own subnet [/ul]

I like to mention this since I worked a case where a forcepointNGFW user  blacklisted his own subnet  causing a priority 1 outage and they didn't follow the above 2 rules :)

Ken Felix

[tanr]

Ken,

I'm curious.  In these blacklist building scripts, did you

- Just rebuild the list of IPs from the SIEM reports?

- Have the list keep getting added to till it was gigantic?

- Put in a method for blacklists to age out? 

I ask because it seems like there could be some problems with any of these options, though aging out seems best.

If the list gets rebuilt, since you were blackholing and therefore not logging the current crop of bad IPs they wouldn't show up on a rebuilt list (depending on your report time frame) and thus would get allowed again.

Just adding to the list makes it too big, plus it will continue to block IPs that might have been dynamic and are now valid.

Aging out the blacklist (or re-generating it from a SIEM report with a large enough time window) seems like the best solution, though I would think you'd want to also use a static list of definite bad actors that doesn't age out.

Any thoughts on this?   

As I move from painfully building a list by hand to scripting it myself seeing what others are doing is very helpful.

I really appreciate your pointer to make sure your own subnet can't get automatically added to the list!  That seems glaringly obvious until you do it to yourself...  ;)

[emnoc]

- Just rebuild the list of IPs from the SIEM reports?

Yes, via SIEM , IDS,   and other means for extraction or  defined  Botnetcontrollists ( aka BCL )

- Have the list keep getting added to till it was gigantic?

yes, see my warning on max  values in  firewal;

- Put in a method for blacklists to age out?

 

Yes,  if you a  ADDR.GROUP and set a DATENAME   or case, you  do a schedule sweep at some  set time.

e.g  maybe every  5 business days, you  delete the older defined  addr.group

[Alexis_G · Accepted solution]

According to feedback, (you dont like to block on router), I would definetely choose third. I m doing this all the time and works fine (normally i do this during new projects (not during production)). BUT !!!

you need to prepare the vdom links, the firewall rules and last to have ready the new routes for the intervdom routing. Because otherwise you can have a downtime in order to migrate.

Its good also if you could prepare a script to configure all routing at once.

[cuneyt]

Hi @jklapas,

Thanks for the reply. This is what I want to learn but previously @neonbit said that in this configuration "..all the WAN sessions will be doubled on the firewall..". Is this correct? 

I serached for fortinet documentation again and find some useful information. In this link:

http://help.fortinet.com/fos50hlp/50/index.html#page/FortiOS%25205.0%2520Help/inter-VDOM.180.15.html

It is about Meshed VDOM configuration with inter-VDOM links. This is what I want actually. I don't need full meshed configuration (in 100+ VDOMs it is too hard to manage) but partial mesh is the exact solution for me. But again @neonbit warning confusing me.

Are there any disadvantages of this topology?

What are your opinions @emnoc, @tanr, @neonbit?