Skip to main content
FortDoog
Explorer
October 18, 2023
Solved

AWS IPSEC on BGP routing (how to control traffic preference for each tunnel?)

  • October 18, 2023
  • 26 replies
  • 20767 views

Good day guys.

I have the following setup.

  • 200F with dual WAN
  • 4 tunnels to AWS (the usual IPSEC with 2 tunnels).
  • Probably, next year, hopefully, I will have AWS Direct Connect.

Let´s begin with the IPSEC tunnels first:

I´m having issues with the ECMP on the AWS TGW, what I need is just failover between the tunnels, but I want to setup the priority in which they are used under normal conditions, like in the following graph:

 

AWS issue.jpg

 

I want to influence the traffic (inbound and outbound) so it has the Tunnels in this order of preference:

 

  1. Main_Tunnel01
  2. Main_Tunnel02
  3. Secondary_Tunnel01
  4. Secondary_Tunnel02

Or

  1. Main IPSEC
  2. Secondary IPSEC

Right now, what I´m having (with ECMP disabled) is that I´m loosing traffic partially if I lose a Tunnel. If I enable ECMP, I get traffic through all the tunnels, and I do not want that.

 

I found the Technical Tip : Difference between asymmetric routing and auxiliary sessions., I will testing that also, but right now I´m confussed with this AWS documentation: 

 

IF I understood correctly the AWS docs, I should use:

  • Outbound traffic: Local Preference, if not, then I should use AS_Path
  • Inbound traffic: Local Preference, if not, MED, if not, then I should use AS_Path

Meaning that I would need two sets of route maps (right?) they would identical in prefix list (my case), but they will differ regarding the Local Preference, MED and AS_Path.

 

My questions are:

  • do I need to setup Local Preference, MED and AS_Path for AWS IPSEC routing inffluencing (all of them, meaning more route maps)?
  • or can I use just one of them? meaning, just two route maps, one with more influence than the other,
  • if so, which one should I use (Local Preference, MED or AS_Path)???

 

Keep in mind that I have to leave the space for the future implementation of the Direct Connect, so, whatever I use, I have to leave it so that in case that the Direct Connect fails, the failover SHOULD be Main IPSEC, if the Main IPSEC fails too, then Secondary IPSEC.

 

Please, I need guidance, oh Wise People of the Community, Help. (FWI: english is my second language, that´s why I´m getting a little confused, sorry about that).

Best answer by BillH_FTNT

Hi, I share something about BGP over IPSec from On-prem to Cloud.
- Support that you use only one VDOM. And in each Tunnel, you have a BPP session. then
1. If you want to influence traffic from AWS back to you through the Tunnel you want, you should use BGP AS-Path. You can use ROute-map to Append more AS into the AS path list and advertise through BGP neighbor. Neighbors with less AS number in the AS Path list will be used to send traffic back to you.
2. For traffic from Fortinet to AWS , the easy way to do is Local preference.

Brs/Bill

26 replies

BillH_FTNT
Staff
Staff
October 18, 2023

Hi, I share something about BGP over IPSec from On-prem to Cloud.
- Support that you use only one VDOM. And in each Tunnel, you have a BPP session. then
1. If you want to influence traffic from AWS back to you through the Tunnel you want, you should use BGP AS-Path. You can use ROute-map to Append more AS into the AS path list and advertise through BGP neighbor. Neighbors with less AS number in the AS Path list will be used to send traffic back to you.
2. For traffic from Fortinet to AWS , the easy way to do is Local preference.

Brs/Bill

FortDoog
FortDoogAuthor
Explorer
October 18, 2023

Thanks Bill, 

But, given that I will have Direct Connect in the future, AWS advices against using AS_Path. Have you seen AS_Path work with Direct Connect? (Fast Connect in OCI, or others?)

BillH_FTNT
Staff
Staff
October 18, 2023

Yeah, they always do the same way that they want customers influence traffic by themself. So, I think when you have Direct connect, there still have 2 cases for you:

1. You have 2 DC circuits for HA or Active /standby. If you want HA using ECMP that is easy one. In case of you want to Active/standby you need to use BGP AS Path for incoming, Local preference or weight for outgoing as I said above.

2. You can use active is DC but standby is ipsec vpn. in that case, you still use bgp attributes to influence traffic as above. 

Now I think you can read about bgp and bgp attributes first, the doc from @hbac is so nice. You can get more links inside that doc too. 

HTH

Brs/Bill

Hong_FTNT
Staff & Editor
Staff & Editor
October 18, 2023
FortDoog
FortDoogAuthor
Explorer
October 18, 2023

Thanks @hbac , it did help to gather more understanding.

I will try both solutions to see what comes back and reply, so this helps any other member out there.

FortDoog
FortDoogAuthor
Explorer
October 24, 2023

One question I forgot to ask.

 

Does these settings work with ECMP enabled or not? that on the AWS side.

FortDoog
FortDoogAuthor
Explorer
January 31, 2024

Good day, I  wanted to add a closure to this. 

Me and my team were able to configure the 4 tunnels in cascade disabling the ECMP on the AWS side (that´s what management wanted so, ye).

 

One thing that came up was the route filtering. Which I think I did not understand the concept well.

 

With the Prefix List, I found somehting. The subnets I have on AWS are all /16. My UNDERSTANDING was that setting the prefix list to accept /11 will actually bring those /16 networks using it in the following way:

edit "AWS_Brazil_in"
config rule
edit 1
set prefix a.b.c.d 255.224.0.0
set ge 11
unset le

 

BUT, it did ot work like that, instead, in the log I was seeing that it was denying the routes because of that filter. I found it odd, because I thought that filter meant: allow all routes that are enclosed by /11, being /16 is inside it.

 

What I DID find out was that I had to setup the filter likes this:

edit "AWS_Brazil_in"
config rule
edit 1
set prefix a.b.c.d 255.224.0.0
set ge 15
set le 25

With that, the firewall filtered the routes correctly. That´s the only thing I found odd, because I wanted to summarize as much as possible to have fewer filters as possible. If anyone can give me a "simple man terms" explanation on how to use it (remember, english is NOT my first language) I would apreciate it.

 

The rest of the instructions given were applied and worked flawlessly.

BillH_FTNT
Staff
Staff
January 31, 2024

Hi FortDoog,

 

What did you configure prefix-list in BGP ? 

 

FortDoog
FortDoogAuthor
Explorer
February 5, 2024

The AWS subnets. Let me get you a screenshot.

 

This happened when using the /11:

 

image (1).png

 

And then this happened when using the /15 and /25

image (2).png

 

What I really want is to receive just the /16 subnets. Because the other networks belong to other networks that I do not want.

Toshi_Esumi
SuperUser
SuperUser
February 6, 2024

I'm not sure if I understand your requirement. But the prefix-list matches what you specify and let matched ones come in (or go out) and take them into BGP table (or send them to neighbors) "as is". It wouldn't summarize to /16 when a /27 matches the prefix-list and place the /16 into the local BGP table.

If you, instead, want to summarize routes when the FGT advertise, like multile /24s into one /16, to its neighbors, you can use "aggregation" below.
https://community.fortinet.com/t5/FortiGate/Technical-Tip-How-to-implement-BGP-route-summary-aggregation-on/ta-p/196101

But I don't know if AWS side support route aggretation on its end. You can probably ask AWS support if they support or not. My guess is they would support though.

Toshi

FortDoog
FortDoogAuthor
Explorer
February 7, 2024

Good day Toshi.

No, no, I don´t the device to summarize. What I want is for the firewall to accept those subnets that are /16, but using the prefix filter set in /11 (roughly speaking).

 

What I assumed, is that if I set the prefix list-in to /11 for certain subnet, it will accept ALL that is enclosed by the /11 subnet, meaning the /16, by setting the ge to 11.

But from what I saw, the prefix filter does not work like that. So I wanted to know how to do this correctly.

Toshi_Esumi
SuperUser
SuperUser
February 7, 2024

Then try this:
config router prefix-list
  edit "AWS-slash16s"
    config rule
      edit 1
        set prefix 10.160.0.0 255.248.0.0
        set ge 16
        set le 16
      next
      edit 2
        set prefix 10.64.0.0 255.240.0.0
        set ge 16
        set le 16
      next
    end
  next
end

Toshi

BillH_FTNT
Staff
Staff
February 7, 2024

Hi,

Hi FortDoog,
I think what you planned to do with BGP prefixes was right. However, to work with the prefix list, I believe you should change it slightly.
I think the common rule for prefix-list is "len <GE <=LE"
In your case the prefix you configured a.b.c.d 255.224.0.0 ==> the len is 8+7 =15 (225 use 8 bits, 224 use 7 bits)
==> It should be "ge 15"
1. you can do "ge 15" only.
2. or ge 15 le 25
(In your first case, ge 11 is an invalid one, I think)
HTH
Bill

FortDoog
FortDoogAuthor
Explorer
February 7, 2024

If I understood correctly, the subnet mask should match the filter ge then? (remember, simple terms, since this is all in english)

BillH_FTNT
Staff
Staff
February 7, 2024

 a.b.c.d 255.224.0.0

=> Subnet mask len is 15

So, the "ge" must be greater 15. 

 

Toshi_Esumi
SuperUser
SuperUser
February 7, 2024

ge and le length need to be longer than the network length. The error message says it all

FGTxxxxx1 (1) # next
Invalid prefix range -- make sure: len < ge-value <= le-valueobject check operator error, -650, discard the setting
Command fail. Return code 1

I think this is common with other routers like Cisco. Or FTNT copied this spec from Cisco.

Toshi

FortDoog
FortDoogAuthor
Explorer
February 8, 2024

Ok, now I understand.

 

For posterity or anyone reading this. The filters (ge and or le) MUST be different than the subnet you are trying to filter.

 

Like Bill and Toshi said:

 

  • set prefix a.b.c.d /20 for example.

The ge would NOT be 20, neither le.

It has to be:

  • in case of ge, 19. So it will take all network greater than 19, those will be, 20, 21, and so on.
  • Or the reverse of it,  in case of le, 21. Is kinda like a math equation.
Toshi_Esumi
SuperUser
SuperUser
February 8, 2024

I think 19 and 20 is prohibited if the network length is /20. Otherwise the error message the system gave me is wrong. Did you test and confirm as the fact?

 

Toshi

FortDoog
FortDoogAuthor
Explorer
February 9, 2024

That´s the thing, it did NOT give me an error. Now, there is one big thing I believe I did not point out, and it was critital to say: the firmware is 6.x.x. (I don´t recall the exact number right now). So most probably, the whole problem was due an old forgotten bug probably. and most most probably corrected in newer versions.

 

I know, you will ask why in 2024 we are running a 200F firewall on 6.x.x firmware... ask my management about that... I decided just to do my best with what I got / what I am allowed to do.