AWS IPSEC on BGP routing (how to control traffic preference for each tunnel?)

[FortDoog]

Good day guys.

I have the following setup.

• 200F with dual WAN
• 4 tunnels to AWS (the usual IPSEC with 2 tunnels).
• Probably, next year, hopefully, I will have AWS Direct Connect.

Let´s begin with the IPSEC tunnels first:

I´m having issues with the ECMP on the AWS TGW, what I need is just failover between the tunnels, but I want to setup the priority in which they are used under normal conditions, like in the following graph:

I want to influence the traffic (inbound and outbound) so it has the Tunnels in this order of preference:

1. Main_Tunnel01
2. Main_Tunnel02
3. Secondary_Tunnel01
4. Secondary_Tunnel02

Or

1. Main IPSEC
2. Secondary IPSEC

Right now, what I´m having (with ECMP disabled) is that I´m loosing traffic partially if I lose a Tunnel. If I enable ECMP, I get traffic through all the tunnels, and I do not want that.

I found the Technical Tip : Difference between asymmetric routing and auxiliary sessions., I will testing that also, but right now I´m confussed with this AWS documentation: 

• https://aws.amazon.com/blogs/networking-and-content-delivery/creating-active-passive-bgp-connections....
• https://repost.aws/knowledge-center/direct-connect-asymmetric-routing
• https://repost.aws/knowledge-center/direct-connect-bgp-communities

IF I understood correctly the AWS docs, I should use:

• Outbound traffic: Local Preference, if not, then I should use AS_Path
• Inbound traffic: Local Preference, if not, MED, if not, then I should use AS_Path

Meaning that I would need two sets of route maps (right?) they would identical in prefix list (my case), but they will differ regarding the Local Preference, MED and AS_Path.

My questions are:

• do I need to setup Local Preference, MED and AS_Path for AWS IPSEC routing inffluencing (all of them, meaning more route maps)?
• or can I use just one of them? meaning, just two route maps, one with more influence than the other,
• if so, which one should I use (Local Preference, MED or AS_Path)???

Keep in mind that I have to leave the space for the future implementation of the Direct Connect, so, whatever I use, I have to leave it so that in case that the Direct Connect fails, the failover SHOULD be Main IPSEC, if the Main IPSEC fails too, then Secondary IPSEC.

Please, I need guidance, oh Wise People of the Community, Help. (FWI: english is my second language, that´s why I´m getting a little confused, sorry about that).

[BillH_FTNT · Accepted solution]

Hi, I share something about BGP over IPSec from On-prem to Cloud.
- Support that you use only one VDOM. And in each Tunnel, you have a BPP session. then
1. If you want to influence traffic from AWS back to you through the Tunnel you want, you should use BGP AS-Path. You can use ROute-map to Append more AS into the AS path list and advertise through BGP neighbor. Neighbors with less AS number in the AS Path list will be used to send traffic back to you.
2. For traffic from Fortinet to AWS , the easy way to do is Local preference.

Brs/Bill

[FortDoog]

Thanks Bill, 

But, given that I will have Direct Connect in the future, AWS advices against using AS_Path. Have you seen AS_Path work with Direct Connect? (Fast Connect in OCI, or others?)

[BillH_FTNT]

Yeah, they always do the same way that they want customers influence traffic by themself. So, I think when you have Direct connect, there still have 2 cases for you:

1. You have 2 DC circuits for HA or Active /standby. If you want HA using ECMP that is easy one. In case of you want to Active/standby you need to use BGP AS Path for incoming, Local preference or weight for outgoing as I said above.

2. You can use active is DC but standby is ipsec vpn. in that case, you still use bgp attributes to influence traffic as above. 

Now I think you can read about bgp and bgp attributes first, the doc from @hbac is so nice. You can get more links inside that doc too. 

HTH

Brs/Bill

[hbac]

Hi @FortDoog, 

I hope this article will help: https://community.fortinet.com/t5/FortiGate/Technical-Tip-How-to-configure-redundant-IPSEC-tunnel-to...

Regards, 

[FortDoog]

Thanks @hbac , it did help to gather more understanding.

I will try both solutions to see what comes back and reply, so this helps any other member out there.

[FortDoog]

One question I forgot to ask.

Does these settings work with ECMP enabled or not? that on the AWS side.

[FortDoog]

Good day, I  wanted to add a closure to this. 

Me and my team were able to configure the 4 tunnels in cascade disabling the ECMP on the AWS side (that´s what management wanted so, ye).

One thing that came up was the route filtering. Which I think I did not understand the concept well.

With the Prefix List, I found somehting. The subnets I have on AWS are all /16. My UNDERSTANDING was that setting the prefix list to accept /11 will actually bring those /16 networks using it in the following way:

edit "AWS_Brazil_in"
config rule
edit 1
set prefix a.b.c.d 255.224.0.0
set ge 11
unset le

BUT, it did ot work like that, instead, in the log I was seeing that it was denying the routes because of that filter. I found it odd, because I thought that filter meant: allow all routes that are enclosed by /11, being /16 is inside it.

What I DID find out was that I had to setup the filter likes this:

edit "AWS_Brazil_in"
config rule
edit 1
set prefix a.b.c.d 255.224.0.0
set ge 15
set le 25

With that, the firewall filtered the routes correctly. That´s the only thing I found odd, because I wanted to summarize as much as possible to have fewer filters as possible. If anyone can give me a "simple man terms" explanation on how to use it (remember, english is NOT my first language) I would apreciate it.

The rest of the instructions given were applied and worked flawlessly.

[BillH_FTNT]

Hi FortDoog,

What did you configure prefix-list in BGP ? 

[FortDoog]

The AWS subnets. Let me get you a screenshot.

This happened when using the /11:

And then this happened when using the /15 and /25

What I really want is to receive just the /16 subnets. Because the other networks belong to other networks that I do not want.