ADVPN - Dual WAN connectivity on spokes

[Mattmans1]

Hi all,

This is my first post on these forums, so hello to everybody :)

I'm going to start by asking a question i don't expect many people to be able to answer but i hope somebody who is familiar with BGP and ADVPN can crack this one.  I have labbed up the below scenario and its working great.  Hub/spoke topology with direct spoke to spoke connectivity on demand.

http://cookbook.fortinet.com/configuring-advpn-in-fortios-5-4-dynamic-hub-and-spoke-vpns/

I have got abit more adventurous and added a secondary WAN connection to each firewall and added a second round of ADVPN config/VPN's to establish tunnels over the new WAN connection in a bid to achieve ADVPN redundancy should the primary VPN's fail.

The interesting bit is that it does work (kind of) - If i shut the VPN's down on the hub it works, both spokes will speak to the hub via the second VPN tunnel and agree new spoke to spoke connectivity over the secondary connection.  However it does not work if i shut the VPN tunnel down on the spokes themselves, i seem to get a recursive lookup error. 

This is the BGP table showing routes for spoke to spoke flow with both VPN's up.

OFFICE-FG-ADVPN-IBGP # get router info routing-table bgp B 10.1.1.0/24 [200/0] via 172.16.1.1, ADVPN, 00:00:01 B 10.2.2.0/24 [200/0] via 172.16.1.2, ADVPN_0, 00:00:01

After i shut the primary WAN VPN down on the hub, it fails over to use the secondary

OFFICE-FG-ADVPN-IBGP # get router info routing-table bgp B 10.1.1.0/24 [200/0] via 172.16.2.1, ADVPN-MPLS, 00:01:42 B 10.2.2.0/24 [200/0] via 172.16.2.2, ADVPN-MPLS_0, 00:00:01

And this is the issue when all VPN's are up o nthe hub and i shut down a VPN on one of the spokes.

OFFICE-FG-ADVPN-IBGP # get router info routing-table bgp B 10.1.1.0/24 [200/0] via 172.16.2.1, ADVPN-MPLS, 00:00:17 B 10.2.2.0/24 [200/0] via 172.16.1.2 (recursive is directly connected, unknown), 00:00:16

The issue looks to be because the route for 10.2.2.0/24 (other spoke) is been learned from the hub but with the next hop (172.16.1.2) that is routed over the same VPN tunnel i just shutdown.  So there is no way it can use 172.16.1.2 as a next hop as there is a static route saying route 172.16.1.0/24 over the VPN interface which is down.   (this route is required according to the above article)

"This is an important special step for the spokes as they need a summary route that identifies all tunnel IP used over your topology to point towards the ADVPN interface. In our example, we use 10.10.10.0/24 (if our network planning expects less than 255 sites). Be sure to adequately plan this IP range as it needs to be hardcoded in the spokes."

I have logged this with TAC support but doubt they will help me with this.  Does anyone know who to fix my issue?

[Mattmans1]

I have managed to progress this slightly by adding a second route out the VPN interface.  I realized that when i shut the VPN interface down the route dropped out, so i added a higher cost route out the other VPN interface solves the recursive issue.  Now i have an issue with asymmetric routing.  If i drop the same tunnel on the spoke 2 as i did on spoke 3 it works.  

edit 3 set dst 172.16.1.0 255.255.255.0 set device "ADVPN

edit 5 set dst 172.16.2.0 255.255.255.0 set device "ADVPN-MPLS"

edit 7 set dst 172.16.1.0 255.255.255.0 set distance 250 set device "ADVPN-MPLS"

edit 8 set dst 172.16.2.0 255.255.255.0 set distance 250 set device "ADVPN"

[tom_maz · Accepted solution]

Hi Mattmans

I was wondering how you ever made out with this. We have about 20 sites around the country that are connected via traditional IPSEC, and I'd like to investigate ADVPN for the full-mesh functionality. All locations have dual-WAN, so we'd want the ability for each office to be able to connect to another office using any combination of wan1 & wan2:

wan1 to wan1

wan1 to wan2

wan2 to wan1

wan2 to wan2

I'm not finding any good documentation on this, and don't have access to any good lab setup at this time to experiment. So, if you ever got your setup completely working, please share any details.

Thanks

[alex_buric]

tom.maz wrote:

Hi Mattmans

I was wondering how you ever made out with this. We have about 20 sites around the country that are connected via traditional IPSEC, and I'd like to investigate ADVPN for the full-mesh functionality. All locations have dual-WAN, so we'd want the ability for each office to be able to connect to another office using any combination of wan1 & wan2:

wan1 to wan1

wan1 to wan2

wan2 to wan1

wan2 to wan2

I'm not finding any good documentation on this, and don't have access to any good lab setup at this time to experiment. So, if you ever got your setup completely working, please share any details.

Thanks

We have some problems. There are 43 sites with two WAN connections on each. We have full-mesh network and it is awful huge thing to support it.

There is good technology in Cisco (Dynamic Multipoint VPN (DMVPN) using GRE over IPSec) but transfer all our network to Cisco devices will be very expensive and no wise.

Does anybody have expirience and want to share any documentations?

[majid]

Dear I have just read your post, actually I have almost the identical problem as that of yours, but it seems you have already the solution of my problem, and you can guide me in this, since you have achieved redundancy on hub site that is what i am trying to achieve in my topology, kindly share the config with me so that I can see where I am making mistake. 

I have setup ADVPN in my current toplogy using the following cookbook recipie ( http://cookbook.fortinet....ic-hub-and-spoke-vpns/ ) I have 21 spoke sites and one hub site. on hub site I have two ISP links with static global routable IPs. on spoke site I have only one ISP with static global routable IP. in current setup all the spokes are making IPSEC with only one global routable IP on Hub site and its working fine. now I want that all my spokes also make redundant IPSEC tunnel with other ISP on the hub site so that if primary ISP goes down on the hub site then secondary or backup IPSEC tunnel takes the charge. now I have tried some configuration and let me share some results.

I have configured hub site and two spoke sites with redundant IPSEC, now what is happening, when I intentionally make the primary ISP link down on the hub site, it does not shift automatically on the backup ISP.

but as I clear the pre-existing primary IPSEC tunnel from the spoke site using the following command "diagnose vpn ike gateway clear Primary ADVPN (Primary tunnel)", then spoke makes IPSEC with other ISP on the Hub site, so it is not automatically falling back to backup ISP.

spokes are making ipsec tunnel with the ISP-1 WAN-1 on hub site. as I unplugged the ISP1 cable physically from the WAN1, spokes do not make IPSEC tunnel with the ISP-2 WAN-2 on hub site, spokes keep on sending IKE phase 1 parameters to the ISP-1 WAN1 on hub site even after I unplugged the WAN1 cable from hub site. But as I remove pre-existing primary IPSEC tunnel from the spoke site using the following command "diagnose vpn ike gateway clear" then spokes make IPESC tunnel with the ISP-2 WAN-2 on hub site.

[thanhletrung85]

Dear Mattmans1 and all,

Did you resolve your problem?

I'm stuck with ADVPN with dual WAN interface on 1 hub. I got recursive issue.

Could you please share your config?

Thanks and regards

[routetehpacketz]

I was having a hard time getting BGP to establish and I think the issue is with the tunnel interface IPs. I found they would not respond to each other as I could see incoming traffic when running a sniffer, but there would be no response (e.g icmp echo but no reply). I could confirm the VPN tunnels were active because if I flushed them the SAs immediately renegotiate.

I think the problem is in the steps for the Hub Fortigate. On each Spoke, the guide directs you to enter a static route for the /24 used for the tunnel interfaces, but this step is missing from the Hub. If I try to ping a Spoke's tunnel IP from the Hub, I get "sendto failed". If I ping from Spoke to Hub, I just lose all of the packets.

I tested this by adding a static route for the /24 used by the tunnel IPs and pointed at the ADVPN interface just like the guide directs you to do for the Spokes. I was then able to ping between these interfaces.

With this in place, BGP finally came up after a little while. However, I could only get BGP to establish between the Hub and one Spoke. Traffic between these two is passing fine, but when I tried to pass traffic from the subnet behind Spoke #2 and the Hub, BGP wouldn't establish. The tunnel is up, but the tunnel interfaces aren't talking to each other. A sniffer on the Hub show that the pings from Spoke #2's tunnel interface IP are making it to the Hub and the Hub is replying, but those replies don't make it to Spoke #2. If I run the sniffer on Spoke #2, I only see the outbound pings from Spoke #2.

Any ideas?

Edit: So I failed to mention that I'm testing this on 6.0.1. I found this 6.0 ADVPN guide, but I'm getting a little confused on some of the config examples.

For example, when it directs to set the remote-ip on the tunnel interface it doesn't include a subnet mask, but my Fortigates force me to provide a subnet mask:

FTG3_Lab (ADVPN) # set remote-ip 
<class_ip&net_netmask> IP address and subnet mask (syntax = 1.1.1.1/24).

I'm not sure if the mask should be a /32 like I have assigned on my tunnel interfaces or a /24 like the static route on the spokes, but neither work.

Also, this guide directs you to use set auto-discovery-psk in the phase 1 config on the hub, but fails to mention this attribute of the phase 1 config isn't available unless you set auth-mode signature. This then requires a certificate attribute to be set. The only mentioning of the word certificate in this entire guide is "if certificates are confgured...". Sorry that I'm mostly complaining at this point, but when a technical guide gives partial config examples and later words them as optional, it makes it difficult to know if you done things right.

To expand upon this (and I literally just noticed it now while typing this edit), the 5.4 guide has comments on the bottom acknowledging command differences between 5.4, 5.6, and 6.0. This acknowledgement directs readers to KB article FD39360

. I am going to read the PDF linked in this KB article and see if I can get this working in 6.0. I apologize if it sounds like I'm complaining. I'm really just trying to help someone else out if they experience the same issues I have been with setting up ADVPN.

[tom_maz]

Hey routetehpacketz,

After reading your post, I just came across something that might be of interest. If you go to this KB article:

http://kb.fortinet.com/kb/documentLink.do?externalID=FD39360

Search for the term "blackhole." Note that they are suggesting to add the static routes for the virtual IPSEC interfaces, but they also add blackhole routes. Maybe that is because BGP won't advertise any routes unless they are already inserted into the routing table.

Anyway, I'm looking into all of this because we want to implement ADVPN. However, we want to do it with redundant WAN interfaces on each hub and spoke. I've yet to find any good docs on how to do that.

[MarcAbaya]

There's already  cookbook recipe on how to do this:

https://cookbook.fortinet.com/configuring-advpn-fortios-5-4-redundant-hubs-expert/

It was created almost 2 years ago. Anybody tried this?

[MBR]

Hi Mattmans,

So did you finally fixed the recursive issue?

if yes, could you please share it with us?