New Member

Question

DUAL HUB SETUP FOR ADVPN and SDWAN FOR BRANCH OFFICES

Forum|Forum|2 years ago
May 25, 2023
7 replies
23404 views

Hi All,

I am planning to setup the below. Using FGT 7.2.4, If you see any bug also please highlight.

Spoke 1 & 2 -------> Hub 1 & Spoke 1 & 2 -------> Hub 2 (Using ADVPN -iBGP)

-------------------

Spoke1 & 2 need to be connected with Hub 1 and Hub2 (Both hubs are running with separate services)

Spoke 1 & Spoke 2 need to be communicated with each other.

Hub 1 and Hub 2 need to be communicated.

All the tunnel interfaces (From Branch to hub 1 and hub 2) in the same Overlay Zone.

--------------------------------

To achieve the above setup using the below protocols.

Between Spoke 1 & 2 ----> Hub 1 & 2 using iBGP (all Spoke and hub locations in the same iBGP AS number)

Between Hub 1 & 2 also using eBGP (or may be iBGP).

------------------------------------

Do you see any challenges on the above setup?

If you have any suggestion then please let me know.

saneeshpv_FTNT

Staff

Spoke1 & 2 need to be connected with Hub 1 and Hub2 (Both hubs are running with separate services)

- Each spoke will have routes learned from HUB1 and HUB2 respectively. I am assuming the same routes are not learned by Spoke from each HUB as you have separate services in each location with different IP allocation. If not, please be careful with the route advertisement and route configuration in the spoke to route the traffic correctly to each HUB.

Spoke 1 & Spoke 2 need to be communicated with each other.

- iBGP route reflector client need to be enabled under neighbor-group in HUB to allow route learned from one iBGP neighbor to another iBGP neighbor.

Hub 1 and Hub 2 can be interconencted using eBGP

Please check this article and will be really helpful with your deployments.

https://community.fortinet.com/t5/FortiGate/Technical-Tip-Fortinet-Auto-Discovery-VPN-ADVPN/ta-p/195698

TT_DUAuthor

New Member

Thanks for your response!!!

Query 1:-

Spoke1 & 2 need to be connected with Hub 1 and Hub2 (Both hubs are running with separate services)

- Each spoke will have routes learned from HUB1 and HUB2 respectively. I am assuming the same routes are not learned by Spoke from each HUB as you have separate services in each location with different IP allocation. If not, please be careful with the route advertisement and route configuration in the spoke to route the traffic correctly to each HUB.

REP: So in this case Can I go with the below setup, because ADVPN (shortcut tunnel) should be enabled across all sites to be communicated with each other and should not be dependent on Hub locations.

Spoke1<------Tunnel over iBGP----->Hub1 (RR1 - for Spoke1)

Spoke2 <------Tunnel over iBGP----->Hub2 (RR1 - for Spoke2)

Hub1<------Tunnel over iBGP----------->Hub2

Query2:

Hub 1 and Hub 2 can be interconencted using eBGP

REP: Assuming if I go with the eBGP setup then shortcut tunnel not formed across the spoke locations. it always depends on the hub location. Correct me If I am wrong.

akristof

Staff

Hello,

I recommend to check this KB:

https://community.fortinet.com/t5/FortiGate/Technical-Tip-Fortinet-Auto-Discovery-VPN-ADVPN/ta-p/195698

It has most of the information.

So in your case, if you will have eBGP (over Ipsec) between HUBs and you enable auto-discovery-forwarder, shortcuts will be formed - very similar to dual-region deployment. I think more simple configuration would be to run OSPF between HUBs, to have connectivity. You don't need to have eBGP/iBGP explicitly to make sure that the spokes will have always routes to all spokes via both HUBs. You just need to adjust routing so HUB1 will be primary, HUB2 will be secondary and whenever HUB1 or HUB1's overlays will be unavailable, HUB2 will be used.

TT_DUAuthor

New Member

Below is the diagram for example, please suggest.

saneeshpv_FTNT

Staff

This is bit complex setup and I would advise you take proper help from our PS team to design this setup with multiple HUB and multiple WAN links.

But here are some suggestions if you wish to try yourself.

- You will have 4 overlay tunnel from each spoke (2 to each HUB across wan1 and wan2).

- Spoke to Spoke shortcuts can be formed using one overlay network per HUB by restricting HUB not to advertise additional path for the same prefixes using command (set additional-path disable).

- IPsec tunnel between HUB1 and HUB2 to allow ADVPN shortcut messages, and eBGP between HUB1 and HUB2 (if not same AS) and iBGP if same AS. This will help you form shortcuts across all spokes and Hubs

TT_DUAuthor

New Member

Thanks!!

To allow shortcut messages between hub 1 and 2 via iBGP, we need to add the command auto discovery forwarder , right?

Also may I know what will happen if we didn't restrict the advertisement from all overlay tunnels.

If any suggestion then please advise.

TT_DUAuthor

New Member

I am able to achieve the below scenarios with above setup mentioned in the diagram.

-----------------------Scenario1 --------------------------------------

SPOKE1 -OL2 down & SPOKE2-OL3 down ----> SPOKE1-OL3 is able to communicate with SPOKE2-OL2

SPOKE1 -OL3 down & SPOKE2-OL2 down ----> SPOKE1-OL2 is able to communicate with SPOKE2-OL3

------------------------------------Scenario2------------------------

SPOKE1 -OL2 down & SPOKE2-OL2 & OL3 are UP ----> SPOKE1-OL3 is not able to communicate with SPOKE2-OL2 or OL3

Can anyone having any idea?

Please suggest what could be the issue, also share the config suggestion.

akristof

Staff

Hi,

First, verify routing. On each Spoke:

- verify that they have routing-table entry for all remaining overlays. Also, spoke1 should have route for OL2 subnet, when its own OL2 is down. It can be static route via OL1 and OL3 as a backup.

- On HUB verify that you have all routes via all available overlays

- Check how Spokes are sending traffic. Which overlay they are selecting? If traffic is leaving OL3, check if HUB is forwarding traffic also over OL3 (you should have basic policy routes on HUB to have overlay stickiness). I would recommend to enable debug flow on all 3 devices at the same time, send 1 ping and you will see where is the problem.

TT_DUAuthor

New Member

I could see the below flow, no reply from SPOKE 2 to HUB because its learning the SPOKE1 subnet via OL3.

Please suggest what could the issue. Should I create basic policy routes on HUB to have overlay stickiness?

Source: 10.101.6.1 - spoke1 subnet

Destination: 10.102.5.1 - spoke 2 subnet

SPOKE1:

2023-05-31 14:35:04.393600 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:35:05.409061 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:35:06.419088 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:35:07.439066 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:35:08.449063 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:35:09.459066 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request

id=65308 trace_id=562 func=print_pkt_detail line=5868 msg="vd-root:0 received a packet(proto=1, 10.101.6.1:38408->10.102.5.1:2048) tun_i
d=0.0.0.0 from local. type=8, code=0, id=38408, seq=10."
id=65308 trace_id=562 func=resolve_ip_tuple_fast line=5956 msg="Find an existing session, id-00244e1d, original direction"
id=65308 trace_id=562 func=ipsecdev_hard_start_xmit line=669 msg="enter IPSec interface INET_OL3_AZ_UAE, tun_id=0.0.0.0"
id=65308 trace_id=562 func=_do_ipsecdev_hard_start_xmit line=229 msg="output to IPSec tunnel INET_OL3_AZ_UAE vrf 0"

HUB:

2023-05-31 14:39:28.585619 INET_OL3_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:28.585676 INET_OL2_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:29.545577 INET_OL3_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:29.545627 INET_OL2_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:30.485620 INET_OL3_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request

id=65308 trace_id=545 func=print_pkt_detail line=5939 msg="vd-root:0 received a packet(proto=1, 10.101.6.1:38408->10.102.5.1:2048) tun_i
d=10.12.14.4 from INET_OL3_AZ_UAE. type=8, code=0, id=38408, seq=4."
id=65308 trace_id=545 func=resolve_ip_tuple_fast line=6027 msg="Find an existing session, id-038d1662, original direction"
id=65308 trace_id=545 func=npu_handle_session44 line=1245 msg="Trying to offloading session from INET_OL3_AZ_UAE to INET_OL2_AZ_UAE, skb
.npu_flag=00000400 ses.state=04010204 ses.npu_state=0x00000100"
id=65308 trace_id=545 func=fw_forward_dirty_handler line=416 msg="state=04010204, state2=00000001, npu_state=00000100"
id=65308 trace_id=545 func=ipsecdev_hard_start_xmit line=662 msg="enter IPSec interface INET_OL2_AZ_UAE, tun_id=0.0.0.0"
id=65308 trace_id=545 func=_do_ipsecdev_hard_start_xmit line=222 msg="output to IPSec tunnel INET_OL2_AZ_UAE_14 vrf 0"

SPOKE2:

2023-05-31 14:38:58.159998 INET_OL2_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:38:59.120018 INET_OL2_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:00.060323 INET_OL2_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:01.079959 INET_OL2_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:02.319901 INET_OL2_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request

id=65308 trace_id=1112 func=print_pkt_detail line=5875 msg="vd-root:0 received a packet(proto=1, 10.101.6.1:38408->10.102.5.1:2048) tun_
id=x.x.x.x from INET_OL2_AZ_UAE. type=8, code=0, id=38408, seq=6."
id=65308 trace_id=1112 func=init_ip_session_common line=6049 msg="allocate a new session-012bb40a, tun_id=x.x.x.x"
id=65308 trace_id=1112 func=ip_route_input_slow line=2267 msg="reverse path check fail, drop"

TT_DUAuthor

New Member

I could see the below flow, no reply from SPOKE 2 to HUB because its learning the SPOKE1 subnet via OL3.

Please suggest what could the issue. Should I create basic policy routes on HUB to have overlay stickiness?

---------------------------------------------------

Source: 10.101.6.1 - spoke1 subnet

Destination: 10.102.5.1 - spoke 2 subnet

---------------------------------------------------------------------------------------

SPOKE1:

2023-05-31 14:35:04.393600 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:35:05.409061 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:35:06.419088 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:35:07.439066 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:35:08.449063 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:35:09.459066 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request

id=65308 trace_id=562 func=print_pkt_detail line=5868 msg="vd-root:0 received a packet(proto=1, 10.101.6.1:38408->10.102.5.1:2048) tun_i
d=0.0.0.0 from local. type=8, code=0, id=38408, seq=10."
id=65308 trace_id=562 func=resolve_ip_tuple_fast line=5956 msg="Find an existing session, id-00244e1d, original direction"
id=65308 trace_id=562 func=ipsecdev_hard_start_xmit line=669 msg="enter IPSec interface INET_OL3_AZ_UAE, tun_id=0.0.0.0"
id=65308 trace_id=562 func=_do_ipsecdev_hard_start_xmit line=229 msg="output to IPSec tunnel INET_OL3_AZ_UAE vrf 0"

HUB:

2023-05-31 14:39:28.585619 INET_OL3_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:28.585676 INET_OL2_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:29.545577 INET_OL3_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:29.545627 INET_OL2_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:30.485620 INET_OL3_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request

id=65308 trace_id=545 func=print_pkt_detail line=5939 msg="vd-root:0 received a packet(proto=1, 10.101.6.1:38408->10.102.5.1:2048) tun_i
d=10.12.14.4 from INET_OL3_AZ_UAE. type=8, code=0, id=38408, seq=4."
id=65308 trace_id=545 func=resolve_ip_tuple_fast line=6027 msg="Find an existing session, id-038d1662, original direction"
id=65308 trace_id=545 func=npu_handle_session44 line=1245 msg="Trying to offloading session from INET_OL3_AZ_UAE to INET_OL2_AZ_UAE, skb
.npu_flag=00000400 ses.state=04010204 ses.npu_state=0x00000100"
id=65308 trace_id=545 func=fw_forward_dirty_handler line=416 msg="state=04010204, state2=00000001, npu_state=00000100"
id=65308 trace_id=545 func=ipsecdev_hard_start_xmit line=662 msg="enter IPSec interface INET_OL2_AZ_UAE, tun_id=0.0.0.0"
id=65308 trace_id=545 func=_do_ipsecdev_hard_start_xmit line=222 msg="output to IPSec tunnel INET_OL2_AZ_UAE_14 vrf 0"

SPOKE2:

2023-05-31 14:38:58.159998 INET_OL2_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:38:59.120018 INET_OL2_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:00.060323 INET_OL2_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:01.079959 INET_OL2_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:02.319901 INET_OL2_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request

id=65308 trace_id=1112 func=print_pkt_detail line=5875 msg="vd-root:0 received a packet(proto=1, 10.101.6.1:38408->10.102.5.1:2048) tun_
id=20.233.48.101 from INET_OL2_AZ_UAE. type=8, code=0, id=38408, seq=6."
id=65308 trace_id=1112 func=init_ip_session_common line=6049 msg="allocate a new session-012bb40a, tun_id=20.233.48.101"
id=65308 trace_id=1112 func=ip_route_input_slow line=2267 msg="reverse path check fail, drop"

TT_DUAuthor

New Member

I could see the below flow, no reply from SPOKE 2 to HUB because its learning the SPOKE1 subnet via OL3.

Please suggest what could the issue. Should I create basic policy routes on HUB to have overlay stickiness?

Source: 10.101.6.1 - spoke1 subnet

Destination: 10.102.5.1 - spoke 2 subnets

SPOKE1:

2023-05-31 14:35:04.393600 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:35:05.409061 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:35:06.419088 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:35:07.439066 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request

id=65308 trace_id=562 func=print_pkt_detail line=5868 msg="vd-root:0 received a packet(proto=1, 10.101.6.1:38408->10.102.5.1:2048) tun_i
d=0.0.0.0 from local. type=8, code=0, id=38408, seq=10."
id=65308 trace_id=562 func=resolve_ip_tuple_fast line=5956 msg="Find an existing session, id-00244e1d, original direction"
id=65308 trace_id=562 func=ipsecdev_hard_start_xmit line=669 msg="enter IPSec interface INET_OL3_AZ_UAE, tun_id=0.0.0.0"
id=65308 trace_id=562 func=_do_ipsecdev_hard_start_xmit line=229 msg="output to IPSec tunnel INET_OL3_AZ_UAE vrf 0"

HUB:

2023-05-31 14:39:28.585619 INET_OL3_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:28.585676 INET_OL2_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:29.545577 INET_OL3_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:29.545627 INET_OL2_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo

id=65308 trace_id=545 func=print_pkt_detail line=5939 msg="vd-root:0 received a packet(proto=1, 10.101.6.1:38408->10.102.5.1:2048) tun_i
d=10.12.14.4 from INET_OL3_AZ_UAE. type=8, code=0, id=38408, seq=4."
id=65308 trace_id=545 func=resolve_ip_tuple_fast line=6027 msg="Find an existing session, id-038d1662, original direction"
id=65308 trace_id=545 func=npu_handle_session44 line=1245 msg="Trying to offloading session from INET_OL3_AZ_UAE to INET_OL2_AZ_UAE, skb
.npu_flag=00000400 ses.state=04010204 ses.npu_state=0x00000100"
id=65308 trace_id=545 func=fw_forward_dirty_handler line=416 msg="state=04010204, state2=00000001, npu_state=00000100"
id=65308 trace_id=545 func=ipsecdev_hard_start_xmit line=662 msg="enter IPSec interface INET_OL2_AZ_UAE, tun_id=0.0.0.0"
id=65308 trace_id=545 func=_do_ipsecdev_hard_start_xmit line=222 msg="output to IPSec tunnel INET_OL2_AZ_UAE_14 vrf 0"

SPOKE2:

2023-05-31 14:38:58.159998 INET_OL2_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:38:59.120018 INET_OL2_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:00.060323 INET_OL2_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request

id=65308 trace_id=1112 func=print_pkt_detail line=5875 msg="vd-root:0 received a packet(proto=1, 10.101.6.1:38408->10.102.5.1:2048) tun_
id=20.233.48.101 from INET_OL2_AZ_UAE. type=8, code=0, id=38408, seq=6."
id=65308 trace_id=1112 func=init_ip_session_common line=6049 msg="allocate a new session-012bb40a, tun_id=20.233.48.101"
id=65308 trace_id=1112 func=ip_route_input_slow line=2267 msg="reverse path check fail, drop

@akristof @knagaraju

akristof

Staff

reverse path check fail, drop - so routing on spoke2 is incorrect. You are using iBGP correct?

Share with me, in files - you can attach them, the following - from all 3 devices:

show router bgp

show vpn ipsec phase1-interface

show system interface

diag ip address list

get router info routing-table all

TT_DUAuthor

New Member

Even after enable the ibgpmultipath command , the issue is looks to be the same. It's still getting reverse path check fail error. Please suggest.

TT_DUAuthor

New Member

Can someone help to mitigate this issue.

anyone having any idea?

freddelm

Explorer

I would be nice if FNT put a full complete guide with working configs.

They have this guide for multiple hubs but is missing key information

https://docs.fortinet.com/document/fortimanager/7.2.0/multiple-datacenters-for-enterprise-primary-primary/503190/introduction

Sign up

Login with SSO

Login to the community

Login with SSO

Scanning file for viruses.

This file cannot be downloaded