AdVPN - ibgp multipath

[Julien87]

Hello Community,

I continue my research on advpn and bgp. I would like to enable ibgp multipath so I can add SDWAN rules and split traffic based on link health for branches.

Before the multipath modification, the routes were well learned by my branches, when I activate either ISP on my HUB.

But after applying multipath configuration in HUB and Branches. Only hub routes are visible in route table database for my branches. The route are received by the bgp protocol from neighbor.

I can not use the new feature self-healing because i have 2 branches with 6.4 version.

I have the same topology as multipath link, but with two tunnel for my branches.

Source multipath : Multipath doc fortinet 

Thanks for your help, you can found below config and info bgp.

Julien

>>>Config HUB bgp

config router bgp
set as 65505
set router-id 1.1.1.1
set ibgp-multipath enable
set additional-path enable

set additional-path-select 2
config neighbor-group
edit "advpn"

set capability-default-originate enable
set link-down-failover enable
set remote-as 65505
set additional-path both

set adv-additional-path 2
set route-reflector-client enable
next
end
config neighbor-range
edit 2
set prefix 10.10.0.0 255.255.0.0
set neighbor-group "advpn"
next
end

>>> Branche BGP config

config router bgp
set as 65505
set router-id 1.1.1.2
set ibgp-multipath enable
set additional-path enable

set additional-path-select 2
config neighbor
edit "10.10.100.254"
set advertisement-interval 1
set link-down-failover enable
set soft-reconfiguration enable
set remote-as 65505
set additional-path both
next
edit "10.10.101.254"
set advertisement-interval 1
set link-down-failover enable
set soft-reconfiguration enable
set remote-as 65505
set additional-path both
next
end

>>> routing table for the hub

FortiGate-HUB # get router info routing-table bgp
Routing table for VRF=0
B 10.19.3.0/24 [200/0] via 10.10.100.2 (recursive via hubwan2-ph1 tunnel 10.10.100.2), 15:50:35
[200/0] via 10.10.101.2 (recursive via hub-ph1-s tunnel 10.10.101.2), 15:50:35
B 10.19.30.0/24 [200/0] via 10.10.100.25 (recursive via hubwan2-ph1 tunnel 10.10.100.25), 15:50:35
[200/0] via 10.10.101.25 (recursive via hub-ph1-s tunnel 10.10.101.25), 15:50:35
B 10.19.103.0/24 [200/0] via 10.10.100.2 (recursive via hubwan2-ph1 tunnel 10.10.100.2), 15:50:35
[200/0] via 10.10.101.2 (recursive via hub-ph1-s tunnel 10.10.101.2), 15:50:35
B 10.19.130.0/24 [200/0] via 10.10.100.25 (recursive via hubwan2-ph1 tunnel 10.10.100.25), 15:50:35
[200/0] via 10.10.101.25 (recursive via hub-ph1-s tunnel 10.10.101.25), 15:50:35
B 10.23.1.0/24 [200/0] via 10.10.100.21 (recursive via hubwan2-ph1 tunnel 10.10.100.21), 15:50:34
[200/0] via 10.10.101.21 (recursive via hub-ph1-s tunnel 10.10.101.21), 15:50:34
B 10.23.101.0/24 [200/0] via 10.10.100.21 (recursive via hubwan2-ph1 tunnel 10.10.100.21), 15:50:34
[200/0] via 10.10.101.21 (recursive via hub-ph1-s tunnel 10.10.101.21), 15:50:34

>> routing table for spoke

FortiGate-SPOKE# get router info routing-table bgp
Routing table for VRF=0
B 10.19.1.0/24 [200/0] via 10.10.100.254 (recursive via spk2-ph1 tunnel x.x.x.x), 15:51:22
[200/0] via 10.10.101.254 (recursive via spk2-ph1s tunnel x.x.x.x), 15:51:22
B 10.19.101.0/24 [200/0] via 10.10.100.254 (recursive via spk2-ph1 tunnel x.x.x.x), 15:51:22
[200/0] via 10.10.101.254 (recursive via spk2-ph1s tunnel x.x.x.x), 15:51:22

>>> Route learn from neighbor for branches

FortiGate-SPOKE# get router info bgp neighbors 10.10.100.254 received-routes
VRF 0 BGP table version is 11, local router ID is 1.1.1.2
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight RouteTag Path
*>i10.19.1.0/24     10.10.100.254                 100      0        0 i <1/->
*>i10.19.3.0/24     10.10.101.2                   100      0        0 i <2/->
*>i10.19.3.0/24     10.10.100.2                   100      0        0 i <1/->
*>i10.19.30.0/24    10.10.101.25                  100      0        0 i <2/->
*>i10.19.30.0/24    10.10.100.25                  100      0        0 i <1/->
*>i10.19.101.0/24   10.10.100.254                 100      0        0 i <1/->
*>i10.19.103.0/24   10.10.101.2                   100      0        0 i <2/->
*>i10.19.103.0/24   10.10.100.2                   100      0        0 i <1/->
*>i10.19.130.0/24   10.10.101.25                  100      0        0 i <2/->
*>i10.19.130.0/24   10.10.100.25                  100      0        0 i <1/->
*>i10.23.1.0/24     10.10.101.21                  100      0        0 i <2/->
*>i10.23.101.0/24   10.10.101.21                  100      0        0 i <2/->

>>> Neighbors info in branches

FortiGate-SPOKE # get router info bgp neighbors 10.10.100.254
VRF 0 neighbor table:
BGP neighbor is 10.10.100.254, remote AS 65505, local AS 65505, internal link
BGP version 4, remote router ID x.x.x.x
BGP state = Established, up for 15:53:20
Last read 00:00:20, hold time is 180, keepalive interval is 60 seconds
Configured hold time is 180, keepalive interval is 60 seconds
Neighbor capabilities:
Route refresh: advertised and received (old and new)
Address family IPv4 Unicast: advertised and received
Address family IPv6 Unicast: advertised and received
Received 1292 messages, 3 notifications, 0 in queue
Sent 1240 messages, 1 notifications, 0 in queue
Route refresh request: received 0, sent 1
NLRI treated as withdraw: 0
Minimum time between advertisement runs is 1 seconds

For address family: IPv4 Unicast
BGP table version 11, neighbor version 10
Index 1, Offset 0, Mask 0x2
Additional Path:
Send-mode: advertised, received
Receive-mode: advertised, received
Inbound soft reconfiguration allowed
Community attribute sent to this neighbor (both)
2 accepted prefixes, 2 prefixes in rib
2 announced prefixes

For address family: IPv6 Unicast
BGP table version 1, neighbor version 1
Index 1, Offset 0, Mask 0x2
Community attribute sent to this neighbor (both)
0 accepted prefixes, 0 prefixes in rib
0 announced prefixes

Connections established 6; dropped 5
Local host: 10.10.100.21, Local port: 21863
Foreign host: 10.10.100.254, Foreign port: 179
Egress interface: 15
Nexthop: 10.10.100.21
Nexthop interface: spk2-ph1
Nexthop global: ::j
Nexthop local: ::
BGP connection: non shared network
Last Reset: 15:53:27, due to BGP Notification received
Notification Error Message: (CeaseUnspecified Error Subcode)

[akristof]

Hi,

Can you share with me following output from spoke that has the problem as hostnames are confusing as they are same for HUB and spokes:

get router info routing-table all

get router info routing-table data

get router info bgp network

[Julien87]

Hi Adrian

i have change hostname in my first post with HUB or SPOKE.

>>> HUB

FortiGate-HUB # get router info routing-table all
Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP
O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
* - candidate default

Routing table for VRF=0
S* 0.0.0.0/0 [1/0] via ISP1, port1
[1/0] via ISP2, port2
S 10.10.100.2/32 [15/0] via hubwan2-ph1 tunnel 10.10.100.2
S 10.10.100.21/32 [15/0] via hubwan2-ph1 tunnel 10.10.100.21
S 10.10.100.25/32 [15/0] via hubwan2-ph1 tunnel 10.10.100.25
C 10.10.100.254/32 is directly connected, hubwan2-ph1
S 10.10.101.2/32 [15/0] via hub-ph1-s tunnel 10.10.101.2
S 10.10.101.21/32 [15/0] via hub-ph1-s tunnel 10.10.101.21
S 10.10.101.25/32 [15/0] via hub-ph1-s tunnel 10.10.101.25
C 10.10.101.254/32 is directly connected, hub-ph1-s
C 10.19.1.0/24 is directly connected, port3
B 10.19.3.0/24 [200/0] via 10.10.100.2 (recursive via hubwan2-ph1 tunnel 10.10.100.2), 17:55:14
[200/0] via 10.10.101.2 (recursive via hub-ph1-s tunnel 10.10.101.2), 17:55:14
B 10.19.30.0/24 [200/0] via 10.10.100.25 (recursive via hubwan2-ph1 tunnel 10.10.100.25), 17:55:14
[200/0] via 10.10.101.25 (recursive via hub-ph1-s tunnel 10.10.101.25), 17:55:14
C 10.19.101.0/24 is directly connected, port4
B 10.19.103.0/24 [200/0] via 10.10.100.2 (recursive via hubwan2-ph1 tunnel 10.10.100.2), 17:55:14
[200/0] via 10.10.101.2 (recursive via hub-ph1-s tunnel 10.10.101.2), 17:55:14
B 10.19.130.0/24 [200/0] via 10.10.100.25 (recursive via hubwan2-ph1 tunnel 10.10.100.25), 17:55:14
[200/0] via 10.10.101.25 (recursive via hub-ph1-s tunnel 10.10.101.25), 17:55:14
B 10.23.1.0/24 [200/0] via 10.10.100.21 (recursive via hubwan2-ph1 tunnel 10.10.100.21), 17:55:13
[200/0] via 10.10.101.21 (recursive via hub-ph1-s tunnel 10.10.101.21), 17:55:13
B 10.23.101.0/24 [200/0] via 10.10.100.21 (recursive via hubwan2-ph1 tunnel 10.10.100.21), 17:55:13
[200/0] via 10.10.101.21 (recursive via hub-ph1-s tunnel 10.10.101.21), 17:55:13
C ISP1/26 is directly connected, port1
C ISP2/26 is directly connected, port2

[akristof]

Hi,

Thanks. Can you now share with me routing-table from spoke? I want to see what is in the routing-table. I see that the routes are received from BGP, so now I need to see routing-table

[Julien87]

Hi,

sorry, my previous message are not send correctly.

I have the same issue with 7.0.6 version.

FortiGate-SPOKE # get router info routing-table all
Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP
O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
* - candidate default

Routing table for VRF=0
S* 0.0.0.0/0 [1/0] via spk2-ph1 tunnel ISP1
[1/0] via spk2-ph1s tunnel ISP2
S 10.10.100.0/24 [5/0] via spk2-ph1 tunnel ISP1
C 10.10.100.21/32 is directly connected, spk2-ph1
S 10.10.100.254/32 [15/0] via spk2-ph1 tunnel ISP1
S 10.10.101.0/24 [5/0] via spk2-ph1s tunnel ISP2
C 10.10.101.21/32 is directly connected, spk2-ph1s
S 10.10.101.254/32 [15/0] via spk2-ph1s tunnel ISP2
B 10.19.1.0/24 [200/0] via 10.10.100.254 (recursive via spk2-ph1 tunnel 147.78.174.134), 18:00:41
[200/0] via 10.10.101.254 (recursive via spk2-ph1s tunnel 147.78.174.225), 18:00:41
B 10.19.101.0/24 [200/0] via 10.10.100.254 (recursive via spk2-ph1 tunnel 147.78.174.134), 18:00:41
[200/0] via 10.10.101.254 (recursive via spk2-ph1s tunnel 147.78.174.225), 18:00:41
C 10.23.1.0/24 is directly connected, port2
C 10.23.101.0/24 is directly connected, port3
S 130.93.6.131/32 [1/0] via ISP1SPK, port1
[1/0] via ISP2SPK, port4
S 130.93.85.62/32 [1/0] via ISP1SPK, port1
[1/0] via ISP2SPK, port4
C 130.93.98.0/24 is directly connected, port1
S 131.93.6.131/32 [1/0] via ISP1SPK, port1
[1/0] via ISP2SPK, port4
S 131.93.85.62/32 [1/0] via ISP1SPK, port1
[1/0] via ISP2SPK, port4
C 131.93.98.0/24 is directly connected, port4
S 147.78.174.134/32 [1/0] via ISP1SPK, port1
[1/0] via ISP2SPK, port4
S 147.78.174.225/32 [1/0] via ISP1SPK, port1
[1/0] via ISP2SPK, port4

FortiGate-SPOKE# get router info routing-table data
Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP
O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
> - selected route, * - FIB route, p - stale info

Routing table for VRF=0
S *> 0.0.0.0/0 [1/0] via spk2-ph1 tunnel ISP1
*> [1/0] via spk2-ph1s tunnel ISP2
S *> 10.10.100.0/24 [5/0] via spk2-ph1 tunnel ISP1
C *> 10.10.100.21/32 is directly connected, spk2-ph1
S *> 10.10.100.254/32 [15/0] via spk2-ph1 tunnel ISP1
S *> 10.10.101.0/24 [5/0] via spk2-ph1s tunnel ISP2
C *> 10.10.101.21/32 is directly connected, spk2-ph1s
S *> 10.10.101.254/32 [15/0] via spk2-ph1s tunnel ISP2
B *> 10.19.1.0/24 [200/0] via 10.10.100.254 (recursive via spk2-ph1 tunnel ISP1), 18:00:54
*> [200/0] via 10.10.101.254 (recursive via spk2-ph1s tunnel ISP2), 18:00:54
B *> 10.19.101.0/24 [200/0] via 10.10.100.254 (recursive via spk2-ph1 tunnel ISP1), 18:00:54
*> [200/0] via 10.10.101.254 (recursive via spk2-ph1s tunnel ISP2), 18:00:54
C *> 10.23.1.0/24 is directly connected, port2
C *> 10.23.101.0/24 is directly connected, port3
S *> ISP1SPK/32 [1/0] via ISP1SPK, port1
*> [1/0] via ISP2SPK, port4
S *> ISP2SPK/32 [1/0] via ISP1SPK, port1
*> [1/0] via ISP2SPK, port4
C *> ISPNETWORK/24 is directly connected, port1
S *> ISP2SPK/32 [1/0] via ISP1SPK, port1
*> [1/0] via ISP2SPK, port4
S *> ISP2SPK/32 [1/0] via ISP1SPK, port1
*> [1/0] via ISP2SPK, port4
C *> ISP2SPK/24 is directly connected, port4
S *> ISP1/32 [1/0] via ISP1SPK, port1
*> [1/0] via ISP2SPK, port4
S *> ISP2/32 [1/0] via ISP1SPK, port1
*> [1/0] via ISP2SPK, port4

FortiGate-SPOKE# get router info bgp network
VRF 0 BGP table version is 11, local router ID is 1.1.1.2
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

Network Next Hop Metric LocPrf Weight RouteTag Path
*>i10.19.1.0/24 10.10.100.254 0 100 0 0 i <1/1>
*>i 10.10.101.254 0 100 0 0 i <1/2>
*>i10.19.101.0/24 10.10.100.254 0 100 0 0 i <1/1>
*>i 10.10.101.254 0 100 0 0 i <1/2>
*> 10.23.1.0/24 0.0.0.0 100 32768 0 i <-/1>
*> 10.23.101.0/24 0.0.0.0 100 32768 0 i <-/1>

Total number of prefixes 4

[akristof · Accepted solution]

Hi,

First thing I would do is split neighbor-group for each ADVPN tunnels.

Potentially it will work like this, but it would be cleaner.

Second, can you run debug on spoke:

diag ip router bgp level info

diag ip router bgp nsm en

diag ip router bgp all en

diag debug console time en

diag debug en

When debug will be enabled, hard-clear one neighbor (on spoke) and wait until it will negotiate and share the full output with me (attach as file).

Then disable debug:

diag debug reset

diag debug disable

[Julien87]

Hi Adrian,

Thanks for your help. I have a problem with my copy/paste... i have not change router-id....

i'm confuse for this easy error.

Julien

[Julien87]

Just last point, you can confirm this config upgrade for separate group ?

config neighbor-group
edit "advpn"
set capability-default-originate enable
set link-down-failover enable
set remote-as 65505
set additional-path both
set route-reflector-client enable
next
edit "advpn1"
set capability-default-originate enable
set link-down-failover enable
set remote-as 65505
set additional-path both
set route-reflector-client enable
next
end
config neighbor-range
edit 2
set prefix 10.10.101.0 255.255.255.0
set neighbor-group "advpn1"
next
edit 1
set prefix 10.10.100.0 255.255.255.0
set neighbor-group "advpn"
next

[akristof]

Hello,

Happy to help. Yes, exactly. To me this is preferred design as you can do some BGP "stuff" for each advpn group separately.