Redistribute OSPF over BGP Between to FotiGates (wrong next hop ip)

[Sowie]

Hi,

I'm trying to redistribute OSPF over BGP. The Neighbors are getting the routes but the routes are using wrong recursive next hop IP on one of the sides...

When you look at the routing table on the right side it is using the WAN IP instead of the tunnel IP

DEFLE-FW01 $ get router info routing-table bgp
Routing table for VRF=0
B 10.1.2.0/24 [200/20] via 172.21.1.6 (recursive via ADVPN tunnel "WAN IP"), 01:06:38
B 10.1.3.0/24 [200/20] via 172.21.1.6 (recursive via ADVPN tunnel "WAN IP"), 01:06:38
B 10.1.4.0/24 [200/20] via 172.21.1.6 (recursive via ADVPN tunnel "WAN IP"), 01:06:38
B 10.1.5.0/24 [200/20] via 172.21.1.6 (recursive via ADVPN tunnel "WAN IP"), 01:06:38
B 10.1.6.0/24 [200/20] via 172.21.1.6 (recursive via ADVPN tunnel "WAN IP"), 01:06:38
B 10.1.90.0/24 [200/20] via 172.21.1.6 (recursive via ADVPN tunnel "WAN IP"), 01:06:38
B 10.1.91.0/24 [200/20] via 172.21.1.6 (recursive via ADVPN tunnel "WAN IP"), 01:06:38
B 10.1.91.100/32 [200/0] via 172.30.0.254 (recursive via ADVPN tunnel "WAN IP"), 02:32:12
B 10.1.100.0/24 [200/20] via 172.21.1.6 (recursive via ADVPN tunnel "WAN IP"), 01:06:38
B 172.21.1.0/30 [200/0] via 172.30.0.1 (recursive via ADVPN tunnel "WAN IP"), 02:32:12
B 172.21.1.4/30 [200/0] via 172.30.0.1 (recursive via ADVPN tunnel "WAN IP"), 02:32:12
B 192.168.4.0/24 [200/0] via 172.30.0.1 (recursive via ADVPN tunnel "WAN IP"), 02:27:33

But when you look on the left side everything seems fine

DKAAR-FW01 $ get router info routing-table bgp
Routing table for VRF=0
B 10.2.2.0/24 [200/20] via 172.21.2.2 (recursive via ADVPN tunnel 172.30.0.2), 01:48:55
B 10.2.3.0/24 [200/20] via 172.21.2.2 (recursive via ADVPN tunnel 172.30.0.2), 01:48:55
B 10.2.4.0/24 [200/20] via 172.21.2.2 (recursive via ADVPN tunnel 172.30.0.2), 01:48:55
B 10.2.5.0/24 [200/20] via 172.21.2.2 (recursive via ADVPN tunnel 172.30.0.2), 01:48:55
B 10.2.6.0/24 [200/20] via 172.21.2.2 (recursive via ADVPN tunnel 172.30.0.2), 01:48:55
B 10.2.90.0/24 [200/20] via 172.21.2.2 (recursive via ADVPN tunnel 172.30.0.2), 01:48:55
B 10.2.91.0/24 [200/20] via 172.21.2.2 (recursive via ADVPN tunnel 172.30.0.2), 01:48:55
B 10.2.91.100/32 [200/0] via 172.30.0.1 (recursive is directly connected, ADVPN), 03:21:57
B 10.2.100.0/24 [200/20] via 172.21.2.2 (recursive via ADVPN tunnel 172.30.0.254), 01:48:55
B 172.21.2.0/30 [200/0] via 172.30.0.1 (recursive is directly connected, ADVPN), 03:21:57

If you have idea on how to fix this please let me know.

Both Fortigates are running version 7.0.11

[Sowie]

Hi Adrian,

I reset the firewalls and got the VPN working again, but I still have the issue with BGP next-hop. Here you have the requested outputs.

Routing table
S* 0.0.0.0/0 [5/0] via 10.192.22.1, wan1, [1/0]
B 10.1.2.0/24 [200/20] via 172.21.1.2 (recursive via ADVPN-SPOKE tunnel ###WANIP###), 01:43:43
B 10.1.3.0/24 [200/20] via 172.21.1.2 (recursive via ADVPN-SPOKE tunnel ###WANIP###), 01:43:43
B 10.1.4.0/24 [200/20] via 172.21.1.2 (recursive via ADVPN-SPOKE tunnel ###WANIP###), 01:43:43
B 10.1.5.0/24 [200/20] via 172.21.1.2 (recursive via ADVPN-SPOKE tunnel ###WANIP###), 01:43:43
B 10.1.6.0/24 [200/20] via 172.21.1.2 (recursive via ADVPN-SPOKE tunnel ###WANIP###), 01:43:43
B 10.1.90.0/24 [200/20] via 172.21.1.2 (recursive via ADVPN-SPOKE tunnel ###WANIP###), 01:43:43
B 10.1.91.0/24 [200/20] via 172.21.1.2 (recursive via ADVPN-SPOKE tunnel ###WANIP###), 01:43:43
B 10.1.91.100/32 [200/0] via 172.30.0.254 (recursive via ADVPN-SPOKE tunnel ###WANIP###), 01:49:59
B 10.1.92.0/24 [200/20] via 172.21.1.2 (recursive via ADVPN-SPOKE tunnel ###WANIP###), 01:43:43
B 10.1.100.0/24 [200/20] via 172.21.1.2 (recursive via ADVPN-SPOKE tunnel ###WANIP###), 01:43:43
O E2 10.2.2.0/24 [110/20] via 172.21.2.2, LACP DEFLE-CSW1, 03:17:07
O E2 10.2.3.0/24 [110/20] via 172.21.2.2, LACP DEFLE-CSW1, 03:17:07
O E2 10.2.4.0/24 [110/20] via 172.21.2.2, LACP DEFLE-CSW1, 03:17:07
O E2 10.2.5.0/24 [110/20] via 172.21.2.2, LACP DEFLE-CSW1, 03:17:07
O E2 10.2.6.0/24 [110/20] via 172.21.2.2, LACP DEFLE-CSW1, 03:17:07
O E2 10.2.90.0/24 [110/20] via 172.21.2.2, LACP DEFLE-CSW1, 03:17:07
O E2 10.2.91.0/24 [110/20] via 172.21.2.2, LACP DEFLE-CSW1, 03:17:07
C 10.2.91.100/32 is directly connected, NETMGMT
O E2 10.2.100.0/24 [110/20] via 172.21.2.2, LACP DEFLE-CSW1, 03:17:07
C 10.192.22.0/24 is directly connected, wan1
B 172.21.1.0/30 [200/0] via 172.30.0.254 (recursive via ADVPN-SPOKE tunnel ###WANIP###), 01:49:59
B 172.21.1.4/30 [200/0] via 172.30.0.254 (recursive via ADVPN-SPOKE tunnel ###WANIP###), 01:49:59
C 172.21.2.0/30 is directly connected, LACP DEFLE-CSW1
S 172.30.0.0/24 [5/0] via ADVPN-SPOKE tunnel ###WANIP###, [1/0]
C 172.30.0.1/32 is directly connected, ADVPN-SPOKE
S 172.30.0.254/32 [15/0] via ADVPN-SPOKE tunnel ###WANIP###, [1/0]
O E2 192.168.0.0/24 [110/20] via 172.21.2.2, LACP DEFLE-CSW1, 03:17:07
C 192.168.1.0/24 is directly connected, lan

diag vpn ike gateway list

name: ADVPN-SPOKE
version: 1
interface: wan1 5
addr: 10.192.22.17:4500 -> ###WANIP###:4500
tun_id: ###WANIP###/::###WANIP###
remote_location: 0.0.0.0
network-id: 0
virtual-interface-addr: 172.30.0.1 -> 172.30.0.254
created: 9958s ago
nat: me
auto-discovery: 2 receiver
IKE SA: created 1/1 established 1/1 time 20/20/20 ms
IPsec SA: created 1/1 established 1/1 time 40/40/40 ms

id/spi: 1 630310eb9f49d396/faa1622092f94387
direction: initiator
status: established 9958-9958s ago = 20ms
proposal: aes128-sha256
key: d67ffbc3d7874fb3-47d28846402d395b
lifetime/rekey: 86400/76141
DPD sent/recv: 00000402/00000239

diag vpn tunnel list

list all ipsec tunnel in vd 0
------------------------------------------------------

name=ADVPN-SPOKE ver=1 serial=1 10.192.22.17:4500->###WANIP###:4500 tun_id=###WANIP### tun_id6=::###WANIP### dst_mtu=1500 dpd-link=on weight=1
bound_if=5 lgwy=static/1 tun=intf mode=auto/1 encap=none/568 options[0238]=npu create_dev frag-rfc role=primary accept_traffic=1 overlay_id=0

proxyid_num=1 child_num=0 refcnt=5 ilast=0 olast=1 ad=r/2
stat: rxp=2689 txp=5077 rxb=540576 txb=3796897
dpd: mode=on-idle on=1 idle=5000ms retry=3 count=0 seqno=1026
natt: mode=keepalive draft=32 interval=10 remote_port=4500
proxyid=ADVPN-SPOKE proto=0 sa=1 ref=6 serial=1 auto-negotiate adr
src: 0:0.0.0.0-255.255.255.255:0
dst: 0:0.0.0.0-255.255.255.255:0
SA: ref=6 options=1a227 type=00 soft=0 mtu=1422 expire=32827/0B replaywin=1024
seqno=13d4 esn=0 replaywin_lastseq=00000a80 qat=0 rekey=0 hash_search_len=1
life: type=01 bytes=0/0 timeout=42901/43200
dec: spi=f1b700f7 esp=aes key=16 581dc979cff130a025311f22d633870a
ah=sha1 key=20 7eef9344de69b0fe184a07a3487db50f7f642f4e
enc: spi=09053b75 esp=aes key=16 59c85a2e11093fb873a501cd3cc9798d
ah=sha1 key=20 a6f270b938efb506b3ae387a4729d381120accb1
dec:pkts/bytes=2690/540636, enc:pkts/bytes=10139/7882174
npu_flag=03 npu_rgwy=###WANIP### npu_lgwy=10.192.22.17 npu_selid=0 dec_npuid=1 enc_npuid=1

Sorry but I have to mask the WAN IP. This is HUB-SPOKE the right side is the SPOKE.

Thanks again for taking a look at this.

[akristof]

Hi,

Can you please add me the following from the HUB:
show vpn ipsec phase1-setting ADVPN-SPOKE

show system interface ADVPN-SPOKE

Please?

[Sowie]

Hi Adrian

phase1-setting doesn't seem to exist. So here is the interface. On the HUB site its called "ADVPN-HUB".

DKAAR-FW01 $ show vpn ipsec phase1-setting ADVPN-HUB

command parse error before 'phase1-setting'
Command fail. Return code -61

DKAAR-FW01 $ show vpn ipsec phase1-interface ADVPN-HUB
config vpn ipsec phase1-interface
edit "ADVPN-HUB"
set type dynamic
set interface "wan"
set peertype any
set net-device disable
set proposal aes128-sha256 aes256-sha256 3des-sha256 aes128-sha1 aes256-sha1 3des-sha1
set add-route disable
set dpd on-idle
set auto-discovery-sender enable
set psksecret ENC Cb0IVdWJfpbb3Iwc3bvWHJBAcgcYhBaAPH+Vr7lFNMKd0csk3XEqs+spjNakSuRpxHIwYhX/y9go+i2ZHSc09Vb5Nu/r+Kj1EFox5ew/0eOv+Ph7h6B4gQmoKPiQMrwEANzJizYXOoSRqsY9iNxnjNjl3bDf3EUTOst5L1ZNaqgObV2wn1A0b0dgjNRrSM5SrnkZLw==
set dpd-retryinterval 5
next
end

DKAAR-FW01 $ show system interface ADVPN-HUB
config system interface
edit "ADVPN-HUB"
set vdom "root"
set ip 172.30.0.254 255.255.255.255
set allowaccess ping ssh
set type tunnel
set remote-ip 172.30.0.253 255.255.255.0
set snmp-index 14
set interface "wan"
next
end

[Sowie]

Hi @akristof @Demir21 

Thanks for the response! I managed to delete the Spoke VPN (and spill Ice Tea into my laptop...)by mistake. I've have been unable to recreate it. So I'll have get that working again before I can check. 

Thinking of just factory resetting both firewalls...

[RaniGome]

When you can, try to create static routes to the network of your IP addresses of the tunnels that are exchanging route, in your case ADVPN. that will solve your problem of wrong interfaces on bgp routes.

[Sowie]

Hi Rani

This already has a static route  As you can see in the picture

[srajeswaran]

Can you share below outputs.
From DKAAR-FW01:
get router info bgp neighbors <DEFLE-FW01 IP> advertised-routes

From DEFLE-FW01:

get router info bgp neighbors <DKAAR-FW01 IP> received-routes

[Sowie]

DKAAR-FW01:

DKAAR-FW01 $ get router info bgp neighbors 172.30.0.1 advertised-routes
VRF 0 BGP table version is 8, local router ID is 172.30.0.254
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete

Network Next Hop Metric LocPrf Weight RouteTag Path
*>i10.1.2.0/24 172.21.1.2 20 100 32768 0 ? <-/->
*>i10.1.3.0/24 172.21.1.2 20 100 32768 0 ? <-/->
*>i10.1.4.0/24 172.21.1.2 20 100 32768 0 ? <-/->
*>i10.1.5.0/24 172.21.1.2 20 100 32768 0 ? <-/->
*>i10.1.6.0/24 172.21.1.2 20 100 32768 0 ? <-/->
*>i10.1.90.0/24 172.21.1.2 20 100 32768 0 ? <-/->
*>i10.1.91.0/24 172.21.1.2 20 100 32768 0 ? <-/->
*>i10.1.91.100/32 172.30.0.254 100 32768 0 i <-/->
*>i10.1.92.0/24 172.21.1.2 20 100 32768 0 ? <-/->
*>i10.1.100.0/24 172.21.1.2 20 100 32768 0 ? <-/->
*>i172.21.1.0/30 172.30.0.254 100 32768 0 i <-/->
*>i172.21.1.4/30 172.30.0.254 100 32768 0 i <-/->
*>i192.168.0.0 172.21.1.2 20 100 32768 0 ? <-/->
*>i192.168.1.0 172.21.1.2 20 100 32768 0 ? <-/->
*>i192.168.4.0 172.30.0.254 100 32768 0 i <-/->

Total number of prefixes 15

DKFLE-FW01:

For some reason I can't do the received-routes command but i can give you he routes which should be the same.

DEFLE-FW01 $ get router info bgp neighbors 172.30.0.254 received-routes
% Inbound soft reconfiguration not enabled
% No prefix for neighbor 172.30.0.254

DEFLE-FW01 $ get router info bgp neighbors 172.30.0.254 routes
VRF 0 BGP table version is 8, local router ID is 172.30.0.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

Network Next Hop Metric LocPrf Weight RouteTag Path
*>i10.1.2.0/24 172.21.1.2 20 100 0 0 ? <-/1>
*>i10.1.3.0/24 172.21.1.2 20 100 0 0 ? <-/1>
*>i10.1.4.0/24 172.21.1.2 20 100 0 0 ? <-/1>
*>i10.1.5.0/24 172.21.1.2 20 100 0 0 ? <-/1>
*>i10.1.6.0/24 172.21.1.2 20 100 0 0 ? <-/1>
*>i10.1.90.0/24 172.21.1.2 20 100 0 0 ? <-/1>
*>i10.1.91.0/24 172.21.1.2 20 100 0 0 ? <-/1>
*>i10.1.91.100/32 172.30.0.254 0 100 0 0 i <-/1>
*>i10.1.92.0/24 172.21.1.2 20 100 0 0 ? <-/1>
*>i10.1.100.0/24 172.21.1.2 20 100 0 0 ? <-/1>
*>i172.21.1.0/30 172.30.0.254 0 100 0 0 i <-/1>
*>i172.21.1.4/30 172.30.0.254 0 100 0 0 i <-/1>
* i192.168.0.0 172.21.1.2 20 100 0 0 ? <-/->
*>i192.168.1.0 172.21.1.2 20 100 0 0 ? <-/1>
*>i192.168.4.0 172.30.0.254 0 100 0 0 i <-/1>

Total number of prefixes 15

[srajeswaran]

Lets take

10.1.2.0/24, this route is advertised with next hop 172.21.1.2 .

We also see 172.21.1.0/30 advertised with next hop 172.30.0.254 

So technically 10.1.2.0/24  is reachable via 172.30.0.254 .

Is 172.30.0.254 the WAN IP on DKAAR-FW01 ?

[Sowie]

No 172.30.0.254 is the VPN hub interface on DKAAR-FW01 see updated picture below.

the difference between the two routes you mentioned is that the 172.21.0/30 is directly connected and the 10.1.2.0/24 is a redistributed route via bgp.