SOLVED: BGP received routes not inserting into main routing table (7.0.18)

[sarlacii]

Greetings all

I have an interesting issue on a FortiGate 40F, with OS 7.0.18 (soon to be upgraded to 7.2, then 7.4.7, to see if that solves the issue). I have done BGP routing before without issues, but over MPLS lines between FortiSwitch ports. In this case we have dual, redundant, IPsec VPN connections over a single WAN interface

The IPsec's run fine, and both Pri and Sec are up.

Then there are the BGP links, which I created via a BGP loopback interface, with primary IP 172.26.200.21 (just did the secondary link first) and secondary IP 172.26.200.1. The BGP links are also running fine, with packets processing, and both advertised and received routes visible on both sides of the VPN.

BGP Pri: 172.26.200.1 (Mine) <> 172.26.200.2 (ISP)

BGP Sec: 172.26.200.21 (Mine) <> 172.26.200.22 (ISP)

However, the received routes are not being inserted into the main routing table, and thus any packet I receive over the IPsec to my VIP fails the return path check and is dropped (since the return routes are not inserted by BGP - there are only the blackhole routes with metric 254). I know that my advertised routes are being applied by the ISP, as I receive packets from the ISP servers on the correct listening IP that I advertise. However, I cannot reply... .

Interface setups...

# show system interface 
config system interface
    edit "wan"
        set vdom "root"
        set ip 1.2.8.8 255.255.255.248
        set type physical
        set description "1.2.8.7 - Primary IP for ISPguys IPsec
1.2.8.8 - Secondary IP for ISPguys IPsec. The Sec is the one we did first, as this FortiGate 40F was setup to be the Sec - we never put the Pri 40F in place... but ISPguys want both IPsecs."
        set alias "WAN1-ISPguys-CX-SEC"
        set role wan
        set snmp-index 1
        set secondary-IP enable
        config secondaryip
            edit 1
                set ip 1.2.8.7 255.255.255.248
            next
        end
    next
    ...
    ...
    ...
    edit "ISPguys-IPsec-SEC"
        set vdom "root"
        set type tunnel
        set snmp-index 12
        set interface "wan"
    next
    edit "VL2112"
        set vdom "root"
        set ip 172.21.12.226 255.255.255.0
        set allowaccess ping
        set description "Radius VLAN 2112"
        set alias "RADIUS_LAN2"
        set role lan
        set snmp-index 13
        set interface "lan2"
        set vlanid 2112
    next
    edit "ISPguys-IPsec-PRI"
        set vdom "root"
        set type tunnel
        set snmp-index 14
        set interface "wan"
    next
    edit "BGP-AS1234"
        set vdom "root"
        set ip 172.26.200.21 255.255.255.255
        set allowaccess ping
        set type loopback
        set alias "BGP"
        set role lan
        set snmp-index 15
        set secondary-IP enable
        config secondaryip
            edit 1
                set ip 172.26.200.1 255.255.255.255
                set allowaccess ping
            next
        end
    next
end

BGP setup...

# show router bgp
config router bgp
    set as 1234
    set router-id 172.26.200.21
    set bestpath-med-missing-as-worst enable
    set ebgp-multipath enable
    set ibgp-multipath enable
    set graceful-restart enable
    config neighbor
        edit "172.26.200.2"
            set passive enable
            set soft-reconfiguration enable
            set interface "ISPguys-IPsec-PRI"
            set prefix-list-in "ISPguys-ALL-prefix"
            set prefix-list-out "Company-prefix-ALL"
            set remote-as 3000
            set weight 20
        next
        edit "172.26.200.22"
            set passive enable
            set soft-reconfiguration enable
            set interface "ISPguys-IPsec-SEC"
            set prefix-list-in "ISPguys-ALL-prefix"
            set prefix-list-out "Company-prefix-ALL"
            set remote-as 3000
            set weight 10
        next
    end
    config network
        edit 1
            set prefix 172.20.8.0 255.255.255.0
        next
        edit 3
            set prefix 1.5.2.3 255.255.255.255
        next
        edit 4
            set prefix 1.5.2.4 255.255.255.255
        next
    end
    config redistribute "connected"
    end
    config redistribute "rip"
    end
    config redistribute "ospf"
    end
    config redistribute "static"
    end
    config redistribute "isis"
    end
    config redistribute6 "connected"
    end
    config redistribute6 "rip"
    end
    config redistribute6 "ospf"
    end
    config redistribute6 "static"
    end
    config redistribute6 "isis"
    end
end

# show router prefix-list 
config router prefix-list
    edit "ISPguys-ALL-prefix"
        set comments "All the ISPguys IP subnets that we want to receive routing details for over BGP"
        config rule
            edit 1
                set prefix 10.10.0.0 255.255.0.0
                unset ge
                unset le
            next
            edit 2
                set prefix 1.2.1.0 255.255.255.0
                unset ge
                unset le
            next
            edit 3
                set prefix 4.3.4.0 255.255.254.0
                unset ge
                unset le
            next
        end
    next
    edit "Company-prefix-ALL"
        set comments "All the Company IP subnets that we want to transmit routing details for over BGP"
        config rule
            edit 1
                set prefix 172.20.8.0 255.255.255.0
                unset ge
                unset le
            next
            edit 2
                set prefix 1.5.2.3 255.255.255.255
                unset ge
                unset le
            next
            edit 3
                set prefix 1.5.2.4 255.255.255.255
                unset ge
                unset le
            next
        end
    next
end

Neighbors, showing the correctly received routes. If you check filtered routes (i.e. get router info bgp neighbors 172.26.200.2 routes), the list is empty, same as it is if you print the main table and grep for "B" - BGP routes...

# get router info bgp neighbors 172.26.200.2 received-routes
VRF 0 BGP table version is 121, local router ID is 172.26.200.21
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight RouteTag Path
*> 10.10.0.0/16     172.26.200.2                           0        0 3000 4000 3000 65555 65555 ? <-/->
*> 10.10.0.1/32     172.26.200.2                           0        0 3000 4000 3000 65555 65555 ? <-/->
*> 4.3.4.0/23    172.26.200.2                           0        0 3000 4000 65555 ? <-/->
*> 1.2.1.0    172.26.200.2                           0        0 3000 4000 65555 ? <-/->

Total number of prefixes 4

# get router info bgp neighbors 172.26.200.2 routes

# get router info bgp neighbors 172.26.200.22 received-route
VRF 0 BGP table version is 111, local router ID is 172.26.200.21
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight RouteTag Path
*> 10.10.0.0/16     172.26.200.22                          0        0 3000 3000 3000 4000 3000 65555 65555 ? <-/->
*> 10.10.0.1/32     172.26.200.22                          0        0 3000 3000 3000 4000 3000 65555 65555 ? <-/->
*> 4.3.4.0/23    172.26.200.22                          0        0 3000 4000 65555 ? <-/->
*> 1.2.1.0    172.26.200.22                          0        0 3000 4000 65555 ? <-/->

Total number of prefixes 4

# get router info bgp neighbors 172.26.200.22 route

Route arrives... route is deleted. LOL

# diagnose debug reset
# diag ip router command show debug nsm kernel
# diag ip router command show debug nsm level info
# diagnose debug console timestamp enable
# diag debug enable

2025-02-25 09:24:09 2025-02-25 09:24:09 NSM: RTM_NEWROUTE ipv6 unicast proto 
2025-02-25 09:24:39 2025-02-25 09:24:39 NSM: netlink_parse_info: netlink-listen type RTM_DELROUTE(25), seq=0, pid=0
2025-02-25 09:24:39 2025-02-25 09:24:39 NSM: RTM_DELROUTE ipv6 unicast proto 
2025-02-25 09:25:43 2025-02-25 09:25:43 NSM: netlink_parse_info: netlink-listen type RTM_NEWROUTE(24), seq=0, pid=0
2025-02-25 09:25:43 2025-02-25 09:25:43 NSM: RTM_NEWROUTE ipv6 unicast proto 
2025-02-25 09:26:44 2025-02-25 09:26:44 NSM: netlink_parse_info: netlink-listen type RTM_DELROUTE(25), seq=0, pid=0
2025-02-25 09:26:44 2025-02-25 09:26:44 NSM: RTM_DELROUTE ipv6 unicast proto 
2025-02-25 09:27:34 2025-02-25 09:27:34 NSM: netlink_parse_info: netlink-listen type RTM_NEWROUTE(24), seq=0, pid=0
2025-02-25 09:27:34 2025-02-25 09:27:34 NSM: RTM_NEWROUTE ipv6 unicast proto

Confirmation of what we already know, no BGP route is inserted.

# get router info routing-table database 
Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP
       O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
       > - selected route, * - FIB route, p - stale info

Routing table for VRF=0
S    *> 0.0.0.0/0 [10/0] via 1.2.8.93, wan, [1/0]
S    *> 10.10.0.0/16 [254/0] is a summary, Null, [1/0]
C    *> 1.2.8.92/29 is directly connected, wan
     *>                 is directly connected, wan
S    *> 4.3.4.0/23 [254/0] is a summary, Null, [1/0]
C    *> 172.18.8.0/24 is directly connected, lan1
C    *> 172.20.8.0/24 is directly connected, lan2
C    *> 172.21.12.0/24 is directly connected, VL2112
C    *> 172.26.200.1/32 is directly connected, BGP-AS1234
S    *> 172.26.200.2/32 [10/0] via ISPguys-IPsec-PRI tunnel 8.25.30.59, [1/0]
S       172.26.200.2/32 [254/0] is a summary, Null, [1/0]
C    *> 172.26.200.21/32 is directly connected, BGP-AS1234
S    *> 172.26.200.22/32 [10/0] via ISPguys-IPsec-SEC tunnel 8.25.30.14, [1/0]
S       172.26.200.22/32 [254/0] is a summary, Null, [1/0]
S    *> 1.2.1.0/24 [254/0] is a summary, Null, [1/0]
S    *> 1.5.2.3/32 [254/0] is a summary, Null, [1/0]

I have checked IP Pools, and the affected sub-nets are not mentioned.

Any thoughts on what to check??? I have exhausted AI suggestions. :)

[funkylicious]

hi,

im kinda confused about your setup.

you are saying that the BGP peers, 172.26.200.2 and 172.26.200.22 are/should send the following prefixes: 10.10.0.0/16 , 10.10.0.1/32,  4.3.5.0/23 and 1.2.1.0/24  and they are not installed in the routing table ?

L.E.

try starting a BGP focus debug:

# diagnose ip router bgp all enable

# diagnose ip router bgp level info

# diagnose debug console timestamp enable

# diagnose debug enable

[sarlacii]

Hi funkylicious - that is correct yes. ISP BGP is correctly advertising the expected routes, and they are being received on my side. But there is some sort of routing/filter issue that is preventing insert into main table.

[Toshi_Esumi]

I haven't used "prefix-list-in/prefix-list-out" with BGP so I can't tell the behavior of them. But the config looks reasonable to me.
Only one thing I noticed odd or something wrong other than the route filtering outcome is it's receiving

4.3.5.0/23

If it's /23, it's supposed to be 4.3.4.0/23. But since you have the same in your static route. So I'm assuming you changed the first 3 octets before posting this but forgot to make the last bit of the 3rd octet 0. So I'll ignore it.

With this config, is the ISP receiving those three routes you advertised?

I would try wrapping the prefix-list in a route-map, then try using "route-map-in/route-map-out" instead to see if that makes any change.
If it still doesn't work, I would open a ticket at TAC to get it looked at if it's production unit.

Toshi

[sarlacii]

---

@Toshi_Esumi wrote:

I haven't used "prefix-list-in/prefix-list-out" with BGP so I can't tell the behavior of them. But the config looks reasonable to me.
Only one thing I noticed odd or something wrong other than the route filtering outcome is it's receiving

4.3.5.0/23

If it's /23, it's supposed to be 4.3.4.0/23. But since you have the same in your static route. So I'm assuming you changed the first 3 octets before posting this but forgot to make the last bit of the 3rd octet 0. So I'll ignore it.

---

You are correct, my apologies. I did indeed modify the IP's and forgot to make the /23 even. Please ignore my error - I have corrected my post. And the IP is correct in the firewall.

---

@Toshi_Esumi wrote:

With this config, is the ISP receiving those three routes you advertised?

---

Yes, my advertised routes are being received, I confirmed.

---

@Toshi_Esumi wrote:

I would try wrapping the prefix-list in a route-map, then try using "route-map-in/route-map-out" instead to see if that makes any change.
If it still doesn't work, I would open a ticket at TAC to get it looked at if it's production unit.

---

I did in fact completely disable the prefix-list (in and out), to remove its effect, as the ISP is only actually sending the routes I expect. But having no prefix-list did not make the routes insert.

As such, I suspect that it is not a prefix issue, but rather a problem with the next-hop. I just wish I could see the issue with route inserts in the debug. The verbose debug settings suggetsed by everyone (same as the ones I usually use myself) show all sorts of details about the BGP connection itself (which TBH is usually the problem people have) but very little about the rooute selection process. I need to see why the received routes fail to insert.

Upgrading to 7.4.7 (from 7.0.17, via 7.2.11) did not solve any issues as I suspected. But I will run debug again and check... .

And yes, next step will be to log a ticket.

[Toshi_Esumi]

I'm still not sure the mechanism how FGT chooses a route to put into RIB when the same route exists in multiple protocols like static, BGP, and OSPF, etc., and what the common metrics are used. So I always avoid that situation to happen.

Try setting a larger subnet mask for those blackhole routes, not to be the same with BGP routes. That would probably mitigate the problem.

Generally the blackhole routes are not necessary when you set "snat-route-change" enabled.
https://community.fortinet.com/t5/FortiGate/Technical-Tip-Using-SNAT-route-change-to-update-existing...
At least we don't use them.

Toshi 

[sarlacii]

Good morning Toshi, and thank you for the additional detail.

I also did test with the blackhole routes removed, and same result.

IMHO the routing calculations should say: a) route exists with metric 254 (blackhole), then b) BGP route received with metric 20 (default), and then choose the BGP route over the blackhole. And so if I look at route database, I will see both, but if I look at route all, or the neighbor routes, I would only see the accepted BGP route.

And obviously if I was to put in a static route, then I could use metric 10 to override them all.

But the BGP routes are not even getting to the point where they are being considered in the routing table. However, as you say, it could just be some wierdness with mask clashes etc.

I will revert with details after the FortiNet support call.

Go well.

[sarlacii · Accepted solution]

Ok, so I finally figured out the issue, before I even got to FortiNet support.

Turns out that a loopback interface is not treated as a "connected" subnet by the FortiGate for the purposes of BGP routes!!!

To verify the BGP routes - run these debug commands:

diagnose ip router bgp all disable # Disable all the BGP debug, so we only see what we actually want!
diagnose ip router bgp updates en # We want (route) updates
diagnose ip router bgp level info # Verbosity "info", "critical", "error", "warn", "none"
diagnose debug enable

Then soft reset the routes, so that you see what happens when they are added back!

execute router clear bgp ip <neighbour ip> soft

Eg.

execute router clear bgp ip 172.26.200.2 soft
BGP: 172.26.200.2-Outgoing [RIB] Update: Prefix 10.10.0.0/16 path_id 0 denied due to non-connected next-hop;
BGP: 172.26.200.2-Outgoing [RIB] Update: Prefix 4.3.4.0/23 path_id 0 denied due to non-connected next-hop;
BGP: 172.26.200.2-Outgoing [RIB] Update: Prefix 1.2.1.0/24 path_id 0 denied due to non-connected next-hop;

The next-hop is connected! But it is a BGP loopback interface... so it is ignored, and you get a next-hop failure.

Solution:

Force the Fortigate to consider the next-hop connected, by setting the "ebgp-enforce-multihop" option for each neighbor.

Eg.

	config router bgp
		config neighbor
			edit "172.26.200.2"
				set ebgp-enforce-multihop enable
			next
			edit "172.26.200.22"
				set ebgp-enforce-multihop enable
			next
		end
	end

See post here: Troubleshooting-Tip-BGP-debug-log-message-denied-due-to-filter 

And here: Technical-Tip-BGP-routes-are-not-installed-in-routing-table-with-loopback 

[Toshi_Esumi]

makes sense.

Toshi