Skip to main content
sarlacii
sarlacii
Explorer
February 28, 2025
Solved

SOLVED: BGP received routes not inserting into main routing table (7.0.18)

  • February 28, 2025
  • 8 replies
  • 8914 views

Greetings all

 

I have an interesting issue on a FortiGate 40F, with OS 7.0.18 (soon to be upgraded to 7.2, then 7.4.7, to see if that solves the issue). I have done BGP routing before without issues, but over MPLS lines between FortiSwitch ports. In this case we have dual, redundant, IPsec VPN connections over a single WAN interface

 

The IPsec's run fine, and both Pri and Sec are up.

 

Then there are the BGP links, which I created via a BGP loopback interface, with primary IP 172.26.200.21 (just did the secondary link first) and secondary IP 172.26.200.1. The BGP links are also running fine, with packets processing, and both advertised and received routes visible on both sides of the VPN.

BGP Pri: 172.26.200.1 (Mine) <> 172.26.200.2 (ISP)

BGP Sec: 172.26.200.21 (Mine) <> 172.26.200.22 (ISP)

 

However, the received routes are not being inserted into the main routing table, and thus any packet I receive over the IPsec to my VIP fails the return path check and is dropped (since the return routes are not inserted by BGP - there are only the blackhole routes with metric 254). I know that my advertised routes are being applied by the ISP, as I receive packets from the ISP servers on the correct listening IP that I advertise. However, I cannot reply... .

 

Interface setups...

 

 

 

# show system interface  config system interface     edit "wan"         set vdom "root"         set ip 1.2.8.8 255.255.255.248         set type physical         set description "1.2.8.7 - Primary IP for ISPguys IPsec 1.2.8.8 - Secondary IP for ISPguys IPsec. The Sec is the one we did first, as this FortiGate 40F was setup to be the Sec - we never put the Pri 40F in place... but ISPguys want both IPsecs."         set alias "WAN1-ISPguys-CX-SEC"         set role wan         set snmp-index 1         set secondary-IP enable         config secondaryip             edit 1                 set ip 1.2.8.7 255.255.255.248             next         end     next     ...     ...     ...     edit "ISPguys-IPsec-SEC"         set vdom "root"         set type tunnel         set snmp-index 12         set interface "wan"     next     edit "VL2112"         set vdom "root"         set ip 172.21.12.226 255.255.255.0         set allowaccess ping         set description "Radius VLAN 2112"         set alias "RADIUS_LAN2"         set role lan         set snmp-index 13         set interface "lan2"         set vlanid 2112     next     edit "ISPguys-IPsec-PRI"         set vdom "root"         set type tunnel         set snmp-index 14         set interface "wan"     next     edit "BGP-AS1234"         set vdom "root"         set ip 172.26.200.21 255.255.255.255         set allowaccess ping         set type loopback         set alias "BGP"         set role lan         set snmp-index 15         set secondary-IP enable         config secondaryip             edit 1                 set ip 172.26.200.1 255.255.255.255                 set allowaccess ping             next         end     next end

 

 

 

 

BGP setup...

 

 

 

# show router bgp config router bgp     set as 1234     set router-id 172.26.200.21     set bestpath-med-missing-as-worst enable     set ebgp-multipath enable     set ibgp-multipath enable     set graceful-restart enable     config neighbor         edit "172.26.200.2"             set passive enable             set soft-reconfiguration enable             set interface "ISPguys-IPsec-PRI"             set prefix-list-in "ISPguys-ALL-prefix"             set prefix-list-out "Company-prefix-ALL"             set remote-as 3000             set weight 20         next         edit "172.26.200.22"             set passive enable             set soft-reconfiguration enable             set interface "ISPguys-IPsec-SEC"             set prefix-list-in "ISPguys-ALL-prefix"             set prefix-list-out "Company-prefix-ALL"             set remote-as 3000             set weight 10         next     end     config network         edit 1             set prefix 172.20.8.0 255.255.255.0         next         edit 3             set prefix 1.5.2.3 255.255.255.255         next         edit 4             set prefix 1.5.2.4 255.255.255.255         next     end     config redistribute "connected"     end     config redistribute "rip"     end     config redistribute "ospf"     end     config redistribute "static"     end     config redistribute "isis"     end     config redistribute6 "connected"     end     config redistribute6 "rip"     end     config redistribute6 "ospf"     end     config redistribute6 "static"     end     config redistribute6 "isis"     end end  # show router prefix-list  config router prefix-list     edit "ISPguys-ALL-prefix"         set comments "All the ISPguys IP subnets that we want to receive routing details for over BGP"         config rule             edit 1                 set prefix 10.10.0.0 255.255.0.0                 unset ge                 unset le             next             edit 2                 set prefix 1.2.1.0 255.255.255.0                 unset ge                 unset le             next             edit 3                 set prefix 4.3.4.0 255.255.254.0                 unset ge                 unset le             next         end     next     edit "Company-prefix-ALL"         set comments "All the Company IP subnets that we want to transmit routing details for over BGP"         config rule             edit 1                 set prefix 172.20.8.0 255.255.255.0                 unset ge                 unset le             next             edit 2                 set prefix 1.5.2.3 255.255.255.255                 unset ge                 unset le             next             edit 3                 set prefix 1.5.2.4 255.255.255.255                 unset ge                 unset le             next         end     next end

 

 

 

 

Neighbors, showing the correctly received routes. If you check filtered routes (i.e. get router info bgp neighbors 172.26.200.2 routes), the list is empty, same as it is if you print the main table and grep for "B" - BGP routes...

 

 

 

# get router info bgp neighbors 172.26.200.2 received-routes VRF 0 BGP table version is 121, local router ID is 172.26.200.21 Status codes: s suppressed, d damped, h history, * valid, > best, i - internal Origin codes: i - IGP, e - EGP, ? - incomplete     Network          Next Hop            Metric LocPrf Weight RouteTag Path *> 10.10.0.0/16     172.26.200.2                           0        0 3000 4000 3000 65555 65555 ? <-/-> *> 10.10.0.1/32     172.26.200.2                           0        0 3000 4000 3000 65555 65555 ? <-/-> *> 4.3.4.0/23    172.26.200.2                           0        0 3000 4000 65555 ? <-/-> *> 1.2.1.0    172.26.200.2                           0        0 3000 4000 65555 ? <-/->  Total number of prefixes 4  # get router info bgp neighbors 172.26.200.2 routes  # get router info bgp neighbors 172.26.200.22 received-route VRF 0 BGP table version is 111, local router ID is 172.26.200.21 Status codes: s suppressed, d damped, h history, * valid, > best, i - internal Origin codes: i - IGP, e - EGP, ? - incomplete     Network          Next Hop            Metric LocPrf Weight RouteTag Path *> 10.10.0.0/16     172.26.200.22                          0        0 3000 3000 3000 4000 3000 65555 65555 ? <-/-> *> 10.10.0.1/32     172.26.200.22                          0        0 3000 3000 3000 4000 3000 65555 65555 ? <-/-> *> 4.3.4.0/23    172.26.200.22                          0        0 3000 4000 65555 ? <-/-> *> 1.2.1.0    172.26.200.22                          0        0 3000 4000 65555 ? <-/->  Total number of prefixes 4  # get router info bgp neighbors 172.26.200.22 route

 

 

 

Route arrives... route is deleted. LOL

 

 

 

# diagnose debug reset # diag ip router command show debug nsm kernel # diag ip router command show debug nsm level info # diagnose debug console timestamp enable # diag debug enable  2025-02-25 09:24:09 2025-02-25 09:24:09 NSM: RTM_NEWROUTE ipv6 unicast proto  2025-02-25 09:24:39 2025-02-25 09:24:39 NSM: netlink_parse_info: netlink-listen type RTM_DELROUTE(25), seq=0, pid=0 2025-02-25 09:24:39 2025-02-25 09:24:39 NSM: RTM_DELROUTE ipv6 unicast proto  2025-02-25 09:25:43 2025-02-25 09:25:43 NSM: netlink_parse_info: netlink-listen type RTM_NEWROUTE(24), seq=0, pid=0 2025-02-25 09:25:43 2025-02-25 09:25:43 NSM: RTM_NEWROUTE ipv6 unicast proto  2025-02-25 09:26:44 2025-02-25 09:26:44 NSM: netlink_parse_info: netlink-listen type RTM_DELROUTE(25), seq=0, pid=0 2025-02-25 09:26:44 2025-02-25 09:26:44 NSM: RTM_DELROUTE ipv6 unicast proto  2025-02-25 09:27:34 2025-02-25 09:27:34 NSM: netlink_parse_info: netlink-listen type RTM_NEWROUTE(24), seq=0, pid=0 2025-02-25 09:27:34 2025-02-25 09:27:34 NSM: RTM_NEWROUTE ipv6 unicast proto

 

 

 

 

Confirmation of what we already know, no BGP route is inserted.

 

 

 

# get router info routing-table database  Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP        O - OSPF, IA - OSPF inter area        N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2        E1 - OSPF external type 1, E2 - OSPF external type 2        i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area        > - selected route, * - FIB route, p - stale info  Routing table for VRF=0 S    *> 0.0.0.0/0 [10/0] via 1.2.8.93, wan, [1/0] S    *> 10.10.0.0/16 [254/0] is a summary, Null, [1/0] C    *> 1.2.8.92/29 is directly connected, wan      *>                 is directly connected, wan S    *> 4.3.4.0/23 [254/0] is a summary, Null, [1/0] C    *> 172.18.8.0/24 is directly connected, lan1 C    *> 172.20.8.0/24 is directly connected, lan2 C    *> 172.21.12.0/24 is directly connected, VL2112 C    *> 172.26.200.1/32 is directly connected, BGP-AS1234 S    *> 172.26.200.2/32 [10/0] via ISPguys-IPsec-PRI tunnel 8.25.30.59, [1/0] S       172.26.200.2/32 [254/0] is a summary, Null, [1/0] C    *> 172.26.200.21/32 is directly connected, BGP-AS1234 S    *> 172.26.200.22/32 [10/0] via ISPguys-IPsec-SEC tunnel 8.25.30.14, [1/0] S       172.26.200.22/32 [254/0] is a summary, Null, [1/0] S    *> 1.2.1.0/24 [254/0] is a summary, Null, [1/0] S    *> 1.5.2.3/32 [254/0] is a summary, Null, [1/0]

 

 

 

 

I have checked IP Pools, and the affected sub-nets are not mentioned.

 

Any thoughts on what to check??? I have exhausted AI suggestions. :)

 

Best answer by sarlacii

Ok, so I finally figured out the issue, before I even got to FortiNet support.

 

Turns out that a loopback interface is not treated as a "connected" subnet by the FortiGate for the purposes of BGP routes!!!

 

To verify the BGP routes - run these debug commands:

diagnose ip router bgp all disable # Disable all the BGP debug, so we only see what we actually want! diagnose ip router bgp updates en # We want (route) updates diagnose ip router bgp level info # Verbosity "info", "critical", "error", "warn", "none" diagnose debug enable


Then soft reset the routes, so that you see what happens when they are added back!

execute router clear bgp ip <neighbour ip> soft

Eg.

execute router clear bgp ip 172.26.200.2 soft BGP: 172.26.200.2-Outgoing [RIB] Update: Prefix 10.10.0.0/16 path_id 0 denied due to non-connected next-hop; BGP: 172.26.200.2-Outgoing [RIB] Update: Prefix 4.3.4.0/23 path_id 0 denied due to non-connected next-hop; BGP: 172.26.200.2-Outgoing [RIB] Update: Prefix 1.2.1.0/24 path_id 0 denied due to non-connected next-hop;

The next-hop is connected! But it is a BGP loopback interface... so it is ignored, and you get a next-hop failure.

 

Solution:

Force the Fortigate to consider the next-hop connected, by setting the "ebgp-enforce-multihop" option for each neighbor.

Eg.

	config router bgp 		config neighbor 			edit "172.26.200.2" 				set ebgp-enforce-multihop enable 			next 			edit "172.26.200.22" 				set ebgp-enforce-multihop enable 			next 		end 	end

 

See post here: Troubleshooting-Tip-BGP-debug-log-message-denied-due-to-filter 

And here: Technical-Tip-BGP-routes-are-not-installed-in-routing-table-with-loopback 

 

8 replies

funkylicious
SuperUser
funkylicious
SuperUser
February 28, 2025

hi,

im kinda confused about your setup.

you are saying that the BGP peers, 172.26.200.2 and 172.26.200.22 are/should send the following prefixes: 10.10.0.0/16 , 10.10.0.1/32,  4.3.5.0/23 and 1.2.1.0/24  and they are not installed in the routing table ?

 

L.E.

try starting a BGP focus debug:

# diagnose ip router bgp all enable

# diagnose ip router bgp level info

# diagnose debug console timestamp enable

# diagnose debug enable

"jack of all trades, master of none"
    sarlacii
    sarlaciiAuthor
    Explorer
    March 3, 2025

    Hi funkylicious - that is correct yes. ISP BGP is correctly advertising the expected routes, and they are being received on my side. But there is some sort of routing/filter issue that is preventing insert into main table.

    Toshi_Esumi
    SuperUser
    Toshi_Esumi
    SuperUser
    February 28, 2025

    I haven't used "prefix-list-in/prefix-list-out" with BGP so I can't tell the behavior of them. But the config looks reasonable to me.
    Only one thing I noticed odd or something wrong other than the route filtering outcome is it's receiving

    4.3.5.0/23

    If it's /23, it's supposed to be 4.3.4.0/23. But since you have the same in your static route. So I'm assuming you changed the first 3 octets before posting this but forgot to make the last bit of the 3rd octet 0. So I'll ignore it.

    With this config, is the ISP receiving those three routes you advertised?

    I would try wrapping the prefix-list in a route-map, then try using "route-map-in/route-map-out" instead to see if that makes any change.
    If it still doesn't work, I would open a ticket at TAC to get it looked at if it's production unit.

    Toshi

      sarlacii
      sarlaciiAuthor
      Explorer
      March 3, 2025
      @Toshi_Esumi wrote:

      I haven't used "prefix-list-in/prefix-list-out" with BGP so I can't tell the behavior of them. But the config looks reasonable to me.
      Only one thing I noticed odd or something wrong other than the route filtering outcome is it's receiving

      4.3.5.0/23

      If it's /23, it's supposed to be 4.3.4.0/23. But since you have the same in your static route. So I'm assuming you changed the first 3 octets before posting this but forgot to make the last bit of the 3rd octet 0. So I'll ignore it.

      You are correct, my apologies. I did indeed modify the IP's and forgot to make the /23 even. Please ignore my error - I have corrected my post. And the IP is correct in the firewall.

       

      @Toshi_Esumi wrote:

      With this config, is the ISP receiving those three routes you advertised?

      Yes, my advertised routes are being received, I confirmed.

       

      @Toshi_Esumi wrote:

      I would try wrapping the prefix-list in a route-map, then try using "route-map-in/route-map-out" instead to see if that makes any change.
      If it still doesn't work, I would open a ticket at TAC to get it looked at if it's production unit.

      I did in fact completely disable the prefix-list (in and out), to remove its effect, as the ISP is only actually sending the routes I expect. But having no prefix-list did not make the routes insert.

       

      As such, I suspect that it is not a prefix issue, but rather a problem with the next-hop. I just wish I could see the issue with route inserts in the debug. The verbose debug settings suggetsed by everyone (same as the ones I usually use myself) show all sorts of details about the BGP connection itself (which TBH is usually the problem people have) but very little about the rooute selection process. I need to see why the received routes fail to insert.

       

      Upgrading to 7.4.7 (from 7.0.17, via 7.2.11) did not solve any issues as I suspected. But I will run debug again and check... .

       

      And yes, next step will be to log a ticket.

      Toshi_Esumi
      SuperUser
      Toshi_Esumi
      SuperUser
      March 3, 2025

      I'm still not sure the mechanism how FGT chooses a route to put into RIB when the same route exists in multiple protocols like static, BGP, and OSPF, etc., and what the common metrics are used. So I always avoid that situation to happen.

      Try setting a larger subnet mask for those blackhole routes, not to be the same with BGP routes. That would probably mitigate the problem.

       

      Generally the blackhole routes are not necessary when you set "snat-route-change" enabled.
      https://community.fortinet.com/t5/FortiGate/Technical-Tip-Using-SNAT-route-change-to-update-existing-NAT/ta-p/198439
      At least we don't use them.

      Toshi 

        sarlacii
        sarlaciiAuthorAnswer
        Explorer
        March 10, 2025

        Ok, so I finally figured out the issue, before I even got to FortiNet support.

         

        Turns out that a loopback interface is not treated as a "connected" subnet by the FortiGate for the purposes of BGP routes!!!

         

        To verify the BGP routes - run these debug commands:

        diagnose ip router bgp all disable # Disable all the BGP debug, so we only see what we actually want! diagnose ip router bgp updates en # We want (route) updates diagnose ip router bgp level info # Verbosity "info", "critical", "error", "warn", "none" diagnose debug enable


        Then soft reset the routes, so that you see what happens when they are added back!

        execute router clear bgp ip <neighbour ip> soft

        Eg.

        execute router clear bgp ip 172.26.200.2 soft BGP: 172.26.200.2-Outgoing [RIB] Update: Prefix 10.10.0.0/16 path_id 0 denied due to non-connected next-hop; BGP: 172.26.200.2-Outgoing [RIB] Update: Prefix 4.3.4.0/23 path_id 0 denied due to non-connected next-hop; BGP: 172.26.200.2-Outgoing [RIB] Update: Prefix 1.2.1.0/24 path_id 0 denied due to non-connected next-hop;

        The next-hop is connected! But it is a BGP loopback interface... so it is ignored, and you get a next-hop failure.

         

        Solution:

        Force the Fortigate to consider the next-hop connected, by setting the "ebgp-enforce-multihop" option for each neighbor.

        Eg.

        	config router bgp 		config neighbor 			edit "172.26.200.2" 				set ebgp-enforce-multihop enable 			next 			edit "172.26.200.22" 				set ebgp-enforce-multihop enable 			next 		end 	end

         

        See post here: Troubleshooting-Tip-BGP-debug-log-message-denied-due-to-filter 

        And here: Technical-Tip-BGP-routes-are-not-installed-in-routing-table-with-loopback 

         

          Toshi_Esumi
          SuperUser
          Toshi_Esumi
          SuperUser
          March 10, 2025

          makes sense.

          Toshi

            Terms & ConditionsAccessibility statement
            Thought Leadership Security Summit. Outpace New Threats with AI - enhanced defense. Tuesday, Septmeber 15, 8:30 AM - 2:30 PM PT. The Golf Club at Newcastle, WA.
            Fortinet Flag the Hack. Wednesday, August 26, 9:00 AM - 5:00 PM ET, COSM, Atlanta, GA.