Fortigate BGP community problem

[Mantovani]

I’m working with a BGP setup involving three FortiGate firewalls:

FW-A

FW-B

FW-C

FW-A has two BGP neighbors with FW-B: one over a point-to-point link and another over an IPsec overlay tunnel. Both neighbors advertise the same routes to FW-A. However, I need FW-B to only advertise the routes it receives via the point-to-point neighbor.

To achieve this, on FW-A I apply an outbound route-map to the point-to-point neighbor with FW-B, matching the route 192.168.10.0/24 and tagging it with the BGP community 65002:200. FW-A also sends this same route to the IPsec overlay neighbor, but without any community tag.

On FW-B, this results in receiving the 192.168.10.0/24 route from both neighbors — one copy with the community 65002:200 (from the point-to-point link), and one copy without the community (from the overlay).

FW-B also peers with FW-C. Between these two, I configure a route-map outbound on FW-B that matches routes with the community 65002:200, so that only routes received from the point-to-point neighbor (the ones tagged with 65002:200) are advertised to FW-C.

Initially, this works as expected: FW-C receives the route with the 65002:200 community, and when the point-to-point BGP neighbor between FW-A and FW-B goes down, FW-C correctly withdraws the route since it's no longer being received with that community.

The issue arises when the point-to-point neighbor comes back up: although FW-A again starts sending the route with the 65002:200 community, FW-C does not receive it anymore. It's as if BGP on FW-B does not re-evaluate or reprocess the route attributes (community) after the neighbor is re-established.

The only workaround that makes FW-C receive the route again is to perform a manual clear bgp all on both FW-B and FW-C.

After FW-C point to point neighbor come up, FW-B has best route over another neighbors to 192.168.10.0/24 and that´s why it insn´t propagating to FW-C. When I keep only point to point neighbor it works as expected. My routing table will be dynamic because of SDWAN over a lot of neighbors. Anyone has any idea how to solve this problem?

[Mantovani]

Here is the FW-B to FW-C route-map out:

FW-B # config router bgp

FW-B (bgp) # config neighbor

FW-B (neighbor) # edit 192.168.101.2

FW-B (192.168.101.2) # show
config neighbor
edit "192.168.101.2"
set capability-graceful-restart enable
set soft-reconfiguration enable
set interface "port2"
set remote-as 65012
set route-map-out "OUT-Comm"
next
end

FW-B (192.168.101.2) # end

FW-B (bgp) # end

FW-B # config router route-map

FW-B (route-map) # edit OUT-Comm

FW-B (OUT-Comm) # show
config router route-map
edit "OUT-Comm"
config rule
edit 1
set match-community "FWA-Routes"
next
end
next
end

FW-B (OUT-Comm) # end

FW-B # config router community-list

FW-B (community-list) #
FW-B (community-list) # edit FWA-Routes

FW-B (FWA-Routes) # show
config router community-list
edit "FWA-Routes"
config rule
edit 1
set action permit
set match "65002:200"
next
end
next
end

FW-B (FWA-Routes) #

[Toshi_Esumi]

It looks fine to me.
But FW-C/Oracle wouldn't care if it came over p2p between A-B or over IPsec between A-B. Only thing it cares is it needs to send the packets to the destination to FW-B. So unless both p2p and IPsec goes down, it would have the next hop as FW-B.
This maybe the basic feature/behavior/design of BGP, not only FGTs.

Toshi

[Toshi_Esumi]

You can confirm this with "get router info neighbor 192.168.101.2 advertised-routes" at FW-B.

[Toshi_Esumi]

Or, you could set a higher local-preference than default 100 for the routes the FW-B received over the p2p, instead of enabling ECMP. You want FW-B to always prefer p2p over IPsec, right?

[Mantovani]

The traffic through p2p is only for oracle networks and some other traffic configured via SDWAN. I need to have this p2p underlay (only oracle traffic) and overlay (local  traffic). Using local pref is not a good choice because all traffic will use this p2p link. If the p2p neighbor goes down and come back, FW-B isn´t sending FW-A routes to FW-C. The only way is to clear IP bgp all on both neighbors which will cause a big disruption. The community cenario is a good choice, but I don´t know why it isn´t sending the routes after p2p come back. Do you have any idea? I tested and it is clear that is because FW-B has a better route via SDWAN, that´s why it isn´t sending the route. I would like to understand why it is happening only with the community. Thank you for your time Toshi

[Toshi_Esumi]

You're talking about only FW-A, how to utilize the p2p and IPsec from its end. And, the traffic destined to Oracle/FW-C need to be nailed down to the p2p. Then you should put a (set of) static route(s) for Oracle traffic toward p2p. BGP should be used only for the traffic that need to change the path. Like failing over to IPsec side.
It has nothing to do with which route the FW-C is seeing, becasue, again, FW-C's just need to route traffic for FW-A to FW-B. Nothing else it needs to know.

Toshi

[Toshi_Esumi]

Above comment covers transactions/sessions initiated from FW-A side. If Oracle itself needs to initiate transactions and you want to route it only to the p2p side, not using IPsec path, when they traverse through FW-B, you need to have a policy routes (or include it in SD-WAN members on FW-B and use rules, which are policy routes).
Since you need to route/direct traffic based on traffic types or sources (Oracle traffic) at FW-B for this direction, BGP is not the way you can accomplish what you want. Those routing protocols, like BGP, OSPF, etc.work based only on destinations.

Toshi