Possible Routing loop with BGP configuration,

[bcote]

Hi,

I have a pair of 1500D(A/P) connecting to our ISP using BGP and seem to be having some routing issues. We have bizarre reports of emails bouncing, web servers being unreachable for some external parties but not all...

The reason I think it might be some type of routing issue is mostly because if I do a traceroute (on a server I can reach), I actually hit the Firewall (BGP Router ID) twice and then a timeout before I actually reach the final destination. So as an example(using fake IP's) :

Tracing route to 8.8.8.8 over a maximum of 30 hops

1 <1 ms <1 ms <1 ms 10.10.10.10 2 <1 ms <1 ms <1 ms 192.168.7.21 3 <1 ms <1 ms <1 ms 192.168.2.2 4 1 ms 1 ms 1 ms 65.28.100.45 5 3 ms 3 ms 3 ms 68.67.63.189 6 7 ms 7 ms 7 ms 68.67.63.251 7 15 ms 7 ms 7 ms 206.108.34.6 8 8 ms 7 ms 7 ms 108.170.250.243 9 21 ms 21 ms 21 ms 216.239.46.162 10 31 ms 31 ms 31 ms 100.100.231.22 -> Fortigate 11 32 ms 31 ms 31 ms 100.100.231.22 -> Fortigate 12 * * * Request timed out. 13 31 ms 31 ms 31 ms 8.8.8.8 -> Final destination

Trace complete.

I hadn't seen this before so I did a bit of research and it seems to be pointing at a routing loop, which would explain some of my issues. I am wondering if anyone using BGP or other routing protocol with a Fortigate, have encountered this type of problem and maybe get some insight on what could cause it. I will open a TAC case, but I wanted to see if anyone had some experience in the matter.

Thanks so much,

Ben

[emnoc]

I would look at the bgp-table if possible. I would  really guess your problem is  really dealing with bgp convergence issues  & where 1> where a link does down 2> the  updates are not populate as fast as the failure 3>and between al bgp routers betweens point A and Z and ASN#s  A thru Z.

So some routers in between let's say two points A &  Z ,  have not   the most correct bgp.path information due to the convergence lag. This is not a flaw in bgp, just how it works. It's fast but not   100% fast like OSPF or EIGRP.

just my 2cts ,

[ul]

                   try to grab the bgp.path tables and see what the bgp.path shows.[/ul][ul]

                    have you reviewed in  bgp-replay and router updates due these issues                     

                    are your links to your ISP provider flapping

                     Is you  provide links to his uplinks flapping[/ul]

Ken

[bcote]

Hi Ken,

I ran the bgp paths command but other than showing the AS number of my provider, it doesn't show me much, at least that I understand : 

Fortigate (root) # get router info bgp paths Address Refcnt Path [0x7fc0c5edc130:0] (24) [0x7fc0c5edbfc8:215] (1) 22652

Currently, although I am configured as an SD-WAN interface to have redundancy, I only have access to one link(provider is physical adding the second fiber in the next weeks). The configuration on the SD-WAN link is to simply weight all to the only available link, but could that cause an issue? I don't see why but I wanted to mention it in case.

I don't see any flapping on my end and last I checked with my provider, it wasn't flapping either, but I do want to get a new ticket open with them so we dig a bit deeper. I am not as familiar with BGP and this is a new configuration with a new provider so it is possible that there is an issue on that end.

As far as the BGP table, that's where I am a bit confused. Here is how the provider is configured.. basically on their end, they allow 0.0.0.0/0 and on my end, I create routes from our owned public subnets. I split it up in /24 routes so to give you an example.

my BGP configuration has networks similar to this:

100.100.224.0/24

100.100.225.0/24

....

100.100.245.0/24

and so on. Initially, I had configured the full /19 that we own, but for some reason, it didn't seem to work properly. Once I switched to /24, all of a sudden, the networks would appear when I used " get router info bgp network" and traffic would flow through. Then I have a static route for each of these /24 going to my internal LAN interface. 

where I think my configuration is bizarre, or maybe I simply don't understand, is that if I run "get router info routing-table bgp", I get nothing. If I run "get router info routing-table all" I get of course all my routes "static and connected" but no BGP. The only time I see a BGP route, is when I run "get router info routing-table database".

Would this imply that not seeing BGP routes when running the other commands, that there is something that isn't configured properly, either on the Fortigate side or on the provider side?

Thanks again for all your help Ken,

Ben

[bcote]

Little update,

finally was able to get in touch with our old provider which still hadn't cleaned properly our subnets from their routes.. meaning that some external clients that actually had them as a provider or even the shortest path would try to go through them and then the traffic would simply die off. It took over a week to get someone from that provider to do the changes. It was like playing whack-a-mole and never getting the right technical person to do the changes.

Thanks,

Ben