Learned MAC address won't be used unless added as a static ARP entry

[umutkacar]

---------------------------------------------

FortiGate 500E on v6.0.4 build0231 (GA)

---------------------------------------------

Hello everyone,

I'm trying to establish a BGP session between our FortiGate and AWS, for the AWS Direct Connect (DX) service.

We have connection across the line. Both FortiGate and AWS peer can discover each other's MAC addresses.

(I can see AWS peer's MAC address on diagnose sniffer output, and confirmed with AWS support that they have our FortiGate's MAC address on their router.)

The trouble is, FortiGate says AWS peer IP is unreachable. Here is a sample output from the diagnose sniffer:

14114.628105 port8 out arp who-has 169.254.237.21 tell 169.254.237.22

14114.665189 port8 in arp reply 169.254.237.21 is-at 12:a1:78:5b:38:a2 14115.628231 VDOM01 out 169.254.237.22 -> 169.254.237.22: icmp: host 169.254.237.21 unreachable

(169.254.237.22 is FortiGate, 169.254.237.21 is AWS peer. MAC address and vdom name are faux for display)

(12:a1:78:5b:38:a2 is not on get system arp output despite learnt actually)

The situation changes if I add a static ARP entry like this:

config system arp-table edit 1 set interface "AWS-DX2" set ip 169.254.237.21 set mac 12:a1:78:5b:38:a2 next end

After applying this and having AWS peer's MAC address on our FortiGate's ARP table and can be displayed on get system arp output.

This way, FortiGate can now reach 169.254.237.21 and BGP session is UP. But only for a while...

Because when I add the static ARP entry, AWS peer can no longer learn our FortiGate's MAC address, and our MAC address stays only for about 20 minutes on AWS peer's router's cache. When it's cleared from AWS peer, they no longer know our MAC, so AWS peer starts to send ARP requests to learn our MAC, but our FortiGate does not reply them when we have a static ARP entry:

1726.313758 port8 in arp who-has 169.254.237.22 tell 169.254.237.21 1742.513386 port8 in arp who-has 169.254.237.22 tell 169.254.237.21 1755.313179 port8 in arp who-has 169.254.237.22 tell 169.254.237.21 1772.912821 port8 in arp who-has 169.254.237.22 tell 169.254.237.21

(it starts to reply AWS peer with our MAC address if I delete the static ARP entry on our side, but then FortiGate won't connect to AWS peer)

hence cannot communicate to us and BGP session goes down like this:

BGP: %BGP-3-NOTIFICATION: sending to 169.254.237.21 4/0 (Hold Timer Expired/Unspecified Error Subcode) 0 data-bytes [] id=20300 logdesc="BGP neighbor status changed" msg="BGP: %BGP-5-ADJCHANGE: neighbor 169.254.237.21 Down Hold Timer Expired" id=20300 logdesc="BGP neighbor status changed" msg="BGP: %BGP-5-ADJCHANGE: neighbor 169.254.237.21 Down BGP Notification FSM-ERR" BGP: [GRST] Timer Announce Defer: Check BGP: 169.254.237.21-Outgoing [FSM] State: Idle Event: 3 BGP: [RIB] Scanning BGP Network Routes... BGP: [RIB] Scanning BGP RIB... BGP: [RIB] Scanning BGP Network Routes... BGP: [RIB] Scanning BGP Network Routes... BGP: [RIB] Scanning BGP Network Routes... BGP: 169.254.237.21-Outgoing [NETWORK] FD=25, Sock Status: 110-Connection timed out

If I delete the static ARP entry for AWS peer and wait for couple of minutes, AWS peer learns our FortiGate's MAC address again.

Shortly after that, if I re-add the static ARP entry for AWS peer, BGP session comes UP, again for only about 20 minutes.

And this goes on...

I need to make FortiGate connect AWS peer without me having to add static ARP entry for AWS peer. It already discovers it by itself. Why do you think it won't add it to its ARP table and connect?

Any help would be appreciated.

Thanks in advance!

[hubertzw]

Can you say more about your setup? I don't understand why arp is sent out of port8, but icmp out of VDOM01. Do you have any routing in place which may route your packet to another VDOM?

14114.628105 port8 out arp who-has 169.254.237.21 tell 169.254.237.22 14114.665189 port8 in arp reply 169.254.237.21 is-at 12:a1:78:5b:38:a2 14115.628231 VDOM01 out 169.254.237.22 -> 169.254.237.22: icmp: host 169.254.237.21 unreachable

[umutkacar]

hubertzw wrote:

Can you say more about your setup? I don't understand why arp is sent out of port8, but icmp out of VDOM01. Do you have any routing in place which may route your packet to another VDOM?

14114.628105 port8 out arp who-has 169.254.237.21 tell 169.254.237.22 14114.665189 port8 in arp reply 169.254.237.21 is-at 12:a1:78:5b:38:a2 14115.628231 VDOM01 out 169.254.237.22 -> 169.254.237.22: icmp: host 169.254.237.21 unreachable

Hello hubertzw, thanks for your reply!

We are working in a VDOM01, so not the default VDOM called root, or vd-root. I think it shows the packets going in and out of VDOM01 as described here as "Traffic should come in and leave the VDOM." Here is a sniff output both showing VDOM01 in and out:

554.595716 VDOM01 out 169.254.237.22 -> 169.254.237.22: icmp: host 169.254.237.21 unreachable 554.595718 VDOM01 in 169.254.237.22 -> 169.254.237.22: icmp: host 169.254.237.21 unreachable

There are no routes for this network in particular, other than the only route that says it's directly connected:

C       169.254.237.20/30 is directly connected, AWS-DX2

Our setup is basically like this (and then some summary of the issue):

[ul]

AWS Direct Connect (DX) cable is connected to our FortiGate's port8

There is a virtual port to tag required VLAN under port8, called AWS-DX2, tagged with VLAN 2013

We got no L2 network connectivity issues, have link between FortiGate and AWS peer fine

Only problem is FortiGate's not getting AWS peer's MAC address on it's ARP table, despite learning it by itself from AWS peer

If we use the workaround of adding a static-arp entry, then AWS peer cannot learn our FortiGate's MAC address, because somehow FortiGate won't answer them when they ask for our MAC, and BGP connection is lost after our MAC address gets deleted from AWS peer router's cache

AWS peer can learn our FortiGate's port8 MAC address fine, if we do not use static-arp entry

We need to find a way to make FortiGate use the MAC address of AWS peer it learned by itself[/ul]

[hubertzw]

Do you see any packets when you run 'diag sniffer' and you initiate ICMP from Amazon?

What is the software version?

[umutkacar]

hubertzw wrote:

Do you see any packets when you run 'diag sniffer' and you initiate ICMP from Amazon?

What is the software version?

I cannot initiate ICMP from AWS since I have no control on the router there. Also the guy from AWS support said he also cannot access a shell of that device but only see logs and outputs etc. 

But I can ping AWS peer if I add static-arp entry for its IP address. So AWS peer should also be able to ping our FortiGate for the 20 minute window, beginning from when I first add the static-arp entry, until our FortiGate's MAC address disappear from their cache.

I mentioned the software version on the first post. That is v6.0.4 build0231 (GA)