Execute traceroute showing first and last hop 127.0.0.1 for connected subnet

[Deep_Banerji]

I have a FortiGate 100e device in which I have taken out one LAN port and set WAN role on it. I have assigned a /30 subnet IP address to the port. The port is up and I can PING it from other zones. However, I cannot PING the remote IP address of the /30 subnet. Execute traceroute shows the only hop as 127.0.0.1. I have deployed ANY-ANY policy from LAN to the above interface but PING from LAN workstation to remote /30 IP address gets DESTINATION HOST UNREACHABLE reply from firewall. I am at my wit's end. Please help.

[boneyard]

is the remote IP address in the arp table?

get sys arp

is the IP 127.0.0.1 configured on the firewall?

[Deep_Banerji]

'get sys arp' is not showing the port in question. I ran 'diag sniff packet <port> 4' which is only showing arp requests. No arp replies. But when I connect the cable to a laptop it works. Btw, this is an Internet link with publicly available IP.  Is there some problem with FGT ARP request which makes the next-hop ignore the ARP request?

No, 127.0.0.1 is not configured on the FGT. Firmware version is v5.6.4

[boneyard]

Deep Banerji wrote:

Is there some problem with FGT ARP request which makes the next-hop ignore the ARP request?

there might be if the configuration isnt correct

if you connect a laptop you say it works, you connect the laptop to the same interface on the internet modem / router?

do you configure an IP on the laptop or use DHCP?

what is the one LAN port configuration?

can you share some of the arp request sniffer output?

also please upgrade, 5.6 is not supported any more. not going to fix this, but just a good idea.

[Deep_Banerji]

Yes I'm connecting the same cable and assigning the same IP (x.y.z.189/30) manually on the FGT-100E and laptop. The LAN port members were port1 to port16 initially. I took out port15 and port16 to assign WAN role. 

edit "lan" set vdom "root" set ip 10.14.50.1 255.255.255.0 set allowaccess ping https ssh snmp http set type hard-switch set alias "LAN1" set device-identification enable set role lan set snmp-index 9 next

edit "port16" set vdom "root" set ip x.y.z.189 255.255.255.252 set allowaccess ping set type physical set role wan set snmp-index 11 next

edit "wan1" set vdom "root" set ip 10.217.7.2 255.255.255.252 set allowaccess ping https ssh http telnet set type physical set role wan set snmp-index 3 next

Next-hop IP is x.y.z.190/30 . I have tried commands like 'set arpforward enable' and 'set l2forward enable' to no avail. WAN1 and WAN2 ports are working fine.

# diag sniff packet port16 none 4 6 interfaces=[port16] filters=[none] 0.620092 port16 -- truncated 802.3ad LACPDU 64 1.644076 port16 -- truncated 802.3ad LACPDU 64 2.668155 port16 -- truncated 802.3ad LACPDU 64 3.692160 port16 -- truncated 802.3ad LACPDU 64 4.716111 port16 -- truncated 802.3ad LACPDU 64 5.740114 port16 -- truncated 802.3ad LACPDU 64

# diag sniff packet any 'arp' 4 interfaces=[any] filters=[arp] 1.067289 port16 out arp who-has x.y.z.190 tell x.y.z.189 1.617282 lan out arp who-has 10.14.50.81 tell 10.14.50.1 1.617602 lan in arp reply 10.14.50.81 is-at 0:50:56:ae:af:cf 2.067292 port16 out arp who-has x.y.z.190 tell x.y.z.189 3.071994 port16 out arp who-has x.y.z.190 tell x.y.z.189 3.905078 lan in arp who-has 10.11.50.10 tell 10.11.50.77 4.067315 port16 out arp who-has x.y.z.190 tell x.y.z.189 4.607669 lan in arp who-has 10.11.50.10 tell 10.11.50.77 4.720917 lan in arp who-has 10.14.50.252 tell 10.14.50.252 5.067294 port16 out arp who-has x.y.z.190 tell x.y.z.189 5.607677 lan in arp who-has 10.11.50.10 tell 10.11.50.77 6.073089 port16 out arp who-has x.y.z.190 tell x.y.z.189 6.950751 lan in arp who-has 10.11.50.10 tell 10.11.50.21 7.067287 port16 out arp who-has x.y.z.190 tell x.y.z.189

I have another ISP link on port15 and it is having the same issue.

[boneyard]

you seem to be sending out or receiving LACP traffic on port16, is port16 part of a link aggregate? or is the ISP router doing LACP? please turn that off for now.

also be sure you turn everything back to default which you have been changing, in the end it might not work because of all the non standard settings.

[Deep_Banerji]

No, I haven't configured port16 as member of any aggregate. I can't do anything about the ISP end config but given that it works on laptop it shouldn't be an issue.

After testing various configurations I always revert to the default/standard state.

I feel that somehow it has got to do with the fact that port16 was extracted from LAN group. That was the time when it was part of any aggregate. I read that FortiOS v5.4 onwards the internal-switch-mode is 'interface' by default so I just had to de-member it from LAN group. The device is suffering from some kind of 'LAN hangover' if I may put it that way. I have tried rebooting the device also. Nothing works. 

[boneyard]

Deep Banerji wrote:

The device is suffering from some kind of 'LAN hangover' if I may put it that way. I have tried rebooting the device also. Nothing works. 

that sounds very interesting but my experience is that things work or not, a port doesn't remember what they used to be. theoretically the change might seem executed but wasn't actually performed. a reboot should then fix that for sure.

lets try to get the exact setup right because you talk about port16, but also mention port15, share the config on wan1 and you arp requests show lan in arp who-has 10.11.50.10 tell 10.11.50.77 which isnt the subnet for lan at all.

how many interface does the ISP router / modem have? please only connect one form the FortiGate.

are you using the copper or SFP part of port16? do you have a SFP inserted?

keep it simple to start with and don't make changes to default configuration. just wondering are you able to factory reset this FortiGate and start over?

parallel to that, please try this, connect port16 to the ISP router / modem.

perform:

show | grep -f port16

show system interface port16

show system virtual-switch

show system switch-interface

execute ping x.y.z.190

get sys arp

get router info routing all

diagnose netlink aggregate list

diagnose hardware deviceinfo nic port16

diag sniff packet port16 '' 4 100 l

[Deep_Banerji]

10.11.50.0/24 is a known subnet and is connected to the LAN subnet through an L3 switch. I don't know how those broadcasts are reaching my LAN port.

# show | grep -f port16 config system interface edit "port16" <--- set vdom "root" set ip x.y.z.189 255.255.255.252 set allowaccess ping set type physical set role wan set snmp-index 11 next end config firewall policy edit 5 set name "test" set uuid <> set srcintf "lan" set dstintf "port16" <--- set srcaddr "all" set dstaddr "all" set action accept set schedule "always" set service "ALL" set logtraffic all set nat enable next end config router static edit 6 set gateway x.y.z.190 set device "port16" <--- next end

# diag sniff packet port16 '' 4 100 l interfaces=[port16] filters=[] 2020-11-08 16:11:45.771877 port16 -- arp who-has x.y.z.190 tell x.y.z.189 2020-11-08 16:11:45.888782 port16 -- truncated 802.3ad LACPDU 64 2020-11-08 16:11:46.776782 port16 -- arp who-has x.y.z.190 tell x.y.z.189 2020-11-08 16:11:46.912853 port16 -- truncated 802.3ad LACPDU 64 2020-11-08 16:11:47.771873 port16 -- arp who-has x.y.z.190 tell x.y.z.189 2020-11-08 16:11:47.936822 port16 -- truncated 802.3ad LACPDU 64 2020-11-08 16:11:48.771879 port16 -- arp who-has x.y.z.190 tell x.y.z.189 2020-11-08 16:11:48.960825 port16 -- truncated 802.3ad LACPDU 64 2020-11-08 16:11:49.776967 port16 -- arp who-has x.y.z.190 tell x.y.z.189 2020-11-08 16:11:49.984818 port16 -- truncated 802.3ad LACPDU 64 2020-11-08 16:11:50.771880 port16 -- arp who-has x.y.z.190 tell x.y.z.189 2020-11-08 16:11:51.008839 port16 -- truncated 802.3ad LACPDU 64 2020-11-08 16:11:51.771879 port16 -- arp who-has x.y.z.190 tell x.y.z.189 2020-11-08 16:11:52.032835 port16 -- truncated 802.3ad LACPDU 64 2020-11-08 16:11:52.776850 port16 -- arp who-has x.y.z.190 tell x.y.z.189 ^C 17 packets received by filter 0 packets dropped by kernel

# show sys int port16 config system interface edit "port16" set vdom "root" set ip x.y.z.189 255.255.255.252 set allowaccess ping set type physical set role wan set snmp-index 11 next end

# show sys virtual-switch config system virtual-switch edit "lan" set physical-switch "sw0" config port edit "port1" next edit "port2" next edit "port3" next edit "port4" next edit "port5" next edit "port6" next edit "port7" next edit "port8" next edit "port9" next edit "port10" next edit "port11" next edit "port12" next edit "port13" next edit "port14" next end next end

# show sys switch-interface config system switch-interface end

# show sys switch-interface config system switch-interface end

# execute ping x.y.z.190 PING x.y.z.190 (x.y.z.190): 56 data bytes

--- x.y.z.190 ping statistics --- 5 packets transmitted, 0 packets received, 100% packet loss

# get sys arp Address Age(min) Hardware Addr Interface 10.14.50.3 0 80:30:e0:84:dc:00 lan 10.14.50.80 0 00:0c:29:2c:10:30 lan 10.14.50.99 0 f4:03:43:b9:cd:a0 lan 10.217.7.5 0 00:03:0f:12:e0:3a wan2 10.14.50.79 0 00:0c:29:d2:45:39 lan 10.217.7.1 0 00:03:0f:12:e0:3a wan1 10.14.50.231 0 c4:00:ad:0b:07:33 lan 10.14.50.81 0 00:50:56:ae:af:cf lan

# get router info routing all Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP O - OSPF, IA - OSPF inter area N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 E1 - OSPF external type 1, E2 - OSPF external type 2 i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area * - candidate default

S* 0.0.0.0/0 [10/0] via x.y.z.190, port16 S 10.11.0.0/16 [10/0] via 10.14.50.3, lan S 10.12.0.0/16 [10/0] via 10.14.50.3, lan C 10.14.50.0/24 is directly connected, lan S 10.217.0.0/16 [10/0] via 10.217.7.1, wan1 [10/0] via 10.217.7.5, wan2 C 10.217.7.0/30 is directly connected, wan1 C 10.217.7.4/30 is directly connected, wan2 C x.y.z.188/30 is directly connected, port16 C a.b.c.112/29 is directly connected, port15

# diag netlink aggregate list List of 802.3ad link aggregation interfaces:

# diag netlink aggregate list List of 802.3ad link aggregation interfaces:

# diag hardware deviceinfo nic port16 Description :FortiASIC NP6LITE Adapter Driver Name :FortiASIC NP6LITE Driver Board :100E lif id :21 lif oid :85 netdev oid :85 Current_HWaddr e8:1c:ba:07:0e:ab Permanent_HWaddr e8:1c:ba:07:0e:ab ========== Link Status ========== Admin :up netdev status :up autonego_setting:1 link_setting :1 speed_setting :10 duplex_setting :0 Speed :1000 Duplex :Full link_status :Up ============ Counters =========== Rx Pkts :0 Rx Bytes :0 Tx Pkts :100724 Tx Bytes :4230454 Host Rx Pkts :163318 Host Rx Bytes :10452352 Host Tx Pkts :100725 Host Tx Bytes :4230496 Host Tx dropped :0

# diag sniffer packet port16 '' 4 100 l interfaces=[port16] filters=[] 2020-11-08 16:25:53.522050 port16 -- arp who-has x.y.z.190 tell x.y.z.189 2020-11-08 16:25:53.765784 port16 -- truncated 802.3ad LACPDU 64 2020-11-08 16:25:54.527077 port16 -- arp who-has x.y.z.190 tell x.y.z.189 2020-11-08 16:25:54.789781 port16 -- truncated 802.3ad LACPDU 64 2020-11-08 16:25:55.522045 port16 -- arp who-has x.y.z.190 tell x.y.z.189 2020-11-08 16:25:55.813793 port16 -- truncated 802.3ad LACPDU 64 2020-11-08 16:25:56.522045 port16 -- arp who-has x.y.z.190 tell x.y.z.189 2020-11-08 16:25:56.837797 port16 -- truncated 802.3ad LACPDU 64 2020-11-08 16:25:57.527064 port16 -- arp who-has x.y.z.190 tell x.y.z.189 2020-11-08 16:25:57.861866 port16 -- truncated 802.3ad LACPDU 64 2020-11-08 16:25:58.522047 port16 -- arp who-has x.y.z.190 tell x.y.z.189 2020-11-08 16:25:58.885802 port16 -- truncated 802.3ad LACPDU 64 2020-11-08 16:25:59.522043 port16 -- arp who-has x.y.z.190 tell x.y.z.189 2020-11-08 16:25:59.909816 port16 -- truncated 802.3ad LACPDU 64 ^C 15 packets received by filter 0 packets dropped by kernel

[boneyard]

thanks for all the output, i don't see anything what looks wrong except these:

2020-11-08 16:25:55.813793 port16 -- truncated 802.3ad LACPDU 64

2020-11-08 16:25:56.837797 port16 -- truncated 802.3ad LACPDU 64

2020-11-08 16:25:56.837797 port16 -- truncated 802.3ad LACPDU 64

you should not see LACPDU on a non link aggregate interface. the fact that it works on the laptop surprises me but perhaps something is configured for it or it auto reacts in a way the ISP router / modem likes better.

the ISP router / modem doesn't have another interface to use?

can you request the ISP to provide an interface on the device without LACP?

if you want to and have the time you could create a link aggregate on the FortiGate, make port16 a member of it and see if that gets a working situation. that does require removing the IP from port16, removing the firewall policy and the route. then putting them back on the link aggregate.