FortiLink over QinQ tunnel

[nreederToN]

Hi All,

We have an ISP providing us layer 2 to several sites via a QinQ tunnel. I have seen having an ISP in between devices is not supported for a L2 FortiLink; however, several people have had success doing so.

The issue is currently that FortiLink establishes for a small period of time if freshly factory reset and mgmt vlan settings are changed to 4094. This will establish the link, then FortiGate sends configs. After a while the CAPWAP tunnel goes down and the switch never comes back online.

FortiNet support mentioned this could be an STP issue, though they weren't willing to further investigate as the ISP devices may be playing a role. I confirmed via packet capture the only STP packets being received are that of the upstream FortiSwitch. 

Any insight to how these setups are working, or configuration that assist this would be helpful! We are very close with the ISP, and they are always willing to work with us, so any insight to what they might be able to do is also appreciated.

[nreederToN]

With set fortilink-p2p enable on the physical interface we can get the switch to come up temporarily.

We get the following output after the switch has gone offline:

Spoiler

date=2023-09-07 time=11:53:57 eventtime=1694105638049591194 tz="-0500" logid="0115022871" type="event" subtype="switch-controller" level="information" vd="root" logdesc="NAC MAC cache sync" user="Switch-Controller" ui="flpold" action="nac-mac-sync" sn="S448EFTF23006719" name="S448EFTF23006719" msg="NAC MAC cache cleared on switch S448EFTF23006719 port (null)"
date=2023-09-07 time=11:53:57 eventtime=1694105638047983320 tz="-0500" logid="0115032606" type="event" subtype="switch-controller" level="warning" vd="root" logdesc="Switch-Controller Tunnel Down" user="Switch-Controller" ui="cu_acd" sn="S448EFTF23006719" name="S448EFTF23006719" msg="CAPWAP Tunnel Down"
date=2023-09-07 time=11:53:57 eventtime=1694105638047957829 tz="-0500" logid="0115022904" type="event" subtype="switch-controller" level="notice" vd="root" logdesc="CAPUTP session status notification" user="Switch-Controller" ui="cu_acd" sn="S448EFTF23006719" name="S448EFTF23006719" msg="S448EFTF23006719 echo message timed out" action="session-leave" srcip=172.17.65.8
date=2023-09-07 time=11:47:45 eventtime=1694105265784772082 tz="-0500" logid="0115022892" type="event" subtype="switch-controller" level="information" vd="root" logdesc="Switch-Controller Switch Sync Complete" user="Switch-Controller" ui="flcfgd" sn="S448EFTF23006719" name="S448EFTF23006719" msg="Config download successful"
date=2023-09-07 time=11:47:31 eventtime=1694105251571350735 tz="-0500" logid="0115032697" type="event" subtype="switch-controller" level="notice" vd="root" logdesc="FortiSwitch switch" user="Fortilink" sn="S448EFTF23006719" name="S448EFTF23006719" msg="Switch-Controller: connected with FortiGate"
date=2023-09-07 time=11:47:31 eventtime=1694105251569771689 tz="-0500" logid="0115032699" type="event" subtype="switch-controller" level="alert" vd="root" logdesc="FortiSwitch system" user="Fortilink" sn="S448EFTF23006719" name="S448EFTF23006719" msg="Configuration is changed in the admin session"
date=2023-09-07 time=11:47:31 eventtime=1694105251566597389 tz="-0500" logid="0115032699" type="event" subtype="switch-controller" level="notice" vd="root" logdesc="FortiSwitch system" user="Fortilink" sn="S448EFTF23006719" name="S448EFTF23006719" msg="Automatic configuration backup to flash disk succeeded"
date=2023-09-07 time=11:47:31 eventtime=1694105251565023009 tz="-0500" logid="0115032699" type="event" subtype="switch-controller" level="notice" vd="root" logdesc="FortiSwitch system" user="Fortilink" sn="S448EFTF23006719" name="S448EFTF23006719" msg="The ntp daemon step adjusted time from Fri Jan  2 15:07:33 1970 to Thu Sep  7 11:47:19 2023 (sync source: 172.17.65.1)"
date=2023-09-07 time=11:47:31 eventtime=1694105251563435888 tz="-0500" logid="0115032699" type="event" subtype="switch-controller" level="notice" vd="root" logdesc="FortiSwitch system" user="Fortilink" sn="S448EFTF23006719" name="S448EFTF23006719" msg="The IPv4 ntp server, 172.17.65.1(172.17.65.1), is determined reachable at Fri Jan  2 14:07:26 1970"
date=2023-09-07 time=11:47:31 eventtime=1694105251561853901 tz="-0500" logid="0115032696" type="event" subtype="switch-controller" level="notice" vd="root" logdesc="FortiSwitch spanning Tree" user="Fortilink" sn="S448EFTF23006719" name="S448EFTF23006719" msg="primary port _FlInK1_MLAG0_ instance 15 changed state from learning to forwarding"
date=2023-09-07 time=11:47:31 eventtime=1694105251560242336 tz="-0500" logid="0115032696" type="event" subtype="switch-controller" level="notice" vd="root" logdesc="FortiSwitch spanning Tree" user="Fortilink" sn="S448EFTF23006719" name="S448EFTF23006719" msg="primary port _FlInK1_MLAG0_ instance 0 changed state from learning to forwarding"
date=2023-09-07 time=11:47:31 eventtime=1694105251558629920 tz="-0500" logid="0115032696" type="event" subtype="switch-controller" level="notice" vd="root" logdesc="FortiSwitch spanning Tree" user="Fortilink" sn="S448EFTF23006719" name="S448EFTF23006719" msg="primary port _FlInK1_MLAG0_ instance 15 changed state from discarding to learning"
date=2023-09-07 time=11:47:31 eventtime=1694105251557009486 tz="-0500" logid="0115032696" type="event" subtype="switch-controller" level="notice" vd="root" logdesc="FortiSwitch spanning Tree" user="Fortilink" sn="S448EFTF23006719" name="S448EFTF23006719" msg="primary port _FlInK1_MLAG0_ instance 0 changed state from discarding to learning"
date=2023-09-07 time=11:47:31 eventtime=1694105251555398984 tz="-0500" logid="0115032696" type="event" subtype="switch-controller" level="notice" vd="root" logdesc="FortiSwitch spanning Tree" user="Fortilink" sn="S448EFTF23006719" name="S448EFTF23006719" msg="primary port _FlInK1_MLAG0_ instance 15 changed role from disabled to designated"
date=2023-09-07 time=11:47:31 eventtime=1694105251553788567 tz="-0500" logid="0115032696" type="event" subtype="switch-controller" level="notice" vd="root" logdesc="FortiSwitch spanning Tree" user="Fortilink" sn="S448EFTF23006719" name="S448EFTF23006719" msg="primary port _FlInK1_MLAG0_ instance 0 changed role from disabled to designated"
date=2023-09-07 time=11:47:31 eventtime=1694105251552177728 tz="-0500" logid="0115032696" type="event" subtype="switch-controller" level="notice" vd="root" logdesc="FortiSwitch spanning Tree" user="Fortilink" sn="S448EFTF23006719" name="S448EFTF23006719" msg="primary port _FlInK1_MLAG0_ instance 0 changed role from designated to disabled"

The behavior seems odd. Again, any insight would be appreciated!

[helpIT-Hartmann]

I have exact the Same issue!

Did you find any solution?

Switch comes online, i do authorize, it gets its name and goes offline... spf+ port is back to factory defaults. No FortiLink Port...

I would appreciate any help!

[SaitAdibelli]

Hi there

i was successfull with the following configuration over mpls --> 

FortiLink over a point-to-point layer-2 network

Starting in FortiSwitchOS 6.4.0, you can run FortiLink mode over a point-to-point layer-2 network. You can form an inter-switch link (ISL) between two FortiSwitch units over a layer-2 device or non-FortiSwitch device (such as a wireless bridge). The LLDP destination MAC address is changed to the broadcast MAC address to bypass middle layer-2 devices. For example:

To create this topology, you configure ports on both ends of the link as described in the following procedure and, optionally, configure the tag protocol identifier (TPID) between the two FortiSwitch units.

NOTE:

• The set fortilink-p2p command is available in FortiLink mode and standalone mode. The set fortilink-p2p-tpid command is available only in FortiLink mode.
• The FS-124E, FS-124E-POE, FS-124E-FPOE, FS-148E, FS-148E-POE, FS-148F, FS-148F-POE, FS-148F-FPOE, FS-124F, FS-124F-POE, and FS-124F-FPOE models support only the default 0x8100 TPID; TPID changes are not supported.

1. Enable the FortiLink point-to-point network on each FortiSwitch unit:

config switch physical-port

edit <port_name>

set fortilink-p2p enable

end

1. Make certain that the FortiLink point-to-point TPID value is the same on each FortiSwitch unit. By default, it is 0x8100.

config switch global

set fortilink-p2p-tpid <0x0001-0xfffe>

end

on switch2 do the following change also 

config switch auto-network

      set mgmt-vlan 4094

      set status enable

end

[helpIT-Hartmann]

I tried p2p too. But didn't come online in the first step (doesn't show up after factoryreset and management vlan set to 4094).