FortiLink over QinQ tunnel

[nreederToN]

Hi All,

We have an ISP providing us layer 2 to several sites via a QinQ tunnel. I have seen having an ISP in between devices is not supported for a L2 FortiLink; however, several people have had success doing so.

The issue is currently that FortiLink establishes for a small period of time if freshly factory reset and mgmt vlan settings are changed to 4094. This will establish the link, then FortiGate sends configs. After a while the CAPWAP tunnel goes down and the switch never comes back online.

FortiNet support mentioned this could be an STP issue, though they weren't willing to further investigate as the ISP devices may be playing a role. I confirmed via packet capture the only STP packets being received are that of the upstream FortiSwitch. 

Any insight to how these setups are working, or configuration that assist this would be helpful! We are very close with the ISP, and they are always willing to work with us, so any insight to what they might be able to do is also appreciated.

[nreederToN]

With set fortilink-p2p enable on the physical interface we can get the switch to come up temporarily.

We get the following output after the switch has gone offline:

Spoiler

date=2023-09-07 time=11:53:57 eventtime=1694105638049591194 tz="-0500" logid="0115022871" type="event" subtype="switch-controller" level="information" vd="root" logdesc="NAC MAC cache sync" user="Switch-Controller" ui="flpold" action="nac-mac-sync" sn="S448EFTF23006719" name="S448EFTF23006719" msg="NAC MAC cache cleared on switch S448EFTF23006719 port (null)"
date=2023-09-07 time=11:53:57 eventtime=1694105638047983320 tz="-0500" logid="0115032606" type="event" subtype="switch-controller" level="warning" vd="root" logdesc="Switch-Controller Tunnel Down" user="Switch-Controller" ui="cu_acd" sn="S448EFTF23006719" name="S448EFTF23006719" msg="CAPWAP Tunnel Down"
date=2023-09-07 time=11:53:57 eventtime=1694105638047957829 tz="-0500" logid="0115022904" type="event" subtype="switch-controller" level="notice" vd="root" logdesc="CAPUTP session status notification" user="Switch-Controller" ui="cu_acd" sn="S448EFTF23006719" name="S448EFTF23006719" msg="S448EFTF23006719 echo message timed out" action="session-leave" srcip=172.17.65.8
date=2023-09-07 time=11:47:45 eventtime=1694105265784772082 tz="-0500" logid="0115022892" type="event" subtype="switch-controller" level="information" vd="root" logdesc="Switch-Controller Switch Sync Complete" user="Switch-Controller" ui="flcfgd" sn="S448EFTF23006719" name="S448EFTF23006719" msg="Config download successful"
date=2023-09-07 time=11:47:31 eventtime=1694105251571350735 tz="-0500" logid="0115032697" type="event" subtype="switch-controller" level="notice" vd="root" logdesc="FortiSwitch switch" user="Fortilink" sn="S448EFTF23006719" name="S448EFTF23006719" msg="Switch-Controller: connected with FortiGate"
date=2023-09-07 time=11:47:31 eventtime=1694105251569771689 tz="-0500" logid="0115032699" type="event" subtype="switch-controller" level="alert" vd="root" logdesc="FortiSwitch system" user="Fortilink" sn="S448EFTF23006719" name="S448EFTF23006719" msg="Configuration is changed in the admin session"
date=2023-09-07 time=11:47:31 eventtime=1694105251566597389 tz="-0500" logid="0115032699" type="event" subtype="switch-controller" level="notice" vd="root" logdesc="FortiSwitch system" user="Fortilink" sn="S448EFTF23006719" name="S448EFTF23006719" msg="Automatic configuration backup to flash disk succeeded"
date=2023-09-07 time=11:47:31 eventtime=1694105251565023009 tz="-0500" logid="0115032699" type="event" subtype="switch-controller" level="notice" vd="root" logdesc="FortiSwitch system" user="Fortilink" sn="S448EFTF23006719" name="S448EFTF23006719" msg="The ntp daemon step adjusted time from Fri Jan  2 15:07:33 1970 to Thu Sep  7 11:47:19 2023 (sync source: 172.17.65.1)"
date=2023-09-07 time=11:47:31 eventtime=1694105251563435888 tz="-0500" logid="0115032699" type="event" subtype="switch-controller" level="notice" vd="root" logdesc="FortiSwitch system" user="Fortilink" sn="S448EFTF23006719" name="S448EFTF23006719" msg="The IPv4 ntp server, 172.17.65.1(172.17.65.1), is determined reachable at Fri Jan  2 14:07:26 1970"
date=2023-09-07 time=11:47:31 eventtime=1694105251561853901 tz="-0500" logid="0115032696" type="event" subtype="switch-controller" level="notice" vd="root" logdesc="FortiSwitch spanning Tree" user="Fortilink" sn="S448EFTF23006719" name="S448EFTF23006719" msg="primary port _FlInK1_MLAG0_ instance 15 changed state from learning to forwarding"
date=2023-09-07 time=11:47:31 eventtime=1694105251560242336 tz="-0500" logid="0115032696" type="event" subtype="switch-controller" level="notice" vd="root" logdesc="FortiSwitch spanning Tree" user="Fortilink" sn="S448EFTF23006719" name="S448EFTF23006719" msg="primary port _FlInK1_MLAG0_ instance 0 changed state from learning to forwarding"
date=2023-09-07 time=11:47:31 eventtime=1694105251558629920 tz="-0500" logid="0115032696" type="event" subtype="switch-controller" level="notice" vd="root" logdesc="FortiSwitch spanning Tree" user="Fortilink" sn="S448EFTF23006719" name="S448EFTF23006719" msg="primary port _FlInK1_MLAG0_ instance 15 changed state from discarding to learning"
date=2023-09-07 time=11:47:31 eventtime=1694105251557009486 tz="-0500" logid="0115032696" type="event" subtype="switch-controller" level="notice" vd="root" logdesc="FortiSwitch spanning Tree" user="Fortilink" sn="S448EFTF23006719" name="S448EFTF23006719" msg="primary port _FlInK1_MLAG0_ instance 0 changed state from discarding to learning"
date=2023-09-07 time=11:47:31 eventtime=1694105251555398984 tz="-0500" logid="0115032696" type="event" subtype="switch-controller" level="notice" vd="root" logdesc="FortiSwitch spanning Tree" user="Fortilink" sn="S448EFTF23006719" name="S448EFTF23006719" msg="primary port _FlInK1_MLAG0_ instance 15 changed role from disabled to designated"
date=2023-09-07 time=11:47:31 eventtime=1694105251553788567 tz="-0500" logid="0115032696" type="event" subtype="switch-controller" level="notice" vd="root" logdesc="FortiSwitch spanning Tree" user="Fortilink" sn="S448EFTF23006719" name="S448EFTF23006719" msg="primary port _FlInK1_MLAG0_ instance 0 changed role from disabled to designated"
date=2023-09-07 time=11:47:31 eventtime=1694105251552177728 tz="-0500" logid="0115032696" type="event" subtype="switch-controller" level="notice" vd="root" logdesc="FortiSwitch spanning Tree" user="Fortilink" sn="S448EFTF23006719" name="S448EFTF23006719" msg="primary port _FlInK1_MLAG0_ instance 0 changed role from designated to disabled"

The behavior seems odd. Again, any insight would be appreciated!