Access to Fortiswitch via Fortilink

[erussell]

Hello all I have recently inherited a network of Fortinet gear.  It was mostly setup reasonably well, but there were a number of issues.  It consists of 2 60F which are setup in HA.  3 FortiSwitch 248E-FPOE, and 6 Forti AP 221E.  I have managed to work out most of what needed to be done.  However one odd thing lingers.

I have all my FortiSwitch online and accessible through FortiLink, as well as Forti AP managed.  The FortiLink address is 10.109.92.1, and my FortiSwitch are 10.109.92.2/3/4.  From any Fortiswitch I can ping and access the other (for instance from 10.109.92.2 I can SSH to one of the other FortiSwitch at 10.109.2.3).  However I cannot from any switch ping the FortiLink interface of 10.109.92.1, nor from the GUI can I access any Fortiswitch via CLI.  My only option right now to access the Fortiswitch directly if I wanted to is to use a console cable.

One challenge I faced was that when I was given this setup the Fortiswitch would not connect via FortiLink.  What I discovered was that NTP was not functional.  The FortiSwitch had the FortiLink IP as NTP but since none of the switches can reach that IP NTP was not functional.  I worked around it temporarily by writing a CLI Firewall policy between FortiLink and the Internet.  I then used a NTP Server on the public internet and then each switch connected via FortiLink.  

So something on my 60F is blocking this traffic.  I can post any config necessary to assist with figuring this out.  I am not sure I explained this properly.

[Markus_M]

Hi erussell,

sounds like a nice present.

Please check whether this fortilink interface is actually a fortilink interface and not some LACP aggregate. Can you reach the switches from the FGT?

This is the interface setting from my managed switch, see that your allowaccess settings are correct.

config system interface
 edit "internal"
  set allowaccess ping https ssh snmp
  set type physical
  set defaultgw enable
 next
end

Also make sure you have the switch on the management VDOM, if you have VDOMs.

Best regards,

Markus

[erussell]

Thanks Markus, I appreciate you taking a stab at it.  Here is the relevant stanza from one of my managed switches.

config system interface
edit "mgmt"
set allowaccess ping https ssh
set type physical
set snmp-index 55
next
edit "internal"
set ip 10.109.92.3 255.255.255.0
set allowaccess ping https ssh
set type physical
set snmp-index 54

Note that I have my internal on my machined switches set to a static IP, the behavior is the same if I change them to DHCP.

It is definitely a Fortilink interface:

edit "FortiLink"
set vdom "root"
set fortilink enable
set ip 10.109.92.1 255.255.255.0
set allowaccess ping fabric
set type aggregate
set member "internal1" "internal2" "internal3" "internal4" "internal5"
set lldp-reception enable
set lldp-transmission enable
set snmp-index 13
set auto-auth-extension-device enable
set switch-controller-nac "FortiLink"
set switch-controller-dynamic "FortiLink"
set swc-first-create 255
set lacp-mode static
next

I am not 100% sure about management VDOM, that you mention, but here is some info from my FGT:

show full system global | grep management-vdom

set management-vdom "root"

To clarify, from the GUI the 3 switches are fully connected via FortiLink, I can get any info I need or configure ports on the 3 switches without issue.  From the FGT I cannot SSH to or even ping any of the 3 switches across the Fortilink.  From the switch side I can reach any switch from any other, but I cannot ping nor access the FGT across the Fortilink.

[erussell]

One other thing I have noticed is that there is a sync error on each of the Fortiswitch:

execute switch-controller get-conn-status
Managed-devices in current vdom root:

FortiLink interface : FortiLink
SWITCH-ID VERSION STATUS FLAG ADDRESS JOIN-TIME NAME
S248EFTF19006074 v7.2.0 (393) Authorized/Up E 10.109.92.3 Tue Jun 21 23:50:33 2022 IFB-SW202
S248EFTF19006082 v7.2.0 (393) Authorized/Up E 10.109.92.2 Tue Jun 21 23:51:16 2022 IFB-SW201
S248EFTF19006253 v7.2.0 (393) Authorized/Up E 10.109.92.4 Wed Jun 22 16:25:18 2022 IFB-SW203

Flags: C=config sync, U=upgrading, S=staged, D=delayed reboot pending, E=config sync error, 3=L3
Managed-Switches: 3 (UP: 3 DOWN: 0 MAX: 16)

Which is likely being caused by this issue.

Does anyone have any further thoughts or troubleshooting paths?

[Markus_M]

You could try to run a packet capture on the FortiSwitch:

diag sniffer packet any 'icmp' 4

And ping the switch you are connected to. You should receive something.

The config-sync error is interesting. What might be easiest is to factory-reset the switch, delete the switch from the FortiGate. Reboot the FortiGate, the FortiSwitch will have rebooted already with the factory-reset. Then re-authorize the switch again.

Config sync errors can also be troubleshooted (troubleshot? :D) with the flcfgd.

diag debug console timestamp enable

diag debug app flcfgd -1

diag debug enable

and then run this on the FortiGate for some minutes, hoping to see some errors pointing to better reasons. I do not know the FortiSwitch well enough, but I guess some similar debug could run on the FortiSwitch as well.

Best regards,

Markus

[erussell]

Thanks for the ideas Markus.  I will give them a try.  I have already went through a factory reset on 1 of the switches.  In fact I did a factory reset full to completely wipe it and go back a version.  Still no joy.

For some reason the Fortigate is firewalling the connection through the Fortilink.  Cannot figure out why.

[Markus_M]

It should not, just the FortiSwitch manager should show the device as authorized (and in sync).

[erussell · Accepted solution]

I wanted to follow up, I was able to solve this.  I took a break from it all for a while and came back with a fresh set of eyes.  Turns out my Fortilink interface had an IP of 10.109.92.1 as I had mentioned.  The switches were all statically IP'd as 10.109.92.2/3/4.  The behavior I was mentioning above was the same even if I moved the switches to DHCP clients.

Anyway, I went from top to bottom on the config and someone had configured SSL VPN Clients to be addressed in 10.109.92.2-254!  Well that's not good is it?  As I had mentioned I inherited this entire setup.  So I moved my Fortinet interface to a new subnet and set my switches to DHCP client and we are all good!  Full access to CLI on the switches now, full physical topology showing, all configs are now in sync also.  

This one took a while to sort, largely due to me not being 100% familiar with Fortinet.  However I now have a fully functioning network consisting of 2 FGT using SD-WAN and HA, 3 switches, 6 AP.  Life is good!