Troubleshooting Tip: FortiSwitch showing offline after the upgrade

Issue:

It is noticed when FortiSwitch in FortiLink mode is upgraded, or possibly also noticed after reboot.

Behavior during the issue:

• FortiSwitch shows offline on the FortiGate.
• The client connected to the offline FortiSwitch can forward traffic.
• Able to SSH to the FortiSwitch via FortiGate.

Reason: 

The FortiSwitch time is not in sync with the current time. NTP is not reachable.

Observation:

• Connect to the FortiSwitch that is seen offline and run the following command:

show system ntp

config system ntp
    config ntpserver
        edit 1
            set server "208.91.112.61"
        next
        edit 2
            set server "208.91.112.63"
        next
    end
        set ntpsync enable
end

• As noticed in the above output, multiple NTP servers were added. However, none of these are reachable.
• If capture is collected on the FortiGate, the one-way traffic will be noticed for port 123 (NTP port no). To validate, run the following command:

diagnose sniffer packet any "port 123" 

• The debug will show that the traffic is denied from FortiGate. To verify, use the following command:

diagnose debug flow filter addr <ntp-server-Ip-address>
diagnose debug console timestamp enable
diagnose debug flow trace start 999
diagnose debug enable

If it is not possible to see the FortiSwitch on the FortiGate, try to manually add the serial numbers under WiFi & Switch Controller -> Managed Switch -> Create New -> FortiSwitch. 

Solution:

1. Either create a policy to have the NTP server reachable from FortiGate.

config firewall policy
    edit 1
        set name "all"
        set srcintf "MCLAG"   <-- FortiLink interface name.
        set dstintf "port1"   <-- FortiGate port toward the internet.
        set srcaddr "all"
        set dstaddr "all"
        set action accept
        set service "all"
        set nat enable
    next
end

Note: A more granular policy can be created to allow specific traffic. 

2. Set the NTP as the FortiLink interface by configuring the NTP server as 'local' or choose 'specify' and provide the IP address of the FortiLink IP. Verify FortiLink is on the listening interface.

In the CLI: 

config system ntp
    set ntpsync enable

    set server-mode enable
    set interface "fortilink"
end 

In the GUI:

Navigate to System -> Settings:

The option is available under the FortiLink interface -> Advanced -> NTP server.

The screenshot below is provided from the FortiLink interface (FortiGate).

Note: It is necessary to reboot the FortiSwitch after the NTP config changes.

From CLI, this config is available under the DHCP server:

show  full system dhcp server

    edit 15
        set dns-service local
        set ntp-service local   <--
        set default-gateway 1.1.1.1
        set netmask 255.255.255.0
        set interface "fortilinktest"
            config ip-range
                edit 1
                    set start-ip 1.1.1.2
                    set end-ip 1.1.1.254
                next
            end
        set vci-match enable
        set vci-string "FortiSwitch" "FortiExtender"
    next
end

When users upgrade the FortiGate firmware to v7.6.1 or above and observe that a FortiSwitch goes offline, it is necessary to check the settings on the FortiLink-enabled interface because the default FortiSwitch discovery method changes.
This can cause FortiSwitch units to appear offline if the LLDP settings were previously changed in FortiLink configuration.

To resolve the issue, enable 'lldp-reception' and 'lldp-transmission' to bring the FortiSwitches online. This is a FortiLink known issue (ID 1113304).

If Link-Layer Detection Protocol (LLDP) is set to disable/VDOM under the FortiLink interface, use the following commands to enable it. This will bring the FortiSwitches online:

config system interface
    edit <port>
        set lldp-reception enable
        set lldp-transmission enable
end

To check the line rate on a FortiSwitch, in the FortiSwitch CLI, the command below can be used.

diagnose switch physical-ports linerate

This command displays the transmit (TX) and receive (RX) packet and rate information for each port, making it possible to monitor traffic flow and identify potential issues like high traffic volume or broadcast storms.

Related articles:

Troubleshooting Tip: FortiSwitch Connection to FortiGate

Troubleshooting Tip: Fix FortiSwitch showing with the 'Offline' status